<PAGE>   1




                    U.S. SECURITIES AND EXCHANGE COMMISSION

                            WASHINGTON, D. C.  20549

                                  FORM 10-QSB

       x   QUARTERLY REPORT UNDER SECTION 13 OR 15(d) OF THE
     ----- SECURITIES EXCHANGE ACT OF 1934

     For the quarterly period ended March 31, 1997

           TRANSITION REPORT UNDER SECTION 13 OR 15(d) OF THE
     ----- SECURITIES EXCHANGE ACT OF 1934
 
     For the transition period from       to 
                                   -------  -------

                         Commission file number 0-13153

                               HABERSHAM BANCORP                            
-------------------------------------------------------------------------------
       (Exact name of small business issuer as specified in its charter)

              Georgia                                  58-1563165
-------------------------------------------------------------------------------
        (State or other jurisdiction of            (I.R.S. Employer
        incorporation or organization)             Identification No.)

      Highway 441 N., P.O. Box 1980, Cornelia, Georgia     30531              
-------------------------------------------------------------------------------
        (Address of principal executive offices)          (Zip code)

                                 (706) 778-1000                             
-------------------------------------------------------------------------------
                (Issuer's telephone number, including area code)

-------------------------------------------------------------------------------
              (Former name, former address and former fiscal year,
                         if changed since last report)

    Check whether the issuer: (1) filed all reports required to be filed by
Section 13 or 15(d) of the Exchange Act during the past 12 months (or for such
shorter period that the registrant was required to file such reports), and (2)
has been subject to such filing requirements for the past 90 days.

                              Yes   X    No 
                                  -----     -----

State the number of shares outstanding of each of the issuer's classes of
common equity, as of the latest practicable date:



    2,369,195 shares, common stock, $1.00 par value, as of March 31, 1997

<PAGE>   2




Item. 1   Financial Statements

HABERSHAM BANCORP AND SUBSIDIARIES
CONDENSED CONSOLIDATED BALANCE SHEETS
(Unaudited)(dollars in thousands)
<TABLE>
<CAPTION>
                                         MARCH 31, 1997  DECEMBER 31, 1996
<S>                                         <C>              <C>
ASSETS                                   
                                         



Cash and due from banks ..................  $ 11,020         $  8,419
Federal funds sold .......................     5,000            6,590
Investment securities available for sale
  (estimated cost of $28,669 at March 31,
  1997 and $31,544 at December 31, 1996)..    28,522           31,629
Investment securities held to maturity
  (estimated market value of $16,398 at
  March 31, 1997, and $18,821 at
  December 31, 1996) .....................    16,381           18,764

Loans held for sale ......................    19,921           23,299

Loans ....................................   193,522          207,568
  Less:  Unearned income .................       (32)             (32)
         Allowance for loan losses .......    (2,262)          (2,261)
                                            --------         -------- 
    Loans, net ...........................   191,228          205,275
                                            --------         --------

Intangible assets ........................     3,727            3,403
Other assets .............................    13,248           12,632
                                            --------         --------
    TOTAL ASSETS .........................  $289,047         $310,011
                                            ========         ========

LIABILITIES
Non-interest bearing deposits ............  $ 25,176         $ 23,192
Interest bearing deposits ................   206,406          201,170
Short-term borrowings ....................       780              502
Other borrowings .........................    19,927           47,909
Other liabilities ........................     8,611            9,569
                                            --------         --------
    TOTAL LIABILITIES ....................   260,900          282,342
                                            --------         --------

SHAREHOLDERS' EQUITY (Note 3)
Common stock, $1.00 par value,
 10,000,000 shares authorized;
  2,403,974 shares issued ................     2,404            2,404
Additional paid-in capital ...............     8,950            8,898
Retained earnings ........................    17,095           16,561
Unrealized (loss)gain on investment
  securities available for sale ..........       (96)              56
Treasury stock (at cost) .................      (206)            (250)
                                            --------         -------- 
    TOTAL SHAREHOLDERS' EQUITY ...........    28,147           27,669
                                            --------         --------
    TOTAL LIABILITIES AND SHAREHOLDERS'
      EQUITY .............................  $289,047         $310,011
                                            ========         ========
</TABLE>


 See notes to condensed consolidated financial statements.
                                       2
<PAGE>   3

HABERSHAM BANCORP AND SUBSIDIARIES
CONDENSED CONSOLIDATED STATEMENTS OF INCOME AND RETAINED EARNINGS
(dollars in thousands, except per share amounts)
For the (Unaudited) Three-Month Periods Ended March 31, 1997 and 1996

<TABLE>
<CAPTION>
                                            MARCH 31, 1997  MARCH 31, 1996
<S>                                           <C>              <C>
INTEREST INCOME
Loans ......................................  $   5,284        $   3,993
Investment securities ......................        609              856
Other ......................................        207              153
                                              ---------        ---------
   TOTAL INTEREST INCOME ...................      6,100            5,002

INTEREST EXPENSE
Deposits ...................................      2,598            2,214
Other ......................................        641               61
                                              ---------        ---------
   TOTAL INTEREST EXPENSE ..................      3,239            2,275

NET INTEREST INCOME ........................      2,861            2,727
Provision for loan losses ..................        123               90
                                              ---------        ---------

NET INTEREST INCOME AFTER
 PROVISION FOR LOAN LOSSES .................      2,738            2,637

Other Income ...............................      1,995              488

Other Expenses:
  Salary and employee benefits .............      2,110            1,420
  Other expense ............................      1,742            1,095
                                              ---------        ---------
    Total other expense ....................      3,852            2,515

INCOME BEFORE INCOME TAXES..................        881              610
Applicable income taxes  ...................        264              123
                                              ---------        ---------
NET INCOME .................................        617              487

Dividends ..................................         83                -
Retained earnings at beginning of period....     16,561           14,932
                                              ---------        ---------
Retained earnings at end of period .........  $  17,095        $  15,419
                                              =========        =========

Per Common and Common Equivalent Share:
NET INCOME .................................  $     .25        $     .20
                                              =========        =========


WEIGHTED AVERAGE NUMBER OF COMMON AND
COMMON EQUIVALENT SHARES OUTSTANDING .......  2,487,892        2,396,646
                                              =========        =========
</TABLE>




See notes to condensed consolidated financial statements.


                                       3
<PAGE>   4



HABERSHAM BANCORP AND SUBSIDIARIES
CONDENSED CONSOLIDATED STATEMENTS OF CASH FLOWS
(dollars in thousands)
For the (Unaudited) Three-Month Periods Ended March 31, 1997 and 1996



<TABLE>
<CAPTION>
                                                         MARCH 31,    MARCH 31,
CASH FLOWS FROM OPERATING ACTIVITIES:                      1997         1996   
<S>                                                     <C>          <C>       
  Net Income ........................................   $   617      $   487   
  Adjustments to reconcile net income to net cash                              
    (used in)provided by operating activities:                                 
      Provision for loan losses .....................       123           90   
      Net loss on sale of investment securities .....         4            -   
      Net gain on sale of loans .....................      (592)           -   
      Depreciation and amortization .................       268          212   
      Proceeds from sale of loans held for sale .....    24,980            -   
      Net (increase)decrease in loans held for sale .   (21,298)           -   
                                                                               
                                                                               
  Changes in assets and liabilities:                                           
      Decrease in other assets ......................      (255)        (203)  
      Decrease in other liabilities .................      (865)        (517)  
                                                         ------       ------   
  Net cash provided by operating activities .........     2,982           69   
                                                         ------       ------   
                                                                               
                                                                               
                                                                               
CASH FLOWS FROM INVESTING ACTIVITIES:                                          
 Investment securities available for sale:                                     
  Proceeds from maturity ............................     5,036        3,692   
  Proceeds from sale ................................     2,826            -   
  Purchases .........................................    (4,956)      (1,371)  
 Investment securities held to maturity:                                       
  Proceeds from maturity ............................     3,768        1,073   
  Purchases .........................................    (1,385)        (354)  
 Business acquisition - Advantage Insurers, Inc. ....      (380)           -   
                                                                               
 Loans:                                                                        
  Proceeds from sale ................................    21,453       (2,945)  
  Net decrease in loans .............................    (7,436)      (5,590)  
 (Increase) decrease in other real estate ...........       514          (23)  
 Net additions of premises and equipment ............      (832)        (487)  
                                                         ------       ------   
  Net cash provided by (used in)                                               
     investing activities ...........................    18,608       (6,005)  
                                                         ------       ------   
</TABLE>



                                      4

<PAGE>   5

<TABLE>
<S>                                                     <C>          <C>
CASH FLOWS FROM FINANCING ACTIVITIES:
  Net increase in deposits ..........................     7,220        8,317
  Net increase(decrease) in short-term borrowings ...       278          (49)
  Net increase(decrease) in other borrowings  .......   (28,091)          19
  Cash dividends paid ...............................       (83)           -
  Sale of treasury stock ............................        97           17
                                                         ------      -------
  Net cash (used in) provided by
     financing activities ...........................   (20,579)       8,304
                                                         ------      -------

Increase in cash and cash equivalents ...............     1,011        2,368

CASH AND CASH EQUIVALENTS: BEGINNING OF PERIOD ......    15,009       16,388
                                                        -------      -------
CASH AND CASH EQUIVALENTS: END OF PERIOD ............   $16,020      $18,756
                                                        =======      =======
</TABLE>

 See notes to condensed consolidated financial statements.

<TABLE>
 <S>                                                      <C>         <C>
 SUPPLEMENTAL DISCLOSURES OF CASH FLOW INFORMATION:
 Cash paid during the period for:
  Interest ............................................   $4,683      $1,968
  Income taxes ........................................      218         165

 SUPPLEMENTAL DISCLOSURE OF NON-CASH INVESTING ACTIVITIES:
 Other real estate acquired through loan foreclosures..   $  195      $  322
 Loan made to sell other real estate ..................        -         (14)
 Unrealized loss on investment securities
  available for sale, net .............................      152         105 
</TABLE>

HABERSHAM BANCORP AND SUBSIDIARIES
Notes to Condensed Consolidated Financial Statements
(Unaudited)

1.  Basis of Presentation

The condensed financial statements contained in this report are unaudited but
reflect all adjustments, consisting only of normal recurring accruals, which
are, in the opinion of management, necessary for a fair presentation of the
results of the interim periods reflected.  Certain information and footnote
disclosures normally included in financial statements prepared in accordance
with generally accepted accounting principles have been condensed or omitted
pursuant to applicable rules and regulations of the Securities and Exchange
Commission.  The results of operations for the interim periods reported herein
are not necessarily indicative of results to be expected for the full year.

The condensed consolidated financial statements included herein should be read
in conjunction with the Company's 1996 financial statements and notes thereto,
and the Independent Auditors' Report included in the Company's Annual Report on
Form 10-KSB for the fiscal year ended December 31, 1996.

Certain reclassifications have been made to the prior periods in order to
conform to classifications adopted in the current period.



                                      5

<PAGE>   6

2.  Accounting Policies

Reference is made to the accounting policies of the Company described in the
notes to consolidated financial statements contained in the Company's Annual
Report on Form 10-KSB for the fiscal year ended December 31, 1996.  The Company
has consistently followed those policies in preparing this report.

3. Recent Business Acquisition

Effective March 31, 1997, Advantage Insurers, Inc., a newly formed subsidiary
of Habersham Bank, acquired substantially all of the assets of Dillard-Scruggs
Insurance Services of Cornelia, Inc. d/b/a Cornelia Insurance Agency.
Advantage Insurers, Inc. will continue to offer a full line of property,
casualty and life insurance products.  This acquisition was  accounted for as a
purchase with a purchase price of $380,000.

Item. 2  Management's Discussion and Analysis of Financial Condition and
Results of Operations.

HABERSHAM BANCORP AND SUBSIDIARIES

Organization

    Habersham Bancorp owns all of the outstanding stock of Habersham Bank
("Habersham Bank"), Security State Bank ("Security Bank"), and The Advantage
Group, Inc. (collectively the "Company").   Habersham Bank owns all of the
outstanding stock of BancMortgage Financial Corp ("BancMortgage"), Appalachian
Travel Service, Inc. ("Appalachian") and Advantage Insurers, Inc. ("Advantage
Insurers").  The Advantage Group, Inc. is a nonbank subsidiary which engages in
the business of providing certain management consulting advice to depository
institutions.  BancMortgage was organized in 1996 as a full service mortgage
lending and servicing company located in the northern Atlanta Metropolitan
area.  Advantage Insurers, which began operations March 31, 1997,  offers a
full line of property, casualty and life insurance products.  The Advantage
Group, Inc., Appalachian and Advantage Insurers do not comprise a significant
portion of the financial position, results of operations or cash flows of the
Company for the quarter ended March 31, 1997.  Management's discussion and
analysis, which follows, relates primarily to Habersham Bank, Security Bank and
BancMortgage.


Material Changes in Financial Condition

    The Company's total assets decreased approximately $21 million during the
first quarter of 1997 primarily due to sales of mortgage loans of approximately
$46 million partially offset by new loan originations and decreases in
investment securities of approximately $5 million.  Funds received from the
sale of loans and investment security activities were used primarily to repay
borrowings from the Federal Home Loan Bank of approximately $28 million.

    At March 31, 1997, loans over 90 day days past due and nonaccrual loans
totaled $1,997,985 or 1.03% of gross outstanding loans, as compared to
$1,358,801 or .65% of gross outstanding loans at December 31, 1996.  This
increase is primarily due to the addition of two large credits totaling
approximately $500,000 to nonaccrual status.  Both of these loans are
collateralized by real estate with an appraised value in excess of loan value.
The balance of the allowance for loan losses is in accordance with the internal
calculation of the allowance for loan losses and accounts for factors such as
classified and past due loans as well as portfolio growth and diversification.
There were no significant writeoffs during the first quarter of 1997.




                                       6
<PAGE>   7

    The Company had impaired loans of $1,750,637 and $981,801 as of March 31,
1997 and December 31, 1996, respectively.  Impaired loans consist of loans in
nonaccrual status and are discussed in the paragraph above.  No allowance was
necessary for such loans.  The interest income recognized on such loans for the
three month periods ended March 31, 1997 and 1996 was not material.

    The Company's other real estate totaled $2,064,365 and $2,403,734 as of
March 31, 1997 and December 31, 1996, respectively.  This decrease is due to
sales of property of approximately $500,000 offset by additions to other real
estate of approximately $200,000.

Material Changes in Results of Operations

    Total interest income for the first quarter of 1997 increased $1,098,000 or
21.95%, when compared to the first quarter of 1996.  Interest income from loans
increased $1,291,000 or 32.33% due to the addition of approximately $527,000 of
income from loans originated by BancMortgage as well as increases in the loan
portfolios of Habersham Bank and Security Bank.  Loan yields decreased by .79%
to 9.32% for the first quarter of 1997 compared to yields of 10.11% for the
first quarter of 1996, due to changes in the composition of loans.

    Interest income from investment securities decreased $247,000 or 28.86% for
the first quarter of 1997 when compared to the first quarter of 1996.  The
decrease in interest income from investments was primarily due to the decrease
in Habersham Bank's investment security portfolio as a result of the maturity
and sale of securities used in funding the lending activities relating to
BancMortgage. Average yields on investment securities increased .34% to 6.29%
for the first quarter of 1997 compared to 5.95% for the first quarter of 1996.

    Total interest expense for the first quarter of 1997 increased $964,000 or
42.37%, when compared to the first quarter of 1996.  Interest expense on
deposits increased $384,000 or 17.34%, when compared to the first quarter of
1996. The increase in interest expense on deposit accounts was primarily due to
the addition of approximately $26 million of interest bearing deposits.
Consolidated average rates on deposits decreased 3.61% to 5.08% during the
first quarter of 1997 compared to the first quarter of 1996.

    Other interest expense increased $580,000 or 950.82% when compared to the
first quarter of 1996.  This increase reflects the increased borrowing at the
Federal Home Loan Bank to fund BancMortgage's lending activities.

    Net interest income increased approximately $964,000 or 42.37%, for the
first quarter of 1997 as a result of the items discussed above.

    The net interest margin of the Company, the spread between interest
income and interest expense, was 4.09% for the first quarter of 1997.  By
carefull management of deposit and loan growth and pricing, the Company
maintains its net interest margin.


    Other income increased $1.5 million or 308.81% for the first quarter of
1997 over the same period in 1996.  This increase was due to the addition of
approximately $592,000 of gain on sale of loans as well as increases in loan
fees of approximately $759,000 and the addition of approximately $278,000 of
revenue from Appalachian Travel Service, Inc., which was acquired during the
third quarter of 1996.  These increases were partically offset by decreases in
other income categories.




                                      7

<PAGE>   8
                                       
    Other expenses increased $1,337,000 or 53.16% for the first quarter of 1997
over the same period in 1996.  This increase was due to increased salary and
employee benefits of approximately $690,000 associated with fully staffing
BancMortgage operations at four locations and the addition of Appalachian
Travel Service, as well as personnel additions at Habersham Bank, Security
Bank and the Company. Overhead expenses, computer and other operating
expenses increased $98,000, $42,000 and $507,000, respectively, due to the
activities of BancMortgage and Appalachian Travel Service.

Liquidity and Capital Resources

    Habersham Bank's liquidity policy requires that the ratio of cash and
certain short-term investments to net withdrawable deposit accounts be at least
20% and Security Bank's liquidity policy requires a minimum of 25%.  The
following table lists the liquidity ratios for the Banks at March 31, 1997.

             Habersham Bank                      23.11%
             Security Bank                       26.48%
                                         

    Habersham Bank has established various sources of funds to be used in the
management of its liquidity position.  Federal Funds and repurchase agreement
guidelines have been provided by Compass Bank and The Bankers Bank.  Also
established is an borrowing arrangement for advances at the Federal Home Loan
Bank.


    At March 31, 1997, Habersham Bank and Security Bank were required to have
minimum Tier 1 and total capital ratios of 4% and 8%, respectively.
Additionally, the Banks are required to maintain a leverage ratio (Tier 1
Capital to average assets) of at least 4%.  The Banks' ratios at March 31, 1997
follows:

                               Habersham          Security
                                  Bank              Bank
          Tier 1                 10.44%            14.46%
          Total Capital          11.47%            15.71%                    
          Leverage                6.82%            12.00%



    Capital expenditures for the construction of new branches for Habersham
Bank in Cleveland, Georgia and Security Bank in Waleska, Georgia totaled
approximately $427,000 and $227,000 for the first quarter of 1997,
respectively.  While management believes that the current level of internal
capital is sufficient for current and foreseeable needs of the Company, capital
needs are continually evaluated by management.




                                       8
<PAGE>   9

Interest Rate Sensitivity


    The objective of asset and liability management is to measure and manage
both the level and volatility of earnings and capital based upon the impact of
changes in interest rates.  Habersham Bank uses interest rate shock tests as
well as traditional asset / liability modeling in order to maintain an optimum
match of maturities and interest rate sensitivity between loans, investment
securities and deposits.



    The interest rate sensitivity analysis below has a three month gap of
approximately $17.6 million (excess of earning assets to interest bearing
liabilities repricing within one year).

<TABLE>
<CAPTION>
                            YIELD/   DUE IN   DUE AFTER     DUE AFTER     DUE AFTER     DUE AFTER     TOTAL
                            RATE     THREE  THREE THROUGH  SIX THROUGH   ONE THROUGH       FIVE
                                     MONTHS   SIX MONTHS  TWELVE MONTHS   FIVE YEARS      YEARS
<S>                        <C>    <C>          <C>          <C>            <C>            <C>       <C>
EARNING ASSETS:                                             
 Federal funds sold        5.20%  $  5,000                                                          $   5,000
 Investment securities     6.29%     2,072     $   596      $ 3,337        $18,016        $20,882      44,903
 Loans                     9.32%   106,395      20,250       33,693         38,926         14,147     213,411
                           ----    -------      ------       ------         ------         ------     -------
  Total earning assets     8.71%   113,467      20,846       37,030         56,942         35,029     263,314

INTEREST BEARING LIABILITIES:
 Deposits
   Money Market and NOW    2.97%    41,931                                                             41,931
   Savings                 2.63%     7,780                                                              7,780
   Certificates of deposit 5.76%    38,961      33,399       52,880         31,455             00     156,695
 Borrowings                6.85%     7,207      10,500           00          3,000             00      20,707
                           ----     ------      ------       ------         ------         ------     -------
  Total interest bearing
   liabilities             5.35%    95,879      43,899       52,880         34,455             00     227,113

INTEREST RATE SPREAD:      4.09%
                           ====

Excess(deficiency) of earning
 assets over(to) interest bearing
 liabilities                      $ 17,588    $(23,053)    $(15,850)      $ 22,487       $ 35,029     $36,201
                                  ========    ========     ========       ========       ========     =======

Cumulative gap                    $ 17,588    $ (5,465)    $(21,315)     $   1,172       $ 36,201

Ratio of cumulative gap to total
 cumulative earning assets           15.50%     (4.07%)     (12.44%)           .51%

Ratio of earning assets to
 interest bearing liabilities       118.34%     47.49%       70.03%         165.26%
</TABLE>





                                       9
<PAGE>   10

                                    PART II
                               OTHER INFORMATION


Item 1.  Legal Proceedings.

                        none

Item 2.  Changes in securities.

                        none

Item 3.  Defaults upon senior securities.

                        none

Item 4.  Submission of matters to a vote of security holders.

                        none

Item 5.  Other information.

                        none

Item 6.  Exhibits and reports on Form 8-K.

                        (a) Exhibits:

                             27. Financial Data Schedule (For SEC Use Only)

                        (b)  none


                                   SIGNATURES


    In accordance with the requirements of the Exchange Act, the registrant has
caused this report to be signed on its behalf by the undersigned, thereunto
duly authorized.


                               HABERSHAM BANCORP
                                  (Registrant)




 Date  May 12, 1997                       /s/ David D. Stovall 
     -----------------                    ---------------------
                                            President and
                                            Chief Financial Officer
                                            (for the Registrant and as the
                                             Registrant's principal financial
                                             and accounting officer)





                                       10