<PAGE>   1

                    U.S. SECURITIES AND EXCHANGE COMMISSION

                            WASHINGTON, D. C.  20549

                                  FORM 10-QSB

     __x__ QUARTERLY REPORT UNDER SECTION 13 OR 15(d) OF THE
           SECURITIES EXCHANGE ACT OF 1934
     For the quarterly period ended September 30, 1996

     _____ TRANSITION REPORT UNDER SECTION 13 OR 15(d) OF THE
           SECURITIES EXCHANGE ACT OF 1934
     For the transition period from _______ to _________

                         Commission file number 0-13153

                               HABERSHAM BANCORP
-------------------------------------------------------------------------------
       (Exact name of small business issuer as specified in its charter)

     Georgia                                        58-1563165
-------------------------------------------------------------------------------
     (State or other jurisdiction of            (I.R.S. Employer
     incorporation or organization)             Identification No.)

Highway 441 N., P.O. Box 1980, Cornelia, Georgia        30531
-------------------------------------------------------------------------------
     (Address of principal executive offices)          (Zip code)

                                 (706) 778-1000
-------------------------------------------------------------------------------
                (Issuer's telephone number, including area code)


-------------------------------------------------------------------------------
              (Former name, former address and former fiscal year,
                         if changed since last report)

     Check whether the issuer: (1) filed all reports required to be filed by
Section 13 or 15(d) of the Exchange Act during the past 12 months (or for such
shorter period that the registrant was required to file such reports), and (2)
has been subject to such filing requirements for the past 90 days.
                                               Yes __x__  No _____


State the number of shares outstanding of each of the issuer's classes of
common equity, as of the latest practicable date:




   2,325,781 shares, common stock, $1.00 par value, as of September 30, 1996






                                  Page 1 of 12
<PAGE>   2



Item. 1   Financial Statements

HABERSHAM BANCORP
CONDENSED CONSOLIDATED BALANCE SHEETS
(dollars in thousands)
<TABLE>
<CAPTION>

                                          September 30, 1996  DECEMBER 31, 1995
ASSETS                                       (Unaudited)
<S>                                              <C>              <C>
Cash and due from banks .......................  $  8,285         $  6,598
Federal funds sold ............................     3,270            9,790
Investment securities available for sale
     (cost of $33,850 at September 30, 1996
     and $40,035 at December 31, 1995).........    33,705           40,313
Investment securities held to maturity
     (estimated market value of $19,043 at
     September 30, 1996 and $17,677 at
     December 31, 1995) .......................    19,017           17,498

Loans held for sale ...........................    20,275

Loans .........................................   203,413          145,434
     Less:  Unearned income ...................       (35)             (44)
            Allowance for loan losses .........    (2,337)          (2,336)
                                                ---------      -----------
     Loans, net ...............................   201,041          143,054
                                                ---------      -----------
Intangible assets .............................     3,501            3,397
Other assets ..................................    10,954            8,936
                                                ---------      -----------
     TOTAL ASSETS .............................  $300,048         $229,586
                                                =========      ===========
LIABILITIES
Non-interest bearing deposits .................  $ 20,015         $ 23,129
Interest bearing deposits .....................   192,700          171,956
Short-term borrowings .........................       866            1,597
Other borrowings ..............................    56,232            3,700
Other liabilities .............................     3,535            3,299
                                                ---------      -----------
     TOTAL LIABILITIES ........................   273,348          203,681
                                                ---------      -----------
SHAREHOLDERS' EQUITY (Note 3)
Common stock, $1.00 par value,
     10,000,000 shares authorized;
     2,403,974 shares issued ..................     2,404            2,404
Additional paid-in capital ....................     8,842            8,837
Retained earnings .............................    16,013           14,955
Unrealized gain (loss) on investment
     securities available for sale ............       (95)             185
Treasury stock (at cost) ......................      (464)            (476)
                                                ---------      -----------
     TOTAL SHAREHOLDERS' EQUITY ...............    26,700           25,905
                                                ---------      -----------
     TOTAL LIABILITIES AND SHAREHOLDERS'
     EQUITY ...................................  $300,048         $229,586
                                                =========      ===========
</TABLE>


     See notes to condensed consolidated financial statements.

                                       2


<PAGE>   3

HABERSHAM BANCORP
CONDENSED CONSOLIDATED STATEMENTS OF INCOME AND RETAINED EARNINGS
(dollars in thousands, except per share amounts)
For the (Unaudited) Three-Month Periods Ended September 30, 1996 and 1995
<TABLE>
<CAPTION>

                                          September 30, 1996  September 30, 1995
<S>                                                <C>              <C>
INTEREST INCOME
Loans (including fees) ........................    $ 5,524          $ 3,788
Investment securities .........................        746              861
Other .........................................         90              107
                                                   -------          -------
     TOTAL INTEREST INCOME ....................      6,360            4,756

INTEREST EXPENSE
Deposits ......................................      2,426            2,090
Other .........................................        547              116
                                                   -------          -------
     TOTAL INTEREST EXPENSE ...................      2,973            2,206

NET INTEREST INCOME ...........................      3,387            2,550
Provision for loan losses .....................         90               39
                                                   -------          -------
NET INTEREST INCOME AFTER
     PROVISION FOR LOAN LOSSES ................      3,297            2,511

Other Income ..................................        357              433

Other Expense .................................      3,185            2,196
                                                   -------          -------

INCOME BEFORE INCOME TAXES.....................        469              748
Applicable income taxes  ......................         91              188
                                                   -------          -------
NET INCOME ....................................        378              560

Dividends .....................................         70
Retained earnings at beginning of period.......     15,705           13,808
                                                   -------          -------
Retained earnings at end of period ............    $16,013          $14,368
                                                   =======          =======

Per share .....................................     $  .16           $  .24
                                                    ======           ======

WEIGHTED AVERAGE NUMBER OF COMMON AND
COMMON EQUIVALENT SHARES OUTSTANDING ..........  2,414,521        2,323,584
                                                 =========        =========
</TABLE>

See notes to condensed consolidated financial statements.


                                       3





<PAGE>   4



HABERSHAM BANCORP
CONDENSED CONSOLIDATED STATEMENTS OF INCOME AND RETAINED EARNINGS
(dollars in thousands, except per share amounts)
For the (Unaudited) Nine-Month Periods Ended September 30, 1996 and 1995
<TABLE>
<CAPTION>
                                          September 30, 1996  September 30, 1995
<S>                                             <C>             <C>
INTEREST INCOME
Loans (including fees) .....................    $14 175          $ 8,995
Investment securities ......................      2,376            2,445
Other ......................................        365              205
                                                -------          -------
   TOTAL INTEREST INCOME ...................     16,916           11,645

INTEREST EXPENSE
Deposits ...................................      6,906            4,938
Other ......................................        774              221
                                                -------          -------
   TOTAL INTEREST EXPENSE ..................      7,680            5,159

NET INTEREST INCOME ........................      9,236            6,486
Provision for loan losses ..................        270               53
                                                -------          -------
NET INTEREST INCOME AFTER
 PROVISION FOR LOAN LOSSES .................      8,966            6,433

Other Income ...............................      1,051              931

Other Expense ..............................      8,434            5,670
                                                -------          -------

INCOME BEFORE INCOME TAXES..................      1,583            1,694
Applicable income taxes  ...................        316              352
                                                -------          -------
NET INCOME .................................      1,267            1,342

Dividends ..................................        209               84


Retained earnings at beginning of period....     14,955           13,110
                                                -------          -------
Retained earnings at end of period .........    $16,013          $14,368
                                                =======          =======


Per share ..................................     $  .53           $  .69
                                                 ======           ======

WEIGHTED AVERAGE NUMBER OF COMMON AND
COMMON EQUIVALENT SHARES OUTSTANDING .......  2,407,574        1,947,105
                                              =========        =========
</TABLE>


See notes to condensed consolidated financial statements.

                                       4


<PAGE>   5
HABERSHAM BANCORP
CONDENSED CONSOLIDATED STATEMENTS OF CASH FLOWS
(dollars in thousands)
For the (Unaudited) Nine-Month Periods Ended September 30, 1996 and 1995

<TABLE>
<CAPTION>
                                                     SEPTEMBER 30, SEPTEMBER 30,
CASH FLOWS FROM OPERATING ACTIVITIES:                     1996           1995
<S>                                                     <C>           <C>

 Net Income ........................................    $ 1,267       $1,342
 Adjustments to reconcile net income to
     net cash provided by operating activities:
     Provision for loan losses .....................        270           53
     Net loss(gain) on sale of investment securities          8            5
     Net loss on other real estate owned ...........         31          100
     Depreciation ..................................        526          403
     Amortization of intangible assets .............        147           52
Changes in assets and liabilities:
     Increase in other assets ......................       (388)        (544)
     Increase (decrease) in other liabilities ......        260          883
                                                        -------      -------
 Net cash provided by operating activities .........      2,121        2,294
                                                        -------      -------

CASH FLOWS FROM INVESTING ACTIVITIES:
 Investment securities available for sale:
  Proceeds from maturity ............................     6,434        3,139
  Proceeds from sale ................................     3,130        2,830
  Purchases .........................................    (6,426)      (6,161)
 Investment securities held to maturity:
  Proceeds from maturity ............................     6,187        3,572
  Purchases .........................................    (4,668)      (4,336)
 Increase in Loans held for sale ....................   (20,275)
 Net increase in loans ..............................   (59,431)      (7,093)
 Decrease in other real estate ......................       663           81
 Net additions of premises and equipment ............    (1,807)        (202)
 Cash and cash equivalents acquired upon acquisition    
  of Security State Bank, net of cash payments ......                  3,219
                                                        -------      -------
  Net cash used by investing activities .............   (76,193)      (4,951)
                                                        -------      -------

CASH FLOWS FROM FINANCING ACTIVITIES:
  Net increase in deposits ..........................    17,630        9,913
  Net increase (decrease) in short-term borrowings ..      (730)          78
  Net increase in other borrowings .......... .......    52,531        2,200
  Sale of treasury stock ............................        17           41
  Cash dividends ....................................      (209)         (84)
                                                        -------      -------
  Net cash provided by financing activities .........    69,239       12,148
                                                        -------      -------

Increase (decrease) in cash and cash equivalents ....    (4,833)       9,491

CASH AND CASH EQUIVALENTS: BEGINNING OF PERIOD ......    16,388        5,471
                                                        -------      -------
CASH AND CASH EQUIVALENTS: END OF PERIOD ............   $11,555      $14,962
                                                        =======      =======
</TABLE>

                                       5

<PAGE>   6
See notes to condensed consolidated financial statements.

<TABLE>
<S>                                                     <C>          <C>
SUPPLEMENTAL DISCLOSURE OF NON-CASH INVESTING ACTIVITY:
OTHER REAL ESTATE ACQUIRED THROUGH LOAN FORECLOSURES..  $ 1,189      $   280
LOAN RELATED TO SALE OF OTHER REAL ESTATE ............       14           16
UNREALIZED GAIN (LOSS) ON INVESTMENT SECURITIES
  AVAILABLE FOR SALE .................................      280        1,498
</TABLE>

     


HABERSHAM BANCORP
Notes to Condensed Consolidated Financial Statements
(Unaudited)

1.  Basis of Presentation

The condensed financial statements contained in this report are unaudited but
reflect all adjustments, consisting only of normal recurring accruals, which
are, in the opinion of management, necessary for a fair statement of the results
of the interim periods reflected.  Certain information and footnote disclosures
normally included in financial statements prepared in accordance with generally
accepted accounting principles have been condensed or omitted pursuant to
applicable rules and regulations of the Securities and Exchange Commission.  The
results of operations for the interim periods reported herein are not
necessarily indicative of results to be expected for the full year.

The condensed consolidated financial statements included herein should be read
in conjunction with the Company's 1995 financial statements and notes thereto,
and the Independent Auditors' Report included in the Company's Annual Report on
Form 10-KSB for the fiscal year ended December 31, 1995.

2.  Accounting Policies

Reference is made to the accounting policies of the Company described in the
notes to consolidated financial statements contained in the Company's Annual
Report on Form 10-KSB for the fiscal year ended December 31, 1995.  The Company
has consistently followed those policies in preparing this report.

3.  Common Stock

Effective May 15, 1995, the Company declared a 5 for 1 split of its common stock
effected in the form of a 400% stock dividend.  In addition, effective April 15,
1995, the Company changed the par value of its common stock from $2.50 to $1.00
per share and increased the number of authorized shares of common stock to
10,000,000 shares.  All references to share and per share amounts have been
retroactively adjusted to reflect this split.  Also retroactively, $875,000 has
been charged to Additional Paid-In Capital and credited to Common Stock to
reflect the stock split and the change in par value.



                                       6

<PAGE>   7



4.  Earnings Per Share

Net income per share is computed based on the weighted average number of common
and common equivalent shares outstanding during the periods.

5.  Acquisition

Effective June 30, 1995, the Company consummated its acquisition of Security
Bancorp, Inc. ("Security") by agreeing to exchange 612,516 shares of its common
stock and $1,990,269 in cash for the outstanding shares of Security common stock
in a merger of Security with and into the Company.  The Company had previously
acquired 5% of the outstanding shares of Security.  The total purchase price
including expenses was approximately $9,257,000.  The merger was accounted for
as a purchase.  The following presents pro forma income statement information
for the nine month period ended September 30, 1995 assuming the merger had
occurred on January 1, 1995 (in thousands, except per share amounts):

<TABLE>
<CAPTION>
                                 Nine Months Ended
                                 September 30, 1995
                                 ------------------
<S>                                 <C>
Total revenues                      $14,723
Net income                          $ 1,423
Net income per common share         $   .61

</TABLE>

The pro forma financial information presented above does not purport to be
indicative of either the results of operations that would have occurred had
the acquisition taken place on January 1, 1995 or of future consolidated
results of operations.

6. New Subsidiaries

On January 2, 1996, Habersham Bank formed a new subsidiary, BancMortgage
Financial Corp. ("BancMortgage").  BancMortgage is a full service mortgage
lender and servicer located in the North Atlanta metropolitan area. BancMortgage
received approval from the Department of Banking and Finance of the State of
Georgia to begin business on January 26, 1996 and began operations on February
1, 1996.

Effective September 1, 1996, Habersham Bank acquired all of the outstanding
stock of Appalachian Travel Service, Inc., Cornelia, GA for $50,000 in cash. The
acquisition was accounted for as a purchase and no significant amount of
intangibles was recorded.

7. New Accounting Pronouncement

In October 1995, the Financial Accounting Standards Board issued Statement of
Financial Accounting Standards (SFAS) No. 123, "Accounting for Stock-Based
Compensation," which became be effective for the Company beginning January 1,
1996.  SFAS No. 123 requires expanded disclosures of stock-based compensation
arrangements with employees and encourage (but does not require) compensation

                                       7
<PAGE>   8


cost to be measured based on the fair value of the equity instrument awarded.
Companies are permitted, however, to continue to apply APB Opinion No. 25,
which recognizes compensation cost based on the intrinsic value of the equity
instrument awarded.  The Company will continue to apply APB Opinion No. 25 to
its stock based compensation awards to employees and will disclose the required
pro forma effect on net income and earnings per share in the 1996
Annual Report.

Item. 2  Management's Discussion and Analysis of Financial Condition and
           Results of Operations.

HABERSHAM BANCORP

Organization

     Habersham Bancorp owns all of the outstanding stock of Habersham Bank
("Habersham Bank"), Security State Bank ("Security Bank"), and The Advantage
Group, Inc. (collectively the "Company").  Habersham Bank owns all of the
outstanding stock of BancMortgage Financial Corp ("BancMortgage") and
Appalachian Travel Service, Inc., both of which are nonbank subsidiaries. The
Advantage Group, Inc. is a nonbank subsidiary which engages in the business of
providing certain management consulting advice to depository institutions.
BancMortgage was organized in 1996 as a full service mortgage lending and
servicing company located in the northern Atlanta Metropolitan area.  Effective
September 1, 1996, Habersham Bank acquired all of the outstanding stock of
Appalachian Travel Service, Inc., Cornelia, GA for $50,000 in cash.  The
acquisition was accounted for as a purchase and no significant amount of
intangibles was recorded.  The Advantage Group, Inc. and Appalachian Travel
Services, Inc., did not comprise a significant portion of the financial
position, results of operations or cash flows of the Company for the period
ended September 30, 1996.   Therefore, management's discussion and analysis,
which follows, relates primarily to the operations of Habersham Bank,
BancMortgage and Security Bank.

Material Changes in Financial Condition

     The Company's total assets increased approximately $70 million during the
first nine months of 1996 reflecting an increase in Loans of approximately $58
million or 40.5% and an increase in loans held for sale of approximately $20
million. BancMortgage originated or had commitments to fund approximately $178
million of loans during the first nine months of 1996.  These originations were
funded by an increase in deposits of approximately $18 million, an increase in
advances from the Federal Home Loan Bank of approximately $53 million, decreases
in investment securities of approximately $5 million and a decrease in federal
funds sold of approximately $7 million during the first nine months of 1996.

     At September 30, 1996, loans over 90 days past due and nonaccrual loans
totaled $1,318,008 or .59% of gross outstanding loans, as compared to $1,280,683
or .88% of gross outstanding loans at December 31, 1995.  The allowance for loan
losses recorded at September 30, 1996 is in accordance with the management's
internal calculation of the allowance for loan losses which

                                       8

<PAGE>   9

considers such factors as classified and past due loans as well as portfolio
growth and diversification.  There were no significant writeoffs during the
first nine months of 1996.

     The Company had impaired loans of $1,087,532 and $1,022,683 as of September
30, 1996 and December 31, 1995, respectively.  No allowance was necessary for
such loans.  The interest income recognized on such loans for the nine months
ended September 30, 1996 was not material.

     Effective May 15, 1995, the Company declared a 5 for 1 split of its common
stock effected in the form of a 400% stock dividend.  In addition, effective
April 15, 1995, the Company changed the par value of its common stock from $2.50
to $1.00 per share and increased the number of authorized shares of common stock
to 10,000,000 shares.  All references to share and per share amounts have been
retroactively adjusted to reflect this split.

     The Company consummated its acquisition of Security Bank as of July 1,
1995.  The acquisition was accounted for as a purchase and the operations of
Security are included in the Company's operations beginning July 1, 1995.  As of
July 1, 1995, Security Bank had total assets of approximately $46 million
consisting of approximately $33 million in loans, $5.7 million in investments,
$5.3 million in cash and cash equivalents and $2 million in other assets.
Liabilities of Security consisted of deposits of $39 million and other
liabilities of $1 million and equity of $5.8 million.

Material Changes in Results of Operations

     Total interest income for the third quarter of 1996 increased $1,604,000 or
33.7%, when compared to the third quarter of 1995.  Total interest income for
the first nine months of 1996 increased $5,271,000 or 45.3% when compared to the
first nine months of 1995.  Interest income and fees from loans increased
$1,736,000 or 45.8% and $5,180,000 or 57.6%, when compared to the third quarter
and the first nine months of 1995, respectively.  These increases are due to the
addition of interest income from Security Bank's loan portfolio and the addition
of approximately $85 million of loans secured by real estate in the Atlanta
Metropolitan area originated by BancMortgage.  Loan yields decreased by .22% to
9.83% for the third quarter of 1996 when compared to yields of 10.05% for the
third quarter of 1995.

     Interest income from investment securities decreased $115,000 or 13.4% for
the third quarter of 1996 when compared to the third quarter of 1995 primarily
due to sales or maturities of investment securities in Habersham Bank's
portfolio during the third quarter of 1996.  Interest income from investment
securities decreased $69,000 or 2.8% for the first nine months of 1996 when
compared to the first nine months of 1995 primarily due to reductions in
Habersham Bank's investment portfolio.   Average yields on investment securities
decreased .18% to 5.96% for the third quarter of 1996 when compared to 6.14% for
the third quarter of 1995.

     Total interest expense for the third quarter of 1996 increased $767,000 or
34.8%, when compared to the third quarter of 1995.  Total interest expense for
the first nine months of 1996 increased $2,521,000 or 48.9%, when compared


                                       9

<PAGE>   10


to the first nine months of 1995.  Interest expense on deposits increased
$336,000 or 16.1%, when compared to the third quarter of 1995 and $1,968,000 or
39.8% for the first nine months of 1996, when compared to the first nine months
of 1995. The increase in interest expense for the third quarter on deposit
accounts is primarily due to the increase of interest bearing accounts of
approximately $21 million.  The increase in interest expense for the first nine
month of 1996 is due to the addition of Security State Bank's deposit portfolio
at June 30, 1995, as well as increases in Habersham Bank's and Security State
Bank's deposit portfolio of approximately $18 million and $5 million,
respectively. Consolidated average rates on deposits also decreased during the
third quarter of 1996, when compared to the third quarter of 1995 as follows;
IMMA & NOW - from 3.41% to 3.33%; Savings - from 3.00% to 3.00%; Certificates
of Deposit and IRA - from 5.89% to 5.61%.

     Other interest expense increased approximately $431,000 or 371.6% for the
third quarter of 1996 when compared to the third quarter of 1995 and increased
approximately $553,000 or 250.2% for the first nine months of 1996, when
compared to the first nine months of 1995.  These increases were primarily due
to increases in Federal Home Loan Bank advances of approximately $53 million
during 1996.

     Net interest income increased approximately $837,000 or 32.8%, for the
third quarter of 1996 when compared to the third quarter of 1995 and increased
approximately $2,750,000 or 42.4%, for the first nine months of 1996 when
compared to the first nine months of 1995, as a result of the items discussed
above.

     Other income decreased $76,000 or 17.6% for the third quarter of 1996 and
increased $120,000 or 12.9% for the first nine months of 1996 when compared to
the same periods in 1995, respectively.  These changes were due to the addition
of other income of Security Bank offset by decreased service charge and
commission activity.

     Other expenses increased $989,000 or 45% for the third quarter of 1996 and
$2,764,000 or 48.7%, for the first nine months of 1996, when compared to the
same periods in 1995, respectively.  The increases for the third quarter were
due to the addition of approximately $488,000 of expenses related to Security
Bank and expenses of $669,000 related to BancMortgage operations.  The increases
for the first nine months of 1996 were primarily due to the addition of
approximately $1,405,000 of expenses related to Security State Bank and expenses
of $1,854,000 related to BancMortgage operations.

     The startup of BancMortgage operations has reduced consolidated net income
and net income per share by $667,328 and $.28, respectively, for the three
months ended September 30, 1996 and $968,560 and $.40, respectively for the nine
months ended September 30, 1996.


                                       10


<PAGE>   11



Liquidity and Capital Resources

     Habersham Bank's liquidity policy requires that the ratio of cash and
certain short-term investments to net withdrawable deposit accounts be at least
20% and Security Bank's liquidity policy requires a minimum of 30%.  The
following table lists the liquidity ratios for the Banks at September 30, 1996.

     Habersham Bank           21.66%
     Security State Bank      28.66%

     At September 30, 1996, Habersham Bank and Security Bank were required to
have minimum Tier 1 and total capital ratios of 4% and 8%, respectively.
Additional, the Banks are required to maintain a leverage ratio (Tier 1 capital
to total assets) of at least 4%.  The Banks' ratios at that date follow:

<TABLE>
<CAPTION>
                          Habersham        Security State
                             Bank              Bank
     <S>                    <C>               <C>
     Tier 1                 10.08%            13.97%
     Total Capital          11.08%            15.22%
     Leverage                7.22%            11.55%

</TABLE>

     While management believes that the current level of internal capital is
sufficient for current and foreseeable needs of the Company, capital needs are
continually evaluated by management.


                                    PART II
                               OTHER INFORMATION

Item 1.  Legal Proceedings.
                     none

Item 2.  Changes in securities.
                     none

Item 3.  Defaults upon senior securities.
                     none

Item 4.  Submission of matters to a vote of security holders.
                     none

Item 5.  Other information.
                     none

Item 6.  Exhibits and reports on Form 8-K.
                     27   Financial Data Schedule (for SEC use only).





                                       11


<PAGE>   12





                                   SIGNATURES


     In accordance with the requirements of the Exchange Act, the registrant
has caused this report to be signed on its behalf by the undersigned, thereunto
duly authorized.


                                          HABERSHAM BANCORP
                                          -----------------
                                            (Registrant)




Date November 13, 1996           /S/ David D. Stovall
    -------------------          -----------------------------------------
                                 President and
                                 Chief Financial Officer
                                 (for the Registrant and as the
                                  Registrant's principal financial and
                                  accounting officer)










                                       12