<PAGE>   1

                                 EXHIBIT 10.4
                                      
              HABERSHAM BANCORP 1996 INCENTIVE STOCK OPTION PLAN






<PAGE>   2

                                      
                              TABLE OF CONTENTS
<TABLE>
<CAPTION>
                                                                                                                     Page
<S>                                                                                                                     <C>
SECTION 1  DEFINITIONS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
         1.1     "Board of Directors" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
         1.2     "Code" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
         1.3     "Committee"  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
         1.4     "Company"  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
         1.5     "Disability" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
         1.6     "Fair Market Value"  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
         1.7     "Option" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
         1.8     "Option Agreement" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
         1.9     "Over 10% Owner" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
         1.10    "Participant"  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
         1.11    "Plan" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   2
         1.12    "Stock"  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   2
         1.13    "Subsidiary" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   2
         1.14    "Termination of Employment"  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   2

SECTION 2  THE PLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   2
         2.1     Purpose of the Plan  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   2
         2.2     Stock Subject to the Plan  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   2
         2.3     Administration of the Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   2
         2.4     Eligibility and Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   3

SECTION 3  TERMS OF OPTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   3
         3.1     Terms and Conditions of Options  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   3
         3.2     Option Price . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   4
         3.3     Option Term  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   4
         3.4     Payment  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   4
         3.5     Conditions to the Exercise of an Option  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   5
         3.6     Special Provisions for Certain Substitute Options  . . . . . . . . . . . . . . . . . . . . . . . . .   5
         3.7     Treatment of Awards Upon Termination of Employment . . . . . . . . . . . . . . . . . . . . . . . . .   5

SECTION 4  GENERAL PROVISIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   5
         4.1     Changes in Capitalization; Merger; Liquidation.  . . . . . . . . . . . . . . . . . . . . . . . . . .   5
         4.2     Right to Terminate Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   6
         4.3     Restrictions on Delivery and Sale of Shares; Legends.  . . . . . . . . . . . . . . . . . . . . . . .   6
         4.4     Nonalienation of Benefits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   7
         4.5     Termination and Amendment of the Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   7
         4.6     Choice of Law. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   7
         4.7     Effective Date of Plan.  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   7
         4.8     Term.  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   7
</TABLE>





                                       2

<PAGE>   3


                               HABERSHAM BANCORP
                        1996 INCENTIVE STOCK OPTION PLAN

                             SECTION 1  DEFINITIONS

         Whenever used herein, the masculine pronoun shall be deemed to include
the feminine, and the singular to include the plural, unless the context
clearly indicates otherwise, and the following capitalized words and phrases
are used herein with the meaning thereafter ascribed:

         1.1     "Board of Directors" means the board of directors of the
Company.

         1.2     "Code" means the Internal Revenue Code of 1986, as amended,
and the rules and regulations promulgated thereunder.

         1.3     "Committee" means a committee composed of members of the Board
of Directors appointed by the Board of Directors to administer the Plan.

         1.4     "Company" means Habersham Bancorp, a Georgia corporation.

         1.5     "Disability" means that condition described in Code Section
22(e)(3), as amended from time to time.  In the event of a dispute, the
determination of Disability shall be made by the Committee and shall be
supported by advice of a physician competent in the area to which such
Disability relates.

         1.6     "Fair Market Value" refers to the fair market value of the
Stock as of a particular date, as determined in good faith by the Company;
provided that the Committee uses a reasonable valuation method in determining
fair market value in accordance with Code Section 422 and the regulations
issued thereunder and provided further that fair market value shall be
determined without regard to any restriction other than a restriction which, by
its terms, will never lapse.

         1.7     "Option" means an incentive stock option, but includes any
option granted under the Plan that is treated as a nonqualified stock option
pursuant to Section 2.4.

         1.8     "Option Agreement" means a written agreement between the
Company and a Participant or other documentation evidencing an award of an
Option.

         1.9     "Over 10% Owner" means an individual who at any time an
incentive stock option is granted owns Stock possessing more than ten percent
(10%) of the total combined voting power of the Company or one of its 
Subsidiaries, determined applying the attribution rules of Code Section 424(d).

         1.10    "Participant" means an individual who receives an Option
hereunder.


                                      3

<PAGE>   4

         1.11    "Plan" means the Habersham Bancorp 1996 Incentive Stock Option
Plan.

         1.12    "Stock" means the Company's common stock $1.00 par value.

         1.13    "Subsidiary" means, with respect to the Company, any
subsidiary corporation within the meaning of Code Section 424(f).

         1.14    "Termination of Employment" means the termination of the
employee-employer relationship between a Participant and the Company and its
Subsidiaries, regardless of the fact that severance or similar payments are
made to the Participant for any reason, including, but not by way of
limitation, a termination by resignation, discharge, death, Disability or
retirement.  The Committee shall, in its absolute discretion, determine the
effect of all matters and questions relating to a Termination of Employment,
including, but not by way of limitation, the question of whether a leave of
absence constitutes a Termination of Employment.

                              SECTION 2  THE PLAN

         2.1     Purpose of the Plan.  The Plan is intended to (a) provide
incentive to certain officers and key employees of the Company and its
Subsidiaries to stimulate their efforts toward the continued success of the
Company and to operate and manage the business in a manner that will provide
for the long-term growth and profitability of the Company; (b) encourage stock
ownership by certain officers and key employees by providing them with a means
to acquire a proprietary interest in the Company by acquiring shares of Stock;
and (c) provide a means of obtaining and rewarding key personnel.

         2.2     Stock Subject to the Plan.  Subject to adjustment in
accordance with Section 4.1, 250,000 shares of Stock (the "Maximum Plan
Shares") are hereby reserved exclusively for issuance pursuant to Options.  At
no time shall the Company have outstanding Options and shares of Stock issued
in respect of Options under the Plan in excess of the Maximum Plan Shares,
determined in accordance with Rule 16b-3(a)(1) as promulgated under the
Securities Exchange Act of 1934, as amended from time to time.

         2.3     Administration of the Plan.  The Plan shall be administered by
the Committee.  The Committee shall have full authority in its discretion to
determine the persons to whom Options shall be granted and the terms and
provisions of Options, subject to the Plan.  Subject to the provisions of the
Plan, the Committee shall have full and conclusive authority to interpret the
Plan; to prescribe, amend and rescind rules and regulations relating to the
Plan; to determine the terms and provisions of the respective Option Agreements
and to make all other determinations necessary or advisable for the proper
administration of the Plan.  The Committee's determinations under the Plan need
not be uniform and may be made by it selectively among persons who receive, or
are eligible to receive, awards under the Plan (whether or not such persons are
similarly situated).  The Committee's decisions shall be final and binding on
all Participants.





                                       4

<PAGE>   5



         The Committee shall consist of two or more directors, each of whom is
not, during the one year prior to service as a member of the Committee, granted
or awarded equity securities of the Company or an affiliate pursuant to the
Plan or any other plan of the issuer or an affiliate, except as may be
permitted under Rule 16b-3(c)(2)(i) promulgated under the Securities Exchange
Act of 1934.

         In addition to any other rights of indemnification that they may have
as directors of the Company or as members of the Committee, the members of the
Committee shall be indemnified by the Company against the reasonable expenses,
including attorneys' fees actually and necessarily incurred in connection with
the defense of any action, suit or proceeding, or in connection with any appeal
thereon, to which they or any of them may be a party by reason of any action
taken or failure to act under or in connection with the Plan or any Option
granted thereunder, and against all amounts paid by them in settlement thereof
(provided the settlement is approved by independent legal counsel selected by
the Company) or paid by them in satisfaction of a judgment in any action, suit
or proceeding, except in relation to matters as to which it shall be adjudged
in the action, suit or proceeding that the Committee member is liable for
negligence or misconduct in the performance of his or her duties; provided that
within 60 days after the institution of any action, suit or proceeding a
Committee member shall in writing offer the Company the opportunity, at its own
expense, to handle and defend the same.

         2.4     Eligibility and Limits.  Options may be granted only to
employees of the Company or any Subsidiary.  In addition, in the event the
aggregate Fair Market Value (determined as of the option grant date) of stock
subject to such Options (under all plans of the Company and Subsidiaries) that
first become exercisable during any calendar year by an amount that exceeds
$100,000, then such Options in excess of the limitation shall not be incentive
stock options and, to the extent such Options were granted pursuant to this
Plan, they shall be treated as nonqualified stock options.

                          SECTION 3  TERMS OF OPTIONS

         3.1     Terms and Conditions of Options.

         (a)     The number of shares of Stock as to which an Option shall be
granted shall be determined by the Committee in its sole discretion, subject to
the provisions of Section 2.2 as to the total number of shares available for
grants under the Plan.  Notwithstanding the preceding, to the extent required
under Code Section 162(m) and regulations thereunder for compensation to be
treated as qualified performance-based compensation, the maximum number of
shares of Stock with respect to which Options may be granted during any single
fiscal year of the Company to any employee shall not exceed 100,000.

         (b)     The date an Option is granted shall be the date on which the
Committee has approved the terms and conditions of the Option and has
determined the recipient of the 


                                      5

<PAGE>   6
Option and the number of shares covered by the Option and has taken all such
other action necessary to complete the grant of the Option.

         (c)     Options shall not be transferable or assignable except by will
or by the laws of descent and distribution and shall be exercisable, during the
Participant's lifetime, only by the Participant; in the event of the Disability
of the Participant, by the legal representative of the Participant; or in the
event of the death of the Participant, by the personal representative of the
Participant's estate or if no personal representative has been appointed, by
the successor in interest determined under the Participant's will.

         (d)     Each Option granted under the Plan shall be evidenced by an
Option Agreement in such form and containing such terms, conditions and
restrictions as the Committee may determine is appropriate, which shall
specify, at a minimum, the number of shares of Stock subject to the grant, the
option price and the option term.  Each Option Agreement shall be subject to
the terms of the Plan and any provision contained in the Option Agreement that
is inconsistent with the Plan shall be null and void.

         3.2     Option Price.  Subject to adjustment in accordance with
Section 4.2 and the other provisions of this Section 3, the exercise price (the
"Exercise Price") per share of Stock purchasable under any Option shall be as
set forth in the applicable Option Agreement and shall in no event be less than
the Fair Market Value of a share of Stock.  With respect to an Option grant to
an Over 10% Owner, the Exercise Price shall in no event be less than 110% of
Fair Market Value of a share of Stock on the date the Option is granted.

         3.3     Option Term.  The term of an Option shall be as specified in
the applicable Option Agreement and shall expire no later than five (5) years
from the date of grant.  In addition, the applicable Option Agreement shall
provide that, in the event of a Termination of Employment, the then unexpired
portion of the Option shall terminate no later than the date of Termination of
Employment; provided, however, that in the case of a Participant whose
Termination of Employment is due to death, the then unexpired portion of the
Option shall terminate no later than one (1) year from the date of Termination
of Employment.

         3.4     Payment.  Payment for all shares of Stock purchased pursuant
to exercise of an Option shall be made in any form or manner authorized by the
Committee in the Option Agreement or by amendment thereto, including, but not
limited to, cash or, if the Option Agreement provides, (i) by delivery to the
Company of a number of shares of Stock which have been owned by the holder for
at least six (6) months prior to the date of exercise having an aggregate Fair
Market Value of not less than the product of the Exercise Price multiplied by
the number of shares the Participant intends to purchase upon exercise of the
Option on the date of delivery; or (ii) in a cashless exercise through a
broker.  The holder of an Option, as such, shall have none of the rights of a
stockholder.

         3.5     Conditions to the Exercise of an Option.  Each Option granted
under the Plan shall be exercisable by whom, at such time or times, or upon the
occurrence of such event or





                                       6

<PAGE>   7


events, and in such amounts, as the Committee shall specify in the Option
Agreement; provided, however, except as otherwise provided by the Plan, that
subsequent to the grant of an Option, the Committee, at any time before its
expiration or cancellation, may accelerate the time or times at which such
Option may be exercised in whole or in part and may permit the Participant or
any other designated person to exercise the Option, or any portion thereof, for
all or part of the remaining Option term notwithstanding any provision of the
Option Agreement to the contrary.

         3.6     Special Provisions for Certain Substitute Options.
Notwithstanding anything to the contrary in this Section 3, any Option issued
in substitution for an Option previously issued by another entity, which
substitution occurs in connection with a transaction to which Code Section
424(a) is applicable, may provide for an exercise price computed in accordance
with such Code Section and the regulations thereunder and may contain such
other terms and conditions as the Committee may prescribe to cause such
substitute Option to contain as nearly as possible the same terms and
conditions (including the applicable vesting and termination provisions) as
those contained in the previously issued Option being replaced thereby.

         3.7     Treatment of Awards Upon Termination of Employment.  Any award
under this Plan to a Participant who suffers a Termination of Employment may be
cancelled, accelerated, paid or continued, as provided in the applicable Option
Agreement or, in the absence of such provision, as the Committee may determine
thereafter.  The portion of any award exercisable in the event of continuation
or the amount of any payment due under a continued award may be adjusted by the
Committee to reflect the Participant's period of service from the date of grant
through the date of the Participant's Termination of Employment or such other
factors as the Committee determines are relevant to its decision to adjust the
award.

                         SECTION 4  GENERAL PROVISIONS

         4.1     Changes in Capitalization; Merger; Liquidation.

         (a)      The number of shares of Stock reserved for the grant of
Options; the number of shares of Stock reserved for issuance upon the exercise
of each outstanding Option; and the Exercise Price of each outstanding Option
shall be proportionately adjusted for any increase or decrease in the number of
issued shares of Stock resulting from a subdivision or combination of shares or
the payment of an ordinary stock dividend in shares of such Stock to holders of
outstanding shares of such Stock or any other increase or decrease in the
number of shares of such Stock outstanding effected without receipt of
consideration by the Company to the extent that Participant's proportionate
interest shall be maintained as before the occurrence of the event.

         (b)     In the event of a merger, consolidation, extraordinary
dividend, reorganization or other change in the capital structure of the
Company or tender offer for shares of Stock, the Committee may make such
adjustments with respect to Options and take such other action as it deems
necessary or appropriate to reflect or in anticipation of such merger,
consolidation,





                                       7

<PAGE>   8


reorganization or tender offer, including, without limitation, the substitution
of new options, the acceleration of Options, the early expiration of Options,
the removal of restrictions on outstanding Options or the cash-out of the
Options, in cash or in kind, based upon the Fair Market Value of the Stock
determined as of a date within thirty (30) days immediately prior to the
transaction, all as may be provided in the applicable Option Agreement or, if
not expressly addressed therein, as the Committee subsequently may determine in
the event of any such merger, consolidation, extraordinary dividend,
reorganization, other change or tender offer.

         (c)     The existence of the Plan and the Options granted pursuant to
the Plan shall not affect in any way the right or power of the Company to make
or authorize any adjustment, reclassification, reorganization or other change
in its capital or business structure, any merger or consolidation of the
Company, any issue of debt or equity securities having preferences or
priorities as to the Stock or the rights thereof, the dissolution or
liquidation of the Company, any sale or transfer of all or any part of its
business or assets, or any other corporate act or proceeding.  Any adjustment
pursuant to this Section 4.1 may provide, in the Committee's discretion, for
the elimination without payment therefor of any fractional shares that might
otherwise become subject to any Option.

         4.2     Right to Terminate Services.  Nothing in the Plan or in any
Option shall confer upon any Participant the right to continue as an employee,
officer, consultant or a member of the Board of Directors of the Company or any
of its affiliates or affect the right of the Company or any of its affiliates
to terminate the Participant's relationship with the Company or any of its
affiliates at any time.

         4.3     Restrictions on Delivery and Sale of Shares; Legends.  Each
Option is subject to the condition that if at any time the Committee, in its
discretion, shall determine that the listing, registration or qualification of
the shares covered by such Option upon any securities exchange or under any
state or federal law is necessary or desirable as a condition of or in
connection with the granting of such Option or the purchase or delivery of
shares thereunder, the delivery of any or all shares pursuant to such Option
may be withheld unless and until such listing, registration or qualification
shall have been effected.  If a registration statement is not in effect under
the Securities Act of 1933 or any applicable state securities laws with respect
to the shares of Stock purchasable or otherwise deliverable under Options then
outstanding, the Committee may require, as a condition of exercise of any
Option or as a condition to any other delivery of such Stock pursuant to an
Option, that the Participant or other recipient of an Option represent, in
writing, that the shares received pursuant to the Option are being acquired for
investment and not with a view to distribution and agree that the shares will
not be disposed of except pursuant to an effective registration statement,
unless the Company shall have received an opinion of counsel that such
disposition is exempt from such requirement under the Securities Act of 1933
and any applicable state securities laws.  The Company may include on
certificates representing shares delivered pursuant to an Option such legends
referring to the foregoing representations or restrictions or any other
applicable restrictions on resale as the Company, in its discretion, shall deem
appropriate.





                                       8


<PAGE>   9


         4.4     Nonalienation of Benefits.  Other than as specifically
provided with regard to the death of a Participant, no benefit under the Plan
shall be subject in any manner to anticipation, alienation, sale, transfer,
assignment, pledge, encumbrance or charge; and any attempt to do so shall be
void.  No such benefit shall, prior to receipt by the Participant, be in any
manner liable for or subject to the debts, contracts, liabilities, engagements
or torts of the Participant.

         4.5     Termination and Amendment of the Plan.  The Board of Directors
at any time may amend or terminate the Plan without stockholder approval;
provided, however, that the Board of Directors may condition any amendment on
the approval of stockholders of the Company if such approval is necessary or
advisable with respect to tax, securities or other applicable laws.  No such
termination or amendment without the consent of the holder of an Option shall
adversely affect the rights of the Participant under such Option.

         4.6     Choice of Law.  The laws of the State of Georgia shall govern
the Plan, to the extent not preempted by federal law.

         4.7     Effective Date of Plan.  The Plan shall become effective upon
the date the Plan is approved by the Board of Directors of the Company;
provided, however, that the grant of any Option prior to stockholder approval
shall be conditioned upon receipt of subsequent stockholder approval within
twelve months of the Plan's effective date.

         4.8     Term.  Unless terminated by the Board of Directors as of any
earlier date, the Plan shall expire on the tenth anniversary of the Plan's
effective date.



                                   HABERSHAM BANCORP

                                   By: /s/ David D. Stovall                     
                                      -----------------------------------------

                                   Title: President and Chief Executive Officer
                                         --------------------------------------
ATTEST:

/s/ Edward D. Ariail      
--------------------------
Secretary
         [CORPORATE SEAL]








                                       9