<PAGE>   1

                                  EXHIBIT 10.6


                                OPTION AGREEMENT



THIS OPTION AGREEMENT (the "Agreement") is made and entered into as of the 11th
day of March, 1998, by and between each shareholder (each, a "Shareholder" and
collectively, the "Shareholders") whose names are set forth on the signature
page(s) to this Agreement and Habersham Bancorp (the "Company"), a Georgia
corporation.

WHEREAS, the shareholders are the record and beneficial owners of an aggregate
of 7,000 shares (the "Shares") of common stock (the "Common Stock") of Empire
Bank Corp. ("Empire"), a Georgia corporation, in the individual amounts set
forth on the signature page(s) to this Agreement; and

WHEREAS, the Company wishes to acquire an irrevocable option (the "Option") to
purchase the Shares from the Shareholders on the terms and conditions set forth
in this Agreement.

                                    ARTICLE I

1.1  Option to Purchase Shares. The Shareholders hereby grant to the Company the
     irrevocable Option to purchase all of the Shares, together with, at the
     Company's election, any additional shares of Common Stock which may be
     acquired by the Shareholders prior to the Closing (as defined in Section
     1.3 below), for a purchase price of one and one-half times the book value
     per share on February 28, 1998 (the "Purchase Price").

1.2  Option Price. In exchange for the Option, the Company has paid the
     Shareholders an aggregate of $100,000 (the "Option Price"). The Option
     Price will not be refundable unless the Closing does not occur due to
     Section 1.8(b), in which event the Option Price will be promptly refunded
     to the Company. At the Closing, the Option Price will be credited toward
     the aggregate Purchase Price.

1.3  Exercise of Option. The Option may be exercised by the Company in whole
     (but not in part) at any time. In the event the Company wishes to exercise
     the Option, the Company shall send a written notice to the Shareholder
     Representative (as defined on the signature page(s) to this Agreement)
     specifying the place, date and time (but not earlier than five business
     days nor later than ten business days from the date such notice is given)
     for the Closing of such purchase (the "Closing").

1.4  Termination of Option. Unless exercised before such time, this Option shall
     terminate upon the earlier of: (a) December 31, 1998 or (b) purchase of all
     the Shares by the Company.

1.5  Option Adjustment. In the event that Empire institutes any change in the
     Common Stock by reason of stock dividends, stock splits, mergers,
     recapitalizations, combinations, conversions, exchanges of shares or the
     like, the number and kind of Shares subject to this Agreement and the price
     to be

<PAGE>   2

     paid for such Shares shall be appropriately adjusted to reflect such
     changes made in the Common Stock.

1.6  Closing Deliveries by Shareholders. At the Closing, the Shareholders will
     deliver certificates representing the number of Shares being purchased in
     proper form for transfer.

1.7  Closing Deliveries by the Company. At the Closing, the Company will deliver
     to the Shareholders a certified or cashier's check in an amount equal to
     the Purchase Price multiplied by the number of Shares being sold.

1.8  Conditions to Closing. The respective obligations of each party to effect
     the Closing will be subject to the following conditions:

     (a)  all approvals shall have been received (and any waiting period,
          including extensions, shall have expired) which are applicable to the
          consummation of the purchase of the Shares under federal and state
          laws and regulations applicable to financial institutions; and

     (b)  no preliminary or permanent injunction or other order by any federal
          or state court of competent jurisdiction that makes illegal or
          otherwise prevents the consummation of the purchase of the Shares
          shall have been issued and remain in effect.

                                   ARTICLE II

2.1  Proposed Empire Transaction.


     (a)  The Company wishes to acquire all of the outstanding common stock of
          Empire (the "Empire Transaction"). The proposed Empire Transaction is
          expected to involve a merger between Empire and the Company, with the
          Company surviving the Merger. The purchase price would be $13.432
          million, payable in Common Stock of the Company at a per share value
          which is equivalent to the Company's 20-day average NASDAQ trading
          price at the date of closing of the Empire Transaction.


     (b)  If Empire is not interested in pursuing the proposed Empire
          Transaction, the Company will so advise the Shareholder
          Representative. In the event the Shareholders locate other Empire
          shareholders who wish to sell their shares to the Company, the Company
          will purchase such shares for $350 per share, payable at the Closing.

     (c)  If the Shareholders arrange for the Company's purchase of an
          additional 1,000 shares of Empire Common Stock, the Company will pay
          the Shareholders a bonus equal to $50,000. If the Shareholders arrange
          for the Company's purchase of an additional 2,000 shares of Empire
          Common Stock, the Company will pay the Shareholders a second bonus
          equal to $50,000 (for a total of $100,000).

     (d)  The Company will not, directly or indirectly, contact any Empire
          shareholders unless and until (i) this Agreement is executed and (ii)
          Empire has indicated its desire not to pursue the Empire Transaction.

<PAGE>   3




2.2  Inconsistent Action. During the term of this Agreement, no Shareholder will
     (a) sell, transfer, pledge, hypothecate or otherwise dispose of or encumber
     such Shareholder's Shares or make any agreement or commitment to do any of
     the foregoing, other than upon exercise of the Option, or (b) take any
     action which would have the effect of preventing or disabling such
     Shareholder from fully performing such Shareholders' obligations under this
     Agreement; provided, however, that nothing in this Section 2.2 shall
     prevent the Shareholders from voting the Shares in favor of the Empire
     Transaction, and the Shareholders have the affirmative obligation to do so.


                                   ARTICLE III

3.1  Title and Authority. Each Shareholder represents and warrants that such
     Shareholder has good, valid and marketable title to such Shareholder's
     Shares, free and clear of all liabilities, claims, liens, options, proxies,
     charges and encumbrances of any kind whatsoever. Each Shareholder
     represents and warrants that such Shareholder has full power, authority and
     individual capacity to enter into this Agreement.

3.2  Acquisition of Title. Each Shareholder represents and warrants that upon
     exercise of the Option, the Company will acquire good, valid and marketable
     title to such Shareholder's Shares, free and clear of all liabilities,
     claims, liens, options, proxies, charges and encumbrances of any kind
     whatsoever.


                                   ARTICLE IV

4.1  No-Shop. During the term of this Agreement, no Shareholder will directly or
     indirectly make, solicit, initiate or encourage proposals or offers from
     any party other than the Company and its affiliates relating to any
     recapitalization, merger, consolidation or acquisition or purchase of
     substantially all of the assets of, or any equity interest in, Empire. Each
     Shareholder will immediately cease and cause to be terminated all such
     contracts or negotiations with third parties, if any. No Shareholder will
     directly or indirectly participate in any negotiations regarding, furnish
     to any other person any information with respect to, assist or participate
     in or facilitate in any other manner any effort or attempt by any other
     person to do or seek any of the foregoing, unless such Shareholder shall be
     advised in writing by counsel that such Shareholder's fiduciary duties, in
     the capacity as a Director of Empire, under applicable law require such
     Shareholder to consider an unsolicited proposal with respect thereto. The
     Shareholder Representative shall promptly notify the Company of any such
     proposal or offer.

4.2  Public Disclosure. No party hereto shall make any public announcement with
     regard to the transactions contemplated by this Agreement without the prior
     approval of the Company or the Shareholder Representative, as applicable,
     except as required by law.


                                    ARTICLE V

5.1  Expenses. The Shareholders and the Company will each pay their own expenses
     incurred in connection with the enforcement or consummation of this
     Agreement.

<PAGE>   4

5.2  Successors. This Agreement will be binding upon and inure to the benefit of
     and be enforced by the parties hereto and their respective heirs,
     executors, representatives, successors and assigns.

5.3  Entire Agreement. This Agreement contains the entire understanding of the
     parties hereto and supersedes all prior agreements and understanding
     between the parties with respect to its subject matter. This Agreement may
     be amended only by a written instrument duly executed by the parties
     hereto.


5.4  Assignment. This Agreement may not be assigned by any Shareholder, and may
     not be assigned by the Company to anyone other than to an affiliate of the
     Company.

5.5  Counterparts. This Agreement may be executed in two or more counterparts,
     each of which shall be an original, but all of which together will
     constitute one and the same Agreement.

5.6  Notices. All notices, requests, claims, demands and other communications
     hereunder shall be in writing and shall be given (and shall be deemed to
     have been duly given if so given) by actual delivery, overnight courier, by
     cable, facsimile transmission, telegram or telex, or by mail (registered or
     certified mail, postage prepaid, return receipt requested) to the
     respective parties at the addresses on the signature page(s) hereto.

5.7  Governing Law. This Agreement shall be governed by and construed and
     enforced in accordance with the laws of the State of Georgia, without
     regard to its principles of conflicts of law.

5.8  Severability. The invalidity or unenforceability of any provision or
     provisions of this Agreement shall not affect the validity or
     enforceability of any other provision of this Agreement, which will remain
     in full force and effect.

5.9  Remedies. Each Shareholder acknowledges that performance of such
     Shareholder's obligations pursuant to this Agreement is of vital importance
     to the Company and that damages are an inadequate remedy for breach of a
     Shareholder's obligations represented hereby and, accordingly, each
     Shareholder agrees that an appropriate, but not exclusive, remedy for
     failure to fulfill such obligations is that of specific performance.

5.10 Further Assurances. From time to time at or after the Closing, at the
     Company's request and without further consideration, each Shareholder will
     execute and deliver to the Company such documents and will take such action
     as the Company may reasonably request in order to consummate more
     effectively the transactions contemplated by this Agreement and to vest in
     the Company good, valid and marketable title to the Shares, including, but
     not limited to, using its best efforts to cause Empire's transfer agent to
     transfer the Shares sold by the Shareholders on the books of Empire.

<PAGE>   5

IN WITNESS WHEREOF, the Company and the Shareholders have caused this Agreement
to be duly executed as of the day and year first above written.


HABERSHAM BANCORP




By: /s/ David D. Stovall
    ------------------------
    President
    Highway 441 North
    Cornelia, Georgia  30531




SHAREHOLDERS



/s/ *Philip E. Manley                  4,202 Shares
----------------------------
308 W. Dame Avenue
Homerville, Georgia  31634



/s/ W. M. Manley                       1,232 Shares
----------------------------
301 Peagler Street
Homerville, Georgia  31634


    Various Others                     1,566 Shares



/s/ Philip E. Manley
----------------------------
* Shareholder Representative