<PAGE>   1

                                                                    EXHIBIT 10.5



                           MORTGAGE BANKING AGREEMENT

      TIES MORTGAGE BANKING AGREEMENT (the "Agreement") is entered into as of
the 2nd day of January, 1996, by and among HABERSHAM BANCORP ("Habersham"), a
Georgia corporation, HABERSHAM BANK (the "Bank"), a bank organized under the
laws of the State of Georgia and a wholly-owned subsidiary of Habersham,
BANCMORTGAGE FINANCIAL CORP. ("BancMortgage"), a Georgia corporation to be a
wholly-owned subsidiary of the Bank, ROBERT S. CANNON ("Cannon") and ANTHONY L.
WATTS ("Watts").

                                   WITNESSETH

      WHEREAS, the parties hereto believe it in their respective best interests
to organize a mortgage banking subsidiary of the Bank;

      NOW, THEREFORE, in consideration of the premises and the mutual agreements
of the parties to this Agreement, and for other good and valuable consideration,
the parties hereto agree as follows:

      1. Capital Structure.

         (a) The Bank shall organize a subsidiary Georgia corporation to be
      known as "BancMortgage Financial Corp." BancMortgage shall have 10,000,000
      authorized shares, of which 200,000 shares shall initially be issued to
      the Bank at $10 per share for an aggregate of $2,000,000.

         (b) The Bank's initial investment is intended to be a maximum
      investment, and any amounts in addition to this initial investment shall
      be on such terms and conditions as the parties agree. In the event the
      parties agree it is necessary or desirable for the Bank to invest more
      than $2,000,000 in BancMortgage, the parties also agree that all of the
      terms of this Agreement shall be subject to renegotiation.

         (c) No additional shares of BancMortgage shall be issued to any entity
      or person other than Habersham or the Bank without first offering such
      shares to Cannon and Watts on the same terms and conditions as to any
      third party.

      2. Directors and Officers.

         (a) The Board of Directors (the "Board") of BancMortgage shall consist
      of five members unless otherwise agreed by the parties hereto.

         (b) The initial Board of BancMortgage shall consist of Edward D.
      Ariail, Robert S. Cannon, D. Michael Sleeth, David D. Stovall and Anthony
      L. Watts. The



                                       37
<PAGE>   2

         initial Executive Committee of the Board and all Board members shall be
         chosen by unanimous agreement among Robert S. Cannon, David D. Stovall
         and Anthony L. Watts.

         (c) The initial officers of BancMortgage shall be as follows: David D.
      Stovall as Chairman of the Board, Anthony L. Watts as President and D.
      Michael Sleeth as Secretary.

         (d) It is the intention of the parties that BancMortgage shall
      cooperate with the Bank relative to operations, but that BancMortgage
      shall have operating autonomy.

      3. Certain Compensation Terms.

         (a) Each of Cannon and Watts:

             (i)   Shall receive an annual base salary of $150,000 paid pursuant
          to the normal payroll practices of the Bank.

             (H)   Shall receive use of a late model luxury automobile,
          gasoline, insurance and repairs. Cannon and Watts shall reimburse
          BancMortgage for their personal use of such vehicle.

             (iii) Shall receive dues, service fees and similar periodic charges
          for their country club memberships (Cherokee Town and Country Club for
          Watts and Country Club of the South for Cannon), and any service clubs
          (such as Rotary International).

             (iv)  Shall be entitled to participate on the same basis as any
          employee of the Bank in the Bank's employee benefit plans.

             (v)   Shall receive term life insurance at least equal to two times
          his base salary to be provided to beneficiaries named by Cannon and
          Watts, respectively.

         (b) Cannon and Watts shall receive 25 % and 25 %, respectively, of
      BancMortgage's net income before taxes. In determining net income before
      taxes for this purpose, any loss carryforwards shall not be deducted, and
      the determination shall be made in accordance with the business plan (the
      "Business Plan") attached hereto and made a part hereof by reference.

         (c) Sleeth shall be employed approximately half-time by BancMortgage
      and approximately half-time by Habersham, with BancMortgage being charged
      for one-half (1/2) of his cash compensation.

         (d) In the future, if the parties agree to use servicing,
      securitization or similar financial techniques to create value not
      otherwise reflected in BancMortgage's net operating income, such value
      shall also be divided as set forth in Section 3(b).



                                       38
<PAGE>   3

      (e) Determination of BancMortgage's net income before taxes shall be
      made as of 11/30 of each calendar year, and the Bank shall make the income
      distribution to Watts and Cannon on or before 12/31 of each calendar year.
      In the event of a buy-out as described in Section 5 or the termination of
      this Agreement, net income before taxes shall be calculated and paid at
      the time of the buy-out or the termination.

4.    Right of First Refusal to Fund Loans, Agreement to Provide Warehouse Ling,
      Performance Criteria.

      (a) The Bank shall provide a warehouse line of credit to BancMortgage in
sufficient amounts to meet the reasonable and necessary needs of BancMortgage's
operation at an adjustable rate agreed upon between the parties or if the
parties cannot agree, at the lower of the Federal Home Loan Bank monthly or
daily adjustable rate.

      (b) The Bank shall have a right of first refusal to fund all loans
(construction, mortgage and other) originated by BancMortgage, such right to be
exercised in a commercially reasonable time period taking into account
BancMortgage's need to fund and quote rates for mortgage loans.

      (c) The Bank and BancMortgage shall create underwriting criteria, policies
and procedures, and all obligations of the Bank to fund or warehouse hereunder
shall be in accordance with such criteria.

      (d) The Bank's obligation to fund and the financial performance criteria
shall be at least to the levels set forth in the Business Plan.

5.    Buy-Out Arrangements.

      (a) Subject to Section 5(d), if the Bank or Habersham should be acquired,
Cannon and Watts shall have the right to acquire BancMortgage at a price equal
to its then book value at the time of acquisition, determined in accordance with
generally accepted accounting principles applicable to the mortgage industry.

      (b) Subject to Section 5(d), (i) if the Bank receives a bona fide offer
(including an offer from Cannon/Watts) for the acquisition of BancMortgage which
it wishes to accept, the Bank shall afford Cannon and Watts a right of first
refusal to acquire BancMortgage for the same price and on the same terms as are
offered by the proposed purchaser; or (ii) if Cannon and Watts agree that the
Bank should accept the offer to acquire BancMortgage, then the total purchase
price paid for BancMortgage shall be distributed to the Bank and to Cannon/Watts
as follows: the total purchase price shall be reduced by the amount of the
Bank's total investment in BancMortgage (its original investment of $2,000,000
plus any additional investment made by the Bank), and 50% of the resulting
amount shall be paid to the Bank, and 25% of the resulting amount shall be paid
to each of Cannon and Watts.



                                       39
<PAGE>   4

      (c) Subject to Section 5(d), if the Bank should decide to discontinue the
business engaged in by BancMortgage, Cannon and Watts shall have a right of
first refusal to acquire BancMortgage for a price equal to its then book value
at the time of exercise, determined in accordance with generally accepted
accounting principles applicable to the mortgage industry.

      (d) In the event one of the foregoing rights is triggered, the Bank shall
offer to sell BancMortgage to both Cannon and Watts, or to either Cannon or
Watts, as may be agreed upon by Cannon and Watts. If Cannon and Watts cannot
agree, the right of first refusal shall terminate. Cannon and Watts must
exercise their right to purchase within ninety (90) days after notice from the
Bank. They shall have up to an additional nine (9) months to close during which
time the Bank shall continue to perform its obligations hereunder. This
nine-month time period is to give Cannon/Watts sufficient time to conclude
negotiations with another partner.

      6. Habersham Stock Options.

      (a) Subject to approval by the Board of Directors of Habersham, Cannon and
Watts shall each be granted a one-time fully vested stock option to purchase
15,000 shares of Habersham common stock at an exercise price equal to the median
price between the bid and asked prices of such stock on the date of grant,
exercisable in whole or in part for a five year period from date of grant, all
in accordance with the terms of Habersham's Incentive Stock Option Plan.

      (b) Cannon and Watts shall each be granted such other options to purchase
shares of Habersham common stock in accordance with such performance criteria as
the parties may from time to time establish.

      7. Interim Operations.

      (a) The parties recognize that it is to their mutual advantage to commence
BancMortgage's business as soon as practically possible and that Cannon/Watts
have already expended sums and shall continue to expend sums in reliance on
these negotiations.

      (b) Accordingly, BancMortgage agrees to promptly reimburse Cannon/Watts
for the reasonable and necessary expenses they have incurred since 12/10/95 to
create BancMortgage, begin hiring employees, obtain space and otherwise to enter
into this Agreement. Further, commencing 1/2/96, BancMortgage shall promptly pay
all such expenses directly or advance the funds to pay such expenses.

      8. Good Faith, Authority. Each of the parties hereto agrees to negotiate
in good faith as necessary and to consummate the transactions contemplated
herein. Each party represents to the other that it has authority to enter into
this Agreement and to pay the consideration provided herein, that it is able to
complete the transactions contemplated hereby, and that as of the date hereof it
does not know of any impediments (legal or otherwise) to the consummation of the
transactions as contemplated hereby.



                                       40
<PAGE>   5

      9.  Cooperation. Each of the parties hereto agrees to cooperate in
obtaining all necessary regulatory approvals for the transactions contemplated
herein.

      10. Confidentiality. The parties hereto shall each maintain the
confidentiality of all of the information received from the other party and
shall use such information only for the purposes contemplated by this Agreement
and for no other purpose. If the transactions contemplated in this Agreement are
not consummated for any reason, the parties shall promptly return to each other
all documents and other written information received from the other party and
shall not retain any copies or summaries thereof. This Section 10 shall survive
the termination of this Agreement.

      11. Public Announcements. Each party agrees to consult with each other
prior to any public announcement relating to this transaction and shall mutually
approve the timing, the content and dissemination of any public announcement
except to the extent that such party is not reasonably able to consult in a
timely manner with, or obtain the approval of, the other party in situations in
which disclosures are required by applicable law.

      12. Time of the Essence. Time is of the essence of this Agreement and each
and an of its provisions.

      13. Laws Governing. This Agreement shall be governed by and construed
pursuant to the laws of the State of Georgia.

      14. Modification and Waiver. Any term or condition of this Agreement may
be waived at any time by the party hereto which is entitled to the benefit of
such term or condition. Any waiver on one occasion shall not be deemed to be a
waiver of the same or of any other term or condition on any future occasion.
This Agreement may be modified or amended only by a writing signed by all of the
parties hereto.

      15. Counterparts and Headings. This Agreement may be executed
simultaneously in any number of counterparts, each of which shall be deemed an
original, but all of which shall constitute one and the same instrument. The
headings set out in this Agreement are for convenience of reference only and
shall not be deemed to be a part of this Agreement.



                                       41
<PAGE>   6

      IN WITNESS WHEREOF, Cannon and Watts have executed this Agreement, and
Habersham, the Bank and BancMortgage have caused this Agreement to be executed
by their respective appropriate officers, under seal, all as of the date and
year first above written.

HABERSHAM BANCORP                         BANCMORTGAGE FINANCIAL CORP.

By:/s/ David D. Stovall                   By:/s/ David D. Stovall
   ------------------------------            -----------------------------------
   David D. Stovall                          David D. Stovall
   President                                 Chairman of the Board

[CORPORATE SEAL]                          [CORPORATE SEAL]

                                          /s/ Robert S. Cannon            (SEAL)
                                          --------------------------------
HABERSHAM BANK                            Robert S. Cannon

By:/s/ David D. Stovall                   /s/ Anthony L. Watts            (SEAL)
   ------------------------------         --------------------------------
   David D. Stovall                       Anthony L. Watts
   President

[BANK SEAL]


                                       42