<PAGE>   1
                                                                   EXHIBIT 10.20


                                 LOAN AGREEMENT


         THIS LOAN AGREEMENT ("Agreement"), made and entered into this 15th day
of December, 1997, by and between TRUST ONE BANK, a Tennessee state banking
corporation with its principal office at 2171 Judicial Drive, Suite #101,
Germantown, Tennessee 38138 ("Lender") and BACK YARD BURGERS, INC., a Delaware
corporation, located at 2678 Colony Park Drive, Memphis, Tennessee 38118
("Borrower").

                                   WITNESSETH:

         WHEREAS, Borrower has requested that Lender extend financial
accommodations to it; and

         WHEREAS, Lender has agreed to extend financial accommodations to
Borrower in the form of a $460,000.00 loan, to be made in accordance with, and
subject to, the terms and conditions set forth below;

         NOW, THEREFORE, in consideration of the terms and conditions contained
herein, and of any loan or other financial accommodation heretofore, now or
hereafter made by Lender to or on behalf of Borrower, the parties hereto hereby
agree as follows:


SECTION I:  AMOUNTS AND TERMS OF THE LOAN:

Lender agrees, based upon the terms and conditions of this Agreement, as well as
the various documents to be executed in connection herewith and the
representations, warranties, covenants and undertakings of the Borrower herein,
to make a secured loan to the Borrower in the amount of $460,000.00 plus
interest, as herein provided.


A.  THE NOTE AND PAYMENT OF INTEREST AND PRINCIPAL:

As evidence for the indebtedness hereunder, the Borrower will execute a
promissory note (the "Note") in the principal sum of $460,000.00, with interest
fixed at nine and three-quarters percent (9.75%) per annum, payable in
fifty-nine (59) monthly principal and interest installments of $6,050.31 each,
based upon a ten (10) year amortization, and one final payment of all accrued
interest and any outstanding principal due under the Note. Interest will be
computed on the basis of actual days elapsed over a 360-day year. In case of any
default with respect to the Note, the Note shall bear interest at a default rate
equal to the highest interest rate allowed by applicable law.

The indebtedness may be pre-paid in whole or in part at Borrower's election at
any time and from time to time without penalty or premium. Pre-payments shall,
at Lender's option, first be applied to any accrued and unpaid interest, with
the remainder being credited to principal. Partial prepayments of principal
shall not have the effect of suspending or deferring any monthly payments due on
the Note, and the same shall continue to be due and payable on each due date
subsequent to such partial payment of principal but shall operate to effect full
payment of the principal at an earlier date.

B.  PLACE OF PAYMENTS:

All payment of principal and interest on the indebtedness evidenced by the Note
and other documents and all payments of fees required hereunder shall be made to
the Lender at 2171 Judicial Drive, Suite #101, Germantown, Tennessee 38138.

C.  PURPOSE OF LOAN:

The loan proceeds shall be applied towards the purchase of the real property
known as 2110 West Street, Germantown, Tennessee and described in attached
Exhibit "A" (the "Land").





<PAGE>   2



SECTION II:  CONDITIONS OF LENDING CONDITIONS PRECEDENT TO MAKING LOAN:

The obligation of the Lender to make the loan or any advances thereunder is
subject to the condition precedent that the Borrower shall have delivered to the
Lender such documents as the Lender shall require including, but not limited to,
the following:

         (a)      A properly executed promissory note in the form attached
                  hereto as Exhibit "B";

         (b)      A title commitment issued by Chicago Title Insurance Company
                  insuring the Lender's first deed of trust on the Land.

         (c)      A properly executed first deed of trust and security agreement
                  (the "Deed of Trust") on the Land and in the personal property
                  thereon in favor of Lender, in the form attached hereto as
                  Exhibit "C";

         (d)      A survey of the Land in standard ALTA form;

         (e)      Properly executed U.C.C. financing statements;

         (f)      Affidavit of Borrower as to the status of the title to the
                  Land in the form attached hereto as Exhibit "D";

         (g)      An environmental survey of the Land, satisfactory to Lender;

         (h)      A certificate of insurance on the Land in a form and with a
                  company satisfactory to the Lender, with Lender named as the
                  mortgage holder; and

         (i)      Payment of a loan commitment fee of $1,150.00.


SECTION III:  COLLATERAL

All advances to Borrower and all obligations of Borrower pursuant to this
Agreement shall be secured by the Deed of Trust.


SECTION IV:   REPRESENTATIONS AND WARRANTIES

Borrower represents and warrants that:

         (a)      Borrower is a Delaware corporation, authorized and doing
                  business in the State of Tennessee and the sole owner of the
                  Land. Borrower is not a party to any contract or other
                  agreement which would prevent it from executing this Agreement
                  or limit his ability to carry out the terms of this Agreement;

         (b)      The Land is free and clear of any and all liens or
                  encumbrances of whatever kind or character.

         (c)      There is no action, suit or proceeding at law or in equity or
                  by or before any governmental instrumentality or other agency
                  now pending, or to the knowledge of the Borrower, threatened
                  against or affecting Borrower or the Land, or any properties
                  or rights of Borrower, which if, adversely determined, would
                  materially affect the financial or any other condition of
                  Borrower or the Land;

         (d)      Borrower is not a party to any agreement or instrument nor is
                  he subject to any charter or other restrictions adversely
                  affecting its business, properties or assets, operations or
                  conditions (financial or otherwise);

         (e)      Borrower is not in default in the performance, observance or
                  fulfillment of any of his obligations, covenants or conditions
                  contained in any agreement or instrument; and

         (f)      The financial statements of Borrower heretofore delivered to
                  the Lender, are true and correct in all respects and no
                  materially adverse change has occurred in the financial
                  conditions reflected therein since the respective dates
                  thereof.


                                      - 2 -

<PAGE>   3


SECTION V:  AFFIRMATIVE COVENANTS OF BORROWER:

Borrower covenants and agrees that from the date hereof and until payment in
full of the principal and interest on the indebtedness evidenced by the Note and
other documents, unless the Lender shall otherwise consent in writing, such
consent to be at the discretion of the Lender, Borrower will:

         (a)      Keep the Land and other Collateral free and clear of all
                  encumbrances, liens, mortgages, security interests and
                  secondary financing, except for matters approved in writing by
                  the Lender, and the Borrower shall not, without the prior
                  written consent of the Lender, sell, transfer or convey all or
                  any part of his interest in the Land or any portion thereof.

         (b)      Not lease any portion of the Land to any person or entity.

         (c)      Notify Lender immediately of any change of address of the
                  Borrower.

         (d)      Pay all of his indebtedness and obligations promptly in
                  accordance with normal terms and practices and pay and
                  discharge or cause to be paid and discharged promptly all
                  taxes, assessments, and governmental charges or levies imposed
                  upon him, the Land or any of the other Collateral, before the
                  same shall become in default, as well as all lawful claims for
                  labor, materials, and supplies which otherwise, if unpaid
                  might become a lien or charge upon such Land or other
                  Collateral or any part thereof, unless such amounts are being
                  properly protested by the Borrower.

         (e)      Provide Lender with quarterly 10Q reports within ten (10) days
                  of their filing, as well as copies of annual audited financial
                  statements within ten (10) days of their preparation.

         (f)      Furnish such other information regarding the operation,
                  business affairs, and financial condition of the Borrower, as
                  the Lender may reasonably request.

         (g)      Maintain its operating accounts with Lender.

         (h)      Maintain a debt service ratio of at least 2:0 to 1:0. (This
                  convenant shall be measured quarterly and shall be calculated
                  as net income, plus depreciation and interest, divided by
                  total banks and leasing debt service requirements.)


SECTION VI:  NEGATIVE COVENANTS

Borrower covenants and agrees that from the date hereof and until payment in
full of the principal of and interest on the indebtedness evidenced by the Note
and other documents, unless the Lender shall otherwise consent in writing, such
consent to be at the discretion of the Lender, Borrower will not:

         (a)      Incur any additional debt, except in the ordinary course of
                  business;

         (b)      Make any loans to others; or

         (c)      Guarantee any indebtedness.

SECTION VII: EVENTS OF DEFAULT

An "Event of Default" of this Agreement shall exist if, at any time, any of the
following shall occur and such occurrence shall continue for ten (10) days after
receipt of notice by Borrower from Lender specifying such default:

         (a)      The Borrower defaults in the time a payment of any installment
                  is due under the terms of the Note;


                                      - 3 -

<PAGE>   4



         (b)      The Borrower defaults with respect to any other agreement with
                  Lender to which he is a party or with respect to any other
                  indebtedness to Lender when due or the performance of any
                  obligation incurred in connection with any indebtedness for
                  borrowed money, if the effect of such default is to accelerate
                  the maturity of such indebtedness, or if the effect of such
                  default is to permit the holder thereof to cause such
                  indebtedness to come due prior to its stated maturity;

         (c)      Any representation or warranty made by Borrower, herein, or in
                  any report, certificate, financial statement or other writing
                  furnished in connection with or pursuant to this Agreement
                  shall prove to be false, misleading or incomplete in any
                  material respect on the date of which made;

         (d)      The Borrower defaults on the performance or observance of any
                  of the covenants, terms or conditions contained herein;

         (e)      Any judgment shall be entered against Borrower which Lender
                  reasonably believes could, in the reasonable judgment of
                  Lender and its counsel, substantially impair the ability of
                  Borrower to perform any of his obligations contained in this
                  Agreement or any of the loan documents;

         (f)      If there is any material adverse change in the property or
                  financial condition of the Borrower; and

         (g)      Borrower shall file for protection under any bankruptcy or
                  state receivership law or shall be placed into bankruptcy or
                  receivership by any outside creditor, and such petition is not
                  dismissed in 60 days.


SECTION VIII:  REMEDIES UPON AN EVENT OF DEFAULT

After notice as required herein to Borrower from Lender of an Event of Default
and failure of Borrower to cure the default within the notice period, Lender
shall have the right to:

         (a)      Accelerate the balance due under the Note;

         (b)      Exercise all rights under the terms of the Note evidencing the
                  indebtedness, the Deed of Trust and any other loan documents;
                  and

         (c)      After any acceleration, as provided above, Lender shall have,
                  in addition to the rights and remedies given it by this
                  Agreement, the Note, the Deed of Trust and any other loan
                  documents, all those rights and remedies allowed by all
                  applicable law including, without limitation, the Uniform
                  Commercial Code as enacted in the State of Tennessee.

The enumeration of Lender's rights and remedies set forth in this Agreement is
not intended to be exhaustive and the exercise by Lender of any right or remedy
and shall not preclude the exercise of any other rights or remedies, all of
which shall be cumulative, and shall be in addition to any other right or remedy
given hereunder, or under any other agreement between Borrower or Lender or
which may now or hereafter exist in law or in equity or by suit or otherwise. No
delay or failure to take action on the part of Lender in exercising any right,
power or privilege shall operate as a waiver thereof, nor shall any single or
partial exercise of any such right, power or privilege preclude other or further
exercise thereof or the exercise of any other right, power or privilege or shall
be construed to be a waiver of any Event of Default. No course of dealing
between Borrower and Lender or its agents or employees, except as may be
otherwise waived in writing by Lender, shall be effective to change, modify or
discharge any provision of this Agreement or to constitute a waiver of any Event
of Default. Lender shall not, under any circumstances or in any event
whatsoever, have any liability for any error, omission or delay of any kind
occurring in the liquidation of the collateral or for any damages resulting
therefrom.

All the remedies provided in this agreement, and any of the other documents
executed pursuant to the terms of this agreement, at law or in equity are
cumulative; and Lender, at its option, may exercise such remedies concurrently
or sequentially in such order as it may choose.




                                      - 4 -

<PAGE>   5


SECTION IX:  MISCELLANEOUS.

         (a)      Borrower covenants, warrants and represents to Lender that all
                  representations and warranties of Borrower contained in this
                  Agreement and any other loan document shall be true at the
                  time of Borrower's execution of this Agreement and the other
                  loan documents and shall survive the execution, delivery and
                  acceptance thereof by the parties thereto and the closing of
                  the transactions described therein or related thereto.

         (b)      This Agreement shall remain in full force and effect until all
                  obligations of Borrower to Lender have been paid in full.

         (c)      The provisions of this Agreement, Deed of Trust, other loan
                  documents and the Note from time to time may be amended,
                  modified, or waived, with the consent of the Lender, if such
                  amendment, modification, or waiver is in writing and signed by
                  the party to be charged with the amendment, modification, or
                  waiver.

         (d)      All notices, requests and demands to or upon the respective
                  parties hereto shall be deemed to have been given or made when
                  delivered in person or deposited in the mail, certified,
                  postage pre-paid, return receipt requested, or in the case of
                  telegraphic notice, when delivered to the telegraph company
                  addressed to the parties described in this Agreement and at
                  such addresses as are set forth above or at such other
                  addresses as may be provided by the parties pursuant to this
                  paragraph.

         (e)      Should any one of more of the provisions of this Agreement be
                  determined to be illegal or unenforceable, all other
                  provisions, nevertheless, shall remain effective and binding
                  on the parties hereto.

         (f)      Borrower shall not assign or transfer his rights or
                  obligations hereunder.


         IN WITNESS WHEREOF, the Borrower and the Lender have caused this
Agreement to be duly executed as of the day and year first above written.


                              Borrower:

                                     BACK YARD BURGERS, INC.


                                     By:
                                        ---------------------------------------


                                     Title:
                                           ------------------------------------



                              Lender:

                                      TRUST ONE BANK


                                     By:
                                        ---------------------------------------


                                     Title:
                                           ------------------------------------



                                    - 5 -