<PAGE>   1
                                                                   Exhibit 10.4

                               SIXTH AMENDMENT TO
                                 LOAN AGREEMENT

         THIS SIXTH AMENDMENT TO LOAN AGREEMENT made and entered into as of
this 24th day of October, 1996, ("this Sixth Amendment"), by and between
SHOLODGE, INC., a Tennessee corporation ("Borrower"); SHONEY'S INN OF LEBANON,
INC., a Tennessee corporation; SHONEY'S INN, INC., a Tennessee corporation;
NASHVILLE AIR ASSOCIATES, INC., a Tennessee corporation; MOORE AND ASSOCIATES,
INC., a Tennessee corporation; SUNSHINE INNS, INC. (successor by change of name
to PenTal Inns, Inc. and successor by merger to Tampa Inn, Inc.), a Tennessee
corporation; LAFLA INN, INC., a Tennessee corporation; SOUTHEAST TEXAS INNS,
INC., a Tennessee corporation; DELAWARE INNS, INC., a Tennessee corporation;
ALABAMA LODGING CORPORATION (successor by change of name to Birmingham Inn,
Inc.), a Tennessee corporation; CAROLINA INNS, INC., a Tennessee corporation;
MIDWEST INNS, INC., a Tennessee corporation; FAR WEST INNS, INC., a Tennessee
corporation; SHONEY'S INN OF BATON ROUGE, a Tennessee general partnership; and
FRONT RANGE SUITES, INC., a Tennessee corporation (individually and
collectively "Co-Obligors"); and FIRST UNION NATIONAL BANK OF TENNESSEE, a
national banking corporation ("Lender").

                                    RECITALS

         WHEREAS, Borrower and Lender entered into that certain Loan Agreement
dated January 7, 1994 ("the Loan Agreement"), the terms of which are
incorporated herein by this reference;

         WHEREAS, Borrower and Lender entered into that certain First Amendment
to Loan Agreement dated March 21, 1994 ("the First Amendment"), the terms of
which are incorporated herein by this reference;

         WHEREAS, Borrower, Lender, and certain of Co-Obligors entered into
that certain Second Amendment to Loan Agreement dated June 20, 1994 ("the
Second Amendment"), the terms of which are incorporated herein by this
reference;

         WHEREAS, Borrower, Lender, and certain of Co-Obligors entered into
that certain Third Amendment to Loan Agreement dated August 17, 1995 ("the
Third Amendment"), the terms of which are incorporated herein by this
reference;


<PAGE>   2



         WHEREAS, Borrower, Lender, and certain of Co-Obligors entered into
that certain Fourth Amendment to Loan Agreement dated November 14, 1995 ("the
Fourth Amendment"), the terms of which are incorporated herein by this
reference;

         WHEREAS, Borrower, Lender, and Co-Obligors entered into that certain
Fifth Amendment to Loan Agreement dated October 18, 1996 ("the Fifth
Amendment"), the terms of which are incorporated herein by this reference;

         WHEREAS, Co-Obligors are Subsidiaries (as defined in the Loan
Agreement) of Borrower and Co-Obligors have and will directly or indirectly
benefit from extensions of credit made by Lender to Borrower;

         WHEREAS, Borrower and Co-Obligors have requested that Lender make an
additional loan to Borrower in the original principal amount of Two Million
Five Hundred Thousand Dollars ($2,500,000) on the terms and conditions of the
Loan Agreement, as amended, and upon the terms and conditions of this Sixth
Amendment;

         WHEREAS, in order to induce Lender to extend the above described
additional credit to Borrower and amend the Loan Agreement, as amended, as
outlined in this Sixth Amendment, Borrower and Co-Obligors have made certain
representations to Lender; and

         WHEREAS, Lender, in reliance on representations and inducements of
Borrower and Co-Obligors, has agreed to make this additional Two Million Five
Hundred Thousand Dollars ($2,500,000) loan to Borrower and further amend the
Loan Agreement as provided in this Sixth Amendment.

         NOW, THEREFORE, for and in consideration of the agreement of Lender to
amend the Loan Agreement as set out in this Sixth Amendment, the mutual
covenants and agreements hereinafter set forth and other good and valuable
consideration, the receipt and sufficiency of which is hereby acknowledged,
Lender, relying on the representations and warranties contained in the Loan
Agreement, as amended, agrees to further amend the Loan Agreement as follows:

         1.       Definitions.  All terms used herein will be further
defined in the Loan Agreement, as amended.



                                       2
<PAGE>   3



         2.       The Loan.  Section 2.1 of the Loan Agreement, as amended,
is hereby deleted in its entirety and the following is substituted
in lieu thereof:

                  2.1      Evidence of Indebtedness; Repayment.

                           (a) The Loan will be evidenced by the Note, which
         will include: (i) Promissory Note dated November 14, 1995, in the
         original principal amount of Twenty Five Million Dollars ($25,000,000)
         made and executed by Borrower and Co-Obligors, payable to Lender and
         having been delivered to Lender by Borrower and Co-Obligors (other
         than Carolina Inns, Inc., Midwest Inns, Inc., Far West Inns, Inc.,
         Shoney's Inn of Baton Rouge, and Front Range Suites, Inc.); (ii)
         Addendum to Promissory Note dated December 27, 1995, executed by
         Carolina Inns, Inc., Midwest Inns, Inc., Far West Inns, Inc., and
         Shoney's Inn of Baton Rouge, as additional co-obligors; (iii) Second
         Addendum to Promissory Note dated October 18,1996, executed by Front
         Range Suites, Inc., as an additional co-obligor; and (iv) First
         Amendment to Promissory Note of even date, wherein the principal
         amount of the Note is increased to Twenty Seven Million Five Hundred
         Thousand Dollars ($27,500,000) made and executed contemporaneously
         herewith by Borrower and Co- Obligors, payable to Lender and having
         been delivered to Lender by Borrower and Co-Obligors. Interest on
         LIBOR Rate Loans will be payable on the earlier of quarterly (i.e.
         January 1, April 1, July 1, and October 1) or the last day of the
         Interest Period. Interest on Prime Rate Loans will be payable
         quarterly (i.e. January 1, April 1, July 1, and October 1). The entire
         outstanding principal amount of the Note, plus all accrued and unpaid
         interest, will be due and payable on or before the Maturity Date. The
         Loan will be repayable in accordance with the additional terms set out
         in the Note.

                           (b) So long as no Event of Default has occurred and
         is continuing hereunder, and so long as no event has occurred that
         with the giving of notice, the passage of time or both, will
         constitute an Event of Default hereunder, and, subject to the
         restrictions set forth herein, Borrower may borrow, repay, and
         re-borrow



                                       3
<PAGE>   4



         funds under the Loan as needed, so long as the maximum principal
         amount outstanding does not exceed Twenty Seven Million Five Hundred
         Thousand Dollars ($27,500,000) during the time period beginning on the
         date hereof, and except as hereinafter provided, ending on the
         Maturity Date. Provided, however, Borrower will be required to make a
         mandatory prepayment in the principal amount of Two Million Five
         Hundred Thousand ($2,500,000) on November 22,1996; Borrower will not
         be permitted to re- borrow this Two Million Five Hundred Thousand
         Dollars ($2,500,000) and thereafter the maximum principal amount that
         may be outstanding under the Note at anytime will be reduced to Twenty
         Five Million Dollars ($25,000,000).

         3. Conditions Precedent.  Notwithstanding any other provisions of this
Sixth Amendment, it is understood and agreed that Lender will have no
obligation to fund at any time unless and until the following conditions have
been met and continue to be met, to the reasonable satisfaction of Lender and
its counsel:

                  (a) Lender will have received such documents, instruments,
and agreements, including, but not limited to, the Sixth Amendment and all
modifications and/or amendments thereto, in form satisfactory to Lender, as
Lender will reasonably request to evidence the agreement set out herein, which
will include a First Amendment to Promissory Note, in form acceptable to
Lender;

                  (b) A Certificate of Existence of Borrower and each of
the Co-Obligors (other than Shoney's Inn of Baton Rouge) issued by
the Secretary of State of Tennessee; and

                  (c) Opinion of counsel for Borrower and each of the Co-
Obligors in form and substance satisfactory to Lender.

         4. Costs.  Borrower agrees to pay all reasonable fees and costs 
incurred by Lender in connection with the preparation, execution, and delivery
of all documentation reasonably required to evidence the agreement set out
herein, including Lender's counsel fees.

         5. Continuing Representation and Warranty by Borrower and Co-Obligors.
Borrower and Co-Obligors represent and warrant to Lender that all of the
covenants, agreements, representations, and


                                       4
<PAGE>   5




warranties made by Borrower and Co-Obligors in the Loan Agreement, as amended,
in any writing delivered pursuant to the Loan Agreement, as amended, and in
this Sixth Amendment, are true and correct, and have been fully complied with,
in all material respects, as of such date.

         6. Continuing Effect.  Except as specifically set forth in this Sixth
Amendment, the Loan Agreement, as amended by the First Amendment, the Second
Amendment, the Third Amendment, the Fourth Amendment, and the Fifth Amendment,
and all of the Loan Documents remain in full force and effect as originally
written.

         IN WITNESS WHEREOF, the parties hereto have caused this Sixth
Amendment to be executed by their duly authorized officers effective as of the
day and year first above written.

                                        LENDER:

                                        First Union National Bank of
                                        Tennessee

                                        By: /s/ Orville W. Kronk
                                            ------------------------------
                                            Orville W. Kronk
                                            Vice President


                                        BORROWER:

                                        ShoLodge, Inc.

                                                    
                                        By: /s/ Leon Moore
                                            ------------------------------ 
                                            Leon Moore, President


                                        CO-OBLIGORS:

                                        Shoney's Inn of Lebanon, Inc.

                                        By: /s/ Leon Moore
                                            ------------------------------
                                            Leon Moore, President


                                      5
<PAGE>   6




                                        Shoney's Inn, Inc.


                                        By: /s/ Leon Moore
                                            ------------------------------ 
                                            Leon Moore, President


                                        Nashville Air Associates, Inc.

 
                                        By: /s/ Leon Moore
                                            ------------------------------
                                            Leon Moore, President


                                        Moore and Associates, Inc.


                                        By: /s/ Leon Moore
                                            ------------------------------
                                            Leon Moore, President


                                        Sunshine Inns, Inc.


                                        By: /s/ Leon Moore
                                            ------------------------------ 
                                            Leon Moore, President


                                        LAFLA Inn, Inc.


                                        By: /s/ Leon Moore
                                            ------------------------------
                                            Leon Moore, President


                                        Southeast Texas Inns, Inc.

                                        By: /s/ Leon Moore
                                            ------------------------------
                                            Leon Moore, President


                                      6
<PAGE>   7



                                        Delaware Inns, Inc.

                                      
                                        By: /s/ Leon Moore
                                            ------------------------------ 
                                            Leon Moore, President


                                        Alabama Lodging Corporation

 
                                        By: /s/ Leon Moore
                                            ------------------------------
                                            Leon Moore, President


                                        Carolina Inns, Inc.

 
                                        By: /s/ Leon Moore
                                            ------------------------------
                                            Leon Moore, President


                                        Midwest Inns, Inc.

 
                                        By: /s/ Leon Moore
                                            ------------------------------
                                            Leon Moore, President


                                        Far West Inns, Inc.


                                        By: /s/ Leon Moore
                                            ------------------------------
                                            Leon Moore, President


                                        Shoney's Inn of Baton Rouge

                                        By:      Two Seventeen, Inc.
                                                 General Partner


                                      7
<PAGE>   8



                                                 By: /s/ Leon Moore
                                                     --------------------------
                                                 Its: President
                                                      -------------------------
                                        By:      Inn Partners, Inc.
                                                 General Partner

                                                 By: /s/ Leon Moore
                                                     --------------------------
                                                 Its: President
                                                      -------------------------

                                        Front Range Suites, Inc.

                                        By: /s/ Leon Moore
                                            -----------------------------------
                                            Leon Moore, President




                                      8
<PAGE>   9
                       FIRST AMENDMENT TO PROMISSORY NOTE

                                  $27,500,000

Nashville, Tennessee                                           October 24, 1996

         WHEREAS, on November 14, 1995, SHOLODGE, INC. ("Borrower"); SHONEY'S
INN OF LEBANON, INC., SHONEY'S INN, INC., NASHVILLE AIR ASSOCIATES, INC., MOORE
AND ASSOCIATES, INC., SUNSHINE INNS, INC. (successor by name change to PenTal
Inns, Inc. and successor by merger to Tampa Inn, Inc.), LAFLA INN, INC.,
SOUTHEAST TEXAS INNS, INC., DELAWARE INNS, INC., ALABAMA LODGING CORPORATION
(successor by name change to Birmingham Inns, Inc.), as co-obligors
(individually and collectively "Co-Obligors"), executed a certain Promissory
Note in the original principal amount of Twenty Five Million Dollars
($25,000,000),payable to FIRST UNION NATIONAL BANK OF TENNESSEE ("Payee") ("the
Note");

         WHEREAS, on December 27, 1995, CAROLINA INNS, INC., MIDWEST INNS,
INC., FAR WEST INNS, INC., and SHONEY'S INN OF BATON ROUGE, as co-obligors
(included in the defined term, individually and collectively, "Co-Obligors"),
executed an Addendum to Promissory Note (included in the defined term "the
Note");

         WHEREAS, on October 18, 1996, FRONT RANGE SUITES, INC., as co-obligor
(included in the defined term, individually and collectively, "Co-Obligors"),
executed a Second Addendum to Promissory Note (included in the defined term
"the Note"); and

         WHEREAS, Borrower and Co-Obligors have requested that Payee make an
additional loan to Borrower in the original principal amount of Two Million
Five Hundred Thousand Dollars ($2,500,000) on the terms and conditions of that
certain Loan Agreement dated January 7, 1994, as amended by: (i) First
Amendment to Loan Agreement dated March 21, 1994, (ii) Second Amendment to Loan
Agreement dated June 20, 1994 , (iii) Third Amendment to Loan Agreement dated
August 17, 1995, (iv) Fourth Amendment to Loan Agreement dated November 14,
1995, (v) Fifth Amendment to Loan Agreement dated October 18, 1996, and
(vi)Sixth Amendment to Loan Agreement of even date with this First Amendment to
Promissory Note.

         NOW, THEREFORE, for and in consideration of the agreement of Payee to
accept this First Amendment to Promissory Note, the mutual covenants and
agreements hereinafter set forth and other good and


<PAGE>   10



valuable consideration, the receipt and sufficiency of which is hereby
acknowledged, the Note is amended as follows:

         Borrower, Co-Obligors, and Payee hereby delete the first paragraph of
the Note in its entirety and substitute the following in lieu thereof:

                  FOR VALUE RECEIVED, the undersigned, SHOLODGE, INC., a
         Tennessee corporation ("Borrower"); SHONEY'S INN OF LEBANON, INC., a
         Tennessee corporation; SHONEY'S INN, INC., a Tennessee corporation;
         NASHVILLE AIR ASSOCIATES, INC., a Tennessee corporation; MOORE AND
         ASSOCIATES, INC., a Tennessee corporation; SUNSHINE INNS, INC.
         (successor by change of name to PenTal Inns, Inc. and successor by
         merger to Tampa Inn, Inc.), a Tennessee corporation; LAFLA INN, INC.,
         a Tennessee corporation; SOUTHEAST TEXAS INNS, INC., a Tennessee
         corporation; DELAWARE INNS, INC., a Tennessee corporation; ALABAMA
         LODGING CORPORATION (successor by change of name to Birmingham Inn,
         Inc.), a Tennessee corporation; FAR WEST INNS, INC., a Tennessee
         corporation; SHONEY'S INN OF BATON ROUGE, a Tennessee general
         partnership; and FRONT RANGE SUITES, INC., a Tennessee corporation
         (individually and collectively "Co- Obligors"), promise to pay to the
         order of FIRST UNION NATIONAL BANK OF TENNESSEE, a national banking
         corporation ("Payee"; Payee and/or any subsequent holder(s) hereof,
         "Holder"), at Payee's address P.O. Box 2648, Nashville, Tennessee
         37219-0648, or at such other place as Holder will designate from time
         to time in writing, the principal sum of Twenty Seven Million Five
         Hundred Thousand Dollars ($27,500,000), or so much thereof as may have
         been advanced, together with interest on the unpaid principal balance
         of such indebtedness from time to time outstanding from the date
         hereof at the rates hereinafter set forth, in lawful money of the
         United States of America, such principal and interest being due and
         payable as provided in the Note.

         Section 3 of the Note is hereby deleted in its entirety and the
following is substituted in lieu thereof:

                  3. Payment Terms.  Interest on LIBOR Rate Loans will be
         payable the earlier of quarterly (i.e. January 1, April 1, July 1, and
         October 1) or the last day of each Interest Period.  Interest on Prime 
         Rate Loans will be payable


                                       2
<PAGE>   11



         quarterly (i.e. January 1, April 1, July 1, and October 1). The entire
         outstanding principal amount of the Note, plus all accrued and unpaid
         interest, penalties, and fees will be due and payable on or before the
         Maturity Date. Borrower will be required to make a mandatory
         prepayment in the principal amount of Two Million Five Hundred
         Thousand Dollars ($2,500,000) on November 22, 1996; Borrower will not
         be permitted to re-borrow this Two Million Five Hundred Thousand
         Dollars ($2,500,000) and thereafter the maximum principal amount that
         may be outstanding under the Note at anytime will be reduced to Twenty
         Five Million Dollars ($25,000,000).

         Except as specifically set out in this First Amendment, the Note will
remain in full force and effect as originally written.

         SIGNED, SEALED, AND DELIVERED, as of the 24th day of October, 1996.

                                         BORROWER:

                                         ShoLodge, Inc.


                                         By: /s/ Leon Moore
                                             --------------------------------
                                             Leon Moore, President
          
 
                                         CO-OBLIGORS:

                                         Shoney's Inn of Lebanon, Inc.


                                         By: /s/ Leon Moore
                                             --------------------------------
                                             Leon Moore, President


                                         Shoney's Inn, Inc.


                                         By: /s/ Leon Moore
                                             --------------------------------
                                             Leon Moore, President 

                                       3
<PAGE>   12



                                         Nashville Air Associates, Inc.

                                         By: /s/ Leon Moore
                                             --------------------------------
                                             Leon Moore, President

                                         Moore and Associates, Inc.

                                         By: /s/ Leon Moore
                                             --------------------------------
                                             Leon Moore, President

                                         Sunshine Inns, Inc.


                                         By: /s/ Leon Moore
                                             --------------------------------
                                             Leon Moore, President

                                         LAFLA Inn, Inc.


                                         By: /s/ Leon Moore
                                             -------------------------------- 
                                             Leon Moore, President
 
                                         Southeast Texas Inns, Inc.

 
                                         By: /s/ Leon Moore
                                             --------------------------------
                                             Leon Moore, President

                                         Delaware Inns, Inc.


                                         By: /s/ Leon Moore
                                             --------------------------------
                                             Leon Moore, President

                                         Alabama Lodging Corporation


                                       4
<PAGE>   13





                                         By: /s/ Leon Moore
                                             --------------------------------
                                             Leon Moore, President

                                         Carolina Inns, Inc.


                                         By: /s/ Leon Moore
                                             -------------------------------- 
                                             Leon Moore, President     

                                         Midwest Inns, Inc.


                                         By: /s/ Leon Moore
                                             --------------------------------
                                             Leon Moore, President

                                         Far West Inns, Inc.


                                         By: /s/ Leon Moore
                                             --------------------------------
                                             Leon Moore, President

                                         Shoney's Inn of Baton Rouge

                                         By: Two Seventeen, Inc.
                                             General Partner

                                             By: /s/ Leon Moore
                                                 ----------------------------
                                             Its: President
                                                 ----------------------------   

                                         By: Inn Partners, Inc.
                                             General Partner



                                       5
<PAGE>   14
                                             By: /s/ Leon Moore
                                                 ----------------------------
                                             Its: President
                                                 ----------------------------   



                                         Front Range Suites, Inc.

                                         By: /s/ Leon Moore
                                             ---------------------------------
                                             Leon Moore, President




         The undersigned Payee hereby consents to this First amendment to
Promissory Note.

                                         PAYEE:

                                         First Union National Bank of
                                         Tennessee

                                         By: /s/ Orville W. Kronk
                                             ---------------------------------
                                             Orville W. Kronk
                                             Vice President


                                       6