<PAGE>   1
                                                                    EXHIBIT 10.4


                               FIFTH AMENDMENT TO
                       LOAN AND NEGATIVE PLEDGE AGREEMENT

         This Fifth Amendment to Loan and Negative Pledge Agreement, made and
entered into as of the 1st day of December, 1999, between First American
National Bank, a national banking association, as Agent for AmSouth Bank, an
Alabama banking corporation ("AmSouth"), First American National Bank ("FANB")
(individually, a "Bank" and, collectively, the "Banks"), and Diversicare
Assisted Living Services NC, LLC, a Tennessee limited liability company (the
"Borrower"),

                              W I T N E S S E T H:

         WHEREAS, pursuant to the terms of a Loan and Negative Pledge Agreement
dated as of October 1, 1997, by and between FANB, AmSouth and Borrower, as
amended from time to time (the "Loan Agreement"), the Banks agreed to make
available to the Borrower, on a nonrevolving basis, up to $34,100,000, to
finance the acquisition of the Facilities (capitalized terms not otherwise
defined herein shall have the meaning ascribed to such terms in the Loan
Agreement); and,

         WHEREAS, the balance outstanding under the Credit Facility is
$9,412,383.87; and,

         WHEREAS, Borrower has requested, and the Banks have agreed, to extend
the Maturity Date of the Credit Facility to February 28, 2000, subject to the
terms and conditions contained herein; and,

         WHEREAS, the Banks, the Borrower and the Guarantors desire to amend the
Loan Agreement to reflect the foregoing,

         NOW, THEREFORE, in consideration of the foregoing premises, and other
good and valuable consideration, the receipt and legal sufficiency of which are
hereby acknowledged, the parties hereto hereby amend the Loan Agreement as
follows:

         1. Definitions. The following definitions set forth in Section 1 of the
Loan Agreement are amended to read as follows:

                  "Maturity Date" means February 28, 2000.

                  "Notes" means the Renewal and Modification Promissory Notes of
even date herewith in the amount of $4,706,191.94 each, executed by the Borrower
in favor of the Banks, together with all renewals, amendments and extensions
thereof.

         2. Credit Facility. The references to the monthly interest payment
dates in Section 2.1 of the Loan Agreement are hereby modified to refer to
December 20, 1999, as the first payment date. The Maturity Date referred to in
Section 2.1 of the Loan Agreement is hereby modified to refer to February 28,
2000.

         3. Guarantors. The Guarantors have joined in the execution of this
Fifth Amendment to acknowledge the renewal and extension of the Loan and to
confirm to the Banks that the terms and provisions of the Guaranty Agreements
remain in full force and effect and to confirm that that the



                                       1
<PAGE>   2

Guaranty Agreements continue to secure the obligations under the Loan Agreement,
in accordance with the terms of the Guaranty Agreements.

         4. Closing Expenses. In consideration for the extension of the Maturity
Date and the other agreements of the Banks set forth herein, Borrower agrees to
pay all out-of-pocket expenses incurred by the Banks in connection with the
renewal and extension of the Loan, including, without limitation, reasonable
attorneys fees.

         5. Ratification. The Borrower hereby restates and ratifies all of the
terms and conditions contained in the Loan Agreement as of the date hereof, and
confirms that the Loan Agreement, as amended hereby, remains in full force and
effect.

                  (Remainder of Page Intentionally Left Blank)








                                       2
<PAGE>   3



         IN WITNESS WHEREOF, the parties hereto have executed this Fifth
Amendment as of the day and date first above written.


FIRST AMERICAN NATIONAL BANK                DIVERSICARE ASSISTED LIVING
                                            SERVICES NC, LLC


BY: /s/ Jeffery E. Lawrence                 BY: /s/ Richard B. Vacek, Jr.
    --------------------------------            --------------------------------
TITLE: Senior Vice President                TITLE: Executive VP--CFO
       -----------------------------               -----------------------------
First American Center                       Chief Executive Office:
Nashville, TN  37237                        277 Mallory Station Road, Suite 130
                                            Franklin, TN 37067


BANKS:

FIRST AMERICAN NATIONAL BANK


BY: /s/ Jeffery E. Lawrence
    --------------------------------
TITLE: Senior Vice President
       -----------------------------


AMSOUTH BANK


BY: /s/ Samuel Ballesteros
    --------------------------------
TITLE: Senior Vice President
       -----------------------------


GUARANTORS:

ADVOCAT, INC., a Delaware
corporation


BY: /s/ Richard B. Vacek, Jr.
    --------------------------------
TITLE: Executive VP--CFO
       -----------------------------





                                       3
<PAGE>   4


DIVERSICARE MANAGEMENT SERVICES
CO., a Tennessee corporation


BY: /s/ Richard B. Vacek, Jr.
    --------------------------------
TITLE: Executive VP--CFO
       -----------------------------


DIVERSICARE LEASING CORP.,
a Tennessee corporation


BY: /s/ Richard B. Vacek, Jr.
    --------------------------------
TITLE: Executive VP--CFO
       -----------------------------


ADVOCAT ANCILLARY SERVICES,
INC., a Tennessee corporation


BY: /s/ Richard B. Vacek, Jr.
    --------------------------------
TITLE: Executive VP--CFO
       -----------------------------


DIVERSICARE CANADA
MANAGEMENT SERVICES CO.,
INC., an Ontario, Canada corporation


BY: /s/ Richard B. Vacek, Jr.
    --------------------------------
TITLE: Executive VP--CFO
       -----------------------------


DIVERSICARE GENERAL
PARTNER, INC., a Texas corporation


BY: /s/ Richard B. Vacek, Jr.
    --------------------------------
TITLE: Executive VP--CFO
       -----------------------------






                                       4
<PAGE>   5

FIRST AMERICAN HEALTH CARE,
INC., an Alabama corporation


BY: /s/ Richard B. Vacek, Jr.
    --------------------------------
TITLE: Executive VP--CFO
       -----------------------------


ADVOCAT DISTRIBUTION SERVICES,
INC., a Tennessee corporation


BY: /s/ Richard B. Vacek, Jr.
    --------------------------------
TITLE: Executive VP--CFO
       -----------------------------


ADVOCAT FINANCE, INC., a
Delaware corporation


BY: /s/ Richard B. Vacek, Jr.
    --------------------------------
TITLE: Executive VP--CFO
       -----------------------------


DIVERSICARE LEASING CORP. OF
ALABAMA, INC., an
Alabama corporation


BY: /s/ Richard B. Vacek, Jr.
    --------------------------------
TITLE: Executive VP--CFO
       -----------------------------


DIVERSICARE ASSISTED LIVING
SERVICES, INC., a Tennessee
corporation


BY: /s/ Richard B. Vacek, Jr.
    --------------------------------
TITLE: Executive VP--CFO
       -----------------------------




                                       5