<PAGE>

                                                                    Exhibit 10.4


                       FOURTH AMENDMENT TO PROMISSORY NOTE

         THIS FOURTH AMENDMENT TO PROMISSORY NOTE (this "Fourth Amendment") is
entered into as of the 1st day of January, 2003, by and between DIVERSICARE
ASSISTED LIVING SERVICES NC II, LLC, a Delaware limited liability company (the
"Borrower"), and GMAC COMMERCIAL MORTGAGE CORPORATION, a California corporation
(the "Lender").

                                    RECITALS

             A. The Borrower executed to the order of the Lender that certain
Promissory Note dated June 4, 1999, in the principal amount of $12,480,000, as
amended by that certain First Amendment to Promissory Note dated July 1, 2002,
as further amended by that certain Second Amendment to Promissory Note dated as
of October 1, 2002, and as further amended by that certain Third Amendment to
Promissory Note dated as of December 1, 2002 (the "Note"). Unless otherwise
defined herein, capitalized terms shall have the meaning assigned to them in the
Note.

             B. The Borrower has requested that the Lender extend the Maturity
Date of the Note, and the Lender has agreed, upon certain conditions, one of
which is the execution of this Fourth Amendment.

                                    AGREEMENT

         NOW, THEREFORE, in consideration of the above Recitals and other good
and valuable consideration, the Borrower and the Lender hereby amend the Note as
follows:

             1. Section 4 of the Note, Maturity Date, is hereby amended to
extend the Maturity Date from January 1, 2003, until April 1, 2003. All
references in the Note to the "Maturity Date" are hereby amended to mean April
1, 2003.

         Except as expressly amended herein, the Note shall remain in full force
and effect in accordance with its terms and conditions.

         Notwithstanding the execution of this Fourth Amendment, the
indebtedness evidenced by the Note shall remain in full force and effect, and
nothing contained herein shall be interpreted or construed as resulting in a
novation of such indebtedness. The Borrower acknowledges and agrees that there
are no offsets or defenses to payment of the obligations evidenced by the Note,
as hereby amended, and hereby waives any defense, claim or counterclaim of the
Borrower regarding the obligations of the Borrower under the Note, as hereby
amended. The Borrower represents that there are no conditions of default or
facts or consequences which will or could lead to a default under the
obligations due from the Borrower under the Note, as amended herein, except as
disclosed by Borrower and Diversicare Management Services Co. in that certain


<PAGE>

Quarterly Compliance Statement & Census Data report and that certain Compliance
Certificate, each for the period ending September 30, 2002, dated November 12,
2002, and signed by Borrower's Chief Executive Officer.



















                                       2
<PAGE>


        IN WITNESS WHEREOF, the Borrower and Lender have caused this Fourth
Amendment to be executed by their respective duly authorized representatives, as
of the date first set forth above.

                                BORROWER:

                                DIVERSICARE ASSISTED LIVING SERVICES NC II, LLC,
                                a Delaware limited liability company


                                By: Diversicare Assisted Living Services NC, LLC
                                Its: Sole Member

                                By:    /s/ William R. Council, III
                                       --------------------------
                                       Name: William R. Council, III
                                       Title: Chief Executive Officer


                                LENDER:

                                GMAC COMMERCIAL MORTGAGE CORPORATION,
                                a California corporation

                                By: Laura Y. McDonald
                                    ----------------------
                                Its: Senior Vice President
                                     ---------------------





                                       3