<PAGE>

                                                                    EXHIBIT 10.4

                   FIFTH AMENDMENT TO RENEWAL PROMISSORY NOTE

         THIS FIFTH AMENDMENT TO RENEWAL PROMISSORY NOTE is made and entered
into by and among AMSOUTH BANK (the, "Bank") and DIVERSICARE ASSISTED LIVING
SERVICES, NC, LLC, a Tennessee limited liability company (the "Borrower").

                              W I T N E S S E T H :

         WHEREAS, Borrower executed to Bank that certain Renewal Promissory Note
dated October 1, 2000, in the original principal amount of NINE MILLION FOUR
HUNDRED TWELVE THOUSAND THREE HUNDRED EIGHTY THREE AND 87/100 ($9,412,383.87)
DOLLARS, as amended by the First Amendment to Renewal Promissory Note executed
by Borrower in December, 2000, as amended by that Second Amendment to Renewal
Promissory Note executed by Borrower and Bank effective as of December 15, 2002,
as further amended by that Third Amendment to Renewal Promissory Note executed
by Borrower and Bank to be effective as of July 11, 2003, as further amended by
that Fourth Amendment to Renewal Promissory Note executed by Borrower and Bank
to be effective as of January 9, 2004 (the "Note"); and

         WHEREAS, Bank has agreed to further modify the Note in accordance with
the terms and conditions set forth herein.

         NOW, THEREFORE, for good and valuable consideration, and payment of an
extension fee in the amount of $5,000.00 by Borrower to Bank upon execution of
this Amendment, the receipt and sufficiency of which are hereby acknowledged,
the parties agree as follows:

         1.       The Note is amended to provide that the Maturity Date, as
defined in the Note, shall be changed from April 16, 2004 to July 16, 2004.

         2.       The Note is amended as stated herein, but no further or
otherwise, and the terms and provisions of the Note, as hereby amended, shall be
and continue to be in full force and effect. Nothing herein is intended to
operate to release or diminish any right of Bank under the Note or with respect
to any collateral securing the Note or with respect to any guaranty or
suretyship agreement for the Note, all of which shall remain in full force and
effect. This instrument constitutes the entire agreement of the parties with
respect to the subject matter hereof.

                                        1

<PAGE>

         IN WITNESS WHEREOF, this instrument has been executed to be effective
on the 16th day of April, 2004.

                                        BORROWER:

                                        DIVERSICARE ASSISTED LIVING SERVICES,
                                        NC, LLC, a Tennessee limited
                                        liability company

                                        By: /s/ William R. Council
                                            -----------------------------------
                                                William R. Council, President

                                        BANK:

                                        AMSOUTH BANK

                                        By: /s/ Tim McCarthy
                                            -----------------------------------
                                                Tim McCarthy, Vice President

                                        2