<PAGE>
                                                                    Exhibit 10.4


This instrument prepared by
and upon recording return to:

Jeffery S. DeArman, Esq.
Walston, Wells, Anderson & Bains, LLP
505 20th Street North, Suite 500
Birmingham, AL 35203
205-251-9600

STATE OF GEORGIA  )

FULTON COUNTY     )

                    SECOND AMENDMENT TO DEED TO SECURE DEBT,
              SECURITY AGREEMENT AND ASSIGNMENT OF RENTS AND LEASES

                  This Second Amendment to Deed to Secure Debt, Security
Agreement and Assignment of Rents and Leases (this "Second Amendment") is
executed as of this 30th day of April, 2002, by and between ROBERTS PROPERTIES
RESIDENTIAL, L.P., a Georgia limited partnership (the "Grantor"), and AMSOUTH
BANK, a state banking corporation (the "Bank").

                                    Recitals

                  A.       The Grantor executed in favor of the Bank that
certain Deed to Secure Debt, Security Agreement and Assignment of Rents and
Leases dated as of August 31, 2001, recorded in the Office of the Clerk of
Superior Court of Fulton County, Georgia, in Deed Book 30956, at page 194, as
amended by that certain First Amendment to Deed to Secure Debt, Security
Agreement and Assignment of Rents and Leases, recorded in the Office of the
Clerk of Superior Court of Fulton County, Georgia in Deed Book 31534, at Page
563 (the "Agreement"). Unless otherwise defined herein, capitalized terms shall
have the meaning assigned to them in the Agreement.

                  B.       The parties desire to extend the maturity date
described in the Agreement, and have agreed to execute this Second Amendment to
reflect such extension.

                                    Agreement

                  NOW, THEREFORE, in consideration of the Recitals, the Grantor
and the Bank hereby amend the Agreement as follows:

         1.       Recital A is hereby deleted in its entirety and replaced with
the following:

                  A.       The Grantor is justly indebted to the Bank in the
principal sum of $3,700,000, as evidenced by an amended and restated promissory
note executed by the Grantor, which note is payable in full on August 30, 2002,
and bears interest as provided therein (the "Note").

         2.       The third paragraph of page 4 of the Agreement is hereby
amended by deleting the date "April 30, 2002", and replacing it with the date
"August 30, 2002".
<PAGE>

                  Except as expressly amended hereby, the Agreement shall remain
in full force and effect in accordance with its terms.

                  The Grantor represents and warrants that the representations
and warranties set forth in the Agreement are as true and correct on the date
hereof as when initially made, except as such representation or warranty
expressly relates to another date.

                  The Grantor acknowledges and agrees that there are no offsets
or defenses to the obligations set forth in the Agreement, as hereby amended,
and represents that there are no Events of Default existing on the date hereof,
nor are there any facts or consequences which will or could lead to an Event of
Default under the Agreement.


                                       2
<PAGE>


                  IN WITNESS WHEREOF, the Grantor and the Bank have caused this
Second Amendment to be executed by their duly authorized respective
representatives as of the date first set forth above.

<TABLE>
<S>                                                  <C>
Signed, sealed and                                   ROBERTS PROPERTIES RESIDENTIAL,
delivered in the                                     L.P., a Georgia limited partnership
presence of:

                                                     By: RobertsRealty Investors, Inc., a Georgia
                                                         corporation, sole general partner
/s/ Adam Corder
-------------------------------------------
Unofficial Witness                                        By:  /s/ Charles R. Elliott
                                                              -------------------------------------------
                                                              Charles R. Elliott, Secretary and Treasurer

/s/ Abigail M. Carmichael                                             (CORPORATE SEAL)
-------------------------------------------
Notary Public

AFFIX SEAL

My Commission expires:

        6/17/05
------------------------



Signed, sealed                                       AMSOUTH BANK, an Alabama banking
and delivered in the                                 corporation
In the presence of:

/s/ Jackie Bullard                                        By /s/ Lawrence Clark
-------------------------------------------                  --------------------------------------------
Unofficial Witness                                          Its Vice President


/s/ Gina Hargrove
-------------------------------------------
Notary Public

AFFIX SEAL

My commission expires:

         6/5/04
------------------------
</TABLE>


                                       3