<PAGE>   1

                                                                    EXHIBIT 10.4

                SECOND AMENDED AND RESTATED EMPLOYMENT AGREEMENT

         AGREEMENT effective November 20, 1997, by and between RFS Managers,
Inc., a Tennessee corporation (the "Company"), and Robert M. Solmson (the
"Executive").

                              W I T N E S S E T H:

         WHEREAS, the Company provides management services to RFS Hotel
Investors, Inc. (the "Parent") pursuant to a Management Services Agreement dated
December 30, 1994 (the "Management Agreement"); and

         WHEREAS, the Company desires to employ the Executive to serve as the
Chief Executive Officer of the Company; and

         WHEREAS, the Company and the Executive have previously entered into an
employment agreement dated January 1, 1996 (the "Original Agreement") and an
Amended and Restated Employment Agreement dated January 1, 1997 (the "Amended
Agreement"); and

         WHEREAS, the parties desire to amend and restate the Amended Agreement
in its entirety effective as of November 20, 1997 as set forth herein.

         NOW, THEREFORE, in consideration of the premises and mutual obligations
hereinafter set forth the parties agree as follows:

         1. EMPLOYMENT. The Company shall employ the Executive, and the
Executive agrees to be so employed, in the capacity of Chief Executive Officer
of the Company to serve for the Term hereof, subject to earlier termination as
hereinafter provided.

         2. TERM. The term of the Executive's employment hereunder shall be
three years and shall commence on January 1, 1997 and shall be extended
automatically, for so long as the Executive remains employed by the Company
hereunder, each January 1 beginning January 1, 1998 for an additional
twelve-month period (such period, as it may be extended from time to time, being
herein referred to as the "Term"), unless terminated earlier in accordance with
the terms of this Agreement, to the effect that on each January 1, the remaining
term of this Agreement and the Executive's employment hereunder shall be three
years.

         3. SERVICES. The Executive shall devote such amount of his time and
attention to the Company's affairs as are necessary to perform his duties to the
Company and to allow the Company to perform its duties specified in the
Management Agreement. Pursuant to the Management Agreement, the Executive shall
have authority and responsibility with respect to the day to day operations and
management of the Parent and RFS Partnership, L.P. (the "Partnership"), for
which the Parent currently serves as sole general partner, as well as
implementation of the long range growth strategy of the Parent and the
Partnership, consistent with direction from the Parent's Board of Directors (the
"Board").

<PAGE>   2


         4. COMPENSATION. (a) During the Term, the Company shall pay the
Executive for his services an annual base salary of $225,000 (the "Base
Salary"), to be paid in semi-monthly payments of $9,375.00, such Base Salary
being subject to any increases approved by the Compensation Committee of the
Board (the "Compensation Committee").

                  (b) In addition to the Base Salary described in Section 4(a)
above, the Executive shall be entitled to a cash bonus ("Base Salary Bonus") for
1996 payable on or before April 1, 1997 determined as follows:

                           (i)   If fully diluted net income per share of common
                           stock of the Parent, for the year ending December 31,
                           1996, as reported in the Parent's audited financial
                           statements for the year ending December 31, 1996, as
                           adjusted as described in the following sentence
                           ("1996 Net Income Per Share") is at least $1.35, the
                           Executive shall be entitled to receive a cash bonus
                           equal to ten percent (10%) of the Base Salary. For
                           purposes of determining the cash bonus under this
                           Section 4(b), fully diluted Net Income per Share of
                           common stock of the Parent shall be exclusive of any
                           gain or loss on the sale of property, any expenses
                           relating to the transactions between the Company and
                           Doubletree Corporation and its affiliates and any
                           expenses which the Compensation Committee deems
                           appropriate to exclude from the calculation of fully
                           diluted net income per share for purposes of
                           determining the cash bonus; and

                           (ii)  For each $.01 increase in Net Income Per Share
                           for 1996 in excess of $1.35, computed by rounding to
                           the closest cent, the Executive shall be entitled to
                           receive an additional bonus equal to three percent
                           (3%) of the Base Salary; and

                           (iii) Notwithstanding the provisions of Section
                           4(b)(ii) above, the maximum cash bonus payable to the
                           Executive pursuant to this Section 4(b) shall be
                           fifty percent (50%) of the Base Salary.

                  (c) In addition to the Base Salary Bonus, the Executive may be
entitled to receive other incentive compensation, including but not limited to,
additional grants of stock options or shares of stock of the Parent, which
awards shall be made (if at all) in consideration of and as an incentive for
services performed solely for the Company, in accordance with rules and criteria
established by the Compensation Committee. Such criteria may include, but not be
limited to, the growth in the Parent's Net Income Per Share and/or other
performance goals.

         5. BENEFITS. The Company agrees to provide the Executive with the
following benefits:


<PAGE>   3


                  (a) Vacation. The Executive shall be entitled each calendar
year to a vacation, during which time his compensation shall be paid in full.
The time allotted for such vacation shall be three (3) weeks.

                  (b) Employee Benefits. This Agreement shall not be in lieu of
any rights, benefits and privileges to which the Executive may be entitled as a
management level employee of the Company, including but not limited to any
retirement, pension, profit-sharing, insurance, hospital or other plans which
may now be in effect or which may hereafter be adopted. The Executive shall have
the same rights and privileges to participate in such plans and benefits as any
other management level employee during the Term.

         6. EXPENSES. The Company recognizes that the Executive will have to
incur certain out-of-pocket expenses, including but not limited to travel
expenses, related to his services and the Company's and the Parent's business
and the Company agrees to reimburse the Executive for all reasonable expenses
necessarily incurred by him in the performance of his duties upon presentation
of a voucher or documentation indicating the amount and business purposes of any
such expenses.

         7. TERMINATION IN CASE OF DEATH OR DISABILITY. In the event of the
Executive's death or a complete physical or mental inability, confirmed by a
licensed physician, to perform the services described in Section 3 above that
continues for a period of one hundred twenty (120) consecutive days) ("Permanent
Disability"), the Company may elect to terminate this Agreement, subject to
continuation of the payments described in Section 10.

         8. DEFINITIONS. For purposes of this Agreement, the following terms
shall have the following definitions:

                  (a) "Voluntary Termination" means, subject to the provisions
of Section 11 hereof, the Executive's voluntary termination of his employment
hereunder, which may be effected by the Executive giving the Board not less than
90 days' prior written notice of the Executive's desire to terminate his
employment or the Executive's failure to provide substantially all the services
described in Section 3 hereof for a period greater than four consecutive weeks
by reason of the Executive's voluntary refusal to perform such services.
Notwithstanding the foregoing, if the Executive gives notice of Voluntary
Termination and, prior to the expiration of the 90-day notice period, the
Executive voluntarily refuses or fails to provide substantially all the services
described in Section 3 hereof for a period greater than two consecutive weeks,
the Voluntary Termination shall be deemed to be effective as of the date on
which the Executive so ceases to carry out his duties. For purposes of this
Section 8, voluntary refusal to perform services shall not include taking
vacation otherwise permitted in accordance with Section 5(a) hereof, the
Executive's failure to perform services on account of his illness or the illness
of a member of his immediate family, provided such illness is adequately
substantiated at the reasonable request of the Company, or any other absence
from service with the written consent of the Board.


<PAGE>   4


                  (b) "Termination Without Cause" means the termination of the
Executive's employment by the Company for any reason other than Voluntary
Termination or Termination With Cause.

                  (c) "Termination With Cause" means the termination of the
Executive's employment by act of the Board for any of the following reasons:

                      (i)   the Executive's conviction for a felony;

                      (ii)  the Executive's theft, embezzlement,
                      misappropriation of or intentional and malicious
                      infliction of damage to the Company's or the Parent's
                      property or business opportunity;

                      (iii) the Executive's intentional and material breach
                      of the noncompetition covenant in Section 11 hereof;

                      (iv)  the Executive's continuous neglect of his
                      duties hereunder or his continuous failure or refusal
                      to follow any reasonable, unambiguous duly adopted
                      written direction of the Board or any duly
                      constituted committee thereof that is not
                      inconsistent with the description of the Executive's
                      duties set forth in Section 3 above; and

                      (v)   the Executive's abuse of alcohol, drugs or
                      other substances, or his engaging in other deviant
                      personal activities in a manner that, in the
                      reasonable judgment of the Board, adversely affects
                      the reputation, goodwill or business position of the
                      Company.

                  (d) "Involuntary Termination" means conduct on the part of the
Company that constitutes continuous and material interference by the Company
with the Executive's performance of his duties as set forth in Section 3 hereof
or the intentional or material breach by the Company of this Agreement.

         9.  VOLUNTARY TERMINATION; TERMINATION WITH CAUSE. If (i) the Executive
shall cease being an employee of the Company on account of a Voluntary
Termination or (ii) there shall be a Termination With Cause, the Executive shall
not be entitled to any compensation after the effective date of such Voluntary
Termination or Termination With Cause (except Base Salary and vacation accrued
but unpaid on the effective date of such event). In the event of a Voluntary
Termination or Termination With Cause, the Executive shall continue to be
subject to the noncompetition covenant contained in Section 11 hereof for the
remainder of the Term.

         10. DEATH OR DISABILITY; TERMINATION WITHOUT CAUSE; OR INVOLUNTARY
TERMINATION. Following (i) the death of the Executive, (ii) Permanent Disability
of the Executive, (iii) an Involuntary Termination, or (iv) a Termination
Without Cause, the Company shall continue to pay the Executive or his heirs,
devisees, executors, legatees or personal 


<PAGE>   5

representatives, as appropriate, the semi-monthly payments of the Base Salary
then in effect for three years from the date of the termination of the
Executive's employment.

         11. CHANGE OF CONTROL COMPENSATION.

                  (a) Compensation. In the event of the termination of the
Executive's employment or the Executive's resignation for Good Reason (as
defined below) on or after a Change of Control (as defined below), the Company
shall, on the date of such termination or resignation, pay the Executive, in
addition to any Base Salary earned but not paid through the date of termination
or resignation, a cash amount (the "Termination Payment") equal to three (3)
times average annual base salary and cash bonus paid to or earned by the
Executive, under this Agreement or otherwise, for the fiscal year in which the
Change of Control occurs and the two preceding fiscal years or, if the Executive
has been employed by the Company for less than this period of time, then the
Termination Payment shall be based on the average annual base salary and cash
bonus paid or earned during the term of the Executive's employment (including
the fiscal year in which the Change of Control occurs). For purposes of
calculating sums hereunder: (i) if the salary and cash bonus are paid or earned
for any period of employment of less than twelve months (including the fiscal
year in which the Change of Control occurs) such sums shall be annualized and in
the case of a cash bonus, it shall be assumed that the bonus shall be the same
as the preceding year, and (ii) no fiscal year for which no base salary or cash
bonus is earned or paid shall be considered. Notwithstanding the first sentence
of this Section 11(a), the Termination Payment shall be calculated and paid
immediately prior to the closing of the transactions constituting a Change of
Control if (i) the Executive receives notice prior to the Change of Control that
his employment will be terminated on or after the Change of Control, or (ii)
regardless of whether the Executive is terminated or receives notice that his
employment will be terminated on or after the Change of Control, the Parent or
any of its affiliates enter into an agreement to effect a transaction
substantially similar to that described in Section 11(b)(v) below. In addition,
the Company or the Parent shall cause the Executive's insurance benefits, as in
effect immediately prior to the Change of Control, to remain in effect for at
least one year following the date of termination of Executive's employment by
the Company or the Executive's resignation for Good Reason.

                  (b) A "Change of Control", for purposes of this Agreement,
shall be deemed to have occurred if, at any time during the Term, any of the
following events occurs:

                           (i) any "person", as that term is used in Section
                  13(d) and Section 14(d)(2) of the Securities Exchange Act of
                  1934, as amended (the "Exchange Act"), becomes, is discovered
                  to be, or files a report on Schedule 13D or 14D-1 (or any
                  successor schedule, form or report) disclosing that such
                  person is, a beneficial owner (as defined in Rule 13d-3 under
                  the Exchange Act or any successor rule or regulation),
                  directly or indirectly, of securities of the Parent
                  representing 50% or more of the combined voting power of the
                  Parent's then outstanding securities entitled to vote
                  generally in the election of directors;


<PAGE>   6


                           (ii)  individuals who, as of the Effective Date,
                  constitute the Board of Directors of the Parent cease for any
                  reason to constitute at least a majority of the Board of
                  Directors of the Parent, unless any such change is approved by
                  the vote of at least 80% of the members of the Board of
                  Directors of the Parent in office immediately prior to such
                  cessation;

                           (iii) the Parent is merged, consolidated or
                  reorganized into or with another corporation or other legal
                  person, or securities of the Parent are exchanged for
                  securities of another corporation or other legal person, and
                  immediately after such merger, consolidation, reorganization
                  or exchange less than a majority of the combined voting power
                  of the then-outstanding securities of such corporation or
                  person immediately after such transaction are held, directly
                  or indirectly, in the aggregate by the holders of securities
                  entitled to vote generally in the election of directors of the
                  Parent immediately prior to such transaction;

                           (iv)  the Parent in any transaction or series of
                  related transactions, sells all or substantially all of its
                  assets to any other corporation or other legal person and less
                  than a majority of the combined voting power of the
                  then-outstanding securities of such corporation or person
                  immediately after such sale or sales are held, directly or
                  indirectly, in the aggregate by the holders of securities
                  entitled to vote generally in the election of directors of the
                  Parent immediately prior to such sale;

                           (v)   the Parent and its affiliates shall sell or
                  transfer of (in a single transaction or series of related
                  transactions) to a non-affiliate business operations or assets
                  that generated at least two-thirds of the consolidated
                  revenues (determined on the basis of the Parent's four most
                  recently completed fiscal quarters for which reports have been
                  filed under the Exchange Act) of the Parent and its
                  subsidiaries immediately prior thereto;

                           (vi)  the Parent files a report or proxy statement
                  with the Securities and Exchange Commission pursuant to the
                  Exchange Act disclosing in response to Form 8-K (or any
                  successor, form or report or item therein) that a change in
                  control of the Parent has occurred; or

                           (vii) any other transaction or series of related
                  transactions occur that have substantially the effect of the
                  transactions specified in any of the preceding clauses in this
                  sentence.

                  (c) Certain Transactions. Notwithstanding the provisions of
Section 11(b)(i) or 11(b)(vi) hereof, unless otherwise determined in a specific
case by majority vote of the Board of Directors of the Parent, a Change in
Control shall not be deemed to have occurred for purposes of this Agreement
solely because (i) the Parent, (ii) an entity in which the Parent directly or
indirectly beneficially owns 50% or more of the voting securities or (iii) any
Parent-sponsored employee stock ownership plan, or any other employee benefit
plan of the 


<PAGE>   7

Parent, either files or becomes obligated to file a report or a proxy statement
under or in response to Schedule 13D, Schedule 14D-1, Form 8-K or Schedule 14A
(or any successor schedule, form or report or item thereon) under the Exchange
Act, disclosing beneficial ownership by it of shares of stock of the Parent, or
because the Parent reports that a Change in Control of the Parent has or any
have occurred or will or may occur in the future by reason of such beneficial
ownership.

                  (d) Good Reason. "Good Reason," for purposes of this
Agreement, shall be deemed to mean any of the following:

                           (i)   a change in the Executive's status, position or
                  responsibilities (including reporting responsibilities) which,
                  in the Executive's reasonable judgment, does not represent a
                  promotion from the Executive's status, position or
                  responsibilities as in effect immediately prior to a Change in
                  Control; the assignment to the Executive of any duties or
                  responsibilities which, in the Executive's reasonable
                  judgment, are inconsistent with such status, position or
                  responsibilities; or any removal of the Executive from or
                  failure to reappoint or reelect the Executive to any of such
                  positions, except in connection with a Termination with Cause
                  as defined in Section 8(c), as a result of the Executive's
                  death or Permanent Disability, or by Voluntary Termination;

                           (ii)  a reduction in the Executive's Base Salary and
                  Base Salary Bonus as in effect on the date hereof or as the
                  same may be increased from time to time;

                           (iii) the relocation of the Company's or the Parent's
                  principal executive offices to a location outside a
                  thirty-mile radius of Memphis, Tennessee or the Company's or
                  the Parent's requiring the Executive to be based at any place
                  other than a location within a thirty-mile radius of Memphis,
                  Tennessee, except for reasonably required travel on the
                  Company's or the Parent's business which is not materially
                  greater than such travel requirements prior to the Change in
                  Control;

                           (iv)  the failure by the Company or the Parent to
                  continue to provide the Executive with compensation and
                  benefits provided for under this agreement or benefits
                  substantially similar to those provided to the Executive under
                  any of the employee benefit plans in which the Executive is or
                  becomes a participant, or the taking of any action by the
                  Company or the Parent which would directly or indirectly
                  materially reduce any of such benefits or deprive the
                  Executive of any material fringe benefit enjoyed by the
                  Executive at the time of the Change in Control;

                           (v)   any material breach by the Company of any
                  provision of this Agreement; and

                           (vi)  the failure of the Company to obtain a
                  satisfactory agreement from any successor or assign of the
                  Company to assume and agree to perform this Agreement.


<PAGE>   8


                  (e) Tax Matters. If the excise tax on "excess parachute
payments," as defined in section 280G of the Internal Revenue Code of 1986, as
amended (the "Code"), will be imposed on the Executive under Code section 4999
as a result of the Executive's receipt of the Termination Payment or any other
payment, benefit or compensation (without regard to the "Additional Amount"
described below) which the Executive receives or has the right to receive from
the Company or the Parent or any of their affiliates (the "Change of Control
Benefits"), the Company shall indemnify the Executive and hold him harmless
against all claims, losses, damages, penalties, expenses, and excise taxes. To
effect this indemnification, the Company shall pay to the Executive the
"Additional Amount" described in this Section 11(e). The Additional Amount shall
be the amount that is sufficient to indemnify and hold the Executive harmless
from the application or Code sections 280G and 4999, including the amount of (i)
the excise tax that will be imposed on the Executive under section 4999 of the
Code with respect to the Change of Control Benefits; (ii) the additional (A)
excise tax under section 4999 of the Code, (B) hospital insurance tax under
section 3111(b) of the Code and (C) federal, state and local income taxes for
which the Executive is or will be liable on account of the payment of the amount
described in item (i); and (iii) the further excise, hospital insurance and
income taxes for which the Executive is or will be liable on account of the
payment of the amount described in item (ii) and this item (iii) and any other
indemnification payment under this Section 11(e). The Additional Amount shall be
calculated and paid to the Executive at the time that the Termination Payment is
paid to the Executive. In calculating the Additional Amount, the highest
marginal rates of federal and applicable state and local income taxes applicable
to individuals and in effect for the year in which the Change of Control occurs
shall be used. Nothing in this paragraph shall give the Executive the right to
receive indemnification from the Company or the Parent for federal, state or
local income taxes or hospital insurance taxes payable solely as a result of the
Executive's receipt of (a) the Termination Payment, or (b) any additional
payment, benefit or compensation other than additional compensation in the form
of the excise tax payment specified in item (i), above. As specified in items
(ii) and (iii), above, all income, hospital insurance and additional excise
taxes resulting from additional compensation in the form of the excise tax
payment specified in item (i), above, shall be paid to the Executive.

         The provisions of this Section 11(e) are illustrated by the following
example:

             Assume that the Termination Payment and all other Change of Control
Benefits result in a total federal, state and local income tax and hospital
insurance tax liability of $180,000; and an excise tax liability under Code
section 4999 of $70,000. Under such circumstances, the Executive is solely
responsible for the $180,000 income and hospital insurance tax liability; and
the Company or the Parent must pay to the Executive $70,000, plus an amount
necessary to indemnify the Executive for all federal, state and local income
taxes, hospital insurance taxes, and excise taxes that will result from the
$70,000 payment to the Executive and from all further indemnification to the
Executive of taxes attributable to the initial $70,000 payment.

         12. NONCOMPETITION. During the Term and for a period of two (2) years
thereafter, the Executive shall not, other than through the Parent or affiliates
of the Parent, own more than a 



<PAGE>   9

10% interest in any hotel property (other than hotels owned by the Parent and
the Partnership), as partner, shareholder or otherwise, or directly or
indirectly, for his own account or for the account of others, either as an
officer, director, shareholder, owner, partner, promoter, employee, consultant,
advisor, agent, manager, or in any other capacity engage in the acquisition,
development, operation or management of any hotel property located within 20
miles of any hotel property owned by the Parent or the Partnership at the time
of termination of employment. The foregoing sentence shall not restrict the
Executive from owning up to 10% of the outstanding securities of any entity,
including any entity whose securities are traded in public securities markets.

         The Executive agrees that damages at law for violation of the
restrictive covenant contained herein would not be an adequate or proper remedy
to the Company, and that should the Executive violate or threaten to violate any
of the provisions of such covenant, the Company, its successors or assigns,
shall be entitled to obtain a temporary or permanent injunction against the
Executive in any court having jurisdiction over the person and the subject
matter, prohibiting any further violation of any such covenants. The injunctive
relief provided herein shall be in addition to any award of damages,
compensatory, exemplary or otherwise, payable by reason of such violation.

         Furthermore, the Executive acknowledges that this Agreement has been
negotiated at arms' length by the parties, neither being under any compulsion to
enter into this Agreement, and that the foregoing restrictive covenant does not
in any respect inhibit his ability to earn a livelihood in his chosen profession
without violating the restrictive covenant contained herein. The Company by
these presents has attempted to limit the Executive's right to compete only to
the extent necessary to protect the Company from unfair competition. The Company
recognizes, however, that reasonable people may differ in making such a
determination. Consequently, the Company agrees that if the scope or
enforceability of the restricted covenant contained herein is in any way
disputed at any time, a court or other trier of fact may modify and enforce the
covenant to the extent that it believes to be reasonable under the circumstances
existing at the time.

         13. NOTICES. All notices or deliveries authorized or required pursuant
to this Agreement shall be deemed to have been given when in writing and
personally delivered or when deposited in the U.S. mail, certified, return
receipt requested, postage prepaid, addressed to the parties at the following
addresses or to such other addresses as either may designate in writing to the
other party:

                  To the Company:   RFS Managers, Inc.
                                    850 Ridge Lake Boulevard
                                    Suite 220
                                    Memphis, TN 38120

                  To the Executive: Robert M. Solmson

                                    ---------------------

                                    ---------------------
<PAGE>   10


         14. ENTIRE AGREEMENT. This Agreement contains the entire understanding
between the parties hereto with respect to the subject matter hereof and shall
not be modified in any manner except by instrument in writing signed, by or on
behalf of, the parties hereto; provided, however, that any amendment or
termination of the covenant of noncompetition in Section 12 must be approved by
a majority of the Directors of the Parent other than the Executive, if the
Executive is then a director of the Parent. This Agreement shall be binding upon
and inure to the benefit of the heirs, successors and assigns of the parties
hereto.

         15. ARBITRATION. Any claim or controversy arising out of, or relating
to, this Agreement or its breach, shall be settled by arbitration in accordance
with the governing rules of the American Arbitration Association. Judgment upon
the award rendered may be entered in any court of competent jurisdiction.

         16. APPLICABLE LAW. This Agreement shall be governed and construed in
accordance with the laws of the State of Tennessee.

         17. ASSIGNMENT. The Executive acknowledges that his services are unique
and personal. Accordingly, the Executive may not assign his rights or delegate
his duties or obligations under this Agreement, except with respect to certain
rights to receive payments as described in Section 10. The Company's rights and
obligations under this Agreement shall inure to the benefit of and shall be
binding upon the Company's successors and assigns.

         18. HEADINGS. Headings in this Agreement are for convenience only and
shall not be used to interpret or construe its provisions.

         19. AMENDED AGREEMENT. This Agreement amends and restates the Amended
Agreement in its entirety as of the effective date of this Agreement.


<PAGE>   11


         IN WITNESS WHEREOF, the parties have executed this Agreement effective
as of the date first above written.

                                    RFS MANAGERS, INC.

                                    By:      /s/ Michael J. Pascal
                                       -------------------------------
                                    Name:    Michael J. Pascal
                                         -----------------------------
                                    Title:   Chief Financial Officer
                                          ----------------------------

                                    EXECUTIVE:

                                    /s/ Robert M. Solmson
                                    ----------------------------------
                                    Robert M. Solmson