<PAGE>   1
                                                                 EXHIBIT 10.31.4

                     MANAGEMENT FEE SUBORDINATION AGREEMENT

         THIS MANAGEMENT FEE SUBORDINATION AGREEMENT (this "Agreement") is made
as of the 23rd day of December, 1997, in favor of NATIONSBANK, N.A. as agent
("Agent") for itself and for certain additional lenders (collectively with the
Agent, the "Lenders") who are or shall be from time to time participating as
lenders in a bank group pursuant to the Amended and Restated Agency Agreement
of even date herewith (as amended, restated or substituted from time to time,
the "Agency Agreement") by SUNRISE EAST ASSISTED LIVING LIMITED PARTNERSHIP, a
Virginia limited partnership (the "Borrower"), and SUNRISE ASSISTED LIVING
MANAGEMENT, INC., a corporation organized and existing under the laws of the
Commonwealth of Virginia, formerly known as Sunrise Terrace, Inc. (the
"Management Company").

                                    RECITALS

         A.      The Borrower obtained from the Agent and certain other lenders
(collectively, the "Original Lenders") a credit facility in the maximum
principal sum of $90,000,000 (the "Original Credit Facility") which was a
non-revolving line of credit pursuant to which the Borrower could obtain
certain construction/interim loans (each a "Facility Loan;" collectively, the
"Facility Loans") for assisted living facilities and independent living
facilities.  The Original Credit Facility has been evidenced by a Master
Promissory Note dated June 13, 1996 as amended pursuant to a First Amendment to
Master Promissory Note dated September 5, 1996 and by a Second Amendment to
Master Promissory Note dated March 31, 1997 (collectively, the "Master Note").

         B.      In connection with the making of each Facility Loan, the
Borrower executed a promissory note in the maximum principal sum of each
Facility Loan (each a "Facility Note" and collectively, the "Facility Notes").
Availability under the Master Note was reduced by the principal sum of each
Facility Note.  In connection with the Original Credit Facility, the Management
Company executed a Management Fee Subordination Agreement dated June 13, 1996
for the benefit of the Original Lenders.

         C.      The Borrower has applied to the Lenders to increase the
maximum principal sum of the Original Credit Facility to $250,000,000 or such
greater amount as the Lenders may from time to time commit to lend pursuant to
the Agency Agreement (such increased and modified credit facility being
hereinafter referred to as the "Credit Facility" or the "Loan") and to provide
that the Credit Facility will be revolving.  Advances or readvances are to be
made pursuant to, and secured by, the provisions of
<PAGE>   2
that certain Amended and Restated Financing and Security Agreement dated the
same date as this Agreement by and between the Agent and the Borrower (as
amended, restated or substituted from time to time, the "Financing Agreement")
and that certain Amended and Restated Master Construction Loan Agreement dated
the same as this Agreement by and between the Agent and the Borrower (as
amended, restated or substituted from time to time, the "Construction
Agreement").

         D.      The Loan is evidenced by that certain Amended, Restated,
Consolidated and Increased Master Promissory Note of even date herewith payable
by the Borrower to Agent on behalf of the Lenders (as amended, restated,
renewed or substituted from time to time, the "Note").

         E.      In accordance with, and pursuant to, that certain management
agreement by and between the Management Company and the Borrower dated
September 5, 1996 covering Facilities (as hereinafter defined) known as Sunrise
of Morris Plains, Sunrise of Old Tappan and Sunrise of Wayne and other
management agreements to be executed by the Borrower or Guarantor Subsidiaries
for other Facilities, the Management Company has agreed or will agree to manage
and operate those assisted living and independent living facilities
(collectively, the "Facilities") owned by the Borrower or the Guarantor
Subsidiaries (the management agreements, together with any and all amendments
thereto, extensions thereof and substitutions therefor are herein collectively
referred to as the "Management Agreement").

         F.      The Management Company and the Borrower have requested that
the Agent enter into the Financing Agreement with the Borrower and have
requested that the Lenders make the Loan to the Borrower pursuant thereto.

         G.      The Lenders have required, as a condition precedent to the
execution and delivery of the Financing Agreement and the making of the Loan
thereunder, the execution and delivery of this Agreement by both the Management
Company and the Borrower.

         H.      All capitalized terms used in this Agreement and not defined
herein shall have the meaning given to such terms in the Financing Agreement.

                                   AGREEMENTS

         NOW, THEREFORE, in order to induce the Agent to enter into the
Financing Agreement and the Construction Agreement whereby the Lenders shall
make advances under the Loan to the Borrower thereunder and for other good and
valuable consideration, the receipt and sufficiency of which are hereby
acknowledged, the Management Company and the Borrower hereby covenants and
agrees





                                       2
<PAGE>   3
with the Agent as follows:

         1.      Recitals.  The Recitals set forth above are incorporated into
this Agreement by reference.

         2.      Subordination.  (a)  To the extent provided herein, the
Management Company hereby subordinates and postpones the payment and the time
of payment of any and all compensatory payments, whether direct or indirect,
due and payable to the Management Company by, or on behalf of, the Borrower or
any Guarantor Subsidiary under, and in connection with, services rendered or to
be rendered by the Management Company under the Management Agreement,
including, without limitation, any and all fees, salaries, compensation,
commissions, bonuses and other payments and distributions due and payable
under, and in connection with, the Management Agreement, to, or for the account
or benefit of, the Management Company, but excluding, however, any and all
payments to the Management Company by the Borrower or any Guarantor Subsidiary
as reimbursement for any and all necessary and out-of-pocket reasonable costs
and expenses incurred and paid by the Management Company in connection with the
Management Company's management and operation of the Facilities to the extent
such payments are properly reimbursable in accordance with the terms and
conditions of the Management Agreement (collectively, the "Subordinated
Management Fees"), to, and in favor of, the payment and the time of payment of
all of the "Obligations" (as hereinafter defined).  As used in this Agreement,
the term "Obligations" shall mean all present and future debts, obligations and
liabilities of the Borrower or any Guarantor Subsidiary to the Lenders arising
pursuant to, and/or on account of, the provisions of the Financing Agreement,
the Construction Agreement, the Note, and any of the other Financing Documents,
including, without limitation, the obligation (i) to pay all principal
(including, without limitation, any principal advanced after the date of the
Financing Agreement and any principal that is repaid and readvanced), interest,
late charges and prepayment premiums (if any) due at any time under the Note;
(ii) to pay all Expenses (as defined in the Financing Agreement),
indemnification payments and other sums due at any time under the Financing
Documents, together with interest thereon as may be provided in the Financing
Documents; and (iii) to perform, observe and comply with all of the terms,
covenants and conditions, expressed or implied, which the Borrower or any
Guarantor Subsidiary is required to perform, observe or comply with pursuant to
the terms of the Financing Documents.

                 (b)      So long as all or any part of the Obligations remain
unpaid, the Management Company shall not, without the prior written consent of
the Agent ask, demand, sue for, set off, accept or receive any payment of all
or any part of the Subordinated Management Fees, except as provided herein.
Any and





                                       3
<PAGE>   4
all amounts paid by the Borrower to the Management Company in accordance with,
and subject to, the terms and conditions of the Management Agreement, shall be
and remain the property of the Management Company; provided, that any such
payments are paid and received in compliance with the terms and conditions of
this Agreement.  Prior to the occurrence of an Event of Default under the
Financing Documents or a default by the Management Company under the Management
Agreement, the Management Company shall be entitled to collect the Subordinated
Management Fees, as and when the same become due and payable pursuant to the
terms of the Management Agreement.

For purposes of this Agreement only, the term "Event of Default" shall mean
that one of the following events has occurred:  (i) the Borrower and the
Guarantor Subsidiary do not have sufficient net cash flow from its operations
or the operating reserves contained in the Total Development Budget to pay the
Subordinated Management Fees, (ii) the Borrower has failed to satisfy one or
more of the financial covenants set forth in the Financing Agreement, and/or
(iii) the Agent has exercised its right to accelerate payment of the Loan.

                 (c)      The Management Company agrees that the failure of the
Borrower to pay any or all of the Subordinated Management Fees as and when
required because of the provisions of this Agreement shall not in any respect
relieve or excuse the Management Company for the prompt and proper performance
and discharge of any of its duties, obligations or responsibilities under the
Management Agreement; and the failure of the Management Company to continue to
perform its duties, obligations and responsibilities under the Management
Agreement in accordance with the terms and conditions of the Management
Agreement shall constitute a default by the Management Company under the
Management Agreement.  The terms and conditions of this Agreement are
incorporated in, and made a part of, the Management Agreement.  In addition,
the Borrower and the Management Company agree not to amend, modify, supplement,
restate, rescind, terminate, waive, or otherwise change any of the terms or
conditions of the Management Agreement without the prior written consent of the
Agent, which consent shall not be unreasonably withheld.

                 (d)      The Management Company agrees not to subordinate,
assign or transfer all or any part of the Subordinated Management Fees or any
of the other rights of the Management Company under, and in connection with,
the Management Agreement, to any other person without the prior written consent
of the Agent, which consent shall not be unreasonably withheld.

         3.      Distributions, etc.  In the event of any distribution,
division or application, partial or complete, voluntary or involuntary, by
operation of law or otherwise, of all or any part





                                       4
<PAGE>   5
of the assets of the Borrower or the proceeds thereof to creditors of the
Borrower or to any indebtedness, liabilities and obligations of the Borrower,
by reason of the liquidation, dissolution or other winding up of the Borrower
or the Borrower's business, or in the event of any sale, receivership,
insolvency or bankruptcy proceeding, or assignment for the benefit of
creditors, or any proceeding by or against the Borrower for any relief under
any bankruptcy or insolvency law or other laws relating to the relief of
debtors, readjustment of indebtedness, reorganizations, compositions or
extensions, then and in any such event any payment or distribution of any kind
or character, either in cash, securities or other property, which shall be
payable or deliverable upon or with respect to all or any part of the
Subordinated Management Fees shall be paid or delivered directly to the Agent
for application to the Obligations (in such order and manner as the Agent may
elect in its sole and absolute discretion; and including, without limitation
any interest accruing subsequent to the commencement of any such event or
proceeding) until the Obligations shall have been fully paid and satisfied.  In
the event the Management Company defaults in the performance of any of the
terms and conditions of this Agreement, the Management Company hereby
irrevocably authorizes and empowers the Agent, and irrevocably appoints the
Agent as the Management Company's attorney-in-fact to demand, sue for, collect
and receive every such payment or distribution and give acquittance therefor
and to file claims and take such other proceedings in the name of the Agent or
in the name of the Management Company or otherwise, as the Agent may deem
necessary or advisable to carry out the provisions of this Agreement.  The
Management Company hereby agrees to execute and deliver to the Agent such
powers of attorney, assignments, endorsements or other instruments as may be
requested by the Agent in order to enable the Agent to enforce any and all
claims upon or with respect to the Subordinated Management Fees, and to collect
and receive any and all payments or distributions which may be payable or
deliverable at any time upon or with respect to any of the Subordinated
Management Fees.


         4.      Receipt of Payments by Management Company.  Should any payment
or distribution not permitted by the provisions of this Agreement or security
or proceeds thereof be received by the Management Company upon or with respect
to all or any part of the Subordinated Management Fees prior to the full
payment and satisfaction of the Obligations, the Management Company will
deliver the same to the Agent in precisely the form received (except for the
endorsement or assignment of the Management Company where necessary), for
application to the Obligations, first, to the payment of those Obligations
which are due and payable, second to any and all unpaid and accrued interest on
the Obligations, and then to such other Obligations which may be outstanding,
and, until so delivered, the same shall be held in





                                       5
<PAGE>   6
trust by the Management Company as property of the Agent.  In the event of the
failure of the Management Company to make any such endorsement or assignment,
the Agent, or any of its officers or employees on behalf of the Agent, is
hereby irrevocably authorized in its own name or in the name of the Management
Company to make the same, and is hereby appointed the Management Company's
attorney-in-fact for those purposes, that appointment being coupled with an
interest and irrevocable.

         5.      Consents, Waivers, etc.  The Management Company hereby
consents that at any time and from time to time and with or without
consideration, the Agent may, without further consent of or notice to the
Management Company and without in any manner affecting, impairing, lessening or
releasing any of the provisions of this Agreement, renew, extend, change the
manner, time, place and terms of payment of, sell, exchange, release,
substitute, surrender, realize upon, modify, waive, grant indulgences with
respect to and otherwise deal with in any manner: (a) all or any part of the
Obligations; (b) all or any of the Financing Documents; (c) all or any part of
any property at any time securing all or any part of the Obligations; and (d)
any person at any time primarily or secondarily liable for all or any part of
the Obligations and/or any collateral and security therefor, all as if this
Agreement did not exist.  The Management Company hereby waives demand,
presentment for payment, protest, notice of dishonor and of protest with
respect to the Obligations, notice of acceptance of this Agreement, notice of
the making of any of the Obligations and notice of default under any of the
Financing Documents.

         6.      Continuing Agreement.  This is a continuing Agreement and
shall remain in full force and effect until (a) all of the Obligations have
been performed and satisfied, (b) the Lenders have no obligation or outstanding
agreement to allow further advances or additional loans to the Borrower, and
(c) the Financing Agreement has been terminated by the Agent, or has otherwise
expired; at which time this Agreement shall automatically terminate without
further action of any kind.

         7.      Transfer or Assignment of Obligations.  If any of the
Obligations should be transferred or assigned by the Agent, this Agreement will
inure to the benefit of the respective transferee or assignee to the extent of
such transfer or assignment, provided that the Agent shall continue to have the
unimpaired right to enforce this Agreement as to any of the Obligations not so
transferred or assigned.

         8.      Further Agreements.  (a)  If the Management Company, contrary
to this Agreement, commences or participates in any action or proceeding
against the Borrower, the Borrower may interpose as a defense or dilatory plea
the making of this





                                       6
<PAGE>   7
Agreement, and the Agent may intervene and interpose such defense or plea in
the name of the Agent or in the name of the Borrower.  Should the Management
Company, contrary to this Agreement, in any way attempt to enforce payment of
the Subordinated Management Fees or any part thereof, the Agent in its own name
or in the name of the Borrower, may restrain the Management Company from so
doing, it being understood and agreed by the Management Company that (i) the
damages of the Borrower, and the Agent from its actions may at that time be
difficult to ascertain and may be irreparable, and (ii) the Management Company
waives any defense that the Borrower or the Agent cannot demonstrate damage
and/or can be made whole by the awarding of damages.

                 (b)      If the Management Company, the Borrower or both
contrary to this Agreement, make, attempt to or threaten to make any payment or
take any action contrary to this Agreement, the Agent may restrain the
Management Company and the Borrower from so doing, it being understood and
agreed by the Management Company and the Borrower that (i) the damages of the
Agent from their actions may at that time be difficult to ascertain and may be
irreparable and (ii) the Management Company and the Borrower waive any defense
or claim that the Agent cannot demonstrate damage and/or cannot be made whole
by the awarding of damages.

                 (c)      The Management Company and the Borrower agrees to
indemnify the Lenders and to hold the Lenders harmless for any and all expenses
and obligations, including attorney's fees, as they arise, relating to actions
of the Management Company, the Borrower or both taken contrary to this
Agreement.

                 (d)      Nothing herein contained shall obligate the Lenders
to grant credit to, or continue financing arrangements with, the Borrower,
except as otherwise provided in any of the Financing Documents.

                 (e)      The provisions of this Agreement are solely for the
benefit of the Lenders and their successors and assigns and there are no other
parties, persons or entities whatsoever (including, without limitation, the
Management Company, its successors and assigns and the Borrower, its successors
and assigns) who are intended to be benefitted in any manner whatsoever by this
Agreement.

                 (f)      No delay or failure on the part of the Agent to
exercise any of its rights and remedies or the rights or remedies of the
Lenders hereunder or now or hereafter existing at law or in equity or by
statute or otherwise, or any partial or single exercise thereof, shall
constitute a waiver thereof.  All such rights and remedies are cumulative and
may be exercised singly or concurrently and the exercise of any one or more of
them will not be a waiver of any other.  No waiver of any of its rights and





                                       7
<PAGE>   8
remedies hereunder, and no modification or amendment of this Agreement shall be
deemed to be made by the Lenders unless the same shall be in writing, duly
signed on behalf of the Agent, and each such waiver, if any, shall apply only
with respect to the specific instance involved and shall in no way impair the
rights and remedies of the Lenders hereunder in any other respect at any other
time.

                 (g)      This Agreement shall be binding upon the Borrower and
the Management Company and the respective successors and assigns of the
Management Company and Borrower and shall inure to the benefit of the Lenders
and their respective successors and assigns.

                 (h)      As used herein, the singular number shall include the
plural and the plural shall include the singular and the use of the masculine,
feminine or neuter gender shall include all genders, as the context may
require. As used herein, the term "person" shall include an individual, a
corporation, an association, a partnership, a trust, an organization and any
other entity.

                 (i)      The paragraph headings of this Agreement are for
convenience only, and shall not limit or otherwise affect any of the terms
hereof.

                 (j)      This Agreement shall be governed and construed in
accordance with the laws of the Commonwealth of Virginia and shall be deemed to
have been executed, delivered and accepted in the Commonwealth of Virginia.

         IN WITNESS THEREOF, the Management Company has caused this Management
Fee Subordination Agreement to be executed by its duly authorized officer and
the Borrower has caused this Management Fee Subordination Agreement to be
executed by its duly authorized general partner under seal as of the date first
written above.


ATTEST:                      SUNRISE ASSISTED LIVING MANAGEMENT, INC.
                             a Virginia corporation, as the Management Company


_/s/ Wayne G. Tatusko__      By:__/s/ James s. Pope___(SEAL)
 --------------------             -----------------
                                         James S. Pope
                                         Vice President





                                       8
<PAGE>   9
WITNESS:                      SUNRISE EAST ASSISTED LIVING LIMITED PARTNERSHIP,
                              a Virginia limited partnership, as
                              the Borrower

                              By:     Sunrise Assisted Living Investments, Inc.,
                                      its general partner


__/s/ Wayne G. Tatusko__      By: _/s/ James s. Pope____(SEAL)
  --------------------             -----------------
                                         James S. Pope
                                         Vice President





                                       9