<PAGE>
 
                                                                 EXHIBIT 10.3(4)

                            [LETTERHEAD OF METLIFE]

March 28, 1997

Chart House, Inc.
115 South Acacia Avenue
Solana Beach, CA  92075

Attention:  William R. Kuntz, Jr.
            Executive Vice President and Chief Financial Officer

Gentlemen:

Reference is hereby made to (x) the Note Purchase and Guarantee Agreement, dated
as of December 30, 1993 (as amended, the "6.69% Note Agreement"), among Chart
House, Inc. (the "Company"), the Guarantors and Metropolitan Life Insurance
Company ("MetLife"), pursuant to which the Company issued its 6.69% Senior
Secured Notes due 2001 (as amended, the "6.69% Notes"), and (y) the Amended and
Restated Note Purchase and Guarantee Agreement, dated as of December 30, 1993
(as amended, the "10.40% Note Agreement" and, collectively with the 6.69% Note
Agreement, the "Note Agreements"), among the Company, the Guarantors and
MetLife, pursuant to which there is outstanding the Company's 10.40% Senior
Secured Notes due 2000 (as amended, the "10.40% Notes" and, collectively with
the 6.69% Notes, the "Notes").  Capitalized terms not otherwise defined herein
shall have the meanings ascribed thereto in the Note Agreements.

As holder of the Notes and party to the Note Agreements, MetLife hereby consents
and agrees that, effective as of the Effective Date (as hereinbelow defined):

     a)  the due date for (i) the $3,000,000 mandatory principal prepayment with
respect to the 6.69% Notes originally payable on January 2, 1997 pursuant to
(S)4.2 of the 6.69% Note Agreement and presently payable on the earlier of (x)
October 1, 1997 and (y) the date of execution of an agreement by the Company
effecting a refinance of all or a portion of the Company's indebtedness to the
Banks or any thereof under the Bank Agreement (the earlier of such dates, the
"Deferred Date"), and (ii) the $3,000,000 mandatory principal prepayment with
respect to the 10.40% Notes originally payable on July 24, 1997 pursuant to
(S)4.2 of the 10.40% Note Agreement and presently payable on the Deferred
<PAGE>
 
Date, shall in each case be modified (and (S)4.2 of the respective Note
Agreements shall be deemed amended mutatis mutandis to reflect such
                                   ------- --------
modification) to be March 31, 1997, and the Company's failure to make such
mandatory principal prepayments with respect to the Notes, together with accrued
interest thereon, on March 31, 1997 shall constitute an Event of Default under
(S)(S)5.1(a) and/or (b) of the respective Note Agreements (after the applicable
grace periods therein have expired);

     b)  the Notes shall be subject to mandatory prepayment (pro rata as between
the 6.69% Notes and the 10.40% Notes) upon any reduction of the Revolving Credit
Commitment Amount (as defined in the Bank Agreement) from and after March 31,
1997 in a pro rata dollar amount with such reduction (and (S)(S)4.2 of the
respective Note Agreements shall be deemed amended mutatis mutandis to reflect
                                                   ------- --------
such mandatory prepayment requirement);

     c)   (S)7.6 of the Note Agreements shall be amended to read in its entirety
as follows:

     "(S)7.6.  Ratio of Consolidated Total Liabilities to Consolidated Tangible
               ----------------------------------------------------------------
     Net Worth.  The Company will not permit the ratio of Consolidated Total
     ---------
Liabilities to Consolidated Tangible Net Worth to exceed 1.25:1 at any time.";

     d)  (S)7.8 of the Note Agreements shall be amended to read in its entirety
as follows:

     "(S)7.8.  Capital Expenditures.  The Company will not (a) make capital
               --------------------
     expenditures for any purpose other than maintenance and repair of existing
     properties and assets and for improvements, or (b) permit the aggregate
     capital expenditures of the Parent and its Subsidiaries, determined on a
     consolidated basis in accordance with generally accepted accounting
     principles (but exclusive of expenditures in respect of Capitalized
     Leases), to exceed (i) $12,000,000 in the 1996 calendar year, (ii)
     $7,000,000 in the 1997 calendar year, (iii) $2,000,000 for the first
     calendar quarter of 1998 or (iv) $0 thereafter.";

     e)   (S)7.32 of the Note Agreements shall be amended to read in its
entirety as follows:

     "(S)7.32.  Minimum EBITDA.  The Company will not permit EBITDA of the
                --------------
     Company and its consolidated Subsidiaries to be less than:

     For the Fiscal Quarter Ending    Required EBITDA
     -----------------------------    ---------------
     December 31, 1996                $ 2,400,000

                                       2
<PAGE>
 
           March 31, 1997                         $ 2,400,000
 
           June 30, 1997                          $ 3,800,000
 
           September 30, 1997                     $ 4,400,000
 
           December 31, 1997                      $ 2,600,000
 
           March 31, 1998                         $2,700,000.
 
           For purposes of this (S)7.32 the term `EBITDA' shall mean
           for each fiscal quarter the sum of (a) net income (or net loss),
           including, without limitation, interest income on the Florida
           Note and the CA/AZ Note (as those terms are defined in that
           certain Consent to Disposition and Agreement for Substitution
           of Collateral dated as of May 14, 1996), plus (b) all amounts
           treated as expenses for interest, amortization, depreciation,
           taxes (to the extent included in the determination of net
           income (or net loss)) and other non-cash charges for such
           fiscal quarter.";

           f) (S)(S)7.7 and 7.31 of the Note Agreements need not be complied
with by the Company for the fiscal quarters ending December 31, 1997 and March
31, 1998, but such sections shall be complied with thereafter.

By their execution hereof, the Company and each of the Guarantors affirms and
agrees that (a) the execution and delivery by such Persons of, and the
performance of their respective obligations under, this consent and agreement
shall not in any way amend, impair, invalidate or otherwise affect any of such
Person's obligations or the rights, remedies and powers of MetLife under the
Loan Documents, including, without limitation, the Security Documents, as the
same are amended hereby, and (b) all Loan Documents remain in full force and
effect.

By their execution hereof, each of the Company, the Parent and Big Wave,
severally and independently as to itself only, represents and warrants that at
the date hereof and at and as of the Effective Date:

     (a)  Except to the extent such were by their terms made solely as of a
prior date, the representations and warranties of such party contained in the
Loan Documents are accurate and complete in all material respects.

     (b)  The execution and delivery by such party of this consent and agreement
and the performance by such party of its obligations hereunder are within the
corporate power of such party, have been (or as of the Effective Date will be)
duly authorized by all necessary corporate action and do not and will not

                                       3
<PAGE>
 
(1) contravene any provision of such party's charter, other incorporation
papers, by-laws or any stock provisions, or any amendment thereof, (2) conflict
with, or result in a breach of any material term, condition or provision of, or
constitute a default under, or result in the creation of any mortgage, lien,
pledge, charge, security interest or other encumbrance upon any of the property
of any of such parties pursuant to, any agreement, deed of trust, indenture,
mortgage or other instrument to which such party is a party or by which any of
its properties are bound, (3) violate or contravene any provision of any law,
regulation, order, ruling or interpretation thereunder or any decree, order or
judgment of any court or governmental or regulatory authority, bureau, agency or
official, (4) require any waiver, consent or approval of any Person other than
such as have been obtained and copies of which have been provided to MetLife, or
(5) require any approval, consent, order, authorization or license by, or giving
of notice to, or taking any other action with respect to, any governmental or
regulatory authority or agency under any provision of law, except those actions
which have been taken or will be taken prior to the Effective Date.

     (c)  This consent and agreement constitutes the legal, valid and binding
obligation of such party enforceable against such party in accordance with its
terms.

     (d)  No Default or Event of Default has occurred or is continuing or will
occur as a result of (1) the execution and delivery of this consent and
agreement, or (2) the consummation of the transactions contemplated hereby.

This consent and agreement shall become  effective upon the date (the "Effective
Date") upon which there shall have been delivered to MetLife each of the
following:

     (a)  a copy or counterpart copies of this consent and agreement, duly
executed by the Company and the Guarantors;

     (b)  a duly executed copy of a Sixth Amendment to Credit Agreement from the
Banks, substantially in the form of Annex 1 hereto, which Sixth Amendment to
Credit Agreement shall be in full force and effect, together with evidence
satisfactory to MetLife of such fact;

     (c)  evidence satisfactory to MetLife that none of the Banks, the Security
Agent or the Agent (as defined in the Bank Agreement), in providing the Sixth
Amendment to Credit Agreement referred to above, has received or will receive
any compensation therefor, by way of amendment or waiver fee or otherwise, or if
in fact any of the Banks, the Security Agent or the Agent has received or will
receive such compensation, an equivalent payment is made to MetLife;

                                       4
<PAGE>
 
     (d)  a counterpart of this consent and agreement, acknowledged and agreed
to by the Agent and the Banks; and

     (e)  from each of Big Wave, the Company and the Parent, certified copies of
such corporate resolutions and authorizations as are furnished to the Agent or
the Banks or as MetLife may reasonably request.

In the event the Effective Date shall not have occurred on or before March 31,
1997, then this consent and agreement shall, at the election of MetLife, as
evidenced by written notice of such election delivered by MetLife to the
Company, terminate and be of no further force or effect.  By its execution
hereof, the Company agrees to promptly notify MetLife if the Sixth Amendment to
Credit Agreement referred to above is terminated pursuant to paragraph 6
thereof.

The representations, warranties, covenants and agreements of the Company, the
Parent and Big Wave set forth herein shall survive the Effective Date.

This consent and agreement shall be governed by and construed in accordance with
the laws of the State of New York.

                                       5
<PAGE>
 
This consent and agreement may be executed in counterparts and such counterparts
shall, when taken together, constitute one and the same agreement.

Very truly yours,

METROPOLITAN LIFE INSURANCE COMPANY

By: /s/ JACQUELINE JENKINS
   ---------------------------------


ACKNOWLEDGED AND AGREED TO
this 28th day of March, 1997.


CHART HOUSE, INC.

By: /s/ JAMES C. WENDLER
    -------------------------------
Name: James C. Wendler
      -----------------------------
Title: Assistant Secretary
      -----------------------------
 

CHART HOUSE ENTERPRISES, INC.

By: /s/ JAMES C. WENDLER
    -------------------------------
Name: James C. Wendler
     ------------------------------
Title: Vice President
      -----------------------------


BIG WAVE, INC.

By: /s/ JAMES C. WENDLER
    --------------------------------
Name: James C. Wendler
     -------------------------------
Title: Treasurer and Asst. Secretary
      ------------------------------


SANWA BANK CALIFORNIA
as Agent and Security Agent

By: /s/ DAVID BEALL
    --------------------------------
Name:  David Beall
      ------------------------------
Title: Vice President
      ------------------------------

                                       6
<PAGE>
 
SANWA BANK CALIFORNIA

By: /s/ DAVID BEALL
    --------------------------------
Name: David Beall
      ------------------------------
Title: Vice President
      ------------------------------


THE FIRST NATIONAL BANK OF BOSTON

By:  /s/ THOMAS F. FARLEY, JR.
    --------------------------------
Name: Thomas F. Farley, Jr.
     -------------------------------
Title: Director
      ------------------------------


THE SUMITOMO BANK OF CALIFORNIA

By: /s/ MATTHEW VAN STEENHUYSE
   ---------------------------------
Name: Matthew Van Steenhuyse
     -------------------------------       
Title: Vice President
      ------------------------------

                                       7