<PAGE>
 
                                                                 EXHIBIT 10.1(6)

August 14, 1996


CHART HOUSE, INC., a Delaware corporation
115 South Acacia Avenue
Solana Beach, California 92075
Attn: William R. Kuntz, Jr.,
      Vice President, Chief Financial Officer and General Counsel


      Re:    Waiver of Specified Defaults; Amendment of Credit Agreement
             -----------------------------------------------------------

Ladies and Gentlemen:

     Reference is hereby made to that certain Amended and Restated Revolving
Credit and Term Loan Agreement dated as of December 17, 1993 (as amended to
date, the "Credit Agreement," and with capitalized terms not otherwise defined
herein used with the meanings given such terms in the Credit Agreement) by and
among Chart House, Inc. (the "Company"), the Guarantors, the Banks and Sanwa
Bank California, as successor to The First National Bank of Boston, as the Agent
for the Banks.

     There exist certain Defaults under the Credit Agreement consisting of the
breach by the Company of certain financial covenants. The Company has requested
the Agent and the Banks to waive such Defaults and the Agent and the Banks have
agreed to do so on the terms and subject to the conditions set forth below:

     1.  Waiver.  On the terms and subject to the conditions set forth herein,
         ------                                                               
effective as of the Effective Date (as defined in Paragraph 8 below), the Agent
and each of the Banks hereby waive the Defaults existing under the Credit
Agreement by virtue of the failure of the Company to have been in compliance
with the financial covenants set forth in (S)11.7 and (S)11.32 thereof as of
July 1, 1996.

     2.  Reduction of Revolving Credit Commitment Amount.  To reflect the
         -----------------------------------------------                 
agreement of the Agent, the Banks and the Company to the reduction of the
Revolving Credit Commitment Amount, effective as of the Effective Date the
definition of "Revolving Credit Commitment Amount" set forth in (S)1 of the
Credit Agreement is hereby amended to read in its entirety as follows:

         "Revolving Credit Commitment Amount.  At any date of determination,
           ----------------------------------                                
     $24,000,000, as such amount is reduced pursuant to (S)2.2 and (S)6.4
     hereof."

                                       1
<PAGE>
 
     3.  Additional Financial Information.  Pursuant to subsection (g) of
         --------------------------------                                
(S)11.9 of the Credit Agreement, the Banks hereby request, and the Company
hereby agrees to provide to the Banks, the following information:

         (a) No later than September 15, 1996, an updated forecast, by month,
consisting of balance sheet, profit and loss and cash flow of the Parent and its
consolidated Subsidiaries for each of September, October, November and December
1996; and

         (b) Within 30 days following the end of each fiscal quarter of the
Parent, consolidated statements of income, balance sheets and cash flow
statements of the Parent and its consolidated Subsidiaries, accompanied in each
case by the Certificates referred to in subsections (c) and (d) of (S)11.9 of
the Credit Agreement.

     4.  Use of Proceeds.  To reflect the agreement of the Company and the
         ---------------                                                  
Guarantors to restrict the use of proceeds of the Revolving Credit Loans,
effective as of the Effective Date (S)11.3 of the Credit Agreement is hereby
amended to read in its entirety as follows:

          "(S)11.3  Use of Revolving Credit Loan Proceeds.  The Company and the
                    -------------------------------------                      
     Guarantors shall use the proceeds of the Revolving Credit Loans to fund
     capital expenditures relating to the maintenance of existing properties and
     assets and completion of pending minor remodeling projects for which
     contractual commitments were made prior to August 14, 1996 and for working
     capital purposes."

     5.  Restriction on Capital Expenditures.  To reflect the agreement of the
         -----------------------------------                                  
Company and the Parent to limit the dollar amount and nature of capital
expenditures of the Parent and its Subsidiaries during calendar year 1996,
effective as of the Effective Date (S)11.8 of the Credit Agreement is hereby
amended to insert the following proviso immediately preceding the period in the
last line thereof:

     "; provided, however, that notwithstanding anything contained herein,
     capital expenditures of the Parent and its Subsidiaries during the 1996
     calendar year shall not exceed $11,000,000 in the aggregate and in no event
     shall at any time following August 15, 1996 funds of the Parent or any of
     its Subsidiaries, including, without limitation, proceeds of the Revolving
     Credit Loans, be utilized for capital expenditures other than expenditures
     for the purpose of the maintenance of existing properties and assets and
     completion of pending minor remodeling projects for which contractual
     commitments were made prior to August 14, 1996."

     6.  Approval of Disposition of Assets.  To reflect the agreement of the
         ---------------------------------                                  
Agent and the Banks with respect to the sale of the assets of the recently
closed Long Beach Chart House, effective as of the Effective Date:

                                       2
<PAGE>
 
         (a) The Agent and the Banks hereby approve the sale thereof at not less
than fair market value; and

         (b) The Agent and the Banks agree that the Net Proceeds of such sale
need not be applied to reduce the principal balance of the Loans as required
pursuant to (S)6.4 of the Credit Agreement.

     7.  Reaffirmation of Obligations; No Additional Waiver or Amendment.  The
         ---------------------------------------------------------------      
Company and the Parent and the Guarantors, by executing this letter in the space
provided below, reaffirm all of their respective obligations under the Credit
Agreement and the other Loan Documents and the Guaranties, as applicable, and
all documents, instruments and agreements executed in connection therewith or
otherwise relating thereto and acknowledge and agree that such obligations are
not altered, modified or affected in any manner or to any extent except to the
extent expressly provided herein.  The Company, the Parent and the Guarantors
further acknowledge that the waiver set forth in Paragraph 1 above is the sole
waiver made by the Agent and the Banks with respect to the existence of Defaults
(or possible occurrence or existence of additional Defaults or Events of
Defaults) under the Credit Agreement and the other Loan Documents and the
Guaranties, regardless of whether the Agent or the Banks are aware of, or would
upon investigation have discovered, the existence of such additional Defaults or
Events of Default.

     8.  Effective Date.  This letter agreement, including, without limitation,
         --------------                                                        
the waiver of the Agents and the Banks set forth in Paragraph 1 above, shall be
effective on the date (the "Effective Date") upon which there shall have been
delivered each of the following:

         (a) To the Agent for distribution to the Banks, a copy of this letter
agreement, duly executed by all signatories set forth below (which may be
counterpart originals); and

         (b) To the Agent for distribution to the Banks pro rata in accordance
with their Commitment Percentages, a waiver fee in the amount of $25,000.

If the Effective Date shall not have occurred on or before August 15, 1996, then
the agreements of the Agent and the Banks set forth herein shall, at the option
of the Agent and the Banks as evidenced by written notice to such effect
delivered by the Agent to the Company, terminate and be of no further force or
effect.

     9.  Authorization; Representations and Warranties.  The Company, the Parent
         ---------------------------------------------                          
and the Guarantors hereby represent and warrant to the Agent and the Banks as
follows:

         (a) The execution, delivery and performance of this letter agreement by
the officers of each of said parties executing the same has been duly authorized
by all 

                                       3
<PAGE>
 
necessary corporate action (each of said parties hereby committing to
deliver to the Agent no later than 10 days following the Effective Date
corporate resolutions and incumbency certificates evidencing the accuracy of
such representation and warranty); and

         (b) Upon the Effective Date and after giving effect to the waiver set
forth in Paragraph 1 above, all representations and warranties of the Company
and the Guarantors contained in the Credit Agreement are accurate and complete
in all material respects on and as of the Effective Date.

     10. Counterparts.  This letter agreement may be executed in counterparts,
         ------------                                                         
all of which taken together shall constitute one and the same instrument.

                         Very truly yours,

                         SANWA BANK CALIFORNIA,
                         as Agent and a Bank



                         By: /s/ DAVID L. BEALL
                            ------------------------------------------
                            David L. Beall, Vice President and Manager


                         THE FIRST NATIONAL BANK OF BOSTON,
                         as a Bank



                         By: /s/ THOMAS F. FARLEY, JR.
                            -------------------------------------
                         Name: Thomas F. Farley, Jr.
                              -----------------------------------
                         Title: Director
                               ----------------------------------


                         THE SUMITOMO BANK OF CALIFORNIA,
                         as a Bank



                         By: /s/ MATTHEW R. VAN STEENHUYSE
                            -------------------------------------
                         Name: Matthew R. Van Steenhuyse
                               ---------------------------------- 
                         Title: Vice President
                               ----------------------------------

                                       4
<PAGE>
 
ACKNOWLEDGED AND AGREED TO
this 14 day of August, 1996


CHART HOUSE, INC.



By: /s/ WILLIAM R. KUNTZ, JR.
   ----------------------------
Name: William R. Kuntz, Jr.
     -------------------------- 
Title: Executive Vice President
      -------------------------


CHART HOUSE ENTERPRISES, INC.


By: /s/ WILLIAM R. KUNTZ, JR.
   ----------------------------
Name: William R. Kuntz, Jr.
     -------------------------- 
Title: Executive Vice President
      -------------------------


BIG WAVE, INC. (formerly known as Islands Restaurants, Inc.)


By: /s/ WILLIAM R. KUNTZ, JR.
   ----------------------------
Name: William R. Kuntz, Jr.
     -------------------------- 
Title: Vice President
      -------------------------

                                       5