<PAGE>   1
                                                                    EXHIBIT 4.20

                                                                  EXECUTION COPY


===============================================================================





                        AMERICAN RESTAURANT GROUP, INC.

                                      AND

                             SUBSIDIARY GUARANTORS

                                      AND

                     U.S. TRUST COMPANY OF CALIFORNIA, N.A.
                                    Trustee




                           _________________________




                              SECOND SUPPLEMENTAL
                                   INDENTURE


                           Dated as of March 13, 1997





===============================================================================



<PAGE>   2

                 SECOND SUPPLEMENTAL INDENTURE dated as of March 13, 1997 among
American Restaurant Group, Inc., a Delaware corporation (the "Company"), the
Subsidiary Guarantors and U.S. Trust Company of California, N.A., a national
banking association, as trustee (the "Trustee") under the Indenture dated as of
December 1, 1993, as amended by a First Supplemental Indenture dated as of
August 28, 1996 (the "Indenture").


                                    RECITALS

                 WHEREAS, pursuant to the Indenture, the Company issued
$50,000,000 original aggregate principal amount of its Senior Secured Notes due
September 15, 1998 (the "Securities");

                 WHEREAS, Section 9.02 of the Indenture provides, among other
things, that the Company, the Subsidiary Guarantors and the Trustee may, with
the consent of the holders of at least a majority in principal amount of the
then outstanding Securities, amend the Indenture and the Securities in certain
respects;

                 WHEREAS, the Company, pursuant to the foregoing authority,
proposes in and by this Second Supplemental Indenture to amend the Indenture
and the Securities in certain respects; and

                 WHEREAS, all things necessary to make this Second Supplemental
Indenture a valid agreement of the Company, the Subsidiary Guarantors and the
Trustee and a valid amendment of and supplement to the Indenture and the
Securities have been done.

                 NOW, THEREFORE, THIS SECOND SUPPLEMENTAL INDENTURE WITNESSETH:

                 For and in consideration of the premises, the Company, the
Subsidiary Guarantors and the Trustee agree as follows:


                                  ARTICLE ONE

                 AMENDMENTS TO THE INDENTURE AND THE SECURITIES

                 SECTION 1.01.  Amendment and Restatement of Section 4.04.
Section 4.04 of the Indenture is hereby amended and restated in its entirety as
follows:

                 "SECTION 4.04.  Maintenance of Consolidated EBITDA.

                                  (a)  The Company shall not permit the sum of
                 (i) Consolidated EBITDA for any period of four consecutive
                 fiscal quarters ending on any date set forth below plus (ii)
                 the Aggregate EBITDA Adjustment Amount as of such date to be
                 less than the amount set forth opposite such date:


<PAGE>   3
                                                                               2

<TABLE>
<CAPTION>
                       Date                                     Amount                  
                       ----                                  -----------                  
                  <S>                                        <C>                        
                  June 30, 1997                              $35,000,000                
                  September 29, 1997                          35,500,000                
                  December 29, 1997                           36,000,000                
                  March 30, 1998                              36,500,000                
                  June 29, 1998                               37,000,000                
</TABLE>

                                  (b)  The Company shall not permit the sum of
                 (i) Consolidated EBITDA for the period of twelve consecutive
                 calendar months ending May 31, 1997 plus (ii) the Aggregate
                 EBITDA Adjustment Amount as of such date to be less than
                 $34,000,000.

                                  (c)  The Company shall not permit the sum of
                 (i) Consolidated EBITDA for the period of two consecutive
                 fiscal quarters ending September 23, 1996 plus (ii) 50% of the
                 Aggregate EBITDA Adjustment Amount as of the last day of such
                 period to be less than $14,000,000.

                                  (d)  The Company shall not permit the sum of
                 (i) Consolidated EBITDA for any period of two consecutive
                 fiscal quarters ending on or after June 30, 1997 plus (ii) 50%
                 of the Aggregate EBITDA Adjustment Amount as of the last day
                 of such period to be less than $13,000,000."

                 SECTION 1.02.  Amendment to Section 4.08.  Section 4.08 of the
Indenture is hereby amended by (a) adding the following sentence to the end of
paragraph (a) thereof:

                 "In addition, no later than June 30, 1997, the Company shall
                 deliver to the Trustee an Officers' Certificate setting forth
                 in reasonable detail a calculation of the sum of (i)
                 Consolidated EBITDA for the period of twelve consecutive
                 calendar months ending on May 31, 1997 plus (ii) the Aggregate
                 EBITDA Adjustment Amount as of such date."

and (b) adding a new paragraph (d) to the end thereof which shall read in its
entirety as follows:

                 (d)  The Company, upon request of any Holder, shall report in
                 reasonable detail regarding the status of the Company's
                 efforts to sell (the "Black Angus Transaction") all or
                 substantially all of the assets constituting its Black Angus
                 division (or the Capital Stock of Subsidiaries holding such
                 assets), provided that the Company will not, pursuant to such
                 reports, be required to provide non-public information with
                 respect to the Black Angus Transaction except in the case of
                 any such reports requested on or after July 1, 1997.





<PAGE>   4
                                                                               3



                 SECTION 1.03.  Amendment to Section 4.25.  Section 4.25 of the
Indenture is hereby amended by changing the amount "$25,000,000" contained
therein to the amount "$15,000,000".

                                  ARTICLE TWO

                                 MISCELLANEOUS

                 SECTION 2.01.  Conditions Precedent; Reaffirmation of
Subsidiary Guarantee.  The effectiveness of this Second Supplemental Indenture
is conditioned upon (a) the receipt by the Trustee of (i) counterparts hereof
executed and delivered by the Company and each Subsidiary Guarantor and (ii) a
satisfactory opinion of counsel stating that this Second Supplemental Indenture
complies with the provisions of Section 9.02 of the Indenture and covering
other customary corporate matters and (b) the payment by the Company of any
invoiced amounts referred to in Section 2.10.  By its execution and delivery of
this Second Supplemental Indenture, each Subsidiary Guarantor reaffirms and
restates its obligation set forth in Article 11 of the Indenture.

                 SECTION 2.02.  Incorporation of Indenture.  All the provisions
of this Second Supplemental Indenture shall be deemed to be incorporated in,
and made a part of, the Indenture; and the Indenture, as supplemented and
amended by this Second Supplemental Indenture, shall be read, taken and
construed as one and the same instrument.

                 SECTION 2.03.  Headings.  The headings of the Articles and
Sections of this Second Supplemental Indenture are inserted for convenience of
reference and shall not be deemed to be a part thereof.

                 SECTION 2.04.  Counterparts.  This Second Supplemental
Indenture may be executed in any number of counterparts, each of which so
executed shall be deemed to be an original, but all such counterparts shall
together constitute but one and the same instrument.

                 SECTION 2.05.  Conflict with Trust Indenture Act.  If any
provision hereof limits, qualifies or conflicts with another provision hereof
which is required to be included in this Second Supplemental Indenture by any
of the provisions of the Trust Indenture Act, such required provision shall
control.

                 SECTION 2.06.  Successors.  All covenants and agreements in
this Second Supplemental Indenture by the Company and each Subsidiary Guarantor
shall bind their respective successors.  All covenants and agreements of the
Trustee in this Second Supplemental Indenture shall bind its successor.

                 SECTION 2.07.  Separability Clause.  In case any provision in
this Second Supplemental Indenture shall be invalid, illegal or unenforceable,
the validity, legality and enforceability of the remaining provisions shall not
in any way be affected or impaired thereby.





<PAGE>   5
                                                                               4



                 SECTION 2.08.  Benefits of Second Supplemental Indenture.
Nothing in this Second Supplemental Indenture, express or implied, shall give
to any person, other than the parties hereto and their successors hereunder and
the Holders, any benefit or any legal or equitable right, remedy or claim under
this Second Supplemental Indenture.

                 SECTION 2.09.  Terms Defined.  All terms defined elsewhere in
the Indenture have the same meanings herein.

                 SECTION 2.10.  Expenses.  The Company agrees to pay the
reasonable costs and expenses of the Trustee and the Holders, including the
reasonable fees and disbursements (not to exceed $25,000) of one firm of
counsel to certain of the Holders, in connection with the negotiation and
execution of this Second Supplemental Indenture.

                 SECTION 2.11.  Acknowledgement of Waiver, etc.  The Company
and the Trustee hereby acknowledge that, in connection with the Consent
Solicitation Statement of the Company dated February 14, 1997, as supplemented
by a letter from the Company dated February 25, 1997, relating to this Fourth
Supplemental Indenture (the "February 1997 Consent Solicitation Statement"),
the holders of a majority in principal amount of outstanding Securities have
waived compliance with (a) any Default or Event of Default under Section 4.04
or 4.25 of the Indenture (as in effect prior to giving effect to this Fourth
Supplemental Indenture), (b) the Consent Period requirement of Section 9.02 of
the Indenture and (c) any restrictions contained in the Indenture on exchanging
the existing global Securities for new global Securities reflecting the
Proposed Modifications (including, to the extent applicable, the Principal
Increase) (as each such term is defined in the February 1997 Consent
Solicitation Statement).  The Company hereby advises the Trustee that the
aggregate stated face amount of Consenting Notes (as defined in the February
1997 Consent Solicitation Statement) is equal to $45,500,000.  Accordingly, as
provided in the February 1997 Consent Solicitation Statement, as of the date
hereof, (i) the aggregate stated face amount of the Principal Increase shall
equal $455,000 and (ii) after giving effect to the Principal Increase, (A) the
aggregate stated face amount of the Securities shall equal $50,455,000 and (B)
the aggregate outstanding amount of the Securities shall equal $42,193,498.
The Company hereby instructs the Trustee to take such actions as may be
necessary to effectuate the transactions described in clause (c) of the first
sentence of this Section 2.11.





<PAGE>   6
                                                                               5



                 IN WITNESS WHEREOF, the parties hereto have caused this Second
Supplemental Indenture to be duly executed, all as of the day and year first
above written.



                                        AMERICAN RESTAURANT GROUP, INC.


                                        By:  /s/  WILLIAM J. McCAFFREY, JR.
                                           ----------------------------------
                                           Name:  William J. McCaffrey, Jr.
                                           Title: Vice President and
                                                  Chief Financial Officer

                                        U.S. TRUST COMPANY OF CALIFORNIA, N.A.


                                        By:  /s/  SANDEE PARKS
                                           ----------------------------------
                                           Name:  Sandee Parks
                                           Title: Vice President


                                        SUBSIDIARY GUARANTORS:

                                        ARG ENTERPRISES, INC.
                                        SPECTRUM FOODS, INC.
                                        SPOONS RESTAURANTS, INC.
                                        ARG PROPERTY MANAGEMENT CORPORATION
                                        GRANDY'S, INC.
                                        LOCAL FAVORITE, INC.


                                        By:  /s/  WILLIAM J. McCAFFREY, JR.
                                           ----------------------------------
                                           Name:  William J. McCaffrey, Jr.
                                           Title: Vice President and
                                                  Chief Financial Officer
                                           (for each of the above-listed
                                           Subsidiary Guarantors)