<PAGE>   1
                                                                    EXHIBIT 4.23

                                                                       EXECUTION


                            AMERICAN RESTAURANT GROUP

                                TWELFTH AMENDMENT
                    TO AMENDED AND RESTATED CREDIT AGREEMENT



          THIS TWELFTH AMENDMENT TO AMENDED AND RESTATED CREDIT AGREEMENT (this
"AMENDMENT") is dated as of September 12, 1997, and entered into by and among
AMERICAN RESTAURANT GROUP, INC., a Delaware corporation ("COMPANY"), the
Subsidiaries of Company listed on the signature pages hereof (the "WORKING
CAPITAL BORROWERS"), the financial institutions listed on the signature pages
hereof ("LENDERS") and BANKERS TRUST COMPANY, as agent for Lenders ("AGENT"),
and, for purposes of Section 3 hereof, Local Favorite, Inc., a California
corporation, and is made with reference to that certain Amended and Restated
Credit Agreement, dated as of December 13, 1993, as amended by that certain
Limited Waiver and First Amendment to Amended and Restated Credit Agreement
dated as of March 23, 1994, that certain Second Amendment to Amended and
Restated Credit Agreement dated as of May 10, 1994, that certain Limited Waiver
and Third Amendment to Amended and Restated Credit Agreement dated as of March
17, 1995, that certain Limited Waiver and Fourth Amendment to Amended and
Restated Credit Agreement dated as of November 1, 1995, that certain Limited
Waiver and Fifth Amendment to Amended and Restated Credit Agreement dated as of
February 27, 1996, that certain Limited Waiver and Sixth Amendment to Amended
and Restated Credit Agreement dated as of August 26, 1996, that certain Limited
Waiver and Seventh Amendment to Amended and Restated Credit Agreement dated as
of September 10, 1996, that certain Eighth Amendment TO Amended and Restated
Credit Agreement dated as of February 25, 1997, that certain Limited Waiver and
Ninth Amendment to Amended and Restated Credit Agreement dated as of February
27, 1997, that certain Tenth Amendment to Amended and Restated Credit Agreement
dated as of June 25, 1997, and that certain Eleventh Amendment to Amended and
Restated Credit Agreement dated as of July 28, 1997 (the "CREDIT AGREEMENT"), by
and among Company, the Working Capital Borrowers, Lenders and Agent. Capitalized
terms used herein without definition shall have the same meanings herein as set
forth in the Credit Agreement.


                                    RECITALS

          WHEREAS, Borrowers have requested Lenders to extend the maturity date
of the Facility Letter of Credit Commitments and Lenders have agreed to make
such extension subject to the terms and conditions set forth herein;


<PAGE>   2
          NOW, THEREFORE, in consideration of the premises and the agreements,
provisions and covenants herein contained, the parties hereto agree as follows:


                                   SECTION 1.
                        AMENDMENT TO THE CREDIT AGREEMENT

1.1       AMENDMENT TO SUBSECTION 2.4: FEES

          Subsection 2.4 of the Credit Agreement is hereby amended to read in
its entirety as follows.

                     "Borrowers jointly and severally agree to pay to Agent for
           distribution to each Facility L/C Lender in proportion to that
           Facility L/C Lender's Pro Rata Share of the Facility Letter of Credit
           Commitments, an amendment fee for the period from and including
           October 15, 1997 to but excluding November 14, 1997, equal to the
           amount of the Facility Letter of Credit Commitment as of September
           12, 1997 multiplied by 1.50% per annum (calculated on the basis of a
           360-day year), such amendment fee to be due and payable in advance on
           September 12, 1997."

1.2       AMENDMENT TO SUBSECTION 2.10: FACILITY LETTERS OF CREDIT

          Subsection 2.10A of the Credit Agreement is hereby amended by
deleting all references to "October 15, 1997" contained therein and substituting
"November 14, 1997" therefor.

1.3       Amendment TO SUBSECTION 6.1: INDEBTEDNESS

          Subsection 6.1 of the Credit Agreement is amended by deleting the
reference to "October 15, 1997" in Clause (ix) thereof and substituting
"November 14, 1997" therefor.


                                   SECTION 2.
                         REPRESENTATIONS AND WARRANTIES

          In order to induce Lenders to enter into this Amendment and to amend
the Credit Agreement in the manner provided herein, each Loan Party represents
and warrants to each Lender that the following statements are true, correct and
complete:

2.1       CORPORATE POWER AND AUTHORITY.

          Each Loan Party has ALL requisite corporate power and authority to
enter into this Amendment and to carry out the transactions contemplated by, and
perform its obligations under, the Credit Agreement as amended by this Amendment
(the "AMENDED AGREEMENT").


                                        2


<PAGE>   3
2.2       AUTHORIZATION OF AGREEMENTS.

          The execution and delivery of this Amendment and the performance of
the Amended Agreement have been duly authorized by all necessary corporate
action on the part of each Loan Party.

2.3       No CONFLICT.

          The execution and delivery by each Loan Party of this Amendment and
the performance by each Loan Party of the Amended Agreement do not and will not
(a) violate any provision of any law or any governmental rule or regulation
applicable to any Loan Party, the Certificate or Articles of Incorporation or
Bylaws of any Loan Party or any order, judgment or decree of any court or other
agency of government binding on any Loan Party, (b) conflict with, result in a
breach of or constitute (with due notice or lapse of time or both) a default
under any Contractual Obligation of any Loan Party, except for conflicts,
breaches or defaults which would not singly or in the aggregate have a Material
Adverse Effect, (c) result in or require the creation or imposition of any Lien
upon any of the properties or assets of any Loan Party (other than any Liens in
favor of Collateral Agent for the benefit of Lenders and the Senior Note
Holders), or (d) require any approval of stockholders or any approval or consent
of any Person under any Contractual Obligation of any Loan Party, except for
such approvals or consents which have been obtained on or before the Twelfth
Amendment Effective Date (as hereinafter defined) or the absence of which would
not singly or in the aggregate have a Material Adverse Effect.

2.4       GOVERNMENTAL CONSENTS.

          The execution, delivery and performance by each Loan Party of this
Amendment and the Amended Agreement do not and will not require any registration
with, consent or approval of, or notice to, or action to, with or by, any
Federal, state or other governmental authority or regulatory body, other than
registrations, consents, approvals, notices and actions that have been taken or
obtained prior to the Twelfth Amendment Effective Date or the absence of which
would not have a Material Adverse Effect.

2.5       BINDING OBLIGATION.

          This Amendment and the Amended Agreement have been duly executed and
delivered by each Loan Party which is a party thereto and are the legally valid
and binding obligations of each such Loan Party, enforceable against each such
Loan Party in accordance with their respective terms, except as may be limited
by bankruptcy, insolvency, reorganization, moratorium or similar laws relating
to or limiting creditors' rights generally or by equitable principles relating
to enforceability.

2.6       INCORPORATION OF REPRESENTATIONS AND WARRANTIES.

          The representations and warranties contained in Section 4 of the
Credit Agreement and each Collateral Document are and will be true, correct and
complete in all material respects on 


                                       3


<PAGE>   4
and as of the Twelfth Amendment Effective Date to the same extent as though made
on and as of that date, except to the extent such representations and warranties
specifically relate to an earlier date, in which case they were true, correct
and complete in all material respects on and as of such earlier date.

2.7       ABSENCE OF DEFAULT.

          As of the Twelfth Amendment Effective Date, except as previously
disclosed to Lenders, no event has occurred and is continuing that would
constitute an Event of Default or a Potential Event of Default.


                                   SECTION 3.
                           ACKNOWLEDGMENT AND CONSENT

          Company is a party to the Company Guaranty pursuant to which Company
has guarantied certain Obligations under the Credit Agreement. Each Subsidiary
of Company is a party to the Subsidiary Guaranty Agreement pursuant to which
each such Subsidiary has guarantied certain Obligations under the Credit
Agreement. Each of the Loan Parties is a party to certain Collateral Documents
pursuant to which the Loan Parties have granted Liens on certain Collateral to
the Collateral Agent, for the benefit of Lenders and the Senior Note Holders.
The Company Guaranty, the Subsidiary Guaranty Agreement and the Collateral
Documents are collectively referred to herein as the "CREDIT SUPPORT DOCUMENTS".

          Each Loan Party hereby acknowledges that it has reviewed the terms and
provisions of the Credit Agreement and this Amendment and consents to the
amendment of the Credit Agreement effected pursuant to this Amendment. Each Loan
Party hereby confirms that each Credit Support Document to which it is a party
or otherwise bound and all Collateral encumbered thereby will continue to
guaranty or secure, as the case may be, to the fullest extent possible the
payment and performance of all "GUARANTIED OBLIGATIONS" and "SECURED
OBLIGATIONS", as the case may be (in each case as such terms are defined in the
applicable Credit Support Document), including without limitation the payment
and performance of all such "GUARANTIED OBLIGATIONS" or "SECURED OBLIGATIONS",
as the case may be, in respect of the Obligations now or hereafter existing
under or in respect of the Amended Agreement.

          Each Loan Party acknowledges and agrees that any of the Credit Support
Documents to which it is a party or otherwise bound shall continue in full force
and effect and that all of its obligations thereunder shall be valid and
enforceable and shall not be impaired or limited by the execution or
effectiveness of this Amendment. Each Loan Party represents and warrants that
all representations and warranties contained in the Amended Agreement and the
Credit Support Documents to which it is a party or otherwise bound are true,
correct and complete in all material respects on and as of the Twelfth Amendment
Effective Date to the same extent as though made on and as of that date, except
to the extent such representations and warranties specifically relate to an
earlier date, in which case they were true, correct and complete in all material
respects on and as of such earlier date.


                                        4


<PAGE>   5
          Each Loan Party (other than Borrowers) acknowledges and agrees that
(a) notwithstanding the conditions to effectiveness set forth in this Amendment,
such Loan Party is not required by the terms of the Credit Agreement or any
other Loan Document to consent to the amendments to the Credit Agreement
effected pursuant to this Amendment and (b) nothing in the Credit Agreement,
this Amendment or any other Loan Document shall be deemed to require the consent
of such Loan Party to any future amendments to the Credit Agreement.


                                   SECTION 4.
                                  MISCELLANEOUS

4.1       REFERENCE TO AND EFFECT ON THE CREDIT AGREEMENT AND THE OTHER LOAN
          DOCUMENTS.

                     (a) On and after the Twelfth Amendment Effective Date, each
           reference in the Credit Agreement to "this Agreement", "hereunder",
           "hereof', "herein" or words of like import referring to the Credit
           Agreement, and each reference in the other Loan Documents to the
           "Credit Agreement", "thereunder", "thereof" or words of like import
           referring to the Credit Agreement shall mean and be a reference to
           the Amended Agreement.

                     (b) Except as specifically amended by this Amendment, the
           Credit Agreement and the other Loan Documents shall remain in full
           force and effect and are hereby ratified and confirmed.

                     (c) The execution, delivery and performance of this
           Amendment shall not constitute a waiver of any existing Event of
           Default or Potential Event of Default or any provision of, or operate
           as a waiver of any right, power or remedy of Agent, the Collateral
           Agent or any Lender under, the Credit Agreement or any of the other
           Loan Documents.

4.2       FEES AND EXPENSES.

          Company acknowledges that all reasonable costs, fees and expenses as
described in subsection 9.3 of the Credit Agreement incurred by Agent and its
counsel with respect to this Amendment and the documents and transactions
contemplated hereby shall be for the account of Company.

4.3       HEADINGS.

          Section and subsection headings in this Amendment are included herein
for convenience of reference only and shall not constitute a part of this
Amendment for any other purpose or be given any substantive effect.


                                       5


<PAGE>   6
4.4       APPLICABLE LAW.

          THIS AMENDMENT SHALL BE GOVERNED BY, AND SHALL BE CONSTRUED AND
ENFORCED IN ACCORDANCE WITH, THE INTERNAL LAWS OF THE STATE OF NEW YORK
(INCLUDING WITHOUT LIMITATION SECTION 5.1401 OF THE GENERAL OBLIGATIONS LAW OF
THE STATE OF NEW YORK), WITHOUT REGARD TO CONFLICTS OF LAWS PRINCIPLES.

4.5       COUNTERPARTS; EFFECTIVENESS.

          This Amendment may be executed in any number of counterparts and by
different parties hereto in separate counterparts, each of which when so
executed and delivered shall be deemed an original, but all such counterparts
together shall constitute but one and the same instrument; signature pages may
be detached from multiple separate counterparts and attached to a single
counterpart so that all signature pages are physically attached to the same
document. This Amendment shall become effective (the "TWELFTH AMENDMENT
EFFECTIVE DATE") upon the execution of a counterpart hereof by each of the
parties hereto and receipt by Company and Agent of written or telephonic
notification of such execution and authorization of delivery thereof.



                  [Remainder of page intentionally left blank.]


                                       6


<PAGE>   7
         IN WITNESS WHEREOF, the parties hereto have caused this Amendment to be
duly executed and delivered by their respective officers thereunto duly
authorized as of the date first written above.

                                    AMERICAN RESTAURANT GROUP, INC.,
                                    a Delaware corporation 
                                    ARG ENTERPRISES, INC., 
                                    a California corporation
                                    SPECTRUM FOODS, INC., 
                                    a California corporation 
                                    SPOONS RESTAURANTS, INC.,
                                    a Texas corporation 
                                    ARG PROPERTY MANAGEMENT 
                                        CORPORATION,
                                    a California corporation
                                    GRANDY'S, INC.,
                                    a California corporation

                                       By:  /s/ WILLIAM J. MCCAFFREY, JR.
                                         -------------------------------
                                          William J. McCaffrey, Jr.
                                          Vice President and
                                          Chief Financial Officer
                                          of each of the foregoing



                                       FOR PURPOSES OF SECTION 3 ONLY:

                                       LOCAL FAVORITE, INC.,
                                       a California corporation


                                       By:  /s/ WILLIAM J. MCCAFFREY, JR.
                                          -------------------------------
                                          William J. McCaffrey, Jr.
                                          Vice President and
                                          Chief Financial Officer



                                       S-1



<PAGE>   8
                                        BANKERS TRUST COMPANY,
                                        individually as Agent and a Lender


                                        By:  /s/   MARY JO JOLLY
                                           -------------------------------
                                             Name:  Mary Jo Jolly
                                             Title: ASSISTANT/VICE PRESIDENT


                                        BANQUE NATIONALE DE PARIS,
                                        as a Lender


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:



                                        BANQUE PARIBAS,
                                        as a Lender


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:



                                       S-2



<PAGE>   9
                                        BANKERS TRUST COMPANY,
                                        individually, as Agent and a Lender


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:


                                        BANQUE NATIONALE DE PARIS,
                                        as a Lender


                                        By:  /s/ J.L. TOURNE
                                           -------------------------------
                                             Name: J.L. Tourne
                                             Title: V.P. & Deputy Manager


                                        By:   /s/ FREDERIQUE MERHAUT
                                           -------------------------------
                                             Name: Frederique Merhaut
                                             Title: V.P. & Credit Manager


                                        BANQUE PARIBAS,
                                        as a Lender


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:



                                       S-2



<PAGE>   10
                                        BANKERS TRUST COMPANY,
                                        individually, as Agent and a Lender


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:


                                        BANQUE NATIONALE DE PARIS,
                                        as a Lender


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:


                                        BANQUE PARIBAS,
                                        as a Lender


                                        By:   /s/ EDWARD V. CANALE
                                           -------------------------------
                                             Name: Edward V. Canale
                                             Title: Senior Vice President


                                        By:  /s/ ALBERT A. YOUNG JR. 
                                           -------------------------------
                                             Name:  Albert A. Young, Jr. 
                                             Title:  Vice President



                                       S-2



<PAGE>   11
                                        SWISS BANK CORPORATION
                                        LOND0N-BRANCH,
                                        as a Lender

                                        By: /s/   James Cullinane
                                           -------------------------------
                                             Name:  James Cullinane
                                             Title: Director/Distressed Debt


                                        By:   /s/  Alfred C. Kellog
                                           -------------------------------
                                             Name: Alfred C. Kellog
                                             Title: Attorney-In-Fact




                                        DRESDNER BANK AG,
                                        as a Lender


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:



                                       S-3



<PAGE>   12
                                        SWISS BANK CORPORATION
                                        CAYMAN ISLANDS BRANCH,
                                        as a Lender

                                        By:
                                           -------------------------------
                                             Name:
                                             Title:


                                        By:
                                           -------------------------------
                                             Name:
                                             Title:




                                        DRESDNER BANK AG, New York Branch
                                        and Grand Cayman Branch as a
                                        Lender


                                        By:   /s/ CHRISTOPHER E. SARISKY
                                           -------------------------------
                                             Name:Christopher E. Sarisky
                                             Title: Assistant Vice President


                                        By: /s/ JOHN W. SWEENEY
                                           -------------------------------
                                             Name: John W. Sweeney
                                             Title: Assistant Vice President



                                       S-3