<PAGE>

                                                                   EXHIBIT 10.20

                            SUBORDINATION AGREEMENT

         SUBORDINATION AGREEMENT (this "Agreement"), dated as of November __,
2000, between FLEET NATIONAL BANK (formerly known as BankBoston, N.A.), a
national banking association having its office at 100 Federal Street, Boston,
Massachusetts 02110, in its capacity as agent (the "Agent") for the Banks (as
hereinafter defined), SAMSTOCK, L.L.C., a Delaware limited liability company
having a principal of business at Two North Riverside Plaza, Suite 600, Chicago,
Illinois 60606 (the "Subordinating Creditor"), CHART HOUSE, INC., a Delaware
corporation having its office at 640 North LaSalle Street, Suite 295, Chicago,
Illinois 60610 (the "Borrower"), Chart House Enterprises, Inc. (the "Parent")
and each of the Subsidiaries of the Borrower set forth on the signature pages
hereto as Guarantors.

         WHEREAS, pursuant to a Revolving Credit and Term Loan Agreement, dated
as of April 26, 1999 (as amended and in effect from time to time, including any
replacement agreement therefor, the "Credit Agreement"), among the lending
institutions party thereto (the "Banks"), the Agent, the Borrower and the
Parent, the Banks have agreed, upon the terms and subject to the conditions
contained therein, to make loans and otherwise to extend credit to the Borrower;
and

         WHEREAS, the Subordinating Creditor has extended or agreed to extend
credit to the Borrower pursuant to a Subordinated Promissory Note and Guaranty
dated as of even date herewith (as amended with the consent of the Agent as
provided herein and in effect from time to time, the "Subordinated Note"),
executed by the Borrower and the Parent in favor of the Subordinating Creditor;
and

         WHEREAS, it is a condition precedent to the Banks' willingness to make
loans and otherwise to extend credit to the Borrower pursuant to the Credit
Agreement that the Borrower and the Subordinating Creditor enter into this
Agreement with the Agent; and

         WHEREAS, in order to induce the Banks to make loans and otherwise
extend credit to the Borrower pursuant to the Credit Agreement, the Borrower and
the Subordinating Creditor have agreed to enter into this Agreement with the
Agent;

         NOW, THEREFORE, in consideration of the foregoing, the mutual
agreements herein contained and other good and valuable consideration, the
receipt and adequacy of which are hereby acknowledged, the parties hereto,
intending to be legally bound, hereby agree as follows:

         1.    Definitions. Terms not otherwise defined herein have the same
               -----------
respective meanings given to them in the Credit Agreement. In addition, the
following terms shall have the following meanings:

         Senior Debt. All principal, interest, fees, costs, enforcement expenses
         -----------
(including legal fees and disbursements), collateral protection expenses and
other reimbursement
<PAGE>

                                      -2-


or indemnity obligations created or evidenced by the Credit Agreement or any of
the other Loan Documents or any prior, concurrent, or subsequent notes,
instruments or agreements of indebtedness, liabilities or obligations of any
type or form whatsoever relating thereto in favor of the Agent or any of the
Banks. Senior Debt shall expressly include any and all interest accruing or out
of pocket costs or expenses incurred after the date of any filing by or against
the Borrower of any petition under the federal Bankruptcy Code or any other
bankruptcy, insolvency or reorganization act regardless of whether the Agent's
or any Bank's claim therefor is allowed or allowable in the case or proceeding
relating thereto.

         Subordinated Debt. All principal, interest (including interest accrued
         -----------------
pursuant to the Subordinated Note), fees, costs, enforcement expenses (including
legal fees and disbursements), collateral protection expenses and other
reimbursement and indemnity obligations created or evidenced by the Subordinated
Note, the Subordinated Guaranty or any prior, concurrent or subsequent notes,
instruments or agreements of indebtedness, liabilities or obligations of any
type or form whatsoever relating thereto in favor of the Subordinating Creditor.

         Subordinated Documents. Collectively, the Subordinated Note, the
         ----------------------
Subordinated Guaranty, any promissory notes executed in connection therewith and
any and all guaranties and security interests, mortgages and other liens
directly or indirectly guarantying or securing any of the Subordinated Debt, and
any and all other documents or instruments evidencing or further guarantying or
securing directly or indirectly any of the Subordinated Debt, whether now
existing or hereafter created.

         Subordinated Guaranty. The Subordinated Guaranty, dated the date
         ---------------------
hereof, made by each Subsidiary of the Borrower in favor of the Subordinating
Creditor.

         2.    General. The Subordinated Debt and any and all Subordinated
               -------
     Documents shall be and hereby are subordinated and the payment thereof is
     deferred until the full and final payment in cash of the Senior Debt,
     whether now or hereafter incurred or owed by the Borrower. The parties
     hereby acknowledge and agree that the Agent and the Subordinating Creditor
     have discussed circumstances in which the Subordinated Debt could be
     prepaid upon the consummation of a recapitalization of the Borrower (or its
     affiliates), including, without limitation, a rights offering, equity
     issuance or sale or issuance of subordinate or mezzanine debt (a
     "Recapitalization Event"). The Subordinating Creditor acknowledges and
     agrees that no terms or conditions of prepayment of the Subordinated Debt
     have been agreed upon and that the Subordinated Debt may only be prepaid in
     connection with any Recapitalization Event with the consent of the Agent
     and the Banks, such consent to be granted or withheld in the sole and
     absolute discretion of the Agent and the Banks.
<PAGE>

                                      -3-

         3.    Enforcement. The Subordinating Creditor will not take or omit to
               -----------
take any action or assert any claim with respect to the Subordinated Debt or
otherwise which is inconsistent with the provisions of this Agreement. Without
limiting the foregoing, the Subordinating Creditor will not assert, collect or
enforce the Subordinated Debt or any part thereof or take any action to
foreclose or realize upon the Subordinated Debt or any part thereof or enforce
any of the Subordinated Documents except to the extent (but only to such extent)
that the commencement of a legal action may be required to toll the running of
any applicable statute of limitation, to defend any challenge to the validity of
the Subordinated Debt, or to file a proof of claim or to make a vote in a
proceeding described in (S)6.1. Until the Senior Debt has been finally paid in
full in cash, the Subordinating Creditor shall not have any right of
subrogation, reimbursement, restitution, contribution or indemnity whatsoever
from any assets of the Borrower or any guarantor of or provider of collateral
security for the Senior Debt. The Subordinating Creditor further waives any and
all rights with respect to marshalling.

         4.    Payments Held in Trust. Until the Senior Debt is paid in full in
               ----------------------
cash, the Subordinating Creditor will hold in trust and immediately pay over to
the Agent for the account of the Banks and the Agent, in the same form of
payment received, with appropriate endorsements, for application to the Senior
Debt any cash amount that the Borrower pays to the Subordinating Creditor with
respect to the Subordinated Debt, or as collateral for the Senior Debt any other
assets of the Borrower that the Subordinating Creditor may receive with respect
to the Subordinated Debt.

         5.    Defense to Enforcement. If the Subordinating Creditor, in
               ----------------------
contravention of the terms of this Agreement, shall commence, prosecute or
participate in any suit, action or proceeding against the Borrower, then the
Borrower may interpose as a defense or plea the making of this Agreement, and
the Agent or any Bank may intervene and interpose such defense or plea in its
name or in the name of the Borrower. If the Subordinating Creditor, in
contravention of the terms of this Agreement, shall attempt to collect any of
the Subordinated Debt or enforce any of the Subordinated Documents, then the
Agent, any Bank or the Borrower may, by virtue of this Agreement, restrain the
enforcement thereof in the name of the Agent or such Bank or in the name of the
Borrower. If the Subordinating Creditor, in contravention of the terms of this
Agreement, obtains any cash or other assets of the Borrower as a result of any
administrative, legal or equitable actions, or otherwise, the Subordinating
Creditor agrees forthwith to pay, deliver and assign to the Agent, for the
account of the Banks and the Agent, with appropriate endorsements, any such cash
for application to the Senior Debt and any such other assets as collateral for
the Senior Debt.

         6.    Bankruptcy, etc.
               ---------------

                    6.1. Payments relating to Subordinated Debt. At any meeting
                         --------------------------------------
         of creditors of the Borrower or in the event of any case or proceeding,
         voluntary or involuntary, for the distribution, division or application
         of all or part of the assets of the Borrower or the proceeds thereof,
         whether such case or proceeding be for the liquidation, dissolution or
         winding up of the Borrower or its business, a
<PAGE>

                                      -4-


         receivership, insolvency or bankruptcy case or proceeding, an
         assignment for the benefit of creditors or a proceeding by or against
         the Borrower for relief under the federal Bankruptcy Code or any other
         bankruptcy, reorganization or insolvency law or any other law relating
         to the relief of debtors, readjustment of indebtedness, reorganization,
         arrangement, composition or extension or marshalling of assets or
         otherwise, the Agent is hereby irrevocably authorized at any such
         meeting or in any such proceeding to receive or collect for the benefit
         of the Banks and the Agent any cash or other assets of the Borrower
         distributed, divided or applied by way of dividend or payment, or any
         securities issued on account of any Subordinated Debt, and apply such
         cash to or to hold such other assets or securities as collateral for
         the Senior Debt, and to apply to the Senior Debt any cash proceeds of
         any realization upon such other assets or securities that the Agent in
         its discretion elects to effect, until all of the Senior Debt shall
         have been paid in full in cash, rendering to the Subordinating Creditor
         any surplus to which the Subordinating Creditor is then entitled.

                    6.2. Securities by Plan of Reorganization or Readjustment.
                         ----------------------------------------------------
         Notwithstanding the foregoing provisions of (S).6.1, the Subordinating
         Creditor shall be entitled to receive and retain any securities of the
         Borrower or any other corporation or other entity provided for by a
         plan of reorganization or readjustment (a) the payment of which
         securities is subordinate, at least to the extent provided in this
         Agreement with respect to Subordinated Debt, to the payment of all
         Senior Debt under any such plan of reorganization or readjustment and
         (b) all other terms of which are acceptable to the Banks and the Agent.

                    6.3. Subordinated Debt Voting Rights. At any such meeting of
                         -------------------------------
         creditors or in the event of any such case or proceeding, the
         Subordinating Creditor shall retain the right to vote and otherwise act
         with respect to the Subordinated Debt (including, without limitation,
         the right to vote to accept or reject any plan of partial or complete
         liquidation, reorganization, arrangement, composition or extension),
         provided that the Subordinating Creditor shall not vote with respect to
         --------
         any such plan or take any other action in any way so as to contest (a)
         the validity of any Senior Debt or any collateral therefor or
         guaranties thereof, (b) the relative rights and duties of any holders
         of any Senior Debt established in any instruments or agreements
         creating or evidencing any of the Senior Debt with respect to any of
         such collateral or guaranties or (c) the Subordinating Creditor's
         obligations and agreements set forth in this Agreement.

         7.    Lien Subordination. The Senior Debt, the Credit Agreement and the
               ------------------
other Loan Documents and any and all other documents and instruments evidencing
or creating the Senior Debt and all guaranties, mortgages, security agreements,
pledges and other collateral guarantying or securing the Senior Debt or any part
thereof shall be senior to the Subordinated Debt and all of the Subordinated
Documents irrespective of the time of the execution, delivery or issuance of any
thereof or the filing or recording
<PAGE>

                                      -5-

for perfection of any thereof or the filing of any financing statement or
continuation statement relating to any thereof.

                    7.1. Further Assurances. The Subordinating Creditor hereby
                         ------------------
         agrees, upon request of the Agent at any time and from time to time, to
         execute such other documents or instruments as may be requested by the
         Agent further to evidence of public record or otherwise the senior
         priority of the Senior Debt as contemplated hereby.

                    7.2. Books and Records. The Subordinating Creditor further
                         -----------------
         agrees to maintain on its books and records such notations as the Agent
         may reasonably request to reflect the subordination contemplated hereby
         and to perfect or preserve the rights of the Agent hereunder. A copy of
         this Agreement may be filed as a financing statement in any Uniform
         Commercial Code recording office.

         8.    Banks' Freedom of Dealing. The Subordinating Creditor agrees,
               -------------------------
with respect to the Senior Debt and any and all collateral therefor or
guaranties thereof, that the Borrower and the Banks may agree to increase the
amount of the Senior Debt or otherwise modify the terms of any of the Senior
Debt, and the Banks may grant extensions of the time of payment or performance
to and make compromises, including releases of collateral or guaranties, and
settlements with the Borrower and all other persons, in each case without the
consent of the Subordinating Creditor or the Borrower and without affecting the
agreements of the Subordinating Creditor or the Borrower contained in this
Agreement; provided, however, that nothing contained in this (S).8 shall
           --------  -------
constitute a waiver of the right of the Borrower itself to agree or consent to a
settlement or compromise of a claim which the Agent or any Bank may have against
the Borrower.

         9.    Modification or Sale of the Subordinated Debt. The Subordinating
               ---------------------------------------------
Creditor will not, at any time while this Agreement is in effect, modify any of
the terms of any of the Subordinated Debt or any of the Subordinated Documents;
nor will the Subordinating Creditor sell, transfer, pledge, assign, hypothecate
or otherwise dispose of any or all of the Subordinated Debt to any person other
than a person who agrees in a writing, satisfactory in form and substance to the
Agent, to become a party hereto and to succeed to the rights and to bound by all
of the obligations of the Subordinating Creditor hereunder. In the case of any
such disposition by the Subordinating Creditor, the Subordinating Creditor will
notify the Agent at least 10 days prior to the date of any of such intended
disposition.

         10.   Borrower's Obligations Absolute. Nothing contained in this
               -------------------------------
Agreement shall impair, as between the Borrower and the Subordinating Creditor,
the obligation of the Borrower to pay to the Subordinating Creditor all amounts
payable in respect of the Subordinated Debt as and when the same shall become
due and payable in accordance with the terms thereof, or prevent the
Subordinating Creditor (except as expressly otherwise provided in (S)3 or (S)6)
from exercising all rights, powers and remedies otherwise permitted by
Subordinated Documents and by applicable law upon a default in the payment of
the Subordinated Debt or under any Subordinated Document, all, however, subject
to the rights of the Agent and the Banks as set forth in this Agreement.
<PAGE>

                                      -6-

The failure of the Borrower to make any payment with respect to the Subordinated
Debt in accordance with its terms by reason of the operation of this Agreement
shall not be construed as preventing the occurrence of a default under the
Subordinated Documents.

         11.   Termination of Subordination. This Agreement shall continue in
               ----------------------------
full force and effect, and the obligations and agreements of the Subordinating
Creditor and the Borrower hereunder shall continue to be fully operative, until
all of the Senior Debt shall have been paid and satisfied in full in cash and
such full payment and satisfaction shall be final and not avoidable. To the
extent that the Borrower or any guarantor of or provider of collateral for the
Senior Debt makes any payment on the Senior Debt that is subsequently
invalidated, declared to be fraudulent or preferential or set aside or is
required to be repaid to a trustee, receiver or any other party under any
bankruptcy, insolvency or reorganization act, state or federal law, common law
or equitable cause (such payment being hereinafter referred to as a "Voided
Payment"), then to the extent of such Voided Payment, that portion of the Senior
Debt that had been previously satisfied by such Voided Payment shall be revived
and continue in full force and effect as if such Voided Payment had never been
made. In the event that a Voided Payment is recovered from the Agent or any
Bank, an Event of Default shall be deemed to have existed and to be continuing
under the Credit Agreement from the date of the Agent's or such Bank's initial
receipt of such Voided Payment until the full amount of such Voided Payment is
restored to the Agent or such Bank. During any continuance of any such Event of
Default, this Agreement shall be in full force and effect with respect to the
Subordinated Debt. To the extent that the Subordinating Creditor has received
any payments with respect to the Subordinated Debt subsequent to the date of the
Agent's or any Bank's initial receipt of such Voided Payment and such payments
have not been invalidated, declared to be fraudulent or preferential or set
aside or are required to be repaid to a trustee, receiver, or any other party
under any bankruptcy act, state or federal law, common law or equitable cause,
the Subordinating Creditor shall be obligated and hereby agrees that any such
payment so made or received shall be deemed to have been received in trust for
the benefit of the Agent or such Bank, and the Subordinating Creditor hereby
agrees to pay to the Agent for the benefit of the Agent or (as the case may be)
such Bank, upon demand, the full amount so received by the Subordinating
Creditor during such period of time to the extent necessary fully to restore to
the Agent or such Bank the amount of such Voided Payment. Upon the payment and
satisfaction in full in cash of all of the Senior Debt, which payment shall be
final and not avoidable, this Agreement will automatically terminate without any
additional action by any party hereto.

         12.   Notices. All notices and other communications which are required
               -------
and may be given pursuant to the terms of this Agreement shall be in writing and
shall be sufficient and effective in all respects if given in writing or
telecopied, delivered or mailed by registered or certified mail, postage
prepaid, as follows:
<PAGE>

                                      -7-

          If to the Agent:

                                        Fleet National Bank
                                        100 Federal Street
                                        Boston, MA 02110

                         Attention:     Thomas P. Tansi

               with a copy to:

                                        Bingham Dana LLP
                                        150 Federal Street
                                        Boston, MA 02110

                         Attention:     Robert A.J. Barry

          If to the Subordinating Creditor:

                                        Samstock, L.L.C.
                                        c/o Equity Group Investments, L.L.C.
                                        Two North Riverside Plaza
                                        Suite 600
                                        Chicago, IL 60606

                         Attention:     Don Liebentritt
                                        Jeff Klein

               with a copy to:

                                        Neal, Gerber & Eisenberg
                                        Two North LaSalle Street
                                        Chicago, IL 60602

                         Attention:     Marc Brenner

          If to the Parent, the
          Borrower or any Guarantor:

                                        Chart House, Inc.
                                        640 North LaSalle Street, Suite 295
                                        Chicago, IL 60610

                         Attention:     Thomas J. Walters

     or such other address or addresses as any party hereto shall have
designated by written notice to the other parties hereto. Notices shall be
deemed given and effective
<PAGE>

                                      -8-

upon the earlier to occur of (i) the third day following deposit thereof in the
U.S. mail or (ii) receipt by the party to whom such notice is directed.

     13.  Governing Law. THIS AGREEMENT SHALL BE GOVERNED BY AND INTERPRETED
          -------------
IN ACCORDANCE WITH THE LAWS OF THE COMMONWEALTH OF MASSACHUSETTS AND SHALL BE A
SEALED INSTRUMENT UNDER SUCH LAWS.

     14.  Waiver of Jury Trial. EACH OF THE SUBORDINATING CREDITOR AND THE
          --------------------
BORROWER EACH HEREBY WAIVES ITS RIGHT TO A JURY TRIAL WITH RESPECT TO ANY ACTION
OR CLAIM ARISING OUT OF ANY DISPUTE IN CONNECTION WITH THIS AGREEMENT, ANY
RIGHTS OR OBLIGATIONS HEREUNDER OR THE PERFORMANCE OF SUCH RIGHTS AND
OBLIGATIONS. EXCEPT AS PROHIBITED BY LAW, EACH OF THE SUBORDINATING CREDITOR AND
THE BORROWER HEREBY WAIVES ANY RIGHT WHICH IT MAY HAVE TO CLAIM OR RECOVER IN
ANY LITIGATION REFERRED TO IN THE PRECEDING SENTENCE ANY SPECIAL, EXEMPLARY,
PUNITIVE OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES OTHER THAN, OR IN ADDITION TO,
ACTUAL DAMAGES. EACH OF THE SUBORDINATING CREDITOR AND THE BORROWER (A)
CERTIFIES THAT NO REPRESENTATIVE, AGENT OR ATTORNEY OF THE AGENT OR ANY BANK HAS
REPRESENTED, EXPRESSLY OR OTHERWISE, THAT THE AGENT OR ANY BANK WOULD NOT, IN
THE EVENT OF LITIGATION, SEEK TO ENFORCE THE FOREGOING WAIVERS AND (B)
ACKNOWLEDGES THAT THE AGENT HAS BEEN INDUCED TO ENTER INTO THIS AGREEMENT BY,
AMONG OTHER THINGS, THE WAIVERS AND CERTIFICATIONS CONTAINED HEREIN.

     15.  Miscellaneous. This Agreement may be executed in several
          -------------
counterparts and by each party on a separate counterpart, each of which when so
executed and delivered shall be an original, and all of which together shall
constitute one instrument. In proving this Agreement, it shall not be necessary
to produce or account for more than one such counterpart signed by the party
against which enforcement is sought. The Agent, acting upon the instructions of
the Required Banks, may, in their sole and absolute discretion, waive any
provisions of this Agreement benefiting the Agent and the Banks; provided,
                                                                 --------
however, that such waiver shall be effective only if in writing and signed by
-------
the Agent and shall be limited to the specific provision or provisions expressly
so waived. This Agreement shall be binding upon the successors and assigns of
the Subordinating Creditor and the Borrower and shall inure to the benefit of
the Agent and the Banks, the Agent's and the Banks' respective successors and
assigns, any lender or lenders refunding or refinancing any of the Senior Debt
and their respective successors and assigns, but shall not otherwise create any
rights or benefits for any third party. In the event that any lender or lenders
refund or refinance any of the Senior Debt, the terms "Credit Agreement", "Loan
Documents", "Event of Default" and the like shall refer mutatis mutandis to the
                                                        ------- --------
agreements and instruments in favor of such lender or lenders and to the related
definitions contained therein.
<PAGE>

                                      -9-

     IN WITNESS WHEREOF, the parties hereto have executed this Agreement as of
the date first above written.

                                BANK: FLEET NATIONAL BANK, As Agent



                                      By:____________________________________
                                           Name:
                                           Title:

              SUBORDINATING CREDITOR: SAMSTOCK, L.L.C.



                                      By:____________________________________
                                           Name:
                                           Title:

                              PARENT: CHART HOUSE ENTERPRISES, INC.



                                      By:____________________________________
                                           Name: William Sullivan
                                           Title: Executive Vice President & CFO

                            BORROWER: CHART HOUSE, INC.



                                      By:____________________________________
                                           Name: William Sullivan
                                           Title: Executive Vice President & CFO


                          GUARANTORS: CHART HOUSE ENTERPRISES OF
                                      IDAHO, INC.


                                      By:____________________________________
                                      Title: Assistant Secretary
<PAGE>

                                      -10-

                                      CHART HOUSE ENTERPRISES OF
                                      PUERTO RICO, INC.


                                      By:____________________________________
                                      Title: Executive Vice President & CFO


                                      CHART HOUSE OF ANNAPOLIS, INC.


                                      By:____________________________________
                                      Title: Executive Vice President & CFO


                                      CHART HOUSE OF MARYLAND, INC.


                                      By:____________________________________
                                      Title: Executive Vice President & CFO


                                      CHART HOUSE ACQUISITION, INC.


                                      By:____________________________________
                                      Title: Executive Vice President & CFO


                                      BIG WAVE, INC.rantor

                                      By:____________________________________
                                      Title: Executive Vice President & CFO


                                      CORK `N CLEAVER, INC.


                                      By:____________________________________
                                      Title: Executive Vice President & CFO
<PAGE>

                                      -11-

                                      ANALOS COMPANY


                                      By:____________________________________
                                      Title: Executive Vice President & CFO
<PAGE>

                                      -12-

                         CERTIFICATE OF ACKNOWLEDGMENT

STATE OF ________________________________)
                                         )  ss.
COUNTY OF _______________________________)

     Before me, the undersigned, a Notary Public in and for the county
aforesaid, on this ____ day of November, 2000, personally appeared ____________
to me known personally, and who, being by me duly sworn, deposes and says that
he is the ___________ of SAMSTOCK, L.L.C., and that said instrument was signed
and sealed on behalf of said limited liability company by authority of its
members, and said _________________ acknowledged said instrument to be the free
act and deed of said limited liability company.



                                        ____________________________
                                        Notary Public
                                        My commission expires: