<PAGE>   1
                                                                   EXHIBIT 10.40

                             SUBORDINATION AGREEMENT


         THIS SUBORDINATION AGREEMENT ("Subordination Agreement"), dated as of
December 18, 1996, is by and between CONGRESS FINANCIAL CORPORATION, a
California corporation ("Senior Creditor", as hereinafter further defined), and
SWISS BANK CORPORATION, a banking corporation organized under the laws of
Switzerland acting through its New York Branch ("Junior Creditor", as
hereinafter further defined). Senior Creditor and Junior Creditor are sometimes
individually referred to herein as "Creditor" and collectively as "Creditors."


                              W I T N E S S E T H:


         WHEREAS, Senior Creditor has entered into financing arrangements with
Hanover Direct, Inc. ("Hanover", as hereinafter further defined) and certain of
its subsidiaries, pursuant to which Senior Creditor has, upon certain terms and
conditions, made loans and provided other financial accommodations to certain
subsidiaries of Hanover, guaranteed by Hanover and certain subsidiaries of
Hanover, secured by substantially all of the assets and properties of Hanover
and of such borrower subsidiaries and guarantor subsidiaries of Hanover (such
subsidiaries, together with Hanover, individually and collectively, "Debtor", as
hereinafter further defined); and

         WHEREAS, pursuant to such financing arrangements, Senior Creditor has
previously arranged for the issuance of the CoreStates Letters of Credit (as
hereinafter defined); and

         WHEREAS, Junior Creditor is about to enter into the Swiss Bank
Reimbursement Agreement (as hereinafter defined) with Hanover, pursuant to which
Junior Creditor will issue the Swiss Bank Letters of Credit (as hereinafter
defined) which will be substituted for the CoreStates Letters of Credit and the
NationsBank Littlestown Letter of Credit (as hereinafter defined), each of which
shall, concurrently with such substitution, be surrendered for cancellation or
otherwise cancelled; and

         WHEREAS, in order to induce Senior Creditor to continue the financing
arrangements with Hanover and certain subsidiaries of Hanover, Junior Creditor
has agreed to the subordination in favor of Senior Creditor as provided herein
of its right to payment of the existing and future obligations of Debtor to
Junior Creditor arising in connection with or relating to the Swiss Bank Letters
of Credit, and related matters as set forth below;

         NOW THEREFORE, in consideration of the mutual benefits accruing to
Creditors hereunder and other good and valuable


<PAGE>   2
consideration, the receipt and sufficiency of which is hereby acknowledged, the
parties hereto do hereby agree as follows:

         1.       DEFINITIONS

         As used above and in this Subordination Agreement, the following terms
shall have the meanings ascribed to them below:

         1.1      "Agreements" shall mean, individually and collectively, the 
Senior Creditor Agreements and the Junior Creditor Agreements.

         1.2      "CoreStates" shall mean CoreStates Bank, N.A., a national 
banking association, and its successors and assigns.

         1.3      "Corestates Letters of Credit" shall mean, individually and
collectively, the CoreStates Series A Letter of Credit, the CoreStates Series B
Letter of Credit and the CoreStates Littlestown Backup Letter of Credit.

         1.4      "CoreStates Littlestown Backup Letter of Credit" shall mean
the Irrevocable Standby Letter of Credit No. 516262P, dated November 14, 1995,
issued in the initial stated amount of $8,560,000 by CoreStates for the account
of certain subsidiaries of Debtor, in favor of NationsBank, National
Association.

         1.5      "CoreStates Series A Letter of Credit" shall mean the
Irrevocable Transferable Letter of Credit No. 516466P, dated December 27, 1995,
issued in the initial stated amount of $10,145,833 by CoreStates for the account
of certain subsidiaries of Debtor, in favor of Norwest Bank Minnesota, N.A., as
Trustee under the Series A Note Agreements.

         1.6      "CoreStates Series B Letter of Credit" shall mean the
Irrevocable Transferable Letter of Credit No. 516467P, dated December 27, 1995,
issued in the initial stated amount of $10,145,833 by CoreStates for the account
of certain subsidiaries of Debtor in favor of Norwest Bank Minnesota Bank, N.A.,
as Trustee under the Series B Note Agreements.

         1.7      "Creditors" shall mean, individually and collectively, Senior
Creditor and Junior Creditor and their respective successors and assigns.

         1.8      "Debtor" shall mean, individually and collectively, Hanover
and each of its existing and future subsidiaries who are or become parties (as
borrower or guarantor or who are or become otherwise obligated for all or part
of the Senior Debt) to the Senior Creditor Agreements, and their respective
successors and assigns, including, without limitation, a receiver, trustee, or
debtor-in-possession on behalf of any such person or on behalf of any such
successor or assign.


                                      - 2 -

<PAGE>   3
         1.9  "Hanover" shall mean Hanover Direct, Inc., a Delaware corporation
and its successors and assigns, including, without limitation, a receiver,
trustee or debtor-in-possession on behalf of such person or on behalf of any
such successor or assign.

         1.10 "Junior Creditor" shall mean Swiss Bank Corporation, a banking
corporation organized under the laws of Switzerland acting through its New York
Branch, and its successors and assigns.

         1.11 "Junior Creditor Agreements" shall mean, individually and
collectively, (a) The Swiss Bank Reimbursement Agreement, (b) the Swiss Bank
Letters of Credit, (c) the rights of Junior Creditor under the Series A Note
Agreements and any Series A Notes at any time held by or for Junior Creditor,
(d) the rights of Junior Creditor under the Series B Note Agreements and any
Series B Notes at any time held by or for Junior Creditor, (e) the rights of
Junior Creditor under the Littlestown IDB Agreements and any Littlestown Bonds
at any time held by or for Junior Creditor, and (f) all agreements, documents
and instruments at any time executed and/or delivered by Debtor or any other
person to, with or in favor of Junior Creditor in connection therewith or
related thereto, as all of the foregoing now exist or may hereafter be amended,
modified, supplemented, extended, renewed, restated or replaced. The Junior
Creditor Agreements shall not, however, mean or include the Richemont Guaranty.

         1.12 "Junior Debt" shall mean all obligations, liabilities and
indebtedness of every kind, nature and description owing by Debtor to Junior
Creditor, including principal, interest, charges, fees, premiums, indemnities
and expenses, however evidenced, whether as principal, surety, endorser,
guarantor or otherwise, arising under or evidenced by or in connection with the
Junior Creditor Agreements, whether now existing or hereafter arising, whether
arising before, during or after the initial or any renewal term of the Junior
Creditor Agreements or after the commencement of any case with respect to Debtor
under the U.S. Bankruptcy Code or any similar statute (and including, without
limitation, any principal, interest, fees, costs, expenses and other amounts,
whether or not such amounts are allowable in whole or in part, in any such case
or similar proceeding), whether direct or indirect, absolute or contingent,
joint or several, due or not due, primary or secondary, liquidated or
unliquidated, secured or unsecured, and whether arising directly, or by way of
subrogation, contribution, reimbursement, indemnification, exoneration or
otherwise, or howsoever acquired by Junior Creditor. The Junior Debt shall not,
however, mean or include any of the obligations, liabilities and indebtedness of
Richemont to Junior Creditor pursuant to the Richemont Guaranty.


                                     - 3 -

<PAGE>   4
         1.13 "Littlestown Bonds" shall mean the Variable Rate Demand Industrial
Development Revenue Refunding Bonds, 1987 Series (Hanover House Industries, Inc.
Project), issued on behalf of Hanover House Industries Inc., now known as
Hanover Direct Pennsylvania, Inc., in the original principal amount of
$8,000,000, as the same now exist or may hereafter be amended, modified,
supplemented, extended, renewed, restated or replaced.

         1.14 "Littlestown IDB Agreements" shall mean, collectively, the
Indenture of Trust, dated as of September 1, 1987, presently between the
Littlestown Industrial Development Authority and Fleet National Bank, as
successor Trustee in connection with the Littlestown Bonds, and all agreements,
documents and instruments at any time executed and/or delivered in connection
therewith, as the same now exist or may hereafter be amended, modified,
supplemented, extended, renewed, restated or replaced.

         1.15 "NationsBank Littlestown Letter of Credit" shall mean the letter
of credit no. 41055, dated November 2 ,1994, issued in the initial stated amount
of $8,560,000 by NationsBank for the account of Hanover Direct Pennsylvania,
Inc., in favor of Fleet National Bank, as Trustee under the Littlestown IDB
Agreements.

         1.16 "Person" or "person" shall mean any individual, sole
proprietorship, partnership, corporation (including, without limitation, any
corporation which elects subchapter S status under the Internal Revenue Code of
1986, as amended), business trust, unincorporated association, joint stock
company, trust, joint venture, limited liability company, limited liability
partnership, or other entity or any government or any agency or instrumentality
or political subdivision thereof.

         1.17 "Remarketing Proceeds" shall have the meaning set forth in Section
2.2(c) hereof.

         1.18 "Richemont" shall mean Richemont Finance S.A., a societe anonyme
organized under the laws of the Grand Duchy of Luxembourg, and its successors
and assigns.

         1.19 "Richemont Guaranty" shall mean the Guaranty, dated of even date
herewith, by Richemont in favor of Junior Creditor with respect to the
obligations of Hanover to Junior Creditor, as the same now exists or may
hereafter be amended, modified, supplemented, extended, renewed, restated or
replaced.

         1.20 "Senior Creditor" shall mean Congress Financial Corporation, a
California corporation, and its successors and assigns.

         1.21 "Senior Creditor Agreements" shall mean, individually and
collectively, the Loan and Security Agreement, dated November 14, 1995, by and
among Senior Creditor, Hanover and


                                      - 4 -

<PAGE>   5
certain subsidiaries of Hanover, as amended (the "Loan Agreement"), the
Guarantee and Waivers, each dated November 14, 1995, by Hanover and certain of
its subsidiaries in favor of Senior Creditor, the General Security Agreements,
each dated November 14, 1995, by Hanover and certain of its subsidiaries in
favor of Senior Creditor and all agreements, documents and instruments at any
time executed and/or delivered by Debtor or any other person to, with or in
favor of Senior Creditor in connection therewith or related thereto, as all of
the foregoing now exist or may hereafter be amended, modified, supplemented,
extended, renewed, restated or replaced.

         1.22 "Senior Debt" shall mean all obligations, liabilities and
indebtedness of every kind, nature and description owing by Debtor to Senior
Creditor and/or its affiliates, or participants, including principal, interest,
charges, fees, premiums, indemnities and expenses, however evidenced, whether as
principal, surety, endorser, guarantor or otherwise, whether arising under the
Senior Creditor Agreements or otherwise, whether now existing or hereafter
arising, whether arising before, during or after the initial or any renewal term
of the Senior Creditor Agreements or after the commencement of any case with
respect to Debtor under the U.S. Bankruptcy Code or any similar statute (and
including, without limitation, any principal, interest, fees, costs, expenses
and other amounts, whether or not such amounts are allowable either in whole or
in part, in any such case or similar proceeding), whether direct or indirect,
absolute or contingent, joint or several, due or not due, primary or secondary,
liquidated or unliquidated, secured or unsecured, and whether arising directly,
or by way of subrogation, contribution, reimbursement, indemnification,
exoneration or otherwise, or howsoever acquired by Senior Creditor.

         1.23 "Senior Debt Default" shall mean any "Event of Default" or
"Incipient Default" as such terms are defined in the Loan Agreement referred to
in the definition of Senior Creditor Agreements.

         1.24 "Series A Note Agreements" shall mean, individually and
collectively, the Series A Note Agreement, dated as of November 9, 1994,
presently between Hanover and Norwest Bank Minnesota, N.A., as Trustee, with
respect to the issuance and sale of the Series A Notes, and all agreements,
documents and instruments at any time executed and/or delivered by Hanover or
any related parties in connection therewith, including, without limitation, the
Second Supplemental Series A Note Agreement, dated as of December 18, 1996, as
the same now exist or may hereafter be amended, modified, supplemented,
extended, renewed, restated or replaced.


                                      - 5 -

<PAGE>   6
         1.25 "Series A Notes" shall mean the Flexible Term Notes, Series A,
originally issued on November 9, 1994, by Hanover in the original aggregate
principal amount of $10,000,000, as replaced or modified through the date hereof
pursuant to the Series A Note Agreements, and as the same may hereafter be
amended, modified, supplemented, extended, renewed, restated or replaced.

         1.26 "Series B Note Agreements" shall mean, individually and
collectively, the Series B Note Agreement, dated as of April 27, 1995, presently
between Hanover and Norwest Bank Minnesota, N.A., as Trustee, with respect to
the issuance and sale of the Series B Notes, and all other agreements, documents
or instruments at any time executed and/or delivered in connection therewith,
including, without limitation, the Second Supplemental Series B Note Agreement,
dated as of December 18, 1996, as the same now exist or may hereafter be
amended, modified, supplemented, extended, renewed, restated or replaced.

         1.27 "Series B Notes" shall mean the Flexible Term Notes, Series B,
originally issued on April 27, 1995, by Hanover in the original aggregate
principal amount of $10,000,000, as replaced or modified through the date hereof
pursuant to the Series B Note Agreements, and as the same may hereafter be
amended, modified, supplemented, extended, renewed, restated or replaced.

         1.28 "Swiss Bank Letters of Credit" shall mean, collectively, the Swiss
Bank Littlestown Letter of Credit, the Swiss Bank Series A Letter of Credit and
the Swiss Bank Series B Letter of Credit, as the same now exist or may hereafter
be amended, modified, supplemented, extended, renewed, restated or replaced.

         1.29 "Swiss Bank Littlestown Letter of Credit" shall mean the Letter of
Credit No. S567171 dated December 18, 1996, issued by Swiss Bank for the account
of Hanover Direct Pennsylvania, Inc. in favor of Fleet National Bank, as Trustee
under the Littlestown IDB Agreements, in the initial stated amount of
$8,560,000, delivered in substitution for the NationsBank Littlestown Letter of
Credit and the CoreStates Littlestown Backup Letter of Credit, as the same now
exists or may hereafter be amended, modified, supplemented, extended, renewed,
restated or replaced.

         1.30 "Swiss Bank Reimbursement Agreement" shall mean the Reimbursement
Agreement, dated as of December 18, 1996, by and between Hanover and Junior
Creditor with respect to the Swiss Bank Letters of Credit, as the same now
exists or may hereafter be amended, modified, supplemented, extended, renewed,
restated or replaced.


                                      - 6 -

<PAGE>   7
         1.31 "Swiss Bank Series A Letter of Credit" shall mean the Letter of
Credit No. S567169 dated December 18, 1996, issued by Swiss Bank for the account
of Hanover in the initial stated amount of $9,638,541, delivered in substitution
for the CoreStates Series A Letter of Credit, as the same now exists or may
hereafter be amended, modified, supplemented, extended, renewed, restated or
replaced.

         1.32 "Swiss Bank Series B Letter of Credit" shall mean the Letter of
Credit No. S567170, dated December 18, 1996, issued by Swiss Bank for the
account of Hanover in the initial stated amount of $9,638,541, delivered in
substitution for the CoreStates Series B Letter of Credit, as the same now
exists or may hereafter be amended, modified, supplemented, extended, renewed,
restated or replaced.

         1.33 All terms used herein which are defined in the Uniform Commercial
Code as in effect in the State of New York, unless otherwise defined herein
shall have the meanings set forth therein. All references to any term in the
plural shall include the singular and all references to any term in the singular
shall include the plural.

          2.  SUBORDINATION OF JUNIOR DEBT

         2.1  Subordination. Except as specifically set forth in Section 2.2
hereof, Junior Creditor hereby subordinates its right to payment and
satisfaction of the Junior Debt, and the payment and satisfaction thereof,
directly or indirectly, by any means whatsoever, is hereby deferred, to the
prior indefeasible payment and satisfaction in full of all Senior Debt.

         2.2  Permitted Payments. Senior Creditor hereby agrees that,
notwithstanding anything to the contrary contained in Section 2.1:

              (a) Debtor may make and Junior Creditor may receive and retain, 
payments by Debtor of bank fees and other expenses set forth in Sections 2.03(a)
through (f) of the Swiss Bank Reimbursement Agreement as in effect on the date
hereof, when due and payable in accordance with such provisions; provided that,
such fees and expenses accrued or incurred through the date hererof do not
exceed $47,500 in the aggregate; and provided further that, in the case of those
fees or expenses accrued or incurred after the date hereof, written notice has
not been given to Junior Creditor to the effect that a Senior Debt Default
exists or has occurred and is continuing at the time of any such payment or
would exist or occur as a result thereof;

              (b) Debtor may make, and Junior Creditor may receive and
retain, payments by Debtor of Debtor's reimbursement obligations owed to Junior
Creditor under the Swiss Bank


                                      - 7 -

<PAGE>   8
Reimbursement Agreement in respect of drawings paid under the Swiss Bank Letters
of Credit for (i) regularly schedule monthly payments of interest when due under
the Series A Note Agreements, Series B Note Agreements and Littlestown IDB
Agreements as each is in effect on the date hereof and (ii) the regularly
scheduled reductions of principal due and payable on October 1, 1997 under the
Series A Notes and Series B Notes, each such reduction to be in the amount of
$500,000; provided that written notice has not been given to Junior Creditor to
the effect that a Senior Debt Default exists or has occurred and is continuing
at the time of any such payment or would exist or occur as a result thereof;

                  (c) if any of the Series A Notes, Series B Notes or the
Littlestown Bonds are purchased with funds drawn under the Swiss Bank Series A
Letter of Credit, the Swiss Bank Series B Letter of Credit and the Swiss Bank
Littlestown Letter of Credit in accordance with the terms thereof and of the
Series A Note Agreements, the Series B Note Agreements and the Littlestown IDB
Agreements, respectively, as each is in effect on the date hereof, the proceeds
(if any) obtained following such purchase from the remarketing of any of the
Series A Notes, Series B Notes and Littlestown Bonds so purchased, as the case
may be, but prior to Junior Creditor's receipt of written notice of the
existence or occurrence of a Senior Debt Default that is continuing
("Remarketing Proceeds"), may be received and retained by the Junior Creditor
(but such amounts permitted to be received and retained by the Junior Creditor
shall not in any event include any payment by Debtor, or amounts collected
from Debtor, in respect of or to purchase any of the obligations arising under
or evidenced by the Series A Notes, Series A Note Agreements, Series B Notes,
Series B Note Agreements, Littlestown Bonds or Littlestown IDB Agreements); and

                  (a) Richemont may make, and Junior Creditor may receive and
retain, payments by Richemont of its obligations when due under the Richemont
Guaranty, whether or not a Senior Debt Default exists or has occurred and is
continuing and whether or not notice thereof is given to the Junior Creditor
hereunder.

         2.3      Distributions.

                  (a) In the event of any distribution, division, or
application, partial or complete, voluntary or involuntary, by operation of law
or otherwise, of all or any part of the assets of Debtor or the proceeds thereof
to the creditors of Debtor or readjustment of the obligations and indebtedness
of Debtor, whether by reason of liquidation, bankruptcy, arrangement,
receivership, assignment for the benefit of creditors, marshalling of assets of
Debtor or any other action or proceeding involving the readjustment of all or
any part of indebtedness of Debtor or the application of the assets of Debtor to
the payment or liquidation thereof (each of the foregoing, an "Insolvency


                                      - 8 -

<PAGE>   9
Proceeding"), or upon the dissolution or other winding up of Debtor's business,
or upon the sale of all or substantially all of Debtor's assets, then, and in
any such event, (i) Senior Creditor shall first receive indefeasible payment in
full in cash of all of the Senior Debt prior to the payment of all or any part
of the Junior Debt, and (ii) Senior Creditor shall be entitled to receive any
payment or distribution of any kind or character, whether in cash, securities or
other property, which is payable or deliverable in respect of any or all of the
Junior Debt.

                  (b) In order to enable Senior Creditor to enforce its rights
under Section 2.3(a) hereof, Senior Creditor is hereby irrevocably authorized
and empowered (in its own name or in the name of Junior Creditor or otherwise),
but shall have no obligation, to enforce claims comprising any of the Junior
Debt by proof of debt, proof of claim, suit or otherwise and take generally any
action which Junior Creditor might otherwise be entitled to take, as Senior
Creditor may deem necessary or advisable for the enforcement of its rights or
interests hereunder.

                  (c) To the extent necessary for Senior Creditor to realize the
benefits of the subordination of the Junior Debt provided for herein (including
the right to receive any and all payments and distributions which might
otherwise be payable or deliverable with respect to the Junior Debt in any
Insolvency Proceeding or otherwise), Junior Creditor shall execute and deliver
to Senior Creditor such instruments or documents (together with such assignments
or endorsements as Senior Creditor shall deem necessary), as may be requested by
Senior Creditor.

         2.4      Payments Received by Junior Creditor. Except for payments
received by Junior Creditor as provided in Section 2.2 hereof, should any
payment or distribution or security or instrument or proceeds thereof be
received by the Junior Creditor in respect of the Junior Debt, Junior Creditor
shall receive and hold the same in trust, as trustee, for the benefit of Senior
Creditor, segregated from other funds and property of Junior Creditor and shall
forthwith deliver the same to Senior Creditor (together with any endorsement or
assignment of Junior Creditor where necessary), for application to any of the
Senior Debt. In the event of the failure of Junior Creditor to make any such
endorsement or assignment to Senior Creditor, Senior Creditor, or any of its
officers or employees, are hereby irrevocably authorized on behalf of Junior
Creditor to make the same.

         2.5      Instrument Legend and Notation. Any instrument at any time
evidencing the Junior Debt, or any portion thereof, shall be permanently marked
on its face with a legend conspicuously indicating that payment thereof is
subordinate in right of payment to the Senior Debt and subject to the terms and


                                      - 9 -

<PAGE>   10
conditions of this Subordination Agreement, and (a) after being so marked
certified copies thereof shall be delivered to Senior Creditor and (b) the
original of any such instrument shall be immediately delivered to Senior
Creditor upon Senior Creditor's request, at any time on or after the
commencement of an Insolvency Proceeding. In the event any legend or endorsement
is omitted, Senior Creditor, or any of its officers or employees, are hereby
irrevocably authorized on behalf of Junior Creditor to make the same. No
specific legend, further assignment or endorsement or delivery of notes,
guarantees or instruments shall be necessary to subject any Junior Debt to the
subordination thereof contained in this Agreement.

         3.       COVENANTS, REPRESENTATIONS AND WARRANTIES

         3.1      Additional Covenants. Junior Creditor and Debtor agree in
favor of Senior Creditor that:

                  (a) except as specifically set forth in Section 2.2 hereof,
Debtor shall not, directly or indirectly, make and Junior Creditor shall not,
directly or indirectly, accept or receive any payment of or any prepayment or
any payment pursuant to acceleration or claims of breach or any payment to
acquire Junior Debt or otherwise in respect of any Junior Debt;

                  (b) notwithstanding any rights or remedies available to it
under the Junior Creditor Agreements, applicable law or otherwise, Junior
Creditor shall not, directly or indirectly, (i) seek to collect from Debtor any
of the Junior Debt or exercise any of its rights or remedies against Debtor upon
a default or event of default by Debtor under the Junior Creditor Agreements or
otherwise or (ii) commence any action or proceeding against Debtor or Debtor's
properties under the U.S. Bankruptcy Code or any state insolvency law or any
similar present or future statute, law or regulation or any proceedings for
voluntary liquidation, dissolution or other winding up of Debtor's business, or
the appointment of any trustee, receiver or liquidator for Debtor or any part of
Debtor's properties or any assignment for the benefit of creditors or any
marshalling of assets of Debtor or (iii) take any other action against Debtor or
Debtor's properties in respect of the Junior Debt;

                  (c) Junior Creditor shall not acquire any guarantees for the
Junior Debt or other agreements under which any person, other than Hanover, is
or may become obligated, directly or indirectly, for all or any portion of the
Junior Debt, except the Richemont Guaranty, and Debtor shall not grant to Junior
Creditor and Junior Creditor shall not acquire any security interest, lien,
claim or encumbrance on any assets or properties of Debtor, except for the
rights and interests of Junior Creditor in any Remarketing Proceeds as permitted
pursuant to Section 2.2(c) hereof and any security interest in any Series A
Notes, Series B


                                     - 10 -

<PAGE>   11
Notes, or Littlestown Bonds Purchased with funds drawn under the Swiss Bank
Letters of Credit, but only to the extent of entitling Junior Creditor to
receive any Remarketing Proceeds arising therefrom;

                  (d) Junior Creditor and Debtor shall not amend, modify, alter
or change in any material respect the terms of any arrangements related to the
Junior Debt;

                  (e) Junior Creditor shall not sell, assign, pledge, encumber
or otherwise dispose of any of the Junior Debt, or subordinate any of the Junior
Debt to any indebtedness of Debtors other than the Senior Debt, except that
Junior Creditor may assign or sell participations expressly subject hereto, a
portion of the Junior Debt to certain other financial institutions
("Participating Banks") so long as (i) Junior Creditor provides Senior Creditor
five (5) days' prior written notice of such participation, (ii) such
Participating Bank is reasonably acceptable to Senior Creditor, (iii)
contemporaneously with such assignment, Junior Creditor causes such
Participating Bank to enter into a written agreement in favor of Junior Creditor
(the "Participating Bank Agreement"), in form and substance satisfactory to
Senior Creditor, pursuant to which such Participating Bank, shall agree, among
other things, to be bound by the terms of this Subordination Agreement, and (iv)
such Participating Bank delivers, in form and substance satisfactory to Senior
Creditor (A) evidence of the adoption and subsistence of authorizing resolutions
of such Particpating Bank approving the execution, delivery and performance by
such Participating Bank of the Participating Bank Agreement and (B) an opinion
of counsel to such Participating Bank addressed to Senior Creditor with respect
to the due authorization, execution, validity and enforceability of the
Participating Bank Agreement, and as to such other matters as Senior Creditor
shall reasonably require;

                  (f) Junior Creditor and Debtor shall, at any time or times
upon the request of Senior Creditor, promptly furnish to Senior Creditor a true,
correct and complete statement of the outstanding Junior Debt; and

                  (g) Junior Creditor and Debtor shall execute and deliver to
Senior Creditor such additional agreements, documents and instruments and take
such further actions as may be necessary or desirable in the opinion of Senior
Creditor to effectuate the provisions and purposes of this Subordination
Agreement.

         3.2      Additional Representations and Warranties. Junior Creditor and
Debtor represent and warrant to Senior Creditor that:

                  (a) Junior Creditor has no guarantee or other agreement from
any person, other than Hanover, under which such


                                     - 11 -

<PAGE>   12
person is or may become obligated, directly or indirectly, for all or any
portion of the Junior Debt, except for the Richemont Guaranty, and Junior
Creditor has no security interest, lien, claim or encumbrance on any assets and
properties of Debtor and the Junior Debt is unsecured, except that Junior
Creditor has a right to receive Remarketing Proceeds as permitted under Section
2.2(c) hereof, and may hold an interest in the Series A Notes, Series B Notes or
Littlestown Bonds purchased with funds drawn under the Swiss Bank Letters of
Credit, but only to the extent of entitling Junior Creditor to receive any
Remarketing Proceeds arising therefrom; and expressly provided that no Series A
Notes, Series B Notes or Littlestown Bonds purchased with drawings under the
Swiss Bank Letters of Credit shall be remarketed, sold or otherwise disposed of
by or for Junior Creditor after receipt of written notice that a Senior Debt
Default exists or has occurred and is continuing;

                  (b) as of the date hereof, no default or event of default, or
event which with notice or passage of time or both would constitute an event of
default exists or has occurred under the Junior Creditor Agreements;

                  (c) Junior Creditor is the exclusive legal and beneficial
owner of all of the Junior Debt;

                  (d) Junior Creditor has established a New York Branch, duly
licensed by the New York Superintendent of Banks, and has obtained all
approvals, licenses and permits from all Federal and State banking and other
governmental entities necessary to issue the Swiss Bank Letters of Credit and to
execute, deliver and perform this Subordination Agreement;

                  (e) none of the Junior Debt is subject to any lien, security
interest, financing statements, subordination, assignment or other claim, except
in favor of Senior Creditor; and

                  (f) this Subordination Agreement constitutes the legal, valid
and binding obligations of Junior Creditor, enforceable in accordance with its
terms.

         3.3      Waivers. Notice of acceptance hereof, the making of loans,
advances and extensions of credit or other financial accommodations to, and the
incurring of any expenses by or in respect of, Hanover or its subsidiaries by
Senior Creditor, and presentment, demand, protest, notice of protest, notice of
nonpayment or default and all other notices to which Junior Creditor and Debtor
are or may be entitled are hereby waived (except as expressly provided for
herein or as to Debtor, in the Senior Creditor Agreements). Junior Creditor also
waives notice of, and hereby consents to, (a) any amendment, modification,
supplement, renewal, restatement or extensions of time of payment


                                     - 12 -

<PAGE>   13
of or increase or decrease in the amount of any of the Senior Debt or to the
Senior Creditor Agreements or any collateral at any time granted to or held by
Senior Creditor, (b) the taking, exchange, surrender and releasing of collateral
at any time granted to or held by Senior Creditor or guarantees now or at any
time held by or available to Senior Creditor for the Senior Debt or any other
person at any time liable for or in respect of the Senior Debt, (c) the exercise
of, or refraining from the exercise of any rights against Debtor or any other
obligor or any collateral at any time granted to or held by Senior Creditor, (d)
the settlement, compromise or release of, or the waiver of any default with
respect to, any of the Senior Debt, and/or (e) Senior Creditor's election, in
any proceeding instituted under the U.S. Bankruptcy Code of the application of
Section 1111(b)(2) of the U.S. Bankruptcy Code. Any of the foregoing shall not,
in any manner, affect the terms hereof or impair the obligations of Junior
Creditor hereunder. All of the Senior Debt shall be deemed to have been made or
incurred in reliance upon this Subordination Agreement.

          3.4 Subrogation; Marshalling. Junior Creditor shall not be subrogated
to, or be entitled to any assignment of any Senior Debt or of any collateral for
or guarantees or evidence of any thereof until all of the Senior Debt is
indefeasibly paid and satisfied in full. Junior Creditor hereby waives any and
all rights to have any collateral or any part thereof granted to or held by
Senior Creditor marshalled upon any foreclosure or other disposition of such
collateral by Senior Creditor or Debtor with the consent of Senior Creditor.

          3.5 No Offset. In the event Junior Creditor at any time incurs any
obligation to pay money to Debtor, Junior Creditor hereby irrevocably agrees
that it shall pay such obligation in cash or cash equivalents in accordance with
the terms of the contract governing such obligation and shall not deduct from or
setoff against any amounts owed by Junior Creditor to Debtor in connection with
any such transaction any amounts the Junior Creditor claims are due to it with
respect to the Junior Debt.

          4.  MISCELLANEOUS

          4.1 Amendments. Any waiver, permit, consent or approval by either
Creditor of or under any provision, condition or covenant to this Subordination
Agreement must be in writing and shall be effective only to the extent it is set
forth in writing and as to the specific facts or circumstances covered thereby.
Any amendment of this Subordination Agreement must be in writing and signed by
each of the parties to be bound thereby.

          4.2  Successors and Assigns.


                                     - 13 -

<PAGE>   14
                  (a) This Subordination Agreement shall be binding upon the
parties hereto and their respective successors and assigns and shall inure to
the benefit of each of Creditors and its respective successors, participants and
assigns.

                  (b) Senior Creditor reserves the right to grant participations
in, or otherwise sell, assign, transfer or negotiate all or any part of, or any
interest in, the Senior Debt and the collateral securing same; provided, that,
Junior Creditor shall not be obligated to give any notices to or otherwise in
any manner deal directly with any participant in the Senior Debt and no
participant shall be entitled to any rights or benefits under this Subordination
Agreement except through Senior Creditor. In connection with any participation
or other transfer or assignment, Senior Creditor (i) may disclose to such
assignee, participant or other transferee or assignee all documents and
information which Senior Creditor now or hereafter may have relating to the
Senior Debt or any collateral and (ii) shall disclose to such participant or
other transferee or assignee the existence and terms and conditions of this
Subordination Agreement.

                  (c) In connection with any assignment or transfer of any or
all of the Senior Debt, or any or all rights of Senior Creditor in any of the
property of Hanover or its subsidiaries (other than pursuant to a
participation), Junior Creditor agrees to execute and deliver an agreement
containing terms substantially identical to those contained herein in favor of
any such assignee or transferee and, in addition, will execute and deliver an
agreement containing terms substantially identical to those contained herein in
favor of any third person who succeeds to or replaces any or all of Senior
Creditor's financing of certain subsidiaries of Hanover, whether such successor
financing or replacement occurs by transfer, assignment, "takeout" or any other
means.

         4.3      Insolvency. This Subordination Agreement shall be applicable
both before and after the filing of any petition by or against Hanover or any of
its subsidiaries under the U.S. Bankruptcy Code and all converted or succeeding
cases in respect thereof, and all references herein to Debtor or any of
Hanover's subsidiaries shall be deemed to apply to a trustee for Hanover or any
of its subsidiaries, as well as to Hanover or any of its subsidiaries as
debtor-in-possession. The relative rights of Senior Creditor and Junior Creditor
to repayment of the Senior Debt and the Junior Debt, respectively, and in or to
any distributions from or in respect of Debtor or any proceeds of Debtor's
property and assets, shall continue after the filing thereof on the same basis
as prior to the date of the petition, subject to any court order approving the
financing of, or use of cash collateral by, Hanover or any of its subsidiaries
as debtor-in-possession.


                                     - 14 -

<PAGE>   15
         4.4      Bankruptcy Financing. If Hanover or any of its subsidiaries
shall become subject to a proceeding under the U.S. Bankruptcy Code and if
Senior Creditor desires to permit the use of cash collateral or to provide
financing to Hanover or any of its subsidiaries under either Section 363 or
Section 364 of the U.S. Bankruptcy Code, Junior Creditor agrees as follows: (a)
adequate notice to Junior Creditor (if required) shall have been provided for
such financing or use of cash collateral if Junior Creditor receives notice two
(2) business days prior to the entry of the order approving such financing or
use of cash collateral and (b) no objection will be raised by Junior Creditor to
any such use of cash collateral or financing. For purposes of this Section,
notice of a proposed financing or use of cash collateral shall be deemed given
when given in the manner prescribed by Section 4.5 hereof to Junior Creditor.

         4.5      Notices. All notices, requests and demands to or upon the
respective parties hereto shall be in writing and shall be deemed to have been
duly given or made: if delivered in person, immediately upon delivery; if by
telex, telegram or facsimile transmission, immediately upon sending and upon
confirmation of receipt; if by nationally recognized overnight courier service
with instructions to deliver the next business day, one (1) business day after
sending; and if mailed by certified mail, return receipt requested, five (5)
days after mailing. All notices, requests and demands are to be given or made to
the respective parties at their addresses set forth below (or to such other
addresses as either party may designate by notice in accordance with the
provisions of this Section:

To Senior Creditor:                 Congress Financial Corporation
                                    1133 Avenue of the Americas
                                    New York, New York 10036
                                    Attention:  Mr. Mark Fagnani

To Junior Creditor:                 Swiss Bank Corporation
                                       New York Branch
                                    10 East 50th Street
                                    New York, New York 10022
                                    Attention: Jorg Rauthe

    with a copy to:                 Whitman Breed Abbott & Morgan
                                    200 Park Avenue
                                    New York, New York 10166
                                    Attention: D. de La Chapelle, Esq.

Either Creditor may change the address(es) to which all notices, requests and
other communications are to be sent by giving written notice of such address
change to the other Creditor in conformity with this Section 4.5, but such
change shall not be effective until notice of such change has been received by
the other Creditor.


                                     - 15 -

<PAGE>   16
          4.6  Counterparts. This Subordination Agreement may be executed in any
number of counterparts, each of which shall be an original with the same force
and effect as if the signatures thereto and hereto were upon the same
instrument.

          4.7  Governing Law. The validity, construction and effect of this
Subordination Agreement shall be governed by the laws of the State of New York
(without giving effect to principles of conflicts of law).

          4.8  Consent to Jurisdiction; Waiver of Jury Trial. Each of the 
parties hereto hereby irrevocably consents to the non-exclusive jurisdiction of
the Supreme Court of the State of New York for New York County and the United
States District Court for the Southern District of New York and waives trial by
jury in any action or proceeding with respect to this Subordination Agreement.

          4.9  Complete Agreement.  This written Subordination Agreement is 
intended by the parties as a final expression of their agreement and is intended
as a complete statement of the terms and conditions of their agreement.

         4.10 No Third Parties Benefitted. This Subordination Agreement is
solely for the benefit of the Creditors and their respective successors,
participants and assigns, and no other person shall have any right, benefit,
priority or interest under, or because of the existence of, this Subordination
Agreement.

         4.11 Disclosures, Non-Reliance. Each Creditor has the means to, and
shall in the future remain, fully informed as to the financial condition and
other affairs of Debtor and neither Creditor shall have any obligation or duty
to disclose any such information to any other Creditor. Except as expressly set
forth in this Subordination Agreement, the parties hereto have not otherwise
made to each other nor do they hereby make to each other any warranties, express
or implied, nor do they assume any liability to each other with respect to: (a)
the enforceability, validity, value or collectability of any of the Junior Debt
or the Senior Debt or any collateral or guarantee which may have been granted to
any of them in connection therewith, (b) Debtor's title to or right to any of
Debtor's assets and properties or (c) any other matter except as expressly set
forth in this Subordination Agreement.

         4.12 Term. This Subordination Agreement is a continuing agreement and
shall remain in full force and effect until the indefeasible satisfaction in
full of all Senior Debt and the termination of the financing arrangements among
Senior Creditor, Hanover and certain subsidiaries of Hanover.


                                     - 16 -

<PAGE>   17
         IN WITNESS WHEREOF, the parties have caused this Subordination
Agreement to be duly executed as of the day and year first above written.


                                              CONGRESS FINANCIAL CORPORATION

                                              By: /s/ John T. Garvey
                                                 ----------------------------
                                              Title: Assistant Vice President
                                                     ------------------------


                                              SWISS BANK CORPORATION,
                                                New York Branch

                                              By: /s/ Guido W. Schuler
                                                  --------------------------
                                              Title: Executive Director
                                                     -----------------------
                                              By: /s/ Jorg Rauthe
                                                  --------------------------
                                              Title: Associate Director
                                                     -----------------------


                                     - 17 -

<PAGE>   18
                                 ACKNOWLEDGMENT


         Each of the undersigned hereby acknowledges and agrees to the foregoing
terms and provisions. By its signature below, each of the undersigned agrees
that it shall, together with its successors and assigns, be bound by the
provisions hereof.

         Each of the undersigned acknowledges and agrees that: (i) although it
may sign this Subordination Agreement, it is not a party hereto and does not and
shall not receive any right, benefit, priority or interest under or because of
the existence of the foregoing Subordination Agreement, (ii) in the event of a
breach by the undersigned of any of the terms and provisions contained in the
foregoing Subordination Agreement, such a breach shall constitute an "Event of
Default" as defined in and under the Senior Creditor Agreements, and (iii) it
shall execute and deliver such additional documents and take such additional
action as may be necessary in the opinion of either Creditor to effectuate the
provisions and purposes of the foregoing Subordination Agreement.

                                         HANOVER DIRECT, INC.

                                         By: /s/ Edward J. O'Brien
                                            __________________________

                                         Title: Senior Vice President
                                                _______________________


                                         HANOVER DIRECT PENNSYLVANIA, INC.

                                         By: /s/ Edward J. O'Brien
                                            __________________________

                                         Title: Vice President
                                                _______________________


                                         BRAWN OF CALIFORNIA, INC.

                                         By: /s/ Edward J. O'Brien
                                            __________________________

                                         Title: Vice President
                                                _______________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                     - 18 -

<PAGE>   19
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                               GUMP'S BY MAIL, INC.

                                               By:   /s/ Edward J. O'Brien
                                                  -------------------------

                                               Title: Vice President
                                                     ----------------------


                                               GUMP'S CORP.

                                               By:   /s/ Edward J. O'Brien
                                                  -------------------------

                                               Title: Vice President
                                                     ----------------------


                                               THE COMPANY STORE, INC.

                                               By:   /s/ Edward J. O'Brien
                                                  -------------------------

                                               Title: Vice President
                                                     ----------------------


                                               TWEEDS, INC.

                                               By:   /s/ Edward J. O'Brien
                                                  -------------------------

                                               Title: Vice President
                                                     ----------------------


                                               LWI HOLDINGS, INC.

                                               By:   /s/ Edward J. O'Brien
                                                  -------------------------

                                               Title: Vice President
                                                     ----------------------


                                               AEGIS CATALOG CORPORATION

                                               By:   /s/ Edward J. O'Brien
                                                  -------------------------

                                               Title: Vice President
                                                     ----------------------


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                     - 19 -

<PAGE>   20
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                          HANOVER DIRECT VIRGINIA INC.

                                          By: /s/ Edward J. O'Brien
                                              ________________________

                                          Title:  Vice President
                                                ______________________


                                          HANOVER REALTY, INC.

                                          By: /s/ Edward J. O'Brien
                                              ________________________

                                          Title:  Vice President
                                                ______________________


                                          THE AUSTAD COMPANY

                                          By: /s/ Edward J. O'Brien
                                              ________________________

                                          Title:  Vice President
                                                ______________________


                                          AEGIS RETAIL CORPORATION  
                                          
                                          By: /s/ Edward J. O'Brien
                                              ________________________

                                          Title:  Vice President
                                                ______________________
                                          
                                          
                                          AEGIS SAFETY HOLDINGS, INC.
                                          
                                          By: /s/ Edward J. O'Brien
                                              ________________________

                                          Title:  Vice President
                                                ______________________
                                          
                                          
                                          AEGIS VENTURES, INC.
                                          
                                          By: /s/ Edward J. O'Brien
                                              ________________________

                                          Title:  Vice President
                                                ______________________
                                          
                        
                     [SIGNATURES CONTINUE ON FOLLOWING PAGE]


                                     - 20 -

<PAGE>   21
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                           AMERICAN DOWN & TEXTILE COMPANY
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           BRAWN WHOLESALE CORP.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           THE COMPANY FACTORY, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           THE COMPANY OFFICE, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           COMPANY STORE HOLDINGS, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           D.M. ADVERTISING, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________


                     [SIGNATURES CONTINUE ON FOLLOWING PAGE]


                                     - 21 -

<PAGE>   22
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                           GUMP'S CATALOG, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           GUMP'S HOLDINGS, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           HANOVER CASUALS, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           HANOVER CATALOG HOLDINGS, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           HANOVER FINANCE CORPORATION
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           HANOVER LIST MANAGEMENT, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________


                     [SIGNATURES CONTINUE ON FOLLOWING PAGE]


                                     - 22 -

<PAGE>   23
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                           HANOVER VENTURES, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           LEICHTUNG OF MICHIGAN, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           LWI RETAIL, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           SCANDIA DOWN CORPORATION
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           TWEEDS OF VERMONT, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                           YORK FULFILLMENT COMPANY, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________


                                           AUSTAD HOLDINGS, INC.
                                           
                                           By: /s/ Edward J. O'Brien
                                              ____________________________
                                           
                                           Title: Vice President
                                                 _________________________
                                           
                                           
                                     - 23 -