<PAGE>   1
                                                                Exhibit 10.41

                             SUBORDINATION AGREEMENT


         THIS SUBORDINATION AGREEMENT ("Subordination Agreement"), dated as of
December 18, 1996, is by and between CONGRESS FINANCIAL CORPORATION, a
California corporation ("Senior Creditor", as hereinafter further defined), and
RICHEMONT FINANCE S.A., a societe anonyme organized under the laws of the Grand
Duchy of Luxembourg ("Junior Creditor", as hereinafter further defined). Senior
Creditor and Junior Creditor are sometimes individually referred to herein as
"Creditor" and collectively as "Creditors."


                              W I T N E S S E T H:


         WHEREAS, Senior Creditor has entered into financing arrangements with
Hanover Direct, Inc. ("Hanover", as hereinafter further defined) and certain of
its subsidiaries, pursuant to which Senior Creditor has, upon certain terms and
conditions, made loans and provided other financial accommodations to certain
subsidiaries of Hanover, guaranteed by Hanover and certain subsidiaries of
Hanover, secured by substantially all of the assets and properties of Hanover
and of such borrower subsidiaries and guarantor subsidiaries of Hanover (such
subsidiaries, together with Hanover, individually and collectively, "Debtor", as
hereinafter further defined); and

         WHEREAS, pursuant to such financing arrangements, Senior Creditor has
previously arranged for the issuance of the CoreStates Letters of Credit (as
hereinafter defined); and

         WHEREAS, Swiss Bank (as hereinafter defined) is about to enter into the
Swiss Bank Reimbursement Agreement with Hanover, pursuant to which Swiss Bank
will issue the Swiss Bank Letters of Credit (as hereinafter defined) which will
be substituted for the CoreStates Letters of Credit and the NationsBank
Littlestown Letter of Credit, each of which shall, concurrently with such
substitution, be surrendered for cancellation or otherwise cancelled; and

         WHEREAS, Junior Creditor is about to execute a Guaranty in favor of
Swiss Bank of Hanover's reimbursement obligations to Swiss Bank and Debtor and
Junior Creditor are about to enter into a reimbursement agreement relating
thereto; and

         WHEREAS, in order to induce Senior Creditor to continue the financing
arrangements with Hanover and certain subsidiaries of Hanover, Junior Creditor
has agreed to the subordination in favor of Senior Creditor as provided herein
of its right to payment of the existing and future obligations of Debtor to
Junior Creditor arising in connection with or relating to the Swiss Bank Letters
<PAGE>   2
of Credit and the Guaranty by Junior Creditor relating thereto, and related
matters as set forth below;

         NOW THEREFORE, in consideration of the mutual benefits accruing to
Creditors hereunder and other good and valuable consideration, the receipt and
sufficiency of which is hereby acknowledged, the parties hereto do hereby agree
as follows:

         1. DEFINITIONS

         As used above and in this Subordination Agreement, the following terms
shall have the meanings ascribed to them below:

         1.1 "Agreements" shall mean, individually and collectively, the Senior
Creditor Agreements and the Junior Creditor Agreements.

         1.2 "CoreStates" shall mean CoreStates Bank, N.A., a national banking
association, and its successors and assigns.

         1.3 "Corestates Letters of Credit" shall mean, individually and
collectively, the CoreStates Series A Letter of Credit, the CoreStates Series B
Letter of Credit and the CoreStates Littlestown Backup Letter of Credit.

         1.4 "CoreStates Littlestown Backup Letter of Credit" shall mean the
Irrevocable Standby Letter of Credit No. 516262P, dated November 14, 1995,
issued in the initial stated amount of $8,560,000 by CoreStates for the account
of certain subsidiaries of Debtor, in favor of NationsBank, National
Association.

         1.5 "CoreStates Series A Letter of Credit" shall mean the Irrevocable
Transferable Letter of Credit No. 516466P, dated December 27, 1995, issued in
the initial stated amount of $10,145,833 by CoreStates for the account of
certain subsidiaries of Debtor, in favor of Norwest Bank Minnesota, N.A., as
Trustee under the Series A Note Agreements.

         1.6 "CoreStates Series B Letter of Credit" shall mean the Irrevocable
Transferable Letter of Credit No. 516467P, dated December 27, 1995, issued in
the initial stated amount of $10,145,833 by CoreStates for the account of
certain subsidiaries of Debtor in favor of Norwest Bank Minnesota Bank, N.A., as
Trustee under the Series B Note Agreements.

         1.7 "Creditors" shall mean, individually and collectively, Senior
Creditor and Junior Creditor and their respective successors and assigns.

         1.8 "Debtor" shall mean, individually and collectively, Hanover and
each of its existing and future subsidiaries who are or become parties (as
borrower or guarantor or who are or become

                                      - 2 -
<PAGE>   3
otherwise obligated for all or part of the Senior Debt) to the Senior Creditor
Agreements, and their respective successors and assigns, including, without
limitation, a receiver, trustee, or debtor-in-possession on behalf of any such
person or on behalf of any such successor or assign.

         1.9 "Hanover" shall mean Hanover Direct, Inc., a Delaware corporation
and its successors and assigns, including, without limitation, a receiver,
trustee or debtor-in-possession on behalf of such person or on behalf of any
such successor or assign.

         1.10 "Insolvency Proceeding" shall have the meaning given in Section
2.3 hereof.

         1.11 "Junior Creditor" shall mean Richemont Finance S.A., a societe
anonyme organized under the laws of the Grand Duchy of Luxembourg, and its
successors and assigns.

         1.12 "Junior Creditor Agreements" shall mean, individually and
collectively, the Richemont Reimbursement Agreement, the rights of Junior
Creditor at any time under any of the Swiss Bank Agreements, or under or by
reason of the Richemont Guaranty, and all agreements, documents and instruments
at any time executed and/or delivered by Debtor or any other person to, with or
in favor of Junior Creditor in connection therewith or related thereto, as all
of the foregoing now exist or may hereafter be amended, modified, supplemented,
extended, renewed, restated or replaced.

         1.13 "Junior Debt" shall mean all obligations, liabilities and
indebtedness of every kind, nature and description owing by Debtor to Junior
Creditor, including principal, interest, charges, fees, premiums, indemnities
and expenses, however evidenced, whether as principal, surety, endorser,
guarantor or otherwise, arising under or evidenced by or in connection with the
Junior Creditor Agreements, whether now existing or hereafter arising, whether
arising before, during or after the initial or any renewal term of the Junior
Creditor Agreements or after the commencement of any case with respect to Debtor
under the U.S. Bankruptcy Code or any similar statute (and including, without
limitation, any principal, interest, fees, costs, expenses and other amounts,
whether or not such amounts are allowable in whole or in part, in any such case
or similar proceeding), whether direct or indirect, absolute or contingent,
joint or several, due or not due, primary or secondary, liquidated or
unliquidated, secured or unsecured, and whether arising directly, or by way of
subrogation, contribution, reimbursement, indemnification, exoneration or
otherwise, or howsoever acquired by Junior Creditor.

         1.14 "Littlestown Bonds" shall mean the Variable Rate Demand
Industrial Development Revenue Refunding Bonds, 1987 Series

                                      - 3 -
<PAGE>   4
(Hanover House Industries, Inc. Project), issued on behalf of Hanover House
Industries Inc., now known as Hanover Direct Pennsylvania, Inc., in the original
principal amount of $8,000,000, as the same now exist or may hereafter be
amended, modified, supplemented, extended, renewed, restated or replaced.

         1.15 "Littlestown IDB Agreements" shall mean, collectively, the
Indenture of Trust, dated as of September 1, 1987, presently between the
Littlestown Industrial Development Authority and Fleet National Bank, as
successor Trustee in connection with the Littlestown Bonds, and all agreements,
documents and instruments at any time executed and/or delivered in connection
therewith, as the same now exist or may hereafter be amended, modified,
supplemented, extended, renewed, restated or replaced.

         1.16 "NationsBank Littlestown Letter of Credit" shall mean the letter
of credit no. 41055, dated November 2 ,1994, issued in the initial stated amount
of $8,560,000 by NationsBank for the account of Hanover Direct Pennsylvania,
Inc., in favor of Fleet National Bank, as Trustee under the Littlestown IDB
Agreements.

         1.17 "Person" or "person" shall mean any individual, sole
proprietorship, partnership, corporation (including, without limitation, any
corporation which elects subchapter S status under the Internal Revenue Code of
1986, as amended), business trust, unincorporated association, joint stock
company, trust, joint venture, limited liability company, limited liability
partnership, or other entity or any government or any agency or instrumentality
or political subdivision thereof.

         1.18 "Richemont Guaranty" shall mean the Guaranty, dated of even date
herewith, by Junior Creditor in favor of Swiss Bank with respect to the
obligations of Hanover to Swiss Bank, as the same now exists or may hereafter be
amended, modified, supplemented, extended, renewed, restated or replaced.

         1.19 "Richemont Reimbursement Agreement" shall mean the Hanover
Indemnity Agreement, dated of even date herewith, by and between Debtor and
Junior Creditor pursuant to which Debtor has agreed to reimburse Junior Creditor
for amounts that are paid by Junior Creditor in connection with the Swiss Bank
Letters of Credit and/or the Richemont Guaranty.

         1.20 "Senior Creditor" shall mean Congress Financial Corporation, a
California corporation, and its successors and assigns.

         1.21 "Senior Creditor Agreements" shall mean, individually and
collectively, the Loan and Security Agreement, dated November 14, 1995, by and
among Senior Creditor, Hanover and certain subsidiaries of Hanover, as amended
(the "Loan Agreement"), the Guarantee and Waivers, each dated November 14,

                                      - 4 -
<PAGE>   5
1995, by Hanover and certain of its subsidiaries in favor of Senior Creditor,
the General Security Agreements, each dated November 14, 1995, by Hanover and
certain of its subsidiaries in favor of Senior Creditor and all agreements,
documents and instruments at any time executed and/or delivered by Debtor or any
other person to, with or in favor of Senior Creditor in connection therewith or
related thereto, as all of the foregoing now exist or may hereafter be amended,
modified, supplemented, extended, renewed, restated or replaced.

         1.22 "Senior Debt" shall mean all obligations, liabilities and
indebtedness of every kind, nature and description owing by Debtor to Senior
Creditor and/or its affiliates, or participants, including principal, interest,
charges, fees, premiums, indemnities and expenses, however evidenced, whether as
principal, surety, endorser, guarantor or otherwise, whether arising under the
Senior Creditor Agreements or otherwise, whether now existing or hereafter
arising, whether arising before, during or after the initial or any renewal term
of the Senior Creditor Agreements or after the commencement of any case with
respect to Debtor under the U.S. Bankruptcy Code or any similar statute (and
including, without limitation, any principal, interest, fees, costs, expenses
and other amounts, whether or not such amounts are allowable either in whole or
in part, in any such case or similar proceeding), whether direct or indirect,
absolute or contingent, joint or several, due or not due, primary or secondary,
liquidated or unliquidated, secured or unsecured, and whether arising directly,
or by way of subrogation, contribution, reimbursement, indemnification,
exoneration or otherwise, or howsoever acquired by Senior Creditor.

         1.23 "Series A Note Agreements" shall mean, individually and
collectively, the Series A Note Agreement, dated as of November 9, 1994,
presently between Hanover and Norwest Bank Minnesota, N.A., as Trustee, with
respect to the issuance and sale of the Series A Notes, and all agreements,
documents and instruments at any time executed and/or delivered by Hanover or
any related parties in connection therewith, including, without limitation, the
Second Supplemental Series A Note Agreement, dated as of December 18, 1996, as
the same now exist or may hereafter be amended, modified, supplemented,
extended, renewed, restated or replaced.

         1.24 "Series A Notes" shall mean the Flexible Term Notes, Series A,
originally issued on November 9, 1994, by Hanover in the original aggregate
principal amount of $10,000,000, as replaced or modified through the date hereof
pursuant to the Series A Note Agreements, and as the same may hereafter be
amended, modified, supplemented, extended, renewed, restated or replaced.


                                      - 5 -
<PAGE>   6
         1.25 "Series B Note Agreements" shall mean, individually and
collectively, the Series B Note Agreement, dated as of April 27, 1995, presently
between Hanover and Norwest Bank Minnesota, N.A., as Trustee, with respect to
the issuance and sale of the Series B Notes, and all other agreements, documents
or instruments at any time executed and/or delivered in connection therewith,
including, without limitation, the Second Supplemental Series B Note Agreement,
dated as of December 18, 1996, as the same now exist or may hereafter be
amended, modified, supplemented, extended, renewed, restated or replaced.

         1.26 "Series B Notes" shall mean the Flexible Term Notes, Series B,
originally issued on April 27, 1995, by Hanover in the original aggregate
principal amount of $10,000,000, as replaced or modified through the date hereof
pursuant to the Series B Note Agreements, and as the same may hereafter be
amended, modified, supplemented, extended, renewed, restated or replaced.

         1.27 "Swiss Bank" shall mean, individually and collectively, Swiss Bank
Corporation, a banking corporation organized under the laws of Switzerland,
acting through its New York Branch, and any other issuer of a letter of credit
or guaranty that hereafter replaces or substitutes for any of the Swiss Bank
Letters of Credit, and their respective successors and assigns.

         1.28 "Swiss Bank Agreements" shall mean, individually and collectively,
(a) the Swiss Bank Reimbursement Agreement, (b) the Swiss Bank Letters of
Credit, (c) the rights of Swiss Bank under the Series A Note Agreements and any
Series A Notes at any time held by or for Swiss Bank, (d) the rights of Swiss
Bank under the Series B Note Agreements and any Series B Notes at any time held
by or for Swiss Bank, (e) the rights of Swiss Bank under the Littlestown IDB
Agreements and any Littlestown Bonds at any time held by or for Swiss Bank and
(f) all agreements, documents and instruments at any time executed and/or
delivered by Debtor or any other person to, with or in favor of Swiss Bank in
connection therewith or related thereto, as all of the foregoing now exist or
may hereafter be amended, modified, supplemented, extended, renewed, restated or
replaced.

         1.29 "Swiss Bank Letters of Credit" shall mean, collectively, the Swiss
Bank Littlestown Letter of Credit, the Swiss Bank Series A Letter of Credit and
the Swiss Bank Series B Letter of Credit, as the same now exist or may hereafter
be amended, modified, supplemented, extended, renewed, restated or replaced.

         1.30 "Swiss Bank Littlestown Letter of Credit" shall mean the Letter of
Credit No. S567171 dated December 18, 1996, issued by Swiss Bank for the account
of Hanover Direct Pennsylvania, Inc. in favor of Fleet National Bank, as Trustee
under the Littlestown IDB Agreements, in the initial stated amount of
$8,560,000, delivered in substitution for the NationsBank

                                      - 6 -
<PAGE>   7
Littlestown Letter of Credit and the CoreStates Littlestown Backup Letter of
Credit, as the same now exists or may hereafter be amended, modified,
supplemented, extended, renewed, restated or replaced.

         1.31 "Swiss Bank Reimbursement Agreement" shall mean the Reimbursement
Agreement, dated of even date herewith, by and between Hanover and Swiss Bank
with respect to the Swiss Bank Letters of Credit, as the same now exists or may
hereafter be amended, modified, supplemented, extended, renewed, restated or
replaced.

         1.32 "Swiss Bank Series A Letter of Credit" shall mean the Letter of
Credit No. S567169 dated December 18, 1996, issued by Swiss Bank for the account
of Hanover in the initial stated amount of $9,638,541, delivered in substitution
for the CoreStates Series A Letter of Credit, as the same now exists or may
hereafter be amended, modified, supplemented, extended, renewed, restated or
replaced.

         1.33 "Swiss Bank Series B Letter of Credit" shall mean the Letter of
Credit No. S567170, dated December 18, 1996, issued by Swiss Bank for the
account of Hanover in the initial stated amount of $9,638,541, delivered in
substitution for the CoreStates Series B Letter of Credit, as the same now
exists or may hereafter be amended, modified, supplemented, extended, renewed,
restated or replaced.

         1.34 All terms used herein which are defined in the Uniform Commercial
Code as in effect in the State of New York, unless otherwise defined herein
shall have the meanings set forth therein. All references to any term in the
plural shall include the singular and all references to any term in the singular
shall include the plural.

         2. SUBORDINATION OF JUNIOR DEBT

          2.1 Subordination. Except as specifically set forth in Section 2.2
hereof, Junior Creditor hereby subordinates its right to payment and
satisfaction of the Junior Debt, and the payment and satisfaction thereof,
directly or indirectly, by any means whatsoever, is hereby deferred, to the
prior indefeasible payment and satisfaction in full of all Senior Debt.

         2.2  Permitted Payments.

                  (a) Senior Creditor hereby agrees that, notwithstanding
anything to the contrary contained in Section 2.1 hereof, Debtor may make and
Junior Creditor may receive and retain from Debtor, a payment consisting of
expenses and attorneys' fees, not to exceed the aggregate amount of $45,000,
required to be reimbursed or paid by Debtor to Junior Creditor

                                      - 7 -
<PAGE>   8
incurred in connection with the issuance of the Swiss Bank Letters of Credit,
plus the fee in the sum of $1,391,854.10, payable to Junior Creditor pursuant to
Section 6 of the Richemont Reimbursement Agreement, as in effect on the date
hereof, but not any other payments whatsoever in respect of the Junior Debt.

                  (b) Debtor may make, and Junior Creditor may receive and
retain, payments by Debtor to Junior Creditor of its indemnification obligations
when due and payable pursuant to the provisions of Section 5 of the Richemont
Reimbursement Agreement, as in effect on the date hereof, excluding those
arising from the payment by Richemont of any amounts to Swiss Bank under the
Richemont Guaranty; provided, that the aggregate amount of such indemnification
obligations permitted to be so paid after the date hereof shall not exceed the
amount of $250,000; and provided, further, that no such payment otherwise
permitted shall be made if any "Event of Default" or "Incipient Default" (as
such terms are defined in the Loan Agreement referred to in the definition of
Senior Creditor Agreements) exists or has occurred and is continuing or would
exist or occur as a result thereof.

                  (c) Nothing herein contained shall restrict the right of
Junior Creditor to convert all or any part of the Junior Debt to common stock of
Hanover in accordance with the Richemont Reimbursement Agreement as in effect on
the date hereof.

         2.3 Distributions.

                  (a) In the event of any distribution, division, or
application, partial or complete, voluntary or involuntary, by operation of law
or otherwise, of all or any part of the assets of Debtor or the proceeds thereof
to the creditors of Debtor or readjustment of the obligations and indebtedness
of Debtor, whether by reason of liquidation, bankruptcy, arrangement,
receivership, assignment for the benefit of creditors, marshalling of assets of
Debtor or any other action or proceeding involving the readjustment of all or
any part of indebtedness of Debtor or the application of the assets of Debtor to
the payment or liquidation thereof (each of the foregoing, an "Insolvency
Proceeding"), or upon the dissolution or other winding up of Debtor's business,
or upon the sale of all or substantially all of Debtor's assets, then, and in
any such event, (i) Senior Creditor shall first receive indefeasible payment in
full in cash of all of the Senior Debt prior to the payment of all or any part
of the Junior Debt, and (ii) Senior Creditor shall be entitled to receive any
payment or distribution of any kind or character, whether in cash, securities or
other property, which is payable or deliverable in respect of any or all of the
Junior Debt.

                  (b) In order to enable Senior Creditor to enforce its rights
under Section 2.3(a) hereof, Senior Creditor is hereby irrevocably authorized
and empowered (in its own name or in the

                                      - 8 -
<PAGE>   9
name of Junior Creditor or otherwise), but shall have no obligation, to enforce
claims comprising any of the Junior Debt by proof of debt, proof of claim, suit
or otherwise and take generally any action which Junior Creditor might otherwise
be entitled to take, as Senior Creditor may deem necessary or advisable for the
enforcement of its rights or interests hereunder.

                  (c) To the extent necessary for Senior Creditor to realize the
benefits of the subordination of the Junior Debt provided for herein (including
the right to receive any and all payments and distributions which might
otherwise be payable or deliverable with respect to the Junior Debt in any
Insolvency Proceeding or otherwise), Junior Creditor shall execute and deliver
to Senior Creditor such instruments or documents (together with such assignments
or endorsements as Senior Creditor shall deem necessary), as may be requested by
Senior Creditor.

          2.4 Payments Received by Junior Creditor. Except for payments received
by Junior Creditor as provided in Section 2.2 hereof, should any payment or
distribution or security or instrument or proceeds thereof be received by the
Junior Creditor in respect of the Junior Debt, Junior Creditor shall receive and
hold the same in trust, as trustee, for the benefit of Senior Creditor,
segregated from other funds and property of Junior Creditor and shall forthwith
deliver the same to Senior Creditor (together with any endorsement or assignment
of Junior Creditor where necessary), for application to any of the Senior Debt.
In the event of the failure of Junior Creditor to make any such endorsement or
assignment to Senior Creditor, Senior Creditor, or any of its officers or
employees, are hereby irrevocably authorized on behalf of Junior Creditor to
make the same.

          2.5 Instrument Legend and Notation. Any instrument at any time
evidencing the Junior Debt, or any portion thereof, shall be permanently marked
on its face with a legend conspicuously indicating that payment thereof is
subordinate in right of payment to the Senior Debt and subject to the terms and
conditions of this Subordination Agreement, and (a) after being so marked
certified copies thereof shall be delivered to Senior Creditor and (b) the
original of any such instrument shall be immediately delivered to Senior
Creditor upon Senior Creditor's request, at any time on or after the
commencement of an Insolvency Proceeding. In the event any legend or endorsement
is omitted, Senior Creditor, or any of its officers or employees, are hereby
irrevocably authorized on behalf of Junior Creditor to make the same. No
specific legend, further assignment or endorsement or delivery of notes,
guarantees or instruments shall be necessary to subject any Junior Debt to the
subordination thereof contained in this Agreement.


                                      - 9 -
<PAGE>   10
         3. COVENANTS, REPRESENTATIONS AND WARRANTIES

          3.1  Additional Covenants.  Junior Creditor and Debtor
agree in favor of Senior Creditor that:

                  (a) Except as specifically set forth in Section 2.2 hereof,
Debtor shall not, directly or indirectly, make and Junior Creditor shall not,
directly or indirectly, accept or receive any payment of or any prepayment or
any payment pursuant to acceleration or claims of breach or any payment to
acquire Junior Debt or otherwise in respect of any Junior Debt;

                  (b) notwithstanding any rights or remedies available to it
under the Junior Creditor Agreements, applicable law or otherwise, Junior
Creditor shall not, directly or indirectly, (i) seek to collect from Debtor any
of the Junior Debt or exercise any of its rights or remedies upon a default or
event of default by Debtor under the Junior Creditor Agreements or otherwise or
(ii) commence any action or proceeding against Debtor or Debtor's properties
under the U.S. Bankruptcy Code or any state insolvency law or any similar
present or future statute, law or regulation or any proceedings for voluntary
liquidation, dissolution or other winding up of Debtor's business, or the
appointment of any trustee, receiver or liquidator for Debtor or any part of
Debtor's properties or any assignment for the benefit of creditors or any
marshalling of assets of Debtor or (iii) take any other action against Debtor or
Debtor's properties in respect of the Junior Debt;

                  (c) Debtor shall not grant to Junior Creditor and Junior
Creditor shall not acquire any security interest, lien, claim or encumbrance on
any assets or properties of Debtor, and Junior Creditor shall not acquire any
guarantees or other agreements under which any person, other than Debtor, is or
may become obligated, directly or indirectly, for all or any portion of the
Junior Debt;

                  (d) Junior Creditor and Debtor shall not amend, modify, alter
or change in any material respect the terms of any arrangements related to the
Junior Debt;

                  (e) Junior Creditor shall not sell, assign, pledge, encumber
or otherwise dispose of any of the Junior Debt, or subordinate any of the Junior
Debt to any indebtedness of Debtors other than the Senior Debt, except that
Junior Creditor may assign, expressly subject hereto, all of the Junior Debt to
an affiliate of Junior Creditor so long as (i) Junior Creditor provides Senior
Creditor five (5) days' prior written notice of such assignment, (ii) such
affiliate of Junior Creditor is reasonably acceptable to Senior Creditor, (iii)
contemporaneously with such assignment, Junior Creditor causes such affiliate to
enter into a written agreement in favor of Junior Creditor (the

                                     - 10 -
<PAGE>   11
"Richemont Affiliate Agreement"), in form and substance satisfactory to Senior
Creditor, pursuant to which such affiliate of Junior Creditor, shall agree,
among other things, to be bound by the terms of this Subordination Agreement,
and (iv) such affiliate of Junior Creditor delivers, in form and substance
satisfactory to Senior Creditor (A) evidence of the adoption and subsistence of
authorizing resolutions of such affiliate of Junior Creditor approving the
execution, delivery and performance by such affiliate of Junior Creditor of the
Richemont Affiliate Agreement and (B) an opinion of counsel to such affiliate of
Junior Creditor addressed to Senior Creditor with respect to the due
authorization, execution, validity and enforceability of the Richemont Affiliate
Agreement, and as to such other matters as Senior Creditor shall reasonably
require;

                  (f) Junior Creditor and Debtor shall, at any time or times
upon the request of Senior Creditor, promptly furnish to Senior Creditor a true,
correct and complete statement of the outstanding Junior Debt; and

                  (g) Junior Creditor and Debtor shall execute and deliver to
Senior Creditor such additional agreements, documents and instruments and take
such further actions as may be necessary or desirable in the opinion of Senior
Creditor to effectuate the provisions and purposes of this Subordination
Agreement.

         3.2 Additional Representations and Warranties. Junior Creditor and
Debtor represent and warrant to Senior Creditor that:

                  (a) Junior Creditor has no guarantee or other agreement from
any person, other than Debtor, under which such person is or may become
obligated, directly or indirectly, for all or any portion of the Junior Debt,
and Junior Creditor has no security interest, lien, claim or encumbrance on any
assets and properties of Debtor and the Junior Debt is unsecured;

                  (b) as of the date hereof, no default or event of default, or
event which with notice or passage of time or both would constitute an event of
default exists or has occurred under the Junior Creditor Agreements;

                  (c) Junior Creditor is the exclusive legal and beneficial
owner of all of the Junior Debt;

                  (d) none of the Junior Debt is subject to any lien, security
interest, financing statements, subordination, assignment or other claim, except
in favor of Senior Creditor; and


                                     - 11 -
<PAGE>   12
                  (e) this Subordination Agreement constitutes the legal, valid
and binding obligations of Junior Creditor, enforceable in accordance with its
terms.

          3.3 Waivers. Notice of acceptance hereof, the making of loans,
advances and extensions of credit or other financial accommodations to, and the
incurring of any expenses by or in respect of, Hanover or its subsidiaries by
Senior Creditor, and presentment, demand, protest, notice of protest, notice of
nonpayment or default and all other notices to which Junior Creditor and Debtor
are or may be entitled are hereby waived (except as expressly provided for
herein or as to Debtor, in the Senior Creditor Agreements). Junior Creditor also
waives notice of, and hereby consents to, (a) any amendment, modification,
supplement, renewal, restatement or extensions of time of payment of or increase
or decrease in the amount of any of the Senior Debt or to the Senior Creditor
Agreements or any collateral at any time granted to or held by Senior Creditor,
(b) the taking, exchange, surrender and releasing of collateral at any time
granted to or held by Senior Creditor or guarantees now or at any time held by
or available to Senior Creditor for the Senior Debt or any other person at any
time liable for or in respect of the Senior Debt, (c) the exercise of, or
refraining from the exercise of any rights against Debtor or any other obligor
or any collateral at any time granted to or held by Senior Creditor, (d) the
settlement, compromise or release of, or the waiver of any default with respect
to, any of the Senior Debt, and/or (e) Senior Creditor's election, in any
proceeding instituted under the U.S. Bankruptcy Code of the application of
Section 1111(b)(2) of the U.S. Bankruptcy Code. Any of the foregoing shall not,
in any manner, affect the terms hereof or impair the obligations of Junior
Creditor hereunder. All of the Senior Debt shall be deemed to have been made or
incurred in reliance upon this Subordination Agreement.

          3.4 Subrogation; Marshalling. Junior Creditor shall not be subrogated
to, or be entitled to any assignment of any Senior Debt or Junior Debt or of any
collateral for or guarantees or evidence of any thereof until all of the Senior
Debt is indefeasibly paid and satisfied in full. Junior Creditor hereby waives
any and all rights to have any collateral or any part thereof granted to or held
by Senior Creditor marshalled upon any foreclosure or other disposition of such
collateral by Senior Creditor or Debtor with the consent of Senior Creditor.

          3.5 No Offset. In the event Junior Creditor at any time incurs any
obligation to pay money to Debtor, Junior Creditor hereby irrevocably agrees
that it shall pay such obligation in cash or cash equivalents in accordance with
the terms of the contract governing such obligation and shall not deduct from or
setoff against any amounts owed by Junior Creditor to Debtor in

                                     - 12 -
<PAGE>   13
connection with any such transaction any amounts the Junior Creditor claims are
due to it with respect to the Junior Debt.

          4. MISCELLANEOUS

          4.1 Amendments. Any waiver, permit, consent or approval by either
Creditor of or under any provision, condition or covenant to this Subordination
Agreement must be in writing and shall be effective only to the extent it is set
forth in writing and as to the specific facts or circumstances covered thereby.
Any amendment of this Subordination Agreement must be in writing and signed by
each of the parties to be bound thereby.

          4.2 Successors and Assigns.

                  (a) This Subordination Agreement shall be binding upon the
parties hereto and their respective successors and assigns and shall inure to
the benefit of each of Creditors and its respective successors, participants and
assigns.

                  (b) Senior Creditor reserves the right to grant participations
in, or otherwise sell, assign, transfer or negotiate all or any part of, or any
interest in, the Senior Debt and the collateral securing same; provided, that,
Junior Creditor shall not be obligated to give any notices to or otherwise in
any manner deal directly with any participant in the Senior Debt and no
participant shall be entitled to any rights or benefits under this Subordination
Agreement except through Senior Creditor. In connection with any participation
or other transfer or assignment, Senior Creditor (i) may disclose to such
assignee, participant or other transferee or assignee all documents and
information which Senior Creditor now or hereafter may have relating to the
Senior Debt or any collateral and (ii) shall disclose to such participant or
other transferee or assignee the existence and terms and conditions of this
Subordination Agreement.

                  (c) In connection with any assignment or transfer of any or
all of the Senior Debt, or any or all rights of Senior Creditor in any of the
property of Hanover or its subsidiaries (other than pursuant to a
participation), Junior Creditor agrees to execute and deliver an agreement
containing terms substantially identical to those contained herein in favor of
any such assignee or transferee and, in addition, will execute and deliver an
agreement containing terms substantially identical to those contained herein in
favor of any third person who succeeds to or replaces any or all of Senior
Creditor's financing of certain subsidiaries of Hanover, whether such successor
financing or replacement occurs by transfer, assignment, "takeout" or any other
means.


                                     - 13 -
<PAGE>   14
          4.3 Insolvency. This Subordination Agreement shall be applicable both
before and after the filing of any petition by or against Hanover or any of its
subsidiaries under the U.S. Bankruptcy Code and all converted or succeeding
cases in respect thereof, and all references herein to Debtor or any of
Hanover's subsidiaries shall be deemed to apply to a trustee for Hanover or any
of its subsidiaries, as well as to Hanover or any of its subsidiaries as
debtor-in-possession. The relative rights of Senior Creditor and Junior Creditor
to repayment of the Senior Debt and the Junior Debt, respectively, and in or to
any distributions from or in respect of Debtor or any proceeds of Debtor's
property and assets, shall continue after the filing thereof on the same basis
as prior to the date of the petition, subject to any court order approving the
financing of, or use of cash collateral by, Hanover or any of its subsidiaries
as debtor-in-possession.

          4.4 Bankruptcy Financing. If Hanover or any of its subsidiaries shall
become subject to a proceeding under the U.S. Bankruptcy Code and if Senior
Creditor desires to permit the use of cash collateral or to provide financing to
Hanover or any of its subsidiaries under either Section 363 or Section 364 of
the U.S. Bankruptcy Code, Junior Creditor agrees as follows: (a) adequate notice
to Junior Creditor (if required) shall have been provided for such financing or
use of cash collateral if Junior Creditor receives notice two (2) business days
prior to the entry of the order approving such financing or use of cash
collateral and (b) no objection will be raised by Junior Creditor to any such
use of cash collateral or financing. For purposes of this Section, notice of a
proposed financing or use of cash collateral shall be deemed given when given in
the manner prescribed by Section 4.5 hereof to Junior Creditor.

          4.5 Notices. All notices, requests and demands to or upon the
respective parties hereto shall be in writing and shall be deemed to have been
duly given or made: if delivered in person, immediately upon delivery; if by
telex, telegram or facsimile transmission, immediately upon sending and upon
confirmation of receipt; if by nationally recognized overnight courier service
with instructions to deliver the next business day, one (1) business day after
sending; and if mailed by certified mail, return receipt requested, five (5)
days after mailing. All notices, requests and demands are to be given or made to
the respective parties at their addresses set forth below (or to such other
addresses as either party may designate by notice in accordance with the
provisions of this Section:

To Senior Creditor:          Congress Financial Corporation
                             1133 Avenue of the Americas
                             New York, New York 10036
                             Attention:  Mr. Mark Fagnani


                                     - 14 -
<PAGE>   15
To Junior Creditor:               Richemont Finance S.A.
                                  35 Boulevard Prince Henri
                                  L 1724 Luxembourg
                                  Attention:  Mr. Alan Grieve

         with a copy to:          Robert P. Wessely, Esq.
                                  Brobeck, Phleger & Harrison LLP
                                  1633 Broadway, 47th Floor
                                  New York, New York 10019

Either Creditor may change the address(es) to which all notices, requests and
other communications are to be sent by giving written notice of such address
change to the other Creditor in conformity with this Section 4.5, but such
change shall not be effective until notice of such change has been received by
the other Creditor.

          4.6 Counterparts. This Subordination Agreement may be executed in any
number of counterparts, each of which shall be an original with the same force
and effect as if the signatures thereto and hereto were upon the same
instrument.

          4.7 Governing Law. The validity, construction and effect of this
Subordination Agreement shall be governed by the laws of the State of New York
(without giving effect to principles of conflicts of law).

          4.8 Consent to Jurisdiction; Waiver of Jury Trial. Each of the parties
hereto hereby irrevocably consents to the non-exclusive jurisdiction of the
Supreme Court of the State of New York for New York County and the United States
District Court for the Southern District of New York and waives trial by jury in
any action or proceeding with respect to this Subordination Agreement.

          4.9  Complete Agreement.

                  (a) This written Subordination Agreement is intended by the
parties as a final expression of their agreement and is intended as a complete
statement of the terms and conditions of their agreement.

                  (b) The obligations of Junior Creditor under this
Subordination Agreement are in addition to, and in no way limited by the terms
of the Subordination Agreement, dated as of December 5, 1996, between Junior
Creditor and Senior Creditor, as acknowledged by Hanover, nor shall any of the
terms thereof be limited or affected by the terms of this Subordination
Agreement.

          4.10 No Third Parties Benefitted. This Subordination Agreement is
solely for the benefit of the Creditors and their respective successors,
participants and assigns, and no other

                                     - 15 -
<PAGE>   16
person shall have any right, benefit, priority or interest under, or because of
the existence of, this Subordination Agreement.

         4.11 Disclosures, Non-Reliance. Each Creditor has the means to, and
shall in the future remain, fully informed as to the financial condition and
other affairs of Debtor and neither Creditor shall have any obligation or duty
to disclose any such information to any other Creditor. Except as expressly set
forth in this Subordination Agreement, the parties hereto have not otherwise
made to each other nor do they hereby make to each other any warranties, express
or implied, nor do they assume any liability to each other with respect to: (a)
the enforceability, validity, value or collectability of any of the Junior Debt
or the Senior Debt or any collateral or guarantee which may have been granted to
any of them in connection therewith, (b) Debtor's title to or right to any of
Debtor's assets and properties or (c) any other matter except as expressly set
forth in this Subordination Agreement.

         4.12 Term. This Subordination Agreement is a continuing agreement and
shall remain in full force and effect until the indefeasible satisfaction in
full of all Senior Debt and the termination of the financing arrangements among
Senior Creditor, Hanover and certain subsidiaries of Hanover.

         IN WITNESS WHEREOF, the parties have caused this Subordination
Agreement to be duly executed as of the day and year first above written.


                                             CONGRESS FINANCIAL CORPORATION

                                             By: /s/ John T. Garvey
                                                ____________________________

                                             Title: Assistant Vice President


                                             RICHEMONT FINANCE S.A.

                                             By: /s/ Jan du Plessis
                                                ____________________________

                                             Title: Director
                                                   _________________________

                                             By: /s/ Howard M.S. Tanner
                                                ____________________________

                                             Title: Director
                                                   _________________________

                                     - 16 -
<PAGE>   17
                                 ACKNOWLEDGMENT


         Each of the undersigned hereby acknowledges and agrees to the foregoing
terms and provisions. By its signature below, each of the undersigned agrees
that it shall, together with their successors and assigns, be bound by the
provisions hereof.

         Each of the undersigned acknowledges and agrees that: (i) although it
may sign this Subordination Agreement, it is not a party hereto and does not and
shall not receive any right, benefit, priority or interest under or because of
the existence of the foregoing Subordination Agreement, (ii) in the event of a
breach by the undersigned of any of the terms and provisions contained in the
foregoing Subordination Agreement, such a breach shall constitute an "Event of
Default" as defined in and under the Senior Creditor Agreements, and (iii) it
shall execute and deliver such additional documents and take such additional
action as may be necessary in the opinion of either Creditor to effectuate the
provisions and purposes of the foregoing Subordination Agreement.

                                   HANOVER DIRECT, INC.

                                   By: /s/ Edward J. O'Brien
                                      __________________________

                                   Title: Senior Vice President
                                          Secretary & Treasurer
                                         _______________________


                                   HANOVER DIRECT PENNSYLVANIA, INC.

                                   By: /s/ Edward J. O'Brien
                                      _________________________

                                   Title: Vice President
                                         ______________________


                                   BRAWN OF CALIFORNIA, INC.

                                   By: /s/ Edward J. O'Brien
                                      _________________________

                                   Title: Vice President
                                         ______________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]

                                     - 17 -
<PAGE>   18
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                      GUMP'S BY MAIL, INC.

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                                      GUMP'S CORP.

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                                      THE COMPANY STORE, INC.

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                                      TWEEDS, INC.

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                                      LWI HOLDINGS, INC.

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                                      AEGIS CATALOG CORPORATION

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]

                                     - 18 -
<PAGE>   19
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                      HANOVER DIRECT VIRGINIA INC.

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                                      HANOVER REALTY, INC.

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                                      THE AUSTAD COMPANY

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                                      AEGIS RETAIL CORPORATION

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                                      AEGIS SAFETY HOLDINGS, INC.

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                                      AEGIS VENTURES, INC.

                                      By: /s/ Edward J. O'Brien
                                          _________________________

                                      Title: Vice President
                                             ______________________


                     [SIGNATURES CONTINUE ON FOLLOWING PAGE]

                                     - 19 -
<PAGE>   20
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                        AMERICAN DOWN & TEXTILE COMPANY

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------


                                        BRAWN WHOLESALE CORP.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------


                                        THE COMPANY FACTORY, INC.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------


                                        THE COMPANY OFFICE, INC.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------


                                        COMPANY STORE HOLDINGS, INC.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------


                                        D.M. ADVERTISING, INC.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------


                     [SIGNATURES CONTINUE ON FOLLOWING PAGE]

                                     - 20 -
<PAGE>   21
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]



                                        GUMP'S CATALOG, INC.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------


                                        GUMP'S HOLDINGS, INC.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------


                                        HANOVER CASUALS, INC.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------


                                        HANOVER CATALOG HOLDINGS, INC.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------


                                        HANOVER FINANCE CORPORATION

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------


                                        HANOVER LIST MANAGEMENT, INC.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                        Title: Vice President
                                               ------------------------



                     [SIGNATURES CONTINUE ON FOLLOWING PAGE]

                                     - 21 -
<PAGE>   22
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]




                                   HANOVER VENTURES, INC.

                                   By: /s/ Edward J. O'Brien
                                       ---------------------------
                                   Title: Vice President
                                          ------------------------


                                   LEICHTUNG OF MICHIGAN, INC.

                                   By: /s/ Edward J. O'Brien
                                       ---------------------------
                                   Title: Vice President
                                          ------------------------


                                   LWI RETAIL, INC.

                                   By: /s/ Edward J. O'Brien
                                       ---------------------------
                                   Title: Vice President
                                          ------------------------


                                   SCANDIA DOWN CORPORATION

                                   By: /s/ Edward J. O'Brien
                                       ---------------------------
                                   Title: Vice President
                                          ------------------------


                                   TWEEDS OF VERMONT, INC.

                                   By: /s/ Edward J. O'Brien
                                       ---------------------------
                                   Title: Vice President
                                          ------------------------


                                   YORK FULFILLMENT COMPANY, INC.

                                   By: /s/ Edward J. O'Brien
                                       ---------------------------
                                   Title: Vice President
                                          ------------------------


                                   AUSTAD HOLDINGS, INC.

                                   By: /s/ Edward J. O'Brien
                                       ---------------------------
                                   Title: Vice President
                                          ------------------------




                                     - 22 -