<PAGE>   1
                                                                    EXHIBIT 10.5


                            SUBORDINATION AGREEMENT


         THIS SUBORDINATION AGREEMENT ("Subordination Agreement"), dated as of
March 24, 2000, is by and between CONGRESS FINANCIAL CORPORATION, a Delaware
corporation ("Senior Creditor", as hereinafter further defined), and RICHEMONT
FINANCE S.A., a societe anonyme organized under the laws of the Grand Duchy of
Luxembourg ("Junior Creditor", as hereinafter further defined).  Senior
Creditor and Junior Creditor are sometimes individually referred to herein as
"Creditor" and collectively as "Creditors."


                              W I T N E S S E T H:


         WHEREAS, Junior Creditor has made a loan in the original principal
amount of up to $25,000,000 to Hanover Direct, Inc. ("Debtor", as hereinafter
defined), which loan is and shall be unsecured; and

         WHEREAS, Senior Creditor has entered into financing arrangements with
Debtor and certain of its subsidiaries, pursuant to which Senior Creditor has,
upon certain terms and conditions, made loans and provided other financial
accommodations to certain subsidiaries of Debtor, guaranteed by Debtor and
certain subsidiaries of Debtor, secured by substantially all of the assets and
properties of Debtor and of such borrower subsidiaries and guarantor
subsidiaries of Debtor; and

         WHEREAS, in order to induce Senior Creditor to continue the financing
arrangements with Debtor and certain subsidiaries of Debtor, Junior Creditor
has agreed to the subordination in favor of Senior Creditor as provided herein
of its right to payment of the existing and future obligations of Debtor to
Junior Creditor arising in connection with a loan of up to $25,000,000 to
Debtor to the prior indefeasible payment of the existing and future obligations
of Debtor to Senior Creditor under the Senior Creditor Agreements, and related
matters as set forth below;

         NOW THEREFORE, in consideration of the mutual benefits accruing to
Creditors hereunder and other good and valuable consideration, the receipt and
sufficiency of which is hereby acknowledged, the parties hereto do hereby agree
as follows:

         1.   DEFINITIONS

         As used above and in this Subordination Agreement, the following terms
shall have the meanings ascribed to them below:
<PAGE>   2
         1.1   "Agreements" shall mean, individually and collectively, the
Senior Creditor Agreements and the Junior Creditor Agreements.

         1.2   "Banking Day" shall mean any day, other than Saturday or Sunday,
when Senior Creditor and commercial banks are open in New York, New York and
Europe.

         1.3   "Creditors" shall mean, individually and collectively, Senior
Creditor and Junior Creditor and their respective successors and assigns.

         1.4   "Debtor" shall mean Hanover Direct, Inc., a Delaware
corporation, and its successors and assigns, including, without limitation, a
receiver, trustee, or debtor-in-possession on behalf of any such person or on
behalf of any such successor or assign.

         1.5   "Event of Default" shall have the meaning given in the Loan
Agreement.

         1.6   "Incipient Default" shall have the meaning given in the Loan
Agreement.

         1.7   "Excess Loan Availability" shall have the meaning given in the
Loan Agreement.

         1.8   "Insolvency Proceeding" shall have the meaning given in Section
2.3 hereof.

         1.9   "Junior Creditor" shall mean Richemont Finance S.A., a societe
anonyme organized under the laws of the Grand Duchy of Luxembourg, and its
successors and assigns.

         1.10  "Junior Creditor Agreements" shall mean, individually and
collectively, the "Hanover Direct, Inc. - Unsecured Line of Credit & Promissory
Note", dated March 1, 2000, by and between Debtor and Junior Creditor in the
original principal amount of up to $25,000,000, and all agreements, documents
and instruments at any time executed and/or delivered by Debtor or any other
person to, with or in favor of Junior Creditor in connection therewith or
related thereto, as all of the foregoing now exist or may hereafter be amended,
modified, supplemented, extended, renewed, restated or replaced.

         1.11  "Junior Debt" shall mean all obligations, liabilities and
indebtedness of every kind, nature and description owing by Debtor to Junior
Creditor under the Junior Creditor Agreements, including principal, interest,
charges, fees, premiums, indemnities and expenses, however evidenced, whether
as principal, surety, endorser, guarantor or otherwise, arising under or
evidenced by or in connection with the Junior Creditor Agreements, whether now
existing or hereafter arising, whether arising before, during or after the
initial or any renewal term of the Junior Creditor Agreements or after the
commencement of any case with respect to Debtor under the U.S. Bankruptcy Code
or any similar statute (and including, without limitation, any principal,
interest, fees, costs, expenses and other amounts, whether or not such amounts
are allowable in whole or in part, in any such case or similar proceeding),
whether direct or indirect, absolute or contingent, joint or several, due or
not due, primary or secondary, liquidated or unliquidated,




                                     - 2 -
<PAGE>   3
secured or unsecured, and whether arising directly, or by way of subrogation,
contribution, reimbursement, indemnification, exoneration or otherwise, or
howsoever acquired by Junior Creditor in connection with the Junior Creditor
Agreements.

         1.12   "Person" or "person" shall mean any individual, sole
proprietorship, partnership, corporation (including, without limitation, any
corporation which elects subchapter S status under the Internal Revenue Code of
1986, as amended), business trust, unincorporated association, joint stock
company, trust, joint venture, limited liability company, limited liability
partnership, or other entity or any government or any agency or instrumentality
or political subdivision thereof.

         1.13   "Senior Creditor" shall mean Congress Financial Corporation, a
Delaware corporation, and its successors and assigns.

         1.14   "Senior Creditor Agreements" shall mean, individually and
collectively, the Loan and Security Agreement, dated November 14, 1995, by and
among Senior Creditor, Debtor and certain subsidiaries of Debtor, as amended
through the Fifteenth Amendment to Loan and Security Agreement, dated as of the
date hereof (the "Loan Agreement"), any related guarantees by Debtor in favor
of Lender, any related security agreements, by Debtor in favor of Senior
Creditor and all agreements, documents and instruments at any time executed
and/or delivered by Debtor or any other person to, with or in favor of Senior
Creditor in connection therewith or related thereto, as all of the foregoing
now exist or, subject to the terms and conditions contained in this
Subordination Agreement, as may hereafter be amended, modified, supplemented,
extended, renewed, restated or replaced.

         1.15   "Senior Debt" shall mean all obligations, liabilities and
indebtedness of every kind, nature and description owing by Debtor or any other
Subsidiary of Debtor to Senior Creditor and/or its affiliates, or participants,
including principal, interest, charges, fees, premiums, indemnities and
expenses, however evidenced, whether as principal, surety, endorser, guarantor
or otherwise, owing in connection with the Senior Creditor Agreements, whether
now existing or hereafter arising, whether arising before, during or after the
initial or, subject to the terms and conditions contained in this Subordination
Agreement, any renewal term of the Senior Creditor Agreements or after the
commencement of any case with respect to Debtor under the U.S. Bankruptcy Code
or any similar statute (and including, without limitation, any principal,
interest, fees, costs, expenses and other amounts, whether or not such amounts
are allowable either in whole or in part, in any such case or similar
proceeding), whether direct or indirect, absolute or contingent, joint or
several, due or not due, primary or secondary, liquidated or unliquidated,
secured or unsecured, and whether arising directly, or by way of subrogation,
contribution, reimbursement, indemnification, exoneration or otherwise, or
howsoever acquired by Senior Creditor in connection with the Senior Creditor
Agreements.

         1.16   All terms used herein which are defined in the Uniform
Commercial Code as in effect in the State of New York, unless otherwise defined
herein shall have the meanings set forth




                                     - 3 -
<PAGE>   4
therein.  All references to any term in the plural shall include the singular
and all references to any term in the singular shall include the plural.

         2.      SUBORDINATION OF JUNIOR DEBT

         2.1   Subordination.  Except as specifically set forth in Section 2.2
hereof, Junior Creditor hereby subordinates its right to payment and
satisfaction of the Junior Debt, and the payment and satisfaction thereof,
directly or indirectly, by any means whatsoever, is hereby deferred, to the
prior indefeasible payment and satisfaction in full of all Senior Debt.

         2.2   Permitted Payments.  Senior Creditor hereby agrees that,
notwithstanding anything to the contrary contained in Section 2.1 hereof,

                 (a)  unless and until Senior Creditor sends written notice to
Junior Creditor of the occurrence of an Event of Default or Incipient Default
under the Senior Creditor Agreements, Debtor may make and Junior Creditor may
receive and retain from Debtor, regularly scheduled payments of interest in
respect of the Junior Debt and of a monthly fee in the amount of $62,500, in
each case as provided by the Junior Creditor Agreements as in effect on the
date hereof; and

                 (b)      Debtor may make, and Junior Creditor may receive and
retain, payments by Debtor to Junior Creditor of the Junior Debt solely out of
cash proceeds pursuant to a rights offering by Hanover Direct, Inc., Hanover
Brands, Inc. or erizon, inc. or any other equity offering(s) or equity private
placement(s) of capital stock of Hanover Direct, Inc,  Hanover Brands, Inc. or
erizon, inc. and Junior Creditor may convert the then outstanding amount of
Junior Debt for capital stock of Hanover Direct, Inc. or Hanover Brands, Inc.
or erizon, inc.; provided, that, (i) Senior Creditor has received not less than
fifteen (15) Banking Days prior written notice from Debtor of the intention to
make such payment out of cash proceeds of such rights offering or other equity
offering or such conversion of the Junior Debt to capital stock, (ii) such
capital stock consists of ordinary common stock as in effect on the date hereof
or of other capital shares if consented to by Senior Creditor, which consent
shall not be unreasonably withheld in Senior Creditor's good faith judgment.
To the extent any such cash proceeds of any such equity offering or conversion
of Junior Debt permitted hereby reduces the amount of any commitment to advance
funds pursuant to the Junior Creditor Agreements, any such net cash proceeds or
conversion shall not also reduce the amount of any commitment to advance funds
pursuant to the Richemont $10,000,000 Credit Agreements as such term is defined
in the Senior Creditor Agreements.

         2.3   Distributions.

                 (a)  In the event of any distribution, division, or
application, partial or complete, voluntary or involuntary, by operation of law
or otherwise, of all or any part of the assets of Debtor or the proceeds
thereof to the creditors of Debtor or readjustment of the obligations and
indebtedness of Debtor, whether by reason of liquidation, bankruptcy,
arrangement, receivership,




                                     - 4 -
<PAGE>   5
assignment for the benefit of creditors, marshalling of assets of Debtor or any
other action or proceeding involving the readjustment of all or any part of
indebtedness of Debtor or the application of the assets of Debtor to the payment
or liquidation thereof (each of the foregoing, an "Insolvency Proceeding"), or
upon the dissolution or other winding up of Debtor's business, or upon the sale
of all or substantially all of Debtor's assets, then, and in any such event, (i)
Senior Creditor shall first receive indefeasible payment in full in cash of all
of the Senior Debt prior to the payment of all or any part of the Junior Debt,
and (ii) Senior Creditor shall be entitled to receive any payment or
distribution of any kind or character, whether in cash, securities or other
property, which is payable or deliverable in respect of any or all of the Junior
Debt.

                 (b)  In order to enable Senior Creditor to enforce its rights
under Section 2.3(a) hereof, Senior Creditor is hereby irrevocably authorized
and empowered (in its own name or in the name of Junior Creditor or otherwise),
but shall have no obligation, to enforce claims comprising any of the Junior
Debt by proof of debt, proof of claim, suit or otherwise and take generally any
action which Junior Creditor might otherwise be entitled to take, as Senior
Creditor may deem necessary or advisable for the enforcement of its rights or
interests hereunder.

                 (c)  To the extent necessary for Senior Creditor to realize
the benefits of the subordination of the Junior Debt provided for herein
(including the right to receive any and all payments and distributions which
might otherwise be payable or deliverable with respect to the Junior Debt in
any Insolvency Proceeding or otherwise), Junior Creditor shall execute and
deliver to Senior Creditor such instruments or documents (together with such
assignments or endorsements as Senior Creditor shall deem necessary), as may be
reasonably requested by Senior Creditor.

         2.4  Payments Received by Junior Creditor.  Except for payments
received by Junior Creditor as provided in Section 2.2 hereof, should any
payment or distribution or security or instrument or proceeds thereof be
received by the Junior Creditor in respect of the Junior Debt, Junior Creditor
shall receive and hold the same in trust, as trustee, for the benefit of Senior
Creditor, segregated from other funds and property of Junior Creditor and shall
forthwith deliver the same to Senior Creditor (together with any endorsement or
assignment of Junior Creditor where necessary), for application to any of the
Senior Debt.  In the event of the failure of Junior Creditor to make any such
endorsement or assignment to Senior Creditor, Senior Creditor, or any of its
officers or employees, are hereby irrevocably authorized on behalf of Junior
Creditor to make the same.

         2.5  Instrument Legend and Notation.  Any instrument at any time
evidencing the Junior Debt, or any portion thereof, shall be permanently marked
on its face with a legend conspicuously indicating that payment thereof is
subordinate in right of payment to the Senior Debt and subject to the terms and
conditions of this Subordination Agreement, and (a) after being so marked
certified copies thereof shall be delivered to Senior Creditor and (b) the
original of any such instrument shall be immediately delivered to Senior
Creditor upon Senior Creditor's request, at any time on or after the
commencement of an Insolvency Proceeding.  In the event




                                     - 5 -
<PAGE>   6
any legend or endorsement is omitted, Senior Creditor, or any of its officers or
employees, are hereby irrevocably authorized on behalf of Junior Creditor to
make the same.  No specific legend, further assignment or endorsement or
delivery of notes, guarantees or instruments shall be necessary to subject any
Junior Debt to the subordination thereof contained in this Agreement.

         3.   COVENANTS, REPRESENTATIONS AND WARRANTIES

         3.1   Additional Covenants.  Junior Creditor and Debtor agree in
favor of Senior Creditor that:

                 (a)  Except as specifically set forth in Section 2.2 hereof,
Debtor shall not, directly or indirectly, make and Junior Creditor shall not,
directly or indirectly, accept or receive any payment of or any prepayment or
any payment pursuant to acceleration or claims of breach or any payment to
acquire Junior Debt or otherwise in respect of any Junior Debt;

                 (b)  notwithstanding any rights or remedies available to it
under the Junior Creditor Agreements, applicable law or otherwise, Junior
Creditor shall not, directly or indirectly, (i) seek to collect from Debtor any
of the Junior Debt or exercise any of its rights or remedies upon a default or
event of default by Debtor under the Junior Creditor Agreements or otherwise or
(ii) commence any action or proceeding against Debtor or Debtor's properties
under the U.S. Bankruptcy Code or any state insolvency law or any similar
present or future statute, law or regulation or any proceedings for voluntary
liquidation, dissolution or other winding up of Debtor's business, or the
appointment of any trustee, receiver or liquidator for Debtor or any part of
Debtor's properties or any assignment for the benefit of creditors or any
marshalling of assets of Debtor or (iii) take any other action against Debtor
or Debtor's properties in respect of the Junior Debt;

                 (c)  Debtor shall not grant to Junior Creditor and Junior
Creditor shall not acquire any security interest, lien, claim or encumbrance on
any assets or properties of Debtor, and Junior Creditor shall not acquire any
guarantees or other agreements under which any person, other than Debtor, is or
may become obligated, directly or indirectly, for all or any portion of the
Junior Debt;

                 (d)  Junior Creditor and Debtor shall not amend, modify, alter
or change in any material respect the terms of any arrangements related to the
Junior Debt;

                 (e)  Junior Creditor shall not sell, assign, pledge, encumber
or otherwise dispose of any of the Junior Debt, or subordinate any of the
Junior Debt to any indebtedness of Debtors other than the Senior Debt, without
the prior written consent of Senior Creditor, which consent shall not be
unreasonably withheld in Senior Creditor's good faith determination;




                                     - 6 -
<PAGE>   7
                 (f)  Junior Creditor and Debtor shall, at any time or times
upon the request of Senior Creditor, promptly furnish to Senior Creditor a
true, correct and complete statement of the outstanding Junior Debt; and

                 (g)  Junior Creditor and Debtor shall execute and deliver to
Senior Creditor such additional agreements, documents and instruments and take
such further actions as may be reasonably necessary or desirable in the opinion
of Senior Creditor to effectuate the provisions and purposes of this
Subordination Agreement.

          3.2   Additional Representations and Warranties.  Junior Creditor and
Debtor represent and warrant to Senior Creditor that:

                 (a)  as of the date hereof, the total principal amount of the
Junior Debt outstanding is $5,000,000;

                 (b)  Junior Creditor has no security interest, lien, claim or
encumbrance on any assets and properties of Debtor and the Junior Debt is
unsecured;

                 (c)  as of the date hereof, no default or event of default, or
event which with notice or passage of time or both would constitute an event of
default, exists or has occurred under the Junior Creditor Agreements;

                 (d)  Junior Creditor is the exclusive legal and beneficial
owner of all of the Junior Debt;

                 (e)  none of the Junior Debt is subject to any lien, security
interest, financing statements, subordination, assignment or other claim,
except in favor of Senior Creditor; and

                 (f)  this Subordination Agreement constitutes the legal, valid
and binding obligations of Junior Creditor, enforceable in accordance with its
terms.

         3.3  Waivers.  Notice of acceptance hereof, the making of loans,
advances and extensions of credit or other financial accommodations to, and the
incurring of any expenses by or in respect of, Debtor or its subsidiaries by
Senior Creditor, and presentment, demand, protest, notice of protest, notice of
nonpayment or default and all other notices to which Junior Creditor and Debtor
are or may be entitled are hereby waived (except as expressly provided for
herein or as to Debtor, in the Senior Creditor Agreements).  Junior Creditor
also waives notice of, and hereby consents to, (a) subject to the terms and
conditions contained in this Subordination Agreement, any amendment,
modification, supplement, renewal, restatement or extensions of time of payment
of or increase or decrease in the amount of any of the Senior Debt or to the
Senior Creditor Agreements or any collateral at any time granted to or held by
Senior Creditor, (b) the taking, exchange, surrender and releasing of
collateral at any time granted to or held by Senior Creditor or guarantees now
or at any time held by or available to Senior Creditor for the



                                     - 7 -
<PAGE>   8
Senior Debt or any other person at any time liable for or in respect of the
Senior Debt, (c) the exercise of, or refraining from the exercise of any rights
against Debtor or any other obligor or any collateral at any time granted to or
held by Senior Creditor, (d) the settlement, compromise or release of, or the
waiver of any default with respect to, any of the Senior Debt, and/or (e) Senior
Creditor's election, in any proceeding instituted under the U.S. Bankruptcy Code
of the application of Section 1111(b)(2) of the U.S. Bankruptcy Code.  Any of
the foregoing shall not, in any manner, affect the terms hereof or impair the
obligations of Junior Creditor hereunder.  All of the Senior Debt shall be
deemed to have been made or incurred in reliance upon this Subordination
Agreement.

         3.4  Subrogation; Marshalling.  Junior Creditor shall not be subrogated
to, or be entitled to any assignment of any Senior Debt or Junior Debt or of any
collateral for or guarantees or evidence of any thereof until all of the Senior
Debt is indefeasibly paid and satisfied in full.  Junior Creditor hereby waives
any and all rights to have any collateral or any part thereof granted to or held
by Senior Creditor marshalled upon any foreclosure or other disposition of such
collateral by Senior Creditor or Debtor with the consent of Senior Creditor.
When the Senior Debt shall have been indefeasibly paid in full and discharged
and all the Senior Creditor Agreements have been terminated, the Junior Creditor
shall, to the extent permitted by law, be subrogated to the rights of the Senior
Creditor to receive payments or distribution of assets in respect of the Junior
Debt.

         3.5  No Offset.  In the event Junior Creditor at any time incurs any
obligation to pay money to Debtor, Junior Creditor hereby irrevocably agrees
that it shall pay such obligation in cash or cash equivalents in accordance
with the terms of the contract governing such obligation and shall not deduct
from or setoff against any amounts owed by Junior Creditor to Debtor in
connection with any such transaction any amounts the Junior Creditor claims are
due to it with respect to the Junior Debt.

         3.6  Certain Amendments to Senior Creditor Agreements. Senior
Creditor shall not make loans or advances to Debtors under the Loan Agreement
that would result in the Senior Debt to be greater than $90,000,000 outstanding
at any one time and Senior Creditor shall not extend the Renewal Date (as such
term is defined in the Loan Agreement) beyond six (6) months without the prior
written consent of Junior Creditor; provided, that, if the Junior Creditor
Agreements are terminated to the extent provided herein, this Section 3.6 shall
automatically and without further action by the parties hereto shall no longer
be deemed to apply and have no further force and effect.

         4.   MISCELLANEOUS

         4.1  Amendments.  Any waiver, permit, consent or approval by either
Creditor of or under any provision, condition or covenant to this Subordination
Agreement must be in writing and shall be effective only to the extent it is
set forth in writing and as to the specific facts or



                                     - 8 -
<PAGE>   9
circumstances covered thereby.  Any amendment of this Subordination Agreement
must be in writing and signed by each of the parties to be bound thereby.

         4.2  Successors and Assigns.

                 (a)  This Subordination Agreement shall be binding upon the
parties hereto and their respective successors and assigns and shall inure to
the benefit of each of Creditors and its respective successors, participants
and assigns.

                 (b)  Senior Creditor reserves the right to grant participations
in, or otherwise sell, assign, transfer or negotiate all or any part of, or any
interest in, the Senior Debt and the collateral securing same; provided, that,
Junior Creditor shall not be obligated to give any notices to or otherwise in
any manner deal directly with any participant in the Senior Debt and no
participant shall be entitled to any rights or benefits under this Subordination
Agreement except through Senior Creditor.  In connection with any participation
or other transfer or assignment, Senior Creditor (i) may disclose to such
assignee, participant or other transferee or assignee all documents and
information which Senior Creditor now or hereafter may have relating to the
Senior Debt or any collateral and (ii) shall disclose to such participant or
other transferee or assignee the existence and terms and conditions of this
Subordination Agreement.

                 (c)  In connection with any assignment or transfer of any or
all of the Senior Debt, or any or all rights of Senior Creditor in any of the
property of Debtor or its subsidiaries (other than pursuant to a
participation), Junior Creditor agrees to execute and deliver an agreement
containing terms substantially identical to those contained herein in favor of
any such assignee or transferee and, in addition, will execute and deliver an
agreement containing terms substantially identical to those contained herein in
favor of any third person who succeeds to or replaces any or all of Senior
Creditor's financing of certain subsidiaries of Debtor, whether such successor
financing or replacement occurs by transfer, assignment, "takeout" or any other
means.

         4.3  Insolvency.  This Subordination Agreement shall be applicable
both before and after the filing of any petition by or against Debtor or any of
its subsidiaries under the U.S. Bankruptcy Code and all converted or succeeding
cases in respect thereof, and all references herein to Debtor or any of
Debtor's subsidiaries shall be deemed to apply to a trustee for Debtor or any
of its subsidiaries, as well as to Debtor or any of its subsidiaries as
debtor-in-possession.  The relative rights of Senior Creditor and Junior
Creditor to repayment of the Senior Debt and the Junior Debt, respectively, and
in or to any distributions from or in respect of Debtor or any proceeds of
Debtor's property and assets, shall continue after the filing thereof on the
same basis as prior to the date of the petition, subject to any court order
approving the financing of, or use of cash collateral by, Debtor or any of its
subsidiaries as debtor-in-possession.

         4.4  Bankruptcy Financing.  If Debtor or any of its subsidiaries
shall become subject to a proceeding under the U.S. Bankruptcy Code and if
Senior Creditor desires to permit the use of cash collateral or to provide
financing to Debtor or any of its subsidiaries under either Section




                                     - 9 -
<PAGE>   10
363 or Section 364 of the U.S. Bankruptcy Code, Junior Creditor agrees as
follows: (a) adequate notice to Junior Creditor (if required) shall have been
provided for such financing or use of cash collateral if Junior Creditor
receives notice two (2) business days prior to the entry of the order approving
such financing or use of cash collateral and (b) no objection will be raised by
Junior Creditor to any such use of cash collateral or financing.  For purposes
of this Section, notice of a proposed financing or use of cash collateral shall
be deemed given when given in the manner prescribed by Section 4.5 hereof to
Junior Creditor.

         4.5  Notices.  All notices, requests and demands to or upon the
respective parties hereto shall be in writing and shall be deemed to have been
duly given or made:  if delivered in person, immediately upon delivery; if by
telex, telegram or facsimile transmission, immediately upon sending and upon
confirmation of receipt; if by nationally recognized overnight courier service
with instructions to deliver the next business day, one (1) business day after
sending; and if mailed by certified mail, return receipt requested, five (5)
days after mailing.  All notices, requests and demands are to be given or made
to the respective parties at their addresses set forth below (or to such other
addresses as either party may designate by notice in accordance with the
provisions of this Section:

To Senior Creditor:                     Congress Financial Corporation
                                        1133 Avenue of the Americas
                                        New York, New York 10036
                                        Attention:  Mr. Laurence S. Forte
                                        Telecopier: 212-545-4283
                                        Phone: 212-545-4280

To Junior Creditor:                     Richemont Finance S.A.
                                        35 Boulevard Prince Henri
                                        L 1724 Luxembourg
                                        Attention:  Mr. J. Alan Grieve
                                        Telecopier: 011-4141-711-7138
                                        Phone: 011-4141-710-3322

       with a copy to:                  Robert P. Wessely, Esq.
                                        Dorsey & Whitney
                                        250 Park Avenue
                                        New York, New York 10036
                                        Telecopier: (212) 953-7201
                                        Phone: (212) 415-9200


Either Creditor may change the address(es) to which all notices, requests and
other communications are to be sent by giving written notice of such address
change to the other Creditor in conformity with this Section 4.5, but such
change shall not be effective until notice of such change has been received by
the other Creditor.




                                     - 10 -
<PAGE>   11
         4.6  Counterparts.  This Subordination Agreement may be executed in
any number of counterparts, each of which shall be an original with the same
force and effect as if the signatures thereto and hereto were upon the same
instrument.

         4.7  Governing Law.  The validity, construction and effect of this
Subordination Agreement shall be governed by the laws of the State of New York
(without giving effect to principles of conflicts of laws).

         4.8  Consent to Jurisdiction; Waiver of Jury Trial.  Each of the
parties hereto hereby irrevocably consents to the non-exclusive jurisdiction of
the Supreme Court of the State of New York for New York County and the United
States District Court for the Southern District of New York and waives trial by
jury in any action or proceeding with respect to this Subordination Agreement.

         4.9  Complete Agreement.

                 (a)  This written Subordination Agreement is intended by the
parties as a final expression of their agreement and is intended as a complete
statement of the terms and conditions of their agreement.

                 (b)  The obligations of Junior Creditor under this
Subordination Agreement are in addition to, and in no way limited by the terms
of the Subordination Agreement, dated as of the date hereof, between Junior
Creditor and Senior Creditor, as acknowledged by Debtor and certain direct and
indirect subsidiaries of Debtor in respect of the Richemont $10,000,000 Credit
Agreements as such term is defined in the Loan Agreement, nor shall any of the
terms thereof be limited or affected by the terms of this Subordination
Agreement.

                 (c)  The obligations of Junior Creditor under this
Subordination Agreement are in addition to, and in no way limited by the terms
of any other subordination agreement, heretofore entered into between Junior
Creditor and Senior Creditor, as acknowledged by Debtor and/or certain of its
subsidiaries and affiliates, nor shall any of the terms of any such
subordination agreement be limited or affected by the terms of this
Subordination Agreement.

         4.10  No Third Parties Benefitted.  This Subordination Agreement is
solely for the benefit of the Creditors and their respective successors,
participants and assigns, and no other person shall have any right, benefit,
priority or interest under, or because of the existence of, this Subordination
Agreement.

         4.11  Disclosures, Non-Reliance.  Each Creditor has the means to, and
shall in the future remain, fully informed as to the financial condition and
other affairs of Debtor and neither Creditor shall have any obligation or duty
to disclose any such information to any other Creditor.  Except as expressly
set forth in this Subordination Agreement, the parties hereto have not
otherwise made to each other nor do they hereby make to each other any
warranties, express or implied, nor do they assume any liability to each other
with respect to:  (a) the enforceability, validity, value or collectability of
any of the Junior Debt or the Senior Debt or any collateral or guarantee which
may have been granted to any of them in connection therewith, (b) Debtor's
title



                                     - 11 -
<PAGE>   12
to or right to any of Debtor's assets and properties or (c) any other
matter except as expressly set forth in this Subordination Agreement.

         4.12  Term.  This Subordination Agreement is a continuing agreement
and shall remain in full force and effect until the indefeasible satisfaction
in full of all Senior Debt and the termination of the financing arrangements
among Senior Creditor, Debtor and certain subsidiaries of Debtor.



               [REMAINDER OF THIS PAGE INTENTIONALLY LEFT BLANK]





                                     - 12 -
<PAGE>   13
         IN WITNESS WHEREOF, the parties have caused this Subordination
Agreement to be duly executed as of the day and year first above written.

                                CONGRESS FINANCIAL CORPORATION

                                By: /s/ Janet Last
                                   --------------------------

                                Title: Vice President
                                      -----------------------

                                RICHEMONT FINANCE S.A.

                                By: /s/ Jan P. du Plessis
                                   --------------------------

                                Title: Director
                                      -----------------------

                                By: /s/ Alan Grieve
                                   --------------------------

                                Title: Director
                                      -----------------------






                                     - 13 -
<PAGE>   14
                                 ACKNOWLEDGMENT


         The undersigned hereby acknowledges and agrees to the foregoing terms
and provisions.  By its signature below, the undersigned agrees that it shall,
together with its successors and assigns, be bound by the provisions hereof.

         The undersigned acknowledges and agrees that: (i) although it may sign
this Subordination Agreement, it is not a party hereto and does not and shall
not receive any right, benefit, priority or interest under or because of the
existence of the foregoing Subordination Agreement, (ii) in the event of a
breach by the undersigned of any of the terms and provisions contained in the
foregoing Subordination Agreement, such a breach shall constitute an "Event of
Default" as defined in and under the Senior Creditor Agreements, and (iii) it
shall execute and deliver such additional documents and take such additional
action as may be necessary in the opinion of either Creditor to effectuate the
provisions and purposes of the foregoing Subordination Agreement.

                                HANOVER DIRECT, INC.

                                By: /s/ Brian C. Harriss
                                   --------------------------

                                Title: SVP & CFO
                                      -----------------------







                                     - 14 -