<PAGE>   1
                                                                   EXHIBIT 10.56


                   SECOND SUPPLEMENTAL SERIES B NOTE AGREEMENT

         This SECOND SUPPLEMENTAL SERIES B NOTE AGREEMENT dated as of December
18, 1996, between HANOVER DIRECT, INC., a Delaware corporation (the "Borrower"),
and NORWEST BANK MINNESOTA, N.A., a national banking association organized under
the laws of the United States of America and having its principal office in
Minneapolis, Minnesota, as trustee and paying agent (the "Trustee" or the
"Paying Agent", as applicable);


                              W I T N E S S E T H :

         WHEREAS, the Borrower and the Trustee entered into the Series B Note
Agreement dated as of April 27, 1995 (the "Series B Note Agreement") pursuant to
which the Borrower issued and sold its interest bearing Flexible Term Notes,
Series B (the "Series B Notes") in the aggregate principal amount of
$10,000,000; and

         WHEREAS, the Series B Note Agreement was modified on December 29, 1995
by the First Supplemental Series B Note Agreement by and between Borrower and
Trustee (the "First Supplemental Series B Note Agreement") to reflect the
delivery of a substitute Letter of Credit;

         WHEREAS, the Borrower has this day, delivered to the Trustee a
Substitute Series B Letter of Credit in substitution for the Series B Letter of
Credit (each as defined in the Series B Note Agreement); and

         WHEREAS, in order to more fully evidence the delivery of the Substitute
Series B of Letter of Credit referenced above, the Borrower and the Trustee
desire to further amend the Series B Note Agreement, subject to the terms and
conditions set forth herein.

         NOW, THEREFORE, the parties hereto agree as follows:

         SECTION 1. DEFINITIONS. All capitalized terms used in this First
Supplemental Series B Note Agreement and not otherwise herein defined shall have
the meaning ascribed to them in the Series B Note Agreement.

         SECTION 2. AMENDMENT TO SECOND RECITAL OF THE SERIES B NOTE AGREEMENT.
The second recital of the Series B Note Agreement on page 1 thereof is hereby
amended by deleting the reference to the "L/C Issuer" in the third line thereof
and replacing it with a reference to the "Bank".
<PAGE>   2
         SECTION 3. AMENDMENTS TO SECTION 1.01 OF THE SERIES B NOTE AGREEMENT.

         (a)      The definition of "BANK" set forth in Section 1.01 of the
Series B Note Agreement is hereby deleted in its entirety and replaced with the
following:

                           "BANK" means the issuer of the Series B Letter of
                  Credit, initially NationsBank of North Carolina, N.A.
                  (predecessor to NationsBank, N.A.) and upon the issuance and
                  delivery of a Substitute Series B Letter of Credit, shall mean
                  the issuer of such Substitute Series B Letter of Credit.

         (b)      The definition of "BANK NOTES" set forth in Section 1.01 of
the Series B Note Agreement is hereby deleted in its entirety and replaced with
the following:

                           "BANK NOTES" means any Series B Notes purchased with
                  proceeds from a draw under the Series B Letter of Credit and
                  pledged to the Bank under the Reimbursement Agreement,
                  including, in the event a book-entry system with respect to
                  the Series B Notes is in effect, any beneficial ownership
                  interest therein; provided that in the event that the Bank
                  fails to honor a drawing under the Series B Letter of Credit
                  to fund such a purchase and the Borrower purchases such Series
                  B Notes with its own funds, "Bank Notes" shall include such
                  Series B Notes, except that such Series B Notes shall not be
                  pledged to the Bank under the Reimbursement Agreement.

         (c)      The definition of "BUSINESS DAY" set forth in Section 1.01 of
the Series B Note Agreement is amended by deleting the last sentence thereof in
its entirety and replacing it with the following:

                  For purposes of this definition, "paying office of the Bank"
                  means the Bank office responsible for making payments under
                  any Series B Letter of Credit.

         (d)      The definition of "OPINION OF COUNSEL" set forth in Section
1.01 of the Series B Note Agreement is hereby deleted in its entirety and
replaced with the following:

                           "OPINION OF COUNSEL" means a written opinion of
                  counsel who is reasonably acceptable to the Trustee, the Bank,
                  the Placement Agent and Remarketing Agent. The counsel may be
                  an employee of or counsel to the Borrower, the Placement
                  Agent, the Remarketing Agent, the Bank or the Trustee.


                                        2
<PAGE>   3
         (e)      The definition of "REIMBURSEMENT AGREEMENT" set forth in
Section 1.01 of the Series B Note Agreement is hereby deleted in its entirety
and replaced with the following:

                           "REIMBURSEMENT AGREEMENT" means, individually and
                  collectively, the Reimbursement Agreement dated as of December
                  18, 1996, by and among the Borrower and certain subsidiaries
                  of the Borrower, and the Bank, pursuant to which, among other
                  things, the Bank has issued the Series B Letter of Credit, and
                  any and all modifications, alterations, amendments and
                  supplements thereto, and any similar agreements between or
                  among the Borrower and the issuer of a Substitute Series B
                  Letter of Credit or Lender providing credit support to such
                  issuer.

         (f)      The definition of "SERIES B LETTER OF CREDIT" set forth in
Section 1.01 of the Series B Note Agreement is amended by (a) deleting the words
"Commonwealth of Pennsylvania" in the third line thereof and replacing them with
"State of New York" and (b) by deleting the word "L/C Issuer" in the fourteenth
line thereof and replacing it with "Bank".

         (g)      Section 1.01 of the Series B Note Agreement is hereby amended
by deleting the following definitions thereto:

                           "ACCOUNT PARTIES" means Hanover Direct Pennsylvania,
                  Inc., Hanover Direct Virginia Inc. and Gump's Corp., and their
                  successors and assigns.

                           "L/C ISSUER" means the issuer of the Series B Letter
                  of Credit, initially CoreStates Bank, N.A., and upon the
                  issuance and delivery of a Substitute Series B Letter of
                  Credit, shall mean the issuer of such Substitute Series B
                  Letter of Credit.

                           "LENDER" means Congress Financial Corporation, its
                  successors and assigns, or other lender that refinances the
                  obligations to Lender under the Reimbursement Agreement and
                  replaces all credit support given by Lender to the L/C Issuer
                  in respect of the Series B Letter of Credit or any Substitute
                  Series B Letter of Credit.

         SECTION 4. AMENDMENT TO SECTION 2.07 OF THE SERIES B NOTE AGREEMENT.
Subsection (b) of Section 2.07 of the Series B Note Agreement is amended by
deleting the third sentence thereof in its entirety and replacing it with the
following:

                  Subject to the provisions of Section 7.09, the Borrower, the
                  Trustee and the Paying Agent will recognize the Securities
                  Depository Nominee, while the


                                        3
<PAGE>   4
                  registered owner of the Series B Notes so held, as the owner
                  of the Series B Notes for all purposes, including (i) payments
                  of principal and Purchase Price of, and interest on, the
                  Series B Notes, (ii) notices and (iii) voting.

         SECTION 5. AMENDMENT TO SECTION 2.08 OF THE SERIES B NOTE AGREEMENT.
Subsection (a) of Section 2.08 of the Series B Note Agreement is amended by
deleting the reference to the "Lender" in the sixth line thereof and replacing
it with a reference to the "Bank".

         SECTION 6. AMENDMENT TO SECTION 3.01 OF THE SERIES B NOTE AGREEMENT.
Subsection (a)(ii) of Section 3.01 of the Series B Note Agreement is amended by
deleting the reference to the "Lender" in the second line thereof and replacing
it with a reference to the "Bank". It is acknowledged that the purported
amendment to Subsection (c) of Section 3.10 of the Series B Note Agreement
pursuant to the First Supplemental Series B Note Agreement was a scrivener's
error and is of no force and effect.

         SECTION 7. AMENDMENT TO SECTION 3.04 OF THE SERIES B NOTE AGREEMENT.
Section 3.04 of the Series B Note Agreement is amended by deleting the reference
to the "Lender" in the third line thereof and replacing it with a reference to
the "Bank".

         SECTION 8. AMENDMENT TO SECTION 3.08 OF THE SERIES B NOTE AGREEMENT.
Subsection (d)(ii) of Section 3.08 of the Series B Note Agreement is hereby
deleted in its entirety and replaced with the following:

                           (ii) All Bank Notes (other than Bank Notes purchased
                  with the Borrower's own funds and not with the proceeds of a
                  draw on the Series B Letter of Credit) will be registered in
                  the name of the Trustee, as agent and bailee of the Bank and
                  subject to the pledge by the Borrower to the Bank, and shall
                  be held by the Trustee pursuant to this Agreement and the
                  Reimbursement Agreement. Upon receipt of Remarketing Proceeds
                  in respect of Bank Notes, the Remarketing Agent shall notify,
                  the Bank, the Trustee and the Borrower of such receipt. Upon
                  its receipt of such notice, the Bank shall, pursuant to the
                  Reimbursement Agreement, notify the Remarketing Agent and the
                  Trustee by telephone, telecopy or telex, promptly confirmed in
                  writing, that the Series B Notes have ceased to be Bank Notes
                  and that the amount of the Letter of Credit has been
                  automatically reinstated as provided therein, whereupon the
                  Remarketing Agent will remit such Remarketing Proceeds as
                  directed by the Bank. The Trustee shall not release the Bank
                  Notes until it receives from the Bank the notice referred to
                  in the


                                        4
<PAGE>   5
                  preceding sentence. The Remarketing Agent shall hold such
                  Remarketing Proceeds in a segregated account of the
                  Remarketing Agent for the benefit of the Bank, except that if
                  the Series B Letter of Credit is not reinstated by an amount
                  equal to the Remarketing Proceeds, then the Remarketing Agent
                  shall hold such funds for the benefit of the purchasers which
                  provided such Remarketing Proceeds.

         SECTION 9. AMENDMENTS TO SECTION 4.03 OF THE SERIES B NOTE AGREEMENT.

         (a) Subsection (c) of Section 4.03 of the Series B Note Agreement is
amended by deleting all references to the "Lender" therein and replacing them
with references to the "Bank".

         (b) Subsection (e) of Section 4.03 of the Series B Note Agreement is
amended by deleting the reference to the "Lender" in the last line thereof and
replacing it with a reference to the "Bank".

         SECTION 10. AMENDMENT TO SECTION 5.02 OF THE SERIES B NOTE AGREEMENT.
Subsection (c) of Section 5.02 of the Series B Note Agreement is amended by
deleting all references therein to the "Lender" and replacing them with
references to the "Bank".

         SECTION 11. AMENDMENT TO SECTION 5.03 OF THE SERIES B NOTE AGREEMENT.
Subsection (b) of Section 5.03 of the Series B Note Agreement is amended by
deleting the reference to the "Lender" in the last line thereof and replacing it
with a reference to the "Bank".

         SECTION 12. AMENDMENT TO SECTION 5.04 OF THE SERIES B NOTE AGREEMENT.
Section 5.04 of the Series B Note Agreement is amended by deleting the second
and third paragraphs thereof in their entirety and replacing them with the
following:

                  When the Series B Letter of Credit terminates or expires in
         accordance with its terms or a Substitute Series B Letter of Credit
         therefor is accepted hereunder, the Trustee shall immediately surrender
         the Series B Letter of Credit to the Bank. The Trustee hereby agrees
         that, except in the case of a redemption in part pursuant to Article
         III hereof or any other reduction in the principal amount of Series B
         Notes Outstanding, it will not under any circumstances request that the
         Bank reduce the amount of the Series B Letter of Credit. If at any
         time, all Series B Notes shall cease to be Outstanding, the Trustee
         shall surrender the Series B Letter of Credit to the Bank, in
         accordance with the terms thereof.


                                        5
<PAGE>   6
                  If at any time, the Bank shall fail to honor a draft presented
         under the Series B Letter of Credit, in conformity with the terms
         thereof, the Trustee shall give immediate telephonic notice thereof to
         the Remarketing Agent and the Borrower.

         SECTION 13. AMENDMENT TO SECTION 7.01 OF THE SERIES B NOTE AGREEMENT.
Subsection (d) of Section 7.01 of the Series B Note Agreement is hereby deleted
in its entirety and replaced with the following:

                  (d) Receipt by the Trustee of written notice from the Bank
                  that an Event of Default has occurred under the Reimbursement
                  Agreement accompanied by a demand by the Bank that the Trustee
                  declare the Series B Notes to be immediately due and payable.

         SECTION 14. AMENDMENTS TO SECTION 7.02 OF THE SERIES B NOTE AGREEMENT.

         (a) Subsection (a) of Section 7.02 of the Series B Note Agreement is
amended by deleting the reference to the "Lender" in the second line thereof and
replacing it with a reference to the "Bank".

         (b) Subsection (b) of Section 7.02 of the Series B Note Agreement is
hereby deleted in its entirety and replaced with the following:

                           (b) Upon the occurrence of any Event of Default
                  specified in Section 7.01(c), the Trustee shall notify the
                  Bank of such Event of Default and shall, by notice to the
                  Borrower, the Paying Agent (who shall promptly give such
                  notice to the holders) and the Remarketing Agent declare the
                  entire unpaid principal of and interest on the Series B Notes
                  immediately due and payable, but only if directed to do so by
                  the Bank, unless the Bank has dishonored a valid draw under
                  the Series B Letter of Credit, in which event the Trustee may
                  declare the entire unpaid principal of and interest on the
                  Series B Notes immediately due and payable and, thereupon, in
                  either case, the entire unpaid principal of and interest on
                  the Series B Notes shall forthwith become due and payable.

         SECTION 15. AMENDMENT TO SECTION 7.03 OF THE SERIES B NOTE AGREEMENT.
Section 7.03 of the Series B Note Agreement is amended (a) by deleting the
reference to the "Lender" in the first sentence thereof and replacing it with a
reference to the "Bank" and (b) by deleting the second sentence thereof in its
entirety and replacing it with the following:


                                        6
<PAGE>   7
                  If such instructions are received by, the Trustee, such draw
                  proceeds and, if necessary, the moneys on deposit in the
                  Interest Reserve Account, shall be immediately applied to the
                  purchase of the Series B Notes, the acceleration of the Series
                  B Notes shall be cancelled, the Series B Notes shall become
                  Bank Notes and the Series B Notes shall be registered in the
                  name of the Trustee, as agent and bailee of the Bank, and
                  pledged under the Reimbursement Agreement as additional
                  security for repayment of the Borrower's obligations under the
                  Reimbursement Agreement.

         SECTION 16. AMENDMENT TO SECTION 7.05 OF THE SERIES B NOTE AGREEMENT.
Section 7.05 of the Series B Note Agreement is amended by deleting the phrase
"and the Account Parties" between the words "Borrower" and "to" in the seventh
line thereof.

         SECTION 17. AMENDMENT TO SECTION 7.06 OF THE SERIES B NOTE AGREEMENT.
Section 7.06 of the Series B Note Agreement is amended (a) by deleting the
references to the "Lender" in the second and fourth lines thereof and replacing
therein with references to the "Bank" and (b) by deleting the reference to the
"Lender" in the second line of subsection (b) thereof and replacing it with a
reference to the "Bank".

         SECTION 18. AMENDMENT TO SECTION 7.09 OF THE SERIES B NOTE AGREEMENT.
It is acknowledged that reference to Section 7.03 in the first line of Section
19 of the First Supplemental Series B Note Agreement entitled "Amendment to
Section 7.09 of the Series B Note Agreement" was a scrivener's error and that
the correct reference should have been to Section 7.09. Section 7.09 of the
Series B Note Agreement is deleted in its entirety and replaced with the
following:

                  SECTION 7.09 BANK DEEMED HOLDER. For all purposes of this
         Article VII (other than receipt of payments), the Bank shall, so long
         as the Series B Letter of Credit shall not have been dishonored (other
         than for failure to receive a drawing in strict compliance with the
         terms thereof or other reason permitted by the Series B Letter of
         Credit), be deemed the holder and registered owner of all Series B
         Notes. As such, the Bank may take all actions permitted by this Article
         VII to be taken by the holders or registered owners of the Series B
         Notes, to the exclusion of the actual beneficial owners and registered
         owners of the Series B Notes. NOTWITHSTANDING ANY PROVISION TO THE
         CONTRARY HEREIN, ON OR AFTER THE EFFECTIVE DATE OF A SUBSTITUTE SERIES
         B LETTER OF CREDIT WHICH RESULTS IN A CREDIT MODIFICATION, THE BANK, AS
         ISSUER OF THE SERIES B LETTER OF CREDIT REPLACED BY SUCH SUBSTITUTE
         SERIES B LETTER OF CREDIT, SHALL BE DEEMED THE "BANK" HEREUNDER FOR
         PURPOSES OF GIVING NOTICE OF DEFAULT UNDER SECTION 7.01(D) AND FOR


                                        7
<PAGE>   8
         PURPOSES OF EXERCISING REMEDIES HEREUNDER, BUT ONLY SO LONG AS (I)
         OBLIGATIONS REMAIN OWING TO THE BANK UNDER THE REIMBURSEMENT AGREEMENT
         OR THE LOAN DOCUMENTS (AS DEFINED IN THE REIMBURSEMENT AGREEMENT) OR
         (II) THERE REMAIN OUTSTANDING HEREUNDER ANY BANK NOTES PLEDGED TO THE
         BANK UNDER THE REIMBURSEMENT AGREEMENT.

         SECTION 19. AMENDMENT TO SECTION 8.05 OF THE SERIES B NOTE AGREEMENT.
Section 8.05 of the Series B Notes Agreement is amended by deleting the
references to the "Lender" in the sixth and tenth lines thereof and replacing
therein with references to the "Bank".

         SECTION 20. AMENDMENT TO SECTION 8.08 OF THE SERIES B NOTE AGREEMENT.
Section 8.08 of the Series B Note Agreement is amended (a) by deleting all
references therein to the "Lender" or "Lenders" and replacing them with
references to the "Bank" or "Bank's", as applicable, and (b) by deleting the
following sentence after the last paragraph thereof:

                  No resignation or removal of the Trustee shall be binding upon
         the L/C Issuer for purposes of the Series B Letter of Credit, and no
         successor Trustee shall have any rights to draw, except upon compliance
         with the transfer provisions therein set forth.

         SECTION 21. AMENDMENT TO SECTION 8.10 OF THE SERIES B NOTE AGREEMENT.
Section 8.10 of the Series B Note Agreement is amended (a) by deleting all
references therein to the "Lender" and replacing them with references to the
"Bank" and (b) by deleting the following sentence after the last paragraph
thereof:

                           No such resignation or removal of the Remarketing
                  Agent or appointment of or assignment to a successor
                  Remarketing Agent shall be binding upon the L/C Issuer for
                  purposes of the Series B Letter of Credit, unless set forth in
                  an amendment to the Series B Letter of Credit issued by the
                  L/C Issuer.

         SECTION 22. AMENDMENT TO SECTION 8.11 OF THE SERIES B NOTE AGREEMENT.
Subsection (c) of Section 8.11 of the Series B Note Agreement is amended by
deleting the reference to the "Lender" in the fourth line thereof and replacing
it with a reference to the "Bank".

         SECTION 23. AMENDMENT TO SECTION 8.12 OF THE SERIES B NOTE AGREEMENT.
Section 8.12 of the Series B Note Agreement is amended by deleting all
references to the "Lender" therein and replacing them with references to the
"Bank".

         SECTION 24. AMENDMENT TO SECTION 8.13 OF THE SERIES B NOTE AGREEMENT.
Section 8.13 of the Series B Note Agreement is


                                        8
<PAGE>   9
amended by deleting all references to the "Lender" therein and replacing them
with references to the "Bank".

         SECTION 25. AMENDMENT TO SECTION 8.14 OF THE SERIES B NOTE AGREEMENT.
Section 8.14 of the Series B Note Agreement is amended by deleting all
references to the "Lender" or "Lender's" therein and replacing therein with
references to the "Bank" or "Bank's", as applicable.

         SECTION 26. AMENDMENT TO SECTION 8.16 OF THE SERIES B NOTE AGREEMENT.
Section 8.16 of the Series B Note Agreement is amended (a) by deleting the
phrase ", to the extent provided in the Reimbursement Agreement," after the word
"shall" at the end of the fourth line thereof and (b) by deleting the reference
to the "Lender" in the fifth line thereof and replacing it with a reference to
the "Bank".

         SECTION 27. AMENDMENT TO SECTION 9. 01 OF THE SERIES B NOTE AGREEMENT.
Subsection (f) of Section 9.01 of the Series B Note Agreement is amended by
deleting the reference to the "Lender" in the fifth line thereof and replacing
it with a reference to the "Bank".

         SECTION 28. AMENDMENT TO SECTION 9.02 OF THE SERIES B NOTE AGREEMENT.
Section 9.02 of the Series B Note Agreement is amended by deleting the reference
to the "Lender" in the fourteenth line thereof and replacing it with a reference
to the "Bank".

         SECTION 29. AMENDMENT TO SECTION 9.06 OF THE SERIES B NOTE AGREEMENT.
The heading and the first sentence of Section 9.06 of the Series B Note
Agreement are hereby deleted in their entirety and replaced with the following:

                           Section 9.06. BANK AND REMARKETING AGENT CONSENT
                  REQUIRED. Except to the extent that the consent of the
                  Remarketing Agent or the Bank is not required for the action
                  that is the subject of the amendment (e.g., removal of the
                  Remarketing Agent, the Trustee or the Paying Agent by the
                  Borrower upon the terms specified herein), an amendment to
                  this Agreement or the Series B Notes shall not become
                  effective unless the Remarketing Agent (but only to the extent
                  that such amendment affects the rights, duties or obligations
                  of the Remarketing Agent hereunder) and the Lender deliver to
                  the Trustee their written consents to the amendment.

         SECTION 30. AMENDMENT TO SECTION 10.01 OF THE SERIES B NOTE AGREEMENT.
Subsection (b) of Section 10.01 of the Series B Note Agreement is amended by
deleting the reference to, and the address of, the "L/C Issuer" and the "Lender"
therein and replacing them with the following:


                                        9
<PAGE>   10
         If to the Bank:                  Swiss Bank Corporation
                                          New York Branch
                                          222 Broadway
                                          New York, New York, 10038
                                          Attention:
                                          Telephone No.:
                                          Fax No.:

         SECTION 31. AMENDMENT TO SECTION 10.09 OF THE SERIES B NOTE AGREEMENT.
Section 10.09 of the Series B Note Agreement is amended by deleting the
reference to the "Lender" in the fourth line thereof and replacing it with a
reference to the "Bank".

         SECTION 32. AMENDMENTS TO EXHIBIT A TO THE SERIES B NOTE AGREEMENT -
FORM OF SERIES B NOTE. Exhibit A to the Series B Note Agreement (the "Form of
Series B Note") is hereby amended as follows:

         (a)      The second boldface paragraph on the first page of the Form of
Series B Note (prior to the text thereof) is hereby deleted in its entirety and
replaced with the following:

                  THIS SERIES B NOTE IS NOT A DEPOSIT OR OBLIGATION OF, OR
                  GUARANTEED BY, SWISS BANK CORPORATION, NEW YORK BRANCH (THE
                  "BANK"), IS NOT INSURED BY THE FEDERAL DEPOSIT INSURANCE
                  CORPORATION, AND IS SUBJECT TO INVESTMENT RISKS, INCLUDING
                  POSSIBLE LOSS OF THE PRINCIPAL AMOUNT INVESTED. ALTHOUGH NOT
                  GUARANTEED BY THE BANK, PAYMENTS OF PRINCIPAL AND INTEREST ON
                  THIS SERIES B NOTE AND, IF REMARKETING PROCEEDS ARE NOT
                  AVAILABLE, THE PURCHASE PRICE OF THIS SERIES B NOTE, WILL BE
                  MADE FROM DRAWINGS UNDER THE SERIES B LETTER OF CREDIT ISSUED
                  BY THE BANK. THE FAILURE OF THE BANK TO HONOR ANY DRAWING
                  UNDER THE SERIES B LETTER OF CREDIT WILL NOT GIVE RISE TO ANY
                  CLAIM OTHER THAN AGAINST THE BANK.

         (b)      The second paragraph of the text of the Form of Series B Note,
beginning on page A-2 thereof, is amended by deleting the first sentence thereof
and replacing it with the following:

                  This Series B Note is one of an issue not to exceed
                  $10,000,000 Hanover Direct, Inc. Flexible Term Notes, Series B
                  (the "Series B Notes"), issued pursuant to a Series B Note
                  Agreement dated as of April 27, 1995 (the "Series B Note
                  Agreement"), as amended by the First Supplemental Series B
                  Note Agreement dated December 29, 1995 and the Second
                  Supplemental Note Agreement dated December 18, 1996 between
                  the Borrower and Norwest Bank Minnesota, N.A., as trustee (in
                  such capacity, the "Trustee") and Paying Agent, for the
                  purpose of refinancing and/or financing certain construction,


                                       10
<PAGE>   11
                  refurbishment and related costs of an approximately 530,000
                  square foot distribution facility of the Borrower located in
                  Roanoke, Virginia and a new retail store of Gump's, Inc., a
                  subsidiary of the Borrower located in San Francisco,
                  California. Pursuant to the Series B Note Agreement, the
                  Borrower has caused Swiss Bank Corporation, New York Branch
                  (the "Bank") to issue its irrevocable Series B Letter of
                  Credit dated the Date of Issuance (as hereinafter defined and
                  as set forth above) of the Series B Notes (the "Series B
                  Letter of Credit") in favor of the Trustee, in an amount
                  sufficient to pay the Series B Facility Amount and unpaid
                  interest on or Purchase Price of the Series B Notes, but not
                  to exceed $9,638,541, pursuant to a Reimbursement Agreement
                  dated as of December 18, 1996 (the "Reimbursement Agreement")
                  by and among the Borrower and the Bank, which Series B Letter
                  of Credit initially expires (subject to extension or earlier
                  termination as provided in the Reimbursement Agreement and the
                  Series B Note Agreement) on February 18, 1998. Substitute
                  letters of credit may be delivered in accordance with the
                  Series B Note Agreement.

         (c)      The seventh paragraph of the text of the Form of Series B
Note, beginning on page A-4 thereof, is amended by deleting the third sentence
thereof in its entirety and replacing it with the following:

                  Subject to the provisions of Section 7.09 of the Series B Note
                  Agreement relating to the Bank as holder of the Series B
                  Notes, the Borrower, the Trustee and the Paying Agent will
                  recognize the Securities Depository Nominee, as hereinafter
                  defined, while the registered owner of the Series B Notes so
                  held, as the owner of the Series B Notes for all purposes,
                  including (i) payments of principal and Purchase Price of, and
                  interest on, the Series B Notes, (ii) notices and (iii)
                  voting, subject to certain qualifications as stated in the
                  Series B Note Agreement.

         (d)      Section 1 of the Form of Series B Note, beginning on page A-5
thereof, is amended by deleting the following definition:

                  "BANK" means, individually and collectively, the Lender and
                  the L/C Issuer.

         (e)      Subsection (f) of Section 3 of the Form of Series B Note,
beginning on page A-9 thereof, is amended by deleting the reference to the
"Lender" in the third line thereof and replacing it with a reference to the
"Bank."


                                       11
<PAGE>   12
         (f)      Subsection (a) of Section 4 of the Form of Series B Note,
beginning on page A-10 thereof, is amended by deleting the reference to the
"Lender" in the fourth line of the last paragraph thereof and replacing it with
a reference to the "Bank."

         (g)      Section 7 of the Form of Series B Note, beginning on page A-13
thereof, is amended by deleting the second sentence of the first paragraph
thereof in its entirety and replacing them with the following:

                  The Series B Note Agreement directs the Trustee to declare an
                  acceleration upon written notice by the Bank of the occurrence
                  and continuance of an event of default under the Reimbursement
                  Agreement and upon the occurrence of certain other Events of
                  Default under the Series B Note Agreement. The Trustee has the
                  right to accelerate the entire unpaid principal of and
                  interest on the Series B Notes in certain events only with the
                  Bank's consent, all as provided in more detail in Article VII
                  of the Series B Note Agreement to which reference is hereby
                  made.

         SECTION 33. DELETION OF EXHIBIT C TO SERIES B NOTE AGREEMENT. The
Series B Note Agreement is amended by deleting Exhibit C thereto.

         SECTION 34. EFFECT OF SECOND SUPPLEMENTAL SERIES B NOTE AGREEMENT.
Except as modified hereby, all of the terms and provisions of the Series B Note
Agreement shall remain in full force and effect.

         SECTION 35. GOVERNING LAW. This Second Supplemental Series B Note
Agreement and the Series B Note Agreement, as amended hereby, shall be deemed to
be contracts made under, and for all purposes shall be construed in accordance
with, the laws of the State of New York.

         SECTION 36. SEVERABILITY. If any provision of this Second Supplemental
Series B Note Agreement shall be determined to be unenforceable by a court of
law, that shall not affect any other provision of this Second Supplemental
Series B Note Agreement.

         SECTION 38. COUNTERPARTS. This Agreement may be signed in several
counterparts, each of which will be an original and all of which together will
constitute the same instrument.


                         [Signatures on following page.]


                                       12
<PAGE>   13
         IN WITNESS WHEREOF, the parties hereto have caused this Second
Supplemental Series B Note Agreement to be duly executed as of the day and year
first above written.

                                             HANOVER DIRECT, INC.


                                             By: /s/ Edward J. O'Brien
                                                --------------------------------
                                             Name: Edward J. O'Brien
                                                  ------------------------------
                                             Title: Senior Vice President
                                                   -----------------------------

[CORPORATE SEAL]

                                             NORWEST BANK MINNESOTA, N.A.
                                             as Trustee and Paying Agent


                                             By: /s/ Marianna C. Sterson
                                                --------------------------------
                                             Name: Marianna C. Sterson
                                                  ------------------------------
                                             Title: Corporate Trust Officer
                                                   -----------------------------


                                       13