<PAGE>   1
                                                         Exhibit 10.57

                        SECOND AMENDMENT TO SERIES B NOTE

     This SECOND AMENDMENT TO SERIES B NOTE dated as of December 18, 1996, is
made by HANOVER DIRECT, INC., a Delaware corporation (the "Borrower"), with the
consent of Norwest Bank Minnesota, N.A., as trustee and paying agent (the
"Trustee" or the "Paying Agent", as applicable);

                              W I T N E S S E T H:

     WHEREAS, the Borrower and the Trustee entered into the Series B Note
Agreement dated as of April 27, 1995, as amended pursuant to that certain First
Supplemental Series B Note Agreement dated as of December 29, 1995 and that
certain Second Supplement Series B Note Agreement dated as of December 18, 1996
between the Borrower and the Trustee (as further amended, restated, supplemented
or otherwise modified from time to time in accordance with its terms, the
"Series B Note Agreement") pursuant to which the Borrower issued and sold its
interest bearing Flexible Term Notes, Series B (the "Series B Notes") in the
aggregate principal amount of $10,000,000 and in the form of Series B Note R-1,
registered in the name of Cede & Co. (as defined in the Series B Note Agreement)
and deposited with the Paying Agent ("Series B Note R-1"); and

     WHEREAS, the Borrower has this day delivered to the Trustee a Substitute
Series B Letter of Credit in substitution for the Series B Letter of Credit
(each as defined in the Series B Note Agreement); and

     WHEREAS, in order to more fully evidence the delivery of the Substitute
Series B of Letter of Credit referenced above, the Borrower and the Trustee
desire to amend Series B Note R-1. subject to the terms and conditions set forth
herein,

     NOW, THEREFORE, the parties hereto agree as follows:

     Section 1. Definitions. All capitalized terms used in this Second Amendment
to Series B Note and not otherwise herein defined shall have the meaning
ascribed to them in the Series B Note Agreement.

     Section 2. Amendments to Series B Note R-1. Series B Note R-1 is hereby
amended as follows:

     (a) The second boldface paragraph on the first page of Series B Note R-1
(prior to the text thereof) is hereby, deleted in its entirety and replaced with
the following:


<PAGE>   2

     THIS SERIES B NOTE IS NOT A DEPOSIT OR OBLIGATION OF, OR GUARANTEED BY,
     SWISS BANK CORPORATION, NEW YORK BRANCH (THE "BANK"), IS NOT INSURED BY THE
     FEDERAL DEPOSIT INSURANCE CORPORATION, AND IS SUBJECT TO INVESTMENT RISKS,
     INCLUDING POSSIBLE LOSS OF THE PRINCIPAL AMOUNT INVESTED. ALTHOUGH NOT
     GUARANTEED BY THE BANK, PAYMENTS OF PRINCIPAL AND INTEREST ON THIS SERIES B
     NOTE AND, IF REMARKETING PROCEEDS ARE NOT AVAILABLE, THE PURCHASE PRICE OF
     THIS SERIES B NOTE, WILL BE MADE FROM DRAWINGS UNDER THE SERIES B LETTER OF
     CREDIT ISSUED BY THE BANK. THE FAILURE OF THE BANK TO HONOR ANY DRAWING
     UNDER THE SERIES B LETTER OF CREDIT WILL NOT GIVE RISE TO ANY CLAIM OTHER
     THAN AGAINST THE BANK.

     (b) The second paragraph of the text of Series B Note R-1, beginning on
page 2 thereof, is amended by deleting the first sentence thereof and replacing
it with the following:

     This Series B Note is one of an issue not to exceed $10,000,000 Hanover
     Direct, Inc. Flexible Term Notes, Series B (the "Series B Notes"), issued
     pursuant to a Series B note Agreement dated as of April 27, 1995 (the
     "Series B Note Agreement"), as amended by the First Supplemental Series B
     Note Agreement dated December 29, 1995 and the Second Supplemental Note
     Agreement dated December 18, 1996 between the Borrower and Norwest Bank
     Minnesota, N.A., as trustee (in such capacity, the "Trustee") and Paying
     Agent, for the purpose of refinancing and/or financing certain
     construction, refurbishment and costs of an approximately 530,000 square
     foot distribution facility of the Borrower located in Roanoke, Virginia and
     a new retail store of Gump's, Inc., a subsidiary of the Borrower located in
     San Francisco, California. Pursuant to the Series B Note Agreement, the
     Borrower has caused Swiss Bank Corporation, New York Branch (the "Bank") to
     issue its irrevocable Series B Letter of Credit dated the Date of Issuance
     (as hereinafter defined and set forth above) of the Series B Notes (the
     "Series B Letter of Credit") in favor of the Trustee, in an amount
     sufficient to pay the Series B Facility Amount and unpaid interest on or
     Purchase Price of the Series B Notes, but not to exceed $9,638,541,
     pursuant to Reimbursement Agreement dated as of December 18, 1996 (the
     "Reimbursement Agreement") by and among the Borrower and the Bank, which
     Series B Letter of Credit initially expires (subject to extension of
     earlier termination as provided in the Reimbursement Agreement and the
     Series B Note Agreement) on February 28, 1998. Substitute letters of credit
     may be delivered in accordance with the Series B Note Agreement.

     (c) The seventh paragraph of the text of Series B Note R-1 beginning on
page 4 thereof, is amended by deleting the third


                                        2

<PAGE>   3

sentence thereof in its entirety, and replacing it with the
following:

     Subject to the provisions of Section 7.09 of the Series B Note Agreement
     relating to the Bank as holder of the Series B Notes, the Borrower, the
     Trustee and the Paying Agent will recognize the Securities Depository
     Nominee, as hereinafter defined, while the registered owner of the Series B
     Notes so held, as the owner of the Series B Notes for all purposes,
     including (i) payments of principal and Purchase Price of. and interest on,
     the Series B Notes, (ii) notices and (iii) voting, subject to certain
     qualifications as stated in the Series B Note Agreement.

     (d) Section 1 of Series B Note R-1, beginning on page 5 thereof, is amended
by deleting the following definition:

     "Bank" means, individually and collectively, the Lender and the L/C Issuer.

     (e) Subsection (f) of Section 3 of Series B Note R-1, beginning on page 9
thereof, is amended by deleting the reference to the "Lender" in the third line
thereof and replacing it with a reference to the "Bank."

     (f) Subsection (a) of Section 4 of Series B Note R-1, beginning on page 10
thereof, is amended by deleting the reference to the "Lender" in the fourth line
of the last paragraph thereof and replacing it with a reference to the "Bank."

     (g) Section 7 of Series B Note R-1, beginning on page 13 thereof, is
amended by deleting the second sentence of the first paragraph thereof in their
entirety and replacing them with the following:

     The Series B Note Agreement directs the Trustee to declare an acceleration
     upon written notice be, the Bank of the occurrence and continuance of an
     event of default under the Reimbursement Agreement and upon the occurrence
     of certain other Events of Default under the Series B Note Agreement. The
     Trustee has the right to accelerate the entire unpaid principal of and
     interest on the Series B Notes in certain events only with the Banks
     consent, all as provided in Article VII of the Series B Note Agreement to
     which reference is hereby made.

     Section 3. Effect of Second Amendment to Series B Note; No Novation. Except
as modified hereby, all of the terms and provisions of Series B Note R-1 shall
remain in full force and effect. This Second Amendment to Series B Note amends
Series B


                                        3

<PAGE>   4

Note R-1 and shall not be construed to constitute a novation thereof in any
manner whatsoever.

     Section 4. Governing Law. This Second Amendment to Series B Note and Series
B Note R-1, as amended hereby, shall be deemed to be contracts made under, and
for all purposes shall be construed in accordance with the laws of the State of
New York.

     Section 5. Severability. If any provision of this Second Amendment to
Series B Note shall be determined to be unenforceable by a court of law, that
shall not affect any other provision of this Second Amendment to Series B Note.

                         [Signatures on following page]


                                        4

<PAGE>   5

     IN WITNESS WHEREOF, the Borrower has caused this Second Amendment to Series
B Note to be duly executed as of the day and year first above written.



                                    HANOVER DIRECT, INC.


                                    By:________________________________
                                    Name:______________________________
                                    Title:_____________________________



[CORPORATE SEAL]


                                        5

<PAGE>   6

                  CONSENT TO SECOND AMENDMENT TO SERIES B NOTE


     Norwest Bank Minnesota, N.A., as Paying Agent, hereby consents to the
amendments to Series B Note R-1 provided for herein.



                              NORWEST BANK MINNESOTA, N.A.,
                               as Trustee and Paying Agent



                              By:_____________________________________
                              Name:___________________________________
                              Title:__________________________________


                                        6