<PAGE>   1
                                                                Exhibit 10.55

                        FIRST AMENDMENT TO SERIES B NOTE


         This FIRST AMENDMENT TO SERIES B NOTE dated as of December 29, 1995, is
made by HANOVER DIRECT, INC., a Delaware corporation (the "Borrower"), with the
consent of Norwest Bank Minnesota, N.A., as trustee and paying agent (the
"Trustee" or the "Paying Agent", as applicable);

                              W I T N E S S E T H:

                  WHEREAS, the Borrower and the Trustee entered into the Series
B Note Agreement dated as of April 27, 1995, as amended pursuant to that certain
First Supplemental Series B Note Agreement dated as of December 29, 1995 between
the Borrower and the Trustee (as further amended, restated, supplemented or
otherwise modified from time to time in accordance with its terms, the "Series B
Note Agreement") pursuant to which the Borrower issued and sold its interest
bearing Flexible Term Notes, Series B (the "Series B Notes") in the aggregate
principal amount of $10,000,000 and in the form of Series B Note R-1, registered
in the name of Cede & Co. (as defined in the Series B Note Agreement) and
deposited with the Paying Agent ("Series B Note R-1"); and

         WHEREAS, the Borrower has this day delivered to the Trustee a
Substitute Series B Letter of Credit in substitution for the Series B Letter of
Credit (each as defined in the Series B Note Agreement); and

         WHEREAS, in order to more fully evidence the delivery of the Substitute
Series B of Letter of Credit referenced above, the Borrower and the Trustee
desire to amend Series B Note R-1, subject to the terms and conditions set forth
herein;

         NOW, THEREFORE, the parties hereto agree as follows:

         SECTION 1. DEFINITIONS. All capitalized terms used in this First
Amendment to Series B Note and not otherwise herein defined shall have the
meaning ascribed to them in the Series B Note Agreement.

         SECTION 2. AMENDMENTS TO SERIES B NOTE R-1. Series B Note R-1 is hereby
amended as follows:

         (a) The second boldface paragraph on the first page of Series B Note
R-1 (prior to the text thereof) is hereby deleted in its entirety and replaced
with the following:

                  PAYMENTS OF PRINCIPAL AND PURCHASE PRICE OF AND
                  INTEREST ON THIS SERIES B NOTE WILL BE MADE FROM
                  DRAWINGS UNDER THE SERIES B LETTER OF CREDIT
                  (HEREINAFTER DEFINED) IF REMARKETING PROCEEDS ARE NOT
<PAGE>   2
                  AVAILABLE OR IF AN EVENT OF DEFAULT HAS OCCURRED. ALTHOUGH THE
                  SERIES B LETTER OF CREDIT IS A BINDING OBLIGATION OF
                  CORESTATES BANK, N.A. (THE "L/C ISSUER"), THIS SERIES B NOTE
                  IS NOT A DEPOSIT OR OBLIGATION OF CORESTATES FINANCIAL CORP OR
                  ANY OF ITS AFFILIATES, INCLUDING THE L/C ISSUER AND CONGRESS
                  FINANCIAL CORPORATION, NOR IS THIS SERIES B NOTE A DEPOSIT OR
                  OBLIGATION OF NATIONSBANK CORPORATION OR ANY OF ITS AFFILIATED
                  BANKS, INCLUDING NATIONSBANK, N.A., AND THIS SERIES B NOTE IS
                  NOT GUARANTEED BY ANY OF THESE ENTITIES. THIS SERIES B NOTE IS
                  NOT INSURED BY THE FEDERAL DEPOSIT INSURANCE CORPORATION AND
                  IS SUBJECT TO CERTAIN INVESTMENT RISKS INCLUDING POSSIBLE LOSS
                  OF THE PRINCIPAL AMOUNT INVESTED.

         (b) The second paragraph of the text of Series B Note R-1, beginning on
page 2 thereof, is amended by deleting the first two sentences thereof and
replacing them with the following:

                  This Series B Note is one of an issue not to exceed
                  $10,000,000 Hanover Direct, Inc. Flexible Term Notes, Series B
                  (the "Series B Notes"), issued pursuant to a Series B Note
                  Agreement dated as of November 9, 1994, between the Borrower,
                  and Norwest Bank Minnesota, N.A., as trustee (in such
                  capacity, the "Trustee") and Paying Agent, as amended by that
                  certain First Supplemental Series B Note Agreement dated as of
                  December 29, 1995 between the Borrower and the Trustee (as
                  further amended, restated, supplemented or otherwise modified
                  from time to time in accordance with its terms, the "Series B
                  Note Agreement"), for the purpose of refinancing and/or
                  financing certain construction, refurbishment and related
                  costs of an approximately 530,000 square foot distribution
                  facility owned and used by certain subsidiaries of the
                  Borrower located in Roanoke, Virginia and a new retail store
                  of Gump's Corp., a subsidiary of the Borrower located in San
                  Francisco, California. Pursuant to the Series B Note
                  Agreement, the Borrower has arranged, through a credit
                  facility established with Lender for certain of Borrower's
                  subsidiaries, and as guarantor of such financing arrangements
                  with Lender, for L/C Issuer to issue for the account of such
                  subsidiaries its irrevocable Series B Letter of Credit dated
                  December 29, 1995 (the "Series B Letter of Credit") in favor
                  of the Trustee, in an amount sufficient to pay the Series B
                  Facility Amount and unpaid interest on or Purchase Price of
                  the Series B Notes, but not to exceed $10,145,833, pursuant to
                  (a) the Application for Irrevocable Standby Letter of Credit
                  dated as of December 22, 1995 executed and delivered to the
                  L/C Issuer by the Account Parties (as defined in the Series B
                  Note Agreement) and the Lender requesting the issuance by the
                  L/C Issuer of the Series B Letter of Credit (the
                  "Application"), (b) the Loan and Security Agreement dated as
                  of November 14, 1995 (the "Loan Agreement"), by and among the
                  Account

                                        2
<PAGE>   3
                  Parties and certain other subsidiaries of the Borrower, and
                  the Lender, acknowledged and agreed to by the Borrower and
                  certain other subsidiaries of the Borrower, pursuant to which,
                  among other things, the Lender has executed the Application
                  pursuant to which the Series B Letter of Credit is issued by
                  the L/C Issuer and delivered to the Trustee, and any and all
                  modifications, alterations, amendments and supplements
                  thereto, (c) the other "Financing Agreements" as defined in
                  the Loan Agreement and (d) any similar agreements between or
                  among the Account Parties, the Borrower and the issuer of a
                  Substitute Series B Letter of Credit or lender providing
                  credit support to such issuer (individually and collectively,
                  the "Reimbursement Agreement").

         (c) The seventh paragraph of the text of Series B Note R-1, beginning
on page 4 thereof, is amended by deleting the third sentence thereof in its
entirety and replacing it with the following:

                  The Borrower, the Trustee and the Paying Agent will recognize
                  the Securities Depository Nominee, as hereinafter defined,
                  while the registered owner of the Series B Notes so held, as
                  the owner of the Series B Notes for all purposes, including
                  (i) payments of principal and Purchase Price of, and interest
                  on, the Series B Notes, (ii) notices and (iii) voting.

         (d) Section 1 of Series B Note R-1, beginning on page 5 thereof, is
amended by adding the following definition:

                  "BANK" means, individually and collectively, the Lender and
the L/C Issuer.

         (e) Subsection (c) of Section 3 of Series B Note R-1, beginning on page
8 thereof, is amended by deleting the reference to the "Bank" in the eighth line
thereof and replacing it with a reference to the "L/C Issuer".

         (f) Subsection (f) of Section 3 of Series B Note R-1, beginning on page
9 thereof, is amended by deleting the reference to the "Bank" in the third line
thereof and replacing it with a reference to the "Lender."

         (g) Subsection (a) of Section 4 of Series B Note R-1, beginning on page
10 thereof, is amended by deleting the reference to the "Bank" in the fourth
line of the last paragraph thereof and replacing it with a reference to the
"Lender."

         (h) Section 7 of Series B Note R-1, beginning on page 13 thereof, is
amended by deleting the second and third sentences of the first paragraph
thereof in their entirety and replacing them with the following:

                  The Series B Note Agreement directs the Trustee to declare an
                  acceleration upon written notice by the Lender of the
                  occurrence and continuance of an event of


                                        3
<PAGE>   4
                  default under the Reimbursement Agreement and upon the
                  occurrence of certain other Events of Default under the Series
                  B Note Agreement.

         SECTION 3. EFFECT OF FIRST AMENDMENT TO SERIES B NOTE; NO NOVATION.
Except as modified hereby, all of the terms and provisions of Series B Note R-1
shall remain in full force and effect. This First Amendment to Series B Note
amends Series B Note R-1 and shall not be construed to constitute a novation
thereof in any manner whatsoever.

         SECTION 4. GOVERNING LAW. This First Amendment to Series B Note and
Series B Note R-1, as amended hereby, shall be deemed to be contracts made
under, and for all purposes shall be construed in accordance with, the laws of
the State of New York.

         SECTION 5. SEVERABILITY. If any provision of this First Amendment to
Series B Note shall be determined to be unenforceable by a court of law, that
shall not affect any other provision of this First Amendment to Series B Note.




                         [Signatures on following page]


                                        4
<PAGE>   5
         IN WITNESS WHEREOF, the Borrower has caused this First Amendment to
Series B Note to be duly executed as of the day and year first above written.

                                             HANOVER DIRECT, INC.


                                             By:  /s/ Edward J. O'Brien
                                                  ----------------------------
                                             Name:    Edward J. O'Brien
                                                    --------------------------
                                             Title:   Executive VP & Treasurer 
                                                    --------------------------

[CORPORATE SEAL]



                                        5
<PAGE>   6
                   CONSENT TO FIRST AMENDMENT TO SERIES B NOTE


         Norwest Bank Minnesota, N.A., as Paying Agent, hereby consents to the
amendments to Series B Note R-1 provided for herein.

                                             NORWEST BANK MINNESOTA, N.A.,
                                               as Trustee and Paying Agent

                                             By:
                                                  ------------------------
                                             Name:
                                                    ----------------------
                                             Title:
                                                    ----------------------


                                       6