<PAGE>   1
                                                                   EXHIBIT 10.49


                        SECOND AMENDMENT TO SERIES A NOTE



         This SECOND AMENDMENT TO SERIES A NOTE dated as of December 18, 1996,
is made by HANOVER DIRECT, INC., a Delaware corporation (the "Borrower"), with
the consent of Norwest Bank Minnesota, N.A., as trustee and paying agent (the
"Trustee" or the "Paying Agent", as applicable);

                              W I T N E S S E T H:

         WHEREAS, the Borrower and the Trustee entered into the Series A Note
Agreement dated as of November 9, 1994, as amended pursuant to that certain
First Supplemental Series A Note Agreement dated as of December 29, 1995 and
that certain Second Supplement Series A Note Agreement dated as of December 18,
1996 between the Borrower and the Trustee (as further amended, restated,
supplemented or otherwise modified from time to time in accordance with its
terms, the "Series A Note Agreement") pursuant to which the Borrower issued and
sold its interest bearing Flexible Term Notes, Series A (the "Series A Notes")
in the aggregate principal amount of $10,000,000 and in the form of Series A
Note R-1, registered in the name of Cede & Co. (as defined in the Series A Note
Agreement) and deposited with the Paying Agent ("Series A Note R-1"); and

         WHEREAS, the Borrower has this day delivered to the Trustee a
Substitute Series A Letter of Credit in substitution for the Series A Letter of
Credit (each as defined in the Series A Note Agreement); and

         WHEREAS, in order to more fully evidence the delivery of the Substitute
Series A of Letter of Credit referenced above, the Borrower and the Trustee
desire to amend Series A Note R-1. subject to the terms and conditions set forth
herein,

         NOW, THEREFORE, the parties hereto agree as follows:

         SECTION 1. DEFINITIONS. All capitalized terms used in this Second
Amendment to Series A Note and not otherwise herein defined shall have the
meaning ascribed to them in the Series A Note Agreement.

         SECTION 2. AMENDMENTS TO SERIES A NOTE R-1. Series A Note R-1 is hereby
amended as follows:

         (a) The second boldface paragraph on the first page of Series A Note
R-1 (prior to the text thereof) is hereby, deleted in its entirety and replaced
with the following:
<PAGE>   2
         THIS SERIES A NOTE IS NOT A DEPOSIT OR OBLIGATION OF, OR GUARANTEED BY,
         SWISS BANK CORPORATION, NEW YORK BRANCH (THE "BANK"), IS NOT INSURED BY
         THE FEDERAL DEPOSIT INSURANCE CORPORATION, AND IS SUBJECT TO INVESTMENT
         RISKS, INCLUDING POSSIBLE LOSS OF THE PRINCIPAL AMOUNT INVESTED.
         ALTHOUGH NOT GUARANTEED BY THE BANK, PAYMENTS OF PRINCIPAL AND INTEREST
         ON THIS SERIES A NOTE AND, IF REMARKETING PROCEEDS ARE NOT AVAILABLE,
         THE PURCHASE PRICE OF THIS SERIES A NOTE, WILL BE MADE FROM DRAWINGS
         UNDER THE SERIES A LETTER OF CREDIT ISSUED BY THE BANK. THE FAILURE OF
         THE BANK TO HONOR ANY DRAWING UNDER THE SERIES A LETTER OF CREDIT WILL
         NOT GIVE RISE TO ANY CLAIM OTHER THAN AGAINST THE BANK.

         (b) The second paragraph of the text of Series A Note R-1, beginning on
page 2 thereof, is amended by deleting the first sentence thereof and replacing
it with the following:

         This Series A Note is one of an issue not to exceed $10,000,000 Hanover
         Direct, Inc. Flexible Term Notes, Series A (the "Series A Notes"),
         issued pursuant to a Series A note Agreement dated as of November 9,
         1994 (the "Series A Note Agreement"), as amended by the First
         Supplemental Series A Note Agreement dated December 29, 1995 and the
         Second Supplemental Note Agreement dated December 18, 1996 between the
         Borrower and Norwest Bank Minnesota, N.A., as trustee (in such
         capacity, the "Trustee") and Paying Agent, for the purpose of
         refinancing and/or financing certain construction, refurbishment and
         costs of an approximately 530,000 square foot distribution facility of
         the Borrower located in Roanoke, Virginia and a new retail store of
         Gump's, Inc., a subsidiary of the Borrower located in San Francisco,
         California. Pursuant to the Series A Note Agreement, the Borrower has
         caused Swiss Bank Corporation, New York Branch (the "Bank") to issue
         its irrevocable Series A Letter of Credit dated the Date of Issuance
         (as hereinafter defined and set forth above) of the Series A Notes (the
         "Series A Letter of Credit") in favor of the Trustee, in an amount
         sufficient to pay the Series A Facility Amount and unpaid interest on
         or Purchase Price of the Series A Notes, but not to exceed $9,638,541,
         pursuant to Reimbursement Agreement dated as of December 18, 1996 (the
         "Reimbursement Agreement") by and among the Borrower and the Bank,
         which Series A Letter of Credit initially expires (subject to extension
         of earlier termination as provided in the Reimbursement Agreement and
         the Series A Note Agreement) on February 28, 1998. Substitute letters
         of credit may be delivered in accordance with the Series A Note
         Agreement.

         (c) The seventh paragraph of the text of Series A Note R-1 beginning on
page 4 thereof, is amended by deleting the third


                                        2
<PAGE>   3
sentence thereof in its entirety, and replacing it with the following:

         Subject to the provisions of Section 7.09 of the Series A Note
         Agreement relating to the Bank as holder of the Series A Notes, the
         Borrower, the Trustee and the Paying Agent will recognize the
         Securities Depository Nominee, as hereinafter defined, while the
         registered owner of the Series A Notes so held, as the owner of the
         Series A Notes for all purposes, including (i) payments of principal
         and Purchase Price of. and interest on, the Series A Notes, (ii)
         notices and (iii) voting, subject to certain qualifications as stated
         in the Series A Note Agreement.

         (d) Section 1 of Series A Note R-1, beginning on page 5 thereof, is
amended by deleting the following definition:

         "BANK" means, individually and collectively, the Lender and the L/C
         Issuer.

         (e) Subsection (f) of Section 3 of Series A Note R-1, beginning on page
9 thereof, is amended by deleting the reference to the "Lender" in the third
line thereof and replacing it with a reference to the "Bank."

         (f) Subsection (a) of Section 4 of Series A Note R-1, beginning on page
10 thereof, is amended by deleting the reference to the "Lender" in the fourth
line of the last paragraph thereof and replacing it with a reference to the
"Bank."

         (g) Section 7 of Series A Note R-1, beginning on page 13 thereof, is
amended by deleting the second sentence of the first paragraph thereof in their
entirety and replacing them with the following:


         The Series A Note Agreement directs the Trustee to declare an
         acceleration upon written notice be, the Bank of the occurrence and
         continuance of an event of default under the Reimbursement Agreement
         and upon the occurrence of certain other Events of Default under the
         Series A Note Agreement. The Trustee has the right to accelerate the
         entire unpaid principal of and interest on the Series A Notes in
         certain events only with the Banks consent, all as provided in Article
         VII of the Series A Note Agreement to which reference is hereby made.

         Section 3. Effect of Second Amendment to Series A Note; No Novation.
Except as modified hereby, all of the terms and provisions of Series A Note R-1
shall remain in full force and effect. This Second Amendment to Series A Note
amends Series A


                                        3
<PAGE>   4
Note R-1 and shall not be construed to constitute a novation thereof in any
manner whatsoever.

         Section 4. Governing Law. This Second Amendment to Series A Note and
Series A Note R-1, as amended hereby, shall be deemed to be contracts made
under, and for all purposes shall be construed in accordance with the laws of
the State of New York.

         Section 5. Severability. If any provision of this Second Amendment to
Series A Note shall be determined to be unenforceable by a court of law, that
shall not affect any other provision of this Second Amendment to Series A Note.

                         [Signatures on following page]


                                        4
<PAGE>   5
         IN WITNESS WHEREOF, the Borrower has caused this Second Amendment to
Series A Note to be duly executed as of the day and year first above written.



                                      HANOVER DIRECT, INC.


                                      By: /s/ Edward J. O'Brien
                                          ________________________________

                                      Name: Edward J. O'Brien
                                            ______________________________

                                      Title: Senior Vice President
                                             Secretary & Treasurer
                                             _____________________________


[CORPORATE SEAL]


                                        5
<PAGE>   6
                  CONSENT TO SECOND AMENDMENT TO SERIES A NOTE


         Norwest Bank Minnesota, N.A., as Paying Agent, hereby consents to the
amendments to Series A Note R-1 provided for herein.



                                       NORWEST BANK MINNESOTA, N.A.,
                                        as Trustee and Paying Agent



                                       By: /s/ Marianna C. Sterson
                                           _____________________________________

                                       Name: Marianna C. Sterson
                                             ___________________________________

                                       Title: Corporate Trust Officer
                                              __________________________________



                                        6