<PAGE>   1
                                                                Exhibit 10.24

                 SIXTH AMENDMENT TO LOAN AND SECURITY AGREEMENT


                  SIXTH AMENDMENT TO LOAN AND SECURITY AGREEMENT, dated as of
December 5, 1996, by and among CONGRESS FINANCIAL CORPORATION, a California
corporation ("Lender"), HANOVER DIRECT PENNSYLVANIA, INC., a Pennsylvania
corporation ("HDPI"), BRAWN OF CALIFORNIA, INC., a California corporation
("Brawn"), GUMP'S BY MAIL, INC., a Delaware corporation ("GBM"), GUMP'S CORP., a
California corporation ("Gump's"), THE COMPANY STORE, INC., a Wisconsin
corporation ("TCSI"), TWEEDS, INC., a Delaware corporation ("Tweeds"), LWI
HOLDINGS, INC., a Delaware corporation ("LWI"), AEGIS CATALOG CORPORATION, a
Delaware corporation ("Aegis"), HANOVER DIRECT VIRGINIA INC., a Delaware
corporation ("HDV"), HANOVER REALTY, INC., a Virginia corporation ("Hanover
Realty"), and THE AUSTAD COMPANY, a South Dakota corporation ("Austad"; and
together with HDPI, Brawn, GBM, Gump's, TCSI, Tweeds, LWI, Aegis, HDV and
Hanover Realty, each individually referred to herein as a "Borrower" and
collectively, "Borrowers") and HANOVER DIRECT, INC., a Delaware corporation
("Hanover"), AEGIS RETAIL CORPORATION, a Delaware corporation, AEGIS SAFETY
HOLDINGS, INC., a Delaware corporation, AEGIS VENTURES, INC., a Delaware
corporation, AMERICAN DOWN & TEXTILE COMPANY, a Wisconsin corporation, BRAWN
WHOLESALE CORP., a California corporation, THE COMPANY FACTORY, INC., a
Wisconsin corporation, THE COMPANY OFFICE, INC., a Wisconsin corporation,
COMPANY STORE HOLDINGS, INC., a Delaware corporation, D.M. ADVERTISING, INC., a
New Jersey corporation, GUMP'S CATALOG, INC., a Delaware corporation, GUMP'S
HOLDINGS, INC., a Delaware corporation, HANOVER CASUALS, INC., a Delaware
corporation, HANOVER CATALOG HOLDINGS, INC., a Delaware corporation, HANOVER
FINANCE CORPORATION, a Delaware corporation, HANOVER LIST MANAGEMENT INC., a New
Jersey corporation, HANOVER VENTURES, INC., a Delaware corporation, LEICHTUNG OF
MICHIGAN, INC., a Michigan corporation, LWI RETAIL, INC., an Ohio corporation,
SCANDIA DOWN CORPORATION, a Delaware corporation, TWEEDS OF VERMONT, INC., a
Delaware corporation, YORK FULFILLMENT COMPANY, INC., a Pennsylvania
corporation, and AUSTAD HOLDINGS, INC., a Delaware corporation (each
individually a "Guarantor" and collectively, "Guarantors").


                              W I T N E S S E T H:


                  WHEREAS, Borrowers, Guarantors and Lender entered into the
Loan and Security Agreement, dated November 14, 1995, as amended by the First
Amendment to Loan and Security Agreement, dated February 22, 1996, the Second
Amendment to Loan and Security Agreement, dated April 16, 1996, the Third
Amendment to Loan and Security Agreement, dated May 24, 1996, the Fourth
Amendment to Loan and Security Agreement, dated May 31, 1996, and the Fifth
Amendment to Loan and Security Agreement, dated September 11, 1996
(collectively, the "Loan Agreement"), pursuant
<PAGE>   2
to which Lender has made loans and advances and provided other financial
accommodations to Borrowers; and

                  WHEREAS, Borrowers and Guarantors have informed Lender that
Richemont Finance S.A. intends to make a loan to Hanover in the original
principal amount of $10,000,000, the proceeds of which will be used solely by
Hanover to make intercompany loan(s) to Revolving Loan Borrowers, to be used by
Revolving Loan Borrowers solely for working capital purposes of Revolving Loan
Borrowers; and

                  WHEREAS, Borrowers and Guarantors have requested that Lender
amend the Consolidated Net Worth financial covenant; and

                  WHEREAS, Lender is willing to agree to the foregoing,
subject to the terms and conditions contained herein; and

                  WHEREAS, the parties to the Loan Agreement desire to enter
into this Sixth Amendment to Loan and Security Agreement (this "Amendment") to
evidence and effectuate such amendments and agreements to the extent set forth
herein, and subject to the terms and conditions set forth herein;

                  NOW, THEREFORE, in consideration of the premises and covenants
set forth herein and other good and valuable consideration, the receipt and
sufficiency of which is hereby acknowledged, the parties hereto agree as
follows:

                  1.       Definitions.

                           (a) Additional Definitions. As used herein or in any
of the other Financing Agreements, the following terms shall have the meanings
given to them below, and the Loan Agreement shall be deemed and is hereby
amended to include, in addition and not in limitation, the following
definitions:

                                    (i) "CoreStates" shall mean CoreStates Bank,
N.A., a national banking association, and its successors and assigns.

                                    (ii) "December 1996 Intercompany Loans"
shall mean the intercompany loans made, contemporaneously herewith, by Hanover
to the Revolving Loan Borrowers, funded with the proceeds of the $10,000,000
Richemont Loan.

                                    (iii) "Richemont" shall mean Richemont
Finance S.A., a societe anonyme organized under the laws of the Grand Duchy of
Luxembourg, and its successors and assigns.

                                    (iv) "$10,000,000 Richemont Loan" shall mean
the loan in the original principal sum of $10,000,000 made by Richemont to
Hanover, contemporaneously herewith.

                                       -2-
<PAGE>   3
                                    (v) "$10,000,000 Richemont Note" shall mean
the Promissory Note, dated of even date herewith, by Hanover payable to
Richemont in the original principal amount of $10,000,000 delivered to evidence
the $10,000,000 Richemont Loan plus interest thereon.

                           (b) Amendment to Definition. Section 1.23 of the Loan
Agreement is hereby amended by deleting the proviso appearing at the end thereof
and substituting the following proviso therefor, effective as of the date
hereof:

                                    "; provided, however, that solely for
                           purposes of calculating Consolidated Working Capital
                           hereunder, the outstanding balance of the Revolving
                           Loans and Term Loans and the outstanding balances of
                           the $10,000,000 IMR Note and the $10,000,000
                           Richemont Note shall not be considered current
                           liabilities."

                           (c) Interpretation. For purposes of this Amendment,
unless otherwise defined herein, all capitalized terms used herein that are
defined in the Loan Agreement, shall have the respective meanings given to such
terms in the Loan Agreement.

                  2. Use of Proceeds and Repayment of Richemont Loan.

                           (a) Hanover agrees to use the proceeds of the
$10,000,000 Richemont Loan solely to make the December 1996 Intercompany Loans
to Revolving Loan Borrowers and Revolving Loan Borrowers agree to use the
proceeds of such intercompany loans from Hanover solely for working capital
purposes of Revolving Loan Borrowers. Hanover and Revolving Loan Borrowers shall
arrange for the disbursement of the $10,000,000 Richemont Loan and the
corresponding December 1996 Intercompany Loans by Hanover to Revolving Loan
Borrowers, by wire transfer(s) directly from Richemont to Lender, for credit to
the applicable Revolving Loan Borrowers' Revolving Loan accounts maintained by
Lender.

                           (b) Notwithstanding Section 6.5(b)(ii) of the Loan
Agreement, Revolving Loan Borrowers may use the proceeds of Revolving Loans to
make payments to Hanover, or directly to Richemont at the direction of Hanover,
in respect of the valid intercompany Indebtedness owed to Hanover in respect of
the December 1996 Intercompany Loans, consisting of principal and accrued
interest, together with an amount equal to Hanover's reimbursement obligations
to Richemont for bank fees, not to exceed $100,000 in the aggregate, incurred by
Richemont in connection with the $10,000,000 Richemont Loan, and in addition
thereto, an amount equal to the attorneys' fees required to be reimbursed or
paid to Richemont incurred in connection with the making of the $10,000,000
Richemont Loan; provided, that, except

                                       -3-
<PAGE>   4
for such amounts paid to Hanover and used to make payment to Richemont of such
attorneys' fees, no such payment shall be made unless (i) prior to any such
payment, each of the following Letter of Credit Accommodations has been
cancelled and surrendered to Lender or CoreStates prior to any drawing under any
of them (other than drawings under the letters of credit referred to in clauses
(B) and/or (C) below that have been reinstated in accordance with the terms of
such letters of credit), and replaced with substitute letters of credit in
amounts and on terms satisfactory to Lender: (A) the Irrevocable Standby Letter
of Credit No. 516262P, dated November 14, 1995, issued by CoreStates for the
account of HDPI, Brawn and GBM in favor of NationsBank in the stated amount of
$8,560,000, which, together with the NationsBank letter of credit referred to
therein, must be replaced by a direct pay letter of credit in favor of Fleet
National Bank, as the successor trustee of certain industrial development
revenue bonds issued by the Littlestown Industrial Development Authority to
finance a project for HDPI, (B) the Irrevocable Transferable Letter of Credit
No. 516466P, dated December 27, 1995, in the initial stated amount of
$10,145,833 issued by CoreStates for the account of HDPI, HDV and Gump's in
favor of Norwest Bank Minnesota, N.A., as trustee, under the Series A Note
Agreement, dated as of November 9, 1994, as amended, with respect to the
issuance of the Series A Notes of Hanover in the original principal amount of
$10,000,000, and (C) the Irrevocable Transferable Letter of Credit No. 516467P,
dated December 27, 1995, in the initial stated amount of $10,145,833 issued by
CoreStates for the account of HDPI, HDV and Gump's in favor of Norwest Bank
Minnesota, N.A., as trustee, under the Series B Note Agreement, dated as of
April 27, 1995, as amended, with respect to the issuance of the Series B Notes
of Hanover in the original principal amount of $10,000,000 and (ii) no Event of
Default or Incipient Default exists or has occurred and is continuing at the
time of any such payment or would exist or occur as a result thereof.

                           (c) Prior to the incurrence by any Borrower or
Hanover of any reimbursement, indemnification or other obligations in favor of
any issuer of any of the replacement letters of credit described in Section 2(b)
hereof or in favor of Richemont as guarantor thereof or Hanover or any other
person in connection therewith, Borrowers and Hanover shall first obtain
Lender's prior written consent thereto. Lender's requirements for such consent
shall include, but are not necessarily limited to (i) Lender's satisfaction with
the terms and conditions of any such obligations and agreements evidencing such
obligations, and (ii) the execution and delivery of a written subordination
agreement in favor of Lender, by each of the obligees of such obligations,
subordinating each such obligee's rights to payment of such obligations to the
prior indefeasible payment and satisfaction in full of the Obligations of
Borrowers to Lender, such written subordination agreement to have terms and
provisions

                                       -4-
<PAGE>   5
substantially in the form of those contained in the subordination agreement by
Richemont delivered under Section 6(b)(ii) hereof and otherwise in form and
substance satisfactory to Lender.

                  3. Consolidated Net Worth. Section 6.20 of the Loan Agreement
is hereby deleted in its entirety and the following substituted therefor:

                           "6.20  Consolidated Net Worth.  Hanover
                  shall, as at the end of each fiscal month, maintain
                  Consolidated Net Worth, calculated on a consolidated basis for
                  Hanover and its Subsidiaries, of at least Seventy Million
                  Dollars ($70,000,000)."

                  4. Amendment Fee. In addition to all other fees, charges,
interest and expenses payable by Borrowers to Lender under the Loan Agreement
and the other Financing Agreements, HDPI shall pay to Lender a fee for entering
into this Amendment in the amount of Fifty Thousand Dollars ($50,000), which
amount is fully earned and payable as of the date hereof and may be charged
directly to HDPI's loan account maintained by Lender in respect of the Revolving
Loans.

                  5. Representations, Warranties and Covenants. Borrowers
represent, warrant and covenant with and to Lender as follows, which
representations, warranties and covenants are continuing and shall survive the
execution and delivery hereof, the truth and accuracy of, or compliance with
each, together with the representations, warranties and covenants in the other
Financing Agreements, being a condition of the effectiveness of this Amendment
and a continuing condition of the making or providing of any Revolving Loans or
Letter of Credit Accommodations by Lender to Borrowers:

                           (a) This Amendment has been duly authorized, executed
and delivered by all necessary action of each of the Borrowers and Guarantors
which is a party hereto, and is in full force and effect, and the agreements and
obligations of Borrowers and Guarantors, as the case may be, contained herein
constitute legal, valid and binding obligations of Borrowers and Guarantors, as
the case may be, enforceable against them in accordance with their terms.

                           (b) Neither the execution and delivery of the
$10,000,000 Richemont Note, the making of the December 1996 Intercompany Loans,
or any other agreements, documents or instruments in connection therewith, nor
the consummation of the transactions therein contemplated, nor compliance with
the provisions thereof (i) has violated or shall violate any Federal or State
securities laws or any other law or regulation or any order or decree of any
court or governmental instrumentality in

                                       -5-
<PAGE>   6
any respect, or (ii) does, or shall conflict with or result in the breach of, or
constitute a default in any respect under any mortgage, deed of trust, security
agreement, agreement or instrument to which Hanover or any other Guarantor or
any Borrower is a party or may be bound, or (iii) does or shall violate any
provision of the Certificate of Incorporation or By-Laws of Hanover or any other
Guarantor or any Borrower.

                           (c) All of the representations and warranties set
forth in the Loan Agreement as amended hereby, and the other Financing
Agreements, are true and correct in all material respects, except to the extent
any such representation or warranty is made as of a specified date, in which
case such representation or warranty shall have been true and correct as of such
date.

                           (d) Within ten (10) days after the date hereof,
Borrowers and Guarantors shall deliver to, or cause to be delivered to, Lender,
in form and substance satisfactory to Lender, the original executed
Subordination Agreement referred to in Section 6(b) hereof, together with (i)
evidence of the adoption and subsistence of authorizing resolutions of Richemont
approving the execution, delivery and performance by Richemont of such
Subordination Agreement and (ii) an opinion of Luxembourg counsel to Richemont
addressed to Lender with respect to the due authorization, execution, validity
and enforceability of such Subordination Agreement, and as to such other matters
as Lender shall reasonably require.

                           (e) After giving effect to the provisions of this
Amendment, no Event of Default or Incipient Default exists or has occurred and
is continuing.

                  6. Conditions Precedent. Concurrently with the execution
hereof, and as a condition to the effectiveness of this Amendment and the
agreement of Lender to the amendments set forth in this Amendment:

                           (a) Lender shall have received an original of this
Amendment, in form and substance satisfactory to Lender and its counsel, duly
authorized, executed and delivered by Borrowers and Guarantors;

                           (b) Lender shall have received, each in form and
substance satisfactory to Lender, (i) a true and complete copy of the
$10,000,000 Richemont Note and all agreements, documents and instruments
relating thereto, and (ii) a written subordination agreement, dated of even date
herewith, among Richemont, Hanover and Lender, pursuant to which, among other
things, Richemont shall have subordinated its right to payment under the
$10,000,000 Richemont Note to the prior indefeasible payment in full of all of
the Obligations, to the extent provided therein,

                                       -6-
<PAGE>   7
each duly authorized, executed and delivered by Hanover and Richemont; and

                           (c)  Lender shall have received, in form and
substance satisfactory to Lender, a letter from Richemont's special New York
counsel providing for (i) such counsel's attestation of the genuineness of the
signatures appearing on the Subordination Agreement described in Section 6(b)
hereof, as executed on behalf of Richemont and delivered to Lender via facsimile
transmission and (ii) such counsel's undertaking to deliver, within ten (10)
days after the date hereof, such original Subordination Agreement executed by
Richemont, together with (A) evidence of the adoption and subsistence of
authorizing resolutions of Richemont approving the execution, delivery and
performance by Richemont of such Subordination Agreement and (B) an opinion of
Luxembourg counsel to Richemont addressed to Lender with respect to the due
authorization, execution, validity and enforceability of such Subordination
Agreement, and as to such other matters as Lender shall reasonably require.

                  7. Effect of this Amendment. This Amendment constitutes the
entire agreement of the parties with respect to the subject matter hereof, and
supersedes all prior oral or written communications, memoranda, proposals,
negotiations, discussions, term sheets and commitments with respect to the
subject matter hereof. Except as expressly provided herein, no other changes or
modifications to the Loan Agreement or any of the other Financing Agreements, or
waivers or consents under any provisions of any of the foregoing, are intended
or implied by this Amendment, and in all other respects the Financing Agreements
are hereby specifically ratified, restated and confirmed by all parties hereto
as of the effective date hereof. To the extent that any provision of the Loan
Agreement or any of the other Financing Agreements conflicts with any provision
of this Amendment, the provision of this Amendment shall control.

                  8. Further Assurances. Borrowers and Guarantors shall execute
and deliver such additional documents and take such additional action as may be
reasonably requested by Lender to effectuate the provisions and purposes of this
Amendment.

                  9. Governing Law. The rights and obligations hereunder of each
of the parties hereto shall be governed by and interpreted and determined in
accordance with the internal laws of the State of New York (without giving
effect to principles of conflicts of laws).


                                       -7-
<PAGE>   8
                  10. Binding Effect. This Amendment shall be binding upon and
inure to the benefit of each of the parties hereto and their respective
successors and assigns.

                  11. Counterparts. This Amendment may be executed in any number
of counterparts, but all of such counterparts shall together constitute but one
and the same agreement. In making proof of this Amendment, it shall not be
necessary to produce or account for more than one counterpart thereof signed by
each of the parties hereto.

                  IN WITNESS WHEREOF, the parties hereto have caused this
Amendment to be executed on the day and year first written.

                                         CONGRESS FINANCIAL CORPORATION

                                         By: /s/ Janet S. Last
                                            _________________________

                                         Title:  Vice President
                                               ______________________


                                         HANOVER DIRECT PENNSYLVANIA, INC.

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title:  Vice President
                                               ______________________


                                         BRAWN OF CALIFORNIA, INC.

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title:  Vice President
                                               ______________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]

                                       -8-
<PAGE>   9
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]

                                         GUMP'S BY MAIL, INC.

                                         By:   /s/ Edward J. O'Brien
                                               -------------------------- 
                                         Title:    Vice President
                                               --------------------------

                                         GUMP'S CORP.

                                         By:   /s/ Edward J. O'Brien
                                               -------------------------- 
                                         Title:    Vice President
                                               --------------------------

                                         THE COMPANY STORE, INC.

                                         By:   /s/ Edward J. O'Brien
                                               -------------------------- 
                                         Title:    Vice President
                                               --------------------------


                                         TWEEDS, INC.

                                         By:   /s/ Edward J. O'Brien
                                               -------------------------- 
                                         Title:    Vice President
                                               --------------------------


                                         LWI HOLDINGS, INC.

                                         By:   /s/ Edward J. O'Brien
                                               -------------------------- 
                                         Title:    Vice President
                                               --------------------------


                                         AEGIS CATALOG CORPORATION

                                         By:   /s/ Edward J. O'Brien
                                               -------------------------- 
                                         Title:    Vice President
                                               --------------------------


                                         HANOVER DIRECT VIRGINIA INC.

                                         By:   /s/ Edward J. O'Brien
                                               -------------------------- 
                                         Title:    Vice President
                                               --------------------------

                       [SIGNATURES CONTINUE ON NEXT PAGE]

                                       -9-
<PAGE>   10
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                        HANOVER REALTY, INC.

                                         By:   /s/ Edward J. O'Brien
                                               -------------------------- 
                                         Title:    Vice President
                                               --------------------------


                                        THE AUSTAD COMPANY

                                         By:   /s/ Edward J. O'Brien
                                               -------------------------- 
                                         Title:    Vice President
                                               --------------------------


By their signatures below, the undersigned Guarantors acknowledge and agree to
be bound by the applicable provisions of this Amendment:

HANOVER DIRECT, INC.

By:   /s/ Edward J. O'Brien
      -------------------------- 
Title:    Senior Vice President
      --------------------------


AEGIS RETAIL CORPORATION

By:   /s/ Edward J. O'Brien
     -------------------------- 
Title:    Vice President
     --------------------------


AEGIS SAFETY HOLDINGS, INC.

By:   /s/ Edward J. O'Brien
     -------------------------- 
Title:    Vice President
     --------------------------


AEGIS VENTURES, INC.

By:   /s/ Edward J. O'Brien
     -------------------------- 
Title:    Vice President
     --------------------------

                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                      -10-
<PAGE>   11
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]

AMERICAN DOWN & TEXTILE COMPANY

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


BRAWN WHOLESALE CORP.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


THE COMPANY FACTORY, INC.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


THE COMPANY OFFICE, INC.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


COMPANY STORE HOLDINGS, INC.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


D.M. ADVERTISING, INC.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


GUMP'S CATALOG, INC.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                      -11-
<PAGE>   12
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]

GUMP'S HOLDINGS, INC.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


HANOVER CASUALS, INC.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


HANOVER CATALOG HOLDINGS, INC.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


HANOVER FINANCE CORPORATION

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


HANOVER LIST MANAGEMENT, INC.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


HANOVER VENTURES, INC.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


LEICHTUNG OF MICHIGAN, INC.

By: /s/ Edward J. O'Brien
   ----------------------------
Title: Vice President
      -------------------------


                       [SIGNATURES CONTINUE ON NEXT PAGE]



                                      -12-
<PAGE>   13
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]

LWI RETAIL, INC.

By: /s/ Edward J. O'Brien
    ---------------------------
Title: Vice President
       ------------------------


SCANDIA DOWN CORPORATION

By: /s/ Edward J. O'Brien
    ---------------------------
Title: Vice President
       ------------------------


TWEEDS OF VERMONT, INC.

By: /s/ Edward J. O'Brien
    ---------------------------
Title: Vice President
       ------------------------


YORK FULFILLMENT COMPANY, INC.

By: /s/ Edward J. O'Brien
    ---------------------------
Title: Vice President
       ------------------------


AUSTAD HOLDINGS, INC.

By: /s/ Edward J. O'Brien
    ---------------------------
Title: Vice President
       ------------------------




                                      -13-