<PAGE>   1
                                                                Exhibit 10.21

                 THIRD AMENDMENT TO LOAN AND SECURITY AGREEMENT


                  THIRD AMENDMENT TO LOAN AND SECURITY AGREEMENT, dated as of
May __, 1996, by and among CONGRESS FINANCIAL CORPORATION, a California
corporation ("Lender"), HANOVER DIRECT PENNSYLVANIA, INC., a Pennsylvania
corporation ("HDPI"), BRAWN OF CALIFORNIA, INC., a California corporation
("Brawn"), GUMP'S BY MAIL, INC., a Delaware Corporation ("GBM"), GUMP'S CORP., a
California corporation ("Gump's"), THE COMPANY STORE, INC., a Wisconsin
corporation ("TCSI"), TWEEDS, INC., a Delaware corporation ("Tweeds"), LWI
HOLDINGS, INC., a Delaware Corporation ("LWI"), AEGIS CATALOG CORPORATION, a
Delaware corporation ("Aegis"), HANOVER DIRECT VIRGINIA INC., a Delaware
corporation ("HDV"), HANOVER REALTY, INC., a Virginia corporation ("Hanover
Realty"), and THE AUSTAD COMPANY, a South Dakota corporation ("Austad"; and
together with HDPI, Brawn, GBM, Gump's, TCSI, Tweeds, LWI, Aegis, HDV and
Hanover Realty, each individually referred to herein as a "Borrower" and
collectively, "Borrowers") and HANOVER DIRECT, INC., a Delaware corporation
("Hanover"), AEGIS RETAIL CORPORATION, a Delaware corporation, AEGIS SAFETY
HOLDINGS, INC., a Delaware corporation ("Aegis Holding"), AEGIS VENTURES, INC.,
a Delaware corporation, AMERICAN DOWN & TEXTILE COMPANY, a Wisconsin
corporation, BRAWN WHOLESALE CORP., a California corporation, THE COMPANY
FACTORY, INC., a Wisconsin corporation, THE COMPANY OFFICE, INC., a Wisconsin
corporation, COMPANY STORE HOLDINGS, INC., a Delaware corporation, D.M.
ADVERTISING, INC., a New Jersey corporation, GUMP'S CATALOG, INC., a Delaware
corporation, GUMP'S HOLDINGS, INC., a Delaware corporation, HANOVER CASUALS,
INC., a Delaware corporation, HANOVER CATALOG HOLDINGS, INC., a Delaware
corporation, HANOVER FINANCE CORPORATION, a Delaware corporation, HANOVER LIST
MANAGEMENT INC., a New Jersey corporation, HANOVER VENTURES, INC., a Delaware
corporation, LEICHTUNG OF MICHIGAN, INC., a Michigan corporation, LWI RETAIL,
INC., an Ohio corporation, SCANDIA DOWN CORPORATION, a Delaware corporation,
TWEEDS OF VERMONT, INC., a Delaware corporation, YORK FULFILLMENT COMPANY, INC.,
a Pennsylvania corporation, and AUSTAD HOLDINGS, INC., a Delaware corporation
(each individually a "Guarantor" and collectively, "Guarantors").


                              W I T N E S S E T H:


                  WHEREAS, Borrowers, Guarantors and Lender entered into the
Loan and Security Agreement, dated November 14, 1995, as amended by the First
Amendment to Loan and Security Agreement, dated February 22, 1996, and the
Second Amendment to Loan and Security Agreement, dated April 16, 1996 (the "Loan
Agreement"), pursuant to which Lender has made loans and advances to Borrowers;
and

                  WHEREAS, Borrowers and Guarantors have requested that Lender
enter into certain amendments to the Loan Agreement and agreements in connection
with the making of a loan by Quadrant
<PAGE>   2
Management, Inc. to Hanover of up to $25,000,000; and

                  WHEREAS, the parties to the Loan Agreement desire to enter
into this Third Amendment to Loan and Security Agreement (this "Amendment") to
evidence and effectuate the foregoing, to the extent set forth herein, and
subject to the terms and conditions set forth herein;

                  NOW, THEREFORE, in consideration of the premises and covenants
set forth herein and other good and valuable consideration, the receipt and
sufficiency of which is hereby acknowledged, the parties hereto agree as
follows:

                  1.       Definitions.

                           (a) Additional Definitions. As used herein or in any
of the other Financing Agreements, the following terms shall have the respective
meanings given to them below, and the Loan Agreement shall be deemed and is
hereby amended to include, in addition and not in limitation, each of the
following definitions:

                                    (i) "Quadrant Management" shall mean
Quadrant Management, Inc., a Delaware corporation, and its successors and
assigns.

                                    (ii) "$25,000,000 Subordinated Note" shall
mean the Subordinated Promissory Note, dated of even date herewith, by Hanover
payable to Quadrant Management in the original principal amount of up to
$25,000,000.

                           (b) Amendments to Definitions.

                                    (i) Section 1.22 of the Loan Agreement is
hereby deleted in its entirety and replaced with the following:

                                    "1.22 "Consolidated Net Worth" shall mean,
                           as to any Person, at any time, in accordance with
                           generally accepted accounting principles, as in
                           effect from time to time consistently applied, on a
                           consolidated basis for such Person and its
                           Subsidiaries, the amount equal to the result obtained
                           by taking total assets and subtracting therefrom
                           total liabilities of such Person and its
                           Subsidiaries; provided, however, (a) that solely for
                           purposes of calculating Consolidated Net Worth of
                           Hanover and its Subsidiaries as at the end of
                           Hanover's 1995 fiscal year, up to $4,500,000 in
                           write-downs of Hanover's deferred taxes asset, as
                           required pursuant to Financial Accounting Standards
                           No. 109, due to losses incurred by Hanover and its
                           Subsidiaries in such fiscal year, shall not be
                           considered reductions of the deferred taxes asset of
                           Hanover and (b) that for purposes

                                       -2-
<PAGE>   3
                           of calculating Consolidated Net Worth of Hanover and
                           its Subsidiaries, the outstanding Indebtedness for
                           Borrowed Money evidenced by the $25,000,000
                           Subordinated Note shall not be considered a liability
                           of Hanover."

                                    (ii) Section 1.23 of the Loan Agreement is
hereby amended by deleting the proviso appearing at the end thereof and
substituting the following proviso therefor:

                           "; provided, however, that solely for purposes of
                           calculating Consolidated Working Capital hereunder,
                           the outstanding balance of the Revolving Loans and
                           Term Loans and the outstanding balance of the
                           $25,000,000 Subordinated Note shall not be considered
                           current liabilities."

                           (c) Interpretation. For purposes of this Amendment,
unless otherwise defined herein, all capitalized terms used herein that are
defined in the Loan Agreement, shall have the respective meanings given to such
terms in the Loan Agreement.

                  2. Proceeds of Subordinated Loan; Assignment. Hanover agrees
to use the proceeds of the loans evidenced by the $25,000,000 Subordinated Note
only for general operating, working capital of Borrowers and for other proper
corporate purposes of Hanover not otherwise prohibited by the terms of the
Financing Agreements. None of the proceeds of the loans evidenced by the
$25,000,000 Subordinated Note shall be used, directly or indirectly, for the
payment of any of the outstanding obligations of Hanover under the 9.25% Notes,
other than, after the assignment to NAR or a wholly-owned subsidiary of NAR the
$25,000,000 Subordinated Note, upon the application thereof to the obligations
of NAR or such subsidiary of NAR to purchase common stock of Hanover, in each
case, to the extent such assignment and application are permitted under the
subordination agreement referred to in Section 4(b) hereof. Following the
assignment of the $25,000,000 Subordinated Note to NAR or a wholly-owned
subsidiary of NAR as permitted under such subordination agreement, Hanover will
not borrow additional amounts under the $25,000,000 Subordinated Note unless and
until such assignee (which shall be the lender thereunder as to future loans
evidenced thereunder) shall have entered into the "NAR Agreement" as defined in
the subordination agreement referred to in Section 4(b) hereof, and delivered
the opinions and certified resolutions to be delivered to Lender pursuant
thereto, all in form and substance satisfactory to Lender.

                  3. Representations and Warranties. Borrowers represent,
warrant and covenant with and to Lender as follows, which representations,
warranties and covenants are continuing and shall survive the execution and
delivery hereof, the truth and accuracy of, or compliance with each, together
with the

                                       -3-
<PAGE>   4
representations, warranties and covenants in the other Financing Agreements,
being a condition of the effectiveness of this Amendment and a continuing
condition of the making or providing of any Revolving Loans or Letter of Credit
Accommodations by Lender to Borrowers:

                           (a)      This Amendment has been duly authorized,
executed and delivered by all necessary action of each of the Borrowers and
Guarantors which is a party hereto, and is in full force and effect, and the
agreements and obligations of Borrowers and Guarantors, as the case may be,
contained herein constitute legal, valid and binding obligations of Borrowers
and Guarantors, as the case may be, enforceable against them in accordance with
their terms.

                           (b)      Neither the execution and delivery of the
$25,000,000 Subordinated Note, nor the consummation of the transactions therein
contemplated, nor compliance with the provisions thereof, (i) has violated or
shall violate any Federal or State securities laws or any other law or
regulation or any order or decree of any court or governmental instrumentality
in any respect or (ii) does, or shall conflict with or result in the breach of,
or constitute a default in any respect under any mortgage, deed of trust,
security agreement, agreement or instruments to which Hanover or any other
Guarantor or any Borrower is a party or may be bound, or (iii) does or shall
violate any provision of the Certificate of Incorporation or ByLaws of Hanover
or any other Guarantor or any Borrower.

                           (c) All of the representations and warranties set
forth in the Loan Agreement as amended hereby, and the other Financing
Agreements, are true and correct in all material respects, except to the extent
any such representation or warranty is made as of a specified date, in which
case such representation or warranty shall have been true and correct as of such
date.

                           (d) After giving effect to the provisions of this
Amendment, no Event of Default or Incipient Default exists or has occurred and
is continuing.

                  4.       Conditions Precedent.  Concurrently with the
execution hereof, and as a further condition to the effectiveness
of this Amendment and the agreement of Lender to the modifica-
tions and amendments set forth in this Amendment:

                           (a)      Lender shall have received an original of
this Amendment, in form and substance satisfactory to Lender and
its counsel, duly authorized, executed and delivered by Borrowers
and Guarantors;

                           (b)      Lender shall have received, in form and
substance satisfactory to Lender, an original written
subordination agreement, dated of even date herewith, between

                                       -4-
<PAGE>   5
Quadrant Management and Lender, duly authorized, executed and delivered by
Quadrant Management, pursuant to which, among other things, Quadrant Management
shall have subordinated its right to payment under the $25,000,000 Subordinated
Note and related obligations to the right of Lender to receive the prior
indefeasible payment in full of all of the Obligations;

                           (c) Lender shall have received the Secretary's
Certificates of Directors' Resolutions evidencing the adoption and subsistence
of corporate resolutions approving the execution, delivery and performance by
Quadrant Management of the subordination agreement described in Section 4(b)
hereof, together with such opinions of counsel to Quadrant Management with
respect thereto, addressed to Lender, as Lender shall reasonably require, all in
form and substance satisfactory to Lender; and

                           (d) each of Borrowers and Guarantors shall deliver,
or cause to be delivered, to Lender a true and correct copy of any consent,
waiver or approval to or of this Amendment, which any Borrower or Guarantor is
required to obtain from any other Person, and such consent, approval or waiver
shall be in a form reasonably acceptable to Lender.

                  5. Effect of this Amendment. This Amendment constitutes the
entire agreement of the parties with respect to the subject matter hereof, and
supersedes all prior oral or written communications, memoranda, proposals,
negotiations, discussions, term sheets and commitments with respect to the
subject matter hereof. Except as expressly provided herein, no other changes or
modifications to the Loan Agreement or any of the other Financing Agreements, or
waivers of or consents under any provisions of any of the foregoing, are
intended or implied, and in all other respects the Financing Agreements are
hereby specifically ratified, restated and confirmed by all parties hereto as of
the effective date hereof. To the extent that any provision of the Loan
Agreement or any of the other Financing Agreements conflicts with any provision
of this Amendment, the provision of this Amendment shall control.

                           (a) Further Assurances. Borrowers and Guarantors
shall execute and deliver such additional documents and take such additional
action as may be reasonably requested by Lender to effectuate the provisions and
purposes of this Amendment.

                           (b) Governing Law. The rights and obligations
hereunder of each of the parties hereto shall be governed by and interpreted and
determined in accordance with the internal laws of the State of New York
(without giving effect to principles of conflicts of laws).

                  6. Binding Effect. This Amendment shall be binding upon and
inure to the benefit of each of the parties hereto and their respective
successors and assigns.

                                       -5-
<PAGE>   6
                  7. Counterparts. This Amendment may be executed in any number
of counterparts, but all of such counterparts shall together constitute but one
and the same agreement. In making proof of this Amendment, it shall not be
necessary to produce or account for more than one counterpart thereof signed by
each of the parties hereto.

                  IN WITNESS WHEREOF, the parties hereto have caused this
Amendment to be executed on the day and year first written.

                                         CONGRESS FINANCIAL CORPORATION

                                         By: /s/ Jennifer S. Last
                                             ------------------------

                                         Title: Vice President
                                                --------------------- 


                                         HANOVER DIRECT PENNSYLVANIA, INC.

                                         By: /s/ Edward J. O'Brien
                                             ------------------------

                                         Title: Vice President
                                                ---------------------


                                         BRAWN OF CALIFORNIA, INC.

                                         By: /s/ Edward J. O'Brien
                                             ------------------------

                                         Title: Vice President
                                                ---------------------


                                         GUMP'S BY MAIL, INC.

                                         By: /s/ Edward J. O'Brien
                                             ------------------------

                                         Title: Vice President
                                                ---------------------


                                         GUMP'S CORP.

                                         By: /s/ Edward J. O'Brien
                                             ------------------------

                                         Title: Vice President
                                                ---------------------


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                       -6-
<PAGE>   7
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                         THE COMPANY STORE, INC.

                                         By: /s/ Edward J. O'Brien
                                             ------------------------

                                         Title: Vice President
                                                ---------------------


                                         TWEEDS, INC.

                                         By: /s/ Edward J. O'Brien
                                             ------------------------

                                         Title: Vice President
                                                ---------------------


                                         LWI HOLDINGS, INC.

                                         By: /s/ Edward J. O'Brien
                                             ------------------------

                                         Title: Vice President
                                                ---------------------


                                         AEGIS CATALOG CORPORATION

                                         By: /s/ Edward J. O'Brien
                                             ------------------------

                                         Title: Vice President
                                                ---------------------


                                         HANOVER DIRECT VIRGINIA INC.

                                         By: /s/ Edward J. O'Brien
                                             ------------------------

                                         Title: Vice President
                                                ---------------------


                                         HANOVER REALTY, INC.

                                         By: /s/ Edward J. O'Brien
                                             ------------------------

                                         Title: Vice President
                                                ---------------------


                                         THE AUSTAD COMPANY

                                         By: /s/ Edward J. O'Brien
                                             ------------------------

                                         Title: Vice President
                                                ---------------------


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                       -7-
<PAGE>   8
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


By their signatures below, the undersigned Guarantors acknowledge and agree to
be bound by the applicable provisions of this
Amendment:

HANOVER DIRECT, INC.
 
By:  /s/ Edward J. O'Brien
    --------------------------

Title: Senior Vice President
       ------------------------  

AEGIS RETAIL CORPORATION

By:  /s/ Edward J. O'Brien
     --------------------------

Title: Vice President
       ------------------------

AEGIS SAFETY HOLDINGS, INC.

By:  /s/ Edward J. O'Brien
     --------------------------

Title: Vice President
       ------------------------


AEGIS VENTURES, INC.

By:  /s/ Edward J. O'Brien
     --------------------------

Title: Vice President
       ------------------------


AMERICAN DOWN & TEXTILE COMPANY

By:  /s/ Edward J. O'Brien
     --------------------------

Title: Vice President
       ------------------------


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                       -8-
<PAGE>   9
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


BRAWN WHOLESALE CORP.

By: /s/ Edward J. O'Brien
   ____________________________

Title:  Vice President
      _________________________


THE COMPANY FACTORY, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title:  Vice President
      _________________________


THE COMPANY OFFICE, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title:  Vice President
      _________________________


COMPANY STORE HOLDINGS, INC.

By: /s/ Edward J. O'Brien 
   ____________________________

Title:  Vice President
      _________________________


D.M. ADVERTISING, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title:  Vice President
      _________________________


GUMP'S CATALOG, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title:  Vice President
      _________________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                       -9-
<PAGE>   10
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


GUMP'S HOLDINGS, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title:  Vice President
      _________________________


HANOVER CASUALS, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title:  Vice President
      _________________________


HANOVER CATALOG HOLDINGS, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title:  Vice President
      _________________________


HANOVER FINANCE CORPORATION

By: /s/ Edward J. O'Brien
   ____________________________

Title:  Vice President
      _________________________


HANOVER LIST MANAGEMENT, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title:  Vice President
      _________________________


HANOVER VENTURES, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title:  Vice President
      _________________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                      -10-
<PAGE>   11
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


LEICHTUNG OF MICHIGAN, INC.

By: /s/ Edward J. O'Brien
    ___________________________

Title: Vice President
       ________________________


LMI RETAIL, INC.

By: /s/ Edward J. O'Brien
    ___________________________

Title: Vice President
       ________________________


SCANDIA DOWN CORPORATION

By: /s/ Edward J. O'Brien
    ___________________________

Title: Vice President
       ________________________


TWEEDS OF VERMONT, INC.

By: /s/ Edward J. O'Brien
    ___________________________

Title: Vice President
       ________________________


YORK FULFILLMENT COMPANY, INC.

By: /s/ Edward J. O'Brien
    ___________________________

Title: Vice President
       ________________________


AUSTAD HOLDINGS, INC.

By: /s/ Edward J. O'Brien
    ___________________________

Title: Vice President
       ________________________




                                      -11-