<PAGE>   1
                                                                Exhibit 10.22

                 FOURTH AMENDMENT TO LOAN AND SECURITY AGREEMENT


                  FOURTH AMENDMENT TO LOAN AND SECURITY AGREEMENT, dated as of
May __, 1996, by and among CONGRESS FINANCIAL CORPORATION, a California
corporation ("Lender"), HANOVER DIRECT PENNSYLVANIA, INC., a Pennsylvania
corporation ("HDPI"), BRAWN OF CALIFORNIA, INC., a California corporation
("Brawn"), GUMP'S BY MAIL, INC., a Delaware Corporation ("GBM"), GUMP'S CORP., a
California corporation ("Gump's"), THE COMPANY STORE, INC., a Wisconsin
corporation ("TCSI"), TWEEDS, INC., a Delaware corporation ("Tweeds"), LWI
HOLDINGS, INC., a Delaware Corporation ("LWI"), AEGIS CATALOG CORPORATION, a
Delaware corporation ("Aegis"), HANOVER DIRECT VIRGINIA INC., a Delaware
corporation ("HDV"), HANOVER REALTY, INC., a Virginia corporation ("Hanover
Realty"), and THE AUSTAD COMPANY, a South Dakota corporation ("Austad"; and
together with HDPI, Brawn, GBM, Gump's, TCSI, Tweeds, LWI, Aegis, HDV and
Hanover Realty, each individually referred to herein as a "Borrower" and
collectively, "Borrowers") and HANOVER DIRECT, INC., a Delaware corporation
("Hanover"), AEGIS RETAIL CORPORATION, a Delaware corporation, AEGIS SAFETY
HOLDINGS, INC., a Delaware corporation ("Aegis Holding"), AEGIS VENTURES, INC.,
a Delaware corporation, AMERICAN DOWN & TEXTILE COMPANY, a Wisconsin
corporation, BRAWN WHOLESALE CORP., a California corporation, THE COMPANY
FACTORY, INC., a Wisconsin corporation, THE COMPANY OFFICE, INC., a Wisconsin
corporation, COMPANY STORE HOLDINGS, INC., a Delaware corporation, D.M.
ADVERTISING, INC., a New Jersey corporation, GUMP'S CATALOG, INC., a Delaware
corporation, GUMP'S HOLDINGS, INC., a Delaware corporation, HANOVER CASUALS,
INC., a Delaware corporation, HANOVER CATALOG HOLDINGS, INC., a Delaware
corporation, HANOVER FINANCE CORPORATION, a Delaware corporation, HANOVER LIST
MANAGEMENT INC., a New Jersey corporation, HANOVER VENTURES, INC., a Delaware
corporation, LEICHTUNG OF MICHIGAN, INC., a Michigan corporation, LWI RETAIL,
INC., an Ohio corporation, SCANDIA DOWN CORPORATION, a Delaware corporation,
TWEEDS OF VERMONT, INC., a Delaware corporation, YORK FULFILLMENT COMPANY, INC.,
a Pennsylvania corporation, and AUSTAD HOLDINGS, INC., a Delaware corporation
(each individually a "Guarantor" and collectively, "Guarantors").


                              W I T N E S S E T H:


                  WHEREAS, Borrowers, Guarantors and Lender entered into the
Loan and Security Agreement, dated November 14, 1995, as amended by the First
Amendment to Loan and Security Agreement, dated February 22, 1996, and the
Second Amendment to Loan and Security Agreement, dated April 16, 1996 (the "Loan
Agreement"), pursuant to which Lender has made loans and advances to Borrowers;
and

                  WHEREAS, Borrowers and Guarantors have requested that Lender
enter into certain amendments to the Loan Agreement and agreements in connection
with the making of a loan of up to $25,000,000 to Hanover by Intercontinental
Mining & Resources
<PAGE>   2
Incorporated ("IMR") to be used in part to repay the loan previously made by
Quadrant Management, Inc. to Hanover in the principal amount of $4,000,000; and

                  WHEREAS, the parties to the Loan Agreement desire to enter
into this Fourth Amendment to Loan and Security Agreement (this "Amendment") to
evidence and effectuate the foregoing, to the extent set forth herein, and
subject to the terms and conditions set forth herein;

                  NOW, THEREFORE, in consideration of the premises and covenants
set forth herein and other good and valuable consideration, the receipt and
sufficiency of which is hereby acknowledged, the parties hereto agree as
follows:

                  1.       Definitions.

                           (a) Additional Definition. As used herein or in any
of the other Financing Agreements, the following term shall have the meaning
given to it below, and the Loan Agreement shall be deemed and is hereby amended
to include, in addition and not in limitation, the following definition:

                                    "$25,000,000 IMR Subordinated Note" shall
mean the Subordinated Promissory Note, dated of even date herewith, by Hanover
payable to IMR in the original principal amount of up to $25,000,000.

                           (b) Amendments to Definitions.

                                    (i) Section 1.22 of the Loan Agreement is
hereby deleted in its entirety and replaced with the following, effective as of
the date hereof:

                                    "1.22 "Consolidated Net Worth" shall mean,
                           as to any Person, at any time, in accordance with
                           generally accepted accounting principles, as in
                           effect from time to time consistently applied, on a
                           consolidated basis for such Person and its
                           Subsidiaries, the amount equal to the result obtained
                           by taking total assets and subtracting therefrom
                           total liabilities of such Person and its
                           Subsidiaries; provided, however, (a) that solely for
                           purposes of calculating Consolidated Net Worth of
                           Hanover and its Subsidiaries as at the end of
                           Hanover's 1995 fiscal year, up to $4,500,000 in
                           write-downs of Hanover's deferred taxes asset, as
                           required pursuant to Financial Accounting Standards
                           No. 109, due to losses incurred by Hanover and its
                           Subsidiaries in such fiscal year, shall not be
                           considered reductions of the deferred taxes asset of
                           Hanover and (b) that for purposes of calculating
                           Consolidated Net Worth of Hanover and its
                           Subsidiaries, the outstanding Indebtedness for
                           Borrowed Money evidenced by the $25,000,000

                                       -2-
<PAGE>   3
                             IMR Subordinated Note shall not be considered a
                             liability of Hanover."

                                    (ii) Section 1.23 of the Loan Agreement is
hereby amended by deleting the proviso appearing at the end thereof and
substituting the following proviso therefor, effective as of the date hereof:

                           "; provided, however, that solely for purposes of
                           calculating Consolidated Working Capital hereunder,
                           the outstanding balance of the Revolving Loans and
                           Term Loans and the outstanding balance of the
                           $25,000,000 IMR Subordinated Note shall not be
                           considered current liabilities."

                           (c) Interpretation. For purposes of this Amendment,
unless otherwise defined herein, all capitalized terms used herein that are
defined in the Loan Agreement, shall have the respective meanings given to such
terms in the Loan Agreement.

                  2. Proceeds of IMR Subordinated Loan; Assignment. Hanover
agrees to use the proceeds of the loans evidenced by the $25,000,000 IMR
Subordinated Note only to repay in full the unpaid principal indebtedness
evidenced by the $25,000,000 Subordinated Note (i.e., the Subordinated
Promissory Note dated May 24, 1996 by Hanover payable to Quadrant Management in
the principal amount of up to $25,000,000), which note shall thereupon be
cancelled, and thereafter such loans shall be used only for general operating,
working capital of Borrowers and for other proper corporate purposes of Hanover
not otherwise prohibited by the terms of the Financing Agreements. None of the
proceeds of the loans evidenced by the $25,000,000 IMR Subordinated Note shall
be used, directly or indirectly, for the payment of any of the outstanding
obligations of Hanover under the 9.25% Notes, other than, after the assignment
to NAR or a wholly-owned subsidiary of NAR the $25,000,000 IMR Subordinated
Note, upon the application thereof to the obligations of NAR or such subsidiary
of NAR to purchase common stock of Hanover, in each case, to the extent such
assignment and application are permitted under the subordination agreement
referred to in Section 4(b) hereof. Following the assignment of the $25,000,000
IMR Subordinated Note to NAR or a wholly-owned subsidiary of NAR as permitted
under such subordination agreement, Hanover will not borrow additional amounts
under the $25,000,000 IMR Subordinated Note unless and until such assignee
(which shall be the lender thereunder as to future loans evidenced thereunder)
shall have entered into the "NAR Agreement" as defined in the subordination
agreement referred to in Section 4(b) hereof, and delivered the opinions and
certified resolutions to be delivered to Lender pursuant thereto, all in form
and substance satisfactory to Lender.

                  3. Representations and Warranties. Borrowers represent,
warrant and covenant with and to Lender as follows,

                                       -3-
<PAGE>   4
which representations, warranties and covenants are continuing and shall survive
the execution and delivery hereof, the truth and accuracy of, or compliance with
each, together with the representations, warranties and covenants in the other
Financing Agreements, being a condition of the effectiveness of this Amendment
and a continuing condition of the making or providing of any Revolving Loans or
Letter of Credit Accommodations by Lender to Borrowers:

                           (a) This Amendment has been duly authorized, executed
and delivered by all necessary action of each of the Borrowers and Guarantors
which is a party hereto, and is in full force and effect, and the agreements and
obligations of Borrowers and Guarantors, as the case may be, contained herein
constitute legal, valid and binding obligations of Borrowers and Guarantors, as
the case may be, enforceable against them in accordance with their terms.

                           (b) Neither the execution and delivery of the
$25,000,000 IMR Subordinated Note, nor the consummation of the transactions
therein contemplated, nor compliance with the provisions thereof, (i) has
violated or shall violate any Federal or State securities laws or any other law
or regulation or any order or decree of any court or governmental
instrumentality in any respect or (ii) does, or shall conflict with or result in
the breach of, or constitute a default in any respect under any mortgage, deed
of trust, security agreement, agreement or instruments to which Hanover or any
other Guarantor or any Borrower is a party or may be bound, or (iii) does or
shall violate any provision of the Certificate of Incorporation or ByLaws of
Hanover or any other Guarantor or any Borrower.

                           (c) All of the representations and warranties set
forth in the Loan Agreement as amended hereby, and the other Financing
Agreements, are true and correct in all material respects, except to the extent
any such representation or warranty is made as of a specified date, in which
case such representation or warranty shall have been true and correct as of such
date.

                           (d) After giving effect to the provisions of this
Amendment, no Event of Default or Incipient Default exists or has occurred and
is continuing.

                  4. Conditions Precedent. Concurrently with the execution
hereof, and as a further condition to the effectiveness of this Amendment and
the agreement of Lender to the modifications and amendments set forth in this
Amendment:

                           (a) Lender shall have received an original of this
Amendment, in form and substance satisfactory to Lender and its counsel, duly
authorized, executed and delivered by Borrowers and Guarantors;

                           (b) Lender shall have received, in form and

                                       -4-
<PAGE>   5
substance satisfactory to Lender, an original written subordination agreement,
dated of even date herewith, between IMR and Lender, duly authorized, executed
and delivered by IMR, pursuant to which, among other things, IMR shall have
subordinated its right to payment under the $25,000,000 IMR Subordinated Note
and related obligations to the right of Lender to receive the prior indefeasible
payment in full of all of the Obligations;

                           (c) Lender shall have received the Secretary's
Certificates of Directors' Resolutions evidencing the adoption and subsistence
of corporate resolutions approving the execution, delivery and performance by
IMR of the subordination agreement described in Section 4(b) hereof, together
with such opinions of counsel to IMR with respect thereto, addressed to Lender,
as Lender shall reasonably require, all in form and substance satisfactory to
Lender;

                           (d) All of the unpaid principal indebtedness
evidenced by the $25,000,000 Subordinated Note (i.e., the Subordinated
Promissory Note dated May 24, 1996, in the principal amount of up to
$25,000,000, by Hanover payable to Quadrant Management) shall have been paid in
full and such note shall have been marked "canceled", and a copy thereof
furnished to Lender, who hereby consents to such payment out of a portion of the
proceeds of the initial loans made by IMR and evidenced by the $25,000,000 IMR
Subordinated Note; and

                           (e) each of Borrowers and Guarantors shall deliver,
or cause to be delivered, to Lender a true and correct copy of any consent,
waiver or approval to or of this Amendment, which any Borrower or Guarantor is
required to obtain from any other Person, and such consent, approval or waiver
shall be in a form reasonably acceptable to Lender.

                  5. Effect of this Amendment. This Amendment constitutes the
entire agreement of the parties with respect to the subject matter hereof, and
supersedes all prior oral or written communications, memoranda, proposals,
negotiations, discussions, term sheets and commitments with respect to the
subject matter hereof. Except as expressly provided herein, no other changes or
modifications to the Loan Agreement or any of the other Financing Agreements, or
waivers of or consents under any provisions of any of the foregoing, are
intended or implied, and in all other respects the Financing Agreements are
hereby specifically ratified, restated and confirmed by all parties hereto as of
the effective date hereof. To the extent that any provision of the Loan
Agreement or any of the other Financing Agreements conflicts with any provision
of this Amendment, the provision of this Amendment shall control.

                           (a) Further Assurances. Borrowers and Guarantors
shall execute and deliver such additional documents and take such additional
action as may be reasonably requested by Lender to effectuate the provisions and
purposes of this Amendment.

                                       -5-
<PAGE>   6
                           (b) Governing Law. The rights and obligations
hereunder of each of the parties hereto shall be governed by and interpreted and
determined in accordance with the internal laws of the State of New York
(without giving effect to principles of conflicts of laws).

                  6. Binding Effect. This Amendment shall be binding upon and
inure to the benefit of each of the parties hereto and their respective
successors and assigns.

                  7. Counterparts. This Amendment may be executed in any number
of counterparts, but all of such counterparts shall together constitute but one
and the same agreement. In making proof of this Amendment, it shall not be
necessary to produce or account for more than one counterpart thereof signed by
each of the parties hereto.

                  IN WITNESS WHEREOF, the parties hereto have caused this
Amendment to be executed on the day and year first written.

                                         CONGRESS FINANCIAL CORPORATION

                                         By: /s/ Janet S. Last
                                            _________________________

                                         Title: Vice President
                                               ______________________


                                         HANOVER DIRECT PENNSYLVANIA, INC.

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title:   Vice President
                                               ______________________


                                         BRAWN OF CALIFORNIA, INC.

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title:  Vice President
                                               ______________________


                                         GUMP'S BY MAIL, INC.

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title:  Vice President
                                               ______________________


                                         GUMP'S CORP.

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title:  Vice President
                                               ______________________

                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                       -6-
<PAGE>   7
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                         THE COMPANY STORE, INC.

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title: Vice President
                                               ______________________


                                         TWEEDS, INC.

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title: Vice President
                                               ______________________


                                         LWI HOLDINGS, INC.

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title: Vice President
                                               ______________________


                                         AEGIS CATALOG CORPORATION

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title: Vice President
                                               ______________________


                                         HANOVER DIRECT VIRGINIA INC.

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title: Vice President
                                               ______________________


                                         HANOVER REALTY, INC.

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title: Vice President
                                               ______________________


                                         THE AUSTAD COMPANY

                                         By: /s/ Edward J. O'Brien
                                            _________________________

                                         Title: Vice President
                                               ______________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                       -7-
<PAGE>   8
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


By their signatures below, the undersigned Guarantors acknowledge and agree to
be bound by the applicable provisions of this
Amendment:

HANOVER DIRECT, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Senior Vice President
      _________________________


AEGIS RETAIL CORPORATION

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


AEGIS SAFETY HOLDINGS, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


AEGIS VENTURES, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


AMERICAN DOWN & TEXTILE COMPANY

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]

                                       -8-
<PAGE>   9
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


BRAWN WHOLESALE CORP.

By: /s/ Edward J. O'Brien
    ____________________________

Title: Vice President
       _________________________


THE COMPANY FACTORY, INC.

By: /s/ Edward J. O'Brien
    ____________________________

Title: Vice President
       _________________________


THE COMPANY OFFICE, INC.

By: /s/ Edward J. O'Brien
    ____________________________

Title: Vice President
       _________________________


COMPANY STORE HOLDINGS, INC.

By: /s/ Edward J. O'Brien
    ____________________________

Title: Vice President
       _________________________


D.M. ADVERTISING, INC.

By: /s/ Edward J. O'Brien
    ____________________________

Title: Vice President
       _________________________


GUMP'S CATALOG, INC.

By: /s/ Edward J. O'Brien
    ____________________________

Title: Vice President
       _________________________



                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                       -9-
<PAGE>   10
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


GUMP'S HOLDINGS, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


HANOVER CASUALS, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


HANOVER CATALOG HOLDINGS, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


HANOVER FINANCE CORPORATION

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


HANOVER LIST MANAGEMENT, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


HANOVER VENTURES, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                      -10-
<PAGE>   11
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


LEICHTUNG OF MICHIGAN, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


LWI RETAIL, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President 
      _________________________


SCANDIA DOWN CORPORATION

By: /s/ Edward J. O'Brien 
   ____________________________

Title: Vice President
      _________________________


TWEEDS OF VERMONT, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


YORK FULFILLMENT COMPANY, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________


AUSTAD HOLDINGS, INC.

By: /s/ Edward J. O'Brien
   ____________________________

Title: Vice President
      _________________________



                                      -11-