<PAGE>   1
                                                                Exhibit 10.20

                 SECOND AMENDMENT TO LOAN AND SECURITY AGREEMENT


                  SECOND AMENDMENT TO LOAN AND SECURITY AGREEMENT, dated as of
April __, 1996, by and among CONGRESS FINANCIAL CORPORATION, a California
corporation ("Lender"), HANOVER DIRECT PENNSYLVANIA, INC., a Pennsylvania
corporation ("HDPI"), BRAWN OF CALIFORNIA, INC., a California corporation
("Brawn"), GUMP'S BY MAIL, INC., a Delaware Corporation ("GBM"), GUMP'S CORP., a
California corporation ("Gump's"), THE COMPANY STORE, INC., a Wisconsin
corporation ("TCSI"), TWEEDS, INC., a Delaware corporation ("Tweeds"), LWI
HOLDINGS, INC., a Delaware Corporation ("LWI"), AEGIS CATALOG CORPORATION, a
Delaware corporation ("Aegis"), HANOVER DIRECT VIRGINIA INC., a Delaware
corporation ("HDV"), HANOVER REALTY, INC., a Virginia corporation ("Hanover
Realty"), and THE AUSTAD COMPANY, a South Dakota corporation ("Austad"; and
together with HDPI, Brawn, GBM, Gump's, TCSI, Tweeds, LWI, Aegis, HDV and
Hanover Realty, each individually referred to herein as a "Borrower" and
collectively, "Borrowers") and HANOVER DIRECT, INC., a Delaware corporation
("Hanover"), AEGIS RETAIL CORPORATION, a Delaware corporation, AEGIS SAFETY
HOLDINGS, INC., a Delaware corporation ("Aegis Holding"), AEGIS VENTURES, INC.,
a Delaware corporation, AMERICAN DOWN & TEXTILE COMPANY, a Wisconsin
corporation, BRAWN WHOLESALE CORP., a California corporation, THE COMPANY
FACTORY, INC., a Wisconsin corporation, THE COMPANY OFFICE, INC., a Wisconsin
corporation, COMPANY STORE HOLDINGS, INC., a Delaware corporation, D.M.
ADVERTISING, INC., a New Jersey corporation, GUMP'S CATALOG, INC., a Delaware
corporation, GUMP'S HOLDINGS, INC., a Delaware corporation, HANOVER CASUALS,
INC., a Delaware corporation, HANOVER CATALOG HOLDINGS, INC., a Delaware
corporation, HANOVER FINANCE CORPORATION, a Delaware corporation, HANOVER LIST
MANAGEMENT INC., a New Jersey corporation, HANOVER VENTURES, INC., a Delaware
corporation, LEICHTUNG OF MICHIGAN, INC., a Michigan corporation, LWI RETAIL,
INC., an Ohio corporation, SCANDIA DOWN CORPORATION, a Delaware corporation,
TWEEDS OF VERMONT, INC., a Delaware corporation, YORK FULFILLMENT COMPANY, INC.,
a Pennsylvania corporation, and AUSTAD HOLDINGS, INC., a Delaware corporation
(each individually a "Guarantor" and collectively, "Guarantors").


                              W I T N E S S E T H:


                  WHEREAS, Borrowers, Guarantors and Lender entered into the
Loan and Security Agreement, dated November 14, 1995, as amended by the First
Amendment to Loan and Security Agreement, dated February 22, 1996 (the "Loan
Agreement"), pursuant to which Lender has made loans and advances to Borrowers;
and

                  WHEREAS, Borrowers and Guarantors have requested that Lender
(a) provide temporary, supplemental revolving loans to HDPI of up to the maximum
amount of Four Million Dollars ($4,000,000) at any one time outstanding, (b)
reduce, on a temporary basis, the required maintenance levels under certain
financial covenants contained in the Loan Agreement, (c) release
<PAGE>   2
a portion of certain loan availability reserves previously established, and
establish a permanent availability reserve in the amount of One Million Dollars
($1,000,000), and (d) enter into certain related amendments to the Loan
Agreement and agreements in connection therewith;

                  WHEREAS, the parties to the Loan Agreement desire to enter
into this Second Amendment to Loan and Security Agreement (this "Amendment") to
evidence and effectuate the foregoing, to the extent set forth herein, and
subject to the terms and conditions set forth herein;

                  NOW, THEREFORE, in consideration of the premises and covenants
set forth herein and other good and valuable consideration, the receipt and
sufficiency of which is hereby acknowledged, the parties hereto agree as
follows:

                  1.       Definitions.

                           (a) Additional Definitions. As used herein or in any
of the other Financing Agreements, the following terms shall have the respective
meanings given to them below, and the Loan Agreement shall be deemed and is
hereby amended to include, in addition and not in limitation, each of the
following definitions:

                                    (i) "Supplemental Revolving Inventory Loans"
shall have the meaning given in Section 2(a) of this Amendment.

                                    (ii) "Hanover Rights Offering" shall mean
the proposed offering by Hanover of rights to purchase shares of common stock of
Hanover, for an aggregate gross issuance price of approximately $40,000,000, as
described in the press release, dated March 7, 1996, issued by Hanover, a copy
of which is annexed as Exhibit A hereto, and the consummation of the
transactions involving the exercise of such rights and the issuance of Hanover's
common stock in respect of such exercise, including the standby purchase of any
of such common stock by NAR.

                                    (iii) "Permanent Availability Reserve" shall
have the meaning given in Section 3(a) of this Amendment.

                                    (iv) "Temporary Loan Period" shall mean the
period commencing on the date hereof and ending on the earlier of (A) June 15,
1996 and (B) the first date of issuance of common stock of Hanover upon the
exercise of the rights under, and/or in the case of NAR, its standby purchase of
common stock of Hanover pursuant to, the Hanover Rights Offering (regardless of
the actual number of rights exercised or shares issued or the amounts received
by Hanover from such issuance or exercise).

                           (b) Interpretation. For purposes of this Amendment,
unless otherwise defined herein, all capitalized terms

                                       -2-
<PAGE>   3
used herein that are defined in the Loan Agreement, shall have the respective
meanings given to such terms in the Loan Agreement.

                  2. Supplemental Revolving Inventory Loan Availability.

                           (a)  Subject to the terms and conditions contained
herein and all of the terms and conditions of the Loan Agreement as amended
hereby, Lender agrees that, after giving effect to the adjustments to certain
availability reserves described in Section 3 hereof, to make available to HDPI
from time to time during the Temporary Loan Period and permit to remain
outstanding during the Temporary Loan Period, additional Revolving Inventory
Loans in the aggregate principal amount of Four Million Dollars ($4,000,000) at
any one time outstanding in excess of the aggregate amount of Revolving
Inventory Loans otherwise determined by Lender to be available to HDPI pursuant
to Section 2.1(b) of the Loan Agreement (the "Supplemental Revolving Inventory
Loans").

                           (b) The Supplemental Revolving Inventory Loans (i)
shall constitute part of and shall be deemed Revolving Inventory Loans, and as
such shall constitute Obligations of HDPI, for all purposes under the Loan
Agreement and the other Financing Agreements, except that during the Temporary
Loan Period the sublimit on Revolving Inventory Loans to HDPI contained in
Section 2.2(b) shall not be applicable to the Supplemental Revolving Inventory
Loans, (ii) shall be subject to (A) Lender's right to establish reserves against
availability of Revolving Loans and Letter of Credit Accommodations pursuant to
Section 2.6 of the Loan Agreement, and all the other terms and conditions set
forth herein and in the Loan Agreement and the other Financing Agreements,
except as expressly set forth in clause (i) of this Section , (iii) shall be
repaid on or prior to the expiration of the Temporary Loan Period in accordance
with the provisions of this Amendment, the Loan Agreement as amended hereby and
the other Financing Agreements, and (iv) shall be secured by all of the
Collateral.

                  3. Release of Part of Existing Availability Reserves;
Establishment of Permanent Availability Reserve.

                           (a) In place of Lender's continued maintenance of the
reserve against Revolving Loan Availability as provided in paragraph 2 of the
letter agreement re: Post-Closing Items, dated November 14, 1995, among Lender
and Borrowers, Lender shall release the remaining unreleased portion of such
reserve, except for $1,000,000 thereof, which shall be maintained at all times
hereafter by Lender as a permanent reserve against the Revolving Loan
availability of HDPI (the "Permanent Availability Reserve"), and the provisions
of such letter agreement providing for release of such previously established
availability reserve shall be of no further force and effect.


                                       -3-
<PAGE>   4
                           (b) The Permanent Availability Reserve shall be in
addition to, and not in limitation of, the rights of Lender to establish other
and further reserves against the availability of Revolving Loans and Letter of
Credit Accommodations under the Loan Agreement and the other Financing
Agreements.

                  4. Amendment Fee. In addition to all other fees, charges,
interest and expenses payable by Borrowers to Lender under the Loan Agreement
and the other Financing Agreements, HDPI shall pay to Lender a fee for entering
into this Amendment in the amount of Forty Thousand Dollars ($40,000), which
amount is fully earned and payable as of the date hereof and may be charged
directly to HDPI's loan account maintained by Lender in respect of the Revolving
Loans.

                  5. Consolidated Working Capital. Notwithstanding Section 6.19
of the Loan Agreement, as of the end of each fiscal month occurring during the
period commencing on the date hereof and ending on the last day of the Temporary
Loan Period, Hanover shall only be required to maintain Consolidated Working
Capital, calculated on a consolidated basis for Hanover and its Subsidiaries, of
not less than Twenty One Million Dollars ($21,000,000). As of the end of each
fiscal month ending after the last day of the Temporary Loan Period, Hanover
shall maintain Consolidated Working Capital as provided in Section 6.19 of the
Loan Agreement.

                  6. Consolidated Net Worth. Notwithstanding Section 6.20 of the
Loan Agreement, as of the end of each fiscal month occurring during the period
commencing on the date hereof and ending on the last day of the Temporary Loan
Period, Hanover shall only be required to maintain Consolidated Net Worth,
calculated on a consolidated basis for Hanover and its Subsidiaries, of at least
Seventy-Five Million Dollars ($75,000,000). As of the end of each fiscal month
ending after the last day of the Temporary Loan Period, Hanover shall maintain
Consolidated Net Worth as provided in Section 6.20 of the Loan Agreement.

                  7. Hanover Rights Offering. Upon receipt of proceeds of the
Hanover Rights Offering, net of commissions and expenses relating thereto,
Hanover shall use all such net proceeds, to the extent necessary to satisfy
fully the following requirement, to make a capital contribution or intercompany
advance (i) to HDPI to be used by HDPI to repay to Lender all Supplemental
Revolving Inventory Loans then outstanding, and (ii) to the Borrowers (including
HDPI), to the extent the outstanding Obligations (excluding the aggregate
outstanding principal amount of the Term Loans) exceeds the aggregate amount of
Revolving Loans determined by Lender pursuant to the lending formulas and
subject to the sublimits and reserves provided for or established pursuant to
the Loan Agreement as amended hereby, to be used by the respective Borrowers to
repay to Lender such excess, in each case under clauses (i) and (ii) before
using any such proceeds for any

                                       -4-
<PAGE>   5
other purpose, whether contemplated by the March 7, 1996 press release annexed
hereto as Exhibit A, or otherwise.

                  8. New Collateral Locations. For purposes of clarifying the
scope of Section 5.7(b) of the Loan Agreement, the movement of Inventory or
Equipment or other Collateral of a Borrower or Guarantor to a location which has
been disclosed on Exhibit C to the Loan Agreement as a location of Collateral of
that type of another Borrower or Guarantor, but not that particular Borrower or
Guarantor, shall be considered the opening of a new location, subject to the
prior notice and other requirements provided in or contemplated by Section
5.7(b). Concurrently herewith, Borrowers and Guarantors shall deliver an updated
Exhibit C to the Loan Agreement and shall execute or cause to be executed and/or
delivered such additional UCC financing statements and other agreements provided
in or contemplated by Section 5.7(b) or in connection with the mergers referred
to in Section 9 hereof, in each case as Lender shall require with respect to any
Collateral locations for particular Borrowers or Guarantors that were not
originally shown on Exhibit C to the Loan Agreement as Collateral locations as
to any type(s) of Collateral for those particular Borrowers or Guarantors.

                  9. Certain Mergers. Anything contained in Section 6.7 of the
Loan Agreement to the contrary notwithstanding, the mergers of certain
Guarantors as described in the footnotes appearing on the updated Exhibit C to
the Loan Agreement delivered pursuant to Section 8 of this Amendment are hereby
acknowledged and approved by Lender, Borrowers and Guarantors as of the
effective dates thereof.

                  10. Pledge of Note Payable to LWI. Upon execution and delivery
of the promissory note to be executed by Woodworkers Supply, Inc., payable to
LWI Holdings, Inc. in connection with the Asset Purchase Agreement, dated March
29, 1996, by and among LWI Holdings, Inc., Hanover and Woodworkers Supply, Inc.,
LWI Holdings, Inc. shall deliver to Lender, as pledgee pursuant to the Loan
Agreement, such note together with an allonge indorsement affixed to such note
providing for the payment of all amounts due thereunder to the order of Lender.

                  11. Representations and Warranties. Borrowers represent,
warrant and covenant with and to Lender as follows, which representations,
warranties and covenants are continuing and shall survive the execution and
delivery hereof, the truth and accuracy of, or compliance with each, together
with the representations, warranties and covenants in the other Financing
Agreements, being a condition of the effectiveness of this Amendment and a
continuing condition of the making or providing of any Revolving Loans or Letter
of Credit Accommodations by Lender to Borrowers:

                           (a) This Amendment has been duly authorized, executed
and delivered by all necessary action of each of the Borrowers and Guarantors
which is a party hereto, and is in full

                                       -5-
<PAGE>   6
force and effect, and the agreements and obligations of Borrowers and
Guarantors, as the case may be, contained herein constitute legal, valid and
binding obligations of Borrowers and Guarantors, as the case may be, enforceable
against them in accordance with their terms.

                           (b) All of the representations and warranties set
forth in the Loan Agreement as amended hereby, and the other Financing
Agreements, are true and correct in all material respects, and except to the
extent any such representation or warranty is made as of a specified date, in
which case such representation or warranty shall have been true and correct as
of such date.

                           (c) After giving effect to the provisions of this
Amendment, no Event of Default or Incipient Default exists or has occurred and
is continuing.

                  12. Conditions Precedent. Concurrently with the execution
hereof, and as a further condition to the effectiveness of this Amendment and
the agreement of Lender to the modifications and amendments set forth in this
Amendment:

                           (a) Lender shall have received an original of this
Amendment, in form and substance satisfactory to Lender and its counsel, duly
authorized, executed and delivered by Borrowers and Guarantors; and

                           (b) each of Borrowers and Guarantors shall deliver,
or cause to be delivered, to Lender a true and correct copy of any consent,
waiver or approval to or of this Amendment, which any Borrower or Guarantor is
required to obtain from any other Person, and such consent, approval or waiver
shall be in a form reasonably acceptable to Lender.

                  13. Effect of this Amendment. This Amendment constitutes the
entire agreement of the parties with respect to the subject matter hereof, and
supersedes all prior oral or written communications, memoranda, proposals,
negotiations, discussions, term sheets and commitments with respect to the
subject matter hereof. Except as expressly provided herein, no other changes or
modifications to the Loan Agreement and no changes or modifications to the
Subordination Agreement dated November 14, 1995 between IMR and Lender, or any
of the other Financing Agreements, or waivers of or consents under any
provisions of any of the foregoing, are intended or implied, and in all other
respects the Financing Agreements are hereby specifically ratified, restated and
confirmed by all parties hereto as of the effective date hereof. To the extent
that any provision of the Loan Agreement or any of the other Financing
Agreements conflicts with any provision of this Amendment, the provision of this
Amendment shall control.


                                       -6-
<PAGE>   7
                  14. Further Assurances. Borrowers and Guarantors shall execute
and deliver such additional documents and take such additional action as may be
reasonably requested by Lender to effectuate the provisions and purposes of this
Amendment.

                  15. Governing Law. The rights and obligations hereunder of
each of the parties hereto shall be governed by and interpreted and determined
in accordance with the internal laws of the State of New York (without giving
effect to principles of conflicts of laws).

                  16. Binding Effect. This Amendment shall be binding upon and
inure to the benefit of each of the parties hereto and their respective
successors and assigns.

                  17. Counterparts. This Amendment may be executed in any number
of counterparts, but all of such counterparts shall together constitute but one
and the same agreement. In making proof of this Amendment, it shall not be
necessary to produce or account for more than one counterpart thereof signed by
each of the parties hereto.

                  IN WITNESS WHEREOF, the parties hereto have caused this
Amendment to be executed on the day and year first written.

                                       CONGRESS FINANCIAL CORPORATION

                                       By: /s/ Janet S. Last
                                          -------------------------

                                       Title: Vice President
                                             ----------------------


                                       HANOVER DIRECT PENNSYLVANIA, INC.

                                       By: /s/ Wayne Garten
                                          -------------------------

                                       Title: /s/ Executive Vice President
                                             ----------------------

                                       BRAWN OF CALIFORNIA, INC.

                                       By: /s/ Wayne Garten
                                          -------------------------

                                       Title: /s/ Vice President
                                             ----------------------

                       [SIGNATURES CONTINUE ON NEXT PAGE]

                                       -7-
<PAGE>   8
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                       GUMP'S BY MAIL, INC.

                                       By: /s/ Wayne Garten
                                          -------------------------

                                       Title: Executive Vice President
                                             ----------------------


                                       GUMP'S CORP.

                                       By: /s/ Wayne Garten
                                          -------------------------

                                       Title: President
                                             ----------------------

                                       THE COMPANY STORE, INC.

                                       By: /s/ Wayne Garten
                                          -------------------------

                                       Title: Vice President
                                             ----------------------


                                       TWEEDS, INC.

                                       By: /s/ Wayne Garten
                                          -------------------------

                                       Title: Vice President
                                             ----------------------


                                       LWI HOLDINGS, INC.

                                       By: /s/ Wayne Garten
                                          -------------------------

                                       Title: President
                                             ----------------------

                                       AEGIS CATALOG CORPORATION

                                       By: /s/ Wayne Garten
                                          -------------------------

                                       Title: President
                                             ----------------------


                                       HANOVER DIRECT VIRGINIA INC.

                                       By: /s/ Wayne Garten
                                          -------------------------

                                       Title: President
                                             ----------------------


                       [SIGNATURES CONTINUE ON NEXT PAGE]

                                       -8-
<PAGE>   9
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                       HANOVER REALTY, INC.

                                       By: /s/ Wayne Garten
                                          -------------------------

                                       Title: President
                                             ----------------------

                                       THE AUSTAD COMPANY

                                       By: /s/ Wayne Garten
                                          -------------------------

                                       Title: Vice President
                                             ----------------------


By their signatures below, the undersigned Guarantors acknowledge and agree to
be bound by the applicable provisions of this Amendment:

HANOVER DIRECT, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title: Vice President
       ----------------------

AEGIS RETAIL CORPORATION

By: /s/ Edward J. O'Brien
    -------------------------

Title: Vice President
       ----------------------

AEGIS SAFETY HOLDINGS, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title: Vice President
       ----------------------

                       [SIGNATURES CONTINUE ON NEXT PAGE]

                                       -9-
<PAGE>   10
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


AEGIS VENTURES, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title: Vice President
       ----------------------


AMERICAN DOWN & TEXTILE COMPANY

By: /s/ Edward J. O'Brien
    -------------------------

Title: Vice President
       ----------------------


BRAWN WHOLESALE CORP.

By: /s/ Edward J. O'Brien
    -------------------------

Title: Vice President
       ----------------------


THE COMPANY FACTORY, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title: Vice President
       ----------------------


THE COMPANY OFFICE, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title: Vice President
       ----------------------

COMPANY STORE HOLDINGS, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title: Vice President
       ----------------------


                       [SIGNATURES CONTINUE ON NEXT PAGE]

                                      -10-
<PAGE>   11
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


D.M. ADVERTISING, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


GUMP'S CATALOG, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------

GUMP'S HOLDINGS, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


HANOVER CASUALS, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


HANOVER CATALOG HOLDINGS, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


HANOVER FINANCE CORPORATION

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


                       [SIGNATURES CONTINUE ON NEXT PAGE]

                                      -11-
<PAGE>   12
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


HANOVER LIST MANAGEMENT, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


HANOVER VENTURES, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


LEICHTUNG OF MICHIGAN, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


LWI RETAIL, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


SCANDIA DOWN CORPORATION

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


TWEEDS OF VERMONT, INC.

By: /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


                       [SIGNATURES CONTINUE ON NEXT PAGE]

                                      -12-
<PAGE>   13
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


YORK FULFILLMENT COMPANY, INC.

By:  /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------


AUSTAD HOLDINGS, INC.

By:  /s/ Edward J. O'Brien
    -------------------------

Title:  Vice President
       ----------------------



                                      -13-