<PAGE>   1
                                                                   EXHIBIT 10.27


                 NINTH AMENDMENT TO LOAN AND SECURITY AGREEMENT


                  NINTH AMENDMENT TO LOAN AND SECURITY AGREEMENT, dated as of
April 18, 1997, by and among CONGRESS FINANCIAL CORPORATION, a California
corporation ("Lender"), HANOVER DIRECT PENNSYLVANIA, INC., a Pennsylvania
corporation ("HDPI"), BRAWN OF CALIFORNIA, INC., a California corporation
("Brawn"), GUMP'S BY MAIL, INC., a Delaware corporation ("GBM"), GUMP'S CORP., a
California corporation ("Gump's"), THE COMPANY STORE, INC., a Wisconsin
corporation ("TCSI"), TWEEDS, INC., a Delaware corporation ("Tweeds"), LWI
HOLDINGS, INC., a Delaware corporation ("LWI"), AEGIS CATALOG CORPORATION, a
Delaware corporation ("Aegis"), HANOVER DIRECT VIRGINIA INC., a Delaware
corporation ("HDV"), HANOVER REALTY, INC., a Virginia corporation ("Hanover
Realty"), and THE AUSTAD COMPANY, a South Dakota corporation ("Austad"; and
together with HDPI, Brawn, GBM, Gump's, TCSI, Tweeds, LWI, Aegis, HDV and
Hanover Realty, each individually referred to herein as a "Borrower" and
collectively, "Borrowers") and HANOVER DIRECT, INC., a Delaware corporation
("Hanover"), AEGIS RETAIL CORPORATION, a Delaware corporation, AEGIS SAFETY
HOLDINGS, INC., a Delaware corporation, AEGIS VENTURES, INC., a Delaware
corporation, AMERICAN DOWN & TEXTILE COMPANY, a Wisconsin corporation, BRAWN
WHOLESALE CORP., a California corporation, THE COMPANY FACTORY, INC., a
Wisconsin corporation, THE COMPANY OFFICE, INC., a Wisconsin corporation,
COMPANY STORE HOLDINGS, INC., a Delaware corporation, D.M. ADVERTISING, INC., a
New Jersey corporation, GUMP'S CATALOG, INC., a Delaware corporation, GUMP'S
HOLDINGS, INC., a Delaware corporation, HANOVER CASUALS, INC., a Delaware
corporation, HANOVER CATALOG HOLDINGS, INC., a Delaware corporation, HANOVER
FINANCE CORPORATION, a Delaware corporation, HANOVER LIST MANAGEMENT INC., a New
Jersey corporation, HANOVER VENTURES, INC., a Delaware corporation, LEICHTUNG OF
MICHIGAN, INC., a Michigan corporation, LWI RETAIL, INC., an Ohio corporation,
SCANDIA DOWN CORPORATION, a Delaware corporation, TWEEDS OF VERMONT, INC., a
Delaware corporation, YORK FULFILLMENT COMPANY, INC., a Pennsylvania
corporation, and AUSTAD HOLDINGS, INC., a Delaware corporation (each
individually a "Guarantor" and collectively, "Guarantors").


                              W I T N E S S E T H:


                  WHEREAS, Borrowers, Guarantors and Lender entered into the
Loan and Security Agreement, dated November 14, 1995, as amended by the First
Amendment to Loan and Security Agreement, dated February 22, 1996, the Second
Amendment to Loan and Security Agreement, dated April 16, 1996, the Third
Amendment to Loan and Security Agreement, dated May 24, 1996, the Fourth
Amendment to Loan and Security Agreement, dated May 31, 1996, the
<PAGE>   2
Fifth Amendment to Loan and Security Agreement, dated September 11, 1996, the
Sixth Amendment to Loan and Security Agreement, dated as of December 5, 1996,
the Seventh Amendment to Loan and Security Agreement, dated as of December 18,
1996, and the Eighth Amendment to Loan and Security Agreement, dated as of March
26, 1997 (the "Loan Agreement"), pursuant to which Lender has made loans and
advances to Borrowers; and

                  WHEREAS, Borrowers and Guarantors have requested that Lender
enter into certain amendments to the Loan Agreement and agreements in connection
with the making of a loan by Richemont to Hanover in the original principal
amount of up to $30,000,000; and

                  WHEREAS, the parties to the Loan Agreement desire to enter
into this Ninth Amendment to Loan and Security Agreement (this "Amendment") to
evidence and effectuate such amendments and agreements, subject to the terms and
conditions set forth herein;

                  NOW, THEREFORE, in consideration of the premises and covenants
set forth herein and other good and valuable consideration, the receipt and
sufficiency of which is hereby acknowledged, the parties hereto agree as
follows:

                  1.       Definitions.

                           (a) Additional Definitions. As used herein or in any
of the other Financing Agreements, the following terms shall have the meanings
given to them below, and the Loan Agreement shall be deemed and is hereby
amended to include, in addition and not in limitation, the following
definitions:

                                (i) "$30,000,000 Richemont Loan" shall mean the
loan made on or about the date hereof by Richemont to Hanover in the original
principal amount of up to $30,000,000.

                                (ii) "$30,000,000 Richemont Note" shall mean the
Promissory Note, dated as of the date hereof, by Hanover payable to Richemont in
the original principal amount of up to $30,000,000 delivered to evidence the
$30,000,000 Richemont Loan plus interest thereon.

                           (b) Amendment to Definition. Section 1.23 of the Loan
Agreement is hereby amended by deleting the proviso appearing at the end thereof
and substituting the following proviso therefor, effective as of the date
hereof:


                                       -2-
<PAGE>   3
                                    "; provided, however, that solely for
                           purposes of calculating Consolidated Working Capital
                           hereunder, the outstanding balance of the Revolving
                           Loans and the Term Loans and the outstanding balance
                           of the $10,000,000 IMR Note and the outstanding
                           balance of the $30,000,000 Richemont Note shall not
                           be considered current liabilities."

                           (c) Interpretation. For purposes of this Amendment,
unless otherwise defined herein, all capitalized terms used herein that are
defined in the Loan Agreement, shall have the respective meanings given to such
terms in the Loan Agreement.

                  2. Use of Proceeds and Repayment of Richemont Loan.

                           (a) Hanover agrees to use all of the proceeds of the
$30,000,000 Richemont Loan solely to make cash capital contributions or
intercompany cash advances to Revolving Loan Borrowers and Revolving Loan
Borrowers agree to use the proceeds thereof from Hanover solely for working
capital or for other proper corporate purposes of Revolving Loan Borrowers not
otherwise prohibited by the terms of the Loan Agreement and the other Financing
Agreements. Hanover and Revolving Loan Borrowers shall arrange for the
disbursement of the $30,000,000 Richemont Loan and the corresponding cash
capital contributions or intercompany cash advances by Hanover to Revolving Loan
Borrowers, by wire transfer(s) directly from Richemont to the Blocked Account(s)
designated by Lender, for credit to the applicable Revolving Loan Borrowers'
Revolving Loan accounts maintained by Lender.

                           (b) Hanover agrees and acknowledges that Richemont's
rights to payment of the $30,000,000 Richemont Loan and certain other related
indebtedness and obligations shall be subordinated in favor of Lender and may
only be paid by Hanover and received by Richemont as and to the extent permitted
by the terms of the subordination agreement delivered pursuant to Section
4(b)(ii) hereof.

                           (c) Hanover and the other Guarantors and Borrowers
hereby acknowledge and confirm that nothing contained herein or in Sections 6.5
and 6.6 of the Loan Agreement shall be construed or interpreted to permit
Revolving Loan Borrowers (i) to repay any intercompany cash advances made by
Hanover to Revolving Loan Borrowers with the proceeds of the $30,000,000
Richemont Loan for the purpose of repaying the indebtedness of Hanover to
Richemont or (ii) to declare or pay any dividend on account of any capital stock
of Revolving Loan Borrowers evidencing any cash capital contributions by Hanover
with the proceeds of the $30,000,000 Richemont Loan or to redeem, retire,


                                       -3-
<PAGE>   4
defease, purchase or otherwise acquire for value any shares of such capital
stock of Revolving Loan Borrowers.

                  3. Representations Warranties and Covenants. Borrowers
represent, warrant and covenant with and to Lender as follows, which
representations, warranties and covenants are continuing and shall survive the
execution and delivery hereof, the truth and accuracy of, or compliance with
each, together with the representations, warranties and covenants in the other
Financing Agreements, being a condition of the effectiveness of this Amendment
and a continuing condition of the making or providing of any Revolving Loans or
Letter of Credit Accommodations by Lender to Borrowers:

                           (a) This Amendment has been duly authorized, executed
and delivered by all necessary action of each of the Borrowers and Guarantors
which is a party hereto, and is in full force and effect, and the agreements and
obligations of Borrowers and Guarantors, as the case may be, contained herein
constitute legal, valid and binding obligations of Borrowers and Guarantors, as
the case may be, enforceable against them in accordance with their terms.

                           (b) Neither the execution and delivery of the
$30,000,000 Richemont Note, the making of the cash capital contributions or
intercompany cash loans by Hanover to Revolving Loan Borrowers contemplated by
Section 2 hereof, or any other agreements, documents or instruments in
connection therewith, nor the consummation of the transactions therein
contemplated, nor compliance with the provisions thereof (i) has violated or
shall violate any Federal or State securities laws or any other law or
regulation or any order or decree of any court or governmental instrumentality
in any respect, or (ii) does, or shall conflict with or result in the breach of,
or constitute a default in any respect under any mortgage, deed of trust,
security agreement, agreement or instrument to which Hanover or any other
Guarantor or any Borrower is a party or may be bound, or (iii) does or shall
violate any provision of the Certificate of Incorporation or By-Laws of Hanover
of any other Guarantor or any Borrower.

                           (c) All of the representations and warranties set
forth in the Loan Agreement as amended hereby, and the other Financing
Agreements, are true and correct in all material respects after giving effect to
the provisions of this Amendment, except to the extent any such representation
or warranty is made as of a specified date, in which case such representation or
warranty shall have been true and correct as of such date.

                           (d) After giving effect to the provisions of this
Amendment, no Event of Default or Incipient Default exists or has occurred and
is continuing.


                                       -4-
<PAGE>   5
                  4. Conditions Precedent. Concurrently with the execution
hereof, and as a condition to the effectiveness of this Amendment and the
agreement of Lender to the modifications, waivers and amendments set forth in
this Amendment:

                           (a) Lender shall have received an original of this
Amendment, in form and substance satisfactory to Lender, duly authorized,
executed and delivered by Borrowers and Guarantors; and

                           (b) Lender shall have received, each in form and
substance satisfactory to Lender,

                                (i) a true and complete copy of the $30,000,000
Richemont Note and all agreements, documents and instruments relating thereto,
and

                                (ii) a written subordination agreement, dated as
of the date hereof, between Richemont and Lender, as acknowledged by Hanover,
pursuant to which, among other things, Richemont shall have subordinated its
right to payment under the $30,000,000 Richemont Note to the prior indefeasible
payment of all of the Obligations, to the extent provided therein, duly
authorized, executed and delivered by Hanover and Richemont, together with an
opinion of Luxembourg counsel to Richemont addressed to Lender with respect to
the due authorization, execution and validity and enforceability of such
subordination agreement, and as to such other matters as Lender shall reasonably
require; and

                                (iii) true and complete copies of all
agreements, documents instruments evidencing the cash capital contributions or
intercompany cash advances by Hanover to Revolving Loan Borrowers contemplated
by Section 2 hereof.

                  5. Effect of this Amendment. This Amendment constitutes the
entire agreement of the parties with respect to the subject matter hereof, and
supersedes all prior oral or written communications, memoranda, proposals,
negotiations, discussions, term sheets and commitments with respect to the
subject matter hereof. Except as expressly provided herein, no other changes or
modifications to the Loan Agreement or any of the other Financing Agreements, or
waivers of or consents under any provisions of any of the foregoing, are
intended or implied by this Amendment, and in all other respects the Financing
Agreements are hereby specifically ratified, restated and confirmed by all
parties hereto as of the effective date hereof. To the extent that any provision
of the Loan Agreement or any of the other Financing Agreements conflicts with
any provision of this Amendment, the provision of this Amendment shall control.


                                       -5-
<PAGE>   6
                  6. Further Assurances. Borrowers and Guarantors shall execute
and deliver such additional documents and take such additional action as may be
reasonably requested by Lender to effectuate the provisions and purposes of this
Amendment.

                  7. Governing Law. The rights and obligations hereunder of each
of the parties hereto shall be governed by and interpreted and determined in
accordance with the internal laws of the State of New York (without giving
effect to principles of conflicts of laws).

                  8. Binding Effect. This Amendment shall be binding upon and
inure to the benefit of each of the parties hereto and their respective
successors and assigns.

                  9. Counterparts. This Amendment may be executed in any number
of counterparts, but all of such counterparts shall together constitute but one
and the same agreement. In making proof of this Amendment, it shall not be
necessary to produce or account for more than one counterpart thereof signed by
each of the parties hereto.

                  IN WITNESS WHEREOF, the parties hereto have caused this
Amendment to be executed on the day and year first written.

                                               CONGRESS FINANCIAL CORPORATION

                                               By:/S/ JANET S. LAST
                                                  -------------------------

                                               Title: Vice President
                                                     ----------------------

                                               HANOVER DIRECT PENNSYLVANIA, INC.


                                               By: /s/ Edward J. O'Brien
                                                  -------------------------

                                               Title: Vice President
                                                     ----------------------

                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                       -6-
<PAGE>   7
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                                  BRAWN OF CALIFORNIA, INC.

                                                      /s/ Edward J. O'Brien
                                                  By:_________________________

                                                          Vice President
                                                  Title:______________________


                                                  GUMP'S BY MAIL, INC.

                                                      /s/ Edward J. O'Brien
                                                  By:_________________________

                                                          Vice President
                                                  Title:______________________


                                                  GUMP'S CORP.

                                                      /s/ Edward J. O'Brien
                                                  By:_________________________

                                                          Vice President
                                                  Title:______________________


                                                  THE COMPANY STORE, INC.

                                                      /s/ Edward J. O'Brien
                                                  By:_________________________

                                                          Vice President
                                                  Title:______________________


                                                  TWEEDS, INC.

                                                      /s/ Edward J. O'Brien
                                                  By:_________________________

                                                          Vice President
                                                  Title:______________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                       -7-
<PAGE>   8
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


                                                   LWI HOLDINGS, INC.

                                                       /s/ Edward J. O'Brien
                                                   By:_________________________

                                                           Vice President
                                                   Title:______________________


                                                   AEGIS CATALOG CORPORATION

                                                       /s/ Edward J. O'Brien
                                                   By:_________________________

                                                           Vice President
                                                   Title:______________________


                                                   HANOVER DIRECT VIRGINIA INC.

                                                       /s/ Edward J. O'Brien
                                                   By:_________________________

                                                           Vice President
                                                   Title:______________________


                                                   HANOVER REALTY, INC.

                                                       /s/ Edward J. O'Brien
                                                   By:_________________________

                                                           Vice President
                                                   Title:______________________


                                                   THE AUSTAD COMPANY

                                                       /s/ Edward J. O'Brien
                                                   By:_________________________

                                                           Vice President
                                                   Title:______________________


By their signatures below, the undersigned
Guarantors acknowledge and agree to be bound
by the applicable provisions of this
Amendment:

HANOVER DIRECT, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Senior Vice President
Title:_________________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                       -8-
<PAGE>   9
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]

AEGIS RETAIL CORPORATION

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


AEGIS SAFETY HOLDINGS, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


AEGIS VENTURES, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


AMERICAN DOWN & TEXTILE COMPANY

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


BRAWN WHOLESALE CORP.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


THE COMPANY FACTORY, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


THE COMPANY OFFICE, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                       -9-
<PAGE>   10
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


COMPANY STORE HOLDINGS, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


D.M. ADVERTISING, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


GUMP'S CATALOG, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


GUMP'S HOLDINGS, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


HANOVER CASUALS, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


HANOVER CATALOG HOLDINGS, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


HANOVER FINANCE CORPORATION

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________

                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                      -10-
<PAGE>   11
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


HANOVER LIST MANAGEMENT, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


HANOVER VENTURES, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


LEICHTUNG OF MICHIGAN, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


LWI RETAIL, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


SCANDIA DOWN CORPORATION

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


TWEEDS OF VERMONT, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


                       [SIGNATURES CONTINUE ON NEXT PAGE]


                                      -11-
<PAGE>   12
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


YORK FULFILLMENT COMPANY, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________


AUSTAD HOLDINGS, INC.

    /s/ Edward J. O'Brien
By:____________________________

        Vice President
Title:_________________________



                                      -12-