<PAGE>
                                                                EXHIBIT 10.4
                                                                [Execution]

            TWENTY-SECOND AMENDMENT TO LOAN AND SECURITY AGREEMENT
            ------------------------------------------------------

        THIS TWENTY-SECOND AMENDMENT TO LOAN AND SECURITY AGREEMENT (this
"Amendment"), dated as of August 16, 2002, is entered into by and among
CONGRESS FINANCIAL CORPORATION, a Delaware corporation ("Lender"), HANOVER
DIRECT PENNSYLVANIA, INC., a Pennsylvania corporation ("HDPI"), BRAWN OF
CALIFORNIA, INC., a California corporation ("Brawn"), GUMP'S BY MAIL, INC., a
Delaware corporation ("GBM"), GUMP'S CORP., a California corporation
("Gump's"), HANOVER DIRECT VIRGINIA INC., a Delaware corporation ("HDV"),
HANOVER REALTY, INC., a Virginia corporation ("Hanover Realty"), THE COMPANY
STORE FACTORY, INC., a Delaware corporation ("TCS Factory"), THE COMPANY
OFFICE, INC., a Delaware corporation ("TCS Office"), SILHOUETTES, LLC, a
Delaware limited liability company ("Silhouettes LLC"), HANOVER COMPANY STORE,
LLC, a Delaware limited liability company ("HCS LLC"), DOMESTICATIONS, LLC, a
Delaware limited liability company ("Domestications LLC"), and KEYSTONE
INTERNET SERVICES, INC., a Delaware corporation ("Keystone Internet"; and
together with HDPI, Brawn, GBM, Gump's, HDV, Hanover Realty, TCS Factory, TCS
Office, Silhouettes LLC, HCS LLC and Domestications LLC, collectively, the
"Borrowers" and each individually, a "Borrower" ), and HANOVER DIRECT, INC., a
Delaware corporation ("Hanover"), AMERICAN DOWN & TEXTILE COMPANY, a Wisconsin
corporation ("American Down"), D.M. ADVERTISING, INC., a New Jersey corporation
("DM Advertising, Inc."), KEYSTONE LIQUIDATIONS, INC., a Delaware corporation,
formerly known as Tweeds of Vermont, Inc. ("Keystone Liquidations"), HANOVER
HOME FASHIONS GROUP, LLC, a Delaware limited liability company ("HHFG LLC"),
ENCORE CATALOG, LLC, a Delaware limited liability company ("Encore LLC"),
CLEARANCE WORLD OUTLETS, LLC, a Delaware limited liability company ("Clearance
World"), SCANDIA DOWN, LLC, a Delaware limited liability company ("Scandia
Down, LLC"), ERIZON, INC., a Delaware corporation ("erizon, inc."), HANOVER
BRANDS, INC., a Delaware corporation ("Hanover Brands"), LACROSSE FULFILLMENT,
LLC, a Delaware limited liability company ("LaCrosse, LLC") , SAN DIEGO
TELEMARKETING, LLC, a Delaware limited liability company ("San Diego LLC"), and
D.M. ADVERTISING, LLC, a Delaware limited liability company ("DM Advertising,
LLC"; and together with Hanover, American Down, DM Advertising, Inc., Keystone
Liquidations, HHFG LLC, Encore LLC, Clearance World, Scandia Down, LLC, erizon,
inc., Hanover Brands, LaCrosse, LLC and San Diego LLC, collectively, the
"Guarantors" and each individually, a "Guarantor").

                             W I T N E S S E T H:
                             --------------------

        WHEREAS, Borrowers, Guarantors and Lender are parties to the Loan and
Security Agreement, dated November 14, 1995, as amended by First Amendment to
Loan and Security Agreement, dated February 22, 1996, Second Amendment to Loan
and Security Agreement, dated April 16, 1996, Third Amendment to Loan and
Security Agreement, dated May 24, 1996, Fourth Amendment to Loan and Security
Agreement, dated May 31, 1996, Fifth Amendment to Loan and Security Agreement,
dated September 11, 1996, Sixth Amendment to Loan and Security Agreement, dated
as of December 5, 1996, Seventh Amendment to Loan and Security Agreement, dated
as of December 18, 1996, Eighth Amendment to Loan and Security Agreement, dated
as of March 26, 1997, Ninth Amendment to Loan and Security Agreement, dated as
of April 18, 1997, Tenth Amendment to Loan and Security Agreement, dated as of
October 31, 1997, Eleventh Amendment to Loan and Security Agreement, dated as
of March 25, 1998, Twelfth Amendment to Loan and Security Agreement, dated as
of September 30, 1998, Thirteenth Amendment to Loan and Security Agreement,
dated as of September 30, 1998, Fourteenth


<PAGE>

Amendment to Loan and Security Agreement, dated as of February 28, 2000,
Fifteenth Amendment to Loan and Security Agreement, dated as of March 24, 2000,
Sixteenth Amendment to Loan and Security Agreement, dated as of August 8, 2000,
Seventeenth Amendment to Loan and Security Agreement, dated as of January 5,
2001, Eighteenth Amendment to Loan and Security Agreement, dated as of November
12, 2001, Nineteenth Amendment to Loan and Security Agreement, dated as of
December 18, 2001, Twentieth Amendment to Loan and Security Agreement, dated as
of March 5, 2002, and Twenty-First Amendment to Loan and Security Agreement,
dated as of March 21, 2002 (as so amended, the "Loan Agreement"), pursuant to
which Lender has made loans and advances to Borrowers;

        WHEREAS, Borrowers and Guarantors have requested that Lender, among
other things,  (a) agree to make an additional term loan to Tranche B Term Loan
Borrowers, consolidate the principal amount of that term loan with the
outstanding principal balance of the existing Tranche B Term Loan and amend and
restate the terms of the Tranche B Term Loan as so consolidated, and (b) amend
certain provisions of the Loan Agreement related to the foregoing; and

        WHEREAS, the parties to the Loan Agreement desire to enter into this
Amendment to evidence and effectuate such agreements and amendments, and
certain other amendments to the Financing Agreements relating thereto, in each
case subject to the terms and conditions and to the extent set forth herein;

        NOW, THEREFORE, in consideration of the premises and covenants set
forth herein and other good and valuable consideration, the receipt and
sufficiency of which is hereby acknowledged, the parties hereto agree as
follows:

        1.  Definitions.
            -----------

                (a)  Additional Definitions.  As used herein, the following
terms shall have the meanings given to them below, and the Loan Agreement and
each of the other Financing Agreements is hereby amended to include, in
addition and not in limitation, the following definitions:

                "Restated Tranche B Term Note" shall mean the Amended and
Restated Tranche B Term Promissory Note, dated as of the date of the
Twenty-Second Amendment, made by Tranche B Term Loan Borrowers payable to the
order of Lender in the original principal amount of $8,410,714.41, as such note
now exists or may hereafter be amended, modified, supplemented, extended,
renewed, restated or replaced.

                "Twenty-Second Amendment" shall mean the Twenty-Second
Amendment to Loan and Security Agreement, dated as of August 16, 2002, by and
among Lender, Borrowers and Guarantors, as the same now exists or may hereafter
be amended, modified, supplemented, extended, renewed, restated or replaced.

                (b)  Amendment to Definitions.

                        (i)  Consolidated Working Capital.  Section 1.23 of the
Loan Agreement shall be deemed deleted in its entirety and replaced with the
following effective as of June 29, 2002:

                "1.23  'Consolidated Working Capital' shall mean as to any
Person, at any time, in accordance with generally accepted accounting
principles as in effect from time to time, consistently applied, on a
consolidated basis for such Person and its Subsidiaries, the amount equal to
the difference between: (a) the aggregate net book value of all assets of such
Person and its Subsidiaries, on a consolidated basis, which would, in
accordance with generally accepted



<PAGE>

accounting principles as in effect from time to time, consistently applied, be
classified as current assets, calculating the book value of Inventory for this
purpose on a first-in-first-out basis, and (b) all Indebtedness (including, for
this purpose, notwithstanding clause (c) of the definition of Indebtedness,
trade accounts payable incurred in the ordinary course of business whether
current or any number of days past due) of such Person and its Subsidiaries, on
a consolidated basis, which would in accordance with generally accepted
accounting principles as in effect from time to time, consistently applied, be
classified as current liabilities; provided, however, that solely for purposes
of calculating Consolidated Working Capital hereunder, the outstanding balance
of the Revolving Loans, the Tranche A Term Loans and the Tranche B Term Loan
shall not be considered current liabilities; provided, further, however, that,
to the extent that Hanover and Borrowers shall have received any cash proceeds
of loans in respect of any subordinated indebtedness under the Richemont Credit
Facilities, other than proceeds of loans under the Richemont $10,000,000 Credit
Agreements, to the extent permitted hereunder, then, solely for the purposes of
calculating Consolidated Working Capital hereunder, the outstanding principal
amount of such loans shall not be considered current liabilities of Borrowers."

                        (ii)  Mortgages.  Section 1.86 of the Loan Agreement
shall be deemed deleted in its entirety and replaced with the following:

                "1.86  'Mortgages' shall mean, individually and collectively,
each of the following (as the same now exist or may hereafter be amended,
modified, supplemented, extended, renewed, restated or replaced): (a) the Deed
of Trust, Assignment and Security Agreement, dated as of November 14, 1995, by
Hanover Realty in favor of Lender with respect to the Real Property and related
assets of Hanover Realty in Roanoke, Virginia, as amended by Amendment No. 1 to
Deed of Trust, Assignment and Security Agreement, dated as of June 26, 1998,
and Amendment No. 2 to Deed of Trust, Assignment and Security Agreement, dated
as of March 24, 2000, and Amendment No. 3 to Deed of Trust, Assignment and
Security Agreement, dated as of the date of the Twenty-Second Amendment, (b)
the Mortgage and Security Agreement, dated as of September 30, 1998, by TCS
Factory in favor of Lender with respect to the Real Property and related assets
of TCS Factory located at 2929 Airport Road, La Crosse, Wisconsin, as amended
by the Mortgage Modification Agreement, dated as of March 24, 2000, and the
Second Mortgage Modification Agreement, dated as of the date of the
Twenty-Second Amendment, and (c) the Mortgage and Security Agreement, dated as
of September 30, 1998, by TCS Office in favor of Lender with respect to the
Real Property and related assets of TCS Office located at 455 Park Plaza Drive,
La Crosse, Wisconsin, as amended by the Mortgage Modification Agreement, dated
as of March 24, 2000, and the Second Mortgage Modification Agreement, dated as
of the date of the Twenty-Second Amendment."

                        (iii)  Tranche B Term Loan.  All references to the term
"Tranche B Term Loan" herein and in the Loan Agreement and the other Financing
Agreements shall be deemed and each such reference is hereby amended to mean
the Obligations evidenced by the Restated Tranche B Term Note.

                        (iv)  Tranche B Term Loan Borrowers.  All references to
the term "Tranche B Term Loan Borrowers" herein and in the Loan Agreement and
the other Financing Agreements shall be deemed and each such reference is
hereby amended to mean, collectively, HDPI, Brawn, GBM, Gump's, HDV, Hanover
Realty, TCS Factory, TCS Office, Silhouettes LLC, HCS LLC, Domestications LLC
and Keystone Internet.

                (c)  Interpretation.  All capitalized terms used herein and not
defined herein shall have the meanings given to such terms in the Loan
Agreement.



<PAGE>


2.  Restated Tranche B Term Loan.
    ----------------------------

                (a)  Tranche B Term Loan Borrowers hereby acknowledge, confirm
and agree that as of the date hereof and immediately before giving effect to
this Amendment, Tranche B Term Loan Borrowers are indebted to Lender for the
Obligations evidenced by the Tranche B Term Note in the principal amount of
$4,910,714.41 (the "Existing Tranche B Term Loan Balance"), plus accrued fees
and interest thereon.  On the date hereof, subject to the terms and conditions
contained herein and in the other Financing Agreements, Lender shall make an
additional term loan to Tranche B Term Loan Borrowers in the amount of
$3,500,000, which, together with the Existing Tranche B Term Loan Balance,
shall be consolidated and evidenced by and be due and payable pursuant to the
terms of the Restated Tranche B Term Note.

                (b)  The Restated Tranche B Term Note shall (i) supersede,
amend and restate the Tranche B Term Note in its entirety and (ii) be secured
by all of the Collateral.

                (c)  All Obligations in respect of the Tranche B Term Loan
shall be due and payable upon the earlier of the termination of the Loan
Agreement and the other Financing Agreements or January 31, 2004.

                (d)  The amendment and restatement contained herein and in the
Restated Tranche B Term Note, shall not, in any manner, be construed to
constitute payment of, or impair, limit, cancel or extinguish, or constitute a
novation in respect of, any of the Obligations evidenced by or arising under
the Financing Agreements, and the liens and security interests securing such
Obligations shall not in any manner be impaired, limited, terminated, waived or
released.

        3.  Fees.  In addition to all other fees, charges, interest and
expenses payable by Borrowers to Lender under the Loan Agreement and the other
Financing Agreements, Borrowers shall pay to Lender the following additional
fees:

                (a)  Amendment Fee.  Borrowers shall pay to Lender,
contemporaneously herewith, an amendment fee in the amount of $35,000, which
fee shall be fully earned and nonrefundable as of the date hereof and may be
charged into the loan account of any Borrower.

                (b)  Tranche B Closing Fee.  Tranche B Term Loan Borrowers
shall pay to Lender, contemporaneously herewith, a closing fee in the amount of
$375,000, which fee shall be fully earned and nonrefundable as of the date
hereof and may be charged into the loan account of any Tranche B Term Loan
Borrower.

                (c)     Tranche B Servicing Fee.  Tranche B Term Loan Borrowers
shall pay to Lender a monthly servicing fee in respect of the Tranche B Term
Loan in the amount of $2,000 for each month or part thereof during the Term of
the Loan Agreement, which fee shall be fully earned and payable on the first
day of each month during the Term and for so long thereafter as any of the
Obligations are outstanding and may be charged into the loan account of any
Tranche B Term Loan Borrower.

        4.  Use of Proceeds.  Tranche B Term Loan Borrowers shall use the
initial proceeds of the Tranche B Term Loan provided by Lender hereunder only
for: (a) general operating, working capital and other proper corporate purposes
of Tranche B Term Loan Borrowers not otherwise prohibited by the terms of the
Loan Agreement, and (b) costs, expenses and fees in connection with the
preparation, negotiation, execution and delivery of this Amendment and the
other Financing Agreements.

<PAGE>

5.  Representations, Warranties and Covenants.  Each Borrower and Guarantor
represents, warrants and covenants with and to Lender as follows, which
representations, warranties and covenants are continuing and shall survive the
execution and delivery hereof, the truth and accuracy of, or compliance with
each, together with the representations, warranties and covenants in the other
Financing Agreements, being a condition of the effectiveness of this Amendment
and a continuing condition of the making or providing of any Revolving Loans or
Letter of Credit Accommodations by Lender to Borrowers:

                (a)  This Amendment and each other agreement or instrument to
be executed and delivered by the Borrowers or Guarantors hereunder have been
duly authorized, executed and delivered by all necessary action on the part of
such Borrowers and Guarantors which are parties hereto and thereto and, if
necessary, their respective stockholders (with respect to any corporation) or
members (with respect to any limited liability company), and is in full force
and effect as of the date hereof, as the case may be, and the agreements and
obligations of Borrowers and Guarantors, as the case may be, contained herein
and therein constitute legal, valid and binding obligations of Borrowers and
Guarantors, as the case may be, enforceable against them in accordance with
their terms.

                (b)  Neither the execution and delivery of this Amendment or
any of the agreements, documents or instruments to be delivered pursuant to
this Agreement  has violated or shall violate any Federal or State securities
laws or any other law or regulation or any order or decree of any court or
governmental instrumentality in any respect applicable to Borrowers or
Guarantors, or  does or shall conflict with or result in the breach of, or
constitute a default in any respect under any mortgage, deed of trust, security
agreement, agreement or instrument to which Borrowers or Guarantors is a party
or may be bound, or  shall violate any provision of the Certificates of
Incorporation, Certificates of Formation, By-Laws or Operating Agreements, as
applicable, of Borrowers or Guarantors.

                (c)  No action of, or filing with, or consent of any
governmental or public body or authority, other than the recording of the
modification agreements with respect to the Mortgages executed and delivered to
Lender pursuant to this Amendment, and no approval or consent of any other
party, is required to authorize, or is otherwise required in connection with,
the execution, delivery and performance of this Amendment and each other
agreement or instrument to be executed and delivered pursuant to this
Amendment.

                (d)  All of the representations and warranties set forth in the
Loan Agreement as amended hereby, and the other Financing Agreements, are true
and correct in all material respects after giving effect to the provisions of
this Amendment, except to the extent any such representation or warranty is
made as of a specified date, in which case such representation or warranty
shall have been true and correct as of such date.

                (e)  After giving effect to the provisions of this Amendment,
no Event of Default or Incipient Default exists or has occurred and is
continuing.

        6.  Conditions Precedent.  Concurrently with the execution and delivery
hereof (except to the extent otherwise indicated below), and as a further
condition to the effectiveness of this Amendment and the agreement of Lender to
the modifications and amendments set forth in this Amendment:

                (a)  Lender shall have received executed originals of the
following documents:

                        (i)  this Amendment, duly authorized, executed and
delivered by each Borrower and Guarantor;

<PAGE>


                        (ii)  the Restated Tranche B Term Note, duly
authorized, executed and delivered by Tranche B Term Loan Borrowers in favor of
Lender;

                        (iii)  the Amendment No. 3 to the Deed of Trust,
Assignment and Security Agreement, duly authorized, executed and delivered by
Hanover Realty;

                        (iv)  the Second Mortgage Modification Agreement, duly
authorized, executed and delivered by TCS Factory; and

                        (v)  the Second Mortgage Modification Agreement, duly
authorized, executed and delivered by TCS Office;

                (b)  Lender shall have received, in form and substance
satisfactory to Lender, updated endorsements to the existing title insurance
policy or a new title insurance policy issued by Lawyers Title Insurance
Corporation acceptable to Lender  insuring the priority and amount of each of
the Mortgages made by Hanover Realty, TCS Factory and TCS Office, as amended,
and  containing any legally available endorsements, assurance or affirmative
coverage requested by Lender for protection of its interests;

                (c)  Lender shall have received, in form and substance
satisfactory to Lender, for each Borrower and Guarantor that is a corporation,
a Secretary's or Assistant Secretary's Certificate of Directors' Resolutions,
Incumbency and Shareholders' Consent (other than in respect of Hanover)
evidencing the adoption and subsistence of corporate resolutions approving the
execution, delivery and performance by such Borrower or Guarantor of this
Amendment and the agreements, documents and instruments to be delivered
pursuant to this Amendment;

                (d)  Lender shall have received, in form and substance
satisfactory to Lender, for each Borrower and Guarantor that is a limited
liability company  a Management and Incumbency Certificate of each such company
identifying all managers, officers or other persons authorized to act on behalf
of such company,  Company Resolutions of each such company, evidencing the
adoption and subsistence of company resolutions approving the execution,
delivery and performance by each Borrower and Guarantor that is a limited
liability company of this Amendment and the agreements, documents and
instruments to be delivered pursuant to this Amendment, in each case signed by
all members of each such company, and  Certificates of the Secretary or
Assistant Secretary of each such company identifying all members of such
company;

                (e)  Lender shall have received the written consent from each
Participant with respect to the transactions contemplated by this Amendment,
each duly authorized, executed and delivered by such Participant;

                (f)  Lender shall have received, in form and substance
satisfactory to Lender, an opinion of counsel to Borrowers and Guarantors with
respect to the transactions contemplated by this Amendment and such other
matters as Lender shall reasonably request, addressed to Lender, in form and
substance and satisfactory to Lender; and

                (g)  Lender shall have received, in form and substance
satisfactory to Lender, a true and correct copy of any consent, waiver or
approval to or of this Amendment, which any Borrower or Guarantor is required
to obtain from any other Person, and such consent, approval or waiver shall be
in a form reasonably acceptable to Lender.

<PAGE>

        7.  Effect of this Amendment. This Amendment constitutes the entire
agreement of the parties with respect to the subject matter hereof, and
supersedes all prior oral or written communications, memoranda, proposals,
negotiations, discussions, term sheets and commitments with respect to the
subject matter hereof. Except as expressly provided herein, no other changes or
modifications to the Loan Agreement or any of the other Financing Agreements,
or waivers of or consents under any provisions of any of the foregoing, are
intended or implied by this Amendment, and in all other respects the Financing
Agreements are hereby specifically ratified, restated and confirmed by all
parties hereto as of the effective date hereof. To the extent that any
provision of the Loan Agreement or any of the other Financing Agreements
conflicts with any provision of this Amendment, the provision of this Amendment
shall control.

        8.  Further Assurances.  Borrowers and Guarantors shall execute and
deliver such additional documents and take such additional action as may be
reasonably requested by Lender to effectuate the provisions and purposes of
this Amendment.

        9.  Governing Law.  The validity, interpretation and enforcement of
this Amendment in any dispute arising out of the relationship between the
parties hereto, whether in contract, tort, equity or otherwise shall be
governed by the internal laws of the State of New York, without regard to any
principle of conflict of laws or other rule of law that would result in the
application of the law of any jurisdiction other than the State of New York.

        10.  Binding Effect.  This Amendment shall be binding upon and inure to
the benefit of each of the parties hereto and their respective successors and
assigns.

        11.  Counterparts.  This Amendment may be executed in any number of
counterparts, but all of such counterparts shall together constitute but one
and the same agreement. In making proof of this Amendment, it shall not be
necessary to produce or account for more than one counterpart thereof signed by
each of the parties hereto.

<PAGE>

        IN WITNESS WHEREOF, the parties hereto have caused this Amendment to be
executed on the day and year first written.


                                CONGRESS FINANCIAL CORPORATION

                                By:   /s/ Janet S. Last
                                  ----------------------------

                                Title:  SVP
                                     -------------------------

                                HANOVER DIRECT PENNSYLVANIA,INC.
                                BRAWN OF CALIFORNIA, INC.
                                GUMP'S BY MAIL, INC.
                                GUMP'S CORP.
                                HANOVER DIRECT VIRGINIA INC.
                                HANOVER REALTY, INC.
                                THE COMPANY STORE FACTORY, INC.
                                THE COMPANY OFFICE, INC.
                                SILHOUETTES, LLC
                                HANOVER COMPANY STORE, LLC
                                DOMESTICATIONS, LLC
                                KEYSTONE INTERNET SERVICES, INC.

                                By:  /s/ Edward M. Lambert
                                     ----------------------
                                Title:  CFO
                                     ---------------------

By their signatures below, the
undersigned Guarantors acknowledge
and agree to be bound by the
applicable provisions of this
Amendment:

HANOVER DIRECT, INC.

By:   /s/ Edward M. Lambert
     ----------------------------

Title:  CFO
     ----------------------------

[SIGNATURES CONTINUE ON NEXT PAGE]

<PAGE>



[SIGNATURES CONTINUED FROM PREVIOUS PAGE]


AMERICAN DOWN & TEXTILE COMPANY
DM ADVERTISING, INC.
KEYSTONE LIQUIDATIONS, INC.
HANOVER HOME FASHIONS GROUP, LLC
ENCORE CATALOG, LLC
CLEARANCE WORLD OUTLETS, LLC
SCANDIA DOWN, LLC
ERIZON, INC.
HANOVER BRANDS, INC.
LA CROSSE FULFILLMENT, LLC
SAN DIEGO TELEMARKETING, LLC
D.M. ADVERTISING, LLC

By:  /s/ Edward M. Lambert
    --------------------------

Title:  CFO
     -------------------------