<PAGE>   1
                                                                    EXHIBIT 10.2
                                                                  Execution Copy


                             SIXTEENTH AMENDMENT TO
                           LOAN AND SECURITY AGREEMENT

             THIS SIXTEENTH AMENDMENT TO LOAN AND SECURITY AGREEMENT, dated as
of August 8, 2000, is entered into by and among CONGRESS FINANCIAL CORPORATION,
a Delaware corporation ("Lender"), HANOVER DIRECT PENNSYLVANIA, INC., a
Pennsylvania corporation ("HDPI"), BRAWN OF CALIFORNIA, INC., a California
corporation ("Brawn"), GUMP'S BY MAIL, INC., a Delaware corporation ("GBM"),
GUMP'S CORP., a California corporation ("Gump's"), LWI HOLDINGS, INC., a
Delaware corporation ("LWI"), HANOVER DIRECT VIRGINIA INC., a Delaware
corporation ("HDV"), HANOVER REALTY, INC., a Virginia corporation ("Hanover
Realty"), THE COMPANY STORE FACTORY, INC., a Delaware corporation ("TCS
Factory"), THE COMPANY OFFICE, INC., a Delaware corporation ("TCS Office"),
TWEEDS, LLC, a Delaware limited liability company ("Tweeds LLC"), SILHOUETTES,
LLC, a Delaware limited liability company ("Silhouettes LLC"), HANOVER COMPANY
STORE, LLC, a Delaware limited liability company ("HCS LLC"), DOMESTICATIONS,
LLC, a Delaware limited liability company ("Domestications LLC"; and together
with HDPI, Brawn, GBM, Gump's, LWI, HDV, Hanover Realty, TCS Factory, TCS
Office, Tweeds LLC, Silhouettes and HCS LLC, each individually referred to
herein as an "Existing Borrower" and collectively, as "Existing Borrowers"), and
HANOVER DIRECT, INC., a Delaware corporation, ("Hanover"), AMERICAN DOWN &
TEXTILE COMPANY, a Wisconsin corporation ("American Down"), D.M. ADVERTISING,
INC., a New Jersey corporation ("DM Advertising"), SCANDIA DOWN CORPORATION, a
Delaware corporation ("Scandia"), YORK FULFILLMENT COMPANY, INC., a Pennsylvania
corporation ("York Fulfillment"), KEYSTONE LIQUIDATIONS, INC., a Delaware
Corporation, formerly known as Tweeds of Vermont, Inc., HANOVER HOME FASHIONS
GROUP, LLC, a Delaware limited liability company ("HHFG LLC"), KITCHEN & HOME,
LLC, a Delaware limited liability company ("Kitchen & Home, LLC"),
DOMESTICATIONS KITCHEN & GARDEN, LLC, a Delaware limited liability company
("Domestications K&G, LLC"), ENCORE CATALOG, LLC, a Delaware limited liability
company ("Encore LLC"), CLEARANCE WORLD OUTLETS, LLC, a Delaware limited
liability company ("Clearance World"), SCANDIA DOWN, LLC, a Delaware limited
liability company ("Scandia Down, LLC"), ERIZON, INC., a Delaware corporation
("erizon, inc."), HANOVER BRANDS, INC., a Delaware corporation ("Hanover
Brands"), ERIZON.COM, INC., a Delaware corporation ("erizon.com"), LACROSSE
FULFILLMENT, LLC, a Delaware limited liability company ("LaCrosse, LLC"), SAN
DIEGO TELEMARKETING, LLC, a Delaware limited liability company ("San Diego LLC";
each individually a "Guarantor" and collectively "Guarantor" and KEYSTONE
INTERNET SERVICES, INC. ("Keystone Internet"). Each Existing Borrower, together
with Keystone Internet shall hereinafter be referred to individually as a
"Borrower" and collectively as "Borrowers".


<PAGE>   2


                              W I T N E S S E T H:

       WHEREAS, Existing Borrowers, Guarantors and Lender are parties to the
Loan and Security Agreement, dated November 14, 1995, as amended by the First
Amendment to Loan and Security Agreement, dated February 22, 1996, the Second
Amendment to Loan and Security Agreement, dated April 16, 1996, the Third
Amendment to Loan and Security Agreement, dated May 24, 1996, the Fourth
Amendment to Loan and Security Agreement, dated May 31, 1996, the Fifth
Amendment to Loan and Security Agreement, dated September 11, 1996, the Sixth
Amendment to Loan and Security Agreement, dated as of December 5, 1996, the
Seventh Amendment to Loan and Security Agreement, dated as of December 18, 1996,
the Eighth Amendment to Loan and Security Agreement, dated as of March 26, 1997,
the Ninth Amendment to Loan and Security Agreement, dated as of April 18, 1997,
the Tenth Amendment to Loan and Security Agreement, dated as of October 31,
1997, the Eleventh Amendment to Loan and Security Agreement, dated as of March
25, 1998, the Twelfth Amendment to Loan and Security Agreement, dated as of
September 30, 1998, the Thirteenth Amendment to Loan and Security Agreement,
dated as of September 30, 1998, the Fourteenth Amendment to Loan and Security
Agreement, dated as of February 28, 2000, and the Fifteenth Amendment to Loan
and Security Agreement, dated as of March 24, 2000 (as so amended, the "Loan
Agreement"), pursuant to which Lender has made loans and advances to Existing
Borrowers; and

       WHEREAS, Borrowers have requested that Lender consent to (i) a proposed
sale of preferred stock of Hanover in the amount of $70,000,000 and the use of a
portion of the proceeds in the amount of up to $35,000,000 of such offering to
reduce the credit limit under each of the Richemont Credit Facilities to zero
dollars and to terminate such facilities and (b) the amendment to the
Consolidated Working Capital and the Consolidated Net Worth covenants;

       WHEREAS, the parties to the Loan Agreement desire to enter into this
Sixteenth Amendment to Loan and Security Agreement (this "Amendment") to
evidence and effectuate such consents, amendments and agreements, and certain
other amendments to the Financing Agreements relating thereto, in each case
subject to the terms and conditions and to the extent set forth herein;

       NOW, THEREFORE, in consideration of the premises and covenants set forth
herein and other good and valuable consideration, the receipt and sufficiency of
which are hereby acknowledged, the parties hereto agree as follows:


<PAGE>   3



1.     Definitions.

(a)           Additional Definitions. As used herein or in any of the other
Financing Agreements, the following terms shall have the respective meanings
given to them below, and the Loan Agreement shall be deemed and is hereby
amended to include, in addition and not in limitation, each of the following
definitions:

(i)                  "Hanover Preferred Stock Offering" shall mean the proposed
sale by Hanover of approximately 1,400,000 shares of preferred stock of Hanover
to Richemont or its affiliates or any other Person for a purchase price in the
aggregate amount of $70,000,000 as described on the "Summary Terms for Preferred
Stock" attached as Annex I to the draft letter, dated August 7, 2000, by
Richemont Finance S.A. to Hanover.

(C)                  "Hanover Preferred Stock Offering Agreements" shall mean,
collectively (as the same now exist and as may hereafter be executed and/or
delivered and as may thereafter be amended, modified, supplemented, extended,
renewed, restated or replaced): a Certificate of the Designations, Powers,
Preferences and Rights of Series A Cumulative Participating Preferred Stock of
Hanover Direct, Inc. to be filed with the Delaware Secretary of State in
connection with the Hanover Preferred Stock Offering, the Securities Purchase
Agreement to be executed and delivered in connection with the closing of the
Hanover Preferred Stock Offering, substantially in the form of Annex II attached
to the draft letter, dated August 7, 2000, by Richemont Finance S.A. to Hanover,
and all related agreements, documents and instruments to be executed, delivered
or filed in connection with, or otherwise evidencing, the Hanover Preferred
Stock Offering.

(b)    Amendments to Definitions.

(i)                  Consolidated Net Worth. Effective as of April 28, 2000,
Section 1.22 of the Loan Agreement is hereby and shall be deemed amended by
adding an additional proviso at the end thereof as follows:

           "provided, further, however, that, to the extent that Hanover and
           Borrowers shall have received any cash proceeds of loans in respect
           of any subordinated indebtedness under the Richemont Credit
           Facilities, other than proceeds of loans under the Richemont
           $10,000,000 Credit Agreements, to the extent permitted hereunder,
           then, solely for the purposes of calculating Consolidated Net Worth
           of Hanover and its Subsidiaries, the outstanding principal amount of
           such loans shall be excluded from the total liabilities of
           Borrowers."

(ii)                 Consolidated Working Capital. Effective as of April 28,
2000, Section 1.23 of the Loan Agreement is hereby and shall be deemed amended
by adding an additional proviso the end thereof as follows:


           "provided, further, however, that, to the extent that Hanover and
           Borrowers shall have received any cash proceeds of loans in respect
           of any subordinated indebtedness under the Richemont Credit
           Facilities, other than proceeds of loans under the Richemont
           $10,000,000 Credit Agreements, to the extent permitted hereunder,
           then, solely for the purposes of calculating Consolidated Working
           Capital, the outstanding principal amount of such loans shall not be
           considered current liabilities of Borrowers."

(a)                  Interpretation. All capitalized terms used herein and not
defined herein shall have the meanings given to such terms in the Loan
Agreement.


<PAGE>   4

2.     Hanover Preferred Stock Offering.

(a)    Lender hereby consents to the issuance of the preferred stock of Hanover
pursuant to the Hanover Preferred Stock Offering, the reduction of the credit
limit under each of the Richemont Credit Facilities to zero dollars and the
termination of such facilities, so long as each of the following terms and
conditions has been satisfied as determined by Lender:

(i)           Hanover shall have delivered to Lender by no later than August 14,
2000, an executed agreement between Richemont and Hanover setting forth
Richemont's firm commitment to purchase 1,400,000 shares of preferred stock of
Hanover having an issuance price of $50 per share for an aggregate gross
purchase price of $70,000,000, less the amount of fees, commissions and expenses
payable under the terms of the Hanover Preferred Stock Offering pursuant to the
Hanover Preferred Stock Offering Agreements;

(ii)          Hanover shall have delivered to Lender evidence that Hanover shall
have received all of the net proceeds payable in connection with the Hanover
Preferred Stock Offering pursuant to the Hanover Preferred Stock Offering
Agreements, which amount shall not be less than $67,500,000, the amounts of
Indebtedness owed to Richemont that are convertible into equity in accordance
with the terms of the Richemont Credit Facilities and each of the Subordination
Agreements, dated as of March 24, 2000, between Lender and Richemont; provided,
that, the net proceeds received by Hanover in the form of cash or immediately
available funds shall not be less than $32,500,000;

(iii)         Hanover shall have delivered to Lender true, correct and complete
photocopies of all of the Hanover Preferred Stock Offering Agreements,
substantially in the form delivered to Lender prior to the date hereof, with
only those changes that do not adversely affect Lender's interests, and to the
extent required by Lender in its good faith judgment, Borrowers and Guarantors
shall execute and deliver to Lender such amendments to the Loan Agreement or any
of the other Financing Agreements in connection with the closing of the Hanover
Preferred Stock Offering; and

(iv)          the transactions contemplated by the Hanover Preferred Stock
Offering shall have occurred by no later than September 22, 2000, unless such
date is extended by Lender in writing.

(b)    In addition to and not in limitation of the conditions set forth in
Section 2(a) hereof, Borrowers and Guarantors hereby acknowledge, confirm and
agree that:

(i)           Lender shall establish a reserve against Revolving Loans that
would otherwise be available to Borrowers under the Loan Agreement in the amount
of $750,000. Such reserve against Revolving Loan availability shall be released
upon consummation of the Hanover Preferred Stock Offering pursuant to the terms
and conditions contained herein as determined by Lender in its good faith
judgment. The reserve against Revolving Loan availability shall be in addition
to, and not in limitation of, the rights of Lender to establish other and
further reserves against the availability of Revolving Loans and Letter of
Credit Accommodations under the Loan Agreement and the other Financing
Agreements.

(ii)          Upon receipt of the proceeds of the Hanover Preferred Stock
Offering, Hanover shall first use such net proceeds to the extent necessary to
make an intercompany advance to HDPI to be used by HDPI to repay to Lender all
Revolving Loans then outstanding, to the extent the outstanding Obligations
(excluding the aggregate outstanding


<PAGE>   5

principal amount of the Term Loans) exceed the aggregate amount of Revolving
Loans available as determined by Lender pursuant to the lending formulas and
subject to the sublimits and reserves provided for or established pursuant to
the Loan Agreement as amended hereby.

(iii)         In addition to and not in limitation of the conditions set forth
in Sections 2(a) and 2(b) hereof, Borrowers and Guarantors hereby acknowledge,
confirm and agree that, notwithstanding anything to the contrary that may be
contained in Sections 6.2, 6.5 and 6.6 of the Loan Agreement or in any other
provisions of the Loan Agreement or in any of the other Financing Agreements, or
in any of the Hanover Preferred Stock Offering Agreements, without the prior
written consent of Lender, Borrowers and Guarantors shall not, and shall not
permit any of their Subsidiaries, directly or indirectly, to make any loans,
advances, dividends, redemptions or other payments in respect of the preferred
stock of Hanover to be issued in connection with the Hanover Preferred Stock
Offering, whether in cash, property or otherwise; provided, that, Hanover may
make dividend payments in kind, but not cash, to the holders of the preferred
stock issued in respect of the Hanover Preferred Stock Offering, so long as each
of the following conditions shall have been satisfied as determined by Lender:
(i) the declaration and payment of such dividends shall be made out of legally
available funds therefor in accordance with applicable law, (ii) the declaration
and payment of such dividends shall not violate any law or any order or decree
of any court or other governmental authority and shall not conflict with or
result in the breach of, or constitute a default under, any indenture, mortgage,
deed of trust, or any other agreement to which any Borrower is a party or may be
bound, and (iii) as of the date of the declaration and payment of such dividends
and after giving effect thereto, no Event of Default or Incipient Default shall
exist or have occurred and be continuing. Hanover and Borrowers and the other
Guarantors hereby covenant with, to and in favor of Lender not to amend, modify
or supplement any of the Hanover Preferred Stock Offering Agreements without the
prior written consent of Lender.

3.     Additional Events of Default. Failure to satisfy the conditions set forth
in Section 2(a) and Section 2(b)(ii) hereof or to maintain the agreements and
covenants contained in Section 2(c) hereof shall, at Lender's option, constitute
an Event of Default.

4.     Representations, Warranties and Covenants. Borrowers and Guarantors
represent, warrant and covenant with and to Lender as follows, which
representations, warranties and covenants are continuing and shall survive the
execution and delivery hereof, the truth and accuracy of, or compliance with
each, together with the representations, warranties and covenants in the other
Financing Agreements, being a condition of the effectiveness of this Amendment
and a continuing condition of the making or providing of any Revolving Loans or
Letter of Credit Accommodations by Lender to Borrowers:

(a)           This Amendment and each other agreement or instrument to be
executed and delivered by each Borrower and/or Guarantor hereunder have been
duly authorized, executed and delivered by all necessary action on the part of
each of Borrower and each Guarantor which is a party hereto and thereto and, if
necessary, their respective stockholders (with respect to any corporation) or
members (with respect to any limited liability company), and is in full force
and effect as of the date hereof, as the case may be, and the agreements and
obligations of each Borrower and/or Guarantor, as the case may be, contained
herein and therein constitute legal, valid and binding obligations of each
Borrower and/or Guarantor, as the case may be, enforceable against them in
accordance with their terms.


<PAGE>   6

(b)           Neither the execution and delivery of this Amendment, any of the
Hanover Preferred Stock Offering Agreements or any other agreements, documents
or instruments to be delivered pursuant to this Agreement has violated or shall
violate any Federal or State securities laws or any other law or regulation or
any order or decree of any court or governmental instrumentality in any respect
applicable to Borrowers or Guarantors, or does or shall conflict with or result
in the breach of, or constitute a default in any respect under any mortgage,
deed of trust, security agreement, agreement or instrument to which Borrowers or
Guarantors is a party or may be bound, or shall violate any provision of the
Certificates of Incorporation or By-Laws of Borrowers or Guarantors.

(c)           No action of, or filing with, or consent of any governmental or
public body or authority, and no approval or consent of any other party, is
required to authorize, or is otherwise required in connection with, the
execution, delivery and performance of this Amendment and each other agreement
or instrument to be executed and delivered pursuant to this Amendment.

(d)           All of the representations and warranties set forth in the Loan
Agreement as amended hereby, and the other Financing Agreements, are true and
correct in all material respects after giving effect to the provisions of this
Amendment, except to the extent any such representation or warranty is made as
of a specified date, in which case such representation or warranty shall have
been true and correct as of such date.

(e)           After giving effect to the provisions of this Amendment, no Event
of Default or Incipient Default exists or has occurred and is continuing.

5.     Conditions Precedent. Concurrently with the execution and delivery hereof
(except to the extent otherwise indicated below), and as a further condition to
the effectiveness of this Amendment and the agreement of Lender to the
modifications and amendments set forth in this Amendment:

(a)           Lender shall have received an executed original or executed
original counterparts of this Amendment, as the case may be, duly authorized,
executed and delivered by Borrowers and Guarantors; and

(b)           after giving effect to the provisions of this Amendment, no Event
of Default or Incipient Default exists or has occurred and is continuing.

6.     Effect of this Amendment. This Amendment constitutes the entire agreement
of the parties with respect to the subject matter hereof, and supersedes all
prior oral or written communications, memoranda, proposals, negotiations,
discussions, term sheets and commitments with respect to the subject matter
hereof. Except as expressly provided herein, no other changes or modifications
to the Loan Agreement or any of the other Financing Agreements, or waivers of or
consents under any provisions of any of the foregoing, are intended or implied
by this Amendment, and in all other respects the Financing Agreements are hereby
specifically ratified, restated and confirmed by all parties hereto as of the
effective date hereof. To the extent that any provision of the Loan Agreement or
any of the other Financing Agreements conflicts with any provision of this
Amendment, the provision of this Amendment


<PAGE>   7

shall control.

7.     Further Assurances. Borrowers and Guarantors shall execute and deliver
such additional documents and take such additional action as may be reasonably
requested by Lender to effectuate the provisions and purposes of this Amendment.

8.     Governing Law. The rights and obligations hereunder of each of the
parties hereto shall be governed by and interpreted and determined in accordance
with the internal laws of the State of New York (without giving effect to
principles of conflicts of laws).

9.     Binding Effect. This Amendment shall be binding upon and inure to the
benefit of each of the parties hereto and their respective successors and
assigns.

10.    Counterparts. This Amendment may be executed in any number of
counterparts, but all of such counterparts shall together constitute but one and
the same agreement. In making proof of this Amendment, it shall not be necessary
to produce or account for more than one counterpart thereof signed by each of
the parties hereto.

                  [REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]


<PAGE>   8


         IN WITNESS WHEREOF, the parties hereto have caused this Amendment to be
executed on the day and year first written.

CONGRESS FINANCIAL CORPORATION

By:

Title:


HANOVER DIRECT PENNSYLVANIA, INC.

By:

Title:



BRAWN OF CALIFORNIA, INC.

By:


Title:



GUMP'S BY MAIL, INC.

By:


Title:



GUMP'S CORP.

By:


Title:



LWI HOLDINGS, INC.

By:


Title:


                       [SIGNATURES CONTINUE ON NEXT PAGE]
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]



<PAGE>   9

                                            HANOVER DIRECT VIRGINIA INC.

                                            By:


                                            Title:



                                            HANOVER REALTY, INC.

                                            By:


                                            Title:



                                            THE COMPANY STORE FACTORY, INC.

                                            By:


                                            Title:



                                            THE COMPANY OFFICE, INC.

                                            By:


                                            Title:



                                            KEYSTONE INTERNET SERVICES, INC.

                                            By:


                                            Title:






                       [SIGNATURES CONTINUE ON NEXT PAGE]


<PAGE>   10


[SIGNATURES CONTINUED FROM PREVIOUS PAGE]


TWEEDS, LLC

By:


Title:



SILHOUETTES, LLC

By:


Title:



HANOVER COMPANY STORE, LLC

By:


Title:



DOMESTICATIONS, LLC

By:



By their signatures below, the
undersigned Guarantors acknowledge and
agree to be bound by the applicable
provisions of this Amendment:

HANOVER DIRECT, INC.

By:


Title:



                       [SIGNATURES CONTINUE ON NEXT PAGE]


<PAGE>   11


[SIGNATURES CONTINUED FROM PREVIOUS PAGE]


AMERICAN DOWN & TEXTILE COMPANY

By:


Title:



D.M. ADVERTISING, INC.

By:


Title:



LWI RETAIL, INC.

By:


Title:



SCANDIA DOWN CORPORATION

By:


Title:



KEYSTONE LIQUIDATIONS, INC.

By:


Title:



YORK FULFILLMENT COMPANY, INC.

By:


Title:


                       [SIGNATURES CONTINUE ON NEXT PAGE]
                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]



<PAGE>   12

HANOVER HOME FASHIONS GROUP, LLC

By:


Title:



KITCHEN & HOME, LLC

By:


Title:



DOMESTICATIONS KITCHEN & GARDEN, LLC

By:


Title:



ENCORE CATALOG, LLC

By:


Title:



CLEARANCE WORLD OUTLETS, LLC

By:


Title:



                       [SIGNATURES CONTINUE ON NEXT PAGE]


<PAGE>   13


                    [SIGNATURES CONTINUED FROM PREVIOUS PAGE]


SCANDIA DOWN, LLC

By:


Title:



ERIZON, INC.

By:


Title:



HANOVER BRANDS, INC.

By:


Title:



ERIZON.COM, INC.

By:


Title:



LA CROSSE FULFILLMENT, LLC

By:


Title:



SAN DIEGO TELEMARKETING, LLC

By:


Title: