<PAGE>   1
                                                                    EXHIBIT 10.6

                     AMENDMENT NO. 4 TO AMENDED AND RESTATED
                           ACCOUNT PURCHASE AGREEMENT


         AMENDMENT NO. 4, made and entered into as of June 28, 1996, by and
among HANOVER DIRECT, INC., a Delaware corporation and the successor-in-interest
to The Hanover Companies ("HDI"); HANOVER DIRECT PENNSYLVANIA, INC., a
Pennsylvania corporation formerly known as Hanover Direct, Inc. and Hanover
Direct Fulfillment, Inc. ("HDPI"); BRAWN OF CALIFORNIA, INC., a California
corporation ("Brawn"); GUMP'S CATALOG, INC., a Delaware corporation formerly
known as GSF Acquisition Corp. ("Gump's Corp."); GUMP'S HOLDINGS, INC., a
Delaware corporation ("Gump's Holdings"); GUMP'S BY MAIL, INC., a Delaware
corporation ("Gump's By Mail"); HANOVER DIRECT VIRGINIA, INC., a Delaware
corporation ("Hanover Virginia"); and TWEEDS, INC., a Delaware corporation
("Tweeds") ("HDI, HDPI, Brawn, GCI, Gump's Corp, Gump's Holdings, Gump's by
Mail, Hanover Virginia and Tweeds being hereinafter collectively and
individually referred to as "Hanover") and GENERAL ELECTRIC CAPITAL CORPORATION
("GE Capital"), a New York corporation.

                              W I T N E S S E T H:

         WHEREAS, HDI, HDPI, Brawn, GCI, Gump's Corp., Gump's Holdings and GE
Capital are parties to an Amended and Restated Account Purchase Agreement dated
as of April 25, 1994, as amended by an Amended No. 1 to Amended and Restated
Purchase Agreement dated as of November 2, 1994, and by a Second Amendment to
Account Purchase Agreement dated as of June 1, 1995, and by a Waiver and
Amendment No. 3 to Amended and Restated Purchase Agreement dated as of November
14, 1995 (collectively, the "Purchase Agreement");

         WHEREAS, it is the mutual desire of Hanover and GE Capital that the
Purchase Agreement be amended in accordance with the terms and conditions
hereinafter set forth;

         NOW, THEREFORE, in consideration of the mutual promises and subject to
the terms and conditions hereinafter set forth, the parties hereto hereby agree
as follows:

         1. Capitalized terms used herein which are not otherwise defined shall
have the same meaning as in the Purchase Agreement.

         2. Gump's By Mail, Hanover VirginIa, and Tweeds shall be made parties
to and shall be bound by the terms of the Purchase Agreement.

         3. All references to "Gump's By Mail" throughout the Purchase Agreement
shall refer to the company now known as Gump's


<PAGE>   2
by Mail, Inc., and shall not refer to the company now known as Gump's Catalog,
Inc.

         4. All references to "Hanover" throughout the Purchase Agreement shall
refer to HDI, HDPI, Brawn, GCI, Gump's Corp., Gump's Holdings, Gump's By Mail,
Hanover Virginia and Tweeds, collectively and individually.

         5. All references to "Gump's" throughout the Purchase Agreement shall
refer to GCI, Gump's Corp., Gump's Holdings, and Gump's By Mail, collectively
and individually.

         6. All references to "GSF" throughout the Purchase Agreement shall
refer to Gump's Corp.

         7. Section 2.1(c)(2) of the Purchase Agreement shall be deleted and
replaced with the following language:

            (2) from any of Gump's By Mail, Gump's Corp., Gump's Holdings or GCI
            if any such party no longer is under the control of Hanover.

         8. In Section 7.1 of the Purchase Agreement, the language at the
beginning of the first sentence reading "HDI, HDPI and Brawn" shall be deleted
and replaced with the following language:

            HDI, HDPI, Brawn, GCI, Gump's Corp., Gump's Holdings, Gump's By
            Mail, Hanover Virginia and Tweeds.

         9. The first sentence of Section 8.20 of the Purchase Agreement shall
be deleted and replaced as follows:

            The Corporate structure in connection with Hanover is as follows:
            (a) Brawn, Gump's Holdings; HDPI, Hanover Virginia, and Tweeds are
            wholly-owned subsidiaries of HDI, (b) International Mail is a
            tradename and catalog asset owned by Brawn, and (c) Gump's By Mail,
            Gump's Corp. and GCI are wholly-owned subsidiaries of Gump's
            Holdings.

        10. Each party shall pay its own out-of-pocket legal expenses in
connection with the execution and delivery of this Amendment and the closing of
the transactions relating thereto.

        11. Except as specifically provided herein, the terms and conditions of
the Purchase Agreement shall continue in full force and effect and shall be
fully binding on the parties hereto. Upon execution of this Agreement, each
reference in the Purchase Agreement to "this Purchase Agreement," "hereunder,"
"hereof," or words of like import, shall mean and be a reference to the Purchase
Agreement as amended hereby. In the event of any conflict between the terms of
the Purchase Agreement and the


                                       -2-

<PAGE>   3
terms of this Amendment, the terms of this Amendment shall prevail.

         12. This Amendment may be executed in any number of counterparts, all
of which taken together shall constitute one and the same amendatory instrument,
and any of the parties hereto may execute this Amendment by signing any such
counterpart.

         IN WITNESS WHEREOF, this Amendment has been duly executed as of the
date and year first above written.

                                        GENERAL ELECTRIC CAPITAL
                                        CORPORATION

                                        By: /s/ J. Alex Aguilar
                                            ---------------------------
                                            Name: J. Alex Aguilar
                                                  ---------------------
                                            Title: Program Manager
                                                   --------------------

                                        HANOVER DIRECT, INC.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                           Name: Edward J. O'Brien
                                                 ----------------------
                                           Title: Senior Vice President
                                                  ---------------------

                                        HANOVER DIRECT PENNSYLVANIA,
                                        INC.

                                        By: /s/ Edward J. O'Brien
                                            ---------------------------
                                           Name: Edward J. O'Brien
                                                 ----------------------
                                           Title: Vice President
                                                  ---------------------

                                        BRAWN OF CALIFORNIA, INC.

                                       By: /s/ Edward J. O'Brien
                                            ---------------------------
                                           Name: Edward J. O'Brien
                                                 ----------------------
                                           Title: Vice President
                                                  ---------------------      
                                                
                                        GUMP'S CATALOG, INC.

                                       By: /s/ Edward J. O'Brien
                                            ---------------------------
                                           Name: Edward J. O'Brien
                                                 ----------------------
                                           Title: Vice President
                                                  ---------------------

                                       -3-


<PAGE>   4
                                         GUMP'S CORP.

                                         By: /s/ Edward J. O'Brien
                                            ____________________________
                                            Name:  Edward J. O'Brien
                                                 _______________________
                                            Title: Vice President
                                                  ______________________


                                         GUMP'S HOLDINGS, INC.

                                         By: /s/ Edward J. O'Brien
                                            ____________________________
                                            Name:  Edward J. O'Brien
                                                 _______________________
                                            Title: Vice President
                                                  ______________________


                                         GUMP'S BY MAIL, INC.

                                         By: /s/ Edward J. O'Brien
                                            ____________________________
                                            Name:  Edward J. O'Brien
                                                 _______________________
                                            Title: Vice President
                                                  ______________________



                                       -4-