<PAGE>

                                                                   EXHIBIT 10.55

                         AMENDMENT TO FACTORING AGREMENT

      AMENDMENT AGREEMENT made as of this 31st day of March, 2005 between R.G.
BARRY CORPORATION, an Ohio corporation ("Borrower") and THE CIT GROUP/COMMERCIAL
SERVICES, INC., a New York corporation ("CIT"), in its capacity as factor under
the Factoring Agreement defined below (in such capacity, "Factor").

                                   BACKGROUND

      Borrower and Factor are party to a certain factoring agreement between
Borrower and Factor dated as of March 29, 2004 (as amended, the "Factoring
Agreement") pursuant to which Borrower agreed to sell to Factor, and Factor
agreed to buy from Borrower, accounts receivable of Borrower, and Factor made
Advances to Borrower, all subject to the terms and conditions of the Factoring
Agreement. All capitalized terms used herein and not otherwise defined have the
meanings ascribed to them in the Financing Agreement.

      Pursuant to Financing Agreement dated as of the date hereof, CIT has
agreed to finance certain accounts receivable of Borrower and, accordingly,
Borrower no longer desires to factor its accounts receivable nor request
Advances from Factor.

      NOW, THEREFORE, in consideration of the foregoing and the mutual covenants
contained herein, the parties hereto agree as follows:

            1. Effective as of the date hereof, (a) Borrower will not sell or
assign to Factor any of its Accounts and Factor will not purchase any Accounts
of Borrower and (b) Factor will no longer extend any Advances to Borrower.

            2. Except as specifically provided for in paragraph 1 hereof, the
Factoring Agreement shall remain in full force and effect in accordance with the
terms thereof.

            3. This agreement shall be governed by the laws of the State of New
York, and unless the context of this agreement otherwise requires, all terms
used herein which are defined in the Uniform Commercial Code shall have the
meanings therein stated.

            4. This Agreement is solely for the benefit of the parties hereto
and their respective successors and assigns. No other person, firm, entity or
corporation shall have any right, benefit, priority or interest under, or
because of the existence of this Agreement.

            5. This Agreement may be executed by the parties hereto in one or
more counterparts, each of which shall be deemed an original and all of which
taken together shall be deemed to constitute one and the same agreement. Any
signature delivered by a party via telecopier shall be deemed to be an original
signature hereto.

<PAGE>

      IN WITNESS WHEREOF, the parties hereto have caused this Amendment to
Factoring Agreement to be executed by their respective corporate officers
thereunto duly authorized as of the day and year first above written.

                               R.G. BARRY CORPORATION

                               By: /s/ Thomas Von Lehman
                                   ---------------------------------------------
                                   Name: Thomas Von Lehman
                                   Title: President and Chief Executive Officer

                               THE CIT GROUP/COMMERCIAL SERVICES,
                               INC., as Factor

                               By: /s/ Marc Theisinger
                                   ---------------------------------------------
                                   Name: Marc Theisinger
                                   Title: Vice President

                                       2