<PAGE>

                                                                    EXHIBIT 10.4

                                 AMENDMENT NO. 3
                                     TO THE
                             R. G. BARRY CORPORATION
                           ASSOCIATES' RETIREMENT PLAN

                  WHEREAS, R. G. Barry Corporation ("Sponsor") has adopted the
R. G. Barry Corporation Associates' Retirement Plan ("Plan"); and

                  WHEREAS, the Plan provides that the Sponsor may amend the Plan
from time to time; and

                  WHEREAS, the Sponsor desires to amend the Plan to freeze
Participants' accrued benefits and participation in the Plan effective as of the
close of business on March 31, 2004;

                  NOW, THEREFORE, the Plan is amended as follows:

                  1.       The following new Section 1.3 shall be added to
                           Section 1 of the Plan:

                  1.3      FREEZE OF THE PLAN

                  Notwithstanding any provision of this Plan to the contrary,
                  effective March 31, 2004, the Plan is frozen. To this end, a
                  Participant's accrued benefit shall only be based on Benefit
                  Service and Compensation earned as a result of such person's
                  employment with the Employer on or prior to March 31, 2004. In
                  addition, a Participant's retirement benefit shall be
                  determined in accordance with Article IV, as modified by this
                  amendment. Participation in the Plan is also limited to only
                  those Eligible Employees who became Participants on or prior
                  to March 31, 2004. However, the determination of a
                  Participant's service for the purpose of determining such
                  person's Vesting Service or eligibility for retirement
                  benefits (e.g., early retirement) shall not be limited by this
                  amendment.

                  2.       The following paragraph shall be added to Section
                           2.1(j) at the end thereof:

     Notwithstanding any provision of this Plan to the contrary, Compensation
     paid for employment periods commencing after March 31, 2004, shall not be
     used in determining a Participant's accrued benefit.

                  3.       The following paragraph shall be added to Section 3.1
                           at the end thereof:

                           Notwithstanding any provision of this Plan to the
                           contrary, no Eligible Employee shall become a
                           Participant in the Plan after March 31, 2004.

                  4.       The following paragraph shall be added to Sections
                           3.5(b) and (c) at the end thereof:

<PAGE>

                           Notwithstanding any provision of this Plan to the
                           contrary, a Participant's Benefit Service shall not
                           include a Participant's period of Vesting Service
                           credited to such Participant after March 31, 2004.

                  5.       Section 5.3 and 5.4 shall be restated as follows:

                           Effective on and after March 31, 2004, if a
                           Participant whose retirement benefits have commenced
                           is reemployed as an Eligible Employee either before
                           or after attaining his Normal Retirement Age, his
                           retirement benefits shall continue to be paid by the
                           Plan.

IN WITNESS WHEREOF, the undersigned has executed this amendment to be effective
as of the date it is executed.

                                         R. G. BARRY CORPORATION

                                         By: /s/ Daniel D. Viren
                                            ------------------------------------
                                         Name: Daniel D. Viren

                                         Title: Senior Vice President - Finance,
                                         Chief Financial Officer

Date: February 20, 2004