<PAGE>   1
                                   EXHIBIT 4.1
                 AGREEMENT TO EXTEND REVOLVING CREDIT AGREEMENT




                                  June 4, 1999


Mr. Michael Flannery
The Bank of New York
One Wall Street - 8th Floor
New York, New York 10286

Mr. Robert Friend
The Huntington National Bank
41 South High Street - HC 0810
Columbus, Ohio 43287

Mr. Tom Redmond
Bank One NA
P. O. Box 710170 - OH1-0170
Columbus, Ohio 43217-0170


         Re: R. G. Barry Corporation, Revolving Credit Agreement Extension


Dear Mike, Bob, and Tom:

I am pleased that we all had an opportunity to meet and review the Company's
1999 Operating Plan on Wednesday, June 2, 1999. Should any of you have any
questions, please let me know.

As you know, the Revolving Credit Agreement between The Bank of New York, The
Huntington National Bank, and Bank One, as successor to NBD Bank, (the "Bank" or
"Banks") dated as of February 28, 1996 (the "Agreement"), currently extends
through December 31, 2000.

In Section 4.11, the Agreement provides for the Company to request an extension
of the Termination Date and the Commitments for a period of one year, by making
a request following delivery of annual financial statements and the Company's
plan for the current year. We are obviously interested in extending the term of
the Agreement.

Please consider this letter as the Company's formal request for an extension of
the Agreement through December 31, 2001, under the same terms and conditions as
outlined in the Agreement.


                               Page 15 of 23 pages
<PAGE>   2

The Bank of New York
The Huntington National Bank
Bank One, NA
June 4, 1999
Page 2


In addition, we discussed the need to permit a Subsidiary of the Company to
incur up to $5 million of Debt. Currently, Section 9 of the Agreement prohibits
a Subsidiary of the Borrower from incurring Current Debt or Funded Debt. Please
consider this letter as an amendment to the Agreement, that will permit a
cumulative amount of up to $5 million of Current Debt or Funded Debt to be
incurred by a Subsidiary or Subsidiaries of the Borrower, provided the Borrower
is in compliance with all other terms of the Agreement.

Kindly acknowledge your Bank's consent to the extension of the Agreement to
December 31, 2001, and your Bank's consent to permit up to $5 million of Current
Debt or Funded Debt, by a Subsidiary or Subsidiaries of the Borrower, by signing
your name at the appropriate location below and returning one copy of this
letter to my attention. Once I have received copies from each of the Banks, I
will circulate originals so that we each have a fully executed set.

Sincerely,
R. G. BARRY CORPORATION

/s/ Michael Krasnoff

Michael S. Krasnoff
Vice President and Assistant Treasurer

/msk

Enclosures

By signing below, each Bank agrees to the extension of the Agreement through
December 31, 2001, and agrees to permit up to $5 million of Debt by a Subsidiary
or Subsidiaries of the Borrower, as outlined above. All other terms of the
Agreement shall remain in force and unchanged.

Acknowledged and Agreed:

/s/ Michael Flannery                                    06/28/99
---------------------------------------------           --------
The Bank of New York, by Mr. Michael Flannery             date


/s/ R. H. Friend                                        06/24/99
---------------------------------------------           --------
The Huntington National Bank, by Mr. Robert Friend        date


/s/ Thomas Redmond                                      06/29/99
---------------------------------------------           --------
Bank One, NA, by Mr. Thomas Redmond                      date


                               Page 16 of 23 pages