<PAGE>   1
                                                                       Exhibit 4


                 SECOND AMENDMENT TO REVOLVING CREDIT AGREEMENT

This Second Amendment to Revolving Credit Agreement (this "Second Amendment") is
entered into at Columbus, Ohio, by and among The Huntington National Bank, The
Bank of New York and Bank One, N. A., as lenders (the "Banks"); The Huntington
National Bank as agent (the "Agent"); and R. G. Barry Corporation, as borrower
(the "Borrower"), as of the 30th day of June, 2000, in order to amend the
Revolving Credit Agreement entered into by and among the Banks and the Borrower
as of the 28th day of February, 1996, which has been amended by an Amendment to
Revolving Credit Agreement dated as of March 17, 2000 (collectively the "Credit
Agreement").

     Whereas, the parties to this Second Amendment desire to amend certain of
the provisions of the Credit Agreement, the Credit Agreement is hereby amended
as follows:

          1. Section 9.17 of the Credit Agreement is hereby amended to
recite in its entirety as follows:

                  EBITDA. Permit Consolidated Net Income PLUS Consolidated Net
                  Interest Expense, consolidated taxes, consolidated
                  amortization and consolidated depreciation ("EBITDA") to be
                  less than the following amounts for each of the following
                  fiscal quarters:

                           Quarter Ending      EBITDA
                           --------------      ------

                             April 1, 2000                   ($6,500,000)
                             July 1, 2000                    ($4,500,000)
                             September 30, 2000               $6,500,000

                  In testing compliance with this section, the Borrower shall be
                  permitted to carry over any amount of EBITDA not exceeding
                  $1,500,000 achieved in the previous fiscal quarter that
                  exceeds the amount required by this section; provided,
                  however, that the amount that may be carried over from the
                  second to the third fiscal quarter shall be calculated as if
                  the required EBITDA amount for the second quarter (i. e., the
                  quarter ending July 1, 2000) was ($3,500,000).

          2. The Borrower represents and warrants that no Event of Default has
occurred and is continuing, nor will any occur immediately after the execution
and delivery of this Second Amendment by the performance or observance of any
provision hereof.

          3. Each reference to the Credit Agreement, whether by use of the
phrase "Credit Agreement," "Agreement," the prefix "herein" or any other term,
and whether contained in the Credit Agreement itself, in this Second Amendment,
in any document executed concurrently herewith or in any loan documents executed
hereafter, shall be construed as a reference to the Credit Agreement as
previously amended and as amended by this Second Amendment.

          4. Except as previously amended and as modified herein, the Credit
Agreement and the Loan Documents shall remain as written originally and in full
force and effect in all respects, and nothing herein shall affect, modify, limit
or impair any of the rights and powers which the Banks may have thereunder.

          5. The Borrower agrees to perform and observe all the covenants,
agreements, stipulations and conditions to be performed on its part under the
Credit Agreement, the promissory notes executed and delivered in connection
herewith, the Loan Documents, and all other related agreements, as amended by
this Second Amendment.


                              Page 14 of 16 pages
<PAGE>   2



          6. The Borrower hereby represents and warrants to the Agent and the
Banks that (a) the Borrower has legal power and authority to execute and deliver
the within Second Amendment; (b) the respective officer executing the within
Second Amendment on behalf of the Borrower has been duly authorized to execute
and deliver the same and bind the Borrower with respect to the provisions
provided for herein; (c) the execution by the Borrower and the performance and
observance by the Borrower of the provisions hereof do not violate or conflict
with the articles of incorporation, regulations or by-laws of the Borrower or
any law applicable to the Borrower or result in the breach of any provision of
or constitute a default under any agreement, instrument or document binding upon
or enforceable against the Borrower; and (d) this Second Amendment and the
promissory notes executed and delivered in connection herewith, constitute valid
and legally binding obligations upon the Borrower, subject to applicable
bankruptcy, insolvency, reorganization or other similar laws affecting
creditors' rights generally, to general equitable principles and to applicable
doctrines of commercial reasonableness.

          7. This Second Amendment shall become effective only upon its
execution by the Borrower, the Banks and the Agent. Execution of this Second
Amendment by the parties hereto may be in any number of counterparts, but all of
such counterparts when taken together shall constitute one and the same
document.

          8. The capitalized terms used herein shall have the same meanings as
the capitalized terms used in the Credit Agreement.

         IN WITNESS WHEREOF, the Borrower, the Banks and the Agent have hereunto
set their hands as of the 30th day of June, 2000.


ATTEST:                                R. G. BARRY CORPORATION


/s/ Joan Faulkner                      By:  /s/ Michael S. Krasnoff
------------------                          ------------------------------
                                            Michael S. Krasnoff, Vice President


                                       THE HUNTINGTON NATIONAL BANK,
                                       Individually and as Agent


                                       By:  /s/ Bud Ward
                                            ------------------------------
                                            Bud Ward, Senior Vice President


                                       THE BANK OF NEW YORK


                                       By:  /s/ Howard F. Bascom, Jr.
                                            ------------------------------
                                            Howard Bascom, Vice President


                                       BANK ONE, N. A.


                                       By:  /s/ Thomas E. Redmond
                                            ------------------------------
                                            Thomas Redmond, Managing Director




                               Page 15 of 16 pages