<PAGE>

                                   EXHIBIT 4.2

                                                            Date: March 29, 2004

R. G. BARRY CORPORATION
13405 Yarmouth Road
Pickerington, Ohio 43147

                         Re: LETTER OF CREDIT AGREEMENT

Ladies and Gentlemen:

         From time to time, in order to assist you in establishing or opening
Letters of Credit with a bank or trust company (herein the "Bank") to cover the
purchase of goods and inventory, you may request us to join in the applications
for such Letters of Credit, and/or guarantee payment or performance of such
Letters and any drafts or acceptances thereunder, thereby lending our credit to
you. These arrangements shall be handled by us subject to the terms and
conditions set forth below.

A.       Our assistance in this matter shall at all times and in all respects be
         in our sole discretion and as provided in the factoring agreement
         entered into between us of even date (the "Agreement"). The amount and
         extent of the Letters of Credit and the terms and conditions thereof
         and of any drafts or acceptances thereunder, shall in all respects be
         determined solely by us and shall be subject to change, modification
         and revision by us, at any time and from time to time. All capitalized
         terms not defined herein shall have the meaning given such terms in the
         Agreement.

B.       Any indebtedness, liability or obligation of any sort whatsoever,
         however arising, whether present or future, fixed or contingent,
         secured or unsecured, due or to become due, paid or incurred, arising
         or incurred in connection with any Letters of Credit, guarantees,
         drafts or acceptances thereunder or otherwise (herein the
         "Obligations") shall be incurred solely as an accommodation to you and
         for your account. Obligations shall include, without being limited to,
         all amounts due or which may become due under said Letters of Credit,
         guarantees or any drafts or acceptances thereunder, all amounts charged
         or chargeable to you or to us by the Bank, other financial institution
         or correspondent bank which opens, issues or is involved with such
         Letter of Credit, any other bank charges; fees and commissions; duties
         and taxes; costs of insurance; all such other charges and expenses
         which may pertain either directly or indirectly to such Letters of
         Credit, drafts, acceptances, guarantees or to the goods or documents
         relating thereto, and our charges as herein provided. We shall have the
         right, at any time and without notice to you, to charge your account on
         our books with the amount of any and all such Obligations when due. Any
         debit balance which may exist at any time or from time to time in your
         account shall be repayable to us on demand and shall incur interest at
         the rate provided in the Agreement. All Obligations are to be repaid to
         us solely in United States currency.

C.       As security for the prompt payment in full of all of your present and
         future indebtedness or obligations whether under Agreement between us,
         any other agreement between us or otherwise, as well as to secure the
         payment in full of all Obligations referred to herein, you hereby
         pledge and grant to us a continuing general lien upon and security
         interest in the following "Collateral", whether now owned or hereafter
         acquired by you, wherever located, whether in transit or not: all
         presently owned and hereafter acquired: (a) warehouse receipts, bills
         of lading, shipping documents, documents of title, chattel paper and

<PAGE>

         instruments, all whether negotiable or not; (b) merchandise, inventory
         and goods which relate to any of the foregoing or which are purchased
         from suppliers located outside of the United States or its territories
         or which relate to letters of credit opened through or with our
         assistance (whether for purchases from domestic or foreign suppliers),
         and all additions thereto, substitutions therefor and replacements
         thereof, in all stages of manufacture, process or production--from raw
         materials through work-in-process to finished goods, together with all
         goods and materials used or usable in manufacturing, processing,
         packaging or shipping same, all wherever located and whether in transit
         or not; and (c) cash and non-cash proceeds of any and all of the
         foregoing, of whatever sort and however arising.

D.       You warrant and represent that we have and shall have at all times a
         valid and effective first and paramount lien on and security interest
         in all said Collateral and that your title to said Collateral is
         unencumbered by any other liens except for Permitted Liens. You also
         warrant and represent that all sales of any goods or inventory covered
         hereby shall be made by you in the ordinary course of business and the
         accounts arising from such sales and proceeds thereof shall be
         transferred and assigned to us pursuant to the Agreement; and you
         confirm that our lien and security interest extends and attaches to
         those accounts and proceeds. Further, you warrant and represent that
         all Letters of Credit are being opened to cover actual purchases of
         goods and inventory solely for your account, and said goods will not be
         sold or transferred, other than as herein provided, without our
         specific prior written consent. You agree to comply with the
         requirements of any and all laws in order to grant to us and maintain
         in our favor, a valid first lien upon and security interest in the
         Collateral subject to Permitted Liens, and to do whatever we may
         request from time to time in order to effect the purposes of this
         Agreement, including, but without limitation, filing financing
         statements, keeping records and making reports on the Collateral to us,
         advising us of the location of all Collateral, marking, labeling and
         segregating such Collateral and obtaining any necessary agreements or
         waivers with regard to the Collateral. To the extent permitted by
         applicable law, you irrevocably authorize us to file financing
         statements and all amendments and continuations with respect thereto,
         all in order to create, perfect or maintain our security interest in
         the Collateral, and you hereby ratify and confirm any and all financing
         statements, amendments and continuations with respect thereto
         heretofore and hereafter filed by us pursuant to the foregoing
         authorization.

E.       You unconditionally indemnify us and hold us harmless from any and all
         loss, claim or liability arising from any transactions or occurrences
         relating to Letters of Credit established or opened for your account,
         the Collateral relating thereto and any drafts or acceptances
         thereunder (collectively "Losses") and all Obligations hereunder,
         including any such losses due to any action taken by any Bank, other
         than Losses arising from or relating to the gross negligence or willful
         misconduct of such Bank or us. You further agree to hold us harmless
         for any errors or omission, whether caused by us, by the Bank or
         otherwise. Your unconditional obligation to us hereunder shall not be
         modified or diminished for any reason or in any manner whatsoever. You
         agree that any charges made to us for your account by the Bank shall be
         conclusive on us and may be charged to your account.

F.       We shall not be responsible for: The existence, character, quality,
         quantity, condition, packing, value or delivery of the goods purporting
         to be represented by any documents; any difference or variation in the
         character, quality, quantity, condition, packing value or delivery of
         the goods from that expressed in the documents; the validity,
         sufficiency or genuineness of any documents or of any endorsements
         thereon, even if such documents should in fact prove to be in any or
         all respects invalid, insufficient, fraudulent or forged; the time,
         place, manner or order in which shipment is made; partial or incomplete
         shipment, or failure or omission to ship any or all of the goods
         referred to in the Letters of Credit or

<PAGE>

         documents; any deviation from instructions; delay, default, or fraud by
         the shipper and/or anyone else in connection with the Collateral or the
         shipping thereof; or any breach of contract between the shipper or
         vendors and yourselves. Furthermore, without being limited by the
         foregoing, we shall not be responsible for any act or omission with
         respect to or in connection with any Collateral.

G.       You agree that any action taken by us, if taken in good faith, or any
         action taken by any Bank, under or in connection with the Letters of
         Credit, the guarantees, the drafts or acceptances, or the Collateral,
         shall be binding on you and shall not put us in any resulting liability
         to you. In furtherance thereof, we shall have the full right and
         authority to clear and resolve any questions of non-compliance of
         documents; to give any instructions as to acceptance or rejection of
         any documents or goods; to execute any and all steamship or airway
         guarantees (and applications therefor), indemnities or delivery orders;
         to grant any extensions of the maturity of, time of payment for, or
         time of presentation of, any drafts, acceptances, or documents; and to
         agree to any amendments, renewals, extensions, modifications, changes
         or cancellations of any of the terms or conditions of any of the
         applications, Letters of Credit, drafts or acceptances; all in our sole
         name, and the Bank shall be entitled to comply with and honor any and
         all such documents or instruments executed by or received solely from
         us, all without any notice to or any consent from you.

H.       Without our express consent and endorsement in writing, you agree not
         to clear and resolve any questions of non-compliance of documents; to
         give any instructions as to acceptance or rejection of any documents or
         goods; to execute any and all applications for steamship or airway
         guarantees, indemnities or delivery orders; to grant any extensions of
         the maturity of, time of payment for, or time of presentation of, any
         drafts, acceptances or documents; or to agree to any amendments,
         renewals, extensions, modifications, changes or cancellations of any of
         the terms or conditions of any of the applications, Letters of Credit,
         drafts or acceptances.

I.       You agree that any necessary import, export or other licenses or
         certificates for the import or handling of the Collateral will have
         been promptly procured; all foreign and domestic governmental laws and
         regulations in regard to the shipment and importation of the
         Collateral, or the financing thereof will have been promptly and fully
         complied with; and any certificates in that regard that we may at any
         time request will be promptly furnished. In this connection, you
         warrant and represent that all shipments made under any such Letters of
         Credit are in accordance with the governmental laws and regulations of
         the countries in which the shipments originate and terminate, and are
         not prohibited by any such laws and regulations. You assume all risk,
         liability and responsibility for, and agree to pay and discharge, all
         present and future local, state, federal or foreign taxes, duties, or
         levies. Any embargo, restrictions, laws, customs or regulations of any
         country, state, city, or other political subdivision, where the
         Collateral is or may be located, or wherein payments are to be made, or
         wherein drafts may be drawn, negotiated, accepted, or paid, shall be
         solely your risk, liability and responsibility.

J.       Any rights, remedies, duties or obligations granted or undertaken by
         you to any Bank in any application for Letters of Credit, or any
         standing agreement relating to Letters of Credit or otherwise, shall be
         deemed to have been granted to us and apply in all respects to us and
         shall be in addition to any rights, remedies, duties or obligations
         contained herein.

K.       You hereby agree that prior to your repayment of all indebtedness and
         Obligations to us, we may be deemed to be the absolute owner of, with
         unqualified rights to possession and disposition of, all Collateral,
         all of which may be held by us as security as herein provided. Should
         possession of any such Collateral be transferred to you, it shall
         continue to serve,

<PAGE>

         as security as herein provided, and any goods or inventory covered
         hereby may be sold, transferred or disposed of only as herein above
         provided.

L.       You agree to maintain insurance on the Collateral under such policies
         of insurance, with such insurance companies, in such amounts and
         covering such risks as are at all times reasonably satisfactory to us,
         but at your expense. All policies covering the Collateral are to be
         made payable to us in case of loss, under a standard non-contributory
         "mortgagee", "lender's" or "secured party" clause and are to contain
         such other provisions as we may require to fully protect our interests
         in the Collateral and to any payments to be made under such policies,
         and all proceeds of such policies are hereby assigned to us. All
         policies are to be delivered to us, premium prepaid; and shall provide
         for not less than ten days prior written notice to us of the exercise
         of any right of cancellation. We shall have the sole right, in our own
         name or your name, to file claims under any such insurance policies and
         to deal with and handle such claims and any payments thereunder in all
         respects.

M.       Upon the occurrence and continuance of an Event of default, if any
         representation or warranty made by you hereunder is incorrect in any
         material respect or if any covenant made by you hereunder is breached,
         we shall have the right, with or without notice to you, to foreclose
         the lien and security interest created herein by any available judicial
         procedure, or to take possession of the Collateral without judicial
         process, and to enter any premises where the Collateral may be located
         for the purpose of taking possession of or removing the Collateral. We
         shall have the right to sell, lease, or otherwise dispose of all or any
         part of the Collateral, whether the goods have arrived or are to
         arrive, in its then condition or after further preparation or
         processing, in your name or in ours, or in the name of such party as we
         may designate, either at public or private sale or at any broker's
         board, in lots or in bulk, for cash or for credit, with or without
         warranties or representations, and upon such other terms and conditions
         as we in our sole discretion may deem advisable, and we shall have the
         right to purchase at any such sale. You agree, at our request, to
         assemble the Collateral and to make it available to us at places which
         we shall select, whether at your premises or elsewhere, and to make
         available to us all of your premises and facilities for the purpose of
         our taking possession of, removing or putting the Collateral in
         saleable form. The proceeds of any such sale, lease or other
         disposition of the Collateral shall be applied first, to the expenses
         of retaking, holding, storing, processing and preparing for sale,
         selling, and the like, and then to the satisfaction of your Obligations
         or other indebtedness to us, application as to particular Obligations
         or as to principal or interest to be in our absolute and sole
         discretion. You shall be liable to us for, and shall pay to us on
         demand, any deficiency which may remain after such sale, lease, or
         other disposition, and we in turn agree to remit to you any surplus
         resulting therefrom. We shall have all rights of a secured party under
         the Uniform Commercial Code. The enumeration of the foregoing rights is
         not intended to be exhaustive and the exercise of any right shall not
         preclude the exercise of any other rights all of which shall be
         cumulative.

N.       Any charges, fees, commissions, costs and expenses charged to us for
         your account by any Bank in connection with or arising out of Letters
         of Credit issued pursuant hereto or out of transactions relating
         thereto will be charged to your account in full as received by us and
         when made by any such Bank shall be conclusive on us. In addition to
         the amounts charged to your account pursuant to the preceding sentence,
         for our services hereunder we shall be entitled to our service fees
         which shall be computed and charged to your account as follows:

                       [SEE LETTER OF CREDIT FEE SCHEDULE]

<PAGE>

         This agreement, which is subject to modification only in writing, is
supplementary to and is to be considered as a part of, the Agreement. If the
foregoing is in accordance with your understanding, please so indicate by
signing and returning the enclosed copy of this letter.

                                    Very truly yours,

                                    THE CIT GROUP/COMMERCIAL SERVICES, INC.

                                    By /s/ Richard Lyons
                                       -----------------------------------------
                                       Name: Richard Lyons
                                       Title: Sr. Vice President

Read and Agreed to:

R. G. BARRY CORPORATION

By /s/ Thomas M. Von Lehman
   ---------------------------------
   Name: Thomas M. Von Lehman
   Title: President and CEO

<PAGE>

                                   SCHEDULE A

                      IMPORT LETTER OF CREDIT FEE SCHEDULE

<TABLE>
<CAPTION>
          TRANSACTION                            FEE                  MINIMUM
--------------------------------    -----------------------------    ---------
<S>                                 <C>                              <C>
ISSUANCE                             1/4% of face amount             $   70.00

     Plus: Processing Fee           $70.00

           Cable Fee                $35.00

AMENDMENTS

     L/C Amount Increase             1/4% of the increased amount    $   70.00

     Plus: Processing Fee

           First four amendments    $ 95.00

           After fourth             $125.00

GUARANTEES                          $ 50.00

     Plus: Processing Fee           $ 50.00

DISCREPANCIES                       $ 50.00

CANCELLATIONS                       $ 50.00

UNUTILIZED L/C FEE                  $100.00

MONTHLY COMMISSIONS                 3% per annum
                                    charged monthly                  $  100.00
PAYMENTS

     Sight                          1/4% of draft amount             $   55.00

     Time                           1 1/2% per annum
                                    on draft amount for term         $   85.00

STANDBY L/C'S

ISSUANCE                            3% per annum of the face
                                    Amount of the standby L/C        $1,000.00

Payment fee                         $1,000.00
Amendment fee                       $1,000.00
</TABLE>