<PAGE>

                                   EXHIBIT 4.3

                          GRANT OF SECURITY INTEREST IN
                        PATENTS, TRADEMARKS AND LICENSES

THIS GRANT OF SECURITY INTEREST IN PATENTS, TRADEMARKS AND LICENSES (herein the
"Agreement") made as of this 29th day of March, 2004, by R. G. BARRY
CORPORATION, an Ohio corporation, with its principal place of business at 13405
Yarmouth Road, Pickerington, Ohio 43147 (herein the "Company"), and THE CIT
GROUP/COMMERCIAL SERVICES, INC., a New York corporation, with offices at 1211
Avenue of the Americas, New York, New York 10036 (herein "CIT").

                              W I T N E S S E T H:

WHEREAS, the Company and CIT are parties to a certain factoring and/or financing
arrangement, pursuant to which CIT may make certain loan and advances to the
Company, all as further set forth in the loan and security documents executed in
connection herewith, dated of even date herewith and as the same may be amended
from time to time (herein collectively, the "Factoring Agreement"), which
Factoring Agreement provides (i) for CIT to purchase, at its discretion certain
accounts receivable of the Company and/or make certain loans, advances and
extensions of credit, all to or for the benefit and account of the Company, and
(ii) for the grant by the Company to CIT, as collateral security therefor, a
security interest in certain of the Company's assets, including, without
limitation, its trademarks, trademark applications and/or registrations, trade
names, goodwill and licenses, and, if applicable, any patents, patent
applications and/or registrations, all as more fully set forth herein and in the
Factoring Agreement;

NOW THEREFORE, in consideration of the premises set forth herein and for other
good and valuable consideration, receipt and sufficiency of which is hereby
acknowledged, the Company agrees as follows:

1.       DEFINITIONS. Capitalized terms used herein and defined in the Factoring
         Agreement shall have the meanings set forth therein unless otherwise
         specifically defined herein.

2.       GRANT OF SECURITY INTEREST. To secure the payment of the Obligations,
         the Company hereby grants to CIT a security interest, effective
         immediately, in all of the Company's right, title and interest in and
         to all of the following described property, whether now owned or
         hereafter acquired (collectively herein the "Intellectual Property
         Collateral"):

(i)      Trademarks, trademark registrations, recordings and/or applications,
         trade names, trade styles, service marks, prints and labels on which
         any of the foregoing have or may appear, designs, general intangibles
         pertaining to any of the foregoing, including, without limitation, the
         trademarks and applications, if any, listed on

<PAGE>

         SCHEDULE B attached hereto and made a part hereof, and any and all
         reissues and/or renewals thereof, and all income, royalties, damages
         and payments now and hereafter due and/or payable in connection
         therewith including, without limitation, damages and payments for past
         or future infringements thereof (all of the foregoing being sometimes
         hereinafter individually and/or collectively referred to as the
         "Trademark Collateral");

(ii)     Patents and patent applications and/or registrations together with the
         inventions and improvements described and claimed therein including,
         without limitation, the patents and applications, if any, listed on
         SCHEDULE A, attached hereto and made a part hereof, and any and all
         reissues and renewals thereof and all income, royalties, damages and
         payments now and hereafter due and/or payable in connection therewith
         including, without limitation, damages and payments for past or future
         infringements thereof (all of the foregoing being sometimes hereinafter
         individually and/or collectively referred to as the "Patent
         Collateral");

(iii)    Any license agreement in which the Company is or becomes licensed to
         use any patents and/or trademarks owned by a third party including,
         without limitation, the licenses, if any, listed on SCHEDULE C attached
         hereto and made a part hereof (all of the foregoing being sometimes
         referred to herein individually and/or collectively as the "License
         Collateral");

(iv)     The goodwill of the Company's business connected with and symbolized by
         the Intellectual Property Collateral;

(v)      Any and all of the Company's rights and interests in any of the
         foregoing as they relate to the Company's Accounts, Inventory,
         Equipment and General Intangibles, or any Collateral bearing any of the
         foregoing, including without limitation the right to sell Inventory,
         goods and property bearing or covered by any of the foregoing
         Intellectual Property; and

(vi)     All cash and non-cash proceeds, royalties and income of the foregoing,
         including without limitation any amounts obtained pursuant to any
         infringement action.

3.       CIT'S RIGHTS. Upon the occurrence and continuance of any Event of
         Default hereunder, CIT shall have all the rights and remedies of a
         secured party under the Uniform Commercial Code and any other
         applicable state or federal laws. CIT will give the Company reasonable
         notice of the time and place of any public sale of the Intellectual
         Property Collateral or the time after which any private sale of the
         Intellectual Property Collateral or any other intended disposition
         thereof is to be made. Unless otherwise provided by law, the
         requirement of reasonable notice shall be met if such notice is mailed,
         postage prepaid to the address of the Company set forth above at least
         ten (10) days before the date of such sale or disposition. In addition
         to the foregoing and all other rights and remedies of CIT upon the
         occurrence and continuance of any Event of Default hereunder, CIT shall
         thereupon have the immediate right to transfer to itself or to sell,
         assign and transfer to any other person all right, title and interest
         in and to all or any part of the Intellectual Property Collateral. A
         formal irrevocable power of attorney (in the form annexed hereto) is
         being executed and delivered by the Company to CIT

<PAGE>

         concurrently with this agreement to enable such rights to be carried
         out. The Company agrees that, in the event CIT exercises its rights
         hereunder and/or pursuant to said power of attorney in accordance with
         its terms, after written notification of such exercise from CIT to the
         Company, the Company shall never thereafter, without the prior written
         authorization of the owner or owners of such Intellectual Property
         Collateral, use any of such Intellectual Property Collateral. The
         condition of the foregoing provision is such that unless and until
         there occurs an Event of Default under this agreement, the Company
         shall continue to own and use the Intellectual Property Collateral in
         the normal course of its business and to enjoy the benefits, royalties
         and profits therefrom provided, however, that from and after the
         occurrence and during the continuance of an Event of Default such right
         will, upon the exercise by CIT of the rights provided by this
         agreement, be revoked and the right of the Company to enjoy the uses,
         benefits, royalties and profits of said Intellectual Property
         Collateral will wholly cease, whereupon CIT or its transferee(s) shall
         be entitled to all of the Company's right, title and interest in and to
         the Intellectual Property Collateral hereby so assigned. This agreement
         will not operate to place upon CIT any duty or responsibility to
         maintain the Intellectual Property Collateral.

4.       FEES. The Company will pay all filing fees with respect to the security
         interest created hereby which CIT may deem necessary or advisable in
         order to perfect and maintain the perfection of its security interest
         in the Intellectual Property Collateral.

5.       REPRESENTATIONS AND WARRANTIES. The Company represents and warrants:
         that the Company lawfully possesses and owns the Intellectual Property
         Collateral and that except for the security interest granted hereby,
         the Intellectual Property Collateral will be kept free from all liens,
         security interests, claims and encumbrances whatsoever, other than
         Permitted Liens, that the Company has not made or given any prior
         assignment, transfer or security interest in the Intellectual Property
         Collateral or any of the proceeds thereof; that the Intellectual
         Property Collateral is and will continue to be, in all respects, in
         full force and effect; and that there are no known infringements of the
         Intellectual Property Collateral. The Company agrees not to take any
         action inconsistent with the terms and intent hereof, provided that the
         Company may enter into licensing agreements in the ordinary course of
         its business on fair and reasonable terms, provided further that no
         Event of Default (as defined herein below) has occurred and that any
         such agreement does not adversely effect CIT' rights and interests
         hereunder. The Company hereby further agrees to provide notice to CIT
         of any hereafter acquired Intellectual Property Collateral, provided
         that any such Collateral shall be automatically subject to the terms
         hereof and provided that the Company shall take any such additional
         action as CIT shall reasonably request with respect thereto.

6.       APPLICATION OF PROCEEDS. The proceeds of any sale, transfer or
         disposition of the Intellectual Property Collateral shall be applied
         first to all costs and expenses, including, but not limited to,
         reasonable attorneys' fees and expenses and court costs, incurred by
         CIT in connection with such sale and the exercise of CIT' rights and
         remedies hereunder and under the Factoring Agreement; next, such
         proceeds shall be applied to the payment, in whole or in part, of the
         Obligations due CIT in

<PAGE>

         such order as CIT may elect; and the balance, if any, shall be paid to
         the Company or as a court of competent jurisdiction may direct.

7.       DEFENSE OF CLAIMS. The Company will defend at its own cost and expense
         any action, claim or proceeding affecting the Intellectual Property
         Collateral or the interest of CIT therein. The Company agrees to
         reimburse CIT for all costs and expenses incurred by CIT in defending
         any such action, claim or proceeding.

8.       RIGHTS CUMULATIVE. This Agreement shall be in addition to the Factoring
         Agreement and shall not be deemed to affect, modify or limit the
         Factoring Agreement or any rights that CIT has under the Factoring
         Agreement. The Company agrees to execute and deliver to CIT (at the
         Company's expense) any further documentation or papers necessary to
         carry out the intent or purpose of this agreement including, but not
         limited to, financing statements under the Uniform Commercial Code.

9.       CONSTRUCTION AND INVALIDITY. Any provisions hereof contrary to,
         prohibited by or invalid under any laws or regulations shall be
         inapplicable and deemed omitted here from, but shall not invalidate the
         remaining provisions hereof.

10.      CHOICE OF LAW. THE COMPANY AGREES THAT THE VALIDITY, INTERPRETATION AND
         ENFORCEMENT OF THIS AGREEMENT AND ALL RIGHTS HEREUNDER SHALL BE
         GOVERNED BY THE LAWS OF THE STATE OF NEW YORK. THIS AGREEMENT TOGETHER
         WITH THE FACTORING AGREEMENT CONSTITUTES THE ENTIRE AGREEMENT OF THE
         COMPANY AND CIT WITH RESPECT TO THE INTELLECTUAL PROPERTY COLLATERAL,
         CAN ONLY BE CHANGED OR MODIFIED IN WRITING AND SHALL BIND AND BENEFIT
         THE COMPANY, CIT AND THEIR RESPECTIVE SUCCESSORS AND ASSIGNS. THE
         COMPANY AND CIT EACH HEREBY EXPRESSLY WAIVES ANY RIGHT OF TRIAL BY JURY
         ON ANY CLAIM, DEMAND, ACTION OR CAUSE OF ACTION ARISING HEREUNDER.

11.      EVENTS OF DEFAULT. Any of the following constitutes an Event of Default
         under this Agreement:

         (i)      The Company fails to perform or observe any agreement,
                  covenant or condition required under this Agreement to be
                  performed by it;

         (ii)     Any warranty or representation made by Company in this
                  Agreement shall be or becomes false or misleading in any
                  material respect; or

         (iii)    The occurrence and continuance of any Event of Default under
                  the Factoring Agreement which is not waived in writing by CIT.

12.      NOTICES. The Company covenants and agrees that, with respect to the
         Intellectual Property Collateral, it will give CIT written notice in
         the manner provided in the Factoring Agreement of:

<PAGE>

         (i)      any claim by a third party that the Company has infringed on
                  the rights of a third party;

         (ii)     any suspected infringement by a third party on the rights of
                  the Company; or

         (iii)    any Intellectual Property Collateral created, arising or
                  acquired by the Company after the date hereof.

13.      FURTHER ASSURANCES. The Company will take any such action as CIT may
         reasonably require to further confirm or protect CIT' rights under this
         agreement in the Intellectual Property Collateral. In furtherance
         thereof, the Company hereby grants to CIT a power of attorney coupled
         with an interest which shall be irrevocable during the term of this
         agreement to execute any documentation or take any action on the
         Company's behalf required to effectuate the terms, provisions and
         conditions of this agreement.

14.      TERMINATION. This agreement shall terminate upon termination of the
         Factoring Agreement and full, final and indefeasible payment in cash of
         all Obligations of the Company thereunder. Upon the Company's request,
         CIT shall within a reasonable time after any such termination execute
         and deliver to the Company (at the Company's expense) such documents
         and instruments as are reasonably necessary to evidence such
         termination and release of the security interest granted herein on any
         applicable public record.

<PAGE>

         IN WITNESS WHEREOF, the parties hereto have duly executed this
Agreement as of the 29th day of March, 2004.

                                    R. G. BARRY CORPORATION

                                    By /s/ Thomas M. Von Lehman
                                       -----------------------------------------
                                       Name: Thomas M. Von Lehman
                                       Title: President and CEO

Agreed and Accepted this
29th day of March, 2004

THE CIT GROUP/COMMERCIAL SERVICES, INC.

By: /s/ Richard Lyons
    --------------------------------
Title: Sr. Vice President