<PAGE>

                                   EXHIBIT 4.5

                                                                  March 29, 2004

R.G. Barry Corporation
13405 Yarmouth Road
Pickerington, Ohio 43147

                          INVENTORY SECURITY AGREEMENT

Ladies and Gentlemen:

This agreement is being executed by you to induce us to make loans or advances
to you and to induce us to enter into or continue a factoring or financing
arrangement with you, and is executed in consideration of our doing or having
done any of the foregoing.

         1.  ADVANCES

1.1 We have agreed to make Advances to you in our sole discretion and from time
to time against your Eligible Finished Goods Inventory as provided in the
Factoring Agreement entered into between us of even date (the "Agreement"). All
capitalized terms not defined herein shall have the meaning given such terms in
the Agreement.

1.2 The amount of the loans and advances made or to be made by us to you, and
the period of time during which they are to remain outstanding shall at all
times be in our sole discretion and as provided in the Agreement.

1.3 Nothing contained herein shall be construed as limiting or modifying, in any
way, our right to: (a) hold any reserve we deem necessary as security for
payment and performance of your Obligations, and/or (b) change the rates of
advance or entirely cease making Advances.

         2. GRANT OF SECURITY INTEREST

2.1 As security for the prompt payment in full of all Obligations (as
hereinafter defined) due by you from time to time to us, in conjunction with the
Agreement, you hereby pledge and grant to us a continuing general lien upon, and
security interest in (herein "Security Interest"), the following described
"Inventory":

         All present and hereafter acquired merchandise, inventory and goods,
         and all additions, substitutions and replacements thereof, wherever
         located, together with all goods and materials used or usable in
         manufacturing, processing, packaging or shipping same; in all stages of
         production -- from

<PAGE>

         raw materials through work-in-process to finished goods -- and all
         proceeds of whatever sort.

2.2      The Security Interest in the Inventory shall extend and attach to:

(a) All Inventory which is presently in existence and which is owned by you or
in which you have any ownership interest, and all Inventory which you may
purchase or in which you may acquire any ownership interest at any time and from
time to time in the future, whether such Inventory is in transit or in your or
our constructive, actual or exclusive possession, or is held by others for your
account;

(b) All Inventory wherever located, including, without limitation, all Inventory
which may be located on your premises or upon the premises of any carriers,
forwarding agents, truckers, warehousemen, vendors, selling agents, finishers,
converters, processors, or other third persons who may have possession of the
Inventory; and

(c) All Inventory and any portion thereof which may be returned, rejected,
reclaimed or repossessed by either of us from your customers, as well as to all
supplies, goods, incidentals, packaging materials, and any other items which
contribute to the finished goods or products manufactured or processed by you,
or to the sale, promotion or shipment thereof.

         3. OBLIGATIONS SECURED

         The Security Interest granted hereunder and any lien or security
interest that we now or hereafter have in any of your other assets, collateral
or property, secure the payment and performance of all of your now existing and
future indebtedness and obligations to us, whether absolute or contingent,
whether arising under the Agreement, this agreement or any other agreement or
arrangement between us, by operation of law or otherwise including ledger debt
(which is indebtedness for goods and services purchased by you from any party
whose accounts receivable are factored or financed by us), and indebtedness
arising under any guaranty, credit enhancement or other credit support granted
by us in your favor, including any accommodation extended with respect to
applications for letters of credit, our acceptance of drafts or our endorsement
of notes or other instruments for your account and benefit (herein the
"Obligations"). Obligations shall also include, without limitation, all
interest, commissions, financing and service charges, and expenses and fees
chargeable to and due from you under this agreement, the Agreement or any other
agreement or arrangement which may be now or hereafter entered into between us.

         4. REPRESENTATIONS, WARRANTIES AND COVENANTS

4.1 You agree to safeguard, protect and hold all Inventory for our account and
make no disposition thereof except in the regular course of your business as
herein provided. You represent and warrant that Inventory will be sold and
shipped by you to your customers only in the ordinary course of your business
and then only on open account and on terms not exceeding the terms currently
being extended by you to your customers, provided that all proceeds of all sales
(including cash, accounts receivable, checks, notes, instruments for the payment
of money and similar proceeds) are forthwith transferred, assigned,

<PAGE>

endorsed, and turned over and delivered to us. Invoices covering sales of
Inventory are to be assigned to us in accordance with the provisions of the
Agreement, and the proceeds thereof (if collected by you) are to be turned over
to us in accordance with the provisions of the Agreement. Cash sales of
Inventory shall not exceed $100,000 in any Period, or sales in which a lien upon
or security interest in the Inventory is retained by you shall only be made by
you with our written approval, and all proceeds of such sales shall not be
commingled with your other property, but shall be segregated, held by you in
trust for us as our exclusive property, and shall be delivered immediately by
you to us in the identical form received by you. Upon the sale, exchange, or
other disposition of the Inventory, as herein provided, the Security Interest
provided for herein shall, without break in continuity and without further
formality or act, continue in, and attach to, all proceeds, including any
instruments for the payment of money, accounts receivable, contract rights,
documents of title, shipping documents, chattel paper and all other cash and
non-cash proceeds of such sale, exchange or disposition. As to any such sale,
exchange or other disposition, we shall have all of the rights of an unpaid
seller, including stopping in transit, replevin, rescission and reclamation.

4.2 You hereby warrant and represent that you are solvent; that except for
Permitted Liens that this Security Interest constitutes and shall at all times
constitute a first and only lien on the Inventory; that you are, or will be at
the time additional Inventory is acquired by you, the absolute owner of the
Inventory with full right to pledge, sell, consign, transfer and create a
Security Interest therein, free and clear of any and all claims or liens in
favor of others; that you will at your expense forever warrant and, at our
request, defend the same from any and all claims and demands of any other
person; and that you will not grant, create or permit to exist, any lien upon or
security interest in the Inventory, or any proceeds, in favor of any other
person.

4.3 You agree to comply with the requirements of all state and federal laws in
order to grant to us a valid and perfected first Security Interest in the
Inventory. To the extent permitted by applicable law, you irrevocably authorize
us to file financing statements and all amendments and continuations with
respect thereto, all in order to create, perfect or maintain our Security
Interest in the Inventory, and you hereby ratify and confirm any and all
financing statements, amendments and continuations with respect thereto
heretofore and hereafter filed by us pursuant to the foregoing authorization.
You agree to do whatever we may reasonably request, from time to time, by way
of; leasing warehouses; filing notices of lien, financing statements,
amendments, renewals and continuations thereof; cooperating with our agents and
employees; keeping Inventory records; obtaining waivers from landlords and
mortgagees; and performing such further acts as we may require in order to
effect the purposes of this agreement.

4.4 You agree to maintain insurance on the Inventory under such policies of
insurance, with such insurance companies, in such amounts and covering such
risks as are at all times reasonably satisfactory to us. All policies covering
the Inventory are to be made payable to us, in case of loss, under a standard
non-contributory "mortgagee", "lender" or "secured party" clause and are to
contain such other provisions as we may require to fully protect our interest in
the Inventory and to any payments to be made under such policies. All original
policies or true copies thereof are to be delivered to us, premium prepaid, with
the loss payable endorsement in our favor, and shall provide for not less than
thirty (30) days prior written notice to us of the exercise of any right of
cancellation. At your request,

<PAGE>

or if you fail to maintain such insurance, we shall arrange for such insurance,
but at your expense and without any responsibility on our part for: obtaining
the insurance, the solvency of the insurance companies, the adequacy of the
coverage, or the collection of claims. The insurance we purchase may not pay any
claims made by you or against you in connection with your Inventory. You are
responsible for the costs of this insurance, including interest and any other
charges we may impose in connection with the purchase of this insurance. The
costs of this insurance may be more than insurance you can buy on your own. You
may still obtain insurance of your own choosing, subject to the terms and
conditions of this paragraph 4.4, on the Inventory. If you provide us with proof
that you have obtained adequate insurance on your Inventory, we will cancel the
insurance that we purchased and refund or credit any unearned premiums to you.
In the event that we purchase such insurance, we will notify you of said
purchase within thirty (30) days after the date of such purchase. If, within
thirty (30) days after the date notice was sent to you, you provide us with
proof that you had adequate insurance on your Inventory as of the date we also
purchased insurance and that you continue to have the insurance that you
purchased yourself, we will cancel the insurance that we purchased without
charging you any costs, interest, or other charges in connection with the
insurance that we purchased. Unless we shall otherwise agree with you in
writing, we shall have the sole right, in our name or yours, to file claims
under any insurance policies, to receive, receipt and give acquittance for any
payments that may be payable thereunder, and to execute any and all
endorsements, receipts, releases, assignments, reassignments or other documents
that may be necessary to effect the collection, compromise or settlement of any
claims under any such insurance policies.

4.5 You agree to pay, when due, all taxes, assessments, claims and other charges
(herein "taxes") lawfully levied or assessed upon the Inventory unless such
taxes are being diligently contested in good faith by you by appropriate
proceedings and adequate reserves are established in accordance with GAAP.
Notwithstanding the foregoing, if such taxes remain unpaid after the date fixed
for the payment thereof, and a lien therefor shall be claimed which in our
opinion would be likely to create a valid obligation having priority over the
rights granted to us herein, we may then, without notice to you, on your behalf,
pay such taxes, and the amount thereof shall be an Obligation secured hereby and
due to us on demand. "GAAP" shall mean generally accepted accounting principles
in the United States of America as in effect from time to time and for the
period as to which such accounting principles are to apply.

4.6 Any and all fees, out of pocket costs and expenses, of whatever kind and
nature, (including any taxes, attorneys' fees or costs for insurance of any
kind), which we may incur in filing public notices; in preparing or filing
documents, making title examinations; in protecting, maintaining, or preserving
the Inventory; in enforcing or foreclosing the Security Interest hereunder,
whether through judicial procedures or otherwise; or in defending or prosecuting
any actions or proceedings arising out of or related to our transactions with
you under this arrangement, shall be borne and paid by you. If same are not
promptly paid by you, we may pay same on your behalf, and the amount thereof
shall be an Obligation secured hereby and due to us on demand.

4.7 You agree to comply with all acts, rules, regulations, and orders of any
legislative, administrative or judicial body or official, applicable to the
Inventory or any part thereof, or to the operation of your business; provided
that you may contest any acts, rules,

<PAGE>

regulations, orders and directions of such bodies or officials in any reasonable
manner which will not, in our opinion, adversely affect our rights or priority
in the Collateral hereunder.

         5. BOOKS AND RECORDS AND EXAMINATIONS

5.1 You agree to maintain Books and Records pertaining to the Inventory in such
detail, form and scope as we shall reasonably require. "Books and Records" means
your accounting and financial records (whether paper, computer or electronic),
data, tapes, discs, or other media, and all programs, files, records and
procedure manuals relating thereto, wherever located.

5.2 You agree that we or our agents may enter upon your premises on reasonable
prior notice (unless there has occurred an Event of Default which is continuing)
at any time during normal business hours, and from time to time, for the purpose
of inspecting the Inventory and any and all Books and Records pertaining
thereto. You agree to notify us promptly of any change in your name, mailing
address, principal place of business or the location of the Inventory. You are
also to advise us promptly, in sufficient detail, of any substantial change
relating to the type, quantity or quality of the Inventory, or any event which
would have a Material Adverse Effect on the value of the Inventory or on the
Security Interest granted to us herein.

5.3 You agree to: execute and deliver to us, from time to time, solely for our
convenience in maintaining a record of the Inventory, such consignments or
written statements as we may reasonably require, designating, identifying or
describing the Inventory pledged to us hereunder. Your failure, however, to
promptly give us such consignments, or other statements shall not affect,
diminish, modify or otherwise limit our Security Interest in the Inventory.

         6. EVENTS OF DEFAULT AND REMEDIES UPON DEFAULT

6.1 It is an "Event of Default" under this agreement if: (a) any representation
or warranty is incorrect in any material respect or you breach any covenant
contained in this agreement; (b) you fail to pay any Obligation when due; or (c)
the occurrence and continuance of an Event of Default (as defined therein) under
the Agreement.

6.2 After the occurrence of an Event of Default which is not waived by us, we
shall have the right, with or without notice to you, to foreclose the Security
Interest created herein by any available judicial procedure, or to take
possession of the Inventory without judicial process, and to enter any premises
where the Inventory may be located for the purpose of taking possession of or
removing the Inventory. We shall have the right, without notice or
advertisement, to sell, lease, or otherwise dispose of all or any part of the
Inventory, whether in its then condition or after further preparation or
processing, in your name or in ours, or in the name of such party as we may
designate, either at public or private sale or at any broker's board, in lots or
in bulk, for cash or for credit, with or without warranties or representations,
and upon such other terms and conditions as we in our sole discretion may deem
advisable, and we shall have the right to purchase at any such sale. If notice
of intended disposition of any said Inventory is required by law, ten (10) days
notice shall constitute reasonable notification. If any Inventory shall require
maintenance, preparation,

<PAGE>

or is in process or other unfinished state, we shall have the right, at our
option, to do such maintenance, preparation, processing or completion of
manufacturing, for the purpose of putting the Inventory in such saleable form as
we shall deem appropriate. You agree, at our request, to assemble the Inventory
and to make it available to us at places which we shall select, whether at your
premises or elsewhere, and to make available to us your premises and facilities
for the purpose of our taking possession of, removing or putting the Inventory
in saleable form. The proceeds of any such sale, lease or other disposition of
the Inventory shall be applied first, to the expenses of taking, holding,
storing, processing, preparing for sale, selling, and the like, and then to the
satisfaction of your Obligations to us, application as to particular Obligations
or as to principal or interest to be in our sole discretion. You shall be liable
to us for, and shall pay to us on demand, any deficiency which may remain after
such sale, lease or other disposition, and we in turn agree to remit to you, or
your successors or assigns, any surplus resulting therefrom. The enumeration of
the foregoing rights is not intended to be exhaustive and the exercise of any
right shall not preclude the exercise of any other rights, all of which shall be
cumulative.

6.3 To the extent that your Obligations are now or hereafter secured by any
assets or property other than the Inventory, or by the guarantee, endorsement,
assets or property of any other person, then we shall have the right in our sole
discretion to determine which rights, security, liens, security interests or
remedies we shall at any time pursue, foreclose upon, relinquish, subordinate,
modify or take any other action with respect to, without in any way modifying or
affecting any of them, or of any of our rights hereunder.

         7. TERMINATION

         The rights and Security Interest granted to us hereunder are to
continue in full force and effect, notwithstanding the fact that the account
maintained in your name on our books may from time to time be temporarily in a
credit position, until termination of the Agreement and the final payment in
full of all Obligations due us by you.

         8. MISCELLANEOUS PROVISIONS

8.1 This agreement and all attendant documentation, as the same may be amended
from time to time, constitutes the entire agreement between us with regard to
the subject matter hereof and supersedes any prior agreements or understandings.
This agreement can be changed only by a writing signed by both of us and our
failure or delay in exercising any of our rights hereunder will not constitute a
waiver thereof, unless such waiver is in writing and signed by us, or bar us
from exercising any of our rights at any time. No course of dealing between us
shall change or modify this agreement. A waiver on any one occasion shall not be
construed as a bar to or waiver of any right or remedy on any future occasion.
The validity, interpretation and enforcement of this agreement shall be governed
by the laws of the State of New York.

8.2 This agreement binds and benefits each of us and our respective successors
and assigns, provided, however, that you may not assign this agreement or your
rights hereunder without our prior written consent.

8.3 If any provision of this agreement is contrary to, prohibited by, or deemed
invalid under, applicable laws or regulations, such provision will be
inapplicable and deemed

<PAGE>

omitted to such extent, but the remainder will not be invalidated thereby and
will be given effect so far as possible.

                [REMAINDER OF THIS PAGE INTENTIONALLY LEFT BLANK]

<PAGE>

         9. JURY TRIAL WAIVER

         TO THE EXTENT PERMITTED BY APPLICABLE LAW, WE EACH HEREBY WAIVE ANY
RIGHT TO A TRIAL BY JURY IN ANY ACTION OR PROCEEDING ARISING DIRECTLY OR
INDIRECTLY OUT OF THIS AGREEMENT, OR ANY OTHER AGREEMENT OR TRANSACTION BETWEEN
US OR TO WHICH WE ARE PARTIES.

If the foregoing is in accordance with your understanding, please so indicate by
signing and returning to us the original and one copy of this agreement. The
agreement shall take effect as of the date set forth above, after being accepted
below by one of our officers after which we shall forward a fully executed copy
to you for your files.

                                    Very truly yours,

                                    THE CIT GROUP/COMMERCIAL SERVICES, INC.

                                    By /s/ Richard Lyons
                                       -----------------------------------------
                                       Name: Richard Lyons
                                       Title: Vice President

Read and Agreed to:

R.G. BARRY CORPORATION

By /s/ Thomas M. Von Lehman
   ---------------------------------
   Name: Thomas M. Von Lehman
   Title: President and CEO

                                    Accepted at:  New York, New York

                                    THE CIT GROUP/COMMERCIAL SERVICES, INC.

                                    By /s/ Stephen Leavenworth
                                       -----------------------------------------
                                       Name: Stephen Leavenworth
                                       Title: Sr. Vice President