<PAGE>

                                                                     Exhibit 4.8

                        PERRY ELLIS INTERNATIONAL, INC.

                         PLEDGE AND SECURITY AGREEMENT

                          Dated as of March 22, 2002

                                     From

                           The Grantors party hereto

                                      to

                      STATE STREET BANK AND TRUST COMPANY
                              as Collateral Agent
<PAGE>

                               TABLE OF CONTENTS
                               ----- -- --------

<TABLE>
<CAPTION>
Section                                                                     Page
-------                                                                     ----
<S>                                                                         <C>
Section 1.   Grant of Security.........................................       1
Section 2.   Security for Obligations..................................       2
Section 3.   Release of Collateral; Pledge of Additional Collateral....       3
Section 4.   Collateral Proceeds.......................................       5
Section 5.   Grantors Remain Liable....................................       5
Section 6.   Representations and Warranties............................       5
Section 7.   Further Assurances........................................       9
Section 8.   Post-Closing Changes; Collections on Licenses.............       9
Section 9.   Place of Perfection; Records..............................      10
Section 10.  Further Actions Regarding Collateral......................      10
Section 11.  Transfers and Other Liens.................................      12
Section 12.  Collateral Agent Appointed Attorney-in-Fact...............      12
Section 13.  Collateral Agent May Perform..............................      13
Section 14.  The Collateral Agent's Duties.............................      13
Section 15.  Remedies..................................................      14
Section 16.  Indemnity and Expenses....................................      15
Section 17.  Amendments; Waivers; Additional Grantors..................      16
Section 18.  Notices...................................................      16
Section 19.  Continuing Security Interest..............................      17
Section 20.  Execution in Counterparts.................................      17
Section 21.  Governing Law.............................................      17
</TABLE>

Schedules

Schedule I   -  Jurisdiction of Formation, Chief Executive Offices and Federal
                Tax Identification Numbers of the Grantors
Schedule II  -  Trademarks
Schedule III -  Licenses

Exhibits

Exhibit A    -  Form of Security Agreement Supplement
Exhibit B    -  Form of Intellectual Property Security Agreement
Exhibit C    -  Form of Intellectual Property Security Agreement Supplement

                                       i
<PAGE>

                         PLEDGE AND SECURITY AGREEMENT

          PLEDGE AND SECURITY AGREEMENT dated as of March 22, 2002 (this
"Agreement") made by Perry Ellis International, Inc., a Florida corporation (the
"Company"), Jantzen Apparel Corp., a Delaware corporation ("Jantzen") and the
Additional Grantors (as defined in Section 16) (the Company, Jantzen and the
Additional Grantors being, collectively, the "Grantors"), to State Street Bank
and Trust Company, a Massachusetts trust company as collateral agent (the
"Collateral Agent") for holders of the Secured Obligations (as defined below).

          PRELIMINARY STATEMENTS.

          (1) The Company has issued $57,000,000 of its 9 1/2% Senior Secured
Notes due 2009 and may issue additional notes (the "Additional Notes") provided
that the aggregate principal amount of all notes does not exceed $75,000,000
(collectively, the "Senior Secured Notes"), pursuant to an Indenture dated as of
March 22, 2002 (as supplemented from time to time, the "Indenture") among the
Company, Jantzen and the Collateral Agent.

          (2) Pursuant to the Indenture, the Grantors are entering into this
Agreement in order to grant to the Collateral Agent, for the equal and ratable
benefit of the holders of the Senior Secured Notes (the "Secured Parties"), a
security interest in certain personal property of the Grantors as provided
herein, now owned or hereafter acquired, to secure, subject to the terms and
conditions of this Agreement, the payment of all of the Grantors' obligations
owing with respect to the Senior Secured Notes, the Indenture and the documents
delivered in connection therewith (the "Debt Instruments").

          (3) Terms defined in the Indenture and not otherwise defined in this
Agreement are used in this Agreement as defined in the Indenture.  Further,
unless otherwise defined in this Agreement or in the Indenture, terms used
herein have the definitions ascribed to such terms in Article 8 or 9 of the
Uniform Commercial Code in effect in the State of New York (the "N.Y. Uniform
Commercial Code").

          NOW, THEREFORE, in consideration of the premises and in order to
fulfill their obligations under the Debt Instruments, each Grantor hereby agrees
with the Collateral Agent for the equal and ratable benefit of the Secured
Parties as follows:

          Section 1.  Grant of Security.

          Each Grantor hereby assigns and pledges to the Collateral Agent for
its benefit and the equal and ratable benefit of the Secured Parties, and hereby
grants to the Collateral Agent for its benefit and the equal and ratable benefit
of the Secured Parties a security interest in, such Grantor's right, title and
interest in and to the following, in each case, as to each type of property
described below, owned by such Grantor, wherever located and existing on the
date of this Agreement (collectively, the "Collateral"):

          (a) all trademarks, service marks, collective marks, trade dress,
     logos, slogans, designs, domain names, trade names, business names,
     corporate names and other

                                       1
<PAGE>

     source identifiers, whether or not registered, whether currently in use or
     not, including, without limitation, all common law rights and registrations
     and applications for registration thereof, including, without limitation,
     the trademark registrations and trademark applications set forth in
     Schedule II hereto (as such Schedule II may be supplemented from time to
     time by supplements to this Agreement, each such supplement being in
     substantially the form of Exhibit C hereto (an "IP Security Agreement
     Supplement"), executed and delivered by such Grantor to the Collateral
     Agent from time to time), and all other marks registered in or applied for
     in the U.S. Patent and Trademark Office or in any office or agency of any
     State or Territory of the United States or any foreign country (but
     excluding any United States intent-to-use trademark application to the
     extent that, and solely during the period in which, the grant of a security
     interest therein would impair the validity and enforceability of such
     intent-to-use trademark applications under applicable law) and all rights
     therein provided by international treaties or conventions, all extensions
     and renewals of any of the foregoing, together in each case with the
     goodwill of the business connected therewith and symbolized thereby, and
     all rights corresponding thereto throughout the world and all other rights
     of any kind whatsoever of such Grantor accruing thereunder or pertaining
     thereto (the "Trademarks");

          (b) all agreements, permits, consents, orders and franchises, in each
     case relating to the Trademarks, including, without limitation, the license
     agreements set forth in Schedule III (as such Schedule III may be
     supplemented from time to time by IP Security Agreement Supplements
     executed and delivered by such Grantor to the Collateral Agent from time to
     time or otherwise), and all proceeds, income, royalties and other payments
     now or hereafter due and/or payable with respect thereto, subject, in each
     case, to the terms of such agreements, permits, authorizations and
     franchises (the "Licenses");

          (c) any and all claims for damages and injunctive relief for past,
     present and future infringement, dilution, misappropriation, violation,
     misuse or breach with respect to the Collateral with the right, but not the
     obligation, to sue for and collect, or otherwise recover, such damages; and

          (d) all proceeds of collateral for, and supporting  obligations
     relating to, any and all of the Collateral (including, without limitation,
     proceeds, collateral and supporting obligations that constitute property of
     the types described in clauses (a) through (c) of this Section 1 and, to
     the extent not otherwise included, all (i) payments under insurance with
     respect to the Collateral (whether or not the Collateral Agent is the loss
     payee thereof), or any damages, indemnity, warranty or guaranty, payable by
     reason of loss or damage to or otherwise with respect to any of the
     foregoing Collateral and (ii) cash proceeds of the foregoing Collateral.

          Section 2.  Security for Obligations.

          This Agreement secures the payment of all obligations of each Grantor
now or hereinafter existing under the Debt Instruments, whether direct or
indirect, absolute or contingent, and whether for principal, reimbursement
obligations, interest, fees, premiums,

                                       2
<PAGE>

penalties, indemnifications, contract causes of action, costs, expenses or
otherwise (all such obligations being the "Secured Obligations").

          Section 3.  Release of Collateral; Pledge of Additional Collateral

          (a) Upon (i) any Collateral Sale (as defined in the Indenture) of any
item of Collateral of any Grantor in accordance with the terms of the Indenture,
or (ii) the making of a Collateral Sale Proceeds Offer (as defined in the
Indenture), the Collateral Agent will, at such Grantor's request and expense,
take all necessary actions to release such item of Collateral from the security
interest granted under this Agreement; provided, however, that:

          (1) at the time of such request and such release no Default (as
          defined in the Indenture) shall have occurred and be continuing under
          the Indenture;

          (2) such Grantor shall have delivered to the Collateral Agent, at
          least 10 Business Days prior to the date of the proposed release, (i)
          a written request for release describing the item of Collateral and
          the terms of the Collateral Sale in reasonable detail, including,
          without limitation, the price and any expenses incurred in connection
          with the transaction, (ii) a form of release for execution by the
          Collateral Agent and (iii) an Officers' Certificate (as defined in the
          Indenture) stating (A) that the Collateral Sale is in compliance with
          the Indenture; (B) except where the Collateral is cash, the Collateral
          Fair Market Value (as defined in the Indenture) of the item of
          Collateral to be released, and (iii) other matters as the Collateral
          Agent may reasonably request; and

          (3) the proceeds of any such Collateral Sale required to be applied,
          or any payment to be made in connection therewith, in accordance with
          the Indenture, to the extent so required, be paid.

          (b) Upon any sale, transfer or other disposition of any license
agreement or the right to receive royalties or payments under such license
agreements of any Grantor in accordance with the provisions of the Indenture to
a Foreign Subsidiary (as defined in the Indenture), the Collateral Agent will,
at such Grantor's request and expense, take all necessary actions to release
such item of Collateral from the security interest granted under the Security
Agreement; provided, however, that:

          (1) at the time of such request and such release, no Default shall
          have occurred and be continuing under the Indenture;

          (2) such Grantor shall have delivered to the Collateral Agent, at
          least 10 Business Days prior to the date of the proposed release, (i)
          a written request for release describing each license agreement or the
          right to receive royalties or payments under such license agreements
          to be released and the terms of the sale, transfer or other
          disposition in reasonable detail, including, without limitation, the
          price and any expenses incurred in connection with the transaction,
          (ii) a form of release for execution by the Collateral Agent and (iii)
          an Officers' Certificate stating (A) that the transaction is in
          compliance with the Indenture; (B) the Collateral Fair Market Value of
          the license agreements to be released or the rights to receive
          royalties or

                                       3
<PAGE>

          payments under the license agreements to be released; and (C) other
          matters as the Collateral Agent may reasonably request; and

          (3) the Company complies with its obligation to pledge Additional
          Collateral in accordance with subsection 3(c) of this Security
          Agreement.

          (c) Each Grantor shall be obligated to pledge additional Collateral
that has a Collateral Fair Market Value, which, together with the Collateral
Fair Market Value of existing Collateral, shall exceed the aggregate principal
amount of Senior Secured Notes outstanding by 100%, and are free and clear of
all Liens other than Liens permitted by the paragraph (b) in the definition of
"Permitted Liens" contained in the Indenture (the "Additional Collateral"):

          (1) within 20 Business Days after the last day of every other fiscal
          year, beginning January 31, 2004, where the Collateral Fair Market
          Value on such date does not exceed the aggregate principal amount of
          Senior Secured Notes outstanding on such date by 100%;

          (2) within 20 Business Days after the closing date of the sale,
          transfer or other disposition of any license agreement or rights to
          receive royalties or payments under such license agreements by a
          Grantor to a Foreign Subsidiary if the amount that is equal to (A) the
          Collateral Fair Market Value (as certified on a date within six months
          prior to such closing date) minus (B) the Collateral Fair Market Value
          of the license agreements or rights to receive royalties or payments
          under all the license agreements released by the Collateral Agent from
          the security interests granted under this Agreement on such closing
          date does not exceed the aggregate principal amount of Senior Secured
          Notes outstanding on such closing date by 100%;

          (3) within 20 Business Days after the closing date of a Collateral
          Sale Proceeds Offer (as defined in the Indenture) whereby (i) less
          than 100% in aggregate principal amount of the Senior Secured Notes
          have been purchased and (ii) the amount that is equal to (A) the
          Collateral Fair Market Value on a date within six months of such
          closing date minus (B) the Collateral Sale Proceeds (as defined in the
          Indenture) used in the Collateral Sale Proceeds Offer does not exceed
          the aggregate principal amount of the Senior Secured Notes outstanding
          on such date by 100%; and

          (4) within 20 Business Days after the closing date of an Additional
          Notes offering if the Collateral Fair Market Value on a date within
          six months of such closing date does not exceed the aggregate
          principal amount of the Senior Secured Notes outstanding on such
          closing date by 100%.

          (d) In addition to this obligation to secure the Senior Secured Notes
with Additional Collateral, at its option, any Grantor may, from time to time,
pledge Additional Collateral to the Collateral Agent.

                                       4
<PAGE>

          Section 4. Collateral Proceeds.

          Subject to the terms of the Indenture and the Inter-Creditor
Agreement, if an Event of Default shall have occurred and be continuing:

          (a) Upon receipt of notice from the Collateral Agent, each Grantor
will (i) immediately instruct each Person obligated at any time to make any
payment relating to the Collateral to such Grantor for any reason (an "Obligor")
to make such payment to an alternative deposit account to be designated by the
Collateral Agent in the notice from the Collateral Agent (the "Other Deposit
Account") and (ii) deposit in the Other Deposit Account, at the end of each
Business Day, all proceeds of Collateral of such Grantor.

          (b) The Collateral Agent shall have the sole right to direct the
disposition of funds with respect to the Other Deposit Account; and it shall be
a term and condition of such Other Deposit Account that no amount will be paid
or released to or for the account of, or withdrawn by or for the account of, the
Company or any other Person from the Other Deposit Account.

          (c) The Collateral Agent may, at any time and without notice to, or
consent from, any Grantor, (i) transfer, or direct the transfer of, funds from
the Other Deposit Account to satisfy any Grantor's obligations under the
Indenture.

          Section 5.  Grantors Remain Liable.

          Anything herein to the contrary notwithstanding, (a) each Grantor
shall remain liable under the contracts and agreements included in such
Grantor's Collateral to the extent set forth therein to perform all of its
duties and obligations thereunder to the same extent as if this Agreement had
not been executed, (b) the exercise by the Collateral Agent of any of the rights
hereunder shall not release any Grantor from any of its duties or obligations
under the contracts and agreements included in the Collateral and (c) no Secured
Party shall have any obligation or liability under the contracts and agreements
included in the Collateral by reason of this Agreement or any other Debt
Instrument, nor shall any Secured Party be obligated to perform any of the
obligations or duties of any Grantor thereunder or to take any action to collect
or enforce any claim for payment.

          Section 6.  Representations and Warranties.

          (a) The Company represents and warrants that all payment and other
     obligations arising under the Revocable Credit Facility Agreement dated May
     26,1995, between Supreme International Corporation (now, the Company) and
     Hamilton Bank N.A. have been paid and satisfied in full and Hamilton Bank
     N.A. has no further claim, whether contingent or otherwise, against the
     Company in respect of such obligations.

          (b) Each Grantor represents and warrants that, except as otherwise set
     forth in the Indenture:

          (1) Such Grantor's exact legal name, as defined in Section 9-503(a) of
          the UCC, is correctly set forth in Schedule I hereto.  Such Grantor is
          located (within the

                                       5
<PAGE>

          meaning of Section 9-307 of the UCC) and has its chief executive
          office at the address specified therefor in Schedule I hereto, as such
          Schedule I may be amended from time to time. Such Grantor's federal
          tax identification number, if any, is set forth opposite such
          Grantor's name in Schedule I hereto.

          (2) Such Grantor is the legal and beneficial owner of the Collateral
          of such Grantor free and clear of any Lien, claim, option or right of
          others, except for the security interest created under this Agreement,
          the Senior Credit Facility Indenture, the Lease, the HSBC L/C and the
          IDB L/C.  No effective financing statement or other instrument similar
          in effect covering all or any part of such Collateral or listing such
          Grantor or any trade name of such Grantor as debtor is on file in any
          recording office, except such as may have been filed in favor of the
          Collateral Agent or as otherwise permitted under the terms of the
          Indenture and the Inter-Creditor Agreement dated March 22, 2002 among
          the Company, Jantzen, the Collateral Agent and the agent for the
          lenders under the Senior Credit Facility, the agent for the funding
          parties under the Lease and the L/C issuers under the HSBC L/C and the
          IDB L/C (the "Inter-Creditor Agreement").  Such Grantor has only the
          trade names listed on Schedule II hereto.

          (3) All filings and other actions necessary or desirable to perfect
          and protect the security interest in the Collateral of such Grantor
          created under this Agreement have been duly made or taken and are in
          full force and effect (or will be made promptly), including, without
          limitation, the filing of financing and continuation statements under
          the Uniform Commercial Code and the recordation of the Intellectual
          Property Security Agreements (defined in Section 10(g) herein) with
          the U.S. Patent and Trademark Office and any other necessary foreign
          authorities and, subject to the terms of the Indenture and the Inter-
          Creditor Agreement, this Agreement creates in favor of the Collateral
          Agent for the benefit of the Secured Parties a valid and, together
          with such filings and other actions, perfected first priority security
          interest in the Collateral of such Grantor, securing the payment of
          the Secured Obligations.

          (4) No authorization or approval or other action by, and no notice to
          or filing with, any governmental authority or regulatory body or any
          other third party is required for (i) the grant by such Grantor of the
          assignment, pledge and security interest granted hereunder or for the
          execution, delivery or performance of this Agreement by such Grantor,
          (ii) the perfection or maintenance of the assignment, pledge and
          security interest created hereunder (including, subject to the terms
          of the Inter-Creditor Agreement, the first priority nature of such
          assignment, pledge or security interest) other than as referred to in
          paragraph (b)(3) above or (iii) for the exercise by the Collateral
          Agent of other rights provided for in this Agreement or the remedies
          in respect of the Collateral pursuant to this Agreement.

          (5) As to itself and its Collateral, except as described in the Inter-
          Creditor Agreement:

                                       6
<PAGE>

          (i)     The rights of such Grantor in or to the Collateral do not
                  conflict with, misappropriate or infringe upon the rights of
                  any third party, and except in the context of routine matters
                  pending before the U.S. Patent and Trademark Office or in
                  corresponding national and international trademark offices
                  which would not have a material adverse effect on such
                  Grantor's business, no claim has been asserted that the use of
                  such Collateral does or may infringe upon the rights of any
                  third party.

          (ii)    Such Grantor is the exclusive owner of the entire and
                  unencumbered right, title and interest in and to the
                  Collateral and is entitled to use all such Collateral without
                  limitation, subject only to (a) recordation with the United
                  States Patent and Trademark Office and corresponding national
                  and international trademark offices, as applicable, of the
                  assignments of registered Trademarks or Licenses to Jantzen or
                  the Company from VF Canada, Inc., in connection with the
                  Jantzen acquisition by the Company and (b) the terms of the
                  Licenses and the Inter-Creditor Agreement.

          (iii)   The Collateral set forth in Schedules II and III hereto
                  includes all of the trademark registrations and applications
                  and Licenses owned by such Grantor.

          (iv)    The Collateral is subsisting and has not been adjudged invalid
                  or unenforceable in whole or part, and to the best of such
                  Grantor's knowledge, is valid and enforceable. Such Grantor is
                  not aware of any uses of any item of Collateral that could be
                  expected to lead to such item becoming invalid or
                  unenforceable.

          (v)     Such Grantor has made or performed all filings, recordings and
                  other acts and has paid all required fees and taxes (other
                  than the recordation of any assignments of Trademarks or
                  Licenses being transferred by VF Canada Inc., to Jantzen or
                  the Company, which recordations the Company shall diligently
                  pursue to completion) to maintain and protect in a
                  commercially reasonable manner its interest in each and every
                  item of Collateral in full force and effect throughout the
                  world, and to protect and maintain in a commercially
                  reasonable manner its interest therein including, without
                  limitation, recordations of any of its interests in the
                  Trademarks with the U.S. Patent and Trademark Office and in
                  corresponding national and international trademark offices.
                  Such Grantor has used proper statutory notice in connection
                  with its use of each Trademark included within the Collateral.

          (vi)    Except in the context of routine matters pending before the
                  U.S. Patent and Trademark Office or in corresponding national
                  and international trademark offices which would not have a
                  material

                                       7
<PAGE>

                  adverse effect on such Grantor's business, no claim, action,
                  suit, investigation, litigation or proceeding has been
                  asserted or is pending or threatened against such Grantor (A)
                  based upon or challenging or seeking to deny or restrict such
                  Grantor's rights in or use of any of the Collateral, or (B)
                  alleging that any services provided by, processes used by, or
                  products manufactured or sold by, such Grantor infringe on or
                  misappropriate any Trademark or any other proprietary right of
                  any third party, or (c) alleging that the collateral is being
                  licensed or sublicensed in violation or contravention of the
                  terms of any license or other agreement. To the best of such
                  Grantor's knowledge, no Person is engaging in any activity
                  that infringes upon or misappropriates, dilutes, misuses or
                  otherwise violates the Collateral or upon the rights of such
                  Grantor therein. Except as set forth on Schedule II or III
                  hereto, such Grantor has not granted any license, release,
                  covenant not to sue, non-assertion assurance, or other right
                  to any Person with respect to any part of the Collateral. The
                  consummation of the transactions contemplated by the Debt
                  Instruments will not result in the termination or impairment
                  of any of the Collateral.

          (vii)   Except as set forth on Schedule III hereto, with respect to
                  each License: (A) such License is valid and binding and in
                  full force and effect and represents the entire agreement
                  between the respective licensor and licensee with respect to
                  the subject matter of such License; (B) such License will not
                  cease to be valid and binding and in full force and effect on
                  terms identical to those currently in effect as a result of
                  the rights and interest granted herein, nor will the grant of
                  such rights and interest constitute a breach or default under
                  such License or otherwise give the licensor or licensee a
                  right to terminate such License; (C) such Grantor has not
                  received any notice of termination or cancellation under such
                  License; (D) such Grantor has not received any notice of a
                  breach or default under such License, which breach or default
                  has not been cured; (E) such Grantor has not granted to any
                  other third party any rights, adverse or otherwise, under such
                  License; and (F) neither such Grantor nor any other party to
                  such License is in breach or default in any material respect,
                  and no event has occurred that, with notice or lapse of time
                  or both, would constitute such a breach or default or permit
                  termination, modification or acceleration under such License.

          (viii)  None of the Collateral is subject to any outstanding order,
                  decree, injunction, judgment or ruling restricting the use
                  thereof or that would impair the validity or enforceability
                  thereof.

                                       8
<PAGE>

          Section 7. Further Assurances.

          (a)  Each Grantor agrees that from time to time, at the sole expense
     of such Grantor, such Grantor will promptly execute and deliver all further
     instruments and documents, and take all further action, that may be
     necessary or desirable, or that the Collateral Agent may request, in order
     to perfect and protect any security interest granted or purported to be
     granted by such Grantor hereunder or to enable the Collateral Agent to
     exercise and enforce its rights and remedies hereunder with respect to any
     Collateral of such Grantor. Without limiting the generality of the
     foregoing, each Grantor shall promptly, with respect to Collateral of such
     Grantor (i) execute and file such financing or continuation statements, or
     amendments thereto, and such other instruments or notices; as may be
     necessary or desirable, or as the Collateral Agent may request, in order to
     perfect and preserve the security interest granted or purported to be
     granted by such Grantor hereunder and (ii) deliver to the Collateral Agent
     evidence that all other action that the Collateral Agent may deem
     reasonably necessary or desirable in order to perfect and protect the
     security interest created by such Grantor under this Agreement has been
     taken.

          (b)  Each Grantor hereby authorizes the Collateral Agent to file one
     or more financing and continuation statements, and amendments thereto, in
     each case without the signature of such Grantor, and to record the
     Intellectual Property Security Agreements, and amendments thereto, relating
     to all or any part of the Collateral of such Grantor without the signature
     of such Grantor where permitted by law. A photocopy or other reproduction
     of this Agreement, the Intellectual Property Security Agreement or any
     financing statement covering the Collateral or any part thereof shall be
     sufficient as a financing statement where permitted by law.

          (c)  Each Grantor will furnish to the Collateral Agent from time to
     time statements and schedules further identifying and describing the
     Collateral of such Grantor and such other reports in connection with such
     Collateral as the Collateral Agent may reasonably request, all in
     reasonable detail.

          Section 8.  Post-Closing Changes; Collections on Licenses.

          (a)  No Grantor will change its name, type of organization,
     jurisdiction of organization, organization identification number or
     location from those set forth in Schedule I of this Agreement without first
     giving at least 30 day's prior written notice to the Collateral Agent and
     taking all action required by the Collateral Agent for the purpose of
     perfecting or protecting the security interest granted by this Agreement.
     No Grantor shall become bound by a security agreement authenticated by
     another Person (determined as provided in Section 9-203(d) of the UCC)
     without giving the Collateral Agent 30 day's prior written notice thereof
     and taking all action required by the Collateral Agent to ensure that the
     perfection and first priority nature of the Collateral Agent's security
     interest in the Collateral is maintained.

          (b)  Except as otherwise provided in this Section (b) and in the
     Inter-Creditor Agreement, each Grantor will continue to collect, at its own
     expense, all amounts due or to become due to such Grantor under the
     Licenses. In connection with such collections,

                                       9
<PAGE>

     such Grantor may take (and, at the direction of the Collateral Agent acting
     in accordance with the terms of the Inter-Creditor Agreement upon the
     occurrence and during the continuance of an Event of Default, will take)
     such action as such Grantor or the Collateral Agent may deem necessary or
     advisable to enforce collection of the amounts due under the Licenses;
     provided, however, that the Collateral Agent shall have the right subject
     to the terms of the Inter-Creditor Agreement, upon the occurrence and
     during the continuance of an Event of Default and upon written notice to
     such Grantor of its intention to do so, to notify the obligors under any
     Licenses of the assignment of such Licenses to the Collateral Agent and to
     direct such obligors to make payment of all amounts due or to become due to
     such Grantor thereunder directly to the Collateral Agent and, upon such
     notification and at the expense of such Grantor, to enforce collection of
     any amounts due under the Licenses, and to adjust, settle or compromise the
     amount or payment thereof, in the same manner and to the same extent as
     such Grantor might have done. After receipt by any Grantor of the notice
     from the Collateral Agent referred to in the proviso to the preceding
     sentence, (i) all amounts and proceeds (including instruments) received by
     such Grantor in respect of the Licenses of such Grantor shall be received
     in trust for the benefit of the Collateral Agent hereunder, shall be
     segregated from other funds of such Grantor and shall be forthwith paid
     over to the Collateral Agent, (ii) such Grantor will not adjust, settle or
     compromise the amount or payment of any License, release wholly or partly
     any obligor thereof, or allow any credit or discount thereon and (iii) such
     Grantor will not permit or consent to the subordination of its right to
     payment under any of the Licenses to any other indebtedness or obligations
     of the obligor thereof.

          Section 9.     Place of Perfection; Records.

          Each Grantor will keep its chief executive office at the location
therefor specified in Section 5(a) or, upon 30 days' prior written notice to the
Collateral Agent, at such other location in a jurisdiction where all actions
required by Section 6 shall have been taken with respect to the Collateral of
such Grantor (and, upon the taking of such action in such jurisdiction, Schedule
I hereto shall be automatically amended to include such other location).  Each
Grantor will hold and preserve its records relating to the Collateral and will
permit representatives of the Collateral Agent at any time during normal
business hours to inspect and make abstracts from such records and other
documents.

          Section 10.    Further Actions Regarding Collateral.

          (a)  Each Grantor shall, within 20 Business Days after the last day of
     every other fiscal year beginning January 31, 2004, deliver to the
     Collateral Agent a certificate signed by its officer and containing the
     Collateral Fair Market Value of its Collateral.

          (b)  With respect to each item of its Collateral, each Grantor agrees
     to take, at its sole expense, all commercially reasonable steps, including,
     without limitation, in the U.S. Patent and Trademark Office and in
     corresponding national and international trademark offices and any other
     governmental authority, to (i) maintain the validity and enforceability of
     each such item of Collateral and maintain each such item of Collateral in
     full force and effect, and (ii) pursue the registration and maintenance of
     each

                                       10
<PAGE>

     trademark registration or application, now or hereafter included in the
     Collateral of such Grantor, including, without limitation, the payment of
     required fees and taxes, the filing of responses to office actions issued
     by any governmental authority, the filing of applications for renewal or
     extension, the filing of affidavits under Sections 8 and 15 of the U.S.
     Trademark Act and applicable law under relevant foreign jurisdictions, the
     filing of renewal applications or extensions, the payment of maintenance
     fees and the participation in opposition, cancellation, infringement and
     misappropriation proceedings. No Grantor shall, without the written consent
     of the Collateral Agent, discontinue use of or otherwise abandon any
     Collateral, or abandon any right to file an application for trademark
     unless such Grantor shall have previously determined that such use or the
     pursuit or maintenance of such Collateral is no longer desirable in the
     conduct of such Grantor's business and that the loss thereof would not be
     reasonably likely to have a material adverse effect on its business, in
     which case, such Grantor will give prompt notice of any such abandonment to
     the Collateral Agent.

          (c)  Each Grantor agrees promptly to notify the Collateral Agent if
     such Grantor learns (i) that any item of the Collateral may have become
     abandoned, placed in the public domain, invalid or unenforceable, or of any
     adverse determination or development regarding such Grantor's ownership of
     any of the Collateral, or its right to register the same or to keep and
     maintain and enforce the same, or (ii) of any adverse determination or the
     institution of any proceeding (including, without limitation, the
     institution of any proceeding before the Trademark Trial and Appeals Board
     of the United States Patent and Trademark Office, corresponding national
     and international trademark tribunals or any court) regarding any item of
     the Collateral.

          (d)  In the event that any Grantor becomes aware that any item of the
     Collateral is being infringed or misappropriated by a third party, such
     Grantor shall promptly notify the Collateral Agent and shall take such
     actions, at its sole expense, as such Grantor or the Collateral Agent,
     acting in accordance with the terms of the Inter-Creditor Agreement, deems
     reasonable and appropriate under the circumstances to protect such
     Collateral, including, without limitation, suing for infringement or
     misappropriation and for an injunction against such infringement or
     misappropriation.

          (e)  Each Grantor shall use proper statutory notice in connection with
     its use of each item of its Collateral.  No Grantor shall do or permit any
     act or knowingly omit to do any act whereby any of its Collateral may lapse
     or become invalid or unenforceable or placed in the public domain.

          (f)  Each Grantor shall take all steps that it or the Collateral
     Agent, acting in accordance with the terms of the Inter-Creditor Agreement,
     deems reasonable and appropriate under the circumstances to preserve and
     protect each item of its Collateral, including, without limitation,
     maintaining the quality of any and all products or services used or
     provided in connection with any of the Trademarks, consistent with the
     quality of the products and services as of the date hereof, and taking all
     steps necessary to ensure that all licensed users of any of the Trademarks
     use such consistent standards of quality.

                                       11
<PAGE>

          (g)  With respect to its Collateral, each Grantor agrees to execute an
     agreement, in substantially the form set forth in Exhibit B hereto (an
     "Intellectual Property Security Agreement"), for recording the security
     interest granted hereunder to the Collateral Agent in such Collateral with
     the U.S. Patent and Trademark Office, corresponding national and
     international trademark offices and any other governmental authorities
     necessary to perfect the security interest hereunder in such Collateral.

          (h)  Each Grantor agrees that, should it obtain an ownership interest
     in any item of the type set forth in Sections 1(a) to (d) that is not on
     the date hereof a part of the Collateral and it is obligated to pledge or
     chooses to pledge Additional Collateral in accordance with the provisions
     of Section 3 of this Agreement, (i) the provisions of Section 1 shall
     automatically apply thereto, (ii) any Additional Collateral and in the case
     of trademarks, the goodwill of the business connected therewith or
     symbolized thereby, shall automatically become part of the Collateral,
     subject to the terms and conditions of this Agreement with respect thereto,
     (iii) such Grantor shall give prompt written notice thereof to the
     Collateral Agent in accordance herewith and (iv) such Grantor shall execute
     and deliver to the Collateral Agent an IP Security Agreement Supplement
     covering such Additional Collateral as "Additional Collateral" thereunder
     and as defined therein, for recordation with the U.S. Patent and Trademark
     Office, corresponding national and international trademark offices and any
     other governmental authorities necessary to perfect the security interest
     hereunder in such Additional Collateral.

          Section 11.    Transfers and Other Liens.

          Each Grantor agrees that it will not (i) sell, assign (by operation of
law or otherwise) or otherwise dispose of, or grant any option with respect to,
any of the Collateral, other than sales, assignments and other dispositions of
Collateral, and options relating to Collateral, permitted under the terms of the
Debt Instruments, or (ii) create or suffer to exist any Lien upon or with
respect to any of the Collateral of such Grantor except for the pledge,
assignment and security interest created under this Agreement and Permitted
Liens, including, without limitation, security interests in favor of the agent
for the lenders party to the Senior Credit Facility.

          Section 12.    Collateral Agent Appointed Attorney-in-Fact.

          Each Grantor hereby irrevocably appoints the Collateral Agent such
Grantor's attorney-in-fact, with full authority in the place and stead of such
Grantor and in the name of such Grantor or otherwise, from time to time and in
accordance with the Inter-Creditor Agreement upon the occurrence and during the
continuance of an Event of Default, in the Collateral Agent's discretion, to
take any action and to execute any instrument that the Collateral Agent may deem
necessary or advisable to accomplish the purposes of this Agreement, including,
without limitation:

          (a)  to ask for, demand, collect, sue for, recover, compromise,
     receive and give acquittance and receipts for monies due and to become due
     under or in respect of any of the Collateral; and

                                       12
<PAGE>

          (b)  to file any claims or take any action or institute any
     proceedings that the Collateral Agent may deem necessary or desirable for
     the collection of any of the Collateral or otherwise to enforce compliance
     with the terms and conditions of any of the rights of the Collateral Agent
     with respect to any of the Collateral.

          Section 13.    Collateral Agent May Perform.

          If any Grantor fails to perform any of its obligations under this
Agreement contained herein, the Collateral Agent may, as the Collateral Agent
deems necessary to protect the security interest granted hereunder in the
Collateral or to protect the value thereof, but without any obligation to do so
(and upon the occurrence and during the continuation of an Event of Default
without notice), itself perform, or cause performance of, such agreement, and
the reasonable expenses of the Collateral Agent incurred in connection therewith
shall be payable by such Grantor.

          Section 14.    The Collateral Agent's Duties.

          (a)  The powers conferred on the Collateral Agent hereunder are solely
     to protect the Secured Parties' interest in the Collateral and shall not
     impose any duty upon it to exercise any such powers.  Except for the safe
     custody of any Collateral in its possession and the accounting for moneys
     actually received by it hereunder, the Collateral Agent shall have no duty
     as to any Collateral, or as to the taking of any necessary steps to
     preserve rights against any parties or any other rights pertaining to any
     Collateral.  The Collateral Agent shall be deemed to have exercised
     reasonable care in the custody and preservation of any Collateral in its
     possession if such Collateral is accorded treatment substantially equal to
     that which it accords its own property.

          (b)  Without limiting the foregoing, the parties agree that:

               (1)  The Collateral Agent undertakes to perform such duties and
          only such duties as are specifically set forth in this Agreement and
          no implied covenants or obligations shall be read in this Agreement
          against the Collateral Agent.

               (2)  The Collateral Agent shall not be required to exercise any
          remedies hereunder unless requested in writing to do so by the Secured
          Parties and in such event subject to the terms of this Agreement and
          only if furnished with indemnity satisfactory to the Collateral Agent
          by the Secured Parties.  The Collateral Agent may consult with
          counsel, including in-house counsel, and shall not be liable for any
          action taken in good faith in reliance upon the advice of counsel.
          The Collateral Agent makes no representation concerning the value or
          validity of the Collateral or the validity or perfection of the pledge
          thereof.

               (3)  The Collateral Agent (i) shall not be obligated hereunder to
          take or hold possession of any Collateral other than the Pledged
          Collateral, (ii) has no responsibility for the sufficiency, contents
          or filing of any financing statement or continuation statement, (iii)
          shall have no responsibility for the genuineness, marketability, or
          sufficiency of or title to any Collateral, (iv) shall have no

                                       13
<PAGE>

          responsibility to take or cause to be taken any action necessary to
          perfect the security interest contemplated hereby (other than to
          retain physical possession of the Collateral in accordance with the
          terms hereof); and (v) shall not be under any duty or obligation to
          determine, compel or enforce performance or compliance by the Grantor
          with this Agreement.

               (4)  No grant of power or authority to the Collateral Agent, and
          no covenant, representation or warranty by the Grantor to the
          Collateral Agent under this Agreement shall be construed to impose or
          imply any duty or obligation on the part of the Collateral Agent to
          exercise or enforce any such power, authority, covenant,
          representation or warranty.

               (5)  The Collateral Agent (i) shall not be obligated to take any
          legal or other action hereunder which might in its judgment involve or
          cause it to incur any expense or liability unless it shall have been
          furnished with acceptable indemnification; (ii) may rely on and shall
          be protected in acting or refraining from acting upon any written
          notice, instruction (including, without limitation, wire transfer
          instructions, whether incorporated herein or provided in a separate
          written instruction), instrument, statement, certificate, request or
          other document furnished to it hereunder and believed by it to be
          genuine and to have been signed or presented by the proper person, and
          shall have no responsibility for determining the accuracy thereof; and
          (iii) shall not be liable to anyone for any action taken or omitted to
          be taken by it hereunder except in the case of the Collateral Agent's
          own gross negligence or willful misconduct in breach of the terms of
          this Agreement.  In no event shall the Collateral Agent be liable for
          indirect, punitive, special or consequential damage or loss (including
          but not limited to lost profits) whatsoever, even if the Collateral
          Agent has been informed of the likelihood of such loss or damage and
          regardless of the form of action.

          Section 15.    Remedies.

          Subject to the terms of the Indenture and the Inter-Creditor
Agreement, if an Event of Default shall have occurred and be continuing:

          (a)  The Collateral Agent may exercise in respect of the Collateral,
     in addition to other rights and remedies provided for herein or otherwise
     available to it, all the rights and remedies of a secured party upon
     default under the N.Y. Uniform Commercial Code (whether or not the N.Y.
     Uniform Commercial Code applies to the affected Collateral) and also may
     (i) require each Grantor to, and each Grantor hereby agrees that it will at
     its expense and upon request of the Collateral Agent forthwith, assemble
     all or part of the Collateral as directed by the Collateral Agent and make
     it available to the Collateral Agent at a place to be designated by the
     Collateral Agent that is reasonably convenient to both parties, (ii)
     without notice except as specified below, sell the Collateral or any part
     thereof in one or more parcels at public or private sale, at any of the
     Collateral Agent's offices or elsewhere, for cash, on credit or for future
     delivery, and upon such other terms as the Collateral Agent may deem
     commercially reasonable, (iii) occupy any premises owned or leased by any
     of the Grantors where the Collateral or any part thereof is

                                       14
<PAGE>

     assembled or located for a reasonable period in order to effectuate its
     rights and remedies hereunder or under law, without obligation to such
     Grantor in respect of such occupation; and (iv) exercise any and all rights
     and remedies of any of the Grantors under or in connection with the
     Licenses or otherwise in respect of the Collateral, including, without
     limitation, (A) any and all rights of such Grantor to demand or otherwise
     require payment of any amount under, or performance of any provision of,
     the Licenses and (B) exercise all other rights and remedies with respect to
     the Collateral, including, without limitation, those set forth in Section
     9-607 of the UCC. Each Grantor agrees that, to the extent notice of sale
     shall be required by law, at least ten days' notice to each Grantor of the
     time and place of any public sale or of the time after which any private
     sale is to be made shall constitute reasonable notification. The Collateral
     Agent shall not be obligated to make any sale of Collateral regardless of
     notice of sale having been given. The Collateral Agent may adjourn any
     public or private sale from time to time by announcement at the time and
     place fixed therefor, and such sale may, without further notice, be made at
     the time and place to which it was so adjourned.

          (b)  All cash proceeds received by the Collateral Agent in respect of
     any sale of, collection from, or other realization upon all or any part of
     the Collateral shall be held by the Collateral Agent as collateral for,
     and/or then or at any time thereafter applied in whole or in part by the
     Collateral Agent against the Secured Obligations.

          (c)  The Collateral Agent may exercise any and all rights and remedies
     of each Grantor in respect of the Collateral.

          (d)  All payments received by each Grantor in respect of the
     Collateral shall be received in trust for the benefit of the Collateral
     Agent, shall be segregated from other funds of each Grantor and shall be
     forthwith paid over to the Collateral Agent.

          Section 16.    Indemnity and Expenses.

          (a)  Each Grantor agrees to indemnify defend and save and hold
     harmless the Collateral Agent and each Secured Party and each of their
     Affiliates, and their respective officers, directors, employees, agents and
     advisors (each an "Indemnified Party") from and against, and shall pay on
     demand, any and all claims, damages, losses, liabilities and expenses
     (including, without limitation, reasonable fees and expenses of counsel)
     that may be incurred by or asserted or awarded against any Indemnified
     Party, in each case arising out of or in connection with or resulting from
     this Agreement (including, without limitation, enforcement of this
     Agreement), except to the extent such claim, damage, loss, liability or
     expense is found in a final, non-appealable judgment by a court of
     competent jurisdiction to have resulted from such Indemnified Party's gross
     negligence or willful misconduct.

          (b)  Each Grantor will upon demand pay to the Collateral Agent the
     amount of any and all reasonable expenses, including, without limitation,
     the reasonable fees and expenses of its counsel and of any experts and
     agents, that the Collateral Agent may incur in connection with (i) the
     administration of this Agreement, (ii) the custody, preservation, use or
     operation of or the sale of, collection from or other realization upon, any
     of the

                                       15
<PAGE>

     Collateral of such Grantor, (iii) the exercise or enforcement of any of the
     rights of the Collateral Agent or the other Secured Parties hereunder or
     (iv) the failure by such Grantor to perform or observe any of the
     provisions hereof.

          Section 17.    Amendments; Waivers; Additional Grantors.

          (a)  No amendment or waiver of any provision of this Agreement, and no
     consent to any departure by any Grantor herefrom, shall in any event be
     effective unless the same shall be in writing and signed by the Collateral
     Agent and then such waiver or consent shall be effective only in the
     specific instance and for the specific purpose for which given.  No failure
     on the part of the Collateral Agent or any other Secured Party to exercise,
     and no delay in exercising any right hereunder, shall operate as a waiver
     thereof; nor shall any single or partial exercise of any such right
     preclude any other or further exercise thereof or the exercise of any other
     right.

          (b)  Upon the execution and delivery by any Person of a security
     agreement supplement in substantially the form of Exhibit A hereto (each a
     "Security Agreement Supplement"), (i) such Person shall be referred to as
     an "Additional Grantor" and shall be and become a Grantor hereunder and
     each reference in this Agreement to "Grantor" shall also mean and be a
     reference to such Additional Grantor, and (ii) the supplemental schedules
     I, II, and III attached to each Security Agreement Supplement shall be
     incorporated into and become a part of and supplement Schedules I, II and
     III respectively, hereto, and the Collateral Agent may attach such
     supplemental schedules to such Schedules; and each reference to such
     Schedules shall mean and be a reference to such Schedules as supplemented
     pursuant to each Security Agreement Supplement.

          Section 18.    Notices.

          All notices and other communications provided for hereunder shall be
in writing (including telecopier communication) and mailed, telecopied, or
delivered to a Grantor, addressed to it at its address set forth opposite such
Grantor's name on the signature page to the Security Agreement or in the case of
each Additional Grantor, addressed to it at its address set forth opposite such
Additional Grantor's name on the Signature Page to the Security Agreement
Supplement pursuant to which it became a party hereto; or, as to any party, at
such other address as shall be designated by such party in a written notice to
the other parties.  All such notices and other communications shall, when mailed
or telecopied, be effective five Business Days after deposited in the mails or
on the Business Day when telecopied, if the transmission of the telecopy is
confirmed before 5:00 P.M. local time on such day, or, if confirmed after that
time, then on the next Business Day after being telecopied, respectively,
addressed as aforesaid; provided, however, that any notices and other
                        --------  -------
communications to the Collateral Agent shall not be effective until received by
the Collateral Agent.  Delivery by telecopier of an executed counterpart of any
amendment or waiver of any provision of this Agreement or of any Security
Agreement Supplement or Schedule hereto shall be effective as delivery of an
original executed counterpart thereof.

                                       16
<PAGE>

          Section 19.    Continuing Security Interest.

          This Agreement shall create a continuing security interest in the
Collateral and shall (a) remain in full force and effect until the release of
all Collateral as provided in Section 3 (b) be binding upon each Grantor, its
successors and assigns and (c) enure, together with the rights and remedies of
the Collateral Agent hereunder, to the benefit of the Collateral Agent, the
Secured Parties and their respective successors, transferees and assigns.

          Section 20.    Execution in Counterparts.

          This Agreement may be executed in any number of counterparts, each of
which when so executed shall be deemed to be an original and all of which taken
together shall constitute one and the same agreement.  Delivery of an executed
counterpart of a signature page to this Agreement by telecopier shall be
effective as delivery of an original executed counterpart of this Agreement.

          Section 21.    Governing Law.

          This Agreement shall be governed by, and construed in accordance with,
the laws of the State of New York.

                                       17
<PAGE>

          IN WITNESS WHEREOF, each Grantor and the Collateral Agent has caused
this Agreement to be duly executed and delivered by its officer thereunto duly
authorized as of the date first above written.

Address for Notices:                            PERRY ELLIS INTERNATIONAL, INC.
-------------------

3000 N.W. 107/th/ Avenue
Miami, FL 33172                                 By:_____________________________
Telecopier No.: (305) 406-0505                     Name:
Attention:      George Feldenkreis                 Title:




3000 N.W. 107/th/ Avenue                        JANTZEN APPAREL CORP.
Miami, FL 33172
Telecopier No.: (305) 406-0505
Attention:      George Feldenkreis              By:_____________________________
                                                   Name:
                                                   Title:

                                       18
<PAGE>

Address for Notices:                            STATE STREET BANK AND TRUST
-------------------                             COMPANY, as Collateral Agent

61 Broadway
15/th/ Floor
New York, NY 10006
Telecopier No.: (212) 612-3202                  By:_____________________________
Attention:      Angelita Pena                       Name:
                                                    Title:

                                       19