<PAGE>

                                                                   Exhibit 10.38


                    FIFTH AMENDMENT TO AMENDED AND RESTATED
                          LOAN AND SECURITY AGREEMENT


          THIS FIFTH AMENDMENT TO AMENDED AND RESTATED LOAN AND SECURITY
AGREEMENT (the "Amendment") is made and entered into effective as of the 14th
day of March, 2002, by and among BANK OF AMERICA, N.A., formerly NationsBank,
N.A., as agent for the financial institutions party to the Loan Agreement (as
hereafter defined) from time to time (the "Agent"), the financial institutions
party to the Loan Agreement from time to time (the "Lenders"), and PERRY ELLIS
INTERNATIONAL, INC., a Florida corporation formerly known as Supreme
International Corporation ("PEI").


                             W I T N E S S E T H :
                             - - - - - - - - - -


          WHEREAS, the Agent, PEI and the Lenders are parties to that certain
Amended and Restated Loan and Security Agreement, dated as of March 26, 1999 (as
amended from time to time, the "Loan Agreement"), pursuant to which the Agent
and the Lenders have agreed to extend certain loans and financial accommodations
to PEI; and

          WHEREAS, an Event of Default exists under Section 12.1(a) of the Loan
Agreement; and

          WHEREAS, PEI has asked the Agent and the Lenders to waive such
existing Event of Default, and the Agent and the Lenders are willing to do so on
the terms set forth herein; and

          WHEREAS, PEI has requested that the Agent and the Lenders consent to
the consummation of the Jantzen Acquisition and the incurrence of the Senior
Note Debt pursuant to the Senior Note Offering (as such terms are hereafter
defined), and the Agent and the Lenders are willing to do so on the terms set
forth herein.

          NOW, THEREFORE, in consideration of the foregoing premises, and other
good and valuable consideration, the receipt and legal sufficiency of which is
hereby acknowledged, the parties hereto hereby agree as follows:

          1.   All capitalized terms used herein and not otherwise expressly
defined herein shall have the respective meanings given to such terms in the
Loan Agreement.
<PAGE>

     2.   In order to induce the Agent and each Lender to enter into this
Amendment and grant the accommodations set forth herein, PEI hereby (a)
represents and warrants that, as of the date hereof, and after giving effect to
the terms hereof, there exists no Default or Event of Default, and (b)
acknowledges and agrees that no right of offset, defense, counterclaim, claim,
causes of action or objection in favor of PEI against the Agent or any Lender
exists arising out of or with respect to any of the Secured Obligations, the
Loan Agreement, or any of the other Loan Documents, or with respect to the
administration or funding of the Secured Obligations.

     3.   PEI hereby acknowledges that, prior to giving effect to this
Amendment, PEI is in default under Section 12.1(a) of the Loan Agreement for the
fiscal period ending on January 31, 2002.  The Agent and the Lenders hereby
waive such Event of Default under Section 12.1(a), but the Agent and the Lenders
expressly reserve their rights and remedies with respect to any other Default or
Event of Default, including, without limitation, any Default or Event of Default
with respect to Section 12.1(a) of the Loan Agreement arising after January 31,
2002.  PEI hereby acknowledges and agrees that the execution and delivery of
this Amendment has not established any course of dealing between PEI and the
Agent and the Lenders or any obligation of the Agent or any Lender with respect
to any future restructuring or modification of the Loan Agreement or the other
Loan Documents or the exercise of the Agent's or any Lender's rights and
remedies thereunder.

     4.   In order to induce the Agent and each Lender to enter into this
Amendment and grant the accommodations set forth herein, PEI hereby covenants
and agrees as follows:

          (a)       Acquisition Documents.  PEI shall furnish to the Agent, (i)
                    ---------------------
promptly following the execution thereof and in any event prior to the Jantzen
Acquisition Date, true, complete and correct copies of the executed Jantzen
Acquisition Agreement, which Jantzen Acquisition Agreement shall be in the form
of the draft Jantzen Acquisition Agreement delivered to the Agent on the date of
this Amendment, except for revisions thereto that do not adversely effect the
Agent or the Lenders in any respect, and (ii) on the Jantzen Acquisition Date,
final versions (in unexecuted form) of such Jantzen Acquisition Documents as the
Agent may request, each of which shall be, upon and after its execution and
delivery, in full force and effect.

          (b)       Senior Note Documents.  PEI shall furnish to the Agent, (i)
                    ---------------------
promptly following the issuance thereof and in any event prior to the incurrence
of the Senior Note Debt, true, complete and correct copies of the Senior Note
Offering Memorandum, which Senior Note Offering Memorandum shall be in the form
of the draft Senior Note Offering Memorandum delivered to the Agent on the date
of this Amendment, except for revisions thereto that do not adversely effect the
Agent or the Lenders in any respect, and (ii) on or prior to the Jantzen
Acquisition Date, final versions (in unexecuted form) of such Senior Note
Documents as the Agent may request, each of which shall be, upon and after its
execution and delivery, in full force and effect.

                                      -2-
<PAGE>

     5.   Based on PEI's representations, warranties, covenants and agreements
set forth in this Amendment and the other Loan Documents, including the
covenants and agreements set forth in Paragraph 4 above, the Agent and the
Lenders hereby consent to the contemporaneous consummation of the Jantzen
Acquisition and the incurrence of the Senior Note Debt on the Jantzen
Acquisition Date, subject to the following conditions:

     (a)  No Default or Event of Default (including without limitation any
Event of Default as described in Paragraph 6 below) shall exist after giving
effect to the consummation of the Jantzen Acquisition and the incurrence of the
Senior Note Debt.

     (b)  After giving pro forma effect to the consummation of the Jantzen
Acquisition and the incurrence of the Senior Note Debt, PEI shall be in
compliance with each of the financial covenants set forth in Section 12.1 (as
amended hereby), and the Agent shall have received a certificate and supporting
calculations from the Financial Officer evidencing such compliance.

     (c)  On the Jantzen Acquisition Date, (i) the Agent shall have received
true and complete executed or conformed copies of the Senior Note Documents;
(ii) the Senior Note Documents shall be in full force and effect and no material
term or condition thereof shall have been amended, modified or waived after the
execution thereof (other than solely to extend the date by which the Senior Note
Debt is to be incurred) except with the prior written consent of the Agent;
(iii) none of the parties to any of the Senior Note Documents shall have failed
to perform any material obligation or covenant required by such Senior Note
Document to be performed or complied with by it on or before the Jantzen
Acquisition Date; (iv) all requisite approvals by Governmental Authorities and
regulatory bodies having jurisdiction over the parties to the Senior Note
Documents in respect of the Senior Note Offering shall have been obtained by
such parties, and no such approvals shall impose any unsatisfied conditions to
the incurrence of the Senior Note Debt; (v) the incurrence of the Senior Note
Debt pursuant to the Senior Note Offering shall be consummated (A) in accordance
with all applicable laws and the terms of the Senior Note Offering Memorandum,
and (B) pursuant to the Senior Note Documents in form consistent with the Senior
Note Offering Memorandum and otherwise satisfactory to the Agent, without any
amendment or waiver of any material provision thereof; (vi) all opinion letters
delivered in connection with the Senior Note Documents and the transactions
contemplated thereby shall be addressed to the Agent, for the benefit of the
Lenders, or accompanied by a written authorization from the firm delivering such
opinion letter stating that the Agent, for the benefit of the Lenders, may rely
on such opinion letter as though it were addressed to it; and (vii) the Agent
shall have received the duly executed and delivered Intercreditor Agreement in
substantially the form attached to this Amendment as Exhibit A (the "Trademark
                                                     ---------
Intercreditor Agreement").

     (d)  On the Jantzen Acquisition Date, (i) the Agent shall have received
true and complete executed or conformed copies of the Jantzen Acquisition
Documents; (ii) the Jantzen Acquisition Documents shall be in full force and
effect and no material term or condition thereof shall have been amended,
modified or waived after the execution thereof (other than solely to extend the
date by which the Jantzen Acquisition is required to occur) except with the
prior written consent of the Agent; (iii) none of the parties to any of the
Jantzen Acquisition

                                      -3-
<PAGE>

Documents shall have failed to perform any material obligation or covenant
required by such Jantzen Acquisition Document to be performed or complied with
by it on or before the Jantzen Acquisition Date; (iv) all material
representations and warranties of the Jantzen Sellers contained in the Jantzen
Acquisition Agreement and the other Jantzen Acquisition Documents shall be true
and correct in all material respects with the same effect as though made on and
as of the Jantzen Acquisition Date; (v) all requisite approvals by Governmental
Authorities and regulatory bodies having jurisdiction over the parties to the
Jantzen Acquisition Agreement in respect of the Jantzen Acquisition shall have
been obtained by such parties (other than the completion of any assignments of
record of Intellectual Property being transferred by VF Canada, Inc. to Jantzen
Apparel or PEI, which assignments PEI shall diligently pursue to completion),
and no such approvals shall impose any unsatisfied conditions to the
consummation of the Jantzen Acquisition; (vi) the Jantzen Acquisition shall have
been consummated (A) in accordance with all applicable laws and the terms of the
Jantzen Acquisition Agreement, and (B) pursuant to the Jantzen Acquisition
Documents in form consistent with the Jantzen Acquisition Agreement and
otherwise satisfactory to the Agent, without any amendment or waiver of any
material provision thereof; (vii) all opinion letters, if any, delivered in
connection with the Jantzen Acquisition Documents and the transactions
contemplated thereby shall be addressed to the Agent, for the benefit of the
Lenders, or accompanied by a written authorization from the firm delivering such
opinion letter stating that the Agent, for the benefit of the Lenders, may rely
on such opinion letter as though it were addressed to it; and (viii) the Agent
shall have received the consent and acknowledgment of the Jantzen Sellers with
respect to the Jantzen Acquisition Documents Assignment.

     (e)  The Agent shall have received a certificate from PEI's President or
Financial Officer, together with such other evidence satisfactory to the Agent,
that each of the conditions set forth in clauses (a) through (d) above shall
have been satisfied.

     (f)  Jantzen Apparel shall have become a Borrower hereunder on the Jantzen
Acquisition Date in accordance with the terms of Section 10.11.

     (g)  The Jantzen Acquisition and the incurrence of the Senior Note Debt
pursuant to the Senior Note Offering shall be consummated in accordance with the
foregoing on or prior to March 30, 2002.

     6.   On the Jantzen Acquisition Date, (a) PEI shall pay to SunTrust Bank,
in immediately available funds, an amount equal to all Secured Obligations owing
to SunTrust Bank under the Loan Agreement, including all outstanding principal,
all accrued interest and all accrued fees, together with an early termination
fee of $30,000, and (b) all of SunTrust Bank's rights and obligations as a
Lender under the Loan Agreement shall terminate, except that SunTrust Bank's
rights under Sections 5.6 and 16.12 of the Loan Agreement, and obligations under
Section 15.7 of the Loan Agreement, shall continue in full force and effect with
respect to all periods prior to the Jantzen Acquisition Date. PEI's failure to
make such payments in full on the Jantzen Acquisition Date shall constitute an
immediate Event of Default under the Loan Agreement and, notwithstanding
anything to the contrary provided in the Loan Agreement, such

                                      -4-
<PAGE>

Event of Default may not be waived, released, or the subject of any amendment or
modification without the written agreement of SunTrust Bank.

     7.   Subject to the satisfaction of the conditions set forth in Paragraph 5
of this Amendment, the Agent hereby agrees to execute and deliver the Trademark
Intercreditor Agreement to the Senior Note Trustee on the Jantzen Acquisition
Date. The Lenders hereby authorize the Agent to execute and deliver the
Trademark Intercreditor Agreement and such other agreements and documents in
connection therewith as the Agent deems necessary or appropriate.

     8.   The Loan Agreement is hereby amended by deleting the definitions of
"Applicable Margin", "Availability", "Funded Indebtedness", "Guarantor", "Letter
of Credit Facility", "Permitted Indebtedness", "Permitted Liens", "Permitted
Purchase Money Indebtedness" and "Restricted Payment" set forth in Section 1.1
and replacing them with the following:

          "Applicable Margin" means, as to (a) Prime Rate Revolving Credit
           -----------------
     Loans, 0.00%, and (b) LIBOR Revolving Credit Loans, 2.75%.

          "Availability" means, as of any date of determination, (a) an amount
           ------------
     equal to the sum of clauses (b)(i) through (iv) of the definition of
                         --------------         ----
     "Borrowing Base" minus (b) the outstanding principal balance of all
     Revolving Credit Loans hereunder as of such date.

          "Funded Indebtedness" means, without duplication, (a) all Indebtedness
           -------------------
     under this Agreement, (b) the aggregate amount at any time of outstanding
     reimbursement obligations with respect to letters of credit (other than the
     Standby Letter of Credit, but including other Letters of Credit) and
     banker's acceptances issued by a Permitted Letter of Credit Bank or any
     other financial institution which have been drawn upon, (c) all
     Indebtedness under the SunTrust Lease, (d) all Senior Note Debt, (e) all
     High Yield Debt, and (f) all other Indebtedness for Money Borrowed.

          "Guarantor" means each of Supreme International (Mexico) and Supreme
           ---------
     Canada.

          "Letter of Credit Facility" means the aggregate amount of $20,000,000,
           -------------------------
     which may be used for documentary Letters of Credit and standby Letters of
     Credit.

          "Permitted Indebtedness" means:  (a) the Loans, (b) letter of credit
           ----------------------
     and steamship guaranty facilities in the aggregate amount of up to
     $60,000,000 at any time with Permitted Letter of Credit Banks, (c)
     Indebtedness of up to $16,300,000 arising under the SunTrust Lease or any
     refinancing thereof on terms satisfactory to the Agent, (d) Permitted
     Purchase Money Indebtedness, (e) the High Yield

                                      -5-
<PAGE>

     Debt in an aggregate principal amount not to exceed $125,000,000, (f) other
     Subordinated Indebtedness, (g) Indebtedness of up to $75,000,000 under the
     Senior Note Indenture, (h) Permitted Acquisition Debt, and (i) Indebtedness
     under Interest Rate Protection Agreements.

          "Permitted Liens" means :  (a) Liens securing taxes, assessments and
           ---------------
     other governmental charges or levies (excluding any Lien imposed pursuant
     to any of the provisions of ERISA) or the claims of materialmen, mechanics,
     carriers, warehousemen or landlords for labor, materials, supplies or
     rentals incurred in the ordinary course of business, but in all cases, only
     if payment shall not at the time be required to be made in accordance with
     Section 10.4; (b) Liens consisting of deposits or pledges made in the
     ordinary course of business in connection with, or to secure payment of,
     obligations under workers' compensation, unemployment insurance or similar
     legislation or under surety or performance bonds, in each case arising in
     the ordinary course of business; (c) Liens constituting encumbrances in the
     nature of zoning restrictions, easements and rights or restrictions of
     record on the use of the Real Estate, which in the sole judgment of the
     Agent do not materially detract from the value of such Real Estate or
     impair the use thereof in the business of the Obligors; (d) Purchase Money
     Liens securing Permitted Purchase Money Indebtedness; (e) Liens in favor of
     any Permitted Letter of Credit Bank, but only if such Liens shall at all
     times be (i) subordinate to the Agent's Liens in all Collateral other than
     the Inventory (and related documents and insurance proceeds of such
     Inventory) acquired by the Borrowers through letters of credit issued by
     such Permitted Letter of Credit Bank, and (ii) subject to an Inventory
     Intercreditor Agreement; (f) Liens in favor of (i) SunTrust Bank, Miami,
     N.A. as security for PEI's obligations arising under the SunTrust Lease,
     provided such Liens are subject to a lien subordination agreement
     acceptable to the Agent and the Lenders, and (ii) any Person providing a
     loan to refinance the SunTrust Lease, provided that such refinancing is
     permitted under clause (c) of the definition of "Permitted Indebtedness",
     such Liens are limited to the real estate and fixtures at PEI's
     headquarters building located in Miami, Florida, and such Liens do not
     secure any other Indebtedness owing to the Person providing such
     refinancing loan; (g) Liens of the Agent arising under this Agreement and
     the other Loan Documents; (h) Liens arising out of or resulting from any
     judgment or award, the time for the appeal or petition for rehearing of
     which shall not have expired, or in respect of which the Obligors are fully
     protected by insurance or in respect of which the Obligors shall at any
     time in good faith be prosecuting an appeal or proceeding for a review and
     in respect of which a stay of execution pending such appeal or proceeding
     for review shall have been secured, and as to which appropriate reserves
     have been established on the books of the Obligors; (i) the interest of the
     Factors in factored Receivables, provided Assignments of Factoring Credit
     Balances are then effective; and (j) Liens in favor of the Senior Note
     Trustee in the "Collateral" as described in the Senior Note Indenture as in
     effect on the original date thereof (but excluding any

                                      -6-
<PAGE>

     "Additional Collateral" as described therein), provided that such Liens
     shall at all times be subject to the Trademark Intercreditor Agreement.

          "Permitted Purchase Money Indebtedness" means Purchase Money
           -------------------------------------
     Indebtedness secured only by Purchase Money Liens and Capitalized Lease
     Obligations up to an aggregate amount outstanding at any time equal to
     $2,750,000.

          "Restricted Payment" means (a) any redemption, repurchase or
           ------------------
     prepayment (including any payment made to any Person to defease payment
     obligations under the High Yield Debt, the Senior Note Debt or any other
     Indebtedness) or other retirement, prior to the stated maturity thereof or
     prior to the due date of any regularly scheduled installment or
     amortization payment with respect thereto, of any Indebtedness of a Person
     (other than the Secured Obligations and trade debt), (b) any redemption,
     retirement or payment with respect to the High Yield Debt or any other
     Subordinated Indebtedness other than in accordance with the subordination
     agreement or provisions applicable thereto, and (c) the payment by any
     Person of the principal amount of or interest on any Indebtedness (other
     than trade debt) owing to an Affiliate of such Person.

     9.   Upon the consummation of the Jantzen Acquisition on the Jantzen
Acquisition Date, the Loan Agreement is hereby amended by deleting the
definition of "Revolving Credit Facility" set forth in Section 1.1 and replacing
it with the following:

          "Revolving Credit Facility" means the facility for the Revolving
           -------------------------
     Credit Loans in the principal sum of up to $60,000,000 or such lesser or
     greater amount as shall be agreed upon from time to time in writing by the
     Agent, the Lenders and the Borrowers.

     10.  The Loan Agreement is hereby amended by adding the following
definitions to Section 1.1:

          "Bank" means Bank of America, N.A., and its successors and assigns.
           ----

          "Inventory Intercreditor Agreement" means an intercreditor agreement
           ---------------------------------
     between the Agent and a Permitted Letter of Credit Bank in form and
     substance acceptable to the Agent, which shall include subordination and
     other provisions acceptable to the Agent with respect to all Collateral
     other than the Inventory (and related documents and insurance proceeds of
     such Inventory) acquired by the Borrowers through letters of credit issued
     by such Permitted Letter of Credit Bank.

          "Jantzen Acquisition" means the Acquisition by PEI from the Jantzen
           -------------------
     Sellers of the assets contemplated by the Jantzen Acquisition Agreement.

                                      -7-
<PAGE>

          "Jantzen Acquisition Agreement" means the Asset Purchase Agreement to
           -----------------------------
     be entered into among the Jantzen Sellers and PEI, together with all
     exhibits and schedules thereto.

          "Jantzen Acquisition Date" means the date on which the Jantzen
           ------------------------
     Acquisition is consummated.

          "Jantzen Acquisition Documents" means, collectively, the Jantzen
           -----------------------------
     Acquisition Agreement and all other documents, agreements and certificates
     executed in connection with the consummation of the transactions
     contemplated by the Jantzen Acquisition Agreement.

          "Jantzen Acquisition Documents Assignment" means the Collateral
           ----------------------------------------
     Assignment of Asset Purchase Agreement, in form and substance satisfactory
     to the Agent, dated the Jantzen Acquisition Date, made by PEI in favor of
     the Agent, for the benefit of the Lenders.

          "Jantzen Apparel" means Jantzen Apparel Corp., a Delaware corporation.
           ---------------

          "Jantzen Sellers" means, collectively, Jantzen Inc., a Nevada
           ---------------
     corporation, and VF Canada, Inc., a Canadian corporation.

          "PEI" means Perry Ellis International, Inc., a Florida corporation.
           ---

          "Permitted Letter of Credit Banks" means Hong Kong Shanghai Bank,
           --------------------------------
     Ocean Bank, Israeli Discount Bank, Commerce Bank, and any other bank
     acceptable to the Agent.

          "Senior Note Debt" means Indebtedness incurred by PEI on the terms set
           ----------------
     forth in the Senior Note Offering Memorandum and pursuant to the Senior
     Note Indenture.

          "Senior Note Documents" means the Senior Note Indenture, all security
           ---------------------
     agreements relating thereto, and all other documents, agreements and
     certificates executed in connection therewith.

          "Senior Note Indenture" means the Indenture to be entered into among
           ---------------------
     PEI, as issuer, certain Subsidiaries of PEI, as guarantors, and the Senior
     Note Trustee, in connection with the Senior Note Debt.

          "Senior Note Offering" means the offering of the Senior Note Debt in
           --------------------
     accordance with the terms of the Senior Note Offering Memorandum.

                                      -8-
<PAGE>

          "Senior Note Offering Memorandum" means the Preliminary Offering
           -------------------------------
     Memorandum with respect to the offering of up to $57,250,000 principal
     amount of PEI's Senior Secured Notes due 2009.

          "Senior Note Trustee" means State Street Bank and Trust Company, as
           -------------------
     trustee under the Senior Note Indenture.

          "Trademark Intercreditor Agreement" has the meaning set forth in the
           ---------------------------------
     Fifth Amendment to this Agreement dated March 14, 2002.

     11.  The parties hereto acknowledge that, effective upon the consummation
of the Jantzen Acquisition on the Jantzen Acquisition Date, a reserve shall be
established against the Borrowing Base in an amount equal to (a) all outstanding
Indebtedness under the SunTrust Lease minus (b) the face amount of all
outstanding Letters of Credit issued in the favor of SunTrust Bank, Miami, N.A.
as security for PEI's obligations under the SunTrust Lease (the "SunTrust Lease
Reserve").  Such reserve shall continue until the termination of the SunTrust
Lease and the payment in full of all Indebtedness thereunder.

     12.  On the Jantzen Acquisition Date, Bank of America, N.A. will assign
$3,750,000 of its Commitment to First Union National Bank.  In connection
therewith, the parties acknowledge that, after giving effect to such assignment
and the provisions of Paragraph 6 of this Amendment, the Lenders' respective
Commitments will be as set forth on Schedule 1 attached to this Amendment.
                                    ----------

     13.  The Loan Agreement is hereby amended by deleting all references
therein to the defined term "Supreme" and replacing them with "PEI", and by
deleting all references therein to the defined term "NationsBank" and replacing
them with "Bank".

     14.  The Loan Agreement is hereby amended by deleting all references
therein to the defined term "Permitted Receivables Disposition", it being the
intent of the parties hereto that such dispositions no longer be permitted
without the consent of the Agent and the Required Lenders.

     15.  The Loan Agreement is hereby amended by deleting Sections 5.2(b) and
(c) and replacing them with the following:

          (b)  Letter of Credit Fees.
               ---------------------

               (i)  The Borrowers agree to pay to the Agent, for the ratable
          benefit of the Lenders, a fee equal to 2.75% per annum based on the
          average daily outstanding Letter of Credit Amount of each Letter of
          Credit. Such fee shall be payable to the Agent, for the ratable
          benefit of the Lenders, in accordance with their respective Commitment
          Percentages, on the first day

                                      -9-
<PAGE>

          of each month in arrears, and shall be calculated based on a year of
          360 days and the actual number of days elapsed.

               (ii)   The Borrowers agree to pay to the Agent, for the account
          of the Bank, the standard fees and charges of the Bank for issuing,
          administering, amending, renewing, paying and canceling Letters of
          Credit, as and when assessed.

          (c)    Commitment Fee. In connection with and as consideration for the
                 --------------
     Lenders' commitments hereunder, subject to the terms hereof, to lend to the
     Borrowers under the Revolving Credit Facility, the Borrowers shall pay a
     fee to the Agent, for the ratable benefit of the Lenders based on their
     respective Commitment Percentages, from the Effective Date until the
     Termination Date, in an amount equal to 0.50% per annum of the average
     daily unused portion of the Revolving Credit Facility (it being understood
     that no such fee shall be due with respect to the amount of the SunTrust
     Lease Reserve, as such reserve is described in the Fifth Amendment to this
     Agreement), payable monthly in arrears on the first day of each month.

     16.  The Loan Agreement is hereby amended by deleting Sections 5.9(a) and
(b) and replacing it with the following:

          (a)    Prepayments from Asset Dispositions. Immediately upon receipt
                 -----------------------------------
     by a Borrower or any of its Subsidiaries of the Net Proceeds of any Asset
     Disposition, the Borrowers shall apply such Net Proceeds in prepayment of
     the Loans as provided in Section 5.9(d); provided, however, that the
     Borrowers shall not be required to make such prepayment to the extent that
     (i) the Net Proceeds from Asset Dispositions of Intellectual Property
     during any fiscal year of the Borrowers do not exceed, in the aggregate,
     $1,000,000, and (ii) the Net Proceeds from all Asset Dispositions during
     any fiscal year of the Borrowers do not exceed, in the aggregate,
     $3,000,000; provided further, however, that the Borrowers shall not be
     required to make any such prepayment from an Asset Disposition of
     Intellectual Property that is subject to a first priority Lien in favor of
     the Senior Note Trustee in accordance with the terms of the Trademark
     Intercreditor Agreement. Concurrently with the making of any such payment,
     the Borrowers' Agent shall deliver to the Agent a certificate of the
     Financial Officer demonstrating the calculations of the amount required to
     be paid.

          (b)    Prepayments from Equity Offerings. In the event that, at any
                 ---------------------------------
     time after the Effective Date, any Borrower or any of its Subsidiaries
     issues capital stock or other securities or receives an additional capital
     contribution in respect of existing capital stock or other securities
     (excluding any such issuance to, or receipt from, another Borrower or a
     consolidated Subsidiary of a Borrower), no later than the third Business
     Day following the date of receipt of the proceeds

                                      -10-
<PAGE>

     from such issuance, the Borrowers shall notify the Agent in writing whether
     they intend to apply such proceeds, net of underwriting discounts and
     commissions and other reasonable costs associated therewith to the
     redemption or repurchase of outstanding High Yield Debt or Senior Note Debt
     or in prepayment of the Loans as provided in Section 5.9(d); provided any
     such redemption or repurchase of outstanding High Yield Debt or Senior Note
     Debt may be made only to the extent permitted under Section 12.6. In the
     event the Borrowers fail to apply such net proceeds to the redemption or
     repurchase of outstanding High Yield Debt or Senior Note Debt within 60
     days after the Borrowers' receipt thereof, the Borrowers shall immediately
     apply such net proceeds as provided in Section 5.9(d).

     17.  The Loan Agreement is hereby amended by adding the following new
Section 9.17:

          Section 9.17   Jurisdiction of Organization. No Borrower shall change
                         ----------------------------
     its jurisdiction of organization without the prior consent of the Agent.

     18.  The Agent and the Lenders acknowledge and agree that (a) the
provisions of Section 10.12 of the Loan Agreement shall not apply to a wholly
owned Subsidiary of PEI that is not organized under the laws of the United
States of America and to which license agreements or the right to receive
royalties or payments under such license agreements are transferred or assigned
(a "License Subsidiary"), provided, that the aggregate payments due under all
the license agreements or rights so transferred each fiscal year shall not, in
the aggregate, exceed 30% of the total payment due under the all the licenses of
PEI and its Subsidiaries in such fiscal year, and (b) notwithstanding anything
to the contrary set forth in the Loan Agreement, so long as no Event of Default
exists, PEI and its Subsidiaries may transfer or assign license agreements or
the right to receive royalties or payments under such license agreements to a
License Subsidiary, free and clear of the Agent's Lien hereunder, provided, that
the aggregate payments due under all the license agreements or rights so
transferred each fiscal year shall not, in the aggregate, exceed 30% of the
total payment due under the all the licenses of PEI and its Subsidiaries in such
fiscal year. The Agent hereby agrees to execute and deliver (and the Lenders
authorize the Agent to execute and deliver) such Lien releases as may be
reasonably requested by PEI to evidence the foregoing.

     19.  The Loan Agreement is hereby amended by deleting Section 11.5(d) and
replacing it with the following:

          (d)  any (i) Default or Event of Default, or (ii) event that
     constitutes or that, with the passage of time or giving of notice or both,
     would constitute a default or event of default by any Borrower or any of
     its Subsidiaries under any material agreement (other than this Agreement)
     to which such Borrower or Subsidiary is a party or by which such Borrower
     or Subsidiary or any of their respective property may be bound (including
     without limitation any agreement

                                      -11-
<PAGE>

     relating to the High Yield Debt, the Senior Note Debt, the SunTrust Lease,
     the factoring arrangements with the Factors, and any letter of credit
     facility with a Permitted Letter of Credit Bank).

     20.  Upon the consummation of the Jantzen Acquisition on the Jantzen
Acquisition Date (and the incurrence of the Senior Note Debt), the Loan
Agreement is hereby amended by deleting clauses (a) and (c) of Section 12.1 and
replacing such clauses with the following:

          (a)  Maximum Funded Indebtedness to EBITDA Ratio. Permit the ratio of
               -------------------------------------------
     Funded Indebtedness to its EBITDA (measured for the preceding four fiscal
     quarters), in each case measured on a consolidated basis for PEI and its
     consolidated Subsidiaries, to be greater than 6.0 to 1 as of any fiscal
     quarter ending after January 31, 2002.

          (c)  Maximum Senior Funded Indebtedness to EBITDA Ratio. Permit the
               --------------------------------------------------
     ratio of (i) Funded Indebtedness minus Subordinated Indebtedness to (ii)
     EBITDA (measured for the preceding four fiscal quarters), in each case
     measured on a consolidated basis for PEI and its consolidated Subsidiaries,
     to be greater than 3.0 to 1 as of any fiscal quarter ending after January
     31, 2002.

     21.  The Loan Agreement is hereby amended by deleting Section 12.3 and
replacing it with the following:

          Section 12.3  Guaranties.  Become or remain liable with respect to any
                        ----------
     Guaranty of any obligation of any other Person, excluding (a) the Guaranty
     by PEI of Sunny Industries' obligations for $600,000, (b) the Guaranty
     Agreements, (c) the Guarantees by the Guarantors of the High Yield Debt,
     and (d) the Guarantees by the Guarantors of the Senior Note Debt.

     22.  The Loan Agreement is hereby amended by deleting Section 12.5 and
replacing it with the following:

          Section 12.5  Capital Expenditures.  Make or incur any Capital
                        --------------------
     Expenditures, except that the Borrowers and their Subsidiaries may make or
     incur Capital Expenditures during any fiscal year in an amount not to
     exceed, in the aggregate, $6,000,000.

     23.  The Loan Agreement is hereby amended by deleting Section 12.6 and
replacing it with the following:

          Section 12.6 Restricted Distributions and Payments, Etc. Declare or
                       ------------------------------------------
     make any Restricted Distribution or Restricted Payment; provided that, the
     Borrowers shall be permitted to redeem or repurchase outstanding High Yield
     Debt and the Senior Note Debt from the proceeds of an equity offering made
     in

                                      -12-
<PAGE>

     accordance with Section 5.9(b) provided (a) no Default or Event of Default
     exists before or after giving effect to such redemption or repurchase, (b)
     the Borrowers are in compliance with the covenants set forth in Section
     12.1 on a pro forma basis after giving effect to such redemption or
     repurchase, and (c) prior to such redemption or repurchase the Borrowers'
     Agent shall deliver to the Agent a certificate of the Financial Officer (i)
     setting forth the calculations required to establish the Borrowers' pro
     forma compliance with the requirements of Section 12.1 after giving effect
     to such redemption or repurchase, and (ii) stating that no Default or Event
     of Default exists before or after giving effect to such redemption or
     repurchase; provided further, that to the extent that any Asset Disposition
     of Intellectual Property that is subject to a first priority Lien in favor
     of the Senior Note Trustee in accordance with the terms of the Trademark
     Intercreditor Agreement is permitted under this Agreement, the Borrowers
     shall be permitted to redeem or repurchase outstanding Senior Note Debt
     from the Net Proceeds of such Asset Disposition. Notwithstanding the above,
     the Borrowers shall also be permitted at any time to repurchase or redeem
     capital stock or other equity securities issued by such Borrowers, provided
     (a) the consideration paid by such Borrowers shall not exceed $7,000,000 in
     the aggregate during the term of this Loan Agreement, (b) no Default or
     Event of Default exists before or after giving effect to such repurchase,
     and (c) the Borrowers are in compliance with the covenants set forth in
     Section 12.1 on a pro forma basis after giving effect to such redemption or
     repurchase.

     24.  The Loan Agreement is hereby amended by deleting Section 12.13 and
replacing it with the following:

          Section 12.13  Amendments of Other Agreements. Amend in any way (a)
                         ------------------------------
     the interest rate or principal amount or schedule of payments of principal
     and interest with respect to any Indebtedness (other than the Secured
     Obligations) other than to reduce the interest rate or extend the schedule
     of payments with respect thereto, (b) the subordination provisions
     applicable to the High Yield Debt or any other Subordinated Indebtedness,
     or (c) any of the other terms and conditions applicable to the High Yield
     Debt or the Senior Note Debt.

     25.  The Loan Agreement is hereby amended by deleting Section 12.14 and
replacing it with the following:

          Section 12.14  Minimum Availability. Permit Availability to be less
                         --------------------
     than $15,000,000 at any time.

     26.  The Loan Agreement is hereby amended by deleting Section 12.15 and
replacing it with the following:

                                      -13-
<PAGE>

          Section 12.15  Limitation on Certain Restrictions on Subsidiaries.
                         --------------------------------------------------
     Create or otherwise cause or suffer to exist or become effective any
     encumbrance or restriction on the ability of any Subsidiary to (a) pay
     dividends or make any other distributions on its capital stock or any other
     interest or participation in its profits owned by a Borrower or any
     Subsidiary of a Borrower, or pay any Indebtedness owed to a Borrower or a
     Subsidiary of a Borrower, (b) make loans or advances to a Borrower or any
     of a Borrower's Subsidiaries, or (c) transfer any of its properties or
     assets to a Borrower, except for such encumbrances or restrictions existing
     under or by reason of (i) applicable laws, (ii) the indenture and other
     agreements applicable to the High Yield Debt, (iii) the Senior Note
     Documents, and (iv) this Agreement and the other Loan Documents.

     27.  The Loan Agreement is hereby amended by deleting Section 13.1(o) and
replacing it with the following:

          (o) Change in Control. At any time after the Agreement Date, (i) any
              -----------------
     "person" or "group" (as such terms are used in Sections 13(d) and 14(d) of
     the Securities Exchange Act of 1934, as amended, and the rules promulgated
     thereunder (collectively, the "Exchange Act")), other than Permitted
     Holders, is or becomes the "beneficial owner" (as defined in Rules 13d-3
     and 13d-5 under the Exchange Act, except that a Person shall be deemed to
     have "beneficial ownership" of all securities that such Person has the
     right to acquire, whether such right is exercisable immediately or only
     after the passage of time), directly or indirectly, of more than 35% of the
     total voting power of all outstanding capital voting stock of PEI, and
     either (A) the Permitted Holders beneficially own, directly or indirectly,
     in the aggregate capital voting stock of PEI that represents a lesser
     percentage of the aggregate ordinary voting power of all classes of the
     capital voting stock of PEI, voting together as a single class, than such
     other person or group and are not entitled (by voting power, contract or
     otherwise) to elect directors of PEI having a majority of the total voting
     power of the Board of Directors of PEI, or (B) such other person or group
     is entitled to elect directors of PEI having a majority of the total voting
     power of the Board of Directors of PEI; or (ii) during any consecutive two-
     year period, individuals who at the beginning of such period constituted
     the Board of Directors of PEI (together with any new directors whose
     election by the Board of Directors of PEI, or whose nomination for election
     by the shareholders of PEI, as the case may be, was approved by a vote of
     66 2/3% of the directors then still in office who either were directors at
     the beginning of such period or whose election or nomination for election
     was previously so approved) cease for any reason to constitute a majority
     of the directors of PEI then in office; or (iii) PEI ceases to own,
     directly or indirectly, 100% of the capital stock of each other Obligor, or
     such ownership shall cease to vest in PEI voting control over any such
     Obligor; or (iv) a "Change of Control" as defined in the Notes Description
     of the High Yield Debt Offering Memorandum, or

                                      -14-
<PAGE>

     any indenture issued in connection therewith, occurs; or (v) a "Change of
     Control" as defined in the Senior Note Indenture occurs.

     28.  The Loan Agreement is hereby amended by adding the following new
Section 15.11:

          Section 15.11 Authorization re Intercreditor Agreements. Without
                        -----------------------------------------
     limiting any of the other provisions of this Article 15, each of the
     Lenders hereby authorizes the Agent to execute and deliver, on behalf of
     itself and all the Lenders, any Inventory Intercreditor Agreement and
     Trademark Intercreditor Agreement, and to take all actions thereunder,
     including any actions relating to the Lien subordinations and releases
     contemplated thereby.

     29.  The Agent and the Lenders waive the requirements set forth in Section
10.12 of the Loan Agreement with respect to PEI's French subsidiary, Perry Ellis
International Europe.

     30.  As consideration for the accommodations set forth herein, PEI shall
pay to the Agent, for the Ratable benefit of the Lenders (other than SunTrust
Bank), a fee of $225,000 on the date hereof.  Such fee shall constitute a
payment for the structuring and approving of the accommodations set forth
herein, and not interest or a charge for the use of money.

     31.  The effectiveness of this Amendment shall be conditioned upon the
satisfaction of the following conditions precedent in a manner acceptable to the
Agent and the Lenders:

          (a) The receipt by the Agent of an Acknowledgment of the terms of this
Amendment from each Guarantor, which Acknowledgments shall be in form and
substance satisfactory to the Agent and the Lenders; and

          (b) The payment of all fees due by PEI in connection with this
Amendment.

     32.  PEI hereby restates, ratifies, and reaffirms each and every term,
condition, representation and warranty heretofore made by it under or in
connection with the execution and delivery of the Loan Agreement, as amended
hereby, and the other Loan Documents, as fully as though such representations
and warranties had been made on the date hereof and with specific reference to
this Amendment.

     33.  As amended hereby, the Loan Agreement shall be and remain in full
force and effect, and shall constitute the legal, valid, binding and enforceable
obligations of PEI to the Agent and the Lenders.

     34.  In addition to the fee set forth in Paragraph 29 of this Amendment,
PEI agrees to pay on demand all costs and expenses of the Agent in connection
with the preparation, execution, delivery and enforcement of this Amendment and
all other Loan Documents and any other

                                      -15-
<PAGE>

transactions contemplated hereby, including, without limitation, the fees and
out-of-pocket expenses of legal counsel to the Agent.

     35.  PEI agrees to take such further action as the Agent or any Lender
shall reasonably request in connection herewith to evidence the amendments
herein contained to the Loan Agreement.

     36.  This Amendment may be executed in any number of counterparts and by
different parties hereto in separate counterparts, each of which, when so
executed and delivered, shall be deemed to be an original and all of which
counterparts, taken together, shall constitute but one and the same instrument.

     37.  This Amendment shall be binding upon and inure to the benefit of the
successors and permitted assigns of the parties hereto.

     38.  This Amendment shall be governed by, and construed in accordance with,
the laws of the State of Georgia, other than its laws respecting choice of law.

                                      -16-
<PAGE>

     IN WITNESS WHEREOF, PEI, the Agent and each Lender have caused this
Amendment to be duly executed by their authorized officers in several
counterparts, all as of the date first above written.


                                     BORROWERS:

                                     PERRY ELLIS INTERNATIONAL, INC., formerly
                                     Supreme International Corporation

                                     By: /s/ Rosemary Trudeau
                                        ----------------------------------------
                                        Rosemary Trudeau
                                        Vice President, Finance
<PAGE>

                                    LENDERS:

                                    BANK OF AMERICA, N.A., formerly
                                    NationsBank, N.A.


                                    By:____________________________________
                                       Name:_______________________________
                                       Title:______________________________


                                    FIRST UNION NATIONAL BANK


                                    By:____________________________________
                                       Name:_______________________________
                                       Title:______________________________


                                    FLEET CAPITAL CORPORATION


                                    By:____________________________________
                                       Name:_______________________________
                                       Title:______________________________


                                    SUNTRUST BANK, formerly SunTrust Bank, N.A.


                                    By:____________________________________
                                       Name:_______________________________
                                       Title:______________________________
<PAGE>

                                    AGENT:

                                    BANK OF AMERICA, N.A., formerly
                                    NationsBank, N.A.


                                    By:____________________________________
                                       Name:_______________________________
                                       Title:______________________________
<PAGE>

                         NOTARY JURAT FOR EXECUTION OF
                                NOTES AND OTHER
                       WRITTEN OBLIGATIONS TO PAY MONEY
                             BY FLORIDA BORROWERS


     On this the ____ day of March, 2002, before me, the undersigned, a Notary
Public in and for the State of __________, County of __________, Rosemary
Trudeau personally appeared, personally known to me or proved to me on the basis
of satisfactory evidence to be the Vice President, Finance of Perry Ellis
International, Inc., who executed the foregoing Fifth Amendment to Amended and
Restated Loan and Security Agreement on behalf of such corporation and
acknowledged to me that such corporation executed the foregoing pursuant to its
by-laws or a resolution of its board of directors, said execution taking place
in the State of __________, County of __________.


__________________________________
        Notary Signature

My Commission Expires:

__________________________________

      [Affix Notarial Seal]
<PAGE>

                         AFFIDAVIT REGARDING DELIVERY


I, ________________________________ hereby certify that I am a _________________
___________ of Bank of America, N.A., as Agent for the Lenders, and that the
foregoing was delivered to me as a representative of Bank of America, N.A., as
Agent for the Lenders, in the State of Georgia, County of Fulton.


                                   _____________________________________________
                                   Signature of Officer or Agent of Lender


     On this the __ day of March, 2002, before me, the undersigned, a Notary
Public in and for the State of Georgia, County of Fulton, ____________________,
personally appeared, personally known to me or proved to me on the basis of
satisfactory evidence to be a _______________ of Bank of America, N.A., a
national banking association, as Agent for the Lenders, who executed the
foregoing affidavit on behalf of such national banking association and
acknowledged to me that such national banking association executed the foregoing
pursuant to its by-laws or a resolution of its board of directors, said
execution taking place in the State of Georgia, County of Fulton.



_______________________________
       Notary Signature
<PAGE>

                                  SCHEDULE 1
                                  ----------

                                  COMMITMENTS
                                  -----------


         ------------------------------------------------------------
             Name of Lender              Commitment      Commitment
                                           Amount        Percentage

         ------------------------------------------------------------
          Bank of America, N.A.          $22,500,000       37.5%
         ------------------------------------------------------------
          Fleet Capital Corporation      $22,500,000       37.5%
         ------------------------------------------------------------
          First Union National Bank      $15,000,000       25.0%
         ------------------------------------------------------------
<PAGE>

                                   EXHIBIT A
                                   ---------

              [attach form of Trademark Intercreditor Agreement]