<PAGE>

                                                                    Exhibit 10.4

                         PERRY ELLIS INTERNATIONAL, INC.
                             3000 NW 107/th/ Avenue
                              Miami, Florida 33172

                                                                February 3, 2003

Mr. Michael J. Setola
44 Sneider Road
Warren, New Jersey 07059

Mr. Setola:

         Reference is made to that certain Agreement and Plan of Merger of even
date herewith being entered into concurrently with the execution and delivery
hereof (the "Merger Agreement"), by and among Perry Ellis International, Inc., a
Florida corporation ("Parent"), Connor Acquisition Corp., a Delaware corporation
("Merger Sub"), and Salant Corporation, a Delaware corporation (the "Company"),
pursuant to which, upon the terms and subject to the conditions set forth
therein, Merger Sub will merge with and into the Company (the "Merger") and the
Company will be the surviving corporation in the Merger (the "Surviving
Corporation"), and become a direct, wholly owned subsidiary of Parent.
Capitalized terms used but not defined herein have the meanings ascribed thereto
in the Merger Agreement.

         This letter agreement (this "Agreement") confirms our mutual agreement
and understanding with respect to the following:

         1. (a) As a material inducement for Parent and Merger Sub to enter into
the Merger Agreement and to consummate the Merger and the transactions
contemplated by the Merger Agreement, and in consideration for the sum of $10.00
and other good and valuable consideration, the receipt and sufficiency of which
is hereby acknowledged, Michael J. Setola ("Setola") hereby agrees that he shall
not, for a period of 21 months from and after the Effective Time (which period
automatically shall be extended for such time as Setola is in breach of any of
the provisions hereof) (the "Restriction Period"), directly or indirectly (i)
solicit for hire, hire, offer employment to, engage, retain or accept the
services of, or otherwise employ, or assist any person, group of persons or any
entity with soliciting for hire, hiring, offering employment to, engaging,
retaining or accepting the services of, or otherwise employing, any person
currently employed, working for or otherwise associated or affiliated (as a
full- or part-time employee) with, the Company, the Surviving Corporation or any
of their respective subsidiaries in any sales, design, marketing, merchandising
or similar capacity (or any combination thereof) and whose annualized
compensation or remuneration (including, without limitation, salary, fees,
bonuses, perquisites, equity and other forms of incentive awards, options or
compensation) is greater than $100,000 (each such person, an "Employee") and
then only to the extent any of the prohibitions described above relate to or are
in connection with a business, venture or activity involving the wholesale or
retail manufacturing, distribution, sale or licensing of apparel goods or
services; (ii) facilitate, entice, encourage or induce in any manner any
Employee to terminate, abandon, resign, cease or otherwise alter his or her
employment, association, consultancy,

<PAGE>

business or other relationship with the Company, the Surviving Corporation or
any of their respective subsidiaries; or (iii) interfere in any manner with the
relationship of the Company, the Surviving Corporation or any of their
respective subsidiaries with any Employee. If, after the Effective Time and
prior to the end of the Restriction Period, (A) any Employee is terminated by
the Surviving Corporation or any of its subsidiaries, or (B) any Employee who is
party to an employment agreement with any of the Company, the Surviving
Corporation or any of their respective subsidiaries terminates his or her
employment for "good reason" (as defined in such employment agreement), Setola
shall be permitted to employ or hire such Employee following the expiration of
six months from such termination (which six-month period may be waived or
reduced by the prior written consent of Parent), provided that Setola is not
otherwise in violation of this Agreement.

              (b) Setola further agrees that for a period ending on December 31,
2005, he shall not, directly or indirectly, whether on his own behalf or on
behalf of or in conjunction with another person, group of persons or entity: (i)
facilitate, entice, encourage or induce in any manner Ocean Pacific Apparel
Corp. or any of its subsidiaries or affiliates ("Ocean Pacific"), as Licensor
under that certain Trademark License Agreement, dated July 20, 2001 as amended
by letter agreements dated May 13, 2002 and November 15, 2002, with the Company,
as Licensee (together with any amendment or modification thereof, and as the
same hereafter may be amended from time to time, the "Ocean Pacific License
Agreement"), to suspend, terminate, limit, modify, not renew, change or
otherwise alter the license granted (including any renewal thereof) and the
respective rights, duties and obligations of the parties under and pursuant to
the Ocean Pacific License Agreement, or otherwise interfere in any manner with
the relationship of the Company, the Surviving Corporation or any of their
respective subsidiaries or affiliates with Ocean Pacific or with the license
granted under and pursuant to the Ocean Pacific License Agreement; or (ii) if
Ocean Pacific shall not have consented to the Merger or shall have terminated
the Ocean Pacific License Agreement pursuant to the terms thereof, "control"
(within the meaning of Section 20 of the Securities Exchange Act of 1934, as
amended (the "Exchange Act")), become an "affiliate" (as such term is defined in
Rule 12b-2 of the Exchange Act) of, or have any financial interest or
participate in the earnings or profits of or become (or have the functional
duties of) an officer, director, partner or managing member of any corporation,
partnership, limited partnership, limited liability company, joint venture,
association or other entity or person (other than Ocean Pacific or any
subsidiary thereof) to which Ocean Pacific shall have granted or assigned a
license to use or any interest in the "Trademark" with respect to one or more of
the "Licensed Products" for use in the "Territory" (all such quoted terms are
defined in the Ocean Pacific License Agreement); provided, however, that nothing
in clause (ii) above shall prohibit Setola from being a passive owner of not
more than 5% of the outstanding shares of any class of securities of any such
corporation, partnership, limited partnership, limited liability company, joint
venture, association or other entity, to the extent such class of securities is
publicly traded, and provided that Setola has no active (or, in relation to such
maximum 5% ownership, disproportionate) participation in or influence over the
management, business policies or affairs of such corporation, partnership,
limited partnership, limited liability company, joint venture, association or
other entity.

         2.   Parent acknowledges and agrees that, at the Effective Time, it
shall pay the lump sum cash payment payable to Setola upon a "Change of Control"
and shall pay any other amounts and shall continue any benefits payable pursuant
to the Employment Agreement, dated

                                       2

<PAGE>

as of May 17, 1999 and as amended as of November 25, 2002 (the "Employment
Agreement"), between Setola and the Company, provided that Setola has given the
notice of termination pursuant to Section 7(e) of the Employment Agreement.
Parent further acknowledges and agrees that, at the Effective Time, it shall:
(i) pay all cash amounts payable and issue all shares of Parent common stock,
$0.01 par value ("Parent Common Stock"), issuable to Setola in exchange for his
outstanding shares of Company common stock, $1.00 par value ("Company Common
Stock") in the Merger upon Setola's delivery of all certificates evidencing
Company Common Stock held by Setola at the Effective Time (together with a
completed letter of transmittal) in accordance with the terms of the Merger
Agreement; and (ii) pay all cash amounts payable and issue all shares of Parent
Common Stock issuable to Setola in respect of the settlement of all outstanding
options to purchase Company Common Stock held by Setola at the Effective Time
upon the surrender of such options to the Company for cancellation in accordance
with the terms of the Optionee Agreement and Release between the Company and
Setola, dated February 3, 2003.

         3. Setola acknowledges and agrees that it is fair and reasonable that
he make the covenants and perform the undertakings set forth in this Agreement
and has done so with the benefit of the advice of legal counsel and other
business advisors. Furthermore, Setola agrees that any breach or attempted
breach or non-observance by him of any of the provisions of this Agreement will
cause material irreparable harm to Parent, the Company, the Surviving
Corporation and their respective subsidiaries and affiliates for which monetary
damages will not be an adequate remedy. Accordingly, Parent, the Company and/or
the Surviving Corporation shall be entitled to apply for and obtain injunctive
and equitable relief (whether temporary, preliminary and/or permanent) to
restrain, prevent and enjoin the breach or threatened breach of, and otherwise
to specifically enforce, the provisions of this Agreement, without the
requirement of the posting of a bond or providing any other form of security.
Nothing contained herein shall be construed as a limitation or waiver of any
other right or remedy that may be available to Parent, the Company and/or the
Surviving Corporation upon such breach, non-observance or threatened breach or
non-observance. For emergency relief (including temporary and preliminary
injunctive relief), an application may be made in any court of competent
jurisdiction, in addition to Parent's, the Company's and/or the Surviving
Corporation's right to seek injunctive, monetary and other forms of relief at
law or in equity as provided for in this Agreement. Setola further agrees that
the subject matter and duration of the restrictions contained herein are
reasonable in light of the facts as they exist today. If any restriction
contained in this Agreement is deemed to be unreasonable in any respect by a
court of competent jurisdiction, such restriction shall be reduced (but not
eliminated), in such manner as such court determines is reasonable (but in all
cases such reduction shall be only to such extent as is legally permissible to
carry out the intention of the parties hereto and to maximize the effect of the
provisions herein agreed to by the parties). Setola agrees that the provisions
of this Agreement are several, such that if any term, provision, covenant or
restriction of this Agreement is held by a court of competent jurisdiction or
other authority to be invalid, void, unenforceable or against its regulatory
policy, the remainder of this Agreement shall remain in full force and effect
and shall in no way be affected, impaired or invalidated.

         4. Each of the parties hereto hereby represents and warrants, as to
himself or itself, that (a) he or it has all requisite capacity, power and
authority to execute and deliver this Agreement and to perform its obligations
hereunder, (b) this Agreement is its legal, valid and binding obligation,
enforceable against him or it in accordance with its terms, (c) if applicable,

                                       3

<PAGE>

his or its entry into this Agreement and performance of the terms hereof has
been duly and validly authorized, (d) his or its entry into this Agreement and
performance of the terms hereof will not breach or conflict with any outstanding
obligation, contractual or otherwise, to which he or it is subject, nor will the
same violate any laws or regulations of any governmental or judicial authority
to which he or it is subject.

         5. This Agreement contains the entire agreement, and supersedes all
prior agreements and understandings, oral or written, between the parties hereto
with respect to the subject matter hereof. This Agreement may not be changed,
modified, extended or terminated except upon written amendment duly approved in
writing by each of the parties hereto.

         6. This Agreement shall inure to the benefit of, and shall be binding
upon, the parties hereto and their respective successors, permitted assigns,
heirs and legal representatives. The rights and obligations of a party hereunder
may not be transferred or assigned, except that Parent may assign its rights and
obligations hereunder to any of its subsidiaries or affiliates.

         7. All questions pertaining to the validity, construction, execution
and performance of this Agreement shall be governed by and construed in
accordance with the laws of the State of Delaware, without regard to the
conflicts of laws provisions thereof. Each of the parties hereto (a) consents to
submit himself or itself to the personal jurisdiction of any Federal court
located in the State of Delaware or any Delaware state court in the event any
dispute arises out of this Agreement or any of the matters that are the subject
of this Agreement, (b) agrees that he or it will not attempt to deny or defeat
such personal jurisdiction by motion or other request for leave from any such
court, and (c) agrees that he or it will not bring any action relating to this
Agreement or any of the matters that are the subject of this Agreement in any
court other than a Federal court sitting in the State of Delaware or a Delaware
state court. The parties irrevocably and unconditionally waive any objection to
the laying of venue of any action, suit or proceeding arising out of this
Agreement or any of the matters that are the subject of this Agreement in any
Federal court located in the State of Delaware or any Delaware state court, and
hereby further irrevocably and unconditionally waive and agree not to plead or
claim in any such court that any such action, suit or proceeding brought in any
such court has been brought in an inconvenient forum.

         8. Each party shall duly execute and deliver, or cause to be duly
executed and delivered, such further agreements, documents and instruments, and
do or cause to be done such further acts as may be reasonably necessary or
proper to effectuate the provisions or purposes of this Agreement and the
transactions contemplated hereby.

         9. This Agreement may be executed in counterparts, each of which shall
be deemed to be an original and all of which together shall constitute one and
the same instrument.

                                       4

<PAGE>

         If this Agreement correctly sets forth our understanding with respect
to the subject matters addressed herein, please execute this Agreement in the
space provided below, whereupon it shall become a binding agreement between us.

                                            Very truly yours,

                                            PERRY ELLIS INTERNATIONAL, INC.

                                            By: /s/ Timothy B. Page
                                               ---------------------------------
                                            Name:  Timothy B. Page
                                            Title: Chief Financial Officer

Agreed to and accepted as of the date first written above:

SALANT CORPORATION


By: /s/ Michael J. Setola
   ------------------------------------
Name:  Michael J. Setola
Title: Chairman & CEO

      /s/ Michael J. Setola
---------------------------------------
           MICHAEL J. SETOLA

                                       5