<PAGE>

                                                                     Exhibit 4.1

                                                                  EXECUTION COPY
                                                                    CONFIDENTIAL

                                 AMENDMENT NO. 1

         AMENDENT NO. 1, dated as of February 3, 2003 ("Amendment"), to the
Rights Agreement, dated as of May 17, 2002, between Salant Corporation, a
Delaware corporation (the "Corporation"), and Mellon Investor Services LLC, a
New Jersey limited liability company (the "Rights Agent") (as amended, the
"Rights Agreement").

         WHEREAS, pursuant to Section 27 of the Rights Agreement, the
Corporation and the Rights Agent may from time to time supplement or amend the
Rights Agreement in accordance with Section 27 thereof; and

         WHEREAS, the Corporation, Perry Ellis International, Inc., a Florida
corporation ("Parent"), and Connor Acquisition Corp., a Delaware corporation and
wholly owned subsidiary of Parent ("Merger Sub"), contemplate entering into an
Agreement and Plan of Merger (the "Merger Agreement"); and

         WHEREAS, the Board of Directors of the Corporation has determined that
an amendment to the Rights Agreement as set forth herein is necessary and
desirable in connection with the execution and delivery of the Merger Agreement
and is consistent with the objectives of the Board of Directors of the
Corporation in adopting the Rights Agreement, and the Corporation and Rights
Agent desire to evidence such amendment in writing; and

         WHEREAS, all actions necessary to make this Amendment valid and
enforceable have been performed and done, and the execution and delivery of this
Amendment by the Corporation and the Rights Agent have been in all respects duly
authorized by the Corporation and the Rights Agent.

         NOW, THEREFORE, in consideration of the premises and the mutual
agreements herein set forth, the parties thereto agree as follows:

         1.   The Rights Agreement is hereby amended as follows:

              (i)  Amendment of Section 1.  Section 1 of the Rights Agreement
is supplemented to add the following definitions in the appropriate locations:

              "Merger Sub" shall have the meaning set forth in the Merger
              Agreement.

              "Merger Agreement" shall mean the Agreement and Plan of Merger,
              dated as of February 3, 2003, by and among Parent, Merger Sub and
              the Corporation, as it may be amended from time to time.

              "Merger" shall have the meaning set forth in the Merger Agreement.

<PAGE>

                                                                  EXECUTION COPY
                                                                    CONFIDENTIAL

              "Parent" shall have the meaning set forth in the Merger Agreement.

              (ii)  Amendment of the definition of "Acquiring Person". The
second sentence of the definition of "Acquiring Person" in subsection 1(a) of
the Rights Agreement is amended and restated in its entirety as follows:

              Notwithstanding the foregoing, (A) the term "Acquiring Person"
              shall not include (i) the Corporation, (ii) any Subsidiary of the
              Corporation, (iii) any employee benefit plan of the Corporation or
              of any Subsidiary of the Corporation, (iv) any Person or entity
              organized, appointed or established by the Corporation for or
              pursuant to the terms of any such plan, (v) the Parent, Merger Sub
              or any of their respective Affiliates or subsidiaries in each case
              solely as a result of the Merger Agreement, the Merger and the
              transactions contemplated thereby (including the public
              announcement or disclosure thereof), (vi) any Person, who or which
              together with all Affiliates and Associates of such Person becomes
              the Beneficial Owner of 15% or more of the then outstanding Common
              Shares as a result of the acquisition of Common Shares directly
              from the Corporation, or (vii) any Grandfathered Stockholder and
              (B) no Person shall be deemed to be an "Acquiring Person" either
              (X) as a result of the acquisition of Common Shares by the
              Corporation which, by reducing the number of Common Shares
              outstanding, increases the proportional number of shares
              beneficially owned by such Person together with all Affiliates and
              Associates of such Person; except that if (i) a Person would
              become an Acquiring Person (but for the operation of this
              subclause X) as a result of the acquisition of Common Shares by
              the Corporation, and (ii) after such share acquisition by the
              Corporation, such Person, or an Affiliate or Associate of such
              Person, becomes the Beneficial Owner of any additional Common
              Shares, then such Person shall be deemed an Acquiring Person, or
              (Y) if a Person did so inadvertently, (i) promptly after such
              Person discovers that such Person would otherwise have become an
              Acquiring Person (but for the operation of this subclause Y), such
              Person notifies the Board of Directors that such Person did so
              inadvertently and (ii) within 2 days after such notification, such
              Person is the Beneficial Owner of less than 15% of the outstanding
              Common Shares.

              (iii) Amendment of the definition of "Interested Stockholder".

              Subsection 1(k) is amended by adding the following sentence at the
end thereof:

                                       2

<PAGE>

                                                                  EXECUTION COPY
                                                                    CONFIDENTIAL

              Notwithstanding the foregoing, the term "Interested Stockholder"
              shall not include the Parent, Merger Sub or any of their
              respective Affiliates or subsidiaries in each case solely as a
              result of the Merger Agreement, the Merger and the transactions
              contemplated thereby (including the public announcement or
              disclosure thereof).

              (iv)  Amendment of the definition of "Distribution Date".
Subsection 3(a) is amended by adding the following sentence at the end thereof:

              Notwithstanding anything in this Agreement to the contrary, a
              Distribution Date shall not be deemed to have occurred as the
              result of (i) the execution, delivery or performance of the Merger
              Agreement or the consummation of the Merger or any of the other
              transactions contemplated by the Merger Agreement, or (ii) the
              public announcement or disclosure of the Merger Agreement or any
              of the other transactions contemplated by the Merger Agreement.

              (v)   Amendment of the definition of "Shares Acquisition Date".
The definition of "Shares Acquisition Date" in Section 1(r) of the Rights
Agreement is amended by adding the following sentence at the end thereof:

                  Notwithstanding anything in this Agreement to the contrary, a
                  Shares Acquisition Date shall not be deemed to have occurred
                  as the result of (i) the execution, delivery or performance of
                  the Merger Agreement or the consummation of the Merger or any
                  of the other transactions contemplated by the Merger
                  Agreement, or (ii) the public announcement or disclosure of
                  the Merger Agreement or any of the other transactions
                  contemplated by the Merger Agreement.

              (vi)  Amendment of the definition of "Triggering Event". The
definition of "Triggering Event" in Section 1(t) of the Rights Agreement is
amended by adding the following sentence at the end thereof:

              Notwithstanding anything in this Agreement to the contrary, a
              Triggering Event shall not be deemed to have occurred as the
              result of and the provisions of Sections 11(a)(ii) and 13 of this
              Agreement shall not apply to (i) the execution, delivery or
              performance of the Merger Agreement or the consummation of the
              Merger or any of the other transactions contemplated by the Merger
              Agreement, or (ii) the public announcement or disclosure of the
              Merger Agreement or any of the other transactions contemplated by
              the Merger Agreement.

                                       3

<PAGE>

                                                                  EXECUTION COPY
                                                                    CONFIDENTIAL

              (vii)  All references in the Rights Agreement (including the
Exhibits thereto) shall, from and after the date hereof, refer to the Rights
Agreement as amended by this Amendment.

         2.   Effectiveness. Each party hereto represents and warrants that (i)
it is duly authorized to execute and deliver this Amendment and to perform
hereunder and (ii) this Amendment constitutes a valid and binding agreement of
such party. This Amendment shall become effective as of the date first above
written. The Rights Agreement shall not otherwise be supplemented or amended by
virtue of this Amendment, but shall remain in full force and effect.

         3.   Counterparts. This Amendment may be executed in any number of
counterparts and each of such counterparts shall for all purposes be deemed to
be an original, and all such counterparts shall together constitute but one and
the same instrument.

         4.   Full Force and Effect. Except as expressly amended hereby, the
Rights Agreement shall continue in full force and effect in accordance with the
provisions thereof on the date hereof.

         5.   Governing Law. This Amendment shall be deemed to be a contract
made under the laws of the State of Delaware and for all purposes shall be
governed by and construed in accordance with the laws of such State applicable
to contracts made and to be performed entirely within such State; except that
all provisions regarding the rights, duties, obligations and immunities of the
Rights Agent shall be governed by and construed in accordance with the laws of
the State of New York applicable to contracts made and to be performed entirely
within such State.

                                       4

<PAGE>

         IN WITNESS WHEREOF, the parties hereto have caused this Amendment to be
duly executed and attested, all as of the date and year first above written.

Attest:                                     SALANT CORPORATION



By:___________________________              By:___________________________
Name:                                          Name:
Title:                                         Title:


Attest:                                     MELLON INVESTOR SERVICES LLC



By:____________________________             By:___________________________
Name:                                          Name:
Title:                                         Title:



[Signature Page to Amendment No. 1 to the Rights Agreement]