<PAGE>

                                                                     Exhibit 4.1



                          SUPPLEMENTAL RIGHTS AGREEMENT
                                       AND
                     FIFTH AMENDMENT TO THE RIGHTS AGREEMENT

     THIS SUPPLEMENTAL RIGHTS AGREEMENT AND FIFTH AMENDMENT TO THE RIGHTS
AGREEMENT, dated as of February 12, 2003 (the "Agreement"), by and between
PHILLIPS-VAN HEUSEN CORPORATION, a Delaware corporation (the "Company"), and THE
BANK OF NEW YORK, a New York banking corporation (the "Rights Agent"), amends
the Rights Agreement, dated as of June 10, 1986 (the "Rights Agreement"), as
amended March 31, 1987, July 30, 1987, June 30, 1992, and April 25, 2000, by and
between the Company and the Rights Agent (as successor thereunder to The Chase
Manhattan Bank, N.A.).

                              W I T N E S S E T H :
                               - - - - - - - - - -

     WHEREAS, the Company proposes to issue up to 10,000 shares of a new series
of Preferred Stock, par value $100 per share, of the Company designated as the
"Series B Convertible Preferred Stock" (the "Series B Preferred Stock") with
such terms as are provided in the Certificate of Designations, Preferences, and
Rights of the Series B Convertible Preferred Stock (the "Certificate") pursuant
to a Securities Purchase Agreement, dated as of December 16, 2002 (the "Purchase
Agreement") by and between the Company and certain investors listed thereto; and

     WHEREAS, on December 15, 2002, the Board of Directors of the Company by
resolutions duly authorized the issuance of the Series B Preferred Stock upon
such time as required pursuant to the Purchase Agreement, the execution and
filing of the Certificate upon such time as required pursuant to the Purchase
Agreement, the execution of the Purchase Agreement, the issuance of Rights in
respect of the Series B Preferred Stock, so long as the Rights have not
previously expired or been redeemed in accordance with the terms of the Rights
Agreement upon such time as required pursuant to the Purchase Agreement and the
amendments to the Rights Agreement upon such time as required pursuant to the
Purchase Agreement;

     NOW THEREFORE, in consideration of the premises and the mutual agreements
herein set forth, the parties hereto hereby agree as follows:

Section 1. Definitions. Capitalized terms used herein and not otherwise defined
herein shall have the meanings ascribed thereto in the Rights Agreement.

Section 2. Issuance of Rights.

     (a) Effective upon the consummation of the Purchase Agreement, the Company
hereby issues that number of Rights in respect of each share of the Series B
Preferred Stock equal to the number of shares of Common Stock into which a share
of the Series B Preferred Stock is convertible, as set forth, and subject to
adjustments as provided, in the

<PAGE>

Certificate, effective upon the closing under the Purchase Agreement, such
Rights having the terms (including, without limitation, the terms relating to
the redemption thereof), being entitled to the benefits of, and being subject to
the conditions of the Rights Agreement, as if such Rights initially were issued
pursuant thereto. Such Rights shall be evidenced by the certificates
representing the Series B Preferred Stock until the Distribution Date, or by the
Rights Certificates in substantially the form of Exhibit A to the Rights
Agreement subsequent to the Distribution Date. Upon conversion of any share of
the Series B Preferred Stock prior to the Distribution Date, the Rights issued
in respect of such share of the Series B Preferred Stock shall cease to exist
and the holder of the Common Stock received upon conversion of such shares of
the Series B Preferred Stock shall be issued Rights in accordance with the
provisions of the Rights Agreement.

     (b) Subject to Section 2(a) above, the number of Rights issued in respect
of each share of the Series B Preferred Stock, the Purchase Price and the number
or kind of shares of capital stock issuable upon exercise of the Rights shall be
subject to adjustment from time to time only in accordance with the terms of the
Rights Agreement.

     (c) In the event that the Rights shall be redeemed by the Board of
Directors of the Company in accordance with their terms while any share of the
Series B Preferred Stock is outstanding, the holder of any share of the Series B
Preferred Stock then outstanding shall have the right to receive the Redemption
Price with respect to each Right then held by such holder.

Section 3. Rights Agent.

     The Company hereby appoints the Rights Agent to act as agent for the
Company and the holders of the Series B Preferred Stock in accordance with the
terms and conditions hereof and of the Rights Agreement, and the Rights Agent
hereby accepts such appointment. The Company may from time to time act as
Co-Rights Agent or appoint such Co-Rights Agent as it may deem necessary or
desirable. The Rights Agent agrees to be bound by the terms and conditions of
and to assume and fulfill the duties and obligations of the Rights Agent under
this Agreement and under the Rights Agreement. Any action which may be taken by
the Rights Agent pursuant to the terms of this Agreement and the Rights
Agreement may be taken by any such Co-Rights Agent.

Section 4. No Effect on Rights Agreement.

     Nothing herein shall be deemed or construed to alter or amend the Rights
Agreement in any respect, except as expressly set forth herein.

Section 5. Supplements and Amendments.

     The Company and the Rights Agent may supplement or amend this Agreement
without the approval of any holders of the Rights Certificates in order to cure
any ambiguity, to correct or supplement any provision contained herein which may
be defective or inconsistent with any other provisions herein, or to make any
other provisions in regard to matters or questions arising hereunder which the
Company and the Rights Agent may deem necessary or

                                       2
<PAGE>

desirable and which shall not adversely affect the interests of the holders of
the Rights Certificates.

Section 6. Further Assurances.

     The Company shall take such action as may be necessary to assure that the
holders of the Series B Preferred Stock shall receive the full benefits of the
Rights, including, without limitation, to assure that the Rights issued pursuant
to this Agreement may be exercised in accordance with Section 13 of the Rights
Agreement in the event of the occurrence of an event specified in Section 13
thereof.

Section 7. Benefits of this Agreement; Rights of Action.

     Nothing in this Agreement except for the provisions of Section 8 hereof
shall be construed to give to any person or corporation other than the Company,
the Rights Agent and the registered holders of the Rights Certificates issued
pursuant to this Agreement (and, prior to the Distribution Date, the registered
holders of the Series B Preferred Stock) any legal equitable right, remedy or
claim under this Agreement; but this Agreement except for the provisions of
Section 8 hereof shall be for the sole and exclusive benefit of the Company, the
Rights Agent and the registered holders of the Rights Certificates issued
pursuant to this Agreement (and, prior to the Distribution Date, the registered
holders of the Series B Preferred Stock). All rights of action in respect of
this Agreement except for the provisions of Section 8 hereof are vested in the
respective registered holders of the Series B Preferred Stock); and any
registered holder of a Rights Certificate issued pursuant to this Agreement (or,
prior to the Distribution Date, any registered holder of the Series B Preferred
Stock), without the consent of the Rights Agent or of the holder of any other
Rights Certificate issued pursuant to this Agreement (or, prior to the
Distribution Date, any other registered holder of the Series B Preferred Stock),
may, in his own behalf and for his own benefit, enforce, and may institute and
maintain any suit, action or proceeding against the Company to enforce, or to
otherwise act in respect of, his rights here-under and his right to exercise the
Rights evidenced by such Rights Certificate in the manner provided in such
Rights Certificate and in the Rights Agreement. Without limiting the foregoing
or any remedies available to the holders of Rights issued pursuant to this
Agreement, it is specifically acknowledged that the holders of Rights issued
pursuant to this Agreement would not have an adequate remedy at law for any
breach of this Agreement or the Rights Agreement and will be entitled to
specific performance of the obligations under, and injunctive relief against
actual or threatened violations of, the obligations of any Person subject to
this Agreement and the Rights Agreement.

Section 8. Amendment of Rights Agreement. The definition of "Acquiring Person"
set forth in Section 1(a) is deleted in its entirety and the following
substituted in lieu thereof:

     (a) "Acquiring Person" shall mean any Person who or which, together with
     all Affiliates and Associates of such Person, shall be the Beneficial Owner
     of securities of the Company constituting a Substantial Block, but shall
     not include (i) any employee

                                       3
<PAGE>

     benefit plan of the Company, (ii) any members of the Lee Family, but only
     for so long as the aggregate number of shares of Common Stock of which the
     Lee Family is deemed to be the Beneficial Owner of is less than or equal to
     30% of the number of shares of Common Stock then outstanding, (iii) (A)
     Apax Managers, Inc., Apax Partners Europe Managers Ltd. (individually or
     collectively, "Apax"), or its or their Affiliates and Associates or any
     investment fund for whom Apax or its Affiliates or Associates serves as a
     general partner or managing member ("Apax Affiliates"), but only for so
     long as the aggregate number of shares of Common Stock of which Apax and
     the Apax Affiliates are deemed to be the Beneficial Owner is less than or
     equal to 49.5% on a fully diluted basis of the number of shares of Common
     Stock then outstanding (including shares of Common Stock issuable upon
     conversion of all outstanding Series B Convertible Preferred Stock, par
     value $100 per share of the Company (the "Series B Stock") and other
     convertible securities) (the "Limitation") or (B) any other Person who
     acquires shares of Series B Stock, Common Stock or securities convertible
     into Common Stock in accordance with Section 3.3(b)(iii) of the Investors'
     Rights Agreement, dated as of February 12, 2003, by and between the Company
     and certain investors listed therein (the "Apax Transferee"), but only for
     so long as the aggregate number of shares of Common Stock of which the Apax
     Transferee is deemed to be the Beneficial Owner is less than or equal to
     the Limitation subject, in each case, to upward adjustment for actions by
     the Company (the "Adjustment", and together with the Limitation, the
     "Permitted Amount"), including, (a) repurchases of Common Stock or
     securities convertible into Common Stock, (b) shares of Common Stock
     issuable upon conversion of the dividends on the Series B Stock, (c)
     adjustments to the conversion price of the Series B Stock, or (d) any
     recapitalization of the Company, and, provided further, Apax or the Apax
     Affiliates shall have a period of 20 business days from the date on which
     such party learns, or receives written notice from the Company, that such
     party holds an amount of shares which exceed the Permitted Amount to
     dispose of all shares which cause such party to exceed the Permitted
     Amount, provided that such party did not intentionally exceed the Permitted
     Amount, or (iv) any Person who acquires a Substantial Block in connection
     with a transaction or series of transactions approved prior to such
     transaction or transactions by the Board of Directors.

Section 9. Counterparts.

     This Agreement may be executed in any number of counterparts and each of
such counterparts shall for all purposes be deemed to be an original, and all
such counter-parts shall together constitute but one and the same instrument.

Section 10. Law Governing.

     This Agreement shall be deemed to be a contract made under the laws of the
State of Delaware and for all purposes shall be governed by and construed in
accordance with the laws of such State applicable to contracts to be made and
performed entirely within such State, provided, however, that the rights and
obligations of the Rights Agent shall be governed by and construed with the laws
of the State of New York.


                                       4
<PAGE>

Section 11. Descriptive Headings.

     Descriptive headings of the several Sections of this Agreement are inserted
for convenience only and shall not control or affect the meaning or construction
of any of the provisions hereof.









                                       5
<PAGE>


     IN WITNESS WHEREOF, the parties hereto have caused this Agreement to be
duly executed as of the date and year first above written.

                                  PHILLIPS-VAN HEUSEN CORPORATION

                                  By:  /s/ Mark D. Fischer
                                       -------------------
                                  Name:   Mark D. Fischer
                                  Title:  Vice President, General Counsel,
                                          Secretary


                                  THE BANK OF NEW YORK

                                  By:  /s/ Edward Timmons
                                       ------------------
                                  Name:   Edward Timmons
                                  Title:  Vice President






                                       6