<PAGE>

                                                                    EXHIBIT 10.4



                     FIRST AMENDMENT TO TERM LOAN AGREEMENT

         This First Amendment to Term Loan Agreement (this "First Amendment"),
is made as of the 12th day of February, 2003, by and between PHILLIPS-VAN HEUSEN
CORPORATION, a Delaware corporation (the "Borrower"), each of the lenders
executing a signature page hereto (each a "Lender" and collectively, the
"Lenders"), and APAX MANAGERS, INC., a New York corporation, as administrative
agent (the "Administrative Agent") for the Lenders.

                                R E C I T A L S:
                                - - - - - - - -

         WHEREAS, pursuant to a Securities Purchase Agreement dated as of
December 16, 2002 (the "Securities Purchase Agreement") among the Borrower and
the investors party thereto (the "Investors"), the Borrower has agreed to issue
and sell to the Investors 10,000 shares of the Borrower's Series B Convertible
Preferred Stock, $100.00 par value per share ("Series B Preferred Stock"), for
an aggregate purchase price of $250,000,000;

         WHEREAS, pursuant to the Securities Purchase Agreement, the proceeds of
the sale of Series B Preferred Stock to the Investors shall be applied to pay a
portion of the purchase price for the Borrower's acquisition of the Capital
Stock of the CK Companies and pay the fees and out-of-pocket expenses relating
to the Contemplated Transactions ("Transaction Costs");

         WHEREAS, in order to enable the Borrower to pay an additional portion
of the purchase price for the Capital Stock of the CK Companies and related
Transaction Costs, pursuant to a Term Loan Agreement dated as of December 16,
2002 (the "Term Loan Agreement") among the parties hereto the Lenders have
agreed to provide a bridge loan to the Borrower in the aggregate principal
amount of One Hundred Twenty Five Million Dollars ($125,000,000), said bridge
loan to be refinanced on a best efforts expedited basis as provided in the Term
Loan Agreement (capitalized terms used herein and not otherwise defined herein
shall have the respective meanings set forth in the Term Loan Agreement); and

         WHEREAS, the Borrower has requested that the Lenders amend the Term
Loan Agreement to permit the Borrower to draw down $100,000,000 of the principal
amount of the Loan on the Closing Date with the option, upon delivery of timely
written notice, to draw down up to the balance of the principal amount of the
Loan which the Lenders have agreed to provide at any time on or before June 30,
2003;

         WHEREAS, the Lenders have agreed to such amendment on the terms and
subject to the conditions set forth herein.

         NOW, THEREFORE, in consideration of the foregoing and of the respective
covenants and undertakings hereunder and for other good and valuable
consideration, the receipt and adequacy of which are hereby acknowledged,
intending to be legally bound, the parties hereto do hereby agree as follows:

<PAGE>


1.       AMENDMENT

         1.1 Section 2.1 of the Term Loan Agreement is hereby amended and
restated in its entirety as follows:

         "2.1 Commitment. Subject to the terms and conditions of this Agreement,
the Lenders hereby agree to provide a bridge loan to Borrower in the form of a
term loan, in the aggregate principal amount of One Hundred Twenty Five Million
Dollars ($125,000,000) (the "Loan"). The Lenders shall advance an aggregate
principal amount of One Hundred Million Dollars ($100,000,000) on the Closing
Date. The amount of the Loan to be advanced by each Lender on the Closing Date
and the principal amount of the Note to be issued to such Lender upon the
Closing Date shall be as set forth on Schedule 2.1. All or any part of the
remaining Twenty Five Million Dollars ($25,000,000) of the principal amount of
the Loan which the Lenders have agreed to provide (such amount being herein
referred to as the "Subsequent Drawdown") will be advanced by the Lenders at any
time on or before June 30, 2003, on fifteen business days prior written notice
by the Borrower to each Lender (any such notice, the "Subsequent Drawdown
Notice"); provided, however, that the Borrower shall be entitled to deliver only
one Subsequent Drawdown Notice. The amount of the Subsequent Drawdown to be
advanced by each Lender following receipt of the Subsequent Drawdown Notice and
the principal amount of the Note to be issued to each such Lender upon the
funding of the Subsequent Drawdown shall be in proportion to the amount of the
Loan advanced by each such Lender on the Closing Date. The Loan is subject to
refinancing on a best efforts expedited basis as provided herein."

         1.2 Schedule 2.1 to the Term Loan Agreement is hereby amended and
restated in its entirety to read as set forth on Schedule 2.1 hereto.

         1.3 Section 2.2(a) of the Term Loan Agreement is hereby amended and
restated in its entirety as follows:

         "(a) Interest on the outstanding principal amount of each Note
("Interest") shall accrue from and including the date of the issuance of such
Note at the rate of 10% per annum through and until the Initial Maturity Date
and, if the Borrower elects to extend the term of the Loan as provided in
Section 2.3, thereafter at the rate of 15% per annum until the Final Maturity
Date. Interest on each Note shall be paid quarterly in arrears on each March 31,
June 30, September 30 and December 31 of each year (each, an "Interest Payment
Date") or, if any such date shall not be a Business Day, on the next succeeding
Business Day to occur after such date, beginning on the first Interest Payment
Date to occur after the date of the issuance of such Note. Interest shall be
computed on the basis of a 360-day year of twelve 30-day months."

         1.4 Section 2.3(a) of the Term Loan Agreement is hereby amended and
restated in its entirety as follows:

         "(a) Subject to the provisions of Section 2.3(b) hereof, the principal
amount of the Loan then outstanding, together with all accrued and unpaid
Interest, shall be due and payable upon the Initial Maturity Date."

<PAGE>



         1.5 Section 2.3(b) of the Term Loan Agreement is hereby amended and
restated in its entirety as follows:

         "(b) The Borrower, at its option, may elect to extend the term of the
principal amount of the Loan then outstanding until the Final Maturity Date by
(i) providing the Administrative Agent irrevocable written notice of such
election at least 30 days and not earlier than 45 days prior to the Initial
Maturity Date and (ii) paying the Lenders on or before the Initial Maturity Date
an extension fee of Four Million Dollars ($4,000,000), in the aggregate (the
"Extension Fee"); such fee to be allocated pro rata among the Lenders in
proportion to the outstanding principal amount of Notes held by the Lenders on
the date of such payment."

         1.6 The Term Loan Agreement is hereby amended by including Schedules
6.1, 6.2, 6.3 and 6.8 to this First Amendment as Schedules 6.1, 6.2, 6.3 and 6.8
to the Term Loan Agreement.

2.       Miscellaneous.

              (a) Limitations. Except as expressly amended hereby, the Term Loan
Agreement shall remain in full force and effect in accordance with its terms.

              (b) Captions. The captions in this Agreement are for convenience
of reference only and shall not be given any effect in the interpretation of
this Agreement.

              (c) Severability. If one or more provisions of this Agreement are
held to be unenforceable under applicable Law, such provision(s) shall be
excluded from this Agreement and the balance of this Agreement shall be
interpreted as if such provision were so excluded and shall be enforceable in
accordance with its terms so long as the economic or legal substance of the
transactions contemplated by this Agreement are not affected in any manner
materially adverse to any party.

              (d) Successors and Assigns. This First Amendment shall be binding
upon and shall inure to the benefit of and be enforceable by, the parties and
their respective successors, transferees and assigns.

              (e) Choice of Law. This First Amendment shall be governed by the
internal laws of the State of New York, without regard to the conflicts of law
principals thereof which would specify the application of the law of another
jurisdiction.

              (f) Counterparts. This First Amendment may be executed in two (2)
or more counterparts, each of which shall be considered an original, but all of
which shall together constitute the same instrument.



<PAGE>


         IN WITNESS WHEREOF, this First Amendment has been duly executed as of
the date first above written.

                                  PHILLIPS-VAN HEUSEN CORPORATION


                                  By:      /s/ Pamela N. Hootkin
                                           ---------------------
                                  Name:   Pamela N. Hootkin
                                  Title:  Vice President, Treasurer


                                  LENDERS:

                                  APAX EXCELSIOR VI, L.P.
                                  By:      Apax Excelsior VI Partners, L.P.,
                                           its general partner
                                  By:      Apax Managers, Inc.,
                                           its general partner


                                  By:      /s/ David H. Landau
                                           -------------------
                                  Name:   David Landau
                                  Title:  Vice President


                                  APAX EXCELSIOR VI-A C.V.
                                  By:      Apax Excelsior VI Partners, L.P.,
                                           its general partner
                                  By:      Apax Managers, Inc.,
                                           its general partner


                                  By:      /s/ David H. Landau
                                           -------------------
                                  Name:   David Landau
                                  Title:  Vice President


                                  APAX EXCELSIOR VI-B C.V.
                                  By:      Apax Excelsior VI Partners, L.P.,
                                           its general partner
                                  By:      Apax Managers, Inc.,
                                           its general partner


                                  By:      /s/ David Landau
                                           ----------------
                                  Name:   David Landau
                                  Title:  Vice President

<PAGE>



                                  PATRICOF PRIVATE INVESTMENT CLUB III, L.P.
                                  By:      Apax Excelsior VI Partners, L.P.,
                                           its general partner
                                  By:      Apax Managers, Inc.,
                                           its general partner


                                  By:      /s/ David H. Landau
                                           -------------------
                                  Name:   David Landau
                                  Title:  Vice President

                                  APAX EUROPE V - A, L.P.
                                  By: Apax Partners Europe Managers Ltd.,
                                      its Investment Manager


                                  By:      /s/ Adrian Beecroft
                                           -------------------
                                  Name:   Adrian Beecroft
                                  Title:  Managing Director


                                  By:      /s/ Clive Sherling
                                           -------------------
                                  Name:   Clive Sherling
                                  Title:  Managing Director

                                  APAX EUROPE V - B, L.P.
                                  By:  Apax Partners Europe Managers Ltd.,
                                       its Investment Manager


                                  By:      /s/ Adrian Beecroft
                                           -------------------
                                  Name:   Adrian Beecroft
                                  Title:  Managing Director

                                  By:      /s/ Clive Sherling
                                           -------------------
                                  Name:   Clive Sherling
                                  Title:  Managing Director



<PAGE>


                                  APAX EUROPE V - C GMBH & CO. KG
                                  By:  Apax Partners Europe Managers Ltd.,
                                       its Investment Manager


                                  By:      /s/ Adrian Beecroft
                                           -------------------
                                  Name:   Adrian Beecroft
                                  Title:  Managing Director

                                  By:      /s/ Clive Sherling
                                           -------------------
                                  Name:   Clive Sherling
                                  Title:  Managing Director

                                  APAX EUROPE V - D, L.P.
                                  By:  Apax Partners Europe Managers Ltd.,
                                       its Investment Manager


                                  By:      /s/ Adrian Beecroft
                                           -------------------
                                  Name:   Adrian Beecroft
                                  Title:  Managing Director

                                  By:      /s/ Clive Sherling
                                           -------------------
                                  Name:   Clive Sherling
                                  Title:  Managing Director

                                  APAX EUROPE V - E, L.P.
                                  By:  Apax Partners Europe Managers Ltd.,
                                       its Investment Manager


                                  By:      /s/ Adrian Beecroft
                                           -------------------
                                  Name:   Adrian Beecroft
                                  Title:  Managing Director

                                  By:      /s/ Clive Sherling
                                           -------------------
                                  Name:   Clive Sherling
                                  Title:  Managing Director



<PAGE>


                                  APAX EUROPE V - F, C.V.
                                  By:  Apax Partners Europe Managers Ltd.,
                                       its Investment Manager


                                  By:      /s/ Adrian Beecroft
                                           -------------------
                                  Name:   Adrian Beecroft
                                  Title:  Managing Director

                                  By:      /s/ Clive Sherling
                                           -------------------
                                  Name:   Clive Sherling
                                  Title:  Managing Director

                                  APAX EUROPE V - G, C.V.
                                  By:  Apax Partners Europe Managers Ltd.,
                                       its Investment Manager


                                  By:      /s/ Adrian Beecroft
                                           -------------------
                                  Name:   Adrian Beecroft
                                  Title:  Managing Director

                                  By:      /s/ Clive Sherling
                                           -------------------
                                  Name:   Clive Sherling
                                  Title:  Managing Director

                                  APAX EUROPE V - 1, L.P.
                                  By:  Apax Partners Europe Managers Ltd.,
                                       its Investment Manager


                                  By:      /s/ Adrian Beecroft
                                           -------------------
                                  Name:   Adrian Beecroft
                                  Title:  Managing Director

                                  By:      /s/ Clive Sherling
                                           -------------------
                                  Name:   Clive Sherling
                                  Title:  Managing Director



<PAGE>


                                  APAX EUROPE V - 2, L.P.
                                  By:  Apax Partners Europe Managers Ltd.,
                                       its Investment Manager


                                  By:      /s/ Adrian Beecroft
                                           -------------------
                                  Name:   Adrian Beecroft
                                  Title:  Managing Director

                                  By:      /s/ Clive Sherling
                                           -------------------
                                  Name:   Clive Sherling
                                  Title:  Managing Director