<PAGE>

                                                                     EXHIBIT 4.1


=============================================================================


                             MODIFICATION AGREEMENT


=============================================================================

<PAGE>

                               Table of Contents

<TABLE>
<CAPTION>
                                                                                                              Page
<S>                                                                                                           <C>
1.     Amendment to Certificate of Designations.................................................................1

2.     Investor Exchange Right..................................................................................1
       2.1      Investor Exchange Right.........................................................................1
       2.2      Exercise of Investor Exchange Right.............................................................1

3.     Closing of Exchange......................................................................................2

4.     Representations and Warranties of the Company to the Investor............................................2
       4.1      Corporate Organization and Authority............................................................2
       4.2      No Conflict.....................................................................................2
       4.3      Governmental and Third Party Consents...........................................................3
       4.4      The Notes.......................................................................................3

5.     Representations and Warranties of the Investor...........................................................3
       5.1      Corporate Organization and Authority............................................................3
       5.2      No Conflict.....................................................................................3
       5.3      Governmental and Third Party Consents...........................................................3
       5.4      Investment......................................................................................4
       5.5      No Public Market................................................................................4
       5.6      Experience......................................................................................4
       5.7      Accredited Investor.............................................................................4

6.     Transfer Restrictions....................................................................................4
       6.1      Transfers in Accordance with this Agreement.....................................................4
       6.2      No Transfer Period..............................................................................5
       6.3      Limitations on Transferability..................................................................5

7.     Miscellaneous............................................................................................5
       7.1      Governing Law...................................................................................5
       7.2      Counterparts....................................................................................5
       7.3      Headings........................................................................................5
       7.4      Notices.........................................................................................5
       7.5      Waiver..........................................................................................5
       7.6      Survival of Warranties..........................................................................6
       7.7      Amendment of Agreement..........................................................................6
       7.8      Expenses........................................................................................6
       7.9      Entire Agreement................................................................................6
       7.10     Successors and Assigns; Benefits................................................................6
</TABLE>

                                      -i-

<PAGE>

                             MODIFICATION AGREEMENT

         MODIFICATION AGREEMENT dated as of May 17, 2002 (the "Agreement")
between St. John Knits International, Incorporated, a Delaware corporation (the
"Company") and Madeleine, L.L.C., a New York limited liability company (the
"Investor").

         WHEREAS, Investor has acquired (the "Acquisition") from Vestar/SJK
Investors LLC, a Delaware limited liability company, Kelly A. Gray and the Gray
Family Trust a total of 250,000 shares (the "Shares") of 15 1/4% Exchangeable
Preferred Stock Due 2010, par value $0.01 per share, of the Company (the
"Exchangeable Preferred Stock").

         WHEREAS, the Exchangeable Preferred Stock is exchangeable at the option
of the Company into 15 1/4% Subordinated Exchange Debentures Due 2010 (the
"Subordinated Debentures") with economic terms comparable to the Exchangeable
Preferred Stock (the "Exchange").

         WHEREAS, the Investor has requested of the Company and the Company has
agreed that in lieu of the Subordinated Debentures, the Exchangeable Preferred
Stock shall be exchangeable into the Company's 15 1/4% Senior Subordinated Notes
due 2009 in the form attached hereto as Exhibit A (the "Notes").

        THE PARTIES AGREE AS FOLLOWS:

1.   Amendment to Certificate of Designations. The Company agrees to use its
reasonable best efforts to amend the Certificate of Designations for the
Exchangeable Preferred Stock (the "Certificate") to provide that the
Exchangeable Preferred Stock is exchangeable into the Notes in lieu of the
Subordinated Debentures. The amendment to the Certificate is attached hereto as
Exhibit B (the "Amendment"). The Company agrees to use its reasonable best
efforts to file the Amendment with the Delaware Secretary of State so that such
Amendment shall be effective as of the date of the Acquisition.

2.   Investor Exchange Right.

     2.1 Investor Exchange Right. Investor shall have the right (the "Investor
Exchange Right"), at any time beginning six months from the consummation of the
Acquisition, to require the Company to exercise an Exchange in accordance with
the terms of the Certificate. If at any time the Investor shall Transfer (other
than to an affiliate) all or any of the Shares, the Investor Exchange Right
shall terminate with respect to the Shares subject to such Transfer.

     2.2 Exercise of Investor Exchange Right. In the event the Investor wishes
to exercise an Investor Exchange Right and require the Company to exercise an
Exchange with respect to any or all of the Shares, the Investor shall send a
written notice to the Company of such exercise (the "Investor Notice"),
specifying the number of Shares subject to such Exchange. Within 15 days of
receipt of the Investor Notice, the Company shall exercise such Exchange in
accordance with the terms of the Certificate.

<PAGE>

                                                                               2

3.   Closing of Exchange.

     (a) At the closing of an Exchange, (i) Investor shall deliver to the
Company the Shares subject to such Exchange by delivery of a duly executed
certificate representing such Shares duly endorsed in blank or accompanied by
stock powers duly executed in blank, and (ii) the Company shall deliver to the
Investor the Notes in an aggregate principal amount equal to (a) the aggregate
liquidation preference of the Shares subject to such Exchange plus (b) an amount
equal to the accrued but unpaid dividends on such Shares plus (c) an amount
calculated pursuant to the last sentence of Section 4(a) of the Certificate.

     (b) At the Closing of such Exchange, Investor shall deliver to the Company
a certificate, dated as of the closing and signed by Investor, certifying that
Investor holds of record and beneficially the Shares subject to such Exchange,
free and clear of all liens and encumbrances and that upon delivery of such
Shares as contemplated herein, Investor will transfer to the Company valid title
to such Shares, free and clear of all liens and encumbrances.

4.   Representations and Warranties of the Company to the Investor. The
Company hereby represents and warrants to the Investor that:

     4.1 Corporate Organization and Authority. The Company has been duly
incorporated and is validly existing and in good standing under the laws of the
State of Delaware and has all requisite corporate power and authority to execute
and deliver this Agreement and execute and file the Amendment and after (i)
approval of the Amendment by both the holders of a majority of the Company's
common stock, par value $.01 per share, and the holders of a majority of the
Exchangeable Preferred Stock (collectively, the "Stockholders Approvals") (which
Stockholder Approvals the Company has obtained) and (ii) the filing of the
Amendment, will have all requisite corporate power and authority to consummate
an Exchange and issue the Notes. The execution and delivery by the Company of
this Agreement, and after the receipt of the Stockholder Approvals, the
consummation of an Exchange and the issuance of the Notes will have been duly
authorized by all requisite corporate action. This Agreement has been duly
executed and delivered by the Company and, assuming the due authorization,
execution and delivery thereof by Investor, constitutes a legal, valid and
binding obligation of the Company, enforceable against the Company in accordance
with its terms, except to the extent that such enforcement may be subject to
applicable bankruptcy, insolvency, reorganization, moratorium, or other laws of
general application relating to or affecting enforcement of creditors' rights
and laws concerning equitable remedies.

     4.2 No Conflict. After the filing of the Amendment, the execution, delivery
and performance by the Company of this Agreement, the consummation by the
Company of an Exchange and the issuance of the Notes do not and will not, with
or without the giving of notice or the passage of time or both, (i) violate the
provisions of any law, rule or regulation applicable to the Company or its
properties or assets; (ii) violate the provisions of the constituent
organizational documents or other governing instruments of the Company; or (iii)
violate any judgment, decree, order or award of any court, governmental or
quasi-governmental agency or arbitrator applicable to the Company or its
properties or assets.

<PAGE>

                                                                               3

     4.3 Governmental and Third Party Consents. Other than the filing of the
Amendment and the receipt of the Stockholder Approvals (which have already been
received), no consent, approval, exemption or authorization is required to be
obtained from, no notice is required to be given to and no filing is required to
be obtained from any third party (including, without limitation, governmental
and quasi-governmental agencies, authorities and instrumentalities of competent
jurisdiction) by the Company, in order (i) for this Agreement to constitute a
legal, valid and binding obligation of the Company or (ii) to authorize or
permit the consummation by the Company of an Exchange and the issuance and
delivery of the Notes.

     4.4 The Notes. The Notes have been duly authorized by the Company and, when
duly executed, issued and delivered as provided herein and exchanged as provided
herein, will be duly and validly issued and outstanding and will constitute a
valid and legally binding obligation of the Company, enforceable against the
Company in accordance with their terms, except to the extent that such
enforcement may be subject to applicable bankruptcy, insolvency, reorganization,
moratorium, or other laws of general application relating to or affecting
enforcement of creditors' rights and laws concerning equitable remedies.

5.   Representations and Warranties of the Investor. The Investor represents
and warrants to the Company as follows:

     5.1 Corporate Organization and Authority. The Investor has been duly
organized and is validly existing and in good standing under the laws of the
State of Delaware and has all requisite limited liability company power and
authority to execute and deliver this Agreement and to perform its obligations
hereunder. The execution and delivery by the Investor of this Agreement, the
performance by the Investor of its obligations hereunder, and the consummation
by the Investor of the transactions contemplated hereby have been duly
authorized by all requisite limited liability company action. This Agreement has
been duly executed and delivered by the Investor and, assuming the due
authorization, execution and delivery thereof by the Company, constitutes a
legal, valid and binding obligation of the Investor, enforceable against the
Investor in accordance with its terms, except to the extent that such
enforcement may be subject to applicable bankruptcy, insolvency, reorganization,
moratorium, or other laws of general application relating to or affecting
enforcement of creditors' rights and laws concerning equitable remedies.

     5.2 No Conflict. The execution, delivery and performance by the Investor of
this Agreement and the consummation by the Investor of the transactions
contemplated hereby do not and will not, with or without the giving of notice or
the passage of time or both, (i) violate the provisions of any law, rule or
regulation applicable to the Investor or its properties or assets; (ii) violate
the provisions of the constituent organizational documents or other governing
instruments of the Investor; or (iii) violate any judgment, decree, order or
award of any court, governmental or quasi-governmental agency or arbitrator
applicable to the Investor or its properties or assets.

     5.3 Governmental and Third Party Consents. No consent, approval, exemption
or authorization is required to be obtained from, no notice is required to be
given to and no filing is required to be obtained from any third party
(including, without limitation, governmental and quasi-governmental agencies,
authorities and instrumentalities of competent jurisdiction) by the Investor, in
order (i) for this Agreement to constitute a legal, valid and binding obligation
of the

<PAGE>

                                                                               4

Investor or (ii) to authorize or permit the consummation by the Investor of the
purchase of the Notes.

     5.4 Investment. This Agreement is made with the Investor in reliance upon
its representation to the Company, which by the Investor's execution of this
Agreement Investor hereby confirms, that the Notes to be received by the
Investor upon the exercise by the Company of an Exchange, including after the
exercise by the Investor of the Investor Exchange Right, will be acquired for
investment for Investor's own account, not as a nominee or agent, and not with a
view to the sale or distribution thereof, and that the Investor has no present
intention of selling, granting any participation in, or otherwise distributing
the Notes. Notwithstanding the foregoing, by making the representations herein,
the Investor does not agree to hold the Shares for any minimum or other specific
term (subject to Section 6 hereof) and reserves the right to dispose of the
Shares, subject to Section 6 hereof, at any time, provided, however, that such
disposition shall be in accordance with or pursuant to a registration statement
or an exemption under the Securities Act and any applicable state securities
laws. By executing this Agreement, the Investor further represents that it has
no contract, undertaking, agreement, or arrangement with any person to sell,
transfer, or grant participation to such person or to any third person, with
respect to the Notes.

     5.5 No Public Market. The Investor understands and acknowledges that the
offering of the Notes pursuant to an Exchange, including after the exercise by
the Investor of the Investor Exchange Right, will not be registered under the
Securities Act on the grounds that the offering and sale of securities
contemplated by this Agreement are exempt from registration pursuant to Section
4(2) and or Section 3(b) of the Securities Act.

     5.6 Experience. The Investor represents that: (a) it has such knowledge and
experience in financial and business matters as to be capable of evaluating the
merits and risks of its prospective investment in the Notes; (b) it believes it
has received all the information it has requested from the Company and considers
necessary or appropriate for deciding whether to exercise the Investor Exchange
Right; (c) it has the ability to bear the economic risks of its prospective
investment; and (d) it is able, without materially impairing its financial
condition, to hold the Notes for an indefinite period of time and to suffer a
complete loss on its investment.

     5.7 Accredited Investor. The Investor presently qualifies and will as of
the closing of the Exchange qualify, as an "accredited investor" within the
meaning of Regulation D of the rules and regulations promulgated under the
Securities Act.

6.   Transfer Restrictions

     6.1 Transfers in Accordance with this Agreement. Investor shall not,
directly or indirectly, transfer, sell, assign, pledge, hypothecate, encumber,
or otherwise dispose of, the Shares or the Notes or any economic interest
therein (including without limitation by means of any participation or swap
transaction) (each, a "Transfer") to any person, except in compliance with the
Securities Act, applicable state and other foreign securities laws and this
Agreement. Any attempt to Transfer either of the Shares or the Notes in
violation of the terms of this Agreement shall be null and void.

<PAGE>

                                                                               5

     6.2 No Transfer Period. The Investor may not Transfer the Shares or the
Notes prior to the first anniversary of the Acquisition of the Shares to any
transferee that is not an affiliate of the Investor. Commencing during the
period beginning on the first anniversary of the Acquisition of the Shares and
ending 180 days thereafter (the "Limited Transfer Period"), the Investor may,
subject to Section 6.1 and 6.3 herein, Transfer the Shares or the Notes provided
that the Investor (together with any affiliate to whom Shares or Notes have been
transferred) retains at least 51% of any voting or consent rights associated
with the Shares and the Notes. Commencing after the Limited Transfer Period, the
Investor may, subject to Section 6.1 and 6.3 herein, Transfer Shares or the
Notes without restriction.

     6.3 Limitations on Transferability. The Investor covenants that in no event
will it dispose of the Shares or the Notes to any transferee that is not an
affiliate of the Investor unless and until (a) the Investor shall have notified
the Company of the proposed disposition and (b) if requested by the Company, the
Investor shall have furnished the Company with an opinion of counsel
satisfactory in form and substance to the Company and the Company's counsel to
the effect that (x) such disposition will not require registration under the
Securities Act and (y) appropriate action necessary for compliance with the
Securities Act and any applicable state, local, or foreign law has been taken;
provided, however, that no such opinion of counsel shall be required if the
Investor provides the Company with assurances, satisfactory in form and
substance to the Company, that the Shares or the Notes can be sold pursuant to
Rule 144(k) under the Securities Act. The Notes shall bear the legend
substantially in the form set forth in Exhibit A hereto.

7.   Miscellaneous.

     7.1 Governing Law. This Agreement shall be governed by, and construed in
accordance with, the laws of the State of New York.

     7.2 Counterparts. This Agreement may be executed in two or more
counterparts, each of which shall be deemed an original, but all of which
together shall constitute one and the same instrument.

     7.3 Headings. The headings of the sections of this Agreement are for
convenience and shall not by themselves determine the interpretation of this
Agreement.

     7.4 Notices. Any notice required or permitted hereunder shall be given in
writing and shall be conclusively deemed effectively given upon personal
delivery or delivery by courier, or on the first business day after transmission
if sent by confirmed facsimile transmission, or five days after deposit in the
United States mail, by registered or certified mail, postage prepaid, addressed
(i) if to the Company, as set forth below the Company's name on the signature
page of this Agreement and (ii) if to an Investor, at the Investor's address on
the signature page of this Agreement, or at such other address as the Company or
the Investor may designate by 5 days' advance written notice to the other
parties hereto.

     7.5 Waiver. Except as provided in the preceding sentence, no action taken
pursuant to this Agreement, including, without limitation, any investigation by
or on behalf of any party, shall be deemed to constitute a waiver by the party
taking such action of compliance with any

<PAGE>

                                                                               6

representations, warranties, covenants or agreements contained herein. The
waiver by any party hereto of a breach of any provision of this Agreement shall
not operate or be construed as a waiver of any preceding or succeeding breach
and no failure by any party to exercise any right or privilege hereunder shall
be deemed a waiver of such party's rights or privileges hereunder or shall be
deemed a waiver of such party's rights to exercise the same at any subsequent
time or times hereunder.

     7.6 Survival of Warranties. The warranties and representations of the
parties contained in or made pursuant to this Agreement shall survive for two
years after the execution and delivery of this Agreement.

     7.7 Amendment of Agreement. Any provision of this Agreement may be amended
by a written instrument signed by the Company and the Investor.

     7.8 Expenses. The Company and the Investor will bear their respective legal
and other fees and expenses with respect to this Agreement and the transactions
contemplated hereby.

     7.9 Entire Agreement. This Agreement contains the entire understanding of
the parties with respect to the subject matter hereof. There are no agreements,
representations, warranties, covenants or undertakings with respect to the
subject matter hereof other than those expressly set forth herein. This
Agreement supersedes all prior agreements and understandings between the parties
with respect to this subject matter.

     7.10 Successors and Assigns; Benefits. The parties to this Agreement may
not assign their respective rights nor delegate their respective obligations
hereunder without the prior consent of the other party hereto. Subject to the
exceptions specifically set forth in this Agreement, the terms and conditions of
this Agreement shall inure to the benefit of and be binding upon the respective
successors and assigns of the parties. Nothing in this Agreement, express or
implied, is intended or shall be construed to give any person other than the
parties to this Agreement or their respective successors or assigns any legal or
equitable right, remedy or claim under or in respect of any agreement or any
provision contained herein.

<PAGE>

                                                                               7

     IN WITNESS WHEREOF, the parties hereto have executed this Agreement as of
the date first above written.


                           ST. JOHN KNITS INTERNATIONAL, INCORPORATED

                           By: /s/ BOB GRAY
                               -------------------------------------------
                               Name:  Bob Gray
                               Title: Chief Executive Officer

                           Address:    St. John Knits International,
                                       Incorporated
                                       17622 Armstrong Avenue
                                       Irvine, California 92614
                                       Attention:  Robert E. Gray



                           MADELEINE, L.L.C.

                           By: /s/ ROBERT DAVENPORT
                               -------------------------------------------
                               Name:  Robert Davenport
                               Title:

                           Address:    Madeleine, L.L.C.
                                       c/o Cerberus Capital Management, L.P.
                                       450 Park Avenue, 28/th/ Floor
                                       New York, NY 10022
                                       Attention: Robert Davenport