<PAGE>

                                                                    EXHIBIT 10.4

                                 FIFTH AMENDMENT

                  TO THE FIRST AMENDMENT AND RESTATEMENT OF THE

                              WILLIAMS-SONOMA, INC.

                         ASSOCIATE STOCK INCENTIVE PLAN

                               (2002 RESTATEMENT)

         Williams-Sonoma, Inc., a California corporation (the "Company"), hereby
makes this Fifth Amendment to the First Amendment and Restatement of the
Williams-Sonoma, Inc. Associate Stock Incentive Plan, generally effective
January 1, 1997, with reference to the following facts:

A.       The Company maintains the Williams-Sonoma, Inc. Associate Stock
         Incentive Plan (the "ASIP"), which was most recently amended and
         restated in its entirety in 2002, for the benefit of eligible
         employees.

B.       The Company wishes to amend the ASIP to set forth expressly the Board's
         authority to delegate its power to amendment the ASIP, and for other
         reasons.

C.       By Section 8.4 of the Plan, the Company has reserved the right to amend
         the ASIP.

         NOW, THEREFORE, the Plan is hereby amended as follows:

         1.       Section 8.4 shall be amended to read as follows:

                                    8.4 Amendments: Subject to Section 8.3
                  (which concerns distributing assets after plan termination)
                  and in accordance with Section 8.5 (which concerns action by
                  the Company), the Company reserves the right in its discretion
                  to amend this Plan at any time and from time to time.
                  Effective for amendments adopted before June 1, 2003, such
                  action shall be by action of the Company's Board of Directors.
                  Notwithstanding any other provision of the Plan, the Company
                  may make any amendment, with or without retroactive effect,
                  that: (a) the Company determines necessary or desirable to
                  comply with ERISA, the Code or other applicable law, (b) is
                  required by the Internal Revenue Service as a pre-condition to
                  the issuance of a favorable determination that the Plan
                  continues to be a qualified plan within the meaning of Code
                  section 401(a), or (c) the Company determines is advisable to
                  clarify the terms of this document, even retroactively, by an
                  amendment accomplishing a good faith correction of any
                  typographical error, inadvertent ambiguity or scrivener's
                  error. Any amendment of the Plan by the Company shall be
                  binding on each Affiliate, without any further action by any
                  such Affiliate. Affiliates shall not have the authority to
                  amend the Plan.

<PAGE>

         2.       A new Section 8.5 shall be added to read as follows:

                                    8.5 Action by the Company: Any action by the
                  Company, including any amendment authorized to be made under
                  Section 8.4 (which concerns the procedure for amending this
                  Plan), shall be made either by a resolution adopted by the
                  Board, or by action taken on behalf of the Company by any
                  committee, person or persons to the extent that such
                  committee, person or persons is authorized by the Board to
                  take such action on behalf of the Company.

This Amendment shall supercede the provisions of the Plan to the extent those
provisions are inconsistent with the provisions of this Amendment.

         In all other respects, the terms and provisions of the Plan are hereby
ratified and declared to remain in full force and effect.

         IN WITNESS WHEREOF, this Amendment has been executed this 30 day of
May, 2003.

                                      WILLIAMS-SONOMA, INC.

                                      By:    /s/ James E. Boike
                                             -------------------------------
                                      Title: Executive Vice President and
                                             Chief Operating Officer

                                       -2-