<PAGE>   1
                                                                    EXHIBIT 10.4

                          AMENDMENT TO TRUST AGREEMENT

         This Amendment to Trust Agreement made as of April 11, 2001, between
Venator Group, Inc., a New York corporation with its principal office at 112
West 34 Street, New York, New York 10120 (formerly Woolworth Corporation) (the
"Company") and The Bank of New York, a New York banking corporation whose
address is One Wall Street, New York, New York 10005 (the "Trustee").

         WHEREAS, the Company and the Trustee are parties to a Trust Agreement
dated November 12, 1987 (the "Trust Agreement"); and

         WHEREAS, on June 11, 1998 the Company amended its Certificate of
Incorporation to change its name from Woolworth Corporation to Venator Group,
Inc.; and

         WHEREAS, the Company has not, prior to the date hereof, has not
provided the Trustee with a Payment Schedule provided for in Section 12(a) of
the Trust Agreement;

         NOW, THEREFORE, the parties do hereby amend the Trust Agreement,
effective the date hereof, as follows:

         A. Section 1(b) of the Trust Agreement is amended to read, in its
entirety, as follows:

         (b) The Trust hereby established shall be revocable by the Company at
         any time until the later of (i) 30 days following the issuance by the
         Internal Revenue Service of tax rulings to be requested by the Company
         in conjunction with the establishment of the Trust to the effect that
         the Company is the owner of the Trust within the meaning of Sections
         671 et. seq. of the Code and that the Executives will not be subject to
         income tax on amounts contributed to the Trust prior to the actual
         receipt of funds from the Trust or (ii) the occurrence of a Potential
         Change in control, as hereinafter defined; thereafter, except as
         provided in Section 11(a)(iii), the Trust shall be irrevocable.
         Notwithstanding the foregoing, the Company may, during any period of
         time that the Trust is revocable, declare the Trust to be irrevocable
         by delivering to the Trustee a certified copy of a resolution to that
         effect which has been adopted by the Compensation Committee of the
         Board of Directors of the Company, or any successor thereto (the
         "Committee").

         B. Section 1(d)(iii) of the Trust Agreement is amended to read, in its
entirety, as follows:

         (iii) any person, other than the Company, a trustee or other fiduciary
         holding voting securities of the Company under an employee benefit plan
         of the Company, is or becomes the "Beneficial Owner" (as defined in
         Rule 13d-3 under
<PAGE>   2
         the Exchange Act), directly or indirectly, of securities of the Company
         representing 20% or more of the combined voting power of the Company's
         then outstanding voting securities; or

         C. Section 11(a)(i) of the Trust Agreement is amended to read, in its
entirety, as follows:

         (a) (i) Except as provided in this Section 11, the Trust Agreement may
         only be amended by a written instrument executed by the Trustee and the
         Company (I) provided that no such amendment shall (A) cause the Trust
         to be revoked after it has become irrevocable in accordance with
         Section 1(b) other than an amendment pursuant to 11(a)(iii) below; or
         (B) other than changes made in accordance with subsection (ii) below,
         or amendments certified by the Chairman or President of the Company to
         be made in accordance with the terms of the Nonqualified Plans, the
         Contract or any Other Plan, as the case may be, or as may be required
         by law, reduce the Nonqualified Plan Benefits, the Contract Benefits or
         Other Benefits, as the case may be, or diminish the rights of any
         nonconsenting Executive or (C) alter or otherwise be inconsistent with
         Section 8(f) or Section 11(b) and (II) provided further that, following
         the occurrence of a Potential Change in Control, the written consent of
         a majority of the Executives who, at the time of such amendment, are
         listed on a Payment Schedule shall be required for such amendment to be
         effective.

         D. The address to which notices may be provided to the Company, as
provided for in Section 13 of the Trust Agreement, shall be:

         TO THE COMPANY AT:

                  Venator Group, Inc.
                  112 West 34th Street
                  New York, New York 10120
                  Attention:  Chief  Executive Officer

         and a copy to:

                  Venator Group, Inc.
                  112 West 34th Street
                  New York, New York 10120
                  Attention:  General Counsel

         E. In all other respects the terms and provisions of the Trust
Agreement are hereby ratified and confirmed.

         IN WITNESS WHEREOF, the Company and Trustee have executed this
Amendment to Trust Agreement as of the date first above written.
<PAGE>   3
                                            VENATOR GROUP, INC.

ATTEST:
                                        By: /s/ John H. Cannon
/s/ Gary M. Bahler                          Title: Vice President and Treasurer
Secretary

                                            THE BANK OF NEW YORK,
                                            as Trustee

ATTEST:
                                        By: Michael T. Shayne
/s/ Mary P. Galligan                        Title: Vice President
Title: Vice President