<PAGE>
                                                                    Exhibit 4.17

                                 TRUST AGREEMENT
                                       OF
                            GOLDMAN SACHS CAPITAL IV


      THIS TRUST AGREEMENT is made as of January 20, 2004 (the "Trust
Agreement"), by and among The Goldman Sachs Group, Inc., a Delaware corporation,
as depositor (the "Depositor"), and The Bank of New York (Delaware), as trustee
( the "Delaware Trustee"). The Depositor and the Delaware Trustee hereby agree
as follows:

      1. The trust created hereby shall be known as Goldman Sachs Capital IV
(the "Trust"), in which name the Delaware Trustee or the Depositor, to the
extent provided herein, may conduct the business of the Trust, make and execute
contracts, and sue and be sued.

      2. The Depositor hereby assigns, transfers, conveys and sets over to the
Trust the sum of $10. The Delaware Trustee hereby acknowledges receipt of such
amount in trust from the Depositor, which amount constitutes the initial trust
estate. The Delaware Trustee hereby declares that it will hold the trust estate
in trust for the Depositor. It is the intention of the parties hereto that the
Trust created hereby constitute a statutory trust under Chapter 38 of Title 12
of the Delaware Code, 12 Del. C. Section 3801, et seq. (the "Statutory Trust
Act"), and that this document constitute the governing instrument of the Trust.
The Delaware Trustee is hereby authorized and directed to execute and file a
certificate of trust with the Delaware Secretary of State in accordance with the
provisions of the Statutory Trust Act.

      3. The Depositor and the Delaware Trustee will enter into an amended and
restated Trust Agreement or Declaration of Trust satisfactory to each such party
and substantially in the form to be included as an exhibit to the 1933 Act
Registration Statement (as defined below), to provide for the contemplated
operation of the Trust created hereby and the issuance of the Capital or
Preferred Securities and Common Securities referred to therein. Prior to the
execution and delivery of such amended and restated Trust Agreement or
Declaration of Trust, the Delaware Trustee shall not have any duty or obligation
hereunder or with respect of the trust estate, except as otherwise required by
applicable law or as may be necessary to obtain prior to such execution and
delivery any licenses, consents or approvals required by applicable law or
otherwise. Notwithstanding the foregoing, the Delaware Trustee may take all
actions deemed proper as are necessary to effect the transactions contemplated
herein.

      4. The Depositor, as sponsor of the Trust, is hereby authorized, in its
discretion, (i) to prepare and file with the Securities and Exchange Commission
(the "Commission") and to execute, in the case of the 1933 Act Registration
Statement and 1934 Act Registration Statement (as herein defined), on behalf of
the Trust, (a) a Registration Statement (the "1933 Act Registration Statement"),
including all pre-effective and post-effective amendments thereto, relating to
the registration under the Securities Act of 1933, as amended (the "1933 Act"),
of the Capital or Preferred Securities of the Trust, (b) any preliminary
prospectus or prospectus or supplement thereto relating to the Capital or
Preferred Securities of the Trust required to be filed pursuant to the 1933 Act,
and (c) a Registration Statement on Form 8-A or other appropriate form (the
"1934 Act Registration Statement"), including all pre-effective and
post-effective amendments thereto, relating to the registration of the Capital
or Preferred Securities of the Trust under the Securities Exchange Act of 1934,
as amended; (ii) if and at such time as determined by
<PAGE>
the Depositor, to file with the New York Stock Exchange, Inc. or other exchange,
or the National Association of Securities Dealers, Inc. ("NASD"), and execute on
behalf of the Trust a listing application and all other applications,
statements, certificates, agreements and other instruments as shall be necessary
or desirable to cause the Capital or Preferred Securities of the Trust to be
listed on the New York Stock Exchange, Inc. or such other exchange, or the
NASD's Nasdaq National Market; (iii) to file and execute on behalf of the Trust,
such applications, reports, surety bonds, irrevocable consents, appointments of
attorney for service of process and other papers and documents that shall be
necessary or desirable to register the Capital or Preferred Securities of the
Trust under the securities or "Blue Sky" laws of such jurisdictions as the
Depositor, on behalf of the Trust, may deem necessary or desirable; (iv) to
execute and deliver letters or documents to, or instruments for filing with, a
depository relating to the Capital or Preferred Securities of the Trust; (v) to
execute, deliver and perform on behalf of the Trust an underwriting agreement
with one or more underwriters relating to the offering of the Capital or
Preferred Securities of the Trust; and (vi) to take all other actions incident
and reasonably related to the performance of its obligations hereunder which are
not specifically provided for in this Section.

            In the event that any filing referred to in this Section 4 is
required by the rules and regulations of the Commission, the New York Stock
Exchange, Inc. or other exchange, NASD, or state securities or "Blue Sky" laws
to be executed on behalf of the Trust by a trustee, the Delaware Trustee, in its
capacity as trustee of the Trust, is hereby authorized to join in any such
filing and to execute on behalf of the Trust any and all of the foregoing, it
being understood that the Delaware Trustee, in its capacity as trustee of the
Trust, shall not be required to join in any such filing or execute on behalf of
the Trust any such document unless required by the rules and regulations of the
Commission, the New York Stock Exchange, Inc. or other exchange, NASD, or state
securities or "Blue Sky" laws.

      5. This Trust Agreement may be executed in one or more counterparts, each
of which shall be deemed an original but all of which together shall constitute
one and the same instrument.

      6. The number of trustees of the Trust initially shall be one and
thereafter the number of trustees of the Trust shall be such number as shall be
fixed from time to time by a written instrument signed by the Depositor which
may increase or decrease the number of trustees of the Trust; provided, however,
that to the extent required by the Statutory Trust Act, one trustee of the Trust
shall either be a natural person who is a resident of the State of Delaware or,
if not a natural person, an entity which has its principal place of business in
the State of Delaware. Subject to the foregoing, the Depositor is entitled to
appoint or remove without cause any trustee of the Trust at any time. Any
trustee of the Trust may resign upon thirty days' prior notice to the Depositor.

      7. The Depositor agrees: (1) to reimburse the Delaware Trustee upon its
request for all reasonable expenses, disbursements and advances incurred or made
by the Delaware Trustee in accordance with any provision of this Trust Agreement
(including the reasonable compensation and the expenses and disbursements of its
agents and counsel), except any such expense, disbursement or advance as shall
be determined to have been caused by its own negligence or bad faith; and


                                      -2-
<PAGE>
            (2) to indemnify the Delaware Trustee for, and to hold it harmless
against, any and all losses, liabilities, damages, claims or expenses including
taxes (other than taxes imposed on the income of the Delaware Trustee) incurred
without negligence or bad faith on its part, arising out of or in connection
with the acceptance or administration of the trust or trusts hereunder,
including the costs and expenses of defending itself against any claim or
liability in connection with the exercise or performance of any of its powers or
duties hereunder.

      8. The Trust may be dissolved and terminated before the issuance of the
Capital or Preferred Securities at the election of the Depositor.

      9. This Trust Agreement shall be governed by, and construed in accordance
with, the laws of the State of Delaware (without regard to conflict of laws
principles).

      10. If the Trust does not issue Capital Securities within ten years of the
date of this Trust Agreement, this Trust Agreement will terminate.


                                      -3-
<PAGE>
      IN WITNESS WHEREOF, the parties hereto have caused this Trust Agreement to
be duly executed as of the day and year first above written.


                                    THE GOLDMAN SACHS GROUP, INC., as
                                    Depositor

                                    By:   /s/ David Weil
                                          __________________________
                                          Name:  David Weil
                                          Title: Treasurer



                                    THE BANK OF NEW YORK (DELAWARE), as
                                     trustee

                                    By:   /s/ William T. Lewis
                                          __________________________
                                          Name:  William T. Lewis
                                          Title: Sr. Vice President


                                      -4-