<PAGE>   1

                                                                  EXHIBIT 10.17



                       COLLATERAL SUBSTITUTION AGREEMENT

                 THIS COLLATERAL SUBSTITUTION AGREEMENT (this "Agreement"),
dated as January 4, 1996, by and among Lehman Commercial Paper Inc. ("Lehman"),
SLT Realty Limited Partnership ("SLTLP"), and SLT Realty Company, L.L.C.
("SLTLLC," together with SLTLP, the "Customers").

                              W I T N E S S E T H:

                 WHEREAS, Lehman has extended to the Customers that certain
mortgage loan funding facility secured by certain mortgage loans and a
participation interest ("Assets") pursuant to that certain Mortgage Loan
Funding Facility, dated as of July 25, 1995, between the Customers and Lehman,
as amended by that certain Amendment No. 1 to the Mortgage Loan Funding
Facility, dated as of October 31, 1995, between the Customers and Lehman (the
"Facility Agreement") and that certain Pledge Agreement, dated as of July 25,
1995, between the Customers and Lehman, as amended by that certain Amendment
No. 1 to the Pledge Agreement, dated as of October 31, 1995, between the
Customers and Lehman (the "Pledge Agreement");

                 WHEREAS, in connection with the Facility Agreement and the
Pledge Agreement, the Customers, Lehman and State Street Bank and Trust Company
of California, N.A. (the "Custodian") entered into that certain Mortgage Loan
Funding Program Tri-Party Custody Agreement, dated July 25, 1995, among the
Customers, Lehman and the Custodian, as amended by that certain Amendment No. 1
to Tri-Party Custody Agreement, dated as of October 31, 1995, among the
Customers, Lehman and the Custodian, as further amended on the date hereof by
that certain Amendment No. 2 to Tri-Party Custody Agreement, dated as of the
date hereof, among the Customers, Lehman and the Custodian (the "Custody
Agreement") pursuant to which the Custodian is holding certain documents in
custody for the benefit of Lehman;

                 WHEREAS, all Advances (as defined in the Pledge Agreement)
made by Lehman pursuant to the Facility Agreement are evidenced by that certain
Promissory Note, dated July 25, 1995, in the original principal amount of
$44,600,000, as increased, amended and restated by that certain Amended and
Restated Promissory Note, dated as of October 31, 1995, in the amount of
$71,000,000 (as amended and restated, the "Promissory Note")

                 WHEREAS, on the date hereof the Customers intend to
restructure the ownership and debt structure on the Mortgaged Property commonly
known as The Grand Hotel in Washington D.C. (the "Grand Hotel"), by having
SLTLP purchase the Grand Hotel and release the mortgages presently encumbering
the Grand Hotel which have been previously assigned to Lehman (the "Old
Mortgages").  SLTLP shall then mortgage the Grand Hotel to Lehman to serve as
security for all present and future Advances made by Lehman to the Customers
evidenced by the Promissory Note up to an amount of $29,360,000 (the "New
Mortgage").  The New Mortgage is also referred to herein as the "Deed of
Trust");
<PAGE>   2

                 WHEREAS, in connection with the acquisition of the Grand Hotel
by SLTLP, SLC Operating Limited Partnership ("SLC"), an affiliate of the
Customers, shall purchase the furniture, fixtures and equipment relating to the
Grand Hotel and operate and manage the Grand Hotel on behalf of SLTLP pursuant
to an operating lease agreement, dated December 28, 1995 (including all
amendments, modifications, supplements, restatements and renewals thereto, the
"Operating Agreement").  In connection with the purchase of the furniture,
fixtures and equipment of the Grand Hotel by SLC, SLC executed and delivered to
SLTLP a secured demand promissory note, dated January 4, 1996 in the amount of
$3,640,000 (the "SLC Note") and a leasehold deed of trust (the "Leasehold Deed
of Trust") to secure the SLC Note, and SLTLP shall pledge to Lehman the
Operating Agreement, Leasehold Deed of Trust and SLC Note as additional
security for all present and future Advances made by Lehman to the Customers
evidenced by the Promissory Note up to an amount of $3,640,000; and

                 WHEREAS, Lehman and the Customers wish to substitute the New
Mortgage and the pledge of the Operating Agreement, the Leasehold Deed of Trust
and SLC Note for the Old Mortgages and have the Customers restate the
representations and warranties contained in the Facility Agreement and the
Pledge Agreement.

                 NOW, THEREFORE, in consideration of the sum of Ten Dollars
($10.00) and other good and valuable consideration the receipt and sufficiency
of which are hereby acknowledged, the Customers and Lehman agree as follows:

                 1.       All capitalized terms not otherwise defined herein
shall have the meaning set forth in the Pledge Agreement.

                 2.       This Agreement shall be deemed an amendment to the
Facility Agreement and the Pledge Agreement, as applicable, in accordance with
the terms and provisions set forth herein.

                 3.       The Customers and Lehman acknowledge and agree that
the documents listed on the "Checklist for Custodian (The Grand Hotel -
Washington, D.C.)" attached to the Initial Certification from the Custodian,
dated October 27, 1995 (the "Old Grand Hotel Documents") shall no longer serve
as security for any Advance made by Lehman to the Customers which was secured
in part by the Grand Hotel and evidenced by the Promissory Note (the "Grand
Hotel Advance").

                 4.       The Customers and Lehman acknowledge and agree that
the New Mortgage, from SLTLP for the benefit of Lehman, dated as of the date
hereof, the Leasehold Deed of Trust, the Operating Agreement and the SLC Note
shall serve as security for the Grand Hotel Advance and replace the Old 
Mortgages and the Old Grand Hotel Documents as security for the Grand Hotel 
Advance.  The Customers and Lehman also acknowledge and agree that in 
addition to the New Mortgage, the Operating Agreement, the Leasehold Deed 
of Trust and the SLC Note, SLTLP has executed with Lehman an Assignment 
of Leases and Rents, UCC-1 financing statements, and such other documents 
as listed on Exhibit A attached hereto as Lehman deemed necessary as 
replacement security for the Old Mortgages and the Old Grand Hotel
Documents and SLTLP has obtained a Mortgagee's title insurance policy in





                                      -2-
<PAGE>   3

an amount equal to the amount of the New Mortgage (the documents referred to in
this Paragraph 4 are hereinafter collectively referred to as, the "New Grand
Hotel Documents").

                 5.       Notwithstanding anything to the contrary set forth in
the Pledge Agreement, on the date hereof Lehman has instructed the Custodian to
release to the Customers the Old Grand Hotel Documents and Lehman, the
Customers and the Custodian have amended the Custody Agreement to provide for
the New Grand Hotel Documents to be held by the Custodian under the Custody
Agreement.

                 6.       The Facility Agreement is hereby amended by adding
the following sentence immediately following the first sentence of Section 1 of
the Facility Agreement:

                 "The aggregate principal amount of Advances with respect to
the Grand Hotel shall not exceed $12,750,000 at any one time."

                 7.       The Facility Agreement is hereby amended by adding
the following paragraph immediately following Section 6(b)(vi) of the Facility
Agreement:

                 "(vii)   In the event that at the time of any Advance, the
outstanding Advances are less than $33,000,000, then, if required by Lehman in
its sole discretion, Lehman shall have received an endorsement to the title
policy obtained in connection with the New Mortgage and the Leasehold Deed of
Trust on the Grand Hotel that, as of the date of the Advance, there are no
Liens on the Grand Hotel other than the Liens existing on the date hereof and
listed on the respective title policy issued on the date hereof.
Notwithstanding the foregoing, in the event that at the time of any Advance,
the outstanding Advances are less than $33,000,000, Lehman may elect, in its
sole discretion, in lieu of the foregoing, to perform searches of title (a
"Title Search") to the Grand Hotel on a quarterly basis at Customers' sole cost
and expense, provided that at the time of each Title Search the amount of
outstanding Advances are less than $33,000,000.  The results of all such Title
Searches shall be satisfactory to Lehman in its reasonable discretion and all
Title Searches shall be conducted by search firms designed by Lehman."

                 8.       For purposes of the definition of "Collateral
Value" and "Market Value" set forth in the Pledge Agreement, the term
"Collateral" as used in such definitions shall apply only to Mortgage Loans and
Participation Interest.

                 9.       The Pledge Agreement is hereby amended by deleting
the definition of "Facility Documents" in its entirety and substituting the
following definition in lieu thereof:

                 ""Facility Documents" means the Facility Agreement; this
Pledge Agreement; the Promissory Note; the Guaranty; the Reaffirmation of
Guaranty dated as of October 31, 1995 by the Guarantor in favor of Lehman; the
Reaffirmation of Guaranty dated as of January 4, 1996 by the Guarantor in favor
of Lehman; the Letter Agreement; the Custody Agreement; the Post-Closing
Agreement; the Post-Closing Agreement dated as of October 31, 1995 among Lehman
and the Customers; the Post-Closing Agreement dated as of January 4, 1996 among
Lehman and the Customers; the Collateral Substitution Agreement; Deed of Trust
and Security





                                      -3-
<PAGE>   4
Agreement from SLTLP for the benefit of Lehman dated January 4, 1996; Security
Agreement dated January 4, 1996 by and between SLTLP and Lehman; Leasehold Deed
of Trust and Security Agreement from SLC Partnership to SLTLP dated January 4,
1996; Assignment of Leases and Rents from SLTLP for the benefit of Lehman dated
January 4, 1996; Lease Agreement dated as of December 28, 1995 between SLTLP
and SLC Partnership; Assignment of Franchise Agreements, Management Agreements,
Agreements, Permits and Contracts dated January 4, 1996 by and between SLTLP
and Lehman; Environmental Indemnity Agreement dated as of January 4, 1996 by
SLTLP and the Guarantor to Lehman; Consent to Assignment of Rents,
Subordination, Estoppel and Attornment Agreement dated January 4, 1996 by and
between SLC Partnership and Lehman; Assignment of Leases and Rents from SLC
Partnership to SLTLP dated January 4, 1996; Secured Demand Promissory Note
dated January 4, 1996 in the amount of $3,640,000 from SLC Partnership, as
borrower, to SLTLP, as lender; Security Agreement dated January 4, 1996 between
SLC Partnership and SLTLP; Assignment of Franchise Agreements, Management
Agreements, Agreements, Permits and Contracts dated January 4, 1996 by and
between SLC Partnership and SLTLP; UCC financing statements; title insurance
policy with respect to the Grand Hotel in an amount equal to the amount of the
New Mortgage."

                 10.      The Pledge Agreement is hereby amended by adding the
following paragraph to Section 2.1 of the Pledge Agreement immediately
following the end of Section 2.1 thereof:

                 ""Collateral" shall also mean all right, title and interest of
Customers in and to the mortgaged property commonly known as The Grand Hotel in
Washington, D.C., as further described in the New Mortgage (the "Grand Hotel"),
the New Grand Hotel Documents and all of the Customers' right, title and
interest in, under and to the items listed in Section 2.1 (a)-(f) related
thereto."

                 11.     The Pledge Agreement is hereby amended by deleting 
Section 2.2 of the Pledge Agreement in its entirety and substituting the
following paragraph in lieu thereof:


                 "2.2     Release of Collateral.  Upon the full satisfaction of
all outstanding principal, accrued interest on, and all other Obligations owing
with respect to any Advance and so long as no Event of Default has occurred and
is continuing, but subject to the rights of any holder of a lien on the items
of Collateral of which Lehman has notice, Lehman shall, and shall direct
Custodian to, release the Collateral related to such Advance.  Upon receipt of
a written request from the Customers, and provided no Event of Default has
occurred and is continuing Lehman shall direct the Custodian to release
Mortgage Loans, Participation Interest and/or the New Grand Hotel Documents, as
the case may be, identified in such request prior to the Facility Termination
Date or Reset Date of an Advance provided that (a) upon release of such
Collateral, the aggregate outstanding Advances shall not exceed the sum of (i)
85% of Collateral Value with respect to the Mortgage Loans and Participation
Interest and (ii), with respect to the Grand Hotel, 50% of the most recent
appraised value of the Grand Hotel according to Lehman's internal valuation
most recently delivered or prepared (the "Loan-to-Value Ratio") (the amount
calculated in Section 2.2 (a)(ii) is hereinafter referred to as the





                                      -4-
<PAGE>   5
"Grand Hotel Advance Limit") and (b) the sum of 85% of the aggregate Collateral
Value and the Grand Hotel Advance Limit remaining is at least $10,000,000
(unless the Facility is paid in full).  So long as an Event of Default has
occurred and is continuing, neither Lehman nor the Custodian shall be required
to release any Collateral unless and until either, (i) the entire outstanding
balance of the related Mortgage Loan or Participation Interest, or the
aggregate principal amount of the SLC Note and the New Mortgage on the Grand
Hotel, as the case may be, requested to be released is paid in full and the
requirements of Section 2.2(a) and (b) continue to be met notwithstanding the
foregoing or (ii) all outstanding principal, accrued interest on, and all other
Obligations owing with respect to any outstanding Advances has been fully
satisfied.  Notwithstanding anything to the contrary set forth herein, Lehman
shall not, and shall not direct the Custodian to, release the SLC Note until
the Customers have fully complied with the provisions of this Section 2.2 and
the terms of Sections 3 and 4 of the Facility Agreement."

                 12.      The Pledge Agreement is hereby amended by adding the
following paragraph at the end of Section 4.12(c) of the Pledge Agreement :

                 "With respect to the Grand Hotel, SLTLP shall also provide
Lehman with quarterly and annual operating statements for the Grand Hotel,
separately disclosing the amounts paid under the Operating Agreement with
respect to the Grand Hotel and including a comparison and reconciliation with
the most recent Annual Operating Budget (as defined below), within 45 days
following the end of each calendar quarter, certified by SLTLP."

                 13.      The Pledge Agreement is hereby amended by adding the
following section to Section 4.12 of the Pledge Agreement immediately following
Section 4.12(e) thereof:

                 "(f)     an annual operating and capital budget for the Grand
Hotel (the "Annual Operating Budget"), including cash flow projections for the
upcoming year, presented on a monthly basis consistent with the quarterly and
annual operating statement referred to in clause (c) above at least 15 days
prior to the start of each calendar year."

                 14.      The Pledge Agreement is hereby amended by deleting
the first sentence of the first paragraph of Section 4.19 of the Pledge
Agreement in its entirety and substituting the following sentence in lieu
thereof:

                 "At any time any Advance is made or shall be outstanding,
Customers shall at all times maintain the aggregate amount of all Advances in
an amount not to exceed the sum of (i) 85% of Collateral Value with respect to
the Mortgage Loans and Participation Interest and (ii), with respect to the
Grand Hotel, the Grand Hotel Advance Limit (such amount, the "Advance
Threshold")."

                 15.      The Pledge Agreement is hereby amended by deleting
the third sentence of the first paragraph of Section 4.19 of the Pledge
Agreement in its entirety and substituting the following sentence in lieu
thereof:





                                      -5-
<PAGE>   6
                 "In the event the outstanding Advances exceed the Advance
Threshold, the Customers shall cure such excess within five (5) Business Days
from receipt of written notice by Customers from Lehman by delivering to Lehman
additional Collateral acceptable to Lehman or cash."

                 16.      The Pledge Agreement is hereby amended by adding the
following section to Article 4 of the Pledge Agreement

                 "4.20    Replacement Reserve.  The Customers shall spend, or
cause to have spent, a minimum spending amount (the "Minimum Spending
Requirement") for the Grand Hotel equal to 4% of the annual Gross Revenues (as
defined below) from the Grand Hotel per annum on repair, replacement and
maintenance of the Grand Hotel during the applicable calendar year, which
Minimum Spending Requirement for calendar year 1996 shall be expended to
complete the work recommended by Oldham & Seltz in the Architecture and
Engineering Report, dated as of December 1, 1995, prepared by Oldham & Seltz, a
copy of which is attached hereto as Exhibit B.  The Minimum Spending
Requirement for each calendar year shall be determined annually by Lehman, and
shall be based on Gross Revenues for the immediately preceding calendar year.
In the event the Customers have spent more than the Minimum Spending
Requirement in any calendar year, Customers may apply such excess, up to an
amount equal to 2% of the annual Gross Revenues from the Grand Hotel to reduce
the amount of the Minimum Spending Requirement that must be spent in the
immediately succeeding calendar year.  The Customers shall deliver evidence
reasonably satisfactory to Lehman no more frequently than monthly, but at least
quarterly, of the portion of the Minimum Spending Requirement that has been
spent; such evidence shall include copies of paid invoices or other receipts
for work done or materials provided and such other information as reasonably
requested by Lehman.  For purposes of this Section 4.20, Gross Revenues shall
mean all income, rents, room rates, additional rents, revenues, issues and
profits and other items including without limitation, all revenues and credit
card receipts collected from guest rooms, restaurants, meeting rooms, bars,
mini-bars, banquet rooms, recreation facilities, vending machines and
concessions derived from the customary operation of the Grand Hotel."

                 17.      As of the date hereof, the Customers hereby confirm,
ratify and restate in their entirety the representations and warranties
contained in Section 5 of the Facility Agreement and in Article 3 of the Pledge
Agreement.

                 18.      The Customers acknowledge and agree that the Facility
Documents are the valid and binding obligation of the Customers, enforceable in
accordance with their respective terms and the Customers agree to be bound by
all covenants and agreements contained in the Facility Documents, as modified
or amended hereby.

                 19.      The Customers acknowledge and agree that a default
hereunder shall constitute an Event of Default (as defined in the Promissory
Note) under the Facility Documents.

                 20.      The Customers acknowledge and agree that an Event of
Default under the Facility Documents shall constitute a default hereunder.





                                      -6-
<PAGE>   7
                 21.      All other terms, conditions, and provisions of the
Facility Agreement and the Pledge Agreement not modified or amended herein are
hereby affirmed and shall remain in full force and effect as written.

                 22.      THIS AGREEMENT SHALL BE CONSTRUED IN ACCORDANCE WITH
THE LAWS OF THE STATE OF NEW YORK, AND THE OBLIGATIONS, RIGHTS AND REMEDIES OF
THE PARTIES HEREUNDER SHALL BE DETERMINED IN ACCORDANCE WITH SUCH LAWS.

                 23.      This Agreement may be executed in one or more
counterparts and by different parties hereto on separate counterparts, each of
which, when so executed, shall constitute one and the same agreement.

                 24.      This Agreement shall inure to the benefit of and be
binding upon Lehman and the Customers, and their respective successors and
permitted assigns under the Facility Agreement and the Pledge Agreement.

                 25.      No modification, amendment, extension, discharge,
termination or waiver of any provision of this Agreement or the New Grand Hotel
Documents, nor consent to any departure by the Customers therefrom, shall in
any event be effective unless the same shall be in a writing signed by the
party against whom enforcement is sought, and then such waiver or consent shall
be effective only in the specific instance, and for the purpose, for which
given.  Except as otherwise expressly provided herein, no notice to, or demand
on the Customers, shall entitle the Customers to any other or future notice or
demand in the same, similar or other circumstances.

                 26.      Neither any failure nor any delay on the part of
Lehman in insisting upon strict performance of any term, condition, covenant or
agreement, or exercising any right, power, remedy or privilege hereunder, or
under the New Grand Hotel Documents, shall operate as or constitute a waiver
thereof, nor shall a single or partial exercise thereof preclude any other
future exercise, or the exercise of any other right, power, remedy or
privilege.

                 27.      SLTLP and SLTLLC shall be jointly and severally
liable for the obligations of the Customers hereunder.

                 28.      The Guarantor's name of "Starwood Lodging Trust" is a
designation of Starwood Lodging Trust and its trustees (as trustees but not
personally) under a Declaration of Trust dated August 15, 1969, as amended and
restated as of June 6, 1988, as further amended on February 1, 1995, as further
amended on June 19, 1995 and as the same may be further amended from time to
time, and all persons dealing with the Guarantor shall look solely to the
Guarantor's assets for the enforcement of any claims against the Guarantor, as
the trustees, officers, agents and security holders of the Guarantor assume no
personal liability for obligations entered into on behalf of the Guarantor, and
their respective individual assets shall not be subject to the claims of any
person relating to such obligations.  The foregoing shall govern all direct and
indirect obligations of the Guarantor under this Agreement.

                    [Signatures commence on following page]





                                      -7-
<PAGE>   8
                 IN WITNESS WHEREOF, the parties have caused their names to be
signed hereto by their respective officers thereunto duly authorized as of the
day and year first above written.

                             LEHMAN COMMERCIAL PAPER INC.

                             By:________________________________________________

                             Name:______________________________________________

                             Title:_____________________________________________



                             Customers:



                             SLT REALTY LIMITED PARTNERSHIP

                             By:     STARWOOD LODGING TRUST
                                     its general partner

                             By:     ___________________________________________

                                     Name: _____________________________________
                                                       
                                     Title: ____________________________________



                             SLT REALTY COMPANY, L.L.C.

                             By:     SLT REALTY LIMITED PARTNERSHIP
                                     its 99% member

                             By:     STARWOOD LODGING TRUST
                                     its general partner

                             By:     ___________________________________________

                                     Name: _____________________________________
                                                       
                                     Title: ____________________________________

                             Telephone #:     (310) 575-3900

                             Facsimile #:     (310) 575-9143





                                      -8-
<PAGE>   9
                                   EXHIBIT A

Deed of Trust and Security Agreement, from SLT Realty Limited Partnership for
the benefit of Lehman Commercial Paper Inc., dated January 4, 1996

Security Agreement, dated January 4, 1996, by and between SLT Realty Limited
Partnership and Lehman Commercial Paper Inc.

Assignment of Leases and Rents, from SLT Realty Limited Partnership for the
benefit of Lehman Commercial Paper Inc.

Assignment of Franchise Agreements, Management Agreements, Agreements, Permits
and Contracts, dated January 4, 1996, by and between SLT Realty Limited
Partnership and Lehman Commercial Paper Inc.

Environmental Indemnity Agreement, dated as of January 4, 1996, by SLT Realty
Limited Partnership and Starwood Lodging Trust to Lehman Commercial Paper Inc.

Consent to Assignment of Rents, Subordination, Estoppel and Attornment
Agreement, dated January 4, 1996, by and between SLC Operating Limited
Partnership and Lehman Commercial Paper Inc.

Leasehold Deed of Trust and Security Agreement, from SLC Operating Limited
Partnership to SLT Realty Limited Partnership, dated January 4, 1996

Assignment of Leases and Rents, from SLC Operating Limited Partnership to SLT
Realty Limited Partnership

Secured Demand Promissory Note, dated January 4, 1996, in the amount of
$3,640,000.

Security Agreement, dated January 4, 1996, between SLC Operating Limited
Partnership and SLT Realty Limited Partnership

Assignment of Franchise Agreements, Management Agreements, Agreements, Permits
and Contracts, dated January 4, 1996, by and between SLC Operating Limited
Partnership and SLT Realty Limited Partnership

Lease Agreement, dated as of December 28, 1995, between SLT Realty Limited
Partnership and SLC Operating Limited Partnership

UCC financing statements

Title insurance policy





                                      -9-
<PAGE>   10
                                   EXHIBIT B

 Architecture and Engineering Report Dated December 1, 1995 prepared by Oldham &
 Seltz





                                      -10-