<PAGE>   1
                                                                    Exhibit 10.3


                           SECOND AMENDMENT TO AMENDED
                           AND RESTATED LOAN AGREEMENT


                  This Second Amendment to Amended and Restated Loan Agreement
is made as of April 25, 1997 between SLT Realty Limited Partnership
("Borrower"), Starwood Lodging Trust (the "REIT"), CP Hotel Realty Limited
Partnership ("CP"), Midland Building Corporation ("Midland") and Lehman Brothers
Holdings Inc., D/B/A Lehman Capital, a Division of Lehman Brothers Holdings Inc.
("Lender"). All capitalized words and phrases not otherwise defined herein shall
have the meanings set forth in that certain Amended and Restated Loan Agreement
dated as of April 26, 1996, as amended by that certain First Amendment To
Amended and Restated Loan Agreement dated as of August 29, 1996, each between
Borrower, the REIT, CP, Midland and the Lender (the "Loan Agreement").

                              Preliminary Statement

                  The Maturity Date of the Loan is April 26, 1997.

                  Borrower, the REIT, CP, Midland and Lender have agreed to
extend the Maturity Date of the Loan and provide for the payment of certain
extension fees.

                  NOW THEREFORE, in consideration of the mutual covenants and
agreement hereinafter set forth and in and for other good and valuable
consideration, the receipt and sufficiency of which are hereby acknowledged, the
parties hereto consent and agree as follows:

                  1. The Loan Agreement is hereby modified and amended as
follows:

                  The definition of Maturity Date is hereby deleted and the
         following substituted therefor:

                  "Maturity Date" shall mean October 26, 1997 or such earlier
         date on which the principal balance of the Loan and all other sums due
         in connection with the Loan shall be due as a result of the
         acceleration of the Loan.

                  2. Borrower shall have the right to extend the Maturity Date
for one additional six (6) month period by giving Lender written notice to
extend no later than September 26, 1997, but no earlier than August 26, 1997,
and by paying an additional extension fee equal to 0.25% of the then outstanding
principal balance of the Loan (the "Additional Extension Fee"). Upon Lender's
receipt of such notice and the Additional Extension Fee, and provided that no
monetary Default or Event of Default has occurred and is continuing, and
Borrower has complied with the terms and conditions of paragraph 4 of this
Agreement, the Maturity Date shall be extended to April 26, 1998.

                  3. Simultaneously with the execution and delivery of this
Agreement and as a condition precedent to the extension of the Maturity Date,
Borrower shall pay to Lender the sum of $234,900.00 as consideration for the
extension of the Maturity Date.
<PAGE>   2
                  4. Notwithstanding anything to the contrary in Section 5.19 of
the Loan Agreement, Borrower and/or Guarantor shall spend (i) the Minimum
Spending Requirement for each Real Property Asset on or before October 26, 1997,
(ii) the Deferred Maintenance Spending Requirement for each Real Property Asset
other than the Real Property Asset identified as the Holiday Inn-Calverton on
Schedule 1 of the Loan Agreement, on or before October 26, 1997 and (iii) the
Deferred Maintenance Spending Requirement for the Holiday Inn-Calverton on or
before June 1, 1997. Prior to May 26, 1997, Borrower shall deliver evidence
reasonably satisfactory the Lender of the amount of the Minimum Spending
Requirement and Deferred Maintenance Spending Requirement that has been spent to
date, together with an itemization of the work for which such amounts were spent
and a budget showing in reasonable detail the remaining work to be done, the
amounts to be spent and the schedule of expenditures, each of which shall be
reasonably satisfactory the Lender. Upon Lender's receipt and review of the
foregoing evidence, Lender may, in its sole discretion, extend the dates by
which the Minimum Spreading Requirement and the Deferred Maintenance Spending
Requirement must be spent to a date no later than April 26, 1998.

                  5. Borrower, the REIT, CP and Midland shall pay all of
Lender's reasonable out-of-pocket costs and expenses, including reasonable
attorneys' fees and disbursements, in connection with the execution of this
Agreement and any documentation in connection with an additional extension of
the Maturity Date.

                  6. Borrower, the REIT, CP and Midland represent, warrant and
covenant that (i) to the best of their knowledge, no Event of Default has
occurred and is continuing and (ii) there are no offsets, counterclaims or
defenses against the Loan, this Agreement, the Loan Agreement or any of the Loan
Document and that Borrower, the REIT, CP and Midland each has full power,
authority and legal right to execute this Agreement and to keep and observe all
of the terms of this Agreement on their part to be observed or performed.

                  7. Except as expressly modified pursuant to this Agreement,
all of the terms, covenants and provisions of the Loan Agreement and the other
Loan Documents shall continue in full force and effect. In the event of any
conflict or ambiguity between the terms, covenants and provisions of this
Agreement and those of the Loan Agreement and the other Loan Documents, the
terms, covenants and provisions of this Agreement shall control.

                  8. This Agreement may not be modified, amended, waived,
changed or terminated orally, but only by an agreement in writing signed by the
party against whom the enforcement of the modification, amendment, waiver,
change or termination is sought.

                  9. This Agreement shall be binding upon and inure to the
benefit of Borrower, the REIT, CP, Midland, Lender, all future holders of the
Note and their respective successors and assigns.

                  10. This Agreement may be executed in any number of duplicate
originals and each such duplicate original shall be deemed to constitute but one
and the same instrument.
<PAGE>   3
                  11. If any term, covenant or condition of this Agreement shall
be held to be invalid, illegal or unenforceable in any respect, this Agreement
shall be construed without such provision.

                  12. This Agreement shall be governed by and construed in
accordance with the laws of the State of New York and the applicable laws of the
United States of America.


                         [NO FURTHER TEXT ON THIS PAGE]
<PAGE>   4
                  IN WITNESS WHEREOF, the parties hereto have executed and
delivered this Agreement as of the date first above written.

                                  SLT REALTY LIMITED PARTNERSHIP

                                  By:    Starwood Lodging Trust, its general
                                         partner

                                         BY:/s/ Ronald C. Brown
                                            ------------------------------------
                                           Name: Ronald C. Brown
                                           Title: Senior Vice President

                                  STARWOOD LODGING TRUST

                                  By:/s/ Ronald C. Brown
                                     ------------------------------------------
                                     Name: Ronald C. Brown
                                     Title:  Senior Vice President

                                 CP HOTEL REALTY LIMITED PARTNERSHIP

                                 By:    SLT Realty Limited Partnership, its
                                        general partner

                                        By: Starwood Lodging Trust, its
                                            general partner

                                            By:/s/ Ronald C. Brown
                                               --------------------------------
                                               Name: Ronald C. Brown
                                               Title: Senior Vice President

                                  MIDLAND BUILDING CORPORATION


                                  By:
                                     ------------------------------------------
                                     Name: Nir E. Margalit
                                     Title: Secretary
<PAGE>   5
                  IN WITNESS WHEREOF, the parties hereto have executed and
delivered this Agreement as of the date first above written.

                                  SLT REALTY LIMITED PARTNERSHIP

                                  By:    Starwood Lodging Trust, its general
                                         partner

                                         By:
                                            ------------------------------------
                                           Name: Ronald C. Brown
                                           Title: Senior Vice President

                                  STARWOOD LODGING TRUST

                                  By:
                                     ------------------------------------------
                                     Name: Ronald C. Brown
                                     Title:  Senior Vice President

                                 CP HOTEL REALTY LIMITED PARTNERSHIP

                                 By:    SLT Realty Limited Partnership, its
                                        general partner

                                        By: Starwood Lodging Trust, its
                                            general partner

                                            By:
                                               --------------------------------
                                               Name: Ronald C. Brown
                                               Title: Senior Vice President

                                  MIDLAND BUILDING CORPORATION


                                  By: /s/ Nir E. Margalit
                                     ------------------------------------------
                                     Name: Nir E. Margalit
                                     Title: Secretary
<PAGE>   6
                                  LEHMAN BROTHERS HOLDINGS INC.  
                                  D/B/A LEHMAN CAPITAL, A DIVISION OF LEHMAN 
                                  BROTHERS HOLDINGS INC., a Delaware corporation


                                  By:/s/ Francis X. Gilhool
                                     ------------------------------------------
                                     Name: Francis X. Gilhool
                                     Title:  SVP
<PAGE>   7
                  IN WITNESS WHEREOF, the parties hereto have executed and
delivered this Agreement as of the date first above written.

                                  SLT REALTY LIMITED PARTNERSHIP

                                  By:    Starwood Lodging Trust, its general
                                         Partner

                                         By:
                                            ------------------------------------
                                            Name: Ronald C. Brown
                                            Title: Senior Vice President

                                  STARWOOD LODGING TRUST

                                  By:
                                     ------------------------------------------
                                     Name: Ronald C. Brown
                                     Title: Senior Vice President

                                 CP HOTEL REALTY LIMITED PARTNERSHIP

                                 By:    SLT Realty Limited Partnership, its
                                        general partner

                                        By: Starwood Lodging Trust, its
                                            general partner

                                            By:
                                               --------------------------------
                                               Name: Ronald C. Brown
                                               Title: Senior Vice President

                                  MIDLAND BUILDING CORPORATION


                                  By:/s/ Nir E. Margalit
                                     ------------------------------------------
                                     Name: Nir E. Margalit
                                     Title: Secretary
<PAGE>   8
                                  LEHMAN BROTHERS HOLDINGS INC.  
                                  D/B/A LEHMAN CAPITAL, A DIVISION OF LEHMAN 
                                  BROTHERS HOLDINGS INC., a Delaware corporation


                                  By:/s/ Francis X. Gilhool
                                     ------------------------------------------
                                     Name: Francis X. Gilhool
                                     Title:  SVP