<PAGE>

                                                                    Exhibit 10.4

                              [Textron Letterhead]

November 17, 2003

Silverleaf Resorts, Inc.
1221 River Bend Drive, Suite 105
Dallas, Texas  75221
Attention:Robert E. Mead, Chief Executive Officer

Re:      $72,000,000 credit facility [Tranche A] (the "Loan") provided to
         Silverleaf Resorts, Inc. ("Borrower") pursuant to that certain Amended
         and Restated Loan, Security and Agency Agreement dated as of April 30,
         2002, by and among Silverleaf Resorts, Inc., the parties, including
         Textron Financial Corporation, a Delaware Corporation, which executed
         and delivered the Agreement, in their respective capacities as lenders,
         and Textron Financial Corporation, as Facility Agent and Collateral
         Agent ("Agent"), as amended by those certain letter agreements dated
         March 27, 2003 and September 25, 2003 by and among Textron Financial
         Corporation ("TFC") and Silverleaf Resorts, Inc., as the same may
         hereafter be amended from time to time (the "Loan Agreement")

Gentlemen:

Reference is hereby made to the Loan. All capitalized terms used herein and not
otherwise defined shall have the meaning ascribed to such terms in the Loan
Agreement.

Borrower has advised Agent of the occurrence of certain events that may
constitute a Default or an Event of Default under the Loan Agreement. These
Events of Default arise from: (i) a $28,711,000 increase in Borrower's allowance
for uncollectible notes reflected in Borrower's 10-Q for the period ended March
31, 2003, which caused Borrower to violate the Interest Coverage and Profitable
Operations covenants set forth in Sections 7.1(cc)(iv) and (v) of the Loan
Agreement; and (ii) the retirement of certain subordinated debt pursuant to that
certain Note Purchase and Release Agreement with eight entities and individuals
affiliated with Harry S. Patten and James E. Lambert, which was entered into
without first obtaining the written consent of Lender and the other senior
lenders in violation of Section 7.2(f) of the Loan Agreement.

Subject to the terms of this letter agreement, Agent, on behalf of the Lenders,
hereby waives any Default or Event of Default under the Loan Documents that may
have occurred or may occur hereafter as a result of the condition described in
item (i) above and/or that may have occurred as a result of the condition
described in item (ii) above on or before the date hereof (each such waived
Default and/or Event of Default a "Waived Default" and, collectively, the
"Waived Defaults").

<PAGE>

Borrower hereby acknowledges that, by its execution of this letter below, Agent
is not waiving, and that Agent has not waived: (i) any Default or any Event of
Default (other than the Waived Defaults), that may occur hereafter by reason of
any facts which exist or existed on or before the date hereof, whether or not
Agent or any Lender has notice or knowledge thereof and whether or not Agent or
any Lender has given notice thereof to Borrower or any of its Affiliates, (ii)
any right or remedy of Agent or any Lender under any Loan Document, at law or in
equity against Borrower, or (iii) any right or remedy contained in any of the
Loan Documents, at law or in equity, other than in respect of the Waived
Defaults.

No Default or Event of Default relating to the Marketing and Sales Expense
covenant as set forth in Section 7.1(cc) (ii) of the Loan Agreement has been
waived as there has been no Event of Default relating to that covenant. By
Letter Agreement dated March 27, 2003 (individually, the "March Letter
Agreement" and collectively, along with the letter agreement executed by TFC in
favor of Silverleaf on September 25, 2003, the "Letter Agreements"), TFC amended
its Loan Agreement to increase the permissible ratio of Marketing and Sales
Expenses to the Borrower's net proceeds to .550 to 1 (a ratio that has not been
exceeded by Borrower). Although the March Letter Agreement was conditioned upon
the execution of similar agreements relating to the Heller and Sovereign Bank
facilities, Borrower, by its acknowledgment of and agreement to this letter
below, has confirmed that those agreements have been obtained and that all of
the conditions of the March Letter Agreement have been satisfied. In this
connection, please note that notwithstanding the fact that the Letter Agreements
were executed by TFC without reference to the capacity in which it was executing
those documents, TFC executed both (i) the March 27, 2003 letter agreement with
the Borrower and (ii) the September 25, 2003 letter agreement with the Borrower,
in its capacity as Agent for each of the Lenders (and not in its individual
capacity) and with the requisite authority under the Loan Agreement.

Borrower hereby releases Agent and the Lender, from liability for any claim,
liability, judgment, obligation, loss, cost, damage or expense arising from, or
relating to any action taken by Agent, any Lender or any of their respective
Affiliates and their respective officers, directors, employees, attorneys and
agents in connection with the Loan.

Furthermore, subject to the terms and conditions contained herein, Agent on
behalf of the Lenders hereby consents to: (i) that certain First Amendment to
Amended and Restated Revolving Credit Agreement dated as of September 30, 2002
by and among Silverleaf Resorts, Inc., Sovereign Bank and Liberty Bank (the
"Sovereign Amendment #1) a copy of which is attached hereto as Exhibit A; (ii)
that certain Second Amendment to Amended and Restated Revolving Credit Agreement
dated as of _________ _____, 2003 by and among Silverleaf Resorts, Inc.,
Sovereign Bank and Liberty Bank (the "Sovereign Amendment #2"), a copy of which
is attached hereto as Exhibit B; (iii) that certain First Amendment to Amended
and Restated Receivables Loan and Security Agreement dated as of August 6, 2002
by and among Silverleaf Finance I, Inc., as Borrower, Silverleaf Resorts, Inc.,
as initial Servicer, Autobahn Funding Company LLC, as Lender, DZ Bank AG
Deutsche Zentral-Genossenschaftsban, Frankfurt AM, MAIN, as Agent for the
Lender, U.S. Bank National Association, as Agent's Bank, and Wells Fargo Bank
Minnesota, National Association, as Backup Servicer (the "DZ Amendment #1"), a
copy of which is attached hereto as Exhibit C; (iv) that certain Second
Amendment and Waiver Agreement dated as of June 19, 2003 by and among Silverleaf
Finance I, Silverleaf Resorts, Inc., as Servicer, Autobahn Funding Company LLC,
as lender, DZ Bank
<PAGE>

AG Deutsche Zentral-Genossenschaftsbank, Frankfurt Am Main, as Agent, U.S. Bank
Trust National Association, as Agent's bank, and Wells Fargo Bank Minnesota,
National Association, as Backup Servicer (the "DZ Amendment #2"), a copy of
which is attached hereto as Exhibit D; (v) that certain Fifth Amendment to
Second Amended and Restated Inventory Loan and Security Agreement and
Modification of Notes dated as of November 21, 2003 by and among Silverleaf
Resorts and Heller Financial, Inc., as lender (the "Heller Amendment #1"), a
copy of which is attached hereto as Exhibit E; and (vi) that certain Third
Amendment to Amended and Restated Receivables Loan and Security Agreement dated
as of November 21, 2003 by and among Silverleaf Resorts and Heller Financial,
Inc., as Lender (the "Heller Amendment #2), a copy of which is attached hereto
as Exhibit F.

In consideration of the waivers set forth herein and in consideration of
Textron's consent to Sovereign Amendment #1, Sovereign Amendment #2, DZ
Amendment #1, DZ Amendment #2, Heller Amendment #1 and Heller Amendment #2 as
set forth herein, Borrower hereby agrees to pay to TFC, individually and in its
capacity as Agent for each of the Lenders under the Loan Agreement and the other
Textron credit facilities, the sum of $50,000.00 (the "Waiver Fee"). This Waiver
Fee shall be payable in consideration not only of the waivers and consents set
forth herein but also in consideration of the same waivers and consents being
granted under the other existing Textron credit facilities (collectively the
"Textron Waivers and Consents"), it being understood that the total amount
Borrower shall be required to pay for the Textron Waivers and Consents shall be
the sum of $50,000.00.

This letter is conditioned upon and shall not be effective unless and until: (i)
similar Agreements are executed relating to the Heller, Sovereign Bank, DZ and
other Textron credit facilities; (ii) payment by the Borrower of the Waiver Fee
is received by Agent in immediately available funds; and (iii) Agent has
received evidence, in form and substance satisfactory to Agent, that the consent
of each party entitled to consent to this letter pursuant to the terms of the
Sovereign Documents, the Heller Documents and any other document evidencing any
other Indebtedness of the Borrower has been obtained.

This letter may be executed in two or more counterparts, all of which together
shall be considered a single instrument. Delivery of an executed counterpart of
a signature page to this letter by facsimile shall be effective as delivery of a
manually executed counterpart of this letter.

Except as expressly set forth herein, this letter does not constitute an
amendment or waiver of any term or condition of the Loan, the Loan Agreement or
any of the Loan Documents, and all such terms and conditions shall remain in
full force and effect. Furthermore, this letter shall not be construed as
establishing any precedent for any future request for a waiver or amendment of
any of the terms or conditions of the Loan, the Loan Agreement or any of the
Loan Documents, which Agent and the Lenders hereby reserve the right to enforce
strictly.

This letter is also entered into with the understanding and upon the condition
that, except for the Waived Defaults, Borrower is not in default under any of
its credit facilities.

<PAGE>

Please confirm your acknowledgement of and agreement with the terms of this
letter by signing in the appropriate space below.

                                              Very truly yours,
                                              TEXTRON FINANCIAL CORPORATION,
                                              a Delaware corporation

                                              As Agent on behalf of itself as
                                              Lender and on behalf of each other
                                              Lender under the Loan Agreement

                                                   /S/ JOHN T. D'ANNIBALE
                                              ----------------------------------
                                              By:  John T. D'Annibale
                                              Its: V.P.

The undersigned party acknowledges its agreement with the terms and conditions
of this letter:

                                              SILVERLEAF RESORTS, INC.

                                                   /S/ ROBERT E. MEAD
                                              ----------------------------------
                                              By:
                                              Its:

Exhibits:

Exhibit A:  Sovereign Amendment
Exhibit B:  Sovereign Amendment #2
Exhibit C:  DZ Amendment #1
Exhibit D:  DZ Amendment #2
Exhibit E:  Heller Amendment #1
Exhibit F:  Heller Amendment #2