<PAGE>

                                                                    Exhibit 10.6

                              [Textron Letterhead]

November 17, 2003

Silverleaf Resorts, Inc.
1221 River Bend Drive, Suite 105
Dallas, Texas  75221
Attention:  Robert E. Mead, Chief Executive Officer

Re:      $10,200,000 credit facility [Tranche C] (the "Loan") provided to
         Silverleaf Resorts, Inc. ("Borrower") pursuant to that certain Loan and
         Security Agreement dated as of April 17, 2001, by and among Silverleaf
         Resorts, Inc. and Textron Financial Corporation ("TFC"), a Delaware
         Corporation, as Lender, as amended by that certain First Amendment to
         Loan and Security Agreement [Tranche C] dated as of April 30, 2002 by
         and among Textron Financial Corporation and Silverleaf Resorts, Inc.,
         and by those certain letter agreements dated March 27, 2003 and
         September 25, 2003 by and among Textron Financial Corporation and
         Silverleaf Resorts, Inc., as the same may hereafter be amended from
         time to time (the "Loan Agreement")

Gentlemen:

Reference is hereby made to the Loan. All capitalized terms used herein and not
otherwise defined shall have the meaning ascribed to such terms in the Loan
Agreement.

Borrower has advised TFC of the occurrence of certain events that may constitute
a Default or an Event of Default under the Loan Agreement. These Events of
Default arise from: (i) a $28,711,000 increase in Borrower's allowance for
uncollectible notes reflected in Borrower's 10-Q for the period ended March 31,
2003, which caused Borrower to violate the Interest Coverage and Profitable
Operations covenants set forth in Sections 7.1(aa)(iv) and (v) of the Loan
Agreement; and (ii) the retirement of certain subordinated debt pursuant to that
certain Note Purchase and Release Agreement with eight entities and individuals
affiliated with Harry S. Patten and James E. Lambert, which was entered into
without first obtaining the written consent of Lender and the other senior
lenders in violation of Section 7.2(f) of the Loan Agreement.

Subject to the terms of this letter agreement, TFC hereby waives any Default or
Event of Default under the Loan Documents that may have occurred or may occur
hereafter as a result of the condition described in item (i) above and/or that
may have occurred as a result of the condition described in item (ii) above on
or before the date hereof (each such waived Default and/or Event of Default a
"Waived Default" and, collectively, the "Waived Defaults").

<PAGE>

Borrower hereby acknowledges that, by its execution of this letter below, TFC is
not waiving, and that TFC has not waived: (i) any Default or any Event of
Default (other than the Waived Defaults), that may occur hereafter by reason of
any facts which exist or existed on or before the date hereof, whether or not
TFC has notice or knowledge thereof and whether or not TFC has given notice
thereof to Borrower or any of its Affiliates, (ii) any right or remedy of TFC
under any Loan Document, at law or in equity against Borrower, or (iii) any
right or remedy contained in any of the Loan Documents, at law or in equity,
other than in respect of the Waived Defaults.

No Default or Event of Default relating to the Marketing and Sales Expense
covenant as set forth in Section 7.1(aa) (ii) of the Loan Agreement has been
waived as there has been no Event of Default relating to that covenant. By
Letter Agreement dated March 27, 2003 (individually, the "March Letter
Agreement" and collectively, along with the letter agreement executed by TFC in
favor of Silverleaf on September 25, 2003, the "Letter Agreements"), TFC amended
its Loan Agreement to increase the permissible ratio of Marketing and Sales
Expenses to the Borrower's net proceeds to .550 to 1 (a ratio that has not been
exceeded by Borrower). Although the March Letter Agreement was conditioned upon
the execution of similar agreements relating to the Heller and Sovereign Bank
facilities, Borrower, by its acknowledgment of and agreement to this letter
below, has confirmed that those agreements have been obtained and that all of
the conditions of the March Letter Agreement have been satisfied.

Borrower hereby releases TFC from liability for any claim, liability, judgment,
obligation, loss, cost, damage or expense arising from, or relating to any
action taken by TFC or any of its Affiliates and/or its officers, directors,
employees, attorneys and agents in connection with the Loan.

Furthermore, subject to the terms and conditions contained herein, TFC hereby
consents to: (i) that certain First Amendment to Amended and Restated Revolving
Credit Agreement dated as of September 30, 2002 by and among Silverleaf Resorts,
Inc., Sovereign Bank and Liberty Bank (the "Sovereign Amendment #1) a copy of
which is attached hereto as Exhibit A; (ii) that certain Second Amendment to
Amended and Restated Revolving Credit Agreement dated as of ____________ ____,
2003 by and among Silverleaf Resorts, Inc., Sovereign Bank and Liberty Bank (the
"Sovereign Amendment #2"), a copy of which is attached hereto as Exhibit B;
(iii) that certain First Amendment to Amended and Restated Receivables Loan and
Security Agreement dated as of August 6, 2002 by and among Silverleaf Finance I,
Inc., as Borrower, Silverleaf Resorts, Inc., as initial Servicer, Autobahn
Funding Company LLC, as Lender, DZ Bank AG Deutsche Zentral-Genossenschaftsban,
Frankfurt AM, MAIN, as Agent for the Lender, U.S. Bank National Association, as
Agent's Bank, and Wells Fargo Bank Minnesota, National Association, as Backup
Servicer (the "DZ Amendment #1"), a copy of which is attached hereto as Exhibit
C; (iv) that certain Second Amendment and Waiver Agreement dated as of June 19,
2003 by and among Silverleaf Finance I, Silverleaf Resorts, Inc., as Servicer,
Autobahn Funding Company LLC, as lender, DZ Bank AG Deutsche
Zentral-Genossenschaftsbank, Frankfurt Am Main, as Agent, U.S. Bank Trust
National Association, as Agent's bank, and Wells Fargo Bank Minnesota, National
Association, as Backup Servicer (the "DZ Amendment #2"), a copy of which is
attached hereto as Exhibit D; (v) that certain Fifth Amendment to Second Amended
and Restated Inventory Loan and Security Agreement and Modification of Notes
dated as of November 21, 2003 by and among Silverleaf Resorts and Heller
Financial, Inc., as lender (the

<PAGE>

"Heller Amendment #1"), a copy of which is attached hereto as Exhibit E; and
(vi) that certain Third Amendment to Amended and Restated Receivables Loan and
Security Agreement dated as of November 21, 2003 by and among Silverleaf Resorts
and Heller Financial, Inc., as Lender (the "Heller Amendment #2), a copy of
which is attached hereto as Exhibit F.

In consideration of the waivers set forth herein and in consideration of
Textron's consent to Sovereign Amendment #1, Sovereign Amendment #2, DZ
Amendment #1, DZ Amendment #2, Heller Amendment #1 and Heller Amendment #2 as
set forth herein, Borrower hereby agrees to pay to TFC, individually under the
Loan Agreement and individually and in its capacity as Agent for each of the
Lenders under the other Textron credit facilities, the sum of $50,000.00 (the
"Waiver Fee"). This Waiver Fee shall be payable in consideration not only of the
waivers and consents set forth herein but also in consideration of the same
waivers and consents being granted under the other existing Textron credit
facilities (collectively the "Textron Waivers and Consents"), it being
understood that the total amount Borrower shall be required to pay for the
Textron Waivers and Consents shall be the sum of $50,000.00.

This letter is conditioned upon and shall not be effective unless and until: (i)
similar Agreements are executed relating to the Heller, Sovereign Bank, DZ and
other Textron credit facilities; (ii) payment by the Borrower of the Waiver Fee
is received by TFC in immediately available funds; and (iii) TFC has received
evidence, in form and substance satisfactory to TFC, that the consent of each
party entitled to consent to this letter pursuant to the terms of the Sovereign
Documents, the Heller Documents and any other document evidencing any other
Indebtedness of the Borrower has been obtained.

This letter may be executed in two or more counterparts, all of which together
shall be considered a single instrument. Delivery of an executed counterpart of
a signature page to this letter by facsimile shall be effective as delivery of a
manually executed counterpart of this letter.

Except as expressly set forth herein, this letter does not constitute an
amendment or waiver of any term or condition of the Loan, the Loan Agreement or
any of the Loan Documents, and all such terms and conditions shall remain in
full force and effect. Furthermore, this letter shall not be construed as
establishing any precedent for any future request for a waiver or amendment of
any of the terms or conditions of the Loan, the Loan Agreement or any of the
Loan Documents, which TFC hereby reserves the right to enforce strictly.

This letter is also entered into with the understanding and upon the condition
that, except for the Waived Defaults, Borrower is not in default under any of
its credit facilities.

<PAGE>

Please confirm your acknowledgement of and agreement with the terms of this
letter by signing in the appropriate space below.

                                              Very truly yours,
                                              TEXTRON FINANCIAL CORPORATION,
                                              a Delaware corporation

                                                 /S/ JOHN T. D'ANNIBALE
                                              ----------------------------------
                                              By:  John T. D'Annibale
                                              Its: V.P.

The undersigned party acknowledges its agreement with the terms and conditions
of this letter:

                                              SILVERLEAF RESORTS, INC.

                                                 /S/ ROBERT E. MEAD
                                              ----------------------------------
                                              By:
                                              Its:

Exhibits:

Exhibit A:  Sovereign Amendment
Exhibit B:  Sovereign Amendment #2
Exhibit C:  DZ Amendment #1
Exhibit D:  DZ Amendment #2
Exhibit E:  Heller Amendment #1
Exhibit F:  Heller Amendment #2