<PAGE>

                                                                    EXHIBIT 10.6

                                 TFC LETTERHEAD

Textron Financial Corporation                            333 East River Drive
Subsidiary of Textron Inc.                               Suite 104
                                                         East Hartford, CT 06108
                                                         (860) 282-7776 Phone
                                                         (860) 282-9053 Fax

Robert E. Mead, Chief Executive Officer
Silverleaf Resorts, Inc..
1221 Riverbend Drive, Suite 105

Dallas, TX  75221

July 30, 2004

RE:   $71,000,000 credit facility [Tranche B] (the "Loan") provided to
      Silverleaf Resorts, Inc. ("Borrower") pursuant to that certain Amended and
      Restated Loan, Security and Agency Agreement dated as of April 30, 2002,
      as amended, by and among Silverleaf Resorts, Inc., the parties, including
      Textron Financial Corporation, a Delaware Corporation, which execute and
      deliver the Agreement, in their respective capacities as lenders, and
      Textron Financial Corporation, as Facility Agent and Collateral Agent (the
      "Loan Agreement")

Dear Bob:

Reference is hereby made to the Loan Agreement. All capitalized terms used
herein and not otherwise defined shall have the meanings ascribed to such terms
in the Loan Agreement.

This letter shall confirm that, in connection with the amendments to the
Sovereign Facility, dated as of July 30, 2004, the List of Sovereign Documents
attached to the Loan Agreement is hereby modified, effective as of July 30,
2004, so that Schedule 1.l(iiii) is deleted in its entirety and in its place is
substituted the attached Amended Schedule 1.l(iiii).

This Letter Agreement shall also confirm that the definition of "Collateral" in
the Loan Agreement is hereby modified, effective as of July 30, 2004, so that
Section 1.1(p)(x) is deleted in its entirety and in its place is substituted the
following:

      "(x) All collateral under the Additional Credit Facility, the Tranche C
      Facility, the Inventory Loan and the Sovereign Facility, as each such term
      is herein defined, except the Inventory Mortgages that are collateral for
      the Sovereign Facility, as that term (Inventory Mortgages) is defined in
      the Inventory and Receivables Revolving Credit Agreement, dated as of July
      30, 2004, among Silverleaf Resorts, Inc., Sovereign, as agent, and
      Sovereign and any other lending institutions that may become party thereto
      as lenders, as amended, amended and restated, or modified and in effect
      from time to time ;"

In addition, this Letter Agreement shall confirm that the Loan Agreement is
hereby modified, effective as of July 30, 2004, so that Section 3 is hereby
amended in part to add the following

<PAGE>

new paragraph:

      "3.14 Heller Receivables Collateral. Immediately upon repayment of the
      Heller Facility, the Borrower hereby covenants and agrees that the
      Borrower will grant to the Agent, for the benefit of the Lenders, a first
      priority security interest in all of the Notes Receivable and the
      Mortgages (each such term as defined in the Heller Documents) pledged to
      Heller to secure the Heller Facility immediately prior to the repayment in
      full of such Heller Facility (the "Heller Receivables Collateral"). The
      Borrower hereby covenants and agrees that, prior to repayment in full of
      the Heller Facility, the Borrower will not, and will not permit any of its
      Subsidiaries to, create or incur or suffer to be created or incurred or to
      exist any lien, encumbrance, mortgage, pledge, charge, restriction or
      other security interest of any kind upon the Heller Receivables
      Collateral, other than in favor of Heller pursuant to the Heller
      Documents. The Agent and the Lenders hereby acknowledge that the Borrower
      has agreed to grant an identical and equal priority lien on the Heller
      Receivables Collateral to Sovereign. The Agent and the Lenders hereby
      acknowledge and agree that (a) until such time as the Heller Facility has
      been repaid in full, the Agent and the Lenders have no title or interest
      in the Heller Receivables Collateral and (b) Heller has no duty or
      obligations to the Agent and the Lenders with respect to the Heller
      Receivables Collateral.

            Notwithstanding the foregoing, in the event that in order to raise
      the funds necessary to repay the Heller Facility, the Borrower intends to
      sell the Heller Receivables Collateral into a securitization or to
      refinance the Heller Facility and pledge the Heller Receivables Collateral
      to the lender providing such refinancing, the Agent and the Lenders hereby
      acknowledge and agree that the Agent and the Lenders shall have no claim
      with respect to such Notes Receivables under this Section 3.14 to the
      extent such Notes Receivable are sold into such a securitization or
      pledged to a lender providing a new revolving credit facility permitted by
      the terms of this Agreement; provided however, that in the event that the
      proceeds of the Heller Receivables Collateral received by the Borrower
      from such sale or refinancing of the Heller Receivables Collateral exceed
      the Borrower's outstanding obligations under the Heller Facility, then,
      subject to the terms of the Intercreditor Agreement, the Borrower shall
      deliver any such excess proceeds to the Agent and any such proceeds shall
      be applied to reduce the principal of the Loans.

            The Borrower hereby agrees to take, at its sole cost and expense,
      such steps as the Agent may request to deliver the Heller Receivables
      Collateral to the Agent and to create and perfect the Agent's first
      priority security interest therein."

This Letter Agreement ("Letter Agreement") may be executed in two or more
counterparts, all of which together shall be considered a single instrument.
Delivery of an executed counterpart of a signature page to this Letter Agreement
by facsimile shall be effective as delivery of a manually

<PAGE>

executed counterpart of this Letter Agreement.

Except as expressly set forth herein, this Letter Agreement does not constitute
an amendment or waiver of any term or condition of the Loan, and all such terms
and conditions shall remain in full force and effect.

Please confirm your acknowledgement of and agreement with the terms of this
Letter Agreement by signing in the appropriate space below.

                                                  Very truly yours,
                                                  TEXTRON FINANCIAL CORPORATION,
                                                  A Delaware corporation

                                                  /s/ John D'Annibale
                                                  ------------------------------
                                                  By:  John D'Annibale
                                                  Its: V.P.

The undersigned party acknowledges its agreement with the terms and conditions
of this Letter Agreement:

                                                  SILVERLEAF RESORTS, INC.

                                                  /s/ Robert E. Mead
                                                  ------------------------------
                                                  By:  Robert E. Mead
                                                  Its: CEO