<PAGE>   1
                                                                    Exhibit 10.6



                                 FIFTH AMENDMENT

              FIFTH AMENDMENT (this "Amendment"), dated as of November 3, 2000,
among TOWN SPORTS INTERNATIONAL, INC., a New York corporation (the "Borrower"),
the various lending institutions party to the Credit Agreement referred to below
(the "Banks"), and BANKERS TRUST COMPANY, as administrative agent (the
"Administrative Agent"). All capitalized terms used herein and not otherwise
defined shall have the respective meanings provided such terms in the Credit
Agreement referred to below.

                                   WITNESSETH:

              WHEREAS, the Borrower, the Banks and the Administrative Agent are
parties to an Amended and Restated Credit Agreement, dated as of October 16,
1997 (as amended, modified or supplemented through, but not including, the date
hereof, the "Credit Agreement"); and

              WHEREAS, the parties hereto wish to amend the Credit Agreement to
permit the Borrower to incur up to $22,500,000 of senior subordinated loans on
the terms and conditions set forth herein;

              NOW, THEREFORE, it is agreed that as of the Fifth Amendment
Effective Date (as defined below):

              1. Section 1.08(d) of the Credit Agreement is hereby deleted and
the following new Section 1.08(d) is inserted in lieu thereof:

              "(d) Interest shall accrue from and including the date of any
         Borrowing to but excluding the date of any repayment thereof and shall
         be payable (i) in respect of each Base Rate Loan, monthly in arrears on
         the last Business Day of each calendar month, (ii) in respect of each
         Eurodollar Loan, on the last day of each Interest Period applicable
         thereto and, in the case of an Interest Period in excess of one month,
         on the date occurring at one month intervals after the first day of
         such Interest Period, and (iii) in respect of each Loan, on any
         prepayment or conversion (other than the prepayment and conversion of
         Revolving Loans that are maintained as Base Rate loans unless such Base
         Rate Loans are to be repaid in full) (on the amount prepaid or
         converted), at maturity (whether by acceleration of otherwise) and,
         after maturity, on demand."
<PAGE>   2
              2. Section 3.01(a) of the Credit Agreement is hereby amended by
deleting the last sentence thereof and inserting the following new last sentence
in lieu thereof:

         "Such Commitment Commission shall be due and payable monthly in
         arrears on the last Business Day of each calendar month and on the date
         upon which the Total Commitment is terminated."

              3. Section 3.01(b) of the Credit Agreement is hereby amended by
deleting the last sentence thereof and inserting the following new last sentence
in lieu thereof:

         "Accrued Letter of Credit Fees shall be due and payable monthly in
         arrears on the last Business Day of each calendar month and upon the
         first day on or after the termination of the Total Commitment upon
         which no Letters of Credit remain outstanding."

              4. Section 3.01(c) of the Credit Agreement is hereby amended by
deleting the last sentence thereof and inserting the following new last sentence
in lieu thereof:

         "Accrued Facing Fees shall be due and payable monthly in arrears
         on the last Business Day of each calendar month and upon the first day
         on or after the termination of the Total Commitment upon which no
         Letters of Credit remain outstanding."

              5. Clause (d) of Section 3.03 of the Credit Agreement is hereby
amended by deleting the text "Restatement Effective Date" appearing in the
second parenthetical contained in such clause (d) and inserting in lieu thereof
the text "Fifth Amendment Effective Date".

              6. Section 6 of the Credit Agreement is hereby amended by
inserting the following new Section 6.23 at the end thereof:

              "6.23 Mezzanine Subordinated Loan Documents. The subordination
         provisions contained in the Mezzanine Subordinated Notes and in the
         other Mezzanine Subordinated Loan Documents are enforceable against the
         Borrower and the holders of the Mezzanine Subordinated Notes, and all
         Obligations are within the definition of "Senior Indebtedness" included
         in such subordination provisions."

              7. Section 7.01 of the Credit Agreement is hereby amended by
inserting the following new clause (i) at the end thereof:

              "(i) Mezzanine Subordinated Loan Documents. Promptly after the
         delivery thereof, copies of all financial information, reports and
         other notices which the Borrower shall deliver to the holders of the
         Mezzanine Subordinated Notes (in each case, to the extent that such
         information is not already delivered to the Banks pursuant to this
         Agreement)."


                                      -2-
<PAGE>   3
              8. Section 8.04 of the Credit Agreement is hereby amended by (i)
deleting the period appearing at the end of clause (i) thereof and inserting ";
and" in lieu thereof and (ii) inserting the following new clause (j) at the end
thereof:

              "(j) unsecured Indebtedness of the Borrower incurred under the
         Mezzanine Subordinated Notes and the other Mezzanine Subordinated Loan
         Documents not to exceed $22,500,000 in aggregate principal amount (less
         any repayments of principal thereof), so long as (i) all of the terms
         and conditions of, and the documentation for, the Mezzanine
         Subordinated Notes and the other Mezzanine Subordinated Loan Documents
         are in form and substance reasonably satisfactory to the Required Banks
         and (ii) prior to the incurrence of each Mezzanine Subordinated Loan
         under the Mezzanine Subordinated Loan Documents, the Borrower shall
         have delivered to the Agent a certificate of the Borrower's chief
         financial officer demonstrating (and showing the calculations therefor
         in reasonable detail) that each such Mezzanine Subordinated Loan may be
         incurred under the incurrence test set forth in the first paragraph of
         Section 4.12 of the indenture for the Senior Notes."

              9. Section 8.06 of the Credit Agreement is hereby deleted in its
entirety and the following new Section 8.06 is inserted in lieu thereof:

              "8.06 Prepayments of Indebtedness, etc. The Borrower will not, and
         will not permit any of its Subsidiaries to, (x) make (or give any
         notice in respect thereof) any voluntary or optional payment or
         prepayment or redemption or acquisition for value of (including,
         without limitation, by way of depositing with the trustee with respect
         thereto money or securities before due for the purpose of paying when
         due) or exchange of any Existing Indebtedness, any Senior Notes or any
         Mezzanine Subordinated Notes or Mezzanine Subordinated Loans; provided,
         that (i) the Borrower and/or its Subsidiaries may prepay Existing
         Indebtedness in an aggregate amount not to exceed $2,000,000, and (ii)
         the Borrower may repay, repurchase or otherwise acquire Senior Notes
         under the clawback provisions of the Senior Notes Documents, (y) amend,
         modify or change in any manner any agreements (including, without
         limitation, the Senior Note Documents, the Mezzanine Subordinated Loan
         Documents and the Existing Indebtedness Agreements) (A) relating to the
         Senior Notes, (B) relating to the Mezzanine Subordinated Notes or (C)
         relating to the Existing Indebtedness, in a manner adverse to the
         interests of the Banks in the case of this clause (C) or (z) amend,
         modify or change in any manner materially adverse to the interests of
         the Banks, the Certificate of Incorporation (including, without
         limitation, by the filing of any additional certificate of designation)
         or By-Laws of the Borrower or any of its Subsidiaries, the terms of any
         of its capital stock or any agreement entered into by the Borrower with
         respect to its capital stock, the Shareholders Agreement, or enter into
         any new agreement in any manner materially adverse to the interests of
         the Banks with respect to the capital stock of the Borrower."

              10. Clause (b) of Section 8.07 of the Credit Agreement is hereby
amended by (i) deleting the word "and" appearing at the end of sub-clause (v)
thereof, (ii) deleting the period


                                      -3-
<PAGE>   4
appearing at the end of sub-clause (vi) thereof and inserting "; and" in lieu
thereof and (iii) inserting the following new sub-clause (vii) at the end
thereof:

              "(vii) the Mezzanine Subordinated Loan Documents."

              11. Sub-clause (ii) of Section 8.08 of the Credit Agreement is
hereby amended by deleting the amount "$250,000" appearing therein and inserting
the amount "$500,000" in lieu thereof.

              12. The definition of "Change of Control" appearing in Section 10
of the Credit Agreement is hereby amended by inserting the following text at the
end of clause (iv) thereof:

              "or the Mezzanine Subordinated Loan Documents".

              13. The definition of "L/C Supportable Obligations" appearing in
Section 10 of the Credit Agreement is hereby amended by inserting the following
parenthetical at the end thereof:

              "(other than obligations in respect of the Mezzanine Subordinated
              Notes and the Senior Notes)".

              14. Section 10 of the Credit Agreement is hereby further amended
by inserting the following new definitions in the appropriate alphabetical
order:

              "Fifth Amendment Effective Date" shall have the meaning set forth
in the Fifth Amendment, dated as of November 3, 2000, to this Agreement.

              "Mezzanine Subordinated Loan" shall mean each loan or advance made
pursuant to the Mezzanine Subordinated Loan Documents.

              "Mezzanine Subordinated Loan Documents" shall mean and include (i)
the Mezzanine Subordinated Notes, (ii) the Subordinated Credit Agreement, dated
on or about November 3, 2000, between the Borrower and CapitalSource Finance,
LLC, and (iii) each of the other documents and agreements entered into relating
to the issuance by the Borrower of the Mezzanine Subordinated Notes, in each
case as the same may be amended, modified or supplemented from time to time
pursuant to the terms hereof and thereof.

              "Mezzanine Subordinated Note" shall mean each note of the Borrower
evidencing the Mezzanine Subordinated Loans.

              15. In order to induce the Banks to enter into this Amendment, the
Borrower hereby represents and warrants that on the Fifth Amendment Effective
Date, both before and after giving effect to this Amendment and the transactions
contemplated hereby, (x) no Default or Event of Default shall exist and (y) all
of the representations and warranties contained in the Credit Documents shall be
true and correct in all material respects, with the same effect as though such
representations and warranties had been made on and as of the Fifth Amendment


                                      -4-
<PAGE>   5
Effective Date (it being understood that any representation or warranty made as
of a specific date shall be true and correct in all material respects as of such
specific date).

              16. This Amendment is limited as specified and shall not
constitute a modification, acceptance or waiver of any other provision of the
Credit Agreement or any other Credit Document.

              17. This Amendment may be executed in any number of counterparts
and by the different parties hereto on separate counterparts, each of which
counterparts when executed and delivered shall be an original, but all of which
shall together constitute one and the same instrument. A complete set of
counterparts shall be lodged with the Borrower and the Agent.

              18. THIS AMENDMENT AND THE RIGHTS AND OBLIGATIONS OF THE PARTIES
HEREUNDER SHALL BE CONSTRUED IN ACCORDANCE WITH AND GOVERNED BY THE LAW OF THE
STATE OF NEW YORK.

              19. This Amendment shall become effective as of the date hereof
(the "Fifth Amendment Effective Date") when the Borrower and the Required Banks
shall have signed a counterpart hereof (whether the same or different
counterparts) and shall have delivered (including by way of telecopier) the same
to the Administrative Agent.


                                      * * *


                                      -5-
<PAGE>   6
              IN WITNESS WHEREOF, each of the parties hereto has caused a
counterpart of this Amendment to be duly executed and delivered as of the date
first above written.

                                  TOWN SPORTS INTERNATIONAL, INC.



                                  By /S/ R.G.Pyle
                                     -------------------------------------------
                                     Title:  CFO



                                  BANKERS TRUST COMPANY,
                                       Individually and as Administrative Agent


                                  By /S/ Gregory Shefrin
                                     -------------------------------------------
                                     Title: Principal



                                  BANK OF SCOTLAND



                                  By /S/  Joseph Fratus
                                     -------------------------------------------
                                     Title: Vice President