<PAGE>   1
                                                                   EXHIBIT 10.73

                       SECOND AMENDMENT TO FOURTH AMENDED
                           AND RESTATED LOAN AGREEMENT


        THIS SECOND AMENDMENT TO FOURTH AMENDED AND RESTATED LOAN AGREEMENT
("Amendment") is made and entered into as of August 10, 2000 by and among: (a)
The Sports Club Company, Inc., a Delaware corporation; Pontius Realty, Inc., a
California corporation; The SportsMed Company, Inc., a California corporation;
LA/Irvine Sports Clubs, Ltd., a California limited partnership; Talla New York,
Inc., a New York corporation; SCC Sports Club, Inc., a Texas corporation; Irvine
Sports Club, Inc., a California corporation; SCC Nevada, Inc. f/k/a Green Valley
Spectrum Club, Inc.; Sports Club, Inc. of California, a California corporation;
SCC Liquidating Company, a California corporation f/k/a Spectrum Liquidating
Corp.; SF Sports Club, Inc., a Delaware corporation; Washington D.C. Sports
Club, Inc., a Delaware corporation; HFA Services, Inc., a California
corporation; and NY Sports Club, Inc., a Delaware corporation (collectively, the
"Borrowers"); (b) Comerica Bank-California ("Comerica"), as the Bank; and (c)
Comerica Bank-California, as agent (in such capacity, the "Agent") for the banks
(the "Banks") that are parties to the Loan Agreement (as defined below), and is
made with reference to the following:

        A. Comerica, the Agent and the Borrowers have entered into that certain
Fourth Amended and Restated Loan Agreement, dated as of April 1, 1999, and
amended pursuant to that certain First Amendment to Loan Agreement (the "First
Amendment"), dated as of December 3, 1999 (as amended, as the same may hereafter
be amended, modified, extended and/or restated, from time to time, the "Loan
Agreement".) Pursuant to the Loan Agreement, the Bank has made certain Loans to
the Borrowers and has committed to make additional Loans in the future upon the
satisfaction of certain conditions.

        B. The Borrowers, the Agent and the Bank wish to amend the Loan
Agreement as more particularly set forth below.

        NOW, THEREFORE, in consideration of the premises and the agreements,
conditions and covenants contained herein, the parties hereby agree as follows:

        1. Defined Terms.

               (a) "Effective Date" shall mean the date on which each of the
conditions precedent set forth in Section 5 of this Amendment is satisfied.

               (b) All capitalized terms used in this Amendment and not
otherwise defined herein shall have the meanings assigned to them in the Loan
Agreement.

        2. Amendments to Loan Agreement. Effective as of the Effective Date, the
Loan Agreement is hereby amended in the following respects:

               2.1 The following new definitions of "Current Assets", "Current
Liabilities", "Current Ratio", "Interest Coverage Ratio", "Interest Expense" and
"Subordinated Indebtedness" are added to Section 1.1 of the Loan Agreement:

                    "Current Assets" means, as of any applicable date of
determination, the sum of (a) all unrestricted cash, CD's or marketable
securities of Borrowers, (b) all net accounts receivable of Borrowers owing by
Persons which are not Affiliates thereof, (c) all United States

<PAGE>   2

Government securities and/or claims against the United States Government, (d)
all Inventories held for sale or lease in the ordinary course of business of
Borrowers and (e) other current assets.

                    "Current Liabilities" means, as of any applicable date of
determination, (a) all liabilities of Borrowers that should be classified as
current in accordance with generally accepted accounting principles consistently
applied, including, without limitation, any portion of the principal of the
indebtedness under this Agreement, the Notes and/or the Loan Documents
classified as current, plus, (b) to the extent not otherwise included, all
liabilities of Borrowers to any of their Affiliates (including officers,
directors, shareholders, Subsidiaries and commonly held companies), whether or
not classified as current in accordance with generally accepted accounting
principles consistently applied unless the same consist of the long term portion
of Subordinated Indebtedness except any deferred rent liability to Millenium
Partners.

                    "Current Ratio" means as of any applicable date of
determination, the Current Assets divided by the Current Liabilities.

                    "Interest Coverage Ratio" means, EBITDA for the applicable
period of determination plus the positive changes in deferred revenues of
Borrowers for the applicable period of determination divided by Interest Expense
less the interest income of Borrowers for the applicable period of determination
plus the capitalized interest of Borrowers for the applicable period of
determination. With respect to the fiscal quarters ending June 30, 2000,
September 30, 2000, December 31, 2000 and March 31, 2001, the measurement period
will be on a cumulative quarterly basis commencing April 1, 2000 to the date of
determination, and with respect to the fiscal quarter ending June 30, 2001 and
all fiscal quarters ending thereafter, the applicable period of determination
will be the four (4) full fiscal quarters immediately preceding the date of
determination.

                    "Interest Expense" means, for any applicable period of
determination, the total interest expense (including that attributable to
capitalized leases) of Borrowers for such period with respect to all of their
indebtedness, including, without limitation, all breakage costs and fees, all
commissions, discounts and other fees and charges owed with respect to letters
of credit and bankers acceptance financing and the accretion of original issue
discount on debt securities, all as determined in conformity with generally
accepted accounting principles consistently applied. The applicable period of
determination will be the four (4) full fiscal quarters immediately preceding
the date of determination.

                    "Subordinated Indebtedness" means, as of any applicable date
of determination, all indebtedness of Borrowers to third parties which has been
subordinated to all indebtedness owing by Borrowers to the Banks pursuant to a
subordination agreement in form and content satisfactory to the Banks.

               2.2 Amendment to Tangible Net Worth Covenant.

                    (a) Section 6.13 of the Loan Agreement is hereby amended and
restated to read in full as follows:

                    "6.13 Tangible Net Worth. Permit Tangible Net Worth to be
less than the amount specified below as of the indicated fiscal quarter-end
date: (i) on June 30, 2000, not less than $78,000,000 (ii) on September 30,
2000, not less than $75,000,000, (iii) on December 31, 2000, not less than
$70,000,0000, (iv) on March 31,

<PAGE>   3

2001, not less than $64,000,000 and (v) on June 30, 2001 and as of the end of
each fiscal quarter thereafter, not less than $59,000,000."

               2.3 Amendment to Ratio of Total Unsubordinated Liabilities to
Tangible Net Worth.

                    (a) Section 6.14 of the Loan Agreement is hereby amended and
restated to read in full as follows:

                    "6.14 Ratio of Total Unsubordinated Liabilities to Tangible
Net Worth. Permit the ratio of Total Unsubordinated Liabilities to Tangible Net
Worth to be greater than the amount specified below as of the indicated fiscal
quarter-end date: (i) on June 30, 2000, to be greater than 2.00:1.00, (ii) on
September 30, 2000, to be greater than 2.00:1.00, (iii) on December 31, 2000, to
be greater than 2.00:1.00, (iv) on March 31, 2000, to be greater than 2.25:1:00
and (v) on June 30, 2001 and as of the end of each fiscal quarter thereafter, to
be greater than 2.50:1.00."

               2.4 Addition of Minimum Interest Coverage Ratio Covenant.

                    (a) Section 6.15 of the Loan Agreement is deleted in its
entirety and replaced in full with the following:

                         "Minimum Interest Coverage Ratio. Permit the Interest
Coverage Ratio to be less than the amount specified below as of the indicated
fiscal quarter-end date: (i) on June 30, 2000, to be less than .20 to 1.00; (ii)
on September 30, 2000, to be less than .50 to 1.00, (iii) on December 31, 2000,
to be less than .50 to 1.00, (iv) on March 31, 2001, to be less than .45 to
1.00, (v) on June 30, 2001 and as of the end of each fiscal quarter thereafter,
to be less than .65 to 1.00."

               2.5 Current Ratio Covenant. The following new covenant is hereby
added at the end of Section 6 of the Loan Agreement:

                         "6.21 Current Ratio. Permit the Current Ratio to be
greater than the amount specified below as of the indicated fiscal quarter-end
date: (i) on June 30, 2000, to be greater than 1.40 to 1.00; (ii) on September
30, 2000, to be greater than .75 to 1.00, (iii) on December 31, 2000, to be
greater than .70 to 1.00, (iv) on March 31, 2001, to be greater than .30 to
1.00, (v) on June 30, 2001 and as of the end of each fiscal quarter thereafter,
to be greater than .30 to 1.00."

        General Amendment. Effective as of the Effective Date, the Loan
Agreement and all other Loan Documents are hereby amended to the further extent
required to give effect to the terms and conditions of the amendments to the
Loan Agreement effected pursuant to Section 2 above.

        Full Force and Effect. Effective as of the Effective Date, each of the
Loan Documents is hereby amended such that all references to the Loan Agreement
contained in any such documents shall be deemed to be references to the Loan
Agreement, as amended by this Amendment. Except as amended hereby, the Loan
Agreement and the other Loan Documents shall remain unaltered and in full force
and effect.

        3. Conditions Precedent. The satisfaction of the following shall be
conditions precedent for the benefit of the Agent and the Banks to the
effectiveness of this Amendment:

<PAGE>   4

               3.1 Restructuring Fee. Borrowers shall have paid to the Agent a
restructuring fee in the amount of Ten (10) basis points representing Fifteen
Thousand Dollars ($15,000).

               3.2 Reaffirmation of Intercreditor Agreement. The Agent shall
have received a Reaffirmation of Intercreditor Agreement in the form of Exhibit
"A" to this Amendment duly executed by each of the parties to the same.

               3.3 Corporate Documents. The Agent shall have received (a) a
certificate of an officer of each Borrower to the effect that such Borrower is
in compliance in all material respects with all material requirements of
applicable law; (b) a certificate of the chief financial officer of each
Borrower stating that, after giving effect to the modifications to the Loan
Agreement effected herein, as of the Effective Date no Default or Event of
Default shall have occurred and be continuing on the Effective Date; and (c)
such additional approvals, documents and other information, in form and
substance satisfactory to the Agent, as the Agent may reasonably request.

               3.4 Bank Expenses. All legal fees, costs and other expenses which
the Agent and the Bank have incurred in connection with this Amendment as of the
Effective Date but which have not previously been reimbursed by Borrowers shall
have been so reimbursed by Borrowers.

        4. Representations and Warranties. Each Borrower hereby represents and
warrants to the Agent and the Banks that each representation and warranty made
by it in Article IV of the Loan Agreement and each representation and warranty
made by it in each other Loan Document, in each case as updated in connection
with the First Amendment, is true and correct on and as of the Effective Date as
though made as of the Effective Date, except to the extent such representations
and warranties relate solely to an earlier date.

        5. Counterparts. This Amendment may be executed in multiple
counterparts, each of which shall constitute an original and all of which, taken
together, shall constitute but one and the same instrument.

        6. Governing Law. This Amendment shall be governed by, and construed in
accordance with, the laws of the State of California.



        [Remainder of page intentionally left blank; signatures follow]

<PAGE>   5

        IN WITNESS WHEREOF, the parties hereto have executed this Amendment by
their respective duly authorized officers as of the date first above written.

BORROWERS:


THE SPORTS CLUB COMPANY, INC.
A Delaware corporation


By:  /s/ Timothy O'Brien
     -----------------------------------
     Timothy O'Brien
     Its:    Chief Financial Officer


PONTIUS REALTY, INC.,                     LA/IRVINE SPORTS CLUBS, LTD.,
a California corporation                  a California limited partnership

                                          By:   Sports Club, Inc. of California,
                                                general partner
By:  /s/ Timothy O'Brien
     -----------------------------------
Its: Chief Financial Officer
                                          BY:    /s/ Timothy O'Brien
                                             -----------------------------------
                                                 Timothy O'Brien
                                                 Its:  Chief Financial Officer

SPORTS CLUB, INC. OF CALIFORNIA,          TALLA NEW YORK, INC.,
a California corporation                  a New York corporation

By:  /s/ Timothy O'Brien                  By:    /s/ Timothy O'Brien
     -----------------------------------     -----------------------------------
     Timothy O'Brien                             Timothy O'Brien
     Its:    Chief Financial Officer             Its:  Chief Financial Officer

IRVINE SPORTS CLUB,INC.,                  SCC NEVADA, INC.,
a California corporation                  a Nevada corporation

By:  /s/ Timothy O'Brien                  By:    /s/ Timothy O'Brien
     -----------------------------------     -----------------------------------
     Its:    Chief Financial Officer             Its:  Chief Financial Officer

THE SPORTSMED COMPANY, INC.,
a California corporation

By:  /s/ Timothy O'Brien
     -----------------------------------
     Its:    Chief Financial Officer

<PAGE>   6

SCC SPORTS CLUB, INC.,
a Texas corporation


By:  /s/ Timothy O'Brien
     -----------------------------------
     Its:    Chief Financial Officer


SCC LIQUIDATING CORPORATION,
a California corporation


By:  /s/ Timothy O'Brien
     -----------------------------------
     Timothy O'Brien
     Its:    Chief Financial Officer

HFA SERVICES, INC.
a California corporation


By:  /s/ Timothy O'Brien
     -----------------------------------
     Its:    Chief Financial Officer

NY SPORTS CLUB, INC.
a Delaware corporation


By:  /s/ Timothy O'Brien
     -----------------------------------
     Its:    Chief Financial Officer

SF SPORTS CLUB, INC.
a Delaware corporation


By:  /s/ Timothy O'Brien
     -----------------------------------
     Its:    Chief Financial Officer

<PAGE>   7

WASHINGTON D.C. SPORTS CLUB, INC.
a Delaware corporation


By:  /s/ Timothy O'Brien
     -----------------------------------
     Its:    Chief Financial Officer


AGENT:

COMERICA BANK-CALIFORNIA,
A California banking corporation, as Agent


By:  /s/ William Phillips
     -----------------------------------
             William Phillips
             Vice President

COMERICA BANK-CALIFORNIA
a California banking corporation


By:  /s/ William Phillips
     -----------------------------------
             William Phillips
             Vice President