<PAGE>   1
                                                                  EXHIBIT 10.61


                       SECOND AMENDMENT TO LOAN AGREEMENT


                 This Second Amendment to Loan Agreement ("Second Amendment")
is dated as of August 14, 1997 and is entered into by and among The Sports Club
Company, Inc., The Spectrum Club Company, Inc., Pontius Realty, Inc., Sports
Club, Inc. of California, Irvine Sports Club, Inc., HealthFitness Organization
of America, Inc., L.A./Irvine Sports Club, Ltd., Talla New York, Inc., SCC
Sports Club, Inc., and Green Valley Spectrum Club, Inc. (collectively,
"Borrowers") and Sumitomo Bank of California ("Bank").

                                R E C I T A L S

         A.      The Sports Club Company, Inc., The Spectrum Club Company, Inc.
("Spectrum"), Pontius Realty, Inc., Sports Club, Inc. of California, Irvine
Sports Club, Inc., HealthFitness Organization of America, Inc., L.A./Irvine
Sports Club, Ltd., Talla New York, Inc., SCC Sports Club, Inc., and Green
Valley Spectrum Club, Inc. as Borrowers, and Bank, as lender, are parties to
that certain Loan Agreement dated as of March 20, 1997 as subsequently amended
(collectively, "Loan Agreement") pursuant to which Bank has provided certain
credit facilities to Borrowers.

         B.      HealthFitness Organization of America, Inc., a Borrower
hereunder ("HealthFitness") has incorporated and is presently the 100%
shareholder of HFA Services, Inc., a California corporation, ("HFA").  HFA has
purchased certain assets of Sports Therapy Systems, Inc.  pursuant to an
Agreement for Purchase and Sale of Assets in the amount of $472,203.00 ("STS
Acquisition").

         C.      Pursuant to the STS Acquisition, HFA has entered into an
"Agreement for Purchase and Sale of Assets" and an Employment Agreement with
the principal of SportsTherapy Systems, Inc., and HealthFitness Organization of
America, Inc. and Larry Schwartz have entered into a Shareholders Agreement.

         D.      Certain funds were advanced by Borrowers to HFA in connection
with the STS Acquisition.  HFA will, however, be a Non-Borrower Affiliate under
the Loan





                                      -2-
<PAGE>   2
Agreement.  Sections 5.6,  6.2 and 6.3 of the Loan Agreement require certain
Bank consents, approvals or waivers with respect to intercompany loans,
Acquisitions and transactions with Non-Borrower Affiliates in connection with
the STS Acquisition on terms and arrangements acceptable to Bank, in its sole
discretion.

         E.      Bank is willing to consent to the STS Acquisition and related
actions as described above on the terms and conditions hereafter provided.


                               A G R E E M E N T

                 NOW, THEREFORE, in consideration of the foregoing, and for
other good and valuable consideration, the receipt and sufficiency of which are
hereby acknowledged, the Borrowers and Bank hereby agree as follows:

                 1.       Defined Terms.  Except as otherwise defined herein,
capitalized terms set forth herein without definition shall have the respective
meanings assigned to them in the Loan Agreement.

                 2.       Consent, Approval and Waiver.  Bank approval and
consent to the STS Acquisition is subject to the following conditions, each of
which is agreed to by the Borrowers:

                          (a)     HealthFitness does hereby pledge all of its
         right, title and interest in the HFA stock to Bank as collateral
         security for all of its Obligations under the Loan Documents.  In
         connection herewith, HealthFitness agrees to execute such pledge
         agreements, financing statements and other documents and agreements as
         may be required by Bank in connection with such pledge of the HFA
         Stock.

                          (b)     The STS Acquisition shall be funded by a loan
         from HealthFitness to HFA in the amount of $472,203.00 ("HFA Loan").
         HFA shall not be required to become a Borrower and the HFA Loan will
         not be included in the aggregate amount of advances to, investments
         in, or commitments to HFA for purposes of Section 5.6 of the Loan
         Agreement.  The HFA Loan shall be evidenced by a promissory note
         ("Note") in form and substance acceptable to Bank.  In addition, such
         Note shall be secured by a perfected first priority





                                      -3-
<PAGE>   3

         security interest in the assets of HFA.  Such Note shall be assigned
         and pledged by HealthFitness to Bank along with all collateral
         security for the payment thereof, as additional security for
         HealthFitness' Obligations under the Loan Agreement.  HFA and
         HealthFitness shall execute such security documents, promissory notes,
         financing statements, assignments and other agreements as Bank may
         require in connection herewith or therewith, such documents to be in
         form and substance acceptable to Bank, including, among other things,
         a restriction on any prepayment of the HFA Loan unless made with cash
         generated from the operation of HFA.

                 3.       Schedules.  Schedules 5.2, 5.6 and 6.10 to the Loan
Agreement are amended as provided in the Schedules attached hereto.

                 4.       Representations and Warranties.  Each representation
made by the Borrowers in the Loan Agreement is true and correct on and as of
the date hereof as though made as of the date hereof, except to the extent such
representations and warranties relate solely to an earlier date or have become
inaccurate without resulting in a breach of the Loan Agreement.

                 5.       Full Force and Effect.  Except as expressly amended
hereby, the Loan Agreement and any other Loan Documents shall remain unaltered
and in full force and effect.

                 6.       Counterparts.  This Second Amendment may be executed
in multiple counterparts, each of which shall constitute an original and all of
which, taken together, shall constitute but one and the same instrument.

                 7.       Amendment Fee.  Borrowers shall pay to Bank,
concurrent with the execution of this Second Amendment, an amendment fee in an
amount equal to one-eighth of one percent of the Commitment ($6,250.00).


                 IN WITNESS WHEREOF, this Second Amendment has been executed as
of the date first above written.


                                   "Borrowers"





                                      -4-
<PAGE>   4
                                     THE SPORTS CLUB COMPANY, INC.

                                     By      /s/ Timothy O'Brien
                                         -----------------------------------

                                     Its     Chief Financial Officer
                                         -----------------------------------

                                     THE SPECTRUM CLUB COMPANY, INC.

                                     By      Timothy O'Brien
                                         -----------------------------------
                                     Its     Chief Financial Officer
                                         -----------------------------------

                                     PONTIUS REALTY, INC.

                                     By      Timothy O'Brien
                                         -----------------------------------

                                     Its     Chief financial Officer
                                         -----------------------------------

                                     SPORTS CLUB, INC. OF CALIFORNIA

                                     By      Timothy O'Brien
                                         -----------------------------------

                                     Its     Chief Financial Officer
                                         -----------------------------------

                                     IRVINE SPORTS CLUB, INC.

                                     By      Timothy O'Brien
                                         -----------------------------------

                                     Its     Chief Financial Officer
                                         -----------------------------------





                                     -5-
<PAGE>   5


                                     HEALTHFITNESS ORGANIZATION OF AMERICA, INC.

                                     By      Timothy O'Brien
                                         -----------------------------------

                                     Its     Chief Financial Officer



                                     L.A./IRVINE SPORTS CLUB, LTD.

                                     By     Sports Club, Inc.. of California
                                            General Partner


                                            By    Timothy O'Brien
                                                  --------------------------

                                            Its   Chief Financial Officer

                                     TALLA NEW YORK, INC.

                                     By      Timothy O'Brien
                                         -----------------------------------

                                     Its     Chief Financial Officer
                                         -----------------------------------

                                     SCC SPORTS CLUB, INC.

                                     By      Timothy O'Brien
                                         -----------------------------------

                                     Its     Chief Financial Officer
                                         -----------------------------------






                                     -6-
<PAGE>   6

                                     GREEN VALLEY SPECTRUM CLUB, INC.

                                     By      Timothy O'Brien
                                         -----------------------------------

                                     Its     Chief Financial Officer
                                         -----------------------------------

                                     ACKNOWLEDGED AND AGREED:

                                     HFA SERVICES, INC.


                                     By      Timothy O'Brien
                                         -----------------------------------

                                     Its     Chief Financial Officer
                                         -----------------------------------

                                     "Bank"

                                     SUMITOMO BANK OF CALIFORNIA

                                     By      /s/ Elizabeth Toda
                                         -----------------------------------

                                     Its     Assistant Vice President
                                         -----------------------------------





                                      -7-