<PAGE>   1
                                                                   EXHIBIT 10.76


                                3/16/98 BANK LOAN

<PAGE>   2
             SECOND AMENDMENT TO AMENDED AND RESTATED LOAN AGREEMENT

        THIS SECOND AMENDMENT TO AMENDED AND RESTATED LOAN AGREEMENT (this
"Amendment"), dated as of March 16, 1998, is entered into by and among THE
SPORTS CLUB COMPANY, INC., a Delaware corporation, and certain of its
subsidiaries identified in the signature pages to this Amendment (collectively,
"Borrowers"), SUMITOMO BANK OF CALIFORNIA, a California banking corporation
("Sumitomo"), COMERICA BANK -CALIFORNIA, a California banking corporation
("Comerica", and collectively with Sumitomo, "Banks"), and Sumitomo in its
capacity as agent for Banks (in such capacity, "Agent"), in light of the
following facts:

                                    RECITALS

        A. Pursuant to that certain Amended and Restated Loan Agreement, dated
as of February 2, 1998 and as amended by a First Amendment to Amended and
Restated Loan Agreement dated as of February 23, 1998 (collectively, the "Loan
Agreement"), Banks are providing Borrowers with certain credit facilities.

        B. Borrowers, Banks and Agent wish to amend the Loan Agreement to, among
other things, provide for the deferment of the termination of the Commitment and
prepayment of the Obligations.

                                    AGREEMENT

        NOW, THEREFORE, in consideration of the foregoing, and for other good
and valuable consideration, the receipt and sufficiency of which are hereby
acknowledged, the Borrowers, the Banks and the Agent hereby agree as follows:

        1. Defined Terms. All initially capitalized terms set forth without
definition in this Amendment (including, without limitation, in the recitals
hereto) shall have the respective meanings assigned thereto in the Loan
Agreement.

        2. Amendments to Definitions. Section 1.1 of the Loan Agreement is
hereby amended such that the definition set forth below shall read in full as
follows:

        "Line B Availability" means, as of each date of determination, the Line
B Commitment minus the aggregate amount of all Line B Loans made by the Banks
(including any Outstanding Standby Letters of Credit, not to exceed the Maximum
Standby Letter of Credit Amount), less any amount by which Borrowers have failed
to comply with the Liquidity Requirement and less any permanent reductions
pursuant to Section 2.2(a).

        2. Standby Letters of Credit.

                (a) Section 2.6(a) of the Loan Agreement is amended and restated
        to read as follows:
<PAGE>   3
               "(a) Subject to the terms and conditions hereof, at any time and
               from time to time from the Closing Date to the earlier of 60 days
               from the Effective Date or June 29, 1998, the Issuing Bank shall
               issue such Standby Letters of Credit as a Responsible Official of
               a Borrower on behalf of the Borrowers may request by a Request
               for Standby Letter of Credit' provided that, upon giving effect
               to such Standby Letter of Credit, (i) the Total Outstanding shall
               not exceed $15,000,000, (ii) the issuance of the Standby Letter
               of Credit shall not result in an amount which exceeds the Line B
               Availability and (iii) the Outstanding Standby Letters of Credit
               shall not exceed the Maximum Standby Letter of Credit Amount. If
               any Standby Letter of Credit is canceled or otherwise expires or
               terminates without I it being drawn upon, such cancellation,
               expiration or termination shall not effect a permanent reduction
               of the Line B Availability pursuant to Section 2.2(a) by the
               amount of such Standby Letter of Credit, but instead such amount
               may be reborrowed. Unless the Requisite Banks otherwise consent
               in writing, the term of any Standby Letter of Credit shall not
               exceed the earlier of 60 days from the Effective Date or June 30,
               1998. If, as of the immediately preceding date, or upon any
               earlier termination or acceleration of the Commitment and any
               Loans thereunder, there exist any Outstanding Standby Letters of
               Credit, Borrowers shall provide to Agent a standby letter of
               credit issued by a bank satisfactory to the Requisite Banks, in
               form and substance satisfactory to the Requisite Banks, in favor
               of the Banks in a face amount equal to Outstanding Standby
               Letters of Credit on that date, or shall make other provisions
               satisfactory to the Requisite Banks for the collateralization or
               settlement of such Outstanding Standby Letters of Credit. No
               Standby Letter of Credit shall be issued except in the ordinary
               course of business of Borrowers or their Subsidiaries. Unless
               otherwise agreed to by the Requisite Banks, the face amount of
               any Standby Letter of Credit shall not be less than $250,000."

        4. Equity or Debt Offering. Section 3.1 (f) of the Loan Agreement is
amended and restated to read as follows:

               "(f) In addition to all other payments hereunder, all
               Obligations, including payment of all indebtedness owing under
               the Notes, shall be fully due and payable upon the consummation
               of SCC, Inc.'s $50,000,000 proposed securities offering or other
               equity or debt offering in any amount by or for the benefit of
               any of the Borrowers, and the Commitment shall terminate;
               provided, however that if (i) the proposed $50,000,000 equity
               offering is consummated on or before April 30, 1998 and (ii)
               within 5 days thereafter, Borrower shall have complied with the
               conditions precedent to this Second Amendment, Banks will defer
               the acceleration of all Obligations pursuant to this Section
               3.1(f) for a period of 60 days following the consummation of such
               offering but in any event not later than June 30,1998.
               Consummation of the $50,000,000 equity offering shall mean SCC,
               Inc.'s successfully concluding the sale of a sufficient number of
               shares of common stock to raise, after deduction for normal
               expenses, not less than $40 million with such proceeds
               irrevocable delivered to SCC, Inc.
<PAGE>   4

        5. Indebtedness, Guaranties and Liens. Section 6.9(c) of the Loan
Agreement is amended and restated to read as follows:

              "(c) Indebtedness and Liens securing obligations incurred in
              connection with the financing or re-financing of any real property
              assets of Borrowers or Non-Borrower Affiliates, to the extent such
              financing has been approved by the Requisite Banks, or (ii)
              incurred in connection with SCC Inc.'s $50,000,000 proposed
              securities offering or other equity or debt offering in any amount
              by or for the benefit of any of the Borrowers, to the extent
              approved by the Requisite Banks."

        6. Conditions Precedent. The effectiveness of Section 4 of this
Amendment ("Effective Date") is subject to the prior satisfaction of each of the
following conditions:

               (a) all obligations owing by any Borrower to AT&T Commercial
        shall have been fully paid, all collateral for such AT&T obligations
        shall have been released, all security interests, Liens or Guaranties in
        favor of AT&T shall have been terminated and the Special Deposit Account
        Agreement and Subordination Agreement shall have been terminated.

        7. Representations and Warranties. Each representation and warranty made
by the Borrowers in Article 4 of the Loan Agreement is true and correct on and
as of the date hereof as though made as of the date hereof, except to the extent
such representations and warranties relate solely to an earlier date.

        8. Full Force and Effect. Each of the Loan Documents is hereby amended
such that all references to the Loan Agreement contained in any of such
documents shall be deemed to be made with respect to the Loan Agreement as
amended by this Amendment. Except as amended hereby, the Loan Agreement and the
other Loan Documents shall remain unaltered and in full force and effect.

        9. Counterparts. This Amendment may be executed in multiple
counterparts, each of which shall constitute an original and all of which, taken
together, shall constitute but one and the same instrument.

<PAGE>   5

        IN WITNESS WHEREOF, the parties hereto have executed this Amendment by
their respective duly authorized officers as of the date first above written.

                                        The "Borrowers"

                                        THE SPORTS CLUB COMPANY, INC.,
                                        a Delaware corporation

                                        By  /s/ Timothy O'Brien
                                            ------------------------------------
                                            Timothy O'Brien
                                            Chief Financial Officer

                                        THE SPECTRUM CLUB COMPANY, INC.,
                                        a California corporation

                                        By  /s/ Timothy O'Brien
                                            ------------------------------------
                                            Timothy O'Brien
                                            Chief Financial Officer

                                        PONTIUS REALTY, INC.,
                                        a New York corporation

                                        By  /s/ Timothy O'Brien
                                            ------------------------------------
                                            Timothy O'Brien
                                            Chief Financial Officer

                                        SPORTS CLUB, INC. OF CALIFORNIA,
                                        a California corporation

                                        By  /s/ Timothy O'Brien
                                            ------------------------------------
                                            Timothy O'Brien
                                            Chief Financial Officer

<PAGE>   6

                                        IRVINE SPORTS CLUB, INC.,
                                        a California corporation

                                        By  /s/ Timothy O'Brien
                                            ------------------------------------
                                            Timothy O'Brien
                                            Chief Financial Officer

                                        THE SPORTSMED COMPANY, INC.,
                                        a California corporation

                                        By  /s/ Timothy O'Brien
                                            ------------------------------------
                                            Timothy O'Brien
                                            Chief Financial Officer

                                        L.A./IRVINE SPORTS CLUB,LTD.,
                                        a California limited partnership

                                        By  /s/ Timothy O'Brien
                                            ------------------------------------
                                            Timothy O'Brien
                                            Chief Financial Officer

                                        TALLA NEW YORK, INC.
                                        a New York corporation

                                        By  /s/ Timothy O'Brien
                                            ------------------------------------
                                            Timothy O'Brien
                                            Chief Financial Officer

                                        SCC SPORTS CLUB, INC.,
                                        a Texas corporation

                                        By  /s/ Timothy O'Brien
                                            ------------------------------------
                                            Timothy O'Brien
                                            Chief Financial Officer
<PAGE>   7
                                        GREEN VALLEY SPECTRUM CLUB, INC.,
                                        a Nevada corporation

                                        By  /s/ Timothy O'Brien
                                            ------------------------------------
                                            Timothy O'Brien
                                            Chief Financial Officer

                                        SPECTRUM CLUB/ANAHEIM HILLS, INC.,
                                        a California corporation

                                        By  /s/ Timothy O'Brien
                                            ------------------------------------
                                            Timothy O'Brien
                                            Chief Financial Officer

                                        The "Agent"

                                        SUMITOMO BANK OF CALIFORNIA,
                                        a California banking corporation

                                        By  /s/ Noel R. Ryan, Jr.
                                            ------------------------------------
                                            Noel R. Ryan, Jr.
                                            Senior Vice President

                                        The "Banks"

                                        SUMITOMO BANK OF CALIFORNIA,
                                        a California banking corporation

                                        By  /s/ Noel R. Ryan, Jr.
                                            ------------------------------------
                                            Noel R. Ryan, Jr.
                                            Senior Vice President

<PAGE>   8
                                        COMERICA BANK - CALIFORNIA,
                                        A California banking corporation

                                        By  /s/ Joseph Yurosek
                                            ------------------------------------
                                            Joseph Yurosek
                                            Vice President

                                        Address:

                                        Comerica Bank - California
                                        301 E. Ocean Boulevard, Suite 1800
                                        Long Beach, California 90802
                                        Attn: Joseph Yurosek, Vice President

                                        Telecopier: (562) 595-8251
                                        Telephone: (562) 590-2530