<PAGE>

                                                                   EXHIBIT 10.40

                                 LIMITED WAIVER
                TO FOURTH AMENDED AND RESTATED CREDIT AGREEMENT


                                                                 August 31, 1999

Granite Broadcasting Corporation
767 Third Avenue, 34th Floor
New York, New York 10017

Ladies and Gentlemen:

          This Limited Waiver (this "Limited Waiver") is dated as of August 31,
1999 and is made with reference to that certain Fourth Amended and Restated
Credit Agreement dated as of June 10, 1998, by and among Granite Broadcasting
Corporation (the "Company"), the lenders party thereto ("Lenders"), Bankers
Trust Company, as Administrative Agent ("Administrative Agent"), The Bank of New
York, as Documentation Agent, and Goldman Sachs Credit Partners L.P., Union Bank
of California, N.A., and ABN Amro Bank N.V., as Co-Agents, as amended by that
certain First Amendment to Fourth Amended and Restated Credit Agreement dated as
of March 23, 1999 (as so amended, the "Credit Agreement").  Capitalized terms
used herein without definition shall have the meanings assigned to such terms in
the Credit Agreement.

          Pursuant to that certain Purchase and Sale Agreement dated as of April
28, 1999, by and among CBS Corporation, a Pennsylvania corporation, Company,
KBVO, Inc. ("KBVO") and KBVO License, Inc. ("KBVO License"), Company, KBVO and
KBVO License have agreed to sell television station KEYE-TV (formerly KBVO-TV)
serving Austin, Texas and the assets relating thereto (the "KEYE Sale") and the
closing of such transaction is expected to occur on or about August 31, 1999.

          Company has informed Lenders that it intends to prepay all outstanding
Loans with the Net Cash Proceeds of the KEYE Sale and to use the Net Cash
Proceeds remaining after such prepayment (the "Excess Proceeds") to redeem or
repurchase a portion of its Subordinated Indebtedness in compliance with the
terms of the Credit Agreement and the indentures pursuant to which such
Subordinated Indebtedness was issued.

          Company has also informed the Lenders that it does not expect to be
able to comply with the provisions of Subsection 7.6A (Maximum Total Debt Ratio)
until it has been able to effect such redemptions and has requested the waiver
of Subsections 7.6A, 7.5(iv) and 7.7 of the Credit Agreement on the terms and
conditions set forth below.

          By their execution of a counterpart of this Limited Waiver, Requisite
Lenders agree to waive the provisions of Subsection 7.6A of the Credit Agreement
and the provisions of Subsections 7.7 and 7.5(iv) of the Credit Agreement
requiring compliance with the provisions of
<PAGE>

Subsection 7.6 and Subsections 7.5(iv)(b) and 7.5(iv)(f)(y), respectively, on a
Pro Forma Basis on the closing of the KEYE Sale and on any redemption or
repurchase of Convertible Preferred Stock; provided, that (i) Company prepays
                                           --------
all Loans outstanding on the date of the closing of the KEYE Sale with the Net
Cash Proceeds thereof, (ii) Company would be in compliance with the provisions
of Subsection 7.6A if pro forma effect were given to the redemption of
Subordinated Indebtedness in an amount equal to the Excess Proceeds after giving
effect to any redemptions and repurchases thereof after the date of the KEYE
Sale, and (iii) the Excess Proceeds are, to the extent required by Subsection
2.4B(iii)(a) of the Credit Agreement, deposited in a collateral account
maintained with the Administrative Agent; provided, further, that the waiver of
                                          --------  -------
Subsection 7.6A set forth herein shall remain in effect until the earlier of (x)
the date when the proposed redemption of Subordinated Indebtedness is effected
in accordance with the terms of the Credit Agreement or (y) the date which is
170 days after the date of the closing of the KEYE Sale.

          Without limiting the generality of the provisions of subsection 10.6
of the Credit Agreement, the waiver set forth herein shall be limited precisely
as written and relate solely to compliance the provisions of Subsection 7.6A of
the Credit Agreement and the provisions of Subsections 7.7 and 7.5(iv) of the
Credit Agreement requiring compliance with the provisions of Subsection 7.6 and
Subsections 7.5(iv)(b) and 7.5(iv)(f)(y), respectively, on a Pro Forma Basis on
the closing of the KEYE Sale and on any redemption or repurchase of Convertible
Preferred Stock for the time period set forth above.  Nothing in this Limited
Waiver shall be deemed to (a) constitute a waiver of any other term, provision
or condition of the Credit Agreement or any other instrument or agreement
referred to therein, (b) prejudice any right or remedy that Administrative Agent
may now have or may have in the future under or in connection with the Credit
Agreement or any other instrument or agreement referred to therein or (c) create
any obligation on the part of Administrative Agent to renew or extend the waiver
contained herein.  Except as expressly set forth herein, the terms, provisions
and conditions of the Credit Agreement and the other Loan Documents shall remain
in full force and effect and in all other respects are hereby ratified and
confirmed.

          Each Subsidiary of Company hereby acknowledges that it has read this
Limited Waiver and consents to the terms thereof and further hereby confirms and
agrees that, notwithstanding the effectiveness of this Limited Waiver, the
obligations of such Subsidiary under the Subsidiary Guaranty shall not be
impaired or affected and the Subsidiary Guaranty is, and shall continue to be,
in full force and effect and is hereby confirmed and ratified in all respects.

          In order to induce the Administrative Agent and Requisite Lenders to
enter into this Limited Waiver, the Company by its execution of a counterpart of
this Limited Waiver, represents and warrants that (a) no Event of Default or
Potential Event of Default exists under the Credit Agreement or the other Loan
Documents after giving effect to the waiver contemplated in this Limited Waiver,
(b) all representations and warranties contained in the Credit Agreement and the
other Loan Documents are true and correct in all material respects on and as of
the date hereof except to the extent such representations and warranties
specifically relate to an earlier date, in which case they were true and correct
in all material respects on and as of such earlier date, and (c) the Company and
its Subsidiaries have performed all agreements to be performed on their part as
set forth in the Credit Agreement.

                                       2
<PAGE>

          This Limited Waiver may be executed in any number of counterparts and
by different parties hereto in separate counterparts, each of which when so
executed and delivered shall be deemed an original, but all such counterparts
together shall constitute but one and the same instrument.  This Limited Waiver
shall become effective as of the date hereof upon the execution of counterparts
hereof duly executed by Company, its Subsidiaries and Requisite Lenders or, in
the case of any Lender, telecopy or telephone confirmation from such Lender of
its execution hereof. Company acknowledges that all costs, fees and expenses as
described in subsection 10.2 of the Credit Agreement incurred by Administrative
Agent and its counsel with respect to this Limited Waiver and the documents and
transactions contemplated hereby shall be for the account of Company.  THIS
LIMITED WAIVER SHALL BE GOVERNED BY, AND SHALL BE CONSTRUED AND ENFORCED IN
ACCORDANCE WITH, THE INTERNAL LAWS OF THE STATE OF NEW YORK, WITHOUT REGARD TO
CONFLICTS OF LAWS PRINCIPLES.

                  [Remainder of page intentionally left blank]

                                       3
<PAGE>

                              ADMINISTRATIVE AGENT:

                              BANKERS TRUST COMPANY, individually and as
                              Administrative Agent and Collateral Agent

                              By:   /s/ Patricia Hogan
                                    ------------------------------------
                                    Patricia Hogan
                                    Principal



ACKNOWLEDGED AND APPROVED BY:

BORROWER:

GRANITE BROADCASTING CORPORATION

By:  /s/ Lawrence I. Wills
     ---------------------------
     Lawrence I. Wills
     Vice President

                                       4
<PAGE>

SUBSIDIARIES:

GRANITE RESPONSE TELEVISION, INC.
KBVO, INC.
KBVO LICENSE, INC.
KNTV, INC.
KNTV LICENSE, INC.
RJR COMMUNICATIONS, INC.
KBJR LICENSE, INC.
SAN JOAQUIN COMMUNICATIONS CORPORATION
KSEE LICENSE, INC.,
WPTA-TV, INC.
WPTA-TV LICENSE, INC.
WTVH, INC.
WTVH LICENSE, INC.
WWMT-TV, INC.
WWMT-TV LICENSE, INC.
WKBW-TV, INC.
WKBW-TV LICENSE, INC.
QUEEN CITY BROADCASTING, INC.
QUEEN CITY BROADCASTING OF NEW YORK, INC.
WEEK, INC.
WEEK LICENSE, INC.
WXON, INC.
WXON LICENSE, INC.
WLAJ, INC.
WLAJ LICENSE, INC.
WEEK-TV LICENSE, INC.
PACIFIC FM INCORPORATED
KOFY-TV LICENSE, INC.

By:  /s/ Lawrence I. Wills
     -----------------------------
     Lawrence I. Wills
     Vice President

LENDERS:

THE BANK OF NEW YORK,
as Documentation Agent and a Lender

By:  _________________________________________
     Name:_______________________________
     Title:_______________________________

                                       5
<PAGE>

GOLDMAN SACHS CREDIT PARTNERS L.P.,
as a Co-Agent and a Lender

By:  _________________________________________
     Name:______________________________
     Title:______________________________


UNION BANK OF CALIFORNIA, N.A.,
as a Co-Agent and a Lender

By:  _________________________________________
     Name:_______________________________
     Title:_______________________________

ABN AMRO BANK N.V., NEW YORK BRANCH,
as a Co-Agent and a Lender

By:  _________________________________________
     Name:_______________________________
     Title:_______________________________

NATEXIS BANQUE BFCE,
as a Lender

By:  _________________________________________
     Name:_______________________________
     Title:_______________________________


COMPAGNIE FINANCIERE DE CIC ET DE
L'UNION EUROPEENNE,
as a Lender

By:  _________________________________________
     Name:_______________________________
     Title:_______________________________


By:  _________________________________________
     Name:_______________________________
     Title:_______________________________

                                       6
<PAGE>

HELLER FINANCIAL, INC.,
as a Lender

By:  _________________________________________
     Name:_______________________________
     Title:_______________________________

PARIBAS,
as a Lender



By:  _________________________________________
     Name:_______________________________
     Title:_______________________________

By:  _________________________________________
     Name:_______________________________
     Title:_______________________________



THE BANK OF NOVA SCOTIA,
as a Lender

By:  _________________________________________
     Name:_______________________________
     Title:_______________________________

BANQUE NATIONALE DE PARIS,
as a Lender



By:  _________________________________________
     Name:_______________________________
     Title:_______________________________

By:  _________________________________________
     Name:_______________________________
     Title:_______________________________



                                       7
<PAGE>

MELLON BANK, N.A.,
as a Lender



By:  _________________________________________
     Name:_______________________________
     Title:_______________________________


BANK OF TOKYO-MITSUBISHI TRUST,
as a Lender



By:  _________________________________________
     Name:_______________________________
     Title:_______________________________

FREMONT FINANCIAL CORPORATION,
as a Lender



By:  _________________________________________
     Name:_______________________________
     Title:_______________________________

SOUTHERN PACIFIC BANK,
as a Lender



By:  _________________________________________
     Name:_______________________________
     Title:_______________________________

                                       8