<PAGE>   1
                                                                    EXHIBIT 4.1


                         HORSESHOE GAMING HOLDING CORP.
                               FOURTH AMENDMENT TO
                                CREDIT AGREEMENT
                                   AND CONSENT


        This FOURTH AMENDMENT TO CREDIT AGREEMENT AND CONSENT (this "Amendment
and Consent") is dated as of July 20, 2001 and entered into by and among
HORSESHOE GAMING HOLDING CORP., a Delaware corporation ("Company"), the
financial institutions listed on the signature pages hereof ("Lenders"), CREDIT
SUISSE FIRST BOSTON as successor to DLJ Capital Funding, Inc., as Syndication
Agent ("Syndication Agent") and CANADIAN IMPERIAL BANK OF COMMERCE, as
Administrative Agent ("Administrative Agent" and together with the Syndication
Agent, "Agents"), and is made with reference to that certain Credit Agreement
dated as of June 30, 1999 (as amended by the First Amendment, Second Amendment
and Third Amendment described below, the "Credit Agreement"), by and among
Company, Lenders, Syndication Agent and Administrative Agent, as amended by that
certain First Amendment to Credit Agreement dated as of November 18, 1999, that
certain Second Amendment to Credit Agreement dated as of November 30, 1999 and
that certain Third Amendment to Credit Agreement dated as of January 20, 2000.
Capitalized terms used herein without definition shall have the same meanings as
set forth in the Credit Agreement.

                                    RECITALS

        WHEREAS, Company has notified Agents that it desires to consummate the
sale (the "Joliet Sale") of all of the capital stock of its wholly owned
subsidiary, Empress Joliet, for estimated net proceeds of $420,000,000 in
accordance with the terms of that certain Agreement and Plan of Merger dated as
of April 12, 2001 (the "Joliet Sale Agreement"), by and among Argosy Gaming
Company, Joliet Acquisition Corporation, Empress Joliet and Company; and

        WHEREAS, Company has notified Agents that it desires to apply the
estimated net proceeds of the Joliet Sale as follows: (a) to prepay the Tranche
B Term Loans in full; (b) to prepay $100,000,000 of the Revolving Loans and
correspondingly reduce the Revolving Loan Commitments in an equal amount and to
reduce the scheduled reductions thereof in forward order of maturity; (c) to
redeem up to $155,700,000 of the New Sub Debt and up to $41,600,000 of the
Existing Subordinated Horseshoe Bonds in accordance with the terms of offers to
redeem such notes as provided in the New Sub Debt Indenture and the Existing
Subordinated Horseshoe Indenture (collectively, the "Asset Sale Offer"); and (d)
to retain any remainder of such proceeds (including the amounts referred to in
(c) to the extent the New Sub Debt and the Existing Subordinated Horseshoe Bonds
are not tendered for redemption or not included in such offers to redeem) for
general corporate purposes or as otherwise permitted by the Credit Agreement
(the application of proceeds set forth in this "Whereas" clause being referred
to herein as the "Application of Joliet Sale Proceeds"); and



<PAGE>   2

        WHEREAS, Company, Lenders and Agents desire to amend the Credit
Agreement to permit the Joliet Sale as long as the proceeds thereof are applied
as set forth herein, including pursuant to the Asset Sale Offer; and

        WHEREAS, Company, Lenders and Agents desire to further amend the Credit
Agreement to permit the Company to make a Restricted Junior Payment to
repurchase all Existing Subordinated Horseshoe Bonds not redeemed in the Asset
Sale Offer and to make certain other requested modifications of the Credit
Agreement.

        NOW THEREFORE, in consideration of the premises and the agreements,
provisions and covenants herein contained, the parties hereto agree as follows:

        SECTION 1. AMENDMENTS TO THE CREDIT AGREEMENT

        1.1 DEFINITIONS

        A. ADDITIONAL DEFINITION. Section 1.1 of the Credit Agreement is amended
by inserting the following definition in alphabetical order:

        "`ASSET SALE OFFER' means the offers to purchase up to $155,700,000 of
New Sub Debt and up to $41,600,000 Existing Subordinated Horseshoe Bonds with
the proceeds of the sale of Empress Joliet on terms as provided in the New Sub
Debt Indenture and the Existing Subordinated Horseshoe Indenture."

        B. MODIFIED DEFINITIONS. Section 1.1 of the Credit Agreement is amended
to modify the definitions of "Consolidated EBITDA" and "Consolidated Leverage
Ratio" as follows:

        1. "CONSOLIDATED EBITDA" is hereby amended to delete the proviso
contained therein in its entirety.

        2. "CONSOLIDATED LEVERAGE RATIO" is hereby amended to add the following
proviso at the end of the existing definition:

        "; provided that all components of Consolidated EBITDA for the period of
four Fiscal Quarters ending at the end of each Fiscal Quarter shall include or
exclude, as the case may be, without duplication, such components of
Consolidated EBITDA attributable to any business or assets that have been
acquired or disposed of by Company or any of its Subsidiaries (including
businesses and assets relating to the sale of Empress Joliet and through mergers
or consolidations but excluding Excluded Asset Sales) after the first day of
such period of four Fiscal Quarters and prior to the end of such period, as
determined in good faith by Company on a pro forma basis for such period of four
Fiscal Quarters as if such acquisition or disposition had occurred on such first
day of such period."

        1.2 SCHEDULED REDUCTIONS OF REVOLVING LOAN COMMITMENTS. The table set
        forth in Section 2.4B of the Credit Agreement is amended to read in its
        entirety as follows:


<PAGE>   3


<TABLE>
<CAPTION>
                                                                SCHEDULED REDUCTION OF
          DATE                                               REVOLVING LOAN COMMITMENTS
          ----                                               --------------------------
<S>                                                                <C>
          December 31, 2003                                        $ 28,125,000
          March 31, 2004                                             40,625,000
          June 30, 2004                                              40,625,000
          September 30, 2004                                         40,625,000
                                                                   ------------
                                                                   $150,000,000
</TABLE>

        1.2 RESTRICTED JUNIOR PAYMENTS. Section 7.5A is amended by deleting the
"and" before clause (v), adding a comma before clause (v) and adding the
following at the end of such section:

        ", (vi) consummate the Asset Sale Offer and (vii) repurchase any and all
Existing Subordinated Horseshoe Bonds not redeemed in the Asset Sale Offer."

        1.3 OPERATING LEASES. Section 7.9 is amended by deleting the existing
section and substituting the following new provision to read in its entirety as
follows:

        "7.9 RESTRICTION OF LEASES.

        Company shall not, and shall not permit any of its Subsidiaries to,
become liable in any way, whether directly or by assignment or as a guarantor or
other surety, for the obligations of the lessee under any Operating Lease (other
than intercompany leases between Company and its wholly-owned Subsidiaries);
unless, immediately after giving effect to the incurrence of liability with
respect to such lease, the Consolidated Operating Lease Rental Payments shall
not exceed $10,000,000 (excluding escalations resulting from a rise in the
consumer price or similar index) for any Fiscal Year; provided, however, that
any calculations made for purposes of this subsection 7.9 shall exclude any
amounts required to be expended for maintenance and repairs, insurance, taxes,
assessments and other similar charges and, provided, further, that any
calculations made for purposes of this subsection 7.9 shall exclude any rental
payments made with respect to leased property disposed of during such Fiscal
Year."

        SECTION 2. CONSENT

        2.1 CONSENT

        A. Subject to the terms and conditions set forth herein and in reliance
on the representations and warranties of the Company herein contained, Lenders
(a) hereby consent effective as of the Fourth Amendment Effective Date (as
defined below) to (i) the Joliet Sale, (ii) the Asset Sale Offer and (iii) the
Application of Joliet Sale Proceeds and (b) hereby waive compliance with the
provisions of (i) subsection 2.4C(iii)(a) (to the extent the Application of
Joliet Sale Proceeds would result in Net Asset Sale Proceeds of the Joliet Sale
being applied other than to prepay Revolving Loans and reduce Revolving Loan
Commitments under the Credit Agreement in excess of $100,000,000), (ii)
subsection 2.4C(iv)(e) (to the extent it would


<PAGE>   4

require the reduction of Revolving Loan Commitments on a pro rata basis rather
than in forward order of maturity), and (iii) sections 7.7 and 7.12 to the
extent, and only to the extent, the Joliet Sale, the Asset Sale Offer or the
Application of Joliet Sale Proceeds would otherwise be prohibited under the
sections set forth in clauses (b)(i), (ii) and (iii).

        B. LIMITATION OF CONSENT. Without limiting the generality of the
provisions of subsection 10.6 of the Credit Agreement, the consent and waiver
set forth in this Section 2 shall be limited precisely as written and is
provided solely with respect to the Joliet Sale, Asset Sale Offer and
Application of Joliet Sale Proceeds and the noncompliance by Company with
subsections 2.4C(iii)(a), 2.4C(iv)(e), 7.7 and 7.12 in the manner and to the
extent described above, and nothing in this Amendment and Consent shall be
deemed to (a) constitute a waiver of compliance by Company with respect to (i)
subsections 2.4C(iii)(a), 2.4C(iv)(e), 7.7 and 7.12 of the Credit Agreement in
any other instance or (ii) any other term, provision or condition of the Credit
Agreement or any other instrument or agreement referred to therein or (b)
prejudice any right or remedy that Agents or any Lender may now have or may have
in the future under or in connection with the Credit Agreement or any other
instrument or agreement referred to therein. Except as expressly set forth
herein, the terms, provisions and conditions of the Credit Agreement and the
other Loan Documents shall remain in full force and effect and in all other
respects are hereby ratified and confirmed.

        SECTION 3. COMPANY'S REPRESENTATIONS AND WARRANTIES

        In order to induce Lenders and Agents to enter into this Amendment and
Consent and to amend the Credit Agreement

        in the manner provided herein, Company represents and warrants to each
Lender that the following statements are true, correct and complete:

        A. CORPORATE POWER AND AUTHORITY. Each of Company and Subsidiary
Guarantors that are parties hereto have all requisite corporate power and
authority to enter into this Amendment and Consent and each of Company and
Empress Joliet have all requisite corporate power and authority to enter into
the Joliet Sale Agreement and to carry out the transactions contemplated by, and
perform its obligations under, the Credit Agreement as amended by this Amendment
and Consent (the "Amended Agreement") and, in the case of Company and Empress
Joliet, the Joliet Sale Agreement.

        B. AUTHORIZATION OF AGREEMENTS. The execution and delivery of this
Amendment and Consent and the Joliet Sale Agreement and the performance of the
Amended Agreement and the Joliet Sale Agreement have been duly authorized by all
necessary corporate action on the part of Company, Empress Joliet and Subsidiary
Guarantors party thereto.

        C. BINDING OBLIGATION. This Amendment and Consent and the Amended
Agreement and the Joliet Sale Agreement have been duly executed and delivered by
Company, Empress Joliet and Subsidiary Guarantors party thereto, and are the
legally valid and binding obligations of Company, Empress Joliet and Subsidiary
Guarantors party thereto, enforceable against Company and Subsidiary Guarantors
party thereto, in accordance with their respective terms, except as may be
limited by bankruptcy, insolvency, reorganization, moratorium or


<PAGE>   5

similar laws relating to or limiting creditors' rights generally or by equitable
principles relating to enforceability.

        D. NO VIOLATION. Company and Empress Joliet are not, and to Company's
best knowledge no other party to the Joliet Sale Agreement is, in violation of
any material term of the Joliet Sale Agreement.

        E. INCORPORATION OF REPRESENTATIONS AND WARRANTIES FROM CREDIT
AGREEMENT. The representations and warranties contained in Section 5 of the
Credit Agreement are and will be true, correct and complete in all material
respects on and as of the date hereof to the same extent as though made on and
as of that date, except to the extent such representations and warranties
specifically relate to an earlier date, in which case they were true, correct
and complete in all material respects on and as of such earlier date.

        F. ABSENCE OF DEFAULT. No event has occurred and is continuing or will
result from the consummation of the transactions contemplated by this Amendment
and Consent that would constitute an Event of Default or Potential Event of
Default.

        G. NO CONFLICT. The execution and delivery by Company and Subsidiary
Guarantors of this Amendment and Consent and the performance by Company of its
obligations under the Amended Agreement, and the execution and delivery by
Company and Empress Joliet of the Joliet Sale Agreement and the performance by
Company and Empress Joliet of their obligations under the Joliet Sale Agreement
do not and will not (i) violate any provision of any law or governmental rule or
regulation applicable to Company or any of its Subsidiaries, the Certificate or
Articles of Incorporation or Bylaws of Company or any of its Subsidiaries or any
of its Subsidiaries, (ii) conflict with, result in a breach of or constitute
(with due notice or lapse of time or both) a default under any Material Contract
of Company or any of its Subsidiaries, (iii) result in or require the creation
or imposition of any Lien upon any of the properties or assets of Company or any
of its Subsidiaries (other than Liens created under any of the Loan Documents in
favor of Administrative Agent on behalf of Lenders), or (iv) require any
approval of stockholders or any approval or consent of any Person under any
Material Contract of Company or any of its Subsidiaries, except for such
approvals or consents which will be obtained on or before the Fourth Amendment
and Consent Effective Date and disclosed in the disclosure schedules to the
Joliet Sale Agreement provided to Agents.

        H. GOVERNMENTAL CONSENTS. The execution and delivery by Company and
Subsidiary Guarantors of this Amendment and Consent and the performance by
Company of its obligations under the Amended Agreement and the execution and
delivery by Company and Empress Joliet of, and the performance of their
obligations under, the Joliet Sale Agreement do not and will not require any
registration with, consent or approval of, or notice to, or other action to,
with or by, any federal, state or other governmental authority or regulatory
body (other than security filings or recordings required by the transactions
contemplated hereunder), except for such registrations, consents, approvals,
notices or other actions which will be obtained or made on or before the Fourth
Amendment and Consent Effective Date and disclosed in the disclosure schedules
to the Joliet Sale Agreement provided to Agents.


<PAGE>   6

        SECTION 4. CONDITIONS TO EFFECTIVENESS

        Sections 1 and 2 of this Amendment and Consent shall become effective
only upon satisfaction of all of the following conditions precedent (the date of
satisfaction of such conditions, which shall be specified in a written notice
from Administrative Agent to Company and Lenders, being referred to herein as
the "Fourth Amendment Effective Date"):

        A. EVIDENCE OF CONSUMMATION OF JOLIET SALE. On or before the Fourth
Amendment Effective Date, Company shall deliver to Lenders evidence reasonably
satisfactory to Agents of the consummation of the Joliet Sale and the repayment
in full of the Tranche B Term Loans and prepayment of $100,000,000 in Revolving
Loans and corresponding permanent reduction of the Revolving Loan Commitments
shall have been made concurrently therewith.

        B. EXECUTION OF AMENDMENT AND CONSENT. On or before the Fourth Amendment
Effective Date, Company, Subsidiary Guarantors and Revolving Lenders with an
aggregate of $76,500,000 in Revolving Loan Commitments (which after giving
effect to this Amendment and Consent shall constitute Requisite Lenders) shall
have executed and delivered copies of this Amendment and Consent to
Administrative Agent.

        C. FEES.

        (i) On or before the Fourth Amendment Effective Date, Company shall have
    paid amendment fees as agreed among the Agents and Company.

        (ii) The Borrower shall have paid to the Administrative Agent all fees
    which are due and payable to the Administrative Agent under the Credit
    Agreement as amended.

        D. OTHER PROCEEDINGS. On or before the Fourth Amendment Effective Date,
all corporate and other proceedings taken or to be taken in connection with the
transactions contemplated hereby and all documents incidental thereto not
previously found acceptable by Agents, acting on behalf of Lenders, shall be
reasonably satisfactory in form and substance to Agents, and Agents shall have
received all such counterpart originals or certified copies of such documents as
Agents may reasonably request.

        SECTION 5. MISCELLANEOUS

        A. RELEASE OF COLLATERAL. Lenders hereby instruct Agents to release the
Collateral pledged by Empress Joliet, and to release the lien on Company's
shares in Empress Joliet, upon the effectiveness of this Amendment and Consent.

        B. REFERENCE TO AND EFFECT ON THE CREDIT AGREEMENT AND THE OTHER LOAN
DOCUMENTS.

        (i) On and after the Fourth Amendment Effective Date, each reference in
    the Credit Agreement to "this Agreement", "hereunder", "hereof", "herein" or
    words of like import referring to the Credit Agreement, and each reference
    in the other Loan


<PAGE>   7

    Documents to the "Credit Agreement", "thereunder", "thereof" or words of
    like import referring to the Credit Agreement shall mean and be a reference
    to the Amended Agreement.

        (ii) Except as specifically amended by this Amendment and Consent, the
    Credit Agreement and the other Loan Documents shall remain in full force and
    effect and are hereby ratified and confirmed.

        (iii) The execution, delivery and performance of this Amendment and
    Consent shall not, except as expressly provided herein, constitute a waiver
    of any provision of, or operate as a waiver of any right, power or remedy of
    either Agent or any Lender under, the Credit Agreement or any of the other
    Loan Documents.

        C. HEADINGS. Section and subsection headings in this Amendment and
Consent are included herein for convenience of reference only and shall not
constitute a part of this Amendment and Consent for any other purpose or be
given any substantive effect.

        D. APPLICABLE LAW. THIS AMENDMENT AND CONSENT AND THE RIGHTS AND
OBLIGATIONS OF THE PARTIES HEREUNDER SHALL BE GOVERNED BY, AND SHALL BE
CONSTRUED IN ACCORDANCE WITH, THE INTERNAL LAWS OF THE STATE OF NEW YORK
(INCLUDING WITHOUT LIMITATION SECTION 5-1401 OF THE GENERAL OBLIGATIONS LAW OF
THE STATE OF NEW YORK), WITHOUT REGARD TO CONFLICTS OF LAWS PRINCIPLES.

        E. COUNTERPARTS. This Amendment and Consent may be executed in any
number of counterparts and by different parties hereto in separate counterparts,
each of which when so executed and delivered shall be deemed and original, but
all such counterparts together shall constitute but one and the same instrument;
signature pages may be detached from multiple separate counterparts and attached
to a single counterpart so that all signature pages are physically attached to
the same document.


<PAGE>   8

        IN WITNESS WHEREOF, the parties hereto have caused this Amendment and
Consent to be duly executed and delivered by their respective officers thereunto
duly authorized as of the date first written above.


                                          HORSESHOE GAMING HOLDING CORP.


                                          By:    /s/ Kirk C. Saylor
                                                 ------------------------------
                                          Name:  Kirk C. Saylor
                                          Title: Chief Financial Officer


<PAGE>   9

        By its execution of a counterpart of this Amendment and Consent, each of
the undersigned, as party to the Subsidiary Guaranty and/or the Collateral
Documents, hereby acknowledges that it has read this Amendment and Consent and
consents to the terms thereof and further hereby confirms and agrees that,
notwithstanding the effectiveness of this Amendment and Consent, any and all
obligations of the undersigned under the Subsidiary Guaranty and the Collateral
Documents shall not be impaired or affected and the Subsidiary Guaranty and the
Collateral Documents are, and shall continue to be, in full force and effect and
are hereby confirmed and ratified in all respects.


BOSSIER CITY LAND CORPORATION                  HORSESHOE GP, INC.

By:    /s/ Roger P. Wagner                     By:    /s/ Roger P. Wagner
       ----------------------------                   -------------------------
Name:  Roger P. Wagner                         Name:  Roger P. Wagner
Title: President                               Title: President


HORSESHOE HAMMOND, INC.                        NEW GAMING CAPITAL PARTNERSHIP

By:    /s/ Roger P. Wagner                     By:    Horseshoe GP, Inc.,
       ----------------------------                   its General Partner
Name:  Roger P. Wagner                         By:    /s/ Roger P. Wagner
Title: President                                      -------------------------
                                               Name:  Roger P. Wagner
                                               Title: President


HORSESHOE ENTERTAINMENT                        ROBINSON PROPERTY GROUP,
                                               LIMITED PARTNERSHIP

By:    New Gaming Capital Partnership,         By:    Horseshoe GP, Inc.,
       its General Partner                            its General Partner
                                               By:    /s/ Roger P. Wagner
                                                      -------------------------
By:    Horseshoe GP, Inc.,                     Name:  Roger P. Wagner
       its General Partner                     Title: President

By:    /s/ Roger P. Wagner
       ----------------------------
Name:  Roger P. Wagner
Title: President


<PAGE>   10


LENDERS:

CREDIT SUISSE FIRST BOSTON,
as successor to DLJ Capital Funding,
individually and as Syndication Agent

By:    /s/ Mark E. Gleason
       ----------------------------
Name:  Mark E. Gleason
Title: Director

By:    /s/ William S. Lutkins
       ----------------------------
Name:  William S. Lutkins
Title: Vice President


CIBC INC
By:    /s/ Paul J. Chakmak
       ----------------------------
Name:  Paul J. Chakmak
Title: Managing Director
CIBC World Markets Inc., AS AGENT


BANK OF SCOTLAND
By:    /s/ Joseph Fratus
       ----------------------------
Name:  Joseph Fratus
Title: Vice President


BANK OF HAWAII
By:    /s/ J. Bryan Scearce
       ----------------------------
Name:  J. Bryan Scearce
Title: Vice President


IMPERIAL BANK
By:    /s/ Eoin Collins
       ----------------------------
Name:  Eoin Collins
Title: Vice President


COMERICA WEST INCORPORATED
By:    /s/ Eoin Collins
       ----------------------------
Name:  Eoin Collins
Title:    Vice President


TRUSTMARK NATIONAL BANK
By:    /s/ Craig E. Sosebee
       ----------------------------
Name:  Craig E. Sosebee
Title: Vice President


CANADIAN IMPERIAL BANK OF COMMERCE,
individually and as Administrative Agent
By:    /s/ Paul J. Chakmak
       ----------------------------
Name:  Paul J. Chakmak
Title: Managing Director
CIBC World Markets Inc., AS AGENT


U.S. BANK NATIONAL ASSOCIATION
By:    /s/ David Walquist
       ----------------------------
Name:  David Walquist
Title: Vice President


WELLS FARGO BANK N.A.
By:    /s/ Virginia S. Christenson
Name:  Virginia S. Christenson
Title: Vice President / Relationship Mgr.


THE CIT GROUP/EQUIPMENT FINANCING, INC.
By:    /s/ Katie J. Saunders
       ----------------------------
Name:  Katie J. Saunders
Title: Sr. Credit Analyst


FLEET NATIONAL BANK
(Formerly Summit Bank)
By:    /s/ John T. Harrison
       ----------------------------
Name:  John T. Harrison
Title: Senior Vice President


HIBERNIA NATIONAL BANK
By:    /s/ Chris Haskew
       ----------------------------
Name:  Chris Haskew
Title:    Vice President


MERCANTILE NATIONAL BANK OF INDIANA
By:    /s/ Daryl D. Pomranke
       ----------------------------
Name:  Daryl D. Pomranke
Title: Vice President, Division Manager
       Corporate Services Division