<PAGE>   1
                                                                     EXHIBIT 4.1


                       THIRD AMENDMENT TO CREDIT AGREEMENT


         This Third Amendment to Credit Agreement (the "Amendment") is made on
this 15th day of November, 2000 by and among Florsheim Group Inc., The Florsheim
Shoe Store Company-Northeast, The Florsheim Shoe Store Company-West, Florsheim
Occupational Footwear, Inc., and L.J. O'Neill Shoe Co. (collectively, the
"Borrowers"), each of the financial institutions from time to time a party
thereto (individually, a "Lender" and collectively the "Lenders") and BT
Commercial Corporation, (individually ("BTCC") and in its capacity as agent (the
"Agent")).

                              W I T N E S S E T H:
                               -------------------

         WHEREAS, the Agent, the Lenders and the Borrowers are parties to that
certain Credit Agreement dated as of August 23, 1999, as amended through the
date hereof (the "Credit Agreement"); and

         WHEREAS, the parties desire to amend the Credit Agreement, as more
fully set forth herein.

         NOW, THEREFORE, in consideration of the mutual agreements herein
contained and the other good and valuable consideration, the adequacy of which
is hereby acknowledged, and subject to the terms and conditions hereof, the
parties hereto agree as follows:

         SECTION 1 DEFINITIONS. Unless otherwise defined herein, all capitalized
terms shall have the meanings given to them in the Credit Agreement.

         SECTION 2 AMENDMENTS TO CREDIT AGREEMENT.

                  2.1 Section 1.1 of the Credit Agreement is hereby amended by
         restating the defined term "Borrowing Base" in its entirety to read as
         follows:

         "BORROWING BASE means, at any time, the sum at such time of

                  (a) Eighty-five percent (85%) of Eligible Accounts Receivable,
         plus

                  (b) the lesser of (i) $50,000,000, and (ii) (1) during the
         months of May, June, July, August, September, and October during the
         term hereof, seventy percent (70%), and (2) at all other times during
         the term hereof, sixty-five percent (65%), of Eligible Inventory, plus


<PAGE>   2

                  (c) the Fixed Asset and Intellectual Property Sublimit, plus

                  (d) the Tranche B Availability, plus

                  (e) the Plan Distribution Availability.

         In addition, the Agent, in the exercise of its Permitted Discretion,
may (i) establish and increase or decrease reserves against Eligible Accounts
Receivable and Eligible Inventory, and (ii) impose additional restrictions (or
eliminate the same) to the standards of eligibility set forth in the definitions
of "Eligible Accounts Receivable" and "Eligible Inventory."

                  2.2 Section 1.1 of the Credit Agreement is hereby further
         amended by restating the defined term "EBITDA" in its entirety to read
         as follows:

                  "EBITDA means, for any period, the Consolidated Net Income of
         the Consolidated Entity (excluding extraordinary items and any non-cash
         income or expense attributable to the termination of the Florsheim
         Retirement Plan or any replacement plan) for such period, plus or minus
         (a) all Interest Expense, income tax expense or benefit, depreciation
         and amortization (including amortization of any goodwill or other
         intangibles) for such period; plus or minus (b) gains and losses
         attributable to any fixed asset sales; plus or minus (c) any other
         non-cash or one-time charges or gains which have been subtracted or
         added in calculating Consolidated Net Income."

                  2.3 Section 1.1 of the Credit Agreement is hereby further
         amended by inserting the following new defined term in proper
         alphabetical sequence:

         "FLORSHEIM RETIREMENT PLAN means the Florsheim Group Inc. Retirement
         Plan."

                  2.4 Section 1.1 of the Credit Agreement is hereby further
         amended by restating the defined term "Fixed Asset and Intellectual
         Property Sublimit" in its entirety to read as follows:

         "FIXED ASSET AND INTELLECTUAL PROPERTY SUBLIMIT means an amount equal
         to $35,000,000; provided, that the Fixed Asset and Intellectual
         Property Sublimit shall be automatically and permanently reduced by an
         amount equal, without duplication, to: (i) on the date which is one
         hundred-twenty (120) days after the end of the fiscal year of Florsheim
         ending December 31, 2000, and the end of each fiscal year of Florsheim
         ending thereafter, in each case by an amount equal to fifty percent
         (50%) of Excess Cash Flow for such fiscal year; (ii) upon consummation
         of any sales of any fixed assets in accordance with Section 8.6(v), in
         an amount equal to the net proceeds thereof; provided, that the Fixed
         Asset and Intellectual Property sublimit shall not be reduced by the
         net proceeds of such sale (other than the sale of the Jefferson City
         Real Estate), to the extent that the aggregate amount of such proceeds
         from such fixed asset sale (x) does not exceed $1,000,000 during any
         fiscal year, or (y) are reinvested in replacement fixed assets within
         270 days of such disposition and which in the Agent's reasonable


                                       2
<PAGE>   3

         determination are of at least substantially equivalent value; (iii)
         $900,000 on the last day of each calendar quarter during the term
         hereof, commencing with the calendar quarter ending June 30, 2000; (iv)
         on the date on which money or property reverts to Florsheim in
         connection with the termination of the Florsheim Retirement Plan the
         lesser of (x) the remainder of (A) the Net Reversion Amount, minus (B)
         $15,000,000 or (y) $4,000,000, and (v) at any time after the Closing
         Date, and after giving effect to any sale of assets permitted by
         subsections (iii) and (v) of Section 8.6 hereof (or the sale by any
         Foreign Subsidiary of all or substantially all of the assets of such
         Foreign Subsidiary) that the Agent determines by re-appraisal that, the
         ratio of the aggregate appraised value of the Borrower's fixed assets
         (excluding Florsheim's real property located in Cape Girardeau,
         Missouri (the "Cape Girardeau Real Estate")) and intellectual property
         (including without limitation, such general intangibles as trade names,
         licenses and franchise value) to the Fixed Asset and Intellectual
         Property Sublimit is less than such ratio as of the Closing Date, an
         amount sufficient to cause such ratio to be at least equal to such
         ratio as of the Closing Date. In no event however shall the Fixed Asset
         and Intellectual Property Sublimit exceed $35,000,000."

                  2.5 Section 1.1 of the Credit Agreement is hereby further
         amended by inserting the following new defined term in proper
         alphabetical sequence:

         "JEFFERSON CITY REAL ESTATE means the real property consisting of land
         and improvements and fixtures attached thereto located at 312 Wilson
         Drive, Jefferson City, Cole County, Missouri."

                  2.6 Section 1.1 of the Credit Agreement is hereby further
         amended by inserting the following new defined term in proper
         alphabetical sequence:

         "NET REVERSION AMOUNT means the aggregate amount of cash and value of
         any property received by Florsheim as a reversion in connection with
         the termination of the Florsheim Retirement Plan as contemplated by
         Sections 6.25 and 7.16 minus (i) the amount contributed or transferred
         by Florsheim to the replacement plan in connection with such
         termination, (ii) the amount of all federal, state and local income,
         alternative minimum, excise and other taxes paid by Florsheim in
         connection with such termination, and (iii) all fees and expenses
         incurred by Florsheim directly relating to such termination not in
         excess of $500,000."

                  2.7 Section 1.1 of the Credit Agreement is hereby further
         amended by inserting the following new defined term in proper
         alphabetical sequence:

         "PDA PARTICIPATIONS means participations in Revolving Loans outstanding
         against Planned Distribution Availability purchased pursuant to that
         certain Contingent Purchase Agreement, dated as of November 15, 2000,
         between Apollo Investment Fund, L.P. and the Lenders, and that certain
         Contingent Purchase Agreement, dated as of November 15, 2000, between
         Artemis America Partnership and the Lenders."




                                       3
<PAGE>   4

                  2.8 Section 1.1 of the Credit Agreement is hereby further
         amended by inserting the following new defined term in proper
         alphabetical sequence:

         "PLAN DISTRIBUTION AVAILABILITY means an amount equal to $15,000,000
         which amount shall automatically and permanently be reduced to zero
         ($0) on the first to occur of (i) August 22, 2002, and (ii) the date on
         which an involuntary proceeding or involuntary petition is commenced or
         filed against Florsheim under any bankruptcy, insolvency or similar law
         seeking the dissolution, or reorganization of Florsheim or the
         appointment of a receiver, trustee, custodian or liquidator for it or
         of a substantial part of its property, assets or business."

                  2.9 Section 1.1 of the Credit Agreement is hereby further
         amended by restating the defined term "Tranche B Availability" in its
         entirety to read as follows:

                  "TRANCHE B AVAILABILITY means an amount equal to $10,000,000,
         which amount shall automatically and permanently be reduced by
         $2,000,000 per month commencing with the last Business Day of November,
         2000, and on the last Business Day of each month thereafter until
         reduced to zero ($0); provided, however, if at any time when a
         reduction is otherwise scheduled to occur, the amount then available to
         the Borrowers under the Borrowing Base (calculated without giving
         effect to Plan Distribution Availability) is less than $5,000,000, or
         would be reduced below $5,000,000 as a result of such scheduled
         reduction, then such scheduled reduction shall be suspended, and such
         reductions shall not be reinstated until such time as the amount
         available to the Borrowers under the Borrowing Base (calculated without
         giving effect to Plan Distribution Availability) is equal to or greater
         than $5,000,000."

                  2.10 Section 4.6(c) of the Credit Agreement is hereby amended
         by restating the first sentence thereof in its entirety to read as
         follows:

         "The Borrowers may reduce or terminate the Line of Credit at any time
         and from time to time in whole or in part; provided that each such
         reduction must be in an amount of not less than $500,000 (and in
         increments of $500,000 in excess thereof)."

                  2.11 Section 4.6 of the Credit Agreement is hereby further
         amended by adding the following thereto as Section 4.6(f):

                  "(f) On or before the Business Day immediately following their
         receipt of amounts representing a reversion in connection with the
         termination of the Florsheim Retirement Plan as contemplated by
         Sections 6.25 and 7.16, the Borrowers shall prepay the Revolving Loans
         in an amount equal to the Net Reversion Amount. Upon any such
         prepayment, an amount equal to the lesser of (i) the outstanding
         principal balance of Revolving Loans attributable to Plan Distribution
         Availability, or (ii) the lesser of (x) $15,000,000, and (y) the Net


                                       4
<PAGE>   5

         Reversion Amount shall be applied to reduce the outstanding principal
         amount of Revolving Loans representing Plan Distribution Availability
         and Plan Distribution Availability will be permanently reduced by the
         amount of such prepayment."

                  2.12 Section 4.10 of the Credit Agreement is hereby amended by
         adding the following sentence immediately after the existing text of
         such Section:

         "Further notwithstanding the foregoing, in the event that the
         Obligations are .declared or become due and payable pursuant to Section
         9.2 hereof at a time when Revolving Loans are outstanding against
         Planned Distribution Availability and PDA Participations have been
         purchased or are thereafter purchased, any distributions (other than
         distributions derived from mandatory prepayments pursuant to Section
         4.6(f) or distributions which, but for an Insolvency Event, would have
         been derived from such a mandatory prepayment) for the ratable payment
         of interest and principal on the Revolving Loans shall be made by the
         Agent as if the portion of the Revolving Loans subject to PDA
         Participations were not outstanding, to the effect that all interest
         and principal on all other outstanding Revolving Loans (other than
         Revolving Loans against Tranche B Availability which shall be pari
         passu with the portion of the Revolving Loans subject to PDA
         Participations) shall be paid prior to the payment of interest and
         principal on any portion of the Revolving Loans subject to PDA Loan
         Participations and any Revolving Loans outstanding against Tranche B
         Availability."

                  2.13 Article 6 of the Credit Agreement is hereby amended by
         adding the following thereto as Section 6.25:

         "6.25 PLAN TERMINATION. In October, 2000, the Board of Directors of
         Florsheim adopted a certain resolution regarding the Florsheim
         Retirement Plan, such resolution directing the officers of Florsheim to
         take the following actions, among others: (i) freeze benefit accruals;
         (ii) secure such Internal Revenue Service ("IRS") and Pension Benefit
         Guaranty Corporation ("PBGC") approvals to make termination of the
         Florsheim Retirement Plan fully effective; (iii) preserve assets of the
         Florsheim Retirement Plan, to effect the purchase of annuities for
         participants and the satisfaction of benefit liabilities; (iv) take
         such actions necessary to reduce the excise tax rate on the reversion
         of assets; and (v) cause the reversion of excess Florsheim Retirement
         Plan assets to Florsheim."

                  2.14 Article 7 of the Credit Agreement is hereby amended by
         adding the following thereto as Section 7.16:

         "7.16 PLAN TERMINATION. Florsheim will cause to be taken all actions
         necessary to effectuate the termination of the Florsheim Retirement
         Plan including, but not limited to: (i) freezing benefit accruals on or
         before January 31, 2001; (ii) causing the required termination forms to
         request approval of the termination of the Florsheim Retirement Plan to
         be filed with the IRS and the PBGC on or before May 31, 2001; (iii)
         terminating the Florsheim Retirement Plan no later than




                                       5
<PAGE>   6

         June 30, 2001; (iv) if the PBGC does not issue a notice of
         noncompliance to the plan administrator of the Florsheim Retirement
         Plan, and the IRS has issued a favorable determination letter to the
         Florsheim Retirement Plan as to its qualified status upon its
         termination, then Florsheim will cause the plan administrator of the
         Florsheim Retirement Plan to commence the final distribution of plan
         assets pursuant to the termination of the Florsheim Retirement Plan in
         accordance with the times set forth in ERISA Section 4041 and the
         regulations promulgated thereunder; and (v) within sixty (60) days of
         the final distribution of plan assets pursuant to the provisions of
         paragraph (iv), above, Florsheim will cause all of the assets of the
         Florsheim Retirement Plan (including any amount constituting excess
         Florsheim Retirement Plan assets) to be distributed."

                  2.15 Section 8.1.1 of the Credit Agreement is hereby restated
         in its entirety to read as follows:

         "8.1.1 SENIOR LEVERAGE RATIO. The Consolidated Entity shall have as of
         the end of each fiscal period set forth below a ratio of Senior
         Indebtedness to EBITDA of not more than the ratio set forth below:

================================================================================
                 Period                                   Ratio
--------------------------------------------------------------------------------
Twelve fiscal months ending with the                     6.4:1.0
fourth quarter of  fiscal 2001
--------------------------------------------------------------------------------
Twelve fiscal months ending with the                     6.0:1.0
first quarter of fiscal 2002
--------------------------------------------------------------------------------
Twelve fiscal months ending with the                     5.0:1.0
second quarter of fiscal 2002, and
each fiscal quarter thereafter for
the then ending 12 month period
================================================================================

                  2.16 Section 8.1.2 of the Credit Agreement is hereby restated
         in its entirety to read as follows:

                                       6
<PAGE>   7
         "8.1.2 TOTAL LEVERAGE RATIO. The Consolidated Entity shall have as of
         the end of each fiscal period set forth below a ratio of total
         Indebtedness to EBITDA of not less than the ratio set forth below:

================================================================================
                 Period                                   Ratio
--------------------------------------------------------------------------------
Twelve fiscal months ending with the                     8.5:1.00
fourth quarter of  fiscal 2001
--------------------------------------------------------------------------------
Twelve fiscal months ending with the                     8.0:1.0
first quarter of fiscal 2002
--------------------------------------------------------------------------------
Twelve fiscal months ending with the                     7.0:1.0
second quarter of fiscal 2002, and
each fiscal quarter thereafter for
the then ending 12 month period
================================================================================

                  2.17 Article 8 of the Credit Agreement is hereby further
         amended by inserting the following new Section 8.1.3"

         "8.1.3 MINIMUM EBITDA. The Consolidated Entity shall have as of the end
         of each fiscal period set forth below EBITDA of not less than the
         amount set forth below:

================================================================================
                 Period                                   EBITDA
--------------------------------------------------------------------------------
Three months ending December 31, 2000                  ($2,000,000)
--------------------------------------------------------------------------------
Three months ending March 31, 2001                      $1,250,000
--------------------------------------------------------------------------------
Six months ending with the second                       $3,900,000
quarter of fiscal 2001
--------------------------------------------------------------------------------
Nine months ending with the third                       $5,400,000
quarter of fiscal 2001
--------------------------------------------------------------------------------
Twelve months ending with the fourth                    $9,400,000
quarter of fiscal 2001
--------------------------------------------------------------------------------
Twelve months ending with the first                    $10,000,000
quarter of fiscal 2002
--------------------------------------------------------------------------------
Twelve months ending with the second                   $12,000,000
quarter of fiscal 2002 and each fiscal
quarter thereafter for the then ending
12-month period
================================================================================




                                       7
<PAGE>   8

                  2.18 Section 8.9 of the Credit Agreement is hereby amended by
         adding the following thereto immediately after the existing text of
         such Section:

         "Notwithstanding the provisions of clause (d) of this Section 8.9, the
         fees otherwise permitted by such clause (d) shall not be paid during
         the period commencing or October 31, 2000 through and including the
         Expiration Date."

         SECTION 3 WAIVER. Agent and Lenders hereby waive the Borrowers'
compliance with covenants contained in Sections 8.1.1 (Senior Leverage Ratio)
and 8.1.2 (Total Leverage Ratio) of the Credit Agreement for the period of the
twelve fiscal months ending with the third quarter of fiscal 2000.

         SECTION 4 AMENDMENT FEES AND WARRANTS. In consideration of the
Agreement of Agent and Lenders to enter into this Amendment, the Borrowers shall
pay to Agent, for the benefit of the Lenders, an amendment fee payable as
follows: an amount equal to Two Hundred Fifty Thousand Dollars ($250,000)
payable on the date hereof, and an additional Two Hundred Fifty Thousand Dollars
payable on each April 15 and November 15 thereafter until such time as (i) Plan
Distribution Availability has been reduced to zero ($0) and, (ii) all Revolving
Loans attributable to Plan Distribution Availability have been paid in full. The
Agent is hereby authorized by the Borrowers to charge such fees to Borrowers'
Loan Account. In further consideration of the agreement of the Agent and Lenders
to enter into this Amendment, Florsheim shall issue to Lenders warrants in the
form attached hereto as Exhibit A (the "Warrants") providing Lenders with the
right to purchase five percent (5%) of the common stock of Florsheim upon the
terms set forth therein.

         SECTION 5 CONDITIONS PRECEDENT. The effectiveness of this Amendment is
expressly conditioned upon satisfaction of the following conditions precedent:

                  5.1 The Borrowers shall have caused the Warrants to be issued
         to the Lenders and shall have paid the amendment fee payable on the
         date hereof as provided in Section 4.

                  5.2 Agent shall have received copies of this Amendment duly
         executed by the Borrowers and the Majority Lenders.

                  5.3 Agent shall have received Contingent Purchase Agreements
         in form and content acceptable to Agent executed on behalf of Apollo
         Investment Fund I ("AIF") and Artemis America Partnership ("Artemis").

                  5.4 Agent shall have received such other documents,
         certificates and assurances as it shall reasonably request.

         SECTION 6 REAFFIRMATION OF BORROWERS. The Borrowers hereby represent
and warrant to Agent and Lenders that (i) the representations and warranties set
forth in Section 5 of the Credit Agreement are true and correct on and as of the
date hereof, except to the



                                       8
<PAGE>   9

extent (a) that any such representations or warranties relate to a specific
date, or (b) of changes thereto as a result of transactions for which Agent and
Lenders have granted their consent; (ii) the Borrowers are on the date hereof in
compliance with all of the terms and provisions set forth in the Credit
Agreement as hereby amended; and (iii) upon execution hereof no Default or Event
of Default has occurred and is continuing.

         SECTION 7 FULL FORCE AND EFFECT. Except as herein amended, the Credit
Agreement and all other Credit Documents shall remain in full force and effect.

         SECTION 8 COUNTERPARTS. This Amendment may be executed in two or more
counterparts, each of which shall be deemed an original, but all of which
together shall constitute one and the same document.

         IN WITNESS WHEREOF, the parties hereto have caused this Credit
Agreement to be executed and delivered in Chicago, Illinois by their proper and
duly authorized officers as of the date first set forth above.



                                           BORROWER:

                                           FLORSHEIM GROUP INC.


                                           By: /s/ Thomas P. Polke
                                              ----------------------------------
                                              Name: Thomas P. Polke
                                              Title: Executive Vice President,
                                                     Chief Financial Officer

                                           THE FLORSHEIM SHOE STORE COMPANY -
                                           NORTHEAST


                                           By: /s/ Thomas P. Polke
                                              ----------------------------------
                                              Name: Thomas P. Polke
                                              Title: Executive Vice President,
                                                     Chief Financial Officer


                                           THE FLORSHEIM SHOE STORE COMPANY -
                                           WEST


                                           By: /s/ Thomas P. Polke
                                              ----------------------------------
                                              Name: Thomas P. Polke
                                              Title: Executive Vice President,
                                                     Chief Financial Officer


                                       9
<PAGE>   10

                                           FLORSHEIM OCCUPATIONAL FOOTWEAR, INC.


                                           By: /s/ Thomas P. Polke
                                              ----------------------------------
                                              Name: Thomas P. Polke
                                              Title: Executive Vice President,
                                                     Chief Financial Officer


                                           L.J. O'NEILL SHOE CO.


                                           By: /s/ Thomas P. Polke
                                              ----------------------------------
                                              Name: Thomas P. Polke
                                              Title: Executive Vice President,
                                                     Chief Financial Officer

                                           AGENT:

                                           BT COMMERCIAL CORPORATION, as Agent


                                           By: /s/ Frank Fazio
                                              ----------------------------------
                                              Name: Frank Fazio
                                                   -----------------------------
                                              Title: Director
                                                    ----------------------------

                                           LENDERS:

                                           BT COMMERCIAL CORPORATION


                                           By: /s/ Frank Fazio
                                              ----------------------------------
                                              Name: Frank Fazio
                                                   -----------------------------
                                              Title: Director
                                                    ----------------------------


                                           LASALLE BANK NATIONAL ASSOCIATION

                                           By: /s/ Bruce S. Lubin
                                              ----------------------------------
                                              Name: Bruce S. Lubin
                                                   -----------------------------
                                              Title: Senior Vice President
                                                    ----------------------------


                                           DIME COMMERCIAL CORP.

                                           By: /s/ Thomas M. Watson
                                              ----------------------------------
                                              Name: Thomas M. Watson
                                                   -----------------------------
                                              Title: Vice President
                                                    ----------------------------


                                       10