<PAGE>   1
                                                                     EXHIBIT 4.1

                      SECOND AMENDMENT TO CREDIT AGREEMENT


         This Second Amendment to Credit Agreement (the "Amendment") is made on
this 15th day of May, 2000 by and among Florsheim Group Inc., The Florsheim Shoe
Store Company-Northeast, The Florsheim Shoe Store Company-West, Florsheim
Occupational Footwear, Inc., and L.J. O'Neill Shoe Co. (collectively, the
"Borrowers"), each of the financial institutions from time to time a party
thereto (individually, a "Lender" and collectively the "Lenders") and BT
Commercial Corporation, (individually ("BTCC") and in its capacity as agent (the
"Agent")).

                              W I T N E S S E T H:

         WHEREAS, the Agent, the Lenders and the Borrowers are parties to that
certain Credit Agreement dated as of August 23, 1999, as amended through the
date hereof (the "Credit Agreement"); and

         WHEREAS, the parties desire to amend the Credit Agreement, as more
fully set forth herein.

         NOW, THEREFORE, in consideration of the mutual agreements herein
contained and the other good and valuable consideration, the adequacy of which
is hereby acknowledged, and subject to the terms and conditions hereof, the
parties hereto agree as follows:

         SECTION 1 DEFINITIONS. Unless otherwise defined herein, all capitalized
terms shall have the meaning given to them in the Credit Agreement.

         SECTION 2 AMENDMENTS TO CREDIT AGREEMENT.

                  2.1 Section 1.1 of the Credit Agreement is hereby further
amended by inserting the following new defined term in proper alphabetical
sequence:

                           "SENIOR INDEBTEDNESS means Indebtedness of the
                           Consolidated Entity, other than Indebtedness which
                           has subordination terms, covenants, pricing and other
                           terms which have been approved in writing by the
                           Majority Lenders."

                  2.2 Section 1.1 of the Credit Agreement is hereby further
amended by deleting the defined term "LIBOR Rate Margin in its entirety, and
inserting the following in its stead:

                           "LIBOR RATE MARGIN means three and one-half percent
                           (3.50%) per annum for each Interest Period for a
                           LIBOR Rate Loan, provided however, that if

<PAGE>   2

                           the ratio of (1) Indebtedness to (2) EBITDA as of the
                           end of any fiscal quarter shall be less than 5.0 to
                           1.0, based on such financial reports delivered by
                           Florsheim to Agent sufficient to Agent's reasonable
                           satisfaction concerning such fact, and so long as no
                           Default or Event of Default then exists, the LIBOR
                           Rate Margin shall, effective on the first day of the
                           calendar month following the month in which the Agent
                           has confirmed such fact, be three percent (3.00%) per
                           annum for any Interest Period occurring during the
                           three (3) calendar months commencing with the month
                           in which the reduction in the LIBOR Rate Margin is
                           effective (except that notwithstanding the foregoing,
                           if a Default or Event of Default shall occur during
                           any period during the term hereof in which the
                           Borrowers are otherwise entitled to the benefit of a
                           reduced LIBOR Rate Margin, no such reduction shall be
                           effective during the pendency of any such Default or
                           Event of Default). At such time as the financial
                           reports delivered by Florsheim to Agent demonstrate
                           that the ratio of total Indebtedness to EBITDA as at
                           the end of any fiscal quarter shall be equal to or
                           greater than 5.0:1.0, then, in such event, the LIBOR
                           Rate Margin shall revert to three and one-half
                           percent (3.50%)."

                  2.3 Section 8.1 of the Credit Agreement is hereby deleted in
its entirety, and the following is inserted in its stead:

                           "8.1 INTEREST COVERAGE RATIO. The Consolidated Entity
                           shall have as of the end of each fiscal period set
                           forth below a ratio of EBITDA to Interest Expense of
                           not less than the ratio set forth below:


<PAGE>   3


-------------------------------------------------------------------
                 PERIOD                                   RATIO
-------------------------------------------------------------------
twelve  fiscal  months  ending  with the                 1.0:1.0
third quarter of fiscal 2000
-------------------------------------------------------------------
twelve  fiscal  months  ending  with the                 1.3:1.0
fourth quarter of fiscal 2000
-------------------------------------------------------------------
twelve  fiscal  months  ending  with the                 1.4:1.0
first quarter of fiscal 2001
-------------------------------------------------------------------
twelve  fiscal  months  ending  with the                 1.4:1.0
second quarter of  fiscal 2001
-------------------------------------------------------------------
twelve  fiscal  months  ending  with the                 1.6:1.0
third quarter of fiscal 2001
-------------------------------------------------------------------
twelve fiscal months ending with the                     1.7:1.0
fourth quarter of fiscal 2001, and each
fiscal quarter thereafter for the then
ending twelve fiscal month period
-------------------------------------------------------------------


                  2.4 Article 8 of the Credit Agreement is hereby amended by
inserting the following new Section 8.1.1:

                           "8.1.1 SENIOR LEVERAGE RATIO. The Consolidated Entity
                           shall have as of the end of each fiscal period set
                           forth below a ratio of Senior Indebtedness to EBITDA
                           of not more than the ratio set forth below:



<PAGE>   4



-------------------------------------------------------------------
                 PERIOD                                   RATIO
-------------------------------------------------------------------
twelve fiscal months ending with the                     11.00:1.0
second quarter of fiscal 2000
-------------------------------------------------------------------
twelve fiscal months ending with the third               7.15:1.0
quarter of fiscal 2000
-------------------------------------------------------------------
twelve fiscal months ending with the                     5.00:1.0
fourth quarter of fiscal 2000
-------------------------------------------------------------------
twelve fiscal months ending with the first               4.50:1.0
quarter of fiscal 2001
-------------------------------------------------------------------
twelve fiscal months ending with the                     4.25:1.0
second quarter of fiscal 2001
-------------------------------------------------------------------
twelve fiscal months ending with the third               4.00:1.0
quarter of fiscal 2001
-------------------------------------------------------------------
twelve fiscal months ending with the                     4.00:1.00
fourth quarter of fiscal 2001
-------------------------------------------------------------------
twelve fiscal months ending with the first               3.50:1.0
quarter of fiscal 2002, and each fiscal
quarter thereafter for the then ending
twelve fiscal month period
-------------------------------------------------------------------


<PAGE>   5




                  2.5 Article 8 of the Credit Agreement is hereby further
amended by inserting the following new Section 8.1.2:

                           "TOTAL LEVERAGE RATIO. The Consolidated Entity shall
                           have as of the end of each fiscal period set forth
                           below a ratio of Indebtedness to EBITDA of not less
                           than the ratio set forth below:

-------------------------------------------------------------------
                 PERIOD                                   RATIO
-------------------------------------------------------------------

twelve fiscal months ending with the                     13.00:1.0
second quarter of fiscal 2000
-------------------------------------------------------------------
twelve fiscal months ending with the third                8.50:1.0
quarter of fiscal 2000
-------------------------------------------------------------------
twelve fiscal months ending with the                      6.15:1.0
fourth quarter of fiscal 2000
-------------------------------------------------------------------
twelve fiscal months ending with the first                5.50:1.0
quarter of fiscal 2001
-------------------------------------------------------------------
twelve fiscal months ending with the                      5.50:1.0
second quarter of fiscal 2001
-------------------------------------------------------------------
twelve fiscal months ending with the third                5.25:1.0
quarter of fiscal 2001
-------------------------------------------------------------------
twelve fiscal months ending with the                      5.25:1.00
fourth quarter of fiscal 2001
-------------------------------------------------------------------
twelve fiscal months ending with the first                5.00:1.0
quarter of fiscal 2002, and each fiscal
quarter thereafter for the then ending
twelve fiscal month period
-------------------------------------------------------------------


<PAGE>   6



                  2.6 Section 8.2 of the Credit Agreement is hereby amended by
deleting the first sentence thereof, and inserting the following its stead:

                           "The Consolidated Entity shall not make payments for
                           Capital Expenditures in excess of $8,500,000 during
                           fiscal 2000 and any fiscal year thereafter."

         SECTION 3. AMENDMENT FEE. In consideration of the agreement of the
Agent and Lenders to enter into this Amendment, the Borrowers shall pay to the
Agent, for the benefit of the Lenders, an amendment fee in the amount of Five
Hundred Thousand Dollars ($500,000) on the date hereof. The Agent is hereby
authorized by the Borrower to charge such fee to Borrowers' Loan Account.

         SECTION 4. CONDITIONS PRECEDENT. The effectiveness of this Amendment is
expressly conditioned upon satisfaction of the following conditions precedent:

                  4.1 The Borrowers shall have paid the amendment fee in the
amount of $500,000 to the Agent for the benefit of the Lenders;

                  4.2 Agent shall have received copies of this Amendment duly
executed by the Borrowers and the Majority Lenders; and

                  4.3 Agent shall have received such other documents,
certificates and assurances as it shall reasonably request.

         SECTION 5. REAFFIRMATION OF BORROWERS. The Borrowers hereby represent
and warrant to Agent and Lender that (i) the representations and warranties set
forth in Section 5 of the Credit Agreement are true and correct on and as of the
date hereof, except to the extent (a) that any such representations or
warranties relate to a specific date, or (b) of changes thereto as a result of
transactions for which Agent and Lenders have granted their consent; (ii) the
Borrowers are on the date hereof in compliance with all of the terms and
provisions set forth in the Credit Agreement as hereby amended; and (iii) upon
execution hereof no Default or Even of Default has occurred and is continuing or
has not previously been waived.

         SECTION 6. FULL FORCE AND EFFECT. Except as herein amended, the Credit
Agreement and all other Credit Documents shall remain in full force and effect.

         SECTION 7. COUNTERPARTS. This Amendment may be executed in two or more
counterparts, each of which shall be deemed an original, but all of which
together shall constitute one and the same document.

<PAGE>   7


         IN WITNESS WHEREOF, the parties hereto have caused this Credit
Agreement to be executed and delivered in Chicago, Illinois by their proper and
duly authorized officers as of the date first set forth above.


                                       BORROWER:

                                       FLORSHEIM GROUP INC.

                                       By:  /s/ Richard J. Anglin
                                           Name:    Richard J. Anglin
                                           Title:   Executive Vice President,
                                                    Chief Financial Officer


                                       THE FLORSHEIM SHOE STORE
                                        COMPANY - NORTHEAST

                                       By:  /s/ Richard J. Anglin
                                           Name:    Richard J. Anglin
                                           Title:   Executive Vice President,
                                                    Chief Financial Officer


                                       THE FLORSHEIM SHOE STORE
                                        COMPANY - WEST

                                       By:  /s/ Richard J. Anglin
                                           Name:    Richard J. Anglin
                                           Title:   Executive Vice President,
                                                    Chief Financial Officer


                                       FLORSHEIM OCCUPATIONAL
                                        FOOTWEAR, INC.

                                       By:  /s/ Richard J. Anglin
                                           Name:    Richard J. Anglin
                                           Title:   Executive Vice President,
                                                    Chief Financial Officer



<PAGE>   8





                                       L.J. O'NEILL SHOE CO.

                                       By:  /s/ Richard J. Anglin
                                           Name:    Richard J. Anglin
                                           Title:   Executive Vice President,
                                                    Chief Financial Officer


                                       AGENT:

                                       BT COMMERCIAL CORPORATION, as Agent

                                       By:  /s/ Frank Fazio
                                           Name:    Frank Fazio
                                           Title:   Director


                                       LENDERS:

                                       BT COMMERCIAL CORPORATION


                                       By:  /s/ Frank Fazio
                                           Name:    Frank Fazio
                                           Title:   Director

                                       LASALLE BANK NATIONAL ASSOCIATION

                                       By: /s/ Steven M. Marks
                                           Name:    Steven M. Marks
                                           Title:   First Vice-President

                                       DIME COMMERCIAL CORP.


                                       By: /s/ Dan Bueno
                                           Name:    Dan Bueno
                                           Title:   Assistant Treasurer