<PAGE>   1
                                                                     EXHIBIT 4.1


                 FOURTH AMENDMENT AND WAIVER TO CREDIT AGREEMENT


         FOURTH AMENDMENT AND WAIVER TO CREDIT AGREEMENT (this "Amendment"),
dated as of July 1, 1999, among FLORSHEIM GROUP INC., A Delaware corporation
(the "Borrower"), the lending institutions from time to time party to the Credit
Agreement referred to below (the "Banks"), and BANKERS TRUST COMPANY, as Agent
(the "Agent"). All capitalized terms used herein and not otherwise defined shall
have the respective meanings provided such terms in the Credit Agreement
referred to below.

                              W I T N E S S E T H:


         WHEREAS, the Borrower, the Banks and the Agent are parties to a Credit
Agreement, dated as of May 9, 1997 (as amended, modified or supplemented to the
date hereof, the "Credit Agreement"); and

         WHEREAS, the parties hereto wish to amend and waive certain provisions
of the Credit Agreement as herein provided, subject to and on the terms and
conditions set forth herein;

         NOW, THEREFORE, it is agreed:

         1.           The Banks hereby waive the requirements of Section 3.03(f)
              of the Credit Agreement solely with respect to the sale by the
              Borrower to Gordon Builders, Inc. of 6.1 acres of land in
              Jefferson City, Missouri consummated on May 12, 1999.

         2.                The Banks hereby waive the requirements of Sections
              9.08, 9.09 and 9.10 of the Credit Agreement solely with respect to
              the amount or ratio appearing in the table of each such Section
              opposite the date "June 30, 1999".

         3.                Section 9.08 of the Credit Agreement is hereby
              amended by deleting the amount "$20,500,000" appearing opposite
              the date September 30, 1999 in the table therein and by inserting
              in lieu thereof the amount "$17,000,000".

         4.                Section 9.09 of the Credit Agreement is hereby
              amended by deleting the ratio "3.75:1.00" appearing opposite the
              date September 30, 1999 in the table therein and inserting in lieu
              thereof the ratio "6.00:1.00".

         5.                Section 9.10 of the Credit Agreement is hereby
              amended by deleting the ratio"2.45:1.00" appearing opposite the
              date September 30,


<PAGE>   2
              1999 in the table therein and by inserting in lieu thereof the
              ratio "1.90:1.00".

         6.                In order to induce the undersigned Banks to enter
              into this Amendment, the Borrower hereby represents and warrants
              that (i) the representations and warranties contained in the
              Credit Agreement are true and correct in all material respects on
              and as of the Fourth Amendment Effective Date (as defined in
              Section 10 of this Amendment) (it being understood and agreed that
              any representation or warranty which by its terms is made as of a
              specified date shall be required to be true and correct in all
              material respects only as of such specified date) and (ii) there
              exists no Default or Event of Default on the Fourth Amendment
              Effective Date, in each case after giving effect to this
              Amendment.

         7.                This Amendment is limited as specified and shall not
              constitute a modification, acceptance or waiver of any other
              provision of the Credit Agreement or any other Credit Document.

         8.                This Amendment may be executed in any number of
              counterparts and by the different parties hereto on separate
              counterparts, each of which counterparts when executed and
              delivered shall be an original, but all of which shall together
              constitute one and the same instrument. A complete set of
              counterparts shall be lodged with the Borrower and the Agent at
              the Notice Office.

         9.                THIS AMENDMENT AND THE RIGHTS AND OBLIGATIONS OF THE
              PARTIES HEREUNDER SHALL BE CONSTRUED IN ACCORDANCE WITH AND
              GOVERNED BY THE LAW OF THE STATE OF NEW YORK.

         10.               This Amendment shall become effective on the date
              (the "Fourth Amendment Effective Date") when the Borrower and the
              Required Banks (i) shall have signed a counterpart hereof (whether
              the same or different counterparts) and (ii) shall have delivered
              (including by way of facsimile transmission) the same to the Agent
              at the Notice Office.

         11.               From and after the Fourth Amendment Effective Date,
              all references in the Credit Agreement and each of the Credit
              Documents to the Credit Agreement shall be deemed to be references
              to the Credit Agreement as amended hereby.

                                      * * *


<PAGE>   3


                  IN WITNESS WHEREOF, each of the parties hereto has caused a
counterpart of this Amendment to by duly executed and delivered as of the date
first above written.

                              FLORSHEIM GROUP INC.


                              By: Richard J. Anglin
                                  ---------------------------------
                                  Name:  Richard J. Anglin
                                  Title: Executive Vice President
                                  Chief Financial Officer



<PAGE>   4



                  IN WITNESS WHEREOF, each of the parties hereto has caused a
counterpart of this Amendment to be duly executed and delivered as of the date
first above written.

                              FLORSHEIM GROUP INC.


                              By:
                                  ---------------------------------
                                  Name:
                                  Title:

                              BANKERS TRUST COMPANY
                              Individually, and as Agent

                              By: James Reilly
                                  ---------------------------------
                                  Name:  James Reilly
                                  Title:  Vice President

                              BANK OF AMERICA NATIONAL
                              TRUST AND SAVINGS ASSOCIATION

                              By:
                                  ---------------------------------
                                  Name:
                                  Title:

                              CREDIT AGRICOLE INDOSUEZ

                              By:
                                  ---------------------------------
                                  Name:
                                  Title:

                              By:
                                  ---------------------------------
                                  Name:
                                  Title:


<PAGE>   5



                              CREDIT LYONNAIS NEW YORK BRANCH

                              By: Attila Koc
                                  ---------------------------------
                                  Name:  Attila Koc
                                  Title: Senior Vice President

                              HARRIS TRUST AND SAVINGS BANK

                              By: Scott F. Geik
                                  ---------------------------------
                                  Name:  Scott F. Geik
                                  Title: Vice President

                              HELLER FINANCIAL, INC.

                              By: Robert M. Regg
                                  ---------------------------------
                                  Name:  Robert M. Regg
                                  Title: Assistant Vice President

                              LASALLE NATIONAL BANK

                              By:
                                  ---------------------------------
                                  Name:
                                  Title:

                              THE SUMITOMO BANK, LIMITED

                              By:
                                  ---------------------------------
                                  Name:
                                  Title:

                              By:
                                  ---------------------------------
                                  Name:
                                  Title:

                              TRANSAMERICA BUSINESS CREDIT CORPORATION

                              By: Perry Vavoules
                                  ---------------------------------
                                  Name:  Perry Vavoules
                                  Title: Senior Vice President