<PAGE>   1






                                                                     EXHIBIT 4.1

                      SECOND AMENDMENT TO CREDIT AGREEMENT


         SECOND AMENDMENT TO CREDIT AGREEMENT (this "Amendment"), dated as of
August 19, 1998, among FLORSHEIM GROUP INC., a Delaware corporation (the
"Borrower"), the lending institutions from time to time party to the Credit
Agreement referred to below (the "Banks"), and BANKERS TRUST COMPANY, as Agent
(the "Agent"). All capitalized terms used herein and not otherwise defined shall
have the respective meanings provided such terms in the Credit Agreement
referred to below.

                              W I T N E S S E T H :

         WHEREAS, the Borrower, the Banks and the Agent are parties to a Credit
Agreement, dated as of May 9, 1997 (as amended, modified or supplemented to the
date hereof, the "Credit Agreement"); and

         WHEREAS, the parties hereto wish to amend the Credit Agreement as
herein provided, subject to and on the terms and conditions set forth herein;

         NOW THEREFORE, it is agreed:

         1.  Notwithstanding anything to the contrary contained in the credit
Agreement, the parties hereto hereby agree that, on the Second Amendment
Effective Date, to the extent the Total Revolving Loan Commitment at such time
is otherwise in excess of $91.6 million, then the Total Revolving Loan
Commitment shall be reduced on such date to $91.6 million. Such reduction to the
Total Revolving Loan Commitment shall be applied to reduce the Revolving Loan
Commitments of the Banks and to reduce the then remaining Scheduled Commitment
Reductions, on the same basis as is provided for reductions to the Total
Unutilized Revolving Loan Commitment pursuant to Section 3.02(a) of the Credit
Agreement. Also on the Second Amendment Effective Date, the Blocked Commitments
(and each of the General Blocked Commitment and the Senior Notes Blocked
Commitment) shall be reduced to $0. In connection with the reduction to the
Total Revolving Loan Commitment specified above in Section 1, the Borrower shall
make such payments, if any, as may be required pursuant to Section 4.02 of the
Credit Agreement immediately after giving effect to such reduction.

         2.  Section 9.09 of the Credit Agreement is hereby amended by (i)
deleting the ratio "5.20:1.00" appearing opposite the date June 30, 1998 in the
table therein and by inserting in lieu thereof the ratio "5.75:1.00", (ii)
deleting the ratio "4.90:1.00" appearing opposite the date September 30, 1998,
in the table therein and by inserting in lieu thereof the ratio "5.50:1.10"" and
(z) deleting the ratio "4.50:1.00" appearing opposite the date December 31, 1998
in the table therein ad by inserting in lieu thereof the ratio "5.00:1.00".

         3.  Section 9.10 of the Credit Agreement is hereby amended by deleting
the ratio "2.20:1.00" appearing opposite the date December 31, 1998 in the table
therein and by inserting in lieu thereof the ratio "2.05:1.00".

         4.  In order to induce the undersigned Banks to enter into this
Amendment, the Borrower hereby represents and warrants (i) the representations
and warranties contained in the Credit Agreement are true and correct in all
material respects on and as of the Second Amendment Effective Date (as defined
in Section 8 of this Amendment) (it being understood and agreed that any
representation or warranty which by its terms is made as of a specified date
shall be required to be true and correct in all material respects only as of
such specified date) and (ii) there exists no Default or Event Default on the
Second Amendment Effective Date, in each case after giving effect to this
Amendment.

         5.  This Amendment is limited as specified and shall not constitute a
modification, acceptance or waiver of any other provision of the Credit
Agreement or any other Credit Document.

         6.  This Amendment may be executed in any number of counterparts and by
the different parties hereto on separate counterparts, each of which
counterparts when executed and delivered shall be an original, but all of which
shall together constitute one and the same instrument. A complete set of
counterparts shall be lodged with the Borrower and the Agent at the Notice
Office.



                                       25
<PAGE>   2







         7.  THIS AMENDMENT AND THE RIGHTS AND OBLIGATIONS OF THE PARTIES
HEREUNDER SHALL BE CONSTRUED IN ACCORDANCE WITH AND GOVERNED BY THE LAW OF THE
STATE OF NEW YORK.

         8.  This Amendment shall become effective on the date (the "Second
Amendment Effective Date") when the Borrower and the Required Banks (i) shall
have signed a counterpart hereof (whether the same or different counterparts)
and (ii) shall have delivered (including by way of facsimile transmission) the
same to the Agent at the Notice Office.

         9.  From and after the Second Amendment Effective Date, all references
in the Credit Agreement and each of the Credit Documents to the Credit Agreement
shall be deemed to be references to the Credit Agreement as amended hereby.

         10. The Borrower hereby covenants and agrees that, so long as the
Second Amendment Effective Date occurs, it shall pay each Bank with executes and
delivers to the Agent a counterpart hereof by the later to occur of (x) the
close of business on the Second Amendment Effective Date or (y) 5:00 p.m. (New
York time) on Friday, August 21, 1998, a cash fee in an amount equal to 5 basis
points (.05%) of an amount equal to the Revolving Loan Commitment of such Bank,
in each case as same is in effect on the Second Amendment Effective Date. All
fees payable pursuant to this Section 10 shall be paid by the Borrower to the
Agent for distribution to the Banks not later than the first Business Day
following the Second Amendment Effective Date.




                                       26
<PAGE>   3



         IN WITNESS WHEREOF, each of the parties hereto has caused a counterpart
of this Amendment to be duly executed and delivered as of the date first above
written.

                     FLORSHEIM GROUP INC.


                     By:      Richard J. Anglin
                              -----------------
                              Name:    Richard J. Anglin
                              Title:   Vice President, Chief Financial Officer


                     BANKERS TRUST COMPANY,
                     Individually, and as Agent


                     By:      G. Andrew Keith
                              ---------------
                              Name:    G. Andrew Keith
                              Title:   Vice President


                     BANK OF AMERICA NATIONAL TRUST AND
                     SAVINGS ASSOCIATION


                     By:      Christine M. Tierney
                              --------------------
                              Name:    Christine M. Tierney
                              Title:   Senior Vice President


                     CREDIT AGRICOLE INDOSUEZ

                     By:      W. Leroy Startz
                              --------------- 
                              Name:    W. Leroy Startz
                              Title:   First Vice President

                     By:      Dean Balice
                              ----------- 
                              Name:    Dean Balice
                              Title:   Senior Vice President

                     CREDIT LYONNAIS NEW YORK BRANCH

                     By:      Attila Koc
                              ----------
                              Name:    Attila Koc
                              Title:   First Vice President

                     HARRIS TRUST AND SAVINGS BANK


                     By:
                        ------------------------------------------
                              Name:
                              Title:

                     HELLER FINANCIAL, INC.


                     By:      Linda W. Wolf
                              -------------  
                              Name:    Linda W. Wolf
                              Title:   Senior Vice President



                                       27

<PAGE>   4


                     LA SALLE NATIONAL BANK


                     By:      Steven M. Marks
                              ---------------
                              Name:    Steven M. Marks
                              Title:   First Vice President

                     SOCIETE GENERALE, CHICAGO BRANCH

                     By:__________________________________________
                              Name:
                              Title:

                     THE SUMITOMO BANK, LIMITED

                     By:      J.H. Broadley
                              -------------
                              Name:    J.H. Broadley
                              Title:   Vice President

                     By:      Brian M. Smith
                              --------------
                              Name:    Brian M. Smith
                              Title:   Senior Vice President &
                                       Regional Manager (East)

                     TRANSAMERICA BUSINESS CREDIT CORPORATION

                     By:      Perry Vavoules
                              --------------
                              Name:    Perry Vavoules
                              Title:   Senior Vice President




                                       28