<PAGE>

                                                                     EXHIBIT 4.1


                             IMPERIAL SUGAR COMPANY


                  FIRST AMENDMENT TO INTERIM WAIVER AGREEMENT



To the Lender Parties to the
 Interim Waiver Agreement
 referred to below

Ladies and Gentlemen:

     We refer to the Interim Waiver Agreement between us dated as of September
30, 2000 (the "Interim Waiver Agreement"), capitalized terms used without
definition below to have the meanings ascribed to them in the Interim Waiver
Agreement.  Upon receipt by the Administrative Agent of counterparts hereof
which, taken together, bear the signatures of the Borrower, the Subsidiary
Guarantors, the Requisite Restructuring Lenders and the Majority Revolving
Credit Facility Lenders, the Interim Waiver Agreement shall be amended by (i)
striking the date "January 8, 2001" wherever such date appears therein other
than in Section 5 thereof and substituting the date "January 14, 2001" therefor
and (ii) Section 5 of the Interim Waiver Agreement shall not be applicable after
January 8, 2001.  Except as specifically amended hereby all of the terms,
conditions and provisions of the Interim Waiver Agreement shall stand and remain
unchanged and in full force and effect.  No reference to this First Amendment to
Interim Waiver Agreement need be made in any instrument or document at any time
referring to the Interim Waiver Agreement, a reference to the Interim Waiver
Agreement in any of such to be deemed to be a reference to the same as amended
hereby.  This First Amendment to Interim Waiver Agreement may be executed in
counterparts and by separate parties hereto on separate counterparts, each to
constitute an original but all of which shall constitute one and the same
instrument.  The amendments provided for herein shall be strictly construed and
limited as herein provided.  This First Amendment to Interim Waiver Agreement
shall be construed in accordance with and governed by the laws of the state of
New York.

     Dated and to Become Effective as of this 8th day of January 2001.
<PAGE>

                                 Imperial Sugar Company (formerly known as
                                   Imperial Holly Corporation)

                                     /s/  MARK Q. HUGGINS
                                 By ________________________________________
                                    Its  CFO
                                        ____________________________________


                                 Biomass Corporation

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: President


                                 Crown Express, Inc.

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: President


                                 Diamond Crystal Brands, Inc.

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President

                                 Diamond Crystal Brands, LP

                                 By:  Savannah Molasses & Specialties Company
                                      Its sole General Partner


                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Diamond Crystal Holdings, Inc.


                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President
<PAGE>

                                 Diamond Crystal Specialty Foods, Inc.


                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Dixie Crystals Foodservice, Inc.

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 DSLT Holding Company

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Food Carrier, Inc.

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Fort Bend Utilities Company

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Great Lakes Sugar Company

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President
<PAGE>

                                 Holly Northwest Company

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: President


                                 Holly Sugar Corporation

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Imperial Distributing, Inc.

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: President


                                 Imperial-Savannah, LP

                                 By:  Savannah Molasses & Specialties Company
                                      Its sole General Partner

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Imperial Sweeteners Distributors, Inc.

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 King Packaging Company, Inc.

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President
<PAGE>

                                 Limestone Products Company

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: President


                                 Menu Magic Foods, Inc.

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Michigan Sugar Company

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Phoenix Packaging Corporation

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Ragus Holdings, Inc.

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: President


                                 Savannah Foods Industrial, Inc.

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President
<PAGE>

                                 Savannah Foods & Industries, Inc.

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Savannah Investment Company

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: President


                                 Savannah Molasses & Specialties Company

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name:
                                 Title:


                                 Savannah Sugar Refining Corporation

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Wholesome Sweeteners Group, Ltd.

                                 By:  Wholesome Sweeteners, LLC
                                      Its sole General Partner

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President


                                 Wholesome Sweeteners, LLC

                                     /s/  W.F. SCHWER
                                 By ________________________________________
                                 Name: W.F. Schwer
                                 Title: Sr. Vice President
<PAGE>

                                 Lehman Commercial Paper Inc., as Syndication
                                   Agent and as a Lender

                                     /s/ MICHELE SWANSON
                                 By ________________________________________
                                 Name:  Michele Swanson
                                 Title: Authorized Signatory


                                 Harris Trust and Savings Bank, as
                                   Administrative Agent, Collateral Agent,
                                   Issuing Lender and as a Lender


                                 By ________________________________________
                                 Name:
                                 Title:
<PAGE>

                                 Wachovia Bank, N.A.

                                     /s/  R.H. MICHALIK
                                 By ________________________________________
                                 Name: Richard H. Michalik
                                 Title: Senior Vice President
<PAGE>

                                 US Bancorp AG Credit, Inc.

                                     /s/  SANDRA A. SAUER
                                 By ________________________________________
                                 Name: Sandra A. Sauer
                                 Title: Vice President
<PAGE>

                                 The Bank of New York

                                     /s/  R.R. REEDY
                                 By ________________________________________
                                 Name: Ronald R. Reedy
                                 Title: Vice President
<PAGE>

                                 Cooperatieve Centrale Raiffeisen-Boerenleenbank
                                   B.A., "Rabobank Nederland" New York Branch

                                     /s/  TIMOTHY J. MOORE
                                 By ________________________________________
                                 Name: Timothy J. Moore
                                 Title: Vice President


                                     /s/  IAN REECE
                                 By ________________________________________
                                 Name: Ian Reece
                                 Title: Senior Credit Officer
<PAGE>

                                 Frost National Bank

                                     /s/  PHIL DUDLEY
                                 By ________________________________________
                                 Name: Phil Dudley
                                 Title: Senior Vice President
<PAGE>

                                 Credit Agricole Indosuez

                                     /s/  LARRY MATERI
                                 By ________________________________________
                                 Name: Larry Materi
                                 Title: Vice President

                                     /s/  PAUL A. DYTRYCH
                                 By ________________________________________
                                 Name: Paul A. Dytrych
                                 Title: Senior Relationship Manager
<PAGE>

                                 Balanced High Yield Fund I Ltd.,

                                 By: BHF (USA) Capital Corporation acting as
                                     attorney-in-fact

                                     /s/  JOHN J. D'ANGELO
                                 By ________________________________________
                                 Name: John J. D'Angelo
                                 Title: Vice President
                                        Asset Management

                                     /s/  CHRISTINE NORMAN
                                 By ________________________________________
                                 Name: Christine Norman
                                 Title: Associate

<PAGE>

                                 Monument Capital Ltd.

                                 By: Alliance Capital Management L.P., as
                                     Investment Manager

                                 By: Alliance Capital Management Corporation,
                                     as General Partner

                                     /s/  JOEL SEREBRANSKY
                                 By ________________________________________
                                 Name: Joel Serebransky
                                 Title: Senior Vice President


                                 Oak Mountain Limited

                                 By: Alliance Capital Management L.P., as
                                     Investment Manager

                                 By: Alliance Capital Management Corporation,
                                     as General Partner

                                     /s/  JOEL SEREBRANSKY
                                 By ________________________________________
                                 Name: Joel Serebransky
                                 Title: Senior Vice President
<PAGE>

                                 Van Kampen CLO I, Ltd.

                                 By:  Van Kampen Management Inc., as Collateral
                                      Manager

                                     /s/  DARVIN D. PIERCE
                                 By ________________________________________
                                 Name: Darvin D. Pierce
                                 Title: Principal


                                 Van Kampen CLO II, Ltd.

                                 By:  Van Kampen Management Inc., as Collateral
                                      Manager

                                     /s/  DARVIN D. PIERCE
                                 By ________________________________________
                                 Name: Darvin D. Pierce
                                 Title: Principal


                                 Van Kampen Senior Income Trust

                                 By:  Van Kampen Investment Advisory Corp.

                                     /s/  DARVIN D. PIERCE
                                 By ________________________________________
                                 Name: Darvin D. Pierce
                                 Title: Principal
<PAGE>

                                 Black Diamond CLO 1998 - 1 Ltd.

                                     /s/  JOHN N. CULLINAME
                                 By ________________________________________
                                 Name: John N. Culliname
                                 Title: Director


                                 Black Diamond International Funding Ltd.

                                     /s/  JOHN N. CULLINAME
                                 By ________________________________________
                                 Name: John N. Culliname
                                 Title: Director
<PAGE>

                                 KZH Sterling LLC

                                     /s/  KIMBERLY ROWE
                                 By ________________________________________
                                 Name: Kimberly Rowe
                                 Title: Authorized Agent
<PAGE>

                                 First Union National Bank

                                     /s/  RON R. FERGUSON
                                 By ________________________________________
                                 Name: Ron R. Ferguson
                                 Title: Senior Vice President
<PAGE>

                                 The Pilgrim Prime Rate Trust

                                 By: ING Pilgrim Investments Inc. as its
                                     Investment Manager

                                     /s/  MICHEL PRINCE, CFA
                                 By ________________________________________
                                 Name: Michel Prince, CFA
                                 Title: Vice President

                                 Pilgrim America High Income Investments, Ltd.
                                   (as Assignee)

                                 By: ING Pilgrim Investments Inc. as its
                                     Investment Manager

                                     /s/  MICHEL PRINCE, CFA
                                 By ________________________________________
                                 Name: Michel Prince, CFA
                                 Title: Vice President