<PAGE>

                                                                     EXHIBIT 4.1

                             IMPERIAL SUGAR COMPANY

                            INTERIM WAIVER AGREEMENT


To the Lender Parties to the Credit
  Agreement Identified Below

Ladies and Gentlemen:

     We refer to the Amended and Restated Credit Agreement dated as of December
22, 1997 by and among Imperial Sugar Company (then known as Imperial Holly
Corporation), the Lenders from time to time party thereto, Lehman Commercial
Paper Inc. as Syndication Agent, Lehman Brothers Inc. as Arranger and Harris
Trust and Savings Bank as Administrative and Collateral Agent, as heretofore or
hereafter amended (the "Credit Agreement"), capitalized terms used without
definition below to have the meanings ascribed to them in the Credit Agreement.
Upon satisfaction of the conditions precedent to effectiveness set forth below,
certain Defaults and/or Events of Default shall be waived, all on and subject to
the conditions and limitations set forth below:

      1.   Interim Waivers with Respect to Certain Financial Covenants. Any
Default or Event of Default occasioned solely by the failure of the Borrower to
be in compliance with Section 7.1(a), 7.1(b), 7.1(e) or 7.1(f) of the Credit
Agreement as of, but only as of, September 30, 2000 is hereby waived but only
for the period to and including the earlier of (i) the Sub Debt Payment Notice
Date and (ii) January 8, 2001 (the earlier of such dates being hereinafter
referred to as the "Expiry Date") and effective on the Expiry Date such waivers
shall be and become null and void and, absent a further waiver of such Defaults
and/or Events of Default by the Required Banks such Defaults and Events of
Default shall once again arise all as though the waivers provided for in this
Section 1 had never been given. The term "Sub Debt Payment Notice Date" shall
mean the date the Borrower serves a notice on the Administrative Agent pursuant
to Section 3 hereof.

      2.   Audit Report Qualifications. Any Default or Event of Default
occasioned solely by the consolidated audit report of the Borrower for the
fiscal year ended September 30, 2000 containing a "going concern" or like
qualification or exception is hereby waived.

      3.   Payments in Respect of Designated Subordinated Debt. The Borrower
agrees that it will not make any payment of principal or interest on Designated
Subordinated Debt or acquire or retire the same or make any deposit of funds or
other property with the trustee for the holders of Designated Subordinated Debt
for the purpose of funding any such payment, acquisition, or retirement unless
the Requisite Lenders have agreed thereto or the Borrower shall have given the
Administrative Agent not less than eight Business Days prior written notice (a
"Sub Debt Payment Notice") of its intention so to do and confirmed that the
Administrative Agent has
<PAGE>

actually received such notice at least eight Business Days prior to funding any
such payment. The term "Designated Subordinated Debt" shall mean the Borrower's
9 3/4% senior subordinated notes due 2007 and issued under an indenture dated as
of December 22, 1997 with The Bank of New York as original trustee.

      4.   No Prepayments. The Borrower will not and will not permit any
Subsidiary to voluntarily prepay or acquire or retire any of the principal or
interest on any Indebtedness (other than Indebtedness hereunder) prior to the
time the same becomes due and payable.

      5.  Availability. In consideration of the waivers hereinabove provided,
the Company has agreed to restrict utilizations under the Revolving Credit
Commitments to amounts based upon its cash flow projections as heretofore
presented to the Lenders. Accordingly, the Borrower hereby agrees that, anything
contained in the Credit Agreement to the contrary notwithstanding, during each
of the periods specified below Total Revolving Extensions of Credit shall not
exceed the lesser of the Adjusted Revolving Credit Commitments or the amount
specified below and the Borrower shall have no right to request a credit
utilization if after giving effect thereto such credit utilizations would exceed
such amounts:

During the period                      Total Revolving Extensions of Credit
                                       shall not exceed following amounts plus
                                       current outstanding amount of letters of
                                       credit
-------------------------------------------------------------------------------
Effectiveness of this Agreement
 to 12/9/00                                          $ 85,000,000
-------------------------------------------------------------------------------
12/10/00 - 1/8/01                                    $125,000,000
-------------------------------------------------------------------------------

      6.  Interest Periods.  Anything contained in the Credit Agreement to the
contrary notwithstanding, the Borrower shall not be entitled to select new
Interest Periods and Eurodollar Loans shall be converted into Base Rate Loans on
the expiration of the Interest Periods currently applicable thereto.

      7.   Mandatory Prepayments.  Anything contained in the Credit Agreement
to the contrary notwithstanding, the proceeds of any mandatory prepayment made
pursuant to Section 2.12 of the Credit Agreement shall be applied pro rata to
the reduction of the Term Loans and the Revolving Credit Commitments and upon
execution of counterparts hereof by the Majority Facility Lenders of the Tranche
B Term Loans Section 2.18 (d) of the Credit Agreement shall be of no force or
effect.

      8. The Restructuring. The Borrower and certain of the Lenders holding
66 2/3% of the obligations of the Borrower under the Credit Agreement which also
constitute a majority in number of the Lenders (the "Requisite Restructuring
Lenders") have agreed to the Summary of Terms and Conditions for Refinancing or
Restructuring of the Credit Agreement dated December 8, 2000 ("Restructuring
Term Sheet") which is attached hereto and incorporated herein. The Borrower and
such Lenders have agreed to all of the terms in the Restructuring Term Sheet
with the following understanding: (a) the Borrower and Requisite Restructuring
Lenders must further agree to the specific terms of (i) the financial covenants;
(ii) the borrowing
<PAGE>

base and the definition of excess cash flow; and (iii) the permitted level of
Capital Expenditures per year (which are to be justified to the Lenders by means
of a Capital Expenditure Program) (collectively hereinafter referred to as
"Implementation Issues"); (b) the Borrower and the Lenders have reserved their
rights and do not agree on two specific pricing issues in the Restructuring Term
Sheet; namely with regard to Pricing Grid, Annex 1A to the Restructuring Term
Sheet, (i) whether there should be eurodollar pricing available at Level III
and, if so, whether the pricing margin should be 4.75% per annum, and (ii)
whether there should be a second Restructuring Fee in the amount of $1,550,000
payable upon the earlier of receipt of the Net Proceeds of an Equity Offering or
one year anniversary of the Effective Date of the Plan of Reorganization for the
Borrower (hereinafter collectively referred to as "Reserved Pricing Matters").
The Borrower, the Lenders and Agent will continue to negotiate the
Implementation Issues and Reserved Pricing Matters in an effort to have them
agreed to by the Borrower and the Requisite Restructuring Lenders The Borrower
acknowledges and agrees that notwithstanding any terms or provisions of this
Agreement or the Credit Agreement, the undersigned Lenders have not waived their
right to receive and shall receive, and the Borrower agrees to pay, interest on
the Loans and the obligations under the Credit Agreement and in respect of
Letters of Credit on and after December 15, 2000, at the rate per annum
applicable under Section 2.15 (d) of the Credit Agreement to amounts overdue
(any notice or other actions by the Lenders necessary to accomplish this are
hereby waived or deemed given by the Borrower). If the Implementation Issues and
Reserved Pricing Matters have been resolved to the satisfaction of the Requisite
Restructuring Lenders and the Agent by January 8, 2001 and the Borrower has
received assurances from Holders of the 9 3/4% Senior Subordinated Notes Due
2007 of the Company ("Subordinated Notes") who in aggregate hold at least
$140,000,000 principal amount of the Subordinated Notes ("Requisite
Restructuring Noteholders") that the Requisite Restructuring Noteholders have
agreed to the terms of the Restructuring Term Sheet by January 8, 2001 then the
immediately preceding sentence shall be voided ab initio as if it had never been
part of this Agreement. Interest in excess of that which would have accrued on
the Loans but for this Agreement shall be due and payable on January 8, 2001
unless voided pursuant to the foregoing. The Borrower represents and warrants
that it has received written assurances from representatives of Informal
Committee of Subordinated Notes who hold collectively at least $140,000,000
principal amount of Subordinated Notes ("Committee") that each member of the
Committee has approved the Restructuring Term Sheet subject to the Reserved
Pricing Matters and the Implementation Issues.

      9. Miscellaneous. Except as specifically modified hereby, all of the
terms, conditions and provisions of the Credit Agreement shall stand and remain
unchanged and in full force and effect. No reference to this Interim Waiver
Agreement need be made in any instrument or document at any time referring to
the Credit Agreement, a reference to the Credit Agreement in any of such to be
deemed to be a reference to the same as modified hereby. This Interim Waiver
Agreement may be executed in counterparts and by separate parties hereto on
separate counterparts, each to constitute an original but all of which shall
constitute one and the same instrument. The Borrower and the Subsidiary
Guarantors hereby confirm that all representations and warranties made by them
in the Loan Documents are true and correct as of the date hereof except to the
extent that any of same expressly relate to any earlier date and acknowledge
that their obligations under the Loan Documents are justly and truly owing
without defense, offset or counterclaim. The waivers provided for herein shall
be strictly construed and limited as
<PAGE>

hereinafter provided. This Interim Waiver Agreement shall become effective upon
receipt by the Administrative Agent of counterparts hereof which, taken
together, bear the signatures of the Borrower, the Guarantors, the Requisite
Restructuring Lenders and the Majority Revolving Credit Facility Lenders,
provided however that upon satisfaction of such conditions such effectiveness
shall relate back to and be deemed effective as of September 30, 2000 all with
the same force and effect as though such conditions precedent to effectiveness
had been satisfied as of and on such date. This Interim Waiver Agreement shall
be deemed to be a "Loan Document" for purposes of the Credit Agreement and the
other Loan Documents. This Interim Waiver Agreement shall be construed in
accordance with and governed by the laws of the state of New York.
<PAGE>

Dated and to become effective as of this 30th day of September 2000.

                                IMPERIAL SUGAR COMPANY (formerly known as
                                 Imperial Holly Corporation)


                                By    /s/ Mark Q. Huggins
                                   -------------------------------------------
                                  Its       CFO
                                      ----------------------------------------


                                BIOMASS CORPORATION

                                By:    /s/ W. F. SCHWER
                                   -------------------------------------------
                                Name:  W. F. SCHWER
                                Title: President


                                CROWN EXPRESS, INC.

                                By:    /s/ W. F. Schwer
                                   -------------------------------------------
                                Name:  W. F. Schwer
                                Title: President


                                DIAMOND CRYSTAL BRANDS, INC.

                                By:    /s/ W. F. Schwer
                                   -------------------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President

                                DIAMOND CRYSTAL BRANDS, LP

                                By: Savannah Molasses & Specialties
                                    Company
                                    Its sole General Partner


                                By:    /s/ W. F. Schwer
                                   -------------------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President




<PAGE>

                                DIAMOND CRYSTAL HOLDINGS, INC.

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                DIAMOND CRYSTAL SPECIALTY FOODS, INC.

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                DIXIE CRYSTALS FOODSERVICE, INC.

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                DSLT HOLDING COMPANY

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                FOOD CARRIER, INC.

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                FORT BEND UTILITIES COMPANY

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President



<PAGE>


                                GREAT LAKES SUGAR COMPANY

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                HOLLY NORTHWEST COMPANY

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: President


                                HOLLY SUGAR CORPORATION

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                IMPERIAL DISTRIBUTING, INC.

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: President


                                IMPERIAL-SAVANNAH, LP

                                By:  Savannah Molasses & Specialties Company
                                     Its sole General Partner

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                IMPERIAL SWEETENERS DISTRIBUTORS, INC.

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President



<PAGE>


                                KING PACKAGING COMPANY, INC.

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                LIMESTONE PRODUCTS COMPANY

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: President


                                MENU MAGIC FOODS, INC.

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                MICHIGAN SUGAR COMPANY

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                PHOENIX PACKAGING CORPORATION

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                RAGUS HOLDINGS, INC.

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: President


<PAGE>


                                SAVANNAH FOODS INDUSTRIAL, INC.

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                SAVANNAH FOODS & INDUSTRIES, INC.

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                SAVANNAH INVESTMENT COMPANY

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: President


                                SAVANNAH MOLASSES & SPECIALTIES COMPANY

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                SAVANNAH SUGAR REFINING CORPORATION

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Sr. Vice President


                                WHOLESOME SWEETENERS GROUP, LTD.

                                By:   Wholesome Sweeteners, LLC
                                      Its sole General Partner

                                By:    /s/ W. F. Schwer
                                   ----------------------------------
                                Name:  W. F. Schwer
                                Title: Manager




<PAGE>

                                WHOLESOME SWEETENERS, LLC

                                By:    /s/ W. F. Schwer
                                    -----------------------------------
                                Name:  W. F. Schwer
                                Title: Member Manager
<PAGE>

                                MONUMENT CAPITAL LTD.

                                By:    Alliance Capital Management L.P., as
                                       Investment Manager

                                By:    Alliance Capital Management
                                       Corporation, as General Partner

                                By:    /s/ Joel Serebransky
                                    -----------------------------------------
                                Name:  Joel Serebransky
                                Title: Senior Vice President


                                OAK MOUNTAIN LIMITED

                                By:    Alliance Capital Management L.P., as
                                       Investment Manager

                                By:    Alliance Capital Management
                                       Corporation, as General Partner

                                By:    /s/ Joel Serebransky
                                    -----------------------------------------
                                Name:  Joel Serebransky
                                Title: Senior Vice President



<PAGE>



                                THE BANK OF NEW YORK

                                By:    /s/ Ronald R. Reedy
                                   ----------------------------------
                                Name:  Ronald R. Reedy
                                Title: Vice President




<PAGE>


                                BLACK DIAMOND CLO 1998--1 LTD.

                                By:    /s/ Debbie Boggess
                                   ----------------------------------
                                Name:  Debbie Boggess
                                Title: Director


                                BLACK DIAMOND INTERNATIONAL FUNDING LTD.

                                By:    /s/ Debbie Boggess
                                   ----------------------------------
                                Name:  Debbie Boggess
                                Title: Director







<PAGE>

                                FARM CREDIT BANK OF WICHITA

                                By:    /s/ Travis W. Ball
                                    -----------------------------------
                                Name:  Travis W. Ball
                                Title: Assistant Vice President


<PAGE>


                                FIRST UNION NATIONAL BANK

                                By:    /s/ Thomas M. Cambern
                                    -----------------------------------
                                Name:  Thomas M. Cambern
                                Title: Senior Vice President



<PAGE>


                                LEHMAN COMMERCIAL PAPER INC., as
                                 Syndication Agent and as a Lender

                                By:   /s/
                                   ----------------------------------
                                Name:
                                Title:


                                HARRIS TRUST AND SAVINGS BANK, as
                                 Administrative Agent, Collateral
                                 Agent, Issuing Lender and as a
                                 Lender

                                By:    /s/ Janet Maxwell-Wickett
                                   ----------------------------------
                                Name:  Janet Maxwell-Wickett
                                Title: Vice President


<PAGE>


                                LEHMAN COMMERCIAL PAPER INC., as
                                 Syndication Agent and as a Lender

                                By:    /s/ Michelle Swanson
                                   ----------------------------------
                                Name:  Michelle Swanson
                                Title: Authorized Signatory


                                HARRIS TRUST AND SAVINGS BANK, as
                                 Administrative Agent, Collateral
                                 Agent, Issuing Lender and as a
                                 Lender

                                By:
                                   ----------------------------------
                                Name:
                                Title:

<PAGE>


                                COOPERATIEVE CENTRALE RAIFFEISEN-
                                 BOERENLEENBANK B.A., "RABOBANK
                                 NEDERLAND" NEW YORK BRANCH

                                By:
                                   ----------------------------------
                                Name:
                                Title:


                                By:    /s/ Stewart Kalish
                                   ----------------------------------
                                Name:  Stewart Kalish
                                Title: Vice President


<PAGE>


                                PILGRIM PRIME RATE TRUST

                                By: Pilgrim Investments Inc. as its
                                     Investment Manager

                                By:    /s/ Michel Prince
                                   ----------------------------------
                                Name:  Michel Prince, CFA
                                Title: Vice President


                                PILGRIM AMERICA HIGH INCOME
                                 INVESTMENTS, LTD. (as Assignee)

                                By:    /s/ Michel Prince
                                   ----------------------------------
                                Name:  Michel Prince, CFA
                                Title: Vice President


                                SEQUILS-PILGRIM 1 LIMITED

                                By: Pilgrim Investments Inc. as its
                                     Investment Manager

                                By:
                                   ----------------------------------
                                Name:
                                Title:


                                PILGRIM CLO 1999--1 LIMITED

                                By: Pilgrim Investments Inc. as its
                                     Investment Manager

                                By:
                                   ----------------------------------
                                Name:
                                Title:


<PAGE>


                                US BANCORP AG CREDIT, INC.

                                By:    /s/ Sandra A. Saver
                                   ----------------------------------
                                Name:  Sandra A. Saver
                                Title: Vice President


<PAGE>


                                WACHOVIA BANK, N.A.

                                By:    /s/ Richard H. Milhauk
                                   ----------------------------------
                                Name:  Richard H. Milhauk
                                Title: Senior Vice President



<PAGE>


                                VAN KAMPEN CLO I, LTD.

                                By: Van Kampen Management Inc., as
                                     Collateral Manager

                                By:    /s/ Darvin D. Pierce
                                   ----------------------------------
                                Name:  Darvin D. Pierce
                                Title: Vice President


                                VAN KAMPEN CLO II, LTD.

                                By: Van Kampen Management Inc., as
                                     Collateral Manager

                                By:    /s/ Darvin D. Pierce
                                   ----------------------------------
                                Name:  Darvin D. Pierce
                                Title: Vice President


                                VAN KAMPEN SENIOR INCOME TRUST

                                By:    /s/ Darvin D. Pierce
                                   ----------------------------------
                                Name:  Darvin D. Pierce
                                Title: Vice President


<PAGE>

                                FROST NATIONAL BANK

                                By:    /s/ Phil Dudley
                                   ----------------------------------
                                Name:  Phil Dudley
                                Title: Senior Vice President