<PAGE>

                                                                 Exhibit 4(b)(3)

AGREEMENT OF RESIGNATION, APPOINTMENT AND ACCEPTANCE, dated as of November 2,
2000 by and among Imperial Sugar Company, formerly known as Imperial Holly
Corporation, a corporation duly organized and existing under the laws of the
State of Texas and having its principal office at One Imperial Square, P.O. Box
9, Sugar Land, Texas 77487 (the "Company"), The Bank of New York, a banking
corporation duly organized and existing under the laws of the State of New York
and having its principal corporate trust office at 101 Barclay Street 21 W, New
York, New York 10286, (the "Resigning Trustee") and United States Trust Company
of New York, a banking corporation and trust company duly organized and existing
under the laws of the State of New York and having its principal corporate trust
office at 114 West 47/th/ Street, New York, NY 10036 (the "Successor Trustee").

                                   RECITALS:

               WHEREAS, there was originally authorized and issued $250,000,000
aggregate principal amount of the Company's 9 3/4% Senior Subordinated Notes due
2007 under an indenture dated as of December 22, 1997 as amended by the
Supplemental Indenture dated as of September 30, 1998 by and between the Company
and the Resigning Trustee (said Notes are hereinafter referred to as
"Securities" and said indenture is hereinafter referred to as the "Indenture");

               WHEREAS, Section 7.08 of the Indenture provides that the Trustee
may at any time resign by giving written notice of such resignation to the
Company, effective upon the acceptance by a successor Trustee of its appointment
as a successor Trustee;

               WHEREAS, Section 7.08 of the Indenture provides that any
successor Trustee appointed in accordance with the Indenture shall execute,
acknowledge and deliver to the Company and to its predecessor Trustee an
instrument accepting such appointment under the Indenture, and thereupon the
resignation of the predecessor Trustee shall become effective and such successor

                                       1
<PAGE>

Trustee, without any further act, deed or conveyance, shall become vested with
all rights, powers, duties and obligations of the predecessor Trustee;

               WHEREAS, the Resigning Trustee was appointed Registrar, Paying
Agent and Note Custodian by the Company;

               WHEREAS, the Company desires to appoint Successor Trustee as
Trustee, Registrar, Paying Agent and Note Custodian;

               WHEREAS, Successor Trustee is willing to accept such appointment
as successor Trustee, Registrar, Paying Agent and Note Custodian under the
Indenture;

               NOW, THEREFORE, the Company, Resigning Trustee and Successor
Trustee, for and in consideration of the premises and of other good and valuable
consideration, the receipt and sufficiency of which are hereby acknowledged,
hereby consent and agree as follows:


                                  ARTICLE ONE

                             THE RESIGNING TRUSTEE
                             ---------------------

               SECTION I. Pursuant to Section 7.08 of the Indenture, Resigning
Trustee hereby notifies the Company that Resigning Trustee is hereby resigning
as Trustee, Registrar, Paying Agent and Note Custodian under the Indenture.

               SECTION II. Resigning Trustee hereby represents and warrants to
Successor Trustee that:

               (a)       No covenant or condition contained in the Indenture has
                 been waived by Resigning Trustee or, to the best of the
                 knowledge of the Responsible Officers of Resigning Trustee's
                 Corporate Trust Group, by the Holders of the percentage in
                 aggregate principal amount of the Securities required by the
                 Indenture to effect any such waiver.

                                       2
<PAGE>

               (b)       There is no action, suit or proceeding pending or, to
                 the best of the knowledge of the Responsible Officers of
                 Resigning Trustee's Corporate Trust Group, threatened against
                 Resigning Trustee before any court or any governmental
                 authority arising out of any action or omission by Resigning
                 Trustee as Trustee under the Indenture.

               (c)       As of the effective date of this Agreement, Resigning
                 Trustee will hold no property or money under the Indenture.

               (d)       Pursuant to Section 2.02 of the Indenture, Resigning
                 Trustee duly authenticated and delivered, on December 22, 1997
                 $250,000,000 aggregate principal amount of Securities of which
                 $250,000,000 is outstanding.

               (e)       Each person who so authenticated the Securities was
                 duly elected, qualified and acting as an officer of Resigning
                 Trustee and empowered to authenticate the Securities at the
                 respective times of such authentication and the signature of
                 such person or persons appearing on such Securities is each
                 such person's genuine signature.

               (f)       This Agreement has been duly authorized, executed and
                 delivered on behalf of Resigning Trustee and constitutes its
                 legal, valid, binding and enforceable obligation.

               (g)       To the best of the knowledge of the responsible
                 Officers of Resigning Trustee's Corporate Trust Group, no event
                 has occurred and is continuing which is, or after notice or
                 lapse of time, or both, would become, an Event of Default under
                 Section 6.01 of the Indenture.

               SECTION III. Resigning Trustee hereby assigns, transfers,
delivers and confirms to Successor Trustee all right, title and interest of
Resigning Trustee in and to the trust under the

                                       3
<PAGE>

Indenture and all the rights, powers and trusts of the Trustee under the
Indenture. Resigning Trustee shall execute and deliver such further instruments
and shall do such other things as Successor Trustee may reasonably require so as
to more fully and certainly vest and confirm in Successor Trustee all the
rights, trusts and powers hereby assigned, transferred, delivered and confirmed
to Successor Trustee as Trustee, Registrar and Paying Agent.

               SECTION IV. Resigning Trustee hereby acknowledges payment or
provision for payment in full by the Company of compensation for all services
rendered by Resigning Trustee under Section 7.07 of the Indenture and
reimbursement in full by the Company of the expenses, disbursements and advances
incurred or made by Resigning Trustee in accordance with the provisions of the
Indenture. Resigning Trustee transfers all property held by it pursuant to
Section 7.08 of the Indenture and acknowledges that it relinquishes any lien it
may have upon all property or funds held or collected by it to secure any
amounts due it pursuant to the provisions of Section 7.07 of the Indenture.

               SECTION V. Resigning Trustee shall deliver to Successor Trustee,
 as of or immediately after the effective date hereof, all of the documents
 listed on Exhibit A hereto.

                                  ARTICLE TWO

                                  THE COMPANY
                                  -----------

               SECTION I.   The Company hereby accepts the resignation of
Resigning Trustee as Trustee, Registrar and Paying Agent under the Indenture.

               SECTION II.  The Company hereby appoints Successor Trustee as
Trustee, Registrar, and Paying Agent and Note Custodian under the Indenture to
succeed to, and hereby vests Successor Trustee with, all the rights, powers,
duties and obligations of Resigning Trustee under the Indenture with like effect
as if originally named as Trustee, Registrar, Paying Agent and Note Custodian
in the Indenture.

                                       4
<PAGE>

               SECTION III. The Company hereby represents and warrants to
Resigning Trustee and Successor Trustee that:

               (a)     The Company is a corporation duly and validly organized
                 and existing pursuant to the laws of the State of Texas.

               (b)     The Indenture was validly and lawfully executed and
                 delivered by the Company and the Securities were validly issued
                 by the Company.

               (c)     The Company has performed or fulfilled prior to the date
                 hereof, and will continue to perform and fulfill after the date
                 hereof, in all material respects each covenant, agreement,
                 condition, obligation and responsibility under the Indenture.

               (d)     No event has occurred and is continuing which is, or
                 after notice or lapse of time, or both, would become, an Event
                 of Default under Section 6.01 of the Indenture.

               (e)     No covenant or condition contained in the Indenture has
                 been waived by Company or, to the best of Company's knowledge,
                 by Holders of the percentage in aggregate principal amount of
                 the Securities required to effect any such waiver.

               (f)     There is no action, suit or proceeding pending or, to the
                 best of Company's knowledge, threatened against the Company
                 before any court or any governmental authority arising out of
                 any action or omission by the Company under the Indenture.

               (g)     This Agreement has been duly authorized, executed and
                 delivered on behalf of Company and constitutes its legal,
                 valid, binding and enforceable obligation.

                                       5
<PAGE>

               (h)     All conditions precedent relating to the appointment of
                 United States Trust Company of New York as Successor Trustee
                 under the Indenture have been complied with by the Company.

                                 ARTICLE THREE

                             THE SUCCESSOR TRUSTEE
                             ---------------------

               SECTION I. Successor Trustee hereby represents and warrants to
 Resigning Trustee and to the Company that:

               (a)     Successor Trustee is not disqualified under the
                 provisions of Article Seven of the Indenture and is eligible
                 under the provisions of Section 7.10 of the Indenture to act as
                 Trustee under the Indenture.

               (b)     This Agreement has been duly authorized, executed and
                 delivered on behalf of Successor Trustee and constitutes its
                 legal, valid, binding and enforceable obligation.

               SECTION II. Successor Trustee hereby accepts its appointment as
successor Trustee, Registrar and Paying Agent under the Indenture and accepts
the rights, powers, duties and obligations of Resigning Trustee as Trustee under
the Indenture, upon the terms and conditions set forth therein, with like effect
as if originally named as Trustee under the Indenture.

               SECTION III. Promptly after the effective date of this Agreement,
the Successor Trustee shall cause a notice, substantially in the form of Exhibit
B annexed hereto, to be sent to each Holder of the Securities in accordance with
the provisions of Section 7.08 of the Indenture.

               SECTION IV. References in the Indenture to "Corporate Trust
Office" or other similar terms shall be deemed to refer to the Corporate Trust
Office of Successor Trustee 114 West 47/th/ Street, New York, NY 10036 or any
other office of Successor Trustee at which, at any particular time, its
corporate trust business shall be principally administered.

                                       6
<PAGE>

                                 ARTICLE FOUR

                                 MISCELLANEOUS
                                 -------------

               SECTION I. Except as otherwise expressly provided herein or
unless the context otherwise requires, all terms used herein which are defined
in the Indenture shall have the meaning assigned to them in the Indenture.

               SECTION II. This Agreement and the resignation, appointment and
acceptance effected hereby shall be effective as of the opening of business on
November 2,2000.

               SECTION III The Company acknowledges its obligation set forth in
Section 7.07 of the Indenture to indemnify Resigning Trustee for, and to hold
Resigning Trustee harmless against, any loss, liability and expense incurred
without negligence or bad faith on the part of the Resigning Trustee and arising
out of or in connection with the acceptance or administration of the trust
evidenced by the Indenture (which obligation shall survive the execution
hereof). The Company acknowledges and reaffirms its obligations to the Successor
Trustee under the Indenture as set forth in Section 7.07 of the Indenture, which
obligations shall survive the execution hereof.

               SECTION IV. This Agreement shall be governed by and construed in
accordance with the laws of the State of New York.

               SECTION V. This Agreement may be executed in any number of
counterparts, each of which shall be an original, but such counterparts shall
together constitute but one and the same instrument.

               SECTION VI. The Company, Resigning Trustee and Successor Trustee
hereby acknowledge receipt of an executed and acknowledged counterpart of this
Agreement and the effectiveness thereof.



                                       7
<PAGE>

     IN WITNESS WHEREOF, the parties hereby have caused this Agreement of
Resignation, Appointment and Acceptance to be duly executed and acknowledged and
their respective seals to be affixed thereunto and duly attested all as of the
day and year first above written.

                                      Imperial Sugar Company

                                      By: /s/ Karen L. Mercer
                                         ----------------------
                                         Name:  Karen L. Mercer
                                         Title: VP and Treasurer

                                      The Bank of New York, as Resigning Trustee


                                      By: /s/ Irene Siegel
                                         ----------------------
                                         Name:  Irene Siegel
                                         Title: Vice President

                                      United States Trust Company of New York,
                                        as Successor Trustee

                                      By: /s/ Bill Barber
                                         ----------------------
                                         Name:  Bill Barber
                                         Title: Authorized Signatory

                                       8