<PAGE>
 
                                                                    EXHIBIT 4.12

                           IMPERIAL HOLLY CORPORATION

                     NONEMPLOYEE DIRECTOR COMPENSATION PLAN


1.  Purpose
    -------

     The purpose of this Nonemployee Director Compensation Plan (the "Plan") of
Imperial Holly Corporation (the "Company") is to promote ownership in the
Company by outside directors of the Company whose services are considered
essential to the Company's continued progress and thus to provide them with a
further incentive to continue to serve as directors of the Company. The Plan is
also intended to assist the Company through utilization of the benefits
provided by the Plan to attract and retain experienced and qualified candidates
to fill vacancies in the Board of Directors (the "Board") which may occur from
time to time.  All awards under this Plan are subject to approval of the Plan by
the affirmative votes of the holders of a majority of the outstanding shares of
the Company's Common Stock, present or represented and entitled to vote at the
1996 annual meeting of the Company's shareholders; provided that, if such
approval is not obtained at the 1996 annual meeting of the Company's
shareholders, this Plan shall terminate and cease to be of any further force or
effect.

2.  Participation in the Plan
    -------------------------

     The Directors of the Company who are not employees of the Company or any
affiliate of the Company, including, without limitation, the Chairman of the
Board ("Eligible Directors"), shall be eligible to participate in the Plan;
provided, however, that the recipient of an award must be serving as an Eligible
Director on the date the award is granted.

3.  Stock Subject to the Plan
    -------------------------

     The stock subject to the Plan initially shall consist of 500,000 shares of
authorized and unissued Common Stock, without par value, of the Company ("Common
Stock").

4.  Stock Awards
    ------------

     On July 26, 1996, and on each subsequent annual election of Directors each
year during the term of this Plan (the "Award Date"), each Eligible Director
shall be issued a number of shares of Common Stock equal to (i) 167% of the
otherwise applicable cash annual retainer payable to such Director, divided by
(ii) the Fair Market Value of the Common Stock on such Award Date, rounded to
the nearest whole share.  Common Stock will not be awarded under this Plan for
committee or Director meeting fees.  For purposes of this Plan, the "Fair Market
Value" of a share on a particular date shall be deemed to be the mean between
the highest and lowest sales price per share of Common Stock on the American
Stock Exchange or, in the discretion of the Board, any other national stock
exchange or transaction reporting system on which Common Stock is listed or
quoted, or if Common Stock is not listed or quoted on any national stock
exchange or transaction reporting system, the mean between the highest closing
bid and lowest closing asked price for Common Stock as reported by the National
Association of Securities Dealers NASDAQ

                                      -1-
<PAGE>
 
System, or if not reported by such system, the mean between the closing bid and
asked price as quoted by such quotation source as shall be designated by the
Board on that date.  If there shall have been no sale on the date in question,
Fair Market Value shall be determined by reference to the last preceding date on
which such a sale or sales were so reported.

     A stock certificate evidencing such shares shall be issued by the Company
to the Eligible Director as soon as practicable after determination of the Fair
Market Value on the Award Date, and such shares, when issued, shall be duly
authorized, validly issued, fully paid and nonassessable.

5.  Stock Restrictions
    ------------------

     Shares of Common Stock issued to an Eligible Director in accordance with
the Plan may not be sold, assigned, transferred, pledged, hypothecated or
otherwise disposed of, until the earliest of the Eligible Director's death,
disability or cessation of status as a Director or the occurrence of a Change in
Control (as defined below) of the Company.  Certificates for shares of Common
Stock issued pursuant to the Plan shall bear an appropriate legend referring to
the restriction.  Any attempt to dispose of any such shares of Common Stock in
contravention of the foregoing restriction shall be null and void and without
effect.  For purposes of this Paragraph 5, a "Change in Control" shall be deemed
to have occurred if any of the following shall have taken place:  (i) change in
control is reported by the Company in response to either Item 6(e) of Schedule
14A of Regulation 14A promulgated under the Securities Exchange Act of 1934, as
amended (the "Exchange Act"), or Item 1 of Form 8-K promulgated under the
Exchange Act; (ii) any "person" (as such term is used in Sections 13(d) and
14(d)(2) of the Exchange Act) is or becomes the "beneficial owner" (as defined
in Rule 13d-3 under the Exchange Act), directly or indirectly, of securities of
the Company representing 40% or more of the combined voting power of the
Company's then-outstanding securities; or (iii) following the election or
removal of directors, a majority of the Board of Directors consists of
individuals who were not members of the Board of Directors two years before such
election or removal, unless the election of each director who was not a director
at the beginning of such two-year period has been approved in advance by
directors representing at least a majority of the directors then in office who
were directors at the beginning of the two-year period.

6.  Assignment
    ----------

     The rights and benefits of an Eligible Director under this Plan may not be
assigned, and any attempted assignment of such rights and benefits shall be null
and void.

7.  Limitation of Rights
    --------------------

A.  NO RIGHT TO CONTINUE AS A DIRECTOR. Neither the Plan, nor the granting of an
    award nor any other action taken pursuant to the Plan, shall constitute or
    be evidence of any agreement or understanding, express or implied, that the
    Company will retain an Eligible Director for any period of time, or at any
    particular rate of compensation.

B.  SHAREHOLDER'S RIGHTS. An Eligible Director shall have no rights as a
    shareholder until the date of the issuance to the Eligible Director of a
    stock certificate for the Common Stock awarded under the terms of the Plan,
    and no adjustment will be made for dividends or other rights for which the
    record date is prior to the date of such issuance.

                                      -2-
<PAGE>
 
8.  Changes in Present Stock
    ------------------------

A.  CORPORATE ACTS. The existence of this Plan shall not affect in any manner
    the right or power of the Company or its shareholders to make or authorize
    any or all adjustments, recapitalizations, reorganizations or other changes
    in the capital stock of the Company or its business or any merger or
    consolidation of the Company, or any issue of bonds, debentures, preferred
    or prior preference stock (whether or not such issue is prior to, on a
    parity with, or junior to the Common Stock) or the dissolution or
    liquidation of the Company, or any sale or transfer of all or any part of
    its assets or business, or any other corporate act or proceeding of any
    kind, whether or not of a character similar to that of the acts or
    proceedings enumerated above.

B.  ADJUSTMENTS. In the event of any subdivision or consolidation of outstanding
    shares of Common Stock or declaration of a dividend payable in shares of
    Common Stock or capital reorganization or reclassification or other
    transaction involving an increase or reduction in the number of outstanding
    shares of Common Stock, then the number of shares of Common Stock reserved
    under this Plan and subject to future awards of Common Stock shall be
    proportionately adjusted to reflect such transaction. Such adjustment to the
    number of shares of Common Stock shall reflect the proportional adjustment
    to the number of shares of Common Stock (or such other capital stock as may
    be issued in a reclassification) that a shareholder who owned an equivalent
    number of shares immediately before the happening of any of the events
    described in the preceding sentence would have owned or been entitled to
    receive after the happening of any of such events. In the event of any
    consolidation or merger of the Company with another corporation or entity or
    the adoption by the Company of a plan of exchange affecting the Common Stock
    or any distribution to holders of Common Stock of securities or property
    (other than cash dividends or dividends payable in Common Stock), the Board
    shall make such adjustments as it may deem equitable, including adjustments
    to avoid fractional shares, to give proper effect to such event; provided
    that such adjustments shall only be such as are necessary to maintain the
    proportionate interest of the Eligible Directors.

9.  Effective Date and Duration of the Plan
    ---------------------------------------

     The Plan shall take effect upon approval by the shareholders of the Company
at the 1996 annual meeting of shareholders.  The Plan shall terminate when all
Common Stock subject to the Plan is awarded (unless earlier discontinued by the
Board).  If, on a date on which Common Stock would normally be awarded, there is
not a sufficient number of shares available to grant each person otherwise
eligible to receive an award on that date the full number of shares to which he
or she would normally be entitled, shares shall be prorated among Eligible
Directors according to the number of shares available on such date of grant.
Such Eligible Directors shall be deemed to have received the full amount due to
them in Common Stock on such date of grant; provided, however that the balance
of any annual retainer fee shall be paid in cash.

                                      -3-
<PAGE>
 
10.  Amendment of the Plan
     ---------------------

     The Board may suspend or discontinue the Plan or revise or amend it in any
respect whatsoever, including, without limitation, to increase the number of
shares of Common Stock authorized under the Plan; provided, however, that (a)
after approval of the Plan by Company shareholders, no amendment or alteration
shall be effective prior to approval by the Company's shareholders of such
amendment or alteration to the extent such approval is then required pursuant to
Rule 16b-3 promulgated under the Exchange Act, in order to preserve the
applicability of any exemption provided by such rule to awards of Common Stock
under this Plan (unless the Eligible Director consents) or to the extent
shareholder approval is otherwise required by applicable legal requirements, and
(b) the Plan shall not be amended more than once every six months to the extent
such limitation is then required pursuant to Rule 16b-3 in order to preserve the
applicability of any exemption provided by such rule (or any successor provision
under the Exchange Act) to awards of Common Stock under this Plan.

11.  Taxes
     -----

     A.  Subject to Subparagraph B below, the Company may make such provisions
         as it may deem appropriate for the withholding of any taxes that it
         determines is required in connection with any Common Stock awarded
         under this Plan; provided, however, that upon approval by the Board,
         any Eligible Director may pay all or any portion of the taxes required
         or allowed to be withheld by the Company by electing to have the
         Company withhold shares of Common Stock, or by delivering previously
         owned shares of Common Stock, having a fair market value, determined in
         accordance with Paragraph 4, equal to the amount required to be
         withheld or paid. Such elections are irrevocable.

     B.  At the time of the Award Date, each Eligible Director shall inform the
         Company, in accordance with procedures established by the Company,
         whether he or she wishes to have taxes withheld at such time, pursuant
         to an election under Section 83(b) of the Internal Revenue Code of
         1986, as amended.

12.  Requirements of Law
     -------------------

     The issuance of shares of Common Stock under this Plan shall be subject to
all applicable laws, rules, and regulations and to such approvals by any
governmental agencies or national securities exchanges as may be required.

                                      -4-
<PAGE>
 
13.  Governing Law
     -------------

     This Plan and all determinations made and actions taken pursuant hereto
shall be governed by the laws of the State of Texas and construed accordingly.


     IN WITNESS WHEREOF, this Plan was adopted by the Board of Directors and
approved by the Company's shareholders on July 26, 1996.


                                        IMPERIAL HOLLY CORPORATION

                                      -5-