<PAGE>   1
                                                                     EXHIBIT 4.1


                   Amendment No. 3 to Stockholders Agreement


                  THIS AGREEMENT is made and entered into as of March ___, 2000
by and among the persons and entities designated on Schedule 1 hereto
(individually a "Stockholder" and collectively the "Stockholders") and CMI
Industries, Inc., a Delaware corporation ("CMI").

                              Background Statement

                  A.       CMI and the holders of the $1.00 par value common
stock of CMI ("CMI Common Stock") are parties to the Stockholders Agreement
dated February 14, 1992, as amended by the Amendment No. 1 to Stockholders
Agreement dated as of December 31, 1993 and Amendment No. 2 to the Stockholders
Agreement dated as of March 1, 1996 (as so amended, the "Original Stockholders
Agreement"). Section 9.2(a) of the Original Stockholders Agreement provides
that it may be amended by the holders of three-fourths or more of the
outstanding shares of CMI Common Stock, and the Stockholders hold in the
aggregate not less than three-fourths of the outstanding shares of CMI Common
Stock.

                  B.       CMI and certain of the Stockholders are parties to a
Stock Acquisition Agreement (the "Stock Acquisition Agreement") dated as of
February 23, 2000, which provides, in part, for the sale by such Stockholders
to CMI of 994,387 shares of CMI Common Stock. The parties hereto desire to
enter into this agreement in order to amend the Original Stockholders Agreement
in light of the transactions contemplated by the Stock Acquisition Agreement.
The Original Stockholders Agreement, as amended hereby, shall be referred to
herein as the "Stockholders Agreement," and the terms defined therein and not
otherwise defined herein shall be used herein as therein defined.

                                   Agreement

                  In consideration of the foregoing, and other good and
valuable consideration, the receipt and sufficiency of which are hereby
acknowledged, the parties hereto hereby agree as follows:

                  1.       Effective Date. This Amendment No. 3 to Stockholders
Agreement shall become effective contemporaneously with the occurrence of the
Closing on the Closing Date (as those terms are defined in the Stock
Acquisition Agreement) and only if such Closing occurs. Notwithstanding the
foregoing, to the extent Article 5 of the Original Stockholders Agreement would
require any party to the Stock Acquisition Agreement to give one or more
notices regarding the transactions contemplated by the Stock Acquisition
Agreement, section 2(a) hereof shall be deemed to have become effective
immediately prior to the execution and delivery of the Stock Acquisition
Agreement. In the event the Closing under the Stock Acquisition Agreement does
not occur for any reason whatsoever, this Amendment No. 3 to Stockholders
Agreement shall be null, void and of no force or effect.

                                       1

<PAGE>   2

                  2.       Amendments to Original Stockholders Agreement.
CMI and the Stockholders hereby amend the Original Stockholders Agreement as
follows:

                  (a)      Section 4.2 of the Original Stockholders Agreement is
amended to include the following subsection (xi), which shall appear
immediately following subsection (x) of Section 4.2:

                                    "(xi)   Pursuant to the Stock Acquisition
                           Agreement between CMI and the MLCP Investors dated
                           as of February 23, 2000."

                  (b)      Subsection (a) of Section 8.1 of the Original
Stockholders Agreement is amended to delete clause (ii)(B) in its entirety and
to add a new subsection (c) immediately following subsection (b) of Section
8.1:

                                    "(ii) The provisions of Articles 2 and 3
                           shall terminate upon the sale, in one or more
                           transactions (whether or not related), by one or
                           more of the MLCP Investors of not less than an
                           aggregate of 60% of the outstanding shares of CMI
                           Common Stock held by all of the MLCP Investors
                           immediately after such Effective Date,".

                  3.       Reference to and Effect on the Stockholders
Agreement. (a) Upon the effectiveness hereof, each reference in the Original
Stockholders Agreement to "this Agreement," "hereunder," "hereof," "herein" or
words of like import referring to the Original Stockholders Agreement, and each
reference in the other documents referred to therein to "the Stockholders
Agreement," "thereunder," "thereof," "therein," or words of like import
referring to the Original Stockholders Agreement, shall mean and be a reference
to the Stockholders Agreement.

                  (b)      The Original Stockholders Agreement, as amended prior
to the date hereof and as amended hereby, shall remain in full force and effect
and is hereby ratified and confirmed.

                  4.       Certain Representations. Each of the Stockholders
hereby represents and warrants to the other Stockholders and to CMI that such
Stockholder holds the number of shares of CMI Common Stock set forth beside its
name on Schedule 1 hereto.

                  5.       Miscellaneous. This agreement shall be governed by
and construed in accordance with the laws of the State of Delaware. This
Agreement shall be binding upon and inure to the benefit of the parties hereto,
their successors and assigns, and, in accordance with Section 9.2(a) of the
Stockholders Agreement, all other parties to the Stockholders Agreement, their
successors and assigns. This agreement, together with the Original Stockholders
Agreement, constitutes the entire agreement among the parties hereto with
respect to the subject matter hereof and thereof and supersedes all prior oral
and written agreements or understandings

                                       2

<PAGE>   3

among the parties with respect thereto. This Agreement may be executed in one
or more counterparts, each of which shall be deemed an original, but all of
which shall constitute one and the same instrument.

                  IN WITNESS WHEREOF, the parties hereto have caused this
Agreement to be duly executed and delivered as of the day and date first above
written.

                              CMI INDUSTRIES, INC.


                              By:
                                 ------------------------------------------
                              Title:
                                    ---------------------------------------



                              MERRILL LYNCH CAPITAL APPRECIATION
                                PARTNERSHIP NO. VII, L.P.

                              By:  Merrill Lynch LBO Partners No. II, L.P.,
                                   as General Partner

                              By:  Merrill Lynch Capital Partners, Inc.,
                                   as General Partner of
                                   Merrill Lynch LBO Partners
                                   No. II, L.P.

                              By:
                                 ------------------------------------------
                              Title:
                                    ---------------------------------------


                              ML OFFSHORE LBO PARTNERSHIP NO. VII

                              By:  Merrill Lynch LBO Partners No. II, L.P.,
                                   as Investment General Partner

                              By:  Merrill Lynch Capital Partners, Inc.,
                                   as General Partner of
                                   Merrill Lynch LBO Partners
                                   No. II, L.P.

                              By:
                                 ------------------------------------------
                              Title:
                                    ---------------------------------------

                                       3

<PAGE>   4

                              ML EMPLOYEES LBO PARTNERSHIP NO. I, L.P.

                              By:  ML Employees LBO Managers, Inc.,
                                   as General Partner

                              By:
                                 -----------------------------------------
                              Title:
                                    --------------------------------------


                              ML IBK POSITIONS, INC.


                              By:
                                 -----------------------------------------
                              Title:
                                    --------------------------------------


                              MERCHANT BANKING L.P. NO. I

                              By:  Merrill Lynch MBP Inc.,
                                   as General Partner

                              By:
                                 -----------------------------------------
                              Title:
                                    --------------------------------------


                              MERRILL LYNCH KECALP L.P. 1986

                              By: KECALP Inc.,
                                  as General Partner

                              By:
                                 -----------------------------------------
                              Title:
                                    --------------------------------------


                              MERRILL LYNCH CAPITAL APPRECIATION
                                PARTNERSHIP NO. B-XVI, L.P.

                              By:  Merrill Lynch LBO Partners No. B-III,
                                   as General Partner

                              By:  Merrill Lynch Capital Partners, Inc.,
                                   as General Partner of Merrill Lynch LBO
                                   Partners No. B-III

                              By:
                                 -----------------------------------------

                                       4

<PAGE>   5
                              Title:
                                    --------------------------------------


                              ML OFFSHORE LBO PARTNERSHIP NO. B-XVI

                              By:  Merrill Lynch LBO Partners No. B-III,
                                   as General Partner

                              By:  Merrill Lynch Capital Partners, Inc.,
                                   as General Partner of Merrill Lynch LBO
                                   Partners No. B-III

                              By:
                                 -----------------------------------------
                              Title:
                                    --------------------------------------


                              MLCP ASSOCIATES L.P. NO. II

                              By:  Merrill Lynch Capital Partners, Inc.,
                                   as General Partner

                              By:
                                 -----------------------------------------
                              Title:
                                    --------------------------------------


                              MERRILL LYNCH KECALP L.P. 1989

                              By:  KECALP INC.,
                                   as General Partner

                              By:
                                 -----------------------------------------
                              Title:
                                    --------------------------------------


                              MERRILL LYNCH KECALP L.P. 1991

                              By:  KECALP INC.,
                                   as General Partner

                              By:
                                 -----------------------------------------
                              Title:
                                    --------------------------------------

                                       5

<PAGE>   6
                              ----------------------------------------------
                              Michael H. deHavenon


                              ----------------------------------------------
                              G. Thaddeus Williams



                              ----------------------------------------------
                              Steve F. Warren



                              ----------------------------------------------
                              J. R. Swetenburg



                              ----------------------------------------------
                              W. James Raleigh



                              ----------------------------------------------
                              Anthony J. Raffo


                              M.S. Bailey & Son, Bankers,
                                 as Trustee for the C. Bailey Dixon II Trust


                              By:
                                 -------------------------------------------
                              Title:
                                    ----------------------------------------


                              M.S. Bailey & Son, Bankers,
                                 as Trustee for Julius A. Swetenburg


                              By:
                                 -------------------------------------------
                              Title:
                                    ----------------------------------------


                              M.S. Bailey & Son, Bankers,
                                 as Trustee for Ada S. Cain

                                       6

<PAGE>   7

                              By:
                                 -------------------------------------------
                              Title:
                                    ----------------------------------------


                              M.S. Bailey & Son, Bankers,
                                 as Trustee for John Richard Swetenburg, III


                              By:
                                 -------------------------------------------
                              Title:
                                    ----------------------------------------



                              ----------------------------------------------
                              Russell G. Vance



                              ----------------------------------------------
                              James D. Barton



                              ----------------------------------------------
                              Joseph L. Gorga



                              ----------------------------------------------
                              John T. Cavanagh



                              ----------------------------------------------
                              James H. Coleman



                              ----------------------------------------------
                              Thomas E. Davenport



                              ----------------------------------------------
                              McArthur A. George



                              ----------------------------------------------
                              Joshua T. Hamilton

                                       7

<PAGE>   8

                              PALBA & CO.,
                                as Trustee for Joshua T. & Marquita C. Hamilton
                                Irrevocable Trust u/a 9-16-94


                              By:
                                 ----------------------------------------------
                              Title:
                                    -------------------------------------------



                              -------------------------------------------------
                              William R. Hill



                              -------------------------------------------------
                              Barry L. Hooks



                              -------------------------------------------------
                              Robert E. Hooks



                              -------------------------------------------------
                              Leon J. McCullough



                              -------------------------------------------------
                              Joseph B. Nelson



                              -------------------------------------------------
                              C. Mack Parsons



                              -------------------------------------------------
                              William E. Stanton



                              -------------------------------------------------
                              H. Jerome Stuckey



                              -------------------------------------------------
                              Harry B. Sullivan

                                       8

<PAGE>   9


                              -----------------------------
                              James A. Ovenden


                              The Warren Family Partnership


                              By:
                                 --------------------------
                              Title:
                                    -----------------------

                              By:
                                 --------------------------
                              Title:
                                    -----------------------

                                       9

<PAGE>   10

                                   SCHEDULE 1

                    Stockholders and Number of Shares of CMI
                               Common Stock Owned

<TABLE>
<CAPTION>
                       Name                                                                   Number of
                       ----                                                                  Shares Owned
                                                                                             ------------
<S>                                                                                          <C>
Merrill Lynch Capital Appreciation Partnership No. VII, L.P.                                    398,900
-------------------------------------------------------------------------------------------------------
ML Offshore LBO Partnership No. VII                                                              10,142
-------------------------------------------------------------------------------------------------------
ML Employees LBO Partnership No. I, L.P.                                                          9,916
-------------------------------------------------------------------------------------------------------
ML IBK Positions, Inc.                                                                           99,401
-------------------------------------------------------------------------------------------------------
Merchant Banking L.P. No. I                                                                       8,902
-------------------------------------------------------------------------------------------------------
Merrill Lynch KECALP L.P. 1986                                                                    2,857
-------------------------------------------------------------------------------------------------------
Merrill Lynch Capital Appreciation Partnership No. B-XVI, L.P.                                  270,399
-------------------------------------------------------------------------------------------------------
ML Offshore LBO Partnership No. B-XVI                                                           162,458
-------------------------------------------------------------------------------------------------------
MLCP Associates L.P. No. II                                                                       4,350
-------------------------------------------------------------------------------------------------------
Merrill Lynch KECALP L.P. 1989                                                                   13,531
-------------------------------------------------------------------------------------------------------
Merrill Lynch KECALP L.P. 1991                                                                   13,531
-------------------------------------------------------------------------------------------------------
Michael H. deHavenon                                                                              4,081
-------------------------------------------------------------------------------------------------------
G. Thaddeus Williams                                                                             97,000
-------------------------------------------------------------------------------------------------------
Steve F. Warren                                                                                  60,000
-------------------------------------------------------------------------------------------------------
J. R. Swetenburg                                                                                 40,002
-------------------------------------------------------------------------------------------------------
W. James Raleigh                                                                                 20,000
-------------------------------------------------------------------------------------------------------
Anthony J. Raffo                                                                                  5,000
-------------------------------------------------------------------------------------------------------
M.S. Bailey & Son, Bankers, as Trustee for the C. Bailey Dixon,                                  46,700
II Trust
-------------------------------------------------------------------------------------------------------
M.S. Bailey & Son, Bankers, as Trustee for Julius A.                                             26,666
Swetenburg
-------------------------------------------------------------------------------------------------------
M.S. Bailey & Son, Bankers, as Trustee for Ada S. Cain                                           26,666
-------------------------------------------------------------------------------------------------------
</TABLE>

                                       10

<PAGE>   11

<TABLE>

<S>                                                                                              <C>
M.S. Bailey & Son, Bankers, as Trustee for John Richard                                          26,666
Swetenburg, III
-------------------------------------------------------------------------------------------------------
Russell G. Vance                                                                                 46,650
-------------------------------------------------------------------------------------------------------
James D. Barton                                                                                   5,000
-------------------------------------------------------------------------------------------------------
Joseph L. Gorga                                                                                   5,000
-------------------------------------------------------------------------------------------------------
John T. Cavanagh                                                                                 20,000
-------------------------------------------------------------------------------------------------------
James H. Coleman                                                                                 10,000
-------------------------------------------------------------------------------------------------------
Thomas E. Davenport                                                                              10,000
-------------------------------------------------------------------------------------------------------
McArthur A. George                                                                                6,000
-------------------------------------------------------------------------------------------------------
Joshua T. Hamilton                                                                               16,000
-------------------------------------------------------------------------------------------------------
Joshua T. & Marquita C. Hamilton Irrevocable Trust u/a 9-16-94                                    4,000
-------------------------------------------------------------------------------------------------------
William R. Hill                                                                                   5,000
-------------------------------------------------------------------------------------------------------
Barry L. Hooks                                                                                   10,000
-------------------------------------------------------------------------------------------------------
Robert E. Hooks                                                                                   6,000
-------------------------------------------------------------------------------------------------------
Leon J. McCullough                                                                                3,000
-------------------------------------------------------------------------------------------------------
Joseph B. Nelson                                                                                  6,000
-------------------------------------------------------------------------------------------------------
C. Mack Parsons                                                                                   3,500
-------------------------------------------------------------------------------------------------------
William E. Stanton                                                                                5,000
-------------------------------------------------------------------------------------------------------
H. Jerome Stuckey                                                                                 6,000
-------------------------------------------------------------------------------------------------------
Harry B. Sullivan                                                                                 5,000
-------------------------------------------------------------------------------------------------------
James A. Ovenden                                                                                 10,000
-------------------------------------------------------------------------------------------------------
The Warren Family Partnership                                                                    60,000
-------------------------------------------------------------------------------------------------------
</TABLE>

                                       11