<PAGE>

                                                                    EXHIBIT 10.4

                             CONSULTING AGREEMENT
                             --------------------


     This Consulting Agreement (the "Agreement") is made and entered into
effective as of the 12th day of August, 1999 (the "Effective Date"), between
George William Swisher, Jr. a/k/a Bill Swisher (the "Consultant"), and CMI
Corporation, an Oklahoma corporation (the "Company").

     1.   Engagement.  As of the Effective Date, the Company shall engage the
          ----------
Consultant, and the Consultant shall accept the engagement of the Company, upon
the terms and conditions set forth in this Agreement.

     2.   Term.  The term of this Agreement shall begin as of the Effective Date
          ----
and shall continue for two (2) years, unless sooner terminated as provided for
in Section 9 below.

     3.   Duties.  The Consultant is hereby retained and appointed to assist the
          ------
Company, at the specific request of either the Board of Directors or the Chief
Executive Officer of the Company, with matters concerning sales of Company
products and/or industry/political committees and associations.  The parties
acknowledge that the Consultant is currently involved in the following industry
committees and associations and, during the term hereof, shall continue his
involvement on a non-exclusive basis as a representative of CMI:

          (i)    American Road & Transportation Builders Association;

          (ii)   Asphalt Reclaiming & Recycling Association;

          (iii)  Construction Industry Manufacturing Association;

          (iv)   Associated General Contractors of America;

          (v)    The Road Information Program;

          (vi)   National Asphalt Pavers Association;

          (vii)  International Road Federation; and

          (viii) American Concrete Paving Association.

     The parties acknowledge and agree that the Company is entitled to designate
one or more of its representatives to serve on (or to serve as a director or
member of) certain of the industry committees and associations listed above.
The Chief Executive Officer of the Company shall, at his discretion, be entitled
to select the Company's designated representative(s) to serve on (or to serve as
a director or member of) each such industry committee or association.
Notwithstanding the preceding sentence, if the Company is entitled to designate
more than one representative to serve on (or to serve as a director or member
of) any such industry committee or association, the Consultant may, at his
option, continue to be included as one of the Company's designated
representatives for such industry committee or association.
<PAGE>

     4.   Extent of Services.  The Consultant shall devote such time, attention
          ------------------
and energies to the business of the Company as are reasonably necessary to
fulfill the Consultant's obligations hereunder.  However, this Agreement shall
not prohibit the Consultant from engaging in other activities or providing
consulting services to other persons, so long as those activities do not unduly
interfere with the ability of the Consultant to carry out his duties, covenants,
and other obligations hereunder and so long as they are not in any way
inconsistent with the interests of the Company or with this Agreement.  In the
event the Consultant proposes to engage in any activity which may be deemed to
be in interference or in competition with the interests of the Company or which
may be construed to interfere with the duties of the Consultant under this
Agreement, the Consultant shall first disclose the same in writing to the Board
of Directors of the Company for its prior approval or disapproval.  If the Board
of Directors of the Company determines in good faith that the activity is in
interference or in competition with the interests of the Company or will
otherwise detract from the duties and obligations of the Consultant, then the
Company shall so inform the Consultant in writing, and the Consultant shall not
engage in such activity.

     5.   Working Facilities.  The Consultant shall be furnished with an office
          ------------------
and secretarial assistance reasonably necessary and suitable for him to carry
out his duties.  The location of said office shall be determined by mutual
agreement of the parties.  In the event said office is not located at the
Company's principal offices, appropriate arrangements will be made to insure
routine connection of calls to Mr. Swisher from the Company's main phone number.

     6.   Compensation.
          ------------

          6.1  Base Fee.  During the term of this Agreement, the Company shall
               --------
pay to the Consultant as compensation for the Consultant's services under this
Agreement a monthly consulting fee of $28,500 (the "Base Fee"), payable in
accordance with the Company's regular payroll schedule. In the event that the
annual base salary of the Chief Executive Officer of the Company is increased
during the term of this Agreement, the Consultant shall receive a proportionate
increase in his Base Fee.

          6.2  Bonus Opportunity.  In addition to the Base Fee, during the term
               -----------------
of this Agreement, the Consultant shall be entitled to participate, on the same
terms and conditions as the Chief Executive Officer of the Company, in any
cash-based incentive bonus plan established for senior management of the
Company. The amount of any bonus payable to the Consultant shall be based on the
Base Fee, and shall be proportionate to the amount of bonus received by the
Chief Executive Officer of the Company as a percentage of his base salary. With
respect to the 1999 fiscal year, the parties acknowledge and agree that the
Consultant shall be entitled to participate in the CMI 1999 Senior Management
Incentive Plan approved by the Board of Directors of the Company on February 25,
1999 as if the Consultant were still an employee of the Company. The parties
further acknowledge and agree that, if this Agreement is not terminated prior to
the expiration of its two-year term, the Consultant shall be entitled to
participate fully in any cash-based incentive bonus plan established for senior
management of the Company for the 2001 fiscal year, and no proration of any
bonus earned thereunder shall occur as a result of the fact that the term of
this Agreement expires on August 11, 2001.

                                      -2-
<PAGE>

          6.3  Health and Life Insurance Benefits.  During the term of this
               ----------------------------------
Agreement, the Company shall pay to the Consultant the amount of premium which
the Company would have paid for health benefits for the benefit of the
Consultant and his dependents to the CMI Corporation Employees Health Benefit
Plan or any other group health insurance plan hereafter maintained by the
Company for the benefit of its employees (the "Plan") if the Consultant was an
employee of the Company.  At the Consultant's option, Consultant may direct that
such payment in lieu of health premiums be made directly to the Plan to pay the
premiums required in order to continue coverage under the Plan. In addition,
during the term hereof, the Company shall continue to maintain coverage for the
Consultant under the Group Term Life Insurance Policy maintained by the Company
for the benefit of the Consultant immediately prior to the execution of this
Agreement.

          6.4  Company Car.  During the term of this Agreement, the Consultant
               -----------
shall be entitled to use the Company car he currently utilizes or a suitable
replacement.

     7.   Business Expenses.  The Company will reimburse the Consultant for all
          -----------------
ordinary and necessary expenses reasonably incurred by the Consultant at the
direction of the Board of Directors or the Chief Executive Officer of the
Company and consistent with the policies established from time to time by the
Company.

     8.   Duty of Loyalty.  The Consultant recognizes that he owes a duty of
          ---------------
loyalty and good faith to the Company and agrees that, during the term of this
Agreement, he will not in any way, directly or indirectly, by himself or through
one or more other persons or entities, take advantage of any corporate
opportunity of the Company, engage in any act of self-dealing, sell or willfully
disclose any confidential or proprietary information of the Company, or have or
obtain any material economic interest in any entity or arrangement which is in
any way competitive with the interests of the Company, without first disclosing
all material facts and details relating thereto to the Board of Directors of the
Company and obtaining the approval of the Board of Directors.

     9.   Termination. This Agreement shall terminate as provided for below:
          -----------

          9.1  By the Company. The Company may terminate this Agreement at
               --------------
any time for cause, upon notice to the Consultant setting forth the early
termination date. For purposes of this Section 9.1, the term "cause" shall mean
(i) a material breach by the Consultant of any term or provision of this
Agreement, which breach is not cured within thirty (30) days after notice of
such breach to the Consultant by the Company setting forth the facts upon which
the breach is based, (ii) a material breach by the Consultant of any term or
provision of any other agreement between the Company and the Consultant, which
breach is not cured within thirty (30) days after notice of such breach to the
Consultant by the Company setting forth the facts upon which the breach is
based, (iii) conviction of a felony, (iv) fraud by the Consultant with respect
to the business or affairs of the Company, (v) alcohol or drug abuse by the
Consultant, or (vi) the engaging in conduct by the Consultant after the
Effective Date which, in the good faith opinion of the Board of Directors of the
Company, is materially detrimental to the Company's interests or which causes or
is likely to cause a material adverse effect on the Company, which conduct and
the effects thereof are not cured to the

                                      -3-
<PAGE>

reasonable satisfaction of the Board of Directors within thirty (30) days after
notice thereof to the Consultant by the Company setting forth the facts upon
which the Board's opinion is based.

          9.2  By the Consultant.  The Consultant may terminate this Agreement
               -----------------
at any time for cause, upon notice to the Company setting forth the early
termination date.  For purposes of this Section 9.2, the term "cause" shall mean
a material breach by the Company of any term or provision of this Agreement or
any other agreement between the Company and the Consultant, which breach is not
cured within thirty (30) days after notice of such breach to the Company by the
Consultant setting forth the facts upon which the breach is based.

          9.3  Death of Consultant.  This Agreement shall terminate effective
               -------------------
immediately upon the death of the Consultant.

          9.4  Change in Control.  This Agreement shall terminate effective
               -----------------
immediately upon a Change in Control (defined below) of the Company.  For
purposes of this Agreement, a "Change in Control" shall be deemed to have
occurred if (i) there shall be consummated (a) any consolidation or merger of
the Company in which the Company is not the continuing or surviving corporation
or pursuant to which shares of the Company's capital stock are converted into
cash, securities or other property (other than a merger in which the holders of
the Company's capital stock immediately prior to the merger beneficially own at
least a majority of the voting capital stock of the surviving corporation
immediately after the merger), or (b) any sale, lease, exchange or other
transfer (whether in one transaction or a series of related transactions) of
all, or substantially all, of the assets of the Company, or (ii) any person (as
such term is defined in Section 13(d) and Section 14(d)(2) of the Securities
Exchange Act of 1934, as amended (the "Exchange Act"), shall become the
beneficial owner (within the meaning of Rule 13d-3 promulgated under the
Exchange Act) of more than 50% of the Company's then outstanding capital stock.

     10.  Rights Upon Termination. Upon termination of this Agreement, all
          -----------------------
rights and obligations of the Company and the Consultant (including any personal
or legal representative or successor in interest to the Consultant if deceased
or under a legal disability) shall terminate and cease, except as provided
below:

          10.1 Vested Compensation and Benefits.  The Consultant shall be
               --------------------------------
entitled to receive any compensation or benefits which the Consultant has earned
and which have vested to the Consultant as of the date of termination.  Payments
of the Base Fee under Section 6.1 shall be prorated to the date of termination.
In addition, if this Agreement is terminated pursuant to Section 9.2 or 9.4, the
Company shall, within ten (10) business days after the Change in Control occurs
or the date of termination by the Consultant, as the case may be, pay to the
Consultant an amount equal to the product of (i) the then current Base Fee times
(ii) the number of months remaining in the original term of this Agreement.  If
this Agreement is terminated pursuant to Section 9.2, the Consultant shall also
be entitled to the bonus opportunity provided in Section 6.2 as if this
Agreement had not been terminated.  If this Agreement is terminated pursuant to
Section 9.1, the Consultant shall not be entitled to receive any bonus under
Section 6.2 with respect to the calendar year in which this Agreement is
terminated or any calendar year thereafter.  If this Agreement is terminated
pursuant to Section 9.3, the Consultant shall be entitled to receive a prorated
portion of

                                      -4-
<PAGE>

any bonus earned with respect to the calendar year in which this Agreement is
terminated; and the Consultant shall not be entitled to receive any bonus under
Section 6.2 with respect to any subsequent calendar year(s).

          10.2 Survival of Remedies.  Each party shall have such additional
               --------------------
rights and remedies as are provided for elsewhere in this Agreement exercisable
either prior to or subsequent to the termination of this Agreement in the event
of any breach by the other party of any material provision of this Agreement.

     11.  Breach and Remedies.
          -------------------

          11.1 General.  In the event either party breaches any material
               -------
provision of this Agreement, then the aggrieved party shall have all rights and
remedies as provided for in this Agreement or as otherwise available at law or
in equity, without limitation.

          11.2 Litigation Expenses.  In any action brought by a party to enforce
               -------------------
this Agreement against the other party, the prevailing party shall be entitled
to collect from the other party the prevailing party's reasonable attorneys'
fees, court costs, and other expenses reasonably incurred in connection with
such action.

     12.  Return of Company Property.  All products, records, plans, manuals,
          --------------------------
memoranda, lists, and other property delivered to or coming into the possession
of the Consultant by or on behalf of or in connection with the Consultant's
consulting relationship with the Company, and all records compiled by the
Consultant which pertain to the business of the Company shall be and remain the
property of the Company and be subject at all times to its discretion and
control.  Likewise, all correspondence with customers or representatives,
reports, records, charts, advertising materials, and any data collected by the
Consultant in connection with his engagement by the Company shall be promptly
delivered to the Company without request by it upon termination of the
Consultant's engagement for any reason.

     13.  Miscellaneous.
          -------------

          13.1 No Guarantee of Engagement.  Except as specifically and expressly
               --------------------------
provided in this Agreement, the Company has no obligation, either express or
implied, to engage or continue the engagement of the Consultant.

          13.2 Notices.  All notices required or permitted under this Agreement
               -------
must be in writing and shall be sufficient if delivered personally or mailed by
certified or registered mail, return receipt requested, to the other party at
the addresses set forth below:

          Company:       CMI Corporation
          -------
                         P. O. Box 1985
                         Oklahoma City, OK 73101
                         ATTN: Chief Executive Officer

                                      -5-
<PAGE>

          Consultant:    Bill Swisher
          ----------
                         1500 Dorchester Drive
                         Oklahoma City, OK 73120


Any notice given by personal delivery shall be deemed made on the first business
day following actual delivery, or if mailed, on the second business day
following date of postmark.  Either party may change the notice person and
address set forth above by giving proper notice thereof to the other party.

          13.3 Independent Contractor; No Authority to Bind the Company.  The
               --------------------------------------------------------
Consultant shall act as an independent contractor, not as an employee, and,
except as otherwise expressly provided herein, shall not be eligible to
participate in any of the Company's employee benefit programs.  The Consultant
shall be responsible for all taxes arising from the consulting fees and other
amounts paid to the Consultant hereunder.  In addition, the parties hereto
understand and agree that the Consultant has no express, implied or apparent
authority to make any commitments on behalf of the Company, and shall not (and
shall not attempt to) bind the Company in any manner whatsoever.

          13.4 Integrated Agreement.  This Agreement, with any schedules and
               --------------------
exhibits attached hereto, contains and constitutes the entire agreement between
the parties hereto and supersedes all prior agreements and understandings
between them relating to the subject matter hereof.

          13.5 Construction.  This Agreement shall be construed, enforced, and
               ------------
governed in accordance with the laws of the State of Oklahoma.  All pronouns and
any variations thereof shall be deemed to refer to the masculine, feminine, or
neuter gender or to the plurals of each, as the identify of the person or
persons or entities or as the context may require.  The descriptive headings
contained in this Agreement are for reference purposes only and are not intended
to describe, interpret, define, or limit the scope, extent or intent of this
Agreement or any provision contained herein.

                                      -6-
<PAGE>

     IN WITNESS WHEREOF, the parties hereto have executed this Agreement, to be
effective as of the Effective Date.


CONSULTANT:                   /s/ George William Swisher, Jr.
----------                    -------------------------------------------------
                              George William Swisher, Jr. a/k/a Bill Swisher



COMPANY:                      CMI CORPORATION
-------


                         By:  /s/ Tom Engelsman
                              -------------------------------------------------
                              Tom Engelsman, Chief Executive
                              Officer

                                      -7-