<PAGE>
                                                                   EXHIBIT 10(m)

                                  WELLMAN, INC.
                      2004 RESTRICTED STOCK GRANT AGREEMENT

SECTION I. GRANT

1.1 This Grant of Restricted Stock is made pursuant to and is subject to the
terms of the Wellman, Inc. Restricted Stock Plan which is effective June 1, 2004
(the "Plan"). All capitalized terms used herein and not defined in this document
have the meaning given such terms in the Plan. Each Grant made pursuant to the
Plan is an Award and shall be subject to all the terms and conditions of the
Plan, which are incorporated herein by reference.

1.2 EFFECTIVE DATE.  This Grant is effective on July 1, 2004 (the "Effective
Date").

SECTION II. CERTAIN DEFINITIONS

2.1 CAUSE in the context of a termination of employment means only one or more
of the following: (i) the commission in the course of employment of any
dishonest or fraudulent act; (ii) a conviction of a felony (from which, through
lapse of time or otherwise, no successful appeal shall have been made) whether
or not committed in the course of employment; (iii) the willful refusal to carry
out reasonable instructions of the Board which could have a material adverse
effect upon the Company or any of its Subsidiaries; and (iv) the willful
disclosure of any trade secrets or material confidential corporate information
to any Person not authorized to know same.

2.2 DISABILITY shall mean a determination by the Committee that the Employee is
unable to perform the duties required of him or her by the Company as a result
of any physical or mental condition.

2.3 INITIAL RESTRICTED PERIOD means the period commencing on June 1, 2004 and
ending on May 31, 2006; provided, however, that upon the written request of the
Participant given to the Company at least six months before the expiration of
the Initial Restricted Period or any Subsequent Restricted Period and prior to
termination of employment (regardless of the reason for termination), the
Restricted Period may be extended for a period of not less than two years from
the date such Restricted Period expires (the "Subsequent Restricted Period").

2.4 RESTRICTED PERIOD means either the Initial Restricted Period or a Subsequent
Restricted Period, as applicable.
<PAGE>
2.5 RESTRICTED STOCK means shares of Common Shares which are issued by the
Company pursuant to this Grant Agreement and which are subject to forfeiture,
restrictions on transfer and other restrictions as are set forth in Section IV
hereof.

2.6 SUBSIDIARY means a corporation of which more than 50% of the total combined
voting power of all classes of stock entitled to vote is owned, directly or
indirectly, by Wellman, Inc.

Unless the context clearly requires otherwise, the masculine pronoun whenever
used shall include the feminine and neuter pronouns, the singular shall include
the plural and the plural shall include the singular.

SECTION III. AWARD

3.1 AWARD.  The Committee has designated the person executing this Grant
Agreement as a Participant eligible to receive 20,000 shares of Restricted Stock
and this Grant is made subject to the terms and conditions in this Grant
Agreement and the Plan. No payment for such shares is required except pursuant
to Section 5.2.

3.2 ACCEPTANCE.  By accepting this Grant of Restricted Stock the Participant
acknowledges and agrees that the shares of Restricted Stock are subject to the
terms and conditions of this Grant Agreement and the Plan.

3.3 ADJUSTMENTS.  In the event of any stock dividend, stock split, combination
or exchange of shares, merger, consolidation, spin-off or other distribution
(other than normal cash dividend) of the Company assets to stockholders, or any
other change affecting shares or the Company's capitalization, such adjustments
as the Committee in its discretion may deem appropriate to reflect such change
or to fairly preserve the intended benefits of the Plan shall be made. In
addition, any shares issued by the Company through the assumption or
substitution of outstanding stock awards or award commitments from an acquired
company or other entity shall not reduce the shares available for issuance under
the Plan.

SECTION IV. TERMS OF RESTRICTED STOCK

4.1 TERMS OF RESTRICTED STOCK

(a) Prior to the expiration of the Restricted Period the Participant shall not
sell, transfer, pledge or otherwise encumber ("Transfer") the Restricted Stock
and any such Transfer shall be void. After the Restricted Period the Participant
shall not Transfer the Restricted Stock in any manner that could violate any
securities laws or result in short swing profits.

(b) The Participant shall not be entitled to delivery of a certificate
evidencing the shares of Restricted Stock until the expiration or termination of
the Restricted Period and the satisfaction of any and all other conditions
specified in this Grant Agreement.
<PAGE>
(c) Any shares of Restricted Stock which are forfeited pursuant to this Grant
Agreement shall be cancelled and all rights of the Participant with respect to
such forfeited shares of Restricted Stock shall terminate without further
obligation on the part of the Company upon the occurrence of any of the events
set forth below in subsection 4.4.

4.2 CUSTODY OF SHARES OF RESTRICTED STOCK; RIGHTS WITH RESPECT TO STOCK.

(a) The Restricted Stock granted hereunder shall be issued and registered in the
Participant's name and shall be held by the Company during the Restricted
Period. The Company shall serve as attorney-in-fact for the Participant during
the Restricted Period with full power and authority in the Participant's name to
assign and convey to the Company any shares of Restricted Stock held by the
Company for such Participant if the Participant forfeits the shares under the
terms of this Grant Agreement. Each certificate representing shares of
Restricted Stock may bear a legend referring to this Grant Agreement and the
Plan and the risk of forfeiture of the shares of Restricted Stock and stating
that such shares of Restricted Stock are nontransferable until all restrictions
have been satisfied and the legend has been removed.

(b) During the Restricted Period, the Participant shall be a stockholder and
have all the rights of a stockholder with respect to such shares, including the
right to vote and receive all dividends or other distributions made or paid with
respect to such shares; provided, however, that such Restricted Stock and any
new, additional or different securities the Participant may become entitled to
receive with respect to such Restricted Stock by virtue of a stock split,
dividend or other change in the corporate or capital structure of the Company,
shall be subject to the restrictions described in this Grant Agreement.

4.3 DISTRIBUTION OF RESTRICTED STOCK.

If the Participant remains in the continuous employment or service of the
Company or any Subsidiary during the entire Restricted Period and otherwise does
not forfeit such shares of Restricted Stock pursuant to subsection 4.4 or any
other terms of the Grant Agreement, all restrictions applicable to the shares of
Restricted Stock shall lapse upon expiration or termination of the Restricted
Period, and a certificate or certificates of unrestricted Common Shares
representing the shares of Restricted Stock shall be delivered to the
Participant.

4.4 FORFEITURE.

(a) If a Participant's service or employment is terminated before the expiration
of the Restricted Period due to Disability or death of the Participant the
restrictions in subsection 4.1 (a) and (b) shall lapse. The certificate or
certificates representing the shares of Restricted Stock upon which the
restrictions have lapsed pursuant to this subsection 4.4(a) shall be delivered
to the Participant (or in the event of the Participant's death, to his
beneficiary or estate) after the Participant or his legal representative
complies with the terms of Section 5.2.

(b) If a Participant's service or employment is terminated by the Company or any
Subsidiary without Cause (other than as a result of death or Disability) before
the expiration of the Initial
<PAGE>
Restricted Period then the restrictions set forth in subsection 4.1 (a) and (b)
shall lapse with respect to a portion of the Restricted Stock determined as
follows: the number of shares of Restricted Stock as to which the restrictions
will lapse (rounded up to the nearest whole share) will be the total number of
shares of Restricted Stock granted to the Participant in this Grant Agreement
multiplied by a fraction, the numerator of which shall be the number of months
the Participant has remained an employee (or in the service of) of the Company
or a Subsidiary since June 1, 2004 and the denominator of which shall be 24. The
remainder of the shares of Restricted Stock will be forfeited. If a
Participant's service or employment is terminated by the Company or any
Subsidiary without Cause after the expiration of the Initial Restricted Period
then all the restrictions shall lapse.

(c) If a Participant's service or employment is terminated before the expiration
of the Restricted Period by the Company or any Subsidiary for Cause or by the
Participant (other than as a result of death or Disability), the Participant
shall forfeit all Restricted Stock.

4.5 CHANGE OF CONTROL.  Upon any Change of Control all restrictions and
forfeiture provisions applicable to shares of Restricted Stock granted to the
Participant hereunder and not previously forfeited shall immediately lapse and
the certificate or certificates representing the Restricted Stock that were
granted to the Participants shall be delivered to the Participants as
unrestricted Common Shares, subject to the provisions of Section 5.2.

4.6 WAIVER OF RESTRICTIONS.  The Committee, in its sole discretion, may at any
time waive any or all restrictions with respect to any shares of Restricted
Stock.

SECTION V. MISCELLANEOUS

5.1 TERMINATION.  The provisions relating to the Restricted Stock granted
hereunder shall continue to apply with respect to all shares of Restricted Stock
until the restrictions lapse or the shares are forfeited.

5.2 WITHHOLDING.  The Participant shall promptly pay to the Company any amount
necessary to satisfy applicable federal, state or local tax withholding
requirements attributable to the grant of Restricted Stock, the issuance of
Restricted Stock hereunder, or upon the vesting of or lapse of restrictions on
such Restricted Stock. If these amounts are not paid when requested, then at the
election of the Committee, shares of Restricted Stock with a value equal to the
amount of tax to be withheld shall be forfeited.

5.3 LEGAL AND OTHER REQUIREMENTS.  The grant of Restricted Stock and the
distribution of shares of Restricted Stock provided by this Grant Agreement
shall be subject to the condition that if at any time the Company determines in
its discretion that the satisfaction of withholding tax or other tax
liabilities, or the listing, registration or qualification of any shares of
Restricted Stock upon any securities exchange or under any federal or state law,
or the consent or approval of any regulatory body, is necessary or desirable as
a condition of, or in connection with such grant or distribution, then in any
such event, such grant or distribution shall not be effective unless such
<PAGE>
liabilities have been satisfied or such listing, registration, qualification,
consent or approval shall have been effected or obtained free of any conditions
not acceptable to the Company.

5.4 CHOICE OF LAW.  This Grant Agreement and the Plan, their validity,
interpretation and administration and the rights and obligations of all Persons
having an interest therein shall be governed by and construed in accordance with
the laws of the State of Delaware, except to the extent that such laws may be
preempted by federal law.

5.5 FRACTIONAL SHARES.  The Company shall not be required to issue or deliver
any fractional share of Restricted Stock issuable under this Grant Agreement.
Fractional shares will be paid in cash.

5.6 NO EMPLOYMENT CONTRACT.  Neither this Grant Agreement nor the Plan shall
confer upon the Participant any right to continued employment by the Company or
in any way interfere with the right of the Company to terminate the employment
of any Participant at any time.

5.7 SECTION 83(b) ELECTIONS.  If a Participant files an election with the
Internal Revenue Service to include the fair market value of any shares of
Restricted Stock in gross income then the Participant shall promptly furnish the
Company with a copy of such election together with the amount of any federal,
state, local or other taxes required to be withheld (if any) to enable the
Company to claim an income tax deduction with respect to such election.

                                        Wellman, Inc.


                                        By: /s/ Keith R. Phillips
                                            ------------------------------------
                                        Name: Keith R Phillips
                                        Title: VP & Chief Financial Officer


Agreed to by:


Participant's Signature:                /s/ Thomas M. Duff
                                        ----------------------------------------
Name (Please Print):                    Thomas M. Duff
Social Security Number:                 XXX-XX-XXXX