<PAGE>

                                                                    Exhibit 10.4

                               September 17, 2003

Ms. Sylvia Metayer
106 Appleton Street
Cambridge, MA 02138-3340

Dear Sylvia:

      We have discussed that Houghton Mifflin Company (the "Company") and
Houghton Mifflin Holdings, Inc. ("Holdings" and, with you and the Company, the
"Parties") desire to obtain your commitment to additional protection of
Confidential Information (defined in Section 4(b) hereof) and goodwill through
the amendment, restatement and extension of certain restrictive covenants
contained in existing agreements between you and the Company and/or Holdings,
effective as of July 1, 2004. We have also discussed the Company's willingness,
in exchange for that commitment from you, to provide you increased severance
benefits in the event that it initiates termination of your employment other
than for Cause (defined in Section 2(a) hereof) on or after that date. To that
end, effective as of July 1, 2004, this Agreement amends any and all prior
agreements between you and the Company and/or Holdings with respect to the
respective rights and obligations of the Parties arising on or after that date
which are the subject of this Agreement, as follows:

      1. CONFIDENTIAL INFORMATION AND RESTRICTED ACTIVITIES. Provided that you
are employed by the Company on July 1, 2004, the following obligations shall
apply, amending and restating those set forth in Sections 6 and 7 of the
Original Retention Agreement (as defined in Section 4 (c) below) as of that
date:

            (a) CONFIDENTIAL INFORMATION. During the course of your employment
and other associations with the Company and its Affiliates, you have learned,
and in the future will learn, of Confidential Information (defined in Section
4(b) below) and you may have developed, and hereafter may develop, Confidential
Information on behalf of the Company and its Affiliates. You agree that you will
not use or disclose to any Person (except as required by applicable law or for
the proper performance of your regular duties and responsibilities for the
Company and its Affiliates) any Confidential Information obtained by you
incident to your employment or any other association with the Company or any of
its Affiliates. You understand and agree that this restriction shall continue to
apply after your employment terminates, regardless of the reason for such
termination. Further, you agree to provide prompt notice to the Company of any
required disclosure of Confidential Information sought pursuant to subpoena,
court order or any other legal requirement and to provide the Company a
reasonable opportunity to seek protection of the Confidential Information prior
to any such disclosure.
<PAGE>
            (b) PROTECTION AND RETURN OF DOCUMENTS AND OTHER PROPERTY. You agree
that all documents, records and files, in any media of whatever kind and
description, relating to the business, present or otherwise, of the Company and
its Affiliates and any copies (including without limitation electronic copies),
in whole or in part, thereof (together, the "Documents" and each individually, a
"Document"), whether or not prepared by you, shall be the sole and exclusive
property of the Company and its Affiliates. You agree to safeguard, and return
to the Company immediately upon termination of your employment and at such other
times as may be specified by the Company, all Documents and all other property
of the Company and its Affiliates, and all documents, records and files of
Persons with whom they do business ("Third-Party Documents" and each
individually a "Third-Party Document") then in your possession or control;
provided, however, that, if a Document or Third-Party Document is on electronic
media, you may, in lieu of surrender of the Document or Third-Party Document,
provide a copy on electronic media (e.g., a properly formatted diskette) to the
Company and delete and overwrite all other electronic media copies thereof. You
also agree that, upon request of any duly authorized officer of the Company, you
will disclose all passwords necessary or desirable to enable the Company to
obtain access to the Documents and Third-Party Documents.

            (c) MAINTENANCE OF RECORDS AND ASSIGNMENT OF RIGHTS TO INTELLECTUAL
PROPERTY. You agree to maintain accurate and complete contemporaneous records
of, and shall immediately and fully disclose and deliver to the Company, all
Intellectual Property (defined in Section 4(e) below). You hereby assign and
agree to assign to the Company (or as otherwise directed by the Company) your
full right, title and interest in and to all Intellectual Property. You agree to
execute any and all applications for domestic and foreign patents, copyrights or
other proprietary rights and to do such other acts (including without limitation
the execution and delivery of instruments of further assurance or confirmation)
requested by the Company to assign the Intellectual Property to the Company and
to permit the Company to enforce any patents, copyrights or other proprietary
rights to the Intellectual Property. You agree not to charge the Company for
time spent in complying with these obligations. All copyrightable works that you
create shall be considered "work made for hire" and shall, upon creation, be
owned exclusively by the Company.

            (d) NON-COMPETITION. You acknowledge that in your employment with
the Company you have had, and will hereafter have, access to Confidential
Information which, if disclosed, would assist in competition against the Company
and its Affiliates and that you also have, and will hereafter, generate goodwill
for the Company and its Affiliates in the course of your employment. You also
acknowledge that the amount of severance pay to be provided you hereunder in the
event of termination of your employment by the Company other than for Cause on
or after July 1, 2004 is twice that to which you were previously eligible,
expressly in exchange for your commitments under this Section 1(d). Further, you
agree that the following restrictions on your activities during and after your


                                      -2-
<PAGE>
employment are necessary to protect the goodwill, Confidential Information and
other legitimate interests of the Company and its Affiliates:

                  (i) While you are employed by the Company and during the
      twenty-four (24) months immediately following termination of your
      employment, regardless of the reason for such termination, (in the
      aggregate, the "Non-Competition Period"), you shall not, directly or
      indirectly, whether as owner, partner, investor, consultant, agent,
      employee, co-venturer or otherwise, compete with the Company or any of its
      Affiliates anywhere in the world or undertake any planning for any
      business competitive with the business of the Company or any of its
      Affiliates. Specifically, but without limiting the foregoing, you agree
      not to work for, or provide services to, in any capacity, whether as an
      employee, independent contractor or otherwise, whether with or without
      compensation, any Person engaged in any business that is competitive with
      the business of the Company or any of its Affiliates, as conducted or in
      planning during your employment with the Company.

                  (ii) You agree that, during the Non-Competition Period, other
      than as required by your duties for the Company and its Affiliates during
      your employment, you will not, and will not assist anyone else to, (A)
      solicit for hiring any employee of the Company or any of its Affiliates or
      seek to persuade any employee of the Company or any of its Affiliates to
      discontinue employment or (B) solicit or encourage any independent
      contractor or other Person doing business with the Company or any of its
      Affiliates at any time during your employment to terminate or diminish its
      relationship with the Company or any of the Company's Affiliates or to
      violate any agreement of such independent contractor or other Person with
      the Company or any of its Affiliates. For the purposes of this Agreement,
      an "employee" of the Company or any of its Affiliates is any Person who is
      then in their employ or who was so employed at any time within the
      preceding six months.

            (e) In signing this Agreement, you give the Company assurance that
you have carefully read and considered all the terms and conditions of this
Agreement, including the restraints imposed on you under this Section 1. You
agree without reservation that these restraints are necessary for the reasonable
and proper protection of the Company and its Affiliates and that each and every
one of the restraints is reasonable in respect to subject matter, length of time
and geographic area. You further agree that, were you to breach any of the
covenants contained in this Section 1, the damage to the Company and its
Affiliates would be irreparable. You therefore agree that the Company, in
addition to any other remedies available to it, shall be entitled to preliminary
and permanent injunctive relief against any breach or threatened breach by you
of any of those covenants, without having to post bond. You and the Company
further agree that, in the event that any provision of this Section 1 is
determined by any court of competent jurisdiction to be unenforceable by reason
of its being extended over too great a time, too large a geographic


                                      -3-
<PAGE>
area or too great a range of activities, that provision shall be deemed to be
modified to permit its enforcement to the maximum extent permitted by law. It is
also agreed that each of the Company's Affiliates shall have the right to
enforce all of your obligations to that Affiliate under this Agreement,
including without limitation pursuant to this Section 1.

      2 TERMINATION OF EMPLOYMENT. Your employment may be terminated by you or
the Company at any time, with or without cause, and nothing contained in this
Agreement obligates the Company or any of its Affiliates to retain your services
for a fixed period. If you are employed by the Company on July 1, 2004, however,
your employment with the Company shall continue thereafter until terminated
pursuant to this Section 2, as follows:

            (a) The Company may terminate your employment for Cause upon notice
to you setting forth in reasonable detail the nature of the Cause. The
following, as determined by the Board of Directors of the Company (the "Board")
in its reasonable judgment, shall constitute Cause for termination: (i) your
continued material failure substantially to perform your duties (other than as a
result of incapacity due to physical or mental illness), (ii) your gross
negligence or willful misconduct in the course of your employment with the
Company, (iii) your conviction of, or plea of nolo contendere to, a felony (or
the equivalent thereof in a jurisdiction outside of the United States), (iv)
your material breach of any material provision of this Agreement, (v) your
material breach of a material employment policy of the Company, (vi) your
misappropriation, embezzlement or material misuse of funds or property belonging
to the Company or any of its Affiliates or (vii) your use of alcohol or illegal
drugs that either interferes with your performance of your duties hereunder or
compromises the integrity and reputation of the Company or its Affiliates, their
employees or their products; provided that, following written notice from the
Company under clause (i), (ii) (iv) or (v) above, you will have ten (10)
business days during which to cure such failure, negligence, misconduct or
breach; but provided further that you will be permitted only one such cure
period.

            (b) The Company also may terminate your employment at any time other
than for Cause upon notice to you; provided, however, that, in the event of such
termination prior to your obtaining a United States permanent resident visa (a
"green card"), the Company shall give you sixty (60) days' notice of any
termination under this Section 2(b).

            (c) You may terminate your employment at any time on thirty (30)
days' notice to the Company. The Company may elect to waive the notice period,
or any portion thereof. Notwithstanding the foregoing, however, in the event of
termination under this Section 2(c) prior to your obtaining a green card, you
may elect instead to give up to a maximum of sixty (60) days' notice of such
termination and the Company agrees to continue your employment throughout that
notice period.


                                      -4-
<PAGE>
            (d) Your employment shall automatically terminate in the event of
your death during employment. In the event you become disabled during employment
and, as a result, are unable to continue to perform substantially all of your
duties and responsibilities for the Company and its Affiliates for more than
ninety (90) days during any period of three hundred and sixty-five (365)
consecutive calendar days, the Company may terminate your employment upon notice
to you; provided, however, that in the event such termination occurs prior to
your obtaining a green card, the Company shall give you at least sixty (60)
days' notice of such termination and agrees to continue your employment until
the end of the notice period. Any leave of absence provided you during any
period of employment with the Company in which you are disabled, and any
continuation of pay or benefits during such leave, will be governed by the
policies of the Company generally applicable to its executives, as in effect
from time to time. If any question shall arise as to whether you are disabled to
the extent that you are unable to perform substantially all of your duties and
responsibilities for the Company and its Affiliates, you shall, at the Company's
request, submit to a medical examination by a physician selected by the Company
to whom you or your guardian, if any, has no reasonable objection to determine
whether you are so disabled and such determination shall for the purposes of
this Section 2(d) be conclusive of the issue. If such a question arises and you
fail to submit to the requested medical examination, the Company's determination
of the issue shall be binding on you.

      3. SEVERANCE PAYMENTS AND OTHER MATTERS RELATED TO TERMINATION. In the
event of termination of your employment on or after July 1, 2004, the Company
shall have no obligation to you, other than as set forth expressly in this
Section 3.

            (a) In the event of any termination of your employment with the
Company, regardless of the reason for such termination, the Company will provide
the following Final Payment to you or, in the event of your death, to your named
beneficiary or, if none, to your estate: (i) base salary earned but not yet
paid, through the date of termination; (ii) pay, at your final base rate of pay,
for any vacation time accrued but not used as of the date of termination; and
(iii) reimbursement, in accordance with its policies, of any business expenses
incurred but not yet reimbursed on the date of termination, provided that such
expenses and any required documentation are submitted within sixty (60) days
following termination of your employment.

            (b) In the event of termination of your employment (i) by the
Company for Cause or (ii) as a result of your death or disability or (iii) by
you prior to obtaining a green card, the Company will have no further obligation
to you, other than for the Final Payment.

            (c) In the event of your termination of your employment pursuant to
Section 2(c) hereof after you have obtained a green card, if the Company elects
to waive the notice period, or any portion thereof, the Company will pay you, in
addition to the Final


                                      -5-
<PAGE>
Payment, your base salary for any portion of the notice period waived, to a
maximum of thirty (30) days' pay.

            (d) In the event of termination of your employment by the Company
other than for Cause in accordance with Section 2(b) hereof, then, in addition
to the Final Payment, the Company will provide you severance pay equal to
twenty-four (24) months of base salary, at the rate in effect on the date of
termination, provided that you sign an effective and timely release of claims in
the form provided by the Company (the "Employee Release"). All severance
payments will be in the form of salary continuation, payable in accordance with
the normal payroll practices of the Company for its executives, and will begin
at the Company's next regular payroll period following the later of the
effective date of the Employee Release or the date the Employee Release, signed
by you, is received by the Company. The first severance payment will be
retroactive to the day immediately following the date of termination of your
employment. Also, subject to your signing an effective and timely Employee
Release and subject to any employee contribution applicable to you on the date
of termination, the Company shall continue to contribute to the premium cost of
your participation in the Company's group health, dental and vision plans for
the period of twenty-four (24) months from the date of termination or, if less,
until you become eligible for coverage under the health plan of another
employer, provided that you are eligible for such continued participation under
applicable law and plan terms. You may be required to exercise your rights under
the federal law known as "COBRA," or other applicable law, in order to be
eligible for the Company's premium contributions hereunder and coverage under
the Company's health, dental and vision plans may be subject to geographically
limited service areas and other limitations, which shall be applicable to you.

            (e) Except for any right you may have to continue participation in
the Company's group health, dental and vision plans in accordance with Section
3(d) above or otherwise at your cost under COBRA, benefits shall terminate in
accordance with the terms of the applicable benefit plans based on the date of
termination of your employment, without regard to any continuation of your base
salary or other payment to you following termination. It is expressly understood
and agreed that you will not have any entitlement to receive benefits under any
other agreement with the Company or any of its Affiliates as a result of
termination of your employment; nor will you have any entitlement to receive
benefits under the Houghton Mifflin Severance Plan or any other plan or
arrangement of the Company or any of its Affiliates providing benefits as a
result of termination of employment, regardless of the reason for such
termination.

            (f) Provisions of this Agreement shall survive any termination if so
provided in this Agreement or if necessary or desirable to accomplish the
purposes of other surviving provisions, including without limitation your
obligations under Section 1 of this Agreement. The obligation of the Company to
make payments to you or on your behalf


                                      -6-
<PAGE>
under Section 3(d) hereof is expressly conditioned upon your continued full
performance of obligations under Section 1 hereof.

      4. DEFINITIONS. For purposes of this Agreement, the following definitions
apply:

            (a) "Affiliates" means all persons and entities directly or
indirectly controlling, controlled by or under common control with the Company,
where control may be by management authority or equity interest.

            (b) "Confidential Information" means any and all information of the
Company and its Affiliates that is not generally available to the public.
Confidential Information also includes any information received by the Company
or any of its Affiliates from any Person with any understanding, express or
implied, that it would not be disclosed.

            (c) "Consolidated Retention Agreement" means your agreement with the
Company captioned "Senior Executive 3X Retention Agreement" effective September
9, 2002 (the "Original Retention Agreement"), as amended by your agreement of
January 29, 2003 with Holdings captioned "Senior Executive 3X Retention
Agreement, Waiver and Election" (the "Rollover Agreement") and as further
amended by your letter agreement with the Company and Holdings dated on or about
September 10, 2003 (the "2003 Compensation Agreement").

            (d) "Employment Agreement" means your letter of employment with the
Company dated September 19, 2002, as amended by the 2003 Compensation Agreement.
(e) "Intellectual Property" means inventions, discoveries, developments,
methods, processes, compositions, works, concepts and ideas (whether or not
patentable or copyrightable or constituting trade secrets) conceived, made,
created, developed or reduced to practice by you (whether alone or with others,
whether or not during normal business hours or on or off Company premises)
during your employment and other association with the Company that relate to
either the products or services of the Company or any of its Affiliates or any
prospective activity of the Company or any of its Affiliates or that make use of
Confidential Information or any of the equipment or facilities of the Company or
any of its Affiliates.

            (f) "Person" means an individual, a corporation, a limited liability
company, an association, a partnership, an estate, a trust or any other entity
or organization, other than the Company or any of its Affiliates.

      5. WITHHOLDING. All payments made by the Company or Holdings under this
Agreement shall be reduced by any tax or other amounts required to be withheld
under applicable law.


                                      -7-
<PAGE>
      6. ASSIGNMENT. Neither you nor the Company nor Holdings may make any
assignment of this Agreement or any interest in it, by operation of law or
otherwise, without the prior written consent of the other; provided, however,
that the Company and Holdings each may assign its rights and obligations under
this Agreement without your consent to one of the Affiliates or to any Person
with whom the Company or Holdings shall hereafter affect a reorganization,
consolidate with, or merge into or to whom either transfers all or substantially
all of its properties or assets. This Agreement shall inure to the benefit of
and be binding upon you, the Company and Holdings, and each of our respective
successors, executors, administrators, heirs and permitted assigns. You
expressly consent to be bound by the provisions of this Agreement for the
benefit of the Company and of any Affiliate, successor or permitted assign to
whose employ you may be transferred, without the necessity that this Agreement
be re-signed at the time of such transfer.

      7. SEVERABILITY. If any portion or provision of this Agreement shall to
any extent be declared illegal or unenforceable by a court of competent
jurisdiction, then the remainder of this Agreement, or the application of such
portion or provision in circumstances other than those as to which it is so
declared illegal or unenforceable, shall not be affected thereby, and each
portion and provision of this Agreement shall be valid and enforceable to the
fullest extent permitted by law. Notwithstanding the foregoing, however, the
Company shall be entitled to enforce its rights under the second sentence of
Section 3(f) hereof.

      8. MISCELLANEOUS. This Agreement sets forth your entire agreement with the
Company and Holdings, and supersedes all prior and contemporaneous
communications, agreements and understandings, written or oral, with respect to
the subject matter hereof, effective as of July 1, 2004. Except as modified
hereby on and after July 1, 2004, however, the Employment Agreement (defined in
Section 4(d) above) and the Consolidated Retention Agreement (also defined in
Section 4(c) above) shall remain in full force and effect in accordance with
their terms. This Agreement may not be modified or amended, and no breach shall
be deemed to be waived, unless agreed to in writing by you and an expressly
authorized representative of the Board and of the Board of Directors of
Holdings. The headings and captions in this Agreement are for convenience only
and in no way define or describe the scope or content of any provision of this
Agreement. This Agreement may be executed in two or more counterparts, each of
which shall be an original and all of which together shall constitute one and
the same instrument. This is a Massachusetts contract and shall be governed and
construed in accordance with the laws of The Commonwealth of Massachusetts,
without regard to the conflict of laws principles thereof.

      9. NOTICES. Any notices provided for in this Agreement shall be in writing
and shall be effective when delivered in person, consigned to an overnight
courier service or deposited in the United States mail, postage prepaid, and
addressed to you at your last


                                      -8-
<PAGE>
known address on the books of the Company or, in the case of the Company and
Holdings, to them at the Company's principal place of business, attention of the
Chair of the Board, or to such other address as either party may specify by
notice to the other actually received.

      If the foregoing is acceptable to you, please sign this letter in the
space provided and return it to me no later than September 17, 2003. At the
time you sign and return it, this letter will take effect as a binding agreement
among you, the Company and Holdings on the basis set forth above. The enclosed
counterpart, which you should also sign and date, is for your records.

Sincerely yours,
HOUGHTON MIFFLIN COMPANY

/s/ Gerald Hughes
-------------------------------
Gerald Hughes, Senior Vice President, Human Resources

HOUGHTON MIFFLIN HOLDINGS, INC.



/s/ Michael Ward
-------------------------------
Michael Ward, Director


Accepted and Agreed:


Signature:  /s/ Marie-Sylvia Metayer
           ----------------------------------
                  Marie-Sylvia Metayer

Date:  September 17, 2003
      ---------------------------------------

                                      -9-