<PAGE>

                                                                    EXHIBIT 10.4

March 19, 2003

Mr. John M. Mullin
907 Castle Pines Dr.
Ballwin, MO 63021

Dear Mr. Mullin:

           This is to confirm your resignation effective immediately from your
employment with Huttig Building Products, Inc. (the "Company") as its Vice
President - Operations and from all positions as an employee, officer and
director of any direct or indirect subsidiary of the Company. In accordance with
our prior discussions, you will be entitled to the following (subject to all
applicable tax withholdings):

          (a) You have a negative balance in your cash subaccount under the
     Company's EVA Incentive Compensation Plan, so you will be entitled to no
     further cash payments under that Plan. Any shares of restricted stock that
     were not vested immediately prior to your termination of employment,
     whether awarded to you under the EVA Incentive Compensation Plan or
     otherwise, as well as any stock options not so vested, shall expire and be
     forfeited as of the date of this letter.

          (b) You will be entitled to receive payment in full promptly after the
     date of this letter of (i) any accrued but unpaid salary and payment for
     any accrued vacation and (ii) reimbursement for any previously unreimbursed
     Company-related business expenses (subject to presentation of adequate
     supporting documentation therefor and compliance with other Company
     policies regarding expense reimbursement).

          (c) During the period beginning on the date of this letter and ending
     September 30, 2003 (the "Severance Period"), you will be entitled to
     receive severance pay in the form of salary continuation, at a rate equal
     to your current rate of base salary, payable in accordance with the
     Company's regular payroll practices. In addition, you will be entitled to
     continue to participate in the Company's health, life and disability
     insurance plans, and the Company will pay the portion of the plan costs
     that the Company would pay if you continued to be an active employee, until
     the earliest of (i) the expiration of the Severance Period or (ii) the date
     you commence other employment.

          (d) All of your compensation and benefits, to the extent accrued and
     vested through but not after the date of this letter, under the Company's
     benefit plans and programs shall be paid to you in accordance with the
     terms of such plans and programs. Without limiting the generality of the
     foregoing, (i) your vested stock options will expire 90 days after the date
     of this letter, in accordance with the terms of the 1999 Stock Incentive
     Plan and the Amended and Restated 2001 Stock Incentive Plan and (ii) you
     will be entitled to continued use of your company car for ninety (90) days
     after the date of this letter. Such ninety (90) day use of the company car
     shall terminate and the car returned to the company upon relocation from
     the St. Louis Missouri area.

          (e) It is expressly understood, acknowledged and agreed that all
     compensation, including fringe benefits payable by the company shall be
     subject to the duty to use your best efforts to mitigate damages by seeking
     other employment and shall be offset by any compensation which you receive
     from such other employment or which you could have received with reasonable
     efforts. The Company may request that you provide information as to the
     actions and status of efforts to mitigate damages hereunder. Upon
     employment you shall notify the Company of such and of the compensation
     terms related thereto.

<PAGE>

          In consideration for the payments and benefits that will be
provided to you under this Agreement, on behalf of yourself and your dependents,
heirs, administrators, representatives, trustees, beneficiaries, executors,
successors, assigns and any other person or entity, you hereby unconditionally
release and forever discharge the Company and its agents, officers, directors,
employees, parents, attorneys, subsidiaries, divisions, affiliates,
predecessors, successors and assigns, all their respective employee benefit
plans and their administrators, trustees and other fiduciaries (severally and
collectively called the "Released Parties") from any and all manner of claims,
demands, debts, rights, disputes, judgments, agreements, losses, costs, damages
and liabilities of any kind whatsoever, in law or in equity, whether known or
unknown, that you or any person or entity acting for you now has or hereafter
may have against any of the Released Parties for any acts, circumstances,
omissions, or events up to and including the date hereof, it being your
intention to effect a general release of all claims. This general release
includes, without in any way limiting the generality of the foregoing, all
claims or causes of action arising out of or relating to your employment or
termination of employment with the Company; and any claims arising from any
alleged violation by any of the Released Parties of any federal, state or local
statutes, ordinances, rules, Executive Orders or regulations, including, but not
limited to, any of the following, as amended: Title VII of the Civil Rights Act
of 1964, the Rehabilitation Act of 1973, the Americans with Disabilities Act,
the Employee Retirement Income Security Act of 1974, the Civil Rights Act of
1991, and the Age Discrimination in Employment Act, and every other source of
legal rights and obligations which may be waived and/or released; provided,
however, that the foregoing release shall not adversely affect your rights under
this letter.

          You acknowledge that you have been given no less than twenty-one (21)
days to consider this letter agreement before executing it. You further
acknowledge that you may revoke this Agreement for a period of seven (7) days
from the date you execute it (the "Revocation Period"), by notifying in writing,
Nick H. Varsam, Vice President - General Counsel, Huttig Building Products,
Inc., 555 Maryville University Drive, St. Louis, Missouri 63141. In the event
you revoke this agreement during the Revocation Period, this agreement shall be
null and void in its entirety and all of the Company's obligations hereunder
shall cease immediately.

          Except as directed by the Board of Directors of the Company or as may
be required by law, you shall keep confidential and shall not divulge to any
other person or entity at any time any of the business secrets or other
confidential information regarding the Company and its affiliates, except to the
extent that such information has become public knowledge through no fault of
yours. Consistent with the Company's policy, you acknowledge that all papers,
books and records of every kind and description relating to the business and
affairs of the Company and its affiliates, whether or not prepared by you, other
than your personal notes, shall remain the sole and exclusive property of the
Company, and you shall return them to the Company promptly after the date of
this letter.

          You agree that you will not make, publish or disseminate any
derogatory statements or comments, whether orally or in writing, about the
Company or its officers or directors, or take any action that a reasonable
person would expect, directly or indirectly, to impair the goodwill, business
reputation or good name of any of them. In addition, during the Severance
Period, you will, if requested by the Company from time to time, consult and
cooperate with the Company and its representatives and answer questions with
regard to the business, management and finances of the Company; provided, that
the Company shall take reasonable measures to ensure that such obligations do
not materially interfere with any employment opportunities or responsibilities
you may have during such Severance Period. Finally, you agree to cooperate with
the Company in any actual or threatened litigation or other proceedings
involving the Company or any of its affiliates. Such cooperation shall include,
but not be limited to, testifying on the Company's behalf at depositions, before
administrative or regulatory bodies or in court or arbitration or similar
proceedings.

<PAGE>

         This agreement shall be governed by the laws of the State of Missouri,
other than the conflict of laws provisions thereof.

               This letter agreement constitutes the entire understanding of
the parties with respect to its subject matter, supersedes all prior agreements
and understandings with respect to such subject matter, and may be terminated or
amended only by a writing signed by all of the parties to this Agreement. The
Change of Control Agreement between you and the Company, dated as of November
19, 2002, and the noncompetition agreement between you and the Company are
hereby terminated in its entirety as of the date of this letter.

                                HUTTIG BUILDING PRODUCTS, INC.,

                                By:      /s/ Michael A. Lupo
                                     --------------------------------------
                                      Michael A. Lupo
                                      President and Chief Executive Officer

Agreed to and accepted by:

  /s/ John M. Mullin
-----------------------
John M. Mullin