<PAGE>

EXHIBIT 10.4                                                  October 6, 2003

Frederick R. Sullivan
One FirstFed Park
Swansea, Massachusetts  02777

Dear Mr. Sullivan:

                  In connection with the anticipated merger (the "Merger") of
FirstFed America Bancorp, Inc. (the "Company") with and into Webster Financial
Corporation (the "Parent") as contemplated by the Agreement and Plan of Merger,
dated as of October 6, 2003, by and between the Parent and the Company (the
"Merger Agreement"), which is entered into as of today, the Company, Company
Bank, the Parent and you hereby enter into this agreement (this "Agreement").
Capitalized terms used but not otherwise defined in this Agreement shall have
the meaning set forth in the Merger Agreement.

                  The parties hereto agree and acknowledge that paragraphs 1(a),
2 and 3 of this Agreement shall become immediately effective upon the execution
of this Agreement; all other provisions of this Agreement shall become effective
only as of the Effective Time (as defined in the Merger Agreement). In the event
that the Effective Time does not occur for any reason, this Agreement shall be
deemed null and void ab initio and of no force and effect, and the Employment
Agreement by and among you, the Company Bank and the Company dated as of January
31, 1997 (the "Company Bank Agreement") and the Employment Agreement between you
and the Company dated as of January 31, 1997 (the "Company Agreement," and
together with the "Company Bank Agreement," the "Prior Agreements") shall be
reinstated effective immediately.

         1.       Compensation and Benefits.

         (a)      Options. Prior to December 1, 2003, you hereby agree to
exercise any and all vested options for the purchase of Company stock
("Options") that you hold as of the date hereof such that all income from such
exercise shall be included in your gross income for 2003. In the event that you
sell the shares subject to the Options that you exercise pursuant to the
preceding sentence prior to the Effective Time on the open market and the amount
per share that you realize upon such sale is less than the Cash Consideration,
on the Effective Date, the Company shall pay you an additional amount per share
equal to the excess, if any, of the Cash Consideration over the amount realized
per share for such shares on the date of disposition of such shares.

         (b)      Effective Date Payment/Representation. On the Effective Date,
the Parent shall pay you a lump sum payment in cash in the amount set forth on
Exhibit A (the "Effective Date Payment"). For the avoidance of doubt, and
notwithstanding anything

<PAGE>

herein to the contrary, the Effective Date Payment shall not be taken into
account in computing any benefits under any plan, program or arrangement of the
Parent, the Company Bank, the Company or their affiliates. You and the Company
represent and warrant that all amounts set forth on Exhibit A and the back up
data provided to the Parent prior to the date hereof for such amounts are true
and correct and that all amounts noted as being estimates are good faith
estimates. Notwithstanding anything in this Agreement or otherwise (including
under the First Federal Savings Bank of America Employee Severance Compensation
Plan or any other severance plan, program, practice or arrangement of the
Parent, the Company Bank, the Company or any of their respective affiliates (the
"Severance Plans")), in no event shall you be eligible to participate in or
receive severance benefits under the Severance Plans.

         2.       Termination of the Prior Agreements; Agreement to Remain
Employed Through Effective Time. Except as provided in paragraph 3 of this
Agreement, you hereby agree that, in consideration for entering into this
Agreement, effective as of the date hereof, the Prior Agreements shall be null
and void and no person or entity shall be obligated to pay to you or any person
any amounts in respect of the Prior Agreements. Further, in consideration of the
benefits conferred upon you and the Company pursuant to this Agreement, you
hereby agree not to terminate your employment with the Company or any of its
subsidiaries prior to the Effective Time, and, prior to the Effective Time, the
Company agrees not to terminate your employment with the Company or its
subsidiaries without the prior written consent of the Parent.

         3.       Withholding and Deductions. The Company and the Parent will
withhold and deposit all federal, state and local income and employment taxes
that are owed with respect to all amounts paid or benefits provided to or for
you by the Company, the Parent or any affiliate pursuant to this Agreement. You,
the Company and the Parent agree that none of the payments and benefits payable
or provided to you or for your benefit in connection with the Merger are
expected to constitute "excess parachute payment" within the meaning of Section
280G of the Code. In the event that any amounts payable or benefits provided
hereunder would be or become subject to the excise tax under Section 4999 of the
Internal Revenue Code of 1986, as amended, such amounts and benefits shall be
treated in the manner set forth under Section 6 of the Company Agreement, the
provisions of which you and the Parent expressly assume under and in accordance
with the terms of this Agreement. You hereby agree to report any amounts paid or
benefits provided under this Agreement for purposes of Federal, state and local
income, employment and excise taxes in a manner consistent with the manner in
which the Parent reports any such amounts or benefits for purposes of Federal,
state and local income, employment and excise taxes and that you shall cooperate
with the Parent in good faith in connection with any valuation of the
restrictions and obligations under this Agreement.

         4.       Successors. This Agreement is personal to you and without the
prior written consent of the Parent shall not be assignable by you otherwise
than by will or the laws of descent and distribution. This Agreement shall inure
to the benefit of and be enforceable by your legal representatives. This
Agreement shall inure to the benefit of and be binding upon the Company, the
Parent and their successors and assigns.

                                       2
<PAGE>

         5.       Waiver. Failure of the Parent to demand strict compliance with
any of the terms, covenants or conditions of this Agreement shall not be deemed
a waiver of such term, covenant or condition, nor shall any waiver or
relinquishment of any such term, covenant or condition on any occasion or
multiple occasions be deemed a waiver or relinquishment of such term, covenant
or condition.

         6.       Governing Law and Jurisdiction. The Agreement is governed by
and construed under the laws of the State of Delaware, without regard to
conflict of laws rules. You, the Company and the Parent (a) hereby consent to
submit to the exclusive personal jurisdiction of any Federal court located in
the State of Connecticut or any court of the State of Connecticut in the event
any dispute arises out of this Agreement or any of the transactions contemplated
by this Agreement, and (b) hereby waive any right to challenge jurisdiction or
venue in such courts with regard to any suit, action, or proceeding under or in
connection with the Agreement. Each party to this Agreement also hereby waives
any right to trial by jury in connection with any suit, action, or proceeding
under or in connection with this Agreement.

         7.       Entire and Final Agreement. This Agreement shall supersede any
and all prior oral or written representations, understandings and agreements of
the parties with respect to their employment relationship (including, but not
limited to all correspondence, memoranda and term sheets and the Prior
Agreements), and it contains the entire agreement of the parties with respect to
those matters. No agreements or representations, oral or otherwise, express or
implied, with respect to the subject matter hereof have been made by either
party which are not set forth expressly in this Agreement. Once signed by the
parties hereto, no provision of this Agreement may be modified or amended unless
agreed to in a writing, signed by you and a duly authorized officer of the
Company and the Parent.

         8.       Assignment. Neither this Agreement nor any of the rights,
obligations or interests arising hereunder may be assigned by you. Neither this
Agreement nor any of the rights, obligations or interests arising hereunder may
be assigned by the Company or the Parent without your prior written consent, to
a person or entity other than an affiliate or parent entity of the Company or
the Parent, or their successors or assigns; provided, however, that, in the
event of the merger, consolidation, transfer, or sale of all or substantially
all of the assets of the Company or the Parent with or to any other individual
or entity, this Agreement shall, subject to the provisions hereof, be binding
upon and inure to the benefit of such successor and such successor shall
discharge and perform all the promises, covenants, duties, and obligations of
the Company or the Parent hereunder.

         9.       Section Headings. The section headings contained in this
Agreement are inserted for purposes of convenience only and shall not affect the
meaning or interpretation of this Agreement.

         10.      Notices. All notices required by this Agreement shall be sent
in writing and delivered by one party to the other by overnight express mail to
the following persons and addresses:

                                       3
<PAGE>

         If to the Company:

         FirstFed America Bancorp, Inc.
         One FirstFed Park
         Swansea, Massachusetts  02777
         Telecopy No.: (508) 235-1800
         Attention: Robert F. Stoico

         With a copy to the Parent.

         If to the Parent:

         Webster Financial Corporation
         Webster Plaza
         Waterbury, Connecticut  06702
         Attention: Harriet Wolfe, Executive Vice President, General Counsel and
         Secretary
         Telecopy No.: (203) 755-5539

         If to you:

         At the most recent address on file at the Company.

         11.      Execution in Counterparts. This Agreement may be executed by
the parties hereto in counterparts, and each of which shall be considered an
original for all purposes.

                                       4
<PAGE>

                  If the foregoing is satisfactory, please so indicate by
signing and returning to the Company and the Parent and the enclosed copy of
this letter whereupon this will constitute our agreement on the subject.

                                       WEBSTER FINANCIAL CORPORATION

                                       By:____________________________

                                       Name: James C. Smith, Chairman & Chief
                                             Executive Officer
                                       Date:________________

                                       FIRSTFED AMERICA BANCORP, INC.

                                       By:____________________________
                                       Name:
                                       Date:________________

                                       FIRST FEDERAL SAVINGS BANK OF AMERICA

                                       By:____________________________
                                       Name:
                                       Date:________________

ACCEPTED AND AGREED TO:

___________________________
Frederick R. Sullivan
Date:______________

                                       5
<PAGE>

                                    EXHIBIT A

Effective Date Payment:                 $1,026,931

                                        6