<PAGE>
 
EXHIBIT 10.4

                      AFFILIATED COMMUNITY BANCORP, INC.


                         SPECIAL TERMINATION AGREEMENT

     SPECIAL TERMINATION AGREEMENT ("Agreement") made as of this 17th day of
April, 1997, by and between Affiliated Community Bancorp, Inc., a Massachusetts
corporation (the "Corporation"), and William J. Gaddis, Jr. (the "Executive").

     1.   Purpose.  In order to allow the Executive to consider the prospect of
          -------                                                              
a Change in Control (as defined in Section 2 hereof) in an objective manner and
in consideration of the services to be rendered by the Executive to the
Corporation and other good and valuable consideration, the receipt and
sufficiency of which are hereby acknowledged by the Corporation, the Corporation
is willing to provide, subject to the terms of this Agreement, certain severance
benefits to protect the Executive from the consequences of a Terminating Event
(as defined in Section 3 hereof) occurring subsequent to a Change in Control.

     2.   Change in Control.  A "Change in Control" shall be deemed to have
          -----------------                                                
occurred in either of the following events: (i) when any "person" (as such term
is used in Sections 13(d) and 14(d)(2) of the Securities Exchange Act of 1934,
as amended (the "1934 Act")) becomes a "beneficial owner" (as such term is
defined in Rule 13d-3 promulgated under the 1934 Act), directly or indirectly,
of securities of the Corporation representing twenty-five percent (25%) or more
of the total number of votes that may be cast for the election of directors of
the Corporation; or (ii) if, as a result of, or in connection with, any tender
or exchange offer, merger or other business combination, sale of assets or
contested election, or any combination of the foregoing transactions, the
persons who were directors of the Corporation immediately before such
transaction shall cease to constitute a majority of the Board of Directors of
the Corporation or of any successor institution.

     3.   Terminating Event.  A "Terminating Event" shall mean (i) termination
          -----------------                                                   
by the Corporation of the employment of the Executive for any reason other than
(A) death, (B) deliberate dishonesty of the Executive with respect to the
Corporation or any subsidiary or affiliate thereof, or (C) conviction of the
Executive of a crime involving moral turpitude; or (ii) resignation of the
Executive from the employ of the Corporation, providing the Executive is not
receiving payments or benefits from the Corporation by reason of the Executive's
disability, subsequent to the occurrence of any of the following events: (A) a
significant reduction in the nature or scope of the Executive's
responsibilities, authorities, powers, functions or duties from the
responsibilities, authorities, powers, functions or duties exercised by the
Executive immediately prior to the Change in Control; (B) a decrease in the
total annual compensation payable by the Corporation to the Executive other than
as a result of a decrease in compensation payable to the Executive and to all
other executive officers of the Corporation on the basis of the Corporation's
financial performance; or (C) a reasonable determination by 
<PAGE>
 
the Executive that as a result of a Change in Control, he is unable to exercise
the responsibilities, authorities, powers, functions or duties exercised by the
Executive immediately prior to such Change in Control.

     4.   Severance Payment.  In the event a Terminating Event occurs within
          -----------------                                                 
three (3) years after a Change in Control, the Corporation shall pay to the
Executive an aggregate amount (the "Severance Payment") equal to two times the
individual's annualized includible compensation for the base period (the "base
amount," as defined in Section 280G(b)(3) of the Internal Revenue Code of 1986,
as amended (the "Code")) applicable to the Executive, payable in one lump-sum
payment on the date of termination or resignation, as the case may be.

     5.   Limitations on Benefits.
          ----------------------- 

          (a) It is the intention of the Executive and of the Corporation that
no payments by the Corporation to or for the benefit of the Executive under this
Agreement or any other agreement or plan pursuant to which he is entitled to
receive payments or benefits shall be non-deductible to the Corporation by
reason of the operation of Section 280G of the Code relating to parachute
payments.  Accordingly, and notwithstanding any other provision of this
Agreement or any such agreement or plan, if by reason of the operation of said
Section 280G, any such payments exceed the amount which can be deducted by the
Corporation, such payments shall be reduced to the maximum amount which can be
deducted by the Corporation. To the extent that payments exceeding such maximum
deductible amount have been made to or for the benefit of the Executive, such
excess payments shall be refunded to the Corporation with interest thereon at
the applicable Federal Rate determined under Section 1274(d) of the Code,
compounded annually, or at such other rate as may be required in order that no
such payments shall be non-deductible to the Corporation by reason of the
operation of said Section 280G.  To the extent that there is more than one
method of reducing the payments to bring them within the limitations of said
Section 280G, the Executive shall determine which method shall be followed,
provided that if the Executive fails to make such determination within forty-
five (45) days after the Corporation has sent him written notice of the need for
such reduction, the Corporation may determine the method of such reduction in
its sole discretion.

          (b) If any dispute between the Corporation and the Executive as to any
of the amounts to be determined under this Section 5(a), or the method of
calculating such amounts, cannot be resolved by the Corporation and the
Executive, either party after giving three (3) days written notice to the other,
may refer the dispute to a partner in the Boston office of a firm of independent
certified public accountants ("Partner") selected jointly by the Corporation and
the Executive.  The determination of such Partner as to the amount to be
determined under this Section 5(a) and the method of calculating such amounts
shall be final and binding on both the Corporation and the Executive.  The
Corporation shall bear the costs of any such determination.

     6.   Employment Status.  This Agreement is not an agreement for the
          -----------------                                             
employment of the Executive, and shall confer no rights on the Executive except
as herein expressly provided.
<PAGE>
 
     7.   Term.  This Agreement shall take effect on the date hereof, and shall
          ----                                                                 
terminate upon the earliest to occur of (a) the termination by the Corporation
of the employment of the Executive because of the occurrence of any of the
events set forth in Section 3(i) of this Agreement, (b) the resignation or
termination of the Executive for any reason prior to a Change in Control, or (c)
the resignation of the Executive after a Change in Control for any reason other
than the occurrence of any of the events enumerated in Section 3 of this
Agreement.

     8.   Withholding.  All payments made by the Corporation under this
          -----------                                                  
Agreement shall be net of any tax or other amounts required to be withheld by
the Corporation under applicable law.

     9.   Arbitration of Disputes.  Any controversy or claim arising out of or
          -----------------------                                             
relating to this Agreement or the breach thereof shall be settled by arbitration
in accordance with the laws of the Commonwealth of Massachusetts by three
arbitrators, one of whom shall be appointed by the Corporation, one by the
Executive and the third by the first two arbitrators.  If the first two
arbitrators cannot agree on the appointment of a third arbitrator, then the
third arbitrator shall be appointed by the American Arbitration Association in
the City of Boston.  Such arbitration shall be conducted in the City of Boston
in accordance with the rules of the American Arbitration Association, except
with respect to the selection of arbitrators which shall be as provided in this
Section 9.  Judgment upon the award rendered by the arbitrators may be entered
in any court having jurisdiction thereof.  In the event that it shall be
necessary or desirable for the Executive to retain legal counsel and/or incur
other costs and expenses in connection with the enforcement of any or all of the
Executive's rights under this Agreement, the Corporation shall pay (or the
Executive shall be entitled to recover from the Corporation, as the case may be)
the Executive's reasonable attorneys' fees and other reasonable costs and
expenses in connection with the enforcement of said rights (including the
enforcement of any arbitration award in court), unless and to the extent the
arbitrators shall determine that under the circumstances recovery by the
executive of all or a part of any such fees and costs and expenses would be
unjust.  The provisions of this Section 9 shall not apply to Section 5(b) of
this Agreement, except in the event that the Corporation and the Executive
cannot agree on the selection of the Partner described in said Section.

     10.  Assignment.  Neither the Corporation nor the Executive may make any
          ----------                                                         
assignment of this Agreement or any interest herein, by operation of law or
otherwise, without the prior written consent of the other party.  This Agreement
shall inure to the benefit of and be binding upon the Corporation and the
Executive, their respective successors, executors, administrators, heirs and
permitted assigns.  In the event of the Executive's death prior to the
completion by the Corporation of all payments due him under this Agreement, the
Corporation shall continue such payments to the Executive's beneficiary
designated in Exhibit A to this Agreement (or to his estate, if he fails to make
              ---------                                                         
such designation).
<PAGE>
 
     11.  Enforceability.  If any portion or provision of this Agreement shall
          --------------                                                      
to any extent be declared illegal or unenforceable by a court of competent
jurisdiction or under any federal or state banking law, regulation or order,
then the remainder of this Agreement, or the application of such portion or
provision in circumstances other than those as to which it is so declared
illegal or unenforceable, shall not be affected thereby, and each portion and
provision of this Agreement shall be valid and enforceable to the fullest extent
permitted by law.

     12.  Waiver.  No waiver of any provision hereof shall be effective unless
          ------                                                              
made in writing and signed by the waiving party.  The failure of either party to
require the performance of any term or obligation of this Agreement, or the
waiver by either party of any breach of this Agreement, shall not prevent any
subsequent enforcement of such term or obligation or be deemed a waiver of any
subsequent breach.

     13.  Notices.  Any notices, requests, demands and other communications
          -------                                                          
provided for by this Agreement shall be sufficient if in writing and delivered
in person or sent by registered or certified mail, postage prepaid, to the
Executive at the last address the Executive has filed in writing with the
Corporation or, in the case of the Corporation, at the main office of the
Corporation, attention of the Clerk.

     14.  Other Plans.  An election by the Executive to resign after a Change in
          -----------                                                           
Control under the provisions of this Agreement shall not be deemed a voluntary
termination of employment by the Executive for the purpose of interpreting the
provisions of any of the Corporation's or any subsidiary's benefit plans,
programs or policies.  Nothing in this Agreement shall be construed to limit the
rights of the Executive under any severance agreement he may then have with the
Corporation, provided, however, that if there is a Terminating Event under
             --------  -------                                            
Section 3 hereof, the Executive may elect either to receive the severance
payment provided under Section 4 or such termination benefits as he may have
under any such severance agreement, but may not elect to receive both.

     15.  Complete Agreement.   This Agreement, contains the entire agreement
          ------------------                                                 
and understanding of the parties with respect to the subject matter. There are
no restrictions, agreements, promises, warranties, covenants or undertakings
between the parties other than those expressly set forth herein. This Agreement
supersedes all prior agreements and understandings between the parties, both
written and oral, with respect to its subject matter.

     16.  Definitions.  As used herein, the term Corporation shall include
          -----------                                                     
but not be limited to, any wholly owned subsidiary of Affiliated Community
Bancorp, Inc. where the context so requires.

     17.  Amendment.  This Agreement may be amended or modified only by a
          ---------                                                      
written instrument signed by the Executive and by a duly authorized
representative of the Corporation.
<PAGE>
 
     18.  Governing Law.  This is a Massachusetts contract and shall be
          -------------                                                
construed under and be governed in all respects by the laws of the Commonwealth
of Massachusetts.

     IN WITNESS WHEREOF, this Agreement has been executed as a sealed instrument
as of the date first above written.

ATTEST:



/S/ JOAN F. WILSON                           /S/ WILLIAM J. GADDIS, JR.
----------------------------------           ---------------------------------
Name: Joan F. Wilson                         Executive: William J. Gaddis, Jr.



ATTEST:                                      AFFILIATED COMMUNITY BANCORP, INC.



/S/ JOHN G. FALLON                           By: /S/ TIMOTHY J. HANSBERRY
----------------------------------           ----------------------------------
John G. Fallon,                                  Timothy J. Hansberry,
Executive Vice President                         President & CEO


[SEAL]