<PAGE>   1
                                                                Exhibit 10.3

                          SPECIAL TERMINATION AGREEMENT

     AGREEMENT dated as of December 23, 1993, as amended and restated as of
January 23, 1997, by and among ANDOVER BANCORP, INC., a Delaware corporation
(the "Company") and its subsidiary, ANDOVER BANK, a Massachusetts savings bank
with its main office in Andover, Massachusetts (the "Bank") (the Bank and the
Company shall be hereinafter collectively referred to as the "Employers"), and
Joseph F. Casey (the "Executive"), an individual presently employed as an
officer of the Company and the Bank.

     1. PURPOSE. In order to allow the Executive to consider the prospect of a
Change in Control (as defined in Section 2) in an objective manner and in
consideration of the services rendered and to be rendered by the Executive to
the Employers and other good and valuable consideration, the receipt and
sufficiency of which is hereby acknowledged by the Employers, the Employers are
willing to provide, subject to the terms of this Agreement, certain severance
benefits to protect the Executive from the consequences of a Change in Control
Terminating Event (as defined in Section 3) or a Merger of Equals Terminating
Event (as defined in Section 3) occurring subsequent to a Change in Control or a
Merger of Equals (as defined in Section 2), respectively.

     2. CHANGE IN CONTROL. A "Change in Control" shall be deemed to have
occurred in any one of the following events:

          (a) any "person," as such term is used in Sections 13(d) and 14(d) of
     the Securities Exchange Act of 1934 (the "Act") (other than the Company,
     any of its direct or indirect subsidiaries, or any trustee, fiduciary or
     other person or entity holding securities under any employee benefit plan
     or trust of the Company or any of its direct or indirect subsidiaries),
     together with all "affiliates" and "associates" (as such terms are defined
     in Rule 12b-2 under the Act) of such person, shall become the "beneficial
     owner" (as such term is defined in Rule 13d-3 under the Act), directly or
     indirectly, of securities of the Company representing 25% or more of the
     combined voting power of the Company's then outstanding securities having
     the right to vote in an election of the Company's Board of Directors
     ("Voting Securities") (in such case other than as a result of an
     acquisition of securities directly from the Company); or

          (b) persons who, as of the date hereof, constitute the Company's Board
     of Directors (the "Incumbent Directors") cease for any reason, including,
     without limitation, as a result of a tender offer, proxy contest, merger or
     similar transaction, to constitute at least a majority of the Board,
     provided that any person becoming a director of the Company subsequent to
     the date hereof whose election or nomination for election was approved by a
     vote of at least a majority of the Incumbent Directors shall, for purposes
     of this Agreement, be considered an Incumbent Director; or

          (c) the shareholders of the Company shall approve (i) any
     consolidation or merger of the Company or any subsidiary of the Company
     where the shareholders of the Company, immediately prior to the
     consolidation or merger, would not, immediately after the consolidation or
     merger, beneficially own (as such term is defined in Rule 13d-3 under the
     Act), directly or indirectly, shares representing in the aggregate a
     majority of 

<PAGE>   2



     the voting shares of the corporation issuing cash or securities in the
     consolidation or merger (or of its ultimate parent corporation, if any),
     (ii) any sale, lease, exchange or other transfer (in one transaction or a
     series of transactions contemplated or arranged by any party as a single
     plan) of all or substantially all of the assets of the Company, or (iii)
     any plan or proposal for the liquidation or dissolution of the Company.

     Notwithstanding the foregoing, a "Change in Control" shall not be deemed to
have occurred:

          (A) for purposes of the foregoing clause (a), solely as the result of
     an acquisition of securities by the Company which, by reducing the number
     of shares of Voting Securities outstanding, increases the proportionate
     voting power represented by the Voting Securities beneficially owned by any
     person to 25% or more of the combined voting power of all then outstanding
     Voting Securities; provided, however, that if any person referred to in the
     preceding clause shall thereafter become the beneficial owner of any
     additional shares of Voting Securities (other than pursuant to a stock
     split, stock dividend, or similar transaction), then a Change in Control
     shall be deemed to have occurred for purposes of the foregoing clause (a);
     or

          (B) as a result of the occurrence of any of the events enumerated in
     the foregoing clauses (a), (b) or (c), if after such occurrence, the
     Incumbent Directors constitute at least one-half of the Board of Directors
     of the Company or any successor institution and the transaction is
     designated a merger of equals transaction by the Incumbent Directors prior
     to the occurrence of such event (a "Merger of Equals").

     3. TERMINATING EVENT. A "Change in Control Terminating Event" shall mean
any termination by the Employers or by the Executive of the Executive's
employment for any reason. A "Merger of Equals Terminating Event" shall mean any
termination by the Employers of the Executive's employment for any reason other
than death or for Cause (as defined below). For purposes of this Agreement, the
term "Cause" shall mean (i) deliberate dishonesty of the Executive with respect
to either of the Employers or any subsidiary or affiliate of either, or (ii)
conviction of the Executive of a crime involving moral turpitude.

     4. SEVERANCE PAYMENT. Subject to the provisions of Section 5 below, in the
event that a Change in Control Terminating Event occurs within two (2) years
after a Change in Control or a Merger of Equals Terminating Event occurs within
two (2) years after a Merger of Equals, the Employers shall pay to the Executive
an aggregate amount equal to (x) three times the "base amount" (as defined in
Section 280G(b)(3) of the Internal Revenue Code of 1986, as amended (the
"Code")) applicable to the Executive, less (y) One Dollar ($1.00), payable in
one lump-sum payment on the date of such termination.

     5. LIMITATION ON BENEFITS.

     (a) It is the intention of the Executive and of the Employers that no
payments by the Employers to or for the benefit of the Executive under this
Agreement and/or any other agreement or plan pursuant to which the Executive is
entitled to receive payments or benefits

                                       2
<PAGE>   3


shall be non-deductible to the Employers by reason of the operation of Section
280G of the Code relating to parachute payments. Accordingly, and
notwithstanding any other provision of this Agreement or any such agreement or
plan, if by reason of the operation of said Section 280G, any such payments
exceed the amount which can be deducted by the Employers in the aggregate, such
payments shall be reduced to the maximum amount which can be deducted by the
Employers. To the extent that payments exceeding such maximum deductible amount
have been made to or for the benefit of the Executive, such excess payments
shall be refunded to the Employers with interest thereon at the applicable
Federal Rate determined under Section 1274(d) of the Code, compounded annually,
or at such other rate as may be required in order that no such payments shall be
non-deductible to the Employers by reason of the operation of said Section 280G.
To the extent that there is more than one method of reducing the payments to
bring them within the limitations of said Section 280G, the Executive shall
determine which method shall be followed, provided that if the Executive fails
to make such determination within forty-five days after the Employers have sent
him written notice of the need for such reduction, the Employers may determine
the method of such reduction in their sole discretion.

     (b) If any dispute between the Employers and the Executive as to any of the
amounts to be determined under this Section 5, or the method of calculating such
amounts, cannot be resolved by the Employers and the Executive, either the
Employers or the Executive after giving three days written notice to the other,
may refer the dispute to a partner in the Boston office of a firm of independent
certified public accountants selected jointly by the Employers and the
Executive. The determination of such partner as to the amount to be determined
under Section 5(a) and the method of calculating such amounts shall be final and
binding on both the Employers and the Executive. The Employers shall bear the
costs of any such determination.

     6. EMPLOYMENT STATUS. This Agreement is not an agreement for the employment
of the Executive and shall confer no rights on the Executive except as herein
expressly provided.

     7. TERM. This Agreement shall take effect January 23, 1997, and shall
terminate upon the termination of employment of the Executive for any reason
prior to a Change in Control or a Merger of Equals.

     8. WITHHOLDING. All payments made by the Employers under this Agreement
shall be net of any tax or other amounts required to be withheld by the
Employers under applicable law.

     9. ARBITRATION OF DISPUTES. Any controversy or claim arising out of or
relating to this Agreement or the breach thereof shall be settled by arbitration
in accordance with the laws of the Commonwealth of Massachusetts by three
arbitrators, one of whom shall be appointed by the Employers, one by the
Executive and the third by the first two arbitrators. If the first two
arbitrators cannot agree on the appointment of a third arbitrator, then the
third arbitrator shall be appointed by the American Arbitration Association in
the City of Boston. Such arbitration shall be conducted in the City of Boston in
accordance with the rules of the American Arbitration Association, except with
respect to the selection of arbitrators which shall be as provided in this
Section 9. Judgment upon the award rendered by the arbitrators may be entered

                                       3

<PAGE>   4


in any court having jurisdiction thereof. In the event that it shall be
necessary or desirable for the Executive to retain legal counsel and/or incur
other costs and expenses in connection with the enforcement of any or all of the
Executive's rights under this Agreement, the Employers shall pay (or the
Executive shall be entitled to recover from the Employers, as the case may be),
the Executive's reasonable attorneys' fees and other reasonable costs and
expenses in connection with the enforcement of said rights (including the
enforcement of any arbitration award in court) regardless of the final outcome,
unless and to the extent the arbitrators shall determine that under the
circumstances recovery by the Executive of all or a part of any such fees and
costs and expenses would be unjust. This provision shall not apply to Section
5(b), except in the event that the Employers and the Executive cannot agree on
the selection of the accounting partner described in said section.

     10. ASSIGNMENT; SUCCESSORS AND ASSIGNS, ETC. Neither the Employers nor the
Executive may make any assignment of this Agreement or any interest herein, by
operation of law or otherwise, without the prior written consent of the other
party. This Agreement shall inure to the benefit of and be binding upon the
Employers and the Executive, their respective successors, executors,
administrators, heirs and permitted assigns. In the event of the Executive's
death prior to the completion by the Employers of all payments due to the
Executive under this Agreement, the Employers shall continue such payments to
the Executive's beneficiary designated in writing to the Employers prior to the
Executive's death (or to the Executive's estate, if the Executive fails to make
such designation).

     11. ENFORCEABILITY. If any portion or provision of this Agreement shall to
any extent be declared illegal or unenforceable by a court of competent
jurisdiction, then the remainder of this Agreement, or the application of such
portion or provision in circumstances other than those as to which it is so
declared illegal or unenforceable, shall not be affected thereby, and each
portion and provision of this Agreement shall be valid and enforceable to the
fullest extent permitted by law. Notwithstanding anything to the contrary
herein, in no event shall any benefits be paid hereunder if the Executive's
employment is terminated (i) due to the removal, suspension or prohibition of
the Executive from participating in the conduct of the Employers' affairs by
order issued under applicable laws and regulations by a federal or state banking
agency having authority over the Employers, or (ii) by reason of the FDIC's
appointment as a conservator or receiver of the Bank pursuant to applicable laws
and regulations or of the Massachusetts Commissioner of Banks' taking possession
of the Bank pursuant to applicable laws and regulations.

     12. WAIVER. No waiver of any provision hereof shall be effective unless
made in writing and signed by the waiving party. The failure of any party to
require the performance of any term or obligation of this Agreement, or the
waiver by any party of any breach of this Agreement, shall not prevent any
subsequent enforcement of such term or obligation or be deemed a waiver of any
subsequent breach.

     13. NOTICES. Any notices, requests, demands and other communications
provided for by this Agreement shall be sufficient if in writing and delivered
in person or sent by registered or certified mail, postage prepaid, to the
Executive at the last address the Executive has filed 


                                       4

<PAGE>   5


in writing with the Employers or, in the case of the Bank, at its main offices
attention of the Clerk or, in the case of the Company, at its main office,
attention of the Secretary.

     14. ENTIRE AGREEMENT. This Agreement is complete, reflects the entire
agreement of the parties with respect to its subject matter, and supersedes all
previous written or oral negotiations, commitments and writings.

     15. ELECTION OF REMEDIES. An election by the Executive to resign after a
Change in Control or a Merger of Equals under the provisions of this Agreement
shall not constitute a breach by the Executive of any employment agreement
between the Employers and the Executive and shall be deemed a voluntary
termination of employment by the Executive for the purpose of interpreting the
provisions of any of the Employers' benefit plans, programs or policies. Nothing
in this Agreement shall be construed to limit the rights of the Executive under
any employment agreement the Executive may then have with the Employers,
PROVIDED, HOWEVER, that if there is a Change of Control Terminating Event or
Merger of Equals Terminating Event under Section 3 hereof, the Executive may
elect either to receive the severance payment provided under Section 4 or such
termination benefits as the Executive may have under any such employment
agreement, but may not elect to receive both.

     16. AMENDMENT. This Agreement may be amended or modified only by a written
instrument signed by the Executive and by duly authorized representatives of
each of the Employers.

     17. ALLOCATION OF OBLIGATIONS BETWEEN EMPLOYERS. The obligations of the
Employers under this Agreement are intended to be the joint and several
obligations of the Bank and the Company and the Employers shall, as between
themselves, allocate these obligations in a manner agreed upon by them.

     18. GOVERNING LAW. This is a Massachusetts contract and shall be construed
under and be governed in all respects by the laws of the Commonwealth of
Massachusetts.



                                       5
<PAGE>   6


     IN WITNESS WHEREOF, this Agreement has been executed as a sealed instrument
by the Company, the Bank and the Executive as of the date first above written.

                              ANDOVER BANCORP, INC.


                              By:/s/Gerald T. Mulligan
                                 --------------------------------------------
                                 Title: President and Chief Executive Officer


                              ANDOVER BANK


                              By:/s/Gerald T. Mulligan
                                 --------------------------------------------
                                 Title: President and Chief Executive Officer


                                 /s/Joseph F. Casey
                              ----------------------------------------------- 
                              Executive:  Joseph F. Casey






                                        6