<PAGE>   1
                                                                    EXHIBIT 10.4



                              BANKNORTH GROUP, INC.


                               SEVERANCE AGREEMENT



      This AGREEMENT, made and entered into as of the 10th day of May 2000,
by and among BANKNORTH GROUP, INC. (the "Company") and Richard E. Johnson
(the "Executive").


                             W I T N E S S E T H:


      WHEREAS, the Executive is employed by the Company in a key executive
capacity and possesses intimate knowledge of the business and affairs of the
Company; and

      WHEREAS, the Company desires to ensure, insofar as possible, that it will
continue to have the benefit of the Executive's services and to protect its
confidential information and goodwill; and

      WHEREAS, the Company recognizes that circumstances may arise in which a
change in the control of the Company occurs, thereby causing uncertainty of
employment without regard to the Executive's competence or past contributions;
and

      WHEREAS, the Company and the Executive wish to provide reasonable security
to the Executive against changes in the Executive's relationship with the
Company in the event of such change in control;

      NOW, THEREFORE, in consideration of the foregoing and of the mutual
covenants and agreements hereinafter set forth, the parties hereto agree as
follows:
<PAGE>   2
      1.    Definitions

            (a)   Accrued Benefits means:

                  (i)   All salary earned or accrued through the date the
Executive's employment is terminated;

                  (ii) reimbursement for any and all monies advanced in
connection with the Executive's employment for reasonable and necessary expenses
incurred by the Executive through the date the Executive's employment is
terminated;

                  (iii) any and all other compensation previously earned and
deferred at the election of the Executive or pursuant to any deferred
compensation plans then in effect together with any interest or desired earnings
thereon;

                  (iv) annual bonus, if any, accrued for a Year prior to the
Year in which employment terminates, but not yet paid to the Executive, under
any bonus or incentive compensation plan or plans in which the Executive is a
participant;

                  (v) a pro rata portion of the maximum bonus payable to the
Executive for the Year in which employment terminates under any bonus or
incentive compensation plan or plans in which the Executive is a participant,
determined as if the Executive had remained in employment for the full Year and
prorated based upon weeks, including partial weeks, of employment during that
Year;

                  (vi) all other payments and benefits to which the Executive
may be entitled under the terms of any applicable compensation arrangement or
benefit plan or program of the Company.

            (b) Act means the Securities Exchange Act of 1934, as amended.




                                       2
<PAGE>   3
            (c) Affiliate of any specified persons means any other person that,
directly or indirectly, through one or more intermediaries, controls, or is
controlled by, or is under direct or indirect common control with such specified
person. For the purposes of this definition, "control" means the possession,
direct or indirect, of the power to direct or cause the direction of the
management and policies of a person, whether through the ownership of voting
securities, by contract or otherwise, and the terms "controlling" and
"controlled" have meanings correlative to the foregoing.

            (d) Annual Compensation shall mean the sum of:

                  (i) the Executive's annual base salary at the rate in effect
on the date of a termination of employment as described in Section 3 or in
Section 7(d) (or, in the event of a termination for Good Reason under Section
1(k)(i)(A) below, the annual base salary as in effect immediately before the
actions giving rise to Good Reason); plus

                  (ii) the greatest of the bonuses either paid or accrued in
either the Year of the Change in Control or the immediately preceding Year.

            (e) Base Amount means an amount equal to the Executive's Annualized
Includable Compensation for the Base Period as defined in Section 280G(d)(1) and
(2) of the Code (as hereinafter defined).

            (f) Cause means (i) the executive's conviction of, or plea of nolo
contendere to, a felony; or (ii) willful and intentional misconduct, willful
neglect, or gross negligence, in the performance of the Executive's duties,
which has caused a demonstrable and serious injury to the Company, monetary or
otherwise.



                                       3
<PAGE>   4
      The Executive shall be given written notice that the Company intends to
terminate his employment for Cause. Such written notice shall specify the
particular acts, or failures to act, on the basis of which the decision to so
terminate employment was made.

      In the case of a termination for Cause as described in Clause (ii), above,
the Executive shall be given the opportunity within 30 days of the receipt of
such notice to meet with the Board to defend such acts, or failures to act,
prior to termination. The Company may suspend the Executive's title and
authority pending such meeting, and such suspension shall not constitute "Good
Reason," as defined in subsection (k) below.

            (g) Change in Control of the Company shall mean a Change in Control
of a nature that would be required to be reported in response to Item 5(f) of
Schedule 14A of Regulation 14A promulgated under the Act or any successor
thereto, provided that without limiting the foregoing, a Change in Control of
the Company also shall mean the occurrence of any of the following events:

                  (i) any "person" (as defined under Section 3(a)(9) of the Act)
or "group" of persons (as provided under Rule 13d-3 of the Act) is or becomes
the "beneficial owner" (as defined in Rule 13d-3 or otherwise under the Act),
directly or indirectly (including as provided in Rule 13d-3(d)(1) of the Act),
of capital stock of the Company the holders of which are entitled to vote for
the election of directors ("voting stock") representing that percentage of the
Company's then outstanding voting stock (giving effect to the deemed ownership
of securities by such person or group, as provided in Rule 13d-3(d)(1) of the
Act, but not giving effect to any such deemed ownership of securities by another
person or group) equal to or greater than twenty-five percent (25%) of all such
voting stock;



                                       4
<PAGE>   5
                  (ii) During any period of twenty four consecutive months,
individuals who at the beginning of such period constituted the Board of
Directors of the Company (including for this purpose any new director whose
election or nomination for election by the Company's shareholders was approved
by a vote of at least a majority of the directors then still in office who were
directors at the beginning of such period) cease for any reason to constitute at
least a majority of the Board of Directors of the Company (excluding any Board
seat that is vacant or otherwise unoccupied).

                  (iii) There shall be consummated any consolidation, merger,
stock for stock exchange or similar transaction (collectively, "Merger
Transactions") involving securities of the Company in which holders of voting
stock of the Company immediately prior to such consummation own, as a group,
immediately after such consummation, voting stock of the Company (or, if the
Company does not survive the Merger Transaction, voting securities of the
corporation surviving such transaction) having less than 50% of the total voting
power in an election of directors of the Company (or such other surviving
corporation).

            (h) Code means the Internal Revenue Code of 1986, including any
amendments thereto.

            (i) Company Retirement Plan is defined in Section 1(r)(vi) below.

            (j) Effective Date means the date this Agreement is executed by the
parties.

            (k) Employment Period means a period commencing on the date of a
Change in Control of the Company and ending on the earlier of (i) the last day
of the twenty-fourth month following the month in which the Change in Control
occurs, or (ii) the Executive's Normal Retirement Date.

            (l) Good Reason means:



                                       5
<PAGE>   6
                  (i) any breach of this Agreement by the Company, including
without limitation (A) any reduction during the employment period in the amount
of the Executive's base salary or aggregate benefits as in effect from time to
time, (B) failure to provide the Executive with the same fringe benefits that
were provided to the Executive immediately prior to a Change in Control of the
Company, or with a package of fringe benefits (including paid vacations) that,
though one or more of such benefits may vary from those in effect immediately
prior to such a Change in Control, is substantially comparable in all material
respects to such fringe benefits taken as a whole, or (C) any other breach by
the Company of its obligations contained in Section 6 below;

                  (ii) without the Executive's express written consent, the
assignment to the Executive of any duties which are materially inconsistent with
the Executive's positions, duties, responsibilities and status immediately prior
to the Change in Control of the Company, a material change in the Executive's
reporting responsibilities, titles or offices as an employee and as in effect
immediately prior to the Change in Control, or a significant reduction, in the
Executive's title, duties or responsibilities, or in the level of his support
services;

                  (iii) the relocation of the Executive's principal place of
employment, without the Executive's written consent, to a location outside the
same metropolitan area in which the Executive was employed at the time of such
Change in Control, or the imposition of any requirement that the Executive spend
more than ninety business days per year at a location other than such principal
place of employment; or

                  (iv) any purported termination of the Executive's employment
for Cause, Disability or Retirement which is not effected pursuant to a Notice
of Termination satisfying the requirements of paragraph (n) below.




                                       6
<PAGE>   7
      Upon the occurrence of any of the events described in (i), (ii), (iii), or
(iv) above, the Executive shall give the Company written notice that such event
constitutes Good Reason, and the Company shall thereafter have thirty (30) days
in which to cure. If the Company has not cured in that time, the event shall
constitute Good Reason.

            (m)   Normal Retirement Date means Normal Retirement Date as
defined in the Banknorth Group, Inc. Retirement Plan.

            (n) Notice of Termination shall mean a notice which shall indicate
the specific termination provision relied upon in this Agreement and shall set
forth in reasonable detail the facts and circumstances claimed to provide a
basis for termination of Executive's employment under the provision so
indicated.

            (o) Person or Group means a "person" or "group," as defined in
Section 1(g)(i) hereof.

            (p) Plan Year with respect to any of the Retirement Plan, the Thrift
Plan (as defined below) or the Employee Stock Ownership Plan, the "plan year" as
defined in such plan.

            (q) Retirement Plan means the Banknorth Group, Inc. Retirement Plan
or any successor plan.

            (r) SERP Agreement means the Supplemental Retirement Agreement
between the Executive and the Company.

            (s) Year means a calendar year unless otherwise specifically
provided.

      2.    Term of Agreement.

This Agreement shall begin on the Effective Date and shall continue until the
third anniversary of such date, provided that, on the first anniversary of the
Effective Date, and on each succeeding





                                       7
<PAGE>   8
anniversary, the term shall be renewed for an additional period of one year
unless either party has given written notice that the term is not so renewed,
which notice must be delivered to the other party at least ninety (90) days
prior to the date of any such renewal, and further provided that if a Change in
Control of the Company occurs during such term, the term shall in all events
continue through the last day of the Employment Period. This Agreement is also
subject to earlier termination as provided in Section 3 below. All rights and
obligations hereunder shall survive to the extent necessary to the intended
enforcement thereof.

      3. Termination of Employment Prior to a Change in Control.

            (a) The Company and the Executive shall each retain the right to
terminate the employment of the Executive at any time prior to a Change in
Control of the Company. In the event the Executive's employment is terminated
prior to a Change in Control of the Company, this Agreement shall, except as
provided in Subsection (b) below, be terminated and of no further force and
effect, and any and all rights and obligations of the parties hereunder shall
cease.

            (b) If the Executive's employment is terminated by the Company prior
to the occurrence of a Change in Control of the Company, and if it can be shown
that the Executive's termination (i) was at the direction or request of a third
party that had taken steps reasonably calculated to effect the Change in Control
of the Company thereafter, or (ii) otherwise occurred in connection with, or in
anticipation of, the Change in Control of the Company, the Executive shall have
the rights described in Section 7(d) below, as if a Change in Control of the
Company had occurred on the date immediately preceding such termination.




                                       8
<PAGE>   9
      4. Employment Following a Change in Control.

      If a Change in Control of the Company occurs when the Executive is
employed by the Company, the Company will continue thereafter to employ the
Executive, and the Executive will remain in the employ of the Company, during
the Employment Period, in accordance with the terms and provisions of this
Agreement.

      5.    Duties.

      During the Employment Period, the Executive shall serve in such capacities
and positions as may be assigned by the Company consistent with the Executive's
capacities and positions on the Effective Date and shall devote the Executive's
best efforts and all of the Executive's business time, attention and skill to
the business and affairs of the Company, as such business and affairs now exist
and as they may hereafter be conducted.

      6.    Compensation.

      During the Employment Period, the Executive shall be compensated by the
Company as follows:

            (a) the Executive shall receive, at such intervals and in accordance
with such standard policies as in effect on the date of the Change in Control of
the Company, an annual base salary not less than the Executive's annual base
salary as in effect on the date of the Change in Control of the Company, subject
to adjustment as hereinafter provided;

            (b) the Executive shall be included in all plans providing incentive
compensation to executives, including but not limited to bonus, deferred
compensation, annual or other incentive compensation, supplemental pension,
stock ownership, stock option, stock appreciation, stock bonus





                                       9
<PAGE>   10
and similar or comparable plans as any such plans are extended by the Company
from time to time to senior corporate officers, key employees and other
employees of comparable status;

            (c) the Executive shall be reimbursed, at such intervals and in
accordance with such standard policies as may be in effect on the date of the
Change in Control of the Company, for any and all monies advanced in connection
with the Executive's employment for reasonable and necessary expenses incurred
by the Executive on behalf of the Company, including travel expenses;

            (d) the Executive shall be included, to the extent eligible
thereunder, in any and all plans providing but not limited to, group life
insurance, hospitalization, disability, medical, dental, pension, profit sharing
and stock bonus plans, and shall be provided any and all other benefits and
perquisites made available to other employees of comparable status and position
at the expense of the Company on a comparable basis;

            (e) the Executive shall receive annually not less than the amount of
paid vacation and not fewer than the number of paid holidays received annually
immediately prior to the Change in Control of the Company or available annually
to other employees of comparable status and position with the Company; and

            (f) During the Employment Period the Board of Directors of the
Company, or an appropriate committee thereof, will consider and appraise, at
least annually, the contributions of the Executive to the Company's operating
efficiency, growth, production and profits and, in accordance with past
practice, due consideration shall be given to the upward adjustment of the
Executive's compensation rate, at least annually, commensurate with increases
generally given to other senior corporate officers and key employees and as the
scope of the Executive's duties expands.




                                       10
<PAGE>   11
      7.    Termination of Employment.

Any termination by the Company or the Executive of the Executive's employment
during the Employment Period shall be communicated by written Notice of
Termination to the Executive if such notice is delivered by the Company and to
the Company if such notice is delivered by the Executive. The Notice of
Termination shall comply with the requirements of Section 1(n).

            (a) Termination for Disability. If during the Employment Period, the
Executive's employment is terminated on account of the Executive's disability,
as determined under the Company's long-term disability plan (as in effect on the
date of a Change in Control of the Company), the Executive shall receive all
Accrued Benefits, if any, and shall remain eligible for all benefits as provided
pursuant to the terms of any long-term disability programs of the Company in
effect at the time of such termination.

            (b) Termination on the Executive's Death. If, during the Employment
Period, the Executive's employment is terminated on account of the Executive's
death, the Executive's estate or his designated beneficiary (or beneficiaries),
as applicable, shall receive the Accrued Benefits.

            (c) Voluntary Termination or Termination for Cause. If, during the
Employment Period, (i) the Executive shall terminate employment with the Company
other than for Good Reason, or (ii) the Executive's employment is terminated for
Cause, the Executive shall receive from the Company only the Accrued Benefits.

            (d) Termination by the Company Without Cause or by the Executive for
Good Reason. If, during the Employment Period, the Executive's employment with
the Company is terminated by the Company other than for Cause, or by the
Executive for Good Reason, then:




                                       11
<PAGE>   12
                  (i) the Executive shall be entitled to receive from the
Company the Accrued Benefits, except that, for this purpose, Accrued Benefits
shall not include any entitlement to severance under any Company severance
policy generally applicable to the Company's salaried employees;

                  (ii) the Executive shall receive from the Company, no less
than ten (10) days following termination of his employment, a lump sum payment
(the "Termination Payment") equal to two (2) times the Annual Compensation;

                  (iii) for purposes of determining the Executive's benefit
under the SERP Agreement, the Executive shall be credited with 24 additional
months of age and of service determined as follows:

                        (A)   The additional 24 months of age and service
shall be applied for purposes of benefit accrual, vesting, eligibility for early
retirement, subsidized early retirement factors, actuarial equivalence and any
other purposes under the Retirement Plan and the SERP Agreement.

                        (B) Any provision under the Retirement Plan or the
SERP Agreement prohibiting the accrual of any additional benefits after the
Executive has been credited with more than a stated number of years of service
shall be disregarded.

                        (C) For purposes of determining the amount of the
Executive's benefit under the SERP Agreement, the reduction in respect of the
benefit paid under the Retirement Plan shall be based on the Executive's actual
Retirement Plan benefit (that is, without any additional deemed service).



                                       12
<PAGE>   13
                        (D) For purposes of determining the Early
Retirement Benefit (as defined in the SERP Agreement) and other optional forms
of benefit under the SERP Agreement, if the Executive is less than fifty-five
(55) years of age, the Executive shall be deemed to be at least fifty-five (55)
years of age on the date his employment with the Company terminates,
notwithstanding the Executive's actual age, if less.

                        (E) the Benefit Computation Base (as defined in the
SERP Agreement) shall be determined as if it were being calculated at the end of
the 24 month period of service credited to the Executive under this Section
7(d)(iii) and as if during such 24 additional month period the Executive's
annualized compensation was the same as such compensation for (I) the Year
during which the Executive's employment is terminated, or, (II) any Year before
the Change in Control occurred, whichever is greater.

                        (F) Any amendment to the Retirement Plan within the
twelve (12) month period prior to the termination of the Executive's employment
shall be disregarded to the extent that the application of such amendment would
decrease the total amount of the benefits provided for in this Section
7(d)(iii).

                        (G) The Executive shall be entitled to a lump sum
distribution of his SERP in all events, and the Company shall not be entitled to
require payment over a longer period. If the Executive elects a lump sum payment
(i) the actuarial equivalent benefit shall be determined in accordance with the
provisions of the Retirement Plan as in effect immediately prior to the Change
in Control, or as in effect on termination of the Executive's employment,
whichever creates the greater benefit, and (ii) payment shall be made within
thirty (30) days following the later




                                       13
<PAGE>   14
of (A) termination of the Executive's employment, or (B) the date the Executive
gives written notice of the Executive's intent to elect a lump sum.

                  (iv) the Executive shall be paid a lump sum amount equal to
(A) the sum of (I) the aggregate contributions and forfeitures allocated to the
Executive's account under the Banknorth Group, Inc. Profit Sharing and Employee
Stock Ownership Plan (the "Employee Stock Ownership Plan") for the Plan Year
ending immediately prior to the Change of Control, or, if different, the Plan
Year ending immediately prior to the termination of the Executive's employment,
whichever Plan Year would produce the greater aggregate value, (II) the total
aggregate value of all contributions, other than elective contributions by the
Executive and employer matching contributions relating thereto, and forfeitures
allocated to the Executive's account under the Banknorth Group, Inc. Thrift
Incentive Plan (the "Thrift Plan") for the Plan Year ending immediately prior to
the Change of Control, or, if different, the Plan Year ending immediately prior
to the termination of the Executive's employment, whichever Plan Year would
produce the greater aggregate value, and (III) (A) the matching contributions
under the Thrift Plan (or its successor) which would have been credited under
such plan on Executive's behalf, if the Executive had contributed the maximum
salary deferral contribution allowable under Section 402(g) of the Code, for the
calendar year in which he terminated employment with the Company, multiplied by
(B) two (2).

                  (v) all rights under any equity or long-term incentive plan
shall be fully vested;

                  (vi) the Executive shall (A) continue to be covered at the
expense of the Company by the same or equivalent hospital, medical, dental,
accident, disability and life insurance




                                       14
<PAGE>   15
coverage as in effect for the Executive immediately prior to termination of his
employment, until the earlier of (I) 24 months following termination of
employment, or (II) the date the Executive has commenced new employment and has
thereby become eligible for comparable benefits; provided that, with respect to
any of the coverages described above, if such coverage is provided through an
insurance policy with an insurance company unaffiliated with the initial
Company, and if under the terms of the applicable policy, it is not possible to
provide continued coverage (including if continued coverage under the policy
would increase the Company's cost allocable to the Executive by more than 100%),
then the Company shall pay the Executive a lump sum cash amount, no later than
sixty (60) days following termination of employment an amount equal to twice the
aggregate allocable cost of such coverage as applicable immediately prior to
termination of employment, such payment to be made without any discount for
present value.

      8. Certain Supplemental Payments by the Company.

            (a) In the event the Executive's employment is terminated pursuant
to Section 7(d) above, and if in connection therewith it is determined that (i)
part or all of the compensation and benefits to be paid to the Executive
constitute "parachute payments" under Section 280G of the Code, and (ii) the
payment thereof will cause the Executive to incur excise tax under Section 4999
of the Code, the Company, on or before the date for payment of such excise tax,
shall pay the Executive, in lump sum, an amount (the "Gross-Up Amount") such
that, after payment of all federal, state and local income tax and any
additional excise tax under Section 4999 of the Code in respect of the Gross-Up
Amount payment, the Executive will be fully reimbursed for the amount of such
excise tax.




                                       15
<PAGE>   16
            (b) The determination of the Parachute Amount, the Base Amount and
the Gross-Up Amount, as well as any other calculations necessary to implement
this Section 8 shall be made by a nationally recognized accounting or benefits
consulting firm selected by the Executive and reasonably satisfactory to the
Company and which has not performed services, other than minor indirect or
incidental services, for either the Company or the Executive for three years
prior to the date the Consultant is retained for this purpose. The Consultant's
fee shall be paid by the Company.

            (c) As promptly as practicable following such determination and the
elections hereunder, the Company shall pay to or distribute to or for the
benefit of the Executive such amounts as are then due to the Executive under
this Agreement and shall promptly pay to or distribute for the benefit of the
Executive in the future such amounts as become due to the Executive under this
Agreement.

            (d) As a result of the uncertainty in the application of Section
280G of the Code at the time of an initial determination hereunder, it is
possible that payments will not have been made by the Company which should have
been made under clause (a) of this Section 8 ("Underpayment"). In the event that
there is a final determination by the Internal Revenue Service, or a final
determination by a court of competent jurisdiction, that an Underpayment has
been made and the Executive thereafter is required to make any payment of an
excise tax, income tax, any interest or penalty, the accounting or benefits
consulting firm selected under clause (b) above shall determine the amount of
the Underpayment that has occurred and any such Underpayment shall be promptly
paid by the Company to or for the benefit of the Executive.




                                       16
<PAGE>   17
      9. Further Obligations of the Executive.

            (a) Confidentiality. During and following the Executive's employment
by the Company, the Executive shall hold in confidence and not directly or
indirectly disclose or use or copy or make lists of any confidential information
or proprietary data of the Company, except to the extent authorized in writing
by the Board of Directors of the Company or required by any court or
administrative agency, other than to an employee of the Company or a person to
whom disclosure is reasonably necessary or appropriate in connection with the
performance by the Executive of duties as an executive of the Company.
Confidential information shall not include any information known generally to
the public or any information of a type not otherwise considered confidential by
persons engaged in the same business or a business similar to that of the
Company. All records, files, documents and materials or copies thereof, relating
to the Company's business which the Executive shall prepare, or use, or come
into contact with, shall be and remain the sole property of the Company and
shall be promptly returned to the Company upon termination of employment with
the Company.

            (b) Non-Solicitation. For the period from the Effective Date until
the second anniversary of the termination of the Executive's employment, the
Executive will not, directly or indirectly, contact, approach or solicit for the
purpose of offering employment to or hiring (whether as an employee, consultant,
agent, independent contractor or otherwise) any officer of the Company, or its
affiliates, other than on the Company's behalf, without the prior written
consent of the Company.




                                       17
<PAGE>   18
      10.   Equitable Relief.

      Executive acknowledges and agrees that in the event of a breach by
Executive of any of the provisions of Section 9 hereof, the Company shall suffer
irreparable harm for which monetary damages alone will constitute an
insufficient remedy. Consequently, in the event of any such breach, the Company
may, in addition to other rights and remedies existing in its favor, apply to
any court of law or equity of competent jurisdiction for specific performance
and/or injunctive or other relief in order to enforce or prevent any violations
of the provisions hereof, in each case without the requirement of posting a bond
or proving actual damages.

      11.   Expenses and Interest.

      If, after a Change in Control of the Company, a good faith dispute arises
with respect to the enforcement of the Executive's rights under this Agreement,
or if any legal or arbitration proceeding shall be brought in good faith to
enforce or interpret any provision contained herein, or to recover damages for
breach hereof, the Executive shall recover from the Company any reasonable
attorney's fees and necessary costs and disbursements incurred as a result of
such dispute, and prejudgment interest on any money judgment or arbitration
award obtained by the Executive calculated at the rate of interest announced by
Peoples Heritage Bank, or the successor thereto, from time to time as its prime
rate from the date that payments to him should have been made under this
Agreement.

      12. Payment Obligations Absolute. The Company's obligation during and
after the Employment Period to pay the Executive the compensation and to make
the arrangements provided herein shall be absolute and unconditional and shall
not be affected by any circumstances, including, without limitation, any setoff,
counterclaim, recoupment, defense or other right which the Company may have
against him or anyone else. All amounts payable by the Company hereunder shall
be paid




                                       18
<PAGE>   19
without notice or demand. Each and every payment made hereunder by the Company
shall be final and the Company will not seek to recover all or any part of such
payment from the Executive or from whomsoever may be entitled thereto, for any
reason whatever except as provided in Section 8(d) above.

      13.   Successors.

            (a) If the Company sells, assigns, or transfers all or substantially
all of its business and assets to any Person, excluding Affiliates of the
Company, or if the Company merges into or consolidates or otherwise combines
with any Person which is a continuing or successor entity, then the Company
shall assign all of its rights, title and interest in this Agreement as of the
date of such event to the Person which is either the acquiring or successor
Company, and such Person shall assume in writing and perform from and after the
date of such written assignment all of the terms, conditions and provisions
imposed by this Agreement upon the Company. Failure of the Company to obtain
such written assignment shall be a breach of this Agreement. In case of such
assignment by the Company and of written assumption and agreement by such
Person, all further rights as well as all other obligations of the Company under
this Agreement thenceforth shall cease and terminate and thereafter the
expression "the Company" wherever used herein shall be deemed to mean such
Person or Persons.

            (b) This Agreement and all rights of the Executive shall inure to
the benefit of and be enforceable by the Executive's personal or legal
representatives, estates, executors, administrators, heirs and beneficiaries.
All amounts payable to the Executive hereunder shall be paid, in the event of
the Executive's death, to the Executive's estate, heirs and representatives.
This Agreement shall inure to the benefit of, be binding upon and be enforceable
by, any successor,




                                       19
<PAGE>   20
surviving or resulting Company or other entity to which all or substantially all
of the Company's business and assets shall be transferred. This Agreement shall
not be terminated by the voluntary or involuntary dissolution of the Company.

      14. Enforcement. The provisions of this Agreement shall be regarded as
divisible, and if any such provisions or any part hereof are declared invalid or
unenforceable by a court of competent jurisdiction, the validity and
enforceability of the remainder of such provisions or parts hereof and the
applicability thereof shall not be affected thereby.

      15. Amendment. This Agreement may not be amended or modified at any time
except by a written instrument executed by the Company and the Executive if such
amendment or modification occurs before any Change in Control, or by the
Executive and the Company after any Change in Control.

      16. Withholding. The Company shall be entitled to withhold from amounts to
be paid to the Executive hereunder any federal, state or local withholding or
other taxes, or charge which it is from time to time required to withhold. The
Company shall be entitled to rely on an opinion of counsel if any question as to
the amount or requirement of any such withholding shall arise.

      17.   Governing Law.

      This Agreement and the rights and obligations hereunder shall be governed
by and construed in accordance with the laws of the State of Maine.

18.   Arbitration.

      Any dispute arising out of this Agreement shall be determined by
arbitration in the State of Maine under the rules of the American Arbitration
Association then in effect and judgment upon any award pursuant to such
arbitration may be enforced in any court having jurisdiction thereof.




                                       20
<PAGE>   21
      19.   Notice.

      Notices given pursuant to this Agreement shall be in writing and shall be
deemed given when received and, if mailed, shall be mailed by United States
registered or certified mail, return receipt requested, addressee only postage
prepaid, to the Company at:

                  Banknorth Group, Inc.
                  P.O. Box 9540
                  One Portland Square
                  Portland, ME 04112
                  Attn:  Clerk

or if to the Executive, at the address included in the Company's records, or to
such other address as the party to be notified shall have given to the other.

      20.   No Waiver.

      No waiver by any party at any time of any breach by another party of, or
compliance with, any condition or provision of this Agreement to be performed by
another party shall be deemed a waiver of similar or dissimilar provisions or
conditions at any time.

      21.   Headings.  The headings herein contained are for reference only
and shall not affect the meaning or interpretation of any provision of this
Agreement.

      22.   Entire Agreement.

                  (a) This Agreement constitutes the entire agreement among the
parties hereto with respect to the subject matter hereof. Except as provided in
paragraph (b) below, all prior correspondence and proposals (including but not
limited to summaries of proposed terms) and all prior promises, representations,
understandings, arrangements and agreements relating to such subject matter
(including without limitation the Change-in-Control Agreement, as defined below)
are merged herein and superseded hereby; provided however, that the terms of the
SERP Agreement,




                                       21
<PAGE>   22
the Retirement Plan, the Thrift Plan, the Employee Stock Ownership Plan, and any
effective applicable employment agreement shall be incorporated herein and made
a part hereof to the extent not inconsistent with the terms hereof.

                  (b) The Company acknowledges that it has agreed, in an
Agreement and Plan of Merger, dated as of June 1, 1999 and amended as of
December 22, 1999, between Banknorth Group, Inc. ("Banknorth") and the Company
(the "Merger Agreement"), to honor the obligations of Banknorth under the
Change-in-Control Agreement, dated as of July 10, 1998 (the "Change- in-Control"
Agreement") between Banknorth and the Executive. Notwithstanding any provision
of this Agreement or the Change-in-Control Agreement to the contrary, (i) in the
event the Executive's employment with the Company is terminated for any reason
by the Executive or by the Company, other than by the Company for cause (as
defined in the Change-in-Control Agreement), by the Executive or by the Company
pursuant to Section 7(d) of this Agreement or as a result of the death of the
Executive, within one year of the Effective Date (the "One-Year Period"), the
Executive shall (x) be entitled to receive the payments and benefits set forth
in Section 3 of the Change-in-Control Agreement (subject to the limitation
contained in Section 3.3 thereof) and not this Agreement, which shall be
terminated, and (y) comply with the requirements of Section 5 of the
Change-in-Control Agreement, and (ii) in the event the Executive's employment
with the Company is terminated by the Executive or by the Company pursuant to
Section 7(d) of this Agreement during the One-Year Period, the Executive shall
be entitled to receive the payments and benefits set forth in this Agreement and
not the Change-in-Control Agreement, which shall be terminated, it being the
express intention of the Company and the Executive that in no event shall the
Executive be entitled to receive benefits under both this Agreement and the
Change-in-Control Agreement. After the One-Year Period, and provided that the
Executive has not become entitled to receive benefits under the





                                       22
<PAGE>   23
Change-in-Control Agreement, the Change-in-Control Agreement shall terminate and
be of no further force and effect without any action on the part of the Company
and the Executive, and the Executive thereafter shall be entitled to receive
such benefits as he may be entitled to receive in accordance with the terms of
this Agreement. The parties agree that this Agreement shall constitute a written
modification to the Change-in-Control Agreement.


      IN WITNESS WHEREOF, the parties have executed this Agreement as of the
date and year first written above.


                                BANKNORTH GROUP, INC.

                                By: /s/ Carol L. Mitchell
                                   ----------------------
                                Carol L. Mitchell

                                Executive Vice President, Secretary and
                                General Counsel

                                /s/ Richard E. Johnson
                                ----------------------
                                Richard E. Johnson





                                       23