<PAGE>

                                                                    Exhibit 10.6

                     CHANGE OF CONTROL SEVERANCE AGREEMENT

          This Change of Control Severance Agreement ("Agreement") dated April
19, 2000, is entered into by and among BSB Bancorp, Inc., a Delaware corporation
("Corporation"), BSB Bank & Trust Company, a New York chartered bank and trust
company ("Employer") and Matthew W. Schaefer ("Executive").

                                  WITNESSETH:

          Whereas, Executive is currently employed by Employer as its Senior
Vice President and Chief Information Officer;

          Whereas, Corporation and Employer desire to provide certain security
to Executive in connection with Executive's employment with Employer;

          Whereas, Executive, Corporation and Employer desire to enter into this
Agreement pertaining to the terms of the security Corporation and Employer are
providing to Executive with respect to his employment;

          Whereas, the Boards of Directors of Corporation and Employer (the
"Boards") have approved and authorized the execution and delivery of this
Agreement by and on behalf of Corporation and Employer;

          Now, therefore, in consideration of the mutual covenants and promises
contained herein, and other good and valuable consideration, the receipt of
which is hereby acknowledged, the parties agree to this Agreement as follows:

          1.   Term.  The initial term of this Agreement extends for a period of
               ----
two years from April 19, 2000, provided that the term of the Agreement shall be
extended automatically for one additional year on each annual anniversary date
of this Agreement, unless either the Boards or Executive provide(s) contrary
written notice to the other not less than 180 days in advance of any such
anniversary date.  In the event either the Boards or Executive provide such
written notice, then the term hereof shall not be extended, but the then current
term shall continue for the period remaining thereunder.  Notwithstanding the
foregoing, this Agreement shall automatically expire and terminate on
Executive's normal retirement date at age 65 or on Executive's early retirement
under the Special Service Retirement provision of Employer's pension plan if
Executive elects to take such early retirement.  The initial term of employment
and all such renewed terms of employment under this Agreement are collectively
referred to herein as the "term of this Agreement."

          2.   Benefits Upon Termination of Employment.
               ---------------------------------------

                (a) If, before a Change in Control, at the request or direction
of the acquiring party, or at any time during the 12-month period following a
Change in Control (1) the employment of Executive with Employer is terminated by
Employer for any reason other than Good Cause, or (2) Executive terminates his
employment with Employer for Good Reason, Employer shall, during the Severance
Period, continue to pay Executive an amount equal to Executive's Base Salary.

                Such amount will be paid during the Severance Period in monthly
or other installments, similar to those being received by Executive at the date
of the Change in Control, and will commence as soon as practicable following the
date of termination of employment. Executive shall receive any and all vested
benefits accrued under any Incentive Plans and Retirement Plans to the date of
termination of employment, the amount, form and time of payment of such benefits
to be determined by the terms of such Incentive Plans and Retirement Plans and
such benefits shall not be reduced by amounts payable under this Agreement.
<PAGE>

                (b) If upon the date of termination of Executive's employment,
Executive holds any options with respect to stock of Corporation, all such
options will immediately become exercisable upon such date and will be
exercisable for not less than 90 days thereafter. To the extent such
acceleration of vesting or exercisability of such options is not permissible
under the terms of any plan pursuant to which the options were granted, Employer
will pay to Executive, in a lump sum, within 90 days after termination of
employment, an amount equal to the excess, if any, of the aggregate fair market
value of all stock of Corporation subject to such options, determined on the
date of termination of employment, over the aggregate option price of such
stock, and Executive will surrender all such options unexercised. For the
purposes of this Agreement, in the event that such stock is listed on an
established national or regional stock exchange, is admitted to quotation on the
National Association of Securities Dealers Automated Quotation System, or is
publicly traded in an established securities market, in determining the fair
market value of the stock, Employer shall use the average of the closing prices
of such stock on such exchange or System or in such market (the highest such
closing price if there is more than one such exchange or market) on the five
trading dates immediately before the date of termination (or, if there is no
such closing price, then the Board shall use the mean between the highest bid
and lowest asked prices or between the high and low prices on such date), or, if
no sale of the Stock has been made on one or more of such dates, on the next
preceding day on which any such sale shall have been made.

                (c) During the Severance Period, Executive and his spouse will
continue to be covered by all Welfare Plans, maintained by Employer in which he
or his spouse were participating immediately prior to the date of his
termination as if he continued to be an employee of Employer; provided that, if
participation in any one or more of such Welfare Plans is not possible under the
terms thereof, Employer will provide substantially identical benefits. If,
however, Executive obtains employment with another employer during the Severance
Period, such coverage shall be provided only to the extent that the coverage
exceeds the coverage of any substantially similar plans provided by his new
employer.

                (d) Notwithstanding any other provision of this Agreement to the
contrary, Executive shall not be entitled to any payment or benefit under this
Agreement (including, without limitation, pursuant to the provisions of Section
2(b) hereof), that, in the reasonable opinion of the independent certified
public accountants of Corporation and Employer, would prevent any transaction
entered into or to be entered into by Corporation or Employer from qualifying as
a "pooling of interests" for accounting purposes.

          3.  No Setoff.   No payment or benefits payable to or with respect to
              ---------
Executive pursuant to this Agreement shall be reduced by any amount Executive or
his spouse may earn or receive from employment with another employer or from any
other source, except as expressly provided in subsection 2(c) of this Agreement.

          4.  Death.  If Executive dies during the Severance Period:
              -----

                (a) All amounts payable hereunder to Executive shall, during the
remainder of the Severance Period, be paid to his surviving spouse.

                (b) The spouse of Executive shall, during the remainder of the
Severance Period, be covered under all Welfare Plans made available by Employer
to Executive or his spouse immediately prior to the date of his death; provided
that, if participation in any one or more of such plans and arrangements is not
possible under the terms thereof, Employer will provide substantially identical
benefits.

                Any benefits payable under this Section 4 are in addition to any
other benefit due to Executive or his spouse or beneficiaries from Employer,
including, but not limited to, payments under any of the Incentive or Retirement
Plans.
<PAGE>

          5.  Termination for Good Cause or Without Good Reason. If the
              -------------------------------------------------
employment of Executive with Employer is terminated (a) for any reason prior to
a Change in Control other than at the request or direction of an acquiror in
connection with a Change in Control, or (b) after a Change in Control by
Employer for Good Cause, or by the voluntary action of Executive without Good
Reason, Executive's Base Salary (at the rate in effect on the date of
termination) shall be paid through the date of termination, and Employer shall
have no further obligation to Executive or his spouse under this Agreement,
except for benefits accrued under Incentive Plans pursuant to subsection 2(a)
above.

          6.  Section 280G Gross-Up Payment.
              -----------------------------

                (a) Anything in this Agreement to the contrary notwithstanding
and except as set forth below in this Section 6(a), in the event it shall be
determined that any payment or distribution by Employer, or any other member of
the affiliated group (as determined for purposes of Sections 280G and 4999 of
the Internal Revenue Code of 1986, as amended (the "Code") of which Employer or
Corporation is a member, to or for the benefit of Executive (whether paid or
payable or distributed or distributable pursuant to the terms of this Agreement
or otherwise, but determined without regard to any additional payments required
under this Section 6) (a "Payment") would be subject to the excise tax imposed
by Section 4999 of the Code or any interest or penalties are incurred by
Executive with respect to such excise tax (such excise tax, together with any
such interest and penalties, are hereinafter collectively referred to as the
"Excise Tax"), then Executive shall be entitled to receive an additional payment
(a "Gross-Up Payment") in an amount such that after payment by Executive of all
taxes (including any interest or penalties imposed with respect to such taxes),
including, without limitation, any income taxes (and any interest and penalties
imposed with respect thereto) and Excise Tax imposed upon the Gross-Up Payment,
Executive retains an amount of the Gross-Up Payment equal to the Excise Tax
imposed upon the Payments. Notwithstanding the foregoing provisions of this
Section 6(a), if it shall be determined that Executive is entitled to a Gross-Up
Payment, but that Executive, after taking into account the Payments and the
Gross-Up Payment, would not receive a net after-tax benefit of at least $10,000
(taking into account both income taxes and any Excise Tax) as compared to the
net after-tax proceeds to Executive resulting from an elimination of the Gross-
Up Payment and a reduction of the Payments, in the aggregate, to an amount (the
"Reduced Amount") such that the receipt of Payments would not give rise to any
Excise Tax, then no Gross-Up Payment shall be made to Executive and the
Payments, in the aggregate, shall be reduced to the Reduced Amount.

                (b) Subject to the provisions of Section 6(c), all
determinations required to be made under this Section 6, including whether and
when a Gross-Up Payment is required and the amount of such Gross-Up Payment and
the assumptions to be utilized in arriving at such determination, shall be made
by PriceWaterhouseCoopers LLP or such other certified public accounting firm
reasonably acceptable to Employer as may be designated in writing by Executive
(the "Accounting Firm") which shall provide detailed supporting calculations
both to Employer and the Executive within 15 business days of the receipt of
notice from Executive that there has been a Payment, or such earlier time as is
requested by Employer. In the event that the Accounting Firm is serving as
accountant or auditor for the individual, entity or group effecting the Change
of Control, Executive shall appoint another nationally recognized accounting
firm reasonably acceptable to Employer to make the determinations required
hereunder (which accounting firm shall then be referred to as the Accounting
Firm hereunder). All fees and expenses of the Accounting Firm shall be borne
solely by Employer. Any Gross-Up Payment, as determined pursuant to this Section
6, shall be paid by Employer to Executive within five business days of the
receipt of the Accounting Firm's determination. Any determination by the
Accounting Firm shall be binding upon Employer and Executive. As a result of the
uncertainty in the application of Section 4999 of the Code at the time of the
initial determination by the Accounting Firm hereunder, it is possible that the
initial Gross-Up Payments made by Employer will be inadequate and that
additional Gross-Up Payments by Employer should have been made ("Underpayment"),
consistent with the calculations required to be made hereunder. In the event
that Employer exhaust its remedies pursuant to Section 6(c) and Executive
thereafter are required to make a payment of any Excise Tax, the Accounting Firm
shall determine the amount of the Underpayment that has occurred and any such
Underpayment shall be promptly paid by Employer to or for the benefit of
Executive.
<PAGE>

                (c) Executive shall notify Employer in writing of any claim by
the Internal Revenue Service that, if successful, would require the payment by
Employer of a Gross-Up Payment. Such notification shall be given as soon as
practicable but no later than 10 business days after Executive receives written
notice of such claim and shall apprise Employer of the nature of such claim and
the date on which such claim is requested to be paid. Executive shall not pay
such claim prior to the expiration of 30 days following the date on which
Executive gives such notice to Employer (or such shorter period ending on the
date that any payment of taxes with respect to such claim is due). If Employer
notifies Executive in writing prior to the expiration of such period that they
desire to contest such claim, Executive shall:

                (i)    give Employer any information reasonably requested by
Employer relating to such claim,

                (ii)   take such action in connection with contesting such claim
as Employer shall reasonably request in writing from time to time, including,
without limitation, accepting legal representation with respect to such claim by
an attorney selected by Employer and reasonably acceptable to Executive,

                (iii)  cooperate with Employer in good faith in order
effectively to contest such claim, and

                (iv)   permit Employer to participate in any proceedings
relating to such claim; provided, however, that Employer shall bear and pay
directly all costs and expenses (including additional interest and penalties)
incurred in connection with such contest and shall indemnify and hold Executive
harmless, on an after-tax basis, for any Excise Tax or income tax (including
interest and penalties with respect thereto) imposed as a result of such
representation and payment of costs and expenses.

                Without limitation on the foregoing provisions of this Section
6(c), Employer shall control all proceedings taken in connection with such
contest and, at their sole option, may pursue or forgo any and all
administrative appeals, proceedings, hearings and conferences with the taxing
authority in respect of such claim and may, at their sole option, either direct
Executive to pay the tax claimed and sue for a refund or contest the claim in
any permissible manner, and Executive agrees to prosecute such contest to a
determination before any administrative tribunal, in a court of initial
jurisdiction and in one or more appellate courts, as Employer shall determine;
provided, however, that if Employer directs Executive to pay such claim and sue
for a refund, Employer shall advance the amount of such payment to Executive, on
an interest-free basis and shall indemnify and hold Executive harmless, on an
after-tax basis, from any Excise Tax or income tax (including interest or
penalties with respect thereto) imposed with respect to such advance or with
respect to any imputed income with respect to such advance; and further provided
that any extension of the statute of limitations relating to payment of taxes
for the taxable year of Executive with respect to which such contested amount is
claimed to be due is limited solely to such contested amount. Furthermore,
Employer's control of the contest shall be limited to issues with respect to
which a Gross-Up Payment would be payable hereunder and Executive shall be
entitled to settle or contest, as the case may be, any other issue raised by the
Internal Revenue Service or any other taxing authority.

                (d) If, after the receipt by Executive of an amount advanced by
Employer pursuant to Section 6(c), Executive becomes entitled to receive any
refund with respect to such claim, Executive shall (subject to Employer's
continued compliance with the requirements of this Section 6) promptly pay to
Employer the amount of such refund (together with any interest paid or credited
thereon after taxes applicable thereto). If, after the receipt by Executive of
an amount advanced by Employer pursuant to Section 6(c), a determination is made
that Executive shall not be entitled to any refund with respect to such claim
and Employer does not notify Executive in writing of their intent to contest
such denial of refund prior to the expiration of 30 days after such
determination (or if any such contest shall be finally determined in a manner
adverse to such refund being allowed), then such advance shall be forgiven and
shall not be required to be repaid and the amount of such advance shall offset,
to the extent thereof, the amount of the Gross-Up Payment required to be paid.

          7.  Definitions.  For purposes of this Agreement:
              -----------
<PAGE>

          (a) "Base Salary" shall mean the higher of Executive's annual base
salary at the rate in effect on the date of a Change in Control or the rate in
effect on the date of termination of employment.

          (b) "Change in Control" shall be deemed to have occurred if there has
been a change of control of a nature that would be required to be reported in
response to Item 6(e) of Schedule 14A of Regulation 14A promulgated under the
Securities Exchange Act of 1934, as amended ("Exchange Act"), or such item
thereof which may hereafter pertain to the same subject; provided that, and
notwithstanding the foregoing, a Change in Control shall be deemed to have
occurred if (i) any "person" (as that term is used in Sections 13(d) and
14(d)(2) of the Exchange Act) is or becomes the beneficial owner, directly or
indirectly, of securities of Corporation or Employer representing 25% or more of
the combined voting power of Corporation's then outstanding securities, (ii)
during any period of two consecutive years, individuals who at the beginning of
such period constituted the Boards of Directors of Corporation and Employer
cease for any reason to constitute at least a majority thereof unless the
election of each Director, who was not a Director at the beginning of the
period, was approved by a vote of at least two-thirds of the Directors then
still in office who were Directors at the beginning of the period, (iii)
Corporation shall cease to be a publicly owned corporation, or (iv) any merger
or consolidation of Corporation with or into another entity shall occur as a
result of which the stockholders of Corporation do not retain or acquire 75% or
more of the capital stock of the resulting entity.

          (c) "Good Cause" shall be deemed to exist if, and only if: (1)
Executive engages in acts or omissions constituting dishonesty, intentional
breach of fiduciary obligation or intentional wrongdoing or malfeasance; (2)
Executive is convicted of a criminal violation involving fraud or dishonesty; or
(3) Executive materially breaches the Agreement (other than by engaging in acts
or omissions enumerated in paragraphs (1) and (2) above), or materially fails to
satisfy the conditions and requirements of his employment with Employer, and
such breach or failure by its nature is incapable of being cured, or such breach
or failure remains uncured for more than 30 days following receipt by Executive
of written notice from Employer specifying the nature of the breach of this
paragraph (3), inattention by Executive to his duties shall be deemed a breach
or failure capable of cure.

     Without limiting the generality of the foregoing, the following shall not
constitute Good Cause: any personal or policy disagreement between Executive and
Employer or any member of the Board of Directors of Employer; or any action
taken by Executive in connection with his duties if Executive acted in good
faith and in a manner he reasonably believed to be in, and not opposed to, the
best interest of Employer and had no reasonable cause to believe his conduct was
unlawful.

     Notwithstanding anything herein to the contrary, in the event Employer
shall terminate the employment of Executive for Good Cause hereunder, Employer
shall give at least 30 days prior written notice to Executive specifying in
detail the reason or reasons for Executive's termination.

          (d)  "Good Reason" shall exist if:

                (1) there is a significant change in the nature or the scope of
Executive's authority;

                (2) there is a reduction in Executive's base salary from the
rate in effect during the fiscal year before the Change in Control;

                (3) Employer changes the principal location in which Executive
is required to perform services to one which is more than 50 miles from his
current location;

                (4) there is a reasonable determination by Executive that, as a
result of a change in circumstances significantly affecting his position, he is
unable to exercise the authority, powers, function or duties attached to his
position;

          (e) "Incentive Plans" shall mean any incentive, bonus deferred
compensation or similar plan or arrangement currently or hereafter made
available by Corporation or Employer in which Executive is eligible to
participate.
<PAGE>

                (f) "Retirement Plans" shall mean any qualified or supplemental
defined benefit retirement plan or defined contribution retirement plan,
currently or hereinafter made available by Corporation or Employer in which
Executive is eligible to participate.

                (g) "Severance Period" shall mean the period beginning on the
date Executive's employment with Employer terminates under circumstances
described in Section 2 above and ending on the first to occur of: (1) the date
24 months thereafter, or (2) the date Executive attains or would have attained
age 65.

                (h) "Welfare Plan" shall mean any health and dental plan,
disability plan, survivor income plan and life insurance plan or arrangement
currently or hereafter made available by Employer in which Executive is eligible
to participate.

          8.  Benefits Unfunded.  All rights of Executive and his spouse or
              -----------------
other beneficiary under this Agreement shall at all times be entirely unfunded
and no provision shall at any time be made with respect to segregating any
assets of Corporation or Employer for payment of any amounts due hereunder.
Neither Executive nor his spouse or other beneficiary shall have any interest in
or rights against any specific assets of Corporation or Employer, and Executive
and his spouse or other beneficiary shall have only the rights of a general
unsecured creditor of Employer.

          9.  Litigation Expense.  Employer shall pay Executive's out of-pocket
              ------------------
expenses, including attorney's fees, in connection with any judicial proceeding
to enforce this Agreement or to construe or determine the validity of this
Agreement or otherwise in connection herewith if Executive is successful in such
litigation.

          10.  Applicable Law.  This Agreement shall be construed and
               --------------
interpreted pursuant to the laws of the State of New York.

          11.  Entire Agreement.  This Agreement contains the entire Agreement
               ----------------
between Employer and Executive and supersedes any and all previous agreements,
written or oral, among the parties relating to his or her employment by
Employer.  No amendment or modification of the terms of this Agreement shall be
binding upon the parties hereto unless reduced to writing and signed by Employer
and Executive.  This Agreement is the exclusive agreement between Corporation,
Employer and Executive regarding payments to Executive in the event of a change
in control of Corporation or Employer.  During the term of this Agreement,
Executive shall not participate in or benefit from any other change of control
severance plan or policy which may be adopted by Corporation or Employer,
provided that thereafter Executive shall participate in such plan or policy if
one has been established by Corporation or Employer.

          12.  No Employment Contract.  Nothing contained in this Agreement
               ----------------------
shall be construed to be an employment contract between Executive and Employer.
Executive is employed at will and Employer may terminate his employment at any
time, with or without cause.

          13.  Counterparts.  This Agreement may be executed in counterparts,
               ------------
each of which shall be deemed an original.

          14.  Severability.  In the event any provision of this Agreement is
               ------------
held illegal or invalid, the remaining provisions of this Agreement shall not be
affected thereby.

          15.  Successors.  This Agreement shall be binding upon and inure to
               ----------
the benefit of the parties hereto and their respective heirs, representatives
and successors.

          16.  Notice.  Notices required under this Agreement shall be in
               ------
               writing and sent by registered mail, return receipt requested, to
               the following addresses or to such other address as the party
               being notified may have previously furnished to the others by
               written notice.
<PAGE>

               If to Employer or Corporation:

               Attention:  Thomas L. Thorn

        1.     BSB Bancorp, Inc.

        2.     BSB Bank & Trust Company
               58-68 Exchange Street
               Binghamton, New York  13902

               If to Executive:

               Matthew W. Schaefer
               329 Doris Avenue
               Vestal, New York  13850
<PAGE>

          In Witness Whereof, Executive has hereunto set his hand, and
Corporation and Employer have caused this agreement to be executed in their
names and on their behalf, all as of the day and year first above written.



                                      BSB BANCORP, INC.

ATTEST:    /s/ Larry G. Denniston     By:  /s/ Thomas L. Thorn
        -------------------------         ----------------------------
          Larry G. Denniston              Thomas L. Thorn
          (Secretary)                     Acting President and Chief
                                          Executive Officer


                                      BSB BANK & TRUST COMPANY

ATTEST:  /s/ Larry G. Denniston       By:  /s/ Thomas L. Thorn
        -------------------------         ----------------------------
         Larry G. Denniston                   Thomas L. Thorn
         Secretary                            Acting President and Chief
                                              Executive Officer

                                           /s/ Matthew W. Schaefer
                                          ----------------------------
                                               Matthew W. Schaefer