<PAGE>
 
                                                                    EXHIBIT 10.6


                    [LETTTERHEAD OF BAY VIEW FEDERAL BANK]





                      OUTSIDE DIRECTORS' RETIREMENT PLAN
                      ----------------------------------








                                                                   February 1992

<PAGE>
 
Article I - Name, Purposes, and Effective Date
----------------------------------------------

     1.1  The Plan set forth in this document is designated as the BAY VIEW 
FEDERAL BANK OUTSIDE DIRECTORS' RETIREMENT PLAN.

     1.2  The Plan was created and will be maintained to provide retirement 
income and pre-retirement death benefits to outside Directors serving on the 
Board.

     1.3  The Plan is a non-qualified defined benefit retirement plan, not 
subject to the Employee Retirement Income Society Act of 1974 (ERISA).

     1.4  The effective date of the Plan is February 27, 1987.  The fiscal year
of the Plan is January 1 through December 31.

<PAGE>
 
Article II - Definitions
------------------------

     2.1  "Advisory Board" means the body established by the Board as a vehicle 
for recruiting, preparing and screening potential new Directors.  Such Advisory 
Board ceased to exist in 1983.

     2.2  "Bank" means Bay View Federal Bank, A Federal Savings Bank and its 
successor, if any.

     2.3  "Board" means the Board of Directors of the Bank.

     2.4  "Board Committee" means a formal committee of the Board.

     2.5  "Director" means an outside, non-employee Director serving the Board.

     2.6  "Effective Date" of the Plan means February 27, 1987.

     2.7  "Normal Form of Payment" means monthly payments for the Payment 
Period.

     2.8  "Pay" means the Retainer in effect at the time the Director becomes a 
Retiree.

     2.9  "Payment Periods" means the period during which the Retiree is 
entitled to receive benefits under the Plan.  If the Retiree should die within 
the Payment Period, payments will continue to the named beneficiary(ies) or to 
the Retiree's estate.  However, such payments shall not continue beyond the 
original Payment Period.

                                       2

<PAGE>

     2.10  "Payment Period Limit" means the maximum number of years in which the
Director is entitled to receive benefits, based on the Director's completed
Years of Service.

     2.11  "Plan" means the Outside Directors' Retirement Plan as described in 
this document.

     2.12  "Retainer" means the annual base amount paid to Directors as a fixed 
amount, exclusive of meeting fees, consulting fees, provision for Directors' 
benefits, and any special director compensation.  The annual base amount shall 
be the amount that is in effect at the time the Director becomes a Retiree.  The
Retainer shall also include the annual fixed amount payable to the Director for 
his/her position as Chairperson of the Board or a Board committee (if 
applicable).  The entitlement to a chair fee shall be accorded any director who 
serves at least three years as the chair of one or more committees of the Board 
or the Board itself, the chair fee sat at the highest level earned by the 
director while acting in such capacity.

     2.13  "Retiree" means a former Director entitles to benefits under the
Plan.

     2.14  "Vesting Schedule" means the applicable percentage of the full 
retirement benefit based on years of service.

     2.15  "Year of Service" means twelve consecutive months of service as a 
Director.  It also includes service on the Advisory Board.

                                       3
 
<PAGE>
 
Article III - Eligibility
-------------------------



     3.1  All non-employee Directors are eligible to participate as of the 
Effective Date.  Directors appointed to the Board following the Effective Date 
of the Plan will be eligible commencing with the effective date of their 
appointment.



Article IV - Retirement Benefit and Vesting Schedule
----------------------------------------------------



     4.1  Directors will be entitled to benefit payments under the Plan 
following completion of three (3) Years of Service, and upon the later of:

          .  attainment of age 65, or

          .  termination of service as a director

The Board in its sole discretion may elect to commence benefit payments on an 
earlier date depending on the facts and circumstances at the time of resignation
or early retirement, but in no event prior to attainment of age 50.

     4.2  The annual Retirement Benefit payable to a Retiree will be equal to 
the Retainer in effect at the time of retirement, and will be subject to the 
Vesting Schedule Applicable Percentage, specified in the following table, 
multiplied by the Retainer.


                                       4

<PAGE>
 
<TABLE> 
<CAPTION> 

          Completed                              Vesting Schedule
          ---------                              ----------------
        Years of Service                       Applicable Percentage
        ----------------                       ---------------------
        <S>                                    <C> 
        Under three (3)                                  0%
     
        Three (3) but less than Five (5)                50%

        Five (5) but less than Ten (10)                100%

        Ten (10) and Over                              100%
     
</TABLE> 



     4.3  The annual Retirement Benefit will be paid to the Retiree or the 
designated beneficiary(ies) for the applicable Payment Period, according to the 
following table:



<TABLE> 
<CAPTION> 

       Completed
       ---------
     Years of Service                       Payment Period Limit
     ----------------                       --------------------
     <S>                                    <C> 
     Under three (3)                                0 Years
     
     Three (3) but less than Five (5)               5

     Five (5) but less than Ten (10)               10

     Ten (10) and Over                             15 if on Board at 
                                                      age 65 or over.
                                                      (10 years if not 
                                                      age 65)

</TABLE> 

                                       5

<PAGE>
 
Article V - Form of Payment
---------------------------

    5.1 The annual Retirement Benefit will be paid in monthly payments which is
the Normal Form of Payment. However, the Board may, at its sole discretion, pay
the actuarial equivalent of the Retirement Benefit in the form of a lump sum
payment. Such decision shall not be at the election of the Retiree, but rather
by the Board, based on the facts and circumstances that exist at the time of the
Director's retirement or termination from the Board. If the decision to effect a
lump sum payment is made, the Board shall also determine the timing of such
payment.


    5.2 For purposes of calculating the lump sum, the interest rate
assumption used in determining the present value equivalent of the Retirement
Benefit shall be the average of the Eleventh District Cost of Funds Index for
the most current calendar quarter.

                                       6

<PAGE>
 
Article VI - Beneficiary Designation
------------------------------------
    6.1 Each Director shall designate one or more beneficiaries to receive 
benefits under the Plan in the event of the director's death.  Such a 
designation may be changed at any time.  Absent a valid designation, or if no 
designated beneficiary survives the Director, benefits under the Plan shall be 
distributed in the following order of priority:  spouse, issue (e.g., children) 
by right of representation, parents, or the estate of the Director.
    
    6.2 Whenever the rights of Directors are stated or limited herein, such 
provisions shall also apply to their beneficiaries.



Article VII - Death/Disability Benefits
---------------------------------------

    7.1 The designated beneficiary (ies) of a Director who dies while on the 
Board, or of a former Director who dies prior to becoming a Retiree under the 
plan, will be entitled to monthly payments equal to one-twelfth of the annual 
Retirement Benefit that the Director would have received had he/she retired on 
the date of death.  Such payments shall commence as soon as practical following 
the death of the Director, and will be paid for the full Payment Period.

    7.2 If a Director should die while a Retiree, the designated 
beneficiary(ies) will be entitled to receive monthly payments for the remaining 
Period of the original Payment Period.

    7.3 There is no payment provision for disability benefits if the Director 
should become totally and permanently disabled.  In that event, the Director 
would be obliged to retire or terminate his/her service.

                                       7

<PAGE>
 
Article VIII - Source of Benefits and Funding
---------------------------------------------

     8.1 The benefits earned and payable under the Plan are a direct obligation 
of the Bank and will be reflected as a financial statement accrued liability.  
This liability will be determined in accordance with accepted actuarial 
assumptions.  No specific assets will be set aside for the designated purpose of
satisfying the Bank's obligation under the terms of the Plan.

Article IX - Effect of Merger/Change in Control
-----------------------------------------------

     9.1 If, at any time prior to retirement of a Director, there is a merger, 
consolidation or combination of the Bank with or into another financial 
institution (other than a merger, consolidation, or combination in which the 
bank is the continuing financial institution and which does not result in the 
outstanding shares of Common Stock being converted into or exchanged for 
different securities, cash or other property, or any combination thereof), or if
there is a change in control of the Bank, or of any company holding more than 50
percent of the outstanding Common Stock of the bank, the Director's vesting 
schedule shall be accelerated to the next highest benefit level, and the annual 
Retirement Benefit will be equal to the amount the Director would have received 
had he/she retired on the effective date of the merger or change in control, and
the Payment Period shall commence as of the date of the merger or change in 
control, notwithstanding the provisions of

                                       8



<PAGE>
 
Section 4.1.   Notwithstanding the foregoing, in no event shall the annual
Retirement Benefit to be paid to a Director under this Section 9.1 exceed two
hundred ninety-nine percent (299%) of the Director's "base" amount of
compensation, as defined in Section 2806(b)(3) of the Internal Revenue Code of
1986, as amended.

     9.2 The term "control" shall refer to the acquisition of 25 percent or more
of the voting securities of the Bank or Holding Company by any person, or 
persons acting as a group within the meaning of Section 13(d) of the Securities 
Exchange Act of 1934, or to such acquisition of a percentage between 10 percent 
and 25 percent if the Board of Directors of the Bank, the Holding Company or the
Office of Thrift Supervision has made a determination that such acquisition 
constitutes or will constitute control of the Bank or the Holding Company. The 
term "person" refers to an individual, corporation, company or other entity.

     9.3 Under the conditions of this Article, the following payment options 
will be available to the Director:
     
     a)  Normal Form of Payment--monthly payments of the annual
         Retirement Benefit due.
     
     b)  Lump Sum Payment as described under Article 5.1.
     
     c)  Deferral of Retirement Benefit until age 65.

                                       9




<PAGE>
 
Article X - Directors' Rights
-----------------------------

     10.1  The Directors of the Bank shall have the right to receive benefits 
earned and payable under the terms of the Plan.  However, the Directors have no 
rights beyond those of a general creditor to receive payment in respect to the 
Bank's obligation as outlined herein.

The Article XI - Amendment and Termination
------------------------------------------

     11.1  The Board reserves the right to amend or terminate the Plan at any 
time it is considered desirable or necessary, by a majority vote of the 
Directors.  The termination, amendment, or modification of the Plan shall in no 
way reduce or eliminate any participant's retirement benefit that vested prior 
to such termination, amendment, or modification.
IN WITNESS WHEREOF, the parties have executed this Plan Document on the 
22nd day of September, 1987.
----        ---------    --

                      BAY VIEW FEDERAL BANK, a Federal Savings Bank

                      By:  /s/ Robert E. Barnes
                         ----------------------
                         Robert E. Barnes
                         President and Chief Executive officer

                      By:  /s/ Bruno A. Malucchi
                         -----------------------
                         Bruno A. Malucchi
                         Chairman of the Board

                                      10
                         
<PAGE>
 
                       OUTSIDE DIRECTORS' RETIREMENT PLAN
                       ---------------------------------

                             ADDITION INFORMATION
                             --------------------

<TABLE> 

<S>                                      <C> 
Employer Sponsoring the Plan             Bay View Federal Bank, 
                                         A Federal Savings Bank
                                         2121 South El Camino Real
                                         San Mateo, California 94403

Employer Identification Number           94-0310961

Plan Number                              007

Plan Administrator                       Compensation/Benefits Committee
                                          Board of Directors

                                         Bay View Federal Bank,
                                         A Federal Savings Bank
                                         2121 South El Camino Real
                                         San Mateo, California 94403

                                         (415) 573-7300

Type of Plan                             This is a Non-qualified defined
                                         benefit Retirement Plan not
                                         subject to the Employee Retirement
                                         Income Security Act (ERISA).

                                         The benefits are not insured by the 
                                         Pension Benefit Guaranty Corporation,
                                         The Employer, or by any other entity.
</TABLE> 

                                      11
<PAGE>
 
                    AMENDMENT TO THE BAY VIEW FEDERAL BANK
                      OUTSIDE DIRECTORS' RETIREMENT PLAN


                                   ARTICLE I
                          PURPOSES AND EFFECTIVE DATE

     1.1  Purposes.  The purposes of this Amendment to the Bay View Federal Bank
          --------                                                              
Outside Directors' Retirement Plan (the "Amendment") are (a) to eliminate any
future economic obligation of Bay View Federal Bank (the "Bank") and Bay View
Capital Corporation (the "Company") under the Bank's Outside Directors
Retirement Plan (the "Plan") other than obligations accruing through the
Effective Date (as hereinafter defined), (b) to freeze the vested accrued
benefits of Participants in the Plan as of the Effective Date, (c) to provide
for distribution to current Retirees and other former outside directors of the
Bank of their vested accrued benefits under the Plan as of the Effective Date in
accordance with the existing terms and provisions of the Plan , and (d) to
provide for the transfer by the Bank to the Grantor Trust on the Effective Date
of cash in an amount equal to the present value of the vested accrued benefits
of the current outside directors of the Bank under the Plan as of the Effective
Date with (i) the rights of such persons and their beneficiaries to benefits
under the Plan being determined under the Grantor Trust and (ii) the
entitlements of such persons being limited to their respective portion of the
funds transferred by the Bank to the Grantor Trust and the earnings, losses and
value adjustments resulting from the investment or revinvestment of such funds
from time to time by the Grantor Trust Trustee subject to the rights of
creditors of the Bank as provided in the Grantor Trust.

     1.2  Effective Date.  The Effective Date of this Amendment (the "Effective
          --------------                                                       
Date") is the date this Amendment is approved by the shareholders of the
Company.  This Amendment has been approved by the Boards of Directors of the
Company and the Bank by action taken by them on March 28, 1996, subject to
approval of this Amendment by the shareholders of the Company. If this Amendment
is not passed by the affirmative vote of a majority of the issued and
outstanding common stock of the Company present, or represented, and entitled to
vote on this Amendment at a meeting of shareholders of the Company at which a
quorum is present, then this Amendment shall have no force or effect and the
Plan will be operated and administered in its existing form without respect or
regard to this Amendment.

                                   ARTICLE II
                             CERTAIN DEFINED TERMS

     The following terms shall be added to the definitions contained in the
Plan:

     2.1  "Change in Control" means (a) a merger, consolidation or combination
of the Company or the Bank with or into another entity (other than a transaction
in which the Company and the Bank are continuing entities with the shares of the
common stock of the Company outstanding immediately prior to the transaction
representing at least 51% of the outstanding common stock of the Company
immediately after the completion of the transaction) unless a majority of the
members of the Board of Directors of the Company immediately prior to such
transaction constitute a majority 
<PAGE>
 
of the Board of Directors of the surviving entity and resulting bank immediately
after the transaction or (b) a change in the individuals who constitute a
majority of the Board of Directors of the Company, by resignation or non-
election of persons nominated by the Board of Directors, occurring after any
person, or group of persons acting in concert, acquires more than 10% of the
outstanding shares of the common stock of the Company.

     2.2  "Grantor Trust" means that certain Trust Agreement for the Bay View
Federal Bank Outside Directors' Retirement Plan to be entered into between the
Bank and the Grantor Trust Trustee in substantially the form attached hereto as
Exhibit I and incorporated herein by reference.

     2.3  "Grantor Trustee" means First Bankers Trust Co., N.A., or such other
independent third party trustee selected by the Bank, its successors in
interest.

     2.4  "Participant" means each person who has a vested accrued benefit under
the Plan as of the Effective Date.

     2.5  "Successor Corporation" means any entity that is the successor to all
or substantially all of the assets of the Company or the Bank; provided however,
if such entity is an affiliate of an ultimate parent entity, then the ultimate
parent entity shall, for all purposes hereof, be the Successor Corporation.

     2.6  "Termination of Service" means termination, for any reason, of the
service of a Participant (but only those persons who are current outside
directors of the Bank as of the Effective Date) as a director of the Bank.
Provided however , if such Participant is also a director of the Company, then a
Termination of Service shall not occur until such Participant is no longer a
director of both the Bank and the Company (including any Successor Corporation).


                                  ARTICLE III
                    FREEZE OF AND ENTITLEMENTS TO BENEFITS

     3.1  Retirees.  A Participant who is either a Retiree or is not a current
          --------                                                            
outside director of the Bank, in either case as of the Effective Date, shall
receive his vested accrued benefit in accordance with the existing provisions of
the Plan without regard to this Amendment.

     3.2  Current Outside Directors.  The present value of the vested accrued
          -------------------------                                          
benefit of each current outside director of the Bank as of the Effective Date
will be fixed at the Effective Date and a corresponding cash amount will be
tendered by the Bank to the Grantor Trustee on the Effective Date for the
benefit of such person (or his or her beneficiaries) and such funds will be
invested and reinvested by the Grantor Trust Trustee for the benefit of such
person (or his or her beneficiaries) subject to the rights of creditors of the
Bank as set forth in the Grantor Trust. Upon a Termination of Service,
distribution of benefits will be made to a Participant who is a current outside
director of the Bank as of the Effective Date in ten (10) installments, with the
first installment being made 

                                       2
<PAGE>
 
within thirty (30) days after Termination of Service and each subsequent
distribution being made annually thereafter, in each case subject to the rights
of creditors of the Bank prior to the making of a distribution, all as more
particularly set forth in the Grantor Trust. Immediately following a Change in
Control or as soon as practicable thereafter, the Grantor Trust Trustee shall
liquidate any investments of the Grantor Trust in common stock of the Company or
securities of any Successor Corporation and reinvest the proceeds therefrom in
permitted investments as described in the Grantor Trust.


                                  ARTICLE IV
                              GENERAL PROVISIONS

     4.1  Revocation of Grantor Trust.  Prior to the occurrence of a Change in
          ---------------------------                                         
Control, the Board of Directors of the Bank may revoke the Grantor Trust and
require the Grantor Trust Trustee to return to the Bank all assets held by the
Grantor Trust.  If such right is exercised, the Bank shall be bound by the terms
and provisions of the Grantor Trust that relate to investment and reinvestment
of funds and the rights and entitlements of Participants therein and thereunder
as if such terms and provisions were a part of the Plan as amended by this
Amendment.

     4.2  Lump-Sum Cash Distribution.  The Board of Directors of the Bank may,
          --------------------------                                          
by affirmative action taken by it prior to a Change in Control, impose a single
lump-sum cash distribution of benefits or remaining benefits (previously
undistributed amounts) immediately following a Change in Control or as soon as
practicable thereafter to those Participants who (a) are current outside
directors of the Bank as of the Effective Date and (b) experience a Termination
of Service prior to or in connection with a Change in Control.  The Grantor
Trust Trustee shall be bound by any such action taken by the Board of Directors
of the Bank. Notwithstanding the foregoing, the Board of Directors of the Bank
shall not impose a single lump-sum distribution to a Participant if, as a result
thereof, the Bank or a Successor Corporation would be subject to a loss or
restriction on tax deductibility by reason of Section 280G of the Internal
Revenue Code of 1986, as amended (the "Code"), or the Participant would be
subject to a penalty tax under Section 4999 of the Code.

     4.3  No Right to Continue as a Director.  Nothing contained in this
          ----------------------------------                            
Amendment shall be deemed to confer upon any Participant any right to continue
to serve as a director of the Bank or the Company.

     4.4  Limitation on Rights Conferred.  No Participant shall have rights
          ------------------------------                                   
beyond those of a general creditor of the Bank to receive payment of benefits
under the Plan or the Grantor Trust.

     4.5  Limitation on Changes.  No changes, amendments or modifications may be
          ---------------------                                                 
made to the Plan after the occurrence of a Change in Control.

                                       3
<PAGE>
 
     4.6  Specific Deletion of Plan Provisions.  Articles VII and  IX of the
          ------------------------------------                              
existing Plan are hereby deleted.

     4.7  Inconsistencies.  The provisions of this Amendment are intended to
          ---------------                                                   
supersede and replace any inconsistent or contrary provisions contained in the
Plan as written prior to this Amendment.  In all cases of conflict between the
provisions of the Plan as in existence prior to this Amendment and the terms and
provisions of this Amendment, the terms and provisions of this Amendment shall
be controlling.

     4.8  Governing Law.  The validity, construction and effect of this
          -------------                                                
Amendment and all rights hereunder will be determined in accordance with the
laws of the State of California.

     4.9  Binding Effect.  This Amendment shall be binding upon the Company, the
          --------------                                                        
Bank and each Successor Corporation.  This Amendment shall inure to the benefit
of the Participants and their respective beneficiaries.

     4.10  Headings.  The headings of section and subsections herein are
           --------                                                     
included solely for convenience of reference and shall not affect the meaning of
any of the provisions of this Amendment or the Plan.

     This Amendment has been executed by the President and Secretary of the
Company and the Bank as of this 28th day of March, 1996, as the act and deed of
such corporate entities.

                                     BAY VIEW CAPITAL CORPORATION

ATTEST:
------ 

                                     By     /s/  Edward H. Sondker
                                       ----------------------------------
                                                President
/s/  Robert J. Flax
---------------------------------
Secretary

                                     BAY VIEW FEDERAL BANK,
                                     A FEDERAL SAVINGS BANK


ATTEST:
------ 
                                     By     /s/  Edward H. Sondker
                                       ----------------------------------
                                                President
/s/  Robert J. Flax
---------------------------------
Secretary

                                       4
<PAGE>
 

                         SUPPLEMENTAL AMENDMENT TO THE

            BAYVIEW FEDERAL BANK OUTSIDE DIRECTORS' RETIREMENT PLAN


     WHEREAS, the Bay View Federal Bank Outside Directors' Retirement Plan was
amended by a written instrument dated March 28, 1996 (the "Amendment"), and

     WHEREAS, the purpose of this document ("Supplemental Amendment") is to
modify the Amendment.

     NOW THEREFORE, in consideration of the premises and for other good and
valuable consideration, the receipt and sufficiency of which is hereby
acknowledged, it is agreed as follows:

     1.   The second sentence of Section 4.1 of the Amendment is hereby deleted
and in lieu thereof the following is inserted:

     If such right is exercised, the Board of Directors of the Bank shall be
     entitled to exercise full investment and reinvestment discretion of the
     funds and full administrative discretion relating thereto, including but
     not limited to, the rights and entitlements of Participants therein and
     thereto.  However, the Bank will be subject to prudent investment standards
     and the distribution of the Normal Form of Benefit, if applicable, may be
     made in common stock of the Company, cash, other assets, or a combination
     of the foregoing, in the same installments that common stock of the Company
     would have been distributed pursuant to the Grantor Trust.  In no event
     will the Bank alter the Post Change in Control Form of Payment set forth in
     the Grantor Trust, except as provided in Section 4.2 of the Amendment.

     2.   All capitalized terms herein shall have the meaning ascribed to them
in the Amendment and/or Grantor Trust.

     3.   The Amendment shall be and hereby is affirmed in its entirety, except
for the changes effectuated by the Supplemental Amendment.
<PAGE>
 
     The Supplemental Amendment has been executed by the President and Secretary
of Bay View Capital Corporation and Bay View Federal Bank, A Federal Savings
Bank as of this 30th day of December, 1996, as the act and deed of such
corporate entities.

                              BAY VIEW CAPITAL CORPORATION

ATTEST:
------ 

                              By   /s/ Edward H. Sondker
                                 --------------------------------
                                    Edward H. Sondker, President

/s/ Robert J. Flax
--------------------------
Robert J. Flax, Secretary



                              BAY VIEW FEDERAL BANK,
                              A FEDERAL SAVINGS BANK
ATTEST:
------ 


                              By   /s/ Edward H. Sondker
                                 --------------------------------
                                    Edward H. Sondker, President

/s/ Robert J. Flax
-------------------------
Robert J. Flax, Secretary
<PAGE>
 
                                TRUST AGREEMENT
                                    FOR THE
                             BAY VIEW FEDERAL BANK
                       OUTSIDE DIRECTORS' RETIREMENT PLAN


     THIS AGREEMENT is entered into as of this 23rd day of May, 1996, by and
between Bay View Federal Bank, A Federal Savings Bank (hereinafter called the
"Bank"), whose address is 2121 South El Camino Real, San Mateo, California
94403, and First Bankers Trust Co., N.A. (hereinafter called the "Trustee"),
whose address is 1201 Broadway, Quincy, Illinois 62301.

                                  WITNESSETH:

     WHEREAS, effective as of February 27, 1987, the Bank adopted the Bay View
Federal Bank Outside Directors' Retirement Plan as revised on April 26, 1990 and
February 27, 1992 (hereinafter called the "Plan") for the benefit of non-
employee directors of the Bank; and

     WHEREAS, the Plan has been amended by action of the Boards of Directors of
the Bank and Bay View Capital Corporation (hereinafter called the "Company"),
the sole stockholder of the Bank, which amendment (the "Amendment") is attached
hereto as Exhibit I and incorporated herein by reference and has been approved
by the shareholders of the Company on the date hereof; and

     WHEREAS, consistent with the Amendment, the Bank does hereby establish a
trust (hereinafter called the "Trust") and shall as hereinafter provided
contribute cash assets to the Trust which shall be held, invested and reinvested
by the Trustee as provided herein, subject to the claims of the Bank's creditors
in the event of Insolvency (as hereinafter defined), until paid to the current
outside directors of the Bank (individually, a "Participant" and collectively,
the "Participants") or their beneficiaries in such manner and at such times as
specified herein; and
<PAGE>
 
     WHEREAS, pursuant to the Amendment, the Bank shall, on the date hereof,
contribute to the Trust cash in an amount equal to the actuarially-determined
present value of the vested accrued benefits of the Participants and such cash
contribution shall, as of the date hereof, satisfy the liabilities of the Bank
under the Plan to such Participants; and

     WHEREAS, it is the intention of the parties that the Trust shall constitute
a part of the Plan and shall not affect the status of the Plan as an unfunded,
non-qualified pension plan, not subject to the Employee Retirement Income
Security Act of 1974, as amended, which Plan has been maintained and is being
maintained for the purpose of providing retirement income benefits to outside
directors of the Bank or their beneficiaries; and

     NOW, THEREFORE, the parties do hereby establish the Trust and agree that
the Trust shall be comprised, held and disposed of as follows:

     1.   Definitions.
          ----------- 

          (a) "Change in Control" means (i) a merger, consolidation or
combination of the Company or the Bank with or into another entity (other than a
transaction in which the Company and the Bank are continuing entities with the
shares of the common stock of the Company (the "Shares") outstanding immediately
prior to the transaction representing at least 51% of the outstanding Shares
immediately after the completion of the transaction) unless a majority of the
members of the Board of Directors of the Company immediately prior to such
transaction constitute a majority of the Board of Directors of the surviving
entity and resulting bank immediately after the transaction or (b) a change in
the individuals who constitute a majority of the Board of Directors of the
Company, by resignation or non-election of persons nominated by the Board of
Directors, occurring after any person, or group of persons acting in concert,
acquires more than 10% of the outstanding Shares.

                                       2
<PAGE>
 
          (b) "Insolvency" or "Insolvent" means (i) the Bank is unable to pay
its debts as they mature, (ii) the Bank is subject to a pending proceeding as a
debtor under the federal bankruptcy laws, or (iii) the Bank is determined to be
insolvent by the Office of Thrift Supervision or other federal banking agency
with jurisdiction to make such determination.

          (c) "Normal Form of Payment" means the payment to a Participant (or
his or her beneficiaries) within thirty (30) days after a Termination of Service
of 1/10th of the Shares allocated to such Participant's account in the Trust
plus any other assets contained in such Participant's account in the Trust that
are not Shares, with the remaining Shares being distributed in nine (9) equal or
substantially-equal annual installments commencing one year after the date of
the first distribution, with any other assets accumulated during each annual
period being distributed on each annual payment date.  The Normal Form of
Payment shall not apply after a Change in Control.

          (d) "Post Change in Control Form of Payment" means (i) in the case of
a Participant who has not experienced a Termination of Service prior to a Change
in Control, the payment of cash in the amount of 1/10th of the principal balance
of his or her account in the Trust within thirty (30) days after a Termination
of Service, with the balance of such principal being paid in nine (9) equal or
substantially-equal annual installments commencing one year from the date of the
first principal payment with earnings accumulating during each annual period
being distributed on each annual payment date; and (ii) in the case of a
Participant (or his or her beneficiaries) who has received one or more payments
prior to a Change in Control, the payment of cash in equal or substantially-
equal annual installments of principal plus accumulated annual earnings on the
same annual payment dates that such Participant (or his or her beneficiaries)
would have otherwise received payments in Shares under the Normal Form of
Payment.

                                       3
<PAGE>
 
          (e) "Termination of Service" means termination, for any reason, of the
service of a Participant as a director of the Bank.  Provided however, if such
Participant is also a director of the Company, then a Termination of Service
shall not occur until such Participant is no longer a director of both the Bank
and the Company (including any successor in interest).

     2.   Establishment of Trust.
          ---------------------- 

          (a) Subject to the claims of its creditors as set forth in Section 4,
the Bank hereby deposits with the Trustee in trust One Million Ninety Thousand
Two Hundred and Twenty-One Dollars ($1,090,221) allocable to each Participant as
set forth on Exhibit II hereto (a "Participant's Initial Principal"), which
funds shall become the initial principal of the Trust to be held, administered
and disposed of by Trustee as provided in this Trust Agreement.

          (b) The Trust hereby established shall be revocable until the
occurrence of a Change in Control and shall become irrevocable thereafter.

          (c) The Trust is intended to be a grantor trust, of which the Bank is
the grantor, within the meaning of subpart E, part I, subchapter J, chapter 1,
subtitle A of the Internal Revenue Code of 1986, as amended (the "Code") and
shall be construed accordingly.

          (d) The principal of the Trust and any earnings thereon (other than
ordinary cash dividends on Shares held by the Trust as described in Section 7
hereinbelow) shall be held separate and apart from other funds of the Bank and
shall be used exclusively for the benefit of the Participants, their
beneficiaries, and general creditors of the Bank, as herein set forth.  The
Participants and their beneficiaries shall have no preferred claim on, or any
beneficial ownership interest in, any assets of the Trust.  Any rights created
under the Plan and this Trust Agreement shall be mere unsecured contractual
rights of the Participants and their beneficiaries against the Bank.  

                                       4
<PAGE>
 
Any assets held by the Trust will be subject to the claims of the Bank's general
creditors under federal and state law in the event of Insolvency.

          (e) No additional contributions shall be made to the Trust by the
Bank.

     3.   Payments to Participants and Their Beneficiaries.
          ------------------------------------------------ 

          (a) Subject to the provisions of Section 4 hereof, in the event a
Participant experiences a Termination of Service prior to a Change in Control,
such Participant (or his or her beneficiaries) shall receive the Normal Form of
Payment.  Provided however, if a Change in Control occurs after the commencement
of the Normal Form of Payment, then following such Change in Control, the Post
Change in Control Form of Payment shall then apply.

          (b) Subject to the provisions of Section 4 hereof, in the event a
Participant experiences a Termination of Service in connection with or after a
Change in Control, such Participant (or his or her beneficiaries) shall receive
the Post Change in Control Form of Payment.

          (c) Subject to the provision of Section 4 hereof, and notwithstanding
anything contained in Sections 3(a) or (b) above or otherwise in this Trust
Agreement to the contrary, upon a determination by the Board of Directors of the
Bank prior to a Change in Control to impose a single, lump-sum cash distribution
of benefits as soon as practicable after a Change in Control pursuant to Section
4.2 of the Amendment, then payment shall be made to Participants (or their
beneficiaries) in a single, lump-sum cash payment as provided in Section 4.2 of
the Amendment.

          (d) The Trustee shall make provision for the reporting and withholding
of any federal, state or local taxes that may be required to be withheld.

          (e) Each Participant shall designate one or more beneficiaries to
receive benefits under this Trust Agreement in the event of such Participant's
death.  Such designation may be 

                                       5
<PAGE>
 
changed at any time. Absent a valid designation, or if no designated beneficiary
survives a Participant, such Participant's remaining undistributed benefits
under this Trust Agreement shall be distributed in the following order of
priority: (i) spouse; (ii) surviving children (or if a child has died leaving
children surviving, then such deceased child's surviving children), all on a per
                                                                             ---
stirpes basis; (iii) parents; and (iv) estate. Whenever the rights of the
-------
Participants are stated or limited herein, such provisions shall also apply to
their beneficiaries. If a Participant should die while receiving benefits
hereunder, such Participant's beneficiaries shall be entitled to receive annual
payments for the remaining term of such Participant's pay-out period.

     4.   Trustee Responsibility Regarding Payments to Participants and their
          -------------------------------------------------------------------
Beneficiaries when Bank is Insolvent.
------------------------------------ 

          (a) In the event of an Insolvency, the Trustee shall cease payment of
benefits to Participants or their beneficiaries consistent with the following
provisions.

          (i) The Board of Directors and the President of the Bank shall have
the duty to inform the Trustee in writing of the Insolvency. If a person
claiming to be a creditor of the Bank alleges in writing to the Trustee that the
Bank has become Insolvent, the Trustee shall independently determine, within
thirty (30) days after receipt of such notice, whether the Bank is Insolvent
and, pending such determination, the Trustee shall discontinue payments to
Participants or their beneficiaries.

                                       6
<PAGE>
 
          (ii)  Unless the Trustee has actual knowledge of the Insolvency, or
has received notice from the Bank or a person claiming to be a creditor alleging
Insolvency, the Trustee shall have no duty to inquire whether the Bank is
Insolvent. The Trustee may in all events rely on such evidence concerning the
Bank's solvency as may be furnished to the Trustee and that provides the Trustee
with a reasonable basis for making a determination concerning the Bank's
solvency.

          (iii) If at any time the Trustee has determined that the Bank is
Insolvent, the Trustee shall discontinue payments to Participants or their
beneficiaries and shall hold the assets of the Trust for the benefit of the
Bank's general creditors. Nothing in this Trust shall in any way diminish any
rights of any Participant or any beneficiary to pursue his or her rights as a
general creditor of the Bank with respect to benefits under this Trust Agreement
or otherwise.

          (iv)  The Trustee shall resume the payment of benefits to Participants
or their beneficiaries in accordance with Section 3 of this Trust Agreement only
after the Trustee has determined that the Bank is not Insolvent (or is no longer
Insolvent).

          (b) Provided that there are sufficient assets, if the Trustee
discontinues payments from the Trust pursuant to Section 4(a) hereof and
subsequently resumes such payments, the first payment following such
discontinuance shall include the aggregate amount of all payments which would
have been made to a Participant (or his or her beneficiaries) in accordance with
the terms of this Trust Agreement during the period of such discontinuance.

          (c) At all times during the continuance of the Trust, the principal
and income of the Trust shall be subject to claims of general creditors of the
Bank under federal and state law in the event of Insolvency.

                                       7
<PAGE>
 
     5.   Payments to Bank.  The Bank shall have the right or power prior a
          ----------------                                                 
Change in Control to direct the Trustee to return to the Bank the assets then
held by the Trust but subject to the terms of the Amendment and the provisions
of this Trust Agreement relating to the investment and reinvestment of the
assets of the Trust and the payment of benefits to Participants or their
beneficiaries; provided however, that such right or power shall cease upon a
Change in Control.

     6.   Investment of the Corpus; Separate Accounts; Adjustments to Shares.
          ------------------------------------------------------------------  
The Trustee shall invest a Participant's Initial Principal as soon as
practicable solely in Shares.  The Trustee shall maintain a separate account,
which account may be for bookkeeping purposes only, for each Participant,
reflecting the number of Shares and other assets, if any, allocated to such
Participant through the investment and reinvestment of such Participant's
Initial Principal and the earnings and proceeds therefrom.  Any adjustments or
additions to, or conversion of, the Shares by virtue of any dividend (other
than an ordinary cash dividend), recapitalization, reorganization, merger,
consolidation, spin-off, combination, repurchase, exchange of shares, stock
split, reverse stock split or other corporate transaction including, but not
limited to, an acquisition of the Company will be appropriately allocated to
each Participant's separate account.

     7.   Disposition of Income.  During the term of this Trust, all income
          ---------------------                                            
received by the Trust, net of expenses and taxes, shall be accumulated and
reinvested and added to the corpus of the Trust.  Notwithstanding the foregoing,
all ordinary cash dividends received by the Trust with respect to the Shares
shall be the sole and exclusive property of the Bank and shall be immediately
tendered by the Trustee to the Bank.

                                       8
<PAGE>
 
     8.   Permitted Investments after a Change in Control.  In the event of a
          -----------------------------------------------                    
Change in Control, all Shares and other non-cash assets held by the Trust
(including those Shares and non-cash assets held for the benefit of a
Participant (or his or her beneficiaries) who is then receiving the Normal Form
of Payment) shall be converted to cash as soon as practicable following the
completion of the Change in Control transaction.  The Trustee shall then invest
the cash proceeds in legally permissible investments that do not involve risk to
principal, with investment options being selected in a manner to permit
installment cash payments consistent with the Post Change in Control Form of
Payment; provided however, that any annual earnings thereon in excess of 120% of
the long-term, applicable federal rate ("AFR") under Section 274(d) of the Code
for the month in which the Change in Control occurs, shall revert to the Bank.

     9.   Accounting by Trustee.  The Trustee shall keep accurate and detailed
          ---------------------                                               
records of all investments, receipts, disbursements, and all other transactions
required to be made, including such specific records as shall be agreed upon in
writing between the Bank and the Trustee. All such accounts, books and records
shall be open to inspection and audit at all reasonable times by the Bank and
Participants (or their beneficiaries). Within sixty (60) days following the
close of each calendar year and within sixty (60) days after the removal or
resignation of the Trustee, the Trustee shall deliver to the Bank and each
Participant (or his or her beneficiaries) a written account of its
administration of the Trust during such year or during the period from the close
of the last preceding year to the date of such removal or resignation, setting
forth all investments, receipts, disbursements and other transactions effected
by it, including a description of all securities and investments purchased and
sold, with the cost or net proceeds of such purchases or sales (accrued interest
paid 

                                       9
<PAGE>
 
or receivable being shown separately), and showing all cash, securities and
other property held in the Trust at the end of such year or as of the date of
such removal or resignation, as the case may be.

     10.  Responsibility of Trustee.
          ------------------------- 

          (a) The Trustee shall act with the care, skill, prudence and diligence
under the circumstances then prevailing that a prudent person acting in a like
capacity and familiar with such matters would use in the conduct of an
enterprise of a like character and with like aims; provided, however, that the
Trustee shall incur no liability to anyone for any action taken pursuant to a
direction, request, or approval given by the Bank which is in conformity with
the terms of the Plan, the Amendment and this Trust Agreement, and is given in
writing to the Trustee.  In the event of a dispute between the Bank and a
Participant (or his or her beneficiaries), the Trustee may apply to a court of
competent jurisdiction to resolve the dispute.

          (b) The Trustee shall not be required to undertake or to defend any
litigation arising in connection with this Trust Agreement, unless it be first
indemnified by the Bank against its prospective costs, expenses and liability,
and the Bank hereby agrees to indemnify the Trustee for such costs, expenses,
and liability.

          (c) The Trustee may consult with legal counsel (who may also be
counsel for the Bank generally) with respect to any of its duties or obligations
hereunder.

          (d) The Trustee may hire agents, accountants, actuaries, investment
advisers and financial consultants or other professionals to assist it in
performing any of its duties or obligations hereunder.

                                      10
<PAGE>
 
          (e) Notwithstanding any powers granted to the Trustee pursuant to this
Trust Agreement or under applicable law, the Trustee shall not have any power to
borrow funds or engage in any activity that could give this Trust the objective
of carrying on a business and dividing the gains therefrom within the meaning of
section 301.7701-2 of the Procedure and Administrative Regulations promulgated
pursuant to the Code.

     11.  Compensation and Expenses of Trustee.  The Trustee shall be entitled
          ------------------------------------                                
to receive such reasonable compensation for its services as shall be agreed upon
by the Bank and the Trustee.  The Trustee shall also be entitled to receive its
reasonable expenses incurred with respect to the administration of the Trust,
including fees incurred by the Trustee pursuant to Sections 10(c) and 10(d) of
this Trust Agreement.  Such compensation and expenses shall be payable by the
Bank.

     12.  Resignation and Removal of Trustee.
          ---------------------------------- 

          (a) The Trustee may resign at any time by written notice to the Bank,
which shall be effective sixty (60) days after receipt of such notice unless the
Bank and the Trustee agree otherwise.

          (b) The Trustee may be removed by the Bank on sixty (60) days notice
or upon shorter notice accepted by the Trustee, except upon a Change in Control,
the Trustee may not be removed until all payments to the Participants or their
beneficiaries have been made pursuant to this Trust Agreement, unless all
Participants (or their beneficiaries) with assets remaining in the Trust shall
unanimously consent in writing to such change of Trustee.

                                      11
<PAGE>
 
          (c) Upon resignation or removal of the Trustee and appointment of a
successor Trustee, all assets shall subsequently be transferred to the successor
Trustee.  The transfer shall be completed within sixty (60) days after receipt
of notice of resignation, removal or transfer, unless the Bank extends the time
limit.
          (d) In the event that the Trustee resigns or is removed, a successor
shall be appointed, in accordance with Section 13 hereof, by the effective date
of such resignation or removal.  If no such appointment has been made, the
Trustee may apply to a court of competent jurisdiction for appointment of a
successor or for instructions.  All expenses of the Trustee in connection with
the proceeding shall be paid by the Bank.

     13.  Appointment of Successor Trustee.  In the event that the Trustee
          --------------------------------                                
resigns or is removed in accordance with Section 12(a) or (b) hereof, the Bank
may appoint any independent third party, as a successor, to replace the Trustee
upon resignation or removal. The appointment shall be effective when accepted in
writing by the new Trustee, which shall have all of the rights and powers of the
former Trustee including ownership rights in the Trust assets.  The former
Trustee shall execute any instrument necessary or reasonably requested by the
Bank or the successor Trustee to evidence the transfer.

     14.  Amendment or Termination.
          ------------------------ 

          (a) This Trust Agreement may be amended at any time and to any extent
by a written instrument executed by the Trustee and the Bank and consented to by
all Participants (or their beneficiaries).

                                      12
<PAGE>
 
          (b) The Trust shall not terminate until the earlier of (i) the
transfer of the assets of the Trust pursuant to a pre-Change in Control
revocation of the Trust by the Bank or (ii)  the date on which no Participant or
beneficiary is entitled to any further benefits under this Agreement.

     15.  Severability and Alienation.
          --------------------------- 

          (a) Any provision of this Trust Agreement prohibited by law shall be
ineffective to the extent of any such prohibition without invalidating the
remaining provisions hereof.

          (b) Benefits payable to a Participant or his or her beneficiaries
under this Trust Agreement may not be anticipated, assigned (either at law or in
equity), alienated or subject to attachment, garnishment, levy, execution or
other legal or equitable process and no benefit actually paid to a Participant
or his or her beneficiaries by the Trustee shall be subject to any claim for
repayment by the Bank or the Trustee.

     16.  Definitions.  Unless the context of this Trust Agreement clearly
          -----------                                                     
indicates otherwise, the terms defined in the Plan and the Amendment shall, when
used herein, have the same meaning in said Plan and Amendment.

     17.  Headings and Subheadings.  The headings and subheadings in this Trust
          ------------------------                                             
Agreement are inserted for the convenience of reference only and are to be
ignored in any construction of the provisions thereof.

     18.  Governing Law.  This Trust Agreement shall be governed by and
          -------------                                                
construed in accordance with the laws of the State of California.

                                      13
<PAGE>
 
     IN WITNESS WHEREOF, the Bank and the Trustee have executed this Agreement
as of the date first above written.

                                  BAY VIEW FEDERAL BANK

ATTEST:

                                      /s/ Edward H. Sondker
                                  -----------------------------------
                                  Edward H. Sondker, President
/s/ Robert Flax
----------------------------
Robert Flax
Secretary



STATE OF CALIFORNIA      )
                         ) to wit:
COUNTY OF SAN MATEO      )

     I HEREBY CERTIFY that on this 8th day of May, 1996, before me, the
subscriber, a Notary Public of the State of California, in and for San Mateo
County, aforesaid, personally appeared Edward H. Sondker, President of Bay View
Federal Bank, and duly acknowledged the foregoing Trust Agreement to be the act
and deed of said corporation.

                                    /s/Sandra H. Bruce
                                  ----------------------------------
                                  Notary Public
My Commission Expires:

     September 3, 1998
----------------------------

                                      14
<PAGE>
 
                                FIRST BANKERS TRUST CO., N.A.

ATTEST:

                                /s/ Carmen Walch
                               -------------------------------------
                               Carmen Walch, Trust Officer
  /s/ Patricia Brink
 -----------------------
Patricia Brink, Cashier



STATE OF ILLINOIS   )
                    ) to wit:
COUNTY OF ADAMS     )


     I HEREBY CERTIFY that on this 3rd day of May, 1996, before me, the
subscriber, a Notary Public of the State of Illinois, in and for Adams County,
aforesaid, personally appeared Carmen Walch, Trust Officer of First Bankers
Trust Co., N.A. of Quincy, Illinois, and duly acknowledged the foregoing Trust
Agreement to be the act and deed of the aforesaid corporation.

                                /s/ Deborah Yakle-Hutter
                               -------------------------------------
                               Notary Public
My Commission Expires:

    7-19-99
-----------------------------

                                      15
<PAGE>
 
                                FIRST AMENDMENT

                                    TO THE

                                TRUST AGREEMENT

                                    FOR THE

           BAY VIEW FEDERAL BANK OUTSIDE DIRECTORS' RETIREMENT PLAN


     WHEREAS,  Bay View Federal Bank, A Federal Savings Bank ("Bank") and First
Bankers Trust Co., N.A. ("Trustee") entered into the Trust Agreement for the Bay
View Federal Bank Outside Directors' Retirement Plan as of May 23, 1996 ("Trust
Agreement"); and

     WHEREAS, the Bay View Federal Bank Outside Directors' Retirement Plan
("Plan") has been amended; and

     WHEREAS, the Bank and the Trustee desire to make certain amendments to the
Trust Agreement.

     NOW THEREFORE, in consideration of the premises and for other good and
valuable consideration, the receipt and sufficiency of which is hereby
acknowledged, it is agreed as follows:

1.  Section 1(c) of the Trust Agreement shall be and hereby is amended by the
addition of the following sentence at the conclusion thereof:

     Notwithstanding the foregoing, a Participant shall be entitled within
     fifteen (15) days after a Termination of Service to elect, by written
     notice to the Trustee, to have his separate account reinvested and
     distributed to him in the Post Change in Control Form of Payment, even
     though no Change in Control shall have occurred.

2.  Section 5 of the Trust Agreement shall be and hereby is amended in its
entirety to read as follows:

     5.  Payments to Bank.  The Bank shall have the right and power prior to a
         ----------------                                                     
     Change in Control to direct the Trustee to return to the Bank the assets
     then held by the Trust, but subject to the terms of the Plan, as amended
     from time to time.

3.  Section 8 shall be and hereby is amended in its entirety to read as follows:

     8.  Permitted Investments.  In the event of a Change in Control, all Shares
         ---------------------                                                  
     and other non-cash assets held by the Trust (including but not limited to
     those Shares and non-cash assets held for the benefit of a Participant (or
     his or her beneficiaries) who is then receiving the Normal Form of Payment)
     shall be converted to cash as soon as practicable following the completion
     of the Change in Control transaction.  In 
<PAGE>
 
     addition, in the event that a Participant who has incurred a Termination of
     Service elects in a timely manner to have his separate account paid to him
     in a Post Change of Control Form of Payment, the Trustee shall liquidate
     all Shares attributable to such Participant's separate account. In either
     of the foregoing events, the Trustee shall then invest the cash proceeds in
     those investments that are legally permitted for trust investment under
     prudent investment standards applicable to trusts under the laws of the
     State of Illinois, with investment options and vehicles being selected by
     the Trustee in a manner to permit installment cash payments consistent with
     the Post Change in Control Form of Payment.

     The Trust Agreement for the Bay View Federal Bank Outside Directors'
Retirement Plan shall be and hereby is affirmed in its entirety, except for the
changes effectuated by this First Amendment.

     IN WITNESS WHEREOF the duly authorized officers of Bay View Federal Bank
and First Bankers Trust Co., N.A. have executed this First Amendment as of the
5th day of February, 1997.



                                    BAY VIEW FEDERAL BANK


ATTEST:


                              By: /s/Edward H. Sondker
                                  ---------------------------          
                                    Edward Sondker, President
/s/ Robert J. Flax
------------------    
Robert J. Flax
Secretary
<PAGE>
 
STATE OF CALIFORNIA   )
                      ) to wit:
COUNTY OF SAN MATEO   )


     I HEREBY CERTIFY that on this 5th day of February, 1997, before me, the
subscriber, a Notary Public of the State of California, in and for San Mateo
County, aforesaid, personally appeared Edward Sondker, President of Bay View
Federal Bank, and duly acknowledged the foregoing Trust Agreement to be the act
and deed of said banking institution.



____________________
Notary Public


My Commission Expires:


_____________________
<PAGE>
 
                                    FIRST BANKERS TRUST CO., N.A.


ATTEST:


                              By:___________________________
                                    Carmen Walch, Trust Officer
__________________________



STATE OF ILLINOIS     )
                      ) to wit:
COUNTY OF ADAMS       )


     I HEREBY CERTIFY that on this ____ day of     ,1997, before me, the
subscriber, a Notary Public of the State of Illinois, in and for Adams County,
aforesaid, personally appeared Carmen Walch, Trust Officer of First Bankers
Trust Co., N.A. of Quincy, Illinois, and duly acknowledged the foregoing Trust
Agreement to be the act and deed of said banking institution.



____________________
Notary Public


My Commission Expires:


_____________________
<PAGE>
 
This First Amendment has been consented to by the Participants in the Plan on
the dates set forth under their names and signatures.



PARTICIPANTS:



____________________________
Name


Date:



____________________________
Name


Date: