<PAGE>

EXHIBIT 10.4

                         BAY VIEW CAPITAL CORPORATION


                     Supplemental Phantom Stock Unit Plan


      1.  Plan Purpose.  The purpose of the Plan is to focus Participants on
          ------------
measures that will lead to the creation of value for the Corporation and its
shareholders and to provide the Participants with the opportunity to earn
significant rewards commensurate with performance and shareholder value
creation.

      2.  Definitions.  The following definitions are applicable to the Plan:
          -----------

          "Affiliate" -- means any "parent corporation" or "subsidiary
corporation" of the Corporation, as such terms are defined in section 425(e) and
(f), respectively, of the Code.

          "Agreement"-- means the written agreement entered into between the
Corporation and a Participant to carry out the Plan with respect to the rights
and entitlements of a Participant to earn Awards in accordance with the Plan's
terms and conditions.

          "Award" -- means the grant by the Committee of a Phantom Stock Unit,
          as provided in the Plan.

          "Change in Control" -- means any of the events specified in the
following clauses (i) through (iii):  (i) any third person, including a "group"
as defined in section 13(d) (3) of the Securities Exchange Act of 1934, shall
become the beneficial owner of shares of the Corporation with respect to which
25% or more of the total number of votes for the election of the Board of
Directors of the Corporation may be cast, (ii) as a result of, or in connection
with, any cash tender offer, merger or other business combination, sale of
assets or contested election, or combination of the foregoing, the persons who
were directors of the Corporation shall cease to constitute a majority of the
Board of Directors of the Corporation, or (iii) the stockholders of the
Corporation shall approve an agreement providing either for a transaction in
which the Corporation will cease to be an independent publicly-owned corporation
or for a sale or other disposition of all or substantially all the assets

                                       1
<PAGE>

of the Corporation.

          "Code" -- means the Internal Revenue Code of 1986, as amended.

          "Committee" -- means the Committee referred to in section 3 hereof.

          "Continuous Service" -- means the absence of any interruption or
termination of service as an Employee.  Service shall not be considered
interrupted in the case of sick leave, military leave or any other leave of
absence approved by the Corporation or in the case of transfers between payroll
locations of the Corporation or between the Corporation, its Affiliates or any
successor.

          "Corporation" -- means Bay View Capital Corporation, a Delaware
corporation, and any successor thereto.

          "Effective Date" -- means December 7, 1998.

          "Employee" -- means an officer who is employed either (i) by the
Corporation or Bay View Bank at the level of senior vice president or above, or
(ii) by an Affiliate (other than Bay View Bank) at the level of executive vice
president or above.

          "Grant Date" -- means the last day of the Quarter in which an Award is
earned.

          "Market Value" -- means the average of the high and low quoted sales
price on the date in question (or, if there is no reported sale on such date, on
the last preceding date on which any reported sale occurred) of a Share on the
Composite Tape for the New York Stock Exchange-Listed Stocks, or, if on such
date the Shares are not quoted on the Composite Tape, on the New York Stock
Exchange, or if the Shares are not listed or admitted to trading on such
Exchange, on the principal United States securities exchange registered under
the Securities Exchange Act of 1934 on which the Shares are listed or admitted
to trading, or, if the Shares are not listed or admitted to trading on any such
exchange, the mean between the closing high bid and low asked quotations with
respect to a Share on such date on the Nasdaq Stock Market, or any similar
system then in use, or, if no such quotations are available, the fair market
value on such date of a Share as the Committee shall determine.

                                       2
<PAGE>

          "Net Interest Margin" -- means net interest spread after including a
factor for the excess of interest-earning assets over interest-bearing
liabilities for the three month period ending on the last day of a Quarter.  Net
interest spread represents the difference between the yield on interest-earning
assets and cost of interest-bearing liabilities adjusted for income derived from
noninterest-earning assets which are funded by interest-bearing liabilities.

          "Non-Employee Director" -- means a director who a) is not currently an
officer or employee of the Corporation or any Affiliate; b) does not receive
remuneration, directly or indirectly, from the Corporation in any capacity other
than as a director; and (c) does not possess an interest in any other
transactions or is not engaged in a business relationship for which disclosure
would be required under Item 404(a) or (b) of Regulation S-K.

          "Participant" -- means an Employee who is selected by the Committee to
participate in the Plan and with whom the Corporation enters into an Agreement.

          "Performance Target" -- means the performance criteria and the
achievement goals established pursuant to section 5 hereof for the granting of
Awards pursuant to section 6 hereof.

          "Phantom Stock Unit" -- means the right to receive on the Grant Date
the Per Unit Value multiplied by the number of hypothetical Shares earned.

          "Per Unit Value" -- means the excess of the Market Value of a Share on
the last day of the Quarter in which the Phantom Stock Unit is earned over
$21.00, but such Market Value shall be limited to $36.25 prior to the earlier of
a Change in Control or the Board of Directors of the Corporation approving a
proposal or transaction that could result in a Change in Control.

          "Plan" -- means the Supplemental Phantom Stock Unit  Plan of the
Corporation.

          "Quarter" -- means the three-month period ending on March 31, June 30,
September 30 or December 31 of each year, commencing on the Effective Date and
ending on the date that the

                                       3
<PAGE>

Plan expires in accordance with Section 16(b) hereof.

          "Share" -- means a share of the common stock of the Corporation.

          "Tangible ROE" -- means the consolidated after-tax net income of the
Corporation (including nonrecurring and extraordinary items) determined in
accordance with generally accepted accounting principles for the twelve month
period ending on the last day of a Quarter, divided by the average tangible
equity of the Corporation for such period.

          "Transaction Account Mix" -- means transaction accounts as a
percentage of total retail deposits on the last day of a Quarter.  Transaction
accounts represent savings, checking and money market accounts including other
deposit accounts which have similar interest rate risks.

      3.  Administration.  The Plan shall be administered by a Committee
          --------------
consisting of two or more individuals, each of whom shall be a Non-Employee
Director.  The members of the Committee shall be appointed, removed and/or
substituted from time to time by the Board of Directors of the Corporation.
Except as limited by the express provisions of the Plan, the Committee shall
have sole and complete authority and discretion to (i) select Participants and
grant Awards; (ii) determine the number of Shares to be subject to types of
Awards generally, as well as to individual Awards granted under the Plan; (iii)
determine the terms and conditions upon which Awards shall be granted under the
Plan; (iv) prescribe the form and terms of the Agreements; and (v) establish
from time to time regulations for the administration of the Plan, interpret the
Plan, and make all determinations deemed necessary or advisable for the
administration of the Plan.  A majority of the Committee shall constitute a
quorum, and the acts of a majority of the members present at any meeting at
which a quorum is present, or acts approved in writing by a majority of the
Committee without a meeting, shall be acts of the Committee.

      4.  Participation.  The Committee may select Employees from time to time
          -------------
to become Participants in the Plan.  As a condition of participation, the
Corporation and the Employee shall enter into an Agreement relating to the
entitlement of such Participant to earn Awards under this Plan.  Employees
approved for participation will be notified of their selection as soon after

                                       4
<PAGE>

approval as practicable.

     5.   Performance Targets.  As soon as practicable after the end of each
          -------------------
Quarter the Committee shall, based upon its review of the Corporation's
financial statements for such Quarter, determine whether one or more Performance
Targets were achieved during such Quarter.  The Committee shall certify in
writing whether a Performance Target was achieved during such Quarter based upon
the actual performance results for such Quarter.  There are three separate
Performance Targets that may be achieved under this Plan.  Each of the three
Performance Targets may be achieved by the Corporation's attainment during any
Quarter of a minimum level of performance in each of three applicable criteria -
Net Interest Margin, Tangible ROE (based upon the last four Quarters)and
Transaction Account Mix - as follows:

<TABLE>
<CAPTION>

Performance       Net Interest        Tangible ROE         Transaction Account
-----------       ------------        ------------         -------------------
  Target             Margin                                      Mix
  ------             ------
<S>               <C>                 <C>                  <C>
  First               3.3%               15.0%                      40%
  Second              3.7%               16.0%                      45%
  Third               4.0%               18.0%                      50%
</TABLE>

     6.   Grant of Awards.
          ----------------

          (a)  Immediately upon the Committee's certification in writing of the
satisfaction of a Performance Target during a Quarter each Employee who was a
Participant in the Plan on the last day of such Quarter shall be granted an
Award of a number of Phantom Stock Units in accordance with the terms of his or
her Agreement.

          (b)  In the event that one or more of the Performance Targets is not
achieved on or before December 31, 2000, each Participant who is an Employee on
December 31, 2000 shall be entitled to an Award of Phantom Stock Units hereunder
equal to the number of Phantom Stock Units that would have been granted to such
Participant if the next Performance Target had been satisfied, multiplied by a
fraction, the numerator of which is the level of performance of the lowest
performing of the three applicable criteria - Net Interest Margin, Tangible ROE
or Transaction Account Mix - on December 31, 2000, and the denominator of which
is the minimum level of performance of the same criteria that would have been
required in order to satisfy

                                       5
<PAGE>

the next Performance Target.

          (c)  In the event that a Participant ceases Continuous Service for any
reason, except as otherwise provided herein, such Participant's right of
participation in this Plan shall thereupon cease and terminate.

     7.   Terms and Conditions of Phantom Stock Units.  A Phantom Stock Unit,
          -------------------------------------------
when granted, shall evidence the right of a Participant to receive in cash the
Per Unit Value, multiplied by the number of hypothetical Shares earned thereby.

     8.   Exercise of Awards.
          ------------------

          (a)  The Per Unit Value shall be paid in cash within 45 days following
the end of the Quarter in which the Phantom Stock Unit is awarded, unless (1)
the Participant elects to defer receipt of such cash payment pursuant to
subsection (b) hereof, or (2) the Committee, in its discretion, elects to delay
the payment of all or a portion thereof until such time as the amount payable
would be deductible to the Corporation or its Affiliate as contemplated by
section 162(m) of the Code.

          (b)  Participants who participate in the Bay View Capital Corporation
Deferred Compensation Plan (the "BVCC Deferred Compensation Plan") may elect to
defer their receipt of cash payment for their Per Unit Value pursuant to the
Plan by their voluntary election to participate in the BVCC Deferred
Compensation Plan.  Based upon the terms and provisions of the BVCC Deferred
Compensation Plan, certain Participants may irrevocably elect to defer the
receipt of all or a portion of their Per Unit Value to a specified future date
as permitted in the BVCC Deferred Compensation Plan.  The election to defer the
Per Unit Value pursuant to the BVCC Deferred Compensation Plan must be in
writing and submitted to the Company's Human Resources Department within the
time period required for deferral under the BVCC Deferred Compensation Plan.

     9.   Effect of Change in Control.  In the event of a Change in Control,
          ---------------------------
each Participant, as soon as practicable following the Change in Control, shall
receive a cash award in lieu of the Phantom Stock Units that such Participant
could have earned in the future under the Plan. The amount of such cash award
shall be equal to the following:

                                       6
<PAGE>

     A)   The total number of Phantom Stock Units that the Participant is
eligible to earn upon the achievement of all three Performance Targets,
multiplied by a fraction, the numerator of which is the number of days from the
Effective Date until the Change in Control, and the denominator of which is 756;
multiplied by

     B)   The Market Value of a Share at the Change in Control, minus $21.00
minus

     C)   The value under section 7 hereof of all Phantom Stock Units previously
awarded to the Participant.

     10.  Adjustments Upon Changes in Capitalization.  In the event of any
          ------------------------------------------
change in the outstanding Shares subsequent to the Effective Date of the Plan by
reason of any reorganization, recapitalization, stock split, stock dividend,
combination or exchange of shares, merger, consolidation or any change in the
corporate structure or Shares of the Corporation, the number of Shares
underlying a Phantom Stock Unit that may thereafter be granted under the Plan,
and the calculation of the Per Unit Value, shall be appropriately adjusted by
the Committee, whose determination shall be conclusive.

     11.  Assignments and Transfers.  No right or interest of any Participant in
          -------------------------
this Plan will be assignable or transferable or subject to any lien or
encumbrance, whether directly or indirectly, by operation of law or otherwise,
including, without limitation, execution, levy, garnishment, attachment, pledge,
or bankruptcy except, in the event of the death of a Participant, by will or the
laws of descent and distribution.

     12.  Employee Rights Under the Plan.  No Employee shall have a right to be
          ------------------------------
selected as a Participant, and no Employee or other person shall have any claim
or right to be granted an Award under the Plan or under any other incentive or
similar plan of the Corporation or any Affiliate.  Neither the Plan nor any
action taken hereunder shall be construed as giving any Employee any right to be
retained in the employ of the Corporation or any Affiliate.

     13.  Withholding Tax.  The Corporation shall have the right to deduct from
          ---------------
all amounts paid in cash with respect to the Per Unit

                                       7
<PAGE>

Value under the Plan any taxes required by law to be withheld with respect to
such cash payments.

     14.  Excise Taxes.  In the event it shall be determined that any payment or
          ------------
distribution by the Corporation or its affiliates to or for the benefit of the
Participant (whether paid or payable or distributed or distributable pursuant to
the terms of the Plan or otherwise, but determined without regard to any
additional payments required under this Section) (a "Payment") would be subject
to the excise tax imposed by Section 4999 of the Internal Revenue Code (the
"Code") or any interest or penalties are incurred by the Participant with
respect to such excise tax(such excise tax, together with any such interest and
penalties, are hereinafter collectively referred to as the "Excise Tax"), then
the Participant shall be entitled to receive an additional payment (a "Gross-Up
Payment") in an amount such that after payment by the Participant of all taxes
(including any interest or penalties imposed with respect to such taxes),
including, without limitations, any income taxes (and any interest and penalties
imposed with respect thereto) and Excise Tax imposed upon the Gross-Up Payment,
the Participant retains an amount of the Gross-Up Payment equal to the Excise
Tax imposed upon the Payment.

     15.  Amendment or Termination.  The Committee, in its sole and absolute
          ------------------------
discretion, may modify or amend any or all of the provisions of this Plan at any
time and from time to time, without notice, and may suspend or terminate it
entirely.  However, no such modification, amendment, suspension, or termination
will be made prior to the Committee determining whether the approval of the
stockholders of the Corporation is required or desirable pursuant to applicable
law or regulation; provided, further, that no such amendment, suspension or
termination shall impair the rights of any Participant, without his consent, in
any Award theretofore earned pursuant to the Plan.

     16.  Effective Date and Term of Plan.
          -------------------------------

          (a)  The Plan shall become effective upon its adoption by the Board of
Directors of the Corporation, effective as of December 7, 1998.

          (b)  Unless sooner terminated under section 15 hereof, the Plan shall
expire as soon as practicable after the earliest to occur of the dates specified
below and the satisfaction of all

                                       8
<PAGE>

of the Corporation's responsibilities and obligations with respect to all Awards
earned through such date:

     (1)  The date on which all three Performance Targets have been achieved.

     (2)  December 31, 2000.

     (3)  A Change in Control.

     17.  Beneficiary Designation.  Each Participant under the Plan may, from
          -----------------------
time to time, name any beneficiary or beneficiaries (who may be named
contingently or successively) to whom any Award under the Plan is to be paid in
case of his death before he receives any or all of such Award.  Each designation
will revoke all prior designations by the same Participant, will be in a form
prescribed by the Committee, and will be effective only when filed by the
Participant in writing with the Committee during his lifetime.  In the absence
of any such designation, benefits remaining unpaid at the Participant's death
will be paid to the Participant's estate.

     18.  No Funding.  Nothing contained in this Plan and no action taken
          ----------
hereunder will create or be construed to create a trust of any kind, or a
fiduciary relationship between the Corporation and any Participant or
beneficiary or any other person.  Amounts due under this Plan at any time and
from time to time will be paid from the general funds of the Corporation.  To
the extent that any person acquires a right to receive payments hereunder, such
right shall be that of an unsecured general creditor of the Corporation.

     19.  Indemnification of Committee.  No member of the Committee shall be
          ----------------------------
liable for any act, omission, or determination taken or made in good faith with
respect to the Plan or any Awards made hereunder; and the members of the
Committee shall be entitled to indemnification and reimbursement by the
Corporation in respect of any claim, loss, damage, or expenses (including
counsel fees) arising therefrom to the full extent permitted by law and under
any directors' and officers' liability or similar insurance coverage that may be
in effect from time to time.

     20.  Binding Effect.  The Plan shall inure to the benefit of the
          --------------
Participants hereunder and their respective heirs, devisees and legal
representatives, and it shall be binding on the

                                       9
<PAGE>

successors of the Corporation and its Affiliates.

     21.  Expenses of the Plan.  The expenses of administering the Plan will be
          --------------------
borne by the Corporation.

     22.  Governing Law.  The Plan will be construed in accordance with and
          --------------
governed by the laws of the State of California, except to the extent that such
laws are preempted by Federal law.

                                       10