<PAGE>   1


                                                                    EXHIBIT 10.4


                        DEFERRED COMPENSATION AGREEMENT

                 This Deferred Compensation Agreement (the "Agreement") is made
and entered into this 19th day of June, 1996 by and between Klickitat Valley
Bank (the "Bank") and Wesley A. Kangas ("Kangas").

                                    RECITALS

                 (1)      The Bank is a state-chartered Washington financial
institution with its principal office in Goldendale, Washington.

                 (2)      Kangas is now employed full-time by the Bank as its
President and Chief Executive Officer, and has served in that position and in
other positions at the Bank since 1968.

                 (3)      The Bank recognizes the contributions that Kangas has
made to the success and profitability of the Bank, and desires to provide
deferred compensation and other consideration to Kangas as compensation for his
services and for the confidentiality covenants set forth in the Agreement.

                 Now, therefore, it is agreed:

                 1.       Deferred Compensation.

                 1.1      Kangas shall become eligible for deferred
compensation under the Agreement upon his retirement as a full-time employee of
the Bank.

                 1.2      As of the date of his retirement (the "Retirement
Date"), Kangas shall be entitled to payment of deferred compensation in the
principal sum of $180,000.00, payable either monthly or annually as determined
by Kangas, and subject to the following conditions:

                          1.2.1   For the calendar year in which Kangas
retires, if Kangas has been employed by the Bank for any portion of such year,
the aggregate of salary received by Kangas in such year plus deferred
compensation under the Agreement may not exceed $100,000.00.

                          1.2.2   For any subsequent calendar year, Kangas may
not receive deferred compensation payments under the Agreement in an amount
exceeding $60,000.00.

                          1.2.3   At least thirty (30) days prior to the
beginning of any calendar year in which Kangas is entitled to receive deferred
compensation under the Agreement, Kangas shall notify the Bank in writing of
(i) the amount of deferred compensation to be paid for such calendar year, and
(ii) whether such deferred compensation is to be paid monthly or annually.
Such election may not be revoked or changed for such calendar year once made.


Page 1 - Deferred Compensation Agreement (Wesley A. Kangas -
         Klickitat Valley Bank

<PAGE>   2
                          1.2.4.  All deferred compensation payments to be made
hereunder must be paid in full to Kangas on or before the date Kangas reaches
the age of 60.

                          1.2.5.  Any deferred compensation to be paid annually
shall be paid on or before the 30th day of January of the calendar year in
which such payment is to be made.  Deferred compensation payments to be paid
monthly shall be paid on the first day of each month in the calendar year in
which such payments are to be made.

                          1.2.6.  At no time shall Kangas be entitled to be
paid interest on any deferred compensation payments to be paid under the
Agreement.

                          1.2.7   During the term of the Agreement, the
deferred compensation to be paid under the Agreement shall be reduced by an
amount equal to any payments Kangas receives by reason of or arising from a
disability, including payments under any disability insurance policy.

                 1.3      As additional consideration under the Agreement, the
Bank shall provide Kangas with all medical, dental, vision and life insurance
coverage which the Bank provides to full-time employees.  Such coverage shall
be provided for the period from the Retirement Date to and including the
earlier of (i) the date on which Kangas reaches the age of 60, or (ii) the date
of Kangas' death.  The Bank shall provide no disability coverage.

                 1.4      For the purposes of the Agreement, the Retirement
Date shall be the date of Kangas' retirement as provided under the Employment
Agreement of June 19, 1996 between the parties (the "Employment Agreement").

                 1.5      If Kangas is terminated by the Bank with cause within
the meaning of the Employment Agreement prior to becoming eligible for deferred
compensation payments under the Agreement, the Agreement shall be null and void
as of the date of such termination, and no deferred compensation payments or
any other benefits under the Agreement shall be due to Kangas.

                 2.       Change of Control.

                 2.1      If there is a change of control of the Bank after the
Retirement Date, Kangas shall continue to be entitled to receive the deferred
compensation provided in the Agreement.

                 2.2      If Kangas is a participant in a restricted stock plan
or share option plan, and such plan is terminated involuntarily as a result of
the change of control, all stock and options shall be declared fully vested and
shall be distributed.  With respect to any unexercised options under any stock
option plan, such options may be exercised within the period provided in such
plan.  Effective as of the date of the change of control, any holding period
established for stock paid as bonus or other compensation shall be deemed
terminated, except as otherwise provided by law.

Page 2 - Deferred Compensation Agreement (Wesley A. Kangas -
         Klickitat Valley Bank


<PAGE>   3
                 2.3      As used in this Section, "control" shall mean the
acquisition of twenty-five percent (25%) or more of the voting securities of
the Bank, or of a holding company owning the shares of the Bank, by any person,
or persons acting as a group within the meaning of Section 13(d) of the
Securities Exchange Act of 1934, or to such acquisition of a percentage between
ten percent (10%) and twenty-five percent (25%) if the Board or the Comptroller
of the Currency, the FDIC, or the Federal Reserve Bank have made a
determination that such acquisition constitutes or will constitute control of
the Bank.  The term "person" refers to an individual, corporation, Bank, bank
holding company, or other entity.

                 3.       Covenants.

                 3.1      Kangas shall at all times fully cooperate with the
Bank and its affiliates in the defense or prosecution of any litigation arising
from or relating to matters about which Kangas has knowledge based on his
employment by the Bank or other work, paid or unpaid, for the Bank or any Bank
affiliate.

                 3.2      Kangas shall at all times keep all confidential and
proprietary information gained from his employment by the Bank or other paid or
unpaid work for the Bank and any Bank affiliate in strictest confidence, and
will not disclose or otherwise disseminate such information to anyone, other
than to Board members or employees of the Bank or its affiliates, except as may
be required by law, regulation or subpoena.

                 4.       Effective Date of Agreement.

                 4.1      The Agreement shall become effective as of the
Effective Date, as that term is defined in that certain Agreement and Plan of
Merger Between Columbia Bancorp and Klickitat Valley Bank of January 13, 1996,
of the acquisition of the Bank by Columbia Bancorp.  The Agreement shall not
become effective if the proposed acquisition does not occur for any reason.

                 4.2      The Agreement, when effective, shall supersede and
replace the Wesley A. Kangas Change of Control Protection Agreement between the
parties of September 26, 1995, and the latter agreement shall be deemed null
and void as of the date the Agreement becomes effective.

                 5.       Miscellaneous.

                 5.1      Kangas' retirement shall not be deemed a retirement
or general termination under any stock incentive plan applicable to Kangas,
except as otherwise provided therein or as provided under law, and shall
therefore not limit the time within which Kangas may exercise his stock option
rights, if any, under such plan unless the plan or applicable law provides to
the contrary.

                 5.2      Each and every portion of the Agreement is
contractual and not a mere recital, and all recitals shall be deemed
incorporated into the Agreement.  The Agreement shall be governed by and
interpreted according to Washington law and any

Page 3 - Deferred Compensation Agreement (Wesley A. Kangas -
         Klickitat Valley Bank


<PAGE>   4
applicable federal law.  The Agreement may not be amended except by a
subsequent written agreement signed by all parties hereto.

                 5.3      The Agreement contains the entire understanding and
agreement of the parties with respect to the parties' relationship, and all
prior negotiations, discussions or understandings, oral or written, are hereby
integrated herein.  No prior negotiations, discussions or agreements not
contained herein or in such documents shall be binding or enforceable against
the parties.

                 5.4      Kangas' rights under the Agreement are in addition
to, and subject to, Kangas' rights under the Employment Agreement, including
without limitation Section 3 of the Employment Agreement.

                 5.5      The Agreement may be signed in several counterparts.
The signature of one party on any counterpart shall bind such party just as if
all parties had signed that counterpart.  Each counterpart shall be considered
an original.  All counterparts of the Agreement shall together constitute one
original document.

                 5.6      All rights and duties of the Bank under the Agreement
shall be binding on and inure to the benefit of the Bank's successors and
assigns, including any person or entity which acquires a controlling interest
in the Bank and any person or entity which acquires all or substantially all of
Bank's assets.  The Bank and any such successor or assign shall be and remain
jointly and severally liable to Kangas under the Agreement.  Kangas may not
assign or transfer Kangas' rights or interests in or under the Agreement to
anyone, including by will or by the laws of descent and distribution.

                 5.7      If Kangas dies prior to receiving all or any portion
of the deferred compensation payable under the Agreement, the Agreement shall
be null and void as of the date of Kangas' death, and no further payments shall
be due hereunder to Kangas' estate, to any assignee, or to any other person or
entity.

                 5.8      Any waiver by any party hereto of any provision of
the Agreement, or of any breach thereof, shall not constitute a waiver of any
other provision or of any other breach.   If any provision, paragraph or
subparagraph herein shall be deemed invalid, illegal or unenforceable in any
respect, the validity and enforceability of the remaining provisions,
paragraphs and subparagraphs shall not be affected.

                 5.9      The deferred compensation and other benefits to be
paid under the Agreement shall be made from the general, unrestricted assets of
the Bank.  No person shall have or acquire any interest in such assets under
the Agreement.  The Bank's obligations hereunder are unfunded and unsecured.
Any right of any person to receive payments or other benefits under the
Agreement shall be no greater than the right of any other unsecured creditor of
the Bank, and shall not constitute a legal or equitable right, interest, lien
or claim in or to any property or assets of the Bank.  Nothing in the
Agreement, and no action by any party hereto, shall create or be deemed to
create a trust of any kind, or a fiduciary relationship between Kangas or
anyone and the Bank.

Page 4 - Deferred Compensation Agreement (Wesley A. Kangas -
         Klickitat Valley Bank


<PAGE>   5
                 5.10     Any dispute, controversy, claim or difference
concerning or arising from the Agreement or the rights or performance of either
party under the Agreement, including disputes about the interpretation or
construction of the Agreement, shall be settled through binding arbitration in
the State of Washington and in accordance with the rules of the American
Arbitration Association.  A judgment upon the award rendered in such
arbitration may be entered in any court of competent jurisdiction.

         [SIG]
______________________________________
Wesley A. Kangas


KLICKITAT VALLEY BANK

         [SIG]
By:____________________________________
      Chairman of the Board



Page 5 - Deferred Compensation Agreement (Wesley A. Kangas -
         Klickitat Valley Bank