<PAGE>   1
                                                                    EXHIBIT 10.4


                                 PROMISSORY NOTE

<TABLE>
<CAPTION>
==============================================================================================================================
Principal         Loan Date      Maturity         Loan No      Call     Collateral    Account     Officer    Initials
<S>               <C>            <C>              <C>          <C>      <C>           <C>         <C>        <C>
$1,000,000.00     01-15-1992     01-16-1999       9001         2A2      STK           4009577     40DXT
==============================================================================================================================
References in the shaded area are for Lender's use only and do not limit the
applicability of this document to any particular loan or item.
==============================================================================================================================
</TABLE>

Borrower: COWLITZ BANKCORPORATION               LENDER: KEY BANK OF WASHINGTON
          927 COMMERCIAL AVENUE                         CORPORATE BANKING
          LONGVIEW, WA 98632                            700 FIFTH AVENUE
                                                        P.O. BOX 90
                                                        SEATTLE, WA 98111-0090
--------------------------------------------------------------------------------

Principal Amount:  $1,000,000.00                 Date of Note:  January 15, 1992

PROMISE TO PAY. COWLITZ BANCORPORATION ("Borrower") promises to pay to KEY BANK
OF WASHINGTON ("Lender"), or order, in lawful money of the United States of
America, the principal amount of One Million & 00/100 Dollars ($1,000,000.00),
together with interest on the unpaid principal balance from January 16, 1992,
until paid in full. The interest rate will not increase above 14.000%.

PAYMENT. Subject to any payment changes resulting from changes in the Index,
         Borrower will pay this loan in accordance with the following payment 
         schedule:

         28 consecutive quarterly interest payments, beginning April 16, 1992,
         with interest calculated on the unpaid principal balances at an
         interest rate of 1.000 percentage points over the index described
         below; 1 principal payment of $115,000.00 on January 16, 1993, with
         interest calculated on the unpaid principal balances at an interest
         rate of 1.000 percentage points over the index described below; 1
         principal payment of $124,775.00 on January 16, 1994, with interest
         calculated on the unpaid principal balances at an interest rate of
         1.000 percentage points over the Index described below; 1 principal
         payment of $135,380.88 on January 16, 1995, with interest calculated on
         the unpaid principal balances at an interest rate of 1.000 percentage
         points over the index described below; 1 principal payment of
         $146,888.25 on January 16, 1996, with interest calculated on the unpaid
         principal balances at an interest rate of 1.000 percentage points over
         the index described below; 1 principal payment of $159,373.75 on
         January 16, 1997, with interest calculated on the unpaid principal
         balances at an interest rate of 1.000 percentage points over the index
         described below; 1 principal payment of $172,920.52 on January 16,
         1998, with interest calculated on the unpaid principal balances at an
         interest rate of 1.000 percentage points over the index described
         below; and 1 principal payment of $145,661.60 on January 16, 1999, with
         interest calculated on the unpaid principal balances at an interest
         rate of 1.000 percentage points over the index described below. This
         estimated final payment is based on the assumption that all payments
         will be made exactly as scheduled and that the index does not change;
         the actual final payment will be for all principal and accrued interest
         not yet paid, together with any other unpaid amounts under this Note.

Interest on this Note is computed on a 365/360 simple interest basis; that is,
by applying the ratio of the annual interest rate over a year of 360 days, times
the outstanding principal balance, times the actual number of days the principal
balance is outstanding. Borrower will pay Lender at Lender's address shown above
or at such other place as Lender may designate in writing. Unless otherwise
agreed or required by applicable law, payments will be applied first to accrued
unpaid interest, then to principal, and any remaining amount to any unpaid
collection costs and late charges.

VARIABLE INTEREST RATE. The interest rate on this Note is subject to change from
time to time based on changes in an index which is the Lender's announced prime
rate (the "Index"). The interest rate will change on the date of each announced
change of the Index within Key Bank of Washington. The Index is not necessarily
the lowest rate charged by Lender on its loans and is set by Lender in its sole
discretion. If the Index becomes unavailable during the term of this loan, the
Lender may designate a substitute index after notifying Borrower. Lender will
tell Borrower the current Index rate upon Borrower's request. Borrower
understands that Lender may make loans based on other rates as well. The
interest rate change will not occur more often than each day that the Index
changes. The Index currently is 6.500% per annum. The interest rate or rates to
be applied to the unpaid principal balance of this Note will be the rate or
rates set forth in the "Payment" section. Notwithstanding any other provision of
this Note, after the first payment stream, the interest rate for each subsequent
payment stream will be effective as of the last payment date of the just-ending
payment stream. Notwithstanding the foregoing, the variable interest rate or
rates provided for in this Note will be subject to the following minimum and
maximum rates. NOTICE: Under no circumstances will the interest rate on this
Note be less than 8.500% per annum or more than (except for any higher default
rate shown below) the lesser of 14.000% per annum or the maximum rate allowed by
applicable law. Whenever increases occur in the interest rate, Lender, at its
option, may do one or more of the following: (a) increase Borrower's payments to
ensure Borrower's loan will pay off by its original final maturity date, (b)
increase Borrower's payments to cover accruing interest, (c) increase the number
of Borrower's payments, and (d) increase the final payment.

PREPAYMENT; MINIMUM INTEREST CHARGE. Borrower agrees that all loan fees and
other prepaid finance charges are earned fully as of the date of the loan and
will not be subject to refund upon early payment (whether voluntary or as a
result of default), except as otherwise required by law. In any event, even upon
full prepayment of this Note, Borrower understands that Lender is entitled to a
minimum interest charge of $50.00. Other than Borrower's obligation to pay any
minimum interest charge, Borrower may pay without penalty all or a portion of
the amount owed earlier than it is due. Early payments will not, unless agreed
to by Lender in writing, relieve Borrower of Borrower's obligation to continue
to make payments under the payment schedule. Rather, they will reduce the
principal balance due and may result in Borrower's making fewer payments.

LATE CHARGE. If a payment is 10 days or more late, Borrower will be charged
5.000% of the regularly scheduled payment or $10.00, whichever is greater.

DEFAULT. Borrower will be in default if any of the following happens: (a)
Borrower fails to make any payment when due. (b) Borrower breaks any promise
Borrower has made to Lender, or Borrower fails to perform promptly at the time
and strictly in the manner provided in this Note or any agreement related to
this Note, or in any other agreement or loan Borrower has with Lender. (c) Any
representation or statement made or furnished lo Lender by Borrower or on
Borrower's behalf is false or misleading in any material respect. (d) Borrower
becomes insolvent, a receiver is appointed for any part of Borrower's property,
Borrower makes an assignment for the benefit of creditors, or any proceeding is
commenced either by Borrower or against Borrower under any bankruptcy or
insolvency laws. (e) Any creditor tries to take any of Borrower's property on or
in which Lender has a lien or security interest. This includes a garnishment of
any of Borrower's accounts with Lender. (f) Any of the events described in this
default section occurs with respect to any guarantor of this Note.

<PAGE>   2
01-15-1992                 PROMISSORY NOTE                                Page 2
Loan No 9001                 (Continued)
--------------------------------------------------------------------------------

LENDER'S RIGHTS. Upon default, Lender may declare the entire unpaid principal
balance on this Note and all accrued unpaid interest immediately due, without
notice, and then Borrower will pay that amount. Upon default, including failure
to pay upon final maturity, Lender, at its option, may also, if permitted under
applicable law, increase the variable interest rate on this Note by 5.000
percentage points. The interest rate will not exceed the maximum rate permitted
by applicable law. Lender may hire or pay someone else to help collect this Note
if Borrower does not pay. Borrower also will pay Lender that amount. This
includes, subject to any limits under applicable law, Lender's attorneys' fees
and legal expenses whether or not there is a lawsuit, including attorneys' fees
and legal expenses for bankruptcy proceedings (including efforts to modify or
vacate any automatic stay or injunction), appeals, and any anticipated
post-judgment collection services. If not prohibited by applicable law, Borrower
also will pay any court costs, in addition to all other sums provided by law.
This Note has been delivered to Lender and accepted by Lender in the State of
Washington. If there is a lawsuit, Borrower agrees upon Lender's request to
submit to the jurisdiction of the courts of KING County, the State of
Washington. This Note shall be governed by and construed in accordance with the
laws of the State of Washington.

DISHONORED ITEM FEE. Borrower will pay a fee to Lender of $10.00 if Borrower
makes a payment on Borrower's loan and the check or preauthorized charge with
which Borrower pays is later dishonored.

RIGHT OF SETOFF. Borrower grants to Lender a contractual possessory security
interest in, and hereby assigns, conveys, delivers, pledges, and transfers to
Lender all Borrower's right, title and interest in and to, Borrower's accounts
with Lender (whether checking, savings, or some other account), including
without limitation all accounts held jointly with someone else and all accounts
Borrower may open in the future, excluding however all IRA, Keogh, and trust
accounts. Borrower authorizes Lender, to the extent permitted by applicable law,
to charge or setoff all sums owing on this Note against any and all such
accounts.

GENERAL PROVISIONS. Lender may delay or forgo enforcing any of its rights or
remedies under this Note without losing them. Borrower and any other person who
signs, guarantees or endorses this Note, to the extent allowed by law, waive
presentment, demand for payment, protest and notice of dishonor. Upon any change
in the terms of this Note, and unless otherwise expressly stated in writing, no
party who signs this Note, whether as maker, guarantor, accommodation maker or
endorser, shall be released from liability. All such parties agree that Lender
may renew, extend (repeatedly and for any length of time) or modify this loan,
or release any party or guarantor or collateral; or impair, fail to realize upon
or perfect Lender's security interest in the collateral; and take any other
action deemed necessary by Lender without the consent of or notice to anyone.

PRIOR TO SIGNING THIS NOTE, BORROWER READ AND UNDERSTOOD ALL THE PROVISIONS OF
THIS NOTE, INCLUDING THE VARIABLE INTEREST RATE PROVISIONS. BORROWER AGREES TO
THE TERMS OF THE NOTE AND ACKNOWLEDGES RECEIPT OF A COMPLETED COPY OF THE NOTE.

BORROWER:

COWLITZ BANCORPORATION


By: /s/ Charles W. Jarrett                 By: /s/ James A. Wills
   -----------------------------------         ------------------------------
   CHARLES W. JARRETT, PRESIDENT & CEO         JAMES A. WILLS, VICE PRESIDENT