<PAGE>   1
                                                                    EXHIBIT 10.4


                    AMENDMENT TO CHANGE OF CONTROL AGREEMENT


        This Amendment to Change of Control Agreement is made this 15th day of
March 2000, by and between LIBERTY BAY FINANCIAL CORPORATION and NORTH SOUND
BANK (hereinafter referred to jointly as the "Bank") and MARK D. FREEMAN, who
agree as follows:

        1. Paragraph 1, "Severance Benefits," is deleted in its entirety and
replaced with the following:

                1. Severance Benefits. The Bank agrees that if there is a Change
        of Control of the Bank, and the Executive leaves the employment of the
        Bank, whether voluntarily or involuntarily (other than discharge for
        cause, as defined below), within twelve (12) months after such Change of
        Control, the Executive shall receive a lump sum payment of $302,837
        payable within fifteen (15) days of termination of employment by the
        Bank.

        2. The following new paragraph is added to the Change of Control
Agreement:

        13. Noncompetition.

                (a) Participation in a Competing Business. While employed by the
        Bank and for twenty-four (24) months after Executive's employment with
        the Bank ends, Executive will not become involved with a Competing
        Business or serve, directly or indirectly, a Competing Business in any
        manner, including, without limitation, as a shareholder, member,
        partner, director, officer, manager, investor, organizer, "founder,"
        employee, consultant, or agent; provided, however, that Executive may
        acquire and passively own an interest not exceeding 2% of the total
        equity interest in any Competing Business.

                (b) No Solicitation. While employed by the Bank and for
        twenty-four (24) months after Executive's employment with the Bank ends,
        Executive will not directly or indirectly solicit or attempt to solicit:
        (1) any employees of the Bank to leave their employment, or (2) any
        customers of the Bank to remove their business from the Bank or to
        participate in any manner in a Competing Business. Solicitation
        prohibited under this Section includes solicitation by any means,
        including, without limitation, meetings, letters or other mailings,
        electronic communications of any kind, and internet communications.

                (c) Employment Outside the Restricted Area. Nothing in this
        Agreement prevents Executive, after Executive's employment with the Bank
        ends, to commence employment outside the Restricted Area (defined below)
        from a Competing Business, as long as Executive will not: (a) act as an
        employee or other representative or agent of the Competing Business
        within the Restricted Area, or (b) have any responsibilities for the
        Competing Business' operations within the Restricted Area.



                                      -1-
<PAGE>   2



                (d) Competing Business. "Competing Business" means any financial
        institution ("financial institution" means a state or national bank, a
        state or federal savings and loan association, a mutual savings bank, or
        a state or federal credit union), trust company or mortgage company
        (including without limitation, any start-up or other financial
        institution, trust company or mortgage company) that competes with, or
        will compete with, the Bank in Kitsap, Jefferson or Clallam Counties in
        the State of Washington (the "Restricted Area").

        3. Except as set forth in this Amendment, all other provisions of the
Change of Control Agreement shall remain unchanged and in full force and effect.



                                      LIBERTY BAY FINANCIAL CORPORATION



                                      By:        /s/
                                          --------------------------------------
                                          Michael J. Clementz, President and CEO



                                                 /s/
                                      -----------------------------------------
                                      MARK D. FREEMAN