<PAGE>

                                                                    EXHIBIT 10.1

                     EXECUTIVE RETIREMENT PAYMENT AGREEMENT

      THIS EXECUTIVE RETIREMENT AGREEMENT ("Agreement"), dated as of the 31st
day of March, 2004, is made and entered into by and between DAVID SMITH
("Executive") and PANHANDLE STATE BANK, an Idaho corporation ("PSB"). This
Agreement has been ratified by INTERMOUNTAIN COMMUNITY BANCORP ("IMCB"), the
parent company of PSB.

                                    Recitals

      A.    Executive is currently employed by PSB as a Senior Vice President.

      B.    Executive and PSB had previously entered into an Executive Severance
            Agreement, dated September 15, 1999 ("Severance Agreement").

      C.    Executive and PSB would like to make provisions for retirement of
            Executive and to replace the Severance Agreement by substituting
            this Agreement in its stead.

      NOW, THEREFORE, for and in consideration of the above, the mutual
covenants, terms and conditions contained herein, and other good and valuable
consideration, the receipt and sufficiency of which is hereby acknowledged, the
parties hereto agree as follows:

                                    Agreement

      1.    Definitions. The following terms shall have the meanings shown
            below, opposite each such term:

            "Affiliate" means, with respect to PSB, any corporation or entity,
            that controls, is controlled by, or is under common control with
            PSB. As used herein, the term "control" means the ownership,
            directly or indirectly, of issued and outstanding shares of stock or
            other equity interests entitled to more than fifty percent (50%) of
            the voting rights represented thereby.

            "Agreement" is defined in the first paragraph hereof.

            "Cause" means the occurrence of any of the following events:

                  (i)   Executive engages in willful misfeasance or gross
                        negligence in the performance of his duties as an
                        employee of PSB or any of its Affiliates;

                  (ii)  Executive is convicted of a crime in connection with the
                        performance of his duties as an employee of PSB or any
                        of its Affiliates; or

                                       1
<PAGE>

                  (iii) Executive engages in conduct demonstrably and
                        significantly harmful to the financial condition of PSB
                        or any of its Affiliates.

                                       2
<PAGE>

            "Disabled" means a physical or mental impairment that both renders
            Executive incapable of substantially performing the essential
            functions of his position and is expected to continue for the
            reasonably foreseeable future, as determined by PSB in its sole
            discretion.

            "Executive" is defined in the first paragraph hereof.

            "PSB" is defined in the first paragraph hereof.

            "Retirement Date" means the later of (i) May 1, 2005, or (ii) the
            date Executive ceases to be an employee of PSB and its Affiliates.

            "Retirement Payment Amount" means the sum of two hundred thousand
            dollars ($200,000), plus up to eight thousand dollars ($8,000) for
            reimbursement of medical and dental benefits.

            "Severance Agreement" is defined in paragraph B of the Recitals
            hereof.

      2.    Termination of Severance Agreement. Executive and PSB hereby agree
            that the Severance Agreement is terminated as of the date hereof,
            without the need for further action on the part of any person.

      3.    Retirement Payments. Under the terms and conditions set forth in
            this Agreement (including, without limitation, paragraph 4),
            Executive shall be entitled to receive from PSB, and PSB shall be
            obligated to pay to Executive, the Retirement Payment Amount. Such
            amount shall be paid at the times and in the manner described below:

                  a.    Timing of Payments. The chief executive officer of PSB
                        shall determine, in his or her sole discretion, the date
                        following the Retirement Date on which PSB shall
                        commence payment of the Retirement Payment Amount, which
                        date shall be no later than May 1, 2006.

                  b.    Manner of Payment. PSB shall pay the two hundred
                        thousand dollar ($200,000) portion of the Retirement
                        Payment Amount to Executive in twenty-four (24) equal
                        monthly installments, without interest, of one
                        twenty-fourth (1/24th) of two hundred thousand dollar
                        ($200,000) each, with the first such installment being
                        due and payable on the date described in paragraph 3.a
                        and subsequent installments being due and payable on the
                        same date in each subsequent month in the payment
                        period. PSB shall pay the reimbursement of medical and
                        dental benefits portion of the Retirement Payment Amount
                        to Executive, from time to time, upon receipt of
                        evidence reasonably satisfactory to it that Executive
                        has paid for such benefits for himself or members of his
                        household.

      4.    Termination for Cause. Notwithstanding any contrary provisions of
            this Agreement, PSB shall have no obligation to pay Executive, and
            Executive shall have no right to

                                       3
<PAGE>

            receive, any part of the Retirement Payment Amount if Executive is
            terminated from employment with PSB or any of its Affiliates for
            Cause.

      5.    Death or Disability. Notwithstanding any contrary provisions of this
            Agreement, if Executive becomes Disabled or dies prior to the
            Retirement Date, PSB shall pay the entire Retirement Payment Amount
            to Executive or Executive's estate, as the case may be, within sixty
            (60) days after such event.

      6.    Not An Employment Agreement. Nothing in this Agreement, expressed or
            implied, is intended to confer upon Executive a right to be employed
            by PSB or any of its Affiliates. Except with respect to the
            Retirement Payment Amount, this Agreement shall have no effect on
            the determination of compensation payable by PSB to Executive or the
            other terms of Executive's employment with PSB. The specific
            arrangements referred to herein are not intended to exclude other
            benefits that may be available to Executive upon a termination of
            employment with PSB pursuant to employee benefit plans of PSB or
            otherwise.

      7.    Withholding. Notwithstanding any contrary provisions of this
            Agreement, PSB shall withhold from the Retirement Payment Amount,
            and pay to the appropriate government agencies, all taxes that PSB
            is required by law to so withhold and pay. Amounts so withheld and
            paid shall be treated as having been paid to Executive hereunder,
            and PSB shall be relieved from further liability to Executive for
            such amounts.

      8.    Assignability. PSB may assign the Agreement, and its rights and
            obligations hereunder, to any corporation, bank or other entity with
            or into which PSB may hereafter merge or consolidate, or to which
            PSB may transfer all or substantially all of its assets, if in any
            such case said corporation, bank or other entity shall by operation
            of law or expressly in writing assume all obligations of PSB
            hereunder, as fully as if it had been originally made a party
            hereto, but may not otherwise assign this Agreement, or its rights
            and obligations hereunder. Executive may not assign or transfer this
            Agreement or any rights or obligations hereunder.

      9.    Entire Agreement. This agreement constitutes the entire
            understanding between the parties concerning its subject matter and
            supersedes the Severance Agreement and all agreements and rights
            superceded by, or waived under, the Severance Agreement.

      10.   Funding. In the event PSB decides to fund its obligations hereunder
            by setting aside assets, crediting a special account, creating
            reserves, purchasing insurance policies or through any other means
            whatsoever, Executive shall have no rights therein, but shall
            remain, at all times, an unsecured general creditor of PSB with
            respect to any claims it may have to payments under this Agreement.

      11.   General Provisions.

                  a.    Choice of Law. This Agreement is made with reference to,
                        and shall be construed in accordance with, the laws of
                        the State of Idaho.

                                       4
<PAGE>

                  b.    Arbitration. Any dispute, controversy or claim arising
                        out of, in connection with, or relating to this
                        Agreement or any breach or alleged breach hereof, shall,
                        upon the request of any party involved, be submitted to,
                        and be settled by, arbitration pursuant to the rules
                        then in effect of the American Arbitration Association
                        (or under any other form of arbitration mutually
                        acceptable to the parties so involved). Any award
                        rendered shall be final and conclusive upon the parties
                        and a judgment thereon may be entered in the highest
                        court having jurisdiction over the parties or the
                        matter. The arbitrator shall render a written decision,
                        naming the substantially prevailing party in the action,
                        and shall award to such party all costs and expenses
                        incurred in the dispute, including reasonable attorneys'
                        fees.

                  c.    Attorney Fees. In the event of a breach of, or default
                        under, this Agreement that results in either party
                        incurring attorney or other fees, costs or expenses
                        (including in arbitration), the substantially prevailing
                        party shall be entitled to recover from the
                        non-prevailing party any and all such fees, costs and
                        expenses, including attorneys' fees.

                  d.    Successors. This Agreement shall bind and inure to the
                        benefit of the parties and each of their respective
                        affiliates, legal representatives, heirs, successors and
                        assigns.

                  e.    Amendment. This Agreement may be amended only by a
                        writing signed by the parties.

                  f.    Headings. The headings of sections and paragraphs of
                        this Agreement have been included for convenience of
                        reference only and shall not be construed to modify or
                        otherwise affect in any respect any of the provisions of
                        the Agreement.

IN WITNESS WHEREOF, each of the parties have executed, or have caused its duly
authorized representative to execute, this Agreement, which shall be effective
as of the date first stated above.

PANHANDLE STATE BANK,
an Idaho corporation

By: /s/ Curt Hecker                        /s/ David Smith
    -----------------------------          -------------------------------
    Chief Executive Officer                David Smith

Date: 3/31/04                              Date: 4/12/04

                                       5
<PAGE>

AGREED TO AND RATIFIED by:

IMCB
Intermountain Community Bancorp

By: /s/ Curt Hecker
    --------------------------------------
    Curt Hecker
    President and Chief Executive Officer

Date: 3/31/04

                                       6