<PAGE>

                                                                    Exhibit 10.4

                                     FORM OF
                        CHANGE IN CONTROL SEVERANCE PLAN
                                       OF
                             JEFFERSON FEDERAL BANK

         1. Plan Purpose. The purpose of the Jefferson Federal Bank Employee
Severance Compensation Plan is to assure for Jefferson Federal Bank (the "Bank")
the services of Eligible Employees of the Bank in the event of a Change in
Control (capitalized terms are defined in section 2 of this Plan) of Jefferson
Bancshares, Inc. (the "Holding Company") or the Bank. The benefits contemplated
by the Plan recognize the value to the Bank of the services and contributions of
Eligible Employees of the Bank and the effect upon the Bank resulting from the
uncertainties of continued employment, reduced employee benefits, management
changes and relocations that may arise in the event of a Change in Control of
the Bank or the Company. The Board of Directors of the Bank believes that it is
in the best interests of the Bank and the Company to provide Eligible Employees
of the Bank and the Company with such benefits in order to defray the costs and
changes in employee status that could follow a Change in Control. The Board of
Directors of the Bank believes that the Plan will also aid the Bank in
attracting and retaining highly qualified individuals who are essential to its
success and the Plan's assurance of fair treatment of the Bank's Eligible
Employees will reduce the distractions and other adverse effects on Eligible
Employees' performance in the event of a Change in Control.

         2. Definitions. Whenever used herein, the following terms shall have
the meanings set forth below:

                    a.     "Affiliate" means any corporation, trade or business,
                           which, at the time of reference, is together with the
                           Bank, a member of a controlled group of corporations,
                           a group of trades or businesses (whether or not
                           incorporated) under common control, or an affiliated
                           service group, as described in Sections 414(b),
                           414(c), and 414(m) of the Code, respectively, or any
                           other organization treated as a single employer with
                           the Bank under Section 414(o) of the Code; provided,
                           however, that, where the context so requires, the
                           term "Affiliate" shall be construed to give full
                           effect to the provisions of Sections 409(l)(4) and
                           415(h) of the Code.

                    b.     "Bank" means Jefferson Federal Bank, or any successor
                           thereto.

                    c.     "Change in Control" means any one of the following
                           events occurs:

                           (i)     Merger: The Company merges into or
                                   consolidates with another corporation, or
                                   merges another corporation into the Company,
                                   and as a result less than a majority of the
                                   combined voting power of the resulting
                                   corporation immediately after the merger or
                                   consolidation is held by persons who were
                                   stockholders of the Company immediately
                                   before the merger or consolidation;

<PAGE>

                           (ii)    Acquisition of Significant Share Ownership: a
                                   report on Schedule 13D or another form or
                                   schedule (other than Schedule 13G) is filed
                                   or is required to be filed under Sections
                                   13(d) or 14(d) of the Securities Exchange Act
                                   of 1934, if the schedule discloses that the
                                   filing person or persons acting in concert
                                   has or have become the beneficial owner of
                                   25% or more of a class of the Company's
                                   voting securities, but this clause (b) shall
                                   not apply to beneficial ownership of Company
                                   voting shares held in a fiduciary capacity by
                                   an entity of which Company directly or
                                   indirectly beneficially owns 50% or more of
                                   its outstanding voting securities;

                           (iii)   Change in Board Composition: during any
                                   period of two consecutive years, individuals
                                   who constitute the Company's Board of
                                   Directors at the beginning of the two-year
                                   period cease for any reason to constitute at
                                   least a majority of the Company.'s Board of
                                   Directors; provided, however, that for
                                   purposes of this clause (iii) each director
                                   who is first elected by the board (or first
                                   nominated by the board for election by
                                   stockholders) by a vote of at least
                                   two-thirds of the directors who were
                                   directors at the beginning of the period
                                   shall be deemed to have been a director at
                                   the beginning of the two-year period; or

                           (iv)    Sale of Assets: Company sells to a third
                                   party all or substantially all of the
                                   Company's assets.

                    d.     "Company" means Jefferson Bancshares, Inc. or any
                           successor thereto.

                    e.     "Eligible Employee" means any Employee who, as of the
                           effective date of the Change in Control has or would
                           have been employed by the Bank for at least one year,
                           and whose employment, within three months prior to a
                           Change in Control, or within one year thereafter is
                           either (i) involuntarily terminated by the Company or
                           any Affiliate, other than for Just Cause, (ii)
                           voluntarily terminated by an Eligible Employee
                           following (A) a relocation of an Eligible Employee's
                           principal place of employment to a location that is
                           more than thirty-five (35) miles from its location
                           immediately prior to the Change in Control, without
                           his or her consent or (B) a reduction in the base
                           salary of the Eligible Employee from the amount being
                           paid as of the date immediately preceding the earlier
                           of their termination date (but only if it occurs
                           within three months of the Change in Control) or the
                           effective date of the Change in Control; or (iii)
                           voluntarily terminated by an Eligible Employee as a
                           result of the failure to offer or employ the Eligible
                           Employee in a "comparable position." For purposes of
                           this Plan, a "comparable position" shall mean a
                           position which (A) requires skills and knowledge
                           similar to those required in the Eligible Employee's
                           position immediately prior to the Change in Control
                           and (B) involves a work

<PAGE>

                           schedule that is substantially similar to the work
                           schedule followed by the Eligible Employee
                           immediately prior to the Change in Control. A
                           position shall not fail to be a comparable position
                           solely as a result of a change following a Change in
                           Control in the Eligible Employee's (A) title, (B)
                           supervisory authority or (C) reporting
                           responsibilities.

                    f.     "Employee" means any person who has been employed by
                           the Company or any Affiliate for at least 120 days,
                           on a full-time salaried basis, immediately prior to
                           the Change in Control, excluding any person who is
                           covered by an employment contract, change in control
                           or severance agreement with the Company or any
                           Affiliate.

                    g.     "Just Cause," with respect to termination of
                           employment, means an act or acts of personal
                           dishonesty, incompetence, willful misconduct, breach
                           of fiduciary duty involving personal profit, failure
                           to perform stated duties, willful violation of any
                           law, rule, or regulation (other than traffic
                           violations or similar offenses) or final
                           cease-and-desist order. In determining incompetence,
                           acts or omissions shall be measured against standards
                           generally prevailing in the banking industry, as
                           determined by the Board of Directors of the Bank or
                           the Company in its sole discretion.

                    h.     "Year of Service" means each consecutive 12 month
                           period, beginning with an employee's date of hire and
                           running without a termination of employment in which
                           an employee is credited with at least one hour of
                           service in each of the 12 calendar months in such
                           period. The taking of an leave of absence shall not
                           eliminate a period of time from being a Year of
                           Service if such period of time otherwise qualifies as
                           such. Further if a particular 12 month period of time
                           would not otherwise qualify under the Plan as a Year
                           of Service because one hour of service is not
                           credited during each month of such period due to the
                           taking of a leave of absence, then such period of
                           time shall be deemed to be a Year of Service for all
                           other sections of this Plan. For purposes of
                           determining a severance benefit under this Plan,
                           partial years will be rounded up to the nearest whole
                           Year of Service.

             3.     Severance Benefit to Eligible Employees.

                    a.     Each Eligible Employee shall be entitled to receive a
                           severance benefit equal to one (1) month's base pay
                           for each Year of Service with the Bank or the
                           Company. Notwithstanding the foregoing, an Eligible
                           Employee shall be entitled to a minimum severance
                           benefit equal to one (1) month base pay and a maximum
                           severance benefit equal to twelve (12) month's base
                           pay. For purposes of this Plan, "base pay" shall mean
                           1/12th of an Eligible Employee's monthly average cash
                           compensation during the

<PAGE>

                           twelve (12) months preceding the Eligible Employee's
                           termination of employment.

                    b.     All severance payments shall be made in a single lump
                           sum payment, without discount, payable within 10 days
                           of termination of employment.

                    c.     Notwithstanding the provisions of paragraph (a)
                           above, if a severance benefit payment to an Eligible
                           Employee who is a "Disqualified Individual" shall be
                           in an amount which includes an "Excess Parachute
                           Payment," when taken together with any other payments
                           or benefits that are paid or provided to the Eligible
                           Employee, the payment to that Eligible Employee shall
                           be reduced to the maximum amount which does not
                           include an Excess Parachute Payment. The terms
                           "Disqualified Individual" and "Excess Parachute
                           Payment" shall have the same meanings as defined in
                           Section 280G of the Internal Revenue Code of 1986, as
                           amended, or any successor provision thereto.

                    d.     The Eligible Employee shall not be required to
                           mitigate damages on the amount of their severance
                           benefits by seeking other employment or otherwise,
                           nor shall the amount of such severance benefit be
                           reduced by any compensation earned by the Eligible
                           Employee as a result of employment after termination
                           of employment hereunder.

             4.     Written Acknowledgment. As a condition to receiving any
payments pursuant to paragraph 3 of this Plan, the Eligible Employee shall
deliver to the Company or any applicable Affiliate on the date of his or her
employment termination a written Acknowledgment signed by the Eligible Employee
stating (i) that the severance payment to be made to the Eligible Employee
pursuant to paragraph 3 above is in full and complete satisfaction of all
liabilities and obligations of the Company and its Affiliates, directors,
officers, employees and agents, except for any tax-qualified plan benefits that
may be due and owing and except for any liabilities or obligations that may be
required by law, and (ii) that the Company or any Affiliate shall not have any
other liabilities or obligation to the Eligible Employee relating to the
Eligible Employee's employment by the Company or any Affiliate.

             5.     Legal Fees and Expenses. All reasonable legal fees and other
expenses paid or incurred by a party hereto pursuant to any dispute or question
of interpretation relating to this Plan shall be paid or reimbursed by the
prevailing party in any legal judgment, arbitration or settlement.

             6.     Required Provisions.

                    a.   The Company or any of its Affiliates may terminate an
                         employee's employment at any time, but any termination
                         by the Company or any of its Affiliates, other than
                         termination for Just Cause, shall not prejudice
                         employee's right to compensation under this Plan.
                         Employee shall not

<PAGE>

                           have the right to receive compensation for any period
                           after termination for Just Cause as defined in
                           Section 2(g) of this Plan.

                    b.     If an Employee is suspended and/or temporarily
                           prohibited from participating in the conduct of the
                           Bank's affairs by a notice served under Section
                           8(e)(3) or 8(g)(1) of the Federal Deposit Insurance
                           Act, 12 U.S.C. (S)1818(e)(3) or (g)(1), the Bank's
                           obligations under this Plan shall be suspended as of
                           the date of service, unless stayed by appropriate
                           proceedings. If the charges in the notice are
                           dismissed, the Bank may in its discretion (i) pay the
                           Employee all or part of the compensation withheld
                           while their contract obligations were suspended and
                           (ii) reinstate (in whole or in part) any of the
                           obligations which were suspended.

                    c.     If an employee is removed and/or permanently
                           prohibited from participating in the conduct of the
                           Bank's affairs by an order issued under Section
                           8(e)(4) or 8(g)(1) of the Federal Deposit Insurance
                           Act, 12 U.S.C. (S)1818(e)(4) or (g)(1), all
                           obligations of the Bank under this contract shall
                           terminate as of the effective date of the order, but
                           vested rights of the contracting parties shall not be
                           affected.

                    d.     If the Bank is in default as defined in Section
                           3(x)(1) of the Federal Deposit Insurance Act, 12
                           U.S.C. (S)1813(x)(1), all obligations of the Bank
                           under this Plan shall terminate as of the date of
                           default, but this paragraph shall not affect any
                           vested rights of the contracting parties.

                    e.     Any payments made to an Eligible Employee pursuant to
                           this Plan, or otherwise, are subject to and
                           conditioned upon their compliance with 12 U.S.C.
                           (S)1828(k) and any regulations promulgated
                           thereunder.

             7.     Administrative Provisions.

                    a.     The administrator of the Plan shall be under the
                           supervision of the Board of Directors of the Bank or
                           a committee appointed by the Board of Directors of
                           the Bank (the "Board"). It shall be a principal duty
                           of the Board to see that the Plan is carried out in
                           accordance with its terms, for the exclusive benefit
                           of persons entitled to participate in the Plan
                           without discrimination among them. The Board will
                           have full power to administer the Plan in all of its
                           details subject, however, to the requirements of
                           ERISA. For this purpose, the Board's powers will
                           include, but will not be limited to, the following
                           authority, in addition to all other powers provided
                           by this Plan: (i) to make and enforce such rules and
                           regulations as it deems necessary or proper for the
                           efficient administration of the Plan; (ii) to
                           interpret the Plan, its interpretation thereof in
                           good faith to be final and conclusive on all persons
                           claiming benefits under the Plan; (iii) to decide all
                           questions concerning the Plan and the eligibility of
                           any person to

<PAGE>

                           participate in the Plan; (iv) to compute the amount
                           of severance benefits payable to any Eligible
                           Employee or other person in accordance with the
                           provisions of the Plan, and to determine the person
                           or persons to whom such benefits will be paid; (v) to
                           authorize severance benefits; (vi) to appoint such
                           agents, counsel, accountants, consultants and
                           actuaries as may be required to assist in
                           administering the Plan; and (vii) to allocate and
                           delegate its responsibilities under the Plan and to
                           designate other persons to carry out any of its
                           responsibilities under the Plan, any such allocation,
                           delegation or designation to be by written instrument
                           and in accordance with Section 405 of ERISA, if
                           applicable.

                    b.     The Board will be a "named fiduciary" for purposes of
                           Section 402(a)(1) of ERISA with authority to control
                           and manage the operation and administration of the
                           Plan, and will be responsible for complying with all
                           of the reporting and disclosure requirements of Part
                           1 of Subtitle B of Title I of ERISA.

             8.     Claims and Review Procedures.

                    a.     If any person believes he is being denied any rights
                           or benefits under the Plan, such person may file a
                           claim in writing with the Board. If any such claim is
                           wholly or partially denied, the Board will notify
                           such person of its decision in writing. Such
                           notification will be written in a manner calculated
                           to be understood by such person and will contain (i)
                           specific reasons for the denial, (ii) specific
                           reference to pertinent Plan provisions, (iii) a
                           description of any additional material or information
                           necessary for such person to perfect such claim and
                           an explanation of why such material or information is
                           necessary and (iv) information as to the steps to be
                           taken if the person wishes to submit a request for
                           review. Such notification will be given within 90
                           days after the claim is received by the Board (or
                           within 180 days, if special circumstances require an
                           extension of time for processing the claim, and if
                           written notice of such extension and circumstances is
                           given to such person within the initial 90 day
                           period). If such notification is not given within
                           such period, the claim will be considered denied as
                           of the last day of such period and such person may
                           request a review of his claim.

                    b.     Within 60 days after the date on which a person
                           receives a written notice of a denied claim (or, if
                           applicable, within 60 days after the date on which
                           such denial is considered to have occurred) such
                           person (or his duly authorized representative) may
                           (i) file a written request with the Board for a
                           review of his denied claim and of pertinent documents
                           and (ii) submit written issues and comments to the
                           Board. The Board will notify such person of its
                           decision in writing. Such notification will be
                           written in a manner calculated to be understood by
                           such person and will contain

<PAGE>

                           specific reasons for the decision as well as specific
                           references to pertinent Plan provisions. The decision
                           on review will be made within 60 days after the
                           request for review is received by the Board (or
                           within 120 days, if special circumstances require an
                           extension of time for processing the requests such as
                           an election by the Board to hold a hearing, and if
                           written notice of such extension and circumstances is
                           given to such person within the initial 60 day
                           period). If the decision on review is not made within
                           such period, the claim will be considered denied.

             9.     Governing Law. This plan shall be governed by the laws of
the State of Tennessee.

             10.    Termination or Amendment. This plan may be amended or
terminated at any time, in the full discretion of the Board of Directors of the
Bank, prior to the Change in Control. This plan may not be terminated or amended
at the time of or after the occurrence of the Change in Control.

             Having been duly adopted by the Board of Directors of the Bank,
this Plan is executed by a duly authorized officer of the Bank on this _____ day
of ______________________, 2003.

ATTEST:                             JEFFERSON FEDERAL BANK


________________________________    By:_________________________________________
                                       For the Entire Board of Directors