<PAGE>
                                                                    EXHIBIT 10.4

                           CHANGE OF CONTROL AGREEMENT

      THIS AGREEMENT (this "Agreement") dated as of the 21st day of July, 2004
(the "Effective Date") is made by and between FIRST NATIONAL BANKSHARES OF
FLORIDA, INC., a Florida corporation (the "Company"), and Kevin C. Hale
("Executive").

                              W I T N E S S E T H:

      WHEREAS, the Board of Directors of the Company (the "Board") believes that
it is in the best interest of the Company and its shareholders to assure that
the Company will have the continued dedication of Executive, notwithstanding the
possibility, threat or occurrence of a Change of Control (as defined below) of
the Company; that it is imperative to diminish the inevitable distraction of
Executive by virtue of the personal uncertainties and risks created by a pending
or threatened Change of Control and to encourage Executive's full attention and
dedication to the Company currently and in the event of any threatened or
pending Change of Control; and that it is appropriate to provide Executive with
compensation and benefits arrangements upon a Change of Control which ensure
that the compensation and benefits expectations of Executive will be satisfied
and which are competitive with those of other corporations in the financial
services industry;

      WHEREAS, in order to accomplish the foregoing objectives, the Company and
Executive desire to enter into this Agreement which, among other things,
provides for the payment of compensation and benefits payable to Executive if
Executive's employment terminates in certain circumstances following a Change of
Control of the Company;

      WHEREAS, the Company and Executive are also entering into an Employment
Agreement dated as of the Effective Date (the "Employment Agreement") pursuant
to which Executive agrees, among other things, that following the termination of
Executive's employment with the Company, Executive will be restricted from
competing with the Company or soliciting the Company's employees and customers;

      WHEREAS, it is a condition of Executive's willingness to enter into the
Employment Agreement that the Company enter into this Agreement;

      NOW THEREFORE, in consideration of the premises and mutual covenants
contained herein and in the Employment Agreement, and intending to be legally
bound hereby, the parties hereto agree as follows:

      1. Definitions. As used herein, the following terms shall have the
meanings set forth below:

      (a) "Change of Control" shall mean the occurrence of any of the following
events:

            (i)   An acquisition (other than directly from the Company) of any
                  voting securities of the Company (the "Voting Securities") by
                  any "Person" (as the term person is used for the purposes of
                  Section 13(d) or 14(d) of the Securities Exchange Act of 1934,
                  as amended as of Effective Date (the "Exchange Act"))
                  immediately after which such Person has Beneficial Ownership
                  (within the meaning of Rule l3d-3 promulgated under the
                  Exchange Act) of greater than 15% of the combined voting power
                  of the Company's then-outstanding Voting Securities; provided,
                  however, in determining whether or not a Change of Control has
                  occurred,

<PAGE>

                  Voting Securities which are acquired in a "Non-Control
                  Acquisition" (as hereinafter defined) shall not constitute a
                  Change of Control. A "Non-Control Acquisition" shall mean an
                  acquisition by (A) the Company, (B) any employee benefit plan
                  (or related trust) sponsored or maintained by the Company or
                  any affiliate of the Company or (C) any Person in connection
                  with a Non-Control Transaction (as hereinafter defined);

            (ii)  Individuals who, as of the date hereof, constitute the Board
                  (the "Incumbent Board") cease for any reason to constitute at
                  least a majority of the Board; provided, however, that any
                  individual becoming a director subsequent to the date hereof
                  whose election, or nomination for election by the Company's
                  shareholders, was approved by a vote of at least a majority of
                  the directors then comprising the Incumbent Board shall be
                  considered as though such individual were a member of the
                  Incumbent Board, but excluding, for this purpose, any such
                  individual whose initial assumption of office occurs as a
                  result of an actual or threatened election contest with
                  respect to the election or removal of directors or other
                  actual or threatened solicitation of proxies or consents by or
                  on behalf of a Person other than the Board; or

            (iii) The consummation of:

                  (A) a merger, consolidation, reorganization or similar
                  corporate transaction with or into the Company or in which
                  securities of the Company are issued, unless such transaction
                  is a "Non-Control Transaction." A "Non-Control Transaction" is
                  a merger, consolidation, reorganization or similar corporate
                  transaction with or into the Company or in which securities of
                  the Company are issued where:

                        I. the shareholders of the Company immediately before
                        such transaction own, directly or indirectly, a majority
                        of the combined voting power of the outstanding voting
                        securities of the corporation resulting from such
                        transaction (the "Surviving Corporation") in
                        substantially the same proportion as their ownership of
                        the Voting Securities immediately before such
                        transaction,

                        II. the individuals who were members of the Incumbent
                        Board immediately prior to the execution of the
                        agreement providing for such transaction constitute at
                        least a majority of the members of the board of
                        directors of the Surviving Corporation or a corporation
                        owning directly or indirectly a majority of the combined
                        voting power of the outstanding securities of the
                        Surviving Corporation, and

                        III. no person other than (x) the Company, (y) any
                        subsidiary of the Company, or (z) any employee benefit
                        plan (or any trust forming a part thereof) maintained by
                        the Company immediately prior to such transaction owns a
                        majority of the combined voting power of the Surviving
                        Corporation's then-outstanding voting securities; or

                                       2

<PAGE>

                  (B) approval by the shareholders of the Company of a complete
                  liquidation or dissolution of the Company; or

                  (C) the sale or other disposition of all or substantially all
                  of the assets of the Company to any Person.

                  Notwithstanding the foregoing, a Change of Control shall not
                  be deemed to occur solely because any Person (the "Subject
                  Person") acquired Beneficial Ownership of more than the
                  permitted amount of the outstanding Voting Securities as a
                  result of the acquisition of Voting Securities by the Company
                  which, by reducing the number of Voting Securities
                  outstanding, increases the proportional number of shares
                  Beneficially Owned by the Subject Person, provided that if a
                  Change of Control would occur (but for the operation of this
                  sentence) and after such acquisition of Voting Securities by
                  the Company, the Subject Person becomes the Beneficial Owner
                  of any additional Voting Securities, then a Change of Control
                  shall occur.

      (b) "Cause" shall have the meaning given such term in the Employment
      Agreement.

      (c) "Good Reason" shall mean:

            (i)   removal of Executive from the position he held immediately
                  prior to the Change of Control (by reason other than death,
                  disability or Cause);

            (ii)  a substantial alteration in the nature or status of
                  Executive's responsibilities following the Change in Control
                  which renders Executive's position to be of less dignity,
                  responsibility or scope;

            (iii) a reduction by the Company in Executive's target annual bonus
                  under the Company's Executive Incentive Compensation Plan, or
                  any successor plan, or the establishment of performance goals
                  under such plan that are unreasonable in comparison to the
                  Company's historical practice under such plan prior to the
                  Change of Control;

            (iv)  the Company requiring Executive to be based anywhere other
                  than the Company's principal executive offices in the city in
                  which Executive is principally located immediately prior to
                  the Change of Control, except for required travel on the
                  Company's business to an extent substantially consistent with
                  Executive's business travel obligations prior to the Change of
                  Control;

            (v)   any material reduction by the Company of the benefits enjoyed
                  by Executive prior to the Change in Control under any of the
                  Company's pension, retirement, profit sharing, savings, life
                  insurance, medical, health and accident, disability or other
                  employee benefit plans, programs or arrangements, or the
                  taking of any action by the Company which would directly or
                  indirectly materially reduce any of such benefits or deprive

                                       3

<PAGE>

                  Executive of any material fringe benefits made available to
                  Executive prior to the Change in Control; or

            (vi)  any material breach by the Company of its obligations
                  contained in this Agreement or the Employment Agreement.

      (d) "Salary and Benefits Continuation" shall mean:

            (i)   payment of an amount of cash equal to the one-twelfth of the
                  Severance Period multiplied by Executive's base salary at the
                  rate of base salary per annum in effect immediately prior to
                  the Change of Control or on the date of the termination of
                  Executive's employment, whichever is higher (such amount shall
                  be payable in equal installments, on a basis consistent with
                  the Company's payroll practices, over a number of months equal
                  to the Severance Period, unless Executive has notified the
                  Company in writing not less than six months prior to the
                  Triggering Event of Executive's election to receive such
                  amount in one lump sum payment, in which event the Company
                  shall make such lump sum payment within 30 days following the
                  Triggering Event);

            (ii)  payment of an amount of cash equal to the one-twelfth of the
                  Severance Period multiplied by the greater of (x) the average
                  annual incentive bonus payment earned by Executive under the
                  Company's Executive Incentive Compensation Plan (or any
                  successor plan) in respect of the three most recent complete
                  fiscal years of the Company preceding the date of the
                  termination of Executive's employment or (y) the target
                  incentive bonus award under the Company's Executive Incentive
                  Compensation Plan (or any successor plan) for the year in
                  which the Change of Control occurs or the year in which the
                  termination of Executive's employment occurs, whichever is
                  higher (such amount shall be payable in equal installments, on
                  a basis consistent with the Company's payroll practices, over
                  a number of months equal to the Severance Period, unless
                  Executive has notified the Company in writing not less than
                  six months prior to the Triggering Event of Executive's
                  election to receive such amount in one lump sum payment, in
                  which event the Company shall make such lump sum payment
                  within 30 days following the Triggering Event);

            (iii) provision to Executive and his eligible dependents of medical,
                  long-term disability, dental and life insurance coverage, to
                  the extent such coverage was in effect immediately prior to
                  the Change of Control, for a period of months equal to the
                  Severance Period;

            (iv)  contribution or crediting by the Company to Executive's
                  account under the Company's defined contribution retirement
                  plans (currently, the Company's Salary Savings Plan and ERISA
                  Excess Profit Sharing and Lost Match Plan) an amount of cash
                  equal to the amount that the Company would have contributed or
                  credited to such plans (including both profit-sharing
                  contributions and Company matching contributions in respect of
                  Executive's

                                       4

<PAGE>

                  contributions to the plan) had Executive continued to be
                  employed by the Company for an additional period of months
                  equal to Severance Period at an annual compensation equal to
                  the sum of (x) Executive's base salary immediately prior to
                  the Change of Control or at the time of the termination of
                  Executive's employment, whichever is higher and (y) the
                  greater of (A) the average annual incentive bonus payment
                  earned by Executive under the Company's Executive Incentive
                  Compensation Plan (or any successor plan) in respect of the
                  three most recent complete fiscal years of the Company
                  preceding the date of the termination of Executive's
                  employment or the date of the Change of Control, whichever is
                  higher or (B) the target incentive bonus award under the
                  Company's Executive Incentive Compensation Plan (or any
                  successor plan) for the year in which the Change of Control
                  occurs or the year in which the termination of Executive's
                  employment occurs, whichever is higher; (and assuming for this
                  purpose that Executive made the maximum permissible
                  contributions to such plans during such period), such
                  contributions being deemed to be made immediately prior to the
                  termination of Executive's employment; and

            (v)   reimbursement to Executive of costs incurred by Executive for
                  outplacement services in the 24-month period following
                  termination of Executive's employment, not to exceed 15% of
                  Executive's base salary immediately prior to the Change of
                  Control or at the time of the termination of Executive's
                  employment, whichever is higher.

      (e) "Severance Period" shall mean the lesser of (x) 36 and (y) the number
      of full months between the date of termination of Executive's employment
      and the date on which Executive would attain the age of 65.

      (f) "Triggering Event" shall mean the occurrence of one of the following
      events:

            (i)   the termination of Executive's employment with the Company by
                  the Company other than for Cause prior to a Change of Control,
                  and the reasonable demonstration by Executive that such
                  termination of employment (x) was at the request of a third
                  party who had taken steps reasonably calculated to effect a
                  Change of Control, or (y) otherwise arose in connection with
                  or in anticipation of a Change of Control; or

            (ii)  at any time during the 24-month period commencing on the date
                  of a Change of Control, either (x) the Company terminates
                  Executive's employment other than for Cause or (y) Executive
                  terminates his employment for Good Reason.

      2. Term. The term of this Agreement shall commence on the Effective Date
and shall expire on the earlier of (i) the date of the termination of
Executive's employment by the Company for any reason prior to a Change of
Control, unless such termination is a Triggering Event under Section 1(f)(i)
hereof; or (ii) the third anniversary of the date on which the Company gives
written notice to Executive of the Company's decision to terminate both this
Agreement and any and all employment agreements in effect between the Company
Executive.

                                       5

<PAGE>

      3. Change in Control Payments and Benefits.

      (a) Upon the occurrence of a Triggering Event:

            (i) the Company shall provide Salary and Benefits Continuation to
            the Executive; provided, however, that in the event medical,
            long-term disability, dental and life insurance benefits cannot be
            provided under appropriate Company group insurance policies pursuant
            to clause (iii) of the definition of "Salary and Benefits
            Continuation," an amount equal to the premium necessary for
            Executive to purchase directly the same level of coverage in effect
            immediately prior to the termination of employment shall be added to
            the Company's payments to Executive pursuant to this Section 3. If
            Executive is required to pay income or other taxes on any medical,
            long-term disability, dental or life insurance benefits provided or
            paid to Executive pursuant to this Section 3, then the Company shall
            pay to Executive an amount of cash sufficient to "gross-up" such
            benefits or payments so that Executive's "net" benefits received
            under this Section 3 are not diminished by any such taxes that are
            imposed with respect to the same or the Company's gross-up hereunder
            with respect to such taxes;

            (ii) all equity compensation awards granted to Executive by the
            Company (e.g., all stock options and shares of restricted stock),
            will immediately become fully vested and fully exercisable;

            (iii) notwithstanding anything to the contrary contained in the
            Company's Basic Retirement Plan (the "BRP"), (i) Executive's
            "Credited Service" under the BRP shall be deemed for all purposes to
            be increased by an amount equal to the one-twelfth of the Severance
            Period; provided, that Executive's Credited Service shall be deemed
            to be a minimum of 10.0 years, and (ii) Executive's "Compensation"
            under the BRP for each such additional year of Credited Service for
            purposes of the BRP shall be deemed to be an amount equal to the sum
            of (x) Executive's base salary at the rate of base salary per annum
            in effect immediately prior to the Change of Control or on the date
            of the termination of Executive's employment, whichever is higher,
            and (y) the greater of (A) the average annual incentive bonus
            payment earned by Executive under the Company's Executive Incentive
            Compensation Plan (or any successor plan) in respect of the three
            most recent complete fiscal years of the Company preceding the date
            of the termination of Executive's employment or the date of the
            Change of Control, whichever is higher or (B) the target incentive
            bonus award under the Company's Executive Incentive Compensation
            Plan (or any successor plan) for the year in which the Change of
            Control occurs or the year in which the termination of Executive's
            employment occurs, whichever is higher;

            (iv) payments to Executive pursuant to the BRP shall begin in
            accordance with BRP election procedures; provided, however that
            payments shall begin no earlier than the later to occur of (x) the
            first month following Executive's attainment of age 55 or (y) a
            number of months following the termination of the employment of
            Executive equal to the Severance Period;

                                       6

<PAGE>

            (v) as to Executive, Section A-1 of Appendix A of the BRP shall be
            amended to read as follows:

                  "A-1. Factors for determining early retirement benefits

<TABLE>
<CAPTION>
     AGE               FACTOR
     ---               ------
<S>                    <C>
55 and above           1.000
</TABLE>

      (b) Executive's subsequent employment, death or disability following
      Executive's termination of employment in connection with a Change of
      Control shall not affect the Company's obligation to provide to Executive
      the payments and benefits pursuant to this Section 3. Executive shall not
      be required to mitigate the amount of any payment provided for in this
      Section 3 by seeking employment or otherwise. The rights to the payments
      and benefits pursuant to this Section 3 shall be in addition to any other
      benefits to which Executive may be entitled under any other agreement or
      compensation plan, program or arrangement of the Company; provided, that
      Executive shall not be entitled to any separate or additional severance
      payments pursuant to any severance plan of the Company then in effect and
      generally applicable to similarly situated Executives.

      4. Employment at Will. Subject to the provisions of any other agreement
between Executive and the Company (including the Employment Agreement),
Executive shall remain an employee at will and nothing herein shall confer upon
Executive any right to continued employment and shall not affect the right of
the Company to terminate Executive for any reason not prohibited by law;
provided, however, that any such removal shall be without prejudice to any
rights Executive may have to receive payments and benefits pursuant to Section 3
hereof.

      5. Gross-Up for Excess Parachute Payments. In the event that any payment
or benefit, or any combination of payment or benefits, to Executive hereunder is
determined to be an "excess parachute payment" pursuant to Section 280G of the
Internal Revenue Code of 1986, as amended (the "Code"), then the Company, at the
time such determination becomes final, shall pay to Executive an amount (the
"gross-up payment") which would equal, after deducting all state and federal
income taxes incurred by the Executive with respect to receipt of the gross-up
payment, the excise tax, if any, imposed on Executive pursuant to Section 4999
of the Code.

      6. Resolution of Differences Over Breaches of Agreement. Except as
otherwise provided herein, in the event of any controversy, dispute or claim
arising out of, or relating to this Agreement, or the breach thereof, or arising
out of any other matter relating to Executive's employment with the Company or
the termination of such employment, the parties may seek recourse only for
temporary or preliminary injunctive relief to the courts having jurisdiction
thereof and if any relief other than injunctive relief is sought, the Company
and Executive agree that such underlying controversy, dispute or claim shall be
settled by arbitration conducted in Naples, Florida in accordance with this
Section 6 of this Agreement and the Commercial Arbitration Rules of the American
Arbitration Association ("AAA"). The matter shall be heard and decided, and
awards rendered by a panel of three arbitrators (the "Arbitration Panel"). The
Company and Executive shall each select one arbitrator from the AAA National
Panel of Commercial Arbitrators (the "Commercial Panel") and those two
arbitrators shall select a third arbitrator; provided, however, that in the
event the two arbitrators cannot agree on a third arbitrator, the AAA shall
select a third arbitrator from the

                                       7

<PAGE>

Commercial Panel. The award rendered by the Arbitration Panel shall be final and
binding as between the parties hereto and their heirs, executors,
administrators, successors and assigns, and judgment on the award may be entered
by any court having jurisdiction thereof. The Company and Executive will each
bear their own costs for legal representation in any arbitration, except that
the Arbitration Panel will have the authority to award all remedies provided by
applicable law, including recovery of attorney fees when so provided by
applicable law. The Company will pay all arbitrators' fees and other
administrative fees in connection with any arbitration hereunder; provided,
however, that the Arbitration Panel may require all or a portion of such fees
and expenses to be paid by Executive in the event the Arbitration Panel
determines that Executive's position in the arbitration proceeding was without
merit.

      7. Release. Executive hereby acknowledges and agrees that prior to
Executive's or his dependents' right to receive from the Company any
compensation or benefit to be paid or provided to him or his dependents pursuant
to Section 3 of this Agreement, Executive may be required by the Company, in its
sole discretion, to execute a release in the form of Exhibit A hereto.

      8. Waiver. The waiver by a party hereto of any breach by the other party
hereto of any provision of this Agreement shall not operate or be construed as a
waiver of any subsequent breach by a party hereto.

      9. Assignment. This Agreement shall be binding upon and inure to the
benefit of the successors and assigns of the Company. The Company shall be
obligated to require any successor (whether direct or indirect, by purchase,
merger, consolidation or otherwise) to all or substantially all of the Company's
business or assets, by a written agreement in form and substance satisfactory to
Executive, to expressly assume and agree to perform this Agreement in the same
manner and to the same extent that the Company would be required to perform if
no succession had taken place. This Agreement shall inure to the extent provided
hereunder to the benefit of and be enforceable by Executive or his legal
representatives, executors, administrators, successors, heirs, distributees,
devisees and legatees. Executive may not delegate any of his duties,
responsibilities, obligations or positions hereunder to any person and any such
purported delegation by him shall be void and of no force and effect with
respect to matters relating to his employment and termination of employment.
Without limiting the foregoing, Executive's rights to receive payments and
benefits hereunder shall not be assignable or transferable, other than a
transfer by Executive's will or by the laws of descent and distribution.

      10. Notices. Any notices required or permitted to be given under this
Agreement shall be sufficient if in writing, and if personally delivered or when
sent by first class certified or registered mail, postage prepaid, return
receipt requested -- in the case of Executive, to his residence address as set
forth in the books and records of the Company, and in the case of the Company,
to the address of its principal place of business, in care of the Chairman of
the Compensation Committee of the Board -- or to such other person or at such
other address with respect to each party as such party shall notify the other in
writing.

      11. Entire Agreement. This Agreement, along with the Employment Agreement,
contains the entire agreement of the parties concerning the matters set forth
herein and therein all promises, representations, understandings, arrangements
and prior agreements regarding the subject matter hereof, other than those set
forth in the Employment Agreement, are merged herein and superseded hereby. The
provisions of this Agreement may not be amended, modified, repealed, waived,
extended or discharged except by an agreement in writing signed by the party
against whom

                                       8

<PAGE>

enforcement of any amendment, modification, repeal, waiver, extension or
discharge is sought. No person acting other than pursuant to a resolution of the
Board of Directors shall have authority on behalf of the Company to agree to
amend, modify, repeal, waive, extend or discharge any provision of this
Agreement or anything in reference thereto or to exercise any of the Company's
rights to terminate this Agreement.

      12. Construction of Agreement.

      (a) Governing Law. This Agreement shall be governed by and construed under
      the laws of the State of Florida without regard to its conflict of law
      provisions.

      (b) Severability. In the event that any one or more of the provisions of
      this Agreement shall be held to be invalid, illegal or unenforceable, the
      validity, legality or enforceability of the remaining provisions shall not
      in any way be affected or impaired thereby.

      (c) Headings. The descriptive headings of the several paragraphs of this
      Agreement are inserted for convenience of reference only and shall not
      constitute a part of this Agreement.

                            [Signature Page Follows]

                                       9

<PAGE>

      IN WITNESS WHEREOF, the parties hereto have executed this Agreement as of
the date first above written.

                                    "COMPANY"

                                    FIRST NATIONAL BANKSHARES OF FLORIDA, INC.

                                    By: /s/ Gary L. Tice
                                        ----------------------------------------
                                        Gary L. Tice, Chairman and Chief
                                        Executive Officer

                                    "EXECUTIVE"

                                    /s/ Kevin C. Hale
                                    --------------------------------------------
                                    Kevin C. Hale

                                       10

<PAGE>

                                    EXHIBIT A

                                 FORM OF RELEASE

      THIS RELEASE is made by the undersigned ("Executive") in favor of First
National Bankshares of Florida, Inc. [OR SUCCESSOR ENTITY] (the "Company") this
____ day of _____, _____.

      WHEREAS, Executive and the Company are parties to that certain Change of
Control Agreement, dated as of _________, 2004 (the "Change of Control
Agreement"); and

      WHEREAS, pursuant to Section 7 of the Change of Control Agreement, it is a
condition to the Company's obligation to make certain payments and provide
certain benefits to Executive pursuant to the Change of Control Agreement
following the termination of Executive's employment that Executive execute and
deliver to the Company a release in the form hereof.

      NOW, THEREFORE, in consideration of the premises and other good and
valuable consideration, the receipt and sufficiency of which are hereby
acknowledged, Executive hereby releases the Company and any affiliated entities
and their respective officers, directors, shareholders, executives and agents,
from any and all claims, demands, suits, causes of action, damages or expenses
which Executive has had or may have in the future, arising out of Executive's
employment as an executive of the Company and Executive's separation from the
Company, including, without limitation:

      (a)   claims under any and all federal, state or local laws or
            regulations, including, but not limited to any labor, employment or
            benefit laws prohibiting any form of discrimination such as the Fair
            Labor Standards Act, the Age Discrimination in Employment Act, as
            amended by the Older Workers Benefit Protection Act, Title VII of
            the Civil Rights Act of 1964, as amended, the Americans with
            Disabilities Act, and the Civil Rights Act of 1991;

      (b)   any right to recover under any claim that may be filed by the Equal
            Employment Opportunity Commission, or state or local human relations
            commission, or any other federal, state or local governmental
            agency; and

      (c)   any claim that Executive is entitled to any payments or benefits
            other than as expressly set forth in the Change of Control
            Agreement.

      IN WITNESS WHEREOF, intending to be legally bound, Executive has executed
this Release on the date first set forth above.

                                             -----------------------------------
                                             Print Name:
                                                        ------------------------