<PAGE>   1
                                                                   Exhibit 10.23



    AMENDMENT TO CHANGE OF CONTROL AGREEMENT DATED MAY 20, 1997 BETWEEN FIRST
                   PALM BEACH BANCORP, INC. AND ALISSA BALLOT



This Amendment is made effective as of November 18, 1997 by and between First
Palm Beach Bancorp, Inc. (the "Holding Company"), and Alissa Ballot
("Executive"), amending that certain First Palm Beach Bancorp, Inc. Change of
Control Agreement dated as of May 20, 1997 between the Holding Company and
Executive (the "Agreement").

WHEREAS, Executive was promoted to Senior Vice President for Legal Affairs of
First Bank of Florida (the "Association") effective October 21, 1997; and

WHEREAS, in light of such promotion, the Board of Directors of the Holding
Company deems it appropriate to extend certain protections afforded by the
Agreement from two years to three years;

NOW THEREFORE, in consideration of the mutual covenants, terms and conditions
herein set forth, the Agreement is amended, effective on the date hereof, as
follows:

1.       Paragraph 3(a) is deleted in its entirety and the following substituted
         therefor:

         (a)      If Executive is terminated by the Holding Company or
                  Association other than in a Termination for Cause pursuant to
                  paragraph 2(c) hereof, or Executive terminates employment for
                  Good Reason, in either case after a Change of Control of the
                  Holding Company as defined in paragraph 2(b) hereof, then in
                  lieu of any further salary payments to Executive for periods
                  subsequent to the date of termination, the Holding Company
                  shall pay as severance to Executive an amount equal to three
                  (3) times the sum of (A) the higher of Executive's base salary
                  in effect immediately prior to the occurrence of the event or
                  circumstance upon which the Notice of Termination is based and
                  Executive's annual base salary in effect immediately prior to
                  the Change of Control of the Holding Company, plus (B) the
                  higher of the highest annual bonus or incentive payment earned
                  by or accrued in respect of Executive in respect of any of the
                  two years immediately preceding that in which the Date of
                  Termination occurs or the highest annual bonus or incentive
                  payment so earned in respect of any of the two years
                  immediately preceding that in which the Change of Control of
                  the Holding Company occurs. Such payment shall be made in a
                  lump sum within five days of the date of termination of
                  Executive's employment.

2.       Paragraph 3(b) is deleted in its entirety and the following
         substituted therefor:

         (b)      Upon the occurrence of a Change of Control of the Holding
                  Company followed at any time during the term of this Agreement
                  by Executive's voluntary (for Good Reason) or involuntary
                  termination of employment, other than a Termination for Cause,
                  the Holding Company shall, for three years or until Executive
                  obtains employment which provides substantially similar
                  benefits, provide Executive and anyone entitled to claim under
                  or through Executive all benefits under any life or other
                  insurance or death benefit plan, medical, group
                  hospitalization, dental, disability insurance or other future
                  or present similar group employee benefit plan or program of
                  the Holding Company or Association for which executive
                  officers are eligible, to the same extent as if Executive had
                  continued to be an employee of the Holding Company or
                  Association during such period and such benefits shall, to the
                  extent not paid under any such plan or program, be paid by the
                  Holding Company. The payments and benefits described in the
                  preceding sentence shall be paid to Executive's beneficiaries
                  by testate or intestate succession in the event of Executive's
                  death during the period during which such payments and
                  benefits are being provided. Executive's "qualifying event"
                  for purposes of continuation coverage under the Consolidated
                  Budget Reconciliation Act ("COBRA") shall occur at the
                  expiration of such three year period.



<PAGE>   2


3.       Executive's address for purposes of the Agreement shall be amended to:
         80 Monterey Pointe Drive, Palm Beach Gardens, Florida 33418.

All other terms and conditions of the Agreement shall remain in full force and
effect.

IN WITNESS WHEREOF, First Palm Beach Bancorp, Inc. has caused this Amendment to
be executed by its duly authorized officer, and Executive has executed this
Amendment, as of the day and year first above written.




ATTEST:                                     FIRST PALM BEACH BANCORP, INC.




                                            By: /s/ LOUIS O. DAVIS, JR.
--------------------------------               --------------------------------


WITNESS



                                               /s/ ALISSA BALLOT
--------------------------------               --------------------------------
                                               Executive