<PAGE>

                                                                    EXHIBIT 99.4

                                 PROMISSORY NOTE


$3,000,000.00

                                                               December 31, 2004

Daniel R. O'Neal ("Borrower") promises to pay to the order of American
Physicians Assurance Corporation ("APA"), in lawful money of the United States
of America, at its principal office in East Lansing, Michigan or wherever else
APA may specify, the sum of Three Million and No/100 Dollars ($3,000,000.00),
with interest on the unpaid principal balance at the rate and on the terms
provided in this Promissory Note (including all renewals, extensions or
modifications hereof, this "Note").

USE OF PROCEEDS. Borrower shall use the proceeds of the loan evidenced by this
Note to acquire shares of Physicians Insurance Company, a Florida stock
insurance corporation ("PIC"), owned by APA.

INTEREST RATE. Interest shall accrue on the unpaid principal balance of this
Note from the date hereof at 8% ("Interest Rate").

DEFAULT RATE. In addition to all other rights contained in this Note, if a
Default (as defined below) occurs and as long as a Default continues, all
outstanding Obligations shall bear interest at the Interest Rate plus 3%
("Default Rate"). The Default Rate shall also apply from acceleration until the
Obligations (as defined below) or any judgment thereon are paid in full.

INTEREST AND FEE(S) COMPUTATION (ACTUAL/360). Interest and fees, if any, shall
be computed on the basis of a 360-day year for the actual number of days in the
applicable period ("Actual/360 Computation"). The Actual/360 Computation
determines the annual effective interest yield by taking the stated (nominal)
rate for a year's period and then dividing said rate by 360 to determine the
daily periodic rate to be applied for each day in the applicable period.
Application of the Actual/360 Computation produces an annualized effective rate
exceeding the nominal rate.

REPAYMENT TERMS. This Note shall be due and payable in 84 consecutive monthly
payments of principal plus accrued interest, commencing on January 31, 2005, and
continuing on the same day of each month thereafter until fully paid in
accordance with the attached repayment schedule. In any event, subject to
"Remedies Upon Default," all unpaid principal and accrued interest shall be due
and payable on the due date of the 84th payment.

APPLICATION OF PAYMENTS. Monies received by APA from any source for application
toward payment of the Obligations shall be applied to accrued interest and then
to principal. If a Default occurs, monies may be applied to the Obligations in
any manner or order deemed appropriate by APA.

If any payment received by APA under this Note or the other Loan Documents is
rescinded, avoided or for any reason returned by APA because of any adverse
claim or threatened action, the returned payment shall remain payable as an
obligation of all persons liable under this Note or other Loan Documents as
though such payment had not been made.

DEFINITIONS. LOAN DOCUMENTS. The term "Loan Documents", as used in this Note,
refers to all documents executed in connection with or related to the loan
evidenced by this Note and may include, without limitation, this Note, the Stock
Pledge Agreement between Borrower and APA of even date herewith (the "Stock
Pledge Agreement"), and any renewals or modifications, whenever any of the
foregoing are executed. OBLIGATIONS. The term "Obligations", as used in this
Note and the other Loan Documents, refers to any and all indebtedness and other
obligations under this Note and all other obligations under any other Loan
Documents. CERTAIN OTHER TERMS. All terms that are used but not otherwise
defined in any of the Loan Documents shall have the definitions provided in the
Uniform Commercial Code.


<PAGE>

SECURITY. The Obligations are secured by the security interest granted to APA by
Borrower pursuant to the Stock Pledge Agreement.

LATE CHARGE. If any payments are not timely made, Borrower shall also pay to APA
a late charge equal to 5% of each payment past due for 5 or more days.

Acceptance by APA of any late payment without an accompanying late charge shall
not be deemed a waiver of APA's right to collect such late charge or to collect
a late charge for any subsequent late payment received. Any late charge paid
shall be treated as liquidated damages and not as a penalty, to compensate APA
for the expenses of administering the Default (as defined below).
Notwithstanding any other provision of this Note, APA's acceptance of the full
amount of any late payment with the accompanying late charge shall be deemed to
cure the Default ascribed to such delinquent payment.

ATTORNEYS' FEES AND OTHER COLLECTION COSTS. Borrower shall pay all of APA's
reasonable expenses incurred to enforce or collect any of the Obligations
including, without limitation, reasonable arbitration, paralegals', attorneys'
and experts' fees and expenses, whether incurred without the commencement of a
suit, in any trial, arbitration, or administrative proceeding, or in any
appellate or bankruptcy proceeding.

USURY. If at any time the effective interest rate under this Note would, but for
this paragraph, exceed the maximum lawful rate, the effective interest rate
under this Note shall be the maximum lawful rate, and any amount received by APA
in excess of such rate shall be applied to principal and then to fees and
expenses, or, if no such amounts are owing, returned to Borrower.

DEFAULT. If any of the following occurs, a default ("Default") under this Note
shall exist: NONPAYMENT; NONPERFORMANCE. The failure of timely payment or
performance of the Obligations under this Note or any other Loan Documents.
FALSE WARRANTY. A warranty or representation made or deemed made in the Loan
Documents or furnished to APA in connection with the loan evidenced by this Note
proves materially false, or if of a continuing nature, becomes materially false.
CROSS DEFAULT. At APA's option, any failure to pay, perform or otherwise comply
with any agreement or obligation under any other loans, contracts or agreements
of Borrower or any Affiliate of Borrower with APA or its affiliates ("Affiliate"
shall have the meaning as defined in 11 U.S.C. Section 101, except that the term
"Borrower" shall be substituted for the term "Debtor" therein), including
without limitation the Stock Pledge Agreement. CESSATION; BANKRUPTCY. The death
of, appointment of a guardian for, appointment of a receiver for, assignment for
the benefit of creditors of, or commencement of any bankruptcy or insolvency
proceeding by or against Borrower or PIC. DEATH; DISABILITY; CESSATION OF
DIRECTOR STATUS. The death or "total disability" of Borrower. For purposes of
this Note, "total disability" refers to a condition resulting from injury or
illness to Borrower which prevents Borrower from performing the duties he or she
has previously performed, and could be reasonably expected to perform, on behalf
of PIC, for a period of 365 consecutive days, and "total disability" shall be
deemed to occur on the first day following the initial 365-day period. MATERIAL
ADVERSE CHANGE. APA determines in good faith, in its sole discretion, that the
prospects for payment or performance of the Obligations are impaired or there
has occurred a material adverse change in the business or prospects of Borrower
or PIC, financial or otherwise.

REMEDIES UPON DEFAULT. If a Default occurs under this Note or any Loan
Documents, APA may at any time thereafter, take any one or more of the following
actions: LIEN. Foreclose its security interest or lien under the Stock Pledge
Agreement without notice. ACCELERATION UPON DEFAULT. Accelerate the maturity of
this Note and, at APA's option, any or all other Obligations, in each case by
giving written notice of the same to Borrower, whereupon this Note and the
accelerated Obligations shall be immediately due and payable; provided, however,
if the Default is based upon a bankruptcy or insolvency proceeding commenced by
or against Borrower, PIC or any guarantor or endorser of this Note, all
Obligations shall automatically and immediately be due and payable. CUMULATIVE.
Exercise any rights and remedies as provided under the Note and other Loan
Documents, or as provided by law or equity.


                                       2
<PAGE>

SET OFF RIGHTS. APA shall have the right at any time to set off any other
indebtedness or obligation that APA owes to Borrower against the Obligations
that are then due and payable.

FINANCIAL AND OTHER INFORMATION. Borrower shall deliver to APA such information
as APA may reasonably request from time to time, including without limitation,
financial statements and information pertaining to Borrower's financial
condition. Such information shall be true, complete, and accurate.

WAIVERS AND AMENDMENTS. No waivers, amendments or modifications of this Note and
other Loan Documents shall be valid unless in writing and signed by an
authorized officer of APA. No waiver by APA of any Default shall operate as a
waiver of any other Default or the same Default on a future occasion. Neither
the failure nor any delay on the part of APA in exercising any right, power, or
remedy under this Note and other Loan Documents shall operate as a waiver
thereof, nor shall a single or partial exercise thereof preclude any other or
further exercise thereof or the exercise of any other right, power or remedy.

Borrower waives presentment, protest, notice of dishonor, demand for payment,
notice of intention to accelerate maturity, notice of acceleration of maturity,
notice of sale and all other notices and formalities of any kind. Further,
Borrower agrees that APA may extend, modify or renew this Note or make a
novation of the loan evidenced by this Note for any period, and grant any
releases, compromises or indulgences with respect to any collateral securing
this Note without notice to or consent of Borrower and without affecting the
liability of Borrower under this Note or any other Loan Document.

MISCELLANEOUS PROVISIONS. ASSIGNMENT. This Note and the other Loan Documents
shall inure to the benefit of and be binding upon the parties and their
respective heirs, legal representatives, successors and assigns. APA's interests
in and rights under this Note and the other Loan Documents are freely
assignable, in whole or in part, by APA. In addition, nothing in this Note or
any of the other Loan Documents shall prohibit APA from pledging or assigning
this Note or any of the other Loan Documents or any interest therein to any
Federal Reserve bank. Borrower shall not assign its rights and interest
hereunder without the prior written consent of APA, and any attempt by Borrower
to assign without APA's prior written consent is null and void. Any assignment
shall not release Borrower from the Obligations. APPLICABLE LAW. This Note and,
unless otherwise provided in any other Loan Document, the other Loan Documents
shall be governed by and construed under the laws of the state of Michigan
without regard to that state's conflict of laws principles. JURISDICTION.
Borrower irrevocably agrees to non-exclusive personal jurisdiction in the state
of Michigan. SEVERABILITY. If any provision of this Note or of the other Loan
Documents shall be prohibited or invalid under applicable law, such provision
shall be ineffective but only to the extent of such prohibition or invalidity,
without invalidating the remainder of such provision or the remaining provisions
of this Note or other such document. NOTICES. Any notices to Borrower shall be
sufficiently given, if in writing and mailed or delivered to Borrower, 2760 NE
23rd Place, Pompano Beach, FL 33062 or such other address as provided hereunder,
and to APA, if in writing and mailed or delivered to APA, 1301 North Hagadorn
Road, East Lansing, MI 48825 or such other address as APA may specify in writing
from time to time. In the event that Borrower changes Borrower's address at any
time prior to the date the Obligations are paid in full, Borrower agrees to
promptly give written notice of said change of address by registered or
certified mail, return receipt requested, all charges prepaid. PLURAL; CAPTIONS.
All references in the Loan Documents to Borrower, guarantor, person, document or
other nouns of reference mean both the singular and plural form, as the case may
be, and the term "person" shall mean any individual or entity. The captions
contained in the Loan Documents are inserted for convenience only and shall not
affect the meaning or interpretation of the Loan Documents.

WAIVER OF JURY TRIAL. TO THE EXTENT PERMITTED BY APPLICABLE LAW, BORROWER BY
EXECUTION HEREOF AND APA BY ACCEPTANCE HEREOF, KNOWINGLY, VOLUNTARILY AND
INTENTIONALLY WAIVE ANY RIGHT THEY MAY HAVE TO A TRIAL BY JURY IN RESPECT OF ANY
LITIGATION BASED ON, OR ARISING OUT OF, UNDER OR IN CONNECTION WITH THIS NOTE,
THE LOAN DOCUMENTS OR ANY AGREEMENT CONTEMPLATED TO BE EXECUTED IN CONNECTION
WITH THIS NOTE, OR ANY COURSE OF CONDUCT, COURSE OF DEALING, STATEMENTS (WHETHER
VERBAL OR WRITTEN) OR ACTIONS OF ANY PARTY WITH RESPECT HERETO. THIS PROVISION
IS A MATERIAL INDUCEMENT TO APA TO ACCEPT THIS NOTE.



                                       3
<PAGE>


BORROWER AND APA AGREE THAT THEY SHALL NOT HAVE A REMEDY OF PUNITIVE OR
EXEMPLARY DAMAGES AGAINST THE OTHER IN ANY DISPUTE AND HEREBY WAIVE ANY RIGHT OR
CLAIM TO PUNITIVE OR EXEMPLARY DAMAGES THEY HAVE NOW OR WHICH MAY ARISE IN THE
FUTURE IN CONNECTION WITH ANY DISPUTE WHETHER THE DISPUTE IS RESOLVED BY
ARBITRATION OR JUDICIALLY.

IN WITNESS WHEREOF, Borrower, on the day and year first above written, has
caused this Note to be executed.





                                    By:  /s/ Daniel R. O'Neal
                                         ------------------------------
                                         Daniel R. O'Neal

                                       4