<PAGE>   1
                                                                    EXHIBIT 10.4

                                 TERM LOAN NOTE

$18,333,333.00                                               Birmingham, Alabama
                                                                    May 10, 2001


         FOR VALUE RECEIVED, PROASSURANCE CORPORATION, a Delaware corporation
(the "Borrower"), promises to pay to the order of AMSOUTH BANK, an Alabama
banking corporation (the "Lender"), the principal sum of EIGHTEEN MILLION THREE
HUNDRED THIRTY-THREE THOUSAND THREE HUNDRED THIRTY-THREE AND NO/100 DOLLARS
($18,333,333.00), or such lesser amount as shall equal the aggregate unpaid
principal amount of all Borrowings disbursed by the Lender to the Borrower
pursuant to Section 2.2 of the Credit Agreement referred to below, on the dates
and in the amounts provided for Borrowings under the Term Loan Facility pursuant
to the Credit Agreement. The Borrower promises to pay interest on the unpaid
principal amount of this note on the dates and at the rate or rates provided for
in the Credit Agreement. Interest on any overdue principal of and, to the extent
permitted by law, overdue interest on the principal amount hereof shall bear
interest at the Default Rate, as provided for in the Credit Agreement. All such
payments of principal and interest shall be made in lawful money of the United
States in Federal or other immediately available funds at the office of
SouthTrust Bank, as Administrative Agent, 420 North 20th Street, Birmingham,
Alabama 35203, or such other address as may be specified from time to time
pursuant to the Credit Agreement.

         All Borrowings under the Term Loan Facility funded by the Lender, the
respective maturities thereof, the interest rates from time to time applicable
thereto, and all repayments of the principal thereof shall be recorded by the
Lender and, prior to any transfer hereof, endorsed by the Lender on the schedule
attached hereto, or on a continuation of such schedule attached to and made a
part hereof; provided that the failure of the Lender to make any such
recordation or endorsement shall not affect the obligations of the Borrower
hereunder or under the Credit Agreement.

         This note is one of the Term Loan Notes referred to in the Credit
Agreement dated as of May 10, 2001 among the Borrower, the Lenders listed on the
signature pages thereof, and SouthTrust Bank, as Lead Arranger, Syndication
Agent, and Administrative Agent, and Bank of America, National Association, as
Co-Arranger (as the same may be amended and modified from time to time, the
"Credit Agreement"). Terms defined in the Credit Agreement are used herein with
the same meanings. Reference is made to the Credit Agreement for provisions for
the optional and mandatory prepayment and the repayment hereof and the
acceleration of the maturity hereof, as well as the obligation of the Borrower
to pay all costs of collection, including reasonable attorneys fees, in the
event this note is collected by law or through an attorney at law. This note is
entitled to the security provided by the Credit Agreement and the Collateral
Documents referred to therein.

         The Borrower hereby waives presentment, demand, notice of dishonor,
protest, notice of demand, protest and nonpayment and any other notice required
by law relative hereto, except to the extent as otherwise may be expressly
provided for in the Credit Agreement.

         THIS NOTE IS MADE AND DELIVERED IN THE STATE OF ALABAMA AND SHALL BE
CONSTRUED IN ACCORDANCE WITH AND GOVERNED BY THE LAWS OF THE STATE OF ALABAMA
WITHOUT REFERENCE TO THE CONFLICTS OR CHOICE OF LAW PRINCIPLES THEREOF.

        [Remainder of Page Intentionally Blank. Signature Page Follows.]


                                        1

<PAGE>   2



         IN WITNESS WHEREOF, the Borrower has caused this note to be duly
executed, under seal, by its duly authorized officer as of the day and year
first above written.


                                        PROASSURANCE CORPORATION


                                        By:  /s/ A. Derrill Crowe
                                            ----------------------------------
                                            Name:   A. Derrill Crowe
                                            Title:  Chairman

[CORPORATE SEAL]


                                        2

<PAGE>   3


                                                                      SCHEDULE A
                                                               TO TERM LOAN NOTE



                         LOANS AND PAYMENTS OF PRINCIPAL

<TABLE>
<CAPTION>
     DATE             BASE RATE BORROWING             AMOUNT             AMOUNT OF            MATURITY             NOTATION
                       OR EURODOLLAR RATE          OF BORROWING          PRINCIPAL              DATE               MADE BY
                           BORROWING                                       REPAID
<S>                 <C>                          <C>                  <C>                 <C>                 <C>

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------

-------------       ------------------------     -----------------    ---------------     ----------------    ------------------
</TABLE>

                                        3