<PAGE>
 
                                                                    EXHIBIT 10.4

$93,747.00                                                   Birmingham, Alabama
                                                                  April 15, 1999

                                PROMISSORY NOTE
                                ---------------

       FOR VALUE RECEIVED, without grace JOHN H. HOLCOMB, III, (the "Borrower"),
promises to pay to the order of ALABAMA NATIONAL BANCORPORATION, a Delaware
corporation (herein called the "Lender," and together with any subsequent holder
of this note called the "Holder"), in the manner set forth below, the principal
sum of Ninety-Three Thousand Seven Hundred Forty-Seven and 00/100 Dollars
($93,747.00), plus interest at the rate set forth below.

       This Note shall bear interest (computed on an Actual/360 Day Basis) on
the unpaid principal balance hereof, from the date of disbursement until payment
in full or complete forgiveness, whichever occurs first, at the rate per annum
equal to the LIBOR-Based Rate (as defined below) adjusted on each Interest Rate
Determination Date (as defined below). If in the Lender's opinion it is
impossible or impractical to determine the LIBOR-Based Rate for a certain year,
this Note shall bear interest at the Prime Rate until the next Interest Rate
Determination Date. Interest payable hereunder shall be payable on each Interest
Rate Determination Date, commencing April 15, 2000.

       Notwithstanding anything to the contrary contained herein, if the
Borrower remains continually employed by the Lender or one of its subsidiaries
or affiliates (hereinafter referred to as a "Qualifying Employer") as of the
first ten (10) anniversaries of this Note, ten percent (10%) of the original
principal balance ($9,374.70) of this Note shall be forgiven as of each such
anniversary, commencing April 15, 2000. In addition, as long as the Borrower
remains continually employed by a Qualifying Employer, upon the occurrence of
(x) a Change in Control (as hereinafter defined), (y) the Borrower's death or
(z) the Total Disability (as hereinafter defined) of the Borrower, the entire
principal balance then remaining unpaid hereunder, shall be immediately forgiven
in full.

       The Borrower further agrees with the Holder as follows:

       SECTION 1 Rules of Construction. This Note is subject to the rules of
                 ---------------------
construction set forth in the Security Documents.

       SECTION 2 Definitions. As used in this Note, capitalized terms that are
                 -----------
not otherwise defined herein have the meanings defined for them in the Security
Documents and the following terms are defined as follows:

       (a)    Actual/360 Day Basis means a method of computing interest and
              --------------------
other charges on the basis of an assumed year of 360 days for the actual number
of days elapsed, meaning that the interest accrued for each day will be computed
by multiplying the interest rate applicable on that day by the unpaid principal
balance on that day and dividing the result by 360.

       (b)    Business Day means any day, excluding Saturday and Sunday, on
              ------------
which the Lender's main office in Birmingham, Alabama, is open to the public for
carrying on substantially all of its banking business.

       (c)    Change in Control of the Lender means (i) any transaction, whether
              -------------------------------------
by merger, consolidation, asset sale, tender offer, reverse stock split, or
otherwise, which results in the acquisition or beneficial ownership (as such
term is defined under rules and regulations promulgated under the Securities
Exchange Act of 1934, as amended) by any person or entity or any group of
persons or entities acting in concert, of fifty percent (50%) or more of the
outstanding shares of Common Stock of the Lender; or (ii) the sale of all or
substantially all of the assets of the Lender; or (iii) the liquidation of the
Lender.

       (d)    Credit Documents means this Note, the Security Documents and all
              ----------------
other documents now or hereafter executed or delivered in connection with the
transactions contemplated thereby.

       (e)    Default Rate means a rate of interest equal to four percentage
              ------------
points (400 basis points) in excess of the highest interest rate that would
otherwise be payable on the principal indebtedness evidenced by this Note from
time to time in the absence of the existence of a default, or the maximum rate
permitted by law, whichever is less.

       (f)    Event of Default is defined in Section 6. An Event of Default
              ----------------
"exists" if an Event of Default has occurred and is continuing.
<PAGE>
 
       (g)    Interest Rate Determination Date means the fifteenth (15) day of
              --------------------------------
April of each year during the term hereof.

       (h)    LIBOR-Based Rate means a fixed rate of one percent (100 basis
              ----------------
points) in excess of the per annum rate of interest most recently published in
The Wall Street Journal as of the close of business on the date hereof and on
and after the most recent Interest Rate Determination Date (being the rate
quoted for the immediately preceding business day) as the London Interbank
Offered Rate for U.S. dollar deposits having a term of ninety (90) days. The
Lender shall determine the LIBOR-Based Rate on the date hereof and on each
Interest Rate Determination Date.

       (i)    Obligors means the Borrower, each other person executing any
              --------
Security Document as a grantor, (if the Borrower or any such grantor is a
partnership) any general partner thereof, and any other maker, endorser, surety,
guarantor or other person now or hereafter liable for the payment or
performance, in whole or in part, of any of the obligations evidenced by this
Note.

       (j)    Prime Rate means a floating interest rate equal to the rate of
              ----------
interest designated by the Lender from time to time as its "prime rate."

       (k)    Security Documents means the Pledge Agreement dated of even date
              ------------------
herewith executed by the Borrower in favor of the Lender and all other documents
now or hereafter securing or guaranteeing the obligations evidenced by this
Note, or any part thereof.

       (l)    Total Disability means the Borrower's inability, as a result of
              ----------------
illness or injury, to perform the normal duties of the Borrower's employment for
a period of ninety (90) consecutive days.

       SECTION 3 Place and Time of Payments.
                 --------------------------
       (a)    All payments by the Borrower to the Holder under this Note shall

be made in lawful currency of the United States and in immediately available
funds to the Lender at its Main Office in Birmingham, Alabama or at such other
address within the continental United States as shall be specified by the Holder
by notice to the Borrower. Any payment received by the Holder after 2:00 p.m.
(Birmingham, Alabama time) on a Business Day (or at any time on a day that is
not a Business Day) shall be deemed made by the Borrower and received by the
Holder on the following Business Day.

       (b)    The amount payable by the Borrower to the Holder under this Note

or any of the other Credit Documents for which a payment date is expressly set
forth herein or therein shall be payable on the specified due date without
notice or demand by the Holder.

       (c)    Payments that are due on a day that is not a Business Day shall be

payable on the next succeeding Business Day, and any interest payable thereon
shall be payable for such extended time at the specified rate.

       SECTION 4 Default Rate. If an Event of Default exists, this Note shall
                 ------------
bear interest at the Default Rate, until the earlier of (a) such time as all
amounts due hereunder are paid in full or (b) no such Event of Default exists.

       SECTION 5 Security Documents. This Note with interest is secured by and
                 ------------------
entitled to the benefits of the Security Documents. Reference to the Security
Documents is hereby made for all of the provisions thereof. This Note shall be
secured by all security documents that by their terms secure this Note, and all
such documents shall constitute Security Documents.

       SECTION 6 Events of Default. The occurrence of any of the following
                 -----------------
events shall constitute an event of default ("Event of Default") under this Note
(whatever the reason for such event and whether or not it shall be voluntary or
involuntary or be effected by operation of law or pursuant to any Governmental
Requirement): (a) any representation or warranty made in any of the Credit
Documents shall prove to be false or misleading in any material respect as of
the time made; or (b) any report, certificate, financial statement or other
instrument furnished in connection with this Note or any of the other Credit
Documents shall prove to be false or misleading in any material respect as of
the time furnished; or (c) default shall be made in the payment when due of any
of the obligations evidenced by this Note or any part thereof; or (d) the
termination of the Borrower's employment with a Qualifying Employer for any
reason with or without cause, whether voluntary or involuntary, other than the
Borrower's death or Total Disability; or (e) any default or event of default, as
therein defined, shall occur under any of the other Credit Documents (after
giving effect to any applicable notice, grace or cure period specified therein).

                                      -2-
<PAGE>
 
       SECTION 7 Acceleration. If an Event of Default exists that does not
                 ------------
already result in the automatic acceleration of this Note under another Credit
Document, the Holder shall have the right without further notice to the Borrower
to declare the entire unpaid principal balance of the indebtedness evidenced by
this Note, with accrued interest, to be immediately due and payable.
Notwithstanding anything in this Note or any other Security Document to the
contrary, the entire unpaid principal balance of the indebtedness evidenced by
this Note shall be immediately due and payable without written notice or demand,
upon the termination of the Borrower's employment with a Qualifying Employer for
any reason with or without cause, whether voluntary or involuntary, other than
the Borrower's death or Total Disability.

       SECTION 8 Certain Waivers and Agreements by Obligors.
                 ------------------------------------------

       (a)    As to the obligations evidenced by this Note, each Obligor
severally (1) waives demand, presentment, protest, notice of protest, suit and
all other requirements necessary to hold liable such Obligor or any of the other
Obligors; (2) waives all exemptions of personal property secured to any Obligor
under the Constitution and laws of the State of Alabama or any other state; and
(3) agrees to pay all costs of collection, including a reasonable attorney's
fee, in the event default should be made in the payment of any of the
obligations evidenced by this Note.

       (b)    Each Obligor severally (1) acknowledges that the Lender has not

made any representations or entered into any agreements with such Obligor to
induce such Obligor to enter into the transactions contemplated by this Note
except as set forth in writing in the Credit Documents; (2) agrees upon request
such Obligor will furnish financial statements to the Holder and grant the
Holder access to such Obligor's books and records; (3) agrees that any
obligations of any Obligor may, from time to time, in whole or in part, be
renewed, extended, modified, accelerated, compromised, discharged or released by
the Holder, and any collateral, lien, right of set-off or other security for the
obligations evidenced by this Note or any other obligations of any Obligor to
the Holder may, from time to time, in whole or in part, be exchanged, sold,
released, satisfied, or terminated, all without notice to, or in any way
affecting or releasing any of the obligations of any other Obligor; and (4)
agrees that the Holder will not be required first to resort to any Security
Document, any guaranty or any other security pledged or granted to the Holder,
but upon a default under this Note or any of the Security Documents, the Holder
may forthwith look to any Obligor for payment hereunder or may look to and
realize upon any other security held by the Holder, in any order the Holder
chooses, until the entire debt evidenced by this Note is paid.

       SECTION 9 Joint and Several Liability. If the Borrower is comprised of
                 ---------------------------
more than one person, all of the Borrower's representations, warranties,
covenants and agreements under this Note shall be joint and several and shall be
binding on and enforceable against either, any or all of the persons comprising
the Borrower. If any one or more of the persons comprising the Borrower is in
default, the Holder my exercise its remedies on default against all of the
persons comprising the Borrower.

       SECTION 10 Independent Obligations. The Borrower agrees that each of the
                  -----------------------
obligations of the Borrower to the Holder under this Note may be enforced
against the Borrower without the necessity of joining any other Obligor, any
other holders of Liens in any Property or any other person, as a party.

       SECTION 11 Heirs, Successors and Assigns. Whenever in this Note any party
                  -----------------------------
hereto is referred to, such reference shall be deemed to include the heirs,
successors and assigns of such party, except that the Borrower may not assign or
transfer its obligations under this Note without the prior written consent of
the Holder; and all obligations of the Borrower under this Note shall bind the
Borrower's heirs, successors and assigns and shall inure to the benefit of the
successors and assigns of the Holder.

       SECTION 12 Governing Law. This Note shall be construed in accordance with
                  -------------
and governed by Title 9 of the U.S. Code and the internal laws of the State of
Alabama except as required by mandatory provisions of law (without regard to
conflict of law principles).

       SECTION 13 Separability Clause. If any provision of the this Note shall
                  -------------------
be invalid, illegal or unenforceable, the validity, legality and enforceability
of the remaining provisions shall not in any way be affected or impaired
thereby.

       SECTION 14 No Oral Agreements. This Note is the final expression of the
                  ------------------
agreement between the parties hereto, and this Note may not be contradicted by
evidence of any prior oral agreement between such parties. All previous oral
agreements between the parties hereto have been incorporated into this Note and
the other Credit Documents, and there is no unwritten oral agreement between the
parties hereto in existence.

                                      -3-
<PAGE>
 
       SECTION 15 Waiver and Election. The exercise by the Holder of any option
                  -------------------
given under this Note or the Security Documents shall not constitute a waiver of
the right to exercise any other option. No failure or delay on the part of the
Holder in exercising any right, power or remedy under this Note or the Security
Documents shall operate as a waiver thereof, nor shall any single or partial
exercise of any such right, power or remedy preclude any further exercise
thereof or the exercise of any other right, power or remedy. No modification,
termination or waiver of any provisions of this Note, nor consent to any
departure by the Borrower therefrom, shall be effective unless in writing and
signed by an authorized officer of the Holder, and then such waiver or consent
shall be effective only in the specific instance and for the specific purpose
for which given. No notice to or demand on the Borrower in any case shall
entitle the Borrower to any other or further notice or demand in similar or
other circumstances.

       SECTION 16 Set-off. While any Event of Default exists, the Lender is
                  -------
authorized at any time and from time to time, without notice to the Borrower
(any such notice being expressly waived by the Borrower), to set off and apply
any and all deposits (general or special, time or demand, provisional or final)
at any time held and other indebtedness at any time owing by the Lender to or
for the credit or the account of the Borrower against any and all of the
obligations evidenced by this Note, irrespective of whether or not the Lender
shall have made any demand under this Note and although such obligations may be
unmatured. The rights of the Lender under this Section 16 are in addition to all
other rights and remedies (including other rights of set-off or pursuant to any
banker's lien) that the Lender may have.

       SECTION 17 Time of Essence. Time is of the essence of this Note.
                  ---------------

       SECTION 18 Submission to Jurisdiction. The Borrower irrevocably (a)
                  --------------------------
acknowledges that this Note will be accepted by the Lender and performed by the
Borrower in the State of Alabama; (b) submits to the jurisdiction of each state
or federal court sitting in Jefferson County, Alabama (collectively, the
"Courts") over any suit, action or proceeding arising out of or relating to this
Note (to enforce the arbitration provisions hereof or, if the arbitration
provisions are found to be unenforceable, to determine any issues arising out of
or relating to this Note) or any of the other Credit Documents (individually, an
"Agreement Action"); (c) waives, to the fullest extent permitted by law, any
objection or defense that the Borrower may now or hereafter have based on
improper venue, lack of personal jurisdiction, inconvenience of forum or any
similar matter in any Agreement Action brought in any of the Courts; (d) agrees
that final judgment in any Agreement Action brought in any of the Courts shall
be conclusive and binding upon the Borrower and may be enforced in any other
court to the jurisdiction of which the Borrower is subject, by a suit upon such
judgment; (e) consents to the service of process on the Borrower in any
Agreement Action by the mailing of a copy thereof by registered or certified
mail, postage prepaid, to the Borrower at the Borrower's address designated at
the end of this Note; (f) agrees that service in accordance with Section 18(e)
shall in every respect be effective and binding on the Borrower to the same
extent as though served on the Borrower in person by a person duly authorized to
serve such process; and (g) AGREES THAT THE PROVISIONS OF THIS SECTION, EVEN IF
FOUND NOT TO BE STRICTLY ENFORCEABLE BY ANY COURT, SHALL CONSTITUTE "FAIR
WARNING" TO THE BORROWER THAT THE EXECUTION OF THIS NOTE MAY SUBJECT THE
BORROWER TO THE JURISDICTION OF EACH STATE OR FEDERAL COURT SITTING IN JEFFERSON
COUNTY, ALABAMA WITH RESPECT TO ANY AGREEMENT ACTIONS, AND THAT IT IS
FORESEEABLE BY THE BORROWER THAT THE BORROWER MAY BE SUBJECTED TO THE
JURISDICTION OF SUCH COURTS AND MAY BE SUED IN THE STATE OF ALABAMA IN ANY
AGREEMENT ACTIONS. Nothing in this Section 18 shall limit or restrict the
Lender's right to serve process or bring Agreement Actions in manners and in
courts otherwise than as herein provided.

       SECTION 19 Usury  Laws.  Any  provision  of this Note or any of the other
                  -----------
Credit  Documents to the contrary  notwithstanding,  the Borrower and the Lender
agree that they do not intend for the interest or other  consideration  provided
for in this Note and the other  Credit  Documents to be greater than the maximum
amount permitted by applicable law.  Regardless of any provision in this Note or
any of the other Credit Documents,  the Lender shall not be entitled to receive,
collect or apply,  as interest on the  Obligations,  any amount in excess of the
maximum rate of interest permitted to be charged under applicable law until such
time, if any, as that  interest,  together with all other interest then payable,
falls within the then applicable maximum lawful rate of interest.  If the Lender
shall receive, collect or apply any amount in excess of the then maximum rate of
interest,  the amount that would be excessive interest shall be applied first to
the reduction of the principal amount of the Obligations then outstanding in the
inverse order of maturity, and second, if such principal amount is paid in full,
any excess shall forthwith be returned to the Borrower.  In determining  whether
the interest paid or payable under any specific  contingency exceeds the highest
lawful rate, the Borrower and the Lender shall, to the maximum extent  permitted
under applicable law, (a)  characterize any nonprincipal  payment as an expense,
fee or premium rather than as interest,  (b) exclude  voluntary  prepayments and
the effects thereof,  (c) consider all the Obligations as one general obligation
of the  Borrower,  and (d) "spread" the total amount of the interest  throughout
the entire term of this Note so that the interest rate is uniform throughout the
entire term of this Note.

                                      -4-
<PAGE>
 
       SECTION 20 Arbitration; Dispute Resolution; Preservation of Foreclosure
                  ------------------------------------------------------------
Remedies
--------

       (a)    The Borrower represents to the Lender that its business and
affairs constitute substantial interstate commerce and that it contemplates
using the proceeds of this Note in substantial interstate commerce. Except as
otherwise specifically set forth below, any action, dispute, claim, counterclaim
or controversy ("Dispute" or "Disputes"), between or among the Lender, the
Borrower or any other Obligor, including any claim based on or arising from an
alleged tort, shall be resolved by arbitration as set forth below. As used
herein, Disputes shall include all actions, disputes, claims, counterclaims or
controversies arising in connection with this Note, any extension of or
commitment to extend credit by the Lender, any collection of any indebtedness
owed to the Lender, any security or collateral given to the Lender, any action
taken (or any omission to take any action) in connection with any of the
foregoing, any past, present and future agreement between or among the Lender,
the Borrower or any other Obligor (including this Note and any Credit Document),
and any past, present or future transactions between or among the Lender, the
Borrower or any other Obligor. Without limiting the generality of the foregoing,
Disputes shall include actions commonly referred to as lender liability actions.

       (b)    All Disputes shall be resolved by binding arbitration in
accordance with Title 9 of the U.S. Code and the Commercial Arbitration Rules of
the American Arbitration Association (the "AAA"). Defenses based on statutes of
limitation, estoppel, waiver, laches and similar doctrines, that would otherwise
be applicable to an action brought by a party, shall be applicable in any such
arbitration proceeding, and the commencement of an arbitration proceeding with
respect to this Note shall be deemed the commencement of an action for such
purposes.

       (c)    Notwithstanding the foregoing, the Borrower and each other Obligor
agrees that the Lender shall have the option, but not the obligation, to submit
to and pursue in a court of law any claim against the Borrower or any other
Obligor for a debt due. The Borrower and each other Obligor agrees that, if the
Lender pursues such a claim in a court of law, (1) failure of the Lender to
assert any additional claim in such proceeding shall not be deemed a waiver of,
or estoppel to pursue, such claim as a claim or counterclaim in arbitration as
set forth above, and (2) the institution or maintenance of a judicial action
hereunder shall not constitute a waiver of the right of any party to submit any
other action, dispute, claim or controversy as described above, even though
arising out of the same transaction or occurrence, to binding arbitration as set
forth herein. If the Borrower asserts a claim against the Lender in arbitration
or otherwise during the pendency of a claim brought by the Lender in a court of
law, the court action shall be stayed and the parties shall submit to
arbitration all claims.

       (d)    No provision of, nor the exercise of any rights under this
Section, shall limit the right of any party (1) to foreclose against any real or
personal property collateral by exercise of a power of sale under any Credit
Document, or by exercise of any rights of foreclosure or of sale under
applicable law, (2) to exercise self-help remedies such as set-off, or (3) to
obtain provisional or ancillary remedies such as injunctive relief, attachment
or the appointment of a receiver from a court having jurisdiction before, during
or after the pendency of any arbitration or referral. The institution and
maintenance of an action for judicial relief or pursuit of provisional or
ancillary remedies or exercise of self-help remedies shall not constitute a
waiver of the right of any party, including the plaintiff in such an action, to
submit the Dispute to arbitration or, in the case of actions on a debt, to
judicial resolution.

       (e)    Whenever an arbitration is required hereunder, the arbitrator
shall be selected in accordance with the Commercial Arbitration Rules of the
AAA. The AAA shall designate a panel of 10 potential arbitrators knowledgeable
in the subject matter of the Dispute. Each of the Lender and the Obligor shall
designate, within 30 days of the receipt of the list of potential arbitrators,
one of the potential arbitrators to serve, and the two arbitrators so designated
shall select a third arbitrator from the eight remaining potential arbitrators.
The panel of three arbitrators shall determine the resolution of the Dispute.

                                      -5-
<PAGE>
 
       IN WITNESS WHEREOF, the undersigned has executed and delivered this Note
dated the date first written above.

                            /s/ John H. Holcomb, III
                            ------------------------
                            Signature of Borrower


                            John H. Holcomb, III
                            --------------------
                            Please Print Name


                            Send Correspondence and Billings to:

                                   John H. Holcomb, III
                                   --------------------------
                                   56 Country Club Blvd      
                                   --------------------------
                                   Birmingham, Alabama  35213
                                   --------------------------

                                      -6-