<PAGE>   1
                                                                    EXHIBIT 10.4



                            THIS EMPLOYMENT CONTRACT

made and entered into effective the 1st day of September, 1996 ("Effective
Date") between Patriot National Bank, a national banking association ("Bank")
and First Patriot Bankshares Corporation, a Virginia corporation ("FPBK") (Bank
and FPBK collectively referred to herein as "Employer") and Carroll C. Markley
("Employee").

         WHEREAS, Employee presently serves as President and Chief Executive
Officer of the Bank and FPBK; and,

         WHEREAS, the parties wish to establish the terms and conditions of
Employee's employment for the term stated herein.

         NOW, therefore, in consideration of the promises and the mutual
covenants and agreements set forth herein, the parties hereby agree as follows:

                          1.  RELATIONSHIP AND DUTIES

         (a)     Employer hereby employs Employee on the Effective Date as
President and Chief Executive Officer of Bank, to hold the title of President
and Chief Executive Officer, and as President and Chief Executive Officer of
FPBK, to hold the title of President and Chief Executive Officer; provided,
however FPBK shall be entitled to place Employee in such different or other
office(s) of FPBK than President and Chief Executive Officer, by an affirmative
vote of two-thirds (2/3) of all members of the Board, whether or not in
attendance at the meeting or voting upon the issue.  Employee shall perform
such services and duties as the respective boards of directors of Bank and FPBK
(collectively, "Boards") may, from time to time, designate during the term
hereof.  Subject to the terms and conditions hereof, Employee will perform such
duties and exercise such authority as are customarily performed and exercised
by persons holding such office, subject to the guidance of the respective
Boards.

         (b)     Employee shall seek and accept election to the Boards, and
shall serve as a member of the Executive Committees thereof and such other
committees as the Boards may designate, subject to the terms hereof.

         (c)     Employee accepts such employment and shall devote his full
time, attention, and best efforts to the diligent performance of his duties
herein specified and as an officer and director of Employer.  While employed by
Employer, the Employee will not, without the prior written consent of the
Boards (which consent shall not be unreasonably withheld) accept employment
with any other individual, corporation, partnership, governmental authority or
other entity, or engage in any other venture for profit which Employer or the
Boards may consider to be in conflict with Employer's best interests or to be
in competition with Employer, or which may interfere in any way with Employee's
performance of his duties hereunder.  It is understood that Employee does have
the right to participate in passive investments including income producing real
estate, not otherwise in conflict with Bank policy.
<PAGE>   2
         (d)     Regarding the relationships and duties of the parties to this
contract, Employee shall not be required by Employer, as a part of his duties,
to perform or to participate in any activity which constitutes a violation of
any state or federal law, rule, ordinance or regulation.

                            2.  TERMS OF EMPLOYMENT

         (a)     The term of this contract shall run from September 1, 1996 to
December 31, 2000, unless terminated earlier pursuant to the terms hereof.
During the year 2000, Employer and Employee shall enter into negotiations for
renewal of Employee's employment.

         (b)     Employee's employment pursuant to this agreement shall be
terminated by the first to occur of any of the following:

         (i)     the death of Employee;

         (ii)    the complete disability of Employee.  "Complete disability" as
used herein shall mean the inability of Employee, due to illness, accident, or
any other physical or mental incapacity, completely to fulfill his obligations
hereunder for an aggregate of ninety (90) days within any period of 180
consecutive days during the term hereof;

         (iii)   the discharge of Employee by Employer for cause.  "Cause" as
used herein shall include, without limitation: dishonesty; theft; conviction of
a crime, which is either a (1) felony or (2) misdemeanor involving moral
turpitude or financial impropriety; unethical business conduct; activity which
is contrary to Employer's interests; gross or repeated negligence in carrying
out Employee's duties; or material violation of Employee's obligations
hereunder. Should Employer deem specific activities contrary to Employer's
interest or that negligence by Employee in carrying out his duties or any
violation of Employee's obligations hereunder has occurred, written notice of
said activity, negligence or violation shall be provided by Employer to
Employee along with a reasonable period of time in which to correct.  Provided
that such activity, negligence or violation is neither dishonest nor criminal,
90 days shall be deemed to be a reasonable time in which to correct such
deficiencies.

         (iv)    Discharge for "cause" will require a two-thirds majority vote
of the Boards, exclusive of Employee.  Termination of Employee's employment for
cause shall include termination as an employee, officer and director of
Employer.

         (c)     Upon a Change in Control of either FPBK or Bank,
notwithstanding anything contained herein to the contrary, Employer may
terminate this agreement without cause at any time upon an affirmative vote of
two-thirds (2/3) of all members of the Board, whether or not in attendance at
the meeting or voting upon the issue.  In the event of such termination without
cause
<PAGE>   3
by Employer, Employee (i) shall be paid a severance pay based upon Employee's
annual base salary in effect at the time of termination, as follows:

During 1996 or 1997        150% of Base Salary
During 1998                140% of Base Salary
During 1999                130% of Base Salary
During 2000                120% of Base Salary then remaining

Such severance pay shall be paid in a lump sum not later than thirty (30) days
following the effective date of termination.  In addition, upon any such
termination Employee shall be entitled to purchase the automobile provided
hereunder at the current book value.  Employee shall not be entitled to a
performance bonus in the year of termination, except as may be awarded in the
sole discretion of the Boards.

         (d)     Employee may cancel this Agreement within ninety (90) days
after a Change in Control, by written notice given to Employer prior to the
Change in Control.

                                3.  COMPENSATION

         For all services which Employee may render to Employer during the term
hereof, Bank shall pay to Employee, subject to such deductions as may be
required by law, according to the schedule set out below:

         (a)     Base Salary.   From the Effective Date hereof, Employee shall
receive for the term of this contract a base salary at an annual rate of
$160,000.05, payable in equal semi-monthly installments, subject to such
deductions as may be required by law.  Employee will receive performance
reviews at least annually at the end of each fiscal year from the Boards, and
Employee's base salary may be increased but not decreased at the sole
discretion of the Boards.

         (b)     Performance Bonuses.   An annual bonus based on the Bank's
performance may be awarded at the sole discretion of the Boards, upon formulae
to be mutually agreed upon by Employer and Employee.

                               4.  OTHER BENEFITS

         During the term of Employee's employment hereunder Bank shall furnish
to Employee: (i) An American automobile of Employee's choice having a cost
(net of trade-in) not to exceed $30,000 which may be leased by the Bank; (ii)
A term life insurance policy providing for death benefits in an amount of up to
two (2) times his annual base salary and having a beneficiary designated by
Employee, or, if such is not obtainable without any rating, a payment of
$300.00 per quarter to Employee;  (iii)  A group health and hospitalization
insurance policy covering the Employee at no
<PAGE>   4
cost to the Employee; (iv) the current disability benefits plan of the Bank
provided to all officers as well as the benefits provided through the
supplemental policy currently in effect; (v) a complete physical examination
for the Employee on an annual basis at the Bank's expense.  The Employee will
be allowed to participate in all other benefits provided to the company's
employees.


                               5.  STOCK OPTIONS

         (a)     Employee shall be granted an option to purchase 6,930 shares
of stock of FPBK at the original issue price of the stock shall be granted on
or before 12/31/96; provided that at the end of the fiscal year, the Bank's
performance, based upon the Bank's return on assets, meets or exceeds the
amount estimated in the annual budgets as approved by the Bank.  The options
may be exercised in whole or in part at any time during the first ten (10)
years after the original issue of stock in FPBK.   For each year following
1996, Employee shall be granted an option to purchase 3,465 shares of stock of
FPBK at a strike price consistent with the market on the date granted, provided
that at the end of the fiscal year, the Bank's performance, based upon the
Bank's return on assets and return on equity, meets or exceeds the amount
estimated in the annual budgets as approved by the Bank.  The options may be
exercised in whole or in part at any time during the first ten (10) years after
the issue of the stock option agreement.

         (b)      If, prior to the Expiration Time of the agreement granted,
the Corporation shall subdivide its outstanding shares of common stock into a
greater number of shares, or declare and pay a dividend on its common stock
payable in additional shares of its common stock, the Exercise Price as then in
effect shall be proportionately reduced, and/or the number of shares of common
stock then subject to exercise under the Stock Option Agreement (and not
previously exercised), shall be proportionately increased as the case may be.

         (c)     If, prior to the Expiration Time of the agreement granted, the
Corporation shall combine its outstanding shares of common stock into a smaller
number of shares, the Exercise Price, as then in effect, shall be
proportionately increased, and the number of shares of common stock then
subject to exercise under this Stock Option Agreement (and not previously
exercised), shall be proportionately reduced.

         (d)     Any options that have not otherwise been accrued and granted
herein shall immediately terminate upon Employee's termination of employment,
whether such termination be voluntary or involuntary; provided, however, in the
event of termination pursuant to paragraph 2(c) of this agreement, Employee
shall be entitled to any options that would have accrued in the year of
termination but for the termination.
<PAGE>   5
                                  6.  EXPENSES

         Employer shall pay, or, upon Employee's presentment to Employer of
expense reports acceptable to Employer and which are in sufficiently detailed
form to comply with standards for deduction of business expenses established
from time to time by the Internal Revenue Service, reimburse Employee for, all
expenses approved by Employer and incurred by Employee in connection with
performance of his duties hereunder, including reimbursement for his Westwood
Country Club dues.

                         7.  POST TERMINATION COVENANTS

         At such time as Employee's employment by Employer terminates, whether
during the initial contract period of employment or thereafter, and whether
such termination be voluntary or involuntary (but excluding a termination as a
result of the nonrenewal of this agreement), Employee agrees that for six (6)
months following such termination he will not engage (either individually or as
an employee or representative of any other person or entity) in banking
activities, in which chartered national or state banks may at that time legally
be engaged, within a five (5) mile radius of the Bank's Headquarters.   The
employees to not furnish, use, or divulge to anyone any confidential
information of Employer acquired by him from Employer and relating to
Employer's business activities and further agrees, in the event there has been
no Change in Control (as defined hereinafter), for six (6) months following
such termination, Employee agrees that he will not, without the prior written
consent of Employer:  (i) furnish anyone with the name of, or any list or
lists which identify, any customers or stockholders of Employer or utilize such
list or information himself; (ii) contact directly or indirectly any customer
of Employer for the purpose of soliciting such person's business for another
bank or similar financial institution; (iii) hire for any other employer
(including himself) any employee of Employer or directly or indirectly cause
such employee to leave his or her employment to work for another; (iv) pursue
an actual or potential business opportunity of interest to and which could be
pursued by Employer which came to the attention of Employee in connection with
his employment with Employer and which Employee had not previously offered in
writing to Employer with sufficient advance notice to allow Employer to examine
and pursue or reject such opportunity.  Excepted from the requirements of
subparagraph (i) in this paragraph is any information which is or becomes
publicly available information through no fault or act of Employee.

         It is understood and agreed by the parties hereto that the provisions
of this paragraph are independent of each other, and to the extent any
provision or portion thereof shall be determined by a court of competent
jurisdiction to be unenforceable, such determination shall not affect the
validity or enforceability of any other provision of this paragraph or the
remainder of this agreement.
<PAGE>   6
                             8.  CHANGE OF CONTROL

         For the purposes of this agreement, as used herein the term "Change in
Control" shall mean the acquisition of control of either FPBK or Bank by a
person, acting directly or indirectly or through or in concert with one or more
persons, through a purchase, assignment, transfer, pledge, or other disposition
of its voting stock.  As used in the definition of Change in Control, the terms
"person" and "control" shall have the meanings assigned to them in 12 USCS
Section 1817(j)(8), as amended from time to time.

                            9.  WAIVER OF PROVISIONS

         Failure by any of the parties hereto to insist, in one or more
instances, on performance by the other in strict accordance with the terms and
conditions of this agreement shall not be deemed a waiver or relinquishment of
any right granted hereunder or of the obligation of future performance of any
such term or condition or of any other term or condition of this agreement,
unless such waiver is contained in writing signed by or on behalf of all the
parties.

                               10.  GOVERNING LAW

         This agreement shall be governed by and construed and enforced in
accordance with the laws of the Commonwealth of Virginia.  If for any reason
any provision of this agreement shall be held by a court of competent
jurisdiction to be void or unenforceable, the same shall not affect the
remaining provisions hereof.

                        11.  MODIFICATION AND AMENDMENT

         This agreement contains the sole and entire agreement among the
parties hereto and supersedes all prior discussions and agreements among the
parties, and any such prior agreements shall, from and after the date hereof,
be null and void.  This agreement shall not be modified or amended except by an
instrument in writing signed by or on behalf of all parties hereto.

                         12.  COUNTERPARTS AND HEADINGS

         This agreement may be executed simultaneously in any number of
counterparts, each of which shall be deemed an original but all of which shall
constitute one and the same instrument. The headings set out herein are for
convenience of reference and shall not be deemed a part of this agreement.
<PAGE>   7
                             13.  INJUNCTIVE RELIEF

         In the event of a breach or threatened breach by Employee of any of
the provisions of paragraph 2 or paragraph 7, and notwithstanding any other
provision in this agreement, Employer, in addition to any other available
rights or remedies, shall be entitled to such temporary restraining orders and
permanent injunctions, as are allowable and authorized by the laws of the
Commonwealth of Virginia based on the facts of the case, to restrain such
breach by Employee and/or any persons directly or indirectly acting for or with
him.  Employee's obligations under paragraph 7 hereof shall remain binding and
enforceable according to its terms notwithstanding expiration or termination of
the other terms of this agreement or the termination of Employee's employment
relationship with the Bank, whether such termination be voluntary or
involuntary, and including a termination as the result of the nonrenewal of
this agreement.

                                14.  SUCCESSORS

         This Agreement shall inure to the benefit of and be binding upon the
Employer, its successors and assigns and upon the Employee, and his heirs and
personal representatives. Neither this agreement nor performance hereunder may
be assigned by Employee or Employer.

         IN WITNESS WHEREOF, the parties hereto have executed this agreement
under seal on the 30th day of August, 1996, to be effective as of the date
first written above.


                                     EMPLOYEE:
                                  
            /s/                                            /s/          (SEAL)
----------------------------------   -----------------------------------
Witness                              Carroll C. Markley
                                  
                                     EMPLOYER:
                                  
                                     PATRIOT NATIONAL BANK,
                                     a national banking association
                                  
                                  
           /s/                       By:                    /s/         (SEAL)
----------------------------------      --------------------------------
                      (SEAL)                 Its Authorized Officer
Witness                           
                                  
                                     FIRST PATRIOT BANKSHARES
                                     CORPORATION, a Virginia corporation
                                  
                                  
         /s/                         By:                    /s/         
----------------------------------      --------------------------------
(SEAL)                            
Witness                                      Its Authorized Officer