<PAGE>

                                                                    EXHIBIT 10.4

Exhibit 10.4 Employment Agreement between Eastern Virginia Bankshares, Inc. and
Joseph H. James, Jr.

                              EMPLOYMENT AGREEMENT

          THIS EMPLOYMENT AGREEMENT is made and entered into as of the eighth
day of January, 2003 by and between Eastern Virginia Bankshares, Inc., a
Virginia corporation, hereinafter called the "Corporation", and Joseph H. James,
Jr. hereinafter called "Employee", and provides as follows:

                                    RECITALS

          WHEREAS, the Corporation is a bank holding company engaged in the
operation of banks; and

          WHEREAS, Employee has been involved in the management of the business
and affairs of the Corporation and, therefore, possesses managerial experience,
knowledge, skills and expertise in such type of business; and

          WHEREAS, the employment of Employee by the Corporation is in the best
interests of the Corporation and Employee; and

          WHEREAS, the parties have mutually agreed upon the terms and
conditions of Employee's continued employment by the Corporation as hereinafter
set forth;

                                        1

<PAGE>

                               TERMS OF AGREEMENT

          NOW, THEREFORE, for and in consideration of the premises and of the
mutual promises and undertakings of the parties as hereinafter set forth, the
parties covenant and agree as follows:

          Section 1. Employment. (a) Employee shall be employed as an executive
officer of the Corporation. He shall perform such services for the Corporation
and/or one or more Affiliates as may be assigned to Employee by the Corporation
from time to time and that are commensurate with his training and experience
upon the terms and conditions hereinafter set forth.

                  (b) References in this Agreement to services rendered for the
Corporation and compensation and benefits payable or provided by the Corporation
shall include services rendered for and compensation and benefits payable or
provided by any Affiliate. References in this Agreement to the "Corporation"
also shall mean and refer to each Affiliate for which Employee performs
services. References in this Agreement to "Affiliate" shall mean any business
entity that, directly or indirectly, through one or more intermediaries, is
controlled by the Corporation.

          Section 2. Term and Renewal. The initial term of this Agreement shall
end December 31, 2004. However, on each December 31, beginning with December 31,
2003, the term of this Agreement shall be renewed and extended by one year
unless Employee or the Corporation gives prior notice to the other in writing
that the term shall not be renewed and extended. This Agreement shall terminate
at the end of its term.

          Section 3. Exclusive Service. Employee shall devote his best efforts
and full time to rendering services on behalf of the Corporation in furtherance
of its best interests. Employee shall comply with all policies, standards and
regulations of the Corporation now or hereafter promulgated, and shall perform
his duties under this Agreement to the best of his abilities and in accordance
with standards of conduct applicable to officers of banks.

          Section 4. Salary. (a) As compensation while employed hereunder,
Employee, during his faithful performance of this Agreement, in whatever
capacity rendered, shall receive an annual base salary of $96,000 payable on
such terms and in such installments as the parties may from time to time
mutually agree upon. The Board of Directors, in its discretion, may increase
Employee's base salary during the term of this Agreement.

                  (b) The Corporation shall withhold state and federal income
taxes, social security taxes and such other payroll deductions as may from time
to time be required by law or agreed upon in writing by Employee and the
Corporation. The Corporation shall also withhold and remit to the proper party
any amounts agreed to in writing by the Corporation and Employee for
participation in any corporate sponsored benefit plans for which a contribution
is required.

                  (c) Except as otherwise expressly set forth hereunder, no
compensation shall be paid pursuant to this Agreement in respect of any month or
portion thereof subsequent to any termination of Employee's employment by the
Corporation.

          Section 5. Corporate Benefit Plans. Employee shall be entitled to
participate in or become a participant in any employee benefit plan maintained
by the Corporation for which he is or will become eligible on such terms as the
Board of Directors may, in its discretion, establish, modify or otherwise
change.

          Section 6. Bonuses. Employee shall receive only such bonuses as the
Board of Directors, in its discretion, decides to pay to Employee.

                                        2

<PAGE>

          Section 7. Expense Account. The Corporation shall reimburse Employee
for reasonable and customary business expenses incurred in the conduct of the
Corporation's business. Such expenses will include business meals, out-of-town
lodging and travel expenses. Employee agrees to timely submit records and
receipts of reimbursable items and agrees that the Corporation can adopt
reasonable rules and policies regarding such reimbursement. The Corporation
agrees to make prompt payment to Employee following receipt and verification of
such reports.

          Section 8. Personal and Sick Leave. Employee shall be entitled to the
same personal and sick leave as the Board of Directors may from time to time
designate for all full-time employees of the Corporation.

          Section 9. Vacations. Employee shall be entitled to twenty (20) week
days of vacation leave each year which shall be taken at such time or times as
may be approved by the Corporation and during which Employee's compensation
hereunder shall continue to be paid.

          Section 10. Termination. (a) Notwithstanding the termination of
Employee's employment pursuant to any provision of this Agreement, the parties
shall be required to carry out any provisions of this Agreement which
contemplate performance by them subsequent to such termination. In addition, no
termination shall affect any liability or other obligation of either party which
shall have accrued prior to such termination, including, but not limited to, any
liability, loss or damage on account of breach. No termination of employment
shall terminate the obligation of the Corporation to make payments of any vested
benefits provided hereunder or the obligations of Employee under Sections 11, 12
and 13.

                  (b) Employee's employment hereunder may be terminated by
Employee upon thirty (30) days written notice to the Corporation or at any time
by mutual agreement in writing.

                  (c) This Agreement shall terminate upon death of Employee;
provided, however, that in such event the Corporation shall pay to the estate of
Employee the compensation including salary and accrued bonus, if any, which
otherwise would be payable to Employee through the end of the month in which his
death occurs.

                  (d) (1) The Corporation may terminate Employee's employment
other than for "Cause", as defined in Section 10(e), at any time upon written
notice to Employee, which termination shall be effective immediately. Employee
may resign thirty (30) days after notice to the Corporation for "Good Reason",
as hereafter defined. In the event the Employee's employment terminates pursuant
to this Section 10(d), Employee shall receive a monthly amount equal to
one-twelfth (1/12) his rate of annual base salary in effect immediately
preceding such termination ("Termination Compensation") in each month for the
remainder of the term of this Agreement. Payments of the Termination
Compensation shall be made at the times such payments would have been made in
accordance with Section 4(a).

          (2) Notwithstanding anything in this Agreement to the contrary:

                  (i)  If Employee breaches Section 11 or 12, Employee will not
thereafter be entitled to receive any further compensation or benefits pursuant
to this Section 10(d); and

                  (ii) If, while he is receiving payments under this Section
10(d), Employee engages in a Competitive Business within the area described in
Section 12(i), such payments will cease and he will not thereafter be entitled
to receive any compensation or benefits pursuant to this Section 10(d) even
though such conduct occurs after the covenants contained in Section 12 have
expired.

                                        3

<PAGE>

          (3) The Corporation shall not be required to make payment of the
Termination Compensation or any portion thereof to the extent such payment is
prohibited by the terms of the regulations presently found at 12 C.F.R. part 359
or to the extent that any other governmental approval of the payment required by
law is not received.

          (4) Except as set forth in Sections 10(d)(2) and 10(d)(3), the
Corporation's obligation to pay the Employee the compensation provided in
Section 10(d)(1) shall be absolute and unconditional and shall not be affected
by any circumstances, including, without limitation, any set-off, counterclaim,
recoupment, defense or other right which the Corporation may have against him or
anyone else. All amounts payable by the Corporation hereunder shall be paid
without notice or demand. Each and every payment made hereunder by the
Corporation shall be final and the Corporation will not seek to recover all or
any part of such payment from the Employee or from whosoever may be entitled
thereto, for any reason whatsoever. The Employee shall not be required to
mitigate the amount of any payment provided for in this Agreement by seeking
other employment or otherwise.

          (5) For purposes of this Agreement, "Good Reason" shall mean:

                  (i)  The assignment of duties to the Employee by the
Corporation which result in the Employee having significantly less authority or
responsibility than he has on the date hereof, without his express written
consent;

                  (ii) Requiring the Employee to maintain his principal office
outside of Essex County, Virginia unless the Corporation moves its operations
center to a place to which the Employee is required to move;

                  (iii) A reduction by the Corporation of the Employee's base
salary, as the same may have been increased from time to time;

                  (iv) The failure of the Corporation to provide the Employee
with substantially the same fringe benefits that are provided to him at the
inception of this agreement;

                  (v)  The Corporation's failure to comply with any material
term of this Agreement; or

                  (vi) The failure of the Corporation to obtain the assumption
of and agreement to perform this Agreement by any successor as contemplated in
Section 14 hereof.

                  (e) The Corporation shall have the right to terminate
Employee's employment under this Agreement at any time for Cause, which
termination shall be effective immediately. Termination for "Cause" shall
include termination for Employee's personal dishonesty, incompetence, willful
misconduct, breach of a fiduciary duty involving personal profit, intentional
failure to perform stated duties, willful violation of any law, rule or
regulation (other than traffic violations or similar offenses) or final
cease-and-desist order, conviction of a felony or of a misdemeanor involving
moral turpitude, misappropriation of the Corporation's assets (determined on a
reasonable basis) or those of its Affiliates, or material breach of any other
provision of this Agreement. In the event Employee's employment under this
Agreement is terminated for Cause, Employee shall thereafter have no right to
receive compensation or other benefits under this Agreement.

                  (f) The Corporation may terminate Employee's employment under
this Agreement, after having established the Employee's disability by giving to
Employee written notice of its intention to terminate his employment for
disability and his employment with the Corporation shall terminate effective on
the 90th

                                        4

<PAGE>

day after receipt of such notice if within 90 days after such receipt Employee
shall fail to return to the full-time performance of the essential functions of
his position (and if Employee's disability has been established pursuant to the
definition of "disability" set forth below). For purposes of this Agreement,
"disability" means either (i) disability which after the expiration of more than
13 consecutive weeks after its commencement is determined to be total and
permanent by a physician selected and paid for by the Corporation or its
insurers, and acceptable to Employee or his legal representative, which consent
shall not be unreasonably withheld or (ii) disability as defined in the policy
of disability insurance maintained by the Corporation or its Affiliates for the
benefit of Employee, whichever shall be more favorable to Employee.
Notwithstanding any other provision of this Agreement, the Corporation shall
comply with all requirements of the Americans with Disabilities Act, 42 U.S.C.
Section 12101 et. seq.

                  (g) If Employee is suspended and/or temporarily prohibited
from participating in the conduct of the Corporation's affairs by a notice
served pursuant to the Federal Deposit Insurance Act, the Corporation's
obligations under this Employment Agreement shall be suspended as of the date of
service unless stayed by appropriate proceedings. If the charges in the notice
are dismissed, the Corporation may in its discretion (i) pay Employee all or
part of the compensation withheld while its contract obligations were suspended,
and (ii) reinstate (in whole or in part) any of its obligations which were
suspended.

                  (h) If Employee is removed and/or permanently prohibited from
participating in the conduct of the Corporation's affairs by an order issued
under the Federal Deposit Insurance Act or the Code of Virginia, all obligations
of the Corporation under this Employment Agreement shall terminate as of the
effective date of the order, but vested rights of the parties shall not be
affected.

                  (i)(1) If Employee's employment is terminated without Cause,
within one year after a Change of Control shall have occurred, or if he resigns
for Good Reason, within one year after a Change of Control shall have occurred,
then on or before Employee's last day of employment with the Corporation, the
Corporation shall pay to Employee as compensation for services rendered to the
Corporation and its Affiliates a cash amount (subject to any applicable payroll
or other taxes required to be withheld) equal to the excess, if any, of 299% of
Employee's "annualized includable compensation for the base period", as defined
in Section 280G of the Internal Revenue Code of 1986 (the "Code"), over the
total amount payable to Employee under Section 10(d) provided that, at the
option of Employee, the cash amount required to be paid hereby shall be paid by
the Corporation in equal monthly installments over the twenty-four (24) months
succeeding the date of termination, payable on the first day of each such month.

                  (2) For purposes of this Agreement, a Change of Control occurs
if, after the date of this Agreement, (i) any person, including a "group" as
defined in Section 13(d)(3) of the Securities Exchange Act of 1934, becomes the
owner or beneficial owner of Corporation securities having 50% or more of the
combined voting power of the then outstanding Corporation securities that may be
cast for the election of the Corporation's directors other than a result of an
issuance of securities initiated by the Corporation, or open market purchases
approved by the Board of Directors, as long as the majority of the Board of
Directors approving the purchases is a majority at the time the purchases are
made; or (ii) as the direct or indirect result of, or in connection with, a
tender or exchange offer, a merger or other business combination, a sale of
assets, a contested election of directors, or any combination of these events,
the persons who were directors of the Corporation before such events cease to
constitute a majority of the Corporation's Board, or any successor's board,
within two years of the last of such transactions. For purposes of this
Agreement, a Change of Control occurs on the date on which an event described in
(i) or (ii) occurs. If a Change of Control occurs on account of a series of
transactions or events, the Change of Control occurs on the date of the last of
such transactions or events.

                  (3) It is the intention of the parties that no payment be made
or benefit provided to

                                        5

<PAGE>

Employee pursuant to this Agreement that would constitute an "excess parachute
payment" within the meaning of Section 280G of the Code and any regulations
thereunder, thereby resulting in a loss of an income tax deduction by the
Corporation or the imposition of an excise tax on Employee under Section 4999 of
the Code. If the independent accountants serving as auditors for the Corporation
on the date of a Change of Control (or any other accounting firm designated by
the Corporation) determine that some or all of the payments or benefits
scheduled under this Agreement, as well as any other payments or benefits on a
Change of Control, would be nondeductible by the Company under Section 280G of
the Code, then the payments scheduled under this Agreement will be reduced to
one dollar less than the maximum amount which may be paid without causing any
such payment or benefit to be nondeductible. The determination made as to the
reduction of benefits or payments required hereunder by the independent
accountants shall be binding on the parties. Employee shall have the right to
designate within a reasonable period, which payments or benefits will be
reduced; provided, however, that if no direction is received from Employee, the
Corporation shall implement the reductions in its discretion.

          Section 11. Confidentiality/Nondisclosure. Employee covenants and
agrees that any and all information concerning the customers, businesses and
services of the Corporation of which he has knowledge or access as a result of
his association with the Corporation in any capacity, shall be deemed
confidential in nature and shall not, without the proper written consent of the
Corporation, be directly or indirectly used, disseminated, disclosed or
published by Employee to third parties other than in connection with the usual
conduct of the business of the Corporation. Such information shall expressly
include, but shall not be limited to, information concerning the Corporation's
trade secrets, business operations, business records, customer lists or other
customer information. Upon termination of employment Employee shall deliver to
the Corporation all originals and copies of documents, forms, records or other
information, in whatever form it may exist, concerning the Corporation or its
business, customers, products or services. In construing this provision it is
agreed that it shall be interpreted broadly so as to provide the Corporation
with the maximum protection. This Section 11 shall not be applicable to any
information which, through no misconduct or negligence of Employee, has
previously been disclosed to the public by anyone other than Employee.

          Section 12. Covenant Not to Compete. During the term of this Agreement
and throughout any further period that he is an officer or employee of the
Corporation, and for a period of twelve (12) months from and after the date that
Employee is (for any reason) no longer employed by the Corporation or for a
period of twelve (12) months from the date of entry by a court of competent
jurisdiction of a final judgment enforcing this covenant in the event of a
breach by Employee, whichever is later, Employee covenants and agrees that he
will not, directly or indirectly, either as a principal, agent, employee,
employer, stockholder, co-partner or in any other individual or representative
capacity whatsoever: (i) engage in a Competitive Business anywhere within a ten
(10) mile radius of the location of the Corporation's principal executive
offices on the date Employee's employment terminates; or (ii) solicit, or assist
any other person or business entity in soliciting, any depositors or other
customers of the Corporation to make deposits in or to become customers of any
other financial institution conducting a Competitive Business; or (iii) induce
any individuals to terminate their employment with the Corporation or its
Affiliates. As used in this Agreement, the term "Competitive Business" means all
banking and financial products and services that are substantially similar to
those offered by the Corporation on the date that Employee's employment
terminates. Employee's obligations under this Section 12 shall terminate on the
date a Change of Control occurs.

          Section 13. Injunctive Relief, Damages, Etc. Employee agrees that
given the nature of the positions held by Employee with the Corporation, that
each and every one of the covenants and restrictions set forth in Sections 11
and 12 above are reasonable in scope, length of time and geographic area and are
necessary for the protection of the significant investment of the Corporation in
developing, maintaining and expanding its business. Accordingly, the parties
hereto agree that in the event of any breach by Employee of any of the
provisions of Sections 11 or 12 that monetary damages alone will not adequately
compensate the Corporation

                                        6

<PAGE>

for its losses and, therefore, that it may seek any and all legal or equitable
relief available to it, specifically including, but not limited to, injunctive
relief and Employee shall be liable for all damages, including actual and
consequential damages, costs and expenses, including legal costs and actual
attorneys' fees, incurred by the Corporation as a result of taking action to
enforce, or recover for any breach of, Section 11 or Section 12. The covenants
contained in Sections 11 and 12 shall be construed and interpreted in any
judicial proceeding to permit their enforcement to the maximum extent permitted
by law. Should a court of competent jurisdiction determine that any provision of
the covenants and restrictions set forth in Section 12 above is unenforceable as
being overbroad as to time, area or scope, the court may strike the offending
provision or reform such provision to substitute such other terms as are
reasonable to protect the Corporation's legitimate business interests.

          Section 14. Binding Effect/Assignability. This Employment Agreement
shall be binding upon and inure to the benefit of the Corporation and Employee
and their respective heirs, legal representatives, executors, administrators,
successors and assigns, but neither this Agreement, nor any of the rights
hereunder, shall be assignable by Employee or any beneficiary or beneficiaries
designated by Employee. The Corporation will require any successor (whether
direct or indirect, by purchase, merger, consolidation or otherwise) to all or
substantially all of the business, stock or assets of the Corporation, by
agreement in form and substance reasonably satisfactory to the Employee, to
expressly assume and agree to perform this Agreement in its entirety. Failure of
the Corporation to obtain such agreement prior to the effectiveness of any such
succession shall be a breach of this Agreement. As used in this Agreement,
"Corporation" shall include any successor to its business, stock or assets as
aforesaid which executes and delivers the agreement provided for in this Section
14 or which otherwise becomes bound by all the terms and provisions of this
Agreement by operation of law.

          Section 15. Governing Law. This Employment Agreement shall be subject
to and construed in accordance with the laws of Virginia.

          Section 16. Invalid Provisions. The invalidity or unenforceability of
any particular provision of this Employment Agreement shall not affect the
validity or enforceability of any other provisions hereof, and this Employment
Agreement shall be construed in all respects as if such invalid or unenforceable
provisions were omitted.

          Section 17. Notices. Any and all notices, designations, consents,
offers, acceptance or any other communications provided for herein shall be
given in writing and shall be deemed properly delivered if delivered in person
or by registered or certified mail, return receipt requested, addressed in the
case of the Corporation to its registered office or in the case of Employee to
his last known address.

          Section 18. Entire Agreement.

                  (a) This Employment Agreement constitutes the entire agreement
among the parties with respect to the subject matter hereof and supersedes any
and all other agreements, either oral or in writing, among the parties hereto
with respect to the subject matter hereof.

                  (b) This Employment Agreement may be executed in one or more
counterparts, each of which shall be considered an original copy of this
Agreement, but all of which together shall evidence only one agreement.

          Section 19. Amendment and Waiver. This Employment Agreement may not be
amended except by an instrument in writing signed by or on behalf of each of the
parties hereto. No waiver of any provision of this Employment Agreement shall be
valid unless in writing and signed by the person or party to be charged.

                                        7

<PAGE>

          Section 20. Case and Gender. Wherever required by the context of this
Employment Agreement, the singular or plural case and the masculine, feminine
and neuter genders shall be interchangeable.

          Section 21. Captions. The captions used in this Employment Agreement
are intended for descriptive and reference purposes only and are not intended to
affect the meaning of any Section hereunder.

          Section 22. Arbitration. Any controversy or claim arising out of or
relation to this Agreement, or the breach thereof, except for those arising
under Sections 11 and 12 hereof, shall be fully and finally settled by
arbitration administered by the American Arbitration Association in accordance
with its Commercial Arbitration Rules then in effect ("AAA Rules"), and judgment
on the award rendered by the arbitrator may be entered in any court having
jurisdiction thereof. The arbitration shall be conducted by one arbitrator
either mutually agreed upon by the Company and the Executive or chosen in
accordance with the AAA Rules. The place of arbitration shall be the County of
Essex, Commonwealth of Virginia. Except as may be required by law, neither a
party nor an arbitrator may disclose the existence, content, or results of any
arbitration hereunder without the prior written consent of both parties.

          IN WITNESS WHEREOF, the Corporation has caused this Employment
Agreement to be signed by its duly authorized officer and Employee has hereunto
set his hand and seal on the day and year first above written.

                                    EASTERN VIRGINIA BANKSHARES, INC.

                                    By: /s/ Joe A. Shearin
                                        ------------------
                                    Joe A. Shearin
                                    Title: President


ATTEST:

/s/ Patricia Gallagher
------------------------
Patricia Gallagher


                                    EMPLOYEE

                                    /s/ Joseph H. James, Jr.  (SEAL)
                                    --------------------------------
                                    Joseph H. James, Jr.
ATTEST:

/s/ Patricia Gallagher
------------------------
Patricia Gallagher

                                        8