<PAGE>

                                                                    EXHIBIT 10.4

                           CHANGE IN CONTROL AGREEMENT

      THIS AGREEMENT is made and entered into effective as of the 17th day of
September, 2003, by and between LESLYE L. GRINDLE (hereinafter "Executive") and
SOUTHERN COMMUNITY BANK, (the "Bank"), a subsidiary of Southern Community
Bancshares, Inc. ("SCBI"), a Georgia bank holding company.

                              W I T N E S S E T H:

      WHEREAS, Executive is employed on an at-will basis as the Chief Financial
Officer of the Bank and SCBI;

      WHEREAS, the parties hereto desire to enter into an agreement setting
forth severance benefits payable to Executive under certain conditions following
a change in control of SCBI;

      NOW, THEREFORE, for and in consideration of the mutual premises and
covenants herein contained, the parties hereto agree as follows:

      1.    CHANGE IN CONTROL. In the event of a "Change in Control", as defined
herein, during Executive's employment and if as a result of any such Change in
Control Executive either (i) is terminated (except "for Cause" as defined
below), during the one-year period after the Change in Control becomes
effective, from her employment hereunder, or (ii) has a "Change in Duties or
Salary" as defined below and resigns, during the one-year period after the
Change in Control becomes effective, as a result of such change, then Executive
shall be entitled to receive severance compensation in an amount equal to one
hundred percent (100%) of her base salary then in effect and any other amounts
owing to Executive at the time of such termination date, which shall be paid in
a lump sum within 14 days following the date of termination or resignation.

            For purposes of this Section 1, "Change in Control" shall mean:

                  (i)   any transaction, whether by merger, consolidation, asset
sale, tender offer, reverse stock split or otherwise, which results in the
acquisition of beneficial ownership (as such term is defined under rules and
regulations promulgated under the Securities Exchange Act of 1934, as amended)
by any person or entity or any group of persons or entities acting in concert,
with the exception of SCBI, SCBI's Board of Directors or SCBI's shareholders, of
50% or more of the outstanding shares of common stock of SCBI or the Bank;

                  (ii)  the sale of all or substantially all of the assets of
the Bank; or

                  (iii) the liquidation of the Bank.

<PAGE>

            For purposes of this Agreement, "Change in Duties or Salary" of
Executive shall mean any of: (i) a change in duties and responsibilities of
Executive from those duties and responsibilities of Executive for the Bank in
effect at the time a Change in Control occurs, which change results in the
assignment of duties and responsibilities inferior to those duties and
responsibilities of Bank at the time such Change in Control occurs; (ii) a
reduction in rate of annual salary from such rate in effect at the time of
Change in Control; or (iii) a change in the place of assignment of Bank from
Fayetteville, Georgia, to any other city or geographical location that is
located further than 30 miles from the principal office of the Bank in
Fayetteville, Georgia.

            For purposes of this Agreement, "for Cause" shall mean for any of
the following reasons:

                  (a)   failure of Executive to follow reasonable written
instructions or policies of the Board of Directors of the Bank;

                  (b)   gross negligence or willful misconduct of Executive
materially damaging to the business of the Bank;

                  (c)   conviction of Executive of a crime involving breach of
trust, moral turpitude, theft or fraud;

                  (d)   the failure by the Executive to perform substantially
her duties other than any failure resulting from incapacity due to physical or
mental illness;

                  (e)   willful commission of (i) acts involving dishonesty or
fraud or (ii) acts causing harm to the Bank;

                  (f)   a willful misrepresentation by the Executive to the
stockholders or the Board of Directors of the Bank which causes substantial
injury to the Bank; or

                  (g)   a request by any state or federal authority regulating
the Bank that the Executive be removed from her office as Chief Financial
Officer of the Bank.

            For purposes of this Agreement, no act, or failure to act, on the
part of the Executive shall be considered "willful" unless done, or omitted to
be done, by her not in good faith and without reasonable belief that her action
or omission was in the best interest of the Bank and the stockholders of the
Bank. The Bank shall notify the Executive in writing of the specific reasons for
the termination for "Cause" and the Executive will be allowed thirty (30) days
to reply in writing to the accusation before any termination for "Cause". If
Executive is terminated for "Cause," she shall receive only her salary and any
other amounts due to her from the Bank (whether pursuant to benefit plans or
otherwise) through the date of termination.

<PAGE>

      2.    ENTIRE AGREEMENT. This Agreement constitutes the entire agreement
between the parties hereto regarding its subject matter, and supersedes and
replaces any prior agreement between the parties.

      3.    NO EMPLOYMENT RIGHTS. No provision or condition contained in this
Agreement shall be construed as creating specific employment rights of Executive
(it being understood that Executive's employment is on an at-will basis) or,
except for the payments contemplated herein, as limiting the right of the Bank
to discharge Executive with or without cause.

      4.    ASSIGNMENT. Neither of the parties hereto may assign this Agreement
without the prior written consent of the other party hereto.

      5.    SEVERABILITY. Each section and subsection of this Agreement
constitutes a separate and distinct understanding, covenant and provision
hereof. In the event that any provision of this Agreement shall finally be
determined to be unlawful, such provision shall be deemed to be severed from
this Agreement, but every other provision of this Agreement shall remain in full
force and effect.

      6.    GOVERNING LAW. This Agreement shall in all respects be interpreted,
construed and governed by and in accordance with the laws of the State of
Georgia.

      7.    RIGHTS OF THIRD PARTIES. Nothing herein expressed or implied is
intended to or shall be construed to confer upon or give to any person, firm or
other entity, other than the parties hereto and their permitted assigns, any
rights or remedies under or by reason of this Agreement.

      8.    AMENDMENT. This Agreement may not be amended orally but only by an
instrument in writing duly executed by the parties hereto.

      9.    NOTICES. Any notice or other document or communication permitted or
required to be given to Executive pursuant to the terms hereof shall be deemed
given if personally delivered to Executive or sent to her postage prepaid, by
registered or certified mail, at ___________________________________, or at any
such other address as Executive shall have notified the Bank in writing. Any
notice or other document or other communication permitted or required to be
given to the Bank pursuant to the terms hereof shall be deemed given if
personally delivered or sent to the Bank, postage prepaid, by registered or
certified mail, at 525 North Jeff Davis Drive, Fayetteville, Georgia 30214, or
at such other address as the Bank shall have notified Executive in writing.

      10.   WAIVER. The waiver by either party hereto of a breach of any
provision of this Agreement by the other shall not operate or be construed as a
waiver of any subsequent breach of the same or any other provision of this
Agreement by the breaching party.

<PAGE>

      IN WITNESS WHEREOF, the parties hereto have caused this Agreement to be
duly executed and delivered as of the day and year first above written.

                                             SOUTHERN COMMUNITY BANK

[BANK SEAL]

                                             By: _______________________________
                                                        Gary D. McGaha
                                                       President and CEO

Attest:

_____________________________
Pam Davis, Secretary

                                             EXECUTIVE

                                             By: _______________________________
                                                       Leslye L. Grindle