<PAGE>

                                                                    EXHIBIT 10.4

                           CHANGE OF CONTROL AGREEMENT

     THIS AGREEMENT ("Agreement") dated April 30, 2006, by and between
Mid-America Bancshares, Inc., a Tennessee corporation (the "Company"), and James
S. Short ("Executive").

                                   WITNESSETH:

     WHEREAS, Executive has been effective in his service to Bank of the South
("Bank"), and the Company recognizes the valuable services that Executive has
rendered and desires to be assured that Executive will continue his active
participation in the business of the Bank and also to serve as executive vice
president of the Company; and

     WHEREAS, Executive is willing to continue to serve both Bank and Company
but desires assurance that, in the event of any change of control of the
Company, or the sale of any of its subsidiary banks (each a "Subsidiary Bank")
after the consummation of the share exchange ("Share Exchange") between the
Company on the one hand and Bank of the South and PrimeTrust Bank on the other,
he will continue to have the responsibility and status he has earned.

     NOW, THEREFORE, in consideration of the premises and the parties's
agreements herein contained, the Company and the Executive hereby agree that:

     1. In order to protect Executive against the possible consequences of a
change of control (as such term is hereinafter defined) of Company or any
Subsidiary Bank and thereby to induce Executive to continue to serve as an
executive officer of the Company and the Bank, the Company agrees that if
control of the Company or any Subsidiary Bank is changed, Executive shall be
entitled to receive within ten (10) business days of the change of control, a
lump sum payment in cash in the amount of $1.00 less than three times the
disqualified individual's base amount of compensation (Internal Revenue Code
Section 280G(b)(2)(A)(ii)). The term "base amount," referred to in this
provision, is the Employee's annualized includible compensation for a base
period, consisting of the most recent five tax years ending before the date on
which the ownership or control of the Company or the Subsidiary Bank changed, or
the portion of this period during which the Employee performed personal services
for or was an employee of the Company or Bank (Code Section 280G(b)(3) and (d)).
"Annualized includible compensation" for the base period is the average annual
compensation that was payable by the Company or the Bank and includible by the
Employee in gross income for the tax years of the base period (Code Section
280G(d)(1)).

     2. Even in the event of termination of Executive's service to the Company
or the Bank, Executive's benefits hereunder shall be considered severance pay in
consideration of his past service, and pay in consideration of his continued
service from the date hereof, and his entitlement thereto shall not be governed
by any duty to mitigate his damages by seeking further employment nor offset by
any compensation which he may receive from future employment.

<PAGE>

     3. As used herein the term "control" shall mean (a) the ownership (whether
directly, indirectly, beneficially or of record) of shares in excess of 50% of
the then-outstanding shares of Common Stock of the Company by a person or group
of persons (including, without limitation, a corporation, trust, partnership,
joint venture, individual or other entity); or (b) any change in the composition
of the Board of Directors of the Company resulting in a majority of the
directors of the Company not constituting a majority two years hence, provided,
that in making such determination directors who were elected by, or on the
recommendation of, such present majority, shall be excluded. Beneficial
ownership shall include all shares directly or indirectly owned by a person or
group of persons or any affiliates or associates of such person or group of
persons, or shares which such person or group of persons, affiliates or
associates have the right to acquire through the exercise of any option, warrant
or right or otherwise and all shares as to which such person or group of
persons, affiliates or associates, directly or indirectly, have or share voting
power or investment power or both. As used herein, the term the "combined
entity" shall mean Company and the person or groups of persons are combined or
separate legal entities; or (c) the sale or other disposition by the Company of
all or substantially all of the assets of any Subsidiary Bank in one or a series
of transactions and/or the sale or other disposition of as much as 50% of any
Subsidiary Bank's common stock.

     4. The specific arrangements referred to above are not intended to exclude
Executive's participation in other benefits available to executive personnel
generally or to preclude other compensation or benefits as may be authorized by
the Board of Directors from time to time.

     5. The Company agrees to require any successor (whether direct or indirect,
by purchase, merger, consolidation or otherwise) to all or substantially all of
the business and/or assets of the Company, by agreement in form and substance
satisfactory to Executive, to expressly assume and agree to perform this
Agreement in the same manner and to the same extent that the Company would be
required to perform it if no such succession had taken place. Failure of the
Company to obtain such agreement prior to the effectiveness of any such
succession shall be a breach of this Agreement and shall entitle Executive to
compensation from the Company in the same amount and on the same terms as
Executive would be entitled under Section 1 hereof. As used in this Agreement,
"Company" shall mean the Company as hereinbefore defined and any successor to
its business and/or assets as aforesaid which executes and delivers the
agreement provided for in this paragraph 5 or which otherwise becomes bound by
all the terms and provisions of this Agreement by operation of law.

     6. This Agreement shall inure to the benefit of and be enforceable by
Executive's personal or legal representatives, executors, administrators,
successors, heirs, distributees, devisees, and legatees. If Executive should die
while any amount would still be payable to Executive hereunder if Executive had
continued to live, all such amounts, unless otherwise provided herein, shall be
paid in accordance with the terms of this Agreement of Executive's devisee,
legatee or other designee, or, if there be no such designee, to Executive's
estate.


                                       -2-

<PAGE>

     7. Any payment or delivery required under this Agreement shall be subject
to all requirements of the law with regard to withholding, filing, making of
reports, and the like, and Company shall use its best efforts to satisfy
promptly all such requirements.

     8. Prior to change of control as herein defined, this Agreement shall
terminate if Executive shall resign voluntarily, retire, become permanently and
totally disabled, voluntarily take another position requiring a substantial
portion of his time, or die. This Agreement shall also terminate if Executive's
employment as an officer of Company shall have been terminated for any reason by
the Board of Directors of Company as constituted prior to any change of control
of Company as herein defined.

     9. The acquisition of a Subsidiary Bank by any company or entity controlled
by the Company shall not trigger the provisions of this Agreement.

     10. This Agreement shall be effective as of the Effective Time of the Share
Exchange. As used herein, the term "Effective Time" means the date and time that
the Share Exchange shall become effective as a result of the filing of articles
of share exchange with respect to the Share Exchange.

     11. This Agreement shall be construed and enforced under the internal laws
of the State of Tennessee without giving effect to the conflict of laws
principles of such jurisdiction.

     IN WITNESS WHEREOF, this Agreement has been executed on April 30, 2006, to
be effective as set forth in paragraph 10 of this Agreement.

                                        MID-AMERICA BANCSHARES, INC.


                                        By: /s/ Gary L. Scott
                                            ------------------------------------
                                            Gary L. Scott, Chairman and CEO


Agreed and Accepted as to all terms and conditions:

/s/ James S. Short
-------------------------------------
James S. Short, Individually


                                       -3-