<PAGE>

                                                                    EXHIBIT 10-4

                                    AGREEMENT


         THIS AGREEMENT (this "Agreement") is entered into as of the 24th day of
January, 2005 by and between James A. Taylor, Jr. ("Executive") and The Banc
Corporation, a Delaware corporation (the "Parent").

                              W I T N E S S E T H:

         WHEREAS, Executive is employed by the Parent as President and Chief
Operating Officer, pursuant to that Employment Agreement dated September 19,
2000, attached hereto as Exhibit A (the "Contract") and is a director of The
Bank, a wholly owned subsidiary of the Parent (the "Bank");

         WHEREAS, a group of potential investors led by C. Stanley Bailey, an
experienced banking executive (the "New Management Team"), has approached the
Parent with a proposal that includes a new strategic plan for the Parent, a
significant equity infusion and the employment of the New Management Team (the
"Proposal");

         WHEREAS, the Proposal was presented to the Parent's Board of Directors
(the "Board") and the Board referred the Proposal to an independent special
committee (the "Committee") for consideration;

         WHEREAS, the Committee and the Board have employed the New Management
Team and adopted the Proposal, and the transactions contemplated by the Proposal
have been closed;

         WHEREAS, this has resulted in the Executive's losing all current
positions with the Parent and the Bank other than as expressly set forth below;

         WHEREAS, the Parties hereto agree that this entitles the Executive to
certain rights and benefits under the Contract;


                                       1
<PAGE>

         WHEREAS, the meaning of certain of the Contract's provisions for the
terms, conditions, extent and timing of certain of those rights and benefits
could be the subject of differing constructions and the Parties hereto seek to
resolve any ambiguities or potential disagreements; and

         WHEREAS, Executive and the Parent desire to fully settle, compromise
and resolve forever all claims, disputes, and potential claims related to the
Contract.

         NOW, THEREFORE, in consideration of the premises, the mutual promises
and covenants contained herein and other good and valuable consideration, the
receipt and sufficiency of which are hereby acknowledged, the parties hereto do
hereby agree as follows:

         1. RESIGNATION OF EMPLOYMENT. Other than as a director of the Parent,
as requested, the Executive hereby resigns, effective immediately, all
positions, whether as director, officer, trustee or otherwise, that he currently
holds with the Parent, the Bank, and any companies, plans or trusts affiliated
with or sponsored by either.

         2. EMPLOYMENT. The Parent agrees that it will maintain the Executive in
its employ to the extent, but only to the extent, necessary for Executive to be
able to exercise any stock options to purchase stock of the Parent for the full
term of such option.

         3. PAYMENT AND OTHER BENEFITS.

         (a) In lieu of any payments that Executive would be entitled pursuant
to Subsection 4(c)(i) of the Contract, the Executive will receive $1,382,872.17
(less any required withholding).

         (b) The Executive will continue to be entitled to the insurance
benefits as provided in Subsection 4(c)(iii) of the Contract; and the
restrictions on his outstanding incentive awards shall lapse and shall become
vested and exercisable during the full term of such award (so long as Executive
continues to be employed by the Parent) and his benefits under deferred
compensation arrangements shall become 100% vested as provided in Subsection
4(c)(iv) of the Contract including the stock in his ESOP account.


                                       2
<PAGE>

         (c) The payments and benefits for which provision is made in
subparagraphs (a) and (b) above are absolute, subject to no contingencies and
will survive the Executive's death or disability.

4.       COVENANTS OF EXECUTIVE.

         (a) General. Executive agrees that for so long as he has any rights or
obligations under this Agreement, he shall continue to conduct himself in a
professional manner that is supportive of the business of the Parent and the
Bank. Without limiting the generality of the foregoing, Executive shall provide
reasonable cooperation in communicating with investors and employees regarding
the transition to the New Management Team.

         (b) Confidential Information. Without limiting the foregoing, Executive
shall not, at any time use or disclose any Confidential Information of the
Parent or the Bank, except in fulfillment of obligations hereunder, for so long
as the pertinent information or data remain Confidential Information, whether or
not the Confidential Information is in written or tangible form. "Confidential
Information" shall mean any information, including but not limited to technical
or non-technical data, a formula, a pattern, a compilation, a program, a device,
a method, a technique, a drawing, a process, financial data, financial plans,
product plans, information on customers, or a list of actual or potential
customers or suppliers, which: (i) derives economic value, actual or potential,
from not being generally known to, and not being readily ascertainable by proper
means by, other persons who can obtain economic value from its disclosure or
use, and (ii) is the subject of efforts that are reasonable under the
circumstances to maintain its secrecy.

         (c) SEC Reporting/Insider Trading Compliance. Executive will cooperate
with the Parent by providing information with respect to all reports required to
be filed by the Parent with the Securities and Exchange Commission as they
relate to required information with respect to Executive. Further, Executive
will remain in compliance with the terms of the Parent's insider trading
program, as such program is applicable to him following the date hereof, with
respect to purchases and sales of the Parent's stock.


                                       3
<PAGE>

(d)      Noncompetition.

                  (i) So long as Executive is receiving any benefits under this
         Agreement, or is providing services to the Parent or the Bank, whether
         as a director, consultant, employee or otherwise, except as
         contemplated by Section 2 of this Agreement, then: Executive agrees
         that he shall not, individually or as an employee, consultant, partner,
         officer, director or shareholder or in any other capacity whatsoever of
         or for any person, firm, partnership, or corporation other than the
         Parent, the Bank or affiliates of either, work as an employee or
         consultant of a financial institution whose deposits are insured by the
         Federal Deposit Insurance Corporation ("FDIC"), or own, manage,
         operate, control or participate in the ownership (other than passive
         ownership of five percent or less of the equity interest of a publicly
         traded entity), management, operation or control of, any business
         headquartered in Alabama or Florida that is in competition with or may
         compete with the then-current business of the Parent, the Bank or
         affiliates of either wherever located.

                  (ii) Anything to the contrary in the immediately preceding
         clause (i) notwithstanding, the Parties hereto acknowledge that the
         Executive is contemplating establishing a de novo bank or banks in the
         following counties in Florida: Bay, Okaloosa, Escambia, Walton and
         Leon, and that the Executive may do so and be an investor, director and
         officer in any such bank provided that Executive resign his position as
         a member of the Parent's Board immediately upon initiating any such
         undertaking.

                  (iii) Executive agrees, further, that he will not, and will
         not cause or influence any other party to, file an objection or appeal
         with any governmental regulatory agency in an attempt to hinder the
         entry by the Parent or the Bank or affiliates of either into any new
         market whether by branching, establishment of a de novo institution, or
         otherwise.

         (e) Nonsolicitation. Executive agrees that so long as he has any rights
or obligations under this Agreement and for one year thereafter, he shall not
either directly or indirectly solicit, induce,


                                       4
<PAGE>

recruit or encourage any of the Parent's employees or consultants to terminate
their relationship with the Parent, or attempt to solicit, induce, recruit,
encourage or take away employees or consultants of the Parent, either for
himself or for any other person or entity. Further, Executive agrees that he
shall not at any time use any Confidential Information of the Parent to
negatively influence any of the Parent's clients or customers from purchasing
Parent or Bank products or services or to solicit or influence or attempt to
influence any client, customer or other person either directly or indirectly, to
direct his or its purchase of products and/or services away from the Parent or
the Bank to any person, firm, corporation, institution or other entity in
competition with the business of the Parent or the Bank.

         (f) Standstill Agreement. Without the prior written consent of the
Parent, Executive shall not sell, offer to sell, or agree to sell, directly or
indirectly, any shares of securities of the Parent owned or controlled by
Executive for one year.

         5. BREACH OF THIS AGREEMENT. Executive acknowledges that upon material
breach of any provision of this Agreement, the Parent and the Bank may sustain
irreparable harm from such breach, and, therefore, Executive agrees that in
addition to any other remedies which the Parent or the Bank may have for any
breach of this Agreement or otherwise, the Parent may be entitled to obtain
equitable relief including specific performance, injunctions and restraining
Executive from committing or continuing any such violation of this Agreement.

         6. APPLICABLE LAW. This Agreement and the rights and obligations of the
parties hereunder shall be construed, interpreted and enforced in accordance
with, and governed by the laws of, the State of Alabama, applied without giving
effect to any conflict-of-laws principles.

         7. CONSIDERATION ACKNOWLEDGMENT. Executive understands and acknowledges
that this Agreement provides Executive with consideration in addition to
anything of value to which Executive was already entitled.

         8. BINDING EFFECT. This Agreement shall inure to the benefit and be
binding upon the Parties and their respective heirs, representatives, successors
and assigns.


                                       5
<PAGE>

         9. SEVERABILITY AND SAVINGS PROVISION. In the event that any one or
more of the provisions contained herein shall for any reason be held to be
unenforceable in any respect under the law or any state or of the United States
of America, such unenforceability shall not affect any other provision of this
Agreement, but, with respect only to that jurisdiction holding the provision to
be unenforceable, this Agreement shall then be construed as if such
unenforceable provision or provisions had never been contained herein.

         10. ENTIRE AGREEMENT. This Agreement constitutes the complete
understanding between Executive, the Parent and the Bank and supersedes any and
all prior or contemporaneous agreements, promises, or inducements, whether oral
or written, concerning these subject matters (including without limitation any
existing employment agreement).

         11. NONDEFAMATION. Executive agrees that he will not slander or libel
the Parent, the Bank or the affiliates of either, or any of their directors,
officers or employees concerning either their conduct regarding his separation
or their conduct at any time during his period of employment with the Parent,
the Bank or the affiliates of either.


                                       6
<PAGE>

         IN WITNESS WHEREOF, the Parent has caused this Agreement to be executed
in its corporate name by its duly authorized officer, and Executive has hereunto
set his hand, and the parties hereto have caused this Agreement to be dated as
of the date first above written.

                                        THE BANC CORPORATION


                                        By:  /s/ Michael E. Stephens
                                             -----------------------------------
                                                     Michael E. Stephens
                                                Chairman of the Compensation
                                             Committee of The Banc Corporation's
                                                     Board of Directors




                                            /s/ James A. Taylor, Jr.  (L.S.)
                                        ----------------------------------------
                                                       EXECUTIVE


                                        Executed by Executive on the 24th day
                                        of January, 2005.


                                                  /s/ Robert E. Garner
                                        ----------------------------------------
                                                         WITNESS


                                       7