<PAGE>

                                                                    EXHIBIT 10.4

                     TERMINATION OF CONSULTING AGREEMENT,
                     -----------------------------------

                    SETTLEMENT AGREEMENT AND MUTUAL RELEASE
                    ---------------------------------------

     This Termination of Consulting Agreement, Settlement Agreement and Mutual
Release (the "Agreement") is made as of May 17, 1999 (the "Effective Date")
between Imperial Bank ("Imperial") and Imperial Financial Group, Inc. ("IFG") on
the one hand (collectively, the "Imperial Parties"), and Second Southern Corp.
("Second Southern"), Ginarra Holdings, Inc. ("Ginarra") and G. Louis Graziadio,
III ("Graziadio") as an individual (the "Graziadio Parties") on the other hand.

RECITALS
--------

     A. Imperial and Second Southern are parties to a Consulting Agreement dated
November 1, 1991. Graziadio is the President, sole shareholder and chief
executive officer of Second Southern. Graziadio is also the chief executive
officer of Ginarra.

     B. Second Southern has made a claim against Imperial for immediate payment
of $4,977,383.03, plus attorneys fees and interest, based upon an alleged
modification of the Consulting Agreement (the "Second Southern Claim"). Ginarra
has made certain claims against Imperial and/or IFG in connection with a
Sublease between Ginarra and IFG dated August 31, 1998 and an Assignment and
Assumption of Lease ("Assignment") between Ginarra and IFG dated August 31,
1998. Graziadio has made certain claims against Imperial and/or IFG for
reimbursement of certain costs incurred in connection with the formation of IFG
and based upon an employment agreement dated September 1, 1998 between Graziadio
and IFG (the "Employment Agreement"). Collectively, the claims of Ginarra and of
Griziadio are referred to as "Graziadio's Claim".

     C. The Board of Directors of Imperial appointed a Special Committee of the
Board to investigate and resolve Second Southern's Claim.

     D. After extensive investigation and deliberation, the Special Committee
determined that it is in the best interests of the Imperial Parties to settle,
upon reasonable terms, Second
<PAGE>

Southern's Claim, Graziadio's Claim and all other claims by any of the Graziadio
Parties against the Imperial Parties in connection with the Consulting
Agreement, the Sublease and the Employment Agreement, and all claims (excluding
any rights to indemnification) of the Graziadio Parties based upon or arising
out of any amounts or obligations owing to and/or claimed by any of the
Graziadio Parties from IFG.

     E.  The parties wish to terminate the Consulting Agreement, the Sublease,
the Assignment and the Employment Agreement, and to settle and compromise all
claims (excluding any rights to indemnification) that have been or could have
been made by the Graziadio Parties, or any of them, against the Imperial
Parties, or any of them, and all claims that have been or could have been made
by the Imperial Parties or any of them against Graziadio Parties or any of them,
based upon or arising out of the Consulting Agreement, the Sublease, the
Assignment and the Employment Agreement or any amounts or obligations owing to
and/or claimed by any of the Graziadio Parties from IFG (excluding any rights to
indemnification) on the terms set forth in this Agreement.

         THEREFORE, in consideration of the recitals and mutual promises and
covenants contained herein, and for other good and sufficient consideration, the
parties agree as follows:

AGREEMENT
---------

     1.  TERMINATION.  The Consulting Agreement, the Sublease, the Assignment
         -----------
and the Employment Agreement and all other agreements, whether oral, written or
otherwise, between the Imperial Parties, or any of them, and the Graziadio
Parties, or any of them, that arise out of or are related to the Consulting
Agreement, the Sublease, the Assignment, the Employment Agreement or any amounts
or obligations owing to or claimed by any of the Graziadio Parties from IFG
(excluding any rights to indemnification), are hereby terminated. This Agreement
is the only agreement between the parties with respect to such matters. As of
the Effective Date, neither party has any remaining rights or obligations
arising out of or based upon the Consulting Agreement, the Sublease, the
Assignment, the Employment Agreement or any other agreement,

                                       2
<PAGE>

whether written or oral, arising out of or relating to such agreements or to
amounts or obligations owing to or claimed by any of the Graziadio Parties from
IFG (excluding any rights to indemnification).

     2.  PAYMENT BY IMPERIAL.  Upon the Effective Date of this Agreement,
         -------------------
Imperial will pay Graziadio the sum of five million dollars ($5,000,000.00) by
electronic or wire transfer in immediately available funds to an account
designated by Graziadio.

     3.  MUTUAL RELEASES
         ---------------

         3.1  The Graziadio Parties' Release of the Imperial Releasees
              --------------------------------------------------------

              3.1.1  The Graziadio Parties, and each of them, hereby release and
forever discharge the Imperial Parties, and each of them, and each of their
respective affiliates, owners, stockholders, divisions, subsidiaries,
predecessors, successors, heirs, assigns, agents, directors, officers, partners,
employees, insurers, representatives, lawyers, or any of them (the "Imperial
Releasees"), of and from any and all manner of action or actions, cause or
causes of action, in law or in equity, suits, debts, liens, liabilities, claims,
demands, damages, loss, cost or expense, of any nature whatsoever, known or
unknown, fixed or contingent (excluding any rights to indemnification) from the
beginning of time to the Effective Date of this Agreement (hereinafter called
"Claims"), that the Graziadio Parties, or any of them, now have or may hereafter
have against the Imperial Releasees, or any of them, arising out of, based upon,
or relating to the Consulting Agreement, the Sublease, the Assignment, the
Employment Agreement or any amounts or obligations owing to or claimed by any of
the Graziadio Parties from IFG, including but not limited to Second Southern's
Claim and Graziadio's Claim. The parties hereto acknowledge and agree that this
is a limited general release, that the parties are releasing only those claims
described above and that except for the Claims specifically set forth above,
none of

                                       3
<PAGE>

the Graziadio Parties are hereby releasing, and hereby expressly reserve, any
and all other Claims, including but not limited to those claims which they hold
in their capacities as shareholders, directors or employees of Imperial or its
parent, Imperial Bancorp.

          3.1.2  ALTHOUGH THIS IS A LIMITED GENERAL RELEASE, THE PARTIES ARE
RELEASING "UNKNOWN" CLAIMS ARISING OUT OF, BASED UPON OR RELATING TO THE
CONSULTING AGREEMENT, THE SUBLEASE, THE ASSIGNMENT, THE EMPLOYMENT AGREEMENT OR
ANY AMOUNTS OR OBLIGATIONS OWING TO OR CLAIMED BY ANY OF THE GRAZIADIO PARTIES
FROM IFG (EXCLUDING ANY RIGHTS TO INDENIFICATION). TO THE EXTENT THE RELEASE OF
SUCH UNKNOWN CLAIMS CONSTITUTES A "GENERAL RELEASE" OF THOSE CLAIMS ONLY, THE
PARTIES AGREE AS FOLLOWS: THE GRAZIADIO PARTIES, AND EACH OF THEM, ACKNOWLEDGE
THAT THEY HAVE BEEN ADVISED BY THEIR LEGAL COUNSEL AND ARE FAMILIAR WITH THE
PROVISIONS OF CALIFORNIA CIVIL CODE SECTION 1542, WHICH PROVIDES AS FOLLOWS:

          "A GENERAL RELEASE DOES NOT EXTEND TO CLAIMS
          WHICH THE CREDITOR DOES NOT KNOW OR SUSPECT
          TO EXIST IN HIS FAVOR AT THE TIME OF EXECUTING
          THE RELEASE, WHICH IF KNOWN BY HIM MUST HAVE
          MATERIALLY AFFECTED HIS SETTLEMENT WITH THE
          DEBTOR."

THE GRAZIADIO PARTIES, BEING AWARE OF SAID CODE SECTION, HEREBY EXPRESSLY WAIVE
ANY RIGHTS THEY MAY HAVE THEREUNDER, AS WELL AS

                                       4
<PAGE>

UNDER ANY OTHER STATUTES OR COMMON LAW PRINCIPLES OF SIMILAR EFFECT.

          3.1.3. The Graziadio Parties, and each of them, represent and warrant
that there has been no assignment or other transfer of any interest in any
Claims which they, or any of them, may have against the Imperial Releasees, or
any of them, and the Graziadio Parties agree to indemnify and hold the Imperial
Releasees, and each of them, harmless from any liability, Claims, demands,
damages, costs, expenses and attorneys' fees incurred by the Imperial Releasees,
or any of them, as a result of any person asserting any such assignment or
transfer. It is the intention of the parties that this indemnity does not
require payment to a third party as a condition precedent to recovery by the
Imperial Releasees, or any of them, against the Graziadio Parties, and each of
them, under this indemnity.

          3.1.4  The Graziadio Parties agree that if they, or any of them,
hereafter commence, join in, or in any manner seek relief through any suit
arising out of, based upon, or relating to any of the Claims released hereunder,
or in any manner assert against the Imperial Releasees, or any of them, any of
the Claims released hereunder, then the Graziadio Parties, and each of them,
shall pay to the Imperial Releasees, and each of them, in addition to any other
damages caused thereby to the Imperial Releasees, all costs and attorneys' fees
incurred by the Imperial Releasees, or any of them, in defending or otherwise
responding to said suit or Claim.

     4.1  The Imperial Parties of the Graziadio Releasees
          -----------------------------------------------

          4.1.1  The Imperial Parties, and each of them, hereby release and
forever discharge the Graziadio Parties, and each of them, and each of their
respective affiliates, owners, stockholders, divisions, subsidiaries,
predecessors, successors, heirs, assigns, agents, directors, officers, partners,
employees, insurers, representatives, lawyers, or any of them (the "Graziadio

                                       5
<PAGE>

Releasees"), of and from any and all Claims from the beginning of time to the
Effective Date of this Agreement, that the Imperial Parties, or any of them, now
have or may hereafter have against the Graziadio Releasees, or any of them,
arising out of, based upon, or relating to the Consulting Agreement, the
Sublease, the Assignment, the Employment Agreement or any amounts or obligations
owing to or claimed by any of the Graziadio Parties from IFG, including but not
limited to, Second Southern's Claim and Graziadio's Claim.

          4.1.2  ALTHOUGH THIS IS A LIMITED GENERAL RELEASE, THE PARTIES ARE
RELEASING "UNKNOWN" CLAIMS ARISING OUT OF, BASED UPON OR RELATING TO THE
CONSULTING AGREEMENT, THE SUBLEASE, THE ASSIGNMENT, THE EMPLOYMENT AGREEMENT OR
ANY AMONTS OR OBLIGATIONS OWING TO OR CLAIMED BY ANY OF THE GRAZIADO PARTIES
FROM IFG (EXCLUDING ANY RIGHTS TO INDEMNIFICATION) TO THE EXTENT THE RELEASE OF
SUCH UNKNOWN CLAIMS CONSTITUTES A "GENERAL RELEASE" OF THOSE CLAIMS ONLY, THE
PARTIES AGREE AS FOLLOWS: THE GRAZIADIO PARTIES, AND EACH OF THEM, ACKNOWLEDGE
THAT THEY HAVE BEEN ADVISED BY THEIR LEGAL COUNSEL AND ARE FAMILIAR WITH THE
PROVISIONS OF CALIFORNIA CIVIL CODE SECTION 1542, WHICH PROVIDES AS FOLLOWS:

     "A GENERAL RELEASE DOES NOT EXTEND TO CLAIMS WHICH THE CREDITOR
     DOES NOT KNOW OR SUSPECT TO EXIST IN HIS FAVOR AT THE TIME OF
     EXECUTING THE RELEASE, WHICH IF KNOWN BY HIM MUST HAVE MATERIALLY
     AFFECTED HIS SETTLEMENT WITH THE DEBTOR."

                                       6
<PAGE>

THE GRAZIADIO PARTIES, BEING AWARE OF SAID CODE SECTION, HEREBY EXPRESSLY WAIVE
ANY RIGHTS THEY MAY HAVE THEREUNDER, AS WELL AS UNDER ANY OTHER STATUTES OR
COMMON LAW PRINCIPLES OF SIMILAR EFFECT.

          4.1.3. The Imperial Parties, and each of them, represent and warrant
that there has been no assignment or other transfer of any interest in any
Claims which they, or any of them, may have against the Graziadio Releasees, or
any of them, and the Imperial Parties agree to indemnify and hold the Graziadio
Releasees, and each of them, harmless from any liability, Claims, demands,
damages, costs, expenses and attorneys' fees incurred by the Graziadio
Releasees, or any of them, as a result of any person asserting any such
assignment or transfer. It is the intention of the parties that this indemnity
does not require payment to a third party as a condition precedent to recovery
by the Graziadio Releasees, or any of them, against the Imperial Parties, and
each of them, under this indemnity.

          4.1.4. The Imperial Parties agree that if they or either of them
hereafter commence, join in, or in any manner seek relief through any suit
arising out of, based upon, or relating to any of the Claims released hereunder,
or in any manner asserts against the Graziadio Releasees, or any of them, any of
the Claims released hereunder, then the Imperial Parties, shall pay to the
Graziadio Releasees, and each of them, in addition to any other damages caused
thereby to the Graziadio Releasees all costs and attorneys' fees incurred by the
Graziadio Releasees, or any of them, in defending or otherwise responding to
said suit or Claim.

     5.  NO PENDING ACTIONS/COVENANT NOT TO SUE.  Each of the Imperial Parties
         --------------------------------------
represents to each of the Graziadio Parties, and each of the Graziadio Parties
represents to each of the Imperial Parties, that such party does not presently
have on file any complaints,

                                       7
<PAGE>

charges or claims (civil, administrative, or criminal) against the other party,
in any court or administrative forum, or before any governmental agency or
entity. Each such party represents that he or it will not hereafter file any
complaints, charges or claims (civil, administrative, or criminal) against the
other party, with any administrative, state, federal or governmental entity,
agency board or court based on any of the Claims released hereunder except to
enforce the agreement.

     6.  FURTHER PROVISIONS
         ------------------

        6.1.  The parties acknowledge that, in entering into this Agreement,
they have had the opportunity to conduct a full and adequate investigation of
the facts and claims in issue and have not relied upon any representations,
warranties, promises or conditions not specifically set forth in this Agreement.
All of the parties represent that they are represented by legal counsel and have
been fully advised as to the meaning and consequence of all of the terms and
provisions of this Agreement.

        6.2.  This Agreement shall be construed and enforced under and in
accordance with the laws of the State of California, except that the provisions
of Civil Code Section 1654 shall not be invoked to affect or alter the
interpretation of this Agreement or any portion thereof.

        6.3.  Any action, whether in contract, tort or otherwise based upon or
arising out of the negotiation, terms or performance of this Agreement, must be
brought in the Superior Court of the County of Los Angeles, California.

        6.4.  In any action arising out of the negotiation, terms or performance
of this Agreement, the prevailing party shall be entitled to recover its
reasonable attorneys' fees and litigation expenses from the other party.

                                       8
<PAGE>

        6.5.  No waiver of the breach of any of the terms or provisions of this
Agreement shall be a waiver of any preceding or succeeding breach of the
Agreement or any other provisions of it.

        6.6.  Except for the provisions of paragraphs 2, 3 and 4, if any one or
more of the provisions contained in this Agreement shall, for any reason, be
held to be invalid, void, illegal, or unenforceable in any respect, such
invalidity, voidness, illegality or unenforceability shall not affect any other
provision of this Agreement, and the remaining portions shall remain in full
force.

        6.7.  This Agreement contains the entire agreement between the parties
hereto with respect to the subject matter of this Agreement. This Agreement
supersedes all prior agreements, negotiations and oral understandings, if any.
This Agreement may not be modified, amended or supplemented except by an
instrument in writing signed by each of the parties.

        6.8. This Agreement is entered into for the settlement and compromise of
disputed claims and shall never be treated as an admission of liability or a
statement or admission as to any particular contention or issue by any party for
any purpose, other than the liability of a party to perform in accordance with
its terms after the execution of this Agreement.

        6.9.  Each of the parties shall execute and deliver any and all
additional papers, documents and other assurances and shall do any and all acts
and things reasonably necessary in connection with the performance of their
obligations under, and to carry out the intent of the parties to this Agreement.

        6.10.  Each and every covenant, term, provision, and agreement in this
Agreement shall be binding upon and inure to the benefit of the successors and
assigns of the respective parties to this Agreement.

                                       9
<PAGE>

     6.11.  Some paragraphs of this Agreement have titles and some do not. The
fact that some paragraphs hereof do not have titles shall have no significance.
The titles are included for ease of reference only and shall not be used to
construe the meaning of this Agreement.

     6.12.  All parties and corporate officers signing this Agreement represent
and warrant that the execution, delivery and performance of this Agreement has
been duly and validly authorized by all necessary corporate action on behalf of
such parties and that this Agreement represents a valid and binding obligation
of such parties enforceable in accordance with its terms.

     6.13.  All notices with respect to this Agreement shall be sent in writing,
by first-class mail, to the parties at the following addresses:

       Second Southern Corporation                Imperial Bank
       2325 Palos Verdes Drive West, Suite 211    9920 South La Cienega Blvd.
       Palos Verdes Estates, CA 90274             Inglewood, CA 90301
       Attn: Louis Graziadio, President           Attn: Richard Baker, Esq.
                                                  Fax: (310) 417 5695

       Louis Graziadio                            Imperial Financial Group, Inc.
       2325 Palos Verdes Drive West, Suite 211    9920 South La Cienega Blvd.
       Palos Verdes Estates, CA 90274             Inglewood, CA 90301
                                                  Attn: Richard Baker, Esq.
                                                  Fax: (310) 417 5695

       Ginarra Holdings, Inc.
       2325 Palos Verdes Drive West, Suite 211
       Palos Verdes Estates, CA 90274
       Attn: Louis Graziadio, President and
       Chief Executive Officer

     6.14.  REFERENCE PROVISION: a. Other than the exercise of provisional
            -------------------
remedies (any and all of which may be initiated pursuant to applicable law),
each controversy, dispute or claim ("Claim") between the parties arising out of
or relating to this Agreement, which is not settled in writing within ten days
after the "Claim Date" (defined as the date on which a

                                      10
<PAGE>

party gives written notice to all other parties that a controversy, dispute or
claim exists), will be settled by a reference proceeding in Los Angeles,
California in accordance with the provisions of Section 638 et seq. of the
                                                            -- ---
California Code of Civil Procedure("CCP"), or their successor section, which
shall constitute the exclusive remedy for the settlement of any Claim, including
whether such Claim is subject to the reference proceeding and the parties waive
their rights to initiate any legal proceedings against each other in any court
or jurisdiction other than the Superior Court of Los Angeles (the "Court"). The
referee shall be a retired Judge selected by mutual agreement of the parties,
and if they cannot so agree within thirty days after the Claim Date, the referee
shall be selected by the Presiding Judge of the Court. The referee shall be
appointed to sit as a temporary judge, as authorized by law. The referee shall
(a) be requested to set the matter for hearing within sixty (60) days after the
Claim Date and (b) try any and all issues of law or fact and report a statement
of decision upon them, if possible, within ninety (90) days of the Claim Date.
Any decision rendered by the referee will be final, binding and conclusive and
judgement shall be entered pursuant to CCP 644 in the Court and such judgement
shall be appealable under CCP 645. All discovery permitted by this Agreement
shall be completed no later than fifteen (15) days before the first hearing date
established by the referee. The referee may extend such period in the event of a
party's refusal to provide requested discovery for any reason whatsoever,
including, without limitation, legal objections raised to such discovery or
unavailability of a witness due to absence or illness. No party shall be
entitled to "priority" in conducting discovery. Depositions may be taken by
either party upon seven (7) days written notice, and, request for production or
inspection of documents shall be responded to within ten (10) days after
service. All disputes relating to discovery which cannot be resolved by the
parties shall be submitted to the referee whose decision shall be final and
binding upon the parties.

                                      11
<PAGE>


     b.  The referee shall be required to determine all issues in accordance
with existing case law and the statutory laws of the State of California. The
rules of evidence applicable to proceedings at law in the State of California
will be applicable to the reference proceeding. The referee shall be empowered
to enter equitable as well as legal relief, to provide all temporary and/or
provisional remedies and to enter equitable orders that will be binding upon the
parties. The referee shall issue a single judgment at the close of the reference
proceeding which shall dispose of all of the claims of the parties that are the
subject of the reference. The parties hereto expressly reserve the right to
contest or appeal from the final judgment or any appealable order or appealable
judgment entered by the referee. The parties expressly reserve the right to
findings of fact, conclusions of law, a written statement of decision, and the
right to move for a new trial or a different judgment, which new trial, if
granted, is also to be a reference proceeding under this provision.

        6.15  This Agreement may be executed in two or more counterparts, each
of which shall be deemed an original, but all of which together shall constitute
one and the same instrument.

                                      12
<PAGE>

     IN WITNESS WHEREOF, the parties hereto have executed this Agreement on the
dates indicated below:

IMPERIAL BANK                            SECOND SOUTHERN CORP.


By: /s/ Daniel R. Mathis                 By: /s/ G. Louis Graziadio, III
   -----------------------------            ---------------------------------
Title:    President                      Title:  President
      --------------------------                -----------------------------

By: /s/ Richard M. Baker                 Dated:   5-14-99
   -----------------------------                -----------------------------
Title:  SVP & Secretary
      --------------------------
Dated: 5-17-99
      --------------------------

IMPERIAL FINANCIAL GROUP, INC.           GINARRA HOLDINGS, INC.

By: /s/ Richard M. Baker                 By:  /s/ G. Louis Graziadio, III
   -----------------------------            ----------------------------------
Title:   Secretary                       Title:  President and Chief Executive
      --------------------------                 Officer
                                                ------------------------------
Dated:  5-17-99                          Dated:  5-14-99
      --------------------------                ------------------------------

                                             /s/ G. Louis Graziadio, III
                                         -------------------------------------
                                         G. Louis Graziadio, III
                                         as an Individual and on behalf of all
                                         Graziadio Parties

                                         Dated:  5-14-99
                                               -------------------------------


                                      13