<PAGE>   1
EXHIBIT 10.23


                         ESOP LOAN AND PLEDGE AGREEMENT
                                     BETWEEN
                           TAYLOR CAPITAL GROUP, INC.
                                       AND
                        TAYLOR CAPITAL GROUP, INC. PROFIT
                   SHARING AND EMPLOYEE STOCK OWNERSHIP TRUST
<PAGE>   2
                         ESOP LOAN AND PLEDGE AGREEMENT

         THIS ESOP LOAN AND PLEDGE AGREEMENT is dated as of November 25, 1998,
by and between Taylor Capital Group, Inc., a Delaware corporation (the
"Corporation"), and Cole Taylor Bank, not in its individual or corporate
capacity, but as trustee under the Taylor Capital Group, Inc. Profit Sharing and
Employee Stock Ownership Trust (the "Borrower" or the "ESOT").

         WHEREAS, the Corporation desires to lend Borrower the sum of Five
Hundred Seventy Six Thousand Dollars ($576,000) (the "ESOP Loan") for the
purpose of enabling the Borrower to purchase from Jeffrey W. Taylor and Cindy
Taylor Bleil (the "Sellers"), 24,000 shares of common stock of the Corporation
(the "Common Stock") pursuant to that certain Stock Purchase Agreement dated as
of the date hereof, by and between the Sellers and the ESOT (the "Purchase
Agreement"); and

         WHEREAS, the parties intend that the ESOP Loan shall constitute an
"exempt loan" within the meaning of Section 4975(d)(3) of the Internal Revenue
Code of 1986, as amended (the "Code"), Treasury Regulation Section 54.4975-7(b),
Section 408(d)(3) of the Employee Retirement Income Security Act of 1974, as
amended ("ERISA"), and Department of Labor Regulation Section 2550.408b-3
(collectively, the "Exempt Loan Rules") and an "Acquisition Loan" within the
meaning set forth in the Plan.

         NOW, THEREFORE, in consideration of the promises herein contained, and
each intending to be legally bound hereby, the parties agree as follows:

SECTION I. DEFINITIONS.

         As used herein:

         "Closing" has the meaning set forth in Paragraph 3.01 hereof.

         "Code" has the meaning set forth in the Preamble.

         "Collateral" means the 24,000 shares of Common Stock, represented by
certificate no. __ that will be acquired by the Borrower with the proceeds of
the loan made hereunder, less those shares which are released pursuant to
Section 8.01.

         "ERISA" has the meaning set forth in the Preamble.

         "ESOP Loan" means the loan described in the Preamble and Section II.

         "ESOP Note" has the meaning set forth in Paragraph 2.01 hereof.

         "Event of Default" has the meaning set forth in Paragraph 7.01 of this
Agreement.


                                       1
<PAGE>   3
         "Laws" means all ordinances, statutes, regulations, orders,
injunctions, writs or decrees of any government or political subdivision or
agency thereof, or of any court or similar entity established by any thereof.

         "Obligations" means the obligation of the Borrower to pay the principal
of and interest on the ESOP Note in accordance with the terms thereof and to
perform all obligations of Borrower arising out of or under or in respect of
this Agreement, as the same may be amended, modified or supplemented from time
to time.

         "Person" means any individual, corporation, partnership, association,
joint-stock company, trust, unincorporated organization, joint venture, court or
government or political subdivision or agency thereof.

         "Plan" means the Taylor Capital Group, Inc. Profit Sharing and Employee
Stock Ownership Plan, effective as of October 1, 1996, as amended.

SECTION II.  THE LOAN.

         2.01 General Terms, Subject to the terms and conditions of this
Agreement, the Corporation agrees to make a loan to the Borrower on the Closing
in the principal amount of Five Hundred Seventy Six Thousand Dollars ($576,000)
evidenced by the promissory note in the form attached hereto (the "ESOP Note")
and payable as provided in the ESOP Note and in Paragraph 2.02. At the Closing,
the Borrower shall execute and deliver the ESOP Note to the Corporation.

         2.02 Payments; Prepayments. On December 15th of each year, while any
principal balance of the ESOP Loan remains outstanding, commencing December 15,
1999, Borrower shall make a payment to the Corporation comprised of equal
principal payments in the amount of One Hundred Fifteen Dollars ($115,000), and
the interest accrued on the unpaid principal balance. If not sooner paid, the
entire principal amount of the ESOP Loan outstanding and all interest accrued
thereon shall be paid in full on or before December 15, 2003.

         The Borrower may at any time prepay the principal amount of the ESOP
Loan outstanding in whole or in part without premium or penalty; provided,
however, that no repayment or prepayment of the ESOP Loan shall be required or
permitted if it would adversely affect (a) the qualification of the Plan or the
Borrower under the Code or ERISA, or (b) the status of the ESOP Loan as an
"exempt loan" as such term is described in Section 4975 (d)(3) of the Code. If
Borrower prepays the entire principal amount of the ESOP Loan outstanding, such
payment shall also include the interest due on the ESOP Loan outstanding to the
date of such prepayment. Each prepayment shall be applied first to the accrued
but unpaid interest on the ESOP Loan and then to reduce the outstanding
principal balance of the ESOP Loan. If any prepayment prepays less than the
entire outstanding principal balance of the Loan, Borrower shall continue to
make payments pursuant to this Section 2.02 until the entire principal balance
of, and all accrued but unpaid interest on, the ESOP Loan has been paid in full.
Any such prepayments shall be applied against the unpaid installments of
principal in the inverse order of their maturity.


                                       2
<PAGE>   4
         2.03 Interest Rate. The outstanding principal amount of the ESOP Loan
shall bear interest at the annual rate of 7.75%. Interest payments shall be made
on the same dates as payments of principal under Section 2.02.

                    If, at any time, the interest to be paid hereunder shall be
deemed by any competent court of law, governmental agency or tribunal to exceed
the maximum rate of interest permitted by applicable Laws, including Treasury
Regulations governing exemptions from prohibited transactions under Section 4975
of the Code, then, for such time as such interest payments are deemed excessive,
their application shall be suspended and there shall be charged instead the
maximum rate of interest permissible under such Laws.

         2.04 Payments to Corporation. All sums payable to the Corporation
hereunder shall be paid directly to the Corporation in immediately available
funds. All payments shall be applied first against any interest due and payable,
and then against the outstanding principal balance of the ESOP Loan.

         2.05 Source of Funds. The ESOP Note and any interest thereon shall be
payable from (a) contributions (other than contributions of "employer
securities" as such term is described in Section 409(l)) of the Code) made to
the Borrower in accordance with the ESOP to enable the Borrower to pay its
obligations under the ESOP Note, earnings attributable to such contributions and
earnings on the Common Stock held in the unallocated share suspense account and
acquired with the proceeds of the ESOP Loan, including but not limited to, cash
dividends received by Borrower with respect to such shares of Common Stock,
and/or (b) the proceeds of a subsequent loan made to repay the ESOP Loan. No
contributions, earnings or dividends may be applied in payment of the ESOP Note
or any interest thereon if such application would cause the Corporation to incur
an excise tax under Section 4975 of the Code or would adversely affect (a) the
qualification of the ESOP or the Borrower under the Code or ERISA, or (b) the
status of the ESOP Loan as an "exempt loan" as such term is described in Section
4975(d)(3) of the Code.


SECTION III.  CONDITIONS PRECEDENT.

                  The obligation of the Corporation to make the ESOP Loan
hereunder is subject to the following conditions precedent:

         3.01 Documents Required for the Closing. The Borrower shall have
delivered to Corporation, prior to the disbursement of the ESOP Loan (the
"Closing"), the following:

         (A) The ESOP Note.

         (B) This Agreement.


                                       3
<PAGE>   5
         (C) Stock certificate no. __ evidencing the Collateral with duly
executed assignments in blank.

         (D) Executed copy of the Trust Agreement between Cole Taylor Bank and
the Corporation.

         (E) An opinion of Alex Sheshunoff & Co. in the form required under the
Purchase Agreement; and

         (F) Such other documents as may be reasonably required by the
Corporation.


SECTION IV.  REPRESENTATIONS AND WARRANTIES.

         4.01 Original. To induce the Corporation to enter into this Agreement,
the Borrower represents and warrants to the Corporation as follows:

         (A) Cole Taylor Bank, the trustee of the ESOT (the "Trustee") (i) is a
duly organized and validly existing trust company in good standing with full
authority to act as trustee of the ESOT and exercise trust powers under the laws
of the State of Illinois, and (ii) has full corporate power and authority to
execute and deliver the Trust Agreement between the Company and Borrower
establishing the ESOT and to carry out the transactions contemplated thereby.

         (B) This Agreement and the ESOP Note have been duly executed and
delivered by the Borrower and constitute the legal, valid and binding obligation
of the Borrower enforceable against the Borrower in accordance with their
respective terms, except as the enforceability thereof may be limited by the
effect of the applicable bankruptcy, insolvency, fraudulent conveyance,
reorganization, moratorium and similar laws affecting creditors' rights
generally and by general principles of equity and ERISA (regardless of whether
considered in a proceeding at law or in equity).

         (C) The execution, delivery and performance of this Agreement and the
ESOP Note will not violate (i) Trustee's Charter or Bylaws, each as amended or
restated to date, (ii) any provision of any agreement, instrument, order, award,
judgment or decree to which the Trustee or the ESOT is a party or by which they
or any of their businesses or properties are bound, or (iii) any statute, rule
or regulation of any federal, state or local government or governmental agency
applicable to the Trustee or the ESOT.

         (D) The Collateral is free and clear of any liens, encumbrances, and
security interests, except for those created by this Agreement.

         (E) All of the proceeds of the ESOP Loan shall be used by the Borrower
to acquire shares of Common Stock.


                                       4
<PAGE>   6
         4.02 Survival. All of the representations and warranties set forth in
Paragraph 4.01 shall survive until all Obligations are satisfied in full.

SECTION V. THE BORROWER'S COVENANTS.

                  The Borrower does hereby covenant and agree with the
Corporation that, so long as any of the Obligations remain unsatisfied, it shall
comply with the following covenants:

         (A) The Borrower shall use the proceeds of the ESOP Loan only for the
purposes of acquiring Common Stock of the Corporation.

         (B) The Borrower shall not knowingly take any action that would
jeopardize the Plan's continued qualification under Sections 401(a) and
4975(e)(7).

SECTION VI.  THE CORPORATION'S COVENANT.

                  Subject to the Corporation's right to amend or to terminate
the Plan, the Corporation agrees to make contributions to Borrower at such times
and in such amounts as are sufficient, when added to earnings and dividends
available for such purpose, to enable Borrower to meet all obligations under the
ESOP Loan; provided, however, that such contributions are deductible by the
Corporation under Section 404 of the Code, do not exceed applicable limitations
under Section 415 of the Code, and, provided further, that the Plan is, at the
time of any such contribution, qualified under Section 401(a) of the Code and
that the Corporation has taken all reasonable action to maintain the
tax-qualified status of the Plan.

SECTION VII.  DEFAULT.

         7.01 Event of Default. An event of default ("Event of Default") shall
occur hereunder if the Borrower shall fail to pay when due any installment of
principal or interest payable hereunder and such default shall continue for a
period of 10 days after written notice by the Corporation; provided, however,
that no Event of Default shall be deemed to have occurred if the Borrower's
failure to make timely payment is caused by the Corporation's breach of its
obligations under Section VI hereof.

         7.02 Rights and Remedies of Corporation. Under no circumstances shall
an Event of Default result in an acceleration of the due date for payment of any
portion of the principal of the ESOP Loan.

SECTION VIII.  PLEDGE OF COLLATERAL; REMEDIES.

         8.01 Pledge. As security for the performance of its Obligations
hereunder, Borrower hereby pledges the Collateral to the Corporation. Collateral
shall be released from this pledge in accordance with the formula prescribed in
Section 7.4 of the Plan for release of shares from the Loan Suspense Account (as
defined in the Plan). The Corporation, as pledgee, shall retain custody 


                                       5
<PAGE>   7
of all pledged Collateral prior to its release, and shall have full power to
assign its rights or prospective rights as Pledgee. So long as no Event of
Default has occurred and is continuing, Borrower shall be entitled to collect
and receive all dividends, income, installment sale proceeds, revenue, and
profits accruing with respect to the Collateral, and shall have full power with
respect to the voting, sale, and other exercise of discretionary shareholder
rights with respect to the Collateral.

         8.02 Remedies. If an Event of Default shall occur and be continuing,
the Corporation, as pledgee, shall be entitled to realize upon the Collateral up
to, but not in excess of, the amount then in default. The Corporation, as
pledgee, is entitled to sell at public or private sale or otherwise dispose of
any part of the Collateral, and after deducting from the proceeds of such sale
or other disposition all expenses (including reasonable attorneys' fees and
expenses), the Corporation may apply any such proceeds toward the satisfaction
of the Obligations. Any remainder of the proceeds after satisfaction of the
Event of Default shall be held as additional collateral for the remaining
obligations, or distributed as required by applicable Laws. Notice of any sale
or other disposition shall be given to the Borrower at least 10 business days
before the time of any such sale or disposition, which the Borrower hereby
agrees shall be reasonable notice of such sale or disposition. If an Event of
Default shall occur and be continuing, the Corporation (or its assignee) shall
be entitled to collect and receive all dividends, income, installment sale
proceeds, revenue, and profits accruing with respect to the Collateral, to be
held as additional collateral or, at the option of the Corporation, applied
against the Obligations as described herein, and the Corporation or its assignee
shall have full power with respect to the voting, sale, and other exercise of
discretionary shareholder rights with respect to that portion of the Collateral
the fair market value of which is not in excess of the amount of the default.

         8.03 Non-Recourse. Except as specifically provided in this Section VIII
with respect to the Collateral, the ESOP Loan shall be without recourse against
Borrower's assets. If an Event of Default shall occur and be continuing, the
Corporation shall have no rights to assets of the Borrower other than shares of
Common Stock acquired with the proceeds of the ESOP Loan which are Collateral
for the ESOP Loan, contributions (other than contributions of "employer
securities" as such term is described in Section 409(l) of the Code) that are
made by the Corporation to enable the Borrower to meet its obligations
hereunder, earnings attributable to the investment of such contributions and
such other assets, if any, as to which recourse may be permitted under Section
4975 of the Code and the regulations thereunder; provided, however, that (a) the
value of the Borrower's assets transferred in satisfaction of the ESOP Loan
shall not exceed the amount of the default (without regard to any acceleration
of the payment schedule that occurs as a result of the default); and (b) the
Borrower's assets shall be transferred to the Corporation only to the extent of
the failure of the Borrower to meet the payment schedule of the ESOP Loan.


                                       6
<PAGE>   8
SECTION IX.  MISCELLANEOUS.

         9.01 Further Assurance. From time to time, the Borrower will execute
and deliver to the Corporation such additional documents and will provide such
additional information as the Corporation may reasonably require to carry out
the terms of this Agreement and be informed of the Borrower's status and
affairs.

         9.02 Applicable Law. Except to the extent preempted by ERISA, the
substantive Laws of the State of Illinois shall govern the construction of this
Agreement and the rights and remedies of the parties hereto.

         9.03 Assignment and Amendment. This Agreement shall inure to the
benefit of, and shall be binding upon, the respective successors and permitted
assigns of the parties hereto. The Borrower has no right to assign any of its
rights or obligations hereunder without the prior written consent of the
Corporation. The Corporation may assign its rights and obligations hereunder to
any Person by giving notice thereof to the Borrower. This Agreement, and the
documents executed and delivered pursuant hereto and in connection herewith,
constitute the entire agreement between the parties and may be amended only by a
writing signed on behalf of each of the Corporation (or its assigns) and the
Borrower.

         9.04 Severability. If any provision of this Agreement shall be held
invalid under any applicable Laws, such invalidity shall not affect any other
provision of this Agreement that can be given effect without the invalid
provision, and, to this end, the provisions hereof are severable.

         9.05 Integration. This Agreement and the ESOP Note contain the entire
understanding of the parties hereto with respect to the subject matter contained
herein and therein. This Agreement supersedes all prior agreements and
understandings between the parties with respect to such subject matter.

         9.06 Notices. Any notice, request or other document to be given
hereunder to any party shall be effective upon receipt (or refusal of receipt)
and shall be in writing and delivered personally or sent by telecopy or
certified or registered mail, postage prepaid to the address indicated on the
signature page.

         9.07 Counterparts. This Agreement may be executed in any number of
counterparts, each of which shall be deemed to be an original, but all of which
together shall constitute but one and the same instrument.

         9.08 Waivers. No failure by the Corporation to exercise any rights or
remedies hereunder shall operate as a waiver thereof, nor shall any waiver of
any right or remedy hereunder preclude any future exercise of such right or
remedy.

         9.09 Construction. It is the intention of the Corporation and the
Borrower that the ESOP Loan shall qualify as an "exempt loan" as defined in 29
C.F.R. Section 2550.408(b)-3 and 26 C.F.R.


                                       7
<PAGE>   9
Section 54.4975-7(b)(iii), and that the Plan shall continue to be qualified
under Sections 401(a) and 4975(e)(7) of the Code. All terms of the ESOP Loan and
the ESOP Note shall be construed in a manner consistent with this intent. In the
event that it is necessary to amend the terms of the ESOP Loan or the ESOP Note
to carry out this intent, the Corporation and the Borrower agree to negotiate
such an amendment (which may have retroactive effect) promptly and in good
faith.

                            [Signature Page Follows]


                                       8
<PAGE>   10
         IN WITNESS WHEREOF, the parties hereto have duly executed this ESOP
Loan and Pledge Agreement as of the day and year first above written.


COMPANY:                              BUYER:

TAYLOR CAPITAL GROUP, INC.            THE TAYLOR CAPITAL GROUP, INC.
                                      PROFIT SHARING AND EMPLOYEE STOCK
                                      OWNERSHIP TRUST

By:      ________________________     By:  Cole Taylor Bank,
Title:   ________________________     Not in its corporate capacity but solely 
                                      as Trustee of The Taylor Capital Group, 
                                      Inc. Profit Sharing and Employee Stock
                                      Ownership Trust
Address:
350 East Dundee                       By:      ________________________
Wheeling, IL 60090                    Title:   ________________________

Attention:  Secretary                 Address:
                                      Cole Taylor Bank
with a copy to                        350 East Dundee Rd.
Helen H. Morrison.                    Wheeling, IL  60090
McDermott, Will & Emery               Attn: __________
Suite 4400
Chicago, IL 60606


                                       9
<PAGE>   11
                                   ESOP NOTE

November 25, 1998                                                $576,000.00

     FOR VALUE RECEIVED, the undersigned promises to pay to the order of TAYLOR 
CAPITAL GROUP, INC., a Delaware corporation (the "Corporation"), at its 
principal office located at 350 East Dundee, Wheeling, Illinois 60090, the 
principal sum of Five Hundred Seventy Six Thousand Dollars ($576,000.00), plus 
interest on the unpaid balance thereof as determined herein.

     This ESOP Note ("ESOP Note") is issued pursuant to a certain ESOP Loan and 
Pledge Agreement dated as of the date hereof by and between the undersigned and 
the Corporation (the "Agreement") reference to which Agreement is made for a 
statement of the additional rights and obligations of the parties hereto. For 
purposes hereof, this ESOP Note, the Agreement and all other documents or 
instruments issued in connection therewith, are collectively referred to as the 
"Loan Documents". All of the terms, covenants, provisions, conditions, 
stipulations, promises and agreements contained in the Loan Documents to be 
kept, observed and/or performed by the undersigned are made a part of this ESOP 
Note and are incorporated herein by this reference to the same extent and with 
the same force and effect as if they were fully set forth herein, and the 
undersigned promises and agrees to keep, observe and perform them, or cause 
them to be kept, observed and performed, strictly in accordance with the terms 
and provisions thereof. In the event of an express conflict between the terms 
hereof and the terms of the Agreement, the terms of the Agreement shall 
prevail. Capitalized terms used herein and not otherwise defined shall have the 
meanings assigned to them in the Agreement.

     Payments of principal and interest shall be made in such amounts and at 
such times as are specified in the Agreement.

     If at any time the rate of interest provided for herein shall be deemed by 
any competent court of law, governmental agency or tribunal to exceed the 
maximum rate of interest permitted by applicable laws, then, for such time as 
the rate would be deemed excessive, its application shall be suspended and 
there shall be charged instead the maximum rate of interest permissible.

     Payments shall be made in immediately available funds and shall be 
applied, unless otherwise agreed, first to interest, then to principal.

     The undersigned may prepay the principal hereof, in whole or in part, at 
any time. Any prepayments of principal shall be applied against installments of 
principal, if any, due hereunder in the inverse order of their maturity. 
Prepayment is permitted in accordance with the terms of the Agreement. This note
is not payable on demand.

                                       1


<PAGE>   12
     The occurrence of an Event of Default as defined in the Agreement shall 
constitute an Event of Default hereunder and the holder shall have such 
remedies as are specified in the Agreement.

     The obligations of the Borrower hereunder are without recourse to the 
Borrower, except to the extent of any Collateral.

     Any payment on this ESOP Note coming due on a Saturday, a Sunday, or a day 
which is a legal holiday in the place at which a payment is to be made 
hereunder shall be made on the next succeeding day which is a business day in 
such place, and any such extension of the time of payment shall be included in 
the computation of interest hereunder.

                                        
                         TAYLOR CAPITAL GROUP, INC.
                         PROFIT SHARING AND EMPLOYEE STOCK
                         OWNERSHIP TRUST


                         By: Cole Taylor Bank
                         Not in its individual or corporate capacity, but
                         solely in its capacity as Trustee of the Taylor Capital
                         Group, Inc. Profit Sharing and Employee Stock
                         Ownership Trust

                         By:
                            -------------------
                         Title:
                               ----------------                                 



                                       2