<PAGE>   1
                                                                 EXHIBIT 10.4

                             SECOND AMENDMENT TO
                           REVOLVING LOAN AGREEMENT

     THIS SECOND AMENDMENT TO REVOLVING LOAN AND SECURITY AGREEMENT dated as of
November 1, 1997 (this "Amendment"), is between TAYLOR CAPITAL GROUP, INC., an
Delaware corporation (the "Borrower"), and LASALLE NATIONAL BANK, a national 
banking association (the "Bank").

                                 WITNESSETH:


     WHEREAS, the Borrower and the Bank entered into a Loan Agreement dated as
of February 12, 1997, as amended by a first Amendment dated February 27, 1997
(as so amended, the "Agreement"); and

     WHEREAS, the Borrower and the Bank have agreed to amend the Agreement as
more fully described herein,

     NOW, THEREFORE, in consideration of the premises and other good and
valuable consideration, the receipt and adequacy of which are hereby
acknowledged, the parties hereto agree as follows:

     1. DEFINITIONS. All capitalized terms used herein without definition shall
have the respective meanings set forth in the Agreement.

     2. AMENDMENTS TO THE AGREEMENT.

        2.1 Amendments to Section 1.1. The following definitions set forth in
Section 1.1 of the Agreement are hereby amended and restated in their
entireties as follows:

           "Revolving Credit Maturity Date" means May 1, 1998.

           "Revolving Note" means a promissory note in the form of Exhibit 3.1
     attached hereto, as amended or replaced from time to time, duly executed 
     by the Borrower.

        2.2 Amendment to Section 3.1. The first sentence of Section 3.1 of the
Agreement is hereby amended and restated in its entirety as follows:

           "The Revolving Loans shall be evidenced by the Revolving Note."

        2.3 Replacement of Exhibit 3.1. Exhibit 3.1 attached to and made a part
of the Agreement is hereby deleted in its entirety and Exhibit 3.1 attached
hereto is hereby substituted therefor.

     3. WARRANTIES. To induce the Bank to enter into this Amendment, the
Borrower warrants that:




<PAGE>   2

        3.1 Authorization. The Borrower is duly authorized to execute and
deliver this Amendment and is and will continue to be duly authorized to
borrow monies under the Agreement, as amended hereby, and to perform its
obligations under the Agreement, as amended hereby.

        3.2 No Conflicts. The execution and delivery of this Amendment and the
performance by the Borrower of its obligations under the Agreement, as amended
hereby, do not and will not conflict with any provision of law or of the
charter or by-laws of the Borrower or of any agreement binding upon the
Borrower.

        3.3 Validitv and Binding Effect. The Agreement, as amended hereby, is a
legal, valid and binding obligation of the Borrower, enforceable against the
Borrower in accordance with its terms, except as enforceability may be limited
by bankruptcy, insolvency or other similar laws of general application
affecting the enforcement of creditors' rights or by general principles of
equity limiting the availability of equitable remedies.

        3.4 No Default. As of the date hereof, no Event of Default under
Section 8 of the Agreement, as amended by this Amendment, or event or condition
which, with the giving of notice or the passage of time, shall constitute an
Event of Default, has occurred or is continuing.

        3.5 Warranties. As of the date hereof, the representations and
warranties in Section 7 of the Agreement are true and correct as though made on
such date, except for such changes as are specifically permitted under the
Agreement.

     4. CONDITIONS PRECEDENT. This Amendment shall become effective as of the
date above first written after receipt by the Bank of the following documents:

        (a)  This Amendment duly executed by the Borrower;

        (b)  Substitute Revolving Note in the form of Exhibit 3.1 attached 
             hereto duly executed by the Borrower;

        (c)  Such other documents and instruments as the Bank reasonably 
             requests.

     5. GENERAL.

        5.1 Law. This Amendment shall be construed in accordance with and
governed by the laws of the State of Illinois.

        5.2 Successors. This Amendment shall be binding upon the Borrower and
the Bank and their respective successors and assigns, and shall inure to the
benefit of the Borrower and the Bank and their respective successors and
assigns.



                                      2

<PAGE>   3


        5.3 Confirmation of the Agreement. Except as amended hereby, the
Agreement shall remain in full force and effect and is hereby ratified and
confirmed in all respects.


LASALLE NATIONAL BANK                      TAYLOR CAPITAL GROUP, INC.



By: /s/                           
   -------------------------------         By: /s/                           
Its: Vice President                           -------------------------------
    ------------------------------         Its:  CEO
                                               ------------------------------





















                                      3