<PAGE>

                                                                    Exhibit 10.4

               AFFILIATE COLLATERAL PLEDGE AND SECURITY AGREEMENT
               --------------------------------------------------

     THIS AFFILIATE COLLATERAL PLEDGE AND SECURITY AGREEMENT ("Pledge
Agreement"), dated as of April 22, 1999, is made by and among Premier Lending
Corporation, a corporation organized and existing under the laws of the State of
Georgia ("Pledgor"), Premier Bank, a bank organized and existing under the laws
of the State of Georgia ("Borrower"), and the Federal Home Loan Bank of Atlanta
("Bank").

     WHEREAS, Borrower is a member and stockholder of the Bank;

     WHEREAS, Pledgor is an Affiliate of Borrower (for purposes of this Pledge
Agreement, Affiliate means any person or company which controls, is controlled
by, or is under common control with, Borrower, including any holding company,
any subsidiary, or any service corporation of the Borrower);

     WHEREAS, Borrower and the Bank have entered into an Agreement for Advances
and Security Agreement with Blanket Floating Lien or an Advances, Specific
Collateral Pledge and Security Agreement, dated as of March 5, 1997 (such
agreement, including any amendments thereto and any successor agreement that may
be entered into by Borrower and the Bank in substitution therefor, hereinafter
the "Borrower Advances Agreement"), pursuant to which the Bank may advance funds
from time to time to Borrower and Borrower may pledge certain collateral from
time to time to the Bank; and

     WHEREAS, at the request of Borrower, and in order to induce the Bank to
make additional advances to Borrower, Pledgor has agreed to pledge certain of
its property as collateral to and for the benefit of the Bank, to secure the
obligations of Borrower to the Bank,

     NOW, THEREFORE, for valuable consideration, the receipt and sufficiency of
which is hereby acknowledged, Pledgor, Borrower, and the Bank agree as follows:

     1.  Pledgor's Receipt of Borrower Advances Agreement; Definitions.  Pledgor
         -------------------------------------------------------------
hereby acknowledges and agrees that it has received a copy of the Borrower
Advances Agreement (including all amendments thereto) and that it is familiar
with the terms and conditions thereof.  Unless otherwise defined herein or the
context otherwise requires, all capitalized terms used herein shall have the
same meanings as in the Borrower Advances Agreement, except that "Borrower" as
used herein shall be synonymous with "Member" as used in the Borrower Advances
Agreement.

     2.  Creation of Security Interest.  As security for all Indebtedness now or
         -----------------------------
hereafter outstanding from the Borrower to the Bank under the Borrower Advances
Agreement, Pledgor hereby assigns, transfers and pledges to the Bank, and grants
to the Bank a security interest in, certain property which is (i) specifically
listed and identified in Attachment A hereto or any amendment thereto or any
substitute Attachment A that may be provided by the Pledgor with the agreement
of the Bank from time to time, or (ii) all of the proceeds of the foregoing
(collectively, the "Pledgor Collateral").
<PAGE>

     The Pledgor Collateral shall constitute Collateral for all purposes under
the Borrower Advances Agreement and, in addition to any rights or duties with
respect to the Pledgor Collateral created by this Pledge Agreement, the Pledgor
and the Bank shall have the same rights and duties with respect to the Pledgor
Collateral as do Borrower and the Bank, respectively, with respect to Collateral
under the Borrower Advances Agreement; provided, however, that if the Bank has
not requested or required delivery of Pledgor Collateral in accordance with
Paragraph 3 below, then (a) Borrower shall, at all times, hold the Pledgor
Collateral in trust for the benefit of, and subject to the direction and control
of, the Bank and physically safeguard the Pledgor Collateral with at least the
same degree of care as the Borrower uses in physically safeguarding its other
property; and, if so requested by the Bank in writing, and (b) Borrower shall
hold each set of First Mortgage Documents and all Other Mortgage Documents which
are a part of the Pledgor's Collateral in a separate file folder with each file
folder clearly labeled with the loan identification number and the name of the
borrower(s).  Each such file folder shall be clearly marked or stamped with the
statement: "The Mortgage/Deed of Trust and Note Relating to this Loan Have Been
Assigned to the Federal Home Loan Bank of Atlanta" or such other statement that
may be approved by the Bank from time to time.  If so requested by the Bank, in
writing, Borrower shall physically segregate First Mortgage Documents and Other
Mortgage Documents which are a part of such Pledgor Collateral from all other
property of the Borrower or the Pledgor in a manner satisfactory to the Bank.

     3.  Delivery.  Upon the Bank's written or oral request, or promptly at any
         --------
time that the Borrower becomes subject to any mandatory collateral delivery
requirements that may be established in writing by the Bank, and in either case
from time to time thereafter, the Pledgor shall deliver (or, in the case of
uncertificated securities, otherwise transfer) to the Bank, or to a custodian
designated by the Bank, Pledgor Collateral in an amount determined by the Bank.
Pledgor Collateral delivered to the Bank or to a custodian designated by the
Bank shall be endorsed or assigned in recordable form by the Pledgor as directed
by the Bank.

     4.  Right of Bank to Proceed Against Pledgor Collateral; Waivers; Borrower
         ----------------------------------------------------------------------
Acknowledgment.
--------------
         (a)  Pledgor agrees that, following an Event of Default, as defined in
     the Borrower Advances Agreement and as modified by this Pledge Agreement,
     the Bank may proceed against the Pledgor Collateral in accordance with the
     terms of the Borrower Advances Agreement as though Pledgor were the Member
     thereunder. Pledgor hereby waives and agrees not to assert: (i) any and all
     right to presentment, protest, demand for payment, notice of default,
     dishonor or nonpayment and all other notices to or upon Borrower or
     Pledgor, including, without limitation, notice as to the making of any
     advance or other extension of credit to Borrower or the exercise of any
     right by the Bank hereunder or under the Borrower Advances Agreement; and
     (ii) any and all right to require the Bank to proceed against Borrower or
     any Collateral pledged by Borrower before enforcing the Bank's rights
     against the Pledgor Collateral, and any other defense based upon an
     election of remedies.

                                       2
<PAGE>

         (b)  By execution hereof, Borrower acknowledges its consent to the
     terms and conditions hereof and Borrower hereby waives and agrees not to
     assert any and all right to require the Bank to proceed against Pledgor or
     Pledgor Collateral before enforcing the Bank's rights against the
     Collateral and any other defense based upon an election of remedies.

     5.  Representations and Agreements by Pledgor.  Pledgor hereby represents,
         -----------------------------------------
warrants to, and agrees with the Bank that:

         (a)  Each item of Pledgor Collateral satisfies all the criteria for
     Qualifying Collateral set forth in the Borrower Advances Agreement, except
     that the Pledgor Collateral is owned by Pledgor, rather than by Borrower,
     free and clear of any liens, encumbrances or other interests other than the
     assignment to the Bank hereunder;

         (b)  Pledgor has full power, right and authority to grant the security
     interest in the Pledgor Collateral created hereby and has taken all
     corporate action necessary to authorize the execution and delivery of this
     Pledge Agreement;

         (c)  The security interest in the Pledgor Collateral created hereby has
     been duly and validly granted by Pledgor and such security interest, and
     this Pledge Agreement, are enforceable in accordance with the terms hereof;

         (d)  This Pledge Agreement has been authorized or ratified and approved
     by Pledgor's Board of Directors and will be maintained continuously among
     Pledgor's official records;

         (e)  A certified copy of the Board of Director's resolution evidencing
     its approval hereof is attached hereto as Attachment "B," the form of which
     has been previously approved by the Bank or its counsel;

         (f)  An opinion of Pledgor's counsel that Pledgor has the power, right
     and authority to grant the security interest in the Pledgor Collateral
     created hereby, that Pledgor has taken all corporate action necessary to
     authorize the execution and delivery of this Pledge Agreement, that there
     is no impediment to the Bank enforcing its interests against the Pledgor
     Collateral under this Pledge Agreement has been provided to and accepted by
     the bank, a copy of which is attached hereto as attachment "C";

         (g)  All information contained in any report, schedule or other
     documentation provided from time to time by Pledgor to the Bank will be
     true and correct as of the time given; and

         (h)  Pledgor agrees to make, execute, record and deliver to the Bank
     such financing statements, notices, assignments, listings, powers and other
     documents with respect to the Pledgor Collateral and the Bank's security
     interest therein in such form as the Bank may require.

                                       3
<PAGE>

     6.  Representation and Warranties by Borrower.  Borrower hereby represents,
         -----------------------------------------
warrants and agrees to and with the Bank that:

         (a)  Borrower has full power, right and authority to enter into this
     Pledge Agreement and has taken all corporate action necessary to authorize
     the execution and delivery of this Pledge Agreement;

         (b)  This Pledge Agreement is enforceable against Borrower in
     accordance with the terms hereof;

         (c)  This Pledge Agreement has been authorized or ratified and approved
     by Borrower's Board of Directors and will be maintained continuously among
     Borrower's official records;

         (d)  A certified copy of the Board of Director's resolution evidencing
     its approval hereof is attached hereto as Attachment "D," the form of which
     has been previously approved by the Bank or its counsel;

         (e)  Borrower agrees to make, execute, record, and deliver to the Bank
     such financing statements, notices, assignments, listings, powers and other
     documents with respect to the Pledgor Collateral and the Bank's security
     interest therein in such form as the Bank may require; and

         (f)  Borrower agrees that a failure by either Borrower or Pledgor to
     perform any of the rights, responsibilities, duties, representations,
     warranties, and agreements under this Pledge Agreement shall constitute an
     Event of Default under the Borrower Advances Agreement.

     7.  Governing Law.  In addition to the terms and conditions specifically
         -------------
set forth herein, this Pledge Agreement shall be governed by the statutory and
common law of the United States and, to the extent state law is applicable, by
the laws of the State of Georgia (without giving effect to choice of law
principles included therein). Notwithstanding the foregoing, the Uniform
Commercial Code as in effect in the State of Georgia shall be applicable to this
Pledge Agreement, to the security interest created hereby, and to the pledge of
Pledgor Collateral hereunder.

     8.  Partial Exercise; Amendment; Severability.  No delay on the part of the
         -----------------------------------------
Bank in exercising any right, power or privilege shall operate as a waiver
thereof, nor shall any single or partial exercise of any such right, power or
privilege preclude other or further exercise thereof or the exercise of any
other right, power or privilege or be construed to be a waiver of any Event of
Default under the Borrower Advances Agreement. No waiver by the Bank of any such
Event of Default shall be effective unless in writing and signed by an
authorized officer of the Bank, and no such waiver shall be deemed to be a
waiver of a subsequent Event of Default under the Borrower Advances Agreement or
be deemed to be a continuing waiver. No course of dealing between Borrower or
Pledgor, respectively, and the Bank or its agents or employees shall be
effective to change, modify or discharge any provision of this Pledge Agreement,
or the Borrower Advances Agreement or to constitute a waiver of any Event of
Default

                                       4
<PAGE>

thereunder. If any provision of this Pledge Agreement is held invalid or
unenforceable to any extent or in any application, the remainder of this
agreement, or application of such provision to different persons or
circumstances or in different jurisdictions, shall not be affected thereby.

     IN WITNESS WHEREOF, each of Pledgor, Borrower, and the Bank has
respectively caused this Pledge Agreement to be signed in its name by its duly
authorized representative as of the date first above mentioned.

<TABLE>
<CAPTION>
FEDERAL HOME LOAN BANK OF ATLANTA                 Premier Lending Corporation
                                                  ---------------------------
                                                  (Pledgor)
<S>                                               <C>

By:                                               By: /s/ George S. Phelps
   --------------------------------                  ---------------------------
   Title:                                            Title:  President
         --------------------------
By:
   --------------------------------
     Title:
           ------------------------

ACCEPTED, ACKNOWLEDGED, AND APPROVED:

Premier Bank
------------
(Borrower)


By:  /s/ Michael E. Ricketson
   -----------------------------------
     Title:  Chief Financial Officer
           ---------------------------

</TABLE>

                                       5