<PAGE>   1
                                                                     EXHIBIT 4.3


===============================================================================


                   SECOND AMENDED AND RESTATED REVOLVING/TERM

                    SECURITY AND COLLATERAL AGENCY AGREEMENT

                           Dated as of July 31, 1996



                                    Between

                    RESOURCE BANCSHARES MORTGAGE GROUP, INC.

                                      and

                            THE BANK OF NEW YORK,

                     as Collateral Agent and Secured Party


===============================================================================
<PAGE>   2
                               TABLE OF CONTENTS


<TABLE>
<CAPTION>
                                                                            Page
                                                                            ----
<S>          <C>                                                             <C>
Section 1.   Security Interest  ..........................................    1
Section 2.   Appointment of Custodian ....................................    3
Section 3.   Release of Collateral  ......................................    3
Section 4.   Information  ................................................    4
Section 5.   Standard of Care of Collateral Agent; Duties;                  
             Indemnification .............................................    4
Section 6.   Fees and Expenses of Collateral Agent  ......................    5
Section 7.   Resignation of Collateral Agent  ............................    6
Section 8.   Availability of Documents  ..................................    6
Section 9.   Representations and Warranties of Pledgor  ..................    6
Section 10.  Covenants of Pledgor ........................................    7
Section 11.  Collection of Collateral Payments  ..........................    8
Section 12.  Authorized Action by Agent and Collateral Agent  ............    9
Section 13.  Default  ....................................................   10
               (a)  Remedies .............................................   10
               (b)  Successor Servicer ...................................   11
Section 14.  Cumulative Rights  ..........................................   12
Section 15.  Waiver ......................................................   13
Section 16.  Binding Upon Successors  ....................................   13
Section 17.  Entire Agreement; Severability ..............................   13
Section 18.  CHOICE OF LAW  ..............................................   13
Section 19.  Amendments, Etc..............................................   13
Section 20.  Notice ......................................................   13
Section 21.  Execution in Counterparts  ..................................   13
Section 22.  Interpretation ..............................................   14
               (a)  Definitions ..........................................   14
               (b)  Other Interpretative Provisions ......................   17
               (c)  Captions .............................................   18 
</TABLE>
<PAGE>   3
                   SECOND AMENDED AND RESTATED REVOLVING/TERM

                    SECURITY AND COLLATERAL AGENCY AGREEMENT

                           Dated as of July 31, 1996


                 WHEREAS, Resource Bancshares Mortgage Group, Inc., a Delaware
corporation (the "Pledgor"), and The Bank of New York, as Collateral Agent and
Secured Party, are parties to an Amended and Restated Revolving/Term Security
and Collateral Agency Agreement, dated as of September 29, 1995 (the "Existing
Security Agreement");

                 WHEREAS, the Pledgor and The Bank of New York are parties to a
Security Agreement dated as of July 14, 1995 (the "Existing Receivables
Security Agreement") and

                 WHEREAS, the Pledgor and the Collateral Agent and Secured
Party wish to amend, restate and combine the Existing Security Agreement and
the Existing Receivables Security Agreement;

                 NOW, THEREFORE, in consideration of the premises and the
execution and delivery of the Second Amended and Restated Secured
Revolving/Term Credit Agreement, dated as of July 31, 1996, by the Banks listed
on the signature pages thereof, Bank One, Texas, N.A., First Bank, N.A.,
NationsBank of Texas, N.A. and Texas Commerce Bank, as Co-Agents, and The Bank
of New York, as Agent and Collateral Agent, the Pledgor hereby agrees with The
Bank of New York, as Collateral Agent and Secured Party, that the Existing
Security Agreement and the Existing Receivables Security Agreement are
combined, amended and restated in their entirety as follows (with certain terms
used herein being defined in Section 22):.

                 Section 1.  Security Interest.  (a)  To secure the payment,
observance and performance of the Secured Obligations, the Pledgor hereby
mortgages, pledges and assigns the Collateral to the Secured Party, and grants
to the Secured Party a continuing security interest in, and continuing lien
upon, the Collateral.
<PAGE>   4

                 (b)  The Pledgor agrees that the Security Interest shall, and
that the Pledgor shall take all action necessary or desirable, or that the
Secured Party may request, including the actions specified on Schedule 1, so
that the Security Interest shall at all times be valid and perfected and
enforceable against the Pledgor and all third parties, in accordance with the
terms hereof, as security for the Secured Obligations and that the Collateral
shall not at any time be subject to any Lien that is prior to, on a parity with
or junior to the Security Interest.  Notwithstanding anything contained in this
Security Agreement to the contrary, the Security Interest created by this
Security Agreement is subject and subordinate to all rights, powers and
prerogatives of FNMA, FHLMC or GNMA under and in connection with the FNMA
Contract, FHLMC Contract or GNMA Contract, as defined in any Acknowledgment
Agreement by and between FNMA, FHLMC or GNMA, as the case may be, and the
Pledgor and to (i) all rights, powers and prerogatives of such other investors
that own Mortgage Loans, or which guaranty payments on securities based on and
backed by pools of Mortgage Loans (the "Investors"); and (ii) to all claims of
an Investor arising out of any and all defaults and outstanding obligations of
the Pledgor to the Investor.  Such rights, powers and prerogatives of the
Investors may include, without limitation, one or more of the following: the
right of an Investor to disqualify the Pledgor from participating in a mortgage
selling or servicing program or a securities guaranty program with the
Investor; the right to terminate contract rights of the Pledgor relating to
such a mortgage selling or servicing program or securities guaranty program;
and the right to transfer and sell all or any portion of such contract rights
following the termination of those rights.  With respect to GNMA, the Pledgor
is entitled to servicing income only so long as it is an issuer in good
standing; that upon the Pledgor's loss of issuer status, the Secured Party's
rights to any servicing income of the Pledgor's will terminate, and that the
pledge of rights of servicing income conveys no rights that are not otherwise
specifically provided for in the Guidelines.

                 (c)  As required by GNMA, the following language is hereby
included:

         "Notwithstanding anything to the contrary contained herein:





                                      -2-
<PAGE>   5

                          (1)  A portion of the property subject to the
         security interest reflected in this instrument is an interest of the
         Debtor in certain mortgages ("Pool Mortgages") pooled under the
         mortgage-backed securities program of the Government National
         Mortgage Association ("Ginnie Mae"), pursuant to section 306(g) of the
         National Housing Act, 12 U.S.C. Section 1721(g);

                          (2)  To the extent that the security interest
reflected in this instrument relates in any way to the Pooled Mortgages, such
security interest is subject and subordinate to all rights, powers and
prerogatives of Ginnie Mae under and in connection with: (i) 12 U.S.C. Section
1721(g) and any implementing regulations; (ii) the terms and conditions of that
certain Acknowledgment Agreement, with respect to the Security Interest, by and
between Ginnie Mae, Resource Bancshares Mortgage Group, Inc. (the "Debtor"),
and The Bank of New York; (iii) the Guaranty Agreements and contractual
agreements between Ginnie Mae and the Debtor; and (iv) the Ginnie Mae
Mortgage-Backed Securities Guide, 5500.1 Rev. 6 and the Ginnie Mae
Mortgage-Backed Securities Guide, 5500.2, and other applicable Guides (the
"Guides"), as each of the Guides is amended from time to time; and

                          (3)  Such rights, powers and prerogatives of Ginnie
         Mae include, but are not limited to, Ginnie Mae's right, by issuing a
         letter of extinguishment to Debtor, to effect and complete the
         extinguishment of all redemption, equitable, legal or other right,
         title or interest of the Debtor in the Pooled Mortgages, in which
         event the security interest as it relates in any way to the Pooled
         Mortgages shall instantly and automatically be extinguished as well."

                 Section 2.  Appointment of Custodian.  The Collateral Agent
hereby agrees to act as agent, bailee and custodian for the exclusive benefit
of the Principals with respect to the Collateral.  The Collateral Agent hereby
further agrees to maintain and hold all Collateral at any time delivered to it
as agent, bailee and custodian for the exclusive benefit of the Principals.
The Collateral Agent acknowledges and agrees that it is acting and will act
with respect to the Collateral for the exclusive benefit of the Principals and
is not, and shall not at any time in the future be, subject, with respect to
the Collateral, in any manner or to any extent, to the direction or





                                      -3-
<PAGE>   6

control of the Pledgor, except as expressly permitted hereunder or under the
Credit Agreement.  The Collateral Agent agrees to act in accordance with this
Agreement and in accordance with any written instructions from any Agent
delivered pursuant to the Credit Agreement.  Under no circumstances shall the
Collateral Agent deliver possession of Collateral to the Pledgor or any other
Person, or otherwise release any Collateral from the Lien created hereby,
except in accordance with the express terms of this Agreement or otherwise upon
the written instruction of all of the Banks.

                 Section 3.  Release of Collateral.  The Collateral Agent
shall, no later than five Business Days following receipt of a written request
therefor from the Pledgor, release from the Security Interest specifically
identified Borrowing Base Servicing Rights for purposes of the sale of such
rights.  Any such release of the Security Interest shall be evidenced by the
execution and delivery by the Collateral Agent, at no cost or expense to the
Collateral Agent or any Principal, of an appropriate form of UCC financing
statement release and such other documents as such Pledgor may reasonably
request (the "Release Documents"); provided, that:

                 (a)  At the time of the delivery of the Release Documents (and
both before and after giving effect to the requested release) no Default or
Borrowing Base Deficiency shall exist and the Agent shall have received from
the Pledgor a certificate to that effect;

                 (b)  The Pledgor shall have delivered to the Collateral Agent
and the Agent (with, so long as the Collateral Agent and the Agent are the same
Person, delivery to the Collateral Agent constituting delivery to the Agent)
concurrently with the written release request evidence satisfactory to said
Persons demonstrating in detail satisfactory to such Persons that at and as of
the date of the requested release and after giving effect thereto, no Borrowing
Base Deficiency would exist; and

                 (c)  The Pledgor shall have delivered to the Collateral Agent
and the Agent (with, so long as the Collateral Agent and the Agent are the same
Person, delivery to the Collateral Agent constituting delivery to the Agent) no
later than three Business Days prior to the requested date for the delivery of
the Release Documents a certified copy of the purchase and sale agreement, as





                                      -4-
<PAGE>   7

amended, and other related agreements pursuant to which the sale of pledged
Servicing Rights will be consummated.

                 Section 4.  Information.  The Collateral Agent shall deliver
promptly to the Agent such reports and information relating to the Collateral
and the performance of the Secured Obligations hereunder as the Agent or any
Bank may from time to time reasonably request.  The Collateral Agent shall
receive reasonable additional compensation from the Pledgor for services
rendered beyond those specifically enumerated in this Agreement.

                 Section 5.  Standard of Care of Collateral Agent; Duties;
Indemnification.  The Collateral Agent is a bailee for hire and shall hold the
Collateral in accordance with customary standards for those engaged as
custodians of commercial documents in similar capacities.  Notwithstanding
anything to the contrary contained herein:

                 (a)  The provisions of this Agreement and the attachments
hereto set forth the exclusive duties of the Collateral Agent and no implied
duties or obligations shall be read into this Agreement against the Collateral
Agent.  The Collateral Agent shall not be bound in any way by any agreement or
contract other than this Agreement and the attachments hereto and any other
agreement to which it is a party.  The Collateral Agent shall not be required
to ascertain or inquire as to the performance or observance of any of the
conditions or agreements to be performed or observed by any other party, except
as specifically provided in this Agreement and the attachments hereto.  The
Collateral Agent disclaims any responsibility for the validity or accuracy of
the recitals to this Agreement and any representations and warranties contained
herein, unless specifically identified as recitals, representations or
warranties of the Collateral Agent.

                 (b)  Throughout the term of this Agreement, the Collateral
Agent shall have no responsibility for ascertaining the value, collectability,
insurability, enforceability, effectiveness or suitability of any Collateral,
the title of any party therein, the validity or adequacy of the security
afforded thereby, or the validity of this Agreement (except as to the
Collateral Agent's authority to enter into this Agreement and to perform its
obligations hereunder).


                                      -5-
<PAGE>   8
                 (c)  The Collateral Agent shall not be under any duty to
examine or pass upon the genuineness, validity or legal sufficiency of any of
the documents constituting part of any Collateral and shall be entitled to
assume that all documents constituting part of such files are genuine and valid
and that they are what they purport to be, and that any endorsements or
assignments thereof are genuine and valid.  The Collateral Agent may rely upon
and shall be protected in acting in good faith upon any notice, resolution,
request, consent, order, certificate, report, statement or other paper or
document appearing on its face to be genuine and to have been signed or
presented by the proper party or parties or by a person or persons authorized
to act on behalf of the proper party or parties.  The Collateral Agent shall
not be liable for any action or omission to act as bailee except for its own
gross negligence or willful misconduct.

                 (d)  No provision of this Agreement shall require the
Collateral Agent to expend or risk its own funds or otherwise incur any
financial liability in the performance of any of its duties hereunder or in the
exercise of any of its rights or powers, if, in its sole judgment, it shall
believe that repayment of such funds or adequate indemnity against such risk or
liability is not assured to it.

                 (e)  The Collateral Agent is not responsible for preparing or
filing any reports or returns relating to federal, state or local income taxes
with respect to this Agreement, other than for the Collateral Agent's
compensation or for reimbursement of expenses.

                 (f)  Neither the Collateral Agent nor any other Principal
shall have any obligation to protect or preserve any Collateral or to preserve
rights pertaining thereto other than the obligation to use reasonable care in
the custody and preservation of any Collateral in its actual possession.  The
Collateral Agent shall be deemed to have exercised reasonable care in the
custody and preservation of any Collateral in its possession if such Collateral
is accorded treatment substantially equal to that which the Collateral Agent
accords its own property.  The Collateral Agent shall be relieved of all
responsibility for any Collateral in its possession upon surrendering it, or
tendering surrender of it, to the Pledgor.



                                      -6-
<PAGE>   9
                 Section 6.  Fees and Expenses of Collateral Agent.  The
Collateral Agent shall notify the Pledgor of all fees, expenses and charges of
the Collateral Agent arising out of the Collateral Agent's execution and
performance of its duties and obligations under this Agreement, and such
reasonable fees, expenses and charges shall be paid promptly by the Pledgor or,
if already paid by the Collateral Agent, the Pledgor promptly shall reimburse
the Collateral Agent therefor.  The Collateral Agent may employ, at the
Pledgor's expense, such legal counsel and other experts as it reasonably deems
necessary in connection with performing its duties and obligations under this
Agreement.

                 Section 7.  Resignation of Collateral Agent.  The Collateral
Agent may, at any time, effective upon 90 days' prior written notice to the
Pledgor and the Agent and the appointment of a successor Collateral Agent,
terminate its agreement to act as the Collateral Agent hereunder.  Upon the
effective date of any such termination, the Collateral Agent shall promptly
deliver the Collateral then held by it or its agents to the successor
Collateral Agent, the Agent or to such person or entity as the Agent may direct
in writing and shall execute and deliver such notices, instructions and
assignments as may be reasonably necessary or desirable to transfer the rights
of the Collateral Agent with respect to the Collateral to the successor
Collateral Agent.  Any successor Collateral Agent shall be designated by the
Required Banks and shall be reasonably acceptable to the Pledgor.

                 Section 8.  Availability of Documents.  Each of the Banks and
its agents, accountants, attorneys and auditors will be permitted during normal
business hours at any time and from time to time upon reasonable notice to the
Collateral Agent to examine (to the extent permitted by Applicable Law) the
files, documents, records and other papers in the possession or under the
control of the Collateral Agent relating to any or all of the Collateral and to
make copies thereof.  As long as no Default shall have occurred and be
continuing, all costs and expenses associated with the exercise from time to
time by any Bank of its rights under this Section 8 shall be for the account of
such Bank; if a Default shall have occurred and be continuing, all costs and
expenses associated with the exercise from time to time by any Bank of its
rights under this Section 8 shall be promptly paid by the Pledgor upon demand.



                                      -7-
<PAGE>   10
                 Section 9.   Representations and Warranties of Pledgor.  The
Pledgor hereby represents and arrants that: (a) the Pledgor is the sole owner
of the Collateral (or, in the case of after-acquired Collateral, at the time
the Pledgor acquires rights in such Collateral, will be the sole owner
thereof); (b) except for the Security Interest and as provided in Section 1(b),
no Person has (or, in the case of after-acquired Collateral, at the time the
Pledgor acquires rights therein, will have) any right, title, claim or interest
(by way of security interest or other lien or charge or otherwise) in, against
or to the Collateral; (c) (i) all Information heretofore, herein or hereafter
supplied to the Collateral Agent, the Agent, the Co-Agents or the Banks by or
on behalf of the Pledgor with respect to the Collateral is or will be accurate
and complete in all material respects and (ii) all information contained on the
Questionnaire is accurate and complete; and (d) no consent of any other Person
is required for the grant of the Security Interest in any of the Collateral
(including any computer software being utilized by the Pledgor pursuant to
license, lease or otherwise), other than in the case of the Borrowing Base
Servicing Rights, Acknowledgement Agreements from FNMA, FHLMC and GNMA, nor
will any consent need to be obtained upon the occurrence of an Event of Default
for the Secured Party to exercise its rights with respect to any of the
Collateral.

                 Section 10.  Covenants of Pledgor.  The Pledgor hereby agrees:
(a) to procure, execute and deliver from time to time any and all endorsements,
assignments, financing statements, notices and other writings deemed necessary
or appropriate by the Agent to perfect, maintain and protect the Security
Interest and the priority thereof and to deliver promptly to the Collateral
Agent all originals of Collateral or proceeds consisting of chattel paper or
instruments; (b) not to surrender or lose possession of (other than to the
Collateral Agent), sell, encumber, or otherwise dispose of or transfer, any
Collateral or right or interest therein other than the sale of Borrowing Base
Servicing Rights as contemplated in Section 3 above; (c) at all times upon the
request of the Required Banks during a Default, to account fully for and
promptly to deliver to the Collateral Agent, in the form received, all
Collateral or Proceeds received, endorsed to the Collateral Agent as
appropriate and accompanied by such assignments and powers, duly executed, as
the Collateral Agent shall request, and until so delivered all Collateral and
proceeds shall be held in trust for the Collateral Agent,



                                      -8-
<PAGE>   11

separate from all other property of the Pledgor and identified as being subject
to the interest of the Collateral Agent, provided that unless an Event of
Default shall have occurred and be continuing, the Pledgor shall have no
obligation to comply with the provisions of this sub-clause (c) with respect to
the Collateral which is such by virtue of clause (c) of the definition of
Collateral or with respect to the proceeds of the sale of Borrowing Base
Servicing Rights so long as Section 1.05(c) of the Credit Agreement is complied
with in connection with the sale or other disposition of the Borrowing Base
Servicing Rights giving rise to such Proceeds; (d) at any reasonable time, upon
demand by the Collateral Agent or any Bank, to exhibit and to allow inspection
by the Collateral Agent or such Bank (or Persons designated by the Collateral
Agent or such Bank) of the Collateral and the records concerning the Collateral
(at no cost to the Pledgor unless there shall have occurred and be continuing a
Default); (e) not to move its chief executive offices or any Collateral
constituting goods to a new location unless (i) the Agent shall have approved
such move in writing or (ii) (A) the Pledgor shall have given the Collateral
Agent not less than 45 days prior notice thereof, (B) the new location shall be
within one of the 50 States of the United States or the District of Columbia
and (C) the Collateral Agent shall have received such evidence reasonably
satisfactory to it as it may reasonably request (including acknowledgment
copies of financing statements and opinions of counsel) that the Secured
Party's rights with respect to the Collateral will not be adversely affected by
such move; (f) to keep the Collateral insured against loss, damage, theft and
other risks customarily covered by insurance, and such other risks as the Agent
may reasonably request; (g) to do all acts to maintain, preserve and protect
the Collateral that a prudent owner of assets of the same type as the
Collateral would deem necessary or desirable therefor (including performing its
obligations under Servicing Contracts); (h) not knowingly to use or permit any
Collateral to be used unlawfully or in violation of any provision of this
Agreement or any Applicable Law or any policy of insurance covering the
Collateral; (i) to pay (or require to be paid) prior to their becoming
delinquent all taxes, assessments, insurance premiums or other charges, now or
hereafter imposed upon or affecting any Collateral other than taxes and
assessments that are being contested in good faith by appropriate proceedings,
as to which adequate reserves in conformance with Generally Accepted Accounting
Principles have been established and which do not





                                      -9-
<PAGE>   12

constitute Liens upon the Collateral; (j) to notify the Collateral Agent before
any such change shall occur of any change in the Pledgor's name, identity or
structure through merger, consolidation or otherwise; (k) to appear in and
defend, at the Pledgor's cost and expense, any action or proceeding which may
affect its title to or the Secured Party's interest in the Collateral; (l) to
keep accurate and complete records of the Collateral and to provide the
Collateral Agent with such records and such reports and information relating to
the Collateral as the Collateral Agent may reasonably request from time to
time; (m) to comply with all laws, regulations and ordinances relating to the
possession, maintenance and control of the Collateral; and (n) upon the request
of the Agent, to use its best efforts as promptly as possible to obtain a GNMA
Acknowledgement Agreement among the Pledgor, the Collateral Agent and GNMA.

                 Section 11.  Collection of Collateral Payments.  (a)
The Pledgor shall, at its sole cost and expense, endeavor to obtain payment,
when due and payable, of all sums due or to become due with respect to any
Collateral ("Collateral Payments" or a "Collateral Payment"), including the
taking of such action with respect thereto as the Collateral Agent may
reasonably request or, in the absence of such request, as the Pledgor may
reasonably deem advisable; provided that the Pledgor shall not, without the
prior written consent of the Collateral Agent, grant or agree to any rebate,
refund, compromise or extension with respect to any Collateral Payment.  Upon
the request of the Required Banks following the occurrence of an Event of
Default, the Pledgor will notify and direct any party which is or might become
obligated to make any Collateral Payment, to make payment thereof to the
Collateral Agent (or to the Pledgor in care of the Collateral Agent) at such
address as the Collateral Agent may designate.  The Pledgor will reimburse the
Collateral Agent promptly upon demand for all out-of-pocket costs and expenses,
including reasonable attorneys' fees and litigation expenses, incurred by the
Collateral Agent in seeking to collect any Collateral Payment.

                 (b)  If a Default shall have occurred and be continuing, upon
the request of the Required Banks, the Pledgor will, forthwith upon receipt,
transmit and deliver to the Collateral Agent, in the exact form received, all
cash, checks, drafts and other instruments for the payment of money (properly
endorsed where required so that such items may be collected by the Collateral
Agent) which may be received by the Pledgor at any time as payment on account
of any Collateral Payment and if such request shall be made, until delivery to
the Collateral Agent,





                                      -10-
<PAGE>   13

such items will be held in trust for the Collateral Agent and will not be
commingled by the Pledgor with any of its other funds or property.  The
Collateral Agent is hereby authorized and empowered to endorse the name of the
Pledgor on any check, draft or other instrument for the payment of money
received by the Collateral Agent on account of any Collateral Payment if the
Collateral Agent believes such endorsement is necessary or desirable for
purposes of collection.

                 (c)  The Pledgor will indemnify and save harmless the
Collateral Agent from and against all liabilities and out-of-pocket expenses on
account of any adverse claim asserted against the Collateral Agent relating to
any moneys received by the Collateral Agent on account of any Collateral
Payment and such obligation of the Pledgor shall continue in effect after and
notwithstanding the discharge of the Secured Obligations and the release of the
Security Interest; provided that the foregoing indemnity shall not be
applicable to any liabilities and expenses suffered by the Collateral Agent to
the extent such liabilities and expenses are determined by a judgment of a
court that is binding on the Pledgor and the Collateral Agent, final and not
subject to review on appeal, to be the result of acts or omissions on the part
of the Collateral Agent constituting willful misconduct or gross negligence.

                 Section 12.  Authorized Action by Agent and Collateral Agent.
The Pledgor hereby irrevocably appoints each of the Agent and the Collateral
Agent as its attorney-in-fact, coupled with an interest, to do (but neither the
Agent nor the Collateral Agent shall be obligated to and shall incur no
liability to any Pledgor or any third party for not so doing), at the request
and direction of the Required Banks at any time while an Event of Default has
occurred and is continuing, any act which such Pledgor is obligated by this
Agreement to do, and to exercise such rights and powers as such Pledgor might
exercise with respect to the Collateral, including the right to (a) collect by
legal proceedings or otherwise and endorse, receive and receipt for all
interest, payments, proceeds and other sums and property now or hereafter
payable on or on account of the Collateral; (b) enter into any extension,
reorganization, deposit, merger, consolidation or other agreement pertaining
to, or deposit, surrender, accept, hold or apply other property in exchange
for, the Collateral; (c) insure, process and preserve the Collateral; (d)
transfer the Collateral to the Agent's or the Collateral





                                      -11-
<PAGE>   14

Agent's own or its nominee's name; (e) sell or otherwise dispose of the
Collateral as provided in Section 13 below; and (f) make any compromise or
settlement, and take any other action it deems advisable with respect to the
Collateral.  Notwithstanding anything contained herein, in no event shall the
Agent, the Collateral Agent or any Principal be required to make any
presentment, demand or protest or give any notice, and neither the Agent, the
Collateral Agent nor any Principal need take any action to preserve any rights
against any prior party or any other Person in connection with the Secured
Obligations or with respect to the Collateral.  The Collateral Agent is, and
shall at all times continue to be, authorized to file financing statements (and
amendments to, and continuation statements in respect of, financing statements)
with respect to the Collateral without the signature of any Pledgor in such
filing offices as the Collateral Agent shall deem necessary or appropriate for
the purpose of perfecting and maintaining the perfection of the Security
Interest provided for herein.  The Collateral Agent shall give the Pledgor a
copy of each filing so made prior thereto or promptly thereafter.

                 Section 13.  Default.  (a) Remedies.  Upon the occurrence of
and during an Event of Default and following the acceleration of the Secured
Obligations as provided in Article 6 of the Credit Agreement, the Collateral
Agent shall, at the request and direction of the Required Banks, without notice
to or demand upon the Pledgor: (i) foreclose or otherwise enforce the Security
Interest in the Collateral in any manner permitted by law or provided for
hereunder; (ii) sell or otherwise dispose of the Collateral or any part thereof
at one or more public or private sales, whether or not such Collateral is
present at the place of sale, for credit or future delivery (without the
assumption of any credit risk), on such terms and in such manner as the Agent
effecting such sale or other disposition may determine to be commercially
reasonable (taking into account the circumstances under which the Collateral is
being sold); (iii) require the Pledgor to assemble the Collateral and make such
available to the Collateral Agent at a place to be designated by such Agent;
(iv) enter onto property where any Collateral is located and take possession
thereof with or without judicial process; (v) prior to the disposition of the
Collateral, prepare it for disposition in any manner and to the extent the
Collateral Agent deems appropriate; and (vi) (A) date each FNMA Power of
Attorney, (B) deliver each FNMA Power of Attorney to FNMA, and





                                      -12-
<PAGE>   15

(C) exercise all of its rights and powers under each FNMA Power of Attorney.
Each Bank shall have the right with respect to any such public sale or sales,
and, to the extent permitted by applicable law, upon any such private sale or
sales, to purchase the whole or any part of the Collateral so sold, and the
Secured Obligations of the Pledgor to such purchaser may be applied as a credit
against the purchase price.  Upon any sale or other disposition pursuant to
this Agreement, the Collateral Agent shall have the right to deliver, assign
and transfer to the purchaser thereof (including any Bank) the Collateral or
portion thereof so sold or disposed of and all proceeds thereof shall be
promptly transmitted to the Collateral Agent and the Collateral Agent may apply
such proceeds to payment of the Loans and other amounts owing under the Loan
Documents, or may hold such proceeds as cash collateral.  Each purchaser
(including any Bank) at any such sale or other disposition shall hold the
Collateral free from any claim or right of whatever kind.  The Pledgor hereby
specifically waives (to the extent permitted by law) all rights of stay or
appraisal which it has or may have under any rule of law or statute now
existing or hereafter adopted.

                 (b)  Successor Servicer.  In furtherance and not in limitation
of the rights of the Collateral Agent set forth in this Section 13, the
Collateral Agent may, with respect to Servicing Rights Collateral, in addition
to any other rights it may have, do one or more of the following, subject to
the terms of any relevant Servicing Contract and any acknowledgement agreement
between the Collateral Agent and any party to a Servicing Contract:  (i)
succeed the Pledgor as servicer of any or all of the Mortgage Loans covered
thereby as absolute assignee thereof and not merely as security; (ii) appoint a
third party as successor servicer of any or all of the Mortgage Loans covered
thereby; (iii) sell to a third party or itself or otherwise transfer the
Pledgor's right, title, interest and obligations thereunder, including the
right to hold the escrow deposits; or (iv) require the Pledgor, notwithstanding
any action taken by the Collateral Agent under clause (iii), to remain as
servicer of any or all of the Mortgage Loans covered thereby.  The Collateral
Agent's rights under clauses (i), (ii) and (iii) of this Section 13(b) shall
respectively include, to the extent permitted under the applicable Servicing
Contract, the applicable Guidelines and any acknowledgement agreement between
the Collateral Agent and any party to a Servicing Contract, the right to
succeed the Pledgor as servicer, appoint a successor servicer or transfer any





                                      -13-
<PAGE>   16

or all Servicing Contracts in the event that the Pledgor, or any successor to
the Pledgor in bankruptcy or similar proceedings, rejects any such Servicing
Contracts.  As successor servicer under clause (i), the Collateral Agent shall
notify all interested Persons thereof and take such further action as it shall
deem necessary or appropriate.  Upon the Collateral Agent (x) succeeding the
Pledgor as servicer under clause (i), (y) appointing a third party as a
successor servicer of any or all of the Mortgage Loans covered by any Servicing
Contracts under clause (ii), or (z) transferring any of the Pledgor's rights,
title, interest and obligations under clause (iii), the Pledgor shall have no
further rights under or with respect to the Servicing Contracts (or to such
rights, title, interest or obligations in the case of a transfer under clause
(iii)), to any other documents pertaining thereto or to the related escrow
deposits.  Upon the exercise by the Collateral Agent of any option hereunder,
the Pledgor shall (i) upon request of the Collateral Agent, deliver to the
Collateral Agent all escrow deposits and other deposits in its possession or
under its control and (ii) cooperate with the Collateral Agent in every respect
in effecting such succession.  In the event that the Collateral Agent or any
appointee of the Collateral Agent succeeds the Pledgor as successor servicer of
any or all of the Mortgage Loans covered by any Servicing Contracts, the
Collateral Agent or the appointee, as the case may be, shall only assume those
obligations which a successor servicer of such Mortgage Loans is obligated to
assume; provided that neither the Collateral Agent nor its appointee shall be
liable for any failure of the Pledgor to perform its obligations relating to
any Servicing Contracts or for any other breach thereof.

                 Nothing contained in this Section 13 shall be construed as an
assumption by the Collateral Agent, the Agent, any other Principal or any of
their respective appointees of any liability of the Pledgor with respect to any
of the Collateral, and the Pledgor shall be and remain responsible, on a joint
and several basis, for all such liabilities.

                 (c)  Application of Proceeds.  All cash proceeds received by
the Collateral Agent upon any sale of, collection of, or other realization
upon, all or any part of the Collateral and all cash held by the Collateral
Agent as Collateral shall, subject to the Collateral Agent's right to continue
to hold the same as Collateral, be applied as follows:





                                      -14-
<PAGE>   17

                 First:  To the payment of all out-of-pocket costs and expenses
         incurred in connection with the sale of or other realization upon
         Collateral, including attorney's fees and disbursements;

                 Second:  To the payment of the Secured Obligations in such
         order as the Secured Party may elect (with the Pledgor remaining
         liable for any deficiency); and

                 Third:  To the extent of the balance (if any) of such 
         proceeds, to the payment to the Pledgor, subject to Applicable Law and
         to any duty to pay such balance to the holder of any subordinate lien
         in the Collateral.

                 Section 14.  Cumulative Rights.  The rights, powers and
remedies of the Collateral Agent, the Agent and the other Principals under this
Agreement shall be in addition to all rights, powers and remedies given to such
Persons by virtue of any statute or rule of law, the Credit Agreement or any
other agreement, all of which rights, powers and remedies shall be cumulative
and may be exercised successively or concurrently without impairing the
Security Interest.  Without limiting the generality of the foregoing, the
Collateral Agent, as Secured Party, shall have all rights of a secured party
under the New York Uniform Commercial Code and other applicable New York law.

                 Section 15.  Waiver.  Any waiver, forbearance, failure or
delay by the Collateral Agent or the other Principals in exercising, or the
exercise or beginning of exercise by the Collateral Agent or the other
Principals of, any right, power or remedy, simultaneous or later, shall not
preclude the further, simultaneous or later exercise thereof, and every right,
power or remedy of the Collateral Agent or the other Principals shall continue
in full force and effect.

                 Section 16.  Binding Upon Successors.  All rights of the
Collateral Agent and the other Principals under this Agreement shall inure to
the benefit of the Collateral Agent and the other Principals and their
successors and assigns, and all obligations of the Pledgor shall bind its
successors and assigns.

                 Section 17.  Entire Agreement; Severability.  This Agreement
contains the entire security agreement and agency





                                      -15-
<PAGE>   18

agreement, with respect to the Collateral, among the Collateral Agent, the
other Principals and the Pledgor.  All waivers by the Pledgor provided for in
this Agreement have been specifically negotiated by the parties with full
cognizance and understanding of their rights.  If any of the provisions of this
Agreement shall be held invalid or unenforceable, this Agreement shall be
construed as if not containing such provisions, and the rights and obligations
of the parties hereto shall be construed and enforced accordingly.

                 Section 18.  CHOICE OF LAW.  THE RIGHTS AND DUTIES OF THE
PLEDGOR, THE AGENT, THE COLLATERAL AGENT AND ANY OTHER PRINCIPAL UNDER THIS
AGREEMENT SHALL, PURSUANT TO NEW YORK GENERAL OBLIGATIONS LAW SECTION 5-1401,
BE CONSTRUED IN ACCORDANCE WITH AND GOVERNED BY THE LAWS OF THE STATE OF NEW
YORK.

                 Section 19.  Amendments, Etc.  No amendment or waiver of any
provision of this Agreement, nor consent to any departure by the Pledgor or the
Collateral Agent herefrom, shall be effective unless the same shall have been
effected in accordance with Section 9.07 of the Credit Agreement.

                 Section 20.  Notice.  Any written notice, consent or other
communication provided for in this Agreement shall be delivered or sent as
provided in Section 9.01 of the Credit Agreement.

                 Section 21.  Execution in Counterparts.  This Agreement may be
executed in counterparts, each of which when so executed shall be deemed to be
an original and all of which when taken together shall constitute one and the
same agreement.

                 Section 22.  Interpretation.  (a)  Definitions.  Except as
otherwise specified herein, terms defined in the Credit Agreement are used
herein with the meanings therein ascribed to them.  For purposes of this
Agreement:

                 "Agreement" means this Agreement including all schedules,
annexes and exhibits hereto.

                 "Agreement Date" means the date set forth as such on the last
signature page hereof, which date is the date executed copies of this Agreement
were delivered by all parties hereto





                                      -16-
<PAGE>   19

and, accordingly, the date this Agreement became effective and, for the first
time, binding upon such parties.

                 "Borrowing Base Servicing Rights" means HFI Borrowing Base
Servicing Rights or HFS Borrowing Base Servicing Rights, as the case may be.

                 "Collateral" means the Pledgor's interest (WHATEVER IT MAY BE)
in each of the following, IN EACH CASE WHETHER NOW OR HEREAFTER EXISTING OR NOW
OWNED OR HEREAFTER ACQUIRED BY THE PLEDGOR AND WHETHER OR NOT THE SAME IS NOW
CONTEMPLATED, ANTICIPATED OR FORESEEABLE, which is subject to Article 8 or 9 of
the Uniform Commercial Code or constitutes Collateral by reason of one or more
than one of the following clauses, AND WHEREVER THE SAME MAY BE LOCATED:

                 (a)      All Servicing Rights Collateral;

                 (b)      All Receivables Collateral;

                 (c)  All equipment, books, records, ledgercards, files,
correspondence, computer programs, tapes, disks and related data processing
software (owned by the Pledgor or in which it has an interest) that at any time
evidence or contain information relating to any Collateral or are otherwise
necessary or helpful in the collection thereof or realization thereupon;

                 (d)  All claims (including the right to sue or otherwise
recover on such claims) (A) to items referred to or included in the definition
of Collateral, (B) under warranties relating to any Collateral and (C) against
third parties for (1)(aa) loss, destruction, requisition, confiscation,
condemnation, seizure, forfeiture or infringement of, or damage to, any
Collateral and, (bb) proceeds payable under or unearned premiums with respect
to policies of insurance relating to any Collateral and (2) breach of any
Contract constituting Collateral; and

                 (e)  All products and Proceeds of Collateral in whatever form.
The inclusion of "proceeds" of Collateral in the definition of "Collateral"
shall not be deemed a consent by the Secured Party to any sale or other
disposition of any Collateral not otherwise specifically permitted by the terms
hereof.



                                      -17-
<PAGE>   20
                 "Collateral Payment" has the meaning ascribed to that term in
Section 11.

                 "Credit Agreement" means the Second Amended and Restated
Secured Revolving/Term Credit Agreement, dated as of July 31, 1996, among
Resource Bancshares Mortgage Group, Inc., the Banks listed on the signature
pages thereof, Bank One, Texas, N.A., First Bank National Association,
NationsBank of Texas, N.A. and Texas Commerce Bank National Association, as
Co-Agents, and The Bank of New York, as Agent and as Collateral Agent.

                 "Designated Purchase and Sale Agreement" means a Purchase and
Sale Agreement delivered by the Pledgor to the Collateral Agent and marked
"Designated Purchase and Sale Agreement", including each confirmation of the
sale of Servicing Rights thereunder (each such confirmation a "Confirmation of
Sale").

                 "FNMA Power of Attorney" means a Power of Attorney in the form 
of Schedule 2.

                 "Guidelines" shall mean the GNMA Guide, the FNMA Selling
Guide, the FNMA Servicing Guide and the FHLMC Sellers' & Servicers' Guide.

                 "Investor" has the meaning ascribed to that term in Section 1.

                 "Pledgor" means Resource Bancshares Mortgage Group, Inc., a 
Delaware corporation.

                 "Proceeds" means whatever is receivable or received when
Collateral is sold, collected, exchanged or otherwise disposed of, whether such
disposition is voluntary or involuntary, and includes all rights to payment,
including return premiums, with respect to any insurance relating thereto.

                 "Questionnaire" means a questionnaire in the form of Schedule 
3.

                 "Receivable" means a right or claim of the Pledgor to the
payment or receipt of money or other form of consideration or compensation of
any kind at any time now or hereafter owing or to be owing or claimed or that
could be claimed to be owing


                                      -18-
<PAGE>   21

(whether, if subject to the Uniform Commercial Code, classified thereunder as
accounts, contract rights, chattel paper, general intangibles, instruments,
securities or otherwise) including all such rights and claims in, to and under:

                 (i) (A) accounts, (B) contracts including guaranties and
         contracts of insurance of all kinds, (C) judgments and (D) all other
         debts, obligations and Liabilities in whatever form now or hereafter,
         and

                 (ii) all causes of action, whether sounding in tort, contract
         or otherwise and whether or not currently in litigation,

in either case arising under, related to, or any way connected with, a
Designated Purchase and Sale Agreement.

                 "Receivables Collateral" means:

                 (a)  All Receivables;

                 (b)  All Designated Purchase and Sale Agreements;

                 (c)  All Bank Accounts; and

                 (d)  All goods and other property, whether or not delivered,
the sale or furnishing of which gives or purports to give rise to any
Receivable.

                 "Release Documents" has the meaning ascribed to that term in
Section 3.

                 "Secured Obligations" means all Liabilities of the Pledgor
owing to, or in favor or for the benefit of, or purporting to be owing to, or
in favor or for the benefit of, the Principals under the Loan Documents to
which the Pledgor is a party, in each case (i) WHETHER NOW EXISTING OR
HEREAFTER ARISING OR ACQUIRED and whether or not jointly owed with others, and
(ii) whether owing to, or in favor or for the benefit of, or purporting to be
owing to, or in favor or for the benefit of, Persons that are Principals as of
the Agreement Date or that become Principals by reason of any succession or
assignment at any time thereafter.





                                      -19-
<PAGE>   22

                 "Secured Party" means the Collateral Agent, acting both on its
own behalf as Collateral Agent and as an agent for and representative (within
the meaning of Section 9-105(m) of the Uniform Commercial Code) of the other
Principals.

                 "Security Interest" means the mortgages, pledges and
assignments to the Secured Party of, the continuing security interest of the
Secured Party in, and the continuing lien of the Secured Party upon, the
Collateral intended to be effected by the terms of this Agreement or any of the
other Loan Documents.

                 "Servicing Rights Collateral" means:

                 (a) (i) All Servicing Rights owned, purchased or otherwise
acquired by the Pledgor, including but not limited to any such Servicing Rights
that arise under Servicing Contracts that relate to pools of Mortgage Loans
that were originated or acquired by the Pledgor, and (ii) all Servicing
Contracts under which such Servicing Rights arise;

                 (b)  All Hedge Contracts relating to Servicing Rights
Collateral referred to clause (a) above;

                 (c)  All accounts, contract rights and general intangibles
constituting, arising under or connected with any of the Servicing Rights
Collateral;

                 (d)  All documents of title, policies and certificates of
insurance, securities, chattel paper and other documents or instruments
evidencing or pertaining to any Servicing Rights Collateral; and

                 (e)  All guaranties, Liens on real or personal property,
leases and other agreements and property that in any way secure or relate to
any Servicing Rights Collateral, or are acquired for the purpose of securing
and enforcing any item thereof.

                 (b)  Other Interpretative Provisions.  (i)  Except as
otherwise specified herein, all references herein (A) to any Person shall be
deemed to include such Person's successors and assigns, (B) to any Applicable
Law defined or referred to herein shall be deemed references to such Applicable
Law or any successor Applicable Law as the same may have been or may be





                                      -20-
<PAGE>   23

amended or supplemented from time to time and (C) to any Loan Document or
Contract defined or referred to herein shall be deemed references to such Loan
Document or Contract (and, in the case of any instrument, any other instrument
issued in substitution therefor) as the terms thereof may have been or may be
amended, supplemented, waived or otherwise modified from time to time.

                          (ii)  When used in this Agreement, the words
"herein", "hereof" and "hereunder" and words of similar import shall refer to
this Agreement as a whole and not to any provision of this Agreement, and the
words "Section", "Annex", "Schedule" and "Exhibit" shall refer to Sections of,
and Annexes, Schedules and Exhibits to, this Agreement unless otherwise
specified.

                          (iii) Whenever the context so requires, the neuter
gender includes the masculine and feminine, the masculine gender includes the
feminine, and the singular number includes the plural, and vice versa.

                          (iv)  Any item or list of items set forth following
the word "including", "include" or "includes" is set forth only for the purpose
of indicating that, regardless of whatever other items are in the category in
which such item or items are "included", such item or items are in such
category, and shall not be construed as indicating that the items in the
category in which such item or items are "included" are limited to such items
or to items similar to such items.

                          (v)   Each power of attorney, license and other
authorization in favor of the Secured Party or any other Person granted by or
pursuant to this Agreement shall be deemed to be irrevocable and coupled with
an interest.

                          (vi)  Except as otherwise indicated, any reference
herein to the "Collateral", the "Secured Obligations", the "Loan Documents", or
any other collective or plural term shall be deemed a reference to each and
every item included within the category described by such collective or plural
term, so that (A) a reference to the "Collateral" or the "Secured Obligations"
shall be deemed a reference to any or all of the Collateral or the Secured
Obligations as the case may be, and (B) a reference to the "obligations" of the
Pledgor under the "Loan Documents" shall be deemed a reference to each and
every obligation under





                                      -21-
<PAGE>   24

each and every Loan Document of the Pledgor, whether any such obligation is
incurred under one, some or all of the Loan Documents.

                 (c)  Captions.  Captions to Sections and subsections are
included for convenience or reference only and shall not constitute a part of
this Agreement for any other purpose or in any way affect the meaning or
construction of any provision of this Agreement.





                                      -22-
<PAGE>   25

                 EXECUTED as of the Agreement Date.



                                       Pledgor

                                       RESOURCE BANCSHARES MORTGAGE GROUP, INC.,
                                         a Delaware corporation


                                       By:
                                           -------------------------------------
                                           Name:
                                           Title:



                                       Secured Party and Collateral Agent


                                       THE BANK OF NEW YORK,
                                         as Secured Party and Collateral Agent


                                       By:
                                           -------------------------------------
                                           Name:
                                           Title:


Agreement Date: July 31, 1996