<PAGE>   1

                                                                 Exhibit 10.6(B)


                           CHANGE OF CONTROL AGREEMENT


         AGREEMENT by and between Resource Bancshares Mortgage Group, Inc., a
Delaware corporation ("RBMG"), and Harold Lewis, Jr. (the "Executive"), dated as
of the 3rd day of May, 2000.

         The Board of Directors of RBMG (the "Board") has determined that it is
in the best interests of RBMG and its shareholders to assure that the Company
(as defined below) will have the continued dedication of the Executive,
notwithstanding the possibility, threat or occurrence of a Change of Control (as
defined below) of RBMG. The Board believes it is imperative to diminish the
inevitable distraction of the Executive by virtue of the personal uncertainties
and risks created by a pending or threatened Change of Control and to encourage
the Executive's full attention and dedication to the Company currently and in
the event of any threatened or pending Change of Control. Therefore, in order to
accomplish these objectives, the Board has caused RBMG to enter into this
Agreement.

         NOW, THEREFORE, IT IS HEREBY AGREED AS FOLLOWS:

         1. Certain Definitions. As used in this Agreement, the following terms
shall have the meanings set forth below:

         (a) "AFR" shall have the meaning set forth in Section 6.

         (b) "Affiliated Company" shall mean any corporation, partnership or
other entity which controls, is controlled by or is under common control with
RBMG.

         (c) "Annual Base Salary" shall have the meaning set forth in Section
1(q)(i).

         (d) "Board" shall have the meaning set forth in the recitals to this
Agreement.

         (e) "Business Combination" shall have the meaning set forth in Section
1(g)(iii).

         (f) "Cause" shall mean:

                  (i) the willful and continued failure of the Executive to
         perform substantially the Executive's duties with the Company (other
         than any such failure resulting from incapacity due to physical or
         mental illness), after a written demand for substantial performance is
         delivered to the Executive by the Board or the Chief Executive Officer
         of RBMG which specifically identifies the manner in which the Board or
         Chief Executive Officer believes that the Executive has not
         substantially performed the Executive's duties; or

                  (ii) the willful engaging by the Executive in illegal conduct
         or gross misconduct which is materially and demonstrably injurious to
         the Company.

<PAGE>   2

         For purposes of this provision, no act, or failure to act, on the part
of the Executive shall be considered "willful" unless it is done, or omitted to
be done, by the Executive in bad faith or without reasonable belief that the
Executive's action or omission was in the best interests of the Company. Any
act, or failure to act, based upon authority given pursuant to a resolution duly
adopted by the Board or upon the instructions of the Chief Executive Officer or
a senior officer of RBMG or based upon the advice of counsel for RBMG shall be
conclusively presumed to be done, or omitted to be done, by the Executive in
good faith and in the best interests of the Company. The cessation of employment
of the Executive shall not be deemed to be for Cause unless and until there
shall have been delivered to the Executive a copy of a resolution duly adopted
by the affirmative vote of not less than three-quarters of the entire membership
of the Board at a meeting of the Board called and held for such purpose (after
reasonable notice is provided to the Executive and the Executive is given an
opportunity, together with counsel, to be heard before the Board), finding that,
in the good faith opinion of the Board, the Executive is guilty of the conduct
described in Section 1(f)(i) or 1(f)(ii) above, and specifying the particulars
thereof in detail.

         (g) "Change of Control" shall mean:

                  (i) The acquisition by any individual, entity or group (within
         the meaning of Section 13(d)(3) or 14(d)(2) of the Securities Exchange
         Act of 1934, as amended (the "Exchange Act")) (a "Person") of
         beneficial ownership (within the meaning of Rule 13d-3 promulgated
         under the Exchange Act) of 20% or more of either (A) the then
         outstanding shares of common stock of RBMG (the "Outstanding RBMG
         Common Stock") or (B) the combined voting power of the then outstanding
         voting securities of RBMG entitled to vote generally in the election of
         directors (the "Outstanding RBMG Voting Securities"); provided,
         however, that for purposes of this Section 1(g)(i), the following
         acquisitions shall not constitute a Change of Control: (W) any
         acquisition directly from RBMG or any corporation controlled by RBMG,
         (X) any acquisition by RBMG or any corporation controlled by RBMG, (Y)
         any acquisition by any employee benefit plan (or related trust)
         sponsored or maintained by RBMG or any corporation controlled by RBMG
         or (Z) any acquisition by any corporation pursuant to a transaction
         which complies with clauses (A), (B) and (C) of Section 1(g)(iii); or

                  (ii) That individuals who, as of the date hereof, constitute
         the Board (the "Incumbent Board") cease for any reason to constitute at
         least a majority of the Board; provided, however, that any individual
         becoming a director subsequent to the date hereof whose election, or
         nomination for election by RBMG's shareholders, was approved by a vote
         of at least a majority of the directors then comprising the Incumbent
         Board shall be considered as though such individual were a member of
         the Incumbent Board, but excluding, for this purpose, any such
         individual whose initial assumption of office occurs as a result of an
         actual or threatened election contest with respect to the election or
         removal of directors or other actual or threatened solicitation of
         proxies or consents by or on behalf of a Person other than the Board;
         or

                  (iii) Consummation of a reorganization, merger or
         consolidation or sale or other disposition of all or substantially all
         of the assets of RBMG or the acquisition of

                                       2

<PAGE>   3

         assets of another corporation (a "Business Combination"), in each case,
         unless, following such Business Combination, (A) all or substantially
         all of the individuals and entities who were the beneficial owners,
         respectively, of the Outstanding RBMG Common Stock and Outstanding RBMG
         Voting Securities immediately prior to such Business Combination
         beneficially own, directly or indirectly, more than 50% of,
         respectively, the then outstanding shares of common stock and the
         combined voting power of the then outstanding voting securities
         entitled to vote generally in the election of directors, as the case
         may be, of the corporation resulting from such Business Combination
         (including, without limitation, a corporation which as a result of such
         transaction owns RBMG or all or substantially all of RBMG's assets
         either directly or through one or more subsidiaries) in substantially
         the same proportions as their ownership, immediately prior to such
         Business Combination, of the Outstanding RBMG Common Stock and
         Outstanding RBMG Voting Securities, as the case may be, (B) no Person
         (excluding any corporation resulting from such Business Combination or
         any employee benefit plan (or related trust) of RBMG or such
         corporation resulting from such Business Combination) beneficially
         owns, directly or indirectly, 20% or more of, respectively, the then
         outstanding shares of common stock or the combined voting power of the
         then outstanding voting securities entitled to vote generally in the
         election of directors, as the case may be, of the corporation resulting
         from such Business Combination (including, without limitation, a
         corporation which as a result of such transaction owns RBMG or all or
         substantially all of RBMG's assets either directly or through one or
         more subsidiaries) except to the extent that such ownership existed
         prior to the Business Combination and (C) at least a majority of the
         members of the board of directors of the corporation resulting from
         such Business Combination were members of the Incumbent Board at the
         time of the execution of the initial agreement, or of the action of the
         Board, providing for such Business Combination; or

                  (iv) Approval by the shareholders of RBMG of a complete
         liquidation or dissolution of RBMG.

         (h) "Change of Control Period" shall mean the period commencing on the
Effective Date and ending on the second anniversary of the Effective Date.

         (i) "Code" shall mean the Internal Revenue Code of 1986, as amended.

         (j) "Company" shall include RBMG and its Affiliated Companies.

         (k) "Date of Termination" means (i) if the Executive's employment is
terminated by RBMG for Cause, or by the Executive for Good Reason, the date of
receipt of the Notice of Termination or any later date specified therein, as the
case may be, (ii) if the Executive's employment is terminated by RBMG other than
for Cause or Disability, the Date of Termination shall be the date on which RBMG
notifies the Executive of such termination and (iii) if the Executive's
employment is terminated by reason of death or Disability, the Date of
Termination shall be the date of death of the Executive or the Disability
Effective Date, as the case may be.

                                       3

<PAGE>   4

         (l) "Disability" shall mean the absence of the Executive from the
Executive's duties with the Company on a full-time basis for 120 consecutive
business days as a result of incapacity due to mental or physical illness which
is determined to be total and permanent by a physician selected by RBMG or its
insurers and acceptable to the Executive or the Executive's legal
representatives.

         (m) "Disability Effective Date" shall have the meaning set forth in
Section 3.

         (n) "Effective Date" shall mean the first date on which a Change of
Control occurs. If a Change of Control occurs and the Executive's employment
with the Company is terminated prior to the date on which the Change of Control
occurs, and if it is reasonably demonstrated by the Executive that such
termination of employment arose in connection with or anticipation of a Change
of Control, then "Effective Date" shall mean the date immediately prior to the
date of such termination of employment.

         (o) "Exchange Act" shall have the meaning set forth in Section 1(g)(i).

         (p) "Executive" shall have the meaning set forth in the recitals to
this Agreement.

         (q) "Good Reason" shall mean:

                  (i) a reduction by the Company in the Executive's annual base
         salary as in effect immediately prior to the Effective Date (the
         "Annual Base Salary");

                  (ii) the Company requiring the Executive to be based at a
         location more than 100 miles from the location at which he is based
         immediately prior to the Effective Date (except for required travel
         which is substantially consistent with travel obligations as of the
         date of this Agreement);

                  (iii) the failure by the Company to pay the Executive any
         portion of the Executive's current compensation within seven days of
         the date such compensation is due, other than an isolated,
         insubstantial and inadvertent failure not occurring in bad faith and
         which is remedied by the Company promptly after receipt of notice
         thereof given by the Executive;

                  (iv) the failure by the Company to continue any material
         benefit plan in which the Executive participates immediately prior to
         the Effective Date (unless the failure did not occur in bad faith and
         is remedied by the Company, promptly after receipt of notice thereof
         given by the Executive, by the Company providing an equitable
         arrangement with respect to such plan);

                  (v) any purported termination by the Company of the
         Executive's employment otherwise than as expressly permitted by this
         Agreement; or

                  (vi) any failure by RBMG to comply with and satisfy Section
         8(c) of this Agreement.


                                      4

<PAGE>   5

         For purposes of this Section 1(q), any good faith determination of
"Good Reason" made by the Executive shall be conclusive.

         (r) "Incumbent Board" shall have the meaning set forth in Section
1(g)(ii).

         (s) "Most Recent Annual Bonus" shall have the meaning set forth in
Section 4(a)(i)(A)(2)(x).

         (t) "Notice of Termination" means a written notice which (i) indicates
the specific termination provision in this Agreement relied upon, (ii) to the
extent applicable, sets forth in reasonable detail the facts and circumstances
claimed to provide a basis for termination of the Executive's employment under
the provision so indicated and (iii) if the Date of Termination is other than
the date of receipt of such notice, specifies the termination date (which date
shall be not more than 30 days after the giving of such notice). The failure by
the Executive or RBMG to set forth in the Notice of Termination any fact or
circumstance which contributes to a showing of Good Reason or Cause shall not
waive any right of the Executive or RBMG, respectively, hereunder or preclude
the Executive or RBMG, respectively, from asserting such fact or circumstance in
enforcing the Executive's or RBMG's rights hereunder.

         (u) "Outstanding RBMG Common Stock" shall have the meaning set forth in
Section 1(g)(i).

         (v) "Outstanding RBMG Voting Securities" shall have the meaning set
forth in Section 1(g)(i).

         (w) "Payment" shall have the meaning set forth in Section 4(c).

         (x) "Person" shall have the meaning set forth in Section 1(g)(i).

         (y) "RBMG" shall have the meaning set forth in the recitals to this
Agreement.

         2. Termination for Cause or Good Reason; Notice of Termination. RBMG
may terminate the Executive's employment with the Company during the Change of
Control Period for Cause and the Executive may terminate employment during the
Change of Control Period for Good Reason. Any termination by RBMG for Cause, or
by the Executive for Good Reason, shall be communicated by Notice of Termination
to the other party hereto in accordance with Section 9(b) of this Agreement.

         3. Termination by Reason of Death or Disability. The Executive's
employment shall terminate automatically upon the Executive's death. If RBMG
determines in good faith during the Change of Control Period that the Disability
of the Executive has occurred, it may give to the Executive written notice in
accordance with Section 9(b) of its intention to terminate the Executive's
employment. In such event, the Executive's employment with the Company shall
terminate effective on the 30th day after receipt of such notice by the
Executive (the "Disability

                                       5

<PAGE>   6

Effective Date"), provided that, within the 30 days after such receipt, the
Executive shall not have returned to full-time performance of the Executive's
duties.

         4. Obligations of RBMG upon Termination. (a) Good Reason; Without
Cause; Death or Disability. If, during the Change of Control Period, RBMG shall
terminate the Executive's employment without Cause, the Executive shall
terminate employment for Good Reason or the Executive's employment terminates
for death or Disability, and subject to the limitation set forth in Section
4(c):

                  (i) RBMG shall pay to the Executive in a lump sum in cash
         within 30 days after the Date of Termination the aggregate of the
         following amounts:

                           A. the sum of (1) the Executive's Annual Base Salary
                  through the Date of Termination to the extent not theretofore
                  paid and (2) the product of (x) the Executive's annual bonus
                  paid or payable, including any bonus or portion thereof which
                  has been earned but deferred, (and annualized for any fiscal
                  year consisting of less than twelve full months or during
                  which the Executive was employed for less than twelve full
                  months) for the most recently completed fiscal year, if any
                  (the "Most Recent Annual Bonus") and (y) a fraction, the
                  numerator of which is the number of days in the current fiscal
                  year through the Date of Termination, and the denominator of
                  which is 365; provided, that, such amount shall not be paid if
                  duplicative of any amounts payable under the Company's annual
                  incentive plans, and (3) any compensation previously deferred
                  by the Executive (together with any accrued interest or
                  earnings thereon) and any unused annual vacation pay, in each
                  case to the extent not theretofore paid; and

                           B. an amount equal to the sum of (1) the Executive's
                  Annual Base Salary and (2) the lesser of (I) the Executive's
                  Most Recent Annual Bonus or (II) $100,000; and

                  (ii) RBMG shall provide medical, dental and vision insurance
         coverage for the Executive and his current spouse and eligible
         dependents until the first to occur of (A) the first anniversary of the
         Date of Termination or (B) the death of the Executive and his spouse,
         that is comparable to the most favorable medical, dental and vision
         insurance coverage provided by the Company for the Executive (and such
         spouse and dependents) during the 120-day period immediately prior to
         the Change of Control, with the Executive (or such spouse) continuing
         to pay the employee portion of the premiums for such coverage (in the
         same pro rata amount as during the 120-day period prior to the Change
         of Control).

         (b) Cause; Without Good Reason. If the Executive's employment shall be
terminated for Cause during the Change of Control Period or if the Executive
voluntarily terminates employment during the Change of Control Period without
Good Reason, this Agreement shall terminate without further obligation to the
Executive other than the obligation to pay to the Executive (i) his Annual Base
Salary through the Date of Termination, and (ii) the amount of any compensation
previously deferred by the Executive.

                                       6

<PAGE>   7

         (c) Notwithstanding anything in this Agreement to the contrary, if it
shall be determined that any payment or distribution by the Company to or for
the benefit of the Executive (whether paid or payable or distributed or
distributable pursuant to the terms of this Agreement or otherwise) (a
"Payment") would constitute an "excess parachute payment" within the meaning of
Section 280G of the Code, then the Payments, in the aggregate, shall be reduced
(in a manner elected by the Executive, or by RBMG if the Executive fails to make
such an election) to the greatest amount that could be paid to the Executive
such that the receipt of Payments would not constitute an "excess parachute
payment."

         5. Non-exclusivity of Rights. Subject to the limitation set forth in
Section 4(c), nothing in this Agreement shall prevent or limit the Executive's
continuing or future participation in any pension, profit sharing, 401(k),
supplemental executive retirement or stock option plan provided by the Company
and for which the Executive may qualify, nor, subject to Section 9(f), shall
anything herein limit or otherwise affect such rights as the Executive may have
under any contract or agreement with the Company. Amounts which are vested
benefits or which the Executive is otherwise entitled to receive under any
pension, profit sharing, 401(k), supplemental executive retirement or stock
option plan or any contract or agreement with the Company at or subsequent to
the Date of Termination shall be payable in accordance with such plan or
contract or agreement except as explicitly modified by this Agreement.

         6. Full Settlement. RBMG's obligation to make the payments provided for
in this Agreement and otherwise to perform its obligations hereunder shall not
be affected by any set-off, counterclaim, recoupment, defense or other claim,
right or action which the Company may have against the Executive or others. In
no event shall the Executive be obligated to seek other employment or take any
other action by way of mitigation of the amounts payable to the Executive under
any of the provisions of this Agreement and such amounts shall not be reduced
whether or not the Executive obtains other employment. RBMG agrees to pay as
incurred, to the full extent permitted by law, all legal fees and expenses which
the Executive may reasonably incur as a result of any contest (regardless of the
outcome thereof) by RBMG, the Executive or others of the validity or
enforceability of, or liability under, any provision of this Agreement or any
guarantee of performance thereof (including as a result of any contest by the
Executive about the amount of any payment pursuant to this Agreement), plus in
each case interest on any delayed payment at the applicable Federal rate
provided for in Section 7872(f)(2)(A) of the Code ( the "AFR"). The Executive
shall repay (with interest at the AFR) any such advanced legal fees and expenses
in the event a court determines that the Executive's claim or action was in bad
faith.

         7. Confidential Information. (a) The Executive shall hold in a
fiduciary capacity for the benefit of the Company all secret or confidential
information, knowledge or data relating to the Company which shall have been
obtained by the Executive during the Executive's employment by the Company and
which shall not be or become public knowledge (other than by acts by the
Executive or representatives of the Executive in violation of this Agreement).
After termination of the Executive's employment with the Company, the Executive
shall not, without the prior written consent of RBMG or as may otherwise be
required by law or legal process, communicate or divulge any such information,
knowledge or data to anyone other than the Company and those designated by RBMG.
In no event shall an asserted violation of the

                                       7

<PAGE>   8

provisions of this Section 7 constitute a basis for deferring or withholding any
amounts otherwise payable to the Executive under this Agreement.

         (b) The Executive agrees and acknowledges that a violation of the
covenants contained in this Section 7 will cause irreparable damage to the
Company, and that it is and will be impossible to estimate or determine the
damage that will be suffered by the Company in the event of a breach by the
Executive of any such covenant. Therefore, the Executive further agrees that in
the event of any violation or threatened violation of such covenants, the
Company shall be entitled as a matter of course to an injunction issued by any
court of competent jurisdiction restraining such violation or threatened
violation by the Executive, such right to an injunction to be cumulative and in
addition to whatever other remedies the Company may have.

         8. Successors. (a) This Agreement is personal to the Executive and
without the prior written consent of RBMG shall not be assignable by the
Executive otherwise than by will or the laws of descent and distribution. This
Agreement shall inure to the benefit of and be enforceable by the Executive's
legal representatives.

         (b) This Agreement shall inure to the benefit of and be binding upon
RBMG and its successors and assigns.

         (c) RBMG will require any successor (whether direct or indirect, by
purchase, merger, consolidation or otherwise) to all or substantially all of the
business and/or assets of RBMG to assume expressly and agree to perform this
Agreement in the same manner and to the same extent that RBMG would be required
to perform it if no such succession had taken place. As used in this Agreement,
"RBMG" shall mean RBMG as hereinbefore defined and any successor to its business
and/or assets as aforesaid which assumes and agrees to perform this Agreement by
operation of law, or otherwise.

         9. Miscellaneous. (a) This Agreement shall be governed by and construed
in accordance with the laws of the State of South Carolina, without reference to
principles of conflict of laws. The captions of this Agreement are not part of
the provisions hereof and shall have no force or effect. This Agreement may not
be amended or modified otherwise than by a written agreement executed by the
parties hereto or their respective successors and legal representatives.

         (b) All notices and other communications hereunder shall be in writing
and shall be given by hand delivery to the other party or by registered or
certified mail, return receipt requested, postage prepaid, addressed as follows:

         If to the Executive:

                  Harold Lewis, Jr.

                  _____________________________________

                  _____________________________________


                                       8

<PAGE>   9

         If to RBMG:

                  Resource Bancshares Mortgage Group, Inc.
                  7909 Parklane Road
                  Columbia, SC  29223
                  Attention:  Chief Executive Officer

or to such other address as either party shall have furnished to the other in
writing in accordance herewith. Notice and communications shall be effective
when actually received by the addressee.

         (c) The invalidity or unenforceability of any provision of this
Agreement shall not affect the validity or enforceability of any other provision
of this Agreement.

         (d) RBMG may withhold from any amounts payable under this Agreement
such Federal, state, local or foreign taxes as shall be required to be withheld
pursuant to any applicable law or regulation.

         (e) The Executive's or RBMG's failure to insist upon strict compliance
with any provision of this Agreement or the failure to assert any right the
Executive or RBMG may have hereunder, including, without limitation, the right
of the Executive to terminate employment for Good Reason pursuant to Section 2,
shall not be deemed to be a waiver of such provision or right or any other
provision or right of this Agreement.

         (f) The Executive and RBMG acknowledge that, except as may otherwise be
provided under any other written agreement between the Executive and the
Company, the employment of the Executive by the Company is "at will" and,
subject to Section 1(n), the Executive's employment may be terminated by either
the Executive or the Company at any time prior to the Effective Date, in which
case the Executive shall have no further rights under this Agreement. This
Agreement shall supersede any other agreement between the parties with respect
to the subject matter hereof.

         (g) This Agreement shall terminate and be of no further force or effect
if a Change of Control does not occur on or before April 3, 2002.

                                       9

<PAGE>   10


         IN WITNESS WHEREOF, the Executive has hereunto set the Executive's
hand and, pursuant to the authorization from its Board of Directors, RBMG has
caused these presents to be executed in its name on its behalf, all as of the
day and year first above written.



                                                   ----------------------------
                                                          Harold Lewis, Jr.



                                                   RESOURCE BANCSHARES MORTGAGE
                                                     GROUP, INC.


                                                   By
                                                      -------------------------
                                                        Douglas K. Freeman
                                                        Chief Executive Officer


                                       10