<PAGE>   1
                                                                   EXHIBIT 10.20

[SEARCH CAPITAL GROUP, INC. LETTERHEAD]

                                  May 24, 1995

                                                   VIA FACSIMILE


Mr. Barry Ridings 
Alex. Brown & Sons 
1290 Avenue of the Americas 
10th Floor 
New York, NY 10104

Dear Barry:

         Attached please find your letter agreement executed as of this date.
My compliments for your changes in a very timely fashion.

         We also will mail a hard copy to follow. Attached also please find for
your information a prospective news release, subject to the finalization of the
written legal settlement today. We should be issuing this tomorrow, if not, at
the latest, next Tuesday.

         Please confirm with me or Linda your clearance related to Alex. Brown's
portion. We will be talking with you next week.

                                   Sincerely,

                                   [ILLEGIBLE]

GCE:li

Attachments

cc:      Joe Dorman
         Robert Idzi

P.S.     Please clear the news release as soon as possible today. Thanks.
<PAGE>   2
[ALEX. BROWN & SONS INCORPORATED LETTERHEAD]

                                  May 23, 1995

Mr. George C. Evans
Chairman of the Board, President
and Chief Executive Officer
Search Capital Group, Inc.
700 N. Pearl Street - Suite 400
Dallas, TX 75201-7490

Dear George:

         This letter agreement (the "Agreement") confirms the understanding and
agreement between Search Capital Group, Inc. ("Search" or the "Company"), and
Alex. Brown & Sons Incorporated ("Alex. Brown" or the "Financial Advisor") as
follows:

         1.      The Company hereby retains Alex. Brown to render financial
                 advisory services to the Company, on the terms and subject to
                 the conditions set forth herein, in connection with the
                 restructuring of certain secured notes (the "Notes") of eight
                 Securitization Subsidiaries of the Company. The Notes
                 currently total approximately $70 million in aggregate
                 outstanding principal as detailed in Exhibit I.  Such proposed
                 restructuring, whether by a solicitation of waivers and
                 consents with respect to the Notes or an exchange offer
                 involving new securities, or a settlement on the existing
                 securities, or other similar transactions is hereinafter
                 referred to as the "Restructuring".

         2.      Alex. Brown's services will include, as necessary:

                 (a)      Evaluating from a financial point of view the
                          Company's business operations and projections;

                 (b)      Determining the Company's and each of the
                          Securitization Subsidiaries' debt capacity in light
                          of its projected cash flows;

                 (c)      Rendering financial advice to the Company and
                          participating in any meetings or negotiations with
                          the Noteholders and/or their representatives in
                          connection with any restructuring, modification or
                          refinancing of the Notes, including any solicitation
                          of waivers, consents or exchanges that may be
                          required as a part of the Restructuring;
<PAGE>   3
Search Capital Group, Inc.
May 23, 1995
Page 2

                 (d)      Rendering financial advice to the Company and
                          participating in any meetings or negotiations with
                          GECC or any other prospective lender in connection
                          with any secured lending facility;

                 (e)      Providing the Company with general capital
                          restructuring advice;

                 (f)      Assisting the Company in the development of a
                          specific restructuring proposal to present to the
                          Noteholders;

                 (g)      Advising the Company on specific tactics and strategy
                          for negotiating with the Noteholders;

                 (h)      Assisting the Company with potential exchange offers;

                 (i)      Advising the Company with respect to the timing, the
                          nature and the terms of securities or other
                          inducements to be offered pursuant to the
                          Restructuring;

                 (j)      Assisting the Company in the preparation of any
                          documentation required in connection with the
                          restructuring of the Notes;

                 (k)      Providing the Company with general restructuring
                          advice from a financial point of view, as required.

         3.      As consideration for Alex. Brown's services as set forth
                 herein, the Company agrees to pay Alex. Brown certain
                 restructuring fees as follows:

                 (a)      Upon acceptance of this Agreement and on the 15th of
                          each succeeding month, the Company shall pay Alex.
                          Brown a monthly cash fee of $60,000.

                 (b)      Upon the completion of the Restructuring, the Company
                          shall pay Alex. Brown a success fee, in the form of
                          cash, equal to 8/10ths of one percent (i.e. 0.8% of
                          the aggregate outstanding principal amount of any
                          Notes participating in such Restructuring. The
                          monthly fees described in 3(a) above will be
                          credited against the fee due, in 3(b). In the event
                          no Restructuring is undertaken, Alex. Brown will only
                          be paid the monthly cash fee, as provided for in 3(a)
                          above.

                 (c)      Upon the completion of the Restructuring, the Company
                          shall pay Alex. Brown a success fee, in the form of
                          common stock of Search Capital Group, Inc., equal to
                          4/10th of one percent (i.e. 0.4%) of the aggregate
                          outstanding principal amount of any Notes
                          participating in such Restructuring. In calculating
                          the number of shares to be provided to Alex. Brown,
                          the success
<PAGE>   4
Search Capital Group, Inc.
May 23, 1995
Page 3

                          fee shall be divided by the average of the per share
                          closing bid price for the 30 consecutive trading days
                          beginning with the first trading day which is 60
                          calendar days after the completion of the
                          Restructuring. In the event that trading prices are
                          not available for any 10 or more of these 30
                          consecutive trading days, Alex. Brown shall have the
                          option to receive 50% of this fee in cash.

                 (d)      In the event that the Restructuring is completed
                          under a pre-packaged bankruptcy filing, the Company
                          shall endeavor to retain Alex. Brown as its financial
                          advisor on terms consistent with this Agreement.

                 (e)      Except as noted in paragraph 2 above, the advisory
                          services and compensation arrangements set forth
                          herein do not encompass other investment banking
                          services, such as raising new debt or equity capital,
                          the solicitation of a third party investor, specific
                          asset sales, of any other similar transaction which
                          might be undertaken by Alex. Brown, nor do they
                          include any expansion of the Financial Advisor role
                          beyond that outlined in this letter. To the extent
                          that the Company requires such additional services,
                          the Alex. Brown will be entitled to compensation
                          relating to such services as mutually agreed to by
                          Alex. Brown and the Company.

         4.      This Agreement may be terminated by the Company or the
                 Financial Advisor by written notice, with or without cause,
                 effective immediately upon receipt. Notwithstanding
                 termination of this Agreement, the Financial Advisor will be
                 entitled to receive: (i) reimbursement of all out-of-pocket
                 expenses through the effective date of termination, (ii) any
                 monthly restructuring fees that have been earned prior to such
                 termination, and (iii) the full benefit of the indemnity and
                 reimbursement provisions detailed in Addendum A which shall
                 remain in full force and effect. Furthermore, if within six
                 months after termination by the Company, the company completes
                 a restructuring substantially similar to such restructuring
                 recommended by the Financial Advisor, then the Financial
                 Advisor shall be entitled to the success fee in 3(b) and 3(c)
                 above.

         5.      In addition to any fees that may be payable to the Financial
                 Advisor, the Company shall reimburse the Financial Advisor for
                 all: (i) out-of-pocket expenses (including travel and lodging,
                 data processing and communications charges, courier and other
                 appropriate expenditures, and any expenses of counsel incurred
                 by the Financial Advisor with the Company's consent) and (ii)
                 fees and expenses (including expenses of counsel, if any) to
                 the extent described in Addendum A. Alex. Brown agrees to
                 promptly notify the Company if it retains counsel. Alex. Brown
                 will also provide the Company with a periodic summary of its
                 expenses and will notify the Company when its total expenses
                 have reached $25,000 and in increments of $10,000
<PAGE>   5
Search Capital Group, Inc.
May 23, 1995
Page 4

                 thereafter. Alex. Brown's expense policy is attached as
                 Addendum B.

         6.      In connection with the Financial Advisor's activities, the
                 Company will furnish the Financial Advisor with all
                 information concerning the Company which is necessary for the
                 Financial Advisor to perform its duties under this Agreement
                 and provide access to the Company's management, and its
                 outside attorneys and accountants (the "Information"). The
                 Company represents that to the best of its knowledge, unless
                 indicated otherwise, all Information will be complete and
                 correct in all material respects and will not contain any
                 materially untrue statement of a material fact. The Financial
                 Advisor will be using and relying on the Information without
                 independent verification thereof or independent appraisal of
                 any of the Company's assets and the Financial Advisor shall
                 bear no liability with respect to any services rendered by the
                 Financial Advisor pursuant to this Agreement, in which it used
                 or relied upon, directly or indirectly, any Information which
                 was incomplete or incorrect in any material respect or
                 contained any untrue statement of a material fact or omitted
                 to state a material fact necessary in order to make the
                 statements therein not misleading in light of the
                 circumstances in which they were made. The Company agrees that
                 it will provide all information necessary to the Financial
                 Advisor to perform its duties including but not limited to
                 adequate and sufficient information as to all material aspects
                 of the Restructuring.

         7.      The Company is signing this Agreement solely for the purpose
                 of inducing the Financial Advisor to act hereunder and
                 undertakes to provide all compensation, reimbursement, as well
                 as indemnification and contribution as provided herein.
                 Accordingly, the Company agrees to the indemnification and
                 contribution provisions (the "Indemnification Provisions")
                 attached to this Agreement as Addendum A and incorporated
                 herein in their entirety, as well as the compensation and
                 reimbursement provisions contained herein.

         8.      The benefits of this Agreement shall inure to the respective
                 successors and assigns of the parties hereto and of the
                 indemnified parties hereunder and their respective successors
                 and assigns and representatives, and the obligations and
                 liabilities assumed in this Agreement by the parties hereto
                 shall survive the termination of the Financial Advisor's
                 engagement and shall be binding upon their respective
                 successors and assigns. Alex. Brown agrees not to assign this
                 Agreement without the prior consent of the Company, which will
                 not be unreasonably withheld.

         9.      The Company will not make any written reference to the
                 Financial Advisor or this Agreement or any statements
                 contained herein in connection with any proposal made by the
                 Company or in any press release or other public disclosure
                 announcing any such proposal or the acceptance thereof or in
                 any public filing by the Company
<PAGE>   6
Search Capital Group, Inc.
May 23, 1995
Page 5

                 without the prior approval of the Financial Advisor, which
                 shall not be unreasonably withheld except to the extent that
                 such disclosure is required by law or requested by any
                 governmental or regulatory agency or body.

         10.     This Agreement and the indemnification letter, attached as
                 Addendum A, incorporate the entire understanding of the
                 parties with respect to the subject matter of this Agreement
                 and supersede all previous agreements should they exist.

         11.     This Agreement may be executed in counterparts, each of which
                 shall be deemed an original and all of which shall constitute
                 one and the same instrument.

         12.     This Agreement may not be amended or modified except in
                 writing executed by the Company and the Financial Advisor, and
                 shall be governed by and construed in accordance with the laws
                 of the State of New York, without regard to principles of
                 conflicts of law.

         If the foregoing letter is in accordance with your understanding of
the terms of our engagement, please sign and return to us the enclosed
duplicate hereof.

                                        Very truly yours,

                                        ALEX. BROWN & SONS INCORPORATED

                                        By: /s/ BARRY W. RIDINGS
                                            ------------------------------------
                                            Barry W. Ridings
                                            Managing Director

Accepted and Agreed:

Search Capital Group, Inc.

By: /s/ GEORGE C. EVANS
    -------------------------------------
    George C. Evans
    Chairman of the Board, President
    and Chief Executive Officer
<PAGE>   7
                                   ADDENDUM A

         In connection with the engagement of Alex. Brown & Sons Incorporated
("Alex. Brown") as a financial advisor, pursuant to a separate engagement
letter (the "Letter") between Alex. Brown and the Company (as defined in the
Letter), the Company agrees to indemnify and hold harmless Alex. Brown and each
of its directors, officers, agents, employees and controlling persons (within
the meaning of the Securities Act of 1933, as amended) against any losses,
claims, damages or liabilities (or actions or proceedings in respect thereof)
related to or rising out of Alex. Brown's engagement and will reimburse Alex.
Brown and each other person indemnified hereunder for all legal and other
expenses as such expenses are incurred in connection with investigating or
defending any such loss, claim, damage, liability, action or proceeding whether
or not in connection with pending or threatened litigation in which Alex. Brown
or any other indemnified person is a party; provided, however, that the Company
will not be liable in any such case (except cases arising out of the use of
information provided by the Company) for losses, claims, damages, liabilities
or expenses that a court of competent jurisdiction shall have found in a final
judgment to have arisen primarily from the gross negligence or willful
misconduct of the person seeking indemnification.

         In case any proceeding shall be instituted involving any person in
respect of whom indemnity may be sought, such person (the "indemnified party")
shall promptly notify the Company, but failure to so notify the Company will
not relieve it from any liability which it may have hereunder or otherwise,
except to the extent such failure materially prejudices the Company's rights.
The Company, upon the request of the indemnified party, shall retain counsel
reasonably satisfactory to the indemnified party to represent the indemnified
party and any others the Company may designate in such proceeding and shall pay
as incurred the fees and expenses of such counsel related to such proceeding.
In any such proceeding, any indemnified party shall have the right to retain
its own counsel at its own expense, except that the Company shall pay as they
are incurred the fees and expenses of counsel retained by the indemnified party
in the event that (i) the Company and the indemnified party shall have mutually
agreed to the retention of such counsel or, (ii) the named parties to any such
proceeding (including any impleaded parties) include both the Company and the
indemnified party and representation of both parties by the same counsel would
be inappropriate, in the reasonable opinion of the indemnified party, due to
actual or potential differing interests between them. The Company will not be
liable, without its consent, for the fees and expenses of more than one counsel
in any proceeding.

         The Company shall not be liable for any settlement of any proceeding
effected without its written consent, but if settled with such consent or if
there is a final judgment for the plaintiff, the Company agrees to indemnify
the indemnified party to the extent set forth herein. In addition the Company
will not, without the prior written consent of Alex. Brown, which consent will
not be unreasonably withheld, settle or compromise or consent to the entry or
any judgment in any pending or threatened claim, action, suit or proceeding in
respect of which indemnification may be sought hereunder (whether or not Alex.
Brown or any indemnified party is an actual or potential party to such claim,
action, suit or proceeding) unless such settlement, compromise or consent
includes an unconditional release of Alex. Brown and each other indemnified
party hereunder from all liability arising out of such claim, action, suit or
proceeding.
<PAGE>   8
         In the event a claim for indemnification hereunder is determined to be
unenforceable by a final judgment of a court of competent jurisdiction, then
the Company shall contribute to the aggregate losses, claims, damages or
liabilities to which Alex. Brown or its officers, directors, agents, employees
or controlling persons may be subject in such amount as is appropriate to
reflect the relative benefits received by each of the Company and the party
seeking contribution on the one hand and the relative faults of the Company and
the party seeking contribution on the other, as well as any other relevant
equitable considerations.

         This indemnification shall apply to the engagement as set forth in the
Letter and any modification of the engagement and the indemnification provided
herein shall survive termination of Alex. Brown's engagement and shall be
binding upon any successors or assigns of the Company.

Acknowledged and Agreed:

Search Capital Group, Inc.

By: /s/ GEORGE C. EVANS
    -------------------------------------
    George C. Evans
    Chairman of the Board, President
    and Chief Executive Officer

Date: 5/24/95
      -------

<PAGE>   9
                                   EXHIBIT I
                             (AS OF MARCH 31, 1995)

<TABLE>
<CAPTION>
                                                                 NOTE
               FUND                                             PAYABLE
                                                                         
               <S>                                            <C>
               ACF                                             $1,948,000
                                                                         
               ACF 91-III                                         665,000
                                                                         
               ACP                                             10,000,000
                                                                         
               ACF 92-II                                       10,000,000
                                                                         
               ACF III                                         15,000,000
                                                                         
               ACF IV                                          10,000,000
                                                                         
               ACF V                                           19,872,000
                                                                         
               ACF VI                                          10,675,000
                                                              -----------
               TOTAL                                          $69,160,000
</TABLE>
<PAGE>   10
                                   ADDENDUM B

                               ALEX. BROWN & SONS
                           EXPENSE POLICY GUIDELINES

         Alex. Brown's charges for expenses to the client are determined
pursuant to Alex. Brown's Travel and Entertainment Policy. Out-of-pocket
expenses incurred by Alex. Brown are charged to a client if the expenses are
incurred in connection with services rendered for such particular client. In
particular, Alex. Brown's internal policies with respect to out-of-pocket
expenses billed to clients are set forth below:

(a)      TRAVEL AND LODGING: All airfare charges billed to a client are based
on coach fare rates. Booking air travel first class is permitted only: (1) when
traveling with clients who are traveling first class, (2) and on domestic red
eye flights. Business class over and coach on the return flight is permitted on
international flights exceeding 7 hours.  Business class round trip is
permitted for international flights exceeding 10 hours. Alex. Brown has only
traveled coach airfare on this assignment.

         With respect to local travel, Alex. Brown's general policy enables
employees to travel by taxi or private car service to and from meetings while
rendering services to a client on a client related matter, for which the client
is charged. This policy is based on Alex. Brown's determination that travel by
taxi or private car service is the most efficient use of a professional's time.
Alex. Brown employees are not permitted to charge commutation expenses to a
client unless the employee is travelling after 8:00 p.m. or on a weekend.
<PAGE>   11
(b)      MEALS: Alex. Brown's general policy permits its employees to bill
lunch or dinner meals to a client if the employee is required to render
services during such meal time to the client due to extreme time constraints.
Alex. Brown employees are also permitted to bill clients for meals while
travelling on behalf of clients and their related matters.  Alex. Brown
employees are permitted to order meals in the office if the Alex, Brown
employee is required to work after 8:00 p.m. or on weekends,

(c)      COMPUTER ACCESS: Alex. Brown bills its clients for time spent by its
employees on computer databases. Alex.  Brown purchases or leases such
databases from third party sources typically for a monthly fee. Such databases
are used to retrieve market price data (ie. such as stock or bond prices) and
financial information. For        example, information from Securities Data
Company, a capital markets database, is required in analyzing comparable
companies.  Alex. Brown charges clients a fee of $250.00 a month for these
computer charges.

(d)      POSTAGE AND MAILING: Messengers and couriers (including Federal
Express) are used by Alex. Brown to deliver hard copy documents relating to the
client matter which require receipt on an expedited basis; otherwise, Alex.
Brown uses the regular postal system. Any charges for either messengers and
couriers are billed to a client at cost.

(e)      PRINTING EXPENSES: Alex. Brown's ability to produce color charts,
large volumes of reports and certain types of presentation slides is limited by
space and mechanical constraints. Outside production companies are used for
projects that can not be handled by Alex. Brown's in-house production
capabilities. Any charges for outside production are billed to a client at
cost.
<PAGE>   12
(f)      DISCLOSURE INFORMATION SERVICES: Outside financial research consists
of charges from outside services, principally Disclosure Information Services,
which supply, for a fee, financial documents to Alex. Brown. Such outside
charges are billed to a client at cost. Financial research services generally
consist of the retail of financial documents filed with the Securities and
Exchange Commission or other governmental and regulatory agencies.

(g)      REPRODUCTION COSTS: Alex. Brown bills photocopying charges at the rate
of $.25 per page.

(h)      COMMUNICATIONS: Alex. Brown employees must stay in contact with their
office while they are travelling. These charges consist of phone cards and air
or rail phone.