<PAGE>

                                                                    EXHIBIT 10.6

                      PROMISSORY NOTE AND PLEDGE AGREEMENT

$922,320.00                                                    December 14, 2001

     FOR VALUE RECEIVED, the undersigned Brian Casey ("Maker"), hereby promises
                                                       -----
to pay to the order of WESTWOOD HOLDINGS GROUP, INC., a Delaware Company
("Payee") or holder hereof, at Payee's address, 300 Crescent Court, Suite 1300,
  -----
Dallas, Texas 75201 or such other address as Payee may designate in writing from
time to time, a principal sum of nine hundred twenty two thousand, three hundred
twenty and No/100 Dollars ($922,320.00) (the "Principal Sum"), together with
                                              -------------
interest thereon, as provided in this promissory note (this "Note").
                                                             ----
     For purposes of this Note, the following terms shall have the following
definitions:

Maturity Date: the earlier of (i) ninety (90) days after Maker's employment
-------------
               with Payee is terminated (A) by Payee for Cause, or (B) by Maker
               without Good Reason, or (ii) December 14, 2010.

Cause:         (i) dishonesty, fraud or gross insubordination with respect to
-----
               the business or affairs of Payee or, (ii) conviction of a crime
               or acceptance of a plea bargain relating to a felony or a
               misdemeanor if, in the opinion of the Board of Directors, the
               employee's ability to carry out his/her responsibilities would be
               materially impaired, or (iii) the violation of any substance
               abuse policy of Payee that would result in discharge under such
               policy as applied to Payee's employees generally.

Change of
---------
Control:       (i) a merger or consolidation of Payee with or into another
-------
               corporation in which Payee shall not be the surviving corporation
               (other than a merger undertaken solely in order to reincorporate
               in another state) (for purposes hereof, Payee shall not be deemed
               the surviving corporation in any such transaction if, as the
               result thereof, it becomes a wholly-owned subsidiary of another
               corporation), (ii) a dissolution of Payee, (iii) a transfer of
               all or substantially all of the assets of Payee in one
               transaction or a series of related transactions to one or more
               other persons or entities, (iv) a transaction or series of
               transactions that results in any entity, "Person" or "Group" (as
               defined below), becoming the beneficial owner, directly or
               indirectly, of securities of Payee representing more than 50% of
               the combined voting power of Payee's then outstanding securities,
               or (v) during any period of two (2) consecutive years commencing
               on or after January 1, 2002, individuals who at the beginning of
               the period constituted Payee's Board of Directors cease for any
               reason to constitute at least a majority, unless the election of
               each director who was not a director at the beginning of the
               period has been approved in advance by directors representing at
               least two-thirds (2/3) of the directors then in office who were
               directors at the beginning of the period; provided, however, that
                                                         --------  -------
               a "Change of Control" shall not be deemed to have occurred if the
               ownership of 50% or more of the combined voting power of the
               surviving corporation, asset transferee or Company (as the case
               may be), after giving effect to the transaction or series of
               transactions, is directly or indirectly held by (A) a trustee or
               other fiduciary under an employee benefit plan maintained by
               Payee, (B) one or more of the "executive officers" of Payee that
               held such positions prior to the transaction or series of
               transactions, or any entity, Person or Group under their control.
               As used herein, "Person" and "Group" shall have the meanings set
               forth in Sections 13(d)(3) and/or 14(d)(2) of the Securities
               Exchange Act of 1934, as amended, and "executive officer" shall
               have the meaning set forth in Rule 3b-7 promulgated under such
               Act.

Collateral:    240 shares of common stock of Westwood Holdings Group, Inc.
----------

Good Reason:   (i) a resignation occurring within ninety (90) days following
-----------
               a Change in Control; (ii) the relocation of Payee's principal
               place of business to a location more than 100 miles from its
               location as of the date of this Note without Maker's consent; or
               (iii) a material reduction in Maker's salary or bonus
               opportunity, or Maker's responsibilities.

Page 1 of 3

<PAGE>

Interest Rate: 3.93% per annum, compounded semi-annually.
-------------

          1.   Background. Maker has executed this Note to evidence Maker's
               ----------
obligation to repay the Principal Sum together with interest thereon, as
provided in Section 2 of this Note, and to provide the Collateral as collateral
security for the obligations of Maker evidenced by this Note.

          2.   Interest. This Note shall bear interest at the Interest Rate set
               --------
forth above. Maker shall pay all accrued and unpaid interest annually, with the
first interest payment date being February 1, 2003, and on the first day of each
February thereafter until the Maturity Date.

          3.   Events of Default; Remedies.
               ---------------------------

               (a)  Each of the following shall constitute an "Event of Default"
          under this Note:

                    (i)   The failure of Maker to pay when due any part of the
               Principal Sum, interest or other payment required to be made
               pursuant to this Note.

                    (ii)  The failure of Maker to observe or perform any other
               obligation set forth in this Note or that certain Stock Purchase
               Agreement between SWS Group, Inc. ("SWS") and Maker, dated as of
               December 14, 2001 (the "SPA");

                    (iii) Any transfer of the Collateral by Maker to SWS as a
               result of an exercise (A) by SWS of any call right, or (B) by
               Maker of any put right, pursuant to the terms of the SPA; or

                    (iv)  If Maker (a) makes a transfer in fraud of creditors,
               or makes an assignment for the benefit of creditors, of a
               substantial part of Maker's property; (b) admits in writing
               Maker's inability to pay his or her debts as they become due; (c)
               has a receiver, trustee or custodian appointed for, or take
               possession of, all or substantially all of Maker's assets, either
               in a proceeding brought by Maker or in a proceeding brought
               against Maker and such appointment is not discharged or such
               possession is not terminated within sixty (60) days after the
               effective date thereof or he or she consents to or acquiesces in
               such appointment or possession; (d) files a petition for relief
               under the United States Bankruptcy Code or any other present or
               future federal or state insolvency, bankruptcy or similar laws
               (all of the foregoing hereinafter collectively called "Applicable
               Bankruptcy Law") or an involuntary petition for relief is filed
               against Maker under any Applicable Bankruptcy Law and such
               involuntary petition is not dismissed within sixty (60) days
               after the filing thereof, or an order for relief naming Maker is
               entered under any Applicable Bankruptcy Law, or any composition,
               rearrangement, extension, reorganization or other relief of
               debtors now or hereafter existing is requested or consented to by
               Maker; (e) fails to have discharged within a period of thirty
               (30) days any attachment, sequestration or similar writ levied
               upon a substantial part of Maker's property; or (f) fails to pay
               within thirty (30) days any final money judgment against Maker.

               (b)  Upon the occurrence and during the continuance of an Event
          of Default under this Note, Payee may declare the entire unpaid
          Principal Sum and accrued and unpaid interest on this Note immediately
          due and payable, without further notice, demand, or presentment,
          foreclose any liens or security interests securing all or any part
          hereof, or exercise any other right or remedy to which Payee may be
          entitled by agreement, at law, or in equity; provided, however, that
                                                       --------  -------
          no Event of Default shall be deemed to have occurred unless notice of
          the event giving rise to a potential Event of Default shall have been
          delivered to Maker and Maker shall have failed to correct or cure such
          event within fifteen (15) days following his receipt of such notice;
          provided further, however, that upon any Event of Default under
          ----------------  -------
          Section 3(a)(iii) and (iv) hereof, this Note shall automatically
          become due and payable without necessity of any notice or action on
          the part of Payee and without the ability of Maker to cure such event.
          Payee shall have all rights and remedies available to it under the
          Uniform Commercial Code as adopted in the State of Texas.

Page 2 of 3

<PAGE>

          5.   Costs of Collection. If this Note is placed in the hands of an
               -------------------
attorney for collection, or if it is collected through any legal proceedings,
Maker agrees to pay court costs, reasonable attorneys' fees, and other costs of
collection of Payee.

          6.   Certain Waivers. Maker waives presentment and demand for payment,
               ---------------
protest, notice of intention to accelerate, notice of acceleration, and notice
of protest and nonpayment, and agrees that his liability on this Note shall not
be affected by, and hereby consents to, any renewal or extension in the time of
payment hereof, any indulgences, or any release or change in any security for
the payment of this Note.

          7.   Pledge Agreement; Partial Release of Collateral. As security for
               -----------------------------------------------
payment of the Principal Sum and any accrued and unpaid interest thereon, Maker
hereby grants to Payee a continuing interest in all of Maker's right, title and
interest in, to and under the Collateral, which shall be retained by Payee, and
all products and proceeds of such Collateral, now existing or hereafter arising
(the "Pledge"). The Collateral shall be released from the Pledge upon Maker's
payment in full of the Principal Sum, together with all accrued interest
thereon. In addition, upon Maker's request, a partial release of the Collateral
shall be made provided, however, that Maker delivers cash to Payee in an amount
              --------  -------
sufficient to pay the pro-rata portion of the outstanding Principal Sum and any
unpaid interest accrued thereon attributable to the Collateral to be released
(the "Pro-Rata Debt Amount").

          8.   ENTIRETY. THIS NOTE REPRESENTS THE FINAL AGREEMENT BETWEEN MAKER
               --------
AND PAYEE AND MAY NOT BE CONTRADICTED BY EVIDENCE OF PRIOR, CONTEMPORANEOUS, OR
SUBSEQUENT ORAL AGREEMENTS BY THE PARTIES. THERE ARE NO UNWRITTEN ORAL
AGREEMENTS BETWEEN THE PARTIES.

          9.   GOVERNING LAW. THE VALIDITY, CONSTRUCTION, AND ENFORCEABILITY OF
               -------------
THIS NOTE SHALL BE GOVERNED BY THE LAWS OF THE STATE OF TEXAS AND THE UNITED
STATES OF AMERICA.

          10.  Parties Bound. This Note is binding upon and shall inure to the
               -------------
benefit of Maker, Payee, and their respective successors and assigns. Maker may
not assign any of its rights or obligations hereunder without Payee's prior
written consent, and any purported assignment thereof without Payee's prior
written consent shall be void and ineffective. Payee, upon prior written notice
to Maker, shall be entitled to assign its rights and duties hereunder to any
subsequent holder of this Note who shall for all purposes hereof thereafter be
"Payee" hereunder the same as if originally named as "Payee" herein.


                                MAKER:

                                 /s/ Brian Casey
                                -----------------------------------------------
                                Brian Casey



                                PAYEE:

                                Westwood Holdings Group, Inc.,
                                a Delaware corporation

                                By: /s/ Susan M. Byrne
                                    -------------------------------------------
                                Name:    Susan M. Byrne
                                    -------------------------------------------
                                Title:   Chairman & CEO
                                      -----------------------------------------

Page 3 of 3