<PAGE>

                                                                    EXHIBIT 10.5

                      PROMISSORY NOTE AND PLEDGE AGREEMENT

$2,766,960.00                                                  December 14, 2001

         FOR VALUE RECEIVED, the undersigned Susan Byrne ("Maker"), hereby
                                                           -----
promises to pay to the order of WESTWOOD HOLDINGS GROUP, INC., a Delaware
Company ("Payee") or holder hereof, at Payee's address, 300 Crescent Court,
          -----
Suite 1300, Dallas, Texas 75201 or such other address as Payee may designate in
writing from time to time, a principal sum of two million seven hundred sixty
six thousand, nine hundred sixty and No/100 Dollars ($2,766,960.00) (the
"Principal Sum"), together with interest thereon, as provided in this promissory
 -------------
note (this "Note").
            ----

         For purposes of this Note, the following terms shall have the following
definitions:

Maturity Date:    the earlier of (i) ninety (90) days after Maker's employment
-------------
                  with Payee is terminated (A) by Payee for Cause, or (B) by
                  Maker without Good Reason, or (ii) December 14, 2010.

Cause:            (i) dishonesty, fraud or gross insubordination with respect to
-----
                  the business or affairs of Payee or, (ii) conviction of a
                  crime or acceptance of a plea bargain relating to a felony or
                  a misdemeanor if, in the opinion of the Board of Directors,
                  the employee's ability to carry out his/her responsibilities
                  would be materially impaired, or (iii) the violation of any
                  substance abuse policy of Payee that would result in discharge
                  under such policy as applied to Payee's employees generally.

Change of
---------
Control:          (i) a merger or consolidation of Payee with or into another
-------
                  corporation in which Payee shall not be the surviving
                  corporation (other than a merger undertaken solely in order to
                  reincorporate in another state) (for purposes hereof, Payee
                  shall not be deemed the surviving corporation in any such
                  transaction if, as the result thereof, it becomes a
                  wholly-owned subsidiary of another corporation), (ii) a
                  dissolution of Payee, (iii) a transfer of all or substantially
                  all of the assets of Payee in one transaction or a series of
                  related transactions to one or more other persons or entities,
                  (iv) a transaction or series of transactions that results in
                  any entity, "Person" or "Group" (as defined below), becoming
                  the beneficial owner, directly or indirectly, of securities of
                  Payee representing more than 50% of the combined voting power
                  of Payee's then outstanding securities, or (v) during any
                  period of two (2) consecutive years commencing on or after
                  January 1, 2002, individuals who at the beginning of the
                  period constituted Payee's Board of Directors cease for any
                  reason to constitute at least a majority, unless the election
                  of each director who was not a director at the beginning of
                  the period has been approved in advance by directors
                  representing at least two-thirds (2/3) of the directors then
                  in office who were directors at the beginning of the period;
                  provided, however, that a "Change of Control" shall not be
                  --------  -------
                  deemed to have occurred if the ownership of 50% or more of the
                  combined voting power of the surviving corporation, asset
                  transferee or Company (as the case may be), after giving
                  effect to the transaction or series of transactions, is
                  directly or indirectly held by (A) a trustee or other
                  fiduciary under an employee benefit plan maintained by Payee,
                  (B) one or more of the "executive officers" of Payee that held
                  such positions prior to the transaction or series of
                  transactions, or any entity, Person or Group under their
                  control. As used herein, "Person" and "Group" shall have the
                  meanings set forth in Sections 13(d)(3) and/or 14(d)(2) of the
                  Securities Exchange Act of 1934, as amended, and "executive
                  officer" shall have the meaning set forth in Rule 3b-7
                  promulgated under such Act.

Collateral:       720 shares of common stock of Westwood Holdings Group, Inc.
----------

Good Reason:      (i) a resignation occurring within ninety (90) days following
-----------
                  a Change in Control; (ii) the relocation of Payee's principal
                  place of business to a location more than 100 miles from its
                  location as of the date of this Note without Maker's consent;
                  or (iii) a material reduction in Maker's salary or bonus
                  opportunity, or Maker's responsibilities.

Page 1 of 3

<PAGE>

Interest Rate:    3.93% per annum, compounded semi-annually.
-------------

     1.   Background. Maker has executed this Note to evidence Maker's
          ----------
obligation to repay the Principal Sum together with interest thereon, as
provided in Section 2 of this Note, and to provide the Collateral as collateral
security for the obligations of Maker evidenced by this Note.

     2.   Interest. This Note shall bear interest at the Interest Rate set forth
          --------
above. Maker shall pay all accrued and unpaid interest annually, with the first
interest payment date being February 1, 2003, and on the first day of each
February thereafter until the Maturity Date.

     3.   Events of Default; Remedies.
          ---------------------------

          (a)  Each of the following shall constitute an "Event of Default"
     under this Note:

               (i)   The failure of Maker to pay when due any part of the
          Principal Sum, interest or other payment required to be made pursuant
          to this Note.

               (ii)  The failure of Maker to observe or perform any other
          obligation set forth in this Note or that certain Stock Purchase
          Agreement between SWS Group, Inc. ("SWS") and Maker, dated as of
          December 14, 2001 (the "SPA");

               (iii) Any transfer of the Collateral by Maker to SWS as a result
          of an exercise (A) by SWS of any call right, or (B) by Maker of any
          put right, pursuant to the terms of the SPA; or

               (iv)  If Maker (a) makes a transfer in fraud of creditors, or
          makes an assignment for the benefit of creditors, of a substantial
          part of Maker's property; (b) admits in writing Maker's inability to
          pay his or her debts as they become due; (c) has a receiver, trustee
          or custodian appointed for, or take possession of, all or
          substantially all of Maker's assets, either in a proceeding brought by
          Maker or in a proceeding brought against Maker and such appointment is
          not discharged or such possession is not terminated within sixty (60)
          days after the effective date thereof or he or she consents to or
          acquiesces in such appointment or possession; (d) files a petition for
          relief under the United States Bankruptcy Code or any other present or
          future federal or state insolvency, bankruptcy or similar laws (all of
          the foregoing hereinafter collectively called "Applicable Bankruptcy
          Law") or an involuntary petition for relief is filed against Maker
          under any Applicable Bankruptcy Law and such involuntary petition is
          not dismissed within sixty (60) days after the filing thereof, or an
          order for relief naming Maker is entered under any Applicable
          Bankruptcy Law, or any composition, rearrangement, extension,
          reorganization or other relief of debtors now or hereafter existing is
          requested or consented to by Maker; (e) fails to have discharged
          within a period of thirty (30) days any attachment, sequestration or
          similar writ levied upon a substantial part of Maker's property; or
          (f) fails to pay within thirty (30) days any final money judgment
          against Maker.

          (b)  Upon the occurrence and during the continuance of an Event of
     Default under this Note, Payee may declare the entire unpaid Principal Sum
     and accrued and unpaid interest on this Note immediately due and payable,
     without further notice, demand, or presentment, foreclose any liens or
     security interests securing all or any part hereof, or exercise any other
     right or remedy to which Payee may be entitled by agreement, at law, or in
     equity; provided, however, that no Event of Default shall be deemed to have
             --------  -------
     occurred unless notice of the event giving rise to a potential Event of
     Default shall have been delivered to Maker and Maker shall have failed to
     correct or cure such event within fifteen (15) days following his receipt
     of such notice; provided further, however, that upon any Event of Default
                     ----------------  -------
     under Section 3(a)(iii) and (iv) hereof, this Note shall automatically
     become due and payable without necessity of any notice or action on the
     part of Payee and without the ability of Maker to cure such event. Payee
     shall have all rights and remedies available to it under the Uniform
     Commercial Code as adopted in the State of Texas.

Page 2 of 3

<PAGE>

     5.   Costs of Collection. If this Note is placed in the hands of an
          -------------------
attorney for collection, or if it is collected through any legal proceedings,
Maker agrees to pay court costs, reasonable attorneys' fees, and other costs of
collection of Payee.

     6.   Certain Waivers. Maker waives presentment and demand for payment,
          ---------------
protest, notice of intention to accelerate, notice of acceleration, and notice
of protest and nonpayment, and agrees that his liability on this Note shall not
be affected by, and hereby consents to, any renewal or extension in the time of
payment hereof, any indulgences, or any release or change in any security for
the payment of this Note.

     7.   Pledge Agreement; Partial Release of Collateral. As security for
          -----------------------------------------------
payment of the Principal Sum and any accrued and unpaid interest thereon, Maker
hereby grants to Payee a continuing interest in all of Maker's right, title and
interest in, to and under the Collateral, which shall be retained by Payee, and
all products and proceeds of such Collateral, now existing or hereafter arising
(the "Pledge"). The Collateral shall be released from the Pledge upon Maker's
payment in full of the Principal Sum, together with all accrued interest
thereon. In addition, upon Maker's request, a partial release of the Collateral
shall be made provided, however, that Maker delivers cash to Payee in an amount
              --------  -------
sufficient to pay the pro-rata portion of the outstanding Principal Sum and any
unpaid interest accrued thereon attributable to the Collateral to be released
(the "Pro-Rata Debt Amount").

     8.   ENTIRETY. THIS NOTE REPRESENTS THE FINAL AGREEMENT BETWEEN MAKER AND
          --------
PAYEE AND MAY NOT BE CONTRADICTED BY EVIDENCE OF PRIOR, CONTEMPORANEOUS, OR
SUBSEQUENT ORAL AGREEMENTS BY THE PARTIES. THERE ARE NO UNWRITTEN ORAL
AGREEMENTS BETWEEN THE PARTIES.

     9.   GOVERNING LAW. THE VALIDITY, CONSTRUCTION, AND ENFORCEABILITY OF THIS
          -------------
NOTE SHALL BE GOVERNED BY THE LAWS OF THE STATE OF TEXAS AND THE UNITED STATES
OF AMERICA.

     10.  Parties Bound. This Note is binding upon and shall inure to the
          -------------
benefit of Maker, Payee, and their respective successors and assigns. Maker may
not assign any of its rights or obligations hereunder without Payee's prior
written consent, and any purported assignment thereof without Payee's prior
written consent shall be void and ineffective. Payee, upon prior written notice
to Maker, shall be entitled to assign its rights and duties hereunder to any
subsequent holder of this Note who shall for all purposes hereof thereafter be
"Payee" hereunder the same as if originally named as "Payee" herein.

                                  MAKER:


                                         /s/ SUSAN BYRNE
                                  ----------------------------------------------
                                  Susan Byrne


                                  PAYEE:

                                  Westwood Holdings Group, Inc.,
                                  a Delaware corporation

                                  By:    /s/ BRIAN O. CASEY
                                     -------------------------------------------
                                  Name:  Brian O. Casey
                                       -----------------------------------------
                                  Title: President & COO
                                        ----------------------------------------

Page 3 of 3