<PAGE>

                                                                    Exhibit 10.4

                           HIGHLANDS INSURANCE GROUP
                          1997 RESTRICTED STOCK PLAN
                       (as amended through May 10, 1999)


1.   Purpose

     This Plan's purpose is to align the interests of management more closely
with those of stockholders, by providing an incentive to invest in Highlands
Insurance Group, Inc. common stock, and by rewarding long service with the
Company and its Subsidiaries.

2.   Definitions

     For purposes of this Plan, the following terms shall have the definitions
set forth below:

     (a) "Board."  The Company's Board of Directors.

     (b) "Committee."  The Compensation Committee of the Board.  No member of
the Committee shall participate in the administration of the Plan unless he is a
nonemployee director described in Rule 16b-3(e)(3) promulgated by the Securities
and Exchange Commission or any successor definition thereto.

     (c) "Company."  Highlands Insurance Group, Inc., a Delaware corporation.

     (d) "Date of Issuance."  The date Restricted Shares are issued to a
Participant.

     (e) "Escrow Agent."  The bank or other institution appointed by the Company
from time to time to hold Restricted Shares during the Restricted Period.

     (f) "Participant."  An employee of the Company or a Subsidiary selected for
participation in the Plan pursuant to Section 4.

     (g) "Plan."  The Highlands Insurance Group 1997 Restricted Stock Plan.

     (h) "Restricted Period."  The period described in Section 6(c).

     (i) "Restricted Shares."  The shares of common stock of the Company
reserved pursuant to Section 3 hereof and any such shares issued to a
Participant pursuant to this Plan.

     (j) "Subsidiary" or "Subsidiaries."  A corporation or corporations of which
the Company owns, directly or indirectly, shares having a majority of the
ordinary voting power for the election of directors.
<PAGE>

3.   Restricted Share Reserve

     (a) Establishment.  The Company shall establish a Restricted Share reserve
to which shall be credited 350,000 shares of the common stock of the Company,
par value $.01 per share.  Should the shares of the Company's common  stock be
increased or decreased, or changed into or exchanged for, a different number or
kind of shares of stock or other securities of the Company or of another
corporation, due to a stock split or stock dividend or combination of shares or
any other change, or exchange for other securities, by reclassification,
reorganization, merger, consolidation, recapitalization or otherwise, the number
of shares then remaining in the Restricted Share reserve shall be adjusted
appropriately to reflect such action.  If any such adjustment results in a
fractional share, the fraction shall be disregarded.

     (b) Adjustments to Reserve.  Upon the grant of shares hereunder, the
reserve shall be reduced by the number of shares so granted.  Upon the
forfeiture of any Restricted Shares, the reserve shall be increased by such
number of shares, and such shares may again be the subject of grants hereunder.

     (c) Source of Restricted Shares.  As the Board shall in its sole discretion
determine, Restricted Shares may be authorized but unissued shares or treasury
shares.  All authorized and unissued shares issued as Restricted Shares in
accordance with the Plan shall be fully paid and non-assessable shares and free
from preemptive rights.

4.   Eligibility

     (a) Eligible Employees.  Any management employee of the Company or any
Subsidiary (including officers and directors, except for persons serving as
directors only) shall be eligible to receive a grant of Restricted Shares
pursuant to the Plan.

     (b) Selection by the Committee.  From the employees eligible to receive
grants pursuant to the Plan, the Committee may from time to time select
employees to receive grants.  The Committee shall base its selections on the
positions and responsibilities of the eligible employees, the value of their
services to the Company and its Subsidiaries and such other factors as the
Committee deems pertinent.

     (c) Participation in Other Stock Plans.  A person who has received options
or other rights to purchase stock of the Company or a Subsidiary may exercise
the same in accordance with their terms, and shall not by reason thereof be
ineligible to receive Restricted Shares under this Plan.  A person who has
received Restricted Shares hereunder shall not be ineligible for that reason to
be granted any option or other rights to purchase stock.

5.   Amounts of Grants

     (a) Participant Election.  The number of Restricted Shares granted to each
Participant shall be equal to the number of shares of common stock of the
Company, if any, that the Participant purchases under the Plan.  Each
Participant shall elect in writing, within

                                      -2-
<PAGE>

the time permitted by the Committee, how many shares of common stock of the
Company to purchase pursuant to the Plan. Such an election shall be made by
delivering to the Secretary of the Company a written election on the form
provided by the Company. Any such election shall be accompanied by a stock power
endorsed in blank in order to permit the Escrow Agent to transfer Restricted
Shares to the Company in the event they are forfeited.

     (b) Purchase of Shares.  The Secretary of the Company shall arrange for
purchase in the market of the number of shares elected by Participants.  The
Secretary shall inform each Participant promptly of the amount needed to
purchase the shares elected by the Participant, and of the date by which payment
must be received.  The election of any Participant who fails to make timely
payment in full to the Secretary shall be null and void.

     (c) Disposition of Purchased Shares.  The Secretary of the Company shall
cause all shares purchased at the election of Participants to be delivered to
the respective Participants or their designees by the broker executing the
transaction.

     (d) Listing on Stock Exchange.  The Company shall take such action as shall
be necessary to cause any Restricted Shares issued pursuant to this Plan and not
previously listed to be listed on the New York Stock Exchange and/or such other
exchange(s) on which shares of the same class as the Restricted Shares are then
listed.

     (e) Delivery of Written Notice.  All elections and notices in writing
required pursuant to this Plan shall be sufficient only if actually delivered or
if sent via registered or certified mail, postage prepaid, to the Company,
attention Secretary, and/or the Escrow Agent at its principal office, and shall
be conclusively deemed given on the third business day following the date of
such mailing, if mailed.

5A.  Exchange Offer

     (a) Offer.  Each Participant shall be entitled to exchange his Restricted
Shares granted under Section 5 for new Restricted Shares in accordance with this
Section 5A.  The number of new Restricted Shares granted shall be 75% of the
number of Restricted Shares that the Participant elects to exchange.

     (b) Participant Election.  Each Participant shall elect in writing, within
the time specified on the Exchange Offer Election form, whether he wishes to
accept the exchange offer described in (a) above.  Such an election shall apply
to all of a Participant's Restricted Shares.  The election shall be made by
delivering to the Secretary of the Company a written election on the form
provided by the Company.  An election to accept the exchange offer shall be
accompanied by a stock power endorsed in blank in order to permit the Escrow
Agent to transfer the new Restricted Shares to the Company in the event they are
forfeited.

     (c) Date of Issuance.  The Date of Issuance of new Restricted Shares issued
under this Section 5A shall be March 10, 1999 for all purposes under this Plan.

6.   Restrictions

                                      -3-
<PAGE>

     (a) Transfer/Issuance.  Restricted Shares shall be issued or transferred to
Participants promptly after completion of the purchases pursuant to Section
5(b), and certificates for such shares shall be issued in the respective
Participants' names.  Each Participant shall thereupon be the record owner of
all the shares represented by the certificate or certificates.  As such, the
Participant shall have all the rights of a shareholder with respect to such
shares, including the right to vote them and to receive all dividends and other
distributions (subject to (b), below) paid with respect to them, provided,
however, that the shares shall be subject to the restrictions in (d), below.
The shares represented thereby may not be sold, exchanged, transferred, pledged,
hypothecated, or otherwise disposed of during the Restricted Period, and each
transfer agent for the common stock shall be instructed to that effect.  In aid
of such restrictions, all certificates for Restricted Shares shall be deposited
by the Company with the Escrow Agent.

     (b) Adjustments to Restricted Shares.  If, due to a stock split, stock
dividend, combination of shares, or any other change or exchange for other
securities by reclassification, reorganization, merger, consolidation,
recapitalization or otherwise, the Participant, as the owner of Restricted
Shares subject to restrictions hereunder, shall be entitled during the
Restricted Period to new, additional, or different shares of stock or other
securities, the certificate or certificates for, or other evidences of, such
new, additional, or different shares or securities also shall be deposited by
the issuer with the Escrow Agent.  If any of the event(s) described in the
preceding sentence occur, all Plan provisions relating to restrictions and lapse
of restrictions shall apply to such new, additional, or different shares or
securities, provided, however, that if the Participant shall receive rights,
warrants or fractional interests in respect of any of such Restricted Shares,
such rights or warrants may be held, exercised, sold or otherwise disposed of,
and such fractional interests may be settled, by the Participant free and clear
of the restrictions hereinafter set forth.

     (c) Restricted Period.  The restrictions set forth in (d), below, shall
apply to Restricted Shares during a period starting on the Date of Issuance of
such shares and ending on the fifth anniversary of such date.

     (d) Restrictions.  The restrictions to which Restricted Shares shall be
subject are the following:

         (i)   During the Restricted Period applicable to Restricted Shares,
none of such shares shall be sold, exchanged, transferred, pledged,
hypothecated, or otherwise disposed of; provided, that Restricted Shares may be
transferred to the Company if the Committee offers an exchange for new
Restricted Shares.

         (ii)  If a Participant's employment with the Company and its
Subsidiaries terminates for any reason, including such Participant's death or
disability, at any time before the third anniversary of the Date of Issuance of
Restricted Shares, the Restricted Shares shall be forfeited and transferred to
the Company.

                                      -4-
<PAGE>

         (iii) If a Participant resigns, or is terminated for cause, from
employment with the Company and its Subsidiaries at any time before the
Restricted Period ends, the Participant's Restricted Shares shall be forfeited
and transferred to the Company.

         (iv)  If the price on the date the Restricted Period ends is less than
161% of the price at the Date of Issuance, all or a portion of the Restricted
Shares shall be forfeited, and transferred to the Company, as determined by the
following table.

         Price at End Compared with
         Beginning of Restricted Period              Forfeited Percentage
         ------------------------------              --------------------
         Less than 140%                                        100%
         140% but less than 150%                                75%
         150% but less than 161%                                50%

     (e) Special Definitions.  For purposes of this Section 6, the following
terms shall have the definitions set forth below.

         (i)   Cause means fraud, dishonesty or similar willful misconduct on
the part of a Participant; a material breach by a Participant of any of his
obligations under any employment agreement with the Company or any of its
Subsidiaries or otherwise in connection with any position that such Participant
holds, or in the future may hold, at the Company or any Subsidiary (provided
that the Participant shall first be notified of and be given a reasonable
opportunity to cure such breach); gross negligence by a Participant in the
performance of the services contemplated by such Participant's employment
agreement (if any) or performed in connection with the Participant's position as
an employee of the Company or its Subsidiary; or conviction of a Participant (or
the entering of a plea of guilty, nolo contendere, or request for deferred
adjudication) for fraud, misappropriation, embezzlement, financial misconduct,
any felony, or any lesser criminal offense which carries a potential penalty of
imprisonment for a term of one year or more and/or a fine of $5,000 or more
(other than such a lesser criminal offense that relates solely to the action or
omission of one or more persons other than the Participant), whether or not a
lesser penalty or fine is assessed.

         (ii)  Price on a given date means the average closing price of the
common stock of the Company on the preceding five trading days, as reported on
the principal national securities exchange on which the common stock is then
listed, or if there were no sales on any of such days, on the most recent five
days on which there were sales.

7.   Delivery of Vested Shares

     (a) Lapse of Restricted Period.  After the end of the Restricted Period,
the Escrow Agent shall transfer to the Company any Restricted Shares that are
forfeited pursuant to Section 6(d), above.  Subject to (b), below, the Escrow
Agent shall release to the respective Participants, free of such restrictions,
the remaining Restricted Shares.

                                      -5-
<PAGE>

     (b) The Escrow Agent shall not release shares from escrow to a Participant
or executor or administrator until the Participant or executor or administrator
has remitted to the Company or its Subsidiary all federal, state and local taxes
that the Company or its Subsidiary determines it is required to withhold.

8.   Death of a Participant

     In the event of the death of a Participant prior to the third anniversary
of the Date of Issuance, the Restricted Shares shall be forfeited and the Escrow
Agent shall transfer them to the Company.  If a Participant dies after the third
anniversary of the Date of Issuance, but before the end of the Restricted
Period, the Restricted Shares that are not forfeited pursuant to Section 6(d),
if any, shall be released as provided in Section 7 to the Participant's executor
or administrator.

9.   Finality of Determination

     The Committee shall administer this Plan and construe its provisions.  Any
determination by the Committee in carrying out, administering, or construing
this Plan shall be final and binding for all purposes and upon all interested
persons and their heirs, successors, and personal representatives.

10.  No Right to Continued Employment

     Neither the Company's action in establishing the Plan, nor any action taken
by it or by the Board or the Committee under the Plan, nor any provision of the
Plan, shall be construed as giving to any person the right to be retained in the
employ of the Company or any Subsidiary.

11.  Amendment, Suspension or Termination of Plan

     (a) The Plan shall be submitted for approval by the stockholders of the
Company at the 1998 annual meeting of stockholders.  If the Plan is not approved
by a majority of shares present in person or by proxy, it shall be void and of
no effect, and all Restricted Shares shall be transferred to the Company.

     (b) The Board may amend, suspend or terminate the Plan in whole or in part
at any time; provided that such amendment shall not affect adversely rights or
obligations with respect to grants previously made.

12.  Governing Law

     The Plan shall be governed by the laws of the State of New Jersey.

                                      -6-
<PAGE>

13.  Expenses of Administration

     All costs and expenses incurred in the operation and administration of this
Plan shall be borne by the Company.

14.  Registration of Restricted Shares

     The Company, at its expense, shall register the Plan under the Securities
Act of 1933.

                                      -7-