<PAGE>
 
                                                                EXHIBIT 10.06(b)
                                                                ----------------

                            STOCKHOLDERS' AGREEMENT
                            -----------------------

           AGREEMENT made this 6th day of November , 1996, by and among (i)
Occupational Health + Rehabilitation Inc, a Delaware corporation (the
"Company"), (ii) the individuals and entities listed under the heading "Holders"
on Schedule I attached hereto, and (iii) those persons whose names are set forth
   ----------
under the heading "Investors" on Schedule I hereto (the "Investors").
                                 ----------

           WHEREAS, the Investors are acquiring up to an aggregate of 1,666,667
shares of the Series A Preferred Stock, $.001 par value per share, of the
Company (the "Series A Preferred Stock") pursuant to a certain Series A
Convertible Preferred Stock Purchase Agreement dated as of November 6, 1996, by
and among the Investors and the Company (the "Purchase Agreement");

           NOW, THEREFORE, in consideration of the mutual covenants herein
contained and other good and valuable consideration, the receipt and sufficiency
of which is hereby acknowledged, the Company, the Holders and the Investors
agree as follows:

           1.        Election of Directors.
                     ----------------------

                     Each of the parties hereto agrees to vote all of the 
Stock (as hereinafter defined) of the Company now owned or hereafter acquired by
such party (and attend, in person or by proxy, all meetings of stockholders
called for the purpose of electing directors), and the Company agrees to take
all actions (including, but not limited to the nomination of specified persons)
to cause and maintain the election to the Board of Directors of the Company, to
the extent permitted pursuant to the Company's Restated Certificate of
Incorporation, as amended, the following:

                     (a)       the Chief Executive Officer of the Company, who 
shall initially be John C. Garbarino;

                     (b)       a person designated by those persons designated 
as Telor Principal Stockholders on Schedule II hereto by a majority in interest 
of Stock held by them, who shall initially be Angus M. Duthie;

                     (c)       a person designated by those persons designated 
as OH+R Principal Stockholders on Schedule II hereto by a majority in interest 
of Stock held by them, who shall initially be Kevin J. Dougherty;

                     (d)       two persons designated by Cahill, Warnock 
Strategic Partners Fund, L.P., one of whom shall initially be Edward L. Cahill 
and the other of whom shall be designated at a later date, and
<PAGE>
 
                                      -2-

                     (e)       two persons who shall be unaffiliated with the 
management of the Company and mutually agreeable to all of the other directors.

                     Each of the parties further covenants and agrees to vote, 
to the extent possible, all shares of Stock of the Company now owned or
hereafter acquired by such party so that the Company's Board of Directors shall
consist of no more than seven (7) members. For the purposes of this Agreement,
"Stock" shall mean and include all Series A Preferred Stock and all shares of
Common Stock, and all other securities of the Company which may be exchangeable
for or issued in exchange for or in respect of shares of Common Stock (whether
by way of stock split, stock dividend, combination, reclassification,
reorganization or any other means).

                     In the absence of any designation from the persons or 
groups so designating directors as specified above, the director previously
designated by them and then serving shall be reelected if still eligible to
serve as provided herein.

                     No party hereto shall vote to remove any member of the
Board of Directors designated in accordance with the aforesaid procedure unless
the persons or groups so designating directors as specified above so vote, and,
if such persons or groups so vote then the non-designating party or parties
shall likewise so vote.

                     Any vacancy on the Board of Directors created by the 
resignation, removal, incapacity or death of any person designated under this
Section 1 shall be filled by another person designated in a manner so as to
preserve the constituency of the Board as provided above.

                     If any party to this Agreement shall fail to vote such
party's Stock as provided in this Agreement, without further action by such
party, the President of the Company shall be, and hereby is, irrevocably
constituted the attorney-in-fact and proxy of such party for the purpose of
voting the shares of such Stock and shall vote the same in accordance with the
terms of this Agreement and is hereby authorized to revoke any proxy providing
for any other vote of such shares with respect to the election of directors.

           2.        Termination. This Agreement, and the respective rights and
                     -----------
obligations of the parties hereto, shall terminate upon the earliest to occur of
the following: (i) the expiration of ten years from the date first written
above; (ii) a Mandatory Conversion pursuant to the terms of Paragraph 6O of the
Company's Certificate of Designations; or (iii) the sale of the Company, whether
by merger, sale, or transfer of more than eighty percent (80%) of its capital
stock, or sale of substantially all of its assets.

           3.        Notices. All notices, requests, demands and other 
                     -------
communications provided for hereunder shall be in writing (including electronic
communication) and delivered personally, or by overnight courier, or by
facsimile or other electronic means or sent by certified or 
<PAGE>
 
                                      -3-

registered United States mail, postage prepaid, return receipt requested and
addressed as follows:

           If to any Investor: at such Investor's address for notice as set
forth in the register maintained by the Company, or, as to each of the
foregoing, at the addresses set forth on Schedule I hereto or at such other
                                         ----------
address as shall be designated by such Person in a written notice to the other
parties complying as to delivery with the terms of this Section, with a copy to
Leslie E. Davis, Esq., Testa, Hurwitz & Thibeault, LLP, High Street Tower, 125
High Street, Boston, Massachusetts 02110.

           If to the Company: at 175 Derby Street, Suite 36, Hingham,
Massachusetts 02043, or at such other address as shall be designated by the
Company in a written notice to the other parties complying as to delivery with
the terms of this Section, with a copy to Donna L. Brooks, Esq., Shipman &
Goodwin LLP, One American Row, Hartford, CT 06103.

           All such notices, requests, demands and other communications shall be
effective three days after deposited in the mails or upon receipt when delivered
electronically, by facsimile, by hand or by overnight courier, respectively,
addressed as aforesaid, unless otherwise provided herein.

           4.        Specific Performance. The rights of the parties under this
                     --------------------
Agreement are unique and, accordingly, the parties shall, in addition to such
other remedies as may be available to any of them at law or in equity, have the
right to enforce their rights hereunder by actions for specific performance to
the extent permitted by law.

           5. Entire Agreement. This Agreement constitutes the entire agreement
              ----------------
among the parties with respect to the subject matter hereof and supersedes all
prior agreements and understandings between them or any of them as to such
subject matter, including, without limitation, that certain Voting Agreement of
the Company dated as of June 6, 1996.

           6. Waivers and Further Agreements. Any of the provisions of this
              ------------------------------
Agreement may be waived with the consent of the Investors holding at a majority
in interest of the issued and outstanding shares of Series A Preferred Stock
(including shares of Common Stock into which any such shares may have been
converted) then held or deemed to be held by all Investors by an instrument in
writing. Any waiver by any party of a breach of any provision of this Agreement
shall not operate or be construed as a waiver of any subsequent breach of that
provision or of any other provision hereof. Each of the parties hereto agrees to
execute all such further instruments and documents and to take all such further
action as any other party may reasonably require in order to effectuate the
terms and purposes of this Agreement. Notwithstanding the foregoing, no waiver
approved in accordance herewith shall be effective if and to the extent that
such waiver grants to any one or more Investors any rights more favorable than
any rights granted to all other Investors or otherwise treats any one or more
Investors differently than all other Investors.
<PAGE>
 
                                      -4-

           7.        Amendments. Except as otherwise expressly provided herein, 
                     ----------
this Agreement may not be amended except by an instrument in writing executed by
(i) the Company, (ii) Investors holding a majority in interest of the issued and
outstanding shares of Series A Preferred Stock (including shares of Common Stock
into which any such shares may have been converted), and (iii) Holders holding a
majority of the shares of Common Stock subject to this Agreement.
Notwithstanding the foregoing, no such amendment shall be effective if and to
the extent that such amendment creates any additional affirmative obligations to
be complied with by any or all of the Investors.

           8.        Assignment; Successors and Assigns.  This Agreement shall 
                     ----------------------------------
be binding upon and shall inure to the benefit of the parties hereto and their
respective heirs, executors, legal representatives, successors and permitted
transferees.

           9.        Severability. In case any one or more of the provisions 
                     ------------
contained in this Agreement shall for any reason be held to be invalid, illegal
or unenforceable in any respect, such invalidity, illegality or unenforceability
shall not affect any other provision of this Agreement, and such invalid,
illegal and unenforceable provision shall be reformed and construed so that it
will be valid, legal, and enforceable to the maximum extent permitted by law.

           10.       Counterparts.  This Agreement may be executed in two or 
                     ------------
more counterparts, each of which shall be deemed an original, but all of which
together shall constitute one and the same instrument.

           11.       Section Headings.  The headings contained in this 
                     ----------------
Agreement are for reference purposes only and shall not in any way affect the
meaning or interpretation of this Agreement.

           12. Governing Law. This Agreement shall be construed and enforced in
               -------------
accordance with and governed by the General Corporation Law of the State of
Delaware as to matters within the scope thereof, and as to all other matters
shall be governed by and construed in accordance with the internal laws of the
Commonwealth of Massachusetts.

           13.       Legend.  The certificates representing the shares of 
                     ------
Series A Preferred Stock shall bear a legend substantially in the following
form:

           "The shares represented by this certificate are subject to the terms
and conditions of a Stockholders' Agreement dated as of November 6, 1996, a copy
of which will be furnished to any interested party upon written request without
charge."

                 [REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]
<PAGE>
 
                                      -5-

           IN WITNESS WHEREOF, the parties hereto have executed this
Stockholders' Agreement as a sealed instrument as of the day and date first
above written.

INVESTORS:                         THE COMPANY:

CAHILL, WARNOCK STRATEGIC          OCCUPATIONAL HEALTH +
PARTNERS FUND, L.P.                  REHABILITATION INC

By:  Cahill, Warnock Strategic
        Partners, L.P.             By:
                                      ___________________________________

By:
   ------------------------------
Title: General Partner
      ---------------------------
                                   HOLDERS:

STRATEGIC ASSOCIATES, L.P.         PRINCE VENTURE PARTNERS III
                                     LIMITED PARTNERSHIP
By:  Cahill, Warnock & Company, 
LLC
                                   By: Prince Ventures, L.P., General Partner
By:
   ------------------------------
Title: Managing Member
      ---------------------------
                                   By:
                                      ___________________________________

AXA U.S. GROWTH FUND, LLC          *THE VENTURE CAPITAL FUND OF
                                      NEW ENGLAND III, L.P.

By:                                By:  FH & Co. III, L.P., Its General Partner
   ------------------------------
Title: Managing Member
      ---------------------------
                                   By:
                                      ___________________________________

U.S. GROWTH FUND PARTNERS,
  C.V.
                                   *BANCBOSTON VENTURES, INC.

By:
   ------------------------------
Title: General Partner
      ---------------------------
                                   By:
                                      ___________________________________

DOUBLE BLACK DIAMOND II, LLC
LLC
                                   --------------------------------------
                                   John C. Garbarino
<PAGE>
 
                                      -6-

By:
   ------------------------------
Title:     Managing Member
      ---------------------------

                                   --------------------------------------
                                   Lynne M. Rosen
ALMANORI LIMITED

                                   *VENROCK ASSOCIATES

By:
   ------------------------------
Title:     Attorney-in-Fact
      ---------------------------
                                   By:
                                      ___________________________________


                                   *VENROCK ASSOCIATES II, L.P.


                                   By:
                                      ___________________________________


                                   *ASSET MANAGEMENT ASSOCIATES,
                                      1989, L.P.

                                   By:  AMC Partners 89, L.P., General 
                                   Partner


                                   By:
                                      ___________________________________

* In their capacities as Holders and Investors hereunder.
<PAGE>
 
                                      -7-

                                                                      SCHEDULE I

                   OCCUPATIONAL HEALTH + REHABILITATION INC

                                    HOLDERS
                                    -------

Prince Venture Partners III Limited Partnership 
The Venture Capital Fund of New England III, L.P.
BancBoston Ventures, Inc.
John C. Garbarino
Lynne M. Rosen
Venrock Associates
Venrock Associates II, L.P.
Asset Management Associates, 1989, L.P.

                                   INVESTORS
                                   ---------

Cahill, Warnock Strategic Partners Fund, L.P.
10 North Calvert Street
Suite 735
Baltimore, Maryland 21202
Attn:  Mr. Edward L. Cahill

Strategic Associates, L.P.
10 North Calvert Street
Suite 735
Baltimore, Maryland 21202
Attn:  Mr. Edward L. Cahill

Axa U.S. Growth Fund, LLC
c/o Partech International
50 California Street
Suite 3200
San Francisco, CA 94111
Attn:  Mr. Thomas G. McKinley

U.S. Growth Fund Partners, C.V.
c/o Partech International
50 California Street
Suite 3200
San Francisco, CA 94111
Attn:  Mr. Thomas G. McKinley
<PAGE>
 
                                      -8-

Double Black Diamond II, LLC
c/o Partech International
50 California Street
Suite 3200
San Francisco, CA 94111
Attn:  Mr. Thomas G. McKinley

Almanori Limited
c/o Partech International
50 California Street
Suite 3200
San Francisco, CA 94111
Attn:  Mr. Thomas G. McKinley

Asset Management Associates, 1989, L.P.
2275 East Bayshore Road
Palo Alto, CA  94303
Attn:  Mr. Craig C. Taylor

Venrock Associates
Room 5508
30 Rockefeller Plaza
New York, New York 10112
Attn:  Mr. Patrick F. Latterell

Venrock Associates II, L.P.
Room 5508
30 Rockefeller Plaza
New York, New York 10112
Attn:  Mr. Patrick F. Latterell

The Venture Capital Fund of New England, III, 
L.P.
160 Federal Street, 23rd Floor
Boston, MA  02110
Attn:  Mr. Kevin J. Dougherty

BancBoston Ventures, Inc.
100 Federal Street
Boston, MA  02110
Attn:  Ms. Marcia T. Bates
<PAGE>
 
                                      -9-

                                                                     SCHEDULE II

                   OCCUPATIONAL HEALTH + REHABILITATION INC

                                             
                          OH+R Principal Stockholders
                          ---------------------------

Prince Venture Partners III Limited Partnership 
The Venture Capital Fund of New England III, L.P.
BancBoston Ventures, Inc.


                         Telor Principal Stockholders
                         ----------------------------

Prince Venture Partners III, Limited Partnership
Venrock Associates
Venrock Associates II, L.P.
Asset Management Associates, 1989, L.P.