<PAGE>   1
================================================================================


                           JEF HOLDING COMPANY, INC.

                             JEFFERIES GROUP, INC.

                                      AND

                             THE BANK OF NEW YORK,

                                    TRUSTEE

                              -------------------

                         SECOND SUPPLEMENTAL INDENTURE

                           DATED AS OF MARCH 26, 1999

                              -------------------


                                  $50,000,000

                          8 7/8% SENIOR NOTES DUE 2004

                              -------------------

              SUPPLEMENTING THE INDENTURE OF JEFFERIES GROUP, INC.
          DATED AS OF APRIL 28, 1994, AND AS AMENDED ON JULY 14, 1994

================================================================================
<PAGE>   2
     SECOND SUPPLEMENTAL INDENTURE, dated as of March 26, 1999, among Jefferies
Group, Inc., a Delaware Corporation ("Group"), JEF Holding Company, Inc., a
Delaware corporation ("New JEF"), and The Bank of New York, a New York banking
corporation, as trustee (the "Trustee").

     WHEREAS, Group and the Trustee executed and delivered that certain
indenture, dated as of April 28, 1994 and as amended on July 14, 1994 by a
First Supplemental Indenture (collectively, the "Indenture"), providing for the
issuance thereunder by Group, and the authentication and delivery by the
Trustee, of Group's 8 7/8% Senior Notes due 2004 (the "Securities"). Any
capitalized terms used herein and not otherwise defined shall have the meanings
given thereto in the Indenture.

     WHEREAS, as of the date hereof, New JEF is a wholly owned subsidiary of
Group.

     WHEREAS, as of March 17, 1999, the Board of Directors of each of Group and
New JEF authorized and approved the execution and delivery of a Distribution
Agreement dated March 17, 1999 between Group and New JEF (the "Distribution
Agreement") and the transactions contemplated thereby;

     WHEREAS, Group has agreed, pursuant to and subject to the conditions set
forth in the Distribution Agreement, to transfer its assets (excluding the
capital stock and assets of Investment Technology Group, Inc., a Delaware
corporation and approximately 80.5% subsidiary of Group as of the date hereof
("ITGI") and ITGI's subsidiaries) and liabilities (excluding the liabilities of
or related to ITGI and ITGI's subsidiaries) to New JEF and a subsidiary of
Group that will become a subsidiary of New JEF in connection with such
transfers (collectively, the "Transfers");

     WHEREAS, it is anticipated that the Transfers will be made on or about
April 21, 1999, except for the limited transfers that are the subject of, and
will be effected thereafter in accordance with, Section 3.01(e) of the
Distribution Agreement;

     WHEREAS, the assets of Group that are the subject of the Transfers
constitute approximately 98% of the consolidated assets of Group, as set forth
in Group's consolidated balance sheet as of December 31, 1998;

     WHEREAS, following completion of the Transfers, Group, in accordance with
the provisions and conditions of the Distribution Agreement, will effect a pro
rata distribution of 100% of the outstanding New JEF common stock owned by
Group to the holders of outstanding common stock of Group (the "Distribution");

     WHEREAS, the Distribution Agreement provides that before the Distribution,
effective as of the date of the Transfers, New JEF shall execute the
Supplemental Indenture;

                                     - 1 -
<PAGE>   3
     WHEREAS, New JEF, Group and the Trustee, by appropriate corporate action,
have determined to supplement the Indenture in the manner described below and
all acts and proceedings required by law, by the Indenture, and by the
Certificate of Incorporation and the Bylaws of New JEF and Group necessary to
authorize and constitute this Second Supplemental Indenture a valid and binding
agreement in accordance with the terms hereof, have been done and taken.

     NOW, THEREFORE, in consideration of the foregoing, each of New JEF and
Group covenants and agrees with the Trustee, for the equal and proportionate
benefit of the respective holders from time to time of the Securities, as
follows:

     1.   Assumption. New JEF hereby agrees that, effective immediately upon
completion of the Transfers (excluding any transfers that may be the subject of
Section 3.01(e) of the Distribution Agreement), New JEF thereby shall succeed
to, be substituted for, and exercise every right and power of Group under the
Indenture, as amended hereby, and shall assume all of the obligations of Group
pursuant hereto and pursuant to the Securities. Group shall be relieved of the
performance and observance of all obligations and covenants of the Indenture,
as amended hereby, and the Securities, including but not limited to the
obligation to make payment of the principal of and interest, if any, on all the
Securities.

     2.   Full Force and Effect; Operative Effect of Amendment and First
Supplemental Indenture. The Indenture, as amended by this Second Supplemental
Indenture, shall be in full force and effect as of the date hereof.

     3.   Governing Law. This Second Supplemental Indenture shall be governed
by and construed in accordance with the laws of the State of New York.

     4.   Duplicate Originals. This Second Supplemental Indenture may be
executed in any number of counterparts and by the parties hereto in separate
counterparts, each of which when so executed shall be deemed to be an original
and all of which taken together shall constitute one and the same agreement.

                  [REMAINDER OF PAGE INTENTIONALLY LEFT BLANK]



                                      -2-
<PAGE>   4
      IN WITNESS WHEREOF, the parties hereto have caused this Second
Supplemental Indenture to be duly executed as of the date first written above.



                               JEF HOLDING COMPANY, INC.



                               By: /s/ JERRY M. GLUCK
                                  --------------------------------
                                  Name: Jerry M. Gluck
                                  Title: General Counsel and Secretary



                               JEFFERIES GROUP, INC.



                               By: /s/ FRANK E. BAXTER
                                  --------------------------------
                                   Name: Frank E. Baxter
                                   Title: Chairman and Chief Executive Officer




                               THE BANK OF NEW YORK,
                               as Trustee



                               By: /s/ MARY BETH LEWICKI
                                  --------------------------------
                                   Name: Mary Beth Lewicki
                                   Title: Assistant Vice President




                                      -3-