<PAGE>

                                                                     EXHIBIT 4.5

                                AMENDMENT NO. 3
                                      TO
                          SECOND AMENDED AND RESTATED
                           INVESTOR RIGHTS AGREEMENT
                           -------------------------

     This Amendment dated as of September 20, 1999 is entered into by and among
Sycamore Networks, Inc., a Delaware corporation (the "Company"), and the persons
and entities listed on Schedule I hereto.

     WHEREAS, the Company has entered into a Second Amended and Restated
Investor Rights Agreement dated as of February 26, 1999 and amended on July 23,
1999 and August 5, 1999 (the "Agreement");

     WHEREAS, the Company and the requisite parties necessary to effect an
amendment to the Agreement pursuant to Section 7(b) thereof desire that the
Agreement be amended in the manner set forth below; and

     WHEREAS, the parties hereto desire to amend the Agreement pursuant to this
Amendment No. 3.

     NOW THEREFORE, in consideration of the mutual covenants contained herein
and for other valuable consideration, the receipt and sufficiency of which is
hereby acknowledged, the parties hereto agree as follows:

1.   The last sentence of Section 2 of Article V of the Agreement is hereby
deleted in its entirety and replaced with the following:

     "Notwithstanding the foregoing, any party hereto may transfer (i) any or
     all of his Shares to any Family Member (as defined below), (ii) any and all
     of his Shares under his will, (iii) any and all of his Shares to a limited
     partnership, limited liability company, trust or other similar entity
     established for the benefit of any Family Member or (iv) up to 10,000 (ten
     thousand)Shares individually or to any third party as a gift.  "Family
     Member" shall mean any child, stepchild, grandchild, parent, stepparent,
     grandparent, spouse, former spouse, sibling, niece, nephew, mother-in-law,
     father-in-law, son-in-law, daughter-in-law, brother-in-law or sister-in-
     law, including adoptive relationships."

     2.   The Agreement, as supplemented and modified by this Amendment,
together with the other writings referred to in the Agreement or delivered
pursuant thereto which form a part thereof, contain the entire agreement among
the parties with respect to the subject matter thereof and amend, restate and
supersede all prior and contemporaneous arrangements or understandings with
respect thereto.

     3.   Upon the effectiveness of this Amendment, on and after the date
hereof, each reference in the Agreement to "this Agreement," "hereunder,"
"hereof," "herein" or words of like import, and each reference in the other
documents entered into in connection with the
<PAGE>

Agreement, shall mean and be a reference to the Agreement, as amended hereby.
Except as specifically amended above, the Agreement shall remain in full force
and effect and is hereby ratified and confirmed.

     4.   This Amendment shall be governed by the laws of the State of Delaware,
notwithstanding the conflict-of-law doctrines of Delaware or any other
jurisdiction to the contrary.

     5.   This Amendment may be executed in any number of counterparts, and each
such counterpart shall be deemed to be an original instrument, but all such
counterparts together shall constitute but one agreement.

     6.   This Amendment shall be binding on all parties to the Agreement as and
when executed by the Company, Founding Stockholders holding at least a majority
by voting power of the shares of capital stock held by the Founding Stockholders
and Investors holding at least a majority of the shares of Common Stock issued
or issuable upon conversion of the shares of the Series A Preferred Stock and/or
the Series B Preferred Stock and/or the Series C Preferred Stock and/or the
Series D Preferred Stock.



               [Remainder of this Page Intentionally Left Blank]

                                       2
<PAGE>

     IN WITNESS WHEREOF the parties hereto have executed this Amendment on the
date first above written.

                                   COMPANY:

                                   SYCAMORE NETWORKS, INC.

                                   By: /s/ Daniel Smith
                                       ---------------------
                                       Daniel Smith
                                       President

                                   Matrix Partners V, L.P.
                                   Bay Colony Corporate Center
                                   1000 Winter Street, Suite 4500
                                   Waltham, Massachusetts 02154

                                   By:   Matrix V Management Co., L.C.C.,
                                         its General Partner

                                   By: /s/ Paul J. Ferri
                                       ---------------------

                                   Matrix V Entrepreneurs Fund, L.P.

                                   Bay Colony Corporate Center
                                   1000 Winter Street, Suite 4500
                                   Waltham, Massachusetts 02154

                                   By:   Matrix V Management Co., L.C.C.,
                                         its General Partner

                                   By: /s/ Paul J. Ferri
                                       ---------------------

                                   /s/ Gururaj Deshpande
                                   -------------------------
                                   Gururaj Deshpande

                                   /s/ Daniel Smith
                                   -------------------------
                                   Daniel Smith

                                       3
<PAGE>

                                   FOUNDERS:

                                   /s/ Gururaj Deshpande
                                   ---------------------
                                   Gururaj Deshpande

                                   Address:  9 Sparta Way
                                             Andover, MA 01810

                                   /s/ Richard Barry
                                   -----------------
                                   Richard Barry

                                   Address:  1284 Beacon Street, # 815
                                             Brookline, MA 02138

                                       4
<PAGE>

                                   Schedule 1
                                   ----------

                                   Investors
                                   ---------

Matrix Partners V, L.P.
Matrix V Entrepreneurs Fund, L.P.
Gururaj Deshpande
Daniel Smith