<PAGE>

                                                                  EXHIBIT 10.10B

                                                               115 Flanders Road


               FIFTH LEASE EXTENSION AND MODIFICATION AGREEMENT
               ------------------------------------------------


     THIS FIFTH LEASE EXTENSION AND MODIFICATION AGREEMENT is made as of October
15, 1997, by and between AETNA REAL ESTATE ASSOCIATES, a Delaware limited
partnership having a mailing address in care of Allegis Realty Investors, LLC.,
242 Trumbull Street, Hartford, CT 06103, Attention: William P. Nikolis
("Landlord"), and BANYAN SYSTEMS, INCORPORATED, a Massachusetts corporation
having its principal place of business at 120 Flanders Road, Westborough, MA
01581-1033 ("Tenant").


                             W I T N E S S E T H:

     WHEREAS, pursuant to certain Lease Agreement (the "Lease Agreement") dated
November 14, 1986 by and between Arthur DiMartino, Jr., Trustee of Flanders
Realty Trust, as landlord ("Flanders"), and Tenant, as tenant, Flanders leased
to Tenant and Tenant hired from Flanders, certain space totalling 64,654
rentable square feet, consisting of all of the rentable area in a building
situated at 115 Flanders Road, Westborough, MA (collectively, the "Premises"),
all as more particularly described in the Lease Agreement;

     WHEREAS, said Lease Agreement has been amended as set forth in (i) a Lease
Addendum dated August 28, 1987, (ii) a Lease Addendum dated January 5, 1988,
(iii) a certain undated Lease Extension and Modification Agreement effective as
of January 1, 1992, and (iv) a certain Lease Extension and Modification
Agreement dated April 15, 1993 (said Lease Agreement, as amended, is referred to
as the "Lease");

     WHEREAS, pursuant to a certain Assignment of Leases dated February 5, 1988,
Flanders assigned its right, title and interest under the Lease to Landlord;

     WHEREAS, the term of the Lease expires on December 31, 1997; and

     WHEREAS, Landlord and Tenant desire to extend the term of the Lease and to
further modify the Lease on the terms and conditions set forth herein.

     NOW, THEREFORE, in consideration of the sum of TEN DOLLARS ($10.00), the
receipt and sufficiency of which are mutually acknowledged, Landlord and Tenant
agree as follows:

     1.   The term of the Lease is hereby extended for sixty (60) months (the
"Extension Term"), from and including January 1, 1998 through and including
December 31, 2002.

     2.   Effective as of January 1, 1998, the Base Rent for the Premises for
the first calendar year of the Extension Term, shall be increased to $484,905.00
per annum payable in twelve (12)
<PAGE>

equal monthly installments of $40,408.75 on the first day of each and every
month; for the second calendar year of the Extension Term, the Base Rent for the
Premises shall be $517,232 per annum payable in twelve (12) equal monthly
installments of $43,102.67 on the first day of each month; for the third
calendar year of the Extension Term, the Base Rent for the Premises shall be
$614,213 per annum payable in twelve (12) equal monthly installments of
$51,184.42 on the first day of each month; for the fourth year of the Extension
Term, the Base Rent for the Premises shall be $711,194 per annum payable in
twelve (12) equal monthly installments of $59,266.17 on the first day of each
month; for the fifth calendar year of the Extension Term, the Base Rent for the
Premises shall be $775,848 per annum payable in twelve (12) equal monthly
installments of $64,654 on the first day of each month. Tenant shall be
responsible for one hundred percent (100%) of all operating, tax, maintenance
and repair expenses as stated in the Lease, as of January 1, 1998, Tenant's
"proportionate share", as such term is used in the Lease, shall be deemed to be
100%.

     3.   Provided that, at the time of such exercise, this Lease is still in
full force and effect without default by Tenant beyond applicable grace periods
and Tenant occupies at least seventy-five percent (75%) of the Premises for its
own business purposes, Tenant shall have the right and option (the "Extension
Option") to extend the Term of this Lease for one (1) extended term of three (3)
years (an "Extended Term").  The Extended Term shall commence on January 1, 2003
and shall end on December 31, 2005.  Tenant shall exercise its Extension Option
for the Extended Term by giving written notice to Landlord of its desire to do
so not later than March 31, 2002.  The giving of such notice by Tenant shall
automatically extend the Term of this Lease for the applicable Extended Term
(except as hereinafter set forth), and no instrument of renewal need be
executed.  In the event that Tenant fails to give such notice to Landlord this
Lease shall automatically terminate at the end of the then current Term and
Tenant shall have no further option to extend the Term of this Lease.  Landlord
and Tenant agree that they have specifically bargained for such option right and
notice provision, that Landlord requires certainty in order to conduct its
business, that any attempt to exercise said option to renew after March 31,
2002, will operate to the detriment of the Landlord and that termination of this
renewal option after such date shall not operate to the detriment of the Tenant.
Accordingly, both parties agree that time is of the essence hereof and that
Tenant shall not be allowed under any circumstances to exercise it renewal
rights after such date even if such failure was due to honest mistake or
otherwise.  The Extended Terms shall be on all the terms and conditions of this
Lease, except (i) during the Extended Term, the extension provisions of this
Section shall not be effective, (ii) during the Extended Term, Base Rent shall
equal the prevalent fair market rent for the space determined at the time of
giving of Tenant's renewal notice.  As used in the immediately preceding
sentence, fair market rent means the rent then prevalent for new rentals of
similar space in similar quality buildings in the Westborough/Route 495 area.
Fair Market Rent will be determined as follows:

     (a)  Landlord and Tenant will have thirty (30) days after Landlord receives
          the option notice from Tenant within which to agree on the monthly
          rent for the Extended Term based upon the fair market rental value of
          the Premises. If they agree on the monthly rent within thirty (30)
          days, they will amend this Lease by stating the monthly rent.

                                       2
<PAGE>

     (b)  If they are unable to agree on the monthly rent for the Extended Term
          within thirty (30) days, the Appraisal Process set forth in Subsection
          (c) below shall be utilized for the determination of fair market rent.

     (c)  (c) Within twenty (20) days after the expiration of the thirty (30)
          day period set forth in Subsection (a) above, Landlord and Tenant will
          each appoint a real estate appraiser with at least five (5) years
          full-time commercial appraisal experience in the area in which the
          Premises are located to appraise the fair market rental value of the
          Premises.  If either party does not appoint an appraiser within such
          twenty (20) day period and notify the other party in writing of this
          selection, the single appraiser appointed will be the sole appraiser
          and will set the fair market rental value of the Premises.  If two (2)
          appraisers are appointed pursuant to this Subsection (c), they will
          meet promptly and attempt to set the then fair market rental value of
          the Premises.  If they are unable to agree within thirty (30) days
          after the second appraiser has been appointed, they will attempt to
          elect a third appraiser meeting the qualifications stated in this
          Subsection (c) within ten (10) days after the end of the aforesaid
          thirty-day period.  If they are unable to agree on the third
          appraiser, either the Landlord or the Tenant, by giving ten (10) days
          prior notice to the other, may apply to the then presiding judge of
          the local circuit or district court having primary jurisdiction for
          civil matters in the area in which the Premises are located for the
          selection of a third appraiser who meets the qualifications stated in
          this Subsection (c).  Landlord and Tenant will each bear one-half (
          1/2) of the cost of appointing the third appraiser and of the third
          appraiser's fee as well as the cost of their respective appointed
          appraisers.  The third appraiser, however selected, must be a person
          who has not previously acted in any capacity for either Landlord or
          Tenant.  Within thirty (30) days after the selection of the third
          appraiser, a majority of the appraisers will set the then fair market
          rental value of the Premises.  If a majority of the appraisers are
          unable to set the then fair market rental value of the Premises within
          thirty (30) days after selection of the third appraiser, the three (3)
          appraisals will be averaged and the average will be the then fair
          market rental value of the Premises.

     Upon determination of the Base Rent for the Extended Term pursuant to the
foregoing, the parties agree to execute an amendment to the Lease acknowledging
the same.  The foregoing Extension Option is granted in lieu of any other
Extension Options, rights of first refusal, or rights of first offer  in the
Lease, which options and rights are hereby deleted in their entirety.

     4.   Tenant acknowledges and agrees that Landlord has completed all tenant
improvements required under the Lease and Tenant hereby accepts the Premises
being leased pursuant to this Fifth Lease Extension and Modification Agreement
on an "AS IS" basis.  Landlord shall not be obligated to construct any tenant
improvements in connection with this Lease.

                                       3
<PAGE>

     5.   Tenant's Proportionate Share as used in the Lease and defined in
Section 4B thereof, shall be 100%.  Further, Sections 5C and D of the Lease are
modified by deleting the term "54%" and substituting therefor "100%". Tenant
further acknowledges that Tenant, and not Landlord, shall be responsible for the
costs of all maintenance and repairs at the Premises, including costs associated
with snow removal, the lobby, parking areas, sidewalks, driveways, landscaping,
HVAC, and other common areas at the Premises, this Lease being on a completely
net basis except as otherwise set forth herein.  Notwithstanding the foregoing,
Landlord shall be responsible for undertaking all building structural repairs
and replacements, including the roof, the exterior walls, the floor slab and the
outside septic system.  Tenant shall be responsible for the reimbursement to
Landlord for the costs of all maintenance and repairs.  In no event, however,
shall the costs to be shared by Tenant hereunder include (1) any capital
expenditure, as determined by generally accepted accounting principles (except
only such as may be made for the purpose and with the effect of achieving a
corresponding reduction of costs which would otherwise be shared by Tenant,
provided however that any such capital expenditure shall be amortized for the
useful life thereof according to generally accepted accounting principles and
such amortized amount shall be payable by Tenant) and (2) any cost for which
Landlord is otherwise entitled to reimbursement from a third party but only when
and as Landlord receive the same; Landlord agrees to use reasonable efforts to
pursue the same, the reasonable cost of such pursuit to be a Tenant expense
hereunder.  Tenant shall be entitled to inspect all books and records of
Landlord reasonably related to the payment of any costs which is to be shared by
Tenant pursuant to the provisions of this Lease.

     6.   Section 9 of the Lease is amended by adding the following sentence:
Tenant shall be responsible for 100% of costs associated with the provision and
usage of all utilities at the Premises including, without limitation, all
electricity, gas, water, sewage, telephone, and telecommunications services.

     7.   Section 10 of the Lease is hereby deleted in its entirety and the
following substituted in lieu thereof:

     10.  ASSIGNMENT AND SUBLETTING.
          -------------------------

          10.1  Landlord's Consent.  Tenant shall not assign this Lease, or
                ------------------
     sublease all or any part of the Premises, or permit the use of the Premises
     by any party other than Tenant, without the prior written consent of
     Landlord which consent will not be unreasonably withheld, delayed or
     conditioned.  When Tenant requests Landlord's consent to such assignment or
     sublease, it shall notify Landlord in writing of the name and address of
     the proposed assignee or subtenant and the nature and character of the
     business of the proposed assignee or subtenant and shall provide financial
     information including financial statements of the proposed assignee or
     subtenant.  Landlord shall not be obligated to consent to or permit any
     proposed sublet or assignment if Tenant is in default under this Lease at
     the time it makes its request for such consent or at the time the
     assignment or sublet is to take effect.

                                       4
<PAGE>

          10.2  Approved Subleases and Assignments. No consent to any assignment
                ----------------------------------
     or sublease shall constitute a further waiver of the provisions of this
     section, and all subsequent assignments or subleases may be made only with
     the prior written consent of Landlord. An assignee of Tenant, at the option
     of Landlord, shall become directly liable to Landlord for all obligations
     of Tenant hereunder, but no sublease or assignment by Tenant shall relieve
     Tenant of any liability hereunder. Any assignment or sublease without
     Landlord's consent shall be void, and shall, at the option of the Landlord,
     constitute a default under this Lease. In the event that Landlord shall
     consent to a sublease or assignment hereunder, Tenant shall pay Landlord's
     reasonable fees per transaction, incurred in connection with the processing
     of documents necessary to the giving of such consent.

          10.4  Except as provided below, the provisions of paragraph 10.1 of
     this Section shall apply to (i) a single transfer or an orchestrated series
     of transfers of a majority of the stock or partnership interests or other
     evidences of ownership of or interest in Tenant as if such transfer were an
     assignment of this Lease and (ii) to any transfer of such stock or
     interests the effect of which would be to constitute an assignment of this
     Lease.  Notwithstanding the foregoing, the provisions of Section 10.1 shall
     not apply to any transactions by Tenant with an entity into or with which
     Tenant is merged or consolidated or with an entity to which substantially
     all of Tenant's assets are transferred or with any entity which controls or
     is controlled by Tenant or is under common control with Tenant or with
     which Tenant, directly or indirectly, has a fifty percent (50%) or greater
     ownership interest, or which directly or indirectly, has a fifty percent
     (50%) or greater interest in Tenant, provided that in any of such events
     (x) Landlord is provided with evidence satisfactory to it that the
     successor to Tenant has a net worth computed in accordance with generally
     accepted accounting principles at least equal to the net worth of Tenant
     upon execution of this Lease, and (y) in the case of any assignment of this
     Lease, the assignee agrees directly with Landlord, by written instrument in
     form reasonably satisfactory to Landlord, to be bound by all the
     obligations of Tenant hereunder including, without limitation, the covenant
     against further assignment and subletting except in accordance with the
     express provisions of this Lease.

     8.   The provisions of Section 12(iv) of the Lease are modified by changing
the amount of liability insurance required from not less than $1,000,000 per
occurrence, to not less than $3,000,000 per occurrence.

     9.   Section 22(c) of the Lease is hereby deleted in its entirety and the
following substituted therefor:

     Any notice, demand, consent, approval, disapproval, or statement
     (collectively, "Notices") from Landlord to Tenant or from Tenant to
     Landlord shall be in writing and shall be deemed duly given (a) if mailed
     by registered or certified mail, postage prepaid, return receipt requested,
     (b) if delivered by recognized national overnight delivery service with
     receipt acknowledged, or (c) only in the case of Notices that are
     Escalation Statements or bills for

                                       5
<PAGE>

     rent, if mailed by first-class mail, postage prepaid, to the address(es)
     for Notices set forth in this Section 22. Notices to Tenant shall be sent
     to Tenant at 120 Flanders Road, Westborough, MA 01581-1033. Notices to
     Landlord shall be sent (i) to the address of Landlord set forth on page 1
     of this Lease Amendment or (ii) to such other address as Landlord shall
     have last designated by notice in writing to Tenant, with copies to
     Trammell Crow Company, 25 First Street, Cambridge, MA 02141, Attention:
     Larry Glazer and to James D. Sperling, Esq., Kassler & Feuer, P.C., 101
     Arch Street, 18th Floor, Boston, MA 02110. Notice shall be deemed given on
     the third business day after depositing same in an office depository of the
     United States Postal Service (or successor organization) or, if given by
     overnight delivery, upon delivery to Landlord or Tenant, as the case may
     be.

     10.  The amount of Security Deposit set forth in Section 2B of the Lease
shall be changed to One Hundred Three Thousand Four Hundred Forty Six and 40/100
Dollars ($103,446.40), which sum shall be due and payable upon Tenant's
execution of this Amendment.  Landlord acknowledges that the current Security
Deposit of $26,802.86 will be credited so that the additional amount of the
Security Deposit to be paid on the date hereof shall be $76,643.54.

     11.  The Lease, as hereinbefore extended and modified, is hereby ratified
and confirmed.

                                      LANDLORD:

                                      AETNA REAL ESTATE ASSOCIATES,
                                      a Delaware limited partnership

Witness:                              By:  Aetna/AREA Corporation
                                           Its General Partner

/s/ Illegible                         By: /s/ Illegible
------------------------------           -----------------------------------
                                         Its
                                         Hereunto duly authorized

                                      TENANT:

                                      BANYAN SYSTEMS INCORPORATED

                                      By:   /s/ Richard M. Spaulding
                                         -----------------------------------
                                      Name:     Richard M. Spaulding
                                           -------------------------
                                      Title:    Vice President and CFO
                                            --------------------------

                                       6
<PAGE>

                              CLERK'S CERTIFICATE
                              -------------------


     I, John P. Mitchell                         , do hereby certify that I am
        -----------------------------------------
the Assistant Clerk of BANYAN SYSTEMS CORPORATION, a Delaware corporation (the
"Corporation") with its principal place of business at 120 Flanders Road,
Westborough, MA 01581-1033, and that I have been duly elected and am presently
serving in that capacity in accordance with the Bylaws of the Corporation.

     I further certify that by unanimous written consent of the directors of the
Corporation, votes in the form of Exhibit A attached hereto were duly adopted by
                                  ---------
all present at a meeting of said directors duly called and as to which notice
was duly given as required by law and the By-laws of the Corporation. The votes
as specified are presently in full force and effect and have not been revoked or
rescinded as of the date hereof.

     I further certify that as of this date the following is the current duly
elected and acting officer of the Corporation who is authorized pursuant to the
attached votes.


              Name                       Title
              ----                       -----


Richard M. Spaulding            Vice President and Chief Financial Officer
--------------------------------------------------------------------------



     IN WITNESS WHEREOF, I have hereunto set my hand and the seal of the
Corporation this   15th   day of October, 1997.
                 --------


                                         /s/ John P. Mitchell
                                         ---------------------------------
                                         John P. Mitchell, Assistant Clerk



[CORPORATE SEAL]