<PAGE>

                                                                   EXHIBIT 10.17

                          BANYAN SYSTEMS INCORPORATED

                         EXECUTIVE RETENTION AGREEMENT


     This Executive Retention Agreement (this "AGREEMENT") between Banyan
Systems Incorporated, a Massachusetts corporation (the "COMPANY"), and Robert D.
Burke (the "EXECUTIVE") is made as of October 16, 1998 (the "EFFECTIVE DATE").
The Company and the Executive shall, at times, be referred to herein each as, a
"PARTY," and together as, the "PARTIES."

                             Preliminary Statement
                             ---------------------

     A.   The Executive presently serves in the employ of the Company as the
Company's Senior Vice President, Worldwide Sales and Service.

     B.   The Executive and the Company have entered into that certain Offer
Letter dated March 19, 1997 (the "PRIOR AGREEMENT").

     C.   The Company recognizes that, as is the case with many publicly held
corporations, the possibility of a change in control of the Company exists and
that such possibility, and the uncertainty and questions which it may raise
among key personnel, may result in the departure or distraction of key personnel
to the detriment of the Company and its stockholders.

     D.   The Board of Directors of the Company (the "BOARD") has determined
that appropriate steps should be taken to reinforce and encourage the continued
employment and dedication of the Company's key personnel without distraction
from the possibility of a change in control of the Company and related events
and circumstances.

     NOW, THEREFORE, as an inducement for and in consideration of the Executive
remaining in its employ, the mutual promises set forth in this Agreement, and
other good and valuable consideration, the receipt and sufficiency of which is
hereby acknowledged, the Company and the Executive hereby further agree as
follows:

     1.   Key Definitions.
          --------------- 

     As used herein, the following terms shall have the following respective
meanings:
<PAGE>
 
          1.1  "CHANGE IN CONTROL" means an event or occurrence set forth in any
one or more of Section 1.1(a) through 1.1(d) below (including an event or
occurrence that constitutes a Change in Control under one of such Sections but
is specifically exempted from another such Section):

               (a) the acquisition by an individual, entity or group (within the
meaning of Section 13(d)(3) or 14(d)(2) of the Securities Exchange Act of 1934,
as amended (the "EXCHANGE ACT")) (a "PERSON") of beneficial ownership of any
capital stock of the Company if, after such acquisition, such Person
beneficially owns (within the meaning of Rule 13d-3 promulgated under the
Exchange Act) 50% or more of either (i) the then-outstanding shares of common
stock of the Company (the "OUTSTANDING COMPANY COMMON STOCK") or (ii) the
combined voting power of the then-outstanding securities of the Company entitled
to vote generally in the election of directors (the "OUTSTANDING COMPANY VOTING
SECURITIES"); provided, however, that for purposes of this Section 1.1(a), the
              --------                                                        
following acquisitions shall not constitute a Change in Control: (i) any
acquisition directly from the Company (excluding an acquisition pursuant to the
exercise, conversion or exchange of any security exercisable for, convertible
into or exchangeable for common stock or voting securities of the Company,
unless the Person exercising, converting or exchanging such security acquired
such security directly from the Company or an underwriter or agent of the
Company), (ii) any acquisition by the Company, (iii) any acquisition by any
employee benefit plan (or related trust) sponsored or maintained by the Company
or any corporation controlled by the Company, or (iv) any acquisition by any
corporation pursuant to a transaction which complies with clauses (i) and (ii)
of Section 1.1(c) hereof; or

               (b) such time as the Continuing Directors (as defined below) do
not constitute a majority of the Board (or, if applicable, the Board of
Directors of a successor corporation to the Company), where the term "CONTINUING
DIRECTOR" means at any date a member of the Board (i) who was a member of the
Board on the date of the execution of this Agreement or (ii) who was nominated
or elected subsequent to such date by at least a majority of the directors who
were Continuing Directors at the time of such nomination or election or whose
election to the Board was recommended or endorsed by at least a majority of the
directors who were Continuing Directors at the time of such nomination or
election; provided, however, that there shall be excluded from this clause (ii)
          --------  -------
any individual whose initial assumption of office occurred as a result of an
actual or threatened election contest with respect to the election or removal of
directors or other actual or threatened solicitation of proxies or consents, by
or on behalf of a person other than the Board; or

               (c) the consummation of a merger, consolidation, reorganization,
recapitalization or statutory share exchange involving the Company or 

                                      -2-
<PAGE>
 
a sale or other disposition of all or substantially all of the assets of the
Company (a "BUSINESS COMBINATION"), unless, immediately following such Business
Combination, each of the following two conditions is satisfied: (i) all or
substantially all of the individuals and entities who were the beneficial owners
of the Outstanding Company Common Stock and Outstanding Company Voting
Securities immediately prior to such Business Combination beneficially own,
directly or indirectly, more than 50% of the then-outstanding shares of common
stock and the combined voting power of the then-outstanding securities entitled
to vote generally in the election of directors, respectively, of the resulting
or acquiring corporation in such Business Combination (which shall include,
without limitation, a corporation which as a result of such transaction owns the
Company or substantially all of the Company's assets either directly or through
one or more subsidiaries) (such resulting or acquiring corporation is referred
to herein as the "ACQUIRING CORPORATION") in substantially the same proportions
as their ownership, immediately prior to such Business Combination, of the
Outstanding Company Common Stock and Outstanding Company Voting Securities,
respectively; and (ii) no Person (excluding the Acquiring Corporation or any
employee benefit plan (or related trust) maintained or sponsored by the Company
or by the Acquiring Corporation) beneficially owns, directly or indirectly, 50%
or more of the then outstanding shares of common stock of the Acquiring
Corporation, or of the combined voting power of the then-outstanding securities
of such corporation entitled to vote generally in the election of directors
(except to the extent that such ownership existed prior to the Business
Combination); or

                    (d) approval by the stockholders of the Company of a
complete liquidation or dissolution of the Company.

          1.2  "CHANGE IN CONTROL DATE" means the first date during the Term (as
defined in Section 2 hereof) on which a Change in Control occurs.  Anything in
this Agreement to the contrary notwithstanding, if (a) a Change in Control
occurs, (b)  the Executive's employment with the Company is terminated prior to
the date on which the Change in Control occurs, and (c) it is reasonably
demonstrated by the Executive that such termination of employment (i) was at the
request of a third party who has taken steps reasonably calculated to effect a
Change in Control or (ii) otherwise arose in connection with or in anticipation
of a Change in Control, then for all purposes of this Agreement the "CHANGE IN
CONTROL DATE" shall mean the date immediately prior to the date of such
termination of employment.

          1.3  "CAUSE" means:

               (a) the Executive's willful and continued failure to
substantially perform his reasonable assigned duties as an officer of the
Company (other than any such failure resulting from incapacity due to physical
or mental illness or any failure after the Executive gives notice of termination
for Good Reason), which failure is not

                                      -3-
<PAGE>
 
cured within 30 days after a written demand for substantial performance is
received by the Executive from the Chief Executive Officer or the Board (or the
Compensation Committee thereof) which specifically identifies the manner in
which the Chief Executive Officer or the Board (or the Compensation Committee
thereof), respectively, believes the Executive has not substantially performed
the Executive's duties; or

               (b) the Executive's willful engagement in illegal conduct or
gross misconduct which is materially and demonstrably injurious to the Company.

     For purposes of this Section 1.3, no act or failure to act by the Executive
shall be considered "WILLFUL" unless it is done, or omitted to be done, in bad
faith and without reasonable belief that the Executive's action or omission was
in the best interests of the Company.

          1.4  "GOOD REASON" means the occurrence, without the Executive's
written consent, of any of the events or circumstances set forth in clauses (a)
through (g) below.  Notwithstanding the occurrence of any such event or
circumstance, such occurrence shall not be deemed to constitute Good Reason if,
prior to the Date of Termination specified in the Notice of Termination (each as
defined in Section 3.2(a) hereof) given by the Executive in respect thereof,
such event or circumstance has been fully corrected and the Executive has been
reasonably compensated for any losses or damages resulting therefrom (provided
that such right of correction by the Company shall only apply to the first
Notice of Termination for Good Reason given by the Executive).

               (a) the assignment to the Executive of duties inconsistent in any
material respect with the Executive's position (including status, offices,
titles and reporting requirements), authority or responsibilities in effect
immediately prior to the earliest to occur of (i) the Change in Control Date,
(ii) the date of the execution by the Company of the initial written agreement
or instrument providing for the Change in Control or (iii) the date of the
adoption by the Board of a resolution providing for the Change in Control (with
the earliest to occur of such dates referred to herein as the "MEASUREMENT
DATE"), or any other action or omission by the Company which results in a
diminution in such position, authority or responsibilities;

               (b) a reduction in the aggregate of the Executive's annual base
salary and ON TARGET EARNINGS (as defined in the Company's Executive Bonus Plan)
as in effect on the Measurement Date or as the same was or may be increased from
time to time;

               (c) the failure by the Company to (i) continue in effect any
material compensation or benefit plan or program (including without limitation
any

                                      -4-
<PAGE>
 
life insurance, medical, health and accident or disability plan) (a "BENEFIT
PLAN") in which the Executive participates or which is applicable to the
Executive immediately prior to the Measurement Date, unless an equitable
arrangement (embodied in an ongoing substitute or alternative plan) has been
made with respect to such plan or program, (ii) continue the Executive's
participation therein (or in such substitute or alternative plan) on a basis not
materially less favorable, both in terms of the amount of benefits provided and
the level of the Executive's participation relative to other participants, than
the basis existing immediately prior to the Measurement Date or (iii) award cash
bonuses to the Executive in amounts and in a manner substantially consistent
with past practice in light of the Company's financial performance;

               (d) a change by the Company in the location at which the
Executive performs his principal duties for the Company to a new location that
is more than 35 miles from the location at which the Executive performed his
principal duties for the Company immediately prior to the Measurement Date;

               (e) the failure of the Company to obtain the agreement, in a form
reasonably satisfactory to the Executive, from any successor to the Company to
assume and agree to perform this Agreement, as required by Section 6.1 hereof;
or

               (f) any failure of the Company to pay or provide to the Executive
any portion of the Executive's compensation or benefits due under any Benefit
Plan within seven days of the date such compensation or benefits are due, or any
material breach by the Company of any employment agreement with the Executive.

     The Executive's right to terminate his employment for Good Reason shall not
be affected by his incapacity due to physical or mental illness.

          1.5  "DISABILITY" means the Executive's absence from the full-time
performance of the Executive's duties with the Company for 180 consecutive
calendar days as a result of incapacity due to mental or physical illness which
is determined to be total and permanent by a physician selected by the Company
or its insurers and reasonably acceptable to the Executive or the Executive's
legal representative.

          1.6  "MATERIAL REDUCTION IN RESPONSIBILITIES" means the occurrence,
without Cause, of any of the following: (i) the Company removes the Executive
from the position of Senior Vice President, Worldwide Sales and Service or as an
officer of the Company or reduces the responsibilities of the Executive from
managing worldwide sales and service, (ii) the Company assigns an individual
other than William P. Ferry as the person to whom the Executive directly
reports, (iii) William P. Ferry is no longer the Chairman of the Board and Chief
Executive Officer of the Company and (iv) the Company reduces the Executive's
base salary and executive 

                                      -5-
<PAGE>
 
bonus. Any voluntary termination by the Executive of his employment with the
Company for a Material Reduction in Responsibilities must occur within 60 days
of such Material Reduction in Responsibilities, and after giving 30 days prior
written notice to the Company.

     2.   Term of Agreement.  This Agreement, and all rights and obligations of
          -----------------                                                    
the Parties, shall take effect upon the Effective Date and shall expire upon the
first to occur of (a) the expiration of the Term (as defined below) if a Change
in Control has not occurred during the Term, (b) the date 12 months after the
Change in Control Date, if the Executive is still employed by the Company as of
such later date, or (c) the fulfillment by the Company of all of its obligations
under Sections 4 and 5.2 hereof if the Executive's employment with the Company
terminates within 12 months following the Change in Control Date.  "TERM" shall
mean the period commencing as of the Effective Date and continuing in effect
through October 16, 2001; provided, however, that commencing on October 17, 2001
                          --------                                              
and each October 17 thereafter, the Term shall be automatically extended for one
additional year unless, not later than 90 days prior to the scheduled expiration
of the Term (or any extension thereof), the Company shall have given the
Executive written notice that the Term will not be extended. Notwithstanding the
foregoing, the provisions of Section 3.1 hereof shall remain in full force and
effect for the duration of the Executive's employment with the Company.

     3.   Employment Status; Termination.
          ------------------------------ 

          3.1  Employment Status.
               ----------------- 

               (a) Not an Employment Contract. The Executive acknowledges that
                   --------------------------
this Agreement does not constitute a contract of employment or impose on the
Company any obligation to retain the Executive as an employee and that this
Agreement does not prevent the Executive from terminating employment at any
time. Notwithstanding the foregoing, the Executive may be entitled to certain
benefits under Sections 3.1(b) or 4 hereof, if the Executive's employment with
the Company is terminated pursuant to the circumstances described therein. If
the Executive's employment with the Company terminates for any reason and
subsequently a Change in Control shall occur, the Executive shall not be
entitled to any benefits under Section 4 hereof, except as otherwise provided
pursuant to Section 1.2 hereof, in which event, any benefits provided pursuant
to Section 4 hereof shall be net of and not in addition to any benefits provided
pursuant to Section 3.1(b) hereof.

               (b) Termination Benefits. Upon the termination of the Executive's
                   --------------------
employment with the Company for any reason other than (v) the Executive's death,
(w) the Executive's Disability, (x) termination for Cause, (y) voluntary
termination at the election of the Executive other than for a Material Reduction
in Responsibilities or (z) any termination pursuant to which the Executive is

                                      -6-
<PAGE>
 
entitled to benefits under Section 4.2 hereof, the Executive shall be entitled
to the following benefits:

                   (i)   For the period nine months subsequent to the Date of
Termination (as defined in Section 3.2 hereof), the Company will pay to the
Executive each month a severance amount calculated as the arithmetic average of
the total cash compensation, including all salary and bonuses, paid by the
Company to the Executive each month for the 12 full calendar months preceding
the Date of Termination. The Company shall deduct from such payments any
applicable deductions for medical, dental, prescription and life insurance
contributions and/or withholdings for taxes or similar governmental payments or
charges.

                   (ii)  Within thirty days of the Date of Termination, the
Company will pay to the Executive all bonuses, if any, earned by the Executive
as of the Date of Termination and the pro rata portion of all guaranteed
bonuses, if any, earned by the Executive during his employment by the Company
through the Date of Termination.

                   (iii) For the period nine months subsequent to the Date of
Termination, the Company will provide continued medical, dental and life
insurance coverage to the Executive and the Executive's family, on terms
substantially as in effect on the Date of Termination, subject to the payment by
the Executive of all applicable employee contributions.

                   (iv)  (A) If this Section 3.1(b)(iv) is applicable due to the
termination of the Executive's employment with the Company for any reason other
than a voluntary termination at the election of the Executive for a Material
Reduction in Responsibilities, each outstanding nonqualified option to purchase
shares of Common Stock of the Company held by the Executive shall become
immediately exercisable as to all shares of Common Stock of the Company subject
to such options for which such option would have become exercisable within the
period nine months subsequent to the Date of Termination; or (B) if this Section
3.1(b)(iv) is applicable due to the termination of the Executive's employment
with the Company due to voluntary termination at the election of the Executive
for a Material Reduction in Responsibilities, the vesting of each outstanding
nonqualified option to purchase shares of Common Stock of the Company held by
the Executive shall, notwithstanding any provisions in such option or the
Company's 1992 Stock Incentive Plan, as amended, to the contrary, continue for
the earlier to occur of (i) the expiration of such option pursuant to its terms
of (II) the date nine months subsequent to the Date of Termination.

               (c) Patent and Confidentiality Agreement.  The Executive hereby
                   ------------------------------------                       
ratifies and confirms in all respects his obligations and responsibilities
pursuant to the Employee Patent and Confidential Information Agreement between
the

                                      -7-
<PAGE>
 
Company and the Executive. Without limiting the generality of the foregoing, the
Executive agrees that all documents, records, techniques, business secrets and
other information which have and will come into his possession from time to time
during his employment with the Company shall be deemed to be confidential and
proprietary to the Company. The Executive shall retain in his confidence any
confidential information known to him concerning the Company and its
subsidiaries and their respective businesses and such information shall not be
disclosed to any person or entity other than employees of the Company and its
subsidiaries who have a need to know such information, nor shall the Executive
use the same for any purpose (other than in the performance of his duties as an
employee of the Company) without written approval by another officer of the
Company, either during or after his employment with the Company, unless and
until such information has become public knowledge without fault by the
Executive.

               (d) Services During Certain Events. In the event that a third
                   ------------------------------  
person begins a tender or exchange offer with respect to the Outstanding Company
Common Stock, circulates a proxy to the Company's stockholders, or takes other
steps seeking to effect a Change in Control, and until such third person has
abandoned or terminated such efforts to effect a Change in Control or until
after such a Change in Control has been effected, the Executive agrees that he
will not voluntarily leave the employ of the Company and will continue to serve
the Company in all capacities in which he served the Company immediately prior
to the date of the taking of such steps to effect a Change in Control.

          3.2  Notice of Termination of Employment.
               ----------------------------------- 

               (a) Any termination of the Executive's employment by the Company
or by the Executive during the Term or, if the Change in Control Date occurs
during the Term, within 12 months following the Change in Control Date (other
than due to the death of the Executive) shall be communicated by a written
notice to the other Party (the "NOTICE OF TERMINATION"), given in accordance
with Section 7 hereof. Any Notice of Termination shall: (i) indicate the
specific termination provision (if any) of this Agreement relied upon by the
Party giving such notice, (ii) to the extent applicable, set forth in reasonable
detail the facts and circumstances claimed to provide a basis for termination of
the Executive's employment under the provision so indicated and (iii) specify
the Date of Termination (as defined below). The effective date of an employment
termination (the "DATE OF TERMINATION") shall be the close of business on the
date specified in the Notice of Termination (which date may not be less than 15
days or more than 120 days after the date of delivery of such Notice of
Termination), in the case of a termination other than one due to the Executive's
death, or the date of the Executive's death, as the case may be.

               (b) The failure by the Executive or the Company to set forth in

                                      -8-
<PAGE>
 
the Notice of Termination any fact or circumstance which contributes to a
showing of Good Reason or Cause shall not waive any right of the Executive or
the Company, respectively, hereunder or preclude the Executive or the Company,
respectively, from asserting any such fact or circumstance in enforcing the
Executive's or the Company's right hereunder.

               (c) Any Notice of Termination for Cause given by the Company must
be given within 90 days of the occurrence of the event(s) or circumstance(s)
which constitute(s) Cause. Subsequent to such event(s) or circumstance(s) and
prior to any Notice of Termination for Cause being given (and prior to any
termination for Cause being effective), the Executive shall be entitled to a
hearing before the Board (or the Compensation Committee thereof) at which he
may, at his election, be represented by counsel and at which he shall have a
reasonable opportunity to be heard. Such hearing shall be held on not less than
15 days prior written notice to the Executive stating the Board's intention to
terminate the Executive for Cause and stating in detail the particular event(s)
or circumstance(s) which the Board believes constitutes Cause for termination.

               (d) Any Notice of Termination for Good Reason given by the
Executive must be given within 30 days of the occurrence of the event(s) or
circumstance(s) which constitute(s) Good Reason.

     4.   Change in Control Benefits to Executive.
          --------------------------------------- 

          4.1  Change in Control.  If the Change in Control Date occurs during
               -----------------                                              
the Term, then, effective upon the Change in Control Date, (a) each outstanding
option to purchase shares of Common Stock of the Company held by the Executive
shall become immediately exercisable as to 50% of the then unvested shares
subject to such option, with such immediate vesting applied ratably to such
unvested shares, and (b) each outstanding restricted stock award held by the
Executive shall be deemed to be fully vested and no longer subject to a right of
repurchase by the Company.

          4.2  Change in Control and Termination of Employment.  If the Change
               -----------------------------------------------                
in Control Date occurs during the Term and the Executive's employment with the
Company terminates within 12 months following the Change in Control Date, the
Executive shall be entitled to the following benefits:

               (a) Termination Without Cause or for Good Reason. If the
                   --------------------------------------------
Executive's employment with the Company is terminated by the Company (other than
for Cause, Disability or Death) or by the Executive for Good Reason within 12
months following the Change in Control Date, then the Executive shall be
entitled to the following benefits:

                                      -9-
<PAGE>
 
                   (i)   the Company shall pay to the Executive in a lump sum in
cash within 30 days after the Date of Termination the aggregate of the following
amounts:

                         (1) the sum of (A) the Executive's earned but unpaid
base salary through the Date of Termination, (B) the product of (x) the annual
bonus paid or payable (including any bonus or portion thereof which has been
earned but deferred) for the most recently completed fiscal year and (y) a
fraction, the numerator of which is the number of days in the current fiscal
year through the Date of Termination, and the denominator of which is 365 and
(C) the amount of any compensation previously earned and deferred by the
Executive, in each case to the extent not previously paid (the sum of the
amounts described in clauses (A), (B), and (C) shall be hereinafter referred to
as the "ACCRUED OBLIGATIONS"); and

                         (2) the amount equal to the sum of (A) the Executive's
annual base salary for the calendar year in which the Change in Control Date or
the Date of Termination occurred, whichever annual base salary is greater, and
(B) the Executive's bonus under the Company's Executive Bonus Plan for the
calendar year in which the Change in Control Date or the Date of Termination
occurred (in each case, assuming for this purpose that all targets requisite to
qualifying the Executive for 100% of On Target Earnings were met and/or
satisfied in full, whether or not such targets were actually met and/or
satisfied), whichever such bonus is greater.

                   (ii)  (a) each outstanding option to purchase shares of
Common Stock of the Company held by the Executive shall become immediately
exercisable in full and (b) each outstanding restricted stock award granted to
the Executive subsequent to the Change in Control Date shall be deemed to be
fully vested and no longer subject to a right of repurchase by the Company.

                   (iii) for 12 months after the Date of Termination, the
Company shall continue to provide benefits to the Executive and the Executive's
family at least equal to those which would have been provided to them if the
Executive's employment had not been terminated, in accordance with the
applicable Benefit Plans in effect on the Measurement Date; provided, however,
                                                            --------
that if the Executive becomes reemployed with another employer and is eligible
to receive a particular type of benefits (e.g., health insurance benefits) from
such employer on terms at least as favorable to the Executive and his family as
those being provided by the Company, then the Company shall no longer be
required to provide those particular benefits to the Executive and his family;

                   (iv)  to the extent not previously paid or provided, the
Company shall timely pay or provide to the Executive any other amounts or
benefits required to be paid or provided or which the Executive is eligible to
receive following

                                      -10-
<PAGE>
 
the Executive's termination of employment under any plan, program, policy,
practice, contract or agreement of the Company and its affiliated companies
(such other amounts and benefits shall be hereinafter referred to as the "OTHER
BENEFITS"); and

                   (v)   for purposes of determining eligibility (but not the
time of commencement of benefits) of the Executive for retiree benefits to which
the Executive is entitled, the Executive shall be considered to have remained
employed by the Company until 12 months after the Date of Termination.

               (b) Resignation without Good Reason; Termination for Death or
                   ---------------------------------------------------------
Disability.  If the Executive voluntarily terminates his employment with the
----------                                                                  
Company within 12 months following the Change in Control Date, excluding a
termination for Good Reason, or if the Executive's employment with the Company
is terminated by reason of the Executive's death or Disability within 12 months
following the Change in Control Date, then the Company shall (i) pay the
Executive (or his estate, if applicable), in a lump sum in cash within 30 days
after the Date of Termination, the Accrued Obligations and (ii) timely pay or
provide to the Executive the Other Benefits.

               (c) Termination for Cause. If the Company terminates the
                   ---------------------
Executive's employment with the Company for Cause within 12 months following the
Change in Control Date, then the Company shall (i) pay the Executive, in a lump
sum in cash within 30 days after the Date of Termination, the sum of (A) the
Executive's annual base salary through the Date of Termination and (B) the
amount of any compensation previously deferred by the Executive, in each case to
the extent earned but not previously paid, and (ii) timely pay or provide to the
Executive the Other Benefits.

          4.3  Taxes.
               ----- 

               (a) Notwithstanding any other provision of this Agreement, except
as set forth in Section 4.3(b) hereof, in the event that the Company undergoes a
"CHANGE IN OWNERSHIP OR CONTROL" (as defined below), the Company shall not be
obligated to provide to the Executive a portion of any "CONTINGENT COMPENSATION
PAYMENTS" (as defined below) that the Executive would otherwise be entitled to
receive to the extent necessary to eliminate any "EXCESS PARACHUTE PAYMENTS" (as
defined in Section 280G(b)(1) of the Internal Revenue Code of 1986, as amended
(the "CODE")) for the Executive. For purposes of this Section 4.3, the
Contingent Compensation Payments so eliminated shall be referred to as the
"ELIMINATED PAYMENTS" and the aggregate amount (determined in accordance with
Proposed Treasury Regulation Section 1.280G-1, Q/A-30 or any successor
provision) of the Contingent Compensation Payments so eliminated shall be
referred to as the "ELIMINATED AMOUNT."

                                      -11-
<PAGE>
 
               (b) Notwithstanding the provisions of Section 4.3(a) hereof, no
such reduction in Contingent Compensation Payments shall be made if (i) the
Eliminated Amount (computed without regard to this sentence) exceeds (ii) 110%
of the aggregate present value (determined in accordance with Proposed Treasury
Regulation Section 1.280G-1, Q/A-31 and Q/A-32 or any successor provisions) of
the amount of any additional taxes that would be incurred by the Executive if
the Eliminated Payments (determined without regard to this sentence) were paid
to him (including, state and federal income taxes on the Eliminated Payments,
the excise tax imposed by Section 4999 of the Code payable with respect to all
of the Contingent Compensation Payments in excess of the Executive's "BASE
AMOUNT" (as defined in Section 280G(b)(3) of the Code, and any payroll taxes).
The override of such reduction in Contingent Compensation Payments pursuant to
this Section 4.3(b) shall be referred to as a "SECTION 4.3(B) OVERRIDE." For
purpose of this Section 4.3(b), the amount of any federal or state income taxes
shall be computed by using the maximum combined federal and state income tax
rate provided by law.

               (c) For purposes of this Section 4.3 the following terms shall
have the following respective meanings:

                   (i)   "CHANGE IN OWNERSHIP OR CONTROL" shall mean a change in
the ownership or effective control of the Company or in the ownership of a
substantial portion of the assets of the Company determined in accordance with
Section 280G(b)(2) of the Code.

                   (ii)  "CONTINGENT COMPENSATION PAYMENT" shall mean any
payment (or benefit) in the nature of compensation that is made (or made
available to) a "DISQUALIFIED INDIVIDUAL" (as defined in Section 280G(c) of the
Code) and that is "CONTINGENT" (within the meaning of Section 280G(b)(2)(A)(i)
of the Code) on a Change in Ownership or Control of the Company.

               (d) Any payments or benefits otherwise due to the Executive
following a Change in Ownership or Control that could reasonably be
characterized (as determined by the Company) as Contingent Compensation Payments
shall not be made until the dates provided for in this Section 4.3(d). Within 30
days after each date on which the Executive first becomes entitled to receive
(whether or not then due) a Contingent Compensation Payment relating to such
Change in Ownership or Control, the Company shall provide notice to the
Executive setting forth the Company's determination (with reasonable detail
regarding the basis for its determination) of: (i) the Contingent Compensation
Payments, (ii) the Eliminated Amount and (iii) whether the Section 4.3(b)
Override is applicable. Within 30 days after delivery of such notice to the
Executive, the Executive shall deliver a response to the Company (the "EXECUTIVE
RESPONSE") stating either (A) that he agrees with the Company's determination

                                      -12-
<PAGE>
 
pursuant to the preceding sentence, in which case he shall indicate, if
applicable, which Contingent Compensation Payments, or portions thereof (the
aggregate amount of which, determined in accordance with Proposed Treasury
Regulation Section 1.280G-1, Q/A-30 or any successor provision, shall be equal
to the Eliminated Amount), shall be treated as Eliminated Payments or (B) that
he disagrees with such determination, in which case he shall set forth (i) which
payments or benefits should not be characterized as Contingent Compensation
Payments, (ii) the Eliminated Amount, (iii) whether the Section 4.3(b) Override
is applicable, and, (iv) which (if any) Contingent Compensation Payments, or
portions thereof (the aggregate amount of which, determined in accordance with
Proposed Treasury Regulation Section 1.280G-1, Q/A-30 or any successor
provision, shall be equal to the Eliminated Amount, if any), shall be treated as
Eliminated Payments. In the event that the Executive fails to deliver an
Executive Response on or before the required date, the Company's initial
determination shall be final and the Contingent Compensation Payments that shall
be treated as Eliminated Payments shall be determined by the Company in its
absolute discretion. If the Executive states in the Executive Response that he
agrees with the Company's determination, the Company shall make the Contingent
Compensation Payments to the Executive within three business days following
delivery to the Company of the Executive Response (except for any such
Contingent Compensation Payments which are not due to be made until after such
date, which such Contingent Compensation Payments shall be made on the date on
which they are due). If the Executive states in the Executive Response that he
disagrees with the Company's determination, then, for a period of 60 days
following delivery of the Executive Response, the Executive and the Company
shall use good faith efforts to resolve such dispute. If such dispute is not
resolved within such 60-day period, such dispute shall be settled exclusively by
arbitration in Boston, Massachusetts, in accordance with the rules of the
American Arbitration Association then in effect. Judgment may be entered on the
arbitrator's award in any court having jurisdiction. The Company shall, within
three business days following delivery to the Company of the Executive Response,
make to the Executive those Contingent Compensation Payments as to which there
is no dispute between the Company and the Executive regarding whether they
should be made (except for any such Contingent Compensation Payments which are
not due to be made until after such date, which such Contingent Compensation
Payments shall be made on the date on which they are due). The balance of the
Contingent Compensation Payments shall be made within three business days
following the resolution of such dispute. Subject to the limitations contained
in Sections 4.3(a) and (b) hereof, the amount of any payments to be made to the
Executive following the resolution of such dispute shall be increased by
interest thereon computed at the prime rate announced from time to time by
Citibank, N.A., compounded monthly from the date that such payments originally
were due.

               (e) The provisions of this Section 4.3 are intended to apply to
any and all payments or benefits available to the Executive under this Agreement
or 

                                      -13-
<PAGE>
 
any other agreement or plan of the Company under which the Executive receives
Contingent Compensation Payments.

          4.4  Mitigation.  The Executive shall not be required to mitigate the
               ----------                                                      
amount of any payment or benefits provided for in this Section 4 by seeking
other employment or otherwise. Further, except as provided in Section 4.2(a)(ii)
hereof, the amount of any payment or benefits provided for in this Section 4
shall not be reduced by any compensation earned by the Executive as a result of
employment by another employer, by retirement benefits, by offset against any
amount claimed to be owed by the Executive to the Company or otherwise.

     5.   Disputes.
          -------- 

          5.1  Settlement of Disputes; Arbitration.  All claims by the Executive
               -----------------------------------                              
for benefits under this Agreement shall be directed to and determined by the
Board (or the Compensation Committee thereof) and shall be in writing.  Any
denial by the Board (or the Compensation Committee thereof) of a claim for
benefits under this Agreement shall be delivered to the Executive in writing and
shall set forth the specific reasons for the denial and the specific provisions
of this Agreement relied upon.  The Board (or the Compensation Committee
thereof) shall afford a reasonable opportunity to the Executive for a review of
the decision denying a claim.  Any further dispute or controversy arising under
or in connection with this Agreement shall be settled exclusively by arbitration
in Boston, Massachusetts, in accordance with the rules of the American
Arbitration Association then in effect.  Judgment may be entered on the
arbitrator's award in any court having jurisdiction.

          5.2  Expenses.  The Company agrees to pay as incurred, to the full
               --------                                                     
extent permitted by law, all legal, accounting and other fees and expenses which
the Executive may reasonably incur as a result of any claim or contest (with
respect to which the Executive is ultimately the prevailing party) by the
Company, the Executive or others regarding the validity or enforceability of, or
liability under, any provision of this Agreement or any guarantee of performance
thereof (including as a result of any contest by the Executive regarding the
amount of any payment or benefits pursuant to this Agreement), plus in each case
interest on any delayed payment at the applicable Federal rate provided for in
Section 7872(f)(2)(A) of the Code.

     6.   Successors.
          ---------- 

          6.1  Successor to Company.  The Company shall require any successor
               --------------------                                          
(whether direct or indirect, by purchase, merger, consolidation or otherwise) to
all or substantially all of the business or assets of the Company expressly to
assume and agree to perform this Agreement to the same extent that the Company
would be required to perform it if no such succession had taken place.  Failure
of the Company to 

                                      -14-
<PAGE>
 
obtain an assumption of this Agreement at or prior to the effectiveness of any
succession shall be a breach of this Agreement and shall constitute Good Reason
if the Executive elects to terminate employment, except that for purposes of
implementing the foregoing, the date on which any such succession becomes
effective shall be deemed the Date of Termination. As used in this Agreement,
"Company" shall mean the Company as defined above and any successor to its
business or assets as aforesaid which assumes and agrees to perform this
Agreement, by operation of law or otherwise.

          6.2  Successor to Executive.  This Agreement shall inure to the
               ----------------------                                    
benefit of and be enforceable by the Executive's personal or legal
representatives, executors, administrators, successors, heirs, distributees,
devisees and legatees.  If the Executive should die while any amount would still
be payable to the Executive or his family hereunder if the Executive had
continued to live, all such amounts, unless otherwise provided herein, shall be
paid in accordance with the terms of this Agreement to the executors, personal
representatives or administrators of the Executive's estate.

     7.   Notice.  All notices, instructions and other communications given
          ------                                                           
hereunder or in connection herewith shall be in writing.  Any such notice,
instruction or communication shall be sent either (i) by registered or certified
mail, return receipt requested, postage prepaid, or (ii) prepaid via a reputable
nationwide overnight courier service, in each case addressed to the Company at
Vice President, Human Resources, Banyan Systems Incorporated, 120 Flanders Road,
P.O. Box 5013, Westboro, Massachusetts  01581-5013, and to the Executive at
Robert D. Burke, 30 Allen Circle, Boxford, Massachusetts  01921 (or to such
other address as either the Company or the Executive may have furnished to the
other in writing in accordance herewith).  Any such notice, instruction or
communication shall be deemed to have been delivered five business days after it
is sent by registered or certified mail, return receipt requested, postage
prepaid, or one business day after it is sent via a reputable nationwide
overnight courier service. Either Party may give any notice, instruction or
other communication hereunder using any other means, but no such notice,
instruction or other communication shall be deemed to have been duly delivered
unless and until it actually is received by the Party for whom it is intended.

     8.   Miscellaneous.
          ------------- 

          8.1  Employment by Subsidiary.  For purposes of this Agreement, the
               ------------------------                                      
Executive's employment with the Company shall not be deemed to have terminated
solely as a result of the Executive continuing to be employed by a majority-
owned subsidiary of the Company.

          8.2  Severability.  The invalidity or unenforceability of any
               ------------                                            
provision of this Agreement shall not affect the validity or enforceability of
any other provision of

                                      -15-
<PAGE>
 
this Agreement, which shall remain in full force and effect.

          8.3  Injunctive Relief.  The Company and the Executive agree that any
               -----------------                                               
breach of this Agreement by the Company is likely to cause the Executive
substantial and irrevocable damage and therefore, in the event of any such
breach, in addition to such other remedies which may be available, the Executive
shall have the right to specific performance and injunctive relief.

          8.4  Governing Law.  The validity, interpretation, construction and
               -------------                                                 
performance of this Agreement shall be governed by the internal laws of the
Commonwealth of Massachusetts, without regard to conflicts of law principles.

          8.5  Waivers.  No waiver by the Executive at any time of any breach
               -------                                                       
of, or compliance with, any provision of this Agreement to be performed by the
Company shall be deemed a waiver of that or any other provision at any
subsequent time.

          8.6  Counterparts.  This Agreement may be executed in counterparts,
               ------------                                                  
each of which shall be deemed to be an original but both of which together shall
constitute one and the same instrument.

          8.7  Tax Withholding.  Any payments provided for hereunder shall be
               ---------------                                               
paid net of any applicable tax withholding required under federal, state or
local law.

          8.8  Entire Agreement.  This Agreement sets forth the entire agreement
               ----------------                                                 
of the Parties in respect of the subject matter contained herein and supersedes
all prior agreements, promises, covenants, arrangements, communications,
representations or warranties, whether oral or written, by any officer, employee
or representative of any Party, including, without limitation, Section G of the
Prior Agreement; and any prior agreement of the Parties in respect of the
subject matter contained herein, including, without limitation, Section G of the
Prior Agreement, is hereby terminated and canceled.

          8.9  Amendments.  This Agreement may be amended or modified only by a
               ----------                                                      
written instrument executed by both the Company and the Executive.

                                      -16-
<PAGE>
 
          IN WITNESS WHEREOF, the Parties have executed this Agreement as of the
day and year first set forth above.


                              BANYAN SYSTEMS INCORPORATED



                              By: /s/ William P. Ferry
                                  -----------------------------------
                                  Name    William P. Ferry
                                  Title:  Chairman of the Board, President and
                                          Chief Executive Officer



                              ROBERT D. BURKE



                              /s/ Robert D. Burke
                              -------------------
                              Individually

                                      -17-