<PAGE>   1
                                                                    Exhibit 10.4



                              EMPLOYMENT AGREEMENT


     AGREEMENT by and between United Asset Management Corporation, a Delaware
corporation (the "Company"), and James F. Orr III (the "Executive"), dated as of
the 15th day of May, 2000.

     WHEREAS, the Board of Directors of the Company (the "Board") has determined
that it is in the best interests of the Company and its shareholders to employ
the Executive as its President and Chief Executive Officer, and the Executive
desires to serve in that capacity;

     NOW, THEREFORE, IT IS HEREBY AGREED AS FOLLOWS:

     1.   EMPLOYMENT PERIOD. The Company shall employ the Executive, and the
Executive shall serve the Company, on the terms and conditions set forth in this
Agreement, for the Employment Period (as defined in the next sentence). The
"Employment Period" shall mean the period beginning on May 15, 2000 and ending
on May 15, 2003; provided, however, that on May 15, 2003 and on each subsequent
May 15 (each such May 15 being hereinafter referred to as a "Renewal Date"),
unless previously terminated, the Employment Period shall be automatically
extended by one year, unless at least 60 days prior to the Renewal Date either
party hereto shall give notice to the other that the Employment Period shall not
be so extended.

     2.   POSITION AND DUTIES. (a) During the Employment Period, the Executive
shall serve as the President and Chief Executive Officer, with such duties and
responsibilities as are customarily assigned to such positions. Effective as of
the first day of the Employment


<PAGE>   2


Period, the Executive shall be appointed as a member of the Board, and
thereafter during the Employment Period he shall be nominated for re-election as
a member of the Board. In addition, beginning September 21, 2000 and thereafter
during the Employment Period, the Executive shall serve as the Chairman of the
Board. At all times during the Employment Period, the Executive shall report
directly to the Board.

          (b)  During the Employment Period, and excluding any periods of
vacation and sick leave to which the Executive is entitled, the Executive shall
devote full business attention and time to the business and affairs of the
Company and use the Executive's reasonable best efforts to carry out such
responsibilities faithfully and efficiently. It shall not be considered a
violation of the foregoing for the Executive to (A) serve on corporate, civic or
charitable boards or committees, (B) deliver lectures, fulfill speaking
engagements or teach at educational institutions and (C) manage personal
investments, so long as such activities are reasonable and do not interfere with
the performance of the Executive's responsibilities under this Agreement.

          (c)  The Executive's services under this Agreement shall be performed
primarily at the Company's headquarters in the Boston, Massachusetts
metropolitan area.

     3.   COMPENSATION. (a) BASE SALARY. During the Employment Period, the
Executive shall receive a base salary (the " Base Salary") at the annual rate of
$1,000,000, payable at such intervals as the Company pays salaries of its senior
executives generally. During the Employment Period, the Base Salary shall be
reviewed for possible increase at least annually. Any increase in the Base
Salary shall not limit or reduce any other obligation of the Company


                                      -2-
<PAGE>   3


under this Agreement. The Base Salary shall not be reduced after any such
increase, and the term "Base Salary" shall thereafter refer to the Base Salary
as so increased.

          (a)  ANNUAL BONUS. (i) The Executive shall be paid a bonus or bonuses
(the "Guaranteed Bonus") for the first twelve months of the Employment Period
equal to $1,000,000.

               (ii) The Executive shall also be eligible to earn, for each
fiscal year ending during the Employment Period, an annual bonus (the "Annual
Bonus") pursuant to the Company's Operating Cash Flow bonus arrangement (the
"Bonus Arrangement"), or any successor shareholder-approved bonus arrangement
that meets the requirements of Section 162(m)(4)(C), based upon the achievement
of objectives approved by a committee of non-employee members of the Board, with
a target Annual Bonus equal to 100% of the Base Salary as in effect at the
beginning of such fiscal year, and a maximum Annual Bonus equal to 200% thereof;
PROVIDED, that the target and maximum Annual Bonus for 2000 shall be pro-rated
based upon the portion of 2000 that is included in the Employment Period; and
PROVIDED, FURTHER, that the Annual Bonuses actually paid for 2000 and 2001 shall
equal the excess (if any) of the amount so determined over the amount of the
Guaranteed Bonus paid in 2000 and 2001, respectively. Notwithstanding the
foregoing, for so long as the Executive's Annual Bonus is to be paid under the
Bonus Arrangement, the maximum Annual Bonus for any fiscal year shall not exceed
the $2 million limit currently in effect under the Bonus Arrangement unless the
Company obtains shareholder approval of a higher limit, and the Company shall
use its reasonable best efforts to obtain such approval if the Executive's Base
Salary is increased above $1,000,000.


                                      -3-
<PAGE>   4


          (c)  STOCK OPTIONS. The Executive shall be granted, on the first day
of the Employment Period, stock options (the "Options") with respect to: (i)
three hundred thousand shares of the Company's common stock under the Company's
Amended and Restated 1994 Stock Option Plan, as amended and restated as of March
31, 1999 (the "First Option Plan") and (ii) subject to approval by the Company's
stockholders of the Company's Second Amended and Restated 1994 Stock Option Plan
(the "Second Option Plan," and collectively with the First Option Plan, the
"Option Plans" ), seven hundred thousand shares of the Company's common stock
under the Second Option Plan, at a per share exercise price equal to the fair
market value of a share of such common stock on the date of grant; provided,
that such one million number of shares shall be subject to adjustment pursuant
to Section 9 of each of the Option Plans. The Options shall vest in four equal
installments on each of the first four anniversaries of the date of grant, shall
have a term of ten years from the date of grant, and shall otherwise be subject
to the standard terms and conditions for options under the Option. Plans. The
Options shall be incentive stock options within the meaning of Section 422 of
the Internal Revenue Code of 1986, as amended, to the extent possible.

          (d)  OTHER BENEFITS. During the Employment Period: (i) the Executive
shall be entitled to participate in all savings, retirement and executive fringe
benefit plans, practices, policies and programs of the Company to the same
extent as other senior executives of the Company; and (ii) the Executive and/or
the Executive's family, as the case may be, shall be eligible for participation
in, and shall receive all benefits under, all welfare benefit plans, practices,
policies and programs provided by the Company (including, without limitation,
medical, prescription, dental, disability, employee life insurance, group life
insurance, accidental


                                      -4-
<PAGE>   5


death and travel accident insurance plans and programs) to the same extent as
other senior executives of the Company.

          (e)  EXPENSES. (i) The Company shall pay to the Executive, within 90
days following the date of this agreement, the sum of $100,000 for (A) the cost
of maintaining temporary housing for himself in the Boston area, (B) the costs
of moving his family and household good to a permanent residence in the Boston
area, and (C) the closing costs associated with purchasing a permanent residence
in the Boston area.

               (ii) During the Employment Period, the Company shall reimburse
the Executive for all reasonable expenses incurred by the Executive in carrying
out the Executive's duties under this Agreement, provided that the Executive
complies with the policies, practices and procedures of the Company for
submission of expense reports, receipts, or similar documentation of such
expenses.

          (f)  VACATION. During the Employment Period, the Executive shall be
entitled to paid vacation of four weeks per year.

     4.   TERMINATION OF EMPLOYMENT.

          (a)  DEATH OR DISABILITY. The Executive's employment shall terminate
automatically upon the Executive's death during the Employment Period. The
Company shall be entitled to terminate the Executive's employment upon written
notice because of the Executive's Disability during the Employment Period.
"Disability" shall mean the absence of the Executive from the Executive's duties
with the Company on a full-time basis for 180 consecutive business days as a
result of incapacity due to mental or physical illness which is determined to be
total and


                                      -5-
<PAGE>   6


permanent by a physician selected by the Company or its insurers and acceptable
to the Executive or the Executive's legal representative.

          (b)  BY THE COMPANY. (i) The Company may terminate the Executive's
employment during the Employment Period for Cause or without Cause. "Cause"
means:

               A.   the willful and continued failure of the Executive
          substantially to perform the Executive's duties under this Agreement
          (other than as a result of physical or mental illness or injury or
          which occurs after the Executive gives a valid Notice of Termination
          for Good Reason), after the Board delivers to the Executive a written
          demand for substantial performance that specifically identifies the
          manner in which the Board believes that the Executive has not
          substantially performed the Executive's duties; or

               B.   illegal conduct that results in material damage to the
          business or reputation of the Company or gross misconduct by the
          Executive.


No act or failure to act on the part of the Executive shall be considered
"willful" unless it is done, or omitted to be done, by the Executive in bad
faith or without reasonable belief that the Executive's action or omission was
in the best interests of the Company. Any act or failure to act that is based
upon authority given pursuant to a resolution duly adopted by the Board, or the
advice of counsel for the Company, shall be conclusively presumed to be done, or
omitted to be done, by the Executive in good faith and in the best interests of
the Company.

               (ii) The Executive's termination for Cause shall be effective
when and if a resolution is duly adopted by the Board, stating that in the good
faith opinion of the Board, the Executive is guilty of conduct that constitutes
Cause under this Agreement, and


                                      -6-
<PAGE>   7


that the Board is therefore terminating the Executive for Cause, and a copy of
such resolution is delivered to the Executive.

          (c)  GOOD REASON. (i) The Executive may terminate employment for Good
Reason or without Good Reason. "Good Reason" means:

               A.   the assignment to the Executive of any duties inconsistent
          in any respect with paragraph (a) of Section 2 of this Agreement, or
          any other action by the Company that results in a diminution in the
          Executive's position, authority, duties or responsibilities, unless,
          in the case of the first such assignment or action to occur, it is
          remedied by the Company promptly after receipt of notice thereof from
          the Executive;

               B.   any failure by the Company to comply with any provision of
          Section 3 of this Agreement, unless, in the case of the first such
          failure to occur, it is remedied by the Company promptly after receipt
          of notice thereof from the Executive;

               C.   any requirement by the Company that the Executive's services
          be rendered primarily at a location or locations other than that
          provided for in Section 2(c) of this Agreement;

               D.   any purported termination of the Executive's employment by
          the Company for a reason or in a manner not expressly permitted by
          this Agreement; or

               E.   any failure by the Company to comply with Section 10(c) of
          this Agreement.

               (ii) A termination of employment by the Executive for Good Reason
shall be effectuated by giving the Company written notice ("Notice of
Termination for Good Reason") of the termination, setting forth in reasonable
detail the specific conduct of the Company that constitutes Good Reason and the
specific provision(s) of this Agreement on which the Executive relies. A
termination of employment by the Executive for Good Reason shall be effective on
the fifth business day following the date when the Notice of Termination for
Good


                                      -7-
<PAGE>   8


Reason is given, unless the notice sets forth a later date (which date shall in
no event be later than 30 days after the notice is given).

               (iii) A termination of the Executive's employment by the
Executive without Good Reason shall be effected by giving the Company written
notice of the termination.

          (d)  NO WAIVER. The failure to set forth any fact or circumstance in a
Notice of Termination for Cause or a Notice of Termination for Good Reason shall
not constitute a waiver of the right to assert, and shall not preclude the party
giving notice from asserting, such fact or circumstance in an attempt to enforce
any right under or provision of this Agreement.

          (e)  DATE OF TERMINATION. The "Date of Termination" means the date of
the Executive's death, the date on which the termination of the Executive's
employment by the Company for Disability, for Cause or without Cause or by the
Executive for Good Reason is effective, or the date on which the Executive gives
the Company notice of a termination of employment without Good Reason, as the
case may be.

     5.   OBLIGATIONS OF THE COMPANY UPON TERMINATION. (a) OTHER THAN FOR CAUSE,
DEATH OR DISABILITY; GOOD REASON. If, during the Employment Period, the Company
terminates the Executive's employment, other than for Cause or Disability, or
the Executive terminates employment for Good Reason, the Company shall pay the
amounts described in subparagraph (i) below to the Executive in a lump sum in
cash within 30 days after the Date of Termination and shall provide the benefits
described in subparagraph (ii). The payments provided pursuant to this Section
5(a) are intended as liquidated damages for a termination of the Executive's
employment by the Company other than for Cause or Disability or for the actions
of


                                      -8-
<PAGE>   9


the Company leading to a termination of the Executive's employment by the
Executive for Good Reason, and shall be the sole and exclusive remedy therefor.

               (i)  The amounts to be paid in a lump sum as described above are:

                    A.   The sum of (1) any portion of the Executive's Base
               Salary through the Date of Termination that has not yet been
               paid; (2) the target Annual Bonus for the fiscal year in which
               the Date of Termination occurs, multiplied by a fraction, the
               numerator of which is the number of days in such fiscal year
               through the Date of Termination, and the denominator of which is
               365; (3) any earned but unpaid Annual Bonuses for prior fiscal
               years; (4) any reimbursements to which the Executive is entitled
               under Section 3(e) for expenses that were incurred before the
               Date of Termination but have not yet been paid, and (5) pay in
               lieu of accrued but unused vacation (such sum, the "Accrued
               Obligations"); and

                    B.   Severance pay equal to 2.5 times the sum of (1) the
               Base Salary at the annual rate as in effect immediately before
               the Date of Termination (but ignoring any decrease thereof that
               served as the basis of a termination by the executive for Good
               Reason) plus (2) the Guaranteed Bonus.

                    (ii) The benefits to be continued as described above are
benefits to the Executive and/or the Executive's family at least as favorable as
those that would have been provided to them under clause (ii) of Section 3(d) of
this Agreement if the Executive's employment had continued for 30 months
following the Date of Termination; PROVIDED, that during any period when the
Executive is eligible to receive such benefits under another employer-provided
plan, the benefits provided by the Company under this subparagraph shall be
secondary to those provided under such other plan.

               (b)  DEATH OR DISABILITY. If the Executive's employment is
terminated by reason of the Executive's death or Disability during the
Employment Period, the Company


                                      -9-
<PAGE>   10


shall pay the Accrued Obligations to the Executive or the Executive's estate or
legal representative, as applicable, in a lump sum in cash within 30 days after
the Date of Termination, and the Company shall have no further obligations under
this Agreement.

               (c)  CAUSE; OTHER THAN FOR GOOD REASON. If the Executive's
employment is terminated by the Company for Cause during the Employment Period,
or if the Executive voluntarily terminates employment during the Employment
Period, other than for Good Reason, the Company shall pay the Executive the Base
Salary through the Date of Termination and pay in lieu of any accrued but unused
vacation, in each case to the extent not yet paid, and the Company shall have no
further obligations under this Agreement.

     6.   NON-EXCLUSIVITY OF RIGHTS. Nothing in this Agreement shall prevent or
limit the Executive's continuing or future participation in any plan, program,
policy or practice provided by the Company for which the Executive may qualify,
nor, subject to Section 10(f), shall anything in this Agreement limit or
otherwise affect such rights as the Executive may have under any contract or
agreement with the Company. Vested benefits and other amounts that the Executive
is otherwise entitled to receive under any plan, policy, practice or program of,
or any contract or agreement with, the Company on or after the Date of
Termination shall be payable in accordance with such plan, policy, practice,
program, contract or agreement, as the case may be, except as explicitly
modified by this Agreement. Notwithstanding the foregoing, the Executive shall
not be entitled to receive pay or benefits under the Company's Salary
Continuation Plan or any other severance plan, policy, practice or program of
the Company.

     7.   FULL SETTLEMENT. The Company's obligation to make the payments
provided for in, and otherwise to perform its obligations under, this Agreement
shall not be


                                      -10-
<PAGE>   11


affected by any set-off, counterclaim, recoupment, defense or other claim, right
or action that the Company may have against the Executive or others. In no event
shall the Executive be obligated to seek other employment or take any other
action by way of mitigation of the amounts payable to the Executive under any of
the provisions of this Agreement and, except as specifically provided in
subparagraph (ii) of Section 5(a), such amounts shall not be reduced, regardless
of whether the Executive obtains other employment.

     8.   CONFIDENTIAL INFORMATION; NONCOMPETITION. (a) The Executive shall hold
in a fiduciary capacity for the benefit of the Company all secret or
confidential information, knowledge or data relating to the Company or any of
its affiliated companies and their respective businesses that the Executive
obtains during the Executive's employment by the Company or any of its
affiliated companies and that is not public knowledge (other than as a result of
the Executive's violation of this Section 8(a)) ("Confidential Information").
The Executive shall not communicate, divulge or disseminate Confidential
Information at any time during or after the Executive's employment with the
Company, except with the prior written consent of the Company or as otherwise
required by law or legal process. In no event shall an asserted violation of the
provisions of this Section 8(a) constitute a basis for deferring or withholding
any amounts otherwise payable to the Executive under this Agreement.

          (b)  During the Noncompetition Period (as defined below), the
Executive shall not, without the prior written consent of the Board, engage in
or become associated with a Competitive Activity. For purposes of this Section
8(b): (i) the "Noncompetition Period" means the period of 30 months following
the termination of the Executive's employment with the Company for any reason;
(ii) a "Competitive Activity" means any business or other endeavor the principal
activity of which is investment management for a


                                      -11-
<PAGE>   12


fee in the United States of America; and (iii) the Executive shall be considered
to have become "associated with a Competitive Activity" if he becomes directly
or indirectly involved as an owner, employee, officer, director, independent
contractor, agent, partner, advisor, or in any other capacity calling for the
rendition of the Executive's personal services, with any individual,
partnership, corporation or other organization that is engaged in a Competitive
Activity. Notwithstanding the foregoing, the Executive may make and retain
investments during the Employment Period in not more than five percent of the
equity of any entity engaged in a Competitive Activity, if such equity is listed
on a national securities exchange or regularly traded in an over-the-counter
market. In no event shall an asserted violation of the provisions of this
Section 8(b) constitute a basis for deferring or withholding any amounts
otherwise payable to the Executive under this Agreement.

     9.   SUCCESSORS. (a) This Agreement is personal to the Executive and,
without the prior written consent of the Company, shall not be assignable by the
Executive otherwise than by will or the laws of descent and distribution. This
Agreement shall inure to the benefit of and be enforceable by the Executive's
legal representatives.

          (b)  This Agreement shall inure to the benefit of and be binding upon
the Company and its successors and assigns.

          (c)  The Company shall require any successor (whether direct or
indirect, by purchase, merger, consolidation or otherwise) to all or
substantially all of the business and/or assets of the Company expressly to
assume and agree to perform this Agreement in the same manner and to the same
extent that the Company would have been required to perform it if no such
succession had taken place. As used in this Agreement, "Company" shall


                                      -12-
<PAGE>   13


mean both the Company as defined above and any such successor that assumes and
agrees to perform this Agreement, by operation of law or otherwise.

     10.  MISCELLANEOUS. (a) This Agreement shall be governed by, and construed
in accordance with, the laws of the Commonwealth of Massachusetts, without
reference to principles of conflict of laws. The captions of this Agreement are
not part of the provisions hereof and shall have no force or effect. This
Agreement may not be amended or modified except by a written agreement executed
by the parties hereto or their respective successors and legal representatives.

          (b)  All notices and other communications under this Agreement shall
be in writing and shall be given by hand delivery to the other party or by
registered or certified mail, return receipt requested, postage prepaid,
addressed as follows:If to the Executive:

               James F. Orr III
               74 Waites landing
               Falmouth, ME

               With a copy to

               William Schmidt
               Hale and Dorr LLP
               60 State Street
               Boston, MA 02109

               If to the Company:

               United Asset Management Corporation
               One International Place
               Boston, Massachusetts 02110
               Attention: General Counsel


                                      -13-
<PAGE>   14


or to such other address as either party furnishes to the other in writing in
accordance with this Section 10(b). Notices and communications shall be
effective when actually received by the addressee.

          (c)  The invalidity or unenforceability of any provision of this
Agreement shall not affect the validity or enforceability of any other provision
of this Agreement. If any provision of this Agreement shall be held invalid or
unenforceable in part, the remaining portion of such provision, together with
all other provisions of this Agreement, shall remain valid and enforceable and
continue in full force and effect to the fullest extent consistent with law.

          (d)  Notwithstanding any other provision of this Agreement, the
Company may withhold from amounts payable under this Agreement all federal,
state, local and foreign taxes that are required to be withheld by applicable
laws or regulations.

          (e)  The Executive's or the Company's failure to insist upon strict
compliance with any provision of, or to assert any right under, this Agreement
(including, without limitation, the right of the Executive to terminate
employment for Good Reason pursuant to Section 4(c) of this Agreement) shall not
be deemed to be a waiver of such provision or right or of any other provision of
or right under this Agreement.

          (f)  The Executive and the Company acknowledge that this Agreement
supersedes any other agreement between them concerning the subject matter
hereof, other than the Change of Control Agreement between the Company and the
Executive of even date herewith.


                                      -14-
<PAGE>   15


          (g)  This Agreement may be executed in several counterparts, each of
which shall be deemed an original, and said counterparts shall constitute but
one and the same instrument.

          IN WITNESS WHEREOF, the Executive has hereunto set the Executive's
hand and, pursuant to the authorization of its Board of Directors, the Company
has caused this Agreement to be executed in its name on its behalf, all as of
the day and year first above written.



                                       /s/ James F. Orr III
                                       -----------------------------------
                                       James F. Orr III


                                       UNITED ASSET MANAGEMENT
                                        CORPORATION


                                       By: /s/ John A. Shane
                                       -----------------------------------
                                       John A. Shane, duly authorized


                                      -15-