<PAGE>
 
                                                                  EXHIBIT 10-F-4
                              EMPLOYMENT AGREEMENT
                              --------------------


     This Agreement entered into as of the 1st day of August, 1996, by and
between HARTMARX CORPORATION, a Delaware corporation ("Company"), and GLENN R.
MORGAN ("Executive").


                                WITNESSETH THAT:
                                ----------------

     WHEREAS, the Executive is a party to an Employment Agreement with the
Company dated August 31, 1995, which also incorporated provisions relating to
severance following a Change in Control of the Company(the "Prior Agreement");

     WHEREAS, the parties hereto desire to enter into this Agreement pertaining
to the terms of Executive's employment by the Company and to provide for
termination of the Prior Agreement; and

     WHEREAS, the Company and the Executive intend to enter into a Severance
Agreement ("Severance Agreement"), coincident herewith; and

     WHEREAS, upon the execution of this Agreement by the Company and Executive
the terms and conditions of this Agreement shall control and govern the
employment relationship between the Company and Executive.

     NOW, THEREFORE, in consideration of the mutual covenants and agreements set
forth below, it is hereby covenanted and agreed by the parties hereto as
follows:

     1.   Agreement Period.  The Company hereby employs Executive and Executive
hereby agrees to remain in the employ of the Company for an employment term
("Agreement Period") beginning on August 1, 1996, and continuing in effect
through December 31, 1998; provided, however, that the Agreement Period shall
automatically be extended for two additional years unless, not later than July
15, 1998, the Company or the Executive shall have given notice not to extend
this Agreement; and further; provided, however, that if the Executive's
employment is terminated following a Change in Control (as defined in the
Severance Agreement), the Agreement Period shall terminate upon such termination
of employment and the Executive's rights with respect to such termination of
employment shall be governed by the provisions of the Severance Agreement. While
Executive is employed by the Company during the Agreement Period, the Company
shall use its
<PAGE>
 
best efforts to have the Board of Directors elect Executive to the offices of
Executive Vice President and Chief Financial Officer of the Company.  The Prior
Agreement shall terminate as of July 31, 1996.

     2.   Performance of Duties.  While employed by the Company Executive shall
devote all his full working time, attention and energies during normal business
hours to the performance of his duties for the Company and its subsidiaries and
shall perform his duties faithfully and efficiently, subject to the direction of
the Company Board of Directors; provided, however, that Executive may become a
director of other corporations and engage in charitable, civic, professional
and other similar pursuits to the extent that such activities do not interfere
with his duties hereunder.

     3.   Compensation.  As compensation for the performance by Executive of his
obligations hereunder:

          (a)  Base Salary.  During the Agreement Period the Company shall pay
Executive an annual base salary of not less than $200,000 ("Base Salary").  Base
Salary shall be paid in accordance with the Company's customary payroll
practices.  Base Salary may be increased at the discretion of the Compensation
and Stock Option Committee of the Company Board of Directors (the "Committee")
and once so increased shall not thereafter be decreased, except for across-the-
board reductions similarly affecting all executives of the Company.

          (b) Management Incentive Plan.  Executive shall participate in the
Company Management Incentive Plan (the "MIP") and/or any successor plan.

          (c) Long Term Incentive Plan. Executive shall participate in any long-
term incentive plan maintained by the Company ("LTI Plan") for such period of
time as such plan may be in effect.

          (d)  Participation in Benefit Plans.  During the Agreement Period the
Executive shall be eligible to participate in all savings, retirement and
welfare benefit plans and programs now or hereafter applicable to any other
senior executives of the Company on a basis no less favorable than is made
available to any other senior executive of the Company.

                                       2
<PAGE>
 
          (e)  Perquisites.  During the Agreement Period, the Company shall make
available to the Executive all perquisites that are made available to Company's
senior executives.

      4.  Termination.  The Executive's employment hereunder may be terminated
under the following circumstances:

          (a) Death. The Executive's employment hereunder shall terminate upon
his death.

          (b)  Disability.  The Company may terminate the Executive's employment
hereunder for "Disability".  Any question as to the existence of the Disability
shall be determined in accordance with the Company's disability plan.

          (c)  Cause.  The Company may terminate the Executive's employment
hereunder for Cause.  For purposes of this Agreement, the Company shall have
"Cause" to terminate the Executive's employment hereunder upon the Executive's:

     (i)       conviction for the commission of a felony; or

     (ii)      willful failure to substantially perform his duties hereunder; or

     (iii)     willful or grossly negligent conduct that is demonstrably and
               materially injurious to the Company or its affiliates; or

     (iv)      material breach of this Agreement, including but not limited to
               Section 7 hereof.

          Notwithstanding the foregoing, no event shall constitute "Cause"
unless the Company shall have notified Executive in writing of the conduct
allegedly constituting Cause and the Executive shall have failed to correct such
conduct within thirty (30) days of the date of his receipt of such written
notice from the Company.

          (d)  Good Reason.  The Executive may terminate his employment
hereunder for Good Reason.  Good Reason shall mean the occurrence (without the
Executive's written consent) of any one of the following acts by the Company, or
failures by the Company to act:

                                       3
<PAGE>
 
     (i)       failure of the Board of Directors of the Company to elect
               Executive to the office(s) described in Section 1 hereof; or

     (ii)      [Intentionally omitted]

     (iii)     any change in (i) the provisions of the Company's bylaws
               describing, or (ii) the relative duties and responsibilities of,
               the office of Executive Vice President and Chief Financial
               Officer; or

     (iv)      the assignment to Executive of any duties inconsistent with
               Executive's status as Executive Vice President and Chief
               Financial Officer or a substantial adverse alteration in the
               nature or status of Executive's responsibilities; or

     (v)       any reduction by the Company in Executive's Base Salary, except
               for across-the-board salary reductions similarly affecting all
               executives of the Company; or

     (vi)      the failure by the Company to pay to Executive any portion of
               Executive's current compensation, or to pay to Executive any
               portion of an installment of deferred compensation under any
               deferred compensation program of the Company, within seven (7)
               days of the date such compensation is due; or

     (vii)     the taking of any action by the Company which directly or
               indirectly causes Executive to cease to be eligible to
               participate in all savings, retirement and welfare benefit plans
               and programs applicable to any other senior executives of the
               Company on a basis no less favorable than is made available to
               any other senior executive of the Company; the failure of the
               Company to make available to the Executive all perquisites that
               are made available to Company's senior executives; the failure by
               the Company to calculate Executive's annual bonus compensation,
               if any, using at least the valuation and number of accountability
               points used to determine the bonus opportunity in any previous
               year during the Agreement Period for any corporate officer
               position held by Executive; or the failure by the Company to
               provide Executive with the number of paid vacation days to which

                                       4
<PAGE>
 
               Executive may then be entitled; except (as to all of the
               foregoing) for changes (including termination) in such benefits
               and/or policies similarly affecting all executives of the
               Company; or

     (viii)    the relocation of the Executive's principal place of employment
               to a location more than 50 miles from the Executive's principal
               place of employment as of the date hereof or the Company's
               requiring the Executive to be based anywhere other than such
               principal place of employment (or permitted relocation thereof)
               except for required travel on the Company's business to an extent
               substantially consistent with the Executive's present business
               travel obligations; or

     (ix)      the giving notice by the Company of its decision not to extend
               this Agreement, in accordance with Section 1; or any purported
               termination of the Executive's employment by the Company other
               than in accordance with this Agreement.

          Notwithstanding the foregoing, no event shall constitute "Good
Reason" unless the Executive shall have notified the Company in writing of the
conduct allegedly constituting Good Reason and the Company shall have failed to
correct such conduct within thirty (30) days of the date of its receipt of such
written notice from the Executive.

     5.   Termination Procedure.
          --------------------- 

          (a)  Notice of Termination.  Any termination of the Executive's
employment by the Company or by the Executive (other than termination pursuant
to Section 4(a) hereof) shall be communicated by written Notice of Termination
to the other party hereto in accordance with Section 9.  For purposes of this
Agreement, a "Notice of  Termination" shall mean a notice which shall indicate
the specific provision in this Agreement relied upon and shall identify in
reasonable detail the reason for termination of the Executive's employment under
the provision so indicated.

          (b)  Date of Termination.  "Date of Termination" shall mean (i) if the
Executive's employment is terminated by his death, the date of his death, (ii)
if the Executive's employment is terminated pursuant to Section 4(b) above, the
date thirty (30) days after Notice of Termination (provided that the Executive
shall not have returned to the performance of his duties on

                                       5
<PAGE>
 
a permanent full-time basis during such thirty (30) day period), (iii) if the
Executive's employment is terminated pursuant to Section 4(c) or 4(d) above, the
date thirty (30) days after Notice of Termination and (iv) if the Executive's
employment is terminated for any other reason, the date specified in the Notice
of Termination which shall be not more than 30 days from the date of such
Notice.

      6.  Compensation Upon Termination.
          ----------------------------- 

          (a)   Termination due to Death or Disability.  If the Executive's
employment is terminated by his death or Disability, except as provided in
Section 6(d) below, the Company shall have no further obligations to Executive
under this Agreement.

          (b)  Termination By Company without Cause or By Executive for Good
Reason.  Upon termination of Executive's employment hereunder during the
Agreement Period by the Company without Cause or by Executive for Good Reason
hereunder, then, in lieu of any further salary, bonus, or LTI Plan payments for
periods subsequent to the Date of Termination and in lieu of any severance
benefit otherwise payable to the Executive:

               (i) The Company shall continue to pay to Executive Base Salary as
          of the Date of Termination (without giving effect to any decrease
          therein which constitutes the basis, or one of the bases, upon which
          the Notice of Termination is based), for a period of twenty-four (24)
          months (the "Severance Period"), payable semi-monthly or more
          frequently, in arrears, commencing on the Date of Termination;
          provided, however, that if a Change in Control occurs during the
          Severance Period, the Company shall pay the Executive by no later than
          5 days following the Change in Control a lump sum in cash equal to the
          sum of the remaining payments that would have been payable to the
          Executive hereunder had no Change in Control occurred, and payments
          hereunder shall terminate.

               (ii) The Company shall pay the Executive a lump sum in cash,
          within 10 days of the Date of Termination, equal to the sum of (A) any
          unpaid incentive compensation (including the cash value, determined
          without regard to any restrictions on the sale thereof, of restricted
          stock) allocated or awarded to Executive under the MIP with respect to
          any fiscal year ending prior to the year in which the Date of
          Termination occurs.

                                       6
<PAGE>
 
               (iii) The Company shall pay the Executive as bonuses any amount
          which would have been payable to Executive under the MIP for the full
          fiscal year in which Executive was terminated and for the full fiscal
          year following such year. Such payments will be made within five days
          of the date on which MIP payments are made to other MIP participants
          after the close of each fiscal year and will include the cash value,
          determined without regard to any restrictions on the sale thereof, of
          restricted stock. If a Change in Control occurs prior to either of
          these two payments, then, in lieu of any further bonus payments, the
          Company shall pay Executive, no later than 10 days following the
          Change in Control, a lump sum in cash calculated at the Step-1 level
          for the then current fiscal year for each MIP bonus payment not yet
          made.

               (iv) The Company shall pay the Executive a lump sum in cash,
          within 10 days of the Date of Termination, equal to the sum of (A) any
          unpaid incentive compensation (including the cash value, determined
          without regard to any restrictions on the sale thereof, of
          restricted stock) allocated or awarded to Executive under the LTI Plan
          with respect to any performance period ending prior to the Date of
          Termination; plus (B) a pro rata portion of the aggregate value of all
          contingent incentive compensation (including the cash value,
          determined without regard to any restrictions on the sale thereof, of
          restricted stock) awards to Executive with respect to any performance
          periods under the LTI Plan which are not completed as of the Date of
          Termination, calculated based on the assumption that the Company's
          results from the beginning of such performance period(s) to the Date
          of Termination would continue at the same rate until the originally
          intended completion date(s) of such performance period(s). The amount
          set forth in item (B) above shall be payable to Executive regardless
          of whether the Company actually achieves the performance level upon
          which the calculation of such amount is based.

               (v) During the Severance Period the Company shall arrange to
          provide the Executive with life, disability, accident and health
          insurance benefits ("Welfare Benefits") substantially similar in all
          material respects to those which the Executive is receiving
          immediately

                                       7
<PAGE>
 
          prior to the Date of Termination (without giving effect to any
          decrease therein which constitutes the basis, or one of the bases,
          upon which the Notice of Termination is based). If the Executive
          receives, or becomes eligible to receive, Welfare Benefits from
          another source, then the Welfare Benefits otherwise receivable by the
          Executive pursuant to this Section 6(b)(v) shall be reduced to the
          extent of such other Welfare Benefits received by, or made available
          to, the Executive during the Severance Period (and any such Welfare
          Benefits received by or made available to the Executive shall be
          reported to the Company by the Executive). Nothing herein shall be
          deemed to limit Executive's rights, if any, to thereafter participate
          in any retiree medical plan then in effect.

               (vi) During the Severance Period, the Company shall arrange to
          provide the Executive with such material perquisites as are provided
          to the Executive immediately prior to the Date of Termination
          (without giving effect to any decrease therein which constitutes the
          basis, or one of the bases, upon which the Notice of Termination is
          based).

               (vii) Effective as of the Date of Termination, all stock options
          (whether or not then fully exercisable) granted to Executive under
          any of the Company's stock option plans prior to the Date of
          Termination shall become immediately exercisable and Executive shall
          be entitled to exercise any or all of such options at any time prior
          to the respective expiration dates of such options as set forth in the
          grant document evidencing same.
 
               (viii) Effective as of the Date of Termination, all restricted
          stock granted to Executive prior to the date Executive's employment
          with the Company is terminated shall become fully vested and all
          restrictions thereon shall lapse.

               (ix) At the time and in the manner such benefits are otherwise
          payable, the Executive shall receive payment of the incremental
          qualified and supplemental defined benefit pension benefits Executive
          would have earned had Executive's employment continued during the
          Severance Period, had he received credit for service for the Severance
          Period for all purposes under the

                                       8
<PAGE>
 
          applicable plans, and had the Executive received compensation during
          the Severance Period of salary, at the annual rate equal to the
          Executive's Base Salary in effect immediately prior to the Date of
          Termination (without giving effect to any decrease therein which
          constitutes the basis, or one of the bases, upon which the Notice of
          Termination is based), and MIP bonuses actually paid during the
          Severance Period. Anything in the applicable plan to the contrary
          notwithstanding, the Executive's benefit (as increased hereunder)
          under any supplemental defined benefit plan maintained by the Company
          ("SERP Benefit") shall be payable to Executive in the form of a single
          life annuity, monthly, at the same times and for the same duration as
          Executive's benefit payments from Hartmarx Retirement Income Plan
          ("RIP"); provided, however, that if the Executive elects to receive
          payment of RIP benefits in the form of a joint and survivor annuity,
          the SERP Benefit shall be paid in the form of a joint and survivor
          annuity, calculated so that the value of such SERP Benefit is the
          actuarial equivalent of the SERP Benefit payable in the form of a
          single life annuity. Notwithstanding the foregoing, in the event of a
          Change in Control then the Executive's SERP Benefit as of such date
          shall be paid to him in a lump sum within 10 days of the Change in
          Control or the Date of Termination, as applicable.

          (c)  Termination by the Company for Cause or By Executive Other than
for Good Reason.  If the Executive's employment shall be terminated by the
Company for Cause or by the Executive other than for Good Reason, then, subject
to Section 6(d) below, the Company shall have no obligations to Executive under
this Agreement.

          (d)  Additional Payments.  Following any termination of Executive's
employment, the Company shall pay the Executive all unpaid amounts, if any, to
which the Executive is entitled as of the Date of Termination under any
compensation plan or program of the Company, at the time such payments are due.
In addition, within ten days of the Date of Termination, the Company shall pay
the Executive, or his legal representative or estate, as applicable, any
amounts accrued but not paid pursuant to Sections 3(a), 3(b) and 3(c) in respect
of periods ending prior to the Date of Termination.

     7.   Confidentiality; Nondisparagement.  The Executive shall hold in a
fiduciary capacity for the benefit of the Company all

                                       9
<PAGE>
 
trade secrets, confidential information, and knowledge or data relating to the
Company and its affiliates which shall have been obtained by the Executive
during the Executive's employment by the Company and which shall not have been
or now or hereafter have become public knowledge (other than by acts by the
Executive or representatives of the Executive in violation of this Agreement).
The Executive shall not, without the prior written consent of the Company, or
as may otherwise be required by law or legal process, communicate or divulge any
such trade secrets, information, knowledge or data to anyone other than the
Company and those designated by the Company.  In addition, the Executive shall
not disparage, discredit or otherwise publicly criticize the Company or its
affiliates or engage in any act, directly or indirectly, for purposes of
disparaging, ridiculing or bringing scorn upon the Company, any affiliate
thereof, or any of their respective officers, directors, businesses, tradenames
or trademarks.  In the event of a breach or threatened breach of this Section
7, the Executive agrees that the Company shall be entitled to injunctive relief
in a court of appropriate jurisdiction to remedy any such breach or threatened
breach, the Executive acknowledging that damages would be inadequate and
insufficient. Any termination of the Executive's employment, Agreement Period or
of this Agreement shall have no effect on the continuing operation of this
Section 7.


     8.   Amendment.  This Agreement may be amended in writing by mutual
agreement of the parties without the consent of any other person and, during the
life of Executive, no person, other than the parties hereto, shall have any
rights under or interest in this Agreement or the subject matter hereof.


     9.   Notice.  Any notice required or permitted to be given under this
Agreement shall be sufficient if in writing and, if sent by registered mail, to
the Company at its principal executive offices, to the attention of its Chief
Executive Officer, or to Executive at the last address filed by him in writing
with the Committee, as the case may be.
  

     10.  Nonalienation.  The interests of Executive under this Agreement are
not subject to the claims of his creditors, other than the Company and its
subsidiaries, and may not otherwise be voluntarily or involuntarily assigned,
alienated or encumbered.

                                       10
<PAGE>
 
     11.  Successors.  This Agreement shall be binding upon, and inure to the
benefit of, the Company and its successors and assigns and upon any person
acquiring, whether by merger, consolidation, purchase of assets or otherwise,
all or substantially all of the Company's assets and business.
 

     12.  Severability.  If, for any reason, any provision of this Agreement is
held invalid, such invalidity shall not affect any other provision of this
Agreement not held so invalid, and each such other provision shall to the full
extent consistent with law continue in full force and effect.  If any provision
of this Agreement shall be held invalid in part, such invalidity shall in no way
affect the rest of such provision not held so invalid, and the rest of such
provision, together with all other provisions of this Agreement, shall to the
full extent consistent with law continue in full force and effect.


     13.  Applicable Law.  The provisions of this Agreement shall be construed
in accordance with the laws of the State of Illinois.

 
     14.  Counterpart.  The Agreement may be executed in two or more
counterparts, any one of which shall be deemed the original without reference to
the others.


     15.  Attorney's Fees.  Company shall reimburse Executive for all reasonable
legal fees and costs and other fees and expenses which Executive may incur in
respect of any dispute or controversy arising under or in connection with this
Agreement; provided, however, that the Company shall not reimburse such legal
fees and costs and other fees and expenses if the fact finder determines that
the action brought by the Executive was frivolous.


     16.  Beneficiaries.  If Executive should die while any amount is payable to
him hereunder, such amount shall be paid to Executive's devisee, legatee or
other designee or, if there is no such designee, to Executive's estate.


     17.  Arbitration.  Any dispute or controversy arising under or in
connection with this Agreement shall be settled exclusively by arbitration,
conducted before a panel of three arbitrators in

                                       11
<PAGE>
 
Chicago, Illinois in accordance with the rules of the American Arbitration
Association then in effect.  The Company and the Executive shall each be
entitled to select one arbitrator, with the two selected arbitrators choosing
the third arbitrator. Judgment may be entered on the arbitrators' award in any
court having jurisdiction.  The expense of such arbitration shall be borne by
the Company.

     18.  Mitigation.  Executive shall not be required to mitigate damages or
the amount of any payment provided for under this Agreement by seeking (and,
except as provided in Section 6(b)(v), no payment otherwise required hereunder
shall be reduced on account of) other employment.


     IN WITNESS WHEREOF, Executive has hereunto set his hand, and the Company
has caused these presents to be executed in its name and on its behalf, and its
corporate seal to be hereunto affixed and attested by its Secretary, all as of
the day and year first above written.


                                         /s/ Glenn R. Morgan
                                         ------------------------------------
                                         GLENN R. MORGAN


Attest:                                  HARTMARX CORPORATION


/s/Mary D. Allen                         By: /s/ E. O. Hand
----------------------                       ---------------------------------
Mary D. Allen                                E.O. Hand, Chairman and
Secretary                                    Chief Executive Officer

                                       12