<PAGE>

                                                                 EXHIBIT 10-H-4
                                                                 --------------


                        FIRST AMENDMENT TO HOMI B. PATEL
                  SUPPLEMENTAL BENEFIT COMPENSATION AGREEMENT
                  -------------------------------------------


       The Homi B. Patel Supplemental Benefit Compensation Agreement effective
as of December 23, 1999 is hereby amended as of the date hereof, unless
otherwise specifically provided, as follows:

  I.   Paragraph 2(b) is amended to read as follows:

       The "Present Value" of Employee's After-Tax Supplemental Benefits shall
       be determined using the interest rate then in effect by reference to the
       IRS 4-Year Weighted Average Interest Rate and the 1983 Group Annuity
       Mortality Table (the "1983 GAM"), as such 1983 GAM may be amended,
       revised or updated from time to time.

  II.  Paragraph 3 is amended as follows:

       A. Paragraph 3(ii) is amended to read as follows:

          (ii) The death of the Employee or the discharge by the Company of
               Employee from the employment of Company or a subsidiary of
               Company without Cause or a termination by Employee for Good
               Reason (Cause and Good Reason shall be defined as defined in the
               most recent Employment Agreement entered into by the Employee and
               the Company ("Employment Agreement");

       B. The first full sentence of the flush paragraph immediately following
          Paragraph 3(ii) is amended to read as follows:

               In the event of a termination of Employee's employment initiated
               by the Employee not for Good Reason (as defined in the
               "Employment Agreement") prior to the expiration of the five-year
               period, the Company shall continue to make the deposits in the
               form and at the times described in the first sentence of this
               paragraph 3 during the remainder of the five-year period.
<PAGE>

       C. The last sentence of Paragraph 3 is amended to read as follows:

               Notwithstanding the foregoing, in the event that (i) the Employee
               is discharged by the Company for Cause as defined in the
               Employment Agreement, (ii) the Employee willfully, intentionally
               and materially breaches the confidentiality and non-disparagement
               provisions of paragraph 7 of the Employment Agreement, or (iii)
               the Employee withdraws funds from the account prior to the
               termination of employment with the Company, no further deposit
               shall be made to the Employee's segregated account by the Company
               under this Agreement.

  III. Except as otherwise provided herein, the Supplemental Benefit
       Compensation Agreement shall remain in full force and effect.

       IN WITNESS WHEREOF, the Company and Employee have executed this Amendment
as of the 27th day of November, 2000.


                                 HARTMARX CORPORATION


                                 By: /s/ GLENN R. MORGAN
                                     --------------------------------
                                     Glenn R. Morgan, Executive
                                     Vice President & Chief Financial
                                     Officer


                                      /s/ HOMI B. PATEL
                                      -------------------------------
                                           HOMI B. PATEL

                                       2