<PAGE>
 
                                                                   EXHIBIT 4-C-4

                                                                       EXECUTION


                             HARTMARX CORPORATION

                     FOURTH AMENDMENT TO CREDIT AGREEMENT


     This FOURTH AMENDMENT TO CREDIT AGREEMENT (this "AMENDMENT") is dated as of
November 30, 1995 and entered into by and among HARTMARX CORPORATION, a Delaware
corporation ("BORROWER"), the LENDERS LISTED ON THE SIGNATURE PAGES HEREOF (each
individually referred to as a "LENDER" and collectively as "LENDERS"), GENERAL
ELECTRIC CAPITAL CORPORATION as Managing Agent and Collateral Agent for Lenders
("MANAGING AGENT") and THE BANK OF NEW YORK and BANKAMERICA BUSINESS CREDIT,
INC. as co-agents ("CO-AGENTS") and, for purposes of Section 3 hereof, the
GUARANTORS IDENTIFIED ON THE SIGNATURE PAGES HEREOF, (collectively the
"GUARANTORS"), and is made with reference to that certain Credit Agreement dated
as of March 23, 1994, among Borrower, Lenders and Agent, as amended by the First
Amendment to Credit Agreement dated as of August 26, 1994 among Borrower,
Lenders and Agent, by the Second Amendment to Credit Agreement dated as of March
20, 1995 among Borrower, Lenders and Agent and by the Third Amendment to Credit
Agreement dated as of June 30, 1995 among Borrower, Lenders, Managing Agent, 
Co-Agents and consented to by Guarantors (the "CREDIT AGREEMENT"; capitalized
terms used herein without definition shall have the same meanings herein as set
forth in the Credit Agreement). Unless otherwise indicated, Section and
subsection references contained herein shall be to the corresponding Sections
and subsections of the Credit Agreement.


                                R E C I T A L S
                                - - - - - - - -

     WHEREAS, Borrower and Lenders desire to amend the Credit Agreement as set
forth below;

     NOW, THEREFORE, in consideration of the premises and the agreements,
provisions and covenants herein contained, the parties hereto agree as follows:
<PAGE>
 
                                   SECTION 1
                                   AMENDMENT

1.1  AMENDMENTS TO THE CREDIT AGREEMENT.

     A.  Minimum Consolidated Debt Service Coverage Ratio. Section 7.6A of the
Credit Agreement is amended by deleting the ratio "1.40:1.00" immediately
adjacent to the reference to "Fourth Quarter 1995" set forth in the table
therein and by substituting the ratio "1.00:1.00" therefor.

     B.  Maximum Consolidated Leverage Ratio. Section 7.6B of the Credit
Agreement is amended by deleting the ratio "5.50:1.00" immediately adjacent to
the reference to "Fourth Quarter 1995" set forth in the table therein and by
substituting the ratio "6.50:1.00" therefor.


                                   SECTION 2
                   BORROWERS' REPRESENTATIONS AND WARRANTIES

     In order to induce Lenders to enter into this Amendment, Borrower
represents and warrants to Lenders that after giving effect to this Amendment in
the manner contemplated by Section 4.6 of this Amendment, each of the following
is true and correct:

     (a) no event has occurred and is continuing which constitutes an Event of
Default or Potential Event of Default;

     (b) the representations and warranties of Borrower and the other Credit
Parties contained in the Credit Agreement and the other Loan Documents are true
and correct on and as of the date hereof and as of the Effective Date (as
defined below) to the same extent as though made on and as of the date hereof
and as of the Effective Date except to the extent such representations and
warranties specifically relate to an earlier date, in which case they are true
and correct in all material respects as of such earlier date;

     (c) each of Borrower and the other Credit Parties has performed all
agreements on its part to be performed prior to the date hereof as set forth in
the Credit Agreement and the other Loan Documents;

     (d) each of Borrower and the Guarantors has all requisite corporate power
and authority to enter into this Amendment and to carry out the transactions
contemplated by, and perform its obligations under, the Credit Agreement;

     (e) the execution of this Amendment has been duly authorized by all
necessary corporate action on the part of Borrower and the Guarantors; and

                                       2
<PAGE>
 
     (f) the execution and delivery by Borrower and the Guarantors of this
Amendment does not and will not (i) violate any provision of any law or any
governmental rule or regulation applicable to Borrower, the Guarantors or any of
their respective Subsidiaries, the Certificate of Incorporation of Borrower, the
Guarantors or any of their respective Subsidiaries or any order, judgment or
decree of any court or other agency of government binding on Borrower, the
Guarantors or any of their respective Subsidiaries, (ii) conflict with, result
in a breach of or constitute (with due notice or lapse of time or both) a
default under any Contractual Obligation of Borrower, the Guarantors or any of
their respective Subsidiaries, (iii) result in or require the creation or
imposition of any Lien upon any of the properties or assets of Borrower, the
Guarantors or any of their respective Subsidiaries (other than any Liens created
under any of the Loan Documents in favor of Collateral Agent on behalf of
Lenders), or (iv) require any approval of stockholders or any approval or
consent of any Person under any Contractual Obligation of Borrower, the
Guarantors or any of their respective Subsidiaries.


                                   SECTION 3
                                   GUARANTORS

     Each of the Guarantors hereby consents to this Amendment and agrees that
each Loan Document to which it is a party shall continue in full force and
effect and shall be valid and enforceable and shall not be impaired or affected
by the execution of this Amendment and is hereby ratified and confirmed.


                                   SECTION 4
                                 MISCELLANEOUS

4.1  REFERENCES TO AND EFFECT ON THE CREDIT AGREEMENT AND OTHER LOAN DOCUMENTS.

     (i)   On and after the Effective Date, each reference in the Credit
Agreement to "this Agreement", "hereunder", "hereof", "herein" or words of like
import referring to the Credit Agreement, and each reference in the other Loan
Documents to the "Credit Agreement", "thereunder", "thereof" or words of like
import referring to the Credit Agreement, shall mean and be a reference to the
Credit Agreement as amended hereby;

     (ii)  Except as specifically amended by this Amendment, the Credit
Agreement and the other Loan Documents shall remain in full force and effect and
are hereby ratified and confirmed; and

     (iii) The execution, delivery and performance of this Amendment shall not,
except as expressly provided herein, constitute a waiver of any provision of, or
operate as a waiver of any right, power or remedy of any Agent or any Lender
under, the Credit Agreement or any of the other Loan Documents.

                                       3
<PAGE>
 
4.2  FEES AND EXPENSES.

     Borrower acknowledges that all costs, fees and expenses as described in
subsection 10.2 of the Credit Agreement incurred by Managing Agent and its
counsel with respect to this Amendment and the documents and transactions
contemplated hereby shall be for the account of Borrower.

4.3  HEADINGS.

     Section and subsection headings in this Amendment are included herein for
convenience of reference only and shall not constitute a part of this Amendment
for any other purpose or be given any substantive effect.

4.4  APPLICABLE LAW.

     THIS AMENDMENT SHALL BE GOVERNED BY, AND SHALL BE CONSTRUED AND ENFORCED IN
ACCORDANCE WITH, THE INTERNAL LAWS OF THE STATE OF NEW YORK, WITHOUT REGARD TO
COMMON LAW CONFLICTS OF LAWS PRINCIPLES.

4.5  COUNTERPARTS.

     This Amendment may be executed in any number of counterparts and by
different parties hereto in separate counterparts, each of which when so
executed and delivered shall be deemed an original, but all such counterparts
together shall constitute but one and the same instrument; signature pages may
be detached from multiple separate counterparts and attached to a single
counterpart so that all signature pages are physically attached to the same
document.

4.6  EFFECTIVENESS.

     This Amendment shall become effective (such date being the "EFFECTIVE
DATE") upon the execution of a counterpart hereof by Requisite Lenders, the
Borrower and the Guarantors and receipt by Borrower and Managing Agent of
written or telephone notification of such execution and authorization of
delivery thereof.

                                       4
<PAGE>
 
     IN WITNESS WHEREOF, the parties hereto have caused this Amendment to be
duly executed and delivered by their respective officers thereunto duly
authorized as of the date first written above.

                     BORROWER:

                     HARTMARX CORPORATION
 
                     By:
                        ----------------------------------------------------
                        Glenn R. Morgan
                        Executive Vice President and
                        Chief Financial Officer



                     GUARANTORS:


                     AMERICAN APPAREL BRANDS, INC.
                     ANNISTON SPORTSWEAR CORPORATION
                     BLLTWELL COMPANY, INC.
                     BRIAR, INC.
                     C.M. CLOTHING, INC.
                     C.M. OUTLET CORP.
                     CHICAGO TROUSER COMPANY, LTD.
                     COUNTRY MISS, INC.
                     COUNTRY MISS INTERNATIONAL LIMITED
                     COUNTRY SUBURBANS, INC.
                     DIRECT ROUTE MARKETING CORPORATION
                     E-TOWN SPORTSWEAR CORPORATION
                     FAIRWOOD-WELLS, INC.
                     GLENEAGLES, INC.
                     TAG APPAREL, INC. (FORMERLY KNOWN AS HENRY GRETHEL
                       APPAREL, INC.)
                     HANDMACHER FASHIONS FACTORY OUTLET, INC.
                     HANDMACHER-VOGEL, INC.
                     HARTMARX INTERNATIONAL, INC.
                     HART SCHAFFNER & MARX
                     HART SERVICES, INC.
                     THOS. HEATH CLOTHES, INC.
                     TAG LICENSING, INC. (FORMERLY KNOWN AS HGA 
                       LICENSING, INC.)
                     HICKEY-FREEMAN CO., INC.


                                      S-1
<PAGE>
 
                     HIGGINS, FRANK & HILL, INC.
                     NOVAPPAREL, INC. (FORMERLY KNOWN AS HMXUS, INC.)
                     HOOSIER FACTORIES, INCORPORATED
                     HSM UNIVERSITY, INC.
                     INTERCONTINENTAL APPAREL, INC.
                     INTERNATIONAL WOMEN'S APPAREL, INC.
                     JAYMAR-RUBY, INC.
                     JRSS, INC.
                     KUPPENHEIMER MEN'S CLOTHIERS DADEVILLE, INC.
                     MEN'S QUALITY BRANDS, INC.
                     NATIONAL CLOTHING COMPANY, INC.
                     106 REAL ESTATE CORP.
                     RECTOR SPORTSWEAR CORPORATION
                     ROBERTS INTERNATIONAL CORPORATION
                     SALHOLD, INC.
                     SEAFORD CLOTHING CO.
                     SOCIETY BRAND, LTD.
                     ROBERT SURREY, INC.
                     TAILORED TREND, INC.
                     THORNGATE UNIFORMS, INC.
                     TRADE FINANCE INTERNATIONAL LIMITED
                     UNIVERSAL DESIGN GROUP, LTD.
                     M. WILE & COMPANY, INC.
                     WINCHESTER CLOTHING COMPANY
                     YORKE SHIRT CORPORATION

                     By:
                        -----------------------------------------------------
                        Glenn R. Morgan
                        Vice President of each of the foregoing (except for
                        Country Miss International Limited) and a Director of
                        Country Miss International Limited

                                      S-2
<PAGE>
 
               LENDERS:

               GENERAL ELECTRIC CAPITAL CORPORATION,
               individually, as Managing Agent and as Collateral Agent

               By:
                  --------------------------------------------------------
               Title:
                     -------------------------------------------------------

                                     S-3 
 
<PAGE>
 
                     THE BANK OF NEW YORK,
                     individually, as Co-Agent and as Issuing Lender for the
                     Letters of Credit (other than the Existing Letters of
                     Credit)



                     By:
                        ---------------------------------------------------
                     Title:
                           --------------------------------------------------

                                     S-4 
 
                     
<PAGE>
 
               BANKAMERICA BUSINESS CREDIT, INC.,
               individually and as Co-Agent


               By:
                  --------------------------------------------------------
               Title:
                     -------------------------------------------------------

                                     S-5
 

<PAGE>
 
                     THE FIRST NATIONAL BANK OF CHICAGO,
                     individually and as Issuing Lender for Existing Letters of
                     Credit


                     By:
                        ----------------------------------------------------
                     Title:
                           ----------------------------------------------------

                                     S-6 
 
<PAGE>
 
                     MANUFACTURERS AND TRADERS TRUST COMPANY


                     By:
                        -------------------------------------------------
                     Title:
                           ---------------------------------------------------

                                     S-7 
 
<PAGE>
 
                     HARRIS TRUST AND SAVINGS BANK


                     By:
                        --------------------------------------------------
                     Title:
                           ---------------------------------------------------

                                     S-8 
 
<PAGE>
 
                     THE NORTHERN TRUST COMPANY


                     By:
                        ----------------------------------------------------
                     Title:
                           ---------------------------------------------------

                                     S-9 
 
<PAGE>
 
                     CHEMICAL BANK


                     By:
                        ----------------------------------------------------
                     Title:
                           ---------------------------------------------------

                                     S-10
 
<PAGE>
 
                     SANWA BUSINESS CREDIT CORPORATION


                     By:
                        ----------------------------------------------------
                     Title:
                           ---------------------------------------------------

                                     S-11