<PAGE>
 
                                                                  EXHIBIT  4-C-5


                              HARTMARX CORPORATION

                      FIFTH AMENDMENT TO CREDIT AGREEMENT



     This FIFTH AMENDMENT TO CREDIT AGREEMENT (this "AMENDMENT") is dated as
of January 30, 1996 and entered into by and among HARTMARX CORPORATION, a
Delaware corporation ("BORROWER"), the LENDERS LISTED ON THE SIGNATURE PAGES
HEREOF (each individually referred to as a "LENDER" and collectively as
"LENDERS"), GENERAL ELECTRIC CAPITAL CORPORATION as Managing Agent and
Collateral Agent for Lenders ("MANAGING AGENT") and THE BANK OF NEW YORK and
BANKAMERICA BUSINESS CREDIT, INC. as co-agents ("CO-AGENTS") and, for purposes
of Section 3 hereof, the GUARANTORS IDENTIFIED ON THE SIGNATURE PAGES HEREOF,
(collectively the "GUARANTORS"), and is made with reference to that certain
Credit Agreement dated as of March 23, 1994, among Borrower, Lenders and Agent,
as amended by the First Amendment to Credit Agreement dated as of August 26,
1994 among Borrower, Lenders and Agent, by the Second Amendment to Credit
Agreement dated as of March 20, 1995 among Borrower, Lenders and Agent, by the
Third Amendment to Credit Agreement dated as of June 30, 1995 among Borrower,
Lenders, Managing Agent, Co-Agents and consented to by Guarantors and by the
Fourth Amendment to Credit Agreement dated as of November 30, 1995 by and among
Borrower, Lenders, Managing Agent, Co-Agents and consented to by Guarantors (the
"CREDIT AGREEMENT"; capitalized terms used herein without definition shall
have the same meanings herein as set forth in the Credit Agreement).  Unless
otherwise indicated, Section and subsection references contained herein shall be
to the corresponding Sections and subsections of the Credit Agreement.


                                R E C I T A L S
                                - - - - - - - -

     WHEREAS, Borrower and Lenders desire to amend the Credit Agreement as set
forth below;

     NOW, THEREFORE, in consideration of the premises and the agreements,
provisions and covenants herein contained, the parties hereto agree as follows:

                                   SECTION 1
                                   AMENDMENT

1.1  AMENDMENTS TO THE CREDIT AGREEMENT.

     A.   Rate of Interest.  Section 2.2.A. of the Credit Agreement is amended
by deleting the third paragraph and the fourth paragraph thereof in their
entirety and substituting the following therefor:
<PAGE>
 
     "The "APPLICABLE MARGIN" for each LIBOR Rate Loan for any fiscal quarter
shall be the percentage which is set forth below next to the most recent fiscal
quarter of Borrower for which a Compliance Certificate is (or should have been)
delivered pursuant to subsection 6.1(iv) ("CALCULATION QUARTER") based upon the
Consolidated Debt Service Coverage Ratio for the four consecutive fiscal
quarters of Borrower ending on the last day of the Calculation Quarter; provided
that notwithstanding anything to the contrary contained herein, the Applicable
Margin shall be 1.75% until the earlier of (i) the date of delivery of a
Compliance Certificate pursuant to subsection 6.1(iv) for the second fiscal
quarter of 1996 of Borrower or (ii) the date such Compliance Certificate should
have been delivered pursuant to subsection 6.1(iv) for the second fiscal quarter
of 1996 of Borrower in the event that such Compliance Certificate is not timely
delivered.

<TABLE>
<CAPTION>
                                                                    APPLICABLE
                                                                    MARGIN FOR
                               CONSOLIDATED DEBT SERVICE            LIBOR RATE
FISCAL QUARTER                      COVERAGE RATIO                    LOANS
<S>                  <C>                                            <C>
------------------------------------------------------------------------------
First Quarter 1996   Not applicable                                      1.75%
------------------------------------------------------------------------------
Second Quarter 1996  Less than 0.76:1.00                                 2.00%
                     ---------------------------------------------------------
                     Greater than or equal to 0.76:1.00 but less                
                     than 0.84:1.00                                     1.875% 
                     ---------------------------------------------------------
                     Greater than or equal to 0.84:1.00 but less                
                     than 1.50:1.00                                      1.75% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.50:1.00                  1.50%
------------------------------------------------------------------------------
Third Quarter 1996   Less than 0.92:1.00                                 2.00%
                     ---------------------------------------------------------
                     Greater than or equal to 0.92:1.00 but less                
                     than 1.02:1.00                                     1.875% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.02:1.00 but less                
                     than 1.50:1.00                                      1.75% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.50:1.00                  1.50%
------------------------------------------------------------------------------
Fourth Quarter 1996  Less than 1.00:1.00                                 2.00%
                     ---------------------------------------------------------
                     Greater than or equal to 1.00:1.00 but less                
                     than 1.11:1.00                                     1.875% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.11:1.00 but less                
                     than 1.50:1.00                                      1.75% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.50:1.00                  1.50%
------------------------------------------------------------------------------
First Quarter 1997   Less than 1.22:1.00                                 2.00%
                     ---------------------------------------------------------
                     Greater than or equal to 1.22:1.00 but less                
                     than 1.36:1.00                                     1.875% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.36:1.00 but less                
                     than 1.50:1.00                                      1.75% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.50:1.00                  1.50%
------------------------------------------------------------------------------
</TABLE> 

                                       2
<PAGE>
 
<TABLE>
<CAPTION>
                                                                    APPLICABLE
                                                                    MARGIN FOR
                               CONSOLIDATED DEBT SERVICE            LIBOR RATE
FISCAL QUARTER                      COVERAGE RATIO                    LOANS
<S>                  <C>                                            <C>
------------------------------------------------------------------------------
Second Quarter 1997  Less than 1.10:1.00                                 2.00%
                     ---------------------------------------------------------
                     Greater than or equal to 1.10:1.00 but less               
                     than 1.20:1.00                                     1.875% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.20:1.00 but less               
                     than 1.50:1.00                                      1.75% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.50:1.00                  1.50%
------------------------------------------------------------------------------
Third Quarter 1997   Less than 1.20:1.00                                 2.00%
                     ---------------------------------------------------------
                     Greater than or equal to 1.20:1.00 but less               
                     than 1.30:1.00                                     1.875% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.30:1.00 but less               
                     than 1.50:1.00                                      1.75% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.50:1.00                  1.50%
------------------------------------------------------------------------------
Fourth Quarter 1997                                                            
 and thereafter      Less than 1.30:1.00                                 2.00% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.30:1.00 but less        1.875%
                     than 1.40:1.00                                          
                     ---------------------------------------------------------
                     Greater than or equal to 1.40:1.00 but less               
                     than 1.50:1.00                                      1.75% 
                     ---------------------------------------------------------
                     Greater than or equal to 1.50:1.00                  1.50%
------------------------------------------------------------------------------
</TABLE>

     For Libor Rate Loans initiated, or converted or continued pursuant to a
Notice of Conversion/Continuation or a telephonic request for conversion or
continuation, at or after the delivery of the applicable Compliance Certificate,
the Applicable Margin shall be adjusted, to the extent required, on the date of
delivery of such Compliance Certificate delivered pursuant to subsection
6.1(iv); provided that the Consolidated Debt Service Coverage Ratios and
Applicable Margins set forth above for any Calculation Quarter shall continue in
effect until the earlier of (i) the date of delivery of a Compliance Certificate
pursuant to subsection 6.1(iv) for the fiscal quarter of Borrower next
succeeding such Calculation Quarter or (ii) the date such Compliance Certificate
should have been delivered pursuant to subsection 6.1(iv) for such fiscal
quarter of Borrower next succeeding such Calculation Quarter in the event such
Compliance Certificate is not timely delivered, at which time such fiscal
quarter of Borrower next succeeding such Calculation Quarter shall be deemed to
be the Calculation Quarter for purposes hereof; provided further that without
limiting any Event of Default or Potential Event of Default that may result
therefrom, in the event Borrower does not deliver any Compliance Certificate
required pursuant to subsec tion 6.1(iv) by the date specified therefor, then
the Applicable Margin shall automatically be adjusted to 2.00%, commencing on
the date such Compliance

                                       3
<PAGE>
 
     Certificate was required to be delivered and expiring on the actual date of
     delivery thereof."

     B.  Prepayments and Reductions from Asset Sales.  (i) Subsection 2.4A(iii)
of the Credit Agreement is amended by deleting the first sentence thereof and
substituting the following sentence therefor:

          "No later than the second Business Day following the date of receipt
     by Borrower or any of its Subsidiaries of (i) Net Cash Proceeds of any
     Asset Sale equal to or greater than $500,000 in the aggregate, Borrower
     shall prepay the Loans in an amount equal to the amount of such Net Cash
     Proceeds which is the highest integral multiple of $100,000 and (ii) Net
     Cash Proceeds of any Asset Sale, the Revolving Loan Commitments shall be
     permanently reduced to the extent required by the proviso set forth
     below."

     (ii) Subsection 2.4A(iii) of the Credit Agreement is further amended by
deleting the proviso thereto and substituting the following therefor:

     "  ; provided, however, that the Revolving Loan Commitments shall only be
     reduced in an amount equal to the Net Cash Proceeds of an Asset Sale only
     to the extent that such amount (or portion thereof) would otherwise be
     required to be applied to payment, redemption or repurchase in respect of
     the Senior Subordinated Notes."

     C.   Minimum Consolidated Debt Service Coverage Ratio.  Section 7.6A of the
Credit Agreement is amended by amending and restating the table set forth
therein as follows:

<TABLE>
<CAPTION>
                                               MINIMUM CONSOLIDATED
                                                   DEBT SERVICE    
          FISCAL QUARTER                          COVERAGE RATIO   
<S>                                            <C>                 
          ---------------------------------------------------------
          Second Quarter 1994                             1.10:1.00
          ---------------------------------------------------------
          Third Quarter 1994                              1.10:1.00
          ---------------------------------------------------------
          Fourth Quarter 1994                             1.10:1.00
          ---------------------------------------------------------
          First Quarter 1995                              1.30:1.00
          ---------------------------------------------------------
          Second Quarter 1995                             1:30:1.00
          ---------------------------------------------------------
          Third Quarter 1995                              1:40:1.00
          ---------------------------------------------------------
          Fourth Quarter 1995                             1.00:1.00
          ---------------------------------------------------------
          First Quarter 1996                              0.70:1.00
          ---------------------------------------------------------
          Second Quarter 1996                             0.67:1.00
          ---------------------------------------------------------
          Third Quarter 1996                              0.82:1.00
          ---------------------------------------------------------
          Fourth Quarter 1996                             0.89:1.00
          ---------------------------------------------------------
          First Quarter 1997                              1.00:1.00
          --------------------------------------------------------- 
</TABLE>

                                       4
<PAGE>
 
<TABLE>
<CAPTION>
                                               MINIMUM CONSOLIDATED
                                                   DEBT SERVICE    
          FISCAL QUARTER                          COVERAGE RATIO   
<S>                                            <C>                 
          ---------------------------------------------------------
          Second Quarter 1997                             1.00:1.00
          ---------------------------------------------------------
          Third Quarter 1997                              1.10:1.00
          ---------------------------------------------------------
          Fourth Quarter 1997                             1.20:1.00
          ---------------------------------------------------------
          First Quarter 1998 and thereafter               1.50:1.00
          --------------------------------------------------------- 
</TABLE>


     D.   Maximum Consolidated Leverage Ratio.  Section 7.6B of the Credit
Agreement is amended by amending and restating the table set forth therein as
follows:

<TABLE>
<CAPTION>
          Fiscal Quarter        Maximum Consolidated Leverage Ratio
<S>                             <C>                                
          ---------------------------------------------------------
          Second Quarter 1994                             6.25:1.00
          ---------------------------------------------------------
          Third Quarter 1994                              6.25:1.00
          ---------------------------------------------------------
          Fourth Quarter 1994                             6.25:1.00
          ---------------------------------------------------------
          First Quarter 1995                              6.00:1.00
          ---------------------------------------------------------
          Second Quarter 1995                             6.00:1.00
          ---------------------------------------------------------
          Third Quarter 1995                              5.50:1.00
          ---------------------------------------------------------
          Fourth Quarter 1995                             6.50:1.00
          ---------------------------------------------------------
          First Quarter 1996                              9.00:1.00
          ---------------------------------------------------------
          Second Quarter 1996                             8.50:1.00
          ---------------------------------------------------------
          Third Quarter 1996                              8.50:1.00
          ---------------------------------------------------------
          Fourth Quarter 1996                             7.50:1.00
          ---------------------------------------------------------
          First Quarter 1997                              7.50:1.00
          ---------------------------------------------------------
          Second Quarter 1997                             7.00:1.00
          ---------------------------------------------------------
          Third Quarter 1997                              7.00:1.00
          ---------------------------------------------------------
          Fourth Quarter 1997                             6.00:1.00
          ---------------------------------------------------------
          First Quarter 1998 and thereafter               5.50:1.00
          --------------------------------------------------------- 
</TABLE>
 
     E. Consolidated Capital Expenditures. Section 7.8 of the Credit Agreement
is amended by amending and restating the table set forth therein as follows:

<TABLE> 
<CAPTION> 
          Fiscal Year     Maximum Consolidated Capital Expenditures
<S>                       <C>       
          ---------------------------------------------------------
          1994                                          $15,000,000
          ---------------------------------------------------------
          1995                                          $16,000,000
          ---------------------------------------------------------
          1996                                          $15,000,000
          ---------------------------------------------------------
          1997                                          $16,000,000
          ---------------------------------------------------------
          1998                                          $17,000,000
          --------------------------------------------------------- 
</TABLE>

                                       5
<PAGE>
 
<TABLE> 
<CAPTION> 
          ---------------------------------------------------------
<S>                                               <C>
          1999                                          $18,000,000
          ---------------------------------------------------------
</TABLE>


                                   SECTION 2
                   BORROWERS' REPRESENTATIONS AND WARRANTIES

     In order to induce Lenders to enter into this Amendment, Borrower
represents and warrants to Lenders that after giving effect to this Amendment in
the manner contemplated by Section 4.6 of this Amendment, each of the following
is true and correct:

     (a) no event has occurred and is continuing which constitutes an Event of
Default or Potential Event of Default;

     (b) the representations and warranties of Borrower and the other Credit
Parties contained in the Credit Agreement and the other Loan Documents are true
and correct on and as of the date hereof and as of the Effective Date (as
defined below) to the same extent as though made on and as of the date hereof
and as of the Effective Date except to the extent such representations and
warranties specifically relate to an earlier date, in which case they are true
and correct in all material respects as of such earlier date;

     (c) each of Borrower and the other Credit Parties has performed all
agreements on its part to be performed prior to the date hereof as set forth in
the Credit Agreement and the other Loan Documents;

     (d) each of Borrower and the Guarantors has all requisite corporate power
and authority to enter into this Amendment and to carry out the transactions
contemplated by, and perform its obligations under, the Credit Agreement;

     (e) the execution of this Amendment has been duly authorized by all
necessary corporate action on the part of Borrower and the Guarantors; and

     (f) the execution and delivery by Borrower and the Guarantors of this
Amendment does not and will not (i) violate any provision of any law or any
governmental rule or regulation applicable to Borrower, the Guarantors or any of
their respective Subsidiaries, the Certificate of Incorporation of Borrower, the
Guarantors or any of their respective Subsidiaries or any order, judgment or
decree of any court or other agency of government binding on Borrower, the
Guarantors or any of their respective Subsidiaries, (ii) conflict with, result
in a breach of or constitute (with due notice or lapse of time or both) a
default under any Contractual Obligation of Borrower, the Guarantors or any of
their respective Subsidiaries, (iii) result in or require the creation or
imposition of any Lien upon any of the properties or assets of Borrower, the
Guarantors or any of their respective Subsidiaries (other than any Liens created
under any of the Loan Documents in favor of

                                       6
<PAGE>
 
Collateral Agent on behalf of Lenders), or (iv) require any approval of
stockholders or any approval or consent of any Person under any Contractual
Obligation of Borrower, the Guarantors or any of their respective Subsidiaries.


                                   SECTION 3
                                  GUARANTORS

     Each of the Guarantors hereby consents to this Amendment and agrees that
each Loan Document to which it is a party shall continue in full force and
effect and shall be valid and enforceable and shall not be impaired or affected
by the execution of this Amendment and is hereby ratified and confirmed.


                                   SECTION 4
                                 MISCELLANEOUS

4.1  REFERENCES TO AND EFFECT ON THE CREDIT AGREEMENT AND OTHER LOAN DOCUMENTS.

     (i) On and after the Effective Date, each reference in the Credit Agreement
to "this Agreement", "hereunder", "hereof", "herein" or words of like
import referring to the Credit Agreement, and each reference in the other Loan
Documents to the "Credit Agreement", "thereunder", "thereof" or words of
like import referring to the Credit Agreement, shall mean and be a reference to
the Credit Agreement as amended hereby;

     (ii) Except as specifically amended by this Amendment, the Credit Agreement
and the other Loan Documents shall remain in full force and effect and are
hereby ratified and confirmed; and

     (iii) The execution, delivery and performance of this Amendment shall
not, except as expressly provided herein, constitute a waiver of any provision
of, or operate as a waiver of any right, power or remedy of any Agent or any
Lender under, the Credit Agreement or any of the other Loan Documents.

4.2  FEES AND EXPENSES.

     Borrower acknowledges that all costs, fees and expenses as described in
subsection 10.2 of the Credit Agreement incurred by Managing Agent and its
counsel with respect to this Amendment and the documents and transactions
contemplated hereby shall be for the account of Borrower.

                                       7
<PAGE>
 
4.3  HEADINGS.

     Section and subsection headings in this Amendment are included herein for
convenience of reference only and shall not constitute a part of this Amendment
for any other purpose or be given any substantive effect.

4.4  APPLICABLE LAW.

     THIS AMENDMENT SHALL BE GOVERNED BY, AND SHALL BE CONSTRUED AND ENFORCED IN
ACCORDANCE WITH, THE INTERNAL LAWS OF THE STATE OF NEW YORK, WITHOUT REGARD TO
COMMON LAW CONFLICTS OF LAWS PRINCIPLES.

4.5  COUNTERPARTS.

     This Amendment may be executed in any number of counterparts and by
different parties hereto in separate counterparts, each of which when so
executed and delivered shall be deemed an original, but all such counterparts
together shall constitute but one and the same instrument; signature pages may
be detached from multiple separate counterparts and attached to a single
counterpart so that all signature pages are physically attached to the same
document.

4.6  EFFECTIVENESS.

     This Amendment shall become effective (such date being the "EFFECTIVE
DATE") upon the execution of a counterpart hereof by Requisite Lenders, the
Borrower and the Guarantors and receipt by Borrower and Managing Agent of
written or telephone notification of such execution and authorization of
delivery thereof.

                                       8
<PAGE>
 
     IN WITNESS WHEREOF, the parties hereto have caused this Amendment to be
duly executed and delivered by their respective officers thereunto duly
authorized as of the date first written above.

                         BORROWER:

                         HARTMARX CORPORATION

                         By: _______________________________________________
                             Glenn R. Morgan
                             Executive Vice President and
                             Chief Financial Officer


                         GUARANTORS:


                         AMERICAN APPAREL BRANDS, INC.
                         ANNISTON SPORTSWEAR CORPORATION
                         BLLTWELL COMPANY, INC.
                         BRIAR, INC.
                         C.M. CLOTHING, INC.
                         C.M. OUTLET CORP.
                         CHICAGO TROUSER COMPANY, LTD.
                         COUNTRY MISS, INC.
                         COUNTRY MISS INTERNATIONAL LIMITED
                         COUNTRY SUBURBANS, INC.
                         DIRECT ROUTE MARKETING CORPORATION
                         E-TOWN SPORTSWEAR CORPORATION
                         FAIRWOOD-WELLS, INC.
                         GLENEAGLES, INC.
                         TAG APPAREL, INC. (FORMERLY KNOWN AS HENRY GRETHEL
                           APPAREL, INC.)
                         HANDMACHER FASHIONS FACTORY OUTLET, INC.
                         HANDMACHER-VOGEL, INC.
                         HARTMARX INTERNATIONAL, INC.
                         HART SCHAFFNER & MARX
                         HART SERVICES, INC.
                         THOS. HEATH CLOTHES, INC.
                         TAG LICENSING, INC. (FORMERLY KNOWN AS HGA LICENSING,
                           INC.)
                         HICKEY-FREEMAN CO., INC.
                         HIGGINS, FRANK & HILL, INC.

                                      S-1
<PAGE>
 
                         NOVAPPAREL, INC. (FORMERLY KNOWN AS HMXUS, INC.)
                         HOOSIER FACTORIES, INCORPORATED
                         HSM UNIVERSITY, INC.
                         INTERCONTINENTAL APPAREL, INC.
                         INTERNATIONAL WOMEN'S APPAREL, INC.
                         JAYMAR-RUBY, INC.
                         JRSS, INC.
                         KUPPENHEIMER MEN'S CLOTHIERS DADEVILLE, INC.
                         MEN'S QUALITY BRANDS, INC.
                         NATIONAL CLOTHING COMPANY, INC.
                         106 REAL ESTATE CORP.
                         RECTOR SPORTSWEAR CORPORATION
                         ROBERTS INTERNATIONAL CORPORATION
                         SALHOLD, INC.
                         SEAFORD CLOTHING CO.
                         SOCIETY BRAND, LTD.
                         ROBERT SURREY, INC.
                         TAILORED TREND, INC.
                         THORNGATE UNIFORMS, INC.
                         TRADE FINANCE INTERNATIONAL LIMITED
                         UNIVERSAL DESIGN GROUP, LTD.
                         M. WILE & COMPANY, INC.
                         WINCHESTER CLOTHING COMPANY
                         YORKE SHIRT CORPORATION


                         By: _______________________________________________
                             Glenn R. Morgan
                             Vice President of each of the foregoing (except 
                             for Country Miss International Limited) and a 
                             Director of Country Miss International Limited

                                      S-2
<PAGE>
 
                         LENDERS:

                         GENERAL ELECTRIC CAPITAL CORPORATION,
                         individually, as Managing Agent and as Collateral Agent

                         By: _______________________________________________

                         Title: ____________________________________________



                                      S-3
<PAGE>
 
                         THE BANK OF NEW YORK,
                         individually, as Co-Agent and as Issuing Lender for 
                         the Letters of Credit (other than the Existing Letters
                         of Credit)


                         By: _______________________________________________

                         Title: ____________________________________________



                                      S-4
<PAGE>
 
                         BANKAMERICA BUSINESS CREDIT, INC.,
                         individually and as Co-Agent


                         By: _______________________________________________

                         Title: ____________________________________________



                                      S-5
<PAGE>
 
                         THE FIRST NATIONAL BANK OF CHICAGO,
                         individually and as Issuing Lender for Existing 
                         Letters of Credit


                         By: _______________________________________________

                         Title: ____________________________________________



                                      S-6
<PAGE>
 
                         MANUFACTURERS AND TRADERS TRUST COMPANY


                         By: _______________________________________________

                         Title: ____________________________________________



                                      S-7
<PAGE>
 
                         HARRIS TRUST AND SAVINGS BANK


                         By: _______________________________________________

                         Title: ____________________________________________



                                      S-8
<PAGE>
 
                         THE NORTHERN TRUST COMPANY


                         By: _______________________________________________

                         Title: ____________________________________________



                                      S-9
<PAGE>
 
                         CHEMICAL BANK


                         By: _______________________________________________

                         Title: ____________________________________________



                                     S-10
<PAGE>
 
                         SANWA BUSINESS CREDIT CORPORATION


                         By: _______________________________________________

                         Title: ____________________________________________



                                     S-11