<PAGE>
                                                                 Exhibit 4(i)
 
                FIFTH MODIFICATION OF SECURED CREDIT AGREEMENT,
                    REVOLVING NOTE AND OTHER LOAN DOCUMENTS
                ----------------------------------------------


     THIS FIFTH MODIFICATION OF SECURED CREDIT AGREEMENT, REVOLVING NOTE AND
OTHER LOAN DOCUMENTS (this "Agreement") is made, and shall be deemed effective,
as of the 9th day of April, 1997 by and between PAUL HARRIS STORES, INC., an
Indiana corporation (herein, together with its successors and assigns, called
the "Borrower") and LASALLE NATIONAL BANK, a national banking association
(herein, together with its successors and assigns, called the "Bank").

     ALL CAPITALIZED TERMS AND PHRASES, UNLESS DEFINED HEREIN, SHALL HAVE THE
SPECIFIC MEANINGS AS ARE SET FORTH IN THAT CERTAIN SECURED CREDIT AGREEMENT
DATED AS OF OCTOBER 28, 1993, BY AND BETWEEN BORROWER AND BANK, AS AMENDED AND
RESTATED BY THAT CERTAIN AMENDED AND RESTATED SECURED CREDIT AGREEMENT DATED AS
OF JANUARY 20, 1994, AS MODIFIED BY THAT CERTAIN FIRST MODIFICATION OF SECURED
CREDIT AGREEMENT, NOTES, MORTGAGE AND OTHER LOAN DOCUMENTS DATED AS OF OCTOBER
31, 1994, AS FURTHER MODIFIED BY THAT CERTAIN SECOND MODIFICATION OF SECURED
CREDIT AGREEMENT, NOTES, MORTGAGE AND OTHER LOAN DOCUMENTS DATED AS OF JANUARY
31, 1995, AS FURTHER MODIFIED BY THAT CERTAIN THIRD MODIFICATION OF SECURED
CREDIT AGREEMENT, NOTES, MORTGAGE AND OTHER LOAN DOCUMENTS DATED AS OF SEPTEMBER
28, 1995, AS FURTHER MODIFIED BY THAT CERTAIN AMENDED AND RESTATED THIRD
MODIFICATION OF SECURED CREDIT AGREEMENT, NOTES, MORTGAGE AND OTHER LOAN
DOCUMENTS DATED AS OF SEPTEMBER 28, 1995, AND AS FURTHER MODIFIED BY THAT
CERTAIN FOURTH MODIFICATION OF SECURED CREDIT AGREEMENT, REVOLVING NOTE AND
OTHER LOAN DOCUMENTS DATED AS OF MAY 8, 1996 (the "Credit Agreement").

     WHEREAS, Borrower has previously requested loans and advances from Bank for
the purpose of funding Borrower's working capital needs, and in connection
therewith, Borrower and Bank entered into and executed the Credit Agreement,
pursuant to which the Bank, inter alia, agreed to make a term loan in an amount
of up to $2,400,000.00 and a revolving credit loan in an amount of up to
$20,000,000.00 to the Borrower; and

     WHEREAS, Borrower has previously executed and delivered to Bank (i) a
Secured Promissory Note (Revolver) dated October 28, 1993, as amended (the
"Revolving Note"), in the principal amount of $20,000,000.00, evidencing an
indebtedness owed by Borrower to Bank in like amount (the "Revolving Loan") and
(ii) a Secured Promissory Note (Term) dated January 20, 1994 (the "Term Note")
in the principal amount of $2,400,000.00 evidencing an indebtedness owed by
Borrower to Bank in like amount (the "Term Loan"); and

     WHEREAS, repayment of the Term Note is secured by, among other items of
collateral, a certain Mortgage, Assignment of Leases and Rents and Security
Agreement dated as of January 20, 1994, made by Borrower to Bank (the
"Mortgage"), recorded on February 1, 1994 in the Office of the Recorder, Marion
County, Indiana as Instrument

<PAGE>
 
Number 94-17807, encumbering the property legally described therein (the
"Premises"); and

     WHEREAS, repayment of the Term Note is additionally secured by a certain
Assignment of Distribution Center Leases and Rents dated as of January 20, 1994
(the "Distribution Assignment"), made by Borrower to Bank; and

     WHEREAS, repayment of the Revolving Note is secured by a certain Security
Agreement and Financing Statement dated as of October 28, 1993, as amended (the
"Security Agreement"), made by Borrower to Bank; and

     WHEREAS, repayment of the Revolving Note is additionally secured by a
certain Assignment of Leases dated as of October 28, 1994, as amended, made by
Borrower to Bank (the "Assignment"), affecting the Premises; and

     WHEREAS, repayment of the Revolving Note and the Term Note (collectively,
the "Notes") is additionally secured by UCC Financing Statements made by
Borrower, as debtor, to Bank, as secured party (the "Financing Statements"); and

     WHEREAS, the Credit Agreement, the Notes, the Mortgage, the Security
Agreement, the Assignment, the Distribution Assignment and the Financing
Statements, together with all other documents and instruments now or hereafter
securing repayment of the Liabilities, or any portion thereof, evidenced by the
Notes are hereinafter collectively referred to as the "Loan Documents"; and

     WHEREAS, Borrower has requested that Bank increase the present Revolving
Loan Commitment, extend the present Revolving Credit Maturity Date and make
other modifications to the Loan Documents, and Bank has so agreed, on the terms
and conditions more specifically set forth herein.

     NOW, THEREFORE, for and in consideration of the foregoing premises and for
other good and valuable consideration, the receipt and sufficiency of which
hereby are acknowledged, Borrower and Bank do hereby agree as follows:

     1.   The preambles to this Agreement are fully incorporated herein by this
reference thereto with the same force and effect as though restated herein.

     2.   Effective as of April 9, 1997 (the "Modification Date"), the Credit
Agreement is modified as set forth below:
 
          a.  The definition of "Borrowing Base" set forth in Section 1.1 of the
     Credit Agreement is amended by adding the following paragraph to the end of
     such definition:

                                      -2-
<PAGE>
 
          "Notwithstanding anything contained in this definition to the
          contrary, during and only during each period from August 1 of any year
          through November 30 of such year, the phrase "seventy percent (70.0%)"
          shall be substituted in the place and stead of each reference to
          "sixty percent (60.0%)" contained in clause (a) of this definition.

          b.   The definition of "Maximum Revolving Credit Availability" set
     forth in Section 1.1 of the Credit Agreement is deleted in its entirety and
     the following definition is substituted therefor:

               "MAXIMUM REVOLVING CREDIT AVAILABILITY" means, as of any given
          date, the lesser of (i) the Borrowing Base minus the Letter of Credit
          Reserve minus the Revolving Loan Balance or (ii) $30,000,000 minus the
          Letter of Credit Reserve minus the Revolving Loan Balance, each such
          amount calculated as of such date and subject further to the
          limitations set forth in Section 2.1."

          c.   The definition of "Revolving Credit Maturity Date" set forth in
     Section 1.1 of the Credit Agreement is deleted in its entirety and the
     following definition is substituted therefor:

               "REVOLVING CREDIT MATURITY DATE" means, with respect to the
          Revolving Credit Commitment, June 30, 1999.

          d.   The definition of "Revolving Loan Rate" set forth in Section 1.1
     of the Credit Agreement is deleted in its entirety and the following
     definition is substituted therefor:

               "REVOLVING LOAN RATE" means the Prime Rate from time to time in
          effect plus one-quarter of one percent (0.25%) per annum."
 
          e.   Section 2.1 of the Credit Agreement is deleted in its entirety
     and the following is substituted in its place and stead:

               "2.1 REVOLVING CREDIT COMMITMENT. On the terms and subject to the
          conditions set forth in this Agreement, Bank agrees to make Revolving
          Loans to Borrower and to issue Letters of Credit, pursuant to Section
          2.3, for the account of the Borrower, from time to time before the
          Revolving Credit Termination Date in such aggregate amounts as
          Borrower may from time to time request but not exceeding at any one
          time outstanding the lesser of (i) the Borrowing Base or (ii)
          $30,000,000; provided, however, that the issuance of standby Letters
          of Credit shall be limited to $1,000,000 in the aggregate.

                                      -3-
<PAGE>
 
          Borrower shall have the right to repay and reborrow any of the
          Revolving Loans in increments of $100,000 (or $25,000 integral
          multiples); provided, however, that it shall be a condition precedent
          to any reborrowing that as of the date of any reborrowing all of the
          conditions to borrowing set forth in this Agreement shall be satisfied
          and all representations and warranties made herein shall be true and
          correct in all material respects as of such date."

          f.   Within clause (i) of paragraph (a) of subsection 2.3.5 of the
     Credit Agreement, the reference to "three-eighths of one percent (0.375%)"
     is deleted in its entirety and the phrase "one-quarter of one percent
     (0.25%)" is substituted in its place and stead.

          g.   The following Section 2.5 is added to the Credit Agreement in the
     appropriate numerical order:

               "2.5  REVOLVING LOAN NON-USE FEE.  Borrower shall pay to Bank, on
          a quarterly basis, a non-use fee for the period beginning April 9,
          1997 and continuing until, but excluding, the Revolving Credit
          Termination Date of one-quarter of one percent (0.25%) per annum on
          the lesser of (i) the excess of (A) $30,000,000.00 over (B) the daily
          average of the aggregate principal amount of (X) all outstanding
          Revolving Loans and (Y) all Letters of Credit for the quarter (or, if
          applicable, shorter period) then ending and (ii) $10,000,000.00.  Such
          non-use fee shall be payable in arrears on the first day of each May,
          August, November and February (commencing on May 1, 1997 for the
          quarter ending April 30, 1997), for the quarters ending on January 31,
          April 30, July 31 and October 31 of each year during the term of this
          Agreement and on the Revolving Credit Termination Date."

          h.   Subsection 6.2.1 of the Credit Agreement is deleted in its
     entirety and the following is substituted in its place and stead:

               "6.2.1  REVOLVING LOAN.  The Revolving Loan Balance may be
          reduced by Bank from time to time as provided in subsection 5.4(b),
          and the entire Revolving Loan Balance, together with all accrued and
          unpaid interest thereon, shall be due and payable in full on June 30,
          1999."

          i.   Subsection 6.2.2 of the Credit Agreement is deleted in its
     entirety and the following is substituted in its place and stead:

                                      -4-
<PAGE>
 
               "6.2.2  ANNUAL REDUCTION OF REVOLVING CREDIT COMMITMENT. All
          amounts outstanding pursuant to the Revolving Loans provisions of the
          Revolving Credit Commitment must be reduced by the Borrower to zero
          ($0.00) for a period of ninety (90) consecutive days during each
          fiscal year of the Borrower, commencing with Borrower's fiscal year
          beginning February 1, 1997."

          j.   Section 9.7 of the Credit Agreement is deleted in its entirety
     and the following is substituted in its place and stead:

               "9.7  INTENTIONALLY OMITTED."

          k.   Section 9.8 of the Credit Agreement is deleted in its entirety
     and the following is substituted in its place and stead:

               "9.8  INTENTIONALLY OMITTED."

          l.   Section 9.9 of the Credit Agreement is deleted in its entirety
     and the following is substituted in its place and stead:

               "9.9  MINIMUM TANGIBLE NET WORTH. Not permit Borrower's Tangible
          Net Worth measured quarterly to be less than $30,000,000 as of January
          31, April 30, July 31 and October 31 of any year, commencing April 30,
          1997."

          m.   Section 9.10 of the Credit Agreement is deleted in its entirety
     and the following is substituted in its place and stead:

               "9.10  INTENTIONALLY OMITTED."

          n.   Section 9.11 of the Credit Agreement is deleted in its entirety
     and the following is substituted in its place and stead:

               "9.11  INTENTIONALLY OMITTED."

          o.   Section 9.13 of the Credit Agreement is deleted in its entirety
     and the following is substituted in its place and stead:

               "9.13  INTENTIONALLY OMITTED."

          p.   Section 9.15 of the Credit Agreement is deleted in its entirety
     and the following is substituted in its place and stead:

                                      -5-
<PAGE>
 
               "9.15  GUARANTIES, LOANS OR ADVANCES. Not become, or be a
          guarantor or surety of, or otherwise become or be responsible in any
          manner (whether by agreement to purchase any obligations, stock,
          assets, goods or services, or to supply or advance any funds, assets,
          goods or services, or otherwise) with respect to any undertaking of
          any other Person, or make or permit to exist any loans or advances to
          any other Person, except for (a) the endorsement, in the ordinary
          course of collection, of instruments payable to it or to its order,
          (b) guaranties of which Bank is the beneficiary or (c) loans to
          officers of Borrower not exceeding $500,000 in the aggregate
          outstanding at any time."

          q.   Section 9.18 of the Credit Agreement is deleted in its entirety
     and the following is substituted in its place and stead:

               "9.18  INTENTIONALLY OMITTED."

          r.   Schedule A to the Credit Agreement is deleted in its entirety and
     the revised Schedule A attached hereto and made a part hereof is
     substituted in its place and stead.

     3.   All references in the Loan Documents to the Credit Agreement hereby
are understood to be to the Credit Agreement as modified hereby.

     4.   Effective as of the Modification Date, paragraph one of page one of
the Revolving Note is hereby deleted in its entirety and the following is
substituted in its place and stead:

                                REVOLVING NOTE


     $30,000,000.00                                        CHICAGO, ILLINOIS
                                                            OCTOBER 28, 1993


          FOR VALUE RECEIVED, PAUL HARRIS STORES, INC., an Indiana corporation
     (together with its successors and assigns, "Maker"), promises to pay to the
     order of LASALLE NATIONAL BANK, a national banking association (together
     with its successors and assigns, "Bank"), on or before June 30, 1999, at
     the Bank's principal office in Chicago, Illinois, the principal sum of
     THIRTY MILLION AND NO/100THS DOLLARS ($30,000,000.00), or, if less, the
     REVOLVING LOAN BALANCE at such time, plus accrued and unpaid interest
     thereon and all other charges applicable thereto, all as set forth more
     fully in that Secured Credit Agreement originally dated as of October 28,
     1993, between Maker and Bank (as the same may be amended,

                                      -6-
<PAGE>
 
     modified, supplemented or restated from time to time, the "Credit
     Agreement").  All capitalized terms used but not elsewhere defined herein
     shall have the same meanings as are ascribed to them in the Credit
     Agreement."

     5.   All references in the Loan Documents to the Revolving Note are hereby
understood to be the Revolving Note as modified hereby.

     6.   In the event of any conflict among the terms of the Credit Agreement
and the other Loan Documents as modified by this Agreement, the terms of the
Credit Agreement as modified by this Agreement shall control.  All terms and
provisions of the Loan Documents corresponding to terms and provisions of the
Credit Agreement prior to the date of this Agreement shall be deemed modified in
accordance with the terms of this Agreement.

     7.   Borrower hereby warrants and represents that (i) Borrower has no
defense, offset or counterclaim with respect to the payment of any sum owed to
Bank, or with respect to any covenant in the Loan Documents; (ii) Bank, on and
as of the date hereof, has fully performed all obligations to Borrower which it
may have had or has on and as of the date hereof; (iii) other than as expressly
set forth herein, by entering into this Agreement, Bank does not waive any
condition or obligation in the Loan Documents and (iv) no individual figure in
Borrower's audited financial statements dated February 1, 1997 will vary by more
than five percent (5%) from the corresponding figure in the Company prepared
financial statements previously delivered to Bank relating to the same period,
it being understood and agreed that any deviation in any number between the two
sets of financial statements of more than five percent (5%) shall constitute an
Event of Default under the Loan Documents.

     8.   Borrower hereby agrees to execute and deliver promptly to Bank, at
Bank's request, such other documents as Bank, in its reasonable discretion,
shall deem necessary or appropriate to evidence the transaction contemplated
herein.

     9.   Borrower agrees to pay all fees and expenses associated with the
consummation of the transactions contemplated in this Agreement, including,
without limitation, fees and expenses of Bank's counsel and related expenses.

     10.  Time is of the essence of this Agreement.  Unless this Agreement is
executed by Borrower and Bank on or before April 30, 1997, it shall become null
and void and shall have no force or effect.

     11.  This Agreement may be executed in any number of counterparts, each of
which shall constitute an original, but all of which, taken together, shall
constitute one and the same Agreement.

                                      -7-
<PAGE>
 
     12.  Except as otherwise set forth herein to the contrary, the Loan
Documents remain unmodified and continue in full force and effect. Borrower
hereby reaffirms, confirms and ratifies each and every covenant, condition,
obligation and provision set forth in the Loan Documents, each as modified
hereby.

                                 [End of Page]

                                       -8-
<PAGE>
 
    IN WITNESS WHEREOF, the undersigned, intending to be legally bound hereby,
have executed and delivered this Agreement as of the day and year first above
written.


                                BORROWER:

                                PAUL HARRIS STORES, INC., an Indiana corporation


                                By: /s/ John H. Boyers
                                   ---------------------------------------------
                                Title:
                                      ------------------------------------------
                                Its: Senior Vice President-Finance and Treasurer
                                    --------------------------------------------



                                BANK:

                                LASALLE NATIONAL BANK, a national banking
                                association


                                By: /s/ Brian Soeldner
                                   -----------------------------
                                Title:
                                      --------------------------
                                Its: Loan Officer
                                    ----------------------------

                                      -9-
<PAGE>
 
                                  SCHEDULE A
                                  ----------

                        ADDITIONAL FINANCIAL COVENANTS
                        ------------------------------



     Operating Cash Flow Before Working Capital Changes based on minimum
cumulative cash flow on a rolling four quarter basis shall not be less than
$12,000,000 on April 30, 1997 and on each July 31, October 31, January 31 and
April 30 thereafter.

                                     -10-