<PAGE>   1
                                                                   EXHIBIT 10.4

                                AMENDMENT NO. 5

                                       TO

                          LOAN AND SECURITY AGREEMENT


        THIS AMENDMENT NO. 5 ("Amendment") is entered into as of September 30,
1996, by and between Apparel Ventures, Inc., a Delaware corporation having its
chief executive office and principal place of business at 204 West Rosecrans
Avenue, Gardena, California 90248 ("Borrower") and Fleet Credit Corporation
f/k/a Shawmut Capital Corporation ("Lender").

                                   BACKGROUND

        Borrower and Lender are parties to a Loan and Security Agreement dated
as of May 23, 1994 (as same has been or may further be amended, supplemented,
restated or otherwise modified from time to time, the "Loan Agreement") pursuant
to which Lender provided Borrower with certain financial accommodations.

        Borrower has requested that Lender amend certain provisions of the Loan
Agreement and Lender is willing to do so on the terms and conditions hereafter
set forth.

        NOW, THEREFORE, in consideration of any loan or advance or grant of
credit heretofore or hereafter made to or for the account of Borrower by Lender,
and for other good and valuable consideration, the receipt and sufficiency of
which are hereby acknowledged, the parties hereto hereby agree as follows:

        1.      DEFINITIONS. All capitalized terms not otherwise defined herein
shall have the meanings given to them in the Loan Agreement.

        2.      AMENDMENTS TO LOAN AGREEMENT. Subject to satisfaction of the
conditions precedent set forth in Section 3 below, the following provisions of
the Loan Agreement shall be amended as set forth below:

                (a) Section 7.2(B) of the Loan Agreement is hereby amended in
its entirety to read as follows:

                "(B) LOANS. Make any loans or other advances of money (other
        than for salary, travel advances, advances against commissions and other
        similar advances in the ordinary course of business) to any Person,
        including, without limitation, any of Borrower's Affiliates, officers or
        employees, except for loans or other advances to AVE which together with
        the value of all assets transferred to AVE by Borrower at any time
        outstanding does not exceed (i) $2,500,000 from the Closing Date through
        January 31, 1996, (ii) $2,630,000 from February 1, 1996 through March
        31, 1996, (iii) $3,000,000 from April 1, 1996 through June 29, 1996,
        (iv) $2,500,000 on
<PAGE>   2
                June 30, 1996, (v) $2,600,000 from July 1, 1996 through
                September 29, 1996, (vi) $2,700,000 from September 30, 1996
                through October 30, 1996, (vii) $3,600,000 from October 31, 1996
                through December 30, 1996, (viii) $3,800,000 from December 31,
                1996 through January 30, 1997, (ix) $4,200,000 from January 31,
                1997 through June 29, 1997 and (x) $3,500,000 on June 30, 1997
                and thereafter." 

                        (b)  Sections 7.3(A) and (B) of the Loan Agreement are
hereby amended in their entirety to provide as follows:
                                
                             "(A) Minimum Adjusted Tangible Net Worth.  (i)
                        Maintain at all times an Adjusted Tangible Net Worth of
                        not less than the amount ("Net Worth Amount") shown
                        below for the Period or Date corresponding thereto:


                        Period or Date                               Amount
                        --------------                         ---------------

                             7/30/95                           ($19,114,000)
                             8/31/95                           ($20,548,000)
                             9/30/95                           ($21,928,000)
                            10/31/95                           ($22,669,000)
                            11/30/95                           ($22,985,000)
                            12/31/95                           ($21,765,000)
                             1/31/95                           ($20,180,000)
                             2/29/96                           ($18,073,000)
                             3/31/96                           ($16,717,000)
                             4/30/96                           ($16,987,000)
                             5/31/96                           ($17,689,000)
                             6/30/96                           ($18,217,000)
                             7/31/96   through 9/29/96         ($19,217,000)
                             9/30/96   through 10/30/96        ($20,000,000)
                            10/31/96   through 11/29/96        ($21,000,000)
                            11/30/96   through 12/30/96        ($21,500,000)
                            12/31/96   through 1/30/97         ($21,000,000)
                             1/31/97   through 2/27/97         ($19,500,000)
                             2/28/97   through 3/30/97         ($18,600,000)
                             3/31/97   through 4/29/97         ($18,400,000)
                             4/30/97   through 5/30/97         ($18,200,000)
                             5/31/97   through 6/29/97         ($18,000,000)
                             6/30/97   through 6/29/98         ($17,500,000)
                             6/30/98   through 6/29/99         ($16,000,000)
                             6/30/99   through 6/29/00         ($14,500,000)

For each Fiscal year, thereafter the Net Worth Amount shall be increased 
by $1,500,000 for each such Fiscal year.   

        Lender acknowledges that the actual Date or ending Date of a period
reflected in the above table may be different by a day or two depending upon the
specific accounting periods utilized by Borrower in the applicable fiscal year.

                                      -2-
<PAGE>   3
                             (B)  Fixed Charge Coverage Ratio.  Maintain a Fixed
                        Charge Coverage Ratio of not less than:

                             (i)     .25 to 1.0 for the four consecutive fiscal
                        quarter period ending on September 30, 1995;

                             (ii)    .37 to 1.0 for the four consecutive fiscal
                        quarter period ending on December 31, 1995;

                             (iii)   .35 to 1 for the four consecutive fiscal
                        quarter period ending on March 31, 1996;

                             (iv)    .75 to 1 for the four consecutive fiscal
                        quarter period ending on June 30, 1996;

                             (v)     .65 to 1.0 for the four consecutive fiscal
                        quarter period ending September 30, 1996;

                             (vi)    .75 to 1.0 for the four consecutive fiscal
                        quarter period ending December 31, 1997; and

                             (vii)   1.0 to 1.0 for the four consecutive fiscal
                        quarter period ending March 31, 1997; and

                             (v)     1.50 to 1.0 for each four consecutive
                        fiscal quarter period ending thereafter."


                3.      Conditions of Effectiveness.  This Amendment shall
become effective as of July 1, 1996, when and only when Lender shall have
received (i) four (4) copies of this Amendment executed by Borrower and (ii)
such other certificates, instruments, documents, agreements and opinions of
counsel as may be required by Lender or its counsel, each of which shall be in
form and substance satisfactory to Lender and its counsel.

                4.      Representations and Warranties.  Borrower hereby
represents and warrants as follows:

                        (a)  This Amendment and the Loan Agreement, as amended
                hereby, constitute legal, valid and binding obligations of
                Borrower and are enforceable against Borrower in accordance with
                their respective terms.

                        (b)  Upon the effectiveness of this Amendment, Borrower
                hereby reaffirms all covenants, representations and warranties
                made in the Loan Agreement to the extent the same are not
                amended hereby and agree that all such covenants,
                representations and warranties shall be deemed to have been
                remade as of the effective date of this Amendments.






                                      -3-
<PAGE>   4
                        (c)  No Event of Default or Default has occurred and is
                continuing or would exist after giving effect to this Amendment.

                        (d)  Borrower has no defense, counterclaim or offset
                with respect to the Loan Agreement.



                5.      Effect on the Loan Agreement.

                        (a)  Upon the effectiveness hereof, each reference in
the Loan Agreement to "this Agreement," "hereunder," "hereof," "herein" or words
of like import shall mean and be a reference to the Loan Agreement as amended
hereby.


                        (b)  Except as specifically amended herein, the Loan
Agreement, and all other documents, instruments and agreements executed and/or
delivered in connection therewith, shall remain in full force and effect, and
are hereby ratified and confirmed.

                        (c)  The execution, delivery and effectiveness of this
Amendment shall not operate as a waiver of any right, power or remedy of
Lender, nor constitute a waiver of any provision of the Loan Agreement, or any
other documents, instruments or agreements executed and/or delivered under or
in connection therewith.

                6.      Governing Law.  This Amendment shall be binding upon
and inure to the benefit of the parties hereto and their respective successors
and assigns and shall be governed by and construed in accordance with the laws
of the State of New York.

                7.      Headings.  Section headings in this Amendment are
included herein for convenience of reference only and shall not constitute a
part of this Amendment for any other purpose.

                8.      Counterparts, Telecopied Signatures.  This Amendment
may be executed in any number of and by different parties hereto on separate
counterparts, all of which, when so executed shall be deemed an original, but
all such counterparts shall constitute one and the same instrument.  Any
signature delivered by a party by facsimile transmission shall be deemed an
original signature hereto.

                IN WITNESS WHEREOF, this Amendment has been duly executed as of
the day and year first written above.


                                        APPAREL VENTURES, INC.



                                        By: /s/ MARVIN L. GOODMAN
                                           ---------------------------
                                           Name:  Marvin L. Goodman
                                           Title: President


                                        FLEET CAPITAL CORPORATION



                                        By:   /s/  WALTER SCHUPPE
                                           --------------------------
                                           Name:  Walter Schuppe
                                           Title: Vice President




                                      -4-