<PAGE>   1
 
                                                                 EXHIBIT 10.40.1
 
                 FIRST AMENDMENT TO LOAN AND SECURITY AGREEMENT
 
     This First Amendment to Loan and Security Agreement ("Amendment") dated as
of November 11, 1996 between L.A. T Sportswear, Inc., a Georgia corporation with
a place of business at 1200 Airport Road, Ball Ground, Georgia 30107
("Borrower") and Mellon Bank, N.A. ("Lender").
 
                                   BACKGROUND
 
     A. On April 29 ,1996, Borrower and Lender entered into a certain Loan and
Security Agreement ("Loan Agreement") and certain related agreements and
documents (all of which are collectively referred to as the "Loan Documents") to
reflect certain loan arrangements between the parties pursuant to which Lender
established for the benefit of Borrower a lending facility in the maximum amount
of $35,000,000, which reduced as of July 31, 1996 to $30,000,000. All
capitalized terms not otherwise defined herein shall have the meaning ascribed
thereto in the Loan Agreement.
 
     B. Borrower and Lender have agreed pursuant to this Amendment to modify
certain terms and conditions of the Loan Documents.
 
     NOW, THEREFORE, with the foregoing background hereinafter incorporated by
reference as if set forth more fully below, and the parties hereto, intending to
be legally bound hereby, promise and agree as follows:
 
          1. Section 1 of the Loan Agreement is hereby amended by deleting the
     definition of "Applicable Base Rate Margin" in its entirety and replacing
     it with the following:
 
             Applicable Base Rate Margin -- one percent (1%) per annum.
 
          2. Effective upon satisfaction of the Conditions to Closing set forth
     in Section 6 of this Amendment, Section 1 of the Loan Agreement shall be,
     on a prospective basis, amended by replacing the definition of "Borrowing
     Base" contained therein with the following:
 
             Borrowing Base -- As of any date of determination, an amount equal
        to the (a) sum of (i) 80% of Eligible Accounts, plus (ii) 60% of
        Eligible Inventory, not to exceed the Inventory Sublimit, plus (iii) the
        Fixed Asset Sublimit, plus (iv) the lesser of $50,000 or 50% of the
        Accounts due from T-Shirt Brokerage, Inc. which are not more than thirty
        (30) days past the invoice date and except as provided in clause (ix) of
        the definition of Eligible Receivables would otherwise constitute
        Eligible Receivables, minus (b) the L/C Reserve amount; provided
        however, that as of May 31, 1997 the percentage of Eligible Inventory
        referred to in clause (a)(ii) above shall be reduced to 50% and that as
        of April 29, 1997, the Fixed Asset Sublimit shall be deleted from the
        calculation of the Borrowing Base.
 
          3. Subsection 2.4(b)(i) of the Loan Agreement is hereby deleted in its
     entirety and replaced with the following:
 
             (b) LIBOR Rate Option
 
                (i) Provided that Borrower has at least $5,000,000 in Base Rate
           Loans outstanding and a Tangible Net Worth of at least $14,000,000,
           and so long as no Event of Default has occurred, Borrower shall have
           the option to have the unpaid principal balance of Loans (in excess
           of $5,000,000) under the Revolving Credit bear interest at the LIBOR
           Based Rate ("LIBOR Rate Option"), provided that LIBOR Rate Loans
           shall be in $1,000,000 increments and in a minimum of Five Million
           Dollars ($5,000,000.00).
<PAGE>   2
 
          4. Section 6.9 of the Loan Agreement is hereby deleted in its entirety
     and replaced with the following:
 
             6.9 Financial Covenants:  Borrower shall maintain and comply with
        the following financial covenants (calculated on the basis of GAAP):
 
                (a) Working Capital:  Borrower shall have and maintain Working
           Capital, measured quarterly at the end of each Fiscal Quarter, of not
           less than:
 
<TABLE>
                <S>                                    <C>
                $8,000,000...........................  as of Fiscal Quarter end September, 1996;
                $7,500,000...........................  as of Fiscal Quarter end December, 1996;
                $7,000,000...........................  as of Fiscal Quarter end March, 1997;
                $7,000,000...........................  as of Fiscal Quarter end June, 1997;
                $7,500,000...........................  as of Fiscal Quarter end September, 1997;
                $8,000,000...........................  as of Fiscal Year end 1997 and
                                                       $11,000,000 for each Fiscal Quarter
                                                       thereafter;
</TABLE>
 
                (b) Tangible Net Worth:  Borrower shall have and maintain a
           Tangible Net Worth, on a consolidated basis, measured monthly at each
           Fiscal Month End, of not less than:
 
<TABLE>
                <S>                                    <C>
                $12,800,000..........................  as of and March, 1996,
                $12,900,000..........................  as of April, 1996,
                $12,950,000..........................  as of May, 1996 through July, 1996,
                $13,500,000..........................  as of August, 1996,
                $12,500,000..........................  as of September, 1996,
                $12,000,000..........................  as of October, 1996,
                $12,000,000..........................  as of November, 1996,
                $11,895,000..........................  as of December 28, 1996,
                $11,500,000..........................  as of January, 1997 through May, 1997,
                $11,625,000..........................  as of June, 1997 through August, 1997
                $12,395,000..........................  as of September, 1997 through November,
                                                       1997,
                $12,895,000..........................  as of Borrower's Fiscal Year End 1997,
                $16,391,000..........................  as of each Fiscal Month End thereafter.
</TABLE>
 
                (c) Net Income:  Borrower shall have and maintain a Net Income,
           on a consolidated basis, measured annually, of not less than
           [$1,750,000] as of the close of Borrower's Fiscal Year End 1996;
           $1,000,000 as of the close of Borrower's Fiscal Year End 1997; and
           $2,000,000 as of the close of Borrower's Fiscal Year End 1998 and for
           each Fiscal Year thereafter.
 
                (d) Current Ratio:  Borrower shall have and maintain a Current
           Ratio on a consolidated basis measured monthly at each Fiscal Month
           End, of not less than: 1.05:1 as of March, 1996 and through August,
           1996; 1.10:1 as of September, 1996 through November, 1996; 1.10:1 as
           of December 28, 1996 through November, 1997; 1.20:1 as of Borrower's
           Fiscal Year End 1997, and 1.25:1 as of each Fiscal Month End
           thereafter.
 
                (e) Debt to Tangible Net Worth:  Borrower shall have and
           maintain a Debt to Tangible Net Worth Ratio on a consolidated basis
           measured quarterly at the end of each Fiscal Quarter not to exceed:
           3.60 as of Fiscal Quarter end March, 1996 and Fiscal Quarter end
           June, 1996; 3.50 as of Fiscal Quarter end September, 1996; 3.50 as of
           December 28, 1996 and for each of Borrower's next three Fiscal
           Quarters; 3.00 as of Borrower's Fiscal Year End December 1997; and
           2.60:1 for each Fiscal Quarter thereafter.
 
                (f) Capital Expenditure:  Borrower shall not expend for Unfunded
           Capital Expenditures in excess of $250,000 during the fiscal year
           ending December, 1996, $400,000 during the fiscal year ending
           December, 1997, and $1,000,000 during the fiscal year ending
           December, 1998, all on a noncumulative basis, except that Borrower
           may carryover no more than $250,000 of permitted but unused Unfunded
           Capital Expenditures for the Fiscal Year End 1997 to the next fiscal
           year.
 
                                        2
<PAGE>   3
 
                (g) Calculation and Establishment of Financial
           Covenants:  Borrower shall not include the reversal of asset reserves
           established as of December 30, 1995 in the calculation of, and for
           the purposes of meeting, the financial covenants set forth above and
           for determining the EBIDTA to Interest Expense calculations referred
           to in this Agreement.
 
          5. Restructure Fee:  Contemporaneously with the execution of this
     Amendment, Borrower shall pay to Lender a Restructure Fee in the amount of
     $50,000, which Restructure Fee is fully earned and nonrefundable as of such
     date.
 
          6. Conditions to Closing:  Lender's obligation to enter into this
     Amendment are subject to the following conditions having been satisfied in
     full to Lender's satisfaction:
 
             (a) Execution and delivery of this Amendment to Lender;
 
             (b) Delivery of a certificate of corporate resolution signed by
        Borrower's Secretary reflecting the authorization to execute, deliver
        and perform under this Amendment and all related agreements and
        documents along with a current incumbency certificate, with specimen
        signatures, identifying all Authorized Officers of the Borrower;
 
             (c) Payment of the Restructure Fee by Borrower to Lender;
 
             (d) Delivery of a certification by the Chief Executive Officer of
        Borrower that there has not occurred any material adverse change, since
        September 30, 1996 in the operations and condition (financial or
        otherwise) of Borrower;
 
             (e) Delivery of such other documentation or documents as Lender may
        reasonable require;
 
             (f) Except with respect to Events of Default arising by Borrower's
        failure to comply with certain covenants contained in Section 6.9 of the
        Loan Agreement, which covenants are being amended pursuant to the terms
        of this Amendment, no Event of Default shall have occurred under the
        Loan Agreement and be continuing and no event shall have occurred which
        with the passage of time, the giving of notice or both would constitute
        an Event of Default under the Loan Agreement; and
 
             (g) Payment or reimbursement to Lender for all legal expenses
        incurred by Lender to analyze, prepare and negotiate and conclude this
        Amendment and all related agreements and transactions described herein.
 
          7. Existing Events of Defaults:  Borrower hereby acknowledges that it
     has not complied with Section 6.9 of the Loan Agreement by failing to meet
     and maintain the financial covenants set forth therein. Upon satisfaction
     of each of the conditions set forth in Section 6 above, Lender shall be
     deemed to have waived any covenant defaults which occurred during the
     fiscal quarter ending September 30, 1996; provided however, that it is
     understood and agreed that this waiver of Borrower's breach of the
     covenants contained in Section 6.9 of the Loan Agreement shall not be
     deemed to be a waiver of any future violations of such section, as amended
     hereby, or any other covenants contained in the Loan Agreement.
 
          8. Confirmation of Indebtedness:  Borrower hereby acknowledges and
     confirms that as of the close of business on November 11, 1996, it is
     indebted to Lender, without defense, setoff, claim or counterclaim under
     the Loan Documents, in the aggregate principal amount of Sixteen Million
     Two Hundred Fifty Nine Thousand and Thirty One Dollars ($16,259,031.00),
     comprised of $16,259,031 outstanding with respect to the Revolving Credit
     Loans and $0 representing the face value of issued and outstanding Letters
     of Credit issued for the benefit of Borrower, plus all fees, costs and
     expenses (including attorneys' fees) incurred to date in connection with
     the Loan Documents.
 
          9. Confirmation of Security Interest:  Borrower hereby confirms that
     all Collateral, liens, and security interests at any time granted by
     Borrower to Lender, shall continue unimpaired and in full force and effect
     and shall continue to cover and secure the Liabilities of Borrower to
     Lender to the full extent set forth in the Loan Agreement, as amended
     hereby. All Collateral remains free and clear of any Liens other than
     Permitted Liens or Liens in favor of Lender. Nothing herein contained is
     intended to in any impair or limit the validity, priority and extent of
     Lender's existing security interest in and Liens upon the Collateral.
 
                                        3
<PAGE>   4
 
          10. Representation and Warranties:  Borrower represents and warrants
     to Lender that:
 
             (a) All warranties and representations made to Lender under the
        Loan Agreement are true and correct as to the date hereof;
 
             (b) The execution and delivery by Borrower of this Amendment and
        the performance by it of the transactions herein contemplated (i) are
        and will be within its powers, (ii) have been authorized by all
        necessary corporate or partnership action, and (iii) are not and will
        not be in contravention of any order of any court or other agency of
        government, of law or any other indenture, agreement or undertaking to
        which Borrower is a party or by which the property of Borrower is bound,
        or be in conflict with, result in a breach of, or constitute (with due
        notice and/or lapse of time) a default under, any such indenture,
        agreement or undertaking or result in the imposition of any lien, charge
        or incumbrance of any nature on any of the Properties of Borrower;
 
             (c) This Amendment and any assignment, instrument, document, or
        agreement executed and delivered in connection herewith, will be valid,
        legal, binding and enforceable in accordance with the respect of terms;
        and
 
             (d) Except with respect to Events of Default arising by Borrower's
        failure to comply with certain covenants contained in Section 6.9 of the
        Loan Agreement, which covenants are being amended pursuant to the terms
        of this Amendment, no Event of Default has occurred under the Loan
        Agreement and that no event has occurred which with the passage of time,
        the giving of notice or both would constitute an Event of Default under
        the Loan Agreement.
 
          11. Incorporation:  The parties acknowledge and agree that this
     Amendment is incorporated into and made a part of the Loan Agreement, the
     terms and provisions of which, unless expressly modified herein, are hereby
     ratified and confirmed and continue unchanged and in full force and effect.
     Any future reference to the Loan Agreement shall mean the Loan Agreement as
     amended hereby. To the extent that any term or provision of this Amendment
     is or may be deemed expressly inconsistent with any term or provision of
     the Loan Agreement, the terms and provisions hereof shall control. All
     other terms and provisions of the Loan Agreement unless expressly modified
     herein shall remain in full force and effect.
 
          12. No Modification:  No modification hereof or of any agreement
     referred to herein shall be binding or enforceable unless in writing and
     signed on behalf of the party against whom enforcement is sought.
 
          13. Waiver:  Nothing herein shall be construed to constitute a waiver
     of any breach of any representation, warranty or covenant made or agreed to
     by Borrower under the Loan Documents as amended hereby, and all of Lender's
     claims and rights resulting from any such breach or misrepresentation by
     Borrower, are expressly reserved by Lender.
 
          14. Governing Law:  This terms and provisions of this Amendment shall
     be governed by, construed and enforced in accordance with the laws of the
     Commonwealth of Pennsylvania.
 
          15. Counterparts:  This Amendment may be executed in any number of
     counterparts, each of which when so executed shall be deemed to be an
     original, and such counterparts together shall constitute one and the same
     respective agreement.
 
                                        4
<PAGE>   5
 
     IN WITNESS WHEREOF, the parties hereto have executed this First Amendment
to Loan and Security Agreement the day and year first above written.
 
                                          MELLON BANK, N.A.
 
                                          By:      /s/ ROGER ATTIX
                                            -----------------------------
 
                                          L.A. T SPORTSWEAR, INC.
 
                                          By:               /s/ ROBERT C.
                                              ALDWORTH
                                            -----------------------------
 
                                          Attest:            /s/ J. DAVID
                                                  KELLER
                                              --------------------------- (Seal)
 
                                        5