<PAGE>   1
EXHIBIT 10.1.1

                   AMENDMENT AND NOTE MODIFICATION AGREEMENT

                                November 5, 1997

Tropical Sportswear Int'l Corporation
4902 West Waters
Tampa, Florida 33634-1302

Gentlemen:

       Reference is made to (i) that certain Loan and Security Agreement dated
September 28, 1994 (as at any time amended, the "Loan Agreement"), between
Fleet Capital Corporation, a Rhode Island corporation, as successor to Shawmut
Capital Corporation (together with its successors and assigns, "Lender"), and
Tropical Sportswear International Corporation ("TSI"), a Florida corporation;
and (ii) that certain Master Equipment Note dated September 28, 1994, in the
original principal amount of $5,000,000 (as at any time amended or modified,
the "Master Note"), by TSI in favor of Lender. Capitalized terms used herein
and not otherwise defined herein shall have the meaning ascribed to such terms
in the Loan Agreement.

       In connection with the execution of the Loan Agreement, Tropical
Acquisition Company ("TAC") and Apparel International Group, Inc. ("AIG")
executed Continuing Guarantee Agreements (the "Guaranties") to guaranty payment
and performance of TSI's obligations under the Loan Agreement and the other
Loan Documents.

       Lender has been advised that TSI, AIG and TAC, have been merged into
Tropical Sportswear Int'l Corporation, a Florida corporation ("TSI/Newco"). On
September 26, 1997, AIG, the sole shareholder of TAC, was merged into
TSI/Newco. Subsequently, on October 23, 1997, TSI and TAC , the sole
shareholder of TSI, were merged into TSI/Newco. TSI/Newco, as the surviving
entity of these mergers, now holds all assets and liabilities of TSI, TAC and
AIG, including all Obligations arising under the Loan Agreement, the Master
Note and the other Loan Documents.

       TSI/Newco has requested that Lender extend the date of the Original Term
from September 30, 1997 to October 31, 1998. Lender is willing to extend the
Original Term, subject to the terms and conditions contained herein.

       NOW, THEREFORE, in consideration of Ten Dollars ($10.00) in hand paid
each to the other and other good and valuable consideration, the receipt and
sufficiency of which is hereby acknowledged, the parties hereto hereby agree as
follows:

       The Loan Agreement is hereby amended by deleting the reference to
"September 30, 1997" that is contained in Section 4.1 of the Loan Agreement and
by substituting in lieu thereof a reference to October 31, 1998."

       The Master Note is hereby amended by deleting the reference to
"September 30, 1997" that is contained in paragraphs (b) and (b) (vi) of the
Master Note and by substituting in lieu thereof, in each instance, a reference
to "October 31, 1998."

       TSI/Newco hereby ratifies and reaffirms each of the Loan Documents and
all of TSI's covenants, duties, indebtedness and liabilities thereunder.
TSI/Newco hereby acknowledges that it is liable for all Obligations, whether
heretofore or hereafter incurred. TSI/Newco acknowledges and stipulates that
the Loan Agreement, the Master Note and all other Loan Documents are legal,
valid and binding obligations enforceable against TSI/Newco in accordance with
the terms thereof and nothing contained herein, except as expressly provided
herein, shall be deemed to amend or modify any provision of any Loan Document,
all of which shall remain in full force and effect. TSI/Newco hereby
acknowledges that (i) TSI/Newco is



                                      
<PAGE>   2

the "Borrower" under the Loan Agreement and the other Loan Documents and (ii)
Lender has a first priority Lien in the Collateral. TSI/Newco hereby grants a
Lien upon the Collateral to Lender as security for the Obligations.

       TSI/Newco hereby agrees to take such further actions as Lender shall
reasonably request from time to time in connection herewith to evidence the
amendments set forth herein on the foregoing Mergers.

       This Amendment shall be governed by and construed in accordance with the
internal laws of the State of Georgia and shall be binding upon and inure to
the benefit of the parties hereto and their respective successors and assigns.

       This Amendment may be executed in any number of counterparts and by
different parties to this Amendment on separate counterparts, each of which,
when so executed, shall be deemed an original, but all such counterparts shall
constitute one and the same agreement. Any signature delivered by a party by
facsimile transmission shall be deemed to be an original signature hereto.

                                  Very truly yours,

                                           FLEET CAPITAL CORPORATION

                                           By: /S/  Elizabeth L. Waller
                                               -------------------------------
                                               Title:     V.P.
                                                     -------------------------

                                           Acknowledged and agreed to:       

                                           TROPICAL SPORTSWEAR
                                           INT'L CORPORATION
                                           ("TSI/Newco")

                                           By: /S/  Sharon L. Perdue
                                               --------------------------------
                                               Title:     Sr. V.P. of Finance
                                                     -------------------------


                                           [CORPORATE SEAL]