<PAGE>   1

                FIFTH AMENDMENT TO LOAN AND SECURITY AGREEMENT


     THIS FIFTH AMENDMENT TO LOAN AND SECURITY AGREEMENT (this "Amendment") is
made and entered into this 16th day of July, 1999, by and between TROPICAL
SPORTSWEAR INT'L CORPORATION, a Florida corporation ("Tropical"), TROPICAL
SPORTSWEAR COMPANY, INC., a Delaware corporation ("TSCI"), SAVANE INTERNATIONAL
CORP., a Texas corporation (formerly known as Farah Incorporated) ("Savane"),
and APPAREL NETWORK CORPORATION, a Florida corporation ("Apparel") (Tropical,
TSCI, Savane and Apparel collectively referred to hereinafter as "Borrowers"
and individually as a "Borrower") each with its chief executive office and
principal place of business at 4902 West Waters Avenue, Tampa, Florida
33634-1302; the various financial institutions listed on the signature pages
hereof and their respective successors and permitted assigns which become
"Lenders" as provided in the Loan Agreement (as defined below); and FLEET
CAPITAL CORPORATION, a Rhode Island corporation, in its capacity as collateral
and administrative agent for the Lenders (together with its successors in such
capacity, "Agent") with an office at 300 Galleria parkway, N.W., Suite 800,
Atlanta, Georgia 30339.


                                   Recitals:

     Borrowers, Agent and Lenders, are parties to a certain Loan and Security
Agreement dated June 10, 1998, as amended by that certain First Amendment to
Loan and Security Agreement dated July 9, 1998, that certain Second Amendment to
Loan and Security Agreement dated August 27, 1998, that certain Third Amendment
to Loan and Security Agreement dated December 31, 1998, and that certain Fourth
Amendment to Loan and Security Agreement dated May 21, 1999 (as at any time
amended, the "Loan Agreement"), pursuant to which Lenders have made certain
revolving credit loans and letter of credit accommodations to Borrowers.

     Borrower is currently in default under the Loan Agreement due to Borrowers
breach of the tangible net worth covenant. Borrowers have requested that Agent
and Lenders waive the existing default under the Loan Agreement and amend the
terms of the Loan Agreement to modify the tangible net worth covenant.

     Agent and Lenders are willing to waive the default and amend the Loan
Agreement on the terms and conditions as hereinafter set forth.

     NOW, THEREFORE, for TEN DOLLARS ($10.00) in hand paid and other good and
valuable consideration, the receipt and sufficiency of which are hereby
severally acknowledged, the parties hereto, intending to be legally bound
hereby, agree as follows:
<PAGE>   2
     1.   DEFINITIONS. All capitalized  terms used in this Amendment, unless
otherwise defined herein, shall have the meaning ascribed to such terms in the
Loan Agreement.

     2.   AMENDMENT TO LOAN AGREEMENT. The Loan Agreement is hereby amended as
follows:

     (a)  By deleting the definition of "Bridge Lender" in Section 1 of the
Loan Agreement and by substituting the following definition in lieu thereof:

          BRIDGE LENDER - Prudential Securities Credit Corporation.

     (b)  By deleting the reference to Section 7.4 in Section 7.3 of the Loan
Agreement and by substituting a reference to Section 7.3 in lieu thereof.

     (c)  By deleting the phrase "Except as set forth on Schedule 9.12 hereto
on the date thereof." in the first sentence of Section 9.1.12 of the Loan
Agreement.

     (d)  By deleting Section 10.3.1 of the Loan Agreement in its entirety and
by substituting the following new Section 10.3.1 in lieu thereof:

                         10.3.1.   CONSOLIDATED TANGIBLE NET WORTH.
          Maintain, as of the end of each Fiscal Quarter, Consolidated Tangible
          Net Worth of not less than the amount shown below for the period
          corresponding thereto:


          PERIOD                                  AMOUNT
          ------                                  ------
          Fiscal Quarter ending                   $99,000,000
          July 3, 1999

          Fiscal Quarter ending                   $101,000,000
          October 2, 1999

          Each Fiscal Quarter thereafter          $101,000,000 plus $4,000,000
                                                  for each additional Fiscal
                                                  Quarter after October 2, 1999

     3.   LIMITED WAIVER OF DEFAULT. An Event of Default has occurred and
currently exists under the Loan Agreement as a result of Borrowers' breach of
Sections 10.3.1 and 12.1.18 of the Loan Agreement (the "Designated Defaults").
The Designated Defaults exist because of (i) Borrowers' failure to maintain the
required Consolidated Tangible Net Worth set forth in Section 10.3.1 for
Borrower's Fiscal Quarter ended July 3, 1999 and (i) Borrower's default under
the

                                      -2-
<PAGE>   3
NationsBank Loan Documents. Each Borrower represents and warrants that the
Designated Defaults are the only Defaults or Events of Default that exists under
the Loan Agreement and the other Loan Documents as of the date hereof. Agent and
Lenders hereby waive the Designated Defaults in existence on the date hereof. In
no event shall such waiver be deemed to constitute a waiver of (a) any Default
or Event of Default other than the Designated Defaults in existence on the date
of this Amendment or (b) each Borrower's obligation to comply with all of the
terms and conditions of the Loan Agreement and the other Loan Documents from and
after the date hereof. Notwithstanding any prior, temporary mutual disregard of
the terms of any contracts between the parties, each Borrower hereby agrees that
it shall be required strictly to comply with all of the terms of the Loan
Documents on and after the date hereof.

     4.   Acknowledgements and Stipulations. Each Borrower acknowledges and
stipulates that the Loan Agreement and the other Loan Documents executed by such
Borrower are legal, valid and binding obligations of such Borrower that are
enforceable against such Borrower in accordance with the terms thereof; all of
the Obligations are owing and payable without defense, offset or counterclaim
(and to the extent there exists any such defense, offset or counterclaim on the
date hereof, the same is hereby waived by each Borrower); the security interests
and liens granted by each Borrower in favor of Agent are duly perfected, first
priority security interests and liens.

     5.   Representations and Warranties. Each Borrower represents and
warrants to Agent and Lenders, to induce Agent and Lenders to enter into this
Amendment, that no Default or Event of Default exists on the date hereof (other
than the Designated Defaults); the execution, delivery and performance of this
Amendment have been duly authorized by all requisite corporate action on the
part of such Borrower and this Amendment has been duly executed and delivered
by such Borrowers; and all of the representations and warranties made by
Borrowers in the Loan Agreement are true and correct on and as of the date
hereof, except to the extent any representation or warranty specifically
relates to an earlier date.

     6.   Expenses of Agent. Borrowers jointly and severally agree to pay, on
demand, all costs and expenses incurred by Agent in connection with the
preparation, negotiation and execution of this Amendment and any other Loan
Documents executed pursuant hereto and any and all amendments, modifications,
and supplements thereto, including, without limitation, the reasonable costs
and fees of Agent's legal counsel and any taxes or expenses associated with or
incurred in connection with any instrument or agreement referred to herein or
contemplated hereby.

     7.   Effectiveness; Governing Law. This Amendment shall be effective upon
acceptance by Agent and Lenders in Atlanta, Georgia (notice of which acceptance
is hereby waived), whereupon the same shall be governed by and construed in
accordance with the internal laws of the State of Georgia.

                                      -3-
<PAGE>   4

     8.   Successor and Assigns. This Amendment shall be binding upon and
inure to the benefit of the parties hereto and their respective successors and
assigns.

     9.   No Novation, Etc.. Except as otherwise expressly provided in this
Amendment, nothing herein shall be deemed to amend or modify any provision of
the Loan Agreement or any of the other Loan Documents, each of which shall
remain in full force and effect. This Amendment is not intended to be, nor
shall it be construed to create, a novation or accord and satisfaction, and the
Loan Agreement as herein modified shall continue in full force and effect.

     10.  Counterparts; Telecopied Signatures. This Amendment may be executed
in any number of counterparts and by different parties to this Agreement on
separate counterparts, each of which, when so executed, shall be deemed as
original, but all such counterparts shall constitute one and the same
agreement. Any signature delivered by a party by facsimile transmission shall
be deemed to be an original signature hereto.

     11.  Further Assurances. Each Borrower agrees to take such further actions
as Agent and Lenders shall reasonably request from time to time in connection
herewith to evidence or give effect to the amendments set forth herein or any
of the transactions contemplated hereby.

     12.  Section Titles. Section titles and references used in this Amendment
shall be without substantive meaning or content of any kind whatsoever and are
not a part of the agreement among the parties hereto.

     13.  Release of Claims. To induce Agent and Lenders to enter into this
Amendment, each Borrower hereby release, acquits and forever discharges Agent
and Lenders, and all officers, directors, agents, employees, successors and
assigns of Agent and Lenders, from any and all liabilities, claims, demands,
actions or causes or actions of any kind or nature (if there be any), whether
absolute or contingent, disputed or undisputed, at law or in equity, or known
or unknown, that such Borrower now has or ever had against Agent and Lenders
arising under or in connection with any of the Loan Documents or otherwise.



                  [Remainder of page intentionally left blank]

                                      -4-
<PAGE>   5
     14.     Waiver of Jury Trial. To the fullest extent permitted by
applicable law, the parties hereto each hereby waives the right to trial by
jury in any action, suit, comparison or preceding arising out of or related to
this Amendment.

IN WITNESS WHEREOF, the parties hereto have caused this Amendment to be duly
executed under seal and delivered by their respective duly authorized officers
on the date firm written above.


                                      BORROWERS:

ATTEST:                               TROPICAL SPORTSWEAR INT'L
                                      CORPORATION

/s/ Regina M. Ifland                  By: /s/ N. Larry McPherson
-----------------------------             -------------------------------------
Assistant Secretary
(CORPORATE SEAL)                          Title: EVP Finance and Administration
                                                 ------------------------------



ATTEST:                               TROPICAL SPORTSWEAR COMPANY,
                                      INC.

/s/ Regina M. Ifland                  By: /s/ N. Larry McPherson
-----------------------------             -------------------------------------
Assistant Secretary
(CORPORATE SEAL)                          Title: EVP Finance and Administration
                                                 ------------------------------


ATTEST:                               SAVANE INTERNATIONAL CORP.
                                      (d/b/a Farah Incorporated)

/s/ Regina M. Ifland                  By: /s/ N. Larry McPherson
-----------------------------             -------------------------------------
Secretary
(CORPORATE SEAL)                          Title:  EVP Finance and Administration
                                                 ------------------------------



                 [Signature is continued on the following page]


                                      -5-
<PAGE>   6
ATTEST:
/s/ Regina M. Ifland
--------------------------       APPAREL NETWORK CORPORATION
Assistant Secretary              By:  /s/ N. Larry McPherson
(CORPORATE SEAL)                    -----------------------------------------
                                 Title: EVP Finance and Administration
                                       --------------------------------------



                                 LENDERS:

                                 FLEET CAPITAL CORPORATION


                                 By: /s/ Elizabeth L. Waller
                                    -----------------------------------------
                                 Title: SVP
                                       --------------------------------------



                                 NATIONSBANC COMMERCIAL CORPORATION



                                 By: /s/ Andrea Jackson
                                    -----------------------------------------
                                 Title: VP
                                       --------------------------------------



                                 FIRST UNION NATIONAL BANK


                                 By: /s/ John T. Trainor
                                    -----------------------------------------
                                 Title: Vice President
                                       --------------------------------------



                                 DEUTSCHE FINANCIAL SERVICES CORPORATION


                                 By: /s/ Paula D. Patrick
                                    -----------------------------------------
                                 Title: Vice President
                                       --------------------------------------



                  [Signatures continued on the following page]


                                      -6-


<PAGE>   7
                                        AGENT:

                                        FLEET CAPITAL CORPORATION,
                                        as Agent


                                        By: /s/ Elizabeth L. Waller
                                           -------------------------------
                                        Title: SVP
                                              ----------------------------



                                      -7-