<PAGE>
                                                                   Exhibit 10.41

                         FIRST AMENDMENT TO AMENDED AND
                      RESTATED LOAN AND SECURITY AGREEMENT

THIS FIRST AMENDMENT TO AMENDED AND RESTATED LOAN AND SECURITY AGREEMENT (this
"Amendment") is made and entered into this 12th day of January, 2004, by and
between TROPICAL SPORTSWEAR INT'L CORPORATION, a Florida corporation
(hereinafter referred to as "Borrower") with its chief executive office and
principal place of business at 4902 West Waters Avenue, Tampa, Florida 33634,
and FLEET CAPITAL CORPORATION, a Rhode Island corporation (hereinafter referred
to as "Lender") with an office at 300 Galleria Parkway, Suite 800, Atlanta,
Georgia 30339.

                                    RECITALS:

Lender and Borrower are parties to a certain Amended and Restated Loan and
Security Agreement dated September 9, 2003 (as at any time amended, the "Loan
Agreement"), pursuant to which Lender has made a certain term loan to Borrower.

         Borrower has requested that Lender make available to Borrower an
additional term loan in the principal amount of $2,000,000. Subject to the terms
and conditions set forth herein, Lender is willing to make such an additional
term loan to Borrower.

NOW, THEREFORE, for TEN DOLLARS ($10.00) in hand paid and other good and
valuable consideration, the receipt and sufficiency of which are hereby
severally acknowledged, the parties hereto, intending to be legally bound
hereby, agree as follows:

         1. DEFINITIONS. All capitalized terms used in this Amendment, unless
         otherwise defined herein, shall have the meaning ascribed to such terms
         in the Loan Agreement.

         2. AMENDMENTS TO LOAN AGREEMENT. The Loan Agreement is hereby amended
         as follows:

         (a) By deleting the definitions of "Applicable Margin," "Mortgage,"
         "Term Loan" and "Term Note" contained in Section 1.1 of the Loan
         Agreement and by substituting in lieu thereof the following new
         definitions of "Mortgage," "Term Loan" and "Term Note":

                           "Applicable Margin" shall mean (a) as to Base Rate
                  Loans, 2.00%, and (b) as to LIBOR Loans, 4.50%.

                           "Mortgage" shall mean the Amended and Restated
                  Florida Mortgage, Fixture Filing, Security Agreement and
                  Assignment of Rents and Leases to be executed by Borrower in
                  favor of Lender on or about the Closing Date and by which the
                  Existing Mortgage shall be modified to, among other things,
                  provide for a Lien on the Tropical Headquarters and reflect
                  the effectiveness of the Loan Assignment.

                           "Term Loan" shall mean, collectively, (i) Term Loan A
                  and (iii) Term Loan B.

                           "Term Note" shall mean, collectively, (i) Term Note A
                  and (ii) Term Note B.

         (b) By adding the following definitions of "First Amendment Closing
         Date," "Term Loan A," "Term Loan B," "Term Note A," and "Term Note B"
         to Section 1.1 of the Loan Agreement, in proper alphabetical sequence:

                           "First Amendment Closing Date" shall mean January
                  12th, 2004.

                           "Term Loan A" shall mean, collectively, (i) the
                  Existing Term Loan, (ii) the New Term Loan.

                                        1

<PAGE>

                           "Term Loan B" shall have the meaning ascribed to it
                  in Section 2.1(b) hereof.

                           "Term Note A" shall mean the Amended and Restated
                  Renewal and Increase Term Note made by Borrower to the order
                  of Lender evidencing the obligation of Borrower to pay the
                  aggregate unpaid principal amount of Term Loan A (and any
                  promissory note or notes that may be issued from time to time
                  in substitution, renewal, extension, replacement or exchange
                  therefor), substantially in the form of EXHIBIT A hereto, with
                  all blanks properly completed, either as originally executed
                  or as the same may from time to time be supplemented,
                  modified, amended, renewed, extended or refinanced

                           "Term Note B" shall mean the Term Note B made by
                  Borrower to the order of Lender evidencing the obligation of
                  Borrower to pay the aggregate unpaid principal amount of Term
                  Loan B (and any promissory note or notes that may be issued
                  from time to time in substitution, renewal, extension,
                  replacement or exchange therefor), substantially in the form
                  of Exhibit A-1 hereto, with all blanks properly completed,
                  either as originally executed or as the same may from time to
                  time be supplemented, modified, amended, renewed, extended or
                  refinanced.

         (c) By deleting Section 2.1 of the Loan Agreement and by substituting
         in lieu thereof the following new Section 2.1:

                  Section 2.1 Existing Term Loan; Manner of Borrowing and
                  Disbursing New Term Loan and Term Loan B.

                                    (a) Subject to and upon the terms and
                  conditions set forth in the Existing Loan Agreement, Bank of
                  America made the Existing Term Loan available to Borrower. The
                  Existing Term Loan was assigned by Bank of America to Lender
                  pursuant to the Loan Assignment. The aggregate principal
                  amount of the Existing Term Loan on the Closing Date was
                  $7,000,000. Subject to and upon the terms and conditions set
                  forth in the Loan Agreement, Lender made a Base Rate Loan to
                  Borrower on the Closing Date in a principal amount equal to
                  $1,000,000 (the "New Term Loan"). The proceeds of the New Term
                  Loan were used by Borrower solely to repay the Swap
                  Obligation.

                                    (b) Subject to and upon the terms and
                  conditions of, and in reliance upon the representations and
                  warranties made under, this Agreement, Lender agrees to make a
                  Base Rate Loan to Borrower on the First Amendment Closing Date
                  in a principal amount equal to $2,000,000 ("Term Loan B").
                  Upon satisfaction of the applicable conditions set forth in
                  Section 8 of the First Amendment to Amended and Restated Loan
                  and Security Agreement, Lender shall make the proceeds of Term
                  Loan B available to Borrower on the First Amendment Closing
                  Date.

                                    (c) Borrower shall not be entitled to
                  reborrow any amounts repaid with respect to the Term Loan.

         (d) By deleting Section 2.2 of the Loan Agreement and by substituting
         in lieu thereof the following new Section 2.2:

                  Section 2.2 Term Note.

                                    (a) The obligation of Borrower to repay Term
                  Loan A shall also be evidenced by Term Note A. Term Note A
                  shall be dated the Closing Date and duly executed and
                  delivered by Borrower.

                                    (b) The obligation of Borrower to repay Term
                  Loan B shall also be evidenced by Term Note B. Term Note B
                  shall be dated the First Amendment Closing

                                        2
<PAGE>

                  Date and duly executed and delivered by Borrower.

         (e) By deleting Section 2.3 of the Loan Agreement and by substituting
         in lieu thereof the following new Section 2.3:

                  Section 2.3 Repayment of Principal of Term Loan.

                                    (a) The principal amount of Term Loan A is
                  due and payable, and shall be repaid in full by Borrower, in
                  consecutive quarterly installments of $200,000 each, payable
                  on the first day of each calendar quarter, commencing on
                  October 1, 2003; provided that the unpaid principal balance of
                  Term Loan A is due and payable, and shall be repaid in full by
                  Borrower, on the Maturity Date or, if sooner, on the
                  Commitment Termination Date.

                                    (b) The principal amount of Term Loan B is
                  due and payable, and shall be repaid in full by Borrower, in
                  consecutive quarterly installments of $50,000 each, payable on
                  the first day of each calendar quarter, commencing on April 1,
                  2004; provided that the unpaid principal balance of Term Loan
                  B is due and payable, and shall be repaid in full by Borrower,
                  on the Maturity Date or, if sooner, on the Commitment
                  Termination Date.

         (f) By adding Exhibit A-1 attached hereto to the Loan Agreement,
         immediately following Exhibit A thereto.

         3. RATIFICATION AND REAFFIRMATION. Borrower hereby ratifies and
reaffirms the Obligations, each of the Loan Documents and all of Borrower's
covenants, duties, indebtedness and liabilities under the Loan Documents.

         4. INTEREST RATE DISCLOSURE. The Base Rate on the date hereof is 4.00%
per annum and, therefore, the rate of interest in effect hereunder on the date
hereof, expressed in simple interest terms is 6.00% per annum with respect to
any portion of the Loans bearing interest as a Base Rate Loan.

         5. ACKNOWLEDGMENTS AND STIPULATIONS. Borrower acknowledges and
stipulates that the Loan Agreement and the other Loan Documents executed by
Borrower are legal, valid and binding obligations of Borrower that are
enforceable against Borrower in accordance with the terms thereof; all of the
Obligations are owing and payable without defense, offset or counterclaim (and
to the extent there exists any such defense, offset or counterclaim on the date
hereof, the same is hereby waived by Borrower); the security interests and liens
granted by Borrower in favor of Lender are duly perfected, first priority
security interests and liens; and the unpaid principal amount of Term Loan A on
and as of January 9, 2004, totaled $7,600,000.

         6. REPRESENTATIONS AND WARRANTIES. Borrower represents and warrants to
Lender, to induce Lender to enter into this Amendment, that no Default or Event
of Default exists on the date hereof; the execution, delivery and performance of
this Amendment have been duly authorized by all requisite corporate action on
the part of Borrower and this Amendment has been duly executed and delivered by
Borrower; and all of the representations and warranties made by Borrower in the
Loan Agreement are true and correct on and as of the date hereof.

         7. REFERENCE TO LOAN AGREEMENT. Upon the effectiveness of this
Amendment, each reference in the Loan Agreement to "this Agreement,"
"hereunder," or words of like import shall mean and be a reference to the Loan
Agreement, as amended by this Amendment.

         8. BREACH OF AMENDMENT. This Amendment shall be part of the Loan
Agreement and a breach of any representation, warranty or covenant herein shall
constitute an Event of Default.

         9. CONDITIONS PRECEDENT. The effectiveness of the amendments contained
in Section 2 hereof are subject to the satisfaction of each of the following
conditions precedent, in form and substance satisfactory to Lender, unless
satisfaction thereof is specifically waived in writing by Lender:

                  (a) No Default or Event of Default shall exist;

                                        3
<PAGE>

                  (b) The Revolver Lenders shall have received a duly executed
         Second Amended and Restated Loan and Security Agreement from Borrower
         and the other borrowers party to the Revolving Loan Agreement, in form
         and substance satisfactory to them;

                  (c) Lender shall have received a duly executed original of
         this Amendment from Borrower, together with a Consent and Reaffirmation
         duly signed by Tropical Sportswear Company, Inc., Savane International
         Corp. and Apparel Network Corporation;

                  (d) Lender shall have received a duly executed original Term
         Note B from Borrower in the form attached to the Loan Agreement as
         Exhibit A-1;

                  (e) Lender shall have received certified copies of resolutions
         of Borrower's board of directors authorizing the execution of this
         Amendment, Term Note B and any other Loan Documents and each document
         required to be delivered by any Section hereof;

                  (f) Lender shall have received a duly executed amendment to
         the Mortgage from Borrower, in form and substance satisfactory to
         Lender;

                  (g) Lender shall have received endorsements or commitments for
         endorsements to the existing mortgagee title insurance policies
         insuring the Liens of the Mortgage, which shall be in form and
         substance satisfactory to Lender and which shall give effect to the
         mortgage amendment described in the foregoing clause (f);

                  (h) Lender shall have received reimbursement from Borrower for
         the payment of all applicable documentary stamp, intangibles,
         recording, note or other similar taxes payable with respect to the
         mortgage amendment described in clause (f); and

                  (i) Borrower shall have taken such other actions as Lender may
         reasonably request.

         10. EXPENSES OF LENDER. Borrower agrees to pay, ON DEMAND, all costs
and expenses incurred by Lender in connection with the preparation, negotiation
and execution of this Amendment and any other Loan Documents executed pursuant
hereto and any and all amendments, modifications, and supplements thereto,
including, without limitation, the costs and fees of Lender's legal counsel and
any taxes or expenses associated with or incurred in connection with any
instrument or agreement referred to herein or contemplated hereby.

         11. EFFECTIVENESS; GOVERNING LAW. This Amendment shall be effective
upon acceptance by Lender in Atlanta, Georgia (notice of which acceptance is
hereby waived), whereupon the same shall be governed by and construed in
accordance with the internal laws of the State of Georgia.

         12. SUCCESSORS AND ASSIGNS. This Amendment shall be binding upon and
inure to the benefit of the parties hereto and their respective successors and
assigns.

         13. NO NOVATION, ETC.. Except as otherwise expressly provided in this
Amendment, nothing herein shall be deemed to amend or modify any provision of
the Loan Agreement or any of the other Loan Documents, each of which shall
remain in full force and effect. This Amendment is not intended to be, nor shall
it be construed to create, a novation or accord and satisfaction, and the Loan
Agreement as herein modified shall continue in full force and effect.

         14. COUNTERPARTS; TELECOPIED SIGNATURES. This Amendment may be executed
in any number of counterparts and by different parties to this Amendment on
separate counterparts, each of which, when so executed, shall be deemed an
original, but all such counterparts shall constitute one and the same agreement.
Any signature delivered by a party by facsimile transmission shall be deemed to
be an original signature hereto.

         15. FURTHER ASSURANCES. Borrower agrees to take such further actions as
Lender shall reasonably request from time to time in connection herewith to
evidence or give effect to the amendments set forth herein or any

                                        4
<PAGE>

of the transactions contemplated hereby.

         16. SECTION TITLES. Section titles and references used in this
Amendment shall be without substantive meaning or content of any kind whatsoever
and are not a part of the agreements among the parties hereto.

         17. RELEASE OF CLAIMS. TO INDUCE LENDER TO ENTER INTO THIS AMENDMENT,
BORROWER HEREBY RELEASES, ACQUITS AND FOREVER DISCHARGES LENDER, AND ALL
OFFICERS, DIRECTORS, AGENTS, EMPLOYEES, SUCCESSORS AND ASSIGNS OF LENDER, FROM
ANY AND ALL LIABILITIES, CLAIMS, DEMANDS, ACTIONS OR CAUSES OF ACTION OF ANY
KIND OR NATURE (IF THERE BE ANY), WHETHER ABSOLUTE OR CONTINGENT, DISPUTED OR
UNDISPUTED, AT LAW OR IN EQUITY, OR KNOWN OR UNKNOWN, THAT BORROWER NOW HAS OR
EVER HAD AGAINST LENDER ARISING UNDER OR IN CONNECTION WITH ANY OF THE LOAN
DOCUMENTS OR OTHERWISE. BORROWER REPRESENTS AND WARRANTS TO LENDER THAT BORROWER
HAS NOT TRANSFERRED OR ASSIGNED TO ANY PERSON ANY CLAIM THAT BORROWER EVER HAD
OR CLAIMED TO HAVE AGAINST LENDER.

                  [Remainder of Page Left Intentionally Blank.]

                                        5
<PAGE>

         18. WAIVER OF JURY TRIAL. TO THE FULLEST EXTENT PERMITTED BY APPLICABLE
LAW, THE PARTIES HERETO EACH HEREBY WAIVES THE RIGHT TO TRIAL BY JURY IN ANY
ACTION, SUIT, COUNTERCLAIM OR PROCEEDING ARISING OUT OF OR RELATED TO THIS
AMENDMENT.

IN WITNESS WHEREOF, the parties hereto have caused this Amendment to be duly
executed under seal and delivered by their respective duly authorized officers
on the date first written above.

ATTEST:                             TROPICAL SPORTSWEAR INT'L CORPORATION

/s/ Karen S. Castillo               By: /s/ Robin J. Cohan
----------------------------            ------------------------------------
KAREN S. CASTILLO, Secretary            ROBIN J. COHAN, Executive Vice
                                        President, Chief Financial Officer
                                        and Treasurer

                                    Accepted in Atlanta, Georgia, this 12th day
                                    of January, 2004:

                                    FLEET CAPITAL CORPORATION

                                    By: /s/ Elizabeth Waller
                                        -----------------------------------
                                      Title: Senior Vice President

                            CONSENT AND REAFFIRMATION

The undersigned guarantors of the Obligations of Borrower at any time owing to
Lender hereby (i) acknowledge receipt of a copy of the foregoing First Amendment
to Amended and Restated Loan and Security Agreement; (ii) consent to Borrower's
execution and delivery thereof and of the other documents, instruments or
agreements Borrower agrees to execute and deliver pursuant thereto; (iii) agree
to be bound thereby; and (iv) affirm that nothing contained therein shall modify
in any respect whatsoever its respective guaranty of the Obligations and
reaffirm that such guaranty is and shall remain in full force and effect.

IN WITNESS WHEREOF, each of the undersigned has executed this Consent and
Reaffirmation as of the date of such First Amendment to Amended and Restated
Loan and Security Agreement.

ATTEST:                             TROPICAL SPORTSWEAR COMPANY, INC.

/s/ Karen S. Castillo               By: /s/ Robin J. Cohan
---------------------                   ------------------
KAREN S. CASTILLO, ASSISTANT            ROBIN J. COHAN, EXECUTIVE VICE

                                       -6-

<PAGE>

SECRETARY                               President and Chief Financial Officer

ATTEST:                             SAVANE INTERNATIONAL CORP.

/s/ Karen S. Castillo               By: /s/ Robin J. Cohan
---------------------------             ------------------------------------
KAREN S. CASTILLO, Secretary            ROBIN J. COHAN, Executive Vice
                                        President, Chief Financial Officer
                                        and Treasurer

ATTEST:                             APPAREL NETWORK CORPORATION

/s/ Karen S. Castillo               By: /s/ Robin J. Cohan
-------------------------------         -------------------------------------
KAREN S. CASTILLO, Secretary            ROBIN J. COHAN, Executive Vice
                                        President, Chief Financial Officer
                                        and Treasurer

                                        7
<PAGE>

                                   EXHIBIT A-1
                                       TO
                AMENDED AND RESTATED LOAN AND SECURITY AGREEMENT
                               FORM OF TERM NOTE B

U.S.$2,000,000                                         January 12, 2004
                                                        Atlanta, Georgia

         FOR VALUE RECEIVED, the undersigned, TROPICAL SPORTSWEAR INT'L
CORPORATION, a Florida corporation (herein called the "Borrower"), hereby
promises to pay to the order of FLEET CAPITAL CORPORATION, a Rhode Island
corporation (herein, together with any subsequent holder hereof, called the
"Holder"), the principal sum of TWO MILLION AND 00/100 DOLLARS ($2,000,000), in
such amounts and on such dates specified in SECTION 2.3 of the Loan Agreement,
in strict accordance with the terms thereof. Borrower likewise unconditionally
promises to pay to Holder interest from and after the date hereof on the unpaid
principal balance hereof at such interest rates, payable at such times and
computed in such manner as are specified in SECTIONS 2.4, 2.5 and 2.10 of the
Loan Agreement, in strict accordance with the terms thereof.

         This Term Note B ("Note") is issued pursuant to, and is a "Term Note"
referred to in, the Amended and Restated Loan and Security Agreement, dated
September 9, 2003, as amended by that certain First Amendment to Amended and
Restated Loan and Security Agreement, dated the date hereof (as at any time
amended, the "Loan Agreement"), between Borrower and Holder, and Holder is and
shall be entitled to all benefits thereof and of all Loan Documents executed and
delivered in connection therewith. All capitalized terms used herein, unless
otherwise defined herein, shall have the meanings ascribed to such terms in the
Loan Agreement.

         This Note is subject to optional prepayment in accordance with the
provisions of SECTION 2.18 of the Loan Agreement. Notwithstanding anything to
the contrary contained herein, the entire unpaid principal balance of and
accrued interest on this Note shall be due and payable immediately on the
Commitment Termination Date.

         All payments of principal and interest shall be made in U.S. Dollars
and in immediately available funds to Holder at the office designated by Holder
from time to time.

         Upon or after the occurrence of an Event of Default, the principal
balance and all accrued interest of this Note may be declared (or shall become)
due and payable in the manner and with the effect provided in the Loan
Agreement. If this Note is collected by or through an attorney at law, then
Borrower shall be obligated to pay, in addition the principal balance and
accrued interest hereof, reasonable attorneys' fees, expenses and court costs.
From and after the occurrence of an Event of Default, the outstanding principal
amount hereof shall bear interest at the Default Rate.

         In no contingency or event whatsoever, whether by reason of advancement
of the proceeds hereof or otherwise, shall the amount paid or agreed to be paid
to Holder for the use, forbearance or detention of money advanced hereunder
exceed the highest lawful rate permissible under any law which a court of
competent jurisdiction may deem applicable hereto; and, in the event of any such
payment inadvertently paid by Borrower or inadvertently received by Holder, such
excess sum shall be, at Borrower's option, returned to Borrower forthwith or
credited as a payment of principal, but shall not be applied to the payment of
interest. It is the

                                        8
<PAGE>

intent hereof that Borrower not pay or contract to pay, and that Holder not
receive or contract to receive, directly or indirectly in any manner whatsoever,
interest in excess of that which may be paid by Borrower under Applicable Law.

         Time is of the essence of this Note. To the fullest extent permitted by
Applicable Law, Borrower, for itself and its legal representatives, successors
and assigns, expressly waives presentment, demand, protest, notice of dishonor,
notice of non-payment, notice of maturity, notice of protest, presentment for
the purpose of accelerating maturity, diligence in collection, and the benefit
of any exemption or insolvency laws.

         Wherever possible each provision of this Note shall be interpreted in
such a manner as to be effective and valid under Applicable Law, but if any
provision of this Note shall be prohibited or invalid under Applicable Law, such
provision shall be ineffective to the extent of such prohibition or invalidity
without invalidating the remainder of such provision or remaining provisions of
this Note. No delay or failure on the part of Holder in the exercise of any
right or remedy hereunder shall operate as a waiver thereof, nor as an
acquiescence in any default, nor shall any single or partial exercise by Holder
of any right or remedy preclude any other or further exercise of such right or
remedy or the exercise of any other right or remedy. Holder, at its option, may
enforce its rights against any collateral securing this Note without enforcing
its rights against Borrower, any Guarantor of the indebtedness evidenced hereby
or any other property or indebtedness due or to become due to Borrower. Borrower
agrees that, without releasing or impairing Borrower's liability hereunder,
Holder may at any time release, surrender, substitute or exchange any collateral
securing this Note and may at any time release any party primarily or
secondarily liable for the indebtedness evidenced by this Note.

         The rights of Holder and obligations of Borrower hereunder shall be
construed in accordance with and governed by the laws (without giving effect to
the conflict of law principles thereof) of the State of Georgia. This Note is
intended to take effect as an instrument under seal under Georgia law.

         IN WITNESS WHEREOF, Borrower has caused this Note to be executed and
delivered by its duly authorized officer, on the date first above written.

ATTEST:                            TROPICAL SPORTSWEAR INT'L CORPORATION

/s/ Karen S. Castillo              By: /s/ Robin J. Cohan
---------------------                  ----------------------------------------
KAREN S. CASTILLO, Secretary           ROBIN J. COHAN, Executive Vice President,
(CORPORATE SEAL)                        Chief Financial Officer and Treasurer

Florida Documentary Stamp Taxes in the amount of $____________________ have been
paid in connection with the recording of the First Amendment to Amended and
Restated Florida Mortgage, Fixture Filing, Security Agreement and Assignment of
Rents and Leases, which modifies the Mortgage that secures this Note.

                                        9