<PAGE>   1



                                                                     EXHIBIT 4.1


                       SECOND AMENDMENT TO LOAN AGREEMENT
                            AND OTHER LOAN DOCUMENTS


         This Second Amendment to Loan Agreement and Other Loan Documents, dated
as of March 27, 2001 (this "Amendment"), is entered into by and between SUPERIOR
UNIFORM GROUP, INC., a Florida corporation (together with all Subsidiaries and
all Affiliates, as herein defined, "Borrower"), and FIRST UNION NATIONAL BANK, a
national banking association ("Lender").

                                    RECITALS

         A.       Borrower and Lender are parties to that certain Loan Agreement
dated as of March 26, 1999, as amended pursuant to the terms of that certain
First Amendment to Loan Agreement and Revolving Credit Note dated as of October
16, 2000 by and between Borrower and Lender (as so amended, the "Loan
Agreement"), pursuant to the terms of which Lender has made (a) a revolving
credit loan to Borrower in the maximum principal amount of $15,000,000.00
("Revolving Loan"), as evidenced by that certain Revolving Credit Note in the
original maximum principal amount of $15,000,000.00 dated March 26, 1999 made by
Borrower payable to the order of Lender ("Revolving Note"), and (b) term loans
to Borrower in the principal amounts of $12,000,000.00 and $5,000,000.00, as
evidenced by that certain Term Note in the original principal amount of
$12,000,000.00 dated as of March 26, 1999 made by Borrower payable to the order
of Lender and that certain Term Note in the original principal amount of
$5,000,000.00 dated as of October 16, 2000 made by Borrower payable to the order
of Lender ("Term Notes," and the Term Notes, together with the Revolving Note,
the Loan Agreement and all other documents executed in connection herewith or
therewith, collectively, the "Loan Documents"). Capitalized terms used herein
but not otherwise defined herein shall have the meanings assigned to such terms
in the Loan Agreement.

         B.       Borrower has requested that Lender renew and amend the terms
of the Revolving Loan pursuant to the terms of that certain Renewal Revolving
Credit Note dated as of the date hereof in the maximum principal amount of
[$15,000,000.00] made by Borrower payable to the order of Lender ("Renewal
Revolving Note"), and Lender has agreed to renew and amend the terms of the
Revolving Loan pursuant to the Renewal Revolving Note, in accordance with, and
conditioned upon, the terms of this Amendment.

                                    AGREEMENT

         In consideration of the foregoing and the mutual covenants and
agreements set forth herein and for other good and valuable consideration, the
receipt and sufficiency of which are hereby acknowledged, the parties, intending
to be bound hereby, agree as follows:

         1.       Modification of Loan Agreement. The Loan Agreement is amended
as follows:

                  (a)      the definition of "Revolving Credit Note" is hereby
deleted in its entirety and replaced with the following:

                  "Revolving Credit Note: that certain Renewal Revolving Credit
                  in the maximum principal amount of [$15,000,000.00] dated as
                  of March 27, 2001 made by Borrower payable to the order of
                  Lender, substantially in the form of Exhibit A hereto,
                  together with any and all amendments, modifications,
                  extensions, substitutions and renewals thereof."

                  (b)      the definition of "Revolving Credit Loan Expiration
Date" is hereby deleted in its entirety and replaced with the following:

                  "Revolving Credit Loan Expiration Date: The "Maturity Date" as
                  defined in the Revolving Credit Note."

<PAGE>   2

                  (c)      Exhibit A to the Loan Agreement is hereby deleted and
replaced with Exhibit A attached to this Amendment.

                  (d)      Section 5.7 (b)(ii) of the Loan Agreement is hereby
deleted in its entirety and replaced with the following:

                  "(ii) the Revolving Credit Loan has terminated pursuant to the
                  terms of this Agreement or Other Agreements including, without
                  limitation, a termination on the Revolving Credit Loan
                  Expiration Date."

         Modification of Other Loan Documents. All other Loan Documents and
Other Agreements are hereby modified as follows:

                  (e)      In addition to all other obligations of payment and
performance by Borrower as secured by such Loan Documents, the Loan Documents
shall secure the payment of the indebtedness evidenced by the Revolving Note, as
renewed and modified by the Renewal Revolving Note; and

                  (f)      The term "Revolving Credit Note" as used in all other
Loan Documents shall mean, the Revolving Note as modified and renewed pursuant
to the terms of the Renewal Revolving Note, together with all modifications,
amendments, extensions and renewals thereof.

         Tax Indemnity. Borrower and Lender have concluded that Florida document
excise taxes are not due in connection with this Amendment or any of the other
Loan Documents because the Loan Documents have been executed by Borrower and the
other signatories, and delivered to Lender, outside the State of Florida.
Nevertheless, Borrower shall pay to Lender in full, on demand, the amount of all
document excise taxes, including interest and penalties, that either Lender or
the Florida Department of Revenue later deem to be due and applicable with
respect to the Notes or any of the other Loan Documents, or any other agreement
between or among Borrower the Subsidiaries and Lender. In addition, Borrower
shall reimburse Lender for any document excise taxes, including penalties and
interest, paid by Lender and all costs and attorney's fees that Lender incurs in
defending against an imposition of such taxes on any of the Notes, this
Amendment, the other Loan Documents and any other agreement between or among
Borrower, the Subsidiaries and Lender.

         Representations and Warranties. Borrower represents and warrants to
Lender that:

                  (g)      all of Borrower's representations and warranties to
Lender in the Loan Documents are true and correct on this date, as if made on
this date, except to the extent any of them expressly relate to an earlier date;

                  (h)      since the date of the most recent financial
statements delivered to Lender, there has not been any material adverse change
in the financial conditions of Borrower or any Guarantor;

                  (i)      Borrower has the full corporate power and authority
to enter into and perform its obligations hereunder and each transaction
contemplated hereby; and

                  (j)      the execution and delivery by Borrower of this
Amendment and each other document contemplated hereby and its performance of its
obligations hereunder and thereunder have been duly authorized by all necessary
corporate proceedings on the part of Borrower.

         Counterparts. The parties may execute this Amendment and any other
agreement executed pursuant to it in counterparts. Each executed counterpart
will be deemed to be an original, and all of them, together, will constitute the
same agreement. This Amendment will become effective as of its stated date of
execution, when each party has signed a counterpart and all the executed
counterparts have been delivered to Lender.

         WAIVER OF CLAIMS. BORROWER HEREBY KNOWINGLY, VOLUNTARILY, IRREVOCABLY,
AND INTENTIONALLY WAIVES AND RELEASES LENDER (AND ITS OFFICERS, DIRECTORS,
SHAREHOLDERS, REPRESENTATIVES, AND AGENTS) FROM: (a) ALL CLAIMS, DEMANDS, SUITS,
AND CAUSES OF ACTION, WHETHER AT LAW OR IN EQUITY, THAT BORROWER EVER HAD, HAS
NOW, OR MIGHT HAVE IN THE FUTURE, BY REASON OF ANY MATTER, CAUSE, OR THING
WHATSOEVER ARISING BEFORE THE DATE AND TIME OF EXECUTION OF THIS AMENDMENT, WITH
RESPECT TO: (i) ANY BREACH BY LENDER (OR AN OFFICER, DIRECTOR, SHAREHOLDER,
REPRESENTATIVE, OR AGENT OF LENDER) OF ITS OBLIGATIONS OR PROMISES UNDER THE
LOAN DOCUMENTS OR OTHERWISE; AND (ii) ANY ACTION OR INACTION BY LENDER (OR AN
OFFICER, DIRECTOR, SHAREHOLDER, REPRESENTATIVE, OR AGENT OF

<PAGE>   3

LENDER) THAT IS ALLEGED TO HAVE HAD AN INJURIOUS EFFECT ON THE BUSINESS,
OPERATION OR MANAGEMENT OF BORROWER; AND (b) ANY DEFENSE, COUNTERCLAIM, SETOFF,
RIGHT OF RECOUPMENT OR ABATEMENT, OR OTHER CLAIM AGAINST LENDER (OR AN OFFICER,
DIRECTOR, SHAREHOLDER, REPRESENTATIVE, OR AGENT OF LENDER) RELATING TO ANY
MATTER, CAUSE, OR THING WHATSOEVER ARISING BEFORE THE DATE AND TIME OF EXECUTION
OF THIS AMENDMENT.

         Ratification of Loan Documents. The parties acknowledge that (except as
expressly amended in this Amendment) the Loan Documents are unaffected,
unchanged, and unimpaired and all such documents and agreements remain
enforceable in accordance with their respective terms. Further, the parties
ratify and confirm all their obligations under the Loan Documents, except as
modified in this Amendment. Neither this Amendment nor any earlier waiver or
amendment of any of the Loan Documents will constitute a novation or have the
effect of discharging any liability or obligation evidenced or secured by the
Loan Documents.

         Transaction Expenses; Taxes. Borrower shall pay all costs and expenses
of Lender (including filing fees, recording fees, document excise and intangible
tax, and reasonable attorney's fees and expenses) in connection with this
Amendment and any related documents.

         Miscellaneous. This Amendment contains the final, complete, and
exclusive expression of the understanding of Borrower and Lender with respect to
the obligations created under it and supersedes any prior or contemporaneous
agreement, understanding, or representation, oral or written, by either of them.
Except as expressly provided herein, this Amendment does not constitute a waiver
of any rights of Lender or obligations of Borrower under the Loan Documents, and
no waiver herein will constitute a continuing waiver or a waiver of any other or
future rights or obligations. A waiver or modification of any provision of this
Amendment is valid only if the waiver or modification is in writing and signed
by each party. The titles and headings preceding the text of the sections of
this Amendment have been inserted solely for convenience of reference and do not
affect this Amendment's meaning or effect. This Amendment is binding on each
heir, assignee, and personal representative of the Borrower, and inures to the
benefit of each assignee and successor of Lender. This Amendment is not
assignable by Borrower, and any attempted assignment by Borrower will not be
valid or effective against Lender. Lender may assign this Amendment, and its
assignee will succeed to all the rights of Lender under it. Words of the neuter
gender in this Amendment are to be construed to include words of the masculine
and feminine genders. This Amendment is a Florida contract, and the parties
intend that it is to be construed according to the laws of the State of Florida.

         IN WITNESS WHEREOF, the parties hereto have executed and delivered this
Amendment as of the date first written above.

                              FIRST UNION NATIONAL BANK,
                                a national banking association

                              By:/s/ Timothy J. Coop
                                 --------------------------------
                                       Name: Timothy J. Coop
                                             --------------------
                                       Title: Vice President
                                              -------------------


                              SUPERIOR UNIFORM GROUP, INC.,
                                a Florida corporation

                              By:   /s/ Andrew D. Demott, Jr.
                                 -----------------------------------------------
                                    Andrew D. Demott, Jr.
                                    Vice President, Chief Financial Officer and
                                    Treasurer