<PAGE>

                                                                   Exhibit 10.84

                                FOURTH AMENDMENT
                                     TO THE
                  GALEY & LORD RETIREMENT SAVINGS PLAN (401(k))

          WHEREAS, Galey & Lord, Inc. (the "Company") maintains The Galey & Lord
Retirement Savings Plan (401(k)), most recently amended and restated as of
January 1, 2000 (the "Plan").

          WHEREAS, the Employer wishes to amend the Plan.

          NOW, THEREFORE, the Plan is amended as follows:

          I. Effective with respect to Salary Deferral Contributions made on or
after March 15, 2003, Matching Employer Contributions will cease. In addition,
Section 3.3 of the Plan is revised in its entirety to read as follows:

          3.3 Employer Contributions: Effective with payrolls beginning on or
     after March 15, 2003, the Employer will not make a Matching Employer
     Contribution unless the Chief Executive Officer of the Company ("CEO"), in
     his discretion, decides otherwise.

          The CEO may, in his discretion, decide to have the Employer make a
     Matching Employer Contribution to the Plan for each payroll period on
     behalf of each Participant who makes Salary Deferral Contributions under
     the Plan.

          The amount of the Matching Employer Contribution will be equal to a
     percentage of the aggregate "eligible Salary Deferral Contributions" made
     on behalf of Participants for the payroll period. For purposes of this
     Section, "eligible Salary Deferral Contributions" with respect to a
     Participant means the Salary Deferral Contributions made on his or her
     behalf for the payroll period in an amount up to, but not exceeding, the
     "match level". For purposes of this Section, the "match level" means a
     percentage of a Participant's Compensation for the payroll period during
     which the Participant makes Salary Deferral Contributions. Each of the
     "percentages" described in this paragraph shall be determined by the CEO.

          At the end of each Plan Year, the Employer shall review the Matching
     Employer Contributions, if any, made during the year on behalf of each
     Participant who is an Employee on the last day of the Plan Year. If the
     total Matching Employer Contribution made on behalf of each such
     Participant for each payroll period during that portion of the Plan Year
     for which Matching Employer Contributions are made is less than the
     "percentages" described above, the Employer shall make a supplemental
     matching contribution on behalf of that Participant so that his total
     Matching Employer Contributions during that portion of the Plan Year for
     which Matching Employer Contributions are made equals the "percentages"
     described above. Notwithstanding the foregoing, if the supplemental
     matching contribution described in this paragraph is less than $1.00, no
     supplemental contribution will be made.

          II. Unless otherwise amended herein, the provisions of the Plan are
hereby ratified and confirmed.

<PAGE>

          III. This Amendment is effective as of March 5, 2003.

THIS AMENDMENT IS EXECUTED this 4th day of March, 2003.

                                    GALEY & LORD, INC.


                                    By: /s/ Arthur C. Wiener
                                        ----------------------------------------
                                            Arthur C. Wiener
                                            Chairman and Chief Executive Officer


ATTEST:


/s/ Leonard F. Ferro
--------------------
    Leonard F. Ferro
    Secretary