<PAGE>

                                                                   EXHIBIT 10.21

                      FIFTH AMENDMENT TO PROMISSORY NOTE


     THIS FIFTH AMENDMENT TO PROMISSORY NOTE (this "Amendment"), dated as of
                                                    ---------
March 31, 2000, is entered into between PILLOWTEX CORPORATION, a Texas
corporation ("Borrower"), and BANK OF AMERICA, N.A. (formerly known as
              --------
NationsBank, N.A.) ("Lender").
                     ------

     A.  Borrower executed that certain Promissory Note, dated May 4,1999, in
the maximum principal amount of $20,000,000, payable to the order of Lender, as
amended and increased to $35,000,000 by that certain First Amendment to
Promissory Note, dated as of July 27, 1999 (said Promissory Note, as amended,
the "Promissory Note"; the terms defined in the Promissory Note and not
     ---------------
otherwise defined herein shall be used herein as defined in the Promissory
Note).

     B.  Borrower and Lender desire to amend the Promissory Note.

     NOW, THEREFORE, in consideration of the covenants, conditions and
agreements hereafter set forth, and for other good and valuable consideration,
the receipt and adequacy of which are all hereby acknowledged, Borrower and
Lender covenant and agree as follows:

     1.  AMENDMENTS TO PROMISSORY NOTE.  The Promissory Note is hereby amended,
         -----------------------------
as follows:

          (a)  Section 1.8(c) is entirely amended, as follows:

               The principal amount of all Advances outstanding hereunder shall
          be due and payable in equal quarterly installments of $87,500,
          together with accrued and unpaid interest thereon, on the last day of
          each March, June, September and December, commencing March 31, 2000,
          with a final payment of all outstanding principal, accrued interest
          and fees thereon, due and payable on the Maturity Date.
          Notwithstanding anything herein to the contrary, each installment
          payment hereunder shall constitute a permanent reduction to the
          Commitment in the amount of such payment.

          (b)  The following definitions set forth in Article III of the
     Promissory Note are hereby amended to read as follows:

          "Base Rate" means an interest rate per annum for each day equal to the
           ---------
     sum of (a) the Base Rate Margin plus (b) higher of (i) the Prime Rate in
     effect on such day or (ii) the Federal Funds Rate in effect on such day
     plus 1/2%, each without notice to Borrower.

          "Eurodollar Rate" means an interest rate per annum equal to the sum of
           ---------------
     (i) the Eurodollar Rate Margin plus (ii) a rate per annum determined
     pursuant to the following formula:

                               London Interbank Rate
                    -----------------------------------------
                      100% - Eurodollar Reserve Percentage
<PAGE>

          "Maturity Date" means the earliest of (a) the occurrence of a
           -------------
     Prepayment Event, (b) January 31, 2002, or (c) the date of termination in
     whole of the Commitment hereunder.

          (c) The following definitions are added to Article III of the
     Promissory Note and inserted in appropriate alphabetical order therein as
     follows:

          "Base Rate Margin" means 3.000%, provided that the Base Rate Margin
           ----------------                -------------
     shall be decreased to 2.500% as of the first day of the calendar month
     following the date the Borrower delivers its annual financial statements
     for Fiscal Year 2000 to the Lender, if such financial statements show
     EBITDA (plus all fees and expenses incurred in connection with the
     restructuring of any Indebtedness of the Borrower and its Subsidiaries
     which have been deducted in calculating EBITDA) of greater than
     $173,000,000 for such Fiscal Year.

          "Eurodollar Rate Margin" means 4.500%, provided that the Eurodollar
           ----------------------                -------------
     Rate Margin shall be decreased to 4.000% as of the first day of the
     calendar month following the date the Borrower delivers its annual
     financial statements for Fiscal Year 2000 to the Lender, if such financial
     statements show EBITDA (plus all fees and expenses incurred in connection
     with the restructuring of any Indebtedness of the Borrower and its
     Subsidiaires which have been deducted in calculating EBITDA) of greater
     than $173,000,000 for such Fiscal Year.

     2.  REPRESENTATIONS AND WARRANTIES TRUE; NO EVENT OF DEFAULT.  By its
         --------------------------------------------------------
execution and delivery hereof, Borrower represents and warrants that, as of the
date hereof and after giving effect to the amendment provided in the foregoing
Section 1 and the Waiver and Seventh Amendment to Amended and Restated Credit
Agreement referred to therein:

     (a) the representations and warranties contained in the Promissory Note are
true and correct on and as of the date hereof as if made on and as of such date;

     (b) no event has occurred and is continuing which constitutes an Event of
Default;

     (c) Borrower has full power and authority to execute and deliver this
Amendment, and this Amendment and the Promissory Note, as amended hereby,
constitute the legal, valid and binding obligations of the Borrower, enforceable
in accordance with their respective terms, except as enforceability may be
limited by applicable debtor relief laws and by general principles of equity
(regardless of whether enforcement is sought in a proceeding in equity or at
law) and except as rights to indemnity may be limited by federal or state
securities laws; and

     (d) no authorization, approval, consent or other action by, notice to, or
filing with, any governmental authority or other Person, is required for the
execution, delivery or performance by Borrower of this Amendment or the
acknowledgment of this Amendment by any Person that executed a Guaranty
Agreement (each such Person being a "Guarantor").
                                     ---------

     3.  CONDITIONS OF EFFECTIVENESS.  This Amendment shall be effective as of
         ---------------------------
March 31, 2000, subject to the following:

                                      -2-
<PAGE>

     (a) Lender shall have received counterparts of this Amendment executed by
Borrower and acknowledged by each Guarantor;

     (b) Lender shall have received an amendment fee from Borrower in
consideration for this Amendment in the amount of $87,500 (which fee shall be
fully-earned when paid, and nonrefundable for any reason); and

     (c) the conditions precedent (other than the effectiveness of this
Amendment) to the effectiveness of that certain Waiver and Seventh Amendment to
Amended and Restated Credit Agreement, dated to be effective as of March 31,
2000, by and among the Borrower, the Guarantors, the lenders party thereto and
Bank of America, N.A., as administrative agent, shall have been satisfied.

     4.  GUARANTOR ACKNOWLEDGMENT. By signing below, each Guarantor (i)
         ------------------------
acknowledges, consents and agrees to the execution, delivery and performance by
Borrower of this Amendment, (ii) acknowledges and agrees that its obligations in
respect of its Guaranty Agreement are not released, diminished, waived,
modified, impaired or affected in any manner by this Amendment or any of the
provisions contemplated herein, (iii) ratifies and confirms its obligations
under its Guaranty Agreement (including, without limitation, with respect to the
Commitment as increased hereby), and (iv) acknowledges and agrees that it has no
claim or offsets against, or defenses or counterclaims to, its obligations under
its Guaranty Agreement.

     5.  REFERENCE TO THE PROMISSORY NOTE.
         --------------------------------

     (a) Upon the effectiveness of this Amendment, each reference in the
Promissory Note to "this Note", "hereunder", or words of like import shall mean
and be a reference to the Promissory Note, as affected and amended by this
Amendment.

     (b) The Promissory Note, as amended by this Amendment, shall remain in full
force and effect and are hereby ratified and confirmed.

     6.  COSTS, EXPENSES AND TAXES.  Borrower agrees to pay on demand all costs
         -------------------------
and expenses of Lender in connection with the preparation, reproduction,
execution and delivery of this Amendment and the other instruments and documents
to be delivered hereunder (including the reasonable fees and out-of-pocket
expenses of counsel and financial consultants for Lender).

     7.  COUNTERPARTS; EXECUTION VIA FACSIMILE.  This Amendment may be executed
         -------------------------------------
in any number of counterparts and by different parties hereto in separate
counterparts, each of which when so executed and delivered shall be deemed to be
an original and all of which when taken together shall constitute but one and
the same instrument.  This Amendment may be validly executed and delivered by
facsimile or other electronic transmission.

                                      -3-
<PAGE>

     8.  GOVERNING LAW; BINDING EFFECT.  This Amendment shall be governed by and
         -----------------------------
construed in accordance with the laws of the State of Texas and shall be binding
upon Borrower and Lender and their respective successors and assigns.

     9.  HEADINGS.  Section headings in this Amendment are included herein for
         --------
convenience of reference only and shall not constitute a part of this Amendment
for any other purpose.

     10.  ENTIRE AGREEMENT. THE PROMISSORY NOTE, AS AMENDED BY THIS AMENDMENT,
          ----------------
REPRESENTS THE FINAL AGREEMENT BETWEEN THE PARTIES AS TO THE SUBJECT MATTER
HEREIN AND MAY NOT BE CONTRADICTED BY EVIDENCE OF PRIOR, CONTEMPORANEOUS, OR
SUBSEQUENT ORAL AGREEMENTS BETWEEN THE PARTIES. THERE ARE NO UNWRITTEN ORAL
AGREEMENTS BETWEEN THE PARTIES.


================================================================================
                  REMAINDER OF PAGE LEFT INTENTIONALLY BLANK
================================================================================

                                      -4-
<PAGE>

     IN WITNESS WHEREOF, the parties hereto have executed this Amendment as of
the date first above written.

                              PILLOWTEX CORPORATION


                              By:  Jaime Vasquez
                                   Vice President and Treasurer



                              BANK OF AMERICA, N.A.



ACKNOWLEDGED AND AGREED:

PILLOWTEX, INC.
PTEX HOLDING COMPANY
PILLOWTEX MANAGEMENT SERVICES COMPANY
BEACON MANUFACTURING COMPANY
MANETTA HOME FASHIONS, INC.
TENNESSEE WOOLEN MILLS
FIELDCREST CANNON, INC.
CRESTFIELD COTTON COMPANY
ENCEE, INC.
FCC CANADA, INC.
FIELDCREST CANNON FINANCING, INC.
FIELDCREST CANNON LICENSING, INC.
FIELDCREST CANNON INTERNATIONAL, INC.
FIELDCREST CANNON SF, INC. (formerly
know as Fieldcrest Cannon Sure Fit, Inc.)
FIELDCREST CANNON TRANSPORTATION, INC,
ST. MARYS, INC.
AMOSKEAG COMPANY
AMOSKEAG MANAGEMENT CORPORATION
DOWNEAST SECURITIES CORPORATION
BANGOR INVESTMENT COMPANY

                      Fifth Amendment to Promissory Note
                                Signature Page
<PAGE>

MOORE'S FALLS CORPORATION
THE LESHNER CORPORATION
LESHNER OF CALIFORNIA, INC.
OPELIKA INDUSTRIES, INC.


By:  Jaime Vasquez
     Vice President and Treasurer


                      Fifth Amendment to Promissory Note
                                Signature Page