<PAGE>

                                                                   EXHIBIT 10.20

                      FOURTH AMENDMENT TO PROMISSORY NOTE


     THIS FOURTH AMENDMENT TO PROMISSORY NOTE (this "Amendment"), dated as of
                                                     ---------
February 15, 2000, is entered into between PILLOWTEX CORPORATION, a Texas
corporation ("Borrower"), and BANK OF AMERICA, N.A. (formerly known as
              --------
NationsBank, N.A.) ("Lender").
                     ------

     A.  Borrower executed that certain Promissory Note, dated May 4,1999, in
the maximum principal amount of $20,000,000, payable to the order of Lender, as
amended and increased to $35,000,000 by that certain First Amendment to
Promissory Note, dated as of July 27, 1999 (said Promissory Note, as amended,
the "Promissory Note"; the terms defined in the Promissory Note and not
     ---------------
otherwise defined herein shall be used herein as defined in the Promissory
Note).

     B.  Borrower and Lender desire to amend the Promissory Note.

     NOW, THEREFORE, in consideration of the covenants, conditions and
agreements hereafter set forth, and for other good and valuable consideration,
the receipt and adequacy of which are all hereby acknowledged, Borrower and
Lender covenant and agree as follows:

     1.  AMENDMENT TO PROMISSORY NOTE.  The Promissory Note is hereby amended by
         ----------------------------
amending the definition of "Maturity Date" set forth in Article III thereof to
read as follows:

          "Maturity Date" means the earliest of (a) the occurrence of a
           -------------
     Prepayment Event, (b) March 31, 2000, (c) the date of termination in whole
     of the Commitment hereunder or (d) the date of termination of the Waiver
     Period, as defined in that certain Sixth Amendment and Waiver to Amended
     and Restated Credit Agreement, dated to be effective as of December 7,
     1999, executed and delivered in connection with the Credit Agreement.

     2.  REPRESENTATIONS AND WARRANTIES TRUE; NO EVENT OF DEFAULT.  By its
         --------------------------------------------------------
execution and delivery hereof, Borrower represents and warrants that, as of the
date hereof and after giving effect to the amendment provided in the foregoing
Section 1 and the Sixth Amendment and Waiver to Amended and Restated Credit
Agreement referred to therein:

     (a) the representations and warranties contained in the Promissory Note are
true and correct on and as of the date hereof as if made on and as of such date;

     (b) no event has occurred and is continuing which constitutes an Event of
Default;

     (c) Borrower has full power and authority to execute and deliver this
Amendment, and this Amendment and the Promissory Note, as amended hereby,
constitute the legal, valid and binding obligations of the Borrower, enforceable
in accordance with their respective terms, except as enforceability may be
limited by applicable debtor relief laws and by general principles of equity
(regardless of whether enforcement is sought in a proceeding in equity or at
law) and except as rights to indemnity may be limited by federal or state
securities laws; and
<PAGE>

     (d) no authorization, approval, consent or other action by, notice to, or
filing with, any governmental authority or other Person, is required for the
execution, delivery or performance by Borrower of this Amendment or the
acknowledgment of this Amendment by any Person that executed a Guaranty
Agreement (each such Person being a "Guarantor").
                                     ---------

     3.  CONDITIONS OF EFFECTIVENESS.  This Amendment shall be effective as of
         ---------------------------
February 15, 2000, subject to the following:

     (a) Lender shall have received counterparts of this Amendment executed by
Borrower and acknowledged by each Guarantor; and

     (b) the conditions precedent (other than the effectiveness of this
Amendment) to the effectiveness of that certain Sixth Amendment and Waiver to
Amended and Restated Credit Agreement, dated to be effective as of February 15,
2000, by and among the Borrower, the Guarantors, the lenders party thereto and
Bank of America, N.A., as administrative agent, shall have been satisfied.

     4.  GUARANTOR ACKNOWLEDGMENT. By signing below, each Guarantor (i)
         ------------------------
acknowledges, consents and agrees to the execution, delivery and performance by
Borrower of this Amendment, (ii) acknowledges and agrees that its obligations in
respect of its Guaranty Agreement are not released, diminished, waived,
modified, impaired or affected in any manner by this Amendment or any of the
provisions contemplated herein, (iii) ratifies and confirms its obligations
under its Guaranty Agreement (including, without limitation, with respect to the
Commitment as increased hereby), and (iv) acknowledges and agrees that it has no
claim or offsets against, or defenses or counterclaims to, its obligations under
its Guaranty Agreement.

     5.  REFERENCE TO THE PROMISSORY NOTE.
         --------------------------------

     (a) Upon the effectiveness of this Amendment, each reference in the
Promissory Note to "this Note", "hereunder", or words of like import shall mean
and be a reference to the Promissory Note, as affected and amended by this
Amendment.

     (b) The Promissory Note, as amended by this Amendment, shall remain in full
force and effect and are hereby ratified and confirmed.

     6.  COSTS, EXPENSES AND TAXES.  Borrower agrees to pay on demand all costs
         -------------------------
and expenses of Lender in connection with the preparation, reproduction,
execution and delivery of this Amendment and the other instruments and documents
to be delivered hereunder (including the reasonable fees and out-of-pocket
expenses of counsel and financial consultants for Lender).

     7.  COUNTERPARTS; EXECUTION VIA FACSIMILE.  This Amendment may be executed
         -------------------------------------
in any number of counterparts and by different parties hereto in separate
counterparts, each of which when so executed and delivered shall be deemed to be
an original and all of which when

                                      -2-
<PAGE>

taken together shall constitute but one and the same instrument. This Amendment
may be validly executed and delivered by facsimile or other electronic
transmission.

     8.  GOVERNING LAW; BINDING EFFECT.  This Amendment shall be governed by and
         -----------------------------
construed in accordance with the laws of the State of Texas and shall be binding
upon Borrower and Lender and their respective successors and assigns.

     9.  HEADINGS.  Section headings in this Amendment are included herein for
         --------
convenience of reference only and shall not constitute a part of this Amendment
for any other purpose.

     10.  ENTIRE AGREEMENT. THE PROMISSORY NOTE, AS AMENDED BY THIS AMENDMENT,
          ----------------
REPRESENTS THE FINAL AGREEMENT BETWEEN THE PARTIES AS TO THE SUBJECT MATTER
HEREIN AND MAY NOT BE CONTRADICTED BY EVIDENCE OF PRIOR, CONTEMPORANEOUS, OR
SUBSEQUENT ORAL AGREEMENTS BETWEEN THE PARTIES. THERE ARE NO UNWRITTEN ORAL
AGREEMENTS BETWEEN THE PARTIES.


================================================================================
                  REMAINDER OF PAGE LEFT INTENTIONALLY BLANK
================================================================================

                                      -3-
<PAGE>

     IN WITNESS WHEREOF, the parties hereto have executed this Amendment as of
the date first above written.

                              PILLOWTEX CORPORATION


                              By:  Jaime Vasquez
                                  Title:  Vice President/Treasurer


                              BANK OF AMERICA, N.A.


                      Fourth Amendment to Promissory Note
                                Signature Page
<PAGE>

ACKNOWLEDGED AND AGREED:

PILLOWTEX, INC.
PTEX HOLDING COMPANY
PILLOWTEX MANAGEMENT SERVICES COMPANY
BEACON MANUFACTURING COMPANY
MANETTA HOME FASHIONS, INC.
TENNESSEE WOOLEN MILLS
FIELDCREST CANNON, INC.
CRESTFIELD COTTON COMPANY
ENCEE, INC.
FCC CANADA, INC.
FIELDCREST CANNON FINANCING, INC.
FIELDCREST CANNON LICENSING, INC.
FIELDCREST CANNON INTERNATIONAL, INC.
FIELDCREST CANNON SF, INC. (formerly
know as Fieldcrest Cannon Sure Fit, Inc.)
FIELDCREST CANNON TRANSPORTATION, INC,
ST. MARYS, INC.
AMOSKEAG COMPANY
AMOSKEAG MANAGEMENT CORPORATION
DOWNEAST SECURITIES CORPORATION
BANGOR INVESTMENT COMPANY
MOORE'S FALLS CORPORATION
THE LESHNER CORPORATION
LESHNER OF CALIFORNIA, INC.
OPELIKA INDUSTRIES, INC.


By:  Jaime Vasquez
     Title:  Vice President/Treasurer


                      Fourth Amendment to Promissory Note
                                Signature Page