<PAGE>   1
                                                                    EXHIBIT 10.3








                                 AMENDMENT NO. 1
                       TO POOLING AND SERVICING AGREEMENT
                                       AND
                         RECEIVABLES PURCHASE AGREEMENT


                          dated as of November 21, 1997


                                      among


                          AVONDALE RECEIVABLES COMPANY,


                              AVONDALE MILLS, INC.,


                                       and


                    MANUFACTURERS AND TRADERS TRUST COMPANY,
                                   as Trustee
<PAGE>   2
         This AMENDMENT No. 1 dated as of November 21, 1997 (this "Amendment")
is made among AVONDALE RECEIVABLES COMPANY, a Delaware corporation, as
transferor ("ARC"), AVONDALE MILLS, INC., a Delaware corporation, as the Seller
and initial Servicer ("Avondale"), and MANUFACTURERS AND TRADERS TRUST COMPANY,
a New York banking corporation, as Trustee (in that capacity, together with any
successor in that capacity, the "Trustee").


                                   BACKGROUND


         1. Avondale, ARC and the Trustee (together, the "Original Parties")
have entered into the Pooling and Servicing Agreement, dated as of April 29,
1996 (the "Pooling Agreement"); and Avondale and ARC have entered into the
Receivables Purchase Agreement, dated as of the same date and amended prior to
the date hereof (as so amended, the "Purchase Agreement"), pursuant to which
Avondale agreed to sell and contribute, and ARC agreed to purchase and receive,
certain Receivables.

         2. Avondale and ARC wish to amend the Pooling Agreement and the
Purchase Agreement as provided herein, and the Trustee is willing to consent (by
its execution hereof) to such amendments subject to the conditions specified
herein.

         NOW, THEREFORE, for good and valuable consideration (the receipt of
which is acknowledged) the parties agree as follows:


                                    ARTICLE I
                                   AMENDMENTS

         SECTION 1.01 Definitions. Capitalized terms used but not otherwise
defined herein have the meanings set forth in Appendix A to the Pooling
Agreement.

         SECTION 1.02 New Definitions. The following new defined terms are added
to Appendix A to the Pooling Agreement and the Purchase Agreement, each in the
proper alphabetical spot:

         "Seller Security Document" means each security document, in form and
substance acceptable to the Buyer, delivered from time to time pursuant to any
provision of the Purchase Agreement, including Section 1.6, as each such
document may be amended, supplemented or otherwise modified from time to time.

         "Transferor Assignment" means an assignment, in form and substance
acceptable to the Trustee, evidencing the transfer of the Transferred Assets by
the Transferor to the Trustee thereunder, as it may be amended, supplemented or
otherwise modified from time to time.

         "Transferor Security Document" means each security document, in form
and substance acceptable to the Trustee, delivered from time to time pursuant to
any provision of the Pooling Agreement, including Section 2.1(e), as each such
document may be amended, supplemented or otherwise modified from time to time.

         SECTION 1.03 Changes to Definitions. (a) The definitions of "Domestic
Person", "Receivable" "Seller Assignment Certificate", "Seller Transaction
Documents" and "Transaction Documents", in Appendix A to the Pooling Agreement
and the Purchase Agreement, are amended and restated to read as follows:

         "Domestic Person" means, with respect to a Receivable, any Person that
has a place of business located in the United States of America, Puerto Rico, or
the Canadian provinces of Alberta, British Columbia, Manitoba, Ontario and
Quebec; provided that such place of business is the billing location for such
Receivables.
<PAGE>   3
         "Receivable" means any right of any Seller to payment, whether
constituting an account, chattel paper, instrument, general intangible or
otherwise, arising from the sale of goods or services (including rights under
Bill and Hold arrangements) by such Seller or Graniteville Company (and
including the right to payment of any interest or finance charges and other
obligations with respect thereto).

         "Seller Assignment Certificates" means assignments by a Seller,
substantially in the form of Exhibit B-1 and B-2 to the Purchase Agreement,
evidencing Transferor's acquisition of the Purchased Assets generated by the
Seller, as they may be amended, supplemented or otherwise modified from time to
time.

         "Seller Transaction Documents" means the Purchase Agreement, the Seller
Assignment Certificates, the Seller Security Documents and the Account
Agreements.

         "Transaction Documents" means each Seller Transaction Document, the
Pooling Agreement, the Seller Guaranty, each Supplement, each PI Agreement, each
agreement to purchase Investor Certificates, each Intercreditor Agreement, the
Transferor Security Documents, the Transferor Assignment and each other
agreement designated as a Transaction Document in any Supplement or PI
Agreement, as any of the same from time to time may be amended, supplemented,
amended and restated or otherwise modified in accordance with the terms thereof.

         (b) The definition of "Excluded Receivable" shall be deleted in its
entirety.

         SECTION 1.04 Changes to Purchase Agreement. (a) The second paragraph of
Section 1.6 of the Purchase Agreement is amended in its entirety to read as
follows:

                  It is, further, not the intention of Buyer or any Seller that
         the conveyance of the Specified Assets by a Seller be deemed a grant of
         a security interest in the Specified Assets by such Seller to Buyer to
         secure a debt or other obligation of such Seller. However, in the event
         that, notwithstanding the intent of the parties, any Specified Assets
         are property of any Seller's estate, then (i) this Agreement also shall
         be deemed to be and hereby is a security agreement under applicable
         law, and (ii) the conveyance by such Seller provided for in this
         Agreement shall be deemed to be a grant by such Seller to Buyer of, and
         such Seller hereby grants to Buyer, a security interest in and to all
         of such Seller's right, title and interest in, to and under the
         Specified Assets to secure (1) the rights of Buyer hereunder and (2) a
         loan by Buyer to such Seller in the amount of the related Purchase
         Price of the Purchased Assets sold by it or the Unpaid Balance of any
         Contributed Receivables and the Related Contributed Assets, as the case
         may be. Each Seller and Buyer shall, to the extent consistent with this
         Agreement, take such actions as may be necessary to ensure that, if
         this Agreement were found to create a security interest in the
         Specified Assets, such security interest would be a perfected security
         interest of first priority (subject to Permitted Adverse Claims) in
         favor of Buyer under applicable law and will be maintained as such
         throughout the term of this Agreement.

         (b)      Clause (e) of Section 6.3 of the Purchase Agreement is amended
in its entirety to read as follows:

                  (e) Change in Name. Such Seller will not (i) change its
         corporate name or (ii) change the name under or by which it does
         business in any manner that would or may make any financing statement
         or any instrument similar in effect filed by such Seller in accordance
         herewith seriously misleading within the meaning of Section 9-402(7) of
         an applicable enactment of the UCC or otherwise ineffective under the
         laws of the applicable jurisdiction, in each case unless such Seller
         shall have given Buyer, the Servicer, the Trustee and the Rating
         Agencies 30 days' prior written notice thereof and unless, prior to any
         change in name, such Seller shall have taken and completed all action
         required by Section 7.3.

         (c)      The final paragraph of Section 7.3 of the Purchase Agreement
is amended in its entirety to read as follows:

                  Each Seller hereby authorizes Buyer or its designee to file
         one or more financing or continuation statements, and amendments
         thereto and assignments thereof and such other instruments or notices,
<PAGE>   4
         relative to all or any of the Receivables and Related Assets of such
         Seller, in each case whether now existing or hereafter generated by
         such Seller. Except for material performance obligations of such Seller
         to any Obligor hereunder or under any of the Contracts, if (i) such
         Seller fails to perform any of its agreements or obligations under this
         Agreement and does not remedy the failure within the applicable cure
         period, if any, and (ii) Buyer in good faith reasonably believes that
         the performance of such agreements and obligations is necessary or
         appropriate to protect its interests under this Agreement, then Buyer
         or its designee may (but shall not be required to) perform, or cause
         performance of, such agreement or obligation and the reasonable
         expenses of Buyer or its designee or assignee incurred in connection
         with such performance shall be payable by such Seller as provided in
         Section 9.1.

         (d)      Each reference in the Purchase Agreement to "Exhibit B" shall
be amended to refer to "Exhibit B-1 and Exhibit B-2".

         (e)      The Purchase Agreement shall be amended by adding Exhibit B-2
attached hereto as Exhibit B-2 thereto.

         SECTION 1.05 Changes to Pooling Agreement.

         (a)      Clause (b) of Section 2.1 of the Pooling Agreement is amended
in its entirety to read as follows:

                  (b) In connection with the transfer described in subsection
         (a), (i) Transferor will from time to time at its own expense, promptly
         execute and deliver such assignments, instruments and documents
         (including the Transferor Assignment) that Trustee may reasonably
         request in order to perfect, protect and more fully evidence such
         transfer and (ii) Transferor and Servicer shall record and file or
         cause to be recorded and filed, as an expense of Servicer paid out of
         the Servicing Fee, financing statements, assignments and such other
         instruments, notices and documents with respect to the Transferred
         Assets meeting the requirements of applicable law in such manner and in
         such jurisdictions as are necessary to perfect the transfer and
         assignment of the Transferred Assets to the Trust. In connection with
         the transfer described in subsection (a), Transferor and Servicer
         further agree to deliver to Trustee each Transferred Asset (including
         any original documents or instruments included in the Transferred
         Assets as are necessary to effect such transfer) in which the transfer
         of an interest is perfected under the UCC or otherwise by possession.
         Transferor or Servicer shall deliver each such Transferred Asset to
         Trustee, as an expense of Servicer paid out of the Servicing Fee,
         immediately upon the transfer of any such Transferred Asset to Trustee
         pursuant to subsection (a).

         (b)      Clause (g) of Section 7.1 of the Pooling Agreement is amended
in its entirety to read as follows:

                  (g) Approvals. All authorizations, consents, orders and
         approvals of, or other action by, any Governmental Authority or other
         Person that are required to be obtained by Transferor, and all notices
         to and filings with any Governmental Authority or other Person, that
         are required to be made by it, in the case of each of the foregoing in
         connection with the transfer of Receivables and Related Transferred
         Assets to the Trust or the execution, delivery and performance by it of
         this Agreement and any other Transaction Documents to which it is a
         party and the consummation of the transactions contemplated by this
         Agreement, have been obtained or made and are in full force and effect
         (other than the filings referred to in Section 2.3(a)(i)(A), all of
         which, at the time required in Section 2.3(a)(i)(A), will be duly
         made), except where the failure to obtain or make any such
         authorization, consent, order, approval, notice or filing, individually
         or in the aggregate for all such failures, would not have a substantial
         likelihood of having a Material Adverse Effect.

         (c)      Clause (f) of Section 8.1 of the Pooling Agreement is amended
in its entirety to read as follows:

                  (f) Approvals. All authorizations, consents, orders and
         approvals of, or other action by, any Governmental Authority or other
         Person that are required to be obtained by Servicer, and all notices to
         and filings with any Governmental Authority or other Person that are
         required to be made by it, in the 
<PAGE>   5
         case of each of the foregoing in connection with the execution,
         delivery and performance by it of this Agreement and any other
         Transaction Documents to which it is a party and the consummation of
         the transactions contemplated by this Agreement, have been obtained or
         made and are in full force and effect, except where the failure to
         obtain or make such authorization, consent, order, approval, notice or
         filing, individually or in the aggregate for all such failures, is not
         likely to have a Material Adverse Effect.

         SECTION 1.06 Changes to Both Purchase Agreement and Pooling Agreement.
The Purchase Agreement and Pooling Agreement shall each be amended as follows:

         (a)      Each reference to "perfected" shall be amended to refer to
"perfected and protected".

         (b)      Each reference to "financing statement" shall be amended to
refer to "financing statement or any instrument similar in effect".

         (c)      Each occurrence of the phrase "within the meaning of the UCC"
shall be amended to read "under applicable law".

         (d)      Each reference to "continuation statements" shall be amended
to refer to "continuation statements or any instrument similar in effect".

         (e)      Each reference to "state law" shall be amended to refer to
"state or foreign law"; provided that such amendment shall not be made to
Exhibit A of the Purchase Agreement.

         (f)      Each reference to "perfection" with respect to the perfection
or protection of a security interest shall be amended to refer to "perfection or
protection".
<PAGE>   6
                                   ARTICLE II
                   CONDITIONS, REPRESENTATIONS AND WARRANTIES

         SECTION 2.01 Conditions Precedent. This Amendment shall be effective
from and after the later of (a) November 21, 1997 or (b) the date upon which all
of the conditions precedent specified below have been satisfied (the "Effective
Date"). The conditions precedent are:

                  (i) The Trustee shall have received from each of Avondale and
         ARC a certificate, dated as of the date hereof, of an Authorized
         Officer as to:

                           (A) resolutions of its board of directors then in
                  full force and effect authorizing the execution, delivery and
                  performance of this Amendment,

                           (B) the incumbency and signature of those of its
                  officers authorized to act with respect to this Amendment,

         upon which certificate the Trustee may conclusively rely.

                  (ii) The Trustee shall have received an opinion of counsel to
         Avondale and ARC that the modifications to the Pooling Agreement and
         the Purchase Agreement made pursuant to this Amendment are legal, valid
         and binding upon each of Avondale and ARC and that such amendments are
         permitted under the terms of the Pooling Agreement and the Purchase
         Agreement.

                  (iii) The representations and warranties of Avondale and ARC
         as set forth in the Transaction Documents shall continue to be true and
         correct, and the Trustee shall have received the certificate of an
         Authorized Officer of each of Avondale and ARC to the effect that the
         representations and warranties continue to be true and correct.

                  (iv) The Trustee shall have received an original counterpart
         of the Seller Assignment Certificate attached hereto as Exhibit B-2.

         SECTION 2.02 Representations and Warranties. Each of Avondale and ARC
represents and warrants to the Trustee that:

         (a) The execution and delivery by it of this Amendment, and the
performance of its obligations under the Pooling Agreement and the Purchase
Agreement as modified by this Amendment, are within its corporate powers, have
been duly authorized by all necessary corporate action, have received all
necessary governmental approvals other than Assignment of Claims Act filings (if
any shall be required), and other consents or approvals and do not and will not
contravene or conflict with, or create any Adverse Claim under, (i) any
provision of law, (ii) its constituent documents, (iii) any court or
administrative decree applicable to it or (iv) any contractual restriction
binding upon it or its property which conflict or adverse claim would have a
substantial likelihood of having Material Adverse Affect.

                  (b) This Amendment has been duly executed and delivered by it,
         and the Pooling Agreement and the Purchase Agreement, as amended, are
         its legal, valid and binding obligations, enforceable against it in
         accordance with its terms except as enforceability may be limited by
         bankruptcy, insolvency, reorganization or other laws affecting the
         enforcement of creditors' rights generally and by general principles of
         equity.

                  (c) The warranties made by it in the Pooling Agreement and the
         Purchase Agreement are true and correct as of the date hereof as though
         made on that date, except to the extent that the warranties
         specifically relate to an earlier date.
<PAGE>   7
                  (d) After giving effect to this Amendment, no Early Event or
         Unmatured Early Amortization Event shall have occurred and be
         continuing.

         SECTION 2.03 Representations and Warranties of Trustee2.03
Representations and Warranties of Trustee. The Trustee represents and warrants
that:

                  (a) it is a banking corporation organized, existing and in
         good standing under the laws of the State of New York,

                  (b) it has full power, authority and right to execute, deliver
         and perform this Amendment, and has taken all necessary action to
         authorize the execution, delivery and performance by it of this
         Amendment, and

                  (c) this Amendment has been duly executed and delivered by the
         Trustee, and is a legal, valid and binding obligation of the Trustee,
         enforceable in accordance with its terms, except as such enforceability
         may be limited by bankruptcy, insolvency, reorganization or other
         similar laws affecting the enforcement of creditors' rights generally
         and by general principles of equity, regardless of whether such
         enforceability is considered in a proceeding in equity or at law.


                                   ARTICLE III
                                  MISCELLANEOUS

         SECTION 3.01 Miscellaneous. (a) THIS AMENDMENT SHALL BE GOVERNED BY,
AND CONSTRUED IN ACCORDANCE WITH, THE LAWS OF THE STATE OF NEW YORK, WITHOUT
REGARD TO CONFLICT OF LAWS PRINCIPLES.

         (b) This Amendment may be executed in any number of counterparts and by
the different parties hereto in separate counterparts, each of which when so
executed shall be deemed to be an original, and all of which together shall
constitute one and the same agreement.

         (c) Any reference to the Pooling Agreement or the Purchase Agreement
contained in any notice, request, certificate or other document executed
concurrently with or after the Effective Date shall be deemed to be a reference
to the Pooling Agreement or the Purchase Agreement as amended hereby. Except as
expressly modified hereby, the Transaction Documents hereby are ratified and
confirmed by the parties hereto. The amended Pooling Agreement, the amended
Purchase Agreement and the other Transaction Documents remain in full force and
effect.
<PAGE>   8
         IN WITNESS WHEREOF, the parties hereto have caused their duly
authorized officers to execute this Amendment as of the day and year first above
written.

                           AVONDALE RECEIVABLES COMPANY,
                           As Transferor



                           By:      /s/ J. Elliott Woodward
                                    --------------------------------------------
                              Title: Assistant Secretary
                                     -------------------------------------------


                           Address:          506 South Broad Street
                                             Monroe, Georgia 30655
                           Attention:        Treasurer
                           Telephone:        707/267-2226
                           Facsimile:        707/267-2543


                           AVONDALE MILLS, INC., as initial Servicer

                           By:      /s/ J. Elliott Woodward
                                    --------------------------------------------
                              Title:  Vice President, Treasurer and Assistant
                                     -------------------------------------------
                                     Secretary
                                     -------------------------------------------


                           Address:          506 South Broad Street
                                             Monroe, Georgia 30655
                           Attention:        Chief Financial Officer
                           Telephone:        707/267-2226
                           Facsimile:        707/267-2543

                           The Trustee joins in this Amendment for purposes of
                           Evidencing its consent thereto.

                           MANUFACTURERS AND TRADERS TRUST 
                           COMPANY, as Trustee

                           By:      /s/ Russell T. Whitley
                                    --------------------------------------------
                              Title  Assistant Vice President
                                     -------------------------------------------

                           Address:          1 M&T Plaza, 7th Floor
                                             Buffalo, New York 14203-2399
                           Attention:        Steve Wattie
                                             Corporate Trust Department
                           Telephone:        716/842-5849
                           Facsimile:        716/842-4474