<PAGE>
 
                                                                       EXHIBIT 1

                 AMENDMENT NO. 3 TO LOAN AND SECURITY AGREEMENT
                 ----------------------------------------------


                                                               November 11, 1997


The Bibb Company
237 Coliseum Avenue
Macon, Georgia  31201

Gentlemen:

     Reference is made to the Loan and Security Agreement (as heretofore
amended, the "Loan Agreement"), dated as of September 12, 1996, by and among
Congress Financial Corporation (Southern), as agent (the "Agent"), the Lenders
parties thereto and The Bibb Company ("Borrower"), together with all other
agreements, documents and instruments at any time executed and/or delivered in
connection therewith or related thereto (as the same now exist, are being
amended hereby and may hereafter be amended, modified, supplemented, extended,
renewed, restated or replaced, collectively, the "Financing Agreements").  All
capitalized terms used herein and not herein defined shall have the meanings
given to them in the Loan Agreement.

     Borrower has requested that Agent and Lenders agree to (a) amend certain
provisions of the Loan Agreement, the Revolving Notes and the Term Notes,
including, among other things, provisions with respect to the Revolving Loan
Limit, the Maximum Credit, the Interest Rate, certain Availability Reserves and
financial covenants, and (b) to waive Borrower's non-compliance with certain of
the financial covenants contained in the Loan Agreement for the Measurement
Periods ended August 31, 1997, September 30, 1997, and October 31, 1997. Agent
and Lenders or Required Lenders, as applicable, are willing to agree to such
amendments and such waivers, subject to the terms and conditions set forth
herein.

     In consideration of the foregoing, the mutual agreements and covenants
contained herein and other good and valuable consideration, the parties hereto
agree as follows:

     1.   Amendments to Definitions.
          -------------------------- 

          (a) After-Tax Fixed Charge Coverage Ratio.  Effective as of November
              -------------------------------------                           
1, 1997, Section 1.3 of the Loan Agreement is hereby deleted in its entirety and
replaced with the following:   "1.3 [Intentionally Omitted.]"

          (b)  Commitments.
               ----------- 

               (i) Effective as of November 1, 1997, the Commitment of Congress,
     as set forth on the signature page of the Loan Agreement, is hereby amended
     by replacing the figure "$78,000,000" and replacing it with "$67,826,090".
<PAGE>
 
               (ii) Effective as of November 1, 1997, the Commitment of
     Transamerica, as set forth on the signature page of the Loan Agreement, is
     hereby amended by replacing the figure "$37,000,000" with "$32,173,910".

          (c) Interest Rate.  Effective as of November 1, 1997, with respect to
              -------------                                                    
interest accruing on and after such date, the definition of "Interest Rate"
contained in Section 1.54 of the Loan Agreement with respect to Eurodollar Rate
Loans is hereby amended by deleting the reference to "three and one-quarter 
(3 1/4%) percent" and replacing that reference with the following:  "three (3%)
percent".

          (d) Maximum Credit.  Effective as of November 1, 1997, Section 1.61 of
              --------------                                                    
the Loan Agreement is hereby deleted in its entirety and replaced with the
following:

          "1.61  "Maximum Credit" shall mean the amount equal to (a)
     $100,000,000 less (b) the aggregate amount of payments and prepayments
                  ----                                                     
     applied to the Term Loan after the Closing Date."

          (e) Pre-Tax Profit.  Effective as of November 1, 1997, Section 1.77 of
              --------------                                                    
the Loan Agreement is hereby deleted in its entirety and replaced with the
following:  "1.77 [Intentionally Omitted.]"

          (f) Revolving Loan Limit.  Effective as of November 1, 1997, Section
              --------------------                                            
1.87 of the Loan Agreement is hereby deleted in its entirety and replaced with
the following:

               "1.87  "Revolving Loan Limit" shall mean the amount of
          $75,000,000."

          (g) Supplemental Capital Expenditure  Reserve.  Effective as of
              -----------------------------------------                  
November 1, 1997, Section 1.94 of the Loan Agreement is hereby deleted in its
entirety and replaced with the following:  "1.94 [Intentionally Omitted.]"

     2.   Amendments to Revolving Notes.
          ----------------------------- 

          (a) Congress Revolving Note.  Effective as of November 1, 1997, the
              -----------------------                                        
Revolving Loan Note, dated September 27, 1996, by Borrower payable to the order
of Congress, in the original principal amount of $61,043,481, is hereby amended
as follows:

               (i) The reference to "$61,043,481" appearing at the top of page 1
     of such note is hereby deleted and replaced with the following:
     "$50,869,568".

               (ii)  Clause (a) contained in the first paragraph appearing on
     page 1 of such note is hereby deleted in its entirety and replaced with the
     following:  "(a) FIFTY MILLION EIGHT HUNDRED SIXTY-NINE THOUSAND FIVE
     HUNDRED SIXTY-EIGHT AND 00/100 DOLLARS ($50,869,568)".

                                       2
<PAGE>
 
               (iii)  The term "Interest Rate" applicable to Eurodollar Rate
     Loans, as defined in the third paragraph appearing on page 1 of such note,
     is hereby amended, with respect to interest accruing on and after November
     1, 1997, by deleting the reference to "three and one-quarter (3 1/4%)
     percent" and replacing that reference with the following: "three (3%)
     percent".

          (b) Transamerica Revolving Note.  Effective as of November 1, 1997,
              ---------------------------                                    
the Revolving Loan Note, dated September 27, 1996, by Borrower payable to the
order of Transamerica in the original principal amount of $28,956,519, is hereby
amended as follows:

               (i) The reference to "$28,956,519" appearing at the top of page 1
     of such note is hereby deleted and replaced with the following:
     "$24,130,432".

               (ii) Clause (a) contained in the first paragraph appearing on
     page 1 of such note is hereby deleted in its entirety and replaced with the
     following:  "(a) TWENTY-FOUR MILLION ONE HUNDRED THIRTY THOUSAND FOUR
     HUNDRED THIRTY-TWO AND 00/100 DOLLARS ($24,130,432)".

               (iii)  The term "Interest Rate" applicable to Eurodollar Rate
     Loans, as defined in the third paragraph appearing on page 1 of such note,
     is hereby amended, with respect to interest accruing on and after November
     1, 1997, by deleting the reference to "three and one-quarter (3 1/4%)
     percent" and replacing that reference with the following: "three (3%)
     percent".

          (c) Revolving Note References.  Effective as of November 1, 1997, all
              -------------------------                                        
references to the term "Revolving Notes" contained in the Loan Agreement, the
Mortgages and the other Financing Agreements are hereby amended to mean the
Revolving Notes as amended hereby.

     3.   Amendments to Term Notes.
          ------------------------ 

          (a) Interest Rate.  Effective as of November 1, 1997, with respect to
              -------------                                                    
interest accruing on or after such date, the term "Interest Rate" applicable to
Eurodollar Rate Loans, as defined in each third paragraph appearing on each page
one of each Term Note is hereby amended by deleting the reference to "three and
one-quarter (3 1/4%) percent" and replacing each such reference in each Term
Note with the following:  "three (3%) percent".

          (b) Term Note References.  Effective as of November 1, 1997, all
              --------------------                                        
references to the term "Term Notes" contained in the Loan Agreement, the
Mortgages and the other Financing Agreements are hereby amended to mean the Term
Notes as amended hereby.

     4.   Supplemental Capital Expenditures Reserve.  Upon the effectiveness of
          -----------------------------------------                            
this Amendment, the Supplemental Capital Expenditures Reserve shall be released
by Agent and Section 2.4(c) of the Loan Agreement is hereby deleted in its
entirety and replaced with the following:  "(c) Intentionally Omitted.]"

                                       3
<PAGE>
 
     5.   Interest Rate.  Effective as of November 1, 1997, Section 3.1(f) of
          -------------                                                      
the Loan Agreement is hereby deleted in its entirety and replaced with the
following:

          "(f)  The Interest Rate with respect to Eurodollar Rate Loans is
     subject to adjustment as follows:

               (i) if Borrower shall have achieved EBITDA of not less than
          $25,000,000 for its fiscal year ending December 31, 1998, then the
          pre-default Interest Rate for Eurodollar Rate Loans will be reduced to
          two and one-half (2 1/2%) percent per annum above the Adjusted
          Eurodollar Rate.

               (ii) If the term of this Agreement is extended beyond December
          31, 1999, and if Borrower shall have achieved EBITDA of not less than
          $30,000,000 for its fiscal year ending December 31, 1999, and Borrower
          previously became entitled to the reduction in the pre-default
          Interest Rate for Eurodollar Rate Loans as described in Section
          3.1(f)(i) hereof, then the pre-default Interest Rate for Eurodollar
          Rate Loans will be further reduced to two and one-quarter (2 1/4%)
          percent per annum above the Adjusted Eurodollar Rate.

               (iii)  If the term of this Agreement extended beyond December 31,
          1999, and if Borrower shall not have achieved EBITDA of at least
          $25,000,000 for its fiscal year ending December 31, 1998, but Borrower
          shall have achieved EBITDA of $30,000,000 or more for its fiscal year
          ending December 31, 1999, then the pre-default Interest Rate for
          Eurodollar Rate Loans will be reduced to two and one-half (2 1/2%)
          percent per annum above the Adjusted Eurodollar Rate.

               (iv) Each adjustment in the Interest Rate for Eurodollar Rate
          Loans shall be effective as to each Interest Period that commences on
          or after the tenth (10th) day after the delivery to Agent of audited
          financial statements of the Borrower for the applicable fiscal year
          showing that the required financial results were achieved and
          accompanied by the unqualified audit report and opinion thereon of
          independent certified public accountants acceptable to Agent.

               (v) No adjustments in the pre-default Interest Rate for
          Eurodollar Rate Loans as described in this Section 3.1(f) shall become
          effective if, at the time an adjustment would otherwise be made under
          this Section 3.1(f), a Default or Event of Default exists or has
          occurred and is continuing."

     6.   Waivers.  Agent and Lenders hereby waive any failure of Borrower to
          -------                                                            
comply with the financial covenants set forth in the Loan Agreement described in
Section 9.14 regarding Changes in Tangible Net Worth, Section 9.16 regarding
EBITDA and Section 9.17 regarding After-Tax Fixed Charge Coverage Ratio with
respect to the Measurement Periods consisting of the eight (8) month period
through August 31, 1997, the nine (9) month period through September 30, 1997,
and the ten (10) month period through October 31, 1997.

                                       4
<PAGE>
 
     7.   Amendments to Certain Financial Covenants.
          ----------------------------------------- 

          (a) Effective as of November 1, 1997, Section 9.14 of the Loan
Agreement is hereby deleted in its entirety and replaced with the following:

               "9.14  Tangible Net Worth.  Borrower shall, at all times,
                      ------------------                                
          maintain Tangible Net Worth of not less than $70,000,000."

          (b)  Effective as of November 1, 1997, Section 9.16 of the Loan
Agreement is hereby deleted in its entirety and replaced with the following:
"9.16  EBITDA. [Intentionally Omitted.]"
       ------                           

          (c)  Effective as of November 1, 1997, Section 9.17 of the Loan
Agreement is hereby deleted in its entirety and replaced with the following:
"9.17  After-Tax Fixed Charge Coverage Ratio.  [Intentionally Omitted.]"
       -------------------------------------                            

     8.   Financial Reporting.
          ------------------- 

          (a) Financial Covenants Schedule.  Effective as of November 1, 1997,
              ----------------------------                                    
the Financial Covenants Schedule to the Loan Agreement is hereby amended by
deleting the first three columns set forth thereon entitled:  "Minimum FIFO
Basis EBITDA", "Minimum After-Tax Fixed Charge Coverage to 1.00" and "Minimum
Increase (Maximum Decrease) in Tangible Net Worth."

          (b)  Covenants Compliance Certificate.  Effective as of November 1,
               --------------------------------                              
1997, the reference to "Sections 9.13 through 9.17" contained in Section 9.6(e)
of the Loan Agreement is hereby deleted in its entirety and replaced with the
following:  "Sections 9.13 through 9.15".

     9.   Amendment Fee.  In addition to all other fees, charges, interest and
          -------------                                                       
expenses payable by Borrower to Agent under the Financing Agreements, Borrower
shall pay to Agent, for the ratable benefit of Lenders, based on their Pro Rata
                                                                       --- ----
Shares, a fee for entering into this Amendment in the amount of $50,000, which
amount is fully earned and payable as of the date hereof and may, at Agent's
option, be charged directly to Borrower's Revolving Loan account maintained by
Agent.

     10.  Condition Precedent.  The effectiveness of the amendments contained
          -------------------                                                
herein shall be subject to the satisfaction of the following condition:  the
receipt by Agent of an original of this Amendment, duly authorized, executed and
delivered by Borrower and each of the Lenders.

     11.  Effect of this Amendment.
          ------------------------ 

          (a) Entire Agreement; Ratification and Confirmation of the Financing
              ----------------------------------------------------------------
Agreements.  This Amendment contains the entire agreement of the parties with
----------                                                                   
respect to the subject matter hereof and supersedes all prior or contemporaneous
term sheets, proposals,

                                       5
<PAGE>
 
discussions, negotiations, correspondence, commitments and communications
between or among the parties concerning the subject matter hereof.  This
Amendment may not be modified or any provision waived, except in writing signed
by the party against whom such modification or waiver is sought to be enforced.
No Events of Default have been or are being waived hereby and, except as
specifically amended or waived pursuant hereto, the Financing Agreements are
hereby ratified, restated and confirmed by the parties hereto as of the
effective date hereof.  To the extent of conflict between the terms of this
Amendment and the other Financing Agreements, the terms of this Amendment shall
control.

          (b) Governing Law.  This Amendment and the rights and obligations
              -------------                                                
hereunder of each of the parties hereto shall be governed by and interpreted and
determined in accordance with the laws of the State of Georgia.

          (c) Binding Effect.  This Amendment shall be binding upon and inure to
              --------------                                                    
the benefit of each of the parties hereto and their respective successors and
assigns.

          (d) Counterparts.  This Amendment may be executed in any number of
              ------------                                                  
counterparts, but all of such counterparts shall together constitute but one and
the same agreement.  In making proof of this Amendment it shall not be necessary
to produce or account for more than one counterpart thereof signed by each of
the parties hereto.

     By the signatures hereto of their duly authorized officers, each of the
parties hereto covenants and agrees as set forth herein.

                                       Very truly yours,                  
                                                                          
                                       CONGRESS FINANCIAL CORPORATION     
                                       (SOUTHERN), as Agent and Lender    
                                                                          
                                       By: /s/ David Stair             
                                          -------------------             
                                                                          
                                       Title: Vice President             
                                             ----------------             
                                                                          
                                                                          
                                       TRANSAMERICA BUSINESS CREDIT       
                                       CORPORATION, as Lender             
                                                                          
                                       By: /s/ Robert L. Heinz           
                                          ---------------------           
                                                                          
                                       Title: Senior Vice President      
                                             -----------------------       

                                       6
<PAGE>
 
AGREED AND ACCEPTED:

THE BIBB COMPANY

By: /s/ Charles R. Tutterow
   -------------------------

Title: Vice President - Finance
      --------------------------

                                       7