<PAGE>

                                                                    EXHIBIT 10.3

                      SECOND AMENDMENT TO CREDIT AGREEMENT


          THIS SECOND AMENDMENT TO CREDIT AGREEMENT (the "Second Amendment")
                                                          ----------------
dated as of December 29, 1999, is to that Credit Agreement dated as of  October
14, 1998, as amended by a First Amendment to Credit Agreement dated May 21, 1999
(as may be subsequently amended and modified from time to time, the "Credit
                                                                     ------
Agreement"; terms used but not otherwise defined herein shall have the meanings
---------
provided in the Credit Agreement), by and among DAN RIVER INC., a Georgia
corporation (the "Borrower"), the Guarantors identified therein, the several
                  --------
banks and other financial institutions identified therein (the "Lenders") and
                                                                -------
FIRST UNION NATIONAL BANK, as administrative agent for the Lenders thereunder
(in such capacity, the "Agent"), BANK ONE, formerly The First National Bank of
                        -----
Chicago, as syndication agent, and WACHOVIA BANK, N.A., as documentation agent.


                              W I T N E S S E T H:

          WHEREAS, the Lenders have established a credit facility for the
benefit of the Borrower pursuant to the terms of the Credit Agreement;

          WHEREAS, the Borrower wishes to amend the Credit Agreement to modify
certain provisions contained therein; and

          WHEREAS, the Required Lenders have agreed to the requested amendment
on the terms and conditions hereinafter set forth;

          NOW, THEREFORE, IN CONSIDERATION of the premises and other good and
valuable consideration, the receipt and sufficiency of which are hereby
acknowledged, the parties hereto agree as follows:

          A.  The Credit Agreement is amended in the following respects:

               1.   Section 1.1 is amended by adding the following definitions
                    thereto in the appropriate alphabetical order:

                    "Apparel Joint Venture" shall mean that certain foreign
                     ---------------------
                    joint venture between a wholly-owned subsidiary of the
                    Borrower, as the minority holder and Grupo Industrial Zaga,
                    S.A. de C.V., as the majority holder.

                    "Textile Joint Venture" shall mean that certain foreign
                     ---------------------
                    joint venture between a wholly-owned subsidiary of the
                    Borrower, as the majority holder and Grupo Industrial Zaga,
                    S.A. de C.V., as the minority holder.

               2.   The definition of "Permitted Liens" in Section 1.1 is hereby
                    amended by adding the following as subclause (xi):

                    (xi) Liens on assets of the Textile Joint Venture arising in
                    connection with Indebtedness permitted pursuant to Section
                    6.1(k) hereof; provided, however that no additional Liens,
                                   --------
                    other than Liens in favor of the Borrower, shall be
                    permitted
<PAGE>

                    in connection with any assets which are collateral for
                    intercompany Indebtedness of the Textile Joint Venture to
                    the Borrower.

               3.   The definition of "Permitted Investments" in Section 1.1 is
                    hereby amended by adding the following subclause (ix)
                    thereto:

                    (ix) Investments in the Apparel Joint Venture in an amount
                    not to exceed $8,000,000 in the aggregate.

               4.   Section 2.7(b)(ii) is hereby amended by making the additions
                    and deletions set forth in bold below:

                         (ii)  Asset Dispositions. Promptly following any Asset
                               ------------------
                    Disposition or series of Asset Dispositions (excluding for
                    the purposes hereof, the Asset Disposition by the Borrower
                    to the Textile Joint Venture permitted pursuant to Section
                    6.5(a)(vi)) in an aggregate amount in excess of $10,000,000
                    in any fiscal year, the Borrower shall prepay the Loans in
                    an aggregate amount equal to the Net Cash Proceeds derived
                    from such Asset Dispositions (such prepayment to be applied
                    as set forth in clause (v) below).

               5.   Section 5.1(a) is hereby amended by making the additions and
                    deletions set forth in bold below:

                         (a) Annual Financial Statements.  As soon as available,
                             ---------------------------
                    but in any event within ninety (90) days after the end of
                    each fiscal year of the Borrower, a copy of the consolidated
                    and consolidating balance sheet of the Borrower and its
                    consolidated Subsidiaries as at the end of such fiscal year
                    and the related consolidated and consolidating statements of
                    income and retained earnings and of cash flows of the
                    Borrower and its consolidated Subsidiaries for such year
                    audited by a firm of independent certified public
                    accountants of nationally recognized standing reasonably
                    acceptable to the Required Lenders, setting forth in each
                    the case of consolidated reports, in comparative form the
                    figures for the previous year, reported on without a "going
                    concern" or like qualification or exception, or
                    qualification indicating that the scope of the audit was
                    inadequate to permit such independent certified public
                    accountants to certify such financial statements without
                    such qualification, provided that for the purposes of this
                                        --------
                    subsection 5.1(a), the term "consolidating" shall be
                    understood to mean the segregation of the financial
                    information of the Borrower and the Guarantors from the
                    financial information of the Subsidiaries which are not
                    Guarantors hereunder;

                                       2
<PAGE>

               6.  Section 5.1(b) is hereby amended by making the additions and
               deletions set forth in bold below:

                         (b) Quarterly Financial Statements.  As soon as
                             ------------------------------
                    available and in any event within forty-five (45) days after
                    the end of each of the first three fiscal quarters of the
                    Borrower, company-prepared consolidated and consolidating
                    balance sheets of the Borrower and its consolidated
                    Subsidiaries as at the end of such period and related
                    company-prepared statements of income and retained earnings
                    and of cash flows (year-to-date) on both a consolidated and
                    consolidating basis for the Borrower and its consolidated
                    Subsidiaries for such quarterly period and for the portion
                    of the fiscal year ending with such period setting forth in
                    comparative form consolidated figures for the corresponding
                    period or periods of the preceding fiscal year (subject to
                    normal recurring year-end audit adjustments), provided that
                                                                  --------
                    for the purposes of this subsection 5.1(b), the term
                    "consolidating" shall be understood to mean the segregation
                    of the financial information of the Borrower and the
                    Guarantors from the financial information of the
                    Subsidiaries which are not Guarantors hereunder; and

               7.   Section 5.9 is hereby amended by adding the word
                    "Restricted" immediately before the word "Subsidiaries" in
                    the second line thereof.

               8.   Section 5.12 is hereby amended by adding the phrase "(other
                    than the Textile Joint Venture)" immediately following the
                    phrase "each of its Subsidiaries" in the first and sixth
                    lines thereof.

               9.   Section 6.1 is hereby amended by adding the following clause
                    (k) to the end thereof:

                    (k) Indebtedness of the Textile Joint Venture which is not
                    Recourse Debt.

               10.  Section 6.3 is hereby amended by making the additions and
                    deletions set forth in bold below:

                    The Borrower will not, nor will it permit any Subsidiary to,
                    enter into or otherwise become or be liable in respect of
                    any Guaranty Obligations (excluding specifically therefrom
                    endorsements in the ordinary course of business of
                    negotiable instruments for deposit or collection) other than
                    (i) those in favor of the Lenders in connection herewith,
                    (ii) Guaranty Obligations by the Borrower or its
                    Subsidiaries of Indebtedness permitted under Section 6.1(g)
                    (except, as regards Indebtedness under subsection (b)
                    thereof, only if and to the extent such Indebtedness was
                    guaranteed on the Closing Date) and (iii) Guaranty
                    Obligations by the Textile Joint Venture permitted under
                    Section 6.1(k).

               11.  Section 6.5(a) is hereby amended by making the additions and
                    deletions set forth in bold below:

                         (a) dissolve, liquidate or wind up its affairs, sell,
                    transfer, lease or otherwise dispose of its property or
                    assets or agree to do so at a future time except the
                    following, without duplication, shall be expressly
                    permitted:

                                       3
<PAGE>

                              (i)    Specified Sales;

                              (ii)   the sale, transfer, lease or other
                    disposition of property or assets (a) to an unrelated party
                    not in the ordinary course of business (other than Specified
                    Sales), where and to the extent that they are the result of
                    a Recovery Event or (b) the sale, lease, transfer or other
                    disposition of machinery, parts and equipment no longer used
                    or useful in the conduct of the business of the Borrower or
                    any of its Subsidiaries, as appropriate, in its reasonable
                    discretion, so long as the net proceeds therefrom are used
                    to repair or replace damaged property or to purchase or
                    otherwise acquire new assets or property, provided that such
                                                              --------
                    purchase or acquisition is committed to within 180 days of
                    receipt of the net proceeds and such purchase or acquisition
                    is consummated within 270 days of receipt of such proceeds;

                              (iii)  the sale, lease or transfer of property or
                    assets (at fair value) between the Borrower and any
                    Guarantor;

                              (iv)   the sale, lease or transfer of property or
                    assets from a Credit Party other than the Borrower to
                    another Credit Party;

                              (v)    the sale, lease or transfer of property or
                    assets not to exceed $10,000,000 in the aggregate in any
                    fiscal year; and

                              (vi)   the sale, lease or transfer of property
                    comprising approximately 46% of the Borrower's U.S. apparel
                    fabrics manufacturing capacity as of December 28, 1999 to
                    the Textile Joint Venture, provided that the sale, lease or
                                               --------
                    transfer is for fair market value and consideration;

                    provided, that in the case of each of (i) through (v)
                    --------
                    above (other than the liquidation of all or substantially
                    all of the assets of the Acquired Business into the Borrower
                    which shall be excluded for purposes hereof) at least 75% of
                    the consideration received therefor by the Borrower or any
                    such Subsidiary is in the form of cash or Cash Equivalents;
                    or

               12.  Section 6.7 is hereby amended by making the additions set
                    forth in bold below:

                         Except (i) as permitted in subsection (iv) of the
                    definition of Permitted Investments, (ii) in the case of the
                    transactions between the Borrower and the Textile Joint
                    Venture, (iii) in the case of the transactions between the
                    Borrower and the Apparel Joint Venture and (iv) otherwise to
                    an extent not judged material by the Required Lenders in
                    their discretion, the Borrower will not, nor will it permit
                    any Subsidiary to, enter into any transaction or series of
                    transactions, whether or not in the ordinary course of
                    business, with any officer, director, shareholder or
                    Affiliate other than on terms and conditions substantially
                    as favorable as would be obtainable in a comparable arm's-
                    length transaction with a Person other than an officer,
                    director, shareholder or Affiliate.

                                       4
<PAGE>

               13.  Section 6.8 is hereby amended by making the additions set
                    forth in bold below:

                         The Borrower will not, nor will it permit any
                    Subsidiary to, create, form or acquire any Subsidiaries,
                    except for (i) Domestic Subsidiaries which are joined as
                    Additional Credit Parties in accordance with the terms
                    hereof, (ii) the Textile Joint Venture and (iii) an indirect
                    wholly-owned Subsidiary which, if formed, shall be a Dutch
                    holding company which will be the majority holder in the
                    Textile Joint Venture and the minority holder in the Apparel
                    Joint Venture. The Borrower will not sell, transfer, pledge
                    or otherwise dispose of any Capital Stock or other equity
                    interests in any of its Subsidiaries, nor will it permit any
                    of its Subsidiaries to issue, sell, transfer, pledge or
                    otherwise dispose of any of their Capital Stock or other
                    equity interests, except in a transaction permitted by
                    Section 6.5.

               14.  Section 6.10 is hereby amended by adding the phrase
                    "except as may be required by the Textile Joint Venture
                    pursuant to its Members Agreement" immediately after the
                    phrase "permit any Subsidiary to" in the first line thereof.

               15.  Section 6.11 is hereby amended by making the additions and
                    deletions set forth in bold below:

                    The Borrower will not, nor will it permit any Subsidiary to,
                    directly or indirectly, declare, order, make or set apart
                    any sum for or pay any Restricted Payment, except (a) to
                    make dividends payable solely in the same class of Capital
                    Stock of such Person, (b) to make dividends or other
                    distributions payable to any Credit Party or in the case of
                    the Textile Joint Venture, to each of its partners (directly
                    or indirectly through Subsidiaries), (c) as permitted by
                    Section 6.12, (d) provided that no Default or Event of
                    Default has occurred and is continuing at such time or would
                    be directly or indirectly caused as a result thereof, the
                    Borrower may pay cash dividends or repurchase shares of its
                    Capital Stock in an aggregate amount not to exceed
                    $5,000,000 in any fiscal year and (e) distributions by the
                    Textile Joint Venture to pay the partnership tax liabilities
                    of its partners.

               16.  Section 6.12 is hereby amended by adding the following
                    proviso to the end thereof:

                    provided, however that the Textile Joint Venture may prepay
                    --------
                    Indebtedness permitted hereunder at any time.

          B.   Except as modified hereby, all of the terms and provisions of the
Credit Agreement (and Exhibits) remain in full force and effect.

          C.   The Credit Parties hereby represent and warrant that (a) the
representations and warranties contained in Article III of the Credit Agreement,
as amended hereby are correct in all material respects on and as of the date
hereof as though made on and as of such date and after giving effect to the
amendments contained herein and (b) no Default or Event of Default exists on and
as of the date hereof and after giving effect to the amendments contained
herein.

          D.   The Guarantors acknowledge and consent to all of the terms and
conditions of this Second Amendment and agree that this Second Amendment and all
documents executed in connection herewith do

                                       5
<PAGE>

not operate to reduce or discharge the Guarantors' obligations under the Credit
Agreement or the other Credit Documents. The Guarantors further acknowledge and
agree that the Guarantors have no claims, counterclaims, offsets, or defenses to
the Credit Documents and the performance of the Guarantors' obligations
thereunder or if the Guarantors did have any such claims, counterclaims, offsets
or defenses to the Credit Documents or any transaction related to the Credit
Documents, the same are hereby waived, relinquished and released in
consideration of the Required Lenders' execution and delivery of this Second
Amendment.

     E.   This Second Amendment may be executed in any number of counterparts,
each of which when so executed and delivered shall be deemed an original and it
shall not be necessary in making proof of this Second Amendment to produce or
account for more than one such counterpart.

     F.   This Second Amendment and the Credit Agreement, as amended hereby,
shall be deemed to be contracts made under, and for all purposes shall be
construed in accordance with the laws of the State of North Carolina.

                                       6
<PAGE>

     IN WITNESS WHEREOF, each of the parties hereto has caused a counterpart of
this Second Amendment to be duly executed and delivered as of the date and year
first above written.


BORROWER:                DAN RIVER, INC.,
--------                 a Georgia corporation


                         By: ______________________________________
                              Name:
                              Title:


                         THE BIBB COMPANY,
                         a Delaware corporation


                         By: ______________________________________
                              Name:
                              Title:


                         DAN RIVER FACTORY STORES, INC.,
                         a Georgia corporation


                         By: ______________________________________
                              Name:
                              Title:
<PAGE>

AGENTS AND LENDERS:      FIRST UNION NATIONAL BANK,
------------------
                         as Administrative Agent and as a Lender


                         By: ______________________________________
                              Name:
                              Title:
<PAGE>

                         BANK ONE,
                         formerly The First National Bank of Chicago,
                         as Syndication Agent and as a Lender


                         By: ______________________________________
                              Name:
                              Title:
<PAGE>

                         WACHOVIA BANK, N.A.,
                         as Documentation Agent and as a Lender


                         By: ______________________________________
                              Name:
                              Title:
<PAGE>

LENDERS:                 SUNTRUST BANK,
                         as a Lender


                         By: ______________________________________
                              Name:
                              Title:
<PAGE>

                         THE BANK OF NOVA SCOTIA,
                         as a Lender


                         By: ______________________________________
                              Name:
                              Title:
<PAGE>

                         COMERICA BANK,
                         as a Lender


                         By: ______________________________________
                              Name:
                              Title:
<PAGE>

                         COOPERATIVE CENTRALE RAIFFEISEN-
                         BOERENLEENBANK B.A. "RABOBANK NEDERLAND", NEW YORK
                         BRANCH, as a Lender


                         By: ______________________________________
                              Name:
                              Title:



<PAGE>

                         CENTURA BANK,
                         as a Lender


                         By: ______________________________________
                              Name:
                              Title:


<PAGE>

                         FLEET BANK, N.A.,
                         as a Lender


                         By: ______________________________________
                              Name:
                              Title:
<PAGE>

                         ABN AMRO BANK N.V.,
                         as a Lender


                         By: ______________________________________
                              Name:
                              Title:
<PAGE>

                         THE BANK OF NEW YORK,
                         as a Lender


                         By: ______________________________________
                              Name:
                              Title:
<PAGE>

                         NATIONAL BANK OF CANADA,
                         as a Lender


                         By: ______________________________________
                              Name:
                              Title:




<PAGE>

                         SOUTHTRUST BANK, N.A.,
                         as a Lender


                         By: ______________________________________
                              Name:
                              Title:
<PAGE>

                         NATIONAL CITY BANK OF KENTUCKY,
                         as a Lender


                         By: ______________________________________
                              Name:
                              Title: