<PAGE>   1
                                                                EXHIBIT 10.18.06



                                FOURTH AMENDMENT
                       TO SECOND RESTATED CREDIT AGREEMENT


         This Fourth Amendment to Second Restated Credit Agreement (this "Fourth
Amendment"), dated as of September 29, 1998, is entered into among Cameron
Ashley Building Products, Inc., a Georgia corporation, Wm. Cameron & Co., a
Georgia corporation, Ashley Aluminum, Inc., a Georgia corporation, CABP, Inc.,
an Arizona corporation, Cameron Ashley Canada, Inc., a Canadian corporation,
NationsBank, N.A., successor by merger to NationsBank of Texas, National
Association, as Issuing Bank and Agent, ABN AMRO Bank, N.V., as Co-Agent,
Canadian Imperial Bank of Commerce, as Canadian Issuing Bank and Canadian Agent,
and each Lender.

                                   BACKGROUND

         Borrowers, Agent, Co-Agent, Issuing Bank, Canadian Agent, Canadian
Issuing Bank and Lenders have entered into the Second Restated Credit Agreement
dated as of January 29, 1997 (such agreement, together with all amendments and
restatements thereof, the "Credit Agreement"). Borrower has requested that
Lenders, Agent, Issuing Bank, Co-Agent, Canadian Agent and Canadian Issuing Bank
amend the Credit Agreement to, among other things, modify certain covenants.

                                    AGREEMENT

         NOW, THEREFORE, in consideration of the covenants, conditions and
agreements hereafter set forth, and for other good and valuable consideration,
the receipt and adequacy of which are all hereby acknowledged, Borrowers, Agent,
Co-Agent, Issuing Bank, Canadian Agent, Canadian Issuing Bank, Lenders and
Guarantors covenant and agree as follows:

1.   Defined Terms. Capitalized terms used herein and not otherwise defined
herein have the meaning given to them in the Credit Agreement.

2.   Amendments. The Credit Agreement is amended as follows:

     (a) Section 1.01 is amended by adding the following in alphabetical order:

     (b) The definition of "EBITDA" is deleted, and the following is
substituted in lieu thereof:

         "EBITDA" means, as of any date of determination, (a) the sum of
     Parent's and its Subsidiaries' (i) pre-tax income or deficit, as the case
     may be (excluding extraordinary items and income from the sale of assets
     other than in the ordinary course of business), plus (ii) cash interest
     expense paid or interest expense accrued; amortization of Debt discounts;
     any payments or fees with respect to letters of credit, bankers'
     acceptances or similar facilities; fees and expenses with respect to
     interest rate swap or similar agreements or foreign currency hedge,
     exchange or similar 



<PAGE>   2

     agreements, plus (iii) depreciation and amortization expense, plus (iv)
     cash interest paid with respect to Capital Leases, plus (v) other non-cash
     charges of Parent and its Subsidiaries for such period deducted from
     consolidated revenues in determining net income for such period, plus (vi)
     all charges taken by Parent during fiscal year 1998 in connection with the
     phase-out of CAFS (not to exceed in the aggregate $3,000,000), plus (vii)
     with respect to Permitted Acquisitions occurring at any time during the
     four fiscal quarters immediately preceding the date of calculation of
     EBITDA, the pro forma trailing 12 months EBITDA (but calculated to exclude
     any increases in EBITDA which would be the result of any expenses that the
     Parent projects to be eliminated by such Permitted Acquisition)
     attributable to such Permitted Acquisition; and minus (b) (i) with respect
     to assets not owned at all times during the four fiscal quarters
     immediately preceding the date of calculation of EBITDA, the EBITDA
     attributable to any assets disposed of during such four fiscal quarters,
     and (ii) non-cash items of Parent and its Subsidiaries for such period
     increasing consolidated revenues in determining net income for such period,
     all calculated on a consolidated basis in accordance with GAAP.

     (c) The definition of "EBITDAR" is deleted, and the following is
substituted in lieu thereof:

         "EBITDAR" means, as of any date of determination, (a) the sum of
     Parent's and its Subsidiaries' (i) pre-tax income or deficit, as the case
     may be (excluding extraordinary items and income from the sale of assets
     other than in the ordinary course of business), plus (ii) cash interest
     expense paid or interest expense accrued; amortization of Debt discounts;
     any payments or fees with respect to letters of credit, bankers'
     acceptances or similar facilities; fees and expenses with respect to
     interest rate swap or similar agreements or foreign currency hedge,
     exchange or similar agreements, plus (iii) depreciation and amortization
     expense, plus (iv) lease payments paid pursuant to Operating Leases, plus
     (v) cash interest paid with respect to Capital Leases, plus (vi) other
     non-cash charges of Parent and its Subsidiaries for such period deducted
     from consolidated revenues in determining net income for such period, plus;
     (vii) all charges taken by Parent during fiscal year 1998 in connection
     with the phase-out of CAFS (not to exceed in the aggregate $3,000,000),
     plus (viii) with respect to Permitted Acquisitions occurring at any time
     during the four fiscal quarters immediately preceding the date of
     calculation of EBITDA, the pro forma trailing 12 months EBITDA (but
     calculated to exclude any increases in EBITDA which would be the result of
     any expenses that the Parent projects to be eliminated by such Permitted
     Acquisition) attributable to such Permitted Acquisition; and minus (b) (i)
     with respect to assets not owned at all times during the four fiscal
     quarters immediately preceding the date of calculation of EBITDA, the
     EBITDA attributable to any assets disposed of during such four fiscal
     quarters, and (ii) non-cash items of Parent and its Subsidiaries for such
     period increasing consolidated revenues in 



                                      -2-
<PAGE>   3

     determining net income for such period, all calculated on a consolidated
     basis in accordance with GAAP.

     (d) Section 5.02 is deleted and the following is substituted in lieu
thereof:

         5.02. Tangible Net Worth. Parent shall not permit Tangible Net Worth as
     at any date to be less than the sum of (a) $35,000,000, plus (b) 50% of
     cumulative Net Income reported for each fiscal quarter of Parent in which
     Net Income is a positive number commencing with the fiscal quarter of
     Parent ending on October 31, 1996, plus (c)100% of Equity Proceeds.

     (e) Section 5.09 is amended by deleting the reference to "$5,000,000" in
parts "(c)" and "(d)" and substituting "$20,000,000" in lieu thereof and by
adding the words "or series of related transactions" after the phrase, "(d)
Permitted Acquisitions provided that the consideration paid in any such
acquisition" in part "(d)".

     (f) Section 5.20(b) is deleted and the following is substituted in lieu
thereof:

         (b) Neither Parent nor any Subsidiary of Parent shall be liable at any
     time for any CAFS Liability or extend credit to or for the benefit of CAFS
     or any CAFS Subsidiary; provided, (i) Parent may (A) acquire for cash
     equity of CAFS (in addition to all equity of CAFS owned by Parent on
     December 3, 1996), and (B) make loans to CAFS, or (c) do any combination of
     activities described in clauses (A) and (B) , and (ii) Cameron may make
     advances to CAFS, the proceeds of which advances will be used by CAFS to
     pay corporate overhead expenses of CAFS; provided, that, the sum of the
     aggregate amount of all equity acquired, unpaid principal and accrued
     interest on all loans referenced above, shall not exceed at any time
     $4,000,000; provided, that, Parent may not make or have outstanding any
     loan to or other extension of credit to or for the benefit of CAFS if at
     any time Parent is not the sole owner of all equity and rights to acquire
     any equity of CAFS; and provided further, that, CAFS shall not engage in
     any activities involving the underwriting and funding of loans other than
     extensions of credit to a CAFS Subsidiary.

     (g) Section 5.22 is deleted and the following is substituted in lieu
thereof:

         5.22 Intercompany Advances. Neither Parent nor any Guarantor or CAFS
     shall make any loans, advances, extensions of credit to, or Investments in
     CA Canada such that the total amount outstanding of such loans, advances,
     extensions of credit or Investments at any one time shall be in excess of
     $30,000,000.



                                      -3-
<PAGE>   4

     (h) A new Section 5.24 is added as follows:

         5.24 Ownership and Structure of Ashley. Notwithstanding any other
     provision of this Agreement, Parent shall not cease to own 100% of Ashley
     or change or allow the organizational structure of Ashley to be changed,
     provided, however, (a) Parent may transfer its ownership interest in Ashley
     to Cameron, and (b) Parent or Cameron may change the organizational
     structure of Ashley to a limited liability company, provided that, (I)
     Parent provide Agent two business days notice of any transfer or
     organizational change, (II) at the time of such transfer and organizational
     change, Parent provide Agent with a new Schedule 4.01-b, and (III) Ashley's
     successor entity assume each of Ashley's Obligations and any other
     responsibilities, including but not limited to reporting responsibilities
     under the Agreement.

     (i) Schedule 4.08-a is deleted and a new Schedule 4.08-a, in the form of
Schedule 4.08-a to this Fourth Amendment, is substituted in lieu thereof.

     (j) Exhibit C is deleted and a new Exhibit C, in the form of Exhibit A to
this Fourth Amendment, is substituted in lieu thereof.

3.   Representations and Warranties. Borrowers and Guarantors, jointly and
severally, represent and warrant to Agent, Issuing Bank, Co-Agent, Canadian
Agent, Canadian Issuing Bank and each Lender that, as of the date hereof and
after giving effect to the amendments in Section 2, the following are true and
correct:

     (a) The representations and warranties contained in the Credit Agreement
and each of the other Loan Papers are true and correct in all material respects
on and as of the date hereof as though made on and as of such date (except as to
representations and warranties which (i) refer to a specific date, (ii) have
been modified by transactions permitted pursuant to the Credit Agreement or any
other Loan Paper, or (iii) have been specifically waived in writing by Agent).

     (b) Each Borrower and each Guarantor has full power and authority to
execute, deliver and perform this Fourth Amendment, and this Fourth Amendment,
the Credit Agreement and each other Loan Paper, constitute the legal, valid and
binding obligation of such Borrower and such Guarantor (with respect to Loan
Papers to which it is a party), enforceable in accordance with their terms
(subject as to enforcement of remedies to any applicable Debtor Relief Laws).

     (c) No authorization, approval, consent or other action by, notice to, or
filing with, any Tribunal or other Person, is required for the execution,
delivery or performance by either Borrower or any Guarantor of this Fourth
Amendment.

     (d) No Obligor has made a material misstatement of fact, or failed to
disclose any fact necessary to make the facts disclosed not misleading, to
Agent, Issuing Bank, Co-Agent, Canadian 



                                      -4-
<PAGE>   5

Agent, Canadian Issuing Bank or any Lender during the course of negotiation of
this Fourth Amendment.

     4. Conditions of Effectiveness. This Fourth Amendment shall be effective on
the date Agent delivers to Borrower written notice that each of the following
has occurred or exists ("Amendment Date"):

     (a) The effectiveness of this Fourth Amendment shall not contravene any Law
applicable to Agent, Issuing Bank, Co-Agent, Canadian Agent, Canadian Issuing
Bank or any Lender.

     (b) No Material Adverse Change, as determined by Agent, shall have occurred
and be continuing since December 31, 1997.

     (c) No Default or Event of Default shall exist.

     (d) Agent, Issuing Bank, Co-Agent, Canadian Agent, Canadian Issuing Bank,
each Lender and each Obligor shall have executed and received counterparts of
this Fourth Amendment.

     (e) Agent shall have received a complete and correct copy of the executed
Third Amendment to Note Purchase Agreement dated as of July 1, 1998.

     (f) Agent shall have received, contemporaneously with Borrowers' execution
of this Fourth Amendment, payment of all fees (including attorneys' fees)
incurred by Agent prior to execution of this Fourth Amendment in the
preparation, negotiation and execution of this Fourth Amendment.

     (g) Agent shall have received, in form and substance satisfactory to Agent
and its counsel, such other approvals, documents, certificates, and instruments
as Agent shall require.

Agent, Issuing Bank, Co-Agent, Canadian Agent, Canadian Issuing Bank and each
Lender may conclusively rely on the certificates delivered pursuant to Section
3.01 of the Credit Agreement until Agent receives notice in writing to the
contrary.

5.   Ratification. Each Borrower and each Guarantor (a) represents and warrants
that it has received and reviewed this Fourth Amendment and (b) ratifies and
affirms its obligations under the Loan Papers, as amended by this Fourth
Amendment.

6.   Reference to the Credit Agreement.

     (a) On the Amendment Date, each reference in the Credit Agreement to "this
Agreement", "hereunder", or words of like import shall mean and be a reference
to the Credit Agreement, as affected and amended hereby.



                                      -5-
<PAGE>   6

     (b) The Credit Agreement, as affected by the amendments referred to above,
shall remain in full force and effect and is hereby ratified and confirmed.

     (c) THE CREDIT AGREEMENT, AS AFFECTED BY THE AMENDMENTS CONTAINED IN THIS
FOURTH AMENDMENT, TOGETHER WITH EACH OTHER LOAN PAPER, REPRESENT THE FINAL
AGREEMENT AMONG THE PARTIES AND MAY NOT BE CONTRADICTED BY EVIDENCE OF PRIOR,
CONTEMPORANEOUS OR SUBSEQUENT ORAL AGREEMENTS OF THE PARTIES. THERE ARE NO
UNWRITTEN ORAL AGREEMENTS AMONG THE PARTIES.

7.   Execution in Counterparts. This Fourth Amendment may be executed in any
number of counterparts and by different parties hereto in separate counterparts,
each of which when so executed and delivered shall be deemed to be an original
and all of which taken together shall constitute but one and the same
instrument.

8.   Governing Law; Binding Effect. This Fourth Amendment shall be governed by
and construed in accordance with the laws of the State of Texas and be binding
upon the parties hereto and their respective permitted successors and assigns.

9.   Headings. Section headings in this Fourth Amendment are included herein for
convenience of reference only and shall not constitute part of this Fourth
Amendment for any other purpose.


--------------------------------------------------------------------------------
             THE REMAINDER OF THIS PAGE IS INTENTIONALLY LEFT BLANK
--------------------------------------------------------------------------------



                                      -6-
<PAGE>   7


         IN WITNESS WHEREOF, the parties hereto have executed this Fourth
Amendment as of September 29, 1998.


PARENT:
                             CAMERON ASHLEY BUILDING PRODUCTS, INC.


                             By:    /s/ J. Andrew Kerner
                                    --------------------------------------------
                                    J. Andrew Kerner  , EVP, CFO & Treasurer
                                    --------------------------------------------
                                      (Print Name)         (Print Title)


CA CANADA:
                             CAMERON ASHLEY CANADA, INC.


                             By:    /s/ J. Andrew Kerner
                                    --------------------------------------------
                                    J. Andrew Kerner,   EVP
                                    --------------------------------------------
                                      (Print Name)         (Print Title)


GUARANTORS:
                             ASHLEY ALUMINUM, INC.


                             By:    /s/ J. Andrew Kerner
                                    --------------------------------------------
                                    J. Andrew Kerner, VP
                                    --------------------------------------------
                                      (Print Name)         (Print Title)


                             WM. CAMERON & CO.


                             By:    /s/ J. Andrew Kerner
                                    --------------------------------------------
                                    J. Andrew Kerner, EVP, CFO & Treasurer
                                    --------------------------------------------
                                      (Print Name)         (Print Title)




                                      -7-
<PAGE>   8


                             CABP, INC.


                             By:    /s/ Ross A. Goolsby
                                    --------------------------------------------
                                    Ross A. Goolsby,   VP
                                    --------------------------------------------
                                      (Print Name)         (Print Title)



AGENT:
                             NATIONSBANK, N.A., successor by merger to 
                             NationsBank of Texas, National Association


                             By:    /s/ Dan Killian
                                    --------------------------------------------
                                    Daniel M. Killian, Vice President


CANADIAN AGENT:
                             CANADIAN IMPERIAL BANK OF COMMERCE


                             By:    /s/  Lori Berg
                                    --------------------------------------------
                                    Lori Berg                , Credit Designer
                                    --------------------------------------------
                                      (Print Name)         (Print Title)


CO-AGENT:
                             ABN AMRO BANK, N.V.


                             By:    /s/ Robert A. Budnek
                                    --------------------------------------------
                                    Robert A. Budnek , Vice President
                                    --------------------------------------------
                                      (Print Name)         (Print Title)


                             By:    /s/ L. K. Kelley
                                    --------------------------------------------
                                    Larry K. Kelley  , Group Vice President
                                    --------------------------------------------
                                      (Print Name)         (Print Title)



                                      -8-
<PAGE>   9

ISSUING BANK:
                             NATIONSBANK, N.A., successor by merger to 
                             NationsBank of Texas, National Association



                             By:    /s/ Dan Killian
                                    --------------------------------------------
                                    Daniel M. Killian, Vice President


CANADIAN ISSUING BANK:
                             CANADIAN IMPERIAL BANK OF COMMERCE


                             By:    /s/ Lori Berg
                                    --------------------------------------------
                                    Lori Berg                , Credit Designer
                                    --------------------------------------------
                                      (Print Name)         (Print Title)


LENDERS:
                             NATIONSBANK, N.A., successor by merger to 
                             NationsBank of Texas, National Association



                             By:    /s/ Dan Killian
                                    --------------------------------------------
                                    Daniel M. Killian, Vice President


                             CANADIAN IMPERIAL BANK OF COMMERCE


                             By:    /s/ Lori Berg
                                    --------------------------------------------
                                    Lori Berg                , Credit Designer
                                    --------------------------------------------
                                      (Print Name)         (Print Title)



                                      -9-
<PAGE>   10

                             ABN AMRO BANK, N.V.

                             By:    /s/ Robert A. Budnek
                                    --------------------------------------------
                                    Robert A. Budnek , Vice President
                                    --------------------------------------------
                                      (Print Name)         (Print Title)


                             By:    /s/ L. K. Kelley
                                    --------------------------------------------
                                    Larry K. Kelley  , Group Vice President
                                    --------------------------------------------
                                      (Print Name)         (Print Title)


                             WELLS FARGO BANK (TEXAS), N.A.


                             By:    /s/ Juan J. Sanchez
                                    --------------------------------------------
                                    Juan J. Sanchez  , Banking Officer
                                    --------------------------------------------
                                      (Print Name)         (Print Title)


                             SUNTRUST BANK, ATLANTA


                             By:    /s/ D. S. Armstrong
                                    --------------------------------------------
                                    Deborah S. Armstrong,   VP
                                    --------------------------------------------
                                      (Print Name)         (Print Title)


                             By:    /s/ F. McClellan Deaver
                                    --------------------------------------------
                                    F. McClellan Deaver, III, Group Vice 
                                                                President
                                    --------------------------------------------
                                      (Print Name)         (Print Title)



                                      -10-