<PAGE>   1
                                                                   EXHIBIT 10.31

                      FOURTH AMENDMENT TO CREDIT AGREEMENT

         THIS FOURTH AMENDMENT TO CREDIT AGREEMENT (the "Fourth Amendment") is
entered into to be effective as of November 21, 1997, by and among F.Y.I.
Incorporated, a Delaware corporation ("F.Y.I."), Imagent Acquisition Corp., a
Delaware corporation ("Imagent"), Researchers Acquisition Corp., a Delaware
corporation ("Researchers"), Recordex Acquisition Corp., a Delaware corporation
("Recordex"), DPAS Acquisition Corp., a Delaware corporation ("DPAS"), Leonard
Archives Acquisition Corp., a Delaware corporation ("Leonard"), Deliverex
Acquisition Corp., a Delaware corporation ("Deliverex"), Permanent Records
Acquisition Corp., a Delaware corporation ("Permanent"), Deliverex Sacramento
Acquisition Corp., a Delaware corporation ("Sacramento"), (F.Y.I., Imagent,
Researchers, Recordex, DPAS, Leonard, Deliverex, Permanent and Sacramento are
collectively referred to as the "Original Borrowers"), B&B (Baltimore-
Washington) Acquisition Corp., a Delaware corporation ("B&B"), Premier
Acquisition Corp., a Delaware corporation ("Premier"), Robert A. Cook
Acquisition Corp., a Delaware corporation ("Cook"), Peninsula Record
Management, Inc., a California corporation ("Peninsula"), RAC (California)
Acquisition Corp., a Delaware corporation ("RAC"), California Medical Record
Service Acquisition Corp., a Delaware corporation ("California Medical"),
Minnesota Medical Record Service Acquisition Corp., a Delaware corporation
("Minnesota Medical"), Texas Medical Record Service Acquisition Corp., a
Delaware corporation ("Texas Medical"), ZIA Information Analysis Group, Inc.,
formerly known as ZIA Acquisition Corp., a Delaware corporation ("Zia"), CH
Acquisition Corp., a Delaware Corporation ("CH") and DISC Acquisition Corp., a
Delaware corporation ("DISC") (B&B, Premier, Cook, Peninsula, RAC, California
Medical, Minnesota Medical, Texas Medical, Zia, CH and DISC are referred to
collectively as the "New Borrowers") (the Original Borrowers and the New
Borrowers are referred to collectively as the "Borrowers"), Banque Paribas, a
bank organized under the laws of the Republic of France, as Agent (the
"Agent"), and the Lenders (as such term is defined in the Credit Agreement, as
hereinafter defined) which are parties hereto.

                                    RECITALS

         A.      The Original Borrowers, the Agent and the Lenders entered into
that certain Credit Agreement dated as of April 18, 1996 (the "Original Credit
Agreement"), pursuant to which, among other things, the Lenders agreed to make
certain loans available to the Original Borrowers upon the terms and conditions
set forth therein;

         B.      The Borrowers, certain of the New Borrowers, the Agent and the
Lenders entered into that certain First Amendment to Credit Agreement dated as
of June 26, 1996, that certain Second Amendment to Credit Agreement dated as of
August 30, 1996, and that certain Third Amendment to Credit Agreement dated as
of December 18, 1996 (the Original Credit Agreement, as amended, is hereinafter
referred to as the "Credit Agreement").





FOURTH AMENDMENT TO CREDIT AGREEMENT                                      Page 1
<PAGE>   2
         C.      The Borrowers, the Agent and the Lenders desire to amend the
Credit Agreement in certain respects as more fully set out herein.

                                   AGREEMENT

         NOW, THEREFORE, for and in consideration of the premises and other
valuable consideration, the receipt and sufficiency of which are hereby
acknowledged, the Borrowers, the Lenders, and the Agent hereby agree as
follows:

         1.      Terms.  All terms used herein which begin with an initial
capital letter shall, unless otherwise expressly defined herein, have the same
definitions assigned to such terms in the Credit Agreement, as modified by this
Fourth Amendment.

         2.      Amendment to Definition of "Term Loans Commitments Termination
Date."  Effective as of the date hereof, the definition of "Term Loans
Commitments Termination Date" contained in Section 1.1 of the Credit Agreement
is hereby amended by deleting the phrase "October 15, 1997" and replacing it
with the phrase "March 31, 1998".

         3.      Amendment to Section 2.1(c) of the Credit Agreement.
Effective as of the date hereof, Section 2.1(c) of the Credit Agreement is
hereby amended to read in its entirety as follows:

                 (c)      Term Loans.  Subject to the terms and conditions of
         this Agreement, each Lender severally agrees to make one or more term
         loans to the Term Loans Borrowers from time to time from and including
         the Closing Date to but excluding the Term Loans Commitments
         Termination Date up to but not exceeding the amount of such Lender's
         Term Loans Commitment.  (Such term loans referred to in this Section
         2.1(c) made by the Lenders to the Term Loans Borrowers are hereinafter
         collectively called the "Term Loans".)  Notwithstanding anything else
         herein or elsewhere to the contrary, the obligation of each Lender to
         make Term Loans shall terminate on the Term Loans Commitments
         Termination Date.

         4.      Amendment to Section 2.3(c) of the Credit Agreement.
Effective as of the date hereof, Section 2.3(c) of the Credit Agreement is
hereby amended to read in its entirety as follows:

                 (c)      Each of the Term Loans Borrowers shall jointly and
         severally pay to the Agent for the account of each applicable Lender
         the outstanding principal of the Term Loans existing as of the Term
         Loans Commitments Termination Date (and the outstanding principal for
         the Term Loans shall be due and payable) in thirteen (13) equal
         quarterly installments on the dates set forth below





FOURTH AMENDMENT TO CREDIT AGREEMENT                                      Page 2
<PAGE>   3
         in an amount sufficient to amortize fully the Term Loans by the Term
         Loans Maturity Date:


                          Payment Date

                          April 15, 1998
                          July 15, 1998
                          October 15, 1998
                          January 15, 1999
                          April 15, 1999
                          July 15, 1999
                          October 15, 1999
                          January 15, 2000
                          April 15, 2000
                          July 15, 2000
                          October 15, 2000
                          January 15, 2001
                          April 15, 2001

         5.      Amendment to Section 2.11(a) of the Credit Agreement.
Effective as of the date hereof, the second sentence in Section 2.11(a) of the
Credit Agreement is hereby amended by deleting the phrase "October 15, 1997"
wherever located in such sentence and replacing it with the phrase "Term Loans
Commitments Termination Date" in each such location.

         6.      Amendment to Section 10.6 of the Credit Agreement.  Effective
as of the date hereof, the chart in Section 10.6 of the Credit Agreement is
hereby amended and restated to read in its entirety as follows:

<TABLE>
<CAPTION>
                 Ending on December 31                              Amount
                 ---------------------                              ------
                 <S>                                              <C>
                 1996                                             $1,750,000
                 1997                                             $2,600,000
                 1998                                             $2,750,000
                 1999                                             $2,950,000
                 2000 and thereafter                              $3,150,000
</TABLE>

         7.      Representations and Warranties.  The representations and
warranties made by the Borrowers in the Loan Documents, as the same are amended
hereby, are true and correct at the time this Fourth Amendment is executed and
delivered, except to the extent that such representations and warranties are
expressly by their terms made only as of the Closing Date or another specified
date.





FOURTH AMENDMENT TO CREDIT AGREEMENT                                     Page 3 
<PAGE>   4
         8.      Costs.  The Borrowers jointly and severally agree to pay all
costs incurred in connection with the negotiation, preparation, execution and
consummation of this Fourth Amendment and the transactions preceding and
contemplated by this Fourth Amendment including, without limitation, the fees
and expenses of counsel to the Agent and the Lenders.

         9.      Miscellaneous.

                 (a)      Headings.  Section headings are for reference only,
         and shall not affect the interpretation or meaning of any provision of
         this Fourth Amendment.

                 (b)      No Waiver.  No failure on the part of the Agent or
         the Lenders to exercise, and no delay in exercising, and no course of
         dealing with respect to, any right, power, or privilege under the Loan
         Documents shall operate as a waiver thereof, and no single or partial
         exercise of any right, power, or privilege under the Loan Documents
         shall preclude any other or further exercise thereof or the exercise
         of any other right, power, or privilege.

                 (c)      Effect of this Fourth Amendment.  The Credit
         Agreement, as amended by this Fourth Amendment, shall remain in full
         force and effect except that any reference therein, or in any other
         Loan Document, referring to the Credit Agreement, shall be deemed to
         refer to the Credit Agreement, as amended by this Fourth Amendment.

                 (d)      Governing Law.  EXCEPT TO THE EXTENT THAT THE CREDIT
         AGREEMENT EXPRESSLY PROVIDES OTHERWISE, THIS THIRD AMENDMENT SHALL BE
         GOVERNED BY, AND CONSTRUED IN ACCORDANCE WITH, THE LAWS OF THE STATE
         OF TEXAS.

                 (e)      Counterparts.  This Fourth Amendment may be executed
         by the different parties  hereto on separate counterparts, each of
         which, when so executed, shall be deemed an original, but all such
         counterparts shall constitute but one and the same Fourth Amendment.

                 (f)      NO ORAL AGREEMENTS.  THE CREDIT AGREEMENT, AS AMENDED
         BY THIS THIRD AMENDMENT, TOGETHER WITH THE OTHER LOAN DOCUMENTS,
         REPRESENTS THE ENTIRE AGREEMENT AMONG THE PARTIES, AND MAY NOT BE
         CONTRADICTED BY EVIDENCE OF PRIOR, CONTEMPORANEOUS, OR SUBSEQUENT ORAL
         AGREEMENTS OF THE PARTIES.  THERE ARE NO UNWRITTEN ORAL AGREEMENTS
         AMONG THE PARTIES.





FOURTH AMENDMENT TO CREDIT AGREEMENT                                      Page 4
<PAGE>   5
         IN WITNESS WHEREOF, the parties hereto have caused this Fourth
Amendment to be executed by their respective duly authorized officers as of the
date first above written.


                                       BORROWERS:

                                       F.Y.I. INCORPORATED


                                       By: /s/ David Lowenstein               
                                           -----------------------------------
                                           David Lowenstein
                                           Executive Vice President

                                       IMAGENT ACQUISITION CORP.
                                       RESEARCHERS ACQUISITION CORP.
                                       RECORDEX ACQUISITION CORP.
                                       DPAS ACQUISITION CORP.
                                       LEONARD ARCHIVES ACQUISITION CORP.
                                       DELIVEREX ACQUISITION CORP.
                                       PERMANENT RECORDS ACQUISITION CORP.
                                       DELIVEREX SACRAMENTO ACQUISITION CORP.
                                       B&B (BALTIMORE-WASHINGTON)
                                         ACQUISITION CORP.
                                       PREMIER ACQUISITION CORP.
                                       ROBERT A. COOK ACQUISITION CORP.
                                       PENINSULA RECORD MANAGEMENT, INC.
                                       RAC (CALIFORNIA) ACQUISITION CORP.
                                       CALIFORNIA MEDICAL RECORD SERVICE
                                         ACQUISITION CORP.
                                       MINNESOTA MEDICAL RECORD SERVICE
                                         ACQUISITION CORP.
                                       TEXAS MEDICAL RECORD SERVICE
                                         ACQUISITION CORP.
                                       ZIA INFORMATION ANALYSIS GROUP, INC.
                                       CH ACQUISITION CORP.
                                       DISC ACQUISITION CORP.


                                       By: /s/ David Lowenstein               
                                           -----------------------------------
                                           David Lowenstein
                                           Executive Vice President, acting 
                                           on behalf of each of the above





FOURTH AMENDMENT TO CREDIT AGREEMENT                                      Page 5
<PAGE>   6

                                       LENDERS:
                                       
                                       BANQUE PARIBAS, as Agent and as a Lender


                                       By:    /s/ Clark C. King, III          
                                              --------------------------------
                                       Name:  Clark C. King, III
                                       Title: Director


                                       By:    /s/ Peter Toal                  
                                              --------------------------------
                                       Name:  Peter Toal
                                       Title: Managing Director


                                       FIRST SOURCE FINANCIAL LLP
                                       
                                       By:    FIRST SOURCE FINANCIAL, INC., its
                                              Agent/Manager

                                              By:    /s/ John Walding     
                                                  -----------------------------
                                                  Name:  John Walding
                                                  Title: Vice President


                                       IBJ SCHRODER BANK & TRUST COMPANY


                                       By:    /s/ Mark Minter                 
                                              --------------------------------
                                              Name:  Mark Minter
                                              Title:   Director





FOURTH AMENDMENT TO CREDIT AGREEMENT                                      Page 6