<PAGE>   1
                                                                  EXHIBIT (10)h.


                           FIFTH AMENDMENT TO MODIFIED
                           AND RESTATED LOAN AGREEMENT

         THIS FIFTH AMENDMENT TO MODIFIED AND RESTATED LOAN AGREEMENT (the
"Fifth Amendment") dated as of November 5, 1999, is to that Modified and
Restated Loan Agreement dated as of September 24, 1997, as amended January 30,
1998, March 31, 1998, August 1, 1998 and December 11, 1998 (hereinafter, such
Loan Agreement as amended hereby, and as further amended or modified from time
to time, the "Loan Agreement"; all terms used but not otherwise defined herein
shall have the meanings provided in the Loan Agreement), by and among GENESCO
INC. (the "Borrower"), the banks and financial institutions on the signature
pages hereto (the "Banks"), BANK ONE, NA (formerly known as The First National
Bank of Chicago), as Co-Agent for the Banks (the "Co-Agent"), and BANK OF
AMERICA, N.A. (formerly known as NationsBank, N.A.), as Agent for the Banks (in
such capacity, the "Agent").

                                   WITNESSETH:

         WHEREAS, the Borrower has requested certain modifications to the Loan
Agreement; and

         WHEREAS, the Banks have agreed to the requested modifications on the
terms and conditions herein set forth;

         NOW, THEREFORE, IN CONSIDERATION of these premises and other good and
valuable consideration, the receipt and sufficiency of which is hereby
acknowledged, the parties hereto agree as follows:

         A.       Effective as of November 5, 1999, Section 7.1(k) shall be
amended in its entirety so that such Section now reads as follows:

                  (k) (i) Liens on accounts receivable which have been sold by
         the Borrower pursuant to that certain Foreign Accounts Receivable
         Factoring Agreement, dated October 13, 1999, by and between the
         Borrower and Suntrust Bank, Atlanta Factoring Division (the "Factoring
         Agreement") provided that (A) the liabilities and obligations incurred
         by the Borrower under the Factoring Agreement (including contingent
         liabilities and obligations) shall not exceed $2,000,000 in the
         aggregate at any time outstanding and (B) the Liens granted by the
         Borrower pursuant to the Factoring Agreement shall only be on the
         accounts receivable sold pursuant to the Factoring Agreement and shall
         not extend to any other assets of the Borrower or any of its
         Subsidiaries; and

                  (ii) Liens on accounts receivable which have been sold or
         discounted by the Borrower by means of a securitization for purposes of
         securing the obligations incurred


<PAGE>   2


         by the Borrower in connection with such sale provided that (A) the
         outstanding amount of accounts receivable so sold or discounted by the
         Borrower in the aggregate at any time shall not exceed 50% of the face
         amount of all such receivables, (B) the accounts receivable so sold or
         discounted are substantially similar in credit quality to the accounts
         receivable retained by the Borrower and (C) the proceeds of such sales
         shall be used to prepay the Obligations and permanently reduce the
         Committed Amounts; and

         B. The Borrower hereby represents and warrants that:

                  (i) any and all representations and warranties made by the
         Borrower and contained in the Loan Agreement (other than those which
         expressly relate to a prior period) are true and correct in all
         material respects as of the date of this Fifth Amendment; and

                  (ii) No Default or Potential Default currently exists and is
         continuing under the Loan Agreement simultaneously with the execution
         of this Fifth Amendment.

         C.       The Borrower will execute such additional documents as are
reasonably requested by the Agent to reflect the terms and conditions of this
Fifth Amendment.

         D.       Except as modified hereby and except for necessary
modifications to exhibits to bring such exhibits in conformity with the terms of
this Fifth Amendment, all of the terms and provisions of the Loan Agreement (and
Exhibits) remain in full force and effect.

         E.       The Borrower agrees to pay all reasonable costs and expenses
in connection with the preparation, execution and delivery of this Fifth
Amendment, including without limitation the reasonable fees and expenses of the
Agent's legal counsel.

         F.       This Fifth Amendment may be executed in any number of
counterparts, each of which when so executed and delivered shall be deemed an
original and it shall not be necessary in making proof of this Fifth Amendment
to produce or account for more than one such counterpart.

         G.       This Fifth Amendment and the Loan Agreement, as amended
hereby, shall be deemed to be contracts made under, and for all purposes shall
be construed in accordance with the laws of the State of Tennessee.

                  [Remainder of Page Intentionally Left Blank]


                                       2
<PAGE>   3


         IN WITNESS WHEREOF, each of the parties hereto has caused a counterpart
of this Fifth Amendment to be duly executed under seal and delivered as of the
date and year first above written.


BORROWER:

                                  GENESCO INC.
                                  a Tennessee corporation



                                  By   /s/ James S. Gulmi
                                      ------------------------------------------

                                  Title  Senior Vice President - Finance
                                         ---------------------------------------

BANKS:

                                  BANK OF AMERICA, N.A.
                                  individually in its capacity as a
                                  Bank and in its capacity as Agent


                                  By  /s/  Timothy H. Spanos
                                      ------------------------------------------

                                  Title  Managing Director
                                         ---------------------------------------


                                  BANK ONE, NA (Main Office -
                                  Chicago, formerly known as
                                  The First National Bank of
                                  Chicago),
                                  individually in its capacity as a Bank and
                                  in its capacity as a Co-Agent


                                  By  /s/ Catherine A. Muszynski
                                     ------------------------------------------


                                  Title  Vice President
                                         ---------------------------------------


                                       3