<PAGE>   1

                                                                 EXHIBIT 4(m) TO
                                                 1997 ANNUAL REPORT ON FORM 10-K



                              MODIFICATION TO NOTE

         This Modification to Note, made as of the 12th day of September 1997,
by and between ELLETT BROTHERS, INC., a South Carolina Corporation (the
"Company"), and THE BANK OF NEW YORK (the "Trustee") as Trustee.

                              W I T N E S S E T H:

         WHEREAS, Lexington County, South Carolina (the "County") issued its
Industrial Revenue Refunding Bonds, Series 1988 (Ellett Brothers Limited
Partnership Project) in the original principal amount of $9,100,000 (the
"Bonds"); and

         WHEREAS, the County loaned the proceeds of the Bonds to Ellett Brothers
Limited Partnership, a North Carolina limited partnership (the "Predecessor
Company") pursuant to a Loan Agreement, dated as of November 1, 1988 (the "Loan
Agreement"), between the County and the Predecessor Company for the purpose of
refinancing certain real and personal property in Lexington County, South
Carolina (the "Project"); and

         WHEREAS, the Predecessor Company evidenced its obligations under the
Loan Agreement by that certain promissory note, dated December 1, 1988 (the
"Note") in the original principal amount of $9,100,000 to the County which the
County assigned to the original trustee by assignment dated December 1, 1988
(the "Assignment"); and

         WHEREAS, the County secured its obligations under the Bonds by
assigning all of its right, title, and interest (except for certain rights to
receive indemnification and the payment of expenses) under the Loan Agreement to
the predecessor to the Trustee pursuant to the terms of the Indenture; and

         WHEREAS, the Predecessor Company secured its obligations under the Note
and the Loan Agreement, as assigned by the County to the Trustee, by giving the
original trustee a Mortgage and Security Agreement, dated as of November 1,
1988, and recorded in Book 1235 at page 192 in the Office of the RMC for
Lexington County, South Carolina (the "Mortgage"), which mortgage constitutes a
first perfected lien and security interest upon the Project; and

         WHEREAS, pursuant to the terms of an Assignment and Assumption
Agreement, dated as of June 9, 1993, the Company has succeeded to all rights and
obligations of the Predecessor Company with respect to the Loan Agreement, the
Note, and the Mortgage; and

         WHEREAS, the Trustee is successor trustee in compliance with the terms
of the Indenture to Citizens and Southern Trust Company (South Carolina),
National Association; and

         WHEREAS, The Depository Trust Company, under its nominee name Cede &
Co., is the


<PAGE>   2

registered owners of all of the outstanding Bonds on behalf of its beneficial
owner The Bank of New York, as custodian (the "Custodian") on behalf of
Municipal Income Opportunities Trust and Municipal Income Trust II, funds
managed by Dean Witter InterCapital, Inc. (collectively, the "Bondholder"); and

         WHEREAS, the Bonds bear interest at a rate of 10.625% and the Company
has requested and the Bondholder has agreed, to reduce the interest rate on the
Bonds to 7.5% and to make certain other modifications to the Bonds; and

         WHEREAS, by ordinance enacted on September 8, 1997, the County Council
of Lexington County, South Carolina authorized the amendment of the Bonds, the
Indenture, and the Loan Agreement to provide for a reduction of the interest
rate and certain other changes as aforesaid and the issuance of Amended Bonds
(the "Amended Bonds") to replace the Bonds; and

         WHEREAS, the Loan Agreement is being amended pursuant to a First
Amendatory Loan Agreement, of even date herewith, executed by and among the
Company, the County, the Trustee, and the Bondholder (the Loan Agreement as
amended by the First Amendatory Loan Agreement is hereinafter referred to as the
"Amended Loan Agreement"); and

         WHEREAS, the Indenture is being supplemented pursuant to a First
Supplemental Trust Indenture, of even date herewith, executed by and among the
Company, the County, the Trustee, and the Bondholder (the Indenture as
supplemented by the First Supplemental Trust Indenture is hereinafter referred
to as the "Supplemented Indenture"); and

         WHEREAS, the Note is being amended pursuant to this Modification (the
Note as amended by this Modification is hereinafter referred to as the "Amended
Note").

         NOW, THEREFORE, the Company and the Trustee, in consideration of the
premises and the sum of one dollar to each in hand paid by the other, a receipt
of which is hereby acknowledged by each, do hereby agree that the Note and the
Assignment should be, the same hereby are, modified and amended as follows:

         1. The recitals are incorporated herein by reference and shall be
deemed to be a part of this Modification.

         2. All references in the Note to the Series 1988 Bonds shall
hereinafter be deemed to refer to the Amended Bonds.

         3. All references in the Note to the Company shall hereinafter mean
Ellett Brothers, Inc., a South Carolina corporation. All references in the Note
to the Indenture shall hereinafter mean the Indenture as supplemented by the
First Supplemental Trust Indenture of even date herewith. All references in the
Note to the Loan Agreement shall hereinafter mean the Loan Agreement as amended
by the First Amendatory Loan Agreement of even date herewith. All references to
the Company shall

                                       2
<PAGE>   3


mean Ellett Brothers, Inc., a South Carolina corporation. All references to the
Trustee shall hereinafter refer to The Bank of New York, its successors and
assigns.

         4. Except as modified as provided in paragraphs 2 and 3 above, all
other terms and provisions of the Note shall remain in full force and effect.

         IT IS MUTUALLY AGREED by and between the parties hereto that this
Modification shall become a part of the Note and the Assignment by reference and
that nothing herein shall waive, annul, vary, or affect any provision,
condition, covenant, or agreement contained in the Note and the Assignment
except as herein amended, nor affect or impair any rights, powers, or remedies
under the Note and the Assignment, as hereby amended.

         The Company promises and agrees to perform all of the requirements,
conditions, and obligations under the terms of the Note and the Assignment as
hereby modified and amended, said documents being hereby ratified and affirmed.

         This Modification shall be binding upon and inure to the benefit of any
assignee or the respective successors and assigns to the parties hereto.

         This Modification may be executed in any number of counterparts, each
of which shall be an original but all of which taken together shall constitute
one and the same instrument and any of the parties hereto may execute any of
such counterparts.

         This Modification shall be construed, interpreted, enforced, and
governed by and in accordance with the laws of the State of South Carolina.

         IN WITNESS WHEREOF, this instrument has been executed under seal by the
parties hereto and delivered as of the date and year first above written.

                                            ELLETT BROTHERS, INC.


                                            By: /s/ E. Wayne Gibson
                                                -------------------------
                                            Its: Corporate Secretary


                                            THE BANK OF NEW YORK, as Trustee


                                            By: /s/ Watson T. Barger
                                                -------------------------
                                            Its: Agent



                                       3