<PAGE>   1
                                                                    Exhibit 10.4

                 AGREEMENT TO SECURE CERTAIN CONTINGENT PAYMENTS


         THIS AGREEMENT TO SECURE CERTAIN CONTINGENT PAYMENTS between Hancock
Fabrics, Inc., a Delaware corporation (the "Company"), and James A. Austin (the
"Executive"), dated as of June 14, 2001.

                                R E C I T A L S :

         WHEREAS, Executive and the Company have entered into that certain
Severance Agreement dated this date (in its present form or as it may hereafter
be amended, extended or renewed, the "Severance Agreement") and that certain
Agreement dated this date ("Deferred Compensation Agreement"); and

         WHEREAS, under such Agreements, certain payments are to be made at
times which are contingent upon a change in control of the Company, and in
certain instances also contingent upon a termination of Executive's employment,
all of which contingencies are set forth in such Agreements; and

         WHEREAS, the Company wishes to make additional provision for the
security of the payment of certain of the contingent amounts under such
Agreements in order to assure itself of continuity of management and to assure
the Executive of payment of the amounts in question,

         NOW, THEREFORE, in consideration of the foregoing and the mutual
covenants contained herein, it is hereby agreed by and between the Company and
the Executive as follows:

         1.       Letter of Credit Required.

                  The Company shall, no later than the date of a change in
control as defined in the Severance Agreement ("Change in Control"), provide
Executive with an irrevocable letter of credit ("Letter of Credit"), under which
Executive shall be the beneficiary, to pay the following amounts:



                                      -1-
<PAGE>   2

                  a.       Severance benefits to be paid by the Company as
provided in Section (5)(a)(I) of the Severance Agreement.

                  b.       "Penalty Taxes" (as defined in Section 10 of the
Severance Agreement) and all other amounts to be paid by the Company as provided
in Section 10 of the Severance Agreement.

                  c.       Payments to be made by the Company under Section 16
of the Deferred Compensation Agreement.

         2.       Amount of Letter of Credit.

                  a. The Letter of Credit shall be in an amount equal to the sum
determined by the Accounting Firm identified in Section 9 (the "Accounting
Firm"), as follows (both initially and recalculated as provided in Section 7)
plus ten percent (10%) of said sum.

                  b.       The Accounting Firm shall determine the sum of:

                           (i)      The amount which would be payable to the
Executive under Section 5(a)(I) of the Severance Agreement assuming a Change of
Control has occurred and that the Executive terminates employment for Good
Reason (as defined in the Severance Agreement) at the time the determination is
made;

                           (ii)     The amount of "Penalty Taxes" and income
taxes, if any, that the Accounting Firm has determined as the total amount
payable by the Company to the Executive under Section 10(a) of the Severance
Agreement; and

                           (iii)    The amount which would be payable to the
Executive under Section 16 of the Deferred Compensation Agreement if he had made
the election to receive benefits under that Section and was in all other



                                      -2-
<PAGE>   3

respects eligible to receive benefits thereunder at the time the determination
is made.

                  c.       The amount so determined shall be stated in a written
notice by the Accounting Firm to the Company; and such notice shall thereupon be
marked "Attachment A" and be attached to and become a part of this Agreement.


                                      -3-
<PAGE>   4

         3.       Form of Letter of Credit.

                  The Letter of Credit shall be in a form agreed to by the
Company and the issuing bank (the "Bank") consistent with the terms of this
Agreement. The Executive shall be entitled to draw on the Letter of Credit by
presenting to the Bank a draft and a certificate in which the Executive
certifies in writing that the requisite events have occurred under the Severance
Agreement and/or the Deferred Compensation Agreement for payment to the
Executive of amounts specified in Section 1.

         4.       Issuer.

                  The Letter of Credit shall be issued by a commercial bank that
is a state or national banking association and which has a stockholders' equity
in excess of $1 billion.

         5.       Single Letter of Credit with Multiple Beneficiaries.

                  At the Company's discretion, the Letter of Credit may provide
for payment of similar compensation and benefits to other executives of the
Company who are parties to agreements similar to this Agreement as long as the
total amount of the Letter of Credit, for all such other executives and for the
Executive, is, at all times, no less than the sum of the required amounts of the
Letter of Credit and of the letters of credit required under the agreements with
such other executives (initially or after recalculation pursuant to Section 7).

         6.       Term of Letter of Credit.

                  The Letter of Credit to be provided as required by Section 1
shall be issued for a term which shall be the maximum term then available for
commercial letters of credit. Until such time as all amounts described in
Sections 1(a), 1(b) and 1(c) have been paid in full, the Company shall, not


                                      -4-
<PAGE>   5

later than thirty (30) days before the Letter of Credit would otherwise lapse or
expire, renew the Letter of Credit or establish a replacement letter of credit
with terms at least as favorable as the initial Letter of Credit. The term
"Letter of Credit" shall mean both the original letter of credit and the renewal
and replacement letters of credit referred to in this Section 6. The period of
time during which a Letter of Credit is required to remain in effect under this
Agreement is herein called the "Secured Period." If the Company does not
establish or renew a Letter of Credit as required by Section 1 and this Section
6, the payment of all amounts described in Section 1 shall be accelerated in
accordance with Section 8 hereof.

         7.       Calculation and Recalculation of Amounts Secured by Letter of
                  Credit and Increase in Amount of Letter of Credit.

                  During the Secured Period, the Accounting Firm shall, no later
than the end of each calendar quarter in the Secured Period, recalculate the
amounts listed on Attachment A and give prompt written notice of such
recalculations to the Executive, the Company and the Bank. Any such recalculated
amounts shall thereafter be deemed to be the amounts listed on Attachment A. If
such recalculated amounts plus ten percent (10%) thereof exceed the amounts
payable under the Letter of Credit then in effect, no later than seven (7) days
after receipt of such written notice of recalculation, the Company shall cause
the amounts payable under the Letter of Credit to be increased to the
recalculated amounts plus 10%.

         8.       Acceleration Under Letter of Credit.

                  If at least 30 days prior to the expiration of a Letter of
Credit established hereunder, which expiration would occur before the end of the
Secured Period, the Executive shall not have received written notice from the
Company that an extension or renewal of the expiring Letter of Credit has


                                      -5-
<PAGE>   6

occurred or a new Letter of Credit has been obtained with the terms and
conditions at least as favorable as the predecessor Letter of Credit, then,
notwithstanding any other provisions of the Severance Agreement, the Deferred
Compensation Agreement, or this Agreement, all amounts which are described in
Section 1 shall be due and payable immediately without regard to any
contingencies or future events and Executive, his beneficiaries or estate shall
be entitled to immediately draw on the Letter of Credit for all such amounts.

         9.       Accounting Firm.

                  The Accounting Firm shall mean the same Accounting Firm as
determined under the provisions of Section 10(h) of the Severance Agreement. All
calculations by the Accounting Firm shall be binding on the Company, the
Executive and the Bank in the absence of wilful misconduct or gross negligence
by the Accounting Firm.

         10.      Expenses.

                  The Company shall pay all expenses and fees of the Accounting
Firm.

         11.      Information.

                  The Company shall provide the Accounting Firm with such
information as the Company has in its possession that the Accounting Firm
believes necessary, in its discretion, to make its calculations and
recalculations under this Agreement.

         12.      Obligation of Executive.

                  Subject to the provisions of Section 8, if Executive receives
payments under the Letter of Credit for any amount in excess of that to which he
is entitled under the Severance Agreement, the Deferred Compensation


                                      -6-
<PAGE>   7

Agreement, or this Agreement, Executive shall immediately repay such amount to
Company.

         13.      Legal Fees, Etc.

                  If Executive incurs any legal fees and expenses as a result of
seeking to obtain or enforce any right or benefit provided by this Agreement,
Company shall pay or reimburse Executive for all such reasonable fees and
expenses.


                                      -7-
<PAGE>   8

         14.      No Waivers.

                  The payment by a Bank on a Letter of Credit established in
accordance with the terms hereof shall not constitute a waiver by the Company
of, or in any way preclude the Company from asserting, any claim the Company may
have against Executive that Executive is not entitled to some or all of such
payment. Drawing upon the Letter of Credit shall not constitute a waiver by
Executive of, or in any way preclude Executive from asserting, any claim
Executive may have against the Company that Executive is entitled to amounts
under this Agreement or the Severance Agreement or the Deferred Compensation
Agreement (except to the extent that such amounts have been paid in full by the
Bank under the Letter of Credit).

         15.      Enforceability.

                  Subject to Section 19, this Agreement shall inure to the
benefit of and be enforceable by Executive's personal or legal representatives,
executors, administrators, successors, heirs, distributees, devisees and
legatees.

         16.      No Change to Agreements.

                  Except as expressly modified by this Agreement, Executive's
rights under the Severance Agreement and the Deferred Compensation Agreement are
unchanged by this Agreement.

         17.      Modification in Writing.

                  No provision of this Agreement may be modified or waived
unless in writing and signed by Executive and such officer of Company as may be
designated or authorized by its Board of Directors.


                                      -8-
<PAGE>   9

         18.      Severability.

                  The invalidity or unenforceability of any provision of this
Agreement shall not affect the validity or enforceability of any other provision
of this Agreement, which shall remain in full force and effect.

         19.      Assignment.

                  Executive's rights under this Agreement and the Letter of
Credit are not assignable.

         20.      Notices.

                  Any notice, report, demand or waiver required or permitted
hereunder shall be in writing and shall be given personally or by prepaid
registered or certified mail, return receipt requested, addressed as follows:

         If to the Company:                 Hancock Fabrics, Inc.
                                            3406 West Main Street
                                            P.O. Box 2400
                                            Tupelo, Mississippi 38003-2400

                                            Attention: Corporate Secretary

         If to the Executive:               James A. Austin
                                            5145 Lackey Lane
                                            Tupelo, Mississippi 38801

A notice shall be effective upon the receipt thereof. The above addresses can be
changed by notice in writing delivered as provided above.

         21.      Additional Benefits.

                  The security provided under this Agreement is in addition to
and not by way of limitation of any rights or benefits to which Executive is
entitled under the Severance Agreement and the Deferred Compensation Agreement.

         IN WITNESS WHEREOF Executive and, pursuant to authorization from its
Board of Directors, Company have executed this Agreement to Secure Certain
Contingent Payments effective as of the date first above written.



                                    --------------------------------------------
                                                                      Executive


                                    HANCOCK FABRICS, INC.

                                    By
                                      ------------------------------------------

                                    Its
                                       -----------------------------------------


                                      -9-