<PAGE>

                                                                    EXHIBIT 10.4

                                 AMENDMENT NO. 3

                                       TO

                                 LOAN AGREEMENT

         AMENDMENT NO. 3 dated and effective as of June 27, 2002, among NICHOLAS
FINANCIAL, INC. ("Borrower"), the financial institutions listed on the signature
pages hereof (the "Lenders") and BANK OF AMERICA, N.A., as agent for the Lenders
(the "Agent").

         WHEREAS, the Borrower, the Agent and the Lenders are parties to a
certain Amended and Restated Loan and Security Agreement, dated as of August 1,
2000 (the "Loan Agreement"), pursuant to which the Lenders have agreed, subject
to the terms and conditions therein set forth, to provide certain financial
accommodations to the Borrower; and

         WHEREAS, at the request of the Borrower, First Tennessee Bank, N.A.
("FTB") has agreed to become a party to the Loan Agreement and a Lender
thereunder, with a Commitment of $15,000,000 and to purchase an assignment of
the interest of Bank One, NA pursuant to an Assignment and Acceptance Agreement
executed simultaneously herewith (the "Assignment");

         WHEREAS, the Borrower desires that the Lenders amend certain provisions
of the Loan Agreement, and the Lenders are willing, subject to the terms and
conditions hereinafter set forth, to do so;

         NOW, THEREFORE, the parties hereto hereby agree as follows:

         SECTION 1. CAPITALIZED TERMS. Capitalized terms used but not defined
herein shall have the respective meanings set forth in the Loan Agreement.

         SECTION 2. AMENDMENTS

                           (a)      Each of the definitions of "Applicable
Margin", "Majority Lenders," "Required Lenders," and "Stated Termination Date"
set forth in the Loan Agreement is hereby amended in its entirety to read as
follows:

                                    " `Applicable Margin' means (i) with respect
                  to Reference Rate Revolving Loans, one quarter of one percent
                  (1/4%) and (ii) with respect to LIBOR Rate Revolving Loans,
                  two percent (2%).

                                    " `Majority Lenders' means at any date of
                  determination (a) Lenders whose Pro Rata Shares aggregate more
                  than 66% as such percentage is determined under the definition
                  of Pro Rata Share set forth herein; or (b) in the event there
                  are only two (2) Lenders under this Agreement, both Lenders or
                  (c) to the extent Bank's Pro Rata Share exceeds 50%, the
                  Lenders other than Bank acting unanimously shall constitute
                  the Majority Lenders for purpose of Section 14.5."

                                    " ` Required Lenders' means at any time (a)
                  Lenders whose

<PAGE>

                  Pro Rata Shares aggregate more than fifty percent (50%) as
                  such percentage is determined under the definition of Pro Rata
                  Share set forth herein, or (b) in the event there are only two
                  (2) Lenders under this Agreement, either Lender or (c) in the
                  event Bank's Pro Rata Share exceeds 50%, any two Lenders."

                                    " `Stated Termination Date' means November
                  30, 2004."

                           (b)      Section 2.2(i)(i) of the Loan Agreement is
hereby amended in its entirety to read as follows:

                                    " `(i) Agent Advances.' (i) Subject to the
                  limitations set forth in the provisos contained in this
                  Section 2.2(i)(i), the Agent is hereby authorized by the
                  Borrower and the Lenders, from time to time in the Agent's
                  sole discretion, (A) after the occurrence of Default or an
                  Event of Default, or (B) at any time that any of the other
                  applicable conditions precedent set forth in Article 10 have
                  not been satisfied, to make Base Rate Revolving Loans to the
                  Borrower on behalf of the Lenders which the Agent, in its
                  reasonable business judgment, deems necessary or desirable,
                  (1) to preserve or protect the Collateral, or any portion
                  thereof, (2) to enhance the likelihood of, or maximize the
                  amount of, repayment of the Loans and other Obligations, or
                  (3) to pay any other amount chargeable to the Borrower
                  pursuant to the terms of this Agreement, including costs, fees
                  and expenses as described in Section 15.7 (any of the advances
                  described in this Section 2.2(i) being hereinafter referred to
                  as `Agent Advances'); provided, that the Required Lenders may
                  at any time revoke the Agent's authorization contained in this
                  Section 2.2(i) to make Agent Advances, any such revocation to
                  be in writing and to became effective prospectively upon the
                  Agent's receipt thereof; provided further that (i) if the Pro
                  Rata Share of the Required Lenders revoking such authorization
                  does not exceed 50%, such revocation shall become effective
                  120 days after Agent's receipt thereof or (ii) if the Default
                  or Event of Default would require consent of all Lenders to
                  waive or amend, such authorization may be revoked by any
                  Lender effective 120 days after Agent's receipt thereof."

                           (c)      Section 3.8 of the Loan Agreement is hereby
amended by deleting the amount of "$20,000" set forth in the first line thereof
and substituting therefor the amount of "$30,000."

                           (d)      Section 4.2 of the Loan Agreement is hereby
amended by:

                                    (i)      deleting the clause "After November
                  30, 2000 but on or prior to November 20, 2001" and
                  substituting therefore the clause "On or prior to November 30,
                  2003"; and

                                    (ii)     deleting the clause "After November
                  30, 2001 but prior to November 30, 2002" and substituting
                  therefor the clause "After November 30, 2003 but prior to
                  November 30, 2004"; and

                                    (iii)    deleting the date of "November 30,
                  2002" appearing in the second paragraph thereof and
                  substituting therefor the date of

                                     - 2 -

<PAGE>

                  "November 30, 2004."

                           (e)      Section 9.18 of the Loan Agreement is hereby
amended by deleting the ratio "1.25.1" set forth therein and substituting
therefor the ratio "1.50 to 1.0."

                           (f)      Section 9.22 of the Loan Agreement is hereby
amended to read in its entirety as follows:

                                    " `9.22 Collateral Adjustment Percent.' The
                  Borrower shall not permit the Collateral Adjustment Percent as
                  of the last day of any month to be greater than sixteen
                  percent (16%)."

                           (g)      Section 11.2(a) of the Loan Agreement is
hereby amended in its entirety to read as follows:

                                    " `Remedies.' (a) If a Default or an Event
                  of Default exists, the Agent may, in its discretion, and
                  shall, at the direction of the Majority Lenders, do one or
                  more of the following at any time or times in any order,
                  without notice to or demand on the Borrower: (i) reduce the
                  Maximum Revolver Amount, or the advance rates against Eligible
                  Contracts used in computing the Borrowing Base, or reduce one
                  or more of the other elements used in computing the Borrowing
                  Base; and (ii) restrict the amount or refuse to make Revolving
                  Loans, If an Event of Default exists, the Agent shall, at the
                  direction of any Lender, declare a Default or Event of Default
                  and give written notice thereof to the Borrower and at the
                  direction of the Majority Lenders, do one or more of the
                  following, in addition to the actions described in the
                  preceding sentence, at any time or times and in any order,
                  without notice to or demand on the Borrower: (A) terminate the
                  Commitments and this Agreement; (B) declare any or all
                  Obligations to be immediately due and payable; provided,
                  however, that upon the occurrence of any Event of Default
                  described in Section 11.1(e), 11.1(f), 11.1(g) or 11.1(h), the
                  Commitments shall automatically and immediately expire and all
                  Obligations shall automatically become immediately due and
                  payable without notice or demand of any kind; (C) pursue its
                  other rights and remedies under the Loan Documents and
                  applicable law and (D) take such action as is required under
                  Section 14.5 hereof."

                           (h)      Clause (iii) of Section 11.2(b) of the Loan
Agreement is hereby amended to read in its entirety as follows:

                                    "(iii) the Agent may sell and deliver any
                  Collateral at public or private sales, for cash, upon credit
                  or otherwise, at such prices and upon such terms as the
                  Majority Lenders deem advisable, in their sole discretion, and
                  may, if the Agent deems it reasonable, postpone or adjourn any
                  sale of the Collateral by an announcement at the time and
                  place of sale or of such postponed or adjourned sale without
                  giving a new notice of sale."

                           (i)      Section 14.5 of the Loan Agreement is hereby
amended in its entirety to read as follows:

                                     - 3 -

<PAGE>

                                    "14.5 Notice of Default. The Agent shall not
                  be deemed to have knowledge or notice of the occurrence of any
                  Default or Event of Default, unless the Agent shall have
                  received written notice from a Lender or the Borrower
                  referring to his Agreement, describing such Default or Event
                  of Default and stating that such notice is a `notice of
                  default.' The Agent will notify the Lenders of its receipt of
                  any such notice. Upon the written request of any Lender, agent
                  shall declare a Default under this Agreement and send notice
                  (`Default Notice') thereof to the Borrower within (10)
                  business days, with a copy provided to each of the Lenders
                  unless such Default is cured or waived. Otherwise, the Agent
                  shall take such action with respect to such Default or Event
                  of Default as may be requested by the Majority Lenders in
                  accordance with Section 11; provided, however, that unless and
                  until the Agent has received any such request, the Agent may
                  (but shall not be obligated to) take such action, or refrain
                  from taking such action, with respect to such Default or Event
                  of Default as it shall deem advisable. Agent and each of the
                  Lenders agree to use reasonable good faith efforts to disclose
                  to each other, as soon as practicable after discovery by a
                  senior officer with direct responsibility for the management
                  of the transactions with Borrower any information or
                  communication (believed to be reliable and substantially
                  accurate) which the disclosing Lender has reason to believe
                  (a) is not known by Agent or the other Lenders (as applicable)
                  and (b) may have a material and adverse effect upon the
                  business or operations of the Borrower and/or upon the
                  collateral security for the Loans, and as a result, may impair
                  the repayment of the Loans as and when due; provided, however,
                  that neither the Agent nor the other Lenders shall have any
                  liability as a result of its or their failure to disclose any
                  information pursuant to this section, nor shall any Lender
                  assert any such failure by Agent or another Lender as a
                  defense to any claim asserted against a Lender under the
                  provisions of this Agreement."

         SECTION 3. EFFECTIVENESS. The amendment made herein shall become
effective when (i) Lenders shall have duly executed and delivered this Agreement
and counterparts hereof shall have been duly executed and delivered to the Agent
by the Borrower; (ii) FTB and Bank One, NA have executed and delivered to Agent
a duly completed Assignment and Acceptance Agreement that has been accepted by
the Agent; (iii) payment of the purchase price of the Assignment has been made
to Bank One, NA by wire transfer of same day available funds; and (iv) payment
by Borrower to Agent of a fee in the amount of $60,000 for the ratable benefit
of the Lenders (other than Bank One, NA).

         SECTION 4. COUNTERPARTS AND GOVERNING LAW. This Agreement may be
executed in counterparts, each of which shall be an original, and all of which,
taken together, shall constitute a single instrument. This Agreement shall be
governed by, and construed in accordance with the law of the State of New York.

         SECTION 5. REFERENCES TO LOAN AGREEMENT. From and after the
effectiveness of this Agreement and the waivers and agreements contemplated
hereby, all references in the Loan Agreement to "this Agreement", "hereof",
"herein", and similar terms shall mean and refer to the Loan Agreement as
certain provisions thereof are amended or supplemented by this Agreement, and
all references in other documents to the Loan Agreement shall mean such
agreement as certain provisions thereof are amended or

                                     - 4 -

<PAGE>

supplemented by this Agreement.

         SECTION 6. INVALIDITY. Whenever possible, each provision of this
Agreement shall be interpreted in such manner as to be effective and valid under
all applicable laws and regulations. If, however, any provision of this
Agreement shall be prohibited by or invalid under any such law or regulation, it
shall be deemed modified to conform to the minimum requirements of such law or
regulation, or if for any reason it is not deemed so modified, it shall be
ineffective and valid only to the extent of such prohibition or invalidity
without the remainder thereof or any of the remaining provisions of this
Agreement being prohibited or invalid.

         SECTION 7. RATIFICATION AND CONFIRMATION. The Loan Agreement is
hereby ratified and confirmed and, except as herein otherwise agreed, remains
unmodified and in full force and effect.

         IN WITNESS WHEREOF, the parties hereto have caused this Agreement to be
duly executed by their respective authorized officers as of the day and year
first above written.

                                          Borrower:

                                          NICHOLAS FINANCIAL, INC.

                                          By: /s/ Peter L. Vosotas
                                              ----------------------------------
                                          Title: CEO

                                          Agent:

                                          BANK OF AMERICA, N.A.,
                                              as Agent

                                          By: /s/ James C. Smith
                                             -----------------------------------
                                          Title: Vice President

                                          Lenders:

                                          BANK OF AMERICA, N.A.

                                          By: /s/ James C. Smith
                                              ----------------------------------
                                          Title: /s/ Vice President

                                     - 5 -

<PAGE>

                                          HIBERNIA NATIONAL BANK

                                          By: /s/ Anita G. Kennedy
                                              ----------------------------------
                                          Title: Asst. Vice President

                                          FIRST TENNESSEE BANK, N.A.

                                          By: /s/ Steven J. Hawkins
                                              ----------------------------------
                                          Title: Senior Vice President

                                     - 6 -