<PAGE>

                                                                   EXHIBIT 10.5

                        THIRD AMENDMENT TO FIFTH AMENDED
                           AND RESTATED LOAN AGREEMENT

         This Third Amendment to Fifth Amended and Restated Loan Agreement,
dated as of June 26, 2003, by and among The J. Jill Group, Inc., a Delaware
corporation ("BORROWER") on the one hand, and Citizens Bank of Massachusetts,
HSBC Bank USA, and Banknorth, N.A. (collectively, "LENDERS") and Citizens Bank
of Massachusetts as agent ("AGENT") for the LENDERS, on the other hand.

                                   WITNESSETH:

         WHEREAS, BORROWER, LENDERS and AGENT are parties to that certain Fifth
Amended and Restated Loan Agreement dated as of June 29, 2001, as amended by
First Amendment thereto dated as of August 28, 2001, and by Second Amendment
thereto dated as of July 25, 2002 (collectively, the "LOAN AGREEMENT"); and

         WHEREAS, BORROWER, LENDERS and AGENT wish to amend the LOAN AGREEMENT
as more particularly hereafter set forth. Capitalized terms used herein without
definition shall have the meanings ascribed to them in the LOAN AGREEMENT.

         NOW, THEREFORE, in consideration of the covenants and agreements herein
contained, the parties hereby agree that the LOAN AGREEMENT is hereby amended as
follows:

         1.  Section 1.01 of the LOAN AGREEMENT is hereby amended by deleting
             the definition of "AGGREGATE COMMITMENT AMOUNT" appearing therein
             and substituting therefor the following:

             "AGGREGATE COMMITMENT AMOUNT shall mean the sum of SIXTY MILLION
             DOLLARS ($60,000,000)."

         2.  Section 1.01 of the LOAN AGREEMENT is hereby amended by adding a
             new definition to said Section as follows:

             "SEC" shall mean the United States Securities and Exchange
             Commission, an agency of the United States government.

         3.  Section 1.01 of the LOAN AGREEMENT is hereby amended by deleting
             the definition of "TERMINATION DATE" appearing therein and
             substituting therefor the following:

             "TERMINATION DATE shall mean June 1, 2005."

         4.  Section 2.01 of the LOAN AGREEMENT is hereby deleted in its
             entirety and the following is substituted therefor:

<PAGE>

             "2.01 Subject to, and upon the terms and conditions herein
             provided, during the AVAILABILITY PERIOD, each LENDER severally to
             the extent of its COMMITMENT and COMMITMENT PERCENTAGE, agrees to
             make ADVANCES to the BORROWER, so long as (a) after giving effect
             to BORROWER'S request for each such ADVANCE, the REVOLVING LOAN
             BALANCE does not exceed Twenty Million Dollars ($20,000,000); (b)
             the CREDIT BALANCE does not exceed the AGGREGATE COMMITMENT AMOUNT;
             and (c) at the time of such ADVANCE the conditions specified in
             Section 2.15 have been and remain fulfilled."

         5.  Section 5.01 of the LOAN AGREEMENT is hereby amended by deleting
             said Section in its entirety and substituting therefor the
             following:

             "5.01 The respective amount of each LENDER'S COMMITMENT AMOUNT and
             its respective COMMITMENT PERCENTAGE with respect to the LOANS
             shall be:

<TABLE>
<CAPTION>
             -------------------------------------------------------------------
                       BANK             COMMITMENT             COMMITMENT
                                          AMOUNT               PERCENTAGE
             -------------------------------------------------------------------
             <S>                        <C>                    <C>
                  Citizens Bank of      $27,000,000               45%
                  Massachusetts
             -------------------------------------------------------------------
                  HSBC Bank USA         $21,000,000               35%
             -------------------------------------------------------------------
                  Banknorth, N.A.       $12,000,000               20%"
             -------------------------------------------------------------------
</TABLE>

         6.  Subsection 8.01(a) of the LOAN AGREEMENT is hereby deleted in its
             entirety and the following is substituted therefor:

             "8.01(a) As soon as practicable and in any event not later than the
             earlier of (i) ninety (90) days after the end of each fiscal year,
             or (ii) the date on which consolidated statements are filed with
             the SEC, consolidated statements, on an unqualified audit basis, of
             income, retained earnings and cash flow of the BORROWER and its
             SUBSIDIARIES for such year, and audited balance sheets of the
             BORROWER and its SUBSIDIARIES as at the end of such year, setting
             forth in each case in comparative form corresponding figures for
             the preceding fiscal year from the preceding annual audit, all in
             reasonable detail and reasonably satisfactory in scope to the
             MAJORITY LENDERS and certified by the ACCOUNTANTS whose certificate
             shall be on an unqualified, audited basis representing an
             unqualified opinion, all in scope and substance satisfactory to the
             MAJORITY LENDERS, and such financial statements shall be prepared
             in accordance with GAAP."

                                       2
<PAGE>

         7.  Subsection 8.01(b) of the LOAN AGREEMENT is hereby deleted in its
             entirety and the following is substituted therefor:

             "8.01(b) As soon as practicable and in any event not later than the
             earlier of (i) forty-five (45) days after the end of each of the
             first three quarterly periods in each FISCAL YEAR, or (ii) the date
             on which consolidated statements are filed with the SEC,
             consolidated statements of income and cash flow of the BORROWER and
             its SUBSIDIARIES for the period from the beginning of the current
             FISCAL YEAR to the end of such quarterly period, and a balance
             sheet of the BORROWER and its SUBSIDIARIES as at the end of such
             quarterly period, setting forth in each case, in comparative form,
             figures for the corresponding period in the preceding FISCAL YEAR,
             all in reasonable detail, and such financial statements shall be
             prepared in accordance with GAAP subject to customary year end
             adjustments and the absence of footnotes. Such quarterly statements
             shall correspond to the information contained in BORROWER'S 10Q
             financial statements as filed with the SEC."

         8.  Subsection 8.01(c) of the LOAN AGREEMENT is hereby amended by
             deleting the words "and consolidating" appearing in the third line
             of said subsection; and Subsection 8.01(d) of the LOAN AGREEMENT is
             hereby amended by deleting said subsection in its entirety and
             substituting therefor the following:

             "8.01(d) Not later than January 31 of each FISCAL YEAR, annual
             projections for such current FISCAL YEAR of BORROWER and its
             SUBSIDIARIES, showing projected monthly balance sheets, income
             statements, and cash flows for such year, all in such form as the
             AGENT shall reasonably require."

             Subsection 8.01(e) of the LOAN AGREEMENT is hereby amended by
             deleting said subsection in its entirety and substituting therefor
             the following:

             "8.01(e) At the time of delivery of the reports required by Section
             8.01(a) and 8.01 (b), a certificate of the Chief Financial Officer,
             Vice President of Finance or Controller of the BORROWER, and within
             five (5) business days of the time of delivery of the reports
             required by Section 8.01(a), a certificate from the ACCOUNTANTS (i)
             stating that, in his, her or their opinion, as the case may be, if
             such be the case, there has been and is existing no DEFAULT or
             EVENT OF DEFAULT hereunder, or if that not be the case, setting
             forth the details of all such DEFAULT or EVENT OF DEFAULT, and (ii)
             in such BORROWER'S officer's certificate showing appropriate
             calculations indicating compliance (or non-compliance) with the
             covenants set forth in Article IX."

         9.  Subsection 8.01(f) of the LOAN AGREEMENT is hereby amended by
             deleting said subsection in its entirety and substituting therefor
             the following:

                                       3
<PAGE>

             "8.01(f) If requested by any of the LENDERS, copies of all notices
             to shareholders, all proxies, reports and any other publicly
             available materials filed with the SEC and all press releases."

         10. Section 9.04 of the LOAN AGREEMENT is hereby deleted in its
             entirety and the following is substituted therefor:

             "9.04 The BORROWER shall not, and shall not permit any SPECIAL
             SUBSIDIARY to, merge or consolidate or be merged or consolidated
             with or into any other corporation or entity (except that nothing
             herein shall prevent any SPECIAL SUBSIDIARY from merging or
             consolidating with any other SPECIAL SUBSIDIARY, or any SPECIAL
             SUBSIDIARY(IES) merging or consolidating with the BORROWER) nor
             shall BORROWER create any SUBSIDIARY which is not in existence on
             the date of this AGREEMENT without obtaining the MAJORITY LENDERS'
             prior written consent."

         11. Section 9.17 of the LOAN AGREEMENT is hereby amended by deleting
             said section in its entirety and substituting therefor the
             following:

             "9.17 BORROWER's TANGIBLE NET WORTH shall not be less than One
             Hundred Nineteen Million Four Hundred Fifty Five Thousand Dollars
             ($119,455,000) from the date of execution hereof through December
             26, 2003; and at December 27, 2003, BORROWER'S TANGIBLE NET WORTH
             shall not be less than the sum of (A) One Hundred Nineteen Million
             Four Hundred Fifty Five Thousand Dollars ($119,455,000) and (B)
             fifty percent (50%) of BORROWER'S net income (but not less than
             zero) for the year then ended (the "TNW MINIMUM BENCHMARK").
             Thereafter, at each year end of BORROWER in each succeeding year,
             BORROWER's TANGIBLE NET WORTH shall not be less than the sum of (A)
             the TNW MINIMUM BENCHMARK for the immediately preceding year and
             (B) fifty percent (50%) of BORROWER'S net income (but not less than
             zero) for the year then ended."

         12. Subsection 12.02(a) of the LOAN AGREEMENT is hereby amended by
             deleting the subsection contained therein beginning with the words
             "Foley, Hoag" and ending with the words "Attn: David R. Pierson,
             Esq." and substituting therefor the following:

                    "Foley Hoag LLP
                    155 Seaport Boulevard
                    Boston, MA 02210
                    Fax: (617) 832-7000
                    Attn: David R. Pierson, Esq."

         13. Subsection 12.02(b) of the LOAN AGREEMENT is hereby amended by
             deleting the name "Christopher M.S. Harrocks, Vice President"
             appearing therein and substituting therefor the name "George
             Ahlmeyer, Senior Vice President".

                                       4
<PAGE>

         14. Subsection 12.02(b) of the LOAN AGREEMENT is hereby further amended
             by deleting the words "Bank of New Hampshire, N.A. c/o BankNorth
             Group, Inc." appearing in said subsection and substituting therefor
             the words "Banknorth, N.A."

         15. Subsection 12.02(b) of the LOAN AGREEMENT is hereby further amended
             by deleting the words ", Suite 700" appearing immediately after the
             words "7 New England Executive Park" appearing in said subsection.

         16. Subsection 12.02 (c) of the LOAN AGREEMENT is hereby amended by
             deleting the subsection contained therein beginning with the words
             "Schnader Harrison Goldstein & Manello" and ending with the name
             "Paul J. Levenson, Esq." and substituting therefor the following:

                    "Adler Pollock & Sheehan, P.C.
                    175 Federal Street
                    Boston, MA 02110
                    Fax: (617) 482-0604
                    Attn: Paul J. Levenson, Esq."

         17. Simultaneously herewith, BORROWER shall pay to AGENT for the
             benefit of the LENDERS on a pro rata basis a facility fee of
             Seventy-Five Thousand Dollars ($75,000) in connection with this
             Third Amendment to Fifth Amended and Restated Loan Agreement.

         18. Simultaneously herewith, BORROWER shall execute and deliver
             respectively to each of the LENDERS an amended and restated
             REVOLVING NOTE in the face amount as set forth in Section 5.01, in
             substitution for (and replacing and superseding) each REVOLVING
             NOTE previously given to each such LENDER, each in form and
             substance satisfactory to AGENT.

Except as hereby amended, the LOAN AGREEMENT is hereby ratified, confirmed and
republished.

                                       5
<PAGE>

         IN WITNESS WHEREOF, the parties hereto have set their hands and seals
as of the date first above written.

<TABLE>
<S>                                                    <C>
CITIZENS BANK OF MASSACHUSETTS                         THE J. JILL GROUP, INC.

By: /s/ Lori B. Leeth                                  By: /s/ Linda L. Trudel
    ---------------------------------------                ---------------------------
    Lori B. Leeth, Senior Vice President               Name: Linda L. Trudel
                                                           ---------------------------
                                                       Title: Vice President/Corporate Controller

HSBC BANK USA                                          BANKNORTH, N.A.

By: /s/ George Ahlmeyer                                By: /s/ Jon R. Sundstrom
    ----------------------------------------               ---------------------------
      George Ahlmeyer, Senior Vice President               Jon R. Sundstrom
                                                           Senior Vice President
</TABLE>




                                       6