<PAGE>

                                                                   EXHIBIT 10.05




           FIRST AMENDMENT TO DEBTOR IN POSSESSION LOAN AND SECURITY
           ---------------------------------------------------------
                                   AGREEMENT
                                   ---------

     This First Amendment to Debtor In Possession Loan and Security Agreement is
made as of this 18th day of July, 2001 by and among

     CASUAL MALE CORP. (the "Borrowers' Representative"), a Massachusetts
     corporation, having a principal place of business at 555 Turnpike Street,
     Canton, Massachusetts 02021, as agent for the Borrowers, being those
     Persons named on Exhibit 1 hereto, debtors and debtors in possession;

     The Borrowers;

     each of the Revolving Credit Lenders party to the Loan Agreement (defined
     below) (together with each of their successors and assigns, referred to
     individually as a "Revolving Credit Lender" and collectively as the
     "Revolving Credit Lenders"), and

     FLEET RETAIL FINANCE INC., as Administrative Agent and Collateral Agent, a
     Delaware corporation, having its principal place of business at 40 Broad
     Street, Boston, Massachusetts 02109; and

     BACK BAY CAPITAL FUNDING LLC, as Tranche B Lender and as Tranche C Lender,
in consideration of the mutual covenants herein contained and benefits to be
derived herefrom.

                              W I T N E S S E T H

     A.   Reference is made to the Debtor In Possession Loan and Security
Agreement (the "Loan Agreement") dated as of May 18, 2001 by and among the
Borrowers' Representative, the Borrowers, the Revolving Credit Lenders, the
Tranche B Lender, the Tranche C Lender, the Administrative Agent and the
Collateral Agent.

     B.   The parties to the Loan Agreement desire to modify and amend certain
provisions of the Loan Agreement, as provided herein.

     Accordingly, the parties hereto agree as follows:

     1.   Definitions.  Capitalized terms used herein and not otherwise defined
          -----------
herein shall have the meanings assigned to such terms in the Loan Agreement.

     2.   Amendment to Article 2.  The provisions of Section 2-16(a) of the Loan
          ----------------------
Agreement are hereby amended by deleting the words "eighteen months" appearing
in the second line thereof and inserting the words "twelve months" in their
stead.
<PAGE>

     3.   Amendment to Article 3.     The provisions of Section 3-3(b) of the
          ----------------------
Loan Agreement are hereby amended by deleting clause (ii) thereof in its
entirety.

     4.   Amendments to Exhibits to the Loan Agreement.  (a) The provisions of
          --------------------------------------------
Exhibit 2:2-12 are hereby amended by adding the following after the chart
appearing therein:

     The Base Margin and the Libor Margin based upon the Pricing Grid shall
     initially be determined on the Pricing Grid Change Date and shall be reset
     every three months thereafter.

     (b)  Exhibit 7:7-11 to the Loan Agreement is hereby deleted in its entirety
and a new Exhibit 7:7-11 in the form annexed to this First Amendment is
substituted in its stead.

     5.   Conditions Precedent to Effectiveness.   This First Amendment shall
          -------------------------------------
not be effective until each of the following conditions precedent have been
fulfilled to the satisfaction of the Administrative Agent:

     a.     This First Amendment shall have been duly executed and delivered by
            the respective parties hereto and, shall be in full force and effect
            and shall be in form and substance satisfactory to the
            Administrative Agent and the Lenders.

     b.     All action on the part of the Borrowers necessary for the valid
            execution, delivery and performance by the Borrowers of this First
            Amendment shall have been duly and effectively taken and evidence
            thereof satisfactory to the Administrative Agent shall have been
            provided to the Administrative Agent.

     c.     The Final Borrowing Order shall have been entered in the
            Proceedings, which Final Borrowing Order shall incorporate, among
            other things, approval of this First Amendment.

     d.     The Borrowers shall have provided such additional instruments and
            documents to the Administrative Agent as the Administrative Agent
            and Administrative Agent's counsel may have reasonably requested.

     6.   Miscellaneous.
          -------------

     a.     This First Amendment may be executed in several counterparts and by
            each party on a separate counterpart, each of which when so executed
            and delivered shall be an original, and all of which together shall
            constitute one instrument.

     b.     This First Amendment expresses the entire understanding of the
            parties with respect to the transactions contemplated hereby. No
            prior negotiations or discussions shall limit, modify, or otherwise
            affect the provisions hereof.
<PAGE>

     c.     Any determination that any provision of this First Amendment or any
            application hereof is invalid, illegal or unenforceable in any
            respect and in any instance shall not effect the validity, legality,
            or enforceability of such provision in any other instance, or the
            validity, legality or enforceability of any other provisions of this
            First Amendment.

     d.     The Borrowers shall pay on demand all costs and expenses of the
            Agents and the Lenders, including, without limitation, reasonable
            attorneys' fees in connection with the preparation, negotiation,
            execution and delivery of this First Amendment.

     e.     The Borrowers warrant and represent that the Borrowers have
            consulted with independent legal counsel of the Borrowers' selection
            in connection with this First Amendment and is not relying on any
            representations or warranties of the Agents, the Lenders or their
            counsel in entering into this First Amendment.
<PAGE>

     IN WITNESS WHEREOF, the parties have duly executed this First Amendment as
of the day and year first above written.


                                    CASUAL MALE CORP.



                                         by
                                           -------------------------
                                         Name: Elizabeth C. White
                                         Title:  Chief Financial Officer

                                    THE BORROWERS
                                         MORSE SHOE, INC.
                                         JBI, INC.
                                         JBI APPAREL, INC.
                                         THE CASUAL MALE, INC.
                                         WGS CORP.
                                         TCMB&T, INC
                                         LP INNOVATIONS, INC.
                                         SPENCER COMPANIES, INC.
                                         BUCKMIN, INC.
                                         ELM EQUIPMENT CORP.
                                         JBI HOLDING COMPANY, INC.
                                         TCM HOLDING COMPANY, INC.
                                         ISAB, INC.
                                         MORSE SHOE INTERNATIONAL,
                                         INC.
                                         WHITE CAP FOOTWEAR, INC


                                         by
                                           --------------------------
                                         Name:  Elizabeth C. White
                                         Title: Chief Financial Officer

                                    FLEET RETAIL FINANCE INC.,
                                         as Administrative Agent, as Collateral
                                         Agent, as Swingline Lender, and as
                                         Revolving Credit Lender


                                         By:___________________________

                                         Name:_________________________
<PAGE>

                                         Title:_______________________


                                    BACK BAY CAPITAL FUNDING, LLC
                                         as Tranche B Lender and Tranche C
                                         Lender


                                         By:________________________

                                         Name:______________________

                                         Title:_______________________

     The following Lenders consent to the amendments set forth herein with the
     exception of the amendment to Exhibit 7:7-11, as to which their consent is
     not required.

                                    FOOTHILL CAPITAL CORPORATION


                                    By.............................

                                    Name..........................

                                    Title..........................


                                    HELLER FINANCIAL, INC.

                                    By................................

                                    Name............................

                                    Title.............................


                                    LASALLE BUSINESS CREDIT, INC


                                    By..............................

                                    Name...........................

                                    Title...........................

                                    NATIONAL CITY COMMERCIAL
                                    FINANCE, INC.
<PAGE>

                                    By................................

                                    Name..............................

                                    Title.............................

                                    IBJ WHITEHALL BUSINESS CREDIT
                                    CORP.


                                    By................................

                                    Name..............................

                                    Title.............................


                                    THE PROVIDENT BANK


                                    By................................

                                    Name..............................

                                    Title.............................
<PAGE>

                       FINANCIAL PERFORMANCE CONVENANTS

     Minimum Cumulative EBITDAR.  The Borrowers shall not permit or suffer the
following, tested as of the last day of each month (commencing with June 30,
2001 and through December 31, 2001) on a cumulative basis dating back to
January 31, 2001, to be less than the following minimum cumulative amount: (a)
Consolidated EBITDAR minus (b) 100% of any adequate protection payments made by
the Borrowers during such period and not reflected in the Business Plan (without
regard to any updating of the Business Plan), to the extent that such adequate
protection payments were not otherwise included as reductions to EBITDAR for the
period of measurement:

--------------------------------------------------------------------------------
Month Ending:                                     Minimum Cumulative EBITDAR
                                                  $ Millions
--------------------------------------------------------------------------------
June, 2001*                                                                 6.0
--------------------------------------------------------------------------------
July, 2001*                                                                 5.3
--------------------------------------------------------------------------------
August, 2001*                                                               5.0
--------------------------------------------------------------------------------
September, 2001*                                                            7.7
--------------------------------------------------------------------------------
October, 2001                                                              10.3
--------------------------------------------------------------------------------
November, 2001                                                             14.2
--------------------------------------------------------------------------------
December, 2001                                                             27.1
--------------------------------------------------------------------------------

     * The Borrowers' compliance with the Minimum Cumulative EBITDAR test for
the subject months will be waived unless Excess Availability is less than
$17,500,000 for any three consecutive days during such period (the "Test
Condition").  If the Test Condition occurs, the Borrower shall be obligated to
comply with the Minimum Cumulative EBITDAR test for the subject month in which
the Test Condition occurs and shall also be required to be in compliance with
the Minimum Cumulative EBITDAR test for each month both preceding, and
subsequent to, the month in which the Test Condition occurs (failing which an
Event of Default shall be deemed to have arisen).  In any event, whether or not
the Test Condition has occurred, the Borrowers must comply with the Minimum
Cumulative EBITDAR test for October, 2001 and each month thereafter.

     Minimum Rolling Four Quarter EBITDAR. The Borrowers shall not permit or
suffer the following, tested as of the last day of each month (commencing with
January, 2002) on a trailing twelve month basis, to be less than the following
minimum cumulative amount: (a) Consolidated EBITDAR minus (b) 100% of any
adequate protection payments made by the Borrowers during such period and not
reflected in the

<PAGE>

Business Plan (without regard to any updating of the Business Plan), to the
extent that such adequate protection payments were not otherwise included as
reductions to EBITDAR for the period of measurement:

--------------------------------------------------------------------------------
Month Ending:                                 Minimum EBITDAR (Rolling 12 Month
                                              Basis):
                                              $ Millions
--------------------------------------------------------------------------------
January, 2002                                                               22.5
--------------------------------------------------------------------------------
February, 2002                                                              22.6
--------------------------------------------------------------------------------
March, 2002                                                                 21.8
--------------------------------------------------------------------------------
April, 2002                                                                 22.0
--------------------------------------------------------------------------------
May, 2002                                                                   23.1
--------------------------------------------------------------------------------
June, 2002                                                                  23.5
--------------------------------------------------------------------------------
July, 2002                                                                  25.7
--------------------------------------------------------------------------------
August, 2002                                                                25.8
--------------------------------------------------------------------------------
September, 2002                                                             25.8
--------------------------------------------------------------------------------
October, 2002                                                               25.9
--------------------------------------------------------------------------------
November, 2002                                                              26.0
--------------------------------------------------------------------------------
December, 2002                                                              25.9
--------------------------------------------------------------------------------
January, 2003                                                               26.0
--------------------------------------------------------------------------------
February, 2003                                                              26.0
--------------------------------------------------------------------------------
March, 2003                                                                 26.0
--------------------------------------------------------------------------------
April, 2003                                                                 26.0
--------------------------------------------------------------------------------
May, 2003                                                                   26.0
--------------------------------------------------------------------------------

     Capital Expenditures. The Borrower shall not permit or suffer its
Consolidated Capital Expenditures to exceed the following:

--------------------------------------------------------------------------------
Fiscal Year              Maximum for Entire                 Maximum for Each
                         Fiscal Year                        Quarter
--------------------------------------------------------------------------------

                                                                  EXHIBIT 7:7-11
                                                                          Page 2





<PAGE>

<TABLE>
--------------------------------------------------------------------------------
  <S>                     <C>                       <C>
  2002                    $5,000,000                $2,500,000
--------------------------------------------------------------------------------

  2003                     6,000,000                 2,500,000
--------------------------------------------------------------------------------
</TABLE>

     Excess Availability.    The Borrower shall at all times have Excess
Availability of not less than the following:

<TABLE>
<CAPTION>
--------------------------------------------------------------------------------
  From                    To                        Minimum Excess
                                                    Availability ($ Millions)
--------------------------------------------------------------------------------
  <S>                     <C>                       <C>
  Closing                 November 30, 2001         5.0
--------------------------------------------------------------------------------
  December 1, 2001        Maturity Date             11.0
--------------------------------------------------------------------------------
</TABLE>



                                                                  EXHIBIT 7:7-11
                                                                          Page 3