<PAGE>

                                                                   Exhibit 10.06

          SECOND AMENDMENT TO DEBTOR IN POSSESSION LOAN AND SECURITY
          ----------------------------------------------------------
                                   AGREEMENT
                                   ---------

     This Second Amendment to Debtor In Possession Loan and Security Agreement
is made as of this 17th day of September, 2001 by and among

     CASUAL MALE CORP. (the "Borrowers' Representative"), a Massachusetts
     corporation, having a principal place of business at 555 Turnpike Street,
     Canton, Massachusetts 02021, as agent for the Borrowers, being those
     Persons named on Exhibit 1 hereto, debtors and debtors in possession;

     The Borrowers;

     each of the Revolving Credit Lenders party to the Loan Agreement (defined
     below) (together with each of their successors and assigns, referred to
     individually as a "Revolving Credit Lender" and collectively as the
     "Revolving Credit Lenders"), and

     FLEET RETAIL FINANCE INC., as Administrative Agent and Collateral Agent, a
     Delaware corporation, having its principal place of business at 40 Broad
     Street, Boston, Massachusetts 02109; and

     BACK BAY CAPITAL FUNDING LLC, as Tranche B Lender and as Tranche C Lender,

in consideration of the mutual covenants herein contained and benefits to be
derived herefrom.

                              W I T N E S S E T H

     A.   Reference is made to the Debtor In Possession Loan and Security
Agreement dated as of May 18, 2001 by and among the Borrowers' Representative,
the Borrowers, the Revolving Credit Lenders, the Tranche B Lender, the Tranche C
Lender, the Administrative Agent and the Collateral Agent (as amended and in
effect, the "Loan Agreement").

     B.   The parties to the Loan Agreement desire to modify and amend certain
provisions of the Loan Agreement, as provided herein.

     Accordingly, the parties hereto agree as follows:

     1.   Definitions.  Capitalized terms used herein and not otherwise defined
          -----------
herein shall have the meanings assigned to such terms in the Loan Agreement.

     2.   Amendments to Exhibits to the Loan Agreement. Exhibit 7:7-11 to the
          --------------------------------------------
Loan Agreement is hereby deleted in its entirety and a new Exhibit 7:7-11 in the
form annexed to this Second Amendment is substituted in its stead.
<PAGE>

     5.   Conditions Precedent to Effectiveness.   This Second Amendment shall
          -------------------------------------
not be effective until each of the following conditions precedent have been
fulfilled to the satisfaction of the Administrative Agent:

     a.     This Second Amendment shall have been duly executed and delivered by
            the respective parties hereto and, shall be in full force and effect
            and shall be in form and substance satisfactory to the
            Administrative Agent and the Lenders.

     b.     All action on the part of the Borrowers necessary for the valid
            execution, delivery and performance by the Borrowers of this Second
            Amendment shall have been duly and effectively taken and evidence
            thereof satisfactory to the Administrative Agent shall have been
            provided to the Administrative Agent.

     c.     The Borrowers shall have provided such additional instruments and
            documents to the Administrative Agent as the Administrative Agent
            and Administrative Agent's counsel may have reasonably requested.

     6.   Miscellaneous.
          -------------

     a.     This Second Amendment may be executed in several counterparts and by
            each party on a separate counterpart, each of which when so executed
            and delivered shall be an original, and all of which together shall
            constitute one instrument.

     b.     This Second Amendment expresses the entire understanding of the
            parties with respect to the transactions contemplated hereby. No
            prior negotiations or discussions shall limit, modify, or otherwise
            affect the provisions hereof.

     c.     Any determination that any provision of this Second Amendment or any
            application hereof is invalid, illegal or unenforceable in any
            respect and in any instance shall not effect the validity, legality,
            or enforceability of such provision in any other instance, or the
            validity, legality or enforceability of any other provisions of this
            Second Amendment.

     d.     The Borrowers shall pay on demand all costs and expenses of the
            Agents and the Lenders, including, without limitation, reasonable
            attorneys' fees in connection with the preparation, negotiation,
            execution and delivery of this Second Amendment.

     e.     The Borrowers warrant and represent that the Borrowers have
            consulted with independent legal counsel of the Borrowers' selection
            in connection with this Second Amendment and is not relying on any
            representations or warranties of the Agents, the Lenders or their
            counsel in entering into this Second Amendment.
<PAGE>

     IN WITNESS WHEREOF, the parties have duly executed this Second Amendment as
of the day and year first above written.

                                    CASUAL MALE CORP.



                                         by
                                            -----------------------
                                         Name:  Elizabeth C. White
                                         Title: Chief Financial Officer

                                    THE BORROWERS
                                         MORSE SHOE, INC.
                                         JBI, INC.
                                         JBI APPAREL, INC.
                                         THE CASUAL MALE, INC.
                                         WGS CORP.
                                         TCMB&T, INC
                                         LP INNOVATIONS, INC.
                                         SPENCER COMPANIES, INC.
                                         BUCKMIN, INC.
                                         ELM EQUIPMENT CORP.
                                         JBI HOLDING COMPANY, INC.
                                         TCM HOLDING COMPANY, INC.
                                         ISAB, INC.
                                         MORSE SHOE INTERNATIONAL,
                                         INC.
                                         WHITE CAP FOOTWEAR, INC


                                         by
                                           -----------------------------
                                         Name:  Elizabeth C. White
                                         Title: Chief Financial Officer

                                    FLEET RETAIL FINANCE INC.,
                                         as Administrative Agent, as Collateral
                                         Agent, as Swingline Lender, and as
                                         Revolving Credit Lender


                                         By:________________________

                                         Name:______________________
<PAGE>

                                         Title:_______________________


                                    BACK BAY CAPITAL FUNDING, LLC
                                         as Tranche B Lender and Tranche C
                                         Lender


                                         By:________________________

                                         Name:______________________

                                         Title:_______________________

The following Lenders consent to the amendments set forth herein with the
exception of the amendment to the financial performance covenants (other than
Excess Availability) set forth on Exhibit 7:7-11, as to which their consent is
not required.


                                    FOOTHILL CAPITAL CORPORATION


                                    By..........................................

                                    Name........................................

                                    Title.......................................


                                    HELLER FINANCIAL, INC.

                                    By..........................................

                                    Name........................................

                                    Title.......................................


                                    LASALLE BUSINESS CREDIT, INC


                                    By..........................................

                                    Name........................................

                                    Title.......................................
<PAGE>

                                    NATIONAL CITY COMMERCIAL FINANCE, INC.


                                    By..........................................

                                    Name........................................

                                    Title.......................................


                                    IBJ WHITEHALL BUSINESS CREDIT CORP.


                                    By..........................................

                                    Name........................................

                                    Title.......................................


                                    THE PROVIDENT BANK


                                    By..........................................

                                    Name........................................

                                    Title.......................................
<PAGE>

                        FINANCIAL PERFORMANCE COVENANTS

     Minimum Cumulative EBITDAR. The Borrowers shall not permit or suffer the
following, tested as of the last day of each month (commencing with June 30,
2001 and through December 31, 2001) on a cumulative basis dating back to January
31, 2001, to be less than the following minimum cumulative amount: (a)
Consolidated EBITDAR minus (b) 100% of any adequate protection payments made by
the Borrowers during such period and not reflected in the Business Plan (without
regard to any updating of the Business Plan), to the extent that such adequate
protection payments were not otherwise included as reductions to EBITDAR for the
period of measurement:

--------------------------------------------------------------------------------
 Month Ending:                                   Minimum Cumulative EBITDAR
                                                 $ Millions
--------------------------------------------------------------------------------
 June, 2001*                                                            6.0
--------------------------------------------------------------------------------
 July, 2001*                                                            5.3
--------------------------------------------------------------------------------
 August, 2001*                                                          5.0
--------------------------------------------------------------------------------
 September, 2001*                                                       7.7
--------------------------------------------------------------------------------
 October, 2001*                                                        10.3
--------------------------------------------------------------------------------
 November, 2001*                                                       14.2
--------------------------------------------------------------------------------
 December, 2001*                                                       27.1
--------------------------------------------------------------------------------

     * The Borrower's compliance with the Minimum Cumulative EBITDAR test for
the subject months will be waived unless Excess Availability is less than
$17,500,000 for any three consecutive days during such period (the "Test
Condition"). If the test Condition occurs; the Borrower shall be obligated to
comply with the Minimum Cumulative EBITDAR test for the subject month in which
the Test Condition occurs and shall also be required to be in compliance with
the Minimum Cumulative EBITDAR test for each month both preceding, and
subsequent to, the month in which the Test Condition occurs (failing which an
Event of Default shall be deemed to have arisen). In any event, whether or not
the Test Condition has occurred, the Borrowers must comply with the Minimum
Cumulative EBITDAR test for October, 2001 and each month thereafter.

     Minimum Rolling Four Quarter EBITDAR. The Borrowers shall not permit or
suffer the following, tested as of the last day of each month (commencing with
January, 2002) on a trailing twelve month basis, to be less than the following
minimum cumulative amount: (a) Consolidated EBITDAR minus (b) 100% of any
adequate protection payments made by the Borrowers during such period and not
reflected in the

<PAGE>

Business Plan (without regard to any updating of the Business Plan), to the
extent that such adequate protection payments were not otherwise included as
reductions to EBITDAR for the period of measurement:

------------------------------------------------------------------------------
 Month Ending:                             Minimum EBITDAR (Rolling 12 Month
                                           Basis):
                                           $ Millions
------------------------------------------------------------------------------
 January, 2002                                                         22.5
------------------------------------------------------------------------------
 February, 2002                                                        22.6
------------------------------------------------------------------------------
 March, 2002                                                           21.8
------------------------------------------------------------------------------
 April, 2002                                                           22.0
------------------------------------------------------------------------------
 May, 2002                                                             23.1
------------------------------------------------------------------------------
 June, 2002                                                            23.5
------------------------------------------------------------------------------
 July, 2002                                                            25.7
------------------------------------------------------------------------------
 August, 2002                                                          25.8
------------------------------------------------------------------------------
 September, 2002                                                       25.8
------------------------------------------------------------------------------
 October, 2002                                                         25.9
------------------------------------------------------------------------------
 November, 2002                                                        26.0
------------------------------------------------------------------------------
 December, 2002                                                        25.9
------------------------------------------------------------------------------
 January, 2003                                                         26.0
------------------------------------------------------------------------------
 February, 2003                                                        26.0
------------------------------------------------------------------------------
 March, 2003                                                           26.0
------------------------------------------------------------------------------
 April, 2003                                                           26.0
------------------------------------------------------------------------------
 May, 2003                                                             26.0
------------------------------------------------------------------------------



                                                                  EXHIBIT 7:7-11
                                                                          Page 2


<PAGE>

     Capital Expenditures. The Borrower shall not permit or suffer its
Consolidated Capital Expenditures to exceed the following:

--------------------------------------------------------------------------------
 Fiscal Year            Maximum for Entire          Maximum for Each
                        Fiscal Year                 Quarter
--------------------------------------------------------------------------------
 2002                   $6,900,000                  $2,500,000, other than for
                                                    the quarter ending on or
                                                    about October 31, 2001, for
                                                    which the maximum shall be
                                                    $4,400,000
--------------------------------------------------------------------------------
 2003                    6,000,000                  2,500,000
--------------------------------------------------------------------------------


     Excess Availability. The Borrower shall at all times have Excess
Availability of not less than the following:

--------------------------------------------------------------------------------
 From                       To                       Minimum Excess
                                                     Availability ($ Millions)
--------------------------------------------------------------------------------
 Closing                    December 6, 2001         5.0
--------------------------------------------------------------------------------
 December 7, 2001           December 13, 2001        6.5
--------------------------------------------------------------------------------
 December 14, 2001          December 20, 2001        8.0
--------------------------------------------------------------------------------
 December 21, 2001          December 27, 2001        9.5
--------------------------------------------------------------------------------
 December 28, 2001          Maturity Date            11.0
--------------------------------------------------------------------------------


                                                                  EXHIBIT 7:7-11
                                                                          Page 3