<PAGE>

                                                                   Exhibit 10.24


                       SEVERANCE COMPENSATION AGREEMENT


     This Severance Compensation Agreement (the "Agreement") by and between
Casual Male Corp., a Massachusetts corporation (the "Company") with its
principal place of business at 555 Turnpike Street, Canton, Massachusetts and
Stuart M. Glasser of 318 Beacon Street, Boston, Massachusetts 02116 (the
"Executive") shall be effective as of the date the Agreement is approved by the
United States Bankruptcy Court for the Southern District of New York (the
"Bankruptcy Court").

     WHEREAS, the Company and certain of its affiliates filed voluntary
petitions to reorganize their businesses under Chapter 11 of the United States
Bankruptcy Code, as amended, on May 18, 2001, which cases have been
administratively consolidated under the heading "Casual Male Corp. et al." and
have been assigned case numbers 01-41403(REG) through 01-41418(REG) (the
"Chapter 11 Case");

     WHEREAS, the Board of Directors of the Company has determined that it is in
the best interests of the Company and its stakeholders that certain key members
of management be provided with the appropriate incentives to cause them to
remain employed by the Company pending the Company's reorganization;

     WHEREAS, the Official Committee of Unsecured Creditors of the Company has
consented to the Company entering into this Agreement;

     NOW THEREFORE, in consideration of the agreements contained herein
including the undertakings of the parties hereto, the receipt and sufficiency of
which are hereby acknowledged by each of the parties hereto, it is covenanted
and agreed as follows:

1.   Severance Compensation upon Termination of Employment.
     ------------------------------------------------------

     In the event that the Executive's employment with the Company or any of its
affiliates is terminated for one of the reasons described below, the Company
shall pay to the Executive the corresponding cash severance payment (the
"Severance Payment"):

(a)       Termination Without Cause or For Good Reason.  In the event the
          --------------------------------------------
     Executive's employment with the Company is terminated (x) by the Company
     without Cause or (y) by the Company or by the Executive for "good reason"
     (as defined in Section 1(d) below), the Severance Payment shall be the
     Executive's Annual Base Salary for a period of 18 months and shall be
     payable to the Executive in accordance with the Company's regular pay
     intervals for its senior executives beginning immediately following the
     Executive's Termination Date. That portion of the Severance Payments made
     pursuant to this Section 1(a) equaling six (6) months

                                       1
<PAGE>

     of the Executive's Annual Base Salary shall be offset by any salary, earned
     income, deferred income or other compensation earned by the Executive from
     other employment during the first six (6) months of the Severance Period,
     it being understood that the Executive shall use reasonable efforts to find
     new employment suitable to, and consistent with, his or her training and
     experience during the first six (6) month period following the Termination
     Date.

(b)       Change of Control.  In the event that the Executive's employment with
          -----------------
     the Company is terminated by the Company or any successor thereto following
     or in connection with a "change of control" as defined in Exhibit A hereto
     then, at the election of the Executive, the Severance Payment shall be
     either: (x) a payment equal to the Executive's Annual Base Salary for a
     period of 18 months, payable in accordance with the Company's normal pay
     intervals for its senior executives, provided, that the portion of such
                                          --------  ----
     Severance Payment equaling six (6) months of Executive's Annual Base Salary
     shall be offset by any salary, earned income, deferred income or other
     compensation earned by the Executive from other employment during the first
     six (6) months of the Severance Period (it being understood that the
     Executive shall use reasonable efforts to find new employment suitable to,
     and consistent with, his or her training and experience during the first
     six (6) month period following the Termination Date) or (y) a payment equal
     to Executive's Annual Base Salary for a period of 12 months made to the
     Executive in a single lump sum cash payment within ten (10) business days
     of the Termination Date, which payment shall not by offset by salary,
     earned income, deferred income or other income earned by the Executive from
     other Employment.

(c)       Liquidation of Company.  In the event that the Executive's employment
          ----------------------
     with the Company is terminated by the Company or its estate following or in
     connection with the conversion of the Company's Chapter 11 Case to a case
     under Chapter 7 of the United States Bankruptcy Code or other liquidation
     of the Company, the Severance Payment shall be a payment equal to
     Executive's Annual Base Salary for a period of 12 months made to the
     Executive in a single lump sum cash payment within ten (10) business days
     of the Termination Date, which payment shall not by offset by salary,
     earned income, deferred income or other income earned by the Executive from
     other Employment.

(d)       Certain Definitions.  For purposes of this Agreement, the following
          -------------------
     words and phrases shall have the following meanings:

     (i)       "Annual Base Salary" shall mean the Executive's base salary in
          effect on the date of this Agreement, as such base salary may be
          increased from time to time.

     (ii)      "Termination Date" shall mean the date upon which the

                                       2
<PAGE>

            Executive formally ceases all regular employment activities on
            behalf of the Company as recognized by the records of the Company.

     (iii)     A termination for "good reason" shall be deemed to have occurred,
            and the Executive shall be entitled to the benefits set forth in
            this Section 1, if the Executive voluntarily terminates his
            employment after the occurrence of any of the following events: (x)
            the assignment to the Executive of any duties inconsistent with the
            highest position (including status, offices, titles and reporting
            requirements), authority, duties or responsibilities attained by the
            Executive during the period of his employment by the Company; (y) a
            relocation of the Executive outside the metropolitan Boston area; or
            (z) a decrease in the Executive's compensation (including base
            salary, bonus or fringe benefits).

     (iv)      "Cause" shall mean (w) willful misconduct with regard to the
            Company intended to have, and having, a material adverse affect on
            the Company, (x) failure by the Employee to cure a material breach
            of this Agreement within 15 days after written notice thereof by the
            Company, (y) refusal to or failure to attempt to follow the
            reasonable written legal direction of the Board of Directors of the
            Company or the Chief Executive Officer of the Company or (z) the
            commission by the Employee of an act constituting a felony.

2.   Effective Date.
     ---------------

     This Agreement shall become effective as of the date on which the
Bankruptcy Court grants its approval thereto.

3.   Release of Liability.
     --------------------

     Notwithstanding anything to the contrary herein, as a condition precedent
to the Company's obligation to make the Severance Payment described in Section 1
hereof, Executive shall execute and deliver to the Company a release and waiver
of employment liabilities in substantially the form attached to this Agreement
as Exhibit B, provided, however, that the Company may modify such form from time
to time in order for such form to comply with the continually developing law of
employment release enforceability.

4.   Confidential Information.
     ------------------------

     The Employee will never use for his own advantage or disclose any
proprietary or confidential information relating to the business operations or
properties of the Company, any affiliate thereof or any of their respective
customers, suppliers, landlords or licensees.  On the Termination Date, the
Executive will surrender and deliver to the Company all documents and

                                       3
<PAGE>

information of every kind relating to or connected with the Company and its
affiliates and their respective businesses, customers, suppliers, landlords and
licensees.

5.   Nonsolicitation of Employees.
     ----------------------------

     (a)  For a one (1) year period following the Termination Date, Executive
will not, in any manner, hire or engage, or assist any company or business
organization by which he is employed or which is directly or indirectly
controlled by him to hire or engage, any person who is or was employed by the
Company or one of its affiliates at any time during his employment with the
Company or during the period of one (1) year thereafter.

     (b)  For a one (1) year period following the Termination Date, Executive
will not, in any manner, solicit, recruit or induce, or assist any company or
business organization by which he is employed or which is directly or indirectly
controlled by him to solicit, recruit or induce, any person who is or was
employed by the Company or one of its affiliates (or is or was an agent,
representative, contractor, project consultant or consultant of the Company or
one of its affiliates) at any time during his employment with the Company, or
during the period of one (1) year thereafter, to leave his or her employment,
relationship or engagement with the Company.


6.   Nonsolicitation of Customers.
     ----------------------------

     (a)  Executive agrees that, for a period of one (1) year following the
Termination Date he will not, in any manner, solicit for business (or assist any
company or business organization by which he is employed or which is directly or
indirectly controlled by him to solicit or do business) any customer or client
of the Company or any of its affiliates with whom Executive has had contact or
for whom Executive has performed services during his employment with the
Company.

     (b)  Executive agrees that, for a period of one (1) year following the
Termination Date, Executive will not, in any manner, solicit for business (or
assist any company or business organization by which he is employed or which is
directly or indirectly controlled by him to solicit or do business) any customer
or client of the Company made known to him by the Company during his employment
with the Company.

7.   Company's Sole Severance Obligation.
     -----------------------------------

     Executive acknowledges and agrees that the Severance Payment set forth in
Section 1 is in lieu of and shall preclude any and all other severance or
termination payments that would otherwise be made by Company to Executive as the
result of Company policy, state or federal laws or regulations or otherwise
(including, without limitation, claims under the Worker Adjustment and
Retraining Act) and that Company's commitment to make said Severance Payment is
subject to and conditioned upon the prior execution by Executive of the release
and waiver described in Section 3 above.

                                       4
<PAGE>

8.   Nonguarantee of Employment.
     ---------------------------

     Nothing contained in this Agreement shall be construed as a contract of
employment between the Company and the Executive, or as a right of the Executive
to continue in the employ of the Company, or as a limitation of the right of the
Company to discharge the Executive with or without cause.

9.   Successors.
     -----------

     (a)  This Agreement shall be binding upon the Company, its successors and
assigns, and in the event of a Change of Control of the Company or in the event
the Company shall be merged or consolidated or otherwise combined into one or
more other corporations or other entities, or substantially all of its assets
are sold or otherwise transferred to one or more other corporations or entities,
this Agreement shall be binding upon the corporation or entity resulting from
such merger or consolidation or to which such assets shall be sold or
transferred and shall be assignable by it by way of transfer of assets, merger,
consolidation or combination to the same extent as if it were the Company.
Except as provided above in this Section 8(a), this Agreement shall not be
assignable by the Company or its successors and assigns. The Company will
require any successor or assign (whether direct or indirect, by purchase,
merger, consolidation or otherwise) to all or substantially all of the business
and/or assets of the Company, by agreement in form and substance satisfactory to
the Executive, expressly, absolutely and unconditionally to assume and agree to
perform this Agreement in the same manner and to the same extent that the
Company would be required to perform it if no such succession or assignment had
taken place.

     (b)  This Agreement shall inure to the benefit of and be enforceable by the
Executive's personal or legal representatives, executors, administrators,
successors, heirs, distributees, devisees and legatees.

10.  Assignment by Executive.
     ------------------------

     This Agreement shall not be assignable by the Executive and shall not be
subject to attachment, execution, pledge or hypothecation.

11.  Notice.
     -------

     For the purpose of this Agreement, notices and all other communications
provided for in this Agreement shall be in writing and either delivered in hand
or by mail by United States registered or certified mail, return receipt
requested, postage prepaid, or by nationally recognized overnight courier, and
shall be deemed to have been duly given the sooner of when actually received or
three (3) days following deposit (a) in the mail by United States registered or
certified

                                       5
<PAGE>

mail, return receipt requested, postage prepaid or (b) with a nationally
recognized overnight courier, as follows:

     If to the Company:

          Casual Male Corp.
           555 Turnpike Street
           Canton, MA  02021
           Attn:  Alan I. Weinstein
                  Chief Executive Officer

           Copy to: General Counsel

     If to the Executive:

           Stuart M. Glasser
           318 Beacon Street
           Boston, MA 02116

or to such other address as either party may have furnished to the other in
writing in accordance herewith, except that notices of change of address shall
be effective only upon receipt.

12.  Modification.
     -------------

     No provision of this Agreement may be modified, waived or discharged unless
such waiver, modification or discharge is agreed to in writing signed by the
Executive and the Company.  No waiver by either party hereto of, or compliance
with, any condition or provision of this Agreement to be performed by such party
shall be deemed a waiver of any other provisions hereof or of any similar or
dissimilar provisions or conditions at the same or any prior or subsequent time.
No agreements or representations, oral or otherwise, express or implied, with
respect to the subject matter hereof have been made by either party which are
not set forth expressly in this Agreement.


13.  Validity.
     ---------

     The invalidity or unenforceability of any provisions of this Agreement
shall not affect the validity or enforceability of any other provisions of this
Agreement, which shall remain in full force and effect.

14.  Governing Law.
     --------------

     This Agreement shall be governed by the laws of the Commonwealth of
Massachusetts without giving effect to the conflicts of law principles thereof.
The parties hereby agree that

                                       6
<PAGE>

Bankruptcy Court shall retain exclusive jurisdiction to determine any disputes
under this Agreement during the pendency of the Chapter 11 Case (or any
conversion of such case to a case under Chapter 7 of the United States
Bankruptcy Court).

15.  Entire Agreement
     ----------------

     This Agreement constitutes the entire understanding of the parties, and
revokes and supersedes all prior agreements between the parties (including,
without limitation, any employment agreements, severance agreements and change
of control agreements) and is intended as a final expression of their Agreement.
This Agreement shall take precedence over any other documents that may be in
conflict therewith.

IN WITNESS WHEREOF, the parties hereto have executed this Agreement as of the
date first above written.

                                         CASUAL MALE CORP.


                                         By:  /s/ Alan I. Weinstein
                                              ---------------------
                                              Alan I. Weinstein
                                              Chief Executive Officer


                                         EXECUTIVE:


                                              /s/ Stuart M. Glasser
                                              ---------------------
                                              Stuart M. Glasser

                                       7
<PAGE>

                                   EXHIBIT A

"Change of Control" shall mean the occurrence of any one of the following
events:

          (i)   any "person," as such term is used in Sections 13(d) and 14(d)
     of the Securities Exchange Act of 1934 (the "Act") (other than the Company,
     any of its Subsidiaries, or any trustee, fiduciary or other person or
     entity holding securities under any employee benefit plan or trust of the
     Company or any of its Subsidiaries), together with all "affiliates" and
     "associates" (as such terms are defined in Rule 12b-2 under the Act) of
     such person, shall become the "beneficial owner" (as such term is defined
     in Rule 13d-3 under the Act), directly or indirectly, of securities of the
     Company representing 30% or more of either (A) the combined voting power of
     the Company's then outstanding securities having the right to vote in an
     election of the Company's Board of Directors ("Voting Securities") or (B)
     the then outstanding shares of Stock of the Company (in either such case
     other than as a result of an acquisition of securities directly from the
     Company); or

          (ii)  persons who, as of the Effective Date, constitute the Company's
     Board of Directors (the "Incumbent Directors") cease for any reason,
     including, without limitation, as a result of a tender offer, proxy
     contest, merger or similar transaction, to constitute at least a majority
     of the Board, provided that any person becoming a director of the Company
     subsequent to the Effective Date whose election or nomination for election
     was approved by a vote of at least a majority of the Incumbent Directors
     shall, for purposes of this Plan, be considered an Incumbent Director; or

     (iii) the stockholders of the Company shall approve (A) any consolidation
     or merger of the Company or any Subsidiary where the shareholders of the
     Company, immediately prior to the consolidation or merger, would not,
     immediately after the consolidation or merger, beneficially own (as such
     term is defined in Rule 13d-3 under the Act), directly or indirectly,
     shares representing in the aggregate 80% or more of the voting shares of
     the corporation issuing cash or securities in the consolidation or merger
     (or of its ultimate parent corporation, if any), (B) any sale, lease,
     exchange or other transfer (in one transaction or a series of transactions
     contemplated or arranged by any party as a single plan) of all or
     substantially all of the assets of the Company or (C) any plan or proposal
     for the liquidation or dissolution of the Company.

     Notwithstanding the foregoing, a "Change of Control" shall not be deemed to
have occurred for purposes of the foregoing clause (i) solely as the result of
an acquisition of securities by the Company which, by reducing the number of
shares of Stock or other Voting Securities outstanding, increases (x) the
proportionate number of shares of Stock beneficially owned by any person to 30%
or more of the shares of Stock then outstanding or (y) the proportionate voting

                                       8
<PAGE>

power represented by the Voting Securities beneficially owned by any person to
30% or more of the combined voting power of all then outstanding Voting
Securities; provided, however, that if any person referred to in clause (x) or
            --------  -------
(y) of this sentence shall thereafter become the beneficial owner of any
additional shares of Stock or other Voting Securities (other than pursuant to a
stock split, stock dividend, or similar transaction), then a "Change of Control"
shall be deemed to have occurred for purposes of the foregoing clause (i).

Further, notwithstanding the foregoing, a "Change of Control" shall not be
deemed to have occurred for purposes of the foregoing clause (i) solely as the
result of the confirmation of a plan of reorganization under Chapter 11 of the
United States Bankruptcy Code pursuant to which the creditors of the Company
become the beneficial owners, directly or indirectly, of securities of the
Company representing 30% or more of either (A) the combined voting power of the
Company's Voting Securities or (B) the then outstanding shares of Stock of the
Company.

                                       9
<PAGE>

                                   EXHIBIT B

                                Form of Release
                                ---------------

     In exchange for the amounts described in the Severance Compensation
Agreement dated as of _________, 2001 and other good and valuable consideration,
the receipt of which is hereby acknowledged, Executive and his representatives,
agents, estate, heirs, successors and assigns, without limitation, hereby
irrevocably and unconditionally release and forever discharge the Company, its
predecessors, successors, parents, subsidiaries, divisions, affiliates, assigns,
and its and their current and former officers, agents, directors, supervisors,
employees, representatives, successors and assigns, and all persons acting by,
through, under, or in concert with, any of them, from any and all charges,
complaints, claims, suits, contracts, causes of action, debts, sums of money,
controversies, agreements, promises, damages, and liabilities of any kind or
nature whatsoever, both at law and equity, known or unknown, suspected or
unsuspected (hereinafter referred to as "claim" or "claims") arising from
conduct occurring up to and through the date of this release, including, without
limitation, any claims incidental to or arising out of Executive's employment
with the Company or the termination thereof. This release is intended by the
parties to be all encompassing and to act as a full and total release of any
claims, whether specifically enumerated herein or not, that Executive has, may
have or has had, that exist or ever have existed, on or prior to the date of
this Agreement, including, but not limited to, any claims based upon any federal
or state law or regulation dealing with either employment or employment
discrimination such as those laws or regulations concerning discrimination on
the basis of age, race, color, religion, creed, sex, sex harassment, sexual
orientation, national origin, ancestry, marital status, handicap or physical
disability, mental disability, medical condition or status as a disabled or
Vietnam-era veteran, veteran status or any military service or application for
military service; any contract, whether oral or written, express or implied; any
tort; or common law.

If applicable:

Waiver of Rights and Claims Under the Age Discrimination in Employment Act of
-----------------------------------------------------------------------------
1967.
-----

     Since Executive is 40 years of age or older, Executive has been informed
and agrees that Executive:

     (a)  has or may have specific rights and/or claims under the Age
Discrimination in Employment Act ("ADEA") of 1967;

     (b)  is, in consideration for the amounts and benefits described in the
Severance Compensation Agreement dated _______, 2001, specifically waiving such
rights and/or claims he might have against the Company, its predecessors,
successors, parents, subsidiaries, divisions, affiliates, assigns, and its and
their current and former officers, agents, directors, supervisors, employees,
representatives, successors and assigns, and all persons acting by, through,
under, or in concert with any of them, to the extent such rights and/or claims
arose prior to the date this release was executed;

                                       10
<PAGE>

     (c)  understands that rights or claims under ADEA which may arise after the
date this release is executed are not waived by him;

     (d)  was advised when presented by the Company with the original draft of
this release that he had at least 21 days within which to consider this release;
this 21-day review period will not be affected or extended by any revisions
which might be made to this release;

     (e)  has been advised to consider the terms of this release carefully and
of his right to consult with or seek advice from an attorney of his choice or
any other person of his choosing prior to executing this release and has not
been subject to any undue or improper influence interfering with the exercise of
his free will in deciding whether to execute this release;

     (f)  has carefully read and fully understands all of the provisions of this
release, and he knowingly and voluntarily agrees to all of the terms set
forth in this release; and

     (g)  after signing this release, Executive may revoke this release for a
period of seven (7) days following said execution. The release shall not become
effective or enforceable under this revocation period has expired.

                                       11