<PAGE>

                                                                   Exhibit 10.32

                       SEVERANCE COMPENSATION AGREEMENT


     This Severance Compensation Agreement (the "Agreement") by and between
Casual Male Corp., a Massachusetts corporation (the "Company") with its
principal place of business at 555 Turnpike Street, Canton, Massachusetts and
Elizabeth C. White of 11 Wycliffe Road, Walpole, MA 02032 (the "Executive")
shall be effective as of the date the Agreement is approved by the United States
Bankruptcy Court for the Southern District of New York (the "Bankruptcy Court").

     WHEREAS, the Company and certain of its affiliates filed voluntary
petitions to reorganize their businesses under Chapter 11 of the United States
Bankruptcy Code, as amended, on May 18, 2001, which cases have been
administratively consolidated under the heading "Casual Male Corp. et al." and
have been assigned case numbers 01-41403(REG) through 01-41418(REG) (the
"Chapter 11 Case");

     WHEREAS, the Board of Directors of the Company has determined that it is in
the best interests of the Company and its stakeholders that certain key members
of management be provided with the appropriate incentives to cause them to
remain employed by the Company pending the Company's reorganization;

     WHEREAS, the Official Committee of Unsecured Creditors of the Company has
consented to the Company entering into this Agreement;

     NOW THEREFORE, in consideration of the agreements contained herein
including the undertakings of the parties hereto, the receipt and sufficiency of
which are hereby acknowledged by each of the parties hereto, it is covenanted
and agreed as follows:


1.   Severance Compensation upon Termination of Employment.
     -----------------------------------------------------

     In the event that the Executive's employment with the Company or any of its
affiliates is terminated for one of the reasons described below, the Company
shall pay to the Executive the corresponding cash severance payment (the
"Severance Payment"):

     (a)       Termination Without Cause or For Good Reason. In the event the
               --------------------------------------------
          Executive's employment with the Company is terminated (x) by the
          Company without Cause or (y) by the Company or by the Executive for
          "good reason" (as defined in Section 1(d) below), the Severance
          Payment shall be the Executive's Annual Base Salary for a period of 14
          months and shall be payable to the Executive in accordance with the
          Company's regular pay intervals for its senior executives beginning
          immediately following the Executive's Termination Date. The Severance
          Payments made pursuant to this Section 1(a) shall be offset by any
          salary, earned income, deferred income or other compensation earned by
          the Executive from other

                                       1
<PAGE>

          employment during the Severance Period, it being understood that the
          Executive shall use reasonable efforts to find new employment suitable
          to, and consistent with, his or her training and experience during
          such 14 month period.


     (b)       Change of Control. In the event that the Executive's employment
               -----------------
          with the Company is terminated by the Company or any successor thereto
          following or in connection with a "change of control" as defined in
          Exhibit A hereto then, at the election of the Executive, the Severance
          Payment shall be either: (x) a payment equal to the Executive's Annual
          Base Salary for a period of 14 months, payable in accordance with the
          Company's normal pay intervals for its senior executives, offset by
          any salary, earned income, deferred income or other compensation
          earned by the Executive from other employment during the Severance
          Period (it being understood that the Executive shall use reasonable
          efforts to find new employment suitable to, and consistent with, his
          or her training and experience during such 14 month period) or (y) a
          payment equal to Executive's Annual Base Salary for a period of 8
          months made to the Executive in a single lump sum cash payment within
          ten (10) business days of the Termination Date, which payment shall
          not by offset by salary, earned income, deferred income or other
          income earned by the Executive from other Employment.

     (c)       Liquidation of Company. In the event that the Executive's
               ----------------------
          employment with the Company is terminated by the Company or its estate
          following or in connection with the conversion of the Company's
          Chapter 11 Case to a case under Chapter 7 of the United States
          Bankruptcy Code or other liquidation of the Company the Severance
          Payment shall be a payment equal to Executive's Annual Base Salary for
          a period of 8 months made to the Executive in a single lump sum cash
          payment within ten (10) business days of the Termination Date, which
          payment shall not by offset by salary, earned income, deferred income
          or other income earned by the Executive from other Employment.

     (d)       Certain Definitions. For purposes of this Agreement, the
               -------------------
          following words and phrases shall have the following meanings:

          (i)            "Annual Base Salary" shall mean the Executive's base
               salary in effect on the date of this Agreement, as such base
               salary may be increased from time to time.

          (ii)           "Termination Date" shall mean the date upon which the
               Executive formally ceases all regular employment activities on
               behalf of the Company as recognized by the records of the
               Company.

          (iii)          A termination for "good reason" shall be deemed to have

                                       2
<PAGE>

               occurred, and the Executive shall be entitled to the benefits set
               forth in this Section 1, if the Executive voluntarily terminates
               his employment after the occurrence of any of the following
               events: (x) the assignment to the Executive of any duties
               inconsistent with the highest position (including status,
               offices, titles and reporting requirements), authority, duties or
               responsibilities attained by the Executive during the period of
               his employment by the Company; (y) a relocation of the Executive
               outside the metropolitan Boston area; or (z) a decrease in the
               Executive's compensation (including base salary, bonus or fringe
               benefits).

          (iv)           "Cause" shall mean (w) willful misconduct with regard
               to the Company intended to have, and having, a material adverse
               affect on the Company, (x) failure by the Employee to cure a
               material breach of this Agreement within 15 days after written
               notice thereof by the Company, (y) refusal to or failure to
               attempt to follow the reasonable written legal direction of the
               Board of Directors of the Company or the Chief Executive Officer
               of the Company or (z) the commission by the Employee of an act
               constituting a felony.

2.   Effective Date.
     ---------------

     This Agreement shall become effective as of the date on which the
Bankruptcy Court grants its approval thereto.

3.   Release of Liability.
     --------------------

     Notwithstanding anything to the contrary herein, as a condition precedent
to the Company's obligation to make the Severance Payment described in Section 1
hereof, Executive shall execute and deliver to the Company a release and waiver
of employment liabilities in substantially the form attached to this Agreement
as Exhibit B, provided, however, that the Company may modify such form from time
to time in order for such form to comply with the continually developing law of
employment release enforceability.

4.   Confidential Information.
     ------------------------

     The Employee will never use for his own advantage or disclose any
proprietary or confidential information relating to the business operations or
properties of the Company, any affiliate thereof or any of their respective
customers, suppliers, landlords or licensees.  On the Termination Date, the
Executive will surrender and deliver to the Company all documents and
information of every kind relating to or connected with the Company and its
affiliates and their respective businesses, customers, suppliers, landlords and
licensees.

5.   Nonsolicitation of Employees.
     ----------------------------

                                       3
<PAGE>

     (a)  For a one (1) year period following the Termination Date, Executive
will not, in any manner, hire or engage, or assist any company or business
organization by which he is employed or which is directly or indirectly
controlled by him to hire or engage, any person who is or was employed by the
Company or one of its affiliates at any time during his employment with the
Company or during the period of one (1) year thereafter.

     (b)  For a one (1) year period following the Termination Date, Executive
will not, in any manner, solicit, recruit or induce, or assist any company or
business organization by which he is employed or which is directly or indirectly
controlled by him to solicit, recruit or induce, any person who is or was
employed by the Company or one of its affiliates (or is or was an agent,
representative, contractor, project consultant or consultant of the Company or
one of its affiliates) at any time during his employment with the Company, or
during the period of one (1) year thereafter, to leave his or her employment,
relationship or engagement with the Company.


6.   Nonsolicitation of Customers.
     ----------------------------

     (a)  Executive agrees that, for a period of one (1) year following the
Termination Date he will not, in any manner, solicit for business (or assist any
company or business organization by which he is employed or which is directly or
indirectly controlled by him to solicit or do business) any customer or client
of the Company or any of its affiliates with whom Executive has had contact or
for whom Executive has performed services during his employment with the
Company.

     (b)  Executive agrees that, for a period of one (1) year following the
Termination Date, Executive will not, in any manner, solicit for business (or
assist any company or business organization by which he is employed or which is
directly or indirectly controlled by him to solicit or do business) any customer
or client of the Company made known to him by the Company during his employment
with the Company.

7.   Company's Sole Severance Obligation.
     -----------------------------------

     Executive acknowledges and agrees that the Severance Payment set forth in
Section 1 is in lieu of and shall preclude any and all other severance or
termination payments that would otherwise be made by Company to Executive as the
result of Company policy, state or federal laws or regulations or otherwise
(including, without limitation, claims under the Worker Adjustment and
Retraining Act) and that Company's commitment to make said Severance Payment is
subject to and conditioned upon the prior execution by Executive of the release
and waiver described in Section 3 above.

                                       4
<PAGE>

8.   Nonguarantee of Employment.
     ---------------------------

     Nothing contained in this Agreement shall be construed as a contract of
employment between the Company and the Executive, or as a right of the Executive
to continue in the employ of the Company, or as a limitation of the right of the
Company to discharge the Executive with or without cause.


9.   Successors.
     -----------

     (a)  This Agreement shall be binding upon the Company, its successors and
assigns, and in the event of a Change of Control of the Company or in the event
the Company shall be merged or consolidated or otherwise combined into one or
more other corporations or other entities, or substantially all of its assets
are sold or otherwise transferred to one or more other corporations or entities,
this Agreement shall be binding upon the corporation or entity resulting from
such merger or consolidation or to which such assets shall be sold or
transferred and shall be assignable by it by way of transfer of assets, merger,
consolidation or combination to the same extent as if it were the Company.
Except as provided above in this Section 8(a), this Agreement shall not be
assignable by the Company or its successors and assigns.  The Company
will require any successor or assign (whether direct or indirect, by purchase,
merger, consolidation or otherwise) to all or substantially all of the business
and/or assets of the Company, by agreement in form and substance satisfactory to
the Executive, expressly, absolutely and unconditionally to assume and agree to
perform this Agreement in the same manner and to the same extent that the
Company would be required to perform it if no such succession or assignment had
taken place.

     (b)  This Agreement shall inure to the benefit of and be enforceable by the
Executive's personal or legal representatives, executors, administrators,
successors, heirs, distributees, devisees and legatees.


10.  Assignment by Executive.
     ------------------------

     This Agreement shall not be assignable by the Executive and shall not be
subject to attachment, execution, pledge or hypothecation.


11.  Notice.
     -------

     For the purpose of this Agreement, notices and all other communications
provided for in this Agreement shall be in writing and either delivered in hand
or by mail by United States registered or certified mail, return receipt
requested, postage prepaid, or by nationally recognized overnight courier, and
shall be deemed to have been duly given the sooner of when actually

                                       5
<PAGE>

received or three (3) days following deposit (a) in the mail by United States
registered or certified mail, return receipt requested, postage prepaid or (b)
with a nationally recognized overnight courier, as follows:

     If to the Company:

          Casual Male Corp.
          555 Turnpike Street
          Canton, MA  02021
          Attn:     Chief Executive Officer

          Copy to: General Counsel

     If to the Executive:

          Elizabeth C. White
          11 Wycliffe Road
          Walpole, MA 02032

or to such other address as either party may have furnished to the other in
writing in accordance herewith, except that notices of change of address shall
be effective only upon receipt.

12.  Modification.
     -------------

     No provision of this Agreement may be modified, waived or discharged unless
such waiver, modification or discharge is agreed to in writing signed by the
Executive and the Company. No waiver by either party hereto of, or compliance
with, any condition or provision of this Agreement to be performed by such party
shall be deemed a waiver of any other provisions hereof or of any similar or
dissimilar provisions or conditions at the same or any prior or subsequent time.
No agreements or representations, oral or otherwise, express or implied, with
respect to the subject matter hereof have been made by either party which are
not set forth expressly in this Agreement.


13.  Validity.
     ---------

     The invalidity or unenforceability of any provisions of this Agreement
shall not affect the validity or enforceability of any other provisions of this
Agreement, which shall remain in full force and effect.


14.  Governing Law.
     --------------

                                       6
<PAGE>

     This Agreement shall be governed by the laws of the Commonwealth of
Massachusetts without giving effect to the conflicts of law principles thereof.
The parties hereby agree that Bankruptcy Court shall retain exclusive
jurisdiction to determine any disputes under this Agreement during the pendency
of the Chapter 11 Case (or any conversion of such case to a case under Chapter 7
of the United States Bankruptcy Court).

15.  Entire Agreement
     ----------------

     This Agreement constitutes the entire understanding of the parties, and
revokes and supersedes all prior agreements between the parties (including,
without limitation, any employment agreements, severance agreements and change
of control agreements) and is intended as a final expression of their Agreement.
This Agreement shall take precedence over any other documents that may be in
conflict therewith.

     IN WITNESS WHEREOF, the parties hereto have executed this Agreement as of
the date first above written.

                                     Casual Male Corp.


                                     By: /s/ Alan I. Weinstein
                                       -----------------------
                                             Alan I. Weinstein
                                             Chairman and
                                             Chief Executive Officer


                                     EXECUTIVE:


                                     /s/ Elizabeth C. White
                                    -----------------------
                                         Elizabeth C. White

                                       7
<PAGE>

                                   EXHIBIT A



"Change of Control" shall mean the occurrence of any one of the following
events:

           (i)   any "person," as such term is used in Sections 13(d) and 14(d)
     of the Securities Exchange Act of 1934 (the "Act") (other than the Company,
     any of its Subsidiaries, or any trustee, fiduciary or other person or
     entity holding securities under any employee benefit plan or trust of the
     Company or any of its Subsidiaries), together with all "affiliates" and
     "associates" (as such terms are defined in Rule 12b-2 under the Act) of
     such person, shall become the "beneficial owner" (as such term is defined
     in Rule 13d-3 under the Act), directly or indirectly, of securities of the
     Company representing 30% or more of either (A) the combined voting power of
     the Company's then outstanding securities having the right to vote in an
     election of the Company's Board of Directors ("Voting Securities") or (B)
     the then outstanding shares of Stock of the Company (in either such case
     other than as a result of an acquisition of securities directly from the
     Company); or

           (ii)  persons who, as of the Effective Date, constitute the Company's
     Board of Directors (the "Incumbent Directors") cease for any reason,
     including, without limitation, as a result of a tender offer, proxy
     contest, merger or similar transaction, to constitute at least a majority
     of the Board, provided that any person becoming a director of the Company
     subsequent to the Effective Date whose election or nomination for election
     was approved by a vote of at least a majority of the Incumbent Directors
     shall, for purposes of this Plan, be considered an Incumbent Director; or

     (iii) the stockholders of the Company shall approve (A) any consolidation
     or merger of the Company or any Subsidiary where the shareholders of the
     Company, immediately prior to the consolidation or merger, would not,
     immediately after the consolidation or merger, beneficially own (as such
     term is defined in Rule 13d-3 under the Act), directly or indirectly,
     shares representing in the aggregate 80% or more of the voting shares of
     the corporation issuing cash or securities in the consolidation or merger
     (or of its ultimate parent corporation, if any), (B) any sale, lease,
     exchange or other transfer (in one transaction or a series of transactions
     contemplated or arranged by any party as a single plan) of all or
     substantially all of the assets of the Company or (C) any plan or proposal
     for the liquidation or dissolution of the Company.

     Notwithstanding the foregoing, a "Change of Control" shall not be deemed to
have occurred for purposes of the foregoing clause (i) solely as the result of
an acquisition of securities by the Company which, by reducing the number of
shares of Stock or other Voting Securities outstanding, increases (x) the
proportionate number of shares of Stock beneficially owned by any person to 30%
or more of the shares of Stock then outstanding or (y) the proportionate voting

                                       8
<PAGE>

power represented by the Voting Securities beneficially owned by any person to
30% or more of the combined voting power of all then outstanding Voting
Securities; provided, however, that if any person referred to in clause (x) or
            --------  -------
(y) of this sentence shall thereafter become the beneficial owner of any
additional shares of Stock or other Voting Securities (other than pursuant to a
stock split, stock dividend, or similar transaction), then a "Change of Control"
shall be deemed to have occurred for purposes of the foregoing clause (i).

Further, notwithstanding the foregoing, a "Change of Control" shall not be
deemed to have occurred for purposes of the foregoing clause (i) solely as the
result of the confirmation of a plan of reorganization under Chapter 11 of the
United States Bankruptcy Code pursuant to which the creditors of the Company
become the beneficial owners, directly or indirectly, of securities of the
Company representing 30% or more of either (A) the combined voting power of the
Company's Voting Securities or (B) the then outstanding shares of Stock of the
Company.

                                       9
<PAGE>

                                   EXHIBIT B


                                Form of Release
                                ---------------

     In exchange for the amounts described in the Severance Compensation
Agreement dated as of _________, 2001 and other good and valuable consideration,
the receipt of which is hereby acknowledged, Executive and his representatives,
agents, estate, heirs, successors and assigns, without limitation, hereby
irrevocably and unconditionally release and forever discharge the Company, its
predecessors, successors, parents, subsidiaries, divisions, affiliates, assigns,
and its and their current and former officers, agents, directors, supervisors,
employees, representatives, successors and assigns, and all persons acting by,
through, under, or in concert with, any of them, from any and all charges,
complaints, claims, suits, contracts, causes of action, debts, sums of money,
controversies, agreements, promises, damages, and liabilities of any kind or
nature whatsoever, both at law and equity, known or unknown, suspected or
unsuspected (hereinafter referred to as "claim" or "claims") arising from
conduct occurring up to and through the date of this release, including, without
limitation, any claims incidental to or arising out of Executive's employment
with the Company or the termination thereof. This release is intended by the
parties to be all encompassing and to act as a full and total release of any
claims, whether specifically enumerated herein or not, that Executive has, may
have or has had, that exist or ever have existed, on or prior to the date of
this Agreement, including, but not limited to, any claims based upon any federal
or state law or regulation dealing with either employment or employment
discrimination such as those laws or regulations concerning discrimination on
the basis of age, race, color, religion, creed, sex, sex harassment, sexual
orientation, national origin, ancestry, marital status, handicap or physical
disability, mental disability, medical condition or status as a disabled or
Vietnam-era veteran, veteran status or any military service or application for
military service; any contract, whether oral or written, express or implied; any
tort; or common law.

If applicable:

Waiver of Rights and Claims Under the Age Discrimination in Employment Act of
-----------------------------------------------------------------------------
1967.
------

     Since Executive is 40 years of age or older, Executive has been informed
and agrees that Executive:

     (a)  has or may have specific rights and/or claims under the Age
Discrimination in Employment Act ("ADEA") of 1967;

     (b)  is, in consideration for the amounts and benefits described in the
Severance Compensation Agreement dated _______, 2001, specifically waiving such
rights and/or claims he might have against the Company, its predecessors,
successors, parents, subsidiaries, divisions, affiliates, assigns, and its and
their current and former officers, agents, directors, supervisors, employees,
representatives, successors and assigns, and all persons acting by, through,
under, or in concert with any

                                      10
<PAGE>

of them, to the extent such rights and/or claims arose prior to the date this
release was executed;

     (c)  understands that rights or claims under ADEA which may arise after the
date this release is executed are not waived by him;

     (d)  was advised when presented by the Company with the original draft of
this release that he had at least 21 days within which to consider this release;
this 21-day review period will not be affected or extended by any revisions
which might be made to this release;

     (e)  has been advised to consider the terms of this release carefully and
of his right to consult with or seek advice from an attorney of his choice or
any other person of his choosing prior to executing this release and has not
been subject to any undue or improper influence interfering with the exercise of
his free will in deciding whether to execute this release;

     (f)  has carefully read and fully understands all of the provisions of this
release, and he knowingly and voluntarily agrees to all of the terms set forth
in this release; and

     (g)  after signing this release, Executive may revoke this release for a
period of seven (7) days following said execution. The release shall not become
effective or enforceable under this revocation period has expired.

                                      11