Exhibit 10.6

MEMORANDUM OF UNDERSTANDING

CASUAL MALE RETAIL GROUP, INC.

NON-QUALIFIED DEFERRED COMPENSATION PLAN

Casual Male Retail Group, Inc. has adopted a nonqualified Deferred Compensation
Plan for certain top paid employees. Such Plan, which shall be referred to
herein as the “Wraparound Plan” is intended to work in conjunction with the
Casual Male Retail Group, Inc. 401(k) Salaried Plan to reduce the administrative
problems associated with refunding excess 401(k) contributions to highly
compensated employees, and allow such employees to defer a higher percentage of
their compensation. The Wraparound Plan shall be operated in good faith
compliance with Internal Revenue Code § 409A and the regulations and other
guidance, until such time as final regulations are released regarding Code §
409A and a final Wraparound Plan document is prepared based on the following
terms and conditions:

 

  1. Adoption and Effective Date

 

  a. Casual Male Retail Group, Inc. adopts a new nonqualified deferred
compensation plan (the Wraparound Plan) with an original Effective Date of
November 1, 2006

 

  b. The provisions of Internal Revenue Code § 409A governing the nature and
timing of elections, withdrawals, and distributions under the Wraparound Plan
shall apply to amounts deferred under the Wraparound Plan.

 

  2. Employer and Plan Information

 

  a. Employer – Casual Male Retail Group, Inc.

 

  b. EIN 04-2623104

 

  c. Associated Plan : Casual Male Retail Group, Inc. 401(k) Salaried Plan

 

  d. Plan Administrator: Casual Male Retail Group, Inc.

 

  e. The Plan Year is the 12-consecutive month period ending December 31 of each
year

 

  f. The first Plan Year is a short year that begins on the original Effective
Date and ends on December 31, 2006.

 

  3. Eligibility to Participate

 

  a. Covered Employment includes only employment with the Employer as a highly
compensated employee designated as eligible to participate in the Wraparound
Plan by the Plan Administrator.

 

  b. In order for an Employee to participate in the Wraparound Plan, he or she
must be in Covered Employment and must also have met the age and service
requirements for participation in the Associated Plan.

 

Page 1 of 5

--------------------------------------------------------------------------------

  4. Employee Deferrals

 

  a. Employee Deferrals will be allowed under the Wraparound Plan.

 

  b. Employee Deferrals for a Plan Year must be made by December 31 of the year
prior to the Plan Year in which the services to which the deferrals relate are
performed.

 

  c. By March 15 of the year following the Plan Year in which Employee Deferrals
are made to the Wraparound Plan, the Employer will perform or cause to be
performed nondiscrimination testing of the Associated Plan to determine the
maximum amount that each Participant in the Wraparound Plan may contribute to
the Associated Plan with respect to the immediately preceding Plan Year. The
Employee Deferrals for each Participant not in excess of that amount is then
contributed from the Wraparound Plan to the Associated Plan. The balance of the
Employee Deferrals for the applicable Plan Year continue to be deferred under
the Wraparound Plan.

Notwithstanding the foregoing, Employee Deferrals will be allowed with respect
to Compensation earned on and after November 1, 2006 and prior to December 31,
2006.

 

  d. Employee Deferrals will be made by means of payroll reduction, in any whole
percentage or whole dollar amount of Compensation. There is no maximum imposed
on Employee Deferrals under the Wraparound Plan.

 

  e. Compensation means earnings required to be reported in the Wages, Tips and
Other Compensation box of Form W-2 excluding Employee Pre-Tax Contributions and
other Elective Deferrals, and elective contributions that are excluded from
income under Code § 125 (cafeteria plan); and reimbursements or other expense
allowances, fringe benefits (cash and non cash), moving expenses, deferred
compensation and welfare benefits.

 

  5. Matching Credits

 

  a. Matching Credits will be added to each Participant’s Account based on the
amount of the Participant’s Employee Deferrals for the Plan Year and shall be
determined annually for each Plan Year by the Employer, in its absolute and sole
discretion, and allocated to Participants in proportion to their Employee
Deferrals for the Plan Year.

 

  b. The Employer intends to limit the Matching Credits to no more than 50% of
the first six percent (6%) of Compensation deferred, including compensation
deferred under the Associate Plan. The Matching Credits for the Wraparound Plan
will be offset by matching contributions made to the Associated Plan. The
Employer reserves the right to change the Matching Credit Formula, as defined in
the Associated Plan.

 

Page 2 of 5

--------------------------------------------------------------------------------

  c. By March 15 of the year following the Plan Year in which Matching Credits
are made to the Wraparound Plan, the Employer will perform or cause to be
performed nondiscrimination testing of the Associated Plan to determine the
maximum amount of Employer Matching Contributions that may be contributed on
behalf of each Participant in the Wraparound Plan to the Associated Plan with
respect to the immediately preceding Plan Year. The Employer Matching Credits
for each Participant not in excess of that amount is then contributed from the
Wraparound Plan to the Associated Plan. The balance of the Employer Matching
Credits for the applicable Plan Year continue to be deferred under the
Wraparound Plan.

 

  d. A Participant will receive Matching Credits described above only if he or
she is employed by the Employer on the last day of the Plan Year.

 

  6. Supplemental Credits

 

  a. Supplemental Credits, without regard to whether any Employee Deferrals have
been made by a Participant for the Plan Year will be allowed under the
Wraparound Plan.

 

  b. Supplemental Credits will be added by the Employer, in its discretion, and
credited to the Participant Accounts in proportion to their Compensation (as
defined in section 4c. above) for the Plan Year.

 

  c. A Participant will receive Supplemental Credits described above only if he
or she is employed by the Employer on the last day of the Plan Year.

 

  d. Notwithstanding section 6c., the last-day employment requirement will not
apply in any case where a Participant’s employment terminates during the Plan
Year due to the Participant’s death, disability, or retirement after normal
retirement age.

 

  7. Vesting and Forfeiture of Benefits

 

  a. Matching Credits and/or Supplemental Credits will be Vested according to
the following schedule based on Years of Service with the Employer:

 

Years of Service

   Percent Vested  

Less than 1 year

   0 %

At least 1 year but less than 2 years

   34 %

At least 2 years but less than 3 Years

   67 %

At least 3 years

   100 %

 

  b. Years of Service for purposes of Vesting shall be calculated under the
actual hours method.

 

  c. Special Vesting Rules – Notwithstanding any vesting schedule set forth
above, a Participant will become 100% Vested in his or her Account if:

 

  i. The Participant’s employment with the Employer is terminated due to death
or disability; or

 

Page 3 of 5

--------------------------------------------------------------------------------

  ii. There is a Change in Control as permitted under Code § 409A.

 

  8. Payment of Benefits

 

  a. Distributions of benefits under the Wraparound Plan may be made on account
of Retirement at or after the Participant’s Normal Retirement Age; disability,
death, termination of employment (i.e., separation from service), a Change in
Control of the Employer, or Unforeseeable Emergency.

 

  b. Distributions of benefits under the Wraparound Plan shall commence on a
date chosen by the Participant at the same time he or she chooses the form of
payment, but commencing no later than 1 year following the Participant’s
termination of employment.

 

  c. Distributions of benefits under the Wraparound Plan shall commence as soon
as practicable in a lump sum after the occurrence of a Change in Control.

 

  d. The forms of distribution of benefits under the Wraparound Plan shall be at
the election of the Participant either in,

 

  i. A single lump-sum,

 

  ii. Installments over a fixed period,

 

  iii. Installments of a fixed dollar amount, or

 

  iv. A combination of lump sum and installment payments, as elected by the
Participant and prescribed by the Plan Administrator.

 

  v. No installment payout period exceeding a period of three years may be
elected, and no installment s of a fixed dollar amount may be elected if it is
deemed likely to result in a payment period in excess of the period of years
specified above, as determined by the Plan Administrator in its sole discretion.

 

  e. If a Participant selects a date for the commencement of benefit payments
and /or a form of benefits, such selection will be made at the time the
Participant first becomes eligible for the Wraparound Plan, and the Participant
may later elect a different payment form and date for commencement of payments,
subject to the limitations of Code § 409A. Any such new election may not be
effective until twelve months after it is filed with the Plan Administrator. In
addition, the benefit payment commencement date must be delayed for at least 5
years from the date of the originally elected commencement date.

 

Page 4 of 5

--------------------------------------------------------------------------------

  f. If a Participant dies before receiving payment of all amounts credited to
his Participant Account, any amounts payable to the Participant’s Beneficiary
will be paid in a lump sum, as soon as practicable after the Participant’s
death.

 

  g. A Participant may apply to the Plan Administrator for a cash distribution
from the Plan, in the event of an Unforeseeable Emergency.

Unforeseeable emergency means a severe financial hardship of the Participant or
beneficiary resulting from an illness or accident of the Participant or
beneficiary, the participant’s spouse or the beneficiary’s spouse or the
Participant’s or beneficiary’s dependent ( as defined in Code section 152(a));
loss of the Participant’s or beneficiary’s property due to casualty (including
the need to rebuild a home following damage to a home not otherwise covered by
insurance, for example, not as a result of a natural disaster); or other similar
extraordinary and unforeseeable circumstances arising as a result of events
beyond the control of the Participant or beneficiary

Distributions because of an unforeseeable emergency must be limited to the
amount reasonably necessary to satisfy the emergency need (which may include
amounts necessary to pay any Federal, state or local income taxes or penalties
reasonably anticipated to result from the distribution). Determinations of
amounts necessary to satisfy the emergency must take into account any additional
compensation that is available if the plan provides for cancellation of a
deferral election upon a payment due to an unforeseeable emergency.

 

Page 5 of 5