<PAGE>   1
                                                                    EXHIBIT 10.1


                                FOURTH AMENDMENT
                          TO THE MEN'S WEARHOUSE, INC.
                  1992 NON-EMPLOYEE DIRECTOR STOCK OPTION PLAN


         THIS AGREEMENT by The Men's Wearhouse, Inc. (the "Sponsor"),

                                   WITNESSETH:

         WHEREAS, the Sponsor maintains the Plan known as "The Men's Wearhouse,
Inc. 1992 Non-Employee Director Stock Option Plan" (the "Plan"); and

         WHEREAS, the Sponsor retained the right in Paragraph 13 of the Plan to
amend the Plan from time to time; and

         WHEREAS, the Board of Directors of the Sponsor approved resolutions on
the 23rd day of March, 2001, to amend the Plan;

         NOW, THEREFORE, the Sponsor agrees that, effective March 23, 2001, but
subject to the condition subsequent of approval of the holders of at least a
majority of the outstanding shares of common stock, $.01 par value, of the
Sponsor,

(i)      Paragraph 3 of the Plan is hereby amended in its entirety to read as
         follows:

                  "3. Option Shares. The stock subject to the Options and other
         provisions of the Plan shall be shares of the Company's Common Stock,
         $.01 par value (or such other par value as may be designated by act of
         the Company's stockholders) (the "Common Stock"). The total amount of
         the Common Stock with respect to which Options may be granted shall not
         exceed in the aggregate 167,500 shares; provided, that the class and
         aggregate number of shares which may be subject to the Options granted
         hereunder shall be subject to adjustment in accordance with the
         provisions of Paragraph 12 hereof. Such shares may be treasury shares
         or authorized but unissued shares.

                  In the event that any outstanding Option for any reason shall
         expire or terminate by reason of the death of the optionee or the fact
         that the optionee ceases to be a director, the surrender of any such
         Option, or any other cause, the shares of Common Stock allocable to the
         unexercised portion of such Option may again be subject to an Option
         under the Plan."


<PAGE>   2
(ii)     Paragraph 4 of the Plan is hereby amended in its entirety to read as
         follows:

                 "4. Grant of Options. Subject to the provisions of Paragraph 16
         and the availability under the Plan of a sufficient number of shares of
         Common Stock that may be issuable upon the exercise of outstanding
         Options, each person who becomes a Non-Employee Director shall be
         granted, on the date he or she becomes a director of the Company, an
         Option under the Plan to purchase 3,000 shares of Common Stock at a
         price per share (the "Option Price") equal to the fair market value of
         the Common Stock on such date. In addition, for so long as this Plan is
         in effect and shares are available for the grant of Options hereunder,
         each Non-Employee Director who is a director of the Company on the last
         Friday of any fiscal year of the Company (the "Final Friday") shall be
         granted an Option to purchase 3,000 shares of the Common Stock at an
         Option Price equal to the fair market value of the Common Stock as of
         such Final Friday.

                  For purposes of this Paragraph 4 and Paragraph 7 below, the
         "fair market value" of a share of Common Stock as of any particular
         date shall mean the closing price of a share of Common Stock on that
         date as reported on the New York Stock Exchange, provided that if no
         closing price for the Common Stock was so reported on that date, then
         the Option Price shall be the fair market value as of the first
         preceding date for which such prices are reported."

(iii)    Paragraph 16 of the Plan is hereby amended in its entirety to read as
         follows:

                  "16. Effective Date of Plan. The Plan shall become effective
         and shall be deemed to have been adopted on February 24, 1992, if
         within one year of that date either (i) it shall have been approved by
         the holders of at least a majority of the outstanding shares of voting
         stock of the Company or if the provisions of the corporate charter,
         by-laws or applicable state law prescribes a greater degree of
         stockholder approval for this action, the approval by the holders of
         that percentage, at a meeting of stockholders or (ii) the Committee
         shall have received an opinion of legal counsel to the effect that such
         approval is not required (a) by law, or (b) in order to make available
         to the optionee with respect to the Option the benefits of Rule 16b-3
         of the Rules and Regulations under the 1934 Act. No Option shall be
         granted pursuant to the Plan after February 23, 2012."

                                      -2-