<PAGE>
 
                                                                    EXHIBIT 10.4

                                                                  EXECUTION COPY


                           RECOURSE PROMISSORY NOTE


March 31, 1999                                                           $48,203


          David McIlquham (the "Employee"), for value received, hereby promises
                                --------                                       
to pay to Sealy Mattress Company, an Ohio corporation (the "Company"), the
                                                            -------       
principal amount of forty-eight thousand two hundred three dollars ($48,203) on
March 31, 2004 (the "Maturity Date").  The Company is a wholly-owned subsidiary
                     -------------                                             
of Sealy Corporation, a Delaware corporation ("Parent").
                                               ------   

          Reference is hereby made to the Stock Purchase Agreement, dated as of
the date hereof, among Bain Capital Fund V-B, L.P. ("Seller"), the Company,
                                                     ------                
Employee, the other "Buyers" listed therein and the other parties named therein
pursuant to which, on the date hereof, Employee has purchased from Seller 8,361
shares of Class A Common Stock and 929 shares of Class L Common Stock (such
shares of Class A Common Stock and Class L Common Stock, the "Purchased
                                                              ---------
Shares"), for an aggregate purchase price of $48,203 (the "Purchase Price").
                                                           --------------    
The Company has loaned the Purchase Price to Employee and Employee has issued
this Note to the Company.

          On the date hereof, Parent and the Employee have entered into an
Executive Stock Agreement (the "Executive Stock Agreement").
                                -------------------------   

          The amounts due under this Note are secured by a pledge of the
Purchased Shares, and the payment of the principal amount and accrued interest
under this Note is subject to certain offset rights under the Executive Stock
Agreement.

          1.   DEFINITIONS.  For purposes of this Note, the following
               -----------                                           
capitalized terms have the following meaning.

          "Class A Common Stock" means Parent's Class A Common Stock, par value
           --------------------                                                
$.01 per share.

          "Class L Common Stock" means Parent's Class L Common Stock, par value
           --------------------                                                
$.01 per share.

          "Employee Shares" means (i) all Purchased Shares, and (ii) all shares
           ---------------                                                     
issued with respect to the Purchased Shares referred to in clause (i) above by
way of a stock dividend or stock split or in connection with any combination,
exchange, conversion, merger, consolidation, recapitalization, or other
reorganization affecting the Class A Common Stock or Class L Common Stock.
<PAGE>
 
          2.   PAYMENT OF PRINCIPAL.
               -------------------- 

          (A)  SCHEDULED PAYMENT.  The Employee will pay the entire unpaid
               -----------------                                          
principal amount of this Note on the Maturity Date.

          (B)  OPTIONAL PREPAYMENT.  Subject to Section 3 hereof, the Employee
               -------------------                                            
may prepay the principal amount of this Note, in whole or in any $1,000
increment, at any time and from time to time.

          (C)  MANDATORY PREPAYMENTS.
               --------------------- 

          (I)  If the Employee sells or otherwise transfers any of the Employee
     Shares, then, on the date of the consummation of such sale or transfer, the
     Employee shall be obligated, to the extent the Employee has obligations to
     the Company under this Note, to pay to the Company an amount equal to the
     gross proceeds received by the Employee for the Employee Shares sold or
     otherwise transferred and such amount paid to the Company shall first
     reduce accrued interest on this Note pursuant to Section 3 hereof and any
     remaining amount paid to the Company shall reduce the principal amount of
     this Note.

          (II) If the Employee receives any cash dividends or other
     distributions with respect to any of the Employee Shares, then, on the date
     of the payment of such cash dividends or other distributions, as the case
     may be, to the Employee, the Employee shall be obligated, to the extent the
     Employee has obligations to the Company under this Note, to pay to the
     Company an amount equal to the amount of such cash dividends or other
     distributions, as the case may be, and such amount paid to the Company
     shall first reduce accrued interest on this Note pursuant to Section 3
     hereof and any remaining amount paid to the Company shall reduce the
     principal amount of this Note.

          3.   INTEREST. Interest will accrue at the rate of five percent (5%)
               --------                                                       
per annum (computed on the basis of a 360-day year, as appropriate, and the
actual number of days elapsed in any year) on the unpaid principal amount of
this Note outstanding from time to time, or (if less) at the highest rate then
permitted under applicable law.  Interest accruing hereunder, will be payable to
the Company in cash, in arrears, prior to the optional or mandatory prepayment
of any principal amount of this Note pursuant to Sections 1(b) or 1(c) hereof,
and, in any event, on the Maturity Date. Interest will accrue on any amount of
principal until such time as payment therefor is actually delivered to the
Company.  In addition, if the Employee receives any bonus from Parent or any of
Parent's subsidiaries (including the Company), then, on the date of the payment
of such bonus to the Employee, the Employee shall be obligated to pay to the
Company an amount equal to the lesser of (i) 25% of the amount of such bonus
(net of the amount of any customary withholding taxes) and (ii) the amount of
accrued and unpaid interest on this Note as of such date, and such amount paid
shall reduce the accrued interest on this Note.  The Employee acknowledges and
agrees that no provision contained herein (including the preceding sentence)
will entitle Employee to receive any bonus from Parent or any of Parent's
subsidiaries (including the Company).

                                       2
<PAGE>
 
          4.   EVENTS OF DEFAULT.
               ----------------- 

          (A)  DEFINITION. An "Event of Default" will be deemed to have occurred
               ----------      ----------------
if:

          (I)       the Employee fails to pay any amount of the principal of or
     interest on this Note as and when required pursuant to the terms hereof; or

          (II)      the Employee makes an assignment for the benefit of
     creditors or admits in writing his or her inability to pay his or her debts
     generally as they become due; or an order, judgment or decree is entered
     adjudicating the Employee bankrupt or insolvent; or any order for relief
     with respect to the Employee is entered under the Federal Bankruptcy Code;
     or the Employee commences any proceeding relating to the Employee under any
     bankruptcy reorganization, arrangement, insolvency, or readjustment of debt
     law of any jurisdiction; or any such petition or application is filed, or
     any such proceeding is commenced, against the Employee and either (A) the
     Employee by any act indicates its approval thereof, consent thereto or
     acquiescence therein or (B) such petition, application or proceeding is not
     dismissed within 30 days.

The foregoing will constitute Events of Default whatever the reason or cause for
any such Event of Default and whether it is voluntary or involuntary or is
effected by operation of law or pursuant to any judgment, decree or order of any
court or any order, rule or regulation of any administrative or governmental
body.

          (B)   CONSEQUENCES OF EVENTS OF DEFAULT.
                --------------------------------- 
 
          (I)   If an Event of Default has occurred, then the aggregate
principal amount of this Note (together with all accrued interest thereon and
all other amounts due and payable with respect thereto) will become immediately
due and payable without any action on the part of the Company.

          (II)  If any Event of Default has occurred and is continuing, the
interest rate on this Note will increase immediately by an increment of 6
percentage points (i.e., 600 basis points), to the extent permitted by
applicable law.  Any such increase of the interest rate resulting from the
operation of this Section 4(b)(ii) will terminate as of the close of business on
the next date on which no Event of Default exists (subject to subsequent
increases pursuant to this Section).

          (III) The Company will also have any other rights which the Company
may have been afforded under any contract or agreement at any time and any other
rights which the Company may have pursuant to applicable law. The Employee
hereby waives diligence, presentment, protest and demand and notice of protest
and demand, dishonor and nonpayment of this Note, and expressly agrees that this
Note, or any payment thereunder, may be extended from time to time and that the
Company may accept security for this Note or release security for this Note, all
without in any way affecting the liability of the Employee thereunder. If the
Employee fails to pay any amounts due

                                       3
<PAGE>
 
hereunder when due, then the Employee shall pay to the Company, in addition to
the amounts due, all costs of collection, including reasonable attorneys fees.

          5.   AMENDMENT AND WAIVER.   None of the terms or provisions of this
               --------------------                                           
Note may be altered, modified or amended except by an instrument in writing,
duly executed by the Company (with approval of Parent's board of directors) and
the Employee.  The Company shall not by any act, delay, omission or otherwise be
deemed to have waived any of its rights or remedies hereunder, and no waiver
shall be valid unless in writing, signed by the Company (with the approval of
Parent's board of directors), and then only to the extent therein set forth.  A
waiver by the Company of any right or remedy hereunder on any one occasion shall
not be construed as a bar to any right or remedy which the Company would
otherwise have on any future occasion.  No failure to exercise nor any delay in
exercising on the part of the Company, any right, power or privilege hereunder
shall preclude any other or further exercise thereof or the exercise of any
other right, power or privilege. The rights and remedies herein provided are
cumulative and may be exercised singly or concurrently, and are not exclusive of
any rights or remedies provided by law.

          6.   CANCELLATION.  After all principal and accrued interest at any
               ------------                                                  
time owed on this Note have been paid in full, this Note will be surrendered to
the Employee for cancellation.

          7.   PAYMENTS.  All cash payments to be made to the Company will be
               --------                                                      
made in the lawful money of the United States of America in immediately
available funds.  Payments of principal and interest in respect of this Note
will be delivered to the Company at the Company's chief executive office.

          8.   DESCRIPTIVE HEADINGS; GOVERNING LAW.  The descriptive headings of
               -----------------------------------                              
the several Sections of this Note are inserted for convenience only and do not
constitute a part of this Note.  This Note shall be governed by and construed in
accordance with the laws of the state of North Carolina, without giving effect
to any rules, principles or provisions of choice of law or conflict of laws.

          9.   BUSINESS DAYS.  If any payment is due, or any time period for
               -------------                                                
giving notice or taking action expires, on a day which is a Saturday, Sunday or
legal holiday in the State of North Carolina, then the payment will be due and
payable on, and the time period will automatically be extended to, the next
business day immediately following such Saturday, Sunday or legal holiday, and
interest will continue to accrue at the required rate under this Note until any
such payment is made.

          10.  USURY LAWS.  It is the intention of the Employee and the Company
               ----------                                                      
to conform strictly to all applicable usury laws now or hereafter in force, and
any interest payable under this Note will be subject to reduction to the amount
not in excess of the maximum legal amount allowed under the applicable usury
laws as now or hereafter construed by the courts having jurisdiction over such
matters.  If the maturity of this Note is accelerated by reason of an Event of
Default, voluntary prepayment by the Employee or otherwise, then earned interest
may never include more than the

                                       4
<PAGE>
 
maximum amount permitted by law, computed from the date hereof until payment,
and any interest in excess of the maximum amount permitted by law will be
canceled automatically and, if theretofore paid, will at the option of the
Company either be rebated to the Employee or credited on the principal amount of
this Note, or if this Note has been paid, then the excess will be rebated to the
Employee. The aggregate of all interest (whether designated as interest, service
charges, points or otherwise) contracted for, chargeable, or receivable under
this Note will under no circumstances exceed the maximum legal rate upon the
unpaid principal balance of this Note remaining unpaid from time to time. If
such interest does exceed the maximum legal rate, it will be deemed a mistake
and such excess will be canceled automatically and, if theretofore paid, rebated
to the Employee or credited on the principal amount of this Note, or if this
Note has been repaid, then such excess will be rebated to the Employee.

          11.  SEVERABILITY.  If any provision of this Note is held by any court
               ------------                                                     
of competent jurisdiction to be illegal, void or unenforceable, such provision
will be of no force and effect, but such holding shall have no effect upon the
enforceability of any other provision.

          12.  GENERAL.  This Note:
               -------             

          (a)  constitutes the entire agreement among the parties with respect
to the subject matter hereof;

          (b)  supersedes any and all prior understandings relating to such
subject matter; and

          (c)  will be binding upon and inure to the benefit of the parties and
their respective heirs, executors, administrators, successors and assigns.

          13.  WAIVER OF JURY TRIAL.  THE EMPLOYEE (AND, BY ITS ACCEPTANCE OF
               --------------------                                          
THIS NOTE, THE COMPANY) HEREBY WAIVES, TO THE EXTENT PERMITTED BY APPLICABLE
LAW, TRIAL BY JURY IN ANY LITIGATION IN ANY COURT WITH RESPECT TO, IN CONNECTION
WITH, OR ARISING OUT OF THIS NOTE OR THE VALIDITY, PROTECTION, INTERPRETATION,
COLLECTION OR ENFORCEMENT THEREOF.

                                   * * * * *

                                       5
<PAGE>
 
    IN WITNESS WHEREOF, the Employee has executed and delivered this Recourse
Promissory Note as of the date specified above.



                                    ______________________________
                                    David McIlquham