<PAGE>

                                                                    Exhibit 10.9


                                 PROMISSORY NOTE

$4,336,445                                                    New York, New York
                                                              January 1, 2000

        FOR VALUE RECEIVED, the undersigned, Frederick J. Rowan II (the
"Payor"), hereby promises to pay to the order of The William Carter Company, or
its successor or assign (the "Holder"), at the office of The William Carter
Company, 1590 Adamson Parkway, Suite 400, Morrow, Georgia 30260, Attn: Michael
D. Casey, in lawful money of the United States of America in immediately
available funds, the principal sum of FOUR MILLION THREE HUNDRED THIRTY-SIX
THOUSAND FOUR HUNDRED FORTY-FIVE DOLLARS ($4,336,445), together with any accrued
and unpaid interest on such principal amount from time to time outstanding,
which sum shall be due and payable in installments of $600,000 (or such lesser
amount as remains outstanding) commencing on March 31, 2002 and thereafter on
each anniversary thereof until such principal amount and all accrued and unpaid
interest thereon has been repaid.

        From the date hereof, interest (i) shall accrue on the outstanding
principal amount hereof at a rate which shall be calculated for each quarter of
part thereof of the Holder's fiscal year, and (ii) shall equal the average rate
paid by The William Carter Company under the revolving portion of its senior
credit facility during such period (or if there are no borrowings outstanding
during such period, the average rate that would have been paid thereunder), and
(iii) shall be calculated on the basis of the actual number of days elapsed over
365, from the date hereof to the date of payment.

        The Payor hereby promises to pay interest, on demand, on any overdue
principal and, to the extent permitted by law, on any overdue interest, and on
any overdue amount under any instrument now or hereafter evidencing or securing
the indebtedness evidenced hereby, at a rate equal to the rate of interest set
forth above, plus 2%, calculated on the basis of the actual number of days
elapsed over 365, from the date such principal or interest was due to the date
of payment.

        The Payor shall have the right at any time and from time to time on any
business day to prepay the principal amount of this Note, together with accrued
and unpaid interest thereon, in whole or in part, without penalty or premium,
upon at least two business days' prior written notice to the Holder hereof, such
notice to specify the prepayment date and the principal amount hereof to be
prepaid. In the event the Payor decides not to so prepay this Note in accordance
with any such notice delivered to the Holder hereof, the Payor shall so notify
the Holder hereof, not less than one business day before such prepayment would
otherwise have been made.

        All amounts paid or prepaid by Payor hereunder shall be applied first to
the accrued and unpaid interest, and then to reduce the outstanding principal
amount under this Note.

        All payment and prepayment of the principal hereof and interest hereon
and the respective dates thereof shall be endorsed by the Holder hereof on the
schedule attached hereto and made a part hereof, or on a continuation thereof
which shall be attached hereto and made a part hereof; PROVIDED, HOWEVER, that
the failure of the Holder hereof to make such a notation or any error in such a
notation shall not affect the obligations of the Payor under this Note.


<PAGE>

        This Promissory Note is secured by that certain Stock Pledge Agreement,
dated as of even date herewith, between the Holder and the Payor (the "Stock
Pledge Agreement").

        Notwithstanding anything to the contrary contained herein, in case of
the happening of any of the following events ("Events of Default"):

               (i) default shall be made in the payment of the principal of or
        interest on this Note when and as the same shall become due and payable,
        whether at the due date thereof or at a date fixed for prepayment
        thereof or otherwise;

               (ii) a breach of any covenant contained in this Note, other than
        the covenant to pay the principal of and interest on this Note, which
        breach shall continue unremedied for thirty days after written notice by
        the Holder;

               (iii) the Payor shall (a) voluntarily commence any proceeding or
        file any petition seeking relief under Title 11 of the United States
        Code or any other Federal or state bankruptcy, insolvency, liquidation
        or similar law, (b) consent to the institution of, or fail to controvert
        in a timely and appropriate manner, any such proceeding or the filing of
        any such petition, (c) apply for or consent to the appointment of a
        receiver, trustee, custodian, sequestrator or similar official for him
        or for a substantial part of his property, (d) file an answer admitting
        the material allegations of a petition filed against him in any such
        proceeding, (e) make a general assignment for the benefit of creditors
        or (f) become unable, admit in writing his inability or fail generally
        to pay his debts as they become due;

               (iv) an involuntary proceeding shall be commenced or an
        involuntary petition shall be filed in a court of competent jurisdiction
        seeking (a) relief in respect of the Payor, or of a substantial part of
        the property of the Payor, under Title 11 of the United States Code or
        any other Federal or state bankruptcy, insolvency, receivership or
        similar law or (b) the appointment of a receiver, trustee, Custodian,
        sequestrator or similar official for the Payor or for a substantial part
        of the property of the Payor; and such proceeding or petition shall
        continue undismissed for 60 days or an order or decree approving or
        ordering any of the foregoing shall continue unstayed and in effect for
        60 days;

               (v) an Approved Sale or an Initial Public Offering (each as
        defined in the Stock Acquisition Agreement, dated as of October 30, 1996
        between Carter Holdings, Inc. and the Payor (the "Stock Acquisition
        Agreement")) of Carter Holdings, Inc., a Massachusetts corporation, or
        The William Carter Company, a Massachusetts corporation, shall be
        consummated;

               (vi) an Event of Default (as defined in the Stock Pledge
        Agreement) shall have occurred under the Stock Pledge Agreement;

               (vii) Payor shall, for any reason, cease to be an employee of
        Holder and the date set for repurchase of the Pledged Shares (as defined
        in the Stock Pledge Agreement) pursuant to Section 4(a), 4(b) or 4(c) of
        the Stock Acquisition Agreement, as the case may be, shall have passed;




                                       2
<PAGE>


then, in any such event (other than an event described in paragraph (iii) or
(iv) above), the Holder hereof may declare the principal amount of this Note
then outstanding to be forthwith due and payable, whereupon the principal
hereof, together with accrued and unpaid interest thereon, shall become
forthwith due and payable both as to principal and interest, without
presentment, demand, protest or any other notice of any kind, all of which are
hereby expressly waived by the Payor (to the extent permitted by law), anything
contained herein to the contrary notwithstanding; and, in any event described in
paragraph (iii) or (iv) above, the principal amount of this Note, together with
accrued and unpaid interest thereon, shall automatically become due and payable
without presentment, demand, protest or other notice of any kind, all of which
are hereby expressly waived by the Payor.

        If any payment under this Note is not made when due, whether at maturity
or by acceleration, Payor shall pay all costs of collection whether or not suit
is filed hereon, on the Stock Pledge Agreement or otherwise, including, but not
limited to, reasonable attorneys' fees and all expenses incurred in connection
with the protection or realization of any collateral.

        Payor hereby certifies and declares that all acts, conditions and things
required to be done and performed and to have happened precedent to the creation
and issuance of this Note, and to constitute this Note the legal, valid and
binding obligation of Payor, enforceable in accordance with the terms hereof,
have been done and performed and happened in due and strict compliance with all
applicable laws.

        The Payor hereby waives (to the extent permitted by law) diligence,
presentment, demand, protest and notice of any kind whatsoever except as
expressly required herein. The nonexercise by the Holder of any of its rights
hereunder or under the Stock Pledge Agreement in any particular instance shall
not constitute a waiver thereof in that or any subsequent instance. The Holder
shall at all times have the right to proceed against any portion of the security
held herefor in such order and in such manner as the Holder may select, without
waiving any rights with respect to any other security. No delay or omission on
the part of the Holder in exercising any right hereunder or under the Stock
Pledge Agreement or other agreement shall operate as a waiver of such right or
of any other right under this Note.

        This Note may be assigned, pledged, hypothecated or otherwise
transferred by the Holder hereof.

        Any notice to be given hereunder shall be in writing and shall be deemed
to have been given (i) when presented personally or (ii) three business days
after being deposited in a regularly maintained receptacle for the United States
Postal Service, postage prepaid, registered or certified, return receipt
requested addressed to the respective party at the address specified herein or
such other address as any party may from time to time designate by written
notice to the other as herein required.

        The Payor irrevocably submits to the jurisdiction of (a) the Supreme
Court of the State of New York, New York County, and (b) the United States
District Court for the Southern District of New York, for the purposes of any
suit, action or other proceeding arising out of or relating to this Note.
Pledgor expressly waives any objection which he may have now or hereafter to the
laying of the venue or to the jurisdiction of any such suit, action or
proceedings.



                                       3
<PAGE>

        All agreements between Payor and the Holder hereof are expressly limited
so that in no contingency or event whatsoever, whether by reason of advancement
of the proceeds of the loan evidenced hereby, acceleration of maturity of the
unpaid principal balance hereof, or otherwise, shall the interest contracted
for, charged or received by the Holder exceed the maximum amount permissible
under applicable law. If, from any circumstance whatsoever, interest would
otherwise be payable to the Holder hereof in excess of the maximum lawful
amount, then IPSO FACTO, the interest payable to the Holder shall be reduced to
the maximum amount permitted under applicable law, and the amount of interest
for any subsequent period, to the extent less than that permitted by applicable
law, shall to that extent be increased by the amount of such reduction. If from
any circumstance the Holder hereof shall ever receive anything of value deemed
interest by applicable law in excess of the maximum lawful amount, an amount
equal to any excessive interest shall be applied to the reduction of the unpaid
principal balance due hereunder and not to the payment of interest, or if such
excessive interest exceeds the unpaid principal balance due hereunder, such
excess shall be refunded to Payor. All interest paid or agreed to be paid to the
Holder hereof shall, to the extent permitted by applicable law, be amortized,
prorated, allocated or spread throughout the full period until payment in full
of the principal balance due hereunder (including the period of any renewal or
extension) so that interest thereon for such period shall not exceed the maximum
amount permitted by applicable law. The provisions of this paragraph shall
control all agreements between the Payor and the Holder hereof.

        Any provision of this Note that is declared invalid, illegal or
unenforceable in any jurisdiction shall not affect in any way the remaining
provisions hereof in such jurisdiction or render that or any other provision of
this Note invalid, illegal or unenforceable in any other jurisdiction.

        This Note has been executed and delivered in the State of New York and
shall be governed and construed, and all rights and obligation hereunder shall
be determined, in accordance with the laws of the State of New York.

        Executed and delivered by the undersigned as of this 1st day of January,
2000 in the County, City and State of New York.



          Address for notices                 Payor
          to the Payor:

                                              /s/ FREDERICK J. ROWAN, II
------------------------------------------    --------------------------------
                                               Frederick J. Rowan, II

------------------------------------------


<PAGE>




                      SCHEDULE OF PAYMENTS AND PREPAYMENTS