Exhibit 10.4

[EXECUTION COPY]

TRADEMARK SECURITY AGREEMENT

This TRADEMARK SECURITY AGREEMENT, dated as of April 23, 2008 (this
“Agreement”), is made by ANNCO, INC., a Delaware corporation (“Grantor”), in
favor of BANK OF AMERICA, N.A., in its capacity as administrative agent for each
of the Lenders now or hereafter party to the Credit Agreement (the “Agent”).

WITNESSETH:

WHEREAS, ANNTAYLOR, INC., a Delaware corporation (“ATI”), Grantor, ANNTAYLOR
DISTRIBUTION SERVICES, INC., a Delaware corporation (“ATDS”), ANNTAYLOR RETAIL,
INC., a Delaware corporation (“ATR”, and together with ATI, Grantor and ATDS,
the “Borrowers” and each, individually, a “Borrower”), the Agent, the lenders
referred to therein, the syndication agents named therein, and the issuing banks
named therein have entered into that certain $175,000,000 Second Amended and
Restated Credit Agreement dated November 14, 2003 (the “Original Credit
Agreement”); and

WHEREAS, at the Borrowers’ request, Lenders, the Agent, and Banc of America
Securities LLC, as Lead Arranger (in such capacity, the “Arranger”) have agreed
to amend and restate the Original Credit Agreement in its entirety pursuant to
that certain Third Amended and Restated Credit Agreement dated as of the date
hereof (as so amended and restated and as such agreement may be further amended,
supplemented, modified, or amended and restated from time to time, the “Credit
Agreement”); and

WHEREAS, each of Grantor, ATDS and ATR is, directly or indirectly, a wholly
owned Subsidiary of ATI and has and will materially benefit from the Loans and
Advances made and to be made, and the Letters of Credit issued and to be issued,
under the Credit Agreement; and

WHEREAS, the Borrowers, AnnTaylor Stores Corporation, a Delaware corporation,
and the Agent have entered into that certain Third Amended and Restated Pledge
and Security Agreement, dated as of the date hereof (as so amended and restated
and as such agreement may be further amended, supplemented, modified, or amended
and restated from time to time, the “Security Agreement”); and

WHEREAS, as collateral security for payment and performance by the Grantor of
its Obligations, the Grantor is willing to continue, amend and grant to the
Agent, for the benefit of the Agent and the Lenders, as the case may be, a
security interest in certain of its personal property and assets pursuant to the
terms of this Agreement, including, without limitation, the Collateral referred
to in Section 2 below;

NOW, THEREFORE, in consideration of the above premises and in order to induce
the Lenders and the Letter of Credit Issuer to amend and restate the Original
Credit Agreement and continue to, respectively, make Loans and issue Letters of
Credit under the Credit Agreement, Grantor hereby agrees with the Agent for its
benefit, and for the benefit of the Lenders and the Letter of Credit Issuer, by
acceptance hereof, as follows:

1. DEFINED TERMS. All capitalized terms used but not otherwise defined in this
Agreement have the meanings given to them in the Security Agreement. All other
undefined terms contained in this Agreement, unless the context indicates
otherwise, have the meanings provided for by the UCC to the extent the same are
used or defined therein.

 

Trademark Security Agreement

--------------------------------------------------------------------------------

2. GRANT OF SECURITY. As security for all Obligations, Grantor hereby pledges,
assigns, charges, mortgages, delivers, transfers and grants to the Agent, for
the benefit of the Agent and the Lenders, a continuing security interest in,
lien on, and right of set-off against, all of the following property and assets
of such Grantor, whether now owned or existing or hereafter acquired or arising,
regardless of where located (the “Collateral”):

(a) all of each Grantor’s right, title and interest, whether now owned or
hereafter acquired, in and to all United States and foreign trademarks, trade
names, trade dress, service marks, trademark and service mark registrations, and
applications for trademark or service mark registration and any renewals thereof
(including without limitation each trademark and service mark registration and
application for registration identified in Schedule A attached hereto and
incorporated herein by reference) and including all income, royalties, damages
and payments now and hereafter due and/or payable with respect thereto
(including without limitation damages for past or future infringements thereof),
the right to recover for all past, present and future infringements thereof, all
rights corresponding thereto throughout the world (but only such rights as now
exist or may come to exist under applicable local law) and all other rights of
any kind whatsoever of each Grantor accruing thereunder or pertaining thereto,
together in each case with the goodwill of the business connected with the use
of, and symbolized by, each trademark and service mark (collectively, the
“Trademarks”); and

(b) any and all Proceeds of the foregoing.

3. RECORDATION. Grantor authorizes and requests that the Commissioner of Patents
and Trademarks and any other applicable government officer record this
Agreement.

4. EXECUTION IN COUNTERPARTS. This Agreement may be executed in any number of
counterparts, each of which when so executed shall be deemed to be an original
and all of which taken together shall constitute one and the same agreement.

5. GRANTS, RIGHTS AND REMEDIES. This Agreement has been entered into in
conjunction with the provisions of the Security Agreement. Grantor does hereby
acknowledge and confirm that the grant of the security interest hereunder to,
and the rights and remedies of, the Agent with respect to the Collateral are
more fully set forth in the Security Agreement, the terms and provisions of
which are incorporated herein by reference as if fully set forth herein.

6. GOVERNING LAW. This Agreement shall be governed by, and construed in
accordance with, the laws of the State of New York.

 

Trademark Security Agreement

- 2 -

--------------------------------------------------------------------------------

IN WITNESS WHEREOF, the parties hereto have executed this Agreement by their
duly authorized officers as of the day and year first above written.

 

ANNCO, INC.     Address for notices to Grantor: (“Grantor”)          

Annco, Inc.

By:

 

Samir Patel

   

c/o AnnTaylor, Inc.

Name:

 

Samir Patel

   

7 Times Square

Title:   Assistant Treasurer     New York, NY 10036       Attention:   General
Counsel       Telephone:         Facsimile:        

With copies to:

     

Annco, Inc.

     

c/o AnnTaylor, Inc.

     

7 Times Square

     

New York, NY 10036

      Attention:   Vice President and Controller      

and:

     

Annco, Inc.

     

c/o Ann Taylor, Inc.

     

7 Times Square

     

New York, NY 10036

      Attention:   Senior Vice President – Chief         Financial Officer BANK
OF AMERICA, N.A.,     Address for notices to Agent: as Agent       (“Agent”)    

Bank of America, N.A.

     

335 Madison Avenue

By:  

Andrew Cerussi

   

New York, New York 10017

Name:   Andrew Cerussi     Attention:   Business Credit – Account Title:   Vice
President       Executive and Legal         Department       Telephone:        
Facsimile:  

 

Signature Page

Trademark Security Agreement

--------------------------------------------------------------------------------

STATE OF                                                              

COUNTY OF                                                              

I,                                                              , a Notary
Public for said County and State, do hereby certify that
                                                              personally came
before me this day and acknowledged that (s)he is
                                         of                             , a
                             , and acknowledged, on behalf of
                                                             , the due execution
of the foregoing instrument.

Witness my hand and official seal, this the              day of
                    ,             .

(Official Seal)

 

  

 

   Notary Public

My Commission expires                                         
                    

 

Signature Page

Trademark Security Agreement