<PAGE>

                                                                 Exhibit 10.5(b)
                                                                 ---------------


                                            August 9, 1999

Mr. Mark Sarvary
Chief Executive Officer
J. Crew Group, Inc.
770 Broadway
New York, NY 10003


Dear Mark:

          Pursuant to our recent discussions regarding the loan which J. Crew
Operating Corp. (the "Company") will make to you in accordance with Section 2(g)
                      -------
of the Employment Agreement dated as of May 3, 1999, among the Company, J. Crew
Group, Inc. and you, we thought it would be useful to lay out certain material
terms and conditions of such loan in this letter for both parties to sign.

I.  The Company Loan

          Immediately prior to the closing of the purchase of your proposed
primary residence (the "Closing"), the Company will lend you one million dollars
                        -------
($1,000,000) (the "Company Loan") to be used by you for the sole purpose of
                   ------------
purchasing your primary residence, located at 7 Fox Run, Purchase, New York (the
"Property").  It is understood that the funding of the Company Loan will take
 --------
place immediately prior to the Closing and will be initially secured by your
pledge to the Company of certain stock options (the "Stock Options") pursuant to
                                                     -------------
the Pledge and Security Agreement to be executed by you, a copy of which is
attached heret as Exhibit B.  Immediately following the Closing, you will grant
the Company a second mortgage on the Property to secure the Company Loan by
executing a second mortgage agreement substantially in the form attached hereto
as Exhibit C (the "Second Mortgage"), and the Company will release its security
                   ---------------
interest in the Stock Options.

II.  Terms and Conditions of the Company Loan

          A.  The obligation of the Company to make the Company Loan is subject
to your providing evidence satisfactory to the Company on the date of the
Closing that:

          (1)  the sum of (x) the Company Loan and (y) the loan to be obtained
     by you from Citibank, N.A. in connection with the purchase of the Property,
     which loan will be secured by a first mortgage on the Property (the "First
                                                                          -----
     Mortgage Loan"), does not exceed ninety percent (90%) of the fair market
     -------------
     value of the Property as appraised by Citibank, N.A. in connection with the
     First Mortgage Loan (a copy of such appraisal to be provided to the Company
     at the Closing);
<PAGE>

          (2)  there are no liens on the Property other than the lien created by
     the First Mortgage Loan (or other items acceptable to the Company
     identified on the Citibank, N.A. lender's title insurance policy, a copy of
     which you will provide to the Company); and

          (3)  The terms and conditions of the First Mortgage Loan do not
     prohibit the Second Mortgage.

          B.  As consideration for the Company making the Company Loan, you
agree that:

          (1) Contemporaneously with the making of the Company Loan, you will
     execute the Promissory Note and the Pledge and Security Agreement,
     substantially in the forms attached hereto as Exhibit A and Exhibit B,
     respectively;

          (2)  You will execute the Second Mortgage in the form attached hereto
     as Exhibit C immediately following the Closing and the Company shall record
     the same in the applicable recording office;

          (3)  You will not grant any liens on the Property without the prior
     written consent of the Company (other than the lien securing the First
     Mortgage Loan); and

          (4)  In the event the Closing does not occur on the date the Company
     makes the Company Loan to you, you will immediately repay the Company Loan
     to the Company.

          If the foregoing correctly reflects your understanding, please sign
the enclosed copy of this letter and return it to me.

                                              Sincerely,


                                              Emily Woods

Agreed to and accepted:


___________________________
Mark Sarvary
Date:
<PAGE>

                                PROMISSORY NOTE

$1,000,000.00                                                    New York, NY
                                                                 August __, 1999

     Mark Sarvary ("Sarvary"), for value received, hereby promises to pay to the
order of J.Crew Operating Corp., a Delaware corporation ("J.Crew"), at its
offices located at 770 Broadway, New York, NY, 10003, or such other place as the
holder hereof may designate by notice to Sarvary, the principal amount of One
Million Dollars ($1,000,000.00), in lawful money of the United States, without
interest, as follows:  (i) $50,000.00 on April 15th in each of the years 2000,
2001, 2002 and 2003; and (ii) the remaining balance of $800,000.00 on June 29,
2004.  Sarvary hereby grants J.Crew the right to deduct all or a portion of the
principal payments when due from any bonus payable to him.  Sarvary acknowledges
and confirms that (i) J.Crew has loaned Sarvary the principal amount of the Note
for the sole purpose of Sarvary purchasing a primary residence located at 7 Fox
Run, Purchase, New York (the "Property") and (ii) he will use the proceeds of
the Note solely for such purpose.

1.   Prepayment
     ----------

     This Note may be prepaid at any time, in whole or in part, without penalty
or premium.  Each partial prepayment shall be applied to installments of
principal in inverse order of maturity.

2.   Events of Acceleration
     ----------------------

     The holder of this Note, by written notice to Sarvary, may declare the
entire principal amount immediately due and payable if any of the following
events ("Accelerated Events") shall have occurred and be continuing, in which
event the maturity of the then unpaid balance of the Note shall be accelerated
and shall become immediately due and payable, without presentment, demand,
protest or notice of any kind, all of which are hereby expressly waived in
accordance with Paragraph 3(f) hereunder.

     (a) Sarvary shall not have paid any installment of principal on this Note
as and when it has become due and payable and such default shall continue for a
period of 10 days after notice to Sarvary;

     (b) Sarvary's employment with J.Crew is terminated for any reason; or

     (c) Sarvary is in default under any other agreement with J.Crew.

     3.  Miscellaneous
         -------------

     (a) Sarvary shall pay all costs and expenses incurred by the holder in
connection with the collection of the Note, including reasonable attorneys'
fees.

     (b) This Note shall be governed by and construed in accordance with the
laws of New York State applicable to agreements made and to be performed therein
and cannot be changed orally. Sarvary irrevocably consents to the sole and
exclusive jurisdiction of the courts of New
<PAGE>

York State and of any federal court located in New York State in connection with
any action or proceeding arising out of or related to this Note.

     (c) No delay or failure on the part of the holder of this Note to exercise
any power or right given under this Notice, including, but not limited to, the
right to accelerate the amounts due, shall operate as a waiver of the power or
right and no right or remedy of the holder shall be deemed abridged or modified
by any course of conduct. All rights and remedies existing hereunder are
cumulative and not exclusive of each other or any rights or remedies otherwise
available.

     (d) All notices and other communications hereunder shall be in writing and
shall be deemed given when delivered personally, three days after being mailed
by registered mail, return receipt requested, or the following day if sent by
overnight courier service, to J.Crew at the address set forth at the beginning
of this Note, attn: General Counsel, and to Sarvary at 770 Broadway, New York,
NY 10003, or such other address as either party may specify by notice given
pursuant hereto.

     (e) To the extent permitted by applicable law, Sarvary hereby waives all
benefit that might accrue by virtue of any present or future moratorium laws
exempting any of the Property, or any other property, real or personal, or any
part of the proceeds arising from any sale of any such property, from
attachment, levy, or sale under execution, or providing for any stay of
execution to be issued on any judgment recovered on this Note (excepting only
any stay of execution).

     (f) SARVARY HEREBY WAIVES PRESENTMENT, DEMAND, DILIGENCE, PROTEST AND
NOTICE OF PROTEST, NOTICE OF INTENT TO ACCELERATE, NOTICE OF ACCELERATION,
DEMAND, DISHONOR AND NON-PAYMENT OF THIS NOTE OR ANY OTHER NOTICE OF ANY KIND
WHATSOEVER.

     (g) If any term or provision of this Note or the application thereof to any
circumstance shall, to any extent, be invalid, illegal or unenforceable, such
term or such provisions shall be ineffective to the extent of such invalidity,
illegality or unenforceability without invalidating or rendering unenforceable
any remaining terms and provisions hereof or thereof or the application of such
term or provision to circumstances other than those as to which it is held
invalid, illegal or unenforceable.

     (h) This Note shall not be transferable, except that J.Crew may transfer
the Note to any other person or entity without Sarvary's consent.



                                  ___________________________
                                          Mark Sarvary


                                       2
<PAGE>

Exhibit "B"
-----------

                         PLEDGE AND SECURITY AGREEMENT

          This PLEDGE AND SECURITY AGREEMENT (as amended, restated, replaced,
supplemented or otherwise modified from time to time, this "Agreement") is dated
                                                            ---------
as of August __, 1999 and entered into by and between Mark Sarvary, in his
individual capacity, ("Grantor") and J.Crew Operating Corp., (together with it
                       -------
successors and assigns, "Secured Party").
                         -------------

                             PRELIMINARY STATEMENTS

          WHEREAS, Secured Party has agreed to loan Grantor the principal amount
of $1,000,000 (the "Loan") in accordance with that certain Promissory Note made
                    ----
by Grantor, dated as of the date hereof (as the same may be amended, restated,
supplemented or otherwise modified from time to time, the "Note") and in
                                                           ----
accordance with that certain letter agreement between the parties, dated August
9, 1999 (the "Letter Agreement") in order to finance Grantor's acquisition of a
              ----------------
certain property located at 7 Fox Run, Purchase, NY 10577 (the "Property");
                                                                --------

          WHEREAS, Grantor is the legal and beneficial owner of those certain
options to purchase 272,000 shares of the common stock of J.Crew Group, Inc.
awarded to him under the J.Crew Group, Inc. 1997 Stock Option Plan (such options
and such plan, the "Pledged Options" and the "Option Plan", respectively) and
                    ---------------           -----------
evidenced by that certain Stock Option Agreement dated as of June 28, 1999 (the
"Option Agreement");
 ----------------

          WHEREAS, it is a condition precedent to the making of the Loan by
Secured Party that Grantor shall have granted the security interests and
undertaken the obligations contemplated by this Agreement; and

          WHEREAS, it is intended that the security interest granted hereunder
secure the Loan until the closing of Grantor's purchase of the Property, at
which time Grantor shall grant to Secured Party a second mortgage on the
Property, substantially in the form agreed to in the Letter Agreement (the
"Second Mortgage"), after which this Agreement shall terminate.
----------------

          NOW, THEREFORE, in consideration of the premises and in order to
induce Secured Party to make the Loan and for other good and valuable
consideration, the receipt and adequacy of which are hereby conclusively
acknowledged, Grantor hereby agrees with Secured Party as follows:

          SECTION 1.  Grant and Pledge of Security.  Grantor hereby assigns and
                      ----------------------------
pledges to Secured Party, and hereby grants to Secured Party a security interest
in, all of Grantor's right, title and interest in and to the following, whether
now or hereafter acquired (the "Pledged Collateral") the Option Agreement,
                                ------------------
Pledged Options, and any interest of the Grantor in the entries on the books of
J.Crew Group, Inc. or any financial intermediary pertaining to the Pledged
Options, and all dividends, cash, warrants, rights, instruments and other
property or proceeds from time to time received, receivable or otherwise
distributable in respect of or in exchange for any or all of the Pledged
Options.  The Grantor shall deliver the original Option
<PAGE>

Agreement to the Secured Party and such Option Agreement shall remain in the
possession of the Secured Party until this Agreement is terminated, at which
time, the Secured Party shall return the Option Agreement to the Grantor.

          SECTION 2.  Security for Obligations.  This Agreement secures, and the
                      ------------------------
Pledged Collateral is collateral security for, all obligations of every nature
of the Grantor now or hereafter existing under the Note and the Letter Agreement
(all such obligations collectively, the "Secured Obligations").
                                         -------------------

          SECTION 3.  No Assumption.  Notwithstanding any of the foregoing, this
                      -------------
Agreement shall not in any way be deemed to obligate Secured Party to assume any
of Grantor's obligations, duties, expenses or liabilities now existing or
hereafter drafted or executed (collectively, the "Grantor Obligations") unless
                                                  ------- -----------
Secured Party or any such purchaser otherwise expressly agrees to assume any or
all of such Grantor Obligations in writing.

          SECTION 4.  Further Assurances and Covenants of Grantor.  Grantor
                      -------------------------------------------
agrees that from time to time, at the expense of Grantor, Grantor will promptly
execute and deliver all further instruments and documents, and take all further
action, that may be necessary or desirable, or that Secured Party may reasonable
request, in order to perfect and protect any security interest granted or
purported to be granted hereby or to enable Secured Party to exercise and
enforce its rights and remedies hereunder with respect to any Pledged
Collateral.  Grantor shall not, without the prior written consent of Secured
Party, which may be granted or withheld in Secured Party's sole discretion,
sell, assign (by operation of law or otherwise), pledge or otherwise dispose of
or hypothecate all or any part of the Pledged Collateral.

          SECTION 5.  Acceleration Event.  In the case of an Acceleration Event,
                      ------------------
as defined in the Note, in addition to all of Secured Party's other rights and
remedies, Secured Party shall have the right, upon five days prior notice to
Grantor, to cause J. Crew Group, Inc. to cancel all or any portion of the
Pledged Option and to apply the "Spread" (as defined below) on each of the
                                 ------
shares of Common Stock underlying the vested Pledged Option which Secured Party
elects to cancel as follows: (i) first to pay Secured Party's expenses
(including reasonable attorney's fees) in connection with collection of the
Note; (ii) second, to apply so much of the remaining Spread as may be necessary
to pay the unpaid principal of the Note; and (iii) third, to pay any remaining
amount of the spread to Grantor.  As used herein, the term "Spread" means the
difference obtained by subtracting the (a) exercise price per share underlying
the Pledged Option, from (b) the Fair Market Value (as defined in the Option
Plan) of a share of J.Crew Group, Inc. Common Stock determined as of the date on
which the Option is cancelled.

          SECTION 6.  Substitution of Pledged Collateral.  In accordance with
                      ----------------------------------
the Letter Agreement, upon Grantor's execution and grant of the Second Mortgage
on the Property and the recordation of such Second Mortgage, the Pledged
Collateral hereunder shall be released to the Grantor and this Agreement shall
terminate.

          SECTION 7.  Continuing Security Interest; Transfer of Loan.  This
                      ----------------------------------------------
Agreement shall create a continuing security interest in the Pledged Collateral
and shall (a) remain in full force and effect until the indefeasible payment in
full of the Secured Obligations or granting and recordation of the Second
Mortgage, (b) be binding upon Grantor, its


                                       2
<PAGE>

successors and assigns, and (c) inure, together with the rights and remedies of
Secured Party hereunder, to the benefit of Secured Party and its successors,
transferees and assigns. Without limiting the generality of the foregoing clause
(c), Secured Party may assign or otherwise transfer the Note to any other
Person, and such other Person shall thereupon become vested with all the
benefits in respect thereof granted to Secured Party herein or otherwise. Upon
the indefeasible payment in full of all Secured Obligations or the granting by
Grantor of the Second Mortgage on the Property (as described above), the
security interest granted hereby shall terminate and all rights to the Pledged
Collateral shall revert to Grantor.

          SECTION 8.  Amendments; Etc.  No amendment, modification, termination
                      ---------------
or waiver of any provision of this Agreement, or consent to any departure by
Grantor herefrom, shall in any event be effective unless the same shall be in
writing and signed by Secured Party, and, in the case of any such amendment or
modification by Grantor, such waiver or consent shall be effective only in the
specific instance and for the specific purpose for which it was given.

          SECTION 9.  Notices.  Any notice or other communication herein
                      -------
required or permitted to be given hereunder shall be given in accordance with
Section 4(d) of the Note.

          SECTION 10.  Failure or Indulgence Not Waiver; Remedies Cumulative.
                       -----------------------------------------------------
No failure or delay on the part of Secured Party in the exercise of any power,
right or privilege hereunder shall impair such power, right, privilege or option
or be construed to be a waiver of any default or acquiescence therein, nor shall
any single or partial exercise of any such power, right, privilege or option
preclude any other or further exercise thereof or of any other power, right,
privilege or option.  All rights and remedies existing under this Agreement are
cumulative to, and not exclusive of, any rights or remedies otherwise available.

          SECTION 11.  Severability.  In case any provision in or obligation
                       ------------
under this Agreement shall be invalid, illegal or unenforceable in any
jurisdiction, the validity, legality and enforceability of the remaining
provisions or obligations, or of such provision or obligation in any other
jurisdiction, shall not in any way be affected or impaired thereby.

          SECTION 12.  Headings.  Section and subsection headings in this
                       --------
Agreement are included herein for convenience of reference only and shall not
constitute a part of this Agreement for any other purpose or be given any
substantive effect.

          SECTION 13.  Governing Law; Terms.  THIS AGREEMENT AND THE RIGHTS AND
                       --------------------
OBLIGATIONS OF THE PARTIES HEREUNDER SHALL BE GOVERNED BY, AND SHALL BE
CONSTRUED AND ENFORCED IN ACCORDANCE WITH, THE LAWS OF THE STATE OF NEW YORK.

          SECTION 14.  Consent to Jurisdiction and Service of Process.  The
                       ----------------------------------------------
provisions of Section 4(b) of the Note are hereby incorporated by reference in
their entirety.

          SECTION 15.  Waiver of Jury Trial.  The provisions of Section 4(f) of
                       --------------------
the Note are hereby incorporated by reference in their entirety.

          SECTION 16.  Counterparts.  This Agreement may be executed in one or
                       ------------
more counterparts and by different parties hereto in separate counterparts, each
of which when so


                                       3
<PAGE>

executed and delivered shall be deemed an original, but all such counterparts
together shall constitute but one and the same instrument; signature pages may
be detached from multiple separate counterparts and attached to a single
counterpart so that all signature pages are physically attached to the same
document.

                  [Remainder of page intentionally left blank]


                                       4
<PAGE>

          IN WITNESS WHEREOF, Grantor and Secured Party have caused this
Agreement to be duly executed and delivered by their respective officers
thereunto duly authorized as of the date first written above.

GRANTOR:                                Mark Sarvary, in his individual capacity



                                        By:
                                            ----------------------------
                                            Name:
                                            Title:

SECURED PARTY:                          J.Crew Operating Corp., a Delaware
                                                          Corporation



                                        By:
                                            ----------------------------
                                            Name:
                                            Title:


                                       5