<PAGE>
 
                                                                   EXHIBIT 10.66

                               ESCROW AGREEMENT
                               ----------------

     THIS ESCROW AGREEMENT is made and entered into as the second day of July,
1998, by and among TARRANT APPAREL GROUP, a California corporation (the
"Company"), GABRIEL MANUFACTURING COMPANY, a Mississippi corporation ("GMC") and
ROCKY APPAREL, INC., a New York corporation ("RAI"), with respect to the
following facts:

     A.   Pursuant to that certain Partnership Interest Purchase Agreement dated
as of July 2, 1998 (the "Purchase Agreement"), by and among the Company, Rocky
Acquisition, LLC, a Delaware limited liability company (the "Subsidiary"),
Limited Direct Associates, LP, a Delaware limited partnership ("Limited"), GMC
and RAI, the Subsidiary, a wholly owned subsidiary of the Company, will purchase
all the outstanding partnership interests in Rocky Apparel, LP, a Delaware
limited partnership ("Rocky"), from Limited, GMC and RAI.

     B.   In partial consideration for the purchase of such partnership
interests, the Company will issue to GMC and RAI an aggregate of 80,890 shares
of the Common Stock of the Company.

     C.   GMC and RAI have agreed, jointly and severally, to defend, indemnify
and hold harmless the Company, the Subsidiary and certain associated persons
from and against certain liabilities, all as more fully set forth in the
Purchase Agreement.

     D.   GMC and RAI each desire to grant each such indemnified person a
security interest in such shares and all distributions thereon as security for
the performance of their obligations under the Purchase Agreement.

     ACCORDINGLY, subject to the terms and conditions set forth below, and on
the basis of the premises, covenants and undertakings contained herein, the
parties hereto agree as follows:

     1.   PLEDGE.

     1.1  Pledge.  GMC and RAI each hereby deliver to the Company, and authorize
          ------                                                                
and direct the Company to hold pursuant to the terms and conditions of this
Agreement, certificates representing 80,890 shares of the issued and outstanding
Common Stock of the Company, as set forth on Schedule 1.1 hereto (the "Shares"),
                                             ------------                       
together with a stock power duly executed in blank, receipt of which hereby is
acknowledged.  The Shares and any other property which may be held by the
Company under Section 1.2 hereof shall be held by it as partial security for the
performance of the obligations of GMC and RAI under the Purchase Agreement.

                                      -1-
<PAGE>
 
     1.2  Additional Collateral.  GMC and RAI each shall deliver or cause to be
          ---------------------                                                
delivered to the Company, and hereby authorize and direct the Company to hold
pursuant to the terms and conditions of this Agreement, all distributions made
during the term hereof on the Shares, whether in the form of securities, cash or
other property, as additional collateral subject to this Agreement. The Shares
and all distributions on the Shares shall be referred to herein as the "Escrow
Fund."

     1.3  Voting Rights and Distributions.  During the term of this Agreement
          -------------------------------                                    
and as long as neither GMC nor RAI is in default under the Purchase Agreement,
GMC and RAI each shall have the right to vote the Shares and any additional
shares of the capital stock of the Company held on their behalf hereunder and to
receive all cash distributions thereon subject to the limitations set forth in
Section 1.2.

     1.4  Notice of Claims.
          ---------------- 

          (a) The Company shall give prompt written notice (a "Demand Notice")
to GMC and RAI of any claim against GMC or RAI under the Purchase Agreement by
the Company, the Subsidiary or any other person entitled to indemnification or
contribution under Section 13 of the Purchase Agreement (an "Indemnified
Person"), including, but not limited to, any claim that (i) any representation
or warranty of GMC or RAI made in the Purchase Agreement or any Ancillary
Agreement (as such term is defined in the Purchase Agreement) was not true and
complete in all respects when made, (ii) GMC or RAI have failed to timely
perform any obligation to be performed by any of them under the Purchase
Agreement or any Ancillary Agreement or (iii) the Company, the Subsidiary or any
Indemnified Person is entitled to indemnification or contribution under Section
13 of the Purchase Agreement.  The Demand Notice shall include a summary
description of the factual and legal bases for the claim and an estimate of the
amount of the claim.  GMC and RAI shall have the right to contest any claim
described in a Demand Notice by giving written notice (a "Dispute Notice") to
the Company within ten (10) calendar days of the Demand Notice.  In the event
GMC or RAI does not give a Dispute Notice within such ten (10) day period, the
description of the claim contained in the Demand Notice (including, but not
limited to, the factual and legal bases therefor and the estimate of the amount
of the claim) shall be deemed conclusively to be true and complete; provided,
                                                                    -------- 
however, that the Company shall be entitled thereafter to submit additional
Demand Notices pursuant to this Section 1.4 with respect to the same claims as
were described in such initial Demand Notice and GMC and RAI shall have the
right to contest any such additional demand Notice, all as set forth above.

          (b) In the event GMC or RAI shall timely deliver a Dispute Notice with
respect to any claim (other than a claim based upon the demand of a person other
than an Indemnified Person (a "third party claim")) and such claim shall not
have been conclusively resolved on or before July 31, 2001, the parties shall
attempt to resolve the dispute through mediation.  In the event such mediation
shall not resolve the claim, either the Indemnified Party or GMC or RAI may
require binding arbitration of the claim pursuant to Section 2.13 of this
Agreement.

                                      -2-
<PAGE>
 
     1.5  Distribution of Escrow Fund.
          --------------------------- 

          (a) The Company shall distribute to GMC and RAI their respective
shares of the Escrow Fund in three equal installments on June 30, 1999, 2000 and
2001 (a "Distribution Date"). Notwithstanding the foregoing, any portion of the
Escrow Fund to be distributed to GMC or RAI on a Distribution Date shall be
reduced by (i) that portion of the Escrow Fund which previously had been
distributed pursuant hereto to an Indemnified Party, and (ii) that portion of
such distribution which has a value, determined pursuant to Section 1.6, equal
as nearly as practicable to the aggregate claims set forth in any unresolved
Demand Notice.  Upon the later to occur of July 31, 2001 and the resolution of
all claims (including third party claims) set forth in all Demand Notices
delivered hereunder on or prior to June 30, 2001, the Trustee shall distribute
to GMC and RAI their respective shares of the Escrow Fund as the same has been
reduced by that portion of the Escrow Fund which previously had been distributed
to an Indemnified Party.

          (b) In the event GMC or RAI shall fail to timely deliver a Dispute
Notice with respect to any claim set forth in a Demand Notice (other than a
third party claim), or such claim is resolved by the agreement of the parties or
by arbitration as set forth in Section 2.13, the Company, promptly after the
last day on which such Dispute Notice could have been timely given or after
receipt by the Company of a written notice of such agreement signed by GMC and
RAI or of such arbitration decision, shall deliver to the Indemnified Party, on
the one hand, or to GMC and RAI, on the other hand, that portion of the Escrow
Fund which has a value, determined pursuant to Section 1.6, equal as nearly as
practicable to the value of the claim as set forth in the Demand Notice, if a
Dispute Notice with respect thereto was not timely delivered, or as determined
by the agreement of the parties or the arbitration, as the case may be.

          (c) In the event a third party claim set forth in a Demand Notice is
finally determined by a decision from which no appeal may be taken, the Company
shall distribute to the Indemnified Party, on the one hand, or to GMC and RAI,
on the other hand, that portion of the Escrow Fund which has a value, determined
pursuant to Section 1.6, equal as nearly as practicable to the value of the
claim as so finally determined.

     1.6  Valuation of Escrow Fund.  The fair market value of a share of the
          ------------------------                                          
Common Stock of the Company shall be determined for purposes of this Agreement
by reference to the closing price on the principal stock exchange on which such
shares are then listed, or, if such shares are not then listed on a stock
exchange, by reference to the closing price (if approved for quotation on the
Nasdaq National Market) or the mean between the bid and the asked price (if
other over-the-counter issue) of a share as supplied by the National Association
of Securities Dealers, Inc. through Nasdaq (or its successor in function), in
each case as reported by The Wall Street Journal, for the business day
                         -----------------------                      
immediately preceding the date on which such determination of value shall be
made (or, if for any reason no such price is available, in such other manner as
the Company may deem appropriate to reflect the then fair market value thereof).

                                      -3-
<PAGE>
 
  2. MISCELLANEOUS.

     2.1  Notices.  Any notice or other communication required or permitted
          -------                                                          
hereunder shall be in writing and shall be deemed to have been given (i) if
personally delivered, when so delivered, (ii) if mailed, one (1) week after
being placed in the United States mail, registered or certified, postage
prepaid, addressed to the party to whom it is directed at the address set forth
on the signature page hereof, or (iii) if given by telecopier, when such notice
or communication is transmitted to the telecopier number set forth on the
signature page hereof and written confirmation of receipt is received.  Each of
the parties shall be entitled to specify a different address by giving the other
parties notice as aforesaid.

     2.2  Entire Agreement.  This Agreement constitutes the entire agreement
          ----------------                                                  
among the parties hereto pertaining to the subject matter hereof and supersedes
all prior agreements, understandings, negotiations and discussions, whether oral
or written, relating to the subject matter of this Agreement.  No supplement,
modification, waiver or termination of this Agreement shall be valid unless
executed by the party to be bound thereby.  No waiver of any of the provisions
of this Agreement shall be deemed or shall constitute a waiver of any other
provision hereof (whether or not similar), nor shall such waiver constitute a
continuing waiver, unless otherwise expressly provided.

     2.3  Headings.  Section and subsection headings are not to be considered
          --------                                                           
part of this Agreement and are included solely for convenience and reference and
are not intended to be full or accurate descriptions of the content thereof.

     2.4  Successors and Assigns.  All of the terms, provisions and obligations
          ----------------------                                               
of this Agreement shall inure to the benefit of and shall be binding upon the
parties hereto and their respective heirs, representatives, successors and
assigns.

     2.5  Governing Law.  The validity, construction and interpretation of this
          -------------                                                        
Agreement shall be governed in all respects by the laws of the State of
California applicable to contracts made and to be performed wholly within that
State.

     2.6  Counterparts.  This Agreement may be executed simultaneously in two or
          ------------                                                          
more counterparts, each one of which shall be deemed an original, but all of
which shall constitute one and the same instrument.

     2.7  Third Parties.  Nothing in this Agreement, expressed or implied, is
          -------------                                                      
intended to confer upon any person other than the parties hereto and their
respective heirs, representatives, successors and assigns any rights or remedies
under or by reason of this Agreement.

     2.8  Attorney's Fees.  In the event any party takes legal action to enforce
          ---------------                                                       
any of the terms

                                      -4-
<PAGE>
 
of this Agreement, the unsuccessful party to such action shall pay the
successful party's expenses (including, but not limited to, attorneys' fees and
costs) incurred in such action.

     2.9  Further Assurances.  Each party hereto shall, from time to time at and
          ------------------                                                    
after the date hereof, execute and deliver such instruments, documents and
assurances and take such further actions as the other parties reasonably may
request to carry out the purpose and intent of this Agreement.

     2.10 Injunctive Relief.  Each of the parties hereto acknowledges and agrees
          -----------------                                                     
that it would be difficult to fully compensate the other parties for damages
resulting from the breach or threatened breach of any provision of this
Agreement and, accordingly, that each party shall be entitled to temporary and
injunctive relief, including temporary restraining orders, preliminary
injunctions and permanent injunctions, to enforce such provisions without the
necessity of proving actual damages or being required to post any bond or
undertaking in connection with any such action. This provision with respect to
injunctive relief shall not diminish, however, the right of any party to any
other relief or to claim and recover damages.

     2.11 Consent To Jurisdiction.  Each party hereto, to the fullest extent it
          -----------------------                                              
may effectively do so under applicable law, irrevocably (i) submits to the
exclusive jurisdiction of any court of the State of California or the United
States of America sitting in the City of Los Angeles over any suit, action or
proceeding arising out of or relating to this Agreement, (ii) waives and agrees
not to assert, by way of motion, as a defense or otherwise, any claim that it is
not subject to the jurisdiction of any such court, any objection that it may now
or hereafter have to the establishment of the venue of any such suit, action or
proceeding brought in any such court and any claim that any such suit, action or
proceeding brought in any such court has been brought in an inconvenient forum,
(iii) agrees that a judgment in any such suit, action or proceeding brought in
any such court shall be conclusive and binding upon such party and may be
enforced in the courts of the United States of America or the State of
California (or any other courts to the jurisdiction of which such party is or
may be subject) by a suit upon such judgment and (iv) consents to process being
served in any such suit, action or proceeding by mailing a copy thereof by
registered or certified air mail, postage prepaid, return receipt requested, to
the address of such party specified in or designated pursuant to Section 2.1.
Each party agrees that such service (i) shall be deemed in every respect
effective service of process upon such party in any such suit, action or
proceeding and (ii) shall, to the fullest extent permitted by law, be taken and
held to be valid personal service upon and personal delivery to such party.

     2.12 Arbitration.  Any controversy arising out of or relating to this
          -----------                                                     
Agreement or the transactions contemplated hereby shall be referred to
arbitration before the American Arbitration Association strictly in accordance
with the terms of this Agreement and the substantive law of the State of
California. The board of arbitrators shall convene at a place mutually
acceptable to the

                                      -5-
<PAGE>
 
parties in the State of California and, if the place of arbitration cannot be
agreed upon, arbitration shall be conducted in Los Angeles. The parties hereto
agree to accept the decision of the board of arbitrators, and judgment upon any
award rendered hereunder may be entered in any court having jurisdiction
thereof. No party shall institute a proceeding hereunder until that party has
furnished to the other party, by registered mail, at least thirty (30) days
prior written notice of its intent to do so. This Section 2.13 shall not limit
the right of any party to seek injunctive relief in the courts of the State of
California or the United States of America.

     2.13 Construction.  This Agreement was reviewed by legal counsel for each
          ------------                                                        
party hereto and is the product of informed negotiations between the parties
hereto. If any part of this Agreement is deemed to be unclear or ambiguous, it
shall be construed as if it were drafted jointly by the parties. Each party
hereto acknowledges that no party was in a superior bargaining position
regarding the substantive terms of this Agreement.

     2.14 Investment Representations.  GMC and RAI each hereby represent,
          --------------------------                                     
warrant and agree to and with the Company as follows:

          (a) It is acquiring the Shares for its own account, for investment
only and not with a view toward the resale or distribution thereof.

          (b) It understands that the Shares have not been registered under the
Securities Act of 1933, as amended (the "Securities Act"), nor qualified under
the securities laws of any state, and neither the Securities and Exchange
Commission nor any state securities commissioner nor any other regulatory
authority has passed upon the accuracy or adequacy of any disclosure made by the
Company to the undersigned or endorsed the merits of a purchase of the Shares.

          (c) It understands and agrees that, prior to consummation of any sale,
transfer, assignment, pledge, hypothecation or other disposition of the Shares,
or any part thereof, a written opinion of counsel must be delivered to the
Company to the effect that, under the Securities Act and the rules and
regulations promulgated thereunder, the proposed disposition may occur without
registration or that all registration requirements of the Securities Act and
             --                                                             
such rules have been or are being met.

          (d) It understands and agrees that the following legends will be
placed on certificates evidencing the Shares:

           THE SHARES REPRESENTED BY THIS CERTIFICATE HAVE NOT BEEN REGISTERED
           UNDER THE SECURITIES ACT OF 1933 (THE "ACT") AND ARE RESTRICTED
           SECURITIES. THE RESTRICTED SECURITIES HAVE BEEN ACQUIRED FOR THE
           HOLDER'S OWN ACCOUNT AND NOT WITH A VIEW TO

                                      -6-
<PAGE>
 
           DISTRIBUTE THEM TO THE PUBLIC. RESTRICTED SECURITIES MUST BE HELD
           INDEFINITELY UNLESS THEY ARE SUBSEQUENTLY REGISTERED UNDER THE ACT,
           OR AN OPINION OF COUNSEL FOR THE PROPOSED TRANSFEROR IS DELIVERED TO
           THE COMPANY, WHICH OPINION SHALL, IN FORM AND SUBSTANCE, BE
           REASONABLY SATISFACTORY TO THE COMPANY AND ITS COUNSEL AND SHALL
           STATE THAT AN EXEMPTION FROM SUCH REGISTRATION IS AVAILABLE.

           THIS CERTIFICATE AND THE SHARES EVIDENCED HEREBY MAY BE SOLD,
           TRANSFERRED, ASSIGNED, HYPOTHECATED, PLEDGED OR OTHERWISE ALIENATED
           ONLY IN ACCORDANCE WITH AND SUBJECT TO THE PROVISIONS OF THAT CERTAIN
           ESCROW AGREEMENT DATED AS OF JULY 2, 1998, A COPY OF WHICH IS ON FILE
           AT THE CORPORATION'S PRINCIPAL OFFICE.

          (e) It understands and agrees that, in order to ensure that no
unlawful resales or other dispositions of the Shares are allowed to occur,
appropriate stop-transfer instructions restricting resale or disposition of the
Shares will be placed against such Shares with the transfer agent of the
Company.

          (f) It understands that the foregoing representations, warranties and
agreements are made in part to induce the Company to rely on certain exemptions
from the registration and qualification provisions of the Securities Act and the
securities laws of various states with respect to the issuance of the Shares
pursuant to the Purchase Agreement.

                                      -7-
<PAGE>
 
     IN WITNESS WHEREOF, the parties hereto have caused this Agreement to be
executed as of the date and year first set forth above.

                              TARRANT APPAREL GROUP



                              By      /s/ Mark B. Kristof
                                  ---------------------------------------
                                    Authorized Representative



                              GABRIEL MANUFACTURING COMPANY



                              By      /s/ Gabriel Zeitouni
                                  -------------------------------------
                                    Authorized Representative



                              ROCKY APPAREL, INC.



                              By      /s/ Gabriel Zeitouni
                                  --------------------------------------
                                    Authorized Representative

                                      -8-