<PAGE>   1
                                                                   EXHIBIT 10.4

                             MODIFICATION OF LEASE

        THIS AGREEMENT is made as of the 15 day of May, 1997, between 1411
TRIZECHAHN-SWIG, L.L.C., a Delaware limited liability company having an office
at c/o TrizecHahn Office Properties Inc., 1411 Broadway, New York, New York
10018 (hereinafter referred to as "Landlord"), and THE SIRENA APPAREL GROUP,
INC., a Delaware corporation having an office at 10333 Vacco Street, South El
Monte, California 91733 (hereinafter referred to as "Tenant").

        WHEREAS, Landlord's predecessor-in-interest, Keystone Associates, and
Tenant's predecessor-in-interest, Consolidated Foods Corp. Sirena Division,
entered into a lease dated as of July 12, 1984, as modified by (i) a
Modification and Extension of Lease dated as of July 18, 1990 and (ii) a
Modification and Extension of Lease dated as of April 14, 1994 (collectively,
the "Lease"), of a portion of the twenty-third (23rd) floor in the building (the
"Building") known as 1411 Broadway, New York, New York, more particularly
described therein (the "Original Demised Premises"); and

        WHEREAS, Tenant desires to hire and take additional space on the
seventeenth (17th) floor of the Building, as shown on the hatched portion of the
floor plan annexed hereto and made a part hereof as Exhibit "A" (the
"Seventeenth Floor Additional Space").

        NOW, THEREFORE, in consideration of the mutual covenants herein
contained, and other good and valuable consideration the receipt and sufficiency
of which are hereby conclusively acknowledged, Landlord and Tenant hereby agree
as follows:

        1.      Effective as of May 15, 1997 (the "Seventeenth Floor
Commencement Date"), the Seventeenth Floor Additional Space shall be added to
and become part of the premises demised under the Lease for a term (the
"Seventeenth Floor Term") that shall commence on the Seventeenth Floor
Commencement Date and expire on July 31, 2002 (the "Seventeenth Floor Expiration
Date"). During the Seventeenth Floor Term, the Original Demised Premises and the
Seventeenth Floor Additional Space shall collectively be deemed to constitute
the "Demised Premises". Landlord and Tenant hereby acknowledge that the
Expiration Date of the term of the Lease with respect to the Original Demised
Premises is August 31, 2004.

        2.      Commencing on the Seventeenth Floor Commencement Date and
continuing through and including the Seventeenth Floor Expiration Date, the
Lease shall be deemed modified as follows:

                (a) the Fixed Minimum Rent Inclusive of Electricity, set forth
in Section 1.01 of the Lease, shall be increased by $94,810.00 per annum and
$7,900.83 per month;

                (b) the Fixed Minimum Rent, set forth in Section 1.01 of the
Lease, shall be increased by $87,325.00 per annum and $7,277.08 per month;

                (c) Tenant's Pro Rata Share, set forth in Section 1.01 of the
Lease, shall be increased by .2363%; and

                (d) the Rentable Area, set forth in Section 1.01 of the Lease,
shall be increased by 2,495 square feet.

                                      -1-
<PAGE>   2
        3.      Effective as of the Seventeenth Floor Commencement Date and
continuing through and including the Seventeenth Floor Expiration Date, with
respect to the Seventeenth Floor Additional Space only, the Lease shall be
modified as follows:

                (a) the Base Tax Year, set forth in Subsection 19.02(c) of the
Lease, shall be deemed to mean the twelve (12) month fiscal year commencing on
July 1, 1996 and ending June 30, 1997;

                (b) the Base Wage Rate, set forth in Subsection 19.02(e) of the
Lease, shall be deemed to mean the Wage Rate in effect on the first day of the
calendar year 1997;

                (c) the date set forth on the third (3rd) line of Subsection
20.03A of the Lease, shall be deemed to mean the Seventeenth Floor Commencement
Date; and

                (d) the calendar year set forth in Section 20.03B of the Lease
shall be deemed to mean the calendar year 1997.

        4.      A.      Tenant shall deliver to Landlord, on the date hereof, an
additional cash security deposit in the amount of $15,801.56 (the "Seventeenth
Floor Security Deposit"), increasing the security deposit held by Landlord under
the Lease to $88,802.69. Tenant shall be obligated to maintain the Seventeenth
Floor Security Deposit with Landlord until the Seventeenth Floor Term shall have
expired (it being acknowledged that the provisions of Subparagraph 8C of the
Modification and Extension of Lease between Landlord and Tenant dated as of
April 14, 1994 shall not apply to the Seventeenth Floor Additional Space and
that Seventeenth Floor Security Deposit shall not be reduced at any time prior
to the Seventeenth Floor Expiration Date).

                B.      Simultaneously with the execution of this Agreement,
Tenant is delivering to Landlord a cash security deposit in the sum of
$20,100.19 in place of the letter of credit previously held by Landlord pursuant
to the provisions of Paragraph 5 of the Modification and Extension of Lease
dated as of July 18, 1990 between Landlord and Tenant. Landlord and Tenant
acknowledge that said letter of credit is null and void.

        5.      Tenant agrees to accept possession of the Seventeenth Floor
Additional Space in "as is" and "where is" condition on the Seventeenth Floor
Commencement Date, and Landlord shall not be obligated to perform any work
whatsoever to prepare the Seventeenth Floor Additional Space or any other
portion of the Demised Premises for Tenant's occupancy thereof, except that
Landlord shall perform the following work ("Landlord's Work") in the Seventeenth
Floor Additional Space: (i) paint the Demised Premises one coat using one
Building standard color per room or open area, (ii) install new Building
standard carpeting in a color selected by Tenant and (iii) demolish one interior
partition. All materials, work, labor, fixtures and installations required for
completion of the Seventeenth Floor Additional Space and the operation of
Tenant's business thereof other than Landlord's Work shall (subject to the
provisions of Article 5 of the Lease) be promptly furnished and performed by
Tenant at Tenant's own cost and expense. All provisions (if any) of the Lease
concerning the performance by Landlord of any work in connection with the
preparation of the Original Demised Premises for Tenant's occupancy thereof and
any related contribution, rent abatement or rent credit shall be inapplicable to
the Seventeenth Floor Additional Space.

        6.      If Landlord shall be unable to give possession of the
Seventeenth Floor Additional Space on any specific date by reason of the fact
that the Seventeenth Floor Additional Space is not ready for occupancy, or by
reason of the failure of a prior tenant or occupant thereof to vacate the same
or deliver possession thereof to Landlord, or for any other

                                      -2-
<PAGE>   3
reason, Landlord shall not be subjected to any liability for the failure to
give possession on said date.  No such failure to give possession on any
specific date shall affect the validity of the Lease or this Agreement or the
obligations of Tenant thereunder or under this Agreement or be deemed to extend
the Term of the Lease, but the rent reserved and covenanted to be paid with
respect to the Seventeenth Floor Additional Space only shall not commence until
possession of the Seventeenth Floor Additional space is given or is available
for occupancy by Tenant, except that if such failure to give possession has
been caused by any act or omission on the part of Tenant, there shall be no
abatement of rent.  If repairs, improvements or decorations of the Seventeenth
Floor Additional Space, if any, as may be expressly provided in this Agreement
to be made by Landlord, are not completed on or before any specific date,
Landlord shall not be subject to any liability for any delay in such completion.


        7.      Landlord hereby grants an aggregate rent credit to be applied
against the first two (2) monthly installments of Fixed Minimum Rent inclusive
of Electricity becoming due under the Lease with respect to the Seventeenth
Floor Additional Space in the amount of $14,554.16, it being understood that
said rent credit shall be applied by granting to Tenant a rent credit in the
sum of $7,277.08 per month for each of said two months.

        8.      Except as expressly modified herein, Landlord and Tenant affirm
that the Lease is in full force and effect, and Tenant certifies that all
obligations of Landlord under the Lease as of this date have been fully
performed and complied with by Landlord.  By entering into this Agreement,
Landlord does not and shall not be deemed either (i) to waive or forgive any
default, rent arrears or other condition with respect to the Lease or the use
of the Original Demised Premises, whether or not in existence or known to
Landlord at the date hereof, or (ii) to consent to any matter as to which
Landlord's consent is required under the terms of the Lease, except such as may
heretofore have been waived in writing or consented to in writing by Landlord.

        9.      All capitalized terms and other terms not otherwise defined
herein shall have the meanings ascribed to them in the Lease.

        10.     Tenant represents and warrants to Landlord that Tenant has not
employed, dealt with or negotiated with any broker other than Fashion Realty
Group, Ltd. in connection with this Agreement and Tenant shall indemnify,
protect, defend and hold Landlord harmless from and against any and all
liability, damage, cost and expense (including attorney's fees and
disbursements) arising out of any claim for a fee or commission by any broker
or other party, other than Fashion Realty Group, Ltd., in connection with this
Agreement.

        11.     The preamble and recitals contained in the "WHEREAS" clauses of
this Agreement are hereby incorporated into this Agreement.

        12.     Except as expressly modified or amended by this Agreement, all
of the terms, covenants and conditions of the Lease are hereby ratified and
confirmed and, except insofar as reference to the contrary is made in any such
instrument, all references to the "Lease" in any


                                      -3-
<PAGE>   4
future correspondence or notice shall be deemed to refer to the Lease as
modified by this Agreement.

        IN WITNESS WHEREOF, Landlord and Tenant have duly executed this
Modification of Lease as of the date first above written.

                     1411 TRIZECHAHN-SWIG, L.L.C., Landlord
                     By: TrizecHahn Office Properties Inc., Agent


                     By: /s/ CAROL A. MEYER
                         ---------------------------------
                       Carol A. Meyer, Assistant Secretary


                     By: /s/ ANTONIO A. BISMONTE
                         ---------------------------------
                       Antonio A. Bismonte, Vice-President


                     THE SIRENA APPAREL GROUP, INC., Tenant


                     By: /s/ HENRY R. MANDELL
                         ---------------------------------
                       Name: Henry R. Mandell
                       Title: EVP, CFO

                     Tenant's Federal I.D. #36-2998726
<PAGE>   5
STATE OF NEW YORK                     )
                                      ) ss.:
COUNTY OF NEW YORK                    )


        On this 9 day of May, 1997, before me personally came Henry R. Mandell,
to me known, who being by me duly sworn, did depose and say that he is the
EVP-CFO of THE SIRENA APPAREL GROUP, INC., the corporation described in, and
which executed the above instrument; and that he signed his name thereto by
order of the Board of Directors of said corporation.


                                   /s/ JENNIFER V. PINTO
                                   ---------------------
                                   Notary Public

                               JENNIFER V. PINTO
                        Notary Public, State of New York
                                 No. 24-4929844
                           Qualified in Kings County
                           Term Expires May 31, 1998
        
<PAGE>   6
                                                                     EXHIBIT A


                                 SEVENTH AVENUE

WEST 39th STREET                                               WEST 40th STREET




                                     M A P