<PAGE>   1
                                                                    EXHIBIT 10.4


                      FIRST AMENDMENT TO SECURITY AGREEMENT



        Reference is made to a certain Security Agreement dated June 5, 1997
(the "Security Agreement") by and between DM Management Company, a Delaware
Corporation, (the "Debtor") and Citizens Bank of Massachusetts (the "Secured
Party").

        In consideration of Secured Party extending additional credit to the
Debtor in the original principal amount of Four Million Three Hundred Thousand
Dollars ($4,300,000.00) and otherwise agreeing to amend the Loan Agreement dated
June 5, 1997 at the request of the Debtor, which the Secured Party is unwilling
to do unless, among other things paragraph 1(e) of the Security Agreement is
amended to confirm that it (i) secures payment and performance of the Loan
Agreement as amended, and (ii) payment and performance of the "Bridge Note" and
"Bridge Mortgage" as defined in the Loan Agreement, the Debtor and the Secured
Party hereby agree that the Security Agreement be and the same hereby is,
amended by deleting Paragraph 1 thereof and replacing it with the following:

                The Debtor hereby grants to Secured Party, a security interest
                in all of the Debtor's present and future right, title and
                interest in and to the property described on Exhibit A (all of
                which is hereinafter called the "Collateral") to secure (a) the
                full payment of the sum of $8,500,000.00, or such lesser amount
                which shall have been advanced, together with interest and other
                charges, all as provided in a certain "Revolving Note" of the
                Debtor to the order of the Secured Party dated June 5, 1997,
                which note was amended and replaced by a certain "Replacement
                Revolving Note" of even date herewith in the face amount of
                $8,500,000.00, all as provided in a certain Loan Agreement dated
                June 5, 1997 as amended and restated as of the date hereof (the
                "Loan Agreement") together with all substitutions or
                replacements therefor and all renewals or extensions thereof and
                the full performance of all other obligations of the maker of
                said note as provided therein; (b) the full payment of the sum
                of $1,650,000.00, as provided in a certain "Real Estate Note"
                (as defined in the Loan Agreement) of the Debtor to the order of
                the Secured Party dated June 30, 1997, executed and delivered by
                the Debtor to the Secured Party, pursuant to the Loan Agreement
                in the face amount of $1,650,000.00, with interest and other
                charges as provided therein, together with all substitutions and
                replacements therefor and all renewals and extensions thereof
                and the full performance of all other obligations of the maker
                of said note as provided therein; (c) the full payment of the
                sum of $3,600,000.00, with interest and other charges, all as
                provided in a certain "Term Note" (as defined in the Loan
                Agreement) of the Debtor to the order of the Secured Party dated
                June 5, 1997, executed and delivered by the Debtor to the
                Secured Party pursuant to the Loan Agreement, in the original
                face amount of $3,600,000.00, together with all substitutions or
                replacements therefor and all renewals or extensions thereof and
                the full performance of all other obligations of the maker of
                said note as provided therein; (d) the full payment of the sum
                of $4,300,000.00, with interest and other charges, all as
                provided in a certain 





<PAGE>   2

                "Bridge Note" (as defined in the Loan Agreement), of the Debtor
                to the order of the Secured Party dated as of the date hereof,
                executed and delivered by the Debtor to the Secured Party
                pursuant to the Loan Agreement, in the original face amount of
                $4,300,000.00, together with all substitutions and replacements
                therefor and all renewals and extensions thereof and the full
                performance of all other obligations of the maker of said note
                as provided therein and under a certain "Bridge Mortgage" (as
                defined in the Loan Agreement) dated as of the date hereof
                executed and delivered in connection therewith; (e) the full
                payment and performance by the Debtor of all indebtedness,
                obligations and liabilities of the Debtor to the Secured Party
                under the Loan Agreement, direct or indirect, absolute or
                contingent, now existing or hereafter arising (including,
                without limitation, all "Obligations", as defined in the Loan
                Agreement) which Loan Agreement provides, among other things,
                for the establishment of a "Revolving Loan" (as defined therein)
                and for the issuance of Letters of Credit pursuant to "L/C
                Applications" (as defined therein) therein pursuant to which
                "Advances" (as defined therein) may be made from time to time,
                and for repayment of all or a portion of the outstanding balance
                of such Advances together with interest and other charges, all
                in accordance therewith, and for the grant of "Loans" (as
                defined therein) as provided therein; and (f) the full payment
                and performance of all covenants and agreements herein contained
                or referred to on the part of the Debtor to be kept and
                performed (collectively hereafter referred to as "Obligations").

        In all other respects, the Security Agreement shall remain in full force
and effect in accordance with its terms.

        Executed as an instrument under seal as of the 31st day of October,
1997.



                                        DM MANAGEMENT COMPANY

/s/ Jacob N. Polatin                    /s/ Peter J. Tulp
---------------------------------       ------------------------------------
Witness                                 By: Peter J. Tulp
                                            Corporate Controller



Agreed:  Citizens Bank of Massachusetts


          By: /s/ Lori B. Leeth, SVP
              ----------------------------------------
              Lori B. Leeth, Senior Vice President