<PAGE>   1
                                                                    EXHIBIT 10.4

                     PROMISSORY NOTE AND SECURITY AGREEMENT

$499,969.08                                                     JANUARY 31, 2000


         FOR VALUE RECEIVED, MICHAEL L. SCHWARZ, who resides at c/o Summit
Properties Inc., 212 South Tryon Street, 500, Charlotte, North Carolina 28281
(hereinafter referred to as the "Employee"), hereby promises to pay to the order
of Summit Properties Inc., a Maryland corporation with its principal place of
business at 212 South Tryon Street, Suite 500, Charlotte, North Carolina
(hereinafter referred to as the "Company"), the principal amount of $499,969.08
together with interest thereon as provided below subject to the terms and
conditions set forth herein.

         1. Purpose and Authority. This Promissory Note and Security Agreement
(the "Note") is entered into for the purpose of financing the Employee's
purchase of shares of common stock, par value $0.01 per share, of the Company
("Common Stock") pursuant to and subject to the terms and conditions of (i) the
Company's Statement of Company Policy on Loans to Executive Officers and
Qualified Employees to Purchase the Company Stock as adopted by the Board of
Directors of the Company on September 8, 1997, as amended from time to time, and
(ii) the Company's 1994 Stock Option and Incentive Plan, as amended from time to
time.

         2. Security. The Employee hereby grants the Company a security interest
in any and all shares of Common Stock purchased by the Employee with the
proceeds of this Note (hereinafter referred to as the "Collateral Stock") and in
any and all distributions and dividends which may from time to time be, paid or
payable on the Collateral Stock (each, a "Distribution"). Employee agrees to
take all such actions and execute all such documents as may from time to time be
reasonably requested by the Company to perfect and maintain the validity and
priority of any security interest granted to the Company pursuant to this Note.
Employee also agrees that a carbon, photographic or other reproduction of this
Promissory Note and Security Agreement may be filed as a financing statement to
the extent that the Company determines that such filing is necessary for the
Company to establish or maintain its security interest in the Collateral Stock.
The Employee shall cause the Collateral Stock to be delivered to the Company and
the Company may retain possession of the Collateral Stock until such time as the
Note has been paid in full.

         3. Payment. All Distributions received by the Employee in cash shall be
applied toward repayment of this Note. The Employee agrees that the Company may
establish and institute any procedure that it deems necessary or advisable to
ensure that each such Distribution shall be applied toward repayment of this
Note, including without limitation, the placement of a restrictive legend on any
check representing a Distribution. Each such payment shall first be applied to
the payment of interest accrued as of the date of such payment and the remainder
thereof, if any, shall then be applied to the payment of outstanding principal.
The Note will bear interest at the rate provided in Section 4 hereof. The entire
principal balance and all accrued and unpaid interest and other charges as may
be due hereunder shall be due and payable on or before the tenth anniversary of
the date of this Note (the "Maturity Date").


<PAGE>   2

         4. Interest. Interest on this Note will be computed on a simple
interest basis and will accrue on the unpaid principal balance due under the
Note until maturity, whether by reason of Default (as defined below) and
acceleration, lapse of time or otherwise ("Maturity"), at the rate of SIX AND
21/100 PERCENT (6.21%) per annum. Prior to Maturity interest shall be payable
solely from Distributions.

         5. Prepayment. The Employee may prepay the whole or any part of the
principal amount of this Note from time to time without premium or penalty.

         6. Default. (a) The occurrence of any of the following events shall
constitute a Default under this Note:

                           (i) the failure by the Employee to deliver or cause
         to be delivered the Collateral Stock to the Company within three
         business days after the purchase of any Collateral Stock;

                           (ii) retention by the Employee of any Distribution,
         which retention continues for a period of ten (10) days;

                           (iii) the failure by the Employee to pay the entire
         outstanding balance of this Note and all accrued interest within one
         hundred and twenty (120) days after termination of the Employee's
         employment with the Company; or

                           (iv) the failure by the Employee to pay the entire
         outstanding balance of this Note and all accrued interest on or before
         the Maturity Date.

                  (b) Upon the occurrence of a Default under this Note, the
outstanding principal balance hereof, together with all reasonable costs of
collection and/or enforcement of the Note, including reasonable attorney's fees,
shall at the option of the Company become immediately due and payable.

                  (c) If the Employee is in Default hereunder, the Company may,
except as otherwise provided herein, exercise the rights and remedies accorded a
secured party by the Uniform Commercial Code as enacted in the State of
Maryland.

         7. Personal Liability. The obligations of the Employee to pay the
unpaid principal balance of this Note, plus accrued interest thereon and other
charges as may be due hereunder, shall be absolute and unconditional, and the
Company shall have full recourse against the Employee's assets (including, but
not limited to, the Collateral Stock) to recover such amounts.

         8. Modification. Neither this Note nor any provision hereof may be
modified, altered, or amended in any manner or form except by an agreement in
writing, executed by a duly authorized officer of the Company and the Employee,
which writing shall make specific reference hereto.

<PAGE>   3

         9. Transfer by Employee. Employee will not sell, assign, transfer or
otherwise dispose of, directly or indirectly, nor grant any option with respect
to, or pledge or grant any security interest in or otherwise encumber any of the
Collateral Stock or any interest therein, except for the security interest
provided for in this Note.

         10. Severability. If for any reason any provision or provisions hereof
are determined to be invalid, unenforceable or contrary to any existing or
future law, such invalidity or unenforcability shall not impair the operation or
affect those portions of this Note which are valid.

         11. Usury, etc. All agreements between the Employee and the Company are
hereby expressly limited so that in no contingency or event whatsoever, whether
by reason Maturity of the indebtedness or otherwise, shall the amount paid or
agreed to be paid to the holder for the use, forbearance or detention of the
indebtedness evidenced hereby exceed the maximum amount which the holder is
permitted to receive under applicable law. If, from any circumstances
whatsoever, fulfillment of any provision of this Note, at the time performance
of such provision shall be due, shall involve payments exceeding such amount,
then the obligation to be fulfilled shall automatically be reduced to the limit
of such maximum amount, and if from any circumstances the holder should ever
receive as interest an amount which would exceed such maximum amount, such
amount which would be excessive interest shall be applied to the reduction of
the principal balance evidenced hereby and not to the payment of interest. As
used herein, the term "applicable law" shall mean the law in effect as of the
date hereof; provided, however, that in the event that there is a change in the
law which results in a higher permissible rate of interest, then this Note shall
be governed by such new law as of its effective date. This provision shall
control every other provision of this Note.

         12. Valuation: Manner of Disposition. Employee acknowledges and agrees
that the Company may not be able to effect a public sale of the Collateral Stock
and, accordingly, agrees that in the event of any sale, collection, realization
or other disposition of or upon the Collateral Stock by the Company, in lieu of
such public sale, the Company may transfer all or any portion of the Collateral
Stock to itself and apply the value of such shares (at a price per share equal
to the average of the daily high and low sales prices, computed to three decimal
places, of the Company's stock as reported on the NYSE for the ten (10) days on
which the NYSE is open and for which trades in the Company stock are reported
immediately preceding the date of such action by the Company or, if one or more
of such days is not a day on which the NYSE is open or the Company's stock is
not traded on the NYSE for the ten (10) days immediately preceding said action
for which the trades are reported) to the amounts due under or in connection
with this Note.

         13. Governing Law. The execution, delivery and performance of this Note
shall be governed by, construed, and enforced in accordance with the laws of the
State of Maryland.

         14. Waivers. The failure of the Company at any time to exercise any
option or right hereunder shall not constitute a waiver of the Company's right
to exercise such option or right at any other time.


<PAGE>   4

         IN WITNESS WHEREOF, this Note has been executed and delivered as a
sealed instrument as of the date first set forth above.




                                                     /s/ Michael L. Schwarz
                                                     -------------------------
                                                     MICHAEL L. SCHWARZ



Executed, sealed and
delivered in the
presence of:


/s/ Susan Wilson
-------------------------
Name of Witness: