<PAGE>
 
                                                                   EXHIBIT 10.80

                                PROMISSORY NOTE
                           SECURED BY DEED OF TRUST

$2,000,000.00                                                 Irvine, California
                                                                  April 13, 1999

     FOR VALUE RECEIVED, the undersigned, G & L REALTY CORP., a Maryland 
corporation ("Borrower"), promises to pay to REESE L. MILNER, as Trustee of The 
Milner Trust ("Lender"), at 439 North Bedford Drive, Beverly Hills, California, 
or at such other place as may be designated in writing by Lender, the principal 
sum of Two Million and No/100 Dollars ($2,000,000.00), together with a loan fee 
equal to Twenty Thousand and No/100 Dollars ($20,000.00), with interest thereon 
at the rate of twelve percent (12%) per annum, compounded monthly. All sums 
owing hereunder are payable in lawful money of the United States of America, in 
immediately available funds.

     1.   Security. This Promissory Note Secured by Deed of Trust (this "Note") 
is secured by, among other things, a Deed of Trust with Assignment of Rents 
(Short Form) dated as of even date herewith (the "Deed of Trust"), executed by 
G&L Realty Partnership, L.P., a Delaware limited partnership, as trustor, in 
favor of Lender, as beneficiary, and encumbering the real property described in 
the Deed of Trust (the "Property"). The holder of this Note shall be entitled to
the benefits of the security provided by the Deed of Trust and shall have the 
right to enforce the covenants and agreements of Borrower contained in the Deed 
of Trust.

     2.   Maturity and Payment. The outstanding principal balance, together with
any accrued but unpaid interest and the above-referenced loan fee, shall be due 
and payable in full on or before May 13, 1999 ("Maturity Date").

     3.   Default. At the option of Lender, without prior notice, and regardless
of any prior forbearance, all sums remaining unpaid under this Note shall become
immediately due and payable upon the occurrence of an Event of Default (as 
defined below). The occurrence of any of the following events shall constitute 
an "Event of Default" under this Note:

          (a)  Borrower shall fail to pay within ten (10) days of when due any 
               sums payable hereunder; or 

          (b)  A Default (as defined in the Deed of Trust) occurs under the Deed
               of Trust or under any obligations secured thereby; or

          (c)  The entire property which is subject to the Deed of Trust, is
               sold, transferred or conveyed, whether voluntarily or
               involuntarily or by operation of law or otherwise, other than as
               expressly permitted by Lender in writing; or

          (d)  If Borrower files, or has filed against it, a petition with the
               bankruptcy court under Title 11 of the United States Code (the
               "Bankruptcy Code"), or otherwise files, or has filed against it,
               any petition or applies to any tribunal for appointment of a
               custodian, trustee, receiver or agent of Borrower, or

                                      -1-
<PAGE>
 
               commences any proceeding relating to Borrower under any
               bankruptcy or reorganization statute or under any arrangement,
               insolvency, readjustment of debt, dissolution or liquidation law
               of any jurisdiction, whether now or hereafter in effect, and such
               proceeding is not dismissed within ninety (90) days after the
               filing thereof.

     4.   Attorneys' Fees. In the event of any action or proceeding to enforce 
this Note, the prevailing party shall be entitled to receive, immediately upon 
demand, all attorneys' fees and all costs incurred by such party in connection 
therewith, together with interest thereon from the date of such demand until 
paid at the rate of interest applicable to the principal balance owing hereunder
as if such unpaid attorneys' fees and costs had been added to the principal.

     5.   Prepayment. Borrower may, at any time, prepay any amounts outstanding 
under this Note in whole or in part without premium or penalty.

     6.   No Implied Waivers. No previous waiver and no failure or delay by 
Lender in acting with respect to the terms of this Note shall constitute a 
waiver of any breach, default, or failure of condition under this Note or the 
obligations secured thereby. A waiver of any term of this Note or of any of the 
obligations secured thereby must be made in writing and shall be limited to the 
express written terms of such waiver. In the event of any inconsistencies 
between the terms of this Note and the terms of any other document related to 
the loan evidenced by this Note, the terms of this Note shall prevail.

     7.   Miscellaneous. If this Note is executed by more than one (1) person as
Borrower, the obligations of each such persons shall be joint and several. 
Time is of the essence with respect to every provision hereof. This Note shall 
be construed and enforced in accordance with the laws of the State of 
California, and all persons and entities in any manner obligated under this Note
consent to the jurisdiction of any federal or state court within the State of 
California having proper venue and also consent to service of process by any 
means authorized by California or federal law. If any provision hereof is found 
to be invalid or unenforceable by a court of competent jurisdiction, the 
invalidity thereof shall not affect the enforceability of the remaining 
provisions of this Note.

     8.   Usury Law Compliance. It is Borrower's and Lender's intention to 
comply with any applicable usury law. If, for any reason whatsoever, fulfillment
of any provision hereof shall be prohibited by law, the obligation to be 
fulfilled shall be reduced to the maximum amount so prohibited, and if for any 
reason Lender should have received as interest an amount which would exceed the 
highest lawful rate, such amount which would be in excess of the permitted 
interest shall, at Lender's option, be applied to the reduction of principal of
this Note and not to the payment of interest, or be refunded to Borrower. All
agreements between Borrower and Lender are expressly limited so that in no
contingency or event whatsoever shall the amount paid or agreed to be paid to
Lender for the use, forbearance or detention of money under this Note exceed the
maximum permissible under applicable law. This provision shall control over any
other provision in this Note or in any other agreement between Borrower and
Lender related hereto.

                        [Signature on following page.]


                                      -2-
<PAGE>
 
         [Signature page to Promissory Note Secured by Deed of Trust.]


                                         "Borrower"

                                         G & L Realty Corp.,
                                         a Maryland corporation
                                         
                                         By:  /s/ Joseph D. Carroll
                                              -----------------------------
                                              Joseph D. Carroll,
                                              Senior Vice President



                                      -3-