<PAGE>

                                                                    Exhibit 10.4

                               VOTING AGREEMENT
 
     This VOTING AGREEMENT (this "Agreement"), dated as of May 7, 1997, is made
and entered into by and among BROOKDALE LIVING COMMUNITIES, INC., a Delaware
corporation (the "Company"), THE PRIME GROUP, INC., an Illinois corporation
("Prime"), PRIME GROUP LIMITED PARTNERSHIP, an Illinois limited partnership
("PGLP"), and PRIME GROUP VI, L.P., an Illinois limited partnership ("PG-VI,
L.P.") (together with Prime and PGLP, the "TPG Group").

                                   RECITALS


     WHEREAS, the Company has filed with the Securities and Exchange Commission
a Form S-1 Registration Statement in connection with the Company's proposed
initial public offering (the "Offering") of its Common Stock, $0.01 par value
per share (the "Common Stock");

     WHEREAS, PG-VI, L.P. has purchased 2,500,000 of the 4,500,000 shares of
Common Stock issued and sold in the Offering, representing approximately 38.5%
of the outstanding Common Stock after giving effect to the Offering;

     WHEREAS, pursuant to the terms of that certain Formation Agreement dated as
of May 7, 1997, the Company has issued an aggregate of 1,703,043 shares of its
Common Stock to Prime and PGLP, representing approximately 26.2% of the
outstanding Common Stock after giving effect to the Offering; and

     WHEREAS, in connection with the Offering, the Company and the TPG Group
have agreed to enter into this Agreement.

                                  AGREEMENTS

     NOW, THEREFORE, in consideration of the recitals and the mutual promises
and covenants herein contained and other good and valuable consideration, the
receipt and sufficiency of which are hereby acknowledged, the parties hereto
hereby agree as follows:

     1.   Agreement to Vote Shares of Common Stock. The TPG Group agrees that,
from and after the completion of the Offering and until this Agreement
terminates pursuant to Section 2 hereof, it will vote all of the TPG Group's
shares of Common Stock at any meeting at which directors of the Company are
elected in favor of that number of independent directors who are not affiliates
of the TPG Group or employees of the TPG Group (any such directors,
"Independent Directors") so that, if such directors were elected,
<PAGE>
 

there would be at least four (4) Independent Directors who are members of the
Board of Directors of the Company. The parties hereto acknowledge and agree that
the agreement contained in the immediately preceding sentence is not, and shall
not be construed as, a guarantee by the TPG Group, or any member thereof, that
any of the Independent Directors for which the TPG Group votes in favor of will
be elected as a member of the Board of Directors of the Company or that there
will be at least four (4) Independent Directors who are members of the Board of
Directors of the Company.

     2.   Term. This Agreement will automatically terminate upon the earlier of
(i) three (3) years from the closing date of the Offering or (ii) the date
the TPG Group owns less than ten percent (10%) of the then outstanding Common
Stock.

     3.   Notices. Any notices required or permitted to be sent hereunder shall
be delivered personally or mailed, certified mail, return receipt requested, or
delivered by overnight courier service guaranteeing next business day delivery,
to the following addresses, or such other addresses as shall be given by notice
delivered hereunder, in each case with applicable postage or delivery charges
prepaid, and shall be deemed to have been given upon delivery, if delivered
personally, three business days after mailing, if mailed, or one business day
after delivery to the courier, if delivery by overnight courier service:

                                                                                
          If to the Company, to:

             Brookdale Living Communities, Inc.
             77 West Wacker Drive
             Suite 3900
             Chicago, Illinois 60601
             Attn:  President

          With a copy to:

             Winston & Strawn
             35 West Wacker Drive
             Chicago, Illinois 60601
             Attn: Wayne D. Boberg

                                                                                
          If to the TPG Group or any member thereof, to:
  
             c/o The Prime Group, Inc.
             77 West Wacker Drive
             39th Floor
             Chicago, Illinois 60601
             Attn: Michael W. Reschke

                                      -2-
<PAGE>
 
          With a copy to:

             The Prime Group, Inc.
             77 West Wacker Drive
             Suite 3900
             Chicago, Illinois 60601
             Attn: Robert J. Rudnik

     4.   Final Agreement.  This Agreement constitutes the final agreement of
the parties concerning the matters referred to herein, and supersedes all prior
agreements and understandings.

     5.   Severability.  Whenever possible, each provision of this Agreement
will be interpreted in such manner as to be effective and valid under applicable
law, but if any provision of this Agreement is held to be prohibited by or
invalid under applicable law, such provision will be ineffective only to the
extent of such prohibition or invalidity, without invalidating the remainder of
this Agreement.

     6.   Descriptive Headings.  The descriptive headings of this Agreement are
inserted for convenience of reference only and do not constitute a part of and
shall not be utilized in interpreting this Agreement.

     7.   Governing Law.  All questions concerning the construction, validity
and interpretation of, and the performance of the obligations imposed by, this
Agreement shall be governed by and construed in accordance with the laws of the
State of Delaware (excluding the choice of law provisions thereof).

     8.   Amendments.  The provisions of this Agreement may be amended or
waived at any time only by the written agreement of the parties hereto.

     9.   Successors and Assigns.  The provisions of this Agreement shall be
binding upon and inure to the benefit of the respective successors and assigns
of the parties hereto, whether so expressed or not, including, without
limitation, any person or entity to whom any member of the TPG Group assigns any
of the Common Stock held by it.

                           [signature page follows]

                                      -3-
<PAGE>
 
     IN WITNESS WHEREOF, the parties hereto have caused this Voting Agreement to
be duly executed and delivered on their behalf as of the date first above
written.

                   
                   
                               THE COMPANY:
 
                               BROOKDALE LIVING COMMUNITIES, INC.,
                               a Delaware corporation
                          
                               By: /s/ Mark J. Schulte
                                 ------------------------------------
                               Name:  Mark J. Schulte
                                    ---------------------------------
                               Title: President
                                     --------------------------------

                                                                    
                               THE TPG GROUP:

                               THE PRIME GROUP, INC.,
                               an Illinois corporation
                               
                               By: /s/ Michael W. Reschke
                                 ------------------------------------
                               Name:  Michael W. Reschke 
                                    ---------------------------------
                               Title: President
                                     --------------------------------


                               PRIME GROUP LIMITED PARTNERSHIP,
                               an Illinois limited partnership

                               By: /s/ Michael W. Reschke
                                 ------------------------------------
                                     Michael W. Reschke,
                                     Managing General Partner

        
                               PRIME GROUP VI, L.P.

                               By: PGLP, Inc.,
                                   its Managing General Partner 
                              
                                   By: /s/ Richard S. Curto
                                       ------------------------------
                                   Name:  Richard S. Curto 
                                         ----------------------------
                                   Title: Vice President
                                         ----------------------------
    
                                      S-1