<PAGE>
                                                                    Exhibit 10.4

                              RAIT INVESTMENT TRUST

                  PHANTOM SHARE PLAN FOR NON-EMPLOYEE TRUSTEES

I.    PURPOSE. The purpose of the Plan is to provide a means whereby RAIT may
grant equivalent rights relating to the Shares of RAIT to members of the Board.
These rights are structured to attract, reward and retain selected Trustees who
are responsible for shaping and carrying out the long-range plans of RAIT. The
Plan is designed to reward designated Trustees for their efforts on behalf of
RAIT and to align the economic interests of such individuals with those of
RAIT's shareholders.

II.   DEFINITIONS. The following definitions shall apply for all purposes of the
Plan:

      2.01  "ACCOUNT" means a bookkeeping account established on RAIT's books
pursuant to Section 5.02. The Account reflects the number of Phantom Shares that
shall be credited to the Participant following an award under Section 5.01.

      2.02  "BENEFICIARY" means the person(s) designated in writing by a
Participant to receive any benefits payable under this Plan subsequent to the
Participant's death. RAIT shall provide a form for this purpose. In the event a
Participant has not filed a Beneficiary designation with RAIT, or no Beneficiary
survives the Participant, the Beneficiary shall be the Participant's estate.

      2.03  "BOARD" means the Board of Trustees of RAIT.

      2.04  "COMMITTEE" means the committee appointed by the Board to administer
the Plan.

      2.05  "EFFECTIVE DATE" means January 29, 2004.

      2.06  "FAIR MARKET VALUE" shall mean, per Share, the average of the
highest and lowest price of a Share on the New York Stock Exchange, or such
other national securities exchange as may be designated by the Committee, on the
applicable date, or, if there are no sales of Shares on such date, then the
average of the highest and lowest price of the Shares on the last previous day
on which a sale is reported.

      2.07  "GRANT AGREEMENT" means a written document issued by the Committee
to a Participant that shall specify the award of Phantom Shares to the
Participant as described in Section 5.01 and such other terms and conditions to
which the grant is subject.

      2.08  "PARTICIPANT" means any Trustee who is designated by the Committee
as eligible to participate in the Plan under Section 4.01 and receives an award
of Phantom Shares. In the event of the death of a Participant, the term shall
mean his or her Beneficiary. In the event of the incompetency of a Participant,
the term shall mean his or her personal representative or guardian.

      2.09  "PHANTOM SHARES" means a phantom interest in RAIT that is awarded to
a Participant to reflect Shares as provided in this Plan.
<PAGE>
      2.10  "PHANTOM SHARE VALUE" means the value of a Phantom Share as of a
particular date based on the Fair Market Value of a Share on such date.

      2.11  "PLAN" means the RAIT Investment Trust Phantom Share Plan for
Non-Employee Trustees as the same is set forth herein and as it may be amended
from time to time.

      2.12  "PLAN YEAR" means the calendar year.

      2.13  "RAIT" means RAIT Investment Trust, a Maryland real estate
investment trust.

      2.14  "SEPARATION FROM SERVICE" means the Trustee's separation from the
Board for any reason, including death or disability. A "Separation from Service"
shall be deemed to have occurred on the last day of the Trustee's membership on
the Board.

      2.15  "SHARES" means common shares of beneficial interest, par value
$0.01, of RAIT.

      2.16  "TRUSTEE" means a non-employee member of the Board.

III.  ADMINISTRATION AND DISCRETION. The Committee shall make grants of Phantom
Shares to the Participants. Further, the Committee shall have full power and
authority to interpret the Plan, to prescribe, amend and rescind any rules,
forms and procedures as it deems necessary or appropriate for the proper
administration of the Plan, to make any other determinations, including factual
determinations, and to take such other actions as it deems necessary or
advisable in carrying out its duties under the Plan. All decisions and
determinations by the Committee shall be final and binding on the Trustees, the
Beneficiaries, RAIT and any other persons having or claiming an interest under
the Plan. All decisions, actions and interpretations of the Committee required
under the Plan shall be in its sole discretion, not in a fiduciary capacity and
need not be uniformly applied to similarly situated individuals. No member of
the Committee shall be personally liable for any action taken with respect to
the Plan, and RAIT shall defend, indemnify, and hold each such individual
harmless, unless circumstances indicate the individual acted with gross
negligence or willful misconduct. The expenses incurred by the Committee in
administering the Plan shall be borne by RAIT.

IV.   PARTICIPATION.

      4.01  IN GENERAL. The Committee shall have the sole authority to
determine, within the limitations of the Plan, each Trustee who shall become a
Participant and who shall be granted Phantom Shares in any Plan Year, the number
of such Phantom Shares and such other terms and conditions relating to the
Phantom Shares if not otherwise specified in the Plan. Participation in the Plan
is conditional upon the Participant's acknowledgement, in writing or by
acceptance of any amount paid from the Plan, that all decisions and
determinations of the Committee shall be final and binding on the Participant,
his or her Beneficiaries and any other person having or claiming an interest
under the Plan on behalf of the Participant.

      4.02  FACTORS TO BE CONSIDERED. In determining whether an individual may
become a Participant under the Plan, the Committee shall take into consideration
the Trustee's present and potential contribution to the success of RAIT and such
other factors as the Committee may deem proper and relevant.

                                       2
<PAGE>
V.    AWARD OF PHANTOM SHARES.

      5.01  AWARD OF PHANTOM SHARES. For each Plan Year, the Committee shall
notify those Participants who are to receive an award of Phantom Shares and
inform such Participants of the number of Phantom Shares each has been awarded.
All Phantom Shares will be fully vested upon award to the Participant. The award
of Phantom Shares to a Participant shall be evidenced by a Grant Agreement that
shall provide the number of Phantom Shares awarded and the terms and conditions
related to the Phantom Shares so awarded. Such awarded Phantom Shares shall be
credited to each Participant's Account as specified in Section 5.02. No Account
shall be credited with a fractional Phantom Share.

      5.02  INTERESTS OF A PARTICIPANT. RAIT shall create an Account on its
books to reflect the number of Phantom Shares credited to each Participant;
provided, however, that no Participant or any other person shall under any
circumstance acquire any property interest in any specific asset of RAIT. In the
event a dividend is declared with respect to the Shares, a cash payment shall be
made to the Participant by RAIT equal to the amount of the dividend that would
have been distributed if those Phantom Shares credited to the Participant's
Account been Shares. Nothing contained in this Plan and no action taken pursuant
hereto shall create or be construed to create a fiduciary relationship between
RAIT or the Committee and any Participant or any other person. To the extent
that any person acquires a right to receive payment from RAIT hereunder, such
right shall be no greater than the right of any unsecured general creditor of
RAIT.

      5.03  REPORT TO PARTICIPANTS. The Committee shall annually provide each
Participant with a statement as to the Phantom Share Value of the Phantom Shares
credited to the Participant's Account as of the last day of the immediately
preceding Plan Year.

VI.   REDEMPTION OF RIGHTS. RAIT shall redeem Phantom Shares credited to a
Participant's Account upon a Participant's Separation from Service for any
reason. The redemption shall apply to all of the Phantom Shares then credited to
the Participant's Account based on the Phantom Share Value of such Phantom
Shares as of the date of the Participant's Separation from Service. Any such
redemption shall be paid within the forty-five (45) day period following the
date of the Participant's Separation from Service in a lump sum in cash.
Notwithstanding the immediately preceding sentence, if, after the Effective
Date, RAIT's shareholders authorize the issuance of Shares under the Plan, the
Committee may determine, in its sole discretion, that the redemption paid to the
Participant after the Participant's Separation from Service will be paid in
Shares not cash.

VII.  MISCELLANEOUS.

      7.01  TRANSFERABILITY. No Phantom Share issued under this Plan shall be
transferred, assigned, pledged, encumbered or exercised by the Participant, and
a Phantom Share may be redeemed during the lifetime of a Participant only by the
Participant.

      7.02  NO RIGHTS AS SHAREHOLDER. No Participant shall have any rights as a
shareholder of RAIT (or any subsidiary or affiliate), including the right to any
cash dividends (except as provided in Section 5.02), or the right to vote, as a
result of the award

                                       3
<PAGE>
      7.03  ADJUSTMENT TO PHANTOM SHARES. Unless determined otherwise by the
Committee, in the event of a share split, share dividend, reclassification,
reorganization, or other capital adjustment of Shares, the number of Phantom
Shares which have been issued under the Plan shall be adjusted in the same
manner as the Shares are adjusted.

      7.04  NO RIGHTS TO CONTINUED SERVICE. Nothing in this Plan, and no action
taken pursuant hereto, shall confer upon any Participant the right to provide
service to RAIT, or affect the right of RAIT to terminate the Participant's
service at any time for any reason.

      7.05  WITHHOLDING TAX. Notwithstanding any other provision of this Plan,
if required RAIT shall be entitled to withhold from, or in respect of, any
payment to be made upon the redemption of Phantom Shares, an amount sufficient
to satisfy all federal, state and local tax withholding requirements relating
thereto. Such withholding may also be made from other amounts due from RAIT to
the Participant or upon the occurrence of any other event that would cause the
Phantom Shares to be deemed taxable compensation to the Participant.

      7.06  NOTICES. Any notice hereunder to be given to RAIT or the Committee
shall be in writing and shall be delivered in person to the Secretary of RAIT,
or shall be sent by registered mail, return receipt requested, to the Secretary
of RAIT at RAIT's executive offices, and any notice hereunder to be given to the
Participant shall be in writing and shall be delivered in person to the
Participant, or shall be sent by registered mail, return receipt requested, to
the Participant at the Participant's last address as shown in the employment
records of RAIT. Any notice duly mailed in accordance with the preceding
sentence shall be deemed provided on the date postmarked.

      7.07  TERMINATION AND AMENDMENT OF THE PLAN/MODIFICATION OF PHANTOM
SHARES. The Plan may be terminated, modified or amended by the Committee at any
time, except with respect to any Phantom Shares then outstanding under the Plan;
provided, however, that the Committee may accelerate the redemption of any
Phantom Shares then outstanding as if a redemption were then being made under
Section VI. Nothing contained in the Plan shall preclude RAIT from entering into
any type of transaction such as a reorganization, merger or recapitalization, or
other change in its capital structure and any such transaction shall not require
the acceleration of payment under this Plan except to the extent expressly
provided in this Plan.

      7.08  OTHER BENEFITS. Payments under this Plan shall not be taken into
account as compensation in determining or calculating other benefits under any
other plan or program of RAIT which bases a benefit on compensation.

      7.09  MISCELLANEOUS.

            (a)   If RAIT shall find that any person to whom any payment is
payable under this Plan is unable to care for such person's affairs because of
illness or accident, or is a minor, any payment due (unless a prior claim
therefor shall have been made by a duly appointed guardian, committee or other
legal representative) may be paid to the spouse, a child, a parent, or a brother
or sister, or to any person deemed by RAIT to have incurred expense for such
person otherwise entitled to payment, in such manner and proportions as RAIT may
determine. Any such payment shall be a complete discharge of the liabilities of
RAIT under this Plan.

                                       4
<PAGE>
            (b)   This Plan shall be binding upon and inure to the benefit of
RAIT, its successors and assigns and the Participant and the Participant's
heirs, executors, administrators and legal representatives.

            (c)   This Plan shall be construed in accordance with, and governed
by, the laws of the State of Maryland.


                                       5