To call it pure risk aversion is unsatisfactory. First, the reasons relate to commitment. You cannot duck in and out of RE investments at will. You need fortitude. You need sizeable entry costs.
Most people have trouble parting with that much money all at once.
You need to submit to a rigorous loan process, and the tribulations of an escrow period. Real estate is also a physically large asset and this fazes many people. Each piece of RE is a unique animal and therefore an investor must personally inspect and analize each potential investment individually. After actually acquiring the property, one has to look after it, and deal with tenats. This is not something for the faint-hearted.