The students receive a lot of city services from the city, and their presence creates the need for provision of special services that makes the area they live in more expensive to serve than predominately family occupied areas. As one example, students tend to have more drivers and cars per living unit than a family unit. This means more police are necessary to maintain traffic and parking order in the area. It is not our intention to exploit students, it is simply our objective to have them pay for the services they use. Indeed this car example also undermines the inelastic demand argument.
Students are generally very mobil.
That would suggest that they have a lot of good apartment options outside of the immediate university area. With lots of good substitutes available, their demand for apartments in the immediate area is probably fairly elastic. Thus, contrary to your argument, the demand might indeed be very inelastic. And furthermore, moving further away from campus might provide lots of substitutes, but it increases the cost of one's living circumstance significantly in non- rental ways. Lots of us have cars, but on most days we leave our car home and walk to campus in 5 minutes.