People across the income spectrum are moving out of the area, but low- and middle-income earners are leaving at higher rates.