They lived in Albany , New York and had four children .
When her brother retired as the President , Mary went to live in Albany , New York .