Over 200,000 were living in New York by 1860, upwards of a quarter of the city's population.