Complex sentence: Many blacks left Manhattan permanently (many moving to Brooklyn), that by 1865 their population fell below 10,000, the number in 1820.
Split into simpler sentences: Many blacks left Manhattan permanently (many moving to Brooklyn). By 1865 their population fell below 10,000, the number in 1820.