African Americans would get on ships and go wherever the ship was going. It was the closest free city to the slave South. The Philadelphia of the period between 1790 and 1830 is what historians have been fond of calling a walking city. It's a city without public transportation, and therefore everybody who lives in it has to be within walking distance of where they work, or where they do whatever they're doing. As a consequence, the arrangement of the city was generally what's called vertical segregation, as much by class as by race.
Owners live on the main streets.
Workers live in the narrow streets and alleys behind. The working population tends to be German, Irish, African American. And they live roughly on the same blocks, which is not the same thing as an integrated neighborhood, but they would have come in contact with each other. They would not have necessarily spent their leisure time in each other's company. They would have been mostly polite to each other on the streets.