Slaveholders wanted to get free people of color out of the South, where they were thought to threaten the stability of the slave societies