The farmers who did not have slaves – yeomen – made up the largest group of whites in the South. Not all Southern whites owned land.