Free people of color were far more likely to be of mixed racial background than slaves