As much as James Madison relied on enslaved black labor, he loathed it as an institution.