Manigault was born in Charleston, South Carolina in 1824. His parents were Joseph and Charlotte Manigault. His grandfather, Peter Manigault, was the richest person in British North America in 1770. Joseph Manigault's great-great-grandfather was Pierre Manigault  (1664–1729), a French Huguenot who was born in La Rochelle, France and settled in Charleston. His mother was both the daughter of Charles Drayton, a South Carolina Lt. Governor, and the granddaughter of Henry Middleton, the second President of the First Continental Congress, whose grandfather, Edward Middleton, emigrated from England via Barbados. Her uncle, Arthur Middleton, was one of the signers of the Declaration of Independence.