De Russy was born in 1795 and grew up in New York City. His father Thomas De Russy had served in the French Navy, moving to the American Navy during the American Revolutionary War (France was at that point allied with the rebels). After returning to France, he left in 1786 as tensions were rising that led to the French Revolution and gone to family lands in Saint-Domingue, the wealthiest French colony in the Caribbean. He settled in Cap Français and married a French colonist, Madeleine Baissiere, there in 1787. Their first two sons, Rene Emile and René Edward De Russy, were born on the island. In 1792, the family had to leave quickly because the city was burning from the Haitian Revolution. They lost all their property but safely reached New York City.