His mother was the only child of his maternal grandparents, Maj. Luther Stoddard (1746–1804) and Abigail Porter (1763–1797), who married in 1781. His grandmother's father was Dr. Joshua Porter (1730–1825), a 1754 graduate of Yale, fought in the Revolutionary War as a colonel. He was at the head of his regiment in October 1777 when John Burgoyne surrendered his 6,000 men after the Battles of Saratoga. After the war, he was elected to various official positions for forty-eight consecutive years. His maternal grand-uncles included Augustus Porter (1769–1849), a member of the New York State Assembly, and Peter Buell Porter (1773–1844), the United States Secretary of War. After his grandmother's death, his grandfather married Mary Wheeler (1765–1853).