In 1813, Ruffin married Susan Hutchings Travis of Williamsburg. The couple moved to a plantation that Ruffin inherited from his namesake grandfather, at Coggin's Point, along the James River in Prince George County, which was noted as the departure point of Benedict Arnold after he switched allegiances and raided along the James River (including Richmond) in 1781. They had eleven children before Susan Ruffin died in 1846, although most would die before the American Civil War ended and their father committed suicide: Edmund Ruffin Jr. (1814-1875), Agnes Ruffin Beckwith (1816–1865), Julian Calx Ruffin (1821–1864, who died at the Battle of Drewry's Bluff), Elizabeth Ruffin Sayre (1821–1860), Rebecca Ruffin Bland (1823–1855), Jane Ruffin Dupuy (1829–1855), Charles Lorraine Ruffin (1832–1870; VMI graduate who survived the war), Ella Ruffin (1832–1855), Ann Ruffin (1844–1863) and George Champion Ruffin (1845-1913).