While recovering from a wound during the Battle of Upperville in the American Civil War in Enterprise, Mississippi, O'Ferrall met Annie Hand, whom he married on February 8, 1865 before returning to active duty. They had two children. After her death, in 1881 he married Jennie Wickliff Knight (1846–1908), with whom he had four more children, and who survived him.