Armistead, known to friends as "Lo" (for "Lothario"), was born in the home of his great-grandfather, John Wright Stanly, in New Bern, North Carolina, to Walker Keith Armistead and Elizabeth Stanly. He came from an esteemed military family. Armistead was of entirely English descent, and all of his ancestry had been in Virginia since the early 1600s. The first of his ancestors to emigrate to North America was William Armistead from Yorkshire, England. Armistead's father was one of five brothers who fought in the War of 1812; another was Major George Armistead, the commander of Fort McHenry during the battle that inspired Francis Scott Key to write "The Star-Spangled Banner", which would later become the national anthem of the United States. On his mother's side, his grandfather John Stanly was a U.S. Congressman, and his uncle Edward Stanly served as military governor of eastern North Carolina during the Civil War.