During the American Civil War, the Richmond, Fredericksburg and Potomac became strategically crucial and suffered extensive damage from both armies. Robinson's firstborn son, Pvt. Leigh Robinson fought with Confederate artillery companies but survived the war, as did Conway Robinson Jr. (1852-1921), who was too young to fight. Two of their brothers died in Confederate military service. William Colston Robinson (1845-1863) died at Bristow in October 1863 and Sgt. Cary Robinson (1843-1864) fought with the 6th Virginia Infantry and died at the Battle of Burgess Mill in October 1864.