Nelson was born in 1852 to Julia Ann Rogers Nelson and her husband, Keating Lewis Simons Nelson (1819-1898) at Cobham, a plantation in Albemarle County, Virginia. He had two older brothers, Dr. Hugh Nelson (1842-1903) and Francis K. Nelson (1845-1864), as well as sisters Margaret Nelson (1843-1858), Elizabeth Harrison Nelson Mason (1846-1945) and Celia Hamilton Nelson (1848-1929) and younger brothers Rev. Keating Simons Nelson Jr. (1856-1919) and William Meade Nelson (1859-1843). His father owned a plantation, and about 35 enslaved people in 1850. His grandfather was U.S. Congressman Hugh Nelson (1768-1836) and his paternal great-grandfather Thomas Nelson (1738-1789) had signed the Declaration of Independence and served as Governor of Virginia. An uncle of the same name, Cleland Kinloch Nelson (1814-1890), was an Episcopal priest in Maryland. His elder brothers both served in the Confederate States Army during the American Civil War. Hugh Nelson rose to become a captain in the 4th Virginia Cavalry and on the staff of General R.S. Ewell, and was commended for his actions at the Battle of Port Republic. His other brother, Francis K. Nelson, enlisted in the Rockbridge Light Artillery battery, transferred to the 1st Virginia Cavalry, then the 2nd Virginia Cavalry and was promoted to sergeant a month before being mustered out and dying in May, 1864.