Elijah White was born in the area of Poolesville, Maryland. In 1855, White moved to Missouri to fight in the border wars with Kansas. The following year, he returned home and bought the Ball farm across the Potomac River in Loudoun County, Virginia, in the vicinity of the Big Spring north of Leesburg.