Davidson was born in Fairfax County, Virginia. He was the son of William B. Davidson, an artillery officer in the United States Army, and the former Elizabeth Chapman Hunter. He was the eldest of four sons; his brothers were Hunter, Roger, and Charles. His father died from disease while serving in Florida during the Second Seminole War in 1840. His mother died ten years later, shortly after remarrying.