After Robinson's birth in Bellevue, Ohio and while still a toddler, his family moved to St. Louis, Missouri where Robinson's father, Bradbury Norton Robinson, Sr., became general baggage agent for the Missouri–Kansas–Texas Railroad. The senior Robinson spent most of his adult life working for railroads. Born in Lowell, Massachusetts, he served one year as a sergeant in Company C of the 6th Massachusetts Infantry at the opening of the Civil War. During the Baltimore Riot of April 19, 1861, one week after Fort Sumter, the 6th Massachusetts became the first Union unit to take casualties in action. After his discharge, he moved his family to Missouri in 1862 to participate in the construction of the Missouri Pacific Railroad from St. Louis to Kansas City.