T. B. Walker was the son of Platt Walker and Anstis Keziah (Barlow) Walker (1814–1883), a sister of New York State Senator Thomas Barlow (1805–1896). He was born in Xenia, Ohio, in 1840, where in 1849 he got his first job in a bakery cutting biscuits. He had accompanied his parents and siblings west from New York when his father died of cholera in 1849 at Westport, Missouri, on their way to the California gold fields to seek their fortune.