Taylor was born into slavery in Shreveport, Louisiana on November 7, 1849 to enslaved parents, Zed and Betty Taylor. He was said to tell his mother that he wanted to be a preacher at the age of four after attending a sermon in Lexington, Kentucky. In 1864, during the American Civil War he enlisted in Company G of the 116th Regiment Infantry U.S. Colored Troops as a drummer and was at the Siege of Petersburg, fall of Richmond, and surrender of Lee. After the end of the war in 1865, his regiment did garrison duty in Texas and New Orleans where he was mustered out a free man. After the war he became a marble engraver and moved to Louisville, Kentucky. Whites in Kentucky refused to work with him, and he instead found work as a train porter for the Louisville &amp; Chattanooga Railroad where he worked for four years.