Since Andrew did not resemble either of his siblings, there are rumors that he may have been fathered by another man. Polly Johnson eventually remarried to a man named Turner Doughtry, who was as poor as she was. Johnson was born in poverty in Raleigh, North Carolina and never attended school. He was apprenticed as a tailor and worked in several frontier towns before settling in Greeneville, Tennessee. The brothers went to Carthage, North Carolina, where Andrew Johnson worked as a tailor for several months.