He was of English, Scots-Irish, and Irish ancestry. He had a brother William, four years his senior, and an older sister Elizabeth, who died in childhood. Johnson's birth in a two-room shack was a political asset in the mid-19th century, and he would frequently remind voters of his humble origins. Polly Johnson worked as a washerwoman and became the sole support of her family. Her occupation was then looked down on, as it often took her into other homes unaccompanied. Since Andrew did not resemble either of his siblings, there are rumors that he may have been fathered by another man.