In 1862, Lincoln appointed him as military governor of Tennessee after most of it had been retaken. Soon after, he purchased Dolly's half-brother Sam. Dolly had three children—Liz, Florence and William. In 1857, Andrew Johnson purchased Henry, who was 13 at the time and would later accompany the Johnson family to the White House. Sam Johnson and his wife Margaret had nine children. Sam became a commissioner of the Freedmen's Bureau and was known for being a proud man who negotiated the nature of his work with the Johnson family.