Duchy of Saxe-Meiningen
Ganz was born on August 18, 1838 in the German town of Walldorf, Thuringia in the then Duchy of Saxe-Meiningen, to Jewish parents Meyer and Hannah Ganz. He was educated in his home country before being apprenticed to a tailor at age 14. Ganz immigrated to the United States in 1858, working as a journeyman tailor in New York City and Philadelphia before settling in Cedartown, Georgia.