Joseph E . Brown Joseph Emerson Brown ( April 15 , 1821November 30 , 1894 ) , often referred to as Joe Brown , was the 42nd Governor of Georgia from 1857 to 1865 , and a U . S . Senator from 1880 to 1891 . Governor Brown was a leading secessionist in 1861 , taking his state out of the Union and into the Confederacy . A former Whig , and a firm believer in states ' rights , he defied the national government ' s wartime policies . He resisted the Confederate military draft , and tried to keep as many soldiers at home as possible ( to fight off invaders , he said ) . He denounced Confederate President Jefferson Davis as an incipient tyrant . Brown challenged Confederate impressment of animals , goods , and slaves . Several other governors followed his lead . Biography . Brown was born 15 April 1821 in Pickens County , South Carolina to Mackey Brown and Sally ( Rice ) Brown . At a young age he moved with his family to Union County , Georgia . In 1840 , he decided to leave the farm and seek an education . Brown , with the help of his younger brother James and his father ' s plow horse , drove a yoke of oxen on a 125 - mile trek to an academy near Anderson , South Carolina , where the impoverished Brown exchanged the oxen for eight months ' board and lodging . In 1844 , Brown moved to Canton , Georgia , where he served as head - master of the academy at Canton . He went on to study law , and in 1847 , he opened a law office in Canton . Brown was elected to the Georgia state senate in 1849 and soon became a leader of the Democratic Party in Georgia . He was elected state circuit court judge in 1855 and governor in 1857 . As governor , he diverted state railroad profits to Georgia ' s public schools . He became a strong supporter of secession from the United States after Lincoln ' s election and South Carolina ' s secession in 1860 . Once the Confederate States of America was established , Brown spoke out against expansion of the Confederate central government ' s powers . He denounced Davis in particular . Brown even tried to stop Colonel Francis Bartow from taking Georgia troops " out " of the state to the First Battle of Bull Run . He objected strenuously to military conscription by the Confederacy . After the loss of Atlanta , Brown withdrew the state ' s militia from the Confederate forces to harvest crops for the state and the army . When Union troops under Sherman overran much of Georgia in 1864 , Brown called for an end to the war . After the war , Brown was briefly held as a political prisoner in Washington , D . C . He was chief justice of the Supreme Court of Georgia from 1865 to 1870 , when he resigned to become president of the Western and Atlantic Railroad . He supported President Andrew Johnson ' s Reconstruction policy , even becoming a Republican " scalawag " for a time . After Reconstruction , he became a Democrat again and was elected to the U . S . Senate in 1880 by the state legislature , according to the laws of the time . Soon after his election to the Senate , Brown became the first Georgia official to support public education for all children — not a popular position at the time . He was re - elected in 1885 , but retired in 1891 due to poor health . While Brown ' s political supporters claimed that he " came to Atlanta on foot with less than a dollar in his pocket after the war and . . . made himself all that he is by honest and laborious methods " , most of his enterprises stemmed from his political connections . He amassed a fortune by using convicts leased from state , county and local government in his coal mining operations in Dade County . His exploitation of leased labor began in 1874 and continued until his death in 1894 , a period that coincided with " the high tide of the convict lease system in Georgia . In 1880 Brown , whose fortune could be estimated conservatively at one million dollars , netted $ 98 , 000 from the Dade Coal Company . By 1886 , Dade Coal was a parent company , owning Walker Iron and Coal , Rising Fawn Iron , Chattanooga Iron , and Rogers Railroad and Ore Banks and leasing Castle Rock Coal Company . An 1889 reorganization resulted in the formation of the Georgia Mining , Manufacturing and Investment Company . This rested largely on a foundation of convict labor . ” " The most powerful politician in Georgia from the 1860s until his death , Brown , still contemptuous of the Emancipation Proclamation , filled his mines with scores of black men forced into the shafts against their will . A legislative committee visiting the sites the same year [ Joel ] Hurt bought them said the prisoners were ' in the very worst condition . . . actually being starved and have not sufficient clothing . . . treated with great cruelty . ' Of particular note to the visiting officials was that the mine claimed to have replaced whipping with the water cure torture - - in which water was poured into the nostrils and lungs of the prisoners - - because it allowed miners to ' go to work right away ' after punishment . " He died in 1894 in Atlanta , Georgia . His statue , in which he is accompanied by his wife , is on the capitol grounds , and his towering tombstone is in Oakland Cemetery . Neither monument mentions his connection with the exploitation of coerced and abused prisoners . His son , Joseph Mackey Brown , would also become governor of Georgia ( twice ) . Joseph E . Brown Hall on the campus of the University of Georgia in Athens is named in his honor . The building was completed in 1932 .