With the outbreak of the American Civil War, he was appointed Quartermaster General of the state of Georgia, a position he continued to hold after the war's end. He remained active in Georgia state politics into the Reconstruction period. Foster was also elected first mayor of Eastman, Georgia. He served in the Georgia House of Representatives, and was elected to the state senates of both Georgia and Alabama.