In October 1863, Clem was captured in Georgia by Confederate cavalrymen while detailed as a train guard. The Confederates confiscated his U.S. uniform, which reportedly upset him terribly, including his cap, which had three bullet holes in it. He was included in a prisoner exchange a short time later, but the Confederate newspapers used his age and celebrity status for propaganda purposes, to show "what sore straits the Yankees are driven when they have to send their babies out to fight us." After participating with the Army of the Cumberland in many other battles, serving as a mounted orderly, he was discharged in September 1864. Clem was wounded in combat twice during the war.