In August 1863, army recruiter George Luther Stearns organized and mustered the 3rd Colored Infantry in Philadelphia, Pennsylvania. Conant was promoted to the rank of captain and was placed in command of Company H. The 3rd Colored fought at the Second Battle of Fort Wagner, capturing the fort shortly after the famous assault by the 54th Massachusetts. In 1864, the regiment assisted in the capture of Jacksonville, Florida, and occupied much of coastal Florida. After the end of the war, the 3rd Colored was assigned to assist the Department of Florida in reorganizing the state.