During the Civil War, Ullman became colonel of the 78th New York Infantry Regiment. Captured at the Battle of Cedar Mountain in August 1862, he was detained at Libby Prison until he was paroled two months later. He later approached President Abraham Lincoln about the possibility of enlisting African Americans as soldiers. After subsequent discussion, in January 1863 Ullman was promoted to brigadier general and sent to Louisiana where he raised five regiments of African Americans as soldiers in a unit that was designated the Corps d'Afrique. He now commanded a brigade made up of those colored infantry regiments and a regiment of colored engineers.