He was promoted to brigadier general on May 27, 1861. In November, 1861, he was transferred to the Department of the Gulf under Major General Benjamin F. Butler, and Phelps left on an expedition to the Gulf of Mexico where his regiment took military possession of Ship Island, Mississippi. His regiment supported Commodore David Farragut's fleet in forcing open the Lower Mississippi in April, 1862. They participated in taking possession of Fort Jackson and St. Philip, Louisiana at the Battle of Forts Jackson and St. Philip, which was instrumental in the capture of New Orleans, the largest city of the Confederacy, on May 1, 1862.