Carroll was promoted to the rank of captain in the 10th U.S. Infantry shortly after the Civil War erupted. In December 1861, he was appointed as colonel of the 8th Ohio Infantry, a three-years' regiment that exclusively saw duty in the Eastern Theater. During the 1862 Valley Campaign, Carroll commanded the 4th Brigade in James Shields' division. Carroll commanded the vanguard of the Union army at the Battle of Cross Keys. He was commended for his performance at the Battle of Cedar Mountain. Transferred with this brigade to the Northern Virginia area, he was severely wounded in the chest in a fight near the Rapidan River. He recovered in time to resume his field command in III Corps before the Battle of Fredericksburg.