Rosser was commissioned a first lieutenant and became an instructor to the famed "Washington Artillery" of New Orleans. He commanded its Second Company at the First Battle of Manassas in July 1861. He was noted for shooting down one of George B. McClellan's observation balloons, a feat that won him promotion to captain. He commanded his battery during the Seven Days Battles of the Peninsula Campaign, and was severely wounded at Mechanicsville. Rosser was promoted to lieutenant colonel of artillery, and a few days later to colonel of the 5th Virginia Cavalry.