He entered the Confederate army as colonel of the 3rd Virginia Infantry Regiment. He was promoted to brigadier general on April 16, 1862. His brigade fought in the Peninsula Campaign and at Second Manassas, where it became detached in the swirling fighting and temporarily operated under Stonewall Jackson. Pryor's command initially consisted of the 2nd Florida, 14th Alabama, 3rd Virginia, and 14th Louisiana. During the Seven Days Battles, the 1st (Coppens') Louisiana Zouave Battalion was temporarily attached to it. Afterwards, the Louisianans departed and Pryor received two brand-new regiments; the 5th and 8th Florida Infantry. As a consequence, it became known as "The Florida Brigade". At Antietam on September 17, 1862, he assumed command of Anderson's Division in Longstreet's Corps when Maj. Gen. Richard H. Anderson was wounded. Pryor proved inept as a division commander, and Union troops flanked his position, causing them to fall back in disorder.