Robert E. Lee became displeased with Drayton's performance. Drayton had failed to get his brigade into action at Second Bull Run, and it was then driven from the field in panic at both South Mountain and Antietam. Lee complained that Drayton was unable to keep his brigade properly organized, failing to file reports and returns, and that in each battle the brigade had been engaged in most of its colonels were AWOL, leaving the regiments commanded by inexperienced majors and captains. The brigade was broken up and its regiments transferred to other brigades. Drayton himself was transferred to the Western Theater to command a brigade in Sterling Price's army. During the final two years of the war, he mainly performed administrative duties in the Trans-Mississippi Theater, although he did briefly command a division in early 1864.