He was promoted to brigadier general on August 29, 1863, to replace the deceased Paul Jones Semmes and commanded the brigade during the Mine Run Campaign. In September 1863, Bryan's Brigade traveled with James Longstreet to Georgia to reinforce the Army of Tennessee and they fought in the Battle of Chickamauga and the Knoxville Campaign. In the spring of 1864 they returned to the Army of Northern Virginia before the start of the Overland Campaign. Bryan fought at Cold Harbor and the beginning of the Siege of Petersburg. On September 20, 1864, he resigned his commission due to chronic ill health and returned home to Georgia.