Palmer was born in Richmond, Virginia. Prior to the outbreak of the Civil War, he was a merchant there. When the Civil War began, Palmer joined the First Virginia Infantry Regiment as a private. At Williamsburg Palmer, by now a Major, took over command of the regiment when his colonel was wounded; however Palmer himself got shot in the arm. In October 1862, he joined the staff of Confederate General A.P. Hill. His shoulder was dislocated in the same friendly fire attack at Chancellorsville that mortally wounded Gen. Thomas "Stonewall" Jackson. Palmer spent the rest of the War with Hill, eventually rising to be the chief of staff for the Army of Northern Virginia's Third Corps.