He was admitted to the bar in 1862, and thereafter served in the Union Army in the American Civil War. He enlisted from Boston as a 2nd lieutenant in Company B, 4th Battalion, Massachusetts Infantry on May 27, 1862, was mustered out a few days later, and commissioned into Company H, 3rd Massachusetts Cavalry on October 7, 1862. He left that unit to accept a commission as a major in the U.S. Volunteers Adjutant General Department on July 25, 1864. Gray was wounded at the Third Battle of Winchester on September 19, 1864, and resigned from the Army on July 14, 1865.