James Ellyson was educated by tutors and in private schools. When he reached legal age in 1863, he left Hampden-Sydney College and enlisted to fight for the Confederacy in the Second Company of Richmond Howitzers, where he served until surrendering with his company at Appomattox Courthouse near the war's end.