Born in Waynesboro, Virginia, to the former Sarah Jane Garber and her husband Lewis Harman (1794-1840), his father died when he was 16 years old. His maternal grandfather, John Cunningham, had been a member of Staunton's Committee of Safety in the American Revolutionary War, as well as captain of a cavalry company in the Virginia militia. John received a private education suitable to his class. He had an elder brother Michael Garber Harman (1823-1877) and younger brothers William Henry Harman (1828-1865), Asher Waterman Harman (1830-1895) and Thomas Lewis Harman (1831-1861). His four brothers would likewise fight in the Confederate States Army, but mostly as quartermasters, and two died. William H. Harman, a prominent local lawyer, would begin the war as a Brigadier General of the Virginia militia and helped secure the Harper's Ferry arsenal immediately after secession. However, health problems caused him to be assigned as an aide-de-camp to Brig. Gen. Edward Johnson and to command reservists in the Shenandoah Valley; he would die in Battle of Waynesboro. Their youngest brother T.L. Harman would become a Lieutenant in the Staunton Artillery of the Confederate States Army, but die in September 1861 of typhoid fever in Prince William County, Virginia.