Hargest was survived by his wife and three children. A fourth child, Geoffrey Hargest, had been killed on 30 March 1944, aged 22 years, during the Battle of Monte Cassino while serving with the 23rd Battalion. He is buried in the Cassino War Cemetery. Another son was killed while on active service during the Malayan Emergency. James Hargest High School, an educational facility in Invercargill, is named after him.