Spencer-Smith was born on 17 March 1883, in Streatham, Surrey, England. He shared a birthday with Lawrence Oates, who died on his return from the South Pole with Robert Falcon Scott on the "Terra Nova" Expedition. His sister, Joan Spencer-Smith (1891–1965), was a notable New Zealand Anglican deaconess and lecturer.