Moss was born in London, son of Alfred Moss, a dentist of Bray, Berkshire, and Aileen (née Craufurd). His grandfather was Jewish, from a family that changed their surname from Moses to Moss. He was brought up at "Long White Cloud" house on the south bank of the River Thames. His father was an amateur racing driver who had come 16th in the 1924 Indianapolis 500. Aileen Moss had also been involved in motorsport, entering prewar hillclimbs at the wheel of a Singer Nine. Stirling was a gifted horse rider as was his younger sister, Pat Moss, who became a successful rally driver in her own right and also married Swedish rally driver Erik Carlsson.