Stewart was born in Milton, Dunbartonshire, Scotland, a village fifteen miles west of Glasgow. Stewart's family were Austin, and later Jaguar, car dealers and had built up a successful business. His father had been an amateur motorcycle racer, and his brother Jimmy was a racing driver with a local reputation who drove for Ecurie Ecosse and competed in the 1953 British Grand Prix at Silverstone.