In 1751, Simon McTavish was born at Stratherrick in the Scottish Highlands, the son of John McTavish (1701–1774), tacksman of Garthbeg, who bore the arms of the McTavishes of Garthbeg. His mother, Mary Fraser (1716–1770) of Garthmore, was descended through Simon Fraser of Dunchea and the Frasers of Foyers, from an illegitimate son of the 1st Lord Lovat. McTavish's father had fought as an officer with the Jacobite armies at the Battles of Culloden and Falkirk Muir, and he was one of the few who were specifically named as to "not" receive a pardon from George II after the Jacobites were defeated.