His father Alexander Erskine, 5th Earl of Kellie, was incarcerated at Edinburgh Castle for supporting the Jacobites in the Jacobite rising of 1745. His mother, Janet Pitcairn, was the daughter of a celebrated physician and poet. Born in Edinburgh, Thomas attended the Royal High School before leaving around 1752 for Mannheim in Germany to study under the elder Johann Stamitz. He returned to Scotland in 1756 as a virtuoso violinist and composer, nicknamed "Fiddler Tam".