Carolina Oliphant was born at the Auld Hoose, Gask, Perthshire (her father's ancestral family home) on 16 August 1766. She was the fourth child of the three sons and four daughters of Laurence Oliphant (1724–1792), laird of Gask, and his wife Margaret Robertson (1739–1774); her mother was the eldest daughter of Duncan Robertson of Struan, the chief of Clan Donnachie, which fought on the Jacobite side in the uprisings of 1715 and '45. Her father was also a staunch Jacobite, and she was given the name Carolina in memory of Prince Charles Edward Stuart.