Charles was born in 1701, most likely at the Jacobite Court at the Château de Saint-Germain-en-Laye, in France. He was the eldest of the five sons of Arthur Dillon and his wife Christina Sheldon. His father was born in 1670 in Ireland, fought for the Jacobites in the Williamite War and had gone to France as the colonel of Dillon's Regiment with the Irish Brigade in April 1690 when Irish troops were sent to France in exchange for French troops sent to Ireland with Lauzun. His father was a younger son of the 7th Viscount Dillon. His father's family was Old English Irish and descended from Sir Henry Dillon who came to Ireland with Prince John in 1185.