His father, who had been a loyal support of Charles I of England during the English Civil War, died in 1655, and Esmé and his mother went into exile in France.
Его отец Джеймс Стюарт, родственник и верный сподвижник короля Англии Карла I Стюарта во время Гражданской войны в Англии, скончался в марте 1655 года.