The castle was sacked by the army of Oliver Cromwell in 1653, during the Wars of the Three Kingdoms. The 4th Earl of Perth was Lord Chancellor of Scotland under King James VII. He began the mansion house in 1689, before being imprisoned following the deposition of King James by William of Orange. He later fled to the exiled Jacobite court in France. The Drummonds continued to support the Jacobite cause in the Jacobite uprisings of 1715 and 1745. The family retained control of the estate until 1750 when the Drummond properties were declared forfeit and seized by the state. The estate was managed by the Commissioners for Forfeited Estates until 1784, when it was sold to Captain James Drummond (later created 1st Baron Perth). He began a number of improvements that were continued by his daughter Sarah and her husband, The 22nd Baron Willoughby de Eresby (1782–1865). These included the formal gardens and terraces in the 1830s. Queen Victoria visited the gardens in 1842.