The castle was used as a prison from the 1650s, and the garrison increased in 1658, and in 1662 it returned to the Crown. George Fox (1624–1691), founder of the Religious Society of Friends was imprisoned there from April 1665 to September 1666 for religious activities viewed as troublesome for Charles II (reigned 1660–1685). The castle declined again: James II (reigned 1685–1688) did not garrison it, he gambled that its defences would be sufficient to resist any Dutch invasion, but the town was seized for William of Orange during the Glorious Revolution that ousted James.