The castle was seized from the de Berminghams permanently in the Cromwellian era (1650s). It was occupied by the Ouseley family (including the famous preacher Gideon Ouseley) until the 19th century.