The original castle on the site, a Norman motte and bailey to the east of the existing ruins, is first recorded in 1262. This castle was later superseded by the current structure. The estate, including the motte and bailey, came into the ownership of the family in 1245, and was in possession of Bogo de Clare (1248-1294), the third son of Richard de Clare, for some years before his death. In 1369, the community of Tregrug was severely affected by one of the later outbreaks of the plague. A new ambitious noble residence, enclosed in a high stone wall, with defensive banks and ditches, was started in the early 14th century, probably by Gilbert de Clare. The castle passed briefly into the hands of the Despenser family; it may have been Hugh Despenser who began the construction of the late medieval structure from which most of the now-standing ruins derive. It was attacked during the revolt of Llywelyn Bren in 1316.