Clifford Castle Clifford Castle is a castle in the village of Clifford which lies four miles to the north of Hay - on - Wye in the Wye Valley in Herefordshire , England ( ) . Early Norman castle and planned settlement . The early motte and bailey castle was built on a cliff overlooking a ford on the River Wye in 1070 by William Fitzpond . The castle was intended to provide protection for a planned Norman settlement above the River Wye of quite a substantial size , with c . 200 plots being intended , running uphill for half a mile towards Llanfair , where the church was sited on the hilltop . St Mary ' s church remains today , much altered in the Victorian era . It has a wooden 13th century effigy of what appears to be a clerical figure . It has been said to resemble another effigy , of Bishop Peter of Aigueblanche , in Hereford Cathedral . A short distance downhill from Llanfair is Priory Farm , built on the site of a Cluniac Priory , believed to have been founded in 1130 . Priory Farm is a very substantial farm building and presumably used much of the stone already on the site in its construction . It is also surrounded by stone wall field boundaries . In the Domesday Survey the settlement was recorded as Cliford . Maps from three centuries later show Herefordshire having three significant settlements ; Hereford , Wigmore and Clifford . The position of the castle adjacent to the River Wye enabled the seasonal flooding of the river to fill the flood plain around the castle , forming a shallow lake or marsh with the assistance of a dam on the western or upstream side of the site . Changing hands . When his heir , Roger de Breteuil , 2nd Earl of Hereford , forfeited his lands for rebellion against the King in 1075 , the castle was granted to Ralph Tosny who held it directly from the Crown . From Ralph , it passed to his son - in - law Richard des Ponts ( more correctly Richard Fitz Pons ) . Richard ' s son Walter Fitz Richard later took the name of Walter de Clifford after he seized the castle from its Tosny overlord before 1162 . Much of the stone castle would seem to have been built before 1162 as it much resembles the Tosny Conches Castle in Normandy . The King ' s mistress . Clifford Castle was the home of Walter Fitz Richard ' s daughter Rosamund Clifford , who was also known as the Fair Rosamund . She was the mistress of Henry II until her death in 1176 or 1177 when she was buried at Godstow in Oxfordshire . A property near the castle is called Rosamund House today . The Clifford Rebellion . In 1233 Walter III de Clifford the grandson of Walter Fitz Richard rebelled against the autocratic rule of King Henry III . In early September the king came and besieged Clifford castle , forcing the garrison to surrender after just a few days . Walter Clifford then made his peace with the Crown and led his troops against Prince Llywelyn ab Iorwerth his father - in - law . Walter ' s volte - face was even more disreputable as Prince Llywelyn had just entered the field to fight the king in support of Walter . Twenty years later Walter nearly rebelled again , when , in a fit of anger , he forced a royal messenger to eat a royal writ , together with the royal wax seal the size of a dinner plate . For this Walter lost many of his Marcher Lord privileges . The ruins . Clifford castle now consists of a great motte as constructed by the men of William Fitz Osbern in the late 1060s . This was later sub - divided and the eastern part was crowned by an ovoid shell keep with five D - shaped towers in its circuit . Its north wall appears to overlie part of William Fitz Osbern ’ s original hall . To the east of the motte is the castle bailey . Most of the walls of this structure have disappeared , but centrally are the remains of a great twin - towered gatehouse probably of the mid - thirteenth century . To the west of the castle is a broken earthwork dam which would have flooded the valley to the south of the castle . With the River Wye to the north the fortress would have been surrounded by water on all sides except for the east . As such it would have been a very difficult fortress to take by storm . Clifford Castle is within the grounds of a family home and has no public access . Destruction . During the rebellion of Owain Glyndŵr the castle was destroyed in 1402 by Welsh forces . Modern ruin . Since its last use in the 15th century it has fallen into ruins with much of the stone pillaged for the older properties seen in the village today . Large dressed stone blocks can be seen in garden walls . The castle is currently on the English Heritage Heritage At Risk Register 2010 . The present owners are working closely with English Heritage to implement a maintenance policy , followed by a program of works to stabilise the current structure and prevent further decay . Most at risk are Rosamund ' s Tower and the arched window in the Hall . A branch of Great Western Railway ran just to the north of the castle , between the castle and the river . The next station to the north on this line was nearby in Clifford , and the next station to the south was Hay - on - Wye railway station . This station was also the terminus of the Great Western line — beyond the station the railroad was the Golden Valley Railway .