Given by William the Conqueror to Gilbert de Gant, De Gant lived in a house "without the town" named Le Burlyn (Old French for wool house), regarded to be built on the site where now stands Low Hall, the manor of Hunmanby became one of the most powerful in the North of England. His son Walter founded Bridlington Priory. Hunmanby was the site of a castle (fortress in some descriptions) which was built by Gilbert de Gant. It was destroyed during the "Battle of Hunmanby" by William le Gros (the Earl of York) and the Constable of Chester, Eustace fitz John during a period of history known as The Anarchy. The site of the castle is now known as Castle Hill.