Before the Norman Conquest, the lands around were controlled by Gilpatrick. In 1069, William the Conqueror granted them to his Breton cousin Alan Rufus, who built a wooden motte-and-bailey castle above the town. By the time of the 1086 Domesday Book, Alan had passed the castle to his brother Ribald. Its earthworks are still visible at William's Hill. Alan also built the castle at Richmond.