The stone castle was built around 1220 and contributed to the development of the town. It was surrounded by a moat and included a watergate giving access to the quay. In the 13th and 14th centuries the castle was involved in the Second Barons' War and Despenser War.