The castle was first mentioned in a certificate of November 2, 1269 documenting a loan from Vranke Stozep van Hildegardsberg. Later in 1343, Heer Kerstant van den Berge owned the church and the castle. However, in 1426, both church and castle were destroyed by the armies of Jacoba of Bavaria, during the Hoekse and Kabeljauwse quarrels. Remnants of the castle dungeon may still be found next to the Hillegondakerk in a corner of the cemetery, which dates from around 1500 in its present form. The structure was designated a national monument in September 1973.