The castle was a residence of the Masovian Dukes up until the fifteenth century. Due to a partial removal of the hillside, in 1532, the castle suffered damage, which was rebuilt. In 1538, the Masovian dukes had resided in the newly built palace, outside the castle walls, allowing the castle complex to be given to the Benedictines. During the times of the wars against the Swedes, the castle was severely damaged, first in 1657, and then in 1705. After the castle's reconstruction, the castle complex became a Baroque architectural style Benedictine Abbey, which existed until 1781. When after the Partitions of Poland, Prussia took over the castle, the authorities ordered some of the defensive walls to be deconstructed. From 1865, the castle hosted spiritual seminars.