The castle fell into disrepair in the late 16th century. The fire of 1616 and damage during The Deluge in 1657 resulted in the further destruction. The fortress was periodically repaired, but due to shifts in the layout of the borders and relations between Poland and its neighbours, it lost much of its importance. After the partitions of Poland, Będzin fell into Prussian control and the castle became property of the Hohenzollern family. In 1807, the nearby lands were transferred to the Duchy of Warsaw, and in 1815, to the Congress Poland. In 1825 the castle was virtually falling apart, and when a piece of the stone crushed a passerby, demolition of the castle was ordered, but before it was started, the castle was declared a monument. In the 1830s the castle was bought by Count Edward Raczyński and partially rebuilt, with a Protestant church temporarily housed inside, but after Raczyński's death in 1845 plans to open an academy or a hospital there were abandoned, and the castle once again fell into disrepair.