At the start of the Hussite rebellions, the Švihovský family searched for havens of supporters of the Catholic side in the district and for their treasures, at Rábí. In 1420–21, the Hussites conquered the castle twice, and legend has it that during the second siege an arrow fired from crossbow hit the trunk of a pear tree and a resulting splinter from the tree hit Jan Žižka in his only good eye. In 1479, the provincial governor Půta Švihovský of Rýzmberk became the owner of the estate and began with a thoughtful remodelling of the castle under the guidance of famous master Benedict Rejt. New living quarters and service buildings were constructed, the castle was enlarged and the fortifications heightened.