In the 16th century the castle was used as a prison. It was destroyed during the Russo-Polish War (1654–1667) and never rebuilt. The territory was granted to a Dominican cloister in 1678 by Marcjan Aleksander Ogiński, Voivode of Trakai. It was not until the 1770s that the monks eventually built their monastery and church; these buildings are also part of the castle ensemble.