The complex consisted of three castles: the Upper, the Lower, and the Crooked (). The Crooked Castle was burned down by the Teutonic Knights in 1390 and was never rebuilt. The Vilnius Castles were attacked several times by the Teutonic Order after 1390, but they did not succeed in taking the entire complex. Its complete capture occurred for the first time during the 1655 Battle of Vilnius. Soon afterwards, the severely damaged castles lost their importance, and many buildings were abandoned. During the Tsarist annexation, several historic buildings were demolished; many more were damaged during the fortress construction in the 19th century.