The following decades were somewhat less stormy. Lida was ravaged by the Crimean Tatars in 1506 and it was stormed by the Russians during the Russo-Polish War in 1659. The Swedes, taking it twice during the Great Northern War, had both towers blown up. In 1794, the castle grounds were the site of a battle between Kościuszko's followers and the Russians.