Kiszka became entangled in a political conflict with royal favorite Michael Glinski and was temporarily removed from the Lithuanian Council of Lords in 1505. In August 1505, Kiszka scored a victory against the invading Crimean Khanate near Haradok. As the Great Hetman, he was supposed to command Lithuanian troops in the Battle of Kletsk against the Crimean Khanate in August 1506 but was unable due to poor health. In summer 1507, Konstanty Ostrogski escaped from Russian service and was reinstated as the Great Hetman. To compensate, Kiszka was appointed as starosta of Hrodna in 1508. When the war with Moscow resumed in 1507, Kiszka organized the defense of Polotsk. During the Glinski rebellion, he defended Minsk and helped to drive out Glinski's forces. His last military commands were near Orsha (18 July 1508) and Vyazma.