In this capacity, in July-August 803, Qasim led a raid into Byzantine Asia Minor. He besieged the strategic fortress of Koron, while his lieutenant al-'Abbas ibn Ja'far ibn Muhammad ibn al-Ash'ath was dispatched to besiege another fortress, known to the Arabs as Sinan. The Byzantines, however, offered to release 320 Muslim prisoners if he departed, and he agreed. In February 808, when Harun departed his residence Raqqa for his second expedition to Khurasan, he left Qasim behind as his deputy in Raqqa, with Khuzayma ibn Khazim as his advisor. After Harun's death and the accession of al-Amin in 809, Qasim was confirmed in his position as governor of the "al-'Awasim" and of "jund Qinnasrin", but was apparently removed from his governorship of the Jazira (Upper Mesopotamia), where Khuzayma ibn Khazim was appointed instead. Eventually, in 810, Qasim was removed by Amin from all his governorships in favour of Khuzayma, and brought to live under close supervision in Baghdad. Amin shortly after forbade his mention in the prayers, along with Abdallah al-Ma'mun, who had risen in near-revolt in Khurasan. In the civil war that followed, Ma'mun was triumphant, defeating and killing his brother Amin in 813. Immediately after, Qasim was formally deposed from his position as successor.