During the 1745 Jacobite Rebellion his father sided with the Young Pretender, despite both James and his brother William then holding commissions under George II; James in the army, William in the navy. Remaining loyal to the Hanoverians, James then served at the Battle of Culloden, fighting on the opposite side to his father. During the rout following the Jacobite defeat, the Earl was captured and taken into the government camp, dishevelled and bareheaded, where he was reportedly recognised by James, who placed his own hat upon his father's head. This was the last time they were to meet, as the Earl was then transported to London where he was tried for treason and executed four months later; forfeiting all his lands and titles and thus depriving James of his inheritance.