After serving in the 1799 Helder Campaign, he was promoted to be Master General of the Ordnance in 1801. This became a key role in the next few years when a French invasion appeared imminent, and he helped oversee Britain's defences including the construction of a series of martello towers. He grew increasingly independent from his brother and did not resign from the cabinet when William Pitt left office in 1801. He held this post under Henry Addington and when his brother returned to office again in 1804. Following his brother's death in 1806, he briefly went into opposition against the Ministry of All the Talents before returning as Master General in the new Tory government of 1807. He also continued his military career, being promoted to Lieutenant General and appointed to several commands in charge of home defence.