He became the 4th Duke of Richmond on 29 December 1806, after the death of his uncle, Charles Lennox, 3rd Duke of Richmond. In April 1807 he became Lord Lieutenant of Ireland. He remained in that post until 1813. He participated in the Napoleonic Wars and in 1815 he was in command of a reserve force in Brussels, which was protecting that city in case Napoleon won the Battle of Waterloo. On 15 June, the night before the Battle of Quatre Bras, his wife held a ball for his fellow officers. The glittering celebration became famous as the Duchess of Richmond's ball and was immortalised by William Makepeace Thackeray in "Vanity Fair" and by Lord Byron in "Childe Harold's Pilgrimage".