Cavendish was born in Westminster, the third son of George Cavendish, 1st Earl of Burlington and Lady Elizabeth Compton, daughter and heiress of the 7th Earl of Northampton. Cavendish was educated at Eton College and Trinity College, Cambridge. He was commissioned as a lieutenant in the 10th Dragoons in 1808 and was deployed to Spain and was wounded at the Battle of Corunna in January 1809 during the Peninsular War. In 1812 he entered Parliament for Derby, a seat which he held until 1834. In 1837, he was appointed Chief Equerry and Clerk Marshal to Queen Victoria, but resigned the post in 1841. On 2 June 1853 he was appointed colonel of The Queen's Bays, a post he held until his death. He was promoted to major-general in 1846, lieutenant-general in 1854 and full general in 1862.