The second son of Lord Augustus FitzRoy and a grandson of the 2nd Duke of Grafton, FitzRoy joined the 1st Regiment of Foot Guards as an ensign in 1752. He fought at the Battles of Minden and Kirchdenkern during the Seven Years' War and rose to the ranks of Captain in 1756 and Lieutenant-Colonel in 1758.