Hamilton-Gordon was the second son of Prime Minister George Hamilton-Gordon, 4th Earl of Aberdeen, by his second marriage to Harriet, daughter of the Hon. John Douglas. Arthur Hamilton-Gordon, 1st Baron Stanmore, was his younger brother. He served in the British Army and saw action at the Battle of Balaclava in October 1854 during the Crimean War. He went on to be General Officer Commanding Eastern District in January 1872.