Educated at Harrow School, Lyttelton-Annesley was commissioned into the 11th Hussars in July 1854. He took part in the Siege of Sebastopol in Winter 1854 and the Battle of the Chernaya in August 1855 during the Crimean War. He went out to India in 1866 and then returned to England to take command of his regiment in 1877 before being appointed Assistant-Adjutant-General of the Horse Guards in 1878 and then Adjutant-General of the Bombay Army in 1883. He went on to command the troops in the North British District in 1888 before retiring in 1893.