Phipps was the son of a Royal Navy officer, Lieutenant Alan Phipps (1915–1943), who was killed ashore at the Battle of Leros, one of the sons of Sir Eric Phipps, a British diplomat descended from the first Earl of Mulgrave. His mother, Veronica Nell (née Fraser; 1920–2005) was a Roman Catholic, the daughter of Simon Fraser, 14th Lord Lovat. In 1946, she married secondly Brigadier Fitzroy Maclean, who raised Phipps, who was educated at Ampleforth and the Royal Military Academy Sandhurst.