Baptised in St Benet's in London, he was the son of Robert Dormer, 1st Earl of Carnarvon and Lady Anna Sophia Herbert, daughter of Philip Herbert, 4th Earl of Pembroke. Dormer was educated at the University of Oxford, where he graduated Master of Arts in 1648. In 1643, on his father's death at the First Battle of Newbury he succeeded to his father's titles and became Hereditary Chief Avenor and Keeper of the King's Hawks. His mother had died a few months earlier.