Olivier was the eldest son and heir of John I, Count of Penthièvre and Margaret de Clisson, and the grandson of Joan, Duchess of Brittany and her husband, Charles of Blois, making him the senior claimant to the Breton throne in the Penthièvre line. In 1411, Olivier joined the Armagnacs in the Armagnac-Burgundian Civil War despite being married to Isabella, a daughter of John, Duke of Burgundy. In this capacity, he fought alongside Louis II, Duke of Anjou, at the siege of Bourges in 1412.