Robert was born around 1276, the third son of the future Charles II of Naples (then heir apparent) and his wife Mary of Hungary. His father was the son of the incumbent King of Naples, Charles of Anjou, who had established an Italian realm a decade earlier in 1266. During the Sicilian Vespers directed against his grandfather Charles, Robert was the hostage of Peter III of Aragon, his grandfather's enemy. In 1285, Robert’s grandfather died at Foggia in Italy, leading to his father (then a hostage) becoming King of Naples as Charles II, with Robert's elder brother, Charles Martel of Anjou as heir apparent.