With the defeat of Manfred, King of Sicily in the Battle of Benevento in 1266, the Swebian hegemony over southern Italy ended and the Castle of Gioia del Colle declined in influence. After the Hohenstaufen, it went under the supremacy of the House of Anjou and of the Crown of Aragon. Manfredwho was born in Gioia del Colle, according to legendlost the property at his death to the princes of Taranto until the 15th century. Then, it passed to the House of Acquaviva from Conversano until the 17th century, and to the princes of Acquaviva until the beginning of the 19th century.