The origins of the castle may be as old as the Phoenician foundation of Oea in the 7th century BCE; excavations have found remains dating back to Roman times. Like the surrounded region, it was taken over by the Vandal Kingdom around 439, recovered by the Roman Empire in 534, overrun by Amr ibn al-As in 642 or 643, reconquered by the Byzantines then again by the Umayyad Caliphate in the late 7th century; then under the rule of the Abbasid Caliphates (750-800), the Aghlabid Emirate of Ifriqiya (800-909), the Fatimid Caliphate (10th century) who refurbished it into a royal residence, the Zirid Emirate of Ifriqiya and Banu Khazrun (11th century-1146), the Norman Kingdom of Africa (1146-58), the Almohad Caliphate (1158-1229), the Hafsid Emirate of Ifriqiya (1234-1321), the Beni Ammar (1321-1401) and again the Hafsid Emirate (1401-1510, with another lapse in the mid-15th century).