During the Crusades, the site was called Blanchegarde, ("White guard"), likely referring to the white rock outcrop next to the site. In 1142, a fort was built on the site by King Fulk. After the Siege of Ascalon in 1153, the castle was expanded and strengthened. It became a lordship in 1166, when it was given to Walter III Brisebarre, lord of Beirut.