The construction of the castle would have started in the 11th century, but it took its current dimensions in the 12th century. It was then a possession of the lords of Anduze, who held the Baronnies of Portes. Having taken the side of their suzerain, the Count of Toulouse, their property including this castle was confiscated by the King of France around 1229, at the end of the Albigensian Crusade. But the Bishop of Mende and the king disputed for a long time the possession of the region. A first agreement in 1265 gave it to the king. In 1307, after 36 years of legal processes between them, the act of "paréage" (treaty) gave it to him definitively. The castle remained the direct property of the Barons de Portes until 1320 when Raymond de Cadoène bought it from them and paid tribute to them for it. The castle nevertheless underwent the vicissitudes of the lordship of Portes.