In the 15th century a 5-storey keep was built by Bishop Cameron, this was later extended with additional fortifications and buildings, constructed by later bishops. Archbishop Beaton added a large corner tower, and Archbishop Dunbar built a round-towered gatehouse in 1544. The central keep served as the residence of the bishop, and was called the Bishop's Palace. The castle played a role in the many political battles during the 16th century, including the protracted struggle between supporters of Mary, Queen of Scots, and her enemies. It changed hands six times between 1513 and 1570, and was occupied by French troops at one point. In 1544 it was defended against Regent Arran and in 1560 defended for Arran.