The existing castle was built around a 15th century tower house or keep and has evolved over 600 years, with significant extensions from the 17th century. William Calder, 6th Thane of Cawdor (orig. Calder) erected the medieval tower in 1454 as a private stronghold. The castle later served as the residence of the Stuarts, until it was abandoned after the death of King James VI in 1603.