The castle dates from the 13th and 14th centuries, when it was occupied by the Comyn, Earls of Buchan, before it was likely slighted in 1308 by Robert the Bruce in the Harrying of Buchan during the First War of Scottish Independence. The property passed to Clan Ross, to Alexander Stewart, Earl of Buchan the Wolf of Badenoch, and to the MacDonald Earls of Ross.