Duffus Castle Duffus Castle , near Elgin , Moray , Scotland , was a motte - and - bailey castle and was in use from c . 1140 to 1705 . During its occupation it underwent many alterations . The most fundamental was the destruction of the original wooden structure and its replacement with one of stone . At the time of its establishment , it was one of the most secure fortifications in Scotland . At the death of the 2nd Lord Duffus in 1705 , the castle had become totally unsuitable as a dwelling and so was abandoned . The wooden castle . Moray was an important part of the Pictish confederation . At the beginning of the 12th century the province was ruled by the mormaer , Angus , grandson of Lulach Macgillecomgan , who had succeeded Macbeth as King of Scots in 1057 . Ferociously autarchic and highly distrustful of the expansion of the monarchy , Angus rebelled in 1130 . The revolt was resolutely quashed by King David I , who immediately began to populate the province with nobles and people of his own choosing . Freskin . One significant arrival was Freskin ( also known as " Freskyn " ) , who already owned an extensive estate in Lothian ( Strabrock , now Uphall ) . He accepted lands at Duffus from King David I of Scotland . Freskin ’ s background is uncertain . The historical consensus amongst historians is that he was of Flemish background , the principal argument being that " Freskin " is a Flemish name . Undoubtedly , King David , himself a Normanized magnate with extensive estates in northern England and Normandy , granted lands to many nobles from Flanders as well as Normans . The unlikely alternatives are that he may have been an Anglo - Saxon or a Scot who fought for King David and his English general Edward Siwardsson in Moray . At that time , when Flemish nobles were referred to in writs by nationality ( almost never ) , they were styled " " Flandrensis " " . Freskin appears in no contemporary sources , and was never referred to by his national origin . By the 13th century his descendents were referring to themselves as ' de Moravia ' ( ' of Moray ' ) and had become one of the more powerful families in northern Scotland . It was Freskin who built the great earthwork and timber motte - and - bailey castle in c . 1140 . The motte was an immense man - made mound with steeply sloping sides and a wide and deep ditch that surrounded the base . Timber buildings would have stood on its flat top and would have been further protected by a wooden palisade placed around the edge of the summit . The motte was accessed from the bailey . This is a wide stretch of earth elevated above the surrounding area but not as high as the motte . At Duffus , the motte would have been reached by steps set into the mound . The bailey contained the buildings necessary to sustain its inhabitants – brew and bake houses , workshops and stables – as well as the living accommodation . The stone castle . Freskin ’ s direct line ended in 1270 and the castle passed into the ownership of Sir Reginald Cheyne , the younger . In around 1350 the last Cheyne died leaving his estate to his daughter who was married to Nicholas , the second son of the 4th Earl of Sutherland . The Sutherlands , themselves were descended from Freskyn and remained in their possession until 1705 when the castle was abandoned . In 1305 , it was recorded that Reginald Cheyne received a grant from King Edward I of England of 200 oaks from the royal forests of Longmorn and Darnaway ' to build his manor of Dufhous ' demonstrating that a large construction project was being carried out . The wood would have been needed for scaffolding , flooring and roofing of the new stone fortress . It is known that the original castle was burned down by Andrew Moray in the summer 1297 who descended on it with what was described as in a letter to King Edward I of England as “ a very large body of rogues ” , because it held a garrison of King Edward ’ s English troops and this had been the impetus for building a more secure castle of stone . A two - storey rectangular tower was built on the motte and was the main residence . The first floor held the lord ’ s hall , with a latrine and bed chambers . The ground floor was the main storage space and also accommodated the lord ’ s household . The tower was built as a defensive structure with a small number of narrow windows . There was only the one entrance on the ground floor which also housed a portcullis . On the second floor , two doors exited onto the walkway of the curtain wall . This wall completely enclosed the bailey . The put - log holes built into the curtain wall indicate the presence of a number of buildings . On the north side a later building was erected that housed a kitchen , a great hall with reception room and the great chamber bedroom . It is possible that this building was constructed by the Sutherlands . It is not known when the serious subsidence took place but evidence of repairs to the tower are evident before it slid down the motte . The tower shows no further repairs and may have collapsed early on but the newer hall became the main residence . This building shows continued alterations over time . In 1689 , John Graham , 1st Viscount of Dundee was a guest of Lord Duffus just before the battle of Killiecrankie and would be one of the last important visitors before the castle ’ s abandonment .