This castle was built mainly from wood and earth. During the siege of Stirling Castle in 1304 the English used Linlithgow as their main supply base. After the Battle of Bannockburn, in 1314, Linlithgow returned to Scottish hands. In 1337 the peill was granted to John Cairns by king David II with orders to build it up for the Kings coming. This he did and King David held court their in 1343 and in later years.
Nothing remains of the early manor house except the name peel .
In 1424 a disastrous fire destroyed most of the town of Linlithgow as well as the parish church and the manor house. King James I set about a programme of building which resulted in the royal palace much as it is today. 2440 Scots are recorded as being spent by John de Waltoun, the Master of Works, and, by June 1428 the place was habitable enough for the king to spend some days there. A high rate of expenditure was maintained until the kings assassination in 1437. James II made little use of his fathers palace before his death at Roxburgh in 1460.