The Abbey fell into decline and after the Reformation it fell into ruin also. Many of the monks remained in the declining monastery. For a number of years the Lady Chapel was used as a parish church. In 1590 Arbroath Town Council granted the stones and timbers from the old dormitory to be used to build a proper church. The Abbey became a quarry for cash conscious burgesses.
Many houses still have interesting carved details which started life in the Abbey.
It was not until 1815 that any steps were taken to preserve the ruins. The Stone of Destiny Arbroath Abbey has one more important date in its history. On Christmas Day 1950 the Stone of Destiny was stolen from Westminster Abbey. On the morning of 11 April 1951 it was deposited on the site of the high alter where it was discovered by the Abbey custodian. Many believe it was not the original stone that was returned or that now sits in Edinburgh Castle.