Various owners leased it from royal landlords and the freehold was the subject of frenzied speculation during the 17th century