The land was originally purchased by Hugh de Raymes, a major Suffolk merchant, in 1296. Hugh purchased the property from what was the estate of an impoverished neighbor in the hopes of increasing his social stature and influence. When Hugh de Raymes died in 1295, his son Robert took over ownership of the property. It was Robert who moved his household and made Aydon Hall their primary residence. The move was believed to be due to opportunities for Robert in the Scottish Wars of Independence, and it is known that Robert fought in Scotland between 1297 and 1298. Robert is believed to have fought at the Falkirk in 1298 against the Scottish forces of William Wallace. Upon Robert's return from fighting in Scotland, he began the construction of the first stone structure. Aydon had previously featured a wooden structure with a two-story chamber block.