The original building dates from 1181 as a monastic site for the monks of the Priory of St Gregory. The footprint of building would have consisted of a chapel and great hall with the working dairy probably where Howfield Farm now stands, along with mills and fishing rights through the Stour valley.