The site of the palace formed part of which Manor in the Middles ages