So much of the castle grounds have eroded over the years - or been removed to build houses below.