The tower ruins are known as the Skull Hole. When the local graveyard was cleared out to make more room (and there's would have been a lot of clearing out to do - it's ancient and full of old ruins), the bones were dumped in the centre of the tower.