Over the ensuing centuries, most of the castle stone was pilfered by locals to build their homes.