The castle walls have been repaired many times over the centuries