When we finally reached the base of the castle, we could look up and see the entire building facing down upon us