Improve deletion over annotations and anchors

I'm not 100% happy about the second commit, as multiple cache hits are quite expensive. But, it is definitely functionally more correct, and more work will be coming soon to further speed up iteration.