Fix the bogus remove
Merge pull request #667 from mozilla-services/fix_annotation_removal

Fix the bogus remove