Method for both removing and adding annotations (useful when integrating with CoreData).