Remove annotations before setting new ones

This puts the responsibility for adding to existing annotations on
the client