Support the clearing of annotation based customizations

Now what's left is to add a way for the user to trigger this and a way
to update all visualizations.