Improves notifications about new annotations

Fixes #7055
We don't need a summary here? Otherwise LGTM.