Compare annotations by name only when clearing/toggling

Otherwise links don't get removed as an empty link is not
comparable to a real link with an href.

Bug: 50461
Change-Id: Id8b421bd44cd5c427d0e5cd738c380bc2b1ea719