Don't rely on DOM traversals to find annotations to update