Doesn't query the server again if the annotation already exists