do not unwrap detached highlights

In case our highlights were removed from the DOM somehow, don't
remove them again.
don't choke when an annotation has no highlights