keep our annotations on others' data

With this fix, it is no longer possible to remove one's annotations from others' data by moving one's own annotated data.

cc: @pwalczysko
--no-rebase