Reset the reference intead of deleting it
Wait a bit after sync. Remove from set if it exists.