Only unsubscribe on pure removal, not detachment