Make unsubscribe + reference modification an atomic operation