Remove participants from handle list when service is gone