How does Uncertainty-aware Sample-selection Help Decision against Action Noise?
Authors: Xingrui Yu
Bo Han
Ivor Tsang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Imitation Learning
State-Dependent Action Noise
Uncertainty-Aware
Negative Learning

Abstract: Learning from imperfect demonstrations has become a vital problem in imitation learning (IL). Since the assumption of the collected demonstrations are optimal cannot always hold in real-world tasks, many previous works considers learning from a mixture of optimal and sub-optimal demonstrations. On the other hand, video records can be hands-down demonstrations in practice. Leveraging such demonstrations requires labors to output action for each frame. However, action noise always occurs when the labors are not domain experts, or meet confusing state frames. Previous IL methods can be vulnerable to such demonstrations with state-dependent action noise. To tackle this problem, we propose a robust learning paradigm called USN, which bridges Uncertainty-aware Sample-selection with Negative learning. First, IL model feeds forward all demonstration data and estimates its predictive uncertainty. Then, we select large-loss samples in the light of the uncertainty measures. Next, we update the model parameters with additional negative learning on the selected samples. Empirical results on Box2D tasks and Atari games demonstrate that USN improves the performance of state-of-the-art IL methods by more than 10% under a large portion of action noise.  

Under review as a conference paper at ICLR 2023
HoW DOES UNCERTAINTY-AWARE SAMPLE-SELECTION HELP DECISION AGAINST ACTION NOISE?
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning from imperfect demonstrations has become vital problem in imitation learning (IL). Since the assumption of the collected demonstrations are optimal cannot always hold in real-world tasks, many previous works considers learning from a mixture of optimal and sub-optimal demonstrations. On the other hand, video records can be hands-down demonstrations in practice. Leveraging such demonstrations requires annotators to output action for each frame. However, action noise always occurs when the annotators are not domain experts, or meet confusing state frames. Previous II. methods can be vulnerable to such demonstrations with state-dependent action noise. To tackle this problem, we propose a robust learning paradigm called USN, which bridges Uncertainty-aware Sample-selection with Negative learning. First, model feeds forward all demonstration data and estimates its predictive uncertainty. Then, we select large-loss samples in the light of the uncertainty measures. Next, we update the model parameters with additional negative learning on the selected samples. Empirical results on Box2D tasks and Atari games demonstrate that USN improves the performance of state-of-the-art II. methods by more than 10% under a large portion of action noise.
INTRODUCTION
Despite the great success of reinforcement learning (RL) (Sutton & Barto 2018) over last few years, designing hand-crafted reward functions can be extremely difficult and even impossible in many real-world tasks (Ng et al. 1999; Amodei et al. 2016 Brown et al. 2019a). A Alternatively, imitation learning (II.) (Russell, 1988, Schaal 1999 Abbeel & Ng. 2004, Argall et al., 2009: Hussein et al., 2017) aims train agent mimic the demonstrations collected from expert, without any access to hand-crafted reward signals. However, it is expensive and difficult to collect high-quality demonstrations in real-world tasks (Silver et al. 2013).
In practice, is much cheaper to collect demonstrations from amateurs (Audiffren et al., 2015). Existing works (Tangkaratt et al. 2019: 2020 Zhang al., (021b) have studied imitation learning from mixture of optimal and non-optimal demonstrations. Specifically, [Tankkaratt et al. (2019) requires that all the actions for demonstration sufficiently small variance. Following works (Tangkaratt et al., 2020) proposed robust imitation learning by optimizing a classification risk with a symmetric loss. The resulting algorithms RIL and RIL_CO still require more optimal demonstrations than non-optimal ones in the dataset.
drawn from the same noisy distribution with
On the other hand, in many practical activities like sport games, it is common for people use camera to record the excellent behaviors of athletes in terms of sequences of pictures and videos. However, usually be hard to get the exact action label for each picture. To leverage such data for imitation learning, we need to recruit annotators to output action labels for pictures in the sequence. Limited by the quality of the annotators, action noise always occur during the action-labeling procedure. amateur annotator may randomly pick action for picture that contains state he never seen before. In this situation, the final demonstration will contain inte-indeeendeet action noise. Besides, even an expert annotator makes mistakes. This is especially true when the annotator meets some similar and confusing states. In this situation, the annotator will output noisy actions that are dependent on the confusing states, resulting demonstration with dependee action noise.
Previous methods (Tangkaratt et al., 2019, 2020; Zhang et al., 2021b) focus on imitation learning from mixture of optimal and non-optimal demonstrations noisy demonstrations with small noise

---

Under review as a conference paper at ICLR 2023
Large-loss samples
Negative Learing
States
Noisy Actions
Speeding throught
Predictive Uncertainty
Change lames
Go straight through
Positive Learning
IL model
Policy
Figure 1: The full procedure of USN.
on the actions. They ignore the fact that both state-independen action noise and state- intteiindependent action noise widely exist practice. Moreover, these methods are evaluated on -wwdimmnsionnl environments, and are hard scale to high-dimensional environments. Therefore, existing methods usually fail learning good policy with action noise, especially in high-dimensional environments.
To tackle this challenge, we first conduct an investigation experiments to study the correlation between the dynamics of loss and predictive uncertainty metrics as the noise rate increase. Then, propose new method called uncertainty-aware sample-sclection with soft negative learning (USN) based on the correlation. As shown in Figure USN trains a policy with additional process of iicertainty-awat aware sample-selection for negative learning (USN). Specifically, in positive learning (marked in blue color), we train the model with any noise-tolerant loss function, and then estimate the predictive uncertainty measures during training. Then, select large-loss samples using the estimated predictive uncertainty measure as the sampling threshold. This is motivated by recent works in the area of learning with noisy labels (Angluin & Laird 1988; |Smyth et al. 1994] Kalai & Servedio 2005 Natarajan et al. 2013 Manwani & Sastry (013). Since deep networks learn easy patterns first (Arpit al. 2017 they would first memorize training data of clean labels and then those of noisy labels with the assumption that clean labels are of the majority in noisy class. Therefore, the large-loss samples can be regarded as noisy actions with high probability (Han et al.,
2018 Yu et al. 2019 Wei et al. 2020; Yao et al. 2020)) Next, employ negative learning (marked in green color) (Kim et al. 2019) on automatically selected large-loss samples, along with positive learning the full dataset.
USN a meta-algorithm that is scalable to both offline imitation learning an online imitation learning. Compared with existing IL methods, our method has advantages in many aspects, including imitation performance, adaptivity and scalability Our method can adaptively select large-loss samples for soft negative learning across different noise rates. Additionally, any extra datasets, models or prior information about the noise model. That means our method avoids the extra efforts and drawbacks of estimating noise rates and transition matrix as previous noise-robust methods. Empirical results show that USN is scalable to Behavioral Cloning, online imitation learning and offline imitation learning.
method does not depend
BACKGROUND AND RELATED WORK
In this section, firstly discuss existing offline and online imitation learning methods. Then, discussed related works of learning with noisy labels.
Behavioral cloning (BC) is probably the simplest offline imitation learning algorithm. For imitation learning in environments with discrete action space, the BC policy T (aas)) is optimized by softmax cross-entropy loss.
2.1 ONLINE IMITATION LEARNING
Generative adversarial imitation learning (GAIL) (Ho & Ermon 2016) is one of the state-of-the- art online IL methods. GAIL treats imitation learning as a distribution matching problem. Built of
top of GAN (Goodfellow et al. 2014), GAIL and its many (robust) variants (Li et al.) 2017] Peng iet al. 2019 Tangkaratt et al. 2001 2020; Wang et al,, 2021) has achieved great success in imitation learning in ow-diimesiional space with noisy demonstrations, However, GAIL fails scale
2

---

Under review as a conference paper at ICLR 2023
to -igh--imensional imitation learning tasks (Brown et al. 2019a Tucker al. 2018). One simple solution to alleviate this issue is to initiate the actor of GAIL using behavior cloning.
Selective Adversarial Imitation Learning (SAIL) Wang al. 2021 proposed to address imperfect demonstration issue, in which good demonstrations can be adaptively selected for training while bad demonstrations are abandoned. Specifically, binary weight {0, 1] is assigned to each expert demonstration to indicate whether to select it for training. The weight set to be determined by the reward function in Wasserstein GAIL (Xiao et al.) 2019) (i.e. higher reward results in higher weight). The resulting algorithm- SAlL-hard is defined as follows:
min max (uaxxEssaa) (s, (ww,ww)rs(ss..... a)ro(s - Kw(s, a)] ((s.a)= (r)(s, (M
(1)
Besides hard binary weighting, they also propose soft weighting scheme with the suggested optimal soft weight as w² Ite 8 --glaaa The corresponding algorithm SAfL- is defined as follows:
min max (s (sss))rr(ss)) + f(w((s a))] ((sa) (ro a)] + (M)())))
(2)
where f(w) = w log(w" w 1) log(1 - w) Kw. We regards GAIL, and its variants RIL, RIL CO (Tangkaratt et al. 2020) and SAIL as baselines, for comparing their robustness against action noise.
2.2
OFFLINE IMITATION LEARNING
Batch-constrained Q-learning (BCQ) (Fujimoto et al. 2019b) is the first continuous control algorithm capable of learning from arbitrary batch data, without exploration. BCQ aims to perform Q-learning while constraining the action space eliminate actions that are unlikely to be selected by the behavioral policy Ty, and are therefore unlikely to be contained the batch. At its core, BCQ uses state-conditioned generative model G: - to model the distribution of data in the batch, G Rs b akin to a behavioral cloning model. As is easier to sample from TV (T(aas)) () than model To(a s) exactly in continuous action space, the policy is defined by sampling N actions a, from G(s) and selecting the highest valued action according to a Q-network.
Discrete BCQ (Fujimoto al. 2019a) extends BCQ to discrete action spaces by computing the probabilities of every action G R Tu ). and instead utilizing some threshold to eliminate actions. To adaptively adjust this threshold, Fujimoto (Fujsoo et al. (2019a) scale it by the maximum probability from the generative model over all actions, to only allow actions whose relative probability is above some threshold. Specifically, Fujimoto et al. (2019a) applies Double DQN (Van Hasselt al., 2016) to select the max valued action with the current Q-network Qa. and evaluate with the target Q-network Qø. This results in an algorithm comparable DQN (Mnih al.) (015)) where the policy is defined by constrained argmax. The Q-network is trained by swapping the max operation with actions selected by the policy:
max C(0) L. (r+7 a/C(al))/ max
Q0r(s', Qo(s.a)).
(3)
where L, defines the Huber loss (Watkins 1989):
(6) {
0.582 <[[0] 0.5k)
if otherwise.
(4)
2019a) maintain the original property of BCQ where setting returns Q-learning and 7 returns an imitator of the actions contained in the batch. The generative model G, effectively a behavioral cloning network, is trained in standard supervised learning fashion, with cross-entropy loss. In this paper, we adopt the (Discrete) BCQ method for offline IL by training intrinsic curiosity module (ICM) Pathak et al. (2017) for generating intrinsic rewards, instead of using the true reward data in the demonstration.
With this threshold r, Fujimoto et al.
Uncertainty weighted actor-critic (UWAC) (Wu al., 2021) is recently proposed offline rein- forcement learning algorithm that is able to detects OOD state state-action pairs and down-weights their contribution in the training objectives accordingly. UWAC is closely related
work, since it also uses the uncertainty estimation to improve the model robustness. Specifically, UWAC uses Monte Carlo (MC) dropout (Gal & Ghahramani 2016) estimate uncertainty of model. The model
3

---

Under review as a conference paper at ICLR 2023
uncertainty is captured by the approximate predictive variance with respect to the estimated Q for T stochastic forward passes:
T
Var[Q(s,a)] ~ o² + Σ
1
Qe(s,a) (e(ss,a) of (I
(5)
T
t=1
Then, UWAC down-weighs the Bellman loss for the Q function by inverse the uncertainty of the Q-target (8' ,a)):
B Err Brr(s,
[(Qe) = -. V VarlQu a')]
(6)
where Err(s,a, _=')= Qo(s,a)= (R(s, a)) + Q (s', a'). This directly reduces the effect that OOD backups has on the overall training process (Wu et al. 2021). We treat UWAC as strong baseline in our paper, and we also show that USN can scale to MC dropout for uncertainty estimation. The original UWAC was built on top of BEAR (Kumar et al. 2019). To enable a fair comparison, we implement UWAC on top of BCQ follows:
((0) La ++ a/G(ala)/ max (G(()) VarlQo (8' a() QQr(s [ (7)
B
max
2.3 LEARNING WITH NOISY LABELS
Learning with noisy labels aims to learn robust classifier f by exploiting training samples with only noisy labels (xi, (=1="
Generalized Cross Entropy (GCE) (Zhang & Sabuncu 2018 a generalization of Cross Entropy (CE) and Mean Absolute Error (MAE). To exploit the benefits of both the noise-robustness provided by MAE and the implicit weighting scheme of CCE, they proposed using the negative Box-Cox transformation (Box & Cox 1964) as loss function:
c,(f()),ez) (1-55(2)9)
(8)
where y E (0, 1]. It is equivalent to CE for limg- (r( and becomes MAF/unhinged loss when g 1.
Label smoothing has been commonly used to improve the performance of deep learning models (Szegedy et al.) 2016; Chorowski & Jaitly 2016 Vaswani et al. 2017 Zoph et al. 2018 Real et al. 2019 Huang et 2019). It serves a regularizer (Li et al. 2020 Pereyra et al. [017]] Müller et al. 2019; Yuan et al. |0200|Zhag Zhang et al. 2222a la) that improves the generalization and model calibration (predictive uncertainty) by using smoothed labels ý that mix the original one-hot labels y with an uniform mixture over all possible labels for a E [0, 1]:
J = (1 a))
1,
K
where is the number of classes. Recently, Lukasik et al. 2020 demonstrated label smoothing is also effective when learning with symmetric label noise.
HOW DOES ACTION NOISE AFFECT IMITATION LEARNING?
In this section, first introduce inte-independent action noise and stale-dependent action noise. Then, we briefly summarize loss estimation and uncertainty estimation methods. Next, we provide comprehensive study on the correlations between loss and uncertainty under multiple action noise.
3.1 ACTION NOISE
State-independent action noise: In traditional imitation learning, consider the demonstrations 1 are collected from some expert demonstrator. In practice, we may recruit annotators give action labels for the recorded sequences of expert behaviors. The datee-ndeeeendett
D {s,, a,, $;+1 R==1
4

---

Under review as a conference paper at ICLR 2023
action noise occurs when amateur annotator randomly pick action for unseen states. The noise generation process is quite similar to the conditinn label noise, where a corrupted label is randomly flipping from other classes (Symmetric noise) or its neighbor (Pairflip noise) (Natarajan et al. 2013 Han et al., 2018). Therefore, we build the noise modeling of state-independen action noise by borrowing the setting of class-conditional label noise. The resulting noisy demonstrations is noted as D {&,, 0g, 844 JE=
State-dependent action noise: The state-dependent ac- tion noise occurs when expert annotator meets confus- ing states. In this situation, we assume the noisy action is dependent on the state feature and is independent on the expert action: P(ã: [4,, s;) = (s), Figure 2 shows a graphical model for statendependent action noise. We generate the state dependenn action noise following the instance-dependent label noise setting in the Appendix
Figure 2: State State-dependent action noise.
B
3.2 CORRELATION BETWEEN LOSS AND PREDICTIVE UNCERTAINTY UNDER ACTION NOISE
In this paper, we provide a comprehensive study about the correlation between the loss, uncertainty estimation of Behavioral Cloning, online imitation learning (GAIL) and offline imitation learning (BCQ) models under diverse action noise in the demonstrations. We consider demonstrations: the first type of demonstration is the demonstrations with inte-inndeendent action noise; and the second type of demonstrations with state-dependent action noise. To study the correlations, we pre-train an imitation learning model with the optimal demonstration, and then estimate the corresponding loss and uncertainty on multiple noisy demonstrations.
types of noisy
Loss estimation. For the Behavioral Cloning model, GAIL actor and BCQ generative model, we use the cross-entropy as the loss estimation method. For the GAIL discriminator, we use the adversarial loss for the expert data for loss estimation.
Uncertainty estimation. Uncertainty estimation has been widely used in machine learning appli- cations, as complement that reflect the degree of trust in the model predictions (Kotelevskii et al.) 2022). Usually, the total uncertainty of a prediction comes from two types of uncertainty: aleatoric and epistemic (Der Kiureghian & Ditlevsen 2009 Kendall & Gal (017). The aleatoric uncertainty, known as data uncertainty, reflects the noise and ambiguity in the data. The epistemic uncertainty, known as model uncertainty, is related to the lack of knowledge about model parameters. Quantifying both types of uncertainty is crucial for safe decision in practical applications (Filos et al. 2020).
One simple aleatoric uncertainty is called MaxProb, which uses maximum softmax probabilities of deep neural network as the uncertainty measure (Nguyen et al. 2015). MC_dropon is based on Bayesian techniques, capturing both types of uncertainty and known to be a more reliable uncertainty estimation method (Gal & Ghahramani] 2016). Recently, a promissing direction of uncertainty estimation methods based a single deterministic neural network has been developed (Lee et al., 2018 Liu et al.) (020). More recently, Kotelevski et al. (2022) proposes nonparametric uncertainty quantification (NUQ) method, which enables uncertainty to disentangled into aleatoric and epistemic uncertainty, and is scalable to large dataset.
Expected Calibration Error (ECE) (Nainn et al. 2015 Guo al. 2017 also widely used to measure the predictive uncertainty (model calibration) of deep network. To approximate the calibration error in expectation, ECE discretizes the probability interval into fixed number of bins, and assigns each predicted probability to the bin that encompasses it. The calibration error is the difference between the fraction of predictions in the bin that are correct (accuracy) and the mean of the probabilities in the bin (confidence). More variants of calibration errors are summarized in the Appendix G
Instead of using the uncertainty for OoD detection (Malinin & Gales, 2018), or down-weighting the Bellman loss for Q function (Wu et al.) 2021) in RL, we leverage uncertainty for robust imitation learning in a novel direction. Namely, we first study the correlation between loss and uncertainty; and then propose an robust imitation learning paradigm called uncertainty with negative learning (USN) based on the correlations. Our method USN is scalable to Behavioral Cloning, online imitation learning and offline imitation learning.
sample selection
5

---

Under review as a conference paper at ICLR 2023
3.2.1 A COMPREHENSIVE STUDY ON THE CORRELATIONS OF LOSS AND UNCERTAINTY
Due to the space limit, only provide samples of the correlations for BC, online and offlie IL in this section. For more comprehensive results, please check the Appendix C
Behavioral Cloning. Figure 3 shows that the loss has positive correlation with the ECE uncertainty as the noise rate increases. This results indicate that the loss and ECE uncertainty of BC model are good identifier of noisy actions.
(a) Loss
(b) ECE
(a) Loss
(b) ECE
Figure 3: The correlations between loss, uncer- tainty of BC and noise rate of the noisy demon- stration in the Acrobot-v1 task.
Figure 4: The correlations between loss, uncer- tainty of BCQ generative model and noise rate of the noisy demonstration in the Qe game.
Online Imitation Learning. Figure and Figure @how that the loss has positive correlation with the ECE uncertainty as the noise rate increases. This results indicate that the loss and ECE uncertainty of GAIL actor and discriminator are good identifier of noisy actions.
In
(a) Loss
(b) ECB
(a) Loss
(b) ECE
Figure 5: The correlations between loss, uncer- tainty of GAIL actor and noise rate of the noisy demonstration in the Q*hert game.
Figure 6: The correlations between loss, uncer- tainty of GAIL. discriminator and noise rate of the noisy demonstration in the hert game.
Offline Imitation Learning. We train the BCQ (Algorithm 2
Appendix A with an ICM module (Pathak et al. 2017) for generating intrinsic rewards as the base offline imitation learning method. BCQ starts by sampling a mini batch of transitions from the input demonstration data D. Then, use the generative model G to eliminate actions (step 5) for updating Q network (step 6). The generative model G is trained using a cross-entropy loss in step 7.
Figure 4 shows that the loss of the generative model G goes up first and then drops as the noise rate increases. Interestingly, the ECE uncertainty drops at first and then increase as the noise rate increases, which shows a negative correlation with the dynamics of G's loss.
Figure 6 and more results in the Appendix we can conclude that the loss estimation be used criteria for detecting noisy actions using the BC model, BCQ's generative model, GAIL actor and discriminator. Specifically, the 'large-loss' samples have high probability to contain noisy actions. For the BC model, GAIL actor and GAIL discriminator, the uncertainty estimation ECE can also be used as a good criterion for detecting noisy actions. However for the offline imitation learning method BCQ, ECE does not show clear positive correlation with the increase of action noise. Instead, the dynamics of ECE in BCQ's generative model has negative correlation to the loss. this paper, we focus on proposing general method that detects noisy actions with high probability, and leverage the selected data to improve the robustness of BC, online IL and offline IL against diverse action noise in the demonstrations, To this end, propose general paradigm called Uncertainty- awwre Sample-selection with Negative learning (USN), for robust training IL models against action noise. USN selects large-loss samples using the uncertainty estimation, e.g. ECE as threshold that decides the number of sample selection.
Analysis. From Figure 3 Figure 4 Figure 5

---

Under review as a conference paper at ICLR 2023
4 METHOD
Algorithm 1 Uncertainty-awee aware Sample-selection with Soft Negative learning (USN)
Input:
mini-batch M of B transitions (8, a, s') from D, negative learning loss weight as Xneg-
model parameter w.
2: Estimate predictive uncertainty, e.g. ECE, and regards it as the sample-selection threshold Tu-
3: Sample the large-loss batch M by selecting the large-loss samples with length of: #neg =
(1 Tu) B.
4: Generate complementary actions for M: ā Randomly select from [AI))a), resulting
a complementary batch for negative learning.
5: w arg min_ (s.a)eM Ppos a) + Aneg L(s.ae (s)
6: Output:
Algorithm| | summarizes the full procedure of USN. Our main goal is develop general robust IL paradigm against diverse type of action noise. To achieve this goal, we design USN to composite of two main steps: uncertainty-awar sample-selection and negative learming for loss correction.
Uncertainty-awar sample-selection aims to select samples that contains noisy actions with high probability. Given mini-batch of demonstration data with size of B, first employ basic IL method with parameter w, for positive learning on the full-batch data. Then second step, we estimate the uncertainty, e.g. on the batch data and regards it as threshold Tu for selecting large-loss samples. Next, we sample the large-loss batch M by selecting the large-loss samples with length of: #neg = (1 Tu) * B. This procedure is called uncertainty-aware sample-selection.
Previous sample selection methods (Hafner al. 2018 Han et al., 2020 Xia et al. 2021) usually assume that the noise rate is known, and design the sample selection threshold using the noise rate. In contrast, ECE). Since the uncertainty estimation dynamically changes during training, the sample selection threshold is automatically updated and is adaptive to different noise rates. In this way, our method can maximally reduces the negative effects of action noise using the adaptive threshold, without requiring any prior knowledge about the noise model.
define the sample selection threshold Tu using the uncertainty estimation (e.g.
Negative learning for loss correction. Positive learning with full-batch data with noisy actions will results in bias in the loss training, and misguides the policy to choose wrong actions. To correct the loss bias, we propose to leverage the selected large-loss sample for negative learning. Intuitively, the selected losge-loss batch M contains noisy actions with high probability. Its complementary set has more chances to contain true actions. Therefore, we generate complementary actions for M by randomly selecting á from (1, Al}Ma) resulting complementary bath M for negative learning. Negative learning the complementary batch of the selected large-loss samples will correct the loss bias from action noise, and therefore improve the imitation learning performance. To further boost the performance, we implement negative learning with label smoothing, resulting Soft Negative learning. Specifically, we employ the following negative log likelihood (NLL) loss for negative learning on the 'large-loss' samples with label smoothing: Lneg = NLL G(als), (1 a) + Aa 1). where à is the complementary action of the 'large-loss' samples, and a is the smooth rate.
EXPERIMENTS AND RESULTS
In this section, we demonstrate how does USN improves the robustness of BC, onfline IL and offline IL when learning with state-independent action noise and state-dependent action noise. For Behavioral Cloning and BC initialized GAIL, we choose the BC as the basic IL model. For offline imitation learning,
obtain the basic IL model BCQ-GGEE, by applying GCE the generative model: Cpos = [1 (I-G(als))) Algorithm in the Appendix A presents an instance of our proposed robust IL method, BCQ-USN. BCQ-USN improves BCQ by updating the generative model G using USN method (Algorithm[1 [ in step 7. USN improves the ability of generative model G for eliminating noisy actions for policy learning. Thus, BCQ-USN is more robust than BCQ and BCQ-GCE, achieving better imitation learning performance across diverse levels of action noise.
7

---

Under review as a conference paper at ICLR 2023
5.1 EXPERIMENTAL SETUP
We construct experiments on the widely used discrete control tasks and Atari games from OpenAl Gym environments. To evaluate the effectiveness of methods, we pre-train expert agent and then generate demonstrations with state-dependent noisy actions using the pre-trained agent. For each setting, we have two demonstrations, D for normal demonstration and D for demonstration with state-dependent noisy actions only, a.k.a. the noisy demonstration. Note that we use ground-truth rewards only to train the expert agent, and we discard the rewards afterward.
In setting 1, we use the classic control tasks from OpenAI Gym Acrobot-vl and LunarLander-v2. We pre-train a DQN policy as the expert agent, and generate demonstrations with 50K steps. This setting is used to evaluate how our method USN improves Behavioral Cloning and offline imitation algorithms under state- cutte-independen action noise and state-dependent action noise. In setting 2, we use widely used Atari games. For online II., we generate one full-episode demonstration using a PPO agent. For offline IL, we first pre-train QRDQN policy the expert agent. Then, we generate 50K-step demonstration dataset. This setting is used for evaluation the robustness of our method USN on online imitation learning algorithm GAIL, and offline imitation learning algorithms BCQ and CQL against diverse action noise.
For Behavioral Cloning, compare USN with BC and GCE. For online imitation learning, we compare our method GAIL-USN to SOTA robust imitation learning algorithms: compare BCQ-USN with BCQ in Algorithm 2 of Appendix [ which is trained with the rewards from an Intrinsic Curiosity Module (ICM). (b) BCQ-GCE, which is a robust BCQ by training generative model with GCE loss. and (c) UWAC, a SOTA robust offline imitation learning that also uses uncertainty estimation. We set the negative learning loss weight Aneg = 0.01 for the classic control tasks and Ang 1.0 for Atari games.
Baseline algorithms:
RIL_CO, SAIL amd the original GAIL.. For offline imitation learning experiments,
5.2 BEHAVIORAL CLONING WITH NOISY DEMONSTRATIONS
In the first experiment, we evaluate the robustness of our method USN on the classic control tasks Acrobot-vl and LunarLander-v2. Figure 7 and Figure & show the performance of BC, GCE and our USN under multiple state indeeeedddednn action noise and steeedde action noise, respectively. In both types of action noise in the demonstrations, method USN achieves better performance than other baselines.
100
63
(a) Acrobot-v1
(b) 2
(a) Acrobot-vl
(b) I Lander_ande--22
Figure 7: Average return of BC, GCE and our USN on classic control tasks with state- independent action noise.
Figure 8: Average return of BC, GCE and our USN on classic control tasks with state- dependent action noise.
5.3 ONLINE IMITATION LEARNING WITH NOISY DEMONSTRATIONS
Then, we show how our method USN can improves the robustness of online imitation learning against action noise. Since most of the previous robust online imitation learning algorithms (Tangkaratt et al. 2020 Wang et al., |02 are based on GAIL, we also choose GAIL as our base method. However, the original GAIL can not achieve good performance in high-dimensional environment like Atari games (Brown et al. 201 9a b). Fortunately, we found that using Behavioral Cloning as an initialization for the actor, GAIL is able to achieve good performance some Atari games, e.g. Seaquest, Q*bert and Hero with only one full-episode demonstration. We apply our method USN on the BC initialization and compares SOTA robust imitation learning algorithms, RII_CO and SAIL with soft weights.
8

---

Under review as a conference paper at ICLR 2023
The results under multiple state-dependent action noise for three Atari games are shown in Figure SAIL totally fails in all the Atari games under action noise in the demonstrations. In the Seaquest game, both our method GAIL-USN and RIL CO achieves the best performance; while RILL RIL_CO fails in the other two games. Our method GAIL--SSN outperforms all the haselines in the other two Atari games. More details are summarized in Appendix P
Herooncti
1900
19499
12500
3:0
3009
12009
CALL
250
03
32
33
E4
6.
2a
0.1
6.5
1.3
0.5
05
29
0.1
1.3
0.5
05
(a) Seaquest
(b) Qu
(c) Hero
Figure 9: The performance of GAIL and baselines with noisy demonstration across different state- dependent action noise on Atari games.
5.4 OFFLINE IMITATION LEARNING WITH NOISY DEMONSTRATIONS
Next, we evaluate the robustmess of our method USN on offline imitation learning with demonstrations under both intee-indeeendent action noise and state-dependent action noise.
Robustness to state-independent action noise. In Figure a))),,), we evaluate our method on two other types of state-independent action noise: symmetric noise and pairflip noise. These two types of noise have been widely studied in the literature of learning with noisy labels. We can that the performance of BCQ drops when noise rate is larger than 0.45 in both Symmetric and Pairflip action noise. Our method BCQ-USN outperforms BCQ-GCE in both cases across diverse noise level. BCQ-USN-PC denotes our method using PC loss Ishida et al. (017) as a negative learning loss. The results show that our method is scalable to multiple choices of negative learning loss.
Lunartanders2 Symmetticc Action Walse
Luartanders2- Pa Action Moise
Lustntanderr2
Booe Rute
Plooe Rube
(a) Symmetric noise
(b) Pairflip noise
(c) State-dependent noise
Figure 10: Average return of BCQ, BCQ-GCE and our BCQ-USN on LunarLander-v2 with state- independent action noise (Symmetric and Pairflip), and state-dependent action noise.
Robustness to state-dependent action noise. Figure 10 c) shows the average return curves of BCQ, BCQ-GCE and our BCQ-USN on LunarLander-v2 across diverse levels of state-dependent action noise. For both types of action noise, BCQ-USN outperforms BCQ-GCE and BCQ across diverse noise levels. In Figure 33 of the Appendix, we show bar charts comparison of BCQ. BCQ-GCE and BCQ-USN for four Atari games. Our method, BCQ-USN consistently outperform BCQ-GCE and BCQ when the noise rate is 0.45. The performance improvements of BCQ-USN compared to other methods are usually significant. For this experiment, compare our method with an related bascline UWAC (Wu et al.) 202I). In addition, we introduce a variants of our method called BCQ-USN-MC MMM_ooopuu,, which uses the MC dropout for uncertainty estimation in USN. The quantitative results are summarized in T
The results show that UWAC only outperforms BCQ-GCE the Assault and AirRaid games, while performs much worse than BCQ-GCE on two other games. Our variant BCQ-USN-MC Coupopu outperforms UWAC and BCQ-GCE two games. This demonstrates that method USN can
9

---

Under review as a conference paper at ICLR 2023
Table 1:
Average return of BCQ, BCQ-GCE, BCQ-USN, OCQ-USN-MCCdroppnn and UWAC when learning with state-dependent action noise with noise rate of 0.45 on four Atari games.
Assault
Aline
AirRaid
NameThisGame
BCQ
646.4
445.6
1,894.5
2,580.4
BCQ-GCE
667.4
655.8
1,858.5
2,615.2
BCQ-USN (Ours)
678.3
833.6
2,170.0
2,907.2
CUUUUUNNMddoooooo (Ours)
699.5
539.6
2,183.3
1,723.6
UWAC (Wu et al. 2021
695.3
510.6
2,491.3
2,317.3
use MC dropout as an uncertainty estimator, while overall it performs worse than the variant using calibration errors. Overall, BCQ-USN outperform BCQ-GCE by 14% across the four Atari games.
Finally, we generalize USN to Q-learning. In the above experiments, USN is applied on top of Behavioral Cloning. To show the scalability of our method, we apply our USN method to a more general Q-learning based II. method without a generative model. We chose the conservative Q- learning (CQL) (Kumar et al., 2020) as our base method. In Appendix [ we implement a robust CQL algorithm, CQL-GCE, by training the Q network with the original conservative loss and additional GCE loss. Similar to BCQ-USN, we apply USN on the positive learning baseline- - CQL-GGE Specifically, estimate calibration error metrics for the Q network, and then the calibration error to determine the large-loss samples for negative learning. Empirically, we evaluate the resulting algorithm CQL-USN on the Q*bert Atari game with stale-dependent action noise, symmetric action noise and pairflip action noise across different noise rates. Figure IT shows the comparative results of CQL. CQL-GCE and CQL-USN on the bert game. Impressively, CQL-USN usually outperform CQL and CQL-GCE by large margin. Therefore, our USN method has good scalability to general Q-learning based II. methods, without requiring the generative model.
1912
7503
2903
2009
U00
1808
PVite
Wole
Htole latts
(a) Symmetricnoise
(b) Pairflip noise
(c) State-dependent noise
Figure II: Average return of CQL, CQL-GCE and CQL-USN on Q*bert game with state- state- indepedent action noise (Symmetric and Pairflip) and state dependenn action noise.
CONCLUSION
In this paper, we have provided comprehensive study of correlations between loss estimation and uncertainty estimation of imitation learning models. Then, based on the correlations, we propose an novel paradigm for robust imitation learning with action noise. Our method, uncertainty-awe aware sample selection for negative learning (USN), scales to Behavioral Cloning, online imitation learning and offline imitation learning with inte-inndepedeent action noise and state deateddepnndent action noise. Moreover, our USN paradigm can use diverse predictive uncertainty for robust imitation learning, showing the nice adapativity and scalability of our method. In addition, the ablation studies in the Appendix Hlass demonstrate that our method is not sensitive to the adaptation of the hyperparameters. In the near future, we are interested in extending our method to continuous control tasks by studying the correlations between loss and regression uncertainty estimations of IL models. Also, it is worthy to investigate the application of our method on actor-critic learning and real-world tasks.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, pp- 1, 2004.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané, Concrete problems in ai safety, arXiv preprint arXiv:1606.06565. 2016.
Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(4):343-370, 1988.
Brenna Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pp. 233-242.
PMLR, 2017.
Julien Audiffren, Michal Valko, Alessandro Lazaric, and Mohammad Ghavamzadeh. Maximum entropy supervipe inverse reinforcement learning. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
George EP Box and David R Cox. An analysis of transformations. Journal of the Royal Statistical Society: Series B (Methodological), 26(2):111-443, 1964.
Daniel S Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International Conference on Machine Learning. 2019a
Daniel S Brown, Wonjoon Goo, and Scott Niekum. Ranking- reward extrapolation without rankings. In Conference on Robot Learning, 2019b.
Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration in sequence to sequence models. arXiv preprint arXiv: 1612.02699. 2016.
Armen Der Kiureghian and Ove Ditlevsen. Aleatory epistemic? does it matter? Structural safety, 31521:105-112, 2009.
Angelos Filos, Panagiotis Tigkas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In International Conference on Machine Learning, pp. 3145-3153 PMLR, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv: 2019a.
Scott Fujimoto. David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019b.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, PMLR, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration modern neural networks. In International Conference Machine Learning, pp. 1321-1330. PMLR, 2017.

---

Under review as a conference paper at ICLR 2023
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, 2018.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. In International Conference on Machine Learning, pp. 4006-4016. PMLR, 2020.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure, arXiv preprint 1rX2v:1812.04660 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565 4573, 2016.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hyoukloong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, et Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Flyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2): 2017.
Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. Learning from complementary labels. arXiv preprint arXiv: (rXXiv1705.075555 075400 2017.
Adam Tauman Kalai and Rocco A Servedio. Boosting in the presence of noise. Journal of Computer and System Sciences, 71(3):266-29) 2005.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Niul: Negative learning for noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 101-110, 2019.
Nikita Kotelevskii, Aleksandr Artemenkov, Kirill Fedyanin, Fedor Noskov, Alexander Fishkov, Aleksandr Petiushko, and Maxim Panov. Nuq: Nonparametric uncertainty quantification for deterministic neural networks. arXiv preprint 22X 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-leanning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint 200,:2090000779, 2020.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribttion samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Weizhi Li, Gautam Dasarathy, and Visar Berisha. Regularization via structural label smoothing. In International Conference on Artificial Intelligence and Statistics, pp. 1453-14455 1463. PMLR, 2020.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. Advances in Neural Information Processing Systems, 30, 2017
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance aware- ness. Advances in Neural Information Processing Systems, 33:7498- 7512, 2020.
12

---

Under review as a conference paper at ICLR 2023
Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pp. 6448-6458. PMLR, 2020.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE transactions on cybernetics, 43(3): 1144-1111,, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 51875500)5529 2015.
Rafael Müller, Simon Korblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. Icml, volume 99, pp. 278-287, 1999.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 427-436, 2015.
Jeremy Nixon, Michael Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mcasuring calibration in deep learning. In CVPR Workshops, volume 2, 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by sulf-supervieed prediction. In International Conference on Machine Learning, pp. 2778-2787, 2017.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. In International Conference Learning Representations, 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint 1.01v:1701.066 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory, pp. 101-103, 1998.
Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6): 233-242, 1999.
David Silver, J Andrew Bagnell, and Anthony Stentz. Learning autonomous driving styles and maneuvers from expert demonstration. In Experimental Robotics, pp. 371-386. Springer, 2013.
Padhraic Smyth, Usama Fayyad, Michael Burl, Pictro Perona, and Pierre Baldi. Inferring ground truth from subjective labelling of venus images. Advances in neural information processing systems, 1994.
Richard S Sutton and Andrew Barto. Reinforcement learning: An introduction (2nd Edition). MIT press, 2018
13

---

Under review as a conference paper at ICLR 2023
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2826, 2016.
Michita Imai Takuma Seno. d3rlpy: An offline deep reinforcement library. In NeurlPS 2021 Offline Reinforcement Learning Workshop, December 2021.
Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, and Masashi Sugiyama. Vild: Variational imitation learning with diverse-quality demonstrations. arXiv preprint (etivv:9900007777 2019.
Voot Tangkaratt, Nontawat Charoenphakdee, and Masashi Sugiyama. Robust imitation learning from noisy demonstrations. arXiv preprint arXiv:2010. 10181, 2020.
Aaron Tucker, Adam Gleave, and Stuart Russell. Inverse reinforcement learning for video games. Workshop on Deep Reinforcement Learning at NeurIPS, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double learning. Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
Yunke Wang, Chang Xu, and Bo Du Robust adversarial imitation learning via adaptively-selected demonstrations. In ICCAI, pp. 3155-3161, 2021.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13726-13735, 2020.
Jiay Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. arXiv preprint 10X. 2021.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140, 2021.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards dnstance-dependent label noise. Advances in Neural Information Processing Systems, 33:7597-7610, 2020.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint arXiv:2106.00444, 00445, 2021.
Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and Thai Hong Linh. Wasserstein adversarial imitation learning. arXiv preprint arXiv: 190660811111 13, 2019.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Tin- Yau Kwok. Searching to exploit memorization effect in learning with noisy labels. In International Conference on Machine Learning, pp. 10789-10798. PMLR, 2020.
Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pp. 7164-7173 PMLR, 2019.
Li Yuan, Francis EH Tay, Guilin Li. Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911, 2020.
14

---

Under review as a conference paper at ICLR 2023
Chang-Bin Bin Zhang, Tan Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, and Ming-Ming Cheng. Delving deep into label smoothing. IEEE Transactions on Image Processing, 30:5984- 5996, 2021a.
Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation learning from demonstrations with varying optimality. Advances in Neural Information Processing Systems, 34, 2021b.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
15

---

Under review as a conference paper at ICLR 2023
A ALGORITHMS
Algorithm 2 and Algorithm 3 shows the procedure of BCQ and BCQ-USN for offline imitation learning. We implement our methods using the Tianshou platform (Weng al. 202T) for offline imitation learning, and train all the algorithms with the default hyperparameters of BCQ.
Algorithm 2 BCQ with ICM for offline imitation learning.
Input: Demonstration D {(8,, Oe, 81+1)] number of iterations T, target_update rate, mini-batch size B, threshold
2: Initialize Q-network Qs. generative model with parameter w and target network Qa with
0' с 0.
3: fort= to T do
4: Sample mini-batch M of B transitions (s,a, a, s') from D.
5: Train an ICM module to generate intrinsic reward: r -= (C,(s,,, s').
6: a' arg maxx"]G(alle")) max (G())=))>r Qo a').
7: 0 arg ming ((aa,s')eM (r + 7Q ( (s', a') a))
8:
w < arg min L(sa)eMM log G(als).
//Update G' using cross- erosspyn loss.
9: If mod target_ cecot__pdat_rraate rate- = 0: 0 <
10: end for
Algorithm 3 BCQ-USN for robust offline imitation learning against action noise.
1: Input: Demonstration mini-batch size B, threshold
{(s;, ag 81+1) number of iterations T, pugget_ppdte_____ _pdate_rate, rate,
2: Initialize Q-network Qu. generative model with parameter w and target network Qo with
8' 0.
3: fo to do
4: Sample mini-batch M of transitions (s, a, s') from D.
5:
Train an ICM module to generate intrinsic reward: r ICM(s,a,
6:
a' arg m
max ) (o(s',a')
7:
0 arg ming E _(sa,x')EEM (i + (s', a') Qo(s,a))
8:
w f (S)
Update using USN approach Algorithm 0: - O.
9:
If mod
10: end for
B DEPENENT ACTION NOISE GENERATION
Algorithm 4 State-Dependent Action Noise Generation
1: Input:
Expert demonstration D {(si, ai 3141) Noise rate: Size of feature: 1 |S];
Number of classes:
[A].
2: Iterations:
3: Sample instance flip rates 4u from the truncated normal distribution (Ce,, 0.12, [0,,0..2, 1]);
4: Sample W E R|A) from the standard normal distribution (0, 12);
5: for 1 to N do
6: p = 8; W
I/ Generate state dependent flip rates. The size of p is 1 x [.A.
7: Pa, -0
I/ Only consider entries different from the expert actions
8:
= qi sofumax
// Let qo. be the probability of getting a wrong action
// Keep expert actions w.p. q: 10: Randomly choose action from the action space as noisy action à; according to p;
9:
Pu, = qi
end for
12: Output: Noisy Demonstrations D {(si ās 1
We simulate the generation of state-dependent action noise by following the instance-dependen label noise setting (Xia al. 2020). Note that it is more realistic that different states have different flip
16

---

Under review as a conference paper at ICLR 2023
rates. Without constraining different states to have same flip rate, it is more challenging to model the label noise and train robust classifiers. In Step 1, in order to control the global flip rate as but without constraining all the states to have same flip rate, sample their flip rates from truncated normal distribution (r, 0.12, 12, [o, 1)). Specifically, this distribution limits the flip rates of states the and the standard deviation
range [0, I]. Their mean and standard deviation are equal to the 0.1 of the selected truncated normal distribution respectively.
In Step we sample parameters W1, W...... We from the standard normal distribution for generating state-dependent action noise. The dimensionality of each parameter is x c, where d denotes the dimensionality of the state. Learning these parameters is critical model state-dependent action noise. However, it is hard to identify these parameters without any assumption. Note that an state with expert action a will be flipped only according to the a-th row of the transition matrix. Thus, in Steps 4 to 7, we only use the a:-th row of the dependep transition matrix for the state Specifically, Steps 5 and 7 are to ensure the diagonal entry of the a- row is Qz- Step is to ensure that the sum of the off-diagonal entries is q:-
C THE CORRELATION BETWEEN LOSS, UNCERTAINTY AND NOISE RATES OR OPTIMALITY OF DEMONSTRATIONS IN THE ATARI DOMAIN
C.I RESULTS OF BC ON CLASSIC CONTROL TASKS.
(a) Loss
(b) ECE
(C) MCE
(d) Entropy
Libertt
Max Porb
(1) Bald
(g)
MC_ MCC_rrpouu
(h) NUQ
Figure 12: The correlations between loss, uncertainty of BC and noise rate of the noisy demonstration the Acrobot-yl task.
17

---

Under review as a conference paper at ICLR 2023
(a) Loss
(b) ECE
(c) MCE
(d) Entropy
Orfhort Jineet seeeere
(c) Max Porb
Bald
(g)
MC_dropout
(h) NUQ
Figure 13: The correlations between loss, uncertainty of BC and noise rate of the noisy demonstration the CartPole-vl task.
(a) Loss
(b) ECE
(c) MCE
(d) Entropy
(c) Max Porb
Bald
(g)
MC_dropout
(h) NUQ
Figure 14: The correlations between loss, uncertainty of BC and noise rate of the noisy demonstration the LundrLander-v2 task.
18

---

Under review as a conference paper at ICLR 2023
C.2 RESULTS OF GAIL ON ATARI GAMES
C.2.1 RESULTS OF GAIL ON A SINGLE ATARI GAME.
(a) Loss
(b) ECE
MCE
Entropy
(c) Max Porb
Bald
(g) MC_dropout
(h) NUQ
Figure 15: The correlations between loss, uncertainty of GAIL actor and noise rate of the noisy demonstration in the Qert game.
(a) Loss
(b) ECE
(c) MCE
(d) Entropy
(e) Max Porb
Bald
(g) MC dropout
(h) NUQ
Figure 16: The correlations between loss, uncertainty of GAIL discriminatorand noise rate of the
noisy demonstration in the bert game.
For the noisy demonstration with state-dependent action noise, we study correlation between loss, uncertainty estimation of GAIL and noise rate of the demonstrations, The results of GAIL actor and discriminator in Q*bert game are shown in Figure 15 and Figure T6, As expected, with the increasing of noise rates in the demonstrations. the loss estimation of the GAIL actor increases accordingly. For the uncertainty estimations, ECE and MCE show the positive correlation with the loss estimation and the increase of noise rate. For the GAIL diseriminator, the loss estimation and three uncertainty estimations (i.e. ECE, MCE and NUQ) hold the positive correlation with the increase of noise rate the demonstrations. The Bald and MC MC_droopuu uncertainty estimations also show light positive correlations. The above observations hold for many Atari games. The results can be found bellow. Therefore, the loss estimation, the uncertainty estimations ECE and MCE, can be used as criteria for detecting noisy actions in the demonstrations using the GAIL actor. Similarly, the loss estimation, the uncertainty estimations ECE, MCE, NUQ and even Bald and MC UCC_dropout, can be used as criteria for detecting noisy actions in the demonstrations using the GAIL. Il discriminator.
19

---

Under review as a conference paper at ICLR 2023
1-
(a) Loss
(b) ECE
(c) MCE
(d) Entropy
(c) Max Porb
(f)
Bald
(g) MC MIC_dropout
(h) NUQ
Figure 17: The correlations between loss, uncertainty of GAIL actor and optimality of the noisy demonstrations in the h*bert game.
(a) Loss
(b) ECE
(C) MCE
(d) Entropy
2.
(e) Max Porb
Bald
(g) MC_dropout
(h) NUQ
Figure 18: The correlations between loss, uncertainty of GAIL discriminatorand optimality of the noisy demonstrations in the heerrt game.
For the noisy demonstration with suboptimal demonstrations, we study correlation between loss, uncertainty estimation of GAIL and optimality of the demonstrations. The results of GAIL actor and discriminator in bert game are shown in Figure 17 and Figure [8] The lower load iter denotes worse optimality. The results show that with the decreasing of the optimality in the demonstrations, the loss estimation of the GAIL actor increases accordingly. For the uncertainty estimations, ECE and MCE show the positive correlation with the loss estimation and the increase of noise rate. Entropy, MaxProb and MC dropout show light positive correlations. For the GAIL discriminator, the loss estimation and many uncertainty estimations (i.e. ECE, MCE, Bald, MC dropout and NUQ) hold the positive correlation with the decrease of optimality in the demonstrations. The above observations hold for many Atari games. The results can be found bellow. Therefore, the loss estimation, the uncertainty estimations ECE, MCE and even MC_ MC_dropout, can be used as creteria for detecting suboptimal data in the demonstrations using the GAIL actor. Similarly, the loss estimation, the uncertainty estimations ECE, MCE, Bald, MC_ cc__dropuut and NUQ, can be used criteria for detecting suboptimal data in the demonstrations using the GAIL discriminator.
20

---

Under review as a conference paper at ICLR 2023
C.2.2 RESULTS GAIL ON MORE ATARI GAMES WITH NOISY ACTIONS
(a) Loss
(b) ECE
(c) MCE
(d) Entropy
Max Porb
Bald
(g)
MC_dropout
NUQ
Figure 19: The correlations between loss, uncertainty of GAIL actor and noise rate of the noisy demonstration in the Seaquest game.
(a) Loss
(b) ECE
MCE
(d) Entropy
(e) Max Porb
( Bald
(g) MC dropout
(h) NUQ
Figure 20: The correlations between loss, uncertainty of GAIL discriminatorand noise rate of the noisy demonstration the Seaquest game.
21

---

Under review as a conference paper at ICLR 2023
(a) Loss
ECE
MCE
(d) Entropy
(c) Max Porb
( Bald
(g) MC_dropout
(h) NUQ
Figure 21: The correlations between loss, uncertainty of GAIL, actor and noise rate of the noisy demonstration in the Hero game.
(a) Loss
(b) ECE
MCE
Entropy
(c) Max Porb
if Bald
(g) MC_dropout
(h) NUQ
Figure 22: The correlations between loss, uncertainty of GAIL discriminatorand noise rate of the noisy demonstration in the Hero game.
22

---

Under review as a conference paper at ICLR 2023
C.2.3 RESULTS GAIL ON MORE ATARI GAMES WITH SUB-OPTIMAL DEMONSTRATIONS
(a) Loss
(b) ECE
(c) MCE
(d) Entropy
Max Porb
(f) Bald
(g)
MC_dropout
NUQ
Figure 23: The correlations between loss, uncertainty of GAIL actor and optimality of the noisy demonstrations in the Seaquest game.
1im
(a) Loss
(b) ECE
MCE
(d) Entropy
(e) Max Porb
( Bald
(g) MC_dropout
(h) NUQ
Figure 24: The correlations between loss, uncertainty of GAIL discriminatorand optimality of the noisy demonstrations in the Seaquest game.
23

---

Under review as a conference paper at ICLR 2023
(a) Loss
ECE
MCE
(d) Entropy
(c) Max Porb
r Bald
(g) MC_dropout
(h) NUQ
Figure 25: The correlations between loss, uncertainty of GAIL. actor and optimality of the noisy demonstrations in the Hero game.
(a) Loss
ECE
MCE
Entropy
(c) Max Porb
f Bald
(g) MC_dropout
(h) NUQ
Figure 26: The correlations between loss, uncertainty of GAIL discriminatorand optimality of the noisy demonstrations in the Hero game.
24

---

Under review as a conference paper at ICLR 2023
C.3 REUSULTS OF BCQ ICM ON ATARI GAMES.
(a) Loss
(b) ECE
(c) MCE
(d) Entropy
Max Porb
11) Bald
(g)
MC_dropout
(h) NUQ
Figure 27: The correlations between loss, uncertainty of BCQ_ICM imitator and noise rate of the noisy demonstration in the Q*bert game.
010
300
2000
(a) Loss
(b) ECE
MCE
(d) Entropy
(e) Max Porb
( Bald
(g) MC_dropout
(h) NUQ
Figure 28: The correlations between loss, uncertainty of BCQ_ICM imitator and noise rate of the noisy demonstration in the NameThisgane game.
25

---

Under review as a conference paper at ICLR 2023
(a) Loss
ECE
MCE
(d) Entropy
(c) Max Porb
( Bald
(g) MC_dropout
(h) NUQ
Figure 29: The correlations between loss, uncertainty of BCQ_ICM imitator and noise rate of the noisy demonstration in the AirRaid game.
(a) Loss
(b) ECE
MCE
(d) Entropy
(c) Max Porb
(f) Bald
(g) MC_dropout
(h) NUQ
Figure 30: The correlations between loss, uncertainty of BCQ_ICM imitator and noise rate of the noisy demonstration in the Alien game.
26

---

Under review as a conference paper at ICLR 2023
10
(a) Loss
ECE
MCE
(d) Entropy
(c) Max Porb
( Bald
(g) MC_dropout
(h) NUQ
Figure 31: The correlations between loss, uncertainty of BCQ_ICM imitator and noise rate of the noisy demonstration in the Assault game.
Zun
(a) Loss
ECE
MCE
(d) Entropy
(c) Max Porb
(f) Bald
(g) MC_dropout
(h) NUQ
Figure 32: The correlations between loss, uncertainty of BCQ_ICM imitator and noise rate of the noisy demonstration in the Seaquest game.
27

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL RESULTS.
D.1 COMPARATIVE RESULTS ACROSS MULTIPLE NOISE RATES.
Table 2 shows the comparative results of BCQ, BCQ-GCE and our BCQ-USN on LunarLander-v2 across different levels of state-dependent action noise.
Table Average return of offline imitation learning on the LunarLander-v2 with diverse levels of stale-dependent action noise.
Noise Rates
Method
0.05
0.25
0.45
0.65
BCQ
210.8
205.1
-370.9
-626.8
BCQ-GCE
178.7
193.4
26.1
113.0
BCQ-USN-ECE
177.2
193.7
0.9
167.1
BCQ-USN-ACE
148.4
211.1
160.4
50.8
BCQ LSS--Max_L CE
188.0
179.2
177.9
165.5
NameThisGame
Noisc Rate
800
BCD
800
BCD-4C2 BCQ-USA
2000
1000
1500
500
025
8.15
0.65
0.85
NamathhaGamm
Noise Rate
games
Figure 33: Average return of BCQ, BCQ-GCE and BCQ-USN LunarLander-v2 v2 and NameTThs- Game across diverse noise rates.
Figure 34: Bar chart comparison of average return of BCQ, BCQ-GCE and BCQ-USN in four Atari games with 45% state-dependent action noise.
Table 3:
Average return of offline imitation learning on the Atari game ThimGamee with diverse levels of state-dependent action noise.
Noise Rates
Game
0.05
0.25
0.45
0.65
0.85
BCQ
3,640.0
3,320.4
2,580.4
1,569.0
906.6
BCQ-GCE
3,605.0
3,195.2
2,615.2
1,309.0
1,033.4
BCQ-USN
3,332.6
3,285.6
2,907.2
1,942.4
1,039.4
E ONLINE IMITATION LEARNING
E.1 GENERATIVE ADVERSARIAL IMITATION LEARNING (GAIL)
Generative adversarial imitation learning (GAIL) was proposed based on the general framework of Generative Adversarial Networks (GANs) Goodfellow et al. (2014). GAIL regards the classical imitation learning problem as an occupancy measure matching between the expert policy and the agent policy. Specifically, GAIL uses discriminator D, to distinguish expert transitions from agent
28

---

Under review as a conference paper at ICLR 2023
transitions, while the agent is to "fool" the discriminator into treating agent transitions expert data. Formally, the objective function of GAIL is written as
min max (s,,) [log Do a)] (ssa)) (log))
(10)
where P9 and pe denote the occupancy measures of agent policies my and the expert Tg, respectively,
GAIL and its many variants (Peng et al., 2019 ei al. 2017) has achieved great success in imitation learning in low-dimensional tasks. In imitation learning with imperfect demonstrations / noisy demonstrations, many previous methods are also based the framework of GAIL. However, GAIL had been demonstrated that does scale well to -ihh-nimnnsioall tasks like Atari games (Tucker et al., 2018 Brown et al. 2019a|b), b). To boost the performance of GAIL in high-dimension simple step by initializing the actor with behavior cloning. We found that GAII, is able achieve demonstration-lelv performance some games by using the BC initialization on the actor. Our implementation based on the "gail atari' repository! []
task, we take a
E.2 GAIL-USN
To extend our USN paradigm to GAII,, directly apply USN on the behavior cloning model in the initialization step.
E.3 EXPERIMENTS AND RESULTS
Experiment setup. For online imitation learning, we consider learning from the noisy environments. Namely, besides to the noisy demonstration data, the data collected from the environment are full of action noise. This happens practice when the environments suffers from system error or the data collecting and transferring protocol under attack. This task is more challenging than previous imitation learning problem with noisy demonstrations only.
Baselines. In this experiment, compare method GAIL-USN GAIL and the following state-of-the-art robust imitation learning methods:
RIL and RIL CO (Tangkaratt al. 2020)
SAAL--oftt and SAIL-hard (Wang al. 2021
Since all the baselines are variants of GAIL, we use the same settings including behavior cloning initialization for all the methods.
Results. As shown in Table SOTA imitation learning method like GAIL and SAILs total fails in this task. Our method GAIL-USN outperforms all the baselines in the Atari game. RIL_CO also performs closely the optimal performance GAIL*, However, it requires more computation cost due to the usage of co-training. 3 shows the performance of GAIL-USN and other baselines on Seaquest game across multiple dependet action noise.
Table 4: Average return of GAIL, GAIL-USN, RIL, RIL_CO SAIL-soft and SAIL-hard on the Atari game with 30% state- dependennn action noise in both the demonstration and the environment.
GAIL*
GAIL-USN (Ours)
GAIL
RIL
RII_CO
SAIL-soft
SAIL-hard
Seaquest 1,553.5
1,564.7
159.4
776.3
1,506.6
29.7
43.2
F EXTENSION TO GENERAL Q-LEARNING METHOD
USN is general method for robust training of imitation learning against action noise. BCQ-USN can easily apply USN on top of the generative model of BCQ methods. Because the generative model similar to a classifier in label-noise lcarning tasks. Therefore, BCQ-USN can naturally employ the noise-tolerant methods for positive learning and negative learning. However, many IL methods
hutps://itthhh.co//nvve///////l
29

---

Under review as a conference paper at ICLR 2023
Table 5: Average return of GAIL-USN and other SOTA baselines the Seaquest game with diverse levels of state dete-dependent action noise in the demonstrations and environments. The reported results are average across 10 random seeds.
Noise Rates
Method
0.1
0.2
0.3
0.4
GAIL* *
1,553.5
1,553.5
1,553.5
1,553.5
GAIL
1,695.7
1,321.3
159.4
1,214.6
GAIL-USN
1,648.6
1,677.6
1,564.7
713.4
RIL
454.0
396.0
776.3
761.9
RIL_CO
1,303.0
993.3
1,506.6
1,538.0
SAIL-soft
0.0
760.6
29.7
455.7
SAIL-hard
0.0
760.6
43.2
455.7
do not have such generative model. How extend USN to such methods is an interesting and essential problem. In this section, we show how to apply USN some general Q-learning based II. methods, e.g. the conservative Q-learning (CQL) (Kumar 2020).
F.1 CONSERVATIVE Q-LEARNING (CQL)
Offline RL algorithms promise to learn effective policies from previously-collected,, static datasets without further interaction. However, in practice, offline RL presents major challenge, and standard off-policy RL methods can fail due to overestimation of values induced by the distributional shift between the dataset and the learned policy, especially when training on complex and multi-modal data distributions. Kumar et al. (2020) proposed conservative Q-learning (CQL), which aims to address these limitations by learning conservative Q-function such that the expected value of policy under this Q-function lower-bounds its true value. They theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees. In practice, CQL augments the standard Bellman error objective with a simple Q-value regularizer which is straightforward existing deep Q-learning and actor-critic implementations. Eq. I addresses the impact of out-of- distribution actions and obtain conservative value estimates. Algorithm presents the procedures of CQL. To use CQL for offline imitation learning, train intrinsic curiosity module (ICM) (Pathak ct al.
implement on top of
2017 for generating intrinsic rewards.
min EEED log Σ exp(((ss -Eunts(atsl(((ss))) ((s, +
B Eo (0- (1I)
Q
where T r(s,,) a) + /B a.) mmax.' Q(s' a')] is the Bellman optimality operator.
Algorithm 5 CQL
Initialize Q-function Qo. and optionally a policy πg.
2: for t = 1, ... do
3: Train the Q-function using objective from Eq. (IT)
4: (only with actor-critic) Improve policy ny via with SAC-style entropy regularization.
5: end for
F.2 CQL-USN
Since there is generative model CQL, we apply USN on top of the Q network. We first introduce a robust IL baseline algorithm, CQL-GCE, by training CQL with additional GCE loss on Q:
(1 a))
Lpo =
(12)
q
30

---

Under review as a conference paper at ICLR 2023
We use CQL-GCE for positive learning the full-batch data. Then, estimate the predictive uncertainty measures (calibration errors) on Q. Next, we set the sample-selection threshold using the calibration errors, and select large-loss samples for soft negative learning. This simple extension called CQL-USN, In CQL-USS (Algorithm 6), the Q network is trained in multi-task manner, predicting values and noise-tolerant action prediction.
Algorithm 6 Robust CQL (e.g. CQL-USN)
1: Initialize Q-function Qo. and optionally policy #d-
2: for = 1,... do
3: Update Q-function parameters by minimizing Eq. (II) and USN in Algorithm
4: (only with actor-critic) Improve policy The via with SAC-style entropy regularization.
5: end for
F.3 EXPERIMENTS AND RESULTS
Settings. We evaluate the effectiveness of CQL-USN on offline datasets Atari games from the d4rl benchmark (Fu al. 2020). We build method on top of d3rlpy library (Takuma Seno 2021), and use its default hyperparameter settings.
Table Average return of CQL,, CQL--GEE and CQL-USN on hert with state-dependen action noise.
Noise rate
0.1
COL CQL-GGE CQL-USN
4,911.0
5,088.5
5,455.0
Table 7: Average return of CQL, CQL-GCE and CQL-USN Q*bert with symmetric action noise.
Noise rate
0.1
0.2
0.3
0.4
CQL 6,096.0
5,050.0
4,000.0
2,721.5
CQL-GCE
6,312.5
5,434.5
4,189.0
4,409.0
CQL-USN
7,752.0
5,615.5
2,975.0
4,933.0
Results. We compare the average return CQL, CQL-GCE and CQL-USN with state-dependent action noise (Table 6), symmetric action noise (Table 7 and pairflip action noise (Table 8). CQL- USN usually outperforms CQL-GCE and CQL on the three types of action noise significantly across multiple noise rates.
G MORE PREDICTIVE UNCERTAINTY DETAILS
Expected Calibration Error (ECE) (Nacini et al. 2015 Guo et al. 2017) is widely used to measure the predictive uncertainty (model calibration) deep network. approximate the calibration error in expectation, ECE discretizes the probability interval into fixed number of bins, and assigns each predicted probability to the bin that encompasses it. The calibration error is the difference between the fraction of predictions in the bin that correct (accuracy) and the mean of the probabilities in the bin (confidence). Intuitively, the accuracy estimates IP(Y = ylğ p). and the average confidence a setting of p. ECE computes weighted average of this error across bins:
ECE Σ (acc(b) (b))
(13)
where n& is the number of predictions in bin b. N is the total number of data points, and ace(b) and conf(b) (b) are the accuracy and confidence of bin b, respectively. ECE as framed in (Nacini et al. 2015) leaves ambiguity in both its binning implementation and how to compute calibration
31

---

Under review as a conference paper at ICLR 2023
Table 8: Average return of CQL,, CQL-GCE and CQL-USN Q*hert with pairflip action noise.
Noise rate
0.1
0.2
CQL
5,830.0
4,438.0
CQL-GCE CQL-USN 7,130.5
5,352.5
4,647.5
5,348.0
for multiple classes. In Guo al. (2017). they bin the probability interval [0; into equally spaced subintervals, and they take the maximum probability output for each datapoint (i.e., the predicted class's probability). We use this for our ECE implementation.
Adaptive Calibration Error (ACE) improves ECE by using adaptive calibration ranges. The motivation is that in order to get the best estimate of the overall calibration error the metric should focus on the regions where the predictions are made (and focus less on regions with few predictions) (Nixon et al. 2019p. ACE uses an adaptive scheme which spaces the bin intervals so that each contains an equal number of predictions. In detail, ACE takes as input the predictions P (usually out of softmax), correct labels, and number of ranges Rt
ACE
lace(r, k) - cons(r, k)).
(14)
KR
where ace(r, k) and con k) are the accuracy and confidence of adaptive calibration range r for class label k, respectively. Thresholded Adaptive Calibration Error (TACE) is identical to ACE, with the only difference being that TACE is only evaluated on values above e.
Static Calibration Error (SCE) is simple extension of Expected Calibration Error to every probability in the multiclass setting. SCE bins predictions separately for each class probability, computes the calibration error within the bin, and averages across bins (Nixon et al., 2019)
K
Tbbk N
SCE
lace(b, k:) - con k)),
(15)
K
where acc(b, k) and conf(b, k) are the accuracy and confidence of bin b for class label k. respectively; ToA is the number of predictions in bin b for class label k; and N is the total number of data points. Unlike ECE, assuming infinite data and infinite bins, SCE is guaranteed be zero if and only if the model is calibrated.
Root Mean Square Calibration Error (RMSCE) (Hendrycks al. 2018 measures the square root of the expected squared difference between confidence and accuracy at a confidence level. The RMSCE is estimated with the numerically stable formula:
Σ n
(ace(b)-com) conf(b))
RMSCE
(16)
N
H ABLATION STUDIES
In this subsection, we study the effects of several essential hyper-parameete and components in LunarLander-2 task with state-dependent action noise = 0.45). This ablation follows the rule of 'study one, fix the others'. The ablation study results are illustrated in Figure and Figure 35
Effect negative learning loss. We evaluate the performance of our method using the classic complementary-labell learning approach, pairwise-comparisonn (PC) loss (Ishida al., (201)) for negative learning. We denote this instance as BCQ-USN-PC. Figure [0(c) and Figure 10 show that BCQ-USN-PC: is comparable the original BCQ-USN when learning with state dependent action noise, symmetric action noise and pairflip action noise. This implies that our method can generate many variants by replacing the negative learning loss.
Effect of sample-selection threshold. Figure shows that BCQ-USN outperforms BCQ-GCE significantly with diverse calibration error metrics when Xnng 1.0. We set sample-selection
32

---

Under review as a conference paper at ICLR 2023
threshold any types of calibration enror metric, which demonstrates the good scalability of our method.
Effect of unlikely action threshold of generative model G. Figure 35(a) shows that BCQ-USN outperforms BCQ-GCE significantly with five calibration error metrics and one fixed value.
Effect of the weight of negative learning loss Aneg. We select Aney from [0.01, 0.1, 1.0]. and find our method BCQ-USN achieves the best performance with Aneg = 1.0, and all three instances outperform BCQ-GCE (the blue dash dot line in Figure Figure(35(b))) Effect of smooth rate To avoid tuning of the smooth rate in soft negative learning loss, we set the smooth rate as the estimated calibration errors. Figure 35(c) shows that ECE and TACE the worst, and the other four instances all outperform BCQ-GCE with a large margin. The above ablations show that our method is not sensitive the choices of myyer-parameters More importantly, the calibration error metrics can be of multiple use, which maintains the robustness of BCQ-USN while avoiding the effort of tuning hyper-parametee
Seylin Daginn
(a)
Threshold
(b) Negative
learning
(c) Smooth rate
weight Aney
Figure 35: The effects of action threshold t, smooth rate o, and negative learning weight Xneg-
H.1
DETAILED ABLATION STUDY RESULTS
Effect o' Selple-selection Threshald
Effect Samale-selection Threshold
171
Aieeagge Pertin
Average Rorutn
150
129
100
100
Sample-seeection Threshold
Sample-selection Threshold
Figure 36: The effect of diverse sample-selectionFiguuee 37: The effect diverse sample-selection threshold тn when Xneg 1.0.
threshold when Xneg 0.01.
Effect of sample-selection threshold. Table 9 and Table shows the ablation results on diverse sample-selection threshold when Aneg 0.01 and Aneg 1.0, respectively. BCQ-USN with Aneg = 1.0 outperforms that trained with Aneg = 0.01 no mater the choice of sample-selection threshold. The corresponding bar chats comparison outperforms the BCQ-GCE by a large margin when using the estimated calibration error metrics (predictive uncertainty) the small-loss sample-selection threshold. Therefore, predictive uncertainty good alternative to the manual tuning sample-selection selectionn threshold, since it requires no prior knowledge about noise and it has good adapbility to diverse noise types and noise rates.
shown in Figure 36 BCQ-USN usually
Effect of threshold т. The ablation results on the unlikely action threshold 7 are shown in Table BCQ-USN significantly outperforms BCQ-GCE when setting different predictive uncertainty
33

---

Under review as a conference paper at ICLR 2023
Table 9: Average return of our method LunarL.ander-v2 with noise rate 0.45, action threshold = 0.3 and negative learning weight Xneg = 0.01 across different small-loss sample-sclection threshold.
Small-loss sample threshold
ECE
ACE
TACE
SCE
RMSCE
Max CE
LunarLander-v2
105.0
180.5
102.3
8.0
50.5
177.9
Table 10: Average return of our method on LunarLLanerr-e with noise rate c c= 0.45, action threshold = 0.3 and negative learning weight Xneg 1.0 across different small-loss sample-selection threshold.
Small-loss sample-selection threshold
ECE
ACE
TACE
SCE
RMSCE
Max_CH
LunarLander-v2
198.1
199.5
185.1
195.4
191.6
203.8
measures except ECE. This means that the predictive uncertainty measures are usually reliable choices for eliminating the unlikely action for policy training in BCQ and its variants.
Effect of Threshold
Averege Renuln
102
ACE
TACE
Mani Ct
Hosd0.33
Action Threshole
Figure 38: The effect of diverse action threshold when Xneg 0.01.
Table 11: Average return of our method on LunarLander-v2 with noise rate 0.45 across different
threshold 7 of BCQ when Äneg = 0.01.
ECE
ACE
TACE
SCE
RMSCE
Max_CE
Fixed 0.3
LumarLander-v2 -103.9
91.2
165.9
173.0
173.0
177.9
177.9
Table 12: Average return of our method on LunarLander-vv2 with noise rate threshold 7 of BCQ when Aneg 1.0.
0.44 across different
ECE
ACE
TACE
SCE
RMSCE
Max CE
Fixed 0.3
Lunarl.ander-v2 208.9
191.2
184.2
189.9
205.5
203.8
203.8
Effect of negative learning weight Aneg- Table 13 shows the ablation results the multiple choices
of negative learning loss weight Anag-
Effect of smooth rate Table shows the ablation results using different calibration error metrics as the smooth rate a in soft negative learning loss.
Effect of negative learning loss. compare the performance of the original BCQ-USN with the one using a classic complementary label learning method, e.g. Pairwise- cww.viii-oomariooonn (PC) loss (Shida et al. 2017) for negative learning:
34

---

Under review as a conference paper at ICLR 2023
Table 13: Average return of our method Lunarl aander-22 with noise 0.45, action threshold
0.3 and smooth rate a = Max CE across different negative learning weight Aneg-
Aneg
0.01
0.1
1.0
LunarLander-v2
177.9
94.9
203.8
Effect Smooth
Fveragg Reruer
325
TACE
Mor
Smooth
Rate
Figure 39: The effect of diverse smooth rate when Xneg 0.01.
Table 14: Average return of our method on Lunarl ander-v2 with noise rate e = 0.45, action threshold
0.3 and negative learning weight Aneg 0.01 across different smooth rate
Smooth Rate
ECE
ACE
TACE
SCE
RMSCE
Max CE
LunarLander-v2
0.9
160.4
17.0
177.9
177.9
177.9
Table 15: Average return of our method LunarLander-v2 with noise rate 0.45, action threshold
T 0.3 and negative learning weight Aneg = 1.0 across different smooth rate a.
Smooth Rate a
ECE
ACE
TACE
SCE
RMSCE
Max CE
LuderLander--v
197.5
201.4
198.6
203.8
203.8
203.8
Ishida et al. (2017) first formulated the problem of complementary-label learmin (CLL) and proposed
a risk minimization framework. The goal of CLL is to learn classifier f that minimizes the classifica- tion risk (17). from only the complementary labeled samples { ] {1r,, (5)) 7)) are drawn independently from unknown probability distribution with density: p(x, 17)
p(x, 1)
where K is the number of classes.
R(f) Epta.v) [C) v)]
(17)
To achieve this goal, [shida et al. (017) proved that the risk (17 can approximated in an unbiased fashion:
R()) = K Σ E(f((, - M1 + Mz.
(18)
n
where e is complementary loss. They considered the one-versus-all (OVA) loss (f))(
R Ensg (x), + E( (x)) and pairwise comparison (PC) loss:
Epo(f(x).
Σ ((gy(x) gz(z)),
(19)
where {(z) is a binary loss which satisfy f(z) 2(-2) 1, such as the sigmoid loss Es(2) T+ LoVa satisfies this condition with M1 = K and M2 = 2. and Σpc satisfies the condition with Mi K(K 1)/2 and M2 K
35

---

Under review as a conference paper at ICLR 2023
Table 16: Average return of offline imitation learning the Lunarl Lnder-v2 with diverse levels of symmetric action noise.
Noise Rates
Method
0.05
0.25
0.45
0.65
0.85
BCQ
207.7
132.1
202.1
-217.6
-153.8
BCQ-GCE
42.3
181.9
90.8
145.0
-228.8
BCQ-USN-PC
194.2
179.9
198.5
195.4
-119.7
BCQ-USN
168.8
191.2
211.1
186.0
-184.7
Results on state-independen action noise. In Table 16 and Table 17 we show the comparative results of BCQ, BCQ-GCE. BCQ-USN and BCQ-USN-PC on Lunarlanderr-v2 with symmetric action noise and pairflip action noise. BCQ-USN is comparable to BCQ-USN-PC, and are usually better than BCQ-GCE and BCQ across diverse noise rates.
Table 17: Average return of offline imitation learning on the LunarLander-vv2 with diverse levels of pairflip action noise.
Noise Rates
Method
0.05
0.25
0,45
0.65
0.85
BCQ
204.3
186.2
142.5
-370.1
-122.9
BCQ-GCE
134.0
120.4
176.8
186.5
-747.0
BCQ-USN-PC
200.9
200.4
186.3
172.5
-799.3
BCQ-USN
193.0
201.2
194.5
168.0
-773.8
Results state-dependent action noise. Training with PC loss, BCQ-USN-PC achieves comparable results to the original BCQ-USN in LunarLander-v2 across multiple levels of state-dependent action noise (Table 18
Table 18: Average return of offline imitation learning the state-dependent action noise.
with diverse levels of
Noise Rates
Method
0.05
0.25
0.45
0.65
0.85
BCQ
208.0
207.4
-335.2
-624.3
-560.2
BCQ-GCE
172.3
188.7
-159.0
67.4
-518.0
BCQ-USN-PC
188.3
175.8
199.0
-34.8
-583.2
BCQ-USN
188.0
179.2
177.9
165.5
-546.2
36


Reviews:
[{"review_id": "vGn0yLz3_aP", "reviewer": "Reviewer_k6dx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents an approach for learning better policies for imitation learning (both offline/online) in the presence of action noise; notably, this work makes a distinction between state-independent action noise (e.g., an annotator picking a random action for a state) vs. state-dependent action noise (e.g., a complex motion in a narrow part of the state space that affects action labels in a correlated way). \n\nTo handle these types of noise, this work proposes uncertainty-aware sample selection with soft negative sampling (USN), a framework that can be plugged into existing imitation learning algorithms like BC-Q; the punchline is to first train an imitation learning policy from all demonstrations, estimate the predictive uncertainty via e.g., Expected Calibration Error (ECE), and use the corresponding threshold to identify “large-loss” examples to inform a negative sampling procedure. The key assumption here is that “large-loss” state-action pairs correspond to transitions with high-uncertainty; rather than “force” a policy to act subject to the given label, it’s better to just replace the given label with a “random” action —> essentially increasing the entropy/model uncertainty at that state, leading to a better policy.\n\nRather than build the results on a “simple” imitation learning algorithm like behavioral cloning, this work instead builds USN (and corresponding results) on BC-Q, training a separate intrinsic reward component (Pathak et. al., 2017) to act as the reward for learning the Q function.\n\nThe results on a few discrete action environments from Atari (Name This Game, Seaquest, Lunar Lander) —> show that USN outperforms BC-Q and online IL approaches as the noise ratio increases (state-independent), and in cases of state-dependent noise.", "summary_of_the_review": "While the motivation behind the approach is well-formed, I don’t think the proposed algorithm for using large-loss examples as negatives to harden imitation learning policies to action noise is evaluated thoroughly. There are a lot of hard-to-swallow assumptions in the current evaluation (using BC-Q with a learned intrinsic reward component — a method that fundamentally is flawed and doesn’t generalize to more stochastic environments), and the choice of only evaluating discrete action spaces is questionable. \n\nI’d love to see 1) experiments using USN on top of traditional Behavioral Cloning, and 2) experiments in continuous action spaces with real-world noise (e.g., from humans) to truly believe the proposed USN framework generalizes.", "strength_and_weaknesses": "I think that this paper presents a neat idea; action noise is a huge problem for imitation learning as a whole; however, the proposed approach and evaluation do not help address this, or truly evaluate the generalizability of the USN approach at all.\n\nIt’s not clear to me why this work chooses to implement USN on top of BC-Q as the base offline IL algorithm, relative to something like standard Behavioral Cloning (BC). BC-Q requires reward annotations to learn the Q-function, and; while this work correctly avoids using the ground truth environment reward (which would implicitly tell you what actions are/aren’t suboptimal; basically turning this into an offline RL problem), they instead **choose to train an intrinsic reward component following Pathak et. al. 2017** — the form of this intrinsic reward is that of a predictive model that uses surprisal against a learned forward dynamics model as a signal. However, in follow-up work by the same authors (Pathak et. al. 2018), this form of intrinsic reward is **explicitly stated to be problematic in the cases of stochastic environments and crucially, when actions are noisy/suboptimal, as it impacts learning the predictive model. These environments are mostly deterministic, and the noise (when injected) is done so in targeted ways that don’t seem to reflect the real-world settings hinted at in the introduction.\n\nI fundamentally don’t trust these results; there are a lot of design choices that seem poorly motivated (why skip straight to Monte Carlo dropout instead of just using entropy of predictions, given actions are discrete), and I think that for there to be a true claim of addressing problems of action noise in imitation learning, we’d need at least one-two experiments in continuous control settings where this problem is more prevalent and damaging.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZLkIPaES3p", "reviewer": "Reviewer_c6qo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work proposes an algorithm, USN (uncertainty aware sample selection for Negative learning) for improving performance results in imitation learning, especially under the presence of high action labelling noise. The work seems to be of the class where the demonstration has to be parsed by some human labor whose errors made on action labels are termed as action noise. The key idea is to figure out data samples from the demonstration which have a high predictive uncertainty (which they show is negatively correlated to generative loss), and that negative learning on such samples can help with imitation learning.", "summary_of_the_review": "I find that the question being investigated by the authors is interesting, which is to accommodate for action noise for Imitation Learning setups, however through their results it seems that the major benefits of this approach are reaped beyond a high enough noise level which I do not believe are typically achieved by amateaur or expert labellers. Among other concerns about the setup, I feel that the work would benefit from a more thorough and consistent experimentation to bolster their claims.\n", "strength_and_weaknesses": "Strengths : \n1. The paper seems to be generally well written, where enough background information was provided for good readability. \n\n2. The paper addresses the problem of accounting for (by being robust to) action labelling noise - which in some sense allows for more amateur labellers - which is a welcome direction [however more on this in the weaknesses].\n\n3. I also appreciate the authors’ attempt to showcase why their intuition behind using predictive uncertainty as a means to realize samples on which they must perform negative learning, makes sense via experiments on  NameThisGame. \n\nWeakness : \n\n1. One of my major concerns is the fact that the experiments show the marginal improvements of USN algorithm over the baselines at high levels of noise. Since the labellers are actually humans (in the idealized setup given by the authors), I do not see how experts or amateur labellers have noise as high as 0.5 or even more (basically they are incorrectly labeling half the time?). Even if the work improves the performance measures beyond 0.5 action noise value, does it actually solve the original problem?\n\n2, Secondly, I find that the experimentation setup is very limited and can benefit from testing across more environment setups. Why has different environments been used for different setups of Robustness to state independent noise (Atari not used?, only Lunar Lander), Robustness to state dependent noise (Atari + Lunar)?(& Section 5.1 uses Q*bert)? \n\n3. The authors provide examples of how amateurs or even experts can end up providing noisy action labels. Firstly, I would have thought that it wouldn’t be as high as 0.5, unless authors performed a user study to validate that such high levels are actually seen. Secondly, even if they are seen I would image that it is maybe because of the large action space like continuous action spaces where getting the exact action maybe hard for the labeller. But the environments chosen for the experiments, especially Atari domains mostly have very limited number of actions. For example, only relevant actions in Assault would be left, right, fire, no-op. I do not see why such settings correctly represent authors original motivation from Section 1 and 2.\n\n4. Although I welcome the discussion in section 3.2, I find that the inferences drawn like the correlation between generative model’s loss vs the predictive uncertainty to be quite ambitious from the perspective of the extent of experimentation. I may agree that the said correlation may exist, but, in my view, experiments on a single domain shouldn’t be used to draw conclusions.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TLOYeVIkOw7", "reviewer": "Reviewer_gQZL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates a practical setting in imitation learning, where a fraction of expert demonstrations are noisy demonstrations. The authors propose to select hard samples by measuring the uncertainty and update the model with the selected samples. The motivation comes from the neural network, which tends to fit easy samples first. The additional training on the selected hard samples can thus guarantee the generalisation of the model. Empirical results show the effectiveness in Box2D and one Atari games.", "summary_of_the_review": "Overall, the paper is well-written and easy to follow.  The method is sound and somewhat novel. However, I have concerns about the generalisation of the method since it seems to rely on the choice of non-optimal demonstrations. Also, some implementation details in the experiment seems missing. As a result, I provide my initial score as ``boardline accept''", "strength_and_weaknesses": "Strengths\n* Imitation learning from noisy demonstrations is a practical setting.\n* The motivation is clear and reasonable.\n* Empirical result shows good performance.\n\nWeaknesses\n* The performance of the method seems to quite rely on the choice of non-optimal demonstrations. The non-optimal demonstrations are defined as demonstrations with noise in the paper, in such a way the uncertainty estimation technology can be used to find the noisy demonstrations. However, what if we define non-optimal demonstrators as early-stage RL training checkpoints. Can the uncertainty estimation scheme still find the non-optimal demonstrations?\n* In Atari domain, the authors only choose 1 game for evaluation. Results on more Atari games can make the results more convincing. Also, more details about the experiments can be given. For example, in Atari domain, how many demonstrations are used for GAIL training? Is there is a need to specially design GAIL since many works point out that directly applying GAIL into Atari domains leads to bad performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject