Adversarially Robust Imitation Learning
Authors: Jianren Wang
Ziwen Zhuang
Yuyang Wang
Hang Zhao
Venue: CoRL2021 Poster (2021)
Keywords: Imitation Learning
Adversarial Learning

Abstract: Modern imitation learning (IL) utilizes deep neural networks (DNNs) as function approximators to mimic the policy of the expert demonstrations. However, DNNs can be easily fooled by subtle noise added to the input, which is even non-detectable by humans. This makes the learned agent vulnerable to attacks, especially in IL where agents can struggle to recover from the errors. In such light, we propose a sound Adversarially Robust Imitation Learning (ARIL) method. In our setting, an agent and an adversary are trained alternatively. The former with adversarially attacked input at each timestep mimics the behavior of an online expert and the latter learns to add perturbations on the states by forcing the learned agent to fail on choosing the right decisions. We theoretically prove that ARIL can achieve adversarial robustness and evaluate ARIL on multiple benchmarks from DM Control Suite. The result reveals that our method (ARIL) achieves better robustness compare with other imitation learning methods under both sensory attack and physical attack.

Adversarially Robust Imitation Learning
Jianren Wang* j jianrenvandree. cmu edu
Ziwen Zhuang
ahuaagzw@shanghaitecchhh edu,
Yuyang Wang yuyangu@andrew. cmu, cmu.edu
Hang Zhao hangzhao01248gmaill com
Modern imitation learning (IL) utilizes deep neural networks (DNNs) as function approximators mimic the policy of the expert demonstrations. However, DNNs can be easily fooled by subtle noise added to the input, which is even non-detectable by humans. This makes the learned agent vulnerable to attacks, especially in IL where agents can struggle to recover from the errors, In such light, we propose sound Adversarially Robust Imitation Learning (ARIL) method. our setting, an agent and an adversary are trained alternatively. The former with adversarially attacked input at each timestep mimics the behavior of an online expert and the latter learns to add perturbations on the states by forcing the learned agent to fail on choosing the right decisions. We theoretically prove that ARIL achieve adversarial robustness and evaluate ARIL multiple benchmarks from DM Control Suite. The result reveals that our method (ARIL) achieves better robustness compare with other imitation learning methods under both sensory attack and physical attack.
Abstract:
Keywords: Imitation Learning, Adversarial Learning
1 Introduction
Imitation learning a powerful and practical alternative to reinforcement learning for learning sequential decision-making policies. Together with the computational advancement brought by deep learning. imitation learning has shown great success in autonomous driving [1, 21, robot control [3], game AI [4], and motion capture [5]. Despite achieving human-level performance on many tasks, the existence of adversarial examples [6] in DNNs and many successful attacks to sequential decision- making policies [7. 8, 10] motivate us study robust imitation algorithms.
The observations by the agent unavoidably contain uncertainty that naturally originates from sensor errors equipment inaccuracy. A policy not robust to such uncertainty can lead to catastrophic failures (e.g., a small piece of white scotch tape on a traffic sign could cause an improper left turn, which will further lead miles of unnecessary travel and even crush of the vehicle in the worst case). Therefore, studying the fundamental principles of robust imitation learning is crucial.
One effective approach to induce robustness is domain randomization [11], method where designer with expertise identifies the components of the model that they are uncertain about. A set randomized, ensuring of training environments is constructed where the uncertain components parametrization of the that the agent is robust on average to this set. However, this requires careful uncertainty set well as hand-designing of the environments. A more easily automated approach to formulate the problem as a zero-sum game and learn an adversary that perturbs the transition dynamics [12, 13, 14]. If Nash equilibrium of this problem is found, then that equilibrium provides a worst-case performance bound under the specified set of perturbations. Specifically, we can search adversary policies and agent policies π, so that min max ye J ʃ), where J(T,, is the expected cumulative reward for the attacker policy. Unfortunately, in most imitation learning settings, quantitative reward from the environment is unavailable.
In this work, we propose a theoretically grounded framework of adversarially robust imitation learning (ARIL), where we build surrogate loss to measure the distance between the actions generated from
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

and the actions from the expert. We consider two ways of attack: sensory attack and physical attack. In the sensory attack setting. the adversary only modifies the inputs to π (e.g., the underlying true state of the environment is untouched as sc). while in the physical attack setting, the adversary directly modifies the actual physical state, i.e., it changes the underlying state to 8/- We consider attacking under interactive imitation learning (II) setting where the expert is available for queries at any time step [15]. We evaluate our method in DM Control Suite [16]. (e.g. HalfCheetah, Swimmer, Ant), under both sensory attack and physical attack settings. Our method significantly improves the robustness under adversarial attacks over the baseline methods.
To summarize, this paper makes four contributions: (1) We study the theory and practice of Adversar- ially Robust Imitation Learning (ARIL). (2) We systematically define sensory attack and physical attack. (3) We prove that ARIL can achieve the adversarial robustness under realizability assump- tion, robust expert assumption, and optimal adversary assumption.
(4) We show the efficiency of our method in both attack stage and defense stage on multiple DM Control Suite benchmarks in comparison with other adversarial attack methods and other robust imitation learning methods.
2 Related Work
Adversarial Attack Adversarial attacks can be divided into white- box attacks and black-bor -ox attacks. In white-box attacks, the attack has access the structure, algorithm, and system parameters of the target model. In black-box attacks, the attack cannot obtain information from the target model. Instead, it can only interact with the target model by querying with the input and observing the output. In our setting, we consider white-box attacks. Szegedy et al.[1]] first pointed out the intriguing property of neural networks and proposed a method called Large-BFGS to find adversarial samples. Goodfellow et al. [18] proposed a Fast Gradient Sign method (FGSM) to generate small perturbations to fool the neural networks by calculating the gradient of the loss function concerning the inputs. Kurakin et al. [19] proposed Targeted FGSM to generate adversarial samples towards a target class and Iterative FGSM to apply FGSM multiple steps with small step sizes. Tramer et [20] proposed a randomized single-step attack to escape the non-smooth vicinity of the data point. Kurakin er al. [19] apply adversarial training on large-scale datasets. Huang el al. [7] showed that adversarial examples generated from FGSM could degrade test-time performance of well-trained NN policy in RL. Sun el al. [21] studied the safe sequential decision-making problem under the setting of adversarial contextual bandits which is one-step RL problem. Lin et al. considered strategically timed attack where the attacker picks specific time steps to attack. Han et [22] considered attacking an RL algorithm during training time. All these previous works mainly focus applying attacks developed from the supervised learning literature to RL setting, while in this work, we focus on learning adversarial attacks in imitation learning.
Robust Imitation Learning Imitation learning aims to learn sequential Dacision-making policies from expert demonstrations. However, the agent can be vulnerable to cascading failures when the trajectory diverges from the demonstrations [23]. Laskey et [24] proposes to inject noise into the supervisor's 's demonstration policies, which forces the supervisor to demonstrate how to recover from errors. Wang et [25] utilizes VAEs [26] and GAIL [27] to learn semantic policy embeddings in imitation learning, which learns robust diverse gaits with limited demonstrations. For robust imitation learning with different- -ureerenntquaiity demonstrations, Tangkaratt al.[ makes use of probabilistic graphical model to estimate the quality of demonstrations. Brown et al. [29] proposes a safe and robust Bayesian reward learning through preferences over imitating demonstrations, Shin et al. [30] performs the derivative- random search optimization method with linear policies. In our work, we focus the robust imitation learning against adversary attacks, which can cause catastrophic failure to DNN-based agents with trivial perturbations.
3 Preliminaries
Before we delve into the details of ARIL, we first outline our terminology. In the rest of the paper, we focus on finite-horizon Markov Decision Process (MDP), which is defined as (S, H, P, r) where S is the state space, A is the action space, r is a reward function, P is the transition and H E N is the finite horizon. policy и S - ((AA is mapping that maps from states to distributions over action space, i.c., T(als) is the probability of picking given s.
2

---

3.1 Sensory Attack vs Physical Attack
For any pair of policy and attack (T, f). we can treat the composition of them as a policy: π S + A. i.e., a ~ (It o (s) is equivalent to first generate stochastic adversarial example s' ~ f(-(s), and then sample action accordingly
x(-ls".. Denote d, as the average state distribution induced by policy x. As f can be understood as policy, drof will be the state distribution induced by a f.
We first introduce the sensory attack setting where the adversary can only modify the sen- sory inputs of the RL agent, but not the underlying physical state. Denote trajectory as = s1 01, sH, Sn, } generated from the composite policy π o f, the likelihood of such trajectory is Paof:
(T) = III t=1
(1)
where we denote P(si) P(s)|so. ao) as the initial state distribution. Note that here adversary modifies state 8, to 8/ and then the policy x only sees s/c-
For physical attack setting, the adversary will have the ability to modify the underlying physical states. Again, we denote a trajectory as т {ss $1, 01..., sH, s ah} generated from the composite policy π o f, the likelihood of such trajectory now is:
p = =II 1=1
(2)
Note the difference between Eq. 2 and Eq. 1, where transition probability is conditioned on attacked state. In physical attack (Eq. 2), the adversary modifies the state 8 to 8', and then the environment actually generates the next state based on s', as the underlying physical state is now s'. The sensory attack is similar attacks considered in today's 's supervised learning setting, where the physical attack is similar to the robust control setting (e.g., a min-max formulation of Hox optimal control [31] under physical disturbances).
Note that under both settings, the attacks are sequential: the adversary can affect the learner's decision via corrupting the state s, which will, in turn, affect the future states. In supervised learning, attack on a given sample will not affect the future samples, since samples are i.idd in supervised learning.
3.2 Expert Robustness
For the expert n", we have:
=
In practice consider the setting where the expert n" is human driver who will not be affected by adversarial examples with small perturbations (e.g., human easily tell a stop sign even with sticker on it). We later will relax the notion of expert robustness to include the possibility that the expert will be affected by adversarial attacks.
We consider interactive imitation learning setting proposed [32] where the expert лf is always available during training time. A learner can query the expert any state during training to get feedback n (s).
4
Adversarially Robust Imitation Learning
The goal of adversarial robust imitation learning (ARIL) is to learn policy that can achieve good performance under all possible attacks:
म arg min max J [TT, f), rel JEF
(4)
where [TT, f') is the expected total cost of executing policy with examples corrupted by attack f:
H
J(T, f) = E
Σ
Elan, n (8n))) ~ (f(()))), sh+1 ~ P(-|an, an) (sensory attack) (5)
J(T, = EE
Elan, n (sn))lan ~ ((((s))) sht1 ~ ((-If(s)), an)
(physical attack) (6)
3

---

We consider a loss function E(a, d') that measures the difference between two actions and a'. For instance,&(o, d') could be zero-one loss if A is discrete, or 12 distance between actions if A is continuous.
If control the long-term prediction error of the learned policy # under the worst possible adversary,
max Bawdaos [aa++())())) [A(a, H" (s))]] < 6, EEJ
then we can upper bound the performance of ये at test time with the existence of an arbitrary adversary F. To see that, let consider the setting where A is discrete and ((a, a') is the zero-one loss:
J(z - J(r = Egnddos
(f(a) (f(ss,), a) - (*),
(8)
I 2Q mia Bondaes aa ar
(f))) (f((s [a + (s)]]
(9)
I 2Q max sond am ((()) [1 + (s)]]
(10)
FEF
= =Q max max Eandyos [Ean))(s) [1,,, म (s))] S 2Qmax
(11)
JEJ
Here, Q denotes the state-action value of the expert policy and Q denotes the upper bound of Q Namely, if is small, the learned policy can achieve similar performance as the expert, even under an adversarial attack f during the execution of ât. This justifies the reason for finding स that minimizes the objective function in Eq. 7.
Let us define the objective as follows:
[(T, f) Eondany [BaNa(((e) [e (a, x (s))]
(12)
We perform an iteratively alternating update for x and f.
4.1 Update æ with fixed f
At iteration t, given the current fi, and we want to update to T++1- We consider the loss function with respect π round t:
(((T) = Bond Bawx(f((s)) [e(a, n- (x))]
(13)
which a classification loss where features are corrupted sampled s dmaffo and labels are computed from expert query n² (s).
Note that for discrete action space, the above loss function simply classification loss of the classifier π, with features generated from the distribution dre and the ground truth label is generated from n.. We perform no-regret update on & (TT) to generate T++1-
T++1 = arg min
Σ ((T). 1=1
(14)
well
Note that the above optimization can be understood sampled from the mixture of distributions {dxzof, Vi with equal weight 1/t, and the corresponding labels are generated based on л" ft for a data point sampled from dx,of, Vi e [t]. In practice, need to samples to replace expectations, which leads us to Data Aggregation style update (DAgger [32]) for π.
classification task where the features are
4.2 Update with fixed
At iteration I, given the current ft and n,, we want to update fa to fe+1- We consider the objective function with respect to f round as:
?.() = Ex-deer [((Te( (s))]
(15)
We want to find / that maximize the above objective function, namely, increasing the classification loss of the current policy n, as much as possible. Note that different from the loss function for updating n, here the expectation is defined with respect the fixed and f. If is stochastic

---

Algorithm Adversarial Robust Imitation Learning (ARIL)
1: Input: MDP M, policy class II, expert policy x", set of adversaries F, a supervised learning algorithm LEANER
2: Initialize a dataset D = 9
3: Initialize a policy mo
4: for t = 0 to NN do
5: ft = arg fax Bx-daror (f))))())); x (s))] (RL. Step)
6: Execute मe o fe to generate set of non- corrupted states {5:]
7: Query expert during execution to get labels {TT(s))
8: Data Aggregation: D D + {f((s)) (x:)]==1
9: TT++1
end for LEARNER(D) (Supervised Learning, e.g., Classification if A is discrete
policy, and f is parameterized as f(s;0), then one can use policy gradient technique to compute the Vol((f) as follows:
Vol((((1))le E E(an, x" (sh)) ETopecto
Vo (In Proof(T))
Σ T' n
(sh))
(16)
H
H elan, 7 (sn))
V. In (a Vof(sn)
(17)
H
E
sn wa mgots an (a
(o (os 0)1o, Q an)]
(18)
where the second step simply uses chain rule, and in the last step simply treat together as a policy that maps from states to actions-hence the Q function defined with respect to n, o ft- Intuitively,
want to optimize fo such that increases the likelihood of generating actions via Te to increase the total "reward" -the long-term prediction error (with respect to π of the policy Tr Using the above gradient perform gradient ascent optimizing fo is similar to classic policy gradient methods and in practice, one can use existing techniques developed for PG here as well (e.g.. natural gradient, actor-critic where the critie is Q*:0/s).
4.3 Algorithm
We summarize the algorithm ARIL in Alg. 1, where every round, update the adversary and the policy x. Given a policy Te, we aim to find the adversary ft from F that increases the long-term prediction error of Ie (w/ expert's prediction) as much possible. Once find such adversary ft. we generate data by executing the composite policy n; o fi on the real system to generate states {s; f==1 d and labels (x*((s; (this procedure requires interactively querying expert, just as DAgger does [32]). We then corrupt the data by applying the latest adversary ft at every data point 8t. and appendix the dataset (fz(s:), = (s:)) the aggregated dataset D. Lastly, train the new policy using the black box supervised learner on the aggregated dataset DD
The data aggregation technique (DAgger [32]) is used to ensure stability in the process of updating and corresponds to the specific no-regret learner FTL as we explained before. However, any other no-regret learner can be used here well, such as Online Gradient Descent [33], with the potential benefit of no need to maintain an ever-growing dataset D.
In theory, optimizing ft is challenging as the objective function could potentially be non-convex here. However, in practice for parameterized adversary ʃ. as we showed the previous section, we can compute the gradient of using classic policy gradient technique, which in turn allows to leverage practically effective techniques such as natural gradient, actor-critic, to optimize St. Note that the optimization procedure for ft does not require any expert data.
5

---

4.4 Analysis
We consider the following setting. First, for optimization in ft, we assume we can achieve near- optimality:
L(ft) 2 max JET
{f) - Erl:
(19)
with 6,1 E Rt. Namely, we do not assume we can find the global optimal adversary at every round. The e here quantifies the performance of the potential RL algorithm in terms of optimizing (ff) with respect to f-
For simplicity, we consider the realizability setting: there exists a policy x E II, such that for any state s E S, Eawx(s)la # (s)] Namely there exists least one policy in policy class that can predict expert's s action at any state s E S. Relaxing the realizability setting to agnostic setting is straightforward here.
Theorem 4.1. As T + co, Alg. I outputs policy ne. E [Te such that:
max
Baws, of FEF
(()) n (s)) Ga Co.
(20)
The above theorem indicates that (1) under the realizability assumption (i.e., policy class II is rich enough to approximate I well), (2) the assumption that the expert is robust with respect to any attack (i.c., Ee is defined as the emror between x (s) and nT (f(s)), which is negligible), and (3) the assumption that we can near-optimally solve the RL-like optimization problem maxser (f( at every round & (i.e., Eyl is small), then Alg. I can achieve adversarial robustness. Please refer to supplementary material for proof.
Experiments
We evaluate proposed adversary and robust imitation algorithm with baseline methods using DM Control Suite [16]. We compare our method with FGSM [18]. Considering FGSM is commonly used in supervised learning setting, we apply FGSM to modify state sh such that it hurts the current one-step prediction of T₁, i.e. (s)) = Sh. + esign V, En (s) (f(a, "aa,,,"( (s))))lnn.
Our algorithm achieves better robustness under both sensory attack and physical attack. The details of the datasets and other experimental settings are described below.
5.1 ARIL with Sensory Attack
In sensory attack, the adversary only modifies the sensory inputs of the agent, but not the underlying physical state, which can be treated as sensory noise in real scenarios.
Environment Setting We investigate the performance of attacks on three continuous control tasks from OpenAl gym robotic benchmarks [34]: Ant-v2, HalfCheetah-v2 and Swimmer-v2. For the convenience of training and testing, we set the horizon H to 1000 in all the benchmark environments.
We only attack the zero-order state the robotics environments, i.e. the positions and angles other than velocities (first-order state). The attack is bounded by the (c is a small positive number), which varies according to the robustness of each environment. The e is chosen to have no more than 40% performance drop with FGSM attack and its values for all experimental setups can be found in supplementary materials. For comparison, all attack methods are bounded to the same e when applying to the same robotics environment.
Training Details Our implementation of ARIL is built on top of rllab [35] and uses Trust Region Policy Optimization (TRPO) [36]. Throughout the experiments, both policy и and adversary fo parameterized with two hidden layers. The output layer of adversary uses tanh as activation function and then multiples to restrict the output in [-e, e]. The standard deviation of J' is initialized to 0.le and the attack to state 8 is sampled from N (Ja(s), 0.1e). We also clip the sampled adversarial noise to [-c, e). Hyperparameters are selected by grid search and the values for all experimental setups can be found in supplementary materials.
Results We first compare our adversary with the baseline Fast Gradient Sign Method (FGSM) [18] and Transferable adversarial perturbations (FGSM'k) [37] In both our method and FGSM, the perturbation added to each state is bounded to a small value such that it's imperceptible as shown in the supplementary material.

---

From Figure I (top), we first observe that FGSM attack in imitation learning works well. By forcing the policy to generate actions that are far away from the expert's, s, FGSM attack is effective and successfully decreases the performance of æ. The performance of FGSM and our attacks in all settings can be found in Table (Adversary). In all tasks, our attack outperforms the FGSM attack. We experimentally demonstrate that through maximizing the total difference to expert policy, the adversarial attack could significantly decrease the total reward. We also show that naively applying adversarial attack techniques developed for supervised learning to sequential decision-makinn agent not effective since the attack is myopic. This result proves the efficiency of our attacks even without ever accessing to reward signals.
Ant
HalfCheetah
Swimmer
1500
4000
100
1250
3000
reward
1000
reward
reward
750
2000
40
1000
250
0
100
200
100
200
100 iterations
300
iterations
iter atrattion
1500
4000
100
3000
reward
1000
reward
reward
60
750
2000
40
500
250
100
iterations
iterations
iterations
No attack
FGSM attack
ARIL attack and defense
Figure I: The first row and second row illustrate the results of sensory attacks and defense on sensory attacks, respective, in Ant, HalfCheetah and Swimmer environment.
We then compare ARIL with another robust imitation learning method, DART [24]. DART learns to inject an optimized level of noise into the expert demonstrations in imitation learning, which forces the expert to demonstrate how to recover from the errors. A noise distribution, P(5 Te. 16), is injected to the demonstration distribution To. (u jr v) where y denotes the sufficient statistics that define the noise distribution. In our experiments, we set Wo. (u (z, x+) Gaussian distribution, .N (N (x), E).
The optimal parameter estimated by the anticipated final robot's error and subsequently scaling the current simulated error to this level, denoted as E p(E Σ. L(u,, Tx. (Ix)). )). For quantitative comparison, train adversary to attack DART using our proposed attacker (ARIL-A), As shown in Table (bottom), our method significantly improves the robustness over DAgger [32] (3rd row and 5th row). ARIL also achieves better robustness compared with DART (4th row and 5th row) in all tasks under the worst possible adversary.
Table 1: Performance of various sensory attack and robust imitation learning methods
Ant-v2
HalfCheetah-v2
Swimmer-v2
Overall (%)
DAgger [15]
1,245
3,276
81
0%
FGSM [18]
945 (-24.1%)
2,336 (-28.7%)
66 (-18.5%)
-23.7%
Adversary
FGSM'k [37]
884 (-28.9%)
1,973 (-39.8%)
57 (-29.6%)
-32.7%
ARIL-A
607 (-51.2%%
830 (-74.7%)
38 (-53.1%)
-59.7%
GAIL [27]
638 (-48.8%%
1,037 (-68.3%)
42(-48.1%)
-55.1%
DR [38]
679(-45.5%)
1.372(-58.1%) 1%)
56(-30.9%)
-44.8%
Robust IL
DART [24]
779 (-37.4%)
1,832 (-44.1%)
49 (-39.5%)
-40.3%
ARIL
965 (-22.5%)
2,974 (-9.2%)
72 (-11.1%)
-14.3%
5.2 ARIL with Physical Attack
In physical attack, the attacker can also affect the underlying transition dynamics. Considering the attacker changes the actual state of in the environment, the transition model can be described as
7

---

8141 f(P(-Is,, az)). Namely, the actual zero-order states are changed under the attack. We can treat physical attack as an external perturbation (e.g. kick on the robot or an unpredicted bump on the road).
Environment Setting Similar to sensory attack, the attack is bounded by the It's worth mentioning that is different for sensory attack and physical attack in the same environment. Since physical attack with the ability to change underlying dynamics is more effective than sensory attack. The e are chosen have more than 40% performance drop with FGSM attack and its values for all experimental setups can be found in supplementary materials.
Ant
HalfCheetah
Swimmer
1500
100
1250
3000
1000
reward
2000
reward
60
750
500
1000
20
100
200
300
400
100
200 iterations
300
400
0
100
200 iterations
300
400
Iterations
1500
4000
100
reward
reward
reward
60
750
2000
500
20
0
200 iterations
100
200 iterations
iterations
No attack
FGSM attack
ARIL attack and defense
The first row and second row illustrate the results of physical attacks and defense on physical attacks, respective, in Ant, HalfCheetah and Swimmer environment.
Figure 2:
Results We first compare our adversary with the FGSM baseline. As might expect, the physical attack is more effective than the sensor attack. Under the physical attack setting, π is more likely to fail. Our experiments support this viewpoint in the sense that for sensory attack needs to be larger than the one for the physical attack to achieve similar attack performance. Again, in all tasks, our attack outperforms the FGSM attack. We also compare ARIL with the DART (Table 2). This result proves the efficiency our robust imitation algorithm even the underlying transition dynamics are changed.
Table 2: Performance of various physical attack and robust imitation learning methods
Ant-v2
HalfCheetab-v2
Swimmer-v2
Overall (%)
DAgger [15]
1,245
3,276
81
0%
Adversary
FGSM [18]
885 (-28.9%)
2,100 (-35.9%)
61 (-24.7%)
-29,8%
ARIL-A
158 (-87.3%)
249 (-92.5%)
16 (-80.2%)
-86.7%
DART [24]
581 (-53.3%)
1,078 (-67.1 16))
27 (-66.7%)
-62,4%
Robust IL
ARIL
903 (-27.5%)
2,183 (-33.4%)
56 (-30.1%)
-30.3%
Conclusion
In conclusion, we propose an adversarially robust imitation learning algorithm. We theoretically prove that under realizability assumption, robust expert assumption, and optimal adversary assumption, ARIL achieves adversarial robustness. We evaluate our method on set of continuous control tasks from OpenAl Gym and show that our method achieve better robustness under both sensory attack and physical attack compare with baselines. We hope that our works can inspire more studies in robust imitation learning methods when the quantitative reward from the environment is unavailable.

---

Acknowledgements. The authors would like to thank Wen Sun, Yan Xu, Yunze Man for fruitful discussion and detailed feedback. This work was supported by the PanGU Young Investigator Award.
References
[1] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, Quan, Sendonaris, Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[2] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. A. Theodorou, and B. Boots. Imitation learning for agile autonomous driving. The International Journal of Robotics Research, 39(2-3)286-302, 2020.
[3] B. Argall, S. Chernova, M. Veloso, and B. Browning. survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-45 483, 2009.
[4] B. Ibarz, J. Leike, T. Pohlen, G. Irving. S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations atari. Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 8022-8034, 2018.
[5] Merel, Y. Tassa, D. TB, Srinivasan, Lemmon, Wang. G. Wayne, Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint 10t.vv:177702200.111 2017.
[6] C. Szegedy, W. Zaremba, Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. International Conference on Learning Representations 2014. URL http://arxis org/abs/ htrg/abs/331... 1312.61999
S. Huang, N. Papemot, I. Goodfellow, Duan, and P. Abbcel. Adversarial attacks on neural network policies. arXiv preprint
2017.
[8] V. Behzadan and A. Munir. Vulnerability of deep reinforcement learning policy induction attacks. In International Conference on Machine Learning and Data Mining Puttern Recognition, pages 262-275 Springer, 2017.
[9] Y.-C. Lin, Z.-W. Hong, Y.-H. Liao, M.-L. Shih, M.-Y. Y. Liu, and M. Sun. Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint 1exXx:1173..00 06748, 2017.
[10] A. Pattanaik, Z. Tang, Liu, G. Bommannan, and G. Chowdhary. Robust deep reinforcement learning with adversarial attacks. In Proceedings of 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2040 2000022022220 2018.
[11] J. Tobin, Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abheel. Domain randomization for transferring deep neural networks from simulation the real world. 2017 IEEE/RSI international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[12] A. Madry, A. Makelov, Schmidt, D. Tsipras, A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[13] H. Zhang, Y. Yu, Jiao, Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pages 7472 7482. PMLR, 2019.
[14] L. Pinto, Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817-2826. PMLR, 2017.
[15] S. Ross. G. Gordon, and D. Bagnell. reduction of imitation learning and structured prediction no-regret online learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635, 2011.
[16] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et Deepmind control suite. 2018.
[17] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties neural networks, arXiv preprint arXiv:1312.6199 2013.
[18] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint www.::i:::426655. 2014.
[19] A. Kurakin, 1. Goodfellow, and S. Bengio. Adversarial machine learning at scale. arXiv preprint arXiv.1611.0126.. 2016.

---

[20] F. Tramer. A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint urXiv:1705.07204 2017.
[21] W. Sun, D. Dey, and A. Kapoor. Safety-aware algorithms for adversarial contextual bandit. In Proceedings of the 34th International Conference on Machine Learning Voluue 70, pages 2280-3288. JMLR. org. 2017.
[22] Y. Han, D. Hubczenko, Montague, O. De Vel. T. Abraham, B. I. Rubinstein, C. Leckie, T. Alpean, and S. Erfani. Adversarial reinforcement learning under partial observability in software-define networking. arXiv preprint arXiv:1902.00062 2019.
[23] A. Hussein, Gaber, Elyan, Jayne. Imitation learning: A survey of learning methods, ACM Computing Surveys (0Q):
5002):1-35. 2017.
[24] Laskey, J. Lee, Fox. A. Dragan, and K. Goldberg. Dart: Noise injection for robust imitation learning. arXiv preprint (r..iv:703.093330 2017.
[25] Z. Wang, Merel, E. Reed, N. Freitas, G. Wayne, and Heess. Robust imitation of diverse behaviors. In Advances in Newral Information Processing Systems, pages 5320-5329, 2017.
[26] D. Kingma and M. Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[27] J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pages 4565-4573, 2016.
[28] V. Tangkaratt, B. Han, M. E. Khan, and Sugiyama. Vild: Variational imitation learning with diverse- quality demonstrations. arXiv preprint arXiv: wwt.:/0099999 06769. 2019.
[29] D. Brown, R. Coleman, R. Srinivasan, and S. Niekum. Safe imitation learning via fast bayesian reward inference from preferences. International Conference on Machine Learning, pages 1165-1177. PMI.,, 2020.
[30] Shin and Kim. Adversarial imitation learning via random search. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2019.
[31] T. Yar. Hinf optimal control and related minimax design problems. Birkhäuser Boston.. 2008
[32] S. Ross, G. J. Gordon, and J. A. Bagnell. No-regret reductions for imitation learning and structured prediction. In A/STATS. Citeseer, 2011.
[33] M. Zinkevich. Online Convex Programming and Generalized Infinitesimal Gradient Ascent. In ICML, 2003.
[34] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
[35] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference machine learning, pages 1329-1338. PMLR, 2016.
[36] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference machine learning. pages 1889-1897, 2015.
[37] W. Zhou, X. Hou, Y. Chen, M. Tang, X. Huang, X. Gan, and Y. Yang. Transferable adversarial perturbations.
In Proceedings of the European Conference Computer Vision (ECCV), pages 452-467, 2018.
[38] D. Seita, A. Ganapathi, R. Hoque, M. Hwang, E. Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali, et al. Deep imitation learning of sequential fabric smoothing from an algorithmic supervisor. In 2020 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 9651-9658. IEEE, 2020.
10

---

A Proof of Theorem 4.1
Proof. Since Alg. 1 runs no-regret online learner update on the sequence of loss functions {4(()), we must have:
Σ
((())
min Tell
Σ
(21)
Add and subtract Ei=o (eTT)) on the left hand side of the above inequality, we have:
T
T
T Σ (((T)
T
((T))
= + min ==
Σ (((T))
(22)
sell
t=
Since operate under the realizability setting, the term inside the parenthesis the RHS of the above inequality is guaranteed to less than or equal to zero. Hence, the above inequality simplifies to:
T Σ (e(T)) Σ (((T)) +o(T)
(23)
t=
1=0
For (e(T)), using the definition of l,, we see:
(TTT Bf [En-o( 7 (s))]] Egndx (f((s) x (
(24)
where we use the expert robustness definition above (i.c., n" (f(s)) (s) for all s and f). Hence, we have:
T Σ (((Te) S Tee + o(T).
(25)
Now we need to lower bound (T). Using the definition of le again, we have:
max [(Tv, f) S [(m: fv) + Gel S E fi) + 41 Σ (((T)) + Gd.
f(m
(26)
where the first inequality comes from the assumption that the RL solver returns 6r near-optimal solution.
Combine all the results above together, we get:
max L(T,, ert + ea + o(T)/T.
(27)
Hence as T' approaches oC, we have that the long-term prediction loss of the learned policy T.+ under the worst possible adversarial attack from F is upper bounded by 6,1 6r1-the error introduced from optimizing ft-
Under agnostic setting, is not guaranteed that there exists a = (s), where a ~ (s) and π E II for Vs S. However, it can be assumed that the error between and x (s) is bounded by small number ea, namely Ea (a o (x)) S ca. Hence Equation 4 is modified as:
(((T)) = Egndroo [ (1L))( (( n5 (x))]] < ca-
(28)
Plugging this to Equation 3, we have:
T (mmms + tEa) +o0))
(29)
Combining the results with Equation 6, we get max L(Tr f) S ent ce + oT)/T.
(30)
Similarly to the realizability setting, as T approaches cc, the long-term prediction loss of the learned policy ni under the worst possible adversarial attack from F is still upper bounded.
11

---

A.1 Training Details
Both attacker and student are trained using Adam optimizer at learning rate of 0.001.
At the start of ARIL algorithm, we initialized the buffer for DAgger algorithm with size 5M timesteps. In each attack stage, it takes 1e6 timesteps for the attacker to collect the trajectories each time. Then at each defense stage, we collected 40 trajectories of student under attack and labeled them with expert actions into the DAgger buffer.
Table Hyperparameters for training attacker
entropy loss coefficient
0.01
value loss coefficient
0.5
clip range
0.2
12


Reviews:
[{"review_id": "zeRhMn5SE1m", "reviewer": "Reviewer_bbzz", "summary": "The paper presents a method for imitation learning that is robust to both sensor noise and physical disturbances, via training adversarial policies with RL. The method is demonstrated on various standard RL benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well-written and easy to follow\n- The method seems reasonable and potentially useful.\n- There are some theoretical results confirming robustness under reasonable assumptions.\n- The experimental results show benefits over simple methods such as FGSM.\n\nWeaknesses\n- This is a minor point, but the authors state \"The sensory attack is similar to attacks considered in today’s supervised learning setting, where the physical attack is similar to the robust control setting (e.g., a min-max formulation of H∞ optimal control [31])\"\nIn fact, in H∞ control it is standard to consider worst-case (adversarial) disturbances to both sensors and the true physical state. Actually, the problem posed in the paper is essentially a nonlinear version of standard problems in H∞ control.\n- The idea of imitation learning is to approximate the optimal policy, but Sec 4 the authors propose comparing alternative policies via a simple loss such as zero-one or l2. But policies should be compared based on their ultimate effect on the system, e.g. via a Q function approximation. Have the authors considered this?\n- The authors compare mainly agains \"vanilla\" FGSM. But this is a very simplistic algorithm and there are many variations that could perform better, e.g. iterative gradient versions.\n- There is some recent work in adversarial RL, e.g. by Russo and Proutiere, which seems similar in spirit. Have the authors compared to this? (Not exactly for imitation learning, but there are similarities)\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I thought the paper was quite interesting and presented a potentially useful method. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I'm not an expert in this area, but it seems quite similar in approach to several other papers.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "jnu2yLWzaRS", "reviewer": "Reviewer_uSZo", "summary": "This paper proposed an adversarial robust imitation learning algorithm to improve imitation robustness. Sensory and physical attack are defined and considered in this paper. The authors theoretically proved that the proposed ARIL algorithm can achieve adversarial robustness. Four experimental benchmark results showed the efficiency of this method compared with other methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\nTheory and implementation of this ARIL method are reasonable.\nWeakness.\n1.\tThere are only 4 benchmark results from DM control suite. I strongly suggest add more experimental results especially those hard tasks like humanoid-v2, which are more sensitive to uncertainty.\n2.\tOther imitation learning algorithms should be considered as comparison such as GAIL like methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "As there is no practical tasks such as self-driving to verify the proposed algorithm, it is hard to assess the theory results in this paper, which are based on strong assumptions.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "More experimental results should be added, such as hose hard tasks like humanoid-v2 or self-driving tasks.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "Y5HWoaCql2", "reviewer": "Reviewer_1bf5", "summary": "The paper proposes an adversarially robust imitation learning (ARIL) method which learns an MDP policy that is robust against state perturbation. This is achieved by iteratively updating a policy to minimize the expected cost and an adversary to maximizes the expected cost under the data aggregation (Dagger) framework. The paper proofs that under certain assumptions, the worst-case performance of the policy against the best adversarial is bounded. ARIL is empirically evaluated against an adversarial attack method (FGSM) and an adversarial defense method (DART) on continuous-control tasks. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n- The paper proposes a promising method for robust imitation learning under state perturbation. The results of this paper should be useful for robust imitation-learning research in the future.  \n- The paper proposes both attack and defense mechanisms and evaluates both of them in the experiments. However, this point is expanded only in the experiments. I suggest to include this point into the discussions in early sections.  \n- The method is supported by a theoretical result showing that the performance of learned policy can be bounded. Still, the result assumes the zero-one loss which is not used in practice. I suggest expanding the results to consider surrogate losses instead of the zero-one loss. \n\n### Weakness\n- There are too few baseline methods in the experiments. The paper should include more methods, especially defensive methods, in the experiments. \n- The paper does not evaluate the method in a physical robot with real-world state perturbation. While this is not a major issue, the paper would be much more convincing with a result on a physical robot. \n- There are mathematical inconsistency and ambiguity, possibly due to typos. The details of these are commented in the Issues section below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes an interesting and promising approach to make imitation learning more applicable to real-world scenarios. While the technical part of the paper is not very new (i.e., the method still follows standard min-max optimization in robust learning), it is a new idea to apply the technique to imitation learning. The derived performance-bound is also a strong addition to the contribution. The major issue of the paper to me is the experiment part, which demonstrates the method only in simulated robots but not in physical robots. The mathematical clarity is also a minor issue, but this can be straightforwardly addressed. Overall, I rate to weak accept the paper. \n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Besides the issues in experiments commented above, the following are mathematical inconsistency and ambiguity that should be addressed in the revision. \n- Line 36 defines the optimization problem $\\min_{\\pi} \\max_{f} J(\\pi, f)$ with the expected reward $J$. This has to be a typo since the policy should be maximizing the expected reward instead of minimizing it. \n- The paper formulates the problem with an MDP with the reward $r$, but later formulates the policy learning problem using the cost $c$. While it can be understood that $c(s,a) = -r(s,a)$, this inconsistency should be addressed for unfamiliar readers. \n- Equation 7 contains the difference $Q^e(s,a) - Q^e(s,\\pi^e(s))$, but the first Q-value should be $Q^\\pi(s,a)$ because it comes from the expected return of the learner $J(\\pi, f)$. In addition, the Q-value and the constant $Q_{\\mathrm{max}}$ is not defined anywhere. While these quantities can be often understood for the context, they should be formally defined in the paper. \n- Equation 13 should have $\\ell_i(\\pi)$ instead of $\\ell_t(\\pi)$ otherwise the summation operator should be removed. \n- Equations 16-17 contains the joint distribution of state-action: $\\pi_t(a_h,s)$. I think it should be the conditional distribution $\\pi_t(a_h|s)$ instead. Otherwise, the gradient is not computable since the joint is unknown and you require the unknown state marginal to compute it using the policy. \n- The error $\\epsilon_e$ in the theorem is not defined anywhere, even though it is claimed to be small when the expert is robust. It is also not defined in the appendix. In fact, $\\epsilon_e$ suddenly appears in equation 5 of the appendix without explanation.\n"}}, {"review_id": "FTEwlTRE7WH", "reviewer": "Reviewer_Bpno", "summary": "This paper considers training an imitation learning agent to be robust to adversarial changes in the observations or actual state. Using a DAgger style algorithm the authors train an agent to follow expert actions which are assumed to be minimally affected by the adversary's changes. The algorithm is shown to work well in terms of robustness and some theory motivating the approach is presented.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n+novel and interesting problem\n+theory justfying method\n+nice empirical results\n\nweaknesses:\n-no robotics experiments\n-no human feedback\n-DAgger is not very practical since it requires human labels for every state.\n-no empirical comparison with domain randomization as a means to robustify an imitation learning algorithm", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper provides a nice proof of concept for generating and defending against adversarial attacks on an imitation learning agent. I think with improvements to the writing and additional experiments comparing with domain randomization this could be a strong paper and I am willing to improve my recommendation is these things are addressed. Another weakness is the lack of any real robotics experiments or human studies.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "In section 3.2 the authors assume the expert is not affected by adversarial examples, but later in Theorem 4.1 the authors include new notation \\eps_e that is undefined and seems to denote that the expert is affected. Please clarify and define assumptions better.\n\nDo you consider this an algorithm that should be used with a human supervisor or an algorithmic supervisor? If human, see below for references on how DAgger is burdensome for a human and how to alleviate this. If algorithmic, how does this method compare to domain randomization applied to imitation learning? There the idea is similar in that you apply different noise and rendering styles to the input images but force the learner to always take the same action regardless of the perturbation. See for example, Seita et al. \"Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor\" IROS 2020. It would be good to compare your results with some form of domain randomization approach to show that optimizing for adversarial noise is advantageous. \n\nClarity / typos\nLine 26 -> \"damage to the vehicle in the worst case\"\nLine 106 -> \"Again, denote ...\"\nLine 130: what is this notation? Is it supposed to be an indicator? maybe use mathbf{1}?\nLine 135: Q is never defined. I assume it is the state-action value, but this should be clarified in the text. What is Qmax? Can this be arbitrarily large? Is this bound vacuous? Is there an equivalent bound for a non-zero one loss?\nLine 149: Is there a reference for Follow the Leader algorithm? I'm not familiar with it.\nLine 151: 1/t isn't equal weight since t is increasing, right?\nLine 6 in Alg 1 doesn't seem necessary given the assumption that the expert isn't affected by the corruption. This assumption or the relaxatino of it should be clarified\nLine 162 and elsewhere: notation is confusing, sometimes uses a and sometimes uses a_h. In eq (15) a_h is undefined. Also s in (16) is not defined should it have a subscript? what is \\rho here. What is ln (\\rho)? what is Q is it wrt the true cost or the prediction error loss?\nLine 196: If the agnostic setting is straightforward then why not do it? At least in the appendix? What is the benefit of the current approach compared to the agnositic setting. It would be good to show the most useful/interesting case in the paper\nLine 239: Reword since it seems that FGSM *is* effective, just not as much as the proposed approach. \n\nEquation (2) is identical to (1) except for a single tick mark. Consider coloring the change in a different color to highlight this difference.\n\nEquation (5): what about physical attacks seems like it should have something in the transitions denoting the changes in states in this case.\n\nThe results in figures 1 and 2 are nice. Please clarify in the text and figure captions what the top and bottom rows of figures are\n\nReferences:\nBrown et al. \"Bayesian Robust Optimization for Imitation Learning\" NeurIPS 2020.\n-also considers robust imitation learning but from a different angle.\n\nZhang et al. \"Query-Efficient Imitation Learning for End-to-End Autonomous Driving\" and Hoque et al. \"LazyDAgger: Reducing Context Switching in Interactive Imitation Learning\"\n-both consider making DAgger more efficient by not requiring human labels at every state. Could something similar be done with this work?\n"}}]

Author rebuttal:

Meta-review: The authors present a system in which policies are trained to be robust to adversarial changes in the observation / state space. Reviewers consistently praised the novelty of the problem formulation as applied to robotic imitation learning (even if such techniques are common in other ML domains), as well as the presentation of theory justifying the proposed method. The reviewers addressed the bulk of reviewer concerns, adding additional (relevant) baselines and overall increasing the quality and clarity of the submission. Multiple reviewers pointed out lack of real-world experiments as the largest weakness of the paper (even after revisions), however two reviewers pointed out (and I agree) that the paper would still be of interest to the CoRL community despite this.
Decision: Accept (Poster)