Preventing Imitation Learning with Adversarial Policy Ensembles
Authors: Albert Zhan
Pieter Abbeel
Stas Tiomkin
Venue:  (2020)
Keywords: Imitation Learning
Reinforcement Learning
Representation Learning

Abstract: Imitation learning can reproduce policies by observing experts, which poses a problem regarding policy propriety. Policies, such as human, or policies on deployed robots, can all be cloned without consent from the owners. How can we protect our proprietary policies from cloning by an external observer? To answer this question we introduce a new reinforcement learning framework, where we train an ensemble of optimal policies, whose demonstrations are guaranteed to be useless for an external observer. We formulate this idea by a constrained optimization problem, where the objective is to improve proprietary policies, and at the same time deteriorate the virtual policy of an eventual external observer. We design a tractable algorithm to solve this new optimization problem by modifying the standard policy gradient algorithm. It appears such problem formulation admits plausible interpretations of confidentiality, adversarial behaviour, which enables a broader perspective of this work. We demonstrate explicitly the existence of such 'non-clonable' ensembles, providing a solution to the above optimization problem, which is calculated by our modified policy gradient algorithm. To our knowledge, this is the first work regarding the protection and privacy of policies in Reinforcement Learning.

Under review as a conference paper at ICLR 2020
PREVENTING IMITATION LEARNING WITH ADVERSARIAL POLICY ENSEMBLES
Anonymous authors Paper under double-blind review
ABSTRACT
Imitation learning can reproduce policies by observing experts, which poses a problem regarding policy privacy. Policies, such as human, or policies on deployed robots, can all be cloned without consent from the owners. How can we protect our proprietary policies from cloning by external observer? To answer this question we introduce a new reinforcement learning framework, where we train ensemble of near-optimal policies, whose demonstrations are guaranteed be useless for an external observer. We formulate this idea by a constrained optimization problem, where the objective is to improve proprietary policies, and at the same time deteriorate the virtual policy of an eventual external observer. We design a tractable algorithm solve this new optimization problem by modifying the standard policy gradient algorithm. Our formulation can be interpreted in lenses of confidentiality and adversarial behaviour, which enables broader perspective of this work. We demonstrate the existence of "non-clonable ensembles, providing a solution the above optimization problem, which is calculated by our modified policy gradient algorithm. To our knowledge, this is the first work regarding the protection of policies in Reinforcement Learning.
INTRODUCTION
Imitation learning and behavioral cloning provide really strong ability to create powerful policies, as
seen in robotic tasks (Laskey al. 2017 Finn et al.) 2017: Codevilla al. 2019 2017: Pomerleaul
1988 Bojarski et al. 2016). Other fields have developed methods to ensure privacy (Al-Rubaie
& Chang. 2019 cloning.
Papernot et al. 2016), however, such work do not offer protection against policy
In this work, we tackle the issue of protecting policies by training policies that aim prevent an external observer from using behaviour cloning. Our approach draws inspiration from imitating human experts, who can near-optimally accomplish given tasks. The setting which analyze is presented in Figure [ We wish to find collection of experts, which as an ensemble can perform a given task well. however, also targets behaviour cloning through adversarial behaviour. Another interpretation is that this collection of experts represents the worst case scenario for behaviour cloning on how to perform a task "good enough".
Imitation learning frameworks generally make certain assumptions of the optimality of the demon- strations (Ziebart et al., 20088: Levine (2018), yet never considered the scenario when the experts specifically attempt to be adversarial to the imitator. We pose the novel question regarding this assumption: does there exist a set of experts that are adversarial to an external observer trying to behaviour clone?
We propose Adversarial Policy Ensembles (APE), a method that simultaneously optimizes the performance of the ensemble and minimizes the performance of policies eventually obtained from cloning it. Our experiments show that APE do not suffer much performance loss from an optimal policy, while causing, average, the cloned policy experience over 5 times degradation compared to the optimal policy.
Our main contributions be summarized follows:

---

Under review as a conference paper at ICLR 2020
Training
Cloning
Pollou tngectory with werpllnng renn cortect
Collatee Dusiset 12
Clore a
pllty
(0s,4a, by Pu (t.)
Exuction edamale modieed grl
Gampl diferent
(1a, (1))
Clon
Otserves Polic
(Us. (r))
Roboutt Faing the canteet over the tuqectar
vnjedbree br deserceet
contodt
(1rs
(1p Fy Rp....))p
Updee coldes
Palicy Eraentiee
Pidacy
Figure I: Confidentiality scheme: Left During training, optimize a Policy Ensemble by estimating gradients using both the policies in the ensemble and the fictitious observer policy. Right When collecting a dataset for cloning, the context variable is marginalized out. Thus cloning the Policy Ensemble can result in useless policy
We introduce a novel method APE, as well as the mathematical justification of the notion of adversarial experts.
By modifying Policy Gradient (Sutton et al. 2000), common reinforcement learning algorithm, we suggest a tractable scheme for finding an optimal solution for this objective.
We demonstrate the solution by numerical simulations, where we show that a cloned policy is crippled even after collecting significantly large number of samples from policy ensemble.
To our knowledge, not only is this the first work regarding the protection of policies in reinforcement learning, but it is also the first to represent adversarial experts.
2 PRELIMINARIES
We develop APE in the standard framework of Reinforcement Learning (RL). The main components we use are Markov Decision Processes, Policy Gradient (Sutton et al. 2000), policy ensembles, and behaviour cloning, which we review below.
2.1 MARKOV DECISION PROCESS
A discrete-time finite-horizon discounted Markov decision process (MDP) M is defined by (S,A,r, r.p, p, Po. 7, T) where S is the state space. A is the action space. r S A - is the reward function, p(s++1 St, at) is the transition probability distribution,
: S + R+ is the ini- tial state distribution, 7 (0, 1) is the discount factor, and T' the time horizon. A trajectory Pr. sampled from and a policy π S x A - Rt, is defined to be the states and actions tuple (sp, co -1, 11, whose distribution is characterized by Pr. Define the return of a trajectory to be r(r) = -1-11""(88 0z) to be the sum of discounted rewards seen along the trajectory, and define value function / S R be expected return of trajectory starting from state under the policy π. The goal of reinforcement learning is to find a policy that maximizes the expected return E,-p. [r(r)].
2.2 POLICY GRADIENT
Policy Gradient (PG) Sutton et al. 2000) aim to directly learn the optimal policy π, parameterized by 8, by repeatedly estimating the gradient of the expected return, in one of many forms, shown in Schulman et al. 2015). In our work, follow notation similar that of (Schulman et al. 2015 2017) and estimate V VolErrpa [r(r]] using the advantage, which is estimated from a trajectory т, (t Rt(t) (st),, where Rt) Ef= xe ''r(ss, ar) is the sum of the reward following action dz-
Here, the value function is learned simultaneously with the the policy, and the advantage will use The as estimate for V*
2

---

Under review as a conference paper at ICLR 2020
2.3 POLICY ENSEMBLE (PE)
A Policy Ensemble (PE) is similar to the notion of contexts and skills (Achiam et al. 2018 Eysenbach
al. 2018 Sharma al. 2019) which we discuss in Section 3 We denote a PE by me. where each Te0 E 2,,2,-. ... represents an expert. To rollout the PE, an expert is chosen at random (in our case uniform), and the expert completes a trajectory. Each expert policy Toto (als) can be viewed as a policy conditioned on a latent variable c, x (als, c)..
Although Fe consists of multiple policies, is important to note that it itself is still a policy.
2.4 BEHAVIOUR CLONING
To behaviour clone expert policy (Widrow & W. Smith 1964), dataset of trajectories DD consisting of state action pairs (s, a) are collected from the the expert rollouts. Then, a policy parametized by ø is trained to maximize the likelihood of action given a state. ((.))eD log (a ().
When cloning Te. D will not contain information of the latent variable and the cloned policy will marginalize it out. Thus, the observer will clone:
To(a 8) = El(el) STTTC (a
We stress that this policy does not exist until Te is behaviour cloned. wa is fictitious policy to represent what would happen in the best case scenario of the observer having access to infinite data from Te to clone into To.
The scope of this paper is to specifically prevent behavioral cloning from succeeding, Other imitation learning approaches such as inverse reinforcement learning (Abbeel & Ng 2004 Ng & Russell 2000
Levine et al., 2011) and adversarial imitation learning (Ho & Ermon 2016 Peng et al. [0188) 8) require rollouts of non-expert policies in the environment, which may be costly, and thus not considered.
RELATED WORK
Adversarial Attacks in RL: Our notion of adversarial policies is inextricably related to other adversarial methods that target RL. such as Lin et al. (2017); [Behzadan & Munir (2017), that add adversarial perturbations to policy input. Other adversarial attacks include poisoning the batch of data used when training RL (Ma et al.) 2019), and exploitation in the multi-agent setting (Glcave eet al. (219)). However, these methods all present active attacks for various learning techniques. Our method, instead, passively protects against cloning.
Privacy in RL: With regards to protection, our work is related differential privacy (Al-Rubaie & Chang 2019). Differential privacy in RL can be used to create private Q-functions (Wang & Hegde 2019) private policies (Balle et al.) 2016), which have private reward functions private policy evaluation. However, we would like to emphasize that our motivation is to prevent cloning, and thus protecting the policies, rather than protecting a dataset. In fact, we make the assumption that the observer can perform behaviour cloning on as much data as desired.
Since we comply the standard imitation learning setting of cloning from dataset with many experts providing the demonstrations, latent variables w.r.t. imitation learning is well-studied. For example, Codevilla et al. (017)) show that conditioning on context representation can make imitation learning a viable option for autonomous driving. Li et al. (2017) demonstrate that the latent contextual information in expert trajectories is often semantically meaningful. As well, Providing extra context variables to condition on also appears in forms of extra queries providing labels (Brown et al.) 2019: de Haan et al. 2019 Hristov et al., 2018). Our method different as instead of experimenting for success in imitation learning, we study how to prevent it.
Imitation Learning:
Multiple Policies: (Achiam et al., 2018 Eysenbach et al. 2018 Sharma et al. 2019) have similar schemes of sampling latent variable and fixing it throughout trajectory, although their latent variables (contexts or skills) are used to solve semantically different tasks. The reason to solve different tasks is due to the objective of using the context variable/skills for learning in an unsupervised setting.
3

---

Under review as a conference paper at ICLR 2020
Our approach differs in both motivation and implementation, same task, and constrain so that observers can not clone the policy.
learn experts that all solve the
A PE can also be viewed assigns probability to the same expert for an entire trajectory. As such, we do not learn the gating network, although it may still be useful to Te special case of mixture of experts where the gating network learns immediately to fix the expert for each trajectory. There are also methods such as OptionGAN (Henderson al. 2018), which uses a mixture of experts model to learn multiple policies as options with access to only expert states.
mixture of experts (Jacobs et al. 1991), except the gating network
Zhang al. (2019) also proposes method train multiple policies that complete the same task but uses the uncertainty of an autoencoder as reward augment. Their motivation is to find multiple novel policies, while our motivation has no connection novelty. Due to these differences in motivation, they train each policy one after the other, while our policies are trained simultaneously.
Policy ensembles are also used in the multi-task and goal conditioned settings in which case the task that is meant to be solved can be viewed as the context. Marginalizing out the context variable (Equation[ [ of these context-conditionn regularizing term for learning new tasks (Goyal et al. 2019) and for sharing/hiding goals (Strouse al. 2018). However, the main motivation is different that both Goyal et al. (2019); [Strouse et al. (2018) use To to optimize mutual information, while we directly optimize its performance.
policies studied in the case of introducing KL divergence
METHOD
4.1 OBJECTIVE
We wish to have experts that can perform the task, while minimizing the possible returns of the cloned policy, denoted in Equation We modify the standard RL objective to be:
arg min Erop., [r(T)] s.t. By-p., (r(7))
where is parameter that lower bounds the reward of the policy ensemble. This translates to maximizing the unconstrained Lagrangian:
J(0) ETrp,, EI -3T-pe. [r(r)]
(3)
where 1/3 is the corresponding Lagrangian multiplier, and subsumed into the returns collected by the policy ensemble. We refer to PE that optimizes this objective as Adversarial Policy Ensembles (APE). There is a natural interpretation "good enough", which is reflected in the constraint. The minimization is simply finding the most adversarial experts.
the objective in Equation 2 Human experts tend to be
Although assume that the observer can only map states actions, it may be the that they can train a sequential policy, which is dependent on its previous states and actions. Our method can be generalized to sequential policies well, and the impact of such observers is discussed in the Section 6
4.2 MODIFIED POLICY GRADIENT ALGORITHM
Intuitively, since there are the returns of two policies that are being optimized, both should be sampled from to estimate the returns.
We show how we can modify PG to train APE, by maximizing Equation 3 The two terms suggest a simple scheme to estimate the returns of the policy ensemble twice: once using Te that we wish to maximize, and second time using w., which approximates the returns of an eventual observer who tries to clone the policy ensemble. Along with our PE, we train value functions V==.09 for each expert, jointly parameterized by ó which estimates V7 BV The loss function for the value functions of two sampled trajectories T1, 72 is
4

---

Under review as a conference paper at ICLR 2020
T1-1 Jn In
T2
1 (st,)
Rr, (t))
V
v
(8tz) + 3R (t)
(4)
2
1=0
t=0
The policy gradient update from N₁ and N2 trajectories is then
VoJn..1 G1 G2
(5)
where
1 4
G1
V log T T (0 (1)
| " (t)
(6)
j=
N T2
G2 = 2
Σ
V log To (a
122 )
(7)
where e identifies the chosen expert of the trajectory,, and A (t) R,,, (t) V0 (st) and Ago (t) -3,,, (t) (8,) are the modified advantage functions. The 3 that is in the advantage in G2 optimizes against the performance of the observed policy To.
The gradient Gi for we is straightforward. However, to estimate the gradient G2 for no which is an fictitious policy, we sample from it by first re-sampling the context of the expert at each state, and then sampling an action from the context. The back-pred proppaaapattiion occurs to Te( (a s) for the context sampled at each state. Practical implementation details can be found in A.2 The intuition is as follow. While sampling To. if selected action causes high return, we should decrease the probability, which lowers the expected reward of n. Combined, the two gradients will cause the PE to select actions that achieves have high reward, and are detrimental to the observer.
Equations 4 and formulate our PG approach of APE, which is summarized in Algorithm
Algorithm 1: PG-APE
Require: 6, Ø, M,
1: for each iteration do:
2: Generate trajectories T1 with Te from M for Equation
3: Generate trajectories 72 with T. from M for Equation 7
4: Calculate Equation 5 to perform gradient update the PE 0 0 0z VoJ (0)
5: Update the value function á + Φb as V To (Ø) as determined by Equation
6: end for
5 EXPERIMENTS
We perform experiments on a navigation task, where the objective is to reach a goal state as fast as possible. The purpose is illustrate that an APE can cause the cloned policy to take significantly longer to reach the goal state. We do so by first training PE and behaviour cloning it. We then compare the performance of the PE that of the clone. We use a discrete environment best demonstrate the validity of the equation. This is because all discrete policies can be parameterized, which is not true in continuous, where typically Gaussian parameterization is used. As such, continuous environments would have to make assumptions about how both the PE and the cloner parameterizes policies, as well as tackle problems of distributional drift, which we would like to avoid. However, with these assumptions, experiments, we use a 10 x 10 grid-world environment as our main testbed. This to have large enough expression that would not be found in smaller grids, while still small enough visualize the behaviour of the APE. The discrete actions will show precisely how the experts can be jointly adversarial.
setting extend to the continuous domain. In our
Using gridworld allows for precise expected return estimates. an environment where closed-fome returns cannot be calculated, approximation emor accumulate through estimating the returns of
5

---

Under review as a conference paper at ICLR 2020
-16
-33
-50
-66
-83
-100
Figure 2: Visualization of APE. We set 3 = 0.6. Arrows indicate action probabilities, and the colour scale represents the hitting time. Yellow indicates expected reward of while purple indicates expected reward of 1000, which is the maximum episode length. The top left corner is the goal state, and the adjacent states that are purple are an example of how APE is adversarial to cloning, as those states will cause the cloned policy to suffer larger losses.
both the trained PE and the clone. This noise would only increase in continuous state space, where the returns of To may not be tractable to estimate due to issues such as distributional drift (Ross et al. 2010 Codevilla et al., 2019 de Haan et al. 2019)..
Our results answer the following questions. How much optimality is compromised? How useless can we make the cloned policy? it possible to use non APE to prevent behaviour cloning?
5.1 TRAINING
Even though our method can compute a policy ensemble with any finite number of experts, chose to visualize a solution with 2 experts, which sufficient to reveal the essential properties of the method. Specifically, we train n 2 tabular experts with PG-APE. Our code written in Tensorflow (Abadi et al. (016), and will be publicly available on GitHub. Training details and hyper- parameters are Section A. of the Appendix.
5.2 ENVIRONMENT
The basic environment is 10 10 grid, with the goal state at the top left corner. The agent spawns random non- non-goal state, and incurs a reward of -1 for each time-step until it reaches the goal. At the goal state, the agent longer receives loss and terminates the episode. The agent is allowed five actions, AA = Up, Down, Left, Right, Stay }. Moving into the wall equivalent to executing a Stay action. We choose this reward function for the benefit of having clear representation of the notion of "good enough", which is reflected in how long it takes to reach the goal state. Having such representation exemplifies how the APE can prevent an observer from cloning good policy.
5.3
VISUALIZATION
Figure shows example of a PE that is trained for the basic gridworld environment. Figure shows the corresponding cloned policy, as well as a comparison to an optimal policy. The colour scale represents the expected return of starting at given state.
In the case of an optimal policy (3 = 0), actions are taken to take the agent to the goal state as fast as possible. However, when 8 > O, such solution is no longer the optimum. Similar to 0, the experts would like to maximize the expected reward, and reach the goal state. However, to minimize the reward of the observed policy, the two expert policies must jointly learn increase the number

---

Under review as a conference paper at ICLR 2020
of steps needed for To to reach the goal state. The expert policies must use adversarial behaviour while reaching the goal state, such as taking intelligent detours or Stay in the same state, which are learned to hinder T, as much possible. These learnt behaviours cause the cloned policy to take a drastically longer time to reach the goal. For example, note the two purple squares at the top-left the goal, which indicates that the experts understand that they should not move to prevent the observer from attaining reward. Even though these sub-optimal decisions are made, on expectation, the experts are "not bad" and achieve an average of -15.27 reward.
5.4 BASELINES
We use behaviour cloning clone PG- APE trained policies. To support our claims preventing even the horizon of infinite data, we collect million timesteps of the trained PE the environment. Further details of be- haviour cloning are in the appendix. Shown in Figure is an optimal policy, and the resulting cloned policy from Section 5.1
-16
As well, we evaluate against other PE, to show that preventing against behaviour cloning is non-trivial. We use several baselines. We first test policies that have approximately the same return as our ensemble by training PG with, and halting early rather than running until convergence. In the Near-Optimal case, ran until the PE had expected returns that matched the average achieved by our method. Conversely. "Random" policies are used as a comparison to show that it is possible to cause the cloned policy to do poorly, but the tradeoff is that the PE itself cannot perform well, which undesirable. These policies are also policies trained with PG, except they are stopped much earlier, when their clones matches the expected returns of our PG-APE. For each PG-APE, we use n = different tabular policies treated we then clone, and average across seeds. For the baselines, we hand-pick the policies, and thus only different policies.
-50
-83
-100
Visualization the cloned APE, The policy obtained from cloning the APE trained has av- erage expected reward 45.18, while the optimal policy has an average expected reward of -9, which is over a 5x increase.
Figure 3:
ensemble, which
PE Returns
Clone Returns
Returns Difference
PG-APE Near- Oatt-Ol PE Random Policy
-16.24 +1.20
-44.27 +1.07
-28.03
-16.74 1132
-16.67 11.31
+0.07
-44.59 #0.52
-44.52 #0.77
+0.07
Table 1:
Comparison of cloned PE. Each policy has their Returns precisely calculated through their closed form solutions. The final column reports the difference between the PE and the Clone, which only significant for our method.
As presented Table I all other PE have an insignificant difference (returns of the PE subtracted from returns of the cloned policy) between the performance of the PE and the cloned policy, except for our method. These empirical findings show that preventing behaviour cloning difficult, but possible using APE.
7

---

Under review as a conference paper at ICLR 2020
6 DISCUSSION & FUTURE WORK
Confidential Policies: There are promising research directions regarding the protection of policies, due to the many applications where confidentiality is crucial. As long as there is a model of the observer, our presented method provides a worst-case scenario of experts.
In our work, we focused on the case where the observer does not use the current trajectory to determine their policy. Instead, may be the case that the observer uses a sequential policy (one that depends on its previous states and/or actions), such as RNN to determine the context of the current expert.
Formally, the observer will no longer learn the policy formulated in Equation I that is solely dependent on the current state, but rather policy that dependent on the current trajectory:
To(a T1:1) == Ep(e)) (Ta))T (a
(8)
We found in our preliminary results that using RNN classifier which outputs (1:) simply ended up in with either optimal policies or crippled policies. In both cases, there was a relatively minor difference in performance between the policy ensemble and the cloned policy.
Unsurprisingly, when the observer has access to a strong enough representation for their policy, then they should be able imitate any policy. In this case, the worst-case set of experts cannot do much to prevent the cloning. We believe that this is an exciting conclusion, and is grounds for future work.
Continuous: Although our methods are evaluated in discrete state spaces, our approach can be generalized to continuous domains.
The Monte Carlo sampling in Equation suggests that the of continuous context may also be possible, given there is a strong enough function approximator to estimate the distribution of cls. We see this as an exciting direction for future work, to recover the full spectrum of possible adversarial policies under the constraint of Equation 2
The Semanties of Reward:
Although the minimization in Equation iipliees a logical equivalence between the success of behaviour cloning to the reward the cloned policy can achieve, it may follow that this not the case. may be the case that useless is defined differently by the expected reward the cloned policy achieves on a different reward function F. For example, a robot that is unpredictable should not be deployed with humans. Since the r functions in Squation[aare disentangled, the reward function that is minimized in Equation can be engineered to fit any definition of uselessness.
We can modify the objective of APE by modifying Equations and to use a different reward function F in the minimization, substituting R(() for i((I)= = Et F(se, or). The rest of the derivation and algorithm remain the same.
We think this is an exciting direction, especially for learning all different possible representations of the worst-case experts.
CONCLUSION
We present APE as well as its mathematical formulation, and show that policy gradient, a basic RL algorithm can be used to optimize a policy ensemble that cannot be cloned. We evaluated APE against baselines to show that adversarial behaviour is not feasible without our method.
This work identifies a novel yet crucial area in Reinforcement Learning, regarding the confidentiality of proprietary policies. The essence of our approach is that policy ensemble can achieve high return for the policy owner, while providing an external observer with a guaranteed low reward, making proprietary ensemble useless to the observer.
The formulation of our problem setup and the algorithm are very general. In this first work we demonstrate the solution in the deliberately chosen simple environments in order to better visualize the essence of our method. In our concurrent work we study thoroughly the application of our method in various domains, which is out of the scope of this introductory paper.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving. Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zhang. Tensorflow: A system for large- machine learning. CoRR, abs/1605.08695, 2016. URL http://arxx..orr//bb///// 08695
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML, 2004.
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. CoRR, abs/1807.10099, 2018. URL/http://axxc...orr hasp///b/// 1807. 10299
Mohammad Al-Rubaie and Morris Chang. Privacy-preserving machine learning: Threats and solutions. IEEE Security & Privacy, 17(2):99-58, 2019.
Borja Balle, Maziar Gomrokchi, and Doina Precup. Differentially private policy evaluation. In International Conference on Machine Learning, pp. 2130-2138, 2016.
Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning policy induction attacks. In International Conference on Machine Learning and Data Mining in Pattern Recognition, pp. 262-275. Springer, 2017.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zicba. End to end learning for self-driving cars. CoRR, abs/1604.07316, 2016. URL
http:://rxiin org.orggg.o....... abs/ 125716600 1604 07316
Daniel S. Brown, Yuchen Cui, and Scott Nickum. Risk-aware active inverse reinforcement learning. CoRR, abs/1901.02161. 2019. httthhttp///xx.... wttp://worggbbb//bb/booo /9011 02161
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoddddeeddee for statistical machine translation. CoRR, ths//140..107. 2014. URL /tlptttp://arxx...or////b/
1078
Felipe Codevilla, Matthias Müller, Alexey Dosovitskiy, Antonio López, and Vladlen Koltun. End- to-end driving via conditional imitation learning. CoRR, abs/1710.02410, 2017. URL httpp
// www.xv.org/ab/////eo 02410
Felipe Codevilla, Eder Santana, Antonio M. López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. CoRR, abs/1904.08988, 2019. URL. http:://axxxii
org/abs 1944. 08980
Pim de Haan, Dinesh Jayaraman, and Sergey Levine Causal confusion in imitation learning. CoRR, abs/1905.1 199999 1979, 2019. URL htllhhtt://aart wwwp/oorggaaaee........... bbrgg//bs/i a105 10t.//999..199999999
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. CoRR, abs/1802.06070, 2018. URL http:
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. CoRR, (as//709904905 2017. URI [.R.:http:://axxivoorr//bb 1709. 04905
Adam Gleave, Michael Dennis, Neel Kant, Cody Wild, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:1905.1661: 2019.
Anirudh Goyal, Riashat Islam, DJ Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick, Sergey Levine. and Yoshua Bengio. Transfer and exploration via the information bottleneck. In International Conference on Learning Representations, 2019. URL // //pp/////ee///iii// net./forummio==rgggee.

---

Under review as a conference paper at ICLR 2020
Peter Henderson, Wei-Di Chang, Pierre-Luc Bacon, David Meger, Joelle Pineau, and Doina Precup. Optiongan: Learning joint reward-policc options using generative adversarial inverse reinforcement learning. In Thirty- AAAI Conference Artificial Intelligence, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. CoRR, abs/1606.0344 2016.
10066666600 03476
Yordan Hristov, Alex Lascarides, and Subramanian Ramamoorthy. Interpretable latent spaces for learning from demonstration. CoRR, abs/1807.065585 2018. URL
2307.06587
Robert A Jacobs, Michael Jordan, Steven J Nowlan, Geoffrey E Hinton, et al. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Diederik Kingma and Jimmy Ba.
Adam: A method for stochastic optimization. CoRR,
abs/1412.6988) 2015.
Michael Laskey, Jonathan Lee, Wesley Yu-Shu Hsich, Richard Liaw, Jeffrey Mahler, Roy Fox, and Ken Goldberg. Iterative noise injection for scalable imitation learning. CoRR, abs/ abs/1703.09327, 2017. URL
wttp://g/aaaaaaaaa////// 09327
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. CoRR, (ww11/85500000 2018. UtL.http:/arxxv..rrrg
00909
Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learning with gaussian processes. 12 2011.
Yunzhu Li, Jiaming Song. and Stefano Ermon. Inferring the latent structure of human decision- making from raw visual inputs. CoRR, abs/1703.088400 2017. URL http::/arxiv..o/ abs/1703. 08840
Yen-Chen Lin, Zhang-Wa Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Yu Liu, and Min Sun. Tacties of adversarial attack deep reinforcement learning agents. CoRR, abs/1703.06748, 2017. URL httttttpp//axii/ wrw.
06748
Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Xiaojin Zhu. Policy poisoning in batch reinforcement learning and control. arXiv preprint /rXiv:///0.0582//, 2019.
Volodymyr Mnih, Adrià Puigdomênech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CoRR, abs/1602.011 2016. URL
abs 002.002.000 01783
Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, pp. 663-670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1-55860-707-2. URL
http://d acm. org/ citation.. wtt.ccmmmmmmm52.a..2 657801
Nicolas Papernot, Martn Abadi, Ifar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data, 2016.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. CoRR, abs/1810.0082) 2018. URL http:///axivv...g/bbbbes
00:21
Dean Pomerleau. Alvinn: An autonomous land vehicle in neural network. In NIPS, 1988.
Stéphane Ross, Geoffrey J. Gordon, and Andrew Bagnell. No-regret reductions for imitation learning and structured prediction. CoRR, abs/ 0tss/1011.000 1.0866 2010. URL abs Abs//111.. 0686
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. 06 2015.
10

---

Under review as a conference paper at ICLR 2020
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707. 107700077377 2017. URL
wttpaawwwi.i..r....aaaoooo abs/
1707.0634
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. CoRR, 0/bs//997.0116/ 2019. URL http:tt/axxx...ooo
01657
DJ Strouse, Max Kleiman-Wect Weiner, Josh Tenenbaum, Matthew Botvinick, and David J. Schwab. Learning to share and hide intentions using information regularization. CoRR, abs/ 1808.00.000 02090 2018. URL httphttp:///l //wpiivaaiiivvaa/aa wwtp:xv..wrr.aaroomo wwg/aba.....ooooooo../.......... 1808. C2093
Richard Sutton, David Mcallester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Ady. Neural Inf. Process. Syst, 12, 02 2000.
Baoxiang Wang and Nidhi Hegde. Private q-learning with functional noise in continuous spaces. arXiv preprint arXiv: wrXiv:191.106334 2019.
B Widrow and F W. Smith. Pattern recognizing control systems. Computer Inf. Sci. (COINS) Proc., 01 1964.
Yunbo Zhang, Wenhao Yu, and Greg Turk, Learning novel policies for tasks. CoRR, abs/1905500 905.05252. 2019. URLphtt:://axxiv..om aabs////////a 905 05252
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. AAAI Conference on Artificial Intelligence, 2008
A
APPENDIX
A.1 TRAINING DETAILS & HYPERPARAMETERS
For training, set ag 0.05, and the value weight to be 0.5, use annealed entropy regularization (Mnih et al., 2016) from 5e 1 to 5e - and set the discount factor = 0.99. Due to the contrasting gradients experienced, large batch sizes are used. In our experiments, take 1 gradient update of AdaM (Kingma& & Ba 2015) per batch of 4096 (containing multiple trajectories), and trained for 3e6 timesteps.
To estimate p[c|s) in Equation| we use a replay buffer that keeps track of the previous 60 contexts seen at each state.
Estimating the quantity in Equation 8 requires memory, which we use single GRU (Cho et al. 2014 as done in Strouse al. (2018), with the exception that only states are fed in as a one-hot. Due to our environment is deterministic, state sequences captures the action sequence information. The single unit then concatenated with the state, which feeds into a fully connected layer of 128, and then a soft-max, to produce the distribution c|s over contexts.
For our behaviour cloning, we collect 1e6 state action pairs, and train a tabular policy with 0.01 learning rate cross entropy softmax loss for 100 epochs. The large amount of data and epochs is to ensure that we can recover T. with little to no variance.
To solve the precise returns of the policies, inject noise of 1e - 9, to ensure hitting time always exists from each state. As well, we clip all the hitting times to -T' = -100.
A.2 ESTIMATING V log To
It is not obvious how Vo log m. should be estimated, since n. is never realized until the policy is cloned. Literally, it is a virtual policy.
Equation I offers straightforward method to back-propagate, similar to that of the Mixture of Experts model Jacobs et al. (191), except using an estimate of c|s instead of a gating network.
However, we can also rewrite Equation as Σ; p(c(() S)TT (a s) = Ee-ptels) [T (a s)], which
results in the gradient update being:
11

---

Under review as a conference paper at ICLR 2020
Vo (og = Vo log pteple)s)) Too (a s)]
(9)
which suggests a method of Monte Carlo sampling the inner expectation with 1 sampled context. Empirically, we use the Monte Carlo sampling method.
12


Reviews:
[{"review_id": "SkeyZ3HTYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of poisoning behavioral cloning using an optimized ensemble of demonstrators. The goals is allow the ensemble to still achieve an expected return above a certain threshold while minimizing the return of a policy trained via behavioral cloning. \n\nThis is a very exciting and novel paper, but it is not yet ready for publication. There are many typos and the paper is difficult to read at times. Also, the experiments are still very basic. While interesting, further experiments in more complicated discrete or continuous domains would greatly enhance the work. \n\nI would recommend not focusing on the privacy of human policies. I think a better motivation is to focus on theoretical ideas of adversarial inputs to behavioral cloning to study robustness as well as potential counter-intelligence strategies for autonomous agents. \n\nThis work has similarities to machine teaching and poisoning attacks. It would be interesting to see if recent methods for machine teaching for IRL [1] or poisoning for RL [2] can be used to solve the proposed problem. It would be good to situate this work within these related works. It seems like the proposed problem can be seen as a kind of anti-machine teaching for IRL where the goal is to find a set of good demonstrations that are maximally uninformative.\n\nSecond paragraph in 2.3: It's unclear what is the point of this paragraph. I would recommend not focusing so much on human demos.\n\nThe min-max approach seems related to GANs and Generative Adversarial Imitation Learning. Can something similar be used to scale this approach to high-dimensional tasks?\n\nEquations (4) and (5) are difficult to unpack. It would be nice add a little more explanation and intuition.\n\nBottom of page 5: What do you mean that continous policies can't be parameterized? Aren't most policy gradient algorithms continous with parameterized policies?\n\nIs the no-op action required to make BC fail?\n\nWhy only ensembles of 2? If you have 3 what happens in the grid env?\n\nThe authors mention that given an expressive enough policy, it should be possible to imitate any policy and thus the worst-case experts cannot prevent cloning. I would argue that a stronger representation such as a deep network would make the problem easier since deep networks are very susceptible to adversarial attacks and will likely over fit and poorly generalize given finite amounts of demonstrations.\n\n[1] Brown et al. \"Machine teaching for inverse reinforcement learning: Algorithms and applications.\"\n[2] Yuzhe et al. \"Policy poisoning in batch reinforcement learning and control.\"\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeehMRpYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to learn an ensemble of policies that is hard to imitate from their rollout trajectories. I like the idea of introducing the problem of privacy in reinforcement learning, and it is quite essential. However, some concerns are raised after checking the draft, and I believe the paper could be improved if some of the questions are addressed:\n\n* The current experiment could be an interesting demonstrative part to show how the algorithm works. However, there are no robust empirical experiments that the proposed method could achieve comparable performance/accumulated return as the policy ensemble (PE). I think some experiments on popular benchmarks like Mujoco simulation environment, robotics learning tasks, and Atari games are needed to make the point. The paper will become more convincing if the argument is proved on those benchmarks. Also, more experts should be explored (n > 2) in the experiments. \n\n* It is better to have some mathematical/theoretical analysis of the learning behavior of APE. For instance, is there a theoretical guarantee that APE could achieve comparable performance as PE?\n\n* The paper should discuss more details/analysis of the algorithm, like the choice of $\\alpha$ and $\\beta$, etc., which I think will affect the algorithm a lot.  \n\n* Some related literature on privacy in machine learning could be discussed in the related work section. \n\n\n====Minor that leads to confusion:\n-No mention about J and M before Alg 1; It is assumed to be the objective function and environment \n-No mention of the hyper-parameter $\\alpha$ after equation 2. \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxgCNX0tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper introduces a method for generating trajectories which prevent behavioral cloning in a policy gradient setting by learning varying experts which try to minimize the ability of a cloned policy. It runs experiments on a grid world to validate empirically that cloning is unsuccessful.\n\nRecommendation: While this is a novel concept and interesting, I cannot recommend acceptance in its current state. The paper was a bit hard to follow and I found the experiments not robust enough to fully characterize the method at this point. It is unclear whether this method really would prevent cloning given an apples-to-apples comparison. My understanding from the paper -- which was a bit hard to follow -- is that cloned policies were tabular while the APE policies were NNs. I would be more confident in results if more environment variations were tested, the cloned policies used more current and apples-to-apples comparisons, and overall if there were more clear details about the methodology. \n\nComments:\n+ It might be worth perusing the differential privacy and adversarial attack literature to think about whether demonstrations can simply be noised to retain information while crashing performance. This work seems relevant for example (it was put online in June which is sufficiently before the September deadline to mention it I believe): Behzadan, Vahid, and William Hsu. \"Adversarial Exploitation of Policy Imitation.\" arXiv preprint arXiv:1906.01121 (2019).\n+ In the discussion: \n\"We found in our preliminary results that using an RNN classifier which outputs p(c|τ1:t) simply ended up in with either optimal policies or crippled policies. In both cases, there was a relatively minor difference in performance between the policy ensemble and the cloned policy.\" --> There are no quantitative results for this so either results should be included and discussed or this should be future work.\n+ The algorithm box doesn't really add a whole lot of information other than saying that trajectories are collected and then gradients are updated. It would be really nice to have a very clear picture of what's happening at each point in the algorithm. In its current state the paper is hard to follow and decipher this sequence.  See for example Algorithm one in: https://papers.nips.cc/paper/6391-generative-adversarial-imitation-learning.pdf . \n+ Notation-wise, R(t) is a bit unusual notation for the RL literature, the advantage is usually r + \\gamma V(s') - V(s), where r+\\gamma V(s') is the action value Q(s,a). Given that the advantage is denoted as A(s,a), it would be clearer I think to use the Q(s,a) notation. Also the notation changes from section 2.2 to section 3.2 from A(s,a) to A(t). Keeping consistent notation would make this paper a lot easier to read.\n+ The related work section is in the middle of the paper. it'd be nice to have it earlier to set the context of the work.\n+ In the multiple policies section, a recent work has shown how to learn multiple policies from multiple experts using a mixture of experts framework -- though they frame it as options: Henderson, Peter, Wei-Di Chang, Pierre-Luc Bacon, David Meger, Joelle Pineau, and Doina Precup. \"Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning.\" In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.\n+ Part of the way this defeats behaviour cloning is through the assumption that there are multiple trajectories to be learned from. It would be interesting to see if methods like the one above or any of the others mentioned can recover optimal performance from noisy trajectories by similarly learning multiple policies. In fact, \n+ \"Policy Gradient (PG) (Sutton et al., 2000) and its variants (Schulman et al., 2015) aim to directly learn\nthe optimal policy π, parameterized by θ.\" --> I think some other citations of variants should be added for the final version instead of only referencing Schulman 2015. There are a lot now, so maybe adding PPO, DDPG, and a few others might be nice. Otherwise you could also just cut out the variants bit since it's not necessary. \n+ All first quotation marks are backwards in the document\n+ I think the experiments ran were a bit lacking in robustness and details. Since this is an adversarial method, I would expect more variance across seeds and 3 seeds may not be enough to characterize this. Table 1 has +/- but does not state what this represents. Standard Deviation or Standard error? Does Table 1 represent returns for rolled out policies after learning or across all episode returns during learning? For the behavioural cloning method, it says a \"tabular policy\" was trained. Does this mean that the experts were trained using policy gradients and neural networks while the behavioural cloning method used a tabular policy? If so, I think this would be at a detriment to the method being tricked. I think it is a necessary condition to validate this method across several gridworld environment variations, seeds, and with more robust cloning methods (if in fact the behavioural policy was underpowered (tabular vs. nn). Overall, it would be great to have more details. While the visualizations of the gridworld itself were nice, I think they took up a lot of space which could be replaced with more detailed explanations and robust quantitative results. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although the reviewers appreciated the novelty of this work, they unanimously recommended rejection.  The current version of the paper exhibits weak presentation quality and lacks sufficient technical depth.  The experimental evaluation was not found to be sufficiently convincing by any of the reviewers.  The submitted comments should help the authors improve their paper.
Decision: Reject