Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary: The paper introduces a method for generating trajectories which prevent behavioral cloning in a policy gradient setting by learning varying experts which try to minimize the ability of a cloned policy. It runs experiments on a grid world to validate empirically that cloning is unsuccessful.

Recommendation: While this is a novel concept and interesting, I cannot recommend acceptance in its current state. The paper was a bit hard to follow and I found the experiments not robust enough to fully characterize the method at this point. It is unclear whether this method really would prevent cloning given an apples-to-apples comparison. My understanding from the paper -- which was a bit hard to follow -- is that cloned policies were tabular while the APE policies were NNs. I would be more confident in results if more environment variations were tested, the cloned policies used more current and apples-to-apples comparisons, and overall if there were more clear details about the methodology. 

Comments:
+ It might be worth perusing the differential privacy and adversarial attack literature to think about whether demonstrations can simply be noised to retain information while crashing performance. This work seems relevant for example (it was put online in June which is sufficiently before the September deadline to mention it I believe): Behzadan, Vahid, and William Hsu. "Adversarial Exploitation of Policy Imitation." arXiv preprint arXiv:1906.01121 (2019).
+ In the discussion: 
"We found in our preliminary results that using an RNN classifier which outputs p(c|τ1:t) simply ended up in with either optimal policies or crippled policies. In both cases, there was a relatively minor difference in performance between the policy ensemble and the cloned policy." --> There are no quantitative results for this so either results should be included and discussed or this should be future work.
+ The algorithm box doesn't really add a whole lot of information other than saying that trajectories are collected and then gradients are updated. It would be really nice to have a very clear picture of what's happening at each point in the algorithm. In its current state the paper is hard to follow and decipher this sequence.  See for example Algorithm one in: https://papers.nips.cc/paper/6391-generative-adversarial-imitation-learning.pdf . 
+ Notation-wise, R(t) is a bit unusual notation for the RL literature, the advantage is usually r + \gamma V(s') - V(s), where r+\gamma V(s') is the action value Q(s,a). Given that the advantage is denoted as A(s,a), it would be clearer I think to use the Q(s,a) notation. Also the notation changes from section 2.2 to section 3.2 from A(s,a) to A(t). Keeping consistent notation would make this paper a lot easier to read.
+ The related work section is in the middle of the paper. it'd be nice to have it earlier to set the context of the work.
+ In the multiple policies section, a recent work has shown how to learn multiple policies from multiple experts using a mixture of experts framework -- though they frame it as options: Henderson, Peter, Wei-Di Chang, Pierre-Luc Bacon, David Meger, Joelle Pineau, and Doina Precup. "Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning." In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
+ Part of the way this defeats behaviour cloning is through the assumption that there are multiple trajectories to be learned from. It would be interesting to see if methods like the one above or any of the others mentioned can recover optimal performance from noisy trajectories by similarly learning multiple policies. In fact, 
+ "Policy Gradient (PG) (Sutton et al., 2000) and its variants (Schulman et al., 2015) aim to directly learn
the optimal policy π, parameterized by θ." --> I think some other citations of variants should be added for the final version instead of only referencing Schulman 2015. There are a lot now, so maybe adding PPO, DDPG, and a few others might be nice. Otherwise you could also just cut out the variants bit since it's not necessary. 
+ All first quotation marks are backwards in the document
+ I think the experiments ran were a bit lacking in robustness and details. Since this is an adversarial method, I would expect more variance across seeds and 3 seeds may not be enough to characterize this. Table 1 has +/- but does not state what this represents. Standard Deviation or Standard error? Does Table 1 represent returns for rolled out policies after learning or across all episode returns during learning? For the behavioural cloning method, it says a "tabular policy" was trained. Does this mean that the experts were trained using policy gradients and neural networks while the behavioural cloning method used a tabular policy? If so, I think this would be at a detriment to the method being tricked. I think it is a necessary condition to validate this method across several gridworld environment variations, seeds, and with more robust cloning methods (if in fact the behavioural policy was underpowered (tabular vs. nn). Overall, it would be great to have more details. While the visualizations of the gridworld itself were nice, I think they took up a lot of space which could be replaced with more detailed explanations and robust quantitative results. 

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper proposes a method to learn an ensemble of policies that is hard to imitate from their rollout trajectories. I like the idea of introducing the problem of privacy in reinforcement learning, and it is quite essential. However, some concerns are raised after checking the draft, and I believe the paper could be improved if some of the questions are addressed:

* The current experiment could be an interesting demonstrative part to show how the algorithm works. However, there are no robust empirical experiments that the proposed method could achieve comparable performance/accumulated return as the policy ensemble (PE). I think some experiments on popular benchmarks like Mujoco simulation environment, robotics learning tasks, and Atari games are needed to make the point. The paper will become more convincing if the argument is proved on those benchmarks. Also, more experts should be explored (n > 2) in the experiments. 

* It is better to have some mathematical/theoretical analysis of the learning behavior of APE. For instance, is there a theoretical guarantee that APE could achieve comparable performance as PE?

* The paper should discuss more details/analysis of the algorithm, like the choice of $\alpha$ and $\beta$, etc., which I think will affect the algorithm a lot.  

* Some related literature on privacy in machine learning could be discussed in the related work section. 


====Minor that leads to confusion:
-No mention about J and M before Alg 1; It is assumed to be the objective function and environment 
-No mention of the hyper-parameter $\alpha$ after equation 2. 


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper addresses the problem of poisoning behavioral cloning using an optimized ensemble of demonstrators. The goals is allow the ensemble to still achieve an expected return above a certain threshold while minimizing the return of a policy trained via behavioral cloning. 

This is a very exciting and novel paper, but it is not yet ready for publication. There are many typos and the paper is difficult to read at times. Also, the experiments are still very basic. While interesting, further experiments in more complicated discrete or continuous domains would greatly enhance the work. 

I would recommend not focusing on the privacy of human policies. I think a better motivation is to focus on theoretical ideas of adversarial inputs to behavioral cloning to study robustness as well as potential counter-intelligence strategies for autonomous agents. 

This work has similarities to machine teaching and poisoning attacks. It would be interesting to see if recent methods for machine teaching for IRL [1] or poisoning for RL [2] can be used to solve the proposed problem. It would be good to situate this work within these related works. It seems like the proposed problem can be seen as a kind of anti-machine teaching for IRL where the goal is to find a set of good demonstrations that are maximally uninformative.

Second paragraph in 2.3: It's unclear what is the point of this paragraph. I would recommend not focusing so much on human demos.

The min-max approach seems related to GANs and Generative Adversarial Imitation Learning. Can something similar be used to scale this approach to high-dimensional tasks?

Equations (4) and (5) are difficult to unpack. It would be nice add a little more explanation and intuition.

Bottom of page 5: What do you mean that continous policies can't be parameterized? Aren't most policy gradient algorithms continous with parameterized policies?

Is the no-op action required to make BC fail?

Why only ensembles of 2? If you have 3 what happens in the grid env?

The authors mention that given an expressive enough policy, it should be possible to imitate any policy and thus the worst-case experts cannot prevent cloning. I would argue that a stronger representation such as a deep network would make the problem easier since deep networks are very susceptible to adversarial attacks and will likely over fit and poorly generalize given finite amounts of demonstrations.

[1] Brown et al. "Machine teaching for inverse reinforcement learning: Algorithms and applications."
[2] Yuzhe et al. "Policy poisoning in batch reinforcement learning and control."

review_title: Official Blind Review #3

Meta-review: Although the reviewers appreciated the novelty of this work, they unanimously recommended rejection.  The current version of the paper exhibits weak presentation quality and lacks sufficient technical depth.  The experimental evaluation was not found to be sufficiently convincing by any of the reviewers.  The submitted comments should help the authors improve their paper.
Decision: Reject