Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel targeted preference-based adversarial attack against deep reinforcement learning (DRL) agents, so that the DRL agents would show extreme behaviors desired by adversaries. In particular, the proposed PALM adopts human preference, an intention policy and a weighting function to guide the adversary. Moreover, the authors also provided a theoretical analysis for the convergence of PALM. Finally, the experimental results suggest that PALM outperforms other baselines under the targeted attack setting.
strength_and_weaknesses: #### **Strengths**:
- This work focuses on a novel targeted attack setting.
#### **Weaknesses**:
- In general, I found this work kind of hard to follow given the complicated notations. Based on my understanding, PALM uses RL to attack an RL agent. 
- Moreover, I am confused by Equation 4. What is the target soft Q-function? If we already know the targeted Q-function \bar{\phi}, why don’t we just use it in Equation 3 directly? If we don’t know \bar{\phi}, how can we optimize \phi with Equation 4?
- On page two, the authors mentioned several works that could improve the robustness of policies. Although I agree that this paper focuses on designing attack instead of defense, I am still wondering whether the proposed attack could surpass such defenses. Could the authors justify or provide some discussions about whether the proposed method could break these defenses or why they didn't run experiments on this? What if the proposed attack could not work under a even simple defense?
- I also have a question about Section 4.3. (please see below in clarity)

clarity,_quality,_novelty_and_reproducibility: - Novelty: This work presents a novel idea of targeted attack against DRL agents by leveraging preference-based RL. 
- Clarity: This work is kind of hard to follow. Moreover, Section 4.3 provides convergence guarantees for the proposed method. But I don’t understand how this section helps explain the efficiency or optimality of the proposed method. If the intention policy is not optimal, could the adversarial policy still be optimal? Especially in Equation 7, if \theta is not optimal, even if the \alpha is the argmin of L_att, does it still imply the optimality of the attack?
Quality: This work has some minor issues in terms of the presentation. 
Reproducibility: The author provides a link to the code and some demo. I trust the authors regarding the experimental results.

summary_of_the_review: Overall, I think this work is interesting. However, as stated above, I didn’t fully understand Equation 4 and its purpose, which might affect my understanding and judgement of this paper and its contribution. Currently, I rate it as 5, but I am willing to increase my score if the authors could answer my questions above. Thank you very much!
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a way to run targeted adversarial attacks against Reinforcement Learning agents. The threat model consist in modifying the state perceived by the agent in order to trigger specific actions. The method is to both find an optimal "intention policy" that achieves the target behavior, then try to match it with an adversarial policy in which states are perturbed. The latter is formulated as bi-level optimization. The authors provide a training procedure for that problem and show that under some conditions it will converge to a critical point. Experimental results show that this attack can achieve the target behavior against multiple RL agents, and outperforms multiple baselines by large margins. Online examples demonstrate the attack's success in Meta-world.
strength_and_weaknesses: **Strengths**: The addressed problem is challenging. The proposed solution is detailed and convincing: it consists of multiple steps, leverages several optimization methods, and is expressed in a complete formalism. This solution also seems largely novel and does not simply extend past work. The experimental setup seems complete and the results very strong. Online demos help grasping better while the paper accomplishes on practical examples.

**Weaknesses**: One weak point is clarity. As a reader very familiar with adversarial attacks but less so with Reinforcement learning, understanding the main points of the paper was difficult. Combining standard RL formalism, intention policy, adversarial policy, victim, multiple rewards and optimizations, etc. lead to a heavy formalism. Given the complexity of the work technicality is necessary, and the authors do provide a diagram of the pipeline, but I think clarity could be improved. Parts of the method remain unclear to me: for instance, I am not sure whether the intention policy is learned before, or jointly to the weighing function and adversarial policy (Figure 2 says jointly, but the method seem to imply before). 

Another point I am not sure about is the attack budget: it is mentioned in appendix, and the introduction says that perturbations are imperceptible, but the actual method does not make clear how that budget is handled or why it is necessary. Overall I'd suggest having a full part dedicated to the outline of the method, keeping technical aspects to a minimum, and moving the rest (theoretical results, learning rates, etc) somewhere else. More detailed diagrams for subparts of the pipeline would also be useful.

Another issue is the limited amount of motivation the authors provide for their work. One sentence in the introduction says that studying adversarial attacks is "crucial", but why some threat models rather than others? With an attack like PALM what could an attacker concretely do in the real world? Motivating or defining informally specific aspects of the work could be useful as well. What is an "imperceptible perturbation"? What imperceptible means on image or speech is clear, but on RL states it is harder to grasp. When it comes to theoretical results, they come with so-called "mild" conditions. Are those conditions typically met? Having convincing answer to those questions would make the paper stronger.

A final (more minor) point: in absence of competing targeted attacks the authors make straightforward modifications to untargeted attacks to get baselines. That this would lead to weak baselines was predictable, and the comparison isn't too fair. Providing it is better than not, but I do not think it is an essential part of the paper, and it could be moved to an appendix without loss.
clarity,_quality,_novelty_and_reproducibility: * **Clarity**: for the general paper outline see weaknesses above. language has some flaws (e.g. "we wonder that" followed by a question in the intro is unnatural) but this is not a major issue.

* **Quality**: the work itself seems very strong.

* **Novelty/Originality**: While not 100% familiar with the state of the art, I do not think targeted attacks on RL agents have been attempted before. The problem is novel, and the solution is complex and original.

* **Reproducibility** is satisfying. Putting the code on anonymous github is very helpful. I did not try to reproduce the results but it would be an easy check.
summary_of_the_review: The paper provides what seems to be a strong work with very good results. I think a subpar paper outline focusing to much on technicality and not enough on motivations undermine the work. I tend to prefer the work accepted, but it would benefit from some rewriting.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a novel method for targeted adversarial attacks on Deep Reinforcement Learning algorithms by modifying agent observations. It learns an adversarial reward function via the preferences of a human-in-the-loop, and uses it to guide an intention policy that itself generates behaviors the adversarial policy eventually learns to reproduce by perturbing agent observations.

strength_and_weaknesses: ### Strengths

Figures and the detailed algorithm are well-designed and help understand some aspects of the work.

Using an intention policy to explore the support of states that lead to maximizing the adversarial human reward is an interesting idea. It would be nice to show how much suboptimal exploration would hinder the same approach without the intention policy.

Experimentation seems to be solid and the proposed method performs better than other benchmarked methods in most tasks.

Having code and videos available is a very nice plus.

### Weaknesses

The title should mention human-in-the-loop in my opinion.

The related work is not complete at all. It is missing its core: targeted attacks for Deep RL. 
A reference comes to mind: Targeted Attacks on Deep Reinforcement Learning Agents through Adversarial Modifications (Hussenot et al, 2020).
While this can be fixed, I find it puzzling that the authors have not found it useful or took the time to include at least a few references in that direction, and better a complete review of the literature. I am willing to give authors the benefit of doubt but this needs to be addressed before I am able to recommend for acceptance.

On the problem setup: 1) “generalized state-adversarial” is not a great name, should be “rewarded state-adversarial” instead, or something that better illustrates that there is a reward component involved, 2) “victim policy” is an offensive terminology, use ‘perturbed’/‘target’ instead maybe?

It is unclear whether the reward learning method proposed is a contribution or not.

$\pi_{\nu\circ\alpha}$ is not defined (though one can figure out what it means).

On the method: the connection to SAC and soft-Q-learning should be mentioned, it is not clear whether it is a contribution or not in the current state, and this choice has no motivation (e.g. continuous control). 

On the objective:
Is it necessary to reweigh the KL-divergence? This is a modification of the standard that should be better motivated. E.g. via an ablation study.

I do not get why $\omega$ is learned via this implicit parameterization. Could it simply be fixed and seen as a hyperparameter? In other words, this is an additional source of complexity (bi-level optimization) of the algorithm and its inclusion should be better motivated.
Another point is that the nature of \pi_\alpha should be clarified sooner in the paper. How does it operate on the observations?? Is $\alpha$ kept fixed?

I think there should be evidence (i.e. empirical, via an ablation study) that the intention policy is needed.
Also, is it not possible to have an epsilon-greedy switch between perturbed and intention instead of the switching heuristic proposed?

Drawbacks include that the intention policy has to be learned semi-online (part of the episode is played according to the intention policy), not simply through off-policy learning. Experiments showing it is not possible would be welcome. 

The main drawback for me is that the writing is not clear enough on key parts of the work. Some super important details (weighting, bi-level optimization, and crucially *the nature of the adversarial modifications*) are hard to grasp or even missing from the current state of the paper.

There are no details on preference collection! This is absolutely crucial to the work.

Nit: the videos are not very impressive, the adversarial command always consists of performing the inverse of the sequence of actions from the policy. Adding other settings would better convey the usefulness of the approach. 

clarity,_quality,_novelty_and_reproducibility: Clarity: can be improved a lot.

Quality: experimentation is well-executed but the quality of the presentation and of the evidence in favor of the proposed method is sub-par.

Novelty: first preference-based method for targeted adversarial attacks on Deep RL agents, though the influence of the preference-based component is not properly quantified.

Reproducibility: the code and some videos are available through anonymized links.

summary_of_the_review: The work features an interesting idea and convincing experimental results.

Though, I feel that the work is not ready for publication yet, and that a proper literature review, improvements to the writing, clarifications and additional ablations would make the work stand out much more (see my detailed comments).
I cannot recommend this work for acceptance in its current state.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of using preference based RL to adversarially attack Deep RL policies.  They propose a novel intention policy as a way to stabilize training which leads to improved performance in this regime
strength_and_weaknesses: Strengths:
* The novelty of the idea of using Preference based RL for adversarial attacks.
it is often hard to define what a successful attack is formally, but you know it when you see it, so this could be a promising direction

* The intention policy seems promising as an approach for solving exploration problems in adversarial attack domains

Weaknesses:
* The paper seems to not test the core claim: that preference Based RL is useful for adversarial attacks

Most of the tasks are tasks for which a hard-coded reward function does exist(maybe all of the tasks, the setup is a bit unclear).  As such, they can't be effectively used to show that preference based RL is useful for adversarial attacks since the problem of specifying the task or solved.

The counterargument to the above presented by the authors (that it would be hard to specify directly specify reward functions for tasks like Go), is contradicted by  Alpha Zero which worked off of a hard-coded human reward, and learned an even more well shaped reward in terms of the value function.  

* It seems that all of the empirical improvements come from the intention policy rather than the preference based RL

-- The preference based model is betting other models which have the ground-truth reward function.  This shouldn't happen if the only change is to use preference based RL, since in the best case preference based RL will reduce to the oracle model.  Thus the only other change (the intention policy), is the probable cause for the difference, but this is not framed as the core contribution of the paper.  If it were, it should be analyzed outside the preference based RL setting, since it does not depend on that.
clarity,_quality,_novelty_and_reproducibility: As far as I can tell the idea of using preference based RL for adversarial attacks is novel.
summary_of_the_review: I'm recommending weak rejection of this paper since it seems like the empirical results do not justify the conclusion, due to the intention policy confounding experiments on preference based RL, and preference based RL not being properly evaluated on its own.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes the Preference-based Advisersarial Manipulation (PALM) algorithm which performs targeted attacks on deep reinforcement learning agents. This method used human preferences feedback and relies on an intention policy and a weighting function to guide the adversarial policy during training. Empirical results show that PALM outperforms several flavors of the adversarial algorithms SA-RL and PA-AD (which, as opposed to PALM, were not designed to manipulate the victim agent policy).
strength_and_weaknesses: The main strengths of this work are:
1. A novel approach that allows to manipulate deep RL agents towards a certain target (as opposed to simply make the agent fail in its task).
2. Supporting theoretical analysis on the convergence of PALM.

The main weaknesses of this work are:
1. An (intuitive) explanation for the role of the intention policy (beyond the given equations) is lacking. What is the motivation behind this? How does it interplay with the adversarial policy and guide its training?
2. In the experiments, the authors compare PALM to existing algorithms that were not designed to manipulate the victim agent policy, in manipulation tasks. It would be interesting to see what happens if the situation is the other way around. How does PALM fair in experiments settings for which SA-RL and PA-AD are designed?
3. Data is collected following a combined policy that depends on the perturbed policy. It is not clear why/how this can be done in practice. What happens if we cannot control according to which policy data is collected?
clarity,_quality,_novelty_and_reproducibility: To allow better assessment of the originality of the paper, it would be useful if the authors could elaborate on how the fact that their approach "emphasizes manipulating victim policy" makes it unique? How far is it from existing approaches? How hard would it be to empower these with the ability of performing targeted attacks (instead of making the agent fail or choose the worst action)?

In terms of clarity, (besides the first weakness listed above), some sentences are not clear. For examples:
- In Section 2, "Unlike previous research, past works focus on making victim policy work fail, while our approach...". Should "Unlike previous research" be removed?
- In Section 3, "The target of GSA-MDP is to solve the optimal adversary pi_alpha*, which enables the victim to achieve the maximum expected return over all states". Why would we want to enable the victim to succeed?
- In Section 3, the context for reward optimization from preferences is not clear at this point. Where will this be used? More context should be given in the section or alternatively, this part could be moved to the next section, where it is used.
- In Section 4, "which indicates the optimal adversarial manipulation without the restriction of the victim". 
- In Section 5, "and we rigorously design for the navigation scenario". What do you design?
Also, some of the notation is not explained in the text. For example, in Section 3, \alpha in \pi_{\alpha}, \hat{A} and \hat{P}, the norm p (which norm is used)?

Smaller comments:
- Figure 1 is not referred to in the text.
- Section 3, 3rd line: is reward function -> is the reward function
- Section 4, \tau_t is the transition at time t: if \tau_t ~ B, isn't it a state? what does transition mean?
- Section 4.1. missing parenthesis around 5 (above equation (7))
summary_of_the_review: In summary, the authors present an interesting approach for target attacks on deep reinforcement learning agents. The main weaknesses of this work are that the level of novelty and originality is not fully clear and it also seems that the empirical results showing that the proposed approach outperforms existing algorithms were performed in "unfair" settings.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers agreed that the paper considers an interesting setting for targeted adversarial attacks against deep RL agents and acknowledged the novelty of using a preference-based RL framework as an attack method. However, the reviewers pointed out several weaknesses and shared a common concern that several claims made in the paper lack strong support (in terms of experiments or theory). We want to thank the authors for their detailed responses; the paper was also discussed among all the reviewers, considering the responses and revision. Based on the raised concerns and follow-up discussions, unfortunately, the final decision is a rejection. Nevertheless, the reviewers have provided detailed and constructive feedback. We hope the authors can incorporate this feedback when preparing future revisions of the paper.
justification_for_why_not_higher_score: The reviewers shared a common concern that several claims made in the paper lack strong support and reached a rejection decision.
Decision: Reject