PALM: Preference-based Adversarial Manipulation against Deep Reinforcement Learning
Authors: Fengshuo Bai
Runze Liu
Yaodong Yang
Yali Du
Venue: Submitted to ICLR 2023 (2023)
Keywords: adversarial attack
deep reinforcement learning
preference-based reinforcement learning
bi-level optimization

Abstract: To improve the robustness of DRL agents, it is important to study their vulnerability under  adversarial attacks that would lead to extreme behaviors desired by adversaries. Preference-based RL (PbRL) aims for learning desired behaviors with human preferences. In this paper, we propose PALM, a preference-based adversarial manipulation method against DRL agents  which adopts human preferences to perform targeted attacks with the assistance of an intention policy and a weighting function. The intention policy is trained based on the PbRL framework to guide the adversarial policy  to mitigate restrictions of the victim policy during exploration, and the weighting function learns weight assignment to improve the performance of the adversarial policy. Theoretical analysis demonstrates that PALM converges to critical points under some mild conditions. Empirical results on a few manipulation tasks of Meta-world show that PALM exceeds the performance of state-of-the-art adversarial attack methods under the targeted setting. Additionally, we show the vulnerability of the offline RL agents by fooling them into behaving as human desires on several Mujoco tasks. Our code and videos are available in https://sites.google.com/view/palm-adversarial-attack.

Under review as a conference paper at ICLR 2023
PALM: PREFERENCE-BASER ADVERSARIAL MANIPU- LATION AGAINST DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep reinforcement learning (DRL) methods are vulnerable to adversarial attacks such as perturbing observations with imperceptible noises. To improve the robust- ness of DRL agents, is important to study their vulnerability under adversarial attacks that would lead to extreme behaviors desired by adversaries. Preference- based RL (PbRL) aims for learning desired behaviors with human preferences. In this paper, we propose PALM, preference-bait based adversarial manipulation method against DRL agents which adopts human preferences to perform targeted attacks with the assistance of intention policy and a weighting function. The intention policy is trained based on the PbRL framework to guide the adversar- ial policy to mitigate restrictions of the victim policy during exploration, and the weighting function learns weight assignment to improve the performance of the adversarial policy. Theoretical analysis demonstrates that PALM converges critical points under some mild conditions. Empirical results on few manipu- lation tasks of Meta-world show that PALM exceeds the performance of state- of-the-art adversarial attack methods under the targeted setting. Additionally, we show the vulnerability of the offline RL agents by fooling them into behaving as human desires several Mujoco tasks. Our code and videos available in attps::/sites: google. com www.wwwwaaa.
INTRODUCTION
Adversarial examples in image classifiers have prompted a new field of studying the vulnerability of deep neural networks (DNN). Recent researches demonstrate that reinforcement learning (RL) agents parameterized by DNN also show vulnerability under adversarial attacks (Huang et al. 2017]] Pattanaik al. 2018 Zhang et al. 2020 1 Sun al., 2022). Adversaries generate impercep- tible perturbations to the observations of victim agent, making agents fail to complete the original behaviors. While adversarial attack an crucial approach evaluate the vulnerability of the agents, targeted attack in RL has received little attention. Recently, embodied intelligence (Gupta et al.)
2021 Liu et al., 2022, Ahn et al., 2022; Reed al. [002 Fan et al. 2022) is considered as a mean- ingful way to improve the cognitive ability of artificial intelligence. These embodied agents show powerful capabilities while possibly exposing vulnerability. Therefore, we wonder that:
How can one manipulate the agent to perform desired behaviors, and whether the embodied agents are robust to adversarial manipulations?
To achieve targeted adversarial attacks, one straight way is to design respective rewards for the adversary agents. However, specifying precise reward function can be challenging. For example, is difficult to design a reward function denoting the goodness the current step in the game of Go. For example, it is difficult in chess games to craft reward function which identify the quality of each move. In preference- -based RL framework, a human only needs to provide binary preference labels over trajectories of the agent (Christiano et al. 2017). Compared reward engineering, preference-basee RI is an easier way to learn policies through human preferences. Meanwhile, RL shows an excellent capacity learn novel behaviors with
recent research on preference-basee few preference labels (Lee et al.) 2021a Park et al. 2022 Liang et al. 2022). and significantly
improves feedback efficiency.
Motivated by this, consider using preference-based RL perform targeted adversarial attacks from the following perspectives. On the one hand, it is difficult to define the desired behaviors to achieve targeted attacks, but humans can implicitly inject intentions by providing binary preferences.

---

Under review as a conference paper at ICLR 2023
Perturbed state 3
Victim policy
-
Perturbation
A
Action
State
Figure 1: Illustration of targeted attack from PALM. The adversary first receives the true state from the environment and perturbs it into Then the victim observes and takes action according to it.
On the other hand, preference-basee preference labels are enough to learn well-behaved policies. As shown in Figure D we emphasize PALM recover the reward function of desired behaviors via preference-based RL framework and manipulate agents to perform human desired behaviors.
RL is data-efficient manipulation method because few
In this paper, we propose Preference- AdversariaL Manipulation (PALM) algorithm, which performs targeted attack from human preferences. PALM includes an adversary to perturb the ob- servations the victim. To achieve targeted attack and better exploration, we introduce an intention policy, which is learned from human preferences, as the learning target of the adversary. Addition- ally, we utilize a weighting function to assist adversary learning by re- re-weighting state examples.
In summary, contributions three-fold. Firstly, to the best of our knowledge, we propose the first targeted adversarial attack method against DRL agents via preference-basee RL. Secondly, theoretically analyze PALM and provide convergence guarantee under some mild conditions. Lastly, we design two scenarios and experiments on Meta-world demonstrate that PALM outper- forms the baselines by large margin. Empirical results demonstrate that both online and offline RL agents are vulnerable to our proposed adversarial attacks.
2 RELATED WORK
Many previous works adversarial attacks study the vulnerability of DRL. agent. Huang et al. (017) computes adversarial perturbations via utilizing the technique of FGSM (Goodfellow et al.) 2015) to mislead the victim policy, not to choose the optimal action. Pattanaik al. (0188) presents an approach that leads the victim to select the worst action based on the Q-function of the vic- tim. Gleave et al. (2020) conducts adversarial attacks under the two-player Markov game instead of perturbing the agent's s observation. ZZhang et al. (2020) proposes the state-adversarial MDP (SA- MDP) and develops two adversarial attack methods named Robust Sarsa (RS) and Maximal Action Difference (MAD). SA-RL (Zhang et al., (02T) directly optimizes the adversary policy to perturb state in the form of end-to-end RL. PA-AD (Sun et al. 2022) designs an RL-baaed "director" to find the optimal policy perturbing direction and construct optimized brimizedd-baedd "actor" to craft perturbed states according to the given direction. Methods untargeted adversarial attack focus on making the victim policy fail, while approach emphasizes manipulating the victim policy. That is to say, victim's 's behaviors are consistent with the preference of the manipulator under attacks. Another line of works (Pinto et al. 2017; Mandlekar et al. 2017 Pattanaik Pataanata al. (018)) consider using adversarial examples to improve the robustness of policies, although it is out of the scope of this paper.
There are few prior researches that focus on targeted attacks on RL agents, [in et al. (2017) first proposes targeted adversarial attack method against DRL agents. It attacks the agent to reach targeted state. [Buddareddygari al.) (2022) also present a strategy to mislead the agent towards to specific state by placing object in the environment. The hijacking attack (Boloor et al.) 2020) is proposed to attack agents to perform targeted actions on autonomous driving systems. Hussenot] et al. (019)) provides new perspective that attacks the agent to imitate target policy. Our method differs that PALM manipulates victim behave as human desire and focuses on the preference baseed RL. Xiao al. (2019) proposes the first adversarial attack method against real world visual naviga- tion robot, Lee et al. (2021) investigates targeted adversarial attacks against the action space of the agent. Our method differs that PALM leverages preference-based RL to avoid reward engineering
2

---

Under review as a conference paper at ICLR 2023
outer-level: optimize w
learning
TT (als)
outer loss /x
(s)
human preferences
env
(als)
inner Ltt
inner-level: optimize &
Figure 2: Framework of PALM. PALM jointly leams intention policy, adversary and weight- ing function under bi-level optimization framework. the inner-level, the adversary is optimized to approach the intention policy which learns via preference- basede-badddd-e RL. In the outer-level, the weight- ing function is updated to maximize the performance of the adversary evaluated by the outer loss.
and learns an intention policy tackle restricted exploration problem, so that PALM can attack the victim policy to perform behaviors far from its original behaviors.
Training agents with human feedback has been investigated several works. Preference- based RL provides an effective way to utilize human preferences for agent learning. Christiano al. (2017) proposes a basic learning framework for preference-basee RL. To further improve feedback effi- ciency, Jbarz et al.) (2018) additionally utilizes expert demonstrations to initialize the policy besides learning the reward model from human preferences. However, previous methods need plenty of hu- man feedhack, which is usually impractical. Many recent works have proposed to tackle this prob- lem. [eee lem. et al. (021a) presents bredback-efficieee preference- based RL algorithm, which benefits from unsupervised exploration and reward relabeling. Park et al.) (2022) further improves feedback efficiency by semi-supervised reward learning and data augmentation, while Liang et al.] pro- poses an intrinsic reward to enhance exploration. To the best of our knowledge, our method the first achieve targeted adversarial attack against DRL agents through preference-basee RL.
PROBLEM SETUP
The Victim Policy. In RI, agent learning can be modeled as a finite horizon Markov Decision Process (MDP) defined as tuple (S,A,R, P,,). S and A denote state and action space, spectively, R : S x A x S + R is the reward function and E (0,1) is the discount factor. P A S [0, 1] denotes the transition dynamics, which determines the probability of trans- ferring to s' given state and action a. We denote the stationary policy T. S - P(A), where are parameters of the victim. We suppose the victim policy is fixed and uses the approximator.
The Adversarial Policy. To study the adversary learning with human preferences, formulate it as rewarded atate-adversarial Markov Decision Process (RSA-MDP). Formally, a RSA-MDP is tuple (S, A, B, R, P, The adversary To S + P(S) perturbs the states before the victim observes them, where a are parameters of the adversary. Specifically, the adversary perturbs the state into š which is restricted by ((s) (i.e., E B(s)). ((s) is defined as small set ($ E S :ll s p c), which limits the attack power of the adversary and e is attack budget. Since directly generating & E (s)) is hard, the adversary learns to produce a Gaussian noise A with (A) less than 1, and we obtain the perturbed state through š = s + e. The victim takes action according to the observed 5, while true states in the environment are not changed. The perturbed policy is denoted as Toon Different from SA-MDP (Zhag al. 2020 RSA-MDP introduces R, which consistent with human preferences. The target of RSA-MDP is to solve the optimal adversary T which enables the victim achieve the maximum expected return over all states. Lemma [ shows that solving the optimal adversary in RSA-MDP is equivalent to finding the optimal policy in MDP M 10=(SAAA Á, R. P. 7), where Á S and P is the transition dynamics of the adversary.
METHOD
In this section, we introduce our method PALM, which leverages preference- based RL to achieve targeted attack against DRL agents. The idea of PALM, on the one hand, is to learn inten- tion policy as the learning target of the adversarial policy to tackle restricted exploration problem. On the other hand, PALM takes advantage of feedhack-efficient preference-bee hased RL method PEB-
3

---

Under review as a conference paper at ICLR 2023
BLE (Lee et al. a to avoid reward engineering. Also, introduce weighting function to improve the performance of the adversary and formulate PALM as a bi-level optimization algorithm. The framework of PALM is shown in Figure and detailed procedure is summarized in Appendix A
4.1 LEARNING INTENTION POLICY
PALM aims to find the optimal adversary that manipulates the victim's behaviors to be consistent with human intentions. However, the victim policy is pre-trained to complete a specific task, directly learning adversary suffer from exploration difficulty caused by the restriction of victim policy, making it hard to find an expected adversarial policy efficiently. Therefore, we introduce an intention policy Te which has unrestricted exploration space to guide adversarial policy training.
To achieve targeted attack and avoid reward en- gineering, inject human intentions into the in- tention policy via preference- based RL framework, which shown in Figure RL, the agent have no access to the ground-truth re- ward function. To learn reward function, humans provide preference labels between two trajectories of the agent and the reward function Fe learns to align with the preferences (Christiano et al. 2017).
leamingg from human preferences
In preference-based
Av
Formally, segment a of length k de- noted as sequence of states and actions (8e+1, ar St4k,, , , ac A human expert is required to give a label y of pair of segments (a", o') to indicate which segment is preferred. where {(0, 1), (1,0), (0.5, 0.5)). Following
(a(s)
replay bufter
Figure Diagram of preference-hased RI..
Bradley- Terry model (Bradley Terry 1952). a preference predictor is constructed in (T]
exp Σ, (sP, a!))
Polo > o'1
(1)
exp
(s) a.) exp Σ, (s)
where 0º > a² denotes d is preferred to 1. This predictor indicates the probability that a segment preferred is proportional its exponential return. Then, the reward function is optimized by aligning the predicted preference labels with human preferences through cross-entropy loss:
[(v)=
(() log [o > - + v(1) log Po o9]]
(2)
(a g)
where D is a dataset of triplets (a", (a",a* ,y,,y) consisting of segment pairs and human preference labels. By minimizing (2). obtain reward function estimator Fut which is used to provide estimated
rewards for agent learning via any RL algorithms. Following PEBBLE (Lee et al.) (2021a). we use an off-policy actor-critic method SAC (Haarnoja et al.) 2018) to learn well-performing policy. Specifically, the Q-function Q is optimized by minimizing the Bellman residual:
JQ(Ø)= E
(Qo(s), a.) - - -V(+))))) -C-
(Se, a,, Fr, St+1) 1) is the transition at time step / is the parameter of the target soft Q-function. The policy my is updated by minimizing 4
where ((s) = Earon [Q] - log To (ar
J (0) B.a log (a,)s)) (s a))
where is the temperature parameter. By learning an intention policy, PALM tackles restricted exploration problem and provides attack target for the following adversary training.
4.2 LEARNING ADVERSARIAL POLICY AND WEIGHTING FUNCTION
To make the victim policy perform human desired behaviors, PALM learns the adversary by mini- mizing the KL divergence between the perturbed policy Toa and the intention policy ιy. However, different states may have various importance to induce the victim policy to the target. To stabilize training process and improve the performance of the adversary, we introduce a weighting function he to re- weight states in adversary training.
4

---

Under review as a conference paper at ICLR 2023
We formulate PALM bi-level optimization algorithm, which alternately updates the adversarial policy Ta and the weighting function hn through inner and outer optimization. In the inner level, PALM optimizes with the importance weights outputted by weighting function h and opti- mizes w in the outer level according to the performance of the adversary. Intuitively, the adversary learns to approach the intention policy in the inner level, while the weighting function learns to im- prove the performance of the adversary by evaluating the performance of the adversary through a meta-level loss. The whole objective of PALM is:
min J=(aww))
(5)
s.t. a(w) arg min Can (a; w. 9).
Inner-level Optimization: Training adversarial policy wa- In the inner-level optimization, given the intention policy To and the weighting function hu, we hope to find the optimal adversarial policy
by minimizing the re-weighted KL. divergence between Tyoa and Tg in 6
Car(a; w 0) Eg (s) (Tuon (s) || TO (s))
(6)
where (s) the importance weights outputted by the weighting function hw Intuitively, the adversarial policy is optimized to make the perturbed policy be close to the intention policy, while hw assigns different weights to states of various importance. With the collaborative assistance of the intention policy and the weighting function, PALM efficiently learns an optimal adversarial policy.
Outer-level Optimization: Training weighting function h As for the outer-level optimization, need to find precise weighting function to balance the state distribution and assign proper weights to propel adversary learning. The weighting function is trained to distinguish the importance of states by evaluating the performance of the perturbed policy. Specifically, the perturbed policy Tooa is evaluated using a policy loss in 7 which is adapted from the policy loss in (4)
Jx (a(w)) = ES, E,,,-E,a (TTooo))
[nl Tuoa(w) (ar/s:) - (s: a:)]
(7)
where (w) denotes implicitly depends on w. Therefore, PALM calculates the implicit derivative
of (w))) with respect to w and finds the optimal w/ by optimizing (7). To make feasible,
we make an approximation of arg min, with the one-step gradient update. (8) obtains an estimated
arg minl with one-step updating and builds a connection between a and w:
d(w) a 7, Ca w, l)....
(8)
According to the chain rule, the gradient of the outer loss with respect w can be expressed as:
VaJr(á(w))a. Vodr(w)l,, = ES(s) Vuh(s)
where f(s) - (V (VJa(a(w))) VaLKL (Tvea (s) || Te())) and detailed derivation can be
found in Appendix B The key to obtain this meta gradient is building and computing the relationship between a and w. Obtaining the implicit derivative, PALM updates the parameters of the weighting
function by taking gradient descent with outer learning rate.
In addition, we theoretically analyze the convergence of PALM in Theorem and In Theorem I we demonstrate the convergence rate of the outer loss, i.e. the gradient of the outer loss with respect to w will convergence to zero. Thus PALM learns more powerful adversary using importance weights outputted by the optimal weighting function. In Theorem 2 we prove the convergence of the inner loss. The inner loss of PALM algorithm converges to critical points under some mild conditions, which ensures the parameters of the adversary can converge to the optimal parameters. Theorems and proofs can be found in Appendix D
EXPERIMENTS
In this section, we evaluate our method on several robotic simulated manipulation tasks from Meta- world (Yu et al. 2020) and continuous locomotion tasks from Mujoco (Todorov et al. al. 2012). Specif- ically, our experiment contains two essential phases. In the first phase, we verify the efficacy of the proposed method through two scenarios: navigation and opposite behaviors. Furthermore, we show the capability of our approach by fooling popular offline RL method, Decision Transformer (Chen jet al. (02T). into acting specific behaviors in the second phase. The detailed description of tasks used in experiments is provided in Appendix F
5

---

Under review as a conference paper at ICLR 2023
5.1 SETUP
Baselines. We include the existing evasion attack methods for comparison study the effectiveness of our approach.
Random attack: this is naive baseline that samples random perturbed observations via uni- form distribution.
SA-RL (Zhang et al. 2021 : this method learns an adversarial policy in the form of an end-to-
end RL formulation.
PA-AD (Sun et al. 2022): this method combines RL-based "director" and non-RL "actor" to find state perturbations, which is the state-of-the-ar adversarial attack algorithm against DRL.
PALM: our proposed method, which collaboratively learns adversarial policy and weighting function with the guidance of intention policy.
Implementation Settings. We compare PALM with existing adversarial attack methods, which at- tack the victim to reduce its cumulative reward rather than manipulate it. To achieve fair comparison, make simple adjustments for SA-RL and PA-AD to suit our settings in the experiments. In their original version, both of these two methods use the negative value of the reward obtained by the vic- tim to train an adversary. We replace with the same estimated reward function Fy as our method uses, which means they learn from human preferences. Following the settings in PEBBLE (Lee) al., 2021a), we use a scripted teacher that provides ground truth preference labels. More details of scripted teacher and preference collection can be found in Appendix F For the implementation of SA-RI and PA-AI we use the released official codebase. For fair comparison, all methods learned via preference-basee RL given the same number of preference labels. In the navigation scenario, we use 9000 labels for all tasks. In the opposite behaviors scenario, we use 1000 for Win- dow Close, 3000 for Drawer Close, 5000 for Faucet Open, Faucet Close and Window Open, 7000 for Drawer Open, Door Lock and Door Unlock. Also, to reduce the impact of preference- additionally add oracle versions of SA-RL and PA-AD, which uses the ground-trus rewards of the targeted task.
RL,
We use the same experimental settings (i.e., hyper-parameters, neural networks) concerning reward learning for all methods. We quantitatively evaluate all methods by comparing the success rate of final manipulation, which is well-defined in Meta-world (Yu et al. (020) for the opposite behaviors scenario, and we rigorously design for the navigation scenario. As in most existing research (Zhang et al. 2020 2021 Sun et al. 2022h, we consider using state attacks with & norm in our experiments, and we report the mean and standard deviation across ten runs for all experiments. We also provide detailed parameerr settings and implementation details in Appendix
5.2 MANIPULATION ON DRL AGENTS
We study the effectiveness of our method compared adversarial attack algorithms, which are adapted to our setting with minimal changes. Specifically, we construct two different scenarios on various simulated robotic manipulation tasks. Each victim agent is well-trained for a specific manipulation task.
Scenarios Navigation. In this scenario, expect the robotic to reach target coordinates instead of completing the original task. Figure 4 shows the training curves of baselines and our method eight manipulation tasks. It shows that the performance of PALM surpasses that of the baselines by large margin based on preference labels. To eliminate the influence of preference- based RL and further demonstrate the advantages of PALM, we additionally train the baseline meth- ods with the ground-truth reward function and denote them as "oracle". We notice that the per- formance of SA-RL (oracle) greatly improves on several tasks over the preference- However, PALM still outperforms SA-RL with oracle rewards on most tasks. These results demon- strate that PALM enables the agent to efficiently learn adversarial policy with human preferences. We also observe that PA-AD is incapable of mastering manipulation, even using the ground-truth rewards.
version.

---

Under review as a conference paper at ICLR 2023
PALM
PA-AD (oracle)
PA-AD
SA-RL (oracle)
SA-RL
Random
100
100
100
100
Success Rain (%)
Success Rate (%)
Success Rato (%)
75
75
75
Success
0.0
0.2
0.8
0.0
022
0.4
0.6
0.8
0.2
0.5
0.8
1.0
0.0
0.2
Environment Steps
Environment Steps
Environment Steps x1o
Environment Steps
(a) Door Lock
(b) Drawer Open
(c) Faucet Open
(d) Window Open
100
Success Rain (%)
Success Rate (%)
Succers Rate (%)
75
Raln
0.0
0.0
0.2
0.8
0.8
0.0
0.2
0.4
0.8
0.8
1.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.8
Environment Steps
Environment Stepe x10"
Environment Steps x11*
Environment Stepe x100
(c) Door Unlock
(f) Drawer Close
(g) Faucet Close
(h) Window Close
Figure 4: Training curves of different methods various manipulation tasks in the navigation scenario. The solid line and shaded area denote the mean and the standard deviation of success rate, respectively, over runs.
PALM
PA-AD (oracle)
PA-AD
SA-RL (oracle)
SA-RL
Random
100
100
Suconcs Rate (96)
Success Rate Rate (%)
Rate (%)
Succers Rate (1))
75
75
50
25
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1,0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
Environment Steps
Enwironment Staps x10"
Environment Steps x10*
Envrronmet Staps x10"
(a) Lock
(b) Drawer Open
(c) Faucet Open
(d) Window Open
100
100
100
100
Sucaasa Rate (%)
Success Rate (%6
Success Rate (56)
75
78
75
Rate
50
80
50
50
25
26
25
0
1.0
Environment Steps x1"
Environment Stnps x10"
Environment Steps x10*
Environment Staps x
(e) Door Unlock
(f) Drawer Close
(g) Faucet Close
(h) Window Close
Figure 5:
Training curves of all methods on various tasks in the opposite behaviors scenario. The solid line and shaded area denote the mean and the standard deviation of success rate over ten runs.
Scenarios on Opposite Behaviors. In the real world, robotic manipulation has good application val- ues. Therefore, we design this scenario to quantitatively evaluate the vulnerability of these agents that masters various manipulation skills. Specifically, we expect each victim to complete the op- posite task under the attack of the manipulator. For example, the victim which masters the skill of opening windows will close windows under targeted attack. As shown Figure 5 PALM presents excellent performance and marginally shows obvious advantages over baseline methods on all tasks. The result again indicates that PALM effective for a wide range of tasks and can efficiently learn adversarial policy with human preferences.
5.3 MANIPULATION ON THE POPULAR OFFLINE RL AGENTS
In this experiment, we show the vulnerability of offline RL agents and demonstrate PALM can fool them into acting human desired behaviors. As for the implementation, choose some online

---

Under review as a conference paper at ICLR 2023
(a) Cheetah-Ran Backwards
(h) Walker-Stand on One Foot
(c) Cheetah-90 Degree Push-up
(d) Walker-Dance
Figure Human desired behaviors behaved by the Decision Transformer under the attack of PALM.
models as victims, which are well-trained by official implementation with D4RL. We choose two tasks, Cheetah and Walker, using expert-level Decision Transformer agents as the victims. As shown in Figure 6 Decision Transformer shows exploitable weaknesses and is misled to perform human desired behavior instead of the original task. Specifically, under the adversarial manipulation, the Cheetah agent runs backwards quickly in Figure 6aa and does 90 degree push-up in Figure 6cc The Walker agent stands on one foot for superior balance in Figure 6b and dances with one leg lifted in Figure 6d The results show that PALM can manipulate these victims to act behaviors consistent with human preferences and embodied agents are extremely vulnerable these well- trained adversaries. We hope this experiment can inspire future work on the robustness of offline RL agents and embodied Al.
5.4 ABLATION STUDY
Contribution of Each Component. We conduct additional experiments to investigate the effect of each component in PALM on Drawer Open, Drawer Close for the navigation scenario and on Faucet Open, Faucet Close for the oppo- site behavior scenario. PALM contains three critical com- ponents: the weight function hw. the intention policy no, and the combined policy. Table [ shows that the intention policy plays essential role in the PALM. As shown in Figure 7 the intention policy can mitigate exploration dif- ficulty caused by the restriction of victim policy and improve the exploration ability of PALM leading to a better adver- sary. We also observe that the combined policy balances the discrepancy between and Tuso on the state distribution and improves the adversary's ' performance. In addition, can economically train the weighting function to distinguish state importance by formulating the adversary learning as bi-level optimization. It can further improve the asymptotic performance of PALM. These empirical results show that key ingredients of PALM are fruitfully wed and contribute to the PALM's success.
Figure 7: visualization
explo- ration space of PALM (red) and PALM without intention policy (blue). The green point denotes the start and the yel- denotes target position.
Table I: Effects of each component. The success rate on four simulated robotic manipulation tasks from Meta-world. The results the average success rate across five runs.
Drawer Open (Navigation)
Drawer Close
Faucet Open (Opposite)
Faucet Close (Opposite)
(Navigation)
PALM
97.2%
86.8%
97.2%
95.5%
PALM without h
93.2%
74.8%
89.4%
86.2%
PALM without no
13.8%
11.6%
0.0%
4.2%
PALM without combination
38.0%
22.3%
51.7%
71.8%
httss::/huugiingffce..ooo
8

---

Under review as a conference paper at ICLR 2023
1.0
0.4
Noralized Weight
0.2
50
005
25
25
-0.2
-0.4
02
100
Average Retur
(a) I-SNE Visualization
(b) Weight Visualization
(c) Heat Map Visualization
Figure 8: (a) A visualization of the weights of trajectories of different qualities is collected by five different policies. (b) Trajectory weights generated by the weighting function from different policies extracted and visualized with t-SNE. (c) A heat map showing the weight distribution and the trajectory of the perturbed agent in 2D coordinates. The red point denotes the start position and the yellow star indicates the targeted position.
To verify the restricted exploration problem, visualize the exploration space of PALM and PALM without intention policy. Figure shows that the intention policy significantly improve the explo- ration ability of PALM.
Effects of the Weighting Function. To further understand the role of the weighting function pro- posed in Section 4 we conduct experimental data analysis and visualization from multiple perspec- tives. Five perturbed policies uniformly sampled with performance increase sequentially before PALM convergence. For each policy, we roll out 100 trajectories and obtain the trajectory weight vectors via the weighting function. By leveraging the technique t-SNE (van der Maaten & Hinton, 2008), the weight vectors of different policies are visualized in Figure Sal From the figure, we can clearly observe clear boundaries between the trajectory weights of various policies, suggesting that the weighting function can distinguish trajectories of different qualities. In Figure [Bb] the darker color indicates trajectories with higher success rates of manipulation. The result shows that the weighting function gives higher weights to better trajectories for improving the adversarial policy performance. To further illustrate the effect the weighting function, we present a heat map of the weight distribution in 2D coordinates and annotate part of the trajectories of the perturbed policy. As Figure Se shows, the weighting function scores the surrounding states trajectories from the perturbed policy higher, especially in the early stage before reaching the target point.
Extensive experiments conducted to analyze and discuss the impact of feedback amount and attack budgets on the performance of PALM the Appendix G
CONCLUSION
In this paper, we propose PALM, preference- based adversarial attack approach against DRL, which mislead the victim to perform desired behaviors of adversaries. PALM involves adversary adding imperceptible perturbations on the observations of the victim, an intention policy learned through preference-basee
RL for better exploration, and weighting function to identify essential states for the efficient adversarial attack. We analyze the convergence of PALM and prove that PALM converges critical points under some mild conditions. Empirically, we design two sce- narios on several manipulation tasks of Morl, and the results demonstrate that PALM out- performs the baselines under the targeted adversarial setting. We further show embodied agents' vuinerability by attacking Decision Transformer on some Mujoco tasks. For future work, we con- sider: (1) further improving the attack efficiency by enhancing the utilization efficiency of human preference and (2) extending the observation space of the victim to the high-dimensional inputs, such images and natural language.
ETHICS STATEMENT
Preference-based RL provides effective way learn agents without carefully designed reward function. However, learning from human preferences means humans need to provide labeled data which inevitably has biases introducing systematic error. There are possible negative impacts when

---

Under review as a conference paper at ICLR 2023
malicious people attack other policies using our methods. However, our approach also makes other researchers aware the vulnerability of policies for Al safety.
REPRODUCIBILITY STATEMENT
The details of experiment settings are provided in Section 4 We provide detailed proofs of theoret- ical analysis in Appendix D more detailed description and implementation setting can be found in Appendix F Meanwhile, we present the link of our source code and videos in the abstract.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov. Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as I can, not as say: Grounding language in robotic affordances. CoRR, 2022.
Adith Boloor, Karthik Garimella, Xin He, Christopher Gill, Yevgeniy Vorobeychik, and Xuan Zhang. Attacking vision-based perception in end-to-end autonomous driving models. Journal of Systems Architecture, 110: 101766, 2020.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Prasanth Buddareddygari. Travis Zhang, Yezhou Yang, and Yi Ren. Targeted attack on deep rl-based autonomous driving with learned visual patterns. In International Conference Robotics and Automation (ICRA), pp. 10571-10577, 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances Neural Information Processing Systems (NeurIPS), volume 34, pp. 15084-15097. Curran Associates, Inc., 2021.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), volume 30. Curran Associates, Inc., 2017.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. CoRR, abs/2206.08853. 2022.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Ad- versarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2020. URL https: http:////ppeeeie..emmm net fest/fooummid HUGEMpVEWB
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1):1-12, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Muchine Learning (ICML), volume 80, pp. 1861-1870, 2018.
Sandy H. Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. In International Conference on Learning Representations (ICLR), 2017.
10

---

Under review as a conference paper at ICLR 2023
Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Targeted attacks deep reinforcement learning agents through adversarial observations. CoRR, abs/1905. 1222,, 2019. URL http:
//axxiv.rrg/aas////......aa 111..1911222.2.111 12282
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems (NeurlPS), volume 31. Curran Associates, Inc., 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedhack-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning (ICML), volume 139, pp. 6152-6163, 2021a.
Xian Yeow Lee, Yasaman Esfandiari, Kai Liang Tan, and Soumik Sarkar. Query-based targeted action-space adversarial policies on deep reinforcement learning agents. Proceedings of the ACM/IEEE 12th International Conference Cyber-Physicul Systems, pp. 87-97, 2021b.
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in mreference-based reinforcement learning. In International Conference Learning Representa- fions (ICLR), 2022.
Yen-Chen Lin, Zhang- Wei Hong, Yuan- Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. In IJCAI, pp. 3756-3762, 2017. URL https:/::://// /ttps::/ddiiio otw/10..rg11... 24963/1 2017/525
Siqi Liu, Guy Lever, Zhe Wang. Josh Merel. S. M. Ali Eslami, Daniel Hennes, Wojeiech M. Czar- necki, Yuval Tassa, Shayegan Omidshafici, Abbas Abdolmalcki, Noah Y. Sicgel, Leonard Hasen- clever, Luke Marris, Saran Tunyasuvunakool, H. Francis Song, Markus Wulfmeier, Paul Muller, Tuomas Haarnoja, Brendan Tracey, Karl Tuyls, Thore Graepel, and Nicolas Heess. From motor control team play in simulated humanoid football. Science Robotics, 7(69)cebb0235, 2022.
Julien Mairal. Stochastic majorization minimization algorithms for large-scale optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 26, 2013.
mngiorzzationmmiiiizzaoo
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction poyssccaly-pllusiiie perturbations. In International Con- ference Intelligent Robots and Systems (IROS), pp. 9932-3939, 2017.
Yu Nesterov. Introductory lectures on convex programming. 1998.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for (eedback-efficiene preference-bas reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. International Conference on Autonomous Agents and MultiAgent Systems (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial rein- forcement learning. In International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pp. 2817-2826. PMLR, 06 Aug 2017.
Scott E. Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards. Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. generalist agent. CoRR, 0ts//2205006155, 2022.
11

---

Under review as a conference paper at ICLR 2023
Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest emy? towards optimal and efficient evasion attacks in deep RL. In International Conference on Learning Representations (ICLR), 2022. URL https: http:///neeeeeeeeee/eeeeeeeee
JN2kFbJvvI
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems (IROS), pp. 5026-5033, 2012.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 986):2577-260 2008.
Chaowei Xiao, Xinlei Pan, Warren He, Jian Peng. Mingjie Sun, Jinfeng Yi, Mingyan Liu, Bo Li, and Dawn Song. Characterizing attacks on deep reinforcement learning. arXiv preprint DrXiv:1907.090 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learn- ing. In Conference on Robot Learning (CoRL), volume 100 of Proceedings of Machine Learning Research, pp. 1094-1100. PMLR, 2020.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsich. Robust deep reinforcement learning against adversarial perturbations on state observations. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 21024-21037. Curran Associates, Inc., 2020.
Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsich. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations (ICLR), 2021.
12

---

Under review as a conference paper at ICLR 2023
A
THE FULL PROCEDURE OF PALM
The Combined Policy. Although guides adversarial policy learning, the discrepancy between Ty and Trea on the state distribution leads to inefficiency. To handle this issue, we design a strategy to construct the behavior policy to collect transitions in our practical implementation. Inspired by Branched rollout (Janner et al. 2019), we combine the intention policy and the perturbed policy Toon, where π 1:h 1-/ π 1:H A+1:H ~ (O, H) and H is task horizon. The combined
policy collects data and stores into the replay buffer during learning.
We provide detailed procedures of our proposed method in Algorithm 1 PALM is implemented based on a popular preference-based RL algorithm PEBBLE (Lee et al. 2021a).
Algorithm PALM
Input: fixed victim policy Tu, frequency of human feedback K, outer loss updating frequency
M, task horizon H
1: Initialize parameters of Qè, Fg. &a and hw
2: Initialize B and To with unsupervised exploration
3: Initialize preference data set D +-
4: for each iteration do
5: if episode is done then
Construct the combined policy
6:
(O,
π 1:h = Poon and 1 Tg 1+1:
7:
8:
end if
9:
Take action 04 ~ and collect 8++1
10:
Store transition into dataset B B U (8t, at, (8)),, 8441)
11:
if iteration K = then
for each query step do Sample pair of trajectories (a", o²)
Query preference
13:
Query preference from manipulator
15:
Store preference data into dataset D + D
end for
17:
for each gradient step do
> Update reward model
18:
Sample batch ((o a² y) from Optimize 2 to update Pv
19:
20:
end for
21:
end if
22:
for each gradient step do
23:
Sample random mini-batch transitions from B Optimize n,, minimize (6) with respect to a end for
24:
b Inner loss optimization
26:
if' iteration % M = 0 then
Sample random mini mini-batch transitions from B Optimize h: minimize 7 with respect to
28:
Outer loss optimization
end if
30:
Update Qo and me according to and (4 respectively. end for
31:
Output: adversarial policy aaa
13

---

Under review as a conference paper at ICLR 2023
B DERIVATION OF THE GRADIENT OF THE OUTER-LEVEL LOSS
In this section, we present detailed derivation of the gradient of the outer loss J/ with respect to the parameters of the weighting function w. According the chain rule, derive that
V (á(W)))
aJn(a(w))
dāi(w) dā(ww) Ow we
(J-( @ār(w) ah(s:w)
Dā(w) la wh(
Ow
(10)
= '/t {J)(((())
0DLL (Tvo To(s))
ah(s;w)
@d((w)
da
Ew
B
(ān(W(
0DRL. (Tuaa(s) TT(s))
@h(s; w)
dā(w)
á
aa
Ow
8~B
For brevity of expression, we let:
f(s) (.)(((()) T 0DRl (Tvoa TT(S))
(11)
aā(w) The gradient of outer-level optimization loss with respect to parameters w is:
dā
@h(s:w)
VaJa - Ef(s)
f(s)
(12)
dw
C CONNECTION BETWEEN RSA-MDP AND MDP
Lemma 1. Given a RSA-MDP = A,, P.7.) and fixed victim policy there exists MDP M A, R, P,7) such that the optimal policy of M equivalent the optimal adversary Ta in RSA-MDP given a fixed victim, where A S and
P(s'(s,a) Σ (a ( fors, E S and â E Ā.
aEA
D THEORETICAL ANALYSIS AND PROOFS
D. THEOREM 1: CONVERGENCE RATE OF THE OUTER LOSS
Lemma (Lemmna 1.2.3 Nesterov (1988) If function f(x) is Lipschitz smooth on R with constant L, then Vx, y E IR, have
1 f(x) (())) (y - x) [0==x]=
2 l1 - x||²
(13)
Proof. Vz, y R", have
f(v) f(x)(( [ f(c+r(y-z)) z)dr
(14)
= + +)+ fflefI((ttttaapt -)) f()) z)dr.
Then we can derive that
f(() - f(x) - fx) T x)| (() (v r)dr
L
[ + 7(V x)) -())) (v x)) s){dt
(15)
/ llf + r(V - x)) fx(x) 1ly 2||d7 dr
/'" TL llv x11² dr 2 || x|
14

---

Under review as a conference paper at ICLR 2023
where the first inequality holds for [." fz))dz & S/2 (f() dr, the second inequality holds for Cauchy-Schwarz inequality, and the last inequality holds for the definition of Lipschitz smoothness.
Theorem 1. Suppose Ja is Lipschitz-smootte with constant L, the gradient of Jx and Lan bounded by Let the training iterations be T, the inner-level optimizution learning rale at min 11 7 for some constant 61 > 0 where o < 1. Let the outer-level optimization learning rate Bt =
T and Ei=1 Be I co, E B S co. The m S for some constant c2 > 0 where C2 S L convergence rate of Jr achieves
min ISST
E
so
VT
(16)
Proof. First,
(1((4+2(4+11)) - ( (J=(()42((W441) - (ārt (wx+1))) + {J (àr+ (wx41)) J (az+1
(17)
Then we separately derive the two terms of 17 For the first term,
I ((142((w++1) - (
V (azz( (02+2((04+)2)))) - det1 (w++1)) + - | uu+1 (11+1))12
2 $ VaJa ( (w)++))) 110))2(((422)) de+1 (Wr 1)]1
1
+ |(b442(Wr+1) - (w+1lW++1)1 (18)
5p-11-me+1 VaLam (a 11 Varan(
L Sm++pp 2 102
where (e+2(we+1) de41 (we-1) - 1 VaCar(C441). the first incquality holds for Lemma 2 the second inequality holds for Cauchy-Schwan inequality, the third inequality holds for S p, and the last inequality holds for a S p. It can be proved that the gradient of w with respect to Jn Lipschitz continuous and we assume the Lipschitz con- stant is L. Therefore, for the second term,
||VaJ (a++((4++(4441))
J (((+1(WW)+1) (x (wi)
(wr++ <V ( (wi+1 wi) + L || well
L
(19)
3.V. V (ār42( VaJ (ārt (wr+) +
2
L 2
=- (B:
Vw. ( (2+++
where W++1 wi
-3, VaJa (w:)), and the first inequality holds for Lemma 2
There-
fore, 17 becomes
L -3,
L
Jg ((642((W(1)) - Jn (ā++1(we)) & nht1p²
2 Ti (3.
2
(20)
Rearranging the terms of 20 we obtain
1-LL) (g, - 2
LB8 [6 s (=(6++1 J J=(0442(4++1)) + m41pp + 12
(21)
15

---

Under review as a conference paper at ICLR 2023
Then, sum up both sides of
21
T
L
(B.
(B.
(a
2
1=1
T
L
(J Jx (wT++))) +
Σ
(1--102
+ 2 )
(22)
1=1
<Jz((2(w))) +
Σ
T
11 + - 12
L
1=1
Therefore,
min E I/V (arel (wx)))) 15tST
-1= (Be 4Bf
V (
<
Σt=1
2
T
1
2J (J2(22()))) +
Σ 1=1
12 + Ink-1
(23, - Li88)
1=
1
2J (az(wi)))
Σ
net 0² Lopes))
2 Br
(23)
5
(az((())))) + +T 22 L)
TB.
2J (āz((W))))
+ 11+1p² L)
TB:
B.
2 (ā2(w11) max
VT
VT + min(1, 9 7 max
(2 L)
T
02
02
2J (á2(w))
c)p²(2 (22+14 L))
CTVT
c2VT
where the second inequality holds according
to
22 the third inequality holds for
D (233- T 2 Ei= 3.
Li
D.2 THEOREM 2: CONVERGENCE OF THE INNER LOSS
Lemma 3. (Lemma A.5 in Mairal 2013 Let (an)n>1, (bn)n21 be two non-negative real sequences
such that the series _m= an diverges, the series Ex=1 anbn converges, and there exists > such that lb,+1 bnl Can Then, the sequence (bn) converges to 0.
Theorem 2. Suppose Jn is Lipschitz-smann smooth with constant L, the gradient of Jx and Lun bounded by p. Let the training iterations be T, the inner-level optimization learning rate т) min(1, } for some constant ei > O where T < 1. Let the outer-level optimization learning rate Be. =
min 2. } for some constant 02 > 0 where 02 S LL, and Eiz Be S 0o, Ei= Bi S oc. Lan achieves
lim E (on::0((a:;::))1 =0
(24)
Proof. First,
Lan(a441(0441) Lan (ar;wz)
(25)
= Lan ([Con(0441:4441) - Can (3++1)w0) + [Ca (Ca((((1))e)) -
16

---

Under review as a conference paper at ICLR 2023
For the first term in
25
Lau (on - (as(1w)) L
VVahan (ort1((04155 (we+1 we) || well
(26)
L
- Waan (1t+1; (are (wi)) + Va (w:))ll
2
where w1+1 wi 3e V VeJ++ ( (w))), and the first inequality holds according to Lemma
For
the second term in 25
Cau Cml(a41www) Lau (axiws)
(((k) - aa) ce)+ + T. |ac41-auk
<V Vahanlariw)) (a++1
Ti Vafaular;))) V Vafaula;:u)) + Laf
(27)
2 [[
T
(n
V (oy;
2
where 04+1 Therefore, 25 becomes
ag mMp V (ar and the first inequality holds according to Lemma 2
Lat (1 Lan (atjur))
L3g 2 (
S 3e WwWwaanlrtiii VaJ (ace(ac(() +
T
(28)
Lai
(np 2 (
Taking expectation of both sides of 28 and rearranging the terms, we obtain
17, [ VV + B.
V || (
L
[aan(ar; - E [[
2 [W (29)
LuE E Carlwwari
Summing up both sides of from t 110 00.
Σ E
Vl (ort | (oetta
T,8
<E Dan (a1;
lim E [Lm 1000
+ Σ
1=1 2 [
(30)
Lufi E
L(m + +0ppp²
+ E [aan s oc,
2
1=1
where the second inequality holds for D vi ( < p. Since
Di= of < xo, [.Lan(rrw)) s p.
B.E (are (oa (at+) < Lp D Σ
Br S
(31)
1=1
Therefore, we have
||V
Lav (an: < <0.
(32)
17

---

Under review as a conference paper at ICLR 2023
Since [[|||] [[
|[b|||| - ||a billa b11, can derive that
E
V (a
E
V San [z (0) w))
E
(ar
V San (ou;
(IV 11 Lau (or;w))))
] |i0 Va
<B
<B
VaLau ( || V Va (
<B
( a Lan (1e4 + ( V (ay
|| <2LpB
-
S2Lpm
[[Va (uJr(ect2(u))ll
S2Ipne
V (ort)(()))))))
32L 222
22v2Lp2mOe 113t.
(33)
Since Ei= !1t c0, according to Lemma 3 we have
lim 81000
E
| [a =.
(34)
E DETAILS OF PBRL
In this section, we present details of the scripted teacher and preference collection. It is crucial part of the PbRL, and PALM follows these settings as [ee et al. (2021a).
Scripted Teacher. To evaluate the performance systemically, useful way is to consider scripted teacher that provides preferences between a pair of agent's 's trajectory segments according to the oracle reward function. Leveraging the preference labels from the human teacher is ideal, while it is hard to evaluate algorithms quantitatively and quickly. Specifically, the scripted teacher can immediately provide ground truth rewards based on the state s and action is a function designed to approximate the human's 's intention.
Preference Collection. During training, we need to query human preference labels regular inter- vals. It samples a batch of segment pairs and calculates the cumulative reward of each segment with rewards provided by the scripted teacher. For a specific segment pair, human prefers the segment with larger cumulative reward. The segment with larger cumulative reward is labelled with 1, and the smaller one labelled with 0. As for the computational cost, we assume that M preference labels required, the segment length is N in run, and the time complexity is ((MN). However, is negligible compared with adversary training, which involves complex gradient computation.
F EXPERIMENTAL DETAILS
In this section, we provide a concrete description of our experiments and detailed hyper- perrmpeareeeers of PALM. For each run of experiments, we run on GeForce RTX 2080 Ti GPUs for training.
F.I TASKS
We conduct experiments eight robotic manipulation tasks from Meta-world (Yu al. 2020) in phase one and two locomotion tasks from Mujoco (Toooov et af. (0022) in phase two. The tasks we used are:
Meta-world
Door Lock: An agent controls simulated Sawyer arm lock the door.
18

---

Under review as a conference paper at ICLR 2023
Door Unlock: An agent controls simulated Sawyer arm unlock the door.
Drawer Open: An agent controls simulated Sawyer arm to open the drawer a target position.
Drawer Close: An agent controls a simulated Sawyer arm to close the drawer to a target posi- tion.
Faucet Open: An agent controls simulated Sawyer arm to open the faucet to target position.
Faucet Close: An agent controls simulated Sawyer arm to close the faucet to a target position.
Window Open: An agent controls simulated Sawyer amm to open the window to target position.
Window Close: An agent controls simulated Sawyer arm to close the window to target position.
Mujoco
Half Cheetah: A 2-dimensional robot with nine links and eight joints aims to learn forward (right) as fast as possible.
Walker: A 2-dimensional two-legged robot aims to move in the forward (right).
F.2 HYPER-PARAMETT SETTING
We choose PEBBLE For SA-RL Zhang et al. 2021 we keep the same parameter setting with parameter and use the same neural network structure as ours. The detailed hyper-parameters of SA- RL is shown in Table 3 For PA-AD (Sun et al. 2022). all hyper-parameters are the same as those of
SA-RL.
Table 2: Hyper-papne parameters of PALM for adversary training.
Hyper-parameter
Value
Hyper-parameter
Value
Number of layers Learning rate Length of segment
Hidden units of each layer
256
0.0003
Batch size
1024
50
Number of reward functions
3
Frequency of feedback
5000
Feedback batch size
128
Adversarial budget
0.1
(31, 82)
(0.9, 0.999)
Table 3: Hyper-paramette of SA-RL for adversary training.
Hyper-parameter
Value
Hyper-parameter
Value
Number of layers
Hidden units of each layer
256
Learning rate Length of segment Frequency of feedback
0.00005
Mini-Batch size Number of reward functions
32
50
5000
Feedback batch size Entropy coefficient Discount
128
Adversarial budget
0.1
0.0
Clipping parameter
0.2
0.99
GAE lambda
0.95
KL divergence target
0.01
F.3 VICTIM SETTING
Our experiment consists of two phases. In the first phase, various simulated robotic manipu- lation tasks from Meta-world, We have two OpenAl Gym MuJoCo continuous control environments in the second phase.
Meta-world, We train the victims on Worldworld by utilizing SAC (Haarnoja et al. 2018) with the original fully connected neural network policy. Detailed hyper-parameters are shown Table
19

---

Under review as a conference paper at ICLR 2023
Table Hyper-parameters of SAC for victim training.
Hyper-parameter
Value
Hyper-parameter
Value
Number of layers
Initial temperature
0.1
Hidden units of each layer
256
Optimizer
Adam
Learning rate
0.0001
Critic target update freq
Discount ?
0.99
Critic EMA
0.005
Batch size
1024
(31, 82)
(0.9, 0.999)
Steps of unsupervised pre-trate pre-training
9000
Discount y
0.99
Mujoco. We directly utilize the well- well-trained model for demonstrating the vulnerability of the Deci- sion Transformer. Specifically, the Cheetah agen and the Walker agen with expert-level.
G EXTENSIVE EXPERIMENTS
Impact of Feedback Amount. We investigate the performance of PALM with differ- ent preference labels. Table 5 shows the results of all methods with various num- bers of labels: {3000, 5000, 7000, 9000} on Drawer Open for the navigation scenario and {1000, 3000, 5000, 7000} on Faucet Close for the opposite behavior scenario. From the experi- mental results in Table 5 we conclude that providing sufficient human feedback learns strong adversary and stabilizes the attack success rate. We notice that the performance of PALM improves with the increase of human preference labels, indicating that the number preference labels has an essential impact on adversary learning, However, the performance of SA-RL and PA-AD is poor even with enough human feedback and PA-AD completely fails in navigation scenario. The reason is that the exploration space of these two methods is limited by the fixed victim policy, while PALM achieves better exploration by introducing an intention policy.
Table 5: Success rate with various amount of preference labels on Drawer Open for the navigation scenario and Faucet Close for the opposite behavior scenario. We report the mean and the standard deviation of the success rate over 30 episodes.
Environment
Feedback
PALM (ours)
PA-AD
SA-RL
3000
65.7% 1 37.1%
0.0% + 0.0%
8.3% 13.2%
Drawer Open (navigation)
5000
86.7% + 18.1%
0.0% + 0.0%
21.3% +: 18.9%
7000
95.7%
13.6%
0.0%
0.0%
28.0% 1: 28.1%
9000
97.0%
6.9%
0.0%
0.0%
13.0% + 18.5%
1000
69.7%
35.2%
16.7% + 9.4%
2.0% + 6.0%
Faucet Close
3000
79.0% + 16.2%
29.0% + 14.0%
6.0% + 11.7%
(opposite behavior)
5000
95.3% + 9.2%
21.3%
12.8%
3.3% 12.7%
7000
95.3% 7.6%
22.7%
12.4%
4.0% 1: 7.1%
Impact of Different Attack Budgets. We also analyze the effect of the attack budget. conduct additional experiments with various attack budgets:
For further understanding, (0.05, 0.075, 0.1, 0.15) on the Drawer Open and {0.02, 0.05, 0.075, 0.1} on Faucet Close for these two scenarios. In the Figure 9 tack budgets. As the experimental results show, the performance of all methods improve as the adversarial budget increases.
we report the performance of baseline and PALM with various at-
https:://ougginngfacc.coo
1ew.dd-eee-rrrr--- expert
20

---

Under review as a conference paper at ICLR 2023
PALM
PA-AD
SA-RL
100
100
100
Drawer Open
Success Rate (%)
Success (%)
Succese Rate (%)
Success Rate (%)
75
75
75
60
80
60
60
25
28
26
0.0
0.2
0.4
0.8
0.0
0.2
0.4
0.6
0.8
100
0.0
1,0
0.0
0.2
0.4
0.8
0.8
Environment Steps xx10
Environment Stops 10
Environment Steps xx10
Environment Steps x100
(a)
0.05
(b) 0.075
(c) 0.1
(d)
0.15
100
100
100
100
Faucet Close
Success Rale (%)
Success Rate (N)
Success Rais (%)
Success Rate (%)
75
75
75
75
50
50
80
25
28
28
0.0
0.
0.0
x110
Environment Steps 1010"
xx10"
Steps x10"
(e) 0.02
(f) 0.05
0.075
(h) 0.1
Figure 9: Training curves of success rate with different adversarial budgets on Drawer Open for the navigation scenario and Faucet Close for the opposite behavior scenario. The solid line and shaded area denote the mean and the standard deviation of the success rate across five runs.
21


Reviews:
[{"review_id": "ddmA68SeR-F", "reviewer": "Reviewer_2Rf7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a way to run targeted adversarial attacks against Reinforcement Learning agents. The threat model consist in modifying the state perceived by the agent in order to trigger specific actions. The method is to both find an optimal \"intention policy\" that achieves the target behavior, then try to match it with an adversarial policy in which states are perturbed. The latter is formulated as bi-level optimization. The authors provide a training procedure for that problem and show that under some conditions it will converge to a critical point. Experimental results show that this attack can achieve the target behavior against multiple RL agents, and outperforms multiple baselines by large margins. Online examples demonstrate the attack's success in Meta-world.", "summary_of_the_review": "The paper provides what seems to be a strong work with very good results. I think a subpar paper outline focusing to much on technicality and not enough on motivations undermine the work. I tend to prefer the work accepted, but it would benefit from some rewriting.", "strength_and_weaknesses": "**Strengths**: The addressed problem is challenging. The proposed solution is detailed and convincing: it consists of multiple steps, leverages several optimization methods, and is expressed in a complete formalism. This solution also seems largely novel and does not simply extend past work. The experimental setup seems complete and the results very strong. Online demos help grasping better while the paper accomplishes on practical examples.\n\n**Weaknesses**: One weak point is clarity. As a reader very familiar with adversarial attacks but less so with Reinforcement learning, understanding the main points of the paper was difficult. Combining standard RL formalism, intention policy, adversarial policy, victim, multiple rewards and optimizations, etc. lead to a heavy formalism. Given the complexity of the work technicality is necessary, and the authors do provide a diagram of the pipeline, but I think clarity could be improved. Parts of the method remain unclear to me: for instance, I am not sure whether the intention policy is learned before, or jointly to the weighing function and adversarial policy (Figure 2 says jointly, but the method seem to imply before). \n\nAnother point I am not sure about is the attack budget: it is mentioned in appendix, and the introduction says that perturbations are imperceptible, but the actual method does not make clear how that budget is handled or why it is necessary. Overall I'd suggest having a full part dedicated to the outline of the method, keeping technical aspects to a minimum, and moving the rest (theoretical results, learning rates, etc) somewhere else. More detailed diagrams for subparts of the pipeline would also be useful.\n\nAnother issue is the limited amount of motivation the authors provide for their work. One sentence in the introduction says that studying adversarial attacks is \"crucial\", but why some threat models rather than others? With an attack like PALM what could an attacker concretely do in the real world? Motivating or defining informally specific aspects of the work could be useful as well. What is an \"imperceptible perturbation\"? What imperceptible means on image or speech is clear, but on RL states it is harder to grasp. When it comes to theoretical results, they come with so-called \"mild\" conditions. Are those conditions typically met? Having convincing answer to those questions would make the paper stronger.\n\nA final (more minor) point: in absence of competing targeted attacks the authors make straightforward modifications to untargeted attacks to get baselines. That this would lead to weak baselines was predictable, and the comparison isn't too fair. Providing it is better than not, but I do not think it is an essential part of the paper, and it could be moved to an appendix without loss.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "_r3GVw3Rqny", "reviewer": "Reviewer_bmdX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of using preference based RL to adversarially attack Deep RL policies.  They propose a novel intention policy as a way to stabilize training which leads to improved performance in this regime", "summary_of_the_review": "I'm recommending weak rejection of this paper since it seems like the empirical results do not justify the conclusion, due to the intention policy confounding experiments on preference based RL, and preference based RL not being properly evaluated on its own.\n", "strength_and_weaknesses": "Strengths:\n* The novelty of the idea of using Preference based RL for adversarial attacks.\nit is often hard to define what a successful attack is formally, but you know it when you see it, so this could be a promising direction\n\n* The intention policy seems promising as an approach for solving exploration problems in adversarial attack domains\n\nWeaknesses:\n* The paper seems to not test the core claim: that preference Based RL is useful for adversarial attacks\n\nMost of the tasks are tasks for which a hard-coded reward function does exist(maybe all of the tasks, the setup is a bit unclear).  As such, they can't be effectively used to show that preference based RL is useful for adversarial attacks since the problem of specifying the task or solved.\n\nThe counterargument to the above presented by the authors (that it would be hard to specify directly specify reward functions for tasks like Go), is contradicted by  Alpha Zero which worked off of a hard-coded human reward, and learned an even more well shaped reward in terms of the value function.  \n\n* It seems that all of the empirical improvements come from the intention policy rather than the preference based RL\n\n-- The preference based model is betting other models which have the ground-truth reward function.  This shouldn't happen if the only change is to use preference based RL, since in the best case preference based RL will reduce to the oracle model.  Thus the only other change (the intention policy), is the probable cause for the difference, but this is not framed as the core contribution of the paper.  If it were, it should be analyzed outside the preference based RL setting, since it does not depend on that.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X8OOiwwNuu2", "reviewer": "Reviewer_nMXe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel targeted preference-based adversarial attack against deep reinforcement learning (DRL) agents, so that the DRL agents would show extreme behaviors desired by adversaries. In particular, the proposed PALM adopts human preference, an intention policy and a weighting function to guide the adversary. Moreover, the authors also provided a theoretical analysis for the convergence of PALM. Finally, the experimental results suggest that PALM outperforms other baselines under the targeted attack setting.", "summary_of_the_review": "Overall, I think this work is interesting. However, as stated above, I didn’t fully understand Equation 4 and its purpose, which might affect my understanding and judgement of this paper and its contribution. Currently, I rate it as 5, but I am willing to increase my score if the authors could answer my questions above. Thank you very much!", "strength_and_weaknesses": "#### **Strengths**:\n- This work focuses on a novel targeted attack setting.\n#### **Weaknesses**:\n- In general, I found this work kind of hard to follow given the complicated notations. Based on my understanding, PALM uses RL to attack an RL agent. \n- Moreover, I am confused by Equation 4. What is the target soft Q-function? If we already know the targeted Q-function \\bar{\\phi}, why don’t we just use it in Equation 3 directly? If we don’t know \\bar{\\phi}, how can we optimize \\phi with Equation 4?\n- On page two, the authors mentioned several works that could improve the robustness of policies. Although I agree that this paper focuses on designing attack instead of defense, I am still wondering whether the proposed attack could surpass such defenses. Could the authors justify or provide some discussions about whether the proposed method could break these defenses or why they didn't run experiments on this? What if the proposed attack could not work under a even simple defense?\n- I also have a question about Section 4.3. (please see below in clarity)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HLzwBMkfbaY", "reviewer": "Reviewer_x6Lt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes the Preference-based Advisersarial Manipulation (PALM) algorithm which performs targeted attacks on deep reinforcement learning agents. This method used human preferences feedback and relies on an intention policy and a weighting function to guide the adversarial policy during training. Empirical results show that PALM outperforms several flavors of the adversarial algorithms SA-RL and PA-AD (which, as opposed to PALM, were not designed to manipulate the victim agent policy).", "summary_of_the_review": "In summary, the authors present an interesting approach for target attacks on deep reinforcement learning agents. The main weaknesses of this work are that the level of novelty and originality is not fully clear and it also seems that the empirical results showing that the proposed approach outperforms existing algorithms were performed in \"unfair\" settings.", "strength_and_weaknesses": "The main strengths of this work are:\n1. A novel approach that allows to manipulate deep RL agents towards a certain target (as opposed to simply make the agent fail in its task).\n2. Supporting theoretical analysis on the convergence of PALM.\n\nThe main weaknesses of this work are:\n1. An (intuitive) explanation for the role of the intention policy (beyond the given equations) is lacking. What is the motivation behind this? How does it interplay with the adversarial policy and guide its training?\n2. In the experiments, the authors compare PALM to existing algorithms that were not designed to manipulate the victim agent policy, in manipulation tasks. It would be interesting to see what happens if the situation is the other way around. How does PALM fair in experiments settings for which SA-RL and PA-AD are designed?\n3. Data is collected following a combined policy that depends on the perturbed policy. It is not clear why/how this can be done in practice. What happens if we cannot control according to which policy data is collected?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ArclLdObCk", "reviewer": "Reviewer_kjbK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel method for targeted adversarial attacks on Deep Reinforcement Learning algorithms by modifying agent observations. It learns an adversarial reward function via the preferences of a human-in-the-loop, and uses it to guide an intention policy that itself generates behaviors the adversarial policy eventually learns to reproduce by perturbing agent observations.\n", "summary_of_the_review": "The work features an interesting idea and convincing experimental results.\n\nThough, I feel that the work is not ready for publication yet, and that a proper literature review, improvements to the writing, clarifications and additional ablations would make the work stand out much more (see my detailed comments).\nI cannot recommend this work for acceptance in its current state.\n", "strength_and_weaknesses": "### Strengths\n\nFigures and the detailed algorithm are well-designed and help understand some aspects of the work.\n\nUsing an intention policy to explore the support of states that lead to maximizing the adversarial human reward is an interesting idea. It would be nice to show how much suboptimal exploration would hinder the same approach without the intention policy.\n\nExperimentation seems to be solid and the proposed method performs better than other benchmarked methods in most tasks.\n\nHaving code and videos available is a very nice plus.\n\n### Weaknesses\n\nThe title should mention human-in-the-loop in my opinion.\n\nThe related work is not complete at all. It is missing its core: targeted attacks for Deep RL. \nA reference comes to mind: Targeted Attacks on Deep Reinforcement Learning Agents through Adversarial Modifications (Hussenot et al, 2020).\nWhile this can be fixed, I find it puzzling that the authors have not found it useful or took the time to include at least a few references in that direction, and better a complete review of the literature. I am willing to give authors the benefit of doubt but this needs to be addressed before I am able to recommend for acceptance.\n\nOn the problem setup: 1) “generalized state-adversarial” is not a great name, should be “rewarded state-adversarial” instead, or something that better illustrates that there is a reward component involved, 2) “victim policy” is an offensive terminology, use ‘perturbed’/‘target’ instead maybe?\n\nIt is unclear whether the reward learning method proposed is a contribution or not.\n\n$\\pi_{\\nu\\circ\\alpha}$ is not defined (though one can figure out what it means).\n\nOn the method: the connection to SAC and soft-Q-learning should be mentioned, it is not clear whether it is a contribution or not in the current state, and this choice has no motivation (e.g. continuous control). \n\nOn the objective:\nIs it necessary to reweigh the KL-divergence? This is a modification of the standard that should be better motivated. E.g. via an ablation study.\n\nI do not get why $\\omega$ is learned via this implicit parameterization. Could it simply be fixed and seen as a hyperparameter? In other words, this is an additional source of complexity (bi-level optimization) of the algorithm and its inclusion should be better motivated.\nAnother point is that the nature of \\pi_\\alpha should be clarified sooner in the paper. How does it operate on the observations?? Is $\\alpha$ kept fixed?\n\nI think there should be evidence (i.e. empirical, via an ablation study) that the intention policy is needed.\nAlso, is it not possible to have an epsilon-greedy switch between perturbed and intention instead of the switching heuristic proposed?\n\nDrawbacks include that the intention policy has to be learned semi-online (part of the episode is played according to the intention policy), not simply through off-policy learning. Experiments showing it is not possible would be welcome. \n\nThe main drawback for me is that the writing is not clear enough on key parts of the work. Some super important details (weighting, bi-level optimization, and crucially *the nature of the adversarial modifications*) are hard to grasp or even missing from the current state of the paper.\n\nThere are no details on preference collection! This is absolutely crucial to the work.\n\nNit: the videos are not very impressive, the adversarial command always consists of performing the inverse of the sequence of actions from the policy. Adding other settings would better convey the usefulness of the approach. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject