Adversarial Cheap Talk
Authors: Chris Lu
Timon Willi
Alistair Letcher
Jakob Nicolaus Foerster
Venue: Submitted to ICLR 2023 (2023)
Keywords: Meta-Learning
Reinforcement Learning
Meta-Reinforcement Learning

Abstract: Adversarial attacks in reinforcement learning (RL) often assume highly-privileged access to the victim’s parameters, environment, or data. Instead, this paper proposes a novel adversarial setting called a Cheap Talk MDP in which an Adversary can merely append deterministic messages to the Victim’s observation, resulting in a minimal range of influence. The Adversary cannot occlude ground truth, influence underlying environment dynamics or reward signals, introduce non-stationarity, add stochasticity, see the Victim’s actions, or access their parameters. Additionally, we present a simple meta-learning algorithm called Adversarial Cheap Talk (ACT) to train Adversaries in this setting. We demonstrate that an Adversary trained with ACT can still significantly influence the Victim’s training and testing performance, despite the highly constrained setting. Affecting train-time performance reveals a new attack vector and provides insight into the success and failure modes of existing RL algorithms. More specifically, we show that an ACT Adversary is capable of harming performance by interfering with the learner’s function approximation, or instead helping the Victim’s performance by outputting useful features. Finally, we show that an ACT Adversary can manipulate messages during train-time to directly and arbitrarily control the Victim at test-time.

Under review as a conference paper at ICLR 2023
ADVERSARIAL CHEAP TALK
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial attacks in reinforcement learning (RL) often assume highty-privileged access to the victim's 's parameters, environment, or data. Instead, this paper proposes a novel adversarial setting called Cheap Talk MDP in which an Adversary can merely append deterministic messages to the Victim's 's observation, resulting in a minimal range of influence. The Adversary cannot occlude ground truth, influence underlying environment dynamics or reward signals, introduce non-stationarity, add stochasticity, see the Victim's 's actions, or access their parameters. Additionally, we present a simple meta-learning algorithm called Adversarial Cheap Talk (ACT) to train Adversaries in this setting. We demonstrate that an Adversary trained with ACT can still significantly influence the Victim's 's training and testing performance, despite the highly constrained setting. Affecting train- time performance reveals a new attack vector and provides insight into the success and failure modes of existing RI. algorithms. More specifically, we show that an ACT Adversary is capable of harming performance by interfering with the learner's function approximation, or instead helping the Victim's 's performance by outputting useful features. Finally, we show that an ACT Adversary can manipulate messages during train- rrin-time directly and arbitrarily control the Victim at test-time.
INTRODUCTION
Learning agents are often trained in settings where adversaries are not able influence underlying environment dynamics or reward signals, but may influence part of the agent's observations. For instance, adversaries may append arbitrary tags to content that will be used train recommender systems. Similarly, they may rent space on interactive billboards near busy traffic intersections to influence data sets used for training self-driving cars. financial markets, adversaries can alter the state of the order-book by submitting orders far out of the money (essentially for free). While these features are 'useless' from an informationn-thorree deep learning to include them as part of the input and let the model learn which features matter. For instance, welf-driving cars typically do not omit useless parts of the visual input but instead learns to ignore them through training. Surprisingly, this paper demonstrates that actor still heavily influence the behaviour and performance of learning agents by controlling information only in these "useless" channels, without knowing anything about the agent's parameters or training state.
point of view, it is common practice in end-to-end
Most past work in adversarial RL assumes that adversaries can influence environment dynamics (Huang et al., 2017 Gleave et al. 22020). For example, perturbing images and observations could obscure or alter relevant information, such as the ball's location in Pong game (Kos and Song 2017). Furthermore, many attacks require access to the trained agent's 's weights and parameters generate the adversarial inputs (Wang et al. 2021). Finally, most of these attacks only cause the victim's 's policy to fail arbitrarily instead of giving the adversary full control over the victim's 's policy at test time (Gu)
et al. 2017 Kiourti et al. 2020 Salem et al. 2020 Ashcraft and Karra, 2021 Zhang et al., 2021).
Instead, we propose novel minimum-viable setting called Cheap Talk MDPs. Adversaries are only allowed to modify "useless' features that are appended to the Victim's 's observation as a deterministic function of the current state. These features represent parts an agent's observations that are unrelated to rewards or transition dynamics. In particular, our model applies to Adversaries adding tags to content in recommender systems, or renting space on interactive billboards, or submitting orders far out of the money in financial markets. The setting is minimal in that Adversaries cannot use these features to occlude ground truth, influence environment dynamics or reward signals, inject stochasticity, introduce non-stationarity, see the Victim's actions, or access their parameters.

---

Under review as a conference paper at ICLR 2023
Cheap Talk MDPs are formalised in Section and we further justify minimality by proving in Proposition that Adversaries cannot influence tabular Victims whatsoever in Cheap Talk MDPs. In particular, it follows that Adversaries can only influence Victims by interfering with their function approximator. We also prove more generally in Proposition Detiittioonnaatt Adversaries cannot prevent Victims with optimal convergence guarantees converge to optimal rewards, even in non-tabular settings.
Despite these restrictions, we show that Adversaries can still heavily influence agents parameterised by neural networks. In Section [5] we introduce a new meta-learning algorithm called Adversarial Cheap Talk (ACT) to train the Adversary. With an extensive set of experiments, we demonstrate in Section 6 that an ACT Adversary can manipulate a Victim achieve a number of outcomes:
1. An ACT Adversary can prevent the Victim from solving a task, resulting in low rewards during training. We provide empirical evidence that the Adversary successfully sends messages which induce catastrophic interference in the Victim's neural network.
2. Conversely, an ACT Adversary can learn to send useful messages that improve the Victim's training process, resulting in higher rewards during training.
3. Finally, we introduce a training scheme that allows the ACT Adversary to directly and arbitrarily control the Victim directly at test-fime.
RELATED WORK
2.1 TEST-TIME ADVERSARIAL ATTACKS
Most work investigating adversarial attacks on deep RL systems focuses on attacks at test-time, i.c. those that attack a fully trained, static policy. Huang al. (2017), Kos and Song (2017), and Zhang et al. (202)) investigate adversarial attacks that influence test time performance by directly perturbing the observation space. Unlike in Cheap Talk MDPs, such perturbations can influence the underlying dynamics by obscuring relevant information. Gleave et al. (2020) investigate ad adversarial attacks that influence test-time performance of reinforcement learning agents that were trained in self-play. In contrast to our method, the adversarial agent can directly interact with the environment and the victim agent. The aforemmntioned test-time attacks largely work by generating perturbations that push the observations out of the Victim's 's training distribution. In contrast, in Cheap Talk MDPs, the Victim trains directly with the static adversarial features; thus, by definition, the Adversary cannot generate
non-stationary inputs. Furthermore, these test-time attacks also assume the ability to directly train against a pre-trained static agent. In contrast, in Cheap Talk MDPs, the adversary is influences learning agent with random initial parameters.
2.2 TRAIN-TIME ADVERSARIAL ATTACKS
In contrast to train-time adversarial attacks in RL, in test-time adversarial attacks the dversary interacts with a learning victim. Pinto et al. (2017) simultancously trains an adversary alongside reinforcement learning agent to robustify the victim's 's policy. Unlike in this work, the adversary is able to directly apply perturbation forces to the environment. We make further comparisons in Section 6.1 Backdoor attacks in reinforcement learning aim to introduce vulnerability during train-time, which can be triggered at test-time. Kiourti et al. (020) and Ashcraft and Karra (2021) assume the adversary can directly and fully modify the victim's observations and rewards in order to discretely insert a backdoor that triggers on certain inputs. This is unlike Cheap Talk MDPs, in which only 'useless' parts of the observations can be modified. Wang al. (202I) considers the multi-agent setting where the adversary inserts a backdoor using its behaviour in the environment. Unlike in this work, the adversary can influence the underlying environment dynamics. Furthermore, each of these backdoor attacks simply cause the victim to fail when triggered. In contrast, we use the backdoor to fully control the victim.
2.3 FAILURE MODES IN DEEP REINFORCEMENT LEARNING
Previous works have shown that using neural networks function approximators in reinforcement learning often results in multiple failure modes due to the non-statinnarity of value function boot- strapping vaan Hasselt al. 2018). In particular, works have shown that catastrophic interference
2

---

Under review as a conference paper at ICLR 2023
(Bengio et al. 2020) and capacity loss Lyle et al. 2022) often occur, within single episode of an environment (Fedus et al., 2020). Song et al. (2020) shows that deep reinforcement learning algorithms often overfit spurious correlations in the observation space. By appending the observation space, we learn to induce the observational failure modes described in these works.
2.4 OPPONENT SHAPING / CHEAP TALK
Our method is closely related to the field of opponent shaping. Originally, most opponent shaping algorithms assumed white-box access to their opponents to shape the flow of the opponent's gradient (Foerster et al. 2018 Letcher et al. 2019a b. Willi et al. 2022). Instead, Lu et al. (2022) introduce method to shape opponents without white-box access. However, they still deploy an agent to
interact directly in the environment. In contrast, we propose a method to shape other agents without having to interact in the environment, solely by appending messages through a cheap talk channel. In reinforcement learning, a cheap talk channel is a part of the state space which can be observed by other agents but does not alter transition dynamics reward functions. Cheap talk channels (Crawford) and Sobel, 1982) in deep reinforcement learning have been used to learn emergent communication (Foerster al., 2016) and solve coordination problems (Cao et al., 2018). To the best of knowledge, this paper is the first to use cheap talk channel (and only a cheap talk channel) to shape the behaviour of learning agents.
BACKGROUND
3.1 REINFORCHMENT LEARNING
A Markov decision process (MDP) consists of tuple D = A,,,,, where S denotes the state space, A represents the action space, P: S function, R: S x A + R is the reward function and [0, denotes the discount factor. At every timestep 1, an agent samples an action from its stochastic policy d, ~ Ti 8t), where a, E A, E S and 0 denotes the policy parameterisation. The agent then receives a reward based on the action taken in the current state: F = R (st, a.). Finally, a new state sampled according to the transition function 8141 P (- 8t, az), ) resulting in trajectory == ((so. do. ro). ($1, a1, r1)..).... The agent's 's goal is to maximise its expected discounted return under policy no:
S - [0, 1] denotes the state transition probability
J(0) E
(1)
3.2 EVOLUTION STRATEGIES
Evolution Strategies (alimans et al. 2017 ES) is derivative- ffree optimisation method. Let F Rd + IR be some function we want to optimise over. Instead of optimising F(x) directly, we first blur the objective to
E [((x ac)],
where a is
dictating the amount of Gaussian noise we add. This is useful because
Vx TxEeeNNo..L) + ac)] (( F(x or)
which allows to optimise non-differentiable function using gradient descent techniques in highly scalable manner.
PROBLEM SETTING
The setting we propose is two agents interacting in Cheap Talk MDP AP which is effectively an MDP with an augmented state space, whereby features (messages) from a Cheap Talk channel M are appended to states. We refer to the agent observing the augmented state the Victim, with transition and reward functions P. R assumed to be independent from M. Formally, this means that P(: s, m, a) = P(- s, m', a) and R(s, m, a) = R(s, m', a) for all messages m. m' M. The agent appending the message called the Adversary, and is endowed
3

---

Under review as a conference paper at ICLR 2023
with deterministic policy f S - M to append messages f(s) and objective function J to optimise (details below).
The Victim is standard reinforcement learning agent, selecting actions according to its policy aj ~ Tol: | s, f(s)), where a E A, E S. The Victim optimises its policy Tg with respect to parameters A in order to maximise its expected return J as defined in Equation [
By contrast, the Adversary may only act by modifying the cheap talk channel features fa(() at at every step, where fo S M is deterministic policy (function) of the current state and ø are the Adversary's parameters. These parameters may only be updated between full training/testing episodes of the Victim; the function remains static during episodes to avoid introducing non-stationarity and thus restrict the Adversary's 's range of influence. The Adversary's 's objective function J may be picked arbitrarily, and need not be diffcrentiable if is optimised using ES.
In our train-time experiments we focus on the fully-adversarial setting where objectives are zero-sum, J -J and the allied setting where objectives are equal, J J. In test- test-time experiments we use an entirely different objective, such as reaching for an arbitrary circle in Reacher (see Figure (igure(5c). This incentivises the Adversary to manipulate the Victim into maximising J, even if this comes at the cost of the Victim's original objective J.
4.1 MINIMALITY OF CHEAP TALK MDPs
To justify our introductory claim that Cheap Talk MDPs only allow for a minimal range of influence, we first prove that Adversaries cannot influence Victims whatsoever in the tabular setting, irrespective of the Victim's 's learning algorithm. It follows that Adversaries can only attack Victims by interfering with their function approximator.
Proposition 1. In any Cheap Talk MDP, the policy of tabular Victim is independent from its Adversary provided uniform initialisation along M, namely na(- 88, my) = To(- 8t. m.)) j.
Proof (Sketch). The main intuition is that policy updates for different states do not interfere with each other in tabular settings. Assuming uniform initialisation and noticing that the only states encountered in the environment are of the form (s, f()).. we deduce that the Victim's policy updates are independent from the Adversary's choice of function f. Formal proof in Appendix A.1
We also prove more generally that Adversaries cannot prevent Victims with optimal convergence guarantees to converge optimal rewards, even in non-tabular settings. They may however still harm the Victim by slowing down their convergence rate significantly.
Proposition 2. A Victim which is guaranteed to converge to optimal policies in MDPs will also converge optimal policies in Cheap Talk MDPs, with expected return equal the optimal return for the corresponding no-channel MDP.
Proof (Sketch). Cheap Talk MDPs are just MDPs with augmented state spaces and transition / reward functions; Victim will therefore converge regardless. Optimality of the expected return follows from the Bellman equation and independence of transition and reward functions from Adversaries. Formal proof Appendix A.2
Finally, we further justify minimality Appendix A.3 by elaborating informally on the Adversaries' range of influence. We also discuss the possibility of further weakening Cheap Talk MDPs and conclude that all such variations either bring no advantage or reduce to regular MDPs.
5
METHOD
5.1
META-TRAINING PROCEDURE
Our method, Adversarial Cheap Talk (ACT), treats the problem setting a meta-learning problem. The Adversary's 's parameters & are only updated after a full training (and testing) run of the Victim's 'ss parameters 0. In other words, is static during the whole training run (inner loop) of 0 and only gets
4

---

Under review as a conference paper at ICLR 2023
updated once the inner loop completes, which prevents the introduction of non-stationarity. In the outer loop, we optimise the Adversary's 's objective J with respect to & using ES as a black-box opti- misation technique. Details, including the Cheap Talk channel sizes and the Victims hyperparameters can be found in Appendix
5.2 TRAIN-TIME INFLUENCE
Algorithm 1 Train-time ACT
When influencing train-time performance, we
1: Set 11 for allied adversarial
set J to be the agent's s mean reward throughout its entire training trajectory. We consider both "Adversarial" and "Allied" versions of ACT, whereby Adversaries try to minimise or max-
2: Initialise Adversary parameters Φ
3: for m = M do
4: Sample o, ~ Φ tcn, with ca 10,
5: for n = 0 to N do
imise J respectively (J =.)). Pseudocode is
6: Initialise Victim parameters 0
provided in Algorithm [] where E is the num-
7:
rewards = [
ber of Victim training episodes and N is the ES
8:
for e = 0 to E do
population size.
9:
s = env.reset)) while not done do
10:
5.3 TEST-TIME MANIPULATION
11:
a ~ T s, fa. (s))
12:
8, done env.step(a) rewards.append(r)
When manipulating test-time behaviour, the goal
13:
of the Adversary is to use the cheap talk features
end while
to maximise some arbitrary objective J during 15:
Update @ with PPO to maximise J end for
the Victim's s test-time; however, the Adversary 16:
may also communicate messages during the Vic-
17:
In = (uin(rewards)/eenrewwwaa
tim's training. Note that J can be any objective,
18:
end for
including minimising or maximising the Victim-
19: Update á using ES to maximise J
sreturn. Because the train-time and test-time behaviour of the Adversary differ significantly,
20:
end for
we parameterise them separately (as & and &b respectively). but optimise them jointly.
As an example, consider the Reacher environment (see 5c). where the Victim is trained to control robot arm reach for the blue circle. During the Victim's training, the train-time Adversary (parameterised by Φ) manipulates the cheap talk features to encode spurious correlations in the Victim's policy. At test-time, the test-time Adversary (parameterised by th) manipulates the cheap talk features to take advantage of the spurious correlations and control the Victim to have it reach for the yellow circle instead (the Adversary's objective J). More concisely, the train-time Adversary wants to create a backdoor make the Victim susceptible to manipulation at test-time. The test-time Adversary wants to use this backdoor to control the Victim. The train- time and test-time Adversaries (ø and 4) are trained end-to-end to maximise J. While such optimisation would be difficult for pradient-based methods due to the long-horizon nature of the problem, ES is agnostic to the length of optimisation horizon.
Note that the test-time Adversary i only gets a single shot to maximise J at the end of the Victim's 's training and does not have access to (and thus cannot train against) the test-time parameters of the Victim 0'. To describe this formally, let T(0 Φ) denote the distribution induced by the inner loop training with the train-time Adversary Φ over Victim 8. Then, in each meta-episode, the test- time Adversary z has to interact with an unseen sample 0' of the distribution over trained Victims
0' T( Φ). In Section 6 we show that the distribution T(O +) has non-trivial variance, suggesting that it is difficult to train against. Moreover, in Figure (Section 6). we provide empirical
evidence that the train-time Adversary learns to reduce the variance of T( ()) to help the test-time
Adversary. Pseudocode is provided in Algorithm 2 Appendix
6 EXPERIMENTS AND RESULTS
We evaluate ACT on three different simple gym environments: Cartpole, Pendulum, and Reacher [Brockman et al. 2016). We also evaluate ACT on Minatar Breakout Young and Tian (2019): Lange (202bb) to test ACT's ability to scale to higher-dimensional environments. The Victim is trained with Proximal Policy Optimisation (Schulman et al. 2017 PPO). The Adversary is trained using ES (Salimans et al. 2017).
5

---

Under review as a conference paper at ICLR 2023
Cartpole
Pendulum
Reacher
180
Return
Return
-400
Zeros
-600
Random
Alled
100
-800
Advensarial
0
2
2 3
0.6
Timesteps
104
Timesteps
l05
Timesseps
(a)
(b)
(c)
Figure Visualisations of the training curves of the Victim across different Adversaries for (a) Cartpole, (b) Pendulum, and (c) Reacher. Error bars denote the standard error across 10 seeds of Victims trained against single trained Adversary.
Minalar Breakout
Cartpole Victim Return
250
Z015:
ACT
40
Bandam
200
RARL
Ishemeriii
neturn
30
6
100
0
6
2
Timestaps
1e6
Updates
1e2
(a)
(b)
Figure 2: (a) Visualisations of the training curves of the Victim in Breakout-Minatar, a higher- dimensional environment, against different Adversaries. (b) Comparing Adversarial ACT RARL. Eventually, the Victim learns to overcome the RARL adversary.
We train thousands of agents per minute on a single V100 GPU by vectorising both the PPO algorithm itself and the environments using Jax (Bradbury et al., 2018). This allows to com-loeille the full training pipeline and perform end-to-end deep RL training completely on GPUs. We adapt the environment implementations from Brockman et al. (016) and Lenton et al. 2021 and use the ES implementation from Lange 2022a). This compute setup allows us to efficiently perform outer-loop ES on the full training trajectories of inner-loop PPO agents. For example, in Cartpole, we can simultaneously train 8192 PPO agents at time on a single V100 GPU. Over 1024 generations of ES, this results in training 8,388,60 608 PPO agents from scratch in 2 hours on 4 V100 GPUs The longest training time, which was the test-time Reacher setting. took 20 hours to train 1024 generations on V100 GPUs Training details are provided in Appendix [E] We also include videos of the Victim's performance and visualisations of the Adversary's 's outputs in the Supplementary Materials.
6.1 TRAIN-TIME INFLUENCE
Figure and Figure 2a show the results of training Victims alongside four different Adversaries.
1. Ally: meta-trained to maximise the Victim's 's mean reward throughout training.
2. Adversary: meta-trained to minimise the Victim's 's mean reward throughout training.
3. Random Adversary: randomly initialise and fix the Adversary's parameters using LeCun Uniform initialisation (LeCun et al. 2012).
4. Zeroes Adversary: appends only zeroes as messages.
Ally. The Ally manages to assist the Victim to learn and converge faster this is likely done by appending useful features of the environment, We do further analysis in Appendix C Figure 7b
Adversary. The Victims trained alongside the Adversaries are vastly outperformed by the baselines, even though the Adversary cannot change the dynamics of the underlying MDP, and cannot add

---

Under review as a conference paper at ICLR 2023
Adversary
Random
Ally
1.0
1.0
Env Segment
2
0.8
Env Segment
2
0.8
Segment
2
0.8
0.6
0.6
0.6
0.4
0..4
0.4
8
8
8
0
6
Env Segment
Env Segment
Env Segment
(b)
(c)
Figure 3: Visualisations of the cosine distance between gradient updates different environment segments in cartpole. We collected each Victim's experience buffer before the agents converge in training and split each into 10 bins, ordered by the time-step within the environment. We then calculate the gradient update the agents perform on each of these bins. This is the technique used in Fedus et al. (2020). For the adversary (a), gradient updates on the early timesteps in an environment interfere with gradient updates on the ending timesteps. For the ally (c), they are positively correlated.
non-stationarity stochasticity. Moreover, since Adversaries cannot influence tabular Victims by Proposition [ this must be accomplished through learnt interference with the Victim's function approximator. We hypothesise that the Adversary may be inducing catastrophic interference within the environment, which was observed by Fedus et al. (2020) in Atari 2600 games. They show that features useful in the early phases of environment episode interfere with learning features for performing well in the later phases of an episode.
Figure 3 suggests that, in the Adversarial setting, the gradient updates performed for transitions sampled early in an episode can interfere with the gradient updates performed for transitions later in episode. Meanwhile, those gradient updates are positively correlated in the Allied setting, suggesting that the gradient updates aid each other.
We also compare our evolutionary meta-optimisation procedure Robust Adversarial Reinforcement Learning (RARL) (Pinto et al. 2017) in Figure 2b] which updates the adversary's 's parameters online using reinforcement learning. We include training details in Appendix [F) In both settings, the Adversary can only communicate over the cheap talk channel; however, RARL is given a larger range of influence. Firstly, RARL introduces stattoattonartty since it is updated online during the opponent's 's learning, unlike ACT. Secondly, RARL is parameterised by a stochastic policy, meaning that can inject stochasticity into the environment, unlike ACT. Thirdly, RARL is able to train directly against the Victim's 's policy online. unlike ACT which cannot view the Victim's policy or actions. However, RARL ultimately underperforms ACT in the adversarial setting since it is unable to consider the long-term evolution of the opponent's policy (it takes greedy updates against the Victim's 's online parameters). Thus, the Victim learns to simply ignore the cheap talk channel. Note that RARL intended be overcome in order to robustify the Victim's policy.
6.2 TEST-TIME MANIPULATION
In test-time manipulation, the Adversary's objective is to maximise the score of the goal-conditioned objectives described in 4n test-time. The Adversary needs to learn introduce backdoor during and use the backdoor during test-time to fully control the Victim as explained in 5.3 To better understand the capability of our model, we investigate four different Adversary- Victim settings. These four settings serve ablations to study the individual and joint performance of the train- and test-time Adversarics.
1. Test-Time Adversary v with Train-Time Adversary o: This is the algorithm described in 2, First, we train Victim 9 alongside train-time Adversary Φ. We then evaluate the return of the test-time Adversary i according to the goal-conditioned return. Both the train- and test-time Adversaries are trained using ES. The test-time Adversary z only gets a single shot against Victim and is thus represented by horizontal line in Figure
7

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
Figure 4: Visualisations of our goal-conditioned environments (a) In Cartpole, the Adversary's target is randomly selected point the x-axis (the yellow box). (b) In Pendulum, the Adversary's goal is a randomly selected angle (the yellow pole). (c) In Reacher, the Adversary's goal a random point, (the yellow circle). The Victim's 's goal the blue circle.
Cartzole
Penduku
Reacher
140
100
50
110
Timestops
Tiresrace
Timashops
(b)
(c)
Figure 5: Training curves of the different agents in (a) Goal-Conditioned Cartpole (b) Goal- Conditioned Pendulum (c) Condiioodd Reacher. The ablations show that the train- and test-time Adversary learn near-optimal performance in comparison to the oracles. Error bars denote the standard error of the mean across 10 seeds of Victim trained against a single trained Adversary.
2. Test-Time Oracle d" with Train- Time Adversary dz First, optimise the Victim by training it alongside the above train-time Adversary ø. Then, instead of ES, we use PPO train the test-time Adversary 2/" against the Victim 0. Unlike the test-time Adversary, the oracle v/ is allowed to train against the pretrained and fixed Victim 0 to maximise its returns, as described in Algorithm in Appendix B
3. Test-Time Oracle v0 with Random Train-Time Adversary Orandom: First, we obtain Victim by training alongside random train-time Adversary, Prandom. with randomly initialised and fixed parameters. Next, we use PPO to train the test-time Adversary vi* to maximise the conditioned return.
4. Direct Oracle:
In this baseline, there is no cheap talk or Victim. We simply train a PPO agent to maximise the goal-conditonne return J. can observe the full state and directly output actions in the environment.
All results are shown in Figure 5 We can compare (1) and (4) to measure how effective the train-time Adversary é and test-time Adversary z are at achieving the maximal possible return jointly. As Figure shows, the train- and test-time Adversaries perform ncar-optimally.
By comparing (2) and (3), the test-time Oracle e cannot achieve max performance with a random train-time Adversary.
observe how effective ø is at shaping 0. In reacher,
that
We can compare (1) and (2) to how effective the test-time Adversary & is exploiting given Victim 0. We can see that &b achieves near-optimal performance even though has never trained against the specific Victim or had access to its parameters. In Figure we show that it can do this
8

---

Under review as a conference paper at ICLR 2023
Mear of Policies
Mean of Policies
1.00
1:00
4
0.78
0.50
0.30
025
025
9.00
0.00
0.50
0.50
-0.75
-0.75
-1.00
-1.00
0.5
0.5
Channel Dimension
Channel Dimension
(a)
(b)
Variance Acress Policies
Variance Acress Policies
0.75
0.75
monrion
0.50
momoionn
0.50
0.00
0.00
0.25
00255
-0.75
-1
-0.75
-1
-1.00
-1.00
-1.0
-0.5
0.0
0.5
-1.0
-0.5
0.0
0.5
Channel monwion C
Channel Dimenviod on D
(d)
Figure 6: We train 10 different Victims alongside the Learned & (a & c), and 10 different Victims alongside randomly generated é (b & d) in the Pendulum environment. (a) and (b) show the mean of the policy output across the 10 Victims as we vary the value of the message a fixed randomly selected state. The policies trained with the learned & achieve much wider range of outputs. (c) and (d) show the variance of the policy output across the 10 Victims. The policies trained with the learned display very little variance.
because the train-time Adversary & not only maximises the range of outputs that the cheap talk can achieve, but also does so in consistent and low-variance way.
7 CONCLUSION & FUTURE WORK
In this paper, propose novel, minimum-viablle adversarial setting for RL agents, where the Adversary can only influence the Victim over messages, and can only do so with deterministic function that only depends the current state. By training an Adversary with adversarial cheap talk (ACT), we show that appending to the observations of a learning agent, even with strong constraints, sufficient to drastically improve or decrease learning agent's train-time performance introduce a backdoor to control the learning agent at test time completely. Furthermore, we provide an in- depth analysis of the behaviour of our Adversaries. At train-time, the Adversary learns induce something akin to catastrophic interference to decrease training performance. In the instant test- time manipulation setting, the train-time Adversary learns reduce the variance of the training process to reduce variance of potential Victims the test-time Adversary could face.
As RL models become more widespread, we believe practitioners should consider this new class of minimum viable attacks. Therefore, we propose using domain knowledge to identify and filter out potentially controllable information as the first defence measure. Identifying these channels without domain knowledge would be challenging: While there has been past work in identifying iask-irrelevant features reinforcement learning (Vischer et al. 2021). ACT features still contain task-relevant information since they functions of the state. To defend against the test-time Adversary. one can detect when the input goes out-of di-diitriitttion (Lin et al. 2017). Future work would be required to build more robust and practical defences.

---

Under review as a conference paper at ICLR 2023
REFERENCES
C. Ashcraft and K. Karra. Poisoning deep reinforcement learning agents with in-distribution triggers. arXiv preprint arXiv:2106.07777 2021.
E. Bengio, J. Pineau, and D. Precup. Interference and generalization temporal difference learn- ing. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 20207
J. Bradbury. R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, VanderPlas, Wanderman- Miine,, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
G. Brockman, V. Cheung, L.. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
K. Cao, A. Lazaridou, M. Lanctot, J. Z. Leibo, K. Tuyls, and S. Clark. Emergent communication through negotiation. In 6th International Conference on Learning Representations, 2018.
V. P. Crawford and J. Sobel. Strategic information transmission. Econometrica, 50(6):1431-1451 1982.
W. Fedus, Ghosh, J. Martin, M. G. Bellemare, Y. Bengio, and H. Larochelle. On catastrophic interference in atari 2600 games. arXiv preprint arXiv: 12499, 2020.
J. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch. Learning with opponent-learnng awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 122-130, 2018
N. Foerster, Y. M. Assael, N. de Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 29, pages 2137-2145, 2016.
A. Gleave, M. Dennis, C. Wild, N. Kant, S. Levine, and S. Russell. Adversarial policies: Attacking deep reinforcement learning. In 8th International Conference on Learning Representations, 2020.
T. Gu, B. Dolan Gavit,, and S. Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.00 2017.
S. H. Huang, N. Papernot, J. Goodfellow, Y. Duan, and P. Abbeel. Adversarial attacks neural network policies. In 5th International Conference on Learning Representations, Workshop Track Proceedings. 2017.
Kiourti, K. Wardega, S. Jha, and W. Li. Trojdrl: Evaluation of backdoor attacks on deep reinforce- ment learning. In 57th ACM/IEEE Design Automation Conference, pages 1-6, 2020.
Kos and Song. Delving into adversarial attacks on deep policies. In 5th International Conference on Learning Representations, Workshop Track Proceedings, 2017.
R. T. Lange. evosax: Jax-based evolution strategies, 2022a.
R. T. Lange. gymnax: JAX-based reinforcement learning environment library, 2022b. URL com/ Robert TLange /gymnxx
Y. LeCun, L. Bottou, G. B. Omt, and K. Müller. Efficient backprop. In G. Montavon, G. B. Orr, and K. Müller, editors, Neural Networks: Tricks of the Trade Second Edition, volume 7700 of Lecture Notes in Computer Science, pages 4- Springer, 2012. doi: 10. 10.10077978-3-442-55522 URL /attps://doi.oo 100 1001 00000/9978333 3- 642 35289-8
D. Lenton, F. Pardo, F. Falck, S. James, and R. Clark. Ivy: Templated deep learning for inter- framework portability, arXiv preprint arXiv:2102.0288. 2021.
A. Letcher, D. Balduzzi, S. Racanière, J. Martens, J. N. Foerster, K. Tuyls, and T. Graepel. Differen- tiable game mechanics. J. Mach. Learn. Res., 20:84:1 14:440, 2019a.
10

---

Under review as a conference paper at ICLR 2023
A. Letcher, J. N. Foerster, D. Balduzzi, T. Rocktäschel, and S. Whiteson. Stable opponent shaping in differentiable games. In 7th International Conference on Learning Representations, 2019b.
Y. Lin, M. Liu, M. Sun, and J. Huang. Detecting adversarial attacks on neural network policies with visual foresight. CoRR, abs/1710.008 2017. URL http::/aaxwv..rg//eeeeaaa 00814
C. Lu, T. Willi, C. Schroeder de Witt, and J. Foerster. Model-free opponent shaping. arXiv preprint arXiv:2205.01447, 2022.
C. Lyle, M. Rowland, and W. Dabney, Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXi::220..09566 2022.
L. Pinto, Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning. In D. Precup and Y. W. Tch, editors, Proceedings of the 34th International Conference on Machine Learning. ICML 2017. Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 2817 2817-2826. PMLR, 2017. URL http: / proceedingsse html
mlr. w.t..press/77//pintoot
A. Salem, R. Wen, M. Backes, S. Ma, and Zhang. Dynamic backdoor attacks against machine learning models. arXiv preprint arXiv:2003.03675. 2020.
T. Salimans, J. Ho. X. Chen, and I. Sutskever. Evolution strategies scalable alternative to reinforcement learning. arXiv preprint 1703::/003.038000 2017.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1..Xiv:700.0000 17071777777777 2017.
X. Song. Y. Jiang, Tu, Y. Du, and B. Neyshabur. Observational overfitting reinforcement learning. In 8th International Conference Learning Representations, 2020.
H. van Hasselt, Y. Doron, F. Strub, M. Hessel, N. Sonnerat, and J. Modayil. Deep reinforcement learning and the deadly triad, arXiv preprint arXiv 2018.
M. A. Vischer, R. T. Lange, and H. Sprekeler. On lottery tickets and minimal task representations in deep reinforcement learning. CoRR, abs/2105.01168, 2021. URL http:::////// /ttps:://axxv.../// abs/210s3 01648
L. Wang. Z. Javed, X. Wu, W. Guo, X. Xing, and D. Song. BACKDOORL: backdoor attack against competitive reinforcement learning. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 3699-3705, 2021.
T. Willi, J. Treutlein, A. Letcher, and J. Foerster. COLA: consistent learning with opponent-learning awareness. arXiv preprint arXiv:2203. 04098, 2022.
K. Young and T. Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments, arXiv preprint 11XXvv/9033.3
2019.
H. Zhang, Chen, D. S. Boning, and C. Hsich. Robust reinforcement learning on state observations with learned optimal adversary. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:// openreview net http:/oormmmisscchbhaaae
11

---

Under review as a conference paper at ICLR 2023
A MINIMALITY OF CHEAP TALK MDPs
A.1 PROOF OF PROPOSITION
Proposition | In any Cheap Talk MDP, the policy of tabular Victim is independent from its Adversary provided uniform initialisation along M, namely To(- 8,, my) = To(- 84,, my.) V j.
Proof. In Cheap Talk MDP (S, A,,, M. J),, a tabular Victim arbitrarily orders states as {81....,88, 8d} and messages as {mm.....,,. where = |||] and k = [M], and stores policies Tel- 82, My) time of the learning process for all i E [ E [k:] The argument follows identically for value functions. Assuming uniform initialisation along the M axis means that
To(- Si, m,) = To |s, m...)
for all j. [k.]. Now consider any two Adversaries f, g and their influence two copies of the same Victim V, W with respective policies æ,, X. The only states encountered in the environment are of the form (s, fs)) and (s, (s)) respectively, Victims only update the corresponding policies TT(. $4, fs;)) and Xe(- 8ts ((.)).
We prove by induction that these quantities are equal for all t. The base case holds by uniform initialisation along M; assume the claim holds for all fixed 0 1 T. The Victims update their policies at time T + according to the same learning rule, function of the transitions and returns under current and past policies Te and X respectively. Transitions take the form (s, fs). a, s', f(')) for V and (8,9()), a, .())) for W, which have identical probabilities and returns because
Te(a 8t. f(x)) Xe(a 8i, gs:)()))):
P(s', f(s)) s, f()) = PPs', s, g))),a); f()) = RR(s, g())
by inductive assumption and independence of P, P.R from M. This implies that the Victims' policies Tr s, fs:)) Xr(- s;. (s:)) are updated identically to
TT+1(- 8,, fs:)) = XT+1(- s, gs.)(
as required to complete induction. Note that this would not necessarily hold in non-tabular settings, where updating parameters of the function approximator for some state (s;, f(s)) may alter the policy on some other state (sj. (s))). ). It now follows that trajectories T = (s ((**,, ), a²),, ). for V
and w = (s*",(s*), )²
y for W have identical probabilities and hence produce identical returns Eyon, [R(T)] = E [R(w)]
any timestep of the learning process, concluding independence from Adversaries.
A.2 PROOF OF PROPOSITION
2
Proposition 2 A Victim which is guaranteed to converge to optimal policies in MDPs will also converge to optimal policies in Cheap Talk MDPs, with an expected return equal the optimal return for the corresponding no-channel MDP.
Proof. By assumption, the Victim is guaranteed converge to an optimal policy स in any given Cheap since a Cheap Talk MDP is itself an MDP with an augmented Talk MDP (S, A,P, R., M.J.J 7)).
state space S M and augmented transition/reward functions that are defined be independent from
M. Now म naturally induces policy on the no-channel MDP, given by x(- s) #(- I fs)), and in particular Q(s, a) = Q(s, f(s), a) by independence of transitions and rewards from M. Optimality of follows directly from the Bellman equation
((.,a) = ((s, P))
+ 7 max Q(s' ((5''1(6)).))) Y'EA
E. =Bgr
r 7 max Q(5 (0)'
d'EA
Now trajectories 7 = (s* [(*)), a²)),, and - (s* a² have identical probability and return under π and i respectively, so the Victim has expected return
Erax [R(F)] Er.a [R(7)]
which is the optimal expected return of the original no-channel MDP.
12

---

Under review as a conference paper at ICLR 2023
A.3 FURTHER INFORMAL DISCUSSION
Consider Cheap Talk MDP (S, 75, For fixed training testing run of the Victim on the MDP, the Adversary outputs a message f(()) at each step according to fixed deterministic function f
M. Below we elaborate informally the claims that Adversaries cannot (1) occlude the ground truth, (2) influence the environment dynamics / reward functions, (3) see the Victim's actions or parameters, (4) inject stochasticity, (5) introduce non-stationarity.
(1) The message is appended to the state and the Victim acts with full visibility of the ground truth (state) according to its policy: a ~ o.. f())).
(2) The transition and reward functions P, R are defined to be independent from M. Formally we have P(- 8, m, a) P. 8, m'', a) for all m, m' . (similarly for R), the Adversary's choice of message m = f()) cannot influence P or R.
(3) f S + .M is defined as function of S only, so the Adversary cannot condition its policy based on the Victim's s actions or parameters (i.e. it cannot see or influence them).
(4) f is a deterministic function, so TT. s. f())) is a distribution only on actions A. The transition and reward functions are independent from f. so they are distributions only state-action pairs S x A. follows that the Adversary injects no further stochasticity into the MDP.
(5) f is static for fixed training testing run, so 9 the run. It follows that any given Victim policy æ is stationary, namely x- Sts f(s)) rT-- sy, ((se)) ) for all = Str. Since P and R are stationary (as defined by standard MDP) and independent from M, their stationarity also preserved.
implies f(s)) = f(se) ) for all timesteps
Finally, discuss the possibility of further weakening components of a Cheap Talk MDP, and conclude that all such variants (A-E) bring no advantage or reduce to regular MDPs.
(A) Removing the channel M or the policy S M would result in the Victim being completely independent from the Adversary, so no adversarial influence could be exerted whatsoever.
(B) Restricting the capacity of M to certain number of bits would further restrict an Adversary's 's range of influence, so one could say that the truly minimum-viable setting is to impose a set of size [M| = 1. However, cheap talk is still cheap talk when varying capacity, and there is no reason to arbitrarily restrict the size to 1 if we are to apply our setting to complex environments likely requiring more than a single bit of communication to witness interesting results.
(C) Not allowing Adversaries to states, namely removing S as inputs f. yields function f {0} + M which always outputs the same message f (0) = m E M. This is equivalent to the previous restriction of imposing set M of size 1, since in this case any function f S - M would have to output the unique element f()) = m for all input states s.
(D) The Adversary must have some objective function J in order for an adversarial setting make sense - removing would remove the Adversary's reason to exist, since would have no incentive to learn parameters that influence the Vietim according to some goal.
E Restricting the function class of objectives J is valid minimisation of the setting, but simply restricts our interest in the setting itself. The setting should at the very least allow for adversarial objectives of the form J =J, as we consider in the train-time setting. In test-time, our aim is to show how Adversaries can exert arbitrary control over Victims despite cheap talk restrictions, and we therefore consider more general objective functions.
13

---

Under review as a conference paper at ICLR 2023
B PSEUDOCODE
Algorithm 2 Test-time ACT
Initialize train-time ACT parameters
2: Initialize test-time ACT parameters x/b
M do 4: Sample on
3: for m
den where 61...,, No0. I)
5: Sample 1h ~ tb + den where 61,...,N EN (0,
6: for n 0 to N do
7: Initialize policy params
8:
rewards=1 [
9:
for e = =0 E do
env.reset( while not done do
11:
m fe 3= [s, ml
13:
5o env.step(a)
15:
end while
17:
Update 0 using PPO to maximise J end for
18:
19:
for i = 0 to / do s env.reeset(
20:
21:
while not done do
22:
m = f (x)
23:
8
24:
me 3)
25:
r. done cnv.step(a)
26:
r R
sewards.append(r>)
28:
end while
29:
end for
30:
end for
31:
Update á using ES to maximise J
32:
Update vi using ES to maximise J
33:
end for
14

---

Under review as a conference paper at ICLR 2023
Algorithm 3 Test-time Oracle PPO ACT
1: Initialize train-time ACT parameters ø
2: Obtain trained d, from Algorithm 2
3: Initialize test-time ACT parameters
4: for = 0 to I do
5: env.reset()
6: while not done do TT- 8) 8= [s, m
7:
8:
9:
(- r. done env.step(())
10:
r R
12:
rewards.appenddr;)
end while
14: Update 21" using PPO to maximise J
end for
15

---

Under review as a conference paper at ICLR 2023
Algorithm 4 Test-time Random Shaper
1: Initialize train-time ACT parameters Grandom
2: Initialize policy params 0
3: rewards
4: for = 0 to E do
5: env.reset()
6: while not done do
7:
f (8) 8 == [s. m
8:
9:
(-
10: s. envvsteppp(
end while
12: Update 9 using PPO to maximise J
end for
14: Initialize test-time ACT parameters x/*
15: for 0 to / do
16: s= s=env.reset(
17: while not done do
18:
TT- s) == [s, ml]
19:
20:
n- r. etepst)
21:
ri R$ as)
23:
wwwards.append(r))
end while
25: Update 2²" using PPO to maximise J
26: end for
16

---

Under review as a conference paper at ICLR 2023
C
ABLATIONS
Cartpole
Cartpole
200
200
150
150
Return
Return
100
Dimensions
100
Dimensions
Zeros
Dimensions
50
Dimensions
50
Allied
Dimensions
Pre-Trained Logits
1
2
Timesteps
1e4
Timesteps
1e4
(b)
Figure 7: (a) Ablations the different number of cheap talk dimensions for the Adversary in Cartpole. We find that for a low-dimensional environment like Cartpole, the Adversary does not achieve much marginal improvement from increasing the number of channels, suggesting that there may be some limit to the amount that it can harm performance. (b) Comparing the ally with an Adversary that outputs pre-trained logits in Cartpole. We find that the allied ACT still performs better, implying that it is outputting features that are more useful than logits from pre- pre-trained policy, Error bars denote the standard error across 10 seeds Victim trained against single meta-trained Adversary.
17

---

Under review as a conference paper at ICLR 2023
D PENDULUM ABLATION
Pendulum
180
Average Return
Zeros
160
No Channel
140
Random
Ally
120
Adversary
100
0
1
2
3
4
5
Timesteps
1e5
Figure 8: Interestingly, it seems like random network features improved performance in Pendulum. To make sure this was not due network initialisation effects, we ran an ablation where we removed the cheap talk channel. It achieves about the same performance channel with zeros, which implies that the performance difference is not due to network initialisation.
18

---

Under review as a conference paper at ICLR 2023
E HYPERPARAMETER DETAILS
We report the hyperparameter values used for each environment in our experiments. Our PPO implementation uses observation normalisation (a common design choice for PPO), which means that the attack is invariant to the range of values outputted over the cheap talk channel.
Table 1: Important parameters for the Cartpole environment
Parameter
Value
State Size
4
message Size
2
message Range
-2m,
2m
Number Environments
4
Maximum Grad Norm
0.5
Number Updates
32
Update Period
256
Outer Discount Factor y
0.99
Number of Epochs per Update
16
PPO Clipping e
0.2
General Advantage Estimation À
0.95
Critic Coefficient
0.5
Entropy Coefficient
0,01
Learning Rate
0.005
Population Size
1024
Number of Generations
2049
Outer Agent (OA) Hidden Layers
2
OA Size of Hidden Layers
64
OA Hidden Activation Function
ReLU
OA Output Activation Function
Tanh
Inner Agent (IA) Actor Hidden Layers
2
IA Size of Actor Hidden Layers
32
IA Number of Critic Hidden Layers
2
IA Size of Critic Hidden Layers
32
IA Activation Function Number of Rollouts
Tanh
4
19

---

Under review as a conference paper at ICLR 2023
Table 2: Important parameters for the Pendulum environment
Parameter
Value
State Size
3
message Size
2
message Range
- 2T
Number of Environments
16
Maximum Grad Norm
0.5
Number Updates
128
Update Period
256
Outer Discount Factor Y
0,95
Number of Epochs per Update
16
PPO Clipping e
0.2
General Advantage Estimation À
0.95
Critic Coefficient
0.5
Entropy Coefficient
0.005
Learning Rate
0.02
Population Size
768
Number of Generations
2049
Outer Agent (OA) Hidden Layers
2
OA Size of Hidden Layers
64
OA Hidden Activation Function
ReLU
OA Output Activation Function
Tanh
Inner Agent (IA) Actor Hidden Layers
IA Size of Actor Hidden Layers
32
IA Number of Critic Hidden Layers
1
IA Size of Critic Hidden Layers
32
IA Activation Function Number of Rollouts
Tanh
4
Table 3: Important parameters for the Reacher environment
Parameter
Value
State Size
10
message Size
4
message Range
-2m, 2m
Number of Environments
32
Maximum Grad Norm
0.5
Number of Updates
256
Update Period
128
Outer Discount Factor y
0.99
Number of Epochs per Update
10
PPO Clipping
0.2
General Advantage Estimation À
0,95
Critic Coefficient
0.5
Entropy Coefficient
0.0005
Learning Rate
0.004
Population Size
128
Number Generations
2049
Outer Agent (OA) Hidden Layers
2
OA Size of Hidden Layers
64
OA Hidden Activation Function
ReLU
OA Output Activation Function
Tanh
Inner Agent (IA) Actor Hidden Layers
2
IA Size of Actor Hidden Layers
128
IA Number of Critic Hidden Layers
2
IA Size of Critic Hidden Layers
128
IA Activation Function
ReLU
Number of Rollouts
4
20

---

Under review as a conference paper at ICLR 2023
Table 4: Important parameters for the Minatar environments
Parameter
Value
State Size
400
message Size
32
message Range
-2T,, 2n
Number of Environments
64
Maximum Grad Norm
0.5
Number Updates
1024
Update Period
256
Outer Discount Factor Y
0,99
Number of Epochs per Update
32
PPO Clipping e
0.2
General Advantage Estimation À
0.95
Critic Coefficient
0.5
Entropy Coefficient
0.01
Learning Rate
3e-4
Population Size
128
Number of Generations
256
Outer Agent (OA) Hidden Layers
2
OA Size of Hidden Layers
64
OA Hidden Activation Function
ReLU
OA Output Activation Function
Tanh
Inner Agent (IA) Actor Hidden Layers
2
IA Size of Actor Hidden Layers
256
IA Number of Critic Hidden Layers
2
IA Size of Critic Hidden Layers
256
IA Activation Function Number of Rollouts
ReLU
21

---

Under review as a conference paper at ICLR 2023
F RARL HYPERPARAMETER DETAILS
Table 5: RARL Cartpole Parameters
Parameter
Value
State Size
4
message Size
2
message Range
-2T, 2m
Maximum Grad Norm
0.5
Total Number of Adversary and Learner Updates
100
Number of Learner Update Steps per Adversary Update 8
Number of Adversary Update Steps per Learner Update
8
Update Period
256
Outer Discount Factor 7
0.99
Number of Epochs per Update
16
PPO Clipping
0.2
General Advantage Estimation À
0.95
Critic Coefficient
0.5
Entropy Coefficient
0.01
Learning Rate
0.005
Population Size
1024
Number of Generations
2049
Outer Agent (OA) Hidden Layers
2
OA Size of Hidden Layers
64
OA Hidden Activation Function
ReLU
OA Output Activation Function
Tanh
Inner Agent (IA) Actor Hidden Layers
2
IA Size Actor Hidden Layers
32
IA Number of Critic Hidden Layers
2
IA Size Critic Hidden Layers
32
IA Activation Function
Tanh
Number of Rollouts
16
22

---

Under review as a conference paper at ICLR 2023
G EXTRA VISUALISATIONS
Pluar of Thadu Trrns mih leancd Hh
Moun of Tinde Tuned with Namsta Pi
0.-
Grep Cirencaan
Chopp Cirencann
(b)
Thate Trannd with Lnamnd
Mnan of Thets Thined Sandam
0.
0.3
Croep
0.5
0.6
64
1.
0.2
0.
1.0
Crapp Oe
Ohapp Oineraan
(c)
(d)
of
0.
0.3
5.0
1.
(e)
(l)
Docar of Thatu harndd with Learne Hh
Moun of This Tained with lanstan Mi
0.-
02
1.0
CA
Lo
Choer 28 Of
Chop: 28 CeraraarO
(g)
(h)
Figure 9: We train 10 different Victims alongside the Learned ø (left column), and 10 different Victims alongside randomly generated (right column) the Pendulum environment. We show the mean of the policy output across the 10 Victims vary the value of the message in multiple randomly selected states. The learned & consistently generates similar policy outputs across different states with respect to the cheap talk channel.
23

---

Under review as a conference paper at ICLR 2023
H
ADDITIVE PERTURBATIONS
Carppole
Reacher
200
180
100
-400
Return
-600
Zeros
120
Random
30
Alled
Adversarial
0.0
0.2
0.4
0.5
0.8
1.0
Timesteps
1n4
105
Timesteps
(b)
(c)
Figure 10: Visualisations of the training curves of the Victim across different Adversaries for (a) Cartpole, (b) Pendulum, and (c) Reacher. Error bars denote the standard error across 10 seeds of Vietims trained against single trained Adversary. In this setting, the Adversary adds the perturbation the input rather than appending. Note that this allows the Adversary to conflate states and influence the optimal policy. Thus, the Adversary can harm performance far more.
Cartpole
200
150
1.00
Alled
Adversoria
0.0
0.5
2.0
1.5
2.0
2.5
3.0
Timesteps
184
(a)
Figure Visualisations of the training curves of the Victim across different Adversaries for Cartpole. Error bars denote the standard error across 10 seeds of Victims trained against a single trained Adversary. In this setting, the Adversary adds the perturbation to the useless features identified in (Vischer et al. 2021) rather than appending. It achieves similar performance to the cheap talk channel attacks.
24


Reviews:
[{"review_id": "qE_wv_ycrf", "reviewer": "Reviewer_fsi4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This study proposes an innovative strategy to attack reinforcement learning algorithms. Importantly, the attack requires minimal, information about the victim, in contrast to many other approaches that assume much more extensive inside information. Such black box approaches are necessarily less effective but ultimately much more dangerous because they can be applied in a generic fashion. ", "summary_of_the_review": "This paper addresses a timely and interesting problem in ML, namely, how to perform adversarial attacks for RL with minimal, if any, information about the victims. The paper is clearly written and shows very promising results in this direction, surpassing benchmarks and introducing a new methodology for black box RL attacks.", "strength_and_weaknesses": "Strengths\n\nWhile there has been extensive work on adversarial attacks in pattern recognition tasks, less work has been devoted overall to the problem of adversarial attacks in reinforcement learning. RL is particularly interesting for adversarial attacks because there are multiple potential attack points, beyond the input to the system itself, such as the policy, value computation, etc. \n\nMost approaches in RL attacks require significant knowledge about the algorithm or the victim. This study focuses on the more challenging (and arguably realistically more critical) form of black box attacks.\n\nFig. 5 is impressive in showing results that are near the oracle predictions for the proposed adversaries. \n\nWeaknesses\n\nRather than a weakness, this is a strength and a question. If I interpret the results correctly, the results for the proposed algorithm in the minatar breakout are particularly devastating. I was somehow expecting the results to be weaker than those in Fig. 1 given the increased complexity of the task. I guess that it is hard to compare quantitatively across completely different tasks and reward values, but is there any intuition for why the algorithm should work perhaps better in Fig. 2a? Is it somehow easier to attack minatar breakout? \n\nMost of the remaining figures focus on the simpler tasks. It would be interesting if possible to expand some of the other figures (goal conditioning, comparison to oracle, showing policies) for the more complex breakout task. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "nBiLm83Ogux", "reviewer": "Reviewer_c6pu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies adversarial attacks in RL. Unlike previous work, the paper considers adversarial attacks that append messages to the victim state observation. In the attack model, the attacker has very limited knowledge and the attacks don't intervene the the system dynamics, rewards, and remain deterministic during an episode. The contribution of the paper is proposing a new attack model called Adversarial Cheap Talk (ACT) that can achieve the attacker's goal with limited influence in the MDP and the victim.", "summary_of_the_review": "The paper studies a novel and interesting problem. But there are caveats in the problem formulation that needs to be filled (Details are provided in section \"Strength And Weaknesses\").", "strength_and_weaknesses": "Strength:\n1. The paper studies a novel and interesting problem, i.e., to craft adversarial attacks that require limited knowledge of the attacker and apply limited influence in the underlying MDP. And the crafted attacks can still achieve the adversary's objective. \n2. The paper is nicely presented and easy to follow. Experiments are illustrative.\n\n\nWeaknesses:\n\nThere are several places in the problem formulation that don't make sense, which needs further justification. \n\n1. Previous work that studies adversarial attacks in the state observation (e.g., [R1]) models the attacks by $s'=s+\\sigma$, where $s$ is the actual state, $s'$ is the observed state, and $\\sigma$ is the attack crafted. Here, the ACT adversary append a message $m(s)$ to the actual state and assume that the victim use $(s,m(s))$ as the observation to generate its actions, which does not make too much sense from a security perspective. (a) Why the victim choose $(s,m(s))$ as the input of its strategy to choose its action? What is the motive? \n(b). Of course, a RL agent does not know the model (the environment dynamics and the rewards). But the dimension of its state space is known beforehand. Appending a message is like changing the dimension of its state space, which would be easy to detect. \n\n2. Does the assumption of stationarity benefit the attacker (to achieve its goal), the victim (to avoid non-convergence), or the authors (to show experimental results that can converge)? The authors did not justify the purpose of requiring the attacker not to introduce non-stationarity. What is the incentive for the attacker to do so If the attacker can achieve its goals by introducing non-stationarity to the learning process?\n\n3. From the figure 1, when the adversary is in the \"Ally\" mode, the authors observe and state that the victim achieves higher rewards in training. But the authors did not explain why this happens? What is the root causes of it? How the attacker or the victim can leverage this phenomenon to do things? \n\n\n\n[R1] Zhang, Huan, et al. \"Robust reinforcement learning on state observations with learned optimal adversary.\" arXiv preprint arXiv:2101.08452 (2021). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gYVeojeqVtp", "reviewer": "Reviewer_HHmW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers train and test time adversarial attacks on deep reinforcement learning. Here, the attacker can append the state observed by the learning agent with messages given by a deterministic policy. In the threat model, the attacker does not influence transition dynamics or reward functions, introduce non-stationarity, or access agents' actions or parameters. The authors develop both types of attacks based on evaluation strategies (ES). The experimental evaluation considers training with PPO for popular gym environments as well as minatar.  The results show that the training-time attacks can interfere with the learning of the neural network used by the agent while test-time attacks provide direct control over the agent.", "summary_of_the_review": "Overall, the paper tackles a less explored but important problem that can promote the development of more robust RL algorithms. However, I found that the threat model and experimental details are not clearly presented due to which I consider the paper below the acceptance threshold for ICLR.", "strength_and_weaknesses": "Strengths:\n1. There do not exist enough studies on the vulnerability of deep reinforcement learning algorithms against training time attacks. New attacks as designed here can promote the development of more robust algorithms.\n\n2. The attack model presented here makes several black-box assumptions that make the attack more feasible.\n\nWeaknesses:\n1. The threat model does not limit the size of the message appended by the adversary.\n\n2. The evaluation is performed only on PPO. However, better algorithms like TD3, and SAC also exist. It is not clear if the proposed attack can affect learning by these algorithms.\n\n3. Experimental details are not clear (as described below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8rENQxhKGPm", "reviewer": "Reviewer_fBNt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies adversarial attacks in reinforcement learning under the setting where the adversary can only append to the agent’s observations. The authors propose a meta learning-based Adversarial Cheap Talk (ACT) method to generate such attacks, such that the agent’s training/test-time performance can be improved or attacked depending on the adversary’s objective. The authors also show that using ACT in training can affect test-time performance. Experiments demonstrate the effectiveness of the proposed ACT method.", "summary_of_the_review": "Overall, I think this work is interesting. Moreover, the authors propose interesting attacks in the RL setting and have discussed train-time and test-time attacks. However, there are certain issues as I mentioned above, I think the paper would be in a better shape after taking the feedback into consideration.", "strength_and_weaknesses": "#### **Strengths**:\n- The appending attack and experiment settings are interesting\n- The authors propose to leverage meta learning for generating attacks\n- The proposed method is straightforward and demonstrates effectiveness\n\n#### **Weakness**:\n- The proposed Adversarial Cheap Talk (ACT) attack match the intuition of appending attack in the RL setting, but it seems the optimization is completely based on evolution strategies (ES), which makes the technical contribution of this work incremental.\n- The two settings of train-time and test-time attacks seem to be less relevant. The former setting trains an adversary along with the victim and the latter setting uses backdoor for test-time manipulation.\n- The test-time attacks introduce backdoor in training for test-time manipulation, however, it lacks details in section 5.3 of how the backdoor is designed and leveraged to fully control the victim.\n- In the train-time influence experiment settings, the selected baseline methods (i.e., random and zero) are too naïve. For the RARL method, authors only present the results in the cartpole experiments, I would like to see more comparison in experiments like pendulum and reacher.\n- In the test-time manipulation experiments (figure 5), why the performance of different settings is not consistent? For example, why does direct oracle reach the maximum return sooner in the reacher experiments?\n- The adversary objective of test-time manipulation is to maximize the goal-conditioned scores, the setting of minimizing the return is not discussed.\n- Finally, I am curious how the proposed attacks perform against simple defense methods as the authors discussed in the last section.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Additional references from AC: 

Zhang, Huan, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. "Robust deep reinforcement learning against adversarial perturbations on state observations." Advances in Neural Information Processing Systems 33 (2020): 21024-21037.

Sun, Yanchao, Ruijie Zheng, Yongyuan Liang, and Furong Huang. "Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL." In International Conference on Learning Representations.

Shen, Qianli, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. "Deep reinforcement learning with robust and smooth policy." In International Conference on Machine Learning, pp. 8707-8718. PMLR, 2020.

Oikarinen, Tuomas, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui-Wei Weng. "Robust deep reinforcement learning through adversarial loss." Advances in Neural Information Processing Systems 34 (2021): 26156-26167.

Korkmaz, Ezgi. "Investigating vulnerabilities of deep neural policies." In Uncertainty in Artificial Intelligence, pp. 1661-1670. PMLR, 2021.

Behzadan, Vahid, and Arslan Munir. "Whatever does not kill deep reinforcement learning, makes it stronger." arXiv preprint arXiv:1712.09344 (2017).

Mandlekar, Ajay, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. "Adversarially robust policy learning: Active construction of physically-plausible perturbations." In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3932-3939. IEEE, 2017.

Pattanaik, Anay, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. "Robust Deep Reinforcement Learning with Adversarial Attacks." In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2040-2042. 2018.

Lütjens, Björn, Michael Everett, and Jonathan P. How. "Certified adversarial robustness for deep reinforcement learning." In Conference on Robot Learning, pp. 1328-1337. PMLR, 2020.

Fischer, Marc, Matthew Mirman, Steven Stalder, and Martin Vechev. "Online robustness training for deep reinforcement learning." arXiv preprint arXiv:1911.00887 (2019).

Kumar, Aounon, Alexander Levine, and Soheil Feizi. "Policy Smoothing for Provably Robust Reinforcement Learning." In International Conference on Learning Representations.

Wu, Fan, Linyi Li, Zijian Huang, Yevgeniy Vorobeychik, Ding Zhao, and Bo Li. "CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing." In International Conference on Learning Representations.

Pan, Xinlei, Chaowei Xiao, Warren He, Shuang Yang, Jian Peng, Mingjie Sun, Mingyan Liu, Bo Li, and Dawn Song. "Characterizing Attacks on Deep Reinforcement Learning." In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp. 1010-1018. 2022.

Tan, Kai Liang, Yasaman Esfandiari, Xian Yeow Lee, and Soumik Sarkar. "Robustifying reinforcement learning agents via action space adversarial training." In 2020 American control conference (ACC), pp. 3959-3964. IEEE, 2020.

Tessler, Chen, Yonathan Efroni, and Shie Mannor. "Action robust reinforcement learning and applications in continuous control." In International Conference on Machine Learning, pp. 6215-6224. PMLR, 2019.

Lee, Xian Yeow, Yasaman Esfandiari, Kai Liang Tan, and Soumik Sarkar. "Query-based targeted action-space adversarial policies on deep reinforcement learning agents." In Proceedings of the ACM/IEEE 12th International Conference on Cyber-Physical Systems, pp. 87-97. 2021.

Behzadan, Vahid, and Arslan Munir. "Vulnerability of deep reinforcement learning to policy induction attacks." In Machine Learning and Data Mining in Pattern Recognition: 13th International Conference, MLDM 2017, New York, NY, USA, July 15-20, 2017, Proceedings 13, pp. 262-275. Springer International Publishing, 2017.

Huang, Yunhan, and Quanyan Zhu. "Deceptive reinforcement learning under adversarial manipulations on cost signals." In Decision and Game Theory for Security: 10th International Conference, GameSec 2019, Stockholm, Sweden, October 30–November 1, 2019, Proceedings 10, pp. 217-237. Springer International Publishing, 2019.

Sun, Yanchao, Da Huo, and Furong Huang. "Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics." In International Conference on Learning Representations.

Zhang, Xuezhou, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. "Adaptive reward-poisoning attacks against reinforcement learning." In International Conference on Machine Learning, pp. 11225-11234. PMLR, 2020.

Rakhsha, Amin, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. "Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning." In International Conference on Machine Learning, pp. 7974-7984. PMLR, 2020.

Decision: Reject