user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CopyCAT: Taking Control of Neural Policies with Constant Attacks

Paper content:
We propose a new perspective on adversarial attacks against deep reinforcement learning agents.
Our main contribution is CopyCAT, a targeted attack able to consistently lure an agent into following an outsider's policy.
It is pre-computed, therefore fast inferred, and could thus be usable in a real-time scenario.
We show its effectiveness on Atari 2600 games in the novel read-only setting.
In the latter, the adversary cannot directly modify the agent's state -its representation of the environment- but can only attack the agent's observation -its perception of the environment.
Directly modifying the agent's state would require a write-access to the agent's inner workings and we argue that this assumption is too strong in realistic settings.
We are interested in the problem of attacking sequential control systems that use deep neural policies.
In the context of supervised learning, previous work developed methods to attack neural classifiers by crafting so-called adversarial examples.
These are malicious inputs particularly successful at fooling deep networks with high-dimensional input-data like images.
Within the framework of sequential-decision-making, previous works used these adversarial examples only to break neural policies.
Yet the attacks they build are rarely applicable in a real-time setting as they require to craft a new adversarial input at each time step.
Besides, these methods use the strong assumption of having a write-access to what we call the agent's inner state -the actual input of the neural policy built by the algorithm from the observations-.
When taking this assumption, the adversary -the algorithm attacking the agent-is not placed at the interface between the agent and the environment where the system is the most vulnerable.
We wish to design an attack with a more general purpose than just shattering a neural policy as well as working in a more realistic setting.
Our main contribution is CopyCAT, an algorithm for taking full-control of neural policies.
It produces a simple attack that is: (1) targeted towards a policy, i.e., it aims at matching a neural policy's behavior with the one of an arbitrary policy; (2) only altering observation of the environment rather than complete agent's inner state; (3) composed of a finite set of pre-computed state-independent masks.
This way it requires no additional time at inference hence it could be usable in a real-time setting.
We introduce CopyCAT in the white-box scenario, with read-only access to the weights and the architecture of the neural policy.
This is a realistic setting as prior work showed that after training substitute models, one could transfer an attack computed on these to the inaccessible attacked model (Papernot et al., 2016) .
The context is the following: (1) We are given any agent using a neuralnetwork for decision-making (e.g., the Q-network for value-based agents, the policy network for actor-critic or imitation learning methods) and a target policy we want the agent to follow.
(2) The only thing one can alter is the observation the agent receives from the environment and not the full input of the neural controller (the inner state).
In other words, we are granted a read-only access to the agent's inner workings.
In the case of Atari 2600 games, the agents builds its inner state by stacking the last four observations.
Attacking the agent's inner state means writing in the agent's memory of the last observations.
(3) The computed attack should be inferred fast enough to be used in real-time.
We stress the fact that targeting a policy is a more general scheme than untargeted attacks where the goal is to stop the agent from taking its preferred action (hoping for it to take the worst).
It is also more general than the targeted scheme of previous works where one wants the agent to take its least preferred action or to reach a specific state.
In our setting, one can either hard-code or train a target policy.
This policy could be minimizing the agent's true reward but also maximizing the reward for another task.
For instance, this could mean taking full control of an autonomous vehicle, possibly bringing it to any place of your choice.
We exemplify this approach on the classical benchmark of Atari 2600 games.
We show that taking control of a trained deep RL agent so that its behavior matches a desired policy can be done with this very simple attack.
We believe such an attack reveals the vulnerability of autonomous agents.
As one could lure them into following catastrophic behaviors, autonomous cars, robots or any agent with high dimensional inputs are exposed to such manipulation.
This suggests that it would be worth studying new defense mechanisms that could be specific to RL agents, but this is out of the scope of this paper.
In this work, we built and showed the effectiveness of CopyCAT, a simple algorithm designed to attack neural policies in order to manipulate them.
We showed its ability to lure a policy into having a desired behavior with a finite set of additive masks, usable in a real-time setting while being applied only on observations of the environment.
We demonstrated the effectiveness of these universal masks in Atari games.
As this work shows that one can easily manipulate a policy's behavior, a natural direction of work is to develop robust algorithms, either able to keep their normal behaviors when attacked or to detect attacks to treat them appropriately.
Notice however that in a sequential-decisionmaking setting, detecting an attack is not enough as the agent cannot necessarily stop the process when detecting an attack and may have to keep outputting actions for incoming observations.
It is thus an exciting direction of work to develop algorithm that are able to maintain their behavior under such manipulating attacks.
Another interesting direction of work in order to build real-life attacks is to test targeted attacks on neural policies in the black-box scenario, with no access to network's weights and architecture.
However, targeted adversarial examples are harder to compute than untargeted ones and we may experience more difficulties in reinforcement learning than supervised learning.
Indeed, learned representations are known to be less interpretable and the variability between different random seeds to be higher than in supervised learning.
Different policies trained with the same algorithm may thus lead to S → A mappings with very different decision boundaries.
Transferring targeted examples may not be easy and would probably require to train imitation models to obtain mappings similar to π in order to compute transferable adversarial examples.

assistant: text: We propose a new attack for taking full control of neural policies in realistic settings.