Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learning

## Abstract

Abstract We study the problem of universal black-boxed reward poisoning attacks against general offline reinforcement learning with deep neural networks. We consider a black-box threat model where the attacker is entirely oblivious to the learning algorithm, and its budget is limited by constraining the amount of corruption at each data point and the total perturbation. We require the attack to be universally efficient against any efficient algorithms that might be used by the agent. We propose an attack strategy called the ‘policy contrast attack.’ The idea is to find low- and high-performing policies covered by the dataset and make them appear to be high- and low-performing to the agent, respectively. To the best of our knowledge, we propose the first universal black-box reward poisoning attack in the general offline RL setting. We provide theoretical insights on the attack design and empirically show that our attack is efficient against current state-of-the-art offline RL algorithms in different learning datasets.

## 1 Introduction

Reinforcement learning (RL) in the offline setting (Levine et al., ; Agarwal et al., ) has become a promising framework for realizing reinforcement learning applications in practice. Offline RL avoids the necessity of potentially expensive online data collection and can work with offline data directly. Taking medical treatment problems as an example, collecting online data would require testing on humans, which is dangerous and infeasible. In contrast, abundant recordings of how a human doctor treats a patient are available for offline training.

A number of effective algorithms have been proposed for the offline RL problem and achieve promising empirical performance on simulated environments. (Bhardwaj et al., ; Kumar et al., ; Fujimoto & Gu, ; Kidambi et al., ; Wu et al., ; Cheng et al., ). However, a practical threat arises in the real world that is usually overlooked in the simulations. In many applications, the offline data is usually based on human feedback (Zheng et al., ; Kiran et al., ), making it possible for an adversary to poison the reward signal in the training dataset. For example, the learning agent may ask a third-party agent to provide offline data. If the third-party agent is dishonest, it may provide human data with subtle malicious changes that are hard to detect but can potentially cause unwanted results for the learning agent. The threat of poisoning attacks even draws attention in recent reinforcement learning from human feedback (RLHF) Baumgärtner et al. (); Wang et al. (); Wu et al. (), and people are concerned that such attacks may greatly influence the performance of current RLHF approaches. The state-of-the-art offline reinforcement learning algorithms mentioned above only consider ideal environments without such threats. A few works Li et al. (); Ye et al. () investigate the settings with the poisoning attack. However, they only consider the attacker adopting some random attack strategies, which may not be enough to evaluate the robustness of an algorithm properly. Therefore, it is important to investigate the vulnerabilities of current algorithms to the poisoning attacks with more efficient attack strategies.

In Fig 1, we show the framework of the attack. Specifically, we focus on a universal black-box reward-poisoning attack with a limited budget such that under the attack, any originally efficient learning algorithm can no longer learn a high-performing policy. Such an attack setting is more realistic due to the following reasons:

- 1.
Universal and Black box: First, it is not in the interest of the learning agent to reveal its algorithm to the attacker. Second, offline RL algorithms are very different in learning strategies and frameworks. Therefore, it is hard for the attacker to assume which kind of learning algorithm the agent will use. The attacker has to work without any knowledge of the algorithm used by the agent and expect the agent to choose any efficient algorithm arbitrarily.
- 2.
Reward poisoning: In RL environments, the reward signals are usually provided by the environment, such as human users, while the state signals are determined by the agent. For example, in reinforcement learning from human feedback (RLHF), the state signals are the prompts the learner gives, while the reward signals are feedback from humans. Therefore, it is more feasible for the attacker to poison rewards.
- 3.
Limited budget: Too much perturbation on the reward signals makes it easier for the learning agent to detect. So, it is more practical for the attacker to work with limited budgets and perturb the training process as little as possible.

We formally introduce our criteria for an attack to be practical in Section 3.

Challenges:
We find the following challenges for developing a practical reward poisoning attack.

- 1.
Fixed training dataset: In the online learning setting, an attacker can mislead the learner to collect data that only covers states and actions with low long-term rewards so that the learner never observes how a high-performing policy would work, making it fail to learn a high-performing policy (Xu et al., ). In contrast, the training dataset is fixed in the offline learning setting. This results in the famous ‘pessimistic’ learning (Levine et al., ) in offline RL, and the learning agent is inclined to stick to the behaviors covered by the dataset. The learner can always observe how an expert would behave (the action) in the dataset regardless of the attack. This contributes to the fundamental difficulty of misleading the learning agent.
- 2.
Universal and Black box attack: A practical attacker should be oblivious to the learning agent and universal efficiency against any algorithms it might use. First, it is challenging to list all possible efficient learning algorithms that an agent might use. Second, without knowing the details of an offline learning algorithm, it would be challenging to predict the exact learning result of the algorithm under the attack.

Contributions: To address the challenges above, our insight is to characterize general efficient offline RL algorithms with the pessimistic learning feature. For a pessimistic algorithm, it will find nearly the best policy well covered by the dataset. Based on the insight, we construct an attack framework called the ‘adversarial reward engineering framework’ such that one can evaluate the efficiency of an attack against universal efficient algorithms in the framework. To find an efficient attack under the framework, the key idea of our attack is to find the high-performing policies covered by the dataset and make them appear low-performing to the agent and vice versa. In this case, the algorithm will believe that some poor-performing policies are near optimal. We build our main attack method, ‘policy contrast attack,’ based on the idea and theoretically analyze the insight behind it based on general assumptions. To the best of our knowledge, this is the first universal black-box reward poisoning attack in the general offline RL setting. We empirically test the efficiency of our attack on various standard datasets from the D4RL benchmark (Fu et al., ) learned by different state-of-the-art offline RL algorithms (Kumar et al., ; Fujimoto & Gu, ; Kostrikov et al., ). Our attack is efficient in most cases with a limited attack budget. We also show that our attack is not sensitive to the choice of hyper-parameters. One can use our attack to assess the practical robustness of offline RL algorithms. We hope that our work can inspire the development of more robust offline RL algorithms.

Figure: Figure 1: Reward poisoning attack framework.
Refer to caption: extracted/5949811/Figures/intro.png

## 2 Related Work

Reward poisoning attack in online RL:
Most studies on reward poisoning attacks against RL focus on the online learning setting. Zhang et al. (); Huang et al. () investigate observation perturbation attack where the attacker modifies the state signals to adversarial examples during training. Rakhsha et al. (); Zhang et al. () theoretically investigate the attack problem in the online RL setting with tabular MDPs. They show that given a specific learning algorithm or an algorithm with certain learning guarantees, attack strategies exist such that the learning algorithm will learn a low-performing policy even if the attack only slightly perturbed the training process. However, many of these are white-box attacks, and it remains unknown how to scale these techniques to general RL environments that require function approximation with deep neural networks. In the more complicated general RL setting, Sun et al. () empirically shows that current state-of-the-art DRL algorithms are vulnerable under the poisoning attack when the attacker is aware of the learning algorithm. Xu et al. (); Xu & Singh () further shows that even in the black-box setting where the attacker is oblivious to both the environment and the learning algorithm, the data poisoning attack can achieve targeted and untargeted attack goals with limited budgets.

Reward poisoning attack and defense in offline RL:
There is limited work on the poisoning attack in offline RL. Ma et al. () theoretically studies the problem for tabular MDPs that cannot scale to the more complicated general RL setting considered in our work. Furthermore, their attack assumes that the learning agent can learn the globally optimal policy, which is not appropriate and practical for recent offline RL algorithms. Li et al. () theoretically and empirically shows that the learning efficiency of an efficient offline RL algorithm is related to the dataset quality regardless of the reward signals, which is closely related to our assumption in Section 4. To defend against the attack, Zhang et al. (); Ye et al. () design algorithms and theoretically prove their robustness. They also empirically test their algorithm against some specific random attack strategies. We empirically show that our attack is much more efficient than such random attacks for different learning scenarios. No attack exists for general offline RL except for the random attacks. Still, we also adapt a state-of-the-art online RL attack from Xu et al. () to our setting as an additional baseline.

## 3 Preliminaries

Offline RL:
We consider a standard offline RL setting (Cheng et al., ) where an agent trains on an offline dataset to learn how to perform well in an RL environment. An RL environment is characterized by an MDP $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})$ where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{P}=\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ is the state transition function, and $\mathcal{R}=\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function. A policy $\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$ is a mapping from the state space to a distribution over action space suggesting the way one behaves in the environment. Without loss of generality, we assume the initial state is always $s_{0}$. Then the performance of a policy $\pi$ for the environment is defined as $J_{\mathcal{R}}(\pi)=\mathbb{E}_{\pi,\mathcal{P}}[\sum_{t=0}^{\infty}\gamma^{t
}\mathcal{R}(s_{t},a_{t})|a_{t}\sim\pi(s_{t})]$, which represents the long-term discounted cumulative rewards for running the policy in the environment.

An offline reinforcement learning dataset is a collection of observation tuples collected from the environment: $\mathcal{D}=\{(s_{i},a_{i},s^{\prime}_{i},r_{i}),i=[0,\ldots,N]\}$ where $s^{\prime}_{i}\sim\mathcal{P}(s_{i},a_{i})$ and $r_{i}\sim\mathcal{R}(s_{i},a_{i})$. The state action distribution $\mu=\{(s_{i},a_{i}),i=[0,\ldots,N]\}$ represents the trajectories covered in the dataset. An offline reinforcement learning agent has access to the offline dataset $\mathcal{D}$ as well as the state and action spaces $\mathcal{S},\mathcal{A}$, and its goal is to find a high-performing policy.

Reward poisoning attack against offline RL:
We consider a reward poisoning attack model where a malicious adversary can poison the reward signals in the offline dataset to mislead the learning agent. Formally, for the $i^{\text{th}}$ observation tuple $(s_{i},a_{i},s^{\prime}_{i},r_{i})$, the attacker can inject a perturbation $\Delta_{i}$ to the reward signal, then the corrupted observation tuple becomes $(s_{i},a_{i},s^{\prime}_{i},r_{i}+\Delta_{i})$. We denote $\mathbf{\Delta}={\Delta_{1},\ldots,\Delta_{N}}$ to be the corruption strategy of the attack.For a practical threat model, the attacker should have limited abilities. Formally, we consider the following constraints on the attacker.

- 1.
No access to the offline training process: Since the training is offline, the learning agent can host the training process on local machines. Therefore, it would be impractical for the attacker to observe the training process of the learning agent.
- 2.
No knowledge of the learning algorithm: There are a variety of efficient offline RL algorithms with very different learning strategies. Unless the learning agent announces its learning algorithm to the public, it would be impractical for the attacker to assume which learning algorithm the agent would choose. Therefore, the attacker should not have any detailed knowledge of the learning algorithm.
- 3.
Limited Budget: To make the attack stealthy, the attacker should make the corruption as small as possible. First, the attacker should limit the maximal perturbation on each reward signal $||\mathbf{\Delta}||_{\infty}=\max|\Delta^{i}|$, as an extreme value of reward would make the data point very suspicious. Second, the attacker should limit the total amount of perturbation $||\mathbf{\Delta}||_{1}=\sum_{i=0}^{N}|\Delta^{i}|$. The reason is that too many slightly problematic data points can also make the agent realize that the dataset is unreliable.

Given a fixed budget, the general goal of the attacker is to make the learning agent learn a low-performing policy. We will formally formulate the attacker’s goal as an optimization problem in the next section.

## 4 Policy Contrast Attack

In this section, we formally define and formulate the universal black-box offline RL attack as an optimization problem. We consider the attacks that are universally effective against any data-driven learning algorithms of high learning efficiency. We highlight the challenges in finding the efficiency of an arbitrary attack and focus on a framework called the ‘adversarial reward engineering framework’ where the behavior of any efficient algorithm becomes predictable under any instance of the framework. Finally, we present our attack algorithm, ‘policy contrast attack,’ which is an efficient instance under the framework.

### 4.1 Universal Attack on General Offline Reinforcement Learning as An Optimization Problem

Here, we formally formulate the universal black-box offline RL attack as an optimization problem. Let $\mathcal{L}$ be a class of offline RL algorithms that a learning agent might use. Let $\pi^{\text{Alg}}(\mathcal{D}\oplus\mathbf{\Delta})$ be a random variable representing the policy learned by a learning algorithm Alg training on the dataset $\mathcal{D}$ with corruption $\mathbf{\Delta}$. By applying a limited amount of perturbation, the attacker should make the agent learn a low-performing policy no matter which algorithm it chooses from $\mathcal{L}$.

$$ $\displaystyle\min_{\Delta_{1:N}}V$ (1) $\displaystyle\text{s.t.},\forall\text{Alg}\in\mathcal{L},\mathbb{E}[\mathcal{J }_{\mathcal{R}}(\pi^{\text{Alg}}(\mathcal{D}\oplus\mathbf{\Delta}))]\leq V$ $\displaystyle\|\mathbf{\Delta}\|_{0}\leq B,\|\mathbf{\Delta}\|_{1}\leq C$ $$

The problem is not complete yet as we have not specified $\mathcal{L}$, the class of algorithms an agent might use. The problem becomes trivial if $\mathcal{L}$ includes all possible policies. In this case, the agent can choose an algorithm that outputs a specific policy with maximal performance regardless of the reward signals poisoned by the attack. Then, all attacks that satisfy the budget constraints are ‘optimal’ solutions. As a result, it is meaningless for the attacker to consider all possible algorithms, especially the ones that are not reward-driven and inefficient. A rational learning agent will only consider using an algorithm that ‘efficiently’ learns the reward signals to find a decent policy based on the dataset. Therefore, it is more important for an attack to focus on the effect of the attack against such algorithms. For this purpose, in this work, we assume $\mathcal{L}$ to include all $\delta$-optimal pessimistic learning algorithms in Assumption 4.1 below.

###### Assumption 4.1 .

($\delta$-optimal pessimistic learning algorithms assumption)
Given an offline RL dataset $\mathcal{D}$ collected from an environment $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})$ with state-action distribution $\mu$. For all efficient offline RL algorithm $\text{Alg}\in\mathcal{L}$ that might be used by the agent, there exists a small value $\delta>0$ and a class of policies $\Pi_{\mu}$ supported by the distribution $\mu$ such that $J_{\mathcal{R}}(\pi^{\text{Alg}}(\mathcal{D}))\geq\max_{\pi\in\Pi_{\mu}}J_{
\mathcal{R}}(\pi)-\delta$. That is, the agent will always use an algorithm that can learn the policy with nearly the highest performance on $\mathcal{M}$ among the supported policies.

Here, we justify that Assumption 4.1 is practical and can represent current efficient offline RL algorithms. It has become common sense that pessimistic learning is the key to efficiency in offline RL Levine et al. (). A pessimistic learning algorithm avoids the policies not supported by the dataset to minimize the chance of reward hacking, that is, outputting a policy that appears promising by the dataset but has poor performance. Li et al. () has the same insight in explaining the ‘survival instinct’ of existing offline RL algorithms against attack: a pessimistic algorithm will stick to the policies covered by the dataset so that it won’t output any low-performing and out-of-distribution policies regardless of the reward signals.

By the optimization problem in Eq 1, we can naturally define the efficiency of an attack as follows.

###### Definition 4.2 .

($(V,B,C)$-efficient attack)
An attack is $(V,B,C)$-efficient if it satisfies the following conditions:

$$ $\displaystyle\forall\text{Alg}\in\mathcal{L},(\mathcal{D}\oplus\mathbf{\Delta} ))\leq V$ (2) $\displaystyle\|\mathbf{\Delta}\|_{0}\leq B,\|\mathbf{\Delta}\|_{1}\leq C$ $$

Given attack budgets $B$ and $C$, finding the optimal attack for Eq 1 is equivalent to finding the most efficient $(V,B,C)$-efficient attack with the minimal value of $V$. Given an arbitrary attack strategy, the attack budgets $B$ and $C$ can be computed by the attack construction. However, evaluating the learning outcome of the agent under the attack is challenging. First, the influence of a universal attack should be evaluated on all algorithms from $\mathcal{L}$, and it is challenging to find all the algorithms that satisfy the near-optimal efficient learning conditions. Second, evaluating the exact influence of an attack on a specific learning algorithm is challenging in the black-box setting, as the neural network used by the agent is unknown to the attacker. In conclusion, it is challenging to evaluate the exact efficiency of an arbitrary attack; hence, it is also challenging to find the optimal attack. To deal with the challenges and find an attack of high efficiency, we adopt an attack framework where one can analyze the efficiency of the attack instances under the framework.

###### Assumption 4.1 .

###### Definition 4.2 .

### 4.2 Adversarial Reward Engineering Framework

###### Definition 4.3 .

(Adversarial reward engineering framework)
In the adversarial reward engineering framework, an instance of attack constructs an adversarial reward function $\widehat{\mathcal{R}}$. The corresponding corruption on the $i^{\text{th}}$ data $(s_{i},a_{i},r_{i})$ from $\mathcal{D}$ satisfies $\Delta_{i}=\widehat{\mathcal{R}}(s_{i},a_{i})-r_{i}$.

In the adversarial reward engineering framework, it is equivalent to say the corrupted dataset is collected from the adversarial environment $\widehat{\mathcal{M}}=(\mathcal{S},\mathcal{A},\mathcal{P},\widehat{\mathcal{R
}})$ with the same state-action distribution $\mu$ as the original dataset. Therefore, we can apply assumption 4.1 to characterize the behavior of the agent under the attack: it will learn a supported policy that is near-optimal on the adversarial reward function $\widehat{\mathcal{R}}$. Formally, the efficiency of an instance of adversarial reward engineering framework is guaranteed in Lemma 4.4 below.

###### Theorem 4.4 .

Let $\widehat{\mathcal{R}}$ be the adversarial reward of an instance of the adversarial reward engineering framework. Let $\hat{\Pi}^{*}:=\{\pi|\pi\in\Pi_{\mu},J_{\widehat{R}}(\hat{\pi}^{*})\geq\max_{
\pi\in\Pi_{\mu}}J_{\widehat{\mathcal{R}}}(\pi)-\delta\}$ be the $\delta$-optimal supported policies in $\widehat{\mathcal{R}}$. The efficiency of the attack satisfies $V\leq\max_{\pi\in\hat{\pi}^{*}}J_{\mathcal{R}}(\pi)$, $B=\max_{i}|\widehat{\mathcal{R}}(s_{i},a_{i})-r_{i}|$ and $C=\sum_{i}|\widehat{\mathcal{R}}(s_{i},a_{i})-r_{i}|$.

The proof for all theorems and lemmas can be found in the Appendix. Recent works about robust offline RL widely consider an attack we call ‘random inverted reward attack’ for empirical evaluation against their learning algorithms Zhang et al. (); Ye et al. (); Li et al. (). Such an attack randomly flips the signs of a portion of rewards in the datasets and can be treated as an instance of the adversarial reward engineering framework. One can find a detailed analysis in the Appendix. Next, we show a way to construct a more efficient instance of the adversarial reward engineering framework and justify the design of our policy contrast attack.

###### Definition 4.3 .

###### Theorem 4.4 .

### 4.3 Towards Efficient Adversarial Reward Engineering Attack: Policy Contrast Attack (PCA)

To ensure that the agent learns a low-performing policy, an efficient instance of the adversarial reward engineering attack needs to make the optimal or near-optimal supported policies in the adversarial environment have poor performance in the true environment. So, our goal is to find a way to construct such an adversarial reward model while limiting the required attack budget. However, it is still impractical to find all near-optimal supported policies and their performance in the true environment as the environment dynamics are unknown to the attacker. Therefore, we focus on finding a feasible way to construct a reward function such that any near-policy policies in the reward function are likely to perform poorly in the true environment.

In Theorem 4.5, we show a sufficient condition and a necessary condition for the adversarial reward function to make the agent learn a low-performing policy with performance less than $V$. The two conditions are the same when $\delta=0$.

###### Theorem 4.5 .

Consider an attack in the adversarial reward engineering framework with $\widehat{\mathcal{R}}$. To make the actual performance of the learned policy by an arbitrary $\delta$-optimal pessimistic learning algorithm lower than a value $J_{\mathcal{R}}(\pi_{0})<V$, a sufficient condition is that the adversarial reward function satisfies

$$ $\displaystyle\exists\pi_{1}\in\Pi_{\mu}\text{. }J_{\mathcal{R}}(\pi_{1})<V, \forall\pi_{2}\in\Pi_{\mu}.J_{\mathcal{R}}(\pi_{2})\geq V,$ $\displaystyle J_{\widehat{\mathcal{R}}}(\pi_{1})>J_{\widehat{\mathcal{R}}}(\pi _{2})+\delta,$ $$

and a necessary condition that it should satisfy is

$$ $\displaystyle\exists\pi_{1}\in\Pi_{\mu}\text{. }J_{\mathcal{R}}(\pi_{1})<V, \forall\pi_{2}\in\Pi_{\mu}.J_{\mathcal{R}}(\pi_{2})\geq V,$ $\displaystyle J_{\widehat{\mathcal{R}}}(\pi_{1})>J_{\widehat{\mathcal{R}}}(\pi _{2})-\delta,$ $$

In both cases, we call $\pi_{1}$ that satisfies the conditions ‘bad policy’ and $\pi_{2}$ ‘good policy.’

Theorem 4.5 suggests that the attacker should make a low-performing policy have a higher performance in the adversarial reward function than any high-performing policies. Formally, denote $\Delta J(\pi)=J_{\widehat{\mathcal{R}}}(\pi)-J_{\mathcal{R}}(\pi)$, to make $J_{\widehat{\mathcal{R}}}(\pi_{1})>J_{\widehat{\mathcal{R}}}(\pi_{2})+\delta$, the attacker needs to ensure that $\Delta J(\pi_{1})-\Delta J(\pi_{2})>J(\pi_{2})-J(\pi_{1})+\delta$. In other words, the attacker needs to make $\Delta J(\pi_{1})$ high for some bad policy $\pi_{1}$ and $\Delta J(\pi_{2})$ low for any good policy $\pi_{2}$. To limit the cost of the attack, the attacker should also determine the rewards to poison that have more influence on $\Delta J(\pi_{1})$ and $\Delta J(\pi_{2})$. This directly inspires the key idea for finding an efficient attack that in the adversarial reward function, the rewards for some low-performing policies are increased so that they appear as high-performing and vice versa.

In terms of finding the bad policies, we show in Lemma B.2 that the attacker can find nearly the worst policy supported by the dataset by training on the dataset of all rewards inverted.

###### Lemma 4.6 .

When training on the dataset $\mathcal{D}$ with the signs of all rewards inverted, the learning agent will learn the policy supported by the dataset with nearly the worst performance $V\leq J_{\mathcal{R}}(\pi_{0})=\min_{\pi\in\Pi_{\mu}}J_{\mathcal{R}}(\pi)+\delta$.

In terms of finding the good policies, Lemma 4.7 shows that policies of high performance will behave similarly to some policies from the good policy set constructed through Alg 2.

###### Lemma 4.7 .

Let $V$ be the performance of the learned good policy in the last iteration of Alg 2. For any policy $\pi\in\Pi_{\mu}$ such that $\min_{s\in\mathcal{S},\pi_{2}\in\Pi_{2}}d_{a}(\pi(s),\pi_{2}(s))>d$, its performance satisfies $J_{\mathcal{R}}(\pi)<V+\delta$.

After finding the good and bad policies, the final step is to make them look bad and good in the adversarial environment. This can be achieved straightforwardly by decreasing or increasing the rewards associated with the actions given by the good or bad policies.

Based on the discussion above, we formulate ‘policy contrast attack.’ The attack consists of two parts as follow:

- 1.
Bad policies look good: The attacker trains on the training dataset with all rewards inverted and learns ‘bad’ policies. Then, the attacker poisons the dataset by increasing the rewards associated with the actions close to that given by the ‘bad’ policies.
- 2.
Good policies look bad: The attacker iteratively learns a set of ‘good’ policies with Algorithm 2. Then, the attacker poisons the dataset by decreasing the rewards associated with the actions close to that given by the ‘good’ policies.

Formally, the policy contrast attack is given in Alg 1. Here $\Delta_{1}$, $\Delta_{2}$, and $d$ are the hyper-parameters of the attack and set to utilize given attack budgets $B$ and $C$ fully. $d_{a}$ is a measure of the distance between actions. For the actions from a continuous space, we choose $d_{a}(a_{1},a_{2})=||a_{1}-a_{2}||_{2}$. The corresponding adversarial reward function in Alg 1 satisfies:

$$ $\widehat{\mathcal{R}}(s,a)=\begin{cases}\mathcal{R}(s,a)+\Delta_{1}\cdot(1- \frac{d_{a}(a,\pi_{1}(s))}{d}),&\text{if }d_{a}(a,\pi_{1}(s_{2}))\leq d\\ -\Delta_{2},&\text{else if }\min_{\pi_{2}\in\Pi_{2}}d_{a}(a,\pi_{2}(s))\leq d \\ \mathcal{R}(s,a),&\text{otherwise}\end{cases}$ $$

Figure: Algorithm 1 Policy Contrast Attack

The first part of the reward function is to increase the rewards associated with the bad actions given by the bad policy. The formation of the reward modification in this part follows Xu & Singh (). The penalty on the reward is a linear function depending on the distance between an action and the bad policy’s action. This is to make it easier for the agent to converge to the bad policy. The second part of the reward function is to decrease the rewards associated with good policies. The penalty here is simply making the reward a low value, as in this case, we merely want the agent to learn something different from the good policies.

Note that there are some cases where the actions given by the good and bad policies are close. There will be a conflict for the aforementioned corruption strategy in this case. Intuitively, these cases are less common as usually good and bad actions should not be similar. In addition, in these cases, a change in reward has a similar influence on the performances of good and bad policies in the adversarial function, suggesting that the corruption in these cases has a limited impact on the ranking of good and bad policies in the adversarial reward. To ensure that the bad policy always performs better in the adversarial environment, we always increase the reward associated with the bad actions whenever there is a conflict.

###### Theorem 4.5 .

###### Lemma 4.6 .

###### Lemma 4.7 .

## 5 Experiments

### 5.1 Performance of Different Attack Strategy

We test the effectiveness of our attack against current state-of-the-art offline RL algorithms training on the standard D4RL offline RL benchmarkFu et al. (). We show that with limited attack budgets, our policy contrast attack is generally more effective.

Experiment setup: The offline datasets we choose from the D4RL benchmark include different combinations of environments and types of trajectories. We consider HalfCheetah, Hopper, Walker2d, and Ant environments. We consider the data collected by a single medium policy (medium), a single expert policy (expert), a single random policy (random), a mixture of medium and expert policies (medium-expert), and the replay buffer of a medium policy (medium-replay).

For the learning algorithms, we choose IQL  Kostrikov et al. (), CQL Kumar et al. (), and TD3-BC Fujimoto & Gu (), which are frequently used as baselines in current offline RL studies. These algorithms are implemented based on CORL library Tarasov et al. (). Recall that the attacker also needs a learning algorithm to learn good and bad policies. To ensure that the attacker has no information about the learning agent, the learning agent and the attacker always use different learning algorithms.

We choose three baseline attacks. The first one is the random invert attack, the only non-trivial attack against offline RL in the current literature. The second and the third ones are the random policy invert (RPI) attack and random policy promote (RPP) attack proposed in Xu et al. () for the online setting, as it is the only existing attack closely related to and can be adapted to work in the offline setting. In the appendix, we show the result of using only one part of the attack strategy from the policy contrast attack.

For the attack budget, following Ye et al. (), we always set the budget for $B$ and $C$ to be the amount required by the random inverted reward attack that corrupts $30\%$ of the data. In the ablation study, we tested the influence of different attack budgets on the attack’s efficiency.

For the hyper-parameters of the attack, the corruption parameters $\Delta_{1}$ and $\Delta_{2}$ are set to fully utilize the budget $B$, and the attack radius $d$ is set to fully use the budget $C$. The number of good policies, i.e., the number of iterations $K$, is set as $5$. In the appendix, we show that the algorithm’s efficiency is not sensitive to the choice of corruption parameters and the number of good policies. We run $5$ times for each experiment with different random seeds and show the average result.

Note that the construction of our attack does not depend on the assumption we made for the learning algorithms. Although the learning algorithms may not strictly satisfy the assumption of efficient learning algorithms, we show that our attack is still effective.

Main Results:
The offline RL algorithms we choose iteratively update their learned policies. To straightforwardly show the effectiveness of attacks, we use the performance of the policy learned by the agent at the end of training to represent its learning efficiency. In the appendix, to intuitively show the details of the training process under the attacks, we show the performance of the policies learned by the algorithm at each iteration during training. We will also show the full training log for the experiments in the ablation studies.

**Table 1: Performance of learning algorithms under the attacks.**
| D4RL Dataset | Normal | Fully inverted | PCA | Rand 30% | RPP | RPI |
| --- | --- | --- | --- | --- | --- | --- |
| HalfCheetah-ME - TD3_BC | 91.0 $\pm$ 2.9 | 13.1 $\pm$ 1.9 | 76.2 $\pm$ 2.1 | 89.8 $\pm$ 3.2 | 88.95 $\pm$ 2.65 | 81.3 $\pm$ 5.1 |
| Hopper-ME - TD3_BC | 105.9 $\pm$ 3.3 | 16.9 $\pm$ 6.3 | 82.1 $\pm$ 6.4 | 99.8 $\pm$ 8.9 | 88.22 $\pm$ 8.37 | 97.4 $\pm$ 6.9 |
| HalfCheetah-MR - IQL | 44.3 $\pm$ 0.4 | 1.07 $\pm$ 0.1 | 37.8 $\pm$ 3.2 | 43.5 $\pm$ 0.8 | 1.72 $\pm$ 0.98 | 43.9 $\pm$ 1.1 |
| Walker2D-M - CQL | 81.8 $\pm$ 0.7 | 67.7 $\pm$ 2.0 | 73.1 $\pm$ 0.2 | 77.4 $\pm$ 2.1 | 77.45 $\pm$ 1.66 | 80.1 $\pm$ 0.4 |
| Walker2D-MR - IQL | 80.2 $\pm$ 3.6 | -0.2 $\pm$ 0.1 | 31.9 $\pm$ 1.5 | 78.7 $\pm$ 5.5 | 60.21 $\pm$ 3.8 | 37.5 $\pm$ 2.8 |
| Hopper-MR - CQL | 85.2 $\pm$ 8.1 | 0.7 $\pm$ 0.01 | 30.3 $\pm$ 4.7 | 25.1 $\pm$ 4.9 | 86.05 $\pm$ 13.64 | 19.2 $\pm$ 10.3 |
| Sum Totals | 488.4 | 99.27 | 331.4 | 414.3 | 402.6 | 359.4 |

To cover as diverse offline datasets as possible, here we use only one learning algorithm for each dataset. Next, we will show the case where we test the performance of different learning algorithms on the same dataset poisoned by our attack. Table 1 shows the main results of running our policy contrast attack in different learning scenarios. The naive baseline is the training with no attack. The ideal effect of the attack is the training with fully inverted rewards, as the learning algorithm tries to learn the worst-performing policy in this case. We observe that our policy contrast attack is usually effective for any combination of datasets and learning algorithms. In contrast, the random inverted attack has almost no effect on the learning result except for the Hopper-Medium Replay dataset, where both attacks are similarly effective. In many cases, our attack is also significantly more efficient than the RPI and RPP baseline attacks.

Here, we briefly discuss why the policy contrast attack should be more efficient than other baseline attacks. 1. random inverted attack: At a high level, both the policy contrast and the inverted reward attack want to inverse the agent’s ranking of good and bad policies. The inverted reward attack treats all state actions equally and inverse the performance of all policies. In contrast, the policy contrast attack focuses on the policy and the more important state actions. This could be the reason for making the policy contrast attack more efficient in terms of budget. 2. RPI and RPP attacks: The PCA attack focuses on the supported policies, while the RPI and RPP attacks focus on arbitrary policies. Since the efficient learning algorithm mostly considers only supported policies, the poisoning of such policies should have more influence on the learning process.

### 5.2 Universal Attack and Robustness Evaluation

To verify that our attack is universal against different learning algorithms, we take the Hopper-medium-expert dataset as an example, apply the policy contrast attack using the same setup as in the main results, and run four different learning algorithms including TD3 BC, CQL, AWAC, and ReBRAC on the corrupted dataset. The results in Figure 2 show that our attack is always effective against all four learning algorithms.

In addition, an important direct application of our attack strategy is using the attack to evaluate the robustness of an algorithm. In this example, we find that compared to other algorithms, the AWAC algorithm performs much better under our attack, suggesting that it is more robust.

Figure: Figure 2: Peformance of different learning algorithms on the same dataset under the attacks.
Refer to caption: extracted/5949811/Figures/robustness.png

### 5.3 Influence of Attack Budget

Attack budget $B$: Here we study the influence of per-step attack budget $B$ on the attack efficiency. Denote $r_{\text{max}}=\max_{i}|r_{i}|$ as the maximal absolute reward from the dataset. In the main results, the budget required by the attacks is $B=2*r_{\text{max}}$. Here, we set the value of $B$ to be $1.5\cdot r_{\text{max}}$ and $1\cdot r_{\text{max}}$ and test on two randomly chosen datasets. The results in Fig 3 show that our policy contrast attack remains efficient with a lower budget of $B$. We note that the attack is slightly more efficient with less value of $B$. Intuitively, although the per-step corruption is less, the attack influences the performance of more policies, making it possible for the agent to learn slightly worse policies. The result also suggests that the efficiency of our attack is not significantly affected by the choice of the corruption parameters $\Delta_{1}$ and $\Delta_{2}$ at each step, as their values depend on $B$.

Figure: Figure 3: Influence of different $B$ budget on the attack.
Refer to caption: extracted/5949811/Figures/B.png

Attack budget $C$:
Here, we study the influence of attack budget $C$, the total amount of corruption, on the efficiency of the attack. We set the value of $C$ to be the amount required by a random inverted attack that corrupts $20\%$ and $40\%$ of data. The results in Fig 4 show that our attack is more efficient with a higher budget of $C$ and remains effective with a lower budget of $C$.

We also study the effect of other hyperparameters on our method. We find that the size of the good policy set has a very small influence on its efficiency. If our method only has the part of making good policy look bad or making bad policy look good, it will be much less efficient than the full method. The details of empirical results can be found in the Appendix.

Figure: Figure 4: Influence of different $C$ budget on the attack.
Refer to caption: extracted/5949811/Figures/C.png

## 6 Conclusion and Limitation

In this work, we propose the first black-box reward poisoning attack in offline reinforcement learning based on our novel theoretical insights. Our empirical results suggest that the attack is effective in various learning scenarios. However, our work is limited in that we only consider black-box attacks, and the attacker requires access to the full offline training dataset.

## 7 Reproducibility

In the main paper, we explain the setting of the problem we study and the threat model we consider. The proofs for all theorems and lemmas can be found in the appendix. The codes we use for the experiments can be found in the supplementary materials.

## Appendix A Proof for Theorems and Lemmas

Proof for Theorem 4.4
By the definition of the attack, the perturbation on the $i^{\text{th}}$ data is $\Delta_{i}=\widehat{\mathcal{R}}(s_{i},a_{i})-r_{i}$. By Assumption 4.1, the agent should learn a near-optimal policy on the reward function of the dataset, which is $\widehat{\mathcal{R}}$ under the attack. Therefore, the policy $\hat{\pi}^{*}$ learned by the agent under the attack satisfies $J_{\widehat{R}}(\hat{\pi}^{*})\geq\max_{\pi\in\Pi_{\mu}}J_{\widehat{\mathcal{R
}}}(\pi)-\delta$.

Proof for Theorem B.2
The adversarial reward function constructed by the fully inverted attack is $\widehat{\mathcal{R}}=-\mathcal{R}$. By Assumption 4.1, the agent will learn a policy $\pi_{0}$ that is near-optimal on $\widehat{\mathcal{R}}$: $J_{\widehat{\mathcal{R}}}(\pi_{0})\geq\max_{\pi\in\Pi_{\mu}}J_{\widehat{
\mathcal{R}}}(\pi)-\delta$. Therefore, the performance of the learned policy on the true reward function satisfies $J_{\mathcal{R}}(\pi_{0})=-J_{\widehat{\mathcal{R}}}(\pi)\leq-(\max_{\pi\in\Pi_
{\mu}}J_{\widehat{\mathcal{R}}}(\pi)-\delta)=\min_{\pi\in\Pi_{\mu}}J_{\mathcal
{R}}(\pi)+\delta$.

Proof for Theorem 4.5
First, we prove the sufficient condition. For any policy $\pi_{2}\in\Pi_{\mu}:J_{\mathcal{R}}(\pi_{2})>V$, we have $\max_{\pi\in\Pi_{\mu}}J_{\widehat{\mathcal{R}}}(\pi)\geq J_{\widehat{\mathcal{
R}}}(\pi)\geq J_{\widehat{\mathcal{R}}}(\pi_{2})+\delta$. Therefore, $\pi_{2}$ is not a near-optimal policy on the adversarial reward function, and by Assumption 4.1, the agent will not learn such a policy.

Next, we prove the necessary condition. Let $\pi_{0}$ be the policy learned by the agent under the attack, and this policy has a performance less than $V$ on the actual reward function $J_{\mathcal{R}}(\pi_{0})<V$. Therefore, we can take $\pi_{1}=\pi_{0}$. By Assumption 4.1, the gap between the performance of $\pi_{1}$ and any other policy on $\widehat{\mathcal{R}}$ must be less than $\delta$. Therefore, for any policy $\pi_{2}\in\Pi_{\mu}:J_{\mathcal{R}}(\pi_{2})>V$, we have $J_{\widehat{\mathcal{R}}}(\pi_{1})>J_{\widehat{\mathcal{R}}}(\pi_{2})-\delta$.

Proof for Theorem 4.7
At the last iteration of Alg 2, the performance of the policy $\pi_{K}$ learned in the iteration on the adversarial reward function $\widehat{\mathcal{R}}_{K}$ in the iteration is no greater than $V$ because the adversarial reward function is strictly less than the actual reward function. By Assumption 4.1, the performance of any policy $\pi\in\Pi_{\mu}$ on $\widehat{\mathcal{R}}_{K}$ satisfies $J_{\widehat{\mathcal{R}}_{K}}(\pi)\leq V+\delta$. For any policy $\pi\in\Pi_{\mu}:\min_{\pi_{2}\in\Pi_{2},s}||\pi(s)-\pi_{2}(s)||_{2}>d$, their performance on the actual reward function and the adversarial reward function are the same. Therefore, their actual performance must be less than $V+\delta$.

## Appendix B Inverted Reward Attack

Here, we discuss the most popular offline reward poisoning attack widely considered in recent RL studies Ye et al. (); Zhang et al. (); Li et al. (). We start from an illustrative example first to help better understanding the attack. Following the idea of reward engineering, one can immediately find an attack that can make the agent learn the worst possible policy under Assumption 4.1.

###### Definition B.1 (Fully inverted reward attack) .

The fully inverted reward attack sets $\widehat{\mathcal{R}}=-\mathcal{R}$ as the adversarial reward function where $\mathcal{R}$ is the true reward function of the underlying environment behind the dataset. The corresponding attack strategy of the fully inverted reward attack satisfies $\Delta_{i}=-2\cdot r_{i}$.

The inverted reward attack flips the sign of all the rewards in the dataset. In Theorem B.2 we show the efficiency of the fully inverted reward attack.

###### Theorem B.2 .

Under the fully inverted reward attack, the learning agent will learn the policy supported by the dataset with nearly the worst performance $V\leq J_{\mathcal{R}}(\pi_{0})=\min_{\pi\in\Pi_{\mu}}J_{\mathcal{R}}(\pi)+\delta$.
The budgets required by the attack are $B=2*\max_{i}|r_{i}|$ and $C=2*\sum_{i}|r_{i}|$.

Theorem B.2 can be derived from Theorem 4.4 directly. Maximizing the cumulative reward on $-\mathcal{R}$ is equivalent to minimizing that on $\mathcal{R}$. Therefore, an efficient learning algorithm will learn the policy with the worst performance on the true reward function among the policies supported by the dataset. Despite achieving the worst outcome for the learner, the inverted reward attack faces the problem of requiring high budgets, which is impractical. Since a practical attack should only corrupt a small portion of data, we consider an attack that can only randomly invert a part of the reward. We call this attack the ‘random inverted reward attack’. This attack strategy has been widely used in previous works as a non-trivial attack Ye et al. (); Zhang et al. ().

###### Definition B.3 (Random inverted reward attack) .

The random inverted reward attack has a parameter $p\in(0,1)$. The attack randomly samples $p\cdot N$ indexes from $[1,\ldots,N]$ with replacement, where $N$ is the size of the dataset. Let the set of indexes be $I$. The attack strategy of the inverted reward attack satisfies $\Delta_{i\in I}=-2\times r_{i}$ and $\Delta_{i\not\in I}=0$.

We cannot find any strong guarantee for the effect of the attack on the learning agent. Empirically, we observe that when the attack inverts $<50\%$ of the states, the learner is usually still able to learn a policy almost as good as the one it learns in the uncorrupted dataset. Since the random inverted reward attack is not practical, we need to find a more efficient attack that can work with less budget yet still makes the agent learn a policy of low performance.

The inverted reward attack treats different state actions from the dataset equally, yet the reward for some state actions can have more influence on the learning process. Therefore, it is possible to construct more efficient attacks that focus the attack budget on the rewards in some specific state actions.

###### Definition B.1 (Fully inverted reward attack) .

###### Theorem B.2 .

###### Definition B.3 (Random inverted reward attack) .

## Appendix C Training Log of Main Results

Here, we show the performance of the learned policies during training in different datasets under different attacks.

Figure: Figure 5: Peformance of learning algorithms on different datasets under the attacks. We take the title of the first figure as an example to explain the meaning of the title. ‘HalfCheetah’ means the RL environment is HalfCheetah; ’Medium Expert’ means the dataset is collected by a mixture of medium and expert policies. ‘IQL attacks TD3_BC’ means the learning algorithm used by the attacker is IQL, and the one used by the learning agent is TD3_BC.
Refer to caption: extracted/5949811/Figures/experiment_main.png

## Appendix D Algorithm for Learning a Set of Good Policies

Figure: Algorithm 2 Learning Set of Good Policies

## Appendix E Additional Ablation Study

Size of good policy set: Here, we study the influence of the size of $\Pi_{2}$ on the efficiency of the attack. We set the sizes of the good policy set to be $3,5,7,10,15$. In Fig 6, we observe that the learning outcomes of the algorithms are similar for different sizes of good policy sets. The results suggest that the number of good policies does not considerably affect our policy contrast attack.

Only Good/Bad Policies attacks: To show that it is beneficial to do both making good policies look bad and bad policies look good, we test the efficiency of the attack strategy with only one part. That is, the attack either only decreases the reward associated with the good policies or increases the reward associated with the bad policy. In Fig 7, we observe that both attacks lose some efficiency compared to the policy contrast attack. Here, we provide empirical insights on why these two attacks do not work alone. For making good policies look bad, it can happen in practice that the good policy set does not cover all policies of high performance, therefore the agent is still able to identify a high-performing policy. For making a bad policy look good, in practice, even if the bad policy already has the highest performance in the adversarial environment, without the good policy attack part, the learning agent may not be able to learn it and still converge to a good policy as the adversarial reward function is more complicated. In this case, breaking the optimality of the good policies makes it easier for the learning agent to converge to the bad policy.

Figure: Figure 6: Influence of different sizes of good polices on the attack.
Refer to caption: extracted/5949811/Figures/K.png

Figure: Figure 7: Attacks that only make bad policies look good and good policies look bad.
Refer to caption: extracted/5949811/Figures/goodbad.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: see weaknesses.
soundness: 3
strengths: - the proposed approach provides a general framework for conducting blackbox reward poisoning attacks, and the reported results demonstrates reasonable performance on limited attack budget
- the general idea is simple and effective: using a set of learnt best and worst policies, and by leveraging a distance on actions each perform w.r.t the sampled action, identify how to corrupt a sampled reward from the dataset.
summary: This work presents a universal blackbox reward poisoning attack for offline RL. The proposed approach extends [ref1] to an offline setting, while providing additional evidence on the universality of the attack from various aspects such as learning algorithm, budget and other hyperparameters used in their approach. The provided ablations demonstrates the attack is effective across multiple environments and offlineRL policies including BC, CQL, and IQL.









[ref1] Xu, Yinglun, and Gagandeep Singh. "Black-Box Targeted Reward Poisoning Attack Against Online Deep Reinforcement Learning." arXiv preprint arXiv:2305.10681 (2023).
weaknesses: - In the main results in Table1, it is unclear what learning algorithm is used to learn best/worst policy leveraged by the attacker. The relation between learning algs used for best or worst policies by the attacker, and the attacked policy are also not studied. Please share this information and discuss the relationship.

- The evaluation framework lacks clarity: looking at the results (e.g, in Table1), it is not immediately clear whether PCA is best performing or not. E.g, total performance of fully inverted is the least, hence, should be considered the best method? why is PCA is better than fully inverted? Is the difference between PCA and RPI significant? what difference is significant? 

- Perhaps a better budget would be to take into account number of poisoned samples, instead of budget. This measure is more universal and in some practical cases may be even more realistic. It is not clear why the budgets introduced are of importance, it seems more like these are constraints of the proposed method and are not universal. This part of evaluation, too, lacks further support and motivation, and needs to be improved by more general and practical constraints. Please provide discussions on the choices of constraints, and provide empirical evidence based on #samples as constraints.

- RL methods are very sensitive to hparam selection. If a non-optimal policy is used to adjust the reward, instead of an adversarial policy, how would such a baseline perform in comparison? How can one shows an adversarial attack as proposed is absolutely necessary to achieve the goal of the attack?

- the paper is not very well structured which makes it hard to read, and many important information are scattered throughout the paper. this work needs to be better presented so that the reader is not confused about the proposed method. For instance, in Eq1, V and J_R are present but they are not introduced until much later in the paper. The same as for Band C (budget limits) introduced in Eq.2. Alg2 seems to be essential and is referenced several times in the main paper, but its definition is in the appendix and it is not explained exactly what it does, when the reader needs it. Please fix the structuring issues, and improve presentation.

- In general, the proposed method is closely related to [ref1], which limits its novelty and contributions. Please discuss how this work is different than [ref1].

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - In line 334, should it be $\hat{r}_i = \hat{r}_i - \Delta_2$?
- How to determine the hyperparameters $d$, $\Delta_1$, $\Delta_2$?
soundness: 3
strengths: 1. The paper is very well-written. The studied problem is clearly stated and motivated.
2. The theoretical analysis is helpful for understanding the essense and challenges of the reward poisoning problem.
3. The proposed algorithm makes intuitive sense.
summary: This paper proposes a black-box reward poisoning method for offline reinforcement learning that can be applied to any efficient offline RL algorithm. The authors first establish theoretical framework on the attack problem, then show sufficient condition and necessary condition of the attacker. Motivated by the conditions and analysis, the paper proposes a Policy Constrast Attack, which learns a bad policy and a good policy set, perturbing rewards such that good policy obtains lower reward and bad policy obtains high reward. Experiments show that the proposed method is effective on D4RL dataset and several different algorithms.
weaknesses: The proposed algorithm is an approximated solution to the reward poisoning problem. But it is not clear how far it is to the optimal solution. Given the experiment results, there seems to be a big gap between the fully inverted attack and PCA attack. Is it possible to show some analysis?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1) What happens when $J_{\hat{R}}(\cdot) = 0$ for all inputs? The necessary condition will always be satisfied in Thm 4.5, but it seems unlikely such a reward function would cause the victim to specifically learn bad policies and not good ones. Are there theoretical limits to the formulation of $J_{\hat{R}}$?

2) Could the authors clarify the nature of the constraint $C$? In Figure 4, it is represented as a percentage, but in Definition 4.2 it functions as a value.

3) What makes the method "universal"? The distinction is unclear from reading the related works and section 5.2.

4) Would the method be feasible if instead of learning "good" policies in Algorithm 2, only "bad" ones were learned, and then the rewards inversed? It would be interesting to know, and if true would reduce concerns about the feasibility of learning all good policies.
soundness: 2
strengths: The paper outlines a difficult problem and provides a fundamentally motivated and novel solution with solid inspiration from existing theoretical methods. 

The intuition behind the approach is also clear and easy to understand.

The constraints on the problem and the importance of addressing them are well-stated.
summary: This paper approaches the problem of attacks in offline RL with quite strong constraints on what the adversary has access to. Without knowing the victim algorithm type, training process, or oracle set of solutions, the proposed approach learns an efficient adversarial reward function that inverses good and bad policy performance. Experiments show the success and efficiency of the method and compare it to similar attacks, even though true 1:1 comparisons don't yet exist.
weaknesses: The terms and notation used in the theoretical contributions are confusing. To list a few:
- $V$ is undefined in eq 1, it is normally a value function but does not appear to be so here.
- in assumption 4.1, $J_R (\pi^{alg}(D))$ and $J_R (\pi)$ are both used. Does the RHS of that equation use a different dataset? 
- equations and definitions such as the $\delta$-optimal pessimistic learning algorithm, Thm 4.5, and $\hat(R)$ on page 7 should be clearly labeled and numbered to communicate how they are used. 
- In Def 4.2, are there theoretical determinations for B and C? or, are they hyperparameters? The difference seems important to the understanding of "efficiency".

The motivation and definitions of the Policy Contrastive Attack seem to contain some self-defeating arguments:  In Challenges, it is stated that listing all efficient learning algorithms is problematic. However, the method relies on the ability to find "good" policies as outlined in Alg 2. This outlines questions about the feasibility of the method.

The authors state that only one learning algorithm for each dataset is shown to increase experimental diversity. However, there is no justification for the choices of each algorithm/dataset combination. It would be helpful to include all pairwise combinations in the appendix. 

Tables and Figures: 
-In black and white, both lines in Figure 2 look the same. A more descriptive caption would aid understanding.
-All tables and figures lack descriptive captions.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: Please see the weaknesses above.
soundness: 1
strengths: - The data security of the offline RL datasets has a significant impact on the RL agent training.

- It is an interesting idea to design a poisoned reward mechanism to obfuscate the actual policy performance.
summary: This paper proposes a universal black-box poisoning attack named "policy contrast attack" against offline reinforcement learning. The authors validate its effectiveness in various datasets from the D4RL benchmark.
weaknesses: 1. The presentation of this paper has a significant issue. Lots of expression in the manuscript are not academically professional. For example, "it is equivalent to say", "Here, ...". Some arguments seem to be overclaimed such as " theoretically analyze the insight behind it
based on general assumptions". I do not see any rigorous theoretical analysis when going through the main text and the appendix. In addition, the meanings of the variables in the math formulas are unclear. I suggest adding "where XXX is ..." after each equation.

2. I am confused about the inconsistency between the attack model (limited budget with the infinity-norm constraint) and the mathematical formulation (zero-norm constraint in Equation (1) and (2)). I assume the authors may want to choose the inifinity-norm?

3. Although the authors did an experiment to vary the corrupted data ratio, it seems the objective does not pose a constraint over the corrupted data ratio. I am wondering if it is necessary to incorporate such a constraint into the final objective. It would be great if the authors can have some discussion.

4. Regarding Theorem 4.4, I do not see any relationship between the efficiency of the attack and the inequalities of  B, V, C. I would appreciate it if the authors could further explain the relationship.

5. Regarding Theorem 4,5, first, it is unclear to me what $\pi_0$ is in the context. Second, the necessary condition gives a very loose condition over the poisoned performance over policy $\pi_1$ and $\pi_2$. When $\delta > 0$, the inequality even cannot ensure the poisoned performance of the "bad policy" is no worse than the "good policy". Third, given the original performance of policy $\pi_1$ and $\pi_2$, it is apparent that $\Delta J(\pi_1)$ should be much higher than $\Delta J(\pi_2)$. I do not think a full paragraph (line 302-line 309) is needed to claim this simple argument.

6. Due to the confusion of Theorem 4.5, Lemma 4.6 and Lemma 4.7 are challenging for me to understand. It would be great if the authors could provide further clarification.

7. For Line 319, $\pi(s)$ outputs an action only when it is a deterministic policy. Otherwise, the output should be a distribution. Therefore, $d_a$ is expected to be measuring the difference between two distributions, e.g., kl-divergence.

8. The compared baseline Xu et al. (2022) is relatively weak. More recent baselines (which are accepted by major ML/RL venues) are expected to be included for comparison. Otherwise, it is difficult to examine the effectiveness of the proposed attack.

9. I took a look over Ye et al. (2023), the authors varied the corruption ratio from 10% to 30% in Appendix D. I am wondering how the proposed attack performs under the ratio of 10%. In reality, corrupting too much data is not realistic and can be detected with high probability.

10. The conclusion for Figure 2 does not seem to be valid. Given the limitation of the original RL algorithm, the difference between the reward under normal training and the reward under PCA attack should be considered. In this view, TD3 algorithm is also robust.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. I understand the motivation of the authors for claiming this to be "universal", however, in my opinion, given that experimental results only show a only a limited subset of combination of offline RL algorithms x environments, claiming this method to be universal seems a little too strong. 

2. Under the challenges section, the authors claim that one of the main challenge is that in offline RL, the agent is learning from a fixed dataset. "The learner can always observe how an expert would behave (the action) in the dataset regardless of the attack. This contributes to the fundamental difficulty of misleading the learning agent." Given that the fundamental idea of the paper is to present the high performing policy as a poor one and vice-versa in the dataset, is this truly a challenge ? I would suggest rephrasing this section as it seems like a trivial challenge that was highlighted specifically just to be solved by the proposed method.

3. In the Related Works section, cited works such as Zhang et al. (2020a); Huang et al. (2017) are more related to state perturbations, rather than reward poisoning.

4. In the Related Works section, the authors claim "However, many of these are white-box attacks, and it remains unknown how to scale these techniques to general RL environments that require function approximation with deep neural networks." Could the authors explain in more detail what do they mean by "scaling" and what challenges do they foresee that prevents the "scaling" of existing methods?

5. In general, what are the challenges of applying existing methods of reward poisoning methods in the online setting directly to offline setting? This needs to be succinctly clarified since the authors have repeatedly claimed that existing methods are not applicable/scalable.

6. In the case of the budget on each step of the reward signal, how will the existing algorithm work if the dataset collected consisted of a sparse reward trajectory where most of the rewards are 0? Would this still maintain the undetectability of the attack? 

7. In Section 4.1, variables V, B and C are not introduced until a later section. These should ideally be introduced earlier to improve the readability. Similarly, in Theorem 4.5, $\pi_{1}$ and $\pi_{2}$ were not formally defined until later

8. I do not understand how this inequality $J_{R}(\pi_{\text{Alg}}(D)) \geq \max_{\pi \in \Pi_{\mu}} J_{R}(\pi) - \delta$ implies the agent will always use an algorithm that can learn the policy with nearly the highest performance among the supported policies. Isn't this inequality merely claiming that the offline RL (LHS) is greater than the online RL (RHS) - $\delta$, which really leaves the offline RL unbounded in terms of performance. How does that relate to the statement?

9. The baselines RPI and RPP should be discussed in a little more detail for the sake of completeness, given that they're the main source of comparisons in this paper. 

10. Could the authors explain how are the combination of algorithm-environment selected? Are they randomly selected?

11. "In many cases, our attack is also significantly more efficient than the RPI and RPP baseline attacks." How are the authors defining efficiency here and where is this statement being supported?

12. In section 5.3, the authors state:"Intuitively, although the per-step corruption is less, the attack influences the performance of more policies, making it possible for the agent to learn slightly worse policies." What do you mean by the attack influences the performance of more policies? Also, how is the initial baseline of B = 2*$r_{max}$ selected? Furthermore, the authors claim that the attack is slightly more efficient with lower values of $B$, which seems counterintuitive. So, what is the main takeway of this section?
soundness: 3
strengths: The paper presents a theoretically motivated idea, applying a well-known concept (altering rewards by flipping the signs of good and bad policies) from adversarial attacks to the offline reinforcement learning (RL) setting. The approach is straightforward and repurposes existing ideas in a new context, broadening the potential applications of adversarial methods. The paper is generally easy to follow, with a clear structure that helps convey the main concepts and findings, making it accessible. While the results are only compared with very simple baselines, they offer initial empirical insights into how this approach might perform within offline RL and suggest possible directions for further exploration.
summary: In this study, the authors studied and develop the idea of developing an efficient black-box reward poisoning attack on efficient offline RL algorithms with the aim of the attack model being universally applicable to all efficient offline RL algorithms. Specifically, the authors proposed two algorithms, one for learning the good policies and one for learning the bad policies, and use swap these two policies to augment the rewards in the offline RL dataset in order to fool to offline RL agent during training. The authors showed that their methods are more effective and efficient compared to baseline methods on several MuJoCo environments and also provided theoretical motivations and backings for their formulation.
weaknesses: The paper’s contribution would be strengthened by a deeper, more comprehensive empirical foundation, as many of the current results are sampled from a subset of possible experiments, limiting the generalizability of the claims. The absence of certain details, such as the definitions of variables and terms early in the text, hinders the flow and clarity of the presentation. Additionally, including additional details on the baselines used as comparison would improve the reader's understanding of the novelty and effectiveness of the proposed method.  Addressing these elements would help establish the reliability and broader applicability of the findings.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: * Can you please explain the results you claim with theorem 4.4? It is hard to understand what the theorem is really stating.
* Why should an attacker implement this form of attack over a backdoor attack, given they have access to the data?
soundness: 1
strengths: The paper has a few strengths
* Researching poisoning attacks against RL, and especially offline RL, is an important direction of research, so I appreciate the authors for aiming to add to the body of literature on this topic.
* The study is motivated within the context of most related work.
* The paper is fairly easy to follow.
* Their attack improves over their proposed baselines.
summary: The authors claim a new, universal reward poisoning attack against offline reinforcement learning algorithms. Their attack assumes no prior information regarding the chosen algorithm of the training, victim party - instead aiming to generate reward perturbations that will be effective against any algorithm. They claim the first, universal poisoning attack against offline reinforcement learning in this black box setting.

They frame their objective as a constrained optimization problem aiming to find reward perturbations that are most likely to minimize the value of policies found by $\delta$-pessimistic offline RL algorithms. The adversary is restricted in the $l_\infty$ and $l_1$ norm of their reward perturbations.

They aim to achieve this objective with two main steps. They first learn a policy set $\Pi_2$ given the dataset $\mathcal{D}$ and then a sub-optimal policy $\pi_1$ given inverted rewards $\hat{r}_i = -r$. Using these policies they then decrease the rewards of "good" actions in the dataset $\mathcal{D}$ which are closer to $\Pi_2$ and increase the reward of "bad" actions close to $\pi_1$ in $\mathcal{D}$. The goal of this is that the victim agent should learn that "good" actions (according to the benign, unpoisoned MDP) are actually bad, while "bad" actions are actually good. Their attack is constrained by budget hyper-parameters $\Delta_1$ and $\Delta_2$.

They evaluate their attack on a suite of D4RL benchmarks and show they can decrease the expected return of the agent a decent amount more than more naive attacks under their reward poisoning constraints.
weaknesses: **Motivation -** The paper's motivation is placed fairly well within the context of related works, but I don't think the attack's objectives are well motivated within the context of a real world attack. This attack formulation is indiscriminate, meaning the agent's performance in all states is harmed. Lets now assume the attack is perfectly successful, minimizing $J_\mathcal{R}(\pi)$. Given this outcome, the party training the agent will be able to observe that the agent is completely failing to solve the desired task, so they'll never actually deploy it in the real world. This essentially makes the offline dataset $\mathcal{D}$ useless after poisoning, which could be one angle for why this attack formulation is interesting, but the trained agent will never actually cause negative outcomes down the line. For this same reason I think the slightly weaker objective of decreasing $J_\mathcal{R}(\pi)$ below some $V$ is also poorly motivated.

I think the authors should better motivate their attack objectives in the context of something a real adversary would hope to induce. I further suggest they compare and justify these objectives against those of backdoor poisoning attacks [1,2,3] which are much stealthier. It may also be interesting if the authors shift their adversarial objective to be less indiscriminate, perhaps only affecting a subset of trajectories or states rather than all states. 

**Theory -** The paper's theoretical results are disconnected from the method that is developed and the claims they make in text. To me Theorem 4.4 is the main theoretical claim of the paper (though this is not completely clear based on the writing), which seems to claim that $J_\mathcal{R}(\pi) \geq V$ given some $\pi$ trained on poisoned rewards $\hat{\mathcal{R}}$. This seems to go against their objective of $J_\mathcal{R}(\pi) < V$. The result doesn't make sense to me and isn't supported by the proof given in the appendix. The proof only claims that the agent will learn a $\delta$-pessimistic policy over $\hat{\mathcal{R}}$, which I agree with, but it doesn't show how this relates to the value of the policy ($V$) under $\mathcal{R}$. Due to this I don't think the claim made in theorem 4.4 is correct. Furthermore, even if Theorem 4.4 is true, the theorem does not connect back to the method presented in algorithm 1, meaning there is nothing meaningful to say about how we should expect the attack to perform. Theorem 4.5 is also a bit confusing since it seems to be just presenting the attack's objectives rather than any guarantees afforded by the attack. 

The paper also claims "universal" poisoning attacks, but only considers $\delta$-pessimistic algorithms in their theory. I don't think their method necessarily relies on targeting $\delta$-pessimistic algorithms (again since the theory doesn't seem to connect to the method), but I still think making the claim of "universal" is too strong without better theoretical results. I suggest the authors consider the effect that their poisoning has on the optimal policy of the MDP rather than performing their analysis over the class of learning algorithms. See [4] for examples of such analysis.

The proofs found in the appendix are also very short and informal, so I am not completely convinced by them. I think they make intuitive sense, but they rely really heavily on Assumption 4.1 doing the heavy lifting.

**Presentation -** the paper has a lot of notation and grammar mistakes. In particular the text claims the attacker wants to minimize the $l_\infty$ norm of their attack, which makes sense, but equations 1 and 2 consider the $l_0$ norm. I think if the authors intended to use $l_\infty$ in these equations, but I could be wrong. Also see line 282 "near-policy policies".

Another issue is that the paper frequently overloads terms and notation. For instance, in the text under equation 1 the authors seem to refer to $\mathcal{L}$ as both a set of policies and a set of algorithms. After reading the paragraph a few times  I believe I understand what the authors are saying, but I think even this adds to the confusion. I agree that one cannot consider all possible algorithms, as I can always design an algorithm that outputs an arbitrary policy, but I don't think these cases are important to the paper's analysis. Instead I recommend the authors just consider the affect their attack has over the optimal policy given the dataset rather than considering a set of algorithms.

There are also a few pieces of the paper which I think are redundant and can be removed to save space. For instance, equations 1 and 2 are nearly identical, along with the two conditions of theorem 4.5. The fact that the attack is black box is stated 3 times in bullet points under "Universal and Black Box", "Universal and Black box attack", and "No knowledge of the learning algorithm".

**Experimental Results -** The paper's experimental results are okay, but they aren't too strong. I think a more novel attack formulation would be required to have stronger results under these constraints. An attack objective which only considers a subset of states may be able to perform better.

**Summary -** I think the study of indiscriminate attacks against offline RL isn't well motivated since they just result in policies that are never deployed. I would be okay with this attack formulation if the theoretical or empirical results were novel, but I think this paper needs stronger theoretical analysis and a better methodology before it can meet that criteria. 

[1] "BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets" https://arxiv.org/abs/2210.04688

[2] "TrojDRL: Trojan Attacks on Deep Reinforcement Learning Agents" https://arxiv.org/abs/1903.06638

[3] "SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents" https://arxiv.org/abs/2405.20539

[4] "Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning" https://arxiv.org/abs/2208.13663