SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xuan Chen $^{1}$ , Wenbo Guo $^{1,2}$ , Guanhong Tao $^{1}$ , Xiangyu Zhang $^{1}$ , Dawn Song $^{2}$
$^{2}$ UC Berkeley
{chen4124, henrygwb, taog, xyzhang}@cs.purdue.edu
{henrygwb, dawnsong}@berkeley.edu
Backdoor attacks pose a severe threat to the supply chain management of deep reinforcement learning (DRL) policies. Despite initial defenses proposed in recent studies, these methods have very limited generalizability and scalability. To address this issue, we propose BIRD, a technique to detect and remove backdoors from a pretrained DRL policy in a clean environment without requiring any knowledge about the attack specifications and accessing its training process. By analyzing the unique properties and behaviors of backdoor attacks, we formulate trigger restoration as an optimization problem and design a novel metric to detect backdoored policies. We also design a finetuning method to remove the backdoor, while maintaining the agent's performance in the clean environment. We evaluate BIRD against three backdoor attacks in ten different single-agent or multi-agent environments. Our results verify the effectiveness, efficiency, and generalizability of BIRD, as well as its robustness to different attack variations and adoptions.
Recent studies [59, 20, 49, 8] demonstrate that an attacker can inject a backdoor into a DRL agent's policy. In particular, the attacker adds a trigger into the environmental state during the training process and forces the agent to take a backdoored/poisoned action that diminishes its total reward given by the environment. To accomplish this, they manipulate the agent's reward function and assign a high reward whenever the agent takes a poisoned action at a poisoned state. We denote the original reward given by the environment as the actual reward and the attacker-manipulated reward function as the poisoned reward function. As a result, the agent learns to take poisoned actions at these states, expecting the environment to assign a high actual reward to it. However, taking the poisoned actions actually diminishes the agent's actual reward and even causes the agent to fail its corresponding tasks.
Given that a backdoorsed agent performs normally at clean states, it is challenging to detect and remove the backdoor from a pretrained policy without accessing the training process, having knowledge about the attack specifications (i.e., trigger, poisoned actions, poisoned reward function), and accessing the poisoned states during testing. Most existing backdoor defenses do not consider the sequential decision-making nature of DRL policies [46, 48, 16, 28, 13, 9, 25, 61, 30, 42, 51, 52] or assume access to the agent training process [11, 66, 56, 66]. Only a few methods provide defense for pretrained policies [2, 15]. However, they have several limitations that hinder their practicality. Specifically, the method proposed in [2] can only be applied to attacks with a perturbation patch as the trigger and cannot detect whether a policy is backdoored or not. Similarly, the defense proposed in [15] is only applicable to attacks where the trigger is an adversarial agent's specific actions.
We propose a generalizable backdoor detection and removal method for DRL policies. Given a pretrained agent's policy network and value function, along with the corresponding clean environment,
we first formulate trigger restoration as an optimization problem. The objective function is crafted as searching for a perturbation to the state representation that forces the agent to take actions that maximize its value function. Since a backdoored agent is trained using a poisoned reward function that assigns high values when the trigger is present, maximizing its value function should identify the trigger. Second, we design a novel metric to detect whether the agent is backdoored or not using the resolved perturbation (denoted as restored trigger). As we will detail in Section 3.3, adding the restored trigger to the clean environment will cause the backdoored agent to receive a much lower actual reward from the environment but increase or not affect the clean agent's actual reward. Based on this behavior difference, we design our metric as the actual reward difference of the agent before and after adding the restored trigger to the environment. Finally, we propose a novel finetuning method to remove the detected backdoor. We design additional regularization terms to the finetuning objective function to maintain the finetuned agent's actual reward in the clean environment. We also introduce a neuron re-initialization mechanism to ensure the backdoor can be successfully removed even when the restored trigger is not exactly the same as the ground truth one in shape and size.
We denote our method as BIRD (Backdoor Identification and Removal for DRL). We extensively evaluate BIRD in ten single-agent or multi-agent RL environments against three prevalent backdoor attacks. First, our results demonstrate that BIRD outperforms existing backdoor detection methods designed for supervised learning in detecting backdoored policies. Moreover, we show the superiority of BIRD over existing methods in removing the backdoor while maintaining the agent's actual reward in the original clean environment. Second, we verify the effectiveness of our key design choices through ablation studies, especially our backdoor detection metric. We demonstrate that compared with the widely used trigger size metric, our metric enables much higher detection accuracy without requiring subtly tuning the threshold. Third, we demonstrate the computational efficiency of BIRD and its insensitivity to hyper-parameter choices. Furthermore, we verify the robustness of BIRD against different variations in attack and two possible adaptive attacks. To the best of our knowledge, BIRD is the first DRL backdoor defense that can detect and remove backdoors in a clean environment without requiring access to the attack specifications and policy training process. Furthermore, BIRD is applicable to all kinds of existing attacks, making it a highly generalizable defense mechanism.
Attacks. Recent research works propose backdoor attacks against single-agent DRL [59, 20, 50, 8]. These attacks, denoted as perturbation-based attacks, add a small perturbation patch to the victim agent's state as the trigger. A backdoored policy performs normally in a clean environment but takes poisoned action at poisoned states, leading to task failure. A follow-up work considers a two-agent setup with an adversarial agent and a victim agent [49]. Rather than perturbing the states, they leverage the adversarial agent's certain actions as the backdoor trigger (denoted as adversarial agent attack). More recent works generalize both perturbation-based attacks and adversarial agent attacks to multi-agent cooperative RL with a team of victim agents [8, 7].
Detection and Defenses. As mentioned in Section 1, only two existing defenses are developed to mitigate backdoors for a pretrained DRL policy. One method, proposed by [2], uses a trigger filter technique that maps possibly poisoned states back to their corresponding clean states. This method is designed for perturbation-based attacks and cannot be applied to adversarial agent attacks. Additionally, it relies on singular value decomposition, which limits its scalability in complicated RL environments with high-dimensional state spaces. Another method [15] also has limited generalizability in that it is only applicable to adversarial agent attacks.
Some other research works also extend adversarial sample attacks [4, 14] to DRL and propose testing-phase perturbation attacks [17, 32, 36, 63, 26, 40, 19], which perturb the environment at certain states to fail a pretrained agent. To defend against such attacks, researchers either generalize existing adversarial defenses designed for supervised learning to DRL [63, 62, 31, 55, 22, 57, 56] or model the attack and defense as a two-player game and train a robust agent to find the Nash equilibrium point [33, 60, 64]. Similarly, some robust RL methods also propose robust training techniques against random perturbations to the agent's observations [65, 66], actions [33, 44], or reward functions [67]. Due to differences in attack setups, these methods cannot be directly applied to our problem.
**Formulation.** Consider a Markov Decision Process (MDP)  $\mathcal{M} = (\mathcal{S},\mathcal{A},P,R,\gamma)$ , where  $\mathcal{S}$  and  $\mathcal{A}$  are the continuous or discrete state and action space.  $P$  is the state transition function,  $R$  is the reward function, and  $\gamma$  is the discount factor. Without facing any attack, the agent's goal is to learn an optimal policy  $\pi$  that maximizes its expected long-term reward,  $\eta(\pi) = \mathbb{E}[\sum_t\gamma^t R(s_t,\pi(s_t))]$ . We also define a stationary state occupancy distribution for  $\pi$ , denoted as  $\rho^\pi(s) = (1 - \gamma)\sum_t\gamma^t p(s_t = s|\pi)$ , and  $\eta(\pi)$  can also be expressed as  $\mathbb{E}_{s \sim \rho^\pi}[V_\pi(s)]$ , where  $V_\pi(s)$  is the state-value function.
Attack model. We follow existing attacks and assume the attacker injects one backdoor into the victim agent's policy. We add a superscript ' to indicate a backdoored/poisoned entity, and the backdoored policy is denoted as  $\pi^{\prime}$ . We consider all three types of existing attacks: single-agent perturbation-based attacks [20], two-agent adversarial-agent attacks [49], and multi-agent perturbation-based attacks [8]. In the following, we use single-agent perturbation-based attacks to derive our technique, and later we discuss the extension of our approach to the other attacks. Under perturbation-based attacks, the attacker designs the trigger as a small perturbation patch denoted as  $\hat{\Delta}$ . At any time  $t$ , the attacker may add  $\hat{\Delta}$  to the current state representation  $s_t$ , causing the backdoored agent to perceive the poisoned state  $s_t'$  and taking a poisoned action  $a_t' = \pi'(s_t')$ . We follow existing works [2, 20] and assume the attacker only perturbs the state representations perceived by the victim agent (victim agent's observation) without altering the actual state (underlying physical state). As such, the state transition still depends on the actual state rather than the perceived/poisoned one, i.e.,  $s_{t+1} \sim P(s_t, a_t')$ . To inject the backdoor, the attacker needs to manipulate the victim agent's reward function, assigning the agent an ultra-high poisoned reward when it takes the poisoned action at poisoned states.
Defense assumption and goals. We do not assume access to the attack training process. Instead, we are provided with an agent with a pretrained policy  $\pi$  and the corresponding state-value network  $V_{\pi}(s)$  or action-value network  $Q_{\pi}(s,\pi (s))$ , and lacks the knowledge of whether the policy is backdoored or not. We are also given a clean environment for testing and debugging purposes. This setup simulates a practical scenario where the defender needs to verify an agent's robustness and safety before deploying it in a potentially poisoned environment, especially in critical fields. We further assume access to limited computational resources that support testing or finetuning a given policy but not training a policy from scratch. We also do not assume any knowledge about attack specifications, including the ground-truth trigger, the poisoned actions, or the reward manipulation strategy employed by the attacker. We aim to develop a full-stack defense with the following three steps: 1) Trigger restoration - restore the potential trigger(s) from the given policy in the clean environment, 2) Backdoor detection - determining whether the restored trigger is associated with an injected backdoor and thus detect whether the given policy is backdoored or not, and 3) Backdoor removal - unlearning the detected backdoor from the backdoored policy without affecting its performance in the clean environment.
3.2 Trigger Restoration
Key insights. We derive the intuition from the training process of backdoor attacks. Recall that the attacker needs to poison the victim agent's reward function. Under the poisoned reward function, the agent receives a high total reward, denoted as  $\eta (\pi^{\prime}) = \sum_{s}\rho^{\pi^{\prime}}(s)\sum_{a^{\prime}}\pi^{\prime}(a^{\prime}|s^{\prime})R^{\prime}(s^{\prime},a^{\prime})$ , when taking trigger actions at the poisoned states. This indicates that given an agent with a fixed policy, adding a proper perturbation (trigger) to its state will activate the backdoor, resulting in a significant increase in the agent's total poisoned reward under the poisoned reward function.
Intuitively, we can restore the backdoor trigger by searching for a small perturbation to the states that maximizes the agent's total poisoned rewards under the poisoned reward function. Since we do not assume accessing to the poisoned reward function, we use the agent's state-value or action-value function to compute the total reward, which is also poisoned during the attack training. Formally, this perturbation can be obtained by solving the following objective function for a given policy  $\pi$
$$
\max  _ {\Delta} \sum_ {s} \rho^ {\pi} (s) \sum_ {a} \pi (s + \Delta) Q _ {\pi} (s + \Delta , \pi (s + \Delta)), \tag {1}
$$
$\Delta$  has the same dimension as  $s$ , where each element in  $\Delta$  represents the perturbation to the corresponding element in  $s$ . Solving Eqn. (1) yields a potential trigger for a backdoored policy but not for
a clean policy. Given that a clean agent's reward and value function is not poisoned, solving Eqn. (1) would likely produce a universal perturbation that guides the agent to take better actions to further increase its reward. However, this is typically challenging for complicated environments where the agent's task is difficult. Besides, we integrate specific designs to eliminate these noisy perturbations.
Technical details. We follow existing perturbation-based attacks [20, 8] and consider the state representation as the snapshot of the current environment,  $s \in \mathbb{R}^{p \times q}$  with each element  $s_{ij}$  is normalized to [0, 1]. Directly solving Eqn. (1) likely results in noisy perturbations for both backdoored and clean policies. To eliminate these noises, we propose two regularization terms to constrain the search space. First, given that the attacker typically uses a small trigger size to stay stealthy, we constrain  $\mathcal{R}_1(\Delta) = ||\Delta||_1$ , which yields a  $\Delta$  with a small number of non-zero elements. This regularization rules out noisy perturbations with large sizes. Second, all existing perturbation-based attacks use dense patches as triggers because they are easily applicable in the physical world. In contrast, noisy perturbations are typically scattered. We add another regularization to constrain the smoothness/density of  $\Delta$ ,  $\mathcal{R}_2(\Delta) = \sum_{i,j} (\Delta_{i+1,j} - \Delta_{i,j})^2 + \sum_{i,j} (\Delta_{i,j+1} - \Delta_{i,j})^2$ . This additional regularization term further rules out more noisy perturbations that are small and scattered.
Besides adding regularizations, we also design the perturbation  $\Delta$  to be produced by a generative model. As demonstrated in previous works [5], assuming a generative model gives a more stable and less noisy result than directly solving the values in an unknown perturbation to neural network input. Since  $s$  is normalized to  $[0,1]$ , this constrains the maximum allowed perturbation to be 1 or  $-1$ , indicating  $\Delta_{ij} \in [-1,1]$ . Since there is no common distribution with a value range of  $[-1,1]$ , we satisfy this constraint by introducing a set variable  $\{p_{ij}\}_{i=1:p,j=1:q}$ . We define  $p_{ij}$  to follow a  $\mathrm{Beta}(\alpha_{ij},\beta_{ij})$  distribution, parameterized by  $\alpha_{ij}$  and  $\beta_{ij}$ . Given that  $p_{ij} \in [0,1]$ , we then compute  $\Delta_{ij} = 2p_{ij} - 1$ , which naturally guarantees  $\Delta_{ij}$  lies within the range of  $[-1,1]$ . The objective function can be rewritten as  $\max_{\alpha_{1:p,1:q},\beta_{1:p,1:q}} \mathbb{E}_{\mathbf{p} \sim \mathbf{B}}[\sum_s\rho^\pi (s)\sum_a\pi (s + 2\mathbf{p} - \mathbf{1})Q_\pi (s + 2\mathbf{p} - \mathbf{1},\pi (s + 2\mathbf{p} - \mathbf{1}))]$ , where  $\mathbf{p} \in \mathbb{R}^{p\times q}$  and  $\mathbf{B}$  stands for the joint distribution  $\prod_{i,j}\mathrm{Beta}(\alpha_{ij},\beta_{ij})$ . By sampling perturbations from the beta distribution and computing the mean as the final result, we can reduce noise and obtain a trigger with higher fidelity. With this generative model, we double the number of parameters needed to be resolved. Given that the beta distribution's output sparsity depends more on the difference between  $\alpha$  and  $\beta$  than their actual value. We further fix the value of  $\alpha$  and design  $\beta_{ij} = \mathbf{e}_{ij} + \alpha$ ,  $\mathbf{e} \in \mathbb{R}^{p\times q}$ , to reduce the number of parameters in our objective function. Supplement S1 provides more insights about this parameter reduction design and Supplement S4 shows our method is not sensitive to the choice of  $\alpha$ .
So far, we consider the trigger's location and pattern to be fixed across different states (time steps). A straightforward attack adaption could change the trigger's location and even its shape at different states. To handle this scenario, we design  $\mathbf{e}$  as a function of each state  $f_{\theta}(s)$ , parameterized by  $\theta$ . This design is also applicable to the original attack, where all the states have the same output for  $f_{\theta}(\cdot)$ . Putting all designs above together, we obtain the trigger restoration objective function.
$$
\max _ {\theta} J (\theta) = \mathbb {E} _ {s \sim \rho^ {\pi}} \left[ \mathbb {E} _ {\mathbf {p} _ {s} \sim \mathbf {B} _ {s}} \left[ \eta_ {s} (\pi (\mathbf {p} _ {s})) + \lambda_ {1} \mathcal {R} _ {1} (\mathbf {p} _ {s}) + \lambda_ {2} \mathcal {R} _ {2} (\mathbf {p} _ {s}) \right] \right], \mathbf {B} _ {s} = \prod_ {i, j} \mathrm {B e t a} (\alpha , \alpha + (f _ {\theta} (s)) _ {i j}),
$$
$$
\eta_ {s} (\pi (\mathbf {p} _ {s})) = \sum_ {a} \pi (s + 2 \mathbf {p} _ {s} - 1) Q _ {\pi} (s + 2 \mathbf {p} _ {s} - 1, \pi (s + 2 \mathbf {p} _ {s} - 1)), \tag {2}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  are hyperparameters.  $\mathbf{p}_s$  and  $\mathbf{B}_s$  is specific for each state  $s$ . Eqn. (2) can not be directly solved because  $\mathbf{p}_s$  is sampled from an unknown distribution  $\mathbf{B}_s$ . We leverage the policy gradient method [21, 18] to compute the gradient  $\nabla_{\theta}J(\theta)$  and solve the Eqn. (2) following the typical on-policy REINFORCE algorithm [41]. Our final perturbation for each state  $s$  is  $\Delta_s = 2\left(\frac{\alpha}{2\alpha + f_\theta(s)}\right) - 1$ , where  $\frac{\alpha}{2\alpha + f_\theta(s)}$  is the mean of the joint beta distribution  $\mathbf{B}_s$ .
To ensure that the perturbations we compute actually influence the agent's actions and affect its reward during execution, rather than generating simple adversarial examples for the value network that cannot affect the agent's actions, we only use the agent's value network to compute average rewards. We do not compute the gradient of  $\theta$  with respect to the value network or update the value network itself. Supplement S1 provides a detailed algorithm for solving Eqn. (2) and also discusses further why we are not solving adversarial examples for the value network.
3.3 Backdoor Detection
Key insights. Our backdoor detection methods draw insights from the unique behaviors of backdoors policies, which perform well in clean environments but poorly in poisoned ones. For a backdoored
policy, if its restored trigger  $\Delta$  has similar efficacy as the ground-truth trigger  $\hat{\Delta}$ , adding it back to the clean environment will significantly reduce the agent's total actual reward. However, this is not the case for a clean agent trained using the actual reward function. Since the value function of a clean agent is not poisoned, solving Eqn. (2) will either result in a perturbation that guides the agent to select better actions and increase its reward or a perturbation that cannot significantly impact the agent's performance, if the agent is already well-trained or the environment is too complicated. In both cases, adding the perturbation to the environment will not reduce the agent's actual reward.
As such, we propose a novel metric to distinguish between backdoored and clean policies, which is based on the agent's actual reward reduction before and after poisoning the environment with restored perturbations. This approach differs from existing detection methods designed for supervised classifiers (e.g., [4, 48, 38, 42]), which rely on the trigger size  $(||\Delta ||_0)$  as the metric. This metric is sensitive to the ground-truth trigger size. Our detection method is designed based on the unique behaviors of backdoor attacks in DRL and is more accurate and robust (demonstrated in Section 4).
Technical details. Given an agent's policy  $\pi$ , we first run it in the environment for  $K$  rounds and record the agent's average actual reward given by the environment, denoted as  $\bar{\eta}(\pi) = \frac{1}{K}\sum_{t}^{(k)}R(s_t^{(k)},\pi(s_t^{(k)}))$ . We then simulate an extreme attack case by adding the restored perturbation  $\Delta_s$  to the agent's policy input at each time step, run the agent for  $K$  rounds, and record the average actual reward, denoted as  $\bar{\eta}(\pi,\Delta) = \frac{1}{K}\sum_{t}^{(k)}R(s_t^{(k)},\pi(s_t^{(k)} + \Delta_{s_t}))$ . For adversarial agent attacks, since  $\Delta$  corresponds to the state representation of the adversarial agent's trigger action, adding  $\Delta_{s_t}$  to  $s_t$  simulates the trigger action. Here,  $\bar{\eta}(\pi,\Delta) = \frac{1}{K}\sum_{t}^{(k)}R(s_t^{(k)} + \Delta_{s_t},\pi(s_t^{(k)} + \Delta_{s_t}))$ , given that the trigger action indeed changes the actual state of the environment. Next, we compute the reward reduction rate  $\phi(\pi,\Delta) = (\bar{\eta}(\pi,\Delta) - \bar{\eta}(\pi)) / \eta_{\max}$ , where  $\eta_{\max}$  is a normalization term, representing the maximum reward difference of the environment.  $\phi(\pi,\Delta) \in [-1,1]$ . If  $\Delta$  is similar to the ground-truth trigger, it will result in a very large reward reduction and thus a negative  $\phi(\pi,\Delta)$  near  $-1$ . Conversely, a  $\Delta$  restored from a clean policy will produce a positive  $\phi(\pi,\Delta)$  or a  $\phi(\pi,\Delta) \approx 0$ .
Therefore, we treat  $\pi$  as a backdoored policy and  $\Delta$  as the trigger if  $\phi(\pi, \Delta) \leq \epsilon$ . We consider a practical scenario where we do not have a set of clean policies to help select a proper threshold. Instead, we need to make a decision for each individual policy by hand. We set an aggressive value with  $\epsilon = -0.9$  to rule out false positives. Here, we consider all of the clean policies and backdoored policies are well-trained, i.e., their reward in the clean environments are nearly optimal. In case that there are some sub-optimal agents that are backdoored, the detection threshold should be more conservative. More discussions on the detection threshold under different situations can be found in Supplement S2. Later, in Section 4, we demonstrate that, due to the significant performance difference between backdoored policies and clean policies under our restored perturbation, even setting  $\epsilon = -0.9$  introduces only negligible false negatives and thus provides a nearly perfect detection performance. Supplement S1 presents our detection algorithm.
3.4 Backdoor Removal through Unlearning
Naive solutions and pitfalls. A naive solution for backdoor removal is to add the restored trigger to the state and retrain the policy under the actual reward function, which penalizes poisoned actions  $a'$ . We test this solution and find that it indeed improves the agent's performance when presented with the restored trigger. However, the retrained policy remains vulnerable to the ground-truth trigger due to differences between the restored and ground-truth trigger. We try to address this issue by varying the restored