SlowGuess/NeurIPS_2023_OCR huggingface dataset:
trigger (e.g., adding random noise) during the training. Our empirical findings suggest that the ground truth trigger is only unlearned when the restored one is very similar to it, which is not always the case. Furthermore, we discover that the performance of retrained policies in the clean environment oftentimes decreases after retraining. These initial explorations highlight two challenges: (1) how to remove the ground-truth trigger when the restored one has some differences from it, and (2) how to maintain the retrained agent's performance in the clean environment.
Figure 1: Overlapped shortcuts between the restored trigger and the ground truth one. The bar plot shows the overlapping percentage among the top  $X$  neurons with the highest value.
Our Designs. As extensively studied in existing research [54, 53, 10], injecting a backdoor in neural networks is equivalent to building one or a few shortcuts from the backdoor trigger to the target out. In DRL, these shortcuts connect the poisoned states with the poisoned action. When the trigger is presented, only the neurons on the shortcuts get activated. Given that our restored trigger could indeed trigger similar poisoned actions as the ground truth one, we hypothesize that even if the restored trigger is not highly similar to the ground truth one, their corresponding shortcuts may have some overlap. To test this hypothesis, we conduct an empirical test where we compared the neurons activated by the restored trigger with those activated by the ground truth trigger. As shown in Fig. 1, the neurons with high activation values have a large overlap for these two triggers, confirming our hypothesis. Guided by this intuition, we propose to remove these most vulnerable and overlapping shortcuts instead of directly unlearning the trigger. In particular, we first add the restored trigger to the backdoored policy input and record the activation values of all the neurons. We select the top  $L$  neurons with the highest activation value and re-initialize their weights to erase the most vulnerable shortcuts. Finally, we retrain the policy in the environment poisoned with the restored trigger under the actual reward function. The objective function can be written as  $\max_{\phi} \eta(\pi_{\phi}, \Delta)$ , where  $\pi_{\phi}$  denotes the retrained policy. We retrain the entire policy network rather than only the reset neurons to give the policy enough capacity to let the agent learn how to perform in the poisoned environment. See Supplement S2 for more details about our neuron selection and re-initialization process.
The second challenge can be addressed by incorporating an additional regularization term, i.e,  $|\eta(\pi_{\phi}) - \eta(\pi')| \leq \epsilon_1$ , where  $\pi'$  is the original backdoored policy. This ensures that the total reward difference between the retrained policy and the original policy is within a certain threshold  $\epsilon_1$ . This constraint helps to maintain the retrained agent's performance in the clean environment, given that the original backdoored agent should have performed well in a clean environment by design. However, this regularization is not feasible as  $|\eta(\pi_{\phi})|$  is intractable during training. To overcome this, we propose to use the KL divergence between  $\pi_{\phi}$  and  $\pi'$  as a proxy since it is much easier to compute. The following inequality states the relationship between KL divergence and value difference.
$$
\left| \eta \left(\pi_ {\phi}\right) - \eta \left(\pi^ {\prime}\right) \right| \leq C \max  _ {s \sim \rho^ {\pi^ {\prime}}} \mathbb {K L} \left(\pi_ {\phi} (s) \| \pi^ {\prime} (s)\right). \tag {3}
$$
See Supplement S1 for the derivation. Based on Eqn. (3), constraining the KL divergence between  $\pi$  and  $\hat{\pi}$  can bound  $|\eta (\pi_{\phi}) - \eta (\pi^{\prime})|$ . Then, our final retraining objective function can be written as
$$
\max  _ {\phi} \eta \left(\pi_ {\phi}, \Delta\right), \text {s . t .} \mathbb {K L} \left(\pi_ {\phi} (s) \| \pi^ {\prime} (s)\right) \leq \epsilon_ {1}. \tag {4}
$$
We leverage the state-of-art PPO algorithm [37] to solve Eqn. (4), which guarantees a fast convergence. During retraining, we periodically add  $\Delta$  to simulate the poisoned states and keep a certain portion of clean states to compute  $\mathbb{KL}(\pi_{\phi}(s)\| \pi '(s))$ . Supplement S1 shows the detailed algorithm of our backdoor removal methods. Supplement S2 states our implementation and hyper-parameter choices.
Note that our proposed defense (from trigger restoration to backdoor removal) can be directly applied to adversarial agent attacks [49] and multi-agent perturbation-based attacks [8] (See Supplement S1 for a detailed discussion).
We compare BIRD with three state-of-the-art backdoor DRL attacks: TrojDRL [20], Backdoorl [49], and Marnet [8]. Following their papers, we select the environments and used their default attack setups to train backdoor policies (see Supplement S2 for more details). For TrojDRL, we select six Atari games from the OpenAI Gym [3]: Breakout, SpaceInvaders (SI), Qbert, and Seaquest, CrazyClimber, Pong. Due to limited space, we put the results of CrazyClimber and Pong in Supplement S3. TrojDRL considers targeted and non-targeted attacks based on whether the backdoored action is prespecified or not. In this section, we report the results of non-targeted attacks, where detection is more challenging, and leave the results of targeted attacks in Supplement S3. For Backdoorl, we choose the You-Shall-Not-Pass (YSNP), Sumo-Humans (SH), and Run-To-Goal-Ants (RTGA) games from the MuJoCo environment [45]. We select the SMAC environment [47] for Marnet and conduct two attacks, one against the policies trained by QMIX [35] (a multi-agent Q-learning algorithm), and the other against the policies trained by COMA [12] (a multi-agent policy gradient algorithm). Note that by default, all the perturbation-based attacks consider the trigger with a fixed shape/size/location, and adversarial agent attacks set the trigger as a sequence of fixed actions taken at the beginning of each game. We include the performance of BIRD against a more recent attack [6] in Supplement S3.
Table 1: Performance of different defenses in the nine attack scenarios. "Clean" and "poisoned" refer to the original clean environment and an environment poisoned by the ground truth trigger. The numbers in each element are mean±std. "-" means that the method is not applicable to the corresponding attack. The Atari games' results are rewards and the rest are winning rates. Supplement S3 shows the p-value for each attack scenario is smaller than 0.001.
Environments	Methods	Breakout	SpaceInvaders	Qbert	Seaquest	QMIX	COMA	YSNP	SH	RTGA
Clean	Original	298±37	610±97	13488±1897	1719±107	74.4%±2.5%	93.0%±2.2%	52.8%±4.5%	28.9%±3.6%	49.7%±4.8%
PD	142±32	238±94	4941±1428	932±97	-	-	-	-	-
Finetuning	317±39	614±104	13151±2233	1785±98	73.6%±2.6%	90.5%±5.4%	53.5%±1.7%	30.1%±2.8%	50.6%±4.4%
BIRD	320±35	529±92	13389±2452	1741±73	70.2%±1.3%	89.2%±3.8%	48.2%±1.8%	27.2%±1.3%	44.2%±3.2%
Poisoned	Original	8.0±4.8	19±87	150±12	146±49	6.5%±1.4%	1.3%±2.7%	16.5%±3.4%	11.2%±2.8%	22.5%±1.8%
PD	140±9	325±53	6180±1817	853±162	-	-	-	-	-
Finetuning	10.6±0.9	21±65	173±24	159±63	6.7%±1.5%	5.31%±8.3%	18.4%±2.1%	12.5%±3.3%	21.5%±1.6%
BIRD	269±49	543±52	12694±2194	1686±83	70.9%±4.8%	90.7%±3.0%	46.7%±3.9%	25.1%±2.6%	43.1%±6.5%
4.1 Trigger Restoration and Backdoor Detection
Baselines. There are no existing works on detecting backdoored policies. $^{1}$  Therefore, we select two backdoor detection techniques designed for supervised classifiers. The first is the widely used NC method [48], and the other is the state-of-the-art method, Pixel [42]. They can be applied to perturbation-based attacks (TrojDRL and Marnet), where the action space is discrete. In particular, we collect a set of states and actions of a given policy and run these methods by treating each possible action as the target class. We then apply their detection method (i.e., using the trigger's  $l_{0}$  norm as the metric to find if there is an outlier trigger) to decide whether the policy is backdoored or not.
Design and Metric. For each of the nine attack scenarios, we trained 10 policies, with five of them being backdoored policies, five being well-trained clean policies. We applied BIRD (with  $\epsilon = -0.9$ ) and the baseline methods to identify the backdoored policies from the 10 policies, and reported the F1 score of the detection result. Both BIRD and the baseline methods make the detection decision for each individual policy without relying on other policies. Therefore, the total number of policies does not affect the detection process.
Results. Fig 2 shows the backdoor detection performance of all three methods in the nine attack scenarios. The
figure first shows that the baseline methods perform poorly in all the attacks where they can be applied. This result confirms that without considering the sequential decision-making nature of DRL, these baseline methods have limited generalizability in detecting DRL backdoor attacks. In contrast, BIRD has a much higher (in some cases perfect or near perfect) F1 score in each setup, demonstrating the effectiveness of our method and its superior advantage over existing methods. As we will elaborate in Section 4.3, both our trigger restoration technique and novel detection metric contribute to this remarkable result. Additionally, we show in Supplement S3 that the triggers obtained from BIRD have higher fidelity than those restored by the comparison baselines.
Figure 2: Backdoor detection F1 scores of the selected methods in the nine attack setups. "SI" stands for the Atari SpaceInvaders environment. "QMIX" and "COMA" stands for the attack against QMIX and COMA policies in the SMAC environment. Given that NC and Pixel cannot be applied to adversarial agent attacks, we do not report their results for those attacks.
4.2 Backdoor Removal
Baselines. We select two baseline methods: 1) direct finetuning of the backdoored policy in the corresponding clean environment, denoted as "Finetuning", and 2) an existing DRL backdoor defense
$^{1}$ Existing DRL defense method [2] applies their defense procedure to each policy without distinguishing whether it is backdoored or not.
Table 2: Ablation study results. "NR" refers to neuron re-initialization. For  $||\Delta ||_0$ , we carefully tune the threshold and report the best result.
Attack scenario	F1 score	Poisoned
φ(π, Δ)	||Δ||0	w. NR	w/o. NR
Qbert	0.909	0.571	12694±2194	0.0%±0.0%
COMA	1.000	0.347	90.7%±3.0%	6.7%±2.1%
YSNP	0.667	0.263	46.7%±3.9%	29.5%±4.1%
Table 3: BIRD against attack variations. Each column represents one variation (e.g., Col.3 means circle,  $3 \times 3$ , 0.1). "Clean" and "Poisoned" refer to the performance in clean and poisoned environments.
Default setup	Shape	Size	PR
circle	triangle	5 × 5	7 × 7	0.2	0.4
F1 score	1.0	0.909	1.0	0.833	0.889	1.0	0.889
Clean	529±29	536±68	609±36	601±85	636±93	540±56	471±83
Poisoned	543±52	501±47	575±24	557±61	629±72	544±61	428±33
designed for single-agent perturbation-based attacks - Provable Defense (PD) [2]. We do not consider another existing defense [15] as it is still a pre-print paper without a public implementation. We also do not include the large body of backdoor unlearning techniques designed for supervised classifiers [34, 25, 24, 61, 39, 68], which cannot be directly applied to our problem.
Design and Metric. For each attack scenario, we apply the selected methods to defend the 5 backdoored policies trained in Section 4.1. We use the agent's average reward/winning rate over 1000 game rounds as the evaluation metric. We report the mean and standard deviation of the metrics over the five policies in the clean and poisoned environment before and after applying the defense. We construct the poisoned environment by adding the ground truth trigger to every state. To demonstrate the statistical significance of our results, we also conduct a paired t-test to compare BIRD with the selected baselines and report the p-value. For BIRD, we select the number of reinitialized neurons  $L \in [10,30]$  for each policy based on the retraining performance in the first few iterations. We reinitialize the values of the selected neurons as zero.
Results. Table 1 presents the defense results of the selected methods. First, we observe that policies directly fine-tuned in clean environments still perform poorly in the poisoned environment, indicating that the backdoor is not removed. Second, PD demonstrates certain defense efficacy, as it increases the agent's reward in the poisoned environments. However, PD significantly reduces the agent's performance in clean environments. We suspect that this is due to the fact that PD project is every state to a possible clean state. This is problematic when PD cannot correctly project the organically clean states, which lowers the quality of the projected states and thus reduces the agent's performance in clean environments. Additionally, PD involves computing singular value decomposition when conducting the projection, which is computationally expensive. The average run time of PD is at least  $2 \times$  larger than BIRD. Finally, the policies retrained by BIRD demonstrate the highest reward/winning rate in poisoned reward across all attacks, which is almost similar to the original backdoored agent's performance in the clean environment, verifying BIRD successfully removed the backdoor. Moreover, BIRD can well maintain the agent's performance in a clean environment. In conclusion, BIRD is much more effective, efficient, and generalizable than existing defenses in robustifying a backdoored policy and maintaining its utility in a clean environment. In addition, we also show that BIRD outperforms two other alternative defenses, where we use the trigger restored by NC and Pixel and apply our backdoor removal method to retrain the backdoored policies (See Supplement S3).
In this experiment, we conduct three ablation studies to verify the efficacy of our three key designs: generative model (Section 3.2), detection metric (Section 3.3), and neuron re-initialization (Section 3.4). We select one game from each attack for this experiment: Qbert (non-targeted attack) and COMA for the perturbation-based multi-agent attack and YSNP for the adversarial agent attack. In the following, we discuss our ablation studies on the detection metric and neuron re-initialization while leaving the experiment on the generative model to Supplement S3.
Detection metric. For each attack scenario, we reuse the ten policies and their restored triggers from Section 4.1. We then replace our reward-based metric  $\phi(\pi, \Delta)$  with the commonly used metric  $-\|\Delta\|_0$  and deem the policy with  $\|\Delta\|_0$  smaller than a certain threshold as the backdoored one. Table 2 (Col. 2&3) shows that using our proposed metric leads to significantly higher detection rates than  $\|\Delta\|_0$ , verifying the effectiveness of our design. As mentioned in [48, 16], detecting backdoors using trigger size is difficult because noisy perturbations or adversarial examples can also have a small size. In contrast, our metric is designed based on behavioral differences between clean and backdoored policies, making it more robust to noisy perturbations. Furthermore, BIRD are insensitive to the threshold  $\epsilon$ , significantly improving its practicality (See Supplement S4).
Neuron Re-initialization. We reuse the five backdoored policies from the three attacks and compare the effectiveness of our proposed neuron re-initialization method against directly retraining the entire policy network with the restored trigger using Eqn. (4). Table 2 (Col. 4&5) shows the retrained policies' performance in the environment poisoned by the ground truth trigger with and without neuron re-initialization. Comparing Col. 2&3 can reflect the efficiency of using total reward as the detection metric and comparing Col. 4&5 can reflect the efficiency of neuron re-initialization. Our results demonstrate the importance of neuron re-initialization in backdoor removal. This also confirms our claim in Section 4.2 that directly retraining the policy with the restored trigger may not fully eliminate the ground truth trigger when they are not very similar.
4.4 Attack Variations and Adaptive Attacks
We evaluate the robustness of BIRD against different attack variations and two potential adaptive attacks in the SpaceInvaders environment of the non-targeted perturbation-based attack.
Attack variations. Based on the default attack setup, where the trigger is a  $3 \times 3$  square, and the state poisoning rate is 0.1, we vary the trigger shape (circle and triangle), size ( $5 \times 5$ ,  $7 \times 7$ ), and poisoning rate (0.2, 0.4). For each variation, we train five backdoored policies, apply BIRD to detect them from the five clean policies mentioned in Section 4.1, and report the detection performance in Table 3. The results show that BIRD maintains a high and stable detection performance when dealing with these variations. Table 3 also presents the backdoor removal performance of BIRD on these attacks, which is consistent with the detection result.
Adaptive attack-1. We first consider a straightforward adaptive by allowing the trigger to vary its shape and locations at different states. However, even after carefully tuning the training parameters, the attack was unable to achieve comparable performance as it is in the static setting. For SpaceIn-vaders, the agent's average reward in the poisoned environment is only reduced from 580 to 510 for non-targeted attack and (to 445 for targeted attack). This motivates future work to design more effective backdoor attacks with dynamic triggers. Nevertheless, even if such an attack succeeds, as discussed in Section 3, BIRD is designed to be robust against it.
Adaptive attack-2. We further consider an adaptive attack against our neuron re-initialization mechanism, a key design for backdoor removal. When training the backdoored policy, we add an additional regularization to constrain the  $l_{2}$  norm of the policy weights, which avoids neurons with ultra-high activation values. We train five backdoored policies with this attack, apply BIRD, and record the backdoor removal performance. The policies' average reward in the clean environment before and after applying BIRD is 582 and 563, respectively, and the average reward in the poisoned environment is 170 (before) and 512 (after). These results demonstrate the robustness of BIRD against this adaptive attack (See Supplement S2 for more implementation details).
Additionally, we discuss generalizing BIRD to a setup where we do not assume accessing the agent's value networks, or a more challenging scenario where the attacker intentionally attaches a benign value network instead of the backdoored one to make the attack more stealthy(See Supplement S3). Moreover, we also demonstrate the computational efficiency of BIRD and its insensitivity to subtle variations in key hyper-parameters  $(\alpha$ , detection threshold  $\epsilon$ , and the number of reinitialized neurons  $L$ ). We present these experiments in Supplement S4.
We present BIRD, a method for detecting and removing backdoors in DRL policies. By analyzing the attack process and unique behaviors of backdoor agents, we formulate the trigger restoration as an optimization problem and design a novel metric to detect backdoored policies. We also propose a novel finetuning method to remove the identified backdoor. Extensive experiments against various attacks demonstrate the effectiveness, efficiency, and generalizability of BIRD, as well as its robustness to different attack variations.
This work points to several promising directions for future research. First, we evaluate two adaptive attacks in Section 4. Our future work will explore other possible attacks (e.g., using watermark triggers) and extend BIRD to defend against these attacks. Second, while our trigger restoration technique may not always yield a restored trigger that perfectly matches the ground-truth trigger, it is still effective in achieving a decent detection and removal performance. We plan to refine our
trigger restoration technique to improve its restoration fidelity in future work. Third, in addition to the detection and unlearning mechanism utilized in BIRD, we will explore alternative solutions to render the backdoor ineffective. For example, we can filter out the trigger before feeding the state representation to the backdoored agent without finetuning it. Fourth, we acknowledge that besides the neuron re-initialization method discussed in Section 3.4, existing works also design other techniques to identify and remove the backdoor shortcuts (e.g., neuron pruning [27, 53, 58], [51]). In future work, we will explore the effectiveness of integrating these techniques into our approach. Fifth, as discussed in Section 2, BIRD differs from other robust RL methods in terms of attack/problem setups. As part of future work, we will investigate generalizing these defenses designed for other attack setups to defend against backdoor attacks. Finally, while we evaluate BIRD in various game environments, we plan to further extend the backdoor attacks and defenses to broader types of environments, such as multi-agent competitive environments [29, 43] and extensive-form environments [23, 1].
This project was supported, in part by ARL Grant W911NF-23-2-0137, IARPA TrojAI W911NF-19-S0012, NSF 1901242 and 1910300, ONR N000141712045, N000141410468 and N000141712947. We also thank the Center for AI Safety for their support of the Compute Cluster. Any opinions, findings, and conclusions in this paper are those of the authors only and do not necessarily reflect the views of our sponsors.
[1] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. arXiv preprint arXiv:1902.00506, 2019.
[2] Shubham Kumar Bharti, Xuezhou Zhang, Adish Singla, and Xiaojin Zhu. Provable defense against backdoor policies in reinforcement learning. In NeurIPS, 2022.
[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[4] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In S&P, 2017.
[5] Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In ICLR, 2019.
[6] Jinyin Chen, Xueke Wang, Yan Zhang, Haibin Zheng, Shanqing Yu, and Liang Bao. Agent manipulator: Stealthy strategy attacks on deep reinforcement learning. Applied Intelligence, 2022.
[7] Shuo Chen, Yue Qiu, and Jie Zhang. Backdoor attacks on multiagent collaborative systems. arXiv preprint arXiv:2211.11455, 2022.
[8] Yanjiao Chen, Zhicong Zheng, and Xueluan Gong. Marnet: Backdoor attacks against cooperative multi-agent reinforcement learning. IEEE Transactions on Dependable and Secure Computing, 2022.
[9] Edward Chou, Florian Tramer, and Giancarlo Pellegrino. Sentinel: Detecting localized universal attacks against deep learning systems. In IEEE Security and Privacy Workshops (SPW), 2020.
[10] Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jieuxiang Gu, Tong Sun, and Xia Hu. Towards interpreting and mitigating shortcut learning behavior of nlu models. arXiv preprint arXiv:2103.06922, 2021.
[11] Min Du, Ruoxi Jia, and Dawn Song. Robust anomaly detection and backdoor attack detection via differential privacy. arXiv preprint arXiv:1911.07116, 2019.
[12] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI, 2018.