BIRD: Generalizable Backdoor Detection and Removal for Deep Reinforcement Learning
Authors: Xuan Chen
Wenbo Guo
Guanhong Tao
Xiangyu Zhang
Dawn Song
Venue: NeurIPS 2023 poster (2023)
Keywords: Backdoor Defense
Deep Reinforcement Learning

Abstract: Backdoor attacks pose a severe threat to the supply chain management of deep reinforcement learning (DRL) policies. Despite initial defenses proposed in recent studies, these methods have very limited generalizability and scalability. To address this issue, we propose BIRD, a technique to detect and remove backdoors from a pretrained DRL policy in a clean environment without requiring any knowledge about the attack specifications and accessing its training process. By analyzing the unique properties and behaviors of backdoor attacks, we formulate trigger restoration as an optimization problem and design a novel metric to detect backdoored policies. We also design a finetuning method to remove the backdoor, while maintaining the agent's performance in the clean environment. We evaluate BIRD against three backdoor attacks in ten different single-agent or multi-agent environments. Our results verify the effectiveness, efficiency, and generalizability of BIRD, as well as its robustness to different attack variations and adaptions.

BIRD: Generalizable Backdoor Detection and Removal for Deep Reinforcement Learning
Xuan Chen', Wenbo Guo Guanhong Tao', Xiangyu Zhang', Dawn Song*
Purruue University
UC Berkeley
{chen4124, henrygub, taog, xyzhang)@cs. purdue. edu {henrygwb, dawnsong)dberkeley edu
Abstract
Backdoor attacks pose a severe threat to the supply chain management of deep reinforcement learning (DRL) policies. Despite initial defenses proposed in recent studies, these methods have very limited generalizability and scalability. To address this issue, we propose BIRD, technique detect and remove backdoors from pretrained DRL policy in a clean environment without requiring any knowledge about the attack specifications and accessing its training process. By analyzing the unique properties and behaviors of backdoor attacks, we formulate trigger restoration as an optimization problem and design novel metric detect back- doored policies. We also design finetuning method to remove the backdoor, while maintaining the agent's performance in the clean environment. We evaluate BIRD against three backdoor attacks in ten different single-agent or multi- agent environments. Our results verify the effectiveness, efficiency, and generalizability of BIRD, as well as its robustness to different attack variations and adaptions.
1 Introduction
Recent studies [59 20 49,8 demonstrate that an attacker can inject a backdoor into a DRL agent's policy. In particular, the attacker adds trigger into the environmental state during the training process and forces the agent to take a backdoored/poisonee action that diminishes its total reward given by the environment. To accomplish this, they manipulate the agent's 's reward function and assign high reward whenever the agent takes a poisoned action at poisoned state. We denote the original reward given by the environment as the actual reward and the attacker teorkkkr-maippllttee reward function as the poisoned reward function. As a result, the agent learns to take poisoned actions at these states, expecting the environment to assign a high actual reward to it. However, taking the poisoned actions actually diminishes the agent's actual reward and even causes the agent to fail its corresponding tasks.
Given that backdoored agent performs normally clean states, it is challenging detect and remove the backdoor from a pretrained policy without accessing the training process, having knowledge about the allack specifications (i.e., trigger, poisoned actions, poisoned reward function), and accessing the poisoned states during testing. Most existing backdoor defenses do not consider the sequential
decisioonmmainng nature of DRL policies [4] 16 28
[3
9 25
61 30 42 51 52 assume access
to the agent training process [11] 66) 56 66) Only a few methods provide defense for pretrained policies [5]. However, they have several limitations that hinder their practicality. Specifically, the method proposed in [ can only be applied to attacks with perturbation patch as the trigger and cannot detect whether policy is backdoored or not. Similarly, the defense proposed [5] is only
applicable to attacks where the trigger is an adversarial agent's 's specific actions.
We propose generalizable backdoor detection and removal method for DRL policies. Given pretrained agent's policy network and value function, along with the corresponding clean environment,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

we first formulate trigger restoration as an optimization problem. The objective function is crafted as searching for a perturbation to the state representation that forces the agent to take actions that maximize its value function. Since a backdoored agent is trained using a poisoned reward function that assigns high values when the trigger is present, maximizing its value function should identify the trigger. Second, we design a novel metric to detect whether the agent is backdoored or not using the resolved perturbation (denoted as restored trigger). As we will detail in Section [.3] adding the restored trigger to the clean environment will cause the backdoored agent to receive much lower actual reward from the environment but increase or not affect the clean agent's 's actual reward. Based design metric the actual reward difference of the agent before
on this behavior difference, and after adding the restored trigger to the environment. Finally, we propose a novel finetuning method to remove the detected backdoor. We design additional regularization terms to the finetuning objective function to maintain the finetuned agent's 's actual reward in the clean environment. We also introduce neuron re-initialization mechanism to ensure the backdoor can be successfully removed even when the restored trigger is not exactly the same as the ground truth one in shape and size.
We denote our method as BIRD (Backdoor Identification and Removal for DRL). We extensively evaluate BIRD in ten single-agent or multi-agent RL environments against three prevalent backdoor attacks. First, our results demonstrate that BIRD outperforms existing backdoor detection methods designed for supervised learning in detecting backdoored policies. Moreover, we show the superiority of BIRD over existing methods in removing the backdoor while maintaining the agent's 's actual reward in the original clean environment. Second, we verify the effectiveness of our key design choices through ablation studies, especially our backdoor detection metric. We demonstrate that compared with the widely used trigger size metric, our metric enables much higher detection accuracy without requiring subtly tuning the threshold. Third, demonstrate the computational efficiency of BIRD and its insensitivity to hyper-paraaee parameter choices. Furthermore, we verify the robustness of BIRD against different variations in attack and two possible adaptive attacks. To the best of our knowledge, BIRD is the first DRL backdoor defense that can detect and remove backdoors in a clean environment without requiring access the attack specifications and policy training process. Furthermore, BIRD is applicable to all kinds of existing attacks, making it highly generalizable defense mechanism.
2 Related Work
Attacks. Recent research works propose backdoor attacks against single-agent DRL [59 20 50 g
These attacks, denoted as perturbation-base attacks, add a small perturbation patch to the victim agent's state as the trigger. A backdoored policy performs normally in clean environment but takes poisoned action at poisoned states, leading to task failure. A follow-up work considers a two-agent setup with an adversarial agent and victim agent 49] Rather than perturbing the states, they leverage the adversarial agent's 's certain actions as the backdoor trigger (denoted as adversarial agent attack). More recent works generalize both perturbation- based attacks and adversarial agent attacks to multi multi-agent cooperative RL with a team of victim agents 7
Detection and Defenses. As mentioned in Section I only two existing defenses are developed to mitigate backdoors for a pretrained DRL policy. One method, proposed by [2], uses a trigger filter technique that maps possibly poisoned states back to their corresponding clean states. This attacks and cannot be applied to adversarial agent attacks.
method is designed for perturbation-ban Additionally, it relies on singular value decomposition, which limits its scalability in complicated RL environments with high-ding sigh-dimensional state spaces. Another method [15] also has limited generaliz- ability in that it is only applicable to adversarial agent attacks.
DRL and propose testing- 40 19], which perturb the environment at certain states fail a pretrained agent. To defend against such attacks, researchers either generalize existing adversarial defenses designed for supervised learning to DRL [63]
Some other research works also extend adversarial sample attacks
phase perturbation attacks 17 32
36
63
26
62
31
55 22,
57 [6] or model
the attack and defense as a two-player game and train a robust agent to find the Nash equilibrium
point 33 60 64 Similarly, some robust RI. methods also propose robust training techniques against random perturbations to the agent's observations [65 66). actions (3 44 or reward functions [67]
Due to differencee in attack setups, these methods cannot be directly applied to our problem.

---

3 Methodology
3.1 Problem Setup
Formulation. Consider Markov Decision Process (MDP) M AS, where S and A are the continuous or discrete state and action space. P is the state transition function, is the reward function, and is the discount factor. Without facing any attack, the agent's goal is to learn optimal
policy π that maximizes its expected long-term reward, 77()) = EE > R(s, T (8:) )) We also define stationary state occupancy distribution for æ,, denoted p (=
(111))) ? pp((s s|T),
and T7(T)) can also be expressed as Ex-p" (+)], where (*) is the state-value function.
Attack model. We follow existing attacks and assume the attacker injects one backdoor into the victim agent's policy. We add a superscript / to indicate backdoorre/poosooeed entity, and the backdoored policy is denoted as x'. We consider all three types of existing attacks: mngle-agent perturbation-bast attacks [20], two-agent adversarial-agen attacks [49], and multi-agent perturbation- based attacks [8]. In the following. we use single-agent perturbation-bart based attacks to derive our technique, and later we discuss the extension of our approach to the other attacks. Under perturbation-
attacks, the attacker designs the trigger as a small perturbation patch denoted as Á. At any time l, the attacker may add Á to the current state representation st, causing the backdoored agent to perceive the poisoned state s and taking a poisoned action af T (s(). We follow existing works 22 200 and assume the attacker only perturbs the state representations perceived by the victim agent (victim agent's observation) without altering the actual state (underlying physical state). As such, the state transition still depends on the actual state rather than the perceived/poisoned one, i.c., S141 ~ P(st, df). To inject the backdoor, the attacker needs to manipulate the victim agent's 's reward function, assigning the agent an ultra-high poisoned reward when it takes the poisoned action at poisoned states.
Defense assumption and goals. We do not assume access to the attack training process. Instead, we are provided with an agent with a pretrained policy и and the corresponding state-value network V (s) or action-value network Q7 (s, T(s)). and lacks the knowledge of whether the policy backdoored or not. We are also given a clean environment for testing and debugging purposes. This setup simulates a practical scenario where the defender needs to verify an agent's robustness and safety before deploying it in a potentially poisoned environment, especially in critical fields. We further assume access to limited computational resources that support testing or finetuning given policy but not training a policy from scratch. We also do not assume any knowledge about attack specifications, including the ground-trun -ruth trigger, the poisoned actions, or the reward manipulation strategy employed by the attacker. We aim to develop a full-stack defense with the following three steps: 1) Trigger restoration restore the potential trigger(s) from the given policy in the clean environment, 2) Backdoor detection determining whether the restored trigger is associated with an injected backdoor and thus detect whether the given policy is backdoored or not, and 3) Backdoor removal- unlearning the detected backdoor from the backdoored policy without affecting its performance in the clean environment.
3.2 Trigger Restoration
Key insights We derive the intuition from the training process of backdoor attacks. Recall that the attacker needs to poison the victim agent's 's reward function. Under the poisoned reward function, the agent receives a high total reward, denoted as n7(x') = =.. pT (s) Ea <'(a'ss) |s') R'((' a'), when taking trigger actions at the poisoned states. This indicates that given an agent with a fixed policy, adding a proper perturbation (trigger) its state will activate the backdoor, resulting in significant increase in the agent's 's total poisoned reward under the poisoned reward function.
restore the backdoor trigger by searching for small perturbation to the states that maximizes the agent's 's total poisoned rewards under the poisoned reward function. Since we do not assume accessing to the poisoned reward function, we use the agent's ' state-value or action-value function to compute the total reward, which is also poisoned during the attack training. Formally, this perturbation can be obtained by solving the following objective function for given policy
Intuitively,
maxa [x((s) 7(s + ()Qx(s + (,
A has the same dimension as 8, where each element in A represents the perturbation to the corre- sponding element in s. Solving Eqn. ① yields potential trigger for backdoored policy but not for
3

---

clean policy. Given that clean agent's 's reward and value function is not poisoned, solving Eqn. would likely produce a universal perturbation that guides the agent take better actions further increase its reward. However, this is typically challenging for complicated environments where the agent's task difficult. Besides, we integrate specific designs to eliminate these noisy perturbations.
Technical details. We follow existing perturbation-based attacks [20 g and consider the state representation as the snapshot of the current environment, s RPx4 with each element sij is normalized to [0, 1]. Directly solving Eqn. ① likely results in noisy perturbations for both backdoored and clean policies. To eliminate these noises, we propose two regularization terms to constrain the search space. First, given that the attacker typically uses small trigger size to stay stealthy, we constrain R1 (A) = ||||| which yields a Á with a small number of non-zero elements. This regularization rules out noisy perturbations with large sizes. Second, all existing perturbation- attacks use dense patches as triggers because they are easily applicable in the physical world. In contrast, noisy perturbations are typically scattered. We add another regularization to constrain the smoothness/density of A, R2(A) = Ex, ( (Ai+1.) A.,)² + E.. (A A.,)² This additional regularization term further rules out more noisy perturbations that are small and scattered.
Besides adding regularizations, we also design the perturbation A to be produced by a generative model. As demonstrated in previous works (5). assuming a generative model gives more stable and less noisy result than directly solving the values in an unknown perturbation to neural network input. Since s is normalized to [0. 1j. this constrains the maximum allowed perturbation to be 1 or -1, indicating Dg E [-], Since there is no common distribution with a value range of [-1, 1]. we satisfy this constraint by introducing a set variable {pty (pt))i=ippjjivv We define Pij to follow Beta(arj. Biz) distribution, parameterized by ouj and Bej. Given that Pij E [0, 1J. we then compute
A.j = 2p.j 1, which naturally guarantees Dij lies within the range of 1-,, 1]. The objective function can be rewritten as maxa 1ig.81 1.g B,,,BiE,''
E. T(s + 2p 1)Qa(s + 2p 1. 1 + 2p- 1))]. where E R8x4 and B stands for the joint distribution II., Beta(ak), (1y)) By sampling perturbations from the beta distribution and computing the mean as the final result, we can reduce noise and obtain a trigger with higher fidelity. With this generative model, we double the number of parameters needed to be resolved. Given that the beta distribution's output sparsity depends more on the difference between a and than their actual value. We further fix the value of and design ßig- = Gij + a, e E R to reduce the number of parameters in our objective function. Supplement S1 provides more insights about this parameter reduction design and Supplement S4 shows our method is not sensitive to the choice of a.
So far, we consider the trigger's location and pattern to be fixed across different states (time steps). A straightforward attack adaption could change the trigger's location and even its shape at different states. To handle this scenario, we design e as a function of each state fs(s,, parameterized by 0. This design is also applicable to the original attack, where all the states have the same output for fo(.). Putting all designs above together, we obtain the trigger restoration objective function.
maxe ((O) Ex-p' [Ep.,B, B, (n.(x(p.)) + X R, (p.) + R2.)(p B. Beta(a, a + (fo(e))a))
(n(.))))) D (s + 2p. 1)Q., (+ + 2p. 1, + 2p. - 1)).
where a and A2 are hyperparameters. Ps and B, is specific for each state 8. Eqn. (2) can not be directly solved because P, is sampled from an unknown distribution B.. We leverage the policy gradient method [21] [8] to compute the gradient V and solve the Eqn. (2) following the typical on-policy REINFORCE algorithm 4T. Our final perturbation for each state is A. a 2( 2o - 1, where fx)) is the mean of the joint beta distribution B.
To ensure that the perturbations we compute actually influence the agent's actions and affect its reward during execution, rather than generating simple adversarial examples for the value network that cannot affect the agent's 's actions, we only use the agent's 's value network to compute average rewards. We do not compute the gradient of 0 with respect to the value network or update the value network itself. Supplement S1 provides a detailed algorithm for solving Eqn. (2) and also discusses further why we are not solving adversarial examples for the value network.
3.3 Backdoor Detection
Key insights. Our backdoor detection methods draw insights from the unique behaviors of backdoored policies, which perform well in clean environments but poorly in poisoned ones. For backdoored

---

policy, if its restored trigger A has similar efficacy as the ground-trun trigger A, adding it back to the clean environment will significantly reduce the agent's 's total actual reward. However, this is not the case for clean agent trained using the actual reward function. Since the value function of a clean agent not poisoned, solving Eqn. 2 will either result in a perturbation that guides the agent to select better actions and increase its reward or perturbation that cannot significantly impact the agent's performance, if the agent is already well-trained or the environment is too complicated. In both cases, adding the perturbation to the environment will not reduce the agent's 's actual reward.
As such, we propose a novel metric distinguish between backdoored and clean policies, which is based on the agent's is actual reward reduction before and after poisoning the environment with restored perturbations. This approach differs from existing detection methods designed for supervised classifiers (e.g., 4 48 38 422), which rely on the trigger size (l [o) as the metric. This metric sensitive to the ground-truth trigger size. Our detection method is designed based on the unique behaviors of backdoor attacks in DRL and is more accurate and robust (demonstrated in Section
Technical details. Given an agent's policy म, we first run in the environment for K rounds and record the agent's average actual reward given by the environment, denoted as 17(T) = (k) R(s{ .T(((())). We then simulate an extreme attack case by adding the restored per- turbation A, to the agent's policy input at each time step, run the agent for K rounds, and record R E(A)
(k) a(s! (R) + A..)). For adversarial agent attacks, since A corresponds to the state representation of the adversarial agent's trigger action,
the average actual reward, denoted as f7 A) = * Exe) (k) R(s,
adding A., to 8y simulates the trigger action. Here, 7T,,, A) k E(e) R(s +A.,. , (s{ + +A,))) given that the trigger action indeed changes the actual state of the environment. Next, we compute the reward reduction rate o(T, A) T,, A) [TT)) /MTuxx where Mmax is normalization term, repre- senting the maximum reward difference of the environment. ó(T, A) E [-1, 1, 1]. If A is similar to the ground-truth trigger, it will result in very large reward reduction and thus negative 6(T, 1 A) near -1 Conversely, a A restored from a clean policy will produce a positive o0 π, A) or a Φ(π, A) R O.
(k)
Therefore, we treat as backdoored policy and A as the trigger if Φ(T,, A) S c. We consider practical scenario where we do not have a set of clean policies to help select a proper threshold. Instead, we need to make decision for each individual policy by hand. We set an aggressive value with = -0.9 to rule out false positives. Here, we consider all of the clean policies and backdoored policies are well-trained, i.e., their reward in the clean environments that there are some sub-optimal agents that are backdoored, the detection threshold should be more conservative. More discussions on the detection threshold under different situations can be found in Supplement S2. Later, in Section 4 we demonstrate that, due to the significant performance difference between backdoored policies and clean policies under our restored perturbation, even setting c = -0.9 introduces only negligible false negatives and thus provides a nearly perfect detection performance. Supplement $1 presents our detection algorithm.
nearly optimal. In case
3.4 Backdoor Removal through Unlearning
Naive solutions and pitfalls. A naive solu- tion for backdoor removal is to add the re- stored trigger to the state and retrain the pol- icy under the actual reward function, which penalizes poisoned actions a'. We test this solution and find that it indeed improves the agent's performance when presented with the restored trigger. However, the retrained policy remains vulnerable the ground- truth trigger due to differences between the restored and ground-truth trigger. We try address this issue by varying the restored trigger (e.g., adding random noise) during the training. Our empirical findings suggest that the ground truth trigger is only unlearned when the restored one is very similar to it, which is not always the case. Furthermore, we discover that the performance of retrained policies in the clean environment oftentimes decreases after retraining. These initial explorations highlight two challenges: (1) how to remove the ground-truth trigger when the restored one has some differences from it, and (2) how to maintain the retrained agent's 's performance in the clean environment.
0.8
Reshored
Backdnoed
0.4
traper
Policy
0.0
25
30
40
Top
Leuurons
Ground mil
digger
deteec kil
Figure 1: Overlapped shortcuts between the restored trigger and the ground truth one. The bar plot shows the overlapping percentage among the top X neurons with the highest value.
5

---

Our Designs. As extensively studied in existing research [54, [33] [00, injecting a backdoor in neural networks equivalent to building one or a few shortcuts from the backdoor trigger to the target out. In DRI, these shortcuts connect the poisoned states with the poisoned action. When the trigger presented, only the neurons on the shortcuts get activated. Given that our restored trigger could indeed trigger similar poisoned actions as the ground truth one, we hypothesize that even if the restored trigger is not highly similar to the ground truth one, their corresponding shortcuts may have some overlap. To test this hypothesis, we conduct empirical test where compared the neurons activated by the restored trigger with those activated by the ground truth trigger. As shown in Fig. [ the neurons with high activation values have a large overlap for these two triggers, confirming our hypothesis. Guided by this intuition, we propose to remove these most vulnerable and overlapping shorteuts instead of directly unlearning the trigger. In particular, the backdoored policy input and record the activation values of all the neurons. We select the top T. neurons with the highest activation value and re-initialize their weights erase the most vulnerable shortcuts. Finally, we retrain the policy in the environment poisoned with the restored trigger under the actual reward function. The objective function can be written max; (TTT, A) , where Te denotes the retrained policy. We retrain the entire policy network rather than only the reset neurons to give the policy enough capacity to let the agent learn how perform in the poisoned environment. See Supplement S2 for more details about our neuron selection and re-initalization process.
first add the restored trigger to
The second challenge can be addressed by incorporating an additional regularization term, i.e. (n(=)) - -())) S 61, where x' is the original backdoored policy. This ensures that the total reward difference between the retrained policy and the original policy is within certain threshold é This constraint helps to maintain the retrained agent's performance in the clean environment, given that the original backdoored agent should have performed well in a clean environment by design. However, this regularization is not feasible as (T(T))) | is intractable during training. To overcome this, we propose to use the KL divergence between Tá and x' as a proxy since is much easier to compute. The following inequality states the relationship between KL divergence and value difference.
(m(=²) 1/(=)) & Cmax, snpp" (L)(m)))]]= (+)).
(3)
See Supplement S1 for the derivation. Based on Eqn. 33) constraining the KI. divergence between ' and A can bound (n(T))) n)(=')). Then, our final retraining objective function can be written
max A), (I((T())]l (s)) <1.
(4)
We leverage the state-of-art PPO algorithm 37 to solve Eqn. 4 which guarantees a fast convergence. During retraining, we periodically add A to simulate the poisoned states and keep a certain portion of clean states compute (T((Ta(s) (s) [| (s)). Supplement $1 shows the detailed algorithm of our choices.
backdoor removal methods. Supplement S2 states our implementation and
Note that our proposed defense (from trigger restoration to backdoor removal) can be directly applied to adversarial agent attacks [49] and multi-agent perturbation- attacks S (See Supplement S1 for detailed discussion).
4 Evaluation
We compare BIRD with three state-of-the-ar backdoor DRL attacks: TrojDRL. 20). Backdoorl 44 and Marnet [8]. Following their papers, select the environments and used their default attack setups to train backdoor policies (see Supplement S2 for more details). For TrojDRL, we select six Atari games from the OpenAl Gym 3: Breakout, SpaceInvaders (SI), Qbert, and Seaquest, CrazyClimber, Pong. Due to limited space, we put the results of CrazyClimber and Pong in Supplement S3. TrojDRL considers targeted and non-targeted attacks based on whether the backdoored action is prespecified or not. In this section, we report the results of non-targeted attacks, where detection is more challenging, and leave the results of targeted attacks in Supplement S3. For Backdoorl, we choose the You-Shall- (SH), and Run-To- To---TT-GGr Goo-Goll-Ama (RTGA) games from the MuloCo Not-Pass (YSNP), We select the SMAC environment 47 for Marnet and conduct two attacks, one environment 455
against the policies trained by QMIX 35 (a multi-agent Q-learning algorithm), and the other against the policies trained by COMA [12 (a multi-agent policy gradient algorithm). Note that by default, all the perturbation-based attacks consider the trigger with fixed shape/size/location, and adversarial agent attacks set the trigger as a sequence of fixed actions taken at the beginning of each game. We include the performance of BIRD against more recent attack (6) in Supplement S3,

---

Table I: Performance of different defenses nine attack scenarios. "Clean" "poisoned" refer to original clean environment and an environment poisoned by the ground truth trigger. The numbers in each element are std. "-" means that method is not applicable the corresponding attack. The Atari games' results are rewards and the rest are winning rates. Supplement S3 shows the p-value for each attack scenario is smaller than 0.001.
Envizonments
Methods
Breakout
Spacelovaders
Qbeet
Seaquest
YSNP
RTGA
Origanl
298::37
630299
13488:189 1897
1719
74.46=2.58
93.09-222
52.86=4.59
26.99:3366
49.78:54.85
Clean
FD
142432
278.944
441111228
932199
Firetuning
817-90
614+104
13151+2233
1785+98
7368-2666
90.99+55.8
53.98-1.76
30.19+288
50,08+4448
BRD
350
929
1338912452
1241873
70.28 1.3%
59.29-18.88
48.29:1.8%
27.25
1.3%
64.2% 3.25
Origini
K0544
19:47
150s:12
1466449
6.661148
1.384227
1636:334
11.294288
22.5%:188
Poisoned
PD
14019
325
6180&1817
$53s:
Finetuning
105:209
21:055
1734244
139283
5.7561555
5.319488.56
18.46
12.5%
21.5%:1166
BRD
269
$43
12994
1696-13
70.98:4.85
90.74+13.8
7.79
25.19446..6
43.16:16.55
4.1 Trigger Restoration and Backdoor Detection
Baselines. There are no existing works detecting backdoored policies Therefore, we select two backdoor detection techniques designed for supervised classifiers. The first is the widely used NC method [48] and the other is the state-of-the-art method, Pixel [42]. They can be applied to perturbation- based attacks (TrojDRL and Marnet), where the action space is discrete.
particular, we collect set of states and actions of given policy and run these methods by treating each possible action as the target class. We then apply their detection method (i.e., using the trigger's lo norm as the metric to find if there is an outlier trigger) to decide whether the policy is backdoored or not.
Design and Metric. For each of the nine attack scenarios, we trained 10 policies, with five of them be- ing backdoored policies, five being well-trained clean policies. We ap- plied BIRD (with = -.9) and the baseline methods to identify the back- doored policies from the 10 policies, and reported the Fl score the detec- tion result. Both BIRD and the base- line methods make the detection deci- sion for each individual policy with- out relying on other policies. There- fore, the total number of policies does not affect the detection process.
NC
Piasl
BIRD
1.0
0..
0.2
0.0
Breakeut
SI
Qbert
Seaquna
QMIX
COMA
YSNP
501
RTGA
Artack Scenarios
Figure 2: Backdoor detection Fl scores of the selected methods nine attack setups. "SI" stands for the Atari SpaceInvaders environment: "QMIX" and "COMA" stands for the attack against QMIX and COMA policies in the SMAC environment. Given that NC and Pixel cannot be applied to adversarial agent attacks, we do not report their results for those attacks.
Results. Fig shows the backdoor de- tection performance of all three meth- ods in the nine attack scenarios. The figure first shows that the baseline methods perform poorly in all the attacks where they can be applied. This result confirms that without considering the sequential decision- making nature of DRL, these baseline methods have limited generalizability in detecting DRL backdoor attacks. In contrast, BIRD has much higher (in some cases perfect effectiveness of our method and its superior advantage over existing methods. As we will elaborate in Section 4.3 both trigger restoration technique and novel detection metric contribute this remarkable result. Additionally, we show in Supplement S3 that the triggers obtained from BIRD have higher fidelity than those restored by the comparison baselines.
perfect) Fl score in each setup, demonstrating the
4.2 Backdoor Removal
Baselines. We select two baseline methods: 1) direct finetuning of the backdoored policy in the corresponding clean environment, denoted as "Finetuning". and 2) an existing DRL backdoor defense
'Existing DRL defense method (2) applies their defense procedure to each policy without distinguishing whether it is backdoored or not.
7

---

Table 2: Ablation study results. "NR" refers to neuron re-initialization. For ||A|lo. we carefully tune the threshold and report the best result.
Table 3: BIRD against attack variations. Each column represents one variation (e.g., Col.3 means circle, 3 x 3. 0.1). "Clean" and "Poisoned" refer to the performance in clean and poisoned environments.
Attack
Fl
Poisoned
scenario
||A||]
w. NR
Delaalt semp
Stape
Obert
0.909
0.571
12694=2194
0.0%=00.0% 6.7%
Cicle
tiinule
7x7
0.2
0.4
COMA
1,000
0.347
90.7% 53.0%
acore
1.0
0,009
1.0
0.533
0.889
1.0
0.589
YSNP
0.667
46.7%
29.5%-+44.1%
Prisoned
Cloon
524-29 543
5012447
596-68
375:34
5571661
636-90
620-72
S46161
540196
471-K1
428:3
designed for single-agent perturbation- based attacks Provable Defense (PD) 2]. We do not consider another existing defense [13] as it is still pre-print paper without public implementation. We
also do not include the large body of backdoor unlearning techniques designed for supervised classifiers 34
25
24
61
39
68
which cannot be directly applied to our problem.
Design and Metric. For each attack scenario, we apply the selected methods to defend the backdoored policies trained in Section 4.1 We use the agent's 's average reward/winning rate over 1000 game rounds as the evaluation metric. We report the mean and standard deviation of the metrics over the five policies in the clean and poisoned environment before and after applying the defense. We construct the poisoned environment by adding the ground truth trigger to every state. To demonstrate the statistical significance of our results, also conduct a paired t-test to compare BIRD with the selected baselines and report the p-value. For BIRD, we select the number of reinitialized neurons
L E [10, 30] for each policy based the retraining performance in the first few iterations. We reinitialize the values of the selected neurons as zero
Results. Table presents the defense results of the selected methods. First, we observe that policies directly fine-tuned in clean environments still perform poorly the poisoned environment, indicating that the backdoor is not removed. Second, PD demonstrates certain defense efficacy, as it increases the agent's 's reward in the poisoned environments. However, PD significantly reduces the agent's performance in clean environments. We suspect that this is due to the fact that PD project s every state to a possible clean state. This is problematic when PD cannot correctly project the organically clean states, which lowers the quality of the projected states and thus reduces the agent's 's performance in clean environments. Additionally, PD involves computing singular value decomposition when conducting the projection, which is computationally expensive. The average run time of PD is at least 2x larger than BIRD. Finally, the policies retrained by BIRD demonstrate the highest reward/winning rate in poisoned reward across all attacks, which is almost similar to the original backdoored agent's performance in the clean environment, verifying BIRD successfully removed the backdoor. Moreover, BIRD can well maintain the agent's 's performance in a clean environment. In conclusion, BIRD is much more effective, efficient, and generalizable than existing defenses in robustifying a backdoored policy and maintaining its utility in a clean environment. In addition, we also show that BIRD outperforms two other alternative defenses, where we use the trigger restored by NC and Pixel and apply our backdoor removal method to retrain the backdoored policies (See Supplement S3).
4.3 Ablation Studies
In this experiment, we conduct three ablation studies to verify the efficacy of our three key designs: generative model (Section 3.2p, detection metric (Section [33)). and neuron re-initialization (Sec- tion 3.4). We select one game from each attack for this experiment: Qbert (non-targeted attack) and COMA for the perturbation-bant
multi-agent attack and YSNP for the adversarial agent attack. In
the following, we discuss our ablation studies on the detection metric and neuron re-initialization while leaving the experiment on the generative model to Supplement S3.
Detection metric. For each attack scenario, we reuse the ten policies and their restored triggers from Section 4.1 We then replace our reward-based metric (( A) with the commonly used metric || and deem the policy with || A smaller than certain threshold as the backdoored one. Table 2 (Col. 2&:3) shows that using our proposed metric leads significantly higher detection rates than || verifying the effectiveness of our design. As mentioned in [48] [6),, detecting backdoors using trigger size difficult because noisy perturbations or adversarial examples can also have a small size. In contrast, our metric is designed based on behavioral differences between clean and backdoored policies, making it more robust to noisy perturbations. Furthermore, BIRD are insensitive to the threshold significantly improving its practicality (See Supplement S4).
8

---

Neuron Re-initialization. We reuse the five backdoored policies from the three attacks and compare the effectiveness of our proposed neuron re-initialization method against directly retraining the entire policy network with the restored trigger using Eqn. 4 Table 2 (Col. 4&:5) shows the retrained policies' performance in the environment poisoned by the ground truth trigger with and without neuron re-initialization. Comparing Col. 2&33 can reflect the efficiency of using total reward as the detection metric and comparing Col. 4&: can reflect the efficiency of neuron re-in re-initialization Our results demonstrate the importance of neuron re-initialization in backdoor removal. This also confirms our claim in Section 4.2 that directly retraining the policy with the restored trigger may not fully eliminate the ground truth trigger when they are not very similar.
4.4 Attack Variations and Adaptive Attacks
We evaluate the robustness of BIRD against different attack variations and two potential adaptive attacks in the SpaceInvaders environment of the non-targeted perturbation-based attack.
Attack variations. Based on the default attack setup, where the trigger is a 3 x 3 square, and the state poisoning rate is 0.1, we vary the trigger shape (circle and triangle), size (5 x 5. x 7), and poisoning rate (0.2, 0.4). For each variation, we train five backdoored policies, apply BIRD to detect them from the five clean policies mentioned in Section 4.1 and report the detection performance in T The results show that BIRD maintains a high and stable detection performance when dealing with these variations. Table 3 also presents the backdoor removal performance of BIRD on these attacks, which is consistent with the detection result.
A ddaptie a We first consider straightforward adaptive by allowing the trigger to vary its shape and locations at different states. However, even after carefully tuning the training parameters, the attack was unable to achieve comparable performance as it is in the static setting. For SpaceIn- vaders, the agent's 's average reward in the poisoned environment is only reduced from 580 to 510 for non-targeted attack and (to 445 for targeted attack). This motivates future work to design more effective backdoor attacks with dynamic triggers. Nevertheless, even such an attack succeeds, as discussed in Section 3 BIRD is designed be robust against it.
Adaptive attack-2. We further consider an adaptive attack against our neuron re-initialization mechanism, a key design for backdoor removal. When training the backdoored policy, we add an additional regularization to constrain the (2 norm of the policy weights, which avoids neurons with ultra-high activation values. We train five backdoored policies with this attack, apply BIRD and record the backdoor removal performance. The policies' average reward in the clean environment before and after applying BIRD is 582 and 563, respectively, and the average reward in the poisoned environment is 170 (before) and 512 (after). These results demonstrate the robustness of BIRD against this adaptive attack (See Supplement S2 for more implementation details).
Additionally, we discuss generalizing BIRD to a setup where we do not assume accessing the agent's value networks, or more challenging scenario where the attacker intentionally attaches a benign value network instead of the backdoored one to make the attack more stealthy(See Supplement S3). Moreover, we also demonstrate the computational efficiency of BIRD and its insensitivity to subtle variations in key hyper-parametes (a, detection threshold e, and the number of reinitialized neurons L). We present these experiments in Supplement $4.
5 Conclusion and Future Works
We present BIRD, a method for detecting and removing backdoors in DRL policies. By analyzing the attack process and unique behaviors of backdoor agents, we formulate the trigger restoration as an optimization problem and design novel metric to detect backdoored policies. We also propose a novel finetuning method to remove the identified backdoor. Extensive experiments against various attacks demonstrate the effectiveness, efficiency, and generalizability of BIRD, well as its robustness to different attack variations.
This work points to several promising directions for future research. First, we evaluate two adaptive attacks in Section 4 Our future work will explore other possible attacks (e.g., using watermark triggers) and extend BIRD to defend against these attacks. Second, while our trigger restoration technique may not always yield a restored trigger that perfectly matches the ground-truth trigger, it is still effective in achieving decent detection and removal performance. We plan to refine our

---

trigger restoration technique to improve its restoration fidelity in future work. Third, in addition to the detection and unlearning mechanism utilized in BIRD, we will explore alternative solutions to render the backdoor ineffective. For example, we can filter out the trigger before feeding the state representation to the backdoored agent without finetuning it. Fourth, we acknowledge that besides the neuron re-initialization method discussed in Section 3.4 existing works also design other techniques to identify and remove the backdoor shortcuts (e.g., neuron pruning (27 53 58), [5 In future work, we will explore the effectiveness of integrating these techniques into our approach. Fifth, as discussed in Section 2 BIRD differs from other robust RL methods in terms of attack/problem setups. As part of future work, we will investigate generalizing these defenses designed for other attack setups to defend against backdoor attacks. Finally, while we evaluate BIRD in various game environments, we plan to further extend the backdoor attacks and defenses to broader types of environments, such as multi-agent competitive environments 29 and extensive-foms environments [23 1
Acknowledgements
This project was supported, in part by ARL Grant 1911NF-23-2-0137, IARPA TrojAI W911NF-19- S0012, NSF 1901242 and 1910300, ONR N000141712045, N000141410466 and N000141712947, We also thank the Center for Al Safety for their support of the Compute Cluster. Any opinions, findings, and conclusions in this paper are those of the authors only and do not necessarily reflect the views of our sponsors.
References
[1] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. arXiv preprint arXiv:1902.006 1002000000000 2019.
[2] Shubham Kumar Bharti, Xuezhou Zhang, Adish Singla, and Xiaojin Zhu. Provable defense against backdoor policies in reinforcement learning. In NeurIPS, 2022.
[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: arXiv:1606.00 (eei:0006014444 2016.
[4] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In S&P, 2017.
[5] Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In ICLR, 2019.
[6] Jinyin Chen, Xueke Wang, Yan Zhang, Haibin Zheng, Shanqing Yu, and Liang Bao. Agent manipulator: Stealthy strategy attacks on deep reinforcement learning. Applied Intelligence, 2022.
[7] Shuo Chen, Yue Qiu, and Jie Zhang. Backdoor attacks multiagent collaborative systems. arXiv preprint arXiv:2211.11455: 2022.
[8] Yanjiao Chen, Zhicong Zheng, and Xueluan Gong. Marnet: Backdoor attacks against coop- erative multi-agent reinforcement learning. IEEE Transactions on Dependable and Secure Computing, 2022.
[9] Edward Chou, Florian Tramer, and Giancarlo Pellegrino. Sentinet: Detecting localized universal attacks against deep learning systems. In IEEE Security and Privacy Workshops (SPW), 2020.
[10] Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, and Xia Hu. Towards interpreting and mitigating shortcut learning behavior of nlu models. arXiv preprini arXiv:2103.06992 2021.
[11] Min Du, Ruoxi Jia, and Dawn Song. Robust anomaly detection and backdoor attack detection via differential privacy. arXiv preprint arXiv:1911.07116. 2019.
[12] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI, 2018.
10

---

[13] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: defence against trojan attacks on deep neural networks. In ACSAC, 2019.
[14] lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hameessing adversar- ial examples. arXiv preprint vrXiv:1412.65722 2014.
[15] Junfeng Guo, Ang Li, and Cong Liu. Backdoor detection in reinforcement learning. arXiv preprint arXiv:2202.03669, 2022.
[16] Wenbo Guo, Lun Wang. Yan Xu, Xinyu Xing, Min Du, and Dawn Song. Towards inspecting and eliminating trojan backdoors in deep neural networks. In ICDM, 2020.
[17] Sandy Huang, Nicolas Papernot, lan Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv::102..000 022844 2017.
[18] Sham M Kakade. A natural policy gradient. In NeurIPS, 2001.
[19] Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adversarial training with langevin dynamics. NeurlPS, 33:8 1888127-1388 2020.
[20] Panagiota Kiourti, Kacper Wardega, Susmit Jha, and Wenchao Li. Trojdrl: Trojan attacks on deep reinforcement learning agents. arXiv preprint arXiv:1903.000 2019.
[21] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In NeurIPS, 1999.
[22] Aounon Kumar, Alexander Levine, and Soheil Feizi. Policy smoothing for provably robust reinforcement learning. In ICLR, 2022.
[23] Marc Lanctot, Edward Lockhart, Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. Openspiel: A framework for reinforcement learning in games. arXiv preprint 0rXhv:1908.09453 2019.
[24] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In NeurlPS, 2021.
[25] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural at- tention distillation: Erasing backdoor triggers from deep neural networks. arXiv preprint arXiv:2101.05930, 2021.
[26] Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning. arXiv preprint arXiv:2210.05927, 2022.
[27] Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks deep neural networks. In RAID, 2018
[28] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In CCS, 2019.
[29] Ryan Lowe, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiiee environments. In NeurlPS, 2017.
[30] Wanlun Ma, Derui Wang, Ruoxi Sun, Minhui Xue, Sheng Wen, and Yang Xiang. The" beatrix"resurrection Robust backdoor detection via gram matrices. arXiv preprint arXiv:2209.11715. 2022.
[31] Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Wei Weng. Robust deep reinforcement learning through adversarial loss. In NeuriPS, 2021.
[32] Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv: (X1i:17112.066 2017.
11

---

[33] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In ICML, 2017.
[34] Ximing Qiao, Yukun Yang, and Hai Li. Defending neural backdoors via generative distribution modeling. In NeurIPS, 2019.
[35] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML, 2018.
[36] Alessio Russo and Alexandre Proutiere. Optimal attacks on reinforcement learning policies. arXiv preprint arXiv: 13548, 2019.
[37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (XXiv:7777.0647 2017.
[38] Ali Shafahi, Mahyar Najibi, Zheng Xu, John Dickerson, Larry S Davis, and Tom Goldstein. Universal adversarial training. In AAAI, 2020.
[39] Guangyu Shen, Yingqi Liu, Guanhong Tao, Qiuling Xu, Zhuo Zhang, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Constrained optimization with dynamic bound-scaling for effective nlp backdoor defense. In ICML, 2022.
[40] Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl. arXiv preprint arXiv: 106.05087, 2021.
[41] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NeurIPS, 1999.
[42] Guanhong Tao, Guangyu Shen, Yingqi Liu, Shengwei An, Qiuling Xu, Shiqing Ma, Pan Li, and Xiangyu Zhang. Better trigger inversion optimization in backdoor scanning. In CVPR, 2022.
[43] J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Gym for multi-agent reinforcement learning. In NeurlPS, 2021.
al. Pettingzoo:
[44] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In ICML, 2019.
[45] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
[46] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In NeurlPS, 2018
[47] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Kuttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint 17Xiv:1708.04798 2017.
[48] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In S&P. 2019.
[49] Lun Wang, Zaynah Javed, Xian Wu, Wenbo Guo. Xinyu Xing, and Dawn Song. Backdoorl: Backdoor attack against competitive reinforcement learning. arXiv preprint arXiv:2105.00579 2021.
[50] Yue Wang, Esha Sarkar, Wenqing Li. Michail Maniatakos, and Saif Eddin Jabari. Stop-and-go go: Exploring backdoor attacks on deep reinforcement learning- based traffic congestion control systems. IEEE Transactions on Information Forensics and Security, 2021.
[51] Zhenting Wang, Hailun Ding, Juan Zhai, and Shiqing Ma. Training with more confidence: Mitigating injected and natural backdoors during training. In NeurIPS, 2022.
12

---

[52] Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma. Rethinking the reverse- engineering of trojan triggers. arXiv preprint arXiv:2210.15127, 2022.
[53] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurlPS, 2021.
[54] Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In ICLR, 2020.
[55] Fan Wu, Linyi Li, Zijian Huang, Yevgeniy Vorobeychik, Ding Zhao, and Bo Li. Crop: Certifying robust policies for reinforcement learning through functional smoothing. In ICLR, 2022.
[56] Fan Wu, Linyi Li, Chejian Xu, Huan Zhang, Bhavya Kailkhura, Krishnaram Kemthapadi, Ding Zhao, and Bo Li. Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks. arXiv preprint arXiv:2203.0883 2022.
[57] Junlin Wu and Yevgeniy Vorobeychik, Robust deep reinforcement learning through bootstrapped opportunistic curriculum. In ICML, 2022.
[58] Kaidi Xu, Sijia Liu, Pin- Yu Chen, Pu Zhao, and Xue Lin. Defending against backdoor attack on deep neural networks. arXiv preprint arXiv:2002. 12162, 2020.
[59] Zhaoyuan Yang, Naresh Iyer, Johan Reimann, and Nurali Virani. Design of intentional back- doors in sequential models. arXiv preprint urXiv:1902.099)) 2019.
[60] Jing Yu, Clement Gehring, Florian Schäfer, and Animashree Anandkumar. Robust reinforcement learning: A constrained game-theoretic approach. In LADC, 2021.
[61] Yi Zeng, Si Chen, Won Park, Z Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. arXiv preprint arXiv:2110.0373. 2021.
[62] Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101.08452, 2021.
[63] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho- Jui Hsich. Robust deep reinforcement learning against adversarial perturbations on state observations. In NeurIPS, 2020.
[64] Kaiqing Zhang, Tao Sun, Yunzhe Tao, Sahika Gene, Sunil Mallya, and Tamer Basar. Robust multi-agent reinforcement learning with model uncertainty. In NeurlPS, 2020.
[65] Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Robust policy gradient against strong data corruption. In ICML, 2021.
[66] Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruttion-obustt offline reinforce- ment learning. In AISTAT, 2022.
[67] Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. In ICML, 2020.
[68] Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. In ECCV, 2022.
13


Reviews:
[{"review_id": "FLsg5nnuxi", "reviewer": "Reviewer_V9nG", "summary": "This paper introduces BIRD (Backdoor Identification and Removal for DRL), a method for detecting and removing triggers in reinforcement learning models. In backdoor attacks, an attacker injects a trigger into the agent's environment during training, leading the agent to take backdoored actions that decrease its actual reward. BIRD addresses the challenge of detecting and removing backdoors from pretrained policies without knowledge of the attack specifications or access to the training process.\n\nBIRD formulates trigger restoration as an optimization problem, identifying the trigger by maximizing the agent's value function. It introduces a novel detection metric based on the actual reward difference before and after adding the restored trigger to the environment. The method effectively detects backdoored agents and employs finetuning with additional regularization terms to remove the backdoor while maintaining performance in the clean environment. Evaluations on various benchmarks demonstrate BIRD's superiority over existing methods, highlighting its generalizability, computational efficiency, and robustness against different attack variations. Overall, BIRD offers an effective solution for detecting and removing triggers in reinforcement learning models, mitigating the vulnerability to backdoor attacks.", "questions": "Does the main idea work efficiently in different scenarios? More experimental results are needed. ", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a groundbreaking method for defending against backdoored models in machine learning. The key innovation lies in exploring the total received reward as a means of detection. This unique approach sets it apart from previous works in the field.\n\nThe experimental results strongly support the efficacy of the proposed method, showcasing its ability to outperform or at least match the performance of existing techniques. The findings demonstrate that the idea of considering the total reward proves to be highly effective in mitigating the impact of backdoor attacks on machine learning models.\n\nOverall, the paper introduces a novel and promising approach to addressing the backdoor vulnerability in machine learning. The method's success, as evidenced by the experimental results, underscores its potential as a robust defense mechanism against backdoored models.", "weaknesses": "However, while the idea of considering the total received reward is novel and promising, there are potential weaknesses that need to be addressed. Firstly, it remains uncertain whether the proposed method's reliance on total reward as a detection mechanism is robust across all scenarios. In particularly noisy or challenging environments, where the model may struggle to learn effectively, it is unclear if the model will consistently receive significantly higher rewards even without the presence of a backdoor.\n\nAdditionally, it would be valuable to investigate the potential trade-off between detecting backdoored models and maintaining overall model performance. Since the method focuses on detecting triggers by maximizing the total reward, there is a possibility that it could inadvertently compromise the model's ability to achieve high performance in non-backdoored scenarios. Understanding the impact of this trade-off and ensuring a balance between detection and model performance would strengthen the applicability and practicality of the proposed method.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive and constructive review. Please see below for our response and clarifications. \n\n**First, the reviewer raised concerns regarding the effectiveness of using reward as the detection metric, particularly for challenging games where even a clean agent may struggle to receive a high reward.** \n\nWe thank the reviewer for raising this concern. We would like to kindly emphasize that our metric does not rely on the absolute value of the agent’s reward. Instead, we compute the reward difference before and after adding the restored trigger to the environment. To offset the influence of the absolute value, we normalize the reward difference (i.e., $\\phi(\\pi, \\Delta) = (\\bar{\\eta}(\\pi, \\Delta) - \\bar{\\eta}(\\pi)) / \\eta_{\\text{max}}$ in Line 210). A low metric $\\phi(\\pi, \\Delta)$ means that the agent’s performance drops after observing the trigger, indicating the agent may contain the backdoor. \n\nRegarding a challenging game, we can consider the following two scenarios.\n\n(1) We have a weak agent with the ability to receive only very low rewards. Consequently, it is unlikely to be selected as the attacker's target, as the agent is already quite weak. Attackers typically do not have an interest in attacking such weak agents, as they cannot inflict significant damage and would only result in wasted time and effort.\n\n(2) We have a sub-optimal agent that can still perform in the environment but cannot receive a very high reward due to the task's complexity. An attacker will have the incentive to attack such an agent with the goal of significantly reducing its reward. In this case, we will still observe a notable reward drop before and after adding the restored trigger to the environment. We admit that the reward drop will not be that significant compared to a well-trained/near-optimal agent. That is, the metric value will not be very close to -1. But our method can still have the potential to identify the backdoored agent from the clean agent by setting a less aggressive threshold. \n\nTo verify this, we conduct an extra experiment using the Atari-SpaceInvaders game. We first prepare $20$ agents. $5$ agents are well-trained clean agents,  $5$ agents are sub-optimal clean agents, $5$ agents are well-trained backdoored agents, $5$ agents are sub-optimal backdoored agents. For all the backdoored agents, we only use the successfully attacked ones. That is, their performance will drop to almost zero after observing the trigger. Otherwise, we treat it as an unsuccessful attack. We mix them together and then apply BIRD to detect the backdoored agents. We vary the detection threshold $\\epsilon=-0.5/-0.6/-0.7/-0.8/-0.9$ and report the detection performance. As we can see from Table 1 in the submitted pdf file, as the $\\epsilon$ increases, we are able to capture more and more sub-optimal backdoored agents. But we will not include any false positives. We thank the reviewer again for pointing this out. In the next version of the paper, we will clarify that we will not select an overly aggressive detection threshold in case there are some sub-optimal agents that are backdoored.\n\nIn addition, we acknowledge that, in general, it is tricky to determine the optimal detection metric. As such, we design our final removal step with the consideration that it will not minimize the effect on the agent’s performance in a clean environment. With that said, conservatively, we can apply the restoration and removal step to all the given agents. In this experiment, as shown in Table 2 in the submitted pdf file, after applying the removal step to all the 20 agents, we can observe that (1) For clean agents, their performance is only marginally affected; (2) For the backdoored agent, their performance in the clean environment keep similar, and their backdoor is removed. This shows that even our detection step fails to capture some backdoored agents. With a conservative solution, the removal step can still remove the backdoor for both optimal and sub-optimal agents. We will emphasize this in the next version.    \n\n**Second, the reviewer also raises the concern that whether our method will affect the agent’s performance in a clean environment.**  \n\nThank the reviewer for pointing this out. First, we want to clarify that our restoration and detection step does not vary with the given agent’s policy and thus will not affect its performance. Our removal step indeed may affect the agent’s performance in the clean environment. As mentioned in Section 3.4 Lines 257-264, we add an additional constraint to avoid this in our retraining objective function. As demonstrated in Table 1 in the paper, for all the selected games, BIRD only introduces negligible performance differences in the clean environment after the removal step. Our extra experiment in Table 2 in the submitted pdf file also demonstrates similar efficacy for suboptimal agents and clean agents. We will emphasize this in the next version. \n\n"}, {"review_id": "N93Re1FaDS", "reviewer": "Reviewer_Mdit", "summary": "This paper studies the backdoor defense problem in deep reinforcement learning (DRL) policies. Specifically, the authors proposed the BIRD method to address the limited generalizability and scalability of current practices. By analyzing the unique properties and behaviors of backdoor attacks, the authors formulated trigger restoration as an optimization problem and design a metric to detect backdoored policies. \nIn addition, a finetuning method is also presented to remove the backdoor, while maintaining the agent’s performance in a clean environment. Experiments are conducted over ten different single-agent or multi-agent environments. ", "questions": "Please see my comments above", "limitations": "Please see my comments above", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The NeurIPS community finds the topic of backdoor threats in Reinforcement Learning highly relevant.\n- The paper is well-written and easy to follow.\n- The empirical evaluation seems to be comprehensive.", "weaknesses": "- Can you please also report the performances in other popular metrics, e.g., ROC, in addition to the F1 score?\n- Can you create some specified baselines tailored to the RL setup? Since I think it might be unfair/inappropriate to apply those defenses for classification models to RL setup.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive and constructive review. Please see below for our response and clarifications. \n\n**First, the reviewer asked for an additional ROC curve for the results in Fig. 2.**\n\nThank the reviewer for the valuable comment. We follow this suggestion and draw the ROC curve in Fig. 1 in the submitted pdf file. We vary the detection threshold $\\epsilon$ from -1 to 1 and plot the ROC curve for BIRD in the selected games. Note that the comparison baselines have a fixed detection threshold. Given that their detection performance is low, we do not draw the ROC curves for NC and Pixel. The results are consistent with Fig. 2 in the main text. \n\n**Second, the reviewer asked for an additional baseline method that is suitable for RL setup.**\n\nThanks for the comment. We totally agree with the reviewer that adding such a baseline can better demonstrate the effectiveness of our method, as the current baselines are not designed for RL. To conduct such a baseline, we can adapt the current baseline to the RL setup. We consider the pixel method [1], which is stronger than NC. To adapt the pixel to RL, we partially leverage the design of our method. That is, we can still model the trigger restoration as solving Eqn.(1) in Lines 135-136. However, instead of modeling the $\\Delta$ as a generative process, we can use the pixel objective to model $\\Delta$. That is $\\Delta = clip(b_p, 0, 1) - clip(b_n, 0, 1)$, where $b_p$ and $b_n$ represents the positive and negative perturbation added to the state. As such, the final objective function for trigger restoration becomes \n\n$\\text{max}\\sum_s\\rho^{\\pi}(s) \\sum_{a} \\pi(s+\\Delta) Q_{\\pi} (s+\\Delta, \\pi(s+\\Delta)) $, where\n\n\n$\\Delta = clip(b_p, 0, 1) - clip(b_n, 0, 1)$. \n\nThis objective function can be resolved by the REINFORCE method. We denote this trigger restoration method as pixel-RL. After restoring the trigger, we can use two metrics for this method of detection – the original metric of the pixel and our proposed method denoted as pixel-RL-trigger-size and pixel-RL-reward.  \n\nWe compare our method with these two adapted baseline under six setups: the two single-agent Atari games: Seaquest and Qbert, with targeted and untargeted attacks, QMIX and You-Shall-Not-Pass in multi-agent games. For each environment, we train 5 clean and 5 backdoored agents. Fig. 2 in the submitted pdf file shows the comparisons of detection results. As we can observe from the figure, BIRD still outperforms both methods (pixel-RL-trigger-size and pixel-RL-reward). We will add this experiment to the next version.   \n\n[1] Better Trigger Inversion Optimization in Backdoor Scanning, CVPR 2022.\n"}, {"review_id": "zhKrmC1Jmn", "reviewer": "Reviewer_jZ8t", "summary": "This paper addresses the threat of backdoor attacks against deep reinforcement learning (DRL) policies. To tackle this problem, the authors propose BIRD, a novel generalizable backdoor detection and removal method for pretrained DRL policies in a clean environment without any knowledge of the attack specifications or access to the training process. They formulate trigger restoration as an optimization problem and introduce a novel metric for detecting backdoored policies. The authors also develop a fine-tuning method to remove the detected backdoor. The environmental results demonstrate the effectiveness and computational efficiency of BIRD, as well as its robustness against various backdoor attacks.", "questions": "In summary, I have few additional questions or concerns. As mentioned in the weaknesses section, there were some suggestions for improvement provided. I am willing to revise my assessment after further discussion.", "limitations": "The paper extensively discusses the future directions of their approach, exploring the potential applications, improvements, and extensions of BIRD. However, it would be beneficial to include more discussions regarding the limitations and practical significance of the proposed method. The potential impact of the work should be better highlighted, offering insights into its practical implications.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality: BIRD, is the first approach to detect and remove backdoors from a pre-trained DRL policy without requiring any prior knowledge. The authors demonstrate the novelty and advantages of each component of the algorithm, highlighting the strong practical value of the proposed technique in addressing backdoor attacks.\n2. Quality and clarity: The paper is well-written and organized, effectively conveying the authors' motivation and the details of the proposed technique. The main contribution lies in the development of a comprehensive defense strategy consisting of trigger restoration, backdoor detection, and removal. The authors provide clear and detailed insights into each part, along with the technical details. The experiments are well-designed, conducted rigorously, and include ablation studies to validate the efficacy of key design choices in BIRD.\n3. Significance: By providing an effective defense approach, the authors contribute valuable insights to adversarial RL defense against backdoor attacks. This work opens up possibilities for further research and development in addressing backdoor attacks, making it a notable contribution to the community.", "weaknesses": "1. Based on my understanding of this paper, the authors' approach focuses on the detection and removal of backdoor attacks rather than training a robust policy from scratch to defend against them. BIRD performs fine-tuning to remove the backdoor, but it does not directly improve the robustness of the RL algorithm itself. However, I do not have any issues with the authors' approach; I am simply highlighting this aspect.\n\n2. Regarding the presentation of experimental results, I noticed that the tables can sometimes be confusing. For instance, in Table 2, directly comparing the results of Qbert, COMA, and YSNP may lead to misunderstandings.\n\n3. I am curious to know whether the choice of different backdoor attack methods would also impact the effectiveness of BIRD. Could the authors provide a brief explanation of this question?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive and constructive review. Please see below for our response and clarifications. \n\n**The reviewer first questioned the presentation of Table 2.**\n\nWe thank the reviewer for the suggestion.  We agree with the reviewer that the results of Qbert, COMA, and YSNP can not be compared directly. In Table 2, we show the ablation study results of two key designs in our method: our proposed reward based detection metric and the neuron re-initialization strategy. Comparing Columns 2&3 and Columns 4&5 at each row can reflect the efficiency of these two designs. Due to the space limit, we combine the results of different games into one table. We will add more clarification to avoid the confusion of comparing the results of different games in the next version. \n\n**The reviewer also asked about BIRD’s effectiveness against different attacks.**\n\nWe thank the reviewer for this interesting question. We would like to kindly highlight that we already consider three different attacks in our work, each for one type of game (RL setup): single-agent games, multi-agent competitive games, and multi-agent cooperative games. To the best of our knowledge, there are no other existing backdoor attacks against multi-agent RL. As such, we found a more recent attack against the single-agent RL [1]. This attack considers a similar attack setup as TrojDRL (the one we evaluated) but with a different method of injecting backdoor.  \n\nWe conduct an extra experiment on the Atari-Breakout and Seaquest game. We train $5$ backdoored agents with this attack and mix them with another $5$ clean agents respectively for each game. We apply BIRD for backdoor detection and removal. The results shown in Table 3 in the submitted pdf file demonstrate the effectiveness of our method against this attack. We will add this experiment to the next version. \n\n[1] Agent Manipulator: Stealthy Strategy Attacks on Deep Reinforcement Learning. Applied Intelligence 2022.\n\n**Finally, the reviewer pointed out that BIRD performs fine-tuning to remove the backdoor, but it does not directly improve the robustness of the RL algorithm itself.**\n\nWe thank the reviewer for pointing this out. We will emphasize in the paper that our goal is to robustify a pre-trained agent against backdoor attacks (i.e., identify backdoored agents and remove the backdoor). We do not consider the setup that aims to improve the robustness of the policy training process to reduce the risk of being backdoored. \n"}, {"review_id": "nJitZj2FUi", "reviewer": "Reviewer_M2QE", "summary": "The paper addresses the challenge of detecting backdoored reinforcement learning policies. The injection of backdoors in RL policies was first studied in [19] and while there has been some work on detection of backdoored policies in [2] and [14] - these methods are limited to settings where the trigger is in the competing agent's actions [14] or limited to perturbation patches [2]. This paper formulates trigger restoration as an optimization problem and designs a novel metric to detect backdoored policies. ", "questions": "* Can you help understand a practical setting where one would necessarily have access to policy as well as the value function for the policy ? Isn't this a very major assumption and severely limits the use of the proposed method? It also makes the problem much more easier to solve. Even if a buyer forces that policy be accompanied by value function -the attacker could produce value function with the trigger removed. Why would an attacker provide a value function that has the trigger behavior embedded in it? \n\n* The part of different triggers having common shortcut is very interesting. But Fig 1 is not clear. What does \"high activation\" mean here? Neurons can have very different scales of activation, particularly across different layers - how do you decide that some activation is high and compare different neurons with each other? What does it mean to reinitialize weights for the highest value neurons - how does it correspond to breaking shortcuts (traditional definition of shortcut is that low layer neurons impacting very high layer neurons). \n\n", "limitations": "There are no concerns of negative broader impact. ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper attempts to solve an important problem on supply chain of machine learning models - detection of backdoored policies and elimination of triggers. ", "weaknesses": "* The formulation has one serious weakness - it assumes that we have access to value function in addition to the agent's policy network and the clean environment. The value function is \"poisoned\" and so, just optimizing against this value function allows one to discover triggers. This is unrealistic. In practice, the RL policies in the supply chain would be procured as a state to action map or sequence of state to a action - that is, one would just have access to the policy and not have access to the value function. \n\nThe entire approach is predicated on this availability of the value function because the trigger recover is framed as an optimization problem that maximizes the value function. So, this is the primary concern of the reviewer. \n\n* Some statements are inaccurate though they are not serious concerns and are more of presentation issue. For example, line 108/109, \"TO inject the backdoor, the attacker needs to manipulate .. reward function .. \" ... This is just one way to inject Trojans - for supervised learning, one can inject Trojans by directly manipulating the network and the same could be feasible for RL. So, it would be better to say, \"existing methods manipulate reward function for injecting backdoor..\"\n\n* The method for trojan elimination uses a number of hyperparams which appear to be critical for success of the method - definition of high activation for a neuron, selection of L top neurons, threshold epsilon_1, etc. In its current presentation, the approach looks rather hacky that might be made to work on 9 scenarios but will be difficult to generalize. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive review. Please see below for our response and clarifications. \n\n**The reviewer first questioned the assumption of accessing the target agent’s value function.**\n\nWe thank the reviewer for the valuable feedback. We would like to kindly point out that this assumption can be practical in many cases. Our assumption simulates a practical scenario that the agent user outsources the policy training process to a third party (model training service provider). The user can request to get both the policy network and value function from the service provider or only choose the provider that is willing to provide the value function. As for service providers, it does not introduce the extra cost of providing the value function to their customers as they will need to train it anyway.  For example, we checked the top 10 models that are most popular in **Hugging Face**, and we found that all of them included the value network.\n\nWe also agree with the reviewer that the attacker could intentionally attach a benign value network instead of the backdoored one to make the attack more stealthy, or we do not have access to the value network. We address this scenario and provide a corresponding solution in Supplement S3.4. In such cases, we restore the trigger by minimizing the actual return of the agent (i.e., change the objective function in Eqn. (1) to \n\n$\\min_\\Delta  \\sum_s\\rho^{\\pi}(s)\\sum_{a}\\pi(s+\\Delta)R(s+\\Delta, \\pi(s+\\Delta)))$, \n\nwhere $R$ is the actual reward function of the RL problem. The insight is for a backdoored agent, its actual return will drop when facing the triggered environment. \n\nFig. S4 in Supplement shows that our method can still outperform all baselines in backdoor detection. Our removal step is not affected as Eqn. (4) does not require the agent’s value function. It optimizes the agent’s total return under the actual reward. Table 4 in the submitted pdf file shows that with the trigger restored using this adaptive method, our backdoor removal is still effective (We select five setups from the nine attack scenarios). We will highlight this in the next version.  \n\n**The reviewer then questioned the sensitivity of our method to key hyper-parameters.**\n\nThank the reviewer for the valuable feedback. In Supplement S4, we systematically evaluate the sensitivity of BIRD to hyper-parameter changes, including the one pointed out by the reviewer – the number of neurons/convolutional kernels selected in each layer for resetting ($L$) in Table S8. The results indicate that our approach is insensitive to the variations in these hyper-parameters. \n\nIn addition, we conduct an additional experiment to evaluate the $\\epsilon_1$ in Eqn.(4). Table 5 in the submitted pdf file shows that our method is robust against the variation of this hyper-parameter. We will add this result in the next version. \n\n**The reviewer then asked for clarifications about our neuron reinitialization process and its connection to breaking shortcuts.**\n\nThank you for pointing this out. Sorry for the confusion. \n\nWe notice that the activation values of neurons at different layers are at different scales. As such, we sort neurons based on their activation value within each layer. Our policy networks involve two types of layers – the convolutional layer and the fully-connected layer. More specifically, as shown in Algorithm 3 in Supplement S1, during the backdoor removal, we first run a warm-up stage to collect a set of $E$ trajectories from the poisoned environment. For each state in these trajectories, we input it into the policy network and record the activation value for each kernel in the convolutional layer and each neuron in the feedforward layer. We then compute the activation mean of each kernel/neuron across all the states. \n\nGiven these mean activation values, we then select $L$ neurons/kernels from each layer in the policy network. Specifically, for each convolution layer, the output for each state is a 3-D tensor with the size of $[C, H, W]$. We first find the highest activation value in each channel (i.e., each $H\\times W$ 2-D activation matrix). Then, we rank these channel-wise highest activation values to select the top $L$ channels with the $L$ highest values. We reinitialize the weights and biases of the kernels corresponding to the selected channels. For each linear layer, we rank the activation value and select the top $L$ neurons. We reinitialize the weight and bias of these selected neurons. Regarding the reinitialization operation, we reset the weights and biases as zero. We are again sorry for the confusion due to these missing details. We will add them together with a more detailed description of Algorithm 3 in the next version. \n\nWe came up with this idea following [1][2]. As discussed in [1], when the trigger is presented in the input, the neurons/kernels with high activation values at each layer typically form the backdoor shortcuts. Resetting their weights and biases as zero can potentially remove the shortcuts and give the model flexibility in learning the correct actions under the poisoned inputs. In our method, we follow this idea to remove backdoor shortcuts and give the policy the opportunity to learn correct policies in the poisoned environment. \n\n\n[1] Training with More Confidence: Mitigating Injected and Natural Backdoors During Training, NeurIPS 2022.\n\n[2] Adversarial Neuron Pruning Purifies Backdoored Deep Models, NeurIPS 2021.\n"}]

Author rebuttal:
We thank the reviewers for the constructive feedback. We addressed all the comments. Below, we summarize our responses:

We have added all experiments mentioned by reviewers (All the results are in the submitted PDF):

1. We demonstrated the effectiveness of BIRD in detecting poisoned/backdoored agents with a sub-optimal performance by varying the detection threshold (Reviewer V9nG).

2. We demonstrated the effectiveness of BIRD’s backdoor removal for suboptimal agents (Reviewer V9nG).

3. We added the ROC curve for the detection results in Fig.2 in the main text (Reviewer Mdit).

4. We compared BIRD with two additional baselines that tailor existing methods for RL setup. We showed BIRD is still better than these two adaptive baselines (Reviewer Mdit).

5. We tested BIRD against a more recent backdoor attack and demonstrated its effectiveness (Reviewer jZ8t).

6. We demonstrated the BIRD can be adapted to a scenario where the value function is not available (Reviewer M2QE).

7. We showed that BIRD is not that sensitive to the changes in $\epsilon_1$ (Reviewer M2QE). 

We have clarified all the questions from reviewers:

**Reviewer V9nG**

1. We clarified that by selecting the proper detection threshold, our method can detect agents with a sub-optimal performance (in complicated games).

2. We demonstrated that our method has a marginal impact on the agent’s performance in the clean environment.

**Reviewer Mdit**

1. We followed the reviewer’s suggestion and added the ROC curves.

2. We followed the reviewer’s suggestion and added two additional baselines that are more suitable for the RL setup.

**Reviewer jZ8t**

1. We followed the reviewer’s suggestion and tested BIRD against another recent attack.

2. We clarified that our method focuses on robustifying a trained agent rather than designing a robust agent training algorithm.

**Reviewer M2QE**

1. We first clarified assuming the availability of value function is a reasonable and practical assumption. We further demonstrate that BIRD is still effective when the value network is not available.

2. We followed the reviewer’s suggestion and added an additional experiment that demonstrates the insensitivity of BIRD to $\epsilon_1$.

3. We clarified our method of identifying and reinitializing neurons during the backdoor removal. 

4. We followed the reviewer’s suggestion and update an imprecise description and provide a negative broader impact. 

We hope this summary can facilitate the reviewers' evaluation and discussion of our paper. We are at your disposal for any further questions. In addition, We would appreciate it if you could kindly consider updating your scores if our rebuttal has satisfactorily addressed the concerns. Thank you again for your time and consideration.


Meta-review: This paper proposes a novel method, BIRD, which allows for both identifying and removing backdoors from Deep RL policies.  The authors were able to effectively address reviewers’ concerns during the rebuttal phase, and performed the additional experiments suggested.  A primary novelty is that the algorithm only requires access to a clean environment, however, reviewer M2QE was originally concerned that the method required access to the value function as well.  However, the authors demonstrated that BIRD can be successfully applied even without such access.  After much discussion, and improvements, reviewers all recommend accepting this paper, and I am happy to recommend acceptance as well.
Decision: Accept (poster)