Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Trustworthy AI is of interest to many as systems are quickly being deployed in the real world. There is a chance that the models being deployed may contain a backdoor policy which acts maliciously when triggered by an interested attacker who may perturb the agents observations. This work attempts to clearly define the objective of such an attacker, describes the limits of their behavior and attempts to defend against such an attack. It introduces the concept of a safe subspace based on testing the provided policy on a clean environment and then uses this empirically derived subspace to sanitize the policy (akin to a 'denoising' step). Bounds on performance of this sanitized policy w.r.t. the optimal policy are given.
strengths_and_weaknesses: Strengths:
+ Intuitive examples (Fig.2) explaining role of backdoor attacker
+ Bounds provided for sanitized policy behavior in clean environment
+ Tackles question of choice when subspace dimension not known (Sec. 5.4)

Weaknesses:
- Experiments on discrete action-spaces, does this generalize to Mujoco environments and robotic problems?
- Analysis is on zero mean state case and claims to generalize to the non-zero mean case, but this is not substantiated. (in my view)
- Assumptions on attacker might not be valid in many cases (limiting to $E^{\perp}$)
questions: 1. Is it right to say that to get the safe subspace, the agent must sample episodes in an environment free of the trigger? (Pg. 4 :L120)
2. To get the sanitation policy, $n$ clean episodes are sampled and 1 state is sampled from each episode. How is this more helpful than choosing correlated samples (say averaging over $n$ entire episodes) ? (Pg 8 :L211)
3. To get a good value of $d$ when it is unknown to the defender, tests will be need to run on the triggered environment (in the presence of the attacker). The experiments in Fig. 5 (Sec 5.4) seem to be testing using a fixed $n=40$. Why is this so low when Breakout clearly needs a large number of samples to approximate the safe subspace (Fig. 4)?
4. How are the backdoor policies trained and the trigger step decided? Is the trigger action hardcoded into the policy? (Sec 5.1, 5.2) 
5. Possible Typos?

    Pg 2
    L46 (which uses autoencoder)
    L51 (have also studies)

    Appendix pg 14 L369 ($s_t$ should be $s$)
limitations: Point on attacker needing to limit in $E^{\perp}$ space (may not be true in practice) and need for clean environment interactions to get sanitization policy is mentioned. It is not certain that this will generalize to continuous action spaces.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a defense algorithm against backdoor attacks for reinforcement learning. Specifically, the defense is based on Singular Value Decomposition of the covariance matrix of state occupancy. The algorithm sanitizes the poisoned policy by projecting states into a ``safe subspace'' formed by a low-rank decomposition of the covariance matrix. Based on several assumptions on the state occupancy structure, the authors show that the proposed algorithm guarantees a bounded different between the optimal policy's value and the sanitized policy's value. Experiments on a vector-input game and a pixel-input game verify the effectiveness of the proposed algorithm. 
strengths_and_weaknesses: Strengths:

The tackled problem, defending against backdoor policy attack, is important. The proposed algorithm applies SVD to filter out triggers, which makes intuitive sense. 

The experiment results are good. It is surprising that the proposed SVD-based defense also works for pixel inputs.

The presentation of this paper is good, and easy to follow. The authors make the attack and defense problem clear, with nice visualization figures. The mathematical notations are also clean to me. 

---

Weaknesses:

The algorithm, although makes intuitive sense, may have some difficulties of scaling up. It is nice that the authors provide hyperparameter check for d in Figure 5, and show that d can be selected based on the eigen gap. But the experiment is only done in one pixel-input task Breakout (which is a relatively simple game), making the results less convincing. More experimental results would be appreciated. 

The explanation of the sample complexity is too simplified. How would the depencence on $d, B, L_E, K, \Sigma$ influence the complexity? They seem to reveal the intrinsic hardness of defending in the enviornment.

questions: 1. What is the $K$ in the sample complexity result?

2. What is the computational complexity of the algorithm in theory and in experiments? 
limitations: The paper briefly discusses the limitations of the work in Section 6.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a defense method to deal with the backdoor attack on reinforcement learning (RL). Instead of re-training the RL policy or directly modifying the model parameters, the proposed method serves as a wrapper that projects the observation onto an estimated safe subspace and thus, eliminates the potential triggers. This helps the RL agent retain its performance even in triggered environments. The authors also provide theoretical analysis for their method.
strengths_and_weaknesses: Strength:
1. Originality: the idea of this paper is novel and the proposed method is a promissing exploration for the defense mechanism against backdoor attack on RL.

2. Clarity: the paper is organized well and the paper is easy to follow.

Weaknesses:
1. Quality: the results of this paper are sound and well supported by the conducted experiments. However, the method seems to rely on a strong assumption (please see detailed discussion in Question section). Also, the method is only tested in two Atari game, which is kind of limited.

2. Significance: the experiments prove the method can work in some scenarios. However, there are no comparisons with any existing works. Therefore, it is unclear how much this method advances the state of the art.
questions: 1. Strong assumption: this paper assumes that the adversary can only inject a trigger in the $E^{\bot}$ subspace of the state space and the proposed sanitization algorithm heavily relies on this assumption. In my personal opinion, this assumption is strong because the $E^{\bot}$ subspace being a convenient place to inject the trigger does not mean the adversary has to do that. Actually, even the adversary injects the trigger in the clean support, it may not affect the performance when the trigger is not present. We can use the Figure 2 as an example. When no trigger is present, the agent will turn left to state E, which is the optimal policy. Then if the adversary injects a trigger in state F and makes the agent think it is in state G (still in clean support), the agent will choose the right direction, which destroys the agent's performance. In this case, the sanitization algorithm may not work because state G is not in the $E^{\bot}$ subspace. In practical, I think it is very hard for the adversary to always inject a trigger in the $E^{\bot}$ subspace and there is also no incentive for them to do that. The adversary only cares whether the backdoor is insert into the victim model or not. For some domains, the successful backdoor may happen to have the trigger patterns that locate in the $E^{\bot}$ subspace. For other domains, the trigger patterns locating in the clean support may already be good enough. Therefore, the proposed method seems to only work for the cases where the triggers are only injected in the $E^{\bot}$ subspace. In addition, I am not sure whether the assumption 2 is practical or not. It seems to be an assumption solely for the theoretical analysis. It would be better if the authors can provide some intuitive explanations/examples to justify it.

2. The comparison with existing works: the experiments do not compare the proposed method with existing works. I agree with the authors that the backdoor defense for RL is different from that for supervised learning because the damage for RL is inflicted via the long term return instead of immediate rewards. However, in the sanitization algorithm, the authors just collect a set of states being visited, perform SVD on the state set, and then directly use the estimated projection operator to sanitize the observation. In the whole process, I do not see the authors explicitly handle the effect of long term return. Therefore, I think it is worthy to compare with the existing works for supervised learning that also pre-process the observations before feeding them into the trained model. For example, the work [12] that uses autoencoder could be a good candidate.
limitations: Please check the Questions section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper formulate backdoor policy attack. It specifies a kind of backdoor policy attack and provides an algorithm to defense such attack. It provide a proof to bound the suboptimality of the policy sanitized by the defense. It also conducts experiments to verify their algorithm.
strengths_and_weaknesses: Strength: Backdoor attack is an interesting problem, and the formulation in this paper is useful for theoretical study and algorithm design. They also provide an algorithm and prove the soundness of the algorithm both theoretical and empirixal.

Weakness: The writing of proof for the main theorem can be improved. Moreover, their algorithm and analysis can only  be used on the simple backdoor attack that consist with their assumption.
questions: 1. The trigger function used in the experiment seems really simple, since the basis of the image of the trigger function is e_i, which greatly reduce the learning difficulty.

limitations: Their formulation only adapts to the case that the support of the image the trigger function is a linear subspace of $R^d$, which might be restricted for real-life application.
review_title: null

Meta-review: metareview: The authors present a novel algorithm for defending against backdoor policies in Reinforcement Learning (RL). 
The main idea is to project observations onto a "safe" subspace which cleans out the backdoor. 

The authors present both empirical finds and theoretical results for their method. 
There was an active discussion between reviewers and authors in which the main concerns were addressed. 

I recommend that the authors follow the suggestion of reviewer iqtz and emphasise the restriction on the attacker ability more clearly to avoid any over-claims about the capability of the defense method.
confidence: Certain
Decision: Accept