Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Overview: The paper introduces a causal mechanism that both creates and explains away noise interventions into observational data fed into RL agents. The authors propose a form of resilient agent, that based on training data containing labeled interventions, learns both Q function and the causal impact of interventions on the Q function. The model architecture consists of a intervention predictor and a split parametrization of the Q function estimation as a function of intervention as shown in L^{CIQ} presented as eq.3. Finally, the authors show the performance of their proposed method on 4 visual based RL agents with two types of interventions (attacks): namely adversarial and blackout against classical baselines such as DQN and DQN with safe actions.

Pros: 
- Clarity: Overall I find the paper well-written and reasonably easy to follow. The problem is well motivated and the related work relevant. 
- Significance/Impact: I think the problem that the paper is trying to solve is relevant and with potentially big impact
- Experimental design: I think the experiments section is relevant and makes a strong case for the method

Cons:
- Though clear at most times, the paper should spend more time explaining the basic causal terminology and the assumptions behind the causal graph introduced in Figure 2a. I find that the authors introduce the causal coneepts in an informal, intuitive way, but that should be followed-up by a clear formalism. 
- I find that the biggest downside of the method is that it needs to be trained with the type of invervention that the agent will be resilient to. It would be interesting to create an intervention detector that is fully unsupervised and that is based more on a state anomaly detection. 

Bordeline:
- Novelty: In terms of novelty, the paper is a relatively straight-forward application of do-calculus to Q-value learning.

Final comments:
	Overall i found the paper interesting and the approach relevant, supported by rigurous experimentation. On the downside, the novelty of the paper does not seem major and the predefined nature of interventions might make it unrealistic in a lot of RW scenarios.
review_title: Interesting paper and relevant approach, supported by rigorous experimentation. On the downside, the novelty of the paper does not seem major and the predefined nature of interventions might make it unrealistic in a lot of RW scenarios.

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################

Summary:

The paper presents a framework for deep reinforcement learning that is motivated by causal inference and with the central objective of being resilient to observational interferences. The key idea is to use interference labels in the training phase to learn a causal model including a hidden confounding state, and then use this model in the testing to make safer decisions and improve resilience. The authors also propose a new robustness measure, CLEVER-Q, which estimates a noise bound of an RL model below which the model's greedy decision would not be altered. The framework is tested extensively over multiple applications and under different types of observational interferences. The results show a clear advantage of the proposed framework over baseline RL methods in terms of resilience to interference.

##########################################################################

Reasons for Score:

The paper addresses an important problem in AI relating to the robustness of the algorithms and paradigms to noisy interference. The proposed framework appears to be sound and the experimental results show superior performance in comparison to other baseline RL methods. That being said, I am not very familiar with the literature on RL and Deep learning, so my decision is more of an educated guess. However, I do have a concern about the causal inference component of the paper (explained below) and this is reflected by the score.

##########################################################################

Cons:

The causal component of the proposed framework is not well-explained. More specifically, the causal graphical model in Figure 2 is introduced at the beginning of Subsection 3.3 very briefly, and the authors don't explain the intuition behind constructing this graph.
The authors say, "We use z_t to denote the latent state which can be viewed as a confounder in causal inference", but there is no explanation for why this makes sense. Is it just an assumption that happens to work?

Moreover, the following phrase appears to be inaccurate "knowing the interference labels it or not corresponds to different levels in Pearl’s causal hierarchy...: the intervention level with the interference labels and the association level without the information". Knowing vs not knowing the interference labels does not correspond to interventional vs associational levels in the causal hierarchy, but simply switching a variable/node in the CGM between observed and latent/unobserved. Such knowledge of a variable in the CGM does not account for an intervention.

##########################################################################

Questions during the rebuttal period:

It would help if the authors can clarify the issue raised in the "Cons" above regarding the clarity of the causal component and its central role, as claimed, in the proposed framework.

##########################################################################

Typos:

- p.1, "the RL agent is asked to learn a binary causation label and *embedded* a latent state into its model": embedded -> embed.
- p.3, "design an end-to-end structure ... and *evaluated* by treatment effects on rewards": The statement does not parse.
-p.3, "where M is a *fix* number for the history": fix -> fixed.
-p.3, "We assume that interference labels i_t *follows* an i.i.d. Bernoulli process": follows -> follow.

##########################################################################

Comments after Discussion:

I appreciate the effort made by the authors to elaborate on the causal formulation behind CIQ. However, the additional discussions in the paper are still confusing and raise soundness concerns. Some of the issues are discussed below.

1- Rubin's Causal Model: The authors reference RCM in Subsection 3.1 for the causal formulation yet the rest of the work does not seem to use the potential outcome notation. Instead, Subsection 3.3 uses graphical models and the do-operator which follows the causal framework by Pearl. Then, in Subsection 3.4, the authors go back to reference RCM. It is not clear why this alternation between the two approaches is employed.

2- If $z_t$ is defined as a function of $x_t$ and $i_t$, shouldn't the CGM reflect that with an arrow from $i_t$ to $z_t$ instead of it being the other way around in Fig.2(a)? Despite the attempt by the authors to elaborate on the causal formulation, I'm unable to map the structural equations such as Eq. (1) and the function of $z_t$ to the given CGM in Fig.2(a).

3- The discussion in Subsection 3.3 leading to Eq. (3) sounds flawed to me. Quoting the authors, "the interference model of Eq. (1) can be viewed as the intervention logic with the interference label it being the treatment information". This statement is elaborating on the formulation of $x_t'$ where $x_t$ is intervened on and replaced by an interfered state when $i_t=1$. Alternatively, $x_t$ is kept intact when $i_t=0$. This intervention on the mechanism of $x_t$ happens whether we obtain $i_t$ and train the DQN with it or not. In this sense, the intervention is not happening under the CIQ framework only, but also when we simply train based on $x_t'$. Accordingly, it is not clear to me how "the learning problem is elevated to Level II of the causal hierarchy" due to the presence of the interference labels. To be clear, I'm not questioning the significance of using the interference labels in the training, but rather the causal story and formulation behind CIQ.
review_title: Interesting framework with promising experimental results

final_score: rating: 7: Good paper, accept
review_content: Paper summary:
The paper makes two main contribution: 1). A metric for evaluating RL agent's relience 2). A framework that uses inteference type as label to enhance relience.

Reasons for score:
Overall, I am towards accepting the paper but I believe there are much improvements to make. The idea of adding interference type as label is quite novel, and the authors provide extensive experiment results to show that CIQ achieves better resilience against interferences. I believe that the proposed framework has the potential to enhance a single agent with multiple types of attcks and I suggest the authors to look into this direction.

Comments:
- The proposed network architecture does not seem to differ from a normal DQN other than an additional interference type output. As such, I am not very convinced by the causal inference insight.
- Is it possible to have multiple interference type during training? The resulting agent would then have the potential to be resilient to all types of interferences during evaluation. Furthermore, it would then be possible to combine different types of adversarial attacks into adversarial training. For real life application, I believe that it is possible that different types of interference could happen simultaneously.
- Computation cost of CLEVER-Q is expensive, and there is no guarantee of the estimated CLEVER-Q score. Also, the advantage of CLEVER-Q over AC-rate is not discussed.


Minor issues:
- Appendix C.1, this is just huber loss instead of quantile huber loss.
- Appendix D, 'Refer to We'.


Questions:
1. If I understand correctly, the difference between DQN-CF and CIQ is just that the inteference loss does not propogate to the Q-network parameters in DQN-CF?
2. Depending on whether there are interference, how different are the outputs of f_2 and f_3? If f_2 output is similar to f_3 even when there is no interference, it would suggest that we only need f_2 and the switching mechanism can be removed.

Post rebuttal:
The authors have addressed most of my main concerns and provided detailed experiment results, hence I increase my score to 7.
review_title: Novel framework but much to do

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a method, the Causal Inference Q-Network (CIQ), for training deep RL agents that are robust to abrupt interferences in observations, such as frame blackouts, Gaussian noise or adversarial perturbations. During training time, a binary interference label is provided to the agent at each time step indicating whether an interference has been applied to the observation; the interference label acts as a switch between two neural networks that process the observation to predict the Q-values. The CIQ agent learns to predict the interference label in a supervised fashion, and at test time it uses the predicted label to switch between networks, rather than the true label. The CIQ agent is shown to learn faster and more effectively when compared to a number of baselines on a selection of OpenAI Gym tasks that are modified to include various types of observational interferences.

While this paper presents a method that is shown to perform well empirically in the setting is aimed to tackle, I cannot recommend it for acceptance because (i) almost no motivation or intuition is given for the architectural choices that seem to be key to the performance of the agent (in particular the switching mechanism between Q-networks), and (ii) the characterisation of the agent as performing causal inference, which is the key message of the paper, is confusing in a number of ways. As a result, it’s not clear what we are supposed to learn conceptually from the paper and how it can guide future research.

Positives:
	•	In order to be able to apply deep RL agents in real-world settings, they need to be robust to noisy observations, and so the paper is tackling an important problem. 
	•	The chosen baselines seem fairly chosen, such as the safe-action DQN and the DQN with concatenated interference label, and the agents are evaluated across a variety of different interference types and levels. A number of different metrics are used to evaluate the performance of each agent. 
	•	The CIQ agent performs the best when compared to the baselines in the interfered observation setting. 

Concerns:
	•	There is little to no motivation given for the architectural choice of having the interference label modulate a switching mechanism between Q-networks, which based on the comparison to baselines and the ablation studies in Appendix E.3 seem to be key to the agent’s performance. The only justifying claim I could identify was the following: “Such a switching mechanism prevents our network from over-generalizing the causal inference state.” Could the authors clarify what exactly they mean by over-generalizing the causal inference state and how can this be inferred from the empirical results? 
	•	Both the CIQ and DQN-CF agent learn to predict the interference probability and use it for predicting Q-values, with the difference that the CIQ uses this predicted probability to switch between two Q-networks, but DQN-CF performs significantly worse on average - what is the intuition for this? 
	•	The causal graph drawn in Figure 2a seems to contradict the experimental setup described in the text: 
	◦	(i) The interference label i_t is stated to be sampled from a Bernoulli process but in the causal graph drawn in Figure 2a, there is an arrow from the latent state z_t towards i_t; in what way does the latent state affect the probability of interference? 
	◦	(ii) In the graph, there is no arrow (and no indirect path) from the interference label to the observed state x’_t; this implies that intervening on the label, which according to Equation 1 directly affects the value of x’_t, would not actually affect x’_t. This is directly contradictory. 
	◦	(iii) In the graph, there is an arrow from i_t to r_t but there is no mention in the paper for how the interference label can directly affect the reward in a given time step. 
It’s possible I have badly misunderstood the translation of the setup into the causal graph provided - in which case, could the authors please explain where I have gone wrong? 
	•	Section 3.3 describes a toy example that is used to show how being provided interference labels during training can lead to better sample efficiency in learning the reward distributions for each of two states (one of which is occasionally subject to interference that switches the observation to the other state). In this example, however, it is not possible to infer the latent state from the potentially interfered observation at test time, while in the DQN agents the advantage of the CIQ agent is stated to come from the fact that it can do just that. It’s not clear here whether the message is that the CIQ agent performs better due to better sample efficiency or due to its ability to infer the latent state at test time. 

Other questions / comments
	•	 As far as I understood, the interference at test time always corresponds to the interference provided during training in the experiments. Were any experiments run to test generalisation to unseen types of interferences at test time? In a real world setting, this could be useful.
review_title: Good experimental results but approach lacking in motivation and explanation

Meta-review: This paper presents a deep reinforcement learning method that aims at ensuring resilience to observational interference. During training labels that indicate presence or absence of interference are available to the algorithm. The training objective is augmented to learn the prediction of interference that is used at test time to infer the interference label. The experimental results show superior performance in comparison to other baseline RL methods.

The main objection raised by the reviewers was on the confusing and possibly unsound causal formulation. The authors' clarifications during the discussion did not eliminate bur rather exacerbated the reviewers' doubts. I read the paper in full myself to understand whether the reviewers' confusion was justified, and whether it could be easily resolved by an improved explanation, or it is a more serious issue.  I did not succeed in clearly understanding the causal formulation nor its relevance, and also have  soundness concerns. Figure 2a does not seem to be a correct explanation of the causal mechanism. It is also not clear from this figure why z is called confounder. More generally, I was not able to reach a coherent and sound causal formulation from the authors' explanation. My conclusion is that the framing of the paper as causal inference based is not well justified. 
Decision: Reject