Causal Inference Q-Network: Toward Resilient Reinforcement Learning
Authors: Chao-Han Huck Yang
Danny I-Te Hung
Yi Ouyang
Pin-Yu Chen
Venue:  (2021)
Keywords: Deep Reinforcement Learning
Causal Inference
Robust Reinforcement Learning
Adversarial Robustness

Abstract: Deep reinforcement learning (DRL) has demonstrated impressive performance in various gaming simulators and real-world applications. In practice, however, a DRL agent may receive faulty observation by abrupt interferences such as black-out, frozen-screen, and adversarial perturbation. How to design a resilient DRL algorithm against these rare but mission-critical and safety-crucial scenarios is an important yet challenging task. In this paper, we consider a resilient DRL framework with observational interferences. Under this framework, we discuss the importance of the causal relation and propose a causal inference based DRL algorithm called causal inference Q-network (CIQ). We evaluate the performance of CIQ in several benchmark DRL environments with different types of interferences. Our experimental results show that the proposed CIQ method could achieve higher performance and more resilience against observational interferences.

Under review as a conference paper at ICLR 2021
CAUSAL INFERENCE - Q-NETWORK: TOWARD RESILIENT REINETRCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep reinforcement learning (DRL) has demonstrated impressive performance in various gaming simulators and real-world applications. In practice, however, a DRL agent may receive faulty observation by abrupt interferences such as black- out, frozen-screen, and adversarial perturbation. How to design a resilient DRL algorithm against these rare but mission-critice and safety-crucial scenarios is an important yet challenging task. In this paper, we consider a resilient DRL framework with observational interferences. Under this framework, we discuss the importance of the causal relation and propose causal inference based DRL, algorithm called causal inference Q-network (CIQ). We evaluate the performance of CIQ in several benchmark DRL environments with different types of interferences. Our experimental results show that the proposed CIQ method could achieve higher performance and more resilience against observational interferences.
INTRODUCTION
Deep reinforcement learning (DRL) methods have shown enhanced performance, gained widespread
applications (Mnih et al., 20155 2016 Ecoffet et al. 2019; Silver al. 2017 Mao et al. (2017), and improved robot learning (Gu et al. 2017) in navigation systems TTai et al. 2017 Nagabandi et al. 2018). However, most successful demonstrations of these DRL methods are usually trained and deployed under well-controlled situations. In contrast, real-world use cases often encounter inevitable observational uncertainty (Grigorescu et al., 2020 Hafner al., 2018 Moreno al.) 2018) from an external attacker (Huang al. (017) or noisy sensor (Fortunato et al. 2018 Lee et al. [20 8). For examples, playing online video games may experience sudden black-outs or frame-skippings due to network instabilities, and driving the road may encounter temporary blindness when facing the sun. Such an abrupt interference on the observation could cause serious issues for DRL algorithms. Unlike other machine learning tasks that involve only single mission a time (e.g., image classification), an RL agent has to deal with a dynamic (Schmidhuber) 1992) and encoded state (Schmidhuber 1991] Kaelbling et al., 1998) and anticipate future rewards. Therefore, DRL-based systems are likely to propagate and even enlarge risks (e.g., delay and noisy pulsed-signals on sensor-fusion (Yurtsever al. 2020 Johansen al. 2015) induced from the uncertain interference.
In this paper, we investigate the resilience ability of an RL agent to withstand unforeseen, rare, adversarial and potentially catastrophic interferences, and recover and adapt by improving itself in reaction to these events. We consider resilient RL framework with observational interferences. At each time, the agent's 's observation is subjected to type of sudden interference at a predefined possibility. Whether or not an observation has interfered is referred to as the interference label.
Specifically, to train a resilient agent, we provide the agent with the interference labels during training. For instance, the labels could be derived from some uncertain noise generators recording whether the agent observes an intervened state at the moment as a binary causation label. By applying the labels as intervention into the environment, the RL agent is asked to learn a binary causation label and embed a latent state into its model. However, when the trained agent is deployed in the field (i.c., the testing phase), the agent only receives the interfered observations but is agnostic to interference labels and needs to act resiliently against the interference.
For RL agent to be resilient against interference, the agent needs diagnose observations to make the correct inference about the reward information. To achieve this, the RL agent has to reason about what leads to desired rewards despite the irrelevant intermittent interference. To equip an RI.

---

Under review as a conference paper at ICLR 2021
Training
Tosting
Navigation
Runvard
Interdered Slaske
Interfendd Label (7)
Intertrend Stanle
Predicted
Bus2d Gaminn Tioes
05 Unlt 3D Banana Collagor
Agant
Latent State (b)
Action
Action
Wi
Raward
Casool briwwe
Causcal Infermnoo Aguesi
(A) Model Framswork
(0) Lundrtando
Figure Frameworks of: (a) the proposed causal inference Q-network (CIQ) training and test framework, where the latent state is an unobserved (hidden) confounder; (b) a 3D navigation task, banana collector (Juliani et al. 2018), and (c) a video game, LunarLander (Brockman et al. 2016).
agent with this reasoning capability, we exploit the causal inference framework. Intuitively, causal inference model for observation interference uses an unobserved confounder (Pearl) 2009 2001 19956 Saunders al., 2018 Bareinboim et al. 2013) capture the effect of the interference on the reward collected from the environment.
When such a confounder is available, the RL agent can focus on the confounder for relevant reward information and make the best decision. As illustrated in Figure | we propose a causal inference based DRL algorithm termed causal inference Q-network (CIQ). During training, when the interference labels are available, the CIQ agent will implicitly learn a causal inference model by embedding the confounder into a latent state. At the same time, the CIQ agent will also train a Q-network on the latent state for decision making. Then at testing, the CIQ agent will make use of the learned model to estimate the confounding latent state and the interference label. The history of latent states is combined into causal inference state, which captures the relevant information for the Q-network to collect rewards in the environment despite of the observational interference.
In this paper, evaluate the performance of our method in four environments: 1) Cartpole-vO the continuous control environment (Brockman et al. 2016); 2) the 3D graphical Banana Collector (Juliani ett al. 2018)); 3) an Atari environment LunarLander--V vn (Brockman al.) 2016), and 4) pixel Cartpole visual learning from the pixel inputs of Cartpole For each of the environments, we consider four types of interference: (a) black-out, (b) Gaussian noise, (c) frozen screen, and (d) adversarial attack.
In the testing phase mimicking the practical scenario that the agent may have interfered observations but is unaware of the true interference labels (i.e., happens or not), the results show that our CIQ method can perform better and more resilience against all the four types of interference. Furthermore, to benchmark the level of resilience of different RL models, we propose a new robustness measure, called CLEVER-Q Q. to evaluate the robustness of Q-network based RL algorithms. The idea is to compute a lower bound on the observation noise level such that the greedy action from the Q-network will remain the same against any noise below the lower bound. According to this robustness analysis, our CIQ algorithm indeed achieves higher CLEVER-Q scores compared with the baseline methods.
The main contributions of this paper include 1) framework to evaluate the resilience of DRL, methods under abrupt observational interferences; 2) the proposed CIQ architecture and algorithm towards training resilient DRL agent, and 3) an extreme- eatreme-value theory based robustness metric (CLEVER-Q) for quantifying the resilience of Q-network based RL algorithms.
2 RELATED WORKS
Causal Inference for Reinforcement Learning: Causal inference (Greenland et al. 1999 Pearl
2009 Pearl et al. 2016 Pearl 2019 Robins et al. 1995) has been used to empower the learning
process under noisy observation and have better interpretability on deep learning models (Shalit et al.
2017 Louizos al. 2017), also with efforts (Jaber et al. [2199] Forney al. 2017: Bareinboim ct al. 2013) on causal online learning and bandit methods. Defining causation and applying causal
inference framework to DRL still remains relatively unexplored. Recent works (Lu et al. 2018
2

---

Under review as a conference paper at ICLR 2021
Tennenholtz et al.) 2019) study this problem by defining action one kind of intervention and calculating the treatment effects on the action. In contrast, we introduce causation into DRL by applying extra noisy and uncertain inventions. Different from the aforementioned approaches, leverage the causal effect of observational interferences on states, and design an end-to-end structure for learning causal-observationns representation evaluating treatment effects rewards.
Adversarial Perturbation: An intensifying challenge against deep neural network based systems adversarial perturbation for making incorrect decisions. Many gradient-based noise-generating methods (Goodiellow et al., 2013; Huang et al. 2017) have been conducted for misclassification and mislead an agent's 's output action. As an example of using DRL model playing Atari games, an adversarial attacker (Lin et al. 2017 Yang et al. 2020) could jam in timely and barely detectable noise to maximize the prediction loss of Q-network and cause massively degraded performance.
Partially Observable Markov Decision Processes (POMDPs): Our resilient RL. framework can be viewed as a POMDP with interfered observations. Belief-state methods are available for simple POMDP problems (e.g.. plan graph and the tiger problem (Kaelbling et al., 1998)). but no provably efficient algorithm available for general POMDP settings (Papadimitriou & Tsitsiklis 1987| Gregor et al. 2018). Recently, [gl et al. (2018) have proposed DRL approach for POMDPs by combining
variational autoencoder and policy-based learning, but this kind of methods do not consider the interference labels available during training in resilient RL framework.
Safe Reinforcement Learning: Safe reinforcement learning (SRL) (Garcia & Fernández, 2012)
seeks learn a policy that maximizes the expected return, while satisfying specific safety constraints. Previous approaches to SRL include reward- shapiin (Saunders et al. 2018). noisy training (Fortunato) 2018), shielding-based SRL (Aishiekh et al., 2018), and policy optimization with confident lower-bound constraints (Thomas et al.) 2015). However, finding these policies in the first place could need to reset the model each time and be computationally challenging. Our proposed resilient RL framework can be viewed as an approach to achieve SRL (Alshiekh et al. (2018), but we focus on gaining resilience against abrupt observation interferences. Another key difference between our framework and other SRL schemes is the novelty in proactively using available interference labels during training, which allows our agent to learn a causal inference model make safer decisions.
RESILIENT REINFORCEMENT LEARNING
In this section, we formally introduce our resilient RL framework and provide an extreme-value theory based metric called CLEVER-Q for measuring the robustness of DQN-based methods.
We consider sequential decision-making problem where an agent interacts with an environment. At each time t, the agent gets an observation Zt, e.g. frame in video environment. As in many RL domains (e.g., Atari games), we view s; (II-M+1,... 2x) to be the state of the environment where M is a fixed number for the history of observations. Given a stochastic policy π, the agent chooses an action a ~ (8t) from a discrete action space based on the observed state and receives a reward rt from the environment. For policy æ, define the Q-function Q" a) = EE| E 7/re|so 8, ag a, π] where y E (0, 1) the discount factor. The agent's 's goal is to find the optimal policy л that achieves the optimal Q-function given by Q* (s(saa) max, Q (s)(sa).
3.1 RESILIENCE BASE ON AN INTERVENTIONAL PERSPECTIVE
We consider resilient RL framework where the observations are subject to interference (as illustrated in Fig (a)) as an empirical process in Rubin's 's Causal Model (RCM) (Kennedy) 2016: Holland 1988 Balke & Pearl 1997 Robins et al. 2003) for causal inference. Given type of interference I.
the agent's s observation becomes:
4/= Fd (x,, in)
I(xx) + i xx.
where i, E (0, 1} is the label indicating whether the observation is interfered at time or not (under the potential outcome estimation (Rubin) 1974h), and I(x;) is the interfered observation.
The interfered state is then given by sy (a'- & follow an i.i.d. Bernoulli process with a fixed interference probability pf as a noise level. For example, when ytt equals to 10%, each observational state has 10% chance to be intervened under
x(). We assume that interference labels
3

---

Under review as a conference paper at ICLR 2021
SF
it
( .))
2r, Zo
1/
F2
XL, Xir..
sample A
pMi.). p(1-
noursl
mocel outputs
multpiication
Figure 2: (a) Causal graphical model (CGM). (b) CIQ architecture. The notation 1r denotes the inference label available during training, whereas i, is sampled during inference is unknown.
perturbation. The agent now needs to choose its actions T(({) based on the interfered state. The resilient RL objective for the agent is to find policy x to maximize rewards in this environment under observation interference.
In this work, we consider four types of interference as described below.
Gaussian Noise. Gaussian noise or white noise is common interference to sensory data (Osband ct al. 2019 Yurtsever et al. 2020). The interfered observation becomes [(x,) = 2z + ny with Gaussian noise ny. The noise variance is set be the variance of all recorded states.
Adversarial Observation. Following the standard adversarial RL attack setting, we use the fast gradient sign method (FGSM) (Szegedy et al., 2014) to generate adversarial patterns against the DQN prediction loss (Huang et al. 2017). The adversarial observation is given by I(xx)) = It esign (V (1),, y; 0)) where y the optimal output action by weighting over all possible actions.
Observation Black-Out. offfthhe-shelf hardware can affect the entire sensor networks as a sensing background (Yurtsever al., 2020) over-shoot with I(x;) (aan et al. 2016). This perturbation is realistic owing to overheat hardware and losing the observational information of sensors.
Frozen Frame. Lagging and frozen frame(s) Kalashnikov al. 2018 often come from limited data communication bottleneck bandwidth. A frozen frame is given by I(z;) = 2t-1- If the perturbation constantly present, the frame will remain the first frozen frame since the perturbation happened.
3.2 CLEVER-Q. Q: ROBUSTNESS EVALUATION METRIC FOR Q-NETWORKS
Here we provide comprehensive score (CLEVER-Q) for evaluating the robustness of a Q-network model by extending the CLEVER robustness score (Weng et al. 20188) designed for classification tasks to Q-network based DRL tasks. Consider an l, ep-norm bounded (p 2 1) perturbation state st. We first derive lower bound 31 on the minimal perturbation to sN for altering the action with the top Q-value, i.e., the greedy action. For a given s, and Q -entwork, this lower bound Br, provides a robustness guarantee that the greedy action at will be the same that of any perturbed state 8, + 6, as long as the perturbation level || S 3r. Therefore, the larger the value 3,, is, more resilience of the Q-network against perturbations can be guaranteed. Our CLEVER-Q score uses the extreme value theory to evaluate the lower bound 3r as a robustness metric for benchmarking different Q-network models. The proof of Theorem 1. is available in appendix B.T
the
Theorem 1. Consider a Q-network Q(s,a) and state St- Let A arg max, (ss a) be the set of greedy (best) actions having the highest Q-value at 8; according to the Q-network. Define (st) Qstt, A²) (s a) for every action o, where Q(st, denotes the best Q-value st- Assume 9a ( is locally Lipschitz continuous 1I with its local Lipschitz constant denoted by Lg where 1/p+ + 1/g = and p 2 1. For any p 2 1, define the lower bound
31 = min Ia (tt)
Then for any 5 such that ||81p BI. we have arg max, Q a) arg max, Q(5; + 8,
Herr locally Lipschitz continuous means 9a (sx) is Lipschitz continuous within the ep ball centered at 8t with radius Rg. We follow the same definition as in (Wng 2018.
4

---

Under review as a conference paper at ICLR 2021
3.3 CAUSAL GRAPHICAL MODEL AND THE BENEFITS ON LEARNING PERFORMANCE
The causal relation of observation, reward, and interference can be described by a causal graphical model (CGM) in Figure 2 We use žt to denote the latent state which can be viewed as confounder in causal inference. Formally, we define 2t = (z &t) be the hidden confounder. Here h is a function which compresses (2z, ie) into a confounder such that the CGM holds. It is clear from Eq. 11 and the MDP definition that the CGM holds with h being the identity function, i.e., 24 = (Iz, 4))- such that is low-dimensional. Similar to Louizos et al. (2017), we aim to learn to predict this low-dimensional hidden confounder by neural network.
We assume that there exists some unknown compression function l
Table Causal hierarchy
Pearl
2009
2019
Bareinboim et al.
2020 in our resilient DRL setting.
Level
Activity
Symbol
Example
(1) Association
Observing
P(r ())
DQN
(III) Intervention
Intervening
P(r (,))))
CIQ (ours)
According to the CGM, different training settings correspond to different levels of Pearl's 's causal hierarchy (Bareinboim et al., 2020 Shpitser& & Pearl 2008 Pearl 2009) as shown in Table ! If only the observations are available, the training process corresponds to Level I of the causal hierarchy, which associates the outcome ry to the input observation x directly by P(ralar). Regular DQN and other algorithms with only observations in training belong this association level. On the other hand, when interference type I and the interference labels i, are available during training, the learning problem is elevated to Level III of the causal hierarchy. In particular, the interference model of Eq. | can be viewed as the intervention logic with the interference label i, being the treatment information. With these information, we can describe the causal inference problem by
P(raldo(a)),, 4,) = P(r]) ( = x1,, 4e)
with the do-operator (Pearl 2019) in the intervention level of the causal hierarchy. Based on the causal hierarchy Theorem (Pearl 2009 we could answer causal questions in the higher Level II given the interference type I and the interference labels i, in the learning process.
We provide an example to analytically demonstrate the learning advantage of having the interference labels during training. Consider an environment of i.i.d. Bernoulli states with P(z; = 1) = P(x: = When dt 1, the agent
0) = 0.5 and two actions and 1. There is no reward taking action 0, pays one unit to have a chance to win a two unit reward with probability gr at state
E {0. 1}.
Therefore, P(r,- = 1 = x, a, = 1) 9s and P(r, = 1 ,. = 2, 0r 1) simple environment is a contextual bandit problem where the optimal policy is to pick a, = 1 at state He = x q. > 0.5, and 0, = if q= S 0.5. If the goal is to find an approximately optimal policy, the agent should take action a 1 during training to learn the probabilities qu and q1. Suppose the environment is subjected to observation black-out x. = 0 with p² = 0.2 when 2, = 1, and no interference when 24 0. Assume go = (3 21)/5/555 5. Then we have P(r, = 1| = 1, az = 1) = @1 and P(r.- = 1/z/ = 0, az = 1) To P(rz = 0jz = 0) + Q P(x: = 1 |0 = 0) = 0.5. If the agent only has the interfered observation x/., the samples for x/ = 0 are irrelevant to learning 9ı because rewards just randomly occur with probability half given x = 0. Therefore, the sample complexity bound is proportional 1/P(x² 1) because only samples with x/ = 1 are relevant. On the other hand, if the agent has access to the labels in during training, even when observed 2/ = O, the agent can infer whether x, = 1 by checking in or not. Therefore, the causal relation allows the agent to learn di by utilizing all samples with
9s- This
1, and the sample complexity bound is proportional to 1/P(x; = 1)= = 2 which is 20% reduction from 1/ 1/p(z' = 1) 2.5 when the labels are not available. Note that 2 (Iz, it) is a latent state for this example, and the latent state and its causal relation is very important to improving learning performance.
From the above discussion, we provide the interference type I the interference labels i, to efficiently train a resilient RL agent with the CGM; however, in the actual testing environment, the agent only has access the interfered observations z/. The CGM allows the agent to infer the latent state 2, and utilize its causal relation with observation, interference and reward to learn resilient behaviors. We will then show how we parameterize this model with novel deep neural network.
5

---

Under review as a conference paper at ICLR 2021
3.4 CAUSAL INFERENCE Q- D-NETWORK
Based on the causal inference graphical model, we propose a causal inference Q-network, referred as CIQ. that able to map the interfered observation sy into a latent state 2t, make proper inferences about the interference condition i,, and adjust our policy based on the estimated interference.
We approximate the latent state by neural network we generate the estimated interference label i, ~ p(ielze) fr(z;;). We denote of (2-- M 2,, it) be the causal inference state. As discussed in the previous sub- section, the causal inference state acts as a confounder between the interference and the reward. Therefore, instead of using the interfered state sy, the causal inference state st contains more relevant information for the agent to maximize rewards. Using the causal inference state helps focus on meaningful and informative details even under interference.
f1(24:01). From the latent state,
With the causal inference state S the output of the Q-network Q(s ()) is set to be switched between two neural networks [x 02) and (,(s(110)) by the interference label. Such a switching mechanism prevents our network from over gererereerll the causal inference state. During training, switching between the two neural networks is determined by the training interference label 2 We assume that the true interference label is available in the training phase so jt it. In the testing, when &e is not available, the two neural networks to use. The design intuition of the inference mechanism is based on the potential outcome estimation theory (Rubin) 1974 Imbens & Rubin 2010 Pearl 2009) in RCM and modeling of the interference scenario as described in Eq. 1 Intuitively. , the switching mechanism (counterfactual inference) from RCM could be considered
use the predicted interference label as the switch decide which of
method to disentangle single deep network into two non- non-parameter parameter-sharin networks to improve model generalization under uncertainty. has shown many advantages for representation learning in regression tasks (Shalit et al. 2017
Louizos et al. 2017 We provide more implementation details in appendix C.1
All the neural networks fi, f2. fa: fo have two fully connected layer& with each layer followed by the ReLU activation except for the last layer in f2, fa and fi. The overall CIQ model is shown in F and 0 = (0,, 02, 0z. Φ) denotes all its parameters. Note that, as common practice for discrete action spaces, the Q-network output (s :0) is an A-dimensional vector where A is the size of the action space, and each dimension represents the value for taking the corresponding action.
Finally, we train the CIQ model (ss;: 0) end-to-end by the DQN algorithm with an additional loss for predicting the interference label. The overall CIQ objective function is defined as:
I, (0,, 02 Os Ø)= = strain I, (01, 82, Φ) + (1 (!(mi))
I, PON (01, 0s, ())
A (i)-ain log p p Ee; 0, σ) + (.rai))) log(1 - p(ie o))),
(4)
where À is scaling constant and is set to for simplicity. The entire CIQ training procedure is de- scribed by Algorithm I
Due to the design of the causal inference state and the switching mechanism, we will show that CIQ can perform resilient behaviors against the observation interferences.
4 EXPERIMENTS
4.1 ENVIRONMENTS FOR DQNS
Our testing platforms were based on (a) OpenAl Gym (Brockman et al., 2016). (b) Unity-3D environments (Juliani et al. 2018), (c) a 2D gaming environment (Brockman et al. 2016). and (d) visual learning from pixel inputs of cart pole. Our test environments cover some major application scenarios and feature discrete actions for training DQN agents with the CLEVER-Q Q analysis.
Vector Cartpole: Cartpole (Sutton et al. 1998) is a classical continuous control problem. The defined environment is manipulated by adding force of +1 or -1 to a moving cart. A pendulum starts upright, and the goal is to balance and prevent it from falling over. We use Cartpole-v0 from Gym (Brockman et al., 2016) with targeted reward 195.0 solve the environment. The observational vector-state consist of four physical parameters of cart position and angle velocities.
Thougg such manner may lead to the myth of wee-pprrameterizztion, our ablation study proves that we achieve better results with almost the same amount of parameters.

---

Under review as a conference paper at ICLR 2021
Algorithm 1 CIQ Training
1: Inputs: Agent, N Now Oracle, max_step. NoisyEnv Bne_test, target, eval_steps
2: Initialize: 1 O, score = O, 8 = Noisy revereset(
3: while mar step and score target do
4: i = Coscle(NossyEntt..
5: o, Agent.accts( (4)
6: 11 Tr. done= = -NoisyyE..steepic
Agent.leam(ss, Ot. Tt. 8(+1> 4t) 8: if eval_stepe then
7:
9: score Agend.eevluaatee siverv.evaluaee(NooiyEEeeee Boissynu_tsss)) if done then
10:
5 = N NoiisyEnu.eeset))
12:
else
8 =
14:
1=
15:
Return Agent
Banana Collector: The Banana collector shown in Figure (b) is one of the Unity baseline (Juliani et al. 2018) rendering by 3D engine. Different from the MuJoCo (Todorov et al. 2012) simulators with continuous actions, the Banana collector controlled by four discrete actions corresponding to moving directions. The state-space has 37 dimensions included velocity and a ray-based perception of objects around the agent. The targeted reward is 12.0 points by accessing correct bananas (+-1).
Lunar Lander: Similar to the Atari gaming environments, Lunar Lander-v2 (Figure (c)) is discrete action environment from OpenAl Gym (Brockman et al.] 2016). The state is an eight- dimensional vector that records the lander's position, velocity, angle, and angular velocities. The episode finishes the lander crashes or comes to rest, receiving a reward -000 or +100 with a targeted reward of 200. Firing ejector costs -0.3 each frame with +10 for each ground contact.
Pixel Cartpole:
To further evaluate our models, we conduct experiments from the pixel inputs in visual learning task. The size of input state is 400 600. We use max-pooling and a convolution layer to extract states as network inputs. The environment includes two discrete actions fleft, right), which is identical to the Cartpole-vO of the vector version.
the cartpole environment
4.2 BASELINE METHODS
In the experiments, we compare our CIQ algorithm with two sets of DRL baselines to demonstrate the resilience capability of the proposed method. We ensure all the models have the same number of 9.7 millions parameters with careful fine-tuning to avoid model capacity issues.
We DQN on interfered state s/. We also evaluate common DQN improvements in appendix C.I and find the improvements have no significant effect against interference.
Pure DQN:
baseline in experiments. The DQN agent is trained and tested
DQN with an interference classifier (DQN-CF): In the resilient reinforcement learning framework, the agent is given the interference label atrain at training. Therefore, would like provide this additional information to the DQN agent for fair comparision. During training, the interfered state xf is concatenated with the true label it as the input for the DQN agent. Since the true label is not available at testing, we train additional binary classifier (CF) for the DQN agent. The classifier trained to predict the interference label, and this predicted label will be concatenated with the interfered state the input for the DQN agent during testing.
DQN with safe actions (DQN-SA): Inspired by chielding-based safe RL (Alshiekh et al. 2018), we consider a DQN baseline with safe actions (SA). The DQN-SA agent will apply the DQN action if there is no interference. However, if the current observation is interfered, it will choose the action used for the last uninterfered observation the safe action. This action-holding method is also typical control approach when there are missing observations (Franklin et al. 1998). Similar to DQN-CE, binary classifier for interference is trained to provide predicted labels at testing.
7

---

Under review as a conference paper at ICLR 2021
CIQ
DQN
DQN-CF
DQN-SA
DVRLQ DVRLQ-CF
1.
(a) ceettrr
(b) Banana" edversorial
(c) Lunar Lunar"dwertt nnwerssrial
(d) Cartpole cartpolendherarce adsersavial pizcel
(e) Wertpoleweebbr
(1) Banana blackont
(g) Lunarhoockout
(h) Cartpole Slackout
Figure 3: Performance of DQNs under potential (20%) adversarial and black-out interference.
DVRLQ and DVRLQ-CF: Motivated by deep variational RL (DVRL) (g et al. 2018). we provide version of DVRL as a POMDP baseline. We call this baseline DVRLQ because we replace the A2C-loss with the DQN loss. Similar to DQN-CR, we also consider another baseline of DVRLQ with a classifier, referred to as DVRLQ-CF, for a fair comparison using the interference labels.
4.3 RESILIENT RL ON AVERAGE RETURNS
We run performance evaluation with six different interference probabilities (p' in Sec. 3.1b. including (0%, 10%, 20%, 30,, 40%, 50%). ). We train each agent 50 times and highlight its standard deviation with lighter colors. Each agent is trained until the target score (shown as the dashed black line) reached or until 400 episodes. We show the average returns for p = 20% under adversarial perturbation and black-out in Figure and report the rest of the results in appendix C.I
CIQ (green) clearly outperforms all the basclines under all types interference, validating the effec- tiveness of our CIQ in learning to infer and gaining resilience against a wide range of observational interferences. Pure DQN (yellow) cannot handle the interference with 20% noise level. DQN-CF (orange) and DQN-SA (brown) have competitive performance in environments against certain interferences, but perform poorly in others. DVRLQ (blue) and DVRLQ-CE (purple) cannot achieve the target reward in most experiments and this might suggest the inefficiency of applying general POMDP approach in a framework with specific structure of observational interference.
4.4 ROBUSTNESS METRICS BASED ON RECORDING STATES
We evaluate the robustness of DQN and CIQ the proposed CLEVER-Q metric. To make the test state environment consistent among different types and levels of interference, we record the interfered states, SN I(Sc), together with their clean states, Sc. We then calculate the average CLEVER-Q for DQN and CIQ based on the clean states Se using Eq. 2 over 50 times experiments for each agent.
We also consider a retrospective robustness metric, the action correction rate (AC-Rate). Motivated by previous off-policy and error correction studies (Dulac-Arnold et al. 2012 Harutyunyan et al. 2016
Lin et al. 2017), AC-Rate is defined the action matching rate R.ct a, and a over an episode with length T. Here a, denotes the action taken by the agent with interfered observations Sv. and at is the action of the agent if clean states So were observed instead. The
E.- [ar=a;)) between
roles of CLEVER Q and AC-Rate are complementary as robustness metrics. CLEVER-O measures sensitivity in terms of the margin (minimum perturbation) required for given state to change the original action. AC-rate measures the utility in terms of action consistency. Altogether, they provide comprehensive resilience assessment.
8

---

Under review as a conference paper at ICLR 2021
Table 2 reports the two robustness metrics for DQN and CIQ under two types of interference. CIQ
attains higher scores than DQN both CLEVER-Q and AC- Aatee reflecting better resilience in CIQ evaluations. We provide more robustness measurements in appendix B.2 and E
Table 2: AC-Rate and CLEVER-O Q robustness analysis under Gaussian (12-norm) and adversarial (loc -noro))m perturbations in the vector Cartpole environment.
T=L2
AC-Rate
CLEVER-Q
Z=Loo
AC-Rate
CLEVER-Q
1%, I
DQN
CIQ
DQN
CIQ
P%, I
DQN
CIQ
DQN
CIQ
10%
82.10%
99.61% %
0.176
0.221
10%
62.23%
99.52%
0.169
0.248
20%
72.15%
98.52%
0.130
0.235
20%
9.68%
98.52% %
0.171
0.236
30%
69.74%
98.12%
0.109
0.232
30%
1.22%
98.10% %
0.052
0.230
4.5
ADDITIONAL ANALYSIS
We also conduct the following analysis to better understand our CIQ model. Environments with a dynamic noise level are evaluated. Due to the space limit, their details in appendix to
Treatment effect analysis: We provide treatment effect analysis on each kind of interference to statistically verify the CGM with lowest errors average treatment effect refutation in appendix D
Ablation studies: We conduct ablation studies by comparing several CIQ variants, each without certain CIQ component. The results verify the importance of the proposed CIQ architecture in appendix E
Test different noise levels: We train CIQ under noise level and test another level. The results show that the difference in noise level does not affect much on the performance in appendix
C.6
Neural saliency map: We apply the perturbation-based saliency map for DRL (Greydanus et al. 2018) in appendix E.4 to visualize the saliency centers and actions CIQ and other baseline agents.
Transferability in robustness: Based on CIQ, we study how well can the robustness of different interference types transfer between training and testing environments. We evaluate two general tings (i) same interference type but different noise levels (appendix C.6 and (ii) different interference types (appendix E.5)
We also provide generalized version of CIQ that deals with multiple interference types at training and testing environments. The generalized CIQ is equipped with common encoder and individual interference decoders to study multi-module conditional inference, as discussed in appendix E.6
Multiple interference types:
CONCLUSION
Our experiments suggest that, although some DRL-based algorithms can achieve high scores under the normal condition, their performance be severely degraded in the presence of interference. DRL
In order to be resilient against interference, we propose CIQ, a novel algorithm. Evaluated on wide range of environments and multiple types of interferences, the CIQ results show consistently superior performance over several RL baseline methods. We also validate the improved resilience of CIQ by CLEVER-O Q and AC-Rate metrics and will open source code.
REFERENCES
Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Prithviraj Ammanabrolu and Mark Riedl. Transfer in deep reinforcement learning using knowledge graphs. In Proceedings of the Thirteenth Workshop on Graph- Based Methods for Natural Language Processing (TextGraphs-13) 13), pp. 1-10, 2019.

---

Under review as a conference paper at ICLR 2021
Alexander Balke and Judea Pearl. Bounds on treatment effects from studies with imperfect compleaane. Journal of the American Statistical Association, (1(439):1171-1117 1997.
Bareinboim, JD Correa, D lbeling, and T Icard. On pearl's hierarchy and the foundations of causal inference. ACM Special Volume in Honor of Judea Pearl (provisional ritle), 2020.
Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: causal approach. In Advances in Neural Information Processing Systems, pp. 1342-1350, 2015.
Yoshua Bengio. Deep learning of representations: Looking forward. In International Conference on Statistical Language and Speech Processing, pp. 1-37. Springer, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.001 01500, 2016.
Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford. John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https
//github. wtw.thtb.com/opeeeeeeg httbbbbcoo/peea//aaaaae. innes 2017.
Gabriel Dulac-Arnold, Ludovic Denoyer, Philippe Preux, and Patrick Gallinari. Fast reinforcement learning with large action sets using error-correcting output codes for mdp factorization. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 180-194. Springer, 2012.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901 190.10995, 2019.
Andrew Forney, Judea Pearl, and Elias Bareinboim. Counterfactual data-fusion for online reinforce- ment learners. In International Conference on Machine Learning, pp. 1156-1164, 2017.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. ICLR 2018, arXiv preprint arXiv: 10295, 2018.
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv: 2015.
Gene F Franklin, David Powell, Michael L Workman, al. Digital control of dynamic systems, volume 3. Addison Addison-wesley Menlo Park, CA, 1998.
Javier Garcia and Fernando Fernández. Safe exploration of state and action spaces in reinforcement learning. Journal of Artificial Intelligence Research, 45:515-555 5644 2012.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
Sander Greenland, Judea Pearl, and James M Robins. Causal diagrams for epidemiologic research. Epidemiology, pp. 37-48, 1999.
Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, and Theophane Weber. Temporal difference variational auto-encoder. arXiv preprint MMXiv:8806.03111 2018.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International Conference on Machine Learning, pp. 1792-1801, 2018.
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3):362-386, 2020.
Shixiang Gu, Ethan Holly, Timothy Lillicrap. and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off off-policy updates. In 2017 IEEE international conference robotics and automation (ICRA), pp. 3389-3396. IEFF, 2017.
10

---

Under review as a conference paper at ICLR 2021
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 2018.
Anna Harutyunyan, Marc G Bellemare, Tom Stepleton, and Rémi Munos. Q lamda with off-policy corrections. In International Conference on Algorithmic Learning Theory, pp. 305-320, Springer, 2016.
Irina Higgins, David Amos, David Pfau, Sehastien Racaniere, Loic Maithey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02255 2018.
Paul W Holland. Causal inference, path analysis and recursive structural equations models. ETS Research Report Series, 1985(1); 1988.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv: 2017.
Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv preprint arXiv:1905.06424, 2019.
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps. International Conference on Machine Learning, pp. 2117- 2126, 2018.
Guido W Imbens and Donald B Rubin. Rubin causal model. In Microeccomometries, pp. 229-241. Springer, 2010.
Amin Jaber, Jiji Zhang, and Elias Bareinboim. Causal identification under markov equivalence: Completeness results. In International Conference on Machine Learning, pp. 2981-2989, 2019.
Tor Johansen, Andrea Cristofaro, Kim Sørensen, Jakob M Hansen, and Thor Fossen. On estimation of wind velocity, angle-of-attack and sideslip angle of small uavs using standard sensors. In 2015 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 510-519, IEEE, 2015.
Arthur Juliani, Vincent-Pierre Berges, Esh Vekay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. arXiv preprint 1rXiv:189.00227. 2018.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint 1806 2018.
Edward H Kennedy. Semiparametric theory and empirical processes in causal inference. In Statistical causal inferences and their applications in public health research, pp. 141-167. Springer, 2016.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Diederik P Kingma and Max Welling. 14226:1312.6114, 2013.
Auto-encoding variational bayes, arXiv preprint
Gilwoo Lee, Brian Hou, Aditya Mandalika, Jeongseok Lee, Sanjihan Choudhury, and Siddhartha S Srinivasa. Bayesian policy optimization for model uncertainty. arXiv preprint arXiv: 1810.01014 2018.
Yen- Cen-Chen Lin, Zhang- Wei Hong, Yuan-Hong Liao, Meng-Li Li Shih, Ming- Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint arXiv:1703.00 1703.0678. 2017.
11

---

Under review as a conference paper at ICLR 2021
Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. In Advances in Neural Information Processing Systems, pp. 6446-6456, 2017.
Chaochao Lu, Bernhard Schölkopf, and José Miguel Hernández-Lobato. Deconfounding reinforce- ment learning in observational settings. arXiv preprint
2018.
Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. Neural adaptive video streaming with pensieve. In Proceedings of the Conference of the ACM Special Interest Group on Data Communi- cation, pp. 197-210. ACM, 2017.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, al. Learning to navigate in complex environments. arXiv preprint /61 1.036667,, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Pol Moreno, Jan Humplik, George Papamakarios, Bernardo Avila Pires, Lars Buesing, Nicolas Heess, and Theophane Weber. Neural belief states for partially observed domains. In NeurIPS 2018 workshop on Reinforcement Learning under Partial Observability, 2018.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt dynamic, real-world environments through meta-reinforreement learning. arXiv preprint arXiv:1803: 1347, 2018.
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepezvari. Satinder Singh, al. Behaviour suite for reinforce- ment learning. arXiv preprint arXiv: 10XXv:1900003560 2019.
Christos H Papadimitriou and John N Tsitsiklis. The complexity of markov decision processes. Mathematics of operations research, (2(3):4114500 1987.
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669-688, 1995a.
Judea Pearl. On the testability of causal models with latent and instrumental variables. In Proceedings of the Eleventh conference on Uncertainty in artificial intelligence, pp. 435-443. Morgan Kaufmann Publishers Inc., 1995b.
Judea Pearl. Causality. Cambridge university press, 2009.
Judea Pearl. The seven tools of causal inference, with reflections machine learning. Communica- tions of the ACM, 62(3):54-66 60, 2019.
Judea Pearl, Madelyn Glymour, and Nicholas Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016
Aditi Raghunathan, Sang Michael Xie, Fanny Yang. John C Duchi, and Percy Liang. Adversarial training can hurt generalization. arXiv preprint 1906 06032, 2019.
Yongming Rao, Jiwen Lu, and Jie Zhou. Attention-aware deep reinforcement learning for video face recognition. In Proceedings of the IEEE international conference computer vision, pp. 3931-3940, 2017.
James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Analysis semiparametric regression models for repeated outcomes in the presence of missing data. Journal of the american statistical association, 90(429):106-12) 1995.
James M Robins, Richard Scheines, Peter Spirtes, and Larry Wasserman. Uniform consistency in causal inference. Biometrika, 90(3):91-515, 2003.
12

---

Under review as a conference paper at ICLR 2021
Kenneth J Rothman and Sander Greenland. Causation and causal inference in AppiemiooogyyAAmericc journal of public health, 95($1):S144-S13) $150, 2005.
Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies.
Journal of educational Psychology, 66(5),688, 1974.
William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without enror.
Towards safe reinforcement learning via human intervention. In Proceedings of the 17th Interna-
fional Conference on Autonomous Agents and MultiAgent Systems, pp. 2067-2069. International
Foundation for Autonomous Agents and Multiagent Systems, 2018.
Jürgen Schmidhuber. Reinforcement learning in markovian and non-markovian environments. In Advances in neural information processing systems, pp. 500-506, 1991.
Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compres- sion. Neural Computation, 4(2):234-24,, 1992.
Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: general- ization bounds and algorithms. In Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 3076-3085. JMLR, org, 2017.
Amit Sharma, Emre Kiciman, et al. DoWhy python package for causal inference. KDD 2019 workshop, 2019.
lyya Shpitser and Judea Pearl. Complete identification methods for the causal hierarchy. Journal of Machine Learning Research, 1441)):14 2008.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy? comprehensive study on the robustness of 18 deep image classification models. In ECCV, pp. 631-648, 2018.
Richard S Sutton, Andrew G Barto, Francis Bach, et al. Reinforcement learning: An introduction. MIT press, 1998.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014.
Lei Tai, Giuseppe Paolo, and Ming Liu. Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 31 31-36. IEEE, 2017.
Guy Tennenholtz, Shie Mannor, and Uri Shalit. Off-policy evaluation in partially observable environ- ments. urXiv preprint arXiv: 03739, 2019.
Philip Thooma, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off-policy evaluation. In Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforeement learning with double q- learning. Thirtieth AAAI conference on artificial intelligence, 2016.
Herke Van Hoof, Nutan Chen, Maximilian Karl, Patrick der Smagt. and Jan Peters. Stable rein- forcement learning with autoencoders for tactile and visual data. In 2016 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3928-3934. IEEE, 2016.
13

---

Under review as a conference paper at ICLR 2021
Tsui-We Wei Weng, Huan Zhang. Pin- Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. Evaluating the robustness of neural networks: An extreme value theory approach. arXiv preprint arXiv:1801.10577 2018.
Chen Yan, Wenyuan Xu, and Jianhao Liu. Can you trust autonomous vehicles: Contactless attacks against sensors of self-driving vehicle. DEFCON24, 2016.
Chao-Han Huck Yang. Jun Qi, Pin- Yu Chen, Yi Ouyang, I-Te Danny Hung, Chin-Hui Lee, and Xiaoli Ma. Enhanced adversarial strategically- attacks against deep reinforcement learning. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3407-3411. IEEE, 2020.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey autonomous driving: Common practices and emerging technologies. IEEE Access, 8:58443-58465 2020.
14

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
INDEX
Our supplementary sections included:
Proof of the CLEVER-O Q Theorem and Additional Robustness Measurements
C. Implementation Details and Additional Results
D. Causal Relation Evaluation and Average Treatment Effects in CIQ Networks
E. Ablation Studies
B PROOF THE CLEVER-Q THEOREM AND ADDITIONAL. ROBUSTNESS MEASUREMENTS
B.1 PROOF OF THE CLEVER-Q Q THROREM
Here provide comprehensive score (CLEVER-OC QQ for evaluating the robustness of a Q-network model by extending the CLEVER robustness score Weng et al. (2018) designed for classification tasks to Q-network based DRL tasks. Consider f--norm bounded (p > perturbation å to the state 8t- We first derive lower bound Br, on the minimal perturbation to St for altering the action with the top Q-value, i.e., the greedy action. For given s, and Q-network, this lower bound Br, provides a robustness guarantee that the greedy action at will be the same as that of any perturbed state s; + 6, as long the perturbation level ||ó||p I 32. Therefore, the larger the value 3g is, the more resilience of the Q-network against perturbations can be guaranteed. Our CLEVER-O score uses the extreme value theory to evaluate the lower bound 3r. as a robustness metrie for benchmarking different Q-network models.
Theorem 2. Consider a Q-network Q(s, a) and a state 8t- Let A = arg max, Q(s;, a) be the set of greedy (best) actions having the highest Q-value at according to the Q-network. Define (s: a) for every action a, where Q(si, A*) denotes the best Q-value at st. Assume ga (st) is locally Lipschitz continuous with its local Lipschitz constant denoted by Lg. where 1/p + 1/q = 1 and p Z 1. Then for any
(a())) = Q(s(,
define the lower bound
BL = min 9a
Then for any 6 such that || S Br.
arg max Q(si, a) arg max Q(si + 5,a)
Proof. Because ga (s:) ) is locally Lipschitz continuous, by Holder's inequality, we have (ga(x) - 9a (v(y)) S Ig[l= - ullp-
(5)
for any æ, y within the Rall centered at st. Now let x = S₁ and y = 8t + 8, where is some perturbation. Then
(st) L.91011 I (st + +) S ga (s 11110
(6)
Note that if (st + 8) 0, then A still remains as the top Q-value action set at state st 8. Moreover, 9a (st) - 16 ||| lp 2 0 implies (st + 5) 2 0. Therefore,
|| (s:)//L
provides a robustness guarantee that ensures Q(*: + 8, A²) Q(+t + d,a) for any satisfying Eq. equation] 5 Finally, to provide robustness guarantee that Q(s; + 5,A") 2 Q(s: + 8, a) for any action a & AA', it suffices take the minimum value of the bound (for each a) in Eq. equation 5 over all actions other than a", which gives the lower bound
B1 minaga (s:)
Here locally Lipschitz continuous means Ia (sx) is Lipschitz continuous within the Ep ball centered at 8t with radius Rg. We follow the same definition as in Weng (2018)
15

---

Under review as a conference paper at ICLR 2021
For computing BL. while the numerator is easy obtain, the local Lipschitz constant Lg cannot be directly computed. In our implementation, by using the fact that Lg is equivalent to the local maximum gradient norm (in Lg norm), theory as proposed in Weng et al. (0188 for estimating I.
the same sampling technique from extreme value
B.2 ADDITIONAL ROBUSTNESS MEASUREMENTS
Following the discussion in Section 4.4 of the main content, provide more experimental results related to CLEVER-Q measurement and use action correction rate (AC-Rate) mentioned in the main content as a reference.
Table SI. reports the two robustness metrics for DQN, CIQ. DQN-CF (a DQN joint-training with an interference classifier, denoted as Q-CF), DQN-SA (a DQN joint-training with safe-action replay, denoted as Q-SA), DVRLQ (a deep variational reinforcement learning framework [gl et al. (2018) with a DQN-loss, denoted as V-Q), and DVRLQ-CF (a V-Q joint-training with an interference classifier, denoted as V-CF) under two types of L, -nom Weng et al. (2018) interference. CIQ attains higher scores than DQN both CLEVER-Q and AC-Rate, reflecting better resilience in CIQ evaluations. The performance of the returns of each agent is shown in Table S2. We observe that variational auto-encoding methods included V-Q and V-CF attaining lower CLEVER Q. Act-Rate, and average returns from Table S1 and S2. From previous studies Van Hoof et al. (20T6), reasons would be difficulties Van Hasselt et al. 2016) of estimation considering temporal information and the various state is hard to disentangle by single network from counterfactual learning (Shalit al. (2017): Louizos et al. (2017). We also conduct an experiment on DQN extension of TD- VAE Gregor al. (2018), but the performance of this extension becomes even lower in all metrics than the V-CI after carefully fine-tuning. We also find that the DQNs would be a benefit on the performance with joint-trained interference classifier shown in Table S1 and S2.
Table 3: AC-Rate and CLEVER-Q Q robustness analysis under Gaussian (Lipschitz 42 norm) and adversarial (Lipschitz A. -orm) Huang et al. (017)) perturbations in the vector Cartpole environment.
I=L2
AC-Rate
CLEVER-O
T=Lxx
AC-Rate
CLEVER-Q
P%, I
DQN
CIQ
DQN
CIQ
P%, I
DQN
CIQ
DQN
CIQ
10%
82.10%
99.61 %
0.176
0.221
10%
62.23%
99.52% %
0.169
0.248
20%
72.15%
98.52%
0.130
0.235
20%
9.68%
98.52% %
0.171
0.236
30%
69.74%
98.12%
0.109
0.232
30%
1.22%
98.10% %
0.052
0.230
PG, I
Q-CF
Q-SA
Q-CF
Q-SA
P%, I
Q-CF
Q-SA
Q-CF
Q-SA
10%
85.10%
83.58%
0.185
0.182
10%
71.82%
67.56%
0.181
0.174
20%
75.23%
74.94%
0.145
0.152
20%
55.28%
51.20%
0.136
0.124
30%
72.25%
71.47%
0.127
0.125
30%
46.45%
43.27%
0.109
0.102
P%, I
V-Q
V-CF
V-Q
V-CF
P%, I
V-Q
V-CF
V-Q
V-CF
10%
83.65%
84.32%
0.184
0.188
10%
61.92%
65.87%
0.173
0.179
20%
71.79%
72.83%
0.123
0.138
20%
51.06%
53.48%
0.117
0.121
30%
69.70%
70.85%
0.094
0.108
30%
36.99%
37.89%
0.087
0.092
C IMPLEMENTATION DETAILS AND ADDITIONAL. RESULTS
C.I BACKGROUND AND TRAINING SETTING
To scale to high-dimensional problems, one can use parameterized deep neural network ((s,, a; 0) to approximate the Q-function, and the network Q(s, a; 8) referred as the deep Q-network (DQN). The DQN algorithm Mnih et al. (2015) updates parameter f according to the loss function:
I DON (0) =E ((soarrr)))))))
DN Q( st. az; 0))
where the transitions (Be, de, Ft. 8141) are uniformly sampled from the replay buffer of previously observed transitions, and y/ PQN Tt + max, ((5++1 a,; 0 is the DQN target with 0 being the target network parameter periodically updated by 0.
16

---

Under review as a conference paper at ICLR 2021
Double DQN (DDQN) Van Hasselt et al. (2016) further improves the performance by modifying the target to Mt DDQN
+ (+++++ arg max, ((s++1, a, 0)) Prioritized replay is another DQN improvement which samples transitions (8t, oe, rt, 81+1) from the replay buffer according to the probabilities Pt proportional to their temporal difference (TD) error: Ps xc lve DDQN - Q(s,, Re; 0) where a is hyperparameter.
We use Pytorch 1.2 to design both DQN and causal inference Q (CIQ) networks in our experiments. Our code can be found in the supplementary material. We use Nvidia GeForce RTX 2080 Ti GPUs with CUDA 10.0 for our experiments. We use the Quantile Huber loss (Dabney et al.] 2018) Lx for DQN models with x = 1 Sup-Eq. 10 which allows less dramatic changes from Huber loss:
{
f K. (lu) 2x),
if |u| S otherwise
(u)
The quantile Huber loss Dabney et al. 2018 is the asymmetric variant of the Huber loss for quantile r E [0, 1] from Sqp-Fq.
(5( = |T - ( (u)
(10)
After the maximum update step in the temporal loss u in Sup-Eq. we synchronize 0, with 0, follow the implementation from the OpenAl baseline Dhariwal et al. (2017) in Sup-Eq I1
Uy (0.) E
DDQN
Q (s. (s.a; 0s)
(11)
Oureei
We use the spdt-up Fox et al.
(2015) to update the DQN target network as in Sup-Eq 12
Orocal T Oreaal + (1 11-77 7) Ourget,
(12)
where 6 fugge and Boeat represent the two neural networks in DQN and J is the soft update parameter depending on the task.
For each enviromment, in additional to the 5 baselines described in Section 4.2. we also evaluate the performance of common DQN improvements such as deep double Q-networks (DDQN) for DDQN with dueling (DDQNa). DDQN with a prioritized replay (DDQNp). DDQN with joint-training interference classifier BODON-CF), and DDQN with a safe action reply (DDQN-SA). We test each model against four types of interference, Gaussian, Adversarial, Blackout, and Frozen Frame, with p' E [10%, 20%, 30%, 40%, 50%]. We also consider a non-stationary noise-level sampling from cosine-wave in range of [0%. 30%] for every ten steps. Results in Table $ show return averaged from the four types of noise in the Env1 to Env4. CIQ shows a better and continuous performance to solve the environments before the noise level attaining 40% and under the cosine- noise. Compared to variational hased DQNs methods, joint-trained DDQN-CF show a much obvious advantages when the noise levels are in the range of 40% to 50%.
C.2 E: CARTPOLE ENVIRONMENT.
We four-layer neural network, which included an input layer, two 32-unit wide ReLU hidden layers, and an output layer (2 dimensions). The observation dimension of Cartpole-v [Bockman al. (2016) is 4 and the input stacks 4 consecutive observations. The dimension of the input layer [4 X 4J. We design a replay buffer with a memory of 100,000, with mini- batch size of 32, the discount factor y is set to 0.99, the r for soft update of target parameter
x 10-3 a learning rate for Adam Kingma & Ba 2014) optimization is 5 x 10-4 regularization term for weight decay is 1 x 10-4, the coefficient a for importance sampling exponent is 0.6, the coefficient of prioritization exponent is 0.4. We train each model 1,000 times for each case and report the mean of the average final performance (average over all types of interference) in Table $4 Env is often recognized as a simplest environment for DQN training. However, over 190.0, when most DQN models attain an over 100.0 score in a 10% noise level. CIQ perform best and competitive results without internal affects from aarrpaaraettriization
observe an stability issue of attaining reward
17

---

Under review as a conference paper at ICLR 2021
Table 4: Performance on return in clean and five different noise level in Env, evaluated by an average of under uncertain perturbation included Gaussian, adversarial, blackout, and frozen frame. All DQN models solve the environment with over 195.0 average returns clean state (a.k.a. no noise).
Model
0%
10%
20%
30%
40%
50%
Cosine
Para.
DQN
195.1
115.0
68.9
32.3
22.8
19.1
42.1
6.9 M
DDQNa
195.1
130.3
85.6
57.2
29.6
23.4
68.2
9.7
DIQN,
195.1
143.6
79.3
56.1
31.2
28.6
60.1
9.7 M
CIQ
195.1
195.1
195.1
195.0
168.2
113.1
195.0
9.7 M
DQN-CF
195.1
192.4
143.6
128.7
68.2
57.8
138.3
9.7
DDQN-CF
195.1
190.2
153.2
138.9
71.4
54.6
141.2
10.7
DVRLQ
195.1
152.6
107.8
76.12
58.12
19.2
72.9
9.7 M
DVRLQ-CF 195.1
163.1
1 119.3
91.3
70.9
25.1
89.2
10.7 M
DQN-SA 195.1
143.2
132.4
112.1
82.9
23.2
107.1
10.7 M
DDQN-SA 195.1
123.4
73.2
59.4
28.1
22.8
62.8
9.7 M
Table 5: Ablation study on parameter of different DQN models using in our experiments training under
four different noise type of noisy environments (P 20%), which included Blackout, Adversarial, Gaussian, and Frozen frame for Env, reported in the main content. The minimal parameters of each model denote as Para. in the Tab. 14
Model
Para.
Gaussian
Adversarial
Blackout
Frozen
DQN
6.9M
67.4
42.5
85.7
62.1
CIQ
9.7M
195.1
195.0
195.1
195.2
DQN-CF
9.7M
149.2
129.1
161.3
167.2
DQN-SA
9.7M
128.9
144.5
109.1
165.8
DVRLQ
9.7M
107.1
87.3
127.56
142.4
DVRLQ-CF
10.7M
112.3
97.82
131.3
152.3
C.3 ENV2: 3D BANANA COLLECTOR ENVIRONMENT.
We utilize the Unity Machine Learning Agents Toolkit Juliani al. 2018). which is open-souree" and reproducible 3D rendering environment for the task of Banana Collector. A reproducible source code, which is designed render the collector agent, has been given in the supplementary for both Linux and Windows systems. A reward of +1 is provided for collecting yellow banana, and reward of -1 provided for collecting a blue banana. We use six-layer deep network. which includes an input layer, three 64-unit fully-connected ReLU hidden layers, soft-attention layer Rao al. (2017) (for all DQNs), and an output layer (2 dimensions). We use [37 4] for our input layer, which composes from the observation dimension (37) and the stacked input of 4 consecutive observations. We design a replay buffer with a memory of 100,000, with mini-batch size of 32, discount factor is equal to 0.99, the for a soft update of target parameter is 10- 3, learning rate for Adam Kingma & Ba 2014) optimization is 5 x 10 10-4, a regularization term for weight decay is 1 x 10 the coefficient for importance sampling exponent is 0.6, the coefficient of prioritization exponent is 0.4.
We train each model 1,000 times for each case and report the mean of the average final performance (average over all types of interference) in Table $6) We report the DVRLQ-CF, which attains a higher performance among DVRLQ and DVRLQ-SA, performs an overall best performance compared to the other baselines. Interestingly. learning an interference improves general performance combined with the joint-training frameworks.
compare with DQN-CF, DDQN-CF. CIQ still
C.4 ENV8: LUNAR LANDER ENVIRONMENT.
The lunar lander-v2 Brockman et al. (016)) is one of the most challenging environments with discrete actions. The observation dimension of Lunar Lander-v2 Brockman et al. (2016) is 8 and the input stacks consecutive observations. The objective of the game is to navigate the lunar lander
Source: http:///iiihaoom///iiiyiiooioogg
18

---

Under review as a conference paper at ICLR 2021
Table 6: Performance on return in clean and five different noise level in Enve evaluated by an average of under uncertain perturbation included Gaussian, adversarial, blackout, and frozen frame. All DQN models solve the environment with over 12.0 average returns in clean state (a.k.a. no noise).
Model
0 0%
10%
20%
30%
40%
50%
Cosine
Para.
DQN
12.0
9.5
5.1
3.4
2.0
2.1
3.2
7.6 M
DDQN
12.0
10.1
5.8
6.8
4.8
2.6
6.9
9.7
DDQN&
12.0
11.8
6.7
7.4
5.4
2.8
6.6
9.7 M
DDQN,
12.0
11.9
9.2
7.2
5,4
3.2
8.3
9.7M
CIQ
12.0
12.0
12.0
12.0
11.8
7.8
12.0
9.7 M
DQN-CF
12.0
12.0
11.5
10.1
9.0
6.9
11.8
9.7 M
DDQN-CF
12.0
12.0
11.8
10.8
10.0
7.1
11.9
10.7 M
DVRLQ-CF
12.0
12.0
11.2
10.5
9.2
6.8
11.6
10.7 M
Table 7: Ablation study on parameter of different DQN models using in our experiments training under four different noise type of noisy environments (P 20%). which included Blackout, Adversarial,
Gaussian, and Frozen frame for Env; reported in the main content. The minimal parameters of each model denote as Para. in the Tab. 14
Model
Para.
Gaussian
Adversarial
Blackout
Frozen
DQN
6.9M
6.1
4.5
5.2
5.7
CIQ
9.7M
12.0
12.0
12.0
12.0
DQN-CF
9.7M
12.0
10.9
11.3
11.8
DQN-SA
9.7M
12.0
11.1
11.1
11.2
DVRLQ
9.7M
12.0
11.3
10.3
11.6
DVRLQ-CF
10.7M
12.0
11.5
10.5
11.6
spaceship to a targeted landing spot without collision. A collection of six discrete actions controls two real-valued vectors ranging from -1 to +1. The first dimension controls the main engine on and off numerically, and the second dimension throttles from 50% to 100% power. The following two actions represent for firing left, and the last two actions represent for firing the right engine. The dimension of the input layer is [8 x 10]. We design 7-layers neural network for this task, which includes input layer, layer of 32 unit wide fully-connected ReLU network, 2 layers deep 64-unit wide ReLU networks, soft-attention layer Rao et al. (217)) (for all DQNs), and output layer (4 dimensions). The replay buffer size is 500,000; the miniumm batch size is 64, the discount factor is 0.99, the for soft update of target parameters is 10-3 the learning rate is 5 10-1 1, the minimal step for reset memory buffer is 50. We train each model 1, 1,000 times for each case and report the mean of the average final performance (average over all types of interference) a challenging task owing to often receive negative reward during the training. We thus consider non-stationary noise-level sampling from a cosine-wave in a narrow range of [0%, 20%] for every ten steps. Results suggest CIQ could still solve the environment before the noise-level going over to 30%. For the various noisy test, CIQ attains best performance over 200.0 the other DQNs algorithms (skipping the table since only CIQ and DQN-CF have solved the environment over 200.0 training with adversarial and blackout interference.)
Table S8 Env3 is
C.5 EN PIXEL CARTPOLE ENVIRONMENT
To observe pixel inputs of Cartpole-v I as states, we use screen- with an original size of [400, 600, 3]. We first resize the original frame into a single gray-scale channel, [100, 150] from the RGN2GRAY function in the OpenCV. The implementation details are shown in the "pixel _toe___tool.p" and paautolle_pixel.py which could be refereed to the submitted supplementary code. Then we stack consecutive gray-scale frames the input. We design 7-layer DQN model, which included input layer, the first hidden layer convolyes 32 filters of a [8 x 8] kernel with stride 4, the second hidden layer convolves 64 filters of [4 x 4] kernel with stride 2, the third layer is fully-connected layer with 128 units, from fourth to fifth layers are fully-connected layer with units, a soft-attention layer Rao et al. (201)) (for all DQNs), and the output layer (2 dimensions). The replay buffer size is
19

---

Under review as a conference paper at ICLR 2021
Table 8: Performance on average return in clean and five different noise level Env, evaluated by an average of under uncertain perturbation included Gaussian, adversarial, blackout, and frozen frame. All DQN models solve the environment with over 200.0 average returns in clean state input (a.k.a. no noise).
Model
0%
10%
20%
30%
40%
50%
Cosine
Para.
DQN
200.0
54.2
-102.1
-134.6
-200.2
-298.7
12.4
12.3M
DDON
200.0
70.1
15.8
-6.8
-124.8
-24.22
32.5
15.5M
DDQNA
200.0
92.2
26.7
-7.4
-167.4
-100.1
43.3
15.5M
DDQN,
200.0
102.1
39.2
-17.2
-189
-00.88
65.0
15,5M
CIQ
200.0
200.0
200.0
107.8
50.1
17.2
200.0
15.5M
DQN-CF
200.0
98.2
81.5
40.1
9.0
-59.9
85.6
15.5M
DDQN-CF
200.0
188.2
91.8
58.2
10.0
-20.1
128.9
15.5M
DVRLQ-CF
200.0
198.2
121.1
80.2
29.2
11.2
165.4
15.5M
500,000; the minimum batch size is 32, the discount factor is 0.99, the for a soft update of target parameters 10-3 the learning rate 5 10-4, the minimal step for reset memory buffer is 1000.
We train each model 1,000 times for each case and report the mean of the average final performance (average over all types of interference) in Table SQ
Table 9: Performance average return in clean and five different noise level in Env, evaluated by an average of under uncertain perturbation included Gaussian, adversarial, blackout, and frozen frame. Only selected DQN models below solve the environment with over 195.0 average returns in a clean state input (a.k.a. no noise).
Model
0%
10%
20%
30%
40%
50%
Cosine
Para.
DDQN
195.0
182.7
99.7
75.5
47.4
13.2
90.4
20.5M
DDQN,
195.0
187.9
139.2
88.3
51.6
15.9
120.3
20.5M
CIQ
195.0
195.0
195.0
157.8
90.1
87.2
195.0
20.5M
DQN-CF
195.0
175.0
112.5
90.0
52.0
39.2
92.8
20.5M
DDQN-CF 195.0
187.2
162.8
117.9
59.8
49.1
102.6
20.5M
DVRLQ-CF
195.0
193.6
121.1
80.2
63.2
41.4
91.2
20.5M
Table 10: Ablation study on parameter of different DQN models using in our experiments training under four different noise type of noisy environments (P 20%), which included Blackout, Adver- sarial, Gaussian, and Frozen frame for Env. reported in the main content. The minimal parameters of each model denote as Para. the Tab.
Model
Para.
Gaussian
Adversarial
Blackout
Frozen
DQN
6.9M
52.6
37.5
70.5
47.6
CIQ
9.7M
195.0
195.0
195.0
195.0
DON-CF
9.7M
119.2
99.1
131.3
137.2
DQN-SA
9.7M
98.0
114.6
79.2
135.2
DVRLQ
9.7M
107.1
87.3
127.56
142.4
DVRLQ-CF
10.7M
121.3
97.82
131.3
152.3
C.6 TRAIN AND TEST ON DIFFERENT NOISE LEVEL
We consider settings with different training and testing noise levels for CIQ evaluation. The (train, test)% case trains with train% noise then tests with test% noise. Their results shown in Table ST1 are similar to the cases with the same training and testing noise level. We observe that CIQ have the capability of learning transformable q-value estimation, which attain succeed score of 195.00 in the noise level 30 + 10%. Meanwhile, other DQNs methods included DDQN-CF, DVRLQ-CF, DDQN-SA perform general performance decay in the test different noise level. This result would be limited to the generalization of power and challenges Bengio 20 Higgins et al. (2018) in as disentangle unseen state of single parameterized deep network.
20

---

Under review as a conference paper at ICLR 2021
Table 11: Stability test of proposed CIQ (Train Noise-Level, Test Noise-Level)
Metrics
(10, 30)%
(30, 10)%
(30, 20)%
(30, 30)%
(30, 40)%
(30, 50)%
Performance
182.8
195.0
195.0
195.0
195.0
185.7
CLEVER-O Q
0.195
0.239
0.232
0.230
0.224
0.215
AC-Rate
91.45 %
98.54%
98.62%
99.45%
98.45%
92.45%
D CAUSAL EFFECTS
In causal learning setting, evaluating treatment effects and conducting statistical refuting experiments are essential to support the underlying causal graphical model. Through resilient reinforcement
learning framework, we could interpret DQN by estimating the average treatment effect (ATE) of each noisy and adversarial observation. We first define how to calculate treatment effect in the resilient RL settings and conduct statistical refuting tests including random common cause variable test (Te), replacing treatment with random (placebo) variable (Tp), and removing random subset of data (T,). The open-source causal inference package Dowhy Sharma et al. (2019) used for analysis.
D.1 AVERAGE TREATMENT EFFECT UNDER INTERVENTION
We refine a Q-network with discrete actions for estimating treatment effects based on Theorem 1 in Louizos et al.) (2017). In particular, individual treatment effect (ITE) can be defined as the difference between the two potential outcomes of a Q-network; and the average treatment effect (ATE) is the expected value of the potential outcomes over the subjects.
binary treatment setting, for a Q-value function (8)) and the interfered state I(s:), the ITE and ATE are calculated by:
Q Qi(s)) (1 pa) + (I(((s))) T ATE Σ t=1
(13)
(I(s)) - E [Q!TE (st)
(14)
where Pt is the estimated inference label by the agent and T is the total time steps of each episode. As expected, find that CIQ indeed attains better ATE and its significance can be informed by the refuting tests based on T.. Tp and T..
To evaluate the causal effect, we follow standard refuting setting Rothman & Greenland (2005):
Pearl et al. (2016); ; Pearl (1995b) with the causal graphical model in Fig. of the main context to run three major tests, as reported in Tab. [3] The code for the statistical test was conducted by Dowhy (Sharma et al. (2019), which has been submitted as supplementary material. (We intend to open source as a reproducible result.)
Pearl Pearl 1995a) introduces "do-operator" to study this problem under intervention. The do symbol removes the treatment tr, which equal to interference I in the Eq. (1) of the main content, from the given mechanism and sets it to a specific value by some external intervention. The notation P(r, do(())) denotes the probability of reward r' with possible interventions on treatment time t. Following Pearl's back-door adjustment formula Pearl (2009) and the causal graphical model in Figure of the main content., it is proved in Louizos et al. (2017) that the causal effect for given binary treatment tr (denoted as a binary interference label in Eq. (1) of the main content), series of proxy variables X (ET- z1) S' (E s(). s in Eq. (1) of the main content, summation of accumulated reward R = (2' 1= re) and a confounding variable Z can be evaluated by (similarly for tr =0):
do PRRSS' do(tr = = p(|s, do(tr 1), Z),(Z) do 1))dZ Jz (R tr 1,Z)p(Z(S')dZZ
(15)
where equality (i) is by the rules of do-calculus Pearl (995a); Pearl et al. 2016) applied to the causal graph applied on Figure of the main content. We extend to Eq. on individual outcome study with DQNs, which is known by the Theorem 1. from Louizos et. al. Louizos et al. (2017) and Chapter 3.2 of Pearl Pearl (2009).
21

---

Under review as a conference paper at ICLR 2021
D.2 REFUTATION TEST:
sampling plan for collecting samples refer as subgroups (i=l, ... k). Common cause variation (T-c) is denoted as ce. which is an estimate of common cause variation within the subgroups in terms of the standard deviation the within subgroup variation:
8: k,
(16)
where i denotes as the number of sample size. We introduce intervention a enror rate n, which is probability to feed error interference (e.g.. feed i, 0 even under interference with a probability n)
and results shown in Table 12
The test (T-p) of replacing treatment with random (placebo) variable is conducted by modifying the graphical relationship in the proposed probabilistic model in Fig. 3 of the main context. The new assign variable will follow the placebo note but with a value sampling from a random Gaussian distribution. The test of removing random subset of data (T-r) is to randomly split and sampling the subset value to calculate an average treatment value in the proposed graphical model. We use the official dowhy dowhy@implende implementationn which includes: (1) confounders effect treatment: how the simulated confounder affects the value of treatment; (2) confounders effect on outcome: how the simulated confounder affects the value of outcome; (3) effect strength on treatment: parameter for the strength of the effect of simulated confounder on treatment, and (4) effect strength on outcome: parameter for the strength of the effect of simulated confounder on outcome. Following the refutation experiment in the CEVAE paper, we conduct experiments shown in Tab. S and $13) with 10 % to 50 % intervention noise on the binary treatment labels. The results in Tab. 112 show that proposed CIQ maintains a lower rate compared with the benchmark methods included logistic regression and CEVAE (refer to Fig. (b) in Louizos al. (017
Through Eq. (9) to (10) and the corresponding correct action rate in the main context, we could interpret deep q-network by estimating the average treatment effect (ATE) of each noisy and adver- sarial observation. ATE Louizos et al. (2017); Shalit et al. (2017) is defined as the expected value of the potential outcomes (c.g.. disease) over the subjects (c.g., clinical fcatures.) For example, in navigation environments, we could rank the harmfulness of each noisy observation against q-network from the autonomous driving agent.
Table 12: Absolute eтror ATE estimate; lower value indicates much stable causal inference under perturbation on logic direction with p 10% and n=error rate of intervention on the binary label.
Model
n=0.1
n=0.2
n=0.3
n=0.4
n=0.5
LR
0.062
0.084
0.128
0.151
0.164
CEVAE
0.021
0.042
0.062
0.072
0.081
CIQ
0.019
0.020
0.015
0.018
0.023
Table 13: Validation of causal effect by three causal refuting tests. The causal effect estimate is tested by random common cause variable test (T-c), replacing treatment with a random (placebo) variable (T-p- lower is better), and removing random subset of data (T-r). Adversarial attack outperforms in most tests.
Noise: : do(Z)
== 0.1
Il 0.2
Method
ATE
w/ T-c
w/ T-p
w/ T-s
ATE
T-T-c
w/ w/T-p
W/ T-s
Adversarial
0.2432
0.2431
0.0294
0.2488
0.0868
0.0868
0.0109
0.0865
Black-out
0.2354
0.2212
0.0244
0.2351
0.0873
0.0870
0.0140
0.0781
Gaussian
0.1792
0.1763
0.0120
0.1751
0.0590
0.0610
0.0130
0.0571
Frozen Frame
0.1614
0.1614
0.0168
0.1435
0.0868
0.0868
0.0140
0.0573
22

---

Under review as a conference paper at ICLR 2021
E ABLATION STUDIES
E. THE NUMBER OF MODEL PARAMETERS
We also spend efforts on a parameter-study on the results of average returns between different DQN-based models, which included DQN, Double DQN (DDQN), DDQN with dueling, CIQ, DQN with a classifier (DQN-CF), DDQN with classifier (DDQN-CF), DQN with a variational nutoencoderrKKngnnne
Welling (2033) (DQN-VAE), NoisyNet, and using the latent input of causal effect variational autoencoder for Q network (CEVAE-Q) prediction. Overall, CEVAE-Q is with minimal-requestee parameters with 14.4 M (in Env1) as the largest model used in our experiments in Tab. 14 CIQ remains roughly similar parameters as 9.7M compared with DDQN, DDQN and Noisy Net. Our ablation study in Tab. indicates the advantages of CIQ are not owing extensive features using in the model according to the size of parameters. CIQ attains benchmark results in our resilient reinforcement learning setting compared to the other DQN models.
Table 14:
Ablation study on parameter of different DQN models using in our experiments in Env1 Env2, EnV3, and Env4. The minimal parameters of each model denote Para. in the Table 10
Model
Para.
Env
Env2
Env3
Env4
DQN
6.9M
20.2
3.1
-13.6
10.8
DDQN
9.7M
41.1
3.5
-123.4
57.9
DDQNa
9.7M
82.9
4.7
-136.3
67.2
CIQ
9.7M
195.1
12.5
200.1
195.2
DQN-CF
9.7M
140.5
12.5
-78.3
120.2
DDQN-CF
12.1M
161.3
12.5
-10.1
128.2
DQN-VAE
9.7M
151.1
7.6
-92.9
24.1
NoisyNet
9.7M
158.6
5.5
50.1
100.1
CEVAE-Q
12.5M
39.8
11.5
-156.5
45.8
DVRLQ-CF
10.7M
107.11 11
9.2
-34.9
42.5
Noisy Nets Fortunato et al. (2018) has been introduced benchmark whose parameters perturbed by parametric noise function. We select Noisy Net in a DQN format as a noisy training baseline with interfered state sf concated with interference label i, from a classifier.
(A)
(C)
(d)
Figure 4: Perturbation-base saliency map on Pixel Cartpole under adversarial perturbation: (a) DQN, (b) CIQ, (c) DQN-CF, and (d) DVRLQ-CF. The black arrows are correct actions and blue arrows are agents' actions. The neural saliency of CIQ makes more correct actions responding to ground actions.
E.2 LATENT REPRESENTATIONS
We conduct an ablation study by comparing other latent representation methods to the proposed CIQ model.
DQN with an variational autoencoder (DQN-VAE): learn important features from observations, many recent works leverage deep variational inference for accessing latent states for feeding into DQN. We provide baseline on training a variational autoencoder (VAE) built upon the DQN baseline, denoted DQN-VAE. The DQN-VAE baseline is targeted to recover potential noisy state and feed the bottleneck latent features into the Q-network.
CEVAE-Q Network: TARNet (Shalit et al. 2017 Louizos al. 2017) is major class of neural net- work architectures for estimating outcomes of a binary treatment on linear data (c.g., clinical reports). Our proposed CIQ uses end-to-end approach to leam the interventional (causal) features. We provide another baseline on using the latent features from a causal variational autoencoder (Louizos) al. 2017) (CEVAE) latent features state inputs followed the loss function in (Louizos et al.
23

---

Under review as a conference paper at ICLR 2021
2017). To get the causal latent model in Q Q-network, approximate the posterior distribution by a neural network 24 ~ p [+)= = (());11). Then we train this neural network, CEVAE-Q, by variational inference using the generative model.
We conduct 10,000 times experiments and fine-tuning on DQN-VAE and CEVAE-Q. The results in Table 15 shows that the latent representation learned by CIQ provides better resilience than other representations.
Table 15: Performance on average return in clean and five different noise level in Env1 evaluated by an average of under uncertain perturbation included Gaussian, adversarial, blackout, and frozen frame. All DQN models solve the environment with over 195.0 average returns in a clean state input (a.k.a. no noise).
Model
0%
10%
20%
30%
40%
50%
Cosine
Para.
DQN
195.1
115.0
68.9
32.3
22.8
19.1
42.1
6.9 M
DDQN
195.
123.4
73.2
59.4
28.1
22.8
62.8
9.7 M
CIQ
195.1
195.1
195.1
195.0
168.2
113.1
195.0
9.7 M
DQN-VAE
195.1
173.5
141.3
124.8
86.5
33.3
101.2
9.7 M
DON-CEVAE
195.1
154.4
111.9
94.8
75.5
48.3
82.1
12.5 M
Table 16: Structure-wiss ablation studies of CIQ in Env1 (noise level P 20%).
Model
Return
CLEVER-O Q
AC-Rate
CIQ
195.1
0.241
97.3
BI: CIQ w/o the concatenation
152.1
0.196
78.2
B2: CIQ w/o the H3 network
150.1
0.182
65.6
B3: CIQ w/o providing grounded for training
135.1
0.142
53.6
E.3 ARCHITECTURE ABLATION STUDY ON CIQ
To study the importance of specific components in CIQ, we conducted additional ablation studies and constructed two new baseline models shown in Table 16 tested in Env, (Cartpole). Baseline 1 (B1) CIQ w/o the concatenation of i, in Sf. This comparison shows the importance of using both the predicted confounder & and the predicted label i, B1 uses label prediction help latent representation but not using the predicted labels in decision-making. The structure is motivated by task-specific (depth-only information from maze environment) DQN network from previous study MMirowski et al. (2016). Baseline 2 (B2) CIQ w/o the 03 network (for testing 0g 's importance) The structure structure|Humplik et al. (2019) motivated by meta-inference reinforcement learning proposed by. Baseline 3 (B3)- CIQ w/o providing grounded i, for training, for testing the importance of the inference loss and joint loss propagation. The superior performance of CIQ validates the proposed model is indeed crucial from the previous discussion in Section 3 of the main content. The setting used for Table is the same as the setting for the third column (noise level 20%) in Table 5 and the third column (noise level = 20%) in Table 15, tested in Env, (Cartpole).
E.4 PERTURBATION- NEURAL SALIENCY FOR DQN AGENTS
To better understand our CIQ model, use the benchmark saliency method DQN agent, perturbation- saliency map, (Greydanus et al. 2018 to visualize the salient pixels, which sensitive to the loss function of the trained DONs. We made case study of input frame under an adversarial perturbation, as shown in Fig. We evaluate DQN agents included DQN, CIQ, DQN-CF, DVRLQ-CE LF and record its weighted center from the neural saliency map. where saliency pixels of CIQ respond to ground true actions more frequent (96.2%) than the other DQN methods.
E.5 ROBUSTNESS TRANSFERABILITY AMONG DIFFERENT INTERFERENCE TYPES
We conduct additional experiments to study robustness transferability of DQN and CIQ when training and testing under different kinds of interference types in Env, Note that both architectures would
24

---

Under review as a conference paper at ICLR 2021
solve a clean environment successfully (over 195.0). The reported numbers are averaged over 20 independent runs for each condition. As shown Table 17 and Table 18 CIQ agents consistently attain significant performance improvement when compared with DQN agents, especially between Gaussian and adversarial perturbation. For example, CIQ succeeded to solve the environment 12 times out of 20 independent runs, with an average score of 165.2 in Gaussian (rriin)Adversariall (test) adaptation. n particular, for CIQ, 12 times out of 20 independent runs are successfully transfered from Gaussian Adversarial perturbation. Interestingly, augmenting adversarial perturbation does not always guarantee the best policy transfer when testing in the Blackout and Frozen conditions, which shows a slightly lower performance compared with training on Gaussian interference. The reason could be attributed to the recent findings that adversarial training can undermine model generalization (Raghunathan al. 2019 Su al. 2018).
Table 17: DQN adaptation: train and test on different interference (noise level 20%) in Env1-
Train / Test
Gaussian
Adversarial
Blackout
Frozen
Gaussian
67.4
38.4
43.7
52.1
Adversarial
53.2
42.5
35.3
44.2
Blackout
46.2
27.4
85.7
50.3
Frozen
62.3
26.2
45.9
62.1
Table CIQ adaptation: train and test on different interference (noise level = 20%) in Env
Train Test
Gaussian
Adversarial
Blackout
Frozen
Gaussian 195.1
165.2
158.2
167.8
Adversarial
162.8
195.0
152.4
162.5
Blackout
131.3
121.1
195.3
145.7
Frozen
161.6
135.8
147.1
195.2
E.6 CIQ WITH MULTI- INTERFERENCE.
Here show how the proposed CIQ model be extended from the architecture shown in Figure to the multi-interference (MI) setting. The design intuition is based on two-step inference by common encoder, infer a clean or noisy observation, followed by an individual decoder tied an interference type, to infer noisy types and activate the corresponding Q-network (named 0₃).
Note that the two-step inference mechanism follows the RCM as two sequential potential outcome es- timation models (Rubin] 1974 Imbens & Rubin 2010h. where interfered observation x;, is determined extended
( from Eq. where it, indicates the presence of interference and 22, indicates which interference type (here we show the case of two types). As a proof of concept, we consider two interference types together, Gaussian noise and adversarial perturbation. In this setting every observation (state) can possibly undergo an interference with either Gaussian noise or Adversarial perturbation. From the results shown in Table. 19 find that the extended version of CIQ, CIQ-MI, is capable of making correct action to solve (over 195.0) ) the environment when training with mixed interference types (last row). Another finding that robustness tranferability (153.9/1 .91.9.115.2) in CIQ-MI is slightly degraded compared to the results (162.8/16) 8/165.12 2) in Table 18 with the same training episodes (500) and runs (20), which could be caused by the increased requirement of model capacity (Ammanabrolu & Riedl 2019 of CIQ-MI.
by two labels i and 22, according to x = 11, (22, (x.) (x+)+(1-1))) (1-1111,0)) 12,
Table 19: CIQ-MI: CIQ agent with an extended (noise level P 20%).
(MI) architecture testing in Env
Train Test
Gaussian
Adversarial
Gaussian Adversarial
Gaussian
195.1
154.2
96.3
Adversarial
153.9
195.0
105.1
Gaussian Adversarial
195.0
195.0
195.0
25


Reviews:
[{"review_id": "qT8kzxTEc2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method, the Causal Inference Q-Network (CIQ), for training deep RL agents that are robust to abrupt interferences in observations, such as frame blackouts, Gaussian noise or adversarial perturbations. During training time, a binary interference label is provided to the agent at each time step indicating whether an interference has been applied to the observation; the interference label acts as a switch between two neural networks that process the observation to predict the Q-values. The CIQ agent learns to predict the interference label in a supervised fashion, and at test time it uses the predicted label to switch between networks, rather than the true label. The CIQ agent is shown to learn faster and more effectively when compared to a number of baselines on a selection of OpenAI Gym tasks that are modified to include various types of observational interferences.\n\nWhile this paper presents a method that is shown to perform well empirically in the setting is aimed to tackle, I cannot recommend it for acceptance because (i) almost no motivation or intuition is given for the architectural choices that seem to be key to the performance of the agent (in particular the switching mechanism between Q-networks), and (ii) the characterisation of the agent as performing causal inference, which is the key message of the paper, is confusing in a number of ways. As a result, it’s not clear what we are supposed to learn conceptually from the paper and how it can guide future research.\n\nPositives:\n\t•\tIn order to be able to apply deep RL agents in real-world settings, they need to be robust to noisy observations, and so the paper is tackling an important problem. \n\t•\tThe chosen baselines seem fairly chosen, such as the safe-action DQN and the DQN with concatenated interference label, and the agents are evaluated across a variety of different interference types and levels. A number of different metrics are used to evaluate the performance of each agent. \n\t•\tThe CIQ agent performs the best when compared to the baselines in the interfered observation setting. \n\nConcerns:\n\t•\tThere is little to no motivation given for the architectural choice of having the interference label modulate a switching mechanism between Q-networks, which based on the comparison to baselines and the ablation studies in Appendix E.3 seem to be key to the agent’s performance. The only justifying claim I could identify was the following: “Such a switching mechanism prevents our network from over-generalizing the causal inference state.” Could the authors clarify what exactly they mean by over-generalizing the causal inference state and how can this be inferred from the empirical results? \n\t•\tBoth the CIQ and DQN-CF agent learn to predict the interference probability and use it for predicting Q-values, with the difference that the CIQ uses this predicted probability to switch between two Q-networks, but DQN-CF performs significantly worse on average - what is the intuition for this? \n\t•\tThe causal graph drawn in Figure 2a seems to contradict the experimental setup described in the text: \n\t◦\t(i) The interference label i_t is stated to be sampled from a Bernoulli process but in the causal graph drawn in Figure 2a, there is an arrow from the latent state z_t towards i_t; in what way does the latent state affect the probability of interference? \n\t◦\t(ii) In the graph, there is no arrow (and no indirect path) from the interference label to the observed state x’_t; this implies that intervening on the label, which according to Equation 1 directly affects the value of x’_t, would not actually affect x’_t. This is directly contradictory. \n\t◦\t(iii) In the graph, there is an arrow from i_t to r_t but there is no mention in the paper for how the interference label can directly affect the reward in a given time step. \nIt’s possible I have badly misunderstood the translation of the setup into the causal graph provided - in which case, could the authors please explain where I have gone wrong? \n\t•\tSection 3.3 describes a toy example that is used to show how being provided interference labels during training can lead to better sample efficiency in learning the reward distributions for each of two states (one of which is occasionally subject to interference that switches the observation to the other state). In this example, however, it is not possible to infer the latent state from the potentially interfered observation at test time, while in the DQN agents the advantage of the CIQ agent is stated to come from the fact that it can do just that. It’s not clear here whether the message is that the CIQ agent performs better due to better sample efficiency or due to its ability to infer the latent state at test time. \n\nOther questions / comments\n\t•\t As far as I understood, the interference at test time always corresponds to the interference provided during training in the experiments. Were any experiments run to test generalisation to unseen types of interferences at test time? In a real world setting, this could be useful.", "extra_scores": {}, "extra_text": {}}, {"review_id": "T5Fv2YXrq0P", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper presents a framework for deep reinforcement learning that is motivated by causal inference and with the central objective of being resilient to observational interferences. The key idea is to use interference labels in the training phase to learn a causal model including a hidden confounding state, and then use this model in the testing to make safer decisions and improve resilience. The authors also propose a new robustness measure, CLEVER-Q, which estimates a noise bound of an RL model below which the model's greedy decision would not be altered. The framework is tested extensively over multiple applications and under different types of observational interferences. The results show a clear advantage of the proposed framework over baseline RL methods in terms of resilience to interference.\n\n##########################################################################\n\nReasons for Score:\n\nThe paper addresses an important problem in AI relating to the robustness of the algorithms and paradigms to noisy interference. The proposed framework appears to be sound and the experimental results show superior performance in comparison to other baseline RL methods. That being said, I am not very familiar with the literature on RL and Deep learning, so my decision is more of an educated guess. However, I do have a concern about the causal inference component of the paper (explained below) and this is reflected by the score.\n\n##########################################################################\n\nCons:\n\nThe causal component of the proposed framework is not well-explained. More specifically, the causal graphical model in Figure 2 is introduced at the beginning of Subsection 3.3 very briefly, and the authors don't explain the intuition behind constructing this graph.\nThe authors say, \"We use z_t to denote the latent state which can be viewed as a confounder in causal inference\", but there is no explanation for why this makes sense. Is it just an assumption that happens to work?\n\nMoreover, the following phrase appears to be inaccurate \"knowing the interference labels it or not corresponds to different levels in Pearl’s causal hierarchy...: the intervention level with the interference labels and the association level without the information\". Knowing vs not knowing the interference labels does not correspond to interventional vs associational levels in the causal hierarchy, but simply switching a variable/node in the CGM between observed and latent/unobserved. Such knowledge of a variable in the CGM does not account for an intervention.\n\n##########################################################################\n\nQuestions during the rebuttal period:\n\nIt would help if the authors can clarify the issue raised in the \"Cons\" above regarding the clarity of the causal component and its central role, as claimed, in the proposed framework.\n\n##########################################################################\n\nTypos:\n\n- p.1, \"the RL agent is asked to learn a binary causation label and *embedded* a latent state into its model\": embedded -> embed.\n- p.3, \"design an end-to-end structure ... and *evaluated* by treatment effects on rewards\": The statement does not parse.\n-p.3, \"where M is a *fix* number for the history\": fix -> fixed.\n-p.3, \"We assume that interference labels i_t *follows* an i.i.d. Bernoulli process\": follows -> follow.\n\n##########################################################################\n\nComments after Discussion:\n\nI appreciate the effort made by the authors to elaborate on the causal formulation behind CIQ. However, the additional discussions in the paper are still confusing and raise soundness concerns. Some of the issues are discussed below.\n\n1- Rubin's Causal Model: The authors reference RCM in Subsection 3.1 for the causal formulation yet the rest of the work does not seem to use the potential outcome notation. Instead, Subsection 3.3 uses graphical models and the do-operator which follows the causal framework by Pearl. Then, in Subsection 3.4, the authors go back to reference RCM. It is not clear why this alternation between the two approaches is employed.\n\n2- If $z_t$ is defined as a function of $x_t$ and $i_t$, shouldn't the CGM reflect that with an arrow from $i_t$ to $z_t$ instead of it being the other way around in Fig.2(a)? Despite the attempt by the authors to elaborate on the causal formulation, I'm unable to map the structural equations such as Eq. (1) and the function of $z_t$ to the given CGM in Fig.2(a).\n\n3- The discussion in Subsection 3.3 leading to Eq. (3) sounds flawed to me. Quoting the authors, \"the interference model of Eq. (1) can be viewed as the intervention logic with the interference label it being the treatment information\". This statement is elaborating on the formulation of $x_t'$ where $x_t$ is intervened on and replaced by an interfered state when $i_t=1$. Alternatively, $x_t$ is kept intact when $i_t=0$. This intervention on the mechanism of $x_t$ happens whether we obtain $i_t$ and train the DQN with it or not. In this sense, the intervention is not happening under the CIQ framework only, but also when we simply train based on $x_t'$. Accordingly, it is not clear to me how \"the learning problem is elevated to Level II of the causal hierarchy\" due to the presence of the interference labels. To be clear, I'm not questioning the significance of using the interference labels in the training, but rather the causal story and formulation behind CIQ.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rx_p9MiDGdL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: The paper introduces a causal mechanism that both creates and explains away noise interventions into observational data fed into RL agents. The authors propose a form of resilient agent, that based on training data containing labeled interventions, learns both Q function and the causal impact of interventions on the Q function. The model architecture consists of a intervention predictor and a split parametrization of the Q function estimation as a function of intervention as shown in L^{CIQ} presented as eq.3. Finally, the authors show the performance of their proposed method on 4 visual based RL agents with two types of interventions (attacks): namely adversarial and blackout against classical baselines such as DQN and DQN with safe actions.\n\nPros: \n- Clarity: Overall I find the paper well-written and reasonably easy to follow. The problem is well motivated and the related work relevant. \n- Significance/Impact: I think the problem that the paper is trying to solve is relevant and with potentially big impact\n- Experimental design: I think the experiments section is relevant and makes a strong case for the method\n\nCons:\n- Though clear at most times, the paper should spend more time explaining the basic causal terminology and the assumptions behind the causal graph introduced in Figure 2a. I find that the authors introduce the causal coneepts in an informal, intuitive way, but that should be followed-up by a clear formalism. \n- I find that the biggest downside of the method is that it needs to be trained with the type of invervention that the agent will be resilient to. It would be interesting to create an intervention detector that is fully unsupervised and that is based more on a state anomaly detection. \n\nBordeline:\n- Novelty: In terms of novelty, the paper is a relatively straight-forward application of do-calculus to Q-value learning.\n\nFinal comments:\n\tOverall i found the paper interesting and the approach relevant, supported by rigurous experimentation. On the downside, the novelty of the paper does not seem major and the predefined nature of interventions might make it unrealistic in a lot of RW scenarios.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7yZFJp9HmuZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\nThe paper makes two main contribution: 1). A metric for evaluating RL agent's relience 2). A framework that uses inteference type as label to enhance relience.\n\nReasons for score:\nOverall, I am towards accepting the paper but I believe there are much improvements to make. The idea of adding interference type as label is quite novel, and the authors provide extensive experiment results to show that CIQ achieves better resilience against interferences. I believe that the proposed framework has the potential to enhance a single agent with multiple types of attcks and I suggest the authors to look into this direction.\n\nComments:\n- The proposed network architecture does not seem to differ from a normal DQN other than an additional interference type output. As such, I am not very convinced by the causal inference insight.\n- Is it possible to have multiple interference type during training? The resulting agent would then have the potential to be resilient to all types of interferences during evaluation. Furthermore, it would then be possible to combine different types of adversarial attacks into adversarial training. For real life application, I believe that it is possible that different types of interference could happen simultaneously.\n- Computation cost of CLEVER-Q is expensive, and there is no guarantee of the estimated CLEVER-Q score. Also, the advantage of CLEVER-Q over AC-rate is not discussed.\n\n\nMinor issues:\n- Appendix C.1, this is just huber loss instead of quantile huber loss.\n- Appendix D, 'Refer to We'.\n\n\nQuestions:\n1. If I understand correctly, the difference between DQN-CF and CIQ is just that the inteference loss does not propogate to the Q-network parameters in DQN-CF?\n2. Depending on whether there are interference, how different are the outputs of f_2 and f_3? If f_2 output is similar to f_3 even when there is no interference, it would suggest that we only need f_2 and the switching mechanism can be removed.\n\nPost rebuttal:\nThe authors have addressed most of my main concerns and provided detailed experiment results, hence I increase my score to 7.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a deep reinforcement learning method that aims at ensuring resilience to observational interference. During training labels that indicate presence or absence of interference are available to the algorithm. The training objective is augmented to learn the prediction of interference that is used at test time to infer the interference label. The experimental results show superior performance in comparison to other baseline RL methods.

The main objection raised by the reviewers was on the confusing and possibly unsound causal formulation. The authors' clarifications during the discussion did not eliminate bur rather exacerbated the reviewers' doubts. I read the paper in full myself to understand whether the reviewers' confusion was justified, and whether it could be easily resolved by an improved explanation, or it is a more serious issue.  I did not succeed in clearly understanding the causal formulation nor its relevance, and also have  soundness concerns. Figure 2a does not seem to be a correct explanation of the causal mechanism. It is also not clear from this figure why z is called confounder. More generally, I was not able to reach a coherent and sound causal formulation from the authors' explanation. My conclusion is that the framing of the paper as causal inference based is not well justified. 
Decision: Reject