Causal Inference Q-Network: Toward Resilient Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( DRL ) methods have shown enhanced performance , gained widespread applications ( Mnih et al. , 2015 ; 2016 ; Ecoffet et al. , 2019 ; Silver et al. , 2017 ; Mao et al. , 2017 ) , and improved robot learning ( Gu et al. , 2017 ) in navigation systems ( Tai et al. , 2017 ; Nagabandi et al. , 2018 ) . However , most successful demonstrations of these DRL methods are usually trained and deployed under well-controlled situations . In contrast , real-world use cases often encounter inevitable observational uncertainty ( Grigorescu et al. , 2020 ; Hafner et al. , 2018 ; Moreno et al. , 2018 ) from an external attacker ( Huang et al. , 2017 ) or noisy sensor ( Fortunato et al. , 2018 ; Lee et al. , 2018 ) . For examples , playing online video games may experience sudden black-outs or frame-skippings due to network instabilities , and driving on the road may encounter temporary blindness when facing the sun . Such an abrupt interference on the observation could cause serious issues for DRL algorithms . Unlike other machine learning tasks that involve only a single mission at a time ( e.g. , image classification ) , an RL agent has to deal with a dynamic ( Schmidhuber , 1992 ) and encoded state ( Schmidhuber , 1991 ; Kaelbling et al. , 1998 ) and to anticipate future rewards . Therefore , DRL-based systems are likely to propagate and even enlarge risks ( e.g. , delay and noisy pulsed-signals on sensor-fusion ( Yurtsever et al. , 2020 ; Johansen et al. , 2015 ) ) induced from the uncertain interference . In this paper , we investigate the resilience ability of an RL agent to withstand unforeseen , rare , adversarial and potentially catastrophic interferences , and to recover and adapt by improving itself in reaction to these events . We consider a resilient RL framework with observational interferences . At each time , the agent ’ s observation is subjected to a type of sudden interference at a predefined possibility . Whether or not an observation has interfered is referred to as the interference label . Specifically , to train a resilient agent , we provide the agent with the interference labels during training . For instance , the labels could be derived from some uncertain noise generators recording whether the agent observes an intervened state at the moment as a binary causation label . By applying the labels as an intervention into the environment , the RL agent is asked to learn a binary causation label and embed a latent state into its model . However , when the trained agent is deployed in the field ( i.e. , the testing phase ) , the agent only receives the interfered observations but is agnostic to interference labels and needs to act resiliently against the interference . For an RL agent to be resilient against interference , the agent needs to diagnose observations to make the correct inference about the reward information . To achieve this , the RL agent has to reason about what leads to desired rewards despite the irrelevant intermittent interference . To equip an RL agent with this reasoning capability , we exploit the causal inference framework . Intuitively , a causal inference model for observation interference uses an unobserved confounder ( Pearl , 2009 ; 2019 ; 1995b ; Saunders et al. , 2018 ; Bareinboim et al. , 2015 ) to capture the effect of the interference on the reward collected from the environment . When such a confounder is available , the RL agent can focus on the confounder for relevant reward information and make the best decision . As illustrated in Figure 1 , we propose a causal inference based DRL algorithm termed causal inference Q-network ( CIQ ) . During training , when the interference labels are available , the CIQ agent will implicitly learn a causal inference model by embedding the confounder into a latent state . At the same time , the CIQ agent will also train a Q-network on the latent state for decision making . Then at testing , the CIQ agent will make use of the learned model to estimate the confounding latent state and the interference label . The history of latent states is combined into a causal inference state , which captures the relevant information for the Q-network to collect rewards in the environment despite of the observational interference . In this paper , we evaluate the performance of our method in four environments : 1 ) Cartpole-v0 – the continuous control environment ( Brockman et al. , 2016 ) ; 2 ) the 3D graphical Banana Collector ( Juliani et al. , 2018 ) ) ; 3 ) an Atari environment LunarLander-v2 ( Brockman et al. , 2016 ) , and 4 ) pixel Cartpole – visual learning from the pixel inputs of Cartpole . For each of the environments , we consider four types of interference : ( a ) black-out , ( b ) Gaussian noise , ( c ) frozen screen , and ( d ) adversarial attack . In the testing phase mimicking the practical scenario that the agent may have interfered observations but is unaware of the true interference labels ( i.e. , happens or not ) , the results show that our CIQ method can perform better and more resilience against all the four types of interference . Furthermore , to benchmark the level of resilience of different RL models , we propose a new robustness measure , called CLEVER-Q , to evaluate the robustness of Q-network based RL algorithms . The idea is to compute a lower bound on the observation noise level such that the greedy action from the Q-network will remain the same against any noise below the lower bound . According to this robustness analysis , our CIQ algorithm indeed achieves higher CLEVER-Q scores compared with the baseline methods . The main contributions of this paper include 1 ) a framework to evaluate the resilience of DRL methods under abrupt observational interferences ; 2 ) the proposed CIQ architecture and algorithm towards training a resilient DRL agent , and 3 ) an extreme-value theory based robustness metric ( CLEVER-Q ) for quantifying the resilience of Q-network based RL algorithms . 2 RELATED WORKS . Causal Inference for Reinforcement Learning : Causal inference ( Greenland et al. , 1999 ; Pearl , 2009 ; Pearl et al. , 2016 ; Pearl , 2019 ; Robins et al. , 1995 ) has been used to empower the learning process under noisy observation and have better interpretability on deep learning models ( Shalit et al. , 2017 ; Louizos et al. , 2017 ) , also with efforts ( Jaber et al. , 2019 ; Forney et al. , 2017 ; Bareinboim et al. , 2015 ) on causal online learning and bandit methods . Defining causation and applying causal inference framework to DRL still remains relatively unexplored . Recent works ( Lu et al. , 2018 ; Tennenholtz et al. , 2019 ) study this problem by defining action as one kind of intervention and calculating the treatment effects on the action . In contrast , we introduce causation into DRL by applying extra noisy and uncertain inventions . Different from the aforementioned approaches , we leverage the causal effect of observational interferences on states , and design an end-to-end structure for learning a causal-observational representation evaluating treatment effects on rewards . Adversarial Perturbation : An intensifying challenge against deep neural network based systems is adversarial perturbation for making incorrect decisions . Many gradient-based noise-generating methods ( Goodfellow et al. , 2015 ; Huang et al. , 2017 ) have been conducted for misclassification and mislead an agent ’ s output action . As an example of using DRL model playing Atari games , an adversarial attacker ( Lin et al. , 2017 ; Yang et al. , 2020 ) could jam in a timely and barely detectable noise to maximize the prediction loss of a Q-network and cause massively degraded performance . Partially Observable Markov Decision Processes ( POMDPs ) : Our resilient RL framework can be viewed as a POMDP with interfered observations . Belief-state methods are available for simple POMDP problems ( e.g. , plan graph and the tiger problem ( Kaelbling et al. , 1998 ) ) , but no provably efficient algorithm is available for general POMDP settings ( Papadimitriou & Tsitsiklis , 1987 ; Gregor et al. , 2018 ) . Recently , Igl et al . ( 2018 ) have proposed a DRL approach for POMDPs by combining variational autoencoder and policy-based learning , but this kind of methods do not consider the interference labels available during training in our resilient RL framework . Safe Reinforcement Learning : Safe reinforcement learning ( SRL ) ( Garcia & Fernández , 2012 ) seeks to learn a policy that maximizes the expected return , while satisfying specific safety constraints . Previous approaches to SRL include reward-shaping ( Saunders et al. , 2018 ) , noisy training ( Fortunato et al. , 2018 ) , shielding-based SRL ( Alshiekh et al. , 2018 ) , and policy optimization with confident lower-bound constraints ( Thomas et al. , 2015 ) . However , finding these policies in the first place could need to reset the model at each time and be computationally challenging . Our proposed resilient RL framework can be viewed as an approach to achieve SRL ( Alshiekh et al. , 2018 ) , but we focus on gaining resilience against abrupt observation interferences . Another key difference between our framework and other SRL schemes is the novelty in proactively using available interference labels during training , which allows our agent to learn a causal inference model to make safer decisions . 3 RESILIENT REINFORCEMENT LEARNING . In this section , we formally introduce our resilient RL framework and provide an extreme-value theory based metric called CLEVER-Q for measuring the robustness of DQN-based methods . We consider a sequential decision-making problem where an agent interacts with an environment . At each time t , the agent gets an observation xt , e.g . a frame in a video environment . As in many RL domains ( e.g. , Atari games ) , we view st = ( xt−M+1 , . . . , xt ) to be the state of the environment where M is a fixed number for the history of observations . Given a stochastic policy π , the agent chooses an action at ∼ π ( st ) from a discrete action space based on the observed state and receives a reward rt from the environment . For a policy π , define the Q-function Qπ ( s , a ) = E [ ∑∞ t=0 γ trt|s0 = s , a0 = a , π ] where γ ∈ ( 0 , 1 ) is the discount factor . The agent ’ s goal is to find the optimal policy π∗ that achieves the optimal Q-function given by Q∗ ( s , a ) = maxπ Qπ ( s , a ) . 3.1 RESILIENCE BASE ON AN INTERVENTIONAL PERSPECTIVE . We consider a resilient RL framework where the observations are subject to interference ( as illustrated in Fig 1 ( a ) ) as an empirical process in Rubin ’ s Causal Model ( RCM ) ( Kennedy , 2016 ; Holland , 1988 ; Balke & Pearl , 1997 ; Robins et al. , 2003 ) for causal inference . Given a type of interference I , the agent ’ s observation becomes : x′t = F I ( xt , it ) = it × I ( xt ) + ( 1− it ) × xt ( 1 ) where it ∈ { 0 , 1 } is the label indicating whether the observation is interfered at time t or not ( under the potential outcome estimation ( Rubin , 1974 ) ) , and I ( xt ) is the interfered observation . The interfered state is then given by s′t = ( x ′ t−M+1 , . . . , x ′ t ) . We assume that interference labels it follow an i.i.d . Bernoulli process with a fixed interference probability pI as a noise level . For example , when pI equals to 10 % , each observational state has a 10 % chance to be intervened under a perturbation . The agent now needs to choose its actions at ∼ π ( s′t ) based on the interfered state . The resilient RL objective for the agent is to find a policy π to maximize rewards in this environment under observation interference . In this work , we consider four types of interference as described below . Gaussian Noise . Gaussian noise or white noise is a common interference to sensory data ( Osband et al. , 2019 ; Yurtsever et al. , 2020 ) . The interfered observation becomes I ( xt ) = xt + nt with a zero-mean Gaussian noise nt . The noise variance is set to be the variance of all recorded states . Adversarial Observation . Following the standard adversarial RL attack setting , we use the fast gradient sign method ( FGSM ) ( Szegedy et al. , 2014 ) to generate adversarial patterns against the DQN prediction loss ( Huang et al. , 2017 ) . The adversarial observation is given by I ( xt ) = xt + sign ( ∇xtQ ( xt , y ; θ ) ) where y is the optimal output action by weighting over all possible actions . Observation Black-Out . Off-the-shelf hardware can affect the entire sensor networks as a sensing background ( Yurtsever et al. , 2020 ) over-shoot with I ( xt ) = 0 ( Yan et al. , 2016 ) . This perturbation is realistic owing to overheat hardware and losing the observational information of sensors . Frozen Frame . Lagging and frozen frame ( s ) ( Kalashnikov et al. , 2018 ) often come from limited data communication bottleneck bandwidth . A frozen frame is given by I ( xt ) = xt−1 . If the perturbation is constantly present , the frame will remain the first frozen frame since the perturbation happened .

Summary: The paper presents a framework for deep reinforcement learning that is motivated by causal inference and with the central objective of being resilient to observational interferences. The key idea is to use interference labels in the training phase to learn a causal model including a hidden confounding state, and then use this model in the testing to make safer decisions and improve resilience. The authors also propose a new robustness measure, CLEVER-Q, which estimates a noise bound of an RL model below which the model's greedy decision would not be altered. The framework is tested extensively over multiple applications and under different types of observational interferences. The results show a clear advantage of the proposed framework over baseline RL methods in terms of resilience to interference.