Causal Inference Q-Network: Toward Resilient Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( DRL ) methods have shown enhanced performance , gained widespread applications ( Mnih et al. , 2015 ; 2016 ; Ecoffet et al. , 2019 ; Silver et al. , 2017 ; Mao et al. , 2017 ) , and improved robot learning ( Gu et al. , 2017 ) in navigation systems ( Tai et al. , 2017 ; Nagabandi et al. , 2018 ) . However , most successful demonstrations of these DRL methods are usually trained and deployed under well-controlled situations . In contrast , real-world use cases often encounter inevitable observational uncertainty ( Grigorescu et al. , 2020 ; Hafner et al. , 2018 ; Moreno et al. , 2018 ) from an external attacker ( Huang et al. , 2017 ) or noisy sensor ( Fortunato et al. , 2018 ; Lee et al. , 2018 ) . For examples , playing online video games may experience sudden black-outs or frame-skippings due to network instabilities , and driving on the road may encounter temporary blindness when facing the sun . Such an abrupt interference on the observation could cause serious issues for DRL algorithms . Unlike other machine learning tasks that involve only a single mission at a time ( e.g. , image classification ) , an RL agent has to deal with a dynamic ( Schmidhuber , 1992 ) and encoded state ( Schmidhuber , 1991 ; Kaelbling et al. , 1998 ) and to anticipate future rewards . Therefore , DRL-based systems are likely to propagate and even enlarge risks ( e.g. , delay and noisy pulsed-signals on sensor-fusion ( Yurtsever et al. , 2020 ; Johansen et al. , 2015 ) ) induced from the uncertain interference . In this paper , we investigate the resilience ability of an RL agent to withstand unforeseen , rare , adversarial and potentially catastrophic interferences , and to recover and adapt by improving itself in reaction to these events . We consider a resilient RL framework with observational interferences . At each time , the agent ’ s observation is subjected to a type of sudden interference at a predefined possibility . Whether or not an observation has interfered is referred to as the interference label . Specifically , to train a resilient agent , we provide the agent with the interference labels during training . For instance , the labels could be derived from some uncertain noise generators recording whether the agent observes an intervened state at the moment as a binary causation label . By applying the labels as an intervention into the environment , the RL agent is asked to learn a binary causation label and embed a latent state into its model . However , when the trained agent is deployed in the field ( i.e. , the testing phase ) , the agent only receives the interfered observations but is agnostic to interference labels and needs to act resiliently against the interference . For an RL agent to be resilient against interference , the agent needs to diagnose observations to make the correct inference about the reward information . To achieve this , the RL agent has to reason about what leads to desired rewards despite the irrelevant intermittent interference . To equip an RL agent with this reasoning capability , we exploit the causal inference framework . Intuitively , a causal inference model for observation interference uses an unobserved confounder ( Pearl , 2009 ; 2019 ; 1995b ; Saunders et al. , 2018 ; Bareinboim et al. , 2015 ) to capture the effect of the interference on the reward collected from the environment . When such a confounder is available , the RL agent can focus on the confounder for relevant reward information and make the best decision . As illustrated in Figure 1 , we propose a causal inference based DRL algorithm termed causal inference Q-network ( CIQ ) . During training , when the interference labels are available , the CIQ agent will implicitly learn a causal inference model by embedding the confounder into a latent state . At the same time , the CIQ agent will also train a Q-network on the latent state for decision making . Then at testing , the CIQ agent will make use of the learned model to estimate the confounding latent state and the interference label . The history of latent states is combined into a causal inference state , which captures the relevant information for the Q-network to collect rewards in the environment despite of the observational interference . In this paper , we evaluate the performance of our method in four environments : 1 ) Cartpole-v0 – the continuous control environment ( Brockman et al. , 2016 ) ; 2 ) the 3D graphical Banana Collector ( Juliani et al. , 2018 ) ) ; 3 ) an Atari environment LunarLander-v2 ( Brockman et al. , 2016 ) , and 4 ) pixel Cartpole – visual learning from the pixel inputs of Cartpole . For each of the environments , we consider four types of interference : ( a ) black-out , ( b ) Gaussian noise , ( c ) frozen screen , and ( d ) adversarial attack . In the testing phase mimicking the practical scenario that the agent may have interfered observations but is unaware of the true interference labels ( i.e. , happens or not ) , the results show that our CIQ method can perform better and more resilience against all the four types of interference . Furthermore , to benchmark the level of resilience of different RL models , we propose a new robustness measure , called CLEVER-Q , to evaluate the robustness of Q-network based RL algorithms . The idea is to compute a lower bound on the observation noise level such that the greedy action from the Q-network will remain the same against any noise below the lower bound . According to this robustness analysis , our CIQ algorithm indeed achieves higher CLEVER-Q scores compared with the baseline methods . The main contributions of this paper include 1 ) a framework to evaluate the resilience of DRL methods under abrupt observational interferences ; 2 ) the proposed CIQ architecture and algorithm towards training a resilient DRL agent , and 3 ) an extreme-value theory based robustness metric ( CLEVER-Q ) for quantifying the resilience of Q-network based RL algorithms . 2 RELATED WORKS . Causal Inference for Reinforcement Learning : Causal inference ( Greenland et al. , 1999 ; Pearl , 2009 ; Pearl et al. , 2016 ; Pearl , 2019 ; Robins et al. , 1995 ) has been used to empower the learning process under noisy observation and have better interpretability on deep learning models ( Shalit et al. , 2017 ; Louizos et al. , 2017 ) , also with efforts ( Jaber et al. , 2019 ; Forney et al. , 2017 ; Bareinboim et al. , 2015 ) on causal online learning and bandit methods . Defining causation and applying causal inference framework to DRL still remains relatively unexplored . Recent works ( Lu et al. , 2018 ; Tennenholtz et al. , 2019 ) study this problem by defining action as one kind of intervention and calculating the treatment effects on the action . In contrast , we introduce causation into DRL by applying extra noisy and uncertain inventions . Different from the aforementioned approaches , we leverage the causal effect of observational interferences on states , and design an end-to-end structure for learning a causal-observational representation evaluating treatment effects on rewards . Adversarial Perturbation : An intensifying challenge against deep neural network based systems is adversarial perturbation for making incorrect decisions . Many gradient-based noise-generating methods ( Goodfellow et al. , 2015 ; Huang et al. , 2017 ) have been conducted for misclassification and mislead an agent ’ s output action . As an example of using DRL model playing Atari games , an adversarial attacker ( Lin et al. , 2017 ; Yang et al. , 2020 ) could jam in a timely and barely detectable noise to maximize the prediction loss of a Q-network and cause massively degraded performance . Partially Observable Markov Decision Processes ( POMDPs ) : Our resilient RL framework can be viewed as a POMDP with interfered observations . Belief-state methods are available for simple POMDP problems ( e.g. , plan graph and the tiger problem ( Kaelbling et al. , 1998 ) ) , but no provably efficient algorithm is available for general POMDP settings ( Papadimitriou & Tsitsiklis , 1987 ; Gregor et al. , 2018 ) . Recently , Igl et al . ( 2018 ) have proposed a DRL approach for POMDPs by combining variational autoencoder and policy-based learning , but this kind of methods do not consider the interference labels available during training in our resilient RL framework . Safe Reinforcement Learning : Safe reinforcement learning ( SRL ) ( Garcia & Fernández , 2012 ) seeks to learn a policy that maximizes the expected return , while satisfying specific safety constraints . Previous approaches to SRL include reward-shaping ( Saunders et al. , 2018 ) , noisy training ( Fortunato et al. , 2018 ) , shielding-based SRL ( Alshiekh et al. , 2018 ) , and policy optimization with confident lower-bound constraints ( Thomas et al. , 2015 ) . However , finding these policies in the first place could need to reset the model at each time and be computationally challenging . Our proposed resilient RL framework can be viewed as an approach to achieve SRL ( Alshiekh et al. , 2018 ) , but we focus on gaining resilience against abrupt observation interferences . Another key difference between our framework and other SRL schemes is the novelty in proactively using available interference labels during training , which allows our agent to learn a causal inference model to make safer decisions . 3 RESILIENT REINFORCEMENT LEARNING . In this section , we formally introduce our resilient RL framework and provide an extreme-value theory based metric called CLEVER-Q for measuring the robustness of DQN-based methods . We consider a sequential decision-making problem where an agent interacts with an environment . At each time t , the agent gets an observation xt , e.g . a frame in a video environment . As in many RL domains ( e.g. , Atari games ) , we view st = ( xt−M+1 , . . . , xt ) to be the state of the environment where M is a fixed number for the history of observations . Given a stochastic policy π , the agent chooses an action at ∼ π ( st ) from a discrete action space based on the observed state and receives a reward rt from the environment . For a policy π , define the Q-function Qπ ( s , a ) = E [ ∑∞ t=0 γ trt|s0 = s , a0 = a , π ] where γ ∈ ( 0 , 1 ) is the discount factor . The agent ’ s goal is to find the optimal policy π∗ that achieves the optimal Q-function given by Q∗ ( s , a ) = maxπ Qπ ( s , a ) . 3.1 RESILIENCE BASE ON AN INTERVENTIONAL PERSPECTIVE . We consider a resilient RL framework where the observations are subject to interference ( as illustrated in Fig 1 ( a ) ) as an empirical process in Rubin ’ s Causal Model ( RCM ) ( Kennedy , 2016 ; Holland , 1988 ; Balke & Pearl , 1997 ; Robins et al. , 2003 ) for causal inference . Given a type of interference I , the agent ’ s observation becomes : x′t = F I ( xt , it ) = it × I ( xt ) + ( 1− it ) × xt ( 1 ) where it ∈ { 0 , 1 } is the label indicating whether the observation is interfered at time t or not ( under the potential outcome estimation ( Rubin , 1974 ) ) , and I ( xt ) is the interfered observation . The interfered state is then given by s′t = ( x ′ t−M+1 , . . . , x ′ t ) . We assume that interference labels it follow an i.i.d . Bernoulli process with a fixed interference probability pI as a noise level . For example , when pI equals to 10 % , each observational state has a 10 % chance to be intervened under a perturbation . The agent now needs to choose its actions at ∼ π ( s′t ) based on the interfered state . The resilient RL objective for the agent is to find a policy π to maximize rewards in this environment under observation interference . In this work , we consider four types of interference as described below . Gaussian Noise . Gaussian noise or white noise is a common interference to sensory data ( Osband et al. , 2019 ; Yurtsever et al. , 2020 ) . The interfered observation becomes I ( xt ) = xt + nt with a zero-mean Gaussian noise nt . The noise variance is set to be the variance of all recorded states . Adversarial Observation . Following the standard adversarial RL attack setting , we use the fast gradient sign method ( FGSM ) ( Szegedy et al. , 2014 ) to generate adversarial patterns against the DQN prediction loss ( Huang et al. , 2017 ) . The adversarial observation is given by I ( xt ) = xt + sign ( ∇xtQ ( xt , y ; θ ) ) where y is the optimal output action by weighting over all possible actions . Observation Black-Out . Off-the-shelf hardware can affect the entire sensor networks as a sensing background ( Yurtsever et al. , 2020 ) over-shoot with I ( xt ) = 0 ( Yan et al. , 2016 ) . This perturbation is realistic owing to overheat hardware and losing the observational information of sensors . Frozen Frame . Lagging and frozen frame ( s ) ( Kalashnikov et al. , 2018 ) often come from limited data communication bottleneck bandwidth . A frozen frame is given by I ( xt ) = xt−1 . If the perturbation is constantly present , the frame will remain the first frozen frame since the perturbation happened .

Summary: Overview: The paper introduces a causal mechanism that both creates and explains away noise interventions into observational data fed into RL agents. The authors propose a form of resilient agent, that based on training data containing labeled interventions, learns both Q function and the causal impact of interventions on the Q function. The model architecture consists of a intervention predictor and a split parametrization of the Q function estimation as a function of intervention as shown in L^{CIQ} presented as eq.3. Finally, the authors show the performance of their proposed method on 4 visual based RL agents with two types of interventions (attacks): namely adversarial and blackout against classical baselines such as DQN and DQN with safe actions.