Provably Efficient Causal Reinforcement Learning with Confounded Observational Data

Empowered by neural networks , deep reinforcement learning ( DRL ) achieves tremendous empirical success . However , DRL requires a large dataset by interacting with the environment , which is unrealistic in critical scenarios such as autonomous driving and personalized medicine . In this paper , we study how to incorporate the dataset collected in the offline setting to improve the sample efficiency in the online setting . To incorporate the observational data , we face two challenges . ( a ) The behavior policy that generates the observational data may depend on unobserved random variables ( confounders ) , which affect the received rewards and transition dynamics . ( b ) Exploration in the online setting requires quantifying the uncertainty given both the observational and interventional data . To tackle such challenges , we propose the deconfounded optimistic value iteration ( DOVI ) algorithm , which incorporates the confounded observational data in a provably efficient manner . DOVI explicitly adjusts for the confounding bias in the observational data , where the confounders are partially observed or unobserved . In both cases , such adjustments allow us to construct the bonus based on a notion of information gain , which takes into account the amount of information acquired from the offline setting . In particular , we prove that the regret of DOVI is smaller than the optimal regret achievable in the pure online setting when the confounded observational data are informative upon the adjustments . 1 Introduction . Empowered by the breakthrough in neural networks , deep reinforcement learning ( DRL ) achieves significant empirical successes in various scenarios [ 19 , 23 , 36 , 37 ] . Learning an expressive function approximator necessitates collecting a large dataset . Specifically , in the online setting , it requires the agent to interact with the environment for a large number of steps . For example , to learn a human-level policy for playing Atari games , the agent has to interact with a simulator for more than 108 steps [ 13 ] . However , in most scenarios , we do not have access to a simulator that allows for trial and error without any cost . Meanwhile , in critical scenarios , e.g. , autonomous driving and personalized medicine , trial and error in the real world is unsafe and even unethical . As a result , it remains challenging to apply DRL to more scenarios . To bypass such a barrier , we study how to incorporate the dataset collected offline , namely the observational data , to improve the sample efficiency of RL in the online setting [ 21 ] . In contrast to the interventional data collected online in possibly expensive ways , observational data are often abundantly available in various scenarios . For example , in autonomous driving , we have access to trajectories generated by the drivers . As another example , in personalized medicine , we have access to electronic health records from doctors . However , to incorporate the observational data in a provably efficient way , we have to address two challenges . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . • The observational data are possibly confounded . Specifically , there often exist unobserved random variables , namely confounders , that causally affect the agent and the environment at the same time . In particular , the policy used to generate the observational data , namely the behavior policy , possibly depends on the confounders . Meanwhile , the confounders possibly affect the received rewards and the transition dynamics . In the example of autonomous driving [ 9 , 22 ] , the drivers may be affected by complicated traffic or poor road design , resulting in traffic accidents even without misconduct . The complicated traffic and poor road design subsequently affect both the action of the drivers and the outcome . Therefore , it is unclear from the observational data whether the accidents are due to the actions adopted by the drivers . Agents trained with such observational data may be unwilling to take any actions under complicated traffic , jeopardizing the safety of passengers . In the example of personalized medicine [ 8 , 29 ] , the patients may not be compliant with prescriptions and instructions , which subsequently affects both the treatment and the outcome . As another example , the doctor may prescribe medicine to patients based on patients ’ socioeconomic status ( which could be inferred by the doctor through interacting with the patients ) . Meanwhile , socioeconomic status affects the patients ’ health condition and subsequently plays the role of the confounder . In both scenarios , such confounders may be unavailable due to privacy or ethical concerns . Such a confounding issue makes the observational data uninformative and even misleading for identifying and estimating the causal effect , which is crucial for decision-making in the online setting . In all the examples , it is unclear from the observational data whether the outcome is due to the actions adopted . • Even without the confounding issue , it remains unclear how the observational data may facilitate exploration in the online setting , which is the key to the sample efficiency of RL . At the core of exploration is uncertainty quantification . Specifically , quantifying the uncertainty that remains given the dataset collected up to the current step , including the observational data and the interventional data , allows us to construct a bonus . When incorporated into the reward , such a bonus encourages the agent to explore the less visited state-action pairs with more uncertainty . In particular , constructing such a bonus requires quantifying the amount of information carried over by the observational data from the offline setting , which also plays a key role in characterizing the regret , especially how much the observational data may facilitate reducing the regret . Uncertainty quantification becomes even more challenging when the observational data are confounded . Specifically , as the behavior policy depends on the confounders , there is a mismatch between the data generating processes in the offline setting and the online setting . As a result , it remains challenging to quantify how much information carried over from the offline setting is useful for the online setting , as the observational data are uninformative and even misleading due to the confounding issue . Contribution . To study causal reinforcement learning , we propose a class of Markov decision processes ( MDPs ) , namely confounded MDPs , which captures the data generating processes in both the offline setting and the online setting as well as their mismatch due to the confounding issue . In particular , we study two tractable cases of confounded MDPs in the episodic setting with linear function approximation [ 7 , 16 , 42 , 43 ] . • In the first case , the confounders are partially observed in the observational data . Assuming that an observed subset of the confounders satisfies the backdoor criterion [ 32 ] , we propose the deconfounded optimistic value iteration ( DOVI ) algorithm , which explicitly corrects for the confounding bias in the observational data using the backdoor adjustment . • In the second case , the confounders are unobserved in the observational data . Assuming that there exists an observed set of intermediate states that satisfies the frontdoor criterion [ 32 ] , we propose an extension of DOVI , namely DOVI+ , which explicitly corrects for the confounding bias in the observational data using the composition of two backdoor adjustments . We remark that DOVI+ follows the same principle of design as DOVI and defer the discussion of DOVI+ to §A . In both cases , the adjustments allow DOVI and DOVI+ to incorporate the observational data into the interventional data while bypassing the confounding issue . It further enables estimating the causal effect of a policy on the received rewards and the transition dynamics with enlarged effective sample size . Moreover , such adjustments allow us to construct the bonus based on a notion of information gain , which takes into account the amount of information carried over from the offline setting . In particular , we prove that DOVI and DOVI+ attain the ∆H · √ d3H3T -regret up to logarithmic factors , where d is the dimension of features , H is the length of each episode , and T = HK is the number of steps taken in the online setting , where K is the number of episodes . Here the multiplicative factor ∆H > 0 depends on d , H , and a notion of information gain that quantifies the amount of information obtained from the interventional data additionally when given the properly adjusted observational data . When the observational data are unavailable or uninformative upon the adjustments , ∆H is a logarithmic factor . Correspondingly , DOVI and DOVI+ attain the optimal√ T -regret achievable in the pure online setting [ 7 , 16 , 42 , 43 ] . When the observational data are sufficiently informative upon the adjustments , ∆H decreases towards zero as the effective sample size of the observational data increases , which quantifies how much the observational data may facilitate exploration in the online setting . Related Work . Our work is related to the study of causal bandit [ 20 ] . The goal of causal bandit is to obtain the optimal intervention in the online setting where the data generating process is described by a causal diagram . The previous study establishes causal bandit algorithms in the online setting [ 26 , 34 ] , the offline setting [ 17 , 18 ] , and a combination of both settings [ 11 ] . In contrast to this line of work , we study causal RL in a combination of the online setting and the offline setting . Causal RL is more challenging than causal bandit , which corresponds toH = 1 , as it involves the transition dynamics and is more challenging in exploration . See §B for a detailed literature review on causal bandit . Our work is related to the study of causal RL considered in various settings . [ 45 ] propose a modelbased RL algorithm that solves dynamic treatment regimes ( DTR ) , which involve a combination of the online setting and the offline setting . Their algorithm hinges on the analysis of sensitivity [ 3 , 27 , 38 , 44 ] , which constructs a set of feasible models of the transition dynamics based on the confounded observational data . Correspondingly , their algorithm achieves exploration by choosing an optimistic model of the transition dynamics from such a feasible set . In contrast , we propose a model-free RL algorithm , which achieves exploration through the bonus based on a notion of information gain . It is worth mentioning that the assumption of [ 45 ] is weaker than ours as theirs does not allow for identifying the causal effect . As a result of partial identification , the regret of their algorithm is the same as the regret in the pure online setting as T → +∞ . In contrast , our work instantiates the following framework in handling confounders for reinforcement learning . ( a ) First , we propose the estimation equation based on the observations , which identifies the causal effect of actions on the cumulative reward . ( b ) Second , we conduct point estimation and uncertainty quantification based on observations and the estimation equation . ( c ) Finally , we conduct exploration based on the uncertainty quantification and achieve the regret reduction in the online setting . Consequently , the regret of our algorithm is smaller than the regret in the pure online setting by a multiplicative factor for all T . [ 25 ] propose a model-based RL algorithm in a combination of the online setting and the offline setting . Their algorithm uses a variational autoencoder ( VAE ) for estimating a structural causal model ( SCM ) based on the confounded observational data . In particular , their algorithm utilizes the actor-critic algorithm to obtain the optimal policy in such an SCM . However , the regret of their algorithm remains unclear . [ 6 ] propose a model-based RL algorithm in the pure online setting that learns the optimal policy in a partially observable Markov decision process ( POMDP ) . The regret of their algorithm also remains unclear . [ 35 ] utilize generative adversarial reinforcement learning to reconstruct transition dynamics with confounder , and [ 40 ] propose a model-based approach for POMDP based on adjustment with proxy variables . [ 30 ] consider offpolicy policy evaluation under one-decision confounding and constructs worst-case bounds with theoretical guarantee . [ 4 ] utilizes states and actions as proxy variables to tackle off-policy policy evaluation with confounders . In contrast , our work utilizes backdoor and frontdoor adjustments to handle confounded observation .

Summary: The paper studies the problem of online reinforcement learning, while making use existing offline data that was previously logged by some behavior policy, where there is unobserved confounding in the logged data. They provide some conditions under which the RL model can be identified from the confounded logged data, and under these conditions propose a variant of optimistic value iteration for online learning that incorporates this offline data as a regularization term, which has an intuitive Bayesian interpretation based on conditioning on this data. Finally, they provide regret bounds for their algorithm, which has a leading term incorporating the effect of the logged data, which decays to zero as the amount of logged data grows to infinity.