Causal Reinforcement Learning using Observational and Interventional Data
Authors: Maxime Gasse
Damien GRASSET
Guillaume Gaudron
Pierre-Yves Oudeyer
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
causality
confounding

Abstract: Learning efficiently a causal model of the environment is a key challenge of model-based RL agents operating in POMDPs. We consider here a scenario where the learning agent has the ability to collect online experiences through direct interactions with the environment (interventional data), but also has access to a large collection of offline experiences, obtained by observing another agent interacting with the environment (observational data). A key ingredient, which makes this situation non-trivial, is that we allow the observed agent to act based on privileged information, hidden from the learning agent. We then ask the following questions: can the online and offline experiences be safely combined for learning a causal transition model ? And can we expect the offline experiences to improve the agent's performances ? To answer these, first we bridge the fields of reinforcement learning and causality, by importing ideas from the well-established causal framework of do-calculus, and expressing model-based reinforcement learning as a causal inference problem. Second, we propose a general yet simple methodology for safely leveraging offline data during learning. In a nutshell, our method relies on learning a latent-based causal transition model that explains both the interventional and observational regimes, and then inferring the standard POMDP transition model via deconfounding using the recovered latent variable. We prove our method is correct and efficient in the sense that it attains better generalization guarantees due to the offline data (in the asymptotic case), and we assess its effectiveness empirically on a series of synthetic toy problems.

Under review as a conference paper at ICLR 2022
CAUSAL REINFORCEMENT LEARNING USING OBSERVATIONAL AND INTERVENTIONAL DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Learning efficiently a causal model of the environment is a key challenge of model-based RL agents operating in POMDPs. We consider here a scenario where the learning agent has the ability to collect online experiences through direct interactions with the environment (interventional data), but also has access a large collection of offline experiences, obtained by observing another agent interacting with the environment (ohservational data). A key ingredient, which makes this situation non-trivial, is that we allow the observed agent to act based on privileged information, hidden from the learning agent. We then ask the following questions: can the online and offline experiences be safely combined for learning a causal transition model? ? And can we expect the offline experiences to improve the agent's s performances ? To answer these, first bridge the fields of reinforcement learning and causality, by importing ideas from the well-established causal framework of do-calculus, and expressing model-based reinforcement learning as causal inference problem. Second, we propose a general yet simple methodology for safely leveraging offline data during learning. In nutshell, our method relies on learning a latent-based causal transition model that explains both the interventional and observational regimes, and then inferring the standard POMDP transition model via deconfounding using the recovered latent variable. We prove our method is correct and efficient in the sense that attains better generalization guarantees due to the offline data (in the asymptotic case), and we assess its effectiveness empirically on a series of synthetic toy problems.
INTRODUCTION
As human beings, key ingredient in our learning process is experimentation: we perform actions in our environment and we measure their outcomes. Another ingredient, maybe less understood, is observation: we observe the behaviour of others acting and evolving in the environment, be it people, animals, or even plants. It is well-known that observation alone is not sufficient to infer how our environment works, or more precisely to predict the outcome of our actions, especially when the behaviours we observe depend on hidden information]] And yet whole field of science, astronomy, heavily relies on the observation of celestial bodies in the sky, on which experimentation is virtually impossible. So which role exactly does observation play during learning ? And in particular, how do we combine observation and experimentation
In the context of reinforcement learning (RL), a related question is the combination of offline data, resulting from observations, with online data resulting from experimentation, in order to improve the performance of a learning agent. In the Markov Decision Process (MDP) setting, where the agent observes the entire state of the environment, the answer is straightforward and practical solutions exist, leading to the fastly growing field of offline reinforeement learning [17 where large databases of demonstrations can be efficiently leveraged. In the more general Partially-Observahh setting however, the question turns out be much more challenging. A typical example is in the context of medicine, where offline data is collected from physicians who may rely on information absent from their patient's medical records, such as their wealthiness or their lifestyle. Suppose that wealthy patients in general get prescribed specific treatments by their physicians, because they can
MDP (POMDP)
'Simply put, correlation does not imply causation. Or, citing Pearl 24. "behind every causal conclusion there must lie some causal assumption that is not testable observational studies".

---

Under review as a conference paper at ICLR 2022
afford it, while being less at risk to develop severe conditions regardless of their treatment, because they can also afford a healthier lifestyle. This creates a spurious correlation called confounding, and will cause naive recommender system to wrongly infer that a treatment has positive health effects. A second example is in the context of autonomous driving, where offline data collected from human drivers who have wider field of vision than the camera on which the rohot driver relies. Suppose human drivers push the brakes when they see a person waiting to cross the street, and only when the person walks in front of the car enters the camera's 's field of vision. Then, again, naive robot might wrongly infer from its observations that whenever brakes are pushed, a person appears in front of the car. Suppose now that the robot's objective is to avoid collisions with pedestrians, they it might get regrettably reluctant to push the brakes. Of course, in both those situations, the learning agent will eventually infer the right causal effects of its actions if collects enough online data from its own interactions. However, in both those situations also, performing many interventions for the sole purpose of seeing what happens is not really realistic, while collecting offline data by observing the behaviour of human agents is much more affordablc.
In this paper we study the question of combining offline and online data under the Partially-Observabbe Markov Decision Process (POMDP) setting. by importing tools and ideas from the well-established field of causality 23 into the model-based RL framework. Our contribution is three-fold:
1. We formalize model-based RL as a causal inference problem using the framework of do- calculus 225 which allows us to reason formally about online and offline scenarios in an intuitive manner (Section 3)
2. We present a generic method for combining offline and online data in model-based RL (Section 4. with a formal proof of correctness even when the offline policy relies on privileged hidden information (confounding variable), and proof of efficiency in the asymptotic case (with respect to using online data only).
3. We propose a practical implementation of our method, and illustrate its effectiveness on three experiments with synthetic toy problems (Section ())
While our proposed method can be formulated outside of the framework of do-calculus, in this paper we hope to demonstrate that it offers a principled and intuitive tool to reason about model- based RL. By relating common concepts from RL and causality, wish that our contribution will ultimately help to bridge the gap between the two communities.
2
BACKGROUND
2.1 NOTATION
In this paper, upper-case letters in italics denote random variables (e.g. X, Y), while their lower-case counterpart denote their value (c.g. y) and their calligraphic counterpart their domain (e.g.. E X). We consider only discrete random variables. To keep our notation uncluttered, with slight abuse of notations and use p(x) to denote sometimes the event probability p(X = r), and sometimes the whole probability distribution of X, which should be clear from the context. In the context of sequential models we also distinguish random variables with a temporal index f, which might be fixed (e.g., op, 01 ), or undefined (e.g., p( 18,, ar) denotes at the same time the distributions p(s) (30, an) and p(s2|s1, )). We also adopt compact notation for sequences of contiguous variables (e.g.. 80-T = (Sp. (S,,..., ST) E ST+1), ), and for summations over sets (Erex - Σx"). ). We assume the reader is familiar with the concepts of conditional independence (X TLL Y Z) and probabilistic graphical models based on directed acyclic graphs (DAGs), which can be found in most introductory textbooks, e.g. Pearl [22]: Studeny (2);: Koller and Friedman
[15]. In the following we will use do-calculus to derive formal solutions to model-based RL in various POMDP settings. We refer the reader to Pearl (25 for thorough introduction, and give a description of rules R1, R2 and R3 used in our derivations in the appendix (Section A.T)
2.2 PARTIALIY-OBSERVABLL MARKOV DECISION PROCESS
We consider Partially- Partially-Obserte Observable Markov Decision Processes (POMDPs) of the form M
(8, O, A, Piniry Pobs: Prrans; r), with hidden states S, observations O, actions a A, ini-
2

---

Under review as a conference paper at ICLR 2022
tial state distribution (s)) state transition distribution Prams 8t, at), observation distribution Pots (org (t), and reward function r O - R. For simplicity we assume episodic tasks with finite horizon H. We further denote a complete trajectory
(Oo. ag..... . 05r). and for convenience introduce the concept of a history at time I, h4 = (oo; ao.....0z) 0z).
common control scenario for POMDPs is when actions decided based on all the available information from the past. We call this the standard POMDP setting. The control mechanism can be represented a stochastic policy π (aa//t [he,, which together with the POMDP dynamies piniss Pobs and Pirass defines a probability distribution over trajectories T,
|-|-1
Pata(T (T) (8nr((8)) Pouu( so)
II 2=
Prrane 188, Of Pr (Ot4 (st+1)-
This whole data-generation mechanism can be represented visually as DAG, represented Figure A key characteristic in this setting is that A, 11 S, H, is always true, that is, every action independent of the current state given the history.
2.3 MODEL- BASED RL
Assuming the objective is the long-term reward, the POMDP control problem formulates as:
π arg max E
Σ r(o.)
(1)
Troplod
2m0
Model- RL relies on the estimation of the POMDP transition model PatlOot11ha az) to solve which decomposes into two sub uub-problems:
1. learning: given a dataset D, estimate transition model @(0++1 [he, ae) Peaa(o [he, az);
2. planning: given history h2 and transition model decide optimal action de-
As will see shortly, the transition model seeked by model-based RL is inherently causal [9] In this work we consider only the first problem above, that is, learning the (causal) POMDP transition model from data.
MODEL-BASED RL AS CAUSAL INFERENCE
Decision problems, such as those arising in POMDPs, can naturally be formulated in terms causal queries where actions directly translate into do statements. For example, given past information about the POMDP process, what will be the causal effect of an action (intervention) on future rewards
3.1 THE INTERVENTIONAL REGIME
In the interventional regime, we assume a dataset Dim of episodes collected the standard POMDP setting from arbitrary decision policy л (a,)hh)
Din Pinits Piransy Pobsy
Let us now adopt causal perspective and reason in terms of interventions in the causal system, depicted in Figure [ Consider that we want to control the system, that is, replace æ with x*, in order to maximize a long-term outcome. Then, evaluating the effect of each action the system is causal inference problem. In order to decide on the best first action ag given by = (oo). one must evaluate series of causal queries in the form Pau(O1 Joo, do(ao)), then Par(Oz op, do(av), 01, do(a1)) )), and so on, and finally using those causal distributions for planning by solving a Bellman equation. Conveniently, in the interventional regime, applying rule R2 of do-calculus the causal DAG results
Without loss of generality we consider the reward to be part of the observation o, to simplify our notation. A guiding example accompanying this section can he found in the appendix (Section A.3
3

---

Under review as a conference paper at ICLR 2022
Proos
Pinnze
Pinii
Pirans
Piuax
S₁
Se+1
Sg
S
Sp+1
Polr
Polni
Pobs
Pola
Pods
Pols
Ou
1
O₂
A
Ot+1
Ou
Ac-1
O2
A
Oc+1
Epro
Figure I: Standard POMDP setting.
Figure 2: Privileged POMDP setting.
in those queries being trivially identifiable from (T)( In fact, those queries exactly boil down to the standard POMDP transition model that model-based RL seeks to estimate,
Pad(oc41 do(ao Psu( (4))
(2)
As such, model-based RL can be naturally reinterpreted in terms of causal inference. Also, convenient property in this regime is that Psus(0++)) //, az) does not depend on the control policy that was used to build the dataset Din- The only requirement, in order estimate transition probabilities for every h,, a combination, is that π has a non-zero chance to explore every action, that is, n7 (Tee (h:) > 0, Var, hts Then, an unbiased estimate of the standard POMDP transition model can be obtained simply via log-likelihood maximization:
|r|-1
arg max JEQ
Σ
log oegl(oii/lee ae).
In some situations it is very reasonable assume an interventional regime, for example when is known to hold by construction. This is the case with online RL data, as the learning agent itself explicitly controls the data-cola data-collection policy x(a h2). But can also be the case with offline RL data, if one knows that the data cola-collection policy did not use any additional information besides the information available to the learning agent, h4, In Atari video games for example, is hard to imagine human player using any kind of privileged information related to the machine's 's internal state 8, other than the video and audio outputs from the game.
3.2 THE OBSERVATIONAL REGIME
In the observational regime, assume dataset Dats of episodes collected in the privileged POMDP setting, depicted in Figure 2 In this setting episodes are collected from an external agent who has access to privileged information, in the extreme case the whole POMDP state 8t, which the learning agent can not observe In this setting we denote the data-generating control policy Tpru (a+) /r, St), such that
Dobs ~ Pinit Prrans: Pobas Tprv-
We denote the whole episode distribution resulting from Pinit: Pirans Pobs and Tpro as Per. Ppr-(T).. ). A key characteristic in this setting is that now A; IL S. H2 can not be assumed to hold any more.
Let us reason here again in terms of causal inference from the causal system depicted in Figure For the purpose of controlling the POMDP in the standard setting, in the light of past information h2 we want evaluate the same series of causal queries as before, the form Ppre Ppr (00 do do)a This time however, those causal queries are not identifiable from Pprr (T). Evaluating them would require knowledge of the POMDP hidden states 8t, which act as confounding variables. For example, identifying the first query at t requires at least the observation of so.
Pprr do(a)))
Σ
(suloo, do (Pprr |su. Og. do(as))
Σ S
(s0 (0:0ssss000 ao)
'Note that our only assumption is that this external agent has access to privileged information. We do not. assume optimally with respect to the learning agent's 's reward, or any other reward.
4

---

Under review as a conference paper at ICLR 2022
Puut
Prosex
Prrany
So
S
St+1
Potu
Pabr
Pohr
Oo
A
O₂
A
Oc+1
Figure Augmented POMDP setting, with policy regime indicator taking values in {0, 1} (I=interventional regime, no confounding, O=observational regime, potential confounding), such that (a Ih, St, = 1) <a, lh, = 1). This additional constraint introduces a contextual independence A 11 S. H, 1.
(R3 and R2 of do-calculus, then Oc+1 IL He St, Ad).
In many offline RL situations, believe that it is common to have access to POMDP trajectories for which A, 1T St H2 can not be assumed, for example when demonstrations are collected from a human agent acting the world (see Section for examples). In such a situation, the observed trajectories may be confounded, and naively learning a causal transition model by solving (3) might result in non-causal model, and non-optimal planning. A natural question is then: what can be done in such situation Are confounded trajectories useless ? Is there still way to this data ?
4
COMBINING OBSERVATIONAL AND INTERVENTIONAL DATA
4.1
PROBLEM STATEMENT
We consider generic situation where two datasets of POMDP trajectories Dinr and Poby are available. sampled respectively in the interventional regime with policy (aa(a)))t), and in the observational (potentially confounded) regime with policy xpr (at 8t). We then ask the following question: is there sound way to use the observational data for improving the estimator of the standard POMDP transition model that would be recovered from the interventional data only ?
4.2 THE AUGMENTED POMDP
We formulate the problem of learning the standard POMDP transition model from Din and Dabe as that inferring a structured vatent-varialle model. Since both datasets are sampled from the same POMDP (Pinit, Pirans and Pobz) controlled in different ways (either Ter auw), the overall data generating process can be represented in the form of an augmented DAG, depicted in Figure 3 We simply introduce an auxiliary variable / {0, 1} that acts a regime indicator (5), for differentiating between observational and interventional data. The augmented POMDP policy then simply becomes π, where T(a St. 0) (apl(a sx) and TTaelhe St, 1) (a (h.).
For simplicity, in the following we will refer to the joint distribution of this augmented POMDP the true distribution p. and with a slight abuse of notation we will consider Dois and D.., two datasets of augmented POMDP trajectories, sampled respectively under the observational regime (r, i) p(T, 0), and the interventional regime (r. p(r, i 1). The causal queries required to control the augmented POMDP can then be identified as
p(0r+1100-sx do()g-.:) = p(0++ 100- do(ag-st)) p(o 1 [ a,
(R1 of do-calculus, then R2 the contextual causal DAG from Figure
4.3 THE AUGMENTED LEARNING PROBLEM
In order to learn the standard POMDP transition model p(o [h, a,, i = 1) from the augmented dataset Daat, U D D p(T, i), propose the following two-step procedure.
5

---

Under review as a conference paper at ICLR 2022
Learning In the first step, fit a latent probabilistic model q to the training trajectories, constrained to respect all the independencies of our augmented POMDP. To do so we substitute the actual POMDP hidden state 8, S by latent variable Z, with Z the discrete latent space of the model. Our learning problem then formulates as standard likelihood maximizatior i.e.,
arg max
(og
(4)
(T.))
with Q the family of sequential latent probabilistic models that respect
21rl+1
17|-1 g(apphe Fes ())(()))(()))))))))))))))))))))) (0000441))4)))
E ((20)g(00120)
II
g(ae)he, 21, == = g(a, he, 1).
Inference In the second step, recover ğ(o (i, dz 1) as an estimator of the standard POMDP transition model. This can be done efficiently with a forward algorithm over the augmented DAG structure which unrolls over time as the RL agent evolves in the environment.
Intuitively, the observational data Dobs influences the interventional transition model q(ot+1lh,, at, = 1) as follows. The learned model q must fit the observational and interven- tional data by sharing the same building blocs ((z),, g( (2t) and q 0z), while only the expert policy q(a,/ht, 2,, 0) offers some flexibility that allows differentiate between both regimes. As result, imposing observational distribution g(rli 0) acts regularizer for the interventional distribution q(7 1).
4.4 THFORETICAL GUARANTEES
In this section we show that our two-step approach is 1) correct, in the sense that yields an unbiased estimator of the standard POMDP causal transition model and efficient, in the sense that it yields a better estimator than the one based on interventional data only (asymptotically in the number of observational data). All proofs are deferred to the appendix (Section A.7).
First we show that the recovered estimator is unbiased, and then we derive bounds for @(ot11/h, a,, i 1) in the asymptotic observational scenario. [Dobs] + oo (regardless of the interventional data Din).
[S], @(o++1 a,,,,, a,, = 1) is an unbiased estimator of Proposition 1. Assuming |Z| p(or41 [h, de, =1.
2
Theorem 1. Assuming [Daual + for any Din the recovered causal model is bounded as follows:
T-1
T
II ((oc (b,, az, 1=,;=11) 1) 2
II p(arthe, 0)p(0r)))))00) the Or, i = 0), and
T-1
T-1
@(or+1 az i= 1) < t=0
II
II
plarlhr, 0)p(or+1 ae, i= = +1 II t=0
plaethe, 0),
t=
Vhr-1,0T-1, T where p(hr-1, ar >
As direct consequence, in the asymptotic case, using observational data ensures stronger generaliza- tion guarantees for the recovered transition model than using no observational data.
The estimator q(ot+1lhu a,, = 1) recovered after solving ( with |Dotel + do offers strictly better generalization guarantees than the with |Datel 0, for any Din-
Corollary 1.
Note that, while the problem of learning structured latent variable models known to be hard in general, there also exists a wide range tools and algorithms available in the literature for solving it approximately, such as the EM algorithm or the method of ELBO maximization. "ee appendix for details (Section A.4

---

Under review as a conference paper at ICLR 2022
5 RELATED WORK
whole body of work exists around the question of merging interventional and observational data in RL, with related results already in econometrics [20]. Bareinboim et al. 2 study a sequential decision problem similar to ours, but assume that expert intentions are observed both in the interventional and the observational regimes, i..., prior to doing interventions the learning agent can ask "what would the expert do in my situation ?" This introduces an intermediate, observed variable à = f(x)) with the property that (0r â. (ā.) 1, which guarantees unconfoundedness in the observational regime (A, S. H)), so that observational data can be considered interventional, and the standard PO-MDP transition model can be directly estimated via 3.. Zhang and Bareinboim [31 34 relax this assumption in the context of binary bandits, and later on in the more general context of dynamic treatment regimes B22 33 They derive causal bounds similar to ours (Theorem [ and propose two-step approach which first extracts causal bounds from observational data, and then uses these bounds in an online RL algorithm. While their method nicely tackles the question of leveraging observational data for online exploration, it does not account for uncertainty in the bounds estimated from the observational data. In comparison, our latent-based approach is more flexible, as never computes explicit bounds, but rather lets the learning agent decide through (4) how data from both regimes influence the final transition model, depending of the number of samples available. Kallus et al. [3 also propose a two-step learning procedure to combine observational and interventional data in the context of binary contextual bandits. Their method however relies on a series of strong parametric assumptions (strong one-way overlap, linearity, non-singularity etc.).
specific instantiation of our framework is off-policy evaluation, i.e., estimating the performance of a policy using observational data only. This corresponds to the specific setting [Dmel = 0, where it can be shown that the causal transition model is in general not recoverable in the presence of confounding variables. Still, growing body of literature studies the question under specific structural or parametric assumptions [19 30 B]. In the context of imitation learning, de Haan et al. (6] attribute the issue of causal misidentifiiaatinn that is, ascribing the actions of agent the wrong explanatory variables, to confounding. We argue that this explanation is erroneous, since their imitated experts are trained in the standard POMDP setting (interventional regime). This reasoning supports Spencer et al. (88). who shows that causal misideniification is simply a manifestation of covariate shift. Finally, other issues orthogonal confounding can appear when combining online and offline data RL, for example the value function initialisation problem [8] or the bootstrapping error problem [6 21
6 EXPERIMENTS
We perform experiments on three synthetic toy problems, each one expressing a different level of complexity and a different form of hidden information.
Door In this toy problem, we consider a closed door, and a light (red or green) indicating which of two buttons (A and B) should be pressed to open the door. The privileged agent perceives the color of the light, while the learning agent doesn't 't (colorblind). This corresponds to simple binary bandit, with time horizon H- = 1 and a hidden state space [S] 33
Tiger In this classical problem from the literature 4 the agent stands in front of two doors, one with a treasure behind (+10 reward) and one with a tiger behind (-100 reward). At each time step the agent can either open one of the doors, or listen (-1 reward) to obtain a noisy estimate of the tiger's position. The privileged agent has full knowledge of the tiger's 's position, while the learning agent doesn't 't. This toy problem is small-scale POMDP, with a time horizon H- 50 and a hidden state space |S| = 6.
This problem is inspired from Alt et al. [. Here the agent starts on the top-left corner of a small 5x5 grid, and tries to get to target placed on the bottom side behind a large wall. The agent can use five actions: top, right, bottom, left and idle, and moves into the desired direction with 50% chances, or randomly remains in the current tile or slips to one of the 4 adjacent tiles otherwise. The privileged agent has full knowledge of its position at each time step, while the learning agent is only revealed this information once in a while, with 20% chances. This toy problem constitutes a more challenging POMDP, with time horizon H 20 and a hidden state space |S] 42.
Gridworld
For each toy problem, we train and evaluate our proposed approach, augmented, using a large amount of observational data Dans (512 samples for door, 8192 for liger and gridworld), and interventional
7

---

Under review as a conference paper at ICLR 2022
data iier of varying size, collected from random explorations. Each time, we compare approach to two baseline methods: no obs, where the Dot is not used at all during training, and naive, where Dobr is naively combined with Diu if there experiment we also report the performance of Kallus et al. [3] (only setting in which it applies). We repeat each experiment 20 random seeds. In the following report main results, and defer the reader to the appendix (Section A.5) for the complete experimental details and results.
confounding. As reference, in the door
6.1
PERFORMANCE OF THE RL AGENTS
door
tiger
gridworld
200
0.60
1.00
no obs nalve
0.55
200
augmonted
Kallus al.
400
0.50
6600
2
2
2
2
2
Iloa scalo)
nints (log scale)
nints (log Scall)
Figure 4: Performance each RL agent on our three toy problems (the higher the better). We report the average cumulative reward (mean + std) obtained on the real environment. Little markers indicate the significance of a two-sided Wilcoxon signed-rank test 7 with < 5%, between our method, augmented, and the baselines no obs (down triangles), naive (squares) and Kallus et al. (up triangles).
In Figure 4 we report the test performance of an RL agent trained on the transition models recovered by each method. Here privileged policy Tpr. consists good but imperfect expert in both door and tiger, and shortest-path expert in gridworld. In all three toy problems, our method successfully leverages the confounded observational data and outperforms the two baseline methods, especially in the law-sample regime (few interventional samples). Most noticeably, the obs baseline converges to the same performance as our method, if given enough interventional samples, while the naive baseline seems suffer from the additional observational data, and converges much slower than the two other methods. As a reference, our approach also performs much better than Kallus et al. 13
6.2 ROBUSTNESS TO DIFFERENT DEGREES OF CONFOUNDING
noisy good expert
perfectly good expert
perfectly bad expert
0.15
0.20
0.20
no obs
naive
0.15
0.15
sugmented
0.10
0.10
0.05
0.05
0.05
0.00
2
2
0.00
(log scale!
mints (log
(log scale!
random expert
positively biased expert
negatively biased expert
0.15
0.15
0.10
0.10
0.05
0.05
0.05
25 scale)
0.00
29
21
27
0.00
nints
Figure 5: Robustness to different degrees of confounding in the door problem (the lower the better). We report the JS divergence (mean + std) between the true and the recovered transition model.
"odee ht/p:p:://gggtt....mm////sslsll
8

---

Under review as a conference paper at ICLR 2022
Our approach is robust any kind of confounding, and does not assumee the observed expert the privileged information in any specific way, or performs well or poorly at the task at hand. To empirically demonstrate this claim, we repeat the door experiment with various expert behaviours, including perfectly good / bad (always never press the correct button), positively negatively biased (overly optimistic pessimistic towards the optimal button), and random control (no confounding). The outcome of this experiment is showcased in Figure where our method always results in the best estimate of the transition model (except in the situation with no confounding, where the naive approach is slightly more effective). In the appendix (Section (A.6.1) we also report gains in terms of reward, except when there is no confounding or when the confounding induces a positive bias, in which case the naive approach performs slightly better.
6.3
FOCUS ON THE GRIDWORLD PROBLEM
JS divergence
reward
1.00
0.5
по abs
0.75
nalve
X
augmented
augmented
0.50
0.2
0.25
0.00
2
27
2
235
27
212
2
nints Hlog Scale)
nints (log scale)
DO obs
naive
the JS divergence and cumulative reward obtained by each method. Right: the initial grid, and a heatmap of the tiles visited by the RI. agents at test time the (Daul 2 mark. At this point, only the augmented method has learned how to pass the wall.
Figure 6: The gridworld experiment. Left:
most complex problem, the gridworld problem, showcased in Figure 6 Our method, augmented, starts obtaining positive rewards with 2 = 128 interventional samples, which is two orders of magnitude better than the two other methods, where positive rewards appear later on at 29 = 512 samples. This impact is clearly noticeable if one looks at the test-time trajectories of the RL agents at the 2 mark (Figure right side). While the transition models learned by the no obs and naive approaches result in the agent being stuck around its starting position, the transition model learned by our approach already allows the agent learn how to pass the wall and reach the bottom side of the grid.
Let us now have particular focus
DISCUSSIONS
In this paper we have presented simple, generic method for combining interventional and observa- tional (potentially confounded) data in model-based reinforcement learning for POMDPs. We have demonstrated that method is correct and efficient in the asymptotic case (infinite observational data), and we have illustrated its effectiveness on three synthetic toy problems. One limitation of our method is that it adds additional challenge top of model-based RL, that of learning a latent-based transition model, which can become problematic in high-dimensional RL settings.. Still, the recent success of discrete latent models for solving complex RL tasks [0] tasks in high-dimenssonal domains [26] lets us envision that this difficulty can be overcome in practice. A first potential extension our work could be use offline data to guide online exploration,
33 34 A second direct extension is to consider that several agents are observed, each with its own privileged policy, leading to multiple observational regimes. This would lead, in the asymptotic case, to a stronger implicit regularizer for the causal transition model. third, obvious extension to develop a similar approach for model-free RL, maybe in the form of value-function regularizer. A fourth direction is to apply the same approach to POMDPs with continuous observation spaces (e.g., pixel-basee problems), which is theoretically very straightforward. Finally, hope that our work will help bridge the gap between the RL and causality communities, and will convince the RL community that causality is an adequate tool to reason about observational data, which is plentiful in the world.
fashion similar to Zhang and Bareinboim BT 32

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Confounding is prevalent issue in human-generated data, and can be an important source of bias in the design of decision policies, if not dealt with properly. This paper, although theoretical, makes a humble step towards more robustness and fairness in Al-based decision systems, by combining causality and statistical learning to address the confounding problem. As such this work has potentially important societal implications, in particular in critical systems where lives are at stake such as medicine or self-driving cars, where -eenanngeeeereed data is prevalent.
REPRODUCIBILITY STATEMENT
Our notations, our POMDP and our causal frameworks are explicitly introduced in Sections 2 and Our problem statement clearly laid down in Section[[Jbefon contribution, and the proofs all our theoretical results are presented the appendix, Section A.7 Our experimental setting is described briefly in the main body of the paper, Section and in details in the appendix, Section A.3 The experimental results presented the paper are reproducible, as both the workflow (scripts, parameters and seeds), and the source code made publicly available alongside the paper.
present contribution
REFERENCES
[1] Bastian Alt, Matthias Schultheis, and Heinz Koeppl. Pomdps in continuous time and discrete spaces. In H. Larochelle, M. Ranzato, Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 13151-13162. Curran Associates, Inc., 2020.
[2] Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: A causal approach. In NIPS, 2015.
[3] Andrew Bennett, Nathan Kallus, Lihong Li, and Ali Mousavi. Off Off-policy evaluation in infinite- horizon reinforcement learning with latent confounders. In AISTATS, 2021.
[4] Anthony R. Cassandra, Leslie P Kaelbling, and Michael L. Littman. Acting optimally in partially observable stochastic domains. In AAAI, 1994.
[5] A. Philip Dawid. tecosiin- foundations for statistical causality, 2020.
[6] Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In NeurlPS, 2019.
[7] Janez Demsar. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res., 7:1-30, 2006. URL te. http://b/rouumaalmm/m/m///m/aa// html #Emlsaresa
[8] Sylvain Gelly and David Silver. Combining online and offline knowledge in uct. In ICML, 2007.
[9] Samuel J Gershman. Reinforcement learning and causal models. In Michael R. Waldmann, editor, The Oxford handbook causal reasoning, chapter 10, pages 295-306. Oxford University Press, 2017.
[10] Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In /CLR, 2021.
[11] Yimin Huang and Marco Valtorta. Pearl's calculus of intervention is complete. In UAI, 2006.
[12) Guido W. Imbens and Donald B. Rubin. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press, 2015.
[13] Nathan Kallus, Aablad Manas Puli, and Uri Shalit. Removing hidden confounding by experi- mental grounding. In NewrlPS, 2018.
10

---

Under review as a conference paper at ICLR 2022
[14] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[15] Daphne Koller and Nir Friedman. Probabilistic Graphical Models Principles and Techniques. MIT Press, 2009.
[16] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In NeurIPS, 2019.
[17] Sascha Lange. Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Reinforcement Learning, volume 12 of Adaptation, Learning, and Optimization, pages 45-73. Springer, 2012.
[18] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint, 2020.
[19] Chaochao Lu, Bernhard Schölkopf, and José Miguel Hermández-Lobar Deconfounding reinforcement learning in observational settings. arXiv preprint, 2018.
[20] Charles Manski. Nomparametric bounds on treatment effects. The American Economic Review, 80(2):319-325 3333333,,,3,, 1990.
[21] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets, 2021.
[22] Judea Pearl. Probabilistic reasoning in intelligent systems networks of plausible inference. Morgan Kaufmann, 1989.
[23] Judea Pearl.
Causality. Cambridge University Press, Cambridge, UK, 2 edition, 2009.
[24] Judea Pearl.
Causal inference in statistics: An overview. Statistics Surveys, 3:96 146, 2009.
[25] Judea Pearl. The do-calculus revisited. In UAI, 2012.
[26] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint, 2021.
[27] Ilya Shpitser and Judea Pearl. Identification of joint interventional distributions in recursive semi-markovian causal models. In AAAI, 2006.
[28] Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J. Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift, 2021.
[29] Milan Studeny. Probabilistic Conditional Independence Structures. Springer, 2005.
[30] Guy Tennenholtz, Uri Shalit, and Shie Mannor. Off-policy evaluation in partially observable environments. In AAAI, 2020.
[31] Junzhe Zhang and Elias Bareinboim. Transfer learning in multi-armed bandits: A causal approach. In IJCAI, 2017.
[32] Junzhe Zhang and Elias Bareinboim. Near-optimal reinforcement learning in dynamic treatment regimes. In NeurIPS, 2019.
[33] Junzhe Zhang and Elias Bareinboim. Designing optimal dynamic treatment regimes: A causal reinforcement learning approach. In Hal Daumé III and Aarti Singh, editors, ICML, volume 119 of Proceedings of Machine Learning Research, 2020.
[34] Junzhe Zhang and Elias Bareinboim. Bounding causal effects continuous outcomes. In AAAI, 2021.
11

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.I DO-CALCULUS
Several frameworks exist in the literature for reasoning about causality 23 12]. Here we follow the framework of Judea Pearl, whose concept of ladder of causation is particularly relevant to answer RI. questions. The first level of the ladder, association, relates to the observation an external agent acting in the environment, while the second level, intervention, relates the question of what will happen to the environment as a result of one's ' own actions, The tool of do-calculus [25] acts as bridge between these two levels, and relates interventional distributions, such p(y (o))) to observational distributions, such as p(y[z), In nutshell, do-calculus allows for measuring changes in the distribution of random variables {X,Y Z. Z........ when one performs an arbitrary intervention do(z) which forces some variables to take values X x regardless of their causal ancestors. It relies complete of rules [1] 27) which allow for the following equivalences when specific structural conditions are met the causal DAG:
causal systems that can be expressed as DAGs.
RI: insertion/deletion observations p(y]do(x), 2, w) = p(yldo(z), w),
R2: action/observation exchange p(y)do(x), do(=), w) p(y]do(zr),
R3: insertion/deletion of actions doy)do(x) do(z), w) p(uldo(x), w).
We refer the reader to Pearl
25 for thorough introduction do-caleulus.
A.2 ABOUT IGNORABILITY AND EXOGENEITY
In this paper we discuss and use great length the concept of confounding, which is idea in Judea Pearl's do-calculus framework. For readers who are more familiar with the framework of potential outcomes from Donald Rubin [12]. the concept of confounding closely relates the concepts of ignorability and exogeneity. Indeed, both those concepts are shown to be equivalent to the unconfoundedness (no confounding) assumption in 23
A.3 GUIDING EXAMPLE: THE DOOR PROBLEM
This section introduces the door problem, and constitutes a guiding example that is meant to accompany the paper.
The door problem Consider a door, a light, and two buttons A and B. The light red 60% of the time, and green the rest of the time. When the light is red, button A opens the door, while when the light is green, then button B opens the door. I am told that the mechanism responsible for opening the door depends both the light color and the button pressed (light - door button), but I am given the mechanism itself. Suppose now that am colorblind, and want to open the door. Which button should press? ? In the do-caleulus framework, the question I am asking
arg max piddor=opem//dobutton)).. buttonE: (A.B)
Interventional regime If am able to observe myself or another colorblind person interacting with the door, then I know that which button is pressed is unrelated to which color the light is (light A button). Then can directly estimate the causal effect of the button on the door,
pldoor==openldo((utonn)) p(door=open button)).
Whichever policy is used to collect (button, door) samples eventually I realise that button A has more chances of opening the door (60%) than button B (40%). and thus the optimal choice.
One assumption though is strict positivity, (bbutton) Vbutton, that both buttons pressed.
12

---

Under review as a conference paper at ICLR 2022
Observational regime Assume now that observe another person interacting with the door. I do not know whether that person is colorblind or not (light + button is possible). Then, without further knowledge, cannot recover the causal queries p(door=opent do (button)) from the observed distribution p(door, button). In the do-calculus framework, the queries are said non identifiabe. However, if that person was to tell me the light color they see before they press A or B, then I could recover those queries as follows,
p(door=apenndo((uuto)))
E lightE (red.green)
pliighppldooozzopenliiggg button).
This formula, called deconfounding, eventually yiclds the correct causal transition probabilities regardless the observed policy given that enough (light, button, door) samples are observed.
door example in light of Theorem Assume this time that observe many (button, door) interactions from a non-colorblind person 0), who's policy i (bbuton==/light-rre)
Merging interventional and observational data Let us now look
0.9 and (button=Allighi==grren 0.4. Then I can already infer from Theorem [ that pidoor=open]dol do (o(bttton=A)) E [0.54, 0.84] and p(door=open do (bb E [0.24, 0.94]. now get chance interact with the door (i = 1), and I decide to press A 10 times and B 10 times. I am unlucky, and my inter- [[door=open)doo(btterr)
= 0.5 and = 0.5. This does not coincide with my (reliable) observational study, and therefore I adjust q( q(dooropenn do (d(bttton=A)) to its lower bound 0.54. now believe that pressing A is more likely to be my optimal strategy.
ventional study results the following probabilities: (orp://ppnnoc((ttoooooso
A.4 RECOVERING THE STANDARD POMDP TRANSITION MODEL.
Recovering ((0441//4 az, i 1) can be done as follows:
g(o++1/he, a = 1) Σ Aielhi i= Σ g( (1
The second and third terms are readily available as components of the augmented POMDP model while the first term can be recovered by unrolling forward algorithm over the augmented DAG structure, First, we have
(a opli 1) ((z))
g(20, ooli 1) (2ol/oo 1)
Σ 0000
ooli
Then, for every i' from to
2
((#411, 0041 his ae, i = 1) = Σ
((rr 1)0( (0)0000
((zv41, 1 ov, 1)
((e+ilhe+1, = 1)
0(2r41, or ar, i = 1)
The strict positivity condition here क (button light)
Vbutton, light.
13

---

Under review as a conference paper at ICLR 2022
A.5 EXPERIMENTAL DETAILS
The code for reproducing our experiments is made available online
We perform experiments three synthetic toy problems: the door problem described earlier (Section A.3). , the classical liger problem from the literature 4 and a 5x5 gridworld problem inspired from Alt et al. [
Data To assess the performance of our method, we consider a large observational dataset Doho of fixed size (512 samples for door, 8192 samples for liger and gridworld), and an interventional dataset Dim of varying size, ranging on an exponential scale from 4 to (Dabel-
Baselines We compare the performance of the transition model q recovered in three different settings: no obs, when only interventional data (D = Dmn) is used for training; naive, when observational data is naively combined with interventional data as if there was no confounding (D - Dim U {(T, 1)) E )); and augmented. our proposed method (D Din U Doba). Note that the only difference between each of those settings the training dataset, all other aspects (learning procedure, model architecture, loss function) begin the same.
Training In all our experiments we use a tabular model for q. that is, we use discrete proba- bility tables for each building blocs of the transition model, q(20), 2(0,)21), g( 2 at), and q(a:)/x ž,, = 0). We use a latent space | of size 32, 32 and 128 respectively for each toy problem, while the true latent space [8] is of size 3, 6 and 42. We train @ by directly minimizing the negative log likelihood (4) via gradient descent. We use the Adam optimizer [14] with a learning rate of 10 and train for 500 epochs consisting of 50 gradient descent steps with minibatches of size 32. We divide the learning rate by 10 after 10 epochs without loss improvement (reduce on plateau), and we derive the optimal policy A* exactly, while in the riger and gridworld experiments we train a "dreamer" RL agent on imaginary samples
stop training after 20 epochs without improvement (early stopping). In the door experiment
q(T 1) obtained from the model, using the belief states q( (h,) as features. We use a simple Actor-Critic algorithm for training, and our agents consist of a simple MLP with one hidden layers for both the critic and the policy parts. RL agents are trained until convergence or with maximum number of 1000 epochs, with a learning rate of 10-?, discount factor y = 0.9 and a batch size of
JS divergence the expected gensen-Shanme divergence between the learned ((,,+1/h,, 1) and the true p(or+1/h, = 1), over transitions generated using a uniformly random policy Frand
To evaluate the general quality of the recovered transition models, we compute
[r]
log p(oo) mo(oo)
p h4, 1)
E Prpar Preau Pohe: Trand
log
2
m i 1)
log ((00) m((on)
g(0.41 |h, 1) log
+ Ere Oowes Raker Stand
2
m( ( 1)
where m..)
2 (Ā(.) + p())) In the first experiment we compute the JS exactly, while in the second experiment we use stochastic approximation over 100 trajectories 7 to estimate each of the expectation terms in the JS empirically.
Reward. To evaluate quality of the recovered transition models for solving the original RL task, that is, maximizing the expected long-term reward, we evaluate the policy #t obtained after planning with the recovered model 9, on the true environment p.
Σ R(oo)
Ereeepant Prave St St
In the first experiment we compute this expectation exactly, while in the second experiment we use stochastic approximation using 100 trajectories T.
acceptance
14

---

Under review as a conference paper at ICLR 2022
A.6 COMPLETE EMPIRICAL RESULTS
A.6.1 DOOR EXPERIMENT
The door experiment corresponds to a simple binary bandit setting, that is, specific POMDP with horizon H 1. The observation space of size |O| O, since the learning agent receives no observation, and the hidden state space is of minimal size |S| 3 to encode both the initial light color and the reward obtained afterwards. The bandit dynamies are described in Table
door
light
light
button
closed
open
A
0.0
1.0
red
green
red
B
1.0
0.0
0.6
0.4
A
1.0
0.0
green
B
0.0
1.0
p(light)
lidho button)
Table 1: Probability tables for our door bandit problem.
We repeat the door experiment in six different scenarios, corresponding to different privileged policies Tpry ranging from a totally random agent to perfectly good and perfectly bad agent. Fach time, we evaluate the performance of the no obs, naive and augmented approaches under different data regimes, by varying the sample size for both the observational data Dobs and the interventional data Din in the range (4, 8, 16, 32, 64, 128, 256, 512).
In each scenario, we report both the expected reward and the JS as heatmaps with [Dnrl and 'Dotxl in the r-axis and y-axis respectively, to highlight the combined effect of the sample sizes each approach. We also provide as a heatmap the difference between our approach, augmented, and the two other approaches in the second row. As a remark, shades of green show gains reward (the higher the better), while shades of purple show gains in JS (the lower the better).
obs and naive. We always plot the expected reward in the first row, and JS
Finally, we also present two plots which provide a focus on the data regime that corresponds to the largest number of observational data (DDatol 512), in the main paper.
15

---

Under review as a conference paper at ICLR 2022
Noisy Good Expert In the noisy good expert setting, the expert plays halfway between perfect and random policy. The diversity of its action leads to a good start for the naive model but the bias it contains is hard to overcome. In contrast, from the start and also able to correct the bias as interventional data come in, eventually converging towards the true transition model.
method makes good use of the observational data
divergence
reward
pro (button (light)
0.15
0.60
button
light
A
B
0.10
pbs naive
0.55
red
0.9
0.1
augmented
green
0.4
0.6
0.05
0.50
2
2
2
27
nints (log scale)
(log scale)
512
10:01s
6666
332
Talee cb 111
6.66
512
sugmeniel dbs in.
0.00
532
sugmorted 19 cbs
512
JMgmurkel -aae
204
26-
-0.0
-006
C5 30
13e
125
04
12
339
10
ro tit
ralve 016+111
sigmmeteed obo- WY
sugmerted no
augmertet nave
312
212
514
0.15
6.60
512
258
5
1112
-0022
0.000
15
120
123
25
04
22 329
Figure 7: Noisy good expert setting. Heatmaps correspond respectively to the expected reward (top row, higher is better) and the JS divergence (bottom row, lower is better).
16

---

Under review as a conference paper at ICLR 2022
Random Expert A random policy naturally results in unconfounded observational data, since it does not exploits the privileged information. Hence, the naive approach is unbiased in this case, and actually makes the best use of the observational data. Our approach, augmented, exhibits an overall comparable performance, only slightly worse at times. We believe this can be explained by the additional complexity of method which tries to disentangle confounded regime in the data, and is not best suited to unconfounded data.
Js divergence
reward
Tpr (button [lihh)
0.15
0.60
button
light
A
B
0.10
no obs naive
red
0.5
0.5
0.55
augmented
green
0.5
0.5
0.00
2
2
2
2
2
nints (log scale)
nints (log scale)
6.00
351
538
600
16
134
1:49
0.45
-133
-33
335
6.05
0.05
0.00
181
215
35
333
125
22
333
26
64 120
16
320
Figure 8: Random expert setting. Heatmaps correspond respectively to the expected reward (top row, higher better) and the JS divergence (bottom row, lower is better).
17

---

Under review as a conference paper at ICLR 2022
Perfectly Good Expert Observing perfectly good expert playing in the door problem induces strong bias, because every observed action always results in positive reward. As such, the naive approach struggles to learn good transition model. The bias however is quickly corrected by augmented approach, which eventually converges to the true transition model faster than the no obs approach.
Js divergence
reward
Tpro (button [light)
0.60
bulton
light
A
B
na abs naive augmented
red
1.0
0.0
green
0.0
1.0
0.05
0.50
0.00
2
2
2
2
2
nints (ag scale)
nints (log scale)
0.5
133
650
0.00
345
645
6/45
16
rakec
Lugmortez
augrecttld
323
0:20
0.29
333
628
26
45 328
124
115
16 128
Figure 9: Perfectly good expert setting. Heatmaps correspond respectively to the expected reward (top row, higher is better) and the JS divergence (bottom row, lower is better).
18

---

Under review as a conference paper at ICLR 2022
Perfectly Bad Expert Similarly to the previous setting, observing expert that always chooses a bad action leads to a strong bias, as every action is associated to a low reward. The behaviour in terms of JS and reward is similar well.
Js divergence
reward
pro (button (lihtt)
button
0.20
light
A
B
0.15
ots naive
0.55
red
0.0
1.0
0.10
augmented
green
1.0
0.0
0.50
2
nints (log scale) sugmarkez oz3
nints scale)
Talvo
augmented:
sagtmeniod
256
335
324
2.5
355
662
6.04
0.6
0.0
-112
-004
120
131
231
281
215
Figure 10: Perfectly bad expert setting. Heatmaps correspond respectively to the expected reward (top row, higher is better) and the JS divergence (bottom row, lower is better).
19

---

Under review as a conference paper at ICLR 2022
Positively Biased Expert Here the expert's policy is considered positively biased in the sense that the agent will only obtain a positive reward when playing button A (with 55% chances) and never by playing button (0% chances). Because playing button A is actually the optimal policy, this strong bias has positive effect on the reward for the naive approach. Hence, although worse in terms of JS than approach, the naive approach always results in very good policy in terms of reward. Our augmented approach, however, seems more conservative.
divergence
reward
pro (button llihtt
0.15
0.60
bulton
light
A
B
0.55
red
0.8
0.2
augmentedd
green
1.0
0.0
0.05
0.50
0.00
2
2
27
nints (log scale)
nints (log scale)
117
ki
0.00
112
rolve 1b6+411
sugmented obs-in:
0.00
sugmerted no
200 sugmental nave
255
1.9
224
1055
350
0.55
255
6665
155
302
Tolvec sto
axgmentel dbc I91
sugmortod
20-
313
204
512
25-
312
355-
330
126
1.723
332
19.6
Figure 11: Positively biased expert setting. Heatmaps correspond respectively to the expected reward (top row, higher is better) and the JS divergence (bottom row, lower is better).
20

---

Under review as a conference paper at ICLR 2022
Negatively Biased Expert In analogous way, negatively biased expert will overuse button A, leading to mixed feelings regarding this button, whereas will always get a positive reward each time it uses button B. This leads to the opposite behavior naive approach always favoring the use button B, and obtaining bad performance in terms of reward. The naive approach only gets better when lot of interventional data is combined with the biased observational data, while our angmented approach is able to overcome this pessimistic bias very early on, and converges faster than both obs and naive.
had in the previous setting, with the
JS divergence
reward
Tpr (wutton light)
0.15
0.60
button
light red green
A
B
0.10
no obs naive
0.55
1.0
0.0
augmented
0.50
0.8
0.2
0.05
0.00
0.40
2
2
27
2
2
27
nints (log scale)
nints (log scale)
359
-02
10
4:4
35 32
04 333
25 32
120
22
313
36
2
64 126
16 31
64 328
Talea dts+itt
nugmerted ozseirt
a.gmented 031
Nagrranted
335
0:1
512
665
635
6.2
14
12 136
124
120
Figure 12: Pessimistic bias expert setting. Heatmaps correspond respectively to the expected reward (top row, higher is better) and the divergence (bottom row, lower is better).
21

---

Under review as a conference paper at ICLR 2022
A.6.2 TIGER EXPERIMENT
The tiger experiment corresponds a synthetic POMDP toy problem proposed by Cassandra al. 4. In short, in this problem the agent stands in front of two doors to open, one of them having tiger behind it (-100 reward), and the other one treasure (+10 reward). The agent also gets noisy observation of the system in the form of the roar from the tiger, which seems to originate from the correct door most of the time (85% chances) and the wrong door sometimes (15% chances). In order to reduce uncertainty the agent can listen to the tiger's roar again, at the cost of a small penalty (-1). We present the simplified POMDP dynamics in Table 2 and in our experiments we impose fixed horizon of size H 50. The observation space is of size |O| 6, to encode the roar location perceived by the agent and the obtained reward, og = (roar,, reward,) ). and the hidden state space is of minimal size [S| to encode both the tiger position and the reward obtained each time step,
St = (tiger,, reward.).
roar
tigero
liger, left
left
right
left
right
0.85
0.15
0.5
0.5
right
0.15
0.85
p(iggro))
p(roariltiger,
liger
reward++
liger,
action:
left
right
liger,
action;
-1
-100
+ +10
listen
1.0
0.0
listen
1.0
0.0
0.0
left
open left
0.5
0.5
left
open left
0.0
1.0
0.0
open right
0.5
0.5
open right
0.0
0.0
1.0
listen
0.0
1.0
listen
1.0
0.0
0.0
right
open left
0.5
0.5
right
open left
0.0
0.0
1.0
open right
0.5
0.5
open right
0.0
1.0
0.0
p ltgerr
pl berevadd
Table 2: Probability tables for the riger problem.
For the tiger experiment we consider four different privileged policies Tpr for the ob- served agent. We then evaluate the performance of the no obs, naive and augmented ap- proaches under different data regimes, by keeping the observational data fixed to (Dabzl = 8192 while varying the varying the number of interventional data for Dan in the range (4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192).
22

---

Under review as a conference paper at ICLR 2022
Noisy Good Expert In this scenario the privileged expert adopts policy that plays the optimal action most of the time (open the treasure door), but also sometimes decides to just listen or to open the wrong door. As be seen, in this scenario the observational data, and is significantly better than both the no obs and naive approaches in the low-sample regime, both in terms of quality of the estimated transition model and obtained reward.
augmented method makes the best use of
J5 divergence
reward
200
0
Agerr
listen
open lelt
open right
no 005
Left
0.05
0.3
0.65
200
right
0.05
0.8
0.15
ugmented
Spri Tpro(action) (action) (iger)
-000
21
2 2
2
2
2 2
Figure 13: Noisy good agent.
Random Expert In the random scenario there is no confounding, and observational data can be safely mixed with interventional data. The naive approach thus does not suffer from any bias, and in fact is the one that converges the fastest to the optimal transition model and policy. Our method, while manages to leverage the observational data to converge faster than no obs, suffers from a worse performance than naive in the low sample regime, most likely because it tries to recover spurious confounding variable to distinguish the observational and interventional regimes, when none actually exists.
JS divergence
reward
0.8
action open left
0
0.6
Ager
listen
open right
nO abs nalve
0.33
right
0.33
0.33
0.33
sgmeettdd
-000
(po((uction] (tgee)
2
2'
2
2
(log scale!
scale)
Figure 14: Random agent.
23

---

Under review as a conference paper at ICLR 2022
Very Good Expert Here the privileged expert never opens the wrong door, and thus never receives the very penalizing -100 reward. As a result the naive approach seems to be overly optimistic with respect to the action of opening door, which strongly affects the expected reward it obtains in the true environment. While our augmented approach seems also to suffer from this bias in the very low sample regime (as be model is still superior to both other approaches, and converges faster to the true transition model.
the reward plot), overall the quality of the recovered transition
divergence
reward
0.8
actioR open
0.5
Ager
open right
550
left right
0.05
0.0
0.95
0.4
0.05
0.95
no obs
1150
0.2
naive
Tpr (acctiio((iigr)
augmented
0.0
2'
250
2
2500
2
nints flog scale)
nints (log scale)
Figure 15: Very good agent.
Very Bad Expert Here the privileged expert never opens the correct door, and thus never receives positive reward (+10). As result, the naive approach seems to be very conservative, and prefers not to take any chances opening a door. It turns out that this strategy is not too bad terms of reward (always listening yields -51 total reward), and such this causal bias seems positively affect the performance of the naive approach in the low sample regime, but prevents it from obtaining better policy in the high sample regime too. Our augmented method, the other hand, is able escape this overly conservative strategy earlier on, and converges to a good-per peroopperooming policy faster than both other approaches.
J5 divergence
reward
0-
Riger
listen
open left
open right
0.95
right
0.05
0.0
0.95
400
Typ (action liger)
augmented
2
2
2
2'
2
2
Figure 16: Very bad agent.
24

---

Under review as a conference paper at ICLR 2022
A.6.3 GRIDWORLD EXPERIMENT
The gridworld experiment, represented in Figure 17 inspired from M. It consists in small 5x5 grid where the agent starts on the top-left corner, and tries to get to target placed on the bottom side behind large wall. The agent can use five actions: top, right, bottom, left and idle, and only receives noisy signal about its current position. At each time step, the agent's position is revealed with 20% chances, and remains completely hidden otherwise. In addition, the agent's actions only have stochastic effect, i.e., the agent moves into the desired direction with 50% chances, and otherwise slips at random to one of the 5 adjacent tiles current tile. case the agent would bump into a wall, it simply remains at its current position. The observation space is of size |O| 44, encode both the agent's location (or the indication that the location hidden) and the reward, and the hidden state space is of size [S] 42 encode both the agent's location and the reward. In this experiment impose a fixed horizon of size 20.
e
X
Figure 17: The gridworld problem
For the gridworld experiment consider a single policy Tpr for the privileged agent, who acts optimally (shortest path from current location to target). We then evaluate the performance of the no obs, naive and augmented approaches under different data regimes, by keeping the observational data fixed to [Dob 8192 while varying the varying the number of interventional data for Din in the range (4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192).
Very Good Expert In this scenario the privileged agent adopts perfect policy, and always chooses an action leading to the shortest path towards the target. As can be seen, here again our augmented method makes the best use of the observational data, and converges faster than both the no obs and the naive approaches for recovering the true transition model. This improvement in the transition model also translates into improvement in terms of the learned policy, which starts converging towards high reward values with fewer samples (2?) than both no obs and naive (29).
divergence
reward
1.00
0.6
0.4
naive augmented
0.50
0.2
0.25
0.00
0.0
2'
200
nints (og scalle)
nints flog scale)
Figure 18: Perfect agent.
25

---

Under review as a conference paper at ICLR 2022
Dine
no obs
naive
augmented
22
23
25
26
27
Figure 19: Average heat-maps over 100 episodes x 10 seeds, of the tiles visited by each trained
agent (no obs, naive, augmented) for different interventional data sizes (22, 23 24, 2 26 27). The augmented approach is the fastest (in terms of interventional data) to learn how to properly escape the top part of the maze through tile (4, 2), and then move towards the treasure on tile (1, 3).
26

---

Under review as a conference paper at ICLR 2022
Dinel
no obs
naive
augmented
28
29
20
2
22
213
Figure 20: Average weat-maps over 100 episodes x 10 seeds, of the tiles visited by each trained agent
(no obs, naive, augmented) for different interventional data sizes (28_ 29, 210 211 212 213). The augmented approach is the fastest (in terms of interventional data) to learn how properly escape the top part of the maze through tile (4, 2), and then move towards the treasure on tile (1, 3).
27

---

Under review as a conference paper at ICLR 2022
A.7 PROOFS.
Proposition 1. Assuming [2] 2 [S), ar, = 1) is an unbiased estimator of p(0r+1 [he 0r, 1).
Proof. The proof is straightforward. First, we have that D p(T, i). Second, we have E Q. because Q is only restricted to the augmented POMDP constraints, and because its latent space is sufficiently large [|2) 2 (S). Therefore, 0(T, i) solution of 44) is an unbiased estimator of p(T, i). and in particular @(0441 (h, az, i = 1) is an unbiased estimator of p(or+ at," 1).
Corollary 1. The estimator i(o [h, art 1) recovered after solving 4 with (Datel Do offers strictly better generalization guarantees than the one with [Dobsl = for any D
Proof. There exists least one history-action couple (br-1,ar-1). T 2 that has non- zero probability in the observational regime. This ensures that there exists value or for which IT p(a, f4,
0)p(o4+1|he a az, 0) is strictly positive, which in turn ensures ((or+1 hy, Qr, = 1) > 0. As a result, the family models {q(0)+1(h,, a,, = 1) q e Q. = 0) p(rli 0)} is strict subset of the unrestricted family {q(0041 [hc ar, 1) e
2). and thus offers strictly better generalization guarantees.
O
Theorem 1. Assuming [Daia] - oc, for any Dia the recovered causal model is bounded as follows:
T-1
T
II ((o+11/ a= 1) t=
II p i 0)p(o++1/hx ae, = 0), and
=
T-1
T-1
T-1 plas 0),
II 1=0
@(o a,, < II
plac i 0)p(0:+1 az, 0) +1
II t=
1=0
T where p(hr-1, 0T 0) >
Proof of Theorem Consider q(T, i) Q any distribution that follows our augmented POMDP constraints. Then, for every T 2 we have
T-1
(
II 1=0
a(a, he, () al,
y(holi)
2
T-1 II
Σ 20-
a(20 ho i)
a(a, 2+11 o4 124122,11 hy i),
t=0
by using A,, Z441, O141 IL Zo--e-1 Zz H,, I, which can be read via d-separation in the augmented POMDP DAG. Likewise, for every t 2 we have
g(ott1lhe,a,, i ai, = Σ g2(+1,
2 glaa = 1) Σ (((++) 0241 hi Oz, = 0), Σ
2
21
Ze41
by using Z, 11. A₁ He, I 1 and Z:11 further write
Zz, A,, Hr. Then for every 2 we can
q(o441/h, 1) Σ g( or
0 Σ
2 hi ha 0).
qlo.lhe-,,,11,,1 --,,
Se+1
By recursively decomposing every q(2,, 0 0-1.2 1) until 0. and finally by using
Zo I1 7 Ho. we obtain that for any T 21
T-1
2 T+1
T-1
Il -
a(0--11h at, = =1) = Σ q 0)
II
a( 141 h4, 0),
20-4
==
28

---

Under review as a conference paper at ICLR 2022
which can be re-expressed
T-1
A²
2T11
T-1
II
g(0c+1 az, - t=0
Σ 20-47
hzol/oo i - 0)
II
g he (04 Me, ar, 0).
t=
By considering the case - 00-77-1 in isolation, and by assuming probabilities positive, we readily obtain our first bound.
T-1 II
T-1
a(oc+ az, 2 t=0
II t=
ga a) =).
In order to obtain our second bound, we further isolate the cases a/ * av. then de au A al + 01 then dg ag A a a1 dg * ag and so on until 4- = 00-77-2 A d * ar-1, which yields
II
a(0441/1,,
II t=0
0)
1=
2
T-1
+
Σ
h2 0) <(0ol=0. ho, 0))
II
(((41) 0141 az, 0)
70-47
==
T-2 2-1
Σ
(<20loo, i = 0) II
g(a,, Ze41, 0c+ 04 h,, 0) (1 p(axl2x, 0))
K=0 20-7
1=0
(1441, h2 ar, 0). K+1
II
Then by assuming probabilities are upper bounded by 1, we obtain
T-1
T-1
II t=
gloc+ [h, az, i- 1) <
II
daceh, 0)((0)+)((00)) ar, = + +1)+1-((aolll) a = 0)
t=
T-2
K-1
Σ II (he 0) k=0t=0
II t=0
glaelheri 0)
Il t=0
clar/heet 0)
T-1
T-1
II q(a i 0) 0)
II
q(a (h, 0).
Finally, with & solution of and [obsl oo we have that DkI (p((Ti = 0) = 0)) =
and thus q(a, lhe, 0) p(a (h,, = 0) and @(0r+ lhe, de, = 0) = p(O (he ar, = 0), which shows the desired result.
29


Reviews:
[{"review_id": "hOwjBrRE4uC", "reviewer": "Reviewer_cYgx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n1. This paper addressed an interesting and important question in RL, i.e., how to use the observational data to improve the performance of online learning.\n\n2. The authors claimed that their setting is non-trivial by considering the unobserved confounders in the observational data.\n\n3. Their method was shown to be valid and promising both theoretically and empirically.\n\n**Weaknesses**\n\n1. Since one major contribution claimed in this paper is to bridge the causal inference with reinforcement learning, I was expecting that the authors could use a more rigorous causal framework and necessary assumptions to ensure the validation of their method and theory. For instance, to replace the do-operator with the conditional probabilities, one should assume ignorability or exogeneity. Please refer to [1] below and add related assumptions. \n\n[1] Pearl, Judea. \"Models, reasoning and inference.\" Cambridge, UK: Cambridge university press 19 (2000).\n\n2. I am not very convinced why online data particularly follows POMDP and offline data follows (possibly) privileged POMDP. Does a pre-testing procedure is required to justify the model assumptions?\n\n3. There are at least two directions of literature that the authors should pay attention to and justify their novelty. \n\n(a). First, a number of works have proposed to combine observational and experimental data (though not for RL), such as [2], [3], etc. The authors may justify why they use the augmentation procedure and will this procedure achieve the usually desired doubly robust property?\n\n[2] Athey, Susan, Raj Chetty, and Guido Imbens. \"Combining experimental and observational data to estimate treatment effects on long term outcomes.\" arXiv preprint arXiv:2006.09676 (2020).\n\n[3] Cooper, Gregory F., and Changwon Yoo. \"Causal discovery from a mixture of experimental and observational data.\" arXiv preprint arXiv:1301.6686 (2013). \n\n(b). There are increasing works regarding combining offline and online data in RL, while it seems that the authors only discussed partial of them. See some works below.\n\n[4] Nair, Ashvin, et al. \"Accelerating online reinforcement learning with offline datasets.\" arXiv preprint arXiv:2006.09359 (2020).\n\n[5] Gelly, Sylvain, and David Silver. \"Combining online and offline knowledge in UCT.\" Proceedings of the 24th international conference on Machine learning. 2007.\n\n4. I don't agree with the statement by the authors that 'Although we would have loved to compare against those approaches, the lack of available code did prevent us from running a fair comparison.' Actually, by searching these cited papers' titles with 'GitHub', I did find their implementations (as follows). Thus, the authors should add the comparison studies to justify their better performance.\n\n[6] Nathan Kallus, Aahlad Manas Puli, Uri Shalit. Removing Hidden Confounding by Experimental Grounding. NIPS 2018.\n\nhttps://github.com/CausalML/RemovingHiddenConfounding\n\n[7] Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: A causal approach. In NIPS, 2015.\n\nhttps://github.com/nanavatirutu/CausalBandits\n\n", "summary_of_the_paper": "This paper considers the model-based reinforcement learning (RL) problem by combining the offline and online data. The online data, i.e., the interventional data, is generated from the standard partially-observable Markov decision process (POMDP), while the offline data, i.e., the observational data, is generated from the privileged POMDP where the offline learner had the access to the state information (i.e., the unobserved confounder) to make an action. The authors proposed an augmented learning procedure to safely combine these two separate different sources and learn a more efficient policy. Their method is shown both theoretically and empirically better than not using offline data. My main concerns lie in their framework and assumptions, novelty compared with existing literature, and comparison studies.", "summary_of_the_review": "I think this is a borderline paper that addressed an important question with reasonably good performance while lacking necessary elaboration and justification. As commented in my 'Main Review', my major concerns to recommend this paper lie in their framework, novelty compared with existing literature, and comparison studies. I am willing to upgrade if my concerns can be addressed during the rebuttal period.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y411ehR8GME", "reviewer": "Reviewer_NLhN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies the evaluation of interventional distributions in a canonical POMDP model with a finite horizon. The target query is P(o_{t+1} | do(a_{0:t}),o_{0:t}) where A_0, ... A_t represent actions from stage t=0, ... t; O_0, ... O_t+1 represents partially observed states from stage t = 0, ..., t+1. This learning setting is general since it could represent most of treatment regimens in medical domains.\n\nThe author first shows that given data collected from interventions, the interventional query P(o_{t+1} | do(a_{0:t}),o_{0:t}) could be consistently estimated using the conditional distribution P(o_{t+1} | a_{0:t},o_{0:t}; I = i) where I represent the intervention policy that generates the data. This is not surprising since the sequential backdoor criterion is entailed in the interventional data. The authors then derive a bound over the product of target query \\prod_{t = 0}^{T-1} P(o_{t+1} | do(a_{0:t}),o_{0:t}) from the observational distribution. The result appears interesting at first, but seems to be a simple application of Manski's bound in (Manski, 1989). \n\nThe authors validate their results through comprehensive simulations. Results show that estimation using both the observational and interventional data consistently outperforms other learning strategies. However, it is unclear how the combination is done. That is, it would be interesting to see how the authors combine the unbiased estimator from the interventional data with the bound derived from the observational data. Unfortunately, this detail is not elaborated in the main manuscript.", "summary_of_the_paper": "This paper studies the problems of evaluating interventional distributions (i.e., system dynamics) of a partially observed Markov decision process (POMDP) from samples collected from a combination of randomized experiments and observations of a privileged expert who could access the latent state. The POMDP is presumed to have a finite horizon, e.g., the physician could only perform a finite number of treatments for the same patients. The authors propose an unbiased estimator for evaluating system dynamics from the experimental data. As for the observational distribution where the unobserved confounding exists, the authors derive bounds over unknown system dynamics, estimable from observations.", "summary_of_the_review": "Overall, the authors study an exciting topic causal identification in POMDP, which is a quite general, and challenging learning setting. My main concern with this paper is its novelty. First, the unbiased estimator in Eq. (4) is not surprising and follows immediately from the backdoor criterion. I am pretty sure many similar MLE estimators have been proposed. Second, the bound in Theorem 1 might be interesting, but appears to be a simple application from the bound in (Manski, 1989). It would be encouraged if the authors could elaborate how to combine these different methods to obtain a more accurate estimation of the target interventional distribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GUhWfo4bOcO", "reviewer": "Reviewer_8zjw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe paper proposes to learn a latent causal transition model explaining both the interventional and observational data and infer the standard POMDP transition model via deconfounding using the recovered latent variable. \n\nThe  paper shows that combing both intervention data and observation data can achieve better generalization guarantees in the asymptotic case. \n\nWeaknesses\nThe setting considered is rather limiting because it assumes that there is no confounder in the model of the POMDP. \n\nThe writing is very poor. The main contribution, section 4.3 is not clearly explained. It is not clear how imposing an observational distribution q(τ|i = 0) acts as a regularizer for the interventional distribution.  \n\nExperiments are only done with very simple toy problems.", "summary_of_the_paper": "The paper considers the problem of learning a causal model in the POMDP setting. It assumes the learning agent has the ability to collect online experiences through direct interactions with the environment and can access a large collection of offline experiences obtained through the observation of another agent. It further assumes that the observed agent can act based on privileged information hidden from the learning agent. The paper formulates model-based reinforcement learning in this setting as a causal inference problem. The paper then proposes to use offline data as a regularizer during learning. The paper presents empirical results on a number of toy problems.", "summary_of_the_review": "The setting considered is very limiting as it assumes there are no latent confounders. See the paper for the tricky issues involved:\nShaking the foundations: delusions in sequence models for interaction and control\nPedro A. Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, Tom Everitt, Corentin Tallec, Emilio Parisotto, Tom Erez, Yutian Chen, Scott Reed, Marcus Hutter, Nando de Freitas, Shane Legg\n\nThe experiments are not performed in any non-trivial settings. \n\nThe writing makes the paper hard to read. For example, it references rule R2 without specifying where it is or first introducing it.\n\n===\nThe authors have made their contributions and assumptions more clear, and will add  results comparing with related work. I am happy to upgrade my rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "7m8brEeRpmC", "reviewer": "Reviewer_Xuy7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concerns are on the experimental side. \n\n- The results on the three very low-dimensional synthetic toy problems are quite limited. It is hard to judge the validity of the proposed method. As we know, the RL problems would become exponentially difficult as the state dimension increases. Also, in the current RL community most RL algorithms take image pixels as input. Without such experiments, it is unclear how the proposed method work in the real world scenarios. \n\n- Comparing with several baselines and SOTA methods is an important way to demonstrate the superiority of the proposed approach. Unfortunately, such a comparison is missing in the paper. I suggest the authors should add some, which would make the paper more convincing. E.g., Rezende et al. (2020), Kallus et al. (2018), Zhang&Bareinboim (2020), etc.", "summary_of_the_paper": "The authors study the POMDP problem from the causal perspective, and the propose to combine offline and online data to infer the transition model via deconfounding. On the theoretical side, they show that the proposed method is correct and efficient in terms of generalization guarantees. On the experimental side, they evaluate the proposed method on three synthetic toy problems. ", "summary_of_the_review": "The experimental results are quite limited so that they are not enough to support the claims in the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors provide a model-based approach for combining experimental and observational data in reinforcement learning, specifically in POMDPs.

The paper was not received very favorably by reviewers, with the main concerns revolving around: (a) writing quality, (b) validation, (c) extent of contribution given existing work on causal RL.

In preparing your revision, in addition to clarifying writing, and adding better validation, I would urge the authors to consult existing causal inference literature on point and partial identification in settings related to RL, such as off-line policy learning.  This will help address issues of novelty by extending their approach to settings with more types of confounding.  In addition to useful references suggested by reviewers, another useful draft may be:

"Path-Dependent Structural Equation Models." Srinivasan, R., Lee, J., Bhattacharya, R., and Shpitser, I.. In Proceedings of the Thirty Seventh Conference on Uncertainty in Artificial Intelligence.
Decision: Reject