Using Confounded Data in Latent Model-Based Reinforcement Learning
Authors: Maxime Gasse
Damien GRASSET
Guillaume Gaudron
Pierre-Yves Oudeyer
Venue: Accepted by TMLR ()

Abstract: In the presence of confounding, naively using off-the-shelf offline reinforcement learning (RL) algorithms leads to sub-optimal behaviour. In this work, we propose a safe method to exploit confounded offline data in model-based RL, which improves the sample-efficiency of an interactive agent that also collects online, unconfounded data. First, we import ideas from the well-established framework of $do$-calculus to express model-based RL as a causal inference problem, thus bridging the gap between the fields of RL and causality. Then, we propose a generic method for learning a causal transition model from offline and online data, which captures and corrects the confounding effect using a hidden latent variable. We prove that our method is correct and efficient, in the sense that it attains better generalization guarantees thanks to the confounded offline data (in the asymptotic case), regardless of the confounding effect (the offline expert's behaviour). We showcase our method on a series of synthetic experiments, which demonstrate that a) using confounded offline data naively degrades the sample-efficiency of an RL agent; b) using confounded offline data correctly improves sample-efficiency.

Published in Transactions on Machine Learning Research (08/2023)
Using Confounded Data in Latent Model-Based Reinforcement Learning
Maxime Gasse
NewviceNow Research
Montréal QC, Canada
Damien Grasset
damicn mmmninnggnaaccd@iifssccteeepppep
IRT Saint Exupéry Canada Montréal QC, Conada
Guillaume Gaudron
guillaunee.gandrrn@@Wwiisss////
Ubisoft La Forge Bordeaur, Prance
Pierre- Yves Oudeyer Inria Bordeaux Sud- Quest Bordewwr, France
pierre- Tovesooudeye@@inrii...
Reviewed on OpenReview: https: // openrevie.. net/ forum? id= #FWRU.JJP
Abstract
In the presence of confounding, naively using off-the-shelf offiine reinforcement learning (RL) algorithms leads to sub-optimal behaviour. In this work, propose safe method to exploit confounded offline data in model-based RI, which improves the sample-efficiency of an interactive agent that collects and learns from online, unconfounded data. First, we import ideas from the well-established framework of do-calculus to express model-based RL as a causal inference problem, thus bridging the gap between the fields of RL and causality. Then, we propose a generic method for lcarning a causal transition model from offline and online data, which captures and corrects the confounding effect using hidden latent variable. We demonstrate that method is correct and efficient, in the sense that it attains better generalization guarantees thanks to the confounded offline data (in the asymptotic case), regardless of the confounding effect (the offline expert's behaviour). We showcase our method on a series of synthetic experiments, which demonstrate that a) using confounded offline data naively degrades the sample-effccincyy of an RL agent collecting and learning from online data; b) using confounded offline data correctly improves its samplo-offciencyy.
1 Introduction
As human beings, understanding cause and effect is crucial successfully navigate our environment. If take aspirin, will my headache go away? Should I better lie down for while? Should I do both? Two key ingredients in our learning process experimentation (we perform actions, and we mcasure their outcomes). While it is well-known that passive observation is not sufficient. to infer cause and effect it is hard to believe that il can provide no learning signal at all. The field of cosmology draws models of the universe by exploiting experiments on earth (particle physics) and passive obscrvations of the sky (astronomy). In our everyday lives, the actions of others (c.g., a coworker taking aspirin) can bring some insight into the effects of our own actions. A key question is
observation (we passively contemplate our environment) and
Simply put, correlation does not imply causation. Or, citing Pearl 29 "behind every causal conclusion there lie rausal assumption that testable observational studies".

---

Published in Transactions on Machine Learning Research (08/2023)
then: which role does observation play when learning cause and effect? This question is at the core of the burgeoning field of causality and reinforcement learning (RI.) 2 40 16
42
26
In this paper we consider the role of confounded data in the generic setting of model-based RL. Imagine an agent trying to solve a sequential decision-making problem, such as a bot in a videogame. The agent can rely on observational data, collected from the passive observation of other agents (e.g.: a dataset of offline traces collected from other bots or humans), and experimental data, collected through the agent's own interactions (e.g., a dataset of online traces collected during learning). We are interested in scenarios where the observational data is confounded, that is, when hidden variable has been the cause of both actions and their effects in the traces (e.g,, when the observed agent had access to privileged information). Our goal is then to understand if and how the use of confounded observational data can improve the sample-efficinncy of an online model-basee RL agent learning from experimental data.
In the Markov Decision Process (MDP) sctting the entire state of the environment is available to the learning agent al each time step, hence there can be no hidden confounder. Because the issue of confounding does not exist, straightforward solutions can leverage large offline datasets safely, leading to the fast-growing field of offline reinforcement learning 20 21. In the more general Eartially-Observabbee MDP (POMDP) setting, however, offline data must be considered with more care, because of the potential presence of confounding, A typical example is in the context of medicine, when offline data is collected from physicians who may rely on information absent from their patient's medical records, such as their wealthiness or their lifestyle. Suppose that wealthy patients in general get prescribed specific treatments by their physicians, because they can afford it, while being less at risk to develop severe conditions regardless of their treatment, because they can also afford a healthier lifestyle. This creates spurious correlation called confounding, and will cause a naive recommender system to wrongly infer that treatment has positive health effects. Another example is in the context of autonomous driving, when offline data is collected from human drivers who have a wider field of vision than the camera on which the robot driver relies. Suppose human drivers push the brakes when they see a person waiting to cross the street, and only when the person walks in front of the car it enters the camera's field of vision. Then, again, a naive robot might wrongly infer from its observations that whenever the brakes are pushed, a person appears in front of the car. Suppose now that the robot's objective is to avoid collisions with pedestrians, then it might get regrettably reluctant to push the brakes. Of course, in both those situations, the learning agent can infer the right causal model by disregarding the (confounded) offline data altogether, and by relying only on online data instead, collected from its own direct interactions. However, in both those situations also, collecting online data can be expensive, impractical, or even unethical, while collecting offline data by observing the behaviour of human agents is much more affordable.
In this paper we study the question of combining confounded offline data with non-confounded, online data in model-based RL, in the general Partially-Observaal Markov Decision Process (POMDP) setting. Our contribution is three-fold:
1. We formalize model-based RL as a causal inference problem using the framework of do-calculun 30 which allows us to reason about confounding in the online and offline scenarios in a formal and intuitive manner (Section 3
2. We present a generic method for combining online and offline data in model-based RL (Sect which we demonstrate is correct even when the offlin policy relies on privileged hidden information (confounding), and is efficient in the asymptotic case (infinite offline data).
3. We illustrate the effectiveness of our method with a practical implementation for the tabular setting, and three experiments on synthetic toy problems (Section
While our proposed method can be formulated outside of the framework of do-caleulus, in this paper we hope to demonstrate that it offers principled and intuitive tool to reason about causality in model-based RL. By relating common concepts from RL and causality, we wish that our contribution will ultimately help to bridge the gap between the two communities.
28ee section 6 for a discussion of related works.
2

---

Published in Transactions on Machine Learning Research (08/2023)
Pirana
Plrans
Piutt
Pinase
Piraux
S₁
S++1
S++1
Ppru
Peru
Pprv
Ppre
Pprs
Pprr
On
1
O₂
A
Or+1
Oo
Ar-1
O
A
O++1
Fold
Wstd
Epro
Frru
Figure 1: Standard POMDP regime.
Figure 2: Privileged regime (confounding).
2 Background
2.1
Notation
In this paper, upper-case letters in italics denote random variables (e.g. X, while their lower-case counterpart denote their value (c.g. y, and their calligraphic counterpart their domain (e.g., x X). For simplicity we consider only discrete random variables. To keep our notation uncluttered, with slight abuse of notations use p(x) to denote sometimes the event probability p(X æ), and sometimes the whole probability distribution of X, which should be clear from the context. In sequential models also distinguish random variables with temporal index f, which might be fixed (e.g.,
), or undefined (e.g-, 1(8,+118,, 4z) denotes at the same time the distributions p(31180, 0n) and p(>2/81, (1)). We also adopt a compact notation for sequenees of contignous variables (e.g., 80-I = (so....,,87) ST) E ST+1 ), and for summations over sets (Erex - Ex". We assume the reader is familiar with the concepts of conditional independence (X I. Y 2) and probabilistic graphical models based on directed acyelic graphs (DAGs), which can be found most introductory textbooks, c.g. Pearl 27 Studeny
301 Koller and Friedman I8
2.2 Partially-Observabll Markov Decision Process
We consider episodic Partially-Observabi Markov Decision Processes (POMDPs) of the form M (S, O, A. Pinit Peranas Ppre, r), with hidden states E S, observations o O, actions E A, initial and transition state distributions Po)a((o)) and ( az), observation distribution (p))(o)(st). and reward function r O + R. For simplicity we assume episodes T (ov. .g.....0y) or) of finite length |T] = T> 0, and history at time I, h4 = (0p, ao...., 0r). The control mechanism represented as
we introduce the concept. stochastic policy π,, which together with the POMDP dynamics Pinsts Ptrons and Perv defines a probability distribution over trajectorics, p(r). In this work consider two types of control policies, which result in two distinct dala-generation regimes.
Definition 1 (Standard POMDP regime). In the standard POMDP regime, actions decided based only the visible information from the past, He, according to standard policy Tadaaa (he). This results in the data-generation process depicted in figure and trajectory distributions that decompose
g1rl+1
77|-1
PTtd(T)= = Σ
Psnaa Pooo
II 1=
ग aad Pa 18 ag (1 (rr b
This standard regime is that the regular POMDP control problem, which formulates as:
arg max E TriPaed
Σ r(o))
(1)
Definition 2 (Privileged POMDP regime). In the privileged POMDP regime, actions con be decided based on the hidden state S. well, according privileged policy Tsta(aellv, st). This results in the data-generation
3

---

Published in Transactions on Machine Learning Research (08/2023)
process depicted in figure &
with trajectory distributions that decompose
gtrli1
1r|-1
PTru(T)
Σ
Pense( Porr((50)prr((00ss0
II t=0
80-111
This privileged regime allows us consider situations where trajectories are collected by observing an external agent who uses privileged information, in the extreme case the entire POMDP hidden state. Such a privileged agent can be for example human driver in the context of autonomous driving, who has access to privileged information not accessible to the learning robot, such as the weather forecast. There lies the origin of the confounding problem in offline RL.
2.3 Causality and do-calculus
Several frameworks exist in the literature for reasoning about causality PN 14 Here follow the framework of Judea Pearl, whose concept of ladder of causation is particularly relevant to answer RL questions. The first level of the ladder, association, relates to the passive observation of an external agent acting in the environment, while the second level, intervention, relates to the question of what will happen to the environment as a result of the observer's own actions. The tool of do-calculus 30, presented in appendix A acts as a bridge between these two levels, and is typically used to answer whether and interventional distribution, such as p(y]do(x)) 2), can be identified from an observational distribution, such p(r, v: 2). In nutshell, in causal systems that can be expressed as DAGs, intervention do(x)) forces the variables in X to take the specific value X = regardless of their causal ancestors in the graph, and queries of form p(y)do(ar), measure the effect of an intervention do(X = 2) on outcome event. Y y, in the context where another event Z = is also observed. In this paper, we will use do-calculus to reason formally about model-based RL in different POMDP data-collection regimes, which entail different causal graphs.
3 Model-based RL as causal inference
In POMDPs, model-based RL relies on measuring the causal effect of immediate interventions, do(ae), on the next. observation, Ot+1, given that past observations, 00-s:: and past interventions, do(ao-rt1), have already happened. Such causal queries are embodied in the causal transition model |00 do) not on the control policy π.
Decision-making problems are inherently causal 10 7
3 which depends ouly on the POMDP dynamics in M, and
(0ee)loo-1,dd(aa0--0 Psta(Or+1 00-se, do(ag->)), WWstd Ppro 100--, do(ao-1))) VT
Together with the initial distribution p(ou), this causal model allows for the evaluation of any standard control policy Ts)) Model-based RL then decomposes the control problem equation ( into two sub-problems:
1. learning: given a dataset D, estimate a model ((0r+1 [he oz) Rs p(o so do(an-)))):
2. planning: given a history h, and the model i derive an optimal action dz.
In this work we consider only the first problem, that is, learning the causal (ransition model from data. Next, we show using do-calculus that this problem can be either trivial or impossible, depending on whether the data is collected using a standard or a privileged control policy.
Guiding example. Consider the door problem illustrated in figure 3 You are sitting in a room with a door, a light that can be red or green, and two bullons that will open the door deperding on the light color. You can collect data samples two ways, either from interventions, i.e., you get up and press the buttons (expensive), or from observations, f.e. you watch someone else press the buttons (cheap). A key detail: you'l 're colorblind
Such a notation be found also
26
4

---

Published in Transactions on Machine Learning Research (08/2023)
Light
Button
Door
Causal DAG
Figure 3: The door problem.
and can't distinguish red from green. Your goal is to find which button is more likely to open the door. The mechanism responsible for opening the door works as follows: when the light is red, button A opens the door, when the light is green, bulton B opens the door. The light is red 60% of the time, and green the rest of the time. You are told nothing about the door's mechanism, except that it depends on both the light color and the button pressed (Light Door + Button). Since you are colorblind you camnot use the light color to make decisions, and the question you are interested in is simply, which button more likely open the door? In the do-caleulus fromework, this question translates to
arg max p( buttonE buttonE(A.B)
You have to estimate two causal queries: pddoor = A)) and p(door=open]dol (uutonn = B)).
3.1 In the standard POMDP regime
In the standard POMDP regime, we assume access to dataset D Psta(T) of episodes collected using an arbitrary standard policy Tat((a)/h)) A key characteristic in this setting is that A, TT S, Hy is always true, that is, every action is independent of the current hidden state given the current history. By applying do-calculus on the causal graph from figure [ the causal model can be shown be trivially identifiable
plo (o((a--)))) = Psta(ot+1/he,00
(2)
Because of this property, any trajectory T (Tta(T) can be interpreted as an interventional trajectory, where the learning agent itself could have decided on each of the action a, in т. Thus, in the remainder of the paper will interchangeably call the standard POMDP regime the interventional regime, and any dataset Datd collected in this regime an interventional dalasel.
Assuming sufficient exploration, which is achieved if the control policy is strictly positive (Tard(ae)(s)) > O, Ya,, he), an estimator the POMDP causal model can be obtained from Data via log-likelihood maximization,
Das |+|-1
i = arg max qeQ
Σ
Σ
log q(0r+1//e az).
(3)
This corresponds to the simplest and most common form of model learning via supervised learning 24 which effectively solves our causal inference problem.
Guiding example. Consider again our door example. If you collect the result of your own for another colorblind person's) interactions with the door, then you know that the light color can not cause which bullon is pressed (Light f Button). Then, you can directly estimate the causal effect of the button on the door,
pl pdoor= do( button)) Petal door= button)
In this regime, regardless of which policy is used to collect (bulton, door) sumples, eventually you realize that button A has more chances of opening the door (60%) than button (40%), and thus the optimal actior
4nne assumption though strict pasitivity, Tualbutton) > Viutton, which ensures that both buttons pressed.
5

---

Published in Transactions on Machine Learning Research (08/2023)
3.2 In the privileged POMDP regime
In the privileged POMDP regime, we assume access to dataset Ppre ~ Ppru(T) of episodes т collected using arbitrary privileged policy T Ihe 8t). In this setting, actions might not be independent of the current hidden state given the current history, i.e., A, IL St H2 might not hold. Because each hidden state St has causal effect on both the current action A, and the next observation O++1, it acts as hidden confounder in the POMDP causal transition model. This confounding effect can not be adjusted for without observing results in the
the hidden states the POMDP, and applying do-calculus on the causal graph from figure iggueercsutts causal model p(oc+1 (00=0))00((0 being non-identifiable from Ppro(T). In particular,
do((o-st)) + Ppro(Or- (1),
Because of this, trajectories T ~ (T) cannot be interpreted as interventional. To better relate to the causality literature, we will interchangeably call the privileged POMDP regime the observational regime, and any dataset D pry collected in this regime an observational dataset.
Note that, a consequence of this non-identifiability, naively applying any off-the-shelf offline RL algorithm 20 21 on an observational dataset such as Dpru is risky endeavour, and might result in biased transition models and value functions, and sub-optimal policies.
Guiding example. Take again the door example in figure S and assume that you observe someone else interacling with the door. You do nol know whelher that person is colorblind not (Light. - Bullon is possible). In this regime, without additional knowledge, you connot recover the causal queries p(door=ope (button)) from the observed distribution doorr, button). In the do-calculus framework, the queries are said non ifibli However, if that person was to tell you the light color they see before they press A or B, then you could recover those queries via deconfounding,
((door==pennloo(utton)))
Σ lighte fred.green)
Pere button).
This formula eventually yields the correct causal transition probabilities regardlees of the observed policy, giver that enough (light, button, door) samples are collected
3.3 Connection to online and offline RL
To relate the concepts of standard (interventional) and privileged (observational) POMDP data to online and offline RL, the key question to ask is, when the samples were collected, could the control policy have used privileged information besides the history h₂? Or, more formally, can we guarantee that A, TL St He did hold in the data-generating process?
online RL, the learning agent explicitly controls the data-collection policy, so by design it can not rely on privileged information, hence A, IL S, H, always holds. Therefore, data collected an online RL setting can be safely treated as interventional, and the causal transition model can be directly estimated using equation 3
In offline RL, the learning agent might have limited knowledge about the data-collection policy, sometimes no knowledge at all. In some settings, it can be shown that the offline policy could not have used any privileged information, and offline data can be treated as interventional. For example, with human replays from Atari video games, is hard to imagine human player having access to more information from the machine's internal state than the regular video and audio outputs from the game. But in more general offline not be dismissed. This is particularly true with human
RL settings, access to privileged information demonstrations collected in the wild, such as in the context of autonomous driving, medical recommender systems (examples in Section D or question answering systems 26 In that case, the offline trajectories can not be considered interventional, and the offline dataset must be treated as observational.
5he strict positivity condition here ppre (eattootoo light) Viutton, Right

---

Published in Transactions on Machine Learning Research (08/2023)
Post
Piruno
Piruna
So
S
S++1
Pole
Podx
Pods
Oo
A
O₂
A
O++1
a(aelee 9, = prr (e: 8t)
T(ae = 1) == mata(ae (a h4)
Figure Augmented POMDP setting, with a policy regime indicator / taking values in {0, 1} (l=inter- ventional regime, no confounding, 0=observational regime, potential confounding), enforcing the contextual independence A, 11 S. H,, =.
4 Combining observational and interventional data
Given enough online data, RL agents can learn optimal policies. But in some situations collecting a large online (interventional) dataset can be expensive (recording a robot driver in the wild), while collecting a large offline (obsorvational) dataset from demonstrations is relatively cheap (recording human drivers in the wild). Is possible then to leverage such oflline data to improve the sample-efficincy of an online RL agent, even in the presence of confounding?
4.1 Problem statement
We consider two datasets of POMDP trajectories, Dand and Drrv sampled respectively the standard (interventional) and the privileged (observational) POMDP regime. We then ask the following question: can the observational dataset Prr be used in combination to the intervent ional dataset Datd, to improve the POMDP causal transition model p(or+1100-4 do(0o=1)) that would be obtained from equation (3) using Dad only? As we will sec, answering this question will require to go beyond the identifiability framework of do-calculus.
4.2 The augmented POMDP
Since both datasets Date and Prro are sampled from the same POMDP (Pinit, Ptrans; Ppru) controlled in two different ways, we introduce a regime indicator variable [7] e (0,1} that controls an augmented control policy æ. This results the augmented data-generating process depicted in figure such that
Dpro ~ p(rlip = 0) == Ppru(T); and D Daca ~ p(T|i = 1) = Pard(T).
Note that the augmented control policy induces the contextual conditional independence At S, Hr, 1, which is implied by the DAG factorization. As direct consequence of equation ( in this augmented POMDP the causal POMDP transition model can be extracted as
p(o 100 do(ao Pata(Ortilhes at) p(0 he, ar, 1).
(4)
4.3
The augmented learning problem
In order to learn the causal transition model p(o:+1 100-1, do(o0-11) propose the following two-step procedure, which relies on fitting a latent probabilistic model & that explains both Data and D prv- Our latent model is constrained to respect the structure of our augmented POMDP, with a latent variable E Z that substitutes the true hidden state 8, S.
Note that consider this question broadest, without further assumptions about the observed offline agent. The offline agent might sub-optimally, optimally according different, reward function than the learning agent.
7

---

Published in Transactions on Machine Learning Research (08/2023)
Learning. Our learning problem formulates as standard likelihood maximization(
D
Dart
arg max REQ
log ((10 0)
Σ
log (T =1),
(5)
with Q the family of latent probabilistic models that respect the augmented POMDP structure,
2 +1
a(Tli 0) Σ
(onic(z))@obal(
Open (ee (tran)((422 (1t, 21 21) (244),; and
70-1=1
7
a(Tli 1)
Σ (imic(=o)
10-111
Note that the recovered model & conveniently decomposes into series of simpler components: the initial latent model ((zo), the observation model ((o))ze), the latent transition model (22+1122 qx), and the behaviour model ((oe/ze h4, i). In practice, the behaviour model in the interventional regime ((x./z, h4 é = 1) = [(ax i 1) can be safely ignored during learning, since does not impact the recovered latent variable nor the causal transition model. Also, the agent behavior model in the observational regime ((a,|2, substituted for a simpler model ((ee),, 0), which further simplifies the model architecture be used when solving ( This leaves only four learnable components: d(=o); ((o/)2)) ((=++1/2,, a4) and @(0s|2,, i= O), each of which can be approxinated using any black-box model, such as a feed-forward neural network.
0) can be
Inference. We recover the causal transition model @(o do(ao-r)) = á [0e ae, = 1) by applying do-calculus on the augmented DAG from figure 4 with 2; instcad of 8t- The procodure conveniently unrolls as a forward algorithın at lest lime, and relies on the recurrent computation of ((z,/4,, i = 1); a.k.a. the agent's belief state time & First, the initial belief state at t = is recovered
@(zolho: i = 1) =)=
Σ (0)ie(20) 4obs (0ol20)
Then, for every 0 St t < T, the causal transition model is recovered
((#+11, or+1 lhe, ae, i = 1) = -3, ((z Ske, t (Z++ at (or4 ((0+ at, = 1) 3z 1(2+11, 0++1 [h,, ar, 1),
E2,,
and the next belief state is updated to
a(z4tlleet,ia = 118411,0411/h ae, 1) 2 E ((21-1,0441/h) oc, 1)
Since the observational distribution Ppru does not appear in the expression of p(ot+1 do(ao-):) in equa- tion (4) how does the observational datasct Dpro influence the causal transition model Í(0 lo0-s,, do(oo-)))? The intuition is as follows. The learned model @ must fit both observational and interventional data by sharing the same latent variables Z,, and the same building blocs (0)) Gobe(Or (2) and (trene az). The privileged behaviour model (p)o(a))2() is the only component that can allow for discrepancies between the two regimes, and it offers a limited flexibility. As result, the observational distribution @(Tli 0) estimated from Dpre acts as a regularizer for the interventional distribution q( 1) estimated from D This regularization helps prevent overfitting when learning from limited interventional data, and improves the generalization performance of the estimated causal transition model. As a side comment, note that our method does not rely on the ifiabiiia of the latent transition model (1rone(8441 |s,, ax), which remains in general non-identifiable from observational data, interventional data, or any of their combinations.
Note that, while the problem of learning structured latent variable models known to be hard in general, there also exists wide range of tools and algorithms available in the literature for solving it approximately, such the EM algorithm or the methor of ELBO maximisation.
8

---

Published in Transactions on Machine Learning Research (08/2023)
4.4 Theoretical analysis
In this section analyse the two-step approach described in the previous section, and we demonstrate that it is 1) correct, in the sense that it yields a consistent estimator of the slandard POMDP causal transition model and 2) efficient, in the sense that it yields a better estimator than the one based on interventional data only (asymptotically in the number of observational data).
First, let us demonstrate how our approach is correct. An important assumption here that the latent space of the model sufficiently large (12) 2 |S|), which ensures enough expressivity to learn the true augmented POMDP distribution, i.c., p e Q. Under this condition, with enough data & converges p, and in particular i( + p(rli 1) when Datd oc. Then, the standard POMDP causal model @(or+1lly, a, - 1) being marginal distribution of 7(rli- = 1), it also converges to Mot+1/Mr, ar, 1). Hence, solving equation (5) with & sufficiently large interventional dataset Dad and a sufficiently large latent space Z converges to the true standard POMDP transition model.
Second, let us demonstrate intuitively how our approach is efficient asymptotically. Our key assumption is that we have big enough observational dataset, [Pyrol + oc, which makes it act as a strong regularizer in equation 5). Our key result is theorem [ which generalizes a famous result in econometrics known Manski's bounds 23 from the contextual bandit setting (T (1=1) to the POMDP setting (T > 1).
Theorem 1. Assuming (Dprol - 00, for any Data the recovered causal model is bounded as follows:
T-1 II
II 100--100-0,,((00-10
placlhe, 0)p(o(+1 ar, 0), and
(=0
T-1 000-1000-
T-1
T-1 II t=
II
p(a,),, 1
+
p(as h4 0),
t=0
t=0
0T--1, where 1, 0) 0.
Proof. See appendix D
Let us denote the family of candidate causal models when solving equation
5 as Ho = (g(0:+1
1:000111000000 00 40-s1.
1) q E 2} when [Dprll = O, and Hox = (4(0++1 1000-tt, 4g- = 1) g E 'Pprel + 20. Because there exists at least one episode = (oo, Go. . OT with p(rli = 0) > 0, theorem implics the non-trivial lower bound q(o,+alo0-1, ag-re, i = 1)
A q(T i = 0) =p(Tli 0)} when
for every tt 1) 0 are allowed in Ho but forbidden in Hoo, and hence H.s Ho. Because this new hypothesis space is strict subset of the original one, it offers better generalization bounds for fixed pprrol or equivalently a better sample-efficinccy with respect to /Dprul.
1, at lcast for the
specific values in r. Therefore, candidate models q such that ((0+41l00-4,:0)
Guiding example. Let now ezamine our door ezample in light of Theorem 1 Assume this time that you observe many (button, door) interactions from non-colorblind person (privileged, i= i=0), 0), who's 's policy is T [(igtton==/igghhtreee) 0.9 and π (bbutton=lllght--nn that p( hitr--opee/dobuuttoo---e---
0.4. Then you already infer from Theorem E [0.54, 0.84] and p( door= do( button-B)) E [0.24, 0.94]. You now get chance to interaci with the door (i =1, and you decide to press A 10 times and B 10 limes. You are unlucky, and your interventional study results in the following probabilities: q( g(door=open) do button=A)) = 0.5 and qi (door=openddd button=B))
0.5. This does not coincide with your (reliable) observational study, and therefore you adjust q( q(door=oeen do( button=A)) to its lower bound 0.54. You now believe that pressing is more likely to be your optimal strategy.
9

---

Published in Transactions on Machine Learning Research (08/2023)
4.5 Limitations of the provided analysis
We would like to acknowledge two limitations of the theoretical results we provide in the previous section. First, it is fairly easy to see that the upper bound in theorem is not tight. For example,
T-1 II @( (ole-illo,,,,(aaa--110 1=0
T-2
II 1=
(0 2,
is always true, and therefore
T-1
K II p(aell 0)p(0++1 or, 0) + KE (O.......
K II p(a i 0)
II ((o0+1100000 do(ao-1))
min
which is a tighter bound and also a generalization of Manski's bounds 23 Still, it is likely that this upper bound is not tight either. The purpose of theorem | is merely to serve as building block in the argument. "observational data creates bounds (in the asymptotic regime) which restrict the hypothosis space for learning". Providing lighter bounds for augmented POMDPs would give valuable insight, and is left for future work.
Second, the results in section 4.4 are restricted to the asymptotic regime [Pprol + oo, and do not provide any practical guarantee in the finite-sample regime. Our intuition is that these hard bounds would translate into some kind of soft prior over the hesot space, which would also improve generalization. Proving this idea formerly and deriving proper generalization bounds in the finite-sample regime is left for future work.
5
Experiments
We run experiments on the three synthetic toy problems described in figure 0 each expressing a different level of complexity and different form of privileged information. In order answer the question raised in section 411 we compare
augmented method against two baselines: obs which discards the observational dataset, and naive which naively combines observational and interventional data as if there was no confounding. We expect two things: 1) in the presence of privileged information (confouning), the observational dataset decreases the performance of the naire agent, compared to the no obs agent; and 2) our augmented agent benefits from the observational dataset, and outperforms both the naive and the no obs agent. The code to reproduce these experiments is available onlind
5.1 Experimental setup
RL procedure. Our augmented model-basod RL procedure is depicted in algorithm We start from a pre-existing dataset of privileged POMDP trajectories (observational data), Dpres an emply datasel of standard POMDP trajectories (interventional data), Datd, and random exploration policy, fudd By fixed increments (c.g., 0, 10, 50,.... .... 1000), the learning agent collects new interventional trajectories by exploring the environment, which complement the interventional dataset Dacs- After each increment a new latent-based model 9 is obtained by solving equation (5), and new (near)-optimal policy Ast is derived from the model @ using an actor-critic algorithm. The newly obtained policy and is then used to collect the next increment of interventional trajectories, supplemented with an e-random noise for exploration.
Model and agent. training. We train all three model-based methods, augmented, no abs and naive, using the same model architecture and training procedure. Each building bloc qinits Itrans: Johs and Sprn consists in tabular logistic model, and equation (5) is solved via mini-batch stochastic gradient descent using Adam [7] Once the POMDP dynamics are recovered extract g(oo) and @ do(ao-+)) to train "dreamer' agent [1 via actor-critic, implemented as food-forward neural network that takes as input the recovered POMDP belief state, ((=:/o--e, do))
Evaluation.
We evaluate on the real test environment 1) the quality of the causal transition model (0041|00-4 do)) in terms of its likelihood on new interventional data (collected via random exploration),
&ooee for the experiments: attp:://gthaab.cc//assseaaaaaaa cow.///assee/casaslllll--tll
10

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm 1 Augmented model-based RL pseudocode
Require: obscrvational datasct: D Pre (potentially privileged), training method (augmented, _o_obs or haive),
training steps N (c.g., N (0, 10, ....... 1000)), exploration noise
Ensure: estimated POMDP dynamics (2)) (obs(or(2z), (1rans(2+41122) at); and optimal policy fsd Initialize Deta + 0, atd f- random exploration policy if training method is no_ _o_ then discard obscrvations (Dpru + 0)
if training method is naive then consider obscrvations as interventions (Dotd + D Dero and Dprv + 1)
n Prrev f-
for all n . do
Collect prre new interventional samples using sid e-random noise, and add them to Deed
Obtain new model using Dstd and prr (equation (5), supervised learning)
Obtain new policy Asd using the model i (actor-critic in "dream" environment)
Pprev n
random
noisy good
perfect good
perfect bad
0.125
0.125
0.125
0.125
moteel likelihood
0.300
0.075
0.075
0.075
0.075
no
0.050
0.025
naive
0.025
0.025
0.025
0.000
1000
1000
500
1000
1000
[Thal
|aad
agent performance
-20-
-20-
-20
-40
-40
-40
-40
500
1000
0
500 1000
500
1000
500
1000
1Dael
DDad
Figure 5: Robustness to different degrees of confounding on the tiger problem. Top: model likelihood on new data (higher is better). Bottom: agent performance in terms of cumulated reward (higher is better). Columns: the different privileged agents used to collect the obscrvational datasct Dobs- The random agent amounts to no confounding, while the noisy good, perfect good and perfect bad agents introduce different. degrees of confounding. We report the mean + standard deviation over 10 random seeds. Our augmented method performs best regardless of the degree of confounding, both in terms of model likelihood and agent performance, and is on par with the naive method when there is confounding (random).
and 2) the performance of the resulting policy aed in terms of its cumulated reward. We evaluate cach model and agent over 10K new trajectories, and we repeat each experiment 10 times with different random seeds account for variability. We defer the reader to appendix B for the complete experimental details.
5.2 Robustness to different degrees of confounding in the tiger problem
Tiger is classic small-scale POMDP from Cassandra al. ! with |S| = 6 hidden states and time horizon T 10. To measure the robustness of our method to different degrees of confounding, we consider different privileged policies, described in detail in appendix B For each experiment we col- lect 'Dprul = 1500 (confounded) observational trajectories, and we train and evaluate at increments (Daall E (0, 50, 100, 150, 200, 300, 400, 500, 600, 700, 800, 900, 1000).
Results are reported figure 5 The first policy, random, amounts a degree 0 of confounding, because the privileged information is effectively not used to generate Dprv- In this setting our method performs on par with the naive method, which also leverages the observational data without. bias due the absence of
11

---

Published in Transactions on Machine Learning Research (08/2023)
confounding. The noisy good policy mostly opens the correct door with the treasure, but sometimes also decides to listen or to open the wrong door at random. This degree of confounding, although rather mild in terms of the bias in the non-causal transition probabilities Ppro(ot+1/Ar, dt), is particularly hurtful to the naive method. The perfect good policy always opens the correct door, which induces non-causal transition probabilities that are completely off the causal ones, and also hurts the performance of the naive method as the Iearned model will tend to be over-optimistic. The perfect bad policy always opens the wrong door, which induces transition probabilities that are completely off' in the other direction, but this time the effect on the performance of the naive method is not as bad. Indeed, the learned naive model this time will tend to be over-pessimistic, which is not such a bad prior in the case of the tiger problem. Our augmented method, always performs better than (or on par with) both no obs and naive, effectively leveraging the observational dataset Pprrll even in the presence of confounding.
5.3 Performance on the gridworld problems
Hidden treasures is a 3x3 grid-world problem inspired from Sutton et al. [37, with |S| = 36 hidden states and a time horizon T = 10. Sloppy dark room is a 5x5 grid-world inspired from Alt et al. [I, with |S| = 21 hidden states and a time horizon T= = 30. In both problems, the privileged agent has complete information at each time step, and uses a shortest path algorithm to decide on next action. For each experiment we collect prnl 8000 (confounded) observational trajectories, and we train and evaluate each method at. increments Datal E (25, 50, 75, 100, 150, 200,300, 400, 600, 800, 1000, 1500, 2000, 3000, 4000, 5000, 6000).
Results are reported in figure 7 where a similar trend can be observed in the two experiments. Initially, when few interventional samples have been collected, the observational data seems to benefit the naive method, despite the presence of confounding, and it exhibits gains both in terms of model likelihood and agent performance compared to the no obs method. But as more samples are collected, the untreated confounding effect in the observational data starts becoming hurtful to the naive method, and eventually better performances are obtained by disregarding the observational data, using the no obs method. In both cases our augmented method makes the best use of the available data, and exhibits a better convergence rate than both naiee and no obs, in terms of model quality and agent performance, despite the confounding effect present in the observational data. In figure 7 right side, we report the density of tiles visited by the agent of each method, at a chosen time step where our augmented method has almost converged, while the two other methods haven't. In hidden treasures we see that our method has learned to cycle through all 4 corners, which is the optimal strategy, while the two other methods still struggle and focus only on a single side or a single corner of the grid. In sloppy dark room, our method succeeds in quickly escaping the upper section and reaching the target, while the two other methods struggle more in the upper section, and reach the target less often. It is interesting to observe very similar outcome in both experiments, while the type of privileged information, hidden to the learning agent, is quite different. In hidden treasures the learning agent knows its position at all times, but is missing information about where the target is located, thus it needs to explore the space at test time, while the privileged agent just has to go straight to the target. In sloppy dark room, the target is fixed hence no exploration of the space is needed at test time, but the agent's 'ss location hidden most of the time and the agent has to learn how to best navigate while being half-blind.
6 Related work
Causal RL. A whole body of work exists around the question of merging interventional and observational data in RL in the presence of confounding, Bareinboim al. [ study sequential decision problem similar to ours, but assume that expert intentions are observed both in the interventional and the observational regimes, i.e., prior to doing interventions the learning agent can ask "what would the expert do in my situation?" This artificially introduces an intermediate, observed variable å = f(o)) with the property that Para(at = âv (a)) = 1, which effectively removes any confounding (A, IL. S, Zhang and Bareinboin 40 E relax this assumption in the context of binary bandits, and later on in the more general context of dynamic treatment regimes AM A2 They derive causal bounds similar to ours (Theorem [], and propose a two-step approach which first extracts causal bounds from observational data, and then uses these bounds in online RI. algorithm. While their method nicely tackles the question of leveraging observational data
12

---

Published in Transactions on Machine Learning Research (08/2023)
tiger
hidden treasures
sloppy dark room
Figure 6: Our three syuthetic toy problems. In tiger, the learning agent receives noisy signal of the liger's position (roar left or roar right). It can wait and listen to new roar at the cost of -1 reward, or decide to move left or right. Treasure gives +10 reward, and tiger gives -100 reward. The privileged agent knows the exact location of the tiger. The game stops when treasure or tiger found, or after a maximum horizon of T= 10. In hidden treasures the agent. must collect a treasure (+1 reward), which randomly located one of the four corners. The privileged agent knows the treasure's position all times, the learning agent doesn't. The treasure is reset to new location when found, and the game stops after fixed horizon of T 10. In sloppy dark room the agent must reach treasure (+1 reward) located behind wall, and slips to random adjacent tile instead of moving to the chosen direction 50% of the time. The privileged agent knows its position at all times, while the learning agent is only revealed its position with 20% chances at each time step, and is blind otherwise (a dummy position is revealed). The time horizon fixed to = 30.
model likelihood
agent performance
density of agent trajectories (at specific time step +)
hidden treasures
Dat 1000
0.03
0.02
1.0
nalve
augmented
2000
4000
6000
2000
4000
6000
nо obs
naive
augmented
17aml
aloppy FOOm
1.5
[Dad = 2000
1.0
0.5
0.0
4000
6000
2000
6000
obs
naive
augmented
(Rau
[Dedl
Figure 7: Performance on the hidden treasures (top row) and sloppy dark room (bottom row) grid- world problems. First column: model likelihood on new data (higher is better). Second column: agent performance in terms of cumulated reward (higher is better). Third column: density of the agent trajectories obtained by each method the unbiased no obs method and the biased naive method, and displays better sample-effciency in terms of of the observational (offline, confounded) data. While the
specific time step *. In both setups, our augmented method outporforms both
interventional (online) data due to its correct naive method sometimes provide some initial gains compared to the no obs method, at some point the confounding effect in the observational dataset (collected from a privileged agent) starts affecting the agent's 's performance negatively, and one better off not using the confounded data at all (no obs method).
13

---

Published in Transactions on Machine Learning Research (08/2023)
for online exploration, it does not account for uncertainty in the bounds estimated from the observational data. In comparison, our latent-based approach is more flexible, as it never computes explicit bounds, but rather lets the learning agent balance through equation 5 how data from both regimes will influence the
final transition model, depending of the number of samples available. Kallus et al. [ also propose a two-step procedure to combine observational and interventional data, which however requires a series of strong parametric assumptions (strong ony-way overlap, lincarity, non-singularity etc.), and only works in the context of binary contextual bandits.
Causal confusion. In the context of imitation learning, the problem of causal misidentification, that is, ascribing the actions of the expert to the wrong explanatory variables, is attributed to confounding by de Haan et al. [ Spencer et al. [ argue instead that it is a manifestation of coveriale shift, which appears more plausible to us. Indeed, it can be shown that in the experiments of de Haan et al. g the experts don't use privileged information, which theoretically circumvents any confounding.
Exploiting offline RL data. Combining online and oflline data in RL also raises additional challenges, such as the value function initialisation problem y or the bootstrapping error accumulation problem [9] 25]. While these challenges could be combined with and amplified by confounding, they originate from fundamentally different issues and require orthogonal treatments. Off-policy evaluation, which is about estimating the performance of a policy π using iissevaaioonll data only, can be seen as a specific instantiation of the framework we present in this paper. It corresponds to the particular setting [D sodl = O, where can be shown that the causal transition model is in general not recoverable in the presence of confounding. Still, a growing body of literature studies the question of learning purely from offiine data in the presence of confounding, under the assumption that the data-generating process respects specific structural or parametric constraints
22 3]
Large sequence models. A recent trend in RL is to apply large sequence models to estimate the enviromment's dynamics in a model-based fashion 32 15 or to parameterize a goal-conditioned policy in a model-free fashion [ 44. While large sequence models appear a promising tool for efficiently combining offline and online datasets, they remain vulnerable to confounding, as pinpointed by Ortega et al. p6 Because our proposed method follows a generic model-based approach, in theory it could be easily combined with a large sequence model to address large-scale RL scenarios, while remaining robust confounding.
7 Discussions
In this paper we have presented simple, generic method for combining interventional and observational (potentially confounded) POMDP trajectories in model-based reinforcement learning. We have demonstrated theoretically that our method is correct and efficient in the asymptotic case (infinite observational data), and we have illustrated its offectiveness empirically on three synthetic toy problems. We have also highlighted the dangers of naively using offline data collected under privileged information in RL, which can effectively hurt the performance of an online RL agent, and reduce its sample-efficiency. The main limitation of our method is that it adds an additional challenge on top of moodel-aasd RL, that of learning a latent-based POMDP transition model, which can become problematic in high-dimensional RL settings. Still, the recent success of discrete latent-based models for solving complex RL tasks I 30
12 or tasks in high-dimensional domains 31 lets us envision that this difficulty can be overcome in practice. A first extension of this work would be to develop a similar approach using latent-free transition models, which would remove the challenge of learning a latent variable model. This seems doable at least in the case T= = 1. A second potential extension to this work could be to consider a setting where several privileged agents are observed, each with its own distinct. policy, leading to multiple observational datasets. This would lead, in the asymptotic case, to stronger regularizer for the causal POMDP transition model, as the implicit bounds implied by theorem would combine into tighter ones. A third, obvious extension would be to develop similar approach within the framework of model-free RL, which could take the form of an explicit or implicit value-function regularizer. A fourth direction to investigate is that of scaling, by extending our method to RL tasks with continuous observation spaces (e.g., pixel-based] and continuous latent spaces, so that it can be applied to a broader range of problems. Finally, we hope that this work will help bridge the gap between the fields of RL and causality, and will convince the RL community that causality is an adequate tool to reason about offline, observational data, which is plentiful in the world.
14

---

Published in Transactions on Machine Learning Research (08/2023)
8 Acknowledgements
We thank the anonymous TMLR reviewers for their comments which helped us improve the quality of this work. We thank David Berger for early discussions and for pointing us to relevant bodics of work. This work was supported by the Canada First. Research Excellence Fund (CFREF), Canada Excellence Research Chairs (CERC), and the DEpendable Explainable Learning (DEEL) transatlantic research program.
Broader Impact Statement
Confounding is prevalent issue in human- menaa-geeeerted data, and can be an important source of bias in the design of decision policies, if not dealt with properly. This paper makes a humble step towards more robustness and fairness Al-based decision systems, by combining causality and statistical learning to address the confounding problem in reinforcement learning from offline data. As such this work has potentially important societal implications, in particular in critical systems where lives are stake, such as medicine or self-driving cars, and where human-generated data is prevalent.
References
[1] Bastian Alt, Matthias Schultheis, and Heinz Koeppl. Pomdps in continuous time and discrete spaces. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Newral Information Processing Systems, volume 33, pages 13151-13162. Curran Associates, Inc., 2020.
[2] Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: A causal approach. In NIPS, 2015.
[3] Andrew Bennett, Nathan Kallus, Lihong Li, and Ali Mousavi. Off-policy evaluation in infinite-horizon reinforcement learning with latent confounders. In AISTATS, 2021.
[4] Anthony R. Cassandra, Leslie P Kaelbling, and Michael L. Littman. Acting optimally in partially observable stochastic domains. In AAAI, 1994.
[5] Anthony Rocco Cassandra. Eract and opprorimate algorithms for partially observable Markou decision processes. PhD thesis, Brown University, 1998.
[6] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via Beygelzimer, Dauphin, P.S. Liang, and Wortman
sequence modeling. In M. Ranzato, Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15084- 15097. Curran Associates, Inc., 2021. URL https://proceedingg. wewpiinngn.neuui....ee. neurips. /c/paper/2021/filee 7 TT4S1522aaab0007755111116666--- Paper.
[7] Philip Dawid. Decisiin--heeretiic foundations for statistical causality. Journal of Causal Inference, 9(1):
39 39-77, 2021. doi: 11--0.01115/ddi2200000000
[8] Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In NewrIPS,
2019.
[9] Sylvain Gelly and David Silver. Combining online and offline knowledge uct In ICML, 2007.
[10] Samuel J Gershman. Reinforcement learning and causal models. In Michael R. Waldmann, editor, The Oxford handbook of causal reasoning, chapter 10, pages 295-306. Oxford University Press, 2017.
[11] Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In ICLR, 2021.
12] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models, 2023.
15

---

Published in Transactions on Machine Learning Research (08/2023)
[13] Yimin Huang and Marco Valtorta. Pearl's calculus of intervention is complete. In UAI, 2006.
[14] Guido W. Imbens and Donald B. Rubin. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press, 2015.
[15] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Newral Information Processing Systems, volume 34, pages Associates, Inc., 2021.
Curran
[16] Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by experimental grounding. In NeurIPS, 2018.
[17] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[18] Daphne Koller and Nir Friedman. Probabilistic Graphical Models Principles and Techniques. MIT Press, 2009.
[19] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In NeurIPS, 2019.
[20] Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Reinforcement Learning, volume 12 of Adaptation, Learning, and Optimizalion, pages 45-73. Springer, 2012.
[21] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offiine reinforcement learning: Tutorial, review, and perspectives open problems, 2020.
[22] Chaochao Lu, Bernhard Schölkopf, and José Miguel Hernández-Lobato Deconfounding reinforcement learning in observational settings, 2018.
[23] Charles F Manski. Nonparametric bounds on treatment effects. The American Economic Review, 80(2): 319 323, 1990.
[24] Thomas M. Moerland, Joost Broekens, and Catholiju M. Jonker. Model-based reinforcement learning: A survey, 2020.
[25] Ashvin Nair, Ablishek Gupla, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement. learning with offline datasets, 2021.
[26] Pedro A. Orlega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, Tom Everitt, Corentin Tallec, Emilio Parisot Tom Erez, Yutian Chen, Scott Reed, Mareus Hutter, Nando de Freitas, and Shane Legg. Shaking the foundations: delusions in sequence models for interaction and control, 2021.
[27] Judea Pearl. Probabilistic reasoning in intelligent systems networks of plausible inference. Morgan Kaufmann, 1989.
[28] Judea Pearl. Causality, Cambridge University Press, Cambridge, UK, 2 edition, 2009.
[29] Judea Pearl. Causal inference statistics: An overview. Statistics Surveys, 3:96 146, 2009.
[00 Judea Pearl. The do-calculus revisited. In UAI, 2012.
[31] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsca Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot lext-to-image generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821-8831. PMLR, 18-24 Jul 2021.
[32] Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Newral Information Processing Systems, volume 34, pages 27580-22375 27597. Curran Associates, Inc., 2021.
16

---

Published in Transactions on Machine Learning Research (08/2023)
[33] Πly Shpitser and Judea Pearl. Identification of joint interventional distributions in recursive semi- markovian causal models. In AAAT, 2006.
[34] Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J. Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift, 2021.
[35] Charlotte Striebel. Sufficient statistics in the optimum control of stochastic systems. Journal of Mathemati- cal Analysis and Applications, 12(3):576 592, 1965. ISSN 0022-247X. doi: 10.1016/0022- (2000000222222(3300000022-222
[36] Milan Studeny. Probobilistic Conditional Independence Structures. Springer, 2005.
[37] Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework
for temporal abstraction in reinforcement learning. Artif. Intell,, 112(1-2):181 211, 1999.
(38) Guy Tennenholtz, Uri Shalit, and Shie Mannor. Off-policy evaluation in partially observable environments.
In AAAI, 2020.
[39] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. Daydreamer: World models for physical robot learning. In Conference on Robot Learning, 2022.
[40] Junzhe Zhang and Elias Bareinboim. Transfer learning in multi-armed bandits: A causal approach. In IJCAI, 2017.
[41] Junzhe Zhang and Elias Barcinboim. Near-optimal reinforcement learning in dynamic treatment regimes. In NewrIPS, 2019.
[42] Junzhe Zhang and Elias Bareinboim. Designing optimal dynamic treatment regimes: A causal reinforce- ment learning approach. In Hal Daumé III and Aarti Singh, editors, ICML, volume 119 of Proceedings of Machine Learning Research, 2020.
43]
Junzhe Zhang and Elias Bareinboim. Bounding causal effects continuous outcomes. In AAAI, 2021.
[44] Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 27042 27059. PMLR, 17-23 Jul 2022.
17

---

Published in Transactions on Machine Learning Research (08/2023)
Appendices
Introduction to do-calculus
18
A.1 The three rules of do-calculus
18
A.2 Note ignorability and exogeneity
18
B Experimental details
19
B. Training
19
3.2 Evaluation
19
B.3 Tiger experiment
20
Additional empirical results
20
D Proof Theorem 1.
23
A Introduction to do-calculus
The framework of do-calculus 30 was proposed as an intuitive tool to answer identifiability questions given a causal graph G, such as, can the interventional distribution p(yldo(x), 2) be recovered from the observational distributions 2))?
A.1 The three rules of do- colcalculus
Do-calculus relies on three graphical rules, which depend solely on the existence specific structural constraints G:
RI: insertion/deletioon of observations, p(yldo(z), w) = p(y]do(z)) w) if Y and Z are d-separated by XUW in g", the graph obtained from g by removing all arrows pointing into variables in X.
R2: cobiionobssevvation exchange, p(yldo(x), do(=), w) p(y)do(#), 2, w) if Y and Z are d-separated by X RUW W in G1, the graph obtained from G by removing all arrows pointing into variables in X and all arrows pointing out of variables in Z.
R3: /deeetionddeetion of actions, ply do(x), do(=), w) - p(y)do(x), w) if Y and Z d-separated by X UWW in Gt, the graph obtained from G by first removing all the arrows pointing into variables in X (thus creating G*) and then removing all of the arrows pointing into variables in Z that are not ancestors of any variable in W in g*.
This set of rules has been shown be complete B 33
and results in an algorithı polynomial in the number of nodes in G to answer identifiability questions, which either outputs "no" or "yes" along with estimate (a recovery formula) based on observational quantities. We refer the reader to Pearl 30 for a thorough introduction to do-calculus.
A.2 Note ignorability and exogeneity
In this paper use at great length the concept of confounding, which is core idea in Judea Pearl's do-calculus framework. For readers who are more familiar with the framework of potential outcomes from Donald Rubin 14, the concept of confounding closely relates to the concepts of ignorability and exogeneity, which can shown to be equivalent to the unconfoundedness (no confounding) assumption 28
18

---

Published in Transactions on Machine Learning Research (08/2023)
B Experimental details
B.1
Training
In all our experiments we use tabular logistic models for each of the components in q. That is, each building bloc ((zu), ((0z) 2t), g(<++1 2t, at), and q(a; h4, 2,, 00) is parameterized using a set of softmax-normalizze scalars vectors. We train ④ via gradient descent using the Adam optimizer [rr by directly minimizing the negative log likelihood of the model (equation
5 on random tani-batches of trajectories sampled from
Dstd pr Agents are trained using the learned model as "dream" environment (by sampling imaginary trajectories ((rli 1)), with simple actor-critic algorithm (REINFORCE with state-value baseline) for fixed number of iterations, also using the Adam optimizer. Both the actor and critic consists of a 2-layers perceptron (MLP) with the same hidden layer size, which take as input the belief state recovered from the model. The training hyperparamcters we used in cach experiment are displayed in table T
hidden treasures
sloppy dark room
tiger
Latent model
latent space size [2]
32
256
128
learning rate
10-3
10
10-3
number of epochs (max)
500
500
500
number of gradient steps per epoch minibatch size (trajectorics r)
50
100
100
32
64
64
Actor-critic agent
exploration noise
0.5
0.2
0.2
hidden layer size learning rate
256
512
256
5 10
5 x
10
x
10
number of epochs
200
400
200
number of gradient steps per epoch minibatch size (trajectorics r) minibatch relurn scaling entropy bonus discount factor ?
50
50
50
32
64
64
yes
n10
no
10 3
10-3
10 10-3
1
Table 1: Training hyperparameters used in each experiment. When learning the model, divide the learning rate by 10 after 10 epochs without loss improvement (reduce on plateau), and we stop training after 20 epochs without improvement (early stopping). We use all available data for training, and we monitor the training loss for early stopping (no validation set).
B.2 Evaluation
Model quality (likelihood). To evaluate the general quality the recovered POMDP model, we compute the likelihood of 9 on a new interventional dataset Drest obtained from the true environment, p with a uniformly random policy Wrands
T II
Bro Propttttoonreeee Proone Tondd
((ov) Tenned
1)
We report an empirical estimate of this measure using 10000 trajectories.
Agent performance (cumulated reward). To evaluate quality of the agent obtained from the model ; for solving the standard POMDP control task, compute the expected cumulated reward of the policy #*
19

---

Published in Transactions on Machine Learning Research (08/2023)
the true environment p.
E Proptntt Perane Paroy
Σ r(or)
We report empirical estimate of this measure using 10000 trajectories.
B.3
Tiger experiment
We present the (compact) POMDP dynamics of the tiger problem in table After conversion to the notation in the paper, the obsorvations become or (roar,, rewardy) the actions remain a, action, and the hidden states are (liger,, rewards) The privileged policies used in the experiments (section 5.2) are reported in table 3
Table 2: Compact POMDP dynamics in the tiger problem.
roor
tigero
tiger, left
left
right
left 0.5
right
0.85
0.15
0.5
right
0.15
0.85
p( tigerr))
[roarr/tiger;
tiger
rewarde+
liger,
aclion.
left
right
liger,
aclion.
-1
-100
+10
listen
1.0
0.0
listen
1.0
0.0
0.0
left
open left
0.5
0.5
left
open left
0.0
1.0
0.0
open right.
0.5
0.5
open right.
0.0
0.0
1.0
listen
0.0
1.0
listen
1.0
0.0
0.0
right
open left
0.5
0.5
right
open left
0.0
0.0
1.0
open right.
0.5
0.5
open right
0.0
1.0
0.0
(1 actions)
pl rewarre liger, action)
Table 3: Privileged policies Appo(action| used in the tiger experiment.
action. left
privileged policy
tiger,
listen
right
left
0.33
0.33
0.33
random
right
0.33
0.33
0.33
left
0.05
0.30
0.65
noisy good
right
0.05
0.80
0.15
left
0.00
0.00
1.00
perfect good
right
0.00
1.00
0.00
left.
0.00
1.00
0.00
perfect bad
right
0.00
0.00
1.00
C
Additional empirical results
20

---

Published in Transactions on Machine Learning Research (08/2023)
DDetal
obs
naive
augmented
100
400
1000
2000
4000
6000
Figure 8: Evolution of the test-time agent trajectories in the hidden treasures experiment. We report heatmap of the tiles visited by each agent (no obs, naive, augmented) at different time steps (number of interventional samples collected) during a single RL run (single seed). Eventually all methods converge to the optimal strategy, which is to cycle through the 4 corners. Our augmented method converges to this behaviour earlier on during training.
21

---

Published in Transactions on Machine Learning Research (08/2023)
DDetal
obs
naive
augmented
100
400
1000
2000
4000
6000
Figure 9: Evolution of the test-time agent trajectories in the sloppy dark room experiment. We report heatmap of the tiles visited by each agent (no obs, naive, augmented) at different time steps (number of interventional samples collected), averaged over 10 RL runs (10 seeds). Eventually all methods manage to consistently overcome the obstacle and reach the target tile. Our augmented method converges to this behaviour earlier on during training.
22

---

Published in Transactions on Machine Learning Research (08/2023)
D Proof of Theorem 1.
Theorem 1. Assuming Dprel + 00, for any Dsta the recovered causal model is bounded as follows:
T
T-1
100-11o0- 1olaao do(ao-st) 2
II
p(ar |4, 0)p(or+1 ar, az, = 0), and
k=0
T
T-1
II t=0
I (=0
p(ae i 10p(00+11/b, 0r, = +
II t=0
p =),
VhT-1, 0T,1, T 2 where 11 0) > 0.
Proof of Theorem 1 Consider q(T, E Q any distribution that follows our augmented POMDP constraints. As an intermediary step, we will start by proving the following
T-1
2
T-1
II 1=0
az, =) at, a
Σ 2 i 0) II
04tkk ar i =0. 0).
(6)
t=0
First, for any 0S1ST-1, we can write the following factorization
ar, q 2++1, o0+1 a a, 1) (zelle, a,, () or+1 (2, hes at, 1).
Because of the augmented POMDP constraints, the independences Zz 11 A Hi, I and Ze41, O441 1l Zr Ar, He hold in which further allows us to write
( 241, O2+1 (y a,, = 1) = g(2, h4, 1)g(()41,, hy ar =0)
(7)
Then, we directly gel
2x2
Σ
)=( hy at, 0)-
(8)
11.1111
Now, let us consider the special case where
T= We can use the constraint Zo 1L Ho write
2²
q /oo ao, i 1 Σ ((zo//o. i 0)g(z1, ho, ao. 0),
which is equation 6 the desired result, for T = 1. In the case where T 2 2, we can reuse equation ( to write
2²
Σ
0T- "r.
2--1-T
Σ
07
2--1-T
T-1 II I=T-2
2²
Σ
27-1-T
Then, we can introduce variable Zr-2 and use equation
again to obtain
II ao0411he i
Σ
(2 21 - ott.::/www....a.....oooo..ooo.a.
=0)
27-2-P
T-1
Σ
((
II
ST-2-T
23

---

Published in Transactions on Machine Learning Research (08/2023)
In the case where T = 2, can use Zu 11 I Ho again to obtain equation 6 In the case where T 3, we can apply the same steps again to obtain
the desired result for T = 2.
T-1 II
T-1
a00t1 ar, a,, (=7 3
Σ
3, = =) I
((2+4,00411 h, 0z, 0).
2--3-T
t=7-3
Now, either = 3 and can use Zo L I Ho to obtain equation 66 or T 2 4 and continue the
decomposition by introducing Zr-4. By following this recursive approach we eventually reach Zo and prove equation 6 for any T.
Let us now re-express equation as follows
T-1
2
(Td
II (10441lh;:041 ac, i =
Σ
( hoo
II 1=0
o he (=0
20--
g
T-1 II
(1
T-1 (0,
Σ
40-4T
T-1
T-1
II (00411/4, 0r, 1) = t=
g (00(())0((())) az, 1= =0) 0)
==
z n
T-11
g((4+11 H4 0))
(1
T-1
+
Σ
( 0)
II
II-IIe 1=
a h4 (1
a(a
20--
By assuming probabilities are positive, we can substitute the second term by 0 to obtain our lower bound
T-1
T-1
Il
a(or4i//r 0z, = =)) 2 t=
II
oarr = 0) ar, 0).
t=0
Then by assuming probabilitics are upper bounded by 1, we can substitute q(0,+ 1124,11 a,, 0) by to obtain our upper bound
T-1
T-1
II c a,, a,, = 1) s 1=0
II
a(a hy
0)g(o a, 0)
t=0
2T
(T-1
a((2+11 (e, he a,, a,, 0))
T-1 (1-I]
+
Σ
q =0
II
50-+
t=
T-1
x II
a(a h4, 10)00+0))00hhhhhmm a,, i= = +1
II
q(a, h4 = 0).
t=
1=
Finally, with solution of 5 and [D (Dprol - have that DRL. (p(r/i 0) = 0)) = 0, and thus g(aelhe,i = 0) = p(ael,r, to conclude.
0 and in particular g(or+1 (ig az i = 0) = p(o:+1 1/e,11/1,000 ar = 0), which allows us
24


Reviews:
[{"review_id": "4Yl7IjrtNs", "reviewer": "Reviewer_ccoi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths\n- The experiments show a clear improvement with <200 samples in the non-confounded dataset, demonstrating usefulness in the small-data regime.\n- the main paper is well written, and the ideas where clearly articulated\n- the method is simple yet empirically effective \n\nMy concerns are mainly about the theory being a little loose.\n\nWeaknesses\n- Unclear whether \"unbiased\" is the right word in proposition 1. Did the authors mean consistent in the sense that given infinite data in both $D_{prv}$ and $D_{std}$, $\\hat{q}$ will converge the truth? \n- The theory statements do not show a quantitative improvement that depends on the size of the $D_{prv}$. Without such statements it is unclear which parts of the algorithm require how much confounded data to improve over the naive/no-privilege case.\n- The proofs were hard for me to follow (see questions). More step by step writing out would save readers a lot of time.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method to leverage offline data that could be confounded along with online data to improve sample-efficiency in model-based RL.\n\nWorking with a kind of POMDP where the hidden state can affect the action being taken, the paper develops a way to use both the confounded data and the data collected in an online manner. The key idea is to introduce a latent variable $z_t$ to model the hidden state $s_t$, and get \"supervision\" for $z_t$ from both the confounded and non-confounded data sources. The paper shows clear experimental improvements over the naive modeling approaches and prove some theoretical evidence toward how the proposed method improves.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n- The authors should provide more intuition about what the results of theorem 1 say about performance improvement.\n  - For example, the upper bound can be vacuous for large $T$ and $a_t \\perp h_t$ because the only negative term gets very small and the upper bound is a number close to 1; so the upper bound reads something like \"probability of trajectory < 1\".\n - Can the authors better explain where and how theorem 1 is useful to think about?\n\nSome proof questions : Can you explain the proof step in the following places\n- The equation after the sentence \"Then for every $t\\geq 1$ we can further write ..\"\n- And the following step about \"By recursively decomposing every .. \"\n\nMaybe for discussion and not necessary for the paper\n\n- The paper would be much better if you were to quantify what \"strictly better generalization\" is. The proof only tells me that a smaller set of models also match the constraints in the data generation $D_{prv}$, but this only tells me that upper bounds on generalization are better. \"Strictly better generalization\" implies that there is a separation in the generalization error for the no privilege case can be lower bounded at a larger value than the generalization error achieved by the case where confounded data is available.\n\nI do understand that such theory can get complicated, especially with multi-step objectives. It might be useful to point out the gap in the discussion and cite some causal inference papers that show more direct guarantees and discuss similar ideas : 1. https://arxiv.org/abs/2103.16689, 2. https://arxiv.org/pdf/2011.08047.pdf\n\nMinor\n\n- In equation 5, why are the log probabilities not standardized by the size of the data?\n- Not sure \"POMDP constraints\" are defined well? I inferred it as graph independence constraints but be more specific please.", "broader_impact_concerns": "All methods that better estimate causal models come with some risk of mis-use. The paper does not contribute risk beyond that.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f2L0Rvl0sQ", "reviewer": "Reviewer_iFKF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. Utilizing confounded observational data for RL is important and interesting. On the one hand, the authors highlighted that naive utilizing the confounded observational data is not effective, which is also demonstrated in the toy examples. On the other hand, the authors proposed a correct way of utilizing confounded observation and justified its effectiveness via theory and experimentation.\n\nWeakness & Questions:\n1. The benefit demonstrated by theory seems limited to me. In particular, the theory shows that the proposed method with privileged observation generalizes better than the proposed method without privileged observation, thus demonstrating the benefit of incorporating confounded observation.  But the latent model is not always required to fit transitions of POMDPs. For instance, it is possible to directly fit the observational model without any latent model [e.g., 1 and plenty of references therein]. Or, one could simply regress observations with respect to history. Thus, the advantage of utilizing confounded data demonstrated by the theory might be limited. \n2. The asymptotic theory requires near-infinite confounded observation to establish benefit over methods without confounded observation. Would the cofounded data be useful if their access is also limited? \n3. Is the latent transition model identifiable with the observational data for generic POMDPs? As a sanity check, I think the latent state transition is not identifiable based purely on the confounded data. Otherwise, one can recover the counterfactual (the do-probability of observations o_{t+1} given do(a_t) and o_t) by marginalizing the transition over the hidden states. The latent states are also not identifiable based purely on the online interventional data for generic POMDPs. Does their combination allows such identification of latent model and how does that happen?\n\n\n\n\n\n[1] Liu et al., When Is Partially Observable Reinforcement Learning Not Scary? 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to utilize offline confounded observational data to enhance the sample efficiency of online reinforcement learning (RL). To this end, (i) the authors model the model-based RL with confounded observational data as a causal inference problem and import do-calculus as the underlying method for such inference. (ii) The authors then propose a generic method to learn the transition model based on both offline confounded and online interventional data.\n\nTo support their proposed method, (i) the authors first prove that their proposed transition is unbiased and generalizes better than inference with online data (asymptotically). (ii) The authors conducted experiments on a toy example environment and show that their proposed method has better performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Adding some discussions to address my concerns raised in the above sections can make this paper stronger. I am particularly interested in the identifiability and non-asymptotic behavior of the proposed algorithm (See Question 2 and 3 of the above section).\n\nMinor: It would be better if the authors could discuss more regarding the details of the algorithm (e.g., adding a pseudocode algorithm in the paper). This could help potential audiences adopt the proposed methods to real-world POMDP challenges.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CAOxxxsLhz", "reviewer": "Reviewer_9ush", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe main strengths of the paper are the novel formulation of how to use interventional and observational data in a model-based setting, together with proving generalization guarantees. This clearly makes a step in an interesting direction, and I believe is of interest to the general audience in RL / causal inference. Also, I find the generalization of Manski's bound as interesting. Overall, I am positive about the paper.\n\n## Weaknesses\n\nThe largest concern is the framing of the paper, which can be easily amended. In particular, note that:\n\n* the paper title is too broad -- I believe it should contain the words \"model-based\"\n* the abstract and introduction do not mention that the setting is model-based (until the contribution bullets); also, the introduction is clearly missing references to existing works in causality & RL (these references are only discussed at the very end of the paper). From reading the intro, the reader may think this is the very first paper on causality & RL, which is not the case.\n\nOtherwise, I am also wondering if Section 3 is a major contribution of the paper (although I do appreciate Sections 4, 5 which follow from it), and if it should be listed as such in the contribution bullets.   \n\n### Minor comments\n\nTypos:\n* \"they it might get\" page 2,\n* $p_{obs}$ appears at start of Section 3, not sure if it is defined at this stage? probably just need to say $p_{prv}$,\n* in a couple of places, there seem to be spaces before punctuation; would be nice to remove those. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discusses an offline reinforcement learning setting in which both interventional and observational (confounded) data are available. The focus of the paper is on a model-based setting, and the paper introduces a latent-variable formulation, which allows a learning procedure that can leverage both the observational and interventional data.\n\nIn order, the main contributions of the paper are:\n1. formulating the latent-variable MLE learning procedure in Eq. (5),\n2. describing an inference procedure for obtaining an estimator (paragraph Inference).\n3. proving that the approach gives (a) an unbiased estimator (Prop. 1), and also has better generalization guarantees compared to using interventional data only,\n4. demonstrating empirically the method, on a range of examples.\n\nOverall, the paper is well-written, and in most places clear and easy to follow. The key theoretical result is contained in Appendix D, which I was able to check, and I believe the proof is correct (although I make minor suggestions below for how to make the proof easier to follow).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n* adding model-based to the paper title,\n* improving the introduction to account for previous work in the field,\n* adding more annotation in Appendix D; for instance, the transition from the first display to the second display on page 23 could have a reference to the first display of the proof on page 22; Furthermore, I wonder if the result in the third display on page 23 could be done sequentially? Also, perhaps adding underbrace annotation would help streamline the proof reading, e.g. using $\\underbrace{q(\\dots)}_{\\leq 1}$. I note that this is an interesting proof, and I would not be surprised if other readers will try to go through it. Making such adaptations could help the readers a bit.", "broader_impact_concerns": "No major concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is overall well written and the reviewers generally agree that it is a useful contribution. I concur, but have some reservations about the quality of the theory. The decision is Accept with Minor Revisions, related to the theory, that I will check. 

The theory has some rigor issues that should be addressed. 
1. Proposition 1 about consistency does not clarify what conditions are required on the probabilistic models. The work [32] is cited about maximum likelihood estimators, to state that MLE will be consistent. But MLE is not always consistent (e.g., what if you lack identifiability?). You should check the conditions for your MLE problem. 

2. Corollary 1 needs to be modified. The claim is that you have a strict subset. This is intuitively true, but its not clear it is actually shown. Is it possible for those two sets to be the same? (The one with q(tau|i = 0) = p(tau | i = 0) and without). It is not clear to me why showing that the qhat(stuff) > 0 automatically ensures you have a strict subset. 

3. One other minor comment. You assume infinite observational data. It is possible a finite sample result could be given, if you use a high probability result instead. This may not be too difficult to obtain, since you just use D_prv being infinite in the last part of the proof of Theorem 1, to say two distributions are the same. You might instead have a probabilistic claim, and add a small error term to account for a lack of perfect equivalence. 

Another option is to avoid writing a formal result in the first place, since that is not the main contribution of this work anyway. It is intuitive that adding the observational data acts like a regularizer. It is also intuitive that MLE should behave as expected. You might simply want to discuss, more informally, why this approach should have the desired properties, without writing explicit propositions and corollaries. You could still keep Theorem 1, if you think it is an important result, but explain a bit more in words why it is important, and maybe even how it relates to the result from Manski (did they also assume infinite observational data? if not, what did they do?).
Decision: Accept (poster)