Adversarial Counterfactual Environment Model Learning
Authors: Xiong-Hui Chen
Yang Yu
Zhengmao Zhu
ZhiHua Yu
Chen Zhenjun
Chenghe Wang
Yinan Wu
Hongqiu Wu
Rong-Jun Qin
Ruijin Ding
Huang Fangsheng
Venue: Submitted to ICLR 2023 (2023)
Keywords: offline environment model learning
reinforcement learning
causal inference

Abstract: A good model for action-effect prediction, i.e., the environment model, is essential for sample-efficient policy learning, in which the agent can take numerous free trials to find good policies. Currently, the model is commonly learned by fitting historical transition data through empirical risk minimization (ERM). However, we discover that simple data fitting can lead to a model that will be totally wrong in guiding policy learning due to the selection bias in offline dataset collection. In this work, we introduce weighted empirical risk minimization (WERM) to handle this problem in model learning.  A typical WERM method utilizes inverse propensity scores to re-weight the training data to approximate the target distribution. However, during the policy training, the data distributions of the candidate policies can be various and unknown. Thus, we propose an adversarial weighted empirical risk minimization (AWRM) objective that learns the model with respect to the worst case of the target distributions. We implement AWRM in a sequential decision structure, resulting in the GALILEO model learning algorithm. We also discover that GALILEO is closely related to adversarial model learning, explaining the empirical effectiveness of the latter. We apply GALILEO in synthetic tasks and verify that GALILEO makes accurate predictions on counterfactual data. We finally applied GALILEO in real-world offline policy learning tasks and found that GALILEO significantly improves policy performance in real-world testing.

Under review as a conference paper at ICLR 2023
ADVERSARIAL COUNTERFACTUAL ENVIRONMENT MODEL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
A good model for action-effect prediction, i.c., the environment model, is essential for sample-efficient policy learning, in which the agent can take numerous free trials to find good policies. Currently, the model is commonly learned by fitting historical transition data through empirical risk minimization (ERM). However, we discover that simple data fitting can lead to a model that will be totally wrong in guiding policy learning due the selection bias in offline dataset collection. In this work, we introduce weighted empirical risk minimization (WERM) to handle this problem in model learning. A typical WERM method utilizes inverse propensity scores to re-weight the training data to approximate the target distribution. However, during the policy training, the data distributions of the candidate policies can be various and unknown. Thus, propose an adversarial weighted empirical risk minimization (AWRM) objective that learns the model with respect to the worst case of the target distributions. We implement AWRM in a sequential decision structure, resulting in the GALILEO model learning algorithm. We also discover that GALILEO is closely related to adversarial model learning, explaining the empirical effectiveness of the latter. We apply GALILEO in synthetic tasks and verify that GALILEO makes accurate predictions on counterfactual data. We finally applied GALILEO in real-world offline policy learning tasks and found that GALILEO significantly improves policy performance in real-world testing.
INTRODUCTION
good environment model is important for sample-efficient pecision-making policy learning tech- niques like reinforcement learning (RL) (James & Johns, 2016). The agent can take trials with this model find better policies, then the costly real-world trial-and-errors can be saved (James & Johns, 2016; Yu et al., 2020) or completely waived (Shi et al., 2019). In this process, the core of the models to answer queries on counterfactual data unbiasedly, that is, given states, correctly answer what might happen if we were to carry out actions unseen in the training data (Levine et al., 2020).
Requiring counterfactual queries makes the environment model learning essentially different from standard supervised learning (SL) which directly fits the offline dataset. In real-world applications, the offline data is often collected with selection bias, that is, for each state, each action might be chosen unfairly. Seeing the example in Fig. (aa), keep the ball following a target line, behavior policy will use a smaller force when the ball's s location is closer to the target line. When a dataset is collected with selection bias, the association between the (location) states and (force) actions will make SL hard to identify the correct causal relationship of the states and actions to the next states respectively. Then when we query the model with counterfactual data, the predictions might be catastrophic failures. In Fig. (cc), it mistakes that smaller forces will increase the ball's next location.
Generally speaking, the problem corresponds testing in another dataset with a shifted distribution (i.e., the dataset generated by counterfactual queries), which is beyond the SL's capability as it violates the independent and identically distributed (i.i.d.) assumption. The problem is widely discussed in causal inference for individual treatment effects (ITEs) estimation in many scenarios like patients' treatment selection (Imbens, 1999; Alaa & van der Schaar, 2018). ITEs are the effects of treatments on individuals, which are measured by treating each individual under a uniform policy and evaluate the effect differences. Practical solutions use weighted empirical risk mininization (WERM) to handle this problem (Jung et al., 2020; Shimodaira, 2000; Hassanpour & Greiner, 2019). In particular, they estimate an inverse propensity score (IPS) to re-weight the training data approximate the data distribution under a uniform policy. Then a model is trained under the reweighted data distribution. The distribution-she problem is solved as ITEs estimation and model training under the same distribution.
challenge of training the model in one dataset but

---

Under review as a conference paper at ICLR 2023
(4, (4, a 0013 (6, 62.5) a -
vañation of next y prediction
1a-ayaaa location
Heal MARM-oracle GAULEE
target line
70
1 (3, a=0.4
(5. 62.3)
50
60
2
SR
(2, 45) 22.45)a=111 11.17
a =0.02
40
30
real
AWRM-oracle GALILEO
20
(1, 10) a =3,5
10
X
-2.0-0.5 0.0 0. 1.0 variation of action value
action
(a) A selection bias example
(b) prediction for training
(c) response curve
Figure I:
An example of selection bias and predictions under counterfactual queries. Subfigure (a) shows how the data is collected: a ball locates in 2D plane whose position is (Iz, M1) at time t. The ball will move to (It+1, V1+1) according to 24+1 = 2e + and yt+1 ~ NN + Qe, 2). Here, a, is chosen by a control policy a, ~ ((66 y,)115, 0.05) parameterized by Ø, which tries to keep the ball near the line = Φ. In Subfigure (a), e is set to 62.5. Subfigure (b) shows the collected training data (grey dashed line) and the two learned models' prediction of the next position of y. All the models discovered the relation that the corresponding next y will be smaller with a larger action. However, the truth is not because the larger at causes smaller yt-11, but the policy selects small a, when is close to the target line. When we estimate the response curves by fixing and reassigning action a with other actions a + Aa, where Aa E [-1, 1] is a variation of action value, the model of SL will exploit the association and give opposite responses, while in AWRM and its practical implementation GALILEO, the predictions are closer to the ground truths. The result is in Subfigure (c), where the darker a region is, the more samples are fallen in.
The selection bias can be regarded as an instance of the problem called "distributional shift" in offline model- based RL, which has also received great attention (Levine al., 2020; Yu at., al., 2020; Kidambi et al., 2020; Chen et al., 2021). However, previous methods, where naive supervised learning is used for environment model learning, ignore the problem in environment model learning but handling the problem by suppressing the policy explo- ration and learning in risky regions. Although these methods have made great progress in many tasks, so far, how to learn a better environment model that can alleviate the problem for faithful offline policy optimization has rarely been discussed.
to Reviewer rQ79
dataret
850 175
KINiM caadd
120
maar square ener
120
In this work, for faithful offline policy optimization, we in- troduce WERM environment model learning. The extra challenge of model learning for policy optimization is that we have query numerous different policies' feedback besides the uniform policy for finding a good policy. Thus the target data distribution to reweight can be various and unknown. To solve the problem, we propose an objective called adversar- ial weighted empirical risk minimization (AWRM). AWRM introduces adversarial policies, of which the corresponding counterfactual dataset has the maximal prediction error of the model. For each iteration, the model is learned to be as small prediction risks as possible under the adversarial counterfactual dataset. However, the adversarial counterfac- tual dataset cannot be obtained in the offline setting, thus we derive an approximation of the counterfactual data dis- tribution queried by the optimal adversarial policy and use a variational representation to give a tractable solution to learn model from the approximated data distribution. As a result, we derive a practical approach named Generative Adversarial offLIne counterfactual. Environment mOdel learning (GALILEO) for AWRM. Fig. shows the differ- ence in the prediction errors learned by these algorithms. We also discover that GALILEO is closely related to existing generative-adversaaial model learning techniques, explaining the effectiveness of the latter.
76
20
adversarial polloy's parameter of the courte/factual dataset
Figure 2: An illustration of the predic- tion error in counterfactual datasets. The prediction risks is measured with mean square error (MSE). The error of SL. is small only in training data (sb 62.5) but becomes much larger in the dataset "far away from" the training data. AWRM-oracle selects the oracle worst counterfactual dataset for train- ing for each iteration (pseudocode is in Alg. 1) which reaches small MSE in all datasets and gives correct response curves (Fig. I(c)). GALILEO approxi- mates the optimal adversarial counter- factual data distribution based on the training data and model. Although the MSE of GALILEO is bit larger than SL in the training data, in the coun- terfactual datasets, the MSE is on the same scale as AWRM-oracle.
Experiments are conducted in two synthetic and two realistic environments. The results in the synthetic environments show that GALILEO can reconstruct correct responses for counterfactual queries. The evaluation results in two realistic environments also demonstrate that GALILEO has
2

---

Under review as a conference paper at ICLR 2023
better ability in counterfactual query compared with baselines. We finally search for policy based on the learned model in a woald online platform. The policy significantly improves performance in concerned business indicators.
RELATED WORK
We give related adversarial algorithms for model learning in the following and leave other related work in Appx. F. GANTIE (Yoon ct al., 2018) uses generator to fill counterfactual outcomes for each data pair and a discriminator to judge the source (treatment group or control group) of the filled data pair. The generator is trained to minimize the output of the discriminator. GANITE is trained until the discriminator cannot determine which of the components is the factual outcome. Bica et al. (2020) propose SCIGAN to extend GANITE continuous treatment effect estimation (a.k.a., dosage-responss estimation) via a hierarchical discriminator architecture. In real-world applications, environment model learning based on Generative Adversarial Imitation Learning (GAIL) has also been adopted for sequential decision- decision-making problems (Ho & Ermon, 2016). GAIL is first proposed for policy imitation (Ho Ermon, 2016), which uses the imitated policy generate trajectories by interacting with the environment. The policy is learned with the trajectories through RI which maximizes the cumulative rewards given by the discriminator. Shi et al. (2019); Chen et al. (2019); Shang et al. (2019) use GAIL for environment model learning by regarding the environment model as the generator and the behavior policy as the "environment" in standard GAIL. These studies empirically demonstrate that adversarial model learning algorithms have better generalization ability for counterfactual queries, while our study reveals the connection between adversarial model learning and the WERM through IPS. Our derived practical algorithm GALILEO is closely related to the existing adversarial model learning algorithms, explaining the effectiveness of the latter.
3 PRELIMINARIES
3.1 SINGLE-STEP INDIVIDUALIZED TREATMENT EFFECTS ESTIMATION AND WEIGHITED EMPIRICAL RISKS MINIMIZATION
We first introduce individualized treatment effects (ITEs) estimation (Rosenbaum & Rubin, 1983), which can be regarded as the scenario in which the environment model has only a single step. ITEs typically defined ITE(2) := EIM'(y (2|z,, A) = 1, = z| E[M (y/æz 0) 0, X = xr. where y is the feedback of the environment M M (ylæ, a). X denotes the state vector containing pre-treatment covariates (such age and weight), A denotes the treatment variable which is the action intervening to the state X, and A should be sampled from a uniform policy. In the two- treatment scenario. is in 10 where 1 is the action intervene and 0 is the action do nothing. A correct ITES estimation should be done in Randomized Controlled Trials (RCT) in which we have the same probability of samples of A use lowercase 2, a and to denote samples of random variables X, A and Y, and use A and y to denote space of the samples. In practice, we prefer to estimate ITEE under observational studies. In observational studies, datasets are pre-collected from the real world by a behavior policy such as a human-expert policy. In this case, a common approach for estimating the ITES can be I7E(2;) a, (Z,, a,)) + (1 a,) (M, 1 a.) y/²) (Shalit et al., 2017) in deterministic prediction, where r, and vf" denote the covariate and factual feedback of the i-th sample, and M E .M denotes an approximated feedback model. M is the space of the model. In this formulation, the training set is an empirical factual data distribution Pr {(z,, a;));" and the testing set is an empirical counterfactual data distribution Par { 1 a:) Jr If a does not sample from a discrete uniform policy, i.e., the policy has selection bias, Pp and PCF will be two different distributions, which violate the i.i.d. assumption of standard supervised learning. In stochastic prediction, rfE(2) [y](,, 1)] RB[(() ar, 0)) and the counterfactual distribution for testing is the dataset with action sampling from a uniform policy.
and A = for each X. Here we
Generally speaking, in ITES estimation, the risks of queries under counterfactual data are caused by the gap between the policy in training and testing data distributions. Without further processing, minimizing the empirical risks cannot guarantee the counterfactual-qqery risks being minimized. Assuming that the policy in training data j satisfies μ(alzz) > 0, Va E A, Vx E X' (often named overlap assumption), classical solution to handle the above problem is weighted empirical risk minimization (WERM) through an inverse propensity scoring (IPS) term w (Shimodaira, 2000; Assaad et al., 2021; Hassanpour & Greiner, 2019; Jung al., 2020):
3

---

Under review as a conference paper at ICLR 2023
Definition 3.1. The learning objective of WERM through IPS is formulated as min L(M)
min E Bz,a.vrrp (1(z, (M) MEM MEM
s.t. w(x, a) = 3()(x))
(I)
plaaz))
where 3 and denote the policies testing and training domains, and p'M- is the joint probability Piss (I, a, y) (())) (y/æ, a) in which po(z) is the distribution of state. M is the model space. is loss function for model learning.
The is also known as importance sampling (IS) weight, which corrects the sampling bias. In this objective, w is to align the training data distribution to the testing data. By selecting different w to approximate w learn the model M. current environment model learning algorithms for ITEs estimation are fallen into the framework. In standard supervised learning and some works for ITEs estimation (Wager & Athey, 2018; Weiss et al., 2015), (<x, a)
the distribution-shift problem is ignored. In Shimodaira (2000); Assaad et al. (2021); Hassanpour & Greiner (2019),
y (i.e., B uniform policy) for balancing treatment and control group, where is approximation of behavior policy μ. Note that it a reasonable weight in ITEs estimation: ITES are defined to evaluate the effect of each state between treatment and control behavior under uniform policy.
3.2 SEQUENTIAL DECISION MAKING SETTING
Decision-making processes in sequential environment often formulated into Markov Decision Process (MDP) (Sutton & Barto, 1998). MDP depicts an agent interacting with the environment through actions. In the first step, states are sampled from an initial state distribution zo po(z)) Then at each time-step t E (0. 1, 2..... J. based on the state 24 E X, then the agent receives a reward rt from a reward function r(1g, 0z) E a,) built in the
the agent takes an action E A through policy I <T (22a E II
R and transits to the next state 2z+1 given by transition function M environment. II, X, and denote the policy, state, and action spaces.
4 METHOD
In this section, we first propose new offline model-learning objective based on Def. 3.1 for policy optimization tasks in Sec. 4.1; In Sec. 4.2, we derive a tractable solution the proposed objective; Finally, we give practical implementation in Sec. 4.3.
4.1 PROBLEM FORMULATION
For offline policy optimization, we require the environment model to have generalization ability in counterfactual queries since we need to query numerous different policies' correct feedback from M. Referring to the formulation of WERM through IPS in Def. 3.1, policy optimization requires M to minimize sounterfactual-query risks under numerous unknown different policies rather than specific target policy B. More specifically, the question is: If /3 is unknown and can be varied, how should we reduce the risks in counterfactual queries? In this article, we call the model learning problem in this setting "counterfactual environment model learning" and propose new objective to handle the problem. To be compatible with multi-step environment model learning, we first define a generalized WERM through IPS based on Def. 3.1.
Definition 4.1. Given the MDP transition function frunsttionfuncctinn M that satisfies M (2'1,, a) > O, Va E X. Ya E A, Va' X and μ satisfies p(a]xx) > O, Va A. Vr X. the learning objective of generalized WERM through IPS is formulated as
min L(M) min Ex, [w(x, a, (',, MEM
MEM
(x,
s.l. w(x, a, x') Pi (2,, (1))'
where pa is the training data distribution (collected by policy μ), e is the testing data distribution (collected by policy B). We define Ens (Ir, a, x') == (xM [x, a) x²) for brevity.
In MDP, given any policy Pîr- (x, a, ( (a) (æ' a) where pī.- (x) denotes the occupancy measure of x for policy x. which can be defined as p5.- (x) == Fx " PPr(a afwo, Mn) (Sutton & Barto, 1998: Ho & Ermon, 2016) where
4

---

Under review as a conference paper at ICLR 2023
Pr" [2:, 2/ M*] is the state visitation probability that the policy visits at time-step by executing in the environment M and starting at the state 2u, and y E [0. 1] is the discount factor. Here we also define PT-- (z, a) := Pir- (a] for simplicity. In this definition, w can be rewritten
P (x.aa)
as: w(x, a, P2-- (2)
Pa-- (xra] In single-step environments, for any policy x,
Po(e) [(el=) and the objective is degraded to Pîr- (x) Po(z)). Then we have w(z, x²) po(z))t((z) p(a/z)
Eq. 1. Therefore, Def. 3.1 is special case of this generalized form.
Since /3 for counterfactual queries is unknown and can be varied in policy optimization, to reduce the risks in counterfactual queries in this scenario, introduce adversarial policies which can induce the worst performance of the model predictions and propose to optimize WERM under the adversarial policies. In particular, we propose Adversarial Weighted empirical Risk Minimization (AWRM) based on Def. 4.1 to handle this problem.
Definition 4.2. Given the MDP transition function M", the learning objective of adversarial Weighted empirical risk minimization through IPS is formulated as
M min max L(P M) MEM BETT
) min max E xx ( MEM BEN
(s,
s.t. w(x, alp2a.) = (s,
where the re-weighting term w(x, alp'2s)))) ) is conditioned on the data distribution Phs- of the adver- sarial policy B. In the following, we will ignore PA and use w(z, a) for brevity.
According the definition of MDP, w(x, a, a,"²) = w(x,a) a) since the transition probability in the ratio can be canceled. Eq. 3 minimizes the maximum model loss under all counterfactual data distributions P (z, a, x²), 3 e II to guarantee the generalization ability for counterfactual data queried by policies in IT.
4.2 TRACTABLE AWRM SOLUTION
In this section, propose tractable solution to optimize Eq. 3.
The full derivations can be found in the Appx. A. We choose lM to be the negative log-likelihood loss and derive the solution to Eq. 3:
L(pas., M) = Egarpi- [u(s,, alp EEM- log M [x, [a.,)))
where EM- [-] denotes (a((aa) [.]. The core problem is how to construct the data distribution
P the the best-response policy 3 in M* as it is costly to get extra data from M real-world applications. Instead of deriving the optimal 3*, our solution is to offline estimate the optimal adversarial distribution Par- with respect to M. then we can construct surrogate objective to optimize M without directly querying the real environment M
4.2.1 OPTIMAL ADVERSARIAL DATA DISTRIBUTION APPROXIMATION
Ideally, given any M, it is obvious that the optimal / is the one that makes P (ar, a) assign all densities to the point that has the largest negative log-likell ligllikelihood. However, searching for the maximum is impractical, especially in continuous space. To give a relaxed but tractable solution, we add an L2 regularizer to the original objective Eq. 3:
min max L( M) min max MEM BEll
Ex Ezawgp (w(x, a) I- log M (x'lIxaa)) - 2/ i (p-r-((--)1) 1|
MEM BEll
(4)
where a denotes the regularization coefficient of P.g- and ( - Sx.A (z, dade.
Then we can approximate the optimal distribution px- via Lemma. 4.3.
Lemma 4.3. Given any M in L(pgr M), the distribution of the ideal best-response policy 3 satisfies:
T (2. _ (DRI(MM (11,, a), ((1,,a))) + Har (z,a)),
(5)
0M where DKL ( a), a)) is the Kullback-eeibler (KL) divergence between M M a) and (-12,.00 a), H (z, a) denotes the entropy of M (-122, a), and a is the regularization coefficient in Eq. 4 and also normalizer of Eq. 5.
5

---

Under review as a conference paper at ICLR 2023
Intuitively, P'M- has larger densities on the data where the divergence between the approximation model and the real model (i.e., DKL ( a). M(- a])) is larger or the stochasticity of the real model (i.e., HM-) is larger. However, the integral process of Dxt. in Eq. 5 is intractable in the offline setting it explicitly requires the conditional probability function of M Our solution to solve the problem is utilizing the offline dataset Dreal as the empirical joint distribution o (æ, a, x') and adopting practical techniques for distance estimation on two joint distributions, like GAN (Goodfellow et al., 2014; Nowozin et al., 2016), to approximate Eq. 5. To adopt that solution, we should first transform Eq. 5 into form under joint distributions. Without loss of generality, we introduce an intermediary policy K, of which μ can be regarded as specific instance. Then we have M [r, a) = Pig (I, a, me')//0% Phr (I, a) for any M if Pir (I, a) > O. Assuming Va E X, Va E A, P%r- (x,, a) > 0f PM-- (2, a) > O, which will hold when &. overlaps with μ4, then Eq. 5 can transform to:
to Reviewer beJS
a0(2),)) (Je Pās- a.) log ai,-(x,,,''dd pās (2, (log (2, HM (1, (1, ) (x,
Vi (2,
tive is the integrated function where ao( a) = aMPir- a). We notice that the form PM-- log Pee in reverse KL divergence, which is an instance of function in f-divergence (Ali & Silvey, 1966). Replacing that form with function, we obtain a generalized representation of par.:
ao (z (Lx
(J.05er par (s,
(,,((, Mig- (I., a,,,,,
((ir(22))) Phr (e, () Hm (z, (2,4))
dx' piee (2.e((x.a)
(6)
where f: 1R+ -> IR is convex and lower semi-continuous (Ls.c.) function. PMr- gives generalized representation of the optimal adversarial data distribution to maximize the error of the model. Based on Eq. 6, we have a surrogate objective of AWRM which can avoid querying M to construct PMT-
Theorem 4.4. Let p as the data distribution of the best-response policy 3r in Eq. 4 under model Me parameterized by 0, then can find the optimal 0 of ming maxsen L( Ma) (Eq. 4)
via iteratively optimizing the objective Bu41 ming L(pas-,Ma where 8 is approximated via the last-iteration model Mo,- Based on Corollary A.7, we derive an upper bound objective for
ming L(P Ma):
(F (x, (2,,a,z)) Pie- a.)
(Pāto,
Pisy (2, Pliee (2)
0.t1 mjn Byyr (o( log Mo(x'|z,0)
Has (Hx-( (2,
where Epgg- [-] denotes Ez 1h is Ls..c function satisfying f'(x) S 0. X, and o0(2,, a) aM, Pig. (x, a).
Thm. 4.4 approximately achieve AWRM by using and pseudo-reweighting module W. W assigns learning propensities for data points with larger differences between distributions Phis, and Pir-- By adjusting the weights, the learning process will exploit subtle errors in any data point, whatever how many proportions it contributes, to correct potential generalization errors counterfactual data.
4.2.2 TRACTABLE SOLUTION TO THM. 4.4
In Thm. 4.4, the term f( Pit, (x,a,a")
Pioo (x,a)
Pig. (z.a) is still intractable. To solve the problem, first, we resort to the first-order approximation of f. Given some (1 - 1, E), > 0, we have
Pice (2.a.))) f( (
s(u) ~ f(1) + (")
where f' the first-order derivative of f. By Taylor's ' formula and the fact that f'(u) of the generator
function is bounded in (1 <. + {), the approximation error is no more than O Let
1) + f(1). v(क] can be approximated by sampling from datasets and f'( q(2)) can be approximated by the
p(x) in Eq. 7, the pattern f( ((((e) in Thm. 4.4 can be converted to f' p(z) p(x)
q(x)
p(x)
p(x)
corresponding variational representation T according Lemma A.9 (Nowozin et al., 2016) (see

---

Under review as a conference paper at ICLR 2023
Appendix for details). Based any specific f-function, we can represent T. with discriminator De. However, Thm. 4.4 holds only when Va: E X, fxx S 0. It can be verified that the instance f(u) = u log u (u + 1) log(u + 1), (u) = log p((u)) satisfy this condition (see Tab. 2 in the Appendix). We select that instance for the tractable solution. In summary, by computing f(-) with first-order approximation (Eq. 7) and leveraging the variational representation Tw- log D. -- to
approximate l'(
(Me we can optimize the surrogate objective in Thm. 4.4 via:
P3
Bc+1 = max Epica
Ar (z, a, x') log Mo |a, a)] + Epjr- [(HM (2, a) - Apg (2, a, x²)) log Mo a)
s.t. pp = arg max Bogr. [log Dero (IT, a, x7 ')] + Brsso [log(1 - Po (z, a, x1)]
(8)
Po
pi arg max Epgy- [og D'1 (2] Epšro, [log(1 - D21 (1),
41
where Apg (I, a, x') = log D4o (Ir, a, x') - log Det (2r, Epg []] is simplification of Ex, H and so and (pi are the parameters of Tip and Twp Based on Lemma A.9, we have f Pāro (2,0,r")
Pii- (m.,,))) = log Deg (2, a, x') and f Pārg, (2.a),
Pis- (x.a) = log (a, Note that in the process, we ignore the term (2, a) for simplifying the objective. The discussion on the impacts of removing ao(I, a) is left in App. B. In Eq. 8, the reeeih--ewiightigg module W in Thm. 4.4 is split into two terms in the RHS of the equation. The first term a generative adversarial training objective regarding Mo as the generator, while the second term is WERM through HM- A
4.3 PRACTICAL IMPLEMENTATION
To give practical implementation of the solution, extra two assumptions introduced in the process of modeling should be handled:
First, the approximation of Eq. holds only when p(z) is close to 1, which might not be satisfied. To
a(x)
handle the problem, we inject a naive supervised learning loss and replace the second term of the objective Eq. 8 when the expectation of Dxpo in Piite far away from 0.5 (1) log 0.5);
Second, the overlap assumption of k. In practice, we need to use the real-world data to construct the distribution Par-- and the generative data to construct Pir.. In the offline model- model-learning setting, we only have a real-world dataset D collected by the behavior policy p. We can learn policy A = l via behavior cloning based on D (Pomerleau, 1991; Ho & Ermon, 2016) and let f be the policy K. Then we can regard D as the empirical data distribution of phi-- and the trajectories collected by e in the model Mo. as the empirical data distribution of Pir.. But the assumption Vr E X, Va E A, p(a[z) > might not be satisfied, To handle the problem, in behavior cloning, we model á with a Gaussian distribution and constrain the lower bound of the variance with a small to keep the assumption holding. Besides, we add small Gaussian noises (0, ED) to the inputs of D.p to handle the mismatch between phr-- and P due to Ege.
to Reviewer beJS
value en
In Eq. 8, HM- is unknown in advance. In practice, we HM, to estimate it. More specifically, the neural network of M, can be modeled with a Gaussian distribution. The variance of the Gaussian distribution is modeled with global variables Σ for each dimension of output. We estimate HM- with the closed-form solution of Gaussian entropy through Σ.
to Reviewer beJS
Based on the above techniques, we propose Generative Adversarial offL counterfactuaL Environment mOdel learning (GALILEO) for environment model learning. GALILEO can be adopted in both step and sequential environment model learning. The detailed implementation and comparison to previous adversarial methods are in Appx. E.
EXPERIMENTS
In this section, we first conduct experiments in synthetic environments (Bica al., 2020) to verify GALILEO on counterfactual queries I and the compatibility of GALILEO in se- quential and single-step environments. We select Mean Integrated Square Error MISE = E [A (Ml(,),, Mfj=,, a))² da as the metric, which is a commonly used metric to mea- sure the accuracy in counterfactual queries by considering the prediction errors in the whole action space. Then we analyze the benefits of the implementation techniques described in Sec. 4.3 and
the code will be released after the paper published.
7

---

Under review as a conference paper at ICLR 2023
GALILEO
GALILED
8
SI.
root MISE
IPW SCIGAN
root MISE
25
SOIANN
5
15
e1_00.2
=1_
e1
00.08
00.2.
60.2
e0.2. 2..0000000
.0.05
od.08 _p0.2
e0.05 $0.005
10,
blnc2
bes20
11. __a.is?
11. baas
17. J_eeee
12 plass
12, blabs
12 place
(a) GNFC
(b) TCGA
Figure 3: Illustration of the performance in GNFC and TCGA. The grey bar denotes the standard
error (x0.3 for brevity) of 3 random seeds.
the problems without them. Finally, deploy GALILEO in two complex environments: MuJoCo in Gym (Todorov ct al., 2012) and a real-world food-delivery platform to test the performance of GALILEO in difficult tasks. The algorithms compared are: (1) Supervised Learning (SL): us- ing standard empirical risk minimization for model learning; (2) Inverse Propensity Weighting (IPW) (Spirtes, 2010): standard implementation of WERM based IPS; (3) eStimating the effects of Continuous Interventions using GANs (SCIGAN) (Bica et al., 2020): an adversarial algorithms for model learning used for causal effect estimation, which can be roughly regarded as partial implementation of GALILEO (Refer to Appx. E.2). We give a detailed description in Appx. G.2.
to Reviewer rQ79
5.1 TEST IN SYNTIETIC ENVIRONMENTS
Test in sequential environments Since there does not exist a task specifically designed for se- quential environment model learning with selection bias, we construct a task, General Negative Feedback Control (GNFC), which can represent classic type of task with policies having selection bias. Fig. 1(a) is also an example of GNFC. We give detailed motivation, the effect of selection bias, and other details in Appx. G.I.I. We construct tasks on GNFC by adding behavior policies with different scales of uniform noise U (-e,, e) with different probabilities p. In particular, with e E {1.0, 0.2, 0.05} and p E {1.0, 0.2, 0.05), , we construct 9 tasks and name them with the format of "e* _p*.. For example, e1 e1_p0.1111111 2 is the task with behavior policy injecting with U(-1, 1) with 0.2 probability. The results of GNFC tasks are summarized in Fig. 3(a) and the detailed results can be found in Tab. 8. The results show that the property of the behavior policy (i.e., and p) dominates the generalization ability of the baseline algorithms. When e = 0.05, almost all of the basclines fail and give completely opposite response curve (see Fig. 4(a) and Appx. H.2). IPW still perform well when 0.2 < e S 1.0 but fails when e = 0.05. <= 0.2. We also found that SCIGAN can reach better performance than other baselines when are unstable. GALILEO is the only algorithm that is robust to the selection bias and outputs correct response curves in all of the tasks. Based the experiment, we also indicate that the commonly used overlap assumption is unreasonable to certain extent especially in real-world applications since it is impractical to inject noises into the whole action space. The problem of overlap assumption being violated should be taken into consideration otherwise the algorithm will be hard to use in practice if is sensitive to the noise range.
0.05, p <= 0.2, but the results in other tasks
roal GALILEO
real
avaraged response
Test in single-step environments Previous experiments on counterfactual environment model learning are based on single-step semi-synthetic data simulation (Bica et al., 2020). Since GALILEO is compatible with single-step en- vironment model learning, we select the same task named TCGA in Bica et al. (2020) to test GALILEO. Based on three synthetic response functions in TCGA, we construct 9 tasks by choosing different parameters of selection bias on μ which is constructed with beta distribution, and de- sign a coefficient to control the selection bias of the beta distribution. We name the tasks with the format of "t? "?__iias" For example, t li_bbaa 2 is the task with the first response functions and c = 2. The detail of TCGA is in Appx. 1.1. The results of TCGA tasks are summarized in Fig. 3(b) and the detailed results can be found in Tab. 9 in Appendix. We found the phenomenon in this experiment is similar to the one in GNFC, which demonstrates the compatibility of GALILEO to single-step environments. We also found that the results of IPW are unstable in this experiment. It might be because the behavior policy is modeled with beta distribution while the propensity score f is modeled with Gaussian distribution. Since IPW directly reweight loss function with z. the results are sensitive
Reccoaa responge Saeee
PW SCIGAN
65
0.2 0.2 0.4 3.6 5.8 1.0 nommblzzd action
02 04 5.6 9.5 1.0 normalized action
(a) c0.05_p0.2
(b) 10 bia220
Figure 4: Illustration of the averaged re- sponse curves.
8

---

Under review as a conference paper at ICLR 2023
Table I: Results of policy performance directly optimized through standard SAC (Haarnoja et al., 2018) using the learned dynamics models and deployed in MuJoCo environments. MAX- MAX-REEURN is the policy performance of SAC in the MuJoCo environments, and "avg. norm." " is the averaged normalized return of the policies in the 9 tasks, where the returns normalized to lie between O and 100, where a score of corresponds to the worst policy, and 100 corresponds to
Task
Hopper
Walker2d
HallCheetah
arg. somm.
Horizon
H=10
H=20
H=H0
H=10
H=20
H=10
H=10
H=20
H=40
GALILEO
13.0
332 0.1
533 1.2
11.7 02
29,9 0.3
612 34
0.2
-1.1
-14.2 4:
51.1
SL.
05
00
46
007=02
20.1
375
04
-1.1
-13.2 +
21.1
IPW
5.4
0.
59
4
28
145
162
0.5088
0.9
19.7
SCIGAN
02
06
462
04
9.1
10=58
0.
41.8
MAX RETURN
0.22.00
13.3
71.0
60.7 4 11.1
2211 299
0.1
13.3 1.1
49.1 2:3
100.0
to the error on ja. GALILEO also models á with Gaussian distribution but the results are more stable since GALILEO does not re-weight through f explicitly.
Response curve visualization We plot the averaged response curves which are constructed by equidistantly sampling action from the action space and averaging the feedback of the states in the dataset as the averaged response. Parts of the results in Fig. 4 (all curves can be seen in Appx. H.2). For those tasks where baselines fail in reconstructing response curves, GALILEO not only reaches better MISE score but reconstructs almost exact responses.
Ablation studies In Sec. 4.3, we introduce several techniques to develop a practical GALILEO algorithm. Based on task e0. 2 2_p0__ 00 of GNFC, we give the ablation studies to investigate the effects of these techniques. As the main-body space is limited, we leave the results in Appx. H.3.
5.2 TEST IN COMPLEX ENVIRONMENTS
In MuJoCo tasks MuJoCo is benchmark task in Gym (Todorov et al., 2012; Brockman et al., 2016) where we need control a robot with specific dynamics to complete some tasks (e.g.. standing or running). We select 3 environment from D4RL (Fu et al., 2020) to construct our model learning tasks. We compare it with a standard transition model learning algorithm used the previous offline model-based RL algorithms (Yu et al., 2020; Kidambi et al., 2020), which is variant of supervised learning. We name the method OFF-SL. Besides, we also implement IPW and SCIGAN as the baselines. In D4RL benchmark, only the "medium" tasks is collected with a fixed policy, i.e., the behavior policy is with 1/3 performance to the expert policy), which is most matching to our proposed problem. So train models in datasets LuufChheeta-mmediine Hopper-medium. We trained the models with the same gradient steps and saved the models.
Talker2d-mediun and
to Reviewer beJS, j5p5, and rQ79
We first verify the generalization ability of the models by adopting them into offline model-based RL. Instead of designing sophisticated tricks to suppress policy exploration and learning in risky regions as current offline model-based RL algorithms (Yu et al., 2020; Kidambi et al., 2020) do, we just use the standard SAC algorithm Haarnoja et al. (2018) to exploit the models for policy learning to strictly verify the ability of the models. Unfortunately, we found that the compounding error will still be inevitably large in the 1,000-step rollout, which is the standard horizon in MuJoCo tasks, leading all models fail derive a reasonable policy. To better verify the effects of models on policy optimization, we learn and evaluate the policies with three smaller horizons: H E {10, 20, 40).
The results are listed in Tab. 1. We first averaged the normalized return (refer to "avg. norm.") under each task, and we can see that the policy obtained by GALILEO is significantly higher than other models (the improvements are 24% to 161%). At the same time, we found that SCIGAN performed better in policy learning, while IPW performed similarly to SL. This is in line with our expectations, since IPW only considers the uniform policy as the target policy for debiasing, while policy optimization requires querying wide variety of policies. Minimizing the prediction risks only under a uniform policy cannot yield a good environment model for policy optimization. Besides, in IPW, the cumulative effects of policy on the state distribution are ignored. On the other hand, SCIGAN, as a partial implementation of GALILEO (refer to Appx. E.2), also roughly achieves AWRM and considers the cumulative effects of policy on the state distribution, so its overall performance is better; In addition, we find that GALILEO achieves significant improvement in 6 of the 9 tasks. But in HalfCheetah, IPW works slightly better. However, compared with MAX-RETURE it can be found that all methods fail to derive reasonable policies because their policies' performances are far away from the optimal policy. By further visualizing the trajectories, we found that all the learned policies just keep the cheetah standing in the same place even going backward. This phenomenon is also similar to the results in MOPO (Yu et al., 2020). In MOPO's experiment in the medium datasets, the suncated-rollout horizon used in Walker and Hopper for policy training is set to 5, while HalfCheetah has to be set to the minimal value: 1. These phenomena indicate that HalfCheetah may still have unknown problems, resulting in the generalization bottleneck of
9

---

Under review as a conference paper at ICLR 2023
noscated cumt cumuulativo wewullliiv treatmintt officds
1.00
GALLD0
mescaled fveerindd opoae orden-lkke
0.5
0.00
resialeo averaged
0.34
0.90
0.88
0.92
0.80
Incemeent poor 2014 LEC Bieday Pal
0.80
0.00
0.0
normalized allocaled allowances
cumulative samples (rescaled to 1)
day number
(a) response curves in City A
(b) AUUC curves
(c) A/B test in the City A
Figure 5: An illustration of the performance in BAT tasks. In Fig. 5(a) demonstrate the averaged response curves of the SL. and GALILEE model in City
expected to be monotonically increasing through our prior knowledge. In Fig. 5(b), the model with larger areas above the "random" line makes better predictions in wewoomiiee-cottooledd---eo shows the daily responses in the A/B test in City A. The complete results are in Appx. H.6.
data (Betlei et al., 2020). Fig. 5(c)
the models. Besides, we also test the prediction error of the learned model in corresponding unseen "expert" and "medium-replay" datasets. The detailed results are in Appx. H.5.
In a real-world platform We finally deploy GALILEO in real-world large-scale food-delivery platform. The goal of the platform is to balance the demand from take-out food orders and the supply of delivery clerks, i.e., helping delivery clerks fulfill more orders by giving reasonable strategies. We focus on Budget A Allocation task to the Time period (BAT) in the platform (see Appx. G.1.3 for details). The goal of the BAT task is to handle the imbalance problem between the demanded orders from customers and the supply of delivery clerks in different time periods by allocating reasonable allowances to those time periods. The core challenge of the environment model learning in BAT tasks is similar to the challenge in Fig. 1. Specifically, the behavior policy in BAT tasks is human-expert policy, which tends to increase the budget of allowance in the time periods with a lower supply of delivery clerks, otherwise tends decrease the budget (Fig. 12 gives a real-data instance of this phenomenon).
We first learn model to predict the supply of delivery clerks (measured by fulfilled order amount) on given allowances, Although the SL model can efficiently fit the offline data, the tendency of the response curve is easily to be incorrect. As can be seen in Fig. 5()), with a larger budget of allowance, the prediction of the supply is decreased in SL, which obviously goes against our prior knowledge. This is because, in the offline dataset, the corresponding supply will be smaller when the allowance is larger. It is conceivable that if we learn a policy through the model of SI., the optimal solution canceling all of the allowances, which obviously incorrect in practice. On the other hand, the tendency of GALILEO's response is correct. Fig. 13 plots all the results in 6 cities.
to Reviewer rQ79 and j5p5
Second, we conduct randomized controlled trials (RCT) in one of the testing cities. Using the RCT samples, we can evaluate the generalization ability of the model predictions via Area Under the Uplift Curve (AUUC) (Betlei et al., 2020). which measure the correctness of the sort order of the model prediction in RCT samples. The AUUC further show that GALILEO gives reasonable sort order on the supply prediction (see Fig. 5(b)) while the standard SL technique fails to complete this task.
Finally, we search for the optimal policy via the cross-entropy method planner (Hafner et al., 2019) based on the learned model and deploy the policy in a real-world platform. The results of A/B test in City A is shown in Fig. 5(c). It can be seen that after the day of the A/B test, the treatment group (deploying our policy) significant improve the five-minute order-taken rate than the baseline policy (the same the behavior policy). In summary, the policy improves the supply from O. 14 1.63 percentage points to the behavior policies in the 6 cities. The details of these results are in Appx. H.6.
DISCUSSION AND FUTURE WORK
In this work, we propose AWRM which handles the generalization challenges of the counterfactual environment model learning. By theoretical modeling, we give a tractable solution to handle AWRM and propose GALILEO. GALILEO is verified synthetic environments, complex robot control tasks, and real-world platform, and shows great generalization ability on counterfactual queries.
Giving correct answers to counterfactual queries is important for policy learning. We hope the work can inspire researchers to develop more powerful tools for counterfactual environment model learning. The current limitation lies in: There are several simplifications in the theoretical modeling process (further discussion is in Appx. B), which can be modeled more elaborately Besides, experiments on MuJoCo indicate that these tasks are still challenging to give correct predictions on counterfactual data. These should also be further investigated in future work.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ahmed M. Alaa and Mihaela van der Schaar. Limits of estimating heterogeneous treatment effects: Guidelines for practical algorithm design. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 129-138, Stockholm, Sweden, 2018.
Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological), 28 (1):131-12,, 1966.
Serge Assaad, Shuxi Zeng, Chenyang Tao, Shounak Datta, Nikhil Mehta, Ricardo Henao, Fan Li, and Lawrence Carin. Counterfactual representation learning with balancing weights. In Arindam Banerjee and Kenji Fukumizu (eds.), The 24th International Conference Artificial Intelligence and Statistics, volume 130 Proceedings of Machine Learning Research, pp. 1972-1980, Virtual Event, 2021. PMLR.
Shai Den-David, John Blitzer, Koby Crammer, and Fermando Pereira. Analysis of representations for domain adaptation. In Bernhard Schölkopf, John C. Platt, and Thomas Hofmann (eds.), Advances in Neural Information Processing Systems 19, pp. 137-144, British Columbia, Canada, 2006. MIT Press.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Mach. Learn., 79(1-2):151-175, 2010.
Artem Betlei, Eustache Diemert, and Massih-Reza Amini Treatment targeting by AUUC maximiza- tion with generalization guarantees. CoRR, abs/2012.09897, 2020.
loana Bica, James Jordon, and Mihaela van der Schaar. Estimating the effects of continuous-valued interventions using generative adversarial networks. In Advances in Neural Information Processing Systems 33, virtual event, 2020.
Greg Brockman, Vicki Cheung. Wojciech Zaremba. Openai gym. CoRR, 1bs//600.01140, 2016.
Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and
Lars Buesing, Theophane Weber, Yori Zwols, Nicolas Heess, Sébastien Racanière, Arthur Guez, and Jean-Baptiste Lespiau. Woulda, coulda, shoulda: Counterfactually-guinn policy search. In 7th International Conference on Learning Representations, New Orleans, LA, 2019.
Jonathon Byrd and Zachary Chase Lipton. What is the effect of importance weighting in deep learning? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 872 881, Long Beach, CA, 2019. PMLR.
Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, and Le Song. Generative adversarial user model for reinforcement learning based recommendation system. In Proceedings of the 36th International Conference Machine Learning, pp. 1052-1061, Long Beach, CA, 2019.
Xiong-Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei Tony Qin, Shang Wenjie, and Jieping Ye. Offline model-based adaptable policy learning. In Advances in Neural Information Processing Systems 34 (NeurIPS'2 21), Virtual Conference, 2021.
Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. In John D. Lafferty, Christopher K. Williams, John Shawe-Taylor, Richard S. Zemel, and Aron Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 4424 British Columbia, Canada, 2010. Curran Associates, Inc.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, hbs//200400229,, 2020.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu. David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27, pp. 2680- Montréal, Canada, 2014.

---

Under review as a conference paper at ICLR 2023
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critics Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, pp. 1856-1865, Stockholmsmässan, Sweden, 2018.
Danijar Hafner, Timothy P. Lillicrap, lan Fischer, Ruben Villegas. David Ha, Honglak Lee, and James Davidson. Learning latent dynamies for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 2555-2565, Long Beach, CA, 2019. PMLR.
Negar Hassanpour and Russell Greiner. Counterfactual regression with importance sampling weights. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 5880-5887, Macao, China, 2019.
J.B. Hirittt-Urruty and C. Lemaréchal. Fundamentals of Convex Analysis. 2001. 9783642564680.
ISBN
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29, pp. 4573, Barcelona, Spain, 2016.
Guido Imbens. The role of the propensity score in estimating dose-response functions. Econometrics eJournal, 1999.
Edward L Ionides. Truncated importance sampling. Journal of Computational and Graphical Statistics, 17(2):255-311, 2008.
Stephen James and Edward Johns. 3D simulation for robot arm control with deep Q-learning, CoRR, abs/1609.03777 (.09,,3759,... 2016.
Fredrik Johansson, Nathan Kallus, Uri Shalit, and David Sontag. Learning weighted representations 2018
for generalization across designs. arXiv preprint arXiv:
Yonghan Jung, Jin Tian, and Elias Bareinboim. Learning causal effects via weighted empirical risk minimization. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33, Virtual Event, 2020.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL Model- based offline reinforcement learning. CoRR, 100520005.0595,,, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, fass2205.011644 2020.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off oof-po reinforcement learning without great exploration. In Advances in Neural Information Processing Systems 33, virtual event, 2020.
XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory, 56(1 1158477-566, 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems 29, pp. 271-279, Barcelona. Spain, 2016.
Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and EssuanTTien Lin (eds.), Advances in Neural Information Processing Systems 33, virtual, 2020.
Dean Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88-97, 1991.
12

---

Under review as a conference paper at ICLR 2023
Joaquin Quinonero-Candela. Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008.
Paul R. Rosenbaum and Donald B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70:41-55, 1983.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning. volume 37, pp. 1889-1897, Lille, France, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 06340, 2017.
Uri Shalit, Fredrik D. Johansson, and David A. Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 3076-3085, Sydney, Australia, 2017.
Wenjie Shang. Yang Yu, Qingyang Li, Zhiwei (Tony) Qin, Yiping Meng, and Jieping Ye. Environment reconstruction with hidden confounders for reinforcement learning based recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Anchorage, AK, 2019.
Mining, pp.
Jing-Cheng Shi, Yang Yu, Qing Da, Mong Chen, and Anxiang Zeng. Virtual-taobao: Virtualizing real-world online retail environment for reinforcement learning. In The 33th AAAI Conference on Artificial Intelligence, pp. 4902-4909, Honolulu, Hawaii, 2019.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Sumedh A. Sontakke, Arash Mehrjou, Laurent Itti, and Bernhard Schölkopf. Causal curiosity: RL agents discovering self-supervised experiments for causal representation learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9848-9858, Virtual Event, 2021. PMLR.
Peter Spirtes. Introduction to causal inference. J. Mach. Learn. Res., 11:1643-1662, aug 2010. ISSN 1532-4435.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning an introduction. computation and machine learning. MIT Press, 1998.
Adaptive
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In Proceedings of the 24th IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 0036,5033, Vilamoura, Portugal, 2012.
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):122--1222, 2018.
Jeremy C. Weiss, Finn Kuusisto, Kendrick Boyd, Jie Liu, and David Page. Machine learning for treatment assignment: Improving individualized risk attribution. In American Medical Informatics Association Annual Symposium, San Francisco, CA, 2015.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. GANITE: estimation of individualized treatment effects using generative adversarial nets. In 6th International Conference on Learning Representations, 2018.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. Advances in Neural Information Processing Systems 33, virtual event, 2020.
13

---

Under review as a conference paper at ICLR 2023
Appendix
Table of Contents
A Proof of Theoretical Results
14
A.1 Proof of Lemma 4.3
15
A.2 Proof of Eq. 6
16
A.3 Proof of Thm. 4.4
17
A.4 Proof the Tractable Solution
20
B Discussion of the Theoretical Results
22
C Societal Impact
22
ooRM-oracle Pseudocode
23
E Implementation
23
E.1 Details of the GALILEO Implementation
23
E.2 Connection with Previous Adversarial Algorithms
25
F Additional Related Work
G Experiment Details
27
G.I Settings
27
G.2 Baseline Algorithms
29
Hyper-parameters
30
G.4 Computation Resources
30
H Additional Results
31
H.l All of the Result Table
31
H.2
................................
Curves
31
H.3
Ablation Studies
31
H.4
Worst-Can Castt-Cas Prediction Error
32
H.5
Detailed Results in the MuloCo Tasks
32
H.6 Detailed Results in the BAT Task
34
A PROOF OF THEORETICAL RESULTS
The overall pipeline to model the tractable solution to AWRM given in Fig. 6.
In the proof section, we replace the notation of E with an integral for brevity. Now we rewrite the original objective I(Pag., M) as:
/ Phr (x,
/ M (x lo M(*la, a)) dx' dadr a o pl (, (1
min max MEM BELI X.A
where w wx. a) Pgr- (1.a)
Pa- (r,aa and |p (.-)) fx.A Pyr (x, dodde which is the squared 12-norm.
3
In MDP, given any policy т, pT-- (x, a, x') = pī (a(x) (x'[x, a) where pūr- (x) denotes the occupancy measure of x for policy n. which can be defined (Sutton & Barto, 1998; Ho & Ermon,
2016) (x) = (1 - E)) [E ''P(2z x|20, M))] where Pr [x, 2 20, MM"] is the state visitation probability that स starts at state In in model M and receive x at timestep and y [0, 1] the discount factor.
14

---

Under review as a conference paper at ICLR 2023
mingreu maxaen Mo M)
estimate the optimal udversarial 3+ distribution Phr- given M (Sec. A.1)
model an eusy-to- estimate distribution of the best-response policy 3
intereeiarry policy
generator function f
derive 3 as generalized rep- resentation of Pir- (Sec. A.2)
Let Pxr- as the distribution of the best-response policy arg maxzeu Lppg- Mf) have rogate objective minsrem L(Ps M) (Sec. A.3)
approximate Par- variational representation
tractable solution, (Sec. A.4)
Figure 6: The overall pipeline to model the tractable solution to AWRM. f is a generator function defined by f-divergence (Nowozin al., 2016). is intermediary policy introduced in the estimation.
A.1 PROOF OF LEMMA 4.3
For better readability, we first rewrite Lemma 4.3 as follows:
Lemma A.1. Given any M in L(MM M). the distribution of the ideal best-response policy 3 satisfies:
3" PM- (z, a)
1 - (I,, aM
+ H (I,
(10)
where DEL (- a), M(-(x,)) is the Kullback-Leiblee (KL) divergence between M (-|2r a) and M-Ixx, a). H (x, a) denotes the entropy of MM a). where DKL (M a), M M(-e a)) the Kullback-Leibler (KL) divergence between M (- a) and M M-112,00 a), H (x, a) denotes the entropy of M (-12r, a), and ooM is the regularization coefficient or in Eq. and also as a normalizer.
Proof. Given transition function M of an MDP, the distribution of the best-response policy 8r satisfies:
Ph aw)((r) / M (²
log M (x'1xx a)) dx' dadr - Illbrr (-, (:,))12
P.. arg Pip.
X.A
Pas. (1, / (7 log M(x' |r. a)) dx' dadx a IP (, ()2))
arg max
Pg X.1
2
g(2,a)
2
(x, a)dadr (1 (:
arg max PM a X.A
2
I
Py (z. a)g(E, addadr (.:
arg max Pgg a X.A
a2
arg max- Px
(-2
Par (I, a)
g(z, a)
d dadx + -1lp- (., .113
l9((,,)11 a
X.A
g(:,))
11 (..-)
arg max- Pape
15

---

Under review as a conference paper at ICLR 2023
We know that the occupancy measure Par is density function with constraint Jx AA (x, a)dadr 1. Assuming the occupancy measure Py.- has an upper bound e, that is 0 I PM- (z,0) I c, Va A, to X, constructing a regularization coefficient am Jx ((DKL(MM')I, ((-x,), + Har. (2, a))drda as a constant value given any M, then we have
3
(I, g( a)
8"
Pas.
aM
Mi dx - Sxx M (x') log M (x'laa, a)dr Sx MM (M')aa a) log (a
OEM
(ML(M a), M(((x,a)) + Has.(x.0)
OM
x (DxL(M M( a)) ))+ + Has- (2. a)):
which is the optimal density function of Eq. with a = aM-
Note that in some particular M*, we still cannot construct 3 that can generate an occupancy specified by g(x, a)/a for any M. We can only claim the distribution of the ideal best-response policy 3 satisfies:
Px. PM- (z. (D,, + Har (x, a)),
(11)
where O' is a normalizer that am =
M(. a)) + H (x, a))dda We give discussion of the rationality of the ideal best-response policy 3 as replacement of the real best-response policy 3 in Remark A.2.
Remark A.2. The optimal solution Eg. 11 relies on g(3,, a). In some particular M it is intractable to derive a B that can generate an occupancy specified by g(x, a)/oMM- Consider the following case: a state D1 in M* might be harder reach than another state T2. e-g.. M (r, a) < M (x))),,,)) a), Vx E X,Ya Va E A, then it is impossible to find 3 that the occupancy satisfies Pa.- (x1,a) > P (x2,0) u). In this case, Eq. 11 can be a sub-optimal solution. Since this work focuses on task-agnostic solution derivation while the solution to the above problem should rely on the specific description of M", we leave it future work. However, point out that Eq. 11 is reasonable re- re-weighting term even as a sub-optimum: 3x gives larger densities on the data where the distribution distance between the approximation model and the real model (i.e., DKL(M", M)) is larger or the stochasticity of the real model (i.e., H ) is larger.
A.2 PROOF OF EQ. 6
The integral process of DKL in Eq. 5 is intractable in the offline setting as it explicitly requires the conditional probability function of M Our motivation for the tractable solution is utilizing the offline dataset Dreal the empirical joint distribution P - (z. a, xx) and adopting practical techniques for distance estimation on two joint distributions. like GAN (Goodfellow et al., 2014; Nowozin et al., 2016), to approximate Eq. 5. To adopt that solution, we should first transform Eq. 5 into a form under joint distributions. Without loss of generality, we introduce an intermediary policy is, of which j can be regarded specific instance. Then have M (x' [x, a) Pir (I, a, ) p% (z, a) for any M if
pi (z. a) > O. Assuming Vz E X, Ya E A, p. (z, a) > if PM (I, a) > O, which will hold when overlaps with μ, then Eq. 5 can transform to:
16

---

Under review as a conference paper at ICLR 2023
M()x,a)) + (M, PMr (v, ( (),
aM 0M /s (x(l))) (108 (x
1
M(x a,)) log M (2'1222 de
SMPfss(2,0 1 PSu- (T, log (1'10,)))))
log M (x (2 dæ'
M(x' 2,
(12)
aMPir- (2c, { P5r. (1,
log (z,- + log (ū)( log M (a'
(<'1 Iz, a) dx
(x,
Pi (r,
aMpis (z) (Jx (2,, log
Pir- (I, a, dx' (2,
(1)
M (a'l'' a)dx' -hss (2, / M (1'2' log M M* (1'1z, dd
Pis. a) log
Pis
(00 a) (s Piis (2, (1) log Pin-(z,a,
1
8 (5, dr' Pir (x, a) log (i)- (2,
+ Pis (z, a) (2))))
(13)
where ao(x, a) = (MMPss-
Definition A.3 (f-divergence). Given two distributions and Q two absolutely continuous density functions p and q with respect base measure dx defined the domain X, define the f-divergence (Nowozin et al., 2016),
p(ar) ((x))
D₂(PI1Q) = {(>)s
dx,
(14)
where the generator functtion f RR + I8 is convex, lower-sseiicontinuuu function.
We notice that the terms Pīr (z, x') log Cir- (m,a,=)) Pi (z,a.0) and Pīy. (z. log Pige (z.a)
Px (2,0) the integrated
functions in reverse KL divergence, which is an instance of function in f-divergence (See Reverse- KL divergence of Tab. in (Nowozin et al., 2016) for more details). Replacing that form g log & with
qf ( we obtain generalized representation of phr-:
p ao a)) (Le Par (2, [
da' piee (z, ( Pir (2,
Har. (ar, (a)
HM
)
P5 (I, (x,,
(15)
A.3 PROOF OF THM. 4.4
We first introduce several useful lemmas for the proof.
Lemma A.4. Rearrangement inequality The rearrangement inequality states that, for two sequences
a1 2 02 2 ... 2 On and by 2 be 2 ... 2 bn, the inequalities
abb azb2 ... + an bn b1)) + a2bx(2) + ... +......... anbr(()) 2 aibn + azbn-1 + + an bi
hold, where (1), π (2),.... ..... (n) is any permutation of 1,2....,
Lemma A.5. For two sequences a1 2 ...2 an and by 2 by ... 2 bn. the inequalities
n
m 1
1
b
0
n
n
hold.
17

---

Under review as a conference paper at ICLR 2023
Proof. By rearrangement inequality, we have
a,b; 2 aιbı + 02b2 + ... + andn
Σ a,b; 2 unbb + azbs an
Σ a,b; 2 a1b3 + 02b4 an
Σ a,;; 2 a,b, + tabi +... aa,bn
Then have
12
12
agby
0,b
1=1
Now we extend Lemma A.5 into the continuous integral scenario:
Lemma A.6. Given X R. for functions f - R and g X - IR that f(x) 2 fly) if and only if g(x) 2 g(v), Vz, E X, the inequality
J (()) 2 / p(z)f(r)dz / p(z))(x)dz
holds, where p: X + R and p(x) 0, E X and fx p(r)dz =1
Proof. Since (f(x) f - ((v)) 2 0, Vr, E X, we have
[ / - -dd 20
HEX
[
+ http://0w000000-)oo))))o/)oo ow 2
2Ex BEX
f VEX
S.ca vEX
[
( plaiptuns(s)sgoohha VEX (olDIo))(( do 2
LEEX (onno///h00000000000000000oo00 JEEK vER
I (pL))s(2))(=)+ / ple)p(w)f(()ll(wda)) de JEEX JWEN XLEMUUrreDdd0==-0000000000000100
VEX
JEEK ple)pll) 2 /ttp:///......//ooo.oooo./o.//..
p(=)s())g(e)de= + Jeex
MEX / (ttp0//0o//0000//0///000/0011000
p(o)s(=)g(()dd
p(u)f()l(()ddy> /
=Ex
MEX p(e)p(1l/(()g((ddddd
2
p(c)f(r)g(z)dz 22
x
VEX JEEK
((o)f(z)g(z)de 22 p(o)S(a)de p(z)g(x)dr
JEX
JeX
p(=)s(e)g(<)de>
plz)S(x)de p(z)g(s)dr
JEEN
18

---

Under review as a conference paper at ICLR 2023
Corollary A.7. Let g( ((a))
q(x)) = - -log p(a)
where p(a) > O, Va E and q(x) > 0, Vo E for
((2)
v. > O, the inequality
Jx ((x)s(( p(x)
)g( p(xx)
dd 2 / (1x))((
P(o)) dd [x (()))( plx) pla)) ((x)
ida,
((x) ((x)
g(x)) g(x))
holds if (a) I O, Vx E X. It is not always satisfied for functions of I-dlivergecee We list comparison of on that condition Tub. 2.
Proof. (x)x) = -ologg
< 0, Vo E X. Suppose ('(x) S 0, Va X, we have f(x) 2 (())
and only if g(x) 2 g(v), Vx,y y E X holds. Thus f(ve(e) ((x) 2 f(u ply) if and only if g( 9(2) 2
((a) gl 9(v) Yx,y E X' holds for all v > O. By defining F(x) f(v ) and G(z) I p(
and
using Lemma A.6, we have:
[ (()))) / / ((x)F(()ddr / ((x)G(x)dx.
Then we know
Sx f())s(v
p(x) P()) ((x) ) p(x) dr 2 / ((x)s()P()) p(x) )d / g(())g(
P(z)) )r q(x)) xx g(z)
((x))
holds.
Table 2: Properties of fx) S 50, E X for f-divergences.
Name
Generator function f(x) If f(() & 0, Vr E X
Kullback-eebble
x log x log (r 1)2
False
Reverse KL
True
Pearson x²
False
Squared Hellinger
(Vi - 1)2
False
Censen-Shannon
(x + 1) log 1tz + x log æ
False
2
GAN
x loog x (x: + i) log(æ + +1)
True
Now, we prove Thm. 4.4. For better readability, we first rewrite Thm. 4.4 as follows:
Theorem A.8. Let P the data distribution of the best- best-response policy 3 in Eq. 4 under model Mg parameterized by 0, then we can find the optimal 0 of ming maxsen L (Eq. 4) iteratively optimizing the objective oc+1 = ming Tp Mo), where Pht is approximated via the last-iteration model Me.. Based Corollary A.7, we have upper bound objective for ming L( B Mo) and derive the following objective
(Piso, (1(Din) (x, pi (z, a,
Piiso (I,
0,41 arg max Epgr-
log Ma(x'|z,a)
+ H (I,
a0(2,a)
p (x)
where a0(x,a) aMM (x, Epr, denotes Ez Ez,ax''mp F. f is the generator function in f-divergence which satisfies f'(x) S 0. E X. and 9 the parameters of M. Ma, denotes probability function with the same parameters as the learned model (i.e., 0 = 0) but the parameter is fixed and only used sampling.
Proof. Let Par- as the data distribution of the best-response policy 3 in Eq. 4 under model Ma parameterized by 0, then find the optimal 0 of ming max T Mo) (Eq. 4) via
19

---

Under review as a conference paper at ICLR 2023
iteratively optimizing the objective 01+1 ming L(p Mo). where per is approximated via the last-iteration model Mo,;
3" (x min (
/ M (x (- log Mo (2' (zz, al) dæ' dadæ
0ct1 X
(16)
min (oaaao
1 ao(2,a)
Piro, (2, a,x')
da' [x M (x'1,,)) log Mo Mo(a'|ar, a))
Pi (2,, a, a,
Pi (2,a)) (s
Plite (2)
Har (ar (ara))
S M (1(l log Mo (o a)) jaat
Pi (x,
min Ax.A au(z)a) (/e (is-((n,
Piio, (or, a,,z)
da' (Maraa
Mo(a'| a) Ma a,
ydr' + Har (a)-(a)))) (1,
Pis- (2, a, 20)
-iie-(zz (s
Pūda, (x)
Har Har-(a,a))) (or
J M (x' (1'(2,)) a)( log Mo | a))dx' Jadd
Pis- (25-(1,
J Jx.A a)
(r,- / M (1' a))
Pisar (ce, a, Pi (2, a,)))
(o )dr (-log
< min
M (x a)
based Corollary A.T
(s
Piate (2,a)
Pir- (2;
-ar- (Nr-(z,))
/ (2'x'xx,)) log Mo a))dx' Joda
Pige (r)
= min
1
Pis- (or, a) /xx
M (2
Prito (2, a,)x) Pās (T,, x.)
(- log Mo(a' lo, (2), dx
Ax a0(
Pir (2,,) (s
Pisee (2)
H. -Hw-(e,a) (e,
A (u'lee log Ma(a' Ir, a))dd' )aadte
Pise (cr,a)
max J Jx.Ax cxo a) pîr- Pār- (i, a, log Mo(a'
(1(fig a, (2ar((),,xx))
(iito (I,, (o,))2)) (3,a
Piee, (2,
Pir- (ir,((,,)) + H (e, de' dada,
8
Pire
(17)
where Mg. is introduced to approximate the term p and fixed when optimizing 0. In Eq. 16, l (11-)12 for Eq. 9 is eliminated as it does not contribute to the gradient of 0. Assume f'(x) s O, Vx E X, let v(z, a) = Pig- (æ.a) > 0, p(=1 Ma (x [x, a), and q(>" (ellar = (z' (a. a). the first inequality can be derived by adopting Corollary A.7 and eliminating the first Has- since does not contribute to the gradient of 0.
Pir,, (2,a)
A.4 PROOF OF THE TRACTABLE SOLUTION
Now we are ready to prove the tractable solution:
Proof. The core challenge that the term J( Pis, (2,a,x")
Pi. (n.0,z7) S( Piso, (z,a) po. (#.e) is still intractable. In the
following, give tractable solution to Thm. 4.4. First, resort to the first-order approximation. Given some E (1 - E, 4 E), > O, we have
f(u) ~ 1()+
(18)
where f' is the first-order derivative of f. By Taylor's formula and the fact that f'(u) of the generator
function f is bounded in (1- (1-6, 1 E), the approximation error is no more than OE22). Substituting
with p(w) in Eq. 18, the pattern f( ((x)) in Eq. 17 can be converted to p(z) [(x)
(' p(w) f((((=) f(1),
v(xr)
((x) ((
20

---

Under review as a conference paper at ICLR 2023
then have:
O+t1 = arg max an an(u, a) SeL ( ( M (ollpor)a)
1
Pittn (2, (2,
(e,- log Mola' ls, a)dx'
Mitee (800, (3,00)
Sx M (u'lor a)la,a log Mo(a')a ajdr'+
Pis (ar
Pāa- (c,, H)) J
log Ma(a' (o, a)de' dade
Pitee (or, (o,,)
Phioz (x, a) / Mo (e(
arg max
log Mo(a' (1, a)dx'
x.A
Pis- a) / M (o'lo,))) (8'
Pāroo (2,, (,,)
-sa) log Mo(a' Mo(a'lar a)dx'
(x,,
Piito (2)
Dis,, (x)
M (x'la, log Mo 1x, a)dx'+
Pir- (2,
(2,00 (r
(p
Piia (5, (1) a))
-(1) / M (o'1az, a) log Mo(a' lx, a)dx'
)
(2,
/ M (0 a) log Mo(a' Mo(2o1lo,, a)dz' dadæ
(2, a) HNr
Piito (e, (x)))
Piien (1,
arg max
1
PMn. (r, (1,
s'
Pār- (Ir, (2,
log Mo(a' Ir, a)dz'daxx+ dadx
X.A.X (o a)
1 a) J (o.. Pās (I, a,)')
8
Pir (II, ())
Piite (ee, aa)
HM (1r a))
log Mo |s, a)dz'dadz. dadz.
Phre (2,,
Pir- (x, (x,
Note that the part Pir- (I. a) in Pig. (I, x') can be canceled because of co M MMr (I, a)) but we choose to keep it and ignore on (z, a). The benefit is that we can estimate 0i.-- (I, a,)x)) from empirical data distribution through data collected by in M directly, rather than from uniform distribution which harder be generated. Although keeping p..- (ær, a) incurs extra bias in theory. the results in our experiments show that it has not made significant negative effects in practice. We leave this part of modeling in future work. In particular, by ignoring ap (z, a), we have:
piso, (0),, (r X.AXX
(Citen (x, (( (Prii
Pise, (2, Phr- (2,a) )
Dct1= arg max
pir. (o,
log Mo a)dz'dadx+ dadx+
(19)
fx Jr Pio (2, (r
Piste (2r, a)) Pir- (Ir
Bisg (Ir, a3,a
8r
+ HM (z, log Mo |a, a)dx' dadæ.
Piir (2, (2,
(20)
(Dsee (2,0) Pi (1,a)
and f (ooog (araa") /he- (150.)))
We can estimate f'
through Lemma A.9.
Lemma A.9 A.9 (f)) estimation (Nguyen et al., 2010)). Given a function T:p : X I. parameterized
by E &, if is convex and lower semi-contimuous, by finding the maximum point of yp in the following objective:
= arg max Em-p(a) (T Exsq(a) (Tx)))
we have f' p(z)
((x)) T (x). fo Fenchel conjugate of f EttiiarrtUUrutt & Lemaréchal, 2001).
In particular,
S arg max Ez [Tip l (2, 2')] - Ex,ant (xT
iPo
p arg max Ex [Tos (z, a)] - Ex,aniioo (T
i1
21

---

Under review as a conference paper at ICLR 2023
Pisp, (x,a,a") Pi-- (1,,,
Tog (2, a, and f (iita (x.a) ( ss Twp (z, Given vó and
then we have I'
ii, let Apg a,) (2, Tei (z, a), then we can optimize 0 via:
0.+1 arg max
N.A.N Piso, (5, (r, Toi (2, log Mo |æ, a)dx' dadx+
J J.LAN (r, (Tps( Top a, x/) + HM- (x, a)) log Mo a)da'dads
arg max
N.A.X Piioo (s, a, Ap (2, a, x') log Ma (x, addr''daaz
JE Pir- 2') (z, a, x') + Has (a, a)) log Mo(x' [z, a)dr' dadr.
Based on the specific f-divergence, we can represent T and (T) with a discriminator D It can be verified that f(u) u llg (u + 1) 1), (u) log (u), and f (T, =
-log(1 (u)()) proposed in Nowozin al. (2016) satisfies the condition f=(x) < 0, Vr & (see
Tab. 2). We select the former in the implementation and convert the tractable solution to:
[Agg (2, a²)) x²) log Me(e' [x, a)] + Ers.. [[Har (xx Avg (2, a, x')) log Mo a)] Oc+1 arg & Epito
s.t.
s = arg max E. [log Do (z, x + Epiro [log(1 D (x, va
ii arg max Bpy. Dog Dap a)) Epssor [log(1 (o,(
(21)
where A Apr (2, = log Deg (2, - log Do; a), Epico F is a simplification of
Ex, x'ppsr uphsn 1..
B DISCUSSION OF THE THEORETICAL RESULTS
We summarize the limitations of current theoretical results and future work as follows:
1. As discussed in Remark A.2, the solution Eq. relies on P (x, a) E [0, Va A, Var X. In some particular M it is intractable to derive 3 that can generate an occupancy specified by (lz.a)/am- If more knowledge of M or 3 is provided some mild assumptions can be made on the properties of M or ,8", we may model in more sophisticated approach alleviating the above problem.
2. In the tractable solution derivation, we ignore the term ao a) aMa, Pi (a, a) (See Eq. 20). The benefit that Pir- (2, a.x") the tractable solution be estimated through offline datasets directly. Although the results in our experiments show that it does not produce significant negative effects in these tasks, ignoring PK (z. a) indeed incurs extra bias in theory. In future work, techniques for estimating PMr (z. (z) (Liu et al., 2020) can incorporated to correct the bias. On the other hand, O,M,, also ignored O'M,, can be regarded as a global rescaling term of the final objective Eq. 20. Intuitively, it constructs an adaptive learning rate for Eq. 20, which increases the step size when the model is better fitted and decreases the step size otherwise. It can be considered further improve the learning process in future work, e.g., cooperating with empirical risk minimization by balancing the weights of the two objectives through oMM,,"
the process.
C
SOCIETAL IMPACT
This work studies a method toward counterfactual environment model learning. Reconstructing accurate environment of the real world will promote the wide adoption decision decision-making policy optimization methods in real life, enhancing our daily experience. We are aware that decision-maker
22

---

Under review as a conference paper at ICLR 2023
policy in some domains like recommendation systems that interact with customers may have risks of causing price discrimination and misleading customers if inappropriately used. A promising way to reduce the risk is to introduce fairness into policy optimization and rules to constrain the actions (Also see our policy design in Sec. G.1.3). We are involved in and advocating research such directions. We believe that business organizations would like embrace fair systems that ultimately bring long-term financial benefits by providing a better user experience.
D AWRM-ORACLE PSEUDOCODE
We list the pseudocode of AWRM-oracle in Alg. 1.
Algorithm 1 AWRM with Oracle Counterfactual Datasets
Input:
@ policy space; N: total iterations
Process:
1: Generate counterfactual datasets {Dn₂} for all adversarial policies To, Φ E A
2: Initialize environment model Mg
3: for iori= I:N do
4: Select Dr, with worst prediction errors through Mg from {Px,)
5: Optimize Mo with standard supervised learning based on T
6: end for
E IMPLEMENTATION
E.1 DETAILS OF THE GALILEO IMPLEMENTATION
The approximation of Eq. 18 holds only when p(z) handle the problem, we inject a standard supervised learning loss
g(z) close to 1, which might not be satisfied. To
arg max Eøg,- [log Mo [2., a)]
(22)
to replace the second term of the above objective when the output probability of D is far away from 0.5 (1) = log 0.5).
In the offline model-learnin setting, we only have a real-world dataset D collected by the behavior policy μ.. We learn a policy á z j via behavior cloning with D (Pomerleau, 1991; Ho & Ermon, 2016) and let js be the policy K. We regard D the empirical data distribution of p.r- and the trajectories collected by j the model Mo, as the empirical data distribution of PiM.," But the
assumption Vr E X' Va E A. p(a]x)) > O might not be satisfied. In behavior cloning, we model f with Gaussian distribution and constrain the lower bound of the variance with small value Cu 0 to keep the assumption holding, Besides, we add small Gaussian noises u ~ N (NO,, e to the inputs of D. to handle the mismatch between 0% and i due to cu- In particular, for 4o and 4 learning. have:
çi = arg max Epp,.. [log Deo (xr + Mr May + ]r))]] + Epian [log(1 D (x + uz + the + u.)
PP
ip = arg max Epg... (log Dap (xr + un a u.)) + Epsio [log(1 Dop (x + Mr. a + un))],
121
where Epscn, al-] is a simplification of Ex " ( and u = [Uzz uay Uz/
On the other hand, we notice that the first term in Eq. 21 is similar to the objective of GAIL (Ho & Ermon, 2016) by regarding Mo as the policy to learn and as the environ- ment to generate data. For better capability
sequential environment model learning, here we introduce some practical tricks inspired by GAIL for model learning (Shi et al., 2019; Shang et al., 2019); we introduce an MDP for k and Mg. where the reward defined by the diseriminator D. ie.. r(z, a, 2') log D(I, a, x²)"). Me is learned to maximize the mulative rewards. With advanced policy gradient methods (Schulman et al., 2015; 2017). the objective is converted to max, [Apt.e] (I. a, 2') log Mo(x, a, z²)]. where A Qise. VM,,
23

---

Under review as a conference paper at ICLR 2023
Algorithm 2 GALILEO pseudocode
Input:
Deeat:
offline dataset sampled from Pi. where j is the behavior policy; total iterations;
N:
Process:
1: Approximate behavior policy j via behavior cloning
2: Initialize an environment model Mg.
3: for 1 NN do
4: Use jú to generate a dataset Dgen with the model Ma,
5: Update the discriminators Dep and Dip, via Eq. 26 and Eq. 27 respectively, where Plato is
estimated by Dgen and Phy- estimated by Dreal
6:
Update Q and V via Eq. 24 and Eq. 25 through D gen D Do and Dap,
7:
Update the model Ma, via the first term of Eq. 23, which is implemented with standard policy gradient method like TRPO (Schulman et al., 2015) or PPO (Schulman et al., 2017).
Record the policy gradient 9pg
8:
if po EDg.. po (Zi, at Tt41) P1 then
9:
Compute the gradient of Mo, via the second term of Eq. 23 and record it as 9st
else
11:
Compute the gradient of Me. via Eq. 22 and record Ial
12:
end
13:
Rescale Isl via Eq. 28
14: Update the model Me, via the gradient Isl and obtain Me
15:
end for
Qi, (x, B 1E (2t, as xr+1) (Ie, ae, T++1) = (z, a, x') k. Me.., and V (z, a) = z²)] a')
A in Eq. can also be constructed similarly. Although it looks unnecessary in theory since the one-step optimal model Mo is the global optimal model in this setting, the technique helpful practice as it makes A more sensitive to the compounding effect one-step prediction errors: would consider the cumulative effects of prediction errors induced by multi-step transitions in environments. In particular, to consider the cumulative effects of prediction errors induced by multi-step of transitions in environments,
EM, Qir, (2,, (x,,
overwrite function Aptipt A 1 Ma, where Qhi,, (x. 2') E E: 7 log Dug (oe, oe, (1 Qe, L++1) = (x, a, x'), K. Mo.] and Ma, (z, = E E 7t log Dep (Iz, (x), at) = (x, a), K, Ma. To give an algorithm for single- step environment model learning, we can just set y in Q and V to O.
fake dataset Darn
dataset [line
(3) sppraximate AWRM through Eq, update the dynamics model
behavior policy A(als)
dynamics model Ma.
(line
discriminators and value functions Dp,(s,a, s'), (8.0),), ? and V
(2) update discriminators and the value functions (Iine
pretrain: aporoximated the behavior policy (line
offinne dataset
Dreai
neural networks
fake dataset
offline dataset
Figure 7: Illustration of the workflow of the GALILEO algorithm.
24

---

Under review as a conference paper at ICLR 2023
By adopting the above implementation techniques, convert the objective into the following formulation
8.+1 arg
Epsive
(o a, log Mo(a'Ia, [x, Bpss- [(HM- (2, Arguet (ar, a, 2²)) log Mo
0
(23)
s.l.
Vito, (2,, (,,,,x)) = E
Σ 7' log Dag (ui, ar, 241)) = (e, a, xx, Mo,
(24)
Vidg, (x, a) = Σ 7' log Dej (mr, ai (a,, as) = (r, a), Ns, Mo.
(25)
10 arg max B [log Dyo (e uas a ua, + ux))] + Boise . [log(1 Dvo (2 + ue + ua, x' u ])]
(26)
p = arg max Epir [log Des (a + uz a + u.)] + Eriso [log(1 Des (xr + ua a + u)) P1
(27)
where A Ago (I, a, x') = Rir, (I, (z,,,xx) Vir (zt, a). In practice GALILEO optimizes the first term of Eq. 23 with conservative policy gradient algorithms (e.g.. PPO (Schulman al., 2017) or TRPO (Schulman et al., 2015)) to avoid unreliable gradients for model improvements. Eq. 26 and Eq. 27 are optimized with supervised learning. The second term of Eq. 23 optimized with supervised learning with re-weighting term - Appg.et + HM- Since H is unknown, we use HM, to estimate it. When the mean output probability of a batch data is larger than 0.6 or small than 0.4, we replace the second term of Eq. 23 with standard supervised learning in Eq. 22. Besides, unreliable gradients also exist in the process of optimizing the second term of Eq. 23. In our implementation, we use the scale of policy gradients term Eq. 23. In particular, we first compute the 12-norm of the gradient of the first term of Eq. 23 via conservative policy gradient algorithms, named |992|22 Then we compute the nn-norm of the gradient of the second term of Eq. 23, name ||ga||2. Finally, we rescale the gradients of the second term 9s1 by
constrain the gradients of the second
119pe 2
9s1 <- Osl max (11ppg 12, (1galll2
(28)
For each iteration, Eq. 23, Eq. 26, and Eq. 27 are trained with certain steps (See Tab. 5) following the same framework GAIL. Based on the above techniques, we summarize the pseudocode of GALILEO in Alg. 2, where po and are set to 0.4 and 0.6 in all of our experiments. The overall architecture shown in Fig. 7.
to Reviewer beJS
E.2 CONNECTION WITH PREVIOUS ADVERSARIAL ALGORITHMS
Standard GAN (Goodfellow et al., 2014) can be regarded as partial implementation including the first term of Eq. 23 and Eq. 26 by degrading them into the single-step scenario. In the context of GALILEO, the objective of GAN is
8,41 arg max Epšto, [Ap (I, (x, 2.) log Mo |z,, a)]
s.t. yo* arg max Epr. [log (2 a, x')) Epr., [log(1 (
io
where Ap (I, x') = log De (I, a, 2'). In the single-step scenario, Phis, (x, a')
x') ) log Ma(x'lz,,) a)] can convert to The term Epi,, [Ap- a,
Eps, [log Dip (x)) by replacing the gradient of Ma, (a' (x. V log Ma(a'I |z, a) with VoMa (a' [z, a) (Sutton Barto, 1998). Previous algorithms like GANITE (Yoon al., 2018) and SCIGAN (Bica et al., 2020) can be regarded as variants of the above training framework.
The first term of Eq. 23 and Eq. 26 are similar to the objective of GAII. by regarding Mg as the "policy" imitate and ja as the "environment" to collect data. In the context of GAL, the
25

---

Under review as a conference paper at ICLR 2023
objective of GAIL is:
0++1 = arg max Boss.
[Ap (a, a, log Ma(aa(a; [x]
Quo (x, a, x') E
Σ 7 log D (are, ac, zet 1 () ac, 24+1) (x, a, Moi
4- -ag max Epg, [ a, x²)] + Eosee, [Log(1 - De(ce, a, 2']]
where Ag- (0, 2,)) Qisn (Ir a, 2²)
Vid (2., a) and Vida (2r, a) ENs, (2,a) (I,
F ADDITIONAL RELATED WORK
to Reviewer j5p5 and rQ79
Our primitive objective is inspired by weighted empirical risk minimization (WERM) based on inverse propensity score (IPS). WERM is originally proposed to solve the generalization problem of domain adaptation in machine learning literature. For instance, we would like to train a predictor (y]rr in a domain with distribution (usin(2) to minimize the prediction risks in the domain with distribution Prest (z), where Piest # Prest- To solve the problem, we can train weighted objective (m) log M (v]x)), which is called weighted empirical risk minimization
with max Ez Purnin Perain
methods (Ben-David et al., 2006; 2010; Cortes et al., 2010; Byrd & Lipton, 2019; Quinonero- Candell al., 2008). These results have been extended and applied to causal inference, where the predictor is required to be generalized from the data distribution in observational studies (source domain) to the data distribution in randomized controlled trials (target domain) (Shimodaira, 2000; Assaad et al., 2021; Hassanpour & Greiner, 2019; Jung al., 2020; Johansson et al., 2018). In this case, the input features include a state x (a.k.a. covariates) and an action a (a.k.a. treatment variable) which is sampled from policy. We often assume the distribution of xr, P(x) is consistent between
the source domain and the test domain, then we have (sec(()) P(a))(aaa)) ((elar) where μ and
Plyaia P(z)u(a(a) j
3 are the policies in source and target domains respectively. In Shimodaira (2000); Assaad et al.
(2021); Hassanpour & Greiner (2019), the policy randomized controlled trials is modeled
uniform policy, then Pawa(()))
p(alz) x "(allz) j(ala) is also known as inverse propensity score (IPS). In Johansson et al. (2018), it assumes that the policy in the target domain is
(z) = P(z)pta( x))
P(a)
3(alz)
predefined as (a(xz) before environment model learning, then it uses _ as the IPS. The differences
between AWRM and previous works are fallen in two aspects: (1) We consider the distribution-shie shift problem in the sequential decision- decision-making scenario. In this scenario, we not only consider the action distribution mismatching between the behavior policy μ and the policy to evaluation /3, but also the follow-up effects of policies to the state distribution; (2) For faithful offline policy optimization, we require the environment model to have generalization ability in numerous different policies. The objective of AWRM proposed to guarantee the generalization ability of M in numerous different policies instead of specific policy.
On a different thread, there are also studies that bring counterfactual inference techniques of causal inference into model-based RL (Buesing et al., 2019; Pitis et al., 2020; Sontakke et al., 2021). These works consider that the transition function is relevant to some hidden noise variables and use Pearl-style structural causal models (SCMs), which is directed acyelic graphs to define the causality of nodes in an environment, to handle the problem. SCMs can help RL in different ways: Buesing et al. (2019) approximate the posterior of the noise variables based on the observation of data, and environment models are learned based on the inferred noises. The generalization ability is improved if we can infer the correct value of the noise variables. Pitis al. (2020) discover several local causal structural models of global environment model, then data augmentation strategies by leveraging these local structures to generate counterfactual experiences. Sontakke et al. (2021) proposes representation learning technique for causal factors, which is an instance of the hidden noise variables, in partially observable Markov decision processes (POMDPs). With the learned representation of causal factors, the performance of policy learning and transfer in downstream tasks will be improved. Instead of considering the hidden noise variables in the environments, our study considers the environment model learning problem in the fully observed setting and focuses on unbiased causal effect estimation in the offline dataset under behavior policies collected with selection bias.
to Reviewer j5p5
26

---

Under review as a conference paper at ICLR 2023
In offline model- RL, the problem is called distribution shift (Yu et al., 2020; Levine et al., 2020; Chen et al., 2021) which has received great attentions. However, previous algorithms do not handle the model learning challenge directly but propose techniques to suppress policy sampling and learning in risky regions (Yu et al., 2020; Kidambi et al., 2020). Although these algorithms have made great progress in offline policy optimization in many tasks, so far, how to learn better environment model in this scenario has rarely been discussed.
G EXPERIMENT DETAILS
G.1 SETTINGS
G.1.1 GENERAL NEGATIVE FEEDBACK CONTROL (GNFC)
The design of GNFC is inspired by a classic type of scenario that behavior policies jr have selection bias and easily lead to counterfactual risks: For some internet platforms, would like to allocate budgets to a set of targets (c.g., customers or cities) to increase the engagement of the targets in the platforms. Our task is to train a model to predict targets' feedback engagement given targets' features and allocated budgets.
In these tasks, for better benefits, the online working policy (i.e., the behavior policy) will tend cut down the budgets if targets have better engagement, otherwise, the budgets might be increased. The risk of counterfactual environment model learning in the task is that: the object with better historical engagement will be sent to smaller budgets because of the selection bias of the behavior policies, then the model might exploit this correlation for learning and get conclusion that: increasing budgets will reduce the targets' engagement, which violates the real causality. We construct an environment and behavior policy to mimic the above process. In particular, the behavior policy JGN FC is
(GNFC(#) (62.5 - mean(r)) +
15
where é is a sample noise, which will be discussed later. The environment includes two parts:
(1) response function Mi(y)2,,0) a):
Mi M1 a) = N (m)can(x) + aa,2)
(2) mapping function M2(w' x., y):
M2 [1 a. = = y- mcan(r)) + +
The transition function M is composite of M (x'12, a) M2(x' [zr, a, Mi (ylar, (ojaraa) a)). The behavior policies have selection bias: the actions taken are negatively correlated with the states, as illustrated Fig. 8(a) and Fig. 8(b). We control the difficulty of distinguishing the correct causality of x, a. and by designing different strategies of noise sampling on e. In principle, with a larger number or more pronounced disturbances, there are more samples violating the correlation between æ and a, then more samples can be used to find the correct causality. Therefore, we can control the difficulty of counterfactual environment model learning by controlling the strength of disturbance. we sample from uniform distribution U U'-e, e) with probability That is, 1 - and ~ U--e,é e) with probability p. Then with larger p. there are more samples in the dataset violating the negative correlation (i.e., (GNFFC), and with larger e, the difference of the feedback will be more obvious. By selecting different e and p. we can construct different tasks to verify the effectiveness and ability of the counterfactual environment model learning algorithm.
particular,
with probability
1.1.2 THE CANCER GENOMIC ATLAS (TCGA)
The Cancer Genomic Atlas (TCGA) is project that has profiled and analyzed large numbers of human tumors to discover molecular aberrations at the DNA, RNA, protein, and epigenetic levels. The resulting rich data provide significant opportunity to accelerate our understanding of the molecular basis of cancer. We obtain features, x, from the TCGA dataset and consider three continuous treatments done in SCIGAN (Bica et al., 2020). Each treatment, a, is associated with set of parameters, Vi, V2, V3. that are sampled randomly by sampling a vector from a standard normal distribution and scaling it with its norm. We assign interventions by sampling treatment, a, from
27

---

Under review as a conference paper at ICLR 2023
50
response
action
timestep
timestep
(a) y in collected trajectories.
(b) a in collected trajectories.
Figure 8: Illustration of information about the collected dataset in GNFC. Each color of the line denotes one of the collected trajectories. The X-axis denotes the timestep of trajectory.
a beta distribution, a x ~ Beta (). a 2 controls the sampling bias and 3 = a-1 + a, where at is the optimal treatment. This setting of 3 ensures that the mode of Beta (a, 3) a".
The calculation treatment response and optimal treatment are shown in Table 3.
Table 3: Treatment response used to generate semi-synthetic outcomes for patient features In the experiments, we set C 10.
Treatment
Treatment Response
Optimal treatment
fx(x,)1) c ((v))* x (v!)) T xaι 12 (v?) xxi)
(w))*=
of 2(v3)"
fe(x; = =C ((v7)* + sin (I v o2)
(vā)* x
2
2(v3)" a5
fa(x, = C ((v))² 12az(ag b)?, where 0.75
irb 0.75, if < 0.75
We conduct experiments on three different treatments separately and change the value of bias a to assess the robustness of different methods to treatment bias. When the bias of treatment is large, which means a is large, the training set contains data with a strong bias on treatment so it would be difficult for models to appropriately predict the treatment responses out of the distribution of training data.
G.1.3 BUDGET ALLOCCATION TASK TO THE TIME PERIOD (BAT)
We deploy GALILEO in a real-world large-scale food-delivery platform. The platform contains various food stores, and food delivery clerks. The overall workflow is as follows: the platform presents the nearby food stores to the customers and the customers make orders, i.e., purchase take-out foods from some stores the platform. The food delivery clerks can select orders from the platform to fulfill. After an order is selected to fulfill, the delivery clerks will take the ordered take-out foods from the stores and then send the food the customers. The platform will pay the delivery clerks (mainly in proportion to the distance between the store and the customers' location) once the orders are fulfilled. illustration the workflow can be found in Fig. 9.
However, there is an imbalance problem between the demanded orders from customers and the supply of delivery clerks to fulfill these orders. For example, at peak times like lunchtime, there will be many more demanded orders than at other times, and the existed delivery clerks might not be able to fulfill all of these orders timely. The goal the Budget Allocation task to the Time period (BAT) is to handle the imbalance problem in time periods by sending reasonable allowances to different time periods. More precisely, the goal of BAT is to make all orders (i.c., the demand) sent in different time periods can be fulfilled (i.e., the supply) timely.
28

---

Under review as a conference paper at ICLR 2023
send
take-out food orders remotely based on the platform
7
delivery clerks take food from the stores
delivery send the food the customers
customers
Lg
delivery clork
5. The platform will delivery fultiliing the order
platform
Figure 9: Illustration of the workflow of the food-delivery platform.
To handle the imbalance problem in different time periods, in the platform, the orders in different time periods & [0. 1, 2..., 23) will be allocated with different allowances 10 A.M. (i.e., = 10), we add 0.5$ (i.e., 0.5) allowances to all of the demanded orders. From 10 A.M.
M+, For example, at
A.M., the delivery clerks who take orders and send food to customers will receive extra allowances. Specifically, if the platform pays the delivery clerks 28 for fulfilling the order, now he/she will receive 2.58. For each day, the budget of allowance budget allocation policy n (clt) of the limited budget C to make as many orders possible can be taken timely.
fixed. We should find the best
To find the policy, we first learn model to reconstruct the response of allowance for each delivery clerk (1 St. Pt, Ct), where 3t-1 is the taken orders of the delivery clerks in state St, Ct is the allowances, Pt denotes static features of the time period f. In particular, the state 8, includes historical order-taken information of the delivery clerks, current orders information, the feature of weather, city information, and so on. Then we use a rule-based mapping function f to fill the complete next time-period states, i.e., Sr+1 fst. Pts Ct. 3t+1). Here define the composition of the above functions M and f as My Finally, we learn a budget allocation policy based on the learned model. For each day, the policy we would like to find is:
max E
S
Σ yel y n
s.t.,
cofe
C
t.EES
In our experiment, we evaluate the degree of balancing between demand and supply by computing the averaged five-minute order-taken rate, that is the percentage of orders picked up within five minutes. Note that the behavior policy is fixed for the long term in this application. So we directly use the data replay with small scale of noise (See Tab. 5) reconstruct the behavior policy for model learning in GALILEO.
Also note that although we model the response for each delivery clerk, for fairness, the budget allocation policy is just determining the allowance of each time period t and keeps the allowance each delivery clerk the same.
G.2 BASELINE ALGORITHMS
The algorithm we compared are: (1) Supervised Learning (SL): training a environment model to minimize the expectation of prediction error, without considering the counterfactual risks; (2) inverse propensity weighting (IPW) (Spirtes, 2010): a practical way to balance the selection bias by re-weighting. It can be regarded
e where je is another model learned to approximate
29

---

Under review as a conference paper at ICLR 2023
the behavior policy; (3) SCIGAN: a recent proposed adversarial algorithm for model learning for continuous valued interventions (Bica et al., 2020). All of the baselines algorithms are implemented with the same capacity of neural networks (See Tab. 5).
G.2.1 SUPERVISED LEARNING (SL)
As baseline, we train a multilayer perceptron model directly predict the response of different treatments, without considering the counterfactual risks. We use mean square enror to estimate the performance of our model so that the loss function can be expressed as MSE 1 E (y+ û.) where n is the number of samples, y is the true value of response and ÿ the predicted response. In practice, we train our SL models using Adam optimizer and the initial learning rate 3e on both datasets TCGA and GNFC. The architecture of the neural networks is listed in Tab.
G.2.2 INVERSE PROPENSITY WEIGHTING (IPW)
Inverse propensity weighting (Spirtes, 2010) is an approach where the treatment outcome model uses sample weights to balance the selection bias by re-weighting. The weights are defined as the inverse propensity of actually getting the treatment, which can be expressed as [(ll))) where x stands for the feature vectors in dataset, is the corresponding action and p ((a(x) indicates the action taken probability a given the features æ within the dataset. f is learned with standard supervised learning. Standard IPW leads to large weights for the points with small sampling probabilities and finally makes the learning process unstable. We solve the problem by clipping the propensity score: + min(ā, 0.05), which is common used in existing studies (lonides, 2008). The loss function can thus be expressed as Σi= MoJ3R (g)+ 2.)². The architecture of the neural networks is listed in Tab. 5.
G.2.3 SCIGAN
SCIGAN (Bica et al., 2020) is a model that uses generative adversarial networks to learn the data distribution of the counterfactual outcomes and thus generate individualized response curves. SCIGAN does not place any restrictions on the form of the treatment-does response functions and capable of estimating patient outcomes for multiple treatments, each with an associated parameter. SCIGAN first trains generator to generate response curves for each sample within the training dataset. The learned generator can then be used to train an inference network using standard supervised methods. For fair comparison, we increase the number of parameters for the the open-source version of SCIGAN so that the SCIGAN model can have same order of magnitude of network parameters as GALILEO. In addition, we also finetune the hyperparameters (Tab. 4) of the enlarged SCIGAN to realize its full strength. We set num goosage_sampaae and = 10.
Table 4: Table of
for SCIGAN.
Parameter
Values
Number of samples
3, 11 0.1, 1, 10, 20
G.3 HYPER- PARAMETERS
We list the hyper-parameter of GALILEO in Tab. 5.
G.4 COMPUTATION RESOURCES
We use one Tesla V100 PCIe 32GB GPU and 32-core Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz to train all of our model.
30

---

Under review as a conference paper at ICLR 2023
Table 5: Table of hyper-parameters for all of the tasks.
Parameter
GNFC
TAGC
MuJoCo
BAT
hidden layers of all neural networks
4
4
5
5
hidden units of all neural networks
256
256
512
512
collect samples for each time of model update
5000
5000
40000
96000
horizon batch size of discriminators
5000
5000
40000
80000
50
100
48 (half-hours)
(also ED)
0.005
0.01
0.05 (0.1 for walker2d)
0.05
times for discriminator update
2
5
times for model update
1
2
20
times for supervised learning update
1
20
learning rate for supervised learning
le-5
1e-5
3e-4
1e-5
Y
0.99
0.0
0.99
0.99
clip-ratio max DKL
NAN 0.001
NAN
NAN
0.1
0.001
0.001
NAN
optimization algorithm (the first term of Eq. 23)
TRPO
TRPO
TRPO
PPO
H ADDITIONAL RESULTS
H.1 ALL THE RESULT TABLE
We give the result of CNFC in Tab. 8, TCGA in Tab. 9, BAT in Tab. 7, and MuJoCo in Tab. 6.
H.2 AVERAGED RESPONSE CURVES
We give the averaged responses for all of the tasks and the algorithms in Fig. 16 to Fig. 23. We randomly select 20% of the states in the dataset and equidistantly sample actions from the action space for each sampled state, and plot the averaged predicted feedback each action. The real response is slightly different among different figure as the fandomly-scledt states for testing is different. We sample 9 points GNFC tasks and 33 points TAGC tasks for plotting.
H.3 ABLATION STUDIES
In Sec. 4.3 and Appx. E.1, we introduce several techniques develop a practical GALILEO algorithm. Based on task e0 0_2_p0.. 05 in GNFC, we give the ablation studies to investigate the effects of these techniques. We first com- pare two variants that do not handle the assumptions viola- tion problems: (1) NO POO_NNJECC NOISE: set Eμ and éD zero, which makes the overlap assumption satisfied;: (2) SINGLE_SI without replacing the second term Eq. 8 with standard supervised learning even when the output probability of D is far away from 0.5. Besides, introduced several tricks inspired by GAIL and give comparison of these tricks and GAIL: (3) ONE_ STEP: use one-step reward instead of cumulative rewards (i.e., Q and V: see Eq. 24 and Eq. 25) for re-weighting, which implemented by set 7 0; (4) STNGF_ remove Te3 (x, a) and replace it with EM, [Tep (I, a, x²)] which is inspired by GAIL that uses a value function as baseline GAIL: instead of using another discriminator; (5) PURE PURE_GAILL
sqn MISE
GAULLEO
SINGLE, DIS ONE_STER STEP
SINGLE SL.
PURE GAIL
NO, JUJCT, NOISE
Figure 10: Illustration of the ablation studies. The error bars are the standard error
remove the second term in Eq. 8. It can be regarded as a naive adoption of GAIL and a partial implementation of GALILEO.
We summarize the results in Fig. 10. Based on the results of NO_ INJECT JOJECT_NOISEE NOISE and SINGLE SL, can see that handling the assumption violation problems is important and will increase the ability
31

---

Under review as a conference paper at ICLR 2023
Table 6: The root square errors on MuJoCo tasks. We bold the lowest error for each task. "medium" dataset is used for training, while "expert" and "medium-replay datasets are just used for testing. + follows the standard deviation of three seeds.
TASK
HalfCheetah
DATASET
medium (train)
expert (test)
medium-replay (test)
GALILEO
0.378 + 0.003 0.404 + 0.001
2.287= 0.005
1.411 = 0.037 2.246 = 0.016
OFF-SL
3.311 + 0.055
TASK
Walker2d
DATASET
medium (train)
expert (test)
medium-replay (test)
GALILEO 0.49 + 0.00 0.467 + 0.004
1.514 + 0.002
0.968 = 0.004 1.239 + 0.004
OFF-SL
1.825 + 0.061
TASK
Hopper
DATASET
medium (train)
expert (test)
medium-replay (test)
GALILEO
0.037 + 0.002
0.322 +: 0.036
0,408 + 0.003
OFF-SL
0.034 + 0.001
0.464 1 0.021
0.574 + 0.008
on counterfactual queries. The results of PURE_GAIL tell us that the partial implementation of GALILEO is not enough to give stable predictions on counterfactual data; On the other hand, the result of ONE ONE_STEP also demonstrates that embedding the cumulative error of one-step prediction is helpful for GALILEO training; Finally, we also found that S STNGLE_DIIS nearly has almost no effect on the results. It suggests that, empirically, we can use EM, [Tpg (z, a, a')] as a replacement for T (2), a), which can reduce the computation costs of the extra discriminator training.
H.4 WORST-CASE PREDICTION ERROR
In theory, GALILEO increases the generalization ability by focusing on the worst-case samples' training to achieve AWRM. To demonstrate the property, we propose a new metric named Mean-Mas Square Error (MMSE): E max (M' (x' [z, a) M (z'|z, a))²] and give the results of MMSE for GNFC in Tab. 10 and for TCGA in Tab. 11.
H.5 DETAILED RESULTS IN THE MuJoCo TASKS
We select environments from D4RL (Fu et al., 2020) construct our model learning tasks. We compare it with typical transition model learning algorithm used in the previous offline model-based RL algorithms (Yu al., 2020; Kidambi et al., 2020), which is a variant of standard supervised learning. We name the method OFF-SL. We train models in datasets HaliCheetah-mediin medium, and Hopper-medium, medium, which are collected by behavior policy with 1/3 performance to the expert policy, then we test them in the corresponding expert dataset. We plot the converged results and learning curves of GALILEO and OFF-SL in three MuJoCo tasks in Tab. 6 and Fig. 11 respectively.
Walker2d-
In Fig. 11, we can see that both OFF-SI. and GALILEO perform well in the training datasets. OFF-SI. can even reach bit lower error in halfcheetah and walker2d. However, when we verify the models through "experr" and "medium-replay" datasets, which are collected by other policies, the performance of GALILEO is significantly more stable and better than OFF-SL. As the training continues, OFF-SL even gets worse and worse. In summary, GALILEO reaches significantly better performances the expert dataset: the averaged declines of root MSE in three environments are 56.5%, 49.2%, and 34.8%. However, whether in GALILEO or OFF-SL, the performance for testing at least 2x worse than in the training dataset. The phenomenon indicates that although GALILEO can make better performances for counterfactual queries, the risks of using the models are still large and still challenging to be further solved.
32

---

Under review as a conference paper at ICLR 2023
1.2
0.5
GAL
0TT-LL
mean square error
133
1.0
1.0
0.8
0.2
0.9
0.38
8.6
0.5
0.8
x32
x10
(a) halfcheetah-meddim (train)
(b) hopper-medium (train)
(c) walker2d-medium (train)
2.2
0.55
3.6
07F-5L
C4LLE9
0.00
3.5
2.0
1001
2.8
GAULCO
0.50
W
2.8
mesa
12
2.6
0.45
1.4
0.40
1.0
3.0
02
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
08
1.5
0.0
0.2
2.4
0.6
0.0
1.0
update steps
update steps
update steps
(d)
balfchhetab-mmciiu--- replay
(e) hopper-mediuum-eplay (test)
(f) walker2d-medium--ree (test)
(test)
GEL
3.5
3.2
mean zauore cieer
0.50
2.6
3.
0.45
2.8
2.7
1004
0.40
sqyt
0FF-51
2.6
0.35
2.4
2.2
025
update
x32
update
update
x10
(g) hallfcheetah-expel expert (test)
(h) hopper-expert (test)
walker2d-expert (test)
Figure 11: Illustration of learning curves of the MuloCo Tasks. The X-axis record the steps of the environment model update, and the Y-axis is the corresponding prediction error. The figures with titles ending in "(train)" means the dataset is used for training while the titles ending in "(test)" means the dataset is just used for testing. The solid curves are the mean reward and the shadow is the standard error of three seeds.
33

---

Under review as a conference paper at ICLR 2023
normalized fulfilled order amount
1.0
0.8
0.6
0.4
0.0
0.2
0.4
0.6
0.8
1.0
normalizee actions
Figure 12: Illustration of relationship between user feedback and the actions of the offline dataset in the real-world food-delivery platform.
H.6 DETAILED RESULTS IN THE BAT TASK
The core challenge of the environment model learning in BAT tasks is similar to the challenge in Fig. 1. Specifically, the behavior policy in BAT tasks is human human-expert policy, which will tend to increase the budget of allowance in the time periods with a lower supply of delivery clerks, otherwise will decrease the budget (Fig. 12 gives instance of this phenomenon in the real data).
Since there is no oracle environment model for querying, we have to describe the results with other metrics.
roscald pearages responte
1.00
1.00
7.00
0.08
0.90
0.06
0.06
0.34
0.92
0.32
0.80
0.1
0.2
0.4
0.6
0.6
1.0
normalized allocated allowances
nomellzed allocated allowances
normalieed allocated allowances
City-A
(b) City-B
City-C
1052880 avengedd 16500766
1.00
regcaled sveraged respooree
1.00
1.00
0.06
0.99
0.88
0.96
3.86
eopalled arenged
0.07
0.88
3.84
0.05
0.98
0.82
G4LL000
0.84
0.56
0.9
normalized allocated allowances
nommalized allocaled allowances
normalzed allocated allowances
(d) City-D
(c) City-E
City-F
Figure 13: Illustration of the response curves in the 6 cities. Although the ground- -ouun curves are unknown, through human expert knowledge, we know that it is expected to be monotonically increasing.
First, review whether the tendency of the response curve is consistent. In this application, with larger budget allowance, the supply will not be decreased. As can be seen in Fig. 13, the tendency of GALILEO's response is valid in 6 cities but almost all of the models of SL give opposite directions to the response. we learn policy through the model of SL, the optimal solution is canceling all of the allowances, which is obviously incorrect in practice.
Second, we conduct randomized controlled trials (RCT) in one of the testing cities. Using the RCT samples, evaluate the correctness of the sort order of the model predictions via Area Under
34

---

Under review as a conference paper at ICLR 2023
rescaled cumulative treatment effects
1.0
0.5
0.0
-0.5
-1.0
0.0
0.2
0.4
0.6
0.8
1.0
cumulative samples (rescaled
Figure 14: Illustration of the AUUC result for BAT.
the Uplift Curve (AUUC) (Betlei et al., 2020). To plot AUUC, we first sort the RCT samples based on the predicted treatment effects. Then the cumulative treatment effects computed by scanning the sorted sample list. If the sort order of the model predictions is better, the sample with larger treatment effects will be computed early. Then the area of AUUC will be larger than the one via random sorting strategy. The result of AUUC show GALILEO gives reasonable sorting to the RCT samples (see Fig. 14).
Finally, we search for the optimal policy via the cross-entropy method planner (Hafner et al., 2019) based on the learned model. We test the online supply improvement in 6 cities. The algorithm compared is human-expert policy, which is also the behavior policy of the offline datasets. We conduct online A/B tests for each of the cities. For each test, we randomly split city into two partitions, one is for deploying the optimal policy learned from the GALILEO model, and the other is as control group, which keeps the human-expert policy as before. Before the intervention, we collect 10 days' observation data and compute the averaged five-minuse of the treatment and control group. named and b respectively. Then we start intervention and observe the five-minute order-taken rate in the following 14 days for the two groups. The results of the treatment and control groups are yf and yf respectively. where i denotes the i-th day of the deployment. (DID):
taker-taken rates the baselines
The percentage points of the supply improvement are computed via
Ef 19) (uf BO) 100,
T
where T is the total days of the intervention and T 14 in our experiments.
Table 7: Results on BAT. We use City-X X to denote the experiments on different cities. "pp" is an abbreviation of percentage points on the supply improvement.
target
City-A
City-B
City-C
supply improvement
+1.63pp
+0.79pp
+0.27pp
target
City-D
City-E
City-F
supply improvement
+0.2pp
+0.14pp
+0.41pp
The results are summarized in Tab. 7. The online experiment is conducted in 14 days and the results show that the policy learned with GALILEO can make better (the supply improvements are from 0.14 to 1.63 percentage points) budget allocation than the behavior policies in all the testing cities. We give detailed results which record the supply difference between the treatment group and the control group in Fig. 15.
35

---

Under review as a conference paper at ICLR 2023
rescaled Eveemndde onder-ateen batern rate
rescalit fre-ninute ordir-ikknn
1,000
0.88
0.06
0.84
0.94
0.82
0.92
0.90
0.88
0.80
0.80
0.88
giour (A
gro.n
diy staniing
2.5
A7
0.806
10
15
10
20
10
day number
day number
day number
(a) City-A
(b) City-B
(c) City-C
resccaled live cetee-iiuuteeee orden-aakon rafe
8.00
1.00
1.00
0.50
0.98
0.68
0.07
0.07
Sonill g1-40
rescaled
andi gous
day number
day number
day number
City-D
(c) City-E
City-F
Figure 15: Illustration of the daily responses in the A/B test in the 6 cities.
36

---

Under review as a conference paper at ICLR 2023
Table 8: VISS results GNFC. We bold the lowest error for each task. + is the standard deviation three random seeds.
el_p
e0.2_pl
c0.05_
GALILEO
5.17 + 0.06
4.73 +: 0.13
4.70 0.02
SL
5.15 + 0.23
4.73 0.31
23.64 + 4.86
IPW
5.22 + 0.09
5.50 - 0.01
5.02 = 0.07
SCIGAN
7.05 + 0.52
6.58 0.58
18.55 + 3.50
el el_p0.2
e0.2_p0.2
e0.05_p0.2
GALILEO
5.03 + 0.09
4.72 0.05
4.87 0.15
SL
5.21 + 0.63
6.74 0.15
33.52 1.32
IPW
5.27 + 0.05
5.69 + 0.00
20.23 + 0.45
SCIGAN
16.07 0.27
12.07 + 1.93
19.27 + 10.72
el cl_00.05
e0.2_p0.05
e0.05 00.05_00.05
GALILEO
5.23 + 0.41
5.01 + 0.08
6.17 + 0.33
SL
5.89 + 0.88
14.25 3.48
37.50 + 2.29
IPW
5.21 + 0.01
5.52 + 0.44
31.95 + 0.05
SCIGAN
11.50 7.76
13.05 + 4.19
25.74 + 8.30
Table 9: VMISE results on TCGA. We bold the lowest error for each task. + is the standard deviation three random seeds.
10 bias 2.0
t0 (ias 20.0
t0 bias 50.0
GALILEO
0.34 + 0.05
0.67 + 0.13
2.04 0.12
SL
0.38 + 0.13
1.50 + 0.31
3,06 = 0.65
IPW
6.57 + 1.16
6.88 + 0.30
5.84 0.71
SCIGAN
0.74 + 0.05
2.74 + 0.35
3.19 0.09
t1 bias 2.0
tl bias_ 6.0
tl bias 8.0
GALILEO
0.43 + 0.05
0.25 + 0.02
0.21 0.04
SL
0.47 + 0.05
1.33 + 0.97
1.18 + 0.73
IPW
3.67 + 2.37
0.54 0.13
2.69 1.17
SCIGAN
0.45 + 0.25
1.08 + 1.04
1.01 + 0.77
bias 2.0
12 biasis__6 6.0
12, bias 8.0
GALILEO
1.46 + 0.09
0.85 + 0.04
0.46 + 0.01
SL
0.81 + 0.14
3.74 2.04
3.59 0.14
IPW
2.94 + 1.59
1.24 0.01
0.99 = 0.06
SCIGAN
0.73 + 0.15
1.20 0.53
2.13 1.75
37

---

Under review as a conference paper at ICLR 2023
Table 10: VMMSE results on GNFC. We bold the lowest error for each task. + is the standard deviation of three random seeds.
el el_p1
e0.2_pl
e0.05_p1
GALILEO
3.86 + 0.03
3,99 0.01
4.07 + 0.03
SL
5.73 + 0.33
5.80 0.28
18.78 + 3.13
IPW
4.02 = 0.05
4.15 0.12
22.66 = 0.33
SCIGAN
8.84 = 0.54
12.62 + 2.17
24.21 = 5.20
el_p0.2
e0.2_p0.2
e0.05_p0.2
GALILEO
4.13 + 0.10
4.11 = 0.15
4.21 + 0.15
SL
5.87 0.43
7.44 1.13
29.13 + 3.44
IPW
4.12 0.02
6.12 0.48
30.96 = 0.17
SCIGAN
12.87 = 3.02
14.59 2.13
24.57 = 3.00
el_p0.05
e0.2_p0.00
e0.05_p0.05
GALILEO
4.39 + 0.20
4.34 0.20
5.26 0.29
SL
6.12 - 0.43
14.88 + 4.41
30.81 + 1.69
IPW
13.60 = 7.83
26.27 + 2.67
32.55 = 0.12
SCIGAN
9.19 = 1.04
15.08 + 1.26
17.52 = 0.02
Table 11: v MSE results on TCGA. We bold the lowest error for each task. + is the standard deviation of three random seeds.
10 bias 2.0
t0 bias 20.0
t0 bias 50.0
GALILEO
1.56 + 0.04
1.96 + 0.53
3.16 + 0.13
SL
1.92 + 0.67
2.31 + 0.19
5.11 0.66
IPW
7.42 + 0.46
5.36 5.96
5.38 = 1.24
SCIGAN
2.11 + 0.47
5.23 0.27
5.59 1.02
t1 _ias 2.0
t1 [iubia__6o 6.0
tl bias 8.0
GALILEO
1.43 + 0.06
1.09 + 0.05
1.36 0.36
SL
1.12 + 0.15
3.65 1.91
3.96 1.81
IPW
1.14 + 0.11
0.90 + 0.09
2.04 = 0.99
SCIGAN
3.32 0.88
4.74 2.12
5.17 2.42
b2as 2.0
12_bias_6.0
t2 2__bias_8.0 8.0
GALILEO
3.77 + 0.35
3.99 + 0.40
2.08 0.60
SL
2.70 + 0.67
8.33 5.05
9.70 + 3.12
IPW
2.92 + 0.15
3.90 0.17
4.47 2.16
SCIGAN
3.82 + 2.12
1.83 + 1.49
3.62 + 4.9
38

---

Under review as a conference paper at ICLR 2023
15
32
pidd
02
0.4
0.6
0.8
action
action
(a) t0_ bias
(b) t0 bias
(c) t0_ 10_bias50
10
averaged responize
prod
pred
15
28
35
12
14
pred
action
action
(d) bias2
(e) bias6
(f) tl bias8
verrgged resporse
30.0
25
27.5
22.5
20.0
3.5
17.
prod
pred
12.
6.8
0.5
action
(g) 12__hhahhe bias
(h) 12, biasó
(i) 12 hias8
Figure 16: Illustration of the averaged response curves of Supervised Learning (SL) in TCGA.
39

---

Under review as a conference paper at ICLR 2023
pied real
pred real
averagee ogeed resporse
60
ped real
83
averagee
averaged
85
50
action
action
(a) el_pl
(b) c0.2_pl
(c) c0.05_p1
avera ged responge
averaged response
50
45
51
40
50
35
action
action
action
(d) el_p0.2
(c) c0.2_p0.2
(f) c0.05_pp..2
everaged reppors
pred
36
awer agedd 1eppo15e
65
pred
54
55
62
60
50
-1
action
action
action
(g) el_ p0.05
(h) e0.2_p0.000
(i) e0.05 p0.05
Figure 17: Illustration of the averaged response curves of Supervised Learning (SL) in GNFC.
40

---

Under review as a conference paper at ICLR 2023
10
aweraged resporse
20
10 12
144
averaged
3000
16
p'ed
real
0.2
0.5
0.4
0.8
0.4
055
1.0
action
action
action
(a) t0 bias
(b) 10 10_ias20
(c) 10 bias50
arod real
16
16
averegee
everaged
13
0.8
action
action
action
(d) tI bias2
(e) tl bias6
tl bias8
pred
30
aeed
averogged response
roal
28
roal
averaged responsse
26
20
15
10
0.8
0.5
0.5
0.7
action
action
action
12_ bias2
(h) 12_bias
(i) t2_ bias8
Figure 18: Illustration of the averaged response curves of Inverse Propensity Weighting (IPW) in TCGA.
41

---

Under review as a conference paper at ICLR 2023
56
pred real
14
real
awerager response
54
overaged
53
51
50
action
action
action
(a) el_pl
(b) e0.2_pl
(c) c0.05_pi
55
pred
60
pred real
everaged resporse
real
averoged response
433
85
15
averaged
12
-1
49
action
action
action
(d) el_ _1_00.2
(e) e0.2_p0.2
(f) e0.05 _0.2
averaged response
overaged response
144
32
30
45
30
action
action
action
(g) el el_p0.05
(h) e0.2_p00.05
(i) c0.05_p0..0
Figure 19: Illustration of the averaged response curves of Inverse Propensity Weighting (IPW) in GNFC.
42

---

Under review as a conference paper at ICLR 2023
13
pidd
02
0.4
0.6
0.8
action
action
(a) t0_ bias
(b) t0_ 00
(c) 10_bias50
overaged rexponze
37
15
0.6
35
averaged
34
10
prod
pred real
pred
teal
action
action
(d) bias2
(e) tl bias6
(f) bias8
Averagee resporse
30.0
pred teal
27.5
255
22.5
20.0
Vveraged
0.5
17.5
prod
15.0
6.8
3.5
action
12_ bias2
(h) 12 bias
(i) 12 2_biis
Figure 20: Illustration the averaged response curves of SCIGAN in TCGA.
43

---

Under review as a conference paper at ICLR 2023
averaped 1830002
prod
averaged response
gred
oged response
54
30
action
action
action
(a) el pl
(b) e0.2 pl
(c) e0.05 pl
35
58
AeeDRRe respent
averagen response
50
pred
53
51
51
action
action
action
(d) el__ep___ p0.2
(e) e0.2_p0.2
(f) e0.05 e0.05_pp.2
aversged response
pred
averaged response
62
avere aaraaae response
60
12
52
50
56
-1
action
action
action
(g) el 10.05.55
c0.2 p0.05
(i) c0.05_p0.00
Figure 21: Illustration of the averaged response curves of SCIGAN in GNFC.
44

---

Under review as a conference paper at ICLR 2023
Sow agee response
auaragedd ruspprnce
12
F
action
62
344
1.8
1.0
action
bias
r0_bias20
(c) t0_ bias50
aweraged response
overaged respanse
0.5 action
action
action
bias2
(e) tl bias6
tl [1_bias8
30
28
resporooe
20
0.3
0.5
0.7
action
action
action
bias2
(h) 12, bias6
12, bias8
Figure 22: Illustration of the averaged response curves of GALILEO in TCGA.
45

---

Under review as a conference paper at ICLR 2023
averaged response
$4
overoged ressoonee
53
53
20
-2
action
2
action
action
(a) el_pl
(b) c0.2_pl
(c) e0.05 pl
pred
averaged resporse
everaged responce
real
overagod response
real
12
52
13
51
action
actin
action
(d) el p0.2
(e) e0.2 p0.2
(I) e0.05 p0.2
55
pee
averaged
Ovaroged
aged
50
-1
action
ection
(g) el pl.00.05
(h) c0.2_p0.05
(i) c0.05_p0..05
Figure 23: Illustration of the averaged response curves of GALILEO GNFC.
46


Reviews:
[{"review_id": "m0r5xkpVmSb", "reviewer": "Reviewer_rQ79", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors address the age old offline RL problem of counterfactual prediction, i.e. what would have happened if an unobserved action would have happened at an unobserved state. To do this, they use a weighted empirical risk minimization framework, which is extended with an adversarial framework to learn a model under a worst-case assumption. \n\nThey verify that their model makes accurate counterfactual predictions on unseen data by evaluating on D4RL and a propriety real-world food delivery task. ", "summary_of_the_review": "The authors fail to show their method is relevant for reinforcement learning, as the method merely learns a prediction of the transition function, but they don't show this leads to better policies or better off-policy evaluation.", "strength_and_weaknesses": "Strengths:\n- addresses an important problem\n- applies model based RL to offline RL, a promising avenue of research\n- applies adversarial worst-case ideas, which should in theory result in better \n\nMajor Concerns:\n\nClarity: It took a lot of reading and re-reading for me to understand what this paper is about. The main thing that made it hard for me to understand is that it's sold by the authors as a RL paper, when in fact their method is more a Domain Adaptation method, where they take data from one domain (observations of a transition function under one policy) and try to make predictions in another domain (behavior of a transition function under another policy). This is confusing, since what one is actually interested in is off policy evaluation, knowing how well a an new policy will work based on observation of an old policy.\n\nQuality: I do not thing the authors claims are supported by experimental evidence. The problem with the presented experimental evidence is three-fold: First, they don't compare to other state of the art domain adaptation methods, even though it's a domain adaption paper. Second, their experiments only talk about prediction accuracy of the transition function. It's totally possible to predict accurately enough an environment, but be very inaccurate at predicting a policy's performance in the environment. Last, the authors claim their method's \"policy significantly improves performance\" but only show it for a non-reproducible setup. In RL, the whole point is to improve a policies performance, and the authors can't demonstrate that their method indeed improves performance.\n\nTo be frank, since the authors were unable, in my view, to support their claims with evidence I didn't carefully study every mathematical detail of their method.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YO2JIEvW5X", "reviewer": "Reviewer_bcJS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the model learning problem in sequential decision making. Traditional methods may learn a model that fails under counterfactual samples (unseen in offline training dataset). To imporve the model's generalizability, the authors introduce adversarial weighted empirical risk minimization (AWRM) which learns a model with adversarially selected test-time policies. Although the original formulation of AWRM is intractable, the paper proposes a surrogate objective with several approximations. The effectiveness of the proposed method is verified by experiments in a synthetic environment, 3 MuJoCo environments, as well as a real-world food-delivery platform.", "summary_of_the_review": "This paper addresses the generalization challenge of model learning in sequential decision making problems. The proposed method is reasonable and sound, with theoretical analysis and empirical evidence. ", "strength_and_weaknesses": "Strengths:\n\n1. The main idea of the paper are well-motivated and interesting.\n2. I am not sure how novel and original the relaxation of the adversarial objective is since I am not familiar with the related work. But the technical insights are sound.\n3. The paper is in general well-written with much details well-organized in the appendix.\n4. The experimental results are convincing in terms of showing the major messages.\n\nWeaknesses:\n\n1. It is not clear how the entire algorithm works and how many networks are learned in total. A high-level summarization or even an architecture figure would be helpful.\n2. Many implementation details are omitted in the main paper. How is $H_{M^*}(x,a)$ implemented?\n3. The presentation could be improved. The notations are sometimes not very clear. Are $\\kappa$ and $\\hat{\\mu}$ (in appendix) the same? The notation $a$ is used several times for different meanings. Also, the layout is too tight. It seems that a negative vspace is used very aggresively.\n4. The experiments only focus on the model learning performance in the form of prediction errors. However, it would be better to demonstrate how such a model can help model-based RL (learn a policy from the fitted model).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "59hAmE1V70", "reviewer": "Reviewer_j5p5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces an approach for learning a model of the environment for model-based RL (MBRL) based on counterfactual data. This enables more stable model performance outside of the training distribution. The added generalization is driven by adversarial training to mitigate selection bias influencing the learned relationships between observations and subsequent actions.", "summary_of_the_review": "This is a good paper and one that warrants publication. I however reserve my enthusiastic recommendation because there are points where the discussion could be improved, specifically around the empirical justifications used to validate the development of GALILEO.", "strength_and_weaknesses": "**Strengths**\n\nThis is a very complete and thoroughly written paper. The extensive efforts made by the authors to clearly justify the steps they took in the development of the proposed GALILEO is appreciated. I do wish there was a more detailed \"related work\" section with more concrete discussion about how the proposed approach builds or differs from prior work. However, this concern is mitigated mostly by how well cited the rest of the paper is. Each addition and concept throughout the paper is properly attributed.\n\nI found the step-by-step derivation of GALILEO through Section 4 to be really easy to follow. The writing was very clear in the development of the final use of AWRM. This was capped off with a great discussion about how the theoretical components of the method are implemented (Section 4.3). Unfortunately, this level of detail (and accessibility) was not carried through the experiments (more on that below). \n\nThe extent of the experiments provide some clear signals that the proposed method carries some \"general\" significance outside of the simple synthetic domains many papers use as a proof of concept. It's clear that the models learned with GALILEO are more accurate than those trained with other approaches.\n\n**Weaknesses**\n\nTo quickly jump to the empirical justification. It's disappointing that the improved models learned with GALILEO aren't shown to improve the overall task performance of the individual domains the models were learned to assist with. I can understand that this is not the major focus of the paper but the conceptual justification for needing generalizable models for MBRL is to provide more robust policy learning. As such the reported results feel only partial in nature. It would be great to see what the knock-on effects of an improved model are when learning a policy in the environment. Is there an improved nature of sample efficiency? How about the performance in more diverse test scenarios?\n\nIt's also disappointing that there aren't very thorough baseline comparisons to contemporary MBRL approaches in the accuracy of the learned models. Aside from the simple synthetic tasks, there are not really any baseline comparisons with \"competing\" approaches. I suppose that the SL baseline is better than nothing but the experiments overlook contributions made by the prior work in learning adequate models. Without a discussion about how these approaches devolve into SL, it's hard to place any significance on the experiments.\n\nAt times the discussion is cast in very formal language that distracts from the actual usage (or motivation) of the concepts. It's technically correct to cast \"error\" as \"risk\" (especially since the paper pursues \"risk minimization\") but it comes across as unnecessarily formal in places. Especially when MSE is used as a metric... This isn't a major concern but it does belie some aspects of where the writing in the paper could be improved to facilitate greater clarity. \n\nThe \"real world\" example is too vague in the initial sections of the paper. There is a lot of weight placed on the contributions of GALILEO working on a real domain but there's not sufficient framing or discussion about why this experiment works or is necessarily best solved by GALILEO (esp. when not compared to relevant baselines). \n\nThere are missing references to prior work using counterfactuals to improve environment modeling. In particular, I would interested to hear from the authors why they omit contributions from prior work such as Sontakke, et al (ICML 2021).\n\n```Sontakke, S. A., Mehrjou, A., Itti, L., & Schölkopf, B. (2021, July). Causal curiosity: RL agents discovering self-supervised experiments for causal representation learning. In International Conference on Machine Learning (pp. 9848-9858). PMLR.```\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject