Adversarial Counterfactual Environment Model Learning
Authors: Xiong-Hui Chen
Yang Yu
Zhengmao Zhu
ZhiHua Yu
Chen Zhenjun
Chenghe Wang
Yinan Wu
Rong-Jun Qin
Hongqiu Wu
Ruijin Ding
Huang Fangsheng
Venue: NeurIPS 2023 spotlight (2023)
Keywords: environment model learning
offline reinforcement learning
off-policy evaluation
individual treatment effects estimation
causal inference
adversarial learning
TL;DR: We propose a new environment model learning techniques with better generalization ability on counterfactual data.

Abstract: An accurate environment dynamics model is crucial for various downstream tasks in sequential decision-making, such as counterfactual prediction, off-policy evaluation, and offline reinforcement learning. 
Currently, these models were learned through empirical risk minimization (ERM) by step-wise fitting of historical transition data. This way was previously believed unreliable over long-horizon rollouts because of the compounding errors, which can lead to uncontrollable inaccuracies in predictions. In this paper, we find that the challenge extends beyond just long-term prediction errors: we reveal that even when planning with one step, learned dynamics models can also perform poorly due to the selection bias of behavior policies during data collection. 
This issue will significantly mislead the policy optimization process even in identifying single-step optimal actions, further leading to a greater risk in sequential decision-making scenarios.
To tackle this problem, we introduce a novel model-learning objective called adversarial weighted empirical risk minimization (AWRM).  AWRM incorporates an adversarial policy that exploits the model to generate a data distribution that weakens the model's prediction accuracy, and subsequently, the model is learned under this adversarial data distribution.
We implement a practical algorithm, GALILEO, for AWRM and evaluate it on two synthetic tasks, three continuous-control tasks, and  \textit{a real-world application}. The experiments demonstrate that GALILEO can accurately predict counterfactual actions and improve various downstream tasks, including offline policy evaluation and improvement, as well as online decision-making.

Adversarial Counterfactual Environment Model Learning
Xiong-Hui Chen Yang Yu 12., Zheng-Mao Zhu Zhihua Yu 12.3 Zhenjun Chen 1,2.3
Chenghe Wang Yinan Wu", Hongqiu Wu Rong-Jun Qin Quuiinn Ding Fangsheng Huang" National Key Laboratory for Novel Software Technology, Nanjing University 2 School of Artificial Intelligence, Nanjing University, Meituan, Polixiraai, Tsinghua University edu. en, (yuzh, [bennj]]@smail www..ne.e....... edu. en
{chenxh, yuy, zhuzm) @lamda wangch@lamda, nju. edu. edu. en, wuyinan02@meituua com, com, {wuhq, qinr @lmlamda nju, edu. en drj 1711maall 1s. ts tsinghua, edu. en, huangfangshenggdneet tuan. com
Abstract
An accurate enviromment dynamics model is crucial for various downstream tasks in sequential decision-making, such counterfactual prediction, off-policy eval- uation, and offline reinforcement learning. Currently, these models were learned through empirical risk minimization (ERM) by step-wise fitting of historical transi- tion data. This way was previously believed unreliable over long-horizon rollouts because of the compounding errors, which can lead uncontrollable inaccuracies in predictions. In this paper, we find that the challenge extends beyond just long- term prediction errors: we reveal that even when planning with one step, learned dynamics models can also perform poorly due to the selection bias of behavior policies during data collection. This issue will significantly mislead the policy optimization process even in identifying single-step optimal actions, further leading to a greater risk sequential decision-making scenarios. To tackle this problem, we introduce a novel model-learning objective called adversarial weighted em- pirical risk minimization (AWRM). AWRM incorporates an adversarial policy that exploits the model to generate data distribution that weakens the model's s prediction accuracy, and subsequently, the model is learned under this adversarial data distribution. We implement practical algorithm, GALILEO, for AWRM and evaluate it on two synthetic tasks, three contimuous-control tasks, and a real-world application. The experiments demonstrate that GALILEO accurately predict counterfactual actions and improve various downstream tasks, including offline policy evaluation and improvement, well online decision decision-making making.
Introduction
good environment dynamics model for action-effect prediction is essential for many downstream tasks. For example, humans agents can leverage this model conduct simulations understand future outcomes, evaluate other policies' performance, and discover better policies. With environment models, costly real-world trial-and-error processes can be avoided. These tasks are vital research problems in counterfactual predictions [52, 1J, off-policy evaluation (OPE) [32, 35], and offline reinforcement learning (Offline RL) [31, 59, 12, 13, 10]. In these problems, the core role of the models is to answer queries on counterfactual data unbiasedly, that is, given states, correctly answer what might happen if we were carry out actions unseen in the training data. However, addressing counterfactual queries differentiates environment model learning from standard supervised learning (SI.) which directly fits the offline dataset for empirical risk minimization (ERM). essence,
'The authors contribute equally
Yang Yu is the corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

variation of next y prediction
y
4 5D) (4. 6 (2.3
62.5)
ball's y-axis location
80
real GAULEE
70
0.4
22 0.13
0,
0.02
2
SL.
5) 459
selection Nas
1.17
trajectory
30
real
AWRM-orade
4
-2
40 (1, 10)
target line
20
GAULEO
PoT 3.5
S33to
netion
SL
2
-1.0 -0.5 0.0 0.5 10
X
action
variation of action value
(a) selection bias example
(b) responses to actions in (c) responses to actions in the training data
the action space
An example of selection bias and predictions under counterfactual queries. Suppose a ball locates in a 2D plane whose position is 8t = (Te, at time L. The ball will move to $141 = (11+1, according I++1 It + 1 and Dt-1 ~ N(,, 0, 2). Here, at is chosen by control policy a ~ (a(s)) = ((( gt)/15, 0.05) parameterized by Ø. which tries to keep the ball near the line y = Φ. In Fig. I(a), the behavior policy j is 66..5. Fig. 1(b) shows the collected training data and the learned models' prediction of the next position of y. Besides, the dataset superfacially presents the relation that the corresponding next y will be smaller with larger action. However, the truth is not because the larger a, causes a smaller yt+1, but the policy selects a small a when Yt is close to the target line. Mistakenly exploiting the "association" will lead to local optima with serious factual errors, e.g., believed that y++1 Xx T (yela) + ag x Φ 14a,,
Figure I:
where T is the inverse function of TT. When we estimate the response curves by fixing y and reassigning action de with other actions + Aa, where Aa E [-1,1] is variation of action value, we found that the model of SI. indeed exploit the association and give opposite responses, while in AWRM and its practical implementation GALILEO, the predictions
closer to the ground truths (yt+1 X Y/ + at). The result is in Fig. I(c), where the darker a region is, the more samples are fallen in. AWRM injects data collected by adversarial policies for model learning to eliminate the unidentifiability between 1++1 X T
1 (yrla) + a, and wyr++ x y + in offline data.
the problem involves training the model on one dataset and testing it on another with shifted distribution, specifically, the dataset generated by counterfactual queries. This challenge surpasses the SL's capability it violates the independent and identically distributed (i.id.) assumption [31].
In this paper, we concentrate on faithful dynamics model learning in sequential decision decision-making settings like RL. Specifically, we first highlight a distinct situ- ation of distribution shift that can easily lead to catas-
moan squarr eneor (MSE) the dastset
160
trophic failures in the model's 'ss predictions: In many real-world applications, offline data is often gathered using a single policy with exhibiting selection bias, meaning that, for each state, actions are chosen un- fairly. As illustrated in Fig. I(a), to maintain the ball's trajectory along target line, a behavior policy applies a smaller force when the ball's location is nearer to the target line. When dataset is collected with such selection bias, the association between the states (loca- tion) and actions (force) will make SL hard to identify the correct causal relationship of the states and actions to the next states respectively (see Fig. 1())). Then when we query the model with counterfactual data, the predictions might be catastrophic failures. Finally, of- fline policy optimization based on this SL model, even for just seeking one-step optimal actions, will select also a totally opposite direction of policy improve- ment, making the offline policy learning system fail. The selection bias can be regarded as an instance of the distributional-shie problem in offline model-based RL, which has also received great attention [31,59, 14, 34). However, previous methods employing naive super- vised learning for environment model learning tend to overlook this issue during the learning process,
122
-5
62. 10 422 1080 112. 1250
adversral poley's parameter 9 the counterfectaal detase!
Figure 2: An illustration of the prediction error in counterfactual datasets. The error of SL is small only in training data (Ø- 62.5) but becomes much larger in the dataset "far away from" the training data. AWRM- oracle selects the oracle worst counterfactual dataset for training for each iteration (pseu- docode is in Alg. 2) which reaches small MSE in all datasets and gives correct re- sponse curves (Fig. 1(c)). GALILEO approx- imates the optimal adversarial counterfactual data distribution based the training data and model. Although the MSE of GALILEO is bit larger than SL in the training data, in the counterfactual datasets, the MSE is on the same scale as AWRM-oracle.

---

addressing it instead by limiting policy exploration and learning in high-risk regions. So far, how to learn faithful environment model that can alleviate the problem directly has rarely been discussed.
In this work, we focus on faithful environment model learning techniques. The work is first inspired by weighted empirical risk minimization (WERM), which is typical solution to solve the selection bias problem in causal inference for individual treatment effects (ITEs) estimation in many scenarios like patients' treatment selection [26, 1, 41J. ITEs measure the effects of treatments on individuals by administering treatments uniformly and evaluating the differences in outcomes. To estimate ITEs from offline datasets with selection bias, they estimate an inverse propensity score (IPS) to reweight the training data, approximating the data distribution under a uniform policy, and train the model under this reweighted distribution. Compared with ITEs estimation, the extra challenge of faithful model learning in sequential decision-makinn settings include: (1) the model needs to answer queries on numerous different policies, resulting in various and unknown target data distributions for reweighting, and (2) the IPS should account for the cumulative effects of behavior policies on state distribution rather than solely focusing on bias of actions. To address these issues, we propose an objective called adversarial weighted empirical risk minimization (AWRM). For each iteration, AWRM employs adversarial policies to construct an adversarial counterfactual dataset that maximizes the model's s prediction error, and drive the model to reduce the prediction risks under the adversarial counterfactual data distribution. However, obtaining the adversarial counterfactual data distribution is infeasible in the offline setting. Therefore, we derive an approximation of the counterfactual data distribution queried by the optimal adversarial policy and provide a tractable solution to learn model from the approximated data distribution. As a result, we propose a practical approach named Generative Adversarial Ine counterfactual, Environment mOdel learning (GALILEO) for AWRM. Fig. 2 illustrates the difference in the prediction errors learned by these algorithms.
Experiments are conducted in two synthetic tasks, three continuous-control tasks, and a real-world application. We first verify that GALILEO can make accurate predictions counterfactual data queried by other policies compared with baselines. We then demonstrate that the model learned by GALILEO is helpful to several downstream tasks including offline policy evaluation and improvement, and online decision- makinn in a large-scale production environment.
Preliminaries
We first introduce weighted empirical risk minimization (WERM) through inverse propensity scoring (IPS), which is commenly used in individualized treatment effects (ITEs) estimation [43]. It can be regarded as a scenario of single-step model learning We define M (2/a,z a) as the one-step environment, where x denotes the state vector containing pre-treatment covariates (such as age and weight), a denotes the treatment variable which is the action intervening with the state x, and is the feedback of the enviromment. When the offline dataset is collected with a behavior policy p(a) (a)z) which has selection bias, a classical solution to handle the above problem is WERM through IPS [48, 3, 29]:
Definition 2.1. The learning objective of WERM through IPS formulated as E MEM
min
Be.aurri' [wxz, a) a) (ylz, a). v)],
where wxx, a) == [(aaz)
p(alz)) and μ denote the policies in testing and training domains, and the joint probability phr (z,, a, y) := Pu(z))(a(x))M) M (y/x,, a) in which Po(x)) is the distribution of state, M the model space. f is a loss function.
The w is also known as importance sampling (IS) weight, which corrects the sampling bias by aligning the training data distribution with the testing data. By selecting different @ to approximate w to learn the model M, current environment model learning algorithms employing reweighting are fallen into the framework. For standard ITFEs estimation, treatments, where ( is an approximated behavior policy μ. Note that it is reasonable weight in ITES estimation: ITEs are defined to evaluate the differences of effect on each state under a uniform policy.
z (i.e., ,3 is a uniform policy) for balancing
In sequential diciiiionmaking setting, decision-making processes in sequential enviromment are often formulated into Markov Decision Process (MDP) [51]. MDP depicts an agent interacting with the environment through actions. In the first step, states are sampled from an initial state distribution zo ~ po(z)) Then at each time-step E (0, 1, 2....... .. the agent takes an action a, E A through a policy TT TTa (2a) e II based on the state 2, E X, then the agent receives reward rt from
3

---

reward function r(x,, ax) E R and transits to the next state 2:+1 given by transition function
M (#++1 12,, a,) built in the environment. II, X. and A denote the policy, state, and action spaces.
Related Work
We give related adversarial algorithms for model learning in the following and leave other related work in Appx. F. In particular, ITEs in Rubin causal model [48] and causal effect estimation in structural causal model [38] attracted widespread attention in recent years [56, 55, 58, 7]. GANTIE [58] uses generator to fill counterfactual outcomes for each data pair and a discriminator to judge the source (treatment group control group) of the filled data pair. The generator is trained to minimize the output of the discriminator. [7] propose SCIGAN to extend GANITE to continuous ITEs estimation via hierarchical discriminator architecture. In woaldw applications, environment model learning based on Generative Adversarial Imitation Learning (GAIL.) has also been adopted for sequential decision-makee making problems [25]. GAIL is first proposed for policy imitation [25]. which uses the imitated policy to generate trajectories by interacting with the environment. The policy is learned with the trajectories through RL which maximizes the cumulative rewards given by the discriminator. [47, 11] use GAIL for environment model learning by regarding the environment model as the generator and the behavior policy as the "environment" in standard GAIL. [16] further inject the technique into unified objective for model-based RL, which joints model and policy optimization. Our study reveals the connection between adversarial model learning and the WERM through IPS, where previous adversarial model learning methods can be regarded as partial implementations of GALILEE, explaining the effectiveness of the former.
4 Adversarial Counterfactual Environment Model Learning
In this section, first propose a new offline model- model-learning objective for sequential decision-make making setting in Sec. 4.1; In Sec. 4.2, we give a surrogate objective to the proposed objective; Finally, we give practical solution in Sec. 4.3.
4.1 Problem Formulation
In scenarios like offline policy evaluation and improvement, it is crucial for the environment model to have generalization ability in counterfactual queries, we need to query accurate feedback from M for numerous different policies. Referring to the formulation of WERM through IPS in Def. 2.1, these requirements necessitate minimizing counterfactual-quess risks for M under multiple unknown policies, rather than focusing on specific targel policy 33. More specifically, the question is: If 3 is unknown and can be varied, how we generally reduce the risks in counterfactual queries? In this article, we call the model learning problem in this setting "counterfactual environment model learning" and propose a new objective address the issue. To be compatible with multi-step environment model learning, we first define a generalized WERM through IPS based on Def. 2.1:
Definition 4.1. In an MDP, given transition function M that satisfies (*'x'axr, > 0, Vz X. Y. E A, Vx' E X' and j satisfies p(alz)) > 0, Va E A.Yz Ya E X, the learning objective of generalized WERM through IPS is:
min E Bra (w(x,, a, x') e (x,,, MEM
where w(x, a, x²) = Cy. (m,,,") Par. (m.0..7) Phr-- and pa.- the training and testing data distributions collected by policy μ and B3 respectively. We define
== (MM [x, a), x²) ) for brevity.
In an MDP, for any given policy म, we have pl- (x, a, x') = Pār- (alr (<''<rr a) where be defined as "Pr( = æf M/") [51, 25] where P [xr = x (zn, M*] is the discounted state visitation probability that the policy visits x at time-step by executing in the environment M and starting at the state xo. Here E [0, 1] is the discount factor. We also define phr- (I, a) := PTr (x) ((a (x) for simplicity.
PMg. (1) denotes the occupancy measure of x for policy π. This measure
7) 1)
With this definition, w can be rewritten: w Pe (a) 2.0) (x,a) In single-step
Pu (=)a(a)2)M 2.0) P (x,a)
environments, for any policy π, PAr. (x) Po (z). Consequently, we obtain w (a(a)((ala) ((a(m)
p)(=) p(a
and the objective degrade Eq. (1). Therefore, Def. 2.1 is special case of this generalized form.
Remark 4.2. w is referred as density ratio and is commonly used to correct the weighting of rewards in off- off-policy datasets to estimate the value of a specific target policy in off-policy evaluation

---

[35, 60]. Recent studies in offline RL also provide similar evidence through upper bound analysis. suggesting that offline model learning should be corrected specific target policies' distribution using [57]. We derive the objective from the perspective of selection bias correlation, further demonstrate the necessity and effects of this term.
In contrast to previous studies, in this article, we would like to propose objective for faithful model learning which can generally reduce the risks in counterfactual queries in scenarios where unknown and can be varied. To address the problem, we introduce adversarial policies that can iteratively induce the worst prediction performance of the current model and propose to optimize WERM under the adversarial policies. In particular, we propose Adversarial Weighted empirical Risk Minimization (AWRM) based on Def. 4.1 to handle this problem.
Definition 4.3. Given the MDP transition function M the learning objective of adversarial weighted empirical risk minimization through IPS is formulated as
min max LPp M) MEM
min max E MEM BEll s.aaarnp [w (l0z,, Yea (2,, a a,,²)),
(3)
where (1r)))/p)))
(2.a(1.a) and the re-weighting term w(x, a (ār-) is conditioned the (r,a)
distribution P.-- of the adversarial policy 3. In the following, we will ignore Ph.-- and use wi a) for brevity.
In nutshell, Eq. (3) minimizes the maximum model loss under all counterfactual data distributions P B II to guarantee the generalization ability for counterfactual queried by policies in II.
4.2 Surrogate AWRM through Optimal Adversarial Data Distribution Approximation
The main challenge of solving AWRM is constructing the data distribution PM of the best-response policy 3* in M since obtaining additional data from M can be expensive in real- world applications. In this paper, instead of deriving the optimal 8", our solution is to offline estimate the optimal adversarial distribution Par- (z, 0, x²) with respect to M, enabling the construction of surrogate objective to optimize M without directly querying the real environment M."
In the following, we select f as the negative log- log-likeldhood loss for our full derivation, instantiating L(P M) in Eq. (3) as: Ez,arpi [w a a Eas- (- log M [z, a))], where Eas- [-] denotes Ey (x')x,0) [-]- Ideally, for any given M, it is obvious that the optimal / is the one that makes 2 (I, a) assign all densities to the point with the largest negative log-likelihood. However, this maximization process is impractical, particularly in continuous spaces. To provide a tractable yet relaxed solution, we introduce an L2 regularizer to the original objective in Eq. (3).
min L L( M) MEM PEN
min max L(Phe- M) MEM
a 2 "1lpss (c-)11.
(4)
where denotes the regularization coefficient of P and 1ley- (111111 Sx.A(Phr- (z, ?
Now we present the final results and the intuitions behind them, while providing a full deriva- _ tion in Appx.A. Suppossing we have PMr- representing the approximated data distribution of the approximated best-response policy 8 under model Ms parameterized by 0, we can find the op- timal 6* of ming maxsen I(p - Ma) (Eq. (4)) through iterative optimization of the objective 0.+1 ming T(p Mo). To this end, approximate P4r via the last-iteration model Mo, and derive an upper bound objective for ming I
Phro, (I,, (,,,,x)) Pig- (zt, (x, a.)"))
Biso, (I,
Oct1 min E
log Mo Mo (o 00(
Has- (x, stochasticity
Phic- (I)
discrepancy
density rratio baseline
W(a,a,a")
(5)
where Epg [:] denotes E Ex H. is a convex and lower semi-continuous (l.s.c.) function
satisfying (x)x S Vv E X, which also called / function in f-divergence [2], oo(2, a) =
aM, 2 (z, a), and HM- (z, a) denotes the entropy of M (112,
5

---

Remark 4,4 (Intuition of W). After derivation, we found that the optimal adversarial data distribution
can be approximated by P (a, a) -xx Piu- (I, a, <' d') (see Appx. A), leading to the upper-bound objective Eq. (5), which is a WERM dynamically weighting by W. Intuitively, W assigns more learning propensities for data points with (1) larger discrepancy between Pitee
(generated by model) and Pa.- (real-data distribution), or (2) larger stochasticity of the real model M The latter is contributed by the entropy HM-, while the former contributed by the first two terms combined. In particular, through the definition of f-divergence, we known that the discrepancy of two distribution P and Q can be measured by fx ((r))(( thus the terms (C'Mo, (x, a, x/) P'r-- (ar, a, x')) can be interpreted as the discrepancy measure unit between P'Ms. (I., a, x') and P-M- (z, a, z²), while f (P'M,, (z, a)) P (z, a)) serves baseline and measured by f to balance the discrepancy contributed by x and a, making M focus on errors on a'.
summary, by adjusting weights W, the learning process will iteratively exploit subtle errors of the current model in any data point, regardless of how many proportions it contributes in the original data distribution, to eliminate potential unidentifubility on counterfactual data caused by selection bias
4.3 Tractable Solution
In Eq. (5), the terms f Mo (z,,, (I, a, (z)) 1 (x, a)/ (o)///ir-- Pirr- a)) are still intractable. Thanks to previous successful practices in GAN [19] and GAiL. [25], we achieve the objective via ddiseriiinnttrpparaaaade
objective through similar derivation. We show the results as follows and leave the complete derivation in Appx. A.4. In particular, by introducing two discriminators Dig (z, x') and Dug (z,,),, we can optimize the surrogate objective Eq. (5) via:
0.11 max (5 AA (z, a, a'x)) log Ma (x' 12, a)] + Eoi, [[Har- (e, Appipt (2, a, x')) log Mo(e'|x, [x)
s.l.
pú max [Bpg [log Do (vc, a, x') + E piig, [log(1 - (xo(
pi = arg max FF1
(Reie- Dlog px (z, a)l + Bosso [log(1 Drs (z)
(6)
a simplification of E = [1) Apg.vt i (x,, a, x') log Deg (x, a, x') log Duy (I, a), and you and şi are the parameters of Do and D',, respectively. We learn a pol- icy j = μ via imitation learning based on the offline dataset Dreat [40, 25]. Note that in the process, we ignore the term (1 a) for simplifying the objective. The discussion on the impacts of removing o0(x,a)) a) is left in App. B.
where Bpg [-]
The overall optimization pipeline illus- tration in Fig. 3. In Eq. (6), the reweighting term W from Eq. (5) is split into two terms in the RHS of the equation: the first term a GAlL-style objective [25], treating Mo the policy generator, as the environ- ment, and A as the advantage function, while the second term WERM through HM- Apj.pp The first term resembles the previous adversarial model learning ob- jectives [46, 47, 11]. These two terms have intuitive explanations: the first term assigns learning weights on data generated by the model Mo., If the predictions of the model appear realistic, mainly assessed by D the propensity weights would be increased, encouraging the model to generate more such kind of data; Conversely, the second term assigns weights on real data generated by M If the model's 's predictions seem unrealistic (mainly assessed by - -Dx;) stochastic (evaluated by HM-), the propensity weights will be increased, encouraging the model to pay more attention to these real data points when improving the likelihoods.
(3) update the dynomics model
advessria! zeweighting W
Offline dhia ply-
Discrimimtor Pg.
Dynamics model Ma
Generated
si,
with benavior policy
(2) update discrininctors
(3) data generation
Figure 3: Illustration of the GALILEO workflow.

---

GALILEO
GAULEO
25
SCKSAN
IPWI
rool MISE
IPW SCIGAN
20
5
5
61,07
01_602
602.p
2.2
a0.2
p0.05
15.05,1 p1
00.05
.0.2
p0.05
10 Ne
B.bia
(a) ONFC
(b) TCGA
Figure 4: Illustration of the performance in GNFC and TCGA. The grey bar denotes the standard
error (x0.3 for brevity) of 3 random seeds.
Based on the above implementations, we propose Generative Adversarial Ine counterfactual Environment mOdel learning (GALILEO) for environment model learning. We list brief of GALILEO in Alg. 1 and the details in Appx. E.
Algorithm 1 Pseudocode for GALILEO
Input:
Dreat:
offline dataset sampled from Phs where y is the behavior policy; N: total iterations;
Process:
1: Approximate behavior policy á via behavior cloning through offline dataset Dreal
2: Initialize an environment model Mø,
3: for t = N do
4: Use generate a dataset Dgen with the model Ma.
5: Update the discriminators Depo and Dup through the second and third equations in Eq. (6)
where Phet, is estimated by D Dgen and pka-- estimated by Dreal
6: Generative adversarial training for Mg. by regarding Aegret as the advantage function and computing the gradient to Adg,, named Ipg. with a standard policy gradient method like
TRPO [44] or PPO [45] based on Pgen-
7: Regard HM-- Aug,se _o the reweighting term for WERM and compute the gradient to Mo.
based on Dreat- Record it as Is)-
8: Update the model 8.41 x 9: + 9pg 9st.
9: end for
Experiments
In this section, we first conduct experiments in two synthetic environments to quantify the performance of GALILEO on counterfactual queries 3 Then we deploy GALILEO in two complex environments: MuJoCo in Gym [53] and real-world food-delivery platform to test the performance GALILEO in difficult tasks. The results are in Sec. 5.2. Finally, to further verify the abiliy GALILEO, in Sec. 5.3, we apply models learned by GALILEO to several downstream tasks including off-policy evaluation, offline policy improvement, and online decision-making in production environment. The algorithms compared are: (1) SL.: using standard empirical risk minimization for model learning: (2) IPW [50]: standard implementation of WERM based IPS; (3) SCIGAN [7]: an adversarial algorithms for model learning used for causal effect estimation, which can be roughly regarded as a partial implementation of GALILEO (Refer to Appx. E.2). We give a detailed description in Appx. G.2.
5.1 Environment Settings
Synthetic Environments
Previous experiments on counterfactual environment model learning are based on single-step semi-synthetic data simulation [7]. As GALILEO compatible with single-step environment model learning, we first benchmark GALILEO in the same task named TCGA as previous studies do [7]. Based on the three synthetic response functions, construct 9 tasks by choosing different parameters of selection bias on f which is constructed with beta distribution, and
codee hutp://giiallccmmioognhuaaalaall
7

---

design coefficient e to control the selection bias. We name the tasks with the format of "t? bias?". For example, t t1_bias2 is the task with the first response functions and 2. The details of TCGA is in Appx. G.1.2. Besides, for sequential environment model learning under selection bias, we construct a new synthetic environment, general negative feedback control (GNFC), which can represent classic type of task with policies having selection bias, where Fig. ((a) is also an instance of GNFC. We construct 9 tasks on GNFC by adding behavior policies j with different scales of uniform noise (--e,e) e) with probabilities p. Similarly, we name them with the format "e?_p?"
Continuous-control Environments We select MuJoCo environments from D4RL [17] to con- struct our model learning tasks. We compare it with a standard transition model learning algorithm used in the previous offline model-based RL algorithms [59, 30J, which is standard supervised learning. We name the method OFF-SL. Besides, we also implement IPW and SCIGAN as the base- lines. In D4RL benchmark, only the "medium" tasks is collected with a fixed policy, i.e., the behavior policy is with 1/3 performance to the expert policy), which is most matching to our proposed problem. So we train models in datasets HalfCheetah-medium Walker2d-mediue
and Hopper-medium.
A Real-world Large-scale Food-delivery Platform We finally deploy GALILEO in a real-world large-scale food-delivery platform. We focus on a Budget Allocation task to the Time period (BAT) in the platform (see Appx. G.1.3 for details). The goal of the BAT task is to handle the imbalance problem between the demanded orders from customers and the supply of delivery clerks in different time periods by allocating reasonable allowances to those time periods. The challenge of the environment model learning in BAT tasks is similar the challenge Fig. 1: the behavior policy is a human- suman-expert policy, which tends to increase the budget of allowance in the time periods with a lower supply of delivery clerks, otherwise tends to decrease the budget (We gives a real-data instance in Appx. G.1.3).
5.2 Prediction Accuracy on Shifted Data Distributions
Test in Synthetic Environments For all of the tasks, we select mean-integrrate--square error MISE E JA (M"')", a) M(x' xr a))² da as the metric, which is a metric to measure the accuracy in counterfactual queries by considering the prediction errors in the whole action space. The results are summarized in Fig. 4 and the detailed results can be found in Appx. H. The results show that the property of the behavior policy (i.c., e and p) dominates the generalization ability of the baseline algorithms. When e = 0.05, almost all of the baselines fail and give a completely opposite response curve, while GALILEO gives the correct response. (see Fig. 5). IPW still performs well when 0.2 S e S 1.0 but fails when e = 0.05. p <= 0.2. We also found that SCIGAN can reach better performance than other baselines when but the results in other tasks are unstable. GALILEO is the only algorithm that is robust to the selection bias and outputs correct response curves in all of the tasks. Based the experiment, we also indicate that the commonly used overlap assumption is unreasonable to a certain extent especially in real-world applications since it is impractical to inject noises into the whole action space. The problem of overlap assumption being violated, c.g., e < in our setting, should be taken into consideration otherwise the algorithm will be hard to use in practice if it is sensitive to the noise range. On the other hand, we found the phenomenon in TCGA experiment is similar to the one in GNFC, which demonstrates the compatibility of GALILEO to single-step environments.
0.05,p p <= 0.2,
65
real
averaged response
50
GALILEO
SL
55
IPW SCIGIN
a5
We also found that the results of IPW unstable in TCGA experiment. It might be because the behavior policy is modeled with beta distribution while the propensity score js is modeled with Gaussian distribution. Since IPW
40
directly reweight loss with 1 the results are sensitive to the error of j..
0.2 0.4 1.0 normalized action
Finally, we plot the averaged response curves which are constructed by equidistantly sampling action from the action space and averaging the feed- back of the states in the dataset as the averaged response. One result is be in Appx. H). For those tasks where base-
Figure 5: Illustration of the averaged re- sponse curves in task e0.05_p00
in Fig. 5 (all curves lines fail in reconstructing response curves, GALILEO not only reaches a better MISE score but reconstructs almost exact responses, while the baselines might give completely opposite responses.
Test in MuJoCo Benchmarks We test the prediction error of the learned model in corresponding unseen "expert" and "medium-replay" datasets. Fig. 6 illustrates the results in halfcheetah. We can see that all algorithms perform well in the training datasets. OFF-SI. can even reach a bit lower error. However, when we verify the models through "expert" and "medium-replay" datasets, which
8

---

14
06--4
1.75
4.00
2.50
130
2:30
2.25
0.0
6.4
update steps
upddate steps
updiate steps
(0) medium (train)
(b) medium-replay (test)
(c) expert (test)
Figure 6: Illustration of learning curves of the halfcheetah tasks (full results are in Appx. H.5). The
figures with titles ending in "(train)" means the dataset is used for training while the titles ending in "(test)" means the dataset is just used for testing. The X-axis records the steps of the environment model update, and the Y-axis is the prediction errors in the corresponding steps evaluated by the datasets. The solid curves the mean reward and the shadow is the standard error of three seeds.
are collected by other policies, the performance of GALILEO more stable and better than all other algorithms. As the training continues, the baseline algorithms phenomenon are similar among three datasets, and we leave the full results in Appx. H.5.
gets worse and worse. The
rescated fie-mieeuu ro--mtuteee ordentzion ride-taten rale
resoaled everajes rasporee
1.00
CALLEO
1.0
1.00
Peesaao oumularive treatment
0.5
0:90
c.94
0.84
0:90
6.88
0.00
00 0.2
08 10 sumulative samples (rescaled 1)
nomalized allocated allowances
day number
(a) response curves in City A
(b) AUUC curves
(c) A/B test the City A
Figure 7: Parts of the results in BAT tasks. Fig. 7(a) demonstrate the averaged response curves of the SL and GALILEO model in City A. is expected to monotonically increasing through our prior knowledge. In Fig. 7(b) show the AUCC curves, where the model with larger areas above the "random" line makes better predictions randomized-ccottrlled---
data [61].
Test in a Real-world Dataset We first learn a model to predict the supply of delivery clerks (measured by fulfilled order amount) given allowances. Although the SL. model can efficiently fit the offline data, the tendency of the response curve is easily to be incorrect. As can be seen in Fig. 7(a), with larger budget of allowance, the prediction of the supply is decreased in SI., which obviously goes against our prior knowledge. This is because, in the offline dataset, the corresponding supply will be smaller when the allowance is larger. the model of SL, the optimal solution is canceling all of the allowances, which is obviously incorrect in practice. On the other hand, the tendency of GALILEO'S ' response is correct. In Appx. H.7. we plot all the results in 6 cities. We further collect some randomized controlled trials data, and the Area Under the Uplift Curve (AUUC) [6] curve Fig. 7(b) verify that GALILEO gives reasonable sort order on the supply prediction while the standard SL technique fails to achieve this task.
conceivable that we learn policy through
5.3 Apply GALILEO to Downstream Tasks
Off-policy Evaluation (OPE) We first verify the ability of the models in MuJoCo environments by adopting them into off-policy evaluation tasks, We use 10 unseen policies constructed by DOPE benchmark [18] to conduct our experiments.
Table I: Results of OPE on DOPE benchmark. We list the averaged performances three tasks. The detailed results are in Appx. H.6. is the standard deviation among the tasks. We bold the best scores for each metric.
Algorithm
Norm. value gap
Rank corr.
Regret
We select three common-used metries: value gap, re- gret@1 1, and rank correlation and aver- aged the results among three tasks Tab. 1. The basclines and the corre-
GALILEO Best DICE
0.37 0.24
0.44 0.10
0.09 0.02
0.48 0.19
0.15 0.04
0.42 0.28
VPM
0.71 1 0.04
0.29 :: 0.15
0.17 1 0.11
FQE (L2)
0.54 1 0.09
1.10
0.34 1 0,03
0.67 1 0.01
-0.40
0,36 0.27
Doubly Rubost
0.57 + 0.07
00.14
0.33 - 0.06

---

Table 2: Results of policy performance directly optimized through SAC [20] using the learned dynamics models and deployed in MuloCo environments. MAX-RETURN is the policy performance of SAC in the MuJoCo environments, and "avg. norm." is the averaged normalized return of the policies in the 9 tasks, where the returns are normalized to lie between 0 and 100, where a score of 0 corresponds to the worst poliey, and 100 corresponds to MAX-RETURN.
Tisk
Hopper
Walker2d
HallCheetah
avg. nomm.
Horioon
I1=10
II=20
1==40
I=:10
II=20
I==00
R==10
Il=20
[==00
GALILEO
13.0 0.1
332 0.1
53.5 1.2
11.7 0.2
29. 0.3
61.2 1.2:1. 34
0:02202
1
-142
51.1
OFF-SL
0.
30=02
0.2
10.7 022
00
37
0.
+1.
-3322
21.1
IPW
5.7
0.
5.2
4,7 1.1
28
14.5 144
0.2
0.5
1.
19.7
SCIGAN
29.2 0.6
46.2
8.
9.2
1.8
0.3
0.3
1.1.4...3
418
MAX RETURN
13.2
33.3 1.2
1.0
149 133
60.71111
2211 198
2.11
13.
49.1
1000
sponding results we used are the same as the one proposed by [18]. As seen in Tab. 1, compared with all the baselines, OPE by GALILEO always reach the better performance with large margin (ar least 23%, 193% and 47% rexpectively), which verifies that GALILEO can eliminate the effect of selection bias and give correct evaluations on unseen policies.
Offline Policy Improvement We then verify the generalization ability of the models in MuJoCo environments by adopting them into offline model- based RL. To strictly verify the ability of the models, we abandon all tricks to suppress policy exploration and learning in risky regions as current offline model-based RI algorithms [59] do, and we just use the standard SAC algorithm [20] to fully exploit the models to search an optimal policy. Unfortunately, we found that the compounding error will still be inevitably large in the 1,000-step rollout, which is the standard horizon in MuJoCo tasks, leading all models to fail to derive a reasonable policy. To better verify the effects of models policy improvement, we learn and evaluate the policies with three smaller horizons: E {10, 20, 40). The results are listed in Tab. 2. We first averaged the normalized return (refer to "avg. norm.") under each task, and we can see that the policy obtained by GALILEO is significantly higher than other models (the improvements are 24% to 161%). But in HalfCheetah, IPW works slightly better. However, compared with MAX-RETURN, it can be found that all methods fail to derive reasonable policies because their policies' performances are far away from the optimal policy. By further checking the trajectories, we found that all the learned policies just keep the cheetah standing in the same place or even going backward. This phenomenon is also similar the results in MOPO [59]. In MOPO's s experiment in the medium datasets, the truncated-rollout horizon used in Walker and Hopper for policy training is set to 5, while HalfCheetah has to be set the minimal value: 1. These phenomena indicate that HalfCheetah may still have unknown problems, resulting in the generalization bottleneck of the models.
Online pecision-making in Production Environment Finally, we search for the optimal policy via model-predict control (MPC) using cross-entrops planner [21] based on the learned model and deploy the policy in real-world platform. The results of A/B test in City A is shown in Fig. 7(c). It can be seen that after the day of the A/B test, the treatment group (deploying our policy) significant improve the five-minute order-taken rate than the baseline policy (the same as the behavior policy). In summary, the policy improves the supply from 0.14 to 1.63 percentage points to the behavior policies in the 6 cities. The details of these results are in Appx. H.7
6 Discussion and Future Work
In this work, we propose AWRM which handles the generalization challenges of the counterfactual environment model learning. By theoretical modeling, we give tractable solution to handle AWRM and propose GALILEO. GALILEO is verified in synthetic environments, complex robot control tasks, and real-world platform, and shows great generalization ability on counterfactual queries.
Giving correct answers to counterfactual queries important for policy learning. We hope the work can inspire researchers to develop more powerful tools for counterfactual environment model learning. The current limitation lies in: There are several simplifications in the theoretical modeling process (further discussion is in Appx. B), which can be modeled more elaborately. Besides, experiments on MuJoCo indicate that these tasks are still challenging to give correct predictions on counterfactual data. These should also be further investigated in future work.
10

---

Acknowledgements and Disclosure of Funding
This work is supported by the National Key Research and Development Program of China (2000AAA0102200) and the National Science Foundation of China (619 (61921006).
References
[1] A. M. Alaa and M. van der Schaar. Limits of estimating heterogeneous treatment effects: Guidelines for practical algorithm design. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 129-138, Stockholm, Sweden, 2018.
[2] S. M. Ali and S. D. Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Stafistical Society: Series B (Methodological), 28(1):131-142, 1966.
[3] S. Assaad, S. Zeng, C. Tao, S. Datta, N. Mehta, R. Henao, F. Li, and L. Carin. Counterfactual representation learning with balancing weights. In The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1972-1980, Virtual Event, 2021.
[4] Ben-David, Blitzer, K. Crammer, A. Kulesza, Pereira, and J. W. Vaughan. theory of learning from different domains. Mach. Learn., 79(1-2):151-175, 2010.
[5] S. Benn-aavid, J. Blitzer, K. Crammer, and F. Pereira. Analysis of representations for domain adaptation. In Advances Neural Information Processing Systems 19, pages 137-144, British Columbia, Canada, 2006. MIT Press.
[6] A. Betlei, E. Diemert, and M. Amini. Treatment targeting by AUUC maximization with generalization guarantees. CoRR, abs/2012.09897, 2020.
[7] Bica, Jordon, and M. van der Schaar. Estimating the effects of continuous-values interven- tions using generative adversarial networks. In Advances in Neural Information Processing Systems 33, virtual event, 2020.
[8] L. Buesing, T. Weber, Y. Zwols, N. Heess, S. Racanière, A. Guez, and J. Lespiau. Woulda, coulda, shoulda: Counterfactuually-gue Learning Representations, New Orleans, LA, 2019.
policy search. In 7th International Conference
[9] J. Byrd and Z. C. Lipton. What the effect of importance weighting in deep learning? In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 872 881, Long Beach, CA, 2019.
[10] X. Chen, B. He, Y. Yu, Q. Li, Z. T. Qin, W. Shang, J. Ye, and C. Ma. Sim2rec: simulator-bast decision-make making approach to optimize real-world long-tern tergtterm&nn user engagement in sequential recommender systems. In 39th IEEE International Conference on Data Engineering, pages 3389-3402, Anaheim, CA, 2023. IEBE.
[11] X. Chen, S. Li. H. Li, S. Jiang. Y. Qi, and L. Song. Generative adversarial user model for reinforcement learning based recommendation system. In Proceedings of the 36th International Conference on Machine Learning, pages 1052-1061, Long Beach, CA, 2019.
[12] X. Chen, Y. Yu, Q. Li. F. Luo, Z. T. Qin. W. Shang, and J. Yc. Offline model-based adaptable policy learning. In M. Ranzato, A. Beygelzimer, N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34, pages 8432 8443, virtual, 2021.
[13) X.-H. Chen, F.-M. Luo, Y. Yu, Q. Li, 7. Qin, W. Shang, and J. Ye. Offline model-based adaptable policy learning for decision-making in out-of-support regions. IEEE Transactions on Pattern Analysis Machine Intelligence, (01):1-16, 5555.
[14] X.-H. Chen, Y. Yu, Q. Li, F.M. Luo, Z. T. Qin, S. Wenjie, and J. Ye. Offline model-based adaptable policy learning. Advances in Neural Information Processing Systems 34, Virtual Conference, 2021.
11

---

[15] C. Cortes, Y. Mansour, and M. Mohri. Learning bounds for importance weighting. In Advances in Neural Information Processing Systems 23, pages 442-450, British Columbia, Canada, 2010. Curran Associates, Inc.
[16] B. Eysenbach, A. Khazatsky, S. Levine, and R. Salakhutdinov. Mismatched no more: Joint model-policy optimization for model-based RI.. CoRR, abs/21 10.02758. 2021.
[17] Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
[18] J. Fu, M. Norouzi, O. Nachum, G. Tucker, Z. Wang, A. Novikov, M. Yang, M. R. Zhang, Y. Chen, A. Kumar, C. Paduraru, S. Levine, and T. Paine. Benchmarks for deep off-policy evaluation. In 9th International Conference on Learning Representations. Virtual Event, Austria, 2021.
[19] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27, pages 26800, Montréal, Canada, 2014.
[20] Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft Actor-Critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning, pages 1856-1865, Stockholmsmässan, Sweden, 2018.
[21] D. Hafner, T. P. Lillicrap. I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2555-2565 Long Beach, CA, 2019.
[22] N. Hassanpour and R. Greiner. Counterfactual regression with importance sampling weights. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 5880-5887, Macao, China, 2019.
[23] Hiriart-Urruty and C. Lemaréchal. Fundamentals of Convex Analysis. 2001.
[24] T. Hishinuma and K. Senda. Weighted model estimation for offline model-based reinforcement learning. Advances in Neural Information Processing Systems 34, pages 17789-17800, virtual, 2021.
[25] J. Ho and Ermon. Generative adversarial imitation learning. In Advances in Neural Informa- tion Processing Systems 29, pages 4565-4573 Barcelona, Spain, 2016.
[26] G. Imbens. The role of the propensity score in estimating dose-response functions. Econometrics eJournal, 1999.
[27] E. L. Tonides. Truncated importance sampling. Journal of Computational and Graphical Statistics, 17(2):295-311, 2008.
[28] F. D. Johansson, N. Kallus, U. Shalit, and D. Sontag. Learning weighted representations for generalization across designs. arXiv preprini wr.Xiv:802..85555 2018.
[29] Y. Jung, J. Tian, and E. Bareinboim. Learning causal effects via weighted empirical risk minimization. In Advances in Neural Information Processing Systems 33, Virtual Event, 2020.
[30] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. MOReL Model-based offline reinforcement learning. CoRR, abs/2005. 0tts/2005.05551,.0 2020.
[31] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643 2020.
[32] Q Liu, I.. Li, Z. Tang, and D. Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. Advances in Neural Information Processing Systems 31, pages 5361-5371, Montréal, Canada, 2018.
12

---

[33] Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill. Provably good batch off-policy reinforcement learning without great exploration. In Advances Neural Information Processing Systems 33, virtual event, 2020.
[34] Lu, P.J. Ball, J. Parker-Holder, M. A. Osborne, and S. J. Roberts. Revisiting design choices in offline model based reinforcement learning. In The 10th International Conference on Learning Representations, Virtual Event, 2022.
[35] O. Nachum, Y. Chow, B. Dai, and L. Li. Dualdice: aehavior-agnostic estimation discounted stationary distribution corrections. In Advances in Neural Information Processing Systems 32, pages 2315-2325, BC, Canada, 2019.
[36] X. Nguyen, M. J. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory, 56(1 1):58477--8, 2010.
[37] S. Nowozin, B. Cseke, and Tomioka. f-GAN: Training generative neural samplers using
variational divergence minimization. In Advances in Neural Information Processing Systems 29, pages 271-279, Barcelona, Spain, 2016.
[38] J. Pearl. Causality. Cambridge University Press, 2009.
[39] S. Pitis, E. Creager, and A. Garg. Counterfactual data augmentation using locally factored dynamics. In Advances in Neural Information Processing Systems 33, virtual, 2020.
[40] D. Pomerleau, Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88 97,888899711777 1991.
[41] T. Qin, T. Wang, and Z. Zhou Budgeted heterogeneous treatment effect estimation. In Proceedings of the 38th International Conference on Machine Learning, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8693-8702, 2021.
[42] J. Citionnero-Cadell. M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. Dataset shift in machine learning. Mit Press, 2008
[43] P. R. Rosenbaum and D. B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70:41-55, 1983.
[44] J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1889-1897, Lille, France, 2015.
[45] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1 707.06377 2017.
[46] W. Shang, Y. Yu, Q. Li, 7. Qin, Y. Meng, and J. Ye. Environment reconstruction with hidden confounders for reinforcement learning based recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 566-576. Anchorage, AK, 2019.
[47] J. Shi, Y. Yu, Q. Da, S. Chen, and A. Zeng. Virtual-taobao: Virtualizing real-world online retail environment for reinforcement learning. The 33th AAAI Conference on Artificial Intelligence, pages 9902-4909, Honolulu, Hawaii, 2019.
[48] H. Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
[49] S. A. Sontakke, A. Mchrjou, L. Itti, and B. Schölkopf. Causal curiosity: RL agents discovering self-supervised experiments for causal representation learning. In Proceedings of the 38th Inter- national Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9848-9888 9858, Virtual Event, 2021.
[50] P. Spirtes. Introduction to causal inference. J. Mach. Learn. Res., 11:1643-1662. aug 2010.
13

---

[51] R. S. Sutton and A. G. Barto. Reinforcement learning an introduction. Adaptive computation and machine learning. MIT Press. 1998.
[52] A. Swaminathan and T. Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. J. Mach. Learn, Res., 16:1731-1755, 2015.
[53] E. Todorov, T. Erez, and Y. Tassa, MuJoCo: A physics engine for model-based control. In Proceedings of the 24th IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026 5026-5033, Vilamoura, Portugal, 2012.
[54] C. Voloshin, N. Jiang, and Y. Yue. Minimax model learning. In A. Banerjee and K. Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1612-1620, Virtual Event, 2021.
[55] Wang, T. Qin, and 7.. Zhou. Estimating possible causal effects with latent variables via adjustment. In International Conference Machine Learning, (CM1.202,) 23-29 July 2023, Honolulu, Hawaïï, USA, volume 202, pages 66308-36333 2023.
[56] T.ZZ. Wang, X.-Z. Wu, S.-J. Huang, and Z.-H. Zhou. Cost-effectively identifying causal effects when only response variable is observable. In Proceedings of the 37th International Conference on Machine Learning, pages 10060-10069, 2020.
[57] S. Yang, S. Zhang, Y. Feng, and M. Zhou. A unified framework for alternating offline model training and policy learning. CoRR, ftss/2210.0592/ 2022.
[58] Yoon, J. Jordon, and M. van der Schaar. GANITE: estimation of individualized treatment effects using generative adversarial nets. In 6th International Conference on Learning Repre- sentations. 2018
[59] Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. MOPO: model- based offline policy optimization. In Advances in Neural Information Processing Systems 33, virtual event, 2020.
[60] R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. In 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
[61] W. Y. Zou, S. Du, J. Lee, and J. O. Pedersen. Heterogeneous causal learning for effectiveness optimization in user marketing. CoRR, hass20044099700 2020.
14

---

Appendix
Table of Contents
A Proof of Theoretical Results
16
A.1 Proof of Lemma A.I
18
A.2 Proof of Eq. (9)
19
A.3 Proof Thm.
20
A.4 Proof the Tractable Solution
23
B Discussion and Limitations of the Theoretical Results
25
C Societal Impact
25
D AoRM-oracle Pscudocode
25
E Implementation
25
E.1 Details of the GALILEO Implementation
25
E.2 Connection with Previous Adversarial Algorithms
28
F Additional Related Work
28
G Experiment Details
30
G.1 Scttings
30
Baseline Algorithms
................................
33
G.3 Hyper-parameters
................................
33
G.4 Computation Resources
33
H Additional Results
34
H.1 Test in Single-step Environments
34
H.2 All of the Result Table
35
H.3 Ablation Studies
................................
35
H.4 Worst-Case Prediction Error
35
H.5 Detailed Results in the MuloCo Tasks
36
H.6 Off-policy Evaluation (OPE) in the MuloCo Tasks
37
H17
Detailed Results in the BAT Task
42
15

---

A Proof of Theoretical Results
In the proof section, we replace the notation of E with an integral for brevity. Now we rewrite the original objective I(phs,,M M) as:
min max / (1,-(1, (2, ())((, / (7 (- log Ma'22, ()) MEM BEll X,
a dz' dade - 2 Ill (s.))
where a) =
P (x)a(xa))
Px- (2.a) and 1PM-((:))1 = Sx.APM- (ar, (2, dada, which is the squared
12-norm. an MDP, given any policy n, PM- (2r, a.), = (x)) M" a) where (x) denotes the occupancy measure of for policy т, which can be defined [51, 25] as Pīr-(a) := (1 - -x) - y Prr((z; = r| Alzo,,M")] where Pr" [x, = x r/z0, MM] is the state visitation probability that starts at state Zg in model M and receive r at timestep and y E [0, is the discount factor.
minMeM maxagn Lip M)
estimate optimal adversarial distribution 3" given M (Sec, A.1)
(model easy-to estimate distribution of best-response policy (*)
intermediary policy x
generator function f
derive Pir- as generalized rep- 3" resentation of Pie. (Sec. A.2)
let Pi as the distribution of best rest-response policy arg maxaen L(Prr. M). we have the rogate objective minare. LiP M) (Sec. A.3)
approximate P with variational representation
tractable solution (Sec. A.4)
The overall pipeline to model the tractable solution AWRM. f is generator function defined by f-divergence [37]. is an intermediary policy introduced in the estimation.
Figure 8:
The overall pipeline to model the tractable solution to AWRM is given in Fig. 8. In the following, will summarize the modelling process based on Fig. 8. We first approximate the optimal distribution
3"
P.. via Lemma. A.I.
Lemma A.1. Given any M in D(p the distribution Pxr (x, a) of the ideal best-response
policy 8 satisfies:
1
M M-12,,)) H (x,a)),
(8)
am
where (M a), M(12x, a)) is the Kullback-Leibler (KL) divergence between (-Ir, lr. a) and M(-1,, a). HM- (z, a) denotes the entropy of M M (-12, a), and ass is the regularization coefficient a in Eq. (4) and also as normalizer of Eq. (8).
Note that the ideal best-response policy 3" is not the real best-response policy 8". The distribution A is approximation of the real optimal adversarial distribution. We give discussion of the rationality of the ideal best-response policy 3 as a replacement of the real best-response policy 3* in Remark A.4. Intuitively, A has larger densities on the data where the divergence between the approximation model and the real model (i.e., (KI(M* (:12, a), M (-1z,a)) is larger or the stochasticity of the real model (i.c., HM.) ) is larger.
However, the integral process of DKL in Eq. (8) is intractable in the offline setting as it explicitly requires the conditional probability function of M Our solution to solve the problem is utilizing the offline dataset Dreal as the empirical joint distribution p (z, x') and adopting practical
16

---

techniques for distance estimation on two joint distributions, like GAN [19, 37], to approximate
Eq. (8). To adopt that solution, should first transform Eq. (8) into form under joint distributions. Without loss of generality. we introduce an intermediary policy K, which μ can be regarded as specific instance. Then we have M (a' [x, a) - Lir (a, a, x x²) (I, a) for any M if pir (a, a) > 0.
Assuming Va X, Va E A. Pir-- (a,, a) 0f PM- (2, O, which will hold when overlaps with 14, then Eq. (8) can transform to:
a0(2;a) (Sx pis- (z, a, log (x, dr' (2s-(2,,) (log (2)-(2, Ha (1, (o.2)) ()) Pi (2,
1
pir(z,a)
where an(az, a) = aMPii- (z, a). We notice that the form Pi-- log fice the integrated function in reverse KI. divergence, which is an instance of function in f-divergence [2]. Replacing that form with / function, we obtain a generalized representation of pxr.:
PKe
FM-- = (o(zz. a) (Je phr ( (z,,, (fi)(,,,xx)) ) dx' pir- (2, (1 Pir (I, (x,,,
pirle,a) Pir (),
Har (7, a) (24,0)
(9)
where 1R - IR is convex and lower semi-continuous (l.s.c.) function. PMr gives generalized representation of the optimal adversarial distribution to maximize the error of the model. Based on Eq. (9), we have a surrogate objective of AWRM which can avoid querying M to construct PMa-:
Theorem A.2. Let PMM- as the data distribution of the best-responss policy 3 in Eq. (4) under model Me parameterized by 6, then we can find the optimal 8 of ming maxsen L(p Mo) (Eq. (4)) via iteratively optimizing the objective Bt41 ming [(Pas,, Mo). where Pag- is approximated via the last-iteration model Mg.- Based on Corollary A.9, we derive an upper bound objective for ming T( Ma):
(1( (iiso, (2,, (0,,azz)) Pir- (Ir, a,,e))
Pise, (2r,a)
Oc+1- = min Byg.
-1 ao(
log Mo(x'(z,,)
+ H (z,
Phre (x)
W(aaaa")
where Epgr [ denotes Ex,a,'> 1. f is a I.s.c function satisfying f'(x) S 0, Vx E X, and ao( a) aMM PMMoPPss- (x, a).
Thm. A.2 approximately achieve AWRM by using x. and pseudo-reweighting module W. W assigns learning propensities for data points with larger differences between distributions Piic, and pig- By adjusting the weights, the learning process will exploit subtle errors in any data point, whatever how many proportions it contributes, correct potential generalization errors counterfactual data. Remark A.3. In practice, need to use weal-world data construct the distribution Pir.. In the offline model-learning setting, we only have a real-world dataset Dreat collected by the behavior policy μ. which the empirical distribution of Phr.. Let K = p. we have
a)( log Mo (a'(a,,, (e,,) (r Piito (e,
Piss, (2),a)
0.41 = min E
P (x,
phis (2,
+ H (1,
W(x.a.x")
which is Eq. (5) in the main body.
In Thm. A.2, the terms fljS,, (z, a, x,,,,x')/P5i-- (x...)) f(pis,, (w,a)//ss- (2, are still in- tractable. Thanks to previous successful practices GAN [19] and GAIL [25], we achieve the objective via a generator-dee dirrrttorddisiiiitttoo paraddigg objective through similar derivation. We show the results as follows and leave the complete derivation in Appx. A.4. In particular, by introducing two discriminators Digg (z, a') and Deg (x. a), letting 14, we can optimize the surrogate objective
Eq. (5) via:
0ct1 max (B (Ara (z, a. ) log Mo (2, a)] + E [(HM (2, Avoiet (2, (, log Mo(2'12,00) (BoA,,
s.t. pp arg max iPo
EE,, [log Dep (2,, a, x')] + Sin,, [log(1 - Deo a, xt (x)))
arg max (B [log Dep (z, a)] + E E [log(1 Des (x))
17

---

where Bpg, [ is a simplification of - Fl. Apg.pt a2)), log Dug (z) a, log Der a), and yPo and 9i are the parameters of pu and Di, respectively. We learn pol- icy f R μ via imitation learning based on the offline dataset Dreal [40, 25]. Note that the process, ignore the term (u a) for simplifying the objective. The discussion the impacts of removing (x a) is left App. B.
A.1 Proof of Lemma A.1
Proof.
Given a transition function M of an MDP, the distribution of the best-response policy 8
satisfies:
P (z,)(())) /
PM
M('l a)) dr' dadr - 2 |l Mr (3.-)11
arg max
Pa
X.A
2
arg max Px
(- log M (2d a)) da' dade
2 11 (, (1.)112
X.A
a(a.a)
2 I
(1,()))))(2, 11 (1 (:,
arg max Prg- a x
2
/
Phr (2, a)ddxr 11pM-((: (:.))11
1l9(..))11
arg max Pig a x
a²
(-2
g(2, a) Par. (Ir a)
dadx + -1pp (1.)112 119(
11
arg max- Pige
X.A
a
a²
g(:,))
arg max - 11 (...).(..)) (:..)
We know that the occupancy measure Pas- is density function with constraint 3 Sx SA Pag. (x, adddr 1. Assuming the occupancy measure Px- has an upper bound c, that is 0 < Par. (2, I c. A.Vo X, constructing a regularization coefficient аı Sx [MDKi(M a), u() + Has- (x, a)).dda as a constant value given any M, then
have
g(z.a)
(I, PM-
8" PM-
a'
Ix (x' a) log (r dx fx M (x' (2, Ma (a'1x, a)dr M(F
OM
(1,, M(1x,, a)) + H (x,
OM
(DKI (M2,,) M(,2z, a)) (x,
which is the optimal density function of Eq. (7) with a = αM-
Note that in some particular M we still cannot construct 8 that can generate an occupancy specified by g(a, aa/a for any M. We can only claim the distribution of the ideal best-response policy 3 satisfies:
8" (z.a) -DKL M(1z, + H (x,
(10)
O.M
where am is a normalizer that aM fx JA (MM (- a)) + H (z, dx)da.. We give discussion of the rationality of the ideal best-response policy 3 as a replacement of the real best-response policy 3 in Remark A.4.
18

---

Remark A.4. The optimal solution Eq. (10) relies on g(x a). In some particular M it is intractable to derive 3 that can generate an occupancy specified by g(z, a)/an. Consider the following case: a state 21 in M might be harder to reach than another state 22, e.g.. (* < (x2lx,)) a), Ve E X,
A, then it is impossible to find that the oceu- pancy satisfies P (21, a) > Pay- (I2,a) In this case, Eq. (10) can be a sub-optimal solution. Since this work focuses on lask-agnostic solution derivation while the solution the above problem should rely on the specific description of M*, we leave as future work. However, we point out that Eq. (10) is reasonable re-weighting term even sub-optimum: Phr- gives larger densities the data where the distribution distance between the approximation model and the real model (i.e.,
D (M", M)) is larger or the stochasticity of the real model (i.e., HM-) is larger.
A.2 Proof of Eq. (9)
The integral process of DEL in Eq. (8) is intractable in the offline setting it explicitly requires the conditional probability function of M Our motivation for the tractable solution is utilizing the offline dataset Dreal as the empirical joint distribution P (z, a,)') and adopting practical techniques for distance estimation on two joint distributions, like GAN [19, 37], to approximate Eq. (8). To adopt that solution, should first transform Eq. (8) into form under joint distributions. Without loss of generality, we introduce an intermediary policy K. of which μ can be regarded as a specific instance. Then have M (x' [ar, a) = Piis (2,, a, x²)/ (1, a) for any M if P%s (2r a) > O.
Assuming Va E X, Va E A. Pir- (x, a) >> 0f PM (x, a) > O, which will hold when x overlaps with μ. then Eq. (8) can transform to:
Par- (1, (x) a) DRI(M a), (1112,))) M( + (M
aM
OM / Mx(x'12,,)) (108 (1
1
M(x (x log M M (2'1(zzz,)) de
aMpis- (x, / (z'((x)) log (1(ll1z,0)
log M M (2'1 dæ'
M(x' a,
(11)
aMpis-(X, [&. PSr (2, (log Pār (I,), + log Pir( log M (e' a) dx
Pir ux,)
Mis (x,
aMpir (x, (Je (ir- (2, log Pig- 2') do'
(I,
Pis. (x, a) log (2)
M (1'222 adde' -his (2, a) / M (1'x'1 log M (I' a)da'
(2,
x
an (00 a) (1 Pir (x, (i,-(1,a,x") dx' pir (x, (z, a) (I,
1
B3 a) + pār (z, a) Har-(z,a))
03 (1,
(12)
where o0(2, a) amppss (Ir a).
Definition A.5 (f-divergence). Given two distributions P and Q, two absolutely continuous density functions and with respect to a base measure do defined on the domain X, we define the f-divergence [37],
p(x) ((x))
D₂(PIIQ) = {(x))
dx,
(13)
where the generator function :. - IR is convex, lower-semicontinuouu function.
We notice that the terms Pig- (z, a, x') log (1,ax)) and pūr- (x, a) log (i. (z.a)
Mi (2.a) are the integrated
Pia Pia (2,0,"))
functions in reverse KL divergence, which is instance of function in f-divergence (See Reverse-
19

---

KL divergence of IaT.. in [37] for more details). Replacing that form q log 2 with of()), we obtain p
8"
generalized representation of Paa-:
1 PM = vor a) (L par (m,, (,,( de' pire (2, (1(2 ( Pir. (2, - Har (ax, a)
Pir (I, ax,
a,)")
(14)
A.3 Proof of Thm. A.2
We first introduce several useful lemmas for the proof.
Lemma A.6. Rearrangement inequality The rearrangement inequality states that, for two sequences ai 02 ... 2 On and bi 2 b2 ... bu. the inequalities
abb + a2b2 +.... an 2 a1bx(1) + a2bx(2) + ....t +..+ an 2 arbn + a2zn-1 + an b1 hold, where T1), π (2)..... (n) is any permutation of 2........
Lemma A.7. For two sequences a1 2 2 2 an and bj 2 b2 2 2 bn. the inequalities oabi
bj
11
1=1
hold.
Proof. By rearrangement inequality, we have
a,b; 2 aibi + a2b2 + an
Σ a;b; 2 a1b2 + aabs + an
Σ a,b; 2 bibs + azbs an
Σ a;b; 2 azb, + tabi + +.. tanb-
Then we have
1
n
71
0:
b
1
a,br
b
n
1=1
Now we extend Lemma A.7 into the continuous integral scenario:
Lemma A.8. Given X C R. for two functions f J - R and g X - TR. that f(x) 2 f(y) if and only if (x)) 2 g(y), Vx, y X, the inequality
[ p(z) (())()))
2 f p(x)f(x)dx [ p(z)g(z)de
holds, where p: X + R and p(x) O, Vx E X and fx p(x)dx= 1
Proof. Since (f(x) f())((())( - ((v)) > Vax E X. we have
20

---

J p(s)p())(((a)) (()))(() s(v))dyde 2 VEX
J. vEX + http:///wwwwowoootteGeeeeaiiiaa -Wteo/))iweeppooow///////a/aaaa 20
JECK I p(o)p(u))())a() + +ttcc)t JeeX DEX (ttpa///eooooooooo///oo///////ad
SOX
[ ZEN (1 ple)p(ul plelp(ll// dr
= DEN JEX
I FEX
(lw) ( /
de / DEN
VEX
S Po0000/ccc000+000000000000666660 TEN DEX
p(()\())((()de + plesppw 2
xEN
ZEN VEN
p(t)s(t)g())de / p(N)s(()g())dd OEX vEX
xEx
DEN
p(t)s())g()dd 22
VEX ZEN
2 p(()\(()g(()d>>> 22 pla)S(a)de / p(z)g(a)dr
ZEX
SEX
FEX
(())s [ p(z)g(o)dr
p(o)((()g(()de>
ZEN
DEX
Corollary A.9. Let g(
p(z)
= - -log p(a)
where p(x) > 0,V E X' and g(x) > 0, Vo E for
g(z)
g(z)
> O, the inequality
Sx g())s((
p(x)
p(z)
dz / ((x))(()
p(x) ((x) dz [ g(z)g( p(x) g(x)
)g(
dd,,
((x)
g(x))
holds if (xx) S O, V. X. It is not always satisfied for J. functions of I-divergence, We list comparison of I on that condition in Tab. 3.
Proof. g' (zxx = - if and only if [g(x)= 2 ,ly), Va,y y E X holds. Thus ((2((a) 2 f(u plu) if and only if g( p(z) q())
< 0, Vo E X. Suppose f((x) S 0, Vo X, we have f(x) 2 f(v)
d(v)
g(z))
p(u) g( (()) Va, y E X holds for e > By defining F(x) = f(u pla ) and G(x) = g (1))) and
(())
using Lemma A.8, we have:
[ (() 2 / ((x))(x)drr / ((x))(x)dx.
Then we know
p(x) f ((x)(((
((x) )g( p(x) )d 2 / g(x)s(( p(x)
/ g(x)g(
p(z) dr
dr
((x))
((x) fx
g(x)
holds.
Now, we prove Thm. A.2. For better readability, we first rewrite Thm. A.2 as follows:
Theorem A.10. Let p as the data distribution of the best- respones policy 3 Eq. (4) under model Me parameterized by 0, then we can find the optimal 0* of ming max L(p Ma) (Eq. (4)) iteratively optimizing the objective Oc+1 = ming Tp Mo), where 2ht is approximated via the last-iteration model Me.. Based Corollary A.9, we have upper bound objective for
21

---

Table 3: Properties of (x) 0, vr X for f-divergences.
Name
Generator function f(x)
If fx)) s 0, X
Kellback-Leibles Reverse KL
x log log r (xr 1)2
False
True
Pearson x2
False
Squared Hellinger
(Vr 1)2
False
Lensen-Shannon
- (x + 1) log L+z + x: log
False
GAN
x logo x (x + 1) log(æ + +1)
True
ming L(PM- Ms) and derive the following objective
log Mo(x'I x a) (s Pieo, (I, (I, Pis (x,, (2),
Piiee (x,
0.41 arg argmaxEpr Eir-
+ Has (I,
c0(2,a) a)
Pis (2t, a)
where (0( O.Mo, Pig (I,,a), a). Epi.. []] denotes a' [. f is the generator function in I-divergence which satisfies ('(x) 0, Vx X, and O is the parameters of M. Ma, denotes probability function with the same parameters as the learned model (i.e. = 0) but the parameter is fixed and only use sampling.
Proof. Let po-- as the data distribution of the best-response policy 3 in Eq. (4) under model Ma parameterized by 0, then we can find the optimal l.+1 ming maxsen T(p - Mo) (Eq. (4)) via iteratively optimizing the objective 0.+1 ming L( Mo), where par is approximated via the i model Mo.:
last-iteration
Dct1 min / (x.a) / M (='laa (- log Mo (x' (,,,,,,,,,, a)) dæ' da'd
(15)
Piso, (ce, (x))
da' I
min SxL a)(
log Ma(a' a))dr'
Pir- (2, (2r a.)
- ( (s
Piiro (1)
H HMr(((r (e) I A (o'le, log Ma a))dd' jadd
Pi (I),a)
= min Jx a) (f (
Pisoo (1e, a,
de' (fr (far (o log M x a)
Mo(a' dd + Har (ve,
plis- (z, a, a,
(2, (r
Piira (x)
J A (a'((' a) log Ma(w' Ir, a))dd' jaad Pisa (x,, (,, Piiee (ar, a,
Har (2e,
Pir- (2,,
Pšs (2, S M (x
Mo(a' xc, a)
< min x.A a)
(- log
da
M- (,)l|xr,
based Corollary A.9
Pii-((1,a)) (s
Pisse, (2)
S M' (x' a), log Mo(a')s,, dade
- (1r (x,
Pir- (x),a)
min L x.A au)e,a) ((, ( (x'laa))
Piieo (x,, a, p (x, a,,
(- log Mo(a' Molor los (e,.)))) dx'
Piste (2)
[ M (a' (2'12,, ()( log Mo(w' Ma a))dz' Jaad )
- (2,))
- H (1r-(2e,)) (z, (z)
Pige (z),a)
max
Ix (olxx, Pār- a. log Mo(e' fir- (
J
(5(
(Biten (2r, (o,,a
Pir- (2,
(Pste, (2,4)
Pig- (2e, ) + H Har- (2. dx' dadæ,
8
(16)
22

---

2 and fixed when optimizing 0. In Eq. (15), where Me. is introduced to approximate the term
1lea-(( (., -)2 for Eq. (7) is eliminated as does not contribute to the gradient of Assume f'(x) S O, Vz E X, let v(z,a) a) == Plz- (v.a) > 0, p(='|z,,) a) Me (a' (x, and q(x' [z, a) (x' [a, a). the first inequality can be derived by adopting Corollary A.9 and eliminating the first Has since does not contribute to the gradient of A.
Pāzo, (2,a)
A.4 Proof of the Tractable Solution
Now we are ready to prove the tractable solution:
Proof. The core challenge is that the term S( Pāroo (z,a,a")
Pāra, (2,a)
Piee (0.0,z") S(
Pies (æ.a) is still intractable. In the
following, we give a tractable solution to Thm. A.2. First, we resort to the first-order approximation.
Given some (1 E, 1 E),E O, we have
f(u) f(1)+ (()(a-)
(17)
where /' is the first-order derivative of f. By Taylor's formula and the fact that f'(u) of the generator
function / bounded in (1 E, 1 + 5). the approximation error is no more than (g22). Substituting
with p(r) in Eq. (17), the pattern f( f(p(e)) in Eq. (16) can be converted to p(x) f ((x)) ((())
f((p(e)
{(2)
v(z)
f(1), then we have:
0.+1 arg max 1
(iten (z, (,,)) (2, a) Xna (arr( J M (e(ellr
SaLa
Pir- (2, (e, x.)
log Mo |s, a)da'
Pisnn (5,00)
I M (a' (x) log Mole a)dr'+
pis (e))))))))
M (e Pis (2)
log Mo(a' (aaaalix,, a)dx dadz
Pido, (2) I (o
Piso, (I, a,,)
arg max
(2,
log Mo(a' la, a)dx'-
X.A
(2n.(2,0) [ M (x'l(()
Plite (ce, (er,, (is-(2,
(2,
- s(a) log Mo(a' (1, a)dx'
[
Di,, (2,
. M a) log Mola' (2,
Pir- (x,
Pirta (I,
-5(1) )
/ M (or' (z, log Mo la, a)dx'+
(2,0 f
pire (2r (2)
(2, (x, (2, f
(a' (x, a) log Mo Mo(e'lr a)dee" dadæ
x
Phite, a,,,)
Piite (I,a)
)) log Mo |s, a)dz'ddd::
arg max
N.A.X wo( Pido (x)
s'
Pir-- (Ir, a,
Pir (2,
S a) (x Pãs
(I)),
Pisp, (x,
Piite (x), a 20)
HM (1, log Mo (x, a)dz' a)dx'doda. dodæ.
Phie (x,
Pir- (x))
Note that the part pfr-- (I, a) in phi-- (I, a, x') can be canceled because of ag ao a) = a., Pfr-- (x,a). a). but we choose keep and ignore on an(2z, (z, a). The benefit is that we can estimate ph. (I. a, x²) from an empirical data distribution through data collected by x in M directly, rather than from a uniform distribution which is harder to be generated. Although keeping (Z,-(x, a) incurs extra bias in theory. the results in our experiments show that it has not made significant negative effects in practice. We
23

---

leave this part of modeling in future work. In particular, by ignoring (z a), we have:
0.+1 = arg max / xxAxx piro, (2, (r
(fiir (fito, (1, (.)..)))) Pīre (z,, a, a,
Pire, (4,0)) ))
log Mo a)dz' dadæ+
Pii- (2,
(18)
Ix.Ad Pis Pis- (25-(2 (p
Pire, (1r, Plie (x)
Pitn (or, a,)))
f
(zr-(z, log Mo (x' a a)dr dadæ.
PSr- (2,
(19)
(isse (2.0) Para [a,a]
Pito, (=,a,)) Pige (2.0,,")
We can estimate /'
and M'
through Lemma A.11.
Lemma A.11 (f estimation [36]) Given a function T.p :x + R parameterized by ip E D i is convex and lower remi-continuous, by finding the maximum point of ip in the following objective:
4" = arg max Enmp(() [Tp(x)] -Bp=q(s) (I)()
we have ('( p(x) = (x)) I is Fenchel conjugate of [23].
In partieular,
G arg max Ez,a,t' [T o (I, a, a')) - Exgant (T('
0 arg max Ex (x, a)] - Exzampiro, (TT (2,
IP1
Pira, (2,0,a")
Pirn (x)a) Pire (x,a)
then we have 1'
Pia- (2.,,
Tap (2), a, and S
Tep a). Given sg and
1 (I, a,, ii. let Apgret
Top (2, Tep (I, a), then we can optimize 0 via:
0.+1 arg max
Piroo (x, (Tea (z. a, - Typ a)) log Ma(a'Ia (1,11,, a)da'd dadx+
X.A,X
J. XxA.X (i5 a, (To a) Two (z, a, x') + H (I, a)) log Mo Ma(w'la d)da' dadæ
/ Piito (2 log Mo le' lx, addr'dadx+ dadx+
arg max
XAX
J. (I, 1 + Has (x, a)) log Me(a' (x, a)dr'dadr: dadr.
Based on the specific f-divergence, we can represent T and (T) with a discriminator D.-. It can be verified that f(u) log (u 1) log(u 1 + 1). Tup (u) log (u),), and f (T(pp( log(1 - (u)) proposed in [37] satisfies the condition f'(x) < 0, Vr E A' (see Tab. 3). We select the former in the implementation and convert the tractable solution to:
0.41 arg max Byite [A (z, a. x') log Me(x' [z, a)] + Eppr- [[H (2, Apg (2, 0, log Ma(e'Ix, s.t.
p arg max E- [log Do (z, x') + Epito [log(1 D (x),,
pi arg max Bpp.- [log Day (z, a)] Epiro [log(1 Day (x)
(20)
where Appiot (z, (.)) log Dyg (x, - log Doy (2, Epi.p a simplification of Ex,,, x' Spstn H..
Remark A.12. In practice, we need to use the real-world data to construct the distribution Par- and the generative data to construct PMr., In the offline model-learning setting, we only have a real-world dataset Dreat collected by the behavior policy We can learn policy fa sJ via imitation learning based on Dreal [40, 25] and let be the policy Then we can regard D, Drest the empirical
24

---

data distribution of ph and the trajectories collected by á in the model Mo, as the empirical data distribution of piMa. Based on the above specializations, we have:
oc+1 max (Enisa,
Ar vr (z, a',) log Ma (a' |r, a)] + E [(Har- App (e, a, x')) )) log Mo(a'|a, [x,a)]]
s.t. pu arg max (Bpz.. [log Dop (ce, a, x'] + E Pire.
[log(1 - D (2, 0, 2x,'1)))
Pi arg max 1P
(E [log Dos [x, Sos, [log(1 (es(tr
which is Eq. (6) in the main body.
B Discussion and Limitations of the Theoretical Results
We summarize the limitations of current thcoretical results and future work as follows:
1. As discussed in Remark A.4, the solution Eq. (10) relies on PMr (z, a) E [0,c], Va E that can generate an
A, Vx E X. In some particular MM,, it is intractable derive occupancy specified by g(z, a)/a If more knowledge of M or 3 is provided or some mild assumptions can be made on the properties of M or 3 we may model p a more sophisticated approach to alleviating the above problem.
2. In the tractable solution derivation, we ignore the term a0( - MMM (ar, a) (See Eq. (19)). The benefit is that Pir-- (It, 0, x²) in the tractable solution can be estimated through offline datasets directly. Although the results in our experiments show that it does not produce significant negative effects in these tasks, ignoring Pi-- (z. a) indeed incurs extra bias in theory. In future work, techniques for estimating P% (x, a) [33] can be incorporated to correct the bias. On the other hand, a'Ma, is also ignored in the process. a.Me, can be regarded global rescaling term of the final objective Eq. (19). Intuitively, constructs an adaptive learning rate for Eq. (19), which increases the step size when the model is better fitted and decreases the step size otherwise. It can be considered to further improve the learning process in future work, e.g., cooperating with empirical risk minimization by balancing the weights of the two objectives through OKMg,
C
Societal Impact
This work studies a method toward counterfactual environment model learning. Reconstructing an accurate environment of the real world will promote the wide adoption of decision decision-making policy optimization methods in real life, enhancing our daily experience. We are aware that vecision-makin policy in some domains like recommendation systems that interact with customers may have risks of causing price discrimination and misleading customers if inappropriately used. A promising way reduce the risk is to introduce fairness into policy optimization and rules to constrain the actions (Also our policy design in Sec. G.1.3). We involved in and advocating research in such directions. We believe that business organizations would like to embrace fair systems that can ultimately bring long-term financial benefits by providing better user experience.
D AWRM-oracle Pseudocode
We list the pseudocode of AWRM-oracle in Alg. 2.
E
Implementation
E.1 Details of the GALILEO Implementation
The approximation of Eq. (17) holds only when p(=)/q(x)) is close to 1, which might not be satisfied. To handle the problem, we inject a standard supervised learning loss
arg max Epg,- [log (a'(2z a)]
(21)
25

---

Algorithm 2 AWRM with Oracle Counterfactual Datasets
Input:
Φ: policy space; N: total iterations
Process:
1: Generate counterfactual datasets {D } for all adversarial policies Tæ, Φ E A
2: Initialize an environment model Ma
3: for i= I:N do
4: Select Dr, with worst prediction errors through Mg from (Da,)
5: Optimize Mp with standard supervised learning based on DT,
6: end for
to replace the second term of the above objective when the output probability of D is far away from 0.5 (f((1) = log 0.5).
In the offline model-learning setting, we only have a real-world dataset D collected by the behavior policy We learn policy ~ via behavior cloning with D [40, 25] and let jú be the policy x. We regard 'D as the empirical data distribution of Pār- and the trajectories collected by jt in the model Ma, as the empirical data distribution of Pli,, But the assumption Væ E X, Va E A, p(alæ) > 0 might not be satisfied. In behavior cloning, we model ja with a Gaussian distribution and constrain the lower bound of the variance with small value tn. 0 to keep the assumption holding. Besides, ~ (N, ED) to the inputs of D&p to handle the mismatch between pia and P due to eqc- In particular, for yoo and (0 learning, we have:
we add small Gaussian noises
00 = arg max Epg... [log Do (x + uz a vay x' + vor)] ] + Episo [log(1 D (x + U2 a ua, + u.))) Ve
ip = arg max Epg. a [log Duy (xr + ue a u.)l Eviro " [log(1 Des (e + ue ua))],
i1
where Epgs, ul-] is a simplification of E ->\(0.) and u [Uz, May Ugy
the other hand, we notice that the first term Eq. (20) is similar to the objective of GAIL. [25] by regarding Ma as the policy to learn and as the environment to generate data. For better capability in sequential environment model learning, here we introduce some practical tricks inspired by GAIL for model learning [47, 46]: we introduce the reward is defined by the discriminator D, i.e., r(x, x²) = log D(x, a, x'). Mg learned to maximize the cumulative rewards. With advanced policy gradient methods [44, 45], the objective is converted to max; [Agg.et (I. a, a.x²) log Mø(a, a, x') where = Qi,, Ma," Q%, (., E E 7'r(21, dt, T1+1) (Iz, at, B++1) (x, a,)x),)),,..) K. Mo,1, and VM, (2, a)
MDP for к and Me. where
EM, Qii, (I. a, x²) A in Eq. (20) can also be constructed similarly. Although it looks unnecessary in theory since the one-step optimal model Ma is the global optimal model in this setting, the technique is helpful in practice as it makes A more sensitive to the compounding effect of one-step prediction errors: we would consider the cumulative effects of prediction errors induced by multi-step transitions in environments. In particular, to consider the cumulative effects of prediction errors induced by multi-step of transitions in environments. we overwrite function Apg.et as A 2hso, VM,, where Ui,, (I. a, z') E [ 7 log (I ae, ( 0e, Zt+1) = (x, a, a'),,, K. Me.] and = E E 7d log Dip (t), at) = (z, a). Mo. To give an algorithm for single- step environment model learning, can just set y in Q and V 0.
VI, (I.
26

---

Algorithm 3 GALILEO pseudocode
Input:
Deaat: offline dataset sampled from par- where y is the behavior policy;
N:
total iterations;
Process:
Approximate behavior policy via behavior cloning
2: Initialize environment model Mp.
3: for / N do
4: Use to generate a dataset Dgen with the model Mo,
5: Update the discriminators Doo and De, via Eq. (25) and Eq. (26) respectively, where Pla
is estimated by D gen and P is estimated by Dreal
6:
Update Q and V via Eq. (23) and Eq. (24) through Dgen. Do and Dep Update the model Ma, via the first term of Eq. (22), which is implemented with standard policy gradient method like TRPO [44] or PPO [45]. Record the policy gradient Spg
7:
8:
if po < ED.... Fo (2t, at Te+1)) )] < P1 then
9:
Compute the gradient of Me. via the second term of Eq. (22) and record it as Gal
10:
else
11:
Compute the gradient of Mo. via Eq. (21) and record as Jat
12:
end if
13: Rescale Is1 via Eq. (27)
14: Update the model Ma, via the gradient Jst and obtain M
15:
end for
Dgen
generate dataset (line
(3) approximate through Eq. update the dynamics model
behavior policy
dynamics model
(line
discriminators and value functions (2.a),
(e, x'), ().a), Q and V
(2) update the discriminators and the value functions (line to
R(rls)
pretrain: aporoximated the behavior policy (line
offiine dataset
Drrot
neural networks
fake dataset
offline dataset
Figure 9: Illustration of the workflow of the GALILEO algorithm.
By adopting the above implementation techniques, we convert the objective into the following formulation
O.t1 = arg max Epito, (2, log Mo(w' a)] Epgr- [[HM- (s, Aug of (u, a, x' )) log (M
(22)
s.t.
Sha, (2,,,z')== =B Σ r't log ae 2e (xt, (,, Mo.
(23)
VMg (x, a)
R
Σ r¹ log Dzy (Zo an) at) (2, Mo.
(24)
( arg max Ep [log D (x + up a uar x' + unr)] Epïco . [log(1 Deoo (x + Mr. + un x u.
40
(25)
i arg max P [log Da, (I. + Uz + ua)) + Eesto a [log(1 Das (x + ux ua))]
(26)
27

---

where Arg (I, a, x') = OM, (z, (2)) VM, (z. a). In practice GALILEO optimizes the first term Eq. (22) with conservative policy gradient algorithms (c.g.. PPO [45] or TRPO [44]) avoid unreliable gradients for model improvements. Eq. (25) and Eq. (26) are optimized with supervised learning. The second term of Eq. (22) is optimized with supervised learning with a re-weighting term - -Apg.p; + M-- Since HM- is unknown, we use HM, to estimate it. When the mean output probability of a batch of data is larger than 0.6 or small than 0.4, we replace the second term of Eq. (22) with standard supervised learning in Eq. (21). Besides, unreliable gradients also exist in the process of optimizing the second term of Eq. (22). In our implementation, we use the scale policy gradients to constrain the gradients of the second term of Eq. (22). In particular, the lg-norm of the gradient of the first term of Eq. (22) via conservative policy gradient algorithms, named 119pe l|2. Then compute the 12-norm of the gradient of the second term of Eq. (22), name 10all2- Finally, we rescale the gradients of the second term Jat by
first compute
119pg 2
961
gal max 1 l l1ga
(27)
For each iteration, Eq. (22), Eq. (25), and Eq. (26) are trained with certain steps (See Tab. 6) following the same framework as GAIL. Based on the above techniques, we summarize the pseudocode of GALILEO in Alg. 3, where Po and P1 are set 0.4 and 0.6 in all of our experiments. The overall architecture is shown in Fig. 9.
E.2 Connection with Previous Adversarial Algorithms
Standard GAN [19] can be regarded partial implementation including the first term of Eq. (22) and Eq. (25) by degrading them into the single-step scenario. In the context of GALILEO, the objective of GAN is
0.41 arg max Epiro, [Ap (2r, a, x') log Mo ls,, a)]
s.l. yo* = arg max Epp. [log (2 a, x')l + Eps,. [log(1-D,ee -
where A.- (x) = log (x, In the single-step scenario, Pir, a,,a, = po(x))(a(2)) (x). The term Epic, [Ap- a, x') log Mo(x'|zx, a)] can convert to Eys, [log D' (x, 2')l by replacing the gradient of Ma, (a') (z. V. log Ma(a'|z,, a) with VoMo (x'|z, a) [51]. Previous algorithms like GANITE [58] and SCIGAN [7] can be regarded as variants of the above training framework.
The first term of Eq. (22) and Eq. (25) are similar to the objective of GAIL by regarding Ma as the "policy" to imitate and j the "environment" to collect data. In the context of GALILEO, the objective of GAIL. is:
act1= = arg max Bosia
[A (a,, a, x²) log Mo [))
s.t. Via, (I, a²) = F Σ rd log Du (Te, at, 34+1) (37e, as, 2++1) = (z,, a, a') x, Mo.
4" arg max Egg,, [log (1, a, a')]]] Eozee [log(1 Dilar, x'))]
where Ap (x, x²) = Qis (a,, a.) x²) Vilr (x, a) and VMo (z, a) = EMs, (2,a) (2, a, x')]
F
Additional Related Work
Our primitive objective is inspired by weighted empirical risk minimization (WERM) based on inverse propensity score (IPS). WERM is originally proposed to solve the generalization problem of domain adaptation in machine learning literature. For instance, we would like to train a predictor (y)rr) in a domain with distribution Purain (z) to minimize the prediction risks in the domain with distribution Ptest (I), where Ptest # Ptest- To solve the problem, we can train weighted objective
with max Ez~Peria Pr (a)) log M (v]=)), which is called weighted empirical risk minimization methods [5, 4, 15, 9, 42]. These results have been extended and applied to causal inference, where the predictor required to be generalized from the data distribution in observational studies (source domain) to the data distribution in randomized controlled trials (target domain) [48, 3, 22, 29, 28].
(reat()
28

---

In this case, the input features include state (a.k.a. covariates) and an action (a.k.. treatment
variable) which is sampled from a policy. We often assume the distribution of z, P(z) consistent
between the source domain and the test domain, then we have (vca(() P(r))t(ap)
P(a))[(a()]
[(a
(arai((r)
x(a=))
where 1 and B9 are the policies in source and target domains respectively. In [48, 22]. the policy in
randomized controlled trials is modeled as a uniform policy, then Pront ((a))(aa)) [aal]]
Piesiu (1n P(a))(((z)
x[a[z]
x
[[d|z]] [[a is also known as inverse propensity score (IPS). In [28], it assumes that the policy in
the target domain is predefined as 3(alzz) before environment model learning, then it uses ft as the
IPS. The differences between AWRM and previous works are fallen in two aspects: (I) We consider the distribution-shin problem in the sequential decision-making scenario. In this scenario, we not only consider the action distribution mismatching between the behavior policy μ and the policy to evaluation B, but also the follow-up effects of policies to the state distribution; (2) For faithful offline policy optimization, we require the environment model to have generalization ability in numerous different d policies. The objective of AWRM is proposed to guarantee the generalization ability of M in numerous different policies instead of a specific policy.
On a different thread, there are also studies that bring counterfactual inference techniques of causal inference into model-based RL [8, 39, 49]. These works consider that the transition function is relevant to some hidden noise variables and use Pearl-style structural causal models (SCMs), which is a directed acyclic graphs to define the causality of nodes in environment, to handle the problem. SCMs can help RL in different ways: [8] approximate the posterior of the noise variables based on the observation of data, and environment models are learned based on the inferred noises. The generalization ability is improved if we can infer the correct value of the noise variables. [39] discover several local causal structural models of a global environment model, then data augmentation strategies by leveraging these local structures to generate counterfactual experiences. [49] proposes a representation learning technique for causal factors, which is an instance of the hidden noise variables, in partially observable Markov decision processes (POMDPs). With the learned representation of causal factors, the performance of policy learning and transfer in downstream tasks will be improved.
Instead of considering the hidden noise variables in the environments, our study considers the environment model learning problem in the fully observed setting and focuses unbiased causal effect estimation in the offline dataset under behavior policies collected with selection bias.
In offline model RL, the problem is called distribution shift [59, 31, 14] which has received great attentions. However, previous algorithms do not handle the model learning challenge directly but propose techniques to suppress policy sampling and learning in risky regions [59, 30J. Although these algorithms have made great progress in offline policy optimization in many tasks, so far, how to learn better environment model in this scenario has rarely been discussed.
We are not the first article use the concept of density ratio for weighting. In off-policy estimation. [32, 35, 60] use density ratio to evaluate the value of a given target policy B. These methods attempt to solve an accurate approximation of w(s,, alp²). The objective of our work, AWRM, for the model to provide faithful feedback for different policies, formalized as minimizing the model error of the density function for any in the policy space II. The core of this problem how to obtain an approximation of the density function of the best best-response 8 corresponding to the current model, and then approximate the corresponding w(s, alp²*) )). It should be emphasized that since 8 is unknown in advance and will change as the induction bias of M is changed, the solutions proposed in [32, 35, 60] cannot be applied to AWRM: Recently. [24, 57] use density ratio weighting to learn model. The purpose of weighting is to make the model adapt to the current policy learning and adjust the model learning according to the current policy, which is the same as the WERM objective proposed in Def. 4.1. [54] also utilizes density ratio weighting to learn a model. Instead of estimating them based on the offline dataset and target policy as previous works do, they propose to design an adversarial model learning objective by constructing two function classes V and W, satisfying the target policy's 's value Vi and density ratio wis are covered, i.e., Va E V and Wg E W. Different from previous articles, our approach uses adversarial weighting to learn a universal model that provides good feedback for any target policy B E II, i.c., AWRM, instead of learning a model suitable to specific target policy.
29

---

G Experiment Details
G.1
Settings
G.1.1
General Negative Feedback Control (GNFC)
The design of GNFC is inspired by a classic type of scenario that behavior policies jt have selection bias and easily lead to counterfactual risks: For some internet platforms, would like to allocate budgets to a set of targets (e.g., customers or cities) to increase the engagement of the targets in the platforms. Our task is to train a model to predict targets' feedback engagement given targets' features and allocated budgets.
In these tasks, for better benefits, the online working policy (i.e., the behavior policy) will tend to cut down the budgets if targets have better engagement, otherwise, the budgets might be increased. The risk of counterfactual environment model learning in the task is that: the object with better historical engagement will be sent to smaller budgets because the selection bias of the behavior policies, then the model might exploit this correlation for learning and get conclusion that: increasing budgets will reduce the targets' engagement, which violates the real causality. We construct an environment and behavior policy to mimic the above process. In particular, the behavior policy PGNFC is
[GNFC(2) (62.5 mean(r))
+,,
15
where é is a sample noise, which will be discussed later. The environment includes two parts:
(1) response function M1 (1 a):
(1 (M(ean(() + a,
(2) mapping function M2(a' xr, w):
a. y y +
The transition function M* is composite of M (x'(1x,, M2 (2(x'lzz [ar, a, Mi (ylæ,, (ylar,a), a)). The behavior policies have selection bias: the actions taken are negatively correlated with the states, as illustrated Fig. 10(a) and Fig. 10(b). We control the difficulty of distinguishing the correct causality of x, a. and by designing different strategies of noise sampling on In principle, with a larger number or more pronounced disturbances, there are more samples violating the correlation between æ and a, then more samples can be used to find the correct causality. Therefore, we can control the difficulty of counterfactual environment model learning by controlling the strength of disturbance. In particular, we sample from uniform distribution U'-e, e) with probability That is, O with probability 1 1- p and ~ U--e,é e) with probability p. Then with larger there are more samples in the dataset violating the negative correlation (i.e., (GNFF)), and with larger e, the difference of the feedback will be more obvious. By selecting different e and p. we can construct different tasks to verify the effectiveness and ability of the counterfactual environment model learning algorithm.
50
response
30
10
timestep
timestep
(a) y in collected trajectories.
(b) in collected trajectories.
Figure 10: Illustration of information about the collected dataset in GNFC. Each color of the line denotes one of the collected trajectories. The X-axis denotes the timestep of a trajectory.
30

---

G.1.2 The Cancer Genomic Atlas (TCGA)
The Cancer Genomic Atlas (TCGA) is project that has profiled and analyzed large numbers of human tumors discover molecular aberrations the DNA, RNA, protein, and epigenetic levels. The resulting rich data provide significant opportunity to accelerate our understanding of the molecular basis of cancer. We obtain features, X, from the TCGA dataset and consider three continuous treatments done in SCIGAN [7]. Each treatment, is associated with set of parameters, Vi, V2, V3. that are sampled randomly by sampling a vector from standard normal distribution and scaling it with its norm. We assign interventions by sampling treatment, a, from beta distribution, x ~ Beta (a, 3). a 2 1 controls the sampling bias and B a=1 + 2 a, where a is the optimal treatment. This setting of ensures that the mode of Beta (a, 3) is a²
The calculation of treatment response and optimal treatment are shown in Table 4.
Table 4: Treatment response used generate semi-synthetic outcomes for patient features In the experiments, we set C' = 10.
Treatment
Treatment Response
Optimal treatment
f1(x,)1)= ((v)) x+ 12 (v))" xa1 12 (v3) xaae
(v))**
oj
2(v1)*
f2(x) C ((vi)* sin (I o2)
(vB)
2
2(v3)"
3
fa(x, 0n) = C ((v9)" x 120g(as b)?, where - 0.75
ifb 0.75, if < 0.75
We conduct experiments on three different treatments separately and change the value of bias a to assess the robustness of different methods to treatment bias. When the bias of treatment is large, which means a is large, the training set contains data with a strong bias on treatment so it would be difficult for models to appropriately predict the treatment responses out of the distribution of training data.
G.1.3 Budget Allocation task to the Time period (BAT)
1. customers send orders remotely based on the platform
2. stores make food
stores
2 customers
7
delivery clerks take food
4, delivery clerks to the customers
delivery
5. The platform will pay delivery cierks for fultiling the order
platfomm
Figure 11: Illustration of the workflow of the food-delivery platform.
We deploy GALILEO in a real-world large-scale food-delivery platform. The platform contains various food stores, and food delivery clerks. The overall workflow is as follows: the platform presents the nearby food stores to the customers and the customers make orders, i.e., purchase take-out foods from some stores on the platform. The food delivery clerks can select orders from platform to fulfill. After an order selected to fulfill, the delivery clerks will take the ordered take-out foods from the stores and then send the food to the customers. The platform will pay the delivery clerks (mainly proportion to the distance between the store and the customers' location) once the orders are fulfilled. An illustration of the workflow can be found in Fig. 11.
31

---

1.0
0.9
normalized fulfilled
0.6
0.4
0.2
normallized actions
Figure 12: Illustration of relationship between user feedback and the actions of the offline dataset in the real-world food-delivery platform.
However, there is an imbalance problem between the demanded orders from customers and the supply of delivery clerks to fulfill these orders. For example, at peak times like lunchtime, there will be many more demanded orders than at other times, and the existed delivery clerks might not be able to fulfill all of these orders timely. The goal of the Budget Allocation task to the Time period (BAT) is to handle the imbalance problem time periods by sending reasonable allowances to different time periods. More precisely, the goal of BAT is to make all orders (i.e., the demand) sent in different time periods can be fulfilled (i.e., the supply) timely.
The core challenge of the environment model learning in BAT tasks is similar to the challenge in Fig. 1. Specifically, the behavior policy in BAT tasks is human-expert policy, which will tend to increase the budget of allowance in the time periods with a lower supply of delivery clerks, otherwise will decrease the budget (Fig. 12 gives an instance of this phenomenon the real data).
To handle the imbalance problem in different time periods, in the platform, the orders in different time periods & [0. 1, 2.... 23) will allocated with different allowances E V+ For example, at 10 A.M. (i.e., = 10), we add 0.5$ (i.e., c 0.5) allowances to all of the demanded orders. From 10 A.M. 11 A.M., the delivery clerks who take orders and send food customers will receive extra allowances. Specifically, if the platform pays the delivery clerks 28 for fulfilling the order, now he/she will receive 2.58. For each day, the budget of allowance budget allocation policy nT (cl() of the limited budget C to make as many orders possible can be taken timely.
fixed. We should find the best
To find the policy, we first learn a model to reconstruct the response of allowance for each delivery clerk M (Vt+1 (t. Pt, Ct), where Vt-1 is the taken orders of the delivery clerks in state st, C is the allowances, Pt denotes static features of the time period t. In particular, the state s, includes historical order-taken information of the delivery clerks, current orders information, the feature of weather, city information, and so on. Then we use rule-based mapping function f to fill the complete next time-period states, i.e., 81+1 f*t, Pt, ct, Mt+1). Here we define the composition of the above functions Mf and f as M, Finally, learn budget allocation policy based the learned model. For each day, the policy we would like to find is:
max E S
Σ 3e 3
s.t.,
Cofi S C
t,sES
In our experiment, we evaluate the degree of balancing between demand and supply by computing the averaged five-minute order-taken rate, that is the percentage of orders picked up within five minutes. Note that the behavior policy is fixed for the long term in this application. So we directly use the data replay with small scale of noise (See Tab. 6) to reconstruct the behavior policy for model learning in GALILEO.
Also note that although we model the response for each delivery clerk, for fairness, the budget allocation policy is just determining the allowance of each time period and keeps the allowance to each delivery clerk the same.
32

---

G.2 Baseline Algorithms
The algorithm we compared are: (1) Supervised Learning (SL): training a environment model to minimize the expectation of prediction enror, without considering the counterfactual risks; (2) inverse propensity weighting (IPW) [50]: a practical way to balance the selection bias by re-weighting. can be regarded as xe. where is another model learned to approximate the behavior policy; (3) SCIGAN: a recent proposed adversarial algorithm for model learning for continuous- valued interventions [7]. All of the baselines algorithms are implemented with the same capacity of neural networks (See Tab. 6).
G.2.1 Supervised Learning (SL)
As baseline, we train a multilayer perceptron model directly predict the response of different treatments, without considering the counterfactual risks. We use mean square eπror to estimate the performance of our model so that the loss function can be expressed as MSF 1 Σ (v+ i.) where n is the number of samples, y is the true value of response and 1 is the predicted response. In practice, train our SL models using Adam optimizer and the initial learning rate 3e on both datasets TCGA and GNFC. The architecture of the neural networks is listed in Tab.
G.2.2 Inverse Propensity Weighting (IPW)
Inverse propensity weighting [50] is an approach where the treatment outcome model uses sample weights to balance the selection bias by re-weighting. The weights are defined as the inverse propensity of actually getting the treatment, which can be expressed as Aalzz where x stands for the feature vectors in a dataset, taken probability of given the features z within the dataset. ji is learned with standard supervised learning. Standard IPW leads to large weights for the points with small sampling probabilities and finally makes the learning process unstable. We solve the problem by clipping the propensity score: f < min(fi, 0.05), which common used in existing studies [27]. The loss function can thus be expressed as Ei= plaila;) (yi - ūi)² The architecture of the neural networks listed in Tab. 6.
the corresponding action and (a(a..)) indicates the action
G.2.3 SCIGAN
SCIGAN [7] is a model that uses generative adversarial networks to learn the data distribution of the counterfactual outcomes and thus generate individualized response curves. SCIGAN does not place any restrictions on the form of the treatment-doe response functions and is capable of estimating patient outcomes for multiple treatments, each with an associated parameter. SCIGAN first trains a generator to generate response curves for each sample within the training dataset. The learned generator can then be used to train an inference network using standard supervised methods. For fair comparison, we increase the number of parameters for the open-source version of SCIGAN so that the SCIGAN model can have same order of magnitude of network parameters as GALILEO. In addition, we also finetune the hyperparameters (Tab. 5) of the enlarged SCIGAN to realize its full strength. We set num num_muummaaaa -amples 9 and À 10.
Table 5: Table of hyper-parameters for SCIGAN.
Parameter
Values
Number of samples
3, 7,9, 11
X
0.1, 1, 10, 20
G.3 Hyper-parameters
We list the hyper-parameter of GALILEO in Tab. 6.
G.4 Computation Resources
We use one Tesla V100 PCIe 32GB GPU and 32-core Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHZ to train all of our models.
33

---

Table Table of hyper- syper-parameters for all of the tasks.
Parameter
GNFC
TAGC
MuJoCo
BAT
hidden layers of all neural networks
4
5
5
hidden units of all neural networks
256
256
512
512
collect samples for each time of model update
5000
5000
40000
96000
batch size of discriminators horizon
5000
5000
40000
80000
50
1
100
48 (half-hours)
En (also ep)
0.005
0.01
0.05 (0.1 for walker2d)
0.05
times for discriminator update
2
5
times for model update
1
2
20
times for supervised learning update
20
learning rate for supervised learning
le-5
le-5
3e-4
le-5
0.99
0.0
0.99
0.99
clip-ratio max D
NAN
NAN
NAN
0.1
0.001
0.001
0.001
NAN
optimization algorithm (the first term of Eq. (22))
TRPO
TRPO
TRPO
PPO
35
GALILEO
GALILEO
30
JPW
25
SCIDAN
10ot MISE
SCIGAN
20
2 60
10
61.p1
07.00.2
05.00
60.2.01
e0._p0..2
00.2
20.22
#0.
60.05
w0.05_po.2
p0.2
e0.05 e0.05_p0.05
blss
bince
bincs
bies?
biusn
biess
(a) GNFC
(b) TCGA
Figure 13: Illustration of the performance in GNFC and TCGA. The grey bar denotes the standard emrr (x0.3 for brevity) of 3 random seeds.
H
Additional Results
H.1 Test in Single-step Environments
The results of GNFC tasks are summarized in Fig. 13(a) and the detailed results can be found in Tab. 11. The results show that the property of the behavior policy (i.c., and p) dominates the generalization ability of the bascline algorithms. When and give completely opposite response curve. IPW still perform well when 0.2 < S 1.0 but fails when e 0.05, p <= 0.2. We also found that SCIGAN can reach better performance than other baselines when 0.05, <= 0.2, but the results in other tasks are unstable. GALILEO is only algorithm that robust to the selection bias and outputs correct response curves all of the tasks. Based on the experiment, we also indicate that the commonly used overlap assumption is unreasonable certain extent especially in real-world applications since it impractical to inject noises into the whole action space. The problem of overlap assumption being violated should be taken into consideration otherwise the algorithm will be hard to use in practice if sensitive to the noise range.
0.05, almost all of the basclines fail
The results of TCGA tasks are summarized Fig. 13(b) and the detailed results can be found in Tab. 12. We found the phenomenon in this experiment is similar to the one in GNFC, which demonstrates the compatibility of GALILEO to single-step environments. We also found that the results of IPW are unstable in this experiment. It might be because the behavior policy is modeled with beta distribution while the propensity score modeled with Gaussian distribution. Since IPW directly reweight loss function with the results are sensitive to the error on GALILEO also models á with Gaussian distribution but the results are more stable since GALILEO does not re- re-weight through á explicitly.
We give the averaged responses for all of the tasks and the algorithms in Fig. 21 to Fig. 28. We randomly select 20% of the states in the dataset and equidistantly sample actions from the action space for each sampled state, and plot the averaged predicted feedback of each action. The real response is slightly different among different figure as the andomly-selecte states for testing different. We sample 9 points in GNFC tasks and 33 points in TAGC tasks for plotting.
34

---

H.2 All of the Result Table
We give the result of CNFC in Tab. 11, TCGA in Tab. 12, BAT in Tab. 9, and MuJoCo in Tab. 10.
H.3 Ablation Studies
In Appx. E.1, we introduce several techniques to de- velop practical GALILEO algorithm. Based task e0 2_p0. 05 in GNFC, we give the ablation studies to in- vestigate the effects of these techniques. We first compare two variants that do not handle the assumptions violation problems: (1) NO_ MNJECT_NOISER set εμ and ED to zero,
sqnt MISE
which makes the overlap assumption not satisfied;; (2)
SINGLE SL: without replacing the second term in Eq. (6)
with standard supervised learning even when the output
probability of is far away from 0.5. Besides, we intro- duced several tricks inspired by GAIL and give compari- son of these tricks and GAIL: (3) ONE ONE_STEP: use one-step reward instead of cumulative rewards (i.e., Q and V; see Eq. (23) and Eq. (24)) for re- weightting, which is imple- mented by set to O; (4) SINGE_DIS: remove Tep (I, a) and replace with EM, [Teg (I, a, x²)], which is inspired by GAIL that uses a value function as a baseline instead of using another discriminator; (5) PURE_ GAIL: remove the second term in Eq. (6). It can be regarded as naive adoption of GAIL and partial implementation of GALILEO.
GAULLED
SINGLE,
ONE_STE STEP
SINGLE,
PURE, GAIL
NO JUJECT NOISE
Figure 14: Illustration of the ablation studies. The error bars the standard error.
We summarize the results Fig. 14. Based on the results of NO INJENT _OTTTIIOIIEEEE and SINGLE, SL, we that handling the assumption violation problems is important and will increase the ability counterfactual queries. The results of PURE_GAIL tell us that the partial implementation of GALILEO not enough to give stable predictions counterfactual data; On the other hand, the result of ONE_STEP also demonstrates that embedding the cumulative error of one-step prediction is helpful for GALILEO training; Finally, we also found that SINGLE nearly has almost no effect on the results. It suggests that, empirically, we can use EM, [Tig (I, a, x') as a replacement for Tep (I, a), which can reduce the computation costs of the extra discriminator training.
H.4 Worst-Case Prediction Error
In theory, GALILEO increases the generalization ability by focusing on the worst-case samples' training achieve AWRM. To demonstrate the property, we propose new metric named Mean-Max Square Error (MMSE): E max.EA (M* (ar (u/|z, a) Mx'12, a))²] and give the results of MMSE for GNFC Tab. and for TCGA in Tab. 14.
35

---

H.S Detailed Results in the MuJoCo Tasks
We select 3 environments from D4RL [17] to construct our model learning tasks. We compare it with typical transition model learning algorithm used in the previous offline model based RL algorithms [59, 30J. which is variant of standard supervised learning. We name the method OFF- SL. Besides, we also implement IPW and SCIGAN as the baselines. We train models in datasets HalfCheetah-mediume
Walker2d-medium, and Hopper-medium policy with 1/3 performance to the expert policy, then we test them in the corresponding expert dataset. For better training efficiency, the trajectories in the training and testing datasets are truncated, remaining the first 100 steps. We plot the converged results and learning curves of the three MuJoCo tasks in Tab. 10 and Fig. 15 respectively.
which are collected by behavior
1.2
0.25
GAL LED
SCISAR
0IT-3L
SOGAN
5000AN
1.0
I7W
PN
Sqpt MSE
0.15
0.6
0.2
0.6
0.4
0.55
0.5
0.4
update
x22
update
x10
halfcheetah- medium (train)
(b) hopper- (train)
(c) walker2d-medium (train)
2.4
GRULLO
0.75
3.6
GALLES
OFF-1
SCEGAN
3.5
PM
My
3.4
2.0
0.50
3.3
2.0
0.55
12
2.6
0.45
0.40
1.0
0.0
0.2
0.0
0.2
0.4
0.5
0.8
2.5
0.0
0.2
update steps
update steps
update steps
(d)
Salffhhctta-mmcdiu---re
(c) hopper-medium-replay (test)
(f) walker2d-mecium--rppha (test)
(lest)
4.00
CALLEO
GALILEO DFF-SL
GALLCCO
DRF
SCIGAN
SOGAN
3.8
325
sqrt MSB
1.7
sqtt
2.75
3.6
2.50
1.5
3.0 KIEP
updatt steps
x30F
update steps
update
(g) halfcheetab-expert (lest)
hopper-expert (test)
walker2d-expert (test)
Figure 15: Illustration of learning curves of the MuJoCo Tasks. The X-axis record the steps of the environment model update, and the Y-axis is the corresponding prediction error. The figures with titles ending in "(train)" means the dataset is used for training while the titles ending in "(test)" means the dataset is just used for testing. The solid curves are the mean reward and the shadow is the standard error of three seeds.
In Fig. 15, we can see that algorithms perform well in the training datasets. OFF-SL and SCIGAN can even reach bit lower error in halfcheetah and walker2d. However, when we verify models through "expert" and "medium-replay" datasets, which are collected by other policies, the performance of GALILEO is significantly more stable and better than all other algorithms. As the training continues, the baseline algorithms even gets worse and worse. However, whether in GALILEO or other basclines, the performance for testing is least 2x worse than in the training
36

---

dataset, and the error is large especially in halfcheetah. The phenomenon indicates that although GALILEO can make better performances for counterfactual queries, the risks of using the models are still large and still challenging to be further solved.
Table 7: Results of policy performance directly optimized through SAC [20] using the learned dynamics models and deployed in MuJoCo environments, MAX-RETURN is the policy performance of SAC in the MuJoCo environments, and "avg. norm." is the averaged normalized return of the
policies in the 9 tasks, where the returns are normalized to lie between 0 and 100, where a score of corresponds to the worst policy, and 100 corresponds to MAX- MAX-RETURN.
Task
Hopger
Walker2d
HalfCheetah
avg. norm.
Horizon
I1-10
H=20
B=40
H=10
11:20
M=:40
M=10
TE=20
B1:40
GALILEO
13.0 0.1
13.2 0.1
53.5 a 1.2
11.7 + 0.2
29.9 = 0.3
61.2 34
0.7 0:2
-11 0.2
-1422 1.4
51.1
OFF-SI.
0.5
0.2
4.3
10.7 0.2
201
375
0.0
-1.1 0.6
-132+:033
21.1
IPW
594
4.-066
59+02
28-39
145 14
+.+
0.5
-113+009
19.7
SCIGAN
12.7
292106
462
8.
9.1
1.0188
1.2103
0..1..
-11.4.0.3
41.5
MAX-RETURN
132+00
333002
71.0005
14.
60.-1111
26+00
133111
49.1
1000
We then verify the generalization ability of the learned models above by adopting them into offline model-based RI.. Instead of designing sophisticated tricks to suppress policy exploration and learning in risky regions as current offline model-based RI. algorithms [59, 30] do, we just use the standard SAC algorithm [20] to exploit the models for policy learning to strictly verify the ability of the models. Unfortunately, we found that the compounding error will still be inevitably large in the 1, 1,000-step rollout, which is the standard horizon in MuJoCo tasks, leading all models to fail to derive reasonable policy. To better verify the effects of models on policy optimization, we learn and evaluate the policies with three smaller horizons: H E {10, 20, 40}.
The results have been listed in Tab. 7, where the learning curve in the dynamics models and the real environments is shown in Fig. 16 and Fig. 17. We first averaged the normalized return (refer to "avg. norm.") under each task, and than other models (the improvements are 24% to 161 6). At the same time, we found that SCIGAN performed better in policy learning, while IPW performed similarly to SL. This is in line with our expectations, since IPW only considers the uniform policy as the target policy for debiasing, while policy optimization requires querying a wide variety of policies. Minimizing the prediction risks only under a uniform policy cannot yield a good environment model for policy optimization. On the other hand, SCIGAN, as a partial implementation of GALILEO (refer to Appx. E.2), also roughly achieves AWRM and considers the cumulative effects of policy on the state distribution, so its overall performance better; In addition, we find that GALILEO achieves significant improvement in 6 of the 9 tasks. But in HalfCheetah, IPW works slightly better. However, compared with it can be found that all methods fail to derive reasonable policies because their policies' performances are far away from the optimal policy. By further checking the trajectories, we found that all the learned policies just keep the cheetah standing in the same place or even going backward and fall down A
that the policy obtained by GALILEO is significantly higher
H.6
Off-policy Evaluation (OPE) in the MuJoCo Tasks
1.
Training and Evaluation Settings
We select environments from D4RL [17] to construct our model learning tasks as Appx. H.5. To match the experiment setting in DOPE [18], here we use the whole datasets to the train GALILEO model, instead of truncated dataset in Appx. H.5, for GALILEO model training.
OPE via a learned dynamics model is straightforward, which only needs to compute the return using simulated trajectories generated by the evaluated policy under the learned dynamics model. Due to the stochasticity in the model and the policy, we estimate the return for a policy with Monte-Carlo sampling. See Alg. 4 for pseudocode, where we use 7 0.995, N 10, H 1000 for all of the tasks.
H.6.2 Metrics
The metrics we use in our paper defined as follows:
'hhe videos can be found in http://ghhhl.cooooooogniccclllao
37

---

342
reture models
200
returns in
40
60
10
20
GALLEO GANTT IPW
CAJLSO
GAWATE
OTT-SL
OFF-SL
OTT-SL
x12
Uime- slep
x1e
Limie-slep
X10
(a) hopper H 10
(b) hopper 20
(c) hopper H 40
20
GALLEE GANITZ
200
returns in models
retumn in models
50
250
40
10
returns
300
IPW
CHP
OPFISL
groundout
time-stap
timo stop
Lime-step
(d) halfcheetah 17 = 10
(e) halfcheetah 17 = 20
halfcheetah 17 = 40
30
retumi models
20
retums in models
GANTT
PW OFF SL
PA)
30
returns
0f1
DAMTE
PW
01F-51
-5
grouns woth sm
time-stop
time-stop
time-stop
(g) halfcheetah H 10
(h) halfcheetah H 20
(1) halfchcetah H = 40
Figure 16: Illustration of offline policy learning curves of the MuloCo Tasks. The X-axis record the steps of the environment model update, and the Y-axis is the corresponding returns in the dynamics models. The solid curves are the mean reward and the shadow the standard error of three seeds.
Absolute Error The absolute error is defined as the difference between the value and estimated value of policy:
AbsErr V
V
(28)
where T is the true value of the policy and 5- is the estimated value of the policy.
Rank correlation Rank correlation measures the correlation between the ordinal rankings of the value estimates and the true values, which can be written as:
RankCorr Cov(VT. ViTT))
(29)
((V.. (V i.n)
where 1 N denotes the indices of the evaluated policies.
38

---

30
evel returns
eval retums
GAURLEO
50
GALILEO
15
OTT-S.
ground SOLEN
GAI LFO
20
GANTTE
IPW OTT-SL
time-step
x20
Lime-slepp
(a) hopper H 10
(b) hopper 20
(c) hopper H 40
250
GALLEO GANIE 2
15
PM
30
eval returns
eval natums
202
GALLEED
GALIEO
-10
time step
K30
tie stop
x10
time-stap
x10
(d) halfcheetah H = 10
(e) halfcheetah H 20
halfcheetah H 40
GAL LFD
10.0
OFF
ground bruth
eval returns
GALEEO GAMTT
PV
OPP-SL
0FF-3L
Srouns muth R72
gruuddouuhh.. cn
-23
time-stop
time-stop
XLE
timo-stop
x10'
(g) halfcheetah H
10
(h) halfcheetah H
20
(i) halfcheetah H 40
Figure 17: Illustration of offline policy learning curves of the MuJoCo Tasks. The X-axis record the steps of the environment model update, and the Y-axis is the corresponding returns in the ground- truth environments. The solid curves are the mean reward and the shadow is the standard error of three seeds.
Regret@k @k Regret@k is the difference between the value of the best policy in the entire set, and the value of the best policy in the top-k set (where the top-k is chosen by estimated values). can be defined as:
Regret @k max V sEL:N
max V
(30)
JEtopk(1:N)
where topk(1 N) denotes the indices of the top K policies measured by estimated values 17
39

---

Algorithm 4 Off-policy Evaluation with GALILEO model
Require: GALILEO model (Ma), evaluated policy number of rollouts N. set of initial states So. discount factor 7, horizon length for i 1 to N do
R; = 0
Sample initial state so So
Initialize T-1 =
for = 0 to H do
Qt ~ (-18t)
$1+1, r2 Mo(-Ist,a.) at)
R; = Rs + rrre
end for end for return
E R2
N
40

---

H.6.3 Detailed Results
The results of OPE on three tasks are in Tab. 8. Firstly, we can see that for all recorded rank- correlation scores, GALILEO is significantly better than all the baseline methods, with least 25 % improvements. As for regret@) I, although GALILEO cannot reach the best performance among all of the tasks, always one the top-2 methods among the three tasks, which demonstrates the stability of GALILEO. Finally, GALILEO has the smallest value gaps except for Halfcheetah. However, for of the top-4 methods, the value gaps in halfcheetah are around 1,200. Thus we believe that their performances value gaps roughly at the same level.
Table 8: Results of OPE DOPE benchmark. + is the standard deviation. We bold the top-2 scores for each metric. The results are tested on 3 random seeds. The results of the baselines are from [18]. Note that the rank correlation is "NAN" for HalfCheetah because the scores not given in [18].
TASK
HalfCheetah
METRIC
Absolute value gap
Rank correlation
Regret 1
GALILEO
1280 83
0.313 0.09
0.12 0.02
Best DICE
1382 = 130
NAN
0.82 = 0.29
VPM
1374 = 153
NAN
0.33 = 0.19
FQE (L2)
1211 = 130
NAN
0.38 = 0.13
IS
1217 + 123
NAN
0.05 = 0.05
Doubly Rubost
1222 = 134
NAN
0.37 = 0.13
TASK
Walker2d
METRIC
Absolute value gap
Rank correlation
Regret @
GALILEO
176 +
0.57 + 0.08
0.08 0.06
Best DICE
273 = 31
0.12 + 0.38
0.27 = 0.43
VPM
426 = 60
0.44 - 0.21
0.08 = 0.06
FQE (L2)
350 = 79
-0.09 = 0.36
0.31 = 0.10
IS
428 60
-0.25 = 0.35
0.70 = 0.39
Doubly Rubost
368 74
0.02 0.37
0.25 + 0.09
TASK
Hopper
METRIC
Absolute value gap
Rank correlation
Regret 1
GALILEO
156 23
0.45 = 0.1
0.08 + 0.08
Best DICE
215 41
0.19 - 0.33
0.18 = 0.19
VPM
433 44
0.13 0.37
0.10 = 0.14
FQE (1.2)
283 73
-0.29 + 0.33
0.32 + 0.32
IS
405 + 48
-0.55 + 0.26
0.32 + 0.32
Doubly Rubost
307 + 73
-0.31 + 0.34
0.38 + 0.28
41

---

H.7 Detailed Results in the BAT Task
The core challenge of the environment model learning in BAT tasks is similar to the challenge in Fig. 1. Specifically, the behavior policy in BAT tasks is human-expen policy, which will tend to increase the budget of allowance in the time periods with a lower supply of delivery clerks, otherwise will decrease the budget (Fig. 12 gives instance of this phenomenon in the real data).
Since there is no oracle environment model for querying, we have describe the results with other metrics.
roscaled aversped nesgorte
GALLCO
GALLOO
GAULDO
0.96
0.90
0.90
0.44
0.94
0.94
0.00
0M2
0.92
0.88
0.90
0.00
0.98
0.81
0.0
0.2
04
08
0.5
00
62
0.4
0.8
0.8
1.0
08
02
0.6
0.5
0.5
normalizad allocated allowances
normalized allocated allowances
normaized allocated allowances
(a) City-A
(b) City-B
(c) City-C
avarrgad respol
1.00
ressoalee oierrgeee respomnee
1.00
0.56
0.00
0.06
0.98
0.54
0.87
0.02
0.90
2.32
100
0.83
0.0
022
54
0.6
0.
1.0
0.0
0.2
0.4
0.6 0.8
1.0
0.
02
B4
0.
0.
1.0
normalized allocated allowances
normelizee allocated allowances
normaized allocated allowances
(d) City-D
(c) City-E
(f) City-F
Figure 18: Illustration of the response curves in the 6 cities. Although the ground-truth curves are unknown, through human expert knowledge, we know that it is expected be monotonically increasing.
First, we review whether the tendency of the response curve consistent. this application, with larger bud- get of allowance, the supply will not be decreased. As can
1.0
be seen in Fig. 18, the tendency of GALILEO's response valid in 6 cities but almost all of the models of SL give opposite directions to the response. we learn a policy through the model of SL, the optimal solution is cancel- ing all of the allowances, which obviously incorrect in practice.
rescaled oummuattvve trealment
GALLED
0.5
-1.0
Second, we conduct randomized controlled trials (RCT) in one of the testing cities. Using the RCT samples, we can evaluate the correctness of the sort order of the model predictions via Area Under the Uplift Curve (AUUC) [6]. To plot AUUC, we first sort the RCT samples based on the predicted treatment effects. Then the cumulative treatment effects are computed by scanning the sorted sample list. If the sort order of the model predictions is better, the sample with larger treatment effects will be computed early. Then the area of AUUC will be larger than the via random sorting strategy. The result of AUUC show GALILEO gives a reasonable sorting to the RCT samples (see Fig. 19).
cumulative samples (rescaled 1)
Figure 19: Illustration of the AUCC re- sult for BAT. The model with larger ar- eas above the "random" line makes bet- ter predictions in randomized-controlle trials data [61].
Finally, we search for the optimal policy via the cross-entropy method planner [21] based on the learned model. We test the online supply improvement in 6 cities. The algorithm compared is human-expert policy, which is also the behavior policy of the offline datasets. We conduct online
42

---

A/B tests for each of the cities. For each test, we randomly split a city into two partitions, one is for deploying the optimal policy learned from the GALILEO model, and the other is control group, which keeps the human-expert policy as before. Before the intervention, we collect 10 days' observation data and compute the averaged five-minute order-taken rates as the baselines of the treatment and control group, named be and & respectively. Then we start intervention and observe the five-minute order-taken rate in the following 14 days for the two groups. The results the treatment and control groups are y and yf respectively, where i denotes the i-th day of the deployment. The (DID):
percentage points of the supply improvement are computed via difference-in-diffeeee
E. b') (vf b²)
100,
where 7 is the total days of the intervention and T 14 in our experiments.
Table 9: Results BAT. We use City-X to denote the experiments different cities. "pp" is an abbreviation of percentage points on the supply improvement.
target city
City-A
City-B
City-C
supply improvement
+1.63pp
+0.79pp
+0.27pp
target city
City-D
City-E
City-F
supply improvement
+0.2pp
+0.14pp
+0.41pp
The results are summarized Tab. 9. The online experiment is conducted in 14 days and the results show that the policy learned with GALILEO can make better (the supply improvements are from 0.14 to 1.63 percentage points) budget allocation than the behavior policies in all the testing cities. We give detailed results which record the supply difference between the treatment group and the control group in Fig. 20.
0.050-440 talto pale
resraled Eve-minele urder-ttlen role
1.0000
0.98
miscaldd me-mintts order-tokknn
0.86
0.84
3.84
0.32
0.90
0.88
0.90
0.86
0.88
Deeateent (GAL satgnna
0.88
All
0.866
15
20
20
10
day number
day number
day number
(a) City-A
(b) City-B
(c) City-C
roszalbe monmmmm 336-3 rato
resocaled five--mn reee--tiuuee ordentaken rale
1.00
1.00
1.000
nocollld orden-aaknn
0.50
0.99
0.50
0.98
8.57
0.00
0.07
aree grop
0.59
(200 glup
day number
day number
day number
City-D
(e) City-E
City-F
Figure 20: Illustration of the daily responses in the A/B test in the 6 cities.
43

---

Table 10: The root square MuJoCo tasks. We bold the lowest error for each task. "medium" dataset is used for training. while "expert" and "medium-replay" datasets are just used for testing. + follows the standard deviation of three seeds.
TASK
HalfCheetah
DATASET
medium (train)
expert (test)
medium-replay (test)
GALILEO
0.378 = 0.003
2.287 0.005
1.411 = 0.037
OFF-SL
0.404 + 0.001
3.311 = 0.055
2.246 = 0.016
IPW
0,513 + 0.033
2.892 0.050
2.058 = 0.021
SCIGAN
0.309 + 0.002
3.813 + 0.133
2.484 + 0.040
TASK
Walker2d
DATASET
medium (train)
expert (test)
medium-replay (test)
GALILEO
0.49 = 0.001
1.514 = 0.002
0.968 + 0.004
OFF-SL
0.467 + 0,004
1,825 = 0.061
1.239 = 0.004
IPW
0.564 + 0.001
1.826 + 0.025
1.282 + 0.007
SCIGAN
0.438 + 0.001
1.825 + 0.031
1.196 = 0.005
TASK
Hopper
DATASET
medium (train)
expert (test)
medium-replay (test)
GALILEO
0.037 0.002
0.322 = 0.036
0.408 + 0.003
OFF-SL
0.034 + 0.001
0.464 = 0.021
0.574 = 0.008
IPW
0.039 + 0.001
0.533 0.00
0.671 = 0.001
SCIGAN
0.039 + 0.002
0.628 = 0.050
0.742 = 0.019
Table 11: VMISE results on GNFC. We bold the lowest error for each task. + is the standard deviation three random seeds.
el_pl
c0.2_pl pl
c0.05_pt pl
GALILEO
5.17 0.06
4.73 + 0.13
4.70 + 0.02
SL
5.15 = 0.23
4.73 0.31
23.64 + 4.86
IPW
5.22 = 0.09
5.50 + 0.01
5.02 + 0.07
SCIGAN
7.05 = 0.52
6.58 + 0.58
18.55 + 3.50
e1 e1_p0.2
e0.2_p0.2
e0.05_p0.2
GALILEO
5.03 0.09
4.72 0.05
4.87 + 0.15
SL
5.21 = 0.63
+ 4
33.52 + 1.32
IPW
5.27 0.05
5.69 + 0.00
20.23 0.45
SCIGAN
16.07 + 0.27
12.07 + 1.93
19.27 + 10.72
e1 e1_50.05
e0.2_p0.05
e0.05_p0.00
GALILEO
5.23 + 0.41
5.01 + 0.08
6.17 + 0.33
SL
5.89 = 0.88
14.25 + 3.48
37.50 2.29
IPW
5.21 = 0.01
5.52 + 0.44
31.95 + 0.05
SCIGAN
11.50 + 7.76
13.05 + 4.19
25.74 8.30
44

---

Table 12: VMISE results TCGA. We bold the lowest error for each task. + is the standard deviation of three random seeds.
10_bias 2.0
t0_bias 20.0
[0_bias_500 _5.0
GALILEO
0.34 - 0.05
0.67 = 0.13
2.04 + 0.12
SL
0.38 + 0.13
1.50 + 0.31
3.06 + 0.65
IPW
6.57 + 1.16
6.88 + 0.30
5.84 + 0.71
SCIGAN
0.74 + 0.05
2.74 + 0.35
3.19 + 0.09
__bias 2.0
t__ias 6.0
[I_bias_8.)
GALILEO
0.43 0.05
0.25
0.02
0.21 + 0.04
SL
0.47 + 0.05
1.33 + 0.97
1.18 + 0.73
IPW
3.67 + 2.37
0.54 + 0.13
2.69
1.17
SCIGAN
0.45 = 0.25
1.08 = 1.04
1.01 + 0.77
12_bias 2.0
12_bias 6.0
2__iaa__8 8.0
GALILEO
1.46 + 0.09
0.85 0.04
0.46 + 0.01
SL
0.81 + 0.14
3.74 + 2.04
3.59 + 0.14
IPW
2.94 + 1.59
1.24 = 0.01
0.99 0.06
SCIGAN
0.73 + 0.15
1.20 = 0.53
2.13 - 1.75
Table 13: v ES results GNFC. We bold the lowest for each task. + is the standard deviation three random seeds.
el_p1
e0.2_pl
e0.05_pl
GALILEO
3.86 + 0.03
3.99 + 0.01
4.07 = 0.03
SL
5.73 + 0.33
5.80 0.28
18.78 + 3.13
IPW
4.02 + 0.05
4.15 + 0.12
22.66 0.33
SCIGAN
8.84 + 0.54
12.62 2.17
24.21 + 5.20
el_20.2
e0.2_p0.2
e0.05_p0.2
GALILEO
4.13 + 0.10
4.11 0.15
4.21 0.15
SL
5.87 + 0.43
7.44 + 1.13
29.13 + 3.44
IPW
4.12 + 0.02
6.12 + 0.48
30.96 0.17
SCIGAN
12.87 + 3.02
14.59 + 2.13
24.57 + 3.00
e1_00.05
e0.2_p0.05
e0.05_p0.00
GALILEO
4.39 + 0.20
4.34 + 0.20
5.26
0.29
SL
6.12 + 0.43
14.88 4.41
30.81
1.69
IPW
13.60 + 7.83
26.27 2.67
32.55 0.12
SCIGAN
9.19 + 1.04
15.08 + 1.26
17.52 + 0.02
45

---

Table 14: VMMSE results on TCGA. We bold the lowest error for each task. + is the standard deviation of three random seeds.
10_bias 2.0
t0_bias 00_ias__0... 20.0
00_bias_500
GALILEO
1.56 + 0.04
1.96 = 0.53
3.16 + 0.13
SL
1.92 + 0.67
2.31 = 0.19
5.11 0.66
IPW
7.42 = 0.46
5.36 = 0.96
5.38 + 1.24
SCIGAN
2.11 + 0.47
5.23 + 0.27
5.59 + 1.02
t1 t1_bass_20 2.0
t1 t1_bias66.
t1 bias_8.0
GALILEO
1.43 + 0.06
1.09 = 0.05
1.36 + 0.36
SL
1.12 0.15
3.65 = 1.91
3.96 1.81
IPW
1.14 1.14
0.90 + 0.09
2.04 + 0.99
SCIGAN
3.32 + 0.88
4.74 + 2.12
5.17 2.42
12_bias 2.0
12_bias_Gr
12 b.0s_8.0
GALILEO
3.77 + 0.35
3.99 = 0.40
2.08 + 0.60
SL
2.70 + 0.67
8.33 + 5.05 5.05
9.70 3.12
IPW
2.92 +: 0.15
3.90 + 0.17
4,47 + 2.16
SCIGAN
3.82 + 2.12
1.83 + 1.49
3.62 + 4.9
34
30
pied
prei real
real
real
0.6
1.0
0.5 action
action
action
(a) bias
(b) bias20
(c) bias50
everagce responge
pred
14
aweraged
33
22
2.0
0.2
3.4
0.6
0.8
1.0
02
0.4
0.6
9.8
02
2.3
0.4
2.5 action
0.5
0.7
0.8
04
action
action
(d) t1 bias2
(e) tI bias6
(f) tl Bibbs
everoged response
30
weeogeed 1630ort2
30
27.5
25.0
20
22.5
24
20.0
22
3.
prad
real
15.0
17.5
Dred tea1
20
Fea) 28
125
0.2
0.4
5.4
5.6
6.8
0.4
0.5 action
0.5
action
action
12_ bias2
(h) t2_ 2_bbi
2_bias8
Figure 21: Illustration of the averaged response curves of Supervised Learning (SL. in TCGA.
46

---

35
pred
55
pred
2veraged resporiee
pred
real
real
real
83
92
averaged
50
30
40
action
(a) el_ el_pl
(b) c0.2_pl
(c) c0.05 pl
averaged response
54
prod
averaged response
aver aged response
60
35
real
real
pic wwwgeeo
55
50
52
45
60
20
50
35
action
action
action
(d) el p0.2
(e) e0.2_p0...... p0.2
(l) e0.05 p0.2
averaged response
pedd
response
55
55
12
averaged
40
86
35
30
-1
action
action
action
(g) el p0.05
c0.2 c0.2_p0.05
(i) e0.05 _0.05_
Figure 22: Illustration of the averaged response curves of Supervised Learning (SL) in GNFC.
47

---

pred
overaged resporse
aweraged respprnse
166
14
32
0.0
0.2
0.4
3.6
0.8
1.0
82
0.4
0.4
0.8
3.0
02
0.4
055
0.7 action
0.6
0.
1.0
action
action
(a) t0 bias
(b) 10 20
(c) 10 bias50
pred
overaged
12
13
13
0.2
0.4
0.5
0.8
01
02
03
0.4
05 06
0.7
38
0.9
0.3
0.5
57
action
action
action
(d) t| bias
(e) tl bias6
(f) tl bias
pred real
30
grod
eversged response
averaged resparse
aweraged
18
16
0.0
5.2
0.4
0.6
3.8
0.1
0.2
03
044 0.5 action
06
077
0.8
0.9
0.2
0.5
3.4
0.5
0.6
0.7
0.8
action
action
(g) 12_ bias2
(h) t2_ biasó
(i) 12_ bias8
Figure 23: Illustration of the averaged response curves of Inverse Propensity Weighting (IPW) in TCGA.
48

---

56
pred
real
averoged response
avernged resporce
24
1.4
53
-22
51
-00
30
action
action
action
(a) el_pl
(b) e0.2_pl
(c) c0.05_p _pp
pred
pred real
overaged resporoe
real
avarrgeed response
122
averaged
-1
action
action
action
(d) el ___2..2
(e) (0.2
(f) e0.05_ _
pred
pred
averaged resporse
averaged resporse
-4
45
-2
30
35
actionn
action
action
(g) el_p0.05
(h) e0.2_p0.05
(i) e0.05_p0.00
Figure 24: Illustration of the averaged response curves of Inverse Propensity Weighting (IPW) in GNFC.
49

---

15
34
12
everrgee
31
0.0
0.2
0.4
0.6
0.8
1.5
0.2
0.4
0.6
0.8
1.0
02
0.3
0.4
0.5
5.6 action
0.7
0.8
0.9
10
action
action
(a) t0 bias
(h) 20
(c) t0 hias50
averaiee resp0155
3.6
27
15
26
32
13
averaged
prod
powd
pred
rea
actioon
(d) tl bias2
(c) tl_ [1_biaó
(f) tl fi_bias
Aversee responne
20.0
prod
30
255
27.5
rual
22.5
20.0
Ivereged
gred
15.0
125
0.0
0.2
0.4
0.6
1.0
0.2
0.4
9.6
0.8
2.5 action
Lio
action
(g) 12_ bias2
(h) 12_ biasó
(i) 12 bias8
Figure 25: Illustration of the averaged response curves of SCIGAN in TCGA.
50

---

pred
averaged response
pre real
real
real
avera
averaged
80
50
action
action
action
(a) el pl
(b) c0.2 pl
(c) e0.05 pl
58
peed
prd real
real
54
56
action
(d) cl p_p22.2
(c) c0.2_p0.2
(f) c0.05 c0.05_pp..
avere perogee responge
94
response
ged ressoonse
52
40
real
real
action
action
action
(g) cl_p0.05
c0.2_p0.05
c0.05_p0.05
Figure 26: Illustration of the averaged response of SCIGAN in GNFC.
51

---

M respporion
64
086
1.0
1.8
10
0.5
action
0.7
0.8
38
1.0
bias
bias20
(c) bias50
averoged resporise
rea1
DVC aoraged rergense
0.3
6-4
action
0.6
0.8
01
32
91
81
action
(d) bias2
(e) bias6
(I) bias8
20
pred real
28
20
0.5 action
0.7
action
action
(g)
bias2
(h) bias6
12 bias8
Figure 27: Illustration of the averaged response of GALILEO in TCGA.
52

---

pree real
prod
144
overagee respornse
averaged respoonse
averaged
822
52
53
action
action
action
(a) cl_pl
(b) e0. c0.2_pl
(c) c0.05 pl
pred
real
64
eepeeeee resporoee
averoged responsee
averaged
52
53
action
action
action
(d) el_ el_p0.2
(e) _0.2_p0.2
(f) e0.05 _
prec
averaged response
53
aversged
01
-1
action
(g) el p0.05
(h) e0.2 p0.05
(i) c0.05_p0.00
Figure 28: Illustration of the averaged response curves of GALILEO in GNFC.
53


Reviews:
[{"review_id": "d0MMIhV1xC", "reviewer": "Reviewer_E1XE", "summary": "This paper introduces an adversarial training approach to model learning that improves performance for counterfactual data that may differ widely from the data used to train the model. This is particularly relevant when training from offline data and expecting the model to generalize when deployed later. This paper extensively presents an adversarially weighted empirical risk minimization objective drawing inspiration from inverse propensity scoring/weighting. A practical algorithm, GALILEO, is introduced and extensively tested across synthetic, continuous control, and real-world data. The paper clearly lays out the advantages of unbiased and accurate counterfactual models in a wide array of use cases in RL. The authors do this to contrast to major limitations of a majority of model-learning approaches that use supervised learning to perform empirical risk minimization.\n\nThis is a complete and well written paper. The development of the proposed method are clearly justified with sufficient grounding in the formal exposition of the equations. The derivation of AWRM was easy to follow with the structure put in place by the authors.\n\nThe included experiments clearly lay out the intended contribution of the proposed approach, that GALILEO provides a more accurate and counterfactually correct model. Impressively, this improved model is shown to have clear benefits for downstream performance in tasks beyond the “pre-training” task.", "questions": "For the MuJoCo experiments it is unclear between Section 5.1 and 5.2 what the set-up is. It appears that the models are trained with the “medium” datasets but are evaluated on “expert” and “medium-replay” datasets. Is this correct?\n\nFor Figure 5b and 5c, what is meant by “update steps” on the horizontal axis? Is this an evaluation of model performance on the test data while training on the training data? This should be made more clear. I was at first inclined to believe that the models were being fine-tuned on the test data…\n\nTitle for Section 5.3 has typo: “Ddownstream”\n\nI was curious why Causal Curiosity (Sontakke, et al 2021ICML) wasn’t used as a baseline? The paper is included among the References but there is no mention of this work in the Related Work section. I would imagine that it would be a relevant baseline to a counterfactually driven model learning approach like GALILEO.\n\n> Sontakke, Sumedh A., et al. \"Causal curiosity: Rl agents discovering self-supervised experiments for causal representation learning.\" International conference on machine learning. PMLR, 2021.\n", "limitations": "As stated by the authors, there are several simplications to the modeling process which may be a cause for the deviation in GALILEO performance when applied to downstream tasks. \n\nAdditionally, as mentioned in the “Weaknesses” section, there is an assumption that counterfactual modeling (on action sets outside the support of the dataset) is admissible. This may eliminate the use of GALILEO among safety critical domains, which would be a majority of real-world settings where model-learning could be useful.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper clearly lays out the advantages of unbiased and accurate counterfactual models in a wide array of use cases in RL. The authors do this to contrast to major limitations of a majority of model-learning approaches that use supervised learning to perform empirical risk minimization.\n\nThis is a complete and well written paper. The development of the proposed method are clearly justified with sufficient grounding in the formal exposition of the equations. The derivation of AWRM was easy to follow with the structure put in place by the authors.\n\nThe included experiments clearly lay out the intended contribution of the proposed approach, that GALILEO provides a more accurate and counterfactually correct model. Impressively, this improved model is shown to have clear benefits for downstream performance in tasks beyond the “pre-training” task.", "weaknesses": "I’m not sure I agree that having real-world offline data being biased is a catastrophic problem. Obviously for model-learning it creates major difficulties but perhaps we can focus on identifying better approaches to using expert data than expecting an RL agent to “figure it out” when there aren’t reliable demonstrations of counterfactuals. There’s likely a good reason why alternative or “exploratory” behaviors are not represented in the data? See Fatemi, et al (2021; NeurIPS) and Killian, et al (2023; TMLR) for a formulation of how we may more adequately think about risk and decision making in such environments.\n\nThe spacing between paragraphs and definitions+equations is really tight. This makes the paper difficult to read. I understand that this was likely a response to the NeurIPS template and page limitations but it should be fixed.\n\nWhile the downstream performance of MBRL methods using GALILEO models is promising. I wish that more analysis was done in the learning dynamics + performance in these downstream tasks regarding sample efficiency and deviations from the behavioral data / optimal policies in these environments. Do the GALILEO agents have predetermined action sequences that they exploit early on or are they flexible to the change in domain/task?\n\n>Fatemi, Mehdi, et al. \"Medical dead-ends and learning to identify high-risk states and treatments.\" Advances in Neural Information Processing Systems 34 (2021): 4856-4870\n\n>Killian, Taylor W., Sonali Parbhoo, and Marzyeh Ghassemi. \"Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning.\" Transactions on Machine Learning Research (2022).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the  detailed feedback on our paper. Your constructive comments are greatly appreciated. We would like to address the concerns and questions you raised as follows:\n\n1. **Concern about Offline Data Bias and other potential solution in Fatemi, et al [1] and Killian, et al [2]**\n\n   We genuinely appreciate the meaningful references you shared.  We believe that  these two solutions are suitable for different scenarios. They [1,2] focus on dead-end discovery, intend to prevent agents from navigating towards potential terminal states, making them highly apt for risk-sensitive scenarios with explicit definitions, such as medical contexts. However, in some other tasks, there isn't a clear dead-end region. The profits or rewards of the tasks are continuous to the action space. To develop an effective policy, we inevitably have to handle the negative influence of biased data on dynamics model predictions, catering to a different set of applications and scenarios between these works. \n\n   Take the task BAT we've explored as an example,  in such tasks, for any order, raising the allowance will increases its acceptance intention. The optimization focus lies in devising a budget allocation policy to maximize the system's overall order intent. In this scenario, no order is truly a \"dead end\"; there are only differences in the speed of acceptance. To develop an effective policy, we have to learn an accurate dynamics model to find an accurately  allocate budgets among different targets.\n\nIn the revised version, we will add these related studies to Appendix F.\n\n2. **Learning Dynamics and Performance Analysis:**  \n\n   Thank you for the suggestion. To provide a more comprehensive view, we will include supplemental results in the revision, including\n\n   (1) The return curve of the algorithms in dynamics models and the testing environments;\n\n   (2) Visualizations of the trajectories of asymptotic policies learned by different models.\n\n   We summarize our finding as follows:\n\n   1. The RL algorithm requires roughly 4e5 samples to find the optimal policies in the dynamics models learned by these algorithms. However,  after policy learning in the dynamics models within about 0.5e5 samples, the policies' performances evaluated the deployment environments will be stable.\n   2. Interestingly, policies trained via IPW and SL either remain standing or fall backward in all three environments, whereas those from GALILEO and SCIGAN demonstrate forward movement in Walker2d and Hopper, and keep standing in HalfCheetah.\n\n\n3. **Omission of Causal Curiosity in the Comparison:**\n\n We did mention this work in Appendix F, recognizing its value in another thread of applying causal inference techniques to RL. In these studies, the researchers consider that the transition function is relevant to some hidden noise variables, where the concept of causal factors in [3] is a special instance of the hidden noise variable. These studies focus on reconstructing the representation of the noise variable, or discovering and estimating the effects of the noise variables.   Our studies and the previous studies using IPS focused on handling the unbiased causal effect estimation problem of actions in the offline dataset under behavior policies collected with selection bias. In this branch of studies, we only consider the environment model learning problem in the fully observed setting thus the **hidden noise variable does not exist**.\n\nIn the revised version, we will split the references into the  main-body part and the appendix part. We then will leave this citation to the appendix reference.\n\n4. **Minor Typos and tight space**\n\n   Thank you for highlighting this. We commit to correcting these in the revised version for a more reader-friendly presentation.\n\n5. **Questions Regarding MuJoCo Experiments:**\n\n   - **Q1: do the GALILEO agents have predetermined action sequences that they exploit early on or are they flexible to the change in domain/task?**\n\n     No. All agents don't employ any predetermined action sequences prior to deployment.\n\n   - **Q2: It appears that the models are trained with the “medium” datasets but are evaluated on “expert” and “medium-replay” datasets. Is this correct?** \n\n     The understanding are correct that models are trained on \"medium\" datasets and assessed on \"expert\" and \"medium-replay\" datasets. We will clarify this in the revision.\n\n   - **Q3: For Figure 5b and 5c, what is meant by “update steps” on the horizontal axis? Is this an evaluation of model performance on the test data while training on the training data? This should be made more clear. I was at first inclined to believe that the models were being fine-tuned on the test data…** \n\n     The observation about \"update steps\" is accurate. We load all datasets, train the model in the medium dataset, and periodically evaluate performance in other datasets. We will elucidate this further in the revised manuscript.\n\n6. **Limitation on Safety-critical Domains:**\n\n   We agree with the observation that in highly safety-critical domains where no risks are allowed, the mentioned works [1,2] might be more appropriate. GALILEO, on the other hand, can be applicable in contexts where safety is essential but certain types of risks are permissible. We believe this distinction clarifies GALILEO's positioning and application scope in the field. We will leave this discussion in Section 6.\n\n   \n\nIn conclusion, your thoughtful feedback has guided us in identifying areas for enhancement and clarification. We believe that these revisions will significantly improve our paper and more effectively convey the contributions and applicability of GALILEO.\n\n\n\n[1] Medical dead-ends and learning to identify high-risk states and treatments.\n\n[2] Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning.\n\n[3] Causal curiosity: Rl agents discovering self-supervised experiments for causal representation learning."}, {"review_id": "yD2TiNUvHy", "reviewer": "Reviewer_So7i", "summary": "The paper proposes a model-learning approach for counterfactual prediction (CP), off-policy evaluation (OPE), and offline reinforcement learning (ORL). The authors introduce the adversarial weighted empirical risk minimization (AWRM) objective to facilitate learning models that accurately evaluate target policies. Additionally, they present the GALILEO algorithm, a generative adversarial training method that approximates the data distribution induced by the optimal adversarial policy.\n", "questions": "In Table 1, it is stated that the returns are normalized to lie between 0 and 100; however, there are negative values in the HalfCheetah column. Is this a typo?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The paper effectively addresses the problem of learning accurate models for CP, OPE, and ORL, which is particularly significant in domains with costly data collection.\n+ The authors provide a comprehensive discussion on the impact of selection bias on CP, which serves as motivation for their objective, AWRM, and the GALILEO algorithm. ", "weaknesses": "- However, it is unclear how novel is the weighted version of empirical risk minimization (ERM), compared to prior research. The main contribution lies in the adversarial aspect. Therefore, I recommend revising the introduction to emphasize and motivate the adversarial part.\n\n- The derivations appear reasonable. However, including a brief discussion about Algorithm 1 could enhance the paper's readability. \n\n- While the experimental results are generally ok, one notable limitation is that the authors only consider three tasks from the D4RL and DOPE benchmarks. In addition, the proposed approach does not outperform other methods in one of these tasks (HalfCheetah). To convincingly demonstrate the efficacy of their algorithm, the paper should include more tasks in their evaluation. Additionally, the constraint of limiting the time horizon to {10, 20, 40} is very strong and lacks proper motivation. \n\n- Furthermore, the proposed algorithm lacks comparison with state-of-the-art algorithms for the D4RL benchmark, such as the one mentioned in https://openreview.net/pdf?id=VYYf6S67pQc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort you have put into reviewing our paper. We appreciate your feedback and would like to address the concerns you've raised as follows:\n\n1. **Experimental Evaluation**:\n\n   1. *Limiting the Time Horizon to {10, 20, 40}*: \n\n      As detailed in Line 328, our motivation for this choice,  was to *rigorously* examine the models' capabilities. By excluding certain tricks that constrain policy exploration and risky region learning which are commonly done in offline model-based RL algorithms like MOPO, we can fully exploit the learned models using standard RL algorithm. However, we observed a large compounding error in the 1,000-step rollout in this setting, making all algorithms fails in learning a reasonable policy. To better verify the effects of models  on policy improvement, we opted for smaller horizons of {10, 20, 40}. We found that after learning in 40-horizon setting, there are significant gaps between the policy learned in real environment and dynamics model, thus we do not further scale up the horizons.\n\n   2. *Choice of Tasks from the D4RL and DOPE Benchmarks. To convincingly demonstrate the efficacy of their algorithm, the paper should include more tasks in their evaluation.*\n\n      As mentioned in Line 263, we opted for the three `medium` datasets in D4RL and DOPE explicitly because these are the only datasets within the benchmark exhibiting selection bias. Other datasets in the benchmark were amassed without an evident selection bias due to the utilization of mixed policies during data collection.\n\n      In addition, we have also incorporated two synthetic benchmarks comprising 18 tasks and a real-world application to demonstrate the effectiveness of our proposed method. **We believe these comprehensive experiments significantly validate our approach, and we hope they aren’t overlooked.**\n\n   3. *Performance on the HalfCheetah Task*:\n\n      As elucidated in Lines 334 to 341, we found certain challenges unique to the HalfCheetah dataset that led to the observed results. All the policies derived, including ours, had the cheetah either standing stationary or moving backward, which means that all policies actually fail in complete the task, even though IPW reaches a bit better performance.\n\n   4. *Comparison with State-of-the-Art Algorithms:* \n\n      We recognize the importance of benchmarking against state-of-the-art methods. Due to the distinct settings in our experiments using D4RL, direct comparisons are challenging. However, we acknowledge the related work highlighted by the reviewer and commit to incorporating it into Appendix F.\n\n   We understand the author's concerns about missing evaluation in standard offlineRL benchmarks. **We would like to emphasize that our experiment is to evaluate the effects of selection bias, and the ability of learned models' predictions in counterfactual actions, instead of purely pursuing the SOTA policy performance in offlineRL benchmarks.** We will revised the paper in the experiment section to clarify this point.\n\n2. **Concern with Table 1**:  In Table 1, only the column `avg. norm.` uses the normalized return, all other scores is the raw returns.\n\n3. **Regarding the Novelty of the WERM and AWRM**: Thanks for the suggestion. We agree that there may have been some confusion regarding the emphasis on the weighted version of ERM (WERM). Our primary contribution is the adversarial objective AWRM. The introduction of the weighted version of ERM served mainly as a scaffold to lead into the discussion of AWRM, ensuring a smoother narrative flow.\n\n   We will revise the introduction section to stress the importance and novelty of the adversarial aspect of our contribution.\n\n4. **Discussion about Algorithm 1 could enhance the paper's readability:** Thanks for the valuable suggestion, we will revise the last paragraph of Section 4.3 to improve the readability of the implementation  of algorithm 1.\n\nIn closing, we are grateful for your insights and will undertake necessary revisions to address the mentioned concerns. We believe that our contributions offer a valuable perspective in the domain of reinforcement learning and hope that the clarifications provided here assuage any reservations."}, {"review_id": "YDivgwz2VB", "reviewer": "Reviewer_h4bm", "summary": "This paper considers the problem of environment modeling. An adversarial method is proposed that the adversarial counterparts is trained to exploit the model to generate a data distribution that weakens the model’s prediction accuracy, and then the model is trained under the adversarial data distribution with a weighted empirical risk minimization objective. Experiments are conducted on synthetic, control and real-world tasks.", "questions": "1. It is known that training GAN-like methods suffer from instability and often the hyperparameters should be carefully tuned. The proposed method involving training two discriminators and I am worried about the stability of the optimization step in Eq. (6) if it is easy to rise collapse problem.\n\n2. It is mentioned that in HalfCheetah, all policies just keep the cheetah standing. I think similar problems might also exist for other tasks, while they are not shown up since reward curves or numbers are compared more often without accessing how well the policies really do in these tasks. Despite this, could any other intuitive information on the performance of the environment modeling be conducted considering a specific task, instead of focusing on the rewards only? For example, in halfcheetah, which transitions are more likely to have a larger discrepancy?\n\nMinor:\n\nTitle of Section 5.3, ‘Ddownstream’ -> ‘downstream’\n\n", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written. The illustration examples are clear and well motivate the problem.\nApplying IPS and the surrogate optimization step is interesting.\nI also agree that conservative or pessimistic offline model-based RL methods often try to limit policy exploration, which might make it hard to obtain an accurate environment model.\n", "weaknesses": "Complexity and stability of the adversarial method might be concerned, where two discriminators should be learned.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for your thoughtful feedback on our submission.  We've taken your concerns to heart and have attempted to address them in the following manner.\n\n**Addressing Stability Concerns of GALILEO:**\n\n1. **Instability of training two discriminators:** The reviewer are right in observing that the GALILEO, much like other GAN-style algorithms, can be challenging to tune. Primarily, the challenge lies in striking a balance between the learning rate of the discriminator and the generator, a known issue in the GAN framework. However, we found that the two discriminators setup did not introduce further complications in this respect. In our practice, we just ensure the uniformity between the two discriminators regarding learning rates, frequency, and network structure, then the GALILEO algorithm can learn the model stably.\n2. **Ablation and Implementation Techniques:** We appreciate your concerns about the stability and effectiveness of the introduced techniques. We have discussed crucial techniques for the GALILEO implementation in Appendix E. Further, ablation studies centered on these techniques can be found in Appendix H.3.\n3. **Code Open-sourcing:** To foster transparency and facilitate reproducibility, we will open-source our code, which would share exhaustive details on the algorithm.\n4. **Add to Limitation:** We agree that the instability is an important issue of GAN-style algorithms, we will mention this limitation in Section 6. This is also in our future work to investigate.\n\n**Algorithm Performance Beyond Reward Metrics**\n\nIn light of your comments, we decided to provide a more comprehensive view of GALILEO's efficacy by delving into its sample efficiency and policy behavior for downstream tasks. We will supplemented our results with trajectory visualizations for asymptotic policies obtained from various models. Interestingly, we found that policies trained via IPW and SL either remain standing or fall backward in all three environments, whereas those from GALILEO and SCIGAN demonstrate forward movement in Walker2d and Hopper, and keep standing in HalfCheetah.\n\nIn the revised version, we will add these visualization results to Appendix H.\n\n**Minor Corrections:**\n\nWe acknowledge the typographical error highlighted by you in Section 5.3 and will rectify it in the final version of the paper.\n\n\n\nIn conclusion, we appreciate the time and effort you've invested in reviewing our work. We've endeavored to address the concerns raised, and we hope our explanations and additions provide clarity. We're optimistic that the enhancements and clarifications enhance the overall quality and impact of our contribution."}, {"review_id": "otjlYEfLXA", "reviewer": "Reviewer_wiCk", "summary": "The paper presents a novel method for improving the accuracy of environment dynamics models for counterfactual prediction, off-policy evaluation, and offline reinforcement learning. Currently, these models learn via empirical risk minimization (ERM), which the authors show can lead to failures in counterfactual action prediction due to selection bias during data collection. To address this, the authors introduce adversarial weighted empirical risk minimization (AWRM), where an adversarial policy weakens the model's prediction accuracy to encourage improvement. They implement this approach via an algorithm named GALILEO, which is evaluated on synthetic tasks, continuous-control tasks, and a real-world application. Results show that GALILEO can accurately predict counterfactual actions and improve several downstream tasks.", "questions": "Please see above. Could you please better situate your work and justify the choice of baseline methods?", "limitations": "The authors didn't address the limitations nor broader societal impacts in the paper. But I didn't see any ethical concerns. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The concept of an adversarial weighted empirical risk minimization (AWRM) is a novel idea that brings together ideas from adversarial training and reinforcement learning. The use of an adversarial policy to manipulate the model's data distribution and improve its learning process is an innovative approach.\n2. The paper provides a theoretical foundation for AWRM and shows its implementation through the GALILEO algorithm. The authors also conduct a variety of tests to assess GALILEO's performance, including synthetic tasks, continuous-control tasks, and a real-world delivery platform.\n3. The paper is generally well-written and the use of figures to illustrate key concepts also enhances the clarity of the paper.", "weaknesses": "1. The authors didn't cover the preliminaries and related works adequately. I'm not too familiar with counterfactual modelling techniques, and think the authors didn't present enough to situate their work.\n2. Justifications for choosing the three baselines are missing. Why not choose the more recent GAIL based methods?\n3. Typo: line 320 Downstream\n4. Formatting issue: The upper margins in page 2, 7, 8 and 9 seem too small. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We'd like to genuinely express our gratitude for your thoughtful feedback and the time you've invested in reviewing our work. We've taken your concerns to heart and have attempted to address them as follows.\n\n**1. Contextualizing the Work:**\n\nWe  acknowledge the feedback on the need for a more detailed presentation of the preliminaries and related works. While we did provide information on counterfactual modeling techniques in Appendix F, including previous techniques based on WERM through IPS, and structural causal models for counterfactual inference.\n\nWe understand that our inclusion of counterfactual modeling techniques in Appendix F might not have been sufficiently prominent. To address this, we commit to elaborating on these concepts within the main sections of the paper (potentially in Section 3) to ensure a more comprehensive and upfront presentation for all readers.\n\n**2. Baseline Justifications:** \n\nThanks for the comment. Our choice of baselines were indeed deliberate:\n\n- **SL**: A representation of a standard algorithm for ERM, which is common-used for model learning in current offline model-based RL algorithms.\n- **IPW**: Chosen for its standing as a benchmark in counterfactual modeling for WERM through IPS, it helps elucidate how AWRM objective compares and advances beyond existing techniques.\n- **SCIGAN**:  An adversarial model learning technique that can be regarded as a partial implementation of GALILEO for optimizing AWRM. While both GAIL and SCIGAN can be seen as partial implementations of GALILEO, the results from SCIGAN gave us a clearer benchmark against which we could measure our algorithm's success.\n\nWe opted for SCIGAN over GAIL because it is well-tuned in [1] and has been tested on TCGA benchmarks. We appreciate the point raised about the potential inclusion of GAIL-based methods. We will further clarify our choice of baselines in the paper, ensuring readers understand the rationale behind our selections in the start of Section 5.\n\nWe believe the baseline selection is representative and comprehensive, and we hope the above clarification can solve your concerns. We commit to introduce GAIL as an additional baseline. It's important to note, however, that the conventional GAIL method focuses on policy learning rather than environment model learning. Given the inherent instability of GAIL-style algorithms, we cannot guarantee optimal results within the constrained timeframe of the rebuttal and discussion period. \n\n**Technical Corrections**\n\nWe acknowledge the typographical error and formatting issue. We assure you that these will be rectified in our revised submission, and we'll double-check the document to preempt any similar issues.\n\n\n\nIn wrapping up, we reiterate our gratitude for your invaluable feedback. We believe these insights will undeniably enhance our paper's clarity and depth. Should you have further queries or need additional clarifications on our revisions, we are open to discussions and more than willing to engage.\n\n[1] Estimating the effects of continuous-valued interventions using generative adversarial networks."}, {"review_id": "UV5zn1CeCP", "reviewer": "Reviewer_qQGz", "summary": "An accurate environment dynamics model is crucial for various downstream tasks, such as counterfactual prediction, off-policy evaluation, and offline reinforcement learning. Currently, these models were learned through empirical risk minimization by step-wise fitting of historical transition data. However, we first show that, particularly in the sequential decision-making setting, this approach may catastrophically fail to predict counterfactual action effects due to the selection bias of behavior policies during data collection. To tackle this problem, the authors introduce a novel model-learning objective called adversarial weighted empirical risk minimization (AWRM). AWRM incorporates an adversarial policy that exploits the model to generate a data distribution that weakens the model's prediction accuracy, and subsequently, the model is learned under this adversarial data distribution. They implement a practical algorithm, GALILEO, for AWRM and evaluate it on two synthetic tasks, three continuous-control tasks, and a real-world application. The experiments demonstrate that GALILEO can accurately predict counterfactual actions and improve various downstream tasks, including offline policy evaluation and improvement, as well as online decision-making.", "questions": "1. I am little bit curious to the details of metrics and downstream tasks used in the experiments. For example, the AUUC, value gap, regret@1, and rank correlation metrics and the details behind the off-policy evaluation experiments. Could you please consider adding some corresponding explanations in the future version?", "limitations": "See the Weaknesses and Questions above.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses the problem of accurate environment dynamics model learning, which exhibits wide impacts in many downstream tasks, like counterfactual prediction, off-policy evaluation, and offline reinforcement learning. This is a very important and meaningful research topic.\n\n2. This is the first research on faithful dynamics model learning in sequential decision-making settings like RL, which demonstrates the novelty of this paper.\n\n3. The analysis to the challenges brought by the conventional empirical risk minimization method is deep and insightful. Especially, the authors use a vivid example to illustrate them. Based on this, the transition to the propose of adversarial weighted empirical risk minimization objective is smooth, which strongly supports the following the Generative Adversarial Offline Counterfactual environment model learning (GALILEO) method.\n\n3. The experiments of this work is sufficient and persuasive. The authors conduct experiments in two synthetic tasks, three continuous-control tasks, and a real-world application. They first verify that GALILEO can make accurate predictions on counterfactual data queried by other policies. Then, they demonstrate that the model learned by GALILEO is helpful to several downstream tasks.", "weaknesses": "1. It can be better if the authors can add several baselines to better validate the superiority of the proposed GALILEO method.\n\n2. The description of the method part is deep and comprehensive. But the authors can consider making it easier to understand if possible. \n\n3. Some typos need to be fixed in the future version, like the subtitle of Section. 5.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude for your constructive feedback and the time you dedicated to evaluating our paper. Your insightful remarks help us in refining our work and emphasizing its importance in the reinforcement learning domain.\n\n**Clarify Experiments and Metrics:** \n\nWe understand the concerns regarding the metrics and downstream tasks utilized in our experiments. While the main body of the paper had constraints regarding space, we have provided a comprehensive explanation of these metrics and tasks in Appendix H.6.1 and Appendix H.7. To make this information more accessible, we will emphasize this section prominently in the main text and provide a direct link ensuring that readers can easily navigate to it.\n\n**Baselines Justifications:** \n\nThank you for raising the point about the selection of baselines. Our choices were deliberate and comprehensive to verify the algorithm:\n\n- **SL**: A representation of a standard algorithm for ERM, which is common-used for model learning in current offline model-based RL algorithms.\n- **IPW**: Chosen for its standing as a benchmark in counterfactual modeling for WERM through IPS, it helps elucidate how AWRM objective compares and advances beyond existing techniques.\n- **SCIGAN**: SCIGAN is perceived as subsets of GALILEO for optimizing AWRM, using SCIGAN provides a more distinct reference to assess the effectiveness of our approach. It is also a good baseline since its rigorous testing on TCGA benchmarks [1].\n\nIn response, we will provide a more detailed explanation of our baseline choices at the beginning of Section 5 to ensure absolute clarity for our readers.\n\nWe believe the baseline selection is representative and comprehensive, and we hope the above clarification can solve your concerns. We commit to introduce GAIL as an additional baseline. It's important to note, however, that the conventional GAIL method focuses on policy learning rather than environment model learning. Given the inherent instability of GAIL-style algorithms, we cannot guarantee optimal results within the constrained timeframe of the rebuttal and discussion period. \n\n**Method Description:**\n\n We genuinely appreciate your feedback regarding the method's description. Our intention was to provide an exhaustive understanding, which, in hindsight, might have resulted in a dense exposition. In light of your observations, we are inclined to simplify certain sections for better clarity.\n\n\n\nIn conclusion, we sincerely thank you for helping us improve our paper's quality. We believe that addressing these points will make the paper more accessible and insightful for the community. We are optimistic that these changes will elevate our work and make it a valuable contribution to the field."}]

Author rebuttal:
We would like to extend our heartfelt appreciation to the esteemed reviewers for their invaluable feedback and insightful comments. Their constructive input has undoubtedly enhanced the quality of our manuscript. The common strengths highlighted across the reviews are encouraging, and we are grateful for the recognition of our efforts:

**Clarity of Presentation**: Multiple reviewers (E1XE, h4bm, wiCk, qQGz) emphasized that the paper is well-articulated, with clear illustrations and examples. The structure, as highlighted by E1XE, made the derivation of AWRM easy to follow, while the use of figures as pointed out by wiCk added to the paper's clarity.

**Novelty and Significance of Methodology**: The introduction of the adversarial weighted empirical risk minimization (AWRM) was recognized as innovative and novel by reviewers So7i, wiCk, and qQGz. The method effectively addresses significant challenges in the domain, including the problem of selection bias as mentioned by So7i.

**Comprehensive Experiments**: Reviewers E1XE, wiCk, and qQGz lauded the comprehensive experimental evaluation provided, noting the wide range of tasks, including synthetic, continuous control, and real-world applications. Particularly, the experiments showcased that GALILEO can accurately predict counterfactual actions and significantly enhance downstream tasks.

**Depth of Analysis**: The in-depth analysis of the challenges posed by conventional empirical risk minimization methods and the subsequent development of our proposed method was highlighted by qQGz as being insightful and backed by strong rationale.

We sincerely thank the reviewers for recognizing these strengths, and we have diligently addressed other feedback and suggestions to further refine our work. 

Meta-review: All of the reviewers agree that the paper makes significant and novel contributions to learning from data with selection biases. This is a well-motivated problem for off-policy bandit and reinforcement learning and counterfactual prediction in general. During the discussion phase, the authors clarified their choice of baselines, and detailed how they trained the discriminators to address the reviewers' concerns about adversarial training stability. These clarifications substantially strengthen the paper's exposition and should be included in the revision.
Decision: Accept (spotlight)