SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We conduct experiments on three different treatments separately and change the value of bias  $\alpha$  to assess the robustness of different methods to treatment bias. When the bias of treatment is large, which means  $\alpha$  is large, the training set contains data with a strong bias on treatment so it would be difficult for models to appropriately predict the treatment responses out of the distribution of training data.
G.1.3 Budget Allocation task to the Time period (BAT)
Figure 11: Illustration of the workflow of the food-delivery platform.
We deploy GALILEO in a real-world large-scale food-delivery platform. The platform contains various food stores, and food delivery clerks. The overall workflow is as follows: the platform presents the nearby food stores to the customers and the customers make orders, i.e., purchase take-out foods from some stores on the platform. The food delivery clerks can select orders from the platform to fulfill. After an order is selected to fulfill, the delivery clerks will take the ordered take-out foods from the stores and then send the food to the customers. The platform will pay the delivery clerks (mainly in proportion to the distance between the store and the customers' location) once the orders are fulfilled. An illustration of the workflow can be found in Fig. 11.
Figure 12: Illustration of relationship between user feedback and the actions of the offline dataset in the real-world food-delivery platform.
However, there is an imbalance problem between the demanded orders from customers and the supply of delivery clerks to fulfill these orders. For example, at peak times like lunchtime, there will be many more demanded orders than at other times, and the existed delivery clerks might not be able to fulfill all of these orders timely. The goal of the Budget Allocation task to the Time period (BAT) is to handle the imbalance problem in time periods by sending reasonable allowances to different time periods. More precisely, the goal of BAT is to make all orders (i.e., the demand) sent in different time periods can be fulfilled (i.e., the supply) timely.
The core challenge of the environment model learning in BAT tasks is similar to the challenge in Fig. 1. Specifically, the behavior policy in BAT tasks is a human-expert policy, which will tend to increase the budget of allowance in the time periods with a lower supply of delivery clerks, otherwise will decrease the budget (Fig. 12 gives an instance of this phenomenon in the real data).
To handle the imbalance problem in different time periods, in the platform, the orders in different time periods  $t \in [0,1,2\ldots,23]$  will be allocated with different allowances  $c \in \mathcal{N}^+$ . For example, at 10 A.M. (i.e.,  $t = 10$ ), we add  $0.5$  (i.e.,  $c = 0.5$ ) allowances to all of the demanded orders. From 10 A.M. to 11 A.M., the delivery clerks who take orders and send food to customers will receive extra allowances. Specifically, if the platform pays the delivery clerks  $2$  for fulfilling the order, now he/she will receive  $2.5$ . For each day, the budget of allowance  $C$  is fixed. We should find the best budget allocation policy  $\pi^{*}(c|t)$  of the limited budget  $C$  to make as many orders as possible can be taken timely.
To find the policy, we first learn a model to reconstruct the response of allowance for each delivery clerk  $\hat{M}(y_{t+1}|s_t, p_t, c_t)$ , where  $y_{t+1}$  is the taken orders of the delivery clerks in state  $s_t$ ,  $c_t$  is the allowances,  $p_t$  denotes static features of the time period  $t$ . In particular, the state  $s_t$  includes historical order-taken information of the delivery clerks, current orders information, the feature of weather, city information, and so on. Then we use a rule-based mapping function  $f$  to fill the complete next time-period states, i.e.,  $s_{t+1} = f(s_t, p_t, c_t, y_{t+1})$ . Here we define the composition of the above functions  $\hat{M}$  and  $f$  as  $\hat{M}_f$ . Finally, we learn a budget allocation policy based on the learned model. For each day, the policy we would like to find is:
$$
\max  _ {\pi} \mathbb {E} _ {s _ {0} \sim \mathcal {S}} \left[ \sum_ {t = 0} ^ {2 3} y _ {t} | \hat {M} _ {f}, \pi \right],
$$
$$
\text {s . t .}, \sum_ {t, s \in \mathcal {S}} c _ {t} y _ {t} \leq C
$$
In our experiment, we evaluate the degree of balancing between demand and supply by computing the averaged five-minute order-taken rate, that is the percentage of orders picked up within five minutes. Note that the behavior policy is fixed for the long term in this application. So we directly use the data replay with a small scale of noise (See Tab. 6) to reconstruct the behavior policy for model learning in GALILEO.
Also note that although we model the response for each delivery clerk, for fairness, the budget allocation policy is just determining the allowance of each time period  $t$  and keeps the allowance to each delivery clerk  $s$  the same.
G.2 Baseline Algorithms
The algorithm we compared are: (1) Supervised Learning (SL): training a environment model to minimize the expectation of prediction error, without considering the counterfactual risks; (2) inverse propensity weighting (IPW) [50]: a practical way to balance the selection bias by re-weighting. It can be regarded as  $\omega = \frac{1}{\hat{\mu}}$ , where  $\hat{\mu}$  is another model learned to approximate the behavior policy; (3) SCIGAN: a recent proposed adversarial algorithm for model learning for continuous-valued interventions [7]. All of the baselines algorithms are implemented with the same capacity of neural networks (See Tab. 6).
G.2.1 Supervised Learning (SL)
As a baseline, we train a multilayer perceptron model to directly predict the response of different treatments, without considering the counterfactual risks. We use mean square error to estimate the performance of our model so that the loss function can be expressed as  $MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2$ , where  $n$  is the number of samples,  $y$  is the true value of response and  $\hat{y}$  is the predicted response. In practice, we train our SL models using Adam optimizer and the initial learning rate  $3e^{-4}$  on both datasets TCGA and GNFC. The architecture of the neural networks is listed in Tab. 6.
G.2.2 Inverse Propensity Weighting (IPW)
Inverse propensity weighting [50] is an approach where the treatment outcome model uses sample weights to balance the selection bias by re-weighting. The weights are defined as the inverse propensity of actually getting the treatment, which can be expressed as  $\frac{1}{\hat{\mu}(a|x)}$ , where  $x$  stands for the feature vectors in a dataset,  $a$  is the corresponding action and  $\hat{\mu}(a|x)$  indicates the action taken probability of  $a$  given the features  $x$  within the dataset.  $\hat{\mu}$  is learned with standard supervised learning. Standard IPW leads to large weights for the points with small sampling probabilities and finally makes the learning process unstable. We solve the problem by clipping the propensity score:  $\hat{\mu} \gets \min(\hat{\mu}, 0.05)$ , which is common used in existing studies [27]. The loss function can thus be expressed as  $\frac{1}{n} \sum_{i=1}^{n} \frac{1}{\hat{\mu}(a_i|x_i)} (y_i - \hat{y}_i)^2$ . The architecture of the neural networks is listed in Tab. 6.
G.2.3 SCIGAN
SCIGAN [7] is a model that uses generative adversarial networks to learn the data distribution of the counterfactual outcomes and thus generate individualized response curves. SCIGAN does not place any restrictions on the form of the treatment-does response functions and is capable of estimating patient outcomes for multiple treatments, each with an associated parameter. SCIGAN first trains a generator to generate response curves for each sample within the training dataset. The learned generator can then be used to train an inference network using standard supervised methods. For fair comparison, we increase the number of parameters for the open-source version of SCIGAN so that the SCIGAN model can have same order of magnitude of network parameters as GALILEO. In addition, we also finetune the hyperparameters (Tab. 5) of the enlarged SCIGAN to realize its full strength. We set num_dosage_samples 9 and  $\lambda = 10$ .
Table 5: Table of hyper-parameters for SCIGAN.
Parameter	Values
Number of samples	3, 5, 7, 9, 11
λ	0.1, 1, 10, 20
G.3 Hyper-parameters
We list the hyper-parameter of GALILEO in Tab. 6.
G.4 Computation Resources
We use one Tesla V100 PCIe 32GB GPU and a 32-core Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz to train all of our models.
Table 6: Table of hyper-parameters for all of the tasks.
Parameter	GNFC	TAGC	MuJoCo	BAT
hidden layers of all neural networks	4	4	5	5
hidden units of all neural networks	256	256	512	512
collect samples for each time of model update	5000	5000	40000	96000
batch size of discriminators	5000	5000	40000	80000
horizon	50	1	100	48 (half-hours)
εμ(also εD)	0.005	0.01	0.05 (0.1 for walker2d)	0.05
times for discriminator update	2	2	1	5
times for model update	1	1	2	20
times for supervised learning update	1	1	4	20
learning rate for supervised learning	1e-5	1e-5	3e-4	1e-5
γ	0.99	0.0	0.99	0.99
clip-ratio	NAN	NAN	NAN	0.1
max DKL	0.001	0.001	0.001	NAN
optimization algorithm (the first term of Eq. (22))	TRPO	TRPO	TRPO	PPO
(b) TCGA
Figure 13: Illustration of the performance in GNFC and TCGA. The grey bar denotes the standard error ( $\times 0.3$  for brevity) of 3 random seeds.
H Additional Results
H.1 Test in Single-step Environments
The results of GNFC tasks are summarized in Fig. 13(a) and the detailed results can be found in Tab. 11. The results show that the property of the behavior policy (i.e.,  $e$  and  $p$ ) dominates the generalization ability of the baseline algorithms. When  $e = 0.05$ , almost all of the baselines fail and give a completely opposite response curve. IPW still perform well when  $0.2 \leq e \leq 1.0$  but fails when  $e = 0.05, p <= 0.2$ . We also found that SCIGAN can reach a better performance than other baselines when  $e = 0.05, p <= 0.2$ , but the results in other tasks are unstable. GALILEO is the only algorithm that is robust to the selection bias and outputs correct response curves in all of the tasks. Based on the experiment, we also indicate that the commonly used overlap assumption is unreasonable to a certain extent especially in real-world applications since it is impractical to inject noises into the whole action space. The problem of overlap assumption being violated should be taken into consideration otherwise the algorithm will be hard to use in practice if it is sensitive to the noise range.
The results of TCGA tasks are summarized in Fig. 13(b) and the detailed results can be found in Tab. 12. We found the phenomenon in this experiment is similar to the one in GNFC, which demonstrates the compatibility of GALILEO to single-step environments. We also found that the results of IPW are unstable in this experiment. It might be because the behavior policy is modeled with beta distribution while the propensity score  $\hat{\mu}$  is modeled with Gaussian distribution. Since IPW directly reweight loss function with  $\frac{1}{\hat{\mu}}$ , the results are sensitive to the error on  $\hat{\mu}$ . GALILEO also models  $\hat{\mu}$  with Gaussian distribution but the results are more stable since GALILEO does not re-weight through  $\hat{\mu}$  explicitly.
We give the averaged responses for all of the tasks and the algorithms in Fig. 21 to Fig. 28. We randomly select  $20\%$  of the states in the dataset and equidistantly sample actions from the action space for each sampled state, and plot the averaged predicted feedback of each action. The real response is slightly different among different figures as the randomly-selected states for testing is different. We sample 9 points in GNFC tasks and 33 points in TAGC tasks for plotting.
H.2 All of the Result Table
We give the result of CNFC in Tab. 11, TCGA in Tab. 12, BAT in Tab. 9, and MuJoCo in Tab. 10.
H.3 Ablation Studies
In Appx. E.1, we introduce several techniques to develop a practical GALILEO algorithm. Based on task e0.2_p0.05 in GNFC, we give the ablation studies to investigate the effects of these techniques. We first compare two variants that do not handle the assumptions violation problems: (1) NO_INJECT_NOISE: set  $\epsilon_{\mu}$  and  $\epsilon_{D}$  to zero, which makes the overlap assumption not satisfied; (2) SINGLE_SL: without replacing the second term in Eq. (6) with standard supervised learning even when the output probability of  $D$  is far away from 0.5. Besides, we introduced several tricks inspired by GAIL and give a comparison of these tricks and GAIL: (3) ONE_STEP: use one-step reward instead of cumulative rewards (i.e., Q and V; see Eq. (23) and Eq. (24)) for re-weighting, which is implemented by set  $\gamma$  to 0; (4) SINGLE_DIS: remove  $T_{\varphi_1^*}(x,a)$  and replace it with  $\mathbb{E}_{M_\theta}\left[T_{\varphi_0^*}(x,a,x')\right]$ , which is inspired by GAIL that uses a value function as a baseline instead of using another discriminator; (5) PURE_GAIL: remove the
second term in Eq. (6). It can be regarded as a naive adoption of GAIL and a partial implementation of GALILEO.
Figure 14: Illustration of the ablation studies. The error bars are the standard error.
We summarize the results in Fig. 14. Based on the results of NO_INJECT_NOISE and SINGLE_SL, we can see that handling the assumption violation problems is important and will increase the ability on counterfactual queries. The results of PURE_GAIL tell us that the partial implementation of GALILEO is not enough to give stable predictions on counterfactual data; On the other hand, the result of ONE_STEP also demonstrates that embedding the cumulative error of one-step prediction is helpful for GALILEO training; Finally, we also found that SINGLE_DIS nearly has almost no effect on the results. It suggests that, empirically, we can use  $\mathbb{E}_{M_{\theta}}\left[T_{\varphi_0^*}(x,a,x')\right]$  as a replacement for  $T_{\varphi_1^*}(x,a)$ , which can reduce the computation costs of the extra discriminator training.
H.4 Worst-Case Prediction Error
In theory, GALILEO increases the generalization ability by focusing on the worst-case samples' training to achieve AWRM. To demonstrate the property, we propose a new metric named Mean-Max Square Error (MMSE):  $\mathbb{E}\left[\max_{a\in \mathcal{A}}\left(M^{*}(x^{\prime}|x,a) - M(x^{\prime}|x,a)\right)^{2}\right]$  and give the results of MMSE for GNFC in Tab. 13 and for TCGA in Tab. 14.
H.5 Detailed Results in the MuJoCo Tasks
We select 3 environments from D4RL [17] to construct our model learning tasks. We compare it with a typical transition model learning algorithm used in the previous offline model-based RL algorithms [59, 30], which is a variant of standard supervised learning. We name the method OFF-SL. Besides, we also implement IPW and SCIGAN as the baselines. We train models in datasets HalfCheetah-medium, Walker2d-medium, and Hopper-medium, which are collected by a behavior policy with 1/3 performance to the expert policy, then we test them in the corresponding expert dataset. For better training efficiency, the trajectories in the training and testing datasets are truncated, remaining the first 100 steps. We plot the converged results and learning curves of the three MuJoCo tasks in Tab. 10 and Fig. 15 respectively.
(a) halfcheetah-medium (train)
(b) hopper-medium (train)
(c) walker2d-medium (train)
(d) halfcheetah-medium-replay (test)
(e) hopper-medium-replay (test)
(f) walker2d-medium-replay (test)
(g) halfcheetah-expert (test)
(h) hopper-expert (test)
(i) walker2d-expert (test)
Figure 15: Illustration of learning curves of the MuJoCo Tasks. The X-axis record the steps of the environment model update, and the Y-axis is the corresponding prediction error. The figures with titles ending in "(train)" means the dataset is used for training while the titles ending in "(test)" means the dataset is just used for testing. The solid curves are the mean reward and the shadow is the standard error of three seeds.
In Fig. 15, we can see that all algorithms perform well in the training datasets. OFF-SL and SCIGAN can even reach a bit lower error in halfcheetah and walker2d. However, when we verify the models through "expert" and "medium-replay" datasets, which are collected by other policies, the performance of GALILEO is significantly more stable and better than all other algorithms. As the training continues, the baseline algorithms even gets worse and worse. However, whether in GALILEO or other baselines, the performance for testing is at least  $2x$  worse than in the training
dataset, and the error is large especially in halfcheetah. The phenomenon indicates that although GALILEO can make better performances for counterfactual queries, the risks of using the models are still large and still challenging to be further solved.
Table 7: Results of policy performance directly optimized through SAC [20] using the learned dynamics models and deployed in MuJoCo environments. MAX-RETURN is the policy performance of SAC in the MuJoCo environments, and "avg. norm." is the averaged normalized return of the policies in the 9 tasks, where the returns are normalized to lie between 0 and 100, where a score of 0 corresponds to the worst policy, and 100 corresponds to MAX-RETURN.
Task	Hopper	Walker2d	HalfCheetah	avg. norm.
Horizon	H=10	H=20	H=40	H=10	H=20	H=40	H=10	H=20	H=40	/
GALILEO	13.0 ± 0.1	33.2 ± 0.1	53.5 ± 1.2	11.7 ± 0.2	29.9 ± 0.3	61.2 ± 3.4	0.7 ± 0.2	-1.1 ± 0.2	-14.2 ± 1.4	51.1
OFF-SL	4.8 ± 0.5	3.0 ± 0.2	4.6 ± 0.2	10.7 ± 0.2	20.1 ± 0.8	37.5 ± 6.7	0.4 ± 0.5	-1.1 ± 0.6	-13.2 ± 0.3	21.1
IPW	5.9 ± 0.7	4.1 ± 0.6	5.9 ± 0.2	4.7 ± 1.1	2.8 ± 3.9	14.5 ± 1.4	1.6 ± 0.2	0.5 ± 0.8	-11.3 ± 0.9	19.7
SCIGAN	12.7 ± 0.1	29.2 ± 0.6	46.2 ± 5.2	8.4 ± 0.5	9.1 ± 1.7	1.0 ± 5.8	1.2 ± 0.3	-0.3 ± 1.0	-11.4 ± 0.3	41.8
MAX-RETURN	13.2 ± 0.0	33.3 ± 0.2	71.0 ± 0.5	14.9 ± 1.3	60.7 ± 11.1	221.1 ± 8.9	2.6 ± 0.1	13.3 ± 1.1	49.1 ± 2.3	100.0
We then verify the generalization ability of the learned models above by adopting them into offline model-based RL. Instead of designing sophisticated tricks to suppress policy exploration and learning in risky regions as current offline model-based RL algorithms [59, 30] do, we just use the standard SAC algorithm [20] to exploit the models for policy learning to strictly verify the ability of the models. Unfortunately, we found that the compounding error will still be inevitably large in the 1,000-step rollout, which is the standard horizon in MuJoCo tasks, leading all models to fail to derive a reasonable policy. To better verify the effects of models on policy optimization, we learn and evaluate the policies with three smaller horizons:  $H \in \{10, 20, 40\}$ .
The results have been listed in Tab. 7, where the learning curve in the dynamics models and the real environments is shown in Fig. 16 and Fig. 17. We first averaged the normalized return (refer to "avg. norm.") under each task, and we can see that the policy obtained by GALILEO is significantly higher than other models (the improvements are  $24\%$  to  $161\%$ ). At the same time, we found that SCIGAN performed better in policy learning, while IPW performed similarly to SL. This is in line with our expectations, since IPW only considers the uniform policy as the target policy for debiasing, while policy optimization requires querying a wide variety of policies. Minimizing the prediction risks only under a uniform policy cannot yield a good environment model for policy optimization. On the other hand, SCIGAN, as a partial implementation of GALILEO (refer to Appx. E.2), also roughly achieves AWRM and considers the cumulative effects of policy on the state distribution, so its overall performance is better; In addition, we find that GALILEO achieves significant improvement in 6 of the 9 tasks. But in HalfCheetah, IPW works slightly better. However, compared with MAX-RETURN, it can be found that all methods fail to derive reasonable policies because their policies' performances are far away from the optimal policy. By further checking the trajectories, we found that all the learned policies just keep the cheetah standing in the same place or even going backward and fall down<sup>4</sup>.
H.6 Off-policy Evaluation (OPE) in the MuJoCo Tasks
H.6.1 Training and Evaluation Settings
We select 3 environments from D4RL [17] to construct our model learning tasks as Appx. H.5. To match the experiment setting in DOPE [18], here we use the whole datasets to the train GALILEO model, instead of truncated dataset in Appx. H.5, for GALILEO model training.
OPE via a learned dynamics model is straightforward, which only needs to compute the return using simulated trajectories generated by the evaluated policy under the learned dynamics model. Due to the stochasticity in the model and the policy, we estimate the return for a policy with Monte-Carlo sampling. See Alg. 4 for pseudocode, where we use  $\gamma = 0.995$ ,  $N = 10$ ,  $H = 1000$  for all of the tasks.
H.6.2 Metrics
The metrics we use in our paper are defined as follows:
4the videos can be found in https://github.com/xionghuichen/GALILEO
(a) hopper  $H = 10$
(b) hopper  $H = 20$
(c) hopper  $H = 40$
(d) halfcheetah  $H = 10$
(e) halfcheetah  $H = 20$
(f) halfcheetah  $H = 40$
(g) halfcheetah  $H = 10$
(h) halfcheetah  $H = 20$
(i) halfcheetah  $H = 40$
Figure 16: Illustration of offline policy learning curves of the MuJoCo Tasks. The X-axis record the steps of the environment model update, and the Y-axis is the corresponding returns in the dynamics models. The solid curves are the mean reward and the shadow is the standard error of three seeds.
Absolute Error The absolute error is defined as the difference between the value and estimated value of a policy:
$$
\operatorname {A b s E r r} = \left| V ^ {\pi} - \hat {V} ^ {\pi} \right|, \tag {28}
$$
where  $V^{\pi}$  is the true value of the policy and  $\hat{V}^{\pi}$  is the estimated value of the policy.
Rank correlation Rank correlation measures the correlation between the ordinal rankings of the value estimates and the true values, which can be written as:
$$
\operatorname {R a n k C o r r} = \frac {\operatorname {C o v} \left(V _ {1 : N} ^ {\pi} , \hat {V} _ {1 : N} ^ {\pi}\right)}{\sigma \left(V _ {1 : N} ^ {\pi}\right) \sigma \left(\hat {V} _ {1 : N} ^ {\pi}\right)}, \tag {29}
$$
where  $1:N$  denotes the indices of the evaluated policies.
(i) halfcheetah  $H = 40$
Figure 17: Illustration of offline policy learning curves of the MuJoCo Tasks. The X-axis record the steps of the environment model update, and the Y-axis is the corresponding returns in the ground-truth environments. The solid curves are the mean reward and the shadow is the standard error of three seeds.
Regret@k Regret@k is the difference between the value of the best policy in the entire set, and the value of the best policy in the top-k set (where the top-k set is chosen by estimated values). It can be defined as:
$$
\operatorname {R e g r e t} @ k = \max  _ {i \in 1: N} V _ {i} ^ {\pi} - \max  _ {j \in \operatorname {t o p k} (1: N)} V _ {j} ^ {\pi}, \tag {30}
$$
where  $\mathrm{topk}(1:N)$  denotes the indices of the top K policies as measured by estimated values  $\hat{V}^{\pi}$ .
Algorithm 4 Off-policy Evaluation with GALILEO model
Require: GALILEO model  $(M_{\theta})$  , evaluated policy  $\pi$  , number of rollouts  $N$  . set of initial states   
 $S_0$  , discount factor  $\gamma$  , horizon length  $H$    
for  $i = 1$  to  $N$  do   
 $R_{i} = 0$    
Sample initial state  $s_0\sim S_0$    
Initialize  $\tau_{-1} = 0$    
for  $t = 0$  to  $H - 1$  do   
 $a_{t}\sim \pi (\cdot |s_{t})$ $s_{t + 1},r_t\sim M_\theta (\cdot |s_t,a_t)$ $R_{i} = R_{i} + \gamma^{t}r_{t}$    
end for   
end for   
return  $\frac{1}{N}\sum_{i = 1}^{N}R_{i}$
H.6.3 Detailed Results