On Trajectory Augmentations for Off-Policy Evaluation
Authors: Ge Gao
Qitong Gao
Xi Yang
Song Ju
Miroslav Pajic
Min Chi
Venue: ICLR 2024 poster (2024)
Keywords: Trajectory augmentation
Off-policy evaluation
Sub-trajectory mining from offline dataset
Primary area: reinforcement learning

Abstract: In the realm of reinforcement learning (RL), off-policy evaluation (OPE) holds a pivotal position, especially in high-stake human-involved scenarios such as e-learning and healthcare. Applying OPE to these domains is often challenging with scarce and underrepresentative offline training trajectories. Data augmentation has been a successful technique to enrich training data. However, directly employing existing data augmentation methods to OPE may not be feasible, due to the Markovian nature within the offline trajectories and the desire for generalizability across diverse target policies. In this work, we propose an offline trajectory augmentation approach to specifically facilitate OPE in human-involved scenarios. We propose sub-trajectory mining to extract potentially valuable sub-trajectories from offline data, and diversify the behaviors within those sub-trajectories by varying coverage of the state-action space. Our work was empirically evaluated in a wide array of environments, encompassing both simulated scenarios and real-world domains like robotic control, healthcare, and e-learning, where the training trajectories include varying levels of coverage of the state-action space. By enhancing the performance of a variety of OPE methods, our work offers a promising path forward for tackling OPE challenges in situations where data may be limited or underrepresentative.

Published as a conference paper at ICLR 2024
ON TRAJECTORY AUGMENTATIONS FOR OFF-POLICY EVALUATION
Ge Gao"
Qitong Gaot
Xi Yang
Song Ju
Miroslav Pajic
Min Chi
ABSTRACT
In the realm of reinforcement learning (RL), off-policy evaluation (OPE) holds a pivotal position, especially in high-stake human-centric scenarios such as learning and healthcare. Applying OPE these domains often challenging with scarce and undernepresentative offline training trajectories. Data augmentation has been a successful technique to enrich training data. However, directly employing existing data augmentation methods to OPE may not be feasible, due to the Marko- vian nature within the offline trajectories and the desire for generalizability across diverse target policies. In this work, we propose an offline trajectory augmentation approach, named OAT, to specifically facilitate OPE in human-involved scenarios. We propose sub-trajectory mining extract potentially valuable sub-trajectories from offline data, and diversify the behaviors within those sub-trajectories by vary- ing coverage of the state-action space. Our work was empirically evaluated in a wide array of environments, encompassing both simulated scenarios and real- world domains like robotic control, healthcare, and e-learning, where the training trajectories include varying levels of coverage of the state-action space. By en- hancing the performance of variety of OPE methods, our work offers a promis- ing path forward for tackling OPE challenges in situations where human-centric data may be limited or underrepresentattve..
INTRODUCTION
Off-policy evaluation (OPE) has been recognized as an important part of reinforcement learning (RL), especially for human-involved RLs (Wu al., 2022; Gao et al. 2023a b(c). in which eval- uations of online policies can have high stakes (Levine et al. 2020). The objective of OPE is evaluate target policies based on offline trajectories collected from behavioral policies differ- ent from the target ones. One major barrier often lies in the fact that the offline trajectories in human-involved tasks often only provide limited coverage of the entire state-action space (Chang et al. 2021; Schweighhofrr et al., 2021). This can be caused by homogeneous behavioral policies; example, during clinical procedures, physicians need to follow certain standardized guidelines. However, a sub-optimal autonomous control agent (e.g., surgical robots under training) may deviate from such guidelines, and thus result in trajectories where the state-action space may not be fully covered by the offline trajectories collected, which introduces great challenges for OPE, as illus- trated in Figure Therefore, to improve the OPE performance, is essential to enrich the offline trajectories.
Data augmentation is powerful tool for data enrichment by artificially generating new data points from existing data. It has shown effectiveness in facilitating learning more robust supervised and unsupervised models (Iwana & Uchida 2021a: Xie et al. 2020). Specifically, generative methods such variational autoencoder (VAE) have achieved superior performance in time-series augmen- tation (Yoon et al.] 2019 Barak et al. 2022). However, an important characteristic of OPE training data is the Markovian nature, as the environments are usually formulated as a Markov decision pro- cess (MDP) (Thomas & Brunskill 2016 Fu et al. (02T). As a result, prior works on time-series augmentation may not be directly applicable to MDP trajectory augmentation. Recently, though data augmentation methods have been extended to facilitate RL policy optimization, most existing
"North Carolina State University, USA. Emails: (ggao5, mchi mchi)@ncsul edu.
I Duke University, USA. Emails: (qitong gao, miroslav. pajiccaaaacc............ 8dckee 1BM Research, USA. Email: yang@ ilm..0ib.mmmmm.mmmmmm.... 1.mm.co.. com.

---

Published as a conference paper at ICLR 2024
works focus enriching the state space, such as adding noise to input images to generate sufficient data and improve the generality of agents (Laskin et al. 2020a Raileanu et al., 202I), but over- look the coverage of the joint state-action distribution over time. More importantly, the goal of data augmentation towards OPE is different from RI. policy optimization. Data augmentation in RI. gen- erally aims quickly facilitate identifying and learning from high-reward regions of the state-action space (Liu et al. 2021; Park et al., (0022). In contrast, the evaluation policies considered by OPE can be heterogeneous and lead varied performance, i.e., the policies be evaluated by OPE do not necessarily perform well; therefore, it is equally important to allow the agent learning from tra- jectories resulted from high- and low-reward regions. As a result, OPE methods prefer training data that provides comprehensive coverage of the state-action space, including the trajectories resulting from low-performing and sub-optimal policies. To the best of our knowledge, there does not exist a method that augments historical trajectories specific to OPE.
In this paper, we propose a framework to facilitate OPE using Augmented Trajectories (OAT). Specifically, moti- vated by the intrinsic nature that human-involved systems (HIS) are often provided limited coverage of the state- action space, while human may behave diversely when following different policies (Yang al., [020bb Wang et al.] 2022). we propose potential sub-trajectories (PSTs) mining to identify uub-trajectories of historical trajecto- ries whose statc-action space is less covered but have great potential to enrich the space. Then generative modeling framework is used to capture the dynamic un- derlying the PSTs and induce augmented aub-trajectories. Based on that, we design the fuse process by simultane- ously taking the augmented sub-trajectories while main- taining the part of the states and actions associated with non-PSTs. The key contributions of this work are summarized as follows: (i) To the best of our knowledge, OAT is the first method augmenting historical trajectories to facilitate OPE in HIS. (ti) We conduct extensive experiments to validate OAT in a variety of simulation and real-world envi- ronments, including robotics, healthcare, and e-learning. (iii) The experimental results present that OAT can significantly facilitate OPE performance and outperform all data augmentation baselines.
GOAL
GOAL
Figure 1: A conceptual illustration of the discrepancy between human demon- strations (left) versus the empirical tra- jectories resulted from a sub-optimal policy (right) to be evaluated by OPE. It can be observed that the autonomous agent may perform maneuvers unseen from the training (demonstration) tra- jectories, and thus can potentially hin- der OPE's performance.
2 OPE WITH AUGMENTED TRAJECTORIES (OAT)
We propose a framework to facilitate OPE with aug- mented trajectories (OAT). Specifically, we first in- troduce offline trajectories and OPE. Then we pro- pose a sub-trajectory mining method that identifies the sub-trajectories of trajectories that have great potential to increase the offline trajectories' cover- age over the state-action space, iee. potential sub- trajectories (PSTs). A generative modeling frame- work is used to capture the dynamics underlying the selected PSTs, followed by fuse process that gen- erates augmented trajectories with augmented sub- trajectories which will be used train the OPE methods.
omhine Trppctores es
PS73
MM
Section 2,1 of Trajectories (PST6)
M0
Section 2.2 Augmentation
f
VAE-MDP
WAE-MPP
Auamened Trjectones
Section 2.2
Fuse Back Augmented PSTs
Offline Trajectories. We consider framing an agent's 's interaction with the environment over a se- quence of maaiing steps as a Markov de- cision process (MDP), which is formulated as a 6- tuple (S,A, S, T, 7). S is the state space. A is the action space. P defines transition dynamics from the current state and action to the next state. Sy defines the initial state distribution. r is the reward function. ? E (0, 1] is discount factor. Episodes are of finite horizon T. At each time-step I, the agent observes the state 8; S of the environment, then
Figure 2: The illustration of OAT. It con- sists of three steps: (i) Mining of poten- tial sub-trajectories (PSTs), where human be- have similarly under behavioral policies at the grey-shaded area and may have more po- tential enrich the coverage of its state- action space; (ii) VAE-MDP for augment- ing PSTs; (iii) Fuse augmented PSTs back to their origins.
2

---

Published as a conference paper at ICLR 2024
chooses action a E A following policy The environment accordingly provides reward = r(8,, (t), and the agent observes the next state S1+1 determined by P. 7(1) is defined as trajectory where 7()) = [..,, (St. 0z, Te, st)),..??-)
Offiine Policy Evaluation (OPE). The goal of OPE to estimate the expected total return over the evaluation (target) policy π, V
relaa ~ T], using set of historical trajectories D collected over a behavioral policy 8 + и. The historical trajectories D = {..., (0) J.. consist of set of N trajectories.
2.1 MINING OF POTENTIAL TEEETOREE (PSTS)
The historical trajectories D collected from HIS are often provided with limited coverage of the state-action space, due to the intrinsic nature that human may follow homogeneous behavioral poli- cies or specific guidelines when performing their professions (Yang et al. 2020b] Wang et al., 2022). For example, a surgeon could perform appendectomy in various ways across patients depending on each patient's 's specific condition; however, they may strictly follow similar steps at the beginning (e.g., disinfection) and the end surgeries (e.g., stitching). Therefore, the resulting trajectories may lead to limited coverage for part of the state-action space representing similar scenarios. However, a sub-optimal autonomous agent, subject to be evaluated by OPE, may visit states unseen from the trajectories collected from the surgeon, e-g., towards the beginning/end of the surgery. As result, we consider augmenting the part of trajectories, i.e., the PSTs, that are more likely to be insuffi- ciently covered by the historical trajectories D. Moreover, the downstream generative models, such as VAEs, do not necessarily need to reconstruct entire trajectories for long horizons and over limited samples which are the common limitations of data collected from HIS (acobyy et al. 2020).
To identify the PSTs that are subject to be augmented, we introduce a three-step approach, i.e., (i) discrete representation mapping, followed by (ii) determining support from discrete representations, where the support is used in step (iii) to identify PSTs to be augmented.
Step (i) Discrete Representation Mapping. Trajectories collected from HIS can be complex, due to the unobservable underlying human mindset and nigh-dimensional state space (Mandel et al.) 2014]. Discrete representation mapping has been recognized as effectively providing abstractions from complex original data and helping capture homogeneous behaviors shared across trajecto- ries (Yang et al. (021). We assume that states 8, E S can be mapped into C clusters, where each 8, is associated with cluster from the set K (K...... Kc}. After mapping, each state si, trajectory (8) is mapped K₁, E K.
Step (it) - Determine Support from Discrete Representations. We assume that each trajectory (1) can be mapped to corresponding temporal discrete sequence K7 = [K.,,..... K.r c ZT, based on the state mapping, where T is the horizon of the environment and Z the set of integers. We also define H = {..., Kto) J..JJ, which is the set of all temporal discrete sequences
mapped from the set of original trajectories D. We define g(i)
c. = [Ki,s; Kic+w- as
a temporal discrete sub-sequence (TDSS) with length W E [1, of K(), where Ç E [1,T
W + 1], denoted as g(6) - E k(, Note that C' is generally greatly smaller than T x N as considered in discrete representation mapping in general (Hallac et al., 2017] Yang et al. [021I) Therefore, it is possible that a temporal discrete sub-sequence 8(1)
5*-C+W-1 is "equal" another - such that g(1) - = (0)
temporal discrete sub-sequence 860)
2 2-++ if every Kr.- =
Kj given Ki,ç;, K j.5s E Z. Though Ç does not necessarily equals to (3, we omit the superscript
(1)
for concise expression. Then, the support (or frequency) of any TDSS cc+W-1 appears in H is the number of K in H. containing the TDSS, i.c.,
NV.
(8(9) support ne (6C.---1))
Σ
1
1(8 1(0 b
(0
(I)
where 1(-) is the indicator function.
(1) Step (iii) Identify PSTs. We denote PCcc+- [.., (8) 5 ".... as sub-
S+W-1
trajectory with length W of (() Given the mapping from trajectory (()) to temporal discrete se-
quence (+) (introduced in the step above), we define that each sub-trajectory f can be
(8)
3

---

Published as a conference paper at ICLR 2024
mapped to a corresponding TDSS 8 -- Now we can identify the PSTs that will be used to train the generative model for reconstructing new sub-trajectories (i.e., augmentation) in Section 2.2 following the definition below.
Definition 2.1 (Potential Sub-Trajectory (PST)), Given historical trajectories D and a threshold &, + is considered as a potential sub-trajectory if the support of its mapped temporal discrete sub- sebbeee g(() -1 satisfies support: (0 2.
a sub-trajectory (8)
An intuitive way to determine the threshold { is ranking the TDSSs by their supports in descending order and picking the top k ones. In this study, we iteratively select the top k distinct TDSSs until the support of the set of G selected TDSSs {6%.c D E [1, G). i.e., the number of K( in H containing any TDSS 8 E [6] (cc+w-11 J. is greater than or equal to .99N, or we early stop at k = 5.
Following the step above, set of PSTs is determined for historical trajectories D, from which we can obtain a set of G distinct corresponding TDSSs ( mapped from the PSTs. Then
we can obtain G sets of PSTs, such that each set T =
where (1)
F E T
satisfy that their corresponding 8(1)
-.C+W-1 = 0% G Each set of PSTs may contain unique information captured from the original historical trajectories D, as previous works have found that the PSTs in the same set, T are in general associated with similar temporal and ross-attributes
correlations (Gao et al. 2021 2022a).
Potential Sub- Sub-Trajectory (PST)
2.2 AUGMENTING THE PSTS
In this section, we introduce how to adapt VAE to cap- ture the MDP transitions, i.e., VAE-MDP, underlying each set of PSTs, T as well as reconstruct new PST
05
samples that will be fused back with the original his- torical trajectories DD for OPE methods to estimate the
Enooding
Z4
Z
P3
returns of evaluation (target) policies. The adaptation mainly consists of three parts: the latent prior, varia- tional encoder, and generative decoder. Given set of PSTs, 7 = f
Original
Trenstion
53
55
Augmented
0g
0.
Transtion
Augmented PST
the formulation of VAE- MDP consists of three major components, i.e., (i) the la- tent prior p(zc) ~ (0, I) representing the distribution of the initial latent states (at the beginning of each PST in the set T)) where I is the identity covariance ma- trix. (ii) the encoder 9. (2, 8:-1; 0,), 8;) that encodes the MDP transitions into the latent space, and (Mii) the de- coders Pr, (z -;- Pr, (xel2e)) Pr, (r,(r_11zz) that re- constructs new PST samples. The detailed setup can be found in Appendix A. and the overall encoding and de- coding processes are illustrated in Figure S
Figure 3: Illustration of VAE-MDP for Sub-trajectory augmentations. The PSTs are extracted by PSTs mining from original offline trajectories. Then VAE-MDP is employed to roll out new PSTs by reconstructing state-action space and inducing rewards. The aug- mented trajectory (colored in green) is formed by fusing back augmented PSTs to offline trajectories.
The training objective for VAE-MDP is to maximize the evidence lower bound (ELBO), which consists of the likelloodd of reconstructing the states and rewards, and regularization of the approximated posterior, i.e.,
L -ELBO(w, 11) Emn | C+W-1 log (, + + S+W-1 > (ogp,(m=112)) K (L(a(xx(ss)]lll
S+W-1 Eit KL KL (2,0l//ll11,,010.111 (1 (p, [x)
(2)
Consequently, given a set of PSTss, T a VAE-MDP to the set can be trained to reconstruct set of new PST samples, denoted as T
The proof of Equation 2 are provided in Appendix A.5
From now we use superscript to replace (+) for 8's, since there may exist multiple TDSSs that are equiv- alent.
From now on we omit the superscripts of 5 for conciseness.
4

---

Published as a conference paper at ICLR 2024
( [1, where P (3,, P7 "(") ".. -C+W- is a augmented PST and V is the total number of augmented PST samples, generated from VAE-MDP, for the set T
2.3 FUSE AUGMENTED PSTs BACK TO THEIR ORIGINS
With augmented sub-trajectories rolled out by the VAE-MDP, we fuse them back to the orig- inal historical trajectories 'D for the OPE methods to leverage. This fusing process is designed to (i) provide enhanced coverage over the state-action space where the corresponding PSTs do not plicitly capture homogeneous behaviors, and still (ii) maintain the part of the covered state-action distribution associated with non-PSTs, since those may indicate object-specific information that is not shared across all trajectories, e.g., the part of the surgical procedure specific to each patient, following from the surgery analogy above. Below introduce how to fuse T with the original trajectories from D. A graphical illustration of this step can be found in Figure 2
Given a trajectory 7() E DD the sets of PSTs {T) TO} mined from D following Section 2.1 and G sets of augmented sub-trajectories {T ..... TG} generated from G' corresponding VAE- MDPs following Section 2.2 augmented trajectory f(0) corresponding to -(() can be obtained
by +(0) = [I(L K. W 1))(5), fT, 8(") V V & k. + W -1) d Te
(0) (4) (1) (9)
(8), az, (., 8(-) E &8 &6 Pi.c+ E To. this study, the &². is selected as the one whose state and action are the closest to the original state and action at step ( + W 1. More details are provided in Appendix A.6
3 EXPERIMENTS
3.1 SETUP
Baselines. We investigate a variety of augmentation methods as baselines, including (i) RL-oriented methods: TDA (Park al. 2022) which originally incorporates with rewards learning by randomly extracting sub-trajectories from trajectories, we replace PST mining by TDA in OAT so that TDA can used for OPE with augmentation; permutation, Gaussian jittering, and scaling have been broadly employed in RL (Laskin et al. 2020b; Liu et al. 20200) Raileanu et al. (02T); (ii) generative methods: TimeGAN (Yoon et al. 2019) and VAE (Barak et al. 2022); (iii) time-series methods: SPAWNER (Kamycki et al., 2019) and DGW Iwaaaa & Uchida 202 lb) that consider similarities
across time series. We implement al--augmentation methods strictly following original algorithms, and use open-sourced code provided by the authors for the generative and time-series augmentation methods. Since generative and time-series methods are not proposed towards trajectories, we treat trajectories as multivariate time series as their input.
Ablations. One ablation of our approach is to apply VAE-MDP to reconstruct entire trajectories augmentations, i.e., without PST mining (Section 2.1) and fusing back (Section 2.3 Moreover, TDA (Park et al.] (022) ) and VAE (Barak et al.) (022) can be considered as two ablations as well, since TDA isolates our PST mining from OAT and VAE augments entire trajectories following the vanilla VAE (Kingma & Welling 2013), Le., without being adapted to the Markovian setting.
OPE methods considered. Outputs from all augmentation methods are fed into five OPE meth- ods to compare the performance achieved with versus without augmentations. The OPE methods we consider include importance sampling (IS) (Precupl 22000) fitted Q-evaluation (FQE) (Le al., 2019), distribution correction estimation (DICE) ( Yang et al. 2020a). doubly robust (DR) (Thomas) Brunskill 2016)) and model-based (MB) (Zhang et al. 2020a). We use the open-sourced imple- mentations provided by the Deep OPE (DOPE) benchmark (Fu et al. 202I).
Standard validation metrics. To validate OPE's performance (for both with and without aug- mentations), we standard OPE metrics introduced in the DOPE benchmark, which include absolute error, Spearman's 's rank correlation coefficient (Spearman 1987). regret@ 1, and regret@ Definitions of the metrics are described in Appendix
B.3 B.3
3.2 ENVIRONMENTS
To evaluate our method, OAT, well as the existing augmentation approaches for OPE, we both simulated and roal-world environments, spanning the domains of robotics, healthcare, and e- learning. The environments are human-involved which is generally challenging with highly limited
5

---

Published as a conference paper at ICLR 2024
quantity of demonstrations containing underrepresented state space, due to homogeneous interven- tions when collecting the historical trajectories.
Adroit. Adriot (Rajeswaran al. 2018) is simulation environment with four synthetic real-world robotics tasks, where a simulated Shadow Hand robot is asked to hammer a nail (nammer), open a door (door), twirl pen (pen), or pick up and move ball (rel reloccae). Each task contains three training datasets with different levels from human (human), induced data from a fine-tuned RL policy (expert), and mixing data with a 50-50 ratio of demonstration and induced data (cloned). We follow the experimental settings provided in Deep OPE benchmark, with 11 DAPG-based evaluation policies ranging from random to expert performance (Fu et al. (021).
numan-involvement.
including full demonstration data
Real-World Sepsis Treatment. We investigate challenging task healthcare, sepsis treatments, which has raised broad attention in OPE (Namkoong et al. 2020 Nie et al., 2022). Specifically, the trajectories are taken from electronic health records containing 221,700 patient visits collected from a hospital over two years. The state space is constituted by 15 continuous sepsis-related clinical attributes that represent patients' health status, including heart rate, creatinine, etc. The cardinality of the action space is 4, i.e., two binary treatment options over {antibs iettiiiiiiio ott.admmiinnmmmmoommcocac.o..iii ciddmnniss rat inn, oxygen_assi ance). Given the four stages of sepsis defined by the clinicians (Delano & Ward 2016). the rewards set for each stage: infection (=5), inflammation (=1)), organ failure (=20), and septic shock (+))). Negative rewards are given when a patient enters a worse stage, and positive rewards given when the patient recovers to better stage. The environment considers discrete time steps, with the horizon being 1160 steps. Five evaluation (target) policies are obtained by training Deep Q Networks (DQNs) (Mnih et al. 2015) respectively over different syper-parameters. More details are provided in Appendix
Real-World Intelligent Tutor, Another important nuuman-involvee task for OPE is intelligent tutor- ing, where students interact with intelligent tutors, with the goal of improving students' engagements and learning outcomes. Such topics have been investigated prior OPE works (Mandel al. 2014] Nie et al. 2022). Specifically, we collect trajectories recorded from 1,307 students' interaction logs with an intelligent tutor, over seven semesters of an undergraduate course at an university. Since students' underlying learning states are unobservable (Mandel et 2014), we consult with domain experts who help defines the state space which constituted by 142 attributes that could possibly capture students' learning status from their logs. The cardinality of the action space is 3, i.e. each problem, the tutor need to decide whether the student should solve the next problem by themselves, study a solution provided by the tutor, work together with the tutor solve on the problem. During the tutoring, each student is required to solve 12 prohlems, thus the horizon of the environment is considered as 12 discrete steps. Sparse rewards are obtained the end of the tutoring, which are defined as students' normalized learning gains (Chi et al. (01I). We use the trajectories collected from six semesters as the training data, where the behavior policy follows an expert policy commonly used in e-learning (Zhou et al. 2019). and test on the upcoming semester. There are 4 evaluation policies, including three obtained by training DQNs over different hyper- parameters respectively, in addition to one expert policy. More details are provided in Appendix
Three sets of PSTs that have significantly improved coverage after augmentation
3.3 RESULTS
The need of PSTs mining. To better understand the need of PSTs mining (Section 2.1) conceptually, visualize the set of augmented trajectories produced by our method, against the original set of historical trajectories D, over the Maze2D- - environment which is a toy navi- gation task requiring an agent to reach a fixed goal lo- cation (Fu et al. 2020). We uniformly down-sample a limited number (i.e., 250) of trajectories from the original dataset provided by D4RL (overall 12k trajectories), and use our method to augment this subset such that the to- tal number of trajectories becomes ten times (x 10) larger. The visualization is shown in Figure It can be observed (x10).
Original
Augmented (X10)
Figure 4: Visualization of trajectories in Maze2D-umaze Left: the original 250 trajectories: Right: augmented data with ten times numbers of trajectories
that there exist 3 sets of PSTs (as cireled in the figure) that have significantly increased state space visitation after augmentation, benefiting from the PSTs mining methodology introduced in Sec-
tion 2.1

---

Published as a conference paper at ICLR 2024
3.3.1 RESULTS OVER ADROIT
QAT
Figure 5 summarizes the averaged improvements across five OPE methods, over all four tasks (i.e. hammer, door, pen, relocate) in Adroit human, quantified by the percentage increases over the four validation metrics achieved by the OPE methods evaluated over the augmented against the original datasets. Overall, our method significantly improves OPE methods in terms of all standard val- idation metrics, and achieves the best performance compared to all augmentation baselines. This il- lustrates the effectiveness and robustness of our proposed methods across environments and tasks. There is no clear winner among baselines, where VAE, TimeGAN, and scaling in general perform bet- ter in terms of MAE, DGW and scaling performs better in terms of rank correlation, permutation and jittering perform better in terms of @ More specifically, besides the fact that all methods can in general improve MAE, most baselines lead nega- tive effects in terms of the other three metrics. De- tailed results are presented in Appendix [.]]
Parm
soe
Soaing
MAE Forc. Improved
Rank Carr. improved
GAT MDF
TDA Hirm
VAE
SPAATEr Dow
-12 Regretig5 Improved
Regreti@1 Improved
Figure 5: OPE improvement results aver- aging across OPE methods and 4 tasks in Adroit human using each augmentation method, Top-left: Mean absolute error (MAE) percentage improved. Top-right: rank correlation improvements. Bottom-left & bottom-right: Regret and @5 improve- ments, respectively.
More importantly. it can be observed that the ab- lation baseline VAE-MDP is significantly outper- formed by OAT across all metrics, which further jus- tifies the importance of augmenting over the PSTs instead of the entire horizon. It can be also observed that VAE-MDP in general outperforms the vanilla VAE without adaptation to the Markovian setting, illustrating the importance of the adapta- tion step introduced in Section 2.22 We also find that generative models achieve the best performance among the baselines over environments that have relatively shorter horizons (e.g., pen), while their performance is diminished when horizons increased. That further indicates the advantage of PSTs mining that provides much shorter and representative sub-trajectories for generative learning.
3.3.2 RESULTS OVER WORLD HEALTHCARE AND E-LEARNING
Figure 6 presents the average MAE improvements across all OPE methods in e-learning (left), and improved rank correlation in healthcare (right). Complete results for empirical study and all val- idation metrics are provided in Appendix E Re- g is not applicable to both environments, since the total number of evaluation policies are less than or equal to five.
E-Leaming
Healthcare
VAE-MDP
TDA
TimecAs
SPWNNER
DGW
Perm.
jittering
Scaling
Overall, our method can significantly improve OPE performance in terms of MAE, rank corre- lation, and regrer@1 1 in both real-world environ- ments. In both e-learning and healthcare, most augmentation basclines lead to neutral to negative percentage improvements over the metrics considered, while OAT significantly improves OPE's per- formance over all baselines, with the ablation VAE-MDP attains the 2nd best performance. A reason for baselines perform worse in real-world environments than in simulations can be that real-world HIS are considered sophisticated, further indicates the importance of extracting underlying information from historical trajectories D, as did in OAT and VAE-MDP, as well as effectively enriching the coverage of state-action space to provide more comprehensive coverage for OPE methods to leverage, powered by the methodologies introduced in Section
130
190
100 Perc. Improved
43 20
28
Cor:. Improved
Figure 6: OPE improvement results averaging across 5 OPE methods in e-learning (left) and healthcare (right).
the human mental states impact their behaviors implicitly. This
7

---

Published as a conference paper at ICLR 2024
3.3.3 MORE DISCUSSIONS
We explore the following two major questions that are commonly involved in analyses over HIS.
Will the level of human involvements affect tra- jectory augmentations for OPE? As presented in Figure 7 we evaluate augmentation meth- ods across the four tasks in Adroit environment with three different levels of human involvements (LoHI) sorted from the most to least, i.e., human, cloned, and expert. The results show that our method achieves the best performance in terms of all validation metrics when humans are involved in data collection (i.e., human, cloned). The per- formance of our method is slightly attenuated (but still effective) when the LoHI decreased, while our ablation VAE-MDP leads MAE when the LoHI is 0% (i.e., expert). Though TDA is effective un- der the case when the LoHI is 0%, it still performs worse than OAT and consistently worse at other levels. Such finding further confirms the effec- tiveness of PST mining. Moreover, most base- lines are ineffective when the LoHI is below 50%. The reason is that the trajectories obtained from demonstrations often provide limited and/or bi- ased coverage over the state-action space (Fu et al.) 2021 Chang et al.) 2021), thus any augmentation methods that can potentially increase the coverage might be able to improve OPE's performance. In contrast, the historical trajectories induced from simulations using fine-tuned policies tend to sult in better coverage over the state-action space in general (Fu et al. 2020), and the augmentation methods that do not consider the Markovian setting generate trajectories that could be less meaning- ful (i.e., providing limited or even unrealistic information) to the OPE methods, making them less effective (e.g., negative effects on OPE when LoHI is 0%). For example, in the realm of surgery, permutation can result in a trajectory with stitching happened before incision, which is unrealistic.
MAE Perc. Improved
Bank Corr. Improved
Regret
Improved
Regret@5 Improved
OAT
TDA
SPRWNES
VAE-HD"
Pers
Scaling
TIMECAN
DOW
Figure 7: OPE improvement results with three involvin levels, i.e., 100% (human), 50% (c (#loned), 0% (expert), averaging across 5 OPE methods and 4 tasks in Adroit.
Can trajectory augmentation facilitate OPE in terms of significance test? OPE validation metrics generally focus on standard error metries as proposed in (Fu et al.) 2021]. while domain experts emphasis statistical signif- icance test for real-world HIS (Robertson & Kaptein 2016 Zhou et al., [022]. For example, rank correlation summarizes the performance of relative rankings of a set of policies using averaged returns; in contrast, statistical significance tests can examine if the relationships being found are due to randomness. Moreover, they can be eas- ier conveyed to and interpreted by domain experts (Guil-
Table 1: Statistical significance test at the level of < 0.05 with bootstrapping on three RL-induced policy Ti, T2. Ty compared to expert policy Texpert from real-world intelligent tutoring. The re- sults that show significance are in bold.
IS result
S1. 1,
f,
No Aug.
7.24
7.07 .0
-4.8 00
OAT
3.10
1.33,
2.11
TDA
10.14
5.82.00
-13.58,0
Perm.
-1.78,08
-1.77 0s
1.77 ,z
Jittering
-1.89.
-1.89 o6
1.90
Ford 1950: Ju al. 2019).
Scaling
VAE
-1.94.0
-1.33.m
-1.92
-1.33.
1.33.nM
1.54
TimeGAN
1.90
05
-2.25 05
One key measurement for RL-induced policies is whether they significantly outperform the expert policy in HIS (Zhou et al., 22199 2022). We conduct t-test over OPE estimations (with and without augmentations) ob- tained from bootstrapping as introduced in (Hao et al. 2021,, and measure whether there is a significant difference between the mean value of OPE esti- mation for each RL-induced policy against the expert policy. Interestingly, the results show that IS performs the best among all 5 OPE methods we considered, in terms of all standard validation met- rics in e-learning experiments, with and without augmentations using each augmentation method. This can be caused by the fact that the behavioral policies are intrinsically similar, where 3 out of the 4 policies (i.e., T2, Ts. Texpert) lead to similar returns (as shown in Appendix E.T) and the horizon (i.e., T=12) is not much long, the unbiased nature of IS estimators could dominate it's high variance downside. Such characteristics of IS made it broadly used in short-horizon settings (Mandel et al.)
SPAWNER
-1.00
-1.00 az
1,00 32
DGW
-1.43.m
-1.43 16
1.43,
Empiriaal result 2.01
0.61.s1
0.20.st
8

---

Published as a conference paper at ICLR 2024
2014 Xie et al.
2019). The statistical significance results are summarized in Table It can be ob- served that, without augmentation, IS estimates that all RL-induced policies performs significantly different from the expert policy. However, in empirical study, only Ti performs significantly better than expert policy, while the other two, i.e., T2 and T3 not. And our proposed method is the only one that improves the IS estimation to be aligned with empirical results across all three policies, while the baselines improve estimation at most one policy. Therefore, the results indicate the effectiveness of our proposed method in terms of both standard OPE validation metrics and significance test.
4 RELATED WORKS
OPE A variety of contemporary OPE methods has been proposed, which can be mainly divided into three categories (Voloshin et al. 2021b): (i) Inverse propensity scoring (Precup 2000 Doroudi
et al., 2017), such as Importance Sampling (IS) (Doroudi et al. 2017). (ii) Direct methods that
directly estimate the value functions of the evaluation policy (Nachum et al. (019] Xie et al. 2019
Zhang et al., [221] Yang et al., [0222] Gao al. [202cc). including but not limited to model -based
estimators (MB) (Paduraru 2013 Zhang et al. 2021). value-based estimators (Munos et al. 2016
Le et al. 2019) such as Fitted Q Evaluation (FQE), and minimax estimators (Liu et al. 2018 Zhang
et al. 2020b Voloshin et al., 2021a) such as DuaIDICE (Yang et al., 2020a). (ii) Hybrid methods
combine aspects of both inverse propensity scoring and direct methods (Jiang & Li 2016 Thomas
& |runskkll 2016). such as DR (Jiang & Li] 2016). However, a major challenge of applying OPE
real-world that many methods can perform unpleasant when human-collected data is highly
limited as demonstrated in (Fu et al. 2020 Gao et al. 2023a 2024). Therefore, augmentation can be an important way to facilitate OPE performance.
Data Augmentation for RL In RL, data augmentation has been recognized as effective to improve generalizability of agents over various tasks (Laskin et al. 20206 a Kostrikov et al. 2020 Liu et al. 2021 Raileanu et al. 2021 Joo et al.) [022] Goyal et al. (002).. For instance, automatic augmen- tation selection frameworks are proposed for actor-critic algorithms by regularizing the policy and value functions (Raileanu et al. (02T). However, most of the prior work only consider image in- put which may not capture temporal dependencies trajectories. More importantly, the prior work is proposed towards RL policy optimization by learning from high-rewart regions of state-action space, while OPE aims to generalize over evaluation policies that can be heterogeneous and lead to varied performance. To the best of our knowledge, prior work has extensively investigated various prior augmentation methods in OPE, nor proposed augmentation towards offline trajectories scaffold OPE in real-world domains. More comprehensive review of related works on OPE and data augmentations in general can be found in Appendix G
5 CONCLUSION & LIMITATION
We have proposed OAT, which can capture the dynamics underlying from historical trajectories that provide limited coverage of the state-action space and induce effec- tive augmented trajectories to facilitate OPE. This is achieved by mining potential sub-trajectories, as well extending a generative modeling framework capture dynamics under the potential sub- tub-trajectories. We have validated OAT in both simulation and real-world environments, and the results have shown that OAT can generally improve OPE performance and outperform variety of data augmentation methods. Latent-modeelbasee models such as VAE have been commonly used for augmentation in offline RL, while they generally rarely come with theoretical error bounds provided (Hafner et al. [0200] Lee et al. 2020 Rybkin et al., (021). Such a challenge also re- mains for many data augmentation methods (Zheng et al. (023). However, once the trajectories are augmented, one can choose to use the downstream OPE methods which come with guarantees, such as DR and DICE. Moreover, prior works found that distribution shift could be a challenge for OPE (Wang et al. 2021 Fu et al. 2021). Though this is beyond the scope of this work, given we use existing OPE methods as backbones to process the augmented trajectories, a potential future work is coming up with new OPE methods that can resolve the distribution shift. We conduct exten- sive experiments to examine the proposed augmentation method, and the results demonstrating its effectiveness. OAT can be stand-alone generate trajectories without any assumptions over target policies. And it can be utilized by built-on-top works such as policy optimization and representation leamning
environments
9

---

Published as a conference paper at ICLR 2024
SOCIAL IMPACTS
All real-world educational and healthcare data employed in this paper were obtained anonymously through exempt IRBBapppoved protocols and were scored using established rubrics. No demographic data or class grades were collected. All data were shared within the research group under IRB, and were de-identifies and automatically processed for labeling. This research seeks to remove societal harms that come from lower engagement and retention of students who need more personalized interventions and developing more robust medical interventions for patients.
ACKNOWLEDGMENTS
This research supported by the NSF Grants: Integrated Data-driven Technologies for Individ- ualized Instruction in STEM Learning Environments (1726550), CAREER: Improving Adaptive Decision Making in Interactive Learning Environments (1651909) Generalizing Data-Driven Tech- nologies to Improve Individualized STEM Instruction by Intelligent Tutors (2013502), CAREER: Foundations for Secure Control of Cyber-Physical Systems (1652544), and National AI Institute for Edge Computing Leveraging Next Generation Wireless Networks (2112562). This research was also sponsored in part by the AFOSR under award number 1199500-191-011 We would also like to thank the anonymous reviewers for insightful comments that lead to improved paper presentations.
REFERENCES
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXiv: 111.:1771100440 2017.
Hamoon Azizsoltani and Yeo Jin. Unobserved is not equal non-existent: Using gaussian processes to infer immediate rewards across contexts. In In Proceedings of the 28th International Joint Conference on Artificial Intelligence, 2019.
Sasan Barak, Ehsan Mirafzali, and Mohammad Joshaghani. Improving deep learning forecast using variational autoencoders. Available SSRN, 2022.
Roger C Bone, Robert A Balk, Frank B Cerra, R Phillip Dellinger, Alan M Fein, William A Knaus, Roland MH Schein, and William J Sibbald. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. Chest, 10166) 1992.
Pedro G Campos, Fernando Díez, and Iván Cantador. Time-aware recommender systems: compre- hensive survey and analysis of existing evaluation protocols. User Modeling and User-Adapted Interaction, 24:67-119, 2014.
Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Advances in Neural Information Processing Systems, 34:965-979, 2021.
Min Chi, Kurt VanLehn, Diane Litman, and Pamela Jordan. Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies. User Modeling and Uxer-Adapted Interaction, 21(1):137-180, 2011.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 113-123, 2019.
Matthew J Delano and Peter A Ward. The immune system's role in sepsis progression, resolution, and long-term outcome. Immunological reviews, 274(1 274(1):330-33) 33:3033353, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In International Conference Learning Representations, 2018.
10

---

Published as a conference paper at ICLR 2024
Shayan Doroudi, Philip S Thomas, and Emma Brunskill. Importance sampling for fair policy selec- tion. Grantee Submission, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprini wrXiv:2004.072//1 2020.
Justin Fu, Mohammad Norouzi, Ofrr Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, et al. Benchmarks for deep off- policy evaluation, arXiv preprint arXiv:2103. 16596, 2021.
Ge Gao, Samiha Marwan, and Thomas W Price. Early performance prediction using interpretable patterns in programming process data. In Proceedings of the 52nd ACM technical symposium computer science education, pp. 342-348, 2021.
Ge Gao, Qitong Gao, Xi Yang, Miroslav Pajic, and Min Chi. A reinforcement learning-informed pattern mining framework for multivariate time series classification. In Proceedings of the Thirty- First International Joint Conference on Artificial Intelligence, IJCA1-22, pp. 2994-3000 2022a.
Ge Gao, Farzaneh Khoshnevisan, and Min Chi. Reconstructing missing ehrs using time- ammre within-and cross-visit information for septic shock early prediction. In 2022 IEEE 10th Interna- tional Conference on Healthcare Informatics (ICHI), pp. 151-162. IEEE, 2022b.
Ge Gao. Song Ju, Markel Sanz Ausin, and Min Chi. Hope: Human-centric off-policy evaluation for e-learning and healthcare. AAMAS, 2023a.
Ge Gao, Yang, and Min Chi. Get a head start: On-demand pedagogical policy selection in intelligent tutoring. In Proceedings of the AAAI Conference Artificial Intelligence, 2024.
Qitong Gao, Ge Gao, Min Chi, and Miroslav Pajic. Variational latent branching model for off-policy evaluation. In The Eleventh International Conference on Learning Representations, 2022c.
Qitong Gao, Ge Gao, Juncheng Dong, Vahid Tarokh, Min Chi, and Miroslav Pajic. Off-policy evaluation for human feedhack. In l'hirty-seventh Conference on Neural Information Processing Systems, 2023b.
Qitong Gao, Stephen L. Schmidt, Afsana Chowdhury, Guangyu Feng, Jennifer J Peters, Katherine Genty, Warren M Grill, Dennis A Turner, and Miroslav Pajic. Offline learning of closed-loop deep brain stimulation controllers for parkinson disease treatment. In Proceedings of the ACM/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-loT Week 2023), pp. 44-55, 2023c.
Anirudh Goyal, Abram Friesen, Andrea Banino, Theophane Weber, Nan Rosemary Ke, Adria Puig- domenech Badia, Arthur Guez, Mehdi Mirza, Peter C Humphreys, Ksenia Konyushova, et al. Retrieval-augmmened reinforcement learning. In International Conference on Machine Learning, pp. 7740-7765. PMLR, 2022.
Joy Paul Guilford. Fundamental statistics in psychology and education. 1950.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
David Hallac, Sagar Vare, Stephen Boyd, and Jure Leskovec. Toeplitz. inverse covariance-oe clustering of multivariate time series data. In ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pp. 215-223, 2017.
Josiah Hanna, Scott Niekum, and Peter Stone. Importance sampling policy evaluation with an estimated behavior policy. In International Conference on Machine Learning, pp. 2605-2613. PMLR, 2019.
Botao Hao, Xiang Ji. Yaqi Duan, Hao Lu, Csaba Szepesvári, and Mengdi Wang. Bootstrapping statistical inference for off-policy evaluation. arXiv preprint arXiv:2102.03600 2021.
Brian Kenji Iwana and Seichh Uchida. An empirical survey of data augmentation for time series classification with neural networks. Plos one, 16())e0225841, 2021a.
11

---

Published as a conference paper at ICLR 2024
Brian Kenji Iwana and Seiichi Uchida. Time series data augmentation for neural networks by time warping with a discriminative teacher. 2020 25th International Conference on Pattern Recog- nition (ICPR), pp. 3558-3565 IEEE, 2021b.
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation reinforcement learning. In International Conference Machine Learning, pp. 652-661. PMLR, 2016.
Ho-Taek Joo, In-Chang Back, and Kyung-Joong Kim. A swapping target q-value technique for data augmentation in offline reinforcement learning. IEEE Access, 2022.
Song Ju, Shitian Shen, Hamoon Azizsoltani, Tiffany Barnes, and Min Chi. Importance sampling to identify empirically valid policies and their critical decisions. EDM (Workshops), pp. 69-78, 2019.
Song Ju, Yeo Jin Kim, Markel Sanz Ausin, Maria E Mayorga, and Min Chi. To reduce healthcare workload: Identify critical sepsis progression moments through deep reinforcement learning. In 2021 IEEE International Conference on Big Data (Big Data), pp. 1640-1646. IEEE, 2021.
Krzysztof Kamycki, Tomasz Kapuscinski, and Mariusz Oszust. Data augmentation with suboptimal warping for time-series classification. Sensors, 20(1): 2019.
Yeo-Jin Kim and Min Chi. Temporal belief memory: Imputing missing data during rnn training. In In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI-201)), 2018.
Diederik P Kingma and Max Welling, 1rXiv:1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The arti- ficial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine, 24(11) 17:1--7777, 2018.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.1364) 2020.
Michael Laskin, Aravind Srinivas, and Picter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, pp. 5639- 5650. PMLR, 2020a.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in neural information processing systems, 33: 19884-19895, 2020b.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Inter- national Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.
Arthur Le Guennec, Simon Malinowski, and Romain Tavenard. Data augmentation for time se- ries classification using convolutional neural networks, In ECMI/PKDD workshop advanced analytics and learning on temporal data, 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio. and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( (6(11):2278-2322 1998.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic:
Deep reinforcement learning with a latent variable model. Advances Neural Information Pro- cessing Systems, 33:741-752, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint /009.020 2015.
12

---

Published as a conference paper at ICLR 2024
Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with rnns: Improved classification clinical time series. In Machine learning for healthcare conference, pp. 253-270. PMLR, 2016.
Bo Liu, Zhenguo Zhang, and Rongyi Cui. Efficient time series augmentation methods. In 2020 13th international congress on image and signal processing, BioMedical engineering and informatics (CISP-BMEI), pp. 1004-1009, IEEE, 2020.
Jinxin Liu, Zhang Hongyin, and Donglin Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning. In International Conference on Learning Representations, 2021.
Qiang Liu, Lihong Li. Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off-policy estimation. Advances in Neural Information Processing Systems, 31, 2018.
Vincent Liu, Gabriel Escobar, John Greene, Jay Soule, Alan Whippy, Derek C Angus, and Theodore Iwashyna. Hospital deaths in patients with sepsis from 2 independent cohorts, Jama, 312(1):90-92, 2014.
Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In AAMAS, volume 2014.
G.S Martin, D.M Mannino, et al. The epidemiology of sepsis in the united states from 1979 through 2000. New England Journal of Medicine, 2003.
Iris Miliaraki, Klaus Berberich, Rainer Gemulla, and Spyros Zoupanos. Mind the gap: Large-scale frequent sequence mining. In Proceedings of the 2013 ACM SIGMOD international conference on management of data, pp. 797-808, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 51 518(7540):529-5 -33333,, 2015.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of dis- counted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019.
Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances in Neural Informa- tion Processing Systems, 33:18819-18833 19-18811, 2020.
Allen Nie, Yannis Flet-Berliac Deon Richmond Jordan, William Steenbergen, and Emma Brunskill. Data-efficient pipeline for offline reinforcement learning with limited data. In Advances Neural Information Processing Systems, 2022.
Cosmin Paduraru. Off-policy evaluation in markov decision processes. 2013.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for fefibeck- preference-based reinforcement learning. In International Conference on Learning Representations, 2022.
Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint 1711::11110960000 2017.
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. Advances in Neural Information Processing Systems, 34:5402-5415
2021. 2021.
13

---

Published as a conference paper at ICLR 2024
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems, 2018.
Judy Robertson and Maurits Kaptein. Modern statistical methods for IICI. Springer, 2016.
Havard Rue and Leonhard Held. Gaussian Markov random fields: theory and applications. CRC press, 2005.
Oleh Rybkin, Chuning Zhu, Anusha Nagabandi, Kostas Daniilidis, Igor Mordatch, and Sergey Levine. Model-based reinforcement learning via latent-space collocation. In International Con- ference on Machine Learning, pp. 1190-92011 PMLR, 2021.
Kajetan Schweighofer, Markus Hofmarcher, Corius-Consttntin Dinu, Philipp Renz, Angela Bitto- Nemling, Vihang Prakash Patil, and Sepp Hochreiter. Understanding the effects of dataset char- acteristics on offline reinforcement learning. In Deep RL Workshop NeurIPS 2021, 2021.
Mervyn Singer, Clifford S Deutschman, Christopher Warren Seymour, Manu Shankar-Hari, Djillali Annane, Michael Bauer, Rinaldo Bellomo, Gordon R Bernard, Coopersmith,
Chiche, Craig M The third international consensus definitions for sepsis and septic shock (sepsis-3). Jama, 31 315(8):801-810, 2016.
Charles Spearman. The proof and measurement of association between two things. The American journal of psychology, 100(3/4):411-471, 1987.
Philip Thomas and Emma Brunskill. Data-efficient off off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 2139-2148. PMLR, 2016.
Masatoshi Uchara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off-
policy evaluation. In International Conference Machine Learning, pp. 9659-96666 PMLR, 2020.
Cameron Voloshin, Nan Jiang, and Yisong Yue. Minimax model learning. In International Confer- ence on Artificial Intelligence and Statistics, pp. 1612-1620. PMLR, 2021a.
Cameron Voloshin, Hoang Minh Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021b.
Lu Wang, Ruiming Tang, Xiaofeng He, and Xiuqiang He. Hierarchical imitation learning via sub- goal representation learning for dynamic treatment recommendation. In Proceedings of the Fif- teenth ACM International Conference on Web Search and Data Mining, pp. 1081-1089, 2022.
Ruosong Wang, Yifan Wu, Ruslan Salakhutdinov, and Sham Kakade. Instabilities of offline rl with pre-trained neural representation. In International Conference on Machine Learning, pp. 10948- 10960. PMLR, 2021.
Jingda Wu, Zhiyu Huang, Wenhui Huang, and Chen Lv. Prioritized experience-based reinforcement learning with human guidance for autonomous driving. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances Neural Information Processing Systems, 33:6256-6268 2020.
Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal off-policy evaluation for rein- forcement learning with marginalized importance sampling. Advances in Neural Information Processing Systems, 32, 2019.
Yaniv Yacoby, Weiwei Pan, and Finale Doshi-Veles Failure modes of variational autoencoders and their effects downstream tasks. arXiv preprint 12Xiv:2007.07124, 2020.
Mengjiao Yang. Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off- evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551-6561, 2020a.
14

---

Published as a conference paper at ICLR 2024
Mengjiao Yang. Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selec- tion under uncertainty. In International Conference on Artificial Intelligence and Statistics, pp- 4376-4396. PMLR, 2022.
Xi Yang, Guojing Zhou, Michelle Taub, Roger Azevedo, and Min Chi. Student subtyping via em-
inverse reinforcement learning. International Educational Data Mining Society, 2020b.
Xi Yang, Yuan Zhang, and Min Chi. Multi-series time-aware sequence partitioning for disease progression modeling. In IICAI, 2021.
Jinsung Yoon, Daniel Jarrett, and Mihacla Van der Schaar. Time-series generative adversarial net-
works. Advances in neural information processing systems, 32, 2019.
Michael R Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Mohammad Norouzi, et al. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference Learning Representations, 2020a.
Michael Zhang, Tom Le Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyu Wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. arXiv preprint arXiv:2104. 13877, 2021.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. In International Conference Machine Learning, pp. 11194- 11203. PMLR, 2020b.
Chenyu Zheng, Guoqiang Wu, and Chongxuan Li. Toward understanding generative data augmen- tation. Phirty-seventhl Conference on Neural Information Processing Systems, 2023.
Guojing Zhou, Hamoon Azizsoltani, Markel Sanz Ausin, Tiffany Barnes, and Min Chi. Hierarchical reinforcement learning for pedagogical policy induction. In International conference artificial intelligence in education, pp. 544-556. Springer, 2019.
Guojing Zhou, Hamoon Azizsoltani, Markel Sanz Ausin, Tiffany Barnes, and Min Chi. Leveraging granularity: Hierarchical reinforcement learning for pedagogical policy induction. International journal of artificial intelligence in education, 32(2):454-500, 2022.
15

---

Published as a conference paper at ICLR 2024
LIST OF APPENDICES
Appendix17 17
More Details on Methodology
17
A.I
Complexity Analysis
17
A.I.I Training Time with VAL-MDP
17
A.2
More Analysis Augmented Trajectories from OAT
18
A.2.1 Scatter Plots on Returns
18
A.3
Details of Discrete Representation Mapping
19
A.3.1 TICC Problem]
19
A.4
Detailed formulation of the VAE-MDP
20
A.5
Proof of Equation 2
21
A.6
More Details of Fusing Back Step
A.7
OAT with Non-tempont Non-temporallyyalgee Trajectories
21
Experimental Setup
22
B.1
Training Resources
22
B.2
Implementation Details & Hyper-Parametens
22
B.3
Evaluation Metrics
22
C
Adroit
23
Detailed Results
23
C.2 Start and End Time of PSTS.
37
D Real-World Sepsis Treatment
39
D.I Task Details
Real-World Intelligent Tutoring
40
E.
Task Details
40
Additional Experiments and Results: Gym-Mujoco
42
F.1
Details of Gym-Mujoco)
42
Results on Gym-Mujoco and Discussions
42
More Related Works
43
16

---

Published as a conference paper at ICLR 2024
A MORE DETAILS ON METHODOLOGY
A.1 COMPLEXITY ANALYSIS
OAT consists of three main steps, including PST's mining, PST's augmentation, and fuse back; thus, the overall complexity depends on the specific techniques used in each step. In PSTs mining, the discrete representation mapping uses TICC, which takes O(TNCC) time assign TNN states (I' is the horizon, N is the number of trajectories) into C clusters (Hallac et al. 2017). To identify PSTs, we use MG-FSM (Miliaraki et al. 2013) for sub-trajectory mining which has O(T) time complexity. We found that providing complexity for PSTss augmentation could be challenging given that training VAE requires stochastic gradient descent algorithms with parameter tuning (e.g. for step size and network architecture). In fusing back, the time complexity is O(N) if the same amount of augmented trajectories added, and we are positive this can be shortened with parallel processing. Overall, as shown in Table 2 average, OAT takes less training time of timeGAN and VAE with the same neural network architecture, respectively, which indicates OAT benefits more in augmenting on much shorter PSTs than the mining process.
For the cost of reaching effectiveness, the total number of TDSSs from which PSTs are selected correlated to the length of the TDSSs which capped at the horizon, i.., the total number of TDSSs is capped at C However, in Adroit-human, we noticed that the longest length of the PSTs found is 10 when & is 10 (less than N/2). so T seems like generous upper bound. For example, in Adroit door-human, PSTs mining takes 224s over 6.7k data points.
Table Average training time of OAT and generative-model--arre methods in Adroit environment.
Training time (s)
Pen
Door
Relocate
Hammer
OAT
1662
688
734
1476
VAE
4247
2266
3184
3756
TimeGAN
1888
3027
4178
3716
A.1.1 TRAINING TIME WITH VAE-MDP
We plot the training time of VAE-MDP vs length of PSTs across all 12 Adroit datasets in Figure obtained from the same architecture of LSTMs and training settings provided in Appendix B.2
From the Figure & we can observe that the training time of VAE-MDP is increased almost linearly ly with the length of PSTs. In experiments, the lengths of PSTs mined from most of the datasets (i.e., 9 out of 12 datasets) are within 11, which only require less than 1 minute to train the VAE-MDP.
Training of (5)
50
20
Length of PST (all Adroit datasets)
Figure 8: Plots of training time of VAE-MDP (in seconds) vs length PSTs across all Adroit tasks.
17

---

Published as a conference paper at ICLR 2024
A.2 MORE ANALYSIS ON AUGMENTED TRAJECTORIES FROM OAT
A.2.1 SCATTER PLOTS ON RETURNS
We present scatter plots plotting the true returns of each policy against the estimated returns. Each point the plot represents one target policy.
FOE
M6
DICE
Before
After aug.
Retum (y-0.995)
Figure 9: Scatter plots of estimate ground truth return before (top) and after (bottom) OAT aug- mentation in Adroit pen-humee human environment.
FOE
MB
15
DICE
Before aug.
After aug.
Retum 0.59
Figure 10: Scatter plots of estimate vs ground truth return before (top) and after (bottom) OAT augmentation in Adroit relocate -omocatteeeeann environment.
18

---

Published as a conference paper at ICLR 2024
FOE M6
DR
DICE
Before aug.
60
aug.
Rcturn (y=0.995)
Figure 11: Scatter plots of estimate vs ground truth return before (top) and after (bottom) OAT augmentation in Adroit hammer-hunee environment.
FOE
IS
DR
DICE
Before
After aug.
Return (p-0.995)
Figure 12: Scatter plots of estimate vs ground truth return before (top) and after (bottom) OAT augmentation in Adroit door-hin -human environment.
A.3 DETAILS DISCRETE REPRESENTATIOE MAPPING
In this work, we leverage Toeplitz inverse covariance-based clustering (TICC) (Hallac et al. 2017) to map states 8t E S into C clusters, where each 8; associated with a cluster from set K= (K,,..., Kc). The states mapped to the same cluster be considered sharing graphical connectivity structure of both temporal and cross-attributes information captured by TICC. There are variations of TICC targeting specific characteristies of data. Specifically, used MT-TICC (Yang et al. 2021) which proposed towards awarenessssss and multi muuti-trajectoriis. The hyperparameter C'aan be determined by calculating silhouette score (Hallac al. 2017).
A.3.1 TICC PROBLEM
Each cluster e [1, C) defined as a Markov random field (Rue & Held 2005). or correlation net- work, captured by its Gaussian inverse covariance matrix Σ R. where m is the dimension of state space. We also define the set clusters K {KI..., Ko} C I as well as the set of inverse covariance matrices Σ (ET) E Then the objective set to
Σ
Σ (
c1s $ Ko))].
(0)
(3)
K
19

---

Published as a conference paper at ICLR 2024
where the first term defines the log-likelihood of (0) coming from Ke as L(8) (0) ; (6
Mok)" (s," ne) + log det _-- / 1og(( with Me being the empirical mean of cluster Ke. the second term [1 & Ke} penalizes the adjacent events that are not assigned to the same cluster and constant balancing off the scale of the two terms. This optimization problem can be solved using the pnactaaionnmaxmmization family of algorithms by updating Σ and K
alternatively Hallac al. (2017).
A.4 DETAILED FORMULATION OF THE VAB-MDP
The latent prior p(z²) .N(0, 1) representing the distribution of the initial latent states (at the beginning of each PST in the set T where I is the identity covariance matrix.
The encoder ( 4-1, is used approximate the posterior distribution 1113-19--11000)
01- p,(2:1s--1,0--115
Ja p(a, 1:0, (Aua)ddy where Z C 1T and m is the dimension.
Given that both distributions qu (2 and (2
(cc-w-22)) = (=(l()))) II (w 11-1, 8t),
follow diagonal Gaussian, where mean and diagonal covariance are determined by multi-layer perceptrons (MLPs) and long short-term memory (LSTM), with neural network weights Thus, one can infer z (=clsc)) 2 a= (=))))) with by fo (hy 11-1, where fo represents LSTM layer and hy represents LSTM recurrent hidden state.
The decoder jectories.
P7l St, 1 1112222211,
is used to sample py Pr(3ç+1s= - where az's are determined follow-
tra-
Given
IIS#W-1 [1S*W-1,,(stx)) II ing the behavioral policy 3, distributions (sr (z)) and py (12-1/22)) follow diagonal Gaussian with mean and covariance determined by MIPs and Pr, (2,1 follows diagonal Gaussian with mean and covariance determined by LSTM.
Thus, the generative process can be formulated as, i.e., at initialization, = ~ p(zc), 3 ~ (=(ss(=") ac Baacs="); followed by z (T())) ri ~ (r-r)=12²?), 2y ~ (s. (++)=?), at ~ B(a))s)).
with hg (h" ="_ 44)1)] where 9n represents ML.P.
20

---

Published as a conference paper at ICLR 2024
A.5 PROOF OF EQUATION 2
The derivation of the evidence lower bound (ELBO) for the joint log-likelihood distribution can be found below.
(4)
log
(5)
log
Sc+hiGt--IE wwkc44ww-11/66-----04000. au
(2 1 ott.::/w........................
(6)
Jensen's a inequolity
g Dlg p(pp)())))))) + - www.:///h-ngggeeee--ww-r.......
(7)
Egg. [ogg(ze) log Pr (scl2<) 3+=+
Σ C+W-1
E
log Py 2t, re
lo (8c) Σ t-(+1
C+W-1
log q (ze 11
(8)
C+W-1 - log brg
-Eg logp(=c) log e (2 (z<( + log +
> c+W-1
(= log aw (2el (2
(9)
=E,y
>
S-W-1 11
log (og + E 1=5+1 log log
C+W-1
|=
KL(g - -=t+1
-E K (1 st) - ac 11)
C+W-1
(10)
A.6 MORE DETAILS OF FUSING BACK STEP
We have ensured that the transition from the original trajectory to the beginning of the augmented PST's are smoothed, by letting the generation of 8ç, the initial state in the PST (e.g., 83 in Figure 3) to be conditioned on $ç-1 which is the last state in the original trajectory (c.g., 82 in Figure which cquivalently set 5ç-1 the underlying initial state for the generated PST. To smooth the end of PSTs, we select the augmented PSTs with the states and actions that have the least distance to the original states and actions, step (+ W In experiments, we use the Euclidean distance as the measure.
A.7 OAT WITH NON-TEMPORALLI--LIONN TRAJECTORIES
In the main context, for notation conciseness, we use horizon T to denote the length of all trajecto- ries for notational simplicity, since our work is the first one introducing sub-trajectory mining and augmentation to OPE. Note that OAT can work with non non-temporaally-ligee jectories with different lengths and varied start and end times. In the PSTs mining step, the length W is flexible and the TDSSs can generated with a small length (e-g., 2) and expanded recursively until reaching maximum length (e.g., T). Since our goal is to extract PSTs whose TDSSs are shared across trajectories with large supports, a threshold ę is used to bound the supports of PSTs so that the PSTs with the greatest supports can be always extracted. Moreover, in experiment, the Adroit-hume human environments contain temporally non-aligned trajectories (e.g., length of trajectories from door and rel relocate, varied from 223 to 300, and 297 to 527, respectively). More environ- mental details are provided in Appendix C And Appendix C.2 shows the the length of trajectories, and start and end time of PSTs found each trajectory can be varied.
trajectories, i.e. tra-
21

---

Published as a conference paper at ICLR 2024
B EXPERIMENTAL SETUP
B.1 TRAINING RESOURCES
We implement the proposed method in Python. Training of our method and baselines are supported by four NVIDIA TITAN Xp 12GB, three NVIDIA Quadro RTX 6000 24GB, and four NVIDIA RTX A5000 24GB GPUs.
B.2 IMPLEMENTATION DETAILS & PARAMEERS
The cluster number for discrete representation mapping can be determined by silhouette score using training data following HHallac et al. 2017), we perform search among [10, 20] for C' in all datasets and the one with the highest silhouette score is selected. In our experiments, C 18, 10, 19, 16 for fpen, door, zelocate, hammer -hummn, respectively; C 20, 10, 10, 10 for {pen, door, relocate, hammer] -ereeerrrcloneddd respectively; C = 11, 10, 10, 10 for (pen, door, locaate, hammer] haamer)-expe -expert, respectively; 14, 17 for e-learning and healthcare, respec- tively.
The experimental results are obtained with selecting the PSTs using the threshold & [2, the top I, i.e. we use the PST with the highest support of its corresponding TDSS, for easier investiga- tion of the PSTs mining and comparison to other augmentation baselines such as TDA, and present straightforward and general effects of our method. The percentage supports of the selected PSTs, i.e, support(-) //N, are all 2 82% across all datasets and all experimental environments, especially can cover 100% trajectories in all Adroit human tasks, which may further indicates the effectiveness of PSTs mining. We choose the neural network architectures as follows.
For the components involving LSTMs, which include (=x 2t-1; Qt-1, 8:) and Pr (r,(e//e-11 at-1), their architecture include one LSTM layer with 64 nodes, followed by dense layer with 64 nodes. All other components do not have LSTM layers involved, they are constituted by neural network with 2 dense layers, with 128 and 64 nodes respectively. The output layers that determine the mean and diagonal covariance of diagonal Gaussian distributions linear and softplus activations, respectively. The ones that determine the mean of Bernoulli distributions (e.g., for capturing early termination of episodes) are configured use sigmoid activations. For training OAT and its ablation VAE-MDP, maximum number of iteration is set to 100 and minibatch size set to 4 (given the small numbers of trajectories, i.e., 25 for each task) in Adroit, and 1,000 and 64 for real-world healthcare and e-learming, respectively. Adam optimizer is used to perform gradient descent. To determine the learning rate, we perform grid search among {le 3e 3, 3e 4, 5e 4, 7 - 4}. Exponential decay is applied to the learning rate, which decays the learning rate by 0.997 every iteration. For OPE, the model-based methods are evaluated by directly interacting with each target policy for 50 episodes, and the mean of discounted total returns (7 = 0.995 for Adroit, 7 0.99 for Healthcare, 7 = 0.9 for e-Iearning) over all episodes is used as estimated performance for the policy.
B.3 EVALUATION METRICS
Absolute error The absolute error is defined the difference between the actual value and estimated value of a policy:
AE = V V7
(11)
where V represents the actual value of the policy and V.TT represents the estimated value of Regret@1 Regret@ @ is the (normalized) difference between the value of the actual best policy, and the actual value of the best policy chosen by estimated values. It can be defined as:
R1
max V iEI:P
max
VT max V
(12)
jebest(1:P)
EEl:
where best(1 P) denotes the index of the best policy over the set of P policies measured by estimated values Va
22

---

Published as a conference paper at ICLR 2024
Table 3: Summary of Adroit.
Task
State Dim.
Action Dim.
Early Term.
Continuous Ctrl.
Dataset Size
Pen-human
24
Yes
Yes
5000
Door-human
Yes
6729
Hammer-human
46
26
No
Yes
11310
Relocate-hunee
39
30
No
Yes
9942
Pen-cloned
45
24
Yes
Yes
* 10
Door-cloned
39
28
No
Yes
10
Hammer-cloned
46
26
No
Yes
106
Relocate-cloned
39
30
No
Yes
10
Pen-expert
45
24
Yes
Yes
5 10
boor-expert
28
No
Yes
10
Hammer-expert
26
No
Yes
10
Relocate-expert
39
30
No
Yes
10
Rank correlation Rank correlation measures the Spearman's 's rank correlation coefficient between the ordinal rankings of the estimated values and actual values across policies:
(ooo(rankVVT.p)) rank V.P)))
=
(13)
(rank(VV (VT.))( (rank (ra)kkkk)))))))) (V..))
where rank(V 1.p) denotes the ordinal rankings of the actual values across policies, and rank (VT denotes the ordinal rankings of the estimated values across policies.
1:P)
C ADROIT
As shown Figure 13 Adriot (Rajeswaran eet al., (2018) is simulation environment with four synthetic real-world robotics tasks, where 24-DoF simulated Shadow Hand robot is asked to hammer a nail (hammer), open a door (door), twirl pen (pen), pick up and move a ball (relocate) Each task contains three training datasets with different levels of numan-involvements, including full demonstra- tion data from human (human), induced data from a fine-tuned RL policy (expert), and mixing data with a 50-50 ratio of demonstra- tion and induced data (c (cloned). Task proper- ties are provided in Table
C.I DETAILED RESULTS
Figure 13: Four tasks in Adroit environment.
23

---

Published as a conference paper at ICLR 2024
Table Summary results averaging across five OPE methods without augmentation and with each augmentation method in Adroit human.
Pen
Relocate
Rank Corr.
Regret@1
Regret@5
Rank Corr.
Regret
Regret@5
NoAug.
3014
-0.144
0.184
0.03
1956.4
0.204
0.434
0.298
OAT
1042.2
0.276
0.166
0.026
537
0.36
0.42
0.158
VAE-MDP
1527.8
0.226
0.204
0.02
430.8
0.142
0.53
0.256
VAE
1302.4
-0.03
0.334
0.062
834.8
0.04
0.654
0.484
TimeGAN
1538.2
0.006
0.216
0.022
1209
0.408
0.604
0.34
SPAWNER
1817.8
-0.192
0.218
0.144
1560.6
0.338
0.436
0.272
DGW
1578
-0.028
0.292
0.054
1226.8
0.164
0.434
0.294
Permutation
1548.6
-0.132
0.27
0.076
1628
0.338
0.51
0.152
Jittering
1632.8
-0.096
0.202
0.076
1407.4
0.038
0.574
0.168
Scaling
1308.2
-0.02
0.382
0.076
1462.8
0.244
0.72
0.212
TDA
1030.4
-0.116
0.262
0.06
832.2
0.182
0.608
0.496
Hammer
Door
MAE
Rank Corr.
Regret 1
Regret@5
MAE
Rank Corr.
Regret
Regret@5
NoAug-
S266
0.344
0.34
0.058
603.8
0.14
0.274
0.004
OAT
3187.8
0,608
0.084
0.018
477.8
0.676
0.152
0.026
VAE-MDP
3418.6
0.02
0.454
0.126
497.2
0.336
0.224
0.052
VAE
3733.2
-0.198
0.47
0.104
642.8
0.482
0.288
0.046
TimeGAN
4681
0.262
0.34
0.24
507
0.392
0.27
0.03
SPAWNER
4244.8
-0.156
0.55
0.37
687.6
0.146
0.37
0.186
DGW
5238.8
0.242
0.296
0.144
578.2
0.342
0.134
0,052
Permutation
4103.2
-0.202
0.534
0.076
583.4
0.264
0.236
0.088
Jittering
4256.4
0.004
0.452
0.102
700.2
0.268
0.342
0.056
Scaling
3832.4
0.166
0.404
0.102
580.8
0.222
0.326
0.078
TDA
3448
-0.298
0.56
0.078
$11.2
0.118
0.376
0.112
24

---

Published as a conference paper at ICLR 2024
Table 5: MAE results of OPE without and with each augmentation method in Adroit human n en- vironment, Results are obtained by averaging over random seeds used for training at discount factor of 0.995, with standard deviations shown after +
FQE
pen
door
relocute
hammer
MB
pen
door
relocate
hammer
NoAug.
140
389:60
593:1113
6000:1612
1218:223
403-1:18
3534:21
3778-1
OAT
1231::33
464
742:17
3500=42
316=112
461:14
298
3372:1288
VAF-MDP
843++44
498+10
419+6
3358
541-310
493++3
423-1
2912+13
TimeGAN
919-:141
459.99
962.196
4353-1-495
1063:1450
4354:34
413-1
58111997
VAB
6144-9
531+77
520+22
3746=16 16
615+14
530+33
421-1
373449
SPAWNER
1508::52
504235
735-1131
6529:1172
110710
473:144
698_4
3561+11
7924:222
320::20
787469
9578:1431
1278:1175
430+:29
810=6
3779::37
Permutation
9241:14
520141
542410
4069=56
998-0
48314
754=4
3324111
Jittering
1092=105
403-330
746::0
552611333
1064119
5391:1
512:100
3257=103
Scaling
875:152
360:20
624
4599
961
478:15
524188
3627-163
TDA
1185::29
4691:
805+10
3407=22
471=2488
470:14
810=4
3402+11
pen
door
relocate
bamunrr
DR
pen
door
relocate
hammer
NoAug.
3926-:128
870-+173
192
7352+1118
2846:4:200
379:66
606:1:116
5768+751
OAT
1364:-97
508:15
436:22
3779=32
1015+1186
494::
450-:227
3815:
VAE-MDP
1315+388
499++6
437+4
3678-83
2954+883
502+10
451+-23
3811+26
TimeGAN
1752:2212
591=12
1995::5
5683=12
13524-282
4944:70
667-1122
4224::138
VAE
1896:187
513:55
93017
3628:114
784=155
51517
545:116
3832-1166
SPAWNER
2769-10
1007=9
2871111
3567=10
870-1:99
4501:86
591::69
4008:1444
DGW
2360-10
541::10
534.188
5289=11
861.199
5451:55
573:1388
4270192
Peruutation
2433:111
$20::13
3093-120
5332-11
787-1499
368::42
613:884
4467-1255
Jittering
2350-t0
1114=22
21114:28
5334=10
1058+117
4194:12
5194-28
38414-774
Scaling
1284:140
5231-8
2118:172
3710=16
822=1322
525-17
642-1:99
3892-1239
TDA
1269+101
572+5
882+25
+34
991+193
477++11
857+ 156
3613+2611
DICE
pen
door
relocate
hammer
NoAug.
3208-1-22
978=100
4304118
3432-16
OAT
1285
462:t55
759+:14
1473:12
VAE-MDP
1986:1-0
494.15
424113
3334-19
TimeGAN
2505:1:5
556:16
2008:115
3334:1.
2603-13
1125:111
1758:110
3726=18
SPAWNER
2835++11
1004++10
2908++49
3559-12
DGW
2599-10
1055:110
3430-163
3278-19
Permutation
2601:+22
1026++11
3138+449
3324=10
Jittering
2600:11
1026:111
3149151
3324=10
Scaling
2599==0
1018+111
3406+60
3334=10
TDA
1236-18
568.15
807114
3400=11
25

---

Published as a conference paper at ICLR 2024
Table 6: Rank correlation results of OPE without and with each augmentation method in Adroit
human environment. Results are obtained by averaging over 3 random seeds used for training at discount factor of 0.995, with standard deviations shown after =.
FQE
pen
door
relocate
bammer
MB
pen
door
relocate
hammer
NoAug.
0.31+0.21
0.07::0.99
0.62:40.11
0.14+
-0.1244.33
0.13--013
0.16=0.10
0,29:40.23
OAT
-0.02440.65
0.55::0.00
-0.05=0.55
0.34:0092
0.67=0.08
0.99=0.01
0.40=0078
0.62::0.47
VAE-MDP
-0.0240.64
0.68:00.4
-0.66+00004
-0.44+0.72
0.12:.0334
0.14=+0.8
0.71-0.16
0.29:40.00
TimeGAN
-0.17+00.28
0.39::020
0.76=0.07
0.37:20.06
0.52=0048
0.18=0011
-0.0240.19
0.16:0.23
VAE
-0.5210.38
0.3911020
-083:10.11
-0.6110.46
0.461:0.28
0.93:10.02
0.25:0.30
0.11100.7
SPAWNER
0.12:10.00
0.4410.117
0.62:10.11
0.1110.31
0.13
0.04:10.79
0.19:0.60
0.82:00.2
DGW
-0.1240.25
0.474-0.11
0.17:00.6
0.47=00.16
-0.02:0..55
0.29::0.19
0.35--031
0.20+0.19
Permutation
0.17.10.18
0.501:0.05
0.49:10.99
0.48 0.03
0.18:10.56
0.02:10.74
0.23:0007
0.85
littering
-0.17440.1
0.45+00.0
-0.33-+0.78
-0 -0.99++0.2.
0.0540004
0.31+0001
0.17-0442
-0.22-0.20 0.20
Scaling
0.36=0.24
0.53+0.06
0.40=00115
0.35:20.34
-0.35=0.38
0.21=0065
0.24=0.33
0.35:+0.15
TDA
0.26:10.68
-0.4100.35
-0,2710.89
-0.12:1.52
0.09:00.99
0.72:0.11
0.71:20.12
-0.21=0.60
pen
hammer
hammer
NoAug.
0.28:1028
0.12::0.35
0.23
0.39
0.36::0.29
0.01
0.65
0.04-10,25
OAT
0.57-10.1
0.81::0.06
0.82
0.75+t0.14
0.04::066
0.45+0046
0.29-0070
0.7440.16
VAE-MDP
0.48=0.36
0.12#00.1
0.22=0.70
-0.28=0.51
0.3420.68
0.50=0.11
0.72=0.18
-0.35=0.79
TimeCAN
0.01110.80
0.38:10.75
-0.8511.011
0.200:20.4
0.53:10.10
0.65=0.08
0.40:10.03
VAE
0.65:0.00
-0.31+0.00
0.31
0.11=00.64
0.29=0.29
+0.04:0.69
0.06:t0.75
SPAWNER
-083:00
0.81:10116
0.09:10.20
0.36:10.26
0.62:00.9
0.47.10.16
DGW
0.03.10.61
0.82
0.251
0.631...5
Permutation
0021140.74
0,37++0.56
-0.70++0.00
0.11-+0.37
0.57+00.14
0.38-0031
0.2740033
Jittring
-0.2840.79
0.124:0.88
0.85=0000
0.23+0073
-0.88=0.51
0.64:00188
0.0440.42
0.18+0884
Scaling
0,24++0.42
0.65-+0.0
0.66-+0.11
-0.15+0.67
0.26+-0.45
0.09-0.67
0.09440.7
TDA 0.11540.53
0.37:40.28
-0.67+0.00
0.81
-0.1120.3
0.18:2034
00,22+0.85
40.28=0.79
DICE
pen
door
relocate
bammer
NoAug.
0.01-+0.39
0.61--0.34
18.1.:44.5
0.94+-0.01
OAT
0.12=0.70
0.58:10.29
0.35=1088
0.52+0.67
VAE-MDP
0.21=0059
0.24++0.53
0.02-10.68
0.38++0.73
TimeGAN
-..11:..99
0.48::0.09
0.27=0.84
0.38+0.73
VAE
0.08:0045
0.151:0.74
0.17:0.76
-0.2420.16
SPAWNER
0.29110.59
.0.11.10.71
0.28
0.32.10.58
DGW
-0.02+0.554
0.35++0.57
0.24++0.87
00.55+0.28
Permutation
0.02.10.64
0.14.10.59
0.28
0.21
Jitering
0.00+0.588
-0.18-0.60
0.27+0084
-0.50-0..5
Scaling
0.04:0.67
-0.13=0.58
0.25=0.87
-0.61+0.30
TDA
-0.1510.64
-0.2720.62
0.32:10.1
-0.2110.84
26

---

Published as a conference paper at ICLR 2024
Table 7: Regret@) results of OPE without and with each augmentation method in Adroit human environment. Results are obtained by averaging over 3 random seeds used for training at a discount factor of 0.995, with standard deviations shown after +
FQE
pen
door
relocate
hammer
MB
pen
door
relocate
hammer
NoAug.
0.07=0.05
0.05=0.08
0.17:10.14
0.46:10.23
0.15::0.15
0.44:10.22
0.73=20.36
0.15::0.17
OAT
0.20-0026
0.26+-0.36
0.99+00.1
0.34+00.8
0.12+00.17
0.00-00.00
0.35+0.46
0.00
VAE-MDP
0.38:0021
0.01:0.01
1.00:00011
0.8110.30
0.1310.11
0.39:10.45
0.41:0.28
0.42:10.16
TimeAN
0.19=0.13
0.23+0.13
0.39::0.24
0.15:1017
0.22::0.25
0.51+00.36
1.00=0.00
0.19:0.24
VAE
0.57-00.00
0.05++0.05
1.00+-0.01
0.73++0.26
0.09+-0.06
0.00=:001
0.03=0002
0.34++0.48
SPAWNER
0.03=0000
0.01-0000
0.27:10.32
0,94:10.11
0.12::0.12
0.35:10.88
0.41:0.43
1.02+:0.00
0.67+00.7
0.14
Permutation
0.37=0.16
0.15#0.16
0.87+10.11
0.05:1:.03
0.19:10.13
0.37+00.7
0.09=0.09
1.02::0.00
Jittering
0.11=0006
0.12+0.00
0.67::0.77
0.73:10.26
0.08:10.07
0.63:10.10
0.48:0.41
0.4510.31
Scaling
0.13:0016
0.1710.11
0.8110.12
0.25:1022
0.50:10.10
0.68+1048
0,94=0.04
0.1410.08
TDA
0.35=0025
0.79=0.29
0.68::0.5
0.81=00.0
0.26:10.23
0.12+0.110
0.31
0.72+0.37
IS
pen
door
relocate
hammer
DR
door
relocate
hammer
NoAug.
0.17+0.15
0.45+0.40
0.63+-0.41
0.19-+0.30
0.09:00.00
0.05++0.09
0.17+0.11
0.46++0.33
OAT
0.12::0.17
0.06::005
0.02:10.02
0.00:10.00
0.20:10.19
0.00:10.01
0.37=0.45
0.05:10.55
VAE-MDP
0.05=0.03
0.37=0.47
0.37+10.45
0.30=10.34
0.16:10.22
0.00:10.01
0.05=0.00
0.71=:0.43
TimGAN
0.38-2022
0.13-00.17
1.00:10.00
1.00-+0:02
0.16:1:021
0.13+40.17
0.26=00.33
0.33-+0.32
VAE
0.44:10.19
0.68:1048
1.00:10.00
0.19:10.25
0.361:0.15
0.21
0.74=0.37
0.37::0.46
SPAWNER
0.57+00.0
1,03--0.00
1.00+-0.00
0.34++0.45
0.12++0.08
0.01-44.01
0.00-0.00
0.02+00.25
DGW
0.44:10.19
0.15:0.16
0.00:10.00
0.59:10.28
0.25:10.24
0.03:10.02
0.68:0..4
0.03::0.00
Permutation
0.44:10.19
0.0510.06
0.9110.13
0.741:0.39
0.1011006
0.0810.11
0,31
0.5010.40
Jittering
0.44:10.19
0.34:0.48
0.68:10.45
0.36:20.46
0.10:10.05
0.0910.11
0.54=0.31
0.35
Scaling
0.57=0.00
0.06:0.05
0.67-10.47
0.26:10118
0.39110.99
0.19::0.26
0.68=0.44
0.6810.44
TDA
0.10-60.13
0.05-00.06
1.00+00.00
0.00:10.01
0.39:10.19
0.35+0046
0.68=0044
0.60+0.43
DICE
door
relocate
hammer
NoAug-
0.44:10.19
0.38:0046
0.47:10040
0.44:10.01
DAT
0.19:0.14
0.4420.09
0.37::0.45
0.03:10.05
VAE-MDP
0.30=:022
0.35:1.15
0.824:0.26
0.03:10.05
TimeGAN
0.13=00.2
0.35++0.1
0.37
0.03
0.21-0015
0.50+0.35
0.50-10.39
0.72-40.37
SPAWNER
0.25:10.11
0.45:0.32
0.50
0.43
DGW
0.21-0011
0.2310.23
0.50.10.39
0.70:10.42
Permutation
0.25-10.11
0,53-0.22
0.37-10.45
0.36:10.4
Jittering
0.28:1015
0.53-0.22
0.5010.39
0.37-:0.3
Scaling
0.32:10.20
0.53-0.22
0.50-10.39
0.69-10.1
TDA
0.21:10.15
0.57:00.16
0.37
0.67++0.7
27

---

Published as a conference paper at ICLR 2024
Table 8: Regret@5 5 results of OPE without and with each augmentation method in Adroit human environment. Results are obtained by averaging over 3 random seeds used for training at a discount factor of 0.995, with standard deviations shown after +
PQE
pen
door
relocate
hammer
MB
pen
door
relocate
hammer
NoAug.
0.01:0002
0.00:0001
0.75:10.22
0.13:10.09
0.03:10.04
0.02:10.03
0.03=00.2
0.01:10.01
OAT
0.01-0002
0.13+0018
0.15+0021
0.07++0.09
0,00--0.00
0.00-:0.00
0.02+00.2
0.00
VAE-MDP
0.06:0008
0.00:0000
0.871:0.10
0,13:10.09
0.01:10.01
0.08:10.12
0.00=0000
0.17:00.55
TImeGN
0.03=0004
0.00::001
0.47:10.35
0.13+10.99
0.00:10.00
0.01+:0.01
0.02=0.02
0.02+0001
VAE
0.06-00.08
0.00-0000
0.75+-0.22
0,13++00.99
0.00:-0.00
0.00-:0.00
0.00=0.00
0.07:+0.09
SPAWNER
0.02:1001
0.01=0000
0.00:10.00
0,01:10.01
0.02+10.01
0.19:10.26
0.07=0.10
0.17:20.05
0.15+00.10
Permutation
0.00:0000
0.00:0.00
0.02:10.02
0.01:10.01
0.10:10.06
0.19:10.26
0.00=0.00
0.23:10.12
Jittering
0.00=0.00
0.00:0.00
0.45
0.02:10.01
0.08:1007
0.01
0.07=0.10
0.05
Scaling
0.00=0000
0.01:0.01
0.03110.2
0.00:10.01
0.0811007
0.08+10.12
0.07=0.11
0.01110001
TDA
0.07=0.08
0.27
0.56:10.41
0.01
0.10:10.06
0.00+00.00
0.07=0.10
0.15
IS
pen
door
relocate
hammer
DR
door
relocate
hammer
NoAug.
0.00
0.00
0.07-+0.10
0.01-+0.02
0.10+H0.08
0.00
0.32-00.46
0.14+0.18
OAT
0.00=:0.0
0.00:0000
0.00:10.00
0.00:10.00
0.06:10.08
0.00:10.01
0.32:0.46
0.01:10.01
VAE-MDP
0.01:0001
0.09
0.30:10.43
0.14:10.18
0.01:10.01
0.00:10.00
0.02=0002
0.1610.16
TimeGN
0.01=:0.01
0.08-001
0.97-10.00
1.02-10.00
0.06:10.08
0.00++000
0.09=00.99
0.00
VAE
0.19=10.7
0.00:0.00
0.95
0.13:20.18
0.05:10.06
0.08:10.12
0.40=0.41
0.1410.18
SPAWNER
0.57+00.00
0.56+-0.00
0.97-+0.00
1.02++0.00
0.04++0.04
0.014-0.01
0.00-0.00
0.60+0043
DGW
0.19:10.27
0.15:00.16
0.00:10.00
0.46:10.40
0.01:10.01
0.01:10.01
0.34=0.45
0.00
Permutation
0.19=:027
0.04=0.06
0.44:10.00
0.00=10.00
0.03110.00
0.00:10.00
0.00=0.00
0,00110.01
Jittering
0.19=:0.27
0.00:0000
0.00:10.00
0.15:10.17
0.05:10.06
0.00=10.01
0.02=0002
0.1
Scaling
0.18=00.0
0.05=0.06
0.32:10:46
0.01=10.01
0.06:10.08
0.08:10.12
0.34=0.45
0.34+0.5
TDA
0.01::0.22
0.00:00011
0.97:40.00
0.00-10.00
0.06:10.08
0.00::0.00
0.56=00.1
0.16+0.16
DICE
door
relocate
hammer
NoAug-
0.01::001
0.00:10.01
0.32-1:.42
0.00:10.00
DAT
0.0620.08
0.00=0.00
0.30:1::43
0.01:10.22
VAE-MDP
0.01::0.11
0.09:0011
0,09:10.09
0.03-10.05
TimeGAN
0.01
0.06
0.15++0.21
0.03++0.05
0.15+t0.16
0.32-4-.42
0.05-10.03
SPAWNER
0.07=00.8
0.16::0.22
0.32:10.42
0.05
DGW
0.03:10.04
0.0910.11
0.30:10.43
0.15:10.17
Permutation
0.06=0008
0.21+0015
0.30-10.43
0.14:10.18
Jittering
0.06=0.08
0.27=0.22
0.30:1..3
0.16110.16
Scaling
0.06-10.00
0.17+0.11
0.30:10.43
0.15-10.17
TDA
0.06=00.8
0.29:0.21
0.32-10.46
0.07++0.05
28

---

Published as a conference paper at ICLR 2024
Table 9: MAE results of OPE without and with each augmentation method in resampled Adroit cloned environment. The data are randomly sampled from original training data as the same data points as the corresponding task in human environment. Results are obtained by averaging over random seeds used for training at discount factor of 0.995, with standard deviations shown after
IQE
pen
dour
relocate
hammer
MB
pen
door
relocate
hanmer
NoAug.
715=11
359-116
3711-5
3714-134
6224:102
1009-92
357
411947
OAT
663-124
336+7
438+:3
3780=20
607=2
537=3
339+1
3743=188
VAE-MDP
323-9
539+6
384+-4
3624=16
813+6744
538-
386
3611+-4
TimeGAN
1237-25
572++7
539+4
5268+24
1065+118
573 573-4
541+1
5268+115
VAE
412-8
56116
384422
3823+14
409:119
561=3
385+1
3814+10
SPAWNER
1173#0
502+:5
443+2
3988=6
1173:10
502-3
446+
3988+
DGW
1176+00
550+55
864+8
4127+6
1176+0
550
871+4
4127+6
Permulation
1176:10
5371:5
822:111
4128::6
1176.10
536
832.14
4
littering
1176.10
537.15
820
4128:16
1176:1
536-3
833.14
4128:16
Scaling
1176210
51711
6961:5
412316
117611
517-3
70514
412316
TDA
610=4
526::
43842
3746=16
59910
527=3
44111
3715=14
IS
pen
door
relocate
hammer
DR
pen
door
hammer
NoAug.
636::21
1072:224
458-13
8162-91
731+115
458+34
475::22
6719+118
OAT
259-9
563+:3
369+-2
3905#99
4224-9
390++41
371+8
166
VAE-MDP
300++177
507+4
406+11
3686+447
493++76
$42+20
415+40
+98
TimeGAN
978::192
57611
695-22
$325.185
7711-191
606.118
696.1.26
131
VAE
300:1:1
517.12
43
3522-11
386.119
$40122
437.120
3764180
SPAWNER
117310
61113
506:11
3989-6
117410
527114
509114
39901-5
DGW
1176:20
56743
114922
412926
117621
597423
1180
41291-5
Permutation
1176:t00
557+33
1107=4
4129=6
1176+0
592+225
1110=
4129+-5
Jittering
1176110
5571:3
1107-4
412916
117610
588124
1115189
41301-5
Scaling
1176#0
5431:3
945:14
412416
117620
580131
968:33
4125:55
TDA
604=6
501+5
445=10
3725-11
261=12
539434
4534-9
3765+142
DICE
pen
door
relocate
hammer
NoAug.
1218-41
1138.17
1841115
3752-18
OAT
1166=38
536113
441:1
3750-11
VAE-MDD
778=5
538::4
160628
3614=115
TimeGAN
1276-42
573+-4
1925+110
5252+118
VAE
1020-14
561+5
10
3813++20
SPAWNER
1173:10
1067:7
185624
398811
DGW
1176:20
1140:133
8694-8
4127-6
Permutation
1176++0
1140+7
832+55
4128+6
Jitteingg
1176:1:0
1140:17
83118
412811
Scaling
1176.10
1098.17
704.88
412316
TDA
601-0
1119419
183526
3731-115
29

---

Published as a conference paper at ICLR 2024
Table 10: Rank correlation results of OPE without and with each augmentation method in resam- pled Adroit cloned environment. The data are randomly sampled from original training data as the same data points as the corresponding task in human environment. Results obtained by eraging over 3 random seeds used for training at discount factor of 0.995, with standard deviations shown after =.
FQE
pen
door
relocate
hammer
MB
pen
door
relocate
hammer
NoAug.
0.29-00.19
0.22+-011
OAT 0.26.10.32
0.9510.02
0.48.10.51
0.38
0.31 0.54
0.77-00.5
0.52.0.15
0.38
0.26+-0.52
0.05+-0.61
TimeGAN
0.3910.43
0.54100.8
0.47.0..41
0.01
0.12.10.63
0.58
0.46
1..30
VAE
SPAWNER
0.19100.8
0.1810.59
0.32..0.73
0.02.10.79
0.05
0.62..0.06
0.7910.19
0.40
0.09--0.25
Penmutation
-0,36=20.69
-0228=0.24
0.12:+0.59
-0.29+:0.79
0.15=0.225
0.60=0.13
0.43=0.75
0.51=0.32
Jittering
-0.474-0.49
0.45+0.44
Scaling
10.19=0.80
0.63=0022
-0.25=00.64
-0.26+0088
0.03=0.62
0.45=0.30
0.67=0.29
0.66=0.16
TDA
0.26++0.77
0.37++0.26
0-887
0.35
0.61-0.27
-0.45+0.62
IS
door
relocate
hammer
pen
door
relocate
hammer
NoAug.
0.00=0.00
0..22=0.59
0.25+0.54
0.79=+0.04
0.42=0.24
0,46=0.72
-0.2240.70
0.29=0.49
OAT
0.78=0.00
0.96:0001
0.99--0.0
0.77+00.1
-0.38+00.00
0.23-00.5
0,33-0.78
0.62-0037
VAE-MDP
0.15=0028
0.55=0.06
0.96=0004
-0.02:+0.74
-0.15=0.60
-0,57:20.06
-0.02.0.1 19
-0.23:0.82
TimeGAN
0.8920.00
-0.7820.27
0.50:10229
0..1010.62
-0.1810.78
VAE
0.61=0.11
-0,23=0066
0.88.0.1 10
0.98=10.11
-0.4220.34
-0.10:20.2
-0.0220.70
-0.20:20776
SPAWNER
0.02-0.64
-0.66:0.64
DGW
-0.45=0.00
-0.67=20.00
-0.05=0.32
-0,3620.31
0.09920.33
-0.20:2.77
Permutation
0..0020.00
-0.67:0.000
0.20110.71
littering
-0:42:20.02
0,31120.26
-0.33:20.39
0.03=0.74
0.0620.78
Scaling
0.67-10.00
-0.24:0.44
0.07:0.55
-0.2010.73
TDA
0.372042
0.41
-0.48=0.355
0.79-0.13
-0,22+0.37
-0,32:20441
-0.05=0.70
-0.25.0.799
DICE
pen
door
relocate
hammer
NoAug.
0.0320.56
0.39
0.05:10.52
0.31
OAT
0.24
00.57
0.36
0.64
VAE-MDP
-0.1100.38
0.26110.5
0.82:10.17
0..81100.79
TimeGAN
0.36.10.36
0.061-0.70
0.77
0.431-0.62
VAE
0..8810.45
0.18110.79
0.30:10.1
-0.11.10.64
SPAWNER
0.30:10.33
0.66
0.18
0.19.10.73
DGW
0.16:10.10
0.38110.88
0.22-10.85
-0.3610..4
Permutation
0.24::0.43
00.8
0.96 0.02
0.20
Jitterig
-0.39110.61
0.37::0.43
-0.30:0.822
-0.1410.77
Scaling
0.14::0.60
0.40::0.55
0.14:10.81
0.16:1-0.70
TDA
-0.02-40.56
-0.11=0.61
0.30++0.3
+0.31+0.78
30

---

Published as a conference paper at ICLR 2024
Table 11: Regret@ results OPE without and with each augmentation method in resampled Adroit clored environment. The data are randomly sampled from original training data as the same data points as the corresponding task in human enviromment. Results are obtained by averaging over random seeds used for training at discount factor of 0.995, with standard deviations shown after
FQE
pen
door
relocate
hammer
MB
door
relocate
hammer
NoAug.
0.10=00.33
0.07=00.04
1.00:00.11
0.03:10.01
0.06:1:.06
0.19:1..1
0.34=0.45
0.21
OAT
0.47+0.00
0.01
0.33++0.47
0.34++0.48
0.10:00.13
0.04++0.03
0.48-00.37
0.13+0.188
VAE-MDP
0.30=0022
0.1310.17
0.67100.7
0.26:1.37
0.20:10.26
0.7911.11
0.02:0.02
0.39:0.31
TimeGAN
0.41=:0.33
0.39::0.5
1.00::0.00
0.40=+0.44
0.31:1:022
0.42:0044
0.35=00.6
0.08
VAE
0.57=0.00
0.4620.41
0.661:0.77
0.47:10.39
0.12::0.66
0.08:10.12
0.33=0.47
0.9410.11
SPAWNER
0.20=:0226
0.44:0.32
0.67+10.77
0.37=10.46
0.13::0.16
0.21
0.26=0.33
0.26:1.337
DGW
0.05
0.43+00.33
0.99+F0.02
0.68++0.48
0.57++0.00
0.52+0.41
0,90+0012
0.01++0.02
Permutation
0.38-0027
0.72+00277
0.741-0.6
0.60+10.43
0.57+:0.00
0.00-10.00
0.40=00.22
0.34
Jittering
0.35=:0.25
0.09:0.11
0.67
0.47=10.39
0.14:1:.11
0.02+00.33
0.81
0.01
Scaling
0.35=10225
0.09
0.67:10.47
0.60:20.43
0.24:10.24
0.19:1..26
0.59=0.39
0.01+0001
TDA
0.35=:022
0.19=0.26
0.67-10:47
0.68:20.48
0.174:0.21
0.47=00.2
0.40=0042
0.81
pen
door
relocate
hammer
DR
pen
door
relocate
hammer
NoAug.
0.07
0.47+-0.22
0.68++0.45
0..48
0.22+-0.25
0.26+10.3.
0.68-00.5
0.26+-0.37
OAT
0.19:0027
0.05::0066
0.00:10.00
0.00-10.01
0.28:10.20
0.2611036
0.37=0.45
0.00
VAE-MDP
0.19=00133
0.07=0004
0.00:10.00
0.35:10.47
0.1710010
0.93+0.11
0,82=0.26
0.37:00.6
TimeGAN
0,57=0.00
1.02=0.00
0.02:10.02
0.85:10.09
0.50:10.05
0.93:1..11
0.82=0.26
0.37::0.46
VAE
0.35=10.24
0.68:10.88
0.00:10.00
0.34:10.46
0.4410.19
0.6240.41
0.67=0.47
0.37::0.46
SPAWNER
0.37+00.46
DGW
0.57=00.0
1.03=0.000
1.00:10.00
1.02:10.00
0.35:10.24
0.93:1.111
0.68=0.45
0.37:10.46
Permutation
0.57+00.00
1.03+0.00
1.00++0.00
1.02-+0.00
0.35++0.25
0,93+0.11
0.68+0.44
0.37+40.46
Jittering
0.57-0.00
L0310000
1.00110000
1.0210.00
0.54110.55
0.6210.41
0.73-0.36
0.37:0.46
Scaling
0.57=0.00
1.03-0.00
0.91=0013
1.02=0.000
0.17:10.21
0.62+0.41
0,81
0.37=0046
TDA
0.17+0021
0.08-0006
1.00+-0.00
0.07++0.09
0.25++0.24
0.62+00.1
0.68 0.68
0.37-40.46
DICE
pen
door
relocate
hammer
NoAug-
0.26:10118
0.34:1.48
0.73:10.36
0.34:10.48
OAT
0.28:1021
0.35=0.48
0.33+10.47
0.03:10.05
VAE-MDP
0.36:10118
0.3620.30
0.03:10.02
0.67=10.47
TimeGAN
0.44:0011
0.6010.25
0.111:0.08
0.68:10.48
VAE
0.31+40.24
0.50+0035
0.35-H0.46
0.65-10.46
SPAWNER
0.26:10.23
0.34=0.48
0.65:10.43
0.40-10.44
DGW
0.29-+0233
0.36+-0.47
0.41+-0.33
0.53++0.35
Permutation
0.50:10.10
0.94:0.12
0.02:10.02
0.40-10.44
Jittering
0.38-1027
0.02:0003
0.68:10.45
0.47:10.2
Scaling
0.50=:0.10
0.34:00.8
0.73400.36
0.40-10.4
TDA 0.06
0.60:1025
0.37:10.45
0.68
31

---

Published as a conference paper at ICLR 2024
Table 12: Regret@5 results OPE without and with each augmentation method in resampled Adroit cloned environment. The data are randomly sampled from original training data as the same data points as the corresponding task in human environment. Results are obtained by averaging over random seeds used for training at discount factor of 0.995, with standard deviations shown after
FQE
pen
door
relocate
hammer
MB
door
relocate
hammer
NoAug.
0.00=0000
0.00=00.00
0.00-100.00
0.00:10.01
0.00:10000
0.00:10.01
0.02=0002
0.01=:0.01
OAT
0.01
0.00
0.02
0.03-+0.05
0,01
0.00
0.02=00.22
0.01++0.01
VAE-MDP
0.07=0.08
0.08:0.12
0.3410.45
0.03+10.05
0.00:10.00
0.02:10.03
0.00=0.00
0.07:10.09
TimeGAN
0.06=00.8
0.00:0.01
0.36:1:0.43
0.07=10.09
0.01:10.02
0.00:10.00
0.00=0.00
0.00
VAE
0.08=0.07
0.00::001
0.45:10.37
0.08+10.09
0.011:0.22
0.00:10.00
0.02=00022
0.0110001
SPAWNER
0.07=0008
0.13:0.17
0.67+:0.7
0.08:10.09
0.05::0.06
0.00::0.00
0.07=0.10
0.03-10.05
DGW
0.31+4024
0,23+-0,24
0.87++0.10
0,16++0.16
0.19++0.27
0.00+00.00
0.35+00.45
0.00
Permutation
0.38-0.27
0.09+0.11
0.15+:0.21
0.16::0.16
0.10:10008
0.00=10.00
0.24-0034
0.00:00.00
Jittering
0.07=:0.08
0.08+00.1
0.17+:0.20
0,16
0.01
0.00
0.64=0031
0.01-+0.01
Scaling
0.35=:0225
0.02+0003
0.39:10.30
0.16::0.16
0.07110.08
0.08:10.12
0.00=0.00
0.00
TDA
0.07=0.08
0.04+0.06
0.17+10.20
0.10=+0.08
0.01::0.02
0.00=10.00
0.09=0.09
0.14:0008
pen
door
relocate
hammer
DR
pen
door
relocate
hammer
NoAug.
0.18+0000
0.25+-0.00
0.63++0.44
0,13++0.18
0.00--0.00
0.13+00.18
0.56-0..4
0.00
OAT
0.00=0.00
0.00:0.00
0.00:10.00
0.00:10.00
0.05:10.06
0.13:10.18
0.24=0.34
0.00
VAE-MDP
0.01:0011
0.00:0001
0.00:10.00
0.14:10.18
0.091:0.06
0.25:10.18
0.02=0002
0.16::0116
TimeGAN
0.02=:002
0.1620.12
0.00:10.00
0.01
0.10:10.08
0.25:10.18
0.40=0.41
0.1410.11
VAE
0.10=0008
0.27+0.23
0.00-10.00
0.00=10.00
0.09:10.06
0,19:10.26
0.32=00.66
0.14
SPAWNER
DGW
0.18=10.00
0.56:0.00
0.97:10.00
0.39:10.00
0.00:10.00
0.15:10.16
0.34=0.45
0.14
Permutation
0.18++0.0
0.56+00.00
0.97++0.00
0.39++0.00
0.05++0.06
0.15++0.16
0.56-0..1
0.14+0.118
Jittering
0.18=10.0
0.5610.00
0.97110.00
0.39:10.00
0.01110.11
0.13:10.17
0,24=0.34
0.13
Scaling
0.18=10.00
0.5620.00
0.11::0.88
0.39=10.00
0.05-10.06
0.13:10.17
0,56=0.41
0.14
TDA
0.01+40.01
0.04+0.06
0.65++0.46
0.00-10.00
0.05++0.06
0.13-10.17
0.34-0.45
0.16++0.16
DICE
pen
door
relocate
hammer
NoAug.
0.05=:0.06
0.02:10.03
0.00:10.00
0.03:10.05
OAT
0.01:0001
0.35=00.48
0.30:10.43
0.01:10.02
VAE-MD MDE
0.06:10.04
0.13:0.117
0.00:1:0.00
0.07=10.05
TimeGAN
0.11=2007
0.1910.26
0.00-10.00
0.08:10.09
VAE
0.03-:0.04
0.13--0.18 18
0.15++0.21
0.07-+0.05
SPAWNER
0.02::002
0.13:10.15 18
0.56:10.41
0.07=10.05
DGW
0.01-:0.01
0.00--0.00
0.30++0.43
0.07++0.04
Permutation
0.00::0.00
0.13:10.18
0.00:10.00
0.07=10.05
Jittering
0.04-0004
0.00::0.00
0.60:10:43
0.05+00.04
Scaling
0.03-:0.04
0.00:0000
0.32--0.46
0.07+40.05
TDA
0.06::0.04
0.23
0.30:1:043
0.07=10.05
32

---

Published as a conference paper at ICLR 2024
Table 13: MAE results of OPE without and with each augmentation method resampled Adroit expert environment. The data are randomly sampled from original training data as the same data points as the corresponding task in human enviromment. Results are obtained by averaging over random seeds used for training at discount factor of 0.995, with standard deviations shown after
FQE
door
relocate
hammer
MB
door
relocate
hammer
NoAug.
1101
1751::52
1729:1557
2822=756
2363: 135
708-188
221
131 131:0:5335
OAT
1254+21
1213-4-55
419+-30
3599=155 159
1212
1233-13
444+2
3383 193
VAE-MDP
467.5
793-36
4681-5
1983:145
1217=1
7461:10
4741:
9890-1
TimeGAN
2012:124
1008:137
994:111
6317=194
3108
1027:11
1007-1:2
19197=1122
1250:112
903--14
1605=8
20391143
3109=1
912-6
1610:16
100311:210
SPAWNER
1247=11
1264:1477
178::10
4738=157
3108=1
2738:16
778:110
2121819915
DGW
1245+-11
1343+-29
762
7264-117 117
3109+1
2885++66
762++8
36422 +-333
Permutation
1249:112
1270:428
7634-8
6350=104
2725:46
31965-4-28
Jittering
1247#:13
1259-143
17724:19
6267 = 158
3108
2
763-18
29471 76
Scaling
12472:1
1500:149
14
5729= 144
2589=367
1478=13
762218
27187::209
TDA
1251=12
1058:145
3420=186
1217=2
1043-13
12350=1943
pen
door
relocate
hammer
pen
door
hammer
NoAug.
1881 4-23
1005-+22
186344-3
3659-159
1632-4783
970--86
18374-52
3262+-855 856
OAT
939:1:1
1089:13
1136-=5
3354:164
597:174
1044=51
2078:186
2989:1614
VAE-MDP
676:124
871:23
1858-142
2225=110
1260=2299
886:171
1837#52
2494=190
TumeGAN
833:1:43
1054:144
1858-142
5929-183
1299=1452
972=7
1837-52
5029 1100
VAE
15794338
1480:141
17411446
248 209
2149=4777
634:163
1734
2179: 158
SPAWNER
4128++6
2384+3309
DGW
2793:1-9
2571:16
1496:116
6935-16
3109=3477
1172=533
14
6556:1643
Permutation
2675+112
2386++6
1497+116
5-20+-6
29974344
1081 -61
1500+117
5380+780
Jittering
2675211
23861:6
14991117
58211:6
3038:1449
1081=50
1503:117
5465-101
Scaling
2683+112
2858:26
1498:11
532016
3046:192
1288=75
1501::16
4909
TDA
733--227
985+-18
2052++113
3542 138
1325-339 398
963--89
2015 161
3141+902
DICE
pen
door
relocate
hammer
NoAug.
3122::106
1250:121
2369:119
4171:147
OAT
1224=73
1148:110
360-:27
38841146
VAE-MDP
1146::53
812:6
473::33
1996:144
TimeGAN
1943194
570110
6941-5
6637-168
VAE
30144-140
428-+1
387-422
SPAWNER
3067-1:143
12941:7
1524:119
4985:1:30
DGW
3103-4-37
13634-7
14924-16
74114-26
Permutation
3108:163
12841:
1495:116
6510-121
Jittering
31042557
128811
14951:1
6510-130
Scaling
3097#:52
15314-8
14941:16
5970-133
TDA
1193::54
1087:1:7
479-1:2
3697:129
33

---

Published as a conference paper at ICLR 2024
Table 14: Rank correlation results of OPE without and with each augmentation method in resam- pled Adroit expert environment. The data are randomly sampled from original training data as the same data points as the corresponding task in human environment. Results are obtained by av- eraging over 3 random seeds used for training at discount factor of 0.995, with standard deviations shown after =.
FQE
pen
door
relocate
hammer
pen
relocate
hammer
NoAng.
0.19--0.22
0.87+0007
-0.38++0.12
0.29-+0.34
0.24-0.30
0.74-0.15 0.15
0.86++0.05
0.06++0.35
OAT
0.22.10.33
0.97..0.03
0.87
0.39.10.90
0.26.00.69
0.81 0.12
0.89 1.002
0.2310.84
VAE-MDP
-0.18-0.53
0.33+0047
00110+0.51
-0.3++0.00
0.29-0.71
0.86-0.00 0.00
0,30+-0.33
-0.27+0041
TimeGAN
-0.78=00.17
0.94:0.02
-0.1020.51
-0.2020.85
0.07=0.75
0.36=0.75
0,30+0.73
-0.25:20.56
VAE
-0.50-0.35
0.63+-0.25
-0.4740.35
-0.44+00.33
0.09-0025
0.54-0.61
-0.15-0.72
-0.79++0.14
SPAWNER
-0.82=0.10
0.96:00.4
-0.2244.500
-0.31#0.91
-0.40:0.53
0.29=0.60
-0.08=0.61
-0.00=20.68
0.2110.14
-0.96:10.03
-0.3240.91
0.06=0.53
0,27::0.33
-0.0120.72
Pormutation
-0.79=0.13
0.80=0.19
-0.25:10.78
-0.31
00.29:0.700
0.13=0.75
0,14#0.74
-0.65=20.32
Jitering
-0822=0.12
0,92:10099
-0.2110.60
0,58=0.09
0.93:10.05
-0.40:0.66
Scaling
0.82.0.111
0.86
0.09
0..1
0.22
0.77 0.29
0.92
-0.37.1..55
IDA
0.8010.25
0.46=0.52
0.55=0.33
0.72:0.24
0.001:.533
IS
door
relocare
hammer
DR
pen
door
relocate
hammer
NoAug. 0.02.10.48
0.08 10.38
0.9610.00
0.34.10.50
0.85.10.03
0.32 0.18
0.28..0.75
0.33
OAT
-0.59=0.11
0.43+00.5
0.84=0.08
0.97+40.22
0.27-+0.26
0.37-0.11
0,21+:0.64
0.34+-088
VAE-MDP
-0.25=0.32
0.28:0002
0.96
1.00:10.00
-083:+0008
0.410.311
-0.28=0.75
0. 0.15:00.3
TimeGAN
0.89:0.03
0.87=0000
0.96=0.00
0.61+00.1
-0.82+00.5
0.42-0.10
-0228-0.75
0.15+00.0
VAE
0.1940.04
0.75=0.00
0.49:20.45
-0.99+0.02
0.35=0.46
-0.2620.38
0.08
SPAWNER
-0.9920.00
-0.04110664
0.10:10144
0,08=0.28
0.23:10885
DGW
-0.32=00.77
-0.59:10.20
-0.8220.03
-0.16:20.28
-0.55=0.188
0.22+:0.84
Permutation
-0.23:20.45
0.2320.59
0.1810083
littering
0.11.10.59
0.50.10.07
0.90.10.01
0.05_ 0.20
0.75
0,22
Scaling
-0.4620.21
0.27:0.62
0.91100.22
-0.5220.16
0.1910.84
TDA
0,31.10.00
0.59
0.90.1..11
0.80.10.14
0.87.10.00
0.31_0.33
0.02
0.282
DICE
pen
door
relocate
hammer
NoAug.
0.234-039
-0.33++0.89
0.11::0.2
0.006-0073
OAT
0.89:10.11
0.18:0081
-0.22:10.01
-.05::0.72
VAE-MDP
0.87:0007
-0.2311.82
-0.17-10.84
0.20110.85
TimeGAN
0.85:10.05
-0.07=0.61
-0.17:20.88
-0,24=0.86
VAE
0.65.10.19
-0.16:10.58
0.2520.67
SPAWNER
0,55.10.50
0.11
0.16.10.67
0.10....8
DOW
0.86:10.06
0.1820.84
Permutation
0.93.10.00
0.13
0.221
0.284
Jinering
Scaling
0.81.10.11 14
0.25 1.881
0.15
1..7
TDA
0.76-40.08
0.17+0.80
0.23+0057
34

---

Published as a conference paper at ICLR 2024
Table 15: Regret@I results OPE without and with each augmentation method in resampled Adroit expert environment. The data are randomly sampled from original training data as the same data points as the corresponding task in human enviromment. Results are obtained by averaging over random seeds used for training at discount factor of 0.995, with standard deviations shown after
FQE
pen
door
relocate
hammer
MB
door
relocate
hammer
NoAug.
0.17=0014
0.05:10.06
0.91=:0.13
0.05:10.04
0.22-10.25
0.07:10.04
0.15=0.10
0.34:00.18
OAT
0.29
0.01
0.33--0.47
0.34++0.48
0,22++0.19
0.04
0.00=0.00
0.33++0.46
VAE-MDP
0.26:1023
0.3510.48
0.6810045
0.6810.48
0.17100211
0.01:10.01
0.41=0.43
0.91-10009
TimeGAN
0.54::0.05
0.02+00.33
0.68:1:0.45
0.60=:0.43
0.24:1:024
0.34:10.48
0.41=0.43
0.73:#0.26
VAE
0.38=0.27
0.30+1.33
1.00:00001
0.60:10.43
0,29110.00
0.261:0.36
0.6620.47
0.94
SPAWNER
0.46=00.9
0.01
0.66:10.47
0.68:10.48
0.41=00117
0.60:1:043
0.57=0.42
0.20::0.16
DGW
0.46++0.09
0.19--0.26
1.00+-0.01
0.68++0.48
0,20++0.19
0,34+0048
0.41+00.3
0.34+00.88
Permutation
0.37-0.16
0.05:40.06
0.67-10.47
0.68:10.49
0.36-10.23
0.67:40.88
0.57=00.2
0.94
Jittering
0.46:00.9
0.01
0.68+:0.45
0.68
0,07::0.08
0.34
0.16=0008
0.67
Scaling
0.50=0.10
0.01:0001
0.68:10.45
0.68:10.48
0.16:1:.22
0.00:10.01
0.02:0002
0.74
TDA
0.37=0.16
0.01-0001
0.68:10:45
0.68:20.48
0.18:10.14
0.34+00.8
0.41=0.43
0.52+0.41
pen
door
relocate
hammer
DR
pen
door
relocate
hammer
NoAug.
0.14-+0.33
0.94+-012
0.00
0.05++0.04
0.37+-0.16
0.00
0.68-00.4
0.34+-0.88
OAT
0.46:20.99
0.00:0.01
0.02:10.02
0.00:10.00
0.43:1:0055
0.00:10.00
0.07=0.10
0.3410.48
VAE-MDP
0.19=0.13
0.39=0.45
0.671:0:47
0.00=10.00
0.37:10.16
0.00+10.01
0.68=0.44
0.20
TameGAN
0.01:00.0
0.69=0.48
0.67:10.47
0.05:10.04
0.46:10.09
0.02:10.03
0.68=0.44
0.3810.45
VAE
0.57=0.00
0.38=0.46
0.82::0.26
0.08:10.09
0.46:10.09
0.34:10.88
0.37=0.45
0.40
SPAWNER
0.37-00.5
0.34+0.48
DGW
0.29:10223
1.03:0.00
0.811:0.26
1.02:10.00
0.37:10.16
0.67:00.88
1.00=0.01
0.35
Permutation
0.14++0.15
1.03+0.00
0,24++0.34
1.02++0.00
0.37++0.16
0.67++0.48
1.00+0.01
0.41 0.41-0.44
Jittering
0.17:0.21
L0310.00
0.17100.0
1.2410.00
0.37:10.16
0.34:10.48
1.00=0.01
0.40:0.44
Scaling
0.36=10.18
1.03-0.00
0.33+10.47
0.86+10.23
0.46:10.09
0.00:10.00
1.00=0.01
0.41#0.44
TDA
0.57+00.0
0.01 00001
0.02++0.02
0.01-10.01
0.37-+0.16
0.00-10.00
0.37-00.5
0.35+00.77
DICE
pen
door
relocate
hammer
NoAug-
0.20=1026
0.69:10.88
0.30.10.43
0.67:10.47
OAT
0.02=0:01
0.34=0.48
1.00:0000
0.67=00.7
VAE- MDE
0.01:0001
0.68:0.48
0.64:10.4
0.66:10.6
TimeGAN
0.00=0.00
0.4210.44
0.6410045
0.6710.47
VAE
0.02-00.01
0.76-t0.36
0.58+00.2
0.40-10.41
SPAWNER
0.06:10.05
0.42:1.44
0.66:10.46
0.67:10.47
DGW
0.02-00.1
0.44
0.58+0.42
0.67++0.47
Permutation
0.01:10.01
0.34:0.48
0,31:10.30
0.67:10.47
Jittering
0.11-0.13
0.34:0048
0.58:1042
0.65:10.6
Scaling
0.02-+0.01
0.34-0.48
0.58+10.2
0.67+00.7
TDA
0.03:10.00
0.34
0.68:10.45
0.67-10.47
35

---

Published as a conference paper at ICLR 2024
Table 16: Regret@5 results OPE without and with each augmentation method in resampled Adroit expert environment. The data are randomly sampled from original training data as the same data points as the corresponding task in human environment. Results are obtained by averaging over random seeds used for training at discount factor of 0.995, with standard deviations shown after
FQE
pen
door
relocate
hammer
MB
door
relocate
hammer
NoAug.
0.02:0001
0.00=00.1
0.24::0.34
0.00=10.01
0.01:10001
0.00:10.01
0.00=0.00
0.07::0.09
OAT
0.05=+0.06
0.00 0.00
0,24++0.34
0.03
0.06++0.08
0.00
0.00=0.00
0.03++0.05
VAE-MDP
0.10=0006
0.00:0001
0.17100.00
0.13:10.09
0.06110.08
0.00:10.00
0.02=0.02
0.03-10.00
TimeGAN
0.08:0006
0.00:0000
0.17:10.20
0,10=:0.08
0.07:10.08
0.00:10.00
0.02=0002
0.13:+0.09
VAE
0.01:0002
0.00::001
0.45:10.37
0.13:10.09
0.05:10.06
0.00:10.01
0.45=0.37
0.29:10.14
SPAWNER
0.12:10.06
0.00:0000
0.26:10.33
0.13:10009
0.09:10.07
0.02:40.00
0.09=0.09
0.07:10.09
DGW
0.14++0.03
0.00
0.95++0.03
0.16++0.16
0.06++0.088
0.00
0.07+0010
0.13++0.18
Permutation
0.14=00.3
0.00:0001
0.45+-0.37
0.13++0.09
0.08:10.06
0.19:10.26
0.24=0.34
0.20::0.16
Jittering
0.08=0006
0.00:0.01
0.17
0.13++0.09
0,00:10.00
0.19::0.26
0.00
0.07
Scaling
0.08=10066
0.00:0001
0.17:10.20
0.13:10.09
0.06:10.08
0.00:10.00
0.00=0.00
0.08
TDA
0.09=0.04
0.00::001
0.241:0.34
0.13++0.09
0.00:10.00
0.00=10.00
0.00=0.00
0.01-10.02
pen
door
relocate
hammer
DR
pen
door
relocate
hammer
NoAug.
0.02+0002
0.23+-0.24
0.00
0.03-+0.05
0.14++0.03
0.00 0.10
0.56-00.41
0.13+-0.18
OAT
0.07=0003
0.00:0.01
0.00:10.00
0.00:10.00
0.05:10.06
0.00:10.00
0.07=0010
0.13
VAE-MDP
0.04:0.04
0,23+0.24
0.65::0:46
0.00=10.00
0.14::0.03
0.00=10.00
0.56=0.41
0.07
TumeGAN
0.00:0000
0.37+0.26
0.65:10.66
0.00:10.00
0.11::0.07
0.00:10.00
0.56=0.41
0.
VAE
0.18=00.0
0.21+0.25
0.79:10.25
0.01=00.01
0.14:10.03 03
0.00:10.00
0.27=0.32
0.13
SPAWNER
0.39++0.00
0.24-0034
0.13
DGW
0.06=10.08
0.56:0.00
0.71110.21
0.39:10.00
0.14:1:003
0.00:10.01
0.58=0039
0.1
Permutation
0.05++0.33
0.56+00.0
0.15+00.21
0.39++0.00
0.14++0.03
0.00
0.03
0.13++0.18
Jittering
0.04:10.04
0.5610.00
0.02-10.02
0.39:10.00
0.14110.03
0.00:10.00
0.70=0.34
0.13
Scaling
0.13=0007
0.56:0.00
0.15:10.11
0.27=10.17
0.14:10.03
0.00:10.00
0.41
0.13
TDA
0.13++0.07
0.00-0.00
0.00--0.00
0.00-10.00
0.14-+0.03
0.00-10.00
0.30=0..3
0.13+0.118
DICE
pen
door
relocate
hammer
NoAug.
0.00=:0.00
0.37:10.26
0.15:10.21
0.04:10.4
OAT
0.00=0.00
0.19:00226
0.30:10.43
0.04110.04
VAE- MDP-MDP
0.00=10.00
0.19::0226
0.05:10.00
0.14:10.18
TimeGAN
0.00=:000
0.21:10.25
0.54100.9
0.07
VAE
0.00-=0.00
0.17-:0.16
0.17-+0.20
0.08=00.08
SPAWNER
0.01::0.01
0.19::0226
0.56:10.41
0.05:10.04
DGW
0.00--0.00
0.19:-0.26
0.32+00.46
0.05++0.04
Permutation
0.00::0.00
0.19::0.26
0.02:10.02
0.07-10.05
Jittering
0.00=0.00
0.19::0.26
0.32-1::46
0.05+10.04
Scaling
0.00=0000
0.19::0.26
0.34-10.45
0.04:00004
TDA
0.00
0.19::0.26
0.02:10.02
0.14:10.18
36

---

Published as a conference paper at ICLR 2024
Table 17: PST information in Adroit pen human.
trajectory index
trajectory length
PTS start time
PST end time
0
200
200
2
200
3
200
0
6
4
200
1
5
200
2
7
200
8
200
9
200
10
200
11
12
200
13
200
6
14
200
15
16
200
17
200
18
200
2
19
20
21
200
O
22
200
7
23
200
24
200
Table 18: PST information in Adroit
trajectory index
trajectory length
PTS start time
PST end time
300
204
217
300
238
251
2
300
202
215
3
300
210
223
4
300
189
202
293
202
6
284
183
196
7
265
161
174
8
265
153
166
9
291
167
180
10
247
151
164
11
236
136
149
12
271
165
178
13
242
144
157
14
277
170
15
257
158
171
16
260
166
179
17
223
134
147
18
257
164
177
19
277
146
159
20
288
189
202
21
289
187
200
22
256
156
169
23
226
146
159
24
225
135
148
C.2 START AND END TIME OF PSTs.
We present that the length of trajectories, and start and end time of found PSTs each trajectory in Adroit human can be varied, as shown in Tables T7 [8] 19 & 20 can be observed that PST can start at different time steps across trajectories, which resolves the non-temporally-alignne cases.
37

---

Published as a conference paper at ICLR 2024
Table 19: PST information in Adroit relocate-humar.
trajectory index
trajectory length
PTS start time
PST end time
0
352
19
26
437
24
31
360
52
59
3
402
30
37
4
346
33
40
5
527
59
66
6
412
28
35
40
8
477
37
44
9
382
32
39
10
385
24
31
11
523
27
34
12
451
33
40
13
382
33
40
14
357
31
38
15
375
39
46
16
439
36
17
378
25
32
18
377
31
38
19
336
42
49
20
512
36
43
333
32
39
22
297
44
51
23
358
36
43
24
328
31
38
Table 20: PST information in Adroit hammer-humae
trajectory index
trajectory length
PTS start time
PST end time
0
537
153
161
399
100
108
2
472
116
124
3
624
115
123
4
469
153
161
5
418
137
145
6
456
110
118
404
232
8
364
123
131
9
529
120
128
10
455
131
139
11
371
108
116
12
496
180
188
13
433
140
148
14
484
129
137
15
398
102
110
16
418
120
17
226
234
18
467
121
129
19
390
104
112
20
348
105
113
464
86
22
425
89
97
23
506
93
101
24
362
82
90
38

---

Published as a conference paper at ICLR 2024
D
SEPSIS TREATMENT
Sepsis, which is defined as tieethheetteing organ dysfunction in response to infection, is the leading cause of mortality and the most expensive condition associated with in-hosiital stay (Liu et al. 2014 Gao et al. 2022b). In particular, septic shock, which is the most advanced complication of sepsis due to severe abnormalities of circulation and/or cellular metabolism (Bone al., [192), reaches mortality rate as high as 50% (Martin et al. 2003). It is critical to find an effective policy that can be followed to prevent septic shock and recover from sepsis.
D.1 TASK DETAILS
Labels. The hospital provided the EHRs over two years, including 221,700 visits with 35 static variables such as gender, age, and past medical condition, and 43 temporal variables including vital signs, lab analytes, and treatments. Our study population is patients with a suspected infection which was identified by the administration of any type of antibiotic, antiviral, antibacterial, antiparasitic, or antifungal, or a positive test result of PCR (Point of Care Rapid). On the basis of the Third International Consensus Definitions for Sepsis and Septic Shock (Singer et al. (016), our medical experts identified septic shock any of the following conditions met:
Persistent hypertension as shown through two consecutive readings (s 30 minutes apart).
Systolic Blood Pressure (SBP) < 90 mmHg Mean Arterial Pressure (MAP) < 65 mmHg Decrease in SBP 2 40 mmHg with an 8-hour period
Any vasopressor administration.
From the EHRs, 3,499 septic shock positive and 81,398 negative visits were identified based on the intersection of the expert sepsis diagnostic rules and International Codes for Disease 9th division (ICD-9); the 36, visits with mismatched labels between the expert rule and the ICD-9 were excluded in our study. 2,205 shock visits were obtained by excluding the visits admitted with septic shock and the long-stap visits and then we did the stratified random sampling from non-shock visits, keeping the same distribution of age, gender, ethnicity, and length of hospital stay. The final data constituted 4,410 visits with an equal ratio of shock and non-shock visits.
States. To approximate patient observations. 15 sepsis-related attributes were selected based on the sepsis diagnostic rules. In our data, the average missing rate across the 15 sepsis-related attributes was 78.6%. We avoided deleting sparse attributes or resampling with regular time interval because the attributes suggested by medical experts are critical to decision making for sepsis treatment, and the temporal missing patterns of EHRs also provide the information of patient observations. The missing values were imputed using Temporal Belief Memory (Kim & Chi 2018) combined with missing indicators (Lipton al. 2016).
Actions. For actions, we considered two medical treatments: antibiotic administration and oxygen assistance. Note that the two treatments can be applied simultaneously, which results in total of four actions. Generally, the treatments are mixed in discrete and continuous action spaces according to their granularity. For example, decision of whether a certain drug is administrated discrete, while the dosage of drug is continuous. Continuous action space has been mainly handled by policy- based RL models such as actor-critic models (Lillicrap et al. 2015). and is generally only available for online RL. Since we cannot search continuous action spaces while online interacting with actual patients, we focus on discrete actions. Moreover, in this work, the RL agent aims to let the physicians know when and which treatment should be given to a patient, rather than suggests an optimal amount of drugs or duration of oxygen control that requires more complex consideration.
Rewards. Two leading clinicians, both with over 20-year experience the subject of sepsis, guided to define the reward function based on the severity of septic stages. The rewards were defined as follows: infection [-5], inflammation [-10], organ failures [-20], and septic shock [-50]. Whenever patient was recovered from any stage of them, the positive reward for the stage was gained back.
The data divided into 80% (the earlier 80% according to the time of the first event recorded in patients' visits) for training and (the later) 20% for test, following the common practice while splitting up time-series for training and testing (Campos al. 2014).
39

---

Published as a conference paper at ICLR 2024
Problem
Ground-truth rewards for four policies deployed in intelligent tutor
Given event A and with p/A)=0.4, p(B)=0.5, and pl~A -8)-0.2. Determine PlA
Tutor
History
[(A)
P(B) = 0.5
R(-An-B)
oquaiion Morgan AnA pres
Reward
betoo:
((AUB) "TARGET
P(-(-An-B)) None
Equations
Raspanse
1 p 201- (iTl (070/ Actiion Thoose
Inuu
171
T12
11
Nexpert
Figure 14: Our intelligent tutor GUI (left) and empirical results with three RL-induced policies and one expert policy (right).
Policies We assume that the clinical care team is well-trained with sufficient medical knowledge and follows standard protocols in sepsis treatments, thus we consider the behavioral policy, param- eterized through behavior cloning (Azizsoltani & Jin 2019), that generates the trajectories above as expert policy. We estimate the behavior policy with behavior cloning in (Fu et al. 2021 Hanna et al. 2019). The evaluation policies were trained using off-policy DQN algorithm with dif- ferent hhper--ppaaeeeeee parameter settings, where DQN was trained using default setting (learingg rate 1e 3, 4, learning rate 5, a different random seed, 7 = 0.9, respectively.
y = 0.99), learning rate
Evaluate Performance of Target Policies Since the RL agent cannot directly interact with patients, only depends on offline data for both policy induction and evaluation. In similar fashion to prior studies (Komorowski et al. [2018] Azizsoltani & Jin 2019 Raghu et al., all](201), the induced policies were evaluated using the septic shock rate. And the OPE validation metric, rank correlation, can be calculated by comparing the ranking by OPE estimations versus the rankings of septic shock rate of target policies. The assumption behind that is (Raghu et al., 2017): when septic shock prevention policy is indeed effective, the more the real treatments the induced policy, the lower the chance the patient would get into septic shock; vice versa, the less the real treatments in a patient trajectory agree with the induced policy (more dissimilar), the higher the chance the patient would get into septic shock. Specifically, we follow the recent design by (Ju jet al., 202I);: We measured agreement rate with the agent policy, which is the number actions of target policy agreed with the agent policy among the total number of actions in a trajectory. Then sort the trajectories by their similarity rate in ascending order and calculate the septic shock rate for the top 10% of trajectories with the highest similarity rate. If the agent policies indeed effective, the more the actually executed treatments agree with the agent policy, the less likely the patient is going to have septic shock.
patient trajectory agree with
E REAL-WORLD INTELLIGENT TUTORING
E
TASK DETAILS
Our data contains total of 1,307 students' interaction logs with web-based ITS collected over seven semesters' classroom studies. The ITS is used an undergraduate STEM course col- lege, which has been extensively used by over 2, 000 students with ~800k recorded interaction logs through eight academic years. The ITS designed to teach entry-level undergraduate students with ten major probability principles, including complement theorem, Bayes' rule, etc. The GUI of the ITS is provided Figure [4
During tutoring, there many factors that might determine indicate students' learn- ing state, but many them are not well understood by educators. Thus, to be conservative, we extract varieties of attributes that might determine or indicate student learning observations from student-system interaction logs. In sum, 142 attributes with both discrete and continuous values are extracted, which can be categorized into the following five groups:
States.
(i) Autonomy (10 features): the amount work done by the student, such as the number of times the student restarted problem:
40

---

Published as a conference paper at ICLR 2024
Learnin
E
E
QAT
VAE MDD
WA
TiMeSAN
SPHANER
Perm:
Scaling
700 Improved
25 Rank Con.. Improved
24 An 24 we Regratig Improved
Figure 15: OPE improvement results averaging across OPE methods in e-learning.
(ii) Temporal Situation (29 features): the time-related information about the work process, such as average time per step;
(iii) Pooviem-Solving (35 features): information about the current coblem--solvigg context, such as problem difficulty;
(iv) Performance (57 features): information about the student's performance during problem- solving, such as percentage of correct entries;
information about the student's hint usage, such as the total number of hints (v) Hints (11 features): requested.
Actions. For each problem, the ITS agent will decide whether the student should solve the next problem, study a solution provided by the tutor or work together with the tutor to solve the problem. For each problem, the agent makes two levels of granularity: problem first and then step. For problem level, it first decides whether the next problem should be worked example (WE), problem solving (PS), or a collaborative problem solving worked example (CPS). In WEs, students observe how the tutor solves problem; in PSs, students solve the problem themselves; in CPSs, the students and the tutor co-construct the solution. a CPS is selected, the tutor will then make step-level decisions on whether to elicit the next step from the student to tell the solution step to the student directly.
Rewards. There was immediate reward but the empirical evaluation matrix (i.e., delayed reward), which was the students' Normalized Learning Gain (NLG). NLG measured students' learning gain irrespective of their incoming competence. NLG is defined as: NLG = scorCpoattent score ppreecat
V scoregretest where I denotes the maximum score for both pre- and post-test that were taken before and after usage of the ITS, respectively.
Policies. The study were conducted across seven semesters, where the first six semesters' data were collected over expert policy and the seventh semester's 's data were collected over four differ- policies (three policies policy randomly picked actions. The three RI--induced policies were trained using off-policy DQN algorithm with different learning rates (r {le 3, le 4, le 5
RL-inddued policies and one was the expert policy). The expert
Target policies are randomly assigned to 140 students who take the Probability course in one semester. During the studies, all students used the same tutor, followed the same general procedure, studied the same training materials, and worked through the same training problems. All students went through the same four phases: 1) reading textbook, 2) pre-test, 3) working on the ITS, and 4) post-test. During reading textbook, students read a general description of each principle, reviewed examples, and solved some training problems to get familiar with the ITS. Then the students took a pre-test which contained a total of 14 single- and multiple- principle problems. Students were not given feedback on their answers, nor were they allowed to go back to earlier questions (so as the post-test). Next, students worked on the ITS, where they received the 12 problems on ITS in the same order. After that, students took the 20 20-problem post-test, where 14 of the problems were isomorphic to the pre-test and the remainders were nomiisoo multiple- multiple-principle principle problems. Tests were auto-graded following the same grading criteria. Test scores were normalized to the range of [0, 1].
Evaluate Performance of Target Policies.
41

---

Published as a conference paper at ICLR 2024
State Dim.
Action Dim.
Early Term.
Continuous Ctrl.
Dataset
Dataset Size
random
999,000
medium-
Halfcheetah
17
No
Yes
replay 201,798
medium 999,000
medium- 1,998,000
expert
expert 999,000
random 999,999
medium- 401,598
Hopper
11
3
Yes
Yes
replay
medium 999,998
medium- 1,998,966
expert
expert 999,061
random 999,999
medium- 301,698
Walker2d
17
Yes
Yes
replay
medium 999,322
medium- 1,998,318
expert
expert
999,000
random
999,427
medium- 301,698
Ant
27
Yes
Yes
replay
medium 999.175
medium- 1,998,
expert
expert
999,036
Table 21: Summary of the Gym-Mujoco environments and datasets used to train OAT and baselines.
F ADDITIONAL EXPERIMENTS AND RESULTS: GYM-MUJOCO
F.1 DETAILS OF GYM-MUJOCO
A total of environments provided by Guy-Muooco, and we follow the guidelines from DOPE benchmark to validate our work and baselines (Fu et al., 2020). Moreover, each environment is provided with 5 training datasets collected using different behavioral policies, resulting in total of 20 sets of tasks. DOPE also provides 1l target policies for each environment, whose performance be evaluated by the OPE methods. Though the Gym-Mujoco environments may not fit our major interests, i.e., human-ine innolved tasks, we provide additional validation of our work on them, given they are popular testbeds and may be interested by readers. Table 21 shows summary of the Gym-Mujoco environments and datasets.
F.2 RESULTS ON GYM-MCIOCO AND DISCUSSIONS
The Table 22 presents summary results Gym-Mujoco environments using OAT and augmentation baselines. The results are obtained by calculating the improved percentage (for MAE), or distance (for Rank Comelation, Regret@ and R5gret@5) after augmentation compared to the original OPE
42

---

Published as a conference paper at ICLR 2024
Halfchoctah
Hopper Rank Corr.
MAB Pere.
Rank Cor:.
Regret@1 1
Regrete5
MAE Pere.
Regreté1
Regreté 5
OAT
25.11
0.10
0.08
0.01
5.60
0.17
0.05
0.05
VAE-MDP
19.01
TDA
15.86
0.04
-0,01
0.00
0.62
0.04
-0.11
0.04
Permutation
10.14
0.03
0.00
0.00
-3.72
0.01
-0.25
0.00
Jittering
12.32
0.00
-0.02
0.01
-3.34
0.00
-0.33
0.01
Scaling
12.08
0.01
0.02
0.00
.4.33
0.02
00.27
0.00
VAE
18.72
0.00
-0.02
-0.01
-3.21
0.14
-0.08
0.04
TimeGAN
17.56
-0.02
-0.08
0.00
-1.19
0.15
-0.08
0.04
SPAWNER
-0.11
-0.15
-0.02
DGW
7.22
-0.12
-0.11
-0.07
-11.38
-0.07
-0.18
-0.15
Walker2d
Ant
MAE Porc.
Rank Conr.
Regret@1
Regret@
MAE Pere.
Rank Corr.
Regret@ I
Regret 5
OAT
3,39
0.15
0.06
0.03
1.50
0.27
0.20
0.01
VAE-MDP
3,37
0.06
0.01
TDA
0.00
0.04
-0,04
0.03
0.42
0.04
-0.03
0.00
Permutation
-5.39
0.03
-0.5
0,02
-0.13
0.05
0.00
0.01
Jinering
-5.62
-0.01
-0.08
Scaling
-6.37
0.04
-0.06
0,01
-0.17
0.00
-0.02
0.00
VAE
TimeGAN
-3.45
-0.03
-0.12
0.01
-0.30
0.08
0.01
0.01
SPAWNER
-14.91
-0.13
-0.14
-0.02
-1.36
-0.19
-0.15
-0.04
DOW
-12.67
-0.10
-0.15
0,00
-2.24
-0.20
-0,22
-0.02
Table 22: Summary results of averaged improvements OPE by OAT and baselines in the Gym- Mujoco environments and datasets.
results without augmentation. OAT can outperforms the baselines in terms of all four evaluation metrics (as bold in Table
G MORE RELATED WORKS
OPE In real-world, deploying and evaluating RL policies online are high stakes in such domains, as a poor policy can be fatal to humans. It's thus crucial to propose effective OPE methods. OPE used evaluate the performance of largel policy given historical data drawn from (alternative) behavior policies. variety of contemporary OPE methods has been proposed, which can be mainly divided into three categories (Voloshin al. 2021b): (i) Inverse propensity scoring (Precup 2000 Doroudi et al. 2017). such as Importance Sampling (IS) (Doroudi et al. 2017). to reweigh the
rewards historical data using the importance ratio between and л. (ii) Direct methods directly
estimate the value functions of the evaluation policy (Nachum et al.) 2019) Uehara et al. 2020
Xie al. 2019 Zhang al.) 202T:) Yang et al. [2022). including but not limited model- based estimators (MB) (Paduraru) 20133 Zhang et al., 2021 that train dynamics and reward models on
transitions from the offline data; basue- estimators (Munos et al. 2016 Le et al.) 2019) such as Fitted Q Evaluation (FQE) which is a policy evaluation counterpart to batch Q learning; minimax estimators (Liu et al., 2018 Zhang et al.] 2020bb) Voloshin et al.) (021a) such as DualDICE that estimates the discounted stationary distribution ratios (Yang et al. (020a). (iii) Hybrid methods combine aspects of both inverse propensity scoring and direct methods (Jiang & Li] (2066] Thomas Brunskill 20166). For example, DR (Jiang & Li, 2016) leverages a direct method decrease the variance of the unbiased estimates produced by IS. However, major challenge of applying OPE to real world is many methods can perform unpleasant when human-collected data is highly limited
as in (Fu et al. 2020 Gao et al. 2023a). augmentation can be an important way to facilitate OPE performance.
Data Augmentation Data augmentation has been widely investigated in various domains, includ- ing computer vision, time series, and RL. computer vision, images are the major target and augmentation have improved downstream models' performance (LeCun et al.) 1998| Deng et al.) 2009 Cubuk et al.) 2019 Xie et al.) (020). However, many image-targeted methods, such as crop and rotate images, will discard important information in trajectories. In time series, variety of data augmentation has been proposed to capture temporal and multivariate dependencies (Le Guen- nec et al. 2016 Kamycki al. 2019 Yoon et al. 2019 Iwana & Uchidal 202 For instance, SPAWNER (Kamycki et al. 2019) and DGW ([wana Uchida (021b) augment time series by capturing group-level similarities to facilitate supervised learning. Generative models such as GAN
43

---

Published as a conference paper at ICLR 2024
and VAE have achieved state-of-the-an performance in time series augmentation for both supervised
and unsupervised learning (Antoniou et al.) 2017] Donahue et al. 20188] Yoon et al. 2019 Barak
et al. 2022). However, those approaches for images and time-series do not consider the Markovian nature OPE training data, and may not directly applicable to MDP trajectory augmentation.
44


Reviews:
[{"review_id": "VxxVBzFGoS", "reviewer": "Reviewer_WbKh", "summary": "The authors propose a novel approach to augment offline datasets for off-policy evaluation. This is achieved by introducing a three-step process (i) select relevant sub-trajectories in the dataset, (ii) use a VAE-based architecture to generate new trajectories, and (iii) add these trajectories back to the dataset.\n\nEmpirically, the authors show that the proposed method outperforms other data-augmentation methods on a diverse set of problems.", "questions": "(i) The unsupported justifications of the results\nIn more than one occasion, the authors (rightly) discuss very specific reasons that could confirm/justify the observed results. However, I feel like in most cases in this work, the justifications are not supported by data. I feel this is better explained through an example. The authors say: \"In contrast, the historical trajectories induced from simulations tend to result in better coverage over the state-action space in general, and the augmentation methods that do not consider the Markovian setting may generate trajectories that could be less meaningful to the OPE methods, making them less effective.\" What do the authors mean by less meaningful? And do they believe that this (i.e., non-Markovian augmentation are worse in scenarios with better state coverage) can be confirmed more generally?\n\nIn my opinion, the authors use phrases like \"we conjecture\", \"a possible reason\" without strictly backing up the claims with evidence (which would in turn greatly improve the quality of the paper)\n\n(ii) The lack of ablations to validate the proposed innovations:\nThe authors compare against a wide set of benchmarks, although, as I'm not an expert in OPE, it is unclear to me whether these are explicitly tailored for the OPE problem or not. Moreover, since the proposed framework is a composed of multiple smaller contributions (discretization, VAE architecture, etc.), the authors should make sure to isolate each of these contributions individually and support the claims with evidence and experiments.\n\n(iii) The relatively narrow-scoped experiments:\nDid the authors consider using this approach within an RL context? How would this perform?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "I am not an expert in the field of data-augmentation for RL, but I enjoyed the thought process behind the development of the framework: (i) select a criterion for what makes a specific region of the state-action space interesting for data augmentation, and (ii) use temporal generative models to sample new sub-trajectories. \n\nFigures are generally informative and overall the paper is well written.", "weaknesses": "In my opinion, the main weaknesses of this work lie in (i) the unsupported justifications of the results, (ii) the lack of ablations to validate the proposed innovations, and (iii) the relatively narrow-scoped experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ykBVo14DUf", "reviewer": "Reviewer_3xn9", "summary": "The paper proposes to use a specialized form of data-augmentation, specifically to improve the accuracy of off-policy evaluation algorithms. It bases its algorithm on looking at promising sub-trajectories and augmenting the dataset with new samples to improve coverage of the state-action space.", "questions": "1. Could one generate better samples by somehow tying in the evaluation policy behavior into the data generation process? It seems like if the goal is to evaluate some policy $\\pi_e$, we should account for what $\\pi_e$ does/samples actions?\n2. How is step 1 in the algorithm actually done? That is, under what basis are states grouped into clusters. Some  representation learning work does this based on behavior similarity [1] etc. How the states are grouped is important for all the remaining steps.\n3. The \"grounding\" process of Eqn 2 is unclear to me. How do you avoid the model from hallucinating and generating samples that cannot occur in the MDP since interaction with the MDP does not happen in the OPE setting?\n\n[1] MICo: Improved representations via sampling-based state similarity for Markov decision processes. Castro et al. 2021.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The paper tackles an important problem of OPE, which receives relatively far less attention than control-based papers.\n2. Adapting data augmentation techniques for the RL setting, and particularly the OPE setting, is a very interesting direction.\n3. The paper applies the proposed idea to more real-life datasets, which is great since lot of OPE work just evaluates ideas on mujoco.", "weaknesses": "1. It feels like the paper was somewhat rushed. There are some confusing parts/writing errors such as: what is \"human-involved RLs\" (third line in intro), \"while human may behave\" (5th line last para of intro), Figure 2 I believe should be referencing Sections 2.1/2.2/2.3 (not 3.X), \"intrinsic nature that human may follow\" (2nd line Section 2.1).\n2. The notation in Section 2.1 is very difficult to parse. I suspect there is an easier way to explain this. Also Figure 3 is very confusing, and is not explained in the text (there are some references in the appendix, but I think there should be much more explanation of it given the complexity of the diagram).\n3. I think drawing connections to how humans behave (such as in Figure 2 caption) is too strong. It is appropriate for intuition purposes but making general statements on how humans behave seems a bit strong.\n4. Biggest concern. I dont think the paper gives a good intuition for why this works. While coverage is important, it seems like that is insufficient. Another challenge is distribution shift. Prior work [1] has discussed that under extreme distribution shift, OPE methods can diverge. However, this work does not discuss this. Moreover, it seems possible to me that this data augmentation technique could introduce samples that worsen the distribution shift, which could worsen accuracy of OPE methods.\n\n[1] Instabilities of Offline RL with Pre-Trained Neural Representation. Wang et al. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RJwJ0RbbfF", "reviewer": "Reviewer_pod6", "summary": "The paper is proposed for the Off-policy evaluation tasks in the RL domain. The proposed method OAT intends to solve the scarce and underrepresented offline trajectory challenges by data augmentation. Specifically, a sub-trajectory mining method and fuse process is designed to find potential exploration space and integrate into existing state-action trajectories seamlessly.", "questions": "1. Could the authors discuss if it's only applicable for the behavior data to be augmented? Or can we also augment the target policies? \n2. When identifying the PST, at the first step, How are these C different clusters divided? And how is the number of c determined?\n3. As shown in Fig3, It is suggested to introduce how the latent prior is constructed, since it is the key step in constructing the new augmented PST.\n4. What is the complexity of the training VAE-MDP? Is there any analysis conducted to show the relations between training time and TDSS length/Potential trajectory scale?\n5. In Fig 4, it is obvious that the trajectories coverages are different on the left hand and right hand corner on the top of the maze, however if these two corners are not the potential area selected to augment, should they maintain the original distribution to the maximum extent? Apart from this illustration, It is suggested that the paper provides quantitative evaluation of the difference between augmented trajectory and ground truth trajectory.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper first proposes to augment the HIS trajectory data for offline RL Off-policy evaluation tasks, which is important for the real-world policy evaluation challenges.\n\n2. The process of detecting the potential sub-sequence is explicitly explained and demonstrated, easy to follow and understand, such as from the discrete representations to determine the support value, and eventually identify the PSTs.\n\n3. Authors applied the VAE-MDP process to generate the potential sub-trajectory, and empirical study shows the overall framework OAT is achieving promising results.", "weaknesses": "1. The main paper omitted some significant parts and placed them in the appendix, however should be explained in the main context, such as how the latent latent prior is constructed. And how the C different clusters are initially divided when identifying the PST is not clear.\n\n2. Even though the experiment could show that the OPE method performance is improved, the paper is suggested to make a fair analysis of how reliable the reward is from the algorithm augmentation, which, however, is significant for the result value of off-policy methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)