STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation
Authors: Hossein Goli
Michael Gimelfarb
Nathan Samuel de Lara
Haruki Nishimura
Masha Itkina
Florian Shkurti
Venue: NeurIPS 2025 spotlight ()
Keywords: Off Policy Evaluation
Reinforcement Learning
Generative Models
Primary area: reinforcement_learning
TL;DR: We introduce STITCH-OPE, a guided-diffusion framework for off-policy evaluation that stitches short behavior-conditioned sub-trajectories, uses negative-behavior guidance to correct distribution shift, and outperforms baselines across all metrics.

Abstract: Off-policy evaluation (OPE) estimates the performance of a target policy using offline data collected from a behavior policy, and is crucial in domains such as robotics or healthcare where direct interaction with the environment is costly or unsafe. Existing OPE methods are ineffective for high-dimensional, long-horizon problems, due to exponential blow-ups in variance from importance weighting or compounding errors from learned dynamics models. To address these challenges, we propose STITCH-OPE, a model-based generative framework that leverages denoising diffusion for long-horizon OPE in high-dimensional state and action spaces. Starting with a diffusion model pre-trained on the behavior data, STITCH-OPE generates synthetic trajectories from the target policy by guiding the denoising process using the score function of the target policy. STITCH-OPE proposes two technical innovations that make it advantageous for OPE: (1) prevents over-regularization by subtracting the score of the behavior policy during guidance, and (2) generates long-horizon trajectories by stitching partial trajectories together end-to-end. We provide a theoretical guarantee that under mild assumptions, these modifications result in an exponential reduction in variance versus long-horizon trajectory diffusion. Experiments on the D4RL and OpenAI Gym benchmarks show substantial improvement in mean squared error, correlation, and regret metrics compared to state-of-the-art OPE methods.

STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation
Hossein G..i
Michael Gimelfarb 122.4 Nathan De Lara 1,2,4 Masha Itkina Florian Shkurti
Haruki Nishimura"
Department of Computer Science, University of Toronto
2 University of Toronto Roboties Institute, Toronto, Canada Toyota Research Institute, Los Altos, California
Veector Institute, Toronto, Canada
hossein. ceaeiggglimmmaa www.mmaii.tooooootoooooooooooooo
mike. gimelfarb@mail wwwronto....... ca haruki ni shimura@tri. wow.eettrigggglall global florian@cs. toronto. edu
nathan. dww.aaamaall...mm...eemmmllllaea wtt.oooooi..oooooooooooooo ca masha. www.iiiiiiiiiiiiiom global
Abstract
Off-policy evaluation (OPE) estimates the performance of target policy using offline data collected from a behavior policy, and crucial in domains such robotics or healthcare where direct interaction with the environment is costly or unsafe. Existing OPE methods ineffective for high-dimensional, long-horizon problems, due to exponential blow-ups in variance from importance weighting compounding errors from learned dynamics models. To address these challenges, we propose STITCH-OPE. model-based generative framework that leverages denoising diffusion long-horizon OPE in high-dimensional state and action spaces. Starting with diffusion model pre-trained on the behavior data, STITCH- generates synthetic trajectories from the target policy by guiding the denoising process using the score function of the target policy. STITCH-OPE proposes two technical innovations that make it advantageous for OPE: (1) prevents over- regularization by subtracting the score of the behavior policy during guidance, and (2) generates long-hooe morgoooizon trajectories by stitching partial trajectories together end-to-end. We provide theoretical guarantee that under mild assumptions, these modifications result in an exponential reduction in variance versus long-horizon trajectory diffusion. Experiments on the D4RL and OpenAl Gym benchmarks show substantial improvement in mean squared error, correlation, and regret metrics compared to state-of-the-art OPE methods
Introduction
Given the slow and risky nature of online data collection, real-world applications of reinforcement learning often require offline data for policy learning and evaluation 23 39). An important problem of working with offline data is off-policy evaluation (OPE), which aims to evaluate the performance of target policies using offline data collected from other behavior policies. One practical advantage of OPE is that it saves the cost of evaluation on hardware in embodied applications in the real world (28) However, central challenge of OPE the presence of distribution shift induced by differences in behavior and target policies 223 3). This can lead to inaccurate estimates of policy values, making it difficult
select between multiple target policies before they are deployed (44 (66).
Numerous approaches have attempted to address the distribution shift in offline policy evaluation by reducing either the variance of the policy value or its bias, but they typically ineffective in
Project website and code: stitch-ope.githuub.o
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

I. Behavior Data
II. Target Policy
M. PGD
IV. Ours
A: Conditional diffusion
improves composition.
Conditional diffusion improves state generalization.
C: Behavior policy guidance addresses distribution
shift.
Table I: A 2D toy problem with Gaussian dynamics illustrates the advantages of STITCH-OPE.
Row A: Behavior data is mixture of two datasets generated by different behavior policies B1 and a piecewise function following 31 on the left half-space and 82 the right half-space. Policy-Guided Diffusion (PGD) in column III estimates the behavior trajectory distribution by training a diffusion model on the behavior dataset, then leverages guided diffusion to generate target policy trajectories
32. The target policy column
[16]. PGD is unable to stitch behavior trajectories correctly while STITCH-OPE is. One explanation is that conditional diffusion provides a higher entropy sampling distribution than full-length diffusion, and thus ensures broader coverage of the modes in the behavior data (see Section 3.3 for details). Row B: The initial state is varied during trajectory generation. As shown in column III, PGD cannot generalize to new initial states. On the contrary, STITCH-OPE is trained on sub-trajectories that start in arbitrary states in the behavior dataset opposed only initial states, which allows it to achieve better generalization. Row C: scenario with severe distribution shift is presented, where the behavior and target policies move the agent different directions. As shown in columns III and IV, the negative behavior guidance term is essential to prevent over-regularization, which can prevent guided diffusion from addressing distribution shift and lead biased value estimation.
high-dimensional long-horizon problems. For example, Importance Sampling (IS) KK estimates the value of the target policy by weighing the behavior policy rollouts according the ratio of their likelihoods. However, it suffers from the so-called curse of horizon where the variance of the estimate increases exponentially in the evaluation horizon [25] More recent model-free OPE estimators reduce or eliminate the explosion in variance by estimating the long-run state-action density ratio d" (s, a)) 2 a) between the target and behavior policy [2] 31 43 yet they have demonstrated poor empirical performance on high-dimensional tasks where the behavior and target policies are different (i.e. the behavior policy is not noisy version of the target policy) [14]
As an alternative approach, model-based OPE estimators typically learn an empirical autoregressive model of the environment and reward function from the behavior data, which is used to generate synthetic rollouts from the target policy for offline evaluation [9 38 46). Some advantages of the model-based paradigm include sample efficiency (24), exploitation of prior knowledge about
the dynamics 12]. and better generalization to unseen states 445) Although model- based OPE
methods often scale well to high-dimensional short-horizon problems owing the scalability of the deep model-based RL paradigm their robustness diminishes in long- long-horizon tasks due to the
compounding of errors in the approximated dynamics model [14] 17 [8]
Driven by the recent successes of generative diffusion in RL 29
35 we propose Sub-
Trujectory Importance- -errrhhtnce-eigghtedd Trajectory Composition for Long- Hogg-Hrizon OPE for model-based off- policy evaluation long-horizon high-dimensional problems. STITCH-OPE first trains diffusion model behavior data, allowing it to generate dynamically feasible behavior trajectories [18).

---

STITCH-OPE differs from prior work by training the diffusion model on short sub- -ub-trajectories instead of full rollouts, where sub-trajectory generation conditioned on the final state of the previous generated sub-trajectory. This enables accurate trajectory "stitching" using short-horizon rollouts, while minimizing compounding error of full-trajectory rollouts, thus bridging the gap between model-based OPE and full-trajectory offline diffusion.
STITCH-OPE explicitly accounts for distribution shift in OPE by guiding the diffusion denoising process 9 18 during inference. This can be achieved by selecting the guidance function to be the difference between the score functions of the target and behavior policies. A significant advantage of guided diffusion is that it eliminates the need to retrain the diffusion model for each new target policy, By pretraining the model on a variety of behavior datasets, generalization can be achieved during guided sampling to produce feasible trajectories under the target policy. leading to robust off- off-policy estimates for target policies that lack offline data. STITCH-OPE contributes the following novel technical innovations that we consider critical to the successful and robust application of diffusion models for OPE:
Trajectory Stitching with Conditional Diffusion. We propose condiitoid guided diffusion model for generating short sub-trajecte uub-trajectories from the target policy (Figure I This significantly improves the quality (Table [ row A) and generalization (Table row B) of trajectory generation in long-horizon tasks. Theorem 3.3 also provides theoretical bounds on the bias and variance of our proposed approach.
Behavior Guidance. We show that the negative score function of the behavior policy mitigates the diffusion model from collapsing to trajectories with large behavior likelihood, thus improving generalization out-of-data (Table row C). This negative score function naturally arises from viewing the likelihood ratio (i.e. in importance sampling) as the connection missed in prior work [66.
classification density in diffusion guidance 9
Robustness Across Problem Difficulty. Finally, we evaluate STITCH-OPE on the OpenAI Gym control suite 4 and the D4RL offline RL suite [13], showing significant improve- ments compared other recent OPE estimators across variety of metrics (mean squared error, rank correlation and regret), problem dimension and evaluation horizon. To our knowledge, STITCH-OPE the first work to demonstrate robust off- ooflpolicy evaluation in high-dimensional long-horizon tasks.
2 Preliminaries
Markoy Decision Processes. We consider the standard Markov Decision Process (MDP), which consists of a 6-tuple (S, A, R, P. do: [34] where: S is the continuous state space, A is the continuous action space, R is the reward function, P is the Markov transition probability distribution, dn is the initial state distribution and 7 E [0, 1] is the discount factor.
A policy т. : S A + oo) observes the current state s, samples an action according to its conditional distribution π (-1ss), and observes the immediate reward R(s, a) and the next state s' ~ P(-|s, a). The interaction of a policy with an MDP generates set of trajectories of states, actions, and rewards. The goal of reinforcement learning is to leam policy that maximizes the expected return:
T-1
J(x) E, Eyopp
Σ ExtR(sn
p=(T) do() II 0=
((alss)
over length-T trajectories 7 (so, ao, 81, ai.a......7 ST) induced by policy æ.
Off-Policy Evaluation. The goal of Off-Policcy Evaluation (OPE) is to estimate the expected return of some target policy given only data set of trajectories Dg from some behavior policy B. To estimate J(x,, ). it is necessary to approximate the distribution over trajectories (T( induced by π using only samples from the distribution PT()). Hence, the phenomenon of distribution shift arises whenever 3 is sufficiently different from n, in which case (T) is not suitable proxy for obtaining samples from (T) and corrections must be made account for the distribution shift.
Denoising Diffusion Models. Denoising Diffussion Probubilistic Models (DDPMs) [36 151 are a class of generative models to sample from given distribution. Given dataset {z, J.N and the
3

---

K- K-step (forward) noise process -² DDPMs are trained to perform the K- K-step (backward) denoising process to recover I; from a ~ (O,01), I). This is accomplished by running the forward process at + x on the original 2; and training the DDPM eo on the denoising process x. | ~ N(u(a k), of to predict the noise e so that the distribution over denoised samples a; matches 24. The standard reparameterization u= k) = (x c (1 - ax))/ok)/ yay maps x² and the predicted noise ca (x k) to x==1 to undo the noise accumulated during the forward process. The following loss function is typically
vagal + ++v--ai age, where ~ (0, I) is independent,
used to train a diffusion model [5]
0) = (x,22, [[|e- [ ==(2,,A))1
(1)
Guided Diffusion. is possible guide the sampling from trained diffusion model maximize some classifier p(y](:)) [9. A key observation diffusion is that the backward diffusion process can be well approximated by a Gaussian when the noise is small, that is, 1f| ~ N(Hu, Σx). Next, observe that p(** + y) x p(a* >ply(=*) by Bayes' rule. Applying the first-order Taylor approximation log p(y) (=*) logp(y)uu, (u) + +(x* px)g(pR) ua, where g(u) V log p(yl (y) l=== it can be shown that:
1
log(p(æ*1a* ++ )p()(=*))
(1000="""*)))(((==")) x - 2 -2(0 uo Eag(u))) 2 (x" ps - Eng(uu)) 2x
log log.N pu Eng(us); Ex)-
In other words, it is possible to sample from the conditional (guided) distribution pla* (otttt (44111 y) by sampling from the original diffusion model with its mean shifted to H4 Eng(ux) Appendix A provides a worked example illustrating guided diffusion for a mixture of Gaussians.
3 Proposed Methodology
The direct method for off-policy evaluation estimates the single-step autoregressive model (( 4x-1) and the reward function A(s,, from the behavior data. Then, draws tar- get policy trajectories
(T) by forward sampling, that is, $g dg. Ag T--lsu), $1 P(-|so, ag).... ST P(-(sT11, -r) However, even small errors in P can lead to significant bias in (T) due to the compounding of errors over long horizon T (20, [7]. STITCH-OPE avoids the compounding problem by generating the partial trajectory in a single (backward diffusion) pass, leading to more accurate OPE estimates over a long horizon.
3.1 Guided Diffusion for Off-Policy Evaluation
is possible to approximate P- using guided diffusion by interpreting each data point mn as a full trajectory Given a behavior policy 3 and corresponding length-T trajectory distribution (T), the corresponding length-T trajectory distribution of target policy æ can be written as:
T-1
a(ae(s)) M(ress))
T-1 (ael()) = (T(T) II t B(ar(s))
(T( = (0n) II t=
B(arls)) P(8e41 0t)
(3)
which is the standard importance sampling correction (33] We address the question of tractably learning PT(T) by training diffusion model (T(T)) the offline behavior data set D3 (18), thus approximating (T) As Pa (T). Specifically, the diffusion model learns to map a trajectory consisting of pure noise, _ = (xb. db ), to a noiseless behavior trajectory 70 = (s0, af....
A key observation is that we can bypass importance sampling in (3) by guiding the generation process (T(T) towards P.r (r) using diffusion guidance (2) [8]. Specifically, let -²x 7 denote noisy behavior trajectory at step & of the forward diffusion process, and let y E {0, 1} be a binary outcome T with pvu = 1|7) x I =(ar)=)) Intuitively, y indicates whether the trajectory 7 generated by the target policy π (y- = 1) or the behavior policy B (y = 0), and the likelihood ratio determines the
3(a,)e))
odds that y 1 given By (3)
(T(T) x <pp(T)p(y 1/7),

---

(ee.. Separ)
E
State Noise
Partial Rollout
Behavior Data
Partial Rollouts
Target Rollouts
Pre-Trained Diffusion
D
J(x) Eyg (R(ona))))
R(secde)
Oft-Polity Evaluation
Reward Preffictor
v E, log x(ocla)) Policy Guidanne
V. E, log ((aslo))
Figure 1: A conceptual illustration of STITCH-OPE, with novel contributions highlighted in orange. A: Behavior data is sliced into partial trajectories of length w. B: The data is fed to a conditional diffusion model taking w-length sequence of Gaussian noise and state 8; as inputs, and applies the backward diffusion process to predict the behavior trajectory of length w beginning in state s,. C: To evaluate policies, STITCH-OPE also trains a neural network on the behavior transitions to predict the immediate reward. D: It then applies guided diffusion on the pretrained diffusion model to generate batch of partial target trajectories of length we where the guidance function incorporates the score function of the target policy and the behavior policy. E: The guided partial trajectories are stitched end-to-end to produce full-length target trajectories. Finally, the guided trajectories are evaluated using the empirical reward function It(s, a), and averaged to estimate the value of the target policy,
and thus the backward diffusion process for generating target policy trajectories for OPE can be approximated with guidance (2:
log Pa (T1) x log(pe(7* k (py = 1/7
ss log (n
(1++1; 1 + Es v log plu = 1/7)| Ex),
where po(r*17**)) (uR. Σx) is the backward diffusion process. Therefore, we can obtain
feasible target policy trajectories using the guidance function:
((T) V logp(y 117) V
log (acls))
V.
log (arlss)
(5)
Given the approximate sampling distribution over the trajectories of the target policy described above, (x(77) = f........ N((* 0, I Px (>*>~117²) dr ... dr.,,, and an empirical reward function
R(s, a), it is straightforward to estimate the expected return (or statistic such as variance quantile) given any target policy, i.c. J(T) = Er-r- E. R(ss,, ai) ~ J(T).
3.2 Negative Behavior Guidance
The target policy score function. Gsimple(F) V, ETT log ((pls)) [66, provides a simple guidance function for OPE. However, it corresponds biased estimator of p+(T) in the context of ( and can generate trajectories that are unlikely under the target policy, illustrated using the GaussianWorld domain Table D (see Appendix B for details). The behavior policy 3 returns positive angle in each state and the target policy returns negative angle, to test the performance of both guidance functions under distribution shift. Conelusions are summarized in row C of Table
[ The omission of the negative guidance term results in a sampling distribution that collapses to high-density region under (7)(7), where P=(7) could be small. other words, behavior guidance prevents the guided sampling distribution P. from becoming oveu-regul
5

---

In our empirical evaluation, we employ the following generalization of (5) to allow fine-granee tine-grained control over the relative importance of the target and behavior policy guidance
T-1
g(r) = aV, Σ log (a(l())) - AV,
log B(ay) (s).
(6)
Ignoring the normalizing constant which does not dependent on T. is equivalent to sampling from the following re- re-weighted trajectory distribution
(8 3(a (()
9- x pp(T)
II t=
which can be interpreted as tempered posterior distribution 22 over trajectories; the importance of the likelihood terms associated with т and 3 are controlled by the choice of a and A, respectively.
Note that the choice good empirical choice because it can push the backward diffusion process too far from the behavior distribution, leading to infeasible trajectories
À 1 reduces to the standard guidance function g(r). This is not
instability. Instead, typical choices satisfy À <
(see Appendix [.1] for an additional experiment confirming this). The choice = 1, = 0 reduces Isimple (7) and is unsuitable for OPE.
3.3 Sub-Trajectory Stitching with Conditional Diffusion
Recent work has shown that full-length diffusion models do not provide sufficient compositionality for accurate Long-horizon sequence generation [6]. In addition, full-length prediction requires the generation of sequences of length 7 (dim(()) + dim(()): ): this may be infeasible or inefficient on resource-constrained systems, when T is large when A or S is high-dimensional.
To tackle these limitations, STITCH-OPE trains a conditional diffusion model to generate behavior sub-trajectories of length < T. To allow for more flexible composition of behavior trajectories during guidance, generation in STITCH-OPE is performed in a semi-autoregressive manner from the diffusion model, which is conditioned on the last state of the previously generated sub uub-trajectory.
Specifically, the conditional diffusion model in STITCH-OPE, denoted as eØ (T kl k)s,), denoises length-w w noisy sub-trajectory Thx = (s at st 15 st $4 a & s conditioned
on the last state of of the previously generated sub-trajectory T Ti-wi Generalizing the loss function of STITCH-OPE is thus
OPEITCH-OPE(O)=
(ie-ca(Thetuu kl k|s
Next, writing p |sP) denote the sampling distribution over fully denoised sub-trajectories Latwwan conditioned on sR, the sampling process of STITCH-OPE can be written as:
T/w-1
10(++11-1 II w=
π (aulsa)
pz (T)
II
3(a,)
where target trajectories are generated by guiding the conditional diffusion model (analogous to (4) according to
(1-11)-1
w((+1)-1
((Tutno((+1)) 1w
Σ
log x (aulsu) A Twtnn(tt11)
Σ
log (aau/sa).
A complete algorithm description of STITCH-OPE is provided in Appendix
To understand the intuition that the conditional diffusion model offers better compositionality than the full-horizon prediction, we decompose the behavior trajectory distribution as mixture over the trajectories Ty in Dg:
(a(st, at,.. 8T (sp, Cg. ... 8t) Σ T,EDg
Pe (tt ot,.. 8T-1/ T3 (p(T))(())))))))) 1s, Oo .......t) -.)).

---

Meanwhile, the conditional diffusion model ignores the full history of past states, i.e.:
Ps(st, at. .T-.....ST111 C.........
St) 2
Σ TEEDs
Pa (5t, a.. ST,1 Tj (s)s))))))).
P(Ty]st) has higher entropy than p(Ty So. a............................a.. st) since it is conditioned on less information (see Appendix for a proof), and thus provides broader coverage of the diverse modes the behavior dataset. This improves the compositionality of guided long-horizon trajectory generation. Row A of Table Tillustrates this claim empirically using the Gaussian World problem. A further claim that the STTTCH-OPE model can generalize better across initial states with low, or even zero, probability under do (see row B of Table D. This occurs because P is trained on sub-trajectories starting in arbitrary states in D3, as opposed to states sampled only from dg- Therefore, sliding windows strike an optimal balance between autoregressive methods and full-length trajectory diffusion, providing good compositionality while avoiding the error compounding in terms of T.
3.4 Theoretical Analysis
We provide theoretical guarantees for our proposed STITCH-OPE method by analyzing its bias and variance. The first prerequisite assumption is standard in OPE [42 27 and limits the ratio π T/3.
Assumption 3.1. There a constant к such that ((a
< k for all s E S, a E A.
[(a
The second prerequisite assumption bounds the total variation between the learned length-w trajectory distribution Pij and the true distribution P under the behavior policy B. Assumption 3.2. TV(23 p3) óg for some constant dg.
Our main result is a bound on the mean squared error of the STTTCH-OPE estimator. We defer the full proofs. technical lemmas, and definitions to Appendix D
Theorem 3.3. Define Pa as the (length-T) T) trajectory distribution of the guided diffusion model, and Ps as the true trajectory distribution under the target policy æ.. Under Assumptions 3.1 and 3.2 the mean squared ervor (MSE) of the STITCH-OPE. return J satisfies:
Ra. [()-(
<
2B
1.008 03
+
T W
B²
0g
8B2 2
200p 0g + Varp. (J),
Bias?
Variance
where B = 1-1 sups,a s.a. [R(s a)l is bound on the maximum length-w discounted return, and J is the return under Pr.
but is expressed in terms of w rather than T (with a more favorable ( dependence). Since w is a fixed hyper-parameter typically chosen to be much smaller than T in practice, STITCH-OPE provides an exponential reduction in MSE versus both importance sampling and length-2 diffusion! In 4 we validate this claim further by showing that STITCH- OPE outperforms full trajectory diffusion (PGD) on most benchmarks (see Appendix L.2 for an additional experiment confirming small > ideal in practice). The MSE decreases as the error in the learned behavior model 8, decreases. In practice, 8g is easier to estimate and control than the error of the target density p². It is also important to note that the variance cannot be reduced below Var,,. (J). the intrinsic variance of the environment and the target policy.
Remarks. Theorem
3.3 resembles the ((ex(eTT) bound of IS-based methods 25
27
Section
4 Empirical Evaluation
Our empirical evaluation aims answer the following research questions:
1. Does the combination of conditional diffusion and negative guidance (as hypothesized in T translate to robust OPE performance standard benchmarks?
2. Is STITCH-OPE robust across problem size (c.g., state/action dimension, horizon)?
3. Is STITCH-OPE robust across different levels of optimality of the target policy and the classes of policies?
7

---

00
Rank Correlation latreaation
0.8
Log RMSE J
Regret@1 4
0.3
6.8
-1.0
a.a
0.2
-15
62
02
RdE
DR
DRE
MBR
PGD
STYCH
0.
44
5
DREE
MBR
STTCH
0.0
Dr
DRE
MBR
PGD
STICH
Figure 2: Mean overall performance of all baselines, averaged across environments. Error bars represent +/- one standard error.
FQE
DR
IS
DRE
MB
PGD
Ours
Hopper
-0.42 0.03
-0.57 + 0.02
-0.48 0.01
+0.42
+.02
-2.33
0.02
Rank Corr. f Log RMSE
Walker2d
0.00
-1.25 0.08
0.001
+0.45
-0.88
-0.32
-1.33
HalfCheetah
10.05
0.01 0.01
-0.84
0.019
0..07
-1.47
=
Pendulum
00.88
-1.02
-0.15
-0.58
0.003
-2.34 00.91
Acrobot
0.00
-0.49 0.06
0.00
0.20
-1.54
-0.13
-2.02 0.05
Hopper
0.05
0.06
-0,06
0.14
0.02
0.36
0.76
Walker2d
0.05
0.50
0.11
0.05
0,04
-0.07
0.63 0.03
HallCheetah
-0.03 0.06
.0,48
0.06
0.80
0.32
0,50
0.87 + 0.01
Pendulum
0.89 0.03
0.02
0.000
0.15
0.06
0.54
0.96 0.02
Acrobot
0.05
0.08
0.00
0.012
0.11
0.43
0.82
Regret 1
Hopper
0.34
0.05 0.02
0.02
0.17
0.04
0.01
Walker2d
0.23 0.04
+
0.06
0.11 0.00
0.05 0.04
0.32
<0.01 0.00
HalfCheetah
0.36 0.00
0.37 0.00
0.03 0.01
<0.01 + 0,00
0.00
0.10
0.08 0.01
Pendulum
0.03 0.03
0.08 0.03
0.98 0.00
0.85 0.13
0.07 a: 0.03
0.13
<0.01 0.01
Acrobot
0.04
<0.01
0.28 +0.06
0.06
0.22
Table 2: Comparison of OPE methods across environments. Error bars represent one standard error
across 5 seeds; any regret shown <0.01 is nonzero but rounds to zero at two decimals.
4.1 Experiment Details
Domains. We evaluate the performance of STITCH- C-PEE in high-dimensional long-horizon tasks us- ing the standard D4RL benchmark [13) and their respective benchmark policies [I4]. Specifically, we Sopper-mediiwe and wal leekdd-meddiie behavior datasets. Each evaluation consists 10 target policies n, n2; ... 11 trained at varying levels ability 144. We also carry similar experiments using classical control tasks (Pendulum and Acrobot) from OpenAI Gym [4], to evaluate the competitiveness STITCH-OPE on standard benchmarks on which other baselines have been extensively evaluated. For this set of environments,
use the halfcheetah-mede
obtain the target policies by running the twin-delayed DDPG algorithm g (see Appendix for details). We set the trajectory length to T 768 for all D4RL problems, T 256 for Acrobot, and T 196 for Pendulum. We also use 7 = 0.99 in all experiments. The domain details are provided in Appendix [ and the training provided in Appendix[]
details of STITCH-OR
Baselines. We include following model-free estimators: Fitted Q-Evaluation (FQE) 22
Doubly-Robust OPE (DR) (37, Importance Sampling (IS) (3 and Density Ratio Estimation
(DRE) [31]. We also include the following model-basee estimators: Model-Based (MB) (19 40)
and Policy-Guided Diffusion (PGD) [16]. The implementation details are provided in Appendix Appendix L.6 provides
proviids an additional comparison against Diffusion World Models (DWM) 10)
Metrics. Each baseline method is evaluated on each pair of behavior dataset and target policy for random seeds. We also generate ground-trute -ruth estimates of each target policy value by running each policy in the environment We evaluate the performance of each baseline using the Log Root Mean Squared Error (LogRMSE), the Spearman Correlation, and the Regret@) calculated as the difference in return between the best policy selected using the baseline policy value estimates and actual best policy. Furthermore, compare metries consistently across tasks, we normalize the returns following [[4] Appendix D C contains the technical details for metric calculation.

---

4.2 Discussion
Table 2 summarizes the performance of each method per domain, while Figure summarizes the aggregated performance averaged across all domains. STITCH-OPE outperforms all baselines in 11 of 15 instances (shown in bold), with general agreement among the different metries. STITCH- OPE soundly outperforms both single-step (MB) and full-trajectory (PGD) model-based methods. This reaffirms our argument in Section 3.3 that intermediate values of w provide a good balance between compositionality and compounding errors. Furthermore, STITCH-OPE performs particularly well according to rank correlation and Regret@ I (with very low standard error) and can accurately rank and identify the best- pest-perforring policy. This suggests that the target policy score function (with the negative behavior term) provides very informative guidance during denoising, allowing it to correctly evaluate target policies of varying levels of ability, even significantly from the behavior policy. Finally, we see that STITCH-OP CPE performance remains consistent across the problem dimension, highlighting the scalability of diffusion when applied to OPE for nigh-dimensional problems.
of those policies deviate
4.3 Off-Policy Evaluation with Diffusion Policies
To demonstrate the ability of STITCH- OPE evaluate more complex policy classes, replace target policies with diffusion policies, which have led to significant advances in robotics 41 (see Appendix [K for details). Since STITCH-OPE only requires the score of the target policy, it is computationally straightforward to perform OPE with diffusion policies, which is not the case for other estimators that require an explicit probability distribution (a(a)))
actions (i.e. IS, DR). D4RL results are provided in T We see that STTTCH-OPE outperforms all other baselines in out of instances, demonstrating robust OPE performance across multiple target policy classes.
FQE
DRE
MBR
PGD
Ours
Hopper
-0.21 0.01
-0.38 = 0.00
-1.56 0.02
-0.89 + 0.00
-1.65 # 0.01
Log RMSE
Walker2d
-0.59 + 0.01
-0.49 = 0.00
-0.81 0.01
0.00
-1.20 a 0.01
HalfChectah
+.01
-1.19 # 0.00
-.024
-0.96 0.00
-0,50 + 0.00
Hopper
0.35 + 0.06
0.35 0.04
0.68 0.02
0.00
0.81 # 0.01
Rank Corr.
Walker2d
0.03 = 0.04
0.45 = 0.03
0.47 # 0.02
0.52
0.46 + 0,09
HalfCheetah
0.59 + 0.01
0,80 # 0.03
0.75 + 0.05
0.46 0.06
0.81 = 0.02
Hopper
0.06 + 0.03
0.41 0.22
0.18 0.00
<0.01 0.00
<0.01 = 0.00
Regret@ |
Walker2d
0.24 + 0.02
0.59 0.13
0.17 + 0.02
0.23 0.00
0.03 + 0.00
HalfCheetah
<0.01 + 0.00
<0.01 + 0.00
0.03 + 0.01
0.02 + 0.01
0.02 + 0.01
Table 3: Comparison of OPE methods across environments when the target policy is diffusion policy; any regret shown as <0.01 is nonzero but rounds to zero at two decimals.
4.4 Ablations
We conduct additional experiments to test the sensitivity of STITCH-OPE to the choice of guidance coefficients (a and A), window size w, and data sets. Due space limitations, we defer results to Appendix We summarize the key findings below:
Sensitivity to Penalty Coefficients a and X (Appendix L.I). The best performance occurs when 0 < a, reaffirming claim in Section 3.2 that optimal regularization occurs for small X.
Sensitivity to Window Size w (Appendix L.2). The best performance occurs for w = 8, showing that STITCH-OPE provides an optimal balance between autoregressive and full- trajectory diffusion.
Robustness Across Datasets (Appendix L.3] We evaluated STITCH-OPE and all baseline methods on Hopper-medium-exper and Hopper-expert data sets. We found that STITCH- OPE outperforms all other baselines by a significant margin in out of instances (metric- dataset combinations) and maintained the most consistent performance across data sets.

---

Horizon
FQE
DR
IS
MB
PGD
Ours
T 128
-2.23 = 0.00
3.49
+3.22 0.01
3.06
3..5
-4.34 0.01
T 256
-1.36
0.00
-2.04
-.09
+3.03
-3.19 0.00
T 512
0.99
-1.19 0.06
-2.17
-..00
-2.65 # 0.01
T 768
-0.81 = 0.00
-1.32
0.94
-1.56
-.08
-1.99 # 0.01
T = 1000
-0.69 = 0.00
0.02
-0.81 = 0.00
-.024
-1.05
-1.61 0.02
Table 4: Log RMSE performance on the Hopper-medium medium dataset across varied evaluation horizon T.
Sensitivity to Evaluation Horizon T. Table shows the log RMSE performance the Hopper-medium data set for all baselines and various values the evaluation horizon T. We also set Ty 0.999 to better the performance differences between the methods for 7 = 1000. The values are normalized based on the global minimum and maximum across all evaluation horizon experiments. STITCH-OPE consistently outperforms all baselines for all values of I' considered, striking an optimal balance between auto-regressive and trajectory diffusion methods. Furthermore, STITCH-OPE performance slowly degrades as T increases, validating the theoretical upper bound of Theorem 3.3
Effect of Reward Estimation (Appendix) L.4). We compare STITCH- performance given access to the true (unknown) reward function and its performance using reward estimation Hopper-mediun minimal effect on the aggregate OPE performance. Furthermore, the error remains stable and uniformly low across all target policies.
The accumulated errors in the reward estimation have
Limitations
The theory STITCH-OPE relies on (standard) Assumptions 3.1 and 3.2 In practice, if there exist many (s, a) pairs such that (aass) 0 but (als) > 0, then the behavior data may be incomplete and diffusion guidance could generate infeasible trajectories and produce biased estimates of (Tr). Diffusion models trained on image data in other applications are often easy to interpret; however, evaluating the fidelity of the trajectories generated from the trained behavior model (.((T) is challenging when the state is complex and difficult to interpret or partially observable. Finally, while STITCH-OPE has demonstrated excellent performance on existing OPE benchmarks, it remains unanswered whether its benefits also apply to domain-specific problems outside robotics.
Conclusion
We presented STITCH- -PE for off- off-policy evaluation in high- nigh-dimensional, long-horizon environments. STITCH-OPE trains a conditional diffusion model to generate behavior sub-trajectories, and applies diffusion guidance using the score the target policy to correct the distribution shift induced by the target policy. Our novelties include trajectory stitching and negative behavior policy guidance, which were shown improve composition and generalization. Using D4RL and OpenAl Gym benchmarks, we showed that STITCH-OPE outperforms fate-of-thhe-at OPE methods across MSE, correlation and regret metrics. Future work could investigate online data collection to address severe distribution shift, explore ways to adapt the guidance coefficients incorporate additional knowledge into the guidance function (c.g. additional structure on the dynamics). also remains an open question whether the advantages of STITCH-OP apply to offline policy optimization.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers and area chairs for their constructive feedback. This work was partially supported by the Toyota Research Institute (TRI).
10

---

References
[1] A. Ajay, Y. Du, A. Gupta, J. B. Tenenbaum, T. S. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Con- ference on Learning Representations, 2023. URL
SPIfo2K9DFG
[2] Alquier and J. Ridgway. Concentration of tempered posteriors and of their variational approximations. The Annals of Statistics, 48(3):1475 1497, 2020.
[3] D. Brandfonbrener, W. Whitney, R. Ranganath, and J. Bruna. Offline r1 without off-policy evaluation. Advances in neural information processing systems, 34:4933-4946, 2021.
[4] G. Brockman. Openai gym. arXiv preprint arXiv:1606.01540 2016.
[5] F. Cérou, P. Héas, and M. Rousset. Adaptive reduced tempering for bayesian inverse problems and rare event simulation. arXiv preprint arXiv:2410. 18833, 2024.
[6] B. Chen, D. M. Monsó, Y. Du, M. Simchowitz, R. Tedrake, and V. Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[7] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273666 2023.
[8] S. Dankwa and W. Zheng. Twin-delayed ddpg: A deep reinforcement learning technique to model a continuous movement of an intelligent robot agent. In Proceedings of the 3rd international conference on vision, image and signal processing, pages 1-5, 2019.
[9] Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
[10] Z. Ding, A. Zhang, Y. Tian, and Q. Zheng. Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning. arXiv preprint arXiv:2402.03577) 2024.
[11] M. Farajtabar, Y. Chow, and M. Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, pages 1447-1456. PMLR, 2018.
[12] M. Fard and J. Pineau, Pac-bayesian model selection for reinforcement learning. Advances in Neural Information Processing Systems, 23, 2010.
[13] Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint 20XX::00004007299, 2020.
[14] J. Fu. M. Norouzi, O. Nachum, G. Tucker, ziyu wang, A. Novikov, M. Yang, M. R. Zhang, Chen, A. Kumar, C. Paduraru, S. Levine, and T. Paine. Benchmarks for deep off-policy evaluation. In International Conference Learning Representations, 2021. URL https //openreview.
[15] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[16] M. T. Jackson, M. Matthews, C. Lu, B. Ellis, Whiteson, and N. Foerster. Policy-guides diffuusion. In Reinforcement Learning Conference, 2024.
[17] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as big sequence modeling problem. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 1273 1286. Curran Associates, Inc., 2021. URL htpp:ttttts///poceeednng neurips wwwpurrps.ccppppper...epeee fbaaper_flles/ppper/ 002////1//9940004440088888442220
[18) M. Janner, Du, J. Tenenbaum, and Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, pages 9902-9915. PMLR, 2022.
11

---

[19] N. Jiang and L. Li. Doubly robust off off-policy value evaluation for reinforcement learning. In International conference on machine learning, pages 652-661. PMLR, 2016.
[20] N. Jiang, A. Kulesza, S. Singh, and R. Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 international conference on autonomous agents and multiagent systems, pages 1181-1189, 2015.
[21] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning, Advances in neural information processing systems, 33:21810-218233, 2020.
[22] H. Le, C. Voloshin, and Y. Yue. Batch policy learning under constraints. International Conference Muchine Learning, pages 3703-3712. PMLR, 2019.
[23] S. Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644. 2020.
[24] G. Li, L. Shi, Y. Chen, Y. Chi, and Y. Wei. Settling the sample complexity of model-based offline reinforcement learning. The Annals of Statistics, 52(1):233- 2024.
[25] Q. Liu, L. Li, Z. Tang, and D. Zhou. Breaking the curse of horizon: infinite-horizen off-policy estimation. Advances in neural information processing systems, 31, 2018.
[26] V. Liu, P. Nagarajan, A. Patterson, and M. White. When is offline policy selection sample efficient for reinforcement learning? arXiv preprint arXiv:23/2.02355, 2023.
[27] Y. Liu, P. Bacon, and E. Brunskill. Understanding the curse of horizon in off-policy evaluation via conditional importance sampling. In International Conference Machine Learning, pages 6184-6193, PMLR, 2020.
[28] Y. Liu, W. Chen, Y. Bai, G. Li, W. Gao, and L. Lin. Aligning cyber space with physical world: comprchensive survey on embodied ai. CoRR, 2024.
[29] C.,,u, J. Ball, Y. W. Teh, and J. Parker-Holder. Synthetic experience replay. In Thirty-sevntth Conference on Neural Information Processing Systems, 2023. URL.
[30] L. Mao, H. Xu, X. Zhan, W. Zhang, and A. Zhang. Diffusion-DICE: In-sample diffusion guidance for offline reinforcement learning. In The eighth Anmual Conference on Neural Information Processing Systems, 2024. URL httppp:://opaarreee.eeee//aartrt EI19qmMmvy
[31] A. Mousavi, I.. Li, Q. Liu, and D. Zhou. Black-box off-policy estimation for infinite-horizon reinforcement learning. In International Conference on Learning Representations, 2020.
[32] O. Nachum, Y. Chow, B. Dai, and I.. Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in neural information processing systems, 32, 2019.
[33] D. Precup, S. Sutton, and S. P. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 759-766. 2000.
[34] M. I.. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[35] M. Rigter, J. Yamada, and I. Posner. World models via policy-guided trajectory diffu- sion. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https / //openreview.
[36] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning, pages 2256-2263 PMLR, 2015.
12

---

[37] P. Thomas and E. Brunskill. Data-efficient off- off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pages 2148. PMLR, 2016.
[38] M. Uehara and W. Sun. Pessimistic model-based offline rl: Pac bounds and posterior sampling under partial coverage. CoRR, 2021.
[39] M. Uchara, C. Shi, and N. Kallus. A review of off-poliey evaluation in reinforcement learning. arXiv preprint arXiv:22/2.06335, 2022.
[40] C. Voloshin, H. M. Le, N. Jiang, and Y. Yue. Empirical study of off-policy policy evaluation for reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[41] Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. The Eleventh International Conference on Learning Representations, 2023. URL ftRttstts:: /ttpa:://ppeeeveeeeeeeee wwwarreiiwwnnn/eorrrrieeee net//forummdd-AAWVDD.....
[42] T. Xie, Y. Ma, and Y.-.. Wang. Towards optimal off- evaluation for reinforcement learning with marginalized importance sampling. Advances in neural information processing systems, 32, 2019.
[43] M. Yang, O. Nachum, B. Dai, L. Li, and D. Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551-6561, 2020.
[44] M. Yang, B. Dai, O. Nachum, G. Tucker, and D. Schuurmans. Offline policy selection under uncertainty. In International Conference on Artificial Intelligence and Statistics, pages 4376- 4396. PMLR, 2022.
[45] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model- based offline policy optimization. Advances in Neural Information Processing Systems, 33: 14129-14142, 2020.
[46] M. R. Zhang, T. Paine, O. Nachum, C. Paduraru, G. Tucker, ziyu wang, and M. Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations, 2021. URL httpps://openreviee....e.. www.peerrrveee..eefforurumm net/forum? id=kmqjgSHXby
[47] R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference on Learning Representations, 2020.
[48] 7. Zhu, H. Zhao, H. He, Y. Zhong, S. Zhang, H. Guo, T. Chen, and W. Zhang. Diffusion models for reinforcement learning: survey. arXiv preprint arXiv:2311.01222, 2023.
13

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All theoretical and empirical claims have been summarized in the abstract and match the results attained in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section 5 discusses the assumptions required obtain the theoretical bound of the method and attain good empirical performance. We also mention that, while the approach performs well on standard benchmark problem sets, its performance domain-specific problems is unclear.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: Appendix [C Appendix D contain all related definitions, theorems and proof' that support the theoretical results stated in the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Appendix H discusses how baseline methods were setup well all hyper- parameters needed to reproduce the results. Appendix G discusses policy calculation. | discusses how the domains were set up, and Appendix discusses how metrics were calculated. Appendix discussses problems. Together, these sections were included allow reproduction of the experiments to the best of our ability.
Appendix
parameters needed to run STITCH-OPP on all
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
15

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Anonymized code is included in the zip file as part of the supplementary material, along with instructions to run the code in readme file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https
//ipps.ccppubii.. wttp://ppppppppu.aeeeeeeeo.op http://uuieeeeeeeeuuuiieeeeeeees http///oooubmsssioooooo sooiiiiiiiiiiiyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The Appendix contains all relevant training and test details, hyper-parameters and other important design considerations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All tables of numerical results and plots clearly show standard error bars and intervals where appropriate.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix M provides the computer resources used and the total time of experiments of our method.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read and adhere the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We believe the current paper is foundational research, thus we do not foresee any direct societal impacts of the work.
17

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The current work uses code bases from other packages. Their accompanying papers were cited, URL links to the codebases were included, and their licenses (where available) were mentioned in the Appendix.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
18

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide code to reproduce the experiments as part of the supplementary zip file. Instructions for running the code are provided in readme file included in the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The research method does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation
Supplementary Material
Abstract
This supplement the paper discusses algorithmic and experiment details that were not included in the main paper due to space limitations. It includes proofs of all main theoretical claims, well as all configurations and parameter settings that are required to reproduce the experiments. It includes additional experiments and ablation studies that were excluded from the main paper due space limitations.
Contents
Pedagogical Example for Guided Diffusion
22
Gaussian World Domain
22
Proof that Conditional Diffusion Increases Entropy
23
Theoretical Analysis
24
D.I Assumptions and Detinitions
24
D.2 Analysis of the Bias
................................
25
D.3 Analysis of the Variance
27
................................
D.4 Proof of the Bias- Variance Decomposition (Theorem)
31
Pseudocode
31
Domains
31
Policies
32
Baselines
33
Metries
35
STTTCH-OPE Training and Hyper-Parameter Details
36
Diffusion Policy Training and Evaluation
37
Additional Experiments
37
L.1
Sensitivity to Guidance Coefficients a and
37
Sensitivity Window Size w
38
L.3
Sensitivity to the Choice of Behavior Policy/ Behavior Datasel
38
Sensitivity to Reward Estimation
40
L.5
Trajectory Visualizations/
40
L.6
Comparison with Diffusion World Models
40
M Computing Resources
41
Related Work
42
21

---

A Pedagogical Example for Guided Diffusion
We consider the simple two-component mixture of Gaussians with density function
p(x) (z; 1, 0.5²) + 0.5.(( -1,00"))
where (2; μt o²) is the density function a (H, o²) distribution. Using the standard substitution
c(x* k) og V logp(z")
in the backward diffusion process, produces the backward diffusion process
x b ~ N(((** + (1 (1-aa) log p(=)(((())))))))))))) oz).
To illustrate the effects of guidance function on the sampling process, we consider the guidance function associated with the (unscaled) score (1, 0.5²) distribution, i.e.
g(x) - (x 1)/0.5².
Then, the guided backward diffusion process has mean:
+ ag V + ogg(t")
Van
x + - (n) (V logp(**) + opg(t*) (ar -x)))
Vas
which corresponds to standard backward diffusion process with the modified score function ogy(" (alaa")""r) /ou/(l okk),
V logg((")
which would place more weight the rightmost mode of the Gaussian mixture during the backward diffusion process.
We run the backward denoising diffusion process using the exact score function V logp(a*)) The sampling distributions of æ² are plotted at various denoising time steps in Figure
50
score of p50
026
1000 of 950
0.02
0.40
So1s
001
001
-1.5
005
0.0
-1000
082
-00
100
-3..0
0.5
1665
Scone of p00)
0.001
0.0
25
25
2005
0.0
25
0.0
2055
00
0.2
Figure 3: Pedagogical example illustrating guided diffusion sample generation for Gaussian + 0.55 0.552)).. 0.5²). Top row: histograms of samples from unguided backward diffusion steps A: 8, 6, 4, O, where V log p(x) is the score of the Gaussian mixture shown in blue. Bottom row: histograms of samples from guided diffusion (2 using the score function of 0.,5) distribution, i.e. g(x) (( 1)/0.52. The modified score function corresponding to the guided diffusion process is shown in blue. The guided score function (the score of the actual sampling density) is significantly shifted and skewed, relative to the original score function, the intermediate denoising time steps (k 6,4). This ensures that the right mode of the Gaussian mixture is sampled more frequently during denoising.
mixture (1,,( 0.5²)
B Gaussian World Domain
The GaussianWorle domain is toy 2-dimensional Markov decision process defined designed to illustrate and compare generalization and compositionality diffusion models (Table is defined as follows:
22

---

Decision Epochs. The decision epochs are t = 2,,, T where we set T = 128 in our experiments.
State Space. S R2 describes all positions (æt, M/) of a particle in space at every decision epoch It is assumed that zy is the x-coordinate and yt is the y-coordinate. The initial state is = (0, 0) unless otherwise specified.
Action Space. = R describes the (counterclockwise) angle of the movement vector of the particle at every decision epoch, relative to the horizontal.
Transitions. Letting be the angle of movement of the particle at time t, the transitions of z; and y are defined as follows:
2o4 2, 0.02 cos(a, + Ex), M++11 = Vr + 0.02 sin(a, + Et) E (0,
Here, is an i..dd Gaussian noise added to the actions before they are applied by the controller.
Reward Function and Discount. The problem not solved so we leave the reward unspecified. We also leave the discount factor unspecified.
C Proof that Conditional Diffusion Increases Entropy
We begin with the following definitions.
Definition C.1 (Entropy). Let p(z) be density function of random variable X with support X. The entropy of X is defined
H(X) = / p(x) log
p(x) dr.
Definition C.2 (Conditional Entropy). The conditional entropy of X given Y support y is defined
as
H(X EEEyyHH(M))) = =y].
Our goal is to prove
Theorem C.3. Let S₁ be the random state at time sampled according to the conditional distribution
= S| = x, A₁ = u). and let Ag be a random action following some conditional distribution
p(A, a|S,- x). Then H(r(S,)
Ag. where is (random) sub-trajectory
beginning in state St.
Proof. First, letting U = Ao,, S ... Ar-1).
observe that:
H(U, T|U
/1 P(U u, U s)
du
pPU u, =s)
/I P(U U. s))og
p(U u|S[ s)p(T(U u. u. ss) ) dudr
-fDUU P(U u|St s)gog
p(U u/S, = s) du
+ / |PUU u/S = p(T|U St =s) log
1
du dr
p(T[U u, St
= HU = + (H(|U = u. = =s]].
Next, using the additivity property of expectation and law of total expectation:
HUU, ((.)) E.es, [H[U)S) = =}]] + E.ES (H(T u, = x)] HUUS))) + U,,
23

---

Next, we prove sub sub-additivity of conditional entropy:
H(U,T|:: = s) - (U]S: = 5) - (T]S = 3)
= -// p(U = u, T|S; = s) log
piU = T = =s} dudr
/ P(U = u|S, = Log
1
du - / [[T]Si=s)loog
1
dr
p(U u/S = 8)
p(T]S: = =s)
// plU = u, T| = log
p(T]S, = +)p(U u|S, = 8)
du
p(U = u, US =s)
- log /f p(U - u, u|s =
[p(T]S:= s)p(U u|st 8)
du
p(U = T|5 = =s)
log
=.
where the inequality in the derivation follows by Jensen's incquality. This implies that H(U, T|; = s) - H(U S; = 8) + HT|SS = =))
Taking expectation of both sides with respect to St, and using the monotonicity and additivity properties of expectation:
(U, =(S.) = E,Ss, (H( =s]
< E,,,, (H(U]S+ 5) + H(([S. s)] +
Finally, putting it all together:
U() H (U,+[()) H(U)))) & H(U[S)) + H([[)) H(USS)) = HTT())),
which completes the proof.
D Theoretical Analysis
D.1 Assumptions and Definitions
We decompose a full trajectory of length T into N T/w non-overlapping sub-trajectories (or chunks), each of length w. Each chunk S; E ((w) is defined as
S = (News Gim; Siai4 Cin+l1 ((1+1)))
Let the full trajectory be defined as
S == S1. SN
We define the boundary state X, as the initial state of chunk Sü;
X, Siws
0.1...M. N,
which form the backbone of the generative process.
We assume the following factored generative process for trajectories
N-1
p(So, SI,..., p(Xo)
II p(S, X.) p(Xi+1 S.).
This implies that the boundary state sequence = (Xo, XI,..., Xx) forms first-order Markov chain
p(X.+1 | (s)) = p(X((+ X²).
Each chunk S, produces scalar discounted return Y,, defined
Y = f(S.) Σ
J R (sww aiw-s))
3=
24

---

where R is a learned reward model, and E [0, 1] is the discount factor.
Given bound Rmax the absolute reward, define the maximum per-chunk return bound
as:
Reaax (1 7m))
B 1= 7 Rmax
|Y]] S Bw.
The cumulative return over the full trajectory is approximated by
Yi,
1=0
and the expected return under the target policy π is:
N 1
Ep. [J]
Epp
w
Definition D.1 (Chunked Behavior Distributions). Let P denote the true distribution over behavior
(we)
((rr) be the learned conditional distribution modeled by the diffusion process. These distributions describe how chunks are generated given boundary states:
chunks S,, and let P3
PB (w) (S, X+),
(S, XX).
Definition D.2 (Total Variation Distance). The total variation distance between two probability distributions P and Q over the same measurable space X' is defined
TV(P,Q) sup PP(A) Q(4))
ACX
We now restate the two assumptions presented in the main text for convenience. Assumption D.3 (Bounded Likelihood Ratio). There is a constant к such that (alss) E S and E A.
S for all B(ass)
Note that this assumption can be easily verified in our experimental setting. Since the action spaces are closed intervals and the behavior and target policy distributions are both represented as truncated Gaussian distributions, the ratio of the two policies is bounded over the action space.
Assumption D.4 (Chunk-wise Model Fit). The total variation distance between the true chunk (w²) and the learned conditional distribution P'3 _((r) bounded by some constant 83 0, distribution P3
TV
Pg
(w)
PP
(w)
5ōg.
D.2 Analysis of the Bias
We begin by bounding the total variation distance between the true target distribution (n)) and the guided model ((w))
Lemma D.5. The total variation distance between the guided model p(_)) and the tree target (w)
distribution pr satisfies
TV (pl (w) p(w) I x²? 6,
Proof. By the definition of total variation distance
TV (w) ((w)) 1 2 2/ /
(1)) (T) p( dr.
Using the reweighted form of each distribution
( ( ((w)) = 2 /
p())) -m)
(D) (m))
(T)
n (a, 87) dr.
3(a (5))
==0
25

---

and applying the bound on the likelihood ratio (Assumption D.3 P/ (2))
TV pl
Ai
/
P3 (w))
(T)
(T) dr n² TV(p3 (w) )) S k: 8g.
2
This completes the proof.
Let the total variation distance between the true target distribution and the guided diffusion model be denoted by
σ = TV .ppp
(w)) p(w) p
By Lemma D.5 we have the bound
5x S RM 63.
We now derive a bound on the absolute bias of the estimated return when sampling chunks from the guided model
(w) p instead of the true target distribution _
Lemma D.6 (Expectation Difference Bound via Total Variation). Let p and be two probability densities on probability space X. Let
1l sup [x)]
=EX
be the supremum norm of a bounded function f + R. and let:
4ll1 1 [((x) (xx) dæ,
|pp
TV(p,) 9) = 1ppp--91 al11.
Then
((epl[(xx) - [f()]] 22 TV.
Proof.
Bp[f] - E,,f] 1/x f()) / f(x)) f(z)
1 f(x) (p(x) - q(x)) dx S / [((0)) [p(x) - g(x)] dx
I ll ((lote) g(x) dr 2llfllo TV(p.q)
This completes the proof.
Lemma D.7 (Marginal TV Bound via Conditional TV). Let p( 8) and D(x 8) be conditional densities over chunk x E T(w), given state E S, and let p(s) denote the marginal distribution over Then
TV (foles (lptrl))uo))) s))(s)dss / ((zl s)u(s)ds) S / TV (pl- 10.00 8)) p(s)ds.
In particular, TV s), 8)) S for all 8, then
TV Se.
Proof. Let p(x) f p(x s)m(s)dss p(x) f p(x s))(s)ds. Then:
TV(p.P) 2 / (p(x) p(x) dz
/ / ((s) [p(z s) - p(x 8)] ds dr
= 2
1
// ((s)
$ 2
|p(x 5) p(æ |s] ds
(by Jensen's inequality)
/ m(s)
[
2 1p(x (s) p(x ()e dr ds
= / m(s) TV(((- | s), p( ss)
(FV [s). 5)) < uniformly, the integral is bounded by
26

---

Theorem D.8 (Bias Bound for STITCH-OPE), The bias of the return estimate under the guided diffusion model satisfies
2B
1p. - J(x))
8g.
70
Proof. The return estimator is:
N-1 Y"r,
where = f(S.)
Σ
7 Riswwty Asw+s).
0
Thus, the bias is:
N Σ
Ba [J] - Ey. [J]
Σ Fm (Ep, [X] - Ep, [Y]]
S
he Bp, [Y]] - Ep, [Yi]]
For each chunk i, Y, depends only on S. with marginal distributions p Pr. respectively. By Lemma D.6 and Lemma D.7
(()))) and pm (w.)) under Pa and
B M
Ep. [У] I 2 sup |Y] TV (m))) p x)
Since | a)l & Rmax the per-chunk return is bounded:
1 7-70 Rmax
Y
S
7 Rimax
?
1=0
Using Lemma D.5 we know that TV (w.,))
p ofmmr)) S dn. Thus we have
Rmax (1 - 70) Epr [Y]] - Ep, [Yll 22
5x.
1
Summing over chunks:
N-1
Rinax (1- 770)
Rmaxx (1 - -
N-1
Ep, [J] - Ep. Ep.15]
- Σ
8 =
iu
--7
1
1=0
1=0
The geometric sum is:
Σ
q
Thus:
2. Rmax (1 - -0))
1
2B 1 7a
Epp [J] 1
[5]
1-7 -r
8g.
70
This completes the proof.
Corollary D.9 (Bias Bound in Terms of Model Fit 5g). Under the assumptions
sup, TV(p (w))). pa
((w))))
< 8n S RM 5g and sup, |5((T) I Ruax the bias satisfies 1-7
2B w k 8g. Eg - ((m) s
D.3 Analysis of the Variance
Lemma D.10 (Conditional Independence of Chunk Rewards). Let X; = 8in be the boundary state at the start of chunk S, and define:
Y = f(S))
Σ rd R(s alw+g))
3=
Assume the generative process satisfies the following properties:
27

---

Xg
X1
X2
XN
So
S
S2
SN-1
Yo
Y1
Y
YN-1
Figure Illustration of the sub-trajectory decomposition. Each chunk St generates a reward sequence Y, and leads to a boundary state Xi+1-
Each chunk St is generated independently given X;
Thhe return Y; is a deterministic funccitoo of S
Then for all # the returns Y; and Y, are conditionally independent given the full boundary state chain Xo, .N.....NN..
Y
Y, ,,
Proof. Refer to the graphical model in Figure 4 The nodes Xo. XI,..., X form Markov chain. Each chunk S; is child of X; and each return Y; is child of S.
Now consider any path from Y, to Yj. Such a path must go through:
Y, S. X, ~n X
X, - -5
All such paths must traverse through at least one boundary node Xx. Since are conditioning all Xo, ... , XN, and these nodes are non-colliders on every path from Y, Y, all such paths are blocked. By the criterion of d-separation (see, e.g. Chapter 8 in [49]), this implies Y; IL Y; Xo...., .NN O
Theorem D.11 (Variance Bound). Let P. denote the trajectory distribution induced by the guided diffusion model, and pz the true trajectory distribution under the target policy. Let J be the return estimator using a learned reward model. Then
2
2B 7,20
Varg, (J) S Varp- (J) 10
B "
8g
+
R² 83,
W
where By, denotes the maximum per-chunk discounted return.
Proof. We begin by applying the law of total variance under the guided model distribution P.
Varg, (J) Ex, [Varp, (J [x1x)] + Vars, (Ep. [J 1x))
Using Lemma D.10 we have that the chunk-level rewards Y, and Y, are conditionally independent given the boundary states Xa, X XI,..., XN:
Y, 1L Y, X,........ Xx for all H
Using this conditional independence, the variance of the total return under pm, factorizes:
N
Varg, X Varg.
Σ
7
Y
X
2iw Varg, (X: X.).
28

---

To bound the difference in conditional variances, we apply the law of variance
Var(Y; X;) B| X,J (E x,))².
Let us define a bound on the per-chunk return magnitude:
Remxx (1 7
B
S Bw: Y B
Using Lemma D.6 ( Expectation Difference Bound via Total Variation), we have Ep [f] Ep, [f]l s 26 Il
Applying this with f Y, and ʃ Yi and using the bound |Yi] S Bu, we obtain:
E Y| Ep,, [Y:]] s 26 Bws
18p. Ep, [Y S 26, B w
We analyze the difference in conditional variances:
Varp, (Yr X,) - Varp, (Y; X;)
Eg. [Y,"] - Ep, [Y - (E [Y]] Ep, [vT?)]
< Epp [Y,²] - Ep, [Y,"] + Epp [Y]] Ep [Y]?
|B,, !Y, - Ex,, [Y?3] + [Bp, [Yi] Ep, [Y]] [Eg, [Y]] B,, [K]]
28, + (25 B..)) (2
66, B
This uses the triangle inequality and the identity |a2 - 6- = blla b). along with the bounds
[Yd < B.. ||sl < B and total variation guarantees from Lemma D.6 Then
Varg, (Yr X,)) Varp, (Y; Xx) < 66 B3.
We now return to bounding the first term in the law total variance
N N-1
Ep-
Varp, (J xx] = Ep.
Σ
7 2010 Varg, (Yr X.)
1=0
Using the bound from the previous step
Varg. (Y: X;) S Varp- (Y:1x)) + 65 B
Taking expectation over P. on both sides
Ep, [Var,, (Y; Xx)] S Es, [Var,, (Y; X:)] + 66,, B
Now, using the expectation difference bound from Lemma D.6 again:
Epp [f] E Ifll 11flloo where f(X.) Varp, (Y, Xx) B2
So
Ep, [Var,, (Y, X:)] S Rp.. [Varp, (Y: Xx)] + 25 B2.
Combining both components
Ep, [Var,, (Y: Xz)) < Ep,. [Varp. (Y: X.)] + 85 B
Summing across all chunks:
N-1
Ep, Varg, (J [ = Σ
7 25w Ep, [Var (Yr X,)]
1=0
N
7 2iw (Bpe [Varp= (Yr X,) + 88 B²))
29

---

We can split the sum and factor out constants:
N -
N-1
Ep-
Varp- (J [x)]
Epr Varp-(( ( + 86, B
1=0
1=0
Let us define the chunk-level return variance
N-1 Σ
Ey, [Varp, (51x) =
7 2iw Ey, Varp, (Yi X:)]
0
Therefore
(aarp, (5 x)] + 86 B
Ep. Varga (J x)] s Ep-
2w
To complete the law of total variance, we now analyze the second term:
Var,., (Ep. [ Varş, (Zp), where Zp
(xx),, _x(x) Ep, (Yr Xx x].
We define the corresponding ideal (true model) version:
Zp Σ (Xn),
[u(x): Ep, DY Xa x].
Our goal is to bound the variance difference:
Amean = Varp, (Zp) Varp, (2p) (Mp Mp) - (mp mp) + mp).
where M3 == p,, [Zg]. mp == pp,, [zp] and similarly for Mp. mp-
Insert and subtract common term:
my mp
Σ
(en [Xx)] Ex (Xx))) +
Σ
(Lpp [xx Ep. Lōx (X+)))
k=
Each term is bounded by 28 Bw so lmg- - mpl S 4No B,.
Expand both squares:
N
2 = Σ
((xx) +2
Σ
Iu (Xx)),((X)),
k=U
0skcesN-1
N-1
22
(Xx)
+2
Th (Xe)).
K=U
OSAcEEN-1
Each term (both diagonal and cross terms) is bounded in total variation with sup-norm B yielding
Mp - Mpl $ 2N B2.
From the bound on the means:
|mpl (mpl NB
[mg mpl S 2N 2NB..
So, the product term:
1mm-- - mp) + mp)1 S B. = 8N B2
Combining both:
Amean |ar (Zp) Varp, (Zp) < 2N 8, B + 8N B 10. 8, B2
which yields
T
Vary. [Ep. [J x1) Varp- (Ep- [J X]) < 10
8 B
30

---

Combining the two components from the law of total variance, we conclude:
Varg, (J) Ep- VVrp, (J xx] + Varg. (Ep_1J 15 |x])
2 Ep- VVarpr (5 [x]] (
+ 86z B
+ Varpm (BpT XJ) + 10
T
8n
B²
72
W
2
86 B
= Varg, (J) 10
B +
2w
By Lemma D.5
T
8B 2
Varp, (J) S Varp, (J) 10
B² nnd 83
8g,
w
and the proof complete.
D.4 Proof of the Bias-Variane Variance Decomposition (Theorem 3.3
Finally, we can bound the mean squared error STITCH-OPE.
Theorem D.12. Under Assumption D.3 and D.4 and using the notation of Thhhoee D.8 and Theorem D.1l the mean squared error of STTTCH- CICCC--PE is bounded by
Ep ( s (1- x1-7))n
2B x= 69)
T W
8B 722
+ 10
B
8g
k x 8g + Varp- (J).
Proof. We start by adapting the standard bias-variance decomposition our setting:
Ep (J - [(T))
Epz (J [Ep_[J] Ep, 1 [J] - [TT))
Epg (J 3J7-Ry
[J)) (J.
+ Eg. (T [J] J J(m))
+ Ep, [ [ Epp [Jp
[J]
J(m))]
Varø. (f) Biasp, (J)² + (E [J] - (E,) 1J Ep. (51)
Varp. (J) Biasp, (J)?
since the last term zero. Plugging in the bounds Theorems D.8 and D.11 completes the proof.
E
Pseudocode
A high-level pseudocode of conditional diffusion model training in STITCH-OPE is provided as Algorithm A pseudocode of the off-policy evaluation subroutine for single rollout is provided as Algorithm Empirically, we have found that per-term normalization of the guidance function (line 9) resulted in more consistent performance, and allowed the guidance coefficients a and À to be more easily tuned.
Domains
We include experiments on the medium datasets from the D4RL offline suite [3], and Pendulum and Acrobot domains from the OpenAl Gym suite [4]. We set the evaluation horizon to T = 768 for D4RL, = 256 for Acrobot and = 196 for Pendulum, and we use 7 = 0.99 in all experiments. Furthermore, Acrobot uses discrete action space and is incompatible with our method, so modified the domain to take continuous actions. Table summarizes the key properties of each domain.
31

---

Algorithm Conditional Diffusion Model Training in STITCH-OPE
Require: diffusion model ca(T, k(s), behavior data DR, 2 O, learning rate 1) 0, f [ and {ax k== positive
1: āg + II--1 Og for l 1...KK
2: initialize randomly
3: repeat
4: sample length-w sub-trajectory 70 (so: 00, 81.... Sw) from D3
5: sample k ~ Uniform(
D Sample denoising time step
6: sample 1)
Sample pure noise sub-trajectory b Gradient descent step on
7:
VeL(0) t Valle 6 c0 (axtt + one, one, k|
8:
+ 0 (L)
9: until converged
return c0
Algorithm 2 Off-Policy Evaluation in STITCH-OPE
Require: diffusion model co(T, (Algorithm 11 empirical reward function R(s, a), behavior
policy B(als), target policy (aas), a 2 0, X w 2 (divides T), {ox) and {ax ) positive
1: -
2: sample =0 dg
b Sample initial state Generation for decision epochs wt to w(t + 1) b Sample pure noise sub-trajectory b Denoising step
3: for 0 to T/w w do
4: sample (1+1 ~ T)
5: for = K to do
6:
Pk-1 + Jak
nk ([t:((+1) Layy (T wott k
Mean of diffusion
9 E - -1 V, log * (a
7:
Compute guidance term
8:
9% - Dw(tt)) w((++))- 11-1 V, log (aal)s))
I Compute 3 guidance term
9:
sample k-1 119 (12) A(g /lg//1ll11 111)
9k a
Compute normalized guidance Apply guided diffusion step
(e+l N (ux + oig T
11: end for
12: J+ J E ((+1))-1
m=mi 7tt 2(55,a0)
b Update return using denoised -
(ttsa(t++1))
end for return
14:
Description
Hopper
Walker
HalfCheetah
Pendulum
Acrobot
state dimension
11
17
17
3
6
action dimension
3
6
range of action
[-1,
[-1,
[-1, 1]
[-2, 2]
[-1,
rollout length T
768
768
768
196
256
discount factor y
0.99
0.99
0.99
0.99
0.99
Table Properties of D4RL. 13 and OpenAl Gym
benchmark problems.
G
Policies
D4RL Offline Suite. Behavior and target policies and their trained procedures are described in [14. and the policy parameters are borrowed from the official repository at htttps://gi t/ub com/ google-resease www.ereeeeeeeeeeeeeeeeeeeeeeeeee heahh/deepopppp (Apache 2.0 licensed). The 10 target policies of varying ability. Te, . Té are obtained by checkpointing the policy parameters 01, f..... 010 at various points during training. Each target policy network models the action probability distribution म; (a]s) using a set of independent Gaussian distributions, predicting the mean and variance (14, of) of each action component a; independently. This allows the score function of the target policy to be easily computed. As discussed in the main text, all policies are derived from the medium datasets in all experiments.
32

---

OpenAI Gym. We model target policies 11. ..... R5 as MLPss and train them in each environment following the Twin- Weinn-elayed DDPG (TD3) [8] algorithm. The total training time set to 50000 steps. and we checkpoint policies every 5000 steps. The behavior policy is set to the target policy x3- The complete list of nyper-parametens provided in Table
Description
Value
number of hidden layers in actor and critic number of neurons per layer in actor and critic
2
256
hidden activation function output activation function
ReLU
tanh
Gaussian noise for exploration
0.1
noise added to target policy during critic update
0.2
target noise clipping
0.5
frequency of delayed policy updates
2
moving average of target O' learning rate Adam optimizer
0.005
0.0003
batch size
256
replay buffer size
1000000
Table 6: crpper-araaeterrs for training target policies
OpenAl Gym domains.
Bounded Action Space. Since the action spaces for all domains compact bounded intervals, need to restrict the action space of the policy networks during evaluation. We accomplish this by applying the tanh transformation each Gaussian action distribution and then scaling the result to the required range. Note that this transformation constrains the action probability distribution of all policies bounded range, and thus satisfies the requirement of Assumption 3.1
H Baselines
The following model-free baseline methods were chosen for empirical comparison with STITCH-
OPE:
Fitted Q-Evaluation (FQE). [22] evaluates target policy π by estimating its Q-value function Qo a) using a neural network. The loss function for f is
(r) = T (aann)))D ( a'war(-l'))
We follow [57 53 and learn a target Q-network (s, a) in parallel for added stability. We use the AdamW algorithm (55) for optimizing the loss function in minibatched setting, with gradient clipping applied to limit the norm of each gradient update to 1. The complete list of hyper-paramette used is provided Table 7
Description
Hopper
Walker
HalfCheetah
Pendulum
Acrobot
number of hidden layers number of neurons per layer
2
2
2
500
500
500
256
100
hidden activation function learning rate of AdamW optimizer
sigmoid
sigmoid
sigmoid
sigmoid
sigmoid
0.001
0.003
0.00003
0.003
0.001
moving average of target /'
0.05
0.05
0.001
0.005
0.05
training epochs (passes over data set)
100
50
70
100
200
batch size
512
256
256
128
512
Table Hyper-parameters for Fitted Q-Evaluation (FQE).
Doubly Robust (DR). 19 37
leverages both importance sampling and value function estimation
to construct a combined estimate that is accurate when either one of the individual estimates is correct. First, we define an estimate Q(s, a) of the Q-value function of policy म, and let V(s) =
Bx ((, a)) be the corresponding value estimate. We also define pt = B(ay(s))
r(ar(s)) (s)) as the policy
33

---

ratio at step t. Then, the DR estimator is defined recursively as
V = V(+)) + Pt (Tx + VDR - ((s),a)))
such that the policy value estimate ( DR. We parameterize both (s,, a) and ((s) as MLPs and train them using AdamW in mini-batched setting, Similar to FQE, we also update target value network improve convergence. The full list of hyper-parameters is provided in Table
Description
Hopper
Walker
HalfCheetah
Pendulum
Acrobot
number of hidden layers number of neurons per layer hidden activation function learning rate of AdamW optimizer moving average of target /' training epochs (passes over data set)
2
2
2
2
500
500
500
256
100
sigmoid
sigmoid
sigmoid
sigmoid
sigmoid
0.0003
0.003
0.003
0.003
0.00003
0.05
0.05
0.05
0.05
0.001
50
50
50
100
100
batch size
32
256
512
256
128
Table Haper-parameters for Doubly Robust (DR) estimation.
33 evaluates the target policy by importance weighting the full trajectory returns in the behavior dataset, i.e.
Importance Sampling (IS).
x (at)st) (II ==
Σ
> R(si, R(st, an)
Jis(T) = ETopo
(t)
=
requires access to the target and behavior policy probabilities in order to compute the weighting. Specifically, we use the per-decision variant of IS (PDIS), i.e.
T-1
f
x (a,l(su) R(se,ai)
(P) E Eycue
Σ
n'
II
B(aulsa)
which has lower variance than IS.
Density Ratio Estimation (DRE). [31 estimates the ratio w(s, a) as/ a) of the discounted state-action occupancies of the target policy π relative to the behavior policy 8. The discounted state-action occupancy of a policy μ {B, π) is defined as
Er of p(s = a.
= lim T'
Σ 7
where p(s; = s, a, μ)) indicates the probability of sampling state-action pair (s, a) from μ at time step 1. We also tested the variants of DICE 43 but found their performance to be unsatisfactory, so they have been omitted from the study. The target policy value estimated as
(( = (e [w[s, a)
wn(s, a) is parameterized as a feedforward neural network and parameters are trained using Adam in mini-batched setting. Fixed hyper-paramedt necessary reproduce the experiment are listed in Table Additionally, since the method requires a kernel function to be specified, we use a Gaussian kernel k(x, xx) exp (-nllæ z' 2'|l²), where and æ are concatenations of the (standardized) state and action vectors. Since this requires setting a kernel bandwidth r7 > which affects the overall this baseline for different values (0.01, 0.1, 1, 10, 100) and report the best performing result (according to RMME).
performance significantly,
The following model-based baseline methods were also chosen for empirical comparison with STITCH-OPÉ. They were chosen to determine the benefits of STITCH-OPE compared to fully autoregressive sampling. i.e. w 1, and non-autoregressive sampling, i.e. w = T.
Model-Based (MB). 19 40 consists of learning dynamics P(s'|s, a). reward function R(s, a) and termination function ((s) trained on the behavior dataset to directly approximate the data-generating
34

---

Description
Value
number of hidden layers of w(s a)
number of neurons per layer of w(ss a) hidden activation function output activation function learning rate of Adam optimizer training epochs (passes over the data set)
256
Leaky ReLU SoftPlus
0.001
20 (D4RL), 200 (Gym)
batch size
512
Table 9: Hyper-pparameters for Density Ratio Estimation (DRE) [3]]
distribution of the target policy, (T(r). P directly predicts the next state s' given the current state and action Both P and R can be found by solving a standard nonlinear regression problem, and D can be found by solving a binary classification problem trained on termination flags in the behavior dataset. We parameterize all functions as nonlinear MLPs and obtain their optimal parameters using Adam in mini-batched setting. Once we obtain their optimal parameters, we estimate the target policy return by generating 50 length-T rollouts from the estimated model, and average their empirical cumulative returns. The necessary hyper-pan pepeepparameters are described in Table 10
Description
Value
number of hidden layers number of neurons per layer
500
hidden activation function
ReLU
learning rate of Adam optimizer training epochs (passes over data set)
0.0003
100
batch size
1024
Table 10: Hyper-parameterr for Model-Based (MB) estimation.
Policy-Guided Diffusion (PGD). [6] takes a generative approach by simulating target policy trajectories using guided diffusion model. We follow the original implementation by training a diffusion model on the behavior data, using the official implementation located at hhttps:// github. cww/
(MIT licensed). We then generate 50 full-length trajectories from the model using guided diffusion [8] with the guidance function Asimple (T) V, E, log (a,]ss), using which estimate the empirical return of the target policy. All hyper-parameters for training the diffuusion models are fixed as per the original paper and codebase (see Appendix A therein for details). However, we found that the policy guidance coefficient a and guidance normalization both have significant effects on performance, thus we ran PGD for different choices of {0.001, 0.01, 0.1, 1.0, 10, 100, 1000) with and without guidance normalization, and report the best performing result (according to log- RMSE).
I Metrics
Let -i,... To be the target policies, (T.).
(T:)) J J2(T)).... Js(T;) be the estimates of the target policy
values across the 5 seeds, and (((m),--- ( be the target policy values estimated using 300 rollouts collected by running the target policies in the environments.
The following metrics were used quantify and compare the performance of STITCH-OPE and all metrics:
Log Root Mean Squared Error (LogRMSE). This is defined as the log root mean squared error using the estimates J, (T1o) and the ground truth returns J (TTO), averaged across seeds j = 1 ...5. Mathematically,
log
Σ
(()))))
10
1=1
t=1
35

---

Spearman (Rank) Correlation. This is defined as the Spearman correlation [62 between the estimates (m(10) and the ground truth returns seeds j = 1.
J (m averaged across
Regret@1. 1. This is defined as the absolute difference in return between the best policy selected using
the baseline policy returns (T,) and the policy selected according to the ground truth estimates (Tx)) averaged across seeds j 1........ i.c:
J (megaa)
where ty argmax Jo J
i= 10..0.10
Normalization. In order to compare metrics consistently across environments, we follow 14 and use the normalized policy values:
(,( - V min
where
Vnúin = min (TT). Vmaz = max (T,,,,
V Vmaa Vmn
where Vmiin and Vmar are the minimum and maximum target policy values, respectively.
Error Bars. All tables and figures report error bars defined as +/- standard error, i.e. å/ vn where ā is the empirical standard deviation of each metric value across seeds and n the number of seeds (fixed to for all experiments).
J STITCH-OPE Training and Hyper-Paramete Details
Diffusion Model Architecture. We follow the configuration used in [18] for training the diffusion model, including architecture, optimizer, and noise schedule. Specifically, parameterize the diffusion process c as a UNet architecture with residual connections [61] trained with a cosine learning rate schedule (56). The UNet contains repeated residual blocks, each consisting of two temporal convolutions followed by group norm and a final Mish nonlinearity. The time step embedding is produced by single fully connected layer and added the activations of the first temporal convolution within each block.
Diffusion Model Training Syper-Parameters. The list of training papertpprameters for the trajec- tory diffusion model is provided in Table II
Description
Value
diffusion architecture denoising time steps
UNet
256
learning rate of Adam optimizer
0.0003
training epochs (passes over the data set) batch size
150
128
training steps per epoch guidance coefficient for i.e. guidance coefficient ratio for / i.e. A window size of sub-trajectories, i.c. w
5000 (D4RL), 2000 (Gym)
0.5 (D4RL), 0.1 (Gym)
0.5 (D4RL.). (Gym)
8 (D4RL), 16 (Gym)
Table I1: Mlyee-parametee for STITCH-OPE.
Reward Function. The reward function approximation R(s, a) two-layer MLP with ReLU activations and 32 neurons per hidden layer, and is trained using Adam with learning rate of 0.001 and batch size of 64.
Guidance Coefficients. For Gym domains, we use a = >= 1, corresponding to the theoretically justified guidance function in Equation g assuming low distribution shift. For D4RL tasks, tempered values a = 0.5 and À = 0.25 to improve sample stability and regularization, which we found empirically helpful in higher-dimensional settings.
Sub-Trajectory Length. We use w for Gym domains and = for most D4RL. tasks. For HalfChectah, we reduce to 4 duc to the environment's 's fast dynamics, which caused degradation
36

---

in stitching fidelity with longer sab-trrajccories, Also to add stability for cheetah, we set the clip denoised flag True during the backward diffusion process.
K
Diffusion Policy Training and Evaluation
Diffusion Policy Loss. We follow [1 and parameterize each target policy T/6 1...
... 10
conditional diffusion model co. (a k(s), whose parameters Φ are learned by optimizing the behavior cloning objective (compare with (T)
C(d+) Ek,, c~,(( sD a-1) [lle Ed, (a k|s)]l2].
Diffusion Policy Guidance Score. In order to use the fine-tuned Ea, (a kiss) as a guidance function for off aff-policy evaluation in STITCH-OPE, we use the following equivalence between score-base models and denoising diffusion [9] (extended trivially to the conditional setting)
Co. (a k(s)
log (((ass))aad
Specifically, this expression cannot be calculated at 0 since oo 0 using the standard param- eterization of diffusion models, so we approximate at and use the resulting gradient in STITCH-OPE.
Implementation and Parametass We implement the diffusion model using the Clean- Diffuser package [50] with official repository at https://githi:
CleanDiffuser (Apache 2.0 licensed). To train the diffusion policies, we first generate rollouts from each of the pre-trained target policies in D4RL [14] and then minimize the behavior cloning objective [();)) above obtain the diffusion policy parameters. The list of relevant hyper-parameters
is provided in Table 12
Description
Value
embedding dimension hidden layer dimension
64
256
learning rate
0.0003
diffusion time steps
32
EMA rate
0.9999
total training steps
10000
number of transitions to generate for each dataset
1000000
training batch size
256
Table 12: Hyper-parameters for training diffusion policies.
L Additional Experiments
L.1 Sensitivity to Guidance Coefficients a and
We evaluate STITCH-OPE across different choices of the guidance coefficients and and plot the resulting trends in Figure [ for Hopper and Figure 6 for Walker2D, Each plot is generated by applying bicubic interpolation the grid evaluations of the Spearman correlation and LogRMSE. The optimal coefficient values of a and X remain consistent across environments. The optimal balance for off-policy evaluation is attained by assigning moderate coefficient for the target policy score (i.e. & < 1) and smaller but positive coefficient to the behavior policy score X, i.e. A <<< a. Recall that À controls the amount of distribution shift we are willing accept during guided trajectory generation. X = 1 is theoretically unbiased, but potentially under-regularized and leads to dynamically infeasible (high-variance and leads to trajectories that are heavily biased towards the behavior policy p=(r). From the plots, we see that a moderate amount of regularization is optimal (around 25% of the value of o), which consistent with regularization in supervised machine learning (i.c., regression).
samples. Meanwhile, X = often over-regularized
37

---

Spearman Correlation (1)
Log RMSE (1)
-1.4
0.8
Correlation Strength
Error Magnitude
- -1.6
0.5
0.5
0.6
8
8
0.1
0.4
0.1
-2.0
2.2
0.01
0.2
0.01
0.25
0.5
0.75
0.25
0.5
0.75
A/a
A/a
Figure Smoothed performance landscape for Hopper. Left: Spearman correlation largest around E [0.1, 0.5). S 0.5a. Right: The LogRMSE is smallest around a E [0.01,0.3] X [0.25a, 0. 057).. These results confirm the optimal range of is < <aa.
Spearman Correlation (1)
Log RMSE (1)
0.6
Correlation Strength
-1.0
Error Magnitude
0.5
0.4
0.5
1.1
8
0.2
0.1
0.1
11.
0.0
-1.3
0.25
0.5
0.75
Figure 6: Smoothed performance landscape for Walker2d. Results are generally consistent with Spearman correlation is largest around E [0.1, 0.5], X 0.25a. Right: The LogRMSE is smallest around a E [0.1, 0.5), À zo 0.75a. These results confirm the optimal range of i 0 < A a.
Hopper. Left:
L.2 Sensitivity to Window Size
To further analyze the sensitivity to w, we evaluate STITCH-OPE across different intermediate values of w, and compare the performance according LogRMSE and Spearman correlation metrics. As illustrated in Figure f Hopper and Pendulum, the best performance is consistently achieved using moderate values of i.e. the main text, based on our analysis Section 3.3 and Section 3.4. low values w provide more flexibility when stitching trajectories and thus promote compositionality. but are more susceptible to the compounding of errors. High values of w are less susceptible to error compounding but at the expense of compositionality and thus less adaptability distribution shift. the current ablation experiment, it is clear that the best balance between compositionality and error compounding occurs using moderate values of w. and the greatest deterioration in performance occurs for very small or very large values. It is also important to note that increasing reduces inference speed due to longer trajectory generations per diffusion step, highlighting a practical trade-off between computational cost and evaluation accuracy.
for Hopper and 16 for Pendulum. As hypothesized in
L.3 Sensitivity to the Choice of Behavior Policy/Behavior Dataset
To examine the sensitivity STITCH-OPE to the choice of behavior policy or behavior dataset, we ran our algorithm and all baselines the medium-exper and expert data sets for the Hopper environment. Across all reported results, we set the hyper-parameters to = 768 and y = 0.99, which allows for easy comparison between the data sets. The results for the medium-expert dataset
38

---

Spearman Correlation 1
0.0
0.8
-0.5
0.6
Log RMSE
-1.0
0.4
-1.5
0.2
-2.0
0.0
128
128
Window Size w
Window Size w
Figure 7: Sensitivity of STITCH-OPE window size in the Hopper-v2 environment. Left: Spearman rank correlation. Right: Log RMSE. Error bars denote one standard error over five random seeds. The overall best performance attained for w 8, suggesting a good balance between compositionality and error compounding.
Spearman Correlation 1
1.00
0.75
Log RMSE
-1
0.50
0.25
-2
0.00
32
128
2
16
32
128
Window Size W
Window Size w
Figure 8: Sensitivity of STITCH-OPE to window size in the Pendulum-vl environment. Left: Spearman rank correlation. Right: Log RMSE. Error bars denote one standard error over five random seeds. The overall best performance is attained for = 16, suggesting good balance between compositionality and error compounding.
shown in Table 13 and the results for the expert dataset shown in Table STITCH-OPE soundly outperforms all other baselines in out of instances (metric-dataset combinations). Furthermore, while the performance of some baseline methods exhibits significant variation between data sets (i.e., FQE, DR, IS, PGD in Log RMSE), the performance of STITCH-OPE remains the most consistent across all metrics and datasets. Finally, we observe steep decline in performance of the baselines on the medium-expert and expert data sets as compared to the medium data set, which suggests that prior OPE work is not robust large distribution shifts between the behavior and target policies.
Metric
FQR
DR
IS
DRE
MB
PGD
Ours
Log RMSE |
-1.14 + 0.03
-0.71 = 0.05
-0.15 = <0.01
-0.18 + <0.01
-1.33 <0.01
-0.55 + 0.02
-2.17 + 0.01
Rank Corr.
0.22
0.14
0.83 <0.01
Regret@1 1 J
c0.01
0.06:0000
0.60
0.03
0.00 # 0.00
0.01 = <0.01
0,00 0.00
Table 13: Comparison of OPE methods for Hopper-medium-expers Error bars represent + one standard error across 5 seeds; any value shown as <0.01 is nonzero but rounds to zero at two decimals.
Metric
FQE
DR
IS
DRE
MB
PGD
Ours
Log RMSE
-0.39 <0.01
0.02 <0.01
-1.05
<0.01
0.02 <0.01
-1,46 <0.01
-0.01
<0.01
-2.33 0.01
Rank Corr.
0.46 0.03
0.20 0.11
0.26 0.1
0.20 = 0.11
0.28 2 <0.01
0.47 = 0.03
0.51 a 0.04
Regret@1
0.12
0.04
0.32 0.07
0.1
0.18<<0.0
0.00 0.00
+ 0.01
Table 14: Comparison of OPE methods Hopper-expert. Error bars represent one standard error
across 5 seeds; any value shown as <0.01 is nonzero but rounds to zero at two decimals.
39

---

L.4 Sensitivity to Reward Estimation
Recall that the immediate reward function STITCH-OPE is modeled by a neural network and trained the offline transitions (s, r, ,...), which can be described as standard (nonlinear) regression problem. We evaluate the sensitivity of STITCH-OPE to this modeling choice by running STITCH- CERECH-PPEEE given the true (unknown) reward function, and comparing this to the original results on the estimated reward function. Table [3 shows the aggregated metrics and the per-target-policce results the hopper-mediun benchmark dataset. In summary, the accumulated errors from reward estimation have a minimal effect on OPE performance as measured using aggregate metrics, and the error remains stable and low across all target policies.
Metric
True Reward
Estimated Reward
Log RMSE 4
-2.30 + 0.01
-2.28 0.01
Rank Corr.
0.77 0.01
0.75 + 0.01
Regret 1
0.10
0.13 = 0.03
Table 15: Aggregated metrics obtained by running STITCH-OPE given the true reward function and the estimated reward function on Hopper-medium. Error bars represent one standard error across seeds.
Policy
2
3
4
5
6
7
8
9
10
True Reward
137.52
229.49
218.71
233.40
268.40
255.46
258.95
239.99
260.88
268.18
Est. Reward
135.43
228.80
217.35
229.55
266.77
253.43
253.87
235.85
256.79
264.88
Rel. Error %
-1.52
-0.30
-0.62
-1.65
-0.61
-0.79
-1.96
-1.72
-1.57
-1.23
Table 16: Perllayget-policy values obtained by running STITCH-OPE given the true reward function and the estimated reward function on Hopper-medium.
L.5 Trajectory Visualizations
We visualize and compare trajectories generated by the guided and unguided versions of STITCH- OPE and Policy-Guided Diffusion (PGD) [16] against both random and optimal policies. These visualizations highlight differences in the quality of generated trajectories, alignment with target policies, and generalization capabilities across various environments. As shown in Figures 9 and STITCH-OPE closely mimics the target policy behavior. On the other hand, PGD performs poorly, significantly overestimating the performance of the random policy. Figure further demonstrates that STITCH-OPE maintains consistent and robust behavior across policy settings.
L.6 Comparison with Diffusion World Models
The Diffusion World Model (DWM) 1T] is an alternative approach that leverages diffusion models for trajectory generation. However, DWMs not well-suited for the off aff-policy evaluation problem, as we demonstrate experimentally in Table [7 In particular, DWM generates only future states and rewards, e.g., P(s P(5 Pirr Sp, ag g). discarding most action information. Consequently, it provides no mechanism for policy guidance and cannot adapt to the significant distribution shift present between the behavior and target policies.
By contrast, STITCH explicitly models the joint state-action trajectory p(r) = p(so- ag...., ST): we therefore effectively mitigate action distribution shift by incorporating policy guidance into trajectory generation (Eq. 3), which steers ao.r toward the target policy. In Markov settings, the DWM diffusion step for s, depends primarily on s;-1 and marginalizes over the behavior policy at - 1, rollouts remain near the behavior occupancy and are harder to steer, making DWM more brittle under action distribution shift.
More generally, while there have been significant advancements in diffusion modeling over the last few years, policy guided diffusion (PGD) [66] is the most representative and recent model-based baseline found the time of this publication which is relevant for OPE. This because addressing distribution mismatch between behavior and target policy trajectories is highly non-trivial problem,
40

---

which only STITCH-OPE (and partially PGD) are able to tackle effectively. For fuller discussion of diffusion-based baselines and other model-basee OPE methods, we refer the reader Section N
Metric
DWM
PGD
Ours
Log RMSE L
-0.82 # 0.20
-1.22 = 0.02
-2.33 = 0.02
Rank Corr. 1
0.41 + 0.10
0.36 0.09
0.76 0.02
Regret@1 1
0.14
0.04 # 0.01
0.11 # 0.04
Table 17: Comparison diffusion modeling approaches on the Hopper-medium benchmark. Error bars represent + one standard error across 5 seeds.
M Computing Resources
Hardware and Software. All experiments were conducted on a local workstation running Ubuntu 20.04 LTS and Python 3.9, with the following hardware:
2x NVIDIA RTX 3090 GPUs (24 GB each)
Unguided
PCD
BITTCH Guiddd
NGD Eacce!
Oytinal
Gudddd
Figure 9: Trajectory visualizations in the Hopper environment. Both STTTCH-OPE and PGD track the optimal policy. PGD significantly overestimates the performance of the random policy, while STITCH-OPE correctly models both the state trajectory and the termination.
41

---

Paley
Ungaided
Guided
RGD Gurod
Oprinal Polcc
BTTTCH Guided
GUobd
Figure 10: Trajectory visualizations in the HalfCheetah environment. STITCH-OPE and PGD both demonstrate consistent behavior across all policy types, highlighting their robust generalization on this task.
Intel(R) Core(TM) 19-9820X CPU @ 3.30GHz (10 cores / 20 threads)
128 GB RAM.
Runtime, Each full training of diffusion model for a D4RL task took approximately 20 hours to complete, depending environment complexity and rollout length. Each OpenAI Gym task took approximately 5 hours. Each evaluation for a D4RL environment took around 18 hours in total (across all 5 seeds) to complete, and each OpenAl Gym environment took around 6 hours to complete.
N Related Work
Off-policy evaluation plays critical role in offline reinforcement learning, enabling the evaluation of policies without directly interacting with the environment. OPE has been studied across wide range of different domains including robotics (52), healthcare (58 60 59 and recommender systems 51 63). Relevant work includes model-free and model-based OPE approaches, including recent generative methods in offline RI.
Model-Free Methods. Model-free methods, such as Importance Sampling (IS) and per-decision Importance Sampling (PDIS) [33] reweight trajectories (or single-step transitions) from the behavior
42

---

Palky
Ungaided
Random
RGD Grodd
Oprinal PoRdy
STITCH Gudld
PGD GUODd
Figure 11: Trajectory visualizations in the Walker2d environment. STITCH-OPE effectively imitates both random and optimal policies. As for the Hopper environment, PGD struggles to correctly imitate the random policy, significantly overestimating its performance.
policy approximate returns under a target policy. However, this class of methods suffers from the so-called "curse of horizon", in which the variance grows exponentially in the length of the trajectory
[19 37 [1] further combine estimation of value functions methods (DICE) and their variants [25] 31 try mitigate the curse-of- of-horio-no by performing importance sampling from the stationary distribution of the underlying MDP. However, these methods perform relatively poorly on high-dimensional long-horizon tasks [14]
25 [27 Doubly Robust (DR) methods
with importance weights, reducing the overall variance. Distribution-correete
32 43 47
Model-Based Methods. Model-based OPE methods estimate the target policy value by learning approximate transition and reward models from offline data and simulating trajectories under the target policy [19 21]. These methods have shown strong empirical performance, especially in continuous control domains [38 46), but they often suffer from compounding errors during rollouts,
which can lead to biased estimates in nigh-dimensional or long-horizon settings 20
Offline Diffusion. Inspired by the recent performance of diffusion models across many areas of machine learning [15] 91 a new stream reinforcement learning has emerged which leverages diffusion models trained on behavior data (2 48 Janner et al. [18] Ajay et al. [1 train diffusion
models on behavior data that can be guided to achieve new goals. Ding al. [10 uses diffusion
models world models. While successful at modeling the behavior distribution over trajectories
43

---

accurately, diffusion world models do not use guidance and thus cannot address the policy distribution shift. Jackson al. [T6], Rigter et al. [35 apply guided diffusion offline policy optimization by setting the guidance function to be the score of the learned policy, while (44] applies guided diffusion to satisfy added safety constraints. Unlike STITCH-OPE these works do not use negative guidance nor stitching, which we found leads to unstable policy values when applied directly for offline policy evaluation over a long-horizon. Mao et al. [30] applies DICE to estimate the stationary distribution of the underlying MDP, which is used as guidance function to correct the policy distribution shift for offline policy optimization. Unlike STITCH-OPE, this work is not directly applicable to offline policy evaluation. Finally, Li et al. [54] introduces variant of trajectory stitching for augmenting behavior data, but does not apply it for offiin policy evaluation. To the best of our knowledge, STITCH-OPE is the first work to apply diffusion models evaluate policies offline data.
References
[49] C. M. Bishop and N. M. Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[50] Z. Dong. Y. Yuan, Hao, F. Ni, Ma, P. Li, and Y. Zheng. Cleandiffuser: An easy-to-use modularized library for diffusion models in decision making. arXiv preprint arXiv:2406.095099 2024. URL wttp::ttt////rreeeeee
org/ ab46 09509.
[51] M. Dudik, D. Erhan, J. Langford, and I.. Li. Doubly robust policy evaluation and optimization. Statistical Science, 29(4):485-511 2014.
[52] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, and S. Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pages 651-673 PMLR, 2018.
[53] I. Kostrikov and O. Nachum. Statistical bootstrapping for uncertainty estimation in off-policy evaluation. arXiv preprint arXiv:2007.13609 2020.
[54] G. Li, Y. Shan, Z. Zhu, T. Long, and W. Zhang. Diffstitch: Boosting offline reinforcement learning with diffusion- trajectory stitching. In International Conference on Machine Learning, pages 28597-28609. PMLR, 2024.
[55] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Con- ference on Learning Representations, 2019. URL https: wtt.://wwwee/oooommiii. Bkg6RicqY7
[56] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations, 2022.
[57] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 51 518(7540):529-533, 2015.
[58] S. A. Murphy, M. J. van der Laan, J. M. Robins, and C. P.P.R. Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96(456):1410-147) 2001. ISSN 01621459. URL httt: //www, jstor. org/ htt///tabbee/305/00///
[59] X. Nie, E. Brunskill, and S. Wager. Learning then-to-treat policies. Journal of the American Statistical Association, 1 116(533)39224400) 409, 2021.
[60] A. Raghu, O. Gottesman, Y. Liu, M. Komorowski, A. Faisal, F. Doshi- and E. Brunskill. Behaviour policy estimation in off off-pl policy evaluation: Calibration matters. International Conference Machine Learning: workshop on Causal Machine Learning, 2018
[61] O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medicul image computing and computer-assisted intervention-MICCAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 234-241. Springer, 2015.
44

---

[62] C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-101, 1904.
[63] G. Theocharous, P. S. Thomas, and M. Ghavamzadeh. Personalized ad recommendation systems for life-time value optimization with guarantees. In International Joint Conference on Artificial Intelligence, 2015. URL hetLtttp::///e
semanticscholar org/ www.oorggg.Doaoao..oo : 80815223
[64] Y. Zheng. Li, D. Yu, Yang, S. E. Li, X. Zhan, and Liu. Safe offline reinforcement learning with feasibility- guideed diffusion model. In The Twelfth International Conference on Learning Representations, 2024. URL http/t:tt///
45


Reviews:
[{"review_id": "dQOzr3xuw2", "reviewer": "Reviewer_YRRV", "summary": "This paper presents STICH-OPE, a novel diffusion-based off-policy evaluation method for reinforcement learning. The algorithm trains a conditional diffusion model to generate short trajectories of the behavior policy. Using the flexible guidance in diffusion models, it is able to sample trajectories of the target policy nicely. The authors show that STICH-OPE performs better than various baselines through extensive experiments, and also provide a derivation of theoretical guarantees of their method.", "questions": "1. Clarification for guidance during the denosing process. Is the guidance applied at every step during the denoising process? The initial state action pairs are quite noisy; would they cause problems for the target and behavior policy score evaluations?\n2. When target policies are diffusion policies, you mentioned that \n    \n    > Specifically, this expression cannot be calculated at $k = 0$ since $\\sigma_0=0$ using the standard parameterization of diffusion models, so we approximate it at $k = 1$ and use the resulting gradient in STITCH-OPE.\n    > \n    \n    Could you elaborate on what this means? Is the same gradient used throughout the whole denoising process of the STICH algorithm? Or is it just for this last denoising step, you use an approximated gradient, and all the other steps utilize the standard calculations of the score? Additionally, do these diffusion policies generate one-step actions or short action trajectories?\n3. Does the trajectory dataset need to contain rollouts that are similar to the behavior policy and target policies? I’m asking this because training generative models like diffusion models is much more data-hungry than other models (like a neural dynamics model). In scenarios that require OPE (without access to simulators or when gathering new data is costly), if the training of the proposed methods requires rollouts similar to the behavior or target policy, then there’s a question of why one would still do OPE when they could already generate rollout data.\n4. How sensitive is the algorithm to the selection of the behavior policy? In other words, do the return values agree when choosing different behavior policies to do the OPE process?\n5. For the visualization, Figures 9 and 11. What is the optimal policy? Does it refer to the target policy being evaluated? For the OPE runs, such as STICH or PGD, are the visualizations rendered with the states’ trajectories or the action trajectories (rollout in sim)?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. Provided a rigorous theoretical analysis of the proposal algorithm.\n2. Evaluated the proposed method on various benchmarks.\n3. Overall, the paper is clear and easy to follow.\n\nWeaknesses:\n\n1. Most evaluations are on simple RL environments; it remains to see how the method performs in more complex and realistic RL tasks.", "was_revised": true, "final_justification": "My final recommendation for this paper is 5 (Accept). I think it is a solid paper with good theoretical backups and good experiments to show the effectiveness of their methods. The rebuttal provided by the authors answered most of my questions, including several clarifications, and provided new insights on the applicability of their method on various target policies, and the target policy doesn't have to be in distribution with the training data. The missing unanswered question is the real-life application of their method. I am not too familiar with the off-policy evaluation applications except in RL algorithms. However, just from the metrics the authors provided, their STICH-OPE is very competitive against other baselines.", "rebuttal": "We appreciate that the reviewer found our paper clear and easy to follow, our theoretical analysis rigorous and insightful, and our empirical evaluation thorough.\n\n---\n\n## (Q1) Is the guidance applied at every step during the denoising process?\n\nYes, you are correct in that we apply guidance at every step of the denoising process. \n\nTo clarify this, we first want to point out an interesting feature of guided diffusion not discussed in the literature. Even with a constant guidance coefficient, the actual strength of the guidance across denoising steps is varied by the denoising schedule, i.e. $\\alpha_k$. In Appendix A, a concrete illustration with the standard parameterization $\\sigma_k^2 = \\beta_k = 1 - \\alpha_k$ shows the guidance strength is $\\sqrt{\\alpha_k}$. \n\n**Theoretical Justification:** Although we agree that the guidance strength is larger at the initial denoising steps, we also want to clarify that this choice is necessary according to theory of classifier diffusion guidance [1]. It also parallels the standard implementation of Langevin dynamics where the ``learning rate'' is annealed over time to achieve (asymptotic) theoretical convergence guarantees. \n\n**Practical Justification:** Annealing the guidance is also intuitive, since we want to take larger steps to speed up convergence when the sample is far from its true distribution at the early denoising steps. Meanwhile, during later denoising steps, it makes practical sense to take smaller steps to ensure convergence and avoid escaping out of the high-density region. Our empirical results did not require additional tricks to improve convergence (beyond guidance normalization), and worked well across diverse target policies and policy classes.\n\n---\n\n## (Q2) Clarification of Diffusion Policy experiments\n\nWhen we refer to approximating the score at $k=1$, we refer to the calculation of the target policy's score function when the target policy is a diffusion policy. Also, our diffusion policies output immediate/single-step actions.\n\n**Details about $k = 1$:** Naturally, when the target policy is parameterized as a diffusion policy, the closed-form expression of its score function is no longer available, so we instead use the known fact that the score at denoising step $k$ is $\\epsilon(a | s, k)/\\sigma_k$. Since we learn $\\epsilon( a|s,k)$ to match the score of the convolution of $\\pi(a|s)$ with $\\mathcal{N}(0, \\sigma_k)$ we would ideally estimate the score function at the last denoising step $k = 0$. But this is practically impossible since $\\sigma_0 = 0$ and so we use the estimate at $k = 1$. Most notably, these calculations are described in terms of the denoising process of the target policy, and not the guided denoising process of STITCH (for sampling trajectories) which is a separate process. Thus, once we derive the approximate score function of the diffusion policy, we use this score function to guide across all denoising steps within STITCH. \n\n**$k = 1$ as a Policy Approximation:** As we show in Table 3, STITCH-OPE still provides competitive OPE performance when the target policy score function must be approximated as discussed above, which introduces some measurement errors into the guidance function (which is often the case in many real-world settings). It also outperforms baselines (such as MBR or DRE) that use the target policy only for sampling actions during OPE, which are technically unbiased samples from the exact target policy. \n\n**Output of the Diffusion Policy:** All diffusion policies we use output single-step (i.e. immediate) actions. This follows exactly the offline diffusion policy literature [2], where single-step actions were also used in their practical implementation.\n\n---\n\n## (Q3) Does the trajectory dataset need to contain rollouts that are similar to the behavior policy and target policies? I'm asking this because training generative models like diffusion models is much more data-hungry than other models.\n\nWe would like to thank the reviewer for raising this important point. We want to clarify that STITCH-OPE does **not** require that rollouts be similar between behavior and target policies. This is due to the following reasons:\n\n**Chunked Diffusion Training:** One of the key technical novelties of STITCH-OPE is training on short overlapping windows sampled from the behavior data. By reducing the prediction horizon, we cut down on the diffusion model's training complexity and data requirements compared to training on full rollouts. More importantly, by observing the same state-action dynamics across different rollouts, they can be reused at different epochs, across multiple trajectories, and across target policies. This means that the behavior data set does not need to contain rollouts that are similar to the target policies, as long as the support of the target policy's immediate action distribution is within the support of the behavior policy -- a standard assumption in OPE literature.\n\n**Compositional Inference:** At inference time, STITCH-OPE stitches together generated trajectory chunks to reconstruct full-horizon rollouts. Together with policy guidance, this process generates novel rollouts that were never observed in the behavior data. Crucially, these rollouts remain faithful to the target policy even when the target policy's action distribution is significantly different from the behavior policy, as long as the target policy stays within the support of the behavior policy. Please see row A of Table 1 for a visual example and Appendix C for a formal proof that the entropy of stitched trajectories is higher than the behavior policy. \n\n**Empirical performance:** We validate these contributions on D4RL and demonstrate that STITCH can reliably predict target policy returns even when the target policy differs from the behavior policy (low MSE, high correlation and low regret aggregated across all target policies in Table 2). We also want to clarify that the construction of the target policies in our paper of varying ability from random to expert, as well as the inclusion of target diffusion policies in our work, makes for a considerably more challenging evaluation benchmark than prior OPE work.\n\n---\n\n## (Q4) How sensitive is the algorithm to the selection of the behavior policy?\n\nWe found that STITCH-OPE is robust to the selection of the behavior policy. To check this, we ran STITCH-OPE and baselines on the medium-expert and expert data sets from the Hopper D4RL benchmark using default parameters ($T = 768, \\gamma = 0.99$). The metrics are shown below:\n\nMedium-Expert\n| Metric | FQE | DR | IS | DRE | MB | PGD | Ours (STITCH) |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Log RMSE ↓ | -1.140 ± 0.027 | -0.709 ± 0.051 | -0.151 ± 0.002 | -0.182 ± 0.001 | -1.334 ± 0.008 | -0.554 ± 0.016 | **-2.174 ± 0.012** |\n| Rank Corr. ↑ | 0.511 ± 0.086| 0.224 ± 0.146 | 0.105 ± 0.062 | 0.132 ± 0.144 | 0.753 ± 0.011 | 0.774 ± 0.044 | **0.826 ± 0.003** |\n| Regret@1 ↓ | 0.023 ± 0.001| 0.064 ± 0.041 | 0.599 ± 0.150 | 0.131 ± 0.045 | 0.000 ± 0.000 | 0.009 ± 0.008  | **0.000 ± 0.000** |\n\nExpert \n| Metric | FQE | DR | IS | DRE | MB | PGD | Ours (STITCH) |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Log RMSE ↓ | -0.391 ± 0.003 | 0.016 ± 0.001 | -1.054 ± 0.006 | 0.016 ± 0.001 | -1.455 ± 0.008| -0.010 ± 0.001 | **-2.328 ± 0.012** |\n| Rank Corr. ↑ | 0.46 ± 0.027 | 0.202 ± 0.113 | 0.261 ± 0.099 | 0.202 ± 0.113 | 0.277 ± 0.004  | 0.473 ± 0.029 | **0.507 ± 0.039** |\n| Regret@1 ↓ | 0.115 ± 0.042 | 0.095 ± 0.037 | 0.321 ± 0.067 | 0.095 ± 0.037 | 0.175 ± 0.001 | **0.000 ± 0.000** | 0.035 ± 0.010 |\n\nSTITCH-OPE outperforms all other baselines across all three metrics using both medium-expert and expert data sets, and performance remains stable across both data sets. This is surprising, given that we would expect the distribution shift for the expert data set to be greater than the medium data set (which contains more diverse rollouts) or the medium-expert data set (which is a mixture of the prior two). This suggests that STITCH-OPE remains effective at composing trajectories across a variety of behavior data sets generated by different behavior policies. Finally, we observe a steep decline in performance of the baselines on the medium-expert and expert data sets as compared to the medium data set (Table 2 in the paper), which suggests that prior OPE work is not robust to large distribution shifts between the behavior and target policies.\n\n---\n\n## (Q5) For the visualization, Figures 9 and 11, what is the optimal policy?\n\nThe random and optimal policies correspond to the worst and best target policies within the 10 target policies evaluated. We selected the best and worst target policies to demonstrate that STITCH-OPE can generate accurate synthetic trajectories even when guided by target policies that are significantly different from each other and from the behavior policy. The trajectories labeled as ``STITCH Guided`` or ``PGD Guided`` are purely synthetic trajectories generated from the trained diffusion model with policy guidance, i.e. there is no simulator involved.\n\n---\n\n1. Dhariwal, Prafulla et al. \"Diffusion models beat gans on image synthesis.\" NeurIPS, 2021.\n2. Wang, Zhendong, et al. \"Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning.\" ICLR, 2023."}, {"review_id": "iHRuK6MkWp", "reviewer": "Reviewer_8yJQ", "summary": "STITCH-OPE introduces a model-based off-policy evaluation method for high-dimensional, long-horizon RL tasks. It leverages guided diffusion models pretrained on behavior data to generate target policy trajectories by \"stitching\" short sub-trajectories end-to-end. It subtracts the behavior policy’s score during guidance to mitigate distribution shift and over-regularization, and splits generating the whole long trajectories into stitching a series of short sub-trajectories. The paper provides both theoretical and experimental analysis to support the effectiveness of STITCH-OPE.", "questions": "1. Could you detail the specific diffusion sampling algorithm (e.g., DDPM, DDIM) and hyperparameters (e.g., diffusion steps, guidance scale, noise schedule) used for generating sub-trajectories? The paper mentions \"conditional diffusion\" but lacks implementation specifics critical for reproducibility.\n\n1. The choice of $T=768$ for D4RL (vs. the standard episode length 1000) appears arbitrary. What empirical or theoretical justification supports this?\n\n1. The paper follows the protocol described in DOPE [1] to train target policies, using SAC on D4RL medium datasets. As offline RL has advanced significantly since 2021, could STITCH-OPE's effectiveness be demonstrated using higher-performing target policies? For instance, evaluating policies trained with modern offline algorithms (e.g., CQL, IQL, TD3+BC) on \"medium-replay\" or \"medium-expert\" datasets would better assess performance on near-expert policies.\n\n1. The model-based (MB) baseline appears outdated. Given recent progress, could you include a stronger diffusion model baseline like Diffusion World Model (DWM) [2], which also performs trajectory-level modeling suitable for OPE? Similarly, please confirm if other baselines (e.g., PGD, DRE, DR) represent the most current competitive methods or could be updated, as current baselines are generally before 2021. Related work in Section N should also be updated.\n\n[1] Fu, Justin, et al. \"Benchmarks for deep off-policy evaluation.\" ICLR 2021\n\n[2] Ding, Zihan, et al. \"Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning.\" preprint 2024.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow.\n- The theoretical analysis provides an exponential reduction in MSE, which allows further improvement following its route.\n- It emperically ourperforms all baselines on commonly-used D4RL benchmarks and various target policies, including complex diffusion policies.\n\nWeaknesses: see questions.", "was_revised": true, "final_justification": "The rebuttal has solved most of my concerns. This paper designs a novel algorithm to the OPE problem, and the results are both significant and solid.", "rebuttal": "We thank the reviewer for the detailed comments. We appreciate that the reviewer found value in our theoretical analysis, which indeed shows an exponential reduction in variance versus existing methods, and our thorough empirical evaluation and competitive OPE performance across all tasks and metrics.\n\n---\n\n## (Q1) Could you detail the specific diffusion sampling algorithm (e.g. DDPM, DDIM) and hyperparameters?\n\nOur implementation of the diffusion model builds on the PyTorch trajectory diffuser implementation by Janner et al. [7], which is a DDPM whose hyperparameters are described in Appendix C of that paper. We are thoroughly committed to reproducible research, and as such have provided key details regarding the diffusion model implementation in Appendix J in the supplementary material (and configuration files with detailed hyperparameters in our attached code base). We will describe all hyperparameters from [7] that we used that were omitted from Table 10 of our appendix in our revised version.\n\n--- \n\n## (Q2) The choice of $T = 768$ for D4RL (vs. the standard episode length 1000) appears arbitrary. What empirical or theoretical justification supports this?\n\nThank you for highlighting an important implementation detail of our work. In summary, the main reason for choosing $T = 768$ as the evaluation horizon for the generative baselines was that time steps beyond this contributed little variance to the cumulative return. We also ran an ablation over $T$ to demonstrate that STITCH-OPE works across different values of $T$.\n\n**Ablation Experiment over $T$:** Since the goal of our paper is to long-horizon correctness of OPE, we conducted an additional ablation over multiple values of $T \\in \\{128, 256, 512, 768, 1000\\}$ on Hopper-medium plus all baselines. We also set the discount factor to $\\gamma = 0.999$, to better see the performance differences across methods for $T = 1000$. \n\nNormalized Log RMSE\n| Method | T=128 | T=256 | T=512 | T=768 | T=1000 |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| STITCH | **-4.342 ± 0.008** | **-3.192 ± 0.003** | **-2.653 ± 0.008** | **-1.985 ± 0.010** | **-1.613 ± 0.019** |\n| MBR | -3.661 ± 0.006 | -2.977 ± 0.006 | -2.165 ± 0.009 | -1.561 ± 0.010 | -1.236 ± 0.011 |\n| IS | -3.819 ± 0.010 | -2.042 ± 0.009 | -1.194 ± 0.007 | -0.944 ± 0.003 | -0.810 ± 0.003 |\n| PGD | -3.652 ± 0.015 | -3.026 ± 0.013 | -2.301 ± 0.007 | -1.816 ± 0.006 | -1.452 ± 0.003 |\n| FQE | -2.228 ± 0.001 | -1.364 ± 0.001 | -0.991 ± 0.001 | -0.811 ± 0.002 | -0.691 ± 0.002 |\n| DR | -3.489 ± 0.024 | -2.454 ± 0.002 | -1.649 ± 0.024 | -1.317 ± 0.012 | -1.140 ± 0.018 |\n\nIn summary: \n1. STITCH-OPE performance degrades very slowly as $T$ increases, validating the theoretical bound of Theorem 3.3 \n2. MBR and IS methods degrade rather quickly as $T$ increases, whereas the performance of STITCH-OPE remains more stable across $T$, and \n3. STITCH-OPE outperforms all other baselines according to MSE across all $T$, striking the optimal balance between fully auto-regressive and trajectory diffusion methods. \n\nWe will include results for all metrics (rank correlation and Regret@1) for this ablation in our revised version.\n\n**Practical Justification for $T = 768$:** We agree that our choice of $T=768$ could benefit from more explicit justification.\n\n1. An important distinction is that the truncation to $T=768$ applies **specifically to generative methods** such as STITCH-OPE and PGD, and not truncation of the data set. In contrast, **non-generative baselines** (e.g., FQE, DRE, DR, IS) use the full offline dataset without truncation, processing entire episodes up to 1000 steps -- our implementation reflects this to ensure fair comparison.\n\n2. The primary rationale for $T=768$ in STITCH-OPE is tied to the discount factor $\\gamma=0.99$, which we borrow from standard work in Offline RL and D4RL benchmarks [2]. With $\\gamma=0.99$, rewards collected towards the end of the rollout contribute negligibly to the discounted return on these problems. Truncation to $T=768$ thus captures nearly all meaningful return variance across all three D4RL benchmark problems. \n\n3. Additionally, 768 was selected as a convenient multiple for implementation efficiency, since it aligns with common batch sizes and diffusion step multiples in our codebase.\n\n---\n\n## (Q3) Could STITCH-OPE's effectiveness be demonstrated using higher-performing target policies?\n\nWe would like to clarify a misunderstanding. Our target policies are not the SAC checkpoints from the D4RL medium datasets; they are trained online to span the full spectrum from near-random to near-optimal. Concretely, our best performing policies achieve the following raw returns, which are in par with the strongest results achieved by recent offline methods on the same tasks (medium-expert dataset). \n\n| Task     | Ours | BCQ | CQL | TD3-BC | IQL |\n|-|-|-|-|-|-|\n| HalfCheetah  | 12696 | 10875  | 9358  | 11154   | 10773  |\n| Hopper  | 2953 | 2566 | 2027  | 2900   | 3564  |\n| Walker-2d  | 4121 | 3426 | 3659  | 3806   | 3537  |\n\nResults are taken from the Robomimic documentation page [8]. \n\nAn added benefit of training online is that these policies are produced with fresh environment interaction (not the behaviour dataset), thus they introduce a larger shift than re‑using policies trained on the same data, making the OPE problem harder and more realistic. In summary, our current protocol already evaluates STITCH‑OPE on high‑return, high‑shift targets while remaining fully compatible with the reviewer’s request for stronger offline policies.\n\n---\n\n## (Q4) The model-based (MB) baseline appears outdated.\n\nWe thank the reviewer for suggesting Diffusion World Models (DWM) [1] as an alternative diffusion model-based approach. We have implemented DWM in our setup and included its results in the hopper-medium benchmark below, where it is clear that STITCH-OPE (and PGD) outperform DWM.\n\n| Metric | DWM | PGD | STITCH-OPE |\n| :--- | :--- | :--- | :--- |\n| Log RMSE ↓ | -0.82 ± 0.20 | -1.22 ± 0.02 | **-2.33 ± 0.02** |\n| Rank Correlation ↑ | 0.41 ± 0.10 | 0.36 ± 0.09 | **0.76 ± 0.02** |\n| Regret @ 1 ↓ | 0.14 ± 0.11 | **0.04 ± 0.01** | 0.11 ± 0.04 |\n\nWe observe that the log MSE values of DWM are worse than PGD [2], and significantly worse than STITCH-OPE which is still by far the best-performing method. This is intuitive, since DWM does not use policy guidance, and thus cannot adapt to the significant distribution shift present between the behavior and target policies. \n\nMore generally, we wish to point out that while there have been significant advancements in diffusion modeling over the last few years, we believe PGD [2] is the most representative and recent model-based baseline we found to date that is **relevant for OPE**. This is because addressing distribution mismatch between behavior and target policy trajectories is a highly non-trivial problem, which only STITCH-OPE (and partially PGD) are able to tackle effectively by using positive and negative policy guidance. \n\nWe will include the Diffusion World Models baseline in our evaluation (with accompanying details) and our related work section accordingly. We will also clarify why some recent diffusion work is not suited for OPE.\n\n---\n\n## (Q4) Please confirm if other baselines (e.g., PGD, DRE, DR) represent the most current competitive methods.\n\nWe thank the reviewer for their concern over fair baselines and believe rigorous evaluation is crucial for advancing OPE. We believe our choice of baselines is up to date.\n\n**Single-Step Baselines:** Within the class of single-step dynamics modeling, the state-of-the-art in OPE still fit a single neural network to predict the next state given the current state and action, and then do Q-learning with the model rollouts [4,5,6]. While models that are one-step and auto-regressive in predicting individual state components are an alternative, Fu et al. [3] shows that using a single MLP to predict the entire next state performs better.\n\n**Trajectory Generation:** We believe that trajectory modeling is considerably more challenging in OPE, as it is largely unclear how to address the distribution shift between the behavior and the target policy. While many general diffusion modeling papers cited in our text could in principle model dynamics over multiple time steps, it is largely unclear how they can be adapted to address the distribution shift. PGD (published in 2024) [2] was the most recent paper and most representative member within this class of methods that is relevant to OPE, since it naturally corrects for the policy distribution shift using target policy guidance.\n\n**DRE, DR and Other Standard Baselines:** The other baselines in our evaluation are considered standard in the OPE literature, as evidenced by the most recent surveys on OPE [4, 5, 6]. They are also considered representative baselines across a variety of diverse classes of OPE methods, i.e. visitation densities, Q-values, doubly-robust, etc. For these reasons, we felt that it would be unfair to exclude these standard baselines, despite that some of them were introduced less recently.\n\n---\n1. Ding, Zihan, et al. \"Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning.\" arXiv 2024.\n2. Jackson, Matthew T., et al. \"Policy-Guided Diffusion.\" RLC 2024.\n3. Fu, Justin, et al. \"Benchmarks for deep off-policy evaluation.\" ICLR 2021.\n4. Kiyohara, Haruka, et al. \"Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation.\" ICLR 2024.\n5.  Uehara, Masatoshi et al. \"A Review of Off-Policy Evaluation in Reinforcement Learning.\" arXiv 2022.\n6. R. Figueiredo Prudencio et al., \"A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems,\" IEEE TNNLS, 2024.\n7. Janner, Michael, et al. \"Planning with diffusion for flexible behavior synthesis.\" arXiv 2022.\n8. Mandlekar, Ajay, et al. \"What Matters in Learning from Offline Human Demonstrations for Robot Manipulation\", CoRL 2021."}, {"review_id": "azTBswibhC", "reviewer": "Reviewer_jw9S", "summary": "This paper proposes to address the off-policy evaluation (OPE) problem using STITCH-OPE, which leverages the denoising diffusion for long-horizon OPE in high-dimensional spaces. The authors claim two technical contributions: (1) the paper proposes a new guidance score function which can avoid over-regularization and (2) generates long trajectories by stitching partial trajectories together. The authors also provide experimental results on the standard D4RL and OpenAI Gym benchmarks and show advantages compared with existing methods on some metrics (mean squared error, correlation, and regret).", "questions": "The authors stress \"long-horizon\" settings, while I understand they are more challenging than \"short-horizon\" cases, but I'm trying to understand how long are the trajectories in the experiments. The authors mention \"T = 768 for all D4RL problems\", does that mean 768 seconds for trajectory (i.e., frequency is 1HZ), or the frequency is not 1HZ?", "limitations": "See weaknesses and questions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. First, the paper studies an important problem, off-policy evaluation, which is critical in domains where deploying a new policy directly might be risky and better understanding on its impact beforehand using behavior data and policy is necessary.\n\n2. The proposed negative behavior guidance is novel and integrating both the target and behavior policy guidance. The theoretical evaluation error upper bound result is encouraging by removing the exponential term in the previous approaches.\n\n3. The experimental design is reasonable with convincing benchmarks and baselines. The empirical results validate the authors' hypothesis. \n\nWeaknesses:\n\n1. I'm not fully convinced by a claimed innovation/contribution, i.e., the \"sub-trajectory stitching using a sliding window\". If my understanding is correct, the authors are using a sliced history trajectory as the diffusion input and output a sliced window future trajectory. I think this is also proposed in literature, e.g., the diffusion policy paper [7]. How is the method in this paper different existing method? More clarifications would be appreciated.\n\n2. Does the authors try different time horizon under the same benchmark? For example, in a D4RL setup, try T from 100, 200, to 700, and (I believe some readers would be curious to) see how the proposed approach work as the horizon becomes longer, compared with baselines. This can also be helpful to better validate and understand the theoretical error upper bound.", "was_revised": true, "final_justification": "The paper studies an important problem and proposes an effective approach -- the rebuttal helps address my previous concerns.", "rebuttal": "We thank the reviewer for your insightful comments. We also agree that the problem of OPE is an important problem that has also not been fully addressed in the context of high-dimensional, long-horizon prediction. We thank the reviewer for acknowledging that our use of negative behavior guidance to address over-regularization and our OPE-specific theoretical bounds were not studied in prior diffusion work, and appreciate the feedback that our empirical results were convincing. Please find our answers to your questions below.\n\n---\n\n## (Weakness 1) I'm not fully convinced by a claimed innovation/contribution, i.e., the \"sub-trajectory stitching using a sliding window\".\n\nThank you for pointing out the apparent similarity between STITCH-OPE and the receding-horizon approach in Diffusion Policy (cited as [7] in our paper). There are a number of key distinctions between STITCH and Diffusion Policies, most notably: \n1. the difference in meaning of sliding windows between the two methods (action vs. state + action prediction), \n2. the difference between training and inference procedures of the two methods, and\n3. OPE-specific innovations in STITCH -- such as theoretical analysis showing exponential reduction in return variance when using sliding windows -- that are not discussed in the diffusion policy paper.  \n\n**Distinct Objectives:** Diffusion Policy and variants apply a sliding window to predict actions for online control, thus all windows are derived from a single demonstration or from the same trajectory. In contrast, the goal of STITCH-OPE is to produce accurate synthetic trajectories (states and actions) for off-policy evaluation that remain accurate over long horizon and in the presence of distribution shift. This is achieved by drawing short segments from different parts of the offline data set, allowing the diffusion model to optimally compose/stitch them end-to-end dependent on the target policy guidance. \n\n**Distinct Training/Inference Procedures:** Diffusion Policy is designed for imitation learning by training on complete demonstration trajectories, slicing them only at inference time. Thus, there is no mechanism to recombine slices from different real trajectories. In contrast, STITCH-OPE trains the diffusion model on overlapping windows (sampled randomly) within demonstration trajectories. This allows us to build an explicit model of the length $w$ transition dynamics that can then generalize well across all possible starting states (see row B of Table 1) and even target policies (see row A of Table 1). \n\n**OPE Specific Innovations:** The unique training and inference setting in STITCH-OPE enables: \n1. bridging states across distinct offline trajectories, \n2. applying positive guidance to correct distribution shift across policies and even policy classes, \n3. applying negative guidance to prevent over-regularization, and\n4. provide a theoretical framework for analysing OPE perforamnce (under weak assumptions) that guarantees **exponential reductions in variance of the return estimate** as compared to other trajectory diffusion models. \n\nThe combination of these innovations, none of which have been discussed by Diffusion Policy variants nor other prior work on trajectory diffusion, are critical to ensure reliable OPE performance across many long-horizon tasks (Figure 2, Table 2).\n\n---\n\n## (Weakness 2) Does the authors try different time horizon under the same benchmark?\n\nThank you for raising this point. To address this, we ran an additional experiment to show that STITCH-OPE remains effective over a large range of evaluation horizons.\n\nSpecifically, we evaluated STITCH-OPE against all baselines on the Hopper-medium data set for various values of $T \\in \\lbrace 128, 256, 512, 768, 1000 \\rbrace.$ All values have been normalized by the global minimum and maximum achieved via the aggregation of value estimates across all the horizons. We used a discount factor of $\\gamma = 0.999$ to show the difference in performance between $T = 768$ and $T = 1000$:\n\nNormalized Log RMSE\n| Method | T=128 | T=256 | T=512 | T=768 | T=1000 |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| STITCH | **-4.342 ± 0.008** | **-3.192 ± 0.003** | **-2.653 ± 0.008** | **-1.985 ± 0.010** | **-1.613 ± 0.019** |\n| MBR | -3.661 ± 0.006 | -2.977 ± 0.006 | -2.165 ± 0.009 | -1.561 ± 0.010 | -1.236 ± 0.011 |\n| IS | -3.819 ± 0.010 | -2.042 ± 0.009 | -1.194 ± 0.007 | -0.944 ± 0.003 | -0.810 ± 0.003 |\n| PGD | -3.652 ± 0.015 | -3.026 ± 0.013 | -2.301 ± 0.007 | -1.816 ± 0.006 | -1.452 ± 0.003 |\n| FQE | -2.228 ± 0.001 | -1.364 ± 0.001 | -0.991 ± 0.001 | -0.811 ± 0.002 | -0.691 ± 0.002 |\n| DR | -3.489 ± 0.024 | -2.454 ± 0.002 | -1.649 ± 0.024 | -1.317 ± 0.012 | -1.140 ± 0.018 |\n\nIn summary, we found that: \n1. STITCH-OPE performance degrades very slowly as $T$ increases, validating the theoretical bound of Theorem 3.3 empirically, \n2. as predicted, model-based and importance sampling based methods degrade rather quickly as $T$ increases, whereas the performance of STITCH-OPE remains more stable across $T$, and \n3. STITCH-OPE outperforms all other baselines according to MSE across all values of $T$, striking the optimal balance between fully auto-regressive and existing trajectory diffusion methods. \n\nWe will include results for all metrics (rank correlation and Regret@1) for this ablation experiment in our revised version.\n\n---\n\n## (Question) Does that mean 768 seconds for trajectory (i.e., frequency is 1HZ), or the frequency is not 1HZ?\n\nWhen we write $T = 768$, we refer to the number of decision epochs in each evaluation episode. Indeed, as you correctly pointed out, it would imply 768 seconds at a frequency of 1Hz."}, {"review_id": "ZycqG4prju", "reviewer": "Reviewer_fKxK", "summary": "This paper proposes a method called Stitch-OPE for evaluating the performance of policies trained on offline data. Stitch-OPE first trains a diffusion model capable of generating sub-trajectories based on data collected by the behavior policy. It then estimates the distribution of the target policy using importance sampling. Finally, it evaluates the performance of trajectories generated under the target policy using a learned reward predictor. This work addresses a key challenge in offline reinforcement learning. However, it may suffer from certain inaccuracies in out-of-distribution (OOD) regions.", "questions": "1. What is the direct relationship between Figure 1 and Off-Policy Evaluation?\n\n2. How does Stitch-OPE handle the OOD (out-of-distribution) issue? Intuitively, since Stitch-OPE trains a reward predictor using behavior data, wouldn't it lead to significant errors in evaluating the performance of the target policy if the trajectories generated by the target policy fall into OOD regions?", "limitations": "This method may suffer from certain inaccuracies in out-of-distribution (OOD) regions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The experimental results are relatively strong.\n2. The paper provides detailed experimental settings.\n\n### Weaknesses\n1. The baselines in Figure 2 and Table 2 are inconsistent. Figure 2 labels the baseline as \"MBR\", while Table 2 uses \"MB\".\n2. The content illustrated in Figure 1 does not seem to be directly related to Off-Policy Evaluation.", "was_revised": true, "final_justification": "The authors' response has addressed most of my concerns.", "rebuttal": "We would like to thank the reviewer for their detailed comments. We appreciate that the reviewer found our empirical results versus a variety of recent model-based and model-free OPE methods convincing, including the detailed ablation studies testing various hyper-parameter choices of STITCH-OPE. Please find our responses to your questions below.\n\n---\n\n## (Q1) What is the direct relationship between Figure 1 and Off-Policy Evaluation?\n\nFigure 1 illustrates that several components and innovations in model-based reasoning are required to achieve robust model-based OPE in practice. This includes:\n1. careful processing of the offline data set to ensure good sample efficiency\n2. chunked prediction, composition and guidance during inference are crucial to handle policy distribution shift, which is central to all OPE methods, and\n3. empirical reward and return estimation, which is the main task in OPE.\n\nIn more details:\n1. Figure 1 (Part A) shows how the offline data is split into overlapping sub-trajectories. Critical to model-based OPE methods, this builds an offline training data set on which accurate short-horizon transition dynamics can be learned using less offline data. \n2. Once these dynamics are learned using a chunked diffusion model, we effectively obtain a model that accurately captures how the state evolves under the behavior policy (Part B). A critical question of all OPE methods is how to tackle the distribution shift that arises between the behavior and target policy. Figure 1 (Part D) shows that this question can be effectively addressed by sub-trajectory composition and target policy guidance. These ideas work synergistically to account for the policy distribution shift, thus accurately approximating what would happen had we executed the target policy in the real world, without access to a simulator or the real world which is a key feature of OPE. \n3. Finally, return estimation involves estimating the empirical reward model from the offline data (Part C), and together with accurate synthetic rollouts from our diffusion model, we can generate accurate estimates of the target policy value, which is the main goal of OPE.\n\n---\n\n## (Q2) How does Stitch-OPE handle the OOD (out-of-distribution) issue? Intuitively, since Stitch-OPE trains a reward predictor using behavior data, wouldn't it lead to significant errors in evaluating the performance of the target policy if the trajectories generated by the target policy fall into OOD regions?\n\nGreat question! We thank the reviewer for raising a very important point about the necessary assumptions for successful implementation and the limitations of STITCH-OPE. \n\nIn summary, we expect STITCH-OPE to generate novel trajectories, but by leveraging the existing structure of the behavior data set through correct composition and stitching of the data, we believe the burden of predicting reward on OOD transitions is significantly reduced because stitching and composition do not involve generating OOD states. We now explain this point in greater detail:\n\n**Clarification on the Meaning of OOD:** When we refer to out-of-distribution in the context of STITCH or other OPE methods, we mean that rollouts simulated from behavior and target policies might be extremely dissimilar (i.e. in terms of standard measure like L2 norm, KL-divergence), but at the level of individual transitions or chunks, they must still share some common structure. This is typically expressed in OPE literature as requiring that the support of the target policy's action distribution (at every individual decision time step) must be within the support of the behavior distribution, which is also required by STITCH-OPE (Section 3.4). Consequentially, in the normal application of STITCH-OPE, we *do* expect to generate entirely novel trajectories through effective composition and stitching, but individual transitions within these rollouts are still expected be connected to the behavior data.\n\n**Stitching Improves Reward Estimation in OOD Regions:**\nWe also argue that successful stitching is a necessary component of accurate reward estimation, since it allows the synthetic rollouts to better match the target rollout distribution, while also respecting the overlapping policy support assumption that is necessary for successful reward prediction (for instance, see row A of Table 1 in our paper).\n\n**Standard Model-Based RL Assumptions:** We agree with the reviewer that estimation of immediate reward can be particularly challenging when it is nonsmooth or non-Lipschitz, i.e. discontinuous or pathological, even with successful stitching. However, our practical assumption is that the reward is both Markovian and sufficiently well-behaved that it can be approximated from the offline data sets in standard RL benchmarks. This assumption also underlies all model-based offline RL methods that leverage deep learning to approximate both the dynamics and the reward function for value estimation. \n\n**Accurate Dynamics Estimation is a Difficult Problem in RL:** We also want to emphasize that learning the state dynamics accurately from purely offline data -- while also avoiding the error compounding problem over long horizons during evaluation -- is very often highly non-trivial in OPE. For instance, even if errors of estimated $P(s' | s, a)$ are moderately small at any given time step, they can quickly blow up over time rendering them ineffective over sufficiently long horizon. Meanwhile, reward estimation error typically does not compound, since rewards are predicted individually at the transition level and thus their error accumulates linearly in $T$. The recent paper of Jiang [1] discusses a number of practical challenges of dynamics estimation that are not present in reward estimation, including error compounding and challenges of accurately learning stochastic dynamics. It also shows generally that the error in value increases quadratically in terms of the dynamics error; this is asymptotically looser than the bound on the value error that grows linearly in terms of the reward error. Thus, we wish to re-emphasize that the main goal of STITCH-OPE is to estimate the dynamics accurately offline through composition and stitching, which tackles a crucial problem in model-based OPE that has been inadequately addressed by prior work. \n\n**Ablation Experiment to Study the Effects of Reward Estimation on D4RL:** To determine the practical effect of reward estimation on the performance of STITCH-OPE on standard offline benchmarks, we ran STITCH-OPE on the Hopper-medium data set using both the true reward function and the empirically estimated reward function. The aggregated performance comparison is provided below:\n\nAggregated Reward Ablation on Hopper-medium\n| Metric | True Reward | Estimated Reward |\n| :--- | :-- | :--- |\n| Log-RMSE | -2.30 ± 0.01| -2.28 ± 0.01 | \n| Rank Correlation | 0.77 ± + 0.01  | 0.75 ± 0.01|\n| Regret@1 | 0.10 ± 0.03  | 0.13 ± 0.03 |\n\nand the per-policy performance is also included:\n\nPer-Target-Policy Reward Ablation on Hopper-medium\n| Target Policy | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Value using True Reward | 137.52 | 229.49 | 218.71 | 233.40 | 268.40 | 255.46 | 258.95 | 239.99 | 260.88 | 268.18 | \n| Value using Est. Reward | 135.43 | 228.80 | 217.35 | 229.55 | 266.77 | 253.43 | 253.87 | 235.85 | 256.79 | 264.88 |\n| Relative Error \\% | -1.52 | -0.30 | -0.62 | -1.65 | -0.61 | -0.79 | -1.96 | -1.72 | -1.57 | -1.23 |\n\nIn summary, we found that the reward estimation error accumulated has a minimal effect on OPE performance measured using aggregate metrics. We also found that the error of the OPE estimation remains stable and relatively low across all target policies, including those that differ the most from the behavior policy ($\\pi_0$, $\\pi_9$) and result in the most significant OOD shift between target and behavior rollouts. This concludes that the prediction errors of the reward were not a significant concern in the Gym and D4RL benchmark suite. We once again thank the reviewer for raising this important point. We will clarify the aforementioned points in our revision and add the previously discussed ablation experiments to our appendix.\n\n---\n\n## (Weakness 1) The baselines in Figure 2 and Table 2 are inconsistent.\n\nWe thank the reviewer for pointing out this typo and we will fix this in our revision.\n\n---\n\n1. Jiang, Nan. \"A note on loss functions and error compounding in model-based reinforcement learning.\" arXiv, 2024."}]

Author rebuttal:

Meta-review: This paper introduces a diffusion framework for off-policy evaluation that stitches shorter sub-trajectories to construct long trajectories and uses a negative behavior guidance mechanism to address distribution shift.

All of the reviewers were positive about the paper for its technical soundness, theoretical analysis, and strong empirical performance on benchmarks. While there were initial questions regarding the novelty of the stitching component, the choice of baselines, and specific hyperparameters, the authors provided a comprehensive and convincing rebuttal, successfully addressing all concerns.

Given the novelty, strong theoretical and empirical backing, and I believe that this work is significant enough to gather interests from the diffusion & reinforcement learning community.
Decision: Accept (spotlight)