Title: STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Off Policy Evaluation

Reinforcement Learning

Generative Models
Abstract: Off-policy evaluation (OPE) estimates the performance of a target policy using offline data collected from a behavior policy, and is crucial in domains such as robotics or healthcare where direct interaction with the environment is costly or unsafe. Existing OPE methods are ineffective for high-dimensional, long-horizon problems, due to exponential blow-ups in variance from importance weighting or compounding errors from learned dynamics models. To address these challenges, we propose STITCH-OPE, a model-based generative framework that leverages denoising diffusion for long-horizon OPE in high-dimensional state and action spaces. Starting with a diffusion model pre-trained on the behavior data, STITCH-OPE generates synthetic trajectories from the target policy by guiding the denoising process using the score function of the target policy. STITCH-OPE proposes two technical innovations that make it advantageous for OPE: (1) prevents over-regularization by subtracting the score of the behavior policy during guidance, and (2) generates long-horizon trajectories by stitching partial trajectories together end-to-end. We provide a theoretical guarantee that under mild assumptions, these modifications result in an exponential reduction in variance versus long-horizon trajectory diffusion. Experiments on the D4RL and OpenAI Gym benchmarks show substantial improvement in mean squared error, correlation, and regret metrics compared to state-of-the-art OPE methods.

STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation

Abstract

Off-policy evaluation (OPE) estimates the performance of a target policy using offline data collected from a behavior policy, and is crucial in domains such as robotics or healthcare where direct interaction with the environment is costly or unsafe. Existing OPE methods are ineffective for high-dimensional, long-horizon problems, due to exponential blow-ups in variance from importance weighting or compounding errors from learned dynamics models. To address these challenges, we propose STITCH-OPE, a model-based generative framework that leverages denoising diffusion for long-horizon OPE in high-dimensional state and action spaces. Starting with a diffusion model pre-trained on the behavior data, STITCHOPE generates synthetic trajectories from the target policy by guiding the denoising process using the score function of the target policy. STITCH-OPE proposes two technical innovations that make it advantageous for OPE: (1) prevents overregularization by subtracting the score of the behavior policy during guidance, and (2) generates long-horizon trajectories by stitching partial trajectories together end-to-end. We provide a theoretical guarantee that under mild assumptions, these modifications result in an exponential reduction in variance versus long-horizon trajectory diffusion. Experiments on the D4RL and OpenAI Gym benchmarks show substantial improvement in mean squared error, correlation, and regret metrics compared to state-of-the-art OPE methods †

1 Introduction

Given the slow and risky nature of online data collection, real-world applications of reinforcement learning often require offline data for policy learning and evaluation [23, 39]. An important problem of working with offline data is off-policy evaluation (OPE), which aims to evaluate the performance of target policies using offline data collected from other behavior policies. One practical advantage of OPE is that it saves the cost of evaluation on hardware in embodied applications in the real world [28]. However, a central challenge of OPE is the presence of distribution shift induced by differences in behavior and target policies [23, 3]. This can lead to inaccurate estimates of policy values, making it difficult to trust or select between multiple target policies before they are deployed [44, 26].

Numerous approaches have attempted to address the distribution shift in offline policy evaluation by reducing either the variance of the policy value or its bias, but they are typically ineffective in

stitch-ope.github.io .

I. Behavior Data

II. Target Policy

III. PGD

IV. Ours

A: Conditional

diffusion

improves

composition.

X

Y

X

Y

X

Y

X

Y

B: Conditional

diffusion

improves state

generalization.

X

Y

X

Y

X

Y

X

Y

C: Behavior

policy guidance

addresses

distribution

shift.

X

Y

X

Y

X

Y

X

Y

high-dimensional long-horizon problems. For example, Importance Sampling (IS) [33] estimates the value of the target policy by weighing the behavior policy rollouts according to the ratio of their likelihoods. However, it suffers from the so-called curse of horizon where the variance of the estimate increases exponentially in the evaluation horizon [25]. More recent model-free OPE estimators reduce or eliminate the explosion in variance by estimating the long-run state-action density ratio d π ( s, a ) /d β ( s, a ) between the target and behavior policy [25, 31, 43], yet they have demonstrated poor empirical performance on high-dimensional tasks where the behavior and target policies are different (i.e. the behavior policy is not a noisy version of the target policy) [14].

As an alternative approach, model-based OPE estimators typically learn an empirical autoregressive model of the environment and reward function from the behavior data, which is used to generate synthetic rollouts from the target policy for offline evaluation [19, 38, 46]. Some advantages of the model-based paradigm include sample efficiency [24], exploitation of prior knowledge about the dynamics [12], and better generalization to unseen states [45]. Although model-based OPE methods often scale well to high-dimensional short-horizon problems - owing to the scalability of the deep model-based RL paradigm - their robustness diminishes in long-horizon tasks due to the compounding of errors in the approximated dynamics model [14, 17, 18].

Driven by the recent successes of generative diffusion in RL [29, 48, 1, 16, 30, 35], we propose S ubT rajectory I mportance-Weighted T rajectory C omposition for LongH orizon OPE for model-based offpolicy evaluation in long-horizon high-dimensional problems. STITCH-OPE first trains a diffusion model on behavior data, allowing it to generate dynamically feasible behavior trajectories [18].

STITCH-OPE differs from prior work by training the diffusion model on short sub-trajectories instead of full rollouts, where sub-trajectory generation is conditioned on the final state of the previous generated sub-trajectory. This enables accurate trajectory 'stitching" using short-horizon rollouts, while minimizing compounding error of full-trajectory rollouts, thus bridging the gap between model-based OPE and full-trajectory offline diffusion.

STITCH-OPE explicitly accounts for distribution shift in OPE by guiding the diffusion denoising process [9, 18] during inference. This can be achieved by selecting the guidance function to be the difference between the score functions of the target and behavior policies. A significant advantage of guided diffusion is that it eliminates the need to retrain the diffusion model for each new target policy. By pretraining the model on a variety of behavior datasets, generalization can be achieved during guided sampling to produce feasible trajectories under the target policy, leading to robust off-policy estimates for target policies that lack offline data. STITCH-OPE contributes the following novel technical innovations that we consider critical to the successful and robust application of diffusion models for OPE:

Trajectory Stitching with Conditional Diffusion. We propose a state-conditioned guided diffusion model for generating short sub-trajectories from the target policy (Figure 1). This significantly improves the quality (Table 1, row A) and generalization (Table 1, row B) of trajectory generation in long-horizon tasks. Theorem 3.3 also provides theoretical bounds on the bias and variance of our proposed approach.

Behavior Guidance. We show that the negative score function of the behavior policy mitigates the diffusion model from collapsing to trajectories with large behavior likelihood, thus improving generalization out-of-data (Table 1, row C). This negative score function naturally arises from viewing the likelihood ratio (i.e. in importance sampling) as the classification density in diffusion guidance [9] - a connection missed in prior work [16].

Robustness Across Problem Difficulty. Finally, we evaluate STITCH-OPE on the OpenAI Gym control suite [4] and the D4RL offline RL suite [13], showing significant improvements compared to other recent OPE estimators across a variety of metrics (mean squared error, rank correlation and regret), problem dimension and evaluation horizon. To our knowledge, STITCH-OPE is the first work to demonstrate robust off-policy evaluation in high-dimensional long-horizon tasks.

2 Preliminaries

Markov Decision Processes. We consider the standard Markov Decision Process (MDP), which consists of a 6-tuple ⟨S , A , R, P, d 0 , γ ⟩ [34] where: S is the continuous state space, A is the continuous action space, R is the reward function, P is the Markov transition probability distribution, d 0 is the initial state distribution and γ ∈ [0 , 1] is the discount factor.

A policy π : S × A → [0 , ∞ ) observes the current state s , samples an action according to its conditional distribution π ( ·| s ) , and observes the immediate reward R ( s, a ) and the next state s ′ ∼ P ( ·| s, a ) . The interaction of a policy with an MDP generates a set of trajectories of states, actions, and rewards. The goal of reinforcement learning is to learn a policy π that maximizes the expected return:

over lengthT trajectories τ = ( s 0 , a 0 , s 1 , a 1 , . . . s T ) induced by policy π .

Off-Policy Evaluation. The goal of Off-Policy Evaluation (OPE) is to estimate the expected return of some target policy π given only a data set of trajectories D β from some behavior policy β . To estimate J ( π ) , it is necessary to approximate the distribution over trajectories p π ( τ ) induced by π using only samples from the distribution p β ( τ ) . Hence, the phenomenon of distribution shift arises whenever β is sufficiently different from π , in which case p β ( τ ) is not a suitable proxy for obtaining samples from p π ( τ ) and corrections must be made to account for the distribution shift.

Denoising Diffusion Models. Denoising Diffusion Probabilistic Models (DDPMs) [36, 15] are a class of generative models to sample from a given distribution. Given a dataset { x i } N i =1 and the

K -step (forward) noise process x k i = √ α k x k -1 i + √ 1 -α k ϵ , where ϵ ∼ N (0 , I ) is independent, DDPMs are trained to perform the K -step (backward) denoising process to recover x i from x K i ∼ N (0 , I ) . This is accomplished by running the forward process x 0 i → x k i on the original x i and training the DDPM ϵ θ on the denoising process x k -1 i | x k i ∼ N ( µ ( x k i , k ) , σ 2 k I ) to predict the noise ϵ so that the distribution over denoised samples x 0 i matches x i . The standard reparameterization µ ( x k i , k ) = ( x k i -ϵ θ ( x k i , k )(1 -α k ) /σ k ) / √ α k maps x k and the predicted noise ϵ θ ( x k i , k ) to x k -1 to undo the noise accumulated during the forward process. The following loss function is typically used to train a diffusion model [15]

Guided Diffusion. It is possible to guide the sampling from a trained diffusion model to maximize some classifier p ( y | x ) [9]. A key observation of diffusion is that the backward diffusion process can be well approximated by a Gaussian when the noise is small, that is, x k | x k +1 ∼ N ( µ k , Σ k ) . Next, observe that p ( x k | x k +1 , y ) ∝ p ( x k | x k +1 ) p ( y | x k ) by Bayes' rule. Applying the first-order Taylor approximation log p ( y | x k ) ≈ log p ( y | µ k )+( x k -µ k ) g ( µ k ) at µ k , where g ( u ) = ∇ x log p ( y | x ) | x = u , it can be shown that:

In other words, it is possible to sample from the conditional (guided) distribution p ( x k | x k +1 , y ) by sampling from the original diffusion model with its mean shifted to µ k +Σ k g ( µ k ) . Appendix A provides a worked example illustrating guided diffusion for a mixture of Gaussians.

3 Proposed Methodology

The direct method for off-policy evaluation [11] estimates the single-step autoregressive model ˆ P ( s t | s t -1 , a t -1 ) and the reward function ˆ R ( s t , a t ) from the behavior data. Then, it draws target policy trajectories τ ∼ p π ( τ ) by forward sampling, that is, s 0 ∼ d 0 , a 0 ∼ π ( ·| s 0 ) , s 1 ∼ ˆ P ( ·| s 0 , a 0 ) , . . . s T ∼ ˆ P ( ·| s T -1 , a T -1 ) . However, even small errors in ˆ P can lead to significant bias in J ( π ) due to the compounding of errors over long horizon T [20, 17]. STITCH-OPE avoids the compounding problem by generating the partial trajectory in a single (backward diffusion) pass, leading to more accurate OPE estimates over a long horizon.

3.1 Guided Diffusion for Off-Policy Evaluation

It is possible to approximate p π using guided diffusion by interpreting each data point x i as a full trajectory τ . Given a behavior policy β and corresponding lengthT trajectory distribution p β ( τ ) , the corresponding lengthT trajectory distribution of target policy π can be written as:

which is the standard importance sampling correction [33]. We address the question of tractably learning p β ( τ ) by training a diffusion model ˆ p β ( τ ) on the offline behavior data set D β [18], thus approximating ˆ p β ( τ ) ≈ p β ( τ ) . Specifically, the diffusion model learns to map a trajectory consisting of pure noise, τ k = ( s k 0 , a k 0 , . . . s k T ) , to a noiseless behavior trajectory τ 0 = ( s 0 0 , a 0 0 , . . . s 0 T ) .

Akey observation is that we can bypass importance sampling in (3) by guiding the generation process ˆ p β ( τ ) towards p π ( τ ) using diffusion guidance (2) [18]. Specifically, let x k = τ k denote a noisy behavior trajectory at step k of the forward diffusion process, and let y ∈ { 0 , 1 } be a binary outcome with p ( y = 1 | τ ) ∝ ∏ T -1 t =0 π ( a t | s t ) β ( a t | s t ) . Intuitively, y indicates whether the trajectory τ is generated by the target policy π ( y = 1 ) or the behavior policy β ( y = 0 ), and the likelihood ratio determines the odds that y = 1 given τ . By (3),

Behavior Data

Partial Rollouts

A

Pre-Trained Diffusion

τ

k

. . .

τ

0

(

s

t

,

/epsilon1

k

)

State + Noise

(

s

t

, . . . s

t

+

w

)

Partial Rollout

B

E

∇

τ

∑

t

log

π

(

a

t

|

s

t

)

∇

τ

∑

t

log

β

(

a

t

|

s

t

)

-

-

Policy Guidance

D

s

t

a

t

ˆ

R

(

s

t

, a

t

)

r

t

Reward Predictor

C

ˆ

J

(

π

) =

E

p

w

π

[

∑

t

γ

t

ˆ

R

(

s

t

, a

t

)]

Off-Policy Evaluation

and thus the backward diffusion process for generating target policy trajectories for OPE can be approximated with guidance (2):

where p β ( τ k | τ k +1 ) = N ( µ k , Σ k ) is the backward diffusion process. Therefore, we can obtain feasible target policy trajectories using the guidance function:

Given the approximate sampling distribution over the trajectories of the target policy described above, ˆ p π ( τ 0 ) = ∫ · · · ∫ N ( τ K ; 0 , I ) ∏ K k =1 p π ( τ k -1 | τ k ) d τ K . . . d τ 1 , and an empirical reward function ˆ R ( s, a ) , it is straightforward to estimate the expected return (or a statistic such as variance or quantile) given any target policy, i.e. ˆ J ( π ) = E τ ∼ ˆ p π [ ∑ t γ t ˆ R ( s t , a t ) ] ≈ J ( π ) .

3.2 Negative Behavior Guidance

The target policy score function, g simple ( τ ) = ∇ τ ∑ T -1 t =0 log π ( a t | s t ) [16], provides a simple guidance function for OPE. However, it corresponds to a biased estimator of p π ( τ ) in the context of (3) and can generate trajectories that are unlikely under the target policy, as illustrated using the GaussianWorld domain in Table 1 (see Appendix B for details). The behavior policy β returns a positive angle in each state and the target policy π returns a negative angle, to test the performance of both guidance functions under distribution shift. Conclusions are summarized in row C of Table 1. The omission of the negative guidance term results in a sampling distribution that collapses to a high-density region under p β ( τ ) , where p π ( τ ) could be small. In other words, behavior guidance prevents the guided sampling distribution ˆ p π from becoming over-regularized.

In our empirical evaluation, we employ the following generalization of (5) to allow fine-grained control over the relative importance of the target and behavior policy guidance

Ignoring the normalizing constant which does not dependent on τ , (6) is equivalent to sampling from the following re-weighted trajectory distribution

which can be interpreted as a tempered posterior distribution [2, 5] over trajectories; the importance of the likelihood terms associated with π and β are controlled by the choice of α and λ , respectively. Note that the choice α = λ = 1 reduces to the standard guidance function g ( τ ) . This is not a good empirical choice because it can push the backward diffusion process too far from the behavior distribution, leading to infeasible trajectories or instability. Instead, typical choices satisfy λ < α (see Appendix L.1 for an additional experiment confirming this). The choice α = 1 , λ = 0 reduces to g simple ( τ ) and is unsuitable for OPE.

3.3 Sub-Trajectory Stitching with Conditional Diffusion

Recent work has shown that full-length diffusion models do not provide sufficient compositionality for accurate long-horizon sequence generation [6]. In addition, full-length prediction requires the generation of sequences of length T · (dim( A ) + dim( S )) ; this may be infeasible or inefficient on resource-constrained systems, when T is large or when A or S is high-dimensional.

To tackle these limitations, STITCH-OPE trains a conditional diffusion model to generate behavior sub-trajectories of length w ≪ T . To allow for a more flexible composition of behavior trajectories during guidance, generation in STITCH-OPE is performed in a semi-autoregressive manner from the diffusion model, which is conditioned on the last state of the previously generated sub-trajectory.

Specifically, the conditional diffusion model in STITCH-OPE, denoted as ϵ θ ( τ k t : t + w , k | s 0 t ) , denoises a lengthw noisy sub-trajectory τ k t : t + w = ( s k t , a k t , s k t +1 , a k t +1 , . . . s k t + w -1 , a k t + w -1 , s k t + w ) conditioned on the last state s 0 t of the previously generated sub-trajectory τ 0 t -w : t . Generalizing (1), the loss function of STITCH-OPE is thus

Next, writing p β ( τ t : t + w | s 0 t ) to denote the sampling distribution over fully denoised sub-trajectories τ 0 t : t + w conditioned on s 0 t , the sampling process (3) of STITCH-OPE can be written as:

where target trajectories are generated by guiding the conditional diffusion model (analogous to (4)) according to

A complete algorithm description of STITCH-OPE is provided in Appendix E.

To understand the intuition that the conditional diffusion model offers better compositionality than the full-horizon prediction, we decompose the behavior trajectory distribution as a mixture over the trajectories τ j in D β :

Meanwhile, the conditional diffusion model ignores the full history of past states, i.e.:

p ( τ j | s t ) has higher entropy than p ( τ j | s 0 , a 0 , . . . s t ) since it is conditioned on less information (see Appendix C for a proof), and thus provides a broader coverage of the diverse modes in the behavior dataset. This improves the compositionality of guided long-horizon trajectory generation. Row A of Table 1 illustrates this claim empirically using the GaussianWorld problem. A further claim is that the STITCH-OPE model can generalize better across initial states with low, or even zero, probability under d 0 (see row B of Table 1). This occurs because p w β is trained on sub-trajectories starting in arbitrary states in D β , as opposed to states sampled only from d 0 . Therefore, sliding windows strike an optimal balance between autoregressive methods and full-length trajectory diffusion, providing good compositionality while avoiding the error compounding in terms of T .

3.4 Theoretical Analysis

We provide theoretical guarantees for our proposed STITCH-OPE method by analyzing its bias and variance. The first prerequisite assumption is standard in OPE [42, 27] and limits the ratio π/β .

Assumption 3.1. There is a constant κ such that

The second prerequisite assumption bounds the total variation between the learned lengthw trajectory distribution ˆ p w β and the true distribution p w β under the behavior policy β .

Assumption 3.2. TV ( p w β , ˆ p w β ) ≤ δ β for some constant δ β .

Our main result is a bound on the mean squared error of the STITCH-OPE estimator. We defer the full proofs, technical lemmas, and definitions to Appendix D.

Theorem 3.3. Define ˆ p π as the (lengthT ) trajectory distribution of the guided diffusion model, and p π as the true trajectory distribution under the target policy π . Under Assumptions 3.1 and 3.2, the mean squared error (MSE) of the STITCH-OPE return ˆ J satisfies:

where B w = 1 -γ w 1 -γ sup s,a | R ( s, a ) | is a bound on the maximum lengthw discounted return, and J is the return under p π .

Remarks. Theorem 3.3 resembles the O (exp( cT )) bound of IS-based methods [25, 27], but is expressed in terms of w rather than T (with a more favorable O ( T 2 ) dependence). Since w is a fixed hyper-parameter typically chosen to be much smaller than T in practice, STITCH-OPE provides an exponential reduction in MSE versus both importance sampling and lengthT diffusion! In Section 4, we validate this claim further by showing that STITCH-OPE outperforms full trajectory diffusion (PGD) on most benchmarks (see Appendix L.2 for an additional experiment confirming small w > 1 is ideal in practice). The MSE decreases as the error in the learned behavior model δ β decreases. In practice, δ β is easier to estimate and control than the error of the target density p w π . It is also important to note that the variance cannot be reduced below Var p π ( J ) , the intrinsic variance of the environment and the target policy.

4 Empirical Evaluation

Our empirical evaluation aims to answer the following research questions:

Does the combination of conditional diffusion and negative guidance (as hypothesized in Table 1) translate to robust OPE performance on standard benchmarks?

Is STITCH-OPE robust across problem size (e.g., state/action dimension, horizon)?

Is STITCH-OPE robust across different levels of optimality of the target policy and the classes of policies?

FQE

DR

IS

DRE

MBR

PGD

STITCH

1.5

1.0

0.5

0.0

Log RMSE

FQE

DR

IS

DRE

MBR

PGD

STITCH

0.0

0.2

0.4

0.6

0.8

Rank Correlation

FQE

DR

IS

DRE

MBR

PGD

STITCH

0.0

0.1

0.2

0.3

Regret@1

4.1 Experiment Details

Domains. We evaluate the performance of STITCH-OPE in high-dimensional long-horizon tasks using the standard D4RL benchmark [13] and their respective benchmark policies [14]. Specifically, we use the halfcheetah-medium , hopper-medium and walker2d-medium behavior datasets. Each evaluation consists of 10 target policies π 1 , π 2 , . . . π 10 trained at varying levels of ability [14]. We also carry out similar experiments using classical control tasks (Pendulum and Acrobot) from OpenAI Gym [4], to evaluate the competitiveness of STITCH-OPE on standard benchmarks on which other baselines have been extensively evaluated. For this set of environments, we obtain the target policies by running the twin-delayed DDPG algorithm [8] (see Appendix G for details). We set the trajectory length to T = 768 for all D4RL problems, T = 256 for Acrobot, and T = 196 for Pendulum. We also use γ = 0 . 99 in all experiments. The domain details are provided in Appendix F, and the training details of STITCH-OPE are provided in Appendix J.

Baselines. We include the following model-free estimators: Fitted Q-Evaluation (FQE) [22], Doubly-Robust OPE (DR) [37], Importance Sampling (IS) [33], and Density Ratio Estimation (DRE) [31]. We also include the following model-based estimators: Model-Based (MB) [19, 40], and Policy-Guided Diffusion (PGD) [16]. The implementation details are provided in Appendix H. Appendix L.6 provides an additional comparison against Diffusion World Models (DWM) [10].

Metrics. Each baseline method is evaluated on each pair of behavior dataset and target policy for 5 random seeds. We also generate ground-truth estimates of each target policy value by running each policy in the environment. We evaluate the performance of each baseline using the Log Root Mean Squared Error (LogRMSE) , the Spearman Correlation , and the Regret@1 calculated as the difference in return between the best policy selected using the baseline policy value estimates and the actual best policy. Furthermore, to compare metrics consistently across tasks, we normalize the returns following [14]. Appendix I contains the technical details for metric calculation.

4.2 Discussion

Table 2 summarizes the performance of each method per domain, while Figure 2 summarizes the aggregated performance averaged across all domains. STITCH-OPE outperforms all baselines in 11 out of 15 instances (shown in bold), with general agreement among the different metrics. STITCHOPE soundly outperforms both single-step (MB) and full-trajectory (PGD) model-based methods. This reaffirms our argument in Section 3.3 that intermediate values of w provide a good balance between compositionality and compounding errors. Furthermore, STITCH-OPE performs particularly well according to rank correlation and Regret@1 (with very low standard error) and can accurately rank and identify the best-performing policy. This suggests that the target policy score function (with the negative behavior term) provides very informative guidance during denoising, allowing it to correctly evaluate target policies of varying levels of ability, even as some of those policies deviate significantly from the behavior policy. Finally, we see that STITCH-OPE performance remains consistent across the problem dimension, highlighting the scalability of diffusion when applied to OPE for high-dimensional problems.

4.3 Off-Policy Evaluation with Diffusion Policies

To demonstrate the ability of STITCH-OPE to evaluate more complex policy classes, we replace target policies with diffusion policies, which have led to significant advances in robotics [7, 41] (see Appendix K for details). Since STITCH-OPE only requires the score of the target policy, it is computationally straightforward to perform OPE with diffusion policies, which is not the case for other estimators that require an explicit probability distribution π i ( a | s ) over actions (i.e. IS, DR). D4RL results are provided in Table 3. We see that STITCH-OPE outperforms all other baselines in 6 out of 9 instances, demonstrating robust OPE performance across multiple target policy classes.

4.4 Ablations

We conduct additional experiments to test the sensitivity of STITCH-OPE to the choice of guidance coefficients ( α and λ ), window size w , and data sets. Due to space limitations, we defer results to Appendix L. We summarize the key findings below:

Sensitivity to Penalty Coefficients α and λ (Appendix L.1). The best performance occurs when 0 < λ < α , reaffirming our claim in Section 3.2 that optimal regularization occurs for small λ .

Sensitivity to Window Size w (Appendix L.2). The best performance occurs for w = 8 , showing that STITCH-OPE provides an optimal balance between autoregressive and fulltrajectory diffusion.

Robustness Across Datasets (Appendix L.3). We evaluated STITCH-OPE and all baseline methods on Hopper-medium-expert and Hopper-expert data sets. We found that STITCHOPE outperforms all other baselines by a significant margin in 5 out of 6 instances (metricdataset combinations) and maintained the most consistent performance across data sets.

Sensitivity to Evaluation Horizon T . Table 4 shows the log RMSE performance on the Hopper-medium data set for all baselines and various values of the evaluation horizon T . We also set γ = 0 . 999 to better see the performance differences between the methods for T = 1000 . The values are normalized based on the global minimum and maximum across all evaluation horizon experiments. STITCH-OPE consistently outperforms all baselines for all values of T considered, striking an optimal balance between auto-regressive and trajectory diffusion methods. Furthermore, STITCH-OPE performance slowly degrades as T increases, validating the theoretical upper bound of Theorem 3.3.

Effect of Reward Estimation (Appendix L.4). We compare STITCH-OPE performance given access to the true (unknown) reward function and its performance using reward estimation on Hopper-medium. The accumulated errors in the reward estimation have a minimal effect on the aggregate OPE performance. Furthermore, the error remains stable and uniformly low across all target policies.

5 Limitations

The theory of STITCH-OPE relies on (standard) Assumptions 3.1 and 3.2. In practice, if there exist many ( s, a ) pairs such that β ( a | s ) = 0 but π ( a | s ) > 0 , then the behavior data may be incomplete and diffusion guidance could generate infeasible trajectories and produce biased estimates of J ( π ) . Diffusion models trained on image data in other applications are often easy to interpret; however, evaluating the fidelity of the trajectories generated from the trained behavior model p β ( τ ) is challenging when the state is complex and difficult to interpret or partially observable. Finally, while STITCH-OPE has demonstrated excellent performance on existing OPE benchmarks, it remains unanswered whether its benefits also apply to domain-specific problems outside robotics.

6 Conclusion

Wepresented STITCH-OPE for off-policy evaluation in high-dimensional, long-horizon environments. STITCH-OPE trains a conditional diffusion model to generate behavior sub-trajectories, and applies diffusion guidance using the score of the target policy to correct the distribution shift induced by the target policy. Our novelties include trajectory stitching and negative behavior policy guidance, which were shown to improve composition and generalization. Using D4RL and OpenAI Gym benchmarks, we showed that STITCH-OPE outperforms state-of-the-art OPE methods across MSE, correlation and regret metrics. Future work could investigate online data collection to address severe distribution shift, or explore ways to adapt the guidance coefficients or incorporate additional knowledge into the guidance function (e.g. additional structure on the dynamics). It also remains an open question whether the advantages of STITCH-OPE apply to offline policy optimization.

Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers and area chairs for their constructive feedback. This work was partially supported by the Toyota Research Institute (TRI).

References

A. Ajay, Y. Du, A. Gupta, J. B. Tenenbaum, T. S. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= sP1fo2K9DFG .

P. Alquier and J. Ridgway. Concentration of tempered posteriors and of their variational approximations. The Annals of Statistics , 48(3):1475-1497, 2020.

D. Brandfonbrener, W. Whitney, R. Ranganath, and J. Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems , 34:4933-4946, 2021.

G. Brockman. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

F. Cérou, P. Héas, and M. Rousset. Adaptive reduced tempering for bayesian inverse problems and rare event simulation. arXiv preprint arXiv:2410.18833 , 2024.

B. Chen, D. M. Monsó, Y. Du, M. Simchowitz, R. Tedrake, and V. Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research , page 02783649241273668, 2023.

S. Dankwa and W. Zheng. Twin-delayed ddpg: A deep reinforcement learning technique to model a continuous movement of an intelligent robot agent. In Proceedings of the 3rd international conference on vision, image and signal processing , pages 1-5, 2019.

P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

Z. Ding, A. Zhang, Y. Tian, and Q. Zheng. Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning. arXiv preprint arXiv:2402.03570 , 2024.

M. Farajtabar, Y. Chow, and M. Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning , pages 1447-1456. PMLR, 2018.

M. Fard and J. Pineau. Pac-bayesian model selection for reinforcement learning. Advances in Neural Information Processing Systems , 23, 2010.

J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

J. Fu, M. Norouzi, O. Nachum, G. Tucker, ziyu wang, A. Novikov, M. Yang, M. R. Zhang, Y. Chen, A. Kumar, C. Paduraru, S. Levine, and T. Paine. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations , 2021. URL https: //openreview.net/forum?id=kWSeGEeHvF8 .

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

M. T. Jackson, M. Matthews, C. Lu, B. Ellis, S. Whiteson, and J. N. Foerster. Policy-guided diffusion. In Reinforcement Learning Conference , 2024.

M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 1273-1286. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/ 2021/file/099fe6b0b444c23836c4a5d07346082b-Paper.pdf .

M. Janner, Y. Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , pages 9902-9915. PMLR, 2022.

N. Jiang and L. Li. Doubly robust off-policy value evaluation for reinforcement learning. In International conference on machine learning , pages 652-661. PMLR, 2016.

N. Jiang, A. Kulesza, S. Singh, and R. Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 international conference on autonomous agents and multiagent systems , pages 1181-1189, 2015.

R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems , 33:21810-21823, 2020.

H. Le, C. Voloshin, and Y. Yue. Batch policy learning under constraints. In International Conference on Machine Learning , pages 3703-3712. PMLR, 2019.

S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

G. Li, L. Shi, Y. Chen, Y. Chi, and Y. Wei. Settling the sample complexity of model-based offline reinforcement learning. The Annals of Statistics , 52(1):233-260, 2024.

Q. Liu, L. Li, Z. Tang, and D. Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. Advances in neural information processing systems , 31, 2018.

V. Liu, P. Nagarajan, A. Patterson, and M. White. When is offline policy selection sample efficient for reinforcement learning? arXiv preprint arXiv:2312.02355 , 2023.

Y. Liu, P. L. Bacon, and E. Brunskill. Understanding the curse of horizon in off-policy evaluation via conditional importance sampling. In International Conference on Machine Learning , pages 6184-6193. PMLR, 2020.

Y. Liu, W. Chen, Y. Bai, G. Li, W. Gao, and L. Lin. Aligning cyber space with physical world: A comprehensive survey on embodied ai. CoRR , 2024.

C. Lu, P. J. Ball, Y . W. Teh, and J. Parker-Holder. Synthetic experience replay. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview. net/forum?id=6jNQ1AY1Uf .

L. Mao, H. Xu, X. Zhan, W. Zhang, and A. Zhang. Diffusion-DICE: In-sample diffusion guidance for offline reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview.net/forum?id= EIl9qmMmvy .

A. Mousavi, L. Li, Q. Liu, and D. Zhou. Black-box off-policy estimation for infinite-horizon reinforcement learning. In International Conference on Learning Representations , 2020.

O. Nachum, Y. Chow, B. Dai, and L. Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in neural information processing systems , 32, 2019.

D. Precup, R. S. Sutton, and S. P. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning , pages 759-766, 2000.

M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

M. Rigter, J. Yamada, and I. Posner. World models via policy-guided trajectory diffusion. Transactions on Machine Learning Research , 2024. ISSN 2835-8856. URL https: //openreview.net/forum?id=9CcgO0LhKG .

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

P. Thomas and E. Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning , pages 2139-2148. PMLR, 2016.

M. Uehara and W. Sun. Pessimistic model-based offline rl: Pac bounds and posterior sampling under partial coverage. CoRR , 2021.

M. Uehara, C. Shi, and N. Kallus. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355 , 2022.

C. Voloshin, H. M. Le, N. Jiang, and Y. Yue. Empirical study of off-policy policy evaluation for reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021.

Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=AHvFDPi-FA .

T. Xie, Y. Ma, and Y.-X. Wang. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. Advances in neural information processing systems , 32, 2019.

M. Yang, O. Nachum, B. Dai, L. Li, and D. Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems , 33:6551-6561, 2020.

M. Yang, B. Dai, O. Nachum, G. Tucker, and D. Schuurmans. Offline policy selection under uncertainty. In International Conference on Artificial Intelligence and Statistics , pages 43764396. PMLR, 2022.

T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Modelbased offline policy optimization. Advances in Neural Information Processing Systems , 33: 14129-14142, 2020.

M. R. Zhang, T. Paine, O. Nachum, C. Paduraru, G. Tucker, ziyu wang, and M. Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum? id=kmqjgSNXby .

R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference on Learning Representations , 2020.

Z. Zhu, H. Zhao, H. He, Y. Zhong, S. Zhang, H. Guo, T. Chen, and W. Zhang. Diffusion models for reinforcement learning: A survey. arXiv preprint arXiv:2311.01223 , 2023.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All theoretical and empirical claims have been summarized in the abstract and match the results attained in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 5 discusses the assumptions required to obtain the theoretical bound of the method and attain good empirical performance. We also mention that, while the approach performs well on standard benchmark problem sets, its performance on domain-specific problems is unclear.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Appendix C and Appendix D contain all related definitions, theorems and proofs that support the theoretical results stated in the main paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Appendix H discusses how baseline methods were setup as well as all hyperparameters needed to reproduce the results. Appendix G discusses policy calculation. Appendix F discusses how the domains were set up, and Appendix I discusses how metrics were calculated. Appendix J discusses hyper-parameters needed to run STITCH-OPE on all problems. Together, these sections were included to allow reproduction of the experiments to the best of our ability.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Anonymized code is included in the zip file as part of the supplementary material, along with instructions to run the code in a readme file.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The Appendix contains all relevant training and test details, hyper-parameters and other important design considerations.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All tables of numerical results and plots clearly show standard error bars and intervals where appropriate.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix M provides the computer resources used and the total time of experiments of our method.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and adhere to the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We believe the current paper is foundational research, thus we do not foresee any direct societal impacts of the work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The current work uses code bases from other packages. Their accompanying papers were cited, URL links to the codebases were included, and their licenses (where available) were mentioned in the Appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide code to reproduce the experiments as part of the supplementary zip file. Instructions for running the code are provided in a readme file included in the code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The research method does not involve LLMs.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation

Supplementary Material

Abstract

This supplement to the paper discusses algorithmic and experiment details that were not included in the main paper due to space limitations. It includes proofs of all main theoretical claims, as well as all configurations and parameter settings that are required to reproduce the experiments. It includes additional experiments and ablation studies that were excluded from the main paper due to space limitations.

Contents

A Pedagogical Example for Guided Diffusion

We consider the simple two-component mixture of Gaussians with density function

where N ( x ; µ, σ 2 ) is the density function of a N ( µ, σ 2 ) distribution. Using the standard substitution in the backward diffusion process, produces the backward diffusion process

To illustrate the effects of a guidance function on the sampling process, we consider the guidance function associated with the (unscaled) score of a N (1 , 0 . 5 2 ) distribution, i.e.

Then, the guided backward diffusion process has mean:

which corresponds to a standard backward diffusion process with the modified score function

which would place more weight on the rightmost mode of the Gaussian mixture during the backward diffusion process.

We run the backward denoising diffusion process using the exact score function ∇ log p ( x k ) . The sampling distributions of x k are plotted at various denoising time steps k in Figure 3.

4

3

2

1

0

1

2

3

4

x

0.00

0.01

0.01

0.01

0.02

0.03

0.03

0.04

p(x)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

score of p(x)

4

3

2

1

0

1

2

3

4

x

0.00

0.05

0.10

0.15

0.20

p(x)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

score of p(x)

4

3

2

1

0

1

2

3

4

x

0.00

0.10

0.20

0.30

0.40

0.50

0.60

p(x)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

score of p(x)

4

3

2

1

0

1

2

3

4

x

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

p(x)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

score of p(x)

4

3

2

1

0

1

2

3

4

x

0.00

0.01

0.01

0.01

0.02

p(x)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

score of p(x)

4

3

2

1

0

1

2

3

4

x

0.00

0.02

0.04

0.06

0.08

0.10

0.12

p(x)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

score of p(x)

4

3

2

1

0

1

2

3

4

x

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

p(x)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

score of p(x)

4

3

2

1

0

1

2

3

4

x

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

p(x)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

score of p(x)

B GaussianWorld Domain

The GaussianWorld domain is a toy 2-dimensional Markov decision process defined designed to illustrate and compare generalization and compositionality of diffusion models (Table 1). It is defined as follows:

Decision Epochs. The decision epochs are t = 0 , 1 , 2 , . . . T where we set T = 128 in our experiments.

State Space. S = R 2 describes all positions ( x t , y t ) of a particle in space at every decision epoch t . It is assumed that x t is the x-coordinate and y t is the y-coordinate. The initial state is s 0 = (0 , 0) unless otherwise specified.

Action Space. A = R describes the (counterclockwise) angle of the movement vector of the particle at every decision epoch, relative to the horizontal.

Transitions. Letting a t be the angle of movement of the particle at time t , the transitions of x t and y t are defined as follows:

Here, ε t is an i.i.d. Gaussian noise added to the actions before they are applied by the controller.

Reward Function and Discount. The problem is not solved so we leave the reward unspecified. We also leave the discount factor unspecified.

C Proof that Conditional Diffusion Increases Entropy

We begin with the following definitions.

Definition C.1 (Entropy) . Let p ( x ) be a density function of a random variable X with support X . The entropy of X is defined as

Definition C.2 (Conditional Entropy) . The conditional entropy of X given Y on support Y is defined as

Our goal is to prove

Theorem C.3. Let S t be the random state at time t sampled according to the conditional distribution p ( S t +1 = s | S t = x, A t = u ) , and let A t be a random action following some conditional distribution p ( A t = a | S t = x ) . Then H ( τ | S t ) ≥ H ( τ | S 0 , A 0 . . . S t ) , where τ is a (random) sub-trajectory beginning in state S t .

Proof. First, letting U = ( S 0 , A 0 , . . . S t -1 , A t -1 ) , observe that:

Next, using the additivity property of expectation and law of total expectation:

Next, we prove sub-additivity of conditional entropy:

where the inequality in the derivation follows by Jensen's inequality. This implies that

Taking expectation of both sides with respect to S t , and using the monotonicity and additivity properties of expectation:

Finally, putting it all together:

D Theoretical Analysis

D.1 Assumptions and Definitions

We decompose a full trajectory of length T into N = T/w non-overlapping sub-trajectories (or chunks), each of length w . Each chunk S i ∈ T ( w ) is defined as

Let the full trajectory be defined as

We define the boundary state X i as the initial state of chunk S i :

which form the backbone of the generative process.

We assume the following factored generative process for trajectories

This implies that the boundary state sequence X = ( X 0 , X 1 , . . . , X N ) forms a first-order Markov chain

Each chunk S i produces a scalar discounted return Y i , defined as

where ˆ R is a learned reward model, and γ ∈ [0 , 1] is the discount factor.

Given a bound R max < ∞ on the absolute reward, we define the maximum per-chunk return bound as:

The cumulative return over the full trajectory is approximated by

and the expected return under the target policy π is:

Definition D.1 (Chunked Behavior Distributions) . Let p ( w ) β denote the true distribution over behavior chunks S i , and let ˆ p ( w ) β be the learned conditional distribution modeled by the diffusion process. These distributions describe how chunks are generated given boundary states:

Definition D.2 (Total Variation Distance) . The total variation distance between two probability distributions P and Q over the same measurable space X is defined as

We now restate the two assumptions presented in the main text for convenience.

Assumption D.3 (Bounded Likelihood Ratio) . There is a constant κ such that π ( a | s ) β ( a | s ) ≤ κ for all s ∈ S and a ∈ A .

Note that this assumption can be easily verified in our experimental setting. Since the action spaces are closed intervals and the behavior and target policy distributions are both represented as truncated Gaussian distributions, the ratio of the two policies is bounded over the action space.

Assumption D.4 (Chunk-wise Model Fit) . The total variation distance between the true chunk distribution p ( w ) β and the learned conditional distribution ˆ p ( w ) β is bounded by some constant δ β > 0 ,

D.2 Analysis of the Bias

We begin by bounding the total variation distance between the true target distribution p ( w ) π and the guided model ˆ p ( w ) π .

Lemma D.5. The total variation distance between the guided model ˆ p ( w ) π and the true target distribution p ( w ) π satisfies

Proof. By the definition of total variation distance

Using the reweighted form of each distribution

and applying the bound on the likelihood ratio (Assumption D.3):

This completes the proof.

Let the total variation distance between the true target distribution and the guided diffusion model be denoted by

By Lemma D.5, we have the bound

We now derive a bound on the absolute bias of the estimated return when sampling chunks from the guided model ˆ p ( w ) π instead of the true target distribution p ( w ) π .

Lemma D.6 (Expectation Difference Bound via Total Variation) . Let p and q be two probability densities on a probability space X . Let be the supremum norm of a bounded function f : X → R , and let:

Then

Proof.

This completes the proof.

Lemma D.7 (Marginal TV Bound via Conditional TV) . Let p ( x | s ) and ˆ p ( x | s ) be conditional densities over chunk x ∈ T ( w ) , given state s ∈ S , and let µ ( s ) denote the marginal distribution over s . Then

In particular, if TV( p ( s ) , ˆ p ( s )) ϵ for all s , then

· | · | ≤ TV( p, ˆ p ) ≤ ϵ.

If TV( p ( · | s ) , ˆ p ( · | s )) ≤ ϵ uniformly, the integral is bounded by ϵ .

Theorem D.8 (Bias Bound for STITCH-OPE) . The bias of the return estimate under the guided diffusion model satisfies

Proof. The return estimator is:

Thus, the bias is:

∣ ∣ For each chunk i , Y i depends only on S i , with marginal distributions ˆ p ( w,i ) π and p ( w,i ) π under ˆ p π and p π , respectively. By Lemma D.6 and Lemma D.7

Since | ˆ R ( s, a ) | ≤ R max , the per-chunk return is bounded:

Using Lemma D.5, we know that TV( p ( w,i ) π , ˆ p ( w,i ) π ) ≤ δ π , Thus we have

Summing over chunks:

Thus:

This completes the proof.

Corollary D.9 (Bias Bound in Terms of Model Fit δ β ) . Under the assumptions sup i TV( p ( w,i ) π , ˆ p ( w,i ) π ) ≤ δ π ≤ κ w · δ β and sup τ | ˆ J ( τ ) | ≤ R max 1 -γ , the bias satisfies

D.3 Analysis of the Variance

Lemma D.10 (Conditional Independence of Chunk Rewards) . Let X i := s iw be the boundary state at the start of chunk S i , and define:

Assume the generative process satisfies the following properties:

X

0

X

1

X

2

X

N

S

0

S

1

S

2

S

N

-

1

Y

0

Y

1

Y

2

Y

N

-

1

Each chunk S i is generated independently given X i

The return Y i is a deterministic function of S i .

̸

Then for all i = j , the returns Y i and Y j are conditionally independent given the full boundary state chain X 0 , X 1 , . . . , X N ,

Proof. Refer to the graphical model in Figure 4. The nodes X 0 , X 1 , . . . , X N form a Markov chain. Each chunk S i is a child of X i , and each return Y i is a child of S i .

Now consider any path from Y i to Y j . Such a path must go through:

All such paths must traverse through at least one boundary node X k . Since we are conditioning on all X 0 , . . . , X N , and these nodes are non-colliders on every path from Y i to Y j , all such paths are blocked. By the criterion of d-separation (see, e.g. Chapter 8 in [49]), this implies Y i ⊥ ⊥ Y j | X 0 , . . . , X N .

Theorem D.11 (Variance Bound) . Let ˆ p π denote the trajectory distribution induced by the guided diffusion model, and p π the true trajectory distribution under the target policy. Let ˆ J be the return estimator using a learned reward model. Then

where B w denotes the maximum per-chunk discounted return.

Proof. We begin by applying the law of total variance under the guided model distribution ˆ p π

Using Lemma D.10 we have that the chunk-level rewards Y i and Y j are conditionally independent given the boundary states X 0 , X 1 , . . . , X N :

̸

Using this conditional independence, the variance of the total return under ˆ p π factorizes:

To bound the difference in conditional variances, we apply the law of variance

Let us define a bound on the per-chunk return magnitude:

Using Lemma D.6 (Expectation Difference Bound via Total Variation), we have

Applying this with f = Y i and f = Y 2 i , and using the bound | Y i | ≤ B w , we obtain:

We analyze the difference in conditional variances:

This uses the triangle inequality and the identity | a 2 -b 2 | = | a -b || a + b | , along with the bounds | Y i | ≤ B w , ∥ Y i ∥ 2 ∞ ≤ B 2 w , and total variation guarantees from Lemma D.6. Then

We now return to bounding the first term in the law of total variance

Using the bound from the previous step

Taking expectation over ˆ p π on both sides

Now, using the expectation difference bound from Lemma D.6 again:

So

Combining both components

Summing across all chunks:

We can split the sum and factor out constants:

Let us define the chunk-level return variance

Therefore

To complete the law of total variance, we now analyze the second term:

We define the corresponding ideal (true model) version:

Our goal is to bound the variance difference:

where M ˆ p := E ˆ p π [ Z 2 ˆ p ] , m ˆ p := E ˆ p π [ Z ˆ p ] , and similarly for M p , m p .

Insert and subtract a common term:

Each term is bounded by 2 δ π B w , so | m ˆ p -m p | ≤ 4 Nδ π B w . Expand both squares:

Each term (both diagonal and cross terms) is bounded in total variation with sup-norm

B 2 w , yielding 2 2 .

From the bound on the means:

So, the product term:

Combining both:

| | | -| ≤ which yields

Combining the two components from the law of total variance, we conclude:

and the proof is complete.

D.4 Proof of the Bias-Variance Decomposition (Theorem 3.3)

Finally, we can bound the mean squared error of STITCH-OPE.

Theorem D.12. Under Assumption D.3 and D.4, and using the notation of Theorem D.8 and Theorem D.11, the mean squared error of STITCH-OPE is bounded by

Proof. We start by adapting the standard bias-variance decomposition to our setting:

since the last term is zero. Plugging in the bounds of Theorems D.8 and D.11 completes the proof.

E Pseudocode

A high-level pseudocode of conditional diffusion model training in STITCH-OPE is provided as Algorithm 1. A pseudocode of the off-policy evaluation subroutine for a single rollout is provided as Algorithm 2. Empirically, we have found that per-term normalization of the guidance function (line 9) resulted in more consistent performance, and allowed the guidance coefficients α and λ to be more easily tuned.

F Domains

We include experiments on the medium datasets from the D4RL offline suite [13], and Pendulum and Acrobot domains from the OpenAI Gym suite [4]. We set the evaluation horizon to T = 768 for D4RL, T = 256 for Acrobot and T = 196 for Pendulum, and we use γ = 0 . 99 in all experiments. Furthermore, Acrobot uses a discrete action space and is incompatible with our method, so we modified the domain to take continuous actions. Table 5 summarizes the key properties of each domain.

Algorithm 1 Conditional Diffusion Model Training in STITCH-OPE

Algorithm 2 Off-Policy Evaluation in STITCH-OPE

G Policies

D4RL Offline Suite. Behavior and target policies and their trained procedures are described in [14], and the policy parameters are borrowed from the official repository at https://github.com/ google-research/deep_ope (Apache 2.0 licensed). The 10 target policies of varying ability, π θ 1 , π θ 2 , . . . π θ 10 , are obtained by checkpointing the policy parameters θ 1 , θ 2 . . . θ 10 at various points during training. Each target policy network models the action probability distribution π i ( a | s ) using a set of independent Gaussian distributions, predicting the mean and variance ( µ i , σ 2 i ) of each action component a i independently. This allows the score function of the target policy to be easily computed. As discussed in the main text, all policies are derived from the medium datasets in all experiments.

OpenAI Gym. We model target policies π 1 , π 2 . . . π 5 as MLPs and train them in each environment following the Twin-Delayed DDPG (TD3) [8] algorithm. The total training time is set to 50000 steps, and we checkpoint policies every 5000 steps. The behavior policy is set to the target policy π 3 . The complete list of hyper-parameters is provided in Table 6.

Bounded Action Space. Since the action spaces for all domains are compact bounded intervals, we need to restrict the action space of the policy networks during evaluation. We accomplish this by applying the tanh transformation to each Gaussian action distribution and then scaling the result to the required range. Note that this transformation constrains the action probability distribution of all policies to a bounded range, and thus satisfies the requirement of Assumption 3.1.

H Baselines

The following model-free baseline methods were chosen for empirical comparison with STITCHOPE:

Fitted Q-Evaluation (FQE). [22] evaluates a target policy π by estimating its Q-value function Q θ ( s, a ) using a neural network. The loss function for θ is

We follow [57, 53] and learn a target Q-network Q θ ′ ( s, a ) in parallel for added stability. We use the AdamW algorithm [55] for optimizing the loss function in a minibatched setting, with gradient clipping applied to limit the norm of each gradient update to 1. The complete list of hyper-parameters used is provided in Table 7.

Doubly Robust (DR). [19, 37] leverages both importance sampling and value function estimation to construct a combined estimate that is accurate when either one of the individual estimates is correct. First, we define an estimate ˆ Q ( s, a ) of the Q-value function of policy π , and let ˆ V ( s ) = E a ∼ π ( ·| s ) [ ˆ Q ( s, a )] be the corresponding value estimate. We also define ρ t = π ( a t | s t ) β ( a t | s t ) as the policy

ratio at step t . Then, the DR estimator is defined recursively as such that the policy value estimate ˆ J DR ( π ) = V 0 DR . We parameterize both ˆ Q ( s, a ) and ˆ V ( s ) as MLPs and train them using AdamW in a mini-batched setting. Similar to FQE, we also update a target value network to improve convergence. The full list of hyper-parameters is provided in Table 8.

Importance Sampling (IS). [33] evaluates the target policy by importance weighting the full trajectory returns in the behavior dataset, i.e.

It requires access to the target and behavior policy probabilities in order to compute the weighting. Specifically, we use the per-decision variant of IS (PDIS), i.e.

which has lower variance than IS.

Density Ratio Estimation (DRE). [31] estimates the ratio w ( s, a ) = d π ( s, a ) /d β ( s, a ) of the discounted state-action occupancies of the target policy π relative to the behavior policy β . The discounted state-action occupancy of a policy µ ∈ { β, π } is defined as where p ( s t = s, a t = a | µ ) indicates the probability of sampling state-action pair ( s, a ) from µ at time step t . We also tested the variants of DICE [43] but found their performance to be unsatisfactory, so they have been omitted from the study. The target policy value is estimated as

w ( s, a ) is parameterized as a feedforward neural network and its parameters are trained using Adam in a mini-batched setting. Fixed hyper-parameters necessary to reproduce the experiment are listed in Table 9. Additionally, since the method requires a kernel function to be specified, we use a Gaussian kernel k ( x, x ′ ) = exp ( -η ∥ x -x ′ ∥ 2 ) , where x and x ′ are concatenations of the (standardized) state and action vectors. Since this requires setting a kernel bandwidth η > 0 which affects the overall performance significantly, we run this baseline for different values η ∈ { 0 . 01 , 0 . 1 , 1 , 10 , 100 } and report the best performing result (according to log-RMSE).

The following model-based baseline methods were also chosen for empirical comparison with STITCH-OPE. They were chosen to determine the benefits of STITCH-OPE compared to fully autoregressive sampling, i.e. w = 1 , and non-autoregressive sampling, i.e. w = T .

Model-Based (MB). [19, 40] consists of learning dynamics ˆ P ( s ′ | s, a ) , reward function ˆ R ( s, a ) and termination function ˆ D ( s ) trained on the behavior dataset to directly approximate the data-generating

distribution of the target policy, p π ( τ ) . ˆ P directly predicts the next state s ′ given the current state s and action a . Both ˆ P and ˆ R can be found by solving a standard nonlinear regression problem, and ˆ D can be found by solving a binary classification problem trained on termination flags in the behavior dataset. We parameterize all functions as nonlinear MLPs and obtain their optimal parameters using Adam in a mini-batched setting. Once we obtain their optimal parameters, we estimate the target policy return by generating 50 lengthT rollouts from the estimated model, and average their empirical cumulative returns. The necessary hyper-parameters are described in Table 10.

Policy-Guided Diffusion (PGD). [16] takes a generative approach by simulating target policy trajectories using a guided diffusion model. We follow the original implementation by training a diffusion model on the behavior data, using the official implementation located at https:// github.com/EmptyJackson/policy-guided-diffusion (MIT licensed). We then generate 50 full-length trajectories from the model using guided diffusion [18] with the guidance function g simple ( τ ) = ∇ τ ∑ t log π ( a t | s t ) , using which we estimate the empirical return of the target policy. All hyper-parameters for training the diffusion models are fixed as per the original paper and codebase (see Appendix A therein for details). However, we found that the policy guidance coefficient α and guidance normalization both have significant effects on performance, thus we ran PGD for different choices of α ∈ { 0 . 001 , 0 . 01 , 0 . 1 , 1 . 0 , 10 , 100 , 1000 } with and without guidance normalization, and report the best performing result (according to log-RMSE).

I Metrics

Let π 1 , . . . π 10 be the target policies, ˆ J 1 ( π i ) , ˆ J 2 ( π i ) , . . . ˆ J 5 ( π i ) be the estimates of the target policy values across the 5 seeds, and J ( π 1 ) , . . . J ( π 10 ) be the target policy values estimated using 300 rollouts collected by running the target policies in the environments.

The following metrics were used to quantify and compare the performance of STITCH-OPE and all metrics:

Log Root Mean Squared Error (LogRMSE). This is defined as the log root mean squared error using the estimates ˆ J j ( π 1 ) , . . . ˆ J j ( π 10 ) and the ground truth returns J ( π 1 ) , . . . . . . J ( π 10 ) , averaged across seeds j = 1 . . . 5 . Mathematically,

Spearman (Rank) Correlation. This is defined as the Spearman correlation [62] between the estimates ˆ J j ( π 1 ) , . . . ˆ J j ( π 10 ) and the ground truth returns J ( π 1 ) , . . . . . . J ( π 10 ) , averaged across seeds j = 1 . . . 5 .

Regret@1. This is defined as the absolute difference in return between the best policy selected using the baseline policy returns ˆ J j ( π i ) and the policy selected according to the ground truth estimates J ( π i ) , averaged across seeds j = 1 . . . 5 , i.e:

Normalization. In order to compare metrics consistently across environments, we follow [14] and use the normalized policy values:

Error Bars. All tables and figures report error bars defined as +/- one standard error, i.e. ˆ σ/ √ n where ˆ σ is the empirical standard deviation of each metric value across seeds and n is the number of seeds (fixed to 5 for all experiments).

where V min and V max are the minimum and maximum target policy values, respectively.

J STITCH-OPE Training and Hyper-Parameter Details

Diffusion Model Architecture. We follow the configuration used in [18] for training the diffusion model, including architecture, optimizer, and noise schedule. Specifically, we parameterize the diffusion process ϵ as a UNet architecture with residual connections [61], trained with a cosine learning rate schedule [56]. The UNet contains 6 repeated residual blocks, each consisting of two temporal convolutions followed by group norm and a final Mish nonlinearity. The time step embedding is produced by a single fully connected layer and added to the activations of the first temporal convolution within each block.

Diffusion Model Training Hyper-Parameters. The list of training hyper-parameters for the trajectory diffusion model is provided in Table 11.

Reward Function. The reward function approximation ˆ R ( s, a ) is a two-layer MLP with ReLU activations and 32 neurons per hidden layer, and is trained using Adam with a learning rate of 0 . 001 and batch size of 64.

Guidance Coefficients. For Gym domains, we use α = λ = 1 , corresponding to the theoretically justified guidance function in Equation 8, assuming low distribution shift. For D4RL tasks, we use tempered values α = 0 . 5 and λ = 0 . 25 to improve sample stability and regularization, which we found empirically helpful in higher-dimensional settings.

Sub-Trajectory Length. We use w = 16 for Gym domains and w = 8 for most D4RL tasks. For HalfCheetah, we reduce to w = 4 due to the environment's fast dynamics, which caused degradation

in stitching fidelity with longer sub-trajectories. Also to add stability for cheetah, we set the clip denoised flag to True during the backward diffusion process.

K Diffusion Policy Training and Evaluation

Diffusion Policy Loss. We follow [41] and parameterize each target policy π ′ i , i = 1 . . . 10 as a conditional diffusion model ϵ ϕ i ( a k , k | s ) , whose parameters ϕ i are learned by optimizing the behavior cloning objective (compare with (1))

Diffusion Policy Guidance Score. In order to use the fine-tuned ϵ ϕ i ( a k , k | s ) as a guidance function for off-policy evaluation in STITCH-OPE, we use the following equivalence between score-based models and denoising diffusion [9] (extended trivially to the conditional setting)

Specifically, this expression cannot be calculated at k = 0 since σ 0 = 0 using the standard parameterization of diffusion models, so we approximate it at k = 1 and use the resulting gradient in STITCH-OPE.

Implementation and Hyper-Parameters. We implement the diffusion model using the CleanDiffuser package [50] with official repository at https://github.com/CleanDiffuserTeam/ CleanDiffuser (Apache 2.0 licensed). To train the diffusion policies, we first generate rollouts from each of the pre-trained target policies in D4RL [14], and then minimize the behavior cloning objective L ( ϕ i ) above to obtain the diffusion policy parameters. The list of relevant hyper-parameters is provided in Table 12.

L Additional Experiments

L.1 Sensitivity to Guidance Coefficients α and λ

We evaluate STITCH-OPE across different choices of the guidance coefficients α and λ , and plot the resulting trends in Figure 5 for Hopper and Figure 6 for Walker2D. Each plot is generated by applying bicubic interpolation to the grid evaluations of the Spearman correlation and LogRMSE. The optimal coefficient values of α and λ remain consistent across environments. The optimal balance for off-policy evaluation is attained by assigning a moderate coefficient for the target policy score α (i.e. α < 1 ) and a smaller but positive coefficient to the behavior policy score λ , i.e. 0 < λ < α . Recall that λ controls the amount of distribution shift we are willing to accept during guided trajectory generation. λ = 1 is theoretically unbiased, but potentially under-regularized and leads to dynamically infeasible (high-variance) samples. Meanwhile, λ = 0 is often over-regularized and leads to trajectories that are heavily biased towards the behavior policy p β ( τ ) . From the plots, we see that a moderate amount of regularization is optimal (around 25% of the value of α ), which is consistent with regularization in supervised machine learning (i.e., regression).

0

0.25

0.5

0.75

1

/

0.01

0.1

0.5

1

Spearman Correlation (

)

0

0.25

0.5

0.75

1

/

0.01

0.1

0.5

1

Log RMSE (

)

0.2

0.4

0.6

0.8

Correlation Strength

2.2

2.0

1.8

1.6

1.4

Error Magnitude

0

0.25

0.5

0.75

1

/

0.01

0.1

0.5

1

Spearman Correlation (

)

0

0.25

0.5

0.75

1

/

0.01

0.1

0.5

1

Log RMSE (

)

0.0

0.2

0.4

0.6

Correlation Strength

1.3

1.2

1.1

1.0

Error Magnitude

L.2 Sensitivity to Window Size w

To further analyze the sensitivity to w , we evaluate STITCH-OPE across different intermediate values of w , and compare the performance according to LogRMSE and Spearman correlation metrics. As illustrated in Figure 7 for Hopper and 8 for Pendulum, the best performance is consistently achieved using moderate values of w , i.e. w = 8 for Hopper and w = 16 for Pendulum. As hypothesized in the main text, based on our analysis in Section 3.3 and Section 3.4, low values of w provide more flexibility when stitching trajectories and thus promote compositionality, but are more susceptible to the compounding of errors. High values of w are less susceptible to error compounding but at the expense of compositionality and thus less adaptability to distribution shift. In the current ablation experiment, it is clear that the best balance between compositionality and error compounding occurs using moderate values of w , and the greatest deterioration in performance occurs for very small or very large values. It is also important to note that increasing w reduces inference speed due to longer trajectory generations per diffusion step, highlighting a practical trade-off between computational cost and evaluation accuracy.

L.3 Sensitivity to the Choice of Behavior Policy/Behavior Dataset

To examine the sensitivity of STITCH-OPE to the choice of behavior policy or behavior dataset, we ran our algorithm and all baselines on the medium-expert and expert data sets for the Hopper environment. Across all reported results, we set the hyper-parameters to T = 768 and γ = 0 . 99 , which allows for easy comparison between the data sets. The results for the medium-expert dataset are

4

8

32

128

w

0.0

0.2

0.4

0.6

0.8

Spearman Correlation

4

8

32

128

w

2.0

1.5

1.0

0.5

0.0

Log RMSE

2

16

32

128

w

0.00

0.25

0.50

0.75

1.00

Spearman Correlation

2

16

32

128

w

2

1

0

Log RMSE

shown in Table 13, and the results for the expert dataset are shown in Table 14. STITCH-OPE soundly outperforms all other baselines in 5 out of 6 instances (metric-dataset combinations). Furthermore, while the performance of some baseline methods exhibits significant variation between data sets (i.e., FQE, DR, IS, PGD in Log RMSE), the performance of STITCH-OPE remains the most consistent across all metrics and datasets. Finally, we observe a steep decline in performance of the baselines on the medium-expert and expert data sets as compared to the medium data set, which suggests that prior OPE work is not robust to large distribution shifts between the behavior and target policies.

L.4 Sensitivity to Reward Estimation

Recall that the immediate reward function in STITCH-OPE is modeled by a neural network and trained on the offline transitions ( s, a, r, s ′ , ... ) , which can be described as a standard (nonlinear) regression problem. We evaluate the sensitivity of STITCH-OPE to this modeling choice by running STITCH-OPE given the true (unknown) reward function, and comparing this to the original results on the estimated reward function. Table 15 shows the aggregated metrics and the per-target-policy results on the Hopper-medium benchmark dataset. In summary, the accumulated errors from reward estimation have a minimal effect on OPE performance as measured using aggregate metrics, and the error remains stable and low across all target policies.

L.5 Trajectory Visualizations

We visualize and compare trajectories generated by the guided and unguided versions of STITCHOPE and Policy-Guided Diffusion (PGD) [16] against both random and optimal policies. These visualizations highlight differences in the quality of generated trajectories, alignment with target policies, and generalization capabilities across various environments. As shown in Figures 9 and 11, STITCH-OPE closely mimics the target policy behavior. On the other hand, PGD performs poorly, significantly overestimating the performance of the random policy. Figure 10 further demonstrates that STITCH-OPE maintains consistent and robust behavior across policy settings.

L.6 Comparison with Diffusion World Models

The Diffusion World Model (DWM) [10] is an alternative approach that leverages diffusion models for trajectory generation. However, DWMs are not well-suited for the off-policy evaluation problem, as we demonstrate experimentally in Table 17. In particular, DWM generates only future states and rewards, e.g., p ( s 1: T , r 1: T | s 0 , a 0 , g ) , discarding most action information. Consequently, it provides no mechanism for policy guidance and cannot adapt to the significant distribution shift present between the behavior and target policies.

By contrast, STITCH explicitly models the joint state-action trajectory p ( τ ) = p ( s 0 , a 0 , . . . , s T ) ; we therefore effectively mitigate action distribution shift by incorporating policy guidance into trajectory generation (Eq. 3), which steers a 0: T toward the target policy. In Markov settings, the DWM diffusion step for s t depends primarily on s t -1 and marginalizes over the behavior policy at t -1 , so rollouts remain near the behavior occupancy and are harder to steer, making DWM more brittle under action distribution shift.

More generally, while there have been significant advancements in diffusion modeling over the last few years, policy guided diffusion (PGD) [16] is the most representative and recent model-based baseline found at the time of this publication which is relevant for OPE. This is because addressing distribution mismatch between behavior and target policy trajectories is a highly non-trivial problem,

which only STITCH-OPE (and partially PGD) are able to tackle effectively. For a fuller discussion of diffusion-based baselines and other model-based OPE methods, we refer the reader to Section N.

M Computing Resources

Hardware and Software. All experiments were conducted on a local workstation running Ubuntu 20.04 LTS and Python 3.9, with the following hardware:

2× NVIDIA RTX 3090 GPUs (24 GB each)

Behavior Policy

STITCH Unguided

PGD Unguided

Random Policy

STITCH Guided

PGD Guided

Optimal Policy

STITCH Guided

PGD Guided

Behavior Policy

STITCH Unguided

PGD Unguided

Random Policy

STITCH Guided

PGD Guided

Optimal Policy

STITCH Guided

PGD Guided

Intel(R) Core(TM) i9-9820X CPU @ 3.30GHz (10 cores / 20 threads)

128 GB RAM.

Runtime. Each full training of a diffusion model for a D4RL task took approximately 20 hours to complete, depending on environment complexity and rollout length. Each OpenAI Gym task took approximately 5 hours. Each evaluation for a D4RL environment took around 18 hours in total (across all 5 seeds) to complete, and each OpenAI Gym environment took around 6 hours to complete.

N Related Work

Off-policy evaluation plays a critical role in offline reinforcement learning, enabling the evaluation of policies without directly interacting with the environment. OPE has been studied across a wide range of different domains including robotics [52], healthcare [58, 60, 59] and recommender systems [51, 63]. Relevant work includes model-free and model-based OPE approaches, including recent generative methods in offline RL.

Model-Free Methods. Model-free methods, such as Importance Sampling (IS) and per-decision Importance Sampling (PDIS) [33] reweight trajectories (or single-step transitions) from the behavior

Behavior Policy

STITCH Unguided

PGD Unguided

Random Policy

STITCH Guided

PGD Guided

Optimal Policy

STITCH Guided

PGD Guided

policy to approximate returns under a target policy. However, this class of methods suffers from the so-called 'curse of horizon', in which the variance grows exponentially in the length of the trajectory [25, 27]. Doubly Robust (DR) methods [19, 37, 11] further combine estimation of value functions with importance weights, reducing the overall variance. Distribution-correction methods (DICE) [32, 43, 47] and their variants [25, 31] try to mitigate the curse-of-horizon by performing importance sampling from the stationary distribution of the underlying MDP. However, these methods perform relatively poorly on high-dimensional long-horizon tasks [14].

Model-Based Methods. Model-based OPE methods estimate the target policy value by learning approximate transition and reward models from offline data and simulating trajectories under the target policy [19, 21]. These methods have shown strong empirical performance, especially in continuous control domains [38, 46], but they often suffer from compounding errors during rollouts, which can lead to biased estimates in high-dimensional or long-horizon settings [20, 17].

Offline Diffusion. Inspired by the recent performance of diffusion models across many areas of machine learning [15, 9], a new stream of reinforcement learning has emerged which leverages diffusion models trained on behavior data [29, 48]. Janner et al. [18], Ajay et al. [1] train diffusion models on behavior data that can be guided to achieve new goals. Ding et al. [10] uses diffusion models as world models. While successful at modeling the behavior distribution over trajectories

accurately, diffusion world models do not use guidance and thus cannot address the policy distribution shift. Jackson et al. [16], Rigter et al. [35] apply guided diffusion to offline policy optimization by setting the guidance function to be the score of the learned policy, while [64] applies guided diffusion to satisfy added safety constraints. Unlike STITCH-OPE, these works do not use negative guidance nor stitching, which we found leads to unstable policy values when applied directly for offline policy evaluation over a long-horizon. Mao et al. [30] applies DICE to estimate the stationary distribution of the underlying MDP, which is used as a guidance function to correct the policy distribution shift for offline policy optimization. Unlike STITCH-OPE, this work is not directly applicable to offline policy evaluation. Finally, Li et al. [54] introduces a variant of trajectory stitching for augmenting behavior data, but does not apply it for offline policy evaluation. To the best of our knowledge, STITCH-OPE is the first work to apply diffusion models to evaluate policies on offline data.

References

C. M. Bishop and N. M. Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006.

Z. Dong, Y. Yuan, J. Hao, F. Ni, Y. Ma, P. Li, and Y. Zheng. Cleandiffuser: An easy-to-use modularized library for diffusion models in decision making. arXiv preprint arXiv:2406.09509 , 2024. URL https://arxiv.org/abs/2406.09509 .

M. Dudık, D. Erhan, J. Langford, and L. Li. Doubly robust policy evaluation and optimization. Statistical Science , 29(4):485-511, 2014.

D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning , pages 651-673. PMLR, 2018.

I. Kostrikov and O. Nachum. Statistical bootstrapping for uncertainty estimation in off-policy evaluation. arXiv preprint arXiv:2007.13609 , 2020.

G. Li, Y. Shan, Z. Zhu, T. Long, and W. Zhang. Diffstitch: Boosting offline reinforcement learning with diffusion-based trajectory stitching. In International Conference on Machine Learning , pages 28597-28609. PMLR, 2024.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id= Bkg6RiCqY7 .

I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations , 2022.

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

S. A. Murphy, M. J. van der Laan, J. M. Robins, and C. P. P. R. Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association , 96(456):1410-1423, 2001. ISSN 01621459. URL http://www.jstor.org/stable/3085909 .

X. Nie, E. Brunskill, and S. Wager. Learning when-to-treat policies. Journal of the American Statistical Association , 116(533):392-409, 2021.

A. Raghu, O. Gottesman, Y. Liu, M. Komorowski, A. Faisal, F. Doshi-Velez, and E. Brunskill. Behaviour policy estimation in off-policy policy evaluation: Calibration matters. International Conference on Machine Learning: workshop on Causal Machine Learning , 2018.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology , 15(1):72-101, 1904.

G. Theocharous, P. S. Thomas, and M. Ghavamzadeh. Personalized ad recommendation systems for life-time value optimization with guarantees. In International Joint Conference on Artificial Intelligence , 2015. URL https://api.semanticscholar.org/CorpusID:8081523 .

Y. Zheng, J. Li, D. Yu, Y . Yang, S. E. Li, X. Zhan, and J. Liu. Safe offline reinforcement learning with feasibility-guided diffusion model. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=j5JvZCaDM0 .

Peer reviews:
comments: SUMMARY:
This paper proposes a method called Stitch-OPE for evaluating the performance of policies trained on offline data. Stitch-OPE first trains a diffusion model capable of generating sub-trajectories based on data collected by the behavior policy. It then estimates the distribution of the target policy using importance sampling. Finally, it evaluates the performance of trajectories generated under the target policy using a learned reward predictor. This work addresses a key challenge in offline reinforcement learning. However, it may suffer from certain inaccuracies in out-of-distribution (OOD) regions.

STRENGTHS AND WEAKNESSES:
### Strengths
1. The experimental results are relatively strong.
2. The paper provides detailed experimental settings.

### Weaknesses
1. The baselines in Figure 2 and Table 2 are inconsistent. Figure 2 labels the baseline as "MBR", while Table 2 uses "MB".
2. The content illustrated in Figure 1 does not seem to be directly related to Off-Policy Evaluation.

QUESTIONS:
1. What is the direct relationship between Figure 1 and Off-Policy Evaluation?

2. How does Stitch-OPE handle the OOD (out-of-distribution) issue? Intuitively, since Stitch-OPE trains a reward predictor using behavior data, wouldn't it lead to significant errors in evaluating the performance of the target policy if the trajectories generated by the target policy fall into OOD regions?

LIMITATIONS:
This method may suffer from certain inaccuracies in out-of-distribution (OOD) regions.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to address the off-policy evaluation (OPE) problem using STITCH-OPE, which leverages the denoising diffusion for long-horizon OPE in high-dimensional spaces. The authors claim two technical contributions: (1) the paper proposes a new guidance score function which can avoid over-regularization and (2) generates long trajectories by stitching partial trajectories together. The authors also provide experimental results on the standard D4RL and OpenAI Gym benchmarks and show advantages compared with existing methods on some metrics (mean squared error, correlation, and regret).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. First, the paper studies an important problem, off-policy evaluation, which is critical in domains where deploying a new policy directly might be risky and better understanding on its impact beforehand using behavior data and policy is necessary.\n\n2. The proposed negative behavior guidance is novel and integrating both the target and behavior policy guidance. The theoretical evaluation error upper bound result is encouraging by removing the exponential term in the previous approaches.\n\n3. The experimental design is reasonable with convincing benchmarks and baselines. The empirical results validate the authors' hypothesis. \n\nWeaknesses:\n\n1. I'm not fully convinced by a claimed innovation/contribution, i.e., the \"sub-trajectory stitching using a sliding window\". If my understanding is correct, the authors are using a sliced history trajectory as the diffusion input and output a sliced window future trajectory. I think this is also proposed in literature, e.g., the diffusion policy paper [7]. How is the method in this paper different existing method? More clarifications would be appreciated.\n\n2. Does the authors try different time horizon under the same benchmark? For example, in a D4RL setup, try T from 100, 200, to 700, and (I believe some readers would be curious to) see how the proposed approach work as the horizon becomes longer, compared with baselines. This can also be helpful to better validate and understand the theoretical error upper bound.\n\nQUESTIONS:\nThe authors stress \"long-horizon\" settings, while I understand they are more challenging than \"short-horizon\" cases, but I'm trying to understand how long are the trajectories in the experiments. The authors mention \"T = 768 for all D4RL problems\", does that mean 768 seconds for trajectory (i.e., frequency is 1HZ), or the frequency is not 1HZ?\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSTITCH-OPE introduces a model-based off-policy evaluation method for high-dimensional, long-horizon RL tasks. It leverages guided diffusion models pretrained on behavior data to generate target policy trajectories by \"stitching\" short sub-trajectories end-to-end. It subtracts the behavior policy’s score during guidance to mitigate distribution shift and over-regularization, and splits generating the whole long trajectories into stitching a series of short sub-trajectories. The paper provides both theoretical and experimental analysis to support the effectiveness of STITCH-OPE.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-written and easy to follow.\n- The theoretical analysis provides an exponential reduction in MSE, which allows further improvement following its route.\n- It emperically ourperforms all baselines on commonly-used D4RL benchmarks and various target policies, including complex diffusion policies.\n\nWeaknesses: see questions.\n\nQUESTIONS:\n1. Could you detail the specific diffusion sampling algorithm (e.g., DDPM, DDIM) and hyperparameters (e.g., diffusion steps, guidance scale, noise schedule) used for generating sub-trajectories? The paper mentions \"conditional diffusion\" but lacks implementation specifics critical for reproducibility.\n\n1. The choice of $T=768$ for D4RL (vs. the standard episode length 1000) appears arbitrary. What empirical or theoretical justification supports this?\n\n1. The paper follows the protocol described in DOPE [1] to train target policies, using SAC on D4RL medium datasets. As offline RL has advanced significantly since 2021, could STITCH-OPE's effectiveness be demonstrated using higher-performing target policies? For instance, evaluating policies trained with modern offline algorithms (e.g., CQL, IQL, TD3+BC) on \"medium-replay\" or \"medium-expert\" datasets would better assess performance on near-expert policies.\n\n1. The model-based (MB) baseline appears outdated. Given recent progress, could you include a stronger diffusion model baseline like Diffusion World Model (DWM) [2], which also performs trajectory-level modeling suitable for OPE? Similarly, please confirm if other baselines (e.g., PGD, DRE, DR) represent the most current competitive methods or could be updated, as current baselines are generally before 2021. Related work in Section N should also be updated.\n\n[1] Fu, Justin, et al. \"Benchmarks for deep off-policy evaluation.\" ICLR 2021\n\n[2] Ding, Zihan, et al. \"Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning.\" preprint 2024.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents STICH-OPE, a novel diffusion-based off-policy evaluation method for reinforcement learning. The algorithm trains a conditional diffusion model to generate short trajectories of the behavior policy. Using the flexible guidance in diffusion models, it is able to sample trajectories of the target policy nicely. The authors show that STICH-OPE performs better than various baselines through extensive experiments, and also provide a derivation of theoretical guarantees of their method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Provided a rigorous theoretical analysis of the proposal algorithm.\n2. Evaluated the proposed method on various benchmarks.\n3. Overall, the paper is clear and easy to follow.\n\nWeaknesses:\n\n1. Most evaluations are on simple RL environments; it remains to see how the method performs in more complex and realistic RL tasks.\n\nQUESTIONS:\n1. Clarification for guidance during the denosing process. Is the guidance applied at every step during the denoising process? The initial state action pairs are quite noisy; would they cause problems for the target and behavior policy score evaluations?\n2. When target policies are diffusion policies, you mentioned that \n    \n    > Specifically, this expression cannot be calculated at $k = 0$ since $\\sigma_0=0$ using the standard parameterization of diffusion models, so we approximate it at $k = 1$ and use the resulting gradient in STITCH-OPE.\n    > \n    \n    Could you elaborate on what this means? Is the same gradient used throughout the whole denoising process of the STICH algorithm? Or is it just for this last denoising step, you use an approximated gradient, and all the other steps utilize the standard calculations of the score? Additionally, do these diffusion policies generate one-step actions or short action trajectories?\n3. Does the trajectory dataset need to contain rollouts that are similar to the behavior policy and target policies? I’m asking this because training generative models like diffusion models is much more data-hungry than other models (like a neural dynamics model). In scenarios that require OPE (without access to simulators or when gathering new data is costly), if the training of the proposed methods requires rollouts similar to the behavior or target policy, then there’s a question of why one would still do OPE when they could already generate rollout data.\n4. How sensitive is the algorithm to the selection of the behavior policy? In other words, do the return values agree when choosing different behavior policies to do the OPE process?\n5. For the visualization, Figures 9 and 11. What is the optimal policy? Does it refer to the target policy being evaluated? For the OPE runs, such as STICH or PGD, are the visualizations rendered with the states’ trajectories or the action trajectories (rollout in sim)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}