Statistical Guarantees for Offline Domain Randomization
Authors: Arnaud Fickinger
Abderrahim Bendahi
Stuart Russell
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Domain Randomization
Sim-To-Real
Primary area: reinforcement learning

Abstract: Reinforcement-learning agents often struggle when deployed from simulation to the real-world. A dominant strategy for reducing the sim-to-real gap is domain randomization (DR) which trains the policy across many simulators produced by sampling dynamics parameters, but standard DR ignores offline data already available from the real system. We study offline domain randomization (ODR), which first fits a distribution over simulator parameters to an offline dataset. While a growing body of empirical work reports substantial gains with algorithms such as DROPO, the theoretical foundations of ODR remain largely unexplored. In this work, we cast ODR as a maximum-likelihood estimation over a parametric simulator family and provide statistical guarantees: under mild regularity and identifiability conditions, the estimator is weakly consistent (it converges in probability to the true dynamics as data grows), and it becomes strongly consistent (i.e., it converges almost surely to the true dynamics) when an additional uniform Lipschitz continuity assumption holds. We examine the practicality of these assumptions and outline relaxations that justify ODR’s applicability across a broader range of settings. Taken together, our results place ODR on a principled footing and clarify when offline data can soundly guide the choice of a randomization distribution for downstream offline RL.

Published as a conference paper at ICLR 2026
STATISTICAL GUARANTEES FOR OFFLINE DOMAIN RANDOMIZATION
Arnaud Fickinger UC Berkeley
Abderrahim Bendahi École Polytechnique
Stuart Russell UC Berkeley
ABSTRACT
Reinforcement-learmin (RL) agents often struggle when deployed from simulation to the real-world. A dominant strategy for reducing the sim-to-real gap is domain randomization (DR) which trains the policy across many simulators produced by sampling dynamics parameters, but standard DR ignores offline data already available from the real system. We study offline domain randomization (ODR), which first fits distribution over simulator parameters to an offline dataset. While a growing body of empirical work reports substantial gains with algorithms such as DROPO (Tiboni et al. 2023), the theoretical foundations of ODR remain largely unexplored. In this work, we cast ODR as a maximum -ilum---iiiioodd estimation over a parametric simulator family and provide statistical guarantees: under mild regular- ity and identifiability conditions, the estimator is weakly consistent (it converges in probability to the true dynamics data grows), and becomes strongly consistent (i.c., it converges almost surely to the true dynamics) when an additional uniform Lipschitz continuity assumption holds. We examine the practicality of these as- sumptions and outline relaxations that justify ODR's s applicability across broader range of settings. Taken together, our results place ODR on principled footing and clarify when offline data can soundly guide the choice of a randomization distribution for downstream offline RL.
INTRODUCTION
In recent years, RI has achieved many empirical successes, attaining human-level performance in tasks such as games (Mnih et al. 2013] Silver et al. (0166), robotics (Kalashnikov et al. 2018) Schulman al., 2015), and recommender systems (Afsar et al. 2021 Chen et 2021). Yet, RL algorithms often require vast amounts of training data to learn effective policies, which severely limits their applicability in real world settings where data collection is expensive, time-consuming, or
unsafe (Levine et al., 2020 Kiran et al. 2020).
Sim-to-real transfer tackles this problem by learning in simulation and transferring the resulting policy to the real world (Sadeghi & Levine 2016 Tan et al., 2018 Zhao et al.) 2020). However, although simulation provides fast and safe data collection, inevitable discrepancies between the simulated dynamics and the real world, commonly termed the sim-to-real gap, typically induce drop in performance upon deployment.
One of the most widely-used approaches to bridge this gap is domain randomization (DR). Rather than training on a single fixed simulator, DR defines family of simulators parameterized by physical factors (e.g., masses, friction coefficients, sensor noise) and at the start of each episode randomly samples one instance from this family for training. DR has enabled zero-shot transfer in robotic control (Tobin et al. 2017; Sadeghi & Levine 2016), dexterous manipulation (OpenAl al. 2018) and agile locomotion (Peng et al. 2017).
'Equal contribution.
Work done during internship at UC Berkeley.
arnaud. tickingerrberkkeeeyy edu
"abderrahim wtterraaaim.beddaaie....e apo @onnnaahhpoyytecchn ytechniiii www.iiuuee... edu Trussell @berreeeeee edu

---

Published as a conference paper at ICLR 2026
Despite this empirical track record, the choice of how to randomize is fundamental challenge. In the original form of DR (Tobin et al., 2017 Sadeghi & Levine 2016), broad uniform ranges that look reasonable for every parameter are chosen. While recent theoretical work (Chen et al. (022) shows that such uniform DR (UDR) can indeed bound the sim-to-real gap, the bound unfavorably scales in O (N3 log(N) ( with respect to the number of candidate simulators, in part because UDR ignores any data already available from the target system.
In contrast, Offline Domain Randomization exploits static dataset from the real environment before policy training to fit a sampling distribution that concentrates on plausible dynamics while remaining stochastic. Empirically, ODR variants such as DROID (Tsai et al. 2021) or DROPO (Tiboni et al. 2023) recover parameter distributions that explain the data and yield stronger zero-shot transfer than hand-tuned UDR. Yet, to the best of our knowledge, ODR lacks principled foundation: we do not know (i) whether the fitted distribution converges to the real dynamics as data grows, nor (ii) how much actually reduces the sim-to-real gap compared with UDR.
Our Contributions:
We formalize ODR as maximum-likelihow estimation over parametric simulator family and prove weak consistency: under mild regularity, positivity, and identifiability assumptions, empirical maximizers converge in probability to the population maximizers.
Weak consistency (Section 4).
Strong consistency ( Adding single uniform Lipschitz continuity assumption on the likelihood, we upgrade convergence to strong consistency: the ODR estimator converges almost surely the true parameter when it is uniquely identified.
A ssumptions in practice: discussion and relaxations (Section (). We analyze when the assumptions hold and provide drop-in relaxations and diagnostics: replacing i.i.d. by strict stationarity and ergodicity for the, weakening mixture positivity via logarithmic tail condition, and giving simple sufficient conditions that imply the uniform Lipschitz requirement.
2 RELATED WORKS
Sim-to-real transfer
The sim-to-real gap has led to extensive research in sim-to-real transfer. Early works exploited system identification or progressive networks adapt controllers online (Floreano et al. 2008 Kober et al. 2013), while more recent efforts have focused on purely offline training in Bughh-idelit simulators. Although zero-shot transfer has been demonstrated for specific settings such as legged locomotion (Peng et al. 2017), dexterous manipulation (Chebotar et al.) 2018 OpenAl al. 2018) and visuomotor control (Rusu et al., 2016) noticeable performance gap persists in unstructured environments. Similar ideas have been explored in autonomous driving (Pouyanfar et al.
2019
Niu et al. 202I).
Domain randomization Domain randomization (DR) varies environment parameters at every training episode with the goal of producing policies that generalize across the induced simulator family. Vision-based DR first showed zero-shot transfer for quadrotor flight from purely synthetic images (Sadeghi & Levine (016)), and dynamics randomization extended this success to legged robots and manipulation (OpenAl et al., 2018). To avoid manual tuning of randomization ranges, online methods adapt the DR distribution using real-world feedback. Ensemble-based robust optimization and Bayesian optimization techniques refine parameters via real rollouts (Rajeswaran al. 2016
Muratore et al., 2020), while meta RL further accelerates adaptation (Clavera et al.) 2018: 8: Arndt al. all.|[2019 2019). However, these require repeated and potentially unsafe- hardware interactions during training.
Offline domain randomization A growing line of work aims to find the best strategy to perform domain randomization from fixed offline dataset, obviating any further real-world trials. DROID (Tsai et al. 2021) tunes simulator parameters using CMA-ES (Hansen Ostermeier 2001 Hansen 2006) with the 2 distance between single human demonstration and its simulated counterpart as objective function. BayesSim (Ramos et al., (201)) trains a conditional density estimator to predict a posterior simulator parameters given offline off-policy rollouts. Most recently, DROPO (Tiboni
2

---

Published as a conference paper at ICLR 2026
et al. 2023) introduces likelihood-base framework that fits both the mean and covariance of Gaussian parameter distribution by maximizing the log-likelihood of the offline data under a mixture simulator. This approach recovers rich uncertainty estimates, handles non-differentiable black-box simulators via gradient-free optimizers, and outperforms DROID, BayesSim and uniform DR in zero-shot transfer standard benchmarks without any on-policy real-world interaction.
Theoretical analyses Let M be the number of candidate simulators and H the horizon length. Chen ct al. (2022) modeled uniform DR as a latent MDP and proved that the performance gap between the optimal policy in the true system and the policy trained with DR scales ( log( (log((M) H in the case where the simulator class is finite and separated and O M log (M) in the finite non-separated simulator class case. Other works have studied the infformmttionthhorettii limit of sim-to-real transfer (Jiang 2018), PAC-style guarantees via approximate simulators (Feng al.) 2019) and generalization in tich-observation MDPs (Zhong et al., 2019 Krishnamurthy et al., 20. But none address the statistical benefits of offlin DR. Our work bridges this gap by providing the first consistency proofs and fnniee gap bounds for offinn DR, thereby unifying empirical successes and theoretical understanding single framework.
3 PROBLEM SETUP AND ODR FORMULATION
Episodic MDPs We consider the episodic RL setting where each MDP corresponds to M = (S,A, R, 81). S is the of states, A is the set of actions, PM: S x A (AA) is the transition probability matrix, R: S x A + [0, 1] is the reward function, H is the number of steps of each episode, and 81 is the initial state step 1; we assume w.l.o.g. that the agent starts from the same state in each episode.
At step [H). the agent observes the current state sn. S, takes action ag E A, receives reward R(sh, 0,), and moves to state 8h+1 with probability PM(Sh+1 Sh, an). The episode ends when state $1+1 is reached.
A policy π is a sequence { ff where each nr maps histories traj, = {(s1 .1.....sh.. to action distributions. Denote by TT the set of all such history-dependen policies. We denote by VM, S + TR the value function at step h under policy on MDP M, i.e., (s) ==
EM Ei=h ((s, an) sh. = s
We use JM denote the optimal policy for the MDP M, and VMA to denote the optimal value under the optimal policy at step h.
We fix simulator class U : RR) of candidate MDPs that share (S,A, H, but can differ in PM via the physical parameter vector E. The unknown real-world environment is M MM.- U. We assume full observability and that the learner can interact freely with any M E L/ in simulation, but never observes & directly.
Sim-to-real Transfer Problem Given access to the simulators in U4 the goal is to output policy m that attains high return when executed in the real-world MDP M We quantify performance via the sim-to-real gap which is defined as the difference between the value of the learned policy и during the simulation phase (or training phase), and the value of an optimal policy for the real world, i.c.
Gap(T) V
Domain Randomization Domain randomization specifies prior distribution over parameters = and thus over U. Sampling & ~ D at the start of every episode induces a latent MDP (LMDP) whose optimal Bayes policy
"DR == arg max RE V [
HEll
In practice approximate HiR with any RL algorithm that trains in the simulator while resampling { ~ V/ each episode.
'The original paper derived looser bound, Section A.I for tighter derivation. "Since the policy ι is allowed to he non Markovian, this quantity can be defined using the history H
(S1, 8h. } as follows: E.,. E/E R(st, az)
Eug B.M
D R(s,, as) Hn
3

---

Published as a conference paper at ICLR 2026
Uniform Domain Randomization
Offline Domain Randomization
sample and train episode on Mg
sample and train one episode Mg
fit. Pg
D
offline data
6
8 ~ Uniff(mins {max]
~ Po(E) fitted from
Figure 1: Conceptual comparison between Uniform Domain Randomization (left) and Offline Domain Randomization (right).
Offline Domain Randomization ODR assumes an offline data D {(s, a,, a,, )= of i.id. transitions collected in the real system M under some unknown behavior policy. The aim is to estimate distribution p* over E that explains the data and can later be used for policy training. We restrict (E) (y, Σ and learn ø by maximum likelihood:
p" (E) arg max PotE)
II (S1,5))441)ED
Eg [Pe(st42 Se. 0e)
(1)
arg max PolE)
Σ
log [Berp(()) [Pe(Se41 st,
(2)
We justify that this formulation well-motivated in wSection A.2 Finally, train a policy with the learned distribution:
TODR arg max Eg-up [M (s:));
well
expecting FODR transfer with lower gap thanks the data-informed parameter distribution.
conceptual comparison between Uniform Domain Randomization and Offline Domain Randomiza- tion is illustrated in Figure 1
4 WEAK CONSISTENCY OF THE ODR ESTIMATOR
4.1 TECHNICAL ASSUMPTIONS
Before stating the theoretical guarantees for ODR, we introduce some mild assumptions of regularity and identifiability that will be usefull for our proofs.
The following assumption assures that Pe is regular in the following sense. Assumption 1 (Simulator Regularity). There exists o o-finite measure on S and a constant K < co such that for all E and (8, a, 8') Pę(ds' s,a) = P (saa) 0 S S (. SK
and & + pe(s' s,a) is continuous.
Notice that when S is finite, and À the counting measure on S, then the first assumption clearly holds with K the mass probability to depend continuously on & in order to verify Assumption I Another case where this continuity holds is the Gaussian case (s'(s,,, a) Nis'; ([f))+ + B(E)a, C(E)), where A(E). B(E), C(E) are matrices that vary continuously in &.
because P5 (s' a, Ps((s') s, a) < 1. In this case, suffices for
"The Gaussian parameterization over & is only modeling choice, not a mathematical necessity. Any other parametric family for Pa that satisfies our upcoming assumptions could he substituted without changing the arguments.
4

---

Published as a conference paper at ICLR 2026
Assumption 2 (Parameter-Spae Spacee Compactness). We fit (1. in {n E 0 Σ mmax/] [ where E is compact, hence & is compact in the product topology.
This is a natural assumption, since in practice one always has prior bounds on each physical parameter, yielding known compact search region.
Furthermore, we assume that all the transitions that appear in our dataset correspond to positive mixture probability. More formally, Assumption (Mixture Positivity). There exists some constant e > O such that the induced kernel
4 s, a) == (E- [Pe(s') ss a)] = / Pe(s s,a) Po(dE),
satisfies go s, a) 2 c 0 for every (s, a, 5') D and every @ E t
This guarantees that every transition in the dataset lies within the support of the simulator under the learned domain randomization distribution, so the log-likelihood always well defined.
Furthermore, we assume that the only mixture distribution which exactly recovers the true transition kernel the degenerate distribution concentrated at the true parameters 5" Assumption 4 (Identifiabilityy. Let μ be the dataset's distribution. If for y-almost every (s, a) (- S, a) = Pg- (- s, a). then Ø = (5*.
4.2 NOTATION FOR ODR
Throughout this work, corresponding lowercase letter, to denote its probability density (or mass) function.
capital letter, P, to denote probability distribution, and the
We define the empirical and population log- lieliikelihoods
N
(N(#) = N
Σ o(),,)),
L(o) == Ex-Pe- [a(X,))) d)],
(5)
where X; (s;, 0g. sf) is the i-th transition in D, and X = (s, a.) is a generic transition. The function defined by
a(x, == log logg s, log / pe(s' s,a)po(E)dE.
(6)
4.3
MAIN THEOREM
The first lemma proves the uniqueness of the maximizer of the population log-likelihood L. A detailed proof of this lemma can be found Section B
Lemma (Uniqueness of the Population Maximizer). Under assumptions) 3 and the population log-likelihood
L(Φ) E(s.a,))) [logqs($ s.a)]
where (s's' [s,))= = -
PE(S' s, (dEE), has the unique maximizer o² (u*,E", (5*, 0).
We now state our first consistency result for ODR.
Theorem 1 (Weak Consistency of ODR). Under Assumptions I 1 and any measurable
maximizer @N E arg max (Ø) satisfies 6x
P
SE#
N
Theorem D guarantees that with a sufficiently large offline dataset, ODR recovers distribution arbitrarily close to the true parameter e*
The following lemma is particularly strong: it establishes uniform convergence in probability of Ln-
Lemma 2. The function è + L(Ø) is uniformly continuous & and furthermore
sup (N( L()) N
(7)
5

---

Published as a conference paper at ICLR 2026
The proof of this lemma relies uniform law of large numbers (ULLN) -in particular the ULLN for Glivenko- classes from Newey& & McFadden (1994)- and is deferred to Section B In contrast, the ordinary law of large numbers only guarantees that for each fixed é one has Lv( L( in probability, i.e., [x((0) - [(6)] - 0 for that particular c. This pointwise convergence does not imply that supee4 (6) [()) - 0, which is exactly what the ULLN provides. Uniform convergence over all Φ E & is crucial to control the behavior of the empirical maximizers and hence establish the consistency of our estimator.
The following lemma formalizes a uniform separation property: any parameter & lying outside an e-neighborhood of the true maximizer o" must have its population log- likllikiiiood at least 1) > below L(Ø").
Lemma 3. Let o³ be the unique maximizer of L. We have
Ve > 0, 0,3n(() > 0, Vé E + 11
Φ 2 = L( ((6) 2 m(() > 0.
(8)
The proof of Lemma deferred to Section
B
Proof of Theorem 7
We consider sequence of measurable maximizers 2N arg max LN( Let E > 0 be a fixed positive real number. Our goal is to prove that
P (1|xx 55| 2
O. N
(9)
Using Lemma we conclude that there exists some 17 0 such that Va E tb if 1| øll then I(Ø") (()) > O. Now, let Fr, be the event
En = {sup (o(()) - [()) < n/33
(10)
then under E, if 110 ~1 2 we have
[Ø²) = (x())) I( + I( 2 - - I(07) I(0)) 2 -n/3+ I(Ø"), (11)
since under E,, - (Ø) L()) 2 -n/3, similarly
-(0))= 2 (()) + = L()) - (v(x) + (o) + 2 (o) L(6) + (o( + (12)
and combining these two inequalities gives LN (Ø") ) 2 Lo) + n/3. This proves that, under En:
ΦN E B(Ø", ,c) == 1 + 16 - 1- < e) thus {1ēN l¹ll 2 c) C Dy which yields
p(lŌN 011 2 E) < P suppLL(()) L()) 2 n/33
By Lemma 2
0.
(13)
The result is weak consistency statement CŌN o* in probability). In Section to almost-sure convergence by adding a Lipschitz regularity assumption.
5 we strengthen this
5 STRONG CONSISTENCY UNDER UNIFORM LIPSCHITZ CONDITIONS
While Theorem guarantees that the ODR estimate converges in probability to the true parameter distribution, many practical settings one desires a stronger, almost sure guarantee. Intuitively, strong consistency asserts that, with probability one, the estimated distribution will converge exactly to the true one as more offline data is observed. In this section we show that, under an additional Lipschitz continuity assumption on the log- likelihhhoood function, ODR enjoys this almost-sure convergence property.
5.1 ADDIOTIONAL ASSUMPTION
The key extra ingredient is a uniform control over how rapidly the single step log-likelihood a(x, の) can change as we vary the distributional parameter (n,, E). Formally:

---

Published as a conference paper at ICLR 2026
Assumption (Uniform Lipschitz Continuity). There exists constant L < such that for every transition I = (s, a, s') and all o, vo E ④ we have aaxx 6) a(x, 4)) - L w 212
This condition ensures that the family {a(,, () Q } is equi-Lipschitz, which -together with compactness of ạ- yields a uniform strong law of large numbers. In turn, this uniform convergence of the empirical log-likelihood to its population counterpart underpins the almost sure convergence of the maximizers.
5.2 MAIN RESULT
We can now state our strong consistency result:
Theorem 2 (Strong Consistency of ODR). Under Assumptions to let 2N E arg ma LN (v( be any measurable maximizer of the empirical log-likelihood, then
ΦN
a.8.
= ())
(14)
N
Le., almost surely the estimated distribution collapses exactly onto the true simulator parameters.
The distinction between convergence probability and almost surely subtle but meaningful: almost-sure consistency implies that, except on set of histories of measure zero, soon as enough data collected the optimizer will never stray from the true maximum again. In contrast, convergence in probability only assures that large deviations become increasingly unlikely.
The heart of the proof the following uniform strong law, which follows from empirical process arguments once have the Lipschitz control:
Lemma 4 (Uniform Strong Law). Under Assumptions 110
the empirical and population log-
likelihoods satisfy supse (Ø) L(d)
8.8.
N
Lemma tells that with probability one the worst-case difference between the finite-sampi sample objective and its ideal limit vanishes. Once this uniform convergence in hand, classical arguments on continuity and compactness show that the maximizers converge almost surely.
Proof (Sketch), We first show supse# (N()) - L(ø)
a.s.
by verifying for each > 0
N
that ΣN P(supaco (Ø) - L()) > 2c) < co. By compactness of & there is finite c/ L-net f Φx} } so that Lipschitz continuity gives (x( (x.)) + ((()) L(ex)) I whenever
|| - æill S e/T. Hence
sup (op][ > c U (Tx( I(())
(15)
and Hoeffding's 's inequality yields
Ne²
P([Nx()))) - I(baal > 6) < 2exp
(16)
2M
where M == max [log K), |log So P(sup, [o( - L(の) 2e) < 2K exp(-c which
summable in N. Borel-Cantelll lemma then gives uniform almost convergence. Finally, the event of uniform convergence one repeats the identification neighborhood argument of Theorem to
conclude ΦN + 6* almost surely.
Full details of the proof are deferred to Section C but the key takeaway is that the Lipschitz assumption upgrades our earlier in probability consistency to the far stronger almost sure statement, giving robust guarantees for ODR even in worst case data realizations.
7

---

Published as a conference paper at ICLR 2026
5.3 A NOTION OF G-INFORMATIVENESS
The strong consistency yields the following.
Lemma 5. Lere of (uN, Ex) - (5*,0) then almost surely there is No so that for all N 2 No. Pow (B(E*,c) 1
Proof of Lemma
Fix 6 > 0 and let ZN ~ NNNN En). Then P(IIZx - E*1 2 () S P(IIZN
pul 2 5)
1 2 5) By Chebyshev's inequality, P(I N -uwl 2 5) < (c/2)"
=1
tr(E.)
(c/2)² Hence Po (B(5", <)) = - P(IIZN - 5*1l 2 2 1- 4tr(Ew] P(I - 5- 2 c/2).
5²
As (uN, - (5*, 0) a.s., we have I1 - -- - 0 and tr(EN) - 0, the right hand side tends to almost surely. Hence Pa (B(5*,e) + almost surely.
The lemma states that when the estimator (UNN Ex) converges almost surely to the true mean with vanishing covariance, the Gaussian distribution fitted by ODR eventually assigns more than half of its probability mass to any fixed e-ball around g*. In other words, ODR is so informative that the learned randomization concentrates near the real world. This observation motivates a general, model-agnostic notion of "informativeeess" for ODR, applicable beyond the Gaussian setting.
Definition 1 (a, e-Informativeness of ODR Algorithm A). Let a E (0, 1) and > 0, algorithm A is a, e-informative if there exists almost surely No 2 such that for all N z No, running A on any collection D {[ss, as. $)) of i.i.d. transitions (from the real system) produces an ODR distribution ΦN such that
Paw (B){),))) 2a
We say algorithu A is a-informative if A is a, e-informative for any >> 0.
Under this language, Lemma 5 states that the Gaussian ODR procedure from Section 3 is a- informative for every 1. When the simulator class E is finite, a-informativeness is equivalent to the almest-sure existence of an index No such that, for all N 2 No, the fitted distribution assigns at least a mass the singleton {E*}, that is, PaN. (5*) a.
ASSUMPTIONS: PRACTICALITY, VIOLATIONS, AND RELAXATIONS
6.1 THE LLD. ASSUMPTION
The i.i.d. assumption the offline dataset D holds whenever the offline dataset is collected using a fixed, stationary behavior policy .- 8). This assumption is stronger than needed for our weak consistency result: we invoke it only to apply a uniform law of large numbers at the end of the proof of Lemma [] As noted after Lemma 2.4 in Newey & McFadden (1994), the same conclusion holds (even for dependent data) for ergodic and strictly stationary sequences {X; (s,, a, s()} which means that the joint distribution of the vector (X....., Xx+m) does not depend on i for any m. This much weaker than the i.i.d. assumption and is satisfied whenever the offline dataset is collected by fixed behavior policy (not necessarily stationary policy). In practice, weak consistency should therefore hold broadly.
6.2 THE MIXTURE POSITIVITY ASSUMPTION
Assumption is a strong requirement: holds if and only if inf, inf& go(x) > 0, i.e., the density is uniformly bounded away from zero over both x and 6. This excludes common light-tailed families (c.g.. Kaussian-like)) for which inf, (x) O. For weak consistency, however, Assumption 3 can be relaxed:
Lemma 6 (Relaxation of Assumption 3. Weak consistency of ODR still holds if Assumption is replaced by the following tail condition: there exists eo > such that for all e E (0)
inf (X) sc) S log(1/e)²
(17)
8

---

Published as a conference paper at ICLR 2026
This assumption is strictly weaker than uniform positivity. The key point is that, to apply the uniform law of large numbers from Newey & McFadden (1994) in the weak-consin consssccns proof, it suffices to have integrable envelope d(æ) with a d(æ) for all Ø, rather than uniform bound in (2e, o), the above tail control yields such an envelope. The proof is deferred to Section D.1
6.3 THE UNIFORM LIPSCHITZ CONTINUITY
Assumption Assumption 5 is not immediately interpretable. We give a simple sufficient condition under which it holds:
Lemma 7 (Sufficient Condition for the Uniform Lipschitz Continuity Assumption). Suppose the following holds for every a,))
1. The function (s( 8, a, is twice continuously differentiable (of class C²),
2. There exists two constants G1 > and G2 >> such that [Veee(' |s,a)] S G1 and VVpe(s' |s,a)] G2.
then Assumption h with I G1 + G2/2
complete proof appears in Section D.2 This sufficient condition is easy to interpret because depends only the simulator's 's transition kernel pę. In practice, is satisfied whenever the simulators are governed by smooth physics.
6.4 THE IDENTIFIABILITY ASSUMPTION
Assumption 3 is coverage condition on the dataset: requires that any mixing Gaussian distribution that reproduces the transition kernel on the state-action pairs observed in D must equal the degenerate Dirac mass at the true parameter. Intuitively, the dataset must visit state-action pairs that are informative about This is information-theoretiallll theoretically minimal: method can distinguish parameters that are observationally identical on supp(μ)).
In the of partial coverage, naturally define the identified under coverage μ follows:
Ø = fØ e d ga(- s, a) = Pe- (- a) for a.e. (s,a))
(18)
It follows from this definition and the proof of Lemma that Lemma 8. The following holds:
2 arg max L(ø).
(19)
Using this notion of identified set, we can generalize Theorem when we relax Assumption 4 as follows:
Theorem 3. Under Assumptions
I 2 and
the following holds, Any measurable maximizer
Ty E arg max (Ø()) satisfies dist ON 2 N
This theorem states that under partial coverage, our estimator does not select a single parameter but converges to an identified set of parameters that are observationally indistinguishable on the state-action pairs visited by the data. The proof deferred to Section DD3] The proof of this theorem is very general. In particular, even in the misspecified case where M & Ut we still have ΦN øt E arg max, L(ø). A more detailed discussion of the this case is deferred to Section D.4
Without any additional assumptions, the only structural result that we can derive on the identified set is:
where dist is the distance defined by dist( Q) inf ver -

---

Published as a conference paper at ICLR 2026
Lemma 9 (Upper Hemicontinuity of Q.). Under Assumptions 7 and The identified set Q is non-empty and compact and the correspondence μ + 2 is upper hemicontinuous with respect to total variation.
The proof of this lemma uses Berge's 's Maximum Theorem and is deferred to Section D.3
In short, this lemma says if we perturb the dataset's 's coverage only slightly (in notal-variation distance), the set of maximizers cannot "jump" to faraway region: any limit of maximizers for the perturbed coverages remains maximizer the limit coverage (upper hemicontinuity). Intuitively, modestly adding or reweighting offline data will not create spurious, distant optima, it keeps the solution set nearby, and, as coverage includes more informative state-action pairs, typically makes tighter.
The main limitation is that, without additional assumptions, we cannot provide a quantitative radius for this a Lipschitz-type bound how much it can move when coverage changes.
CONCLUSION
In this paper, we present a rigorous framework for ODR, bridging the gap between empirical success and theoretical understanding in sim-to-real transfer. By casting ODR as maximum likelihood es- timation over parametric family of simulator distributions, we proved that, under mild regularity conditions, the learned distribution is weakly consistent, concentrating on the true dynamics as the offline dataset grows. With the addition of a uniform Lipschitz continuity assumption, we further established strong consistency. Beyond these core results, we scrutinized the practicality of the assumptions and provided diagnostics and ceixxations-replcciin i.i.d. with stationarity/ergodiiity for the ULLN, weakening mixture positivity via logarithmic tail condition, and giving checkable smoothness criteria that imply the uniform Lipschitz requirement- guuireement-thereby justifying ODR's s applica- bility across a broader range of settings. By demonstrating that offline logs are not merely passive datasets but a powerful tool for principled domain randomization, we hope our formulation and analysis provide insight that paves the way for safer, more data-efficient sim-to-real pipelines in robotics, autonomous vehicles, and beyond.
'A set set-velued map F is upper hemicontinuous at 20 if, whenever 2n, - xo and yn E F(x.) with yn. + then y F(ro). Equivalently: for every open U with F(zo) C U, there exists neighborhood V of zo such that F(x) C U for all V.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Mohammad Mehdi Afsar, Trafford Crump, and Behrouz H. Far. Reinforcement learning based recommender systems: A survey. CoRR, abs/2101.06286, 2021. URLhttts::/axxi http/xxxxiivor..//.
abs/2101. 06286
Karol Arndt, Murtaza Hazara, Ali Ghadirzadeh, and Ville Kyrki. Meta reinforcement learning for sim-to-real domain adaptation. CoRR, abs/1909. 12906, 2019. URL http://arxiv..og/ abs/1909. 12906
Pierre C Bellec and Cun-Hui Zhang. Second order stein: Sure for sure and other applications in 04121
nigh-dimensional inference, 2020. /ul.:tttss::/axxi.orrr////a
Claude Berge. Topological Spaces: Including Treatment of Multi- Functions, Vector Spaces, and Convexity. Oliver and Boyd, Edinburgh, 1963. Translated from the French by E.M. Patterson.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac. Nathan D. Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. CoRR, abs//810..eo 180.00587, 201 8. /Rtpbtte://arxiv......
abs
05687
Xiaocong Chen, Lina Yao, Julian J. Mc.Auley, Guanglin Zhou, and Xianzhi Wang. A survey of deep reinforcement learning recommender systems: systemattic review and future directions. CoRR, abs/2109.00 093400909044, 2021. /tt.tttpe////ar.....rr./aa/aaaaa
03540
Xiaoyu Chen, Jiachen Hu, Chi Jin, Lihong Li, and Liwei Wang. Understanding domain randomization for sim-to-real transfer, 2022.
Ignasi Clavera, Anusha Nagabandi, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt: Meta-leaming for model-based control. CoRR, abs/ ht//1803.111 1347, 2018.
URL http:://aa//iiiee org/ ha/p//ass/111/// 11347
Fei Feng, Wotao Yin, and Lin F. Yang. Does knowledge transfer always help to learn a better policy? CoRR, abs/1912.02286, 2019. URL/http:/aarxvv.or
02986
Dario Floreano, Phil Husbands, and Stefano Nolfi. Evolutionary Robotics, pp. 1423-1451. 01 2008.
ISBN 978-3-540-23957.4, doi: 10. 10.1007/978-3-540-300000
Nikolaus Hansen. The CMA Evolution Strategy: A Comparing Review, pp. 75-102. Springer Berlin Heidelberg, Berlin, Heidelberg, 2006. ISBN 978-3-540-32440 doi: 10. 10.1007/3-540-324444444 URL
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 95)):159-195, 2001. doi: 10.1 10.1162/00336560150000000000
Nan Jiang. Pac reinforcement learning with imperfect model. Proceedings of the AAAI Conference on Artificial Intelligence, 32, 04 2018. doi: 10.1609/aaai. 1..09//aaii.2221.1..... v3291.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scal- able deep reinforcement learning for visedn-based robotic manipulation. CoRR, abss1806. 10293, 2018. URL /tttp:://xxii....///// org. als 1806.
Bangalore Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Kumar Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. CoRR, abs/2002.00444, 2020. URL htl_ttpps//aax....rr//////222222
Jens Kober, J. Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The Interna- tional Journal of Robotics Research, 32: 12:1243-1274, 09 2013. doi: 10.1177027736649114497711
Akshay Krishnamurthy, Alckh Agarwal, and John Langford. Pac reinforcement learning with rich observations. Advances in Neural Information Processing Systems, 29, 2016.
S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79 86, 1951. doi: 10. / URL ht https://dooo /oo. wrt./o..1...1..... aoms 11177296777
11

---

Published as a conference paper at ICLR 2026
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020. URL https://
arxiv.org/ wttp://////.//.......0.0
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URL/cto::/aaxxiv wwtp:oo/gga/aaaaaaa 5602
Fabio Muratore, Christian Eilers, Michael Gienger, and Jan Peters. Bayesian domain randomization for sim-to-real transfer. CoRR, abs/2003.02471, 2020. URL http::://aazivv.or////bbb
2003. 02471
Whitney K. Newey and Daniel McFadden. Chapter 36 large sample estimation and hypothesis testing. volume 4 of Handbook of Econometrics, pp. 2111-2245. Elsevier, 1994. doi: https:
URL http::/wwwwwsccccceeeeeeeee com/
Haoyi Niu, Jianming Hu, Zheyu Cui, and Yi Zhang. DR2L: surfacing corner cases to robustify autonomous driving via domain randomization reinforcement learning. CoRR, abs/2107.11762. 2021.
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. CoRR, 0bss/180..00177, 2018, URL /ett::/aarxivorgg
abs 00177
Xue Bin Peng. Marein Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. CoRR, abs/1 100./170..0637, 2017. URL URLhttp:
Samira Pouyanfar, Muneeb Saleem, Nikhil George, and Shu-Ching Chen. Roads: Randomization for obstacle avoidance and driving in simulation. In 20/9 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1267-1276. 2019. doi: 10.1109/CVR...009.
00166.
Aravind Rajeswaran, Sarvject Ghotra, Sergey Levine, and Balaraman Ravindran. Epopt: Learning robust neural network policies using model ensembles. CoRR, abs/1610.01283, 2016. URL
http://arxi:: wttp://wwggp//p//o//o/////// 01283
Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain randomization
via probabilistic inference for robotics simulators. CoRR, abs/1906.01728, 2019. URL http
7/arxiv. //arxiv.org/abb/11 9006. 01728
Andrei A. Rusu, Matej Vecerík, Thomas Rothörl, Nicolas Manfred Otto Heess, Razvan Pascanu, and
Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. ArXiv, abs/ 610.0426. 2016. UtLpttss: api.
Fereshteh Sadeghi and Sergey Levine. (Srd) Real single-image flight without a single real image. CoRR, abs/1611.04201, 2016. URL http:
04201
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR,
2015. http:::p//aaaxv.o////aaaaa
05477
David Silver, Aja Huang, Christopher Maddison, Arthur Guez, Laurent Sifre, George Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-489, 01 2016. doi: 10. 1038/nature1166
Charles M. Stein. Estimation of the Mean of a Multivariate Normal Distribution. The Annals of Statistics, 96)): 135 1151, 1981. doi: 11211/////11173445555
htRpst:s: h/tp/.ooogrg... /
10. 1 4
12

---

Published as a conference paper at ICLR 2026
Jie Tan, Tingnan Zhang. Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. CoRR, 2es/1804.10332,. 2018. http:////aaxxi wtt.aawwwwwwv......oaaaa..baaa abwa 10332
George Tauchen. Diagnostic testing and evaluation of maximum likelihood models. Jour-
nal of Ecomometrics, 30(1):415-443, 1985. ISSN 0304-4076. doi: https://doi.or//10
10160030440066633911196 URL http::/ww...iieccccreeccccc wwtp///tt...oocc/ccccceeoeeeeeee atticee/AL//3040768590333
Gabriele Tiboni, Karol Arndt, and Ville Kyrki. Dropo: Sim-to-real transfer with offline domain randomization, 2023. URL htt.t:ttp///oomm /tatttp://ariv.... /ttp:///////oooo////aaabb////a// wtw.aa////..00000 .82344
Joshua Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel.
Domain randomization for transferring deep neural networks from simulation to the real world.
CoRR, 0oss/703300607,, 2017. URL /
06907
Ya- Yen Tsai, Hui Xu, Zihan Ding, Chong Zhang, Edward Johns, and Bidan Huang. DROID: minimizing the reality gap using single-shot human demonstration. CoRR, abs/2102. 10033 2021. URL /tttps://arxiv.oo org/ abs/ 2102. 11003
David Williams. Probability with Martingales. Cambridge University Press, 1991.
Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep
inforcement learning for robotics: a survey. CoRR, abs/2009.13303, 2020. URL https
//axxii.org//bs/2200000 13303
Yuren Zhong, Aniket Anand Deshmukh, and Clayton Scott. PAC reinforcement learning without real-world feedback. CoRR, abs/1909. 10449, 2019, URL http: owg/abg......ooooo.... /109 1909
10449
13

---

Published as a conference paper at ICLR 2026
A ADDITIONAL PRELIMINARIES
A.1 REFINED ANALYSIS OF THE UNIFORM DR SIM-TO-REAL GAP
In this section, tighten the worst-case sim-to-real gap bound in the finite, d-separable setting originally proved by Chen et al. (2022).
In the proof of Lemma of the paper, Inequality (47) yields with probability least do.
(M so, ao) PM, (s' 80: ao)
ng
Σn In S'E
log((oo))) log (2/ón) - Vno log (2/60) /c 20.Sn
(20)
The objective is to find setting of parameters that guarantee with probability at least MH*
PM (s'
so, An)
Σ In
0.
PM. (s'
80: aa)
S'EW
sufficient to have the right term positive Equation
20 i.e.
noo²
log(1/a) 2no log (2/6))
V // log (2/5o) /c- 2aSno > 0.
(21)
Setting
8a
6u
(the same values as the paper), this term becomes
85
MH
nnd2
log(1 /a) 2ng log (2/6o) v no log (2/6o) /c- 20Sno
(22)
2
no8²
82
SS
82
log(
2no log (2MH) v no log (2MH)
4 110
(23)
2
82
nod2
8.
log(
2no log (2MH)
v log (2MH)/C
(24)
82
82
4
85 82
Vno
/no
log(
/ (2MH) Vlgg (2MH) /c
(25)
4
82
hence the condition 21 becomes equivalent to
/no
log(2MH)
(V2log
885
+
(26)
52
83
vc)
or, equivalently,
no
84
16
log (2MH)
(V2log
88
82
+
Ve
1
(27)
Thus, there exists valid setting that satisfies condition 21 which can expressed as
8²
1
no log((HH) log2((//''
(28)
8S
MH
84
for some constant 6o > 0 sufficiently large.
With this setting, the result of the Lemma of the paper becomes
DM2 DM log(MH log
Bihol < o
(29)
84
The proof of Theorem 5 of the paper is not affected by the new expression of no and gives
(81,,((11) ( s O + D) = 0 DM2 (M)
(30)
84
14

---

Published as a conference paper at ICLR 2026
Combining this result with Lemma 1 of the paper leads
( (DAM (og(MH) log (S/62)
(DM
Gap(#bn, U) O
(31)
84
This shows that the regime where H and M are relatively large, the ((M log" (M))) bound of Chen et al. (2022) be tightened O(M² (o)(MH)).
A.2 INSIGHTS INTO THE ODR OBJECTIVE
In this section, we explain why the formal ODR problem in Equation 2 corresponds exactly to fitting the simulator parameter distribution that maximizes the likelihood of our offline dataset.
We seek the parameter á of the distribution Po(E) that maximizes the probability of observing the triples (8,, a,, 84) of our dataset, i.e., to solve
ai arg max (88,, 1Ø),
(32)
This probability corresponds to P (n ((5,, ai sf)} () and since the data is i..dd, o² can be rewritten as follows
N
arg max II
P a(((ss,ai,s/) Φ)-
(33)
Now, we can approximate P([(s;, as s/} E i.e,),
Φ) by the expected transition probability over all
N
arg max
I
E Pe (P((
s a.))
(34)
Since the logarithm increasing, this is equivalent to
Ø² arg max
Σ log Ee [P((5]] 8 s,,a;)],
(35)
which recovers exactly the empirical log-likelihood objective stated Equation
B
OMITTED PROOFS IN SECTION 4
Proof of Lemma
7
We have
L(Φ)= = E(e,a) Ey upe (1aaa log (e-moce)(PE((' (1,))() s,a))].
(36)
We rewrite the inner expectation as follows
log (s s)
Ey' (Il,a)
log go' s, a))
Per (s' s, + log log p (8 ss
(37)
Notice that
J go(d' (o,6'\s,))\(d5))- J I pe(s' (s)))))d)E)((d))
(38)
and using "ubini-Tonell 's theorem, it follows,
[ 1x5S/ { | (s) /
Jee3 po(EE) / Ju pe(s' asa))(dss).
(39)
15

---

Published as a conference paper at ICLR 2026
Since P5(: s,a) and p& probability densities, their total mass is 1, which yields
go(d' 8, a.))(dd))= WES Jes
[ Po(dE) 1.
(40)
Hence go(- | 8, a) is a probability density, and one can rewrite L( using Kallback-Leibler (KL)
divergence (defined in Kullback & Leibler (155]] follows
L(6) (s.a)) - DeL (pe. (. a) s, a)) Ex [logpe- (5' s, ا]
(41)
(s,) (- DxL (pe- (. 8, (.)ol 8, a.)]
H(")),
(42)
where ((5*) = (s (ls ups (.1a,a) Pe Pe s, a)] independent of Φ, and for fixed (s, a).
DEL (pc- 8, a. 8, a)))) 2 0 with equality and only if p.- (- s, a) = go s,a).
Hence, for all @ 4 ((o) < (E"), and
(s, Dx (pe- ( s, a) (9o( s, a))] =
(43)
For almost every (s, a), DxL (pe: (. 8, (. 8, s,a)) =0
(44)
For almost every (s, a). per 8,a) gol. |8,a)
(45)
(5", 0),
(46)
where the last equivalence follows from Assumption
This concludes proof.
Proof of Lemma We begin by stating and proving few intermediate lemmas that will simplify the proof.
The following lemma states that convergence of & implies convergence in distribution of Pa-
Lemma 10. Let {dr} {(uns Enn} E & sequence that converges to & (H, 2) (i.e. [| - All - 0 and 12:.. - Sillo + 0). Then Pon, converges weakly to Po (Pon = Pa).
Proof of Lemma 10
We denote
G = N En),
G= G=N(u,))
(47)
The characteristic function of G, is
{ 1 [Innt) tERd.
PG.. (t) exp
T Nn
(48)
2
For every fixed IRd, we have
1 T (t)
Paa (t) n
exp
it p 2
(49)
By Lévy's continuity theorem (see Williams
1991) we have Pa.. Po.
Notice that the result holds also in the case where Σ 0. In that case, (t) = exp (it v)) which is the characteristic function of the degenerate distribution 5, AN(,,,
This result will be used to derive the continuity of the function @ a(x, Φ) in the following lemma.
Lemma 11. For some fixed x (s, a, and ④. the function
+ alx, Φ) = log pe(s'1 8, 8,)))s(E)dE
continuous on +
16

---

Published as a conference paper at ICLR 2026
Proof of Lemma 11 For E, we denote (5) pe(s' s,
h2 is continuous on E (by Assumption D and bounded on E, because
vy E =, (E)l = ( s, a.) - M
(again by Assumption !
Let {Øn] = (( _m) } E g a sequence that converges to Φ == (1, E). Notice that
( s,a)ps.. (E)dE = Pon [/2]]
(50)
and since Po.. Ps. (from Lemma 10 then EPo.. [he]] n Ep [hz]-
We then compose by the logarithm function which is continuous on (0). log Ep,,, [1x] log Ep, [ba] Equivalently,
This yields
a(z, Qn) n a(x,
(51)
This concludes the proof by the sequential characterization of continuity.
Now prove Lemma 2
We have (N( N E a(X,, Φ). where X; (s;, ais st) iid Pg..
& compact (by Assumption 2) and by Lemma 11 for each x, + alx, ()) is continuous on $
Additionally, the following holds for any ø E +
Ja(z, 6) log pe(s' s,))p
(52)
By Assumptions and have 1g (5 (E)) S K. Hence
|a(z, Ø)1 s M == max flog cl, |log K}}.
(53)
Since L(Ø) Ex-P. [a(X, 2) this implies (by Lemma 2.4 from Newey & McFadden (1994)
which is implied by Lemma from Tauchen (1985) that L is continuous on @ and thus uniformly continuous since @ is compact by Heine-Cantor theorem. Furthermore,
sup [o(()) -()) #E
(54)
N
Proof of Lemma 3 Let é >> 0. We consider the set defined as follows
Cer.e == &Ø || o"I 2 e).
(55)
Cor. compact because it can written as the intersection of a compact set
C
f (Ie, (().
(56)
is where denote far d || ø* Indeed, T is compact (by Assumption and for ((c, oo)) closed as the inverse image of the closed set [c, oc) by the continuous function far.
The function g Φ + L(ó*) L()) is continuous (by Lemma 2p on the compact set Crr... hence by the extreme value theorem, attains its minimum on Co+,e ,e in some & 4.
Thus
ve Cor -(0")-L(00
- L(0) 2 g(0).
(57)
By Lemma 1 2 0 on & and
g(0) -0 - Φ = $*
(58)
Since Ø * 0² (because & E Ca. d. we have g(0) > 0. Thus, the lemma holds with the choice of n(c) ((و) 0.
17

---

Published as a conference paper at ICLR 2026
C
OMITTED PROOFS IN SECTION 5
Before proving Lemma 4 we state and prove a few preliminary lemmas.
Notation for Strong Consistency
We define the diameter of ④
by
Diam ( sup
1| 3/11-
(59)
Φ.vE#
We begin with the following technical lemma, which gives an upper bound on the number of closed balls of radius L/L needed to cover Φ.
Lemma 12. Let 0 < < 2 Diam(() L, and let Ne be the minimum number of closed balls of radius
r = required to cover 4. Then
N₂ s 44( (Diam())
(1) Lid
(60)
Proof of Lemmal 12 We construct a sequence Φ1, 22 ... in 4 satisfying
Vi j, ||
ااهه >
(61)
This process must terminate after finitely many steps; denote the final index by K. Indeed, if it were infinite, then compactness & would yield convergent subsequence of for), ). contradicting equation 61
By construction,
c U B(oL.))
(62)
k=1
for otherwise we could pick some Φ & U B(ok, r) to continue the process, contradicting the definition of K. Hence N. S K.
Next, observe that the closed balls B(du, r/2), h = 1..... K, are pairwise disjoint: if there were Φ B(o,, r/2) n B(0j.r/2) r2) with i * j. then
| - - - || øll + |1 أل1l s r/2 + rr2
(63)
contradicting equation 61
Moreover, for each k;
T(Ox.T/2) C (( Diam ( +r2),,
(64)
since if || - dall S r/2 then || - Φ1ll I b dall ll + 11 Φ1l |l s r/2 + Diam
Thus
K
U B(ou.//2)
C B(0i, (iam($)- - -r/2).
(65)
and by comparing volumes of disjoint balls in IRd we get
(r (4/2)a74/2
(Diam () + +r/2)
K T( +1)
(66)
r(g +1)
Hence
N < K < (1+ 2 Diam(
Diami ())
<
(67)
where the final inequality < 2 Diam( L.
L
In the following lemmas establish sufficient condition for the almost convergence.
18

---

Published as a conference paper at ICLR 2026
Lemma 13. Ler (At)e21 be sequence of events. We have
P
Ae
-0
ve P(Ar) - 0.
(68)
F21
Proof of Lemma 13 If P (Uez1 At)
1, then for all & 1, we have clearly P(As) -
P (Urg) Ar) = 0 and P(Ae) 0.
If P(A)) 0 for every f 2 0, then we have by Boole's inequality,
P
A
P(Ar) = O.
(69)
121
E20
Lemma 14. Let (Zu) be sequence of random variables. We have
Ve>a O,
Σ B21
=
Zn a.s. n2-00
0.
(70)
Proof of Lemma 14
We have by definition of the almost sure convergence, Zn a.a.
0 if and only
P
lim Zn = 0 70
Equivalently,
P (Ac > 0, En vm 2 n, 1. <c)= =1
(71)
and since we can replace by any sequence of positive real numbers that converges to 0, the previous condition equivalent to
P ( Zull x (
-1.
(72)
m2n
Considering the complementary event, this is equivalent to
(unu
1 | 1
p
(73)
(21n21m2n
Using Lemma 13 in order to have the almost sure convergence of Zn to O, it is sufficient to prove that
Zul 2 1 = 0. Now suppose that for all > Enz1 P P(|Zal 2 c) 00. co.
n nnn
3A 1,
(74)
R21 m2
This implies that for all 2 1, we have
Zn 2 1)
Σ P n2l
< 00.
(75)
Using Borel-Cantelly lemma, this implies that
(
111 =
AA 11
2ul
(76)
m21m2n
This concludes the proof.
19

---

Published as a conference paper at ICLR 2026
Lemma 15. For any fixed ó E Φ, and e > 0. we have
Ne²
P((L((0) -( 2 2exp
(77)
2M2
Proof of Lemma 15
We have
Lx(6)= = N Σ o), L(d) = Ex-p- [a(X,
(78)
where X, XI,,...... XN lid Pe..
We already establish that |a(x,, Φ0) - M (see 53 hence
P[(LN(0) 2()))2))=
Σ
(a Φ)
Ex~p (a(x,,)) 2 Ne
(79)
2(NNc)2
$ 2xxp
(80)
(2,7))
N
$ 2xxp
(81)
2M2
where Inequality 80 results from Hoeffding's 's inequality.
Proof of Lemma
Let < < 2DL. We cover + by N, closed balls of radius e/L, i.e.
B(
DL d
for some .I,..., ON. E O, where N₂ < 4d
by Lemma 12
For all Φ E 4, there exists an integer 1 < k(Ø)= & N. such that ||0 - Ok(O)) from Assumption that
I r, hence it follows
Vx,
- ((x,) L 110 2n(e) |1 S sr
(82)
We have
T. - I(d)] - (Ø)) - IN ((n(())) - Lx (o)(o)) - [((r(()))) + [((n())) I(Ø)
The first term can be bounded using Inequality 82 as follows,
Lx(Ø) L
1 N
Σ
(((X,,)) o(X)) Se.
(83)
Similarly, the third term satisfies
((br())) - L()) = | oM()) - Exa(X, Ex (a(X, - a(X,,) Se,
where the first equality holds from Jensen's 's inequality.
Putting these inequalities together yields
sup Φc
(o) L()) <
max I....NN
LN(0,)) L(ø.) 2c.
(84)
20

---

Published as a conference paper at ICLR 2026
This implies that
P sup dEG
(N(6) - L(6)]
3e
SP
max
[N((,))
10(0L(()2 2c)
(85)
i=,...,,
N,
P(]LN(0)) [(0i) 20
(86)
.e²
exp
(87)
2M
1=1
Ne²
2N, exp
(88)
2M
DL
d
Ne²
S 2 4a
exp
(89)
2M
where Equation (86) uses union bound, Equation (87 follows from Lemma 12
follows from Lemma 13 and the last inequality
This yields when N DO
1
P
sup [T) I(0)] 2 3c = 0E4
(90)
N2
This assures that ΣNZ1 P (supsen (Ø)) L(Ø) 2 3e) < xc, which gives by Lemma 14
sup
[N( L(ø) 3.3. 0.
(91)
sc#
N
Proof of Theurem
2 By the preceding lemma we have the event
={w> sup ( - L(Ø)] N 07) =1.
P Oo = w
(92)
@E
Fix w So and, let e > 0. From Lemma there exists 77 0 such that
Váb E +; | +ll 2 = I( T(()) 2 m
(93)
Since w E 2g there exists a random index No(w, 17) with
sup de@
Lx(Ø)- - L(d) L(d)) < n/3 VN > No(w,n).
(94)
Take N > No 1)) and suppose, towards a contradiction, that ([Ō - 601 2 c. Then, using
equation 93 equation 94
(w)) <
[(6)) - - < (dd))).
(95)
which contradicts the maximality of @w(W)) Hence, for all N 2 No(w,n)
(xŌ(w)) -
([][)) 6611EE. <E
(96)
This implies that Qo c (w: GN((w) N-000 }+
Since P(Lo) we conclude
ΦN
a.s. 0²
(97)
N-00
21

---

Published as a conference paper at ICLR 2026
D OMITTED PROOFS IN SECTION 6
D.1 RELAXATION OF THE MIXTURE POSITIVITY ASSUMPTION
Lemma 6 The weak consistency of ODR still holds we replace Assumption 3 with the following (weaker) assumption:
P inf go(() (z) [c) - (log(E))² for e sufficiently small.
(98)
Proof of Lemma We start by proving these two elementary lemmas.
Lemma 16. For any almost surely non-negative random variable Z, i.e., P(Z 2 0) = 1, we have
Z = |0
P(Z 2 a) da.
(99)
Proof of Lemma T6
We have
P(Z 2 a)
E m2a da
(100)
1 122 dP(z)
(101)
12a da
dP(z)
(102)
0=0
da dP(=)
(103)
xdP(2)
(104)
0=
E[Z].
(105)
where Equality equation 102 follows from follows from the non-negativity the random variable Z.
's theorem, and Equality equation 105
Lemma 17. For any positive function
(0, o0) defined on some interval IR we have
sup log (xx = log sup f(x).
(106)
Proof of Lemma 17 For any x E I we have by monotonicity of the logarithm function
log fx) < log sup f(x),
(107)
hence, sup,, log fx) < log sup, f(x). Furthermore,
f(x) log f(x) sup log f(x)
(108)
and taking the supremum over x E 7 yields sup,, (xx S esu, log [(x), thus
log sup fxx I sup, log f fxr),
(109)
which concludes the proof.
Note that the only passage of the proof of Theorem I in which we use Assumption 3 when we derive uniform bound on the function in Inequality 53 More precisely, we proved that
Vz, Vo E 4 |a(x, の) M max | log(M))
(110)
While this is sufficient to apply Lemma 2.4 from to bound a(x, Φ) by some quantity d(x:) that is independent of Φ and integrable in
from/rrwer Newey McFadden (1994), this lemma only require
22

---

Published as a conference paper at ICLR 2026
We have
la(2, 6) = log q (z)
(111)
(log 2 (x)) + (log (x))
(112)
where =t and = denote respectively the positive and negative parts of
We have
(log go (x)) max(0, log (x()))
(113)
max 0. 1o log { pe(o' (8,))p(()dd))
(114)
[log(M)). Thus, the first term of equation 112 is bounded by log(M)) which is independent of Φ and integrable in x.
and by Assumption 1 (s [ s, a) < M, hence (log (loggo(z)) (z))
Furthermore,
(log (x))) max(0, - log (x(x))
(115)
0, log (x(x)
(116)
1
0, 0, log
(117)
q (x)
1
0, log sup (
(118)
max
0, 0,g
(119)
inf 1s (x:)
where Equality equation 118
The last quantity is independent of o, so we only need it be integrable order for the weak consistency result to hold.
follows from Lemma 17
Since this quantity is non-negative, Lemma 16 yields
E
max
0, log inf @ (x(x))
P max 0, 0,
inl gp(xr)
22) da
(120)
0
P
log inf go(x)) 2 da
(121)
00
P inf (o( S se da,
(122)
and hence we only need to have the convergence of this integral. The integrand is bounded (between O and 1), the integral is always convergent (0, 1]. Hence, it is sufficient have the convergence of the integral on [1, o0), e.g., one sufficient condition might be
inf (x( $ Seaa
for a sufficiently large, a
(123)
equivalently,
p inf (z) <
(log(E))² for sufficiently small.
(124)
Notice that Assumption 3 implies this condition, since it implies that inf (x) > 0 and hence for sufficiently small e > 0 we have
(0)=<())) 005 (log(e))²*
P inf as (x) sc)
(125)
23

---

Published as a conference paper at ICLR 2026
D.2 SUFFICIENT CONDITION FOR THE UNIFORM LIPSCHITZ CONTINUITY ASSUMPTION
In this section, we prove a practical sufficient condition for Assumption 5 More formally, the following holds:
Lemma 7 (Sufficient Condition for the Uniform Lipschitz Continuity Assumption). Suppose the following holds for every x (8, 0, s')
1. The function + (s(s') s, a) twice continuously differentiable class class C²),
2. There exists two constants 11 > and G2 > such that [VePe(' | ss S Gi and (e ss S G2.
G1 + G2/2
then Assumption hlllds with L
Before proving this result, state and prove technical lemma that use in our proof. Lemma 18. For any >> 0, the logarithm function log is --Lisschitz |c, oc).
Proof of Lemma 18 Let and y be two real numbers such that S y. We have
[log(y) log(2)) log(y) log(x) log
1/
log
1
y/ x
1
(126)
x
and since follows
1
(log(y) log(z)] S - (y - x) -2)
- x
(127)
Notice that this result can also be proved using the mean value inequality.
Proof of Lemmal 7 Our goal is to prove that under the two assumptions of Lemma 7 we have
VΦ = (μ, E), o'= = (,'EE' Σ') E 4 Va, |a(a, の) - a(x, Φ') - L 110 o'1l2-
(128)
First, notice that using Lemma 18 and Assumption 3 we have
- - fa(o')
la(z, Φ) a(z, 0')l -
log(f
(129)
where we used the notation f=( == (s / a) = Eg (pe(s' 8, a)). Hence, it is sufficient to prove that (Ø) fe(d') < L ||0 6' for every x for some constant L ->0 0.
We start by treating the case where V and >' are non-singular.
Case 1: non-singular covariance matrices. In the case where is son-singular,
fe(Ø) = f (EW E)dg,
(130)
where h2(()) = (5 (8' s, and W E) == (2m) del (E))(((2))))))))))) (E)= ( - Since 1a + TT and E E-1 are continuously differentiable respectively on Rdd and C.,.(R).. then the function Φ + N Σ) is C as long as Σ > 0 with
V -
(131)
and using the matrix-calculus identities
dlog det E tr(E-1dd))
d(E-l) -E- -
(132)
24

---

Published as a conference paper at ICLR 2026
compute
dlog. = d - log det E ] (1) n)TT 15- [!]
(133)
- dE) ) nn d(==-)(( - 11)
(134)
ir(E (Σ) + 1 - n1) m (d -w).
(135)
Since dN N Wolog.. N, we get
dN = 2 - - (5 n1) tr(E
(136)
Rewriting in Frobenius inner product form,
dV = tr (3 ())(( p)(5 1)
- dΣ
(137)
Thus the gradient is
VxN(E;,, E) N Σ) (E- - -
(138)
On each compact subset K of 1 n ((E, 2) E 0}. we have by the sub-muliipicattivtyy of the norm
V (.V p. E) - M | l lk 2ll ((; μ4, 2),
(139)
ans since the function & + - 2 1 - W μ4, Σ) is continuous on K, it attains its maximum in some point of K, hence, there exists some μn and Eo such that for all Φ E K,
Σ)| - M ز II5 - Holl W[; Mo, Eo), where the right term is integrable in & since Ex~N (EsHo Eo) ||IX - poll] < cc. Furthermore,
(140)
(EV E - M. Σ) (|| 11=--(E - -m)(5 - - || + || (141)
2
and || (5 T can be rewritten as follows
Σ |1 S || 11 |F ( - 10(55 - -1) |
The middle factor
|[E ( 11) ir [(E ME(E TTT (5-N)(E- -m) n)
(142)
= tr [(5 - 11) (E - -1)(E - [4) - [1]]
TT)
(143)
115 -m1l.
(144)
which yields
[hac()V E)1 - 1 MN p, 2) (1 Ile -ml2 - -
(145)
Again, the function & - (| 11 I15 1112 + + -|>~||1) (1)) (E: 14, Σ) continuous on K, attains
its maximum in some point of K, hence, there exists some #n and En > 0 such that for all K,
(e) u, E) S 2 M (1ET in 12 + | NE p1, Ex),
(146)
where the right term in integrable in since the Gaussian distribution has finite second order moment. Using Leibniz integral rule, the function Φ + fa(() is C² and we may interchange differentiation and integration to get
V vo) = (((E)
(147)
ho(E))= (OND--((WWW(EERAA 2) de
(148)
I E)]
(149)
((N E)
(150)
EE-N(( (V))
(151)
25

---

Published as a conference paper at ICLR 2026
where Equation 150)
follows from an integration by part Furthermore,
Vxl=(()=
(152)
1 (E- p)(E - 11) w ------1 ((Eii - 23
(e)
(153)
2
E EE-NN (x) [(5 -1(s 1¹) - -2]] =
E
(154)
E 1/2 B/ ((x(E) (D-1/2(5 -1) (/-1/2(5 - -1)) Id] [a] Ia|I=-1/
(155)
2
E 1- 2/-1/EEENN(LLE) [=x Ia)] Σ- 1/2
(156)
2
where 1 is the unique positive definite square root of 1- 2 == シー (/ and
g(z) ==
hz(E) = hr (E + u). Using the Iterated Stein formula (Bellec & Zhang have
2020
Stein 1981 we
B [o(x)(2x] - Ia)] = =Ee-Nin,>)
V [=)
(157)
= EE
V (c)
(158)
-E (e)
(159)
Combining this equation with Equation (156) yields
Vzfa(o)
Eg ((.))) [
(160)
Since fa is C when Σ > 0, for any two points Φ, o' E t such that Σ and E' > 0, there is on the segment joining them (and thus E > 0 so that by the mean-value theorem
fz(Ø) f=(ø') (Vafz(0), o- -²).
(161)
In particular
(2( - - (ofz( 110 6'll-
(162)
By assumption (ii) /| S G1 and |. S G2. Hence
[+)] = [ S G1. |V>x/(00|l = 1 (2)] < _
(163)
Assembling the two blocks,
(o) -
f (o') < (G1 + C2) | - 6'1l.
(164)
Therefore fa is Lipschitz in o. with constant T' G G2/2, and by Lemma 18 so is a(x,
log fØ) with constant == G1 + G2/2
General case. For the case where we no longer suppose that Σ and Σ' are non-singular, we use the density of the set of invertible matrices in (R.). More precisely, there exists two sequences of non-singular matrices (Ex), y and (Ely } n such that Ex - E and Σyy + E' when N + co. We
denote &N = (n, Ex) and d'y = xx) The previous result yields
NA 2 0, Væ, Ja(z, 2N) - a(x, d'x)1 )1 < L I1 d'x11.
(165)
and thus, when N + get
Vz, Ja(z, Φ) - a(x, Φ')l < L ||Φ 0'11 where we used the continuity of the function o + a(x, Φ) on +b (Lemma 11 This concludes the proof.
(166)
The first term of the integration by part vanishes since ha (E) Σ) MN((:;4,) E)
16ll+oo
7 Indeed, there exists E [o, 1] such that E = E + (1 - +)E' where E > and E' > 0, thus for any
IRd. Ez tz Σ> (1 - +)2 E'2 > 0.
26

---

Published as a conference paper at ICLR 2026
Notice that even if in many robotic systems have strongly non-smooth dynamics in (s, a) due to hard contacts and friction. However, this non-smoothness concerns the map (s, a) H Pe s, a) (for example, if the state encodes the position and velocity of robot arm, the probability of next-step positive velocity can change discontinuously). However, the map < Ps 8, a) typically remains smooth with respect the physical parameters. Intuitively, if slightly perturb masses, friction coefficients, or gains, we expect the transition probabilities to change only slightly, even though the contact dynamics in (s, a) are themselves non-smooth.
hard contact
D.3 WEAK CONSISTENCY UNDER PARTIAL COVERAGE
Theorem 3 Under Assumptions
7 and
3
the following holds, Any measurable maximizer
ON E arg max (Ø) satisfies dist
27) N
6E4
Proof of Theorem
As Theorem I the uniform law of large numbers holds:
sup
[x I( 2 0.
(167)
Lemma proves that 2 is nonempty and compact.
Fix s > 0 and define the separation (margin) outside the e-neighborhood of O
m()) inf (L(0)) L(Ø) o- O dist(o, (7)) 2 e).
Because L is continuous and &Ø E & dist( 2x)) 2 2} is compact, we have n7))) > 0. By equation 167 there exists sequence of events EN with P(Ex) - such that on EN.
sup (x(Ø) 1(d) m()).
On EN, for any & with dist(o, 21,) and any 6* 2j.
LN(Ø) I L(Ø) + (n() < L( - nT=) (n(E) -(0")-(-' - -3n(E) < sup LN(V),
w
where the last inequality uses LN (1)) 2 - - no maximizer Ln can lie outside e-neighborhood of 0j on EN. Equivalently,
L(ő*) -
m(()) for any & O; Therefore,
dist dist 27, < on EN-
Since P(Ex) + 1 and & > 0 arbitrary, we conclude dist dist 21,) E, 0.
Lemma Under Assumptions 7 and 3 The identified 2 non-empty and compact and and the correspondence + 2 is upper hemicontinuou 19 with respect to total variation.
Proof of Lemma 9
Write
L(b.p) E(S,A)~ Es (1.1 1 aols,a)
where
a, S'),
where dist is the distance "A set-valued map F is upper hemicontinuous at 20 if, whenever 2n, + xo and yn E F(x.) with yn. - then y F(ro). Equivalently: for every open (/ with F(zo) C U, there exists neighborhood V of zo such that F(x) C U for all e V.
defined by dist 2) = v || w|ll
27

---

Published as a conference paper at ICLR 2026
Step 1: Finite-valued and continuity in Φ. We have
sup (a(x, @E
V M for all x (,)
Therefore (fo(s, & M for all (s and Ø, and [(), E R. Moreover, Lemma 11 gives continuity of à + a(x, Φ) for each fixed x. By dominated convergence with the uniform bound M, we obtain continuity (hence upper semicontinuity)
+ L(ø, p)) on $
Step 2: Uniform TV-continuity in p. Let + in total variation. Then, for any á E 2,
/ fo(s,a) (Hn (ds da)
[L mn) L))
(168)
S /
(1) (ds
(169)
S M |l ullTV.
(170)
Taking the supremum over Φ E yields
sup [L (a) L(0: - M 11 1llTV 20000 $E
0.
(171)
Step Joint continuity of L. Let (fhn - (cb, (1) with on - s and pn - 1 TV. Then
LL Mun) L(d, p)1 S [(Dn:Kn) - L(óns 11] + [L( - L(0,
By uniform TV-coniinuty in f (from (a(x, 6) s M),
sup [L,,, (m) L(b,, will S Mlmm p/|TV - 0, WED
n-000
(n,) - L(dn, p1) 0. By continuity in è fixed μ (dominated convergence with the hence [L(nn,
same bound), [L μ)) - L( μ}) + 0. Therefore L( (() + L(ø, p), i.., (6. μ)) + L μ) is jointly continuous.
Hence, by Berge's 's Maximum Theorem (Berge 1963). for each l the argmax set Ø arg maxşe# L(Ø, ()) is nonempty and compact, and the correspondence + 27. is upper hemicon- tinuous (in total variation).
D.4 MISSPECIFICATION AND REPRESENTATIVE MDPs
In this subsection, we study the case where we no longer suppose that the true dynamics M belongs the simulator class U. Given Markov kernel P: x A + ASS). write M(P) for the MDP H1, and we denote the value of policy æ at the initial state 8j by
V (51) (s1).
We use (= (p(5) {s')) for the total variation distance between distributions q over S.
Lemma 19 (Value stability under kernel perturbations). Let S ASS) be two transition kernels defined on the same state-action space, with a common reward function R. S A [0, and horizon H. Define
A(P.Q) sup (x.0)ESXA
TV(P( (V.())ss.), Q( s,,)),
where TV(p. : (p(s') q(s')) denotes the total variation distance between two proba-
bility distributions p.q on S. Then, for any policy x and initial state $
(s) Va.(s:)) H2 Q)
28

---

Published as a conference paper at ICLR 2026
Proof. For h E {1...., H,, let df and di denote the distributions over state action pairs (5h, 04) at step h when running policy т in the MDPs M(P) and M(Q) respectively, both initialized from the same state $1. In particular, df = dl. We first control the evolution of the occupancy measures. By definition of the dynamics,
(o+ Σ ()))a)) P(s))) 8,
and analogously for Q. Hence
|1dh+11
dbttal1= Σ
Σ
dh ())(a)) ( 8, dg as) (a' [raja)((s' ss,a)
8'.0' 8,a
<
(8, a) - dg aT [rajj P(s' 8,a)
8',0' 8.0
ΣΣ E a) traja) 5, Q(s' s,,)
,',''
dh (8. a) dR (8, a)
Σ
P(s' s. Q(d ss,a)
sup
(s,a)
[d d |11 + 2 sup TV (P(: s, ss,a)
(a,a)
| - d 11 + (P,
Since l|dp
891 1 = 0, an induction on h yields
| - df S 2(h 1) Q().Q)
4h I...., 1.
Next, write the value policy æ under kernel P as
V2 = Σ E (on a(sh)
and similarly (81) with d. Since R(s, a) E [0. 1).
Bag [R] Bgg [R] I ll - dh 112.
Therefore,
(si) (s)) 5 Σ
Egg [R]] [
H
laff dR111
h=1
<
2(h 1)A(P.Q) HPA(P.Q).
h=1
where the last inequality uses 1h (HH < /f2 This concludes the proof.
Theorem 4. Let 46 be the learned mixture kernel from offline data, Māy MM be the training MDP ODR uses, and A. be the learned policy using any RL algorithm in Mā then have:
Gap S GaPM, (MN) + AHHA 92p
Proof. We have
( - V
= (81) - VMa (81)
(81) + (81) - VAN
1 (81) + VAN
MM (81) - VMTE
29

---

Published as a conference paper at ICLR 2026
By maximality of VMa My ($1) we have:
(51, Vi vi (51) S (M++,
Vi
Man (81)
(172)
V
M Ow (81)
(173)
S APAPP (6xx)
(174)
where the last inequality follows from Lemma 19 Using the same lemma we have:
VM (81) ( & H2A(P" Vox
Hence,
1 - (M S VM (81) - VM, 1 + 211 A(P ,g,x)
(81)
The first term is the suboptimality of n the learned mixture MDP (that ODR actually optimizes against), while the second term is closeness penalty measuring how well the fitted mixture kernel Jā approximates the real dynamics P* in total variation. Under the well-specified and identifiable assumptions of our main results, 26x converges to P*, so the penalty term vanishes as N + co. Under misspecification, the penalty converges to the best approximation error achievable within the simulator family.
30


Reviews:
[{"review_id": "iasupccrOk", "reviewer": "Reviewer_sZX1", "summary": "The authors present a rigorous theoretical analysis on the offline domain randomization (ODR) framework, which enables offline datasets of real-world data to be used for inference of simulation parameters and ultimately for training effective RL policies in simulation.\nAs current ODR methods are vastly empirical, the current manuscript studies the theoretical implications and assumptions of such setting, including assumptions on identifiability of dynamics parameters, and implications on when posterior distributions are guaranteed to converge to degenerate zero-variance point-estimates.", "questions": "- How much of the derivations in this work are restricted by assumptions on simulator dynamics behaving as a parametric family of distributions? And what about assumptions on Gaussian distributions in particular?\n- The original empirical works (e.g. DROPO) consider the case of unmodeled phenomena, i.e. where the parametric family of simulators may not perfectly match the entire dataset of offline transitions, hence maximizing the log-likelihood may not converge to such degenerate zero-variance distributions. Can the results on weak and strong consistency extend to such cases, or do the authors consider this setting?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles a relevant framework in the field of sim-to-real transfer (e.g. ODR) that gained traction in recent years but lacked a thorough theoretical understanding. Such setting opens applications of domain randomization that are arguably safer and more sample efficient than uniform domain randomization methods.", "weaknesses": "- Restricting Gaussian assumption: recent empirical works further extend the ODR framework by considering normalizing flows or neural density estimators over dynamics parameters [1]. It's unclear how much the presented analysis is restricted to (1) simulators that follow a Gaussian parameter distribution and to (2) transition functions that are also assumed to be Gaussian.\n\n[1] Muratore, Fabio, et al. \"Neural posterior domain randomization.\" Conference on robot learning. PMLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pLdiea8Dmj", "reviewer": "Reviewer_EobY", "summary": "The paper “Statistical Guarantees for Offline Domain Randomization” establishes a theoretical foundation for Offline Domain Randomization (ODR), a variant of Domain Randomization that leverages offline data to better align simulated and real-world dynamics. The authors formulate ODR as a maximum-likelihood estimation problem and derive statistical guarantees on its consistency, with the goal of improving sim-to-real transfer in Reinforcement Learning (RL).", "questions": "1. How much offline data is needed to achieve meaningful consistency in practice? Are there any finite-sample bounds?\n2. Can your theoretical framework explain why existing DR approaches (like DROPO or DROID) sometimes work even without theoretical consistency?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. By framing ODR as a maximum-likelihood estimation (MLE) problem, the paper elevates it from a purely empirical heuristic to a method with formal statistical grounding, establishing properties such as consistency.\n\n2. The paper provides a clear exposition of its underlying assumptions, such as i.i.d. sampling, mixture positivity, and Lipschitz continuity, and thoughtfully discusses possible relaxations, which helps clarify the scope and applicability of the theoretical results.", "weaknesses": "1. The theoretical framework assumes that the true environment dynamics $𝑀^∗$ lie within a known parameterized simulator family \n{$𝑀_𝜉$}, and that a representative dataset of real-world transitions is available. In practice, however, the true parameterization is unknown, and it is rarely possible to guarantee that the simulator family adequately captures real-world behavior. This makes the theory elegant but largely non-operational in realistic settings.\n\n2. The proposed ODR framework relies on access to real-world data to fit the randomization distribution, which partly contradicts the original motivation for offline RL, to minimize or avoid costly real-world data collection.\n\n3. The paper does not include empirical results or analyses addressing robustness under model misspecification, i.e., when the true environment lies outside the assumed simulator family.\n\n4. The work sidesteps the central open challenge in domain randomization: how to design or learn an appropriate simulator distribution when the true real-world distribution is unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gwYKhqwpxA", "reviewer": "Reviewer_Fgrz", "summary": "The paper provides a rigorous theoretical foundation for offline domain randomization (ODR), an increasingly important topic in bridging the sim-to-real gap in reinforcement learning. The authors frame ODR as a maximum-likelihood estimation problem over a parametric family of simulators and derive formal convergence guarantees. Specifically, they show that under mild regularity and identifiability assumptions, the estimator is weakly consistent, and under additional uniform Lipschitz continuity, it becomes strongly consistent, converging almost surely to the true dynamics as the amount of offline data increases. The proofs are technically solid and clearly structured, and the paper does a commendable job explaining not only the mathematical results but also their practical meaning.", "questions": "Could the authors illustrate, even with a toy example, how the weak and strong consistency results manifest as offline data increases?\n\nGiven that many of the primary applications for ODR, such as robotics, are dominated by non-smooth dynamics (e.g., hard contact, stiction-friction) where Assumption 5 may not hold, could the authors elaborate on the theoretical implications of this gap?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper represents one of the first attempts to formally establish statistical guarantees for ODR, an area that has previously relied primarily on empirical evidence (e.g., algorithms such as DROPO). The theoretical treatment is rigorous yet well-motivated, and the authors are careful to analyze the realism of their assumptions, providing insightful discussions on how they could be relaxed to cover broader scenarios. This combination of solid mathematical grounding and practical reflection significantly strengthens the paper’s contribution. In particular, the convergence results provide valuable theoretical reassurance that incorporating real offline data into domain randomization is not only empirically beneficial but also statistically sound.", "weaknesses": "The analysis assumes that environment parameters are predefined, but in practice, it may be more realistic to start with a broader set of perturbable parameters and iteratively remove those with small variance as data accumulates. It would be helpful to discuss whether the current proofs would still hold, or require modification, under such an adaptive parameter-selection procedure. \n\nWhile the theoretical contribution stands well on its own, the paper could be strengthened by adding a few illustrative experiments (perhaps simple synthetic tests) that empirically confirm the convergence behavior or highlight the limits of the stated assumptions. Such additions would make the results more accessible to a wider audience beyond theory-focused researchers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wPBhZh0TcE", "reviewer": "Reviewer_i6We", "summary": "This paper provides the first theoretical analysis of offline domain randomization (ODR), formulating it as maximum-likelihood estimation over simulator parameters and proving weak consistency and strong consistency under various regularity conditions. The work bridges the gap between empirical ODR methods like DROPO and theoretical understanding.", "questions": "1. Do the assumptions (particularly 3 and 5) hold in standard benchmark environments (e.g., MuJoCo with mass/friction randomization)?\n\n2. Can the framework extend beyond Gaussian to mixtures or other distribution families?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel theoretical contribution: First formal consistency guarantees for ODR, addressing an important gap between practice and theory\n\n2. Practical considerations: Section 6 thoroughly examines when assumptions hold and offers relaxations (e.g., ergodic processes instead of i.i.d., weaker tail conditions)\n\n3. Clear presentation: Problem setup is well-motivated and mathematical framework is clearly articulated", "weaknesses": "1. No empirical validation: The paper is purely theoretical with no experiments demonstrating when assumptions hold in practice or how consistency rates scale with data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)