Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples
Authors: Weiqin Chen
Xinjie Zhang
Sandipan Mishra
Santiago Paternain
Venue: Accepted by TMLR ()

Abstract: Offline reinforcement learning (RL) learns effective policies from a static target dataset. The performance of state-of-the-art offline RL algorithms notwithstanding, it relies on the size of the target dataset, and it degrades if limited samples in the target dataset are available, which is often the case in real-world applications. To address this issue, domain adaptation that leverages auxiliary samples from related source datasets (such as simulators) can be beneficial. However, establishing the optimal way to trade off the limited target dataset and the large-but-biased source dataset while ensuring provably theoretical guarantees remains an open challenge. To the best of our knowledge, this paper proposes the first framework that theoretically explores the impact of the weights assigned to each dataset on the performance of offline RL. In particular, we establish performance bounds and the existence of the optimal weight, which can be computed in closed form under simplifying assumptions. We also provide algorithmic guarantees in terms of convergence to a neighborhood of the optimum. Notably, these results depend on the quality of the source dataset and the number of samples in the target dataset. Our empirical results on the well-known Procgen and Mujoco benchmark substantiate the theoretical contributions in this work.

Xinjie Zhang Published in Transactions on Machine Learning Research (02/2026)
Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples
Weiqin Chen Rensselaer Polytechnic Institute
chenw 1 &rrip edu
1232366ccouumiia.cco edu
Columbia University
Sandipan Mishra Rensselaer Polytechnic Institute
MISHRS2drpi.edh
Santiago Paternain
paters@rpi.ed.
Rensselaer Polytechnic Insfälute
Reviewed on OpenReview: /t opwreevee......ee.eo net/ forum? id- Zog8ThcKuuy
Abstract
Offline reinforcement learning (RL) learns effective policies from a static target dataset. The performance of Maleeofftheeart offline RL algorit hums notwithstandingg relies on the size of the target dataset, and degrades if limited samples in the target dataset are available, which is often the case in real-world applications. To address this issuc, domaín adaptation that leverages auxiliary samples from related source datasets (such simulators) can be beneficial. However, establishing the optimal way to trade off the limited target dataset and the but-biset source dataset while ensuring provably theoretical guarantees remains an open challenge. To the best our knowledge, this paper proposes the first framework that theoretically explores the impact of the weights assigned to each dataset on the per- formance of offline RL. In particular, we establish performance bounds and the existence of the optimal weight, which can be computed in closed form under simplifying assumptions. We also provide algorithmic guarantees in terms of convergence to neighborhood of the optimum. Notably, these results depend on the quality of the source dataset and the mum- ber of samples in the target dataset. Our empirical results on the well-known Proegen and MuJoCo benchmarks substantiate the theoretical contributions in this work.
1 Introduction
Deep reinforcement learning (RL) has demonstrated impressive performance in wide variety of applications, such as strategy games (Mnih et al.) 2013 2015), robotics (Levine et al. 2016 Duan et al.) 2016). and recommender systems {Afsar et al. 2022 Lin et al. 2023). RT. aims to learn an optimal policy that maximizes the expected discounted cumulative reward. To achieve this goal, the online RI agent learns and improves the policy by actively interacting with the environment. However, this poses critical challenge for the real-world applications of R.,, as interactions with the real-world can be significantly dangerous and expensive (Kumar let al. 2020 Levine et al.] 2020 Chen et al. 2024). In this context, offline RL has emerged as promising alternative framework for the real-world applications of RL, where the agent learns effective policies from a static and previously collected dataset.
Recent advances in offline RL algorithms have shown remarkable success across a diverse array of problems and datasets (Fujimoto et al. 2019a Kumar al. 2020; Kostrikov al.) 2021 Chen al. (021). Never- theless, their effectiveness depends the quality and size of the dataset. More concretely, it is worth noting

---

Published in Transactions on Machine Learning Research (02/2026)
that even state-of-the-art (SOTA) offiine RL algorithms like BCQ FFjjimoo et al., 2019a), CQL Kumar
et al. 2020), IQL, (Kostrikov et al. 2027), DT (Chen et. al. 2021 demonstrate poor performance given limited offline RL dataset, as training on a small number of samples may lead to overfitting (Fu et al. 2019 Kumar et al.] 2019). In this work, we are interested in offine RL that learns from a static dataset with limited samples. We now proceed by introducing a series of related works.
1.1 Related Work
Offline reinforcement learning with dataset distillation. Dataset distillation (Wang et al. 2018) proposes a framework for synthesizing a smaller and more efficient dataset by minimizing the gradient discrepancy of the samples from the original dataset and the distilled dataset. Syuthetic (Light al. 2024) is the first work that applies dataset distillation to offline RL and achieves comparable performance with the original large offline RT, dataset. Specifically, it synthesizes a small distilled dataset by minimizing the gradient matching loss between the original offline RL datasct and the synthetic dataset. However, generating the synthetic dataset necessitates access to the original large offline RL dataset, which is oflen impractical in weal-world scenarios. In particular, this work focuses on scenarios where only a limited number of samples are accessible at all times.
Offline reinforcement learning with domain adap- tation. To avoid overfitting by learning from the löm- ited target dataset, domain adaptation techniques (Redko jet al. 2020 Farahani et al. 2021) propose to leverage an auxiliary large source dataset (such as simulators) with unlimited samples. H2O (Niu al. 2022) assumes ac- cess to an unrestricted simulator, and the process of train- ing on simulators still requíres interacting with the envi- ronment online. On the other hand, ORIS (Hou et al. 2024) proposes to generate a new (source) dataset from the simulators, where a generative adversarial network (GAN) model is employed to approximate the state dis- tribution of the original larget dataset. Starting from the initial state provided by GAN, the new (source) dataset is generated by interacting with the simulator and reweighted by an additional discriminator model. These approaches emphasize the algorithmic design by employ- ing sample-dependent weights to combine the source and target datasets during the learning process, e.g., through weighting based on the transition ratio between the source and target domains.
S 35
Source
Agent
Target
Figure 1: Schematic: the target dataset has lim- ited samples (red), whereas the source dataset has unlimited samples (green) with dynamics gap from the target dataset. How to strike a proper balance between the two datasets?
Given the unlimited source dataset and the limited target dataset, however, striking a proper balance between the two datasets prior to the actual learning process remains challenging problem in offline RL (see Figure 1. Straightforward strategies involve either combining both datasets equally or using only one of them, nevertheless, neither approach offers optimality guarantecs. Accordingly, this work focuses on exploring the impact of the weights assigned to each dataset on the performance of offline RL, particularly from a provably theoretical perspective.
Offline reinforcement learning with dynamics gap. Dynamics gap in the domain adaptation and transfer for offline RL is widely acknowledged as significant challenge. From a practical standpoint, DARC (Eysenbach et al. 2020) and DARA (Liu et al. 2022 train dynamics gap-related penalty by minimizing the divergence between the real and simulator trajectory distributions, and combine it with the simulator reward during the online/ofliiee training of simulators. On the other hand, HTRL (Qu el al., 2024) theoretically investigates the sample efficiency of a hybrid transfer RL framework with a dynamics gap, which, however, requires access to the entire target domain. To our knowledge, no prior work has theoretically investigated the impact of the dynamics gap on selecting the weights assigned to the limited target dataset and the unlimited source dataset.
2

---

Published in Transactions on Machine Learning Research (02/2026)
1.2 Main Contributions
The main contributions of this work are summarized as follows.
To the best of our knowledge, this is the first work that proposes an swoorithm-agnoste framework of domain adaptation for offline RL prior to the actual learning process, which exhibits a provably theoretical trade-off between the number of samples in limited larget dataset and the dynamics gap (or discrepancy) between the target and source domains.
We establish the (expected and worst-case) performance bounds and the convergence to a neighbor- hood of the optimum within our framework. We further identify, under simplifying assumptions, the existence of an optimal weight for balancing the two datascts, which is typically not one of the trivial choices: treating both datasets equally or using either dataset only. All theoretical guarantees and the optimal weight will depend on the quality of the source dataset (refer to the dynamics gap) and the size of the target dataset (i.e., number of samples).
A series of numerical experiments conducted on the well-known Procgen and MuJoCo benchmarks substantiate our theoretical contributions.
Notice that the primary locus of this paper is on the theoretical analysis of our proposed framework, while algorithm development is reserved for future research.
2 Balancing Target and Source Datasets
In this section, we consider the mathematical formalism offline RL (Levine et al., 2020), namely Markoe Decision Process (MDP) (Sutton & Barto 2018). This work centers on the tabular MDP defined by a tuple where S and A finite state and action spaces (a standard assumption in RL
M = P,r,,, for thcoretical guarantecs), P S A S - [0,1] denotes the transition probability that describes the dynamics of the system, r: S x A + R denotes the reward function that evaluates the quality of the action, : S + [0, 1] denotes the initial state distribution, represents the discount factor, and T E 10,1,2,... ... defines the horizon length.
A policy is represented by the probability distribution over actions conditioned on states each state 8, the agent selects an action a based on the policy (a]s)) and generates the next state s'. The tuple (s, s') is referred to as the transition data. Offline RL considers employing a behavior policy Ty to collect an offline and static dataset with N transitions (N E N+), Lee., D.r = {(s;, 0;, + The transitions the dataset Dur are collected from a domain D. The goal of RL to learn an optimal policy x² that maximizes the expected discounted cumulative reward, Lee,
T Σ
π arg max
E
(1)
p. ((sls()
P(
In the context of offline RL, the policy needs to be learned from the static dataset D,, exclusively. Offline RL algorithms (Fujimoto et al. 2019a Levine et al. 2020 Kostrikov et al. 2021) typically train an action-value function (or Q-function) by minimizing the temporal difference (TD) error iteratively. To be formal, let B
be a Bellman operator. This operator takes different forms depending on the specific algorithm considered. For instance, in Q-learning type methods the operator takes the form
BQ (s, a) r(s, (1s.a)+1)
+
E o' ((rlsa)
max Q*
(2)
whereas for actor-critic methods takes the form
BTQ² (s, a) = r(s,
s' s al mla(ks) [Q"(6', (s))
(3)
3

---

Published in Transactions on Machine Learning Research (02/2020)
Given any transition (s, s'), let us define the TD error for any Q-function at the step & as
(s,,,))))))) = (Q(s.a)-
where B denotes the stochastic approximation of the Bellman operator, namely version of (2) where the expectation replaced by evaluating the random variable at single of realization s'. Note that. we have dropped the dependence on & on the left hand side of the above expression for simplicity, We further define the expected TD error in the domain D as
(Q(
=
E (s,a,8')=
where (8, 8') D denotes that the transition as,a,'') is drawn from the probability distribution Po(8, 8') in the domain D. With slight abuse of notation, we also denote (s, D to represent that the state-action pair (s, a) is drawn from the probability distribution Pp(s, a) in the domain D. With these definitions, one can define the iterations of a large class of offline RL algorithms through the following optimization problem
QK+1 arg (Q) Ak E.
(6)
Tt is worth pointing out that the specific forms of 2 and 3 can result. in poor performance in offline RT attributed to the issues with bootstrapping from out-of-distribue
(OOD) actions (Fujimoto et al. 2019b Kumar el al. 2019 2020 Levine et al. 2020). This typically leads to an overestimation of the Q-value. To avoid this overestimation, prior works consider solely using in-distribution state action pairs to maintain the Q-function (Fujimoto ct al. 2019a), or constraining the learned policy remain closely aligned with the
behavior policy (Levine al.
2020
Note that the expectation in 5 poses challenge in solving problem 66: it requires visiting every transition infinite times. In practice, onc defines the empirical version of the TD error ED(Q) in
1 N (Q) = N
Σ (s,,,,)
(7)
where the samples are from the dataset Dr.. Then, the offline RL algorithm is defined as the minimization of the stochastic approximation of problem
@++1 arg min (Q),), Vk N.
Q
It has been widely demonstrated that SOTA offline RL algorithms, such BCQ (Fujimoto et al. 2019a CQL (Kumar et al. 2020 IQL (Kostrikov et al. 2021), and DT (Chen et al. 2021 are capable of solving problem (6) given sullicient transition samples from the domain D. Nevertheless, ess, this assumption may not realizable practice, e.g., healthcare (Tang et al. 2022) and autonomous driving (Pan et al. 2017) where data collection is challenging. In this regime, Q++1 generally demonstrates poor performance in approximating Q++1, as training on a small number of samples can lead overfitting (Fu et al. 2019 Kumar et al. 2019).
On the other hand, in certain applications, larger number of samples D' = {(,,,, ay, ,5 with N' > N. It is worth noting that, in general, D' will differ from D in terms the state distribution and transition probabilities. Similar to (5), we define
rely on simulators (or related datasets) that provide a
E (p)(Q) (e (D)DD)
[E(Q, (sg,
where (s), ag. 5/) D' denotes that the transition (g),a,,8)) is drawn from the probability distribution Po (8j, 86) in the source domain D'.
4

---

Published in Transactions on Machine Learning Research (02/2026)
We then explore in this paper a general scheme to combine the limited target dataset and the large- but-biased source dataset
Q = arg mig (1 () XED vke
(10)
where E [0, 1] denotes the weight that trades off (g) from the limited target dataset and ED (Q) from the large-but- but-biased source dataset. Particularly, À x 1 prioritizes the minimization of the TD error corresponding to the source dataset. This approach is suitable in cases where the source dataset is similar to the taryet (or coming from the same domain in an extreme case). On the other hand, À 0 focuses on minimizing the TD error in the target dataset. This method is appropriate for scenarios where data is abundant (sufficiently large N) or where dynamics gaps between the target and source datasets are too large.
Given these observations, it is expected that different values of can attain the optimal performance, depending on the interplay between the available number target samples and the dynamics gap ween the two datasets. The following section formalizes this expectation.
3 Performance and Convergence Guarantees
We start. this section by discussing the necessary assumptions develop our theoretical results concerning the generalization of the solution to problem (10).
Assumption 1. There erists B 0 such that, for any (s,a) E S x A, |r(s,a) < B.
Assumption 1 is common in the literature (Azar et al. 2017 Wei ct al. 2020 Zhang et al. 2021 b. In
particular, in the case of finite state-action spaces, it is always possible to design the reward to avoid the possibility of being unbounded. Further notice that this assumption, along with the fact that a geometric series with ratio 7 E [0, 1) converges to 1/(1 7) implies that
B
max Q(s,a) S
(11)
(A
Assumption 2. g can be computed given a sufficiently large amount of samples from source domain D'.
To be precise, in practice one must work with the empirical counterpart of (9) Note, however, that since N' >> N, the generalization error arising from the finite number of samples in the source dataset is negligible. For this reason and to simplify the exposition, slight abuse of notation, denote D' to refer to both the souree dataset and the souree domain.
assume that (9 can be computed directly and, with a
To proceed, we denote by (ar a) and Pp(s, a) the probability of a state-action pair (s,a) a) from the source and target domains, Let us define D = [(ss a) E S x A (s, a) E Dud as the transition-erctuu excluded dataset of Dar, (ss E 10,1....,NN as the number of the specific (s, a) transitions in the target datasct Dez, and Po(s, a) as.a)
Assumption 3. For any (8.a) E S x A, PD(s,a) a) and Ppr(s, a) are positive. Given any dataset Der and its corresponding transition-ezce eezclued dataset D, there exist constants Ba 2 31 > such that ()(s,u)/Por) a) [3, (ssaa) E D.
3 posils that every state-action pair (s) in the space S x A has non-zero probability of occurring in the source domain D' and the target domain D, thus yielding a bounded ratio Po(s, a)/ (s, a), V(s a) E A.A. This is a mild assumption in the sense that if Pp(s, 0 or (s. essentially implies that the state-uction pair (s, a) can be ignored in the corresponding domain of interest. ii) Nevertheless, since the limited dataset D is untikely to cover the entire space of S x A, we define 3r and for all (8,a) D. In this
Remark 1. i) Assumption
Bu as the lower and upper bounds, respectively, of the ratio for any (s, E
sense, Pp(s,a)= = ((s,a)/N
It is worth noting that the transition sample 7 is independent and identically distributed (i.i.d),, as the dataset Der is shuffled and the transition data are sampled i..d. Alternatively, one can think of the dataset random samples from the occupancy measure, followed by random transition.
5

---

Published in Transactions on Machine Learning Research (02/2026)
Remark 2. It is crucial to note that
10 relies solely on the limited target dataset when X 0, i.e., e arg ming (1 (Q) + AE compule Q for any state action pair outside the trunsition eransilioneexcudeed dalaset D. Hence, Uhroughoul this XPrr a) > O. work, we assume À E (0, for any (s, a) E E A which guerantees that (1 (1-A)Pp(s,0) (XP
arg mingin (g( In this context, it becomes infensible to
Having formally stated all assumptions in this work, we are now in conditions to proceed with the analysis, where we start by deriving the analytical expressions of Q++1 and Ox+1
Proposition 1. Let Assumptions and 3 hold. Recall the empirical Bellman operator B in 4 Denote by BD (BD, ) the Bellman operator in D (D'). Note that Q++1 and Q represent the solutions to (6) and [10). Given any dataset Der and its corresponding consition-ezcluded dataset D. denote by N and (s,a) the total number of samples and the amount of a,,a.)) transition in Der- At each iteration (k- = 0,2,2.... it holds that
(3) in which s' follows the transition probability of the domain
(8, (s, a), V(s,a) ES 1-1
(12)
Q+ (s, N E (a..a)
By (*, (s (B) ((s,a)
2
(13)
Proof. Refer to Appendix
In addition to the analytical expressions Q++ (s.a) a) and Q (s, a). Proposition implics that O (s. a) (the right hand side of (33) reduces BD Q* (s, a), an analog of Q a) solely replacing D by D'.
We proceed by exploring the performance of Q++1, as in our problem of interest (10). To do so, we next define two quantities that will play important. roles in the thaoretical guarantees of Q Start by defining the dynamics gap (or discrepancy) between the target and source domains
=
max (s
BD 1" (Bb0
(14)
where, for notation simplicity, we omit the dependence of & on the iteration k. Indeed, if the transition probabilitics the source and target domains match, the quantity & in 14) becomes zcro. Then, given any dataset D.r and its corresponding transition-exccudeel dataset D, we define a measure of the variability
L (8.
== max (s.a)ED
(15)
N(8,a)
where of (By (B, denotes the variance the empirical Bellman operator. The term & above is the maximum normalized variance in the given target dataset. It is significant to note that the number of samples to keep the value constant is proportional to the variance.
With these definitions and the analytical expression of Øt+1, we are now in conditions of establishing the bound the expected TD error of O*++1 over the target domain.
Theorem 1 (Expected Performance Bound). Recoll & in min(s.a)eSxAA Po(s, a). Let the conditions of Proposition hold. Given any dataset Der. it holds at each iteration (k = 0, 0,1,..... that
14
in 15 and define C
[ - -ED(Q 1 ) ++ (1-17A/8))
2 ()- + ) to-No) E. (16)
8'
Proof. Refer to Appendix A.2
Recall that (Q*+ represents the optimal TD error of the offine RL at iteration k. which cannot be computed in practice due to the finite number of samples in the target dataset. The significance of Theorem

---

Published in Transactions on Machine Learning Research (02/2026)
is to establish bound on the difference between the optimal TD error ED(Q*+)) and the expected TD error of Of+² (the solution to (10 with respect to the next-state 8 (from the limited target dataset) over the target domain transition probability Pb(3' s,a).
Notice that the bound in (16) arises from the general Assumption 3 which implies that the limited dataset D does not cover the entire cente-action space S x A. As a result, this bound is loose in some scenarios. Before proceeding with a more detailed explanation, we formally present tighter bound in the following theorem, under stronger version of Assumption This new result will make the discussion the looseness of the bound in [16) more explicit.
Theorem (Tighter Expected Performance Bound). Let the conditions of Theorem 1 hold. Suppose that for any (8,a) E E x A, Po(s,a), (s,(s,a) and (s(s,a) are positive and there exist constants 3u 2 3 > O such that (s (o(ss))/PP P//PP//sP/a) E [u). v(s,a) E S A. Given any dataset Den, it holds at each iteration
(k: = 0,,, that
[Ex(Q)+)) EE 1- (1-1+XB..)
)'+ s+ (1- 31
2
E 8' (PPPo())
+ g.
(17)
Proof. Refer to Appendix
Theorems and 2 imply that the expected performance bounds of Q in both 16 and 17 exhibit an
intuitive form for any E (0, I], as jointly depends on the variance (s) of the limited target dataset and the bias (E) introduced by the large source dataset when À (0, 1), and reduces to the dependence on é
alone when À = (i.c., solcly considers the source dataset D' in 10
Notwithsstnnding we also note that the bound in 16 may be loose when X = 0, since it depends both and E, even though only the limited target dataset is employed in (IOp. Notably, such looseness does not manifest in the bound presented in Theorem undder À = O, which provides a tighter bound than that in (66
Yet, Theorem 2 relies on the stronger assumption that the limited dataset D covers the entire state-action space A, which rarely the case practice.
In addition establishing the expected performance bounds of Q Theorems 1 and 2 imply the bias- variance trade-off sought by combining the two datasets with different weight À in (10p. Indeed, the optimal weight X that minimizes the right hand side of (16) and (17) is discussed formally by the following corollaries.
Although Theorem 2 provides a tighter bound, it is worth highlighting that Theorems and 2 yield the same
X", as the extra term e-NCg in d16b is independent of X.
Corollary 1. Under the assumptions of Theorem or Theorem the optimal weight X that minimizes the bounds in 16 or 17 respectively is X when ç = 0 and X when { = O.
Proof. Refer to Appendix A.4
Corollary 2. Under the assumplions of Theorem 1 or Theorem iʃ 31 = 3n = =3 > 0, the oplimal weight X' that ménimizes the bound in 16 or 17 respectively takes the form
BC
X
(18)
35 E
Proof. Refer to Appendix A.5
Recall that 3 and B. denote the lower and upper bounds of the ratio (o(s,a)/PP/(( V(s E D. Thus, the assumption & B. = B in Corollary 2 implics that Pp(s, ())pp(ss) (s) is fixed ratio for any (s, E D. This may occur when the sampling of the target dataset proportionally follows the source distribution for all (s,a) E D. Since DD (relates to the target dataset) comprises fewer Mlate-actin pairs than S x A (relates to the source domain), the fixed ratio 3 will be greater than 1.
It is significant to highlight that both corollaries above recover the intuition that the target dataset with variation (or the number of samples in the target dataset is sufficiently large), ie., 5 0, encourages
7

---

Published in Transactions on Machine Learning Research (02/2026)
to consider the target dataset only in 10 i.e,, X = 0. On the other hand, when the two domains are close (5 ~ 0), the optimal value of X is one, suggesting that one should use the source dataset solely. Although intuitive, the expected performance bounds in Theorems and 2 are insufficient to claim any generalization guarantees as the tails of the distribution could be heavy. We address this concern in the next theorem by providing the generalization bound (worst-case performance bound). Moreover, since the stronger assumption in Theorem 2 is unlikely to hold practice, we center on Theorem from now on, upon which the remainder of this work is built, to maintain the generality of our results.
Theorem 3 (Worst-Case Performance Bound), Denole by Sy the upper bound of Po(s; a)/ (Dr(1,,0 a) for any (s, a) E S x A. Let the conditions of Theorem 1 and Assumption 1 hold. Given any dataset Der, the following bound holds at each iteration (I- 0.1,2.... with probability at least 1 1-6
tD( ED((
1-1
2
A
2
5
& E
A
A/Bu
X) B1
S|AA
Bi
2(1 -11-)//B/
( ( (BB))BB(BB/(11 -)) + 2AVE (1-
+ 2
log
(19)
VN
31 Bi
+
1-7
Proof. Refer to Appendix A.6
The above theorem provides the worst-case bound of solving (0), which demonstrates the bias-variance trade-off by the two datasets with different weight as well. Most importantly, both the expected and worst-case performance bounds, as shown in (16) and (19), imply that the optimal trade-off between the source and target datasets is not always trivial, indicating that X may not belong to {0, 0.5, 1}. The optimal trade-off for the expected performance depends on the number of samples in the target dataset (corresponding to ç and N), the dynamics gap (or discrepancy) between the two domains (corresponding to E), and the bounds of (s (a(s,,)//Ha) aa/ (////>/ss, a) (corresponding to B1 and B.). In addition, the optimal weight for the worst-case performance bound will depend on more factors such as the reward bound B, the discount factor y, the size of the state and action spaces [S] and |A|, and the bound of Po(s, a) (s, a) (see Remark [ Ph some of which might be highly challenging to estimate in practice. Therefore, the worst-case performance bound in our work is primarily of conceptual interest. In addition, we focus mainly on the theoretical analysis of our proposed framework that balances the limited target dataset and the large- but- buse-but- source dataset. Developing a practical and efficient algorithm to learn an approximate optimal weight remains a promising direction for future research.
Having established various performance bounds of solving 10 we are in the stage of providing the con- vergence guarantee. We formalize it in the next theorem, which relics the following two quantities: the maximun of the dynamics gap over all iterations and the maximum of the variance over all ilerations, given any dataset D.r and its corresponding transition-excluded dataset D
Emax
sup E(Q*), Smax = sup <(Q²). KEN
(20)
KEN
Theorem 4 (Convergence). Let the conditions of Theorem hold. Given any dataset Drr, it holds at each iteration (k = 0,1,2.... that
E 8' Po('
E (s, ()) S
(21)
(s,a)>
nk1 E (s [1I" a) ('( 1 gkt1 1-7
1- 11-7(11177 11-1+1/3,
> A) Bi
+ +o-Nc)
V Smar"
V
Proof. Refer to Appendix A.7
8

---

Published in Transactions on Machine Learning Research (02/2026)
The previous theorem implics that the solution Øx+1 of solving (10 is guaranteed in expectation to converge to neighborhood of the optimal Q-function, Lee, Q* as l - 00. This neighborhood is presented as follows
1 ( (I- C --
1-
NC NC) Smax + (1 A) + te +e-Ne)
VVmmxx
(22)
Apart from the discount factor 7: the neighborhood C depends the weight X, the maximal dynamics gap Smax; the maximal variance Smax, and the bounds B1 and Bu.
4 Numerical Experiments
Although this work primarily focuses theoretical analyses, present in this section series of numerical experiments that demonstrate the performance of solving (I0) under different weight À and validate the corresponding thcoretical contributions in the provious section.
4.1 Procgen Experiments
4.1.1
Environments
We consider the well-known offline Procgen benchmark (Mediratta et al. 2023 which is often used assess the domain Cllaptation/genraaliii capabilities of oflline RL. We select live games revironnnnss from Procgen: Caveftyer, Climber, Dodgeball, Maze, Miner, whose descriptions are provided in Appendix A.8
4.1.2 Experimental Setup
Our implementations as well as the datasots that have been usod in this work are basod on (Modiratta al., 2023). Instead of training on single dataset, this work trains an offline RL agent on two different dalasets from the source and target domains.
Backbone algorithms. Recall that our framework is algorithm-agnostic implying that various SOTA RL algorithms can apply. In this work, we select CQL (Kunar et al. 2020 and IQL. (Kostrikov et al. 2021 as representative algorithms due to their promising and robust performance across a variety of offline RL tasks.
Datasets. Note that Proegen employs procedural content generation create adaptive levels upon episode reset. Each level corresponds a specific seed (non-negative integer) and has distinct layouts (such as the amount and position of various entities, see c.g., Figure 13) (Modiratta. et al. 2023 As result, the same action taken in the same state can lead to different successor states depending on the level (e.g., being blocked by an entity in one level but not in another), yielding level-dependent transition dynamics. In each environment, we select the target domain to span levels [100, 199), and consider three distinct source domains defined over the level ranges [0, 99), [25, 124] and [50, 149], respectively. Recall that the target dataset is expected to contain significantly fewer samples than the source datasel, i.e., & N'. Typically, N' is considered be ten times larger than N. Therefore, we consider three different sizes of target datasets from levels [100, 199] with N E (1000, 2500, 4000), and sct N² 40000.
Hyperparameters. To ensure a fair comparison, we retain all hyperparameters consistent, e.g., batch size, learning rate and network size, and solely change the weight assigned to each dataset. Key hyperparameters for the datasets and algorithms are summarized in Table (refer to Appendix A.9
4.1.3 Results
Recall that the worst-case performance bound in 19 can be overly conservative, particularly when the state and action spaces are large, as the bound scales with the dimensionality of the spaces. Therefore, this subsection focuses on the expected performance bounds (16) or 177 as well as its corresponding corollaries. It is crucial to note that the variance and the dynamics gap & are challenging to measure or estimate precisely, as requires access to the entire source and target domains. This not feasible within the scope of our problem of interest. Nevertheless, one can still investigate how these factors influence the expected bounds, which provide insights into the expected performance of offline RL. Specifically, examine the
9

---

Published in Transactions on Machine Learning Research (02/2026)
impact of each of s, & and À the expected performance. To achieve this, vary one of these factors at a time while keeping the other two parameters constant. We present our findings as follows.
Impact of the trade-off between the source and target datasets (A). We consider seven discrete values of from f.,0.2, 0.4, 0.5, 0.6, 0.8, where À E {0, represents the three trivial choices: con- sidering the limited target dataset only, employing the large-but-biased source dataset solely, treating both datasets equally. Notably, the expected performance bounds and Corollary 2 reveal that the optimal weight may not be the three trivial choices. To validate this, in each of the Procgen environments, target dataset comprising 1000 samples from levels [100, 199] and source dataset with 40000 samples from levels [0, 99). Figure 2 depicts the results of two offline RI. algorithms under various A: CQL (upper row) and IQL (lower row). Indeed, observe that only two out of ten environments have the optimal weight to be the trivial choice, i.., X = 0.5 in Dodgeball for both CQL and IQL. This further underscores the importance of striking proper trade-off between the two datasets, and reveals that trivial balancing strategies, e.g., A {0, 0.5, 1} are consistently effective and can, sometimes, be catastrophic (scc c.g., À 1.0 in Miner).
consider
1.75-
1.50
0.8
E 1.25
Retur
0.8
Return
0.6
0.4
0.4
1.00
0.75
0.25
0.50
02
0.50
0.25
0.00
0.0
0.2 4.666.66688
0.0
0.0 0.2 0.5 0.6 0.8
0.0
0. 0.2 0.4 050.608...
0.0 0.2 0.4 0.5 06
0.00
0.0 0.000.4444 204 0506681.
Lambda
Lambda
Lambda
Lambda
Lambda
Cavefiyer
(b) Climber
(c) Dodgeboil
(d) Maze
Miner
2.5
1.2 0.8
2.5-
2.0-
Return tu A
1.5-
1.0
0.4
0.5
0.0
0.0 02 0,05060,81
0.0 0.00 0.
0.0
0.0 0.0.4
0.5.0.00 05000880
0.0 022 0.0.5. 0.6 0.000 Lambda
0.0 0.4 0.6 0.8 1.0
Lambda
Lambda
Lambda
Lambda
(f) Cavefiyer
(g) Climber
(h) Dodgeball
(i) Maze
(i) Miner
Figure 2:
The performance of offline RL across five Procgen games. The source dataset contains 40000
samples from levels [0, 99). while the target dataset. comprises 1000 samples from levels [100, 199). We consider seven weights, A E 0.,0 0.5, 0.6, 0.8, 1.0}, to trade off the source and target datasets with the star marking the optimal weight. Upper row: CQL as the backbonc; Lower row: IQL as the backbone.
Table 1: The performance across all À E {0, 0.2, 0.5, 0.,. corresponding to different ç. Left: CQL as the backbone algorithm; Right: IQL as the backbone algorithm.
1.0) (mean and std) with fixed N 1000
Game
E1
$a
Game
E1
E2
Caveftyer
1.23 + 0.23
1.28
0.26
1.37
0.34
Caveffyer
2.29 0.13
2.30 + 0.13
2.56
0.13
Climber
0.57 + 0.10
0.73 + 0.16
0.85 0.21
Climber
1.74 0.36
2.14
2.20
0.60
Dodgeball
0.67 0.17
0.87 0.21
1.29 0.42
Dodgeball
0.83 183:100 0.20
0.97
1.39 0.47
Maze
3.52 + 0.41
3.97 + 0.42
4.35 +: 0.52
Maze
5.91 + 1.02
6.47 + 1.29
7.33 1.37
Miner
1.27 + 0.48
2.05 + 0.55
3.12 + 0.80
Miner
3.58 + 1.51
5.31 + 2.01
7.24
2.81
Impact of the dynamics gap between the source and target domains (E). We consider three source datasets of the same size but from different domains: levels [o, 99), levels [25, 124], [50, 149), and let E1, fa and fa represent the dynamics gap between levels [o, 99] and [100, 199], between levels [25, 124] and [100, 199]. and between levels [50, 149] and [100, 199), respectively. Thus, we obtain & 2 & 2 &a, as more overlap between the levels of the two datasets demonstrates smaller discrepancies between them. Note that the bound in [66
17 decreases with smaller values of implying improved expected performance of offline RT. Our
10

---

Published in Transactions on Machine Learning Research (02/2026)
The performance across all E 0.2, (., 0.5, 1.0, (mean and std) with fixed Ea corre- sponding to different N. Left: CQL as the backbone algorithm; Right: IQL as the backbone algorithm.
Table 2:
Game
N 1000
N = 2500
N 4000
Game
N = 1000
N 2500
N 4000
Caveftyer
1.37 + 0.34
1.52 + 0.26
1.57 + 0.19
Caveflyer
2.56 + 0.13
2.82 + 0.30
2.90 0.32
Climber
0.85 + 0.21
0.89 + 0.29
0.95 + 0.26
Climber
2.20 + 0.60
1.98 + 0.65
2.22 0.63
Dodgeball
1.29 + 0.42
1.34 + 0.30
1.42 1: 0.18
Dodgeball
1.39 0.47
1.40 + 0.29
1.57 = 0.29
Maze
4.35 + 0.52
4.75 + 0.59
5.21 + 0.69
Maze
7.33 1.33+131
7.73 + 1.30
8.23 1.22
Miner
3.12 = 0.80
3.28 0.59
3.33 0.75
Miner
2
8.10 + 2.53
8.18 2.20
numerical results of five games across three different & values are summarized in Table which supports the implication from the bound in 16 or 17
Impact of the size of the target dataset (s). It is worth highlighting that the normalized variance ç in (15) decreases as (s, a) increases. Given the positive proportional relationship between N and (ss, a), larger N practically leads to smaller < (in expectation). Analogous to &, the bound in (16] [17]
decreases with smaller values of and/or e-NC, thus indicating an enhanced expected performance of oflline RL with larger N (smaller and/or smaller e eNCC). Our numerical results of five games across three different values of N are summarized in Table 2 and Figure 4 which validate the implication from the bound in (16) or (17). Nevertheless, offline RL may fail due to the coverage gaps rather than an insufficient sample size when the random sampling non-uniform. For instance, having large amount of data concentrated in a small portion of the target domain can lead catastrophie performance, due to the limited generalization capability of RL (Packer el. al.
2018
Kirk el. al. 2023
2.00
2.0
Return
1.0
1.75
1.50
Return
Return
125
1.0
0.6
0.75
1.00
0.50
025
0.0
0.0 0.2 0200011009100
0.0 0.2 0.4 0.5 0.6 1.0
0.00
0.0 0.20 0.
0.0 0.2 0.4 0.5 0.6 0.8
0.0 0.002 0.4 0.5 0.6
0.8 10
Lambda
Lambda
Lambda
Lambda
Lambda
(a) Cavefayer
(b) Clömber
(c) Dodgeball
(d) Maze
(c) Miner
3.5
3.0
2.5
Return
Retum
Retur
m
2.0
2.0
1.5.
1.0
0.5-
0.25
0.0
0.0
0.2
0.40
0.5500
0.8
0.0
0.2 4
0.5
0.6 0.8 1.0
0.00
0.0 0. 0,000001881.
0.0 0.544.55 0.6 1.0 Lambda
0.0 0.2 0.4 0.5 0.6 0.8
Lambda
Lambda
Lambda
Lambda
(f) Cavefiyer
(g) Climber
(h) Dodgeball
(1) Maze
(i) Miner
The performance of offline RL across five Procgen games. The target dataset comprises 1000 samples from levels [100, 199], and three source datasets are considered, each containing 40000 samples from levels [0, 99] (green, Ei); [25, 124] (blue, E2), and [50, 149] (red, Ea), respectively. Seven weights, E {0, 0.2, 0.4, 0.5, 0.8, 1.0], are evaluated to trade off the source and target datasets with the star marking the optimal weight for each ş. Upper row: CQL as the backbone; Lower row: IQL as the backbone.
Figure 3:
Practical optimal trade-off between the source and target datasets (X*). Notice that Theorems I and 2 as well as Corollaries I and 2 explicitly demonstrate that a smaller drives X closer to 1, while a smaller shifts X* closer to O. In what follows, we substantiate this implication through empirical ev- idence. i) To explore how X varies with &, we fix N 1000 in each of the five Procgen games and select three different source datascts with & 2 & 2 Ea- We then implement this using seven values of 0.2, A E 0.2, 0.4, 0.5, 0.6, 0.,, Our numerical results in Figure 3 and Table demonstrate that the opti- mal weight X within 0.,, 0.4, 0.5, 0.6, 0.8, 1} increases (closer to 1) as & decreases. This substantiates
11

---

Published in Transactions on Machine Learning Research (02/2020)
2.5
2.5
2.0-
28 m
1.50
1.25
2.0
Return
Retur
1.5
1.00
1.0
0.75
0.5
0.5
0.5
0.0 0.0 022 0405060810
0.00
0.0 0.05060810 1.0
0.0
0.0
0.2
0.4
0.5 0.
0.0 0.2 0.4 0.5 0.8
0.0 02 0.4 0.5 0.8 1.0
Lambda
Lambda
Lambda
Lambda
Lambda
Cavefiyer
(b) Climber
(c)
Dodgeboil
(d) Maze
(e) Miner
3.5
3.0
3.0
12 10
14 Return
2.0
Retum
tu m m
1.0
0.0 0.2 0.4 0.5 0.8
0.0 0.0 0.2
0.5
0.5
0.8 0.000
0.0
0.0 02
0.4 0.6088100
0.0 02 0.4 0.5.5 0.6 1.0
0.0 02 0.4 06
Lambda
Lambda
Lambda
Lambda
Lambda
(f) Cavefiyer
(g) Climber
(h) Dodgeball
(i) Maze
(i) Miner
Figure 4: The performance of offiine RL across five Procgen games. The source dataset comprises 40000 samples from levels [50, 149], and target datasets from levels [100, are considered with three different sample sizes: N = 1000 (green), N = 2500 (blue), and N = 4000 (red). Seven weights, E (0,0.2, 0.4, 0.5, 0.6, 0.8, 1.0), are evaluated to trade off the source and target datasets with the star marking the optimal weight for each N. Upper row: CQL as the backbone algorithm; Lower row: IQL as the backbone algorithm.
Corollary (aay/aann and the intuition that greater emphasis should be placed on the source dataset when its discrep- ancy from the target domain is smaller. In the extreme case, where the source and target domains become identical, one should consider the large source dataset only. ti To explore how varies with s, we fix the dynamics gap be &3 in each of the five Procgen games. Since larger N corresponds to smaller s, select three different values of N (1000, 2500, 4000). It is worth noting that N' is consistently main- tained at least ten times larger than N, ensuring that the source dataset always comprises significantly larger amount of samples than that of the target dataset. We then implement this using seven values of E {0, 0.2, 0.4, 0.5, 0.6, 0.8, 1}. Our numerical results in Figure 4 and Table demonstrate that the optimal weight within 0.2, 0.4, 0.5, 0.6, 0.8, 1} decreases (closer to 0) as N increases (s decrcasos). This validates Corollary 2 and the intuition that greater emphasis should be placed on the larget dataset when com- prises larger number of samples. In the extreme case where the target dataset contains infinite samples, it becomes optimal to rely exclusively on the target dataset.
Table 3: The optimal weight X within (0, 0.2, 0.4, 0.5, 0.6, 0.8, 13 corresponding to different g. Left: CQT as the backbone algorithm; Right: IQL as the backbone algorithm.
Game
E (10, 99])
(25, 124))
{500, 149))
Game
{ 99))
{2((,, 124])
Es((5, 149])
Cavefyer
0.4
0.4
0.5
Caveflyer
0.4
0.4
Climber
0.6
0.8
1.0
Climber
0.6
0.6
Dodgeball
0.5
0.6
0.8
Dodgeball
0.5
0.6
Maze
0.4
0.6
0.8
Moze
0.4
0.6
0.8
Miner
0.4
0.6
0.8
Miner
0.2
0.6
0.8
4.2 Auxiliary MuJoCo Experiments
In addition the Proegen benchmark, we conduct auxiliary experiments on two continuous-control tasks from the MuJoCo benchmark: HalfCheetah and HumanoidStandup. This study is intended further sub- stantiate theoretical contributions in this work.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: The optimal weight X within 0.2, 0.4, 0.5, 0.6, 0,8, 1) corresponding to different N. Left: CQL as the backbone algorithm; Right: IQL as the backbone algorithm.
Game
N 1000
N 2500
N
4000
Game
N
1000
N
2500
N 4000
Caveflyer
0.5
0.5
0.2
Caveftyer
0.5
0.5
0.2
Climber
1.0
0.8
0.4
Climber
1.0
0.8
0.8
Dodgeball
0.8
0.6
0.5
Dodgeball
0.8
0.6
0.5
Maze
0.8
0.4
0.2
Moze
0.8
0.4
0.2
Miner
0.8
0.8
0.6
Miner
0.8
0.6
0.6
Environments. For each task, the target domain corresponds to the standard environment. We construct three source domains (É1, &2, {3) by scaling a subset of physics parameters: gravity and friction scales. With the target domain defined by (gravity_scall friction_sceett are specified by (gravity_scaee scale,, friction friction_scale) E {00.85, 0.85), (0.90, 0.90), (0.95, 0.95) for (2, Ea), respectively. In this setting, {3 exhibits the smallest dynamics gap relative the target domain, whereas & induces the largest.
= (1.0, 1.0), the three source domains
Experimental Setup. We select IQL as the backbone algorithm and sweep À E {0, 0.2, 0.4,0 0.6,0.8, 1.).. We collect the limited target dataset with N samples and the large source datasets with N' > N in each of the source domains. In HalfCheetah, we choose N' = 80000 while N can vary in {3000, 5000, 8000). In HumanoidStanduu, we select N' = 100000 while N can vary in {6000, 8000, 10000). Note that different MuJoCo tasks may require distinct amounts of source and target data, depending on the task complexity.
Results. Figure 5 demonstrates that the optimal weight X the MuJoCo experiments does not coincide with any of the three trivial choices {0, 1, 0.5), further highlighting the importance of striking an appropriate trade-off between the two datasets. Figure 6 reveals that the optimal weight X shifts towards larger values as the dynamics gap between the souree and target domains decreases, and Figure 7 exhibits that increasing shifts the optimal weight X* Loward smaller values. These observations are consistent with those in the Procgen experiments, continuing to validate our theoretical findings in this paper.
1750
Return (x109)
1500
1.2-
c1250
51000
0.8
Re 750
0.6
500
0.4
250
0.2
0
0.0 0.4 0.5 0.5.060.88....
0.81.0
0.00
20
0.5
0.8.0.6.8888
6.8
Lambda
Lambda
HalfCheetah
(b) HumonoidStandap
Figure 5: The performance of offline RL across two MuJoCo tasks. The source dataset contains N' samples from the farthest source domain Éi, while the targel dataset comprises N samples from the target domain (HalfCheetah: N 3000, N' 80000; HumanoidStandup: N 6000, N' 100000). We consider seven weights, X E {0, 0.20.2.00 0.4, 0.5, 0.6, 0.8, 1.0}, to trade off the source and target datascts with the star marking the optimal weight.
5 Conclusion
The performance of offline RL highly dependent on the size of the target datasct. Even stato-of-the- art offline RL algorithms often lack performance guarantees under a limited number of samples. To tackle offline RL with limited samples, domain adaptation can be employed, which considers related source datasets,
13

---

Published in Transactions on Machine Learning Research (02/2020)
2
2000
Return 6
1.75
(150000
Returg
1500
1.5
1.25
10000
0.75
500
0.5
0.25
0.00.2 0..60...1
0.0 0. 0.6.81.0
Lambda
Lambda
(a) HolfCheetah
(b) HumanoidStandap
Figure 6: The performance of offline RL across two MuJoCo tasks. The target dataset comprises N samples from the target domain, and three source domains are considered, each containing N' samples with gravity_scall frictionnscallll
E {(0.85, 0.85), (0.90, 0.90), (0.95, 0.95)} (green-E1; blue-f;; red- Ea) (HalfCheetah: N = 3000, N' = 80000; HumanoidStandup: À E {0,0.2, 0.4, 0.5, 0.6, 0.8, 1.0), are evaluated to trade off the source and target datasets with the star marking the optimal weight for each ç.
= 6000, N' = 100000). Seven weights,
2000
N: 5200
Return [ x110)
2
1000
0200
1.75
10000
Return
E1500
1.5
1.25
1000
0.75
500
0.5
0.25-
0.00.20.A
4.
0.5 0.8 1.0
2.00.20.4
5.5
0.6 0.8 1.0
Lambda
Lambda
(a) HalfCheotah
(b) HumanoddStandaa
The performance of offline RL across two MuJoCo tasks. The source domain fixed to the closest setting Es, and target datasets from the target domain are considered with three different sample sizes: HalfCheetah N E (3000, 5000, 8000} (green, blue, red) with N' 80000, and HumanoidSSanduu e {6000, 8000, 10000} (green, blue, red) with N' 100000. Seven weights, A 0., 0.4, 0.5, 0.6, 0.8, 1.0), are evaluated to trade off source and target datasets with the star marking the optimal weight for cach N.
Figure 7:
e.g., simulatoos that typically offer unlimited (or a sufficiently large number of) samples. To the best of our knowledge, we propose in this work the first framework that theoretically explores the domain adaptation for offline RL with limited samples. Specifically, we establish the expected and worst-case performance bounds, as well as a convergence neighborhood under our framework. Moreover, this work provides the optimal weight for trading off the unlimited source dataset and the limited target datasot. demonstrates that the optimal weight is not necessarily one of the trivial choices: using either dataset solely combining the two datasets equally. Although this work centers the theoretical analyses of our framework, conduct series of numerical experiments the renowned Procgen and MuJoCo benchmarks, which substantiate theoretical contributions. Last but not least, our established optimal weight is unlikely to be computed in practice, as depends on quantities that are challenging to estimate. Hence, developing a practical and efficient algorithm to learn an approximate optimal weight remains a promising direction for future research.
Acknowledgments
This work was sponsored by the Office of Naval Research (ONR), under contract number N00014-23-1-223
14

---

Published in Transactions on Machine Learning Research (02/2026)
References
M Mchdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys, 55(7): 38, 2022.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement PMLR, 2017.
learning. In International conference on machine learning, pp.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15044-15097, 2021.
Weiqin Chen, Dharmashankar Subramanian, and Santiago Paternain. Probabilistic constraint for safety- critical reinforcement learning. TEFF Transactions on Aulomatic Control, 2024.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. 2448-0056. PMLR, 2020.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Picter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine leurning, pp. 1329 1338. PMLR, 2016.
Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. arxiv preprint arXiv:2006.1330 13916, 2020.
Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. A brief review of domain adaptation. Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020, pp. 877-894, 2021.
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q-learning algorithms. In International Conference on Machine Learning, pp. 2021 2030. PMLR, 2019.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavanzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint Xrv:::1900.0 01708, 2019a.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052 2062. PMLR, 2019b.
Yiwen Hou, Haoyuan Sun, Jinming Ma, and Feng Wu. Improving offline reinforcement learning with inac- curate simulators. arXiv preprint arXiv:2405. 04307, 2024.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76:201-22 1611,,24441 2023.
llya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXi::2110.061 166.6669,, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy q-lcarning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 33:1179-1191. 2020.
Sergey Levine, Chelsea Finn, Trevor Darroll, and Picter Abboel. End-to-ond training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334 1373, 2016.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. Xixio preprint 2005:20055.01443, 2020.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Jonathan Light, Yuanzhe Liu, and Ziniu Hu. Dataset distillation for offiine reinforcement learning. arXiv preprint ar arXio:2477200990 2024.
Yuanguo Lin, Yong Liu, Fan Lin, Lixin Zou, Pengcheng Wu, Wenhua Zeng, Huanhuan Chen, and Chunyan Miao. A survey reinforcement learning for recommender systems. IEFE Transactions Newral Networks and Learning Systems, 2023.
Jinxin Liu, Hongyin Zhang, and Donglin Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning. arXiv preprint 0ab.2203.06666 2022.
Ishita Mediratta, Qingfei You, Minqi Jiang, and Roberta Raileanu. The generalization gap in offline rein- forcement learning. orXiv preprinl axXiv:2312.05742, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXie preprinl arXiv:1312.5600 2013.
Volodymyr Maūh, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7544):559-533) 2015.
Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When to trust your sim- ulator: Dynamics-aware hybrid ofline-and-online reinforcement learning. Advances in Neural Information Processing Systems, 35:36599 366122, 2022.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint 1EG.11810.122500 2018.
Xinlei Pan, Yurong You, Ziyan Wang, and Cewu Lu. Virtual to real reinforcement learning for autonomous driving. arXiv preprint 1744:2200.03952, 2017.
Chengrui Qu, Laixi Shi, Kishan Panaganti, Pengcheng You, and Adam Wierman. Hybrid transfer reinforce- ment learning: Provable sample efficiency from hhifted-dynamiee data. arXiv preprint arXi::2411.03810, 2024.
levgen Redko, Emilie Morvant, Amaury Habrard, Mare Sebban, and Younès Bennani. A survey on domain adaptation theory: learning bounds and theoretical guarantees. arXiv preprint arXiv:200,.11899 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi-Velers and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems, 35:34272-34286, 2022.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprinl 0rX9v:1811.1099,, 2018.
Chen-Ye Yu Wei, Mehdi Jafarnia Jahromi, Haipeng Luo, Hiteshi Sharma, and Rahul Jain. Model-free reinforce- ment learning in infmite-horizon average-reward markov decision processes. In International conference on machine learning, pp. 10170 10180. PMLR, 2020.
Zihan Zhang, Simon Du, and Xiangyang Ji. Near optimal reward-free reinforcement learning. In International Conference on Machine Learning, pp. 12402 12412. PMI.R, 2021.
16

---

Published in Transactions on Machine Learning Research (02/2026)
A Omitted Proofs
A.1 Proof of Proposition
Proposition i Let Assumptions and 3 hold. Recall the empirical Bollman operator B (4. Denoto by BD (BD,) the Bellman operator in or 3 in which s' follows the transition probability of the domain D (D'). Note that Q++1 and 2x+1 represent the solutions (6) and (10p. Given any dataset Der and its corresponding transition-excluded dataset. D. denote by N and [(s, a) the total number of samples and the amount of (s,a, a). transition in Drr- At each iteration (lt = 0,2,,,...) it holds that
(s, a), ((s,a) E SxA. (s = BBQ
(23)
L Vi B (s0 v(s.a)
(s
(24)
(1- (s) APp
Proof. Note that
12
well-known in RL, however, we provide its proof here for completeness. For any Q,
note that
Ep (Q)
E (8,0,5))mo
(25)
Σ PD(s,a) Σ Pols' s,,a) (Q(s, ( a'rD
(26)
where (a) holds by definitions and (b) follows from the definition of conditional expectation.
The derivative of (Q) w.r.t. (s, ((, E x A is given by
④E)(Q) 2Pp(s, a) Σ Pp(s'(s,a) (Q(s. a) - -B, (s,
(27)
aQ(s,a) a)
2Po(s, a) (Q(s,a)-BppC)) - Bp
(28)
where (a) follows by taking the derivative, leveraging the fact that each Q(s, a) is only present in exactly
one term the summation 28 (b) follows from the facts that Q(s, a) is independent of s' and by definition
(D a) = Ee-ro(ss)ss.)
B. Qh (s. a)} (see (2 or (3)). Since the objective 6 strongly convex, its
minimizer Q++1 the unique point that satisfies
(ED(Qh+1)
aQ(s.a) = O, V(s () ES A.
(29)
Combining the previous equation with 28 and the fact that Po(s, a) > (by Assumption 3 yields
Q (8.0) a) = (DQ a), ((s,a)
(30)
This completes the proof of 12 in Proposition
We now turn our attention to proving
Note that
N (1 X) Σ
- (Q=))=(())- + XED
(0)
as) - By - + AED (Q)
(31)
1=1
N(s.))
b
(1
Σ
((s,,))BB,,00 (x, + XED (Q),
(32)
(s.A)ED
where (a) follows from the definitions in and 7 (b) re-arranges
31 by summing first the N(s.a)
samples of the next state corresponding to a given (s, a) pair the dataset D.
The derivative of (1 - (Q) AED w.r.t. Q(s, a) for any (s, E S A can be computed under scenarios:
17

---

Published in Transactions on Machine Learning Research (02/2026)
(i) For any (s.a) 8 A A D, the derivative is given by
0 ((1 (Q) AE (Q) (s. (Q(s,a) BD QD.
(33)
aQ(s,a)
where the previous equation follows from an analog of (28 replacing D by D' and the fact that the first term in 432) depends solely on the state-action pairs in the dataset D.
(ii) For any (s,a) D. the derivative is given by 0 - ()) + AE (Q))
8Q(s, a)
N(s.a)
(1 X)
Σ
BD 2APp (Q( (s.
(34)
j=1
N(x.a)
6
(1 1)
2 N ((s,a))(s,a)-
Σ (sy0*(s.a) + 2XPD ==1
(Q(s, - Bo
(35)
where the first term in (a) is obtained by taking the derivative, leveraging the fact that each ( a) only present in exactly one torm of the summation (35), and the sccond term in (a) is analogous to the derivation of σE (Q))0QQ)(())))) a) (see 28 replacing D by D', (b) due the fact that
Ex ((s,a) a(s,))
It worth highlighting that (35 roduces to 33 when N(s, a) i.c., when (s,a) S A A (sce Remark D.. Consequently, by combining the two scenarios of (8, a) E D and (s,a) E x A D, it holds for any (s, a) E x A that
((1 (Q) + AED((())
(Q(s,a)
((s.a)
2 = -(1- (1- N
((s,a)Q(s,u)- - Σ By (x, + 2XPo (Q(s,)) B Qz
(36)
Similar to 29 since Q+++ is the unique solution to the convex problem 100. the gradient of the objective is zero. Equating the right hand side of the previous equation to zero follows that
1 1 A N(s.a)
(s APo
Σ By Qz
(37)
N
where in the above expression we use (s(s.a) = N(s, a)/N. holds by Remark that (1- - (s, + (s) > Then we further obtain
(s, N Σy= (i By Q + ()) V(s,a) e SxA.
(38)
1-X) a) AP'
This completes the proof of Proposition
A.2 Proof of Theorem i
Theorem
1 (Expected Performance Bound). Recall & in 14p.
in 15 and define C
min(s..) (s. a). Let the conditions of Proposition hold. Given any dataset Der. it holds at each
iteration (k = 0, 1, 1,,,....) that
[D(())1) - -E0())+) 1 1-x+1/B s
(
+ -e-No- E. (39)
E 8'
2) +x
18

---

Published in Transactions on Machine Learning Research (02/2026)
Proof. We start by writing Ex(QX*1) ED(Q*+!) using the definitions in
04 and (5)
E ED(Q)t1)= (s,,
[Q* (s,a) B. - BB (s.,
(40)
Expanding the squares in the above expression follows that
ED( - EQ(ttt1
E (0 (0 (s,+) 28, (s, () 2x (s. ())
(41)
Conditioning on (s,a), using the fact that only B. depends on s', and that by definition (p (s) (DQ a), the above equation reduces to
- (s,
E (a.0)D
(s, (s) (Q)t
2BBDQ(ss,a) (Q)+)(s,a) - Of (s.al) ].
(42)
Replacing Q with (DQ (by
12 in Proposition in the previous equation yields
Ep(( - ED(Q
(s, a))² (s, (B,Q* - (s,a)) [10f (s, (s, (BDQ
E
(43)
(s,a)
(s, a))² [10 (BDQ (s, a))² 2B (x. (s.
E (s,a)D
(44)
We next take the expectation on 44 with respect to S' (from the dataset Der), using the fact that only 2 (s, depends on and the conditional expectation follows that
E 3'~Po(8')a.aa
[ED(QA+))] (bb((ttt)
[ott (s, (o** mi)]
= E (s,0)D 8- Pa
(Q)t (s, + (s)) 2B (mPo(d")s,a)
(45)
Using the definition of the variance of random variable, we rewrite the first term on the right hand side of the above expression
E
(ott
o (o*+*(s,a)) + (
E [Qxt (s.0))
(46)
Substituting the previous expression in 45 yields
E (PPP(8'
(ED(QA+))) - (Qt+))
E (s.a)~
0 (s,+)(s,a))
[ott'(s, (s, [
E
(0)a))))==))) + (B)))
-BBDQ*(s.a)
E
@ui @ (8.
(47)
8NPP s.a)
19

---

Published in Transactions on Machine Learning Research (02/2026)
Note that the last three terms the right hand side of the above expression the square of a difference. Hence, the previous equation reduces to
E 8 (s,
[E(( En((t)1)
2
(s,t))) ( 1
E (a.0)~D
0g
2
(Qxt=1(,,0)) +
E N'PPols' s.0)
(48)
We next work on of (0) (s. a)) and U1 separately by focusing on o (Qx+1ss, as) first.
(i) For any (x,a) E x A \ D. + (s,a) in 13 reduces to (x which is independent of 8'. Then we have
o (s, = ox ((,,))) 0.
(49)
(ii) For any (8,a) E D, i.e., M(s,a) > 0, we note that
N Σ Nis,a) Q²) (Prr (B
08 (Q+
(0x+
(8, a))
j=l
(50)
(1 P) + (8,
N Σ j=1 B2 (s.a)
N(a.0)
(51)
11 () (8,0) 0) + (D.
() [o.aa) N(,,,)) Bg (*
oz.
(52)
(1 P) a) A
2 20 (BxQ((s,a))
(c)
-1- ()
(53)
(s,p( + XPD (s,a)
((s.a)
where (a) follows from 13 in Proposition (b) is obtained by the theorem of variance of a shifted
random variable, since A B) (s, a) does not depend on 8', (c) is due to o² = (x) where X denotes any random variable and is a constant.
Therefore, we have
0,
V(s,a) E S A\D,
of (a- (1-))pp(ss.)) + (Po
203 20 (Bx N(s,a)
(54)
(1 (1- ())((s)) (i--A--APPbaaa
V(s,a)
Notice that 48
can be rewritten as
E 8'~ (PP'(8')s.a)
[ED( (Q+4
E
(Q)+1 (s,a))
+ E
U?
(55)
(s.))D
(saa)D
Σ
Po (ott((s,a))
E
U2
(56)
(s.a)D
(a,0)88xA
Σ
Po(s,a)oz
(Q) (s,a))
Σ
Pols, (0)+((s,)))
+ E
Up]
(57)
(0.0)
(R.a)D
(
20

---

Published in Transactions on Machine Learning Research (02/2026)
Substituting 54
into the previous equation yields
E 8' (''-Pos'ls,,a)
[ED(QX - -ED(Q
2 20 (B..))))
(1- (L)))
Σ a)(s,a)
+
E
U?
(58)
(1-A) XPD
N(s,a)
(0.0DD
(s:a)ED
It then follows from the law of total probability that
E [ED(QA+1) )] ED( 3' 3' (8.)a)
Σ
(Po(s,a (s,a) E D) P((s,a)
E D) + Po(ssa (s.a) P(s.))))xs
(S,0)D
203 (Bx
(1- A)Pp(s,a) APb
E (s,a)-
UR
(59)
a(s,a)
203 (Bx()(s,aa)
(1 (1-))Pp(s1) a)
Σ (s,a)ED
P (s,a) E E) ()),,))))))
(1- + (P
N(8,a)
+
U?
(60)
(a.0)D
where the last equation holds by the fact that PD(s, a (s,a) E A D)) = 0 as all (s, E D now. follows by the definition of conditional expectation and p((s,a) D) that
[ED(Q) (Q(ht))
8'~Pos''lla (s,))
(1 (1- ()-))p(ss (11 (1- ()) (1-)))p()))) (P
20 (B,)))s,a))
[U
E (
+ E
(61)
N(s,a)
(M.O)D
We now turn our attention to U1. For any (8, E A, obtain
L N(x.)) By (5, + as)
U1
E (=(3's.))
BDQ
(62)
( APp:
b Nf ( (s BDQ (s,u)
(63)
(1 (P
(c) (1 (s B) BDQ
(64)
(d) (s (Br -B0
(s
(65)
(1-A)P a) (s,
where (a) follows from 13 in Proposition (b) follows by the definition of BDQ*(s,a) (sec 2
3
(c)
follows by the definition (o(s,a) ((s, and (d) follows from combining and canceling terms.
Substituting the expression of U1 in 65 into 61 results in
203 (B..)((.aa))
(
(1 - (1-A)) 2)Po(s,a) XPor(s.a)
E
[E0(O)+1)]
-E)()))))
E (1,0)<D11
8 (PPllsl)
N(s.0)
2 (1-)))3(ssa) APD
APpr(s,
E (+.0)<
(BDQ)(s, (D)
(66)
21

---

Published in Transactions on Machine Learning Research (02/2026)
The previous inequality can be simplified
E 8'mPo(8' s.a)
[ED(Q - {D(
20 (B,))(
(1 - 1-)))((ss (1-x)) (1-x11
(s,0)~D1
+ AP (s,
N(s.a)
(s
Σ ao(s,a)
(BDD" - (b,a)))
(67)
(1- (1=s)) + (P
(a,))8XAA
20 (B.))
(1 (1-1)) (1-x)) + AP (5,
(1- 1)) a)
E (ID)
N(s.u)
(s
Σ ap(s,a)
(1-))P + (P
(S,))D
? (1-))) 2PP -B000((s))-BB-000003 (68)
(P
Σ
Pp(s,a)
BD
Tt follows from the law of total probability that
E
[ED(QA+)] (Q(+t))
20z (Ba))(s,a))
(1 - (s, a) (1 - P) + XP (s,a)
<
E
(M.))a/D
((s,))
+
Σ
(Po(s,a
D) (s, E
(s,a)
a((s,a)= E +D) + Po(s
E (
(S,a)D
AP
as) (B)
(69)
(1 () (PP
Σ
I p((s,a) E D) + Pols,a (8, E ES A D) ((ss,))
(s,))SSxii
(1 () + (P) ) (BBO - (s.)
XPr (x,
(70)
Notice that Po(s,a a (5.a) 6 x A D) 0 for all (s E D in 69 and Po(s, a (s, E D) for all
(s, S x A D in 70). Subsequently, the previous inequality reduces to
E 8' P'Po'3'1s.))
Ex(( - (1)(
003 (B,
(1 (s.
<
(1 - () (1 XPP
N(s,a)
BD Q (BD))) P((ss)) ((sss(((s (1-)))p((sa) (P ) ((564(5)))-B000066000) (s.
(s
)
Σ
(s,a)D
APrr 2
Σ
Sx D) P((s.a) { (1-x))pss (P
(Sa))SxAlD
(BDQ (aosaa-BB--B
BD Q (s, a))²
(71)
22

---

Published in Transactions on Machine Learning Research (02/2026)
Moreover, it holds that
((( 1
(72)
(a)
P((s,a) A (1 Po(s,a))* S CNNNP(() NNC
<c-No
(73)
where N denotes the size of the target datasel, (a) follows from log(1 aa) < -a, Va E (0,1), and (b) holds
by the monotonicity of e and C' := (a, Pp(s,a). Substituting 72)
and 73 into yields
E
[ED(QX+)) - -Q((
8'"Pp((((s,a)
20 (B,))()))a))
( (1- (1-))) (1-)))p((sa)) + (s
E (s,a)~D1D)
N(s,a)
+
Σ
(Pols,a ( (s,a) =))
(s,a)
(P
(Bo
(1- (s)( + AP (8,
(8,a)D
AP
Σ
Po(s,a (8, E cSx D) NC.
(1- (1- (P
(s.0)ESxA15
(74)
It holds by the definition of conditional expectation and Pp(s,a) for any (s, Sx A D that [ED(QX+1) [] (Q)
8' P'''a(8')
2 (B,0)().a))
(1 (s, () (P
E (s,))ai)
N(s,a)
0,
APD (s, a)
(B)Q*(s,a) -B..)))))))))
E (
A)Po a) APo
U/3
E
e
NC as, 1z
(75)
(s.))D(SSAAD
Dividing both the numerator and denominator by ap(s,a) in U2 and by Por a) in Us yields
E
[ED(O E0(())1)
8'~ 8'~'p('(ss.a)
- 203 (Bx0
<
E (s,0)<DIb
1- x AP (s, /)/ Pp(ss
N(s.a)
X
+
E (s.0)>DID
BD -B
(1 ()Pp(
+
E NC
(s)
(76)
(Dxa)DD(SxAVD
Assumption implics that both (osssa)P(((s,p and Po(s, a) (s) are bounded, i.c.,
1 as) < 3 S
Po(s,a)
<
(s) S Bn, V(s @D.
(77)
p) Br
23

---

Published in Transactions on Machine Learning Research (02/2026)
Substituting the previous expressions into
76
yields
203 (Bx0)((.0))
E
[D(()
ED((Ut))
E (aa)>DiD
1
1 - x + 3/3uu
N(8,a)
E (s,a)aDIE)
(BDQ) - BD ( (*,
(1 - A)B1 +
+
E (s.))D1SxA10)
e -NC. -NC. .BBD(" a) - (s,
(78)
og (*x
1-1
E (s,0)~DID
1+1/311
N(s,a)
A
E
(s, By Bb
(1 A) +
(+.N)DID
+ NN-NC
E
-b.
(79)
(aa))Di(SxA)
Given any dataset Drr and its corresponding transition dataset D) it holds that
c (s,
oB (B.))(((.aa) S max (s.a)ED
N(s,a) (8:00(500) -Bb)
E
(80)
(I
a(s,a)
E (Sa)DDID
(BoQ*(s,a) -B (s,
max (s.)ED
(81)
max (s,a)ESxAl
(BBDQ"(s, -so
(82)
E (s.a)DD($xa)bb (BDQ*(s, - By - (s)
max
(s,
(83)
(s,a)ESxA(()
max (a.
(84)
Then by the definitions of & in 14h and ç in
15
79 is equivalent to
(8'Po()')s.a) (ED(Q)+)) -
1
2
c
A
2
E+ e NCE. G+e-NE.
(85)
1-
1
(1 A) + A
1-A
2
2
s+
+
E.
(86)
1-
(1 A) +
This completes the proof of Theorem
A.3 Proof of Theorem
2
Theorem 2 (Tighter Expected Performance Bound). Let the conditions of Theorem 1 holdd Suppose that for any (s, a) S A, a)(s,a), (s such that Po a) (P E [,, V(s,a) E S x A. Given any dataset Drr. it holds at each iteration (k = 0, 1, ... that
a) and Pp(s, a) are positive and there exist constants B, 3 > 0
E (''PP(E''s..)
[ED(Q)t1)] - (p(t)t)
1- N/B )² S (1
1
A
2
g.
(87)
2)
24

---

Published in Transactions on Machine Learning Research (02/2026)
Proof. Analogous to the proof of Proposition 1 it holds with the stricter version of Assumption 3 that
2 2 (s, N Σj= ا By Q" (s, a) + a)) Q*, v(s,a) E SxA.
1-A ((s.a)
(88)
(1 +)
It follows from
48 in the proof of Theorem
I that
E 8'~ P~Polf'aa0 (a,00)
-
]
=
E (a.0)D
T (s,a) +
E 8'mPold' a.a))
- (
(89)
81
where
(1-A))3(s,0) 200 (B)))(s.a)
o3 (0)+)(s) ( (1
((s,a) AxA,
(90)
N(s,a)
U1 = AP a) -B.0 ((s,a) ESX.
(91)
(1- (sp
It then holds for any (8,a) E x A that
203 (B.))
(1- (),
(ED(Q)+1)) - (
( 1))p)))))) + (P,
(>~Po(3'ls.a)
N(s,a)
(1)))) ( (BDQ*(s,a) ())
XP
2
E
(r.a)D
(92)
Since Pp(s, a) > 0 and (s) > for any (s E S x A, dividing both the numerator and denominator by P5(8, a) in the first term above and by Po (8,a) in the second term above yields
E 8/ (p''
[ED( - (
20z (B.)
1 1
(s,a)
1-
a) PP
N(s,a)
(B,Q*(s,a) BD (B)
+ (s,a)>
(93)
Theorem 2 assumes that both (o.(s,))PPfl a) and Po(s, a) (s are bounded, i.c.,
1 as)(s, < A <
Po(s,a)
<
S Ba, v(s,a) Exx.
(94)
a) 3
Por(s,a) a)
25

---

Published in Transactions on Machine Learning Research (02/2026)
It then holds that
[ED(QA+1)] - (Q((t1)
8'~Po(El|s,a)
oz (s,
1
(a-x))+ +X
-B)
E (e,0)/D)
E (aj))
1- +
((s,0)
(BxQ*(s,a))
>
I-
1-1
2
(s,a)=
3
a(s.a)
1)
d
E (#.0)-
(BDQ"ss, a) (s, a))²
(95)
Given any dataset Drr and its correspouding transition-excluuled dataset D. holds that
og (Bx.Q²
o3 (BxxQ((s,a))
E (s,0)~D
max <
(96)
N(8,a)
(s,a)DD
N(8,a)
E
(BoQ*(s,a)
max (s.0)ESxA
(s.)) -B.00
(97)
(e,0)/DL
Then by the definitions of & 14h and in
15
95
equivalent
E E0( - Ep(( s
11 s+
1-
)
A
2
g.
(98)
P
(1- A) + A
This completes the proof of Theorem
A.4 Proof of Corollary 1
the optimal weight X that minimizes the bounds in (16] 17 respectively is X = 0 when ş 0 and X when & = 0.
Corollary 1 Under the assumptions of Theorem or Theorem 2
Proof. Notice that Theorems 1 and yield the same X", the extra term e NC in Hence, we, herein, present the proof for (16), noting that the proof with respect to
16
is independent of X.
17
follows identically.
We start the proof by considering ş = 0. In this case, the bound in 16 reduces to
x) + & ENE-NCE
(99)
where e -NC g is a constant. Since the minimum of the squared term 0, achicved by À 0, it holds that letting À = 0 minimizes the bound [1=X)]++X ç, thus minimizing the bound in (16).
We now turn our attention the scenario of & = Likewise, the bound in (16 reduces to
1- 2 5. - 1- A/B, )
(100)
Therefore, setting I minimizes the bound above as makes the squared term equal to 0, thus minimizing the bound in 16 These complete the proof of Corollary
26

---

Published in Transactions on Machine Learning Research (02/2026)
A.5 Proof of Corollary 2
Under the assumptions of Theorem I or Theorem 2 if Br 3u B 0, the optimal weight X that minimizes the bound in 16
Corollary 2
17 respectively takes the form
X
(101)
3s +
Proof. Notice that Theorems 1 and yield the same X the extra term e N in [16] is independent of X.
Hence, we, herein, present the proof for [16), noting that the proof with respect to 17 follows identically.
Note when 3 = Bu = 3 that the right. hand side of 16 reduces to
(1 -- )'+ s+
( (1 A)B +
E+e -Nc..
(102)
1
i.C.,
(1 (1-1)8 1)B (1 (1- ) 5+
2
(1- + ) E+e-NoG.
(103)
Taking the derivative of the previous bound with respect to À yields
(1 (1-1)3
-B((1 - A)B + + - (1 2)3(1 - -B)
Derivative = 25
(1 - 1)B
((1 1)3 + A)²
(1 (1-A)8+ 1(1 3)
+ 25
(104)
(1 (1-
((1- + A)²
By combining and canceling terms the previous derivative reduces to
(A-1)
A6
Derivative 25 ((1 A)B + A)
+ 2
(105)
((1 - X)B + A)
+ >> 0 due to that $ > 0 and A E [0, 1]. We further solve the Notice that in the previous equation (1- + optimal weight X by letting the above derivative be zero, i.e.,
Bc 3c + E
(106)
This completes the proof of Corollary
0
A.6 Proof of Theorem
3
Theorem 3 (Worst-Case Performance Bound). Denote by 8y the upper bound of Pp(s, (b(s,a)/Pr((s a) for any (s, E x A. Let the conditions of Theorem and Assumption [ hold. Given any dataset Drr, the following bound holds at each iteration (k = 0, 2, ...) with probability at least 1
- - (Qhtt)
X+ 1/3w )²
1
2
A) B1 + )
A
2
5
E &+e-NOE
-
(1
$1 2
1
log
8
1
|S|IAI
VN
(1- +
3r
2(1 -\nBB
1-1
1 1-7
()B/)) - -7)) (4- 22VE
(
(1 (1 +
(107)
27

---

Published in Transactions on Machine Learning Research (02/2026)
Proof. Note that
ED( (Q(
=E((0xt))
E 3'mPo(5'la,a) [ED(QX)1)
+
E 8'mPb(8 (s,a)
[ED(OX - -D(((t))
(108)
(1-X7N/8 s+ (1- (1- (1- E+e
& ED((
[ED(Qx))]] +
(109)
8'~Po(8')s,))
where the previous inequality follows from Theorem Thus, we are left to bound ED(QRt1) ( To proceed, we rely on the following technical lemma.
E (PPD(''ss.))
Lemma 1 (MeDiarmid Inequality). Let TI.... :n be independent random voriables taking on values in set H and let -1, Cn be positive real constants. If p H" - IR satisfies
sup Thy TTt
I " Th. ,T) - ((TI.' -.. .. , S C.
(110)
for 1 Usis n, then it holds that
-2 1==1
(Ip(T) ,T))
(T.) - E " " ,T,)) 2) $ exp sexp
(111)
To obtain the similar generalization bound akin to the above, we aim to compute the bound of - Eo( where Ox+1 and @ (both see (13)) differ in a single sample of only. More specifically, they take the sequences of random samples (.),... 8y, "., 8y) and (8%, .., 8%) respec- tivcly, where 8) 8 for any j (1,2.... ,N} and j f i. Note that
ED( Ep((st1))
(a)
(of (s. - - -s.a)))))
E (8,0,5))
(112)
(0)
(s))t - 10x+ a)) + 2B a) (Q)+1(s, a) Qx+ (s.a))
E
(113)
(s,0,s")~
where (a) follows from the definition in and 5 (b) follows from expanding the square and canceling the terms.
Notice that By (s, a) is the only term in the previous expression that depends on s'. To proceed, we condition on (s, a) in the previous equation, i.e.,
ED - - -
(a)
E (s,a)DD
[00 (s,)) (s, + (s (s)+t
(Q++1 (s,a) 2 (s.a)) + Q4 (s,a) (s.a) - 2B
-
(114)
(b)
E (a.a)D
(Q -2 (s. 1 (s.a)) (o) (s.a)
(115)
E (s,0~D
- (s,a))
(o (s, (s, +
(s,a) (
(116)
61
where (a) follows by definition (DQ (x-rp(()sss)) combining the terms, and (c) follows by Jensen's inequality.
B (*,
(see 2 or (3)), (b) follows by
28

---

N(e.)) Published in Transactions on Machine Learning Research (02/2026)
We next work G1 and G2 individually. Substituting the expressions of (s,' and 2 (s. a) from 13 into G1 and G2 yields
(1 - 1)/N (B:
G1 =
(117)
(1 (1-A)PA(ssa) (s,
1 N(s.a) N
Eド=1
(Br + (x. (0) 2BDQ
-
(118)
(1-)) a) (P
By combining and canceling the terms G2 be further simplified as
Σ 1=1
(Be, (8,,00(s, B 2(1 ()),
N
G2
(1- a) XPrr(s,a)
2AP (8, (B00))
(119)
(1 P 8, a) (s.
Note that Assumption
and
11 combining with
2
implies
B
B
Bx
(s, 2
(s, a)
S {
+
)
Vi
(120)
-
1-9
1 1-
B
s B +7
Vj
(121)
1-7
1-7"
1
N(s.)
N(s.0)
B
Σ
8x a)
< Σ
(x.
s SN(ss a)
(122)
1-7
j-1
j=1
Applying the above bounds to G1 and G2 yields
(1 1)/N
ByB
G - (1-)))((s,a) (s)
(123)
2 N(s,0) B - - () 1-
G2 5
(1 +
28,a: (B
+
(124)
+
2(1 () +
2xP (BD -B
(125)
(1 (1-))P((s,0) a) (P
(1 (1- +
(1 ())B - 2) ( (8
2XP (Bo -
(126)
Substituting G, and G2 back into
116 yields
( (
(1- (1-N
BB
E (s.a)<
(1- 1) (P
(1 - ()B + 2XP - (sQ*(s,a))
(127)
() (s.p
29

---

Published in Transactions on Machine Learning Research (02/2026)
We rewrite the previous inequality using the definition of the expectation
()(( - ED(()+1)
(1- A)/N
ByB
Σ a
(1 (1- 1) XPrr (s, -
(s.))ESxA
(1
(4B 22P a)
BD (s, BuQ (s)
(128)
(1- (1-A) (5,0) 0) + A
PD(s,a) a)
2(1 2(1-A)7B
Σ
(1 -1- 1) (s.
1-7
(e,0)ESAA
(1 - (4B((s,))4BB(111)))) 2APp a) BD Q- (s. BDQ -B) (1 -1 ()-A)) + (P
(129)
Dividing both the numerator and denominator by (s, in the right hand side of the previous expression, and re-ordering terms yields
-
1
Σ
Po(s,a)/Po
2(1 - X)yB
(1 ()) 1-7
(KO)ESxA
(1 - 1PP33/30/PPooo0000000---01221111 (s))
(130)
(1 (
By using Assumption 3 and the definition of &y the previous inequality reduces to
- - Eb(()+1)
3r
2(1 - 1)
Σ
N
X) +
1-7
(s.a)ESxA
- A4Ba/4//BBB 7)) + 22
1B (s, BDQ
(131)
(1 (1 A) B1 +
31
2(1 A)7B
Σ
N
() X
1-
(s.a)ESxA
(1 (4B
2X
max ( A
(132)
+
By the definition of & in the above expression equivalent
- (2(11)))
+1
En
8
2(1 - -)nB)
(11 ((--))) A)B, 7)) + 2AVE)
Σ
(133)
X) B1 +
1 1-7
(1-
(a.0)SSAA
|S||AA
3r.
2(1 -))BB
1
(1 A)BB((BB//(11))))) 2XVE
(134)
N
A)
31
+
1-7
(1 - 1 +
c.
(135)
where is the bound shown in Lemma
Lemma implies that
30

---

Published in Transactions on Machine Learning Research (02/2026)
P (bllttt)-
E
[Eb(QX+)) )0)= s <xxp Σ c
-26²
(136)
i.e.,
P
(ED(O
[E(( <
21 exp
-2e
(137)
8'~Po(8'
N
Let the right hand side of the previous expression to be 1 - 6. Then,
=
log(
NNC
(138)
2
SIAII
B
2(1 1)
( (1- (LL-))a(4BB(L) 7) + 2AVE
log
(139)
VN
A)
1-7
(1 +
Therefore, the following bound holds with probability at least 1-
Eo(( 10((t)1)
$ 1
s+
& &
N/BuB
A) B1 +
1 S|A
3y
2(1 21-AhBB
( (11-)),,(1B/111 7)) + 2AVE
log(
(140)
VN
(1 - 1) +
1-7
(1
which completes the proof of Theorem 3
A.7 Proof of Theorem 4
To proceed, we rely on the following technical lemma. Lemma 2. Given any dataset Der and its corresponding transition- eeneludee--ecllueed dataset D, let define Emaz
suPkeN E(Q²) and Smar supkeM <(Q²). Then, it holds that
E E (~PP(('lssa)
[11 (s, (3,0(s
s I 1 V VSmaz (1 A) 3 Emaz
1
A
+e + NC VEmazi Vk E N.
(141)
Proof. Given any dataset Drr and its corresponding transition-exchuded dataset D, holds that
E
E (s.a)D
[110++((sa) (s, (DD
8 8~~Po(Fllso)
(a)
E
(s. - Bp (bp00()) [1,0)+((s,a)+
(142)
(s,a)
(''Pp(8'1ss.a)
1-1 M(, (4
(b)
N 2i=1
By (
(143)
f'mPo(8' (s.a)~D
8,0)
x)
L-A Eド=
N(saa)
E 8' 8' (s,a)
(s.a)oD
(1 +
AP (B) -BD (1-))P (
(144)
31

---

Published in Transactions on Machine Learning Research (02/2026)
where (a) follows from swapping the two expectations and the fact that 11-11> =|1- of + a) (see (13)) that operates over (x, a), (c) follows by combining the terms.
(6) follows by definition
By using the triangle inequality the previous expression reduces to
[11Q (s. a) - BDQX(s,
8' (8"Po((1ss.a)
(aie)D
L (ssa) BR (x) (1 (s, (3B N
S
E
8' 3'~Po(E'laso (s.a)>D
(1 (s.
(s, (B) (ss BB
+
E (e,0)~D
B
(145)
8 (NNP(F'1a.00 8.0)
(1- (s.)) (PP (s.
By the definition Po(s,a) a)N, the previous inequality is equivalent to
E
E (ssa)D
[ a) (D
8' 8'('((()),a))
)) Po(a.a) Σ N(..a) Ms,a) j=
$
E (s.a)~/
E
8' S'~Po($'|s.o (x.0)
(1 (1- ()) +XPp((s,a)
(P a) (B (s, BB
EE (a.0)~D ('''Po('
E
(146)
(1 2) (8, a) + (s,
A) Pa(s.a) (s..a)
((.a)) 2ÿ=1 - ())
E
E 8'> P>>Po( [s,a]
(s.a)~DIb
(1 (1- P (8. a) (s,r
E
E
(e.0)-D(D
8 P'''Pa (s,a)
(1 ())
E 5 D
E
[lle [lle-NC. Ne-N
(147)
where the previons incquality follows from the proof of Theorem
Dividing both the numerator and denominator by Po a) the first term and by (D) in the second term of the right hand side of the previous inequality, and re-ordering terms yields
E [aagDD (a,0)
E
[1 (s) (s) -BDO
8' 3'~Pp(3'le.
(1 A) M(s.jj N(m.a) By (2(s,a) - -BDQ((sa)
<
E (s,a)<DID
E ('MPo(3311sa)
1 1- A + AP (8, a) P (s,
[| (
E
E (('PP((81a..))
(s.0) D
[ (1-))Pp(s,a///bp(s))))))
E
[lle-NC. (BD. B
(148)
1''NPol8' (sx))DD(SxAD
s.a)
Assumption 3 implies that both Po a)) and Po(s, P)/Pp are bounded, i.e.
1 (s)
1
Po(s,a)
< <
3
S Bu, V(s ED.
(149)
B, p)(s,a)
Bi'
Ppr
32

---

Published in Transactions on Machine Learning Research (02/2026)
Applying the previous bounds to 148 yields
E
E ((s,)/D
11Qe + (s, -BDQX
8' (8
(1- 2) (N(aa) Σド=1 Ex(a By -
E (s,a)~DiD
E ('MPo((3lea))
1
(1 a)) - (x
E
E
(s,0)/DID
88' (P
(1 A) X
(Bx [lle . BDQX(s,
(150)
DxxA ENPPol' s.0)
Extracting the constant terms in the above expression outside expectations and using the fact that BDQ (s. a) and BD Qx do not depend on s' (at the step h + 1) yields
E
E
11122t((s,, (s. (D
8'^ P
(G.a)D
["|
N(s.a)
1-X
E
E
((s.a)
1
Σ
-
||]
x X/ BB (s.))
d'P (s.a)
+ (1 (1- A) (s,a)DD
E
[1((BoQ((s,a) BD (
+ -NN
NC
(s.))DD(SxA [I((BDQ((s.a) - (x
E
(151)
|| (Má..))
N(s.a)
B ( - B008(60a)) - |
1-X
$ 1 X A/BB (s,a)ED
max
E Pa (s,)
((s.o)
Σ ==1
+
(1 A) + te- NC
A
(I((Bo()(s,a) Bp Q2
(152)
(s.0)ESAA
By definition of Smax and Emax we obtain
E 8'~Po (e,0)
E (s.a)aD
(1 - (DQ
5 1 A/B v Smax (1- A) 8r + X Smax
+ e NNC Vmmax: Vk N.
(153)
This completes the proof of Lemma 2
Having introduced Lemma 2 we are in the stage of proving Theorem 4
Theorem (Convergence). Let the conditions of Theorem hold. Given any dataset Dez, it holds at each iteration (I: = 0,1,2..... that.
E 8f Pol'
E 11,00==(0_))))_____00"""))))")) ((), S
(s,a)>D
1- gh+1
1 Vmaxx +
((1 (a-x) Bi + x + +o-Nc) Vmma
xh1 E [s,a)DD
(lIQ((s,)) Q((, 1 1-7
11-1+1/3
(154)
33

---

Published in Transactions on Machine Learning Research (02/2026)
Proof. From the triangle inequality obtain
E
E ((sa))/
(s,a) - [112**(s,a))
Q""
(155)
8 P~Po(slaa.)
$ P' a.a) ((s,a)n
E [ (s,
- Q²,ss
(156)
Given that Q" is the fixed point of the Bellman optimality operator 2 (i.e., (s Qs,, obtain
E
E
[1Q (s,a) (s, - Q= (s, [|o
8' s'mPo(('(a,a)
(a.a)~D
<
E ('
11 - (Bot:)))-oc(ffllllllll-00000000 (BDQ (s)
(157)
Applying the contraction property (Sutton Barto 2018 of the Bellman optimality operator to the previous inequality and re-arranging the terms yields
[1IQ] [s, -x
8' (8')a()')s
(s.a)D
E
[1 (s,a) (1l
(158)
(2,a)D 3'~Po(3(
E
E
|| (s,a) - BD. -B..))BB((((s
3~~Po(8(a.a)
(s.a)~
+7 8'
E
E (e.0)DD
-
(159)
Unrolling the previous incquality until Q yields
(8, ((s0)
E
E (s.a)D
8¹ 3'~Po(8(sa.a)
<
E
(A.))D [11Q - BD
E
8/ 8'~'o(('
+ +7 5' - Po
E
E
[1[ a) (s,
(N,O)D
+
(s.))D [1, a) (=)
8'~Po((((s.a)
+ ++1 k+1
[s, Q* (s.
(160)
(("Pa((')_..) (s,a)D
Since Q is the initial Q-function, holds that
(s, = ()) S
(161)
34

---

Published in Transactions on Machine Learning Research (02/2026)
Employing Lemma
2 further implics that
[ - Qs, s
8' 3/~P' "BPP'8' (8))a (B.e)DD
1- A
+ A Buu Smaxt
(1- A) + + + =oo
A
V /max +...
1
V Smax +
(1- + +o- NC
v Smax
+ +g
[1l - Q*(ss
X + A/Bu
(e,0)D
(162)
= =g k+l E
(s, - (8())
(s,a)~D
1 1-1
1
+E-No) VEmax)
v Smax +
(163)
1-7
1-
3
A) Br +
This completes the proof of Theorem
A.8 Procgen Environments
We select five Proegen games (Cobbe et al. 2020 to substantiate our theoretical contributions in this work, whose details are provided below.
Description of Caveflyer (Cobbe et al., 2020). "The player needs traverse complex network of caves to reach the exit. Player movement reminiscent of the classic Atari game "Asteroids" where the ship can rotate and propel forward or backward along its current axis. The primary reward is granted upon successfully reaching the end of the level, though additional reward can be earned by destroying target objects with the ship's lasers along the way. The level is fraught with both stationary and moving lethal obstacles, demanding precise navigation and quick reflexes."
Figure 8: The screenshot of Caveflyer
Cobbe et al. 2020
Description of Climber (Cobbe et al.) 2020). "The player needs to climb a series of platforms, collccting stars scattered along the path. A small reward granted for each star collected, with a substantial reward provided for gathering all stars within level. If every star is collected, the episode terminates. The level is also populated with lethal flying monsters, adding extra challenges to the player's journey."
35

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 9: The screenshot Climber
Cobbe et al. 2020
Description of Dodgeball (Cobbe et al., 2020). "Inspired by the Atari game "Berzerk", the player spawns in a room with a randomly generated configuration of walls and enemies. Contact with wall results in an immodiate game over, terminating the episode. The player moves slowly, allowing for careful navigation throughout the room. Enemies, moving slowly too, throw balls at. the player. The player can retaliate by throwing balls as well, but only in the direction they are facing. Once all enemies are eliminated, the player can advance the unlocked platform, carning substantial level complction bonus."
Figure 10: The screenshot of Dodgeball (Cobbe et al. 2020
Description of Maze (Cobbe et al., 2020). "The player, embodying a mouse, needs to navigate a maze locate the sole piece of cheese and obtain a reward. The mazes, generated using Kruskal's algorithm, vary in size from 3 x 3 25 x 25, with dimensions uniformlyy sampled across this range. To navigate the maze, the player can move up, down, left, or right."
Figure 11: The screenshot of Maze Cobbe al. 2020).
Description of Miner (Cobbe et al. 2020). "Inspired by the game "BoulderDash", the player (robot) can dig through dirt navigate the world. The game world is governed by gravity, where dirt supports both boulders and diamonds. Boulders and diamonds fall through free spaces and roll off each other. If either boulder diamond falls the player, the game terminates immediately. The objective is to collect
36

---

Published in Transactions on Machine Learning Research (02/2026)
all the diamonds in the level and then reach the exit. The player carns small reward for each diamond collected and a huge reward for successful completion in the level."
Figure 12: The screenshot of Miner (Cobbe et al. 2020
Proegen levels. An example (Maze) with different Procgen levels is provided in Figure 13
+
F
Figure 13: Maze with different levels.
A.9 Procgen Experimental Hyperparameters
Key hyperparamoters for the datasets and algorithms are summarized in Table 5
Table 5: Experimental hyperparameters.
Hyperparameters
Value
Target domain levels
100, 199]
Source domain levels
[0, 99). [25, 124), [50, 149)
Number of target samples (N)
1000, 2500, 4000
Number of source samples (N')
40000
Weight (A)
{0. 0.2, 0.4, 0.5, 0.6, 0.8, 1}
Number of episodes for evaluation
500
Learning rate
0.0005
Batch size
256
Neural network hidden size
256
Discount factor (Y)
0.99
CQL conservativeness constant (a:)
4
Gradient norm clip
0.1
IQL expectile (Texp)
0.8
IQL temperature (3)
0.1
37


Reviews:
[{"review_id": "9WpQAktDAG", "reviewer": "Reviewer_cXkD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes the first framework that theoretically explores the impact of the weights assigned to each dataset on the performance\nof offline RL.  Experiments on the Procgen benchmark prove the point.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In the 1.1 related work, Offline reinforcement learning with a domain adaptation method ignores the weight's importance. The paper discusses the impact of the weights assigned to the two datasets on the performance of offline RL.", "audience_yn": "Yes", "audience_explanation": "A good solution for offline RL applications that don't have more data.", "requested_changes": "Several problems need to be clarified,\n1. The theoretical results characterize the optimal $\\lambda^\\*$ in terms of $\\varsigma$, $\\xi$, and $\\beta$, but these quantities are not directly observable in practice. Do you have a concrete, implementable procedure to choose \\lambda without access to the true MDP dynamics?\n2. Have the authors attempted any preliminary experiments outside Procgen, and if so, what were the outcomes?\n3. In experiment setups, are there concrete examples where $\\lambda = 0$ fails primarily because of coverage gaps rather than sample size?", "broader_impact_concerns": "No more Broader Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WkyMcF1YO9", "reviewer": "Reviewer_99q4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies hybrid offline reinforcement learning, where the agent optimizes its policy with a small set of offline data from a target domain alongside a substantially larger dataset from a source domain. Since the target data is scarce, it is insufficient to learn a meaningful policy on its own. However, leveraging the data from the potentially mismatched source domain can introduce generalization gaps in the final policy. Therefore, a straightforward choice is to coordinate the usage of both datasets with a mixing hyperparameter, denoted as $\\lambda$ in this paper. \n\nMotivated by this, this paper theoretically studies the relationship between $\\lambda$, the size of the target dataset, and the dynamic bias. The major results include the expected and worst-case bounds on the differences between the empirical TD error and the expected TD error in the target domain. Besides, under simplifying conditions, this bound also implies the existence of an optimal weight $\\lambda$. Finally, the experimental section subsequently validates the theoretical corollaries, particularly the trend of the optimal $\\lambda$ as a function of the target dataset size and the dynamics gap.", "claims_and_evidence_yn": "No", "claims_explanation": "Part of the claims presented in this paper are supported by convincing theoretical analysis and empirical results, while others require clarification or correction. Here is a detailed comment. \n\n1. Existing hybrid RL methods, including H2O and ORIS, are **mispresented** and therefore compromise the significance of the optimization objective studies in this paper. This paper studies a scheme where the critic loss is constructed by mixing the loss from the target domain dataset and the loss from the large source domain dataset using a constant mixing weight $\\lambda$.  It then claims that existing hybrid RL methods, such as H2O and ORIS, can be recognized as instantiations of this approach with specific choices of $\\lambda$. However, this claim is not correct. To be specific, H2O utilizes an importance-sampling-like weight to account for the dynamics mismatch between the source domain ($D'$) and the target domain ($D$). Specifically, it learns a discriminator to capture the transition ratio and uses this ratio to weight the samples from the source domain (as shown in Equation 8 of [1]):\n$$\\mathbb{E}\\_{s'\\sim P_{D'}(s'|s, a)}[\\frac{P_D(s'|s, a)}{P_{D'}(s'|s, a)}\\mathcal{E}(Q, s, a, s')]$$\nHowever, this paper claims that H2O is using the unweighted expected error:\n$$\\mathbb{E}\\_{s'\\sim P_{D'}(s'|s, a)}[\\mathcal{E}(Q, s, a, s')]$$\nSimilarly, ORIS uses a state-specific weight that quantifies the divergence between the dynamics at state $s$ to adjust the weight for each sample strategically. The constant weighting scheme employed and analyzed in this paper cannot encompass H2O and ORIS and may not be suitable for practical applications.\n\n2. The final theoretical results implied the scaling trend between the optimal weight $\\lambda$ and the dataset size and dynamics gap, respectively. This is supported and validated by the experiment section. \n\n\n[1] When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning.", "audience_yn": "Yes", "audience_explanation": "Yes. The problem studied in this paper is hybrid RL, where a probably mismatched simulator and a handful of offline interactions from the target domain are available for reinforcement learning. This combination effectively mitigates the critical challenges associated with using either data source independently: direct learning in the target domain is often prohibitively sample-costly, while merely learning within the simulator suffers from significant sim-to-real gaps. The final theoretical finding, although conceptually straightforward, establishes a connection between the mixing weight and the dataset/simulator conditions, which can potentially guide the hyperparameter choice prior to the actual learning process.", "requested_changes": "1. (Major concern) Please address my concerns about the misrepresentation of established hybrid RL methods if such misrepresentation does exist. Besides, to demonstrate the necessity and efficacy of the simpler weighting scheme analyzed in this paper, please explicitly compare and contrast it with the more complex, sample-specific weighting used by H2O and ORIS.\n\n2. The current interpretation of the dynamics gap across different Procgen task levels is unclear. For the reader to properly interpret your theoretical claims and empirical results, please articulate how the dynamics vary between levels.\n\n3. Experiments on locomotion or manipulation tasks (e.g., those in Gym-MuJoCo or DeepMind Control Suite) are strongly recommended. These environments are established testbeds for hybrid and domain-adaptation RL, as prior works (including H2O and ORIS) have focused on them. It is easy to create a series of tasks that differ in transition dynamics by tweaking certain parameters of the simulator, such as the gravity and friction coefficient.", "broader_impact_concerns": "N/A.", "additional_comments": "Minor issue: It would be clearer to separate the notation used for the domains (the underlying dynamics) and the datasets (the empirical collections of transitions). Currently, they all share the notation of $\\mathcal{D}$ (with or without hats), and this can lead to confusion when distinguishing between empirical operators and expected operators.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3mouYdq5QB", "reviewer": "Reviewer_YFb3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies offline reinforcement learning where you have a small, target dataset and an unlimited but biased source dataset, e.g., a simulator, and asks how to optimally combine them. The authors propose an algorithm-agnostic framework which updates Q by minimizing a $\\lambda$-weighted TD loss over target and source transitions and develop a theory that decomposes error into variance from limited target samples and bias from the dynamics gap between source and target MDPs. Within a tabular setting and under standard coverage and boundedness assumptions, they derive expected and worst-case performance bounds, show the algorithm converges to a neighborhood of the optimal $Q^{\\star}$, and, under simplifying assumptions on sampling ratios, obtain a closed-form optimal weight $\\lambda^{\\star}$ which balances variance and bias and is typically not one of the trivial choices using only source, only target, or equal weighting. They have also conducted experiments on the offline Procgen benchmark with CQL and IQL backbones showing that performance as a function of $\\lambda$ qualitatively matches the theoretical predictions and that naive weighting choices are often suboptimal, though the theoretically optimal $\\lambda$ depends on quantities (eg. dynamics gap) which are hard to estimate in practice.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper presents a neat and unifying formulation, and is overall well written. The derivations in the tabular setting under exact minimization and coverage clearly support the basic theoretical claims: 1) that $\\\\lambda$-weighted TD updates induce a bias–variance tradeoff between source and target data, 2) that one can obtain expected and worst-case bounds on error, and 3) that there is, under simplifying assumptions, a closed-form optimal $\\\\lambda^*$ balancing dynamics gap and variance. The main results appear technically sound at a high level. However, all the provided guarantees depend on a set of relatively strong assumptions: a finite tabular MDP, exact minimization of the TD objective, full coverage of the state–action space, and effectively unlimited source data such that expectations under $D'$ can be treated as known. In my opinion, another challenge is thereby introduced in establishing the practicality of the proposed algorithm.", "audience_yn": "Yes", "audience_explanation": "I think the paper offers a useful, theoretically grounded perspective on a problem of broad interest in the community. It is well written and, in my opinion, would add value to the current body of literature.", "requested_changes": "The worst case performance bound, Theorem~3, has a dependence on $|\\mathcal{S}||\\mathcal{A}|$, a number of $\\\\beta$-type constants, and the usual $1/\\sqrt{N}$ and thus is almost certainly vacuous in non-tiny settings. The analysis itself appears to be standard in spirit (Hoeffding-type plus a union bound) though the paper does not illustrate numerically how large these bounds are even for modest $|\\mathcal{S}|$ and $|\\mathcal{A}|$. Furthermore, the bound is used mainly to argue that ``we have a generalization guarantee,'' rather than to inform algorithmic design or the choice of parameters. Given the TMLR focus, such work is strengthened either by (1) providing a small-scale example where the bound is numerically non-trivial or (2) explicitly stating that the worst case bound is primarily of conceptual interest.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)