MOORL: A Framework for Integrating Offline-Online Reinforcement Learning
Authors: Gaurav Chaudhary
Washim Uddin Mondal
Laxmidhar Behera
Venue: Accepted by TMLR ()

Abstract: Sample efficiency and exploration remain critical challenges in Deep Reinforcement Learning (DRL), particularly in complex domains. Offline RL, which enables agents to learn optimal policies from static, pre-collected datasets, has emerged as a promising alternative. However, offline RL is constrained by issues such as out-of-distribution (OOD) actions that limit policy performance and generalization.
To overcome these limitations, we propose Meta Offline-Online Reinforcement Learning (MOORL), a hybrid framework that unifies offline and online RL for efficient and scalable learning. While previous hybrid methods rely on extensive design choices and added complexity to utilize offline data effectively, MOORL introduces a meta-policy that seamlessly adapts across offline and online trajectories. This enables the agent to leverage offline data for robust initialization while utilizing online interactions to drive efficient exploration. Importantly, MOORL addresses the key challenges of hybrid RL in terms of being design-free.
Our theoretical analysis demonstrates that the hybrid approach enhances exploration by effectively combining the complementary strengths of offline and online data. Furthermore, we demonstrate that MOORL learns a stable Q-function without relying on extensive design choices. Extensive experiments on 28 tasks from the D4RL and V-D4RL benchmarks validate its effectiveness, showing consistent improvements over state-of-the-art offline and hybrid RL baselines. With minimal computational overhead, MOORL achieves strong performance, underscoring its potential for practical applications in real-world scenarios.

Published in Transactions on Machine Learning Research (05/2025)
MOORL: A Framework for Integrating Offline-Online Rein- forcement Learning
Gaurav Chaudhary
paurauch@ittk.cc..
Deportment of Electrical Engineering Indian Institute of Technology Konpur
Washim Uddin Mondal
umondalWiitk..u
Department Electrical Engineering Indian Institute of Technology Kanpur
Laxmidhar Behera
Ibehera Britk. accitk..c..k.
Department of Electrical Engineering Indian Institute of Technology Kampur
Reviewed on OpenReview: /ttps://openwee opw.revveepre..ee.e.mmmm net/ forum? id- FH2
Abstract
Sample efficiency and exploration remain critical challenges in Deep Reinforcement Learn- ing (DRL), particularly in complex domains. Offline RL, which enables agents to learn optimal policies from static, pre-collected datasets, has emerged as a promising alternative. However, offline RI. is constrained by issues such as out-of-distribution (OOD) actions that limit policy performance and generalization. To overcome these limitations, we propose Meta Offline-Olline Reinforcement Learning (MOORL), hybrid framework that unifies oflline and online RI for efficient and scalable learning. While previous hybrid methods rely on extensive design components and added computational complexity to utilize offline data effectively, MOORL introduces a meta-policy that seamlessly adapts across offline and online trajectories. This enables the agent to leverage offline data for robust initialization while utilizing online interactions to drive efficient. exploration. Our theoretical analysis demonstrates that the hybrid approach enhances exploration by effectively combining the complementary strengths of offiine and online data. Furthermore, demonstrate that MOORL learns stable Q-function without added complexity. Extensive experiments 28 tasks from the D4RL and V-D4RL benchmarks validate its effectiveness, showing con- sistent improvements over state-of-the-art offline and hybrid RL baselines. With minimal computational overhead, MOORL achieves strong performance, underscoring its potential for practical applications real-world scenarios.
1 Introduction
Deep Reinforcement Learning (DRL) has been tremendously successful solving a variety of complex problems, including robotics (Tang al. 2025), autonomous driving (Kiran et al. 2021), healthcare Yu et al. 2021 game-playing (Silver et al.) 2017 Vinyals et al. 2019), intelligent perception system (Chaudhary et al. 2023 and finance [Charpentier et al. 2021p. However, algorithms is their sample inefficiency, i.c., the number of state-action-state transition samples they require to train a policy. Typically, these algorithms require millions of such interactions, making them impractical in real-world scenarios, particularly in safety-critical domains like robotics and autonomous driving (Kiran ct al. 2021). Learning policies controlled simulation environments can offer a partial remody, as these policies often fail generalize to real-world situations due to the well-known simulation-to-rue to-oonnttt---rrtyyy (sim2real) gap (Tobin et al. 2017
of the primary drawbacks of DRL

---

Published in Transactions on Machine Learning Research (05/2025)
An effective strategy to mitigate these problems is Offine Reinforcement Learning (RL) (Levine et al, 2020), which enables policy learning from historical datasets without requiring online exploration. Offline RT, can train policies safely and cost-effectively using pre-collected human demonstrations or previously logged interactions. Although it alleviates some concerns related to sample complexity, the reliance on static offline data introduces new challenges, such as extrapolation errors and out-of-distribution (OOD) actions (Bai ct al., 2022), which can result in sub-optimal behaviors when policies are tested in real environments or unfamiliar states (Kim al. 2024). Ofline-to-Online (O20) RL (Lee al. 2022; Wagenmaker & Pacchiano 2023) dilutes the limitations of purely offline RL to some extent. 020 RL setups typically pre-train the agent using offline data, followed by fine-tuning via limited online interactions. However, this approach often experiences performance drops due to compounded Bellman errors (Sun et al.
to changes in reward distributions and distributional shifts between oflline data and online interaction FFarhhmad et al. 2010
Munos 2005).
In this article, we aim to tackle the inherent challenges faced by offiine RL and online RL algorithms. We believe that directly integrating oflline data into online RL training can lead to more stable learning and mitigate issues such out-of-distributin (OOD) actions and inefficient exploration. While recent efforts, such as RLPD (Ball et al. 2023) and Hy-Q (Song et al., 2022), have attempted to address offlinc-online integration, each comes with its own challenges. RLPD, which combines offline and online data, introduces extensive design components (refer to Appendix E RLPD employs a large Q-ensemble and a high Update- to- Data (UTD) ratio to stabilize learning and optimize performance. These dependencies add complexity to the algorithm, making it more challenging and computationally intensive, limiting its scalability.
On the other hand, Hy-Q offers a more streamlined approach by integrating offline data into online train- ing without necessitating extensive design components. However, Hy-Q has its own limitations; it requires maintaining separate Q-value functions for each timestep within fixed horizon, significantly increasing computational and memory overhead, especially in environments with longer or variable horizons. Addition- ally, Hy-Q's reliance on a predefined horizon makes it less adaptable to tasks with dynamic episode lengths, limiting ils scalability and flexibility across diverse RL scenarios. While Hy-Q reduces the need for many design components compared to RLPD, it still struggles with computational efficiency and adaptability in more complex or heterogencous environments.
Further, as highlighted by (Furuta et al. 2021 Engstrom et al. 2019 Henderson et al. 2018 the RL algorithms are difficult to optimize and tune, where minor hyperparameter changes can have a non con-trivial impact on performance. We believe it is important to limit RL algorithm design components. These limitations underscore the necessity for a hybrid approach that effectively integrates offline and online data and maintains computational efficiency and adaptability across various tasks. With this motivation in this work, we propose a framework called Meta Onlline-Onliie RL (MOORL) that addresses the stated issues by utilizing a unified set of design components without the need for large Q-ensemble, high UTD, and separate Q-function per horizon step, offering more robust and generalizable solution for hybrid reinforcement learning. In particular, our contributions can be summarized as follows.
We provide theoretical insights showing that mixing offline and online data can affect performance and provide a performance bound on the expected reward.
We leverage the off-policy RL framework, Soft- Critic (SAC) HHarnoja et al. 2018 to seamlessly integrate offline and online data via meta-learning for efficient design-free policy learning without introducing any new hyperparameters.
Our proposed framework, MOORL, uses meta-learning principles Finn et al. 2017 Nichol & Schul- man 2018) to train policies under a single meta-objective, enabling the dynamic balancing of offline and online data. The learned meta-policy adapts across varying distributions, minimizing the impact of distributional shifts and extrapolation errors.
We validate our methodology through 28 comprehensive experiments on benchmark D4RL (Fu et al., 2020) and V-D4RL. (Lu et al. 2022) environments, demonstrating that MOORL outperforms state- of-the-art methods in reward accumulation while being stable across diverse tasks, including dense and sparse reward scenarios.
2

---

Published in Transactions on Machine Learning Research (05/2025)
2 Preliminaries
2.1 Markov Decision Process
P),,, where S indicates the state space, We consider a Markov Decision Process (MDP) M (S, R.
A denotes the action space, T: S - (S) represents the state transition dynamics (where ASS) defines the collection all probability distributions over S), R AxAxS + R the reward function, 7 E (0, 1) is the discount factor, H the length of the horizon of the episodes, and p A(S) is the initial state distribution. At each time instant I, the agent observes the state se; execules an action at, and a result, transitions to the next state st+1 T(-]st, (e)), ae), and receives reward learning is to learn (stochastic stationary) policy и : S + ((A) that maximizes the expected cumulative reward J = E | 7're p where the expectation is obtained over x-induced trajectories of length и that start from the initial state distribution, p. The state- action distribution under policy и is defined as d= (s, (1 7) Er 7a P(s, 8,0, r)) The state-value and state-action value functions are defined, respectively, (s) E - 80 = and Q (s, a) E Dito >'r, 80 8,00 a, where the expectations are obtained over π n-induced trajectories of length H. Note that Jn = Eepp [V* In this paper, we obtain the optimal policy that maximizes J= by combining offline and online data using ideas from mela-reinforcement learning, which is described below.
R St+1). The goal in reinforcement
2.2 Reta-reeifoorcement Learning
Meta-RL aims to solve distribution of tasks given by PT(-), rather than a single fixed task, where each task is characterized by an MDP M₁ = Ai, T;, H;, R2, Yi, Ps). Tn our setting, the meta-learning process is used to combine offline and online data. Following the meta-RL paradigm, we maintain separate replay bullers for each lype of data, denoted as Dottine and Dontine respectively, which store transition tuples in the form (st, 04, T4, s(). The training process alternates between sampling from these replay buffers update the policy while adapting to the changing data as the agent gathers more online experiences. A meta-cpisode consists of sampling data from either of these replay bulfers and forming the associated trajectories to update the meta policy. A significant challenge arises from the distributional shift between offline and online data, which can lead to instability in the training process. To mitigate this issue, our meta-RL RL approach employs gradient- wrasient-based meta-learning to effectively balance updates between the offline and online data sources, enhancing the robustness of the policy against. distribution mismatches.
3 Why Meta Learning?
Metta-lerrning or 'learning to learn is a powerful paradigm that performs well across diverse task distri- butions (Finn al.) 2017 Nichol & Schulman 2018). By leveraging meta-learning to integrate offline and online data, Meta Offline Reinforcement Learning (MOORL) can achieve the following benefits:
Reduced Extrapolation Error and Improved Distribution Generalization: MOORL adapts to changing data distributions by training meta-policy over online and offline data. This approach minimizes extrapolation errors by optimizing the meta-policy to generalize across distributions rather than being confined to a single dataset (Finn et al. 2017 Garcia & Thomas 2019).
Improved Credit Assignment: MOORL utilizes a meta-objective that optimizes task perfor- mance across various data sources, helping isolate beneficial behaviors. By employing gradient-based met meea-leanni techniques, the meta-policy can assign credit more accurately, focusing on the tra- jectory aspects that generalize well between online and offline data (Finn et al. 2017 Al-Shedivat ct al. 2018
Multi objectivity: MOORL streamlines the learning process by employing a single meta-objective that integrates agent and expert data without extensive design choices. This meta-objective is robust across varying distributions, automating many design decisions that would require manual adjustments (Chen et al. 2019 Ye et al. 2021
3

---

Published in Transactions on Machine Learning Research (05/2025)
4 Integrating Online RL with Offline Data
This work aims to bridge the gap between online and offline reinforcement learning by presenting unified approach that integrates both paradigms without introducing extensive design elements and large computa- tional overhead. Our method utilizes an off-policy reinforcement learning algorithm (Haarnoja et al. 2018) to effectively leverage data from various distributions. The proposed framework, MOORL, designed to handle variations in offline data quality, making it adaptable to various scenarios. Additionally, MOORL demonstrates consistent performance across different problem settings, including environments with state- based or pixel-based observations, well as those with dense, sparse; this, first offer theoretical insights into why combining offline data with online agent data may be a more effective strategy than relying solely on online learning. We then introduce the MOORL framework, highlighting its simplicity and computational efficiency (refer to Appendix
binary rewards. To support
4.1 Expected Reward and its Performance Bound
In this section, we examine the expected cumulative reward when sampling trajectories from mixed dis- tribution D, which combines data generated by an offline policy l and an online policy л. We derive a performance bound for the expected reward difference relative to the online policy, leveraging the total vari- ation distance as measure of distributional discrepancy between the offline and online state-action visitation distributions.
4.1.1 Problem Setup
Consider offline policy μ,, which generates the offline dataset, and online policy æ,, which interacts with the environment. Let 'D denote mixed dataset of trajectories, where fraction À E [o, 1] of the trajectories are sampled from μ, and the remaining induced by a trajectory randomly sampled from D is defined as:
are sampled from x. The state-action visitation distribution
(s Ad a) + (1- (s,"
(1)
Here, d' d' a) and (s)(sa) represent the state-action visitation distributions under policies μ and x, respec- tively, over the state-action space S x A. The expected cumulative reward generated by trajectories sampled from D is given by (Durukaar 2023
[
Σ ((s,))
(2)
(s.a)
Here, R(s,a) (R(s,
Our goal is to quantify the performance difference between the expected reward under the mixed distribution D and that under the online policy x:
AR = ED( [R(
(3)
4.1.2 Performance Gain Expression
The expected reward under the online policy и is:
E R
Σ
d' d',ss)))(ss
(4)
(s,a)
Thus, the performance gain AR becomes:
AR
ds, (d, (o5 R(s.a)
(5)
(5.a)
4

---

Published in Transactions on Machine Learning Research (05/2025)
Substituting (s. Ad (s. + (1 A)d a), we obtain:
AR
À
Σ (+.a)
(s) dsss
(6)
This expression indicates that the performance gain depends on the difference wetween the offline and online distributions, weighted by R scaled by X.
4.1.3 Bounding the Performance Gain with Total Variation Distance
To bound AR, we introduce the Total Variation CCoer 1999 distance between the distributions d' d a) and a). The Total Variation distance between d" (8, a) and ds(s,a) is defined as:
TV(d", d²))
Σ (*.0)
d (s d' (s.
The total variation distance quantifies the maximum difference probability assigned to any event by the two distributions and satisfies < TV(d", dd"d",d") I 1. It provides a natural metric for comparing d= ds and ds, a) in the context of performance analysis.
For any bounded function f(s,a):
Σ (d) - (xx, (8.a)
$
Σ (s, dx 1/ (e.0)
(8)
where | (a.)) R(s, a). If the reward function is taken be bounded, i.c., |R(s, a)l S Ruuxx v(s,a), then from equation and the definition of TV (equation we arrive at the following.
Σ (d) dt (s,a)) 2 TV((V ( Raax (s,a)
(9)
This leads the following bound on the performance gain.
2XRmax
AR < (d" d') 1 1-7
(10)
Pinsker's inequality (Pinsee 1964 provides an upper bound on the TV distance between two probability distributions in terms of their Kullback-Leibler (KT.) (Kulbbck & Leibler 1951 diverreeee
TV d²) < V Drt (dtT || du)
(11)
Using this definition, the final performance bound can be defined as:
XRmax dd). 1-7
AR S 22Dx (d
(12)
The above result suggests that the performance difference between the mixed distribution D and the online policy и is influenced by the total variation distance TV dd)) and the mixing parameter X. Specifically, when (V d"') is small, that the offline and online policies induce similar state-action distributions, the expected reward under D closely approximates that under w, suggesting stability in integrating the two data sources. Conversely, a large TV(d", dd) implies a potentially significant deviation in cumulative rewards, highlighting the risk of distributional mismatch when combining offline and online data.

---

Published in Transactions on Machine Learning Research (05/2025)
Importantly, the performance gain AR depends on the relative quality of the offline policy com- pared to the online policy т. For instance, if μ is an expert policy, Ep[R] > [x[R] is expected early in training when т is suboptimal, resulting in a positive AR > However, as training progresses and x improves, TVdd", dtt) may decrease, reducing the benefit of mixing. This dynamic suggests that the value of offline data changes over time, with the greatest benefits often realized the initial training phase.
Crucially, this analysis is not limited to expert offline datasets. For non-expert datasets, where μ may be suboptimal or diverse, the evolving difference between dx and de still governs the effectiveness of data mixing. In such cases, an adaptive approach is essential to manage the shifting relationship between offline and online distributions, ensuring that the policy can leverage useful information from oflline data while avoiding negative impacts as π matures, This motivates the design of MOORL, which employs meta-learning dynamically balance offlinc and online data, optimizing performance across diverse datasct qualities and training stages.
4.2 MOORL: Meta Offline-Onlinn RL
This section introduces the Meta Offline-Online Reinforcement Learning (MOORL) framework, which ad- dresses the challenges of integrating offline and online data in off-policy reinforcement learning. MOORL combines the strengths of off-policy learning (Haarnoja et al. (0188) and meta-learning (Finn al. 2017) leveraging offline data for efficient learning while enabling online exploration, all while ensuring stable Q- learning updates. By employing meta-learning strategy to dynamically adapt Q-function updates, MOORL mitigates issues such distributional mismatch, overestimation bias, and instability in Q-learning,
4.2.1 Problem Definition
The task to learn a policy using two distinct data distributions:
Offline Data (Doffine): This data consists of trajectories previously collected from one more policies. While offline data may include high-reward sequences, it is often derived from sub-optimal or outdated policies, leading to potential biases. Direct incorporation of this data into training may cause overestimation bias in learned Q-values due to limited diversity and representativenes of experiences.
Online Data (Dontine): This data is collected through interactions with the environment based on the current policy. Initially, online data may yield low rewards due to early-stage exploration, but typically improves the policy refines.
The primary challenge comes from the distributional mismatch between offline and online data. Directly mixing these two distributions in off-policy RL algorithms can lead instability in Q-learning, as the value estimates can become biased lowards the high-reward oflline data, resulting in overestinated Q-values. The proposed MOORL framework addresses this by learning meta Q-function (s, a; metta), parameterized by fmeta. that aims to generalize across both offline and online data distributions. The meta Q-function is optimized using a meta-Q objective using Reptile (Nichol & Schulman 2018) meta-learning algorithm, balancing contributions from offline and online data.
4.2.2 Meta Q-Function Learning
MOORL learns robust meta Q-values that generalize across oflline and online data distributions to mitigate overestimation bias. The MOORL framework aims to learn meta Q-function Q(s, o; @meta); parameterized by metaa that generalizes across both offline and online distributions. The paramcter fmeta can be interpreted as solution to the Bellman error minimization problem stated below.
[((ss, (meta)) - (r 7---BeTTTafl max Q(s', a'; motaa) (mmocs))) }))²
min E (s.a.na// Ometa
(13)
6

---

Published in Transactions on Machine Learning Research (05/2025)
Algorithm MOORL: Meta Ofline-Online Reinforcement Learning
1: Initialize: Mcta-policy paramcters actor 9meta, critic metaa: and temperature
2: Offline dataset Dofline and empty online buffer Donline.
3: Meta-loarning rate 1/meta; inner-loop learning rate 1, number of iterations N.
4: for n = 1 to N do
Select Buffer: Choose Dotline or Donline as the data buffer.
Inner-loop Adaptation:
Collect transition in online environment and store in Dolline.
8:
Sample mini-batch from the selected data buffer.
9:
Perform K inner actor o and critic 0 updates using data from Dj.
10:
Meta-update:
11: Update meta-policy parameters of both actor and critic using Pmeta t Ometa - Mmmta Practa [C(5]
and
metaa < metta Tmmeta Dmeeta [C))),, respectively.
12: end for
where D is a dataset containing both offline and online data.
The combined loss ensures that Q(: metaa) provides consistent value estimates across both offline and online data. However, this objective is similar to mixing offline and online data distribution as done by RLPD (Ball et al. 2023). which simply mixes data from different distributions and learns a Q-function that can generalize across data distributions, which requires different design choices to stabilize learning. In this work, we apply meta-learning perspective. Specifically, our algorithm progresses multiple epochs, performing one meta- policy update each epoch. At the start of an epoch, we randomly choose either the offline or the online replay buffer and perform K inner updates for distribution adaptation, followed by a mela-update.
Specifically, for given data distribution, i {offline, online), we take mini-batch B, D, of length B, and define the following loss functions for inner loop adaptation of the sampled distribution.
Caritic (0) =(...,,,")))B; [Q(s, a; 0) (r + QQss ('s((('''''))))) d':
(14)
where 0' is the parameter of target function that is used for stabilizing the learning. Following Q-learning paradigm, 9 and 6' are synced periodically. The above loss is used to update the critic parameter, 6, via a gradient-descent approach. The weights of the critic in the inner update loop are initialized with mela-crilic weights, followed by K inner loop gradient steps for distribution adaptation. where r) denotes a learning parameter. Mathematically, each gradient descent step is defined as follows.
0 - 1-0--VVee V.
(15)
Let the final critic parameter (after K inner loop steps) be O. We similarly define actor loss function follows, where denotes the parameter of the actor (policy approximator), and a is a tunable parameter.
Cactor (Φ, 0) = (en (a log (als; Φ) -Q(s, 0z
(16)
Starting from meta-policy weights, the parameter ò is also updated K times as follows.
& - v Cactor (g.
(17)
Here, we use the principle of Soft Actor Critic Haarnoja et al., 2018) that maximizes the expected reward (Q-value) while ensuring sufficient exploration through entropy regularization controlled by the temperature parameter a. The entropy lerm encourages the policy to remain stochastic, promoting diverse actions and balancing exploration and exploitation. Let the final value of the actor parameter be After inner loop distribution adaptation, in the outer loop, we update the meta actor and critic paramcters Quetas Queta using the updated inner loop paraneters follows, where Tmmeta is a tunable learning parameter.
0 <- Ometa Mmeta Vo (()
(18)
Ometa @meta Ameta V Ummor L( 01]
(19)

---

Published in Transactions on Machine Learning Research (05/2025)
For computing meta-updates Pwgeta and meeta in the last step in the direction of @meta - ~ and meta 0. we treat Ometa á and O - as gradient similar to (Nichol & Schulman 2018) and plunge it into an adaptive algorithm such as Adam (Kingma) 2014]. In summary, the learning process consists of two steps: first, adapting the distribution through K inner updates, followed by a meta-update aimed at generalizing across distributions
Hence, by leveraging meta-learning capabilities of task adaptation (Finn et al. 2017;; we enable adaptation across diverse data distributions generated by different policies. This distribution adaptation strategy allows the learned Q-function to approximate combined Bellman Q-function. Further, Figure | the Q-values learned via MOORL and RLPD exhibit similar convergence trends. This demonstrates that learning a combined Bellman function, as done by RLPD or exploring a meta Q-function, yields comparable Q-values. However, MOORL offers the advantage of not relying on specific design choices for stability, highlighting its robustness and simplicity.
illustrates that
5 Experiments
We evaluate the proposed MOORL approach on the D4RL benchmark (Fu al. 2020) and V-D4RL (Lu et al. 2022), comparing its performance against state-of-the-art methods, including hybrid RT approach RLPD (Ball et al. 2023), Hy-Q (Song et al. 2022) and offline RL method ReBRAC (Tarasov et al. 2024 for completeness. Each baseline has distinct design choices and operational paradigms that provide val valuable context for evaluating MOORL's strengths in hybrid offline-online RL settings. Through our chosen baselines and benchmark tasks, we aim to address the following questions:
Can MOORL effectively integrate offline data into an online RL sctting without extensive design- specific configurations?
Does MOORL ensure stable learning across diverse data distributions, minimizing the need for task-specific tuning?
Can MOORL be extended to high-dimensional image-based observation data?
Does MOORL maintain consistent performance across varying offline data qualities by learning stable Q-function?
5.1 Evaluation on Offline D4RL Tasks
To evaluate MOORL's performance, we select a range of D4RL tasks to assess robustness across diverse data dist risut
D4RL Locomotion(Pu et al. 2020): This set includes 15 dense-reward locomotion tasks, with offline data covering varying levels of optimality, from expert to random trajectories.
D4RL Maze-Navigation(FN) et al. 2020 We utilize 6 AntMaze navigation tasks with sparse binary reward structures, each with different complexities.
D4RL Adroit Fuu et al. 2020 The tasks in this set (Pen, Door, Hammer) involve complex manipulation and sparse rewards, with offline data consisting of expert-level trajectorics.
5.1.1 Choice of Baselines
Hybrid RL: We evaluate the performance MOORL against current state-of-the-art hybrid RL ap- proaches inchuding RLPD (Ball et al.) 2023) and Hy-Q (Sogg al. 2022). These approaches aim to inte- grate offline and online learning. To ensure stable learning, they introduce many design elements, resulting in computational overhead and limiting their broader impact real-world scenarios.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: The performance comparison across Locomotion tasks presented as the average normalized score across 10 seeds. The symbol = denotes the standard error of the mean.
Task Name
TD3+BC
ReBRAC
Hy-Q
RLPD (UTD=20)
MOORL, our
half-cheetab-expeett
93.4 + 0.1
105.9 + 0.6
84.0
111.1 + 0.1
105.9 + 0.3
89.1 1.9
101.1 1.7
86.0
105.8 0.1
103.4 +
45.0 0.4
51.0 + 0.3
89.0
72.2 + 0.1
96.9 0.7
cheetah-
54.7 = 0.3
65.6 1 0.3
88.0
85.4 + 0.1
99.2 + 0.7
half-cheetah-randee
30.9 = 0.1
29.5 - 0.5
80.0
85.1 - 3.3
99.0 1.6
hopper-expert
109.6 + 1.2
100.1 + 2.8
54.0
101.7 + 5.5
111.6 + 1.4
hopper-meedhum-exper
87.8 3.5
107.0 + 2.1
100.0
97.1
101.5 +
hopper-mechuu-replay
55.1 + 10.6
98.1 + 1.8
77.0
81.3 + 0.8
99.6 1.4
hopper-medium
60.9 + 2.5
102.0 + 0.3
106.0
90.8 + 0.7
107.9 + 0.8
hopper-random
8.5 + 0.2
8.1 + 0.8
80.0
92.9 + 0.9
101.9 1.1
wa walker2d-expert
110.0 + 0.2
112.3 + 0.1
112.0
127.5 + 1.9
123.6 + 1.2
walker2d-medium expprt
110.4 + 0.2
111.6 0.1
95.0
128.0 - 1.8
117.2 - 2.5
waller2dd-medimn--plll
68.0 6.4
77.3 + 2.6
103.0
105.7 + 2.4
111.2 1.2
walker2d-medium
77.7 1.0
82.5 + 1.2
86.0
115.1 + 2.1
114.1 + 0.6
walker2d-random
2.0 + 1.2
18.4 E 1.5
90.0
73.7 10.6
93.8 + 0.3
Average
66.9 + 3.0
78.0 + 1.0
88.7
98.2 + 3.7
105.8 + 1.1
Table 2: The performance comparison across AntMaze tasks is reported as the average normalized score over 10 seeds. The symbol + represents the standard error of the mean. For RLPD, results with UTD=20 and UTD=5 correspond to 300K timesteps, whereas UTD=1 results are based on 1M timesteps.
Task Name
TD3+BC ReBRAC
Hy-Q
RLPD
MOORL, our
UTD=20
UTD=5
UTD=1
autmaze-umare
66.3 + 2.1
97.8 + 0.3
99.0 : 0.2
99.0 + 0.2
88.2 + 9.3
99.2 + 0.2
autruaze-mmazzediveese
53.8 1 2.8
88.3 4.3
97.8 0.3
98.5 0.3
93.1 + 1.2
99.0 + 0.3
antmaze-medium- play
26.5 6.1
84.0
1.4
25.0
98.5 = 0.2
97.5 +.5
94.4
1.2
98.2 + 0.3
untrusze-nediuu-ddiveee
25.9 15.9 5.1
76.3 4 4.5
2.0
98.0 0.3
97.5 0.3
93.6 + 1.4
98.5 + 0.4
antmaze-lagge-ply
+
60.4
88.0
0.8
75.0 8.3
37.8
5.4
82.3
antmaze-laggoddiverse
0.0 + 0.0
54.4 + 8.4
0.0
87.5 + 1.2
77.5 + 4.2
50.2 + 6.9
85.6 + 2.1
Average
28.7 2.7
76.8 4.6
94.8 0.5
90.8 2.3
79.6
3.5
93.8
For the offline RL method, RL and investigates the effect of many design elements the performance highlighting how different design elements enhance the performance of oflline RL agents. ReBRAC is primarily designed for oflline RL, while it is less relevant to MOORL's hybrid offline-online framework, and is included for completeness.
Offline RI:
ReBRAC (Tarasov et al. 2024 which builds offline
These approaches use minimalistic approach for learning from offline data. To demonstrate the effectiveness of MOORL against the Behavior Cloning (BC) regularized (Bain & Sammut 1995) RL method, we use TD3+BC (Fujimoto & Gu 2021) and DrQ+BC (Yarats et al. 2021 for state and pixel-based tasks.
RL+BC:
5.1.2 Empirical Results
The baselines TD3+BC and ReBRAC results are taken from (Tarasov et al. 2024) while Hy-Q and DrQ+BC results are taken from [Nakamoto et al. 2024). Our results demonstrate that MOORL achieves competitive performance with RLPD, Hy-Q, and ReBRAC across the D4RL. and V-D4RL tasks while being simple and computationally efficient without introducing many design components. MOORL exhibits stable learning in hybrid RT setting, i.e. offline-online learning, and achieves robust cumulative rewards.

---

Published in Transactions on Machine Learning Research (05/2025)
Table 3:
The performance comparison across Adroit tasks is presented as the average normalized score across 10 seeds. The symbol denotes the standard error of the mean.
Task Name
BC
TD3+BC
ReBRAC
RLPD (UTD=20)
MOORL, our
pen
85.1
146.3 + 2.4
154.1 + 1.8
137.8 + 2.4
150.0 + 0.8
door
34.9
84.6 + 14.8
104.6 + 0.8
105.5 + 2.1
107.1 + 0.7
hammer
125.6
117.0 + 10.3
133.8 + 0.2
140.3 + 2.8
137.2 1.1
Average
81.9
116 + 9.2
130.8 + 1.0 127.9 + 2.4 131.4 + 0.9
On D4RL locomotion benchmarks, MOORL achieves the highest average performance. For the tasks with suboptinal data, hybrid approaches such as MOORL, Hy-Q, and RLPD highlight the advantage of using a hybrid learning approach. It is evident from Table [ that MOORL performs most optimally across tasks, specifically where offline data is suboptimal. MOORL achieves 8 - 10% performance improvement over RLPD without using large critic ensembles, high UTD, and layer normalization. Further, MOORL high- lights that for the lask, such as halfcheelul--rundon,, it achieves significant performance gain. We believe that this advantage comes from MOORL's adaptation strategy. Unlike other hybrid approaches, MOORL alternates between offiine and online data sources rather than directly mixing them. This design prevents the policy from being overly constrained by the offline dataset at each adaptation step, the inner-loop fine-tuning is performed separately for each data source. Consequently, MOORL shifts its reliance toward online interactions, resulting in superior policy.
The performance of RLPD is competitive to MOORL on the AutMaze navigation task, but as highlighted in Table [ the performance of RLPD drops significantly with a lower UTD ratio (UTD=1), whereas MOORL performs significantly superior. The performance of RLPD (UTD=5), with similar number of gradient steps, is similar to MOORL. However, MOORL being larger Q-ensemble free highlights its computational efficiency. For the D4RL Ant navigation tasks, MOORL avoids incorporating design choices such Clipped Double Q-learning (CDQ) (Fujimoto et al.) 2018) and Entropy Backup similar to RLPD, which are used in other tasks. These design choices tend to perform poorly in the sparse reward structure of AntMaze tasks, leading to overly conservative learning and suboptimal policy performance. While RLPD demonstrates the best performance under standard configurations, achieving a modest -% improvement over MOORL, its reliance on a high Update-to- (UTD) ratio makes sensitive to this hyperparameter. With UTD=1, RLPD suffers substantial performance drop, whereas MOORL maintains stable learning, achieving 13% 17% improvement over RLPD at 1M timesteps. This highlights MOORL's ability to maintain effective learning without reliance on aggressive update schodules, making it particularly advantageons in scenarios with limited computational budgets or where low UTD ratios are preferred.
Adroit tasks pose significant challenges due to their high-dimensional action spaces and sparse reward struc- tures. To evaluate the performance of various approaches, we conducted experiments on Adroit tasks using high-quality offline expert data. As shown in Table 3 the performance of all methods, including MOORL, remains consistent across tasks, with no single approach demonstrating definitive advantage. Depending the specific task, RLPD, ReBRAC, and MOORL occasionally outperform one another. However, the differences in performance are not statistically significant. This lack of clear improvement can likely be attributed to the inherent complexity of the Adroit tasks, which makes it challenging for any single method to achiove a distinct edge over others in this domain.
5.2 MOORL's Adaptability to Pixel-Based Observation Spaces
This section investigates MOORL's ability to operate effectively with high-dinensional image-based observa- tions. Unlike methods that require extensive hyperparameter tuning or specialized architectural adjustments, MOORL seamlessly extends to this challenging domain by utilizing a shared feature extraction encoder within its actor-critic framework. This encoder processes raw pixel input into meaningful feature representations, enabling efficient. learning even in visually complex environments. We evaluate MOORL's performance on
10

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: The performance comparison across V-D4RL Locomotion tasks is presented as the average normal- ized score across 10 seeds. The symbol + denotes the standard error of the mean.
Task Name
BC
DrQ+BC
ReBRAC
RLPD
MOORL, our
walker-walk-expett
91.5 + 1.3
68.4 + 2.5
81.4 + 3.3
91.5 + 1.1
94.6 + 0.2
walker-walk-mediung
40.9 + 1.3
46.8 = 0.8
52.5 + 1.1
84.9 - 0.7
87.9 - 1.8
cheetaa--ru-expert
67.4 + 2.3
34.5
2.8
35.6 + 1.8
68.3 1.0
53.2 + 2.5
cheetab-run-mmdium
51.6 + 0.5
53.0 + 1.0
59.0 + 0.2
49.0 + 1.3
49.9 + 1.6
Average
62.9 + 1.3
50.7 + 1.8
57.1 + 1.6
73.4 + 1.0
71.4 + 0.9
high-dimensional DeepMind Control Suite (DMC) (Tassa et al. 2018 tasks, leveraging datasets with pixel- based observations that test robustness and adaptability. As shown in Table MOORL outperforms RLPD on 3 out of 4 DMC tasks, while RLPD achieves slightly better average cumulative performance. Though RLPD remains competitive, its reliance on large Q-ensembles makes computationally expensive, particu- larly for image-based tasks. MOORL, on the other hand, achieves comparable results without using large critic-ensembles, emphasizing its practicality for resource-constrainedd scenarios.
These findings highlight MOORL's capacity for generalization and computational efficiency in high- dimensional reinforcement learning tasks. By maintaining strong performance without complex design alter- ations or fine-tuning, MOORL demonstrates its suitability for visually demanding environments, reaffirming its flexibility and reliability across diverse settings
5.3 Does MOORL learn Stable Q-Values
AmtMaza-Hediu--Divene
AmHMaza-Meedumm-lly
MOCRL
MOORL
To this end, we demonstrate that MOORL ables stable lcarning without relying on exten- sive design choices. Specifically, we present the mean Q-values Figure to showcase the sta- bility of MOORL's lcarning architecture. De- spite the data quality, MOORL consistently learns stable meta Q-function, translating into stable motacyolicy.
RLPD
RUPD
-20
-30
-10
022
066
0.5
Timestaps
Timestaps
(a)
(b)
Figure Learning curves showing the mean Q-values for the AntMaze- Medium task on Diverse and Play datasets. Figures [a] and [b depict the performance of the MOORL and RLPD frameworks, respectively, demonstrating their learning stability and effectiveness across both datasets.
In Figure D we evaluate MOORL stabil- ity in the challenging and sparsely rewarded AntMaze navigation task. The diverse dataset is created by assigning random goal locations in the maze and directing the agent navigate to them. In contrast, the play dataset consists of trajectories from specific, hand-picked initial positions to hand-picked goal locations. Remarkably, MOORL achieves similar Q-value trajectories across these datasets, emphasizing its robustness to variations in data quality.
For comparison, we conducted a similar analysis on RLPD to further substantiate the stability of Q-value learning across different frameworks. This analysis highlights MOORL's capability to maintain stable learn- ing dynamics even under diverse and challenging data conditions without requiring specific design choices.
6
Related Work
6.1
Offline Reinforcement Learning
Offline reinforcement learning (RL) has gained significant attention for its ability learn policies from pre-collected datasets without additional environmental interaction. However, distributional shift remains
11

---

Published in Transactions on Machine Learning Research (05/2025)
fundamental challenge, as highlighted by Levine et al. (2020), designing algorithms that generalize effectively from limited data is crucial. Several approaches address this challenge by incorporating regularization techniques. TD3+BC (Fujimoto & Gu 2021 integrates behavioral cloning into the actor loss to constrain policy learning, while CQL (Kumar et al. 2020] enforces conservatism by penalizing the critic for assigming high values to out-of-distributiio (OOD) actions. IQL. (Kostrikov et al., 2021 lakes a different approach by leveraging advantago- -arranttg--wighhtedd regression to avoid sampling OOD actions altogother. More recent methods further improve policy learning by incorporating representation learning, such as pre-training action encoders (Akimov et al., 2022. Chen et al. 2022) estimating dataset uncertainty using VAEs (Wu et al.) 2022) and RND (Nikulin et al. 2023). Another line of work improves policy robustness by leveraging mnsemble-based
uncertainty estimation. SAC-N (An et al. 2021) achieves strong results using large Q-function ensembles, though some Lasks require ensembles of up to 500 members, making it computationally expensive. MSG Ghhasemipour et al. 2022) mitigates this by introducing independent target updates, reducing the ensemble size to four in MuJoCo tasks but still requiring 64 members for AntMaze.
Further, Fujimoto & Gu
2021 shows that non-algorithmic factors significantly influence performance, and 2024) performs a detailed analysis to understand the effect of design choice
ReBRAC (Tarasov et al.
the performance of offline RL. Offline RL also struggles when datasets lack full coverage, making pessimism
2021 Zhang et al. 2022) and often requiring strong representation conditions (Xie et al.) 2021p. To overcome these limitations, hybrid approaches incorporating limited online
computationally challenging (Jin et al.
interaction have been proposed as a promising alternative.
6.2 Bridging Online and Offline RL
Integrating online and offline reinforcement learning (RL) is a critical research area. Empirical studies have examined how online learners can leverage logged data to improve performance (Rajeswaran et al. 2017 Nair
et al. 2017 Hester et al. 2017 Ball et al. 2023 Nakamoto et al.) 2024 Zheng et al. 2023). While practical
beneftts are evident, formal guarantees in this setting remain limited. (Ross & Bagnell 2012) proposed a
framework where a learner can choose wetween execuling logging policy oror an alternative online policy,
effectively bridging the gap between online and offline data exploitation. (Xie et al., 2021] demonstrated
that no approach could achieve strictly better sample complexity than purely online or offline methods when using data collected from a logging policy. This result highlights the challenges of balancing online exploration with offline data utilization. In contrast, our research assumes access to a pre-collected offline dataset and the ability to interact online, enabling the refinement of a near-optimal policy while minimizing online interactions. (Song et al., 2022) proposed 'Hybrid RL' using the Hybrid Q-lcarning algorithm (Hy- Q) for low bilinear rank Markov Decision Processes (MDPs) (Du et al.] 2021]. p. Under certain conditions, Hy-Q can achieve optimal policies efficiently. However, the method's performance may degrade when offline data coverage of the optimal policy is insufficient, illustrating the importance of comprehensive offline data. Further, a study by (Xie et al., 2022) delves into purely online contexts or frameworks involving offline datasets, offering insights into the concealability coefficient-a a parameter critical in establishing guarantees for offine RL. This approach bridges the analytical methods used in online and offline settings. Recent work by OO'Donoghue et al.) 20188 Wagenmaker & Pacchiano| (022), and others explore synergies between these methodologies, revealing strategies for effectively integrating offline data with online exploration.
Our approach builds on these foundations by addressing sample complexity when merging offline datasets with online learning. We aim to enhance the unified integration of offline data into online reinforcement learning without extensive design choices and added computational complexity (Ball et al. 2023 Song et al. 2022), which limits the border applicability of such hybrid-RL approaches.
7 Conclusion
In this work, we demonstrated that off-policy RL can effectively integrate oflline and online data distribu- tions. By learning a meta-policy over these distributions, our approach enables stable Q-value estimation independent of data quality. Extensive experiments across 28 diverse tasks, spanning both state and pixel observations, validated the efficacy of our method. Specifically, we showed that a hybrid RL policy improves sample efficiency and ensures robust performance across varying data qualities. By limiting design compo-
12

---

Published in Transactions on Machine Learning Research (05/2025)
nents and computational overhead, approach generalizes well across different environments, making it scalable and practical solution for real-world applications. These findings highlight the potential of combining offline and online learning to address key challenges in reinforcement learning. While MOORL demonstrates robustness in handling diverse offiine datasets, highly biased data may still constrain exploration. Future work could explore adaplive mechanisms to miligate such biases by dynamically adjusting the influence of offline data to further enhance learning stability and generalization.
References
Dmitriy Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov, and Sergey Kolesnikov. Let of- fline RL flow: Training conservative agents in the latent space of normalizing flows. arXiv preprint arXiv:2211.11096 2022.
Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, llya Sutskever, Igor Mordatch, and Pieter Abbeel. Contin- uous adaptation via meta-learning in nonstationary and competitive environments. ICLR, 2018.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified Q-ensemble. Advances in neural information processing systems, 34:7436 7447, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offiine reinforcement learning. arXiv preprint arXiv:2202. 11566, 2022.
Michacl Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103 129, 1995.
Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. International Conference on Machine Learning, pp. 1577-1594. PMLR, 2023.
Arthur Charpentier, Romuald Flie, and Carl Remlinger. Reinforcement learning in economics and finance. Computational Economies, pp. 38, 2021.
Gaurav Chaudhary, Laxmidhar Behera, and Tushar Sandhan. Active perception system for enhanced visual signal recovery using deep reinforcement learning. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023. doi: 10.1109/CCASSP49333.22200 10097084.
Xi Chen, Ali Ghadirzadch, Märten Björkman, and Patric Jensfelt. Meta-learning for multi-objective re- inforcement learning. In 2019 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 977-983. IEEE, 2019.
Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. Lapo: Latent-variable advantage- weighted policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:36902 36913, 2022.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Simon Shaolei Du, Sham M. Kakade, Jason D. Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, and Ruosong Wang. Bilinear classes: structural framework for provable generalization RL. ArXiv, abs/2103 12.11.00177 2021. URL https://api. semanticscholes
Ishan Durugkar. Estimation and control of visilation distributions for reinforcement learning. PhD thesis, 2023.
Logan Engstrom, Andrew Tyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep RL: A case study PPO and TRPO. In International conference learning represendations, 2019.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. Advances in neural information processing systems, 23, 2010.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data- driven reinforcement learning. CoRR, abs/2004.07219, 2020. URL https://arxiv, httpe////arii..org///////00/// 07219
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20144 2021.
Scolt Fujimoto, Herke Hool, and David Meger. Addressing function approxination error in actor-critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott. Fujimoto, Edoardo Conti, Mohammad Ghavanzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint Xit::191000011 708, 2019.
Hiroki Furuta, Tadashi Kozuno, Tatsuya Matsushima, Yutaka Matsuo, and Shixiang Shane Gu. Identi- fying co-adaptation of algorithmic and implementational innovations in deep reinforcement learning: A taxonomy and case study of inference- based algorithms. arXiv preprint arXiv:2108. 17258, 2021.
Francisco Garcia and Philip Thomas. A meta-mdp approach to exploration for lifelong reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline RL through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35:18267-18281 2022.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint Xiv:1881 05905, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Todd Hester, Matej Vecerík, Olivier Pielquin, Mare Lanciot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John P. Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep Q-learning from demonstrations. In AAAI Conference on Artificial Intelligence, 2017. URL https://api. www.atticccooooor.oooooooooo :
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning, pp. 5084-5096 PMLRR 2021.
Taewoo Kim, Ho Suk, and Shiho Kim. Chapter nine offline reinforcement learning methods for real- world problems. In Shiho Kim and Ganesh Chandra Deka (eds.), Artificial Intelligence and Machine Learning for Open-world Novelty volune 134 of Advances in Computers, pp. 285 315. Elsevier, 2024. doi: https:///oi.orr//10.001111 articcle/pp//000644482000000
200..203.20000011. URL htttps://wwi htttps//www.scienneediccc. www.sseccenneerrtt.cc.m cttpstiicccccccccccccccccc
Diederik P Kingma. Adam: A method for stochastic optimization. arXio preprint arXiv:1412.6988, 2014.
B Ravi Kiran, Tbrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement lcarning for autonomous driving: A survey, IEEE Transactions on Inlelligent Transporlation Systems, 23(6) 4926, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning.
arXiv preprint or Xix::2110.00 0616666,, 2021.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statislics, 22(1):79 16099-866111 1951.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline rein- forcement learning. Advances in Newral Information Processing Systems, 33:1179-1191, 2020.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic Q-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:20550010 01643, 2020.
Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offine reinforcement learning from visual observations. arXiv preprint urXiv:220600477 77779, 2022.
Rémi Munos. Error bounds for approximate value iteration. In Proceedings of the National Conference on Artificial Intelligence, volume 20, pp. 1006. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2005.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and P. Abbeel. Overcoming explo- ration in reinforcement learning with demonstrations. 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6292-6299, 2017. URL https://api semanticscoolar. oaw.aaamiiisscllaaooooooooooollp org/CorpusID:
3543784
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offiine RL pre-training for efficient online fine-tuning. Advances in Newral Information Processing Systems, 36, 2024.
Alex Nichol and John Schulman. Reptile: scalable metalearning algorithm. arXiv preprial arXiv:1803.0200 2(3):4, 2018.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International Conference on Machine Learning, pp. 26228 26244. PMLR, 2023.
Brendan O'Donoghue, Ian Osband, Remi Munos, and Volodymyr Mnih. The uncertainty Bellman equation and exploration. In International conference on machine learning, pp. 3836-3845, 2018.
Mark S Pinsker. Information and information stability of random variables and processes. Holden Holden-Day, 1964.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. RSS, 2017.
Stephane Ross and Andrew Baguell. Agnostic system identification for model-based reinforcemnett learning. arXiv preprint Xiv: 1K:3:2003100 2012.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXie preprint 1112: 01815, 2017.
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. arXiv preprint X:Xiu:2210.06718 2022.
Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Modcl-Bellman inconsistency for model-bace based offine reinforcement learning. In International Conference on Machine Learning, pp. 33177-33194. PMLR, 2023.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martin-Martin, and Peter Stone. Deep reinforcement. learning for robotics: A survey of real-world successes. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 39, pp. 28694 28698, 2025.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offiine reinforcement learning. Advances in Newral Information Processing Systems, 36, 2024.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Morel, Andrew Lefrancq, Timothy Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, hbs/1801.00663 2018. URL http://aaxxvv..rrabaa/eeee/
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/B RSSJ International Conference on Intelligent Robots and Systems (TROS), pp. 23-30, 2017. doi: 10.1109/IRO5 001..82..
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojciech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al. Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind blog, 2:20, 2019.
Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data in online reinforcement learning, ArXiv, abs abs/2211.04974, 2022. URL http:::/api..emanticccess www.etiisccoooo..oooo....ooo.oot
wrggCorpgsstaooooooooooooooo 253420756
Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data. in online reinforcement learning. In International Conference on Machine Learning, pp. 35300 35338. PMLR, 2023.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offinn reinforcement learning. Advances in Neural Information Processing Systems, 35:31278-31291, 2022.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample- efficient offline and online reinforcement learning. Advances in newral information processing systems, 34: 27395 27407, 2021.
Tengyang Xie, Dylan J Foster, Yu Bai, Nan Jiang, and Sham M Kakade. The role of coverage in online reinforcement learning, arXiv preprint ar 0rX1::211000115,, 2022.
Denis Yarats, Tlya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixcls. In International conference learning representations, 2021.
Feiyang Ye, Baijiong Lin, Zhixiong Yue, Pengxin Guo, Qiao Xiao, and Yu Zhang. Multi-objective meta learning. Advonces in Neural Information Processing Systems, 34:21338-21351, 2021.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement lcarning in healthcare: A survey.
ACM Computing Surveys (CSUR), 55(1):1-38 36, 2021.
Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offiine reinforcement learning. In International Conference on Artificial Intelligence and Slatislics, pp. 5757-5773. PMLR, 2022.
Han Zheng, Xulang Luo, Pengfei Wei, Xuan Song, Dongsheng Li, and Jing Jiang. Adaptive policy learning for offlnnetooonllne reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11372-11380, 2023.
16

---

Published in Transactions on Machine Learning Research (05/2025)
haltcheetah-expett
hallcheetah-modumm
halfchedtah-random
239
-20
Normalize Return
Normaltoue Festurm
Normalizeet Foeiturr
No Mormallze Rreuur
NLFO
AUPO MOOAL
RIPD MOORL
NLPO
338950 293000
20000 250030
259000
59900 304030 290030
hoppen
hogger
hagoer-random
Normalzd Reeum
No Nommalltzed Ruttum
No Rutuum
Nerm Noraaloos www.dlllddddooooooooooo Return
03
ALPO
ALPO
RUPD
ALPO MOORL
MOORL
MOORL
MOORL
Time Steps
Time Stops
Time Steps
Time Steps
walker 2d-espert
wullee28-medinn-eelly
waller2d-medium
waller26-random
120
Normal fooed Naturn
Nonta lowd Rotturn
Normaa Rowdd Rosturn
Norma loodd Roturn
MOOKL
MODAL
MOOKL
MOOHL
59000 189099
128903 230000 253900 389000
50583 300000
1906S 200006 20000 300090
500ce 300900 Time Steps
189990 200000 251000 300000
39900 100989 126909 306090 259000 389000
Time Steps
Time Skeps
Time Steps
antrma.ze ummaze
cntmaze-umaace diverse
antimaze mediim--ppy
antmaaee wwwreeeeeeddm--iiie diverse
Norm alloed ineurn
100
Norm awfeeed Ruatum
Norm alloddddddd ltuttum
100
Norm allood Reeurm
RLPO
RUFO
RLPD MOORL
RLPO
1990 J8GERS 250000 2900a 103000
16500 SREED KREG WEIce 2005s VEEeSS
v266 11020 193939 29R00 252ne seEco
1990 J8OEES 200000 290031
walkerssyalk-eeppet
Normalize Returm
Normallzz lied Retur
Normalbed Raluum
Normallzzs lild Retuum
ALPO
ALPO MOORL
RLPD
ALPO MOORL
MOORL
MOORL
Time
Time
Time
Time
halfcheetah medium-expart
happir-mediun-erppert
antr--ze-large-play
Nor moltelted Retur
Nor Normaltrod Battum nettum
Nor Pnoemmaltoen nettum
Nor mal ledd Retumm
MOORL
MOORL
MOORL
MOORL
Time Stops
Time Steps
Time Stops
Time Steps
antmage- diverse
pen expert
doer expert
hammer expert
100
Normalized weturn
Normalized Baturn
Normallzad Beturnn
Normallzed Return
RUPO
RUP
MOORL
MOORL
33000 200000 158083 300000 350930 303000
50360 20000 LO00S J00006 20036-20003
50300 160000 150003 300030 200830 38369) 5206 200320 330000 380000 45300 300006 Time Steps
Figure 2: The plots show learning the mean performance across 10 random seeds, with shaded regions indicating the standard deviation. The normalized return at each point is computed as the average over 10 evaluation episodes. All tasks are evaluated over 300K timesteps.
with normalized returns the y-axis. Each curve represents
17

---

Published in Transactions on Machine Learning Research (05/2025)
haltcheetah-expet
hallcheettah-mddium-eeee
haltchestah- medium-replay
halfcheetah-
168
Normalize Preuurm
Normallton Festurr
Normalizeet Festuur
Normallze Rreuurn
K-4
Km4
Kmd
K-4
K-6
K-6
K-6
338950
253800
30000 1500 2000 20000
3500e 300320
Time 253000 300000
59900 336000 353030
antmaze-umace diverse
antinaze medium--ppe play
antmage medium diverse
Noraalzzd Retur
No aww. Ruttum
200
K-4
No altiaaaddd Retuur
K=4
Norm Normalloo alodddlddddooooooooo Retur
-30
Kmd
K-6
K-6
Km6
60-
K
Time Steps
Time Stops
Time Stops
Time Steps
Figure 3: The plots illustrate the impact the inner-lopp adaptation step on learning. The y-axis represents the normalized return, while the x-axis denotes timesteps. The solid curves show the mean return across random seeds, with shaded regions indicating the standard deviation. Each evaluation point is computed the average return over 10 episodes. All tasks evaluated over 300K timesteps.
A
Ablation Study: Impact of Inner-Loop Adaptation Steps
The number of inner-loop adaptation steps (K) plays crucial role in our meta-learning framework, influ- encing the policy's ability to adapt to distributional shifts. To assess its impact, conduct an ablation study (Figure 3 by varying K E {2, 42,4,6) while keeping all other hyperparameters fixed.
Our results highlight the importance of selecting an appropriate K for effective adaptation. With K = 2, the model struggles align with the target distribution, leading high variance and degraded performance. This suggests that too few updates hinder the agent's ability to capture distributional shifts, resulting in unstable learning dynamics. Conversely, K = 4 provides the best balance between stability and adaptability, yielding optimal performance. However, increasing K further to K 6 offers no significant performance gains. Instead, we observe diminishing returns, where additional updates increase computational overhead without substantial improvements. This likely due to overfitting to recent experiences, reducing the learned policy's generalization capability.
This aligns with prior meta-learning findings Finn et al., 2017 Nichol & Schulman, 2018 which indicate that a moderate number of inner-loop steps maximizes generalizating while mainlaining efficient, adapta- tion. Overall, our findings suggest that balanced adaptation strategy caatee than aggressive inner-loop optimization is key to achieving strong generalization in hybrid offline-oninee reinforcement learning.
B Implementation Details
entropy-regularized SAC (Haarnoja et al. 2018) as the base RL algorithm and apply Reptile (Nichol & Schulman 2018) for meta-updates, improving generalization across distributions. The policy and Q-nelworks are 2-layer MLPs with 256 hidden units and ReLU activations. We use the Adam optimizer (Kingma 2014) for inner-loop adaptation and meta-updates. Each training iteration consists of K update. A learning rate of 3 x 10-4 for inner-loop adaptation is used, while for meta-updates, the learning rate is dynamically adjusted (Nichol & Schulman 2018) based the ratio of the current timestep to
To implement the proposed framework,
SAC updates followed by Reptile-style (Nichol & Schulman, 2018) meta-
18

---

Published in Transactions on Machine Learning Research (05/2025)
Table 5: MOORL Hyperparameters
Parameter
Value
Batch size Discount (?)
256
0.99
Optimizer
Adam
Learning rate
3 x 10 4
Critic EMA Weight (p)
0.005
Inner Gradient Steps (K)
Network Width
256 Units
Number of Layers
Initial Entropy Temperature (a)
1.0 dim (M) 2
Target Entropy
total timesteps. We maintain an exponentially moving average larget Q-uetwork with an update weight of p 0.005. Hyperparameters are summarized in Table
C Inner-Loop Adaptation in MOORL vs. UTD in RLPD
It is essential to distinguish the role of inner-loop adaptation steps (K) from the Update--To-Date (UTD) ratio in RLPD (Ball et al., 2023). The UTD ratio dictates the number of gradient updates per environment step, primarily influencing sample efficiency off-policy RT, by controlling the degree of data reuse. Tn contrast, our adaptation steps (K) govern the number of inner-loop within each outer-loop meta-update iteration, directly shaping the adaptation dynamics rather than the frequency of gradient updates. While high UTD enables more updates per collected transition, K determines how effectively the learned policy aligns with the target distribution across offline and online phases. Thus, K plays a fundamental role in optimizing meta-adaptation rather than modulating sample efficiency, making it a distinct and crucial design choice in our framework. These distinctions become evident in Figure [ where we show that both RLPD and MOORL learn similar Q values. While MOORL inherently adapts to changing distribution through inner and meta updates, RLPD uses layer norm and a large Q-ensemble stabilize learning, essentially learn similar Q-values.
D Computational Cost Comparison
We compare the computational cost of our methods with RLPD on Adroit hand tasks, including pen door and hammer. The RLPD for these tasks uses a 3-layer MLP, whereas MOORL uses a 2-layer MLP, which is consistent across all the tasks as highlighted in Table 7 Also, RLPD uses a high UTD ratio (20) while maintaining large Q-ensemble (10), which is in contrast to MOORL, which maintains simple Double-Q network architecture. RLPD is trained for 300K timesteps at each timestep, it performs 20 gradient steps given by UTD, essentially resulting in 6M gradient steps over 300K timesteps. MOORL performs 4 inner update adaptation gradient steps and 1 meta update gradient step at each timestep, resulting in a total of 1.5M gradient steps over 300K timesteps. RLPD takes approx 0.5sec while MOORL takes 0.05sec per timestop when run on a single RTX A4000 GPU.
E MOORL: Embracing Simplicity in Design
Recent works have demonstrated that leveraging offline data can significantly enhance reinforcement learning performance. State-of-the-art methods in hybrid RL (e.g., RLPD (Ball et al. 2023 and offline RL (e.g..
19

---

Published in Transactions on Machine Learning Research (05/2025)
ReBRAC (Tarsoov et al. 2024)) typically rely extensive component selection and fine-tuning of design elements to address offline data challenges. In contrast, MOORI. adopts a simpler algorithmic structure that requires fewer design components.
MOORL integrates offine and online data using an architecture that minimizes reliance on extensive tun- ing-aavoding avoiding the need for deep network ensembles or high UTD ratios while still effectively handling distribution shifts and limited exploration. As illustrated in Tables c and 7 MOORL delivers robust perfor- mance with streamlined set of design choices compared to ReBRAC and RLPD.
Although the MOORL. algorithm is not entirely free of design decisions, the simplicity of MOORL's 's approach enhances its adaptability across diverse tasks, providing a more generalizable and robust solution relative to methods that depend heavily on intricate design configurations.
Table 6: Comparison of Design Choices with ReBRAC Tarasov et al. 2024
Component
ReBRAC MOORL
Deeper Networks
X
Larger Batches
X
Layer Normalization
X
X
Decoupled Penalization
Adjusted Discount Factor
Table 7: Comparison of Design Choices with RLPD (Ball et al.
2023
Component
RLPD MOORL
Sampling Strategy
Layer Normalization
Random Ensemble Distillation
X
V
Clipped Double Q-Learning
X
Network Architectures
Update to Data Ratio
Entropy Backup
F Detailed Task Definition
F.1 D4RL: Locomotion
In these tasks, the reward is dense and based on the agent's forward velocity, penalizing large control inputs to encourage stable movement. The goal is to maximize the cumulative reward over the episode by learning an efficient and stable gait. The standard evaluation metric is the normalized score, which computed by normalizing the agent's return relative expert and random policies, as defined by (Fujimoto al. 2019). The datasets generated from policies of varying expertise, including random, mediumm, medium-replay, meeiimm-eppert, and expert trajectories. Episodes typically last for 1,000 timesteps without early termination.
F.2 D4RL: AntMaze
In these tasks, the reward is sparse and binary, indicating whether the agent has reached the goal. Upon reaching the goal, the episode terminates. The normalized return is measured as the proportion of successful trials out evaluation trials following prior work. The dataset consists of play-based and diverse demon- strations, where the former includes goal-directed trajectorics, and the latter contains broader movement data. The challenge this domain arises from long-horizon credit assignment and the need for effective exploration in sparse reward setting.
20

---

Published in Transactions on Machine Learning Research (05/2025)
F.3 D4RL: Adroit
The Adroit suile consists of dexterous hand manipulation lasks that require controlling a 24-Dof simulated Shadow Hand robot to perform complex actions such as hammering a nail, opening door, twirling a pen. This domain is specifically designed to assess the impact of narrowly distributed expert demonstrations on learning in a high-dimennsioall robotic manipulation setting with sparse rewards. Unlike standard Gym MuJoCo tasks, Adroit exhibits several distinct characteristics. First, the dataset is sourced from human demonstrations. Second, solving these tasks with online RL alone proves challenging due the sparse reward structure and inherent exploration difficulties. Lastly, the high-dimensional nature of these tasks introduces additional complexity in representation learning.
F.4 V-D4RL: DeepMind Control Suite (DMC)
The DMC tasks involve controlling physics-based agents with dense rewards that encourage smooth, efficient movement. The standard evaluation metric is the normalized score, computed using the agent's return normalized against the performance of a well-trained SAC policy. The datasets include expert and medium policies, allowing evaluation of an agent's ability to learn from varying data quality, Episodes typically run for 1,000 timesteps without carly termination.
21


Reviews:
[{"review_id": "gPg8O2JiBr", "reviewer": "Reviewer_xTLd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths** \n\n1. Novel integration of meta-learning with hybrid RL, enabling dynamic adaptation to distribution shifts.\n2. Comprehensive experiments across diverse tasks (locomotion, navigation, manipulation, pixel-based).\n3. Strong theoretical grounding in analyzing performance bounds and KL divergence effects.\n\n**Weaknesses**\n\n1. Computational costs of meta-learning (e.g., inner-loop updates) are not quantified. The meta-learning approach introduce extra computation burden.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces MOORL, a meta-learning framework that unifies offline and online reinforcement learning. It formulates policy adaptation as a bi-level optimization, leveraging KL-regularized performance bounds to balance exploration and exploitation. MOORL uses a Reptile-style meta-update to integrate offline pre-training with online fine-tuning without additional hyperparameters. Theoretical analysis reveals how mixing offline and online data influences policy improvement. Empirical results across 28 tasks demonstrate superior stability and sample efficiency over prior methods. Unlike existing approaches, MOORL automates the balance between data sources, providing a general and principled solution for hybrid reinforcement learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Discuss how pathological offline data affects MOORL. If offline data comes from a poor policy, does MOORL still help, or can it mislead learning? The success on “random” datasets suggests that diverse but low-return data aids exploration. However, if offline data is highly biased, could it restrict exploration due to KL penalties? Consider whether MOORL avoids this issue by alternating data sources instead of mixing them directly. Conclude that MOORL may naturally shift to online data if offline data is suboptimal, highlighting this as a limitation or future work discussion point.\n2. Include Ablation Studies (Meta-Parameters): Since MOORL introduces the meta-learning procedure, an ablation or sensitivity analysis on its hyperparameters would be very insightful. Two suggestions: (a) Vary the number of inner-loop gradient steps N (e.g., try $N=1$ vs $N=4$ vs $N=8$) to see how it affects performance. And discuss the relationship of UTD from RLPD.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iCGoOo4n2Q", "reviewer": "Reviewer_Xnhu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Empirical Evaluation: The experimental results are comprehensive, covering a wide range of tasks, including state and pixel based observation spaces\n- Relevant Problem: The effective integration of offline data to jumpstart the RL process is an open area of research and is highly important for the greater adoption of RL. \n- MOORL appears to be robust to task changes and doesn't need any task specific design choices. It is preferable to have algorithms that can be applied to new problems without needing extensive tuning.\n\n\nWeaknesses\n- Although the paper claims to be \"design-free,\" meta-learning itself introduces hyperparameters (e.g., meta, n, K in Algorithm 1). The paper doesn't discuss the sensitivity of MOORL to these hyperparameters. \n- The paper makes repeated claims about RLPD that is \"requires tailored design parameters and careful task-specific tuning\". However they do not say that they needed extensive tuning of RLPD for the different tasks and (in the absence of contrary evidence) appear to have used a single set of hyperparam choices for all tasks.\n- See questions below", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Meta Offline-Online Reinforcement Learning (MOORL), a framework designed to integrate offline and online reinforcement learning. The core idea is to train a \"meta-policy\" that can effectively adapt to the distributional shifts between offline and online data sources. MOORL aims to improve sample efficiency and stability compared to purely online or offline RL methods, and to do so without requiring extensive, task-specific design choices or hyperparameters. The authors provide a theoretical analysis on the benefits of mixing offline/online data as a regularization, and empirically evaluate MOORL on a variety of benchmark tasks (D4RL and V-D4RL), including both state-based and pixel-based environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Could the authors provide details about the number of training/environment steps for each task as well as learning curves if possible?\n- I think it is important to have the comparison between ReBRAC, RLPD and MOORL in the main text of the paper. It is a core claim of the paper that MOORL is design free and the evidence to support this should be in the main text and not the appendix.\n- The authors make a note about the increased computational overhead of running RLPD however there aren't any quantitative numbers for this claim, such as training time, gradient steps, memory usage, etc. It isn't necessary to provide all of these, but at least some quantitative metrics would be useful.\n- Could the authors compare performance on versions of Adroit with suboptimal/random data mixed in? For example with 20% and 50% of random data mixed with the expert human demonstrations? It would strengthen their claim that \"MOORL performs most optimally across tasks, specifically where offline data is suboptimal.\" Especially since there isn't a clear leader in performance in this task.\n- Could the authors provide the standard deviation of the Q Values for the experiment in Fig 1?\n- Are there any meta RL baselines that are relevant for comparison?\n- Since the algorithm randomly chooses at every epoch which buffer to perform a fixed number of inner gradient steps, wouldn't this be a problem when the buffers are of significantly different sizes? Wouldn't this bias the algorithm ?\n- The hyperparams given in Appendix A are for MOORL and not RLPD I assume? Could you provide the hyper parameters for the RLPD/other baselines and if they needed per task tuning?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4aDw7nBmsB", "reviewer": "Reviewer_uLsg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper addresses a timely topic in RL and proposes a simple yet effective solution.\n- The paper provides a comprehensive benchmark of MOORL on 28 tasks from four different suites.\n\n### Weaknesses\n- The paper is not well-written. Specifically, the language and notation are not rigorous and sometimes lead to confusion. Here are some examples:\n    - In multiple instances, the authors refer to an \"offline expert dataset,\" but the offline dataset can also be task-irrelevant, such as a random dataset.\n    - In the captions of Tables 1–4, it is unclear what the authors mean by \"unseen seeds.\"\n    - On page 8, second paragraph, I assume you mean $\\theta_{\\mathrm{meta}} - \\widetilde{\\theta}$ since $\\widetilde{\\theta}_{\\mathrm{meta}}$ is not defined anywhere.\n    - In the pseudocode, line 7, are you sure you only update the actor-critic once after completing full trajectories of environment interactions? I believe most works update per environment step.\n    - In the methods section, MOORL involves two learning rates: $\\eta$ for the inner loop and $\\eta_{\\mathrm{meta}}$ for the outer loop. However, in the appendix, only one learning rate is listed.\n- One of the core contributions of the paper—the theoretical analysis of why the offline dataset benefits online RL—appears problematic. I am not an expert in theory, but the authors seem to use $r(s, a)$, $Q^\\pi(s, a)$, and $\\log d^\\pi(s, a)$ interchangeably throughout the derivation without any stated assumptions.\n- One of the claims made by the authors is that MOORL is \"design-free,\" which I find unconvincing. For example, both RLPD and MOORL explore the design of an offline-online algorithm, but from different perspectives: RLPD focuses on component selection, while MOORL focuses on algorithmic structure. In my opinion, neither of them is truly design-free. I agree that MOORL is a simple algorithm, but not that it is design-free.\n- In Section 3, the authors list four benefits of incorporating meta-learning, but none of these claims are supported by references or experiments.\n- The paper lacks ablation studies on the new hyperparameter, namely the inner-loop steps $K$. Since meta-learning effectively takes $K+1$ gradient steps to update the actor-critic once, $K$ plays a similar role to the Update-to-Data (UTD) ratio in RLPD. Given that $K$ is set to 4 in this paper, it would be more informative to compare it fairly to AntMaze with UTD=5 in RLPD.\n- Although MOORL leverages an offline dataset, it remains an online RL method. However, the paper does not provide learning curves, which are crucial for evaluating the sample efficiency of an online RL method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an interesting perspective on viewing offline-online RL as a meta-learning problem. The proposed algorithm, MOORL, treats learning from the offline dataset and the online buffer as two separate meta-learning tasks and applies meta-learning to update the actor-critic. Benchmark experiments on 28 tasks demonstrate that MOORL can match or surpass state-of-the-art baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I request the authors to revise the paper thoroughly to ensure that the language and notation are rigorous and consistent.\n- Please provide learning curves showing normalized returns for MOORL alongside hybrid baselines such as Hy-Q and RLPD. This will help demonstrate MOORL's sample efficiency and clarify how the reported numbers in the tables were obtained.\n- The theoretical analysis of why the offline dataset benefits online RL should be revised. Specifically, the authors should clearly differentiate between $r(s, a)$, $Q^\\pi(s, a)$, and $\\log d^\\pi(s, a)$ in the derivation and explicitly state any assumptions made.\n- The claim that MOORL is \"design-free\" should be reconsidered or better justified. Both RLPD and MOORL involve design choices—RLPD focuses on the component level while MOORL focuses on the algorithmic level. The authors should clarify this distinction and avoid overstating the generality of MOORL.\n- In Section 3, the authors list four benefits of incorporating meta-learning, but none of these claims are supported by literature or experiments. Please support these claims with references to prior work or empirical evidence through experiments.\n- Please provide the ablation study on the inner-loop steps $K$. Additionally, make a fair comparison with RLPD on AntMaze if you intend to argue that its good performance is due to taking more gradient steps.\n- Since RL is known to be sensitive to hyperparameters and random seeds, please provide sufficient implementation details, and ideally, also release the code.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows an interesting perspective on combining offline and online training in RL. After extensive adjustments based on the reviews and the discussion and the resulting improvements, the paper can be accepted. 

However, a revision is still necessary. The paper still needs to be proofread very carefully. Here is a list, which is not exhaustive:

* Question marks must be resolved, e.g. “(?Nair et al. ”and “(?)”
* “Antmaze” -> ‘AntMaze’

* In the bibliography, the entry for Adam is incorrect: “D Kinga, Jimmy Ba Adam, et al. A method” -> "D Kinga, Jimmy Ba, Adam:. A method“
”California;, 2015." -> “California, 2015.”

* In many places letters are incorrectly written in lower case, “bellman”, “rl”, ‘ppo’, “q”

* In Tables 1 to 4, the $\pm$ sign is used together with the standard deviation. The $\pm$ sign should exclusively be used to express an  uncertainty, e.g. the 95% confidence interval, or the standard deviation of the mean (also called standard error). I suggest to express the standard error instead of the standard deviation by dividing the standard deviation by sqrt(10 - 1) = 3. E.g. “93.4$\pm$ 0.2” instead of “93.4 $\pm$ 0.4” (uncertainties are always rounded up and given with one or two valid digits, the number of decimal places of the measured value (i.e. the number before the $\pm$ sign) must correspond to the number of decimal places of the uncertainty).
If the authors insist on specifying the standard deviation instead of an uncertainty, then brackets should be used instead of the $\pm$ sign, for example, or a separate column should be used.
Decision: Accept (poster)