Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
Authors: Jongchan Park
Mingyu Park
Donghwan Lee
Venue: NeurIPS 2025 poster ()
Keywords: Offline Reinforcement Learning
Data Efficiency
Primary area: reinforcement_learning

Abstract: Offline reinforcement learning (RL) aims to learn a policy from a fixed dataset without additional environment interaction. However, effective offline policy learning often requires a large and diverse dataset to mitigate epistemic uncertainty. Collecting such data demands substantial online interactions, which are costly or infeasible in many real-world domains. Therefore, improving policy learning from limited offline data—achieving high data efficiency—is critical for practical offline RL. In this paper, we propose a simple yet effective plug-and-play pretraining framework that initializes the feature representation of a $Q$-network to enhance data efficiency in offline RL. Our approach employs a shared $Q$-network architecture trained in two stages: pretraining a backbone feature extractor with a transition prediction head; training a $Q$-network—combining the backbone feature extractor and a $Q$-value head—with *any* offline RL objective. Extensive experiments on the D4RL, Robomimic, V-D4RL, and ExoRL benchmarks show that our method substantially improves both performance and data efficiency across diverse datasets and domains. Remarkably, with only **10\%** of the dataset, our approach outperforms standard offline RL baselines trained on the full data.

Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
Jongchan Park" Hyundai Motor Company jepark1 10hyundai. com
Mingyu Park
Donghwan Lee KAIST
KAIST
mingyu@kaist.ac. ac.
donghwanCkaist. ac. kr
Abstract
Offline reinforcement learning (RL) aims to learn policy from fixed dataset without additional environment interaction. However, effective offline policy learning often requires large and diverse dataset mitigate epistemic uncertainty. Collecting such data demands substantial online interactions, which are costly or infeasible in many real-world domains. Therefore, improving policy learning from limited offline data-achieving high data efficiency-is critical for practical offline RL. In this paper, we propose simple yet effective plug-and-play pretraining framework that initializes the feature representation of Q-network to enhance data efficiency in offline RL. Our approach employs shared Q-network architecture trained in two stages; pretraining backbone feature extractor with transition prediction head; training a Q-network- wwt.wooo--ommbingg....ee the backbone feature extractor and Q-value weth-with any offline RL objective. Extensive experiments on the D4RL, Robomimic, V-D4RL, and ExoRL. benchmarks show that our method substantially improves both performance and data efficiency across diverse datasets and domains. Remarkably, with only 10% of the dataset, our approach outperforms standard offline RL baselines trained on the full data.
Introduction
Sample efficiency is a long-standing challenge in reinforcement learning (RL). Typical RL algorithms rely on online learning process that alternates between collecting experiences through interactions with the environment and improving the policy [55]. However, acquiring a large number of online interactions is often impractical, since data collection can be costly and risky. Offline reinforcement learning (RL) has emerged as promising alternative to address this issue by decoupling data collection from policy learning, enabling agents to learn solely from pre-collected datasets [B7]. Learning an offline policy from static dataset allows the agent to focus on how effectively it can extract optimal behaviors from a fixed data distribution--unl expand its experience data through active exploration. Nevertheless, learning optimal policy from limited experience data remains a fundamental challenge in both online and offline RI..
online RI,, which must continuously
However, prior offline RL approaches have largely focused on improving policy learning within a fixed dataset through policy constraints [8] 32]. conservative regularization [3], and model-based uncertainty estimation [28 to mitigate distributional shift. Other studies, such as maniplaanin
strategies
27
67 61 and afffine-to-onling frameworks [43 59 47 3], provide alternative perspec-
tives on improving policy performance with limited data sources. However, understanding how an offline agent learns an effective policy with minimal data usage offers distinct perspective. To address this, we define data efficiency as the ability of offline RI. agent to learn an optimal policy from as little offline data as possible, which is distinct from sample efficiency in online RI. Furthermore, we consider truly data-efficient offline RL as learning an optimal policy across datasets
Equal Contribution. Correspondence to donghwan kr...
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Que(5,
Pretraining
RL Training
Collect Minimal Data
Transition Head
Q-value Head
fa
fe
Environment
h c
h
D
Dataset
Feature Extractor
Feature Extractor
(s,a,r,s))
i a
Figure 1: Overview of the pretraining framework. Our approach decomposes the original Q- network into two core architectures: a shared backbone network from the concatenated state-action input (s, a). and two shallow head networks for learning the transition model and estimating Q-values, respectively.
that extracts the representation
that vary in size, coverage, and behavioral optimality, rather than focusing performance within single fixed dataset.
In this work, we propose simple yet effective plug-and-play framework that pretrains a shared Q-network via two-stage learning strategy toward data-efficient offline RL. As illustrated Figure [ our shared Q-network consists of a backbone feature extractor ha and two shallow head networks: for next-state prediction and fo for Q-value estimation. In the pretraining stage, we train h.p jointly with dv through transition prediction task, encouraging the backbone to encode dynamics-relevant representations. the subsequent RL training stage, we fine-tune h alongside fo using any standard offline RL objective. This modular design enables seamless integration with existing offline RL algorithms while improving representation quality for value estimation.
We theoretically analyze the effect of such pretraining using the projected Bellman equation under linear function approximation. Our analysis reveals that initializing the feature matrix (backbone feature extractor) with pretrained representations increases its rank, which tightens the upper bound on the optimal Q-value estimation error. Consequently, the pretrained Q-network achieves faster and more accurate convergence than conventional methods. Empirically, this structural property is validated by observing higher feature matrix ranks and lower Q-value proxy errors, as shown in Figure 3 and T
Finally, extensive experiments demonstrate that our approach substantially enhances both the perfor- mance and data efficiency of existing offline RL algorithms across diverse benchmarks, including D4RL 15), Robomimic [40], V-D4RL. (38. and ExoRL. (62]. Our method maintains strong per- formance even with limited data subsets and under varying data qualities and collection strategies. Notably, with only 10% of the dataset, our pretrained Q-network outperforms standard baselines trained on the full dataset. Moreover, our method surpasses both offline model-based and representa- tion learning approaches on reduced datasets, confirming its general effectiveness in data-efficient offline policy learning.
Further discussions how data efficiency in offline RL differs from sample efficiency in online RL are in Appendix A Additionally, the source code is available in our GitHub repository With only few additional lines top of the original TD3+BC3) impplimmntatiiooonn method is simple to implement and easily adaptable to other offline RI. algorithms.
demonstrate that
2 Related Works
Offline RL. Offline RL aims to learn policies solely from a fixed dataset without further interaction with the environment. A major challenge in this setting is distribution shift, where queries to
httpper//gtthal.ccma//aspppppppp git httpe::/ggicabb..com//ttiiiiii

---

the Q-function on out-of-distribute actions can lead to overly optimistic value estimates during
training [8] 32 37 33
16
29
Recent studies have explored scaling offline RL algorithms to larger
datasets and model capacities
44
56
well as tefline-to-online RL paradigms that pretrain
agents offline before fine-tuning them online to enhance sample efficiency 4 59
Beyond these standard formulations, other works have investigated diverse data conditions, such as imbalanced or corrupted datasets and the use of unlabeled data within the offline RL framework [27 30 33 have evaluated performance on data subsets, few have explicitly addressed data efficiency that is, how well an offline RL agent can learn with minimal data, In contrast, our work directly targets this problem. We propose a simple yet effective plug- and play pretraining method that enhances data efficiency by initializing a shared Q-network for improved policy learning from limited static datasets.
67 61] Although several studies [
Sample-Efficient RI.. A persistent challenge in most RL algorithms is sample inefficiency, as learning optimal policies typically requires extensive online interactions. Addressing this limitation has been a long-standing focus of RL research [65] 66 [2]. One prominent approach model- based RL, which improves efficiency by learning a (possibly latent) dynamics model to generate additional synthetic transitions [34
21
25 Alternatively, techniques such as representation have shown strong empirical gains in sample
pretraining (50] [6] and data augmentation 34
65
efficiency by enhancing feature reuse and robustness.
More recently, offline-to-online RL 36 3
43 and foundation model approaches
6
have emerged to mitigate the poor sample efficiency of online RL. These methods leverage large- scale offline data or pretrained models to accelerate subsequent online adaptations, underscoring the growing convergence between sample-efficient and data-driven RL paradigms.
Data-Efficient Offline RL. In this work, we define data efficiency offline RL as the ability of an algorithm to learn an optimal policy from minimal set of pre-collected samples. This differs from sample efficiency in online RL, which concerns minimizing environment interactions. While prior works [50] 51) have discussed
RL, their primary focus was online settings thus addressing sample efficiency rather than true offline data efficiency. These methods typically rely on self- prediceedeiiive representation learning in latent spaces, often combined with techniques like data augmentation (65) or momentum target encoders [26]-
By contrast, our method employs self-supervised pretraining within shared network architecture to improve representation quality, without requiring auxiliary techniques or additional data trans- formations. Through extensive experiments under various dataset qualities and distributions, we demonstrate that our approach consistently improves performance in offline RL, effectively addressing the data efficiency problem defined in this work.
In Appendix we provide additional discussions on related approaches and broader connections to representation learning and model-based methods in RL.
3 Pretraining Q-network with Transition Prediction Improves Data Efficiency
In this paper, we propose simple yet effective pretraining framework that transfers learned transition features into the initialization of Q-network to improve data efficiency in offline RL. To this end, we design shared Q-network architecture combining a backbone feature extractor hyp and two shallow head networks: a transition head Jy for next-state prediction and a Q-value head fo for estimating Q-value. We further introduce a two-stage learning strategy- - pretraining and an RL training- built upon the shared Q network for data-efficient offline RL. During the pretraining stage, the transition model gy o hip predicts the next state given state-action pairs (s, a):
(g+ (s,)s, (s. S x A.
(1)
where 8' denotes the predicted next state, and ge is parameterized linear function.
In the subsequent RL training stage, the same backbone network hip is shared with the Q-vauue head fe. forming the Q-network:
Qs. a) = (fo (s) (8, S
where So represents a linear output layer, and hi corresponds to the fully connected layers shared with the transition model in I.. The overall shared architecture is illustrated in Figure] [
3

---

Algorithm Pretraining a shared Q-network scheme for offline RL
1: Input: Dataset D of transition (8, a, 8'), learning rate initialized parameters yp; 1
2: for each gradient step
3: Sample mini-batch B ~ D
4: Compute the next-state prediction error
Lpre(
Σ (s,a,)")EB
(8' (gv o (s)
5: Update the weights of the backbone feature extractor and the transition prediction head of the shared network
q - g- aV pfp v - a ctrr 1p)
6: end for
7: Output: Pretrained weights of the backbone feature extractor of the shared network
We pretrain the transition model o hip via self-supervised regression by minimizing the mean- squared prediction error:
Lere v))
Σ 1 (8)
(s,0,8))D
where D denotes the static dataset of transition tuples (s, s').
After pretraining, the parameters yp be fine-tuned frozen when training standard RL algorithms using the Q-network structure above, without requiring any architectural modification. By default, fine-tune the backbone feature extractor during the RL training stage, and report results with frozen backbone in Appendix The complete pretraining procedure summarized in Algorithm
3.1 Analysis Based on the Projected Bellman Equation and a Proxy Q Error
In this section, we analyze how our method improves data efficiency through the lens of the projected Bellman equation. For clarity, we assume discrete and finite state-action spaces with deterministic transitions. However, the core principles naturally extend to continuous domains.
Our analysis begins by noting that the Q-function parameterized by neural networks can be expressed as in (2). We decompose the network into two parts: a feature extractor hep and a linear function approximator Letting
(s, RT the Q-function
can be rewritten as Qp.((s,a) = Σ 0chppa(ssa) = (0, (s, where (s,a) E S A.
When &o is fixed, then the above structure can be viewed linear function approximation with the feature function hyp Our method pre- trains h by minimizing the prediction loss in (3). Here, the latent feature = corresponds to the MLP output before the final layer, while 0 parameterizes the linear output layer; i.c., Q0.p(s, a) (s, a). Thus, interpreting our network under the linear approximation framework provides useful model explain its improved data efficiency.
Qp.a
((H,))
Figure Reduced approximation error through the expanded column space of H.. In linear ap- proximation, the true value function Q may lie outside the column space of H., The projected Bellman equation addresses this by projecting Q onto its closest representation IIQ within the col- umn space of H.p
well known that under linear function ap- proximation, the standard Bellman equation
Q0.((5.) R(s, a) + Σ 18'(18 a) Σ Qp.o(s'aa)
s'ES
a'EA
may not admit a solution in general. However, typical TD-learning algorithms are known to converge to the unique fixed point of the projected Bellman equation [4]] In particular, considering the vector

---

form of the Bellman equation, Q4.0 = RR the projected Bellman equation is known to admit solution
Qp.0= = 7 p..))
where denotes the projection operator onto the column space ((H)) of the feature matrix Hep defined as
Hp == hp(s,a)
The corresponding approximation error is bounded by
11Q - Q" 11
[IQ" Q2
(4)
where Q" is the true Q-function corresponding to the target policy и. This bound highlights that the estimation error depends on the expressiveness of H.. A richer feature representation-iae a column space ((Hp) that better spans Q Icads a smaller Bellman error (Figure 2p.
To empirically validate this interpreta- tion, compare the rank of the la- tent feature space between the vanilla TD3+BC and pretrained TD3+BC models using 512 samples. Following (31, measure the hard rank (instead of soft rank used in (3I). As shown in Figure 3] method yields signifi- cantly higher feature-space rank, indi- cating that expands C(Hp) and thus captures larger subspace of IR Sx.A), This expansion enables more precise Q- function estimation with the same num- ber of
hallf heetahhehhhhh medium-eapprt--t2
boppes-edium-)
203
260
200
220
202
TD3+8C
TD3+BC
TD1+4CC
our
300
110
100
150
200
100 150
200
Figure 3: The Rank of the latent space in the Qnetwork during training. We compare the rank of the latent repre- sentations between vanilla TD3+BC and TD3+BC+Ours using 512 samples. Our method consistently maintains a higher latent-space rank, indicating a richer feature repre- sentation and reduced approximation error.
more ac- curate learning with less data.
To further verify that our method reduces Bellman error under limited data, we define a proxy Q-error based the left-hand side of (4). We estimate Q" using the critic of TD3+BC trained with the full dataset, and estimate Qu from TD3+BC and TD3+BC+Ours trained with only 10% of the data. Across D4RL benchmarks, T shows that our method consistently yields lower proxy Q-error than the baseline. These results demonstrate that our pretrained representation facilitates more accurate Q-function estimation, leading to superior data efficiency in offline RL.
Table Comparison of proxy Q-error between TD3+BC and TD3+BC+Ours. We compare the proxy Q-error of vanilla TD3+BC and TD3+BC+Ours, both trained on 10% of the D4RL datasets. The reference optimal Q-value is estimated using the critic of TD3+BC trained on the full datasets. Our method consistently achieves lower proxy Q-error, demonstrating more accurate Q-function estimation with substantially less data hata--highlightinn its superior data efficiency.
TD3+BC
TD3+BC Ours
HalfCheetah
503.6851
287.7740
Medium
Hopper
536.2078
472.9333
Walker2d
299.1773
138.0320
HalfCheetah
1032.6671
119.9188
Medium Replay
Hopper
320.1790
319.9531
Walker2d
379.4456
49,9275
HalfCheetah
522.4211
112,2337
Medium Expert
Hopper
437.8861
254.9155
Walker2d
392.7360
148.2040
5

---

4 Experiments
Table 2: Average normalized scores on the D4RL benchmark. Each column corresponds to a different RL baseline. The values on the left represent the baseline scores reported in the original literature, while the values on the right show the results of our method combined with each bascline. Performance improvements over the original baselines are highlighted in blue. All results are reported with the mean and standard deviation scores over five random seeds.
AWAC
CQI
IQL.
TD3-C
HallChecrah
217::99-33194222
103+09 + 18.3-00
2.8-00-+188-03
Randoen
Hopper
107201-391223
94404 107404
117=12-311622 6:00
Walker24
78::02--111133
27#12--196245
79405
89407
59=10-=11:2551
HallCheetah
48.42.011-54..61133
5722403-39994183
466402-2489400
48.5:011-99220:
Hopper
184-888-100077-0..
2426:03--906-222
789453-7756-122
610-119-7755-22
Walker2d
53.01::3.2-895555
$7.5283-8871177
818115-8361111
1.11
HallChootah
4194111-+46610
414103-455402
400:05-+58::03
Medium Replay
Hopper
101.:::0.6-10067-:::-0
286::09--966:12.11
962-19-999111
729:733--0002:1.
Walker2a
16.12:55-1100::2..1
158126-88.7213
779121-880117
380:13-+920:116
HaliCheetah
764228-9901::11
27.1#39-878816
948202-4955
768=18-9999200
Malium Expert
Hopper
1150207--11522220
111412-1111
1018475-
1022296-1113:-22
Walker2d
10532153--11111133
88.11133-99161--2
111:6:07-+112::100
1095202-+1116200
9644088-9335-.11
824474-9911410
964+02 97.40011
940-02-9900066
Expert
Hopper
1128504--112951 112421-1111100
1131 111.3:25
1130:01-1114403
Walker2d
1004200-11.2204
1858776-1116-003
110.703 -3 1128+1.1
1099203-111.0:02
In this section, we evaluate the effectiveness of our method across a range of offline RL benchmarks, including standard D4RL, the more complex Robomimic domain, and the image-based V-D4RL environment. We further examine data efficiency by evaluating performance on partial subsets of D4RL and ExoRL datasets. We begin by describing the experimental setup and the baselines used for each experiment. The experimental evaluation is structured as follows: first, we compare performance improvements standard offline RL benchmarks; second, analyze data efficiency across varying dataset qualities; and finally, we investigate performance across different dataset distributions.
Experimental setup and Baselines. We consider heterogeneous tasks and diverse datasets to ensure comprehensive evaluation. For locomotion tasks, we evaluate our method on the D4RL. benchmark [15] with three agents (HalfCheetah, Hopper, Walker2d) and five dataset types (random, medium-replay, medium, medium-expert, expert). Our method is applied to popular offline RL algorithms-AWe AWAC [2). CQL (33, IQL [29]. and TD3+BC [16]-and scores between the baseline and the baseline augmented with our pretraining framework.
compare normalized
For tabletop manipulation tasks, we use the Robomimic benchmark [40] with Lift and Can tasks and mixed-quality machine- machine-generaiee generated (MG) datasets. We compare the success rates of IQL, TD3+BC, IRIS [3], and BCQ [18] with and without our method.
For high-dimensional vision-based tasks, we evaluate on Cheetah Run and Walker Walk in V- D4RL [38] building our method on top of DrQ+BC, which applies the same regularization of TD3+BC into DrQ-v2 (63)
For data-efficient offline RL, we investigate both the impact of dataset quality and dataset collection strategy. We evaluate reduced D4RL locomotion datasets MOPO (68] MOBILE (53, and ACL 600, and reduced ExoRL datasets 662 (walker walk: SMM (35), RND {8) ICM 146); point
point mass maze: Proto 164%, DIAYN (13)) on TD3 [17 and CQL. Detailed experimental and implementation
settings are provided in Appendix D and Appendix C
4.1 Performance Improvement in Offline RL Benchmarks
To validate the effectiveness of our approach, we evaluate it on the D4RL and Robomimic benchmarks. Table 2 compares the normalized scores of baseline algorithms and their counterparts augmented with our method across various environments and datasets. When integrated with existing offline RL methods (i.e., AWAC, CQL, IQL, and TD3+BC), our approach consistently improves performance in most settings. The blue-highlights scores in Table 2 indicate gains over the corresponding baseline algorithms. Notably, AWAC shows an average performance improvement of + 140.37% compared to its original version.

---

halfdh uocaa mwd www.-oodddmmmmeooommmom modlyy
hoopper mode
Wal www.wmeeee.............
8
Noma netum
Normateed mitum
8
18
8
8
100 protrams
Eraestpp (ched)
Frresdog (cbed)
Figure 4: Learning curves of TD3+BC. We represent the normalized scores of the vanilla TD3+BC (blue) and TD3+BC (orange) with our pretraining method, respectively. The vertical red dashed lines indicate the transition between the pretraining and main training phases. After pretraining. TD3+BC with our method rapidly surpasses the vanilla baseline by significant margin.
w/a pretraining
w pretraining (ours)
Env: Lift
Env: Can
0.6
0.5
Success rate
0.4
0.3
0.2
0.0
IQL
TD348C
IRIS
BCQ
KGL
IRIS
BCQ
Algoritim
Algorithm
Figure 5: Average success rates the Robomimic benchmark. We compare the baseline methods without pretraining (blue) against those augmented with our pretraining approach (orange) over three seeds. In seven out of cight tasks, our method yields a substantial improvement in success rate across both environments.
Figure presents the learning of TD3+BC, demonstrating the clear benefit of pretraining strategy. After the pretraining phase (denoted by the red vertical lines), the pretrained agent quickly outperforms the vanilla TD3+BC and achieves higher returns throughout training. These results indicate that our method accelerates convergence and enhances asymptotic performance with only minimal architectural provided in Figure 12 of Appendix
algorithmic modifications. Complete learning curves for TD3+BC are
We further evaluate method large-scale robotic manipulation tasks from the Robomimic bench- mark to assess its effectiveness in complex, real-world scenarios. These tasks include suboptimal transitions, providing challenging testbed beyond the D4RL benchmark. Figure [5] reports the averaged success rates of four offline RL baselines, both with and without our pretraining method. As shown, incorporating our approach consistently improves performance in seven out of eight cases, demonstrating its robustness in complex tasks.
Additional experiments the Adroit benchmark (24-DOF control) presented in Appendix B We apply our method to AWAC, IQL, and TD3+BC across twelve settings (i.e., four environments x three datasets) and evaluate each over five random seeds. In most cases, our method achieves clear performance gains, further confirming its effectiveness in high-dimenssonal control.
Finally, we examine the scalability of our approach to high- eigh-dimensional visual input using the V- D4RL benchmark (38]. Similar to other rision-based offline RL methods, our framework integrates seamlessly by replacing the state input with latent representations extracted from visual encoder. As shown in Figure g our method consistently enhances the performance of DrQ+BC 63 validating its applicability image environments.

---

Table Average episode returns on the V-D4RL benchmark. We evaluate our approach in the image-based environment over three seeds. The results show that integrating our method consistently improves the performance of DrQ+BC.
DrQ+BC
DrQ+BC Ours
Walker walk
306.93:1.8.211
338.77-129.55
Medium
Chectah Run
340.33:7.55
379 899.804455.8
Humanoid Walk
12.57::..3
20.03 3.80
Walker walk
30.09:10.75
28.68:42.229
Medium Replay
Cheetah
21.15-+:2.04
25.13-4:2.04
Humanoid Walk
40.76:11627
19.38:16110
Walker walk
352.46:1:7..15
099.66-120.866
Medium Expert
Cheetah Run
251.52++334.7
258. 50.333
Humanoid Walk
4.11+22.72
5.12+1.889
Data Amount
BCrsurs
100
TD3, BC+oours
TD3 BC-sours 10%
TD3,
BCHours 100%
Avg DARL Score
random--22
medum-nuplay-22
impert-22
Data Quality
Figure 6: Average normalized scores across varying dataset sizes and qualities. We present the performance of our method on progressively reduced datasets 17,, 3%, 10%, 30%, 100%) across three D4RL environments: HalfCheetah, Hopper, and Walker2d. We demonstrate that our method remains highly data-efficient, achieving strong performance even with only 10% of the data and as little 1% for random datasets and 3% for medium datasets regardless of data quality.
Data Efficiency across Data Qualities
To evaluate the data efficiency of our method across different dataset gualities, we tested it with TD3+BC on progressively reduced subsets of the D4RL datasets (1%, 3%, 10%, 30%. and 100%) spanning various data qualities (random, medium, mediumm-replas medium-experr, expert). Each reduced dataset constructed by uniformly sampling transition segments (s, a, r, s') from the full dataset, followed by both pretraining and RL training using these subsets.
As shown in Figure 14 our method demonstrates remarkable data efficiency. On the random datasets, training with only 1% of the data surpasses the performance of the vanilla TD3+BC trained on the full dataset for the HalfCheetah and Walker2d environments. Similarly, on the medium datasets, our method achieves comparable or higher performance with only 3% of the data. For higher-quality datasets (medium-replay,
aedium-expert, and expert), our method using merely 10% of the data consistently outperforms the vanilla TD3+BC trained on the entire dataset. Overall, as summarized in Figure 6, our method delivers robust performance with as little as 10% of the original dataset, confirming its strong data efficiency in offline RL.
We further compare our approach with representative offline model-based and representation--earmin methods. Experiments are conducted on medium, medium-replay, and medium-experi datasets of D4RL over three seeds. Figure 7 presents the aggregate results, while Figure 15 provides detailed comparisons. The results show that our method preserves high performance under reduced data conditions, unlike competing methods that incur additional training overhead (c.g., transition model

---

hoftheorah
booper
velleral
INZNEL VINOO Vosile
Aggo T2SHIC CLISS HOPP
N550
Monde
Modll
Kon Srow
AiQ D4TL Score
13
20
Date Amnouni
Dala Arroust
Data Arnount
Figure Comparison with other approaches on the D4RL benchmark. We compare our method against model-free offline RL baselines, model-based offline RLs (MOPO, MOBILE), and a representation learning approach (ACL) across three random seeds. The results are averaged over medium, medium-replan and medium-expert datasets. Our method consistently maintains high performance under severe data reduction particularriittytl % of the dataset-where competing methods degrade significantly.
453
TDS
E00
'03
T23
TDS Cura
720
105 Cus
123 Outo
1000
800
520
700
230
8og
400
198
338
400
15%
1007
10%
100%
15
120%
Data Amounnt
Data Artount
Data Amount
Figure 8: Average episode returns in reduced datasets across data collection strategies. We evaluate our method under different dataset collection strategies (SMM, RND, ICM). Across all cases, TD3 combined with method consistently outperforms vanilla TD3, even when trained with only 10% of the data- surpa-sur the performance the full-data baseline. These results demonstrate that method achieves strong data efficiency regardless of the underlying data distribution.
Across
training and inference). Consequently, our method emerges as more effective and computationally efficient choice for data-efficient offline RL.
4.3 Data Efficiency across Data Distributions
We hypothesize that smaller datasets induce distributional shifts compared to larger ones, they often exhibit narrower coverage of the visited state space. To examine this effect, we evaluate our method across datasets generated by different collection strategies, each producing distinct data distributions. Using the ExoRL benchmark, we select TD3 as the baseline and consider datasets collected with SMM, RND, and ICM for the walker walk task. As reported in 62 ICM achieves the highest performance among the three, followed by RND and SMM. We compare vanilla TD3 and TD3 augmented with our method using reduced subsets of the datasets (1%, 10%, 100%) over three random seeds. Reduced datasets are constructed by taking the initial segments of the trajectories, and both pretraining and RL training are conducted on these subsets. As shown in Figure 8] our method consistently outperforms vanilla TD3 across all dataset types, when using only 10% of the data. Notably, on the RND dataset, training with just 1% of the data achieves remarkably high returns, exceeding the full-data baseline.
We further investigate the robustness of our method on datasets with highly limited state coverage using the point mass maze environment from ExoRL. Figure 9 visualizes the trajectories from reduced datasets collected via DIAYN and Proto strategies (1% of DIAYN, 7% of Proto). Compared to Figure 2 in (62). our settings exhibit even narrower state support. For example, the DIAYN dataset shows sparse trajectories near the top-right goal, while the Proto dataset shows limited coverage near the bottom-right goal. To assess performance under such constrained coverage, evaluate

---

79 cM
Reach bottn right
Prote
131
Figure 9: Effectiveness of our method on datasets with narrow state coverage. (Left) Visualization of goal-reaching agents and their trajectories under different goals, dataset fractions, and exploration strategies. (Right) Average episode returns of CQL trained with two datasets, with and without our pretraining method. Our approach yields significant performance gains even when the available data has limited state coverage.
CQL with and without our pretraining method on these datasets for both short-horizon (reach top- right) and long horizon (reach bottom-right) goals. As shown in Figure 9 our method significantly improves performance even under severe state- distrriittiittion limitations. These results collectively demonstrate that approach achieves strong data efficiency and remains effective across diverse and distributionally shifted offinn datasets.
Conclusion
In this paper, we propose simple yet effective data-efficient offline reinforcement learning method that pretrains shared Q-network through a transition prediction task. The proposed framework leverages shared network architecture that jointly predicts the next state and the Q-value, allowing efficient feature reuse between the transition model and value function. This design makes our approach easily applicable a wide range of existing offline RL. algorithms and substantially improves data efficiency, maintaining strong performance even when trained on limited data.
To verify the effectiveness of method, we analyzed it under the framework of the projected Bellman equation and performed extensive experiments across diverse offline RL benchmarks, including D4RL, Robomimic, and V-D4RL. The results demonstrate that our approach consistently enhances the performance of existing offline RL methods. Furthermore, evaluations on reduced datasets and shifted data distributions confirm that our method is robustly data-efficient across varying data qualities and distributions.
Limitations & Future Works. This study focuses standard model-free offline RL settings, where popular algorithms primarily rely on Q-function learning. Consequently, our design is tailored to complement such architectures. Nonetheless, prior works (51 (88) suggest that offline pretraining can be beneficial broader contexts, such as unsupervised learning or goal-conditioned RL. Owing to its simplicity and plug-and-plas compatibility, Future research will extend our framework to more general settings, including offline-to-online RI., goal-conditioned RL, and real-world control scenarios.
method has strong potential for broader applications.
10

---

References
[1] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforcement learning. In International conference on machine learning, pages 114. PMLR, 2020.
[2] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng. i say: Grounding language in robotic affordances. (arXIv:2204.01691) Aug.
Do i can, 2022. atXiv:2204.0169 [cs].
[3] J. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577-1594, PMLR, 2023.
[4] C. Bhateja, D. Guo, D. Ghosh, A. Singh, M. Tomar, Q. Vuong, Y. Chebota,,, S. Levine, and A. Ku- mar. Robotic offline from internet videos via value-function pre-training. (arXiv:2309.1304) Sept. 2023. 1XXi:23399.3044 [es].
[5] M. Botvinick, S. Ritter, J. X. Wang. Z. Kurth-Nelson, C. Blundell, and D. Hassabis. Reinforce- ment learning, fast and slow. Trends in cognitive sciences, 23(5):408-444 -22)22242222 2019.
[6] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, Fu, M. G. Arenas. K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lec, S. Levine, Y. Lu, H. Michalewski, Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-2: Vision-language action models transfer web knowledge to robotic control. (arXiv:2307.158 July 2023. arXiv:2307.155: [cs].
[7] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog. J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manju- nath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control scale. (arXiv:2212.06811
Aug. 2023.
[8] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. / 12894, 2018
[9] Y. Chebotar, Q. Vuong, K. Hausman, F. Xia, Y. Lu, A. Irpan, A. Kumar, T. Yu, A. Herzog, K. Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In Conference on Robot Learning, pages 3909-3928. PMLR, 2023.
[10] J. Cheng, R. Qiao, G. Xiong, Q. Miao. Y. Ma, B. Li, Y. Li, and Y. Lv. Scaling offline model- based rl via jointl--optimizz world-action model pretraining. arXiv preprint arXiv:2410.00564, 2024.
[11] M. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference machine learning (ICML-11), pages 465-472, 2011.
[12] P. D'Oro, M. Schwarzer, E. Nikishin, P.-... Bacon, M. G. Bellemare, and A. Courville. Sample- efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
[13) B. Eysenbach, A. Gupta, Ibarz, and S. Levine. Diversity is all you need: Learning skills without reward function. arXiv preprint arXiv:1802.06070, 2018.
11

---

[14] Y. Feng, N. Hansen, Z. Xiong, C. Rajagopalan, and X. Wang. Finetuning offline world models in the real world. Oct. 2023.
[15] Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint (XXi::00040022199 2020.
[16] S. Fujimoto and S. Gu, A minimalist approach offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, page 20132-20145. Curran Associates, Inc., 2021.
[17] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[18] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without ploration. In Proceedings of the 36th International Conference on Machine Learning, page 0052-2062 PMLR, May 2019.
[19] Z. Guo, M. G. Azar, B. Piot, B. A. Pires, and R. Munos. Neural predictive belief representa- tions. arXiv preprint arXiv: /81 1.06407, 2018
[20] T. Haarmoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:7812.05905. 2018.
[21] D. Hafner, T. Lillicrap, Ba, and M. Norouzi. Dream control: Learning behaviors by latent imagination. (arXiv: 191.00v691,.,,,33,, 2019.
[22] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, page 2555-2565. PMLR, 2019.
[23] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint xxXv:2301..44044, 2023.
[24] N. Hansen, H. Su, and X. Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.1683 2023.
[25] N. Hansen, X. Wang, and H. Su. Temporal difference learning for model predictive control. S04iv:22003049555) July 2022. arXiv:2203.0493 009955 [cs].
[26) K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9733 2020.
[27] Z.-W. Hong, A. Kumar, S. Karnik, A. Bhandwaldar, A. Srivastava, J. Pajarinen, R. Laroche, A. Gupta, and P. Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. Advances in Neural Information Processing Systems, 36:4985-5009, 2023.
[28] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810-21823, 2020.
[29] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning (arXiv:2110.06169) Oct. 2021. arXiw:2110.06169 [cs].
[30] A. Kumar, R. Agarwal, D. Ghosh, and S. Levine. Implicit under- paraeetarramttiiizaatoon inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010. /4498, 2020.
[31] A. Kumar, R. Agarwal, T. Ma. A. Courville, G. Tucker, and S. Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. arXiv preprint arXiv:2112.04716, 2021.
[32] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via boot- strapping error reduction. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
12

---

[33] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. (arXiv:0006.44799, Aug. 2020, arXiv:2006.04779 [cs, stat].
[34] M. Laskin, K. Lee, A. Stooke, L. Pinto, Abbeel, and A. Srinivas. Reinforcement learning with augmented data, Advances in neural information processing systems, 011111884-1999 2020.
[35] L. Lee, B. Eysenbach, Parisotto, E. Xing. S. Levine, and R. Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint 190..00664.974, 2019.
[36] S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic g-ensemble. In Proceedings of the 5th Conference on Robot Learning, page 1702-1712. PMLR, Jan. 2022.
[37] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems.
(tt1:20005011443) Nov. 2020. arXiv:2005.01e: 0.6643 [es, stat].
[38] C. Lu, P. J. Ball, T. G. Rudner, J. Holder, M. A. Osborne, and Y. W. Teh. Challenges and opportunities offline reinforcement learning from visual observations. arXiv preprint arXiv:2206.04777 2022.
[39] A. Mandlekar, F. Ramos, B. Boots, S. Savarese, L. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420. IEEE, 2020.
[40] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín- Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.0:22 2021.
[41] S. Melo and M. Ribeiro. Q-learning with linear function approximation. 2007.
[42] A. Nair, A. Gupta, M. Dalal, and Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006. 09359, 2020.
[43] M. Nakamoto, S. Zhai, A. Singh, M. Sobol Mark, Y. Ma, C. Finn, A. Kumar, and S. Levine. Cal-ql: Calibrated offline pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36, 2024.
[44] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models, arXiv preprint 131::2310008866 2023.
[45] Y. Pan, J. Mei, A.-m. Farahmand, M. White, H. Yao, M. Rohani, and J. Luo. Understanding and mitigating the limitations of prioritized experience replay. In Uncertainty in Artificial Intelligence, pages 1561-1571, PMLR, 2022.
[46] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017.
[47] R. Rafailov, K. B. Hatch, V. Kolev, J. D. Martin, M. Phielipp, and C. Finn. Moto: Offline pre-training to online fine-tuning for model-based robot learning. In Conference Robot Learning, pages 3654-3671. PMLR, 2023.
[48] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 700:::109000888 2017.
[49] T. Schaul, J. Quan, Antonoglou, and D. Silver. Prioritized experience replay. arXiv preprint arXiv:5511.05552 2015.
[50] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.00 05929, 2020.
13

---

[51] M. Schwarzer, N. Rajkumar, M. Noukhovitch, A. Anand, L. Charlin, R. D. Hjelm, P. Bachman, and A. C. Courville. Pretraining representations for data-efficient reinforcement learning. 2021.
Advances in Neural Information Processing Systems,
[52] Y. Seo, K. Lee, S. James, and P. Abbeel. Reinforcement learning with action- action-free pre-training from videos. International Conference on Machine Learning, page 19561-19579, PMLR, 2022.
[53] Y. Sun, J. Zhang, C. Jia, H. Lin, J. Ye, and Y. Yu. Model-bellman inconsistency for model- based offline reinforcement learning. In International Conference on Machine Learning, pages 33177-33194. PMLR, 2023.
[54] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 264):160-163, 1991.
[55] R. Sutton, A. G. Barto, al. Introduction to reinforcement learning. vol. 135, 1998.
[56] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman,
C. Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024.
[57] Z. Wang. T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning. pages 1995-2003. PMLR, 2016.
[58] P. Wu, A. Majumdar, K. Stone, Y. Lin, I. Mordatch, P. Abbeel, and A. Rajeswaran. Masked trajectory models for prediction, representation, and control. In International Conference on Machine Learning, pages 37607-37623. PMLR, 2023.
[59] T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395-27400
2021.
[60] M. Yang and O. Nachum. Representation matters: Offline pretraining for sequential decision making. In International Conference on Machine Learning, pages 11794-17994 PMLR, 2021.
[61] R. Yang, H. Zhong, Xu, A. Zhang, C. Zhang, L. Han, and T. Zhang. Towards robust offline reinforcement learning under diverse data corruption. arXiv preprint arXivv2310.12955, 2023.
[62] D. Yarats, D. Brandfonbrener, H. Liu, M. Laskin, P. Abbeel, A. Lazaric, and L. Pinto. Don't t change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprini 12Xi::2201.33444 2022.
[63] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning, arXiv preprint arXiv:2107.09645 2021.
[64] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Reinforeement learning with prototypical representations. In International Conference on Machine Learning, pages 11920-11931. PMI.R 2021.
[65] D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning representations, 2021.
[66] D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pineau, and R. Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the aaai conference on artificial intelligence, volume 35, pages 10674-10681, 2021.
[67] T. Yu, A. Kumar, Y. Chebotar, K. Hausman, C. Finn, and S. Levine. How to leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning, pages 25611 1-66355 PMLR, 2022.
[68] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou,, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129- 14142, 2020.
14

---

A Comparison over Sample Efficiency and Data Efficiency
In this section, we further expand the discussion on comparing the sample efficiency and data efficiency. In both online and offline RL, learning an optimal policy from limited experience data remains a fundamental challenge. Agents in both paradigms aim to extract maximal information from a finite set of interactions with the environment. This shared motivation arises from the practical constraints of data acquisition, as gathering experience in real-world or high-fidelity simulation environments is expensive and time-consuming. Consequently, both settings emphasize effective data utilization through methods such as representation learning [50] 5T 66 60). model-based
RL [2 21 28 68 and off-policy optimization [20] 18), which aim to accelerate learning without
proportional increases in data volume. In essence, both sample efficiency in online RL and data
efficiency in offline RL quantify how effectively an agent transforms its available experience- whether gathered online provided offline-into improved decision-makinn performance.
Despite this shared goal, their underlying desiderata differ substantially. online RL, an agent alternates between two intertwined phases: experience gathering and policy update. The policy controls the distribution of collected data, which in turn affects subsequent policy updates. As a result, the experience distribution is non-stationary and tightly coupled with the evolving policy, making it theoretically and practically challenging to analyze or control 49 45]. In contrast, offline RL learns from fixed dataset collected by a separate behavior policy, where data distribution is static but often limited in coverage. While online RL suffers from issues such as incremental network updates and weak inductive biases (5), offline RL must contend with distribution shift and extrapolation errors (37. which hinder generalization beyond the support of the dataset.
B Extended Discussion on Methodological Connections
In this section, we address potential concerns regarding the novelty of our method, given its conceptual connection to several prior approaches in related research areas. We provide detailed comparisons and clarifications across two primary themes: representation learning and model-based reinforcement learning.
Representation Learning. Recent years have witnessed surge of research on predictive repre- sentations in reinforcement learning. Our agproach-pretraiingg a shared Q-network through next-state prediction task shaskssharess the spirit of prior work on representation learning for improving data efficiency [50 19). yet differs in key methodological aspects.
Specifically, Schwarzer et al. [50) proposed an online self-supervised pretraining scheme based on latent-space prediction, relying heavily auxiliary design choices such as data augmentation [65] and target encoders [26). In contrast, our method adopts a supervised pretraining objective directly on the next-state prediction task, avoiding such additional mechanisms. This simplicity enables our approach to be seamlessly integrated with diverse offline RL algorithms while consistently improving data efficiency and performance across locomotion and manipulation benchmarks.
Similarly, Guo et al. 19 introduced an unsupervised belief-state encoder for partially observable settings (POMDPs). Their focus lies in inferring the hidden state from trajectory using recurrent GRU-based network that predicts future observations. In contrast, our approach operates in the fully observable MDP setting using a simple MLP-based architecture that predicts the next state 8,411 from the current state-action pair (8t, az). Thus, while both approaches leverage predictive modeling, our contribution lies in unifying this principle with offline RL through shared Q-network architecture that enhances data efficiency without sequential modeling or partial observability assumptions.
Model-based RL. While method employs transition prediction objective. its design philosophy and application differ fundamentally from conventional model-based RL. Classical model-based approaches [54] explicitly use the learned dynamics for planning policy improvement, whereas our approach leverages transition prediction solely for representation pretraining.
Recent methods such as TD-MPC (25] and TD-MPC2 24 integrate model-based objectives by recursively feeding outputs of a shared encoder for transition and value learning. Our approach instead introduces a dueling-style shared architecture [57] where distinct shallow heads are used for
the transition model and Q-value estimation. Moreover, we propose a two-phase training scheme: first, transition transition-prediction prediction pretraining phase that shapes the shared backbone; and second, standard
15

---

RL training phase that fine-tunes the Q-network initialized from the pretrained backbone. This staged training framework reduces complexity and training cost while yielding substantial data efficiency gains.
JOWA 10 also employs shared Transformer-basee backbones for multi- offline RL through sequence modeling. However, while JOWA focuses on scaling across tasks and environments with few-shot fine-tuning, our objective improve data efficiency in conventional single-task offline RL without additional architectural or training overhead.
Dreamer [33 further advanced model- RL with sophisticated latent world models and recon- struction objectives for planning. Although highly effective, such designs introduce considerable computational and data requirements. In contrast, our method provides lightweight, plug -nn--nd-pp play alternative that enhances sample efficiency within existing offline RI. frameworks, offering a practical balance between architectural simplicity and empirical performance.
Summary. In essence, our work departs from both model-based and paradigms by introducing a shared Q-network architecture with two-phase supervised pretrain- ing. This simple yet powerful mechanism enhances feature reuse, reduces approximation error, and consistently improves data efficiency-withoe the need for auxiliary objectives, sequential modeling, or complex multi-stage optimization.
C Implementation Details
This section provides the detailed implementation setup used in our experiments. Since our proposed
method plug-and-play pretraining approach applicable to popular offline RL algorithms, we build directly open-source implementations for fair and consistent comparisons. Specifically, we adopt publicly available PyTorch-based repositories for each baseline:
D4RL
AWAC
CQLF
- IQL
-TD3+BC
*
Robomimic
Official Robomimic repository for all baselines !
We restrict our comparisons to PyTorch-basr baselines to ensure implementation consistency.
For D4RL experiments, each agent is trained for million gradient steps per environment across five random seeds. Evaluation conducted every 5k gradient steps for AWAC, CQL, and TD3+BC, and every 10k steps for IQL, using five rollouts per evaluation. For Robomimic experiments, each agent trained for 200k gradient steps per environment and evaluated using 50 rollouts across five seeds. All reported results in tables and figures correspond to the best evaluation scores achieved during training. All experiments were conducted on a single NVIDIA RTX A5000 GPU for both training and evaluation.
D Tasks and Datasets
In this section, we describe the experimental setups for the tasks and datasets used in our study. The corresponding environments are illustrated in Figure 10
http:s//gghaabbboo///aaarrrrarap
hatpaa//ggtthbbbbcom/tttijiiyaaa
16

---

(a) HalfCheetah
(b) Hopper
(c) Walker2d
(d) Lift
(e) Can
(1) Walker Walk
Figure 10: Illustrations of tasks. (a-c) Locomotion tasks on the OpenAl Gym MuJoCo and D4RL benchmark; (d-e) Tabletop manipulation tasks on the Robomimic benchmark; (f) Locomotion task on ExoRL benchmark.
D.1 D4RL
The D4RL benchmark consists of eight distinct task families. For our main experiments, we focus on the OpenAl Gym MuloCo continuous control suite, which includes four environments: HalfCheetah, Walker2d, Hopper, and Ant.
Each environment provides five datasets that differ in data quality and collection strategy:
Random (IM samples): Collected using randomly initialized policy.
Expert (IM samples): Collected from a policy fully trained with SAC.
Medium ( (IM samples): Collected from partially trained policy, achieving roughly one-third the expert's performance.
Medium-Expert (MM samples): A 50-50 combination of the medium and expert datasets.
Medium-Rephe Replay ( 3M samples): Collected from replay buffer of the medium-level agent during training.
All environments have an episode horizon of 1000 steps, and each agent's objective is to maximize forward velocity while avoiding instability. More details can be found in the official D4RL. repository: https://gitub..cmm/aaa Faraaarrraaaoouattiiooooo/
D.2 Robomimic
The Robomimic benchmark provides a large and diverse collection of robotic manipulation demon- strations, including both human and machine-generated data of varying quality. In our experiments, we use the machine-genneatee (MG) datasets, which are produced by training SAC agents on each task and saving demonstrations from intermediate checkpoints obtain mixed-quality data. We select
17

---

these datasets because our method consistently demonstrates strong performance suboptimal data, as observed in D4RL. Each environment has an episode length of 400 steps. The Lift task requires the robot to lift a cube above a designated height, whereas the Can task requires placing can into the appropriate container. More details are available at: http://sttttt.oooo/////////ttttt
D.3 ExoRL.
The ExoRL benchmark provides exploratory datasets for six domains from the DeepMind Control Suite:
Cartpole, Cheetah, Jaco Arm, Point Mass Maze, Quadruped, and Walker, comprising total of 19 tasks. Each dataset is collected using nine unsupervised RL algorithms-AP3 APT, DIAYN, Disagreement, ICM, ProtoRL, Random, RND, and SMM- MeMM--mmleeeeted in the Unsupervised Reinforcement Learning Benchmark (URLB), each trained for 10 million steps. Further information can be found in the official ExoRL repository: http://hhhcccooo//eyeyy////////d
file
E
Experiments in Dexterous Manipulation
We further evaluate our method on the Adroit benchmark from D4RL [15). to examine its applicability complex domains speraincilpecillllllly...l dexterous manipulation. An illustration of the Adroit environment shown in Figure [T] The Adroit domain involves controlling a 24-DoF robotic hand perform four distinct manipulation tasks: Pen, Door, Hammer, and Relocate. Each task provides three datasets of varying quality:
Human: 25 expert demonstrations collected from human teleoperation, as provided in the DAPG repository [48
Cloned: A 50-50 combination of human demonstrations and 2,500 trajectories generated by a behavior-cloned policy trained on the demonstrations. The demonstration trajectories are duplicated match the number of cloned trajectories.
Expert: 5,000 trajectories collected from an expert policy that successfully solves each task, also provided in the DAPG repository.
Figure 11: Dexterous manipulation tasks of Adroit hands. (Top-left) Pen aligning pen with target orientation; (Top-right) Door opening door with a door handle: (Bottom-left) Hammer hammering nail into board; (Bottom-right) Relocate moving a ball to target position.
For evaluation, we compare AWAC, IQL, and TD3+BC, with and without our pretraining method, across five random seeds. Table 4 reports the averaged normalized scores for each task. Across all three algorithms, integrating pretraining phase consistently improves performance. These
18

---

results demonstrate that our approach generalizes effectively to complex, high nigh-dimensional do- mains extending beyond tabletop manipulation dexterous hand control tasks.
Table 4: Average normalized scores on Adroit. Each column corresponds to a different RL baseline. The values the left represent the baseline scores reported in the original literature, while the values on the right show the results of our method combined with each baseline, Performance improvements the original baselines are highlighted in blue. All results are reported with the mean and standard deviation scores over five random seeds.
AWAC
IQL
TD3+BC
13.133 10187:114.34-1100...6
Human
Hammer
11981941-+3669511
1..3=_5222+17.78119
2.4010.16-72.3..00111
000-40000000001100000
Relocare
1.31=:1.555331=221111
1.20=1.5-+1.322.1111
-.29:0.0--+011+0.11
165377-419-:14448:1111
W...::66133-997762611166
99691118555-48888:111111
Cloned Hammer
0137:17..88+12.611.11
1041-2007-11138144444
0.59:0017771177.001
2.5-42297-:9.9947777
561-3302-3500::144
0.2:4.11--:0.:::000
Relocare
0.04:0.0-+0018:0000 18:0211
091=044-:1.00=0444
4.0.+...9-000033:..000
16::99=
H8.3822.6-114779455.00
131.735:1115-111100211
Expert
Hammer
139.66-0442-1295511:00
13.661.34611199776.12222
01666711..2--1166555-...11
106455_0.29-111861111100
Relocare
0,57:0333-0:22201133
Total
0856671:77.3--997726--2
732.400-48.27-7777.7778
F RL Training with Pretrained Frozen Backbone Feature Extractor
In this section, we investigate the effect of shallow linear heads of the shared network architecture. Specifically, pretrained TD3+BC and subsequently froze all network parameters except for the final linear layer during the RL training stage. The blue-colored entries in Table 5 indicate performance improvements relative the original TD3+BC results.
Interestingly, even when only the final linear layer was trained and the shared backbone remained frozen, the model achieved higher performance than the vanilla CQL baseline. Furthermore, this frozen variant consistently outperformed others on suboptimal datasets (i.e., random, medium, and medium-replas suggesting that the pretrained shared representation captures sufficiently rich features for effective downstream value learning, even without full fine-tuning.
Table 5: Average normalized scores of RL training with the frozen backbone on the D4RL benchmark. Performance improvements over the original baselines are highlighted in blue. All results are reported with the mean and standard deviation scores over five random seeds.
AWAC
CQL
IQL
TD3+BC
freezed TD3+BC
HalfCheetah
2.6
21.7
10.3
10.2
6.03.42.65
Random
Hopper
28.6
10.7
9.4
11.0:1.1
10.59
Walker2d
7.8
2.7
7.9
1.4=16
7.18
HalfCheetah
48.4
37.2
46.6
42.8+-0.3
42.644+1.19
Medium
Hopper
88.4
44.2
76.9
99.5+1100
67.16+43.56
Walker2d
53.0
57.5
83.8
7..7+
72.03+40.78
HalfCheetah
46.1
41.9
43.4
43.3
40.2110.79
Medium Replay
Hopper
101.3
28.6
96.2
31.413.0
64.41119.54
Walker2d
88.1
15.8
77.9
25.2+
41.02++2.05
HalfCheetah
76.4
27.1
94.8
97.9.14.4
47.35.18.73
Medium Expert
Hopper
113.0
111.4
101.8
112.2.10.2
95.07-L15.27
Walker2d
103.3
68.1
111.6
101.119.3
74.75.10.59
HalfChectah
94.4
82.4
96.4
105.7+1.9
10.73+
Expert
Hopper
112.8
111.2
113.1
112.2:0.2
113.13-10.99
Walker2d
110.4
103.8
110.7
105.7-12.7
47.14.L44.96
19

---

G Learning Curves
In this section, we provide the full learning curves in Section 4.1 for further insights.
tanckll epplay
E:1 adveupen
heperrandon
roppermedin
uxdiar
waleora moken
ndllaai cucrt
Nowralmn
linesop- bebel
(inculop 193251
Figure 12: Learning curves of TD3+BC on the D4RL benchmark. We represent the normalized scores of the vanilla TD3+BC and TD3+BC with our method on progressively reduced datasets (3%, 10%, 30%), respectively. The vertical red dashed lines indicate the transition between the pretraining and main training phases.
H Rank of Latent Space during the Learning Time
We further depict the rank of the latent feature space across tasks and datasets in Section 3.1 for comprehensive view.
Stal
Figure 13: Rank of the latent feature space of Q-network during the entire training. We provide the rank of the vanilla TD3+BC and TD3+BC with our method, respectively. The horizontal red dashed lines stand for the full rank of the latent feature space.
20

---

I Experiments with Varying Data Qualities and Sizes
This section provides more details on ablating the data quality and size, which is an extension of
Section 4.2 All experimental results are reported with the mean and standard deviation normalized
scores over five random seeds, following the same configuration in Section
WWWCC
Figure 14: Average normalized scores across dataset optimal quality and sizes. We compare performance of method with TD3+BC in progressively reduced datasets (i.e., 1%. 3%, 10%, 30%, 100% of each dataset) to vanilla TD3+BC across the data qualities (i.e, random, medium, medium replay, medium expert, expert) D4RL. Aggregated results (Bottom Right) suggest that our method guarantces better performance even in 10% of the datasets regardless of the data quality of the dataset.
(a) Medium
(b) Medium Replay
(c) Medium Expert
(d) Average
Figure 15: Comparison with offline model-based RL and representation approaches. We compare TD3+BC, AWAC, CQL with ours to offline model-based RLs (i.e., MOPO, Mobile) and a representation RL (i.e., ACL) on D4RL over three seeds. The graph shows results over medium. medium-replay, medium-expe expert datasets. The results show that method maintains the performance in reduced datasets, especially %, unlike the other approaches.
21

---

Table Average normalized scores of AWAC across dataset sizes and qualities.
w/o pretrain
w/ pretrain, 10%
w/ pretrain, 30%
w/ pretrain, full
HalfCheetah
2.6
9.71
36.37=1 ,47
51.10#0.89
Random
Hopper
28.6
97.05
93.35=16.34
59.47=13.779
Walker2d
7.8
8.57-t0.47
8.36++
13.11+3.91
HalfCheetah
48.4
53.47=1.522
56.64
54.6311144
Medium
Hopper
88.4
101.28=00.78
101.32:10.20
101.73:10.20
Walker2d
53.0
95.14
91.38:11.77
89.5110.88
HalfCheetah
51.00-+0.69
52.12-0076
55.75
Medium Replay
Hopper
101.3
103.67=1.81
107.69
106.67+t0.99
Walker2d
88.1
104.10
105.42-1.97
100.3112.11
HalfCheetah
76.4
83.18
86.55=00.44
90.05
Medium Expert
Hopper
113.0
113.01=0.71
113.34=0.09
113.23+0.22
Walker2d
103.3
117:26=1.77
114.68=2.18
111.88+0.28
HalfCheetah
94.4
91.5411.04
93.46:10.54
93.48.10.11
Expert
Hopper
112.8
113.02-10.17
113.18
112.86:10.10
Walker2d
110.4
117.92=2.07
112.55=0.56
111.224:0.5
Table 7: Average normalized scores of IQL across dataset sizes and qualities.
w/o pretrain
w/ pretrain, 10%
w/ pretrain, 30%
w/ pretrain, full
HalfCheetah
10.3
6.92-40.63
12.65:42..53
18. 18.28+11.02
Random
Hopper
9.4
8.17::0.54
9.93-11.19
10.67=10.4
Walker2d
7.9
8.26++0.64
9.08++0096
8.88++0.71
HalfCheetah
46.6
46.51=:0.18
47.87=1..21
48.85=10.16
Medium
Hopper
76.9
75.72+3.23
80.76+33.51
78.6242222
Walker2d
83.8
82.62
83.89++1.99
83.63+
HalfCheetah
43.4
33.4911.22
41.16:10.50
45.48.10.17
Medium Replay
Hopper
96.2
80,59118.25
91.0811..67
99.43.11.71
Walker2d
77.9
39.08
75.33
87.9511.668
HalfCheetah
94.8
87,44+22.52
93.66++0.46
95.25 14
Medium Expert
Hopper
101.8
93.89-100.7
91.05118.78
105.77111.31
Walker2d
111.6
111.23
111.65
112.09
HalfCheetah
96.4
77.85+3.82
95.88=0.44
97.40+0.13
Expert
Hopper
113.1
109.16::3225
112.85
113.34
Walker2d
110.7
113.76=2.55
112.53#1.35
112.80+1.08
In addition depicting results on varying dataset qualities and sizes, we numerically compare the performance of baselines for a comprehensive view in Table 6llle Table 7
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we summarize our work and explain the goal of this research.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We describe the limitations of our work and potential future work to overcome those issues in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [No]
Justification: In Section 3 we support the implication on how pretraining the shared Q- network with the transition model prediction task affects the convergence of Q-value via linear function approximation and the projected Bellman equation. However, we do not provide any theorems, assumptions, or proofs for addressing rigorous connection between prior theorems and our method.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We explain experiment and implementation details in Appendix Didnd C We further provide a pseudo code for outlining the two-phase learning strategy in Algorithm
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide reference webpages for implementing baselines in Appendix C Additionally, describe training details with a desirable computing resource.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guides httpai///iide///ooeuuuiiisssssss oolliccyyly for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.c/ppie http:l/iigguidde///eeeeeeeeeeeee
semniisssoniiii ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: In Section Appendix D and Appendix C we explain the experimental setup and training details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the statistical significance via the mean and standard deviation over few random seeds in every experimental result-figures and tables.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix denote the computing resource used for training and evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: All authors thoroughly confirm the NeurlPS Code of Ethics, and there are no violations in our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
26

---

Justification: We present shared Q-network architecture via two-phase learning strategy for data-efficieet offline RL. We conduct extensive experiments on diverse offline RL benchmarks, including D4RI., Rohomimic, and ExoRI.. We further suggest the theoretical justification for how our pretraining strategy improves the convergence of Q-value learning and data efficiency. However, we do not expect any negative societal impacts regarding our work, since our results mainly rely on simulated continuous control tasks.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not provide any checkpoints or source codes for reproducing the results in this paper. Since our method can be readily reimplemented top of any off-policy offline RL methods with a few lines of modification, we have decided to refrain from releasing an official source code.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
27

---

Justification:
We cite the previous works on relevant fields in the Reference. Additionally, we explicitly provide the original repository of each baseline in Appendix
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssii thcode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We provide pseudo code block in Section 3 We further provide an example code in https:/gtttuu.coommmaapppp/////s
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing research with human subjects. We only conduct extensive experiments on simulated locomotion and manipulation tasks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
28

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve crowdsourcing research with human sub- jects. We only conducted extensive experiments on MuJoCo locomotion and Robomimic manipulation simulation tasks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Our work is not related LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non stn-standard components.
Please refer to our LLM policy pattps://neurre cc/Conferencees22225 /LLM) for what should or should not be described.
29


Reviews:
[{"review_id": "pfqggXGz63", "reviewer": "Reviewer_42nb", "summary": "This paper addresses the challenge of improving data efficiency in offline reinforcement learning (RL), where agents must learn from static datasets without further environment interaction. The authors propose a simple, plug-and-play pretraining method that initializes a shared Q-network to predict both the next state and Q-value, enhancing performance across various offline RL algorithms. Empirical results on D4RL, Robomimic, V-D4RL, and ExoRL benchmarks show that the method significantly improves performance, even outperforming standard approaches using only 10% of the dataset.", "questions": "see weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1\tThe idea of pre-training with next-state prediction is interesting and intuitively sound.\n2\tThe proposed method significantly improves upon existing offline reinforcement learning approaches.\n3\tThe paper provides an analysis based on the projected Bellman equation, which adds theoretical depth to the work.\n\nWeaknesses:\n1\tThe use of next-state prediction as a pre-training task is not novel, and the paper currently lacks important comparisons and citations. For example, prior works [1, 2] have also employed next-state prediction for pre-training or co-training policy networks. It is recommended that the authors discuss these related studies to better position their contribution. [1] Wu, Hongtao, et al. \"Unleashing large-scale video generative pre-training for visual robot manipulation.\" arXiv preprint arXiv:2312.13139 (2023). [2] Radosavovic, Ilija, et al. \"Humanoid locomotion as next token prediction.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024).\n2\tThe paper does not include real-world experiments, which limits the evaluation of its practical applicability.\n3\tIt is not clear what is the datasets that are used for pre-training.", "was_revised": true, "final_justification": "Thanks to the authors for their response and clarifications; will retain my score leaning towards acceptance.", "rebuttal": "## **Weakness 1:** \n>Reviewer Comment:\nLack of related comparisons and citations. For example, prior works [1, 2] have also employed next-state prediction for pre-training or co-training policy networks. It is recommended that the authors discuss these related studies to better position their contribution.\n\n**Our Response:**\n\nWe appreciate the reviewer’s thorough observation. It is true that our method employs a next-state prediction task for pretraining the shared network for data-efficient offline RL, which resembles how suggested works train their network. We agree that referencing such works will strengthen the related work section and better position our contribution in the broader landscape of the next state prediction task for pretraining.\n\nHowever, we would be grateful if we could clarify a potential misunderstanding regarding our method for pretraining the shared network with the next state prediction task. We introduce **a two-stage strategy for data-efficient offline RL** training by:\n- **Pretraining the shared network with the forward dynamics (next state) prediction** task\n- Training the weight-initialized, pretrained Q network with **the underlying RL training** task\n\nOn the contrary, suggested references pretrain the generative large Transformer with the next token prediction task via an autoregressive reconstruction objective. As we discuss in Appendix F, involving auxiliary representation learning in policy learning is non-trivial since effective representation learning strategy often requires **domain and modality-specific techniques **(e.g., data augmentation, Transformer architecture). On the other hand, our method empirically demonstrates performance gains using **only 10% of the original dataset for both pretraining and RL, outperforming baselines trained on full datasets**.\n\nWe will revise the manuscript to reinforce the contribution of our method by offering a broader, deeper discussion on comparing with the next token prediction task.\n\n---\n\n## **Weakness 2:** \n>Reviewer Comment:\nThe paper does not include real-world experiments, which limits the evaluation of its practical applicability.\n\n**Our Response:**\n\nWe appreciate the reviewer’s attention to expanding the applicability of our method into the real world. We select **D4RL, Robomimic, V-D4RL, and ExoRL** benchmarks–which offer **high-fidelity proxies for real-world dynamics** in the simulated environment– for the performance comparison across popular offline RL baselines:\n- D4RL: Most popular benchmark in offline RL.\n- Robomimic: Data from diverse robot executions in manipulation tasks.\n- V-D4RL: High-dimensional visual input that expands the applicability.\n- ExoRL: Non-i.i.d. collection strategies that mimic real-world variability.\n\nAcross these domains, our method consistently:\n- Improves **data efficiency** (1–10% data usage)\n- **Outperforms baseline methods trained on full datasets**\n\nThese findings position the potential practicality of our method under real-world constraints. Subsequently, we believe that future work could explore diverse real-world RL tasks by pretraining the Q network with our method to enable data-efficient offline RL.\n\nWe will strengthen our Limitations section to explicitly discuss transfer to physical platforms and outline planned future experiments on real robots.\n\n---\n\n## **Weakness 3:** \n>Reviewer Comment:\n It is not clear what is the datasets that are used for pre-training.\n\n**Our Response:**\n\nWe thank the reviewer for the thoughtful comment. We agree that the current manuscript could confuse the reader about the sort of datasets used for pretraining.\nIn every experiment, we **pretrain the shared Q-network on the same static dataset** that the offline RL algorithm later consumes.\n**No external data or additional transitions** are introduced at any stage.\nThis design is central to our core claim of true **data efficiency**: by leveraging **simple** supervised next-state prediction, we extract more value from existing data **without expanding the dataset or raising computational requirements**.\n\nWe will explicitly state in the Methodology section that pretraining and RL training share an identical dataset in the revised paper. Additionally, we will highlight this point in the Experimental Setup, emphasizing its importance for resource-constrained applications.\n\n---\n### **Reference**\n[1] Wu, Hongtao, et al. \"Unleashing large-scale video generative pre-training for visual robot manipulation.\" arXiv preprint arXiv:2312.13139 (2023).\n\n[2] Radosavovic, Ilija, et al. \"Humanoid locomotion as next token prediction.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)."}, {"review_id": "wVx5Iz7yBA", "reviewer": "Reviewer_9vaj", "summary": "This paper proposes a pertaining objective for offline RL, specifically a forward dynamics surrogate loss that pretrains the critic network before TD learning. The pretraining objective mitigates loss of expressivity (shown through rank reduction in the network outputs) in the critic network cause by using TD learning from scratch. Experiments show that this pretraining objective is useful in improving the sample efficiency of existing offline RL algorithms across a range of domains. The authors highlight the their method trained with 10-50% of the dataset can match baseline offline RL algorithms trained on the whole dataset, validating their claims.", "questions": "From weaknesses\n- Comparing with existing pretraining objectives\n- Comparing on more image based tasks\n\n\nGeneral Suggestions and Questions\n- They mention that they don't reconstruct the full image but instead the extracted features from a pretrained encoder. Is this encoder fine-tuned during the RL phase or is it always frozen?\n- It would be helpful to aggregate performance increases (average % increase perhaps) per algorithm/task in Table 1 to provide a quick summary", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The authors perform extensive experiments in state-based environments to validate their claims. They compare their surrogate objective on a range of offline RL benchmarks, showing that their approach is generally applicable. \n- The experiments show strong performance uplifts across tasks.\n- The experiments on comparing performance across data scales provides strong evidence supporting their claim of improved sample efficiency. \n- Comparing performance under different data distributions highlights the robustness of the objective. \n- The method is simple to implement, aiding in the broader adoption of their approach\n- The curves showing the difference in the effective rank of the network is a good addition to the paper.  \n\n\nWeakness\n- Lack of comparison with existing pretraining objectives: The authors do not compare with SGI [1], or ATC [2]. While they have shown that their approach is beneficial, it would be good to have a point of reference with existing literature. There is an extended discussion of related works in the appendix which I appreciate, but I do think actual experimental comparisons on some environments/tasks would be helpful. Also I disagree that SGI is an online method, since SGI has experiments with offline datasets in Atari. This is the most importance weakness that I wish the authors would address for changing my rating.\n- Lack of experiments in the image domain. Observation reconstruction is much harder in image based environments compared to state based environments like D4RL. While the authors have compared two envs from V-D4RL, I would like to see at least a few more experiments in image based tasks (from V-D4RL or visual OGBench). The data scaling experiments on image based tasks would be useful too.\n- The authors use the phrase \"our method guarantees better performance\" in Fig 7 and in the text below which is too strong I feel given that they have provided mostly empirical evidence. I would suggest they tone down that language\n- Clarity in writing: The paper is a bit hard to follow at times with odd phrasing and repeated language. \n\n[1] Schwarzer, Max, et al. \"Pretraining representations for data-efficient reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 12686-12699.\n[2] Stooke, Adam, et al. \"Decoupling representation learning from reinforcement learning.\" International conference on machine learning. PMLR, 2021.", "was_revised": true, "final_justification": "The authors have resolved my major concerns with additional experiments and explanations. The broad applicability of the method and easy integration make it relevant to the research community.", "rebuttal": "## **Weakness 1:**\n>Reviewer Comment:\nThe authors do not compare with SGI [1], or ATC [2]. While they have shown that their approach is beneficial, it would be good to have a point of reference with existing literature.\n\n## **Question 1:**\n>Reviewer Comment:\nComparing with existing pretraining objectives\n\n**Our Response:**\n\nWe appreciate the reviewer’s feedback for pointing out the connection between our work and existing literature with a pretraining objective in RL. We agree that the current manuscript lacks comprehensive explanations on comparing pretraining objectives in reinforcement learning, especially for offline RL. Before we outline our plans to revise the manuscript, we would like to clarify the differences between our method and the suggested pretraining objectives, which highlight the unique novelty of our approach.\n\n> Comparison with existing pretraining objective\n\nPretraining strategy has been actively investigated in the reinforcement learning field to boost the performance of an underlying policy. The target of pretraining is often the visual feature extractor — the encoder — that extracts a latent vector from high-dimensional sensory input. SGI [1] and ATC [2] are representative methods of how the pretraining objective enables efficient downstream vision-based RL training. However, as we discuss in Appendix F, we clarify that the novelty of our method is based on its simplicity and adaptability without enforcing any modification on the base RL algorithm nor any constraint on problem formulation.\n\nTypical pretraining method for visual RL involves an unsupervised learning task (e.g., contrastive learning) that relies on a synthetic technique for generating a learning signal. For instance, ATC employs random-shift data augmentation to obtain positive and negative samples for computing InfoNCS loss, which has been proven effective in improving sample efficiency in visual RL. However, this domain-specific data augmentation technique may hinder the broader applicability of an RL algorithm by limiting the domain of the algorithm can solve. On the other hand, we demonstrate our method’s generalizability to diverse problem formulations through extensive empirical experiments, including offline, online, and visual RL. Additionally, we observe that our method improves the data efficiency of offline RL across data collection strategies and constrained distribution of state space.\n\nWe will reinforce the novelty of our method in the revised paper as follows:\n\n* **We will reconstruct the related work section in the main paper** to clearly distinguish our method from self-predictive pretraining methods.\n* **We will expand the discussion about related work in Appendix F**, which offers a broader and comprehensive understanding of pretraining objectives in diverse RL formulations. \n\nWe believe that these modifications will reposition the contribution of our method as an effective pretraining strategy for data efficiency.\n\n> Our argument on SGI is an online method\n\nWe thank the reviewer for thorough feedback. It is indeed true that **SGI is not limited to a method that can only be applied to an online RL setting**. As the reviewer notes, SGI validates its strong performance by considering diverse sources of offline data to pretrain the encoder in the Atari-100k benchmark. To explain this misidentification, SPR is a self-predictive pretraining method for online RL, as we discuss in Appendix F, and we have confused SPR with SGI at the time of writing the manuscript. We agree that the manuscript delivers a misunderstanding of SGI and degrades the reliability of our method.\n\nWe will fix this incorrectness in the introduction and related work section in the revised paper.\n\n**Reference**\nSchwarzer, Max, et al. \"Data-efficient reinforcement learning with self-predictive representations.\" arXiv preprint arXiv:2007.05929 (2020).\n\n\n---\n## **Weakness 2:**\n>Reviewer Comment:\n Lack of experiments in the image domain. Observation reconstruction is much harder in image based environments compared to state based environments like D4RL. While the authors have compared two envs from V-D4RL, I would like to see at least a few more experiments in image based tasks (from V-D4RL or visual OGBench). The data scaling experiments on image based tasks would be useful too.\n## **Question 2:** \n>Reviewer Comment:\nComparing on more image based tasks\n\n**Our Response:**\nWe thank the reviewer for suggesting thorough comments. We agree that the current results in the image-based environment may visualize a limited context of which our method can improve the base visual RL algorithm across tasks.\n\nTo address the reviewer’s concern about the image-based experiment, we conduct additional experiments on the humanoid_walk task across data quality. Additionally, we consider an extra dataset quality, medium_replay, during the additional experiment for a broader understanding of the effect of the policy optimality. The episodic return is averaged over three random seeds for each data quality in the table below.\n\n### **Extra image-based tasks performance**\n| Environment     | Data           | DrQ-v2             | DrQ-v2 + Ours       |\n|-----------------|----------------|--------------------|---------------------|\n| walker_walk     | medium          | 306.93 ± 28.21     | **338.77 ± 29.55**  |\n| | medium_replay   | **30.09 ± 0.75**       | 28.68 ± 2.29    |\n| | medium_expert   | 352.46 ± 37.15     | **369.66 ± 20.86**  |\n| cheetah_run     | medium          | 340.33 ± 7.55      | **379.80 ± 45.83**  |\n| | medium_replay   | 21.15 ± 2.04       | **25.13 ± 2.04**    |\n| | medium_expert   | 251.52 ± 34.37     | **258.76 ± 50.33**  |\n| humanoid_walk   | medium          | 12.57 ± 6.73       | **20.03 ± 3.80**    |\n| | medium_replay   | **40.76 ± 16.27**      | 19.38 ± 6.10    |\n| | medium_expert   | 4.11 ± 2.72        | **5.12 ± 1.89**     |\n||||\n\nIn summary, our method indicates improved performance in seven out of nine cases across three tasks and three data qualities. We believe these additional results support the applicability of our method to the image-based domain.\n\nWe will incorporate these empirical observations into the revised paper to make the novelty of our method more concrete.\n\n---\n## **Weakness 3:** \n>Reviewer Comment:\nThe authors use the phrase \"our method guarantees better performance\" in Fig 7 and in the text below which is too strong I feel given that they have provided mostly empirical evidence. I would suggest they tone down that language.\n\n\n**Our Response:**\nWe appreciate the reviewer on suggesting meticulous feedback. In Figure 7, we conduct data scale experiments to verify data efficiency of our method across various scales. We acknowledge that the current manuscript would deliver unintentionally strong tone of our argument.\nWe will revise the expression “From the overall results, we conclude that our method guarantees better performance” to ”The empirical observations support the data efficiency of our method”.\n\n---\n\n## **Weakness 4:**\n>Reviewer Comment:\nThe paper is a bit hard to follow at times with odd phrasing and repeated language.\n\n**Our Response:**\nWe thank the review for their thorough comment. We apologize for some inconvenient writings. We will revise the overall manuscript to provide clear, coherent idea on our method.\n\n---\n\n## **Question 3:**\n>Reviewer Comment:\nThey mention that they don't reconstruct the full image but instead the extracted features from a pretrained encoder. Is this encoder fine-tuned during the RL phase or is it always frozen?\n\n\n**Our Response:**\nWe appreciate the reviewer on insightful question. Our method is designed to be *plug-and-play*, meaning it can be integrated into existing RL algorithms without requiring architectural modifications. To preserve this modularity, we applied our approach by altering only the Q-function loss, without changing other components of the underlying algorithm.\n\nFor image-based tasks, we adopted **DrQ-v2** as the backbone RL algorithm. As described in the DrQ-v2 framework, the visual encoder is continusouly trained during the RL phase other than kept frozen. We follow this protocol and update the encoder during policy training. This design choice aligns with our goal of preserving compatibility with established pipelines while evaluating the effectiveness of our Q loss formulation.\n\n---\n## **Question 4:** \n>Reviewer Comment:\nIt would be helpful to aggregate performance increases (average % increase perhaps) per algorithm/task in Table 1 to provide a quick summary.\n\n\n**Our Response:**\nWe thank the reviewer on improving readability of the manusciprt. Through this rebuttal, we aggregate experimental results on two benchmarks, D4RL and V-D4RL. We calculate the average improvements using blow equation:\n\n> Improvement = ((before -after)/before) x 100 (%)\n\nIn below tables, we average the improvements over each environment and dataset.\n\n### **Average Improvement (%) by Task and Algorithm (D4RL)**\n|    | AWAC         | CQL          | IQL          | TD3+BC       |\n|---------------|--------------|--------------|--------------|--------------|\n| HalfCheetah   | +325.89% | +42.48%  | +15.26%  | +101.56% |\n| Hopper        | +22.13%  | +88.97%  | +4.61%  | +42.77%  |\n| Walker2d      | +28.67%  | +195.83% | +6.74%  | +36.85%  |\n| Overall Avg. | +140.37% | +138.89% | +9.72%   | +67.58%  |\n||||||\n\n### **Average Improvement (%) by Task (V-D4RL)**\n| Walker walk | Cheetah run | Humanoid walk |\n|-------------|-------------|----------------|\n| +3.52%   | +11.13%  | +10.49%     |\n||||\n\n---\n### **Reference**\n[1] Schwarzer, Max, et al. \"Pretraining representations for data-efficient reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 12686-12699.\n\n[2] Stooke, Adam, et al. \"Decoupling representation learning from reinforcement learning.\" International conference on machine learning. PMLR, 2021."}, {"review_id": "Hk1N9ccGTW", "reviewer": "Reviewer_KRdK", "summary": "The paper proposes a simple yet effective pretraining method for offline reinforcement learning (Offline RL) to address the data efficiency issue in offline RL. The method initializes the feature extraction of a Q-network by pretraining a shared Q-network, which outputs predictions of the next state and Q-value. Specifically, the shared Q-network consists of a shared deep neural network layer and two separate shallow output layers, one for predicting the next state (transition model) and the other for outputting the Q-value. During the pretraining phase, the shared network is trained using a supervised regression task to predict the transition model. After pretraining, the shared network is connected to the Q-network's shallow output layer and trained with existing offline RL value learning methods.", "questions": "1. Why did several reported scores of baselines significantly lower than that in widely-used implementations or original papers?\n2. Would axuiliary training of dynamic during the training of Q be better or worse than the method proposed in the main paper? It seems you directly add the dynamic loss without tuning the loss weight in Eq.6 in appendix. So it's hard to say which method is better.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The experiments are widely performed on different alogirthms and benchmarks.\n2. The method is easy to follow.\n\nWeakness\n1. The method seems a self-supervised RL (SSRL) method. Therefore, I believe the authors should at least discuss the relationships between the proposed method and existing SSRL  methods such as SAC+AE[1] and CURL[2]. Besides, in existing works such as [3] have discussed the representations effectiveness in offline RL. What's the contribution of the proposed method compared with these methods?\n2. Some baseline results significantly underperform compared to widely-used implementations. For instance, CORL(https://github.com/tinkoff-ai/CORL) 's CQL achieves over 90 normalized score on halfcheetah-medium-expert, yet Table 1 reports only 27.1. BCQ could achieve success rate of 91 and 75 on Lift(MG) and Can(MG) in the original Robomimic paper, but in Figure 5 the reported results are lower than 50. Such discrepancies raise questions about the proposed method's effectiveness.\n3. The writing should be significantly improved. In line 43, $g$ should refer to next state prediction and $f$ should refer to Q-value. But it seems the author write them in reverse. In line 214, 'our' should be 'Our'. In title 4.2 and 4.3, the capitalization of initial letters in words are inconsistent.\n\n[1] Improving sample efficiency in model-free reinforcement learning from image.\n\n[2] Representation learning with contrastive predictive coding.\n\n[3] Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learning.", "was_revised": true, "final_justification": "The authors addressed my concern. Although the idea of leveraging dynamic model's representation is not quite novel in online RL, applying it in offline setting and improving different offline rl algorithms could be a valuable contribution.", "rebuttal": "## **Weakness 1:**\n>Reviewer Comment:\nThe method seems a self-supervised RL (SSRL) method. Relationships between the method and SSRL (e.g., SAC+AE[1] or CURL[2])? Besides, in existing works such as [3] have discussed the representations effectiveness in offline RL. What's the contribution of the proposed method compared with these methods?\n\n**Our Response:**\n\nWe thank the reviewer for highlighting the connection between self-supervised learning methods and our approach. We agree that the current Related Work section could be expanded to provide a broader contextualization, which would enhance the clarity and reliability of our empirical findings.\n\nThat said, we would like to clarify a possible misunderstanding regarding the nature of our method. Specifically, our approach does not fall under self-supervised learning. In our offline pretraining phase, we use a supervised forward dynamics prediction task, where the model is trained to predict the next state given a state-action pair from the replay buffer. As we discuess in Appendix F, this is a regression task at the state level, in contrast to typical self-supervised approaches such as CURL, which focus on learning representations in the latent space via contrastive objectives.\n\nNonetheless, we fully acknowledge the importance of learning good representations. As discussed in Section 3, when the pretrained encoder is optimized to increase the rank of the feature matrix—an implicit form of learning structured representations—the resulting Q-function initialized from this encoder demonstrates significantly improved accuracy, leading to stronger downstream performance. We believe that integrating a more advanced representation learning strategy could further enhance our method’s effectiveness across diverse environments.\n\nWe will revise the related work section to reflect clear distinctions from self-supervised learning and discuss more details on how our method differentiates with the conventional self-supervised learning.\n\n---\n## **Weakness 2:**\n>Reviewer Comment:\nSome baseline results significantly underperform compared to widely-used implementations. For instance, CORL(https://github.com/tinkoff-ai/CORL) 's CQL achieves over 90 normalized score on halfcheetah-medium-expert, yet Table 1 reports only 27.1. BCQ could achieve success rate of 91 and 75 on Lift(MG) and Can(MG) in the original Robomimic paper, but in Figure 5 the reported results are lower than 50. Such discrepancies raise questions about the proposed method's effectiveness.\n\n## **Question 1:**\n>Reviewer Comment:\nWhy did several reported scores of baselines lower than other implementations?\n\n**Our Response:**\n\nWe appreciate the reviewer’s attention to the reliability and consistency of our experimental results. We acknowledge that some of the reported scores may differ from those available in public repositories.\n\nEnsuring fairness and reproducibility has been a top priority in conducting our experiments across various baselines. To this end, we built our method on top of widely used offline RL algorithms that offer official, author-maintained implementations. Below, we provide a detailed analysis of why some reproduced scores differ from those reported in the original implementations:\n\n> CORL’s CQL Performance\n\nTo validate the reason for lower score, we first evaluated our method based on the CORL implementation of CQL, running experiments with three random seeds per task and comparing the resulting D4RL scores to those reported in the original CORL paper. As shown in the table below, our method consistently outperforms the baseline, validating the effectiveness of our method across different implementations.\n\n**CORL implementation: original CQL vs CQL + Ours**\n\n| **Task** | **Implementation** | **Medium** | **Medium Replay** | **Medium Expert** |\n| --- | --- | --- | --- | --- |\n| halfcheetah | original CQL | 47.04 ± 0.22 | 45.04 ± 0.27 | 95.63 ± 0.42 |\n| | CQL + Ours | **47.28 ± 0.30** | **46.10 ± 0.07** | **96.06 ± 0.08** |\n| hopper | original CQL | 59.08 ± 3.77 | 95.11 ± 5.27 | 99.26 ± 10.91 |\n| | CQL + Ours | **72.10 ± 11.55** | **101.49 ± 0.11** | **109.00 ± 1.14** |\n| walker2d | original CQL | 80.75 ± 3.28 | 73.09 ± 13.22 | 109.56 ± 0.39 |\n| | CQL + Ours | **83.83 ± 0.61** | **86.08 ± 0.76** | **110.99 ± 0.61** |\n\nAdditionally, we found that our implementation differs from CORL's in terms of hyperparameters in **the table below**. For example, the CORL implementation uses a deeper network, weight initialization, and a policy learning rate of 3e-5, whereas our version follows a commonly used setting in deep RL, including a policy learning rate of 3e-4.\n\n**Comparion on the hyperparameters across implementations**\n\n| **Hyper Parameter** | **Impl. in CORL** | **Impl. in Our Paper** |\n| --- | --- | --- |\n| Policy Learning Rate | 3e-5 | 3e-4 |\n| CQL Alpha | 10.0 | 5.0 |\n| Orthogonal Init | O | X |\n| Hidden Layers | 3 | 2 |\n\n> Robomimic’s BCQ Performance\n\nAccording to Appendix G of the Robomimic paper (Mandlekar et al., 2021), policy performance can vary significantly depending on factors such as input modality (state vs. image), dataset quality (machine-generated vs. human-demonstrated), and task difficulty (easy vs. hard). As further shown in Table 2 of Luo et al. (2021), the originally reported scores for CQL and BCQ are often significantly higher than their reproduced counterparts.\n\nWe recognize this as an important issue to address in future work. To mitigate potential confusion, we will clearly indicate which scores come from original papers and which are our reproduced results in the revised paper.\n\n**Reference**\n\nLuo, Jianlan, et al. \"Action-quantized offline reinforcement learning for robotic skill learning.\" Conference on Robot Learning. PMLR, 2023.\n\nMandlekar, Ajay, et al. \"What matters in learning from offline human demonstrations for robot manipulation.\" arXiv preprint arXiv:2108.03298 (2021).\n\n---\n## **Weakness 3:**\n>Reviewer Comment:\nThe writing should be significantly improved. In line 43,  should refer to next state prediction and  should refer to Q-value. But it seems the author write them in reverse. In line 214, 'our' should be 'Our'. In title 4.2 and 4.3, the capitalization of initial letters in words are inconsistent.\n\n**Our Response:**\n\nWe appreciate the reviewer’s suggestions for improving readability. As pointed out, Line 43 lists the outputs of the shallow layers in reverse order. We thank the reviewer for catching this detail in the shared network architecture and will revise the manuscript to prevent such potential misunderstandings.\n\n> Regarding capitalization\n\nWe also appreciate the reviewer’s attention to grammatical consistency. We acknowledge the presence of inconsistent expressions and capitalization in the current draft and will revise the manuscript to ensure grammatical correctness and a more professional tone throughout.\n\n---\n## **Question 2:**\n>Reviewer Comment:\nWould auxiliary training of dynamics during the training of Q be better or worse than the method proposed in the main paper? It seems you directly add the dynamic loss without tuning the loss weight in Eq.6 in the appendix. So it's hard to say which method is better.\n\n**Our Response:**\n\nWe appreciate the reviewer’s insightful suggestion regarding the integration of the pretraining objective into downstream RL training. We agree that incorporating the forward dynamics prediction task into the learning process could offer additional potential benefits for downstream offline policy learning.\n\nTo support this discussion, we present a detailed comparison between our method and a variant that incorporates the forward dynamics prediction as a regularization term during training, as described in Eq. 6:\n\n**Comparison Between Our Method and the Eq. 6 Variant**\n\n| **Data** | **HalfCheetah w/ Eq.6** | **HalfCheetah Ours** | **Hopper w/ Eq.6** | **Hopper Ours** | **Walker2d w/ Eq.6** | **Walker2d Ours** |\n| --- | --- | --- | --- | --- | --- | --- |\n| Random | 11.45 ± 0.51 | **14.8 ± 0.5** | 31.54 ± 0.42 | **31.6 ± 0.2** | **13.46 ±  6.58** | 11.2 ± 5.1 |\n| Medium | 48.23 ± 0.33 | **49.2 ± 0.3** | 70.86 ± 2.17 | **71.5 ± 2.2** | 82.65 ± 1.65 | **87.1 ± 0.6** |\n| Medium Replay | 44.93 ± 0.29 | **45.8 ± 0.3** | 90.39 ± 7.34 | **100.2 ± 1.6** | 86.11 ± 1.54 | **92.0 ± 1.6** |\n| Medium Expert | 93.55 ± 1.00 | **96.9 ± 0.9** | **113.44 ± 0.35** | 113.0 ± 0.2 | **111.88 ± 0.63** | 111.6 ± 0.4 |\n| Expert | 96.59 ± 0.25 | **98.9 ± 0.6** | 113.28 ± 0.20 | **113.4 ± 0.3** | 110.98 ± 0.22 | **111.0 ± 0.2** |\n\nThese results demonstrate that our method achieves superior performance in 12 out of 15 settings, supporting our hypothesis that pretraining the shared network with a supervised regression task significantly improves downstream policy learning.\n\nIn addition, our method is more computationally efficient than the Eq. 6 variant. While Eq. 6 requires computing both the TD loss and the forward dynamics loss throughout the entire training process, our method computes the forward dynamics loss only during the pretraining phase.\n\nWe agree that adaptively tuning the strength of the forward dynamics loss in the combined objective (i.e., Eq. 6) may further enhance performance. However, we would like to emphasize that our key empirical insights are derived from the design choice to pretrain the shared network via the forward dynamics prediction task, other than jointly optimizing it during policy training. Hence, this line of investigation lies outside the scope of this work.\n\nWe will revise the manuscript to clearly highlight the improvements of our method over the Eq. 6 baseline and better articulate this distinction."}, {"review_id": "yF7goDKW3p", "reviewer": "Reviewer_XMkN", "summary": "This paper proposes to pre-train the representation of a Q-network by learning a transition model before running model-free offline RL.\nThe authors show that this representation pre-training improves the performance of various model-free offline RL algorithms across multiple benchmarks, including Robomimic, D4RL, and V-D4RL. \nThe authors also provide a preliminary analysis showing that the latent space learned through pre-training has a higher rank, which may contribute to better Q-learning.", "questions": "**Q1. Q-error for Figure  3**\n* Would it be possible to include the plot of Q-error to support the claim?  \n* Alternatively, if the rank itself is the more critical factor, it would * be helpful to clarify why it serves as a meaningful proxy for Q-error in this context.\n\n**Q2. Clarification on Figure 8** \n* Could you include the results of model-free baselines (e.g., TD3+BC, AWAC) in Figure 8? \n* This would help clarify whether the observed gains in sample efficiency stem from the proposed pre-training method or simply from the underlying model-free algorithms.\n\n**Q3. Details on the experiment of Section 3.1**\n* Could you elaborate on how the rank of the latent space is computed? \n* Are you following a methodology used in prior works (e.g., effective rank used in [4])?\n* Providing more detail would help to understand the results and make it reproducible.\n\n[4] Aviral Kumar et al., DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization, ICLR 202", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n**S1. Strong Empirical Results**\n* The proposed method consistently improves the performance across a range of offline RL algorithms and benchmarks. \n* Notably, it also demonstrates strong gains on complex tasks, such as Robomimic and pixel-based V-D4RL. \n* In addition, the method significantly improves sample efficiency; on the 1% D4RL datasets, the method outperforms prior model-based approaches by 20%–40%.\n\n**S2. Simplicity**\n* The approach is simple and modular---it can be easily plugged into existing offline RL pipelines. \n* Moreover, it does not introduce significant computational overhead during training nor inference.\n\n**S3. Writing**\n* The paper is easy to read.\n\n&nbsp;\n\n**Weaknesses**\n\n**W1. Novelty**\n* The idea of learning representations through model-based pre-training have been actively explored (e.g., [1], [2], [3]).\n* However, it is also true that its application in the context of offline RL remains underexplored. \n\n**W2. About the analysis in Section 3.1**\n* The analysis claims that to reduce the Q-error, the column space of the feature vectors and the Q-values should be closer, so having higher rank on the feature vector is important.\n* Why do we need to indirectly illustrate the rank of the feature vectors, instead of directly showing the Q-error itself? \n* Please correct me if I’m wrong for this point.\n\n[1] Max Schwarzer et al., Pretraining Representations for Data-Efficient Reinforcement Learning, NeurIPS 2021\n[2] Zichen Jeff Cui et al., In-Domain Dynamics Pretraining for Visuo-Motor Control, NeurIPS 2025\n[3] Quentin Garrido et al., Learning and Leveraging World Models in Visual Representation Learning, Arxiv 2024.", "was_revised": true, "final_justification": "The paper introduces a modular approach, which pretraining a representation with world model training.\nThe concern on the discrepancy between the rank analysis and the actual Q-error was resolved. \n\nI think this will impact offline RL (especially offline MBRL) areas on understanding how world model can benefit policy learning and Q-learning.", "rebuttal": "## **Weakness 1:** \n>Reviewer Comment:\nPretraining representations w/ model has been explored ([1,2,3]),  but it is also true that its application in the context of offline RL is underexplored.\n\n**Our Response:**\n\nWe are grateful to the reviewer for pointing out the novelty of our method and the analysis in Section 3.1. We agree that the model-based self-predictive pretraining strategy has been widely investigated under diverse domains. However, we would like to distinguish our method from the prior approaches, conceptually and technically.\n\nIn SGI, DynaMo, and I-JEPA, the encoder is pretrained with self-supervised learning for downstream policy learning. **SGI** and **I-JEPA** require an auxiliary **domain-specific data augmentation** to generate the synthetic self-predictive target for representation learning, whereas **DynaMo** only requires **raw sequential observations** for training forward and inverse dynamics in embedding space. Additionally, the **Transformer architecture** is enforced for the encoder and policy head to consume the sequential data structure for DynaMo.\n\nIn contrast, **our method pretrains the shared network via a forward dynamics prediction** task, which is a supervised regression task that does **not necessitate domain-specific augmentation or network architecture**.\n\nThis **plug-and-play ad-hoc architecture** exhibits several advantages:\n- It boosts the practical adoption of **many offline RL algorithms**, improving the reusability **without major modifications**.\n- It **avoids to necessitate of domain-specific techniques** for improving the underlying performance.\n\nFurthermore, as the reviewer rightly notes, our empirical evaluation spans several offline RL benchmarks (D4RL, V-D4RL, Robomimic, ExoRL). Beyond coverage, however, our primary contribution lies in demonstrating **strong performance in data-constrained settings**.\n\nWe introduce a **simple yet effective architecture built on a shared Q-network pre-trained via forward dynamics prediction**. Across all experiments, our method consistently surpasses baselines—even when **trained with only 10% of the data**, demonstrating **strong data-efficiency in offline RL**. To support this, we offer **theoretical justification via the Projected Bellman Equation**, which helps explain why our approach facilitates more accurate Q-value estimation.\n\nWe will revise the manuscript to clearly highlight these distinctions and contributions of our method, positioning the unique novelty of domain-agnostic applicability and strong data efficiency of our pretraining method compared to model-based pretraining strategies.\n\n---\n\n## **Weakness 2:** \n>Reviewer Comment:\nThe analysis claims that to reduce the Q-error, the column space of the feature vectors and the Q-values should be closer, so having higher rank on the feature vector is important.\nWhy do we need to indirectly illustrate the rank of the feature vectors, instead of directly showing the Q-error itself?\n\n## **Question 1:**\n>Reviewer Comment:\nWould it be possible to include the plot of Q-error to support the claim? Alternatively, if the rank itself is the more critical factor, it would * be helpful to clarify why it serves as a meaningful proxy for Q-error in this context.\n\n**Our Response:**\n\nWe appreciate the reviewer’s attention to connecting our theoretical justification with empirical observations. We recognize that the current manuscript could deliver a questionable insight on understanding the relation between the rank of the feature matrix and the Q error.\n\nIn fact, it is impossible to compute the true Q error through the experiment since the Q value is approximated with the neural network, and the state and action spaces are continuous.\n\nHence, per the reviewer’s suggestion, we now include **proxy Q-error comparisons** in the **Table 1.** below. We define a proxy Q error using **the left-hand side of Equation 5** in the manuscript ($||Q_{\\varphi,\\theta} - Q^\\pi||_\\infty$). We compare the proxy Q-error of **TD3+BC (10%)** and **TD3+BC + Ours (10%)**, where the percentile stands for the amount of dataset used for training:\n- **TD3+BC (10%)**: trained on a **10% subset** of the full dataset\n- **TD3+BC + Ours (10%)**: pretrained/trained on a **10% subset** of the full dataset\n\nTo compute **the optimal Q value**, $Q^\\pi$ in Equation 5, we use the Q network of **TD3+BC (100%)** for each Q error, which is **trained on the full dataset**. Since the true Q value is not available in D4RL tasks, **we consider the fully trained Q network as the best optimal Q value**.\n\nNotably, **our method achieves the lowest Q-error over all tasks**. These results indicate that our approach **learns a more accurate Q-function**, enabling truly **data-efficient** offline RL.\n\nWe will revise the paper to concretely support our theoretical justification by providing the rank of the matrix and the proxy Q error in Figure 3.\n\n\n**Table 1. Proxy Q-error**\n| Task                          | TD3+BC (10%) | Ours (10%) |\n|------------------------------|------------|-------------|\n| halfcheetah-medium-v2        | 503.6851   | **287.7740**  |\n| halfcheetah-medium-replay-v2 | 1032.6671  | **119.9188**  |\n| halfcheetah-medium-expert-v2 | 522.4211   | **112.2337**  |\n| hopper-medium-v2             | 536.2078   | **472.9333**  |\n| hopper-medium-replay-v2      | 320.1790   | **319.9531**  |\n| hopper-medium-expert-v2      | 437.8861   | **254.9155**  |\n| walker2d-medium-v2           | 299.1773   | **138.0320**  |\n| walker2d-medium-replay-v2    | 379.4456   | **49.9275**   |\n| walker2d-medium-expert-v2    | 392.7360   | **148.2040**  |\n||||\n\n---\n\n## **Question 2:** \n>Reviewer Comment:\nCould you include the results of model-free baselines (e.g., TD3+BC, AWAC) in Figure 8?  This would help clarify whether the observed gains in sample efficiency stem from the proposed pre-training method or simply from the underlying model-free algorithms.\n\n\n**Our Response:**\n\nWe thank the reviewer for constructive feedback. We will add model-free baselines (e.g., TD3+BC, AWAC) to Figure 8 in the revised manuscript.\n\n---\n\n## **Question 3:** \n>Reviewer Comment:\nCould you elaborate on how the rank of the latent space is computed? Are you following a methodology used in prior works (e.g., effective rank used in [4])? Providing more detail would help to understand the results and make it reproducible.\n\n\n**Our Response:**\n\nWe appreciate the reviewer’s attention to contributing reproducible research. Following the protocol in DR3 [4], we compute the rank of the feature matrix $h(s,a)$ by **feeding batches of state–action pairs to the hidden layers** of the Q network. Unlike DR3, which reports soft rank (s-rank), we calculate the **hard rank** of the feature matrix.\n\nSince we use 256 hidden units and a batch size of 256, the full rank of the resulting feature matrix is 256. We will clarify this distinction in Section 3.1 and include implementation details in the Appendix for reproducibility.\n\n---\n### **Reference**\n[1] Max Schwarzer et al., Pretraining Representations for Data-Efficient Reinforcement Learning, NeurIPS 2021 \n\n[2] Zichen Jeff Cui et al., In-Domain Dynamics Pretraining for Visuo-Motor Control, NeurIPS 2025 \n\n[3] Quentin Garrido et al., Learning and Leveraging World Models in Visual Representation Learning, Arxiv 2024.\n\n[4] Aviral Kumar et al., DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization, ICLR 2022"}]

Author rebuttal:

Meta-review: The paper got a pretty positive reception overall: reviewers agreed it’s a simple but effective idea, easy to plug into existing offline RL pipelines, and the empirical gains, especially under low-data regimes, are convincing. The main criticisms centered on novelty (next-state pretraining isn’t new), missing comparisons to prior pretraining objectives (SGI, ATC, SSRL methods), and some baseline discrepancies. Reviewers also noted limited real-world or image-based experiments and suggested the writing could be cleaned up. The authors’ rebuttal addressed most of these, adding new experiments, clarifying dataset use, and contextualizing novelty, which helped shift reviewer scores upward. In short, the work is meaningful, well-executed, and broadly useful, with clear room for polishing and positioning in relation to existing literature.
Decision: Accept (poster)