Action-Free Offline-To-Online RL via Discretised State Policies
Authors: Natinael Solomon Neggatu
Jeremie Houssineau
Giovanni Montana
Venue: ICLR 2026 Poster (2026)
Keywords: Action free Offline to Online Reinforcement Learning
Online Reinforcement Learning
Offline Reinforcement Learning
Primary area: reinforcement learning

Abstract: Most existing offline RL methods presume the availability of action labels within the dataset, but in many practical scenarios, actions may be missing due to privacy, storage, or sensor limitations. We formalise the setting of action-free offline-to-online RL, where agents must learn from datasets consisting solely of $(s,r,s')$ tuples and later leverage this knowledge during online interaction. To address this challenge, we propose learning state policies that recommend desirable next-state transitions rather than actions. Our contributions are twofold. First, we introduce a simple yet novel state discretisation transformation and propose Offline State-Only DecQN (OSO-DecQN), a value-based algorithm designed to pre-train state policies from action-free data. OSO-DecQN integrates the transformation to scale efficiently to high-dimensional problems while avoiding instability and overfitting associated with continuous state prediction. Second, we propose a novel mechanism for guided online learning that leverages these pre-trained state policies to accelerate the learning of online agents. Together, these components establish a scalable and practical framework for leveraging action-free datasets to accelerate online RL. Empirical results across diverse benchmarks demonstrate that our approach improves convergence speed and asymptotic performance, while analyses reveal that discretisation and regularisation are critical to its effectiveness.

Published as a conference paper at ICLR 2026
ACTION-FREE OFFLINE-TO-ONLINE RL VIA DISCRE- TISED STATE POLICIES
Natinael Solomon Neggatu, Jeremie Houssineau, & Giovanni Montana"
(nati. i neggatu, g g.montanaa @ewa rwi ck ac. jeremie e houssi neau@ntu. edu.sg
ABSTRACT
Most existing offline RL methods presume the availability action labels within the dataset, but in many practical scenarios, actions may be missing due to privacy, storage, or sensor limitations. We formalise the setting of action-free offline-oo- online RI, where agents must learn from datasets consisting solely of (s, &') tuples and later leverage this knowledge during online interaction. To address this challenge, propose learning state policies that recommend desirable next- state transitions rather than actions. Our contributions are twofold. First, we introduce simple yet novel state discretisation transformation and propose Offline State-Only DecQN (OSO-DecQN), a value-basee algorithm designed to pre- pre-train train state policies from action-free data. OSO-DecQN integrates the transformation to scale efficiently to high-dimensional problems while avoiding instability and overfitting associated with continuous state prediction. Second, we propose a novel mechanism for guided online learning that leverages these pre-trained state policies accelerate the learning online agents. Together, these components establish a scalable and practical framework for leveraging action-free datasets to accelerate online RL. Empirical results across diverse benchmarks demonstrate that our approach improves convergence speed and asymptotic performance, while analyses reveal that discretisation and regularisation are critical its effectiveness.
INTRODUCTION
Offline reinforcement learning (RL) focuses on learning effective policies entirely from fixed dataset gathered by an unknown behaviour policy, avoiding the need for risky or costly online exploration. While most offline RL research assumes that all necessary training information is readily available in the dataset, in this work, we explore the scenario where action data is completely absent. Such missing or corrupted actions can occur due to privacy requirements. sensor failures, or simply because actions were never logged. For example, in healthcare settings (Kushida et al. 2012 Yu et al. [022 explicit treatment decisions may be excluded from patient records for legal or confidentiality reasons, leaving only patient states (symptoms, vitals, lab results). In finance (Liu et al. 2022 Loukides & Gkoulalas-Divanis 2012), the exact trades or order flows might be withheld to protect proprietary strategies, so only partial portfolio states are available. In robotics or industrial control, sensor logs are sometimes kept without recording the precise torque or command signals, either to reduce storage overhead or because that information is considered proprietary. In these scenarios, agents must learn from action-free datasets that contain only (s, rr,') tuples. This raises a fundamental question: can we learn effective policies and transfer them to online RL without ever observing offline actions? We call this setting action-free offline-to-onling RL and propose a scalable framework to address it by learning state policies over next-state transitions rather than actions.
Recent methods address parts of this problem but leave fundamental limitations: Earlier Q-value formulations over target states (Edwards al., 22200] Hepburn et al. 2024 require actions during training, preventing their application in action-free settings. Transformer-ban based state policies (Zhu al. 2023) attempt to remove this dependency, but they incur high computational costs and have
* Unniversity of Warwick
Nanyang Technological University
The Alan Turing Institute

---

Published as a conference paper at ICLR 2026
not demonstrated consistent improvements when guiding online RL across diverse environments and dataset qualities. Moreover, existing discretisation strategies (Seyde et al. 2022) have been designed for actions with bounded ranges, making them unsuitable for action-free datasets. None of these works has demonstrated a practical scalable approach that can (a) learn from action-free offline data, (b) scale to high-dimensional control problems and (c) be used for guided online learning to accelerate online performance across wide range of environments.
In this work, we structure our contribution into two complementary stages. First, we present simple yet effective strategy for pre-training offline state policies; then we propose a pipeline that leverages these policies for guided online learning that addresses all three criteria.
For our pre-training strategy, we introduce novel state discretisation transformation and offline RL algorithm, Offline State-Only DecQN (OSO-DecQN), designed to learn state policies that predict discretised state differences from action-free datasets. The algorithm integrates our discretisation mechanism with conservative regularisation adapted to the discrete state domain. This design avoids the instability of continuous regression, mitigates overestimation bias, and constrains predictions to reachable states. Unlike Seyde et al. (2022). our discretisation method is inherently scale-invariant, facilitating learning across state dimensions with varying numerical ranges.
This leads our second contribution, where we develop a novel mechanism that leverages the pre-trained state policy to accelerate online RL. This mechanism introduces:
An online-trained inverse dynamics model (IDM) to translate predicted state differences (As) into executable actions, trained from scratch and learned jointly with the online agent.
A policy-switching strategy that blends offline- offline-guided actions with the online agent's own policy, enabling smooth integration of offline knowledge without destabilising learning.
Through extensive experiments across diverse range of tasks and datasets of varying quality, show that our approach can effectively guide exploration online, improving the performance online algorithms beyond existing action-free methods wide range of tasks, including environments with up to 78 state dimensions. To our knowledge, no prior work has combined an action-frec free offline state policy learner with mechanism guide online learning to produce consistent gains across diverse control environments.
To better understand why our approach provides valuable signals for enhancing online learning, undergo novel offline analysis comparing our method to alternative algorithms that pre-train state policies. Through this comparative analysis, we reveal that:
The naive approach of training state policies using continuous next-state targets results in state policies with poor prediction capabilities.
Our proposed discretisation mechanism enables us to sidestep the difficulties inherent in continuous regression.
Despite the loss of information via discretisation, OSO-DecQN is capable of suggesting trajectories that outperform the empirical performance of the dataset.
Regularisation is critical component in OSO-DecQN to prevent overestimation bias and ensure prediction error of state policies is small.
2 RELATED WORK
Offiiie RL Offline RL methods focus on learning policies using a fixed dataset generated by an unknown behaviour policy. large proportion of methods focus on constraining policies to stay close to the support of the data during training with the aim of mitigating the issues of distributional shift.
Although this has been achieved in various ways in the literature, most methods fall into two broad categories. Policy-constraint methods (Fujimoto & Gu 2021 Wu et al. 2019 Fujimoto et al. 2019 Kumar et al. 2019) encompass the set of algorithms that look to explicitly constrain the policy to ensure that the agent stays close to the data. Alternatively, conservative methods (An et al. 2021 Bai et al. 2022 Kumar et al. 2020) look at penalising the Q valuue estimate of out-of--disribution (OOD) actions. OSO-DecQN extends these concepts the action-free setting, adapting conservative
2

---

Published as a conference paper at ICLR 2026
regularisation techniques to constrain state transitions rather than actions, that additionally addresses the challenge of state reachability that is unique to action-free free offline RL.
State-only RL Edwards al. 2020) develop novel form of value function, QSS', which estimates the return of transitioning to a target state rather than a selected action. Despite estimating the Q-value of transitioning to next-states, they rely actions make next state predictions that can be used to train the critic. [epbbrrn et al.] (2024) extend this method to the offline setting, positing that it can be more effective develop policies from offline data by estimating the critic values of target states instead of actions. However, like Edwards et al. (2020). they still rely on using actions to train their networks. In contrast, Zhu et al. (2023) removes the need for action labels entirely and instead uses decision transformers to learn a state policy from action-fre free datasets offline. They build off the work of Chen et al. (202T) and use entire trajectories instead of single state to estimate the next state in the sequence. Decision transformers have also been proposed in an offline setting when actions are available (Bhargava et al. 2024), but several offline algorithms (Fujimoto & Gu 2021 An et al. 2021 Beeson & Montanal 2024) outperform them while also being less computationall expensive. Other action-free approaches adapt transformers (Zhou et al.) 2023] 2024, Luo et al. 2024 for visual data and unsupervised pretraining (Seo et al. 2022). However, (hou et al. (2023);
Luo et al. y023); Seo et al. (2022), assume that the visual demonstrations are of expert or near-expert quality, which enables purely mitaaionnlearning objectives. In contrast, in our offline RL setting, the data generating process is of unknown (and potentially mixed) quality. Zhu et al. (2023) can be viewed as an adaptation of Zhou et al. (2023) to the offline RL setting, using a similar implicit reward mechanism to guide online agents. Furthermore, our work differs by employing a simple discretisation strategy that circumvents the need for computationally intensive transformer- based architectures, while providing more stable performance through effective regularisation that addresses the critical reachability constraints overlooked in previous work.
Single agent action value decomposition Seyde et al. (0022) investigate solving continuous state-action problems using algorithms designed for discrete action settings. They develop a novel algorithm, Decoupled Q-Networks (DecQN), that combines deep Q-learning (Mnih et al.) 2015) with value decomposition (Sunehag et al. 2018; Rashid et al.) 2020). studied in multi-agent RL (MARL), to learn utility values for each action dimension. In their work, they treat each action dimension independently, allowing them to reduce the computational complexity from scaling exponentially with the action dimension to linearly. Ireland & Montanal (2024) build DecQN by showing that the use of ensembles can reduce variance, improving sample efficiency. [Beeson et al. (2024) explore the use of DecQN in the offline setting for continuous action environments. They generate several discrete action datasets across a wide range of continuous-action
tasks to benchmark offline variants of DecQN. [Tang al. (022) also look at using a linearly decomposed critic for offline RL, although they focus solely on discrete action environments. These efforts collectively show that decomposing the critic can be highly effective in both online and offline scenarios, which motivates our extension of decomposed Q-learning to action-free datasets.
3 BACKGROUND
3.1 PRELIMINARIES
We model the RL problem as a Markov decision process (MDP) that can be completely specified by the tuple (S, P. P.r,pe r.p,)), 7). where S C M and A = II=h A, C RN are, respectively, the state and action space. We assume the continuous action space has this Cartesian product structure, with no additional coupling constraints between dimensions. P(s" ss a) defines the environment dynamics, p(su) describes the initial state distribution, r S x A + R is the reward function, and y E (0, 1] is the discount factor. The agent's behaviour within the environment is specified by policy (alp iss and the expected return for given policy can be described using the action value function Q" (8, = E,, E120 7're|s0 = 8, 4g a).
3.2 DECOUPLED Q-NETWORKS
Both Seyde et al. 2002) and Tang et al. (002) propose the idea of decomposing the Q function, used in Deep N-Ntwoor (DQN) (Mnih et al. 2015). into utility functions Us(s, a.) for each action
3

---

Published as a conference paper at ICLR 2026
dimension. The overall action value is then estimated the average the utility functions
Qo(8, N Σ Ug (z,
where (a1, (di..... aN) E RN and U computes the utility value for the jth action dimension. This approach effectively treats each action dimension independently. reducing what would otherwise be an exponential complexity in discrete action settings to linear in N.
We define the argument of the maximum over a in an element-wise fashion as
arg max (s,s. arg max U, (1, a1)...., arg max U (8, aN) aN
The Q function is then trained by minimising the following loss function
(
Σ
L(y" Q0(s0:
(2)
B
(50, 1BB
where is the sampled batch and y" = Σj=o Pry + 2" Q a is the n-step Q-learning target, with 9 the lagged parameter of the target Q network. The action a* can be retrieved efficiently as a arg maxa Qa(s, a). The most common choices of L have been the Huber and MSE functions.
METHODOLOGY
4.1 DISCRETISING THE TARGET STATE SPACE
Our goal for pre-training is to learn state policy that predicts how each state dimension should cvolve to yield high returns. Instead directly regressing to continuous next states, an approach prone to instability and overfitting, state dimension, we learn whether will increase, decrease, or remain unchanged. Formally, we define 8 (s, s') as vector in (, with i-th component
introduce simple discretisation transformation. For each
{
if
< -c
(s,
{
if >
otherwise.
This transformation enables us to replace the instability of continuous regression with discrete pre- diction problem. In turn, allows us to leverage the well-studied value-based Q-learning framework to effectively learn from action-free datasets. We henceforth use the shorthand notation As to denote the discretised state difference.
Unlike the action discretisation approach of Seyde et al. (2022). discretise state differences after first applying z-score normalisation to each state dimension, yielding an effectively scale-invariant transformation. This eliminates dependence tuning, leading to a more robust performance across heterogeneous state spaces.
numerical ranges and avoids costly per-dimension
Discretising state differences simplifies prediction but inevitably introduces approximation error. To quantify this effect, we establish the following bound.
Theorem (Value bound under is evenly spaced bins in As). the discretised increment space As is partitioned into i evenly spaced bins per coordinate, then
HVM
-V-
where V and Vÿ are the optimal value functions of the original and discretised MDPs. Here M denotes the state dimension and H = 6t x - 8 denotes the range of the per-coordinate mean increments. The required assumptions, and the proof are provided in Appendix
This bound shows that discretising each state dimension into finite number of bins preserves the value function up controllable that diminishes as the discretisation becomes finer.
4

---

Published as a conference paper at ICLR 2026
4.2 OSO-DECQN
Our algorithm, Offline State Only DecQN (OSO-DecQN), adapts the DecQN framework to the action-free setting by replacing discrete actions with discretised state differences. Specifically, we define Q-values the form Q(s, As), where As E {-1,0, 1 represents the discretised change in each state dimension. This reformulation allows us to estimate action-free Q-values directly from (s, r, s') tuples, without requiring explicit action labels.
While DecQN decomposes Q-values acToss action dimensions, OSO-DecQN instead decomposes across state-difference dimensions, enabling the same scalability benefits in high-dit nigh-dimensional state spaces. We
ensemble variant of DecQN for improved sample efficiency, and employ a variant of double Q-learning (Van Hasselt et al. (016) to stabilise learning. Importantly, we further introduce regularisation term Rg, detailed in Section 4.3 that mitigates overestimation bias and addresses state reachability constraints, a challenge unique to pre-training from action-free learning and not addressed in standard DecQN.
Algorithm OSO-DecQN
1: Load dataset D with samples the form (s, s')
2: for i I... N do
3: Sample mini batch of transitions (s, s') ~
4: Sample As' based on the softmax of Qo (s',
5: Compute target y² r 7Qpss',
6:
Update parameters:
7:
θ arg arg Σ [v- Qo(s, + aR
8:
end for
4.3 REGULARISATION
Various regularisation techniques have been used in offline RL constrain action policies from deviating away from the support of the data. Many such methods rely the actor-critic framework that separates policy and critic learning. Value-basse based methods such as DecQN, however, implicitly learn the policy by training the critic, and thus cannot utilise such regularisation methods with the same convergence guarantees. Kumar et al. (2020) propose a penalty term in the critic loss to conservatively learn critic values in the actor-critic framework, which Luo et al. (2023) show equivalent to the negative fog-likelihood behavioural cloning (BC) loss the discrete action setting. Denoting exp((o(s, (s.
Ese exp(Q9(s, As)), we adapt this regularisation term for our discrete state difference, R2 = =E E( (e)d log (xp(QQ(ss.)) ))) Qo As), in Algorithm to mitigate the issues of both overestimation bias and state reachability constraints.
4.4 GUIDED ONLINE LEARNING
As offline trained state policies can not be directly used to interact with the environment, we instead explore how they can be used guide online learning. In particular, we identify several key factors that must be taken into account for guided online learning. Namely, we consider how to leverage the offline Q-values that define the state policy to guide training online, which type of online agent to guide using our method, and finally how to train an inverse dynamics model (IDM), from scratch, concurrently with the online agent to translate next state predictions. We address each of these points in the following sections.
Leveraging Q-values To demonstrate how our offline algorithm can enhance online performance, its critic to guide exploration within standard actor-critic algorithm. Specifically, introduce a policy-switching mechanism that integrates the pre- pre-trained state policy predictions into the online agent's decision- makkng. enabling more targeted and efficient exploration.
5

---

Published as a conference paper at ICLR 2026
We an IDM, denoted I, that is trained concurrently from seratch, to convert the state policy into actions and switch between the online policy and offline policy during training as follows:
(8 arg max (sa, As)) < <<8 3
A
a =
(s)
otherwise,
where Ç U(, and is a hyperparameter. The samples are stored in the online agent's replay buffer, where they are used to train both the IDM and the online agent.
Alternatielly, Zhu et al. ) propose using an intrinsic guiding reward related the discrepancy between the planned and true next state. However, without convergence guarantees, it is not clear whether this approach can generalise across tasks. Instead, we focus on leveraging Q-values to guide action selection and leave exploring reward shaping methods to future work.
Training an IDM We incorporate an IDM to translate next state predictions into actions that can be used to improve the online agent's behaviour. While more complex alternatives may exist for mapping next state predictions to actions, we deliberately adopt minimal IDM as the simplest approach that integrates with our method. This choice ensures low computational overhead and keeps our focus on the state policy's 's contribution rather than the translator architecture.
To train the IDM, we use supervised learning. We find that the L1 loss is better at achieving lower per-dimension errors than MSE. This is because the L1 loss approximates the median whereas the L2 loss approximates the mean, making the L1 loss more robust to outliers. In addition to the samples obtained online, we also leverage samples from the offline dataset that we denote (Soft, Asoft, Toff, Roft). )- Here, Roff represents the observed discounted return. Since assume that there are no actions in the offline dataset, we first assign an action using the online agent's policy as off Ton Ton (foff) ). We then determine what the IDM would predict by computing (s Asott). Consequently, the final loss is ( afn Ig(s, AS II + ||a Lo(s, As)]
Mapping state discretisations to actions Our guided online mechanism assumes that there exists sufficiently regular inverse relationship between actions and local state transitions in the environment. In particular, for most states encountered online, the mapping from action next state is locally smooth and predictable, so that a lightweight inverse dynamics model can learn ( As) that approximately realises the target discretised transition As. This requires local continuity in the dynamics (small changes in action induce coherent changes in the resulting state change) and discretisation that not so coarse to collapse distinct next-state transitions induced by different actions into the same As. When these conditions fail, more expressive IDM may be required to ensure that guidance benefits do not diminish. This can occur under strongly discontinuous dynamies or highly multi-modal inverse mappings where the same (8, As) corresponds to many distinct actions and their smoothed average does not realise As.
EXPERIMENTAL RESULTS
In this section, investigate the performance of OSO-DecQN at guiding online learning across wide range of continuous and discrete action tasks. We pre-train the discrete state policies on the datasets from the D4RL suite (Fu et 2020) and the Action-factorised DeepMind Suite (Beeson iet al.) 2024), removing all action information beforehand. These tasks present a set of challenging control problems that allow us assess the effectiveness and robustness of our algorithm. We provide more details on environments in Appendix C.I
5.1 GUIDED ONLINE LEARNING
In Figure [ we show that our method of guiding online agents using the offline trained state policies is capable of improving both asymptotic performance and convergence speed. This demonstrates the practical value of our action-free frre approach, as the learned state policies effectively transfer knowledge despite never having access to actions during offline training. We use TD3 (Fujimoto et al.) 2018) as the agent we focus on improving for the continuous action datasets and an ensemble variant of DecQN that denote DecQN N for the discrete action datasets.

---

Published as a conference paper at ICLR 2026
Halfcheetah
Walker2D
Hopper
Normalised returns
Normalised returns
Normalised
De..he medium
Timestops
Timesto ps
Timostops
(a) D4RL with TD3 as base algorithm.
Quadruped-Walk
Cheetah-Run
Normalised returns
Normalised returns
000. Destl-mmalar
200 Deotl-mnctare coperttrrtt
Destl-ecout
Timesteps
Timesteps
(b) DeepMind Control Suite with DecQN N as base algorithm.
Figure I: Online learning curves comparing the performance of guided online learning with state policies pre-trained on datasets of different qualities against baselines over 1M timesteps. The solid line corresponds to the mean normalised return across 5 seeds with the shaded area corresponding to 1 standard deviation away from the mean.
We observe that our method shows the smallest improvement the hopper environment, which we believe is result of being the simplest task with the fewest actions and observation dimensions, meaning the online agent able to learn quickly without guided exploration. Despite this, we are still able obtain slightly higher asymptotic performance and improve convergence speeds.
For Quadruped- -alkk, the environment with the highest state dimension (78 in total), we achieve a consistent performance improvement during the early stages of learning, demonstrating the scalability of our approach. We observed that the performance of the online agent DecQN .N without guided learning highly sensitive the choice of seed, leading the average performance is lower than the results reported in Treland & Montana (2024). We found that this was the result of minor changes we made to the hyperparameters and architecture they provided, which led to increased variability in performance. Further details on our hyperparameter choices are provided Appendix C Although our main experiments focus on enhancing TD3, Appendix demonttrates that the same method and set of hyperparameters can be applied to SAC with comparable improvements, showing that our approach can integrate seamlessly with other online algorithms and requires minimal tuning.
variability in the results. As result,
In conclusion, we observe that in both continuous and discrete action settings, OSO-DecQN is able to pre-train state policies that meaningfully accelerate and improve online learning, demonstrating that valuable knowledge can be extracted from offline datasets even without action labels.
IDM analysis To demonstrate the robustness of our method to the design of the IDM, we keep the IDM architecture and learning fixed across all environments in this section. Additionally, in Appendix E.I we conduct an ablation study varying the IDM architecture and batch size. The results show that, across both smaller and larger architectures and batch sizes, performance remains similar. This indicates, in our tested settings, that the IDM has minimal effect online guidance, allowing us to focus primarily on the predictive qualities the state policy. In addition to these experiments, we also examine the sensitivity of guided learning to 3 (Appendix D.3), demonstrating that our improvements persist across range of hyperparameter values. Finally, we provide an empirical analysis of IDM learning dynamics (Appendix (..)). Together, these results suggest that, in our tested settings, the IDM can reliably translate state policy predictions across diverse environments while incurring minimal computational overhead.
7

---

Published as a conference paper at ICLR 2026
Benchmark comparison We compare method with that of Zhu al. (2023) in Figure which presents an alternative approach to guided online learning using state policies. To keep the plots uncluttered, only include TD3 trained from scratch (we also provide combined plot with all results in Figure ( and note that, across all three environments, their method consistently underperforms relative TD3, in contrast ours. This further highlights the significance of our method in consistently providing a more effective signal for guiding online agents. Since our approach does not rely on a decision transformer, we can substantially reduce the time pre-train state policies on action-free free datasets. Furthermore, our algorithm scales efficiently with state dimensionality, enabling us to guide agents environments with up 78 state dimensions.
Halfcheetah AF guide
Walker2D AF guide
Hopper AF guide
Normallsed returns
Normalised returns
Gast-matim
Normalised
Timnssepss
Timestepp
Timesteps
Figure 2: Comparison of Af-guide (Zhu al. 2023) against TD3 baseline. The solid line corresponds to the mean normalised return across seeds with the shaded area corresponding to s.d.
6 PRE-TRAINING ANALYSIS AND ABLATION
In this section, we analyse our decision to discretise the state space and explain why the Q-values defining our state policy provide valuable signal that improves online sample efficiency. We also examine the effect of regularisation on state-reachability constraints and overestimation bias.
Inverse dynamics model (analysis only) As we learn state policies from action- datasets, we have no way to directly evaluate their performance in the environment. In this section, for the sole purpose of comparing the predictive capabilities of state policies pre-trained using different algorithms, we use an expert-level IDM to translate predicted next-state targets (As, s', s' -) s) into actions so that they can be evaluated in the environment. By using an expert-level IDM, we minimise the bias introduced when mapping state policy predictions to actions, ensuring that observed differences in performance can be attributed to the state policies themselves. The expert IDM trained independently of our method and all action-free baselines and is used only for the offline analysis in this section. It is not used anywhere in our main algorithm or online experiments.
Why discretisation? To understand why we discretise the state space, investigate the perfor- mance of state policies trained using imitation learning. We use imitation learning to pre-train and compare the performance of state policies that predict the next state s', the continuous state difference - s, and the discrete state difference that we denote As across several datasets in the first four columns of Table [able]]eelo.. below. As reference. we also provide the performance of BC policy that is trained on the same dataset with actions. We denote the name of each imitation method as BC + the target is trying to mimic.
Table shows that the policy trained to mimic As targets is capable of generating trajectories that yield returns that match the performance of imitation policy trained with action labels. In contrast, directly mimicking the continuous target state or state difference from the dataset results in poor evaluation performance.
To further understand why performance is poor when directly imitating s' o' - 8, we introduce an additional evaluation metric: the discrete state difference error per step, averaged over an entire trajectory. This metric, reported in Table 2 (Appendix A). complements the return-based evaluation in Table by directly measuring the predictive accuracy of state policies. To compute this metric, we transform the true state difference syb. Sobe- into discrete state difference using the procedure outlined in Equation 3 and compare it to the predictions, As ASpredicts of the imitation policies at each time step during episode. We compute the discrete difference error, Asobs A Spredict for each
8

---

Published as a conference paper at ICLR 2026
time step and average the error across entire episode for each imitation learning algorithm. We expect that for algorithms that fail to replicate trajectories, this error will be much higher. For random policy, the average discrete difference eπror is roughly equal to the number of dimensions in the state space, since the average error per state dimension is approximately 1. We include the discrete difference error for random policy in Table 2 as reference.
Table I: Normalised average returns on D4RL and Factorised action tasks. Scores are averaged across 5 seeds with 10 episodes per seed. Ensemble- based methods use N = 5. Where relevant, we report the mean + standard error.
Reference
Why discretisation?
Why regularisation?
Dataset
BC
BC s'
BC
BC As
DeeQN N
OSO-DecQN
Hopper
medium-replay
26.6
5,8
4.9
29.2
+ 3.7
7.7 + 1.3
65.7 +: 2.6
-medium
51.5
3,5
31.4
47.5
= 1.2
1.2 + 0.31
55.8 + 1.3
-medium-expert
51.3
2.9
15.6
#.1
1.1 0.33
110 1.3
-expert
110.6
2.2
9.9
106.9
= 2.4
1.9 + 0.45
111.6 + 0.08
Halfcheetah
-medium-replay
36.3
-0.9
-0.58
40.2 + 0.43
0072
43 + 0.54
-medium
41.9
-0,2
-0.41
42.8 + 0.21
-1.9 0.15
44.6 + 0.11
-medium-expert
60.1
-0.25
-0.25
54.7
+ 3.9
-1.6 0.13
87.8 + 2.7
-expert
92.6
-1.7
-1
95.4 + 0.63
-1.9 0.09
95.1 + 0.26
Walker2d
-medium-replay
23.5
6,4
-0.32
37.5 37.5 6.1
-0.41
= 0.26
84.8 +
-medium
71.8
9.1
-0.76
60.1 +4
60.1
-0.33 = 0.01
77 + 1.3
mediumm-xppert
107.7
2.3
-0.71
84.4 - 3.4
-0.24 0.06
108.8
+ 0.13
-expert
108.2
17.1
-0.69
108 + 0.12
0.86 = 0.94
108.1
+ 0.13
Cheetah-
randomm-nen ceetonm-meiummmm- medium-expert
41.5
-0.16
-0.16
1.2 0.09
+ 0.8
-medium
40.4
-0.69
-0.69
41.2 0.4
0,95 0.13
41.8
-medium-expert
61.6
1.7
1.7
+
1.2 0.32
90 + 3.8
-expert
99.9
-0.56
-0.56
93.6 + 3.7
2, 0.02
98.4 +
Quadruped- Walk manddoommmdium-expprt
28
6.6
6.6
39.6 = 6.3
2.3 0.1
45.4 + 6.8
-medium -eddium-expert -expert
39.2
8.4
8.4
47.8
7.2
2.9 + 0.32
52.7 7.8
63.4
4.7
4.7
8.1
0.19 + 0.03
91.2 5.8
97.7
6.6
6.6
96.7
6.4
1 0.04
100.7
1.2
Comparing the two tables, we see a strong correlation between the performance in Table and predictive accuracy in Tf state policies. Specifically, the imitation policies trained using s' and have errors comparable to the random policy several benchmarks, whilst our transformation procedure allows us keep errors down across all datasets. We also observe that the errors produced by our method are marginally larger for the more diverse datasets, medium-replay and medium setiium-expprt.. This is result of the imitation policy averaging across wider range of discrete next states available in these datasets.
Why regularisation? We conduct an ablation study to understand the impact of regularisation. To do this, we compare our algorithm, OSO-DecQN, with state-only variant of DecQN that is identical in all respects except that it omits the regularisation term. To control for ensemble size, we configure the benchmark use the same number of crities our method and refer this variant as DecQN N. We also compare against the imitation learning methods discussed in the previous section demonstrate that our method is capable of using reinforcement learning to improve upon supervised learning methods.
the last two columns of Table we clearly see that OSO-DecQN is capable of outperforming both action and discrete state imitation learning methods. We believe this is clear sign that the loss of information from our state discretisation transformation has minimal impact on our method's 's ability to reinforcement learning to improve the dataset trajectories. We also that DecQN_N struggles to learn, indicating that regularisation is critical for offline performance and mitigating overestimation bias. In Table we observe that the lack of regularisation in DecQN N also results

---

Published as a conference paper at ICLR 2026
in the trained state policy not being able accurately predict next states, indicating the policy also suffers from state reachability constraints.
Summary of findings From our analysis. we can conclude that both discretisation and regularisation are key components to ensuring that we can pre-train state policies effectively from action-free datasets. Table 2 shows that OSO-DecQN can accurately predict the next state, addressing state reachability constraints, whilst Table T shows that OSO-DecQN is capable of suggesting next states that yield high long- term returns, addressing overestimation bias. This explains why state policies trained using our method provide beneficial signals for guided online learning that result in enhanced sample efficiency.
In addition to the above analysis, Appendix presentss an ablation study on the sensitivity of pre- training performance the discretisation threshold and a comparison between 2-bin and 3-bin discretisation. These results demonstrate that our method is robust to a broad range of e values and discretisation granularities.
7 DISCUSSION
Leveraging datasets without action labels presents a fundamentally challenging problem where stan- dard methods cannot be directly applied, yet such scenarios are common in domains like healthcare, finance, and robotics. Our work establishes scalable and principled framework for action-free offline-to-onlin RL, showing that effective policies can be trained directly from (s, r, s') tuples and then successfully transferred online agents. By discretising state differences with scale- invariant transformation and applying conservative regularisation, OSO-DecQN avoids the instability of continuous regression and constrains predictions to reachable states. Coupled with a lightweight guided online learning mechanism, this framework consistently improves convergence speed and asymptotic performance across diverse benchmarks, scaling to complex environments with up to 78 dimensions. These results provide the first evidence that reliable guidance can be extracted from action-fic free datasets at scale. Furthermore, our analysis in Section ( demonstrates that our learned state policy provides useful signals for online guidance. We believe this analysis will be valuable to the community for benchmarking and validating alternative action-free state policies.
Looking forward, potential avenue could be to adopt adaptive discretisation mechanisms such as (Seyde et al.) (2024) investigate whether more fine-tuned control can enhance pre-training performance and online guidance. Another is to evaluate more expressive translation mechanisms that go beyond the current minimal IDM, potentially improving robustness and performance in more complex domains.
several promising directions for further developing approach. One
Another promising avenue of research would be to adapt our method to learn from video comprised of suboptimal demonstrations similar those available in vectorised state format in the D4RL and action- factorised datasets. While there exist methods (include citations) that leverage existing encoders to transform images into vector representations compatible with our approach, additional work is needed on how to reliably extract reward information from visual data to execute RL. algorithms.
In complimentary direction, would also be interesting to combine our method with recent work that replaces value-function regression by classification over discretised value supports (Farebrother et al., (0244). Incorporating such a cross-entropy based objective into our framework could further improve the stability and scalability of the critic, particularly when utilising larger architectures or solving problems in more complex domains.
ACKNOWLEDGEMENNS
This work was supported by the Engineering and Physical Sciences Research Council (2674942) and UKRI Turing AI Acceleration Fellowship (EP/V024868/I), Jeremie Houssineau is supported by the Singapore Ministry of Digital Development and Information under the AI Visiting Professorship Programme, award number AIVP-2024-004.
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
We provide code as supplementary material to enable reproduction of all experiments presented in this paper. Detailed hyperparameter settings are provided in Appendix all assumptions and complete proofs for the theorems stated in this work in Appendix
C Additionally, we include
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-bant offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7444 -447, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- ran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference Learning Representations, 2022.
Alex Beeson and Giovanni Montana. Balancing policy constraint and ensemble size in uncertainty- based offline reinforcement learning. Machine Learning, 113(1):443 488, 2024.
Alex Beeson, David Ireland, and Giovanni Montana. An investigation of offline reinforcement learning in factorisable action spaces. Transactions on Machine Learning Research, 2024.
Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, and Amy Zhang. When should we prefer decision transformers for offline reinforcement learning? In The Twelfth International Conference on Learning Representations, 2024.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 14:1558444 1084-15097, 2021.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience, edition, 2006.
Richard M. Dudley. Real Analysis and Probability. Cambridge University Press, 2002.
Ashley Edwards, Himanshu Sahni, Rosanne Liu, Jane Hung, Ankit Jain, Rui Wang, Adrien Ecoffet, Thomas Miconi, Charles Isbell, and Jason Yosinski. Estimating q (s. s') with deep deterministic dynamics gradients. In International Conference on Machine Learning, pp. 2825-2835, PMLR, 2020.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. In International Conference Machine Learning, pp. 13049-13071. PMLR, 2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/9 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 14-20132-20144 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International conference on machine learning, pp. 2052 2062. PMLR, 2019.
Charles A Hepburn, Yue Jin, and Giovanni Montana. State-constrained offline reinforcement learning.
arXiv preprint 24t.:24005113774 2024.
David Ireland and Giovanni Montana. Revalued: Regularised ensemble value-decomposition for factorisable markov decision processes. In The Twelfth International Conference on Learning Representations, 2024.
11

---

Published as a conference paper at ICLR 2026
Iyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
Clete A. Kushida, Deborah A. Nichols, Rik Jadmicek,, Ric Miller, James K. Walsh, and Kara Griffin. Strategies for de-identification and anonymization of electronic health record data for use in multicenter research studies. Medical Care, 50(Suppl:s82--1111 2012. doi: 10.1097/MLR. 06013e3182585355.
Xiangwen Liu, Xia Feng, and Yuquan Zhu. Transactional data anonymization for privacy and information preservation via disassociation and local suppression. Symmetry, 14(3):472, 2022.
Grigorios Loukides and Aris Gkoulalas-Divanis, Utility-preserving transaction data anonymization with low information loss. Expert systems with applications, (9(10):9764-977)
Hao Luo, Bohan Zhou, and Zongqing Lu. Pre-trained visual dynamies representations for efficient policy learning. In European Conference on Computer Vision, pp. 249-267. Springer, 2024.
Jianlan Luo, Perry Dong, Jeffrey Wu, Aviral Kumar, Xinyang Geng, and Sergey Levine. Action- quantized offline reinforcement learning for robotic skill learning. In Conference on Robot Learning, pp. 1348-1361. PMLR, 2023.
Max Sobol Mark, Ali Ghadirzadeh, Xi Chen, and Chelsea Finn. Fine-tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 518(7540):529-33 533, 2015.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36:62244-62269 2023.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action- free pre-training from videos. In International Conference on Machine Learning, pp. 19561-19579, PMLR, 2022.
Tim Seyde, Peter Werner, Wilko Schwarting, Igor Gilitschenski, Martin Riedmiller, Daniela Rus, and Markus Wulfmeier. Solving continuous control via q-learning. arXiv preprint arXiv:2210.12566. 2022.
Tim Seyde, Peter Werner, Wilko Schwarting, Markus Wulfmeier, and Daniela Rus. Growing q networks: Solving continuous control tasks with adaptive control resolution. In 6th Annual Learning for Dynamics & Control Conference, pp. 1646-1661. PMLR, 2024.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki. Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decompoositoon networks for cooperative multi-agent learning based on team reward. Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2085-2087, 2018.
Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi- Velez, and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems, 15:34272-34286, 2022.
12

---

Published as a conference paper at ICLR 2026
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012. doi: 10.1 20.109//0R0..2222.68800000
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-
learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint 1 1361, 2019.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healtheare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
Bohan Zhou, Ke Li, Jiechuan Jiang, and Zongqing Lu. Learning from visual observation via offline pretrained state-to-go transformer. Advances in Neural Information Processing Systems, 36: 59585-59605, 2023.
Bohan Zhou, Jiangxing Wang, and Zongqing Lu. Nolo: Navigate only look once. arXiv preprint arXiv:2408.01388, 2024.
Deyao Zhu, Yuhui Wang. Jürgen Schmidhuber, and Mohamed Elhoseiny. Guiding online reinforce- ment learning with action-free free offline pretraining. arXiv preprint arXiv:2301.12876, 2023.
13

---

Published as a conference paper at ICLR 2026
A STATE POLICY PREDICTION ERROR
Table 2: Average discrete difference error, |A predictt) across an entire episode. Results are averaged across 5 seeds with 10 episodes per seed. We report reference the expected error of random policy denoted "Rnd". Where relevant, we report the mean + standard error.
Reference
Why discretisation?
Why regularisation?
Dataset
Rnd
BC s'
BC
BC As
DecQN.N OSO-DecQN
Hopper
medium-rcplay
11
12.1
7.9
0.68 0.04
11.8 + 0.54
0.99 0.04
-medium
10.5
1.9
0.25 0.01
6.8 -
0.18 0.008
-medium-expeee expeert
10.8
0.18
10.7 + 0.49
0.16 0.004
-expert
11.5
0.15 0.02
0.004 7.2 1.3
Halfcheetah
mediuum-rppla
17
17.1
2.9
1.1
11.1 + 0.2
1.42 + 0.35
-medium
17
15
3.5
0.39
16.8 2 0.19
0.76 0.01
medium-expert
17
17
2.3
0,46 + 0.04
16.8 0.01
0,43
+ 0.02
-expert
17
16.9
4.2
0.38 + 0.03
16.9 0.36
0.36
0.009
Walker2d
medium--eppay
17
16.2
16.6
2.9
17.3 + 0.81
2.6 0.04
-medium
17
15
16.6
1.8 0.3
13.3 - 0.55
1.6 0.02
-medium-expert
17
20.3
16.6
0.87 = 0.08
9.7 = 2.2
0.75 +:55:0000
-expert
17
17.2
17.1
0.45 + 0.009
14.4 0.48
0.55 + 0.02
Cheetah-Run
-aadomm-medin medumm---iiu
17
16.8
6.5
2.6 0.06
15.3 0.76
1.89 0.01
-medium
17
16
5.1
1.4 0.02
16.7 + 0.13
1.07 0.01
-medium-expert
17
15.8
13.5
1.6 0.02
12 - 0.63
1.05 + 0.09
-expert
17
16,9
4.1
1.2 + 0.09
17 + 0.02
0.86 + 0.06
Quadruped- Walk mindom-medium-expert
78
74.4
19.8
0.39
77.1 0.03
15.8 +
-medium
78
77.6
19.8
13 0.27
77.2
0.21
12 0.25
-ediium-expee expert
78
74.5
10.6
14.7
0.34
78.2
- 0.26
13.8 0.17
-exeert
78
77
7.8
13.5
0.49
77.2 - 0.04
12.2 1 0.06
Table 2 compares the predictive capabilities of state policies trained using different algorithms.
B PROOFS
Assumptions and Definitions.
Throughout, we define and assume:
M = AS,, is an MDP with rewards bounded within [rmin fmax]] and we define mmax
A
The discount factor satisfies 1).
For each action a E A, discretisation is defined by projection b: - Ap mapping a continuous action its discrete bin. Without loss of generality, assume this mapping is independent of the state.
.MD = (S, Ap, P. rry)) 7) denotes the MDP obtained from M by discretising its action space.
V and VB are the optimal value functions M and its discretisation Ma respectively.
εKr. the symmetrised KI. mismatch of next-state distributions and is defined as
<KL = sup sup min ( a) P( M()),, Dku(P(-la (())ss,())) P(lss a))) eES a
Lemma (Control of expectation differences via TV and KL). Let μ, v be probability measures on a measurable space and let f be bounded and measurable. Then
/ du / du sp(f)TV(u,
sp(ʃ) == sup f(x) - iinf f(x),
14

---

Published as a conference paper at ICLR 2026
see Dudley 2002). Moreover, by Pinsker's inequality (Cover & Thomas 2006
TV S V DRI(ullP).
Lemma 2 (KL bound for discretised actions). If M is discretised by binning each action + b(a). then
IV Vi S
A. 7
4r
EKL.
1-y
(1 (1-
2
Proof.
Let the Bellman operators be
max + Ma aEd
=
and
(TDV)(s) = max acAn
r(s, Pta (V))))
By 7-contraction of Tp.
(1 11>) |V VD
max Q- (.,a) a) max Q' @'(.ā))le.
aEA
aeAp
Fix and let a" arg max, (" a); set b(a") the bin representative. Then
max Q2, max aEA
(" a) s (s,saa") - Q*
aEAD
Expanding Q" gives
= ( (Ess)(()a)) - E P(rp))))
+
Define A, such that A, Fmax
Tmin-
By the TV dual bound and Pinsker (Lemma
E Bx (V* (s') Ex VV (s') s spp(V") v DKL (PL a)).
Taking suprema and inserting the definition of CKL,
(1->)llv- - Vollo N A, + sp(V")
CKL
Finally, apply sp(V*) < A, 7) conclude
IV Vollo0 5 1-
Ar + (1 7)
A, Arv
EKL
2
as claimed.
Lemma 3 Lis-binning bound in continuous space). If the continuous increment 8s is discretised into As by a binning rule, then
IV Volloc
A + <
KL 2
(1
Proof. L R and da RM + B be a binning rule. Define the discretised increment As = (8) and denote by P(As a) the induced distribution. For each state s, form bins C, by the equivalence relation a ~, a' iff P(As s,a) = P(As 8, a')., and pick a representative âc E C for each C C.
Construct the discretised MDP My by mapping each a E C to its representative (s). Applying Lemma 2 with this mapping and the above definition of eKI. yields the claimed bound.
15

---

Published as a conference paper at ICLR 2026
Theorem 2 (Value bound under & evenly spaced bins in As). Assume the continuous increment satisfies
Σa(s) > 0 independent of and there exist constants min < max such that, for every coordinate i == l,.... ,.....,M M and all (s, a),
ôs (s,a) ~ (Haa(s, a), Ea(s)),
8min S (1, Smax-
If the discretised increment space As is partitioned into k evenly spaced bins per coordinate, then
| Vollo O HvM M
Proof. For each 8, partition the mean-increment box
[dsmin, 88max)] M
into k equal bins per coordinate (evenly spaced). For each bin C, choose the representative ac so that (a(s, āc) is the bin center. Let
H
H == 8smax 8smins
cell
=
V,, 2k
As == sup Amax [Sa(s)
with Amax (-) the largest eigenvalue of its argument. By construction, for any E C,
[a (a(s, ac)l2 S rell
Since the covariances are equal and independent of DKI(N(H, Σ) [[ Σ)) _ (p 11) E- n²) I >max 11- 4 11112.
Apply this with μ (s, a), n² pa(s, E [s)s), and then take suprema over C, C, and s to obtain the symmetrised mismatch
As MH
As (ras)2
CKL
12
Plug into Lemma
3. Theorem gives
V - Vbllo
A
A
EKL
D
7
A, v As
H
1-7
11
4(1
7)2
which grows at the claimed order.
C EXPERIMENTAL DETAILS
C.I ENVIRONMENT DETAILS
D4RL suite The D4RL Fu al. 2020) suite comprises datasets generated across of tasks that use the MuloCo Todorov et al. 2012) physics engine. For each task, rewards are provided according the agent's 's ability learn precise control and balance of robotic model whilst motion. For each problem, several datasets are generated using different quality policies. The different quality data allow us investigate how robust our algorithm the getta-ggnerating process.
Action-factorised DeepMind suite Beeson et al. (2024) investigate offline learning in continuous action environment with datasets generated using discrete actions. We use their suite as an opportunity to investigate how method handles offline learning with smaller datasets and environments with larger state and action dimensions than those available the D4RL suite.
Dataset sizes Table 3 summarises the number of offline state-only samples used for each dataset. For datasets that share the same size across environments, we simply indicate the dataset quality (medium, medium-expert, expert...eeeeee...........
16

---

Published as a conference paper at ICLR 2026
Dataset names
Dataset size
D4RL Hopper mediumm-replas
402k samples
D4RL HalfCheetah medium-replat
202k samples
D4RL Walker2D medium-repla,
302k samples
D4RL medium
1M samples
D4RL medium-expert
2M samples
D4RL expert
IM samples
Action factorised mandom-medium-expen
200k samples
Action factorised medium
IM samples
Action factorised medium-expee expert
2M samples
Action factorised expert
1M samples
Table Number of offline state-only samples in each dataset used to pre-train the offline state policy.
Table List of general hyperparameters used.
Hyperparameter
Value
Optimiser
Adam
Mini-batch size
256
Target update rate
lc-3
General Hyperparameters
Gamma
0.99
Hidden dim
512
Hidden layers
3
Activation function
ReLU
discrete epsilon (e)
le-4
Offline DecQN Ensemble size
Learning rate
le44
n step returns
Ensemble size
Minimum exploration
0.05
Online DecQN_N
decay rate
0.999
step returns
Learning rate
5c-4
Ensemble size
n step returns
Policy noise
0.2
Online TD3
Policy noise clipping
(-0.5.0.5)
Policy update frequency
2
Critic learning rate
Se-4
Actor learning rate
5c-4
Learning
lc-3
IDM
Mini-batch size
512
C.2 OSO-DECQN HYPERPARAMETERS
We provide detailed list of hyperparameters used for our experiments in Tables and
C.3 IMPLEMENTATION DETAILS
We run all tests on a single GeForce RTX 3080. We implement our algorithm with Python 3.10 using the pytorch framework. We run all offline experiments for 3 million time steps and all online experiments for million time steps. In T and 7 we provide the computational cost running our method on the Walker2d/Cheetah environment for offline pre-training and online learning, In Table g on all environments. We note slightly higher runtimes for guided online method overhead of concurrently train an IDM. For the online hyperparameter 3. we start with the initial value of 3max and decrease the value every 100k timesteps by Baccrement until 3mii reached. We
we provide a comparison of total runtime cost for our method compared to baseline methods
result of the
17

---

Published as a conference paper at ICLR 2026
Table 5: Regularisation (a) weights used during offline learning.
Darasct
Hopper
-mediom-meplay
-medium
8
-medium-espert
-esper
Haffeheccth
medium--rppll
2
medium
-mediun-expert
espert
Wallierd
-medium-rppll
10
-medium
-medium-cxpert
-espert
Chectab expert
-edium
medium expert
-expert
Quadraped Walk condken-moiiu--cpprr -eedium
-eddium-sspert
fix 8max = 0.5, Sderrement = 0.1 and Bmin = 0 across all environments and dataset qualities. In Table we list the regularisation (a) values used. line with how the datasets were curated, for Q learning,
MSE loss for D4RL suite and Huber loss for the factoriicd DeepMind suite. The general hyperparameters Table 4 apply to all network architectures unless specified otherwise. All networks standard feed-forwane neural network architecture.
Runtime offline Runtime offfine GPU Mem. (offline) (MiB)
(s/epoch")
(s/epoch")
OSO-DecQN (N=5)
44.9
710
BC As
28
542
II epoch 10,000 gradient steps
Table Computational cost of pre-training on Walker2d/Cheetah environments.
Runtime online GPU Mem. online (s/epoch*)
(MiB)
Guided learning
88.1
638
(Section 4.4)
DecQN N (N=5)
52.7
578
TD3 (N=2)
51
540
1 epoch = 10,000 gradient steps
Table 7: Computational cost for (guided) online learning Walker2d/Cheetah environments.
C.4 BASELINES
We used implementation for all algorithms used in the main paper besides Behavioural Cloning (BC) with actions. We report the results of BC with actions from Beeson et al.) (2024) for the factorised action datasets.
18

---

Published as a conference paper at ICLR 2026
Hopper
HalfChectah
Walker2D
Cheetah-Run Quadruped Walk
OSO-DecQN
2.9 1.8
3.7 + 1.9
3.7 1.7
3.3 + 1.9
5 5 2.3
offline online
DecQN_N
1
1.4
(N=5)
TD3
1.1
1.2
1.1
(N=2)
Table 8: Total cost for our method on all environments compared to baselines (in hours). For OSO-DecQN we separate time into (pretraining) + (online).
D ABLATION STUDIES
D.1 IMPACT OF CHANGING
We provide analysis demonstrating the effect of varying the value of that specifies the range of values that are converted to the discretisation process. We look at how performance varies when the threshold value is set to (I 10
10 10 5, 10
Table Ablation studying the effects of varying c. Results averaged across 3 seeds + s.e.
x 10m4
10"4
x 10
x 10-6
hopper medium expert
107.5
110 4 1.3
112 0.09
111.7 + 2.3
halfcheetah medium replay halfcheetah medium expert
42.3 0.73
43.1 0.1
41.2
1,25
0.77
87.8 2.7
93.8 + 2.3
87.6 + 5.3
walker2d medium walker2d expert
1.
77 - 1.3
84.6 4 0.26
79.7 + 2.8
109 = 0.24
108.1 = 0.13
108.8 - 0.12
108.7 + 0.07
D.2 SETTING TO 0
The parameter e is introduced as a tolerance for declaring "no change" in a state dimension after normalisation. This helps avoid spurious sign flips caused by small numerical noise. In this section we look at the impact of removing 0 from the discrete options, instead discretising state difference into bins instead of three. Despite only using two bins, the number of possible As available from any given state is 2d)m(S), We show in Table 10 that this is also sufficient amount of granularity for our method to learn from an offline dataset. We do however observe some notable degradation in performance, namely Walker2d medium-replay and Hopper medium-replay indicating setting = 0 slightly hurts some environments because every tiny fluctuation is treated as a change. We also observe, on average, higher variation across seeds. We do minor improvements on performance in some datasets, most notably Walker2d-medine which we believe could be as a result of better balance between overfitting and information loss for these less diverse datasets.
We stick to only investigating 2 or bins, as these are the only obvious ways to discretise the state difference s' s. Using 1 bin would result in learning. and increasing the number of bins beyond would require setting thresholds for each bin, which could likely have vary for each state dimension.
Table presents the average discrete difference error when using two and three bins. For reference, we include a column showing the expected error under random policy for both bin configurations. see that the error lower when using fewer bins. Additionally, we report the discrete difference percentage of the corresponding random policy's is error (in brackets),
enror of OSO-DecQN providing normalised metric for comparison.
D.3 EMPIRICAL ANALYSIS TO 8 SENSITIVITY
In this section we analyse the sensitivity of our method to the choice of by comparing a linearly annealed schedule to keeping / fixed throughout training
19

---

Published as a conference paper at ICLR 2026
Table 10: Comparing the performance of our algorithm when discretising As using two and three bins
Dataset
BC As (3 bins)
OSO-DecQN (2 bins)
OSO-DecQN (3 bins)
Hopper
-medium-replay
29.2 1: 3.7
61.5 + 7.27
65.7 2.6
-medium
47.5 1: 1.2
52.2 + 1.31
55.8 i: 1.3
-medium-expert
51.9 2.1
110.4 = 1.21
110 1.3
-expert
106.9 2.4
111.3 0.43
111.6 0.08
Halfcheetah
-ediuu--replay
40.2
43.5 0.32
0.54
meediu
42.8 0.21
44
44.6 + 0.11
-medium-expert
54.7 + 3,9
86.3 = 1.57
87.8 + 2.7
-expert
95.4 + 0.63
97.9 + 0.26
95.1 + 0.26
Walker2d
medium-replay
37.5 6.1
73.8 3.8
84.8 2.2
-medium
60.1 +
81.7 + 0.76
77 1.3
-eedium-expert
84.4 +
108.5 0.15
108.8 0.13
-expert
108 + 0.12
108.3 = 0.09
108.1 1 + 0.13
Table 11: Average discrete difference error, S ASpredictl across an entire episode. Results averaged across seeds with 10 episodes per seed. We report reference the expected error of a random policy denoted "Rnd". In brackets we provide a percentage ratio of our algorithm error compared the random policy using the respective number bins. Where relevant, we report the mean + standard eπror.
Dataset
Rnd (3 bins)
Rnd (2bins)
OSO-DecQN (2 bins)
OSO-DecQN (3 bins)
Hopper
-medium-replay
11
5.5
0.51 + 0.02 (9.27%)
0.99 +:2 0.04 (9%)
-medium
11
5.5
0.1 + 0.004 (1.82%)
0.18 = 0.008 (1.64%)
-eddium-expert
11
5.5
0.07 + 0.004 (1.27%)
0.16 0.004 (1.45%)
-expert
5.5
0.06 + 0.001 (1.09%)
0.12 0.004 (1.09%)
Halfcheetah
-medium-replay
8.5
0.68 0.004 (8
1.42
(8.35%)
-medium
8.5
0.35 1 0.002 (4.12%)
0.76 -: 0.01 (4.47%)
-medium-expert
17
8.5
0.24 + 0.004 (2.82%)
0.43 - 0.02 (2.53%)
-expert
17
8.5
0.22 + 0.009 (2.59%)
0.36 - 0.009 (2.12%)
Walker2d
-medium-replay
17
8.5
0.02 (17.4%)
2.6 = 0.04 (15.5%)
-medium
8.5
0.81 0.02 (9.52%)
1.6 0.02 (9.59%)
-medium-expert
0.41 1 0.02 (4.82%)
0.75 0.02 (4.41%)
-expert
8.5
0.27 + 0.01 (3.18%)
0.55 4 0.02 (3.24%)
Walker2D
Halfcheetah
Hopper
Normalised returns
Normalssed returns
Normallsed returns
m/
050. Desth eopteetttttttooo bett
050 beed eoteeetpttttttt beta =3.
DecON-empnt beta arnodingg
Timessopp
Tiimestops
Timesteps
Figure 3: Comparison of fixed versus linearly annealed 3 in guided online learning for Walker2D and Hopper.
In Figure 3 we observe that in the simpler Hopper environment, where the online agent converges rapidly, the choice of B has the smallest impact performance. In contrast, in both HalfCheetah and
20

---

Published as a conference paper at ICLR 2026
Walker2D environments, performance degrades substantially when 3 = 0.8 is kept fixed throughout training. In these cases, a large proportion of actions originate from the offline state policy while the IDM is still inaccurate, leading poor early exploration that hinders the online agent.
This degradation mitigated when smaller values are used. For example, in HalfChectah, per- formance with 0.2 and / = 0.5 is similar during learning, but annealing / results in slightly improved asymptotic performance. In Walker2D, annealing from 3 = 0.5 yields the best results, combining strong early exploration from guided actions with the flexibility to rely more the agent's own learned policy later in training.
Our interpretation is that B controls the exploration-explootation balance induced by the offline state policy. A larger 3 increases the probability that the agent follows suggestions from the offline state policy rather than its current online policy, which encourages offline-guided exploration early in training but can lead to under-exploitation of the learned policy later on. However, when / is too large, the offline guidance becomes dominated by noise in the IDM predictions, leading to the performance degradation observed in Walker2D and HalfCheetah.
In contrast, the annealed schedule starts from a relatively high 3, leveraging the offline state policy to diversify exploration when the online agent is still weak, and then gradually reduces , so that the agent increasingly exploits own learned policy. This dynamic shift from offline-guided exploration to exploitation of the online policy improves both stability and final performance. While annealed initially learns slower than = 0.2 (due to the noisy IDM predictions), it quickly surpasses 0.2 as the IDM becomes accurate, achieving improved returns. This supports our view that annealing 3 provides more favourable
trade-off than any fixed choice.
E ONLINE IDM ANALYSIS
E. ARCHITECTURAL AND BATCH SIZE ABLATION
In Table 12 we that varying the IDM depth and minibatch size produced only moderate change in guided online learning performance. While three layers with a batch size of 512 achieved the highest returns in both environments, the gains over other configurations were relatively small, indicating that the IDM capacity is not constraining learning performance. This supports our design choice of using a lightweight IDM. as performance is driven primarily by the quality of the offline state policy rather than fine-tuning performance.
Table 12: Effect of varying number of layers and minibatch size of the online IDM on final online performance for HalfCheetah medium and Walker2D medium. Results averages over seeds + s.c.
HalfCheetah medium
Walker2D medium
Layers
Batch 384
Batch 512
Batch 768
Layers
Batch 384
Batch 512
Batch 768
117.9 5.9
116.7 5.39
113.6 11.8
2
115 7.3
113.3 + 3.2
112.7 15
3
111.6 5
116.4 + 4.7
116.4 + 8.2
3
110.3 + 12.1
120 + 5.1
114.7 = 7.7
103.2 + 2.1
104.4
114.4 + 6.3
113.1 + 2.3
116.8 + 6.9
119.8 13.4
Consistent with the observation, across IDM architectures we find only weak correlation between IDM loss and final online return (Pearson r = 0.19 for HalfCheetah-medine and r = -0.09 for Walker2D-medium, indicating the variation in IDM prediction error explains little of the variation in performance.
E.2 LEARNING DYNAMICS
In Figure 4 look at the evolution of the online IDM loss, defined as:
LIDM |aDM - offll + ||a I(s, As)
over the course of online training. Across all environments, the IDM loss converges quickly and remains stable for the remainder of learning. To quantify the final accuracy of the online IDM, Table 13 compares the final online IDM loss after training with the loss of the expert-level IDM used in
21

---

Published as a conference paper at ICLR 2026
Section 6 We note that the online IDM loss is composed of two terms, making it roughly twice as large as the expert IDM loss. By considering approximately half the total online IDM loss, the online IDM converges to level comparable to the expert IDM across all environments. This demonstrates that, even without action-labelled pre-training, the online IDM can rapidly achieve expert-level mapping accuracy, ensuring it is not limiting factor in guided online learning.
Hopper
Halfcheetah
Walker2D
IDM Loss
IDM Loss
IDM
Timestops
Timestops
Timestops
Figure 4: A look at the IDM loss function during online training
Table 13: Comparison of final online IDM loss and expert IDM loss. The online IDM loss is the of two terms which makes it approximately twice large the expert IDM loss. This shows that the online IDM loss converges to a similar performance as the expert IDM by the end of online learning. Results are averaged over 5 seeds + s.d.
Dataset
Final expert IDM loss
Final online IDM loss
Hopper
0.13
0.22 1 0.022
HallCheetah
0.12
0.196 + 0.028
Walker2D
0.12
0.21 + 0.04
We that the IDM is included solely lightweight translator from (s, As actions for the purpose of online guidance, and is not part of OSO-DecQN itself. The loss curves here and 3- sensitivity results (in Section 3 reported here demonstrate that our improvements are robust to variations in IDM learning speed and scheduling, reinforcing that the core performance gains stem from the offline state policy rather than the IDM's specific architecture optimisation.
F GUIDANCE WITH SAC
In Figure we demonstrate that our method enhances online learning when combined with SAC, compared to SAC without guidance, using the same hyperparameters as Table [5] Since the hyperparameters were tuned primarily for TD3, our goal was to test whether the method can still guide SAC without additional tuning, underscoring robustness.
Unlike TD3, which rapidly learns optimal behaviour in the Hopper environment even without guidance, SAC learns more slowly, making the benefits of guidance in accelerating learning more pronounced. Consistent with the results for TD3, we also observe that guidance improves both convergence speed and asymptotic performance in HalfCheetah and Walker2D when applied with SAC.
G COMPARISON TO OFFLINE-TO-ONIEEE METHODS WITH ACTION LABELS.
In Figure we compare DecNN to strong dffline-to-onlnne baselines that use action labels (O3F (Mark et al. 2022). IQL et 2021). CAL-QL (Nakamoto et al. 2023), AWAC (Luo et al.) 2023)) on Walker2D. As a result of training an action policy offline offline- tonne-tt--oline methods start online fine-tuning with much higher initial performance than guidance with state-only datasets.
22

---

Published as a conference paper at ICLR 2026
Halfcheetah
Walker2D
Hopper
Normalised returns
Normalised returns
Normalised returss
Dw..oooChh medtum
Timestops
Timesto
Timostops
Figure 5: Online learning curves on D4RL, each with corresponding legend. We compare our method of guided learning to the performance of an online agent trained using SAC, without guidance over 1M timesteps. The solid line corresponds to the mean normalised return across seeds with the shaded area corresponding to standard deviation away from the mean.
walker2d-expert
walkerr2d-medlumexxpett
walker
Normalised returns
Normalised returns
Normalised
Timesteps
Timesteps
Timesteps
walker2d meedu-raam
Norrallseed returns
122
Timesteps
Figure 6: Normalised online returns Walker2D for different dataset qualities (expert, medium- expert, medium, medium-replay) We compare our action-free method OSO-DecQN, which is pre-trained only state-only data, offline-to-online baselines that use full action labels (O3F (Mark et al., (022), IQL(Kostrikov et al. (02T), CAL-QL (Nakamoto et al. 2023), AWAC (Luo et al.) 2023)). Curves show the mean over 5 seeds; shaded regions indicate steps.
over 250k training
H COMBINED COMPARISON OF AF-GUIDE AND OSO-DECQN.
Figure 7 overlays the learning curves AF-Guide, TD3 and OSO-DecQN showing that our action- free method consistently outperforms AF-Guide.
T ABLATION EFFECT OF IDM BOOTSTRAPPING TERM.
Table 14 reports an ablation of the IDM bootstrapping term. We compare our full method, which includes the additional loss on offline (s, s') pairs using actions from the current policy, with variant trained without this term. Result are normalised returns (mean + s.e over 5 seeds). The full method generally attains better or comparable final return, yielding higher total average return (115.8 vs 112.7). This indicates the bootstrapping provides useful extra signal that improves long-term performance.
LLM USAGE
We L.LMs in work solely for grammatical purposes.
23

---

Published as a conference paper at ICLR 2026
Happer
Halfcheetah
Walker2D
Normallsed returns
Normalises returns
Timesteps
Timesteps
Timesteps
Figure Normalised online returns (mean + s.e. over seeds) for Hopper, HalfCheetah and Walker2D across different dataset qualities. We plot AF-Guide, TD3 and OSO-DecQN on the same axes. OSO-DecQN consistently achieves comparable or higher final performance than AF--Guide.
Table 14: Effect of IDM relabelling (bootstrapping) final performance. Normalised returns (mean + s.e. over seeds) for OSO-DecQN with the full IDM loss and for an ablated variant without the bootstrapping term.
Dataset
OSO-DecQN (full)
OSO-DecQN (w/o bootstrapping)
Hopper
-medium-replay
111.4 + 0.79
113 + 0.48 88.1 +
-medium
113.2 + 1.5
-medium-expert
113.1 0.46
109.3 2.5
-expert
111.4 + 1.7
114.2 = 0.64
Halfcheetah
-medium-replay
112.4 4.1
108.6 1 0.61
-medium
116.4 4.7
108.9 + 1.6
-medium-expert
4
119.3 = 1,4
-expert
113
117.6 2.2
Walker2d
mediuum-repla
123.3 5.4
3.1 118.1 = 2.4
medium
120 = 5.1
-ediium-expee eeppert -expert
112.6 + 1.5
113.2 2.9
120.7 + 2.3
121 + 1.4
Cheetah- Run
121.5 +
124.7 +7
-medium -medium-expert -expert
120 = 4.4
124.1 + 3.5
132,7 + 1.43
115.5 + 10.2
124.2 1 4.9
109.9 1 10
Quadruped-Wai Walk candom-medium-exppe
114,2 + 5.3
111.8 - 6,9
-medium -medium-expert
87.4 + 13.2
95.6 4: 13.4
117.5 + 2.7
106.9 - 7.9
-expert
117.7 + 1.4
117.2 +
Total Average
115.8
112.7
24


Reviews:
[{"review_id": "EcwwfKxw7h", "reviewer": "Reviewer_cXZg", "summary": "This paper proposes an algorithm for learning value functions from action-free datasets in an offline manner and using these value functions to improve the learning speed in a subsequent online phase. The key parts are discretization method for the state space to assist learning in the offline phase. Experiments with standard offline RL datasets (removing action information) demonstrate the utility of the method.", "questions": "- Why is DecDQN specifically chosen tobe used rather than a simpler DQN variant given that the method seems to be agnostic to the value-based learning algorithm?\n\n\n- For the IDM's loss, the $L_1$ loss is used. Why not use the Huber loss if robustness is an issue? \n\n- How does the performance of the method compare to offline-to-online methods that do have access to action information in the offline dataset? We would expect worse performance of course but it would be interesting to see how large the gap really is, it may turn out to be quite small.\n\n\n- Concerning the discretization step for the state space, it has been found for value learning that regression and predicting a continuous output is inferior to using classification-style losses (the histogram loss) [1]. Have you tried using the histogram loss instead? The loss converts continuous regression targets into a classification target. Then predictions can be done by outputting probabilities over discrete outputs in the support and taking an average. It can be used with a variable number of support points.\nThis method could also potentially be used for the inverse dynamics model.\n\n\n- Line 277: Why do we need to use generated actions from the policy to relabel transitions in the offline dataset with the current policy to train the IDM? This would seem to induce a feedback loop since the IDM is trained on its own actions. Could an ablation be run on including this part or not?\n\n\n- Is the $\\epsilon$ hyperparameter in the discretization important? In the appendix, it indicates it is set to be 1e-4 which seems quite small but an ablation showed that this was important to include for some environments.\n\n- How important is the policy-switching strategy? How sensitive is it to the $\\beta$ parameter? \n\n- Does the $Q(s, \\Delta s)$ value network also get updated during the online phase?\n\n\n- It is fairly surprising that an inverse dynamics model would be able to use the $\\Delta s$ discretized state change effectively since there would be many actions that could map to the same discretized state difference. Why do you think this is effective? Is this exploiting some structure specific to simulated robotics tasks? \nAn experiment that would help clarify this is to test whether following the action given by the IDM actually leads to the expected dicretized state change. Verifying that the IDM can do this would be interesting.\n\n\nMinor points (not impacting the score):\n- The notation for the discretized state is introduced as $\\delta(s,s')$ but in other places it is $\\Delta s$ that is used. \n\n- In Theorem 1, M is used to denote the number of bins but it is also used to denote the MDP. It would also be clearer to include the definition of $M$ and $H$ in the theorem statement. \n\n- Table 1: The font is a bit small. \n\n[1] Stop Regression: Training Value Functions via Classification for Scalable Deep RL", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The problem setting is interesting, action-free datasets would seem to be more common in practice and understudied.\nThe discretization technique for the state space and converting to actions using an IDM is simple and surprisingly effective.", "weaknesses": "Generally, there are no glaring issues but some experimental and design choices are unclear to me.\nI have included these questions below in the \"Questions\" section.\nMany of these are clarification questions or ablation suggestions and I am happy to increase my score after having more information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hExCsFOLDv", "reviewer": "Reviewer_cTFm", "summary": "In this paper, the authors formalise the setting of action-free offline-to-online RL, where agents must learn from datasets consisting solely of (state, reward, next state) tuples and later leverage this knowledge during online interaction. To address this challenge, they propose learning state policies that recommend desirable next-state transitions rather than actions. First, the authors introduce a simple yet novel state discretisation transformation and propose Offline State-Only DecQN (OSO-DecQN), a value-based algorithm designed to pre-train state policies from action-free data. OSO-DecQN integrates the transformation to scale efficiently to high-dimensional problems while avoiding instability and overfitting associated with continuous state prediction. Second, they propose a novel mechanism for guided online learning that leverages these pre-trained state policies to accelerate the learning of online agents. Together, these components establish a scalable and practical framework for leveraging action-free datasets to accelerate online RL. Empirical results across diverse benchmarks demonstrate that their approach improves convergence speed and asymptotic performance, while analyses reveal that discretisation and regularisation are critical to its effectiveness.", "questions": "Please see the weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation for action-free RL is strong, while the explanation for the motivation is clear in the paper.\n2. The authors propose an algorithm with novel technical designs.\n3. The experimental results are generally good. The ablation study part is helpful.", "weaknesses": "1. The paper directly decouples the Q function. Under the case where the real Q function depends on the correlation between different dimensions, could this approximation perform well? Meanwhile, for the argmax of the action, I am wondering what is the choice when $\\mathcal{A}$ is not a product space. For instance, if $\\mathcal{A}$ is a unit ball, how to define the argmax on each dimension?\n\n2. For the training loss of IDM, could you please provide some motivation for that? Does the success of such approach implicitly depend on the Lipschitzness of the Q value function over actions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KLIcezSp7H", "reviewer": "Reviewer_KioB", "summary": "This paper proposes a new offline-to-online reinforcement learning (RL) framework designed for action-free offline datasets. The authors introduce a discretized state representation that serves as an action surrogate, enabling the training of a discretized Q-function purely from state transitions. During online learning, an inverse dynamics model (IDM) is trained to map the predicted next-state differences into executable actions. A policy-switching strategy blends the IDM-based actions with the online policy, while a regularization term is used to stabilize and constrain the offline Q-function.\nExperiments on D4RL and the Action-Factorized DeepMind Control Suite demonstrate faster convergence and better asymptotic performance for both TD3 and DecQN baselines. Ablation studies further show that both discretization and regularization are crucial for performance.", "questions": "1. Are there plans to extend this framework to visual or multimodal inputs, where the state-space dimension exceeds 1000?\n2. Would the method still be effective if the offline dataset and the online environment come from slightly different dynamics distributions?\n3. How many offline data samples are used in training, and how does the amount of offline data influence the overall training stability and performance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is conceptually clear and technically sound, addressing a problem where offline datasets lack action labels.\n2. The methodology is well-designed, combining existing offline RL techniques with novel discretization and regularization components in a coherent way.\n3. The ablation studies are comprehensive and clearly demonstrate the contribution of each component.", "weaknesses": "1. The motivation for the action-free offline data setting is not clearly justified. The paper should better explain when and why such data would realistically occur.\n2. The examples in the introduction are not entirely convincing. The most plausible application would be robotics from video, but the experiments only consider relatively small state spaces (up to 78 dimensions), which limits the realism of the claim.\n3. The baseline coverage is limited — the paper primarily compares against a single online RL algorithm (TD3) and lacks comparison with stronger modern offline-to-online or action-free RL baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YsfBx7hssn", "reviewer": "Reviewer_5mWV", "summary": "This paper addresses learning from datasets that lack action labels, containing only (state, reward, next-state) tuples. It proposes OSO-DecQN, which learns policies over discretized states offline and uses them to guide online learning through an inverse dynamics model (IDM).", "questions": "1. Zhu et al. 2023 [1] (AF-guide) seems to adopt different approaches for both offline and online training. In Figure 2, is the online and offline training of Af-guide entirely following Zhu et al. 2023? If so, can you provide results for the case of Af-guide (offline) + your method (online) and your method (offline) + Af-guide (online)? This would give a more fine-grained performance comparison of each training process (online/offline).\n\n[1] Deyao Zhu, Yuhui Wang, J ̈urgen Schmidhuber, and Mohamed Elhoseiny. Guiding online reinforcement learning with action-free offline pretraining. arXiv preprint arXiv:2301.12876, 2023.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper tackles practical scenarios where data has no action labels by proposing an offline policy training method that treats a discretized state change as an action and applies offline Q-learning, and demonstrates its effectiveness through various D4RL tasks.\n\n1. The work provides a solution to further improve models pretrained with offline data via online learning by incorporating an IDM (inverse dynamics model), which maps a state change to an action.", "weaknesses": "1. Limited comparison with other action-free methods. It seems that in Figure 2, it only compares with the case without pre-training on offline datasets (TD3) and the action-free method (Zhu et al., 2023 [1]). However, given there are more action-free offline training methods [2,3,4], it would be better to compare with all these baselines to fully validate the effectiveness of the proposed action-free training based on state discretization.\n\n2. The presentation of Figure 2 is confusing because although it is for comparing with another action-free offline method (Af-guide), it doesn't directly compare the proposed one and Af-guide. Instead, it is comparing with another baseline (TD3) and indirectly claims that the proposed one is better than AF-guide because it is better than TD3, which I found confusing. It would be better to put all algorithms to be compared in the figure. Also, I wonder if the training/evaluation setup for your method, Af-guide, and TD3 is the same for all experiments.\n\n3. The state discretization seems coarse, which could lose a lot of meaningful state transition information depending on state representation and tasks.\n\n[1] Deyao Zhu, Yuhui Wang, J ̈urgen Schmidhuber, and Mohamed Elhoseiny. Guiding online reinforcement learning with action-free offline pretraining. arXiv preprint arXiv:2301.12876, 2023.\n\n[2] Bohan Zhou, Ke Li, Jiechuan Jiang, and Zongqing Lu. Learning from visual observation via offline pretrained state-to-go transformer. Advances in Neural Information Processing Systems, 36: 59585–59605, 2023.\n\n[3] Hao Luo, Bohan Zhou, and Zongqing Lu. Pre-trained visual dynamics representations for efficient policy learning. In European Conference on Computer Vision, pp. 249–267. Springer, 2024.\n\n[4] Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with actionfree pre-training from videos. In International Conference on Machine Learning, pp. 19561–19579. PMLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QiEI2fNRLi", "reviewer": "Reviewer_EJh5", "summary": "This paper studies a novel and practically motivated setting in reinforcement learning (RL): action-free offline-to-online RL, where the agent must learn from datasets containing only tuples of the form state-reward-next state datasets, without action labels. Such a setting arises naturally in domains like healthcare, finance, and robotics, where action logs may be unavailable due to privacy, storage, or sensor constraints. The paper asks: Can an agent learn useful knowledge from such datasets and transfer it to online learning?\nTo address this, the authors propose a two-stage framework built around a new offline algorithm, Offline State-Only DecQN (OSO-DecQN), that learns state policies rather than action policies. Instead of predicting actions, the algorithm predicts discretised state differences (i.e., the direction of state change) and uses these predictions to guide online RL. \nEmpirically, the authors show that OSO-DecQN pre-trained on action-free datasets can accelerate online learning and improve final performance across a range of continuous- and discrete-control tasks (D4RL, DeepMind Control Suite), outperforming existing action-free baselines (e.g., AF-Guide by Zhu et al., 2023). Ablation studies further highlight the importance of discretisation and regularisation, and theoretical results (Theorem 1–2) provide a discretisation-dependent value approximation bound.", "questions": "1.\tSince the annealed beta schedule significantly affects results (Fig. 3), analysing it in terms of exploration-exploitation balance could clarify its effect.\n2.\tIt would be better to show a correlation between IDM prediction error and final online performance to support the claim that IDM design is not a major factor.\n3.\tIt would be better if the paper includes comparison of total pretraining + online runtime versus baseline methods to substantiate claims of scalability.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper clearly identifies a real-world gap between existing offline RL methods (which assume full action observability) and practical domains where actions are missing. The authors articulate why action-free learning is challenging and outline an elegant conceptual framework that combines discrete state prediction with conservative regularisation and online guidance. This direction feels both original and valuable for RL research, especially as large unlabelled state datasets become more common.\nThe proposed OSO-DecQN is conceptually simple yet technically grounded. Overall, the algorithm is a clean extension of DecQN into an action-free paradigm.\nThe authors provide a series of formal results bounding the approximation error introduced by state discretisation. In particular, Theorem 1 and Theorem 2 derive the difference between the original and discretised optimal value functions, showing that the discretisation granularity controls the value loss. The proofs (Appendix B) are rigorous and grounded in classical contraction and KL-divergence arguments, enhancing the credibility of the approach.", "weaknesses": "Although the integration of discretisation, conservative regularisation, and decoupled Q-learning is novel in the action-free context, each component individually builds on well-known ideas (DecQN, CQL-style penalties, inverse dynamics models).\nThe contribution is therefore more conceptual unification than a fundamentally new RL mechanism.\nWhile the paper claims the IDM is lightweight, its reliability is central to the online guidance mechanism. The IDM must map predicted state differences back to executable actions accurately; errors in this translation could affect results. Although the authors show robustness across architectures (Table 10–11), a more direct comparison of IDM accuracy versus final online performance would better quantify this dependency.\nThe theoretical analysis focuses entirely on the offline discretisation error.\nThere is no corresponding guarantee or convergence analysis for the guided online learning phase (e.g., stability of β-switching or IDM-based guidance).\nSome formal justification, perhaps connecting it to off-policy improvement or safe exploration, would make the work more complete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)