State-Constrained Offline Reinforcement Learning
Authors: Charles Alexander Hepburn
Yue Jin
Giovanni Montana
Venue: Accepted by TMLR ()

Abstract: Traditional offline reinforcement learning (RL) methods predominantly operate in a batch-constrained setting. This confines the algorithms to a specific state-action distribution present in the dataset, reducing the effects of distributional shift but restricting the policy to seen actions. In this paper, we alleviate this limitation by introducing state-constrained offline RL, a novel framework that focuses solely on the dataset’s state distribution. This approach allows the policy to take high-quality out-of-distribution actions that lead to in- distribution states, significantly enhancing learning potential. The proposed setting not only broadens the learning horizon but also improves the ability to combine different trajectories from the dataset effectively, a desirable property inherent in offline RL. Our research is underpinned by theoretical findings that pave the way for subsequent advancements in this area. Additionally, we introduce StaCQ, a deep learning algorithm that achieves state-of-the-art performance on the D4RL benchmark datasets and aligns with our theoretical propositions. StaCQ establishes a strong baseline for forthcoming explorations in this domain.

Published in Transactions on Machine Learning Research (05/2025)
State-Constrained Offline Reinforcement Learning
Charles A. Hepburn Yue Jin?, Giovanni Montana 2.3,4
Mathematics Instätute, University of Warwick, Coventry, UK
Warwick Manufacturing Group, University of Warwick, Coventry, UK Department of Statistics, University of Warwick Coventry, UK Alan Turing Instttute, London, UK
(charlie.hepbung yue.jin.3, g.monlana) @warwick..c.coo
Reviewed on OpenReview: https: openreview. net/ forum? id= (d=XcB8ykFF..
Abstract
Traditional offline reinforcement learning (RL) methods predominantly operate in batch- constrained setting. This confines the algorithms to a specific state-action distribution present in the dataset, reducing the effects of distributional shift but restricting the policy to seen actions. In this paper, we alleviate this limitation by introducing slate-constrmined offline RL, novel framework that focuses solely on the dataset's state distribution. This approach allows the policy to take high-quality out-of-distributiin actions that lead to in- distribution states, significantly enhancing learning potential. The proposed setting not only broadens the learning horizon but also improves the ability to combine different trajecto- ries from the dataset effectively, a desirable property inherent in offline RL. Our research underpinned by theoretical findings that pave the way for subsequent advancements in this area. Additionally, we introduce StaCQ,, a deep learning algorithm that achieves state- of-tho-art performance on the D4RL benchmark datasets and aligns with our thcoretical propositions. StaCQ establishes strong baseline for forthcoming explorations in this do- main.
1 Introduction
Offiine RL aims to derive an optimal policy solely from a fixed dataset of pre-collected experiences, without further interactions with the environment (Lange et al. 2012 Levine et al. 2020). It prohibits online exploration, meaning that effective policies must be constructed solely from the evidence provided by an unknown, polentially sub-optimal behaviour policy active in the environment. This approach especially suited for real-world scenarios where executing sub-optimal actions can be dangerous, time-consuming, or costly, yet there is abundance of prior data. Relevant applications include robotics (Singh et al.
2020:
Kumar et al. 2021 Sinha et al. 2022), long-term healthcare treatment plans (Tang & WWinn
2021
Tang
et al. 2022 Shiranthike et al. 2022), and autonomous driving (Shi et al. 2021 Fang et al. 2022 Diehl et al.
2023 1 Despite its practical appeal, offline RL faces a significant distributional shift challenge (Kumar et al.
2019
) This phenomenon arises when attempting to estimate the values actions not present in the dataset,
often manifesting as an overestimation. As a result, out-o--disttibutiin (OOD) actions are perceived as more valuable than they truly are, causing the agent to select sub-optimal actions and leading to the accumulation of errors (Fujimoto et al. 2019).
Currently available offline RL techniques mitigate the issue of distributional shift through two primary strategics. The first constrains the policy, during training, to take actions close to datasct actions (Fujimoto
ee al. 2019 Kumar et al.] 2019 Wu et al., 2019 Siegel et al.] 2020 Kostrikov et al.] 2021a Zhou el al.,
2021 Fujimoto & Gu| 2021 ). The second revolves around conservatively estimating the value of OOD actions
[Kumar et al. 20200] Yu et al. 2021 An et al. 2021). Both strategics aim to anchor the learning algorithm
the state-action pairs within the dataset, an approach termed catch-constrained (Fujimoto et al. 2019). By doing so, these methods seek to minimise the impact of distributional shift. However, adhering strictly to the

---

Published in Transactions on Machine Learning Research (05/2025)
State
Action
a
Reachable state space
SG
Reachablc states
S Full state space
S
Figure 1: Illustration of state reachability in continuous state space. States (s) and next states (s') from the dataset are shown, with yellow circles representing additional reachable states. The state-constrained method identifies high-quality reachable states to mitigate distributional shift while improving policy performance.
state-action distribution can be restrictive when the optimal action is not found in the dataset. Thus, recent techniques seek balance between exploiting potentially valuable OOD actions and minimising distributional shift repercussions.
Given the constraints of the statc-action distribution, a pivotal question arises: can we confine our methods to the state distribution alone and still counteract distributional shift??
Constraining exclusively to the dataset's state distribution could significantly reduce the requisite dataset size. Instead of needing all state-action pairs, the focus shifts to states alone. This approach allows the agent to initiate OOD actions, provided they lead to known state, a form of safe action exploration which avoids distributional shift. Such flexibility can enhance the capability of offlie RL algorithms in trajectory stitching combining sub-optimal experiences into improved trajectories. Rather than connecting based on actions from different trajectories, the state- constrained approach facilitates stitching by leveraging adjacent states. This method relies on understanding these proximate states, a concept we label as reachability, which refers to the ability reach particular state from another state within the dataset.
This idea is illustrated in Figure 1 which shows the benefits of a stale-consttrineed methodology. Most current offline RL methods constrain to the specific transition, (s, or s')air,,,, found in the dataset. Our state-constrained mothodology can instead find alternative next states (yellow circles in Figure D and select the highest-value one. This approach effectively increases the number of transitions for the agent to learn from. It avoids distributional shift by updating on in-distribution states and leads to higher-quality policy, as the policy is no longer restricted to explicitly observed actions.
In the state-constrained approach, understanding reachability is crucial for effectively stitching together sub- optimal trajectories and improving the overall policy. Tn some environments, reachability is easily delineated; for example, in grid-based environments like mazes, where the agent can move directly between adjacent cells. In other environments, reachability must be ascertained from the dataset, such as in complex robotic systems where state transitions depend on intricate dynamics.
In this work, we make several key contributions. First, building on the concept of reachability, we introduce state-constrained offline RL. Unlike batch-constrained RL, which anchors learning to specific state-action pairs, saate-constrained RL focuses states, allowing more flexible action selection while still mitigating distributional shift. We also provide thcoretical convergence guarantees under deterministic assumptions, showing that our framework yields policies whose actions are of higher or equal value compared to batch- constrained approaches. Another main contribution StaCQ, a novel deep learning algorithm that learns a state-constrained value function and updates the policy to stay close to the highest-quality reachable states. We demonstrate competitive performance on multiple D4RL tasks, surpassing many state-of of the-art methods in locomotion and Antmaze. This positions StaCQ robust baseline for future work state-constraines
2

---

Published in Transactions on Machine Learning Research (05/2025)
RL, similarly to how BCQ FFjimoto et al. 2019) has served as strong baseline for batch-constrained techniques, thereby advancing offline RL, research.
2 Preliminaries
The RL framework involves an agent interacting with an environment, which is typically represented as a Markov Decision Process (MDP). Such an MDP is defined as M = (S, P. R),,, where S and A denote the state and action spaces, respectively; P = p(s''ss,a a) represents the environment transition dynamics; R- r(s, ((s.0 a. s') is the reward function for transitioning; and 0 S 1 is the discount factor (Sutton & Barto 2018). As in Fujimoto et al. (2019), we focus on the deterministic MDP, where p(s'|s,a) [0,1] and rss a, 8¹) 8') r(,=='','') In RL, the agent's objective is to identify an optimal policy, x()), that maximises the future discounted sum of rewards Ex=r"' "ir(s,, s().
QSA-values, Q(s,a) = E, T"r 0z)|su = s, 3o = al. are the expected sum of future discounted rewards for executing action a in state 8 and there after following policy æ. Q-learning (Watkins & Dayan 1992), denoted as QSA-learning in this paper, estimates the QSA-values under the assumption that future actions are chosen optimally,
(1- + + aar(s,,,ss)) max Q(ss
(1)
The optimal policy derived from QSA-learning identifies the best action that maximises the QSA-values, (s) arg max, Q(s, a). In deterministic contexts, QSA-learning parallels QSS-learning (Edwards et al. 2020), which estimates the expected rewards upon transitioning from state to 8', followed by optimal decisions:
((s,s') + (1 (1-a)((sss 8') + a ([r(ss,))) +ymax Q (())
(2)
QSS-lcarning enables QSS-values to be learned for transitioning, without evaluating actions. The optimal policy via QSS-learning discerns the most advantageous subsequent state to maximise QSS-values, (5) arg max, Q(sss').. An action can then be retrieved from inverse dynamics model strategy originally designed to address QSA-llarning's challenges in redundant action spaces, i.e. where multiplc actions lead to the same next state. The methods presented in this paper build upon QSS-learning, using it to avoid evaluating OOD actions between reachable state pairs.
I(s, (s)),
In offline RL, the agent aims to discover an optimal policy; however, it must solely rely on a fixed dataset without further interactions with the environment (Lange al.) 2012 Levine al. 2020). The dataset comprises trajectories made up of transitions consisting of the current state, action, next state, and the reward for transitioning, where actions are chosen based on unknown behaviour policy, тT. Batch- Constrained QSA-learning (BCQL) (Fujimoto et al. 2019) adapts QSA-learning for the offline setting by restricting the optimisation to state-action pairs present in the dataset:
max + (alssa)+(f(((s0))) a') (' ad),,a))
(3)
This approach is restrictive since convergence the optimal value function in all states only ensured if every optimal (s, pa-ir from the MDP resides within the dataset. Such limitation aims to sidestep extrapolation errors resulting from distributional shifts. Nonetheless, this framework places significant constraint any learning algorithm. Subsequent sections relax this constraint: instead of adhering strictly to state-action pairs (batch-constrainee), the learning process is only bound by states (state-constrained) In the following section, we introduce the stale-constrained framework and provide new algorithm called state-constrained QSS-learning (SCQL). SCQL is based from QSS- QSS-learning which useful to evaluate state and reachable next state pairs, where both states exist in the datasct, avoiding the distributional shift issuc in offlie RL. We show that, under minor assumptions, SCQL converges to the optimal QSS-value and produces less restricvtee policy than BCQL.
3

---

Published in Transactions on Machine Learning Research (05/2025)
3 State-constrained QSS-learning
In this section, we provide formal introduction to state-constrained QSS-lcarning and establish its conver- gence to the optimal QSS-value under a set of minimal assumptions. Within this framework, the learning updates are restricted exclusively to the states present in the dataset. rigorous definition of state reacha- bility is indispensable for this setting:
Definition 3.1. (State reachability) In a deterministic MDP M, a state s' is considered reachable from state and only if there exists an action a such that p(s"(s, = 1. We denote the set states reachable from 8 as SRM( (8), where E (s).
Definition 3.1 implies that a state reachable if there exists an action that, when executed the environment, leads to that state. This definition allows multiple reachable next states to be evaluated rather than a single state-action pair. Stale-constrained QSS-learning enables more flexible learning updates by evaluating multiple reachable next states, as shown in Figure where the agent can now learn from all reachable states rather than the single explicit next state. This flexibility leads to more robust policies that can better generalise beyond the specific transitions seen in the dataset.
where
3.1 Theoretical foundations
In this section, initially adapt the theorems presented for BCQL (Fuiimoto al. 2019 to suit the state- constrained context (Theorems 3.4 3.6 All our subsequent theorems are proposed based on learning an optimal QSS-value. However, it is important to note that these theorems still hold if QSA-learning is used instead, with an inverse model defined to evaluate actions between states and reachable next states.
Our theory operates under the following assumptions: (A1) the environment is dcterministic; (A2) the rewards are bounded such that (r(,ss))((s,s S c; (A3) the QSS-values, Q(s,s'), are initialised to finite values; and (A4) the discount factor is set such that 0 2 1.
First, we show that under these assumptions, QSS-lcarning converges to the optimal QSS-value. Theorem 3.2. Under assumptions A1-4, and with the training rule given by
max Q(s, + r(s, ( +7 8² s.t. (( Q 8"),
(4)
assuming each ss, pair is visited infinitely often, let Qn s') be the value from the nihh update of Eg. then (n(s,s') converges to the optimal QSS-value, (" s'), n for all s'.
Proof. This follows from the convergence of QSA-learning in deterministic MDP MMitchell 1997). For brevily and clarily, the full proof is given in the Appendix.
The convergence of QSS-learning in a deterministic MDP, as shown in Theorem 3.2 is crucial for establishing the convergence of SCQL. Since SCQL is based on QSS-learning and operates in deterministic MDP, Theorem 3.2 provides the foundation for proving the convergence and optimality of SCQL under certain assumptions. We now demonstrate that learning the value function from the dataset D is equivalent to determining the value function of an associated MDP, denoted as Ms. Intuitively, Ms is the MDP where all transitions are possible between reachable states found in D.
Definition 3.3. (State-constrained MDP) Let the state-constrained MDP Ms (S R, Here, both and A remain identical to those in the original MDP, M and Sterminal an additional terminating stale. The transition probability is given by: a = (s,,8)) 6
ps( = -(0
if (s, E D and s' E (R)(ss) or (s $ D and s' Sterminal) otherwise.
The reward function and discount factor remain the same as the original MDP. Except for the terminal state where rss, Sterminal), is set to the initialised value of Q(,, 8').
4

---

Published in Transactions on Machine Learning Research (05/2025)
The transition probability ps(s)(s,a) a) in the state-constrained MDP is defined such that transitions are possible only between states that are reachable from one to the other according to the state reachability definition. This ensures that the state-constrained MDP captures the essential dynamics of the original MDP while focusing on the states present in the datasct. The reward r(s, s') is assigned as the original reward defined in M. For the case where 8 absent from the dataset, the rewards are set the initialised values ('),s). Importantly, the and s' in Definition 3.3 both cxist the dataset but may not exist as a pair (s, 8'); this means that more transitions exist under this definition than in the batch-consttained MDP defined in Fujimoto et al. (219
Theorem 3.4. By sampling 8 from D, sampling & from SR)(() and performing QSS-learning on all reachable state-nert state pairs, QSS-learning converges to the optimal value function of the state-construined MDP Ms.
Proof. Given in Appendix
Theorem 3.4 establishes that performing QSS-lcarning all reachable state-next state pairs from the dataset converges to the optimal value function of the state-constrained MDP. This result is crucial for understanding the convergence and optimality properties of SCQL, as it shows that QSS-learning elfectively learns the optimal value function of the state-constrained MDP, which is closely related to the original MDP.
We are now ready to define the ttaat-constraineee QSS-learning (SCQL) update which similar to the BCQL formulation except now the maximisation is raistr to the states rather than state-action pairs in the dataset. This formulation allows the maximisation be taken over more values composing more accurate Q-values while still staying close to the dataset:
(1 + aa) +7
max s" ".s. 8"ESRM('')
Q(d',s")
(5)
SCQL, Eq. (5), converges under the identical conditions as traditional QSS-learning, primarily because the state-constrained setting is non-limiting whenever every state in the MDP observed.
Theorem 3.5. Under assumptions A1-4 and ossuming every state s is encountered infinitely, let Qu(s, s') be the value from the nth update of Eq. 5), the update rule of SCQL, then Qn(s, converges to the optimal QSS-colue (s, as n + oo for all ss,s").
Proof. This follows from Theorem 3.2 noting the state-constraint is non-restrictive with a datasct which contains all possible states.
Theorem 3.5 is a reduction the restriction compared to BCQL as now we only require every state to be encountered infinitely rather than every (s, -pair.
The optimal policy for our state-constraiindd approach can be formulated as:
(s) arg max (" S. s'ESR((s)
(6)
Here, the maximisation is laken over next stales from the dataset and that are reachable from the current state. We now demonstrate that Eq. 66) represents the optimal state-constrained policy.
Theorem 3.6. Under assumptions A1-4 and assuming every state s is encountered infinitely, let Qn(s, 8') be the value from the nth update of (5. , the update rule of SCQL, then Qn(s,s) converges to Q3(s, s'), the oplimal QSS-value compuled from states from D, with the optimal stale-comstruined policy defined by Eq. c where s E D and s' E =SRM(s) N..
Proof. Given in Appendix
A
5

---

Published in Transactions on Machine Learning Research (05/2025)
Theorem 3.6 has important practical implications for the performance of SCQL in real-world scenarios with limited datasets. It suggests that SCQL. can learn an optimal state-constrainnd policy even when the dataset does not contain all possible state-action pairs, as long as every state is visited infinitely often. The state-constrained approach greatly reduces the limitations placed on the learning algorithm compared to the batch-constrained method. In the raiiie approach, the necessity for every state-action pair to be present in the dataset demands dataset of size at least [S] x |A]. On the other hand, the state-constrained method only mandates that each state be visited, which minimally requires a dataset size of [S].
We will now show that BCQL is special case of SCQL, and that policies produced by SCQL will never be worse than policies produced by BCQL.
Theorem 3.7. Let BBQL and EsCQL be the policies produced by BCQL and SCQL respectively. Then, in a deterministic MDP, Vasccu (s) 2 V Vazcau(s), Vs E.
Proof. Given Appendix A
From Theorems 3.6 and 3.7 the state-constraiind approach poses fewer restrictions than the batch- constrained counterpart while also always producing a policy least as good. SCQL can perform more Q-value updates and maximise over more states than BCQL because it considers all reachable stxte-next state pairs from the dataset, rather than being limited to the explicit state-action pairs present in the datasel. This allows SCQL to exploit the structure of the state-constrainet MDP more effectively and learn better policy, with less required data. In the next section, this improvement is illustrated through an example which shows SCQL excelling even with a limited dataset.
3.2 An illustrative example: maze navigation
To elucidate the advantages of SCQL in comparison to BCQL, we explore a simple maze environment backed by a dataset with few trajectories. This pedagogical illustration aims to show how SCQL utilises the concept of state reachability which enables improved performance over BCQL (which has no mechanism to utilise state reachability). Figure 2 shows the full maze environment, visualised a 10 by 10 grid. The available states are shown as white squares, while the red squares represent the maze walls. Each state in this environment is denoted by the (I, coordinates within the maze, and the agent's available actions comprise of simple movements: left, right, up, and down. With each movement, the agent incurs a reward penalty of a small negative value (rpen = -0.1), but upon successfully navigating to the gold star, it is rewarded with a large positive value (Fgoal = 10). Consequently, the agent's 's primary objective becomes devising the most direct route to the star. In this deterministic environment, the outcomes of transitioning a state and executing an action are congruent, meaning Eq. (T) and (2) mirror each other. This allows for a direct comparison between BCQL and SCQL in terms of their performance and ability to leverage limited datasets. This experiment converges after 100 steps with a learning rate of a 0.25 and a discount factor of 7 0.99.
Figure 2a provides a visual representation of the maze and accompanying dataset. Here, trajectories start at the white circle and end at the black cirele. Despite its simplicity, the dataset contains a scant number of states wherein multiple actions have been taken. Within this specific framework, the notion of state reachability is straightforward: any state that is just one block away from the current state is considered reachable. Formally the state reachability is, SRM((1 y)) {(x + 1, (x 1) (x, + 1), (ær -))-- Although this definition includes the wall states reachable, SCQL can only perform updates where s' is reachable and in the dataset, thus the walls are avoided. In other words, a state is considered reachable from the current state if it is one block away in any of the four cardinal directions (left, right, up, or down). This simple definition of state reachability is sufficient for demonstrating the advantages of SCQL in this illustrative example.
Applying BCQL with training as per Eqq.
3 and policy extraction as per
n
max Q(+, a),
s.t. (s.0)D
we obtain the policy depicted in Figure D where arrows indicate policy direction. Conversely, for SCQL, training via Fq. (5) and policy training through Eq. ( yields the policy in Figure 2 For clarity, if
6

---

Published in Transactions on Machine Learning Research (05/2025)
(a) Maze and Dataset
(b) BCQL Final Policy
(c) SCQL Final Policy
Figure 2: Comparison of BCQL and SCQL methods simple maze environment. (a) The maze is 10 by 10 grid where the coordinate values (z, y) represent the state. The high reward region is represented by star and maze walls are represented by red grid squares. The dataset is made of trajectories represented by the dotted lines where the white circle is the starting state and the black circle the final state. (b) The final policy when applying BCQL to the dataset. (c) The final policy when applying SCQL to the dataset.
no optimal movement can be found in the state due to no information in the dataset, it left blank. SCQL able to leverage the sparse dataset more efficiently, enabling the agent reach the gold star from any dataset state position. BCQL, however, succeeds in only 11 of the 57 unique datasel states, which is equivalent to the performance of behavioural cloning. SCQL's advantage stems from its ability to use state adjacency information for trajectory stitching, enabling the agent to construct optimal trajectories even in sparse datasets. By contrast, BCQL's reliance on previously observed state-action pairs limits its performance when faced with a limited dataset. In many practical settings state reachability cannot be naturally assumed from the environment. As a result, in the following sections, deep learning implementation of SCQL is defined where state reachability learned from the dataset.
4 Practical implementation: StaCQ
In this section, we aim provide practical implementation of SCQL. Through this method we seek benefit from the state-constrained framework, where Q-values updated all state and reachable next state pairs; and the policy uses the reachable states to find more diverse, safe and closer to optimal policy. This requires practical method to learn state reachability, as well an actor-crilie method that exploits the benefits of the state reachability.
As a result, we introduce StaCQ Chaatt-Conttaineed deep QSS-learning), our particular deep learning implementation of SCQL. We present. a simple, yet. elfective, approach to learning state reachability, using dynamics models of the environment. StaCQ is policy constraint method that regularises the policy towards the best reachable next state in the dataset. Our specific implementation learns QSS-function that is aligned with the underlying theory. However, state-constrained methods can be adapted from the theory in various ways, similar to how many techniques modify the batch constraint in BCQ (Fujimoto et al. 2019). For instance, alternative state-constrainet methods might incorporate different policy regularisation techniques, such as those based on divergence measures or constraint violation penalties, or explore different approaches to estimating state reachability using available data and domain knowledge. The purpose here is illustrate one specific implementation of the broader state-constrained framework, this implementation can be extended modified in several ways.
4.1 Estimating state reachability
Central the state-constrained approach the concept of state reachability, as per Definition 3.1 Because the environments used in benchmarks do not provide this information, need to estimate reachability

---

Published in Transactions on Machine Learning Research (05/2025)
from the dataset. In our implementation, we consider s' reachable to if we predict the action that reaches 8'. For this, we introduce a forward dynamics model, far (8,a), and an inverse dynamics model, (), Both models are implemented as neural networks and trained via supervised learning on triplets (s, a,as s') D. The loss function for the forward model is given by
Lw, = B(s.as)) (s, (s.a) =s'
(7)
while the loss for the inverse model is:
Lwa E "ss,""")==
a)
(8)
Consistent with prior research, we adopt an ensemble strategy to account for the epistemic uncertainty (uncortainty model paramcters) (Buckman et al.) 2018 Chua et al.] 2018 Janner et al.] 2019 Argenson & Dulac-Arnold 2020; Yu al., 2020 2021]. Each model within the ensemble possesses distinct set of parameters. The ensemble's 's final predictions for s' and
computed by taking the average of their respective outputs. It is important to note that these models relatively simple, and incorporating more complex architectures, such as those proposed in Zhang et al. 2021). could significantly improve both model predictions and state reachability estimates.
Using these models, we propose an estimator of state reachability, denoted by SRM, as:
(8,8)) 8' E (R)(s) ifl [s,, (8, 8)) - 81 S.
(9)
This criterion suggests that 8' is reachable from 8 only if an action can be predicted that transitions the agent from 8 to S'. Our practical implementation of state reachability, shown in Eq. 9 aims to satisfy Definition 3.1 Without access to the true enviromment dynamics, we must model these transitions using available data. We use the -orm between observed and predicted states to ensure reachability determined uniformly across all state dimensions. While other norms, such as I." and L2, can be used, our experiments showed no significant difference in the policy produced by StaCQ between the L and _ norms. An ablation study the effects of norm choice and threshold distance is provided in Appendix
It is important to note that e, in Eq. 99. is small positive value used to account for potential model inaccuracies, and we set = 0.1 for all datasets. In Appendix B we also present a method for reducing the computational complexity of this calculation.
4.2 StaCQ
StaCQ uses actor-critic framework, where the critic, represented by the Q(s, value, is trained by minimising the mean square error (MSE) between predicted and true QSS-values. In line with the theory, the QSS-values are updated across all pairs of states and their reachable next states:
[ (1(8),8) + 7Q (8,, (8, ny (5))) (8,
Qo(s, (s) -o(s, )
Lo
(10)
BND ((SRM((s
In this equation, the target actor produces an action, To (8),, which passed into the forward model fur before being evaluated by the QSS-function. The specific reward for the reachable pair (s) may be unseen in the dataset, and if the reward function is unknown, it must be approximated using a neural network. Similar to the forward and inverse dynamics models, this can be achieved through supervised learning by minimising the MSE between the predicted and actual rewards:
Lug E (e..,, (( (8, -))))
(11)
Past rescarch suggests that this type of reward model is effective at estimating rewards for previously unseen transitions (Hepburn & Montana| 2024). In Eq. 10), 0' represents the parameters of the target Q-value, which are incrementally updated towards 0: 0' < 70 + (1 T)0', where is the soft update coefficient.
8

---

Published in Transactions on Machine Learning Research (05/2025)
To train the deterministic actor, 8 A, aim to maximise the current QSS-values while staying close to the best next state in the dataset. Therefore, the loss we minimise is:
C = E (non--=1Q0000 AQo(s, fw, (s, mo(s))) + (fws (s, (o))) - 87
(12)
where the hyperparameter controls the level of regularisation. The 8' represents the best reachable next state from 8 according to the QSS-value,
arg max Q0(8, (SERR((s)
This is, therefore, similar policy extraction method TD3+BC (Fujimolo & Gu 2021 as both methods use MSE regulariser. However, StaCQ leverages the state reachability metric, allowing the maximisation to be performed over multiple states, whereas TD3+BC is limited to single state-action pair.
Similar to bateh-constrained offline RL methods, StaCQ aims to balance staying close to the dataset while max- imising the Q-value. The hyperparameter À controls the trade-off between producing actions that lead to high-value OOD states and staying close to states the dataset, specifically the highest value reachable state. Since the forward model is fixed, small amount of Gaussian noise is added to the policy action, to ensure the policy does not overfit to the forward model, resulting in more ro- bust policy, The full algorithm is presented in Algo- rithm I Notably, all three constitucnt models are trained using supervised learning, making them straightforward sub-processes within the overall architecture. The code can be found at https://github. com hot.a//ooo///aaeeeeeaaablll St:to--oonstraaaall-------------
Algorithm StaCQ
Input: Datasot D, T iterations, T
Initialise: Wi; w2, W3, 0, 0' Φ
Pre-train fu, & Lug: Eqs. &
4: Pre-train reachability criteria SRM
for t = 1.,.. do
Optimise reward function: Eq.
11
Optimise QSS-value: Eq.
10
8:
Optimise policy: Eq.
12
9:
Update target networks:
8' + T0 (1-
o' T + -1-
10: end for
Learning
5 Related work
of the pioncering offline deep RL methodologies (Fuji- moto et al.] 2019). It posits that having all (s,a) pairs in the MDP enables the construction of optimal QSA-value by focusing updates only on the (s, a) pairs present in the dataset. As a result, BCQ restricts the policy to these observed (s, a) pairs while allowing minimal perturbations to facilitate marginal improve- ments. Following BCQ, numerous algorithms have been introduced to develop alternative ways to constrain the policy to seen (8,a) a) pairs, either by restricting to the support of the (s, a)-distribution (Kumar el al. 2019
Model-free offline RL. BCQ has emerged
Wu al.] 2001 Siegel al. 2020 Kostrikov et al.) 2021a Zhou et al. 2021 Brandfoubrener et al.
2021 or directly limiting it to the (s.0) pairs (Fujimoto & Gu. 2021
In contrast to policy constraints, some methods tackle distributional shifts by pessimistically evaluating the value function on OOD (s, pairs (Kumar et al.) 2020 Yu et al. 2021 An et al. 2021). There is also a subset of offline RL mothods that combincs both pessimistic value evaluation and policy constraints (Dadashi
et al. 2021 Beeson Montana, 2024). Alternatively, implicit Q-learning (IQL) (Kostrikov et al. 2021b) and policy-guided offline RI. (POR) (Xu al. 2022) aim to learn an optimal value function through expectile regression, with POR estimating a state-value function and IQL estimating an action-value function. Both methods learn the optimal policy (referred to as the guide policy in POR) via advantage weightte regression. All of these approaches adopt the batch-constrained method, prioritising explicit (s,a) pairs (s,s') pairs found in the datasct over unobscrved actions. Our state-constrained method, however, relaxes the batch- constrained objective by requiring constraints only on reachable states in the dataset rather than explicit (s, a) pairs.
Model-based offline RL. Model-based methods learn the dynamics of the environment support policy learning (Sutton 1991 Janner et al. 2019). There are generally two main approaches to model-based
9

---

Published in Transactions on Machine Learning Research (05/2025)
offline RL. The first approach learns a pessimistic model of the environment and performs rollouts with this augmented model, effectively increasing the dataset size (Yu et al.) 2020 Kidambi et al., 2020: Rigter
et al. 2022). The second approach uses the models for planning, allowing the agent to look ahead during
evaluation to determine the optimal path (Argenson & Dulac-Arnold 2020; Zhan et al. 2021 Janner ct al.
2022 Diehl et al. 2021). In contrast, StaCQ uses models to implement our concept of state reachability;
thus, StaCQ does not generate new states or involve planning during evaluation.
State reachability. The state-constrained approach relies heavily on the concept of state reachability.
Prior studies on state reachability (Hepburn & Montana 2022; 2024] have employed a Gaussian distribution to determine the likelihood of stitching from to s' In contrast, our method treats a state as either reachable or not, avoiding the use of continuous probability distributions. State reachability is closely related to state similarity, well-cxplored concept in the literature (Zhang et al. 2020 Agarwal et al. 2021 Le Lan et al., 2021). One way to measure state similarity is through bisimulattion metrics, which are based on the environment's dynamics [Ferns et al.) 2012). Traditional bisimulation methods often require full state cnumeration (Chen et al. 2012; Bacci et al. 20133 b), leading to the development of more scalable pseudometric approaches (Castro) 2020). While incorporating a pseudometric into our state reachability framework could be a potential avenue, we leave this for future work. Our current state reachability metric is simple, intuitive, and grounded in the core definition.
Trajectory stitching. A key feature of offline RL is its ability to stitch together trajectories (Kostrikov et al.) 2021b), i.c., combining previously observed trajectories create a novel one that completes the task. Imitation learning methods, such as behavioural cloning (Pomerleau, 1988 1991, , struggle with stitching due to their inability to distinguish between optimal and sub-optimal states. Similarly, the Decision Transformer (DT) (Chen et al.] (2021) formulates the offline RL problem as supervised learning task using a goal- conditioned policy. While DT exhibits weak stitching capabilities, efforts have been made to address this by integrating offline RL principles (Yamagata et al. 2023: Wu et al., 2023). Combining DT with StaCQ's policy extraction approach could yield further improvements in stitching. However, our current work focuses on a straightforward implementation of state-consttaineed oflline RL.
6 Experimental results
We evaluate StaCQ against several model-free and model-basod baselines the D4RL benchmarking datasets from the OpenAl Mujoco tasks (Todorov et al. 2012: Fu et al. 2020). The model-free baselines we compare against are BCQ (Fujimoto et al. 2019), TD3+BC (Fujimoto & Gu 2021), and IQL (Kostrikov et al. 2021b). BOQ is the foundational catth-coustrained offline RL method on which many subsequent mothods are based, making it the most direct theoretical comparison. TD3+BC is the most similar to StaCQ in terms of implernentation, as both use a BC-style regularisation process. IQL, a current SOTA model-free method, provides a strong comparison point.
The model-based baselines include MBTS (Hepburn & Montana 2024). Diffuser (Janner et al. 2022 and
RAMBO RRigeer et al. 2022). MBTS employs a data augmentation strategy with different state reachabil- ity approach compared to StaCQ. Diffuser is a planning-bas based method that uses a diffusion model, making its dynamics model more complex than StaCQ's. RAMBO, a SOTA model-based method, performs rollouts using a pessimistic dynamics model of the environment. Although StaCQ does not use models for planning or rollouts, we include these model-based algorithms for comparison.
Additionally, we devised one-step version of StaCQ (Brandfonbrener et al. 2021), detailed in Appendix D to showcase the fiexibility of the state-constrained framework.
Results for both StaCQ and the one-step version of StaCQ are shown in Table 1 For the Antmaze tasks, it is unnecessary to Irain a reward model since these tasks feature sparse rewards, meaning the reward for being in s' remains constant. As all are in the dataset, the reward, r(s,s'), is always known for all s where s' E (s). Also for the Antmaze tasks, StaCQ deploys independent target Q-value estimates. This is a techniques from Ghasemipour et al. (2022) that improves the performance of the policy in the medium and large Antmaze tasks. Despite potential model errors in estimating state reachability, StaCQ performs
10

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Average normalised scores on the D4RL datasets. StaCQ results have been obtained by taking an average 5 sceds. The bolded scores are within 95% of the highest performing method.
Model-free baselines
Model-based baselines
Our methods
BC
BCQ TD3+BC IQL
MBTS Diffuser RAMBO
StaCQ
OneStep StaCQ
Rand
6.2
7.6
21.6
17.5 + 12.8
7.5
Hopper
Med-Rep
22.5
51.0
60.9
94.7
50.2
93.6
96.6
99.1 = 1.3
99.4 1 0.6
Med
60.9
59.3
66.3
64.3
74.3
92.8
100.2 3.0
93.3 3.2
Med-Exp
54.2
85.9
98.0
91.5
94.8
103.3
83.3
111.9 + 0.2
92.1 + 9.2
Walker2D
Rand
1.4
4.4
1.6
11.5
4.4
6.5 1 0.9
Med-Rep
60.7
81.8
73.9
61.5
70.6
85.0
87.2 7.3
88.1 + 5.3
Med
39.4
73.7
83.7
78.3
78.8
79.6
86.9
92.2 = 3.0
85.7 + 10.4
Med-Exp
90.5
94.5
110.1
109.6
108.8
106.9
68.3
116.2
107.6 8.5
Halichootah
Rand
2.1
2.2
11.0
40.0
2..3
2.6 1.6
Med-Rep
34.5
41.1
44.6
44.2
39.8
37.7
68.9
52.2 0.5
46,4 + 0.4
Med
42.4
46.6
48.3
47.4
43.2
42.8
77.6
57.6 0.6
50.0 + 0.2
Med-Exp
66.6
87.8
90.7
86.7
86.9
88.9
93.7
96.4 + 2.9
94.9 1 1.0
Total (excl. rand)
432.4
602.2
677.4
692.6
628.3
697.7
753.1
813.0
757.5
Umaze
53.4
70.0
78.6
87.5
25.0
89.8 + 5.0
75.6 75.6 3.8
Antmaze
U-diverse
64.6
44.0
71.4
62.2
0.0
64.0 1
67.2 13.8
M-play
0.0
0.0
3.0
71.2
16.4
47.2 = 27.1
20.6 + 12.5
M-diverse
0.8
0.0
10.6
23.2
47.4
13.0 - 4.2
T-play
0.0
0.0
0.0
39.6
0.0
31.4 + 6.0
5.2 5.2
L-diverse
0.0
0.0
0.2
47.5
2.4
40.0 + 22.4
2.2 1 1.6
Total (Antmaze)
118.8
114.0
163.8
378.0
67.0
319.8
183.8
remarkably well against the baselines. The learning curves for all tasks in Table 1 are shown in Appendix
StaCQ outperforms model-free baselines in the locomotion tasks (Hopper, Halfchootah, and Walker2d) across the entire range of datasets (random, medium-replay, medium, and medium-expert). These envi- ronments are complex robotics tasks, with the random and mediun-replay consisting of few good-quality episodes. StaCQ also surpassos most model-based baselines in the locomotion tasks, with the exception of RAMBO, which outperforms StaCQ in the Halfcheetah and random tasks. However, average StaCQ is the highest-performing method across all locomotion tasks, consistently ranking either the top second-best performing method. TD3+BC uses a consistent hyperparameter across all datasets, whereas our StaCQ methods use an environment- -deeennttteeeeeentt hyperparameter (see Appendix C To enable a fair comparison, Appendix compares TD3+BC and StaCQ using the same hyperparameter tuning strategy.
Through utilising the higher number of reachable states and thus finding higher quality OOD actions that lead to in-distribution states, StaCQ performs well across most of the locomotion tasks. However its performance in the Antmaze tasks is comparatively lower. StaCQ does outperform RAMBO in the Antmaze tasks, which are notoriously challenging for methods involving dynamics models (Wang et al. 2021 Rigter et al. 2022), and performs compelitively against the model-free baselines. Notably, StaCQ outperforms BCQ and TD3+BC, which are the most comparable baselines terms of both theory and implementation. Methods such as IQL, which rely on
weightte policy extraction, may have an advantage these tasks due their ability handle sparse rewards more effectively. Nonetheless, StaCQ either the highest or second highest performing method the Antmaze tasks against the comparative methods.
11

---

Published in Transactions on Machine Learning Research (05/2025)
7 Discussion and conclusions
In this paper, we introduced a method for state-constrained offline RL. Most prior offline RL approaches are bateh-consttraine,, limiting the learning of the Q-function policy to (5,a) a) pairs or the (s, distrrbbbbtttn in the dataset. In contrast, our stale-constrained methodology confines learning updates to states within the dataset, offering significant potential reductions in dataset size required for achieving optimality. By focus- ing on states rather than state-action pairs, the stato-constrained approach enables more efficient learning updates and can achieve optimality with smaller datasets compared to batch-constrained methods.
Central to the proposed state-constrained approach is the concept of state reachability, which we define and illustrate in a simple maze environment. This example underscores the improved stitching capability of state-constrained methods over match-constrainet ones, particularly in datasets with limited trajectories. Additionally, we provide theoretical backing that affirms the potential for our method's convergence based on dataset states rather than state-action pairs. Similar to BCQ (Fujimoto et al. 2019), our theory assumes a deterministic MDP. This focus on deterministic environments simplifies the theoretical development (Fu- jimoto et al., 2019: Jin et al.] 2021: Shi et al.] 2022), and provides a fair comparison with the BCQ theory, while also providing clear guarantocs for convergence, making an appropriate starting point for the state- constrained approach. Our theoretical goal is demonstrate convergence and provide comparison with BCQ. As result, Theorem 3.6 shows that SCQL converges to the optimal policy with a smaller requisite dataset size than BCQ; and, Theorem 3.7 shows that the policy produced by SCQL can always be preferred to the policy produced by BCQL.
Based from these theoretical developments, our practical implementation also defines reachable states de- terministically states are considered reachable or not, regardless of potential stochastic transitions. De- terministic settings align with many practical applications, such as robotic control or industrial systems, where transitions can be accurately modelled. However, some real-world environments are stochastic, where actions lead to distribution of possible next states. Future work will extend state reachability to account for stochastic transitions by incorporating probabilistic models of the environment's dynamics. Theoretically, the definition of state reachability can be exlended to stochastic MDPs, for example s' is reachable from 8 if is the most likely next state when performing action a, i.c s' E (s) if p(s'|s, ())aa) = maxx maxxs/>((-1s.a) p($/ [s, a). Practically, this will involve redefining reachability in terms of the likelihood of reaching a state under certain actions, possibly using techniques such as probabilistic graphical models. The key challenge will be maintain- ing computational efficiency while handling the increased complexity of probabilistic transitions. Extending the framework in this way will allow the stato-constrained approach to be applied to more complex, uncertain environments, improving its robustness and real-world applicability.
We also introduced StaCQ, an implementation of state-constrained deep offline RL. StaCQ leverages QSS- learning (Edwards et al., 2020) to estimate the value of transitioning from s to s', which is particularly advantageous in a state-conntraained context as it avoids action dependency. StaCQ determines state reach- ability using forward and inverse dynamics models, trained via supervised learning on the dataset. While the performance of these models can inffuence StaCQ, the algorithm consistently demonstrates strong per- formance in complex environments, often surpassing baseline cothh-constrained methods. StaCQ relies on an ensemble of forward and inverse dynamics models. Inereasing the size of the ensemble would enable more accurate estimates, however the number of models is consistent with prior research (Janner et al., 2019) and is accurate enough for StaCQ to perform well, as shown in Table I StaCQ uses an ensemble of four critics whereas OneStep StacQ uses a single critic. These values were chosen as they enable stable estimates across all datascts.
There are several potential improvements to explore in future work. Despite the simplicity of the models we used, our reachability measure delivered SOTA performance. Future enhancements could improve this by adding more complexity to the dynamics models, reducing model error, and capturing the true environment dynamics more accurately. Alternatively, developing entirely new state reachability criteria independent of dynamics models could significantly enhance the algorithm's effectiveness. Potential ideas include leveraging graph-based approaches or using reinforcement learning techniques to learn reachability directly from data. Additionally, the policy extraction process could be improved by incorporating elements such as a Decision Transformer (Chen et al.] 2021) diffiusion policies (Ajay et al. 2022). These techniques have shown
12

---

Published in Transactions on Machine Learning Research (05/2025)
promise in offline RL by leveraging the structure of the dataset to generate more diverse and effective policies. Integrating them into StaCQ could allow the algorithm to exploit the available data more effectively.
Similar to batch-constrained methods, further advances might arise from relaxed state constraints, albeit with tailored considerations for this setting. Using ensembles to assess the uncertainty of QSA-values (An ett al. 2021 Ghasemipour et al. 2022 Beeson & Montana, 2024) and constraining actions to specific regions has proven advantageous in the batch-constrained domain. Similar methodologies could be extended to state-constrained approaches, such as employing ensembles of QSS--functoons to evaluate uncertainty.
Beyond improving state reachability estimates and policy extraction mechanisms, the potential for advance- ment in state-constrained methodologies is vast. One intriguing direction is to refine state reachability estimates to focus solely on states, excluding action predictions. Such advancements could lead to algo- rithms that rely entirely on state-only datasets, which is particularly relevant for real-world scenarios like learning from video data, where states are observable (as images), but the actions leading transitions are unknown. Expanding the notion of state reachability another promising avenue. Our current definition, based one-step state reachability (3a such that p(s"|s, a) = =1), could be extended to multiple steps. For example, two-step reachability could be defined as s" E SRM(ss if 3a, 02 such that p(s"|s, a1) = 1 and (ss ,02) = 1. This principle can be further extrapolated, providing foundation for novel model-based offline RT. algorithms. Extending this notion of state reachability will allow for a higher number of reachable states to found, while being sure they lead, in k-steps, to in-distribution states. Appendix H shows the theoretical benefits of a multi-step state reachable method on a simple maze environment. However, a practical method that incorporates model error is left for future work. Lastly, the state-constrained method- ology holds significant potential for multi-agent offline RL scenarios, where complexity grows with the action space size. By reducing action dependence, the state-constrained approach appears well-suited to tackle such challenges.
In conclusion, we have ventured into the realm of state-constrained offline RL, introducing a novel paradigm that emphasises states over state-action pairs in the dataset. While this paper lays the foundational ground- work, the path forward calls for further investigation, refinement, and integration with other emerging techniques. We believe that these endeavours can push the boundaries of current offinee RL methodologies and offer unprecedented solutions to complex problems.
13

---

Published in Transactions on Machine Learning Research (05/2025)
References
Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning, arXiv preprint arXiv:2101.0555 2021.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is condi- tional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.16577 2022.
Gaon An, Seungyong Moon, Bang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified Q-ensemble. Advances in newral information processing systems, 34:7436 7447, 2021.
Arthur Argenson and Gabriel Dulac-Arnold. Model-based offiine planning, arXiv preprint arXiv:2008.05556 2020.
Giorgio Bacci, Giovanni Bacci, Kim G Larsen, and Radu Mardare. Computing behavioral distances, compo- sitionally. In Mathematical Foundations of Computer Science 2013: 88th International Symposium, MFCS 2013, Klosterneubury, Austria, August 26-30, 2019. Proceedings 98, pp. 74 15-5. Springer, 2013a.
Giorgio Bacci, Giovanni Bacci, Kim Larsen, and Radu Mardare. On-the-fly fly exact computation of bisimi- larity distances. In International conference on tools and algorithums for the construction and analysis of systems, pp. 1-15. Springer, 2013b.
Alex Beeson and Giovanni Montana. Balancing policy constraiit and ensemble size uncertainty-based offline reinforcement learning. Machine Learning, 113(1):4434483) 2024.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. Advances neural information processing systems, 34:4933-4946, 2021.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-effiiiet rein- forcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 31, 2018.
Pablo Samuel Castro. Scalable methods for computing state similarity deterministic markov dccision processes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 10069 10076, 2020.
Di Chen, Franck van Breugel, and James Worrell. On the complexity of computing probabilistic bisimilarity. In Foundations of Software Science and Computational Structures: 15th International Conference, FOS- SACS 2012, Held as Part of the European Joint Conferences Theory and Practice of Software, ETAPS 2012, Tallinn, Estonia, March 24-April 1, 2012. Proceedings 15, pp. 437-451. Springer, 2012.
Lili Chen, Kevin La, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement lcarning via sequence modeling. Advances in newral informalion processing systems, 34:15084-1555
2021.
Kurtland Chua, Roberto Calandra, Rowan McAAlister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advonces in neural information processing syslems, 31, 2018
Robert Dadashi, Shideh Rezaeifar, Nino Vieillard, Léonard Hussenot, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning with pseudometric learning. In International Conference on Machine Learn- ing, pp. 2307-2318. PMLR, 2021.
Christopher Diehl, Timo Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram. Um- brolla: Uncertainty-aware modcl-basod offline reinforcement learning leveraging planning. arXiv preprint arXiv:2111.11097 2021.
Christopher Dichl, Timo Scbastian Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram. Uncertainty-awan model-based offline reinforeement learning for automated driving. IEEE Robotics and Automation Letters, 8()) 2023.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Ashley Edwards, Himanshu Sahni, Rosanne Liu, Jane Hung, Ankit Jain, Rui Wang, Adrien Ecoffet, Thomas Miconi, Charles Isbell, and Jason Yosinski. Estimating Q(s, s') with deep deterministic dynamics gradients. In International Conference on Machine Learning, pp. 2825 2835. PMLR, 2020.
Xing Fang, Qichao Zhang, Yinfeng Gao, and Dongbin Zhao. Offine reinforcement learning for autonomous driving with real world driving data. In 2022 IEEE 25th International Conference Intelligent Trans- portation Systems (ITSC), pp.
IEEE, 2022.
Norman Ferns, Prakash Panangaden, and Doina Precup. Metries for finile markov decision processes. arviv preprint Xiv: 107.41277,114 2012.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data- driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20144 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International conference on machine learning, pp. 2052 2062. PMLR, 2019.
Kamyar Chasemipour, Shixiang Shane Gu, and Ofrr Nachum. Why so pessimistic? estimating uncertainties for oflline RT. through ensembles, and why their independence matters. Advances in Neural Information 2022.
Processing Systems,
Antonin Guttman. R-trees: A dynamic index structure for spatial searching. In Proceedings of the 1984 ACM SIGMOD international conference on Management of data, pp. 47-57, 1984.
Charles A Hepburn and Giovanni Montana. Model-based trajectory stitching for improved offline reinforce- ment learning. arviv preprint xrXiv:2211.11603 2022.
Charles A Hepburn and Giovanni Montana. Model-based trajectory stitching for improved behavioural cloning and its applications. Machine Learning, 113(():641766) 674,4,744744447 2024.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXim:2205.0000 099911 2022.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference Machine Learning, pp. 5084 5096. PMLR, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-based offline reinforcement lcarning. Advances in neural information processing systems, 33:21810-21823, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ar preprint arXiv:1412.690 2014.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference Machine Learning, pp. 5774-5783. PMT.R
2021a.
llya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. arXiv preprint arXi::2110.066 06.66669,, 2021b.
via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019. Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy Q-learning
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline rein- forcement learning. Advances in Neural Information Processing Systems, 13:1179 19-1179-1199 1191, 2020.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model- free robotic reinforcement learning. arXiv preprint or Xrv:21109. 10813, 2021.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-ard pp. 45 73. Springer, 2012.
Charline Le Lan, Mare G Bellemare, and Pablo Samuel Castro. Metrics and continuity in reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 8261-8269, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 2tt::00050016455 2020.
Tom Mitchell. Machine Learning. McGraw Hill, 1997.
Dean A Pomerleau. Alvinm: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88 97, 1991.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-RL: Robust adversarial model-based offline reinforce- ment learning. Advances in neural information processing systems, 35:16082 16097, 2022.
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic Q-learning for offline reinforcement learning: Towards optimal sample complexity. In International conference on machine learning, pp. 20025. PMLR., 2022.
Tianyu Shi, Dong Chen, Kaian Chen, and Zhaojian Li. Offline reinforcement learning for autonomous driving with safety and exploration enhancement. Xiv preprint arXiw:2110.07000 2021.
Chamani Shiranthika, Kuo-Wo Wei Chen, Chung- Yih Wang, Chan-Yun Yang, BH Sudantha, and Wei-Fu Li. Su- pervised optimal chemotherapy regimen based on oflline reinforcement learning. IEEE Journal of Biomed- ical and Health Informatics, 26(9):4763 17722-7777,, 2022.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe; Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning, arXiv preprint 2xXiu:2002.0836, 2020.
Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. COG: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010.14500 2020.
Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4RL: Surprisingly simple superviini for offline reinforcement learning in robotics. In Conference Robot Learning, pp. 907-917. PMLR, 2022.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160-163, 1991.
Richard S Sutton and Andrew G Barto. Reinforcement leurning: An introduction. MIT press, 2018
Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Machine Learning for Healthoore Conference, pp. 2 2-. PMLR, 2021.
Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi- Velez, and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems, 35:34272 15:34222-34286, 2022.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- based control. In 2012 IEFF/RSJ international conference intelligent robots and systems, pp. 5026 5033. IEFF, 2012.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offlie re- inforcement learning with reverse model-based imagination. Advances in Newral Information Processing Systems, 34:29420 29432, 2021.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279- 292, 1992.
Yifan Wu, George Tucker, and Ofrr Nachum. Behavior regularized offline reinforcement learning. arXiv preprint xrXi::1911.11961 2019.
Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. ar preprint arXiv:2307.024684 2023.
Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for oflline reinforcement learning. Advances in Neural Information Processing Systems, 35:4085-4099, 2022.
Taku Yamagata, Ahmed Khalil, and Raul Santos- Routros-. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling offline RL. In International Conference on Machine Learning, pp. 38989 39007. PMLR, 2023.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MÓPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129 14142, 2020.
Tianhe Yu, Aviral Kumar, Rafacl Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative oflline model-based policy optimization. Advances in neural information processing systems, 0122954-289677 2021.
Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model- based offline planning with trajectory pruning. anXiv preprint XrXi::2105. 07351, 2021.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant repre- sentations for reinforcement learning without reconstruction. arXiv preprint arXi::2006.100 10744 2020.
Michael R. Zhang, Tom Le Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyu Wang, and Mo- hammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. arxiv preprint orXie:2144.18877 2021.
Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pp. 1719-1735. PMLR, 2021.
17

---

Published in Transactions on Machine Learning Research (05/2025)
A Missing proofs
Since each ss) pair is visited infinitely often, consider consecutive intervals during which each (s, s') transition occurs at least once. We want to show the max error over all entries in the Q table is reduced by at least a factor of y during each such interval.
Proof of Theorem 3.2
Let An by the max error in On On == maxs,s' [Qn(s, 8¹) (*(s, Then,
Qu+1(8,8)) - Q* (8, = [(((8,8) 8)) + max Qn (s', 5") - (r(8, + max ('(s',
(13)
max max Qn(6', 8") max ('(5', 66,)"))
(14)
> max Qn (ss'''") max Q'(s', sS"!
(15)
8"
8"
S max
1Qn(s' ,") - Q"(s'
(16)
S y max
Qn((, Q* 8
(17)
8,0"
Which implics 1Q++1(8,8)) (s, S A
Where for brevity the maxinisation max," is the shorthand for max a.t. (ESRu())) Also, Eq- 13 is using the training rule; Eqs.
14 and 15) are simplifying and rearranging: (16) uses the fact that max, fi()) - max, f2(2)] I max, (x) fz(=)): Eq. (17) we introduced a new variable & for which the
maximisation is performed this is permissible as allowing this additional variable to differ will always be at least the maximum value.
Thus, the updated (n+ for any s' of is at most y times the maximum error in the Qn table, A. The largest error in the initial table, Ao, is bounded because the values of Qo(8, and (* are bounded Vs, Now, after the first interval during which each s is visited the largest error will be at most 70g- After k such intervals, the error will be at most 7t Ag. Since each state is visited infinitely often, the number of such intervals is infinite and An + 0 as +C.
1.Define deterministic state-constrained MDP Ms. We define Ms (S,A,Pss R, where S and A are the same as the original MDP. We have the transition probability where
Proof of Theorem 3.4,
ps((')s,a) (1 if (s, s' E' D and s' SRM(()) or (s & D and s' Sterminal)
otherwise.
this is deterministic transition probability for 8, s' that are in the dataset and reachable. If pair exists but is not in the dataset we set the rewards to be the initialised ((s, values, otherwise they have the reward seen in the dataset.
2. We have all the same assumptions under Ms as we do under M, apart from infinite (s, 8') visitation, so we need that and then it follows from Theorem 3.2
Note that sampling under the dataset with uniform probability satisfies the infinite state- visi- tation assumptions of the MDP Mg- For reachable pair (s, & D [this may be due to s & D or s' $ D). Q(s, s') will never be updated and will correspond to the initialised value. So sampling from D is equivalent to sampling from the MDP Ms and QSS-learning converges to the optimal value under Ms by following Theorem 3.2
This follows from Theorem 3.2 noting the state-constraint is nestriitrietive with a dataset which contains all possible states.
Proof of Theorem 3.5.
Proof of Theorem 3.6 This result follows from the fact that every s visited infinitely then due the definition of state reachability we can evaluate over all (s, s')-pairs in the datasct. Then following Theorem 3.4 which states QSS-learning learns the optimal value for the MDP Ms for s, E D. However, the deterministic Ms corresponds to the original M in all seen state and reachable next-state pairs. Noting
18

---

Published in Transactions on Machine Learning Research (05/2025)
that state-constrained policies operate only on s, E D, where Ms corresponds to the true MDP, it follows that x² will be the optimal state-constrained policy from the optimality of QSS-learning.
Theorem A.I. In deterministic setting, QSA-malue equivalent to QSS-values
Proof. See Theorem 2.2.1 in Edwards et al. 2020
Proof of Theorem 3.7 Case 1: In this situation, for all states in the dataset, we do not have any extra
reachable states, other than the pairs i... As D, (s)) {s'}, where (s.s') D. In this case the state- constrained MDP, Definition 3.3 is equivalent to the batch-comsttrinee MDP in (Fujimoto et al. 2019). The condition of the state-constrained MDP,
8, E and s' (R)(s)
becomes (s) E D as s' (s) only exists where (s, D. From this, the probability transition function for the state-constrained and batch-constrained MDPs are equivalent and thus are the MDPs themselves. Finally, for this case just need to show that the SCQL and BCQL updates are the same. So under this case condition the SCQL update becomes
((),8)) (1-a) + u((((s,'') +7
0(6',5") s.t. (s
From Theorem A.1 for transition (s, a, s') D have Q(s, a) = Q(s, s') and thus for the transition (8', a', D we have Q(s', Q(s'((('))) 8"), therefore the SCQL update is equivalent to
((s,a)
a)Q(s, a (s) +y st. (of (E)
((st
Therefore, for case 1 BCQL and SCQL have the same Q-value update and therefore have the optimal policy, So, (s) TscQu(s), Q(s, TBCQL(s)) Q(s, mscQu(s)), and thus VISCQL (s) Vrsequ (s), Vs e D.
Case 2: In this case, for a single state in the dataset, we have one reachable next state that un- seen as a pair the dataset, ie. E8,8' E s.t. s' (5) and (x, 8') & D. In this case the ctate-constrained MDP probability transition function becomes: for [(s, A
p)($(1,a)= =(0 if ((s,8')))) & D and s' Sterminal) or (s and 8' = 8) otherwisc.
This the transition function for the batch-constrained MDP but allowing for an extra transition from to S'. Now comparing BCQL and SCQL Q-value updates, we have all equivalent values for the transition (s, La,ss) s') except for the trajectory that contains the state s. For the converged QSS-value, Q", let $-1 be the state in the trajectory previous to and g' be the next state after in the trajectory,
@"(8-1,8) (1 (1- (8-1,)) (3 + max max (3, Q²()
8') then again BCQL is equivalent to SCQL. However, if (3,))") (3, then SCQL will have higher values than BCQL for all states previous to and therefore will produce higher quality policy by the policy improvement theorem. In this case, as (²((,5") < (3 Es such that Q(s, (s)) > Q(s, (s)), thus Vzscou (8) > V=scou Vs E.
If (5, 2 (3,
Then without loss of generality Case 2 can be extended to all cases where we have multiple reach- able next states for multiple dataset states, that have higher value according to the optimal QSS-value.
19

---

Published in Transactions on Machine Learning Research (05/2025)
B Reducing the complexity of state reachability estimation
Directly computing SRMM for every state would entail comparing each state in the dataset against all others, an approach that is computationally prohibitive for larger datasets. To address this challenge, we calculate the range of potentially reachable states for each state dimension. This calculation is based a set of random actions, denoted as {arand
For a given state 8, we first determine its range by calculating the minimum and maximum values using say Specifically, the minimum range Ruin(s) (s) is computed as min; fw, (s, aa and the maximum range Ruix is max; Say (s, lana). Subsequently, we construct smaller set of states within the range (Rmin Rmax (+)), using an R-tree (Guttman 1984), a data structure that efficiently identifies states within a specified hyper- rectangle (our range). This approach, leveraging the R-tree's efficient search capability, dramatically reduces the size of the dataset. Consequently, the models are applied only to this refined set of states, leading to a significant reduction in computational complexity.
C Implementation details
For our experiments we use = 0.1 for the state reachability criteria where the LHS is also scaled by the maximum of the difference in state range, i.e our state reachability estimate for state is determined by
fwi (s, Lwo (s. s))) 5 Rmax (s) - Rmiu
<c
This means that the maximum state dimension model prediction error must be within 10% of the true state range. Due this we also normalise all states which gives more accurate model prediction and is done in the same way TD3+BC (Fujimoto & Gu 2021 i.c let s; be the ith feature of state
81 M4i
0 6
where μr and o; are the mean and standard deviation of the ith feature across all states and 6, 10-3 is small normalisation constant. Similar to TD3+BC, we do not normalise states for the Antmaze tasks as this is harmful for the results.
For the MuJoCo locomotion tasks we evaluate our method over seeds each with evaluation trajectories; whereas for the Antmaze tasks we also evaluate over 5 seeds but with 100 evaluation trajectories. Just like TD3+BC (Fujimoto & Gu 2021 we scale our hyperparameter by the average Q- Q-value across the minibatch
a
(ss, 8))
where N is the size of the minibatch (in case N 256) and s is the next state where the policy action leads from We aim to find consistent hyperparameter across the different environments. We find that the optimal consistent hyperparameters are a (1,5, 5, 10) for Hopper, Walker2d and Halfcheetah respectively. However for Halfcheetah -meiium expert we use = 0.5 due to the significant improvement. For the Antmaze tasks, each maze size is a new environment and we use a = {2, 10, 19} for the umaze, medium and large environments respectively. Across all environments, we add a small amount of zero mean Gaussian noise to the policy action before being input into the forward model, we use a fixed variance of 0.1. It should be noted that making this policy noise vary across the different environments could improve results greatly, however aim to keep this fixed so that we have general method that can be easily optimised other environments.
The actor, critic and reward model are represented as neural networks with two hidden layers of size 256 and ReLU activation. They are trained using the ADAM optimiser (Kingma Ba 2014) and have learning rates 4, the actor also has cosine scheduler. We use an ensemble minimum value across the networks. Also we use soft parameter updates for the target critic network with parameter 0.005, and use a discount factor of 7 0.99. For the locomotion tasks a shared
critic networks and take the
20

---

Published in Transactions on Machine Learning Research (05/2025)
target value to update the critic towards, whereas for the Antmaze tasks we use independent target values for each critic value. Both the inverse and forward dynamics models are represented as neural networks with three hidden layers of size 256 and ReLU activation. They are trained using the ADAM optimiser with a learning rate of 4e-3 -3 and a batch size of 256. We use an ensemble of 7 forward models and 3 inverse models and then take a final prediction as an average across the ensemble. Also our forward model predicts the state difference rather than the next state directly, which improves model prediction performance.
To attain the results for BCQ, re-implemented the algorithm from the paper (Fujimoto al. 2019) and trained the version 2 datasets. Following the same practises StaCQ evaluate BCQ over 5 seeds each with 10 evaluations on the MuJoCo locomotion tasks and 5 seeds each with 100 evaluations on the Antmaze lasks. All other results in Table [ were obtained from the original authors' papers. Our experiments were performed with a single GeForce GTX 3090 GPU and an Intel Core 19-11900K CPU at 3.50GHz.. Each run takes on average 2.5 hours, where models are pre-trained and state reachability given (the same models and reachability lists are given to each run). We provide runs for each dataset (18) which gives a total run time of 225 hours.
One-step method
StaCQ, Algorithm 1 that we have introduced in this paper is an actor-critic method that learns the QSS- value and policy together. Alternatively, the QSS-value can be learned directly from the data, in an ou-policy fashion, then a policy can be extracted directly from this on-policy QSS-value. These approaches are known one-step methods (Brandfonbrener et al. 2021). In this section adapt StaCQ into a one-step method.
D.1 Estimating QSS-values
So that the QSS-values are learned on-policy while still taking advantage of the state-constrained framework we introduce a small modification to Eq. following approximation:
5 Since the pair (s', s") is unseen in the dataset, we use the
max ss ED " (s
fr(s', 8") + ,Q(s", )"
(18)
to replace max ED ((",s"). This adjustment ensures that the QSS-value is only evaluated on explicit ((s')
state-next-state pairs, thereby avoiding OOD (s, s')--asrs. Although this approach diverges slightly from the theoretical method, where QSS-value updates performed every pair, it provides a practical solution.
Using Eq. 18 the approximation of state reachability and the reward model Eq. [1] the on-policy state- constrained QSS-values can be refined by reducing the MSE between the target and the actual QSS-values. The target is determined by identifying the maximum value across reachable states:
Co E (1(6)6)) +7
(rag QQ0 "" (u""" -Qlesst))
max s" s..ls"" "s"ss""""" JE, JD, (ss))
(19)
Here, 0' represents the parameters for a target Q-value which are incrementally updated towards 0: + 7B + (1 7), with being the soft update coefficient.
D.2 Policy extraction step
Eq. gives a one-step optimal state-constrained QSS-value. However, this equation alone does not produce an optimal action. To determine the optimal action, we need to add a policy extraction step. We use the same policy extraction mothod as StaCQ, a state behaviour cloning regularised policy update similar
to TD3+BC (Fujimoto Gu 2021):
L. = EAND AQo(s, Juy (o())) + (Ja, (s, mo - -8'2]]
(20)
21

---

Published in Transactions on Machine Learning Research (05/2025)
where
8' arg (May(8,55) + 70ols'' s's.t.s's""ED s ('E5R((s)
A Algorithm 2 StaCQ (One Step RL version)
1: Input: Dataset D, T number of iterations,
2: Initialise: parameters wi, w2, wa. 0, 0',
3: Pre-train fw, and Lw, using Eqs. and
4: Pre-train reachability criteria SRM
5: for t= i..,..,,T do
6: Optimise reward function according to Eq.
11
7: Optimise QSS-value according to Eq.
19
8: Update larget networks: - 70 + (1 00'
9: end for
10: for 1......, do
Optimise policy: Eq. 20
12: end for
Again, due to the forward model being fixed, a small amount of Gaussian noise is added to the policy action before being input into the model. This creates a more robust policy by ensuring the policy does not exploit the forward model. The complete procodure for the one step version of StaCQ provided in Algorithm 2
D.3 One step method implementation details
In the original OneStepRL paper [Brandfonbrener et al. 2021), they evaluate their method over 10 seeds where the hyperparameter has been tuned over the first 3 seeds and then evaluated with the X fixed for the remaining 7. However as we want consistent hyperparameter for each environment we choose a = (0.1, 1.0, 5.0) for Hopper, Walker2d and Halfcheetah respectively and for the Antmaze tasks we choose a (10, 40, 100} for the umaze, medium and large environments respectively. The one step method also uses single critic, we found that increasing the mumber of critics deteriorated results. All other implementation details are the same StaCQ, Appendix C
E Consistent hyperparameter experiment
The single StaCQ hyperparameter, À in the policy update, is tuned per environment. (for example, using the same X for all Hopper datasets). This ensures StaCQ is not over-tuned to each dataset while still producing realistic, high-quality results. Most saate-of-thh-arr methods have different hyperparameters for each dataset
(Yu al. [020 Kidambi al.] 2020 Brandfonbrener al. 2021 Wang al. 2021: Ghasemipour et al. 2022 Rigter et al.) 20022 Xu et al.) 2022); this is not the case for StaCQ. However, TD3+BC and IQL both use consistent hyperparameter across all datasets. TD3+BC uses an ensemble of two critics, whereas StaCQ has ensemble of four. To compare fairly with these methods, perform two comparisons. First, we fix StaCQ's ' hyperparameter across all locomotion tasks. Second, we tune TD3+BC's hyperparameter for each environment and supplement it with an ensemble of four critics. Table 2 shows these comparisons.
From Table 2 we can see that with consistent hyperparameter tuning strategies StaCQ still vastly outper- forms BC (and IQI.). This now provides clear comparison showing how more Q-value evaluations through the state reachability metric allows for higher-quality policies than the batch-constrained counter- parts.
22

---

Published in Transactions on Machine Learning Research (05/2025)
Fixed hyperparameter
Per environment tuning & 4 critics StaCQ
IQL
TD3+BC StaCQ
TD3+BC
Random
Halfcheetah
11.0
15.8
18.0
24.3 1.3
Hopper
8.5
8.6
7.6 0.7
17.5 + 12.8
Walker2d
1.8
3.5
0.9 + 0.2
4.4 6.4
Halfcheetah
47.4
48.3
50.7
49.0 1:0 0.3
57.6 0.6
Med
Hopper
66.3
59.3
100.1
61.5
100.2 = 3.0
Walker2d
78.3
83.7
88.6
83.6 4.8
Med-Exp Med-Rep
Halfcheetah
44.2
44.6
46.5
44.6 0.7
52.2 0.5
Hopper
94.7
60.9
98.7
61.5 : 23.8
99.1 + 1.3
Walker2d
73.9
81.8
85.5
84.0 + 3.0
87.2 + 7.3
Halfchectah
86.7
90.7
92.4
90.9 2.2
96.4 2.9
Hopper
91.5
98.0
94.0
104.5 + 11.8
111.9 + 0.2
Walker2d
109.6
110.1
114.2
110.6 + 0.5
116.2 + 1.5
Total
692.6
698.7
798.6
716.7
859.2
Table 2: Average normalised scores for StaCQ, TD3+BC and IQL with consistent hyperparameter tuning strategies. The left hand side compares StaCQ with IQL and TD3+BC where there is a consistent hy- perparameter across all datasets. The right hand side compares StaCQ with TD3+BC with 4 critics and hyperparameters tuned per dataset. The bold scores are within the 95% of the highest performing method in cach datasct. Scores represent mean over seeds of 10 evaluation + the standard deviation.
F Ablation study: exploring the effect of the reachability metric
In this section, explore the effect of changing the reachability metric for StaCQ. That is, for fixed datasets, Walker2d- and Maaleerdd-MediumEppere we explore how the change of reachability norm and threshold for reachable states, effects the average normalised score.
Table shows the comparison of the reachability metric with different thresholds. For the Li-norm with 0.01 and 0.05 and jo--norm with = 0.01 on the Walker2d-Medium dataset, the threshold was too small to find any reachable states. In these cases, the reachable states were the explicit next state in the dataset, therefore Eq. [10) and [12 where only updated using (s, D. This is essentially the same method TD3+BC, except using Q(s, s') instead of Q(s,a),, a). In these cases, StaCQ performs par with TD3+BC, as expected, but much worse than StaCQ with larger threshold.
Otherwise, for all metrics increasing the threshold leads to an increased score up to a certain point (e.g. = 0.1 for 1.2-norm) where too many states are considered reachable, when they are not, at which point the average score decreases. On the Walkor2d-medium datasct, StaCQ scems fairly robust a change in reachability as even with the large thresholds the scores do not decrease far as TD3+BC. On the dataset, when the threshold is too large, = 0.5 for the L2 and - -norms, the performance degrades massively. This is because too many states are considered reachable which are in fact not.
As a result of this study, the reachability metric chosen across all datasets the L with e 0.1, as these values produced the most reliable scores. It should be noted that other norms and thresholds could have been used to achieve similar or greater scores, however across all datasets these hyperparameters performed well.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Norm
Threshold (c)
Walker2d-Medium
Walker22AMeedhmxxppet
norm
0.01
94.1 3.1
116.0 1.8
0.05
94.4 + 1.4
104.5 #
0.1
92.1 + 1.0
94.0 + 22.8
0.5
91.9 + 2.5
0.3 = 0.6
-
0.01
82.6 + 1.3
115.5 1.551.
0.05
82.6 = 1.3
114.6 1.8
0.1
84.1 + 17.0
113.7 + 5.0
0.5
90.3 + 2.5
107.8 11.1
-
0.01
82.6 + 1.3
115.7 1.3
0.05
92.6 + 3.6
114.2 1.9
0.1
92.2 + 3.0
116.2 + 1.5
0.5
91.4 : 5.8
6.7 12.4
Table 3: Comparison of different reachability metric norms and thresholds on the Walkor2d-Meduum and Walker2d-Medium Expert datasets. The normalised score averaged for 10 evaluations providing a standard deviation over seeds.
G Learning curves for StaCQ
In this scction we provide the learning curves of StaCQ, across all datasets. During training, at 5000 gradient step intervals, the mean of 10 evaluations (for the locomotion lasks) or 100 evaluations (for the Antmaze tasks) StaCQ had been recorded across seeds. This is used to monitor the performance of StaCQ during training and the environment interactions are used for the policy Q-value updates.
24

---

Published in Transactions on Machine Learning Research (05/2025)
120
120
100
100
Norme 18c Score
Normatoos Store
80
hoppper- emed um sepertiv
hopperiaa penperrrandmma 22
walker2d modium expert
hopper-med
walkerdd rendom v2
walkerrdd menluum-replay-2
walker modium-v2
WWW
(a) Hopper
(b) Walker2d
100
80
Normallodd Score
No Normalised Score
hallcheetah--eee www.ettth-meeddrr expertor
halfcheetah-- wacaaaaa-dddddn anddmms
haltcheetan medium replay v2
hallcheetah-meduum
entiaaee Ummeze 4/2
antimaze diversee-e
0.2
0.6
1.0
0.2
0.6
3.0
Ileaatioss
Rerations
Halfcheetah
(d) Antmaaze Umaze
70
70-
60
50
50-
Nomalssd
40
30-
mul
10-
antmaae medium play
10
antmane playsvz
antmereelee aeweeeeeeeeeeee
0.0
0.2
0.4
0.6
0.8
0.0
0.4
0.6
0.8
Inerations
torations
(c) Antmaze-Medium
(f) Antmazo-Large
Figure Learning curves for StaCQ across all datasets, grouped together by environment. The mean normalisod score, over 10 evaluations (for locomotion tasks) 100 evaluations (for Antmaze tasks), is recorded for every 5000 steps during training. The solid line is the mean score across 5 seeds and the shaded region is the standard deviation.
25

---

Published in Transactions on Machine Learning Research (05/2025)
H Mult reachability: potential extensions and an illustrative example
The SCQL framework leverages one-step state reachability, A natural and promising direction for future work involves extending this concept to multi-step reachability. The core idea is to use a learned forward dynamics model, fWi to simulate k-step trajectories (sg, ao...., ak-1, sy) starting from a state sg. The crucial constraint remains that the trajectory must terminate at a state 8 that belongs the original dataset DD This allows the agent states ($1..... Sk-1) that might be out-of-distributie distribution, while still anchoring the final value estimate to reliable in-distribution data (8s= E D).
potentially bridge larger gaps in the dataset by traversing through intermediate
The two-step case reachable states ending in the dataset from a state s as:
Let us examine the specific case of two-step reachability. We define the set of 2-step
( {s2 E D Bao, a1 s.t. p(si|s, oo) and p(s2 $1, a1) =1.
Here, the intermediate state $1 does not need to be in D. Practically, reachability would be estimated using the learned model fan, denoted SR (s).
To utilise this, a modified Q-lcarning update could estimate the value ((s,s') basod on the best 2-step reachable state from s'. conceptual sketch, analogous to Eq. might look like:
max ((s,s') + (1sa) (()
v(2)
ED
(SRM.
In this sketch, ((2) (s',s") represents the estimated value of the optimal 2-step path initiating from s' and terminating at 8 E D. Accurately estimating ( using the learned model fw, (involving rewards for the first step 5 + s and the value/reward for the second step s" + s")) is a key challenge, requiring careful handling of model predictions and potential accumulated errors.
Extracting an optimal policy x (s) based on k-step reachability also more complex than the 1-step case. Conceptually, the goal is to select the first action ag that initiates the sequence leading the highest-value k-step reachable state 8% E D:
1. Find the best k-step endpoint: $ arg max
(s,)s,
SR (e)
(k)
2. Determine the first action aj on the optimal path from to s
Step 2 typically involves some form of planning (e.g., sampling action sequences with the model) or learning policy specifically optimised for this multi-step objective.
Illustratoo: maze two-step reachability The potential benefit of multi-step reachability is illustrated in the simple maze environment shown in Figure 4 The dataset provided (Figure 4 is sparse, preventing the standard onc-step SCQL from finding a path to the goal from all datasot states (Figure 4p). However, by considering two-step reachability (k = 2), the agent can plan through an intermediate state that may not be in the dataset. (indicated by dotted arrows). Assuming perfect knowledge of the dynamics for this illustrative example, hypothetical two-step state-constrained policy successfully connect all starting points to the goal region (Figure 4-), demonstrating improved trajectory stitching capabilities.
Potential challenges While promising, extending stato-constrained RL to multiple steps also introduces significant practical challenges, primarily stemming from reliance on the learned dynamics model fuy- Key challenges and potential research directions include:
Model error accumulation: Errors in fw, compound over longer rollouts (I 1), leading inaccurate state predictions and potentially unreliable value estimates for the k-step paths.
Handling uncertainty: Effectively leveraging multi-step rollouts requires managing this model uncertainty. Potential approaches include incorporating pessimism into value estimates based
26

---

Published in Transactions on Machine Learning Research (05/2025)
(a) Maze and Dataset
(b) SCQL: 1-step reachable policy
(c) SCQL: 2-step reachable policy
Figure 4: Comparison of one-step and two-step state reachability SCQL methods on simple maze envi- ronment. (a) The maze a 10 by 10 grid where the coordinate values (x, y) represent the state. The high reward region represented by a star and maze walls are represented by red grid squares. The dataset is composed of Irajectories represented by the dotted lines where the white circle the starting state and the black circle is the final state. (b) The final policy when applying one-step reachable SCQL the dataset. (c) The final policy when applying two-step reachable SCQL to the dataset, where dotted arrows represent transitions through state that is not in the dalaset.
rollout uncertainty Yu al. 2020 Rigter al. 2022). using model ensembles to quantify confidence, dynamically adapting the rollout horizon k based model reliability.
Computational cost: Finding and evaluating all k-step reachable paths terminating in can computationally demanding. Efficient scarch or sampling techniques would be crucial. Graph-basod search algorithms operating on the state space defined by the learned model could offer an alternative to simple rollouts for finding paths back to D.
Algorithmic integration: Developing robust Q-learning or policy optimization methods that correctly incorporate these multi-step, state-constrained value estimates is non-trivial, especially when considering the complexities of planning or diffcrentiating through model rollouts.
Exploring these multi-step extensions involves balancing the benefit of connecting more distant states against the cost and unreliability of longer model-based rollouts. This intersect ion of model-based planning and offline constraints presents a rich area for future investigation within the state-constrained RL paradigm.
27


Reviews:
[{"review_id": "GGWqO0tbEW", "reviewer": "Reviewer_72cw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** This work proposes a new paradigm for offline RL: focusing on the state distribution of the dataset rather than the state-action distribution. This state-constrained approach is a departure from nearly all existing offline RL methods, which are batch-constrained (i.e. they confine the policy or value updates to logged $(s,a)$ pairs). In contrast, the authors allow the agent to consider out-of-distribution actions as long as those actions lead to in-distribution states. This is a novel relaxation of the offline RL constraints – essentially enabling the agent to explore unseen actions safely by keeping it within the support of seen states. Prior methods either strictly prohibit unseen actions (to avoid distribution shift) or heavily penalize them, which can prevent the agent from improving beyond the behavior policy. By confining only to states, the proposed framework permits the policy to reach new combinations of state transitions that were never directly observed, as long as those states themselves are known. This idea has direct implications for trajectory stitching: the algorithm can piece together fragments of suboptimal trajectories from the dataset into a better trajectory that achieves higher reward. Rather than being limited to following the dataset action-by-action, the agent can stitch via adjacent states – effectively jumping from one trajectory to another at a common or nearby state. This capability is a unique contribution and addresses a known challenge in offline RL, where improving beyond the given trajectories is hard. The paper illustrates this advantage in concept with Figure 1: the state-constrained method identifies high-value reachable states (yellow circles) that were not directly connected in the dataset and uses them to improve the policy. This is a clear innovation over batch-constrained methods that “lock” the policy to recorded actions and cannot leverage such alternate transitions. \n\n**Weakness:**  \n\n1. Assumption of Deterministic Dynamics: The theoretical framework and experimental setup assume deterministic environments, which simplifies the definition of reachability (i.e., an action deterministically leads to a specific next state). Most real-world environments have stochastic transitions due to noise, uncertainty, or partial observability. In such cases, defining and verifying whether a state is “reachable” becomes less clear.\n\n2. One-Step Reachability Only: The method only considers one-step reachability, i.e., whether a state can be reached from another in a single action. This limits the ability to stitch longer sub-trajectories, potentially capping performance in long-horizon or sparse-reward settings.\n\n3. Limited Exploration of Stochastic or Realistic Benchmarks: All experiments were done in relatively deterministic simulators (MuJoCo, AntMaze, grid-worlds). The generalizability to real-world applications or highly stochastic tasks is uncertain.\n\n4. No Direct Safety Guarantees: Although StaCQ only considers actions that lead to known states, it does not provide formal guarantees about safety or worst-case performance in unknown situations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a formal state-constrained offline RL framework grounded in new theoretical insights. It defines state reachability (Definition 3.1) as the set of next states obtainable from a given state via some action in a deterministic MDP. This notion allows the Q-function to consider multiple possible next states per state, rather than a single observed state-action pair. Based on this, the authors formulate QSS-learning, where the Q-value is defined over state-to-state transitions $Q(s,s’)$ instead of state-action pairs. Under standard assumptions ((A1)-(A4): deterministic environment, bounded rewards, finite initial Q-values, $0\\le\\gamma<1$), they prove that this QSS-learning converges to the optimal state-to-state value function (Theorem 3.2) by a straightforward extension of classic Q-learning proofs. The theoretical development is rigorous, adapting known results (e.g. convergence of Q-learning in deterministic MDPs) to the new setting with appropriate justification. Proofs are provided (many deferred to the appendix for brevity), and they appear logically sound given the assumptions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please answer the questions for the weakness part in above.", "broader_impact_concerns": "No Broader Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TrVsxYgznl", "reviewer": "Reviewer_XLFq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n•\tThe proposed idea is conceptually straightforward.\n\n•\tThe method effectively constrains learning within a reachable state space, which may improve robustness in offline RL.\n\n**Weaknesses:**\n\n1.\t**Theoretical Limitations:**\n\n•\tThe theoretical analysis is confined to deterministic MDPs, which limits its applicability to broader RL settings.\n\n•\tThe provided analysis only establishes the possibility of convergence but lacks a quantitative learning efficiency analysis. This makes the theoretical contribution more of a descriptive validation of QSS learning rather than a comparative study with traditional QSA learning. A theoretical comparison, particularly in terms of convergence rate and sample complexity, would strengthen the contribution.\n\n2.\t**Writing and Mathematical Rigor:**\n\n•\tThe key theorems are stated in natural language rather than using formal mathematical expressions. To improve clarity and rigor, I recommend expressing these results mathematically.\n\n3.\t**Maze Navigation Experiment Issues:**\n\n•\tThe direct given of the state-reachable domain in the maze navigation task appears to be a form of implicit cheating, as it directly provides environment information rather than learning it from data.\n\n•\tThis approach may also introduce errors, as it does not consider environmental constraints such as walls blocking the agent’s movement.\n\n•\tA fairer comparison should involve learning the state-reachable domain from the dataset rather than assuming it a priori.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a simple yet effective idea—QSS learning operates within a restricted state-reachable domain.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "•\tExtend the theoretical analysis beyond deterministic MDPs and provide quantitative learning efficiency guarantees.\n\n•\tPresent key theoretical results using formal mathematical expressions.\n\n•\tModify the maze navigation experiment to ensure that the state-reachable domain is learned rather than manually provided.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GbIeVIszA7", "reviewer": "Reviewer_xzyQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The paper is well-written. I enjoyed my reading of this work.\n- The proposed method is novel and intuitive. Constraining on reachable states allows more action generalization than the batch constraints. \n- The illustrative experiment presents the motivation of the SCQL.\n- The proven convergence and optimality properties allow the comparison of the SCQL and BCQ, showing the theoretical advantage of SCQL.\n\n# Weaknesses\n- The implemented agent of SCQL applies ensembles of forward, inverse, and critic models. In addition, the implementation of the Locomotion task is different from that of the Antmaze task in using a shared/independent target network, and whether to use a reward model. Although using these techniques does not count for weaknesses, their effect on performance is not discussed in this paper, which makes it vague whether the proposed method itself is the main source of the presented performance improvement (Table 1)\n\n- The comparison shown in Table 1 uses different hyperparameter tuning strategies for each agent. For example, the TD3-BC (Fujimoto et al.[1]) and IQL (Kostrikov et al.[2]) are taken from the original papers, where similar hyperparameter values are used for all Locomotion tasks. However, StaCQ uses domain-specific values for its hyperparameters. Some baseline agents might perform better with their parameters tuned for each environment. Based on this, Table 1 does not present a clear comparison.\n\n## Reference\n[1] Fujimoto, Scott, and Shixiang Shane Gu. \"A minimalist approach to offline reinforcement learning.\" Advances in neural information processing systems 34 (2021): 20132-20145.\n[2] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline Reinforcement Learning with Implicit Q-Learning.\" International Conference on Learning Representations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel algorithm constraining the policy to produce actions that lead to reachable states within the datasets. While popular batch-constrained methods confine the policy to the state-action pairs from the dataset, this work proposed constraint allows for state-action pairs that do not exist in the datasets but are reasoned from the dataset. The optimality related to the proposed method is proofed, and experiments on the D4RL benchmark present better performance than the selected baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- A performance comparison between agents using a consistent hyperparameter tuning (see weakness 1) is needed. Either using domain-specific hyperparameters or similar values for all environments.\n\n- A discussion about the StaCQ components (See weakness 2) and their influence on the performance is needed.\n\n- A missing definition of $s_{terminal}$ (Definition 3.3). Is $s_{terminal}$ used in this work similar to the definition from Fujimoto et al., [1]? Please define this notation in this work.\n\n- Could you provide the environment penalty and reward values of the maze environment (Section 3.2). Currently, the paper only mentions *\"a small negative value\"* and *\"rewarded with a large positive value\"*. Providing these values and the training details (e.g. agent hyperparameters) will help the readers further understand the experiment.\n\n### Minors\n\n- In the Introduction section, *\"However, adhering strictly to the state-action distribution can be restrictive. Thus, recent techniques...\"* The first sentence makes a claim that is not discussed. A brief discussion about \"can be restrictive\" is needed here.\n\n- Section 3.2, *\"With each movement, the agent incurs a reward penalty of a small negative value, but upon...\"*, what is the \"reward penalty\"? Here, it should be a penalty, given the context.\n\n- Consistency of using \"co-ordinate\" and \"coordinate\"; \"black circle\" \"white ones\" and \"closed circle\" \"open circle\"\n\n- The notation $Q(s, \\pi(s))$ or $f(s, \\pi(s))$ implies a deterministic policy $\\pi(s)$. Although this paper considers deterministic policy in their experiments, please explicitly define the deterministic policy $\\pi(s)$ in Section 4.2.\n\n## Reference\n[1] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052–2062. PMLR, 2019.", "broader_impact_concerns": "I believe that this work does not raise concerns regarding Broader Impact", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SE6oGFDY01", "reviewer": "Reviewer_vVkQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed approach is interesting and worth studying.\n- The paper is well-written and explains necessary background (QSA-learning and QSS-learning) at an appropriate level of detail.\n- The paper contains theoretical and experimental results showing the efficacy of the proposed approach and the implementation.\n- The mathematical notation is clean and understandable.\n\nWeaknesses:\n\n- Overall, this is, in my opinion, solid work and suitable for publication in TMLR. There are only minor weaknesses, which I list in the requested changes section of my review.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses offline reinforcement learning with the additional constraint that the states need to exist in the training dataset. Existing theoretical results by Fujimoto et al. are extended to the proposed state-constrained QSS-learning (SCQL) approach and a practical implementation of SCQL (called StaCQ) is evaluated on the D4RL environments for offline RL and compared with model-free baselines (BC, BCQ, TD3+BC, IQL) as well as model-based baselines (MBTS, Diffuser, RAMBO). A variant of their implementation that learns the values and policies together (called OneStep StaCQ) is also discussed and evaluated.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The claims of this work should be stated in a more clearer way. I suggest the authors add a section that clearly states the claims/contributions made in this work.\n- Formatting of the references can be improved at some points (e.g., “q(s, s’)”, “D4rl”, “offline rl”, “q-learning”)\n- Minor comments regarding notation:\n    - \\argmax should be properly defined. Please, do not use \\arg \\max.\n    - In Eq. (5) and (6) the subscripts to the max could be improved: Either write this as a intersection of the two sets $\\mathcal{D}$ and $\\mathcal{SR}_\\mathcal{M}(s)$ or add a logical and-operator.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As previously mentioned, most of the pushback on this paper was of the form 'please do more', which shouldn't count against its acceptance, as what is there is sound and relevant.
Decision: Accept (poster)