Sample Efficient Offline RL via T-Symmetry Enforced Latent State-Stitching
Authors: Peng Cheng
Zhihao Wu
Jianxiong Li
Ziteng He
Haoran Xu
Wei Sun
Youfang Lin
Yunxin Liu
Xianyuan Zhan
Venue: ICLR 2026 Poster (2026)
Keywords: sample efficiency
representation learning
fundamental symmetry for dynamic modeling
Primary area: reinforcement learning
TL;DR: we propose a sample-efficient offline RL algorithm that achieves strong OOD generalizability, significantly outperforming existing offline RL methods in a wide range of challenging small-sample tasks.

Abstract: Offline reinforcement learning (RL) has achieved notable progress in recent years. However, most existing offline RL methods require a large amount of training data to achieve reasonable performance and offer limited out-of-distribution (OOD) generalization capability due to conservative data-related regularizations. This seriously hinders the usability of offline RL in solving many real-world applications, where the available data are often limited. In this study, we introduce TELS, a highly sample-efficient offline RL algorithm that enables state-stitching in a compact latent space regulated by the fundamental time-reversal symmetry (T-symmetry) of dynamical systems. Specifically, we introduce a T-symmetry enforced inverse dynamics model (TS-IDM) to derive well-regulated latent state representations that greatly facilitate OOD generalization. A guide-policy can then be learned entirely in the latent space to optimize for the reward-maximizing next state, bypassing the conservative action-level behavioral regularization adopted in most offline RL methods. Finally, the optimized action can be extracted using the learned TS-IDM, together with the optimized latent next state from the guide-policy. We conducted comprehensive experiments on both the D4RL benchmark tasks and a real-world industrial control test environment, TELS achieves superior sample efficiency and OOD generalization performance, significantly outperforming existing offline RL methods in a wide range of challenging small-sample tasks.

Published as a conference paper at ICLR 2026
SAMPLE EFFICIENT OFFLINE RL VIA T-SYMMETRY ENFORCED LATENT STATE-STITCHING
Peng Cheng*11.4 Zhihao Wu" 1.4 Jianxiong Li,, Ziteng He², Haoran Xu² Wei Sun", Youfang Lin Xianyuan Zhan
1 Beijing Jiaotong University, 2 Tsinghua University, 3 University of Texas at Austin, Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence ce12nn66126.. com, { (zhwu, yflin) @bjtu. www..u...c.m cn, whanai aunxianyuan@aii wetpeeaarrrssiiinnne tsinghua. eechhhuaaedu..... cn
ABSTRACT
Offline reinforcement learning (RL) has achieved notable progress in recent years. However, most existing offline RL methods require large amount of training data to achieve reasonable performance and offer limited out-of-distributiin (OOD) generalization capability due conservative data-related regularizations. This seriously hinders the usability of offline RL in solving many real-world appli- cations, where the available data are often limited. In this study, we introduce TELS, a highly sample-effiieent offline RI. algorithm that enables state- sate-stitching in a compact latent space regulated by the fundamental time-reversal symmetry (T-symmetry) of dynamical systems. Specifically, we introduce T-symmetry enforced inverse dynamics model (TS-IDM) to derive well-regulated latent state representations that greatly facilitate OOD generalization. be learned entirely in the latent space to optimize for the reward maximizing next state, bypassing the conservative action-level behavioral regularization adopted in most offline RI. methods. Finally, the optimized action can be extracted using the learned TS-IDM, together with the optimized latent next state from the guide-policy. We conducted comprehensive experiments on both the D4RL benchmark tasks and real-world industrial control test environment, TELS achieves superior sam- ple efficiency and OOD generalization performance, significantly outperforming existing offline RL methods in a wide range of challenging small- -samle tasks.
solide-poicy then
INTRODUCTION
Offline reinforcement learning (RL) has seen rapid progress in recent years. It directly utilizes pre-collected datasets for policy learning, making them ideal for many real-world tasks that lack high-fidelity simulators or have restrictions on environment interaction (Levine et al., 2020; Zhan al., 2022a). However, offline RL is also known to be prone to value overestimation, caused by extrapolation error when evaluating out- distributto (OOD) samples and amplified through the bootstrapped update procedure in RL (Kumar et al., 2019; Fujimoto et al., 2019). In the past few years, quite a few offline RI. methods have been proposed, which commonly adopt the pessimism principle using strategies such as adding explicit or implicit policy constraints to prevent the selection of OOD actions (Kumar et al., 2019; Wu et al., 2019; Fujimoto et al., 2019; Fujimoto & Gu, 2021; Li et al., 2022), penalizing value function on unseen samples (Kumar et al., 2020; Bai et al., 2021; Lyu et al., 2022), adopting in-samlle learning to implicit regularize policy optimization (Kostrikov et al., 2022; Xu et al., 2023; Mao et al., 2024b; Wang et al., 2024; Zheng et al., 2024). Adopting such action-level constraints, although helpful stabilize offline value and policy learning, also leads to over-conservatism and crippled OOD generalization performance (Li et al., 2022; Cheng et al., 2023). Most of the existing offline RL methods only perform well when trained with sufficiently large offline datasets with reasonable state-action space coverage (c.g., million samples for simple
"Equal contribution.
"Correspondigg Author.
A Work done during internship Institute for AI Industry Research (AIR), Tsinghua University.

---

Published as a conference paper at ICLR 2026
D4RL benchmark tasks (Fu al., 2020)). This forms stark contrast to the reality in most real- world scenarios, such as industrial control (Zhan et al., 2022a; 2025), robotics (Sinha et al., 2022), and healthcare (Tang et al., 2022), where the real-world data are often scarce, and scaling up data collection can be rather costly.
Enhancing sample efficiency and OOD generalization capability is essential to making offline RL widely applicable to real-world applications. This is particularly important for small dataset settings, as most of the state-action space will become OOD regions. Several recent attempts have been made to improve the generalization performance of offline RL, which mainly follow three directions. The first direction builds upon the empirical observation that deep value functions interpolate well but struggle to extrapolate, thus allowing exploitation on interpolated OOD actions to promote generalization (Li et al., 2022). However, this method has a smoothness assumption on the offline dataset geometry and only applies to the continuous action space. The second class of methods avoids the conservative action-level constraint and instead performs reward maximization on the state-space (Xu et al., 2022a; Park et al., 2024), which allows exploitation of OOD actions long as the corresponding state transitions are reachable (also referred to as "state-stitchingg) ") Although these methods offer some promising generalization capabilities, they still require the state-action space to have reasonable data coverage to enable valid state-stitching. Finally, the last direction to learn compact and robust latent representations to enhance sample efficiency. Most methods in this direction focus on extracting statistical-level information from the data, using techniques such as contrastive learning (Laskin al.. 2020; Agarwal et al., 2021a; Yang & Nachum, 2021; Uehara et al., 2021). Due to insufficient consideration of the underlying dynamics inside sequential data, these methods still struggle to provide generalizable information beyond data distribution. Some recent methods propose to learn representations that extract fundamental symmetries of dynamics to facilitate policy learning (Weissenbacher et al., 2022; Cheng et al., 2023), such as the time-reversal symmetry (T-symmetry) (Cheng et al., 2023), i.e., the underlying physical laws should not change under the time-reversal transformation. By leveraging such universally held symmetries in dynamical systems, it is possible maximally promote OOD generalization without information. Although promising, these methods are
being restrained by data distribution-relan built upon offline RL backbone algorithms with action-level constraints (e.g., CQL (Kumar et al., 2020) or TD3+BC (Fujimoto & Gu, 2021)), which still suffer from the over-conservatism issue.
In this paper, find that enabling state-stitching in coherent, fundamental symmetry-enforeee latent space can lead to a surprisingly strong sample-efficien offline RL algorithm. We refer to our method as Offline RL via T-symmetry Enforced Latent State-Stitching (TELS). Specifically, we introduce a T-symmetry enforced inverse dynamics model (TS-IDM) that can not only learn well-behaved and OOD generalizable latent state and action representations, but also facilitate effective action inference. Within the learned well-behaved latent state space, we can optimize a T-symmetry regularized guide-policy to output the next latent state that maximizes the accumulated reward, bypassing the conservative action-level behavioral regularization adopted in most offline RL algorithms. Lastly, the optimized action can be easily extracted by plugging the output of the guide-policy as the goal state in the learned TS-IDM. We evaluate TELS on both the challenging reduced-size D4RL benchmark tasks and a real-world industrial control test environment. Through comprehensive experiments, show that TELS achieves state-of-the-an art (SOTA) sample efficiency and OOD generalization capability, significantly outperforming existing offline RL algorithms on small datasets.
2 PRELIMINARIES
Offfine RL.. We consider the standard Markov decision process (MDP) setting (Sutton & Barto, 2018), which is represented as a tuple M = (S,A, ".,., 7). and dataset D, which consists of trajectories
{so, do, $1, 01 ... ST, ar). Here S and A denote the state and action spaces, r(s, a) is a scalar reward function, P(s' s, a) and p denote the transition dynamics and initial state distribution respectively. and E (0, 1) is discount factor. Our goal is to learn policy T(als) based on dataset DD by maximizing the expected return in the MDP: E. 1E2 7 r(st, at)l
Offline policy optimization in the state space. Instead of adopting conservative action-level constraints for offline policy learning, Policy-guided Offline RL (POR) (Xu et al., 2022a) proposes to decompose the conventional reward-maximizing policy into a guide-policy and an execute policy.
2

---

Published as a conference paper at ICLR 2026
The guide-policy only works in the state space to find the optimal next state that maximizes the state-value function, and the execute-policy policy is learned as an inverse dynamics model (Xu et al., 2022a) goal-condit imitative policy (Park et al., 2024). Such methods only need to learn state-only value function V using the IQI- expectile regression, as proposed by Kostrikov et al. (2022), or the sparse value learning objective discussed in (Xu et al., 2023). We present the former as follows:
V = arg min B(annYND b2 (r())+ + +V
1(x (T- < 0)(2² is the asymmetric expectile regression loss and V denotes the target value network. Based the learned state-value function, can learn guide-policy Tg (5'(s) [s) to serve as a prophet by telling which state the agent should (high reward) and can (logical generalization) go to, without being constrained to state-action transitions seen in the dataset. This can be achieved by leveraging an advantage weighted regression (AWR) objective (Neumann & Peters, 2008; Peng et al., 2019) to maximize the value while implicitly constraining Rg to s 8 transitions observed in the dataset (i.e., state-stitching):
where Lī(x)
Tg = arg max E(s. exp(a A(s, s')) log Tg(s') ss)
(2)
where the advantage -r the temperature parameter to prioritize value maximization over state-wise imitation.
V(s) ((())
serves as the behavior cloning weight, and a is
For the execute-policy Te, POR employs a supervised learning framework and trains ne by maximiz- ing the likelihood of the actions given the states and next states: max [ss.,,""~= xe (a s, s,,')) During evaluation phase, given the current state s, we can sample the optimized next state s' from T (6'(s), and get final action simply as a* = TT (a 8, Ty (s"[s))
Time-reversal symmetry for generalizable offline RL. Recently, leveraging fundamental, univer- sally held symmetries of dynamics like T-symmetry discovered in classical and quantum mechan- ies (Lamb & Roberts, 1998; Huh et al., 2020) has been shown to be promising approach to enhance the generalization of offline RL (Cheng et al., 2023; Zhan et al., 2025). Specifically, if we model the system dynamics with measurements x set of non-linear first-order differential equations dt = F Fx). dynamical system is said to exhibt time-reversal symmetry there is an invertible transformation I that reverses the direction of time: i.e., dT /dt For the discrete-time MDP setting, the T-symmetry can be extended as learning a pair of ODE forward dynamics F(s,a) a) + 8 and reverse dynamics G(s', a) - 8, and require them to satisfy F(s,a) -G(s, a), where the time-derivative of state à ds is approximated s' 8..
(ODEs) expressed as dx
F[[(x))
Based on this intuition, TSRL (Cheng et al., 2023) constructed an encoder-decoder structured T- symmetry enforced dynamics model (TDM) for representation learning, which embeds a pair of latent ODE forward and reverse dynamics to enforce T-symmetry. TSRL achieves impressive performance under
settings, and its variant has been successfully deployed for real-world industrial control (Zhan et al., 2025), but it still has some limitations. First, TSRL only uses the learned encoder from TDM to derive the latent representations, without fully exploiting the rich dynamics- related information for downstream policy learning. Second, its representation learning scheme uses both state and action as inputs, forcing TSRL to involve policy-induced actions during policy optimization, which inevitably requires adding conservative action-level behavioral constraint as in TD3+BC (Fujimoto & Gu, 2021) to stabilize training. Moreover, involving action as an input for representation learning is also prone to capturing biased behaviors in the behavioral policy, which could impede learning fundamental, distribution-agnn aggoostic dynamics patterns in data. Please refer to Appendix A for more detailed comparison and discussion.
3 OFFLINE RL VIA T-SYMMETRY ENFORCED LATENT STATE-STITCHING
We present proposed method, TELS, which comprises T-symmetry enforced inverse dynamics model (TS-IDM) integrated with an effective offline policy optimization procedure operated in latent state space (illustrated in Figure 1).
3

---

Published as a conference paper at ICLR 2026
T-ymmeetry enforced inverse dynamic model (TS-IDM)
Latent state value function learning
Leariing the Ideett space
(.).(5) $ 8 (9)
min Beun"") (4
$
hyud
T-symmetry regularized guide-poliey optimization
max Risa')- Ael (Rg(te) -n 11 (fo(ee)
=I]
Er-syer (ze, Aew (z..
or
max (ea)) - (exp(ar- Alto 2, logzy (15115)) by
Taymmerry regulraase asttionn
heys
Ey Syrs Pam (g (z,)
Action inference
Extract optimized
latent
State encoer
Latent Inverse dynomics
Latent ODE forward and reverse dynamics
State and action decoders
9o Aese (3..7
Figure I: Overview of T T-ymmeerry Enforced Latent State- Statc-Stitching (TELS) framework. Left: The illustration TS---D structure. Right: The process of training T-symmetry regularized guided-policy.
3.1 T-SYMMETRY ENFORCED INVERSE DYNAMIC MODEL
As illustrated in Figure 1, if inspecting the input and output of our proposed TS-IDM, it functions similarly to an inverse dynamics model that takes current and next state (s, s') as input and outputs the predicted action a. However, in its interior, TS-IDM comprises state encoder (s = 2, and corresponding decoder (2.) = 8; latent inverse dynamics module hinn Ze') = 2a followed by an action decoder va(=)) à: and most importantly, a pair of T-symmetry enforced latent ODE forward and reverse dynamics predictors f za) = å, and ( za) = 2g.. All these sub-components are implemented as simple 2-layer ML.ss In the following, we will dive into the design intuitions and learning objectives of these components.
Encoding and decoding. As previously discussed, constructing an informative and well-structured latent space is critical for sample-efficient offline policy optimization. To this end, we introduce state encoder &, (s) = Ze to map a state 8 into corresponding latent representation zg. and also a state decoder (2.) = to reconstruct the original state from its latent embedding, ensuring that the learned latent representations remain faithful to the original state space and avoid excessive distortion.
We then construct latent inverse dynamics module hine (Eey 2 Za. which infers the latent action za from the latent state transitions (2,, 2. By inferring actions from state transitions, the learned latent space implicitly encodes the underlying dynamics of the environment. Moreover, the inverse dynamic module hive can be integrated with a pair of latent ODE dynamics predictors ( and hyna) derive the T-symmetry property of the system, which we will introduce in more detail shortly. Finally, to ensure that the inferred actions are both meaningful and interpretable, we employ an action decoder (2a)
map the latent action back to original action space. We can thus formulate the reconstruction loss for the states and actions as follows:
(s, a, 5')
(e. (s) s11 1 (hino )) all
reconstruction loss of states
reconstraction lres of actions
Latent ODE forward and reverse dynamics. Drawing inspiration from previous research that integrates physics-informed insights into dynamical systems modeling (Brunton et al., 2016; Cham- pion et al., 2019; Huh et al., 2020; Cheng et al., 2023), we embed pair of latent ODE forward and reverse dynamics h w 2 2, and (t 2.) -2, to separately capture the forward and reverse time evolution in the latent states. We are interested in modeling ODE systems because encourages learning parsimonious models helpful to uncover fundamental properties from the data that can maximally promote generalization (Brunton et al., 2016; Champion et al., 2019). Note that based on the chain rule, we can derive the supervision signal for the latent dynamics modules with 2, dz do.. ds V 8 V.O,(s) á enforce the ODE property. Therefore, we introduce the following training losses for o and bros:
(s))
[[ 25 2 112 + II(V of 25 ((
2-2-)11
latent ODE forward dynamics
latent ODE roverse dynamics
(.))(s) ()dd(2., 2a 20)112 |
V. (s'((''') (rs (2s1, 20)1
(4)
4

---

Published as a conference paper at ICLR 2026
where the latent action 2. is obtained from the latent inverse dynamics module (2s 22)).
ODE property enforcement on state decoder. Note that in (sya(s, s'), we actually implicitly enforced the ODE property on the state encoder 6ss the same should also apply to the state decoder V to ensure compatibility with the T-symmetry formalism, i.c. the time-derivative of the state encoder do.(s) and decoder (va(=a) should behave in the same way as éa and s. Similar to the dt
previous treatment on the state encoder, as á dv,(s,) dv.(ss) da V., dgs we can dz, de use the following objective to enforce the ODE property for the state decoder eb
dt
Pode (8, enforce ODE of ve, on Fo
| 2, (2e( 2.
812 + |1 sot (2x/) (-2.) - (-8)112
enforce ODE v., hyr.
=| 2, (2,) ( za) $11 + V V,,, (a(z)) hrrs 2a) + -112-
(5)
Notably, the ODE property enforcement in Eq. (5) is not considered in the T-symmetry enforced dynamics model (TDM) proposed by TSRL (Cheng et al., 2023). In other words, TDM only enforces the ODE properties for encoders but not for decoders. This can cause inconsistency between the learned dynamics and the underlying ODE structure, leading to inaccurate ODE representations.
T-symmetry enforcement. To further regularize the learned latent representations, enforce T-symmetry by requiring h/wa(2s) za) = - hrve (Zg', 2a), which corresponds to the following loss:
(T-yy za) = [| I za) + hrve (Zs + (zad za), za)
(6)
where we use the fact that Zg' = 24 + dg =e + f and hrve (2g + h za); [a)
-2,- -h za) to further couple the learning process of h fod and hres. Moreover, given latent state-action pair (2e. Za). the above T-symmetry consistency loss can also serve as an evaluation metric assess their agreement with the learned TS-IDM. A large T-symmetry loss indicates that the latent state-action representation (z,, za) induced by some (s, s') may not satisfy the fundamental dynamics pattern, making it more likely to be problematic non-generalizable sample.
Overall learning objective. Finally, the complete training loss function of TS-IDM is as follows:
LTS-IDM
Σ
free + B- (Pdyn Lode + (T-sy)) (s, a.,.s)).
(s,D,S")E
where B is hyperparameter that balances extracting fundamental dynamics properties and ensuring the interpretability of the learned representation. Note that we employ a single shared 3 for laya- lode: and fTaym terms. This is to ensure that the ODE property and T-symmetry regularization are enforced on a consistent scale for these strongly coupled loss terms, while also reducing the number of unnecessary hyperparameters. This actually critical, as we have empirically shown in Appendix B.5, using a shared | enables stable training, while adopting separate weights can cause a substantial performance drop. Despite containing multiple sub-modules, our proposed TS-IDM is actually quite small (based on several simple MLP layers), which can be efficiently and stably learned owing to its highly coupled design. The entire training process can be completed in merely 20 minutes and minutes in our PyTorch and JAX implementations, respectively (see Table 12 in Appendix).
3.2 LATENT SPACE OFFLINE POLICY OPTIMIZATION
Once we have learned TS-IDM, we can extract three highly useful components from to facilitate sample-efficient downstream offline policy optimization, including: 1) the state encoder 6(s) that provides an ideal, well-behaved latent space for state-stitching; 2) T-symmetry consistency as an additional regularizer to prevent erroneous generalization when learning guide-policy in the latent state space; and 3) the TS-IDM itself can serve as an execute-policy as in POR (Xu et al., 2022a) to extract optimized action given a learned guide-policy.
Latent state-value functions learning. Based on the state encoder (s) from the learned TS- IDM, we can convert the entire offline policy optimization process into the latent state space, which enjoys both stable learning process and generalizability due to more compact and well-behaved representations. Specifically, we can use a similar expectile regression loss as in Eq. (1) learn state-value function V22,), but in the latent state space:
min ((er)s)) LE (r + ( (
(8)
5

---

Published as a conference paper at ICLR 2026
T-symmetry regularized guide-policy optimization. A key benefit of learning within the T- symmetry preserving latent space is that, as T-symmetry captures what is essential and invariant about the dynamical system, it can provide generalizable information even for OOD samples beyond the offline dataset. This naturally favors learning a swwardmmaimizzing guide-policy "g in the latent space, which can enjoy more effective state-stitching. Moreover, by leveraging the T-symmetry consistency term (T-yy((-) in Eq. (6) as an additional regularizer, we can prevent Ty from outputting problematic and non-getn non-generaliiaale latent next state, thereby further enhancing logical state-wise OOD generalization. In TELS, we provide two instantiations for guide-polics optimization, depending on the choice of using deterministic policy Ty (2s) or stochastic policy Ty (2ar
Deterministic policy: vll (aV((o(t))))))) - - s'12 - frssym (Zer Nine (Zar Th (z(())))
max (D (Tg(())) 8
(9)
Stochastic policy:
max E (s expla- A(zs, 2x))) log (2s 2x) &y (2y, hino (2,, Tg og ((1())))
(10)
Fa
where 2g = Ez. = (s')). and A(2,, (er) = (2s/) -V V- (2&). For the deterministic policy (2,), optimize the guide-policy by maximizing the latent state-value function V weighted by normalization term Aa. together with two extra regularization terms. The first regularizes the next state decoded from the guide-policy using state decoder ei, should not deviate too much from the next state s' in the dataset. The second term regularizes the guide-policy induced latent state-action pair (i.e., (Z,: =a) (zg. (2,, m (2,))) to comply with the T-symmetry consistency specified in the learned TS-IDN -D-M.MM. For the stochastic guide-policy Tg(2z/|2)) we adopt a similar AWR-style objective as in Eq. (2), while also incorporating the T-symmetry consistency regularization as in the deterministic version. In our experiments, we find that the deterministic version of the objective Eq. (9) works well for the MuloCo locomotion tasks, while the stochastic version Eq. (10) works better for more complex Antmaze tasks, potentially due to the more stochastic nature of the task environment.
Action inference. After learning the guide-policy Tg. can further use it to extract the optimized action for control. To do this, we can simply use the optimized latent next state 2 obtained from guide-policy ng (2x) or x-(-12e) ) as the goal state, and plug it into the learned latent inverse dynamics module hime 22r) ) in TS-IDM to replace 2,' The final action can be extracted by decoding the resulting latent action from hiv using the action decoder tha
a" = tha (hino (Zo, ag (2+))).
(11)
Note that there is no training process needed for this stage. Moreover, throughout our policy optimization process, actions are not involved, allowing TELS to completely bypass the conservatism issue caused by the action-level regularization. Please refer to Algorithm 1 in Appendix C for the detailed implementation, as well as the training and inference procedure of TELS..
EXPERIMENTS
In this section, we present the evaluation results of TELS on the D4RL benchmark tasks (Fu et al., 2020) against behavior cloming (BC), and existing offline RL methods: TD3+BC (Fujimoto & Gu, 2021), CQL (Kumar et al., 2020), IQL. (Kostrikov et al., 2021), DOGE (Li et al., 2022), POR (Xu et al., 2022a), model-based methods MOPO (Yu et al., 2020) and COMBO (Yu et al., 2021b), diffusion-based method IDQL (Hansen-Estree current SOTA method in small-sample settings. To demonstrate the effectiveness of TELS in solving real-world tasks, we also validate TELS in a real-world industrial control environment, which is a data center (DC) cooling control testbed built by recent work (Zhan et al., 2025). Moreover, conduct additional experiments to evaluate the OOD generalizability of TELS on a challenging task, and the strengths of the representations learned with TS-IDM in improving small-sample performance.
et al., 2023), and TSRL (Cheng et al., 2023), the

---

Published as a conference paper at ICLR 2026
Table Normalized scores reduced size D4RL datasets (averaged over the final 10 evaluations with 5 seeds). We report standard deviations after + sign. Numbers at or above 95% the best value the row are highlighted in bold.
Task
Size (ratle)
BC
TD3-BC
MOPO
COMBO
CQL
RQU.
DOGE
IBQU
POR
TSRL
TRLS
Reyperon
(19)
55..2.
10..1 2611
4111.416
467161
442 10.1
64211711
16.41.11
63.0133
10.7
Hupperine
16: (2.5%
1M
7.3611
65:003
179.445
317670
174462
21,5482
412 M
Dopperoee
1K (1.3)
37.8 06.
138.38
56..58
139 220
19.045
363006
505 55.
433144
1.61
5091KA
100. 1.88
Hilcheerah-m
(1%)
M:4:73
4.1.4
358-38
2994013
363334
964:13
3333632
164431
(85)
143.28
1999.9
1.2.4.52
118
81194
227184
263.1.
27,136
263435
362 1.0
Haldbeera--me
10K (1.58)
191-94
1564407
111-14
53-466
105488
4.7.4.66
188--18
54.4-66
M991211
41.
36:111
WelkerMerrr
10 (1.9)
1.4419
57188
X1-627
42 156
6.903.0
107111
135 846
12.24005
148143
2601113
5488 6/
Waleralme
10k (0.59)
21.28822
7.991
06
01:01
191844
265386
38. 13.6
2188148
201136
46.41174
(1%)
0.
0.9
0.0
53
6 10.
073
61
7 15.6
Antronnd
H6 (19)
201 153
16,37
164
0.0
01:000
146
185
41.7
951.4.90 16.
550.4
Antame--d
1000 (109))
0.0
0.0
0.0
00
00
4.
0.0
90:33.
0.0
00
4 173
Antmare-ii
TIMR (1015)
0.0
0.
0.0
000
00
4.
0.0
16144444
00
00
19.8 14.1
Antrennipp
(8095)
0.
0.0
1:43
00
98.188
0V0
Pen-herron
(1078))
9.7
62.6
16.1
661 25.1
(1025))
-02
403
-11.4
011.1.0 0.1
05 1.000
11.8
Reloeato-hersnn
9: (1005)
0.
-03
-02
-0.
0.2
0.8
011 0.2
0.7 0.01
01:101
0.1 0.111:
Normalized Score
125
Normalized Score
125
100
300
25-
25
25
DC
CTL
KL.
DOGE
1001
PDR
T5RL
T115
BC
TOS+ MCMOFFOOOOO
COL
105.
DOGE
1041.
POR
TSRL
T13
Hopper
2M
100%
50k
10k
WalkerZd-me
Figure 2: Performance of TELS against baselines under different data sizes. The error bars represent the standard deviation calculated over 5 random seeds.
4.1 COMPARATIVE EVALUATION ON SMALL-SAMPLE SETTING
Evaluation on D4RL benchmarks. In Table we evaluate TELS against baseline methods on challenging reduced-size D4RL datasets (5k~1 samples, about 0.5~100% of their original sizes)'. These amall-sample tasks are particularly challenging for offline RL algorithms, as the data only sparsely cover the state-action space and require strong OOD generalization capability for algorithms to achieve reasonable performance. Results on full D4RL datasets can be found in Appendix B.I.
As shown in Table 1, most baselines fail to learn reasonable policies under small datasets, especially in the challenging 100k Antmaze medium/large datasets. For example, conventional offline RL methods like TD3+BC and CQL perform poorly small datasets, primarily due to their over-conservative data-related policy constraints. Model-hased methods also perform badly due to insufficient samples to learn accurate dynamics models and the use of problematic model rollout data. Baselines that have generalization promotion designs, such as DOGE and TSRL, perform slightly better but still fail miserably in the challenging Antmaze-m/l tasks, they still adopt conservative action-level constraints to stabilize policy learning. Recent diffusion-based methods like IDQL, although perform well on large datasets, struggle learn when given limited data. By contrast, TELS dominates the chart and outperforms all other baselines in all tasks, sometimes by large margin. This is attributed to the leverage of fundamental, data Instribution-agnnooe which greatly improves the OOD generalization performance. This is evident when observing the huge performance difference between POR and TELS, as the former shares a similar policy optimization procedure but does not use the T-symmetry enforced representation and policy regularization.
T-symmetry property for policy learning,
We also evaluate the performance of the algorithms across different dataset sizes in Figure 2. The results show that TELS can robustly maintain reasonable performance even with only 5k samples,
"We use the same seduced-size MuJoCo datasets from the TSRL paper (Cheng et al., 2023), and randomly sub-sample 100k Antmaze datasets for experiments. We use the original Adroit-human datasets for evaluation, they are already small.
7

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation results in the real-world DC cooling control testbed (6-hour length experiments). Results with the lowest ACLF under zero thermal safety violations are highlighted bold.
Testbed
CQL
IQL
TSRL
TELS
Server energy consumption (kWh)
41.44
39.80
40.30
40.61
ACU energy consumption (kWh)
4.16
16.27
10.95
8.19
Energy efficiency measure: ACLF (the lower better)
10.3%
40.89%
27.16%
20.17% 17% 4
Percentage of thermal safety violation (the lower the better)
40.99%
0.00%
0.00%
0.00%
with various
Pollcy
Policy
Pollcy rollouts:
Deletion
100k Antmaze-m- n dataset with multiple deletion areas
Deletion 70%
Start point
Deletion retio: 100%
End point
ZZA Deletion area
X-coordiatt
X-ccorcinate
X-coordnate
X-coordnans
Illustration of the 100k Antmaze-m-d task with multiple deletion areas, where the red denotes the start point, the yellow star denotes the goal locations, and the red shaded areas denote the data deletion regions. Right: Visualization of the training dataset and policy rollout trajectories generated by trained policies from various algorithms under varying deletion ratios.
Figure 3: Left:
surpassing all the other methods, while most baseline methods suffer from significant performance drop when training samples are decreased.
Evaluation on real-world industrial control test environment. To further demonstrate the ef- fectiveness of TELS in solving real-world industrial control tasks, we deploy TELS real-world DC cooling control testbed (Zhan et al., 2025) and compare against CQL, IQL, and TSRL. The testbed comprises 22 servers with oscillating server loads and an Air-Cooling Unit (ACU) for cooling control. A small historical operational dataset (43k real-world samples collected over 61 days) with 105 state-action features is used for policy learning. The goal is to improve the energy efficiency of the DC's cooling systems (minimizing the Air-side Cooling Load Factor (ACLF), calculated as the ratio of energy consumption of ACU to servers), while satisfying thermal safety constraints (no overheating). We follow the same real-world experiment setup as in (Zhan al., 2025) and present the details in Appendix D.2. As shown in Table 2, under a similar server energy consumption level, TELS learns the best control policy, achieving 20.17% ACLF while maintaining zero thermal safety violations. CQL learns naive policy that achieves lower ACLF but with significant thermal safety violations. This shows TELS's effectiveness solving real-world complex industry control tasks.
OOD generalization capability. To further examine the OOD generalizability of TELS, we construct very challenging task based on the reduced-size 100k Antmaze-m-d dataset, as illustrated in Figure: 3. Specifically, we randomly remove samples within 5 critical regions along the critical paths from the start to the goal locations. This task requires extremely strong OOD generalization capability to solve, as the vital information for the optimal trajectory is extremely scarce completely OOD. We train IQL, POR, and TELS on the remaining data and plot their policy rollouts over 20 episodes for performance evaluation and behavior analyses (due to page limit, include results for IDQL, DOGE, TSRL in Appendix B.3). As shown in Figure 3, IQL can only achieve some success when the deletion ratio is 0%, and POR fails reach the goal in all cases. By contrast,
8

---

Published as a conference paper at ICLR 2026
Normalized Score
100
Normalized Score
xno
ab-
Normalizze
20
Hopger-me Hellcheetah-me
Hopger-me
Waleerddmme
Hopperme
Walker2d-me
Figure 4: Left: Performance of IQL and TD3+BC on 10k datasets with or without using the representation from TS-IDM. Right: The performance of TELS with different representation models on 10k datasets. The error bars represent the standard deviation calculated over 5 random seeds.
Figure Impact éray on policy optimization.
TELS consistently learns an optimal policy even with 70% and 100% deletion rates. It can effectively utilize the limited information provided in the sparse remaining data samples at the boundaries of the deletion areas for policy learning. These highlight the extraordinary OOD generalization capability of TELS in extremely challenging low-data regimes.
4.2 ANALYSIS AND ABLATION OF TELS
Ablations on the design of TS-IDM. To examine the impact of each sub-module in TS-IDM, evaluate various variants of TS-IDM, starting with a vanilla latent inverse dynamics module with en- coder and decoders, denoted
6/U+ hine", " gradually adding latent forward and reverse dynamics "hfwd, fyee" ODE property enforcement "Lade", " and eventually the T-symmetry consistency loss "Lrsym", ", resulting in the full TS-IDM. Results on 10k datasets shown in Table 3. We observe that the naïve autoencoder-basee Incorporating dynamics-related information via latent dynamics is helpful, but the performance gain remains mild. Enforeing ODE proper- ties on decoders greatly enhances the quality of learned representations. Lastly, enforcing T- symmetry consistency proves to be the strongest performance improvement factor, which greatly enhances the quality of the learned representa- tions for downstream policy learning.
inverse dynamics module fails to provide reasonable representations.
Table Ablation on components of TS-IDM. The standard deviations are noted by +.
Hopper-me
Halfchectah-me me Walker2d-me
0/1++ + hive
17.2 +: 7.0
29.7 3.6
24.5 10.1
+ Pt Funds Ryas
35.5 +: 7.3
31.3 1.1
33.6 9.2
Fode
61.4
31.2
+ 18.1
Taya
100.9 1: 6.8
40.7 + 1.2
87.4
Effectiveness of the learned representations.
As demonstrated in Figure 4(left), we further verify the effectiveness of the learned latent rep- TS-IDM's state encoder (s) the representation
resentation in TS-IDM. Specifically, learning module on top of two offline RL methods: IQL and TD3+BC. The results reveal significant performance improvements and variance reduction when IQL and TD3+BC trained within the latent state space induced by (s), suggesting that TS-IDM learns compact and generalizable representations that benefit policy learning. To further evaluate the quality of TS-ID''s representa- tions, in Figure 4(right), we replace TS-IDM in TELS with other representation learning methods, including autoencoder ("AE-rep"),, variational autoencoder ("VAE-rep") (Kingma & Welling, 2014), and contrastive learning method SimCLR ("Contras-rep") (Chen et al., 2020). The results show that the TS-IDM representation achieves substantially better performance as compared to AF, VAE, and contrastive representations.
Ablations on regularizer terms in policy optimization. We also conduct ablation experiments in Figure 5 to validate the effectiveness of the T-symmetry consistency regularizer term fT-ymm during the quide-policy optimization process of TELS. The results demonstrate that incorporating this term can effectively enhance performance while reducing variance, highlighting the importance of utilizing T-symmetry consistency regularization to promote OOD generalization and learning stability. We present more ablation experiment results in Section B.5.

---

Published as a conference paper at ICLR 2026
5
RELATED WORK
Offline RL faces unique challenges in mitigating the risk of OOD exploitation. Evaluating value functions in OOD regions often results in inaccurate estimates, which can lead to severe value overestimation and misguiding policy learning. To mitigate this, most offline RL methods leverage data-related regularizations stabilize the learning process. These include explicit behavior constraint techniques that penalize action divergence (Wu et al., 2019; Kumar et al., 2019; Fujimoto & Gu, 2021; Li et al., 2022; Liu et al., 2024), value regularization schemes discourage policies from selecting OOD actions via modifying Bellman update (Kumar et al., 2020; Xu et al., 2022b; Bai al., 2021; Lyu al., 2022; Niu et al., 2022; 2025) introducing uncertainty penalities (Wu et al., 2021; An et al., 2021; Bai et al., 2021), and in-sample learning methods (Brandfonbrener et al., 2021; Kostrikov et al., 2022; Xu et al., 2023; Mao et al., 2024b; Li al., 2023: Wang et al., 2024; Zheng et 2024), which stabilize training by only using in-sample data for value and policy learning. While these methods perform reasonably well on datasets with sufficient state-action coverage, they often struggle in small-sample settings where exploiting OOD generalization is vital for achieving good performance. Recently, leveraging expressive model architectures such as Transformers and diffusion models (Wang et al., 2022; Ajay et al., 2022; Janner et al., 2022; Hansen-Estruch et al., 2023; Li et al., 2024; Mao et al., 2024a; Zheng et al., 2024; Liu et al., 2025) have gained popularity in offline RL, due to their strong capability fit complex data distributions. However, these models are overly heavy and require extensive amounts of data to learn (Wang et al., 2025), making them hardly usable for the small-sample setting.
CONCLUSION
We propose highly sample-efficient offline RL algorithm that Iearns optimized policy within the latent space regulated by the fundamental T-symmetry property. Specifically, we develop T- symmetry enforced inverse dynamics model, TS-IDM, to construct well-behaved and generalizable latent space, effectively mitigating the challenges of OOD generalization. By learning T-symmetry regularized guide-policy within this latent space, we can obtain the reward-maximizing next state to serve as the goal state input in the learned TS-IDM for optimal action extraction. Through extensive experiments, we show that TELS achieves strong OOD generalization capability and SOTA small-sample performance. Moreover, we empirically show that TS-IDM can also function representation learning model to provide informative representations and enhance the performance of existing methods under the small-sample setting. One potential limitation of TELS is that strong ODE and T-symmetry property regularizations, although helpful for capturing fundamental patterns in data, sometimes could limit the model's expressive power (sce Appendix B.5). Furthermore, the current TELS framework is primarily optimized for deterministic dynamics. Future studies could explore improved designs to optimally balance fundamental pattern extraction with model expressivity and investigate the adaptation of TS-IDM to capture T-symmetry properties within stochastic environments.
ACKNOWLEDGMENTS
This work is supported by GDS Holdings Limited and funding from Wuxi Research Institute of Applied Technologies, Tsinghua University under Grant 20242001 120, and Xiongan Al Institute.
REFERENCES
Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Marc G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference on Learning Representations, 2021a.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29377 2021b.
10

---

Published as a conference paper at ICLR 2026
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is con- ditional generative modeling all you need for decision maiiin? arXiv preprint arXiv:2211.15657 2022.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- ram Wang. Pessimistic bootstrapping for uncertainty-driver offline reinforcement learning. In International Conference on Learning Representations, 2021.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in Neural Information Processing Systems, 34:4933-4946, 2021.
Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932-333 3937, 2016.
Kathleen Champion, Bethany Lusch, Nathan Kutz, and Steven L Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 1 116(45); 22445-22451, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020.
Peng Cheng, Xianyuan Zhan, Wenjia Zhang, Youfang Lin, Han Wang, Li Jiang, al. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl. Advances in Neural Information Processing Systems, 36, 2023.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Philippe Hansen-Estruch Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit g-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 1057:3330 2023.
In Huh, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Time-reversal symmetric ode network. Advances in Neural Information Processing Systems, 13:19016-19027, 2020.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pp. 12519-12530, 2019.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine, Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09) 2009999999099 2022.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Newral Information Processing Systems (NeurIPS), 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR 2014: International Conference on Learning Representations (ICLR) 2014, 2014.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2026
Iyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pp. 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.
Jeroen SW Lamb and John AG Roberts. reversav symmetry in dynamical systems: a survey.
Physica D: Nonlinear Phenomena, 11((1-2):1-39, 1998.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference machine learning, pp. 5639-5650. PMLR, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learmin Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0166 2020.
Guanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long, and Weinan Zhang. Diffstitch: Boosting offline reinforcement learning with diffusion-based trajectory stitching. In Forty-first International Conference on Machine Learning, pp. 28597 28609, PMLR, 2024.
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning, The Eleventh International Conference on Learning Representations, 2022.
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, Qing-Shin Shan Jia, and Ya-Qin Zhang. Mind the gap: Offline policy optimization for imperfect rewards. In The Eleventh International Conference on Learning Representations, 2023.
Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive advantage soantagge-guiedd policy regularization for offiine reinforcement learning. Forty-firs International Conference on Machine Learning, volume 235, pp. 31406 31424. PMLR, 2024.
Tenglong Liu, Jianxiong Li, Yinan Zheng, Haoyi Niu, Yixing Lan, Xin Xu, and Xianyuan Zhan. Skill expansion and composition in parameter space. In The Thirteenth International Conference on Learning Representations, 2025.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 35:1711-1724, 2022.
Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. Odice: Revealing the mystery of distribution correction estimation via orthogonal-gradien update. In The Twelfth International Conference on Learning Representations, 2024b.
Gerhard Neumann and Jan Peters. Fitted q-iteration by advantage weighted regression. Advances in neural information processing systems, 21, 2008
Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When to trust your simulator: Dynamics- aware hybrid aodfine--nddollinee reinforcement learning. Advances in Neural Information Processing Systems, 35:36599-366 366122, 2022.
-aware
Haoyi Niu, Tianying Ji, Bingqi Liu, Haocheng Zhao, Xiangyu Zhu, Jianying Zheng, Pengfei Huang. Guyue Zhou, Jianming Hu, and Xianyuan Zhan. H2o+: an improved framework for hybrid offline-and ande---nd--niee rl with dynamics gaps. In 2025 IEEE International Conference on Robotics and
Automation (ICRA), pp. 1421-1428. IEEE, 2025.
12

---

Published as a conference paper at ICLR 2026
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal-conditioned with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighhted regression Simple and scalable off- off-po reinforcement learning. arXiv preprint arXiv:1910.00117,, 2019.
Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems, 35:16082-16097, 2022.
Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4r1: Surprisingly simple self-supervision for offline reinforcement learning in robotics. In Conference on Robot Learning, pp. 907-917. PMLR, 2022.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi-Veles Velez, and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healtheare. Advances in neural information processing systems, 35:34272 -42266, 2022.
Masatoshi Uchara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline rl in low-rank mdps. arXiv preprint arXiv:21 10.04552, 2021.
Guan Wang, Haoyi Niu, Jianxiong Li, Li Jiang, Jianming Hu, and Xianyuan Zhan. Are expressive models truly necessary for offline ri? In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 11062-21070, 2025.
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 34:29420- 29400, 2021.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforcement learning with implicit global-to-local value regularization. Advances in Neural Information Processing Systems, 36, 2024.
Zhendong Wang. Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06 191,, 2022.
Matthias Weissenbacher, Samarth Sinha, Animesh Garg, and Kawahara Yoshinobu. Koopman q- learning: Offline reinforcement learning via symmetries of dynamics. In International Conference on Machine Learning, pp. 23645-23667, PMLR, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint Xrxiv: 1911.1131, 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdi- nov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In International Conference on Machine Learning, pp. 11319 1328. PMLR, 2021.
Haoran Xu, Jiang Li, Jianxiong Li, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022a.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022b.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang. Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2023.
Mengjiao Yang and Ofir Nachum. Representation matters: offline pretraining for sequential decision making. In International Conference on Machine Learning, pp. 11784-11799 PMLR, 2021.
Tao Yu, Cuiling Lan, Wenjun Zeng. Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34:5276-5289, 2021a.
13

---

Published as a conference paper at ICLR 2026
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Neural Information Processing Systems (NeurIPS), 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offfine model-based policy optimization. Advances in neural information processing systems, 34:28954 289667, 2021b.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022a.
Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning. In Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI- 22. pp. 3716-3722, 2022b.
Xianyuan Zhan, Xiangyu Zhu, Peng Cheng, Xiao Hu, Ziteng He. Hanfei Geng, Jichao Leng, Huiwen Zheng, Chenhui Liu, Tianshun Hong, Yan Liang, Yunxin Liu, and Feng Zhao. Data center cooling system optimization using offline reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025.
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang. Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility diffusion model. In The Twelfth International Conference on Learning Representations, 2024.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A ADDITIONAL DISCUSSION ON RELATED WORKS
In this section, we present detailed discussion of the connections and differences between our proposed method TELS with TSRL (Cheng et al., 2023), POR (Xu et al., 2022a), and conventional model-based offline RL approaches.
(.44(s) 8 (.)
V/
Aymd
Rpad
S"-
Rrss
V.'d"((") (-8))
State act
Latent ODE forward riverse dynamics
State and action decaders
State encoder
Latent Inverse
Latent ODE forward and roverse dynamics
State and action
enzoder
(x) T-symmetry enforced dynamic model (TDM) in TSRL
(b) Our proposed T-symmetry enforved inverse dynamic model (TS-IDM)
Figure 6: Comparison of the architecture hetween TDM in TSRL and our proposed TS-IDM in TELS.
Connection and differences with TSRL. illustrated Figure 6, both TSRL and TELS leverage the T-symmetry consistency enforcement construct the latent space. Specifically, in Figure 6(a), TSRL employs a T-symmetry-enfine -enforree dynamics model (TDM), which models system dynamics by incorporating paired latent ODE forward and reverse dynamics enforce T-symmetry. In contrast, Figure 6(b) illustrates our proposed T-symmetry-en enforred inverse dynamics model (TS-IDM), which integrates T-symmetry constraints into both forward and reverse dynamics while incorporating an inverse dynamics model. We emphasize the main differences between TELS and TSRL as follows:
Architecture: As presented in Figure 6(a), TDM jointly encodes state-action pairs to form the latent space, which may capture behavioral biases from the dataset (e.g., expert-specifie action patterns) and impede learning fundamental, distribution-agnosie dynamics patterns in data. In contrast, Figure 6(b) illustrates that TS-IDM overcomes these limitations by adopting state-only modeling approach, focusing the underlying latent state variations. Additionally, the only useful component the learned TDM for downstream policy learning is its encoder Φ(s, a). wasting the dynamics-related information captured by the model. In contrast, TS-IDM trains an inverse dynamics model within the syymmetry-enforcee latent space, which can be reused as an execute-policy to extract optimal actions.
Detailed model design: As shown in Figure 6(a). TDM only enforces the ODE property for its encoder but not the decoder, which could lead to inconsistency between the learned dynamics and the underlying ODE structure, resulting in inaccurate misaligned ODE representations. To address this problem, we introduce the loss term Eade in Eq. (5) specifically to achieve this goal. This design is very important it can greatly enhance the coupling among the different elements in the model and result in more stable learning process.
Training procedure: TSRL, the TDM encoder and decoders must be pre-trained before joint training on other components to avoid stability issues. In contrast, our proposed TS-IDM does not require pre- pre-training: all components can be learned jointly in a single stage. Additionally, TDM requires adding L1-norm regularization the parameters of the latent forward and reverse dynamics models to stabilize the learning process. This is unnecessary in TS-IDM (see Eq. (7)). as the design of our proposed TS-IDM enables strongly coupled and consistent relationships among all its internal components. The learning curves of TS-IDM can be found in Appendix F.
Policy optimization: Since TDM requires both state and action as inputs to derive the latent representations, it is constrained to Q-function maximization for policy optimization. Consequently, TSRL adopts the TD3+BC framework as its backbone for policy optimization, which inherently suffers from over-conservative action-level constraints, particularly in small dataset settings. In
15

---

Published as a conference paper at ICLR 2026
contrast, TELS performs policy optimization entirely within the compact and generalizable latent state space derived from TS-IDM, enabling state-level optimization that avoids the limitations of action-space constraints.
Connection and differences with POR. share similarities in utilizing state-stitching approach in state space for policy optimization, they exhibit the following fundamental differences:
As discussed in Section 2, while both POR and TELS
Original space-space vs. latent state-space optimization: POR relies policy optimization in the original state space, which inherently requires sufficient state-action coverage for valid state- stitching. In contrast, TELS mitigates this limitation by constructing compact and generalizable latent space via TS-IDM.
Unregularized T-symmetry vs. T-symmetry regularized policy optimization: POR optimizes the guide-policy solely through an AWR formulation (Neumann & Peters, 2008; Peng et al., 2019), constraining Ty to stay close to the dataset via state-stitching additional regularization to ensure generalizable state transitions. In contrast, TELS enforces an additional T-symmetry consistency regularization f--smm which plays critical role in preventing Ty from outputting problematic and non- non-generalizable latent next states, thereby enhancing its OOD generalizability.
in Eq. (2), but lacks
Naïvely combining TSRL and POR does not work. Simply combining TSRL and POR actually performs notably worse than each method alone, shown in Table 4. This performance degradation stems from a fundamental incompatibility between the TDM in TSRL and POR's 's state-stitching mechanism. In contrast, our proposed TELS successfully exploits both T-symmetry and state-stitching. leading to substantial improvements over all baselines.
Differences from model-based approaches. We emphasize that our proposed TELS framework fundamenially differs from MBRL methods (Janner et al., 2019; Kidambi et al., 2020; Yu et al., 2021 lb; Wang et al., 2021; Yu al., 2021a; Zhan al., 2022a;b; Rigter et al., 2022). Conventional MBRL methods prioritize learning forward dynamics models to predict future states and generate rollouts for policy learning. contrast, our proposed TS-IDM is primarily designed for state representation learning and action extraction via inverse dynamics, rather than for data generation. Furthermore, as evidenced by Table 1, in the small-sample setting, limited data samples are insufficient for the model- based approach to learn accurate dynamics model, causing high approximation errors during model rollouts, which significantly deteriorate policy learning performance.
Table 4: Performance comparison between TELS, TSRL, POR, and TSRL+POR on reduced- size D4RL datasels. The highest score in the row is bolded.
Task
TELS
TSRL
POR
TSRL+POR
Hopper-m
77.3 10.7
62.0 + 3.7
46.4+ 1.7
38.5
Hopper-me
4: 8.2
17.44
25.9
Hoppar-me
100.9 = 6,8
50.9 + 8.6
37.94 6.1
30.3:99.7
Halfeheetah-m
40.8 :: 0.6
38.4 4: 3.1
33.34 3.2
35.24 7.5
Halfehectah-mr
33.2 1.0
28.1 + 3.5
27.54 3.6
28.3+t
Halfchectah-me
40.7 -1.2
39.9 21.1
34.74 2.6
38.9 1.6
Walker2d-m
62.4 +: 5.3
49.7 10.6
22.24 3.6
25.74 16.9
Walker2d-mr
54.8 + 6.0
26.0 +: 11.3
14.8::4.2
12.9 3,2
Walker2d-me
87.4 13.3
46.4 :: 17.4
20.1 8.6
23.8: 9.8
Autmaze-u
88.7 + 7.7
76.1 15.6
42.1
10.4++18.1
Antmaze-u-d
60.9 :: 16.9
52.2 22.1
6.14
6.7::3.11
Antmaze-m-d
47.2 17.3
0.0
0.0
0.0
Antmaze-m-p
62.9
0.0
Autmaze-I-d
39.8 14.1
0.0
Antmaze-I-p
47.3 : 13.1
0.0
0.0
0.0
16

---

Published as a conference paper at ICLR 2026
B ADDITIONAL RESULTS
B.1 EVALUATION ON THE FULL DATASETS
We also evaluate the performance of TELS on the original full datasets of D4RL tasks, and the results are presented in Table 5. Our proposed method achieves comparable or better performance than existing offline RL. methods. Note that although TSRL also adopts a similar T-yymmetry regularized representation learning scheme as ours, it performs poorly in Antmaze medium and large datasets. Primarily due to its use of the conservative TD3+BC backbone for policy optimization.
Table 5: Normalized scores on full-size D4RL datasets (averaged over the final 10 evaluations with 5 seeds). The highest score in the row is bolded.
Tisk
BC
TD3+BC
MOPO
COMBO
COL
IQL
DOGE
IDQL
POR
TSRL
TELS (ours)
Hopper-m
52.9
28.0
97.2
58.5
66.3
2.1
78.6
9674877
Hopper-me
18.1
60.9
67.
89.5
95.0
96.7
76.21177
87.4
98.92211
28722281
Hogperaaze
52.5
98.0
23.7
111.1
105.4
91.5
102.7 5.2
105.3
90.0 12.1
95.9
105.4 8.5
Hollchertabm
45.3d
48.2
44.3:2004
Halichorsa--rr
36.6
44.6
53.1
55.1
45.5
46.2
43.8 6:06
45.1
43.52009
42.2 # 3.5
41.1 0.1
Huitchertah
35.2
63.3
90.0
78.748.
94.7122
92.0
Wol
75.
81.7
178
81.9
72.5
78.3
86.8 0.8
80.2
81:1 2.3
77.5 2:45
81.32
Walker2d mr
26.0
81.8
29.0
56.0
77.2
73.9
N7.3 à: 2,3
79.8
76.6 3: 6.9
66.14:120
96,0 3.3
107.5
44.6
108.8
100.4:1.5
109.1 0.7
10988331111
Antraaze-a
65.0
0.0
90.6
19.2
94.5 # 10.3
Animate-u-d
45.6
71.4
0.0
57.3
43.4
66.7
63.5 9.9.
62.0
71.3 11.3=:10 12.1
76.3 29.7
79.7 1: 15,3
Anmmws-aud
0.0
54.011.7
74.6633.2
77.6166.1
79.24:311
R2.4
Antmane-m-p
0.0
0.0
0.0
65.22:488
90,41553
80.61.5
83.5
84.6 14..6
0.0
86.7
Ammare-lsd
0.0
0.0
0.0
0.0
316629.5
45.61376
16.4 19.11
56.4
75.4 28.5
0.0
15.7 1: 11.2
Ammusl-p
0.0
0.0
18.5:15.3
43.5:84.5
48.2
57.0
58.0
08.7 13.3
B.2 ADDITIONAL RESULTS ON ADROIT TASKS
We conduct additional experiments on Adroit-llonddeexper tasks. Since these tasks have much larger datasets (500k) as compared to Adroit-human tasks (5k samples), substantially reducing the learning difficulty, we therefore test our methods against baselines on a more challenging reduced-size setting with 10k samples. The results in Table 6 demonstrate TELS still achieves strong performance.
Table 6: Performance comparison of TELS against baseline algorithms Adroit tasks with limited data (10k).
Numbers at or above 95% of the best the row are highlighted in bold.
Tisk
Size (salie)
BC
TD3:BC
MOPO
COMBO
COL.
JQL.
DOGE
IDQL
POR
TSRL
TELS
Penedlood
(29)
07
15:48
355:305
41.6
09.74
Per-espert
27.51.21
52127
12103
25..05
$61.43
143
51. 195
1056.1..
6.2.1.200
05.6
105.7
(26
0.
08801
01801
03201
02201
0.200.2
033
0.:00
0.100.
06:00
0.6:2.0
Hammer expert
(29)
0.2
05:02
61
08:01
18:11
303
0.3003
27
17.6 01.3
Dax closed
(29)
0.
08.01
6.200
0.1.08
02
1.:08
0.
0.1
0.1:0:
0.1
7.0.2..8
Dxpen
(29)
1.8+11
58+31
15511
35:-1
7984888
054.0
0.33
0.700
46.3 12.5
101.8:8855
Reiosate exper
(26)
0.1....1
0.1 8.1
6.
1.
02201
a1.1288
0.3
87.5 12.7
0.2
46.2 15.3
85.0
B.3 ADDITIONAL OOD GENERALIZABILITY VALIDATION EXPERIMENTS
We further investigate the generalization capabilities of DOGE (Li et al., 2022), IDQI. (Hansen- Estruch et al., 2023), and TSRL (Cheng et al., 2023) under the variation deletion degrees in the Antmaze environment. Specifically, we train each algorithm on the modified dataset after the deletion operation. We then evaluate their behaviors by visualizing rollouts over 20 evaluation episodes.
As illustrated in Figure 7, only IDQL occasionally succeeds in reaching the goal under the 0% deletion setting, while both DOGE and TSRL fail consistently. As the deletion ratio increases to 70% and 100%, none of the three methods achieves meaningful policy learning. These results highlight the inherent challenges of this setting, which requires both compact yet expressive latent representation
17

---

Published as a conference paper at ICLR 2026
Data with various deletion rebios
Policy rollouts: IDOL
Policy rollouts: DOGE
Policy rollouts: TBRL
Delation
0%
100k anttaaze-m-d dataset with multiple deletion areas
Deletion ratio: 70%
Viccoran
Start point End point
Deletion
ratio: ratio:100% 100%
ZZZ
Deletion area
X-ccordintt
Figure 7: Left: Illustration of the 100k Antmaze- m-d task with multiple deletion areas, where the red cross denotes the start point, the yellow star denotes the goal locations, and the red shaded areas denote the data deletion regions. Right: Visualization of the training dataset and policy rollout trajectories generated by trained policies from various algorithms under varying deletion ratios.
space and highly generalizable policy capable of operating with extremely sparse and limited data. While TSRL integrates TDM to distill underlying patterns from the dataset, the scarcity of available data undermines its action-level constraints approach, preventing it from deriving viable policy.
B.4 ADDITIONAL. RLIABLE P1,OSS
To further statistically justify the performance of TELS, we use Rliable (Agarwal et al., 202 to plot the aggregate results across all locomotion tasks with 10k avaliable dataset. As shown in Figure 8, the results demonstrate that TELS consistently yields the highest score with the minimal optimality gap compared to all baselines.
Median
IQM
Mean
Optimality Gap
MORD
COMBO
COL.
DOGE
I0C
TELS
Normalized Score
Figure 8: Rliable plots for locomotion tasks with 10k dataset 5 random seeds.
B.5 ADDITIONAL ABLATION EXPERIMENTS
Ablations of in TS-IDM. We find that good weighting parameter / value typically corresponds to low TS-IDM training loss Eq. (7). As shown in Figure 9, = 0.1 yields the lowest loss for HalfCheetah, while 3 1 is better for Hopper and Walker2d. Table further confirms that the same setting with = gives the highest score on Hopper and Walker2d, whereas 3 0.1 performs best HalfCheetah. This shows that the better the IDM is trained, the higher quality latent representation can be learned to facilitate downstream policy optimization. This is important, as indicates that do require any environmental interaction policy evaluation for 3 selection.
18

---

Published as a conference paper at ICLR 2026
hopper--edium--rpplay.22
halfchetah-mmeiuu--reaaaay
Waakerz2--mediu--reeaaaa--
0.4
1.5
0.15
0.3
1.0
0.10
10.2
Crs
0.5
0.05
0.1
0.00
0.0
0.0
250
500
750
1000
250
500
750
1000
250
500
750
1000
Training epoch
Training epoch
Training epoch
0.1
B B=1
B 10
The learning curves for training TS-IDM on 10k dataset with different 3 hyperparameter. The standard deviations are noted by 1. Numbers at or above 95% of the best in row are highlighted bold.
Figure 9:
Table 7: TELS performance with different under sample setting. Numbers at or above 95% the best in the column are highlighted in bold.
Table 8: TELS performance under various weight- ing terms on 10k Hopper-me me dataset.
Weight of Edgn
Weight of Ende
Weight of Eramm
Evaluation scores
8 10
8 1 (Used)
B = 0.1
Hopper-m
77.3 5.4
77.3 10.7
61.4
1
1
100.9 1 6.8 (Used)
Hopper-mr
15.3
43.2 = 3.5
19.7 #3.4
1
1
0.1
56.4 +:066
Hopper-me
37.6 +: 17.9
100.9 +: 6.8
64.7 d: 3.3
0.1
1
51.1 d: 2.2
Halfehestab-m
32.9 2,3 2,3
40.8 = 0.6
41.2 1.1
0.1
1
d: 3.1
Halfehestah-mr
8.6 d: 1.8
33.2 = 1.0
34.9
0.1
0.1
56.3 :: 3.2
Halfeheetah-me
7.5 + 2.2
40.7
41.5
0.1
1
0.1
$4.8 1: 11.4
Walker2d-m
37.2 +: 7.9
62.4 5,3
$4.6 d.6:::. 8.2
0.1
0.1
56.1 +: 5.7
Walker2d-mr
17.19229
54.8 = 6.0
39.2 : 8.6
Walker2d-me
20.4 + 10.4
87.4 13.3
44.7 9.8
Simply looking at the supervised training loss of TS-IDM on offline datasets will already provide good sense of the proper scale.
Empirically, we observe that smaller datasets benefit from relatively larger 8, whereas in large datasets, a smaller 3 is typically required to reduce training loss. This is as expected, as f controls the strength of T-symmetry and ODE regularization. Large datasets contain sufficient information from data samples, thus requiring less regularization, while small datasets benefit from stronger regularization to enable the extraction of additional information from limited samples. To keep simple, in our main results, we use B = 1 for all MuJoCo locomotion tasks in the small-sample setting without hyperparameter tuning. For tasks with large datasets and other domains, select from the set (0.01, 0.1, }} as the one with the lowest TS-IDM training loss.
Furthermore, as have discussed in the main paper, we need layn. fade, and sym To provide some evidence, we also conducted an extra ablation experiment by re-weighting each term in Eq. (7) differently as in Table 8. The results confirm that inconsistent I weighting schemes lead to significant degradation in policy performance and unstable learning, ultimately resulting poor outcomes. By contrast, simply using the same 3 value achieves doubled evaluation scores. The reason behind this is what we have explained in the main paper, the internal components of TS-IDM are strongly coupled and have to regulated the same strength (i.e., using the same B). Specifically, both the latent ODE forward and reverse dynamics modules (h htro)) use the same latent actions z, from /i as input. The T-symmetry consistency &T is also enforced on both h fwdtrr These make the latent forward, reverse, and inverse dynamics modules strongly coupled. Moreover, the state encoder d, and decoder V also need to satisfy the ODE property as required in the h fwd and hres, as enforeed through the loss terms laya and fode respectively. Hence, different levels of regularizations are applied to these loss terms, internal inconsistency will emerge and impair the learning of TS-IDM.
single shared / for loss terms
19

---

Published as a conference paper at ICLR 2026
Impact of regularizer terms 7) in policy optimization. The hyperparameter 7 governs the strength of regularization in TELS, balancing exploration and adherence to dataset states during policy updates. To evaluate the robustness of TELS, we test multiple 17 values (1) {1, 5, 10)) to examine its sensitivity to the state-level behavioral constraint in Eq. (9). Higher 1 values impose stronger constraints the guide-policy, requiring generated states s' to align closely with dataset states. As shown in Figure 10, TELS demonstrates consistent robustness across 1) settings, achieving reliable performance under varying constraint strengths.
Impact of each component in TS-IDM for policy optimiza- tion. To further validate the impact of the T-symmetry regular- izer &-syym in Eq. (10), we conduct additional ablation studies on 100k-sample Antmaze tasks. From the evaluation results presented in Table 9, the naïve auto-encoder based inverse dy- namics module "6/1) + hise" fails form reasonable latent space, yielding 0 average normalized scores across all Antmaze tasks. The introduction of latent dynamics models "/ fwd and "hrus" " provides marginal improvements by capturing partial system dynamics, yet remains insufficient for effective policy learning. Notably, enforcing ODE properties on decoders and applying T-symmetry consistency emerge as the most significant factors driving performance improve- ments, substantially enhancing the reliability of learned representations for downstream guide-policy optimization.
Normalized Score
100-
Q-5
n=10
60-
20
Halfcheetah-me Walker2d-me
Figure 10: TELS with various T).
Table 9: Ablations the components of TS-IDM with 100k Antmaze datasets. Numbers at above 95% of the best in the row are highlighted bold.
Antmaze-m-d
Antmaze-m-P
Antmaze-I-d
Antmaze-l-p
/ 1 hinv
0
0
0
0 7.8 + 3.4
hfwd, hrus
23.6 + 18.4
30.4 + 9.3
14.4 + 5.6
+ Eade
34.1 + 15.7
48.7 = 13.3
20.1 = 8.9
22.6 - 16.7
+ Tsym
47.2 1 17.3
62.9 + 17.8
39.8 1 14.1
47.3 1 13.1
Impact of T-symmetry regularizer term in guide-policy optimization with stochastic policy instantiation. We further conduct ablation experiments in Figure [1 to validate the effective- ness of the T-symmetry consistency regularization term &r-ym during the stochastic pooiey optimization process of TELS. The results demonstrate that in stochastic policy optimization schemes, integrating this term significantly improves performance while reducing variance, underscoring the critical role of T-symmetry consistency regularization in enhancing OOD generalization and training stability.
Effectiveness of learned representations for guide-policy optimization with stochastic policy instantiation. As illustrated Figure (rightt,, we evaluate TELS across diverse representation learning approaches in Antmaze tasks. The results demonstrate that baseline models struggle to construct meaningful latent spaces as task complexity increases and data scarcity intensifies (with only 100k usable samples). In contrast, TS-IDM uniquely learns a compact, well-structured latent space that remains informative and generalizable, providing a more reliable latent space for policy learning.
B.6 VISUAL ANALYSIS ON THE LEARNED LATENT SPACE
Effectiveness of learned latent state space. illustrate the compactness and effectiveness of the learned latent state space through TS-IDM, in Figure 12, we plot the t-SNE visualization of the original data trajectories of the Hopper-m 10k task, as well as the rollout trajectories of learned TELS and IQL policies on both the original state space and the latent state space (encoded using our TS-IDM state encoder). We can clearly observe that the learned latent space is much more compact and well- bell-behaved. The policy rollout trajectories form clear, continuous line patterns in our learned latent space, but can be quite noisy in the original state space. Such a more compact and structured latent space greatly facilitates robust policy learning via latent stitching and OOD generalization.
20

---

Published as a conference paper at ICLR 2026
Normallzed aaaallzzdd Score
Norma lized Score
60-
lizd
40-
00-
20-
Antmaze-m-d
Antrnaze-m-p
Antmaaz---d
Antmaze-l-p
Antrmaze-m-d
Antmaaze-m--
Antmeze-i-d
Antmaze-tpp
W/o Ir-rpe
with fr-
AE-rep
VAE-rep
Contras-rep
TS-IDM-ep
Figure 11: Left: Impact of &--ym on policy optimization with 100k Antmaze datasets. Right: Performance of TELS with different representation models on Antmaze 100k datasets.
Original State Space
Latent State Space
Dr ginial data points TELS roltura data
Ohiginal cata punts TELS rollbuts cata
120
Reward Value
50
-50
-100
Original data points
Original cata peints sullouts data
date
120
Reward Value
-100
-150
-100
-50
50
1.00
150
-150
-1.0
50
Figure The t-SNE visualization of state representations on the Hopper-medium 10k The plots compare the original state space (left) with the latent space encoded by the pre- pre-trained TS-IDM IDM encoder d. (right). The data trajectory samples are colored by reward value. Overlaid points represent rollout trajectories (two episodes of 1,000 steps each) generated by the TELS policy (green) and the IQL. policy (bluc).
From the t-SNE visualization, we can observe that the IQL's rollout trajectories deviate substantially from the data distribution, generating numerous OOD states that violate the offline dataset distribution boundaries. Even when projected through the pre-trained TS-IDM encoder, these irrational states remain outside the meaningful latent manifold, elucidating the primary cause of IQL's performance degradation data-scarce scenarios. In contrast, TELS demonstrates superior state-space utilization by maintaining good alignment with the dataset distribution while effectively navigating toward high-reward regions.
C IMPLEMENTATION DETAILS
C.I ALGORITHM PSEUDOCODE
The pseudocode of TELS is listed in Algorithm 1.
21

---

Published as a conference paper at ICLR 2026
Algorithm 1 Offline RL via T-symmetry Enforced Latent State-Stitching (TELS),
Require: Offline dataset D.
1: //TS-IDM learning
2: Learning the state encoder d.. state decoder dh,, action decoder Va. latent inverse dynamics hen.. latent
forward and reverse dynamies y and byve using the TS-IDM learning objective Eq. (7).
3: Initialize Ve. Vgr. Re
4: // /Pooicyy training
5: = .M training steps do
6: Sample transitions (,,,,"" 6') 'D and compute their representations (2g, a./) using the state encoder Φs-
7: Use (z., v; Zgr) to update the latent state-value function using (8).
8: Use (2,, ) to update the latent guide-policy Ty using Eq. (9) or (10).
9: end for
10:
//Evalaation
Get initial state from environment
12: while not done do
13: Get optimized next state = using guide-policy Tg.
14:
Extract action using Eq. (11).
15:
end while
C.2 IMPLEMENTATION DETAILS OF TS-IDM
Network structure. For all MuJoCo locomotion and Antmaze tasks, we deployed 3-layer feed- forward neural networks for the state encoder 6s, latent inverse dynamics module hinos forward and reverse dynamics models f and hrus. and decoder models 1. and vh for the latent states and actions. The activation function is ReLU and uses the Adam optimizer to update the parameters. We present the hyperparameter details of training TS-IDM in Table 10, including the details of the structure we have implemented as well as the deployed hyperparameters.
ODE property enforcement on o, and v/.. We adopt similar approach to TSRL (Cheng et al., 2023) to train the ODE enforced forward and reverse dynamic models. Specifically, we compute the time-derivative of the state encoder (s) by calculating its Jacobian matrix through vmap () function in Functorch 1, This allows us to derive the supervision values dé.(s). á and do. (-8)
de
ds' in Eq. (4). This
for the forward dynamics module and reverse dynamics module respectively approach implicitly enforces the ODE property on the state encoder Φ. as the encoder required to produce state representations that satisfy the ODE constraints. Unlike TSRL, which enforees ODE properties only on the encoders and not on the decoders, our method further regularizes the state decoder v/. Specifically, vi, is trained to decode the predicted latent state variables generated by hfwa(Zs, 2g, Ra) = åN and hyve (2e', = - 2, ensuring that it also satisfies the ODE constraints in Eq. (5). To achieve this, apply the same approach compute de. (+,) and train the state decoder accordingly.
C.3 IMPLEMENTATION DETAILS OF T-SYMMETRY REGULARIZED GUIDE-POLICY
Network structure. For all D4RL MuJoCo-v2 and Antmaze-vl tasks, we deployed 2-layer feed- forward neural networks for the guide-policy Ty and the value function V. The activation function is ReLU and uses the Adam optimizer to update the parameters. The parameter details are presented in Table 11.
Hyperparametes for policy optimization. Under both small-sample and full datasets settings, we employ deterministic policy update strategy for MuJoCo locomotion tasks, defined in Eq. (9), with learning rates of le-4 for both value and policy functions. The normalization term À is computed where a controls the trade-off between value maximization and policy regularization and N denotes the number of samples in the training batch. For Antmaze tasks, we employ a stochastic policy optimization strategy, as outlined in Eq. (10), with learning rates of 1e-3 for both the value and policy functions.
as >a = a/
22

---

Published as a conference paper at ICLR 2026
Table 10: Hyperparameters of TS TS-IDM.
Hyperparameters
Value
State encoder hidden units
512 x 256
State encoder activation function
ReLU
Latent forward module hidden units
256 x 256
Latent forward module activation function
ReLU
Latent reverse module hidden units
256 x 256
Latent reverse module activation function
ReLU
TS-IDM Architecture
latent inverse module hidden units
1024 1024
Latent inverse module activation function
ReLUU
Latent inverse module dropout
True
Latent inverse module dropout rate
0.1
State decoder hidden units
256 x 512 ReLU
State decoder activation function
Action decoder hidden units
512 512
Action decoder activation function
ReLU
Optimizer type
Adam
Weight of Euee Learning rate
3e-4
Batch size
256
Training Parameters
Training epoch
1000
State normalize
True
Selected from (0.01,0.1.1) as the one with the lowest TS-IDM training loss (see Figure 9)
Weight of 8
(MuJoCo locomotion setting)
Weight decay
le-5 (Other tasks)
Table 11: Structure and training parameters of guide-policy optimization.
Hyperparameters
Value
Value network hidden units
1024 1024
Guide-policy structure
Value network activation function
ReLU
Policy network hidden units
1024 x 1024
Policy network hidden units
ReLU
Optimizer type
Adam
Target Value network moving average
0.05
Batch size
256
Training steps
100,000
Training
State normalize
True
Perparameters
0.9
(Antmaze tasks)
Weight of r
0.7 (Other tasks)
10
(Antmaze tasks)
Weight of
0.01 (Other tasks)
C.4 MODEL COMPLEXITY AND TRAINING TIME.
As we presented the model structure details in Table 10, TS-IDM is actually relatively small model, consisting of only 2-layer MI.P sub-modules. Its parameter size (~2.8M parameters) significantly smaller compared to many recent Transformer-based (-12M parameters) and diffusion-based (-16M parameters) offline RL methods. further demonstrate the learning speed TELS, we present a comparative analysis of training times with other baseline methods the hopper-mediun--22 10k dataset, utilizing the official codebases. All the algorithms are trained on a workstation with an AMD Ryzen 9 7950X 16-Core Processor, NVIDIA GeForce RTX 4090 GPU, and 16GB of memory, running on Ubuntu 22.04.2 LTS 64-bit. As illustrated Table 12, the Jax implementation TELS completes training in merely 20 minutes, whereas the Torch version requires 120 minutes. This
23

---

Published as a conference paper at ICLR 2026
result not only matches but often surpasses the efficiency of most baseline methods, underscoring the exceptional training efficiency the method.
Table 12: Training time cost comparison on 10k Hopper-m datasets across various algorithms.
Dynamics model training (min)
Policy optimization (minn)
Total time (min)
Algorithm
Evaluation scores
TELS (IAX)
15
20
75.2 = 6,3
TELS (PyTorch)
20
100
120
77,3 1: 10.7
TSRL (pichub. con/ pcheng2/19RR)
30
130
160
62.0 3.7
POR (githab.com) wyanktx/PGR
450
450
46.4 d: 1.7
IDQL (githhh how/bcoo//hhhh (e011eegeeeee-0/1000
470
470
44.2 # 12,1
DOGE wtw.thubcc.//aaeeeeeeeeeeeeeeeee httpa/////o/eeeeseeeeaa////00000
410
410
44.2 + 10.2
IQL (github. com/ httpa//www.....ooeaee///e/aaaaaa
46.7 6.5
CQL (githubb http:///......iaaaaaaa//22////00
780
780
43.1 + 24.6
COMBO CcMM0///gi....aac//////////////d
1200
1200
30.2 28.0
MOPO (gy httpg/thh..c////////////////////
780
780
5.5
TD3+BC wtt.://abc.o///aaaaaaaa/aaoo/aa/
18.6
29.7 11.7
D DETAILED EXPERIMENT SETUPS
D.1 EXPERIMENT SETUP FOR SIMULATION BENCHMARK TASKS
Reduced-size dataset generation. To create reasonably reduced-size D4RL datasets for a fair comparison, we use the identical small samples as in the TSRL paper (Cheng et al., 2023) for the locomotion tasks training. For Antmaze tasks, we adopt a similar approach by randomly sub- sampling trajectories from the original dataset to construct smaller training datasets. Specifically, for the "Antmaze-umaze" tasks, we randomly sample 10k data points for training, and for the "Antmaze- medium" and "Antmaze-large" tasks, we utilize 100k random samples as the training dataset of TELS.
The rationale behind this adjustment is the "medium" and "large" environments are significantly more expansive than the "umaze" environment. Sampling only 10k data points would likely result in trajectories that lack the fundamental information necessary to describe the task. Therefore, relax the small-sample constraints for these environments to ensure that the reduced datasets at least contain enough successful trajectories for effective training.
Experiment setups for various representation learning. To validate the effectiveness of the representations learned by TS-IDM, integrate it as the representation module in two offline RL frameworks (IQL and TD3+BC), verifying the usability of the learned latent space as illustrated in Figure 4(left). Specifically, we process the original states and next states s' from the dataset using the pre-trained state encoder o, of TS-IDM to derive the latent representations: (,(s) + 2, and (s') Ze. Then, train IQL and TD3+BC within the latent space evaluate their performance under the small-sample setting.
Furthermore, in Figure 4(right), we benchmark TELS against three established representation learning baselines ("AE-repp", "Contras-rep" and "VAE-rep") to rigorously assess TS-IDM 's representation quality. Implementation details for baseline models are provided below:
We implement a naïve autoencoder-basee inverse dynamics framework, consisting of state encoder and decoders d, and th, to construct the latent state space. As in TELS, the inverse dynamics model hine is built within this latent space, serving the execute-policy. For fair comparison, we use the same network parameters for the encoder, decoder, and inverse dynamics module in TS-IDM The "AE-rep" model is trained with reconstruction loss capture the essential features of the input, and the inverse dynamics model simultancously trained on the latent representations predict actions.
"AE-rep":
24

---

Published as a conference paper at ICLR 2026
4'sensor
PDU
back
Rack B02 front
front
Rack A02
back
sesesor
FDU
sesnno
outlet
ACU
k inlet--
"sensor
4'senoor
sensoo
back Rack B01 front
front Rack A01
back
glass
sliding
glass
doors
doors
doors
Figure 13: The layout illustration of the real-world DC cooling control testbed environment.
The variational autoencoder (VAE) (Kingma & Welling, 2014) built based on the "AE-rep" model by introducing additional KL divergence loss terms. Specifically, the encoder outputs parameters of Gaussian distribution in the latent space, and the latent representations are sampled using the reparameterization trick. The VAE is trained using a combined loss function that includes both the reconstruction loss and the KL divergence loss, which regularizes the latent space to follow a prior distribution. The inverse dynamic module is trained simultancously with the VAE, sharing the latent space and optimizing for both the reconstruction the input data and the prediction of actions.
VAE-repp":
"Contras-rep": We utilize the NT-Xent loss (Normalized Temperature-Seall Scaledd Cross Entropy Loss) used in SimCLR (Chen et al., 2020) within the latent representation space on top of the "AE-rep" model. The overall loss function combines the contrastive loss with the reconstruction loss, ensuring that the latent space not only captures the structure of the data but also learns semantically meaningful representations that are robust variations. The inverse dynamic module is trained simultaneously within the latent space to predict actions.
D.2 EXPERIMENT DETAILS OF REAL-WORLD INDUSTRIAL CONTROL TEST ENVIRONMENT.
We adapted the figure from (Zhan et al., 2025) to illustrate the layout structure the real-world DC cooling control testbed. As shown in the Section D, the testbed comprises 22 server units and an inter-rack air conditioning unit (ACU) positioned between Rack and Rack 2, supplemented by 24 temperature and humidity sensors (organized into six monitoring sets) to capture spatial thermal dynamics within the environment. Notably, the ACU employs compressor-driven cooling, with fan operation and compressor workload constituting the primary sources of energy expenditure. The thermal regulation is achieved by modulating the ACU's entering air temperature (EAT) setpoint to maintain the cold aisle temperature (CAT) below predefined safety threshold. The energy-saving objective is to improve the energy efficiency of the DC's s cooling systems (minimizing the ACLF) while satisfying thermal safety constraints.
We leverage dataset of 43k real-world operational samples recorded at 2-minute intervals over 61 days with 105 state-action features. During the training process, we utilize the identical reward function and follow the same experimental protocols outlined in (Zhan et al., 2025). To ensure rigorous benchmarking, we adopt same challenging thermal constraint (set the CAT threshold as C2cC) for comparative evaluation of TELS performance. Following the testing protocol in (Zhan et al., 2025), we ran our RL policy on the testbed continuously for hours, which issues control commands every minutes. We collected and aggregated all the energy- measurements 2-minute intervals to calculate the final ACLF metric.
E BROADER IMPACT
While training reinforcement learning (RL) agents on large-scale offline datasets has been extensively studied, real-world applications often face prohibitive data scarcity and collection costs. This necessitates offline RL methods that achieve reliable performance in small-sample regimes. To address this challenge, introduce highly sample-efficient offline RL algorithm to learn high- performing policies from extremely limited data. We empirically validate its efficacy through deployment real-world data center cooling control testbed, establishing its practical viability.
25

---

Published as a conference paper at ICLR 2026
Our approach highlights promising pathway for advancing sample-efficiem offline RL in resource- constrained settings. A potential limitation is the inherent risk of unreliable or unsafe actions within historical datasets, which may mislead policy learning.
F LEARNING CURVES
The following the learning curves of TS-IDM and the T-symmetry regularized guide-policy optimization in TELS on the reduced-size D4RL MuloCo and Antmaze datasets. We evaluate the policy with 10 episodes over 5 random seeds.
26

---

Published as a conference paper at ICLR 2026
Hopper-m-v2 (10k)
Hopper-m-v2 (10K)
Hopper-m-v2 (10k)
0.30
0.20
g000
0.40
0.30
0015
0.344
0.10
0.2
600
0.00
0.30
0/50
3.30
02:6
6i6e
3.5
1.00
060
636
3.00
0.26
1.0
0.30
8.57
0.15
Training Steps (x10")
Training Steps (x107)
Training Steps (x10*)
Hopper-m-v2 (10k)
Hopper-m-v2 (10k)
Hopper-m-v2 (10K)
0.10
1.50
00020
0015
0.00
0010
0.03
Trahiin (x10")
Training Steps pX10P)
Training Steps (x10²)
Hopper-m-r-v2 (10k)
Hopper-m-r-v2 (10k)
Hopper-m-r--2 (10k)
0.53
0.00
0010
0.35
100
3.35
0.000
0:34
0.04
0.00
0.22
0.00
0.0
0.20
0.53
8.5g
335
03c
3.30
0.25
120
0.61
03%
Training Steps (x10")
Training Sleps (x10²)
Training Steps (x10")
Hopper-m-r-v2 (10k)
Hopper-m-r-v2 (10k)
Happer-m-e-v2 (10k)
1.025
0.15
1.000
0.23
0.204
0.24
0:1
0.00
0:3
0.000
0.2
0.2
0.50
025
3.50
0.75
1.00
0.00
0.30
6.75
Training Steps (110")
Training Steps (x10")
Training Staps (x10P)
Hopper-m-e-v2 (10k)
Hopper-m-e-v2 (10k)
Hopper-m-e-v2 (10k)
000
0.00
Zone00
3.54
oue
0.02
0.32
0.00
0.30
0.000
0.30
0.26
678
335
1.00
000
036
3.50
0.26
140
830
0.25
050
120
1.00
Training Steps (x10")
Training Steps (x10²)
Training Steps (x10%)
Hopper-m-e-v2 (10k)
Hopper-m-e-v2 (10k)
Hopper-m-e-v2 (10k)
2.000
0.004
0.225
0.000
0/11
1007
0.05
Training ))10))
Training Steps (x102)
Training Steps (x10²)
Figure 14: Learning curves of the overall and each individual loss terms in TS-IDM for Hopper tasks.
27

---

Published as a conference paper at ICLR 2026
Halfcheefahh-m-2 (10k)
Hallcheettah-m-22 (10k)
Halfcheetah-m-rZ (10K)
0.2
6.11
0.004
01004
0.2
0.30
0000
0.30
0226
1000
000
0.3
3.91
10:05
116
0.00
0.26
8:5
1.0
Training Steps (x10")
Training Steps (x107)
Training Steps (x10*)
Halfcheetah-m-w2 (10k)
Halfcheetah-m-v22 (10k)
Halfchholaah-m-22 (10k)
0.4
0.3
0.3
0.000
0.2
0.04
0.1
0:1
0.0
033
0.000
0.0
0.25
0.75
025
3.00
0.75
1.00
0.75
Training Steps (x10")
Training Staps (x10")
Training Stepe (x10P)
Hallccheetahmm-r-22 (10k)
Hallcheetah-m-rr22 (10k)
Hallcheetah-m-rr22 (10K)
0.000
0.35
0.15
0.34
0.000
0.004
0.32
0.00
0.30
0/5
0.30
0256
6is
37%
1.00
000
035
3.00
0.75
1.0
0.30
8.57
0:15
1.0
Training Steps (x10")
Training Steps (x107)
Training Steps (x10*)
Halfcheetah-mm-r-22 (10k)
Halicheetah-m---22 (10k)
Hallcheotah-mm-xx2 (10k)
0.45
0.0
0.5
0.0
3.35
0.000
0.30
0:15
0.004
6/6
035
8.0
0.0
0.00
0.30
0.25
0.00
275
1.00
0.00
025
0.50
0.75
1.00
0.50
0.20
0.00
0.75
1.00
Training Steps (x10%)
Training Steps (x10*)
Training Steps (x10²)
Hallcheefah-m-eee (10k)
Hallcheetah-m--e-22 (10k)
Hallcheetah-m-r-22 (10K)
6.50
0.000
0.34
0.004
0.8
0.30
0.00
0.30
0026
Dike
1,00
000
023
2.50
1.00
0.00
0.25
8.3
5.75
1.0
Training Stops (x10")
Training Staps (x107)
Training Steps (X202)
Hallcheetah-mn-e-2 (10k)
Hallcheetah-m--e-22 (10k)
Hallcheetah-m---422 (10K)
0.4
0.115
6.30
030
620
0.5
20205
0.15
0.13
662
0:3
0.50
0.75
Training Steps (x10")
Training Staps (c107)
Training Steps (x10*)
Figure 15: Learning curves the overall and each individual loss terms in TS-IDM Halfcheetah tasks.
28

---

Published as a conference paper at ICLR 2026
Walker2d-m-v2 (10k)
Walker2d-m-12 (10k)
Walker2d-m-22 (10k)
0.14
0.30
6:15
0.00
1.08
1.00
0.34
00004
610
0.0
0.30
0.80
0.33
0226
036
1.00
010
036
3.0
0.26
1.0
0.30
8:57
0:15
100
Training Steps (x10")
Training Steps (x107)
Training Steps (x10*)
Walker2d-m-v2 (10k)
Walker2d-m-v2 (10k)
Walker2d-m-v2 (10k)
0.10
0.004
1.00
0.00
2.00
0:0
0.00
0.2
6.2
0.58
0.75
0.00
025
0.50
0.75
1.00
0.50
0.25
0.30
0.75
1.00
Training Steps (x10F)
Training Steps (x10*)
Training Steps (x10*)
Walkke2dd---r-22 (10k)
Walker2d-m-rr-r2 (10k)
Walker2d-m-r-v2 (10k)
0.10
0.00
0.04
0:54
0/00
640
0.01
0:00
0.00
0.30
0220
1000
000
623
3.9
116
0.00
0.5
8.3
0.75
1.0
Training Steps (x10")
Training Steps (x107)
Training Steps (x207)
2alker2d-m-e-2 (10k)
Walker2d-m-r-w22 (10k)
Walker24-m-r-v2 (10k)
0.7
0.007
0.00
0333
0.00
0.00
0:1
0.01
0:0
632
035
0.50
6.75
0100
U10
0/76
1.0
Training Steps (x10")
Training Siepe (x10*)
Training Steps (x10*)
Walkke22-m-e--v2 (10k)
Walker2d-m-e--22 (10k)
Wilkker2d-m-e-v2 (10k)
0.000
0.12
0.15
0.000
2..00
0..04
0.00
Training Steps (x10")
Training Slepe (x10*)
Training Steps (x10*)
Walker2d-m-e-w22 (10k)
Walker2d-m-ee-12 (10k)
Walkker2d---e-v2 (10K)
0.70
0.206
2.70
0.00
0.00
0.32
0.80
Training (110")
Training Stops (X10")
Training Steps (x10=)
Figure 16: Learning curves of the overall and each individual loss TS-IDM for Walker2d tasks.
29

---

Published as a conference paper at ICLR 2026
Hopper-m-v2 (10K)
Hopper-m-r-22 (10K)
Hopper-m-e-v2 (10k)
Normalized Score
8
Normalized Score
800
800
Training Steps
Training (d10%)
Training (d10f)
Hallcheetah-m-v2 (10k)
Hallcheetahhmm-rw2 (10k)
Halfcheetah-m-eww (10k)
Normalized Score
Normalized Score
5
Normalized
825
11c
110
Training Steps (x10%)
Training (x10%)
Training Steps (x10")
Walker2d-m-v2 (10k)
Walker2d-m-rww2 (10k)
Walker2d-m-e-v2 (10k)
Norralizzd Score
Normalized Score
100
Normalized
0190
USe
523
3.50
100
8.0
0.25
0:50
1.00
Training Steps (x10f)
Training Steps (x10")
Training Steps (X10*)
Antmaze-u-v2 (10k)
Antmaze-u-d-12 (10k)
Antmaze-m-d-v2 (10k)
105
8
Normalized Score
Nermalized Scare
2 2 8 8 8 a 8.
22 8
Nomalzzd
mm
g 8 8 8 8
025
0.20
377
1.00
0.55
0.72
1.00
8.ce
332
3.5
0.55
1.00
Training Steps (x10")
Training Steps (x10F)
Training Slepe (x10*)
Anlmaze-m-p-v2 (10k)
Antmaaze-Id-v22 (10k)
Antmaaz---p-v2 (10k)
Normalized Score
Normalized Score
y
y
8
Nomalized
puly p Mr a phy
8.
8.
2
800
0.26
U80
5.7
1.00
823
3.30
1.00
323
3.5
0.79
1.00
Training Stops (x10%)
Training Steps (x10")
Training Staps (x10%)
Figure 17: Learning curves of policy optimization in TELS for D4RL MuloCo and Antmaze tasks with reduced- size datasets. We evaluate the policy within 10 episodes over 5 random seeds.
30


Reviews:
[{"review_id": "J9hfiBvoAZ", "reviewer": "Reviewer_Kxst", "summary": "The paper proposes TELS, an offline RL algorithm addressing poor sample efficiency and limited OOD generalization in existing methods. It leverages time-reversal symmetry (T-symmetry) and latent state-stitching via a T-symmetry Enforced Inverse Dynamics Model (TS-IDM), which learns generalizable latent representations by enforcing ODE properties. TELS optimizes a latent guide-policy to avoid action-level conservatism and extracts actions via TS-IDM. Experiments on reduced-size D4RL benchmarks and a real-world data center testbed confirm its SOTA small-sample performance.", "questions": "TELS uses reverse/backward dynamics for T-symmetry and representation learning, similar to PlayVirtual (Yu et al., NeurIPS 2021), which leverages backward dynamics for RL generalization.  Could the authors clarify the similarities and differences between TELS and PlayVirtual?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel and Principled Design: Integrating T-symmetry (a fundamental physical property) and latent state-stitching is clever. The ODE perspective—regularizing both encoders and decoders (unlike TSRL’s encoder-only focus)—ensures distribution-agnostic dynamics capture, while separating latent guide-policy optimization from action extraction avoids over-conservatism, making the method theoretically sound.\n2. Comprehensive Experiments: TELS is rigorously tested on 5k–100k sample D4RL tasks (0.5–10% of original sizes) and outperforms 11 baselines. It also works in real-world data center cooling (20.17% ACLF, no thermal violations). Ablations (TS-IDM components, T-symmetry regularization) and OOD tests (Antmaze critical sample deletion) validate robustness.\n3. Clear Computational Efficiency: The appendix addresses ODE-related overhead concerns, showing TELS (2-layer MLPs, compact latent space) is efficient—JAX implementation trains in 20 minutes on 10k samples, outpacing baselines like TSRL (160 mins) and CQL (780 mins).", "weaknesses": "1. Unaddressed Dynamics Assumptions: TELS assumes invertible, deterministic environment dynamics (required for T-symmetry and inverse modeling), but real-world settings often have non-invertible (e.g., irreversible heat dissipation) or non-deterministic (e.g., noisy sensors) dynamics. No discussion of how TELS performs here limits its applicability.\n2. Limited Latent Interpretability: While TS-IDM learns “well-behaved” latent representations, there is no qualitative analysis (e.g., dimensionality reduction, correlation with physical variables) to show what these representations capture, weakening claims about their structure driving generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gNMrWetayb", "reviewer": "Reviewer_AstW", "summary": "This paper introduces a T-symmetry enforced inverse dynamics model (TS-IDM) that can learn well-behaved and OOD generalizable latent representations, and facilitate action inference. The proposed method outperforme existing offline RL algorithms on small datasets.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The paper is clearly written and logically structured. \nThe proposed method outperforme some offline RL algorithms on small datasets.", "weaknesses": "1.\tThe paper appears to be a straightforward combination of the POR and TSRL.\n2.\tThe compared baselines are relatively outdated and do not include comparisons with some recent and important baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DGgPsS9qfz", "reviewer": "Reviewer_qR8u", "summary": "This work introduces a novel T-symmetry enforced latent state-stitching (TELS) algorithm designed to improve the sample efficiency and OOD generalization in offline RL. While existing offline RL methods requires massive dataset to learn reliable policies, and perform poorly when an action has not been observed in training data, TELS addresses those by focusing on maximizing the utility of limited dataset. \n\nThe main technical contribution is the TELS mechanism, which learns a low-dimensional latent state representation. It can help creates a robust set of imagined trajectories, which serves as an implicit form of data augmentation and leads to a more reliable policy.", "questions": "1 See Weaknesses.\n\n2 Regarding overall learning objective (7) and Table 7: Selecting same weight $\\beta$ for $\\ell_{dyn}$ and $\\ell_{ode}$ is natural while selecting the same weight for the last term $\\ell_{T-sym}$ is not intuitive. a) It seems Table 7 is on Hopper-me rather than 10k Hopper-m (Check Table 6 performance score). b) Provide further evidence for weights $(1,1,5)$, $5,5,5$, $(0.5,0.5,1)$ and $(0.5,0.5,0.5)$ could be helpful to show the rationale for selecting a shared $\\beta$. A better illustration would be a performance score figure where x axis is weight of $\\ell_{dyn},\\ell_{ode}$ , y axis is $\\ell_{T-sym}$ , and z axis is the score.\n\n3 Can the latent state encoder learned by TELS transferable? What is the performance by re-using the trained encoder from related tasks?\n\n4 It would be great to demonstrate the idea of state-stitching by toy examples to show successful (or unsuccessful) stitched trajectories projected back into the original observation space. This would help the reader to understand the concept of state-stitching.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1 The TELS demonstrates superior performance, such as D4RL's medium replay datasets, showing its effectiveness in data-limited settings.\n\n2 The TELS policy is trained to navigate a wider and more reliably state-action space, reducing the conservativeness required by typical OOD regularization techiniques.\n\n3 The ability to stitch data allows the algorithm to handle disparate parts of the state space, making it highly suitable for real-world applications where data is collected sporadically.", "weaknesses": "1 TELS algorithms requires a complex model, including both RL components and particularly a separate high-quality latent state model with symmetry regularization. \n\n2 The performance of TELS depends heavily on the learned latent space being an accurate and consistent representation. \n\n3 While latent stitching is meant to improve generalization, if the environment dynamics violate the T-symmetry assumption, the learned policy may rely on imagined transitions that are impossible in practice, which may ultimately japardize its performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4iWgR3Yym7", "reviewer": "Reviewer_GJr5", "summary": "This paper proposes TELS, a sample-efficient offline reinforcement learning (RL) algorithm that leverages time-reversal symmetry (T-symmetry) to learn latent state and action representations. \nBy enforcing T-symmetry in an inverse dynamics model and optimising a guide policy entirely in latent space, the method improves out-of-distribution generalisation and sample efficiency compared to existing offline RL approaches. \nThe paper is clearly written, well-motivated, and supported by thorough experiments on both D4RL benchmarks and a real-world industrial control task, demonstrating strong empirical gains in small-sample regimes.", "questions": "1. One thing that I think has been overlooked is that this method won't work if the environment's transition dynamics are stochastic. Please can the authors comment on this?\n2. What is $h\\_{ivs}$ in Equation 9?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clarity and presentation:\n\n    The paper is clearly written and well-organised. The introduction and preliminaries effectively motivate the work, set the stage for the proposed approach, and position it well in relation to prior research. The method and experimental sections are easy to follow, and the paper maintains a strong logical flow throughout.\n\n- Methodological contribution:\n\n    Although the proposed method is a relatively straightforward extension of existing offline RL techniques, the authors demonstrate that this extension—enforcing time-reversal symmetry in the latent dynamics model—has a substantial positive impact on performance and generalisation.\n\n- Empirical evaluation:\n\n    The experiments are comprehensive and well-designed. The results show strong performance, particularly in data-limited offline RL settings, and provide convincing evidence of the method’s effectiveness and sample efficiency. The inclusion of both D4RL benchmarks and a real-world industrial control environment strengthens the paper’s practical relevance.", "weaknesses": "- Captions lacking detail:\n\n    In general, table and figure captions do not provide enough information.\n\n    - Figure 1: The caption should explain how to interpret the figure.\n\n    - Table 1: The caption should clarify what “±” represents (e.g., standard deviation?), what the bolded values indicate (best method overall or under a statistical test such as a paired t-test?), and where readers can find details about the reduced-size datasets.\n\n    - Table 2: In the third row, two values are bolded—please clarify what bolding signifies in the caption.\n\n    - Table 3: The caption should explain what each row represents, as this is not clear on first reading.\n\n- Missing normalisation details:\n\n    The paper does not specify how the normalised scores are computed. The authors should include this information, ideally in the captions of tables or figures where normalised scores are reported.\n\n- Figure 2 presentation:\n\n    Figure 2 could be improved by aggregating results across both environments (and potentially more). The authors may find the rliable package useful for this purpose. I recommend reporting the interquartile mean along with 95% stratified bootstrap confidence intervals.\n\n- Clarification on thermal safety violations (Line 358):\n\n    The paragraph beginning on Line 358 does not make it clear what causes thermal safety violations or what they reveal about the algorithm’s behaviour. Please add a brief sentence providing intuition.\n\nMinor issues:\n- Line 78: When introducing “latent representations,” clarify that these refer to latent state and action representations.\n- Line 102: The citation should be presented textually (i.e., integrated into the sentence rather than in parentheses).\n- Line 311: Clarify the meaning of the tilde (~) symbol. Should this instead read “5k–100k”?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SJGZiFrQ1J", "reviewer": "Reviewer_96gJ", "summary": "The paper proposes a sample-efficient offline RL method, TELS. The method builds on a T-symmetry-enforced inverse dynamics model (TS-IDM) for learning a latent space and offline policy optimization in the latent space. A guide policy selects a target for the next state, and the final action is extracted from the latent inverse dynamics module. The method is benchmarked on D4RL in a small-sample setting and on a real-world industrial control environment, and the method is found to outperform existing offline RL methods on these benchmarks.", "questions": "- Eq3: Do you use stop-gradient at all, e.g. for z_s, z_{s'} in the second term?\n- Would it be possible to ablate the second term of the deterministic policy loss by replacing it with a BC-like term in the action space?\n- Would you expect this method to be suitable for offline-to-online adaptation? I interpret that the l_{T-sym} term in the policy loss is essentially preventing the policy from going where the dynamics model is inaccurate, and if you fine-tune the encoder, you might run into non-stationarity issues. Is my understanding correct?\n- Could you discuss the potential trade-offs of using the T-symmetry prior? It could be a problem in manipulation tasks or other tasks with impacts. On the other hand, based on Table 1, the method works reasonably well in Adroit. Evaluating also on *-cloned and/or *-expert would strengthen the case.\n- Would the computation of the Jacobians be a bottleneck in the case of, for example, visual offline RL tasks, where the networks would benefit from being scaled up? Probably manageable, if you use a frozen backbone? If you have time and resources, V-D4RL experiments, for instance, could add value to the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method with the latent ODE and symmetry design is well-motivated and connected to prior work. The method appears to be novel and sound.\n- The method shows significant sample efficiency improvements over the chosen baselines.\n- The ablations in 4.2 are highly interesting, and they clearly illustrate the value of the ODE property enforcement and the T-symmetry consistency loss. The OOD experiments in Figure 3 are likewise impressive, and the method seems to be fast to train.\n- The fact that the learned representation also benefits IQL and TD3+BC supports the claim that the latent-space learning is valuable and generalizable.\n- Even though the overall dynamic model learning objective (Eq. 7) consists of multiple terms, equal weighting of all the loss terms is nice and speaks to the robustness of the method.", "weaknesses": "- No code made available.\n- You could consider citing and comparing to existing trajectory stitching methods in offline RL like DiffStitch [1].\n- The real-world experiment (Table 2) is very cool, but its significance is somewhat hard to assess. Given that the experiments were run once (as I understand the paper), there are no error bars, and the differences in results could simply be due to, e.g., how well the initial hyperparameter guess worked out.\n- The paper claims to \"completely bypass the conservatism issue caused by the action-level regularization\". However, the deterministic policy loss (Eq. 9) simply replaces the TD3+BC-like action-regularization with a next state-regularization term. The stochastic variant uses AWR over the next states. It is still somewhat unclear if this next state-regularization leads to different behavior and/or learning dynamics than action-level regularization in practice. The empirical results indicate that the difference-maker is the latent space and the advantage-weighed regression for the maze tasks.\n- The weight of \\alpha (Table 10) differs greatly between environments, raising questions about hyperparameter sensitivity.\n- As acknowledged by the authors, the ODE and T-symmetry regularizations can limit the model's expressive power, making this method specifically useful for small datasets.\n- Minor: Some issues with presentation & typos, for instance: Equations 9, 10 use h_{ivs} instead of h_{inv}, Appendix E \"Border Impact\", Zhan et al. 2025a and 2025b is the same paper cited twice, comparision (Table 11 caption), exhibt on L126, Training Perparameters (Table 9). \n\n[1] Li, G., Shan, Y., Zhu, Z., Long, T., & Zhang, W. (2024). Diffstitch: Boosting offline reinforcement learning with diffusion-based trajectory stitching. ICML.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)