Less Is More: Clustered Cross-Covariance Control for Offline RL
Authors: Nan Qiao
Sheng Yue
Shuning Wang
Yongheng Deng
Ju Ren
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning;offline RL; OOD area; Clustering-based RL;
Primary area: reinforcement learning
TL;DR: Squared error in offline RL induces TD cross covariance. We mitigate it with clustered replay $C^4$ and a gradient penalty, keeping guarantees and improving stability with up to 30% higher returns.

Abstract: A fundamental challenge in offline reinforcement learning is distributional shift. Scarce data or datasets dominated by out-of-distribution (OOD) areas exacerbate this issue. Our theoretical analysis and experiments show that the standard squared error objective induces a harmful TD cross covariance. This effect amplifies in OOD areas, biasing optimization and degrading policy learning. To counteract this mechanism, we develop two complementary strategies: partitioned buffer sampling that restricts updates to localized replay partitions, attenuates irregular covariance effects, and aligns update directions, yielding a scheme that is easy to integrate with existing implementations, namely Clustered Cross-Covariance Control for TD ($C^4$). We also introduce an explicit gradient-based corrective penalty that cancels the covariance induced bias within each update. We prove that buffer partitioning preserves the lower bound property of the maximization objective, and that these constraints mitigate excessive conservatism in extreme OOD areas without altering the core behavior of policy constrained offline reinforcement learning. Empirically, our method showcases higher stability and up to 30% improvement in returns over prior methods, especially with small datasets and splits that emphasize OOD areas.

Published as a conference paper at ICLR 2026
LESS Is MORE: CLUSTERED CROSS-COVARIANCE CONTROL FOR OFFLINE RL
Nan Qiao Sheng Yue? Shuning Wang Yongheng Deng" & Ju Ren Central South University, 2Snn Yat-sen University, Tsinghua University
ABSTRACT
A fundamental challenge in offline reinforcement learning is distributional shift. Scarce data or datasets dominated by outo---distrbuttion (OOD) areas exacerbate this issue. Our theoretical analysis and experiments show that the standard squared error objective induces a harmful TD cross covariance. This effect amplifies in OOD areas, biasing optimization and degrading policy learning. To counteract this mechanism, we develop two complementary strategies; partitioned buffer sampling that restricts updates localized replay partitions, attenuates irregular covariance effects, and aligns update directions, yielding a scheme that is easy to integrate with existing implementations, namely Clustered Cross- Cross-Covariant Covariance Control for TD (C*). We also introduce an explicit gradient- based corrective penalty that cancels the covariance induced bias within each update. We prove that buffer partitioning preserves the lower bound property of the maximization objective, and that these constraints mitigate excessive conservatism in extreme OOD areas without altering the core behavior of policy constrained offline reinforcement learning. Empirically, our method showcases higher stability and up to 30% improvement in returns over prior methods, especially with small datasets and splits that emphasize OOD areas. The implementation code is available at whhttss::/ /ttpt/gtttt///////////////t com com/NanMu2/CN
INTRODUCTION
Offline reinforcement learning (RL) learns policies from fixed datasets without further interaction, which is essential when exploration is risky or expensive (Sutton & Barto 2018). Although large
benchmarks with millions of transitions report strong performance (Agarwal al. 2019 Fujimoto
& Gu 2 Kumar et al. 2020a Chen et al. 2023), real deployments usually offer much smaller datasets with narrow state and action coverage (Nguyen-Taan & Arora 2024 Cheng al.) 2023).
Limited coverage enlarges cuttof-dditributiin (OOD) areas and stresses standard training pipelines
(Tkachuk al. 2024 Foster al., 2022: Jia et al. 2024).
To mitigate distribution shift and provide safety margins, conservative objectives have been widely adopted (Fujimoto & Gu, 202T] An al. 2021 Lyu et al. 2022 Kostrikov al., 2021a| la Peng et al.
2019). These methods implicitly assume that the dataset covers the relevant parts of the space (Kumar al. 2020ai Chen et al., 2024). Under weak coverage they can become overly cautious exactly where improvement is needed, and in extreme cases, this overconservatism can destabilize policy
learning (Cheng et al. 2023 Li et al. 2023b).
A second and less analyzed failure mode arises from the value fitting objective itself. Under distribu- tion shift and limited data, the procedure of temporal difference (TD) learning induces detrimental bias and feature co-adaptation that can culminate in training collapse, noted by prior work (Kumar et al. 2022 Yue et al. 2023). We identify the core cause: TD learning that minimizes the second moment of the residual, E generates harmful tross-time covariance of gradient features, which becomes dominant under severe OOD area. Specifically, our theory and experiments show that in OOD areas, TD updates induce three implicit regularizers. Two are beneficial for generalization, akin to the beneficial implicit regularization produced by noise in supervised learning (Mulayoff & Michaeli 2020 Damian et al. 2021). The third is cross-time covariance of gradient features, and
Part of this work was done when Nan Qiao and Shuning Wang worked at Tsinghua University. "orresponniig to: yuesh5@ma1 i 1 wysu.edu.co

---

Published as a conference paper at ICLR 2026
acts against the intended optimization objective and, under severe OOD, causes pronounced gradient interference and instability.
We address this challenge with two complementary strategies that operate locally on the geometry of the replay data. First, we partition the buffer by gradient features and train with single-cluster mini-batches, which removes the between-partition mean covariance. Second, we add an explicit gradient- corrective penalty with a tunable coefficient that mitigates the covariance-driver bias within each update. To prevent conservative objectives from becoming over-restrictive in extreme out- of-distribution areas, we include a lightweight divergence-based term that is neutral on distribution and activates only in OOD areas, which reduces unnecessary suppression while preserving the core behavior of existing conservative methods.
Putting these pieces together, our contributions are threefold. First, we identify a data-limited failure mode in which the squared TD objective induces a harmful implicit regularizer that degrades generalization and can trigger training collapse. Second, we propose C+2, which constrains cross- region covariance to significantly curb this effect, and we introduce gradient-based corrective penalty that further cancels within-cluster covariance. Third, while C*4 calls for small adjustments to sampling and loss in practice, it remains effectively pellu-and-play" for numerous offline RL algorithms with the optimization goals preserved. In experiments on small datasets and -ephassaaiieed splits, C+ delivers substantial and stable gains, with improvements exceeding 30% on several benchmarks.
RELATED WORK
Reinforcement learning with small static datasets. Traditional RL suffers from poor sample efficiency, and offline RL aims to address this issue by learning policies from fixed, pre-collected datasets without any interaction with the environment (Lillicrap et al. 2019 Lu et al. 2022). Under this offline learning paradigm, conventional off-policy RL approaches are prone to substantial value overestimation when there is a large deviation between the policy and data distributions (Kumar et al.
2020b Qiao et al.) 2026). Existing offline RL methods address this issue by following several directions, such as constraining the learned policy to be "close" to the behavior policy (Fujimoto et al., Kumar et al.) 2019 Fujimoto& & Gu (02I), regularizing value function on OOD samples (Kumar 2020a Kostrikov et al.) 2021b), enforcing strict in-sample learning (Brandfonbrener et al. Kostrikov et al. 2022). and performing pessimistic policy learning with uncertainty- based reward or value penalties Yu et al.) 2020 Ann et al., (021). Most existing offline RL methods adopt the pessimism principle and avoid policy evaluation on OOD samples (Fujimoto et al., [019 Yu et al. a0. This approach curbs error accumulation, but small or weakly covered datasets, it can become overly conservative and cause large performance drops (Li et al., (0236). This suggests a renewed bottleneck in sample efficiency. Recent work, such DOGE and TSRL, mitigates the issue by admitting carefully chosen iitto--diittibttion samples, for example, those within a convex hull or those that are cenaamcs--epplinatiee (Li et al., 2023b Cheng et al. 2023 However, these methods operate at the level of data selection rather than RL itself.
2019
ct al.
2021
Implicit regularization in deep reinforcement learning. Deep RL, driven by the deadly triad, often exhibits overestimation, out of distribution representation coupling, and value divergence.
effects that intensify when data are scarce and noisy (Sutton & Barto 2018 Baird 1995 Tsitsiklis & Van Roy 1997 van Hasselt et al. 2018 Yue al. [222]] Li et al. 2023b). Classical stabilizers such as target networks, Double Q. TD3, and normalization, together with linear and dynamical analyses, mostly mitigate symptoms without addressing the mechanism that fuels self excitation and OOD coupling (Mnih et al. al.
2015 Hasseli 2010; Fujimoto et al.) 2018: Bhhtt et al. 2024 Toffe & Szegedy
2015) AAchiam
2019 In the offline regime, policy constraint and pessimistic approaches
regulate what is learned to reduce OOD evaluation errors and error accumulation (Fujimoto et al.
2019 Kumar et al. 2019 Fujimoto & Gu 2021 Peng et al. 2019 Kumar et al., An et al.
2021). DR3 instead acts on feature geometry by penalizing the inner product over two features,
which reveals and counters an implicit regularizer implicated these instabilities (Nikulin et al..
2023 Kang et al. 2023 Kumar et al. 2023 2022). LayerNorm provides consistent stabilization with
NTK and spectral contraction explanations and with scale decoupling that further weakens this effect,
aligning with observations on representation stability and implicit bias (Ghosh & Bellemare 2020
Kumar et al. 2021 Durugkar & Stone 2018 Yue et al. (202)). Current work has not yet recognized
that this effect can also be suppressed at the sampling level.
2

---

Published as a conference paper at ICLR 2026
Clustering-based reinforcement learning Some recent work partitions heterogeneous offline datasets into interpretable behavior clusters to enable stable learning in local in-distribution regions (Mao et al.) 2024; Wang al. 2024] Hu et al. 2025 Wang et al. 2023). SORL alternates trajectory clustering and policy updates in an expectation maximization manner to reveal diverse high-quality behaviors (Mao et al.] 2024). Behavior-aware deep clustering extracts near single-peaked subsets and improves stability and returns (Wang et al. (2024). Probabilistic approaches model latent behavior policies with Gaussian mixtures and derive closed-form improvement operators for the implicit clustering (Li et al., 2023a). Diffusion QL fits multi- peaked behavior policy distributions with diffusion models and mitigates mode mixing bias (Wang al., 2023). Online skill discovery shows the value of learning separable skills in latent space and informs the design of clustering offline (Achiam et al. 2018). However, most recent efforts focus on diverse policy training and have not yet connected to offline reinforcement learning under small datasets.
3 PRELIMINARY
Offiine reinforcement learning. We consider the standard Markov decision process (MDP) M A, T,r, rr. do 7), with state space S, action space A, transition dynamics T S + P(S), reward function R : S x A + [o, 1). and initial state distribution μ : + P(S), where P(S) represents the set of distributions over S. Subsequently. policy za s) induces the discounted occupancy dt (s((s a) (1 - 7) Ei 7 Pr(s: s,a, a| 1T and maximizes the return E[_nn 7'r(Se, a)]
with S ~ [- 5t, at). In the offline setting a fixed dataset D = {(s, a, r', a a()) is collected by behavior policy Tg. Its empirical distribution de approximates tT and serves the notion of data support. The key challenge is extrapolation error which tends to assign spuriously high values to actions outside the support of dg. This motivates us to design both policy evaluation and policy improvement remain near the behavior distribution.
Policy evaluation in offline reinforcement learning. from a fixed dataset. The (s, a) is obtained by solving the temporal-difference regression problem
In offline RI., value functions are estimated
min (r) = E (s....r))) (s,s, a) - [r(s, a) + + - Qa a')) 2
(I)
where Q denotes the target critic corresponding to Qá. We have the temporal difference residual 8 = r(s, a) + + (aa= (s', a') (s a). Thus, the Problem [ is equivalent to minimizing the second moment of the temporal difference residual 5 under the dataset distribution, i.e.min E 8²), To make the evaluation robust near the dataset support, we reason about small perturbations of the critic through its layer features. For a unit direction w and a small magnitude 2 O, the first order expansion of the head gives
(x + kw) = (x(x) + l (w, V Qx)) for sb E (')
(2)
Policy improvement in offline reinforcement learning. Offline RL improves the ac- tor ıt on the dataset states by maximizing expected value under the learned critie, i.e. max, EgwD Bawr(-s) s)) [s,(s,a)) To mitigate extrapolation error, we add an explicit proximity regularizer that keeps action selection close to the behavior policy, which yields the generic objective
max Eg-D Ea- (s(s.a) - aa
s), TT(- [s)]
where a >> 0 controls the regularization strength and D(,,) measures divergence or distance between action distributions. Different algorithms instantiate with KL or Rényi divergences and use MSE
or MMD as practical proximity surrogates (Wu et al. 2019 Jaques al.) 2019: Metelli et al. 2020
Fujimoto & Gu 202T: Kumar et al. 2019). In addition, some methods adopt implicit behavior
regularization (Kumar et al.) [220bb] Yu et al. 2021b Lyu et al. 2022 An al. 202
We provide
details of these behavior regularizers in the Appendix
CROSS COVARIANCE EFFECTS IN THE TD SECOND MOMENT
In this section, we turn attention to conventional technique from reinforcement learning, temporal difference learning, and analyze how its second moment changes when the evaluation point moves slightly in the feature space toward the OOD area, following two observations.
3

---

Published as a conference paper at ICLR 2026
Mard& Sade
6 2
+U
MM MA
64
200
189
3.4
100
3444
1.0
03L
220
200
289
0.0
200
483
(a) Var dominated grad updates
(b) Implicit regularization beneficial and harmful
Figure 1: Left four panels report cosine similarities between VE[6"] and Varl( versus V(El))) under
dominates across henchmarks. Right four panels track Var 6] 2 k2 2k'' C and score, where larger A, B and smaller C correlate with better performance, indicating A and B act as beneficial implicit regularizers while C is harmful.
B|8a] = ( Var[ő], showing that the variance
Observation 1: Variance of TD residual plays an important role in Problem
To connect the TD loss in Eq. I with a feature space view, we start from the identity
E[6²] (El))) + Varlo),
which reduces the task to understanding how the variance changes under small displacements. As indicated by the left four panels of Fig. (a)) our analysis focuses on how Var responds to small perturbations in feature space. To probe out of distribution directions while keeping the input domain of Q+ fixed, we consider directional displacements x + x + k w. which sets up the subsequent Taylor analysis. To probe of distribution directions without changing the input domain of Qa. evaluate the heads at displaced features x + kw. Then, using the first-order Taylor approximation, the sample variance of the Q-values an OOD area, along w can be represented
Var(Qu(1 kw)) = Var(Qu(1) + k(w, (=)x()))) Var(Q(z) + k(w, ((())))))
= k22 Var((w, V Qx))))
(5)
Similarly, we have (ar(Qarss' (s')) = (k')Var((w'' Va (x')) where x' = (s',a') denotes
the next state action pairs of x (s, a) come from the dataset (detailed proof in Appendix B This keeps the analysis attached to the empirical support while we probe into support behavior virtually.
Observation 2: Implicit regularization of covariance should be well controlled.
We now state the main result that decomposes the variance change into a supervised style part and term that is unique to temporal difference learning.
w,,,,, Theorem 1. All expectations, variances, and covariances below are taken over k, k', w, w'. With the first order approximation for Q in feature space, the variance satisfies
Varlo] se (2(22)² Var((w', Qe) + 122 Var( (w, (=((())))
implicit regularizer noisy rupervised leurning, denote Term/A) and Term
22kk'' Cov( V Qu
V (z))
(6)
additional
unique TD learning, denote Termi
where x and x' are drawn from D, with x' being the next state action pair that follows x. By Eq. this variance decomposition directly controls the TD second moment minimized by Eq.
(4).
Sketch of proof. Expand Qy at x' and Qo at using the first order rule in feature space, separate the zero displacement part and the linear part, apply variance rules and bilinearity of covariance, then drop higher order terms. Full details are given in the Appendix B
Equation (6) has a natural interpretation. The first bracket penalizes large feature gradients and recovers the implicit regularizer from noisy supervised learning, which reduces Var[Q] in out-of- distribution regions per Eq. (5) and improves generalization. Same as supervised learning, noise
induces beneficial implicit regularization, and our effect is analogous(Mulayofi & Michaeli 2020
4

---

Published as a conference paper at ICLR 2026
Damian al. 2021). The second bracket is TD-specific because it couples feature gradients across a transition. This cross-term is misaligned with optimization and, since it enters Eq. (6) with a negative sign while the TD loss is minimized, updates tend to mecraase it, turning it into harmful implicit regularizer that can drive collapse in pronounced OOD regimes. Fig. [(b) cobroobrres this by showing that the first terms act beneficially while the cross term grows under TD minimization, with A, B, and C' approximated by traces of denoised gradient covariance.
Although Eq. (6) may look close the conclusion of (Kumar al. 2022 Yue al. 2023), our analysis and takeaways are different. We derive the result directly from the TD loss and the second moment identity rather than from Lyapunov style or gradient stability arguments, and we not assume optimizer specific behavior or noise alignment. Our decomposition retains two next state contributions that are missing prior work, namely Var (w', V.rQe/ (x'))) and its scaling by 2 which clarify when sensitivity to the future head dominates even if current state terms are controlled. We also pair (zr, x²) from the dataset instead of policy rollouts so the analysis stays on empirical support while out of distribution effects are introduced by virtual feature displacements inside the head. experiments, this pairing choice and the explicit treatment of the next state variance and the TD cross covariance improve stability and returns.
5 C²:: CLUSTERED CROSS-COVARIANCE CONTROL FOR TD
This section turns the TD>variance model into two control objectives and develops EM-style procedure that clusters gradient pairs and samples within a single cluster per update. Subsection 5. derives a matrix target from TD variance and sets the size and sign control objectives, Subsection 5.2 introduces clustering of stacked gradient pairs and shows why single-cluster sampling removes the between-clustee driver while within-cluster alignment controls the sign. Subsection[533 specifies sppesiiffss mixture- ceatuurrregulrrizeed objective, minibatch estimators, and the overall training procedure. All formal proofs deferred to Appendix
5.1 PROBLEM FORMULATION: FROM TD VARIANCE TO A MATRIX TARGET
For a transition with feature-space gradients g' = Fe.Qut (x') and g = Vz, (x)), the one-step variance admits
Var[ó:] Ri 2 (Ww + 222 Var (WW9)) 27yk'' Cov( (w',g'), (w,9)).
(7)
implicit regularizer as in noisy supervised learning
TD cross term
and, under a minibatch sampling law,
Cov((w) (ww((w, g'), (w.g)) ||O||2 < |||||,
C Cov(lg E ER*"
(8)
Thus, make the TD cross term harmless, (i) shrink size proxy of C (trace/spectral norm). and (ii) adding a penalty to offset the covariance - -2ykk' Cov ((w','' .g),, (w,g)).
Before Clustering
Clustering
Clustering
Clustering
15-
10
10
0-
-5
-10
-10
Surel Dee
-10
-5
Xarin
-10
10
Figure 2: Intuition behind clustering and TD covariance. The left panel shows that without clustering, the overall covariance ellipse mixes within-clusee cluster spread and between between-cluster offsets, so the TD cross term couples unrelated modes and its sign drift. Right panel clusters the stacked gradients minibatch from single cluster, which removes the betwcen-cluster driver and leaves updates governed by local within-cluster covariance C., The result more local TD updates, weaker spurious coupling across modes, and improved stability in OOD directions.
[g g) and samples each
5

---

Published as a conference paper at ICLR 2026
5.2 CLUSTERING THE STACKED GRADIENT PAIRS
Building on the tarre formulation in Section 5.1 where the TD variance decomposes as in Eq. (7) and the cross is controlled by the matrix C via Eq. (8). Fig. shows that treating the whole dataset as a single cloud mixes within-cluster spread and between- cluttr offsets, which makes the TD cross term unstable. This motivates clustering the stacked gradient pairs and sampling single-cluster minibatches so that updates are governed by local within-cluste statistics.
Let y/ [g's 9] E 22mm and partition the dataset into clusters in y-space. For cluster 2, define means 2 Ps. variances Σ'z, E,, and cross covariance C₂ Cov(g', 9 Z = z).
Theorem (Single-cluster sampling removes the between-cluster driver). With cluster label Z, the cross covariance decomposes
C C=
Hz)(
(9)
If each minibatch is drawn from a single cluster 2, the between-cluster term in Eq. that batch and, for any unit w', w.
vanishes in (9)
2-ykk Cov ((w', :g),, (w,g)) 22 ||C=1l2 S 22kk' vtr E Er.
(10)
Sketch of proof. Apply the law of total covariance to obtain Eq. Then use la Mb
|M and
the operator Cauchy-Seie Sccharrr inequality.
The theorem splits the source of the cross covariance into within-cluster component and a be- tween cluster component. If each minibatch is drawn from a single cluster, the newween-clustee term vanishes in that update, so the TD cross term is fully determined by the within-cluster cluster statistics C.. This yields the batch-level bound 2-kk' [C2 S 2-ykk' //tr 2 /tr Eg Thus single-cluster cluster sampling together with penalty on C²2ll stabilizes TD updates by suppressing the harmful cross term scale controlled by within-clusee variances.
5.3 MIXTURRF-ROUULARIEEDD OBJECTIVE AND TRAINING PROCEDURE
We fit K comoppnen Gaussian mixture on {4:} with parameters
[/] M2
>> C. C
p(y) Σ p= Nyy Mz, S2), μz
Ω2
(11)
Σg
Coupling TD fitting with spectral proxy gives
min St.
(TD(#)
Ps ||
Ps 220, 0,
1 Ω 0,
(12)
where || = tr(C.c upper bounds ||C,|| and is easy to estimate per minibatch.
Given single-cluster minibatch B C. estimate
Cz( = Covn(g', ,9), Reross (B) - -) (
(13)
JØ,, B) LTD(O, B) + Reroas (B),
(14)
which implements Eq.
12 stochastically and, by Eq.
10 controls the harmful term in each update.
C repeatedly clusters stacked gradient pairs to estimate within-cluster cross covariance C₂ and then performs critic updates using single-cluster minibatches. Each update minimizes the TD loss plus Frobenius-she penalty C., By Theorem 2 this bounds the harmful cross-term batch-wise. The result a minibatch distribution tailored to reduce |[C'| and stabilize TD in OOD directions.
TRAINING ON CLUSTERED BUFFERS
The last section examines periodic dataset clustering to control cross-covariance. The clustering design remains open challenge. In offline RI, TD updates occur during evaluation, and policy

---

Published as a conference paper at ICLR 2026
Algorithm 1 C*: Single-Cluster Cluster Offline Update for TD (EM-style)
Input offline dataset D. number clusters K, regularizers X, 3. iterations Initialize mixture (P.. S2 (2,) K
critic Φb.
for t= 1..... IT' do Compute gradients: for each sample i, form g', and stack = [g', E step: Fiz P= N(w M2, R2), normalize 2, Pia = 1.
M- Mtstep: Pa - ; E. Piz, Critic minibatch: sample cluster ~ (p=(((p)). draw minibatch B with weights ris-
+ N; E: Pre Ω2 + N- E. Pi -aaamaa ua) + el, extract Ca.
Penalty and update: compute (B) Cov (g'. g). minimize batch loss [(b, B) Eq. (14.
end for
Output trained critic é and mixture (P2, M2, St2).
improvement imposes policy constraints. Periodic clustering can reshape data geometry and shift support across clusters, which may compromise these constraints during improvement. Designing clustering that preserves them is an important direction.
To this end, this section explains why training with clustered buffers (single-cluster minibatches as in Algorithm [ has limited impact on the policy improvement objective and, in fact, provably optimizes computable lower bound of the canonical mixture objective. We specialize the discussion to the stye improvement surrogate and connect each step to Appendix
CQL improvement target and global lower bound. For a policy x and state consider the transformed CQL target
Ucqu ;8) = (*))) ((I-7-")- x² (T((
(15)
where x2 is the Pearson divergence (Definition D. Lemma 2 in the Appendix implies the statewise lower bound
(TQU(T::) 2 [=(, - (mlTs)
(16)
and the same form holds when KL penalty or its nonnegative combination is used. Thus, within short local window where V is linearized by E, ()) policy updates that increase the r.h.s. of
Eq. (16) improve a global lower bound to Eq. (55 This shows that replacing the long-horizon transformed penalty by tractable per-state divergence primarily tightens the bound and does not qualitatively alter the improvement direction.
Mixtures, clustering, and why per-cluster training is safe. Let the behavior mixture v = M
=1=1 Wm Vm (e.g., a Gaussian mixture induced by clustered replay). Lemma in the Appendix shows f-divergences are convex in the second argument:
(T(l
Wm (T(TT
(17)
with the special cases (Pearson, KL) holding verbatim. Plugging Eq. 17 into Eq. (16) yields the cluster-decomposed lower bound
Ex ExIQ(s, [)] - -f(2
> Wm ( + +3 Σ wm KL (Pm))))))))
(18)
m
m
for any nonnegative combination of Pearson and KI.. Consequently, optimizing the per-cluster surrogate
[x) = E, a)]
+ (KL()
(19)
gives an unbiased stochastic gradient of the weighted sum Σ., Wmn (x) in in Appendix), which is a computable lower bound to the mixture objective with v inside each divergence. Thus, partitioning the buffer and training per cluster does not bias the direction: it maximizes a principled lower bound to the original improvement target, with the gap controlled by divergence convexity. Additionally, the f-divergence case is illustrative rather than exclusive. Any policy constraint satisfying (( Σ. w. v2)) < Σ. w, D(T ([2)) can be used in our algorithm. See Appendix A for admissible D choices.
and sampling (Lemma
7

---

Published as a conference paper at ICLR 2026
Stability from adding KL and quantitative caps. Proposition in the Appendix proves that adding a KL term yields strong concavity of the local surrogate and quantitative step caps:
IV
near 0g= strong concavity 2 m33,
110 - 0ell S
(20)
mBp
For the Gaussian-mean case.
# وg + K* Egg. (2ape Rapp 3p) K² =1
K²
(21)
3p
B
and Pearson inflation is bounded by ( S (2())
when > 0. By convexity in the
second argument (LemmaB)) the same caps hold per cluster and therefore under cluster sampling in expectation. This shows that partitioning the buffer does not compromise the known CQL stabilizing effects. If anything, makes the constants local and often tighter.
Takeaway for practice. Equations 16 (8) show that training with clustered buffers maximizes a certified lower bound to the original mixture objective (mixture placed inside the divergence), with unbiased gradients under random cluster selection. KL caps Eq. (20) 2T) carry over per cluster, so the induced change to the policy improvement direction is small (same direction, cluster-adaptive step). In the CQL special case, the p-weighted divergence still upper-bounds the propagated penalty, and convexity guarantees let replace the mixture by sum over cluster penalties without loosening control. Therefore, splitting the buffer into clusters has a limited effect on the improvement rule while improving its computability and stability, exactly the properties exploited by C#4 in Algorithm [
Pointers to Appendix. Formal statements and proofs are in Appendix D Lemma (global operator frrezze), ( in the second argument and unbiased cluster gradients). Proposition 2 (isotropic penalties and local update direction), and Proposition 3 (KL. stabilization and caps). Remark quuantiiiss poor-coverage regimes and explains why adding KI prevents runaway steps. The same reasoning applies per cluster.
7 EXPERIMENTAL RESULTS
In this section, we empirically evaluate our proposed method CA.. The experiments are organized to address the following questions:
Q1. How does it perform on offline RL relative to existing approaches across standard bench- marks, particularly under reduced-data regimes?
Q2. How is performance influenced by factors such as the number of initial clusters and the quality of the data?
Q3. Can the plug-and-play method C adapt to different types of algorithms?
7.1 IMPLEMENT
For fair and comprehensive evaluation, we compare our method against Behavior Cloning (BC)
and a broad set of state-of-the-art offline reinforcement learning algorithms. For standard offline RL backbones, we include COL (Kumar al. 2020a), TD3+BC (or TD3BC) (Fujimoto & Gu 202T).
and IQL (Kostrikov et al. 2022). To directly probe performance in reduced data regimes (Q1), we further consider data-efficient algorithms such as DOGE (Li et al. (023b) and TSRL (Cheng et 2023). DOGE is selected for its strong out-of-distributi generalization through cate-conditioned distance functions, while TSRL exploits temporal symmetry in system dynamics to improve sample efficiency. In addition, we include recent high-performing methods, BPPO (Zhuang et al. (023)). which leverages PPO-style clipping for monotonic improvement, and A2PR (Liu et al. 2024), which employs adaptive regularization with VAE-augmented policy.
We further study the plug-and and-play nature of C+ and its relation to other regularization techniques by comparing against methods that act generic regularizers share similar design principles. In particular, to mitigate gradient collapse in sparse data regimes, we evaluate DR3 (Kumar et al. 2022) and LN (Yue et al. 2023). We additionally consider SORL (Mao al. 2024) which is closely
8

---

Published as a conference paper at ICLR 2026
Table 1:
Normalized scores on MuJoCo locomotion tasks using reduced-sizz datascts (10k samples), Abbrevia- tions fr, mr, and me denote full-replay, medium-replas and medium-expert respectively.
Task
TD3+BC
CQI.
IQI.
DOGE
ВPPO
TSRI.
A2PR
Ours
Ant-me
12.0=18.2
74.0425.0
66.0=10.5
82.0#16.4
85.5+13.7
83.6::1244
66.7::00.
100.9=5.0
Ant-m
46.0=17.4
62.01:2211
56.0199.3
69.0
78.0+119
12.2+:00.6
64.0:9.3
84.5:-6.1
Ant-mr
31.0=15.5
36.0116.3
41.0=10.8
46.0112.9
52.0110.8
49.42:3.1
44.0=9.1
65.816699
Ant-e
72.0=25.0
4
82.0=15.0
98.04:20.3
103.0=11.2
100.74126
88.0110.6
109.6:2.77
Ant-fr
70.0:224.0
92.01:26.5
80.0=15.0
96.01:20.0
102.0=11.7
99.81:1..0
86.0110.4
107.6:3.2
Hopper-m
30.7=13.2
50.11:22.3
61.016.2
55.6=8.3
55.0=7.8
60.914411
55,948.4
69.7
Hopper-mr
11.3:44.7
13.2420
16.24330
19.1=3.3
45.1=8.7
23.54888
12.5+599
Hopper-me
22.6:13.9
43.2:6.9
$1.747..0
36.81:44.5
27.9115.2
$6.6113.9
49.71:10.7
81.3:6600
Hopper-e
53.6+17.11
56.14-26.4
60.9419.6
62.2+-2.7
85.0+17.9
76.7+2004
80.0+16.8
107.0
Hopper-fr
32.0=13.5
45.0122.0
56.016.3
54.0=8.4
60.0=9.7
53,41113
55.0=8.9
65.3:19.4
Walker2d-m
11.2=19.2
54.11:5.5
34,245..2
53.74:2..6
54.71114
17.3
5.9:5522
65,9177.8
Walker2d
9.3+66.6
13.8+5.3
17.74889
15.5 9.2
2975-8.7
27.6+122
34.4+8899
5.94
Walker2d-me
12.4=15.7
26.0111.0
38.0=12.2
42.5
61.3112.2
50.9126.4
$6.5
96.3
Walker2d-e
29.5=23.5
46.0+29.4
16.243322
8122+18.6
102.0-19.7
104,9=10.6
98.0=7.9
009
Walker2d-fr
14.2:19.5
55.0+16.0
36.01:5.6
55.5+:123
$2.0
14.3
48.0=9.6
77.
Halfeheetah-m
25.9:18.4
41.7=222
35.6:12.9
42.8=2.9
28.5=332
43.3#2.8
37.1=2.7
46.3-1..1
Halfcheetah-mn me
29.1:18.3
16.3:249
34,116.33
26.3=3.1
34.4
27.713.8
23.6:14.7
43.1
Halfchectah-me
23.5=13.6
39.7=6.4
14.347.3
33.1-888
22.3-9.6
37.2#14.9
32.4-8.3
46.013.55
Halfcheetah-e
26.44-4.2
5.8+1.3
-1.14388
144-3.1
6.54344
42.0+26.4
36.0+7.88
75,8--5,2
Halfcheetah-fr
28.0:18.6
45.0=2.4
33.013..0
43.0=3.1
37.0=3.6
41.0:13.0
39.0:24.1
58.1133.4
Locomotion-Avg.
31.5
46.0
41.4
50.7
56.1
57.2
50.6
75.7
AntMaze-Arg.
10.7
21.4
20.5
22.0
16.2
27.0
Maze2D-Avg.
103.6
106.1
120.6
115.7
Adroit-Arg.
15.2
8.4
23.1
15.7
21.6
related in motivation performs data clustering, but does so at the trajectory level rather than in the gradient space O does. Across all comparisons, C is instantiated as plug-in module on top of existing backbones, allowing us to isolate its effect on performance.
7.2 MAIN RESULTS
To answer Q1, focus primary evaluation data-scarce regime. Specifically, restrict each D4RL MuJoCo locomotion task to only 10k state-action pairs (approximately 1% of the full dataset). This setting places all methods challenging low-data regime, thereby highlighting their generalization capabilities. As summarized in Table we benchmark a wide spectrum of algorithms, including standard backbones (TD3+BC, CQL, IQL), data-efficient methods (DOGE, TSRL), and stronger OOD-awand baselines (BPPO, A2PR). Table primmrrly reports the normalized scores on the locomotion benchmarks and summarizes the main results on AntMaze, Maze2D,, and Adroit, while detailed per-task results for these three domains are deferred to Tables 3 and in Appendix F To complement the tabular comparison, Figure Soauee3proviinn provides a holistic visualization by plotting normalized scores across tasks. On the MuJoCo locomotion benchmarks, despite the extreme data sparsity, approach recovers nearly 75% expert performance on average and consistently outperforms all competing baselines, achieving average improvement of more than 30 % over the best alternative. This demonstrates that C substantially enhances data efficiency offline RL.
Ours
LN
DR3
SORL
110
110
52
72
28
mE
ms
Hopper
Halfcheetah
Walker2d
Figure Radar charts comparing normalized scores on D4RL MuJoCo locomotion tasks (10k samples).
To examine Q2, move beyond performance and analyze computational efficiency and sensitivity to key hyperparameters. Tigure44repont wall-clock training time over 300K optimization steps, comparing both full algorithms and plug-in regularizers. Incorporating C into standard

---

Published as a conference paper at ICLR 2026
Runtime of stepsstpps
Runt 10XK steps
73
61.0
1030015501
542
60-
0002
T010G-C1
Normelized Sore.
COL-CC1
40.3
26.5
1031C
5
20
60
20
30
21
2.5
0.
10
50
5.0
25 30
100
25
30
56 12.
Figure 4: Wall-clock runtime comparisons, and performance sensitivity hyperparameterss À and ax
backbones such as CQL and TD3+BC introduces only moderate overhead, yielding a runtime profile comparable to other lightweight regularizers (LN, DR3, SORL) and substantially more efficient than complex data-efficient baselines such as TSRL and DOGE. We also study the sensitivity C* to its regularization strength À and the base algorithm coefficient a. Performance remains stable over a broad range of these values, indicating that C does not require delicate tuning. Additional ablations in Appendix E further show that reasonable variations in the number of initial clusters and the quality/coverage of the dataset lead to smooth changes in performance, supporting the robustness of the gradient-space clustering mechanism.
Finally, to directly address Q3 regarding the plug-and-play property of C+,, we evaluate it as an add-on regularizer for both CQL and BD3+BC. Using identical training protocols, we compare each backbone to its variants augmented with LN, DR3, and C (Figures 5 and Across both algorithm families, incorporating C yields the most consistent and substantial improvements throughout training, whereas LN and DR3 provide only moderate or task aaskdeeendent gains. This indicates that C effectively targets the variance components in gradient space that limit offline RL performance, while preserving the inductive biases of the underlying algorithm. In practice, Oa can therefore be treated as a drop-in module that robustly enhances a variety of existing offline RL methods.
Ha.Fcheetah-&lllrredaaa
Happet-radim-eepeett
Antemedum--eplay
Wi lder2dde
Normalized Sooe
1.00
CQL-DR3
CQL-C4
100
100
100
CQL-Lh
75-
75
EQL
50-
25
25
25-
25
50
100
150
200
50
100 Epoch
150
200
50
100
150
200
0
50
100
150
200
#poch
Epoch
Epoch
Figure 5: Performance comparison on CQL vs. (+LN), (+DR3) and (+C*)).
HalCchetah--fll-eea
Hopaes-sddmm-expett
Antemedum--uplay
Walker2d 2d-epert
Normalized Score
1.00
TB3RC+061
1230CICe
100
100
100
TD3DC+
1225C
50-
25
25
25-
25
75
150
225
300
75
150 Fporh
225
300
150
225
300
0
75
150
225
300
Spoch
Epoch
Fpoch
Figure 6: Performance comparison on TD3BC vs. (+LN), (+DR3) and (+C*).
Additionally, extended implementation. benchmark descriptions, experimental details, complete results, and ablations are provided in Appendix E due space constraints.
8 CONCLUSION
This work identifies harmful TD cross covariance as a key driver of instability under weak coverage in offline RL. Ca counters this effect with partitioned buffer sampling that localizes updates and with an explicit gradient- based penalty that offsets bias while preserving the objective's 's lower bound. The method reduces excessive conservatism, improves stability, and integrates with standard pipelines without heavy tuning. Experiments on small data and splits that emphasize out-o--distribution states show consistent gains in return, with improvements up to about 30% and smoother learning dynamics. These results indicate that clustered cross-covariance control for TD is a practical and effective approach for achieving robust offline RL under weak coverage.
10

---

Published as a conference paper at ICLR 2026
9 ACKNOWLEDGEMENTS
This research was supported in part by the National Natural Science Foundation of China under Grants 62432004 and 62572496, the Shenzhen Science and Technology Program under Grant JCYJJ025000445550000 the Young Elite Scientist Sponsorship Program by CAST under Contract 2B2025-218, and grant from the Guoqiang Institute, Tsinghua University.
10 REPRODUCIBILITY STATEMENT
We release the full codebase supplementary files. The repository contains scripts for end-to-end runs, environment setup instructions, and exact configurations. The paper provides the algorithm pseudo-code and complete list of hyperparameters. We will include data preparation steps, evalua- tion scripts, random seed control, and instructions to regenerate all main tables and figures.
11 ETHICS STATEMENT
This work targets decision-making under distribution shift in offline reinforcement learning. The method can improve reliability, yet misuse may create harm. Potential risks include job displacement, unsafe behaviors in autonomous systems, privacy exposure when training on sensitive data, and misleading outputs from generative components. We recommend careful auditing, limited deployment in controlled settings, and human oversight. Use should focus on supporting decisions rather than replacing human judgment. Privacy protection, transparency, continuous monitoring, and rollback procedures are necessary. We will provide usage checklist encourage responsible application.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms, 2018.
www.r/////b////// ab0/aasss.11.. 10299
Joshua Achiam, Ethan Knight, and Pieter Abbeel. Towards characterizing divergence in deep Q-learning, 2019.
1903. 08894
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle Seattle, WA, USA, Tech. Rep, 32:96, 2019.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified Q-ensemble, 2021.
1110001544
Leemon C. Baird. Residual algorithms: Reinforcement learning with function approximation. In Anmand Prieditis and Stuart J. Russell (eds.). International Conference on Machine Learning. pp. 30-37, Tahoe City, California, USA, 1995. Morgan Kaufmann, ISBN 1-55860-377-88 doi: 10.1016//978-
Aditya Bhatt. Daniel Palenicek. Boris Belousov, Max Argus, Artemij Amiranashvili. Thomas Brox, and Jan Peters. Crossq: Batch normalization deep reinforcement learning for greater sample efficiency and simplicity, 2024. htthpsts:://arxx wwwpaawwwwww./a/////////a///
00630.055665555 05605
Christopher Bishop and Nasser Nasrabadi. Pattern recognition and muchine learning, volume Springer, 2006.
David Brandfonbrener, William F Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL battps:/openrevieer
Jayy Chen, Bhargav Ganguly, Yang Xu, Yongsheng Mei. Tian Lan, and Vaneet Aggarwal. Deep generative models for offline policy learning: Tutorial, survey, and perspectives on future directions, 2024. URL JRI.tttss:: Pnttps://arxx www..org/gaa.... /2402. 13777

---

Published as a conference paper at ICLR 2026
Liting Chen, Jie Yan, Zhengdao Shao, Lu Wang, Qingwei Lin, Saravan Rajmohan, Thomas Mosci- broda, and Dongmei Zhang. Conservative state value estimation for offline reinforcement learning, 2023. URL httttt/s::/aaatt http://axivvv.ooo abs http//bb//333000 22002 06884
Peng Cheng, Xianyuan Zhan, Wenjia Zhang, Youfang Lin, Han Wang, Li Jiang, al. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl. Advances in Neural Information Processing Systems, 36:7612-7631, 2023.
Alex Damian, Tengyu Ma, and Jason D. Lee. Label noise sgd provably prefers flat global minimizers, 2021. [tttsss:://aaxx org/ abs 0ttpb///2116.665600
Ishan Durugkar and Peter Stone. TD learning with constrained gradients, 2018. URL https
Dylan J. Foster, Akshay Krishnamurthy, David Simchi-Levi, and Yunzong Xu. Offline reinforcement learning: Fundamental barriers for value function approximation, 2022. URL htttttss////xiii
org/ abs//bb//a 21111 10919
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep
data-driven reinforcement learning, 2021. URL htt.:htttss///rtttt wwwp:xivvvrrrreeooo org http://b///200.0.. 9
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018. httll/tppe//aarvv.orrraaa/////// 09477
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Dibya Ghosh and Marc G. Bellemare. Representations for stable off-policy reinforcement learning, 2020. URL
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110(2)399-4166 2021.
Hado Hasselt. Double Q-learning. In Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23, Curran Asso- ciates, Inc., 2010. URL http:: httptpp:://prroceeeeenneeeee ings neur cc wwtppper.ppppeeeeeeeee faterr__ffesepprrrrrra BDDD1f/feeeSSSSSeccoooa2333333a3
Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation, 2017. (KK.:httpps//arrxiiorr/////////1
08475
Hao Hu, Xinqi Wang, and Simon Shaolei Du Policy-based trajectory clustering in offline reinforce- 09202
ment learning. 2025.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelaaatnnn deep network training by reducing internal covariate shift, 2015. ULL.::tss//axx...orr///////
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of
implicit human preferences in dialog, 2019.
Zeyu Jia, Alexander Rakhlin, Ayush Sekhari, and Chen-Yu Yu Wei. Offline reinforcemeentteamii leamnng Role of state aggregation and trajectory data, 2024. UBLLattpss///xxxvvccr////////2/2 17091
Bingyi Kang, Xiao Ma, Yirui Wang, Yang Yue, and Shuicheng Yan. Improving and benchmarking offline reinforcement learning algorithms, 2023. URL
6
00972
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning 2021a.
12

---

Published as a conference paper at ICLR 2026
Iyy Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization, 2021b. URL htt.ttps//aax...ooo/////a//
08050
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Qleaamnin. In International Conference on Learning Representations, 2022.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizzigg off-polcyyy Q-learning via bootstrapping error reduction, 2019.
00949
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-eeannin for offline reinforcement learning, 2020a. URL /tt.:///////....///////////////
http://axxivv.oo
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp. 1179-1191, 2020b.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under paraettrrttttnnnnooonnnn inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021. URL
net. wtt..eeeoorommmmo-00ooii.oo
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR3: Value-based deep reinforcement learning requires explicit regularization. In International Conference Learning Representations, 2022. URL httthtttt///peeeeeeee wwt.:/ewww.eee.tteeeeeeeeeee
1d=POvMvL191F
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline Q- learning on diverse multi-task data both scales and generalizes, 2023. URI. https://arxiv
org/abs 2211. 15144
Jiachen Li, Edwin Zhang. Ming Yin, Qinxun Bai, Yu-Xiang Wang, and William Yang Wang. Offline reinforcement leamning with closed-form policy improvement operators, 2023a. URL. https
//arxiv.org/alt http///bb//21..... /211 15956
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In International Conference on Learning Representations, 2023b. CtRI.https:/// //tp::ss///ppeeeiieee net.iiiwwweeeeeeeeeee..e..eeeeee forum?
[CL-1MOTCCooo
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforeement learning, 2019.
URL https: /tap::///arxivy wrw.iiivorrrrreeee wttg/aa//ggaa.../ 02971
Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive advantage-guided policy regularization for offline reinforcement learning. In Forty-first International Conference Machine Learning, 2024. URL httpt::://///...tt. /ttp::::////eeeeeee.... net http://ooommmiiiV3333303
Rui Lu, Andrew Zhao, Simon S. Du, and Gao Huang. Provable general function class representá- tion learning in multitask bandits and mdps, 2022. URL [RRLattts:///axi.rrrppp
15701
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative Q-learning for offline rein- forcement learning. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL het/forum? 1
Yihuan Mao, Chengjie Wu, Xi Chen, Hao Hu, Ji Jiang, Tianze Zhou, Tangjie Lv, Changjie Fan, Zhipeng Hu, Yi Wu, al. Stylized offline reinforcement learning: Extracting diverse high-quality behaviors from heterogeneous datasets. In International Conference on Learning Representations, 2024.
Alberto Maria Metelli, Matteo Papini, Nico Montali, and Marcello Restelli. Importance sampling techniques for policy optimization. Journal of Machine Learning Research, 11 2020.
13

---

Published as a conference paper at ICLR 2026
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks, 2018. URL
05957
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 2015.
Rotem Mulayoff and Tomer Michaeli. Unique properties of flat minima in deep networks, 2020. 04710)
URL htk.ttttss://aattt
Thanh Aguyen-Tang and Raman Arora. On sample-effccient offline reinforcement learning: Data diversity, posterior sampling, and beyond, 2024. URL https://arxiv..
2201
03301
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, Dmitry Akimov, and Sergey Kolesnikov,
Q-ensemble for offline rl: Don't scale the ensemble, scale the batch size, 2023. URL https
//arxiv. abs http//bs/222100 11.2211110001
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-wcighted regression Simple and scalable off-policy reinforcement learning, 2019. URL htttss://ax.v..rrg
abs/1910. 00177
Nan Qiao and Sheng Yue. FORLER: Federated offline reinforcemmet learning with Q-ensemble and actor rectification, 2026. (RLpa:ts:://axttt
hwtp:///g//aaaaeeeee.e http://///aaaa...oo///////////// 02055
Nan Qiao, Sheng Yue, Ju Ren, and Yaoxue Zhang. FOVA: Offline federated reinforcement learning with mixed-quality data. IEEE Transactions on Networking, 34:2031-2046, 2026. doi: 10.1109/
TON.2025.3637043
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http://pp//cccoo.....eeeeaeeeees html
Volodymyr Tkachuk, Gellért Weisz, and Csaba Szepesvári. Trajectory data suffices for statistically efficient learning in offline with linear q" gerrealizability and concentrability, 2024. URL https
7/arxiv. ott/ abs 2405 6809
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033, 2012.
J.N. Tsitsiklis and Van Roy. analysis of temporal-difference difference learning with function approxima- tion. IEEE Transactions Automatic Control, 42(5):674-690, 1997. doi: 10.1 18909988777
Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Somnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad, 2018. URL httLptttts////xxv..oor
1812.022 02.422648
Qiang Wang, Yixin Deng, Francisco Roldan Sanchez, Keru Wang, Kevin McGuinness, Noel O'Connor, and Stephen Redmond. Dataset clustering for improved offline policy learning.
2024. URL.tttps: //apxii////xxvv arxiv wwrg/aaaa... 2202 09550
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive pol- icy class for offline reinforcement learning, 2023. URL httpps:/aaavv...oo/bbb///////// wrtp//////bbaaaaa///////////
06193
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, 2019. BRtthttss///axiii http://axwvvvo../eee//// wwtp/aa//gg......///////
Ming Yin, Mengdi Wang, and Yu-Xiang Wang. Offline reinforcement learning with differentiable
function approximation is provably efficient, 2022. URL http///pps///aatt.oo////////////
00750
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-brie based offline policy optimization, 2020. URL https: //arxivv
orgg/bbs/200 2005. 13239
14

---

Published as a conference paper at ICLR 2026
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offiiee reinforcement learning. Advances in Neural Information Processing Systems, 1111111111116,, 2021a.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offfine model-based policy optimization. Advances in neural information processing systems, 34:28954 28967, 2021b.
Yang Yue, Bingyi Kang, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Value-consistent represen- tation learning for data-efficient reinforcement learning, 2022. /ttp
abs/2206. 12542
Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, and Gao Huang. Understanding, predicciin and better resolving Q-value divergence in offline-rl. 2023. URL https::/aaxx...oorr/bb/////
04411
Zifeng Zhuang, Kun LEI, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. In International Conference Learning Representations, 2023. URL https;
7 //openreview. net htte://ormmm1dd31133pp
15

---

Published as a conference paper at ICLR 2026
OUTLINE
Introduction
2 Related Work
3
Preliminary
Cross Covariance Effects in the TD second moment
C²:
Clustered Cross-Covariance Control for TD
Problem formulation: from variance to matrix target
Clustering the stacked gradient pairs
5.3
Mixture-regulanzeed objective and training procedure
6
Training on Clustered Buffers
6
7
Experimental Results
8
Implement
8
7.2 Main Results
8
Conclusion
10
Acknowledgements
11
10 Reproducibility Statement
Ethics Statement
11
Common Policy Proximity Constraints Offline RL
17
Appendix: proof details Theorem
18
Appendix: Proofs for Section
19
Mixture of Policies
20
Experiment Details
23
Extended implimmmennt implementation
23
Benchmark
23
E
Evaluation more benchmarks with reduced datal
23
Generalization and compatibility with SOTA offline RL methods
................................
24
Scalability consistency
25
6
Covariance control under
26
E.7 7
Robustness and efficieccy
27
Use of Large Language Models
34
16

---

Published as a conference paper at ICLR 2026
A COMMON POLICY PROXIMITY CONSTRAINTS IN OFFLINE RL
In this section, we measure per-state proximity between the learned policy я and the behavior policy Ty and average over the dataset state distribution PD. and the global constraint is
C(T||==)
Ex-pp
(
8), 11: TTa. ())]
(22)
Furthermore, the mixture upper bound we will check is (T| E, 1FTT.) S E, w; (. wi z
Σ, w/ 1 in Corollary
KL. The Kullback-Leibler divergence is
DKL (T (T))) = f x(a () log
x(a s) TT(aa 3)
da,
(23)
finite only when supp(x) S supp(T;), and the reverse form (Te(Te]|T)) is behavior-sample estimable, and as an diviergeence KL is convex in the second argument, hence satisfies
wi,xs) DKL. (-)) W
E: wi (T]|((/((TT))
Rényi. The Rényi divergence of order 0,000111 is
log / x(a Tz(a s)1- da, a
D (ग][TT)
(24)
A
DR π Σ, W.Ta) and D + DKL as a + 1. and for a E (0, 1] it is convex in the second argument and obeys
Σ. u; (F((TT))) a (0, 1], while for a > this inequality is not
generally guaranteed.
d-div. The didverggence family generated by a convex x with x(1)
(x(T(TT)= f
m(ass) T ns(a|s), dr,
(25)
with the Pearson chi-square special case
- ( TT(aas))
(
(((al))
)
(x)(T]) = Exnq
(26)
as fdiieergences they are convex in the second argument and satisfy Dj (T Ee (iTT.)
5
Σ, W. (F
JSD. The Jensen-Shannon divergence is
JSD(T|| (F) = 1/2 (T) + 1/2 DKL( n' n'= (T + xg).
(27)
symmetric and bounded in [0, log 2]. and being an f f-divergence it convex in each argument and thus JSD ( Σ, WiTTs)
< E w,JSD( [[
W1. The Wassersstnin distance admits the Kantorovich Rubinsteen dual
W (T,, πT) = sup (E== - Ey-olf(v)l) If 1pp
(28)
which is supremum of affine functionals and hence convex in each argument, and therefore it obeys
W1 (म. E ); < Σ, w. WT Tg.).
MSE. A practical surrogate for discrete actions is the mean squared error between probability vectors
MSE( (T TB)
Σ
(T(a s) (a(a s))
(29)
LAI
or parameter-Ste space MSE for Gaussian policies continuous actions, and since ग. ++ || - 1 is convex have MSE (T, Σ, w;Tr,) ) - Σ; wo MSF( T..)-
17

---

Published as a conference paper at ICLR 2026
MMD. With positive- definite kernel & and feature map ó in the associated RKHS, the maximum mean discrepancy is
MMD mp) = -I Marg
7 Mr = [B===((()))
(30)
and because the kernel mean embedding is linear while the squared norm convex, it satisfies MMD² (T,, E, s; S E, (r,MD
Corollary 1. The mixture upper bound is valid for KL, all X-divergences including JSD, W,, MSE and MMD, and for Rényi only when E (0, 1]. and it is not generally valid for Rényi with a > 1.
B APPENDIX: PROOF DETAILS FOR THEOREM 1
Assumption 1 (Local Smoothness). Let D denote the dataset support. There exist constants L, > such that for any E DD and perturbation k with [k] S. p. the critic Qs is differentiable (almost everywhere) and has an L-Lipschitz gradient. Under this assumption, Taylor's 's theorem gives
Q0 kw) Qx(z) k(w, V Vu S - k2.
Thus, the expansion in Eq. (2) is standard first-order approximation with controlled O(k)) remainder. [T] Similar local differentiability assumptions are common in recent theoretical work on standard offline RL (Yin et al. 2022 Li al. 2023a Qiao & Yue 2026).
Proof Theorem / Fix a dataset pair (x, x'). All expectations, variances, and covariances this proof taken with respect the perturbation proposal (k.,', w, w'), while (xr,²') is held fixed. The perturbed error is
r 7 Q (x' +kw) Q0(x + kw).
(31)
Apply the first-order expansion in feature space and discard O(k2 k²2) terms:
(x + k(w2) Qx(x')
w/2 V. Qar (z')),
(32)
Qs(x + kw) Qo(x) w. v=Qx(=)).
(33)
Suppose kt w/, is OOD action (same (nn al. 2021 and (s' a') D. plugging (s', (Ts)))
(s', a') + ki wi into above:
Qs ((s', (s')) + k/w1) = ((s((s a') + k(w + k)w') ~ Qs (w', Qo (x'))
where k/ w/ + K/w's k'w' and x' (s',a') denotes the next state action pairs (s, a) come from the dataset. Thus,we have
~ + Q2 (x') - (x) (k'(w', Valor - k(w, (=Qa()))
(34)
6parn
(u)
To show this, we suppose the Q-value predictions for the in-distribution state-action pairs coincide, (e)Qu(z), vb ~ {ő. )'). which is common used in RL (An et al. 2021). That is, the base term òpase a constant. Therefore Var(ōbase) and Cov (Sbase, Olin) O, so
Var [8] Var Var(fin(u))
(35)
Expanding the variance and using bilinearity of covariance yields
Var 8) ? Var K (w', V= + Var (R(w, Var(M(w, V=Qo(z)))
27 Cov K'(w', V (x')), k(w, v=Qo(z)))
(36)
When k 1e' are treated as fixed scalars inside the proposal, Eq. (36) reduces exactly to the three terms in the main text's Eq. (6). with moments taken over (w, w'). By the identity E[8?] (E[ 2 Var 0 the corresponding second-moment statement follows.
'Wile deep ReLU networks are not globally smooth, they are locally Lipschitz almost everywhere (Hein & Andriushchenko 2017). Our theoretical model captures the dominant first-order effects of cross-covariance within this local trust region, which aligns with empirical observations of instability.
18

---

Published as a conference paper at ICLR 2026
Remark 1 (on the base term and the vanishing covariance). In the per-sample proof above the randomness is only over the perturbation u = (k, k,, W, w') while (z,x') is fixed, so the base term
òbase + (xx) - Qx(x)
a constant and therefore Var(dvase) and Cov (ōbaee d 0 exactly. When viewing Eq. (4) at the dataset level (averaging over (I,z') D), within the short locality window used in the main text on-support predictions are already accurate and conditional reward noise is negligible; hence Sbase is approximately constant over D. In that the k- d-dependent covariance term vanishes to first order, and all displacement dependence arises from Var(ônn).
Proposition 1 (block form under perturbation randomness). Let
S Cov ('), S = Cov (oo(ww), N Cov (l'w''' w', kw),
(37)
where all covariances with respect to the perturbation proposal. Then the variance contribution in Eq. (36) admits the quadratic form
Var [6] 72 Q0 S2 V Ve Qo((()))) + S₁ voQ((z)
27 Qv)) N (V))x()).
(38)
Corollary (cqual-direction simplification). Ifk = w w' w, and l Cov(w).
then
Var 8] k2 (vV (vV==QQr(x''))) V (x), D 1000--22ol V=Qolz)))
(39)
In particular, if w isotropic on 9m-1 so that -. I, then
1.2
[ar
(c - v=Qu(x))1
(40)
C APPENDIX: PROOFS FOR SECTION 5
Proof of Theorem 2 Let C' = Cov(g' g). With cluster label Z E K.
Covlg' = [[(g' - Eg' (g - Eg) || = E B ((g' Egy' (g - Eg) T z
E (g] 2lz + 2lz - Eg') - pz + pz - Eg) T 2
E Cov((' 9 |) Cov(('zz uzz)
(41)
which is Eq. 9. If minibatch is drawn from fixed z, then the effecctive covariance is Cz. For
any unit w', w. |w" w/ & ||Ca1l2 by the definition the operator norm. Morcover, ||Cx|le S
|>
1/2
E
2
1/2 by the operator Cauchy- Schwarz incquality, and | I tr(E) PSD Σ, yielding
Eq.
Lemma 1 (Alignment inside a cluster controls sign and magnitude). Let the SVD of C2 be C2 UEVT with singular values ai 2 02 2 ... 2 O. For unit w' w,
w C.C. w 2 01 coso' cos - 02 sin 6' sin
(42)
8' = (w',
0 = (w,,
Choosing w' and w vn yields w C2W or 2 0. The cross term in Eq. (8) is then
nonpositive. Sketch of proof. Expand the singular bases and bound the residual with 02 via Cauchy-Schwarz.
Proof. Let C2 = T with Σ = diag (01, .2.... ). For unit w', w, w TT C.C. (UT (UT'') T TV w) Eaz(u] (u, w).
(43)
Let cos 0² = uu w' and cos = oJ w. By Cauchy-Schwart and Σ Enzr(u] w/2 = sin 8¹, Ejze(1, w)2 sin² 0, obtain
Σ w/ (v) w) 2 - 02 122
[(uJ wryz j22
ΣOo (o) w)2 = - o sin oin 0' sin 0. (44)
J22
Combining Eq. 43 and Eq. 44 gives Eq.
Choosing w' Un w = 11 yields w T C.w
a 2 0. Thus, the proof is completed.
19

---

Published as a conference paper at ICLR 2026
D MIXTURE OF POLICIES
Definition (f divergence and special cases). Let f -0) be a proper lower semicontinuous convex function with f(1) = 0. Let and be probability measures on the action space that admit densities with respect to a common reference measure. Assune π < v so that tN is well defined almost everywhere. The f divergence
[(a)
D=(T|lu)=
da.
v(a)
satisfies Dy(=[l)) 2 0 and Dy(= if and only if = v almost everywhere whenever f strictly convex at 1. Special cases: are
x{a)²
(x(1|() = D D(c-1)=(=111)==
v(a) = da
and
KL(T|Pp) Dz (T][[)] T(a) log Taa) v(a)
da.
Lemma 2 (Global operator freeze and lower bound). For any nonnegative measurable function and any policy
- (T->PT--11 Σ
DC
1 (PT)'h 1 11
- For the transformed CQL improvement target at a state Let p== T-y
(*((s) a - -=
one has the global statewise lower bound
(*) - (=(l(a))) 2 (x( a a sup (=
The same bound holds with KL or any nonnegative combination of x2 and KL.
Proof. Monotonicity and the geometric series bound yield
1 1- -11111000
(I (I->Pm)-
I1
1=0
Set hi.)) = (() then linearize (*(s) to E,, (Q(s, a)) at the working point. Both penalties are nonnegative, hence the inequality is preserved.
Lemma (Convexity in the second argument and clusterwise training). Let satisfy the definition above. Let v Em=r Wmmm with Win 2 0 and Σm Wm = Assume & Um for every m with Wm > 0. Then
M Σ m=1
(y(T|lu) S
Wm (y(T(((mm)))
Consequently the surrogate
- - I xm (T|[mm) + Σ Wm KL( (Pm) m
m
is computable lower bound to the same expression with the mixture placed inside each divergence. Sampling a cluster index M w and updating with the per cluster objective
T [Q) ( ( + [TK((T|PMM))
gives an unbiased estimator of J
Special cases: For Pearson and for Kullback Leibler the same inequality holds, therefore the same lower bound and the same stochastic cluster training rule apply when V' are Gaussian components of mixture.
20

---

Published as a conference paper at ICLR 2026
Proof. Define the perspective g(u, f(u/v) for 0 and 0. Since f is convex, 9 is jointly convex. For each a,
( (())),
Σ m
Wm Wm (a) S
Σ
Wmg (=(a), Pm Pm(a))
m
Integrate over a to obtain
(T(TT))) 1
Σ Wmn Dy (T(
Taking expectation with respect to M wo proves that the per cluster gradient unbiased estimator of the gradient of the weighted sum objective.
Proposition 2 (General isotropic penalty and update direction). Fix state and let the policy class be zy with a mean parameter p. Let Σ be positive definite and assume the penalty has the isotropic form
(p mb) = v ((u 14a) ( Σ (1u 1
148))
with dp differentiable and strictly increasing on [0, 20). Let Q(+ a) linearized at = pg with gradient g(s). Consider
max
A:= pc, c e 0.
Any maximizer satisfies
μ² p²(s)= - up + x (s))
where (s))s= 0 is the unique solution of
2A v/ (x) 1=
R: g(8) Eg(s).
Special cases: (f)(r)= = - which corresponds to Pearson under equal covariances then
R
(s) (s)=-1
W
2R
2X
where W is the Lambert function given by W (z)e W() = The argument R2(2X²) is positive when
R > 0 and X > therefore the principal branch gives a positive x². If (f)( = à which corresponds to Kullback Leibler under equal covariances then
K (8)
X
For a Gaussian mixture behavior one applies the convexity lemma to replace the mixture penalty by the weighted of per cluster penalties and solves the same scalar equation per cluster with the corresponding E and μag.
Proof. Let A = μ and define
F(A)= = AT g AMAA E-)A).
The first order stationarity condition is
vaF(A)= = 2\ T - A) - - = 0.
Therefore A is colinear with Eg. Let A kEg and R Eg. Then
2A A (k2 R) K 1. Strict increase of v implies unique positive root. Substituting (br) = - gives the Lambert equation
1
K E
2X
which reduces to the stated form by y = x2R R and ye2u = R/(2X?). Substituting ((r) = r/2 gives 1/A.
21

---

Published as a conference paper at ICLR 2026
Proposition 3 (Adding KL gives stabilization and quantitative bounds in Gaussian cases). Consider for a fixed state s the surrogate
[= a)] ((ax2(=11 ( (T((T))))
0, B> 0.
Assume the policy class admits a local quadratic lower bound on KL around reference parameter 8g
(L(TTe|TTA)) z ||0 83ll²
for some constant > 0. Then near 0g the surrogate is m3p strongly concave in 0 and the unique maximizer satisfies
II
10 - 0sll -
m33
Special cases: If n and T are Gaussians with equal covariance Eg and parameter 0 μs, then the maximizer has the form
F pB + K Egg.
(2ape" x² I +
+.3p)
=1,
R= g Egg.
This implies the step size cap
1 1 1-7
K²
30 8
and if a >> O one also has
x2 (TT
R <
8
1.
2a
For Gaussian mixture behavior policy the convexity lemma yields
KL
Σ
Wm
Σ Wm ( m
x²
Wm Mm
S
Σ
wmx (TT((
YY
So the same lower bound structure and per-cluster step control apply.
Proof. The local quadratic lower bound on KL implies that subtracting 3pKL. adds curvature at least m33p, which gives strong concavity and uniqueness 0g. The parameter step bound follows from the basic inequality for the maximization of a strongly concave function. In the Gaussian mean case
(L (p) EB)) マ (u T Ep (μ 14))
and the stationarity condition becomes
g 2ape
23 A 3p2, A -0
With A ng this reduces to the scalar equation in and yields the stated bounds. For mixtures. apply convexity in the second argument to both divergences and optimize per cluster.
Remark (Effect under poor coverage and relation to mixture training). a measurable set A satisfies La Taa 8) da = 8 > 0 while A = Tz(a s) da is small, then
8²
(
This shows the sensitivity of a pure Pearson penalty when coverage is poor. Adding the KL term enforces curvature and gives the cap
3 --14 2a
(TH S
which stabilizes the update in low density regions. When the behavior policy is a Gaussian mixture, convexity in the second argument of both divergences yields additive upper bounds that justify training by data clusters with random cluster selection while increasing a computable lower bound to the true mixture objective.
22

---

Published as a conference paper at ICLR 2026
E EXPERIMENT DETAILS
E. EXTENDED IMPLEMENTATION
Our method is designed to be plug-and-play and can be readily incorporated into numerous existing offline RL. frameworks. In the experiments, we instantiate on top of both CQL and TD3+BC. Unless explicitly stated, CQL serves as the base algorithm, and our method is indicated with the notation "Ours". We implement all experiments in PyTorch 2. 1.2 on Ubuntu 20.04.4 4 LTS with four NVIDIA GeForce RTX 3090 GPUs. The actor and critic are ReLU multilayer perceptrons with four hidden layers of width 256. We train with Adam using batch size 256, learning rate 1 x 10 for the actor, learning rate x 10-1 1 for the critic, and discount factor y = 0.99. Unless stated otherwise, we use default hyperparameters following CQL (Kumar et al. 2020a). In addition, we use the network's penultimate layer (the learned features) as a computationally tractable surrogate for vx)) We follow the D4RL protocol for normalized scores. Given task score and the corresponding random and expert scores (as reported in Table 15), the normalized score is defined
score - random score
normalized score = 100 x expert score random score
Unless noted, we report the mean normalized score over to 5 repetitions per setting.
E.2 BENCHMARK
We evaluate method on the widely recognized offline RL benchmark D4RL (Fu et al. 2021 Todorov et al., 2012), which encompasses several domains, including Locomotion, Maze2D, AntMaze, Adroit, and Kitchen as illustrated in Figure 7 The locomotion domain includes four continuous robotic control tasks ("HalfCheetah", "Hopper", "Walker2d", and "Ant"), each providing four dataset quality levels (expert, medium-expert, medium, and medium-replay). The Maze2D domain requires 2D agent to navigate to fixed goal positions across three maze sizes ("umaze", "medium", and "large"), featuring both dense and sparse reward variants. In addition, we consider the Adroit suite of high-dimensional robotic manipulation tasks ("pen", "door", "relocate", and "hammer"), using both "human" and "cloned" datasets. We further include the AntMaze benchmark for long-horizon navigation with sparse rewards, covering eight maze configurations ("umaze", "umaze-ddversee. "medium-play", "medium-diverse", "large-play", "large-diverse", "ultra-play", and "ultra-diverse") Finally, we evaluate the Franka Kitchen environment, 9-DoF multi-task manip- ulation benchmark, using the "mixed", "partial", and "complete" settings, which require the agent to accomplish multiple conliconditioned sub-tasks within single episode.
Subsequently, we evaluate 4 by addressing the following key questions:
1. Evaluation on more benchmarks with reduced data: How does C² compare against recent state-of-the-art offline RL, particularly within the challenging 10k-sample setting?
2. Generalization: How does method perform across broader suite of datasets, especially in extremely small data regimes?
3. Scalability and consistency: there a performance trade-off on large datasets, and does our method consistently outperform baselines across the entire spectrum of data regimes?
4. Covariance control under 04: Under the C+ intervention, can the covariance be effectively maintained a low level?
5. Robustness and efficiency: How sensitive is the method to initialization? What do ablation studies reveal, and does C²4 introduce additional computational overhead?
E.3 EVALUATION MORE BENCHMARKS WITH REDUCED DATA
To rigorously evaluate performance in the small-data regime, the primary motivation for our approach,
Table 2 reports normalized D4RL scores across tasks containing at most 10k transitions. This
"To avoid expensive double backpropagation, we use the feature layer proxy. This controls input sensitivity via the chain rule, akin to layer-wise Lipschitz, constraints (Miyato et al. 2018 Gouk et al., (202T). "e treat this a practical engineering approximation. Our code provides both versions, and we recommend the exact V, -haseed implementation for theoretical rigor.
23

---

Published as a conference paper at ICLR 2026
Halfcheetah.
(b) Hopper.
Walker2d.
Ant.
(e) Maze2D-open.
Maze2D-umaze.
Maze2D-medium
(h) Maze2D-
2F
ii Antmaze-umaz
(i) Antmaze-medium
(k) Antmaze-late large.
(I) Antmaze-ultra
(m) Adroit-pen.
(n) Adroit-door.
(o) Adroit-hammer.
(p) Adroit-relocate.
Figure 7: Locomotion, Maze2D, AntMaze and Adroit tasks.
selection encompasses the sample-reduced locomotion tasks (Hopper, HalfCheetah. Walker2d) as well as naturally sparse datasets: Kitchen-complete, Pen-human, and Door-human. Our method demonstrates superior generalization, achieving the highest average score of 58.1 and consistently outperforming strong baselines like BPPO and TSRL in these strictly lata-constrained environments.
Furthermore, we evaluated additional D4RL datasets beyond locomotion, including Adroit, AntMaze, and Maze2D, with results reported Fig. and Tables 3 4 and The results show that our method delivers consistently strong performance across these diverse benchmarks and achieves remarkable gains in the vast majority of small data regimes.
E.4 GENERALIZATION AND COMPATIBILITY WITH SOTA OFFLINE RL METHODS
Improvements on standard baselines. We first examine the generalization capability of C as plug-in module for established baselines. comparess CQL with and without the C module on the full D4RL locomotion datasets. Rather than degrading performance, Cdd increases the total normalized score from 695.5 to 742.1. In tasks where the base algorithm already performs strongly C maintains competitive performance. while in more challenging moeree2d-mmiiuu-rpp..
(e.g., Coopper-ediummexeee
settings (c.g., Hopper-mediun suggest that data density increases, the gradient distribution becomes more stable and the C
yields notable gains. These observations
24

---

Published as a conference paper at ICLR 2026
Table 2: Normalized D4RL scores on tasks with 10k samples per dataset. We compare performance on reduced-same sample locomotion tasks and inherently small datasets (Kitchen, Pen, Door).
Task (510k)
BC
TD3BC
CQL
IQL
DOGE
BPPO
TSRL
A2PR
Ours
Hopper-m
28.8
30.7
50.1
61.0
55.6
55.0
60.9
55.9
69.2
Hopper-mr
19.7
11.3
13.2
16.2
19.1
45.1
23.5
12.5
45.9
Hopper-me
38.2
22.6
43.2
51.7
36.8
27.9
56.6
49.7
81.3
Halfcheetah-m
40.2
25.9
41.7
35.6
42.8
28.5
43.3
37.1
46.3
Halfcheetah-m mr
25.2
29.1
16.3
34.1
26.3
34,4
27.7
23.6
43.1
Halfcheetah-mn
33.7
23.5
39.7
14.3
33.1
22.3
37.2
32.4
46.9
Walker2d-m
25.4
11.2
54.1
34.2
53.7
54.7
47.3
5.9
65.9
Walker2d-mr
2.5
13.8
17.7
15.5
29.5
27.6
34.4
55.4
Walker2d-me
35.1
38.0
42.5
61.3
50.9
56.5
96.3
Kitchen-c
33.8
51.0
10.2
5.7
8.3
55.5
9.5
71.5
35.8
117.8
85.7
-0.1
85.3
Door-h
0.6
10.7
4.3
-1.1
25.9
0.3
-0.3
5.8
Average
24.4
15.5
31.8
35.8
30.9
49.5
38.9
26.3
58.1
Pen-human Data)
Deor-human Data)
Hammer-humon [Sk Data)
Relocate-human (5k
100
Normal leedallzedd Score
Normal Score
Normalloos lood Score
DRG
Ous
Normalized
60
40
20
ma
ToD dggh
sceet poae
moree
3
Figure 8: Comparison on Adroit tasks over different data sizes.
penalty naturally adjusts, avoiding over-regularization and retaining the strengths of the underlying algorithm.
Compatibility with stronger offline RL methods. Beyond standard baselines, we investigate whether C can further enhance recent sfate-of-the-art (SOTA) algorithms,, specifically BPPO (Zhuang al. 2023) and A2PR et al. 2024). While these methods achieve strong performance on standard benchmarks, they remain susceptible overfitting in strictly limited data regimes (e.g.. the 10k- sample setting). Theoretically, both algorithms rely on temporal-difference (TD) learning for value estimation: A2PR utilizes Q-learning updates, while BPPO employs SARSA-style value warm-up. Consequently, they inherit the vulnerability to high cross-covariance terms in the TD error when data is scarce. We postulate that C serves as a complementary regularizer to these approaches by explicitly controlling the covariance induced by TD updates. To test this, we integrate C" into BPPO and A2PR without altering their core mechanisms.
The results, summarized in Tables 7 and g indicate substantial performance gains. For BPPO (Table 7). adding C consistently improves performance across diverse tasks, raising the average normalized score from 38.3 to 53.5 relative improvement of 39.6%. For A2PR (Table 8). the effect is even more pronounced: the method achieves an average score 57.6 compared to the baseline of 31.0, corresponding to 85.9% relative gain. These findings suggest that uncontrolled cross-covariance is a fundamental bottleneck even for advanced offline RL methods, and 4 provides robust, algorithm-agnen agnostic solution to mitigate this issue in small-data regimes.
E.5 SCALABILITY AND CONSISTENCY
central question for any regularization technique designed for data searcity is whether it hinders learning when data is abundant. In this subsection, we examine whether our method imposes performance trade-off on large datasets and how consistently scales across varying data budgets.
Adaptation to large-scale benchmarks. To address the first question, we assess how our plug-in approach compares against state-of-the-ar methods specifically tuned for general offline RI. bench-
25

---

Published as a conference paper at ICLR 2026
Table 3: Average normalized score on D4RL Adroit tasks with reduced-size datasets (5 10k samples).
Task
Size
TD3BC
CQL
IQL
LN
DR3
SORL
DOGE
ВРРО
TSRL
A2PR
Ours
Pen-human
5k
9.5
41.2
71.5
45.3
44.1
68.1
35.8
117.8
85.7
-0.1
85.3
Hammer-human
5k
1.7
4.9
4.3
5.4
4.6
1.6
-1.0
2.4
0.3
-0.3
5.8
Door-human
10.7
2.9
11.8
10.4
4.9
-0.8
25.9
0.6
-0.2
12.6
Relocate-human
5k
-0.1
0.3
1.5
0.33
0.28
0.1
-0.2
-0.1
0.2
0.1
3.2
Pen-cloned
10k
-0.2
1.8
35.9
32.5
2.0
30.1
33.5
30.4
38.4
-0.1
58.2
Hammer-cloned
10k
0.5
1.5
0.6
0.4
0.7
-0.2
8.4
0.4
-0.3
Door-cloned
10k
-0.3
-0.6
2.6
-0.2
-0.3
-0.6
0.0
-0.1
0.1
-0.3
2.8
Relocate-cloned
10k
-0.2
-0.3
1.4
00.2
-0.1
-0.3
0.0
0.2
-0.2
0.2
1.4
Average
7.3
15.2
12.0
7.7
13.1
8.4
23.1
15.7
-0.1
21.6
Table Average score on D4RL, Maze2D tasks with reduced-size datasets (10k samples).
Task
TD3BC
CQL
IQL
LN
DR3
SORL
DOGE
BPPO
TSRL
A2PR
Ours
Maze2D-open
12.3
52.3
63.6
22.1
37.0
58.0
70.1
75.4
57.0
35.7
83.0
Maze2D-umaze-dee
104.5
74.1
103.6
86.5
117.6
63.4
115.1
51.4
138.0
125.5
130.3
Maze2D-unaze
13.5
35.9
68.6
17.9
76,9
72.5
52.8
58.9
76.9
100.6
Mare2D--meiummdeeeee
100.1
68.8
85.7
119.4
100.9
81.2
119.9
108.7
Maze2D-medium
76.6
80.1
104.4
107.3
93.2
96.4
127.7
132.0
118.0
156.0
122.3
MazeeD-large-denan
91.9
78.5
181.7
105.8
133.5
115.4
149.3
217.2
160.0
155,9
166.1
D-Iaree
28.5
42.7
103.2
115.4
82.2
58.3
126.5
228.1
140.0
97.3
152.9
Average
49.7
57.4
103.6
74.8
89.4
83.3
106.1
120.6
115.7
111.4
126.9
marks using full replay buffers (containing millions of transitions). Table the normalized scores on locomotion tasks. CQL+C" demonstrates robust performance, surpassing standard base- lines like IQL and TD3BC, and remaining highly competitive with recent strong methods such as BPPO and A2PR. While primary contribution lies in solving the small- mallesample dilemma, these results confirm that C is a safe and versatile plug-in: it significantly boosts data efficiency in sparse regimes while automatically adapting large-scale datasets without requiring manual tuning or deactivation.
Consistency across varying data budgets. To scrutinize scaling behavior more granularly, we visualize the performance trends as the training set size grows. Finst, we benchmark eight offline RL algorithms on the challenging Adroit tasks (Pen, Hammer, Door, and Relocate) under cloned regimes, shown in Fig. 10 Each curve traces the performance across budgets of {1, 5, 10, 100, 500} thousand state-action pairs. Our method attains the highest returns on most manipulation tasks and scales favorably with data availability, pointing to strong robustness and transfer capability under contact-rich dynamics.
Separately, assess Hopper, Walker2d, and Ant) across diverse dataset qualities (medium-expert. expert, medium, medium-replay, and full-replay), shown in Fig. 9 These curves utilize budgets of (3, 10, 50, 100) thousand pairs. While all baselines naturally benefit from larger datasets, our approach consistently achieves the top trajectories and exhibits the clearest gains on replay datasets. This suggests that C not only adapts to scale but also generalizes better to varied dynamics and distribution shifts throughout the learning spectrum.
offline RL algorithms the standard Gym domains (HalfCheetah,
E.6 COVARIANCE CONTROL UNDER 4
We monitor the cross covariance between current and next-state input gradients throughout training. For a mini-batch B, define gi = V V.Qo(r:), V. Qu (zz)). The empirical cross covariance matrix
Σg - Eien (gf ()) g) We report the dimension-normaliin trace tr, tr,())) = m Er))) which targets the harmful TD component in the variance decomposition Var(š] x A
1
26

---

Published as a conference paper at ICLR 2026
Table 5: Normalized score on D4RL AntMaze tasks with reduced-size datasets (10k samples),
Task
TD3BC
CQL
IQL
LN
DR3
SORL
DOGE
BPPO
TSRL
A2PR
Ours
AntMaze-umaze
20.9
21.1
69.8
48.1
41.1
65.3
78.9
53.3
74.3
69.2
68.7
MuiMaze-umaze-divee
16.8
43.3
58.1
29.7
0.5
39.1
43.8
45.1
57.6
27.5
65.9
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.3
0.0
0.0
7.2
AmtMaze-medium-pla
0.0
0.0
0.4
0.0
0.0
0.0
0.0
0.0
0.0
0.3
2,9
AntMaze- lagge--verre
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
9.8
AntMaze- laMMaze-largeplay
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.2
7.3
Average
6.3
10.7
21.4
13.0
6.9
17.4
20.5
16.5
22.0
16.2
27.0
Table 6: Comparison of CQL with and without C* on full-size (see Table 15) D4RL locomotion
datasets. The module improves the total score without inducing performance degradation.
Method
Hop.-m
Hop.mmr
Hop-me
Half.-m
Half--mr
Half-me
Wal.mm
Wal.-mr
Wal.-me
COL
58.5
95.0
105.4
41.0
45.5
91.6
72.5
77.2
108.8
CQL + C*
85.9
100.7
89.4
48.5
44.7
91.6
81.8
90.9
108.6
B B- 2ykk' Cor( (w', Vo (w, voQu(x))) In Fig. 11 we visualize tr (2i)) over
training. Across Locomotion tasks, C keeps tr, (Eg) substantially lower than LN, CQL, and DR3 during most of training, with the gap most pronounced on replay datasets. This indicates that C+ suppresses the cross covariance while preserving stable learning.
E.7 ROBUSTNESS AND EFFICIENCY
In this subsection, we empirically study the robustness of our method to key hyperparameters through ablation studies and analyze its computational efficiency. We also provide visual insights into the clustering mechanism.
Cluster Visualization. To intuitively understand the learned representations, we visualize the clustering structure for CQL+C* and TD3BC+C*4 in Fig. 12 and Fig. 13 respectively. For each method, we sample snapshots across the full training process and plot the stacked gradient pairs after t-SNE reduction to two dimensions, with points colored by their mixture assignments. The figures demonstrate that our method effectively separates distinct gradient modes throughout representative stages of training.
Sensitivity to the number of clusters K. We first investigate the impact of the number of clusters K on performance. (0] a sensitivity analysis on 1ak-sa D4RL tasks. We observe that performance improves rapidly K increases from and quickly plateaus around K 3-5. Performance remains stable even when is as large as 20, indicating that the method not overly sensitive to this choice. Based on these results, recommend K low-coverage datasets.
robust default for small,
For larger sample regimes, we further examine the effective number of clusters used by the algorithm when initialized with K = 20. As shown in Table 11 only a handful of clusters are effectively occupied, and this number decreases as the dataset becomes denser (from 10* to 106 samples). This suggests that for large or high-coverage datasets, K can be safely reduced without sacrificing performance.
Sensitivity to the penalty weight X. We next study the effect of the penalty weight À in the cross-covariance regularizer. Table summarizes the results across medium ("-m"), medium-replay ("-mr"), and medium-expert ("-me") D4RL tasks. We find that the optimal À depends quality and the dispersion the feature space. For high-quality datasets (e.g.. medium exxiut--epper), trajectories tend to form locally dense clusters with small within- covariance, and a relatively small penalty (A E [0.05, 0.1) suffices. In contrast, replay-style datasets or those with pronounced distribution shift exhibit more scattered features and larger cross-covariances, benefiting from larger
dataset
27

---

Published as a conference paper at ICLR 2026
Table 7: Performance of BPPO with and without C4 on 10k-sample datasets (normalized scores).
Method
Ant
Halfcheetah
Hopper
Walker2d
AntMaze
Pen
Door
Kitchen
Avg.
BPPО
85,5
22.3
27.9
61.3
63.4
30.4
-0.1
15.9
38.3
BPPO C*
87.1
36.1
56.0
85.2
66.2
74.5
-0.1
23.1
53.5 (+39.6
Table 8: Performance of A2PR with and without C 10k-sample datasets.
Method
Ant
Halfcheetah
Hopper
Walker2d
AntMaze
Pen
Door
Kitchen
Avg.
A2PR
66.7
32.4
49.7
56.5
42.5
-0.1
-0.3
0.3
31.0
A2PR C
92.6
49.0
83.6
101.3
75.8
47,9
0.1
10.2
57.6 (+85.9%)
penalties (e.g. [0.3, 0.5)). practical heuristic is to choose such that the weighted penalty term is on the same order of magnitude as the
loss.
Computational efficiency of clustering. We now analyze the computational profile of the gradient- space clustering procedure. Theoretically, implement clustering via an appectation-Maximizaai algorithm with complexity O(K N for N samples, K clusters, feature dimension m, and number of EM iterations & (Bishop & Nasrabadi 2006). Importantly, do not re-cluster at every gradient step: we subsample representative points from the replay buffer and perform clustering only periodically. Once clusters are formed, computing the localized covariance penalty during each critic update is linear in the batch size and adds negligible overhead.
Empirically, Table reports the proportion of total training time spent in clustering across tasks and dataset scales. For our primary target regime of scarce data (e.g.. 109-10' samples), the overhead is about 1% of the total training time, which is effectively negligible. On larger, million-scale datasets, the overhead increases to roughly 15-20% which we consider reasonable trade-off given the substantial stability and performance improvements (up to 30% gains return) observed in our main results. Overall, the cost of C'4 remains comparable to that of commonly used regularizers.
Clustering update frequency. Finally, we examine how often clusters need to be updated. Table 14 reports the performance of CQL+C* when varying the clustering frequency, defined the number of critic updates between two clustering runs. We observe a clear "sweet spot" around 100-200 updates: updating too frequently (e.g., every 20 steps) can destabilize the critic by introducing rapidly changing group assignments, while updating too infrequently (e.g., every 1000 steps) leads to stale cluster statisties and degraded performance.
Consequently, we adopt moderate default frequency of 200 critic updates, which captures distribu- tional shifts in the replay buffer while maintaining stable learning. Importantly, this robustness to update frequency directly supports our efficiency claims: because optimal performance is achieved with intermittent rather than per-step clustering, the clustering cost can be heavily amortized over many critic updates, keeping the overall runtime overhead low (cf. Table 13).
Overall, these analyses show that Ca robust to reasonable choices K, X, and clustering frequency, and that it introduces only modest computational overhead large-scale offline RL benchmarks.
28

---

Published as a conference paper at ICLR 2026
Table Normalized scores on locomotion tasks with full datasets (~ 10 samples).
Task
BC
TD3BC
IQL
DOGE
TSRL
ВPPО
A2PR
CQL
CQL+C*
Hopper-m
52.9
59.3
66.3
98.6
86.7
93.9
100.8
58.5
85.9 (+27.4)
Hopper-mr
18.1
60.9
94.7
76.2
78.7
92.5
101.5
95.0
100.7 (+5.7)
Hopper-me
52.5
98.0
91.5
102.7
95.9
112.8
112.1
105.4
89.4 (-16.0)
Hopper-e
108.0
100.1
99.3
107.4
110.0
113.2
115.0
110.1 (+11.7)
Halfehoetah-m
42.6
48.3
47.4
40.6
48.2
44.0
68.6
41.0
48.5 (+7.5)
Halfcheetah-mm mr
55.2
44.6
44.0
42.8
42.2
41.0
56.6
45.5
44.7 (-0.8)
Halfchectah me
55.2
90.7
86.7
78.7
92.0
92.5
98.3
91.6
91.6
Halfcheetah-e
92.2
82.1
88.9
93.5
94.3
95.3
103.2
95.6
(-2.4)
Walker2d-m
75.3
83.7
78.3
86.8
77.5
83.6
89.7
72.5
(49.3)
Walker2d-mr
26.0
81.8
73.9
87.3
66.1
77.6
94.4
77.2
90.9 (+13.7)
Walker2d-me
107.5
110.1
109.6
110.4
106.4
113.1
114.7
108.8
108.6 108.6 (-0.2)
Walker2d-e
107.9
108.2
109.7
107.3
110.2
113.8
114.2
110.3
109.7 (-0.6)
Average
66.1
80.7
82.5
86.1
84.0
89.4
97.4
83.3
87.9 (+4.6)
Table 10: Sensitivity of CQL+C* to the number of clusters on 10k-sample D4RL tasks.
Method
K
Ant
Halfchectah
Hopper
Walker2d
AntMaze
Pen
Kitchen
CQL
21.0
39.7
43.2
26.0
21.1
1.8
0.6
CQL+C
59.3
41.3
68.3
51.2
55.1
29.8
11.7
CQL+C*
2
66.5
48.2
75.8
82.6
62,9
36.0
17.1
CQL+C*
70.6
47.5
90.8
87.8
51.3
19.5
CQL+C*
5
71.3
46.0
85.3
96.3
68.7
58.2
20.9
CQL+C*
69.6
46.4
58.0
19.3
CQL+C
10
68.5
46.3
78.6
90.6
68.5
52.4
18.0
CQL+C*
44.9
73.3
87.3
62.8
54.0
19.8
Table 11: Average number of effective clusters for CQL+C* when initialized with K = 20.
Samples
Ant
Halfcheetah
Hopper
Walker2d
AntMaze
Pen
Kitchen
10k
6
4
4
5
5
6
samples
3
2
2
4
3
Table 12: Sensitivity of CQL+C* to the penalty weight X.
Task name
= 0.0
À 0.05
À
0.1
A
0.3
X 0.5
1.0
Halfcheetah-m
30.1
38.9
44.1
46.3
45.0
40.3
Hopper-m
55.6
68.6
75.0
69.2
69.6
62.8
Walker2d-m
55.3
57.9
61.1
65.9
65.2
63.7
Halfcheetah-mr
33.6
32.3
37.8
35.6
43.1
36.6
Hopper-mr
25.0
35.5
49.6
51.0
45.9
50.2
Walker2d-mr
20.1
35.3
38.1
44.7
39.0
Halfcheetah-me
52.2
46.9
27.4
27.0
Hopper-me
65.0
74.2
81.3
70.8
67.6
63.5
Walker2d-me
63.6
95.9
96.3
89.9
87.9
79.2
Table 13: Proportion of total training time spent in gradient- space clustering for CQL+C4.
Samples
Ant
HalfCheetah
Hopper
Walker2d
AntMaze
Pen
Door
Kitchen
103
1.0%
0.9%
1.2%
1.2%
2.9%
1.7%
1.5%
2.0%
~ 106
18.7%
15.3%
14.4%
13.8%
22.6%
17.9%
15.1%
16.4%
29

---

Published as a conference paper at ICLR 2026
Halftheetaah-medeee expert
Hopper-medium
Weller2d-medimm d-mediu--eeeeett
Ant Ant-mediuum-eee
238
220
100
100
Norrallzd scIP
-Q-
Cuars
40
28
20
100
Hopper expert
Wbllker2d expert
Ant: expert
Normal score
303
Halfcheelah-medium
Nomma I2oe Score
60
40
40
50
100
10
50
300
30
50
100
309
topremedium-reppyy
Wakker2dmmedu---ellll
Art-medium--replay
222
100
100
Norralized score
48
40-
20
20-
100
10
300
Halfcheatah-full fullaatttaa-ull-l-paa
Hopper.fulrreplay
Walker26-Ull-eppla
Arts/ul--rppay
Nomnalaoo score
100
100
48-
40
Samples
Samples
Samples
Figure 9: Comparison on Locomotion tasks over different data size.
Hanmmarclored
Pan-cloned
Doonslored
Relocatecclared
Nommirent Score
40
20
0
20
Dataset Size
(1)
TDS
COL
Figure 10: Comparison of Adroit tasks over different data sizes.
30

---

Published as a conference paper at ICLR 2026
1.2
1.0
1.5
1.0
0.8
5.8
0.8
0.4
22
0.2
5.2
0.2
0.2
150
0.0
15
160
5.0
0.0
25
150
Happem ved aec.ernrggg regeet
1.0
Hepp open
3.0
Heperrecber
1.0
Happe 1aprr48mmd
Ooveterns
84-
0.4
9.4
0.6
0.4
0.2
922
0.2
1.0
Thes of Contura
0.8
9.6
0.6
0.8
64-
0.4
9.4
0.6
0.4
Artopet
of
0.5-
0.6-
0.2
0,2
0.2
15
30
0.0
15
10
15
45
62
15
Ezcch
Ezoch
Spoch
Epoch
Ssech
Figure 11: Trace of covariance during training on MuJoCo locomotion. Tasks HalfCheetah Hopper Walker2d Ant. Dataset regimes: medium-expert, expert, medium, medium-replay. full-replay. The x-axis epoch. The axis is the trace of the empirical cross covariance, lower better. Methods include Ours (CQL+C*), LN (CQL+LN), CQL, and DR3 (CQL+DR3).
Table 14: Sensitivity of CQL+C4 to clustering frequency (number of updates between clustering).
Task name
Freq. 20
Freq. 50
Freq.
100
Freq.
200
Freq.
500
Freq. 1000
Halfcheetah-m
39.1
36.7
41.0
46.3
36.6
22.7
Hopper-m
66.9
65.0
64.7
69.2
60.8
61.4
Walker2d-m
58.9
64.1
66.5
65.9
57.8
46.5
Halfcheetah-mr
29.4
29.1
38.5
43.1
37.2
36.9
Hopper-mr
51.8
54.0
54.6
45,9
36.1
37.7
Walker2d-me mr
55.4
35.9
22.3
Halfchectah-ne
27.6
55.0
63.6
46.9
44.3
23,9
Hopper-me
45.5
79.7
87.7
81.3
73.0
61.6
Walker2d-me
69.3
74.1
91.7
96.3
75.1
74.9
31

---

Published as a conference paper at ICLR 2026
Dens CLaosttc
Dete KURSOM
Figure 12: Clustering visualization for CQL+04.
TOEE SUNN MAALTE
Figure 13: Clustering visualization for TD3BC+C4
32

---

Published as a conference paper at ICLR 2026
Table 15: Datasets overview.
Task Name
Samples
# Traj
random
expert
Maze2D
maze2d-open
106
22148
0.01
20.7
maze2d-umaze
106
12460
23.9
161.9
maze2d-medium
10
11889
13.1
277.4
maze2d-large
106
16727
6.7
274.0
AntMaze
antmaze-umaze
106
10154
0.0
1.0
autmaze-umaze-divers
106
1035
0.0
1.0
antmaze-medium-P
106
10767
0.0
1.0
autmazz-meedium-ddveeeee
106
2958
1.0
antmaze-large-plas
106
13516
0.0
1.0
antmaze-large-diverss
106
7188
0.0
1.0
antmaze-ultra-play
106
10536
0.0
1.0
antmaze-ultra-diverse
106
6076
0.0
1.0
Gym-MuloCo
hopper-expert
106
1028
-00.3
3234.3
hopper-medium
106
2187
-20.3
3234.3
hopper-meedium-rpllay
402000
2041
-20.3
3234.3
hopper-meediumeepper
1999906
3214
-20.3
3234.3
hopper-fuul-replay
106
3515
-20.3
3234.3
halfcccetah--exertt
106
1000
-280.2
12135.0
halfcheetah-mmeduum
106
1000
-280.2
12135.0
halfcheetab-mediiu--rraaay
202000
202
-280.2
12135.0
halfchhetaab-meddum---ppe
106
2000
-280.2
12135.0
rell
106
1000
-280.2
12135.0
walker2d-expert
106
1001
1.6
4592.3
walker2d-medium
106
1191
1.6
4592.3
walker2dd-mediiu--rplay
302000
1093
1.6
4592.3
walker20l-mecium-eeyy expert
1999995
2191
1.6
4592.3
-aul
106
1888
1.6
4592.3
ant-expert
106
1035
-325.6
3879.7
ant-medium
106
1203
-325.6
3879.7
ant-medium-replay
302000
485
-325.6
3879.7
ant-medium-expprt
1999946
2237
-325.6
3879.7
ant-full-replay
106
1319
-325.6
3879.7
Adroit
pen-human
5000
25
96.3
3076.8
pen-cloned
105
3755
96.3
3076.8
pen-expert
499206
5000
96.3
3076.8
hammer-hume
11310
25
-274.9
12794.1
hammer-cloned
106
3606
-274.9
12794.
hammer-expert
106
5000
-274.9
12794.
door-human
6729
25
-56.5
2880.6
door-cloned
106
4358
-56.5
2880.6
door-expert
106
5000
-56.5
2880.6
relocate-human
9942
25
-6.4
4233,9
relocate-clone cloned
106
3758
-6.4
4233.9
relocate-expert
106
5000
-6.4
4233.9
33

---

Published as a conference paper at ICLR 2026
F USE OF LARGE LANGUAGE MODELS
In this manuscript, the authors make limited use of large language models, including OpenAl's GPT-5 and DeepSeek, mainly for proofreading, language polishing, improving textual clarity, and reviewing the logical flow of arguments. In addition, Anthropic's Claude is used as an auxiliary tool to assist with coding tasks during the research process. The use of these models is restricted supportive roles in writing and technical implementation; they do not contribute to research design, data collection, data analysis, or the formulation of scientific claims. All substantive content and conclusions remain entirely the responsibility of the authors.
34


Reviews:
[{"review_id": "X1ScE54uiJ", "reviewer": "Reviewer_Ykhx", "summary": "This paper investigates a fundamental source of instability in temporal-difference (TD) learning for offline reinforcement learning (RL). The authors argue that the primary cause is the cross-time covariance between feature gradients of consecutive transitions, which acts as a harmful implicit regularizer. This covariance term biases TD updates and leads to critic collapse, particularly under weak data coverage or out-of-distribution (OOD) regions. To address this, the paper proposes C4 (Clustered Cross-Covariance Control), a simple yet theoretically grounded method that clusters gradient pairs *$(g',g)$* and applies covariance regularization within each cluster. This localized control effectively suppresses cross-covariance while preserving the TD objective’s lower bound. Empirical results on D4RL MuJoCo benchmarks demonstrate substantial performance gains—especially in low-data regimes—showing improved stability and data efficiency.\n\nAlthough this issue may appear specific to small datasets, the underlying mechanism suggests that it will become even more significant as environments grow larger or more complex. When the environment’s state or action space expands while the dataset size remains fixed, coverage sparsity worsens, amplifying the harmful covariance effects. Hence, the insight and methodology presented here are expected to remain increasingly relevant as future offline RL benchmarks evolve toward more realistic and high-dimensional settings.", "questions": "**Cluster sensitivity**:\n\nHow sensitive is performance to the number of clusters *$K$*? Did you observe degradation or instability for smaller/larger *$K$* values? Could an adaptive mechanism (e.g., based on gradient variance or cluster entropy) help automate cluster selection?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper impressively connects a theoretically grounded understanding of TD learning instability with a practical, well-performing algorithm. The C4 framework stands out for translating a nontrivial analytical insight into empirically robust improvements, showing both conceptual clarity and engineering maturity.\n\n**Strong theoretical foundation**: The paper begins from a clear theoretical diagnosis of a subtle but critical problem in temporal-difference (TD) learning — the emergence of harmful cross-time covariance under distributional shift and limited data. The authors derive this effect formally and distinguish it from other, beneficial forms of implicit regularization. This analytical depth provides a precise and original explanation for instability in offline RL, rather than relying on empirical speculation.\n\n**Elegant link from theory to practice**: One of the most compelling aspects is how the theoretical insight leads directly to a concrete and empirically effective solution. The transition from analyzing TD covariance to designing the Clustered Cross-Covariance Control (C4) algorithm shows an excellent bridge between mathematical reasoning and algorithmic implementation.\n\n**Consistent and plentiful empirical validation and ablation studies**: The experiments on D4RL MuJoCo benchmarks demonstrate that the theory translates into tangible performance benefits: smoother learning curves and more than 30% improvement in return under small data and OOD-emphasized splits. Even though the authors do not deeply explore the optimal number of clusters, the empirical results with *$K=5$* show that the method works reliably across tasks without extensive tuning—indicating good robustness in practice. \n\nApart from that, the paper includes extensive ablation studies on different factors such as dataset size, which reinforce the soundness and robustness of the proposed approach.", "weaknesses": "I have only on weakness to point out.\n\n**Sensitivity to the number of clusters K**:\n\nThe proposed method critically depends on the number of clusters used to partition the gradient feature space. If *$K$* is too small, dissimilar gradient modes are mixed, leaving between-cluster covariance unremoved; if *$K$* is too large, covariance estimation becomes noisy due to small sample counts per cluster. However, the paper fixes *$K$* (typically 5) for all experiments without analyzing sensitivity or scalability. It remains unclear how robust C4 is to this choice or how *$K$* should scale with data size or environment complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AZMLU6Yd1p", "reviewer": "Reviewer_99jv", "summary": "The paper analyzes the standard temporal difference (TD) loss used for training in offline reinforcement learning (RL), and identifies a cross-covariance term that acts as a harmful implicit regularizer. Based on this observation, the paper proposes an algorithm $C^4$ (Clustered Cross-Covariance Control) to mitigate the cross-covariance term through buffer partitioning and an explicit cross-covariance penalty term. Theoretical results demonstrate that these modifications are compatible with standard offline RL policy update methods, and experimental results demonstrate that these modifications lead to significant improvement across a range of offline RL benchmarks when small datasets are considered.", "questions": "**[Q1]** What is the intuition behind why the cross-covariance term present in the standard TD loss is harmful for performance?\n\n**[Q2]** Is there a performance tradeoff on large datasets in order to achieve strong performance on small datasets, or does the method automatically adjust to the large dataset scenario? Please include experimental results on the standard dataset sizes to help understand this. Experiments on the default size of benchmark datasets would also make it easier to compare the results in this paper to other results available in the literature.\n\n**[Q3]** I would recommend reorganizing Sections 5 and 6 to clearly highlight the main contributions upfront, and only including results and discussions that are most critical for providing theoretical support and intuition of the approach. This would provide additional space to include a more meaningful experiments section in the main paper, rather than deferring most of the results to the appendix.\n\n**[Q4]** Additional clarifications on implementation and results:\n- Does the fact that the TD loss is only optimized with respect to the current Q function (gradients are stopped for the TD target) impact the analysis and conclusions?\n- Why does the proposed regularization term in (13) contain two components?\n- What is the justification for approximating the gradient w.r.t. the input $x$ as the network’s penultimate layer (line 1091)? My understanding is this was done in DR3 (Kumar, 2022) to approximate the gradient w.r.t. the Q function network parameters, not the gradient w.r.t. the Q function input.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**[S1]** The paper presents novel and interesting analysis that identifies a cross-covariance term that acts as a harmful implicit regularizer in the standard TD loss function, and proposes algorithmic modifications to address this issue in offline RL.\n\n**[S2]** Theoretical results are provided that justify the algorithmic design choices and demonstrate their impact on other components of standard offline RL algorithms such as the policy improvement update.\n\n**[S3]** Extensive experimental results across several offline RL benchmarks demonstrate the strong performance of $C^4$ on small datasets compared to several baseline methods (unfortunately, most of this analysis is deferred to the appendix).", "weaknesses": "**[W1]** Experiments primarily focus on small datasets and do not include results on the benchmarks for the standard dataset size (e.g., Figure 6 considers a max dataset size that is 10% of the full dataset in most cases), so it is difficult to understand if there are performance tradeoffs on large datasets in order to achieve robust performance on small datasets.\n\n**[W2]** The organization and presentation of the work could be improved. In particular, the organization of Sections 5 and 6 required the reader to extract why results are important and how they relate to Algorithm 1. Some of the notation makes it difficult to follow the results (e.g., $C$ and $c$ are used in multiple ways throughout Section 5). The length of Sections 5 and 6 also led to a very brief experiments section in the main paper, even though the authors have done rather extensive experimental analysis that would be valuable to include instead of deferring it to the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zDXaa3wVSV", "reviewer": "Reviewer_FJyE", "summary": "This paper studies instability in offline RL under distributional shift, particularly in small-data regimes.\nThe authors identify a previously under-analyzed phenomenon: the TD loss induces a harmful cross-time gradient covariance that biases optimization in OOD regions.\nTo mitigate this, the paper introduces Clustered Cross-Covariance Control (C4), composed of two mechanisms:\n(1) Clustered buffer sampling — partitions the replay buffer by gradient features and samples mini-batches from single clusters to remove between-cluster covariance;\n(2) Gradient-based corrective penalty — penalizes the Frobenius norm of within-cluster cross-covariance to reduce harmful coupling.\n\nTheoretical analysis shows that C4 preserves the lower-bound property of conservative objectives (e.g., CQL) and bounds TD instability, while experiments on reduced-size D4RL tasks (10k samples) show improved performance and stability compared to CQL, IQL, and DOGE.", "questions": "Why were recent offline RL algorithms such as A2PR (arXiv:2405.19909) and On-Policy Regularized IQL excluded?\n\nHow would C4 perform against them under the same 10k-sample setting?\n\nHow often are gradient clusters updated during training? Does dynamic re-clustering significantly affect performance or computational cost?\n\nCan the proposed gradient-space clustering scale to high-dimensional visual inputs or large replay buffers (e.g., 1M samples)? What is the per-iteration complexity?\n\nHow should λ and K be chosen in practice? Is there a heuristic based on dataset size or OOD ratio?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality: The work offers a novel theoretical perspective linking TD error variance to cross-time gradient covariance. \n\nTechnical quality: The derivations are mathematically sound and clear. Theorem 1 decomposes the TD variance into beneficial (variance) and harmful (covariance) components, while Theorem 2 proves that single-cluster sampling removes between-cluster interference.\nThe proposed loss (Eq. 12–14) is principled and compatible with existing algorithms.\n\nClarity: The paper is clearly structured, easy to follow, and supported by illustrative figures (e.g., Fig. 2). Algorithm 1 is concise and implementable.\n\nSignificance: The idea of controlling TD-induced covariance is relevant for improving stability and generalization in offline RL with small datasets. The method is plug-and-play and conceptually simple, which could make it attractive for integration into broader offline RL pipelines.", "weaknesses": "Outdated Baselines: The empirical evaluation does not include recent state-of-the-art methods, particularly A2PR (Adaptive Advantage-Guided Policy Regularization, arXiv:2405.19909), which achieves significantly higher returns on the same D4RL tasks.\nWithout comparing to such strong baselines, the claimed 30% improvement is not convincing. **Please answer the concern in the response.**\n\nExperimental Scope and Depth: Experiments are limited to low-dimensional MuJoCo tasks with 10k samples; no results on real-world robot control or discrete domains (e.g., Atari, Adroit).\n\nThe main text lacks ablation studies (e.g., number of clusters K, λ penalty weight, clustering frequency).\n\nThe claimed “computational efficiency” of clustering is not quantified.\n\nTheoretical Assumptions: The first-order Taylor expansion (Eq. 2) assumes smooth linearity in feature space; this may not hold in deep nonlinear critics, yet this limitation is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FSKxiKw4CJ", "reviewer": "Reviewer_ko2m", "summary": "The paper analyzes the TD (temporal-difference) regression objective by decomposing the TD second moment and shows that directly minimizing the squared TD loss induces a harmful cross-time covariance between current and next state–action gradients. Building on this diagnosis, the authors propose two complementary fixes: (i) clustered, single-cluster minibatch sampling over stacked gradient pairs, which removes between-cluster covariance; and (ii) an explicit gradient-based corrective penalty that offsets within-cluster covariance during critic updates. They argue these modifications are “plug-and-play” for common offline RL pipelines and preserve policy-constraint objectives. Experiments on reduced-data D4RL settings report consistent gains (often sizeable) over several baselines, particularly under scarce data and OOD-emphasized splits. The paper also provides proofs and a training recipe.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear theoretical diagnosis of a TD-specific failure mode. The decomposition explicitly isolates the cross-time covariance term unique to TD learning, clarifying why standard TD minimization can be destabilizing under weak coverage. \n\n\nThe clustered sampling + Frobenius-norm penalty is well-motivated, easy to integrate, and provably controls the harmful covariance while keeping the optimization intent intact.\n\nOn reduced-size D4RL locomotion (10k samples), the method substantially outperforms listed baselines; Table 1 and the accompanying discussion highlight strong average gains and improved stability.", "weaknesses": "While the theory shows TD minimization tends to raise the cross-term, the paper could better quantify how much covariance growth translates into return degradation and in what regimes (e.g., through controlled interventions that vary covariance at fixed value error).\n\n\nDespite some Maze2D/AntMaze numbers in the appendix, there is no focused evaluation on challenging sparse-reward benchmarks like AntMaze with standard sparse settings and strong recent baselines, which would directly test the claimed robustness under weak signal\n\nBaseline set skews older in places. Several comparisons rely on classical offline RL baselines; inclusion of more recent small-data/OOD-aware methods would make the empirical case more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)