Calibration Enhanced Decision Maker: Towards Trustworthy Sequential Decision-Making with Large Sequence Models
Authors: Haoyuan Sun
Bo Xia
Yifu Luo
Tiantian Zhang
Xueqian Wang
Venue: Accepted by TMLR ()

Abstract: Offline deep reinforcement learning (offline DRL) has attracted considerable attention across various domains due to its ability to learn effective policies without direct environmental interaction. Although highly effective, the trustworthiness of agent concerns the community. Offline DRL can be categorized into three principal paradigms: model-based algorithms, model-free algorithms, and trajectory optimization. While extant research predominantly concentrates on calibration enhancement of model-based and model-free algorithms, calibration of trajectory optimization remains a rather rare topic. In this paper, we introduce the concept of Expected Agent Calibration Error (EACE), a novel metric designed to assess agent calibration. Furthermore, we rigorously prove its theoretical relationship to the state-action marginal distribution distance. Subsequently, we introduce the Calibration Enhanced Decision Maker (CEDM), which employs a binning executor to process feature distribution histograms as input for the large sequence model, thereby minimizing the state-action marginal distribution distance and enhancing the agent's calibration. A series of in-depth case studies of CEDM are carried out, with application on Decision Transformer, Decision ConvFormer, and Decision Mamba. Empirical results substantiate the robustness of EACE and demonstrate the effectiveness of CEDM in enhancing agent calibration, thereby offering valuable insights for future research on trustworthy sequential decision-making.

Published in Transactions on Machine Learning Research (01/2026)
Calibration Enhanced Decision Maker: Towards Trustworthy Sequential Decision-Making with Large Sequence Models
Haoyuan Sun
ww..hygS@mmal..ssiihuuue......
Tsinghua Shenzhen International Groduate School, Tsinghua Universily
Bo Xia
niab21 @fozmail com
College of Electronic Science and Technology, National University of Defense Technology
Yifu Luo
Tsinghua Shenzhen International Groduate School, Tsinghua University
Tiantian Zhang"
zhang. wtw.st..innnnnaa.eeeeee. edu.cn
Tsinghua Shenzhen International Groduate School, Tsinghua University
Xueqian Wang
wwng.zzy@ss.tsiingguue edu.crr
Tsinghua Shenzhen International Gruduate School,
Tsinghua University
Reviewed OpenReview: https: https://opeerreviau openreview. net/ formfo id= b6WcmPEb48
Abstract
Offline deep reinforcement learning (offline DRL) has attracted considerable attention across various domains due to its ability to learn effective policies without direct environmental in- tcraction. Although highly effective, the trustworthiness of agent concerns the community. Ofline DRL can be categorized into three principal paradigms: model-based algorithms,, model-free algorithms, and trajectory optimization. While extant research predominantly concentrates calibration enhancement of model-based and model-free algorithms, cali- bration of trajectory optimization remains rather topic. In this paper, introduce the concept of Expected Agent Calibration Error (EACE), novel metric designed to assess agent calibration. Furthermore, we rigorously prove its theoretical relationship to the state- action marginal distribution distance. Subsequently, we introduce the Calibration Enhanced Decision Maker (CEDM), which employs a binning executor to process feature distribu- tion histograms as input for the large sequence model, thereby minimizing the state-action marginal distribution distance and enhancing the agent's calibration. A series of in-depth case studies of CEDM are carried out, with application Decision Transformer, Decision ConmFrrm and Decision Mamba. Empirical results substantiate the robustness of FACE and demonstrate the effectiveness of CEDM in enhancing agent calibration, thereby offering valuable insights for future research on trustworthy sequential decision-maan makinn..
Introduction
"A good decision is based on knowledge and not on numbers."
Plato
"Corresponding authors: Bo Xia, Tiantian Zhang

---

Published in Transactions on Machine Learning Research (01/2026)
Deep reinforcement learning, employing the trial-and-crror mechanisms to learn and optimize specific reward signals, has emerged as powerful learning strategy executing autonomous data collection across various domains, such as robotic motion control (Singh et al. 2022 Tang et al. 2024), autonomous driving (Kiran ict al. 2021 Zhao et al. 2024), large language model tunint (Ouyang et al.) 2022: Guo et al. 2025 and so on. Furthermore, is the aspiration of many roboticists to program a robot with a Lask in the evening and return the following morning to discover capable of effectively solving the task. Then, offline deep reinforcement learning (offline DRL) has garnered considerable interest in the community (Gürtler et al., 2023), primarily on account of its capacity to acquire effective strategies without the need for interaction with the environment. Such capability is particularly advantageous in scenarios where the cost or risk associated with real-time environmental engagement is substantial. According to survey compiled by Prudencio et al. it can be classified into three primary aspects: (1) model-based algoritluss (Luo el al., 2024 /. (2) model-free algorithms (Swazinna et al. 2022 (3) trajectory optimizations (Janner et al. 2021 Hu et al, 2024
Espocially, for the third aspect, trajectory optimization approaches reformulate reinforcement learning prob- lems as sequence modeling tasks, leading to improved performance and generalization. A large proportion of sateectory-optmmizatee
approaches adopt high-capacity large sequence models, which effectively address several long-standing challenges in conventional reinforcement learning methods, including sample efficiency, credit assignment, and partial observability. Within this paradigm, Decision Transformer (Chen et al., 2021 stands as a pioneering contribution. As shown in Figure [d), Decision Transformer processes a sequence that is composed of states, past actions, and desired returns within an autoregressive framework, and sub- sequently infers actions aimed at achieving the specified return, with the Transformer functioning as the policy model. Furthermore, following the advancements in large sequence models, adopting more powerful backbone architectures for sequential docision-making has also drawn incrcasing attention, exemplified by Decision ConvPormer (Kim et al. 2024) and Decision Mamba (Ola 2024).
Meanwhile, trustworthiness of the agent has also standed as paramount concern within the community (Yu et al. 2025). It is widely acknowledged that miscalibrated trust can lead to misuse of agents (Wei et al. 2025). Therefore, to foster trustworthy and reliable agents; it imperalive to emphasize their calibration. In the aforementioned taxonomy of offiine DRL, the first two categories already have corresponding works that consider calibration. Calibrated model-based DRL (Malik et al. 2019) demonstrates method for endowing agents with a calibrated world model that accurately represents true uncertainty and enhances planning in high-stakes scenarios. Calibrated Q-learning (Nakamoto et al. 2024) learns conservative value functions (Kumar et al. 2020) that are calibrated with respect to behavior policy. However, current research on trajectory optimization has never incorporated calibration considerations, which poses a risk of misuse in future real-world applications ions. In this study, we primarily concentrate on offline DRL algorithms with large sequence models (such as Transformer (Vaswani] 2017), MetaFormer (Yu al.] 2022) and Mamba (Gu &
Dao ).24)). Therefore, we raise the following question:
Major Question
Can we enhance calibration of offline DRL algorithms that utilize large sequence models to further bolster their trustworthiness in future real-world applications?
To address this question, it is essential to first define a more accurate metric for agent calibration. Drawing inspiration from the definition of Expected Calibration Error (Guo et al. 2017) in prior model calibration literature, we introduce the Expected Agent Calibration Error (EACE) as a rigorous metric to quantitatively characterize the calibration properties of agents. Nonetheless, implementing measurement in practical as- sessments is challenging due to safety concerns. Therefore, we further theorctically prove the relationship between the EACE and the state-action marginal distribution distance: differences in the EACE are bounded by the total variation distance between the state-action marginal distributions of two agents; and under mild assumptions, this can be extended to the Wasserstein-1 distance. In other words, smaller Wasserstein-1 distance between the state-action marginal distributions of two agents indicates a smaller difference in their EACE values, implying superior agent calibration.
Based on these foreshadowings, we introduce the Calibration Enhanced Decision Maker (CEDM). Unlike traditional approaches that rely solely on the "number" of returns-to-go as input for the large sequence model, CEDM incorporates a binning executor to process feature distribution histograms, thereby providing
2

---

Published in Transactions on Machine Learning Research (01/2026)
av the cullrated syent I sheald diep right swe octer might nor highest opur vake durrisation durinnc and ExcE
p-1
Causol
on the wnnallionns ejest. I oon has mine famard moves er collding pediettinn otfere the light petort el noturn
Binning Executor
(a) Beyond return: Calibration also counts
Feature
Feature
Outtur
Torget
D-7
6-1
Action Accuracy
Gup
Ralett
(c) Calibration Enhanced Decision Maker
Probobility
Causal Transformer
Confidence Expected Agent Callbration Error
Feature (X-Velocity)
State-Action Marginal Distribution bistance
6-7
b-7
b-7
(b) Calibration Measurement
(d) Decision Transformer
Figure 1: (a) Motivation: In real-world applications, especially in high-risk scenarios, trustworthiness of agent's actions also of great importance. (b) Measurement: We define the Expected Agent Calibration Error; furthermore, for practical applications, establish its relationship with the state-action marginal distribution distance. (c) Methodology: We introduce the Calibration Enhanced Decision Maker paradigm, which employs a binning executor to process the distribution histograms as inputs to the large sequence model. (d) Prototype: Decision Transformer has been a groundbreaking approach in offline deep reinforcement learning that treats decision-making as a sequence modeling problem.
richer "knowledge" for the agent. Intuitively, CEDM enables the agent under training to more accurately model the state-action marginal distribution represented the offline dataset. By minimizing the distance of the distributions, it consequently decreases the differences in the EACE, thereby enhancing agent calibration. Such paradigm partially inspired by the previous studies on state-marginal matching (Lee et al. 2019; Ghasemipour et al., (020):: they conduct a qualitative assessment of distribution matching outcomes within designated state dimensions (e.g., zy-positions); furthermore, it can be extended to the velocity (e.g., velocity) to obtain richer "knowledge". Categorical Decision Transformer (Furuta et al. 2022) has shown the potential of similar paradigm within the Transformer architecture and has explained its effectiveness from the perspective of hindsight. information matching. Moreover, we demonstrate the applicability of such paradigm to a wider range of large sequence models, which reduces the distance between state-action marginal distributions and thereby enhances the agent calibration.
Furthermore, we perform case studies by applying the CEDM paradigm to Decision Transformer (Chen al. 2021), Decision ConvFormer (Kim et al. 2024), and Decision Mamba (Otal 2024 Based on a comprehensive review of these practical implementations, several conclusions can be drawn. Firstly, the CEDM paradigm effectively decreases the W distance between the state-action marginal distributions of the target rollout and the agent's behavior. Secondly, performance of the agent (e.g., final return) can be improved by utilizing the CEDM paradigm. Finally, application of the CEDM paradigm to ultra-long context sequence models (e.g., Mamba) may encounter limitations inherent the offline dataset itself, providing insights for future efforts deploy more advanced sequence models in decision- decision-making tasks.
In summary, our contributions are follows:
We introduce the agent calibration metric: Expected Agent Calibration Error. Furthermore, facilitate practical implementation, we theoretically establish the relationship between the EACE and the state-action marginal distribution distance.
3

---

Published in Transactions on Machine Learning Research (01/2026)
We introduce the Calibration Enhanced Decision Maker paradigm, which utilizes a binning executor to process distribution histograms as inputs to the large sequence model. This paradigm effectively reduces the state-action marginal distribution distance, thereby enhancing the agent's is calibration. To the best of our knowledge, this is the first study on the enhancement of agent calibration in the field of offline DRL with large sequence models.
We perform case studies, applying the CEDM paradigm to Decision Transformer, Decision Con- vFormer, and Decision Mamba. Experimental results demonstrate the effectiveness of CEDM and provide insights into its future applications in more advanced sequence models.
2 Related Work
2.1 Offline DRL with Large Sequence Models
Decision Transformer (Chen et al. 2021 represents a pioneering stride in reinforcement learning, reframing the decision-making as sequence modeling using the Transformer architecture to predict optimal actions by processing trajectories of state, action, and relurns-to-go. Such an innovative training paradigm has sparked a burgeoning trend leveraging advanced large sequence models to address decision-making tasks. Deci- sion ConvFormer (Kim et al. (2024) utilizes the architectural framework of MetaFormer @Yu et al. 2022).
employing localized convolutional filtering as the token mixer, adeptly capturing the intrinsic local depen-
dencies within the trajectories. Decision S4 (David et al. 2023 employs RNN-like S4 (Gu et al. 2022
architecture for inference, while Decision Mamba. (Otal 2024 directly substitutes the attention mechanism
with Mamba (Gu & Dao 2024). Furthermore, this paradigm has also been increasingly applied in various domains, including robotic motion control (Gajewski et al., 2024), autonomous driving (Li et al.) 2025 and large language model fine-tuning (Hu et al. (202)). Despite the highly promising results of these endeavors,
none of them have addressed the issue of agent calibration, thereby posing a significant risk to their reliability and trustwsrthiness in further real-world applications.
2.2 Model Calibration
Model calibration refers to the alignment between a model's predicted probabilitics and the true likelihood of the events (Guo el al.] (017). For example, if the model assigns a 70% confidence level to a prediction, we would expect the predicted outcome to occur in approximately 70% of instances in practical applica- tion. Unlike accuracy, calibration emphasizes trustworthiness of confidence, tackling challenges such overconfidence. This guarantees probabilistic reliability, which is of paramount importance in high-cost and high-stakes domains. The concept of calibration has found purchase in a multitude of domains, such autonomous driving (Fu et al. 2012), computer vision (Guo et al. 2017), natural language processing (Zhao et al. 2021 and so on.
In the domain of offline DRL, rescarchers have also made significant strides enhancing calibration. In the survey (Prudencio al.) 2023 offline DRL can be classified into three primary aspects: model-based algo- rithms, model-free algorithms, and trajectory optimizations. Calibrated model-based reinforcement learning, detailed in (Malik et al. 2019), provides agents with refined world model that accurately reflects in- herent uncertainty, thereby improving planning in high-stakes secnarios. Calibrated Q-lcarning (Cal-QL) [Nakamolo el al. (2024) leverages a modified CQL framework to ensure learned policy Q-values remain above a lower bound and below upper bounds for suboptimal reference policies, thereby calibrating the reasonable scale. They have shown promising applications of calibration both model- based and model-free algorithms; however, there still lack of effort to enhance calibration in trajectory optimizations. Tn this study, we aim to provide insights into bridging this gap.
lcarned Q-values
4

---

Published in Transactions on Machine Learning Research (01/2026)
3 Preliminaries
3.1 Offline DRL with Trajectory Optimization
In the scenario of offline DRL, our target is to identify an optimal policy x (a[ss,, maximizing the expected cunulative reward or=]_ The stale 8 and the action a are dictated by the behavior policy; while the next state 8t+1 and the reward ry are determined by the transition dynamics function. This enables the construction of an offline trajectory dataset
{(st, dts $141, Tt)i), where presents the timestep in the episode. Trajectory optimization methods recast the goal as minimizing the action reconstruction loss due to the lack of interaction with the environment:
L ET
L = EI CMSE/CE((;;a))))
(1)
where a; = K R2 k+i ay- -x)-=1 donates the reconstruction action: K context length of the sequence model; R Fr .r the returns-to-go. During the testing phase, a target returns-to- go is manually predefined to encapsulate the intended performance benchmarks. Trajectories from last K timesteps are input into the model, which subsequently generates action for the current timestep. Based on this, the environment provides the next state and reward, after which the returns-to-go is updated accordingly. Then, these new elements are also input into the model.
3.2 Expected Calibration Error
Expected Calibration Error (Nacini et al. 2015) has been common metric for describing the calibration performance of models, which quantifies the discrepancy between the model's predicted confidence and its actual accuracy. A model is deemed as "perfectly calibrated" when there a precise alignment between its confidence levels and its empirical performance:
Vp [0, 1), p)
p)
where Y is the output of network prediction and P is its associated confidence (i.e. probability correctness). Furthermore, the Expected Calibration Error (F.CF) is defined discrepancy between the empirical accuracy and the corresponding confidence level:
ECE Ep [P rIP= =p) =
(2)
4 Methodology
In this scction, we aim to address the challenge of enhancing agent calibration of offline DRL algorithms that utilize large sequence models, thereby improving their trustworthiness. Firstly, we introduce a metric called Expected Agent Calibration Error describe the agent calibration. Furthermore, we theoretically prove that the difference between Expected Agent Calibration Error is bounded by the total variation distance and the Wasserstein-1 distance (under mild assumptions) between the learned policy and the hidden policy. Building upon them, we introduce Calibration Enhanced Decision Maker, which further leverages the distribution histograms as input for the large sequence model, ultimately yielding the calibration enhanced agent.
4.1 Expected Agent Calibration Error
To address the aforementioned challenge, the first step is to formulate precise description of agent calibra- tion. Inspired by the concopt of "perfect calibration" " and incorporating the inherent mechanisms of offline DRL, we formally define the concept of "perfect policy calibration" in Definition []
Definition 1. Let (II,) be metric space, where II denotes the space of all stochastic policies, and d II II Rzo is policy distance metric. Defining U = 1 [dis (T1, σ2) I 6), where I(-) is the the indicator function, U=1 1 denotes that the distance between policy лı and policy is smaller than the threshold 8. Hence, for the agent (model), perfect calibration (with full confidence) can be expressed as:
Vp E [0, P (U 112 = p) p-
(3)
5

---

Published in Transactions on Machine Learning Research (01/2026)
Intuitively, "perfect policy calibration" refers the scenario: two policies are sufficiently similar with the probability p; and exactly a proportion of practical predictions are indeed sufficiently similar.
Building upon this and drawing inspiration from ECE (Guo et al. 2017 we define the Expected Agent Calibration Error in Definition 2 Intuitively, it describes the average discrepancy between the agent's true probability p of "U=1" and the stated probability P when asserts that "I am P confident. that U=1":
Definition 2 (Expected Agent Calibration Error (EACE)). Define the miscalibration of the agent by com- puting the expectation of calibration error predicted confidence P:
EACE = Ep [[P(U= 1|. = p) pl
(4)
Nevertheless, we find that although EACE is theoretically complete and grounded, it is computationally impractical in real-world applications. Notably, in high-risk scenarios, particularly when the agent's true probability p is relatively low, such measurement become exceedingly costly, the estimation demands a substantial number of rollouts under the policy.
Furthermore, as illustrated in Figure [d), in the context of offline DRL with large sequence models, the input tokens generally consist of states, actions, and their associated returns-to-go. Therefore, the subsequent action is essentially determined by the knowledge embedded in the preceding states and the preceding actions. This further inspires us to delve deeper into the relationship between the EACE and the state-action marginal distribution distance. Ultimately, we elaborate on this relationship in Theorem I Specifically, the differences in FACE are bounded by the total variation distance between the state-action marginal distributions of two agents, as formally expressed in Theorem !
Theorem 1. Suppose Te, (als) and nez (als) are separately two policies of two agents, pa (s and p (s, a) are the stte-acctio marginal distributions of the two agents, then:
EACE(TT,) - EACE(TT)) S E [4 TV..VV(("" (s, (s,
(5)
where TV(-) the total variation distance.
Proof.
Before presenting a dotailed proof of Theorem 1 we first introduce two lemmas. The derivation of
Lemma primarily utilizes vector transformations and the absolute value inequality; and the derivation of primarily relies on the Hölder's inequality. Their detailed proofs are provided in Appendix A
Lemma 2
Lemma 1. Suppose a, b, are three vectors, then we have that:
(la - bl, 2b) - {la - cl, 2c) S (b+ c la b) +aa cl, c))
(6)
Lemma 2. and are two vectors, and each of their terms is nonnegative. Then, get:
(a. < [|b|)1,||b|lo))
where ||a]] represents the Ln-norm of vector and |ibloc represents the Lox norm of vector b. We now proceed to the proof of Theorem
For any state-action marginal distribution, we set that p p= p p= p a) without loss of generality. As - (s, = pT* (s) (a s); and given policy n, the action is sampled with the policy Te(als). Thus, we have:
EACE(*0) mT) =Ep [P(U = pl
- P(U = 1 (s)
A a) - (x,aa
(8)
- E pT IPU
= (s, - A= a) - -p- pm a)
We then abbreviate the conditional distribution P(U =
A a) as p Then, we have:
FACE E E(lp" n0 (s,a) n0 (8, a))
(9)

---

Published in Transactions on Machine Learning Research (01/2026)
where 10a -pth (s. a)l and p=a a) are vectors and {pp= - (s, a)l, of 100 pa (7o(8))) a)l and pa (s, a). Further, let's ' compare the EACE of two agents 01 02 E e:
(s) represents the inner product
EACE( (AC E <110~~pp0 - p (s,a)), pT (s) {pp p=m (s) (st
(10)
According Lemma we can get:
EACE((T)) EACE((TO)
(s, (s)) p= (s,a)
-s, - -s. 10=
(11)
p (s.
2
According to the Lemma 2 we can have that:
EACE(TT.) ) - EACL((T.))
1 (8, a) + ~ (s + 100 (s, pm0d + ()) p (s,
<
rr (s, - -000 (s.
2
pals (8, + pma (8, + (0 a) - ( (8, 10 - (s,
< (s,0 (s,=
2
(12)
Setting that:
1a0, (8, (8, + p (8) (8, pr (s. + (s, p= (s,
m(To To., m)
2
For the sake that each term of the distributions pTa, (s, a), nT (s, a) and p= a) are bounded in [0, 1]; hence, it is evident that m( Toz, T) $ 2. Therefore, we can get that:
EACE(*),) EACE(TO.) < E [2 ||e= (s, a), p= (s.
(13)
= E [ TV (pTx (s, a), -T (s, a))l
which completes the proof.
Moreover, in practical applications, the Wasserstein-1 distance is often considered more applicable than the total variation distance, as it can better capture geometric properties of the distributions. Therefore, we extend the total variation distance in Theorem [ to the Wasserstein-1 distance according to the inequality between them. Further defining the two policies as the anticipated ideal policy and the current policy, we can obtain the Proposition | It also worth noting that our analysis restricted the discrete-only setting, under the structural assumption that the finite discrete support admits strictly positive inner point distance in the underlying metric space.
Proposition 1. Suppose T(als) is the anticipated ideal policy, Ta(a)s) is the policy of the agent, p= a.) is the state-action marginal distribution of the anticipated ideal agent and p (s, a) is the state-action marginal distribution of the current agent, then:
EACE(TT) EACE(T) SE
(s,
(14)
dmii
where W1 (s.e(s,)) a), p²t (s) represents the Wasserstein-1 distance between (5, and p²((sa);; let pT0 (s) E and 1² (8, a) EM, selling Ω = supp(p) U supp(u), dmin = pn -1 - -='l
Proof. According to the work [Panarctos & Zemel 2019), we introduce the following Lemma establishes relationship between the TV distance and the Wasserstein-1 distance.
which
Lemma 3 (Panaretos & ZZmel 2019)). ). Setting X, Y are finitely discrete random variables, and they are bounded; X E a and Y E v, Ω supp(u) U supp(v); dmin =nffzyest 1|x yll then have that:
1 .W,
TV (XXY)) dmin
(15)

---

Published in Transactions on Machine Learning Research (01/2026)
Herein, let's consider the proof of Proposition !
According to Theorem we have that:
EACE(TT) - EACE(T) S E (C (1,,
According to Lemma setting that (s, 14 (8,0)) v, Ω supp(u) U supp(u), dman pnf -1 p"Il
derive that:
EACE(TT)) EACE(T)
S [4 (s,
4
(16)
E
W1 (p (s). ()
dmin
which completes the proof.
In summary, under mild assumptions, the EACE discrepancy between the anticipated ideal policy and the current policy is bounded by the Wasserstein-1 distance between their state-action marginal distributions. Hence, we consider minimizing the upper bound, specifically the Wasserstein-1 distance, to reduce their FACE discrepancy.
4.2 Calibration Enhanced Decision Maker
Based on the comprehensive theoretical analysis presented above, we introduce the Calibration Enhanced Decision Maker (CEDM) paradigm, which employs categorical state-action marginal distribution histograms as the input token to the large sequence model. As illustrated in Figure [ c), a binning executor utilized transform the feature distribution into a categorical approximation of the original continuous distributtin. Previous research has also explored the binning paradigm to enhance input information density: building upon prior work on the mtage-marginal matching (Lee et al. 2019 Ghasemipour et al. 2020), the Categorical Decision Transformer (Furuta et al. 2022) demonstrates the potential of binning distributions, particularly reward distributions, within the Transformer architecture. Moreover, as demonstrated by our theoretical analysis, the CEDM paradigm can effectively enhance the agent's calibration and is applicable to all large sequence models instead of Transformer only.
Furthermore, as Plato stated, "a good decision based on knowledge and not on numbers". The CEDM paradigm equips the agent with richer "knowledge", rather than relying solely on the "number" of returns- to-go. In practical applications, agents to the specific requirements of each task. For example, in autonomous driving, the planar velocity (z- velocity) should be prioritized for consideration; in obstacle-crossing robots, the vertical velocity (e-velocity) is also crucial; in robotic arms, the joint angular velocity is of primary importance; and in more elaborate applications, such as missile tracking or space robots, the acceleration should also be considered. The introduced CEDM paradigm enables further selection of the appropriate state-action marginal distribution for binning, thereby allowing customization meet specific practical requirements.
be endowed with different dimensions of "knowledge" according
5 Case Studies
This section presents case studies applying the CEDM paradigm to Decision Transformer Chen et al. 2021 Decision ConvFormer (Kim et al.) 2024), and Decision Mamba (Ota 2024), with reports on final returns and the Wasserstein-1 distances between state-action marginal distributions to validate its effectiveness and enhancement on agent calibration. These three architectures represent three distinct levels of dependence on historical information within the decision-making process. We focus on the following three questions:
1) Can the CEDM paradigm effectively reduce Wassorstein-1 distances, thereby enhance the agent's calibration?
2) Can the CEDM paradigm improve the agent's return?
3) Would the improvements of the CEDM paradigm be affected by different degrees of dependence on historical information during the decision-making process?
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: WT Wi_Di Dis and return comparison of Calibration Enhanced Decision Transformer (CEDT) and Deci-
sion Transformer (DT). Bolded text indicates that the results of CEDT are superior those of DT.
CEDT
DT
W1 Dis
Med
Med-Exp
Med-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HC
0.19=0.01
0.68=0.29
0.31+
0.39
0.20+0.02
1.06=0.34
0.32=0011
0.53
Hp
0.14+0.02
0.13-0.03
0.26++0118
0.18
1.19
0.14+00.02
0.13=0.03
0.70+#0.8
0.32
1.39
Wk
0.31
0.08=0.02
1.66+0.80
0.62
0.43+0.11
0.08=0.04
1.12=0.62
0.54
Return
HC 12.77:+0.26 86.6442..0
3..59-0.25
56.33
12.88+0..29 81.05+22.22
39.09+0039
55.34
Hp 59.85=0.52 78.88=16.25 62.11+17.99 19.6
70.96
200.27
57.80+2.41 90.16=11.58 20.34+5.51
56.10
182.11
Wk 73.7744.28 107.70+0.70 97.48+17.88.
72.98
72.87+6.22 107.49+1.11 31.66-14.22
70.67
Table 2:
Dis and return comparison of Calibration Enhanced Decision ConyFormer (CEDC) and De-
cision ConvFormer (DC). Bolded text indicates that the results of CEDC are superior to those DC.
CEDC
DC
W1 Dis
Med
Med-Exp
Med-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HC
0.20=0.05
0.46=0.25
0.09
0.25
0.22+0.06
0.50=0.29
0.24=0006
0.32
Hp
0.02
0.12
0.28.1.19
0.18
0.69
0.15
0.15
0.93
0.41
1.16
0.27+0.10 Wk
0.07=0.01
0.45++0.32
0.26
0.46+00133
0.23-0.31
0.60+0.22
0.43
Return
HC 42.92.1.15 88.5111.78
41.1710.53
57.53
42.8110.22
88.16:2..53
39.6910.21
56.89
Hp
62.67+1.62 75.0648.86
75.14+13.39
70.96
204.75
59.1343.228 76.57=19.08 30.81+21.95
55.51
189.08
Wk 76.0912.56 108.05.10.76 14.64119.10
76.26
74.87.12.83 104.3617.38 50.81
76.68
5.1
Experimental Setup
In our case studios, experiments are conducted with the OpenAI Gym environment (Brckkman et al., 2016
MuJoCo tasks (Todorov et al. 2012) which serve as the most typical beuchrnark for reinforcement learning. Specifically, we utilize the task of Hhleeahht (HC), Hopper (Hp), Walker2d (Wk). Tn terms of datasets, we utilize medium (Med), predium-expers (Med-Exp), and medium-replay (Med-Rep) datasets that are from the D4RL benchmark (Fu et al.) M.re More details provided in Appendix C To ensure consistency, we utilize the distributions of v-velocity throughout our study. Furthermore, to facilitate unified representation of returns across tasks, we perform normalization on the scores:
Score - Random Score
Return = 100 X Expert Score Random Score'
(17)
where the terms "Random Score" and "Expert Score" are adopted from the D4RL benchmark. When selecting the binning executor, we utilize the same binning method as that used in the Categorical Decision Transformer (Furuta al. 2022). For cach dataset in every task, we identify the best trajectory; and the results are reported as the "mean + standard deviation" over five random seeds. Additionally, we calculate the Average performance for each task across the three corresponding datasets, well the Total results
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table
W1 Dis and return comparison of Calibration Enhanced Decision Mamba (CEDMamba) and
Decision Mamba (DMamba). Bolded text indicates that results of CEDMamba are superior to those of DMamba.
CEDMamba
DMamba
W1 Dis
Med
Med-Exp
Med-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HC
0.21110.05
0.8911027
0.11.10.04
0.40
0.18110.04
0.66.10.13
0.31
0.38
Hp
0.15+0.03
0.10+#.02
0.39+0.23
0.21
0.86
0.14+00.03
0.15+00.2
0.65++0.18
0.31
1.33
Wk
0.30=10.55
0.06:10.02
0.39:10.15
0.25
0.50
0.88+0.25
0.55+0.27
0.64
Return
HC
42.87-+0.23
85.59+-2.05
40.86+00.2
56.44
42.97++0.12
86.73+1.299
39.26+00.54
56.32
Hp
59.73#2.28
90.28+9.008
78.19+2.12
76.07
212.94
69.39:t7.25 97.47=10.47
41.03+
69.30
196.72
76.69+2.40 108.05+1.00 56.56+5.12
80.43
70.4747.21 57.20-55.48
55. 55.63+7.22
71.10
Medium
Medium-Exppe Expert
Medium-Replie
Medium
Medium-Expart Expert
Medium
Medium-Ruie RediummRReplay
Medium Experrt Replay
HC
Hp
Wk
Toeget
Rallut
Torget
Rollou
Reloot
Calibration Enhanced Decision Transformer
Calibration Enhanced Decision ConvFormer
Calibration Enhanced Decision Mamba
Figure Visualizations of the distribution histograms for the highest-return trajectory of CEDT (left), CEDC (middlee) and CEDMamba (right), presented separately for the three tasks across the three datasets.
of the average values. The training hyperparameters and implementation details are consistent with those reported in the original paper.
5.2 Comparison Results
In Table [ Table and Table 3. we present the comparative results of Wasscrstein-1 distance and return by applying the CEDM paradigmn to Decision Transformer, Decision ConvFormer, and Decision Mamba, respectively. Moreover, the three architectures represent distinct levels of dependence on historical infor- mation within the decision-making process: Decision ConvFormer primarily utilizes local information, while Decision Mamba emphasizes global information.
Firstly, we analyze the impact of the CEDM paradigm on the agents' Wasserstein-1 distances to address the first question. From an overall perspective, specifically in comparison to the "Total" values, is indicated that the Calibration Enhanced Decision Makers effectively reduce the Wasserstein-1 distance across all architectural configurations relative to the baseline methods. To be specifie, Table damoossdeeosstraaee that CEDT
10

---

Published in Transactions on Machine Learning Research (01/2026)
CEDC
CEDMamba
Figure 3: Example rollout (blue) visualizations of CEDC (left) and CEDMamba (right) on the HallCheetah medium-expert dataset for the same target (violct).
achieves reduction of 14.4% in the total Wasserstein-1 distance, decreasing it from 1.39 to 1.19; Table 2 indicates that CEDC attains a reduction of 40.5% in the total Wasscrstein-1 distance, lowering it from 1.16 to 0.69; Table 3 reveals that CEDMamba accomplishes a reduction of 35.3% in the total Wasserstein- 1 distance, reducing from 1.33 to 0.86. Consequently, the CEDM paradigm is empirically demonstrated to effectively reduce the Wasserstein-1 distance between state-action marginal distributions, thereby enhancing the agent calibration. Furthermore, to facilitate intuitive comprehension, we present representative visualizations of the distribution histograms for them, as depicted in Figure 2 Overall, the CEDM paradigm exhibits good alignment with the distributions from the original dataset, thereby highlighting efficacy of CEDM paradigm.
Secondly, let's proceed to answer the second question. Although primary motivation of the CEDM paradigm is to enhance agent calibration, it is also crucial to evaluate its potential impact on the final return. By comparing the "Total" return values, it is evidont that the Calibration Enhanced Decision Makers consistently oulperform the baseline methods, yielding superior final returns across all archilectural configurations. To be specific, Table demonstrates that CEDT achieves an improvement of 10.0% in the total final return, elevating it from 182.11 to 200.27; Table 2 indicates that CEDC attains an improvement of 8.3% in the total final return, enhancing it from 189.08 to 204.75; Table 3 reveals that CEDMamba accomplishes improvement of 8.2% in the total final return, raising it from 196.72 to 212.94. Notably, observe that the improvements achieved by the CEDM paradigm are particularly pronounced on the Hopper- Pooper--Mddum-RRpllee datasct: specifically, CEDT exhibits an increase from 20.34 to 62.11; CEDC improves from 30.84 to 75.14; and CEDMamba demonstrates a rise from 41.03 to 78.19. According to the study (Ajay et al. 2023 the dataset exhibit propensity for diminished smoothness.
action trajectorics in the Dopper-Mediun-Repue Therefore, the CEDM paradigm still capable of maintaining remarkable efficacy even when implemented datasets of relatively lower quality.
Finally, we advance to answer the third question. Within the results, it is observed that the CEDM paradigm occasionally exhibits instances of failure, particularly in relation to CEDMamba. We argue that this phe- nomenon attributable to the inherent ultra-long context capabilities of Mamba. In Figure 3 we present an example that demonstrates the differences rollouts between CEDC and CEDMamba for the same target. For the CEDC, its convolutional architecture endows it with an enhanced capacity to capture local informa- tion, thereby exhibiting superior fitting performance within high-probability regions. However, CEDMamba predominantly emphasizes global information, thereby making its modeling of high-probability regions par- ticularly vulnerable to biases stemming from the datasct. This further highlights two critical considerations for future real-world applications: firstly, the selection of an appropriate backbone should be tailored to the specific dataset and task; secondly, the quality of offline datasets plays a crucial role in shaping effectiveness of the CEDM paradigm.
6 Conclusion, Limitation, and Future Work
In this study, we focus on the crucial challenge of enhancing calibration in offline DRL algorithms that lever- age large sequence models, with the aim of enhancing their trustworthiness in future real-world applications.
11

---

Published in Transactions on Machine Learning Research (01/2026)
We begin by introducing the Expected Agent Calibration Error, metric that deseribes the agent calibra- tion. Furthermore, to facilitate practical assessments, we rigorously establish its theoretical relationship with the state-action marginal distribution distance: the EACE discrepancy between the anticipated ideal policy and the current policy is bounded by the Wasscrstein-1 distance between the agent's 's state-action marginal distributions under the two policies. Subsequently, we introduce the Calibration Enhanced Decision Maker paradigm, which leverages a binning executor process distribution histograms as inputs to the large se- quence model. This paradigm aims at reducing the discrepancy between state-action marginal distributions, thereby enhancing agent's 's calibration. Moreover, we perform case studies, applying the CEDM paradigm to Decision Transformer, Decision ConvFormer, and Decision Mamba. Experimental results highlight effec- tiveness of the CEDM paradigm and shed light on its future application in more advanced large sequence models.
Limitation. Although the CEDM paradigm has demonstrated significant advancements trustworthiness and has exhibited promising performance across simulation environments, its effectiveness in real-world applications necessitates further investigation and exploration.
Future Work. Going forward, efforts in the future could primarily focus on the following three points. Firstly, the applications of the CEDM paradigm to real-world scenarios warrant further exploration, par- ticularly in the domains of robotic motion control and autonomous driving. Secondly, it is also worthwhile to further investigate the applications of diverse physical quantities beyond velocity, such as joint angular velocity and acceleration, within the CEDM paradigm. Lastly, future research should continue to investigate approaches in the domain of trajectory optimization that could enhance agent calibration.
Broader Impact Statement
stated in Section 1.2 the proposed CEDM paradigm allows flexible customization meet specific practical requirements. Therefore, it holds great potential for real-world applications and is expected to be further extended to areas including robotics, autonomous driving, and aerospace engineering.
Acknowledgments
This work was supported by the Natural Science Foundation of Shenzhen (No. JCY12022305011116000000 No. 120024081311200000000 the Natural Science Foundation of Guangdong Province (No. 2024A1515010003) and Cross-disciplinary Fund for Research and Innovation (No. JC2024002) of Tsinghua SIGS.
References
Anurag Ajay, Yilun Du, Abhi Gupla, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.013: 01540, 2016.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:1500
2021.
Shmuel Bar David, Itamar Zimerman, Eliya Nachmani, and Lior Wolf. Decision s4: Efficient sequence-based RL via state spaces layers. In The Eleventh International Conference on Learning Representations, 2023.
Bo Fang, Yuxin Song, Qianggiang Wu, Haoyuan Sun, Wenhao Wu, and Antoni B Chan. Viss-rl: Self- supervised reinforcement video reasoning. arXiv preprint orXix:2511.1305, 2025.
Justin Fu, Aviral Kumar, Ofr Nachum, George Tucker, and Sergey Levine. D4rl: Datasots for deep data- driven reinforcement learning. arXio preprint orXiv:2004.07219, 2020.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Xianping Fu, Xiao Guan, Eli Peli, Hongbo Liu, and Gang Luo. Automatic calibration method for driver's head orientation in natural driving environment. TEEE Transactions on Intelligent Transportation Systems, 14(1):303 312, 2012.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for oflline hindsight information matching. In International Conference on Learning Representations, 2022.
Paul Gajewski, Dominik Zurek, Marcin Pietroñ, and Kamil Faber. Solving multi-gos robotic tasks with
decision transformer. arXiv preprint arXiv:24 10. 10.06377, 2024.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference robot learning, pp. 1259-1277. PMLR, 2020.
Albort Gu and Tri Dao. Mamba: Lincar-time sequence modcling with selective state spaces. In First
Conference on Language Modeling, 2024.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. International conference on machine learning, pp. 1321 1330. PMLR, 2017.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseck-r1 incentivizes reasoning in Ilms through reinforcement learning. Nature, 645(8081))6333 608, 2025.
Nico Gürtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier, Manuel Wüthrich, Stefan Bauer, Bernhard Schölkopf, and Georg Martius. Benchmarking offine reinforcement learning on real-robot hardware. In The Eleventh International Conference on Learning Representations, 2023.
Jian Hu, Li Tao, June Yang, and Chandler Zhou. Aligning language models with oflline learning from human feedback. arXiv preprint arXiw:2308. 12050, 2023.
Shengehao Hu, Li Shen, Ya Zhang, Yixin Chen, and Dacheng Tao. On transforming reinforcement learning with transformers: The development trajectory. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-122 -1876, 2021.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neurol information processing systems, 30, 2017.
Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making. In The Twelfth International Conference on Learning Representations, 2024.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE transactions on intelligent transportation systems, (36)):900-4922 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce-
ment learning. Advances in Newral Information Processing Systems, 33:1179-1191, 1191, 2020.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, al. A tutorial on energy-based learning. Predicting structured data 1(0), 2006.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching, arviv preprinl arXiv: 100::1906.05274 2019.
Zenan Li, Fan Nie, Qiao Sun, Fang Da, and Hang Zhao. Uncertainty-awann decision transformer for stochastic driving environments. In Conference Robot Learning, pp. 364-386. PMI.R, 2025.
Fan-Ming Luo, Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, and Yang Yu. A survey on model-based reinforcement learning- Science China Information Sciences, 67(2):121101, 2024.
Yifu Luo, Penghui Du, Bo Li, Sinan Du, Tiantian Zhang, Yongzhe Chang, Kai Wu, Kun Gai, and Xueqian Wang. Sample by step, optimize by chunk: Chunk-level grpo for text-to-image generation. arXiv preprint WrXiv:2510.21556 2025a.
Yifu Luo, Xinhao Hu, Keyu Fan, Haoyuan Sun, Zeyu Chen, Bo Xia, Tiantian Zhang, Yongzhe Chang, and Xueqian Wang. Reinforcement learning meets masked generative models: Mask-grpo for lext-lo-image generation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b.
Xufei Lv, Kehai Chen, Haoyuan Sun, Xueleng Bai, Min Zhang, and Houde Liu. The hidden link wetween rlhf and contrastive learning. arXiv preprint 2ttd:2500.225111 2025.
Ali Malik, Volodymyr Kuleshov, Jiaming Song, Danny Nemer, Harlan Seymour, and Stefano Ermon. Cali- brated model-based deep reinforcement learning. In International Conference on Machine Learning, pp. 4314-4323. PMLR, 2019.
Mahdi pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference artificial intelligence, volume 29, 2015.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline pre-training for efficient online fine-tuning, Advances in Newral Information Processing Systems, 36, 2024.
Toshihiro Ota. Decision mamba: Reinforcement learning via sequence modeling with selective state spaces. arXiv preprint Xix: 1198.9999 19925, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human leedback. Advances in neural information processing systems, 35:27730 27744, 2022.
Victor M. Panaretos and Yoav Zemel. Statistical aspects of wasscrstein distances. Annual Re- view of Slatistics and Ils Application, 6(1):405-433 March 2019. ISSN 2326-831X. doi: 10.1146/
anmurev-statisties--5e 080718- 104938.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline rein- forcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Kai Qin, Jiaqi Wu, Jianxiang He, Haoyuan Sun, Yifei Zhao, Bin Liang, Yongzhe Chang, Tiantian Zhang, and Houde Liu. Distribution preference optimization: A fine-grained perspective for 1lm unlearning. arXiv preprint XrX::22510.0077 2025.
Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications: comprehensive survey. Artificial Intelligence Review, 55(2):945 900,9990,00 2022.
Haoyuan Sun, Bin Liang, Bo Xia, Jiaqi Wu, Yifei Zhao, Kai Qin, Yongzhe Chang, and Xueqian Wang. Diffuoson-rannbooaa:: Improvements integrated preference alignment for diffusion- based text-to-image gen- cration. Transactions on Machine Learning Research, 2025a.
Haoyuan Sun, Jiaqi WU, Bo Xia, Yifu Luo, Yifei Zhao, Kai Qin, Xufei Lv, Tiantian Zhang, Yongzhe Chang, and Xueqian Wang. Reinforcement fino-tuning powers reasoning capability of multimodal large language models. Xiviv preprint or Xim:2500 18536, 2025b.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Haoyuan Sun, Bo Xia, Yongzhe Chang, and Xueqian Wang. Generalizing alignment paradigm of text- to-image generation with preferences through F-iveegence minimization. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 39, pp. 27644 27652, 2025c.
Haoyuan Sun, Bo Xia, Yifei Zhao, Yongzhe Chang, and Xueqian Wang. Identical human preference alignment paradigm for text-to-image models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025d.
Haoyuan Sun, Bo Xia, Yifci Zhao, Yongzhe Chang, and Xueqian Wang. Positive enhanced preference alignment for lext-to-image models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). TEFE, 2025c.
Phillip Swazinna, Steffen Udluft, Daniel Hein, and Thomas Runkler. Comparing model- free and model-based algorithms for offline reinforcement learning. IFAC- PPAC-PaperSOnIine, 35(15):19-25) 2022.
Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martin-Martin, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. Anruaal Review of Control, Robotics, and Autonomous Systems, 8, 2024.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RS] international conference on intelligent robots and systems, pp.
IEEE, 2012.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Zeming Wei, Tianlin Li, Xiaojun Jia, Yihao Zhang, Yang Liu, and Meng Sun. Position: Agent-specific trustworthiness risk as a research priority. In ICML 2025 Workshop on Reliable and Responsible Foundation Models, 2025.
Bo Xia, Haoyuan Sun, Bo Yuan, Zhiheng Li, Bin Liang, and Xueqian Wang. A delay-robust method for enhanced real-time reinforcement learning, Newral Networks, 181:106769, 2025.
Jun Yin, Pengyu Zeng, Haoyuan Sun, Yuqin Dai, Han Zheng, Miao Zhang, Yachao Zhang, and Shuai Lu. Floorplan-llama: Aligning architects' feedback and domain knowledge in architectural floor plan generation. In Proceedings of the 63rd Annual Meeting of the Associalion for Compulational Linguistics (Volume 1: Long Papers), pp. 6640 6662, 2025.
Miao Yu, Fanci Meng, Xinyun Zhou, Shilong Wang, Junyuan Mao, Linsey Pan, Tianlong Chen, Kun Wang, Xinfeng Li, Yongfeng Zhang, al. A survey trustworthy Ilm agents: Threats and countermeasures. In Proceedings of the 31st ACM SIGKDD Conference Knowledge Discovery and Dala Mining V. 2, pp. 6216 6226, 2025.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10819 10829, 2022.
Rui Zhao, Yun Li, Yuze Fan, Fei Gao, Manabu Tsukada, and Zhenhai Gao. A survey on recent advancements in autonomous driving using deep reinforcement learning: Applications, challenges, and solutions. IEEE Transactions on Intelligent Transportation Systems, 2024.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few- shot performance of language models. In International conference on machine learning, pp. 12697-12706.
PMLR, 2021.
15

---

Published in Transactions on Machine Learning Research (01/2026)
A Detailed Proof of Lemma 1 and Lemma 2
Lemma 1. Suppose a, b, c are three vectors, then we have that:
(|a - b/, 2b)
(la ala-cc,20) 2l,20) s (b + c la bi + 1/ cl)
(18)
Proof. Consider the following inequation:
(la cl c 1 cl) (|a b), b |c (c3bb) b])
(19)
According the nature of the absolute value, it is obvious that b- c- |b cl S 0 and that c- 6 6) 2 0, which shows the constancy of Equation (19) is obvious.
Transform Equation
19h, we can get that:
(la c), b) + (1a b),b b 1b <l) (1a c/.ct 1b - c) + (c, la b))
(20)
Based on the absolute value inequality, we can get:
la - - bb el s la cl:
cl + 1b
d N la bl
(21)
Henco, we have that:
(la - bl - - cl,b) + {1a- bl,b - 1b <|> s {/a - cl.b) (Ia bj.b - bb c)
$ (Ia cl (-- + (c. la b))
(22)
S {Ia c/. -- d) + (c. la + 16 d)
Combine some items of the same kind:
(la bl cl,8) b) + {/a- bl, b -cc)) s {|a cl + cl + (1a c 1b (b-c)
(23)
Thus, we have that:
2 I|a bl,b) 2 {|a- c) S (b,1b- d) (c, c)) (/a 61, 1b c) {1a c), 1b -0- c)
(24)
Then, we have that:
- 2b) - (1a 2l, s (b +b+ la bl b/+ + c c))
(25)
which completes the proof.
For the proof of Lemma 2, we rely on the following fact, which also denoted as the Hölder's inequality.
Fact 1 (Hölder's inequality). Set p 1,1/p 1/g 1, if a1 a2...a, and by by..b, is nonnegative, then have:
6
(26)
Lemma 2. and are two vectors, and each of their terms is nonnegative. Then, we get:
(a {[a]) [bbl)=))
(27)
where ||a|| represents the -oom of vector and |ibl represents the noom of vector b.
Proof. Setting p as oo and q 1, then according to the Equation 26 (Hölder's inequality), we can have:
a,b. |6llo ||a||
(28)
Thus, have that:
(a b) < (la]),, [|b])-))
(29)
which completes the proof.
16

---

Published in Transactions on Machine Learning Research (01/2026)
B Additional Visualizations of the Distribution Histograms
In this section, provide more visualizations of the distribution histograms of four additional trajectories. In each figure, each row represents results on the specific dataset: medium as Med; medium-expert as Med-Exp; medium-replay as Med-Rep.
CEDT: Figure (HalfChectah), Figure (Hopper), Figure 6 (Walkordd).
Med
Med
Exp
Med
Rep
Target
Rollout
Figure 4: Within the CEDT, for the HalfChootah task, additional visualizations distribution histograms across three datasets.
Med
Med
Exp
Med
Rep
Target
Rollout
Figure Within the CEDT, for the Hopper task, additional visualizations of distribution histograms across three datasets.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Med
Med
Exp
Med
Rep
Target
Rollout
Figure 6: Within the CEDT, for the Walker2d task, additional visualizations of distribution histograms across three datasets.
CEDC:
Figure
(HalCCheetah), Figure (Hopper), Figure 9 (Waleerd).
Med
Med
Exp
Med
Rep
Target
Rollout
Figure 7: Within the CEDC, for the HalfCheetah task, additional visualizations of distribution histograms across three datasets.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Med
Med
Exp
Med
Rep
Target
Rollout
Figure 8: Within the CEDC, for the Hopper task, additional visualizations of distribution histograms across three datasets.
Med
Med
Exp
Med
Rep
Target
Rollout
Figure 9: Within the CEDC, for the Walker2d task, additional visualizations of distribution histograms across three datasets.
19

---

Published in Transactions on Machine Learning Research (01/2026)
CEDMamba: Figure 10 (HalfCheetah), Figure
11
(Hopper), Figure
12
(Walkerdd).
Med
Med
Exp
Med
Rep
Target
Rollout
Figure 10: Within the CEDMamba, for the HalfCheetah task, additional visualizations of distribution histograms across three datasets.
Med
Med
Exp
Med
Rep
Target
Rollout
Figure 11: Within the CEDMamba, for the Hopper task, additional visualizations of distribution his- tograms across three datasets.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Med
Med
Exp
Med
Rep
Target
Rollout
Figure 12: Within the CEDMamba, for the Walker2d task, additional visualizations of distribution histograms across three datasets.
C Details of Case Studies
C.1 Dataset and Benchmark.
HalfCheetah
Hopper
Walker2d
In our case studies, we focus on three specific tasks the MuJoCo environment: HalfCheetah, Hopper, and Walker2d.
Figure 13:
The MuJoCo environment tasks (HalfCheetah, Hopper, Walker2d) (Tooorov et al. 2012 have been part of the D4RL benchmark (Fu et al.) (2020), encompassing a series of continuous locomotion tasks characterized by dense reward signals. Specifically, the three tasks utilized our case studies are defined as follows:
HalfCheetah: The HalfCheetah is a two-dimensional robot consisting of nine interconnected body segments linked by eight joints, including a pair of paws. Its objective is to apply torques to these joints drive the cheetah forward (to the right) as quickly as possible. Additionally, the HallCheetah task has a 17-dimensional state space and 6-dimensional action space.
Hopper:
The Hopper is two-dimensional single-legged robot composed of four primary segments: the torso the top, the thigh in the middle, the leg at the bottom, and single foot on which the entire body rests. Its objective is to produce forward (rightward) locomotion by driving the three
21

---

Published in Transactions on Machine Learning Research (01/2026)
joints linking these segments with appropriately applied torques, enabling it to perform continuous hopping motions. Furthermore, the Hopper task features an 11-dimensional state space and a 3- dimensional action space.
The Walker is a two-dimensional bipedal robot composed of seven principal segments: a single upper torso from which the two legs diverge, pair of thighs positioned below the torso, a pair of legs exlending from the thighs, and two feet. that provide ground support. The task is to make it walk forward (to the right) by applying torque to the six hinges linking these seven segments. Moreover, the Walker2d task has 17-dimensional state space and 6-dimensional action space.
Walker2d:
For each of these, we utilize three different v2 datasets, each representing a different level of data quality: medium, medium-expert, and medium-replay. The medium dataset obtained by first training a policy online with the Soft. Actor-Critic algorithm, carly-stopping the training process, and then collecting one million samples from the resulting partially trained policy. The medium-expert dataset is constructed by mixing equal amounts of expert demonstrations with suboptimal trajectories, where the suboptimal data is generated either from partially trained policy or by unrolling medium-replay dataset is created by recording every sample accumulated in the replay buffer over the course of training, up until the point at which the policy reaches the medium level of performance. Overall, MuJoCo offers an excellent testing ground to investigate how datasets, originating from policies with varying skill levels, influence learning and performance.
policy. And the
C.2 Baselines.
Various domains, from computer vision (Sun et al. 2025a b Luo et al.
2025a b Yin et al. 2025
Fang et al. 2025 to natural language processing (Lv et al. 2025 Qin et al. 2025 increasingly leverage deep reinforcement learning (Xia et al. 2025) as a core paradigm for model training. Within this broad research landscape, trajectory optimization methods have emerged as a particularly promising direction. our study, we adopt Decision Transformer, Decision ConvFormer, and Decision Mamba as representative bascline approaches.
Decision Transformer (DT) frames offline reinforcement learning a sequence modeling problem, and leveraging the Transformer architecture. Instead of optimizing the value functions or the learning policies directly, DT treats the trajectories as sequences composed of stales, actions, and returns-lo-go (the sum of future rewards from each step). The agent is trained to predict the next action, conditioning on the previous states, actions, and desired return of the trajectory. At cach timestep, input into the Transformer consists of a tuple: the current stale, the previous action, and the larget tourrns-to-oo. then autoregressively predicts the next action that should be taken to maximize the likelihood of achieving the return. Such an approach effectively turns policy generation into a sequence generation task, similar to how language models predict the next word in sentence.
Decision ConvFormer (DC) serves as a novel offline reinforcement learning predictor that fundamentally rethinks how sequential dependencies are captured in offline reinforcement learning trajectories. While the traditional DT models offline reinforcement learning as a sequence modeling problem using the Transformer's self-attention mechanism, DT's reliance on attention can be suboptimal for offline RI. tasks. Furthermore, DC replaces the global attention mechanism with local, causal 1D convolutional filters. Specifically, DC employs three independent convolutional mixers for states, actions, and returns, each operating over local temporal window (e.g., 6 timesteps). This design ensures that predictions made using only nearby tokens, effectively capturing the local, Markovian structure of RL data, while also dramatically reducing model complexity and computational cost.
Decision Mamba (DMamba) further replaces the self-attention mechanism in DT with the Mamba GGu & Dao 2024) block. The architecture comprises token-mixing and channel-mixing layers, with the Mamba block acting as the primary token-mixing module. By leveraging the structured state space modeling, DMamba potentially offering greater efficiency and the ability to model longer dependencies.
22

---

Published in Transactions on Machine Learning Research (01/2026)
C.3 Implementation Details.
To ensure the reproducibility of the experiment, have included all codes in the supplementary materials. Herein, we encounter important hyperparameters and implementation details. All experiments are conducted on a single NVIDIA GeForce RTX 3090 graphics card. Hyperparamoters are mainly set with reference to the Categorical Decision Transformer (Furuta et al., 2022). In Table 4 we list detailed hyperparameters of the CEDT. In Table we list detailed hyperparameters of the CEDC. In Table we list. detailed hyperparameters of the CEDMamba.
Table 4: Hyperparameters for the CEDT.
Hyperparameters
Value
Number of Bins
31
Number of Layers
Number of Attention Head
Embedding Dimension
128
Batch Size
64
Context Length
20
Dropout
0.1
Learning Rate
1c4
Grad Norm Clip
0.25
Weight Decay
le-4
Warmup Steps
10000
Activation Function
ReLU
Gamma
1.0
Max Training Iterations
10
Random Scods
0, 1, 2, 3, 4
Table 5: Hyperparameters for the CEDC.
Hyperparameters
Value
Number of Bins
31
Number of Layers
3
Convolution Window Size
6
Embedding Dimension
128
Batch Size
64
Context Length
20
Dropout
0.1
Learning Rate
1e-4
Grad Norm Clip
0.25
Weight Decay
1c-4
Warmup Steps
10000
Activation Function
ReLU
Gamma
1.0
Max Training Iterations
10
Random Seeds
0, 1, 3, 4
23

---

Published in Transactions on Machine Learning Research (01/2026)
Table 6: Hyperparameters for the CEDMamba.
Hyperparameters
Value
Number of Bins
31
Number of Layers
Embedding Dimension
128
Batch Size
64
Context Length
20
Dropout
0.1
Learning Rate
1e-4
Grad Norm Clip
0.25
Weight Decay
le-4
Warmup Steps
10000
Activation Function
ReLU
Gamma
1.0
Max Training Iterations
10
Random Seeds
0. 1, 2, 3,
D Evaluations Grounded in More Metrics
D.1 An implementation of EACE
For a more thorough evaluation, this section introduces a tractable approximation. The EACE is defined as:
EACE = EB [P(UU= = 1 = -pl
(30)
Overall, the primary bottleneck is that the true hidden policy æ inaccessible, and is difficult to ascertain whether the discrepancy between the two policies is smaller than 8. Therefore, approximate solutions are adopted. Firstly, given that the policy æ expressed in terms of its action outputs, estimate it by performing rollouts to sample its behavior. Then, the accuracy is practically assessed according to:
U [[[[@pred agell 8).
(11)
Furthermore, following standard practices in uncertainty estimation (Kendall & Gal 2017 LeCun et al. 2006), map the action prediction error to the confidence score through exponential kernel:
i =
lgtll?).
(32)
Then, predictions are assigned confidence bins By {ilps [bn, bx+1),, and within each bin, the mean confidence score and the corresponding actual accuracy are evaluated by:
1
P
Σ D B
and
accs
U
(33)
B
B
SE
Finally, the computation of EACE is conducted as:
M
B& N
EACE = Σ
1Px acckl.
(34)
A=
where N is the number of samples. In the subsequent implementation, the hyperparameter settings for the
evaluation are configured as: a = 10, = 0.1, and N 10. The experimental results are presented
Table Table g and Table
24

---

Published in Transactions on Machine Learning Research (01/2026)
Table 7: EACE comparison of CEDT and DT. Bolded text indicates that the results of CEDT performs
well better than DT.
CEDT
DT
EACE 4
Med
Med-Exp
Med-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HС
0.2611.02
0.22:10.01
0.1210.01
0.20
0.26:40.01
0.27+0.01
0.13:10.01
0.22
Hp
0.22:10.02
0.1810.01
0.12:10.00
0.17
0.49
0.21110.02
0.1810.01
0.10110.02
0.16
0.50
Wk
0.16110.3
0.1710.02
0.02:10.00
0.12
0.16:10.03
0.16100.2
0.0310.000
0.12
Table 8: EACE comparison of CEDC and DC. Bolded text indicates that the results of CEDC performs
as woll as or better than DC.
CEDC
DC
EACE 4
Med
Med-Exp
Med-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HC
0.12:00.1
0.0310000
0.0310.01
0.06
0.1310.01
0.1010.03
0.0410.01
0.09
Hp
0.04.1001
0.04
0.02.10.01
0.03
0.12
0.07-10.01
0.05.10.01
0.02.10.00
0.05
0.17
Wk
0.0410.01
0.0410.01
0.0011.000
0.03
0.05110.02
0.05
0.00110.00
0.03
Table EACE comparison of CEDMamba and DMamba. Bolded text indicates that results of CEDMamba performs well as or better than DMamba.
CEDMamba
DMamba
EACE 1
Med
Med-Exp
Med-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HC
0.04+0.03
0.0210.01
0.02+0000
0.03
0.08+0001
0.05 0.02+00.01
0.05
Hp
0.03:10.2
0.05110.01
0.02:10.00
0.03
0.07
0.05:10.01
0.0410001
0.03100.00
0.04
0.11
0.01+:0.00
0.02:10.01
0.00+10.00
0.01
0.04+:0.01
0.03+ 0.00++0.00
0.02
D.2 KL Divergence
To further validate the effectiveness of our method, we introduce KL divergence in this section for additional
evaluation. The experimental results are presented in Table 10 Table 11
Table 12
Table 10: KL divergence comparison of CEDT and DT. Bolded text indicates that the results of CEDT performs as well as or better than DT.
CEDT
DT
KL Divergence
Med
Med-Exp
Med-Ren Rep-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HC
0.04+0.01
0.1940.13
0.09+00.01
0.11
0.05++0.01
0.24+00.99
0.166+0.11
0.15
Hp
0.10+0.02
0.10:0.03
0.40+0.38
0.20
0.65
0.14+00.2
0.15+00.08
0.88+00.41
0.39
1.04
Wk
0.1810.06
0.0410001
0.8110.70
0.34
0.33.10.19
0.57.00.68
0.60.10.29
0.50
25

---

Published in Transactions on Machine Learning Research (01/2026)
Table 11: KL divergence comparison of CEDC and DC. Bolded text indicates that the results of CEDC performs well as or better than DC.
CEDC
DC
KL Divergence
Med
Med-Exp
Med-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HC
0.0410.01
0.19.1.27
0.03110001
0.09
0.06:10.02
0.19:10.21
0.14110.09
0.13
Hp
0.1111003
0.0810.03
0.38.10.36
0.19
0.48
0.17:10.02
0.1210.05
0.89110.26
0.39
1.02
0.21+40144
0.04+:.01
0.35++0.13
0.20
0.33++0.11
0.68+.0.00
0.49-100088
0.50
Table 12: KL divergence comparison of CEDMamba and DMamba. Bolded text indicates that results of CEDMamba performs as well as or better than DMamba.
CEDMamba
DMamba
Divergence
Med
Med-Exp
Med-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HC
0.0410.01
0.67.10.30
0.03
0.25
0.05:10.01
0.46:1.222
0.15:10.07
0.22
Hp
0.10:10.03
0.0610.01
0.82:10.69
0.33
0.73
0.11100.03
0.1910.07
0.95110.40
0.42
1.19
Wk
0.19::0.05
0.03:+0.01
0.23+10.06
0.15
0.35++0.17
0.95++0.61
0.36++0.11
0.55
D.3 TV Divergence
Furthermore, we also conduct additional evaluation with the TV divergence. The experimental results are
shown in Table B Table 14 and Table 15
Table 13: TV divergence comparison of CEDT and DT. Bolded text indicates that the results of CEDT performs well as or better than DT.
CEDT
DT
TV Divergence
Med
Med-Exp
Med-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HC
0.1011.02
0.2510.09
0.1310.01
0.16
0.1010.00
0.2910.06
0.16100.07
0.18
Hp
0.18++0.02
0.15+-0.03
0.31+00.1
0.21
0.65
0.19+:0.03
0.19+00.07
0.53++0.13
0.30
0.82
Wk
0.24.1.05
0.10.10.22
0.4910.18
0.28
0.27-1010
0.33
0.42100.33
0.34
26

---

Published in Transactions on Machine Learning Research (01/2026)
Table 14: TV divergence comparison of CEDC and DC. Bolded text indicates that the results of CEDC performs well as or better than DC.
CEDC
DC
TV Divergence
Med
Med-Exp
Med-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HC
0.10.1001
0.19110.11
0.13
0.12
0.13:10.03
0.23
0.15.10.04
0.17
Hp
0.17:0.03
0.15110.3
0.3110.14
0.21
0.51
0.22:10.02
0.1710.04
0.51100.09
0.30
0.82
0.22+40.88
0.09:0.044
0.24+0.10
0.18
0.28++0.05
0.41+00300
0.35++.0.06
0.35
Table 15: TV divergence comparison CEDMamba and DMamba. Bolded text indicates that results of CEDMamba performs as well as or better than DMamba.
CEDMamba
DMamba
TV Divergence
Med
Med-EEp
Med-Rep
Average
Total
Med
Med-Exp
Med-Rep
Average
Total
HC
0.1111.02
0.3910.12
0.0610.02
0.19
0.12:00.02
0.351
0.17100.0
0.21
Hp
0.17:10.02
0.12+0001
0.47::0.18
0.25
0.59
0.18++0.03
0.21+00.1
0.54+:0.12
0.31
0.88
Wk
0.1710.06
0.0810.02
0.2111004
0.15
0.2710006
0.5310.16
0.29110111
0.36
E
Evaluations for Different Number of Bins
To further validate the effectiveness of our proposed approach, this section examines the impact of different bin counts on the resulting W1 Wi_Dis return, and EACE. All experiments were conducted over five random seeds. As the main text already includes the results for the setting with 30 bins, we omit this case in the tables below and present only the results for bin=15, bin=60, and bin=120. The experimental results averaged over five random seeds are presented in Table 16 Table 17 and Table 18 while the corresponding
standard deviations are reported in Table 19 Table 20 and Table 21
Table 16: Mean Evaluation of Different Numbers of Bins CEDT.
HalfCheetah
Hopper
Walker2d
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rep
bin=15 15
0.22
0.76
0.26
0.15
0.18
0.19
0.27
0.09
0.42
W1 Dis
bin=60
0.22
0.74
0.24
0.16
0.11
0.16
0.24
0.09
1.12
bin 120
0.24
0.74
0.28
0.14
0.15
0.21
0.31
0.11
0.55
bin=15 15
42.82
86.65
39.83
56.53
70.38
67.61
76.63
107.57
38.99
Return
bin=60
42.75
86.08
39.76
63.29
91.13
72.11
77.24
107.41
36.10
bin= 120
42.91
86.08
39.95
62.91
67.67
73.35
75.29
107.00
39.42
bin=15
0.27
0.24
0.13
0.20
0.17
0.10
0.15
0.16
0.02
EACE
bin=60
0.28
0.21
0.12
0.22
0.17
0.12
0.15
0.15
0.00
bin 120
0.27
0.22
0.19
0.21
0.16
0.11
0.16
0.15
0.02
27

---

Published in Transactions on Machine Learning Research (01/2026)
Table 17: Mean Evaluation of Different Numbers of Bins on CEDC.
HallCheetah
Hopper
Walker2d
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rep
bin=15
0.24
0.63
0.15
0.11
0.09
0.27
0.27
0.10
0.33
W1 Dis
bin=60
0.22
0.50
0.18
0.16
0.10
0.18
0.18
0.11
0.39
bin= 120
0.23
0.40
0.15
0.13
0.10
0.18
0.18
0.08
0l.0
bin=15
43.07
87.14
40.86
59.98
81.08
67.18
77.02
107.40
45.18
Return
bin=60
42.99
87.97
40.57
62.74
83.75
80.76
76.49
106.95
47.19
bin 120
43.11
88.98
40.49
62.03
84.52
80.41
77.95
107.73
45.44
bin=15 15
0.10
0.03
0.03
0.07
0.04
0.03
0.06
0.04
0.00
FACE
bin=60
0.10
0.05
0.04
0.05
0.05
0.02
0.04
0.04
0.00
bin 120
0.11
0.06
0.03
0.07
0.05
0.03
0.05
0.04
0.00
Table 18: Mean Evaluation of Different Numbers of Bins on CEDMamba.
HalfCheetah
Hopper
Walker2d
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rep Rep
bin=15
0.24
0.78
0.16
0.10
0.12
0.14
0.23
0.12
0.20
W1 Dis
bin=60
0.21
0.76
0.13
0.13
0.13
0.40
0.23
0.09
0.59
bin=120 120
0.21
0.80
0.17
0.15
0.11
0.24
0.18
0.09
0.42
bin=15 15
42.96
86.85
40.73
64.56
79.67
77.83
78.96
106.68
53.49
Return
60
43.08
86.15
40.87
59.43
89.46
82.57
76.68
107.20
47.03
bin 120
43.01
85.97
40.76
60.99
95.48
73.52
77.35
107.28
56.70
bin=15 15
0.03
0.01
0.01
0.03
0.03
0.02
0.01
0.00
EACE
bin=60
0.04
0.01
0.03
0.04
0.04
0.02
0.02
0.01
0.00
bin 120
0.07
0.01
0.03
0.03
0.04
0.02
0.02
0.02
0.00
Table 19: Standard Derivation Evaluation of Different Numbers of Bins on CEDT.
HalfCheetah
Hopper
Walker2d
Med
Med-Fxp
Med-Rep
Med
Med-Fxp
Med-Rep
Med
Med-Exp
Med-Rep
bin=15
0.04
0.16
0.04
0.04
0.06
0.10
0.10
0.04
0.09
WL Dis
bin=60 60
0.05
0.04
0.07
0.03
0.02
0.04
0.10
0.04
1.02
bin= 120
0.03
0.18
0.03
0.02
0.05
0.10
0.04
0.04
0.16
bin=15 5
0.24
1.25
0.17
1.51
14.79
16.91
1.44
0.56
16.55
Return
bin=60
0.10
0.75
0.55
2.72
11.36
13.31
1.97
1.09
15.12
bin=120 120
0.17
1.67
0.20
4.22
15.57
10.51
3.44
0.85
7.50
bin=15
0.02
0.01
0.01
0.02
0.01
0.01
0.02
0.02
0.00
EACE
bin=60
0.04
0.04
0.01
0.03
0.02
0.01
0.02
0.03
0.01
bin 120
0.02
0.02
0.01
0.02
0.01
0.01
0.02
0.02
0.00
28

---

Published in Transactions on Machine Learning Research (01/2026)
Table 20: Standard Derivation Evaluation of Different Numbers of Bins on CEDC.
HallCheetah
Hopper
Walker2d
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rep
bin=15
0.05
0.30
0.05
0.05
0.03
0.22
0.09
0.01
0.15
W1
Dis
bin=60
0.04
0.32
0.07
0.04
0.08
0.07
0.07
0.07
0.13
bin=122
0.06
0.13
0.07
0.02
0.03
0.09
0.03
0.03
0.19
bin=15
0.17
2.12
0.31
3.33
19.64
12.74
5.21
0.89
2.80
Return
bin=60
0.21
2.44
0.36
3.79
5.15
8.55
1.78
1.55
9.92
bin 120
0.20
1.41
0.77
1.67
14.08
12.02
0.99
0.55
7.33
bin=15 15
0.04
0.01
0.01
0.02
0.01
0.00
0.02
0.02
0.00
FACE
bin=60
0.01
0.02
0.02
0.01
0.01
0.00
0.02
0.01
0.00
bin=120
0.03
0.03
0.01
0.02
0.01
0.00
0.02
0.01
0.00
Table 21: Standard Derivation Evaluation of Different Numbers of Bins on CEDMamba.
HalfCheetah
Hopper
Walker2d
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rep Rep
bin=15
0.03
0.27
0.02
0.04
0.05
0.09
0.02
0.04
0.10
W1 Dis
60
0.06
0.16
0.05
0.03
0.05
0.19
0.06
0.03
0.28
bin= 120
0.05
0.22
0.07
0.05
0.02
0.10
0.03
0.02
0.08
bin=15 15
0.24
1.64
0.22
4.23
13.83
11.52
0.40
0.90
10.93
Return
60
0.11
1.52
0.26
3.00
16.37
5.40
2.31
0.85
5.46
bin= 120
0.18
1.91
0.45
3.97
9.71
15.42
3.55
0.48
13.50
bin=15 15
0.02
0.01
0.01
0.02
0.01
0.01
0.02
0.02
0.00
EACE
bin=60
0.04
0.04
0.01
0.03
0.02
0.01
0.02
0.03
0.01
bin= 120
0.02
0.02
0.01
0.02
0.01
0.01
0.02
0.02
0.00
F
Evaluations for Different Size of Datasets
In this section, we investigate how different dataset sizes influence performance of the method. Specifically, we construct subsets by sampling 20%, 40%, and 80% of the trajectorics from the medium datasets cach environment. We abandon using medium-expert and medium-replay for this analysis, as the trajectories these datasets exhibit large inter-trajectory discrepancies that could confound the evaluation.
29

---

Published in Transactions on Machine Learning Research (01/2026)
Table 22: Evaluation of Different Size of Datasets on CEDT and DT.
CEDT
DT
HalfCheetah
Hopper
Walker2d
HallChoetah
Hopper
Walker2d
20%
0.24+0.06
0.1340.04
0.23=0.02
0.25#0.04
0.20=#.06
0.23+0.06
W1 Dis
40%
0.19110.04
0.13:10.06
0.32.10.6
0.2011000
0.1410.03
0.31110.06
80%
0.19++004
0.12++0.01
0.26+0008
0.25+00.3
0.16+0.03
0.34+0.12
20%
42.87.10.49
60.9113.21
76.35.1..5
42.50.10.59
55.22-6.24
77 77.55:12.43
Return
40%
42.83:10.59
60.8416.24
74.2011243
42.78:10.39
59.20-5.66
76.80:11.97
80%
42.84+0.23
59.05+2.007
76.37+3.51
12.85=00.17
57.71=4.08
75.69+.2.35
Table 23: Evaluation of Different Size of Datascts on CEDC and DC.
CEDC
DC
HalfCheetah
Hopper
Walker2d
HalfCheetah
Hopper
Walker2d
20%
0.25:10.08
0.1310.02
0.2310.04
0.23:0004
0.1910.03
0.26110.4
W1 Dis
40%
0.21
0.13:00.03
0.26=0.06
0.2310.03
0.15#0.00
0.3810.15
80%
0.21110.11
0.13.10.03
0.2510.05
0.2410.07
0.1410.04
0.37
20%
42.90:10.39
63.95:15.49
79.10:0.31
43.00=2..00
58.69-2.24
77.7910.83
Return
10%
12.95+10.27
63.3411.64
77.89+1.900
43.1111.299
60.95-1.62
77.32
80%
42.95:10.11
66.3211.666
77.9913.03
43.07
62.17-5.03
78 78.6211225
Table 24: Evaluation of Different Size of Datasets on CEDMamba and DMamba.
CEDMamba
DMamba
HalfChoetah
Hopper
Walker2d
HalfCheetah
Hopper
Walker2d
20%
0.23
0.15+0.05
0.26+0.05
0.24=10.04
0.15=00.3
0.38+00.15
W1 Dis
40%
0.18110.01
0.1410..5
0.22110.07
0.2410003
0.1110..3
0.36.10.07
80%
0.27:10.03
0.12+10.3
0.24+0.08
0.23+0.05
0.12+00.33
0.5610.52
20%
42.81-10.33
59.74116.33
78.31
42.80.1.37
65.77114.84
75.49
Return
40%
42.95:10.23
61.8114.71
77.6410.64
42.8110.25
69.6911.71
75.7311.55
80%
13.13-10.26
61.90+43.09
78.23+1.72
43.22+0.16
72.10+8.20
68.78+11.100
G
Comparison with Cal-QL and Other Q-learning Methods
To further demonstrate the offectiveness of CEDM, we include comparisons against the offline part Cal-QL NNakamoto el al. 2024), , BCQ (Fujimoto el al. 2019), and CQL (Kunar el. al. 2020). Specifically, results of Cal-QL are sourced from Table 5 its original publication, whereas the results for BCQ and CQL are obtained from the Table 2 of D4RL (Fu et al. 2020
30

---

Published in Transactions on Machine Learning Research (01/2026)
Table 25: Comparison of Return with Cal-QL and Other Q-leariing Methods (with Total).
HalfCheetah
Hopper
Walker2d
Total
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rep
Med
Med-Exp
Med-Rey
CEDT
42.77
86.64
39.59
59.85
78.88
62.11
73.77
107.70
37.48
588.79
CEDC
42.92
88.15
41.17
62.67
75.06
75.14
76.09
108.05
44.64
613.89
CEDMamba
42.87
85.59
40.86
59.73
90.28
78.19
76.69
108.05
56.56
638.82
DD
42.88
84.05
39.09
57.80
90.16
20.34
72.87
107.49
31.66
546.34
DC
42.81
88.16
39.69
59.13
76.57
30.84
74.87
104.36
50.81
567.24
DMamba
42.97
86.73
39.26
69.39
97.47
41.03
70.47
87.20
55.63
590.15
Cal-QL
52
54
51
89
69
76
75
96
52
614
BCQ
40.7
64.7
38.2
54.5
110.9
33.1
53.1
57.5
15.0
467.7
CQL
44.4
62.4
46.2
58.0
98.7
48.6
79.2
111.0
26.7
575.2
31


Reviews:
[{"review_id": "M0sMzuigFn", "reviewer": "Reviewer_KM5j", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The Authors provide a procedure for estimating the reliability of sequential  actions for  Offline Deep  Reinforcement Learning. This is important for addressing the reliability of rare occurrences. The Manuscript  show promising results improving the robustness of the trained systems.  \n\nThe procedure is based on binning the policies according to their frequencies in the training datasets. The experimental histogram of offline data and the trained policy for the same are compared with Wasserstein distance. The distributions of the histograms show how reliable the action sequence is on average, their distance between the distributions show how well the trained data matches the experimental data. To me KL-divergence of the probability histogram would be easier to calculate the distribution distances.\n\nThe procedure is mathematically sound and well written.  The results show that it improves the results, but is sensitive length of the frame of actions till RTG.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The definition of the procedure is clear, proven and concise providing estimators  Expected Agent Calibration Error and Calibration Enhanced Decision Maker.", "audience_yn": "Yes", "audience_explanation": "The problem of robustness becomes more and more significant and Authors provide a rather simple and well explained method estimate this. However, a topic of interest would be the influence of data set size. It is expected that larger data sets combined with larger neural networks have emergent capabilities.  s this also reflected in the estimators?", "requested_changes": "1) The language does not sound English.  Please ask a native speaker to check it out.\nExamples:\n\"remains a paramount concern within the community\" -> concerns the community.\n\"While extant research\". -> prior work\n\"remains a comparatively underexplored avenue of investigation\" ->  remains a rather rare topic.\n\"we pioneer the concept of \"  we introduce the concept of\n\"A series of in-depth case studies are undertaken to examine CEDM \". A series of In-depth case studies of CEDM are carried out, with applications on Decision Transformer, ....\n\n2) I would like to see if using KL-divergence (i.e cross - entropy)  would change the results when used for estimating the similarites of the distributions. \n\n3) What is the effect of the size of data sets on the performance?", "broader_impact_concerns": "This Manuscripts provides means to ease ethical problems on rare incident handling of high risk cases.  However, it does not ensure  a guarantee, but indication. The Authors provide an appropriate statement on Broader Impact Concerns", "additional_comments": "None", "extra_scores": {}, "extra_text": {}}, {"review_id": "QsPeC76ahf", "reviewer": "Reviewer_y6VS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method for calibration in offline deep reinforcement learning, with a particular focus on trajectory-optimization approaches that leverage large sequence models. They provide theoretical justification for their method and present experiments across three different models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1) The paper is well-written and well-structured, combining theoretical analysis with practical applications.\n\n2) The proposed CEDM method is simple, general, and appears compatible with multiple sequence-model architectures.\n\n3) Theoretical contributions, including Theorem 1, offer conceptual justification for why reducing state–action distribution divergence can improve calibration.\n\n4) The empirical evaluation is thorough and performed across three different model families, showcasing the applicability and generality of CEDM.\n\n5) The experimental results demonstrate consistent improvements in Wasserstein-1 distance and return across multiple D4RL benchmarks.", "audience_yn": "Yes", "audience_explanation": "The paper focuses on Offline Reinforcement Learning, large sequence-model architectures, and issues of trustworthiness and calibration, making it well aligned with the interests of the TMLR community.", "requested_changes": "Questions:\n1. In Definition 1, you introduce the condition *dis(π₁, π₂) ≤ δ*, but the paper does not specify how this distance is defined.  \n   - What exact metric is used to measure the distance between two policies?  \n   - How is δ chosen, and does the metric satisfy properties such as symmetry or triangle inequality?\n\n2. While the paper provides theoretical results regarding EACE (Definition 2, Theorem 1), the analysis focuses only on the *expected* calibration error.  \n   - Have you examined the variance of calibration error, either empirically or theoretically?  \n   - Does CEDM reduce not only the mean EACE but also the variability of calibration across different states or confidence bins?\n\n3. Proposition 1 relates total variation distance to the Wasserstein-1 distance.  \n   - Do you have empirical results illustrating the relationship between empirical TV distance and W1 in your setting?  \n   - Can you provide evidence that the bound is non-vacuous or practical comparisons between TV and W1)?\n\n4. Do you have any results comparing your method with Q-learning? Including such a comparison could strengthen the paper significantly.\n\n\nImprovements in writing:\n\n5. There are multiple inconsistencies in capitalization and abbreviation usage. Examples include:\n\n- *Deep reinforcement learning (DRL)* vs *Deep Reinforcement Learning (DRL)*  \n- *Offline deep reinforcement learning (offline DRL)* even though DRL was already defined  \n- Inconsistent usage of abbreviations:\n  - *Expected Agent Calibration Error (EACE)* is sometimes reintroduced or repeated  \n  - *Return-to-go (RTG)* must be *Return-To-Go (RTG)*  to be consistent \n  - Similar inconsistencies appear with *Calibrated Q-learning (Cal-QL)*, *Calibration Enhanced Decision Maker (CEDM)*, etc.\n\nThe paper should define each abbreviation once, apply consistent capitalization, and avoid redefining abbreviations throughout the text.\n\n6. Several references are listed only as *arXiv preprints*.  \nFor works that have been published in conferences or journals, please cite the official publication venue instead of the arXiv version.  \nThis is important to maintain citation quality and formality during the review process.", "broader_impact_concerns": "NA", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}, {"review_id": "rzmYkXYpGw", "reviewer": "Reviewer_h8Tc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, author propose a method for calibration in offline Deep Reinforcement Learning, particularly focusing on trajectory optimization methods that utilize large sequence models.\nThis paper has following main contributions\n1. the paper introduces a new metric called Expected Agent Calibration Error (EACE) for assessing agent calibration, which is similar to the ECE metrics but tailored for agents. \n2. A theoretical contribution is the derivation of a bound, which shows that the difference in EACE is bounded by the Wasserstein-1 distance between the agent's and the ideal policy's state-action marginal distributions\n3. authors introduce the Calibration Enhanced Decision Maker (CEDM), CEDM modifies the architecture of sequence models via a binning executor that processes feature distribution histograms as additional inputs. The goal is to provide the information of the data distribution. Therefore, minimizing distribution distance and hence calibration error.\n4. CEDM is evaluated on the D4RL tasks using Decision Transformer, Decision ConvFormer, and Decision Mamba architecure. The results show that CEDM reduces the W1 distance and improvement in normalized returns\n\n\nStrengths:\n1. The proposed framework and binning mechanism are applicable to different sequence modeling architecture as demonstrated by the experiments.\n2. Using binning mechanism as addtional input is easy to implement and can easily added to existing sequence models\n3. This paper solve the challenges of calibration  Deep Reinforcement Learning (DRL) sequence models which is a timely and important problem.\n\n\nWeakness: \n1. The paper claims to improve calibration but relies entirely on a proxy metric which is w1 for validation. It does not measure EACE to verify that calibration actually improved.\n2. The evaluation compares CEDM incorporated models only against corresponding base models. but it does not compared them with existing calibration methods such as Cal-QL, making it difficult to assess relative performance.\n3. The paper does not explore/mentioned different choices such as the number of bins, which features are binned, or where the histogram tokens are added in the sequence and how it affects the results. The paper reports W1 distance and returns, but it does not show any calibration plots or EACE values.", "claims_and_evidence_yn": "No", "claims_explanation": "The authors main argument is that the proposed method improves the calibration of the agent. However, the experimental evidence provided does not directly support this claim.\n\n1. The authors prove that EACE is upper-bounded by the W1 distance. In the experiments, they demonstrate that CEDM reduces the W1 distance. However, minimizing an upper bound does not mathematically guarantee that the objective (EACE) is minimized. Without directly measuring EACE on the test set, the claim that calibration has been enhanced remains a theoretical hypothesis rather than an empirically verified fact. the paper must empirically show calibration improves when W1 decreases on the same experimental runs. \n\n2. The authors argue that measuring EACE is impractical in real-world applications. While true for deployment, the experiments are conducted on standard D4RL tasks. In this environment, the authors could have estimated the calibration error to validate their theoretical derivation. The absence of this direct validation weakens the paper significantly.", "audience_yn": "Yes", "audience_explanation": "The paper targets a highly relevant intersection of topics: Offline Reinforcement Learning, Large Sequence Models, and Trustworthiness/Calibration, which is useful for the TMLR audience. As the authors mention, the calibration of trajectory optimization is a comparatively under explored topic, and hence this paper addresses an important aspect. The theoretical perspective connecting calibration error to state-marginal distribution matching (i.e., W1 distance) provides a fresh perspective. The application of the method across different architectures adds to its potential for wider acceptance.", "requested_changes": "1. The current evaluation relies entirely on a proxy W1 distance metrics, based on the theoretical upper bound from Theorem 1. Minimizing an upper bound does not guarantee that the actual calibration error is reduced. You must define a tractable approximation of EACE and report it empirically for the D4RL tasks. Since these are simulation-based benchmarks, the safety concerns mentioned do not valid to the evaluation phase. You must demonstrate that CEDM actually achieves lower calibration error than the baselines, not just better distribution matching.\n\n2. To prove the theoretical work, you should empirically validate the relationship between the W1 distance and the calibration error. Please provide scatter plots or correlation analysis showing the relationship between the measured W1 Dis and the measured calibration error. This is crucial to confirm that W1 Dis is a valid proxy for calibration in this domain.\n\n3. Try different number of bins and show sensitivity of W1, return, and EACE.\n\n4. If possible, Compare to existing calibration approaches such as Cal-QL", "broader_impact_concerns": "NA", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}, {"review_id": "lqWXv6xX0Z", "reviewer": "Reviewer_M59X", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce the Expected Agent Calibration Error (EACE), a new metric for systematically measuring and improving calibration in trajectory optimization-based offline reinforcement learning with large sequence models, and propose the Calibration Enhanced Decision Maker (CEDM) method, which augments agent inputs with distribution histograms to provide richer context; together, EACE and CEDM enable minimizing the difference between the state-action distributions of the agent's policy and the dataset, leading to empirically demonstrated improvements in both calibration and performance on simulated D4RL MuJoCo benchmarks", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors evaluate their approach using the D4RL MuJoCo benchmark suite, which consists of simulated continuous control tasks, including HalfCheetah, Hopper, and Walker2d, each with datasets representing different levels of agent expertise. To prove the validity and effectiveness of their proposed EACE metric and the CEDM calibration method, the authors implement these ideas within leading sequence-model-based offline RL algorithms—Decision Transformer, ConvFormer, and Decision Mamba—and systematically show both improved calibration (lower EACE and Wasserstein distance) and agent performance compared to the original baselines using these diverse simulated tasks.", "audience_yn": "Yes", "audience_explanation": "The community that is working on the offline RL and the high-risk and costly environments where offline RL is important, such as Robotics, AV.", "requested_changes": "The paper is well structured, provides benchmarks, and algorithms. could be improved by incorporating more graphical visuals and figures to help readers intuitively grasp key concepts, metric behaviors, and results.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SbfLmcrTG6", "reviewer": "Reviewer_s24B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis work proposes a new framework to enhance agent calibration of offline Deep Reinforcement Learning (DRL) algorithms that utilize Large Sequence Models (LSM) to improve the overall trustworthiness. In terms of contribution, it Introduces Expected Agent Calibration Error (EACE) to quantify agent calibration and gives a theory linking EACE to the state-action marginal distance in $W_{1}$ and proposes Calibration Enhanced Decision Maker (CEDM), a framework that feeds distribution histograms into LSM to reduce the state-action marginal distance and improve calibration. The method is applied to Decision Transformer, Decision ConvFormer, and Decision Mamba over OpenAI Gym and MuJoCo to show empirical performance. \n\n---\n\n**Strength**\n\n1. To my knowledge, this method is new and the authors positioned as first to target agent calibration for offline DRL with large sequence models.\n2. The motivation is clear and the problem they studied seems interesting. \n3. I believe CEDM should be implemented easily, and overall method reads simple and neat for me.\n4. Extensive experiment over different LSM backbone is conducted. \n\n---\n\n**Weaknesses**\n\n1. There are some theoretical limitation, see my Requested Changes for detail.\n2. Theoretical setup and experiment setup are not clearly written, which is unfriendly to the readers who are unfamiliar with those RL tasks. \n3. The method is unclear whether can be scalable to larger models, as the use of Wasserstein naively inside the training loop does not scale well.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Empirically, yes. The paper clearly states the three evaluation questions and reports performance over 5 seeds on HC/Hp/Wk (Med/Med-Exp/Med-Rep), showing return gains and reduced $W_{1}$. However, some theoretical results lacks clarity.", "audience_yn": "Yes", "audience_explanation": "This paper could attract readers from alignment community, as the issue studied in this paper is correlated to some obstacles faced by alignment community. The theoretical formulations might inspire the researchers from the alignment community to build upon it.", "requested_changes": "- The full introduction is for DRL. Please consider to restructure it by adding some part for Large Sequence Models, e.g., with a clear explanation of decision transformer.  \n- Some unclear definition and theoretical derivation:\n    1. In Definition 1, what is the metric space used to evaluate the distance between two policies? Please clearly write it out in the definition. Also, please add a further detail to explain why EACE is computationally costly (under the paragraph below eq 4), as this is directly related to your motivation. \n    2. For $\\hat P$, what is the exact confidence and how should I set it? As this is related to the central part of your design, it definitely requires further discussion, analysis and even ablation to improve the clarity. Arbitrarily set it as $\\rho_{\\pi_{\\theta}}(s,a)$ doesn't seem to be a model-reported confidence about the event. \n   3. For Proposition 1, I am concerned regarding its validity under continuous space. For discrete-only analysis, I believe it requires further stronger structural assumptions on finite discrete supports with strictly positive inner point distance under your metric space. \n\n- Please expand $\\S~5.1$, explaining what is each task doing and its metric performance in detail.", "broader_impact_concerns": "This is more like a theoretical-grounded method applied to several limited scenarios, and I do not believe a broader impact statement is required here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)