Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns
Authors: Dong Tian
Onur Celik
Gerhard Neumann
Venue: ICLR 2026 Poster (2026)
Keywords: Soft Actor-Critic (SAC)
Transformer-based Critic
Sequence Chunking
N-step Returns
Critic Alignment
Double Q-Learning
Deep Reinforcement Learning
Primary area: reinforcement learning
TL;DR: T-SAC trains a transformer-based SAC critic on windowed sequence chunks with N-step TD returns and critic alignment, improving temporal credit assignment, stability, and sample efficiency on long-horizon, multi-phase, and sparse-reward tasks.

Abstract: We introduce a sequence-conditioned critic for Soft Actor--Critic (SAC) that models trajectory context with a lightweight Transformer and trains on aggregated $N$-step targets. Unlike prior approaches that (i) score state--action pairs in isolation or (ii) rely on actor-side action chunking to handle long horizons, our method strengthens the critic itself by conditioning on short trajectory segments and integrating multi-step returns without the need of importance sampling (IS). The resulting sequence-aware value estimates capture the critical temporal structure for extended-horizon and sparse-reward problems. On multiple benchmarks, we further show that freezing critic parameters for several steps makes our update compatible with CrossQ's core idea, enabling stable training without a target network. Despite its simplicity, a 2-layer Transformer with $128$--$256$ hidden units and a maximum update-to-data ratio (UTD) of $1$, the approach consistently outperforms standard SAC and strong off-policy baselines, with particularly large gains on long-trajectory control. These results highlight the value of sequence modeling and $N$-step bootstrapping on the critic side for long-horizon reinforcement learning.

Published as a conference paper at ICLR 2026
CHUNKING THE CRITIC: A TRANSFORMER-BASED SOFT ACTOR-CRITIC WITH N-STEP RETURNS
Dong Tian" Onur Celik Gerhard Neumann
Karlsruhe Institute of Technology (KIT)
ABSTRACT
We introduce sequence-conditioned critic for Soft Actor-Critic (SAC) that mod- els trajectory context with a lightweight Transformer and trains on aggregated N- step targets. Unlike prior approaches that (i) score state-action pairs in isolation (ii) rely on actor-side action chunking to handle long horizons, our method strengthens the critic itself by conditioning on short trajectory segments and in- tegrating multi-step returns without the need of importance sampling (IS). The resulting sequence-aware value estimates capture the critical temporal structure for serended-horizonn and sparse-reward problems. On multiple benchmarks, we further show that freezing critic parameters for several steps makes update compatible with CrossQ's core idea, enabling stable training without a target net- work. Despite its simplicity, 2-layer Transformer with 128-256 hidden units and a maximum update-to-data ratio (UTD) of 1, the approach consistently out- performs standard SAC and strong off-policy baselines, with particularly large gains on long-trajectory control. These results highlight the value of sequence modeling and N-step bootstrapping on the critic side for long-horizon reinforce- ment learning. Code is available on GitHub, and full test logs can be viewed on Weights Biases.
Ules dts-sl
Que decess
Qr Onson
1.0
Success Rate IQM
0.6
Transformer Critic (Causality Mask)
0.4
0.2
SAC
piDE
PPP
8t
ai
a+++1
0+++
5
(x10*)
(a) Transformer critic processes segments of actions rather than single action, using causal self-attention so token attends only to timesteps S i, preventing leakage.
(b) Aggregated success Meta-World MLI (50 tasks) comparing T-SAC to step-based and episodic baselines. Curves show IQM success rate with 95% bootstrap confidence intervals: unless noted, results average 8 seeds and count success only the final timestep.
Figure 1: T-SAC overview and aggrogate soradWorld MLI results.
INTRODUCTION
Off Off- Poe-policy actor-critic methods are the workhorses of continuous control. Soft Actor (SAC) (Haarnoja et al., 2018a) is notable for its sample efficiency and stability, driven by tightly controlled bootstrap targets and mechanisms that mitigate value overestimation.
"Corresponding author: dong. tian@cut look. de. This work was conducted at KIT and supported solely by the Institute for Anthropomatics and Robotics (IAR) through the Autonomous .earning Robots (ALR) Lab.

---

Published as a conference paper at ICLR 2026
Yet accurate value estimation remains difficult in sparse-reward, long-horizon, and high- -iseesiinnsi settings. Towards this end, multi-step targets (N (N-step returns) (Sutton et al., 1998) can reduce boot- strapping bias and accelerate credit assignment, but in off off-policy settings they typically require im- portance sampling (IS) (Precup et al., 2000). IS introduces high variance and often destabilizes train- ing in practice, which limits the effective horizon of multi-step supervision in modern actor-critic methods (Munos et al., 2016).
More recently, critics and policies have been extended operate on temporally extended actions, in- cluding movement primitives (Otto et al., 2023a; Li et al., 2024b) and action chunking (Zhang et al., 2022: Li al., 2025). By evaluating or executing short open-loop action sequences, these methods can improve exploration and speed up value propagation. However, in online reinforcement learning they introduce important trade-offs: fixing a chunk length reduces control frequency and reactivity, couples performance to horizon hyperparameter, and complicates learning when different phases of task benefit from different temporal resolutions. As a result, action-chunking methods have not consistently produced robust gains in online, off-policy settings, despite their success in structured or episodic domains.
Recent work has shown that the stability of off-policy actor-critic methods can be substantially im- proved through careful control of critic optimization dynamics. Techniques such as slowing critic updates, partial parameter freezing, and function-class regularization reduce target drift and mit- igate overestimation (Vincent et al., 2025; Piché et al., 2021; Gallici et al., 2024). In this vein, CrossQ (Bhatt et al., 2019) demonstrates that appropriate normalization-via Batch Renormaliza- tion (BRN) (loffe, 2017) combined with bounded activations- stabilize bootstrapped value learning the extent that target networks can be removed altogether. These results suggest that scaling up off-policy RL does not fundamentally require architectural changes to the policy, but rather careful design of the critic's 's update and normalization schemes.
Transformer-based Soft Actor-Critic (T-SAC). Our primary contribution is T- SAC, a step-based Soft Actor-Critic in which the standard MLP critic replaced by a sequence- conditioned Transformer critic. The critic is trained on short trajectory segments and supervised with pariable-horize N-step returns. For each segment, it predicts values for all state-action pre- fixes and is jointly trained across multiple temporal horizons. By modeling temporal structure inside the critic through attention over brief actionn windows, T-SAC improves long- nonghhorizon credit as- signment while keeping the policy strictly one-step and the update rule free of importance sampling.
This paper:
Around this core design, we introduce several supporting choices that improve stability and prac- ticality. First, we use causal masking and lightweight Transformer architecture, following TOP- ERL (Li et al., 2024a) in spirit but adapted to a step-based SAC setting. Second, we employ a gradient-averaged N-step loss. Instead of averaging N-step targets, which can attenuate sparse long-horizon rewards, we compute a loss for each horizon and average their gradients. Because adjacent horizons correspond to overlapping prefixes and produce correlated gradient estimates, this reduces update variance while preserving long-horizon signal. Third, we adopt a lightweight critic parameter-freezing schedule that enables stable training without target networks, in contrast to Polyak averaging as used in the original SAC (Haarnoja et al., 2018a). Empirically, T-SAC preserves SAC-style stability and remains sample efficient, solving most Mera-World tasks in approximately 5M interactions, achieving 96.8 percent success on dense Box- fox-Pushing, and remaining stable at low update-to-date ratios across benchmarks.
2 RELATED WORK
2.1 TRANSFORMERS FOR RL
Transformer critics for episodic vs. step-based control. Episodic RL (ERL) replaces per-step actions with trajectory-level primitives (Otto et al., 2023a; Li et al., 2024b), easing long-horizon reasoning but complicating temporal credit assignment, especially off-policy. TOP-ERL (Li et al., 2024a) partitions each episode into fixed-length segments and trains Transformer critic that attends across segments. With truncated N N-step targets (Sutton et al., 1998), the critic predicts per-se ser-segment returns, enabling officcllcy replay while exploiting attention for partial observability and long- range dependencies. On manipulation benchmarks, POP-ERL improves over prior ERL-based, step- based, and on-policy value-based baselines (Li et al., 2024a).
2

---

Published as a conference paper at ICLR 2026
TOP-ERL (Li et al., 2024a) is an early Transformer-critic method for episodic control: ProDMP policy (Li et al., 2023) outputs full trajectories. and the critic evaluates segment-level returns along them. Replanning is discussed but not implemented in the released experiments. The method addi- tionally relies on a Trust Region Projection Layer (TRPL) (Otto et al., 2021), typically uses ~00 interactions, and still underperforms on some multi-phase Meta- Meta-World tasks (Yu et al,, 2020) (e.g., Assembly, Disassemble) and Box-Pushing at tight tolerances (Otto et al.).
By contrast, T-SAC remains in the standard step-based, closed-loop regime: the policy outputs an action at every time step from the current state, and the Transformer critic is prefix-conditioned on short stte-action windows sampled from replay. It is trained with non-soft N N-step TD targets without importance sampling, keeping it closer to conventional off off-po actor actor-critic methods than to episodic ERI:: temporal abstraction lives in the critic's ss conditioning and targets, not in an open- loop policy.
Transformer policies for offline RL. Decision Transformer and related offline sequenee-modee
approaches (Chen et al., 2021; Janner et al., 2021) instead perform policy-side sequence modeling on fixed datasets, mapping past trajectories and target returns directly to actions. These methods are complementary to T-SAC: we use a Transformer only for the critic, in an online, off-policy setting. In principle, an offline Decision Transformer policy could be paired with a T-SAC-style critic, or our critic architecture could be adapted to evaluate trajectories generated by such sequence policies.
2.2 TRAINING WITHOUT TARGET NETWORK
Target networks stabilize bootstrapped critics but slow value propagation and add complexity (Kim et al., 2019; Piché et al., 2021). Recent work instead limits target drift or smooths backups. The strongest result, CrossQ, achieves state-of-the-ar (SOTA) sample efficiency in continuous control by removing the target network and stabilizes single bootstrapped critic with BRN (loffe, 2017). Related target-free strategies include value smoothing (mellowmax) (Asadi & Littman, 2017; Kim al., 2019). constrained/proximal updates (Durugkar & Stone, 2018; Ohnishi et 2019), function- space regularization and partial freezing (Piché et al., 2021; Asadi et al., 2024; Vincent et al., 2025) feature decorrelation (Mavrin et al., 2019). Theory unifies these mechanisms alternatives to target networks via partial freezing, regularization, and separation of optimization dynamics (Fellows et al., 2023).
2.3 ACTION CHUNKING IN REINFORCEMENT LEARNING
Action chunking replaces per-step control with short open-loop sequences of actions ("chunks"") which can capture temporal structure, accelerate value propagation via longer effective horizons, and promote temporally coherent exploration (Kalyanakrishnan et al., 2021; Zhang et al., 2022). The trade-off is reduced reactivity within a chunk (Liu et al., 2024), but for long-horizon, sparse- reward manipulation this bias often pays off (Zhang et al., 2021; Gupta et al., 2019).
Reinforcement Learning with Action Chunking (Q-chunking) (Li et al., 2025) applies TD-based actor-critt learning directly in the chunked action space: the policy proposes an H-step action se- quence and the critic evaluates Q(st, 00: enabling unbiased H-step backups and efficient up- dates (Li et al., 2025). In their implementation, the critic is simple MLP that ingests the state con- catenated with the proposed action chunk (rather than a sequence model), which keeps the method lightweight while still reaping the benefits of temporally extended actions (Li et al., 2025).
PRELIMINARIES
Off-Policy Reinforcement I Learning. Reinforcement learning (RL) (Sutton et al., 1998) formalizes
sequential decision making as Markov decision process (MDP) (S, A, P.r,7): at time i. an agent observes st, selects 0t ~ (- sr), receives r; = r(t,, at), and transitions to 8:+1 ~ P(- 8t, at); the goal is to learn a policy maximizing J ( = E,, 1 7*r,] using value functions (*((s) and
Q=(ssa a) that satisfy Bellman consistency, with Q* inducing the optimal policy. Algorithms differ in how they estimate and improve these quantities Caalntttess-vlue---ee
learning (Watkins & Dayan, 1992; Hessel et al., 2018; Van Hasselt et al., 2016; Rummery & Niranjan, 1994), actor-critic (Mnih et al., 2016; Schulman et al., 2017; 2015a; Fujimoto al., 2018), or direct policy optimization (Kakade,
3

---

Published as a conference paper at ICLR 2026
2001: Peters & Schaal, 2008)- while managing exploration vs. exploitation (Sutton et al., 1998). Off-po RL learns a target policy x from data generated by a (possibly different) behavior policy μ4, reusing transitions (s,a, a, s') via replay buffers and bootstrapped Bellman updates; distribution mismatch when evaluating from ju-data can be corrected (e.g., with IS (Sutton et al., 1998)). This decoupling enables efficient experience Dayan, 1992) and the SAC (Haarnoja et al., 2018a) family.
and underpins methods like Q-learning (Watkins &
Soft Actor-Critic. Let aa s) be a stochastic policy with parameters 0. Let Qu(s, a) be the critic with parameters vb, and let Q be its target network (e.g., Polyak-averaged copy of Qw). SAC (Haamoja al., 2018a) maximizes a maximum-entropp objective improve robustmess and exploration:
(
Σ 0
7 (Tr + ((T {
where y is the discount factor.
The Bellman target is
V/ r2 Bey Bou Q0(sEt1, a log (a' sct1)];
and the critic minimizes the squared error
(1) ( ao Ba (t) ve)
The actor minimizes
Ja (0) EgwD, a a log Ta(a [s) Qu(s,a)
The temperature tuned match a target entropy by minimizing
[(a) E amap (-1s) a< log (a () + R2)]
(Haarnoja et al., 2018b).
Recent work studies more expressive MaxEnt actors beyond Gaussians, e.g., energy-based policies trained with Stein variational updates and practical entropy estimation AS: Messaoud et al.), and diffusion- policies that optimize a lower bound on the maxinum-entropp objective (DIME; (Celik al., 2025)). These approaches are largely orthogonal to our critic-side focus.
N-step Returns and IS. Using the same notation as above, on on-policy N-step targets speed up credit assignment (Sutton et al., 1998; Schulman et al., 2015b; Mnih et al., 2016), i.e.
(n) St.soft
x Earps [re+k a log no (ar4k) sc+k)] + + E (Q
km
With off-policy data drawn from a behavior policy μ & TT. per-decision importance ratios
[ Sc+R)
Pt++
p(ac+k setk)
can be used to correct the distributional mismatch (Sutton et al., 1998), i.e.,
G
II pe+s
[re+h loo To(actk [ + II pets Qa attn).
with the convention that an empty product equals 1. When μ wa. all p's are I and ((f) reduces to the standard N-step target. Pure IS can introduce high variance, therefore the step length n cannot be chosen to be very large (Precup et al., 2000; Sutton et al., 1998; Espeholt et al., 2018).
Averaged N-step Returns for Critie Updates. Using N-step returns is standard way to reduce tar- get bias for the critic (Sutton et al., 1998). For starting index t and horizon n E [1, max length), following Zhang et al. (2022) we define the N-step target as
((n)) non- ......, a0+n-1)
Tress + 7" Vo(se+a))
(1)
4

---

Published as a conference paper at ICLR 2026
(0, 1] and a target network parameterized by o. Here Vo(s) (e)))(()s) (s(s,a) is the (non-entropy) bootstrap value under the current policy. While larger reduces bootstrapping bias, the variance of Gka) typically grows with (Precup al., 2000). A practical variance reduction is to average partial returns (Konidaris et al., 2011; Daley et al., 2024):
with discount γ
((n)
=
(2)
1=1
This averaging lowers the variance of the reward-sum component from O toward roughly 0(n/4)-0(n/3) (decreasing with n, depending on reward correlations), and makes the value- estimation term decay 1/n; under the same assumptions (Daley al., 2024), the full proof appears in App. C. This motivates using multiple horizons during critic training (see $ 4.2). How- ever, in T-SAC implementation,
do not average N-step returns directly, this strategy performs poorly in sparse-reward settings (see App. F).
4 TRANSFORMER-BASE BASED SOFT ACTOR-CRITH (T-SAC)
4.1 N-STEP RETURNS FOR CRITIC UPDATES
4.1.1
GRADIENT-LEVEL AVERAGING OF N-STEP RETURNS
For horizon i, define the prefix-conditioned online critic output Q Qst a....., a1pi-1). Directly averaging targets dilute sparse reward signals (App. F). In- stead, we form per-horizon losses
Notation.
(.(+) = 122 GO))
i= i=
(3)
where shared- weights online critic outputs Q for each prefix (st, a....., 4143-1) (st and a, separate embedding layers). We then average gradients across horizons:
V
(.).
(4)
Because adjacent horizons have overlapping targets and correspond to adjacent decoder posi- tions in the same network, their per-paranne per-parameter gradient contributions are positively- not per- cocll-correelted, Averaging therefore reduces update variance while preserving sparse signals (App. D, F; Fig. 2).
4.1.2 STABLE CRITIC LEARNING WITHOUT IMPORTANCE SAMPLING
Standard off-policy N-step TD presumes that post-a; actions are drawn from the current policy To, which mismatches replay generated by behavior policy μ. Per-decision IS with Pe+k mala ((511))))
placta [8+44) corrects this but injects high variance (Precup et al., 2000; Sutton et al., 1998; Es- peholt et al., 2018).
Similarly Li al. (2024a), instead change the target: the critic predicts prefix-conditioned values for realized prefixes from replay,
{ ((((s,)at4iii))) Hez
ama4i-1)
with i-step targets
((), =
Neets + Vo(st)))
(5)
3=0
and the loss
Lentic = B( 1)
1 Σ
(rote-)) - ( (1)+)-1))
(6)
5

---

Published as a conference paper at ICLR 2026
As rewards follow the recorded prefix 0t:t+i-1, assumption that actions came from xe is needed, and hence, no is required. Only the bootstrap at t+ii depends on X via (31+s)
Supervising short windows with multi-horizon targets and averaging their gradients yields stable up- dates and preserves sparse signals, enabling "multi-step supervision, one-step policy update" without IS (Fig. 2, 9b).
4.1.3 CONNECTION TO STANDARD N-STEP TD AND THEORETICAL GUARANTEES
Equations 5-6 can be viewed as standard multi-step TD update in MDP where each action prefix arati-1 is treated as an extended action. For a fixed horizon i, we define
(8t, ora+i-1);
use equation 5 as the -tep target G (I), and minimize the squared TD error
(Qx - (z))
exactly as in classical N-step Q-learning.
The key difference to off-policy N-step TD with importance sampling (IS) what the critic is asked to predict. IS-corrected targets are (in principle) unbiased for QT, but have high variance and typically require clipping when behavior and target policies differ. Our critic instead learns the value of realized prefixes under the replay distribution.
From a theoretical perspective, conditioned on a given state & and realized prefix au:+i-i. the dis- tribution over future rewards is fully determined by the environment dynamics and does not depend on how this prefix was generated (behavior versus target policy). Empirically this yields more stable long-horizon learning. See App. E for the formal connection existing N-step TD theory.
4.2 CRITIC NETWORK AND OBJECTIVE
Our critic is a causal Transformer that ingests (st, at at+1..... a(+n-1) and outputs the n prefix- conditioned values ( at...., (,tss11)))))) (Fig. 1). For mini-batch of L trajectories. ran- dom start index t [0, N - n]. and horizons i E {1,...,m} n) with n sampled uniformly from (min mningeength, ... wam.mxalleee lenghh), the training objective is the mean-squared error over all hori- zons:
L n
1 ((v) Ln
ΣΣ
....... of d
During backpropagation we apply the gradient-level averaging across { described above. This construction leverages multi-horizon targets and inherits their variance-reduction benefits with- target-level signal dilution.
4.3 POLICY NETWORK AND OBJECTIVE
Following Ba et al. (2016); Parisotto et al. (2020) and Plappert et al. (2017), apply Layer Normalization to the policy's 's hidden layers (before the nonlinearity); Plappert et al. (2017) report this configuration to be useful for continuous-control actor -cittec especially when exploration noise injected. The objectives remain
(+) EgnD, aworo log (a 8) (s(s ])
(8)
J(a)= E,ND, antil - (log (a + H))
(9)
with target entropy -H (typically -dim(AA) and automatic temperature tuning (Haarnoja et al., 2018b). Unlike canonical SAC, our critic does not include entropy in the target; it estimates the standard (non-soft) action-value. The policy is optimized with an entropy-regularized objective, so exploration and regularization are handled entirely by the policy. This "non-soft critic policy- side regularization" " design is also used in MPO (Abdolmaleki et al., 2018), AWR/AWAC (Peng et al., 2019; Nair et al., 2020), and IQL/DQQL (Kostrikov et al., 2021; Hansen-Estrude Throughout this paper, all value targets are standard (non-soft) action-values; the entropy term appears only in the policy objective and not included in the critic targets.
al., 2023).

---

Published as a conference paper at ICLR 2026
4.4 CRITIC- CRITIC-PARAMETER PARAMETER FREEZING ENABLES TARGET-FREE TRAINING
CrossQ (Bhatt et al., 2019) removes target networks via batch renormalization (loffe, 2017) and bounded activations. In contrast, We eliminate Polyak updates with a short critic-freezing schedule: at the start of each critic segment we snapshot the online critic (Ø + 1), precompute and cache bootstrap targets (s) for all windows in that segment, and then freeze this snapshot while opti- mizing the online critic against the cached targets for the next K updates (reusing each segment across N₂ windows; Gymnasium MuJoCo (Towers et al., 2024): K=20). This lightweight decou- pling curbs target drift without batch renormalization or constrained activations, and on locomotion and sparse-reward tasks (e.g., Box-Pushing-Sppea Sparse (Otto et al.)) the resulting hard-copy schedule yields stable training that matches exceeds Polyak updates.
Our scheme introduces a single hyperparameter, the freezing interval K, i... the number critic updates for which we reuse single value snapshot Vg. Because targets are computed once per seg- ment before we enumerate windows, the minimum effective freezing interval is the segment length Tewg (for locomotion tasks, Lveg 20). Sweeping K (20, 100, 1000, 10000) on Gymnasium MuloCo Walker2d (Fig. 4g), we observe largely stable performance with only mild degradation for the largest K, suggesting that segment-level target caching already provides useful stabilization and that K is not brittle hyperparameter in our setting.
5 EXPERIMENTS
Positioning T-SAC. Prior value-based RL largely splits into (i) step-based methods (e.g.. SAC (Haarnoja et al., 2018a), CrossQ (Bhatt et al., 2019)) that dominate locomotion tasks (e.g.. Ant (Towers et al., 2024)), and (ii) episodic/tajjecton-lleeee methods (c.g.. BBRL (Otto et al., 2023a), TOP-ERL (Li ct al., 2024a)) that excel on long-horizon problems (c.g.. Box-Pushing (Otto et al.), Meta-World (Yu et al., 2020)). T-SAC partially narrows the gap between these regimes: retains one-step policy updates while using a sequence-condiiee conditioned Transformer critic, and em- pirically matches standard SAC on locomotion benchmarks while outperforming existing Trans- former-based approaches (e.g., GTrXL-style policies and TOP-ERL) on our long-horizon tasks.
Environments and Seeds. We evaluate T-SAC on 57 tasks spanning Meta-World MI.1 (50) (Yu et al., 2020), Gymnasium MuloCo locomotion (5) (Towers et al., 2024), and Box-Pushing (dense/sparse; 2) (Otto et al.). Meta-Worid probes task generalization; Gymnasium MuJoCo covers standard locomotion; and Box-Pushing stresses precise, contact-rich manipulation. Unless noted otherwise, we report means over 8 seeds (ablations use 4) with 95% bootstrap confidence inter- vals (Agarwal et al., 2021). Training time per IM environment steps, compared to off-policy base- lines, is shown in App. I. Baseline implementations and hyperparameters are detailed App. K and App. L, with environment details in App.
h
5.1 HETA-WORLD RESULTS
We run Meta-World MLI with UTD- 1, policy delay= 5, batch size 512; training time is ~-3 h per 1M env steps. Across 50 tasks, T-SAC solves most within ~5M steps and yields stronger aggregated IQM than strong baselines (per-task curves in App. A). On the hardest multi-phase tasks (Assembly, Disassemble, Hammer, Stick-Pull) T-SAC particularly strong (Fig. 2). In contrast, TOP-ERL (Li et al., 2024a) typically requires 20M steps reach similar aggregates. All comparisons use 5M env steps for T-SAC, while many baselines use larger budgets (Fig. 7, 8). Success is evaluated only at the final step (App. J), and our aggregates compute IQM per task and then average across tasks (unlike pooled-tase task IQM in TOL-
5.2 BOx PUSHING (DENSE AND SPARSE)
We evaluate dense and sparse variants of FANCYGYM (Otto et al.) Box-Pushing with tight success tolerances (position +5 cm, orientation +0.5 rad). Under dense shaping, T-SAC attains 96.8% success (Fig. 2), exceeding prior baselines (< 85%) under the same protocol. Under sparse re- wards-where these terms apply only at the terminal step-I T-SAC with the hard-copp copy critic reaches 60% success, compared to TOP-ERL's 70%. Thus T-SAC is state-of-the -het--rrt on Meta- World ML1 and dense Box-Pushing, and competitive under sparse rewards.

---

Published as a conference paper at ICLR 2026
Assemblyy
Disassemble
Hammee
StikPPll
1.0-
1.0-
1.0-
1.0
1014
9.8
8.8-
Serowee TOW
0.8
0.8
9.6
9.6-
0.6
Syouug rat
0.6
9.4
0.2
0.2
0.2
0.2
3.0
0.0
0.0
0.0
0.0
0.5
1.0
1.5
2,0
1.0
1:3
2,0
0.0
0.5
1
1,5
2.0
0.0
0.
1.3
1.5
2.0
Number
Interaction
Number Interudiion
Number
Interection
Number
Interaction
Box Pushing Dencn Rewwadd
Box Pushing Beward
1.0-
1.0
0.8
T-SAC (ours)
PPO gSDE
GTrXI((PO)
9.4
0.4
SAC
CrossQ
0.
0.2
TCP
TOP-ERL
0.0
0.5
0.0
0.5
3.0
1.5
BBRL
Number of Eav Interaction
Number Eay Interaction
Figure 2: Success- rate IQM vs. environment interactions on challenging Meta- World ML1 tasks and FANCY- GYM Box- Pushing. Panels show Assembly. Disassemble, Hammer. and Stick-Pull, plus Box Box-Pushing under dense and sparse rewards. Success is counted only at the final timestep.
T-SAC (Hard Copy)
T-SAC (Soft Copy)
CrossQ
SAC
TD3
100 Roswoot 2004
330k
14
230x
Manoered Soaentse
Nusber Inaau
Number Uas
Number Exy Interacion
Figure Episode return (IQM) vs. environment interactions for Ant, HalfCheetah, HumanoidStandup. Hop- per, and Walker2d. Evaluation follows Gymnasium v4 native shaping/temination (no reward normalization); we report undiscounted return and use cetrministic-poliqg evaluation.
5.3 GYMNASIUM MuJoCo
lightweight pritic-parameterrreeeee schedule ($ 4.4, App. B) enables target-free training: we re- move the target network while retaining SAC-style stability at low update rates (UTD & 0.75) and consistently match or surpass Polyak updates. Across the five Gymnasium MuloCo tasks, T-SAC competitive with or better than SAC Ant, Hopper, and Walker2d, with the largest gains HumanoidStandup and HalfCheetah (Fig. 3), and we do not observe slower early convergence de- spite conditioning the critic multi-step sequences from early exploratory policy. Because episodes have variable length, we apply simple action mask when constructing N-step targets from fixed-length windows avoid bootstrapping across episode boundaries (App. H); this mask is an implementation detail rather than a core component T-AC and does not degrade performance.
5.4 ABLATION STUDY
We conduct targeted ablations on FANCYGYM Box-Pushing (dense) and MUJoCo WALKER2D. These ablations are structured disentangle the effect of the sequence- conditioned Transformer critic-di our main algorithmic contribution- group, all settings are identical except for the component under test; across groups, minor differences (e.g., training budget step_length) arise from compute limits and are stated explicitly.
supporting design choices. Within each ablation
Transformer Components. We ablate three parts of the Transformer critic blocks, the causal mask, and self-attention--hlding all other settings fixed (Fig. 4a). Removing only self- attention invalidates the segment--oodittoned objective and typically diverges. Removing self- attention together with the ResNet and causal mask reduces the critie plain MLP: compare this baseline $ 5.4 and Fig. 4c.
Step Length and Min Length (Reuse factor). We sweep step_length and min_length, which bound the multi-horizon (N-step) supervision window. At each update we sample n e {min n length...., step length); by default Uni 16). Using fixed horizon
8

---

Published as a conference paper at ICLR 2026
Box Pushing Dease Boward
Box Puching Dease Reward
Box Pushing Dease Roward
1.0-
1.0
0.8-
0.8
0.8
0.6-
Syonne PMA
0.6
0.4
0.2
oreal Mol
0.2
0.2
0.0
0.0
0.0
0.0
0.5
1.0
15
2.0
0.0
0.5
1.0
15
2.0
0.0
0-4
0.3
1.2
3.6
Numbur Interaction
Number Eny Internction
Number Bov Interaction
(a) Transformer components: ef-
(b)
Multi-horizon window: (min_length, Jexghh) min 1, max 16; min 2, max 2).
super- varying (c.g..
(c) Policy-side action chunking (MLP critic): chunk length (1,2,3,5,8,16;
feet of removing ResNet blocks, vision the causal mask, and/or self- attention.
Pruhiig
Pruciing
Puching
1.0.
1.0
LiTh des
0.8.
Sugnnee raje
0.66
0.5.
0.4-
0.4
0.4
0-2
0.0
0.5
1:0
135
2.0
0.2
0.4
0.5
Number of Euv Intoraction
Number Eay Interaction
Number of Eay Intormatioon
(d) Policy-side action chunking (Transformer critic); chunk length (1,2, 4,5, 8, 163.
(e) Per-step target sampling: super- vision windows per environment step 2,,,,3,4 3, 4); training budget held constant.
(f) Critic backbone: Transformer vs. GRU and LSTM.
10M
Meen Rewarnds 1OM
4.5k
Los piace 10.3
4.5k
4.5%
3k-
3k-
Moun
1.kk
0.5k
Moun
0
0.0
0.4
U.5
1.2
3.5
2,0
0.0
0.6
0.5 L.2 1.5
0.0
0.2
0.4
11.6
5.5
1.0
Mamber Tntecnotion
Mamber Internction
(g) Effect of freeze interval K on Walker2d: i-SAC stable down K = 20, while very large K slows learning.
Effect of reusc factor (step_length) on lacomo-
(h)
(i) Robustness on Walker2d under noise (0, 0.05). false termina- tions (p = 0.1), and partial observ- ability (last half of observation re- moved).
tion: moderate values are robust; larger step- yields only mild gains.
Figure Ablations: transformer--ritic design and training settings. Within each panel. methods share the same interaction budget and differ only in the ablated component.
L smooths optimization but slightly reduces final performance (Fig. 4b), partly because the last
L-1 states of each segment never serve as starting indices an effect amplified for large L (e.g., 16). Despite standard guidance to keep n S 5 (Precup et al., 2000; Sutton et al., 1998; Espeholt et al., 2018), our Transformer critic with gradient-level averaging is stable up to n=16 and benefits from longer windows (Fig. 4b). An analogous sweep under the hard-copy scheme on Gymnasium Walker2d shows consistent trends (Fig. 4h).
Comparison to Multi-step MLP (Reinforcement Learning with Action Chunking). With 16M environment steps (vs. our default 20M), a multi-step MLP critic consistently underperforms the Transformer critic, and podecy-side chunking with an MLP yields only modest gains (Figs. 5, 4c). These policy-side baselines follow Q-Chunking (QC) (Li et al., 2025): we keep the action- -ettonnchuuking policy but drop the offline behavior-cloning constraint to match our online off-policy setting. In contrast, chunking the Transformer critic is consistently beneficial: the best configuration (chunk length 4) converges by x x10 steps, reaches 99.5% final success, and shows no late-stage divergence across seeds (Fig. 4d). Since the policy is identical across QC-style settings, comparing Fig. 4c to Fig. 4d isolates the advantage of a sequence-conditionn
critic under the same chunked policy. We include these results primarily to show compatibility between Transformer critic and policy-side chunking; policy chunking itself is not our focus.
This improvement is not explained by richer input features alone. Even with single-step temporal resolution (min length=max length=1), the Transformer critic in Fig. 4b still outperforms the chunki ng 1 length=1 baseline in Fig. 4c, suggesting the gain comes from integrating N-step

---

Published as a conference paper at ICLR 2026
returns and averaging gradients over short trajectories. The MLP critie also suffers causal leakage: because it consumes fixed-length segments, Q(s,, at) is effectively conditioned on (s,, ags+n)) ). i.c., it can "peek" at future actions Gt+1:t+n (Li et al., 2025). Our Transformer critic avoids this via causal mask (each token attends only to l' & 1) and outperforms an unmasked ablation (Fig. 4a). Finally, the MIP critic requires fixed-length inputs and couples the policy chunk length to the critic, whereas the Transformer critic removes both constraints.
Number of Samples Generated per Step. We vary the number of super- vision windows sampled per environment step ("per-step target samples"), For this study we use step_length conventional SAC (one target per step), T-SAC benefits from generating mul- tiple windows (Fig. 4e); our default is 4. In practice, use four vectorized envs or collect single trajectory of length 4xmax_length and slice it (App. H). Intuitively, multiple windows raise the share of fresh samples in each batch: with one window, once selected there are none left; with four, three remain.
Oct+o
[1. 8] (standard: [1, 16).. Unlike
MLP
Figure 5: Policy
GRU/LSTM as the Critic. We replace the Transformer critic with GRU
chunking (adapted from ei al. (2025)).
and LSTM variants under identical training (10M env steps; standard: 20M). Although recurrent crities model action sequences,
gradient-level av- eraging analysis ($ 4.1.1; App. D) does not directly apply, and parallelism is reduced (Fig. 12). Empirically, both GRU and LSTM underperform the Transformer critic Box-Pushing setting (Fig. 4f).
Robustness under noise and partial observability. We evaluate robust- ness to injected noise on actions and states, stochastic early termination, and partial observability via short observation windows. T-SAC degrades gracefully under action and state noise, retaining a clear performance mar- gin over SAC and CrossQ. Stochastic termination and partial observability lead to larger drops and higher variance, but T-SAC remains at least as sta- ble as these baselines, suggesting that sequence- conditioned critics help mitigate such effects.
Qboai
-
PC
FC
Figure 6: Structure of the recursive network used our experiments.
CONCLUSION AND FUTURE WORK
On Meta-World MLI multiphase and FANCYGYM box-pushing tasks, T- SAC with Transformer critic attains state-of-the-ard success rates under fixed training budgets (5M and 20M environment steps, respectively) and common evaluation protocol (success over 8 seeds; see $5). The sequence-conditee conditioned critic provides smoother value estimates and more coherent long- horizon credit assignment than both largely open-loop multiphase pipelines and standard step-based value methods, yielding higher-quality continuous control.
Our study is restricted online continuous control with low-dimensional observations. Extending T-SAC to discrete-action domains and to pixel-based or strongly partially observable settings (e.g., with visual belief-state encoders) high sensitivity to architectural and optimization choices. Applying T-SAC to real-robot tasks and developing theory for when critic-side chunking provably helps, including representation analyses and shared Transformer backbones for actor-critic, are important directions for future work.
cettrivaal-prelimmnne
experiments revealed instabilities and
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We made substantial efforts ensure reproducibility. The paper and appendix detail the soft- ware/hardware environment, evaluation protocols, and all hyperparameters used. We additionally release a public GitHub repository containing the full implementation of our proposed algorithms, along with training and evaluation scripts and configuration files: here. We also provide Weights & Biases dashboard with the complete set of test results, logs, and model artifacts: T-SAC test results. Finally, the appendix includes step-by-step description of the experimental setup, including exact configuration details, to facilitate independent reimplementation and verification.
ETHICS STATEMENT
No human participants were involved; all data were generated in simulation. Consequently, the work does not process personal or sensitive information. We took care to avoid introducing bias in our simulated setups, but acknowledge that broader fairness and safety considerations arise in real-world deployments of such methods. We are not aware of any security, dual-use, or legal concerns stem- ming from this work. The authors report no conflicts of interest external sponsorship influencing the results.
USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language models (LLMMs) as general- purppsse assistant during writing and develop- ment. Its roles included:
grammar and spell-checking, language polishing, and minor stylistic edits;
drafting and rewriting multi parli-paraggaph text (e.g., introductions, preliminaries, and parts of experimental write-ups) based on author- suthor-provided outlines and results;
high-level suggestions for debugging strategies and hyperparameter choices;
assistance with literature search (proposing search queries and surfacing candidate papers).
All BibTgX entries were copied from Google Scholar; the LLMs did not generate or edit biblio- graphic entries. The L.LMs did not originate the paper's main idea, problem formulation, algorith- mic design, or experimental plan, and it was not used to generate or alter data, results, or figures. All citations were selected and verified by the authors against the original sources. All LLMs outputs were reviewed and, when necessary, edited or discarded. No confidential or proprietary data were shared with the LLMs.
CONTRIBUTIONS
Dong Tian led the research and implementation, including the design of the Transformer Critic, theoretical development, manuscript writing, software development, and ongoing code maintenance.
Onur Celik provided code for the experimental pipeline (e.g., CrossQ, SAC, TD3), integrating Slurm-based execution via Hydra, and sharing practical insights on state-of-the-art methods (c.g.. SimbaV2, BroNet, DIME)
Gerhard Neumann initiated and supervised the project, provided mentorship and guidance in Re- inforcement Learning to the lead author, proposed adapting Transformer Crities from movement- primitive-based methods to SAC, secured computational resources. and provided comprehensive feedback on the manuscript and experimental evaluations (including MetaWorld and MuJoCo).
ACKNOWLEDGMENTS
We thank Dr. Ge Li and Hongyi Zhou for insightful discussions and for sharing testing data from other projects. We thank Xin Ye for guidance on scientific writing. We are grateful to Alcksandar Taranovie for organizational support, including help in addressing computational constraints dur- ing the rebuttal phase. D.T. thanks Heiko Junker, Dr. Muhammed Türker, and Kevin Sklubal for professional guidance during his internship.
Furthermore, the first author would like to express his sincere gratitude to Gerhard Neumann and Onur Celik for their professional mentorship and for setting standard of excellence in scientific
11

---

Published as a conference paper at ICLR 2026
rigor. We also thank the anonymous reviewers OpenReview for their constructive feedback and suggestions, which significantly improved this work.
The authors gratefully acknowledge the computing time made available to them the high- performance computer HoreKa at the NHR Center NHR @ KIT. This center is jointly supported by the Federal Ministry of Education and Research and the Ministry of Science, Research and the Arts of Baden- Württemberg, as part of the NHR joint funding program. O.C. and G.N. are supported by the European Research Council (ERC) under the European Union's Horizon Europe programme through the project SMARTI (Grant Agreement No. 101171393).
REFERENCES
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- fion processing systems, 0-29304-29320, 2021.
Kavosh Asadi and Michael L. Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, pp. 243-252. PMLR, 2017.
Kavosh Asadi, Yao Liu, Shoham Sabach, Ming Yin, and Rasool Fakoor: Learning the target network in funetion space. arXiv preprint arXiv:2406.018888, 2024.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06600 2016.
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. arXiv preprint arXiv: Kiiv:/900 1902.05605, 2019.
Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palenicek. Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. Dime: Diffusion-basee maximum entropy reinforcement learning. arXiv preprint arXiv:2502.0233 2025.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 1 15084-15097 2021.
Brett Daley, Martha White, and Marlos C Machado. Averaging n-step returns reduces variance in reinforcement learning. arXiv preprint vrXivv2402.03900 2024.
Ishan Durugkar and Peter Stone. Td learning with constrained gradients. 2018
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference machine learning, pp. 1407-1416. PMLR, 2018
Mattie Fellows, Matthew JA Smith, and Shimon Whiteson. Why target networks stabilise temporal difference methods. In International Conference on Machine Learning, pp. 9886-9909. PMLR, 2023.
Scott Fujimoto. Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811 048/1, 2024.
12

---

Published as a conference paper at ICLR 2026
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving ong-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. Pmlr, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:/812.05900 2018b.
Philippe -struch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idgl: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. In gi 1o/2022/03/ ICLR Blog Track, 2022. URL https:://sat http://ictrbblo-ttrckggiiiii tinckkkiittbbbb//000000225ppppp 25/ 25/ppo- imp empplmmmentiinn interatiiindnnddd...... implementation--etaalli.
Sergey Ioffe. Batch renormalization:
Towards reducing minibatch dependence in hathh-normalized models. Advances in neural information processing systems, 30, 2017.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances neural information processing systems, 34:1273-1286, 2021.
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Shivaram Kalyanakrishnan, Siddharth Aravindan, Vishwaject Bagdawat, Varun Bhatt, Harshith Goka, Archit Gupta, Kalpesh Krishna, and Vihari Piratla. An analysis of frame-skipping in rein- forcement learning. arXiv preprint 202.x::110203::
2021.
Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Deepmellow: removing the need for a target network in deep q-learning. In Proceedings of the twenty eighth international joint conference artificial intelligence, 2019.
George Konidaris, Scott Niekum, and Philip S Thomas. Td gamma: Re-evaluating complex backups in temporal difference learning. Advances in Neural Information Processing Systems, 24, 2011.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint 10.0v:2110.061699 2021.
Ge Li, Zeqi Jin, Michael Volpp, Fabian Otto, Rudolf Lioutikov, and Gerhard Neumann. Prodmp: A unified perspective dynamic and probabilistic movement primitives. IEEE Robotics and Automation Letters, 8(4):2325 2332, 2023.
Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, and Gerhard Neumann. Top-erl: Transeormer-based off off-policy episodic reinforcement learning, arXiv preprint arXiv:2410.09536, 2024a.
Ge Li, Hongyi Zhou, Dominik Roth, Serge Thilges, Fabian Otto, Rudolf Lioutikov, and Gerhard Neumann. Open the black box: Step-based policy updates for temporally correlated episodic reinforcement learning. arXiv preprint arXiv:2401.11433 2024b.
lemporally-correltted
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. arXiv preprint (it::2557.00969 2025.
13

---

Published as a conference paper at ICLR 2026
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg. Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In International Conference Machine Learning (ICML), 2018. URL https://
mrxiv.org/go httpp///11/21911111
Yuejiang Liu, Jubayer Ibn Hamid, Annie Xie, Yoonho Lee, Maximilian Du, and Chelsea Finn. Bidirectional decoding: Improving action chunking via closed-loop resampling. arXiv e-prints, pp. arXiv-2408, 2024.
Borislav Mavrin, Hengshuai Yao, and Linglong Kong. Deep reinforcement learning with decorrela- tion. arXiv preprint 17XXiv:903.077000 2019.
Safa Messaoud, Billel Mokeddem, Zhenghai Xue, Linsey Pang. Bo An, Haipeng Chen, and Sanjay Chawla. 2 ac: Energy-based reinforcement learning with stein soft actor critic. In The Twelfth International Conference Learning Representations.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PmLR, 2016.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off off-policy reinforcement learning. Advances neural information processing systems, 29, 2016.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.0913 2020.
Shota Ohnishi, Eiji Uchibe, Yotaro Yamaguchi, Kosuke Nakanishi, Yuji Yasui, and Shin Ishii. Con- strained deep q-learning gradually approaching ordinary q-learning. Frontiers in neuroroboties, 13:103, 2019.
Fabian Otto, Onur Celik, Dominik Roth, and Hongyi Zhou. Fancy gym. URL Utthttps://githbb com/ALRhub/ faec//aanccyyyye
Fabian Otto, Philipp Becker, Ngo Anh Vien, Hanna Carolin Ziesche, and Gerhard Neumann. Dif- ferentiable trust region layers for deep reinforcement learning. arXiv preprint arXiv:2101.99207, 2021.
Fabian Otto, Onur Celik, Hongyi Zhou, Hanna Ziesche, Vien Anh Ngo, and Gerhard Neumann. Deep black-box reinforcement learning with movement primitives. In Conference on Robot Learning. pp. 1244-1265. PMLR, 2023a.
Fabian Otto, Hongyi Zhou, Onur Celik, Ge Li, Rudolf Lioutikov, and Gerhard Neumann. Mp3: Movement primitive-based (re-) planning policy. arXiv preprint arXiv:2306. 12729, 2023b.
Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. Probabilistic move-
ment primitives. Advances in neural information processing systems, 26, 2013.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pp. 7487-7498. PMLR, 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint arXiv: 2019.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4):682-697, 2008.
Alexandre Piché, Valentin Thomas, Rafael Pardinas, Joseph Marino, Gian Maria Marconi, Christo- pher Pal, and Mohammad Emtiyaz Khan. Bridging the gap between target networks and func- tional regularization. arXiv preprint arXiv:2 02613, 2021.
Matthias Plappert. Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: 706.0/905, 2017.
14

---

Published as a conference paper at ICLR 2026
Doina Precup, Richard S Sutton, and Satinder Singh. Eligibility traces for off- off-policy policy evalua- tion. 2000.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http: jmmr..rrrooooo/mmmmmmmmm org patrr//appr///2/1
20- 1064.664.. html.
Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Conference robot learning, pp. 1634-1644. PMLR, 2022.
Thomas Rückstieß, Martin Felder, and Jürgen Schmidhuber. State-dependent exploration for policy gradient methods. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 234 249. Springer, 2008.
Thomas Rückstiess, Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, and Jürgen Schmidhuber. Exploring parameter space in reinforcement learning, Paladyn, 1(1):14-24, 2010.
Gavin A Rummery and Mahesan Niranjan. On-line Q-learning using connectionisl systems, vol- ume 37. University of Cambridge, Department of Engineering Cambridge, UK, 1994.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint (tl.:1506.02338 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. 1(1), 1998.
Mark Towers, Ariel Kwiatkowski, Jordan Terry John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.1700 2024.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double 4- learning. Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Théo Vincent, Yogesh Tripathi, Tim Faust, Yaniv Oren, Jan Peters, and Carlo D' D'Eramo. Bridging the performance gap between target-free and target-based based reinforcement learning with iterated q-learning, arXiv preprint arXiv:2506. 043398 2025.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):799-292, 1992.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning, pp. 1094-1100 PMLR, 2020.
Haichao Zhang, Wei Xu, and Haonan Yu. Generative planning for temporally coordinated explo- ration in reinforcement learning, arXiv preprint a7X1v:2201.09765, 2022.
Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. arXiv preprint arXiv: 101.0521,,, 2021.
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX: INDIVIDUAL META-WORLDY TESTS RESULTS
T-SAC (ours)
PPO
GTXXL(PPO) TOP-ERL
gSDE BBRL
SAC
CrossQ
TCP
Aggregated Results IQM
1.0
10M
0.8
0.6
0.4
0.2
0.0
0.5
1.0
2,0
Number of Env Interaction
Biniicking
SenClose
0.8
8.8
0.0
0.6
9.6-
SUCTEMNN rale
0.6
0.6
9.4-
Succerg
0.4-
0.4
0.4
0.0
0.0
0.0
Number Interaction
Number Intecuation
Number Interaction
Number Intersetion
attoonPrees
BuiteniironWWill
0.88
8.8
0.8
0.6-
0.8-
0.8
0.8
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.5
1.0
2.0
2.0
0.5
1.0
1.5
2.0
0'5
1.9
L.S.
2,0
Number
Coff:cButton
CoffcePull
CoffcePush
Dialaam
3.4
1.0
1.0
1.0
0.81
0.a
Sucooss tite D0M
0.4
02
0.0
0.0
0.0
0.5
1.0
1.5
2.0
0.0
05
1
155
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.3
1.5
1.5
2.0
Number
Interaction
Number Internation
Number
Interaction
Number Bin Intersstion
Disuseemble
DoarClose
Deorlack
1.0
1.0
1.0
1.0
Success oate 100M
0.6
SurTwA men
0.0
0.4
9.4
0.4
8.4
1
1.5
1.0
0.5
1.3
1.5
Number of Eav Interaction
Number of Tay Intertation
Number of Bas Intervction
Number
Bow Intersttion
DrawerOpen
1.0
1.0
1.0
0.6
0.6
0.6
Swosso Pade
0.6
0.4
0.4
0.4
0.4
1/0
Number Eav Interactios
Numberr Intacuction
Number
Interetioon
Number Bov Interection
Figure 7: Success Rate IQM of each individual Meta- World tasks. (Part 1)
16

---

Published as a conference paper at ICLR 2026
PauceiDpen
Thmmer
HandInoort
HondlePrees
1.0
1.0
1.0
1.0
1004
3.8
0.8
0.8
Syouug pate T0W
0.0
0.6
0.2
0.2
0.2
3.0
0.0
0.0
0.0
0.0
0.5
1.0
1.5
2,0
0.0
1.0
2,0
0.0
0.5
1
1,5
2.0
0.0
0.5
1.3
1.5
2,0
Number
Interaction
Number Interuciion
Number
Interection
Number
Interaction
Hundeauull
Handilel-lliiiiee
LeverPvll
1.0
1.0
1.0
1.0-
0.8
0.8
Succed cate BOME
0.88
0.4-
0.4
0.4
0.4
0.2
0.2
0.
3.22-
0.0
0.5
1.0
0.5
0.5
1.00
1.5
0.0
1.5
0.0
0.5
1.3
1.5
Number of Eav Interaction
Numberr or of Eay Internotion
Number of Eas Interection
Number of Eov Interoction
Pegtheplogtide
1.0
1.0
Subo0g rate T0M
8.6-
0.4-
0.4-
D.
6.4
K
0.5
1.5
1.5
Number of Eav Interaction
Number of Eay Interaction
Number Eas Interection
Number Ens Interaction
Wilk
8.8
8.8
Success rade TONE
0.6
0.6
0.8
0.6
0.6
0.4
0.4
0.4
Number Interaction
Number Intecnation
Number Interaction
Number Internetion
PlateSlideSlde
Push
PushBack
PushWAL
1.0
Sucosss rate So0
0..1
8.81
4.16
0.0
0.0
0.0
6.0
0.5
2.0
2.0
0.5
1.0
1.5
2.0
0'3
1.9
L.
2.9
Reach
BeochWall
ShalfPazee
Socoer
3.0
1.0
1.0
1.0
3.8
0.8
0.8
0.8
Surrawe man
0.6
Sucooss rate
0.6
9.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
13.%
2.0
0.0
0.5
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.6
8'5
1.3
1.5
2.0
Number Interection
Numbber
Intterutiion
Number
Interuction
Number Bnw Internatioon
StikPPitt
StokPussh
Sweeptnto
3.0
1.0-
1.0
1.0
Suousse pate T0ME
8.6
8.6
0.4
0.4
0.8
0.4
0.2
0.2
0.5
1.0
0.5
1.0
1.5
Number of Eav Interaction
Number of Eny Interuction
Numbber of Eas Interaction
Number of Env Internection
Sp0000o rate 10N4
0.8-
0.6-
8.6-
0.4-
0.44
Number of Eay Interaction
Numbee Eay Interaation
Figure 8: Success Rate IQM of each individual Meta- World tasks. (Part
17

---

Published as a conference paper at ICLR 2026
B APPENDIX: DETAILED ALGORITHM FLOWCHART
Algorithm 1: T-SAC
Initialize: Critic params Ø; target critic Orarget 0; policy params 0; temperature replay buffer B; environment reset - Sy.
Input: Segment length cap M; window length bounds (Emin: fmax); updates per iteration U; critic steps No: policy steps Np; soft update t; warmups: policy temperature.
repeat
Collect segment (length M);
store 80: / 0;
while < MM do
sample
TT(- st); step env - (Tt, St+1, de);
append (st, ae, re, de, 8t41) temporary buffer; if d, then break;
141+1; 88411
end
push the whole segment (5o:M, 40:M-1,70:M Mo:M-1) to B; d, then reset env >> 80:
else 8M8
Parameter updates; for u f to U do
sample a batch of segments {(50:M, 40:M-1 10:M-1, "o d1)M- from B; precompute segment-wise bootstrapped targets using Q & reuse across N₂ windows
for k to No do
for each sequence the batch, draw start i uniformly over valid indices and draw
{~ (-iimm m s.t. i+t segment end;
form windows ($as+6+1, agi+e Fisté, des+e) and the corresponding N-step returns;
update critic parameters & with the Transformer critic on these windows; critic update
Prarget <- 7 + (1 7) Oragett
D soft (or hard) target update
end
step > policy warmup then sample a (fresh) batch of states from B; for k to N, do
update policy 0 by maximizing the SAC objective using Q if step temperature warmup then update temperature o;
end end
until convergence;
18

---

Published as a conference paper at ICLR 2026
C APPENDIX: PROOF OF THE VARIANCE REDUCTION PROPERTY OF AVERAGING OF N-STEP RETURNS
convenient variance identity.
Under the equicorrelation model,
X[X m, Var(X;) v, Cov(XR, Xt) p (I: + t), p 0,
any weighted sum S = Ek=0 N-1 Xx satisfies
N-1
N-1 a
N-1 P
Var(S) = Σ
of
+
(10)
k=0
k==
This follows by expanding Var and collecting diagonal/off-diagone terms.
Reward part with discount < 1. triangular average by
Define the single N-step discounted reward sum and its
N-
i-1
N-1
RN(?)
Tki
Ry(7)
A
1
T
Σ
whTks
Wk
N
N
R=
i=1 k=0
k=
Let
N-1
1- 22
2N 7
N-1
1 1- 1 N
So = (N
&
Σ
2k
To = ToN, -
Σ
7
1 1-7 ?
k=0
k=0
Also define the (discounted, triangular) weight aggregates
N-1
2
Ay
W
(N 6N-
By
Σ
wk
Ay.
(11)
N2
Lemma (Variance formulas for the discounted reward part). Under the reward ussumptions stated in the setup,
Var( = 05 pTO - So)], Var Rx(1)(()) 0² -2 + B.).
Proof. Apply equation 10 with weights ag = n* for RN (7) and a = we for RN (7), and use the definitions of Sq, To, By.
Proposition 1 (Reward-side variance ratio and bounds). Define
Ry(N) A Var(Rx So + PTT So)
(ar(Rx(1)
Ay + +pB,
Then for all N 2 1, p 2 O. and E (0,
S RN(N) < 4.
Moreover, Mor R,(N) 7 4 as + xo; for any fixed y E (0, 1). (N) - 1. When > O, (N) is strictly increasing in for foroor 7 it need not be monotone.
Proof (sketch). The 7 proof carries through verbatim after replacing the unweighted triangular weights j by the discounted weights W4 = ((N k)/N)>* the same algebraic positivity arguments
yield the bounds. For the limits, when 7 < we have we + 7* pointwise as N - 3o and dominated convergence gives A, + So and Σx A wP + To. hence (N) + 1. For = 1, the standard triangular identities imply R (N) 4.
Bootstrap value part. assume
Let 2, A Vohan (8e4s) denote the (target) values used for bootstrapping and
E E[Zj
v4 Var(Z)) = Cov 72) xT² (1
2 0,
19

---

Published as a conference paper at ICLR 2026
and that {rk} and }Z,} are independent unless stated otherwise. The single N-step bootstrap term and its triangular average are
1 N
By(7) _ 7 ZN.
BN(7)
Z'z.
Let
A 72(1-92N S S
N
N
-11--/)
C= C=N,N) -
-- 7/
72
1
im1
Lemma (Variance of the averaged bootstrap part). With Cov 2n) = 7² for
1ori> $j.
72 N2
Var[BN(1)]=
S (c²
Proof. Apply equation 10 with a, = -
Proposition 2 (Bootstrap-side variance ratio, bounds, and condition). Define
RB(N, A Vr)(Bx(T)]
2222N
Si + x(2² -i)
Then for any [0, 1).
N2
N2
dRB
S (A(N,),)) I
< 0.
C
S
DK
In particular, averaging reduces bootstrap variance (RB 2 1) whenever
N2,2N 2N S1
K S N. (N, 2
C - Si
For the uncorrelated case (K 0), (N(,, 7, 0) N
Proof. Monotonicity in k immediate from the denominator. The bounds follow from S 81 + x Si) < C Solve RB for get K...
Putting the parts together. With Gx(1) = Rx(T) + Br(?) and Gx(r) = Rx(7) + BN(1), and assuming independence between rewards and bootstrap values,
Var[RN (7)] Var[BN(1) Var Gx(()) Var (7) Var Bx (1)]
Var[G>((7)]
(12)
Since all terms nonnegative,
Var[Gx(T)] Var[Gx(+1]
min R, Re(N, S
S max R,)(), RN,,,,,,))
Consequently:
Because (N) 2 (Prop. 1), RR 7, 2 (e.g., K Kx), then averaging N-step targets strictly reduces total variance.
Even if (2,,N,,, 7, R) < 1, the overall ratio in equation 12 remains 2 whenever the reward-side gain dominates:
Var[Bx(()] SI + xx -t)
(N( > Varl
N2
Dependence between rewards and bootstrap values. If Cov(RN(1) Bw(r)) and Cov( (T), (w)))) are nonzero, the numerator/denominator of equation 12 each acquire an additional covariance term. The sandwich bound above still applies after inserting these, and a crude control is [Cov(X, Y) I above conclusions unless the cross-covariances are pathologically large.
CCuuch-SShwaaze which cannot overturn the
20

---

Published as a conference paper at ICLR 2026
Useful closed forms.
Besides So. Tu, C' above, one has
Σ (N- -k) 1 (N + 1)7 + N k=
(1-
A closed form for =k=0 (N 2 (hence A, via equation 11) follows from the standard iden- tities for Σ kr and Σ 1²r after the change k N - k; we omit it as not needed for the bounds above.
N-1
Remarks. Setting 7 + recovers the undiscounted results: So, To. S1, - N and Ay, B.y A/N2, B/N², where A, B are the non-discounted triangle sums.
(ii) As N - 00 with fixed y < 1, S + >2 72/(1 7²) and C + 1/(11 - 7) while N222N + 0; thus RN(N, k) O. For typical RL regimes (7 Z 0.95, moderate N). k., (NV,) 7) is positive and large, so averaging still reduces bootstrap variance over a wide range of k.
(ii) R, (N) is horizon- and discount-agnostic in the sense of the bound < (N) < 4: for y 1, its large-N limit is 1. It is the principal driver of the overall variance reduction.
On the equicorrelation assumption. We assumed an equicorrelation (exchangeable) model for the reward noise and for the bootstrapped values: identical variances and common pairwise corre- lation (o and K, respectively). This is a standard device that yields closed forms while capturing the empirically relevant regime of positively correlated temporal signals in RL trajectories.
The key conclusions above are robust to relaxing equicorrelation. Let be any covariance matrix for (ru....,TN-1) with nonnegative entries (i.e., nonnegative autocovariances). For any nonnegative weight vector w, the variance is w Ew and increases monotonically with each off-diagonal entry. Since the triangular weights have strictly smaller & norm and smaller sum than the flat weights of the single N-step sum, the reward-side variance reduction persists under a wide range of stationary, positively correlated processes (including Toeplitz/lag-epende
models such Cov o² Pik-2 with Pa 2 0). The specific constant 4 in the upper bound is tight for the exchangeable model; with general lag structure the same [1, 4) bracket continues to hold under mild bounded- correlation conditions (e.g. supaye Corr(rk, re) S 1), while the uncorrelated case (pa- recovers the (1,3) limit.
For the bootstrap part, assuming a common correlation i: across {7:) } is likewise a tractable approx- imation: the explicit ratio R YN is decreasing in K, weaker dependence only strengthens the variance reduction. More general lag-dependent models Cov(Z,, 2,) = F with Ka 2 0 lead to the same qualitative behavior (smaller weights and partial averaging reduce variance), with our equicorrelation formulas serving as convenient upper/lower benchmarks.
When to be cautious. If the process exhibits strong negative or oscillatory correlations (e.g. al- ternation effects), equicorrelation overstates the benefit of averaging; in such cases, replacing the common p (or k) by a small set of lag-specific parameters (p1, .2....) is safer. Empirically, one can estimate the sample autocovariance and plug into T Ewe verify the inequalities numerically.
21

---

Published as a conference paper at ICLR 2026
D APPENDIX: VARIANCE REDUCTION FROM GRADIENT-LEVEL AVERAGING WITH A WWIGHTS TRANSFORMER CRITIC
Setup. Fix trajectory position t. A Transformer critic with shared parameters & outputs
(m)
where @ predicts the i-step return for the same prefix (st, -t---.,, (.tii-)) Let G denote the i-step target and define the per-horizon MSE
(((b) - L(t) 2
L.(8).
1=1
In implementation we average their gradients during backprop:
1 Σ ( G(1) O
V. L(v)
n
Local gradient factorization under weight sharing. rule.
Let be any scalar entry of th. By the chain
(:(w) A
aL, dw
(
(
G
aQ
Ow
20
For linear modules (affine maps in attention/FFN), the Jacobian has the standard local form aQ
aL g(0) (input activation x upstream error). Because the same w is shared across decoder positions, the sequence (w)),, are n gradient contributions for the same parameter, drawn from adjacent positions of one forward pass, and therefore generally positively correlated.
convenient covariance model (exchangebbleegiiorreltte,, For fixed w, we use the stan- dard homoscedastic equicorrelation approximation (also common in mini-batch analyses):
Varlgs(w) = oà:
(ov(g(w):: (w)) Pw oe
(i (),
Per [0,
This captures the empirically relevant regime where adjacent horizons produce positively correlated gradients and yields tight, closed-form variance expressions.
Lemma 3 (Variance of the averaged per- parameerr update). With the model above, the averaged
update ((w) 1 E
11 (w)) satisfies
1 1+ (n 1)p
Varlg(w)] n²
Varlgh] + Cov(((g 2
if1
In particular, Varlg(w)] < o for any Pw 1. Corollary 3 (Effective batch size and asymptotics). Define the effective sample size Nell (w) & 1 (1) Then Varlg(w)) /nat / (w)) with S Nett (wor(w) n, strictly increasing in n, and lom Varlg(w)] = Pro of (the inpelaation-iiposse variance floor). Proposition 4 (Uniform horizon averaging is optimal under exchangeability). Among all unbiased linear combinations Σ (ug((w) with Σ, 0z 1, the variance is minimized by the uniform weights o; whenever Cov(gi, 9j) ) is exchangeable (same diagonalloff (ingommmloff-dddgona)).
Proof. For exchangeable covariance Enp oz [(1 p.w) + pw. 11 Var Oaggi) a Enna minimized under ITa by a- 11.
Why Per 2 is natural. Both multiplicative factors of g:(w) vary smoothly with (i) the tar- gets G( share overlapping reward sums and a common bootstrapped tail; and (ii) the Jacobians _ v /Ow come from adjacent decoder positions of the same Transformer. This induces positive correlation among (w)) putting squarely in the regime of Lemma 3.
22

---

Published as a conference paper at ICLR 2026
Connection to target-side variance (discounted rewards and bootstrap). discount. Write the single N-step reward sum and its triangular average as
Let E (0, 1] be the
N
Ry(7)
Rx(c)
N
N
k=0
Under the equicorrelated reward model (mean μ4, variance o², pairwise corr. p 2 0),
Var War[RN(>>)] = [S0 + p (T -So]] Var Rar)Rx(1)] = o² + By]
with
So
N-1
2k
1 1-
To
N-1
Σ
7
k
1 -
A,
Σ k=0
Wki
2
By
Σ k=0
N-1
Wk
2
A
1
k= The reward-side ratio
k=0
Var[Ry (7)
So + p PT So))
(N(N)
Var Var[Ry (7)
A BB,
satisfies the uniform bound 1 S R2 < 4 for all N 2 1, 2 0, 7 E (0, 11. and R2(N) 7 4 as
N + DC.
For the bootstrapped values Z, V Oear (St+e) with Var(Z,) 7² and Cov (Zi, 2,) xT² (i
j, K E [0, 1)). define
1 N
By(1) = N ZN.
BN(?) = N
Σ 2 Zi;
and let 5 Disl? C' Ein 7" Then
,2 72 Var (B - [S + k(C² si)],
Var Var(By BN(BBN
N2g2N
RB(N, &
Var (ar[Bn(>))
S1 + k{C² - S1)
Rg is decreasing in and obeys
22,2N
N2g2N
S (B 7, <
C
S
In particular, averaging the bootstrap part reduces variance whenever N S1
S (N(N,)
C2-
Theorem 5 (Main: gradient averaging reduces update variance; compounded by target-side smooth- ing). Let w be any scalar parameter of the shared-weights Transformer critic and suppose (g are homoscedastic and equicorrelated with Pu. 1. Then
TaL
= o² + (n 1)P < <o = Var TaL
Var
Aj E (1..1....,,))
Ow
dw
Moreover, writing (w( = e00 (w) and (mildly) assuming {E(9) and (w)) are indepen- dent across i with bounded second moments, there exist constants ane byr (depending only on vb) such that
Var[g<(w]] S aw Var(G() + bw-
Consequently, replacing a single horizon by the triangularly averaged target across horizons 1:N reduces the reward-side variance by at least a factor reduces the bootstrup-side variance by factor Ra RA(N ()) Thus, in addition to the across- horizon gradient averaging gain 1+(n-1)p= the per-horizon variance term o itself decreases with N via tareet-side smoothing, yielding compounded reduction.
(1/4, 1), and (when K.) also
Practical notes. (i) The gradient-level algebra is agnostic to discount :: only the target-side con- stants (So. To, Ay. B,)) and (S), C)) change with 7. (ii) Under exchangeability, uniform averaging Berizon-weighte are needed for variance
across horizons variance-optimad (Prop. 4); no learned reasons. (iii) As grows, the residual variance floor is Pu o2. (Cor. 3); lower temporal correlation between parizon-gradient directly improves this floor. (iv) If horizon horizon-gradients are not perfectly
02
exchangeable, the bound Varlg(w)) Corr(gi, 93) Pm for all # j.
(1 (n P) still holds whenever Var(gi] o oz. and
n
23

---

Published as a conference paper at ICLR 2026
E
APPENDIX: CONNECTION TO MULTI-STEP TD THEORY
Motivation: Q mixes environment and policy. In an MDP M (S, 7) and policy x,
(s, = r(s, a)
+
B (s (s))) (*(s) = Banx(1s) (Q" (0' (s, a)].
env dynamics
dynamics policy via V
The one-step reward term is fully environmenntdeetrminnd (via (P,r)), while the bootstrap term depends both the environment and the continuation policy through V Our construction makes this separation explicit by defining critics that condition on a longer realized action prefix, thereby pushing all voliey-dependence to single boundary bootstrap term discounted by by
E.1 PREFIX-ONNDITIONEL (EXTENDED-ACTION) Q
Fix a horizon i 2 and write an action prefix as
00:-1 (0t, 0t+1,-.... (t+i-1) A';
Oct+i-1 = a0:i--
Define the prefix-conditioned action-value under continuation policy n by
Q7 a0:-11 = E
Prts + (²V*((8++) 8t, 0c:ti-11 A0:1-1
(13)
Conditioned on (st, a0:i-)). the random variables (reati-1, 814s) are distributed only according the environment dynamics induced by executing the prefix open-loop: the policy enters solely through the single bootstrap term V* ("(844)).
Environment/policy decomposition.
Let the environmentt-eterminnee prefix return be
17 (st; a0.i-1) = E
Prtti 8t, Oprti-1 a0:i-1
Then equation 13 becomes
Q7 (st, a0:-1) R.T (8), (st, a0.-1) + ((+4)) Oct+i-1 = ags-1] As grows, more of the return is captured by realized cenvironment-defines rewards, while the policy- deeendet component is confined to the boundary and attenuated by 7²
(14)
E.2 BELLMAN VIEW VIA AN AUGMENTED MDP (EXTENDED ACTION SPACE)
For fixed i, define an augmented MDP M: = AS, Po,ri, where an extended action ag:i-1 executed open-loop for steps:
(pss1)),nns-1) Pr(stti = 8
8 = 8, Octti-1 =
(15)
i-1
(s 10
rtt
8t = 8, O = a0:i-1
(16)
Then equation is exactly the one-step Bellman equation in M₁
Of (s. a) ra(s, a) + ( V V' (s')
(17)
Hence, learning Q from replayed prefixes is simply multi-step TD on an MDP whose actions are length-i open-loop prefixes.
E.3 WHY IMPORTANCE SAMPLING NEEDED (BY CONSTRUCTION)
Let be any behavior policy producing replay. Given sampled window/prefix
(st, an Peati-1, st-e), define the target (as Eq.(5) of the main text, using a target value net- work Vo))
Σ res + +
(18)
24

---

Published as a conference paper at ICLR 2026
Proposition 6 (Conditional unbiasedness; no dependence on pt). For any fixed (st, a0:i-1)
B
Pruss + (V(V(( 8t, Out-i-11 a0:i-1
Q (8): aus--))
In particular, this conditional expectation is determined by (P,r) and x (only through V*), and is independent of the behavior policy μ that generated the prefix.
Proof sketch. Conditioning on (st, d a0.i-1) fixes the executed actions. The induced
distribution of (fea+i-1; St+s) ) is therefore the environment distribution under open-loop execution of the prefix from St; y is irrelevant after conditioning. Taking expectations yields equation 13. C
Contrast with standard off-policy i-step evaluation of (T(s;, a:). If one instead targets the classical Q" (St, at) in the original MDP, then intermediate actions O++1:-ii-1 must be integrated under æ, while replay provides them under ja, creating distribution mismatch that typically quires per-decision importance sampling. Here, the intermediate actions are part of the conditioning variable (we learn Q (8t, a0;--1)), so there no such mismatch correct.
E.4 RELATION THE STANDARD STE- Q"
This definition is consistent with the classical action-value:
(s, a) = (s,
For > 1, the original Q" (s, a) is recovered by marginalizing the future actions under m:
Q (s, a) = Eota-1on S1
QT (s. (a, a1. (.))))........))) a.-))
(19)
Thus QT can be viewed refinement that conditions longer realized prefix, moving more of the return into environment-rcalized rewards and leaving only a discounted boundary bootstrap term to carry policy dependence.
25

---

Published as a conference paper at ICLR 2026
Box Pushing Sparse Reward
Box Pushing Sparse Reward
1.0
1.0
TSAC Standard Setting
Success rate IQM
Random target
Success rate IQM
Using Avaraging N-Stop Returns
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.4
0.8 1.2
1.6
2.0
0.0
0.2
0,4
0.6
0.8
1.0
Number of Env Interaction
Number of Env Interaction
Effect of target-value construction under sparse rewards. Randomly selecting one of two double-Q Q targets leads to high-variance updates and occasional collapse, whereas the conservative minimum-basse target yields stable learning curve with the hard- copy critic.
(b) Naively averaging N-step targets across horizons destabilizes learning and can erase progress, confirm- ing the need for gradient-level averaging.
Figure 9: Meta- Meta-World World Box Pushing (Sparse Reward), Ablations on (a) target-value construction and (b) return-propen propagationnnn schemes for T-SAC.
F APPENDIX: ADDITIONAL EXPERIMENTAL RESULTS
Post-figure summary. Figure 9a shows that the instability originally observed Box-Pushing- Sparse is explained by the high-variance target estimator: with the conservative min- imum-based target, hard-copy T-SAC is stable and reaches the best success rates. Figure 9b fur- ther illustrates that naive N-step target averaging can derail optimization, motivating our choice of rradient-averaged multi horizon losses. Seeds: 4. Results under IQM with 95% confidence intervals.
26

---

Published as a conference paper at ICLR 2026
Q(s. (s)..s))
( anzz2
Linear
Add & Norm
Feed
Forward
Add & Norm
xN
Masked Multi-Head
Attention
Positional Encoding
State Embedding
Action Embedding
St
at
a++1
a++2
at+n
Figure 10: T-SAC Critic Detailed Structure: causal Transformer short state-action segments. Given (St, at, ... Oc++a) ), the network produces n scalar outputs {Qu (St, aty..ry Oc++) 1a li=1- Colors and block styling follow the Transformer diagram conventions of Vaswani al. (2017).
G APPENDIX: TRANSFORMER CRITIC DETAILED STRUCTURE
Implementation details. We follow the TOP-ERL-Style Transformer critic design adopted in this work (see Li al. (2024a) for the schematic), i.e.. masked multi-heaid self eelf-attention stack with positional encodings and residual Add&Norm blocks; the critic ingests (st, a..... 4t+n) and jointly predicts all 1... n step returns. State and action tokens use separate one-layer linear embeddings (no bias), consistent with our training objective that conditions on realized action prefixes; the output head linear map without bias that emits one scalar per decoder position. No dropout used anywhere in the critic. The causal mask ensures each position only attends to S tokens, aligning network outputs with the i-step targets used for learning.
27

---

Published as a conference paper at ICLR 2026
States
SU
$:
8'
90
$:
S33
$4
80
8-2
83
80
Actions
On
@11
02
033
0p
0
0/2
03
ag
Q1
022
03
0g
@11
412
4,3
Action Mask
Rewords On
ro
72
To
122
1 1
0
0
Dones 0
0
0
0
0
0
0
0
O
0
0
(a) Fixed-Iength segments stored the replay buffer. Each sample is window (5L.44, (a
(b) Action mask for a sampled segment. The binary mask m marks valid action positions: my = 1 for steps before or at the first terminal in the window and my 0 after any d, terminal (including those filled from a new trajectory) are masked out so losses/targets and attention never cross episode houndaries.
If terminal d, appears before the window is full. we immediately continue saving from the start the next trajectory until the segment length is reached.
Entries that occur after
Figure 11: T-SAC segment construction and masking.
H APPENDIX: SEGMENT CONSTRUCTION AND THEORETICAL BENEFITS CRITIC CHUNKING
H.1 TRAJECTORIES SAVED IN REPLAY BUFFER AND ACTION MASK DESIGN
To efficiently process sequence data and maximize hardware utilization, T-SAC stores experience in the replay buffer as continuous, fixed-length segments rather than isolated single transitions. As il- lustrated Figure 11, typical stored segment consists of sequence of states, actions, rewards, and done flags: (Stst+L, I, J.-t+I--1, Tr:r+- f--, d..+..11).. ) In environments with variable-length episodes, a trajectory may terminate before the fixed segment capacity L is reached. Rather than padding the re- mainder of the buffer with whioh wastes memory and compute we immediately append the initial transitions of the subsequent episode the same segment until the length is satisfied. This strategy ensures constant tensor shapes, heavily optimizing batch processing for the Transformer critic.
However, packing multiple disjoint trajectories into a single contiguous segment introduces math- ematical risk: the critic might incorrectly bootstrap values or attend to states across episode bound- aries. To strictly prevent this, we implement straightforward binary action mask,
As detailed in Figure 11, the mask identifies the first terminal flag within sampled window. We set m, = for all timesteps up to and including the first terminal transition, and strictly enforce m, = 0 for all subsequent steps within that specific window. This action mask is applied when constructing the aariablehhoizon N-step targets, the mask zeroes out the TD error for any horizon that attempts to bootstrap across the boundary.
By doing so, any entries occurring after terminal state entirely ignored in the loss formulation. This masking acts as an implementation detail that resolves the mismatch between variable-length environments and fixed-length sequence models. It fully preserves the Markov properties of the MDP by preventing boundary leakage without degrading performance.
H.2 THE BENEFIT OF CHUNKING ON THE CRITIC SIDE
We analyze chunking under sparse rewards and state coverage.
Setup. Consider fixed segment of N transitions (81, 01, .,,..., 8841). A chunked sample is obtained by: (i) drawing a start index p uniformly from (1....,N N); ); (ii) drawing a window length
L uniformly from the integers {fmin: ... ) (with lmin 2 1). The window is truncated at the episode boundary, yielding an effective sequence length
h min I., N p 1}.
28

---

Published as a conference paper at ICLR 2026
For the fixed start state Sp. the Transformer critic predicts the ä-step returns for every horizon {I....,h} ,,. The multi-step target for horizon is:
GO Σ rpts + (Vp4p))))
(20)
Unlike standard multi-step methods that compute single return, T-SAC computes distinct tar- gets- with a different horizon and bootstrap
a single sampled start state 8p-
Connection state coverage (selection) probability. The probability that given transition is covered by the sampled window (i.e. E [p. + h 1--1) is
Pr(j covered)
NA max max(lmin: j - p 13 1
(21)
where A = mmax Emin + 1 and [++ max ,00 For the common case min 1 and writing
m Lmax, this simplifies
j6 111-1)
j
1 S m,
N
2m
Pr(j covered)
(22)
m
+
< N,
2N
i... ramp near the start followed by flat plateau. This higher coverage (vs. 1-step sampling) provides a denser learning signal across the trajectory.
Sparse rewards: how far does single reward propagate? Assume only the terminal transition N carries a non-zero reward (the sparse-reward setting). For sampled window starting at p with effective length h, the targets ( target if and only if the window reaches the end of the episode (i.e., p + h will be included in exactly one horizon's 's target (specifically, i = - 1-p-- + 1).
..... ((h) are computed. The terminal reward rx is included in N). Crucially, it
Therefore, the expected number of reward- beardd-bbbrnggggggg updates per sampled window is exactly the prob- ability that the sampled window covers the terminal transition:
Hupdates including terminal reward] = Pr(N covered).
(23)
Assuming N mmax, this is simply the plateau value of the coverage probability:
LLL
Hupdates including terminal reward]
(24)
N A
N
L=Emin
In the standard setting with Emin = 1:
mmax + 1
E Hupdates including terminal reward]
(25)
2N
representing an fold amplification (e.g., an 8.5-fold increase for Emax = 16) over uniform 1-step TD, which touches the terminal reward only when p N (a 1/N probability).
Takeaways. Chunking yields two critical critic-side benefits: (i) Denser state coverage: Instead of updating single state-action value per sample, each sampled window generates E ELL] distinct multi-horizon updates starting from 8p- This dramatically increases the coverage probability of states across the trajectory, accelerating value propagation.
(ii) Sparse-reward propagation: When only the last transition is rewarded, chunking increases the proportion of updates that incorporate the true reward by a factor of [E[.]. By naturally wrapping the terminal reward into the exact-horizon target, this mechanism explains why T-SAC excels in loriznorizon, sparse-reward settings without suffering from extreme off-policy variance.
29

---

Published as a conference paper at ICLR 2026
I APPENDIX: COMPUTATIONAL COSTS AND SAMPLE EFFICIENCY
Training time is reported for 1M environment steps (UTD= 1), unless otherwise stated. All benchmarks were run on an NVIDIA A100 (40 GB) GPU and an Intel Xeon Platinum 8368 CPU. Network architectures are config- ured as follows:
CroseQ
SAC
GRUCriti
6.2
LSTMChte
6.3
T-SAC (ours): MLP policy with two 256-unit hidden layers; Transformer critic with layers X 256 units.
Figure 12: Training time is reported for 1M environment steps (UTD= 1), unless other- wise stated. All benchmarks were run on an NVIDIA A100 (40 GB) GPU and Intel Xeon Platinum 8368 CPU.
GRU/LSTM: Same policy; 2-layer RNN critic (256 units).
SACC and CrossQ: Default configurations.
Table 1: Sample efficiency on long-horizon benchmarks, measured as the numher of environment steps (in millions) required to reach a fixed performance threshold on each task. Thresholds are defined as 90% of SAC's final return on Box- Pushing (dense) and MLL, and 90% of T-SAC's final return Box-P Box-Pushing (sparse).
Lower better. Values are means over seeds.
Task
SAC
CrossQ
GTrXL policy
TOP-ERL
T-SAC (ours)
Box-Pushing (dense)
15M
10M
20M
2M
4M
Box-Pushing (sparse)
N.A.
N.A.
N.A.
4M
17M
ML1
4M
N.A.
N.A.
4M
1M
Table Effect of minimum and maximum sequence length T-SAC performance and wall-clock training time on Box- nox-Pushing (dense). All runs use the same number environment steps (1M) for the standard setting.
min length
length
Return (mean + s.e.)
Wall-clock time
Peak GPU Mem (GB)
-78.45 6.89
2h35m03s
2.37
4
4
-66.63 + 3.09
2h39m23s
2.37
1
4
-74.80 = 7.69
2h38m56s
2.37
1
16
-65.500 055.054 + 0.20
3h06m58s
2.37
Table 3: Computational cost comparison for different methods Box-Pushing (dense) for fixed number of environment steps and matched (or explicitly stated) update-to-data (UTD) ratios.
Method
Params (M)
UTD
Wall-clock time (hours)
SAC
0.2
1
1.47
CrossQ
10
1
2.44
T-SAC (ours)
3,3
1/4
0.77
TOP-ERL
3.3
1/10
0.35
30

---

Published as a conference paper at ICLR 2026
Table 4: Performance fixed data budgets long- horizon tasks. Entries rate = standard error seeds after a given portion of environment steps. methods are trained with the same number of transitions.
Task
Method
25 %
50 %
100 %
Box-Pushing (dense)
SAC
1% 1%
5% 1%
12% 3%
(max 20M steps)
CrossQ
3% = 10%
10% + 20%
n.a. 10% 1%
GTrXL policy
1% = 1%
2% 1%
T-SAC (ours)
20% + 3%
80% + 3%
96% + 1%
Box-Pusian Box-Pushing (sparse)
SAC
0% = 1%
1% + 1%
1% + 1%
(max 20M steps)
Cross
0% = 0%
1% 1%
n.a. 1% + 1%
GTrXL policy
0% = 1%
1% + 1%
T-SAC (ours)
2% = 1%
15% + 10%
60% + 25%
ML1
SAC
53% + 7%
60% + 10%
68% + 10%
(max 5M steps)
CrossQ
50% 5%
50% + 5%
50% 5%
GTrXL policy
18% + 5%
35% + 5%
53% + 13%
T-SAC (ours)
68% 12%
78% 10%
86% + 5%
Train Tasks
Test Tasks
ossembly
basketbal
button press 1opdown
buttor press button press
button press
coffee outton
coffee pull
coffee push
bin picking
diai tur
discssemble
door diose
door open
drower olace
drower
foucet open
foucet close
hommer
box close
handle press
handle press
hondle pul
hondle pull
lever pull
Insert
Peg unplug
pick out of
pick place
door lock
pick place
plate slide
picte slide
plate slide
piate side
push bock
push
push wal
reach
apor ock
reach wall
shelf place
soccer
stick push
stick pull
Sleep Into
Sweep
Nindow
window close
hand Insert
Figure 13: Meta-Worlas tasks (Yu et al., 2020).
J EXPERIMENT DESCRIPTION
J.1 META- WORLD ML1
Meta-Wone (Yu et al., 2020) an open-source simulated benchmark for meta-reinffrreemntt learn- ing and multi-task learning in robotic manipulation. It comprises 50 distinct tasks spanning skills such as grasping, pushing, and object placement, each posing different perception-contee chal- lenges. By covering broader skill spectrum than narrowly scoped benchmarks, Meta-Worla is well-suited for evaluating algorithms that aim to generalize across diverse behaviors. Figure 13 enumerates all 50 tasks and illustrates their variety and difficulty.
Success criterion. To better approximate real-world deployment, we adopt a stringent evaluation rule: episode is counted successful only if the environment's success condition is satisfied the final timestep; intermediate achievements do not count toward success.
31

---

Published as a conference paper at ICLR 2026
J.2 Box PUSHING
Setup. A 7-DoF Franka Emika Panda arm with a rod pushes a box table target pose. At episode start, initial and target box poses are sampled with minimum 0.2 m separation:
I, E [0.3,000
y e 0.4],
[ 2m].
Success (for evaluation) position error I 0.05 m and orientation emoor v 0.5 rad.
Observations Actions. Observations: robot joint posi- tions/velocities (q. q), box position/orientation (p, r), and target (Purget: Flarget). Actions: joint torques a, E IR?.
Termination. Fixed horizon T' 100 steps; no early termination.
Figure 14: task (Otto el.).
Box
Pushing
Dense reward.
At each step,
Reoral = Ryod 0.02 - err(q. q) 350 R
position 200 R Ryotation
Subterms are
Ryod clip([1p
hppo 0.05, 10) clip( & arccos Mhor-hol 0.25, 2),
(26)
Te Σ (a))
(27)
i=1
err(q, Q)
Σ
(19:l -1901)
Σ
(lésl -a31)
(28)
4: 11:1111001
8: 1
R Reosition p Prungetll
(29)
Reutation
1 arccos|r- Trunggel
(30)
Here, pos the rod tip position, and fore hg are rod orientations (quaternions). Sparse reward.
Only the task terms applied the final step:
-Reeed Ryod - 0.02 T. crr(q, q): Ryod - 0.02 T; crr(q, q) 350 R position 200 Ryotation:
< T,
Riotal
(31)
t =
J.3 GYMNASIUM MuJoCo
We evaluate on the Gymnasium MuloCo v4 suite- Ant-v4, Hal fCheetah-v4, Hopper-v4, Walker2d-va and Humanoi ietooidttaan aSSaandu-vv (Fig. 15). We use the default observation and tion spaces and the native v4 reward shaping and termination rules (no reward normalization). Per- formance reported as undiscounted episode return. Unless noted otherwise, evaluation uses the deterministic policy over 152 episodes and aggregates results across multiple random seeds using IQM with 95% bootstrap confidence intervals; full hyperparameters and seeds are provided in App. L.
32

---

Published as a conference paper at ICLR 2026
(a) Ant
(b) HalfChectah
(c) Hopper
(d) Walker2d
(e) HumanoidStandup
Figure 15: MuJeCo (Towers et al., 2024) tasks used in our experiments.
K APPENDIX: ALGORITHM IMPLEMENTATIONS
Proximal Policy Optimization (PPO) (Schulman et al., 2017) is an on-policy, step-based method that constrains policy updates to remain close the behavior policy. Two variants are com- mon: PPO-Penaliy (KL regularization) and PPO-Clip (clipped surrogate). We evaluate PPO-Clip given its prevalence and robustness, following the reference implementation in Raffin et (2021).
PPO
Seeds: 20.
SAC
Soft Actor-Critie (SAC) (Haarnjj al., 2018;;b) is an off off-policy actor-critic with twin Qenetworks to mitigate overestimation and an entropy term to encourage exploration. We use the Bhatt al. (2019) implementation, which includes SAC. Seeds: 20 (Meta-World MI11), 5 (Gym MuJoCo).
TD3 Twin Delayed DDPG (TD3) (Fujimoto et al., 2018) addresses overestimation and instability via (i) clipped double Q-learning, (ii) delayed policy updates, and (iii) target policy smoothing. Our TD3 follows standard practice adapted from Raffin et al. (2021), including Polyak averaging and action noise for exploration. Seeds:
GTTXL Gated Transformer- XL (GTrXL) (Parisotto et al., 2020) stabilizes Transformer training for partially observable control. We build the PPO GTrXL implementation from Liang et al. (2018) and add minibatch advantage normalization plus a sate-inneeendent log- sog-sttnnardddevineee head following Huang et al. (2022). Seeds: 4.
gSDE Generalized State-Dependent Exploration (gSDE) (Raffin al., 2022; Rückstieß et al., 2008; Rückstiess et al., 2010) replaces i.i.d. Gaussian action noise with state dependent, temporally smooth exploration. Concretely, disturbances are generated as 6, Os, where the last hidden layer's activation and @ is resampled from Gaussian every steps according to the SDE sampling frequency. We evaluate gSDE with PPO using the reference implementation of Raffin et (2022); for stability on some tasks we employ linear schedule for the PPO clipping range. Seeds: 20.
Reinforcement Learning (BBRL) (Otto et al., 2023a;b) performs episodic, tra- jectory-level search by parameterizing policies with ProMPs (Paraschos et al., 2013). This han- dles sparse and MarkMarko rewards but can reduce sample efficiency. We consider both di-
BBRL
33

---

Published as a conference paper at ICLR 2026
agonal-covariance (BBRL-Std) and collrcoveee (BBRL-Cov) Gaussian policies, paired with ProDMP (Li et al., 2023). Seeds: 20.
TCP Temporally--oorelated Episodic RL (TCP) (Li al., 2024b) augments episodic policy up- dates with step-level signals, narrowing the gap between episodic and step-based RL while retaining smooth, parameter- space exploration. Seeds: 20.
TOP-ERI, Trajectory-Opiimiie Policy for Episodic RL (TOP-ERL) optimizes distribution over pootion-primitive parameters with (i) KL-constrained trust region and (ii) a temporally struc- tured covariance that induces smooth, correlated exploration across the episode. Our instantiation uses ProDMP (Li et al., 2023) as the trajectory generator; unless stated, we adopt an adaptive scale (entropy) schedule and per- dimensinno normalization of primitive parameters. Seeds: 8.
Cross CrossQ (Bbatt et al., 2019) is an off-policy SAC variant that removes target networks and applies BRN in the critic, enabling strong sample efficiency an update-to-data ratio of UTD 1. We follow the authors' reference implementation: a single batch norem-nomallzedd critic (no target networks), default temperature tuning, and recommended hyperparameters unless stated otherwise. Seeds: 4 (Meta-Work) MLI), 5 (Gym MuJoCo and Box-Pushing! Training Box- was capped at 10 M steps due to the experiment budget; by that point, wall-clock time exceeded 24 h.
34

---

Published as a conference paper at ICLR 2026
L APPENDIX: HYPERPARAMETERS OF THE ALGORITHMS
Baseline provenance. For BBRL, TCP, PPO, gSDE, GTrXL, TOP-ERL, and SAC on Meta-World MLI, report numbers from prior publications and/or official released runs/configurations under settings comparable to ours; we did not perform additional large-scale sweeps for these baselines in this paper (see citations in the main text and Appendix K).
Methods tuned in this work. We tuned SAC on Gym/FANCyGYMM, the full CrossQ implemen- tation, and TD3, including optimizer selection and hyperparameters (e.g., learning rates).
Trans- former-critic depth (number of attention layers), number of heads, dimensions per head, learn- ing rates (polieylcritic/a), supervision-window settings (fixed variable horizons; mi min_length step.length)) number of per-step target windows, and policy-side chunk length (for the com- patibility study). Where appropriate, we initialized choices from publicly reported configurations: Transformer hyperparameter ranges from TOP-ERI (Li et al., 2024a), the catropy-temperauur term from Celik et al. (2025), and the optimizer family from CrossQ (Bhatt et al., 2019). Final settings and search grids are listed in the appendix tables.
Our tuning for T-SAC. For T-SAC, conducted targeted grid search
35

---

Published as a conference paper at ICLR 2026
Table 5: Hyperparameters for the Meta-World experiments. Episode Length T 500
PPO
ESDE
GTTXI.
SAC
CroesQ
TCP
BARI.
TOP-ERI.
T-SAC
number samples
16000
19000
1000
GAE
0.95
0.95
0.95
n.D.
n.A.
0.95
n.a
n.A.
2..
discount fattor
0.99
0.99
0.99
0.99
0.99
1
1.0
0.99
"i
8.8.
0.a.
n.a
ex:
B.A.
24.
8.a.
n.8.
0.0005
0,0005
0.0005
n4.
trust region loss coef.
n...
n.a.
10
1.0
n.2
optimizer
adam
adamw
spochs
10
1000
100
15
20
learning rate
3e-4
le-3
2o-4
3e-4
3e-4
3o-4
Se-4
1o-3
2.5
use critic
True
True
True
True
Troe
True
Thae
Tre
True
spochs criti
10
1000
100
50
100
learning
3e-4
5e-5
number minibatches
n.a.
n.n.
n.a
n.a.
batch size
n.
500
1021
256
256
n.a.
n.a
256
512
butler size
a.a.
n.a
leb
1c6
n.a.
n.a.
3000
5000
125
learning stors
100000
5000
0
2
200
lemperature warmup
0
0
10000
polysk weiggtt
So-3
5o-3
SDE sampling frequency
n.d.
m.d.
n.a
n.A.
m..
n.d.
n..
n...
entropy coefficient
auto
2010
n.a.
auto
normalized observations
False
Fulse
False
False
normalized rewards
Palse
Palse
False
observation clip
10.0
244
m.a
n.a.
n.a.
n.a.
n.a.
n...
21.
reward clip
10.0
10.0
10.0
n.n.
n.a.
n.n.
n.n.
na.
n.a.
critic clip
0.2
Iin 0.3
10.0
n.n.
n.n.
n...
importance rañio clip
hidden layers
[128, 128]
[128, 128]
1a.1.
[256, 256]
[256, 256]
[128, 128)
[32, 32]
[ [128, 128]
1288, 128)
hidden layens critic
[128, 128)
[128, 128)
(256, 256]
[2048, 2048]
[128, 128)
[32, 32]
n.a
a.a.
hidden activation
lanh
Lanh
relu
relu
relu
rolu
leaky _rela
leaky .rolu
orihugomal insiaaiiation
fanin
1.0
number hoads
8
dims per head
16
16
32
nomber of attention layers
2
2
Tpsk-specific settings (Meta-World). For T-SAC, we initialize the policy's log standard devia- tion as log -5. The replay buffer stores 5,000 segments of length 125 (i.e., 5,000 125 625,000 transitions). The sampler retrieves 4 segments of length 125 (i.e., 4 125 500 transi- tions).
36

---

Published as a conference paper at ICLR 2026
Table 6: Hyperparameters for the Box Pushing Dense, Episode Length T
100
PPO
ESDE
GTXXI
SAC
Crossi)
TCP
BHRI.
TOP-ERI
T-SAAC
number samples
80000
8000
GAR A
0.95
0.95
0.95
0.a.
n.a.
0.95
n.a
n.a.
n.a.
discount factor
1.0
1.0
0.99
0.99
0.99
1.0
1.0
1.0
0.99
n.a.
n.a.
I.a.
0.a.
11.b:
0.05
0.1
0.005
0.a.
0.0005
0,00025
0,0005
loss coef.
1.0
optimizer
adam
adlmm
adlmm
adlmm
adam
adam
adam
adamm
udomw
epochs
10
15
learning
2.5e
use critic
True
Trme
Trme
Trme
True
True
True
Thee
Thue
epochs critie
10
10
5
1
50
10
30
100
learning critic
1o-4
1o-4
2o-1
3o-4
3e-4
1o-3
3e-4
So-5
2.5e-5
number minibstches
n.a.
batch size
n.a.
2000
1000
512
256
n.a.
n.a
512
buffer size
n.a.
n.a.
2e6
lef
n.a.
7000
20000 100
learning starts
0
0
le5
5000
8000
5000
temperature warmup
polyak_weight
n.a.
U.a.
n.a.
5o-3
1.0
n.a.
n.a.
Se-3
5o-3
SDB sampling frequency
n.a.
0.a.
m.a
0.9.
n.a.
n.a.
entrogr coefficient
0.01
0
20to
auto
0
normalized observations
True
Trme
Falke
Falke
False
True
Palse
Falke
False
normalized rewards
True
Thie
0.1
Falke
False
False
Falke
Falke
False
observation
10.0
clip
n.a.
n.a.
n.a.
critic clip
0.2
10.
n..
n.a.
n.a.
n.a
n.a.
n.a.
importance ratio clip
0.2
0.2
0.1
n.a
n.a.
n.a.
n.a.
n.a.
hidden layers
[512, 512]
[256, 2561
n..
[256, 256]
[256, 256]
[128, 128]
[128, 128]
[256, 256]
layers x 512] n.a.
hädden layers critic
[512, 512]
[256, 256]
n.a.
[256, 256]
[256, 156]
[256, 256]
[256, 256]
n.a.
hädden activation
tanh
tunh
relu
tanh
tanh
leaky
leaky
leaky_relu Yes
relu
orthogonal
fanin
fanin 1.0
initial
1.0
0.05
1.0
1.0
1.0
1.0
vumber of heads
4
8
dims bead
16
16
number attention layens
4
2
MP type
value
n.a.
n.a.
ProDMP
ProDMP
ProDMP
number basis functions
value
weight
value
n.a.
n.a.
goal scale
n.a.
n.a.
value
n.a.
n.a.
0.3
0.3
0.3
n..
37

---

Published as a conference paper at ICLR 2026
Table 7: Hyperparameters for the Box Pushing Sparse, Episode Length T 100
PPO
gSDE
GTXXI
SAC
CrosqQ
TCP
BRRI.
TOP-ERI.
T-SAC
number samples
9000
76
GAB
0.95
0.95
0.95
0.8
n...
0.95
n.a.
n..
F.B.
discount factor
1.0
1.0
1.0
0.99
0.99
1.0
1.0
1.0
1.0
1..0.
B.a
maa
0.3.
B.a.
0.005
<5
0.3.
n3.
n.8.
n.3.
B.8
0.0005
0.00025
0.0005
11.h
trust region loss coef.
0.4.
8..
n.8.
n.a.
P.8
10
1.0
n.a.
optimizer
epochs
50
20
learning rute
Se-4
le.4
2e-4
3e-4
3e
3e-4
3e-4
3e-4
2.5e-4
use critic
True
True
Trae
True
True
True
True
True
True
spochs critic
10
50
10
30
100
learning critic
le.4
le4
2e-4
3e-4
3e-4
3e-4
3e-4
Se-5
3.0e-4
n.a
n.a.
B.
n.a.
n.a.
n.
butch size
n.a.
2000
1000
512
512
na.
n.a.
512
buffer size
n.a.
n.a.
n.B.
2e6
206
n.a.
n.a.
7000
20000 100
learning starts
0
0
0
le5
1c5
0
0
100
2000
lomperature warmup
0
0
0
kolyakkwwighh
So-3
1.0
campling froquency
n.a.
n.a.
n.a.
8.a
n.a.
n.a.
n..
entropy coefficient
0.01
0
auto
auto
0
normalized observations
Time
True
False
False
False
Thue
False
False
False
False
observation clip
10.0
na
Falso
n.a
n.a.
n...
n.a.
na
1.S.
reward clip
10.0
10.0
10.0
n.a.
n.a
n.a.
n.a
na.
n.D.
critic clip
0.2
0.2
10.0
n.a.
n.a.
n.a
n..
importunce raiio clip
0.2
0.2
0.1
hiddlen lnyer
[122,
[256, 256]
n.l.
[256, 256]
[256, 256]
[128. 128]
[128, 128]
[256, 256]
[4 layers 512]
hädden layer critic
[512, 512]
[256, 256)
n.a.
[256, 256]
[2048, 2048]
[256, 256]
[256, 256]
n.a.
hiddenn activation
Ianh
tanb
rolu
tanh
rolu
leaky rrclu
leaky
leaky
leaky urllu
oethogonna intiiaiization
savice
Fanin
famin
initial std
number bocads
4
dims head
16
16
64
number of attention layers
2
type
11.0.
B.a
value
B.S.
B.A.
ProDMP
ProDMP
PreDMP
m.B.
number basis Functions
11.0
B.a
value
1.3.
n.a.
8
8
8
weight scale
0.a.
0.a
value
n.a.
n.A.
0.3
0.3
0.3
scale
value
0.3
0.3
0.3
38

---

Published as a conference paper at ICLR 2026
Table 8: Hyperparameters for the Gymnasium MuloCo, Episode Length T 1000
TD3
CrossQ
SAC
T-SAC (Soft Copy)
T-SAC (Hard Copy)
number samples
4 20
4 * 20
GAE À
discount factor
0.99
0.99
optimizer
adam
adam
adam
adamn
adamw
epochs
12
learning rate
3e-4
le-3
3e-4
3e-4
3e-4
use critic
True
True
True
True
True
epochs critic
60
60
learning rate critic
3e-4
le-3
3e-4
3e-4
3e-4
batch size
256
256
256
buffer size
le6
le6
le6
le5
le5 20
leamning starts
5000
5000
5000
10000
10000
temperature warmup
0
10000
10000
weihht
5e:3
1.0
5e-3
5e-3
1.0
entropy coefficient
hidden layers
[256, 256]
[256, 256]
[256, 256]
[256, 256]
[256. 256]
hidden layers critic
[256, 256]
[2048, 2048]
[256. 256]
hidden activation
relu
relu
relu
relu
orthogonal initialization
fanin
fanin
fanin
fanin
fanin
initial std
1.0
1.0
1.0
1.0
1.0
number of heads
4
4
dims per head number of attention layers
64
64
2
Tpek-ip settings (Gymnasium MuJoCo). For T-SAC, we use the same task-agnostic set- tings across all Gymnasium MuJoCo tasks: the initial policy log-standard deviation set to 55, and the target entropy is set to the SAC default H = - dim(A).
39


Reviews:
[{"review_id": "43HWg2dfRy", "reviewer": "Reviewer_qLQQ", "summary": "This paper proposes T-SAC (Transformer-based Soft Actor-Critic), an off-policy RL method whose critic is a lightweight causal Transformer trained on short trajectory segments with aggregated (N)-step returns. Unlike standard SAC, which evaluates single state–action pairs, T-SAC conditions the critic on action prefixes $(s_t, a_t,\\ldots,a_{t+n-1})$ and predicts prefix-conditioned values for multiple horizons without importance sampling. The method averages gradients across N-step targets and introduces a parameter-freezing schedule for the critic in locomotion tasks. Experiments on Meta-World ML1, Box-Pushing, and Gymnasium MuJoCo show improved sample efficiency, especially on long-horizon and sparse-reward tasks; e.g., 96.8% success on Box-Pushing (dense) and solving most ML1 tasks within ~5M interactions at UTD=1.", "questions": "1. Can T-SAC be evaluated on discrete-action domains (e.g., Atari) or more strongly partially observable settings?\n2. Do the benefits of sequence-conditioned critics carry over to image-based ML1 or partially observed Box-Pushing variants?\n3. How does T-SAC compare with Transformer-based policies and offline RL approaches (e.g., Decision Transformer)?\n4. For GRU/LSTM critics, how extensive was the hyperparameter and architecture search (depth, hidden size, normalization, teacher forcing, sequence length)?\n5. Have you tested robustness under observation/action noise, stochastic terminations, or mixed-policy replay buffers? This seems particularly relevant given the “off-policy without IS” choice.\n6. What aspects of the design (e.g., window sampling, bootstrap lag, prefix length) most limit performance in sparse-reward settings, and could targeted modifications (e.g., auxiliary returns, adaptive horizons) mitigate the observed instability?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The causal Transformer predicts prefix-conditioned values aligned with realized action prefixes, which plausibly improves temporal credit assignment. Using a “non-soft” critic with entropy regularization on the policy side preserves the standard SAC actor objective and simplifies the critic target, improving implementability.\n2. Gradient-level averaging is supported by Lemma 3 and Theorem 5, establishing $\\mathrm{Var}[\\nabla_\\psi \\bar L] < \\sigma_w^2$ under equicorrelation assumptions. The paper analyzes target-side variance reduction for both reward and bootstrap components, providing bounds $1 \\le R_\\gamma(N) < 4$ and conditions for reducing bootstrap variance. The IS-free formulation avoids high-variance importance ratios by conditioning on realized prefixes and is clearly derived from the N-step return objective.\n3. The evaluation spans 57 tasks across Meta-World ML1, Gymnasium MuJoCo, and Box-Pushing. Gains are pronounced on difficult tasks (e.g., 96.8% on Box-Pushing (dense), outperforming step-based baselines ≤85%) and on multi-phase ML1 tasks such as Assembly, Disassemble, Hammer, and Stick-Pull.", "weaknesses": "1. The core idea—Transformer critic trained with N-step returns—resembles TOP-ERL; the manuscript acknowledges this but does not sufficiently sharpen the technical distinctions. The claim of “bridging step-based and episodic regimes” appears overstated: the policy remains step-based while sequence conditioning is confined to the critic. Several components (causal Transformer, parameter freezing, N-step averaging) draw on prior art, but the paper does not isolate which design choices constitute the principal contribution.\n2. Comparisons with traditional off-policy N-step methods using IS are empirical only; formal connections to prior analyses are not developed.\n3. Reported instabilities on Box-Pushing-Sparse lead to a fallback on soft targets, limiting generality. The action-mask requirement and performance variability on Ant/Hopper/Walker2d suggest fragility and sensitivity to dynamics/constraints.\n4. T-SAC trails CrossQ on Hopper and Walker2d; the explanation “action mask needed may degrade performance” lacks systematic investigation. On Box-Pushing-Sparse, success is 58%, below TOP-ERL’s 70%; the claim of being “competitive under sparse feedback” does not reconcile this gap.\n5. No ablations on freeze length $K$ or reuse factor $N_c$ are provided to delineate stability regions.\n6. Equation (1) defines $G^{(n)}$ with $V_\\phi(s_{t+n})$ but does not clarify whether this is a soft or standard value; later (Sec. 4.3) it states the critic estimates the standard (non-soft) action-value. The notation should reflect this distinction.\n7. All benchmarks are simulated continuous-control tasks; the paper does not evaluate discrete action spaces, stronger partial observability (beyond action history), or real-robot settings that motivate the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4WGmdU1kzC", "reviewer": "Reviewer_m1ZH", "summary": "This paper addresses the challenge of long-horizon credit assignment in standard step-based reinforcement learning. It introduces T-SAC, an algorithm that replaces the standard MLP critic in Soft Actor-Critic (SAC) with a lightweight Transformer. This new critic conditions on short sequences of actions from the replay buffer and is trained on N-step returns across multiple horizons without requiring importance sampling, using a gradient averaging scheme to ensure stability. On long-horizon and sparse-reward control benchmarks like Meta-World ML1 and Box-Pushing, T-SAC is shown to achieve superior sample efficiency and final performance compared to strong off-policy baselines.", "questions": "1, Could you please elaborate on the specific role and necessity of the \"action mask\" mentioned as the cause for performance degradation on \"high trajectory variability\" tasks like Hopper and Walker2d ? Experimental result(e.g. with action mask vs without action mask) would be helpful to understand your claim.\n\n2, The paper's core mechanism is averaging gradients rather than targets. The result shows that naive target averaging fails in sparse-reward settings (Fig. 10b, App. E). Could you provide more intuition as to why averaging gradients is more effective at preserving the sparse reward signal?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1, T-SAC significantly outperforms strong off-policy baselines such as SAC and CrossQ on the Meta-World ML1 benchmark and the FANCYGYM Box-Pushing tasks. Its 58% success rate on the sparse-reward Box-Pushing variant is particularly notable, clearly demonstrating the effectiveness of its long-horizon reasoning capabilities.\n\n2, While N-step returns are known to suffer from high variance as n increases, the proposed gradient averaging technique enables stable training for horizons as long as n=16. This challenges the conventional wisdom of using small n and provides a robust method for learning long-term dependencies.\n\n3, The paper provides a clear justification for its architectural choices through rigorous ablation studies. It demonstrates the critical importance of the self-attention and causal mask components in the Transformer critic and shows its superiority over recurrent backbones like GRU and LSTM.", "weaknesses": "1, The performance of T-SAC is not uniform across all benchmarks. On Gymnasium MuJoCo, it performs worse than the standard SAC baseline on Hopper and Walker2d.The authors attribute this to the need for an \"action mask\" on tasks with \"high trajectory variability\", which points to a limitation in the method's generality.\n\n2, The critic conditions on multi-step action sequences (at, ..., at+n-1) generated by the current, often random, policy. Early in training, these sequences are noisy and suboptimal, potentially introducing significant variance into the critic's learning target. This may slow down the initial convergence speed compared to standard critics that condition only on a single (s, a) pair. This effect could be particularly pronounced in dense-reward settings like mujoco, where the benefit of a clean, immediate one-step TD target is high.\n\n3, The proposed critic-parameter freezing schedule (Sec 4.4), which enables training without a target network, is only demonstrated on locomotion tasks. The authors admit it is unstable in sparse-reward settings, where the paper's main results are achieved using conventional Polyak averaging. This makes the target-free contribution a secondary point with limited applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1xVfRLBtQV", "reviewer": "Reviewer_2mBX", "summary": "This paper proposes T-SAC (Transformer-based Soft Actor-Critic), which enhances standard SAC by introducing a sequence-conditioned critic that models short trajectory segments via a lightweight Transformer and trains with aggregated N-step returns—without importance sampling. The approach enables long-horizon credit assignment while keeping the actor update one-step. A simple critic-freezing schedule further removes the need for target networks. Experiments on various benchmarks show consistent improvements over SAC, CrossQ, and other baselines, demonstrating that sequence modeling on the critic side can improve sample efficiency and stability in long-horizon control.", "questions": "1. How sensitive is T-SAC to the choice of N-step horizon and freezing interval?\n\n2. Would the approach extend to visual or partially observable environments?\n\n3. Could actor and critic share a Transformer backbone for further efficiency?\n\n4. How does T-SAC compare computationally to CrossQ and TOP-ERL at scale?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of chunking the critic rather than the actor is novel and conceptually clean, offering a new perspective on temporal abstraction in off-policy RL.\n\n- The removal of importance sampling through prefix-conditioned targets is a practical and effective simplification.\n\n- The paper combines Transformer critics, multi-horizon learning, and critic freezing in a coherent design that improves stability without target networks.\n\n- Experiments are broad and rigorous, with thorough ablations verifying each design component.\n\n- Writing is clear, structured, and technically sound.", "weaknesses": "- The novelty is largely architectural; theoretical justification is limited, and the benefits of critic-side chunking lack deeper analysis.\n\n- All results use low-dimensional state inputs; no experiments on visual or partially observable tasks.\n\n- Statistical significance of gains and computational cost comparisons are not deeply analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fxyQ1Uc3AU", "reviewer": "Reviewer_9zeX", "summary": "This article extends the work in the literature [1] by introducing a sequence-conditioned critic for SAC that models trajectory context with a light-weight Transformer and trains on aggregated N-step returns. Empirical results on standard online RL benchmarks show that the proposed method sometimes outperforms other algorithms.\n\n\n[1]Qiyang Li, Zhiyuan Zhou, and Sergey Levine.Reinforcement learning with action chunking.arXiv  preprintarXiv:2507.07969,2025.", "questions": "1. What is the subset of MDPs that have the property that the Transformer-based action chunking can be provably useful?\nThis is an important question. As you can see, the proposed method does not perform well in Mujoco.\n2. Could you discuss the computational cost and scalability of the proposed method in detail, especially as the sequence length increases? \n3. Could you provide a more detailed analysis of how your Transformer-based action chunk gain was observed in your experiments? In particular, in addition to the comparison of rewards in the paper, is there an experiment that more intuitively demonstrates the significant improvement in sample efficiency?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is overall well-organized and easy to follow. \n2. The proposed transformer critic is intuitive and is supported by several theoretical results.", "weaknesses": "1. The improvement of the algorithm in the experiments is not significant for a basic benchmark, Mujoco.\nThe authors claim that an action mask is needed for T–SAC and may degrade performance with high trajectory variability.\nT-SAC is better than other methods in Meta-World tasks, however, there is higher trajectory variability in Meta-World tasks than that in Mujoco.\n2. The authors of literature [1] claim that we might desire a final optimal Markovian policy, the exploration problem can be better tackled with non-Markovian and temporally extended skills, and that action chunking offers a very simple and convenient recipe for obtaining this. \n*However,* they did not provide any theoretical explanation for the aforementioned phenomenon. They also did not offer any intuitive examples to demonstrate the effectiveness of action chunking in MDP tasks. The same problems are shown in this paper.\n3. I think the major contribution of this work is the proposed causal Transformer. Thus, in addition to the performance gain by inserting the causal Transformer into SAC, it would be good to discuss other metrics that can directly evaluate the \"quality\" or \"informativeness\" of representations learned in the causal Transformer. Such metrics might include Centered Kernel Alignment and Mutual Information Neural Estimation.\n4. The contribution is ambiguous. An alternative interpretation of the benefit from action chunking and T-SAC is that the model is simply given a richer input feature set, which allows a powerful function approximator (like a deep neural network) to better fit a complex value or policy function. \n5. The new method should be compared with QC, which is proposed in [1].\n\n\n*If my concerns and questions are all addressed, I will raise the score.*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)