Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns

## Abstract

Abstract We introduce a sequence-conditioned critic for Soft Actor–Critic (SAC) that models trajectory context with a lightweight Transformer and trains on aggregated N N -step targets. Unlike prior approaches that (i) score state–action pairs in isolation or (ii) rely on actor-side action chunking to handle long horizons, our method strengthens the critic itself by conditioning on short trajectory segments and integrating multi-step returns—without importance sampling (IS). The resulting sequence-aware value estimates capture the critical temporal structure for extended-horizon and sparse-reward problems. On local-motion benchmarks, we further show that freezing critic parameters for several steps makes our update compatible with CrossQ’s core idea, enabling stable training without a target network. Despite its simplicity—a 2-layer Transformer with 128 128 – 256 256 hidden units and a maximum update-to-data ratio (UTD) of 1 1 —the approach consistently outperforms standard SAC and strong off-policy baselines, with particularly large gains on long-trajectory control. These results highlight the value of sequence modeling and N N -step bootstrapping on the critic side for long-horizon reinforcement learning.

## 1 Introduction

Off–policy actor–critic methods are the workhorses of continuous control. SAC (Haarnoja et al., 2018a) is notable for sample efficiency and stability, driven by tightly controlled bootstrap targets and mechanisms that curb overestimation. Recent work shows that similar stability can be achieved by slowing critic updates or regularizing its function class (Vincent et al., 2025; Piché et al., 2021; Gallici et al., 2024). In this vein, CrossQ (Bhatt et al., 2019) removes target networks by pairing Batch Renormalization (BRN) (Ioffe, 2017) with bounded activations, yielding strong locomotion results.

A complementary line of work uses *temporally extended actions* for temporal abstraction. Movement primitives and action chunking can aid exploration and produce smoother trajectories than purely step–based control (Otto et al., 2023a; Li et al., 2024b; Zhang et al., 2022). However, methods that jointly predict actions and repetition horizons (Sharma et al., 2017) or replace the policy with recurrent sequence models (Zhang et al., 2022) have not consistently produced robust gains.

Accurate value estimation remains difficult in sparse-reward, long-horizon, high-dimensional settings. Multi-step targets ($N$-step returns) (Sutton et al., 1998) can reduce bias at the cost of higher variance and typically require off-policy IS (Precup et al., 2000), which complicates—and can destabilize—training (Munos et al., 2016).

Transformers have become strong backbones for RL: self-attention supports sequence modeling, long-range credit assignment, and flexible conditioning on history (Vaswani et al., 2017; Parisotto et al., 2020; Agarwal et al., 2023). They parameterize policies and critics, serve as world models, and enable offline RL via trajectory/return conditioning (Chen et al., 2022; 2021; Janner et al., 2021).

##### Transformer critics for episodic RL.

TOP-ERL (Li et al., 2024a) is an early successful Transformer-critic approach, but it targets episodic control: the policy outputs full trajectories via ProDMPs (Li et al., 2023), and the critic helps evaluate whole trajectories. While replanning is discussed, it is not implemented in released experiments. The method additionally relies on a Trust Region Projection Layer (TRPL) (Otto et al., 2021), typically uses $\sim$20M interactions, and underperforms on multi-phase Meta-World tasks (Yu et al., 2020) (e.g., *Assembly*, *Disassemble*) as well as Box–Pushing at tight tolerances (Otto et al.,).

##### This paper: Transformer-based Soft Actor-Critic (T-SAC).

We introduce T-SAC, a *step-based* Soft Actor–Critic whose Transformer critic is trained on *short trajectory segments*. By *chunking* temporal structure inside the critic—attending over brief state–action windows—we enable long-horizon credit assignment while keeping policy updates strictly one step (no trajectory-level outputs or replanning). The method preserves SAC-style stability and is sample efficient, solving most Meta-World tasks in $\sim$5M interactions. Training remains stable at low update-to-data (UTD) ratios (Meta-World $=1$, Gymnasium MuJoCo $=0.75$, Box–Pushing $=0.25$) and reaches $96.8\%$ success on Box–Pushing (dense). With $1$M interactions at UTD$=1$, runs typically complete in $\sim$3 hours. Our critic follows TOP-ERL’s Transformer design in spirit (causal masking, lightweight depth) but is derived from an $N$-step-return objective and optimized for step-based control—*without* IS. On locomotion benchmarks (e.g., Gymnasium MuJoCo), a lightweight critic parameter-freezing schedule enables stable training *without* target networks (like Deep Q-Learning (Van Hasselt et al., 2016)), outperforming Polyak averaging.

## 2 Related Work

### 2.1 Transformer-based Critics for Episodic RL

Episodic RL (ERL) replaces per-step actions with trajectory-level primitives (Otto et al., 2023a; Li et al., 2024b), improving long-horizon reasoning but complicating temporal credit assignment—especially off-policy. TOP-ERL (Li et al., 2024a) partitions each episode into fixed-length segments and trains a Transformer critic that attends across segments. Using truncated $N$-step targets (Sutton et al., 1998), the critic predicts per-segment returns, enabling off-policy replay while leveraging attention to handle partial observability and long-range dependencies. In diverse manipulation tasks, TOP-ERL outperforms other ERL-based, step-based or on-policy value-based RL baselines (Li et al., 2024a).

### 2.2 Training without Target Network

Target networks stabilize bootstrapped critics but slow value propagation and add complexity (Kim et al., 2019; Piché et al., 2021). Recent work instead limits target drift or smooths backups. The strongest result, CrossQ, achieves state-of-the-art (SOTA) sample efficiency in continuous control by removing the target network and stabilizes a single bootstrapped critic with BRN (Ioffe, 2017). Related target-free strategies include value smoothing (mellowmax) (Asadi & Littman, 2017; Kim et al., 2019), constrained/proximal updates (Durugkar & Stone, 2018; Ohnishi et al., 2019), function-space regularization and partial freezing (Piché et al., 2021; Asadi et al., 2024; Vincent et al., 2025) feature decorrelation (Mavrin et al., 2019). Theory unifies these mechanisms as alternatives to target networks via partial freezing, regularization, and separation of optimization dynamics (Fellows et al., 2023).

### 2.3 Action Chunking in Reinforcement Learning

Action chunking replaces per-step control with short open-loop sequences of actions (“chunks”), which can capture temporal structure, accelerate value propagation via longer effective horizons, and promote temporally coherent exploration (Kalyanakrishnan et al., 2021; Zhang et al., 2022). The trade-off is reduced reactivity within a chunk (Liu et al., 2024), but for long-horizon, sparse-reward manipulation this bias often pays off (Zhang et al., 2021; Gupta et al., 2019).

Reinforcement Learning with Action Chunking (Q-chunking) (Li et al., 2025) applies TD-based actor–critic learning directly in the chunked action space: the policy proposes an $H$-step action sequence and the critic evaluates $Q(s_{t},a_{t:t+H-1})$, enabling unbiased $H$-step backups and efficient updates (Li et al., 2025). In their implementation, the critic is a simple MLP that ingests the state concatenated with the proposed action chunk (rather than a sequence model), which keeps the method lightweight while still reaping the benefits of temporally extended actions (Li et al., 2025).

## 3 Preliminaries

##### Off-Policy Reinforcement Learning.

Reinforcement learning (RL) (Sutton et al., 1998) formalizes sequential decision making as a Markov decision process (MDP) $(\mathcal{S},\mathcal{A},P,r,\gamma)$: at time $t$ an agent observes $s_{t}$, selects $a_{t}\sim\pi(\cdot\mid s_{t})$, receives $r_{t}=r(s_{t},a_{t})$, and transitions to $s_{t+1}\sim P(\cdot\mid s_{t},a_{t})$; the goal is to learn a policy maximizing $J(\pi)=\mathbb{E}_{\pi,P}\!\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\right]$ using value functions $V^{\pi}(s)$ and $Q^{\pi}(s,a)$ that satisfy Bellman consistency, with $Q^{\star}$ inducing the optimal policy. Algorithms differ in how they estimate and improve these quantities—value-based learning (Watkins & Dayan, 1992; Hessel et al., 2018; Van Hasselt et al., 2016; Rummery & Niranjan, 1994), actor–critic (Mnih et al., 2016; Schulman et al., 2017; 2015a; Fujimoto et al., 2018), or direct policy optimization (Kakade, 2001; Peters & Schaal, 2008)—while managing exploration vs. exploitation (Sutton et al., 1998). Off-policy RL learns a target policy $\pi$ from data generated by a (possibly different) behavior policy $\mu$, reusing transitions $(s,a,r,s^{\prime})$ via replay buffers and bootstrapped Bellman updates; distribution mismatch when evaluating $\pi$ from $\mu$-data can be corrected (e.g., with IS (Sutton et al., 1998)). This decoupling enables efficient experience reuse and underpins methods like Q-learning (Watkins & Dayan, 1992) and the SAC (Haarnoja et al., 2018a) family.

##### Soft Actor-Critic

Let $\pi_{\theta}(a\mid s)$ be a stochastic policy with parameters $\theta$.
Let $Q_{\psi}(s,a)$ be the critic with parameters $\psi$, and let $Q_{\phi}$ be its
target network (e.g., a Polyak-averaged copy of $Q_{\psi}$).
SAC (Haarnoja et al., 2018a) maximizes a maximum-entropy objective to improve robustness and exploration:

$$ $J(\pi_{\theta})=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma^{t}\big(r_{t}+\alpha\,\mathcal{H}(\pi_{\theta}(\cdot\mid s_{t}))\big)\Big].$ $$

where $\gamma$ is the discount factor.

The Bellman target is

$$ $y_{t}=r_{t}+\gamma\,\mathbb{E}_{a^{\prime}\sim\pi_{\theta}(\cdot\mid s_{t+1})}\!\big[\,Q_{\phi}(s_{t+1},a^{\prime})-\alpha\log\pi_{\theta}(a^{\prime}\mid s_{t+1})\,\big],$ $$

and the critic minimizes the squared error

$$ $J_{Q}(\psi)=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\!\Big[\tfrac{1}{2}\big(Q_{\psi}(s_{t},a_{t})-y_{t}\big)^{2}\Big].$ $$

The actor minimizes

$$ $J_{\pi}(\theta)=\mathbb{E}_{s\sim\mathcal{D},\;a\sim\pi_{\theta}(\cdot\mid s)}\!\big[\alpha\log\pi_{\theta}(a\mid s)-Q_{\psi}(s,a)\big].$ $$

The temperature $\alpha$ is tuned to match a target entropy $\bar{\mathcal{H}}$ by minimizing

$$ $J(\alpha)=\mathbb{E}_{s\sim\mathcal{D},\;a\sim\pi_{\theta}(\cdot\mid s)}\!\big[-\alpha\big(\log\pi_{\theta}(a\mid s)+\bar{\mathcal{H}}\big)\big]\quad\text{~\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{haarnoja2018asoft}{\@@citephrase{, }}{})}}.$ $$

##### N-step Returns and IS

Using the same notation as above, on-policy N-step targets speed up credit assignment (Sutton et al., 1998; Schulman et al., 2015b; Mnih et al., 2016), i.e.,

$$ $y_{t,\text{soft}}^{(n)}=\sum_{k=0}^{n-1}\gamma^{k}\,\mathbb{E}_{a_{t+k}\sim\pi_{\theta}}\!\big[r_{t+k}-\alpha\log\pi_{\theta}(a_{t+k}\!\mid s_{t+k})\big]+\gamma^{n}\,\mathbb{E}_{a\sim\pi_{\theta}(\cdot\mid s_{t+n})}\big[Q_{\phi}(s_{t+n},a)\big].$ $$

With off-policy data drawn from a behavior policy $\mu\neq\pi_{\theta}$, per-decision importance ratios

$$ $\rho_{t+k}=\frac{\pi_{\theta}(a_{t+k}\mid s_{t+k})}{\mu(a_{t+k}\mid s_{t+k})}$ $$

can be used to correct the distributional mismatch (Sutton et al., 1998), i.e.,

$$ $\hat{G}_{t,\text{soft}}^{(n)}=\sum_{k=0}^{n-1}\left(\gamma^{k}\prod_{j=0}^{k-1}\rho_{t+j}\right)\big[r_{t+k}-\alpha\log\pi_{\theta}(a_{t+k}\!\mid s_{t+k})\big]+\left(\gamma^{n}\prod_{j=0}^{n-1}\rho_{t+j}\right)Q_{\phi}(s_{t+n},a_{t+n}),$ $$

with the convention that an empty product equals $1$. When $\mu=\pi_{\theta}$, all $\rho$’s are $1$ and $\hat{G}_{t}^{(n)}$ reduces to the standard N-step target. Pure IS can introduce high variance, therefore the step length $n$ cannot be chosen to be very large (Precup et al., 2000; Sutton et al., 1998; Espeholt et al., 2018).

##### Averaged N-step Returns for Critic Updates

Using N-step returns is a standard way to reduce target bias for the critic (Sutton et al., 1998).
For a starting index $t$ and horizon $n\in[1,\texttt{max_length}]$, following  Zhang et al. (2022) we define the N-step target as

$$ $G^{(n)}(s_{t},a_{t},\dots,a_{t+n-1})\;=\;\sum_{j=0}^{n-1}\gamma^{j}r_{t+j}\;+\;\gamma^{n}\,V_{\phi}(s_{t+n}),$ (1) $$

with discount $\gamma\in(0,1]$ and a *target* network parameterized by $\phi$.
Here $V_{\phi}(s)\coloneqq\mathbb{E}_{a\sim\pi_{\theta}(\cdot\mid s)}\!\big[\,Q_{\phi}(s,a)\big]$ is the (non-entropy) bootstrap value under the current policy.
While larger $n$ reduces bootstrapping bias, the variance of $G^{(n)}$ typically grows with $n$ (Precup et al., 2000).
A practical variance reduction is to average partial returns (Konidaris et al., 2011; Daley et al., 2024):

$$ $\bar{G}^{(n)}\;=\;\frac{1}{n}\sum_{i=1}^{n}G^{(i)}.$ (2) $$

This averaging lowers the variance of the reward-sum component from $\mathcal{O}(n)$ toward roughly $\mathcal{O}(n/4)$–$\mathcal{O}(n/3)$ (decreasing with $n$, depending on reward correlations), and makes the value-estimation term decay as $1/n$; under the same assumptions as  (Daley et al., 2024), the full proof appears in App. [C](https://arxiv.org/html/2503.03660v3#A3). This motivates using multiple horizons during critic training (see § [4.2](https://arxiv.org/html/2503.03660v3#S4.SS2)). In our T-SAC implementation, we did not average N-step returns.

## 4 Transformer-based Soft Actor-Critic(T-SAC)

### 4.1 N-step Returns for Critic Updates

#### 4.1.1 Gradient-Level Averaging of N-Step Returns

Notation. For horizon $i$, define the prefix-conditioned online critic output
$Q_{\psi}^{(i)}\coloneqq Q_{\psi}(s_{t},a_{t},\ldots,a_{t+i-1})$.
Directly averaging targets can dilute sparse reward signals (App. [E](https://arxiv.org/html/2503.03660v3#A5)). Instead, we form per-horizon losses

$$ $L_{i}(\psi)\;=\;\tfrac{1}{2}\bigl(Q_{\psi}^{(i)}-G^{(i)}\bigr)^{2},\qquad i=1,\dots,n,$ (3) $$

where a shared-weights *online* critic outputs $Q_{\psi}^{(i)}$ for each prefix $(s_{t},a_{t},\ldots,a_{t+i-1})$ ($s_{t}$ and $a_{t}$ use separate embedding layers). We then *average gradients* across horizons:

$$ $\nabla_{\psi}\bar{L}\;=\;\frac{1}{n}\sum_{i=1}^{n}\nabla_{\psi}L_{i}(\psi).$ (4) $$

Because adjacent horizons have overlapping targets and correspond to adjacent decoder positions in the same network, their per-parameter gradient contributions are positively—but not perfectly—correlated. Averaging therefore reduces update variance while preserving sparse signals (App. [D](https://arxiv.org/html/2503.03660v3#A4),  [E](https://arxiv.org/html/2503.03660v3#A5); Fig.  [3](https://arxiv.org/html/2503.03660v3#S5.F3)).

#### 4.1.2 Stable Critic Learning Without Importance Sampling

Standard off-policy N-step TD presumes that post-$a_{t}$ actions are drawn from the current policy $\pi_{\theta}$, which mismatches replay generated by a behavior policy $\mu$. Per-decision IS with $\rho_{t+k}=\tfrac{\pi_{\theta}(a_{t+k}\mid s_{t+k})}{\mu(a_{t+k}\mid s_{t+k})}$ corrects this but injects high variance (Precup et al., 2000; Sutton et al., 1998; Espeholt et al., 2018).

Similarly to Li et al. (2024a), we instead change the target: the critic predicts *prefix-conditioned* values for realized prefixes from replay,

$$ $\{\,Q_{\psi}(s_{t},a_{t:t+i-1})\,\}_{i=1}^{n},$ $$

with $i$-step targets

$$ $G^{(i)}(s_{t},a_{t:t+i-1})=\sum_{j=0}^{i-1}\gamma^{j}r_{t+j}+\gamma^{i}V_{\phi}(s_{t+i}),$ (5) $$

and the loss

$$ $\mathcal{L}_{\text{critic}}=\mathbb{E}_{(s_{t},a_{t:t+n-1})\sim\mathcal{D}}\Bigl[\tfrac{1}{n}\sum_{i=1}^{n}\bigl(Q_{\psi}(s_{t},a_{t:t+i-1})-G^{(i)}(s_{t},a_{t:t+i-1})\bigr)^{2}\Bigr].$ (6) $$

As rewards follow the *recorded* prefix $a_{t:t+i-1}$, no assumption that actions came from $\pi_{\theta}$ is needed, and hence, no IS is required. Only the bootstrap at $t{+}i$ depends on $\pi_{\theta}$ via $V_{\phi}(s_{t+i})$.

Supervising short windows with multi-horizon targets and averaging their gradients yields stable updates and preserves sparse signals, enabling “multi-step supervision, one-step policy update” *without* IS (Fig. [3](https://arxiv.org/html/2503.03660v3#S5.F3), [10(b)](https://arxiv.org/html/2503.03660v3#A5.F10.sf2)).

### 4.2 Critic Network and Objective

Our critic is a causal Transformer that ingests $(s_{t},a_{t},a_{t+1},\dots,a_{t+n-1})$ and outputs the $n$ prefix-conditioned values $\{Q_{\psi}(s_{t},a_{t},\dots,a_{t+i-1})\}_{i=1}^{n}$ (Fig. [1](https://arxiv.org/html/2503.03660v3#S0.F1)).
For a mini-batch of $L$ trajectories, a random start index $t\in[0,\,N-n]$, and horizons $i\in\{1,\dots,n\}$ with $n$ sampled uniformly from $\{\texttt{min_length},\dots,\texttt{max_length}\}$, the training objective is the mean-squared error over all horizons:

$$ $\mathcal{L}(\psi)=\frac{1}{L\,n}\sum_{k=1}^{L}\sum_{i=1}^{n}\Bigl(Q_{\psi}(s^{k}_{t},a^{k}_{t},\dots,a^{k}_{t+i-1})-G^{(i)}(s^{k}_{t},a^{k}_{t},\dots,a^{k}_{t+i-1})\Bigr)^{2}.$ (7) $$

During backpropagation we apply the gradient-level averaging across $\{L_{i}\}_{i=1}^{n}$ described above. This construction leverages multi-horizon targets and inherits their variance-reduction benefits without target-level signal dilution.

### 4.3 Policy Network and Objective

Following  Ba et al. (2016); Parisotto et al. (2020) and  Plappert et al. (2017), we apply Layer Normalization to the policy’s hidden layers (before the nonlinearity);  Plappert et al. (2017) report this configuration to be useful for continuous-control actor–critic, especially when exploration noise is injected. The objectives remain

$$ $\displaystyle J_{\pi}(\theta)$ $\displaystyle=\mathbb{E}_{s\sim\mathcal{D},\,a\sim\pi_{\theta}}\big[\;\alpha\,\log\pi_{\theta}(a\mid s)-Q_{\psi}(s,a)\;\big],$ (8) $\displaystyle J(\alpha)$ $\displaystyle=\mathbb{E}_{s\sim\mathcal{D},\,a\sim\pi_{\theta}}\big[\;-\alpha\,(\log\pi_{\theta}(a\mid s)+\bar{\mathcal{H}})\;\big],$ (9) $$

with target entropy $-\bar{\mathcal{H}}$ (typically $-\mathrm{dim}(\mathcal{A})$) and automatic temperature tuning (Haarnoja et al., 2018b).
Unlike canonical SAC, our critic does not include entropy in the target; it estimates the standard (non-soft) action-value. The policy is optimized with an entropy-regularized objective, so exploration and regularization are handled entirely by the policy.
This “non-soft critic + policy-side regularization” design is also used in MPO (Abdolmaleki et al., 2018), AWR/AWAC (Peng et al., 2019; Nair et al., 2020), and IQL/IDQL (Kostrikov et al., 2021; Hansen-Estruch et al., 2023).

### 4.4 Critic–parameter freezing schedule enables target–free training (Local-motion only)

CrossQ (Bhatt et al., 2019) removes target networks via batch normalization (Ioffe, 2017) and bounded activations. In contrast, we eliminate Polyak updates with a short *critic–freezing* schedule: at the start of each critic segment we *snapshot* the online critic ($\phi\leftarrow\psi$), precompute and cache bootstrap targets $V_{\phi}(s)$ for all windows to be sampled, then freeze this target snapshot for the next $K$ updates while optimizing the online critic against the cached targets (reuse across $N_{c}$ windows; Gymnasium MuJoCo (Towers et al., 2024): $K{=}20$; see App. [B](https://arxiv.org/html/2503.03660v3#A2)). This lightweight decoupling curbs target drift without BRN or constrained activations. On local–motion benchmarks it yields stable training and matches or exceeds Polyak updates; a hard refresh every $K$ steps performs on par with—or better than—a soft copy. In very sparse–reward regimes (e.g., Box–Pushing–Sparse (Otto et al.,)) we occasionally observe instability; in those cases we reinstate the conventional target update (see App. [E](https://arxiv.org/html/2503.03660v3#A5)). Overall, the freezing schedule is a simple drop–in path to training without continuous soft target updates in our setting; intuition is discussed in App. [G](https://arxiv.org/html/2503.03660v3#A7).

## 5 Experiments

##### Positioning T–SAC.

Prior value–based RL largely split into (i) *step–based* methods (e.g., SAC (Haarnoja et al., 2018a), CrossQ (Bhatt et al., 2019)) that dominate local–motion tasks (e.g., Ant (Towers et al., 2024)), and (ii) *episodic/trajectory–level* methods (e.g., BBRL (Otto et al., 2023a), TOP–ERL (Li et al., 2024a)) that excel on long–horizon problems (e.g., Box–Pushing (Otto et al.,), Meta–World (Yu et al., 2020)). T–SAC bridges these regimes by retaining one–step policy updates while using a sequence–conditioned critic, yielding strong long–horizon performance.

### 5.1 Environments and Seeds

Figure: Figure 2: Training time for 1M environment steps (UTD=1). Benchmarks ran on an NVIDIA A100 (40 GB) GPU and an Intel Xeon Platinum 8368 CPU. T -SAC: MLP policy with two 256-unit hidden layers; Transformer critic with 2 layers × 256 units. Gru/Lstm: same policy; 2-layer RNN critic (256 units). Sac and CrossQ: default configurations.
Refer to caption: x3.png

We evaluate T–SAC on 57 tasks spanning Meta–World ML1 (50) (Yu et al., 2020), Gymnasium MuJoCo locomotion (5) (Towers et al., 2024), and Box–Pushing (dense/sparse; 2) (Otto et al.,).
Meta–World probes task generalization; Gymnasium MuJoCo covers standard locomotion; and Box–Pushing stresses precise, contact–rich manipulation.
Unless noted otherwise, we report means over 8 seeds (ablations use 4) with 95% bootstrap confidence intervals (Agarwal et al., 2021).
Training time per 1M environment steps, compared to off–policy baselines, is shown in Fig. [2](https://arxiv.org/html/2503.03660v3#S5.F2).
Baseline implementations and hyperparameters are detailed in App. [I](https://arxiv.org/html/2503.03660v3#A9) and App. [J](https://arxiv.org/html/2503.03660v3#A10), with environment details in App. [H](https://arxiv.org/html/2503.03660v3#A8).

### 5.2 Meta–World Results

Figure: Figure 3: Success-rate IQM vs. environment interactions on challenging Meta-World ML1 tasks and FANCYGYM Box-Pushing. Panels show Assembly, Disassemble, Hammer, and Stick-Pull, plus Box-Pushing under dense and sparse rewards. Success is counted only at the final timestep.
Refer to caption: x4.png

We run Meta–World ML1 with UTD$=1$, policy delay$=5$, batch size $512$; training time is $\sim$3 h per 1M env steps. Across 50 tasks, T–SAC solves most within $\sim$5M steps and yields stronger aggregated IQM than strong baselines (per–task curves in App. [A](https://arxiv.org/html/2503.03660v3#A1)). On the hardest multi–phase tasks (Assembly, Disassemble, Hammer, Stick–Pull) T–SAC is particularly strong (Fig. [3](https://arxiv.org/html/2503.03660v3#S5.F3)). In contrast, TOP–ERL (Li et al., 2024a) typically requires 20M steps to reach similar aggregates. All comparisons use 5M env steps for T–SAC, while many baselines use larger budgets (Fig. [8](https://arxiv.org/html/2503.03660v3#A1.F8),  [9](https://arxiv.org/html/2503.03660v3#A1.F9)). Success is evaluated only at the final step (App. [H](https://arxiv.org/html/2503.03660v3#A8)), and our aggregates compute IQM *per task* and then average across tasks (unlike pooled–task IQM in TOP–ERL).

### 5.3 Box Pushing (Dense and Sparse)

We evaluate dense and sparse variants of FancyGym (Otto et al.,) Box–Pushing with tight success tolerances (position $\pm 5\,\mathrm{cm}$, orientation $\pm 0.5\,\mathrm{rad}$). Under dense shaping (positional and angular terms at every step), T–SAC attains 96.8% success (Fig. [3](https://arxiv.org/html/2503.03660v3#S5.F3)), exceeding previously reported step–based baselines on this task ($\leq\!85\%$) under the same protocol. Under sparse rewards—where these terms apply only at the terminal step—T–SAC reaches 58% (Fig. [3](https://arxiv.org/html/2503.03660v3#S5.F3)). Sparse feedback induces *hitting–time* dependence (credit only upon goal achievement), which disproportionately challenges step–based algorithms (SAC, PPO, GTrXL, CrossQ). Episodic ERL methods that score full trajectories are less sensitive; e.g., TOP–ERL achieves 70% in this regime. T–SAC is state–of–the–art on dense rewards and competitive under sparse feedback. Success is measured at the terminal step using the tolerances above; full task and reward definitions are in App. [H](https://arxiv.org/html/2503.03660v3#A8). Sparse–reward results use 4 random seeds due to compute limits.

### 5.4 Gymnasium MuJoCo

Figure: Figure 4: Episode return (IQM) vs. environment interactions for Ant, HalfCheetah, HumanoidStandup, Hopper, and Walker2d. Evaluation follows Gymnasium v4 native shaping/termination (no reward normalization); we report undiscounted return and use deterministic-policy evaluation.
Refer to caption: x10.png

For locomotion, a lightweight *critic–parameter freezing* schedule (§ [4.4](https://arxiv.org/html/2503.03660v3#S4.SS4), App. [B](https://arxiv.org/html/2503.03660v3#A2)) enables target–free training: we remove the target network while retaining SAC–style stability and effective value propagation, matching or surpassing Polyak updates at low update rates ($\mathrm{UTD}\!\approx\!0.75$). Across five tasks, T–SAC is strong on HumanoidStandup and HalfCheetah, middling on Ant, and weaker on Hopper and Walker2d (Fig. [4](https://arxiv.org/html/2503.03660v3#S5.F4)). The latter three exhibit high trajectory variability; an action mask is needed for T–SAC and may degrade performance (App. [G](https://arxiv.org/html/2503.03660v3#A7)).

### 5.5 Ablation Study

We conduct targeted ablations on FancyGym Box–Pushing (dense). Within each ablation group, all settings are identical except for the component under test; across groups, minor differences (e.g., training budget or step_length) arise from compute limits and are stated explicitly.

Figure: (a) Transformer components: effect of removing ResNet blocks, the causal mask, and/or self-attention.
Refer to caption: x15.png

#### 5.5.1 Transformer Components

We ablate three parts of the Transformer critic—ResNet blocks, the causal mask, and self-attention—holding all other settings fixed (Fig. [5(a)](https://arxiv.org/html/2503.03660v3#S5.F5.sf1)). Removing only self-attention invalidates the segment-conditioned objective and typically diverges. Removing self-attention *together with* the ResNet and causal mask reduces the critic to a plain MLP; we compare this baseline in § [5.5.3](https://arxiv.org/html/2503.03660v3#S5.SS5.SSS3) and Fig.  [5(c)](https://arxiv.org/html/2503.03660v3#S5.F5.sf3).

#### 5.5.2 Step Length and Min Length

We sweep step_length and min_length, which define the multi-horizon ($N$-step) supervision window. At each update we sample
$n\in\{\texttt{min_length},\dots,\texttt{step_length}\}$;
by default $n\sim\mathrm{Unif}\{1,\ldots,16\}$.
Using a *fixed* horizon $L$ smooths optimization but can lower final performance (Fig. [5(b)](https://arxiv.org/html/2503.03660v3#S5.F5.sf2)), partly because the last $L{-}1$ states of each segment never appear as starting indices. Large fixed $L$ (e.g., $16$) amplifies this effect (see Fig. [5(b)](https://arxiv.org/html/2503.03660v3#S5.F5.sf2)). Despite the usual advice to keep $n\!\leq\!5$ (Precup et al., 2000; Sutton et al., 1998; Espeholt et al., 2018), our Transformer critic with gradient-level averaging is stable up to $n{=}16$ and benefits from longer windows (Fig. [5(b)](https://arxiv.org/html/2503.03660v3#S5.F5.sf2)).

#### 5.5.3 Comparison to Multi-step MLP (Reinforcement Learning with Action Chunking)

With 16 M environment steps (default budget: 20 M), a multi-step MLP critic underperforms the Transformer critic; policy-side chunking with an MLP yields only modest gains (Figs. [6](https://arxiv.org/html/2503.03660v3#S5.F6), [5(c)](https://arxiv.org/html/2503.03660v3#S5.F5.sf3)). In contrast, adding chunking to the Transformer critic helps: the best setting uses chunk length $4$, converges by $\approx\!10$ M steps, reaches $99.5\%$ final success, and shows no late-stage divergence across seeds (Fig. [5(d)](https://arxiv.org/html/2503.03660v3#S5.F5.sf4)). We report this primarily to demonstrate compatibility between a Transformer critic and
policy-side chunking; chunking itself is not our focus. The MLP critic also suffers causal leakage: because it consumes fixed-length segments, $Q(s_{t},a_{t})$ is

Figure: Figure 6: Policy chunking (adapted from Li et al. (2025)).
Refer to caption: x21.png

effectively conditioned on $(s_{t},a_{t:t+n})$, thereby “peeking” at future actions $a_{t+1:t+n}$ (Li et al., 2025). Our Transformer critic applies a causal mask so each token attends only to positions $t^{\prime}\!\leq\!t$, preserving causal correctness (see Fig. [1](https://arxiv.org/html/2503.03660v3#S0.F1)) and outperforming an ablation without masking (Fig. [5(a)](https://arxiv.org/html/2503.03660v3#S5.F5.sf1)). Finally, the MLP critic imposes design constraints—fixed-length inputs and a policy chunk length tied to the critic—whereas the Transformer critic avoids these constraints.

#### 5.5.4 Number of Samples Generated per Step

We vary the number of supervision windows sampled per environment step (“per-step target samples”). For this study we use step_length $=[1,8]$ (standard: $[1,16]$). Unlike conventional SAC (one target per step), T-SAC benefits from generating multiple windows (Fig. [5(e)](https://arxiv.org/html/2503.03660v3#S5.F5.sf5)); our default is $4$. In practice, use four vectorized envs or collect a single trajectory of length $4{\times}\texttt{max_length}$ and slice it (App. [G](https://arxiv.org/html/2503.03660v3#A7)). Intuitively, multiple windows raise the share of fresh samples in each batch: with one window, once selected there are none left; with four, three remain.

#### 5.5.5 GRU/LSTM as the Critic

Figure: Figure 7: Structure of the recursive network used in our experiments. FC = fully connected layer.
Refer to caption: x22.png

We replace the Transformer critic with GRU and LSTM variants under identical training (10M env steps; standard: 20M). Although recurrent critics can model action sequences, our gradient–level averaging analysis (§ [4.1.1](https://arxiv.org/html/2503.03660v3#S4.SS1.SSS1); App. [D](https://arxiv.org/html/2503.03660v3#A4)) does not directly apply, and parallelism is reduced (Fig. [2](https://arxiv.org/html/2503.03660v3#S5.F2)). Empirically, both GRU and LSTM underperform the Transformer critic on Box–Pushing in our setting (Fig. [5(f)](https://arxiv.org/html/2503.03660v3#S5.F5.sf6)).

## 6 Conclusion

On Meta-World ML1 multiphase and FancyGym box-pushing tasks, our T-SAC framework with a Transformer critic achieves state-of-the-art performance under a fixed training budget (5M environment steps for Meta-World ML1 and 20M for box-pushing) tasks and consistent evaluation protocol (success rate over 8 seeds). Empirically, the Transformer critic yields smoother value estimates and more temporally coherent credit assignment, producing higher-quality continuous control sequences. In contrast, many multiphase pipelines rely on largely open-loop plans, while step-based value methods often underutilize historical control signals; our approach combines the strengths of both while mitigating their respective trade-offs. To our knowledge, few prior methods achieve this combination without incurring the usual drawbacks of either paradigm; we provide comparisons to representative baselines in §[5](https://arxiv.org/html/2503.03660v3#S5).

#### Reproducibility Statement

We made substantial efforts to ensure reproducibility. The paper and appendix specify environments, evaluation protocols, and all hyperparameters used. Upon acceptance, we will release a public GitHub repository containing the implementation of the proposed algorithms, experiment scripts, and trained models. Detailed descriptions of the experimental setup, including configuration files, are provided in the appendix to enable independent reimplementation during review.

## References

- Abdolmaleki et al. (2018)
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller.
Maximum a posteriori policy optimisation.
*arXiv preprint arXiv:1806.06920*, 2018.
- Agarwal et al. (2023)
Pranav Agarwal, Aamer Abdul Rahman, Pierre-Luc St-Charles, Simon JD Prince, and Samira Ebrahimi Kahou.
Transformers in reinforcement learning: a survey.
*arXiv preprint arXiv:2307.05979*, 2023.
- Agarwal et al. (2021)
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare.
Deep reinforcement learning at the edge of the statistical precipice.
*Advances in neural information processing systems*, 34:29304–29320, 2021.
- Asadi & Littman (2017)
Kavosh Asadi and Michael L Littman.
An alternative softmax operator for reinforcement learning.
In *International Conference on Machine Learning*, pp. 243–252. PMLR, 2017.
- Asadi et al. (2024)
Kavosh Asadi, Yao Liu, Shoham Sabach, Ming Yin, and Rasool Fakoor.
Learning the target network in function space.
*arXiv preprint arXiv:2406.01838*, 2024.
- Ba et al. (2016)
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
Layer normalization.
*arXiv preprint arXiv:1607.06450*, 2016.
- Bhatt et al. (2019)
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters.
Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity.
*arXiv preprint arXiv:1902.05605*, 2019.
- Celik et al. (2025)
Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palenicek, Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann.
Dime: Diffusion-based maximum entropy reinforcement learning.
*arXiv preprint arXiv:2502.02316*, 2025.
- Chen et al. (2022)
Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn.
Transdreamer: Reinforcement learning with transformer world models.
*arXiv preprint arXiv:2202.09481*, 2022.
- Chen et al. (2021)
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.
Decision transformer: Reinforcement learning via sequence modeling.
*Advances in neural information processing systems*, 34:15084–15097, 2021.
- Daley et al. (2024)
Brett Daley, Martha White, and Marlos C Machado.
Averaging $n$-step returns reduces variance in reinforcement learning.
*arXiv preprint arXiv:2402.03903*, 2024.
- Durugkar & Stone (2018)
Ishan Durugkar and Peter Stone.
Td learning with constrained gradients.
2018.
- Espeholt et al. (2018)
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al.
Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.
In *International conference on machine learning*, pp. 1407–1416. PMLR, 2018.
- Fellows et al. (2023)
Mattie Fellows, Matthew JA Smith, and Shimon Whiteson.
Why target networks stabilise temporal difference methods.
In *International Conference on Machine Learning*, pp. 9886–9909. PMLR, 2023.
- Fujimoto et al. (2018)
Scott Fujimoto, Herke Hoof, and David Meger.
Addressing function approximation error in actor-critic methods.
In *International conference on machine learning*, pp. 1587–1596. PMLR, 2018.
- Gallici et al. (2024)
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin.
Simplifying deep temporal difference learning.
*arXiv preprint arXiv:2407.04811*, 2024.
- Gupta et al. (2019)
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman.
Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning.
*arXiv preprint arXiv:1910.11956*, 2019.
- Haarnoja et al. (2018a)
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.
In *International conference on machine learning*, pp. 1861–1870. Pmlr, 2018a.
- Haarnoja et al. (2018b)
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.
Soft actor-critic algorithms and applications.
*arXiv preprint arXiv:1812.05905*, 2018b.
- Hansen-Estruch et al. (2023)
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine.
Idql: Implicit q-learning as an actor-critic method with diffusion policies.
*arXiv preprint arXiv:2304.10573*, 2023.
- Hessel et al. (2018)
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.
Rainbow: Combining improvements in deep reinforcement learning.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 32, 2018.
- Huang et al. (2022)
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang.
The 37 implementation details of proximal policy optimization.
In *ICLR Blog Track*, 2022.
URL [https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/).
https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/.
- Ioffe (2017)
Sergey Ioffe.
Batch renormalization: Towards reducing minibatch dependence in batch-normalized models.
*Advances in neural information processing systems*, 30, 2017.
- Janner et al. (2021)
Michael Janner, Qiyang Li, and Sergey Levine.
Offline reinforcement learning as one big sequence modeling problem.
*Advances in neural information processing systems*, 34:1273–1286, 2021.
- Kakade (2001)
Sham M Kakade.
A natural policy gradient.
*Advances in neural information processing systems*, 14, 2001.
- Kalyanakrishnan et al. (2021)
Shivaram Kalyanakrishnan, Siddharth Aravindan, Vishwajeet Bagdawat, Varun Bhatt, Harshith Goka, Archit Gupta, Kalpesh Krishna, and Vihari Piratla.
An analysis of frame-skipping in reinforcement learning.
*arXiv preprint arXiv:2102.03718*, 2021.
- Kim et al. (2019)
Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris.
Deepmellow: removing the need for a target network in deep q-learning.
In *Proceedings of the twenty eighth international joint conference on artificial intelligence*, 2019.
- Konidaris et al. (2011)
George Konidaris, Scott Niekum, and Philip S Thomas.
Td_gamma: Re-evaluating complex backups in temporal difference learning.
*Advances in Neural Information Processing Systems*, 24, 2011.
- Kostrikov et al. (2021)
Ilya Kostrikov, Ashvin Nair, and Sergey Levine.
Offline reinforcement learning with implicit q-learning.
*arXiv preprint arXiv:2110.06169*, 2021.
- Li et al. (2023)
Ge Li, Zeqi Jin, Michael Volpp, Fabian Otto, Rudolf Lioutikov, and Gerhard Neumann.
Prodmp: A unified perspective on dynamic and probabilistic movement primitives.
*IEEE Robotics and Automation Letters*, 8(4):2325–2332, 2023.
- Li et al. (2024a)
Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, and Gerhard Neumann.
Top-erl: Transformer-based off-policy episodic reinforcement learning.
*arXiv preprint arXiv:2410.09536*, 2024a.
- Li et al. (2024b)
Ge Li, Hongyi Zhou, Dominik Roth, Serge Thilges, Fabian Otto, Rudolf Lioutikov, and Gerhard Neumann.
Open the black box: Step-based policy updates for temporally-correlated episodic reinforcement learning.
*arXiv preprint arXiv:2401.11437*, 2024b.
- Li et al. (2025)
Qiyang Li, Zhiyuan Zhou, and Sergey Levine.
Reinforcement learning with action chunking.
*arXiv preprint arXiv:2507.07969*, 2025.
- Liang et al. (2018)
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica.
RLlib: Abstractions for distributed reinforcement learning.
In *International Conference on Machine Learning (ICML)*, 2018.
URL [https://arxiv.org/pdf/1712.09381](https://arxiv.org/pdf/1712.09381).
- Liu et al. (2024)
Yuejiang Liu, Jubayer Ibn Hamid, Annie Xie, Yoonho Lee, Maximilian Du, and Chelsea Finn.
Bidirectional decoding: Improving action chunking via closed-loop resampling.
*arXiv e-prints*, pp. arXiv–2408, 2024.
- Mavrin et al. (2019)
Borislav Mavrin, Hengshuai Yao, and Linglong Kong.
Deep reinforcement learning with decorrelation.
*arXiv preprint arXiv:1903.07765*, 2019.
- Mnih et al. (2016)
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.
Asynchronous methods for deep reinforcement learning.
In *International conference on machine learning*, pp. 1928–1937. PmLR, 2016.
- Munos et al. (2016)
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare.
Safe and efficient off-policy reinforcement learning.
*Advances in neural information processing systems*, 29, 2016.
- Nair et al. (2020)
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine.
Awac: Accelerating online reinforcement learning with offline datasets.
*arXiv preprint arXiv:2006.09359*, 2020.
- Ohnishi et al. (2019)
Shota Ohnishi, Eiji Uchibe, Yotaro Yamaguchi, Kosuke Nakanishi, Yuji Yasui, and Shin Ishii.
Constrained deep q-learning gradually approaching ordinary q-learning.
*Frontiers in neurorobotics*, 13:103, 2019.
- (41)
Fabian Otto, Onur Celik, Dominik Roth, and Hongyi Zhou.
Fancy gym.
URL [https://github.com/ALRhub/fancy_gym](https://github.com/ALRhub/fancy_gym).
- Otto et al. (2021)
Fabian Otto, Philipp Becker, Ngo Anh Vien, Hanna Carolin Ziesche, and Gerhard Neumann.
Differentiable trust region layers for deep reinforcement learning.
*arXiv preprint arXiv:2101.09207*, 2021.
- Otto et al. (2023a)
Fabian Otto, Onur Celik, Hongyi Zhou, Hanna Ziesche, Vien Anh Ngo, and Gerhard Neumann.
Deep black-box reinforcement learning with movement primitives.
In *Conference on Robot Learning*, pp. 1244–1265. PMLR, 2023a.
- Otto et al. (2023b)
Fabian Otto, Hongyi Zhou, Onur Celik, Ge Li, Rudolf Lioutikov, and Gerhard Neumann.
Mp3: Movement primitive-based (re-) planning policy.
*arXiv preprint arXiv:2306.12729*, 2023b.
- Paraschos et al. (2013)
Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann.
Probabilistic movement primitives.
*Advances in neural information processing systems*, 26, 2013.
- Parisotto et al. (2020)
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al.
Stabilizing transformers for reinforcement learning.
In *International conference on machine learning*, pp. 7487–7498. PMLR, 2020.
- Peng et al. (2019)
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine.
Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.
*arXiv preprint arXiv:1910.00177*, 2019.
- Peters & Schaal (2008)
Jan Peters and Stefan Schaal.
Reinforcement learning of motor skills with policy gradients.
*Neural networks*, 21(4):682–697, 2008.
- Piché et al. (2021)
Alexandre Piché, Valentin Thomas, Rafael Pardinas, Joseph Marino, Gian Maria Marconi, Christopher Pal, and Mohammad Emtiyaz Khan.
Bridging the gap between target networks and functional regularization.
*arXiv preprint arXiv:2106.02613*, 2021.
- Plappert et al. (2017)
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz.
Parameter space noise for exploration.
*arXiv preprint arXiv:1706.01905*, 2017.
- Precup et al. (2000)
Doina Precup, Richard S Sutton, and Satinder Singh.
Eligibility traces for off-policy policy evaluation.
2000.
- Raffin et al. (2021)
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann.
Stable-baselines3: Reliable reinforcement learning implementations.
*Journal of Machine Learning Research*, 22(268):1–8, 2021.
URL [http://jmlr.org/papers/v22/20-1364.html](http://jmlr.org/papers/v22/20-1364.html).
- Raffin et al. (2022)
Antonin Raffin, Jens Kober, and Freek Stulp.
Smooth exploration for robotic reinforcement learning.
In *Conference on robot learning*, pp. 1634–1644. PMLR, 2022.
- Rückstieß et al. (2008)
Thomas Rückstieß, Martin Felder, and Jürgen Schmidhuber.
State-dependent exploration for policy gradient methods.
In *Joint European Conference on Machine Learning and Knowledge Discovery in Databases*, pp. 234–249. Springer, 2008.
- Rückstiess et al. (2010)
Thomas Rückstiess, Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, and Jürgen Schmidhuber.
Exploring parameter space in reinforcement learning.
*Paladyn*, 1(1):14–24, 2010.
- Rummery & Niranjan (1994)
Gavin A Rummery and Mahesan Niranjan.
*On-line Q-learning using connectionist systems*, volume 37.
University of Cambridge, Department of Engineering Cambridge, UK, 1994.
- Schulman et al. (2015a)
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.
Trust region policy optimization.
In *International conference on machine learning*, pp. 1889–1897. PMLR, 2015a.
- Schulman et al. (2015b)
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel.
High-dimensional continuous control using generalized advantage estimation.
*arXiv preprint arXiv:1506.02438*, 2015b.
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Sharma et al. (2017)
Sahil Sharma, Aravind Srinivas, and Balaraman Ravindran.
Learning to repeat: Fine grained action repetition for deep reinforcement learning.
*arXiv preprint arXiv:1702.06054*, 2017.
- Sutton et al. (1998)
Richard S Sutton, Andrew G Barto, et al.
Reinforcement learning: An introduction.
1(1), 1998.
- Towers et al. (2024)
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al.
Gymnasium: A standard interface for reinforcement learning environments.
*arXiv preprint arXiv:2407.17032*, 2024.
- Van Hasselt et al. (2016)
Hado Van Hasselt, Arthur Guez, and David Silver.
Deep reinforcement learning with double q-learning.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 30, 2016.
- Vaswani et al. (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
*Advances in neural information processing systems*, 30, 2017.
- Vincent et al. (2025)
Théo Vincent, Yogesh Tripathi, Tim Faust, Yaniv Oren, Jan Peters, and Carlo D’Eramo.
Bridging the performance gap between target-free and target-based reinforcement learning with iterated q-learning.
*arXiv preprint arXiv:2506.04398*, 2025.
- Watkins & Dayan (1992)
Christopher JCH Watkins and Peter Dayan.
Q-learning.
*Machine learning*, 8(3):279–292, 1992.
- Yu et al. (2020)
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.
Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.
In *Conference on robot learning*, pp. 1094–1100. PMLR, 2020.
- Zhang et al. (2022)
Haichao Zhang, Wei Xu, and Haonan Yu.
Generative planning for temporally coordinated exploration in reinforcement learning.
*arXiv preprint arXiv:2201.09765*, 2022.
- Zhang et al. (2021)
Jesse Zhang, Haonan Yu, and Wei Xu.
Hierarchical reinforcement learning by discovering intrinsic options.
*arXiv preprint arXiv:2101.06521*, 2021.

## Appendix A Appendix: Individual Meta-World Tests results

Figure: Figure 8: Success Rate IQM of each individual Meta-World tasks. (Part 1)
Refer to caption: x23.png

Figure: Figure 9: Success Rate IQM of each individual Meta-World tasks. (Part 2)
Refer to caption: x44.png

## Appendix B Appendix: Detailed Algorithm Flowchart

Figure: Algorithm 1 T-SAC

## Appendix C Appendix: Proof of the variance reduction property of averaging of N-step returns

##### A convenient variance identity.

Under the equicorrelation model,

$$ $\mathbb{E}[X_{k}]=m,\quad\mathrm{Var}(X_{k})=v,\quad\mathrm{Cov}(X_{k},X_{\ell})=\rho\,v\ \ (k\neq\ell),\qquad\rho\geq 0,$ $$

any weighted sum $S=\sum_{k=0}^{N-1}a_{k}X_{k}$ satisfies

$$ $\mathrm{Var}(S)=v\!\left(\sum_{k=0}^{N-1}a_{k}^{2}\;+\;\rho\!\left[\Bigl(\sum_{k=0}^{N-1}a_{k}\Bigr)^{2}-\sum_{k=0}^{N-1}a_{k}^{2}\right]\right).$ (10) $$

This follows by expanding $\mathrm{Var}$ and collecting diagonal/off-diagonal terms.

##### Reward part with discount 𝜸 < 𝟏 \bm{\gamma<1} .

Define the single $N$-step discounted reward sum and its triangular average by

$$ $R_{N}(\gamma)\;\triangleq\;\sum_{k=0}^{N-1}\gamma^{k}r_{k},\qquad\bar{R}_{N}(\gamma)\;\triangleq\;\frac{1}{N}\sum_{i=1}^{N}\sum_{k=0}^{i-1}\gamma^{k}r_{k}\;=\;\sum_{k=0}^{N-1}w_{k}r_{k},\quad w_{k}\triangleq\frac{N-k}{N}\,\gamma^{k}.$ $$

Let

$$ $S_{0}\equiv S_{0}(N,\gamma)\triangleq\sum_{k=0}^{N-1}\gamma^{2k}=\frac{1-\gamma^{2N}}{1-\gamma^{2}},\qquad T_{0}\equiv T_{0}(N,\gamma)\triangleq\sum_{k=0}^{N-1}\gamma^{k}=\frac{1-\gamma^{N}}{1-\gamma}.$ $$

Also define the (discounted, triangular) weight aggregates

$$ $A_{\gamma}\;\triangleq\;\sum_{k=0}^{N-1}w_{k}^{2}\;=\;\frac{1}{N^{2}}\sum_{k=0}^{N-1}(N-k)^{2}\gamma^{2k},\qquad B_{\gamma}\;\triangleq\;\Bigl(\sum_{k=0}^{N-1}w_{k}\Bigr)^{2}-A_{\gamma}.$ (11) $$

###### Lemma 1 (Variance formulas for the discounted reward part) .

Under the reward assumptions stated in the setup,

$$ $\mathrm{Var}\!\big[R_{N}(\gamma)\big]=\sigma^{2}\bigl[S_{0}+\rho\,(T_{0}^{2}-S_{0})\bigr],\qquad\mathrm{Var}\!\big[\bar{R}_{N}(\gamma)\big]=\sigma^{2}\bigl[A_{\gamma}+\rho\,B_{\gamma}\bigr].$ $$

*Proof.* Apply equation [10](https://arxiv.org/html/2503.03660v3#A3.E10) with weights $a_{k}=\gamma^{k}$ for $R_{N}(\gamma)$
and $a_{k}=w_{k}$ for $\bar{R}_{N}(\gamma)$, and use the definitions of $S_{0},T_{0},A_{\gamma},B_{\gamma}$. ∎

###### Proposition 1 (Reward-side variance ratio and bounds) .

Define

$$ $R_{\gamma}(N)\;\triangleq\;\frac{\mathrm{Var}[R_{N}(\gamma)]}{\mathrm{Var}[\bar{R}_{N}(\gamma)]}=\frac{S_{0}+\rho\,(T_{0}^{2}-S_{0})}{A_{\gamma}+\rho B_{\gamma}}.$ $$

Then for all $N\geq 1$, $\rho\geq 0$, and $\gamma\in(0,1]$,

$$ $1\ \leq\ R_{\gamma}(N)\ <\ 4.$ $$

Moreover, for $\gamma=1$, $R_{\gamma}(N)\nearrow 4$ as $N\to\infty$; for any fixed $\gamma\in(0,1)$, $R_{\gamma}(N)\to 1$. When $\rho>0$, $R_{\gamma}(N)$ is strictly increasing in $N$ for $\gamma=1$; for $\gamma<1$ it need not be monotone.

*Proof (sketch).* The $\gamma=1$ proof carries through verbatim after replacing the unweighted triangular weights $j$ by the discounted weights $w_{k}=((N-k)/N)\gamma^{k}$; the same algebraic positivity arguments yield the bounds. For the limits, when $\gamma<1$ we have $w_{k}\to\gamma^{k}$ pointwise as $N\to\infty$ and dominated convergence gives $A_{\gamma}\to S_{0}$ and $\sum_{k}w_{k}\to T_{0}$, hence $R_{\gamma}(N)\to 1$. For $\gamma=1$, the standard triangular-sum identities imply $R_{1}(N)\nearrow 4$. ∎

###### Lemma 1 (Variance formulas for the discounted reward part) .

###### Proposition 1 (Reward-side variance ratio and bounds) .

##### Bootstrap value part.

Let $Z_{i}\triangleq V_{\phi_{\mathrm{tar}}}(s_{t+i})$ denote the (target) values used for bootstrapping, and assume

$$ $\mathbb{E}[Z_{i}]=\nu,\quad\mathrm{Var}(Z_{i})=\tau^{2},\quad\mathrm{Cov}(Z_{i},Z_{j})=\kappa\,\tau^{2}\ \ (i\neq j),\qquad\kappa\geq 0,$ $$

and that $\{r_{k}\}$ and $\{Z_{i}\}$ are independent unless stated otherwise.
The single $N$-step bootstrap term and its triangular average are

$$ $B_{N}(\gamma)\triangleq\gamma^{N}Z_{N},\qquad\bar{B}_{N}(\gamma)\triangleq\frac{1}{N}\sum_{i=1}^{N}\gamma^{i}Z_{i}.$ $$

Let

$$ $S_{1}\equiv S_{1}(N,\gamma)\triangleq\sum_{i=1}^{N}\gamma^{2i}=\frac{\gamma^{2}(1-\gamma^{2N})}{1-\gamma^{2}},\qquad C\equiv C(N,\gamma)\triangleq\sum_{i=1}^{N}\gamma^{i}=\frac{\gamma(1-\gamma^{N})}{1-\gamma}.$ $$

###### Lemma 2 (Variance of the averaged bootstrap part) .

With $\mathrm{Cov}(Z_{i},Z_{j})=\kappa\,\tau^{2}$ for $i\neq j$,

$$ $\mathrm{Var}[\bar{B}_{N}(\gamma)]=\frac{\tau^{2}}{N^{2}}\Bigl[S_{1}+\kappa\,(C^{2}-S_{1})\Bigr].$ $$

*Proof.* Apply equation [10](https://arxiv.org/html/2503.03660v3#A3.E10) with $a_{i}=\gamma^{i}/N$. ∎

###### Proposition 2 (Bootstrap-side variance ratio, bounds, and condition) .

Define

$$ $R_{B}(N,\gamma,\kappa)\;\triangleq\;\frac{\mathrm{Var}[B_{N}(\gamma)]}{\mathrm{Var}[\bar{B}_{N}(\gamma)]}=\frac{N^{2}\,\gamma^{2N}}{\,S_{1}+\kappa(C^{2}-S_{1})\,}.$ $$

Then for any $\kappa\in[0,1]$,

$$ $\frac{N^{2}\gamma^{2N}}{C^{2}}\ \leq\ R_{B}(N,\gamma,\kappa)\ \leq\ \frac{N^{2}\gamma^{2N}}{S_{1}},\qquad\frac{\partial R_{B}}{\partial\kappa}<0.$ $$

In particular, averaging reduces bootstrap variance ($R_{B}\geq 1$) whenever

$$ $\kappa\ \leq\ \kappa_{\star}(N,\gamma)\ \triangleq\ \frac{N^{2}\gamma^{2N}-S_{1}}{C^{2}-S_{1}}\,.$ $$

For the uncorrelated case $(\kappa=0)$,
$R_{B}(N,\gamma,0)=N^{2}\gamma^{2N}/S_{1}.$

*Proof.* Monotonicity in $\kappa$ is immediate from the denominator. The bounds follow from $S_{1}\leq S_{1}+\kappa(C^{2}-S_{1})\leq C^{2}$. Solve $R_{B}\geq 1$ for $\kappa$ to get $\kappa_{\star}$. ∎

###### Lemma 2 (Variance of the averaged bootstrap part) .

###### Proposition 2 (Bootstrap-side variance ratio, bounds, and condition) .

##### Putting the parts together.

With
$G_{N}(\gamma)=R_{N}(\gamma)+B_{N}(\gamma)$
and
$\bar{G}_{N}(\gamma)=\bar{R}_{N}(\gamma)+\bar{B}_{N}(\gamma)$,
and assuming independence between rewards and bootstrap values,

$$ $\frac{\mathrm{Var}[G_{N}(\gamma)]}{\mathrm{Var}[\bar{G}_{N}(\gamma)]}=\frac{\mathrm{Var}[R_{N}(\gamma)]+\mathrm{Var}[B_{N}(\gamma)]}{\mathrm{Var}[\bar{R}_{N}(\gamma)]+\mathrm{Var}[\bar{B}_{N}(\gamma)]}.$ (12) $$

Since all terms are nonnegative,

$$ $\min\!\bigl\{R_{\gamma}(N),\,R_{B}(N,\gamma,\kappa)\bigr\}\ \leq\ \frac{\mathrm{Var}[G_{N}(\gamma)]}{\mathrm{Var}[\bar{G}_{N}(\gamma)]}\ \leq\ \max\!\bigl\{R_{\gamma}(N),\,R_{B}(N,\gamma,\kappa)\bigr\}.$ $$

Consequently:

- •
Because $R_{\gamma}(N)\geq 1$ (Prop. [1](https://arxiv.org/html/2503.03660v3#Thmtheorem1)), if $R_{B}(N,\gamma,\kappa)\geq 1$ (e.g., $\kappa\leq\kappa_{\star}$), then averaging N-step targets strictly reduces total variance.
- •
Even if $R_{B}(N,\gamma,\kappa)<1$, the overall ratio in equation [12](https://arxiv.org/html/2503.03660v3#A3.E12) remains $\geq 1$ whenever the reward-side gain dominates:
$R_{\gamma}(N)\ \geq\ \frac{\mathrm{Var}[\bar{B}_{N}(\gamma)]}{\mathrm{Var}[B_{N}(\gamma)]}=\frac{S_{1}+\kappa(C^{2}-S_{1})}{N^{2}\gamma^{2N}}.$

##### Dependence between rewards and bootstrap values.

If $\mathrm{Cov}\!\big(R_{N}(\gamma),B_{N}(\gamma)\big)$ and
$\mathrm{Cov}\!\big(\bar{R}_{N}(\gamma),\bar{B}_{N}(\gamma)\big)$ are nonzero, the numerator/denominator of equation [12](https://arxiv.org/html/2503.03660v3#A3.E12) each acquire an additional covariance term. The sandwich bound above still applies after inserting these, and a crude control is
$|\mathrm{Cov}(X,Y)|\leq\sqrt{\mathrm{Var}(X)\mathrm{Var}(Y)}$
(Cauchy–Schwarz), which cannot overturn the above conclusions unless the cross-covariances are pathologically large.

##### Useful closed forms.

Besides $S_{0},S_{1},T_{0},C$ above, one has

$$ $\sum_{k=0}^{N-1}(N-k)\gamma^{k}=\frac{1-(N+1)\gamma^{N}+N\gamma^{N+1}}{(1-\gamma)^{2}}.$ $$

A closed form for $\sum_{k=0}^{N-1}(N-k)^{2}\gamma^{2k}$ (hence $A_{\gamma}$ via equation [11](https://arxiv.org/html/2503.03660v3#A3.E11)) follows from the standard identities for $\sum kx^{k}$ and $\sum k^{2}x^{k}$ after the change $k\mapsto N-1-k$; we omit it as not needed for the bounds above.

##### Remarks.

(i) Setting $\gamma\to 1$ recovers the *undiscounted* results: $S_{0},T_{0},S_{1},C\to N$ and $A_{\gamma},B_{\gamma}\to A/N^{2},B/N^{2}$, where $A,B$ are the non-discounted triangle sums.
(ii) As $N\to\infty$ with fixed $\gamma<1$, $S_{1}\to\gamma^{2}/(1-\gamma^{2})$ and $C\to\gamma/(1-\gamma)$ while $N^{2}\gamma^{2N}\to 0$; thus $R_{B}(N,\gamma,\kappa)\to 0$. For typical RL regimes ($\gamma\gtrsim 0.95$, moderate $N$), $\kappa_{\star}(N,\gamma)$ is positive and large, so averaging still reduces bootstrap variance over a wide range of $\kappa$.
(iii) $R_{\gamma}(N)$ is horizon- and discount-agnostic in the sense of the bound $1\leq R_{\gamma}(N)<4$; for $\gamma<1$, its large-$N$ limit is $1$. It is the principal driver of the overall variance reduction.

##### On the equicorrelation assumption.

We assumed an equicorrelation (exchangeable) model for the reward noise
and for the bootstrapped values: identical variances and a common
pairwise correlation ($\rho$ and $\kappa$, respectively). This is a
standard device that yields closed forms while capturing the empirically
relevant regime of positively correlated temporal signals in RL
trajectories.

The key conclusions above are *robust* to relaxing equicorrelation.
Let $\Sigma$ be any covariance matrix for $(r_{0},\dots,r_{N-1})$ with
nonnegative entries (i.e., nonnegative autocovariances). For any
nonnegative weight vector $w$, the variance is $w^{T}\Sigma w$
and increases monotonically with each off-diagonal entry. Since the
triangular weights have strictly smaller $\ell_{2}$ norm and smaller
sum than the flat weights of the single $N$-step sum, the reward-side
variance reduction persists under a wide range of stationary,
positively correlated processes (including Toeplitz/lag-dependent
models such as $\mathrm{Cov}(r_{k},r_{\ell})=\sigma^{2}\rho_{|k-\ell|}$ with
$\rho_{d}\geq 0$). The specific constant $4$ in the upper bound is
tight for the exchangeable model; with general lag structure the same
$[1,4)$ bracket continues to hold under mild bounded-correlation
conditions (e.g. $\sup_{k\neq\ell}\mathrm{Corr}(r_{k},r_{\ell})\leq 1$),
while the uncorrelated case ($\rho_{d}\equiv 0$) recovers the
$[1,3)$ limit.

For the bootstrap part, assuming a common correlation $\kappa$ across
$\{Z_{i}\}$ is likewise a tractable approximation: the explicit ratio
$R_{B}(N,\gamma,\kappa)$ is decreasing in $\kappa$, so weaker
dependence only strengthens the variance reduction. More general
lag-dependent models $\mathrm{Cov}(Z_{i},Z_{j})=\tau^{2}\kappa_{|i-j|}$ with
$\kappa_{d}\geq 0$ lead to the same qualitative behavior (smaller weights
and partial averaging reduce variance), with our equicorrelation
formulas serving as convenient upper/lower benchmarks.

*When to be cautious.* If the process exhibits strong *negative*
or oscillatory correlations (e.g. alternation effects), equicorrelation
overstates the benefit of averaging; in such cases, replacing the common
$\rho$ (or $\kappa$) by a small set of lag-specific parameters
($\rho_{1},\rho_{2},\dots$) is safer. Empirically, one can estimate the
sample autocovariance and plug it into $w^{T}\hat{\Sigma}w$ to verify
the inequalities numerically.

## Appendix D Appendix: Variance reduction from gradient-level averaging with a shared-weights Transformer critic

##### Setup.

Fix a trajectory position $t$. A Transformer critic with shared parameters $\psi$ outputs

$$ $Q^{(1)}_{\psi},\,Q^{(2)}_{\psi},\,\dots,\,Q^{(n)}_{\psi},$ $$

where $Q^{(i)}_{\psi}$ predicts the $i$-step return for the same prefix $(s_{t},a_{t},\dots,a_{t+i-1})$.
Let $G^{(i)}$ denote the $i$-step target and define the per-horizon MSE

$$ $L_{i}(\psi)=\tfrac{1}{2}\!\big(Q^{(i)}_{\psi}-G^{(i)}\big)^{2},\qquad\bar{L}(\psi)\triangleq\frac{1}{n}\sum_{i=1}^{n}L_{i}(\psi).$ $$

In implementation we *average their gradients* during backprop:

$$ $\nabla_{\psi}\,\bar{L}(\psi)\;=\;\frac{1}{n}\sum_{i=1}^{n}\nabla_{\psi}L_{i}(\psi)\;=\;\frac{1}{n}\sum_{i=1}^{n}\big(Q^{(i)}_{\psi}-G^{(i)}\big)\,\nabla_{\psi}Q^{(i)}_{\psi}.$ $$

##### Local gradient factorization under weight sharing.

Let $w$ be any scalar entry of $\psi$. By the chain rule,

$$ $g_{i}(w)\;\triangleq\;\frac{\partial L_{i}}{\partial w}=\underbrace{\big(Q^{(i)}_{\psi}-G^{(i)}\big)}_{\varepsilon^{(i)}}\;\underbrace{\frac{\partial Q^{(i)}_{\psi}}{\partial w}}_{\psi^{(i)}(w)}.$ $$

For linear modules (affine maps in attention/FFN), the Jacobian has the standard local form
$\tfrac{\partial Q^{(i)}_{\psi}}{\partial w}=a^{(i)}\,\delta^{(i)}$ (input activation $\times$ upstream error).
Because the same $w$ is *shared* across decoder positions, the sequence
$\{g_{i}(w)\}_{i=1}^{n}$ are $n$ gradient contributions for the *same* parameter,
drawn from adjacent positions of one forward pass, and are therefore generally
*positively correlated*.

##### A convenient covariance model (exchangeable/equicorrelated).

For fixed $w$, we use the standard homoscedastic equicorrelation approximation
(also common in mini-batch analyses):

$$ $\mathrm{Var}[g_{i}(w)]=\sigma_{w}^{2},\qquad\mathrm{Cov}(g_{i}(w),g_{j}(w))=\rho_{w}\,\sigma_{w}^{2}\quad(i\neq j),\qquad\rho_{w}\in[0,1).$ $$

This captures the empirically relevant regime where adjacent horizons produce positively
correlated gradients and yields tight, closed-form variance expressions.

###### Lemma 3 (Variance of the averaged per-parameter update) .

With the model above, the averaged update $\bar{g}(w)\triangleq\tfrac{1}{n}\sum_{i=1}^{n}g_{i}(w)$ satisfies

$$ $\mathrm{Var}\!\big[\bar{g}(w)\big]=\frac{1}{n^{2}}\!\left(\sum_{i=1}^{n}\mathrm{Var}[g_{i}]+\!\!\sum_{i\neq j}\!\!\mathrm{Cov}(g_{i},g_{j})\right)=\sigma_{w}^{2}\,\frac{1+(n-1)\rho_{w}}{n}.$ $$

In particular, $\mathrm{Var}[\bar{g}(w)]<\sigma_{w}^{2}$ for any $\rho_{w}<1$.

###### Corollary 3 (Effective batch size and asymptotics) .

Define the *effective sample size*
$n_{\mathrm{eff}}(w)\triangleq\dfrac{n}{1+(n-1)\rho_{w}}.$
Then $\mathrm{Var}[\bar{g}(w)]=\sigma_{w}^{2}/n_{\mathrm{eff}}(w)$ with
$1\leq n_{\mathrm{eff}}(w)\leq n$,
strictly increasing in $n$, and
$\lim_{n\to\infty}\mathrm{Var}[\bar{g}(w)]=\rho_{w}\,\sigma_{w}^{2}$
(the correlation-imposed variance floor).

###### Proposition 4 (Uniform horizon averaging is optimal under exchangeability) .

Among all unbiased linear combinations $\sum_{i=1}^{n}\alpha_{i}g_{i}(w)$ with $\sum_{i}\alpha_{i}=1$,
the variance is minimized by the *uniform* weights $\alpha_{i}=\tfrac{1}{n}$
whenever $\mathrm{Cov}(g_{i},g_{j})$ is exchangeable (same diagonal/off-diagonal).

*Proof.* For an exchangeable covariance $\Sigma_{w}=\sigma_{w}^{2}[(1-\rho_{w})I+\rho_{w}\mathbf{1}\mathbf{1}^{T}]$,
$\mathrm{Var}(\sum_{i}\alpha_{i}g_{i})=\alpha^{T}\Sigma_{w}\alpha$
is minimized under $\mathbf{1}^{T}\alpha=1$ by $\alpha^{\star}=\tfrac{1}{n}\mathbf{1}$. ∎

###### Lemma 3 (Variance of the averaged per-parameter update) .

###### Corollary 3 (Effective batch size and asymptotics) .

###### Proposition 4 (Uniform horizon averaging is optimal under exchangeability) .

##### Why ρ w ≳ 0 \rho_{w}\gtrsim 0 is natural.

Both multiplicative factors of $g_{i}(w)$ vary smoothly with $i$:
(i) the targets $G^{(i)}$ share overlapping reward sums and a common bootstrapped tail; and
(ii) the Jacobians $\partial Q^{(i)}_{\psi}/\partial w$ come from *adjacent* decoder positions
of the same Transformer. This induces positive correlation among $\{g_{i}(w)\}$, putting us squarely
in the regime of Lemma [3](https://arxiv.org/html/2503.03660v3#Thmlemma3).

##### Connection to target-side variance (discounted rewards and bootstrap).

Let $\gamma\in(0,1]$ be the discount.
Write the single $N$-step reward sum and its triangular average as

$$ $R_{N}(\gamma)=\sum_{k=0}^{N-1}\gamma^{k}r_{k},\qquad\bar{R}_{N}(\gamma)=\frac{1}{N}\sum_{i=1}^{N}\sum_{k=0}^{i-1}\gamma^{k}r_{k}=\sum_{k=0}^{N-1}\underbrace{\frac{N-k}{N}\gamma^{k}}_{w_{k}}r_{k}.$ $$

Under the equicorrelated reward model (mean $\mu$, variance $\sigma^{2}$, pairwise corr. $\rho\geq 0$),

$$ $\mathrm{Var}\!\big[R_{N}(\gamma)\big]=\sigma^{2}\!\left[S_{0}+\rho\,(T_{0}^{2}-S_{0})\right],\qquad\mathrm{Var}\!\big[\bar{R}_{N}(\gamma)\big]=\sigma^{2}\!\left[A_{\gamma}+\rho\,B_{\gamma}\right],$ $$

with

$$ $S_{0}=\sum_{k=0}^{N-1}\gamma^{2k}=\frac{1-\gamma^{2N}}{1-\gamma^{2}},\quad T_{0}=\sum_{k=0}^{N-1}\gamma^{k}=\frac{1-\gamma^{N}}{1-\gamma},\quad A_{\gamma}=\sum_{k=0}^{N-1}w_{k}^{2},\quad B_{\gamma}=\Big(\sum_{k=0}^{N-1}w_{k}\Big)^{2}-A_{\gamma}.$ $$

The reward-side ratio

$$ $R_{\gamma}(N)\;\triangleq\;\frac{\mathrm{Var}[R_{N}(\gamma)]}{\mathrm{Var}[\bar{R}_{N}(\gamma)]}=\frac{S_{0}+\rho\,(T_{0}^{2}-S_{0})}{A_{\gamma}+\rho B_{\gamma}}$ $$

satisfies the *uniform bound* $\;1\leq R_{\gamma}(N)<4\;$ for all $N\geq 1$, $\rho\geq 0$, $\gamma\in(0,1]$,
and $R_{\gamma}(N)\nearrow 4$ as $N\to\infty$.

For the bootstrapped values $Z_{i}=V_{\phi_{\mathrm{tar}}}(s_{t+i})$ with $\mathrm{Var}(Z_{i})=\tau^{2}$ and
$\mathrm{Cov}(Z_{i},Z_{j})=\kappa\,\tau^{2}\ (i\neq j,\ \kappa\in[0,1])$, define

$$ $B_{N}(\gamma)=\gamma^{N}Z_{N},\qquad\bar{B}_{N}(\gamma)=\frac{1}{N}\sum_{i=1}^{N}\gamma^{i}Z_{i},$ $$

and let $S_{1}=\sum_{i=1}^{N}\gamma^{2i}$, $C=\sum_{i=1}^{N}\gamma^{i}$.
Then

$$ $\mathrm{Var}[\bar{B}_{N}(\gamma)]=\frac{\tau^{2}}{N^{2}}\big[S_{1}+\kappa(C^{2}-S_{1})\big],\qquad R_{B}(N,\gamma,\kappa)\triangleq\frac{\mathrm{Var}[B_{N}(\gamma)]}{\mathrm{Var}[\bar{B}_{N}(\gamma)]}=\frac{N^{2}\gamma^{2N}}{S_{1}+\kappa(C^{2}-S_{1})}.$ $$

$R_{B}$ is decreasing in $\kappa$ and obeys

$$ $\frac{N^{2}\gamma^{2N}}{C^{2}}\ \leq\ R_{B}(N,\gamma,\kappa)\ \leq\ \frac{N^{2}\gamma^{2N}}{S_{1}}.$ $$

In particular, averaging the bootstrap part reduces variance whenever
$\kappa\leq\kappa_{\star}(N,\gamma)\triangleq\dfrac{N^{2}\gamma^{2N}-S_{1}}{C^{2}-S_{1}}.$

###### Theorem 5 (Main: gradient averaging reduces update variance; compounded by target-side smoothing) .

Let $w$ be any scalar parameter of the shared-weights Transformer critic and suppose
$\{g_{i}(w)\}_{i=1}^{n}$ are homoscedastic and equicorrelated with $\rho_{w}<1$.
Then

$$ $\mathrm{Var}\!\left[\frac{\partial\bar{L}}{\partial w}\right]=\sigma_{w}^{2}\,\frac{1+(n-1)\rho_{w}}{n}<\sigma_{w}^{2}=\mathrm{Var}\!\left[\frac{\partial L_{j}}{\partial w}\right],\quad\forall j\in\{1,\dots,n\}.$ $$

Moreover, writing $g_{i}(w)=\varepsilon^{(i)}\psi^{(i)}(w)$ and (mildly) assuming
$\{\varepsilon^{(i)}\}$ and $\{\psi^{(i)}(w)\}$ are independent across $i$ with bounded second moments,
there exist constants $a_{w},b_{w}\geq 0$ (depending only on $\psi$) such that

$$ $\mathrm{Var}[g_{i}(w)]\ \leq\ a_{w}\,\mathrm{Var}[G^{(i)}]\ +\ b_{w}.$ $$

Consequently, replacing a single horizon by the triangularly averaged target across horizons $1{:}N$
reduces the reward-side variance by at least a factor $R_{\gamma}(N)^{-1}\!\in(1/4,1]$, and (when
$\kappa\leq\kappa_{\star}$) also reduces the bootstrap-side variance by a factor $R_{B}(N,\gamma,\kappa)^{-1}$.
Thus, in addition to the *across-horizon gradient averaging* gain
$\frac{1+(n-1)\rho_{w}}{n}$, the per-horizon variance term $\sigma_{w}^{2}$ itself decreases with $N$
via target-side smoothing, yielding a compounded reduction.

###### Theorem 5 (Main: gradient averaging reduces update variance; compounded by target-side smoothing) .

##### Practical notes.

(i) The gradient-level algebra is agnostic to discount $\gamma$; only the target-side constants
$(S_{0},T_{0},A_{\gamma},B_{\gamma})$ and $(S_{1},C)$ change with $\gamma$.
(ii) Under exchangeability, uniform averaging across horizons is *variance-optimal*
(Prop. [4](https://arxiv.org/html/2503.03660v3#Thmtheorem4)); no learned horizon-weights are needed for variance reasons.
(iii) As $n$ grows, the residual variance floor is $\rho_{w}\,\sigma_{w}^{2}$ (Cor. [3](https://arxiv.org/html/2503.03660v3#Thmtheorem3));
lower temporal correlation between horizon-gradients directly improves this floor.
(iv) If horizon-gradients are not perfectly exchangeable, the bound
$\mathrm{Var}[\bar{g}(w)]\leq\dfrac{\bar{\sigma}_{w}^{2}}{n}\big(1+(n-1)\bar{\rho}_{w}\big)$
still holds whenever $\mathrm{Var}[g_{i}]\leq\bar{\sigma}_{w}^{2}$ and $\operatorname{Corr}(g_{i},g_{j})\leq\bar{\rho}_{w}$ for all $i\neq j$.

## Appendix E Appendix: Additional Experimental Results

Figure: (a) *Hard-copy* under sparse rewards accelerates convergence and attains high success rates, albeit with pronounced training volatility.
Refer to caption: x74.png

Post-figure summary.
Figure [10(a)](https://arxiv.org/html/2503.03660v3#A5.F10.sf1) shows a stability–efficiency trade-off: *hard-copy* yields strong, sample-efficient gains but exhibits oscillations (Fig. [10(a)](https://arxiv.org/html/2503.03660v3#A5.F10.sf1)), while naive $N$-step averaging can derail optimization (Fig. [10(b)](https://arxiv.org/html/2503.03660v3#A5.F10.sf2)). Given its appealing peak performance, *hard-copy* warrants further study—e.g., adaptive mixing schedules, target-network smoothing, or variance-reduction techniques—to retain its benefits while improving stability.
Seeds: 4. Results under IQM with 95% confidence interval.

## Appendix F Appendix: Transformer Critic detailed Structure

Figure: Figure 11: T-SAC Critic Detailed Structure: a causal Transformer over short state–action segments. Given $(s_{t},a_{t},\dots,a_{t+n})$, the network produces $n$ scalar outputs $\{Q_{\psi}(s_{t},a_{t},\dots,a_{t+i})\}_{i=1}^{n}$. Colors and block styling follow the Transformer diagram conventions of Vaswani et al. (2017).
Refer to caption: x76.png

Implementation details.
We follow the TOP–ERL–style Transformer critic design adopted in this work (see Li et al. (2024a) for the schematic), i.e., a masked multi–head self–attention stack with positional encodings and residual Add&Norm blocks; the critic ingests $(s_{t},a_{t},\ldots,a_{t+n-1})$ and jointly predicts all $1\ldots n$ step returns.
State and action tokens use *separate* one–layer linear embeddings (no bias), consistent with our training objective that conditions on realized action prefixes; the output head is a linear map without bias that emits one scalar per decoder position. No dropout is used anywhere in the critic. The causal mask ensures each position $i$ only attends to $\leq i$ tokens, aligning the network outputs with the $i$-step targets used for learning.

## Appendix G Appendix: Trajectories saved in Replay Buffer and Action Mask Design

Figure: (a) Fixed-length segments stored in the replay buffer. Each sample is a window $(s_{t:t+L},a_{t:t+L-1},r_{t:t+L-1},d_{t:t+L-1})$. If a terminal $d_{\tau}=1$ appears before the window is full, we immediately continue saving from the start of the next trajectory until the segment length $L$ is reached.
Refer to caption: x77.png

#### The benefit of chunking on the critic side

We analyze chunking under sparse rewards and target–value reuse.

##### Setup.

Consider a fixed segment of $N$ transitions $(s_{t},a_{t},r_{t},\dots,s_{t+N})$.
A *chunked* sample is obtained by:
(i) drawing a start index $p$ uniformly from $\{t{+}1,\dots,t{+}N\}$;
(ii) drawing a window length $L$ uniformly from the integers $\{\ell_{\min},\dots,\ell_{\max}\}$ (with $\ell_{\min}\geq 1$);
and defining the window end

$$ $q\;=\;\min\{\,p+L-1,\ t{+}N\,\}.$ $$

For every $i\in\{p,\dots,q{-}1\}$ we form the truncated multi-step target that bootstraps at $q$:

$$ $G_{i}\;=\;\sum_{j=i}^{q-1}\gamma^{\,j-i}\,r_{j}\;+\;\gamma^{\,q-i}V_{\phi}(s_{q}).$ (13) $$

Hence the *same* value $V_{\phi}(s_{q})$ is reused across the $(q{-}p)$ TD updates inside the window.

##### How often is a particular value reused?

Write $\Delta=\ell_{\max}-\ell_{\min}+1$, and (for clarity) index the segment states by $1,\dots,N$.
Let $\text{reuse}_{j}$ denote the number of updates in a sampled window whose target bootstraps at $V_{\phi}(s_{j})$ (i.e., with $q=j$).
Then, for all interior states $j<N$,

$$ $\mathbb{E}[\text{reuse}_{j}]=\frac{1}{N\,\Delta}\!\!\sum_{k=\ell_{\min}-1}^{\min(\ell_{\max}-1,\,j-1)}k=\begin{cases}0,&j<\ell_{\min},\[6.0pt] \dfrac{(j-\ell_{\min}+1)(j+\ell_{\min}-2)}{2N\,\Delta},&\ell_{\min}\leq j<\ell_{\max},\[10.0pt] \dfrac{\ell_{\min}+\ell_{\max}-2}{2N},&\ell_{\max}\leq j\leq N{-}1.\end{cases}$ (14) $$

Thus, away from the left boundary, the expected reuse $\mathbb{E}[\text{reuse}_{j}]$ *plateaus* at
$\tfrac{\ell_{\min}+\ell_{\max}-2}{2N}$ for all $j\in[\ell_{\max},\,N{-}1]$.

The right boundary $j=N$ is special because of truncation ($q=\min\{p{+}L{-}1,N\}$).
In this case,

$$ $\mathbb{E}[\text{reuse}_{N}]=\frac{1}{N}\sum_{h=1}^{\ell_{\min}}(h{-}1)\;+\;\frac{1}{N\,\Delta}\sum_{h=\ell_{\min}+1}^{\ell_{\max}}(h{-}1)(\ell_{\max}-h+1),$ (15) $$

which exceeds the interior plateau and concentrates more bootstrap reuse at the end of the segment.
A transparent special case is $\ell_{\min}=1$:

$$ $\mathbb{E}[\text{reuse}_{N}]\;=\;\frac{\ell_{\max}^{2}-1}{6N}.$ (16) $$

Averaging equation [14](https://arxiv.org/html/2503.03660v3#A7.E14) across all $j$ yields the compact relation

$$ $\frac{1}{N}\sum_{j=1}^{N}\mathbb{E}[\text{reuse}_{j}]\;=\;\frac{\mathbb{E}[L]-1}{N},$ (17) $$

i.e., per sampled window the expected reuse scales linearly with the average window length.

##### Connection to state coverage (selection) probability.

The probability that a given state $j$ is *covered* by the sampled window (i.e., $j\in[p,\;q]$) is

$$ $\Pr(j\ \text{covered})=\frac{1}{N\,\Delta}\sum_{p=1}^{j}\bigl[\ell_{\max}-\max\{\ell_{\min},\,j-p+1\}+1\bigr]_{+},$ (18) $$

where $[\cdot]_{+}=\max\{\,\cdot,0\,\}$.
For the common case $\ell_{\min}=1$ and writing $m=\ell_{\max}$, this simplifies to

$$ $\Pr(j\ \text{covered})=\begin{cases}\dfrac{1}{N}\!\left(j-\dfrac{j(j-1)}{2m}\right),&1\leq j\leq m,\[10.0pt] \dfrac{m+1}{2N},&m<j\leq N,\end{cases}$ (19) $$

i.e., a ramp near the start followed by a flat plateau. This higher coverage (vs. 1-step sampling) underlies the critic-side gains below.

##### Sparse rewards: how far does a single reward propagate?

Assume only the terminal transition carries non-zero reward (the sparse-reward setting).
An update’s target contains that reward iff the sampled window reaches the segment end ($q=N$), in which case *all* $(N{-}p)$ updates inside the window include it.
Therefore, the expected number of reward-bearing updates per sampled window equals

$$ $\mathbb{E}\big[\#\text{updates including terminal reward}\big]=\frac{1}{N}\sum_{h=1}^{\ell_{\min}}(h{-}1)\;+\;\frac{1}{N\,\Delta}\sum_{h=\ell_{\min}+1}^{\ell_{\max}}(h{-}1)(\ell_{\max}-h+1),$ (20) $$

which coincides with equation [15](https://arxiv.org/html/2503.03660v3#A7.E15). In particular, for $\ell_{\min}=1$,

$$ $\mathbb{E}\big[\#\text{updates including terminal reward}\big]\;=\;\frac{\ell_{\max}^{2}-1}{6N},$ (21) $$

representing a $\approx\ell_{\max}^{2}/6$-fold amplification over uniform 1-step TD (which touches the terminal reward only in the single $(N{-}1)\!\to N$ update, i.e., $1/N$ of samples).

##### Takeaways.

Chunking yields two critic-side benefits:
(i) *Target–value reuse:* each sampled window reuses a single bootstrap $V_{\phi}(s_{q})$ across $\mathbb{E}[\text{reuse}_{j}]$ updates, reaching a plateau of $\tfrac{\ell_{\min}+\ell_{\max}-2}{2N}$ for interior states and an even larger value at the terminal state due to truncation equation [15](https://arxiv.org/html/2503.03660v3#A7.E15). This may help explain why, in our setting, training remains stable even *without* a target network in local-motion tasks.
(ii) *Sparse-reward propagation:* when only the last transition is rewarded, chunking increases—often quadratically in $\ell_{\max}$ when $\ell_{\min}=1$—the share of updates that incorporate the true reward, substantially shortening effective credit-assignment horizons. This mechanism helps explain why T-SAC performs well under sparse-reward settings.

## Appendix H Experiment Description

### H.1 Meta-World ML1

Figure: Figure 13: Meta-World tasks (Yu et al., 2020).
Refer to caption: figures/metaworld_task.png

Meta-World (Yu et al., 2020) is an open-source simulated benchmark for meta-reinforcement learning and multi-task learning in robotic manipulation. It comprises 50 distinct tasks spanning skills such as grasping, pushing, and object placement, each posing different perception–control challenges. By covering a broader skill spectrum than narrowly scoped benchmarks, Meta-World is well-suited for evaluating algorithms that aim to generalize across diverse behaviors. Figure [13](https://arxiv.org/html/2503.03660v3#A8.F13) enumerates all 50 tasks and illustrates their variety and difficulty.

Success criterion. To better approximate real-world deployment, we adopt a stringent evaluation rule: an episode is counted as successful only if the environment’s success condition is satisfied at the *final* timestep; intermediate achievements do not count toward success.

### H.2 Box Pushing

Figure: Figure 14: Box Pushing task (Otto et al.,).
Refer to caption: figures/robot_box_push_task.png

##### Setup.

A 7-DoF Franka Emika Panda arm with a rod pushes a box on a table to a target pose. At episode start, initial and target box poses are sampled with a minimum $0.2$ m separation:

$$ $x_{i}\!\in\![0.3,0.6],\quad y_{i}\!\in\![-0.4,0.4],\quad\theta\!\in\![0,2\pi].$ $$

Success (for evaluation) is position error $\leq 0.05$ m and orientation error $\leq 0.5$ rad.

##### Observations & Actions.

Observations: robot joint positions/velocities $(\bm{q},\bm{\dot{q}})$, box position/orientation $(\bm{p},\bm{r})$, and target $(\bm{p}_{\text{target}},\bm{r}_{\text{target}})$. Actions: joint torques $\bm{a}_{t}\in\mathbb{R}^{7}$.

##### Termination.

Fixed horizon $T=100$ steps; no early termination.

##### Dense reward.

At each step,

$$ $R_{\text{total}}=-R_{\text{rod}}-0.02\,\tau_{t}-\text{err}(\bm{q},\bm{\dot{q}})-350\,R_{\text{position}}-200\,R_{\text{rotation}}.$ $$

Subterms are

$$ $\displaystyle R_{\text{rod}}$ $\displaystyle=\operatorname{clip}\!\left(\|\bm{p}-\bm{h}_{\text{pos}}\|,\,0.05,\,10\right)+\operatorname{clip}\!\left(\tfrac{2}{\pi}\arccos\!\left|\bm{h}_{\text{rot}}\!\cdot\!\bm{h}_{0}\right|,\,0.25,\,2\right),$ (22) $\displaystyle\tau_{t}$ $\displaystyle=\sum_{i=1}^{7}(a_{t}^{i})^{2},$ (23) $\displaystyle\text{err}(\bm{q},\bm{\dot{q}})$ $\displaystyle=\sum_{i:\,|q_{i}|>|q_{i}^{b}|}\!\left(|q_{i}|-|q_{i}^{b}|\right)+\sum_{j:\,|\dot{q}_{j}|>|\dot{q}_{j}^{b}|}\!\left(|\dot{q}_{j}|-|\dot{q}_{j}^{b}|\right),$ (24) $\displaystyle R_{\text{position}}$ $\displaystyle=\|\bm{p}-\bm{p}_{\text{target}}\|,$ (25) $\displaystyle R_{\text{rotation}}$ $\displaystyle=\tfrac{1}{\pi}\arccos\!\left|\bm{r}\!\cdot\!\bm{r}_{\text{target}}\right|.$ (26) $$

Here, $\bm{h}_{\text{pos}}$ is the rod tip position, and $\bm{h}_{\text{rot}},\bm{h}_{0}$ are rod orientations (quaternions).

##### Sparse reward.

Only the task terms are applied at the final step:

$$ $R_{\text{total}}=\begin{cases}-R_{\text{rod}}-0.02\,\tau_{t}-\text{err}(\bm{q},\bm{\dot{q}}),&t<T,\[2.0pt] -R_{\text{rod}}-0.02\,\tau_{t}-\text{err}(\bm{q},\bm{\dot{q}})-350\,R_{\text{position}}-200\,R_{\text{rotation}},&t=T.\end{cases}$ (27) $$

### H.3 Gymnasium MuJoCo

Figure: (a) Ant
Refer to caption: figures/Ant.png

We evaluate on the Gymnasium MuJoCo v4 suite—Ant-v4, HalfCheetah-v4, Hopper-v4, Walker2d-v4, and HumanoidStandup-v4 (Fig. [15](https://arxiv.org/html/2503.03660v3#A8.F15)). We use the default observation and action spaces and the native v4 reward shaping and termination rules (no reward normalization). Performance is reported as undiscounted episode return. Unless noted otherwise, evaluation uses the deterministic policy over 152 episodes and aggregates results across multiple random seeds using the IQM with 95% bootstrap confidence intervals; full hyperparameters and seeds are provided in App. [J](https://arxiv.org/html/2503.03660v3#A10).

## Appendix I Appendix: Algorithm Implementations

##### PPO

Proximal Policy Optimization (PPO) (Schulman et al., 2017) is an on–policy, step–based method that constrains policy updates to remain close to the behavior policy. Two variants are common: *PPO–Penalty* (KL regularization) and *PPO–Clip* (clipped surrogate). We evaluate PPO–Clip given its prevalence and robustness, following the reference implementation in Raffin et al. (2021). Seeds: 20.

##### SAC

Soft Actor–Critic (SAC) (Haarnoja et al., 2018a; b) is an off–policy actor–critic with twin Q–networks to mitigate overestimation and an entropy term to encourage exploration. We use the Bhatt et al. (2019) implementation, which includes SAC. Seeds: 20 (Meta–World ML1), 5 (Gym MuJoCo).

##### TD3

Twin Delayed DDPG (TD3) (Fujimoto et al., 2018) addresses overestimation and instability via (i) clipped double Q–learning, (ii) delayed policy updates, and (iii) target policy smoothing. Our TD3 follows standard practice adapted from Raffin et al. (2021), including Polyak averaging and action noise for exploration. Seeds: 5.

##### GTrXL

Gated Transformer–XL (GTrXL) (Parisotto et al., 2020) stabilizes Transformer training for partially observable control. We build on the PPO + GTrXL implementation from Liang et al. (2018) and add minibatch advantage normalization plus a state–independent log–standard–deviation head following Huang et al. (2022). Seeds: 4.

##### gSDE

Generalized State–Dependent Exploration (gSDE) (Raffin et al., 2022; Rückstieß et al., 2008; Rückstiess et al., 2010) replaces i.i.d. Gaussian action noise with state–dependent, temporally smooth exploration. Concretely, disturbances are generated as $\epsilon_{t}=\Theta s$, where $s$ is the last hidden layer’s activation and $\Theta$ is resampled from a Gaussian every $n$ steps according to the SDE sampling frequency. We evaluate gSDE with PPO using the reference implementation of Raffin et al. (2022); for stability on some tasks we employ a linear schedule for the PPO clipping range. Seeds: 20.

##### BBRL

Black–Box Reinforcement Learning (BBRL) (Otto et al., 2023a; b) performs episodic, trajectory–level search by parameterizing policies with ProMPs (Paraschos et al., 2013). This handles sparse and non–Markovian rewards but can reduce sample efficiency. We consider both diagonal–covariance (BBRL–Std) and full–covariance (BBRL–Cov) Gaussian policies, paired with ProDMP (Li et al., 2023). Seeds: 20.

##### TCP

Temporally–Correlated Episodic RL (TCP) (Li et al., 2024b) augments episodic policy updates with step–level signals, narrowing the gap between episodic and step–based RL while retaining smooth, parameter–space exploration. Seeds: 20.

##### TOP–ERL

Trajectory–Optimized Policy for Episodic RL (TOP–ERL) optimizes a distribution over motion–primitive parameters with (i) a KL–constrained trust region and (ii) a temporally structured covariance that induces smooth, correlated exploration across the episode. Our instantiation uses ProDMP (Li et al., 2023) as the trajectory generator; unless stated, we adopt an adaptive scale (entropy) schedule and per–dimension normalization of primitive parameters. Seeds: 8.

##### CrossQ

CrossQ (Bhatt et al., 2019) is an off–policy SAC variant that removes target networks and applies BRN in the critic, enabling strong sample efficiency at an update–to–data ratio of $\mathrm{UTD}=1$. We follow the authors’ reference implementation: a single batch–normalized critic (no target networks), default temperature tuning, and recommended hyperparameters unless stated otherwise. Seeds: 4 (Meta–World ML1), 5 (Gym MuJoCo and Box–Pushing). Training on Box–Pushing was capped at 10 M steps due to the experiment budget; by that point, wall–clock time exceeded 24 h.

## Appendix J Appendix: Hyperparameters of the algorithms

##### Baseline provenance.

For BBRL, TCP, PPO, gSDE, GTrXL, TOP–ERL, and SAC on Meta–World ML1, we report numbers from prior publications and/or official released runs/configurations under settings comparable to ours; we did not perform additional large–scale sweeps for these baselines in this paper (see citations in the main text and Appendix [I](https://arxiv.org/html/2503.03660v3#A9)).

##### Methods tuned in this work.

We tuned SAC on Gym/FancyGym, the full CrossQ implementation, and TD3, including optimizer selection and hyperparameters (e.g., learning rates).

##### Our tuning for T–SAC.

For T–SAC, we conducted a targeted grid search over Transformer–critic depth (number of attention layers), number of heads, dimensions per head, learning rates (policy/critic/$\alpha$), supervision–window settings (fixed vs. variable horizons; min_length, step_length), number of per–step target windows, and policy–side chunk length (for the compatibility study). Where appropriate, we initialized choices from publicly reported configurations: Transformer hyperparameter ranges from TOP–ERL (Li et al., 2024a), the entropy–temperature term from Celik et al. (2025), and the optimizer family from CrossQ (Bhatt et al., 2019). Final settings and search grids are listed in the appendix tables.

Table: Table 1: Hyperparameters for the Meta-World experiments. Episode Length $T=500$

##### Task-specific settings (Meta-World).

For T–SAC, we initialize the policy’s log standard deviation as $\log\sigma=-5$.
The replay buffer stores $5{,}000$ segments of length $125$ (i.e., $5{,}000\times 125=625{,}000$ transitions).
The sampler retrieves $4$ segments of length $125$ (i.e., $4\times 125=500$ transitions).

Table: Table 2: Hyperparameters for the Box Pushing Dense, Episode Length $T=100$

Table: Table 3: Hyperparameters for the Box Pushing Sparse, Episode Length $T=100$

Table: Table 4: Hyperparameters for the Gymnasium MuJoCo, Episode Length $T=1000$

##### Task-specific settings (Gymnasium MuJoCo).

For T–SAC, the initial policy log-standard deviation is set to $-5$ for Ant, HumanoidStandup, and HalfCheetah, and to $-10$ for Hopper and Walker2d. For Hopper and Walker2d only, the target entropy is
$H_{\text{target}}=-4\cdot\mathrm{dim}(\mathcal{A})$; unless otherwise noted, other tasks use the SAC default $H_{\text{target}}=-\,\mathrm{dim}(\mathcal{A})$.

## Appendix: Use of Large Language Models (LLMs)

We used large language models (LLMs) as a general-purpose assistant during writing and development. Its roles included:

- •
grammar and spell-checking, language polishing, and minor stylistic edits;
- •
drafting and rewriting multi-paragraph text (e.g., introductions, preliminaries, and parts of experimental write-ups) based on author-provided outlines and results;
- •
high-level suggestions for debugging strategies and hyperparameter choices;
- •
assistance with literature search (proposing search queries and surfacing candidate papers).

All BibTeX entries were copied from Google Scholar; the LLMs did not generate or edit bibliographic entries. The LLMs did *not* originate the paper’s main idea, problem formulation, algorithmic design, or experimental plan, and it was not used to generate or alter data, results, or figures. All citations were selected and verified by the authors against the original sources. All LLMs outputs were reviewed and, when necessary, edited or discarded. No confidential or proprietary data were shared with the LLMs.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Can T-SAC be evaluated on discrete-action domains (e.g., Atari) or more strongly partially observable settings?
2. Do the benefits of sequence-conditioned critics carry over to image-based ML1 or partially observed Box-Pushing variants?
3. How does T-SAC compare with Transformer-based policies and offline RL approaches (e.g., Decision Transformer)?
4. For GRU/LSTM critics, how extensive was the hyperparameter and architecture search (depth, hidden size, normalization, teacher forcing, sequence length)?
5. Have you tested robustness under observation/action noise, stochastic terminations, or mixed-policy replay buffers? This seems particularly relevant given the “off-policy without IS” choice.
6. What aspects of the design (e.g., window sampling, bootstrap lag, prefix length) most limit performance in sparse-reward settings, and could targeted modifications (e.g., auxiliary returns, adaptive horizons) mitigate the observed instability?
soundness: 3
strengths: 1. The causal Transformer predicts prefix-conditioned values aligned with realized action prefixes, which plausibly improves temporal credit assignment. Using a “non-soft” critic with entropy regularization on the policy side preserves the standard SAC actor objective and simplifies the critic target, improving implementability.
2. Gradient-level averaging is supported by Lemma 3 and Theorem 5, establishing $\mathrm{Var}[\nabla_\psi \bar L] < \sigma_w^2$ under equicorrelation assumptions. The paper analyzes target-side variance reduction for both reward and bootstrap components, providing bounds $1 \le R_\gamma(N) < 4$ and conditions for reducing bootstrap variance. The IS-free formulation avoids high-variance importance ratios by conditioning on realized prefixes and is clearly derived from the N-step return objective.
3. The evaluation spans 57 tasks across Meta-World ML1, Gymnasium MuJoCo, and Box-Pushing. Gains are pronounced on difficult tasks (e.g., 96.8% on Box-Pushing (dense), outperforming step-based baselines ≤85%) and on multi-phase ML1 tasks such as Assembly, Disassemble, Hammer, and Stick-Pull.
summary: This paper proposes T-SAC (Transformer-based Soft Actor-Critic), an off-policy RL method whose critic is a lightweight causal Transformer trained on short trajectory segments with aggregated (N)-step returns. Unlike standard SAC, which evaluates single state–action pairs, T-SAC conditions the critic on action prefixes $(s_t, a_t,\ldots,a_{t+n-1})$ and predicts prefix-conditioned values for multiple horizons without importance sampling. The method averages gradients across N-step targets and introduces a parameter-freezing schedule for the critic in locomotion tasks. Experiments on Meta-World ML1, Box-Pushing, and Gymnasium MuJoCo show improved sample efficiency, especially on long-horizon and sparse-reward tasks; e.g., 96.8% success on Box-Pushing (dense) and solving most ML1 tasks within ~5M interactions at UTD=1.
weaknesses: 1. The core idea—Transformer critic trained with N-step returns—resembles TOP-ERL; the manuscript acknowledges this but does not sufficiently sharpen the technical distinctions. The claim of “bridging step-based and episodic regimes” appears overstated: the policy remains step-based while sequence conditioning is confined to the critic. Several components (causal Transformer, parameter freezing, N-step averaging) draw on prior art, but the paper does not isolate which design choices constitute the principal contribution.
2. Comparisons with traditional off-policy N-step methods using IS are empirical only; formal connections to prior analyses are not developed.
3. Reported instabilities on Box-Pushing-Sparse lead to a fallback on soft targets, limiting generality. The action-mask requirement and performance variability on Ant/Hopper/Walker2d suggest fragility and sensitivity to dynamics/constraints.
4. T-SAC trails CrossQ on Hopper and Walker2d; the explanation “action mask needed may degrade performance” lacks systematic investigation. On Box-Pushing-Sparse, success is 58%, below TOP-ERL’s 70%; the claim of being “competitive under sparse feedback” does not reconcile this gap.
5. No ablations on freeze length $K$ or reuse factor $N_c$ are provided to delineate stability regions.
6. Equation (1) defines $G^{(n)}$ with $V_\phi(s_{t+n})$ but does not clarify whether this is a soft or standard value; later (Sec. 4.3) it states the critic estimates the standard (non-soft) action-value. The notation should reflect this distinction.
7. All benchmarks are simulated continuous-control tasks; the paper does not evaluate discrete action spaces, stronger partial observability (beyond action history), or real-robot settings that motivate the approach.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1, Could you please elaborate on the specific role and necessity of the "action mask" mentioned as the cause for performance degradation on "high trajectory variability" tasks like Hopper and Walker2d ? Experimental result(e.g. with action mask vs without action mask) would be helpful to understand your claim.

2, The paper's core mechanism is averaging gradients rather than targets. The result shows that naive target averaging fails in sparse-reward settings (Fig. 10b, App. E). Could you provide more intuition as to why averaging gradients is more effective at preserving the sparse reward signal?
soundness: 4
strengths: 1, T-SAC significantly outperforms strong off-policy baselines such as SAC and CrossQ on the Meta-World ML1 benchmark and the FANCYGYM Box-Pushing tasks. Its 58% success rate on the sparse-reward Box-Pushing variant is particularly notable, clearly demonstrating the effectiveness of its long-horizon reasoning capabilities.

2, While N-step returns are known to suffer from high variance as n increases, the proposed gradient averaging technique enables stable training for horizons as long as n=16. This challenges the conventional wisdom of using small n and provides a robust method for learning long-term dependencies.

3, The paper provides a clear justification for its architectural choices through rigorous ablation studies. It demonstrates the critical importance of the self-attention and causal mask components in the Transformer critic and shows its superiority over recurrent backbones like GRU and LSTM.
summary: This paper addresses the challenge of long-horizon credit assignment in standard step-based reinforcement learning. It introduces T-SAC, an algorithm that replaces the standard MLP critic in Soft Actor-Critic (SAC) with a lightweight Transformer. This new critic conditions on short sequences of actions from the replay buffer and is trained on N-step returns across multiple horizons without requiring importance sampling, using a gradient averaging scheme to ensure stability. On long-horizon and sparse-reward control benchmarks like Meta-World ML1 and Box-Pushing, T-SAC is shown to achieve superior sample efficiency and final performance compared to strong off-policy baselines.
weaknesses: 1, The performance of T-SAC is not uniform across all benchmarks. On Gymnasium MuJoCo, it performs worse than the standard SAC baseline on Hopper and Walker2d.The authors attribute this to the need for an "action mask" on tasks with "high trajectory variability", which points to a limitation in the method's generality.

2, The critic conditions on multi-step action sequences (at, ..., at+n-1) generated by the current, often random, policy. Early in training, these sequences are noisy and suboptimal, potentially introducing significant variance into the critic's learning target. This may slow down the initial convergence speed compared to standard critics that condition only on a single (s, a) pair. This effect could be particularly pronounced in dense-reward settings like mujoco, where the benefit of a clean, immediate one-step TD target is high.

3, The proposed critic-parameter freezing schedule (Sec 4.4), which enables training without a target network, is only demonstrated on locomotion tasks. The authors admit it is unstable in sparse-reward settings, where the paper's main results are achieved using conventional Polyak averaging. This makes the target-free contribution a secondary point with limited applicability.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How sensitive is T-SAC to the choice of N-step horizon and freezing interval?

2. Would the approach extend to visual or partially observable environments?

3. Could actor and critic share a Transformer backbone for further efficiency?

4. How does T-SAC compare computationally to CrossQ and TOP-ERL at scale?
soundness: 3
strengths: - The idea of chunking the critic rather than the actor is novel and conceptually clean, offering a new perspective on temporal abstraction in off-policy RL.

- The removal of importance sampling through prefix-conditioned targets is a practical and effective simplification.

- The paper combines Transformer critics, multi-horizon learning, and critic freezing in a coherent design that improves stability without target networks.

- Experiments are broad and rigorous, with thorough ablations verifying each design component.

- Writing is clear, structured, and technically sound.
summary: This paper proposes T-SAC (Transformer-based Soft Actor-Critic), which enhances standard SAC by introducing a sequence-conditioned critic that models short trajectory segments via a lightweight Transformer and trains with aggregated N-step returns—without importance sampling. The approach enables long-horizon credit assignment while keeping the actor update one-step. A simple critic-freezing schedule further removes the need for target networks. Experiments on various benchmarks show consistent improvements over SAC, CrossQ, and other baselines, demonstrating that sequence modeling on the critic side can improve sample efficiency and stability in long-horizon control.
weaknesses: - The novelty is largely architectural; theoretical justification is limited, and the benefits of critic-side chunking lack deeper analysis.

- All results use low-dimensional state inputs; no experiments on visual or partially observable tasks.

- Statistical significance of gains and computational cost comparisons are not deeply analyzed.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. What is the subset of MDPs that have the property that the Transformer-based action chunking can be provably useful?
This is an important question. As you can see, the proposed method does not perform well in Mujoco.
2. Could you discuss the computational cost and scalability of the proposed method in detail, especially as the sequence length increases? 
3. Could you provide a more detailed analysis of how your Transformer-based action chunk gain was observed in your experiments? In particular, in addition to the comparison of rewards in the paper, is there an experiment that more intuitively demonstrates the significant improvement in sample efficiency?
soundness: 2
strengths: 1. The paper is overall well-organized and easy to follow. 
2. The proposed transformer critic is intuitive and is supported by several theoretical results.
summary: This article extends the work in the literature [1] by introducing a sequence-conditioned critic for SAC that models trajectory context with a light-weight Transformer and trains on aggregated N-step returns. Empirical results on standard online RL benchmarks show that the proposed method sometimes outperforms other algorithms.


[1]Qiyang Li, Zhiyuan Zhou, and Sergey Levine.Reinforcement learning with action chunking.arXiv  preprintarXiv:2507.07969,2025.
weaknesses: 1. The improvement of the algorithm in the experiments is not significant for a basic benchmark, Mujoco.
The authors claim that an action mask is needed for T–SAC and may degrade performance with high trajectory variability.
T-SAC is better than other methods in Meta-World tasks, however, there is higher trajectory variability in Meta-World tasks than that in Mujoco.
2. The authors of literature [1] claim that we might desire a final optimal Markovian policy, the exploration problem can be better tackled with non-Markovian and temporally extended skills, and that action chunking offers a very simple and convenient recipe for obtaining this. 
*However,* they did not provide any theoretical explanation for the aforementioned phenomenon. They also did not offer any intuitive examples to demonstrate the effectiveness of action chunking in MDP tasks. The same problems are shown in this paper.
3. I think the major contribution of this work is the proposed causal Transformer. Thus, in addition to the performance gain by inserting the causal Transformer into SAC, it would be good to discuss other metrics that can directly evaluate the "quality" or "informativeness" of representations learned in the causal Transformer. Such metrics might include Centered Kernel Alignment and Mutual Information Neural Estimation.
4. The contribution is ambiguous. An alternative interpretation of the benefit from action chunking and T-SAC is that the model is simply given a richer input feature set, which allows a powerful function approximator (like a deep neural network) to better fit a complex value or policy function. 
5. The new method should be compared with QC, which is proposed in [1].


*If my concerns and questions are all addressed, I will raise the score.*


Decision: Accept (Poster)