Title: Less is More: an Attention-free Sequence Prediction Modeling for Offline Embodied Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Offline Reinforcement Learning

Embodied Learning
Abstract: Offline reinforcement learning (offline RL) is increasingly approached as a sequence modeling task, with methods leveraging advanced architectures like Transformers to capture trajectory dependencies. Despite significant progress, the mechanisms underlying their effectiveness and limitations remain insufficiently understood. We conduct a thorough analysis on the representative Decision Transformer (DT) model using an entropy analysis and identify the inconsistencies in state-action-reward ($\langle s, a, R \rangle$) distributions causing attention ``dispersal". To address this, we propose a hierarchical framework that decomposes sequence modeling into intra-step relational modeling—handled by a Token Merger that fuses each $\langle s, a, R \rangle$ triplet—and inter-step modeling—handled by a Token Mixer across timesteps. We investigate several Token Merger designs and validate their effectiveness across various offline RL methods. 
Furthermore, our theoretical analysis and experimental results suggest that while Token Mixers are important, lightweight architecture can also achieve even better performance to more complex ones. We therefore propose a parameter-free Average Pooling Token Mixer, which, combined with a convolutional Token Merger, forms our final model, Decision HiFormer (DHi). DHi achieves a \textbf{73.6\%} improvement in inference speed and an \textbf{9.3\%} gain in policy performance on the D4RL benchmark compared to DT. DHi also generalizes well to real-world robotic manipulation tasks, offering both practical benefits and insights into sequence-based policy design for offline RL. Code and models are public at \href{https://wei-nijuan.github.io/DecisionHiFormer/}{project page}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper employs an entropy-based analysis to show that the Decision Transformer (DT) suffers from attention dispersal, which stems from inconsistent attention allocation across the state–action–reward triplet, and that DT also struggles to capture dependencies across timesteps, ultimately limiting its performance. Motivated by these findings and supported by extensive comparisons, the authors propose DHi. The model first uses a convolutional Token Merger to handle intra-step dependencies by fusing each triplet into a single token, and then applies a parameter-free pooling Token Mixer to model inter-step dependencies. DHi achieves strong results on multiple D4RL tasks and real-world robotic manipulation benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Stengths: \n- This work is well-motivated. The authors conduct a thorough analysis that pinpoints the shortcomings of DT and, based on these insights, devise the new DHi architecture, which improves DT’s performance.\n- A large suite of experiments on both D4RL benchmarks and real-world tasks convincingly demonstrates DHi's effectiveness.\n- The design of DHi is further supported by theoretical analysis and comprehensive ablation studies.\n- The paper is well written and easy to follow.\n\n## Weaknesses:\n- While Figure 2 conveys the main insights about attention-entropy, several finer details remain confusing.\n- The Theorem 3.1 would benefit from additional discussion or intuition explaining its assumptions.\n- The design of DHi seems to assume that decision making in offline RL is mainly driven by short-range dependencies, which could hinder DHi's performance on tasks requiring long-range planning (e.g., navigation in more complex maze environments).\n\nQUESTIONS:\n- **Q1:** The heat map on the right-hand side of Figure 2 is a bit confusing. If I understand correctly, in the upper plot each \"$s_i$\" actually corresponds to three columns (or rows), yet DT attends only to the middle column representing the \"state\". Moreover, the lower plot shows temporal attention scores, but DHi is attention-free; does this mean the figure merely replaces DT's triplet input at each timestep with the single token produced by the convolutional token merger? Finally, how is DT's \"inter-step modeling entropy\" across different timesteps computed? Figure 2 does not appear to include a corresponding illustration.\n\n\n- **Q2:** When introducing Theorem 3.1, the authors posit the assumption: “Assuming that the total attention entropy remains unchanged after adding the Token Merger.” What is the intuition behind this assumption? Furthermore, because the merged token changes in form after the Token Merger, the distribution of attention scores might also change. In that case, does the decomposed form stated in Theorem 3.1 still hold?\n\n- **Q3:** Building on Weakness 3, how would DHi perform on maze environments more complex than umaze? For long-horizon planning tasks, might an attention-based mixer be more appropriate?\n\n- **Q4:** How would DHi perform on Gym’s random and expert datasets?\n  \n\n**Other suggestion:** It might be better to replace Table 1 in the main text with Table 6 from the appendix. In the current Table 1, using a convolutional mixer seems to nullify, or even harm, the effectiveness of every merger. What could be causing this?\n\nLIMITATIONS:\nPlease see weakness 3.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to decompose sequence modeling for RL into a “token merger” that combines states, actions, and rewards for a specific timestep and a “token mixer” that combines across timesteps. They study various token mixers and mergers and propose the best one as Decision HiFormer, which uses pooling for the mixer and convolution for the merger. They show that empirically this outperforms DT on D4RL.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed hierarchical architecture is novel to the best of my knowledge, and it is interesting that you can have a version of DT that does not use attention\n- The authors perform extensive analysis of attention entropy and systematically consider multiple different ways of combining the tokens\n- The authors show that their model outperforms DT and other baselines on D4RL\n\n\nWeaknesses\n- This work seems incremental in the sense that how to combine the states, actions, and rewards seems more like a design choice/hyperparameter. It is not surprising that by adding a bit of complexity you can improve performance, but this is a tradeoff. The main appeal of DT is its simplicity.\n- It’s confusing that the choice of pooling for the mixer does not seem supported by Table 1, which shows that linear attention would actually be better. The text says that “There is limited performance variation between different token mixers” but the table seems to show significant variations in performance. Could the authors explain these discrepancies? \n- The main advantage of mechanisms that handle long-range dependencies is in tasks that require temporally extended reasoning. How do you ensure you do not lose this kind of reasoning ability with an architecture focused on local dependencies?\n- Empirically the proposed model performs very similarly to DC, which suggests that the benefit of the hierarchical structure might not be large\n\nQUESTIONS:\nSee Strengths and Weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Less is More: an Attention-free Sequence Prediction Modeling for Offline Embodied Learning\" introduces Decision HiFormer (DHi), an alternative to Transformer-based models in offline reinforcement learning.\n\nThe authors analyze the Decision Transformer's (DT) attention mechanism, finding \"attention dispersal\" due to inconsistent state-action-reward distributions, which hinders local dependency modeling.\n\nDHi is a hierarchical framework that addresses this by:\n1.  **Intra-step modeling:** A \"Token Merger\" (convolutional) fuses each (state, action, reward) triplet into a single representation.\n2.  **Inter-step modeling:** A lightweight, parameter-free \"Token Mixer\" (Average Pooling) captures dependencies across timesteps, preserving the Markov property.\n\nDHi significantly outperforms DT on the D4RL benchmark, achieving a 73.6% improvement in inference speed and a 9.3% gain in policy performance. It also demonstrates strong generalization in real-world robotic manipulation, yielding smoother trajectories and faster, more efficient inference.\n\nIn summary, the paper offers a new understanding of attention in offline RL, proposes an efficient and effective hierarchical model (DHi), and validates its benefits in both simulated and real robotic tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n*   **Significance:** Addresses crucial offline RL limitations (Transformer inefficiencies, local dependency issues), boosting performance (9.3% gain) and efficiency (73.6% faster inference), and demonstrating real-world robotic applicability.\n*   **Originality:** Introduces \"attention entropy\" for analyzing Transformer limitations. Proposes a novel hierarchical architecture (DHi) with a unique convolutional \"Token Merger\" for intra-step and a parameter-free \"Average Pooling Token Mixer\" for inter-step modeling.\n*   **Clarity:** Well-structured, clearly written, and effectively uses visualizations and tables to explain complex concepts and present results.\n*   **Quality:**\n    *   **Thorough Analysis of DT's Limitations:** The paper's empirical analysis of DT's attention mechanism using attention entropy provides a solid foundation for its proposed solution. The correlation between high attention entropy and performance drops is clearly demonstrated.\n    *   **Systematic Design and Evaluation of DHi Components:** The authors systematically investigate various Token Merger designs (straightforward, concatenation, self-attention, convolution, pooling, average) and Token Mixers, providing ablation studies (Table 1) to justify their architectural choices (convolutional Token Merger + Average Pooling Token Mixer).\n    *   **Strong Empirical Results:** DHi consistently outperforms various state-of-the-art baselines on the challenging D4RL benchmark across different domains (Gym, Adroit, Kitchen, AntMaze). The real-world robotic experiments further strengthen the empirical validation.\n    *   **Theoretical Justification:** The paper offers theoretical analysis, including Theorem 3.1 on entropy reduction with the Token Merger and proofs in the appendix for the pooling layer's expressiveness in Markov problems, bolstering the architectural decisions.\n\n\n\n### Weaknesses:\n\n*   **Quality:**\n    *   **Scope of \"Markov Property\" Preservation:** The claim of the pooling layer preserving the Markov property (Line 60) and its theoretical proof (Appendix A, B) are significant, but the implications of this for long-term credit assignment or complex, non-Markovian real-world tasks could be more thoroughly discussed. \n    *   **Robustness to Diverse Trajectory Quality:** The D4RL benchmark has varying trajectory qualities (e.g., medium-expert vs. medium-replay). While DHi performs well overall, a more nuanced discussion on how the attention dispersal issue and DHi's solution specifically manifest or perform across different levels of trajectory quality could provide deeper insights.\n\n*   **Significance:**\n    *   **Limited Generalization to Other RL Paradigms:** The paper explicitly states \"A limitation of this study is the lack of evaluation on the transferability of our approach to other RL paradigms, which we leave for future investigation\" (Lines 349-350). This suggests that while the hierarchical modeling is effective for offline RL (specifically sequence prediction), its broader applicability to online RL, model-based RL, or different policy representation forms remains an open question. It would be helpful to specify the type of problem where this method performs best.\n\nQUESTIONS:\n*   **Relation to other \"Attention-Free\" Transformers:** While the paper compares to DC (Decision ConvFormer) which uses convolutions, a more in-depth discussion of how DHi's hierarchical decomposition (Merger then Mixer) differentiates it from other non-Transformer architectures that also aim to reduce attention (e.g., S4, Mamba) would strengthen the originality claims. The specific combination of a convolutional merger and pooling mixer is novel, but the underlying ideas of local processing and simple mixers have precursors. \n\n\n*   **Computational Cost of Training vs. Inference:** While inference speed improves, the paper doesn't detail training time or resources needed for DHi versus DT or other baselines. This is vital for practical use and could affect cost-effectiveness.\n\nLIMITATIONS:\nPlease see the weakness section above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the limitations of Decision Transformer (DT) in offline RL, attributing its performance bottleneck to attention dispersal, i.e., high attention entropy that prevents effective local dependency modeling. To address this, the authors propose a hierarchical architecture, Decision HiFormer (DHi), composed of a Token Merger (fusing ⟨state, action, reward⟩ triplets at each timestep) and a Token Mixer (capturing inter-step dependencies via lightweight average pooling). The authors show that attention entropy correlates negatively with rewards, and their Token Merger reduces this entropy, improving performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The entropy–reward correlation is well-motivated and supported by attention map visualizations. Authors re-identify the issue of local dependency modeling and mitigates it with proposed Merger and Mixer methods.\n- Strong empirical results across D4RL tasks, with thorough ablations on important design decisions (e.g., Token Merger and Mixer variants).\n\nWeakness:\n- No standard errors or confidence intervals are reported, which is crucial for offline RL benchmarks with high variance. Table 2 has some values underscored and it's not clear what that signifies.\n- The proposed approach is essentially a variant of DT and thus inherits its core limitation of being a single-step method, lacking the ability to perform multi-step dynamic programming. As a result, its performance is expected to degrade significantly on datasets with very few or no near-optimal  trajectories, such as antmaze-medium and antmaze-large.\n\nQUESTIONS:\n- Why does avg pooling inplace of attention even work? It's only capturing information from adjacent previous tokens and that seems to be sufficient. Is this because the hopper task which is locomotion does not really require much past history context and hence pooling works fine? Consequently some other tasks that might require history (in terms of credit assignment) would give different results?\n- Instead of self-attention in tokenmerger if the authors could use cross attention with a learned query I wonder if it might give better performance?\n- Does dataset for real robot experiment has any suboptimal trajectories? \n- I'm open to raising my score if authors provide confidence intervals.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-23"}