Title: Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems
Venue: NeurIPS 2025
Accepted: yes
Keywords: Attention Mechanism

Transformers

Hierarchical Data

Multi-modal Data

Geometric Deep Learning
Abstract: Transformers and their attention mechanism have been revolutionary in the field of Machine Learning. While originally proposed for the language data, they quickly found their way to the image, video, graph, etc. data modalities with various signal geometries. Despite this versatility, generalizing the attention mechanism to scenarios where data is presented at different scales from potentially different modalities is not straightforward. The attempts to incorporate hierarchy and multi-modality within transformers are largely based on ad hoc heuristics, which are not seamlessly generalizable to similar problems with potentially different structures. To address this problem, in this paper, we take a fundamentally different approach: we first propose a mathematical construct to represent multi-modal, multi-scale data. We then mathematically derive the neural attention mechanics for the proposed construct from the first principle of entropy minimization. We show that the derived formulation is optimal in the sense of being the closest to the standard Softmax attention while incorporating the inductive biases originating from the hierarchical/geometric information of the problem. We further propose an efficient algorithm based on dynamic programming to compute our derived attention mechanism. By incorporating it within transformers, we show that the proposed hierarchical attention mechanism not only can be employed to train transformer models in hierarchical/multi-modal settings from scratch, but it can also be used to inject hierarchical information into classical, pre-trained transformer models post training, resulting in more efficient models in zero-shot manner.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new mechanism attention that inherently supports both multi-modal and multi-scale problems. It starts by introducing a formalism of multi-scale problems as trees of nested signals. It then uses this formalism and the following logic to obtain a new formulation of the attention mechanism. First, given key K and queries Q, one can express the conditional entropy of Q given K, and try to minimize it to increase the information. Second, because this entropy is intractable, one expresses an upper bound using variational distribution. By considering a Boltzman distribution and computing the gradient descent for the query, we can obtain new formulations of the attention mechanism which depend on the energy function of the Boltzman prediction. With a particular energy function we can even retrieve an attention quite similar to the original softmax attention. Using another energy function one can rewrite the attention matrix as a block matrix. It means, still considering the hierarchical structure of the problems as a tree, attention value is uniform across leaf elements that belong to different branches. This effectively reduces the number of values to learn.\n\nThe first experiment evaluates the performance of this new hierarchical attention on sentiment analysis. Texts are split into paragraphs, sentences and tokens. The hierarchical attention outperforms the traditional one. The second experiment evaluate news classification in a multi-modal context. The proposed attention yield to slightly worse results but for a much cheaper computation.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper presents an original per-block attention that natively  support for multi-modal and hierarchical settings\n- Experimental results show the new attention mechanism performs on par with the traditional one for a cheaper cost.\n    \n## Weaknesses\n\n- The paper is dense, which makes it difficult to follow despite of the clarity of the different exposition steps. The paper may gain in readability with some reorganisation. Some parts of the mathematical justification behind the new attention mechanism may be superfluous. For instance, is it necessary to emphasize so much how to find back with the new formulation an attention that is close to the traditional one? The per-block attention matrix that supports hierarchical settings should be brought forward sooner in the paper.\n    \n- This new attention formalism has the potential to impact many problems due to its support for hierarchical and multi-modal settings. The experimental settings could be larger to assess this impact. For instance, it could include comparison with other methods that were design with hierarchy in mind like Swin Transformer.\n\nQUESTIONS:\n- Could comparison with models dedicated to hierarchical settings like Swin Transformer be added?\n- Would it be possible to reduce the section about finding back a similar attention as the traditional softmax one, to bring forward the per-block attention matrix? It would be also the opportunity to bring forward the dynamic programming method to compute the hierarchical attention, which is pointed as a major contribution but relegated to the appendix.\n\nLIMITATIONS:\nMore experimental evaluation of the hierarchical self-attention would help assessing its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper introduced hierarchical structure into the soft-max attention mechanism in a mathematically interpretable manner. The authors first derived the softmax attention layer as similar to one step gradient update of an entropy minimization problem. Then generalize the attention layer to nested signals by introduce hierarchical structure into the variables/nodes

STRENGTHS AND WEAKNESSES:
Strengths:
+ The network architecture are derived from unrolling, which leads to an interpretable network.
+ Introduce the hierarchical structure into the attention map.
+ Clear performance gain and also computations cost reduced due to hierarchical structure

Weaknesses:
+ Lack experiments to demonstrate the interpretability of the model, Ex: do more than 1 gradient steps ?, plot reduced entropy objective for various number step/layers ?
+ The signal hierarchy are not always obvious for various datasets/application: graphs signals. The paper don't introduce any mathematical methods to create hierarchy from data and mainly use intuition.

QUESTIONS:
NA

LIMITATIONS:
+ Have the authors try to do more than 1 gradient step update ?
+ How the performance vary according to different signal hierarchy ? Is the a measurement to quantify the robustness of one hierarchy to another aside from doing the training process.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel mathematical concept termed nested signal to formally represent hierarchical and multi-modal information within the Transformer framework. Based on this, this paper derives Hierarchical Self-Attention (HSA), a generalization of the standard Softmax attention mechanism that incorporates hierarchical structure. The paper further presents an efficient dynamic programming algorithm to compute HSA and demonstrates its effectiveness on both unimodal and multimodal tasks. Empirical results on IMDB, Elec, and N24News datasets show performance improvements over baselines.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n\nThe nested signal concept is new and interesting.\n\nThe design of Hierarchical Self-Attention appears principled and technically sound.\n\nAddressing multi-level, multi-modal learning within a unified Transformer-based framework is an important direction, especially given the increasing complexity of modern AI tasks.\n\nThe proposed method shows empirical improvements on several benchmarks (IMDB, Elec, N24News), indicating its potential practical utility.\n\n# Weaknesses:\n\nBased on my understanding, the proposed method requires the hierarchy depth to be explicitly pre-defined. This rigid design may limit its flexibility and scalability, especially when dealing with tasks or datasets where the hierarchical structure varies or is not clearly defined. This contrasts with traditional Transformers, which often benefit from scaling data or model size without making strong structural assumptions.\n\nAlthough the paper claims to address multi-modal scenarios, only the N24News dataset is used to demonstrate this aspect. Moreover, the reported improvements over baselines like FSA are relatively modest, and the chosen baselines are not particularly strong or recent.\n\nWhile I agree with the authors' experimental design principle of isolating contributing factors, limiting multimodal evaluation to a single dataset with weak baselines undermines the strength of the multi-modal claim. Additional, more convincing evidence is needed to support this aspect.\n\nQUESTIONS:\nIs it possible to dynamically infer or adapt the hierarchy depth rather than requiring it to be pre-specified? How would the method generalize to scenarios with unknown or evolving hierarchical structures?\n\nCan the authors provide further empirical evidence for the multi-modal applicability of the proposed method, ideally on more diverse and competitive benchmarks?\n\nHow does HSA interact with modern large-scale pre-trained vision-language models or other Transformer variants? Could HSA be integrated into such models, and if so, what would be the expected benefit?\n\nLIMITATIONS:\nThe requirement to pre-define the hierarchy limits the method's flexibility and may restrict its applicability to tasks with fixed or clearly known hierarchical structures.\n\nThe experimental evaluation for multi-modal scenarios is limited, relying on a single dataset with relatively weak baselines, making it hard to draw strong conclusions about multi-modal performance.\n\nThe computational overhead of the proposed HSA, despite claimed efficiency improvements, is not thoroughly analyzed or compared to standard attention in practical large-scale settings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper introduces hierarchical self attention (HSA) to reason over data with hierarchies or multiple modalities. The paper introduces nested signal model for such data, and then perform entropy minimization to obtain HSA. They demonstrate that such a representation is KL-closest block-constrained approximation of standard soft max attention. The authors show that HSA increases accuracy on long-text sentiment tasks (ex: IMDB) and outperforms baselines such as DeepSet and vanilla transformer on multimodal news classification benchmark.

STRENGTHS AND WEAKNESSES:
Strengths
1. Main strength of the paper it's clean derivation from first principles with strong grounding provided by KL-optimality proof. 
2. The dynamic programming implementation wit black box sub attention for plugging in fast approximation helps bring down the computation complexity significantly. 
3. It can be used as dtop in replacement for softmax attention without increasing memory footprint.

Weaknesses
1. Main supervised tasks are on a limited scale (ex: classification), and there is no token level evaluation of generation limiting the ability to understand the qualitative improvements obtained for generalized inference.
2. The baselines such as DeepSet and flat attention for multimodal tasks are sparse, how does it perform on more complex tasks such as VQA, Image Description etc. where the model inference and training becomes complex. 
3. Main limiting factor of the proposed approach seem to be assumption of a fixed hierarchy. Is it possible to learn the hierarchy or how can this method be scaled where no existing parser is available.

QUESTIONS:
Please see weakness section

LIMITATIONS:
None
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-08