Title: Relieving the Over-Aggregating Effect in Graph Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: graph transformers

message passing

graph neural networks
Abstract: Graph attention has demonstrated superior performance in graph learning tasks. However, learning from global interactions can be challenging due to the large number of nodes. In this paper, we discover a new phenomenon termed over-aggregating. Over-aggregating arises when a large volume of messages is aggregated into a single node with less discrimination, leading to the dilution of the key messages and potential information loss. To address this, we propose Wideformer, a plug-and-play method for graph attention. Wideformer divides the aggregation of all nodes into parallel processes and guides the model to focus on specific subsets of these processes. The division can limit the input volume per aggregation, avoiding message dilution and reducing information loss. The guiding step sorts and weights the aggregation outputs, prioritizing the informative messages. Evaluations show that Wideformer can effectively mitigate over-aggregating. As a result, the backbone methods can focus on the informative messages, achieving superior performance compared to baseline methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discovers a new concept, over-aggregating, which aggregates a large volume of messages with less discrimination.\nTo address it, the authors propose Wideformer, a plug-and-play method for graph attention. Wideformer divides the aggregation of all nodes into parallel processes and guides the model to focus on specific subsets of these processes.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths\n1. The paper identifies a novel phenomenon called \"over-aggregating\" in graph transformers, where a large volume of messages is aggregated into a single node with less discrimination, leading to potential information loss. This insight is important for understanding and improving graph transformer models.\n‌Proposed Wideformer Method‌:\n2. The Wideformer method is an effective approach to relieve the over-aggregating effect. By dividing the aggregation of all nodes into parallel processes and guiding the model to focus on informative subsets, Wideformer can maintain the global receptive field while reducing information loss.\n3. The paper provides a theoretical analysis to justify the over-aggregating effect and the effectiveness of the proposed Wideformer method. This enhances the credibility of the claims.\n\n* Weaknesses & Questions\n1. In Sec. 3.3, I noticed that over-smoothing and over-aggregating have different definitions. The first concern: Is their influence on graph learning different?  I also noticed sentences in P4.L156-157. Since the authors think higher entropy suggests similar attention scores, I believe that for each node/token whose representation seems to be similar after this layer. Isn't it a low-pass filtering effect?\n2. When and how do the authors use Algorithm 1? Before training or during training? Are centers in different layers different?\n3. The paper uses a simplistic KMeans++ clustering method to divide the nodes into clusters. A more sophisticated clustering approach based on learned node representations may lead to better performance.\n4. While the experiments cover a diverse set of datasets, the evaluation on large-scale graphs is limited. Evaluating Wideformer on graphs with millions of nodes would provide insights into its scalability.\n5. Question about algorithm 1. In the loop, $\\\\mathbf{C}$ becomes a matrix, not a vector anymore. I'm confused about it.\n\nQUESTIONS:\nAbove.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper defines the concept of attention entropy, proposes a new algorithm that divides global aggregation by clusters and guides the aggregation by the significance of the clusters, which is supposed to reduce the attention entropy. Experiments show that the algorithm is effective in improving the performance of several graph transformers on various datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: Define the concept of attention entropy; propose a new algorithm that divides global aggregation by clusters and guides the aggregation by the significance of the clusters. It is written in plain language and has a handful of plots that attempt to corroborate the effectiveness of its method.\n\nWeaknesses: \n1. Theorem 3.1 (at line 122) is a correct but not meaningful statement. You can always show that entropy is lower bounded by 0, which is obtained when $\\alpha$ has a point mass distribution. This statement does not suffice to support the exact claim (at line 126) \"graph attention will have higher attention entropy when aggregating more graph nodes\", and fig.1(b) does not conform to the monotonicity in the Theorem when the # of nodes increases from 1e+4 to 1e+6.\n2. The paper does not show a clear causal relation (or correlation) between reduction of attention entropy and better generalization. Empirically, the algorithm reduces the attention entropy of all three architectures significantly, but the test accuracy is improved marginally. Moreover, the most severe reduction occurs on polynormer, but the improvement of polynormer is not superior to that of the other two architectures.\n3. The idea of using cluster information in the network data is not unique, even in my package of reviewed assignments.\n\nQUESTIONS:\n1. Is there a possibility that over-aggregation is not a significant factor for the generalization of graph transformer models, or, on the contrary, why is it a significant one?\n2. Can you find an approach to demonstrate that reducing the attention entropy deterministically improves the generalization of the model?\n3. In your abstract, you mentioned that your method reduces information loss. Can you be more specific about the information loss in the original transformer structures? How do you measure the information loss? To what extent does your algorithm help reduce it?\n\nLIMITATIONS:\nThere is no negative societal impact of the work.  \nThe limitations of the paper: See the questions and weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies over-aggregating, a tendency of graph-transformer attention to assign nearly uniform weights when many nodes are present, which “dilutes” salient messages. It measures the effect with attention-entropy and shows that the entropy lower-bound grows with graph size. To counter this, the authors introduce Wideformer, a plug-and-play module that divides the global receptive field into m query-based clusters using a K-means++ seeding routine (Alg. 1) and aggregates each cluster independently. It guides attention by re-weighting and sorting the cluster outputs with a second soft-max so each node can focus on informative clusters. Wideformer is inserted atop three linear graph-transformer backbones (GraphGPS, SGFormer, Polynormer) and evaluated on 13 node-classification datasets spanning 1 K – 170 K nodes. It consistently lowers attention-entropy and yields accuracy or ROC-AUC gains of ~(0.2–1.2) pp on most benchmarks, with linear time/space O(n m) and modest wall-time overhead. An ablation shows both division and guidance are needed for peak gains. Limitations and societal impact are discussed.\n\nSTRENGTHS AND WEAKNESSES:\nQuality\t\nStrengths: Solid empirical study on 13 datasets; ablations and complexity analysis support claims; open-sourced code and reproducibility checklist.\nWeaknesses: Experiments focus only on node classification; no link-prediction or graph-level tasks; statistical significance of gains not reported. Limitation of K-means++ choice. The divide step uses K-means++, which may be suboptimal due to its assumptions. Have the authors considered differentiable clustering or adaptive-k strategies? An ablation comparing clustering algorithms or at least discussing these trade-offs would strengthen the paper.\n\nClarity\t\nStrengths: Motivation, method and equations are easy to follow; limitations clearly stated.\nWeaknesses: (i) Several figures are hard to interpret: e.g., in Figure 6 the numbers actually label clusters, but this is never explained in the caption or legend, so a reader cannot tell what they mean without digging into the text. More descriptive legends or color-coding would help. (ii) Fonts in many plots are tiny. (iii) Multiple spelling glitches (“Cluter”, “Large”, “compexity”) and inconsistent capitalization of “Wideformer.”\n\nSignificance\nStrengths: Tackles a practical performance bottleneck in linear graph transformers and can be adopted by many models; modest but consistent gains on real data.\t\nWeaknesses: Incremental relative to existing sparse-attention or cluster pooling work; gains may be negligible in applications where entropy is already low.\n\nOriginality\nStrengths: Introduces the specific notion of over-aggregating and couples a two-step divide-and-guide scheme not present in prior work.\nWeaknesses: Concept overlaps with known uniform-attention phenomena; division uses standard K-means++ rather than a novel algorithm.\n\nQUESTIONS:\nThe work is technically sound, addresses a practical issue, and shows consistent improvements with minimal cost. Expansion to broader tasks and deeper statistical analysis would elevate its impact. \n\nGenerality beyond node classification. Evaluate on graph-level tasks or link prediction.\n\nStatistical robustness. Add paired-t tests or confidence intervals; some gains lie within one standard deviation.\nFigure interpretability. Revise Figure 4 (requires description) and Figure 6 lacks a informative legend explaining that numeric labels denote clusters: similar plots, enlarge fonts. \n\nProofreading. Fix spelling errors (\"Cluter\", \"compexity\", \"Targe\") and standardize “Wideformer” capitalization.\n\nThe brief description of over-smoothing in §3.3 is hard to follow. Please consider rewriting this paragraph with a concise definition, one illustrative example\n\nLIMITATIONS:\nAuthors explicitly acknowledge that (i) the current division and guidance strategies are simple and may fail in some regimes and (ii) only a limited search over clustering and weighting strategies is explored. I concur and suggest emphasizing scalability to million-node graphs and tasks beyond classification. Yes, limitations are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Wideformer, a plug-and-play enhancement to graph attention mechanisms designed to address a newly identified challenge in graph transformers: over-aggregating. The authors observe that as the number of nodes increases, attention scores tend to become uniformly distributed, leading to diminished discriminatory power during message aggregation. Wideformer mitigates this issue by employing a divide-and-conquer strategy, in which parallel aggregation modules each focus on restricted subsets of nodes. This design reduces the information burden per aggregation unit and alleviates message dilution, thereby enhancing the model’s ability to capture salient information in large graphs.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\n1. I personally enjoyed Wideformer’s design—partitioning attention aggregation and enforcing selective focus—which mirrors established strategies in managing information overload in attention-based models (e.g., mixture-of-experts). The method is described clearly with intuitive steps: parallel aggregation, cluster-wise processing, and attention-guided selection.\n\n2. The paper is built upon a solid motivation which effectively situates Wideformer among two dominant solution categories for attention scalability: sparse attention and linear attention. It convincingly argues that while these approaches reduce complexity, they may exacerbate over-aggregation under large node counts.\n\n3. The paper is based on strong experimental results which validates that Wideformer achieves state-of-the-art performance across 13 well-known benchmarks in Graph Processing.\n\n[Weaknesses]\n\n1. Although \"over-aggregating\" is introduced as a distinct problem, its symptoms resemble aspects of over-squashing, over-smoothing, and over-dilution (where important messages are lost due to limited representation capacity). I have not been convinced that \"over-aggregating\" is a distinct and unique problem that requires special solution sets.\n\n2. Wideformer mitigates over-aggregation by increasing the number of parallel aggregation units. But I would have enjoyed a more thorough analysis on the computational overhead or memory cost. How does it compare in runtime efficiency after its \"late-aggregation\" style where it runs at subsets first and aggregated later. A brief discussion or analysis of efficiency vs. accuracy trade-offs is necessary, especially since scalability is a key motivation.\n\n3. The most major concern is that while the problem is introduced in Graph Neural Network, this is a central issue in both Graph-based [1] and Transformer-based methods that could be considered as fully connected graphs. In this aspect, I personally do not think that Wideformer is either unique nor novel, since simply restricting the attention scope into a subset of the input token has been presented in multiple previous works in Transformer attention [2,3,4].\n\n[1] Even Sparser Graph Transformers (NeurIPS 25, https://neurips.cc/virtual/2024/poster/95681)\n[2] Stabilizing Transformer Training by Preventing Attention Entropy Collapse (ICML'23, https://proceedings.mlr.press/v202/zhai23a/zhai23a.pdf)\n[3] Attention is not all you need: pure attention loses rank doubly exponentially with depth (ICML'21, https://arxiv.org/pdf/2103.03404)\n[4] In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness\n[5] ENACT: Entropy-based Clustering of Attention Input for Reducing the Computational Needs of Object Detection Transformers (https://arxiv.org/abs/2409.07541)\n[6] Scratching Visual Transformer's Back with Uniform Attention (ICCV'23, https://arxiv.org/pdf/2210.08457)\n\nQUESTIONS:\nMy major concerns are written in the weakness section above.\n\nLIMITATIONS:\nNo. Refer to my weaknesses please.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-07"}