Title: Rethinking Tokenized Graph Transformers  for Node Classification
Venue: NeurIPS 2025
Accepted: yes
Keywords: graph Transformer

token swapping

token sequence

node classification
Abstract: Node tokenized graph Transformers (GTs) have shown promising performance in node classification. The generation of token sequences is the key module in existing tokenized GTs which transforms the input graph into token sequences, facilitating the node representation learning via Transformer. In this paper, we observe that the generations of token sequences in existing GTs only focus on the first-order neighbors on the constructed similarity graphs, which leads to the limited usage of nodes to generate diverse token sequences, further restricting the potential of tokenized GTs for node classification. To this end, we propose a new method termed SwapGT. SwapGT first introduces a novel token swapping operation based on the characteristics of token sequences that fully leverages the semantic relevance of nodes to generate more informative token sequences. Then, SwapGT leverages a Transformer-based backbone to learn node representations from the generated token sequences. Moreover, SwapGT develops a center alignment loss to constrain the representation learning from multiple token sequences, further enhancing the model performance. Extensive empirical results on various datasets showcase the superiority of SwapGT for node classification.
Code is available at https://github.com/JHL-HUST/SwapGT.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe tokenized graph Transformer has been a popular architecture of graph Transformers for node classification in recent years. While this paper reveals a common limitation of existing tokenized graph Transformer that these methods only leverage limited graph information to construct the token sequences, resulting in unsatisfied performance in several situations (e.g., sparse training data). To address this issue, the authors propose a new method called SwapGT which introduces a novel operation, token swapping. This proposed operation can generate diverse token sequences based on the initial token sequence, like a data augmentation strategy. In the end, the generated token sequences enhance the performance of graph Transformers for node classification, especially in the data sparsity situation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThis paper is well organized and easy to follow.\n2.\tThe proposed token swapping operation is novel.\n3.\tThe authors provide the corresponding theoretical analysis of the generated token sequences.\n4.\tThe empirical results show promising performance.\n\nWeaknesses:\n\n1.\tSome modules lack motivation.\n2.\tExperimental results need deeper and more insightful discussions.\n3.\tThe baselines need necessary introductions, and the reason of selecting baselines is also required.\n\nQUESTIONS:\n1.\tBased on the previous tokenized graph Transformers (e.g., NAGphormer), the readout function is also an important module. However, the proposed readout function in Eq. (7) lacks necessary motivations. Please provide the insights for the proposed strategy.\n2.\tMoreover, do authors try other readout functions, such as attention-based method in NAGphormer?\n3.\tAlthough the experimental results in both dense splitting and sparse splitting show the promising results, these results lack deeper discussion. For instance, I notice that SwapGT show more significant improvement in the sparse splitting. Can you explain? \n4.\tThere is no introduction for baselines. Why do you choose these methods? Can you provide the corresponding reasons?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposes a novel tokenized Graph Transformer for node classification by developing a novel token swapping operation that flexibly swaps tokens in different token sequences,called SwapGT,significantly outperforming several GTs and GNNs on extensive experiments.

STRENGTHS AND WEAKNESSES:
Strengths:
- S1. The method proposed in this paper enhances the model’s ability to capture rich node representations.
- S2. Experiments on real-world datasets demonstrate that SwapGT achieves better performance on node classification, compared with several representative GNNs and GTs.

Weaknesses:
- W1. Limited Novelty. The novelty is not clearly articulated, and the distinction from prior tokenized GTs needs further clarification. This paper also lacks a discussion between the proposed method and GTs with global attention.
- W2. Performance Improvement. Although the proposed SwapGT addresses the performance degradation in sparse splitting, it seems that GNNs do not suffer from sparse splitting, which weakens the significance of performance improvement.
- W3. Missing limitations. The limitations of the proposed method are not adequately discussed, potentially limiting readers’ understanding of its scope and applicability.

QUESTIONS:
See the weakness part.

LIMITATIONS:
No. The authors do not adequately address the limitations of their study and its potential future optimization directions of their work.

- Cons of the novel token swapping operation proposed by SwapGT?

- Any other more efficient method to enhances the model’s ability to capture rich node representations?
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel tokenized graph Transformer named SwapGT for the node classification task. Specifically, the authors firstly analyze the drawbacks of token generation in existing tokenized graph Transformers. Then, they design a new token generator called token swapping. By swapping tokens between different token sequences, the proposed token generator can generate diverse token sequences which carry rich graph information. In addition, the authors also develop a center-alignment loss to help the model learn node representations from multi-token sequences. Extensive results indicate that SwapGT outperforms other mainstream methods in the node classification task, such as GNNs and GTs.\n\nSTRENGTHS AND WEAKNESSES:\nThe strengths and weaknesses are summarized as follows:\n\nStrengths:  \n1. The motivation of SwapGT is clear.\n2. The idea of swapping token between different token sequences is new and interesting.\n3. The paper is well written.\n4. The experiments of performance comparison are extensive.\n\nWeaknesses:  \n1. The authors only consider standard Transformer.\n2. The experimental section lacks additional ablation studies.  \n3. The analysis of the key hyper-parameter is also required.\n\nQUESTIONS:\nThe questions are summarized as follows:\n\n1. According to Fig. 2, the proposed SwapGT can be regarded as a generalized framework where the Transformer layer is just utilized to learn node representations from input token sequences. In this way, the learning module is optional. I am curious about the performance of SwapGT when integrated with other Transformer variants, such as linear Transformer and PolyFormer.  \n2. Based on Q1, I suggest authors conduct additional ablation studies to validate the influence of learning module on model performance, just like the ablation studies in GraphGPS.\n3. I have noticed that $\\alpha$ in Eq. (8) is an important hyper-parameter. So, can you provide the necessary experiments to explore the influence of $\\alpha$ on model performance? \n4. In A.4, the authors provide the results of SwapGT with different token sampling size. I have found that sparse splitting prefers larger sampling size. Can you provide some insightful discussions for this phenomenon? \n5. What is the value of the swapping probability $p$ in practice?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies a key limitation in existing tokenized Graph Transformers (GTs), where the token sequences for a target node are generated only from its immediate (1-hop) neighbors on a pre-constructed similarity graph. The authors argue this restricts the diversity of information available to the Transformer, limiting its performance, especially in sparse data settings. To address this, they propose SwapGT, a novel tokenized GT framework.\n\nThe core contribution is a token swapping operation, a data augmentation technique that creates multiple, diverse token sequences for each node by iteratively swapping tokens with those from their neighbors' token sets, effectively sampling from multi-hop neighborhoods on the similarity graph. Extensive experiments show that SwapGT consistently outperforms a wide range of GNN and GT baselines, particularly in sparse training scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n-  The paper does an excellent job of identifying a concrete and sensible limitation in the current design of tokenized GTs—the restricted sampling space for token generation. The framing of this process as a 1-hop neighborhood selection on a k-NN graph is insightful and provides a strong, clear motivation for the proposed solution.\n\n- The proposed token swapping operation is an original and well-designed solution to the identified problem. It serves as an elegant, graph-specific data augmentation technique that expands the model's view to multi-hop semantic neighbors in a controlled fashion.\n\n- The quality of the proposed method is demonstrated through rigorous experiments on eight benchmark datasets, including both homophilous and heterophilous graphs. The inclusion of standard deviation in the result tables further strengthens the reliability of these claims.\n\n- The paper is well-written, clearly structured, and easy to follow. The overall framework is well-illustrated in Figure 2, and the token swapping algorithm is precisely detailed in Algorithm 1. The authors provide thorough ablation studies for each component of their method and include detailed experimental settings in the appendix, which aids in understanding and reproducibility\n\nWeaknesses\n- The method introduces several new hyperparameters, including the swapping times t, swapping probability p, number of augmented sequences s, and the fusion coefficient α. The parameter analysis in Figures 5 and 6 shows that model performance is sensitive to \nt and s, and their optimal values vary across datasets. This reliance on careful tuning might limit the method's practical, out-of-the-box applicability.\n\n- The paper includes a probabilistic analysis in Fact 1, showing that the swapping mechanism preferentially samples lower-hop neighbors on the k-NN graph. While this is a nice sanity check to show the process isn't completely random, it is a very light theoretical contribution. It does not provide any formal guarantees on the quality of the learned representations or why this specific sampling strategy leads to better generalization compared to others.\n\n- The study on token generation strategies in Section 5.5 compares SwapGT against two \"naive\" variants: using a single, longer sequence (SwapGT-L) and randomly sampling from a larger pool (SwapGT-R). While SwapGT outperforms them, these baselines were designed by the authors for the ablation. The study would be more compelling if it compared the token swapping strategy against other established multi-hop or graph augmentation techniques from the literature, adapted to the tokenized GT context.\n\nQUESTIONS:\n- The entire method depends on the initial k-NN graph. How sensitive is the model's performance to the choice of k in this initial step? Does a poor choice of k disproportionately harm SwapGT compared to a standard 1-hop GT?\n\n- The token swapping operation is presented as the best way to leverage multi-hop information. How does this strategy compare, both conceptually and empirically, to other established multi-hop sampling techniques?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}