SGHormerVQ: Bridging Graph Transformers and Spiking Neural Networks via Spiking Vector Quantization

Authors: Huizhe Zhang, Jintang Li, Yuchang Zhu, Liang Chen, Zibin Zheng
Primary area: learning on graphs and other geometries & topologies
Keywords: graph transformer, spiking neural network, vector quantization

Abstract:
Graph Transformers (GTs), which simultaneously integrate message passing and self-attention mechanisms, have achieved promising empirical results in some graph prediction tasks. Although these approaches show the potential of Transformers in capturing long-range graph topology information, issues concerning the quadratic complexity and high computing energy consumption severely impair the scalability of GTs on large-scale graphs. Recently, as brain-inspired neural networks, Spiking Neural Networks (SNNs) provide an energy-saving deep learning option with lower computational and storage overhead via their unique spike-based event-driven biological neurons. Inspired by these characteristics, we propose SGHormerVQ, which bridges efficient Graph Transformers and spiking neural networks via spiking vector quantization. Spiking vector quantization generates implied codebooks with smaller sizes and higher codebook usage to assist self-attention blocks in performing efficient global information aggregation. SGHormerVQ effectively alleviates the reliance on complex machinery (distance measure, auxiliary loss, etc.) and the \textit{codebook collapse} present in previous vector quantization-based GNNs. In experiments, we compare SGHormerVQ with other state-of-the-art baselines on node classification datasets ranging from small to large. Experimental results show that SGHormerVQ has achieved competitive performances on most datasets while maintaining up to 518× faster inference speed compared to other GTs.

Decision: reject
Reviewer scores: 3, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730250877475

thread_type:
official_review

<thread_object>
creator: reviewer_nqbn

<content>
summary:
The paper introduces SGHormerVQ, a Graph Transformer model that integrates spiking neural networks through spiking vector quantization (SVQ) to enhance efficiency in graph learning tasks. By using spiking neurons as quantizers, SGHormerVQ achieves faster inference and reduces computational overhead while maintaining competitive performance on node classification benchmarks.

strengths:
The paper is very well-written and looks into an important research question.

weaknesses:
1. I am not sure what is the key novelty of the work - it seems the authors have just used Spiking Vector Quantization (which is itself not a new concept) for spiking graph transformers (also not new)

2. The experimental section is very weak as most of the datasets used for the results are very simple datasets.

3. The paper gives a lot of context and discusses a lot about the prior works. i would recommend to push them to the supplementary section and add more experimental results to show where this model actually works and where it fails- as that is not really clear from the experiments

questions:
1. I am confused by the statement "SGHormerVQ with better performance infers faster thanotherGTs by up to 518x" as the Fig 4 shows that it has higher latency than the standard GAT with slightly higher accuracy? Also, the authors only show this for the simplest dataset (Physics dataset) - I would recommend the authors to also show the latency results for the more complex datasets. Also, how are you getting the latency numbers? Is it just the average for all the inferences in the test set?

2. I would recommend the authors to give a detailed analysis of energy and memory usage would give solid proof of the paper's efficiency claims.The authors could measure the average number of spikes per inference and that would be a good measure for the memory reductions, especially compared to standard Graph Transformers and SNN-based models. 

3. I would also recommend the authors further break down the SVQ module to explore the effects of specific components (e.g., varying quantization levels, spike neuron types, or normalization techniques) could highlight why SVQ is particularly effective and reveal which configurations yield optimal performance.

4. I am curious - does the degree of homophily play a role in the SVQ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730268453100

thread_type:
official_review

<thread_object>
creator: reviewer_v4s6

<content>
summary:
Authors propose SGHormerVQ, a graph transformer model that combines Graph Transformers (GTs) with Spiking Neural Networks (SNNs) through Spiking Vector Quantization (SVQ). SGHormerVQ leverages the spiking mechanism of SNNs to encode compact, low-precision spiking vectors that efficiently capture global graph information. The model was evaluated on multiple node classification datasets, demonstrating superior performance and computational efficiency over various state-of-the-art baselines, with up to a 518x increase in inference speed on some datasets.

strengths:
1.SGHormerVQ introduces SNNs into graph transformers and uses SVQ to address issues of high computational complexity and resource demands in graph data processing. This interdisciplinary model design showcases a novel application of SNNs in graph structures.
2.The paper provides thorough experimental comparisons with various advanced models, and detailed ablation studies support the design decisions behind SGHormerVQ.imited Applicability: The model performs less effectively on datasets with high node degrees (e.g., ogbn-products), suggesting that SVQ may suffer from information loss in highly connected graphs.
Lack of Technical Detail: The practical impact of SVQ and CGSA on computational complexity, hardware requirements, and performance trade-offs is not sufficiently analyzed, potentially limiting the model's real-world applicability.
Unclear Advantage of Innovation: The paper could better articulate the distinct advantages of SVQ over traditional quantization methods, especially regarding its unique application in SNN-based quantization.

weaknesses:
1.limited Applicability: The model performs less effectively on datasets with high node degrees (e.g., ogbn-products), suggesting that SVQ may suffer from information loss in highly connected graphs.
2.Lack of Technical Detail: The practical impact of SVQ and CGSA on computational complexity, hardware requirements, and performance trade-offs is not sufficiently analyzed, potentially limiting the model's real-world applicability.
3.Unclear Advantage of Innovation: The paper could better articulate the distinct advantages of SVQ over traditional quantization methods, especially regarding its unique application in SNN-based quantization.
4.The authors should set up ablation experiments to verify the model improvement on the speed of inference.
5.The description of how the node information of a graph is converted into a codebook is poor.

questions:
Same as Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730554361641

thread_type:
official_review

<thread_object>
creator: reviewer_budy

<content>
summary:
This paper proposes a spike-based graph transformer model SGHormerVQ with spiking vector quantization for graph node classification. It builds on transformer with vector quantization and proposes to leverage spiking vector quantization to replace the pre-defined and fixed codebook. Experiments on various datasets demonstrate the competitive performance and faster inference speed of the proposed method.

strengths:
1. This paper proposes a new spiking vector quantization for efficient graph transformers.

2. The proposed model achieves competitive or superior node classification performance with much faster inference speed than graph transformer baselines.

weaknesses:
1. The motivation for *spiking* vector quantization is not clear enough. 

- If it aims at the potential energy efficiency of SNNs, there is no discussion on the issue of deployment on neuromorphic hardware, because SNNs rely on it to obtain real energy efficiency and hardly have efficiency on GPU. Since spiking vector quantization is only a small part of the model and other parts like normalization, MPNN, and self-attention are real-valued artificial neural networks, the model is not a pure SNN model and has problems for deployment. It is also not proper to mark the model as spike-based since some major parts are not spike-based.

- If it just wants to solve the problems of VQ on GPU, why consider spiking neurons? The process is similar to previous random feature propagation, what if directly use it without the spiking function? No experiment demonstrates any advantage of considering the spiking property.

2. The claim “518$\times$ faster inference speed compared to other GTs” is not proper since SGFormer has a similar inference time. It is also unclear why inference speed is largely improved.

3. For the proposed model, spiking vector quantization seems to adapt previous random feature propagation to the spiking function setting for implicit codebook, and codebook guided self-attention also follows previous works. Can the authors distinguish the contributions of this work more clearly?

4. There is no theoretical analysis for the proposed method.

questions:
For SNNs, spiking is only one of the properties. Very early works focused on analyzing their stronger computational power induced by the temporal dimension of spiking time [1], and recent works have introduced this thought to graph AI tasks [2]. This paper only considers rate coding that loses a lot of information of spike trains. Is it possible to introduce these spiking time information to better enhance the codebook coding?

[1] Networks of spiking neurons: the third generation of neural network models. Neural Networks, 1997.

[2] Temporal Spiking Neural Networks with Synaptic Delay for Graph Reasoning. ICML 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716663609

thread_type:
official_review

<thread_object>
creator: reviewer_oc9c

<content>
summary:
In this work, the authors proposed SGHormerVQ, which is a graph transformer with spiking neural networks. SGHormersVQ utilizes spiking vector quantization with a codebook. According to the authors’ experiments, they could achieve 518x faster inference speed compared to other models.

strengths:
- Through experiments and competitive results
- Incorporating graph Transformer and spiking neural networks through spiking vector quantization

weaknesses:
This study presents a method to combine spiking neural networks and graph Transformers through spiking vector quantization to reduce the computational complexity, but it seems that the vector quantization technique using codebook has serious concerns in terms of soundness.

This study utilizes a method to reduce the computational complexity of self-attention through vector quantization using a codebook. Vector quantization is implemented using spiking neurons as quantization functions. This causes the codebook to change dynamically, which is expressed in the manuscript as the term “implicit codebook.” However, this is actually just a spike train, which is the activation value of a spiking neuron, and it is difficult to apply a quantization technique that uses a codebook because it changes dynamically.

In addition, in general, in order to improve efficiency due to quantization, the representation space of the codebook should be narrower than the representation space of the activation. However, in this study, since these two are used identically, it is correct to say that there is no improvement due to the codebook. In this paper, vector quantization using an implicit codebook simply processes information with spike activation. It is unreasonable to view the improvement as being due to a codebook vector quantization. The main reason for the improvement is believed to be due to the characteristics of spiking neural networks that process information with binary spikes. The authors need to clarify this. 

Furthermore, a significant portion of the manuscript is devoted to explaining existing research. It is necessary to concisely revise it and devote more parts to explaining the proposed model. Overall, it is recommended that the authors revise the manuscript.

- Miscellaneous
  - Redundant definition of abbreviations - SNNs, GT, SVQ
  - Used before abbreviations are defined - GNNs (line 107)
  - Used undefined abbreviations - EMA (line 148)

questions:
- Why is the title SGHormerVQ?
- What algorithm was used for learning?
- What surrogate function was used?
- What is the effect of the norm function in Equation 11?
- What is R in g{V,e,X} → g{V,e,R} in Figure 2?
- What does U in Equation 13 represent? It is a one-hot matrix, so what information does it contain?
- Line 305: “we construct random features” → What does it mean?
- Is SVQ the only part that uses spiking neurons? It seems that information is not processed by spikes in other parts. Then, how can this model work on neuromorphic hardware?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732265556061

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
Dear Area Chairs and Reviewers, 

We thank the reviewers for their thorough reviews and constructive suggestions. We have carefully considered your comments and incorporated more discussion into the manuscripts. Besides, we add  new experiment results to further strenghthen our contributions. Here is a summary of the key revisions:


1. We have corrected some typo errors in the revision.
2. We provided more explainations about our observations and experimental results to detail the motivation and key novelty for our method. The updated contents are highlighted in orange.
3. For better addressing the raise problems, we conduct additional experiments and update results in the Appendix A~D.

Below we restate our main motivations and contributions to facilitate the discussions. In this study, we investigate the role of spiking neurons in current SGNNs and present the following three observations: 
- **[O1]:** For message propagation-based models, the trained embeddings of nodes within the same class tend to exhibit similar distributions.
- **[O2]:** Messages received from the multiple propagation steps can be converted into the same spike trains via spike neurons. Decoding the outputs into firing rates or spike counts, different nodes can be represented by the same low-precision vectors.
- **[O3]:** The precision of decoding node representations can be adjusted by spiking neurons.

Corresponding visualization results are demonstrated in Appendix A. The above observations drive us to push the field of spiking graph neural networks, our contributions include:
- **[C1] New Vector Quantization Variant:** Constructing spiking-based vector quantization mitigates codebook collapse and injects graph inductive bias.
- **[C2] New Linear-time Graph Transformer:** Utilizing message propagation patterns guides Graph Transformer in generating node representations with linear complexity.
- **[C3] Performance:** SGHormerVQ is able to achieve performance on par with both advanced baselines while demonstrating significant efficiency advantages in terms of codebook usage, inference latency, memory usage, and energy consumption.

In the following individual response, we provide answers to each raised weakness/question point and will incorporate the suggestions and new results in our revised paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732549429524

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gentle Reminder to Reviewers

comment:
Dear Reviewers,

We sincerely appreciate your insightful review and feedback comments. All comments have greatly enhanced our paper.

**As the author-reviewer discussion deadline is approaching, we would like to check if you have any other remaining concerns about our paper.** We have faithfully responded to all your comments and revised our paper according to your suggestions. If you have any further questions, we will be happy to answer them. And we eagerly await your feedback on whether our responses have satisfactorily resolved your concerns. We understand that you have a demanding schedule, and we appreciate the time and effort you dedicate to reviewing our paper.

Kind regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733313052539

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
Dear AC and reviewers,

First and foremost, we greatly appreciate all reviewers who voluntarily spend their time in order to improve the quality of scientific research in our field. The insightful comments and constructive suggestions are indeed helpful for improving our work. During the rebuttal period, we have tried to address **ALL** the reviewers’ concerns adequately and believe most of the concerns have been covered. To facilitate grasping the key points of the entire rebuttal, we provide the following summary:

- **More explanations regarding observations of existing spiking graph neural networks (SGNNs).** We update a number of visualization results in Appendix A of the revision to help readers better understand the necessity of spiking neurons in SGNNs and the motivation of SGHormerVQ. 
- **More implementation details of SVQ.** We restate how SVQ reconstructs the integer codebook from floating point node embeddings step by step. These implementation details are provided in the related sections of the revision to clarify potential misunderstandings.
- **Comprehensive analysis of energy efficiency.** For those concerns about the energy consumption, memory usage and inference latency raised by most reviewers, we have provided a detailed energy efficiency analysis comparing the state-of-the-art GTs with SGHormerVQ in Appendix B.
- **Additional technical details.** We conduct additional experiments including the effects of different components, the choice of coding mechanisms, and performances on heterophilic datasets, to demonstrate the practical applicability of SGHormerVQ. Corresponding results are presented in Section 5.4 and Appendix C-D.

We confidently believe that this work dives deep into the field of SGNNs, our contributions include:

- A series of observations analyze the role of spiking neurons in existing SGNNs.

- The spiking vector quantization module (SVQ) represents nodes from narrower latent spiking embeddings space to alleviate the codebook collapse.

- The Graph Transformer module (CGSA) with linear complexity generates node embeddings indirectly guided by message propagation patterns.
  
- Empirical experiments show that SGHormerVQ effectively addresses the issues presented in previous VQ-based GNNs. SGHormerVQ achieves comparable or even faster inference speed compared to other GT baselines.

We understand that the discussion during the rebuttal period is limited due to time constraints. **We hope our response can spark more discussion during the AC-reviewer phase, to confirm whether our rebuttal has adequately addressed reviewers' concerns.**

Best regards,

the Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work applies spiking neural networks to improve the efficiency of graph transformers. The empirical success in reducing computational overhead and the presentation quality are appreciated by all reviewers.

The discussion has highlighted a few key issues which remain unresolved. The concept of implicit codebook and other key novelties should be explained in more detail in the writing. How the spiking vector quantization mechanism helps to mitigate codebook collapse and graph inductive bias should be better clarified. Overall, the submission has merit to be published and can benefit from another round of revision to address these issues.