A Graph is Worth 1-bit Spikes: When Graph Contrastive Learning Meets Spiking Neural Networks
Authors: Jintang Li
Huizhe Zhang
Ruofan Wu
Zulun Zhu
Baokun Wang
Changhua Meng
Zibin Zheng
Liang Chen
Venue: ICLR 2024 poster (2024)
Keywords: Graph contrastive learning
spiking neural networks
binarized graph representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a novel spiking graph contrastive learning framework to learn binarized 1-bit representations for graphs, making balanced trade-offs between efficiency and performance.

Abstract: While contrastive self-supervised learning has become the de-facto learning paradigm for graph neural networks, the pursuit of higher task accuracy requires a larger hidden dimensionality to learn informative and discriminative full-precision representations, raising concerns about computation, memory footprint, and energy consumption burden (largely overlooked) for real-world applications. This work explores a promising direction for graph contrastive learning (GCL) with spiking neural networks (SNNs), which leverage sparse and binary characteristics to learn more biologically plausible and compact representations. We propose SpikeGCL, a novel GCL framework to learn binarized 1-bit representations for graphs, making balanced trade-offs between efficiency and performance. We provide theoretical guarantees to demonstrate that SpikeGCL has comparable expressiveness with its full-precision counterparts. Experimental results demonstrate that, with nearly 32x representation storage compression, SpikeGCL is either comparable to or outperforms many fancy state-of-the-art supervised and self-supervised methods across several graph benchmarks.

Published as a conference paper at ICLR 2024
A GRAPH IS WORTH 1-BIT SPIKES: WHEN GRAPH CONTRASTIVE LEARNING MEETS SPIKING NEURAL NETWORKS
Jintang Li Huizhe Zhang', Ruofan Wu², Zulun Zhu", Baokun Wang",
Changhua Meng", Zibin Zheng', Liang Chen'
Sun Yat-sen University
2 Ant Group
Naayynn Technological University
[1ijt55, zhanghzh33 3 @mail2. wys.uu.ss.uuuu....m. edu. en;
truofan wrf, yike. wbk, changhua mch) @antgroup.com com;
ZULUN00 wtt.oo0.eeee........mmmmmm edu.ss
ABSTRACT
While contrastive self-supervised learning has become the de-facto learning paradigm for graph neural networks, the pursuit of higher task accuracy requires a larger hidden dimensionality learn informative and discriminative full-precision representations, raising concerns about computation, memory footprint, and energy consumption burden (largely overlooked) for real-world applications. This work explores promising direction for graph contrastive learning (GCL) with spiking neural networks (SNNs), which leverage sparse and binary characteristics learn more biologically plausible and compact representations. We propose SPIKEGCL, a novel GCL framework to learn binarized 1-bit representations for graphs, making balanced trade-offs between efficiency and performance. We provide theoretical guarantees to demonstrate that SPIKEGCL has comparable expressiveness with its uill-precision counterparts. Experimental results demonstrate that, with nearly 32x representation storage compression, SPIKEGCL is either comparable to or outperforms many fancy state-of-the-arr supervised and self-supe methods across several graph benchmarks.
INTRODUCTION
Graph neural networks (GNNs) have demonstrated remarkable capabilities in learning representations of graphs and manifolds that are beneficial for wide range of tasks. Especially since the advent of recent self-super learning techniques, rapid progress toward learning universally useful representations has been made Liu et al. (2021) I); Yu et al. a2021a]b). Graph contrastive learning (GCL), which aims to learn generalizable and transferable graph representations by contrasting positive and negative sample pairs taken from different graph views, has become the hotspot in graph self-supervised learning. As an active area of research, numerous variants of GCL methods have been proposed to achieve state-of-the-an performance in graph-based learning tasks, solving the dilemma of learning useful representations from graph data without end-to-end supervision Li et al. 2023a):
Velickovic et al. (2019); Thakoor et al. (2021) Zheng al. (2022).
In recent years, real-world graphs have been scaling and growing even larger. For example, Amazon's product recommendation graph has 1.50M users and 350M products Chen al. (0226), while Microsoft Academic Graph consists of more than 120 publications and related authors, venues, organizations, and fields of study (Sinha al. (015). New challenges beyond label annotations have arisen in terms of processing, storage, and deployment in many industrial scenarios Chen et al. (2022a). While GNNs-based contrastive learning has advanced the frontiers in many applications, current -tate-of- oftte-----h--rtt ahe-arts mainly requires large hidden dimensions to learn generalizable full-precision
Work done during an internship at Ant Group.
1 "oorrsponniig author.

---

Published as a conference paper at ICLR 2024
representations Mo et al. (2022), Li et al. (2023a), making them memory inefficient, storage excessive,
and computationaly expensive especially for resource-constrald edge devices Zhou et al. (2023).
In parallel, biological neural networks continue to inspire breakthroughs in modern neural network performance, with prominent examples including spiking neural networks (SNNs) Malcolm (SNNs)Malcolm & Casco Rodriguez (.023). J. SNNS are a class of brain-inspired networks with asynchronous discrete and sparse characteristics, which have increasingly manifested their superiority in low energy consumption and inference latency [Fng al. (2022). Unlike traditional artificial neural networks (ANNs), which use floating-poing outputs, SNNs use more biologically plausible approach where neurons communicate via sparse and binarized representations, referred to as 'spikes' Such characteristics make them promising choice for low powerr mobile, or otherwise bardware-constraiee In literature Zhou et al. (2022); Zhu et al. 2022), SNNs are proven to consume ~000 less energy compared with modern ANNs on the neuromorphic chip (e.g. ROLLs Indiveri et al. (2015).
settings.
Inspired by the success of vision research Kim et al. (2021); Zhou et al. (2022), some recent efforts have been made toward generalizing SNNs to graph data. For example. SpikingGCN [hhu et al.] (2022) and SpikeNet Li et al. (2023b) are two representative works combining SNNs with GNN architectures, in which SNNs are utilized as the basic building blocks to model the underlying graph structure and dynamics. Despite the promising potentiality, SNNs have been under-appreciated and under-investigated in graph contrastive learning. This has led to an in- teresting yet rarely explored research question:
32
GCL
SpiceGCL
Inputggrph.
Comparison between conven- tional GCL and SPIKEGCL Instead of us- ing full-precision (32-bit) representations, SPIKEGCL produces sparse and compact 1-bit representations, making them more memory-friendee and computationally effi- cient for cheap devices with limited resources.
Figure 1:
Can we explore the possibilities of SNNs with contrastive learning schemes to learn sparse, binarized yel generalizable representations?
Present work. Deviating from the large body of prior works on graph contrastive learning, in this paper we take a new perspective to address self-supervised and binarized representation learning on graphs. We present SPIKEGCL, principled GCL framework built upon SNNs learn binarized graph representations in a compact and efficient manner. Instead of learning full- full-precision node representations, we learn sparse and compact 1-bit representations to unlock the power of SNNs on graph data and enable fast inference. In addition, we noticed the problem of vanishing gradients (surprisingly overlooked) during direct training of SNNs and proposed a blockwise training approach to tackle it. Although SPIKEGCI uses binarized 1-bit spikes as representations, it comes with both theoretical guarantees and comparable performance in terms of expressiveness and efficacy, respectively. Compared with floating point counterparts, SPIKEGCL leads to less storage space, smaller memory footprint, lower power consumption, and faster inference speed, all of which are essential for practical applications such as edge deployment.
Overall, our contributions can be mainly summarized as follows:
We believe our work is timely. In response to the explosive data expansion, we study the problem of self-supervised and binarized representations learning on graphs with SNNs, whose potential is especially attractive for low-power and resource-constrained settings.
We propose SPIKEGCL to learn binarized representations from large-scale graphs. SPikEGCL exhibits high hardware efficiency and significantly reduces memory consump- tion of node representations by ~32x and energy consumption of GNNs by up to ~~xx respectively. SPIKEGCL is theoretically guaranteed framework with powerful capabilittess in learning node representations.
We address the challenge of training deep SNNs with a simple blockwise learning paradigm. By limiting the backpropagation path to a single block of time steps, we prevent SNNs from suffering from the notorious problem of vanishing gradients.
Extensive experiments demonstrate that SPIKEGCL performs on par with better than advanced full precision competitors, across multiple graphs of various scales. Our results hold great promise for accurate and fast online inference of graph-based systems.
sometimes
2

---

Published as a conference paper at ICLR 2024
We leave two additional remarks: (i) SPIKEGCL is designed to reduce computation, energy, and storage costs from the perspective of compressing graph representations using biological networks, which is orthogonal to existing advances on sampling Hamilton et al. (2017). data condensation Jim et al. ) Zeng et al. (2020) and architectural simplifications Thakkoor et al. (2021); |o et al. (2022); ; Zheng et al. 2022). (ii) To the best of knowledge, our work is the first to explore the feasibility of implementing graph self-supervised learning with SNNs for learning binarized representations. We hope our work will inspire the community explore more promising algorithms.
RELATED WORK
Graph contrastive learning. Self-supervised representation learning from graph data is quickly evolving field. The last years have witnessed the emergence promising self-superrvied learning strategy, referred to graph contrastive learning (GCL) Liu et al. (202I). Typically, GCL works by contrasting so-called positive samples against negative ones from different graph augmentation views, which has led to the development of several novel techniques and frameworks such as DGI Velickovic et al. (019)) GraphCL You et al, (2020), GRACE Zhu et al. (020), and GGD Zheng et al. (2022). Recent research has also suggested that negative samples not always necessary for graph contrastive learning, with prominent examples including BGRL Thakoor et al. (2021) and
(CA-SSG/Zhnng al. (2021) Negative-sample-ree approach paves the way to simple yet effective
GCL method and frees the model from intricate designs. We refer readers to Liu et al. (2021) for comprehensive review of graph contrastive learning. Despite the achievements, current GCL designs mainly focus on task accuracy with a large hidden dimensionality Mo et al. (2022) ; Li et al. (2023a), and lack consideration of hardware resource limitations to meet the real-time requirements of edge application scenarios Zhou et al. (2023).
Spiking neural networks. SNNs, which mimic biological neural networks by leveraging sparse and event-driven activations, are very promising for low-power applications. SNNs have been around for while, and although they have not gained the same level of popularity GNNs, they have steadily increased their influence in recent years. Over the past few years, SNNs have gained significant popularity in the field of vision tasks, including image classification Zhou al. (2022), objection detection Kim et al. (2020), and segmentation Kim et al. (2021). Efforts have been made to incorporate their biological plausibility and leverage their promising energy efficiency into GNN learning. SpikingGCN Zhu et al. (222) is a recent endeavor on encoding the node representation through SNNs. As SNNs are intrinsccally dynamic with abundant temporal information conveyed by spike timing. SpikeNet Li et al. (2023c) then proceeds to model the temporal evolution of dynamic graphs via SNNs. In spite of the increasing research interests, the benefits of SNNs have not been discovered in GCL yet.
Binarized graph representation learning. Model size, memory footprint, and energy consumption are common concerns for many real-world applications Bahri et al. (202T). In reaction, binarized graph representation learning has been developed in response to the need for more efficient and compact representations for learning on large-scale graph data. Currently, there are two lines of research being pursued in this area. One line of research involves directly quantizing graphs by using discrete hashing techniques Yang et al. (2018 8 Bahri et al. (2021) or estimating gradients of non-differentiable quantization process Chen et al. (2022b). Another research line to produce binary representations using deep learning methods are those binarized networks, which are designed for fast inference and small memory footprint, in which binary representation is only a by-product Wang et al. (202T). On one hand, binarized graph representation learning is a novel and promising direction that aims to encode graphs into compact binary vectors that can facilitate efficient storage, retrieval, and computation. On the other hand, SNNs that use discrete spikes rather than real values to communicate between neurons are naturally preferred for learning binary representations. Therefore, it is intuitively promising explore the marriage between them.
3 PRELIMINARIES
Problem formulation. Let g {v:b/1 and & CVV x V are a set of nodes and edges, respectively. We focus primarily on undirected graphs though it is straightforward to extend our study to directed graphs. G is associated with an
EV, denote attributed undirected graph where V
3

---

Published as a conference paper at ICLR 2024
d-dimensional attribute feature matrix X { E R RNd In the self-supervised learning setting of this work, our objective is to learn an encoder Sa IRd - (0. 1 parameterized by 0, which maps between the space of graph 9 and their low-dimensional and binary latent representations Z = [ M such that fo(G) Z E {0, 1 given d the embedding dimension. Note that for simplicity we assume the feature dimensions are the same across all layers.
Spiking neural networks. Throughout existing SNNs, the integrate- fire (IF) Salinas & Sejnowski (2002) model and its variants are commonly adopted to formulate the spiking neuron and evolve into numerous variants with different biological features. As the name suggests, IF models have three fundamental characteristics: (i) Integrate. The neuron integrates current by means of the capacitor over time, which leads to charge accumulation; (ii) Fire. When the membrane potential has reached or exceeded given threshold Vin, it fires (i.e., emits spike). (iii) Reset. After that, the membrane potential is reset, and here we introduce two types of Rueckauur et al. (2016): reset to zero which always sets the membrane potential back to a constant value Vreset < Vun typically zero, whereas reset by subtraction subtracts the threshold Vin from the membrane potential at the time where the threshold is exceeded: We use unified model to describe the dynamics of IF-based spiking neurons:
Integrate:
V V(V'-1,,"
Fire:
St = O(V* - -vn,,
S'V + -1-557 S*) reset to zero, reset by subtraction,
Reset:
V' (*(1 Vin) + (1-
where Ie and V denote the input current and membrane potential (voltage) at time-step respectively, The decision to fire a spike in the neuron output is carried out according to the Heaviside step function
O()). which defined by ((x) = 1 if x 2 0 and 0 otherwise. The function (-()) Eq. [ describes how the spiking neuron receives the resultant current and accumulates membrane potential. We have
|| Sejnowski (2002) and its variant Leaky Integrate-and- (LIF) Gerstner et al. (2014)
model, formulated as follows:
IF:
V'= V--1 + 1
(4)
V' V'=v-1 1 (r' (
LIF:
Vreset)
(5)
where Tm in Eq. 5 represents the membrane time constant control how fast the membrane potential decays, leading to the membrane potential charges and discharges exponentially in response to the current inputs. Typically, Tm can also be optimized automatically instead of manually tuning to learn different neuron dynamics during training, which we referred to as Parametric LIF (PLIF) Fang et al. (2020). In this paper, the surrogate gradient method is used to define O'(x) o(xx) during error back-propagation, with (-) denote the surrogate function such as Sigmoid and a the smooth factor Li et al. (023bb)
4 SPIKING GRAPH CONTRASTIVE LEARNING (SPIKEGCL)
In this section, we introduce our proposed SPIKEGCL framework for learning binarized representa- tions. SPIKEGCL is simple yet effective derivative of the standard GCL. framework with minimal but effective modifications on the encoder design coupled with SNNs. In what follows, we first shed light on building sequential inputs for SNNs from single non-temporal graph ($ 4.11) and depict how to binarize node representations using SNNs $ 4.2). Then, we explain the architectural overview and detailed components of SPIKEGCL one by one 4.3). Finally, we employ blockwise learning for better training of deep SNNs (S 4.4).
4.1 GROUPING NODE FEATURES
Typically, SNNs require sequential inputs to perform the integrated-and-fit fire process to emit spikes. One major challenge is how to formulate such inputs from non-temporal graph. A common practice introduced in literature is to repeat the graph multiple times (i.e., a given time window T), typically followed by probabilistic encoding methods (e.g.. Bernoulli encoding) to generate diverse input graphs Zhu et al. (2022); Xu et al. (202I). However, this will inevitably introduce high computational and memory overheads, becoming major bottleneck for SNNs scale to large graphs
4

---

Published as a conference paper at ICLR 2024
In this work, we adopt the approach of partitioning the graph data into different groups rather than repeating the graph multiple times to construct the sequential inputs required for SNNs. Given a time window T (T >> 1) and graph G. we uniformly partition the node features into the following T groups x = X ....., ,'] along the feature dimension, where X E R R & consists of the group of features in the t-th partition. cases where d cannot be divided by T, we have X E NN(a///T) fort T and X E Vx mad T) Thus we have T subgraphs as
g6 9T) = [(V, x, (V,E,
Each subgraph in 9 shares the same graph structure but only differs node features. Note that [X...... ."" are non-overlapping groups, which avoids unnecessary computation input features. Our strategy leads to huge memory and computational benefits as each subgraph in the graph sequence only stores a subset of the node features instead of the whole set. This is a significant improvement over previous approaches such SpikingGCN Zhu al. (2022). as it offers substantial benefits in
terms of computational and memory complexity.
4.2 BINARIZING GRAPH REPRESENTATIONS
In GCL, GNNs are widely adopted as encoders for representing graph-structured data. GNNs generally follow the canonical message passing scheme in which each node's representation is computed recursively by aggregating representations ("messages") from its immediate neighbors Kipt & Welling (2016); |aamlloo et al. (2017). Given an L. layer GNN fo, the updating process of the 7-th layer could be formulated as:
h ! COMBINE ((NS-1, "MM-"I,,GGRGATTI ((MS-D.UEN.)))
(7)
where AGGREGATE() is an aggregation function that aggregates features of neighbors, and COMBINE(.) denotes the combination of aggregated features from a central node and its neighbors.
Na is the embedding of node the I-th layer of GNN, where / E (1,..., L} and initially (0)
Tui NC, is the set of neighborhoods of u. After rounds of aggregation, each node
obtains its representation vector h (L) The final node representation is denoted as H = [h] (L) N6)
We additionally introduce SNNs to the encoder for binarizing node representations. SNNs receive continuous values and convert them into binary spike trains, which opens up possibilities for mapping graph structure from continuous Euclidian space into discrete and binary one. Given graph sequence se = [ ,..,,................... GJ), we form graph encoders [ff,, .,, f accordingly such that each encoder fā. maps a graph 9, to its hidden representations H' Here we denote Se = [fb,:-... [ with slightly abuse of notations. Then, spiking neuron is employed on the outputs to generate binary spikes in dynamic manner:
= (VV(t-) - Vn), H' = fg(g',
(8)
where (-) denotes spiking function that receives the resultant current from the encoder and accumulates membrane potential to generate the spikes, such as IF or LIF introduced in $ St e {0, 1) Nx is the output spikes at time step I. In this regard, the binary representations are derived by taking historical spikes in each time step with concatenate pooling Li et al. (003b)) i.e.,
= [S (S)).
4.3 OVERALL FRAMEWORK
We decompose our proposed SPIKEGCL from four dimensions: (i) augmentation, (ii) encoder, (iii) predictor head (a.k.a. decoder), and (iv) contrastive objective. These four components constitute the design space of interest in this work. We present an architectural overview of SPIKEGCL in Figure 2(a) as well the core step of embedding generation in Figure 2(b)
Augmentation. Augmentation is crucial for contrastive learning by providing different graph views for contrasting. With given graph G. SPIKEGCL involves bi-level augmentation techniques, i.e.,
Exploring non-uniform partition with clustering methods important direction for future work.
5

---

Published as a conference paper at ICLR 2024
Encodm
pritive
Grouphag
Sp TONTTO
head
greoh.g
Group Si
Augmentationn Comption
Share porameters
low
Concel
3-btt Spies
Input graph 5
Greesing
Eicoder
Predicor
head
respike
Group S,
(a) SpikeGCL framework
(b) SpikeGCL embedding (spikes) generation
Figure 2: Overview of SPIKEGCL framework. (a) SPIKEGCL follows the standard GCL philosophy, which learns binary representations by contrasting positive and negative samples with margin ranking loss. (b) SPIKEGCL first partitions node features into T von-overlapping groups, each of which is then fed an encoder whereby spiking neurons represent nodes of graph as 1-bit spikes.
topology (structure) and feature augmentation, to construct its corrupted version G. To be specifie, we harness two ways for augmentation: edge dropping and feature shuffling. Edge dropping randomly drops a subset of edges from the original graph, while feature shuffling gives a new permutation of the node feature matrix along the feature dimension. In this way, we obtain the corrupted version of an original graph 9 (V, X}. where & C & and X denotes the column- wise shuffled feature matrix such that X [[: P] with P the new permutation of features. Since we partition node features in sequential manner that is sensitive to the permutation of input features, feature shuffling able to offer hard negative samples for contrastive learning.
Encoder. As detailed in 4.2 our encoder is series of peer GNNs corresponding to each group of input node features, followed by a spiking neuron to binarize the representations. Among the many variants of GNNs, GCN Kipf & Welling 2016) the most representative structure, and we adopt as the basic unit of the encoder in this work. The number of peer GNNs in the encoder is relative to the number of time steps T. which makes the model excessively complex and potentially lead to overfitting if T is large. We circumvent this problem by parameter sharing. Note that only the first layer different in response to diverse groups of features. For the remaining L- 1 layers, parameters are shared across peer GNNs to prevent excessive memory footprint and the overfitting issue.
Predictor head (decoder). A non-linear transformation named projection head is adopted to map binarized representations to continuous latent space where the contrastive loss is calculated, as advocated in Thakoor al. (2021) : ZZang et al. (2021). As a proxy for learning discrete spikes, employ a single-layer perceptron (MLP) to the learned representations, i.e., (2(2) ) = MLP(2.,,, Vu E V. Since the hidden representations are binary, we can instead use 'masked summation' [i al. (2023b) to enjoy the sparse characteristics and avoid expensive matrix multiplication computations during training and inference.
Contrastive objective. Contrastive objectives are widely used to measure the similarity or distance between positive and negative samples. Rather than explicitly maximizing the discrepancy between positive and negative pairs as most existing works on contrastive learning have done, the 'contrastive- ness' in this approach is reflected in the diverse 'distance' naturally measured by parameterized model (-). Specifically, we employ a margin ranking loss (MRL) Chen et al. 2020 to (+):
J
Σ m0 (z() - go(2.) ) m), WEV
(9)
with the margin m hyperparameter to make sure the model disregards abundant far (easy) negatives and leverages scarce nearby (hard) negatives during training. Zu and म, is the representation of node obtained from original graph sequence 9 and its corrupted one, respectively. MRL forces the score of positive samples to be lower (towards zero) and assigns a higher score to negative samples by a margin of at least Therefore, positive samples separated from negative ones.
4.4 BLOCKWISE SURROGATE GRADIENT LEARNING
From an optimization point of view, SNNs lack straightforward gradient calculation for backward propagation, and also methods that effectively leverage their inherent advantages. Recent attempts

---

Published as a conference paper at ICLR 2024
have been made surrogate gradient learning Lee et al. (2016), alternative of gradient descent that avoids the non-differentiabiliyy of spike signals. As backpropugattionlik. like training method, it approximates the backward gradients of the hard threshold function using smooth activation function (such as Sigmoid) during backpropagation. At present, the surrogate learning technique plays extremely important role in advanced methods learn SNNs properly (Zhu al. (2022) Xu
et al. 2022)); Li et al. (0033); Fang et al. (2021) 2020).
Despite the promising results, surrogate gradient learning has its own drawbacks. Typically, SNNs require relatively large time steps to approximate continuous inputs and endow the network with better expressiveness Zhu et al. (2022). However, large time step often leads to many problems such as high overheads and network degradation Fang et al. (2021). Particularly, the training of SNNs also comes with serious vanishing gradient problem in which gradients quickly diminish with time steps. We have further discussion with respect to the phenomenon in [Appendix B] These drawbacks greatly limit the performance of directly trained SNNs and prevent them from going 'deeper' with long sequence inputs.
In this paper, we explore alternatives to end-to-end back- propagation in the form of surrogate gradient learning rules, leveraging the latest advances in self-supervised learning Siddiqui al. 2023) to address the above lim- itation in SNNs. Specifically, we propose a blockwise training strategy that separately learns each block with a local objective. We consider one or more consecu- tive time steps as a single block, and limit the length of the backpropagation path to each of these blocks. Pa- rameters within each block are optimized locally with a contrastive learning objective, using stop- -raddiett to prevent the gradients from flowing through the blocks. A technical comparison between the existing end-to- end surrogate learning paradigm and our proposed local training strategy is shown in Figure
Erocder
Decader
Bick
prossgation calt
Oubut
(a) End-to-end backpropagation
(b) Blockwise backpropegation
Figure 3: A comparison between two back- propagation learning paradigms. The back- propagation path during blockwise train- ing limited to a single block of networks to avoid a large memory footprint and van- ishing gradient problem.
5 THEORETICAL GUARANTEES
There are several works on designing practical graph-based SNNs that achieve comparable per- formance with GNNs Li et al. (2023c); [uu et al. y Zhu et al.] (2022). However, the basic principles and theoretical groundwork for their performance guarantees are lacking, and related research only shows the similarity between the IF neuron model and the ReLU activation Rueckauer et al. (016)). In this section, we are motivated to bridge the gap between SNNs and GNNs. We present an overview of theoretical results regarding the approximation properties of SPIKEGCL with respect to its full-precision counterparts through the following theorem:
Theorem (Informal). For any full-precision GNN with hidden dimension of d/T, there exists corresponding SPIKEGCI such that its approximation error, defined as the &z distance between the firing rates of the SPIKEGCL representation and the GNN representation at any single node, of the order (1/TT).
In the order relation O ()), we hide factors dependent upon the underlying GNN structure, network depth as well as characteristics of the input graph, which will be stated precisely in Theorem Theorem 2 Appendix A| Theorem suggests that with a sufficiently large node feature dimension such that we may set a large simulation length T, we are able to (almost) implement vanilla GNNs with a much better computational and energy efficiency. Moreover, the approximation is defined using the firing rates of SNN outputs, which measures only a restricted set of inductive biases offered by SPIKEGCL. Consequently, we empirically observe that a moderate level of T might also provide satisfactory performance in our experiments. Our analysis is presented to shed insights on the connection between the computational neuroscience model (e.g.. SNNs) and the machine learning neural network model (e.g., GNNs). This connection has been analytically proven under certain conditions and empirically demonstrated through our experiments (see 6). It can serve as the theoretical basis for potentially combining the respective merits of the types of neural networks.
7

---

Published as a conference paper at ICLR 2024
6 EXPERIMENTS
In this section, we perform experimental evaluations demonstrate the effectiveness of our proposed SPIKEGCL framework. Due space limitations, we present detailed experimental settings and
additional results in Appendix and Appendix F
Datasets. We evaluate SPIKEGCL on several graph benchmarks with different scales and properties. Following prior works Zhang al. (202T): Thakoor al. (202T) /Zhu al. (2022), we adopt 9 common benchmark graphs, including two co-purchase graphs i.c., Amazon- Photto,,PPooo,, Amazon- SemnuuterShchurr Shchur al. 2018), two co-author graphs, i.e., Coauthor-CS and CCauthor-PhyssccsSheeeee Shcchr
Computer
et al. (2018), three citation graphs, i.e., Cora, CiteSeer, PubMed Sen et al. (2008). as well as two large-scale datasets ogbn-arXiv and ogbn-MAG from Open Graph Benchmark Hu 22020 The
detailed introduction and statistics of these datasets are presented in Appppddixx E.
Baselines. We compare proposed methods to wide range of baselines that fall into four categories: (i) full-precision (supervised) GNNs: GCN Kipt & Welling (2016) and GAT Veličkovid
et al. 2018); (ii) 1-bit quantization-basee based GNNs: Bi-GCN WWan et al. (2021), BinaryGNN Bahri
et al. 202T and BANE Yang et al. (2018); (iii contrastive methods: DGI Velickovic et al. (2019).
GRACE Zhu et al. (2020), CCA-SSG Zhang et al. (2021), BGRL Thakoor /TaLoThakoor et (2021). SUGRL Mo
et al. (2022), and GGD Zheng et al. (2022); ; (iv) Graph SNNs: SpikingGCNN Zhu et al. (2022).
SpikeNet Li al. (2023b). GC-SNN and GA-SNN Xu et al. (202I). SpikeNet initially designed for dynamic graphs, we adapt the author's implementation to static graphs following the practice in Zhu al. 2022]; Xu et al. (202T). The hyperparameters of all the baselines were configured according to the experimental settings officially reported by the authors and were then carefully tuned in our experiments to achieve their best results.
Table 1: Classification accuracy (%) on six large scale datasets. The best result for each dataset is highlighted red. The missing results are due to the out-of-ment out-of-meoory error on a GPU with 24GB memory. (U: unsupervised or self-supervised: S: spike-based; B: binarized)
U
S
B
Computers
Photo
CS
Physies
arXiv
MAG
GCN Kipf & Welling 2016
86.5
92.4_a.2
92.5
95.7
70.4103 70.610.3
30.1 30.5
GAT
Veličković et
2018
86.910.2
92.5
92.310.2
95.41o.
SpikeNet Li
2023b
88.02o.7
92.9_a.1
93.4
95.810.
66.8
SpikingG Zhu al.
2022
86.9 86.910.3
92.61..7
92.610.3
94.310.
55.8
GC-SNN Xu et al 402
88. 88.2
92.8-a.1
93.040.4
95.640.7
GA-SNN Xu ct al.
2021
88.1
93.5
92.2
95.8
Bi-GON Wang al. |202
86.420.3
92.1
91.0+0.7
93.3+1.1
66.0=a.a
28.2+0.4
BinaryGNNY Bahn et al.
2021
87.8
92.4_a.2
91.2
95.3
67.220.9
BANE Yang el al, 2018
72.7=0.3
78.2=n.3
92.8+0.1
93.420.4
>ddyys
>3days
DGI Velickovic et ul 2019
/
84.0 84.0
91.6_n.2
92.2
94.5
65.1
31.410.3
GRACE Zhu el al 2020
86.3
92.2-n.2
92.940.a
95.3
68.7
31.5
ZCA-SSG/Zhang et 2021
88.7
93.1
93.3
95.710.1
71.210.
31.810.4
BGRL. Thakoor al.
90.3
93.2
93.3
95.7
71.610.1
31.110.1
SUGRL Mo et 2022
88,9
93.2-0.4
93.4
95.240.0
68.820.4
68.8
32.4
GGD Zheng cl 2022
88.02o.1
92.9
93.1
95.310.0
71.620.5
31.710.7
SPIKEGCI.
V /
88.9
93.0
92.8
95.240.6
70.9
32.0ga.3
Overall performance. The results on six graph datasets are summarized in Table T We defer the results Cora, CiteSeer, and PubMed Appendix FJ Table shows significant performance gap between full-precision methods and 1-bit GNNs, particularly with the unsupervised method BANE, Even supervised methods Bi-GCN and binaryGNN struggle match the performance of full-precision methods. In contrast, SPIKEGCL competes comfortably and sometimes outperforms advanced full-prees precisiio methods. When compared to full-precision GCL methods, our model approximates about 95%~99% of their performance capability across all datasets. Additionally, SPIKEGCL performs comparably to supervised graph SNNs and offers the possibility to scale to large graph datasets (e.g., arXiv and MAG). Overall, the results indicate that binary representation does not necessarily lead to accuracy loss as long as it is properly trained.
Efficiency. We first provide a comparison of the number of parameters and theoretical energy consumption of SPIKEGCI and full-precision GCL methods. The computation of theoretical energy
8

---

Published as a conference paper at ICLR 2024
Table 2: The parameter size (KB) and theoretical energy consumption (mJ) of various GCL methods. The row in 'Average' denotes the averaged results of full-preciie full-precision GCL baselines. Darker color in SPIKEGCL indicates larger improvement in efficiency over the baselines.
Computers
Physics
arXiv
MAG
#Paramm
Energy
iPuram]
Energy
Param,
Energy
#Param]
Energy J #Puram]
Energy
DGI
917.5
0.5
4008.9
8
4833.3
6
590.3
590.3
568
GRACE
656.1
1.1
3747
17
4571.9
13
328.9
21
328.9
4463
CCA-SSG
262.4
1808.1
152
2220.2
352
98.8
340
BGRL
658.4
25
3749.8
163
4574.2
373
331.2
180
331.2
787
SUGRL
193.8
13
2131.2
147
2615.1
342
99.5
26
99.5
117
GGD
254.7
3747.3
30.0
Average
490.4
11.9
3198.7
104.5
3906.0
237.6
246.4
68.3
246.4
1279.1
SpikeGCL
60.9
0.038
460.7
0.048
564.4
0.068
7.3
0.2
6.6
0.18
Acoursyy
88
300
03
Mcmorr (16))
Eneeggo (0L)
Time
200
86
0.2
100
84
5
10
15
20
30
10
15
20
25
30
10
15
20
25
30
0.0
6
10
15
20
25
30
Time
Time slop
Time
Time
step
Figure 4: Comparison of SPIKEGCL and other graph SNNs in terms of accuracy (%), training time (s), memory usage (GB), and energy consumption (mJ), respectively.
consumption follows existing works Zhu et al. (2022); Zhou et al. (2022): Wang al, (202T) and detailed in [Appendix E Table summarizes the results in terms of model size and energy efficiency compared to full-pree cull-precision GCL methods. SPIKEGCL consistently shows better efficiency across all datasets, achieving ~7x less energy consumption and ~ 1/60 model size MAG. As the time step T a critical hyperparameter for SNNs to better approximate real-valued inputs with discrete spikes, we compare the efficiency of SPIKEGCL and other graph SNNs in terms of accuracy, training time, GPU memory usage, and theoretical energy consumption with varying time steps from to 30. The results on the Computers dataset are shown in Figure 4 It is observed that SPIKEGCL and other graph SNNs benefit from increasing time steps, generally improving accuracy as the time step increases. However, a large time step often leads to more overheads. our experiments, larger time step can make SNNs inefficient and become major bottleneck to scaling to large graphs, as demonstrated by increasing training time, memory usage, and energy consumption. Nevertheless, the efficiency of SPIKEGCL is less affected by increasing time steps. The results demonstrate that SPIKEGCL, coupled with compact graph sequence input and blockwise training paradigm, alleviates such a drawback of training deep SNNs. Overall, SPIKEGCL is able significantly reduces computation costs and enhance the representational learning capabilities of the model simultaneously.
CONCLUSION
In this work, present SPIKEGCL, principled graph contrastive learning framework learn binarized and compact representations for graphs at scale. SPIKEGCL leverages the sparse and binary characteristies of SNNs. as well as contrastive learning paradigms, to meet the challenges of increasing graph scale and limited label annotations. The binarized representations learned by SPIKEGCL require less memory and computations compared to traditional GCL, which leads to potential benefits for energy-efficient computing. We provide theoretical guarantees and empirical results to demonstrate that SPIKEGCL is an efficient yet effective approach for learning binarized graph representations. In our extensive experimental evaluation, SPIKEGCL is able to achieve performance on par with advanced baselines using full- aall-precision or 1-bit representations while demonstrating significant efficiency advantages in terms of parameters, speed, memory usage, and energy consumption. We believe that our work is promising from a neuroscientific standpoint, and hope it will inspire further research toward efficient graph representation learning.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDDGEMENT
The research is supported by the National Key R&D Program China under grant No. 2022YFF0902500, the Guangdong Basic and Applied Basic Research Foundation, China (No. 2023A15 1050), Shenzhen Research Project (K122020231023044550000)00 Ant Group Research Intern Program.
REFERENCES
Mehdi Bahri, Gaétan Bahl, and Stefanos Zafeiriou. Binary graph neural networks. CVPR, pp. 1492-9501. Computer Vision Foundation IEEE, 2021.
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: Fast learning with graph convolutional networks via importance sampling. In ICLR (Poster). OpenReview.net, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML,, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020.
Yankai Chen, Huifeng Guo, Yingxue Zhang, Chen Ma, Ruiming Tang, Jingjie Li, and Irwin King. Learning binarized graph representations with multi-faccted quantization reinforcement for top-k recommendation. In KDD, pp. 168-178. ACM, 2022a.
Yankai Chen, Huifeng Guo, Yingxue Zhang, Chen Ma, Ruiming Tang. Jingjie Li, and Irwin King. Learning binarized graph representations with multi-faceted quantization reinforcement for top-k recommendation. In KDD, pp. 168-178. 2022b.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. arXiv preprini 20V::2007.00788. 2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In NeurlPS, pp. 11056-21069) 2021.
Lang Feng, Qianhui Liu, Huajin Tang, De Ma, and Gang Pan. Multi-level firing with spiking ds-resnet: Enabling better and deeper directly-trained spiking neural networks. IJCAI, pp. 2471-2477. ijcai.org, 2022.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons networks and models of cognition. Cambridge University Press, 2014.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurlPS, pp. 1024-1034, 2017.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687, 2020.
Giacomo Indiveri, Federico Corradi, and Ning Qiao. Neuromorphic architectures for spiking deep neural networks. In 2015 IEEE International Electron Devices Meeting (IEDM), pp. 4-2. IEEE, 2015.
Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph condensation for graph neural networks. In ICLR, 2022.
Sei Joon Kim, Scongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for
object detection. In AAAL, pp. 11270-11277, AAAI Press, 2020.
Youngeun Kim. Joshua Chough, and Priyadarshini Panda. Beyond classification: Directly training spiking neural networks for semantic segmentation. arXiv preprint 1 2021.
10

---

Published as a conference paper at ICLR 2024
Thomas N Kipf and Max Welling. Semi- supervised classification with graph convolutional networks. arXiv preprint wrXiv:1609.0290) 2016.
Junhaeng Lee, Tobi Delbrück, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. CoRR, abs/1608.00 00780 2016.
Jintang Li, Ruofan Wu, Wangbin Sun, Liang Chen, Sheng Tian, Liang Zhu, Changhua Meng, Zibin Zheng, and Weiqiang Wang. What's behind the mask: Understanding masked graph modeling for graph autoencoders. In KDD, pp. 1268-1279. ACM, 2023a.
Jintang Li, Zhouxin Yu, Zulun Zhu, Liang Chen, Qi Yu, Zibin Zheng, Sheng Tian, Ruofan Wu, and Changhua Meng. Scaling up dynamic graph representation learning via spiking neural networks. In AAAI, 2023b.
Jintang Li. Zhouxin Yu, Zulun Zhu, Liang Chen, Qi Yu, Zibin Zheng, Sheng Tian, Ruofan Wu, and Changhua Meng. Scaling up dynamic graph representation learning via spiking neural networks. In AAAI, 2023c.
Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S. Yu. Graph self-supervised learning: A survey. CoRR, abs/2103.00111, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster). OpenReview.net, 2019.
Kai Malcolm and Josue Casco-Rodriguez. A comprehensive review of spiking neural networks: Interpretation, optimization, efficiency, and best practices. CoRR, abs/2303.10780, 2023.
Yujic Mo, Liang Peng, Jie Xu, Xiaoshuang Shi, and Xiaofeng Zhu. Simple unsupervised graph representation learning. In AAAI, pp. 7797-7805. AAAI Press, 2022.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. ICLR. OpenReview.net, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, pp. 8024 0844-8035,00 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 525-542, Cham, 2016. Springer International Publishing.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In ICLR. OpenReview.net, 2020.
Bodo Rueckauer, Iulia- Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer: Theory and tools for the conversion of analog to spiking convolutional neural networks. CoRR,
2016.
Emilio Salinas and Terrence J Sejnowski. Integrate-and-fire neurons driven by correlated stochastic input. Neural computation, 14(9):211-2155: 2002.
Prithviraj Sen, Galilco Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Ran Rad.. Collective classification in network data. Al magazine, 29(3):93 93, 2008.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. Relational Representation Learning Workshop, NeurlPS, 2018.
Shoaib Ahmed Siddiqui, David Krueger, Yann LeCun, and Stéphane Deny. Blockwise self-supervised learning at scale. CoRR, abs/2302.01647,, 2023.
11

---

Published as a conference paper at ICLR 2024
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June (Paul) Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In WWW, www '15 Companion, pp. 243-246. Association for Computing Machinery, 2015. ISBN 9781450334730. doi: 10.1 00.1145524400082744283339
Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Remi Munos, Petar Veličković, and Michal Valko. Bootstrapped representation learning on graphs. In ICLR 2021 Workshop on Geometrical and Topological Representation Learning, 2021.
Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3):4, 2019.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
Junfu Wang, Yunhong Wang, Zhen Yang. Liang Yang. and Yuanfang Guo. Bi-gen: Binary graph convolutional network. In CVPR, pp. 1561-1570. Computer Vision Foundation / IEEE, 2021.
Mingkun Xu, Yujie Wu, Lei Deng, Fagiang Liu, Guoqi Li, and Jing Pei. Exploiting spiking dynamics with spatial-temporal feature normalization in graph learning. In IJCAI, pp. 3207-3213, IJCAT, 2021. Main Track.
Hong Yang, Shirui Pan, Peng Zhang, Ling Chen, Defu Lian, and Chengqi Zhang. Binarized attributed network embedding. In ICDM, pp. 1476-1481. IEEE Computer Society, 2018.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS, 2020.
Junchi Yu, Tingyang Xu, Yu Rong, Yatao Bian, Junzhou Huang, and Ran He. Graph information bottleneck for subgraph recognition. In ICLR, 2021a.
Junchi Yu, Tingyang Xu, Yu Rong, Yatao Bian, Junzhou Huang, and Ran He. Recognizing predictive substructures with subgraph information bottleneck. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021b.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graphsaint: Graph sampling based inductive learning method. In ICLR. OpenReview.net, 2020.
Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S. Yu. From canonical correlation analysis to self-supervised graph neural networks. NewrIPS, 2021.
Yizhen Zheng, Shirui Pan, Vincent C. S. Lee, Yu Zheng, and Philip S. Yu. Rethinking and scaling up graph contrastive learning: An extremely efficient approach with group discrimination. In NeurIPS, 2022.
Ao Zhou, Jianlei Yang, Yingjie Qi, Yumeng Shi, Tong Qiao, Weisheng Zhao, and Chunming Hu. Hardware- aware graph neural network automated design for edge computing platforms. CoRR, Abs/2303.108855 2023.
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. CoRR, abs/2229..15423 2022.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep Graph Contrastive Representation Learning. In ICML Workshop Graph Representation Learning and Beyond, 2020.
Zulun Zhu, Jiaying Peng, Jintang Li, Liang Chen, Qi Yu, and Siqiang Luo. Spiking graph con- volutional networks. In IJCAI, pp. 1434-2440 7 2022. doi: 10.24963/ijcai 2022/338. Main Track.
12

---

Published as a conference paper at ICLR 2024
A THEORY AND PROOF
To establish the expressiveness the proposed model, we define an oracle GNN which performs standard message passing operations with input node feature being the raw features, with per-layer hidden dimension 7. For simplicity, we assume the oracle GNN is instantiated by stacking L GCN layers, with parameters 0. = -
We additionally make two assumptions that are
usually satisfied in graph modeling scenarios:
Assumption 1. The input graph G has its degree bounded from above by D.
Assumption 2. The operator norm of all the weight matrices of the oracle GNN model is bounded from above by i.e., supie[L] Assumption 3. The membrane potential any time step lower bounded by Vi with |Visl I Vv. with some >>> 1.
||W!" llop Vv
!!)
Remark 1. Assumption is actually not necessary and is stated primarily for a sleeker presentation of results. In particular, given assumption we are able to derive lower bound that depends on the GNN matrices' operator norms, length ' as well as the characteristics of the underlying GNN.
For some node v, denote 2, + 8u the firing rate derived from the binaried representation learned by SNNs. The following theorem states that with a sufficiently large time window T, the proposed SNN can implement the oracle GNN with small approximation error.
Theorem 2. Assume that assumptions 3 holdd for any input graph G, let zy be the representation of node v produced by some L-layer oracle GCN model. Under the IF model and reset by subtraction mechanism, there exists SNN such that its firing rate Ey has the following approximation property:
sup sup E 21112 Vdk (VI+ Dv) "
(10)
FEV
TVT
With slight abuse of notation, hereafter we will denote Vi-) either a scalar with value equal to Vit) or vector of length & with each coordinate being V(O), , the meaning shall be clear from the context. The proof relies on the following simple Lemma:
Lemma 1. For any node v and < < L, let (I) and (),()) be the membrane potential and input current at time step < t < T and layer I, further let N.(T) Σt st( be the number of fired spikes throughout the T steps. We have the following inequality:
I
(I(T) = Σ H - N((T)M
(11)
The Lemma follows from the trivial fact that each time upon firing, the membrane potential reduces by exactly Vin-
Proof of Lemma 7
We will first give the construction of the approximation SNN. Recall that the parameter configuration of an L-layer SNN is given by:
Proof of theorem 2
1st-layer Since there are distinct GNN encoders, denote the corresponding weight matrices as w(1) W!!l)
2nd to Lth layer (if exists) We equip each layer with a single weight matrix W 2 I L.
Now construct the approximation SNN as follows: Given the oracle GNN with parameter 0, (w!") w!').. In the first layer, we partition w(!) row-wise into T blocks, with each block having an identical number of rows if d is divided by T' otherwise the first T - 1 blocks constructed with row d//T with the last one having d mod T' rows, denote the resulting blocks (1)
(1)
...,..., (1) r. Then we set ((()) (1(1) 4,
< < T. In the subsequent layers, set
( V4. w! For notational simplicity, we define aav
(IN. for some node pair
13

---

Published as a conference paper at ICLR 2024
(u, r). Now fix some arbitrary node V. Recall the aggregation equation at the I-th layer of the
SNN, with time step
(1(()= -
EmENLUron our w + y EVENSUtO) Our w( a
2 L if
(12)
Here note that in the first layer, the pre-activation output of the oracle GNN is El= (t)
+ Er= (t) for any node v. Now we average Eq. 12 at the L-th (final) layer according to all timesteps, yielding
T
(t)
1
E
Ouv Vn W!!
(4)
Σ su (1)
(13)
T =
T
uEA.U(M)
E WEN.U(O)
aue Vnn W
W! (L) 2/-1
(14)
Now use Lemma
1 at layer
(T)
T
(T)
2g
24
(t)
(15)
T
Vín
T
1=1
auv W (1) (T)
Σ
(16)
TV
wENSU(o)
VT) (17)
Σ
Gue ( ReLU (2
TV
vEN.U(e)
Ti
where the last equality follows since the firing rates are by definition non-negan non-negative. Next we further analyze T, apply Lemma 1 layer L - yields:
VLL ((r)(T)
== Σ
aNo W ReLU
auw W
2
(18)
TV
mEN.U(e)
wENaU(M)
Σ mEN.Ufe)
Ouu !
Σ wEA.Ufat
auw W((-1))
+ aue (!
(19)
Σ
W ReLU
Σ
auw w.'--" ReLU (26-2)
Σ
Ouv W! (1)
aue
WEN.Ufe)
WEN,
mENoU(e)
RL
(20)
where in the remainder term Rd R.1 we define
V (TT
rL-1- ReeU
auw (-(11))
ReLU
auw w!
((b-1))1
Tv
(x)
wENLU(u)
(21)
We will first obtain an upper bound of the l--norm of r specifically, using the inequality |ReLU( + +) - ReLU(x) S lvl.
VT-1(T)
V (T)
d k
||T--1||2 5
V
(22)
TVn
TV
T
14

---

Published as a conference paper at ICLR 2024
where the last inequality follows from the fact the membrane potential never exceeds Ví. and
assumption 3 Next we bound the term R under the GCN model, we further define deg(v) = [Wel-
We have:
| 2
r
(23)
uEN.u(r) V deg(()) deg(v))
K
(24)
uEN,u(r) V deg (1) deg(v))
T
d
SV1+ Duy TT
(25)
where the second inequality follows by assumption? 2 and the last incquality follows by the bounded
degree assumption and deg(u) 2 0. Now we further unravel the right hand side of Eq. 20 untill there are altogether L remainders. is straightforward to check that the leftmost term would be Zy. combining with Eq.
17 write
2
+....R.. R RR,
(26)
with R- =
(2) TVv being the remainder term in Eq. that the remainder terms satisfy:
17
Following similar arguments, we can show
|R'1l2 I (V1 (VI+Du) 1-1 d K
I
(27)
TT
Consequently we have:
[ ==2ll2 5 R + ......... +||2211
(28)
S V d k
[1+..+ (VI+ Dv) 6-1
(29)
TT
ddk (vI+ DDv²
(30)
VTT
Remark The upproximation bound Eg. To dependent on the specifie form of GCN, and therefore has the scaling term O ((vII+ D which is exponential in L. With alternative underlying GNN model, we may get better approximations, for example in the SAGE model Hamilton et al. (017)). following similar arguments, we may prove that we may remove the exponential dependence over (root)-degre bounds, i.e., the scaling factor reduces to O (Lv").
Remark 3. Theorem Theorem@yelies on the reset by subtraction mechanism, for the reset to constant (zero) mechanism, stated in Rueckauer et al. (2016), we have sacrifice error term that does not vanish as grows even in the i.i.d setting.
B VANISHING GRADIENTS IN SURROGATE LEARNING
The vanishing gradient problem is a well-known issue that hinders convergence in modern neural networks. To address this problem, ReLU activation functions were introduced alternatives to Sigmoid and Tanh to avoid dead neurons during backpropagation. However, in SNNs, the vanishing gradient problem can arise again when using a layer of spiking neurons to learn long sequences, similar to the vanishing problem in deep artificial neural networks. We attribute the vanishing gradient problem in SNNs the surrogate gradient learning technique, wherein surrogate functions (typically smooth Sigmoid or Tanh) are used to approximate the gradient of the Heaviside step function in spiking neurons. While surrogate learning plays a crucial role in directly training SNNs, it inherits defects of surrogate functions, resulting in poor training performance and slow convergence. Recent works, such as Fang et al. (2021): Feng et al. (2022), have also noted this issue in SNNs and reached the same conclusion.
15

---

Published as a conference paper at ICLR 2024
Addressing the vanishing gradient problem is essential for unlock- ing the full potential of deep SNNs and enabling them to learn and generalize effeetiielyy Recent success in local learning [Siddiqui et al. (2023) has pointed out new way to train deep modern net- works without suffering severe network degradation and vanishing gradients problems. Specifically, the network is divided into sev- eral individual blocks that are trained locally with gradient isolation during training to avoid a long backpropagation path. Inspired by this approach, we are motivated explore an interesting research direction in SNNs: Is it possible to address the issue of vanishing grudients by adjusting the backprepagation path reasonable way?
Avg. Grad Norm
0.4
Time Step
Figure 5: Average gradient norms of IF and LIF neurons, with and without the block- wise learning strategy.
In this work, we present blockwise training strategy to address the vanishing gradient problem in SNNs. Unlike conventional end- to-end learning, we horizontally divide the encoder network into several individual blocks along the time dimension We limit the length of the backpropagation path to each of these blocks and apply a self-supervised contrastive loss locally at different blocks, using ttop-gradient to ensure that the loss function from block does affect the learned parameters of other blocks. We argue that the blockwise learning rule is unlikely to cause vanishing gradients as long as each block has reasonable depth. We plot the averaged gradient norm of IF and LIF in Figure Sland observe that signals in both IF and LIF neurons are prone to vanishing with an increasing time step. However, the backward gradients exhibit healthy norms in IF and LIF, and the vanishing gradient problem is largely alleviated by incorporating our blockwise optimization paradigm.
C ALGORITHM
Algorithm Spiking Graph Contrastive Learning (SPIKEGCL)
Input: Graph = (V, E, X), encoder fo(-)
[f ..... fū
predictor head (.).), time step T;
Output: Learned encoder fe(-);
1: while not converged do;
2:
Ć = (V, X) < corruption(
3:
for t = 1 to do
4:
Z1 < fb (9");
5:
H, 9s (Zz);
6:
Z₂ < fb. (g*);
7:
H2 + go (Z2):
8:
Calculate contrastive loss J over H1, H2;
9:
Update 0,, ő by gradient descent;
end for
11: end while:
return fo(:);
"Our work differs from Siddiqui
2023 in which the network vertically divided from bottom top.
16

---

Published as a conference paper at ICLR 2024
Algorithm 2 PyTorch-style pseudocode for SPIKEGCL
f: set of peer GNNs
# h: SNN neuron
g: MLP network
# m: margin
perm: random permutation of node features
# T: time steps
G: input graph
# x: node features
x1 torch.chunk (x, T, dim=1)
G1
corruption
x2 torch.chunk chunk (x[:, perm) T, dim=1)
G2 drop_edge (G)
for t in range (T):
z1 f[t] (xl[t,, G1)
# gradient isolation
21 h(zl.detach ())
21 - g (21)
22 = f (x2[t), G2)
# gradieanettt isolation
z2 = h (z2. (e2achht
22 = g (22)
# compute loss & backpropagation
loss mamarinn ranking loss (21, 22, margin=m) loss. (assbaacward())
D DISCUSSION
D.1 DISCUSSION ON COMPLEXITY
As SPIKEGCL uses relatively simple predictor head (a single-layer MLP), main complexity bottleneck lies in the GNN-based encoder network. Therefore, we discuss the overall computation and parameter complexity SPIKEGCL from the perspective of the encoder. Recall that our encoder consists of T peer GNNs coupled with spiking neuron. The computation and parameter complexity of our method is generally times larger than standard GCL. methods that use a single GNN as an encoder. Here omit the bias term in GNN and learnable parameters (if have) in spiking neurons as they do not significantly affect the computation and parameter complexity. Note that we divide the input features into 7' groups and share parameters across different GNNs. Each GNN actually has only F size of parameters with a hidden dimension of compared to the standard case. Therefore, our method achieves similar computation and parameter complexity as traditional GCL methods that use a single GNN as an encoder. Since each GNN accepts each group of features as input and performs message aggregation individually, the computation can be trivially parallelized, which accelerates computations on larger graphs. This further reduces the complexity of our method and endows with desirable scalability not only to larger dataset sizes but also to larger embedding dimensions.
D.2 LIMITATION
We note certain limitations of our work. (i) First, the linear evaluation used our experiments assumes that the downstream task can be solved by linear classifier, which may not always be the case. addition, the quality the learned graph representations may depend on the specific downstream task of interest, and linear evaluation may not be able to capture all relevant aspects of the learned representations for the task. (ii) Second, one common assumption behind the theoretical guarantees is that the neuron adopts 'reset by subtraction' mechanism, which may not always hold
17

---

Published as a conference paper at ICLR 2024
for spiking neurons that the 'reset to zero' mechanism, also mentioned in Remark (iii) Third, our empirical evaluations on real-world datasets do not include any significantly large-scale datasets (N ~ 10 or higher), although our collection of 9 datasets is a broad selection among those commonly used in related research. These aspects mark potential areas for future enhancements and investigations
E DETAILED EXPERIMENTAL SETTINGS
Table 3: Dataset statistics.
Computers
Photo
CS
Physies
Cora
CiteSeer
PubMed
arXiv
MAG
#Nodes
13.752
7,650
18,333
34,493
2,708
3,327
19,717
16,9343
736,389
#Edges
491,722
238,162
163,788
495,924
10,556
9,104
88.648
2,315,598
10,792,672
#Features
767
745
6,805
8,415
1,433
3,703
500
128
128
#Classes
8
15
5
3
40
349
Density
0.144%
0.082%
0.023%
0.407%
0.260%
0.049%
0.042%
0.008%
0.002%
Datasets. We evaluate SPIKEGCL on several graph benchmarks with different scales and properties. Following prior works Zhang al. (202T); Thakoor al. (202T): [Zhu al. (2022). we adopt 9 common benchmark graphs, including two co-purchase graphs, i.e., Amazon- Amazon- ComputerSSccurr Computer Shchur al. (2018), two co-author graphs, i.e., Coauthor-CS and Coauthor-Physics PhysiccSShhhur
Shhchr et al. (2018). three citation graphs, i.e., Cora, CiteSeer, PubMed Sen et al. (208)) as well as two large-scale datasets ogbn-arXiv and ogbn-MAG from Open Graph Benchmark Hu et al. (2020). Dataset statistics are listed in Table 3] For three citation datasets, we evaluate the models on the public full splits introduced in Rong et al. (2020); Chen et al. (2018). we adopt the random 1:1:8 split for Amazon-Computer, Amazon-Photar Coauthor-CS, and Seauthor-Physiee We use stratified sampling to ensure that the class distribution remains the same across splits.
Baselines.
We compare our proposed methods to a wide range of baselines that fall into four
categories:
Full-precision (supervised) GNNs: GCN Kipf & Welling 2016) and GAT Veličkovi et al.
2018))
(2021) BinaryGNN Bahri et al. (2021) and BANE et al. 2018). Bi-GCN and Binary GNN are both supervised methods
1-bit quantization-based GNNs: Bi-GCN Wang et al.
while BANE an unsupervised one.
Fuel-preeision
contrastive methods: DGI Velickovic al. (2019). GRACE Zhu al.
2020), CCA-SSG /Zhag et al. (202T), BGRL Thakoor et al. (2021), SUGRL Mo et al.
2022). and GGD Zheng et al. (022) DGI, GRACE, SUGRL, and GGD are negative- sampling based methods, whereas BGRL and CCA-SSG are negative-sampll--iee ones.
Graph SNNs: SPikingGCN Zhu al. (022)) SpikeNet Li et al. (2023b). GC-SNN and GA-SNN Xu et (202 Note that SpikeNet is initially designed for dynamic graphs, we adapt the author's implementation to static graphs following the practice in Zhu et al. (2022)
Xu et al. (202T). Although these methods are also based on the combination of SNNs and GNNs, they do not fully utilize the binary nature of SNNs to leam binaried representations in an unsupervised fashion.
The hyperparameters of all the baselines were configured according to the experimental settings officially reported by the authors and were then carefully tuned in our experiments to achieve their best results.
Implementation details. We present implementation details for our experiments for reproducibility. We implement model as well as the baselines with PyToorch Paszke et al. (2019) and Py PyTorch Geometric Fey & Lenssen (2019), which are open-source software released under BSD-style B and MIT license, respectivlly All datasets used throughout experiments are publicly available
ht httpss//thhha.oooooooo//cccchhht
18

---

Published as a conference paper at ICLR 2024
PyTorch Geometric library. All experiments conducted NVIDIA RTX 3090 Ti GPU with 24 GB memory unless specified. Code is available at wetttts:///// /tttps:/ggitie httpg/tthhhhh.... com/ www..////iiieeeeeeeeerrreee SpikeGCLJor reproducibility.
Hyperparameter settings. The embedding dimension for each time step is searched in {4, 8, 16, 32, 64). We carefully tune the time step 7 from 8 to 64 to better approximate the full-precision performance. The firing threshold is tuned within {5e-4, 5e-3, 5e-2, 5e-1, 1.0). The margin m is tuned in {0, 0.5, 1.0, 1.5, 2.0]. We initialize and optimize all models with default normal initializer and AdamW optimizer| Loshehilov & Hutter (2019). For all datasets, we follow the practice in [i et al.. (20236) and increase the sparsity of the network using PLIF |LIF|Fang et al. (2020) units, a bio-inspired neuron model that adopts a learnable Tm = 1 to activate sparsely in time solely when crossing a threshold. We adopt Sigmoid as the surrogate function during backpropagation, with smooth factor a = 2.0. Exploration on parameters of SPIKEGCL including time step T, encoder architecture, spiking neurons, and threshold Va are elaborated in Appendix H
Evaluation protocol. For unsupervised or self-supervised methods, including SPIKEGCL, we follow the linear evaluation scheme as described in literature Velickovic et al. (2019) Thakoor et al. (202T): (i) we first train the model on all the nodes in graph with self-defined supervision to learn node representations; (ii) we then train a linear classifier (e.g., logistic regression model) on top with the learned representations under the supervised setting to evaluate the performance. We report averaged accuracy with standard deviation across 10 different trials with random seeds.
Computation of theoretical energy consumption. Different from common ANNs implemented on GPUs, SNNs are designed for neuromorphic chips that adopt synaptic operations (SOPs) to run neural networks in low power consumption manner. However, training SNNs directly neuromorphic chips has been rarely explored in the literature. To investigate the energy consumption of SNN- methods on neuromorphic chips, we follow previous works /huu et al. (2022): Zhou al. (2022) and adopt an alternative estimation approach, which involves counting the total number of spikes generated during the embedding generation process. The computation of energy consumption involves two parts. The first one comes from the spike encoding process, which converts the full-precision node representations into discrete spikes to run on neuromorphic chips (e.g., Bernoulli encoding). is estimated by multiply-and -nd--ann-ccuuulaa (MAC) operations. The second one is the spiking process, which receives the resultant current and accumulates the membrane potential to generate the spikes. Therefore, the theoretical energy consumption can be estimated as follows:
E = Bencoding + Espiking
(31)
=
EMAC
Nd+ ESOP ΣΣS, S/
where S/ denotes the output spikes at time step and layer L. Note that SPIKEGCL has only one spiking neuron, so we count the number of spikes the last layer. According to Zhu et al. (2022); Zhou et al. (2022). EMAC and Esor are set to 4.6pJJ and 3.7pJ, respectively. We also detail how to calculate the theoretical energy consumption of methods other than graph SNNs. For 1-bit GNNs, the processing time required to execute single cycle operation, encompassing one multiplication and one addition, can be utilized to perform 64 binary operations effectively Rastegari et al. (0016). Therefore, use the following formula derived from Wang et al. (2021) to calculate the theoretical energy consumption of 1-bit GNNs (specifically GCN); 1 N de,, Jout + +2N + (8(d 64
E
EMAC
(32)
where d',, and d' are the input and output dimensions of the representation at the 1-th layer. We assume d',, = d'ut = d in this paper. For the other full-precision models deployed on GPUs, we count the number of MAC operations to calculate the theoretical energy consumption. In practice, we split the message passing into two steps: embedding generation and aggregation. In the embedding generation step, the vanilla GNN projects features into a low-dimensional embedding space. For example, given the node features X and a weight matrix W E Bd Xdout, it executes Ndin multiplication and Nd,,.dout addition operations. In the aggregation step, the number of multiplication and addition operations can be simply considered as [E and E'dout- For fair comparison, we only consider the energy consumption of the encoder in GCL, as it is often the main bottleneck.
19

---

Published as a conference paper at ICLR 2024
Table 4: Classification accuracy (%) three citation datasets. The best result for each dataset is highlighted in red. Darker colors indicate larger performance gaps between SPIKEGCL and the best results. (U: unsupervised or self-supervised: S: spike- B: binarized)
Methods
U
S
B
Cora
CiteSeer
PubMed
GCN Kipf & Welling 2016
86.1 10.2
75.9 10.4
88.2 86.8
GAT Veličković et al. 2018
86.710.7
18.5
GC-SNN Xu et al. (0221)
83.8 10.4
13.4
85.3
GA-SNN
Xu et al
2021
86.4
72.8
84.5.0.6
SpikeNet
Li et al.
2023b)
83.5
71.4
83.9=0.4
SpikingGCN Zhu et al.
(2022)
87.7
77.7
87.5
Bi-GCN Wang et al.
(202)
80.1
72.8+0.7
88.6=1.0
Binary GNN Bahri al. (2021)
85.0.0.9
71.9
86.8 20.7
BANE Yang et al.
2018
65.0
64.3
68.8
DGI Velickovic et al. (2019)
86.3
78,9
86.2
GRACE Zhu et (2020)
87.210.2
74.5.00.1
87.3
CCA-SSGZZamp et al. 2021
84.610.7
75.4
88.4
BGRL Thakoor el al. (2021)
87.3
76.0+0.2
88.3
SUGRI Mo et al.
2022
88.0
77.640.4
88.2
GGD Zheng et al.
2022
86.2
75.5
84.2=
SPIKEGCL
87.440.6
77.640.6
88.8
Table 5: The parameter size (KB) and theoretical energy consumption (mJ) of various methods.
Computers
Photo
Physies
arXiv
MAG
#Param.)
Energy
#Param.
Energy
+Parmm
Energy.
*Parmm]
Energy
*Parnm]
Energyl
#Param]
Energyl
GC-SNN
195.9
0.53
258.9
0.29
1812.2
0.16
2221.6
0.11
1812.2
1094.4
SpikeNet
293.6
0.71
484.1
0.62
1790.2
0.46
2197.1
0.23
194.0
0,41
SpikingGCN
7.6
0.68
5.9
0.36
102.0
0.20
42.0
0.21
Average
190.5
0.61
251.9
0.38
1388.7
0.24
1388.7
0.16
194.0
0.41
BL-GON
$29.9
0.67
$17.6
0.33
3623.9
3.4
218.1
376.6
12.3
BinaryGNN
105.3
$81.9
1086.7
Averago
319.1
0.93
311.1
0.41
2252.9
1.86
2765.0
6.3
124.3
1.50
376.6
12.3
DGI
917.5
0.5
906.2
0.21
4008.0 9
4833.3
590.3
590.
568
GRACE
656.1
1.1
644.88
01:47
3747.5
4571.9
13
328.9
21
378
4463
OCA-SSG
22201.2
340
BGRL
787
SUGRL
193.8
189.4
2131.2
147
2615.1
342
99.5
26
99.5
117
GGD
254.7
249.2
3747.3
140
4571.6
340
30.0
100
30.0
1400)
Averae
490.4
11.9
482.2
3198.7
104.5
3906.0
237.6
246.4
68.3
246.4
1279.
SPIKEGCL 60.9
460.7
0.18
TAveraged resalts of baseline methods each section.
F ADDITIONAL EMPIRICAL RESULTS
Performance comparison. We show additional results of SPIKEGCL and baselines three common citation benchmarks: Cora, CiteSeer, and PubMed. We can observe from Table 4 that SPIKEGCL obtains competitive performance compared to state-of-the-art baselines across all datasets, including those using full-precision or binarized representations. This highlights that SPIKEGCL is versatile and effective method for learning node representations in citation networks. It is worth noting that SPIKEGCL achieves these results with significantly fewer bits compared to full-precision representations, which demonstrates its efficiency and scalability. Additionally, SPIKEGCL does not require end-to-end supervision, which makes it more practical for real-world applications.
Parameter size and energy consumption. We calculate the parameter size and the theoretical energy consumption of all methods on six different datasets, as shown in Table Note that BANE, a factorization-bast based approach, is not feasible for estimating its energy consumption, and therefore is omitted from the comparison. demonssaemsssssttss the energy efficiency of graph SNNs and 1-bit GNNs over conventional GCL methods, which have significantly lower energy consumption with
20

---

MAG Published as a conference paper at ICLR 2024
60
0.00
(ccussy (5)
Mcmoryy (GB)
Frange (P-))
Time 54
60
18
0.04
88
5
10
15
20
20
30
10
15
20
25
30
5
10
15
20
25
30
0.00
15
20
25
30
Time stap
Time step
Time stupp
Time
str
Figure 6: Effect of time step T on Computers, Photo, CS, and Physics datasets.
increasing graph scale. In particular, SPIKEGCL exhibits an average of ~50x, ~000x and 10000 less energy consumption compared to graph SNNs, 1-bit GNNs, and GCL baselines, respectively. These results demonstrate that implementing advanced SNNs on energy-efficient hardware can significantly reduce the energy consumption of SPIKEGCL. While we have series of peer GNNs in our encoders corresponding to each group of imputs, SPIKEGCL also exhibits low complexity in parameter size due to parameter sharing among them. Compared GCL baselines, the parameter size of SPIKEGCL has decreased by a factor of 8 on average. Additionally, SPIKEGCL has a smaller model size even compared to 1-bit GNNs and graph SNNs. These results demonstrate that SPIKEGCL not only has strong representation learning abilities but also requires fewer parameters compared other methods.
Effect of time step T. Figure 6 shows the effect of time step T in terms of accuracy performance, running time, memory usage, and energy consumption. The running time is measured as the training time of SPIKEGCL until converged (with early stopping) and the memory usage refers to the maximum GPU memory usage during the training process. We can observe that as 7 increases, the accuracy performance of SPIKEGCL gradually improves until it reaches plateau (c.g., T 25). With the increase of T,, the rumning time and memory usage also slightly increase, which may lead to longer training times and higher hardware requirements. Additionally, the energy consumption of SPIKEGCL also increases with larger values of T, which can be concern for energy-efficieen applications. Therefore, the choice T should strike a balance between accuracy performance and computational efficiency, taking into account the specific requirements of the application hand.
Table 6: Classification accuracy (%) SPIKEGCL on six datasets with different encoder architectures. The best result for each dataset is highlighted in red.
Computers
Photo
CS
Physics
arXiv
SAGE GAT GCN
84.4+uu9
91.4
90.8
94.6=0.1
65.7
28.2
87.9
92.4
90.5
92.6
68.0=a.1
30.8
88.9
93.0
92.8
95.2
70.9
32.0
Effect of encoder architectures. Different GNN architectures offer SPIKEGCL different capabilities in learning graph structure data. In our experiments, we used GCN the default architecture in the encoder, our framework allows various choices of GNN architectures though. In order to explore the performance of SPIKEGCL, we conduct ablation
the impact of different GNN architectures studies on three citation graphs with different GNN encoders, including SAGE Hamilton et al. (0017).
GAT Veličković et al. (2018). and GCN Kipf & Welling (0116). Table 6 shows that GCN is the most effective architecture for SPIKEGLL, which is consistent with prior works VVlickovic et al. 2019);
Li al. (2023a) Thakoor al.) (202T): Zhang al. (202T). However, it is worth noting that other GNN architectures may still provide valuable insights in different scenarios or for different types of data. Therefore, the choice of GNN architecture should be made based on the specific requirements of the task at hand.
Effect of spiking neurons. We provide series of spiking neurons as building blocks for SNNs. including IF Salinas & Sejnowski (2002), 2), LIF Gerstner et al. (2014), and PLIF Fang et al. 2020). The results are shown in Table It is observed that simple IF neuron is sufficient for SPIKEGCL to achieve good performance. By introducing a more biologically plausible leaky term, LIF increases the sparsity of output representations and achieves better performance. Additionally, the leaky term be a learnable parameter, which endows the network with better ficxibility and biological plausibility. Therefore, in most cases, PLIF achieves slightly better performance than LIF. Overall, the choice of
21

---

Published as a conference paper at ICLR 2024
Table 7: Classification accuracy (%) of SPIKEGCL six datasets with different spiking neurons. The best result for each dataset is highlighted in red.
Computers
Photo
CS
Physics
arXiv
MAG
80.6
92.9+0.2
91.2++0.6
95.0+0.5
68.2+
31.0+0.5
LIF
88.610.1
92.910.1
92.1 92.1
10.3
95.3 1o.1
68. 18.8
29.8 29.8
PILF
88.910.3
93.0
92.8
95.210.6
70.910.1
32.0
spiking neuron should be based on the specific requirements of the task and the available hardware. For example, IF neurons may be more suitable for tasks with lower computational requirements, while LIF PLIF neurons may be more suitable for tasks that require higher levels of sparsity or greater biological plausibility.
60
Accuracy (%)
Sparsity (%)
Ouepator
20
Pro
5e
5e 5e 5e
1.0
5e-1
5e
5e
5e-1
1.0
Threshod
Threshod
Figure 7: Accuracy (%) and output sparsity (%) of SPIKEGCI in four datasets.
Effeet of threshold Vun The neuron threshold directly controls the firing rates of spiking neurons and, thereby, the output sparsity. We conducted an ablation study on Vn, on four datasets to investigate its effect, as shown in Figure / We observed that the performance of SPIKEGCL is sensitive to the value of Vnn on two dense datasets, Computers and Photo. When Vnn is too low, the output sparsity may be too high, leading to a loss of information and compromised performance. When Vn is too high, the firing rates of the spiking neurons may be too low, resulting in a lack discriminative power and reduced performance. In contrast, the performance of SPIKEGCL is stable with different Vi on CS and Physics, two relatively sparse datascts. This suggests that a dense datasct requires a smaller Vn capture the underlying structure. We also that the output spikes become sparser as Vnn increases. Overall, our ablation study demonstrates the importance carefully selecting the value of Vn achieve optimal performance with SPIKEGCL. The optimal value of Vnn may depend on the specific dataset; therefore, it should be chosen based on the empirical evaluation.
Convergence. We compared the convergence speed of SPIKEGCL with full-precision GCL baselines on Computers, which shown Figure g It is observed that negative-sample-fice methods generally converged faster than regatve-sample-based ones. However, they still required sufficient epochs gradually improve their perfor- mance. In contrast, SPIKEGCL that trained blockwise training paradigm demonstrates significantly faster convergence speed over full-precision methods. In particular, 1-2 epochs are sufficient for SPIKEGCL learn good representations. Overall, our experiments demonstrate the effectiiveness of SPIKEGCL in improving the con- vergence speed of SNNs and provide insights into the benefits of the blockwise training paradigm.
Accuracy (%)
80
GCD GRAZ
70
85R)
20
40
60 Epoch
80
100
Figure 8: Convergence speed comparison among SPIKEGCL and full-precision methods. Blue: negative- sample-based methods; green: Sggativ-sammpl--fre methods
Connections between original input features and output spikes. To gain a deeper understanding of SPIKEGCL, we assess the sim- ilarity between input features (partitioned by T groups) and the corresponding output spikes using the Centered Kernel Alignment metric. Centered Kernel Alignment (CKA) Nguyen et al. (2021) serves a representation similarity metric extensively utilized for comprehending the representations learned by neural networks. CKA takes two representations X and Y input and calculates their normalized similarity, measured in
22

---

Published as a conference paper at ICLR 2024
94
10
Meroon (CB)
10
15
20
25
30
10
15
20
25
30
10
15
20
25
30
70
15
20
25
30
Time stop
Timeo stri
Time step
(a) Photo
80
Accuraty 1%6
Memooyy (CB)
10
10
16
20
25
30
10
15
20
16
20
25
30
AlRp
(b)
rccurpyy (5))
Energr 1111
0.4
8
144
Time
Memory
82
0.0
(c) Physics
Figure 9: Comparison of SpikeGCL and other graph SNNs in terms of accuracy (%), training time (s), memory usage (GB), and energy consumption (mJ), respectively.
terms of the Hilbert-Schmidt Independence Criterion (HSIC):
HSTC (K, L)
CKA(K, /HSICo K HHSICOLL, L)
(33)
Where K and L are similarity matrices of X and Y respectively. The results on Computers and Photo are presented in Figure TO As evident from the results, each group of features exhibits strong correlation with the corresponding output spikes while demonstrating minimal correlation with spikes other time steps. This reflected the CKA similarity matrices, with values along the diagonal being notably larger than those elsewhere. The results have also suggested that the learned binary representations are disentangled from each other, thereby providing improved expressiveness in representing the input features.
Computers
Photo
Input Feature Groups
25
0.20
Imput Feature Groups
25
0.175
20
0.150
0.16
0.125
15
0.10
0.100
10
0.06
0.050
0.025
20
25
Output Spike Groups
Output Spike Groups
Figure 10: CKA similarity between different input feature groups and output spike groups.
Effect of reset mechanisms. We additionally include the ablation results on reset mechanisms of different spiking neurons, shown in Table As can be observed, the performance of SPIKEGCI.
23

---

Published as a conference paper at ICLR 2024
Table 8: Classification accuracy (%) of SPIKEGCL four datasets with different reset mechanisms and spiking neurons. The best result for each dataset is highlighted in red.
Computers
Photo
CS
Physics
zero-reset + IF subtract-reset IF zero-reset + LIF subtract-reset LIF
88.0
90.7
91.240.11
95.0
88. 10.1
92.9
91.210.1
95.010.1
88.5
92.910.3
92.010.2
95.240.1
88.6
92.940.1
92.1
95.3+0.2
zero-reset PLIF SPIKEGCL (subtract-reset + PLIF)
89.0 +0.3
92.5
10.1
95.2
88.9
93.0
92.8
95.2
does not necessarily depend on the specific implementation details of the reset mechanisms. The performance gap between the best architecture and the worst is not significant. Even simple IF neuron with zero-reset mechanism can achieve satisfactory performance.
24


Reviews:
[{"review_id": "Fo5JXcyc1I", "reviewer": "Reviewer_n3GK", "summary": "This paper presents a novel graph contrastive learning (GCL) framework called SPIKEGCL, which leverages sparse and binary characteristics to learn more biologically plausible and compact representations. The proposed framework outperforms many state-of-the-art supervised and self-supervised methods across several graph benchmarks, achieving nearly 32x representation storage compression. The paper also provides experimental evaluations and theoretical guarantees to demonstrate the effectiveness and expressiveness of SPIKEGCL.", "questions": "check the comments above", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper propose a novel GCL framework called SPIKEGCL that leverages sparse and binary characteristics to learn more biologically plausible and compact representations. \n2.\tThis paper provides theoretical guarantees to demonstrate the expressiveness of SPIKEGCL.\n3.\tSpikeGCL nearly 32x representation storage compression and outperforming many state-of-the-art supervised and self-supervised methods across several graph benchmarks. \n4.\tExtensive experimental evaluations to demonstrate the effectiveness of the proposed framework.", "weaknesses": "1.\tIn Section 4.1, to reduce the complexity of SNNs by sampling from each node, the authors uniformly partition the node features into T groups, which is unreasonable. Features of different dimensions may represent different meanings, and operations after grouping these features may lead to inconsistencies in the feature space between different groups. On the contrary, the traditional mask method retains most features by randomly masking some features, ensuring the consistency of feature distribution. Therefore, in this section, the author can consider using random masks to reduce computational complexity while ensuring the consistency of data distribution.\n2.\tIn table 2, the authors compare the parameter size and energy consumption between proposed method with traditional unsupervised/self-supervised mehtods. Howvere, from table 1, the performance of spikeGCL is worse than the spike-based mehtods in most cases, there’s no evidence that SpikeGCL is better than other mehtods. The authors should add the comparision between SpikeGCL with spike-based mehtods in Table 2.\n3.\tAn intuitive question: Contrastive learning usually generates rich features from multiple perspectives to represent the target. However, spike-based methods usually lose a large amount of data, that is, a large number of learnable features are lost. Why can SpikeGCL still achieve similar results compared with traditional contrastive learning methods?\n4.\tGenerally speaking, combining Spiking and GCL is a good idea, but the novelty is not enougt. Compared with traditional methods, SpikeGCL only groups features and then uses the traditional GCL method for learning, which does not present the special nature of contrastive learning in the scenario where spike and graph are combined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OBngzI0iUZ", "reviewer": "Reviewer_jihE", "summary": "The paper addresses the challenge of learning full-precision representations in graph neural networks,\nwhich can be computationally and resource-intensive. The authors propose a new approach that\ncombines graph contrastive learning with spiking neural networks to improve efficiency and accuracy.\nThe proposed framework, SPIKEGCL, learns binarized 1-bit representations for graphs and provides\ntheoretical guarantees to demonstrate its comparable expressiveness with full-precision counterparts.", "questions": "Please see the weakness section for detailed questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The motivation is clear. The paper combines graph contrastive learning with spiking neural\nnetworks to improve efficiency and accuracy.\n2. The proposed method is tested on several benchmarks.\n3. The paper is well-written and provides a promising direction for graph contrastive learning with\nspiking neural networks.", "weaknesses": "1. The author divided the original graph in time in the feature dimension and obtained T graph\nstructures with the same structure and reduced the node feature dimension to N/T. Compared with\ncopying T copies, it saves storage resources. However, the author did not explain the reason for\nthis approach. For example, from my personal understanding, the author's approach can be\nunderstood as for a 1xd feature vector, there is a temporal relationship between the 0th value and\nthe N/T-th value, which we can’t understand.\n2. The author used the SNN method to compress the original representation. One problem is that\nSNN considers the accumulation in time and does not take into account the distribution\ncharacteristics in time. What I mean is, if the characteristics of time T-1 and time T-2 are\nexchanged. It seems that the value of time T will not be affected, but they will become two\ncompletely different vectors. In this way, will there be a many-to-one situation during the\ncompression process?\n3. I think the article lacks some quantitative analysis, such as what is the connection between the\ncompressed binary vector and the original vector, what is the distribution of the conventional", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BzSZ4ISiU8", "reviewer": "Reviewer_HWYD", "summary": "The paper presents SpikeGCL, a GCL framework built upon SNNs to learn 1-bit binarized graph representations and enable fast inference. The authors shows that Spike GCL achieves high efficiency and reduces memory consumption, and is also theoretically guaranteed with powerful capabilities to learn representations. Extensive experimental results verified that spikeGCL achieves comparable or superior performance to full-precision competitors.", "questions": "How much does the result rely on detailed implementation of the SNN (such as reset to 0/reset by subtraction/IF or LIF)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a new framework for learning on graph data, the spikeGCL. The paper is well written with clear introduction of the model and the learning algorithm to prevent the vanishing gradient problem, and presents both theoretical guarantees and extensive numerical results to demonstrate the capabilities of the model.", "weaknesses": "The paper focuses on the learning algorithm and performance of SpikeGCL, I think it would be interesting to further explore the properties of the 1-bit node representations themselves and compare them with other baseline models, to better understand why the learned graph representations are superior to other binary GNNs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r1i31ADlgH", "reviewer": "Reviewer_bEwP", "summary": "This work presents SpikeGCL which targets on optimizing GCL with SNN. They provide detail experiments to demonstrate the efficiency of the proposed method.", "questions": "1.\tIn Sec3, does the problem formulation special for this work, or it is general for all GCL application? Please make it clearly. \n2.\tI think author should formulate the GCL problem in background (instead of giving a brief introduction). Also, it is better to highlight which part is optimized by the proposed methods.\n3.\tWhy T encoders? Usually, neurons adopt the same weight among T time-steps, this design may increase the model size. Also, it is not clear how computation complex relates to time step size, since the author claim that they partition the feature dimension into T blocks. In my opinion, the computation complexity would not change when modifying T, which is not consist to Fig4.\n4.\tFig3(b) ‘y1->y2’. Also, it is not clear how the entire backpropagation work. whether all y1…yn can directly receive gradient from the loss? Sec 4.4 introduces too much previous studies, it is better to clarify the gradient diagram in revision, i.e. for different blocks, where the gradient come from\n5.\tHow SPIKEGCL can reduce parameter size? Usually, SNN can reduce the activation size but keep parameter size unchanged. Author should provide a diagram of how to compute the parameter size.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tConnect SNN and GNN is a very important topic, since GNN is closer to the neuron system and SNN is closer to the neuron dynamic.\n2.\tIt is interesting to combine feature dimension and temporal axis.", "weaknesses": "1.\tThe detailed background does not provide in background section, and there are too many existing work descriptions in the method sections.\n2.\tSome design choices do not present.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)