FiGURe: Simple and Efficient Unsupervised Node Representations with Filter Augmentations
Authors: Chanakya Ekbote
Ajinkya Deshpande
Arun Iyer
SUNDARARAJAN SELLAMANICKAM
Ramakrishna B Bairi
Venue: NeurIPS 2023 poster (2023)
Keywords: Graph Neural Networks
Unsupervised Representation Learning
Graph Filters
TL;DR: This work proposes filter based augmentations for learning node representations within graph-structured data.

Abstract: Unsupervised node representations learnt using contrastive learning-based methods have shown good performance on downstream tasks. However, these methods rely on augmentations that mimic low-pass filters, limiting their performance on tasks requiring different eigen-spectrum parts. This paper presents a simple filter-based augmentation method to capture different parts of the eigen-spectrum. We show significant improvements using these augmentations. Further, we show that sharing the same weights across these different filter augmentations is possible, reducing the computational load. In addition, previous works have shown that good performance on downstream tasks requires high dimensional representations. Working with high dimensions increases the computations, especially when multiple augmentations are involved. We mitigate this problem and recover good performance through lower dimensional embeddings using simple random Fourier feature projections. Our method, FiGURe, achieves an average gain of up to 4.4\%, compared to the state-of-the-art unsupervised models, across all datasets in consideration, both homophilic and heterophilic. Our code can be found at: https://github.com/Microsoft/figure.

FiGURe: Simple and Efficient Unsupervised Node Representations with Filter Augmentations
Chanakya Ekbote"
Ajinkya Pankaj Deshpande
Microsoft Research India chanakyackoote@gmalll com
Microsoft Research India aj ajinkya. pat. www.ppanddd66@e@maa1......
com
Arun Iyer Microsoft Research India ari ariy@microsoft.. crosoft. com
Ramakrishna Bairi
Sundararajan Sellamanickam Microsoft Research India com
Microsoft Research India
rkbairi@@mail.cc com
asrajanSun www.oft.oooooooomoom www.ojaoo...ooott.....
Abstract
Unsupervised node representations learnt using contrastive learning- -based methods have shown good performance downstream tasks. However, these methods rely on augmentations that mimic low-pass filters, limiting their performance on tasks requiring different eigen-spectrum parts. This paper presents simple ilter-based augmentation method to capture different parts of the eigen-spectrum. We show significant improvements using these augmentations. Further, show that sharing the same weights across these different filter augmentations is possible, reducing the computational load. In addition, previous works have shown that good performance on downstream tasks requires high dimensional representations. Working with high dimensions increases the computations, especially when multiple augmentations are involved. We mitigate this problem and recover good performance through lower dimensional embeddings using simple random Fourier feature projections. Our method, FiGURe, achieves an average gain of up to 4.4%, compared to the satte-of-th--as art unsupervised models, across all datasets in consideration, both homophilic and heterophilic. Our code can be found at: https: / / thub. com/ hicrosoft/figuzee
1 Introduction
Contrastive learning is powerful method for unsupervised graph representation learning, achieving notable success in various applications [44, 15]. However, these evaluations typically focus on tasks exhibiting homophily, where task labels strongly correlate with the graph's 's structure. An existing edge suggests the connected nodes likely share similar labels these scenarios. However, these representations often struggle when dealing with heterophilic tasks, where edges tend to connect nodes with different labels. Several papers [7, 17, 4, 28) have tackled the problem of heterophily by leveraging information from both low and high-frequency components. However, these methods operate in the suni-supervieed setting, and the extension of these ideas in unsupervised learning still needs to be explored. Inspired by the insights in these papers, we propose a simple method incorporating these principles.
Our approach introduces filter banks as additional views and learns separate representations for each filter bank. However, this approach faces two main challenges: Firstly, storing representations from each view can become prohibitively expensive for large graphs; secondly, contrastive learning
Both authors contributed equally to this work. Work done while the authors were Research Fellows at Microsoft Research India.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

methods typically demand high-ding dith-eiinnii representations, which increase both the computational cost of training and the storage burden. We employ shared encoder for all filter banks to tackle the first challenge. Our results confirm that a shared encoder performs on par with independent encoders for each filter bank. This strategy enables us to reconstruct filter-specffic representations as needed, drastically reducing the storage requirement. For the second challenge, we train our models with low-dimensional embeddings. Then, we use random Fourier feature projection [40] to lift these low-dimensional embeddings into a higher-dimennional space. Kernel tricks [23] were typically used in classical machine learning to project low-dimensional representation to high dimensions where the labels can become linearly separable. However, constructing and leveraging the kernels in large dataset scenarios could be expensive. To avoid this issue, several papers [40, 41, 20, 36, 26] proposed to approximate the map associated with the kernel. For our scenario, we use the map associated with Gaussian kernel [40]. We empirically demonstrate that using such simple approach preserves high performance for downstream tasks, even in the contrastive learning setting. Consequently, our solution offers a more efficient approach to unsupervised graph representation learning in computation and storage, especially concerning heterophilic tasks. The proposed method exhibits simplicity not only in the augmentation of filters but also in its ability to learn and capture information in a low-dimensional space, while still benefiting from the advantages of digendimensional embeddings through Random Fourier Feature projections.
Our contributions in this work are, 1] We propose simple scheme of using filter banks for learning representations that can cater to both heterophily and homophily tasks, 2] We address the computa- tional and storage burden associated with this simple strategy by sharing the encoder across these various filter views, 3] By learning a low-dimensional representation and later projecting it to high dimensions using random Fourier Features, further reduce the burden, 4] We study the perfor- mance of our approach on four homophilic and seven heterophilic datasets. Our method, FiGURe, achieves an average gain of up to 4.4%, compared to the state-of-the-art unsupervised models, across all datasets in consideration, both homophilic and heterophilic. Notably, even without access to task-specific labels, FiGURe performs competitively with supervised methods like GCN [25].
2 Related Work
Several unsupervised representation learning methods have been proposed in prior literature. Ran- dom walk-based methods like Vede2 [14] and DeepWalk [37] preserve node proximity but tend to neglect structural information and node features. Contrastive methods, such as DEEP GRAPH INFOMAX (DGI) [44], maximize the mutual information (MI) between local and global representa- tions while minimizing the MI between corrupted representations. Methods like MVGRL [15] and GRACE [45] expand on this, by integrating additional views into the MI maximization objective. However, most of these methods focus on the low frequency components, overlooking critical insights from other parts. Semi-supervised methods like GPRGNN [7], BERNNET [17], and PPGNN [28] address this by exploring the entire eigenspectrum, but these concepts are yet to be applied in the unsupervised domain. This work proposes the use of filter bank to capture information across the full eigenspectrum while sharing an encoder across filters. Given the high-dimensional representation demand of contrastive learning methods, we propose using Random Fourier Features (RFF) project lower-de siwer-dimensional embeddings into nigher-dimension sacrificing performance. The ensuing sections define our problem, describe filter banks and random feature maps, and explain our model and experimental results.
spaces, reducing computational load without
3 Problem Setting
In the domain of unsupervised representation learning, our focus lies graph data, denoted as g = (V,),, where V is the set of vertices and E the set of edges (E C V x V). We associate an adjacency matrix with G, referred to A A (0, 1 | where n | corresponds to the number of nodes. Let X E Rn be the feature matrix. We use Ar to represent A + I with I is the identity matrix, while DAi signifies the degree matrix of A₁. We also define A, as 11/ A, A, 1/2
No additional information is provided during training. The goal is to learn parameterized encoder, Es Raax x Baxd, + nxd where d' & d. This encoder produces a set of node representations Eg X, An) = (h1, h2. .... }nd where each h2 E Rdd represents a rich representation for node i. The
2

---

subsequent section will provide preliminary details about filter banks and random feature maps before we discuss the specifics of the proposed approach.
4
Preliminaries
Our approach relies on filter banks and random feature maps. In this section, we briefly introduce these components, paving the way for detailed explanation of our approach.
4.1 Filter Banks
Graph Fourier Transform (GFT) forms the basis of Graph Neural Networks (GNNs). A GFT is defined using a reference operator R which admits spectral decomposition. Traditionally, in the case of GNNs, this reference operator has been the symmetric normalized laplacian Ln = I An or the An as simplified in [25]. A graph filter is an operator that acts independently on the entire eigenspace of diagonalisable and symmetric reference operator R. by modulating their corresponding eigenvalues. [43, 42]. Thus, a graph filter H is defined via the graph filter function g(.) operating on the reference operator as = (
U9(A)U". Here, A diag([A1- A2 .... Anl). where A; denotes the eigenvalues of the reference operator. We describe filter bank as a set of filters, denoted as F (F,, F2 2,,... F Fx). Both GPRGNN [7] and BERNNET [17] employ filter banks, comprising of polynomial filters, and amalgamate the representations from each filter bank to enhance the performance across heterophilic datasets. GPRGNN uses a filter bank defined as
GGRRNNN (I, An...., A, K-1), while BERNNET = (Bo. B,..... B characterizes the filter bank utilized by BERNNET. Here, B, = अ (K-) -
Each filter in these banks highlights different parts of the eigenspectrum. By tuning the combination downstream tasks, it offers the choice to select and leverage the right spectrum to enhance performance. Notably, unlike traditional GNNs, which primarily emphasize low-frequency components, higher frequency components have proved useful for heterophily [4, 7, 17, 28]. Consequently, a vital takeaway is that for comprehensive representations, must aggregate information from different parts of the eigenspectrum and fine-tune it for specific downstream tasks.
4.2 Random Feature Maps for Kernel Approximations
Before the emergence of deep learning models, the kernel trick was instrumental in learning non-linear models. A kernel function, k IR IRd + R. accepts two input features and returns real-valued score. Given a positive-defintt kernel, Mercer's Theorem [30] assures the existence of feature map (-)), such that k(x, y) (ő(x), (vy)) Leveraging the kernel trick, researchers combined Mercer's 's theorem with the representer theorem [23], enabling the construction of non-linear models that remain linear in These models created directly using instead of the potentially complex d, outperformed traditional linear models. The implicit maps linked with these kernels projected the features into a significantly high-dimensional space, where targets were presumed to linearly separable. However, computational challenges arose when dealing with large datasets. Addressing these issues, subsequent works [41, 20, 36, 40] introduced approximations of the map associated with individual kernels through random projections into highher-iieensional spaces (ø' (.)). This approach ensures that (Ø' (x), ø' (yy)) s k(2, y). These random feature maps are inexpensive to compute and affirm that simple projections to higher-ding digeer-dimensional spaces can achieve linear separability. The critical insight is that computationally efficient random feature maps, such as Random Fourier features (RFF) [40], exist. These maps project lower-dimensional representations into higher dimensions, enhancing their adaptability for downstream tasks..
Proposed Approach
The following section delineates the process of unsupervised representation learning. Then, we detail the use of filter bank representations in downstream tasks with random feature maps.
5.1 Unsupervised Representation Learning
Our method FiGURe (Filter-based Graph Unsupervised Representation Learning) builds on concepts introduced in [18, 44], extending the maximization of mutual information between node and global
3

---

filter representations for each filter in the filter bank F = F2. In this approach, we employ fltterbased augmentations, treating filter banks as "additional views" within the context of contrastive learning schemes. In the traditional approach, alternative baselines like DGI have employed a single filter in the GPRGNN filter bank. Additionally, MVGRL. attempted to use the diffusion kernel; nevertheless, they only learned a single representation per node. We believe that this approach is insufficient for accommodating a wide range of downstream tasks. We construct an encoder for each filter to maximize the mutual information between the input data and encoder output. For the 7th filter, we learn an encoder, Eg Xr + X. denoted by learnable parameters In
this context, X; represents set of examples, where each example [X.j. Fiyl E X; consists of a filter F, its corresponding nodes and node features, drawn from an empirical probability distribution P,, That is, P. captures the joint distribution of a filter F,, its corresponding nodes and node features ij: Fey] denote nodes, node features and edges sampled from the 2th filter (X, F.D basis the probability distribution P, Note that X.j E R.V' xd and Fij E R xN' X defines the set of representations learnt by the encoder on utilizing feature information as well as topological information from the samples, sampled from the joint distribution P,, The goal, aligned with [29, 18, 44], is to identify that maximizes mutual information between [X, Fyj and Eg (X, F,). or I (X, F, a). (X, F.).. While exact mutual information (MI) computation is infeasible due to unavailable exact data and learned representations distributions, we can estimate the MI using the Jensen-Shannon MI estimator [9, 32], defined as:
FX) Note that ixi
IS (X, F. (x : spp ([ Fuyl ( Fo))
(I)
xP, ( ([Xi), F Fijl, Eg (Xij FiyI
Here, To : X3 x X"; - R represents a discrimi- nator function with learnable parameters Note that [X.,. is an input sampled from P., which denotes a distribution over the corrupted input data (more details given below). The function sp(.) cor- responds to the softplus function [10]. Addition- ally, To.. ((h,ph. [h.yl2) ,.,, o (R((hg)))) 12, where R denotes the readout function responsible for summarizing all node representations by aggre- gating and distilling information into a global filter representation. We introduce a Icarnable discrimina- tor Dw. where (.,..) represents the joint probabil- ity score between the global representation and the node-specific patch representation. Note that [h.y]] 1. denotes representations obtained after passing sam- ples sampled from the original distribution P., to the encoder, and [hsj]e, denotes representations obtained after passing samples sampled from the original dis- tribution Pr or samples sampled from the corrupted distribution P. to the encoder. Intuitively, Eq. 1 im- plies that we would want to maximize the mutual information between the local (patch) representations of the nodes and their corresponding global graph representation, while minimising the mutual information between a global graph representation
Fiter Banks
Shared Enceding
Shared Encoder Weight Matnx
F
Node Frribeddings
e
Nodes with
Mutual Information Maximization
edges
eiF,x0)
Hr
Fittered Graph
Droweddingg)
maxx MA(A
ReOCOLt
Graph
MI(A
Figure 1: Unsupervised learning of node em- beddings by maximizing mutual information between node and graph representations over the graphs from the filter bank. Note that the parameter θ is shared across all the filters.
and the local representations of corrupted input data. Note that: [X.j Fyg] denotes a corrupted version of input features and given filter. More details with regards to this will be given below. In our approach, we first obtain node representations by feeding the filter-specfiic topology and
associated node features into the encoder: H.j- = Eo (X.j, F.y) = {hr ,12 ..... h ). Note that H. has the following dimensions IR N' xd To obtain global representations, we employ a readout function R R xd + R. which combines and distills information into global representa- tion 1 R(H,,) (R(Fa) XX,; F.y)). Instead of directly maximizing the mutual information between the local and global representations, we maximize (.....). This joint score should be

---

higher when considering global and local representations obtained from the same filter. opposed the joint score between the global representation from one filter and the local representation from corrupted (Xij. Fij). To generate negative samples for contrastive learning, we employ
N x RN' XN', which yields corrupted samples denoted as ix Ful = C(X.j, The designed corruption function generates data decorrelated with the input data. Note that [Xij. i Fúy] denote nodes, node features and edges sampled from the corrupted version of 7th filter, basis the probability distribution P,, The corruption function can be designed basis the task at hand. We employ simple corruption function, whose details are present in the experimental section. Let the corrupted node representations be follows:
corruption function C IR-' xd x R
H,y = (C(C(Xi, i; Fú)) = Ea (X.,, Fiy) (11 F. 112 Fs ,.... 1 J. In order to learn representations across all filters in the filter bank, we aim maximise the average estimate of mutual information (MI) across all filters, considering K filters, defined by Ip.
K
IF
1 K
E
JSD 1,0 ( E, (X(,F.))
Maximising the Jenson-Shannon MI estimator can be approximately optimized by reducing the binary cross entropy loss defined between positive samples (sampled from the P,) and the negative samples (sampled from P,)) Therefore, for each filter, the loss for defined as follows:
LF,1
1 2N P.l-PP))
DD Σ log( Du (h Fu " ())) )) + log(1 (1- Dhu Ps n [
(3)
Note that for Eq. 3, the global representation (h is generated by R(Eg(Xij) Fiy)). The local
representations (h+ V k) are constructed by passing the sampled graph and features through cortuption function (see H;y). Therefore to learn meaningful representations across all filters the following objective is minimised:
K K
C
LF:
Managing the computational cost and storage demands for large graphs with distinct node repre- sentations for each filter poses challenge, especially when contrastive learning methods require high dimensions. To address this, we employ parameter sharing, inspired by studies like [7] and [17]. This approach involves sharing the encoder's parameters and the discriminator's parameters w across all filters. Instead of storing dense filter-specific node representations, we only store the shared encoder's parameters and the first-hop neighborhood information for each node per filter, reducing storage requirements. To obtain embeddings for downstream tasks, we reconstruct filter-specific representations using a simple lay-r GNN. This on-demand reconstruction significantly reduces computational and storage needs associated with individual node representations. Fig 1 illustrates such simple encoder's mutual information- based learning process. To address the second challenge, we initially train our models to produce now-dimensionss embeddings that capture latent classes, as discussed in [2]. These embeddings, while informative, lack linear separability. To enhance separability, we project these low-dimensional embeddings into higheradimensional space using random Fourier feature (RFF) projections, inspired by kernel methods (see Section 4.2). This ap- proach improves the linear separability of latent classes, confirmed by experimental results in Section 6.2, demonstrating the retention of latent class information these embeddings.
5.2 Supervised Representation Learning
After obtaining representations for each filter post the reconstruction of the node representations, learning an aggregation mechanism to combine information from representations that capture different parts of the eigenspectrum for the given task is necessary. We follow learning schemes from [7, 17, 28),
5

---

Filber Banks
Encode from Trained
Node Embbeddings
Higer Dimm-nDiennn Random Fourler Fealures
R
Projection
+8 Nodes with self-edges
cos(w' A
Embeddinng
Fusion
H4
Projection
probsbiitiee
(N
clovolier
ros(w" Ie
Fittered Graph
Emibeddings
o(rixo)
Projection
CoOTw W
Filtered Graph
Figure 2: Supervised Learning: Using the trained parameter θ, we generate the node embeddings by encoding the filtered graphs that get consumed in the classification task.
learning weighted combination of filter-specific representations. The combined representations for downstream tasks, considering K filters from filter bank F, are as follows:
K a ((Fo((X,F)))
Z =
(5)
The parameters o;'s 's' are learnable. Additionally, the function d'(.) represents either the RFF projec- tion or an identity transformation, depending on whether Egg (X. F,) low-dimensional or not. classifier model (e.g. logistic regression) consumes these embeddings, where train both the and the weights of the classifier. Fig 2 illustrates this process. Notably, semi-supervised methods like [28, 7, 17] differ as they learn both encoder and coefficients from labeled data, while our method pre-trains the encoder and learns task spec-fie combinations of filter-specific representations.
Experimental Results
Training Details: We define a single-layer graph convolutional network (GCN) with shared weights (9) across all filters in the filter bank (F) as encoder. Therefore, the encoder be expressed as follows: (X, F,) = (F))... is important to note that F; represents a normalized filter with self-loops, which ensures that its cigenvalues
within the range of [0, 2]. The non- non-linearity function i refers to the parametric rectified linear unit [16]. As we work with single graph, we obtain the positive samples by sampling nodes from the graph. Using these sampled nodes, we construct adjacency list that only includes the edges between these sampled nodes filter F,- On the other hand, the corruption function operates on the same sampled nodes. However, it randomly shuffles the node features instead of perturbing the adjacency list. In essence, this involves permuting the node features while maintaining consistent graph structure. This action introduces form of corruption, altering each node's feature to differ from its original representation in the data. Similar to [44], we employ straightforward readout function that involves averaging the representations across all nodes for specific filter F,: RHH))
"1 where denotes the sigmoid non-linearity. We utilize a bilinear scoring function (.,.,,), whose parameters are also shared across all filters, where (h F. hr.) a (1.F22 Wh. We learn the encoder and discriminator parameters by optimising Eq. 4. While we could use various filter banks, we specifically employ the filter bank corresponding to GPRGNN (F gprgnN) for all our experiments. However, conduct an ablation study (see 6.5) comparing FGPRGNN with F BBrrNNer Additional training details are available 8.3.
We conducted series of comprehensive experiments to evaluate the effectiveness and competitiveness of our proposed model compared SOTA models and methods. These experiments address the following research questions: [RQ1] How does FiGURe, perform compared to SOTA unsupervised models? [RQ2] Can we perform satisfactorily even with lower dimensional representations using projections such as RFF? [RQ3] Does shared encoder decrease performance? [RQ4] What is the computational efficiency gained by using lower dimensional representations compared to methods that rely higher dimensional representations? [RQ5] Can alternative filter banks be employed to

---

recover good quality representations? [RQ6] How does FiGURe combine information from different filters (in F GPRRNNN) task-dependent manner?
Datasets and Setup: We evaluated our model on a diverse set of real-world datasets, which include both heterophilic and homophilic networks, to assess its effectiveness. Similar to previous works, utilized the node classification task as a proxy to evaluate the quality of the learned representations. Please refer 8.2 for detailed information about the benchmark datasets.
In our comparison against baselines, we considered common unsupervised approaches, such as DEEPWALK and NODE2 and state-of-the- mutual information- basodd methods, namely DGI, MVGRL, GRACE, and SUGRL. We also include the performance numbers of the widely used GCN for reference. Unless stated otherwise, all reported results, following prior research. For additional details please refer to 8.3 and 8.4.
Baselines:
5 512-dimensional representation size for
6.1 RQ1: FiG versus SOTA Methods
Table 1: Contains node classification accuracy percentages on homophilic and heterophilic datasets. FiGURe RFP and FiGURe
RFF refer to FiGURe trained with 32 and 128 dimensional representations, respectively, and then projected using RFF. Other models, including FiGURe512 and all baselines are trained at 512 dimensions and do not utilize RFF. Higher numbers indicate better performance. It is worth noting that FiGURe512 outperforms or remains competitive with the baselines in all cases. The rightmost column Av. A gain represents the average accuracy % gain of FiGURes12 over the model that row, averaged across the different datasets. If model is labeled as 'OOM' for a dataset, it's excluded from Av. Again calculation. Magenta, Green and Blue represent the 1 2nd and 3rd best performing models, for a particular dataset.
HETEROPLLLC DATASETS
HOSOPHILIC
DATASETS
SQUIBEL
CHAMELON
BOMAN EMPRRE
MINESWEETER
YEAR
CORA
CIVES2ER
TU3MED
OGBN-ABE ABXIV
Av. Ague
DEaPWALK
(1.44)
(0:08)
41.05
5035
None7Vic
(1.59
54:23 (2:30)
12.12.0.90
50000 (0:00)
9:09 (0.09)
(1.19
57.45 (6.44)
71.34
00.20
15.78
DGI
39.63 (1.51)
59.25 (1.25)
47.54 (0.76)
$2.51 (0:47)
40.59 (0.05)
(1.22)
(1.61)
86.57(0.52)
65.58
6.61
MVGRL
54.61 (1.39)
00M
(1.30)
75.02
87.12
GRACE
53.151.100
(0.55)
(0:42
00M
(1.7911551
(2.00
(0.44)
SUGRI
(1.13
58.60 (2.04)
59.40 49)
$2.4000.550
36.06 (0. 19)
(2.00)
(1.50(1.62)
86:00 (0.54)
65.80
8.64
FOURR!!
(1.55
65.66
6461
(0:71)
(0.30(021)
(0.57)
71.25
83.91
RIGURe
48.75 (2.48)
66.03 (2.19)
67.01 (0.56)
35.16 (0.58)
41.94 (0.15)
(1.13)
73.34 (1.91)
83.56 (0.34)
FGGUR
(2.23((1.1))
68:55 (1:87)
70.99 (0.53)
(0.49)
42.36.00.00
87.00
74.77 (2.00)
85 (0.44)
69.99
0.00
Table 2: Comparing node classification with the widely-used GCN, FiGURes12 achieves strong results even without task-specift labels, highlighting its ability to learn quality representations.
SQUIRREL
CHAMELEON
ROMAN-
MINESWBEPER
ARXIV-YEAR
CORA
PUBMED
OGBN-ARXX
GCN
(2.13
61.43 (2.70)
73.69 (0,74)
89.75 (0.52)
46.02 (0.26)
87.36 (0.91)
76.47 (1.34)
41 (0.46)
69.37 (0.00)
FGUURes (1.19)
(0.52)
(0.49)
42.26 20)
74.77 (2.00)
Table 3: Mean epoch time (in seconds) on the two large datasets for different embedding sizes. For lower dimensional embeddings, there a significant speedup.
512 dims
128 dims
32 dims
XRAIV-YEAR
1.24s
0.75s
0.72s
OGBN-ARXIV
0.92s
0.74s
0.72s
We analyzed the results in Table and made important observations. Across homophilic and het- erophilic datasets, FiGURes12 consistently outperforms several SOTA unsupervised models, except in few cases where it achieves comparable performance. Even on the large-scale datasets ARXIV- YEAR and OGBN-ARXIV FiGURes12 performs well, demonstrating the scalability of our method. Two baseline methods MVGRL and GRACE run into memory issues on the larger datasets and are accordingly reported OOM in the table. We want to emphasize the rightmost column of the table, which shows the average percentage gain in performance across all datasets. This metric compares the improvement that FiGURes provides over each baseline model for each dataset and averages these improvements. This metric highlights that FiGURe512 performs consistently well across diverse datasets. No other baseline model achieves the same consistent performance across all datasets as FiGURe512. Even the recent state-of-the- the-terrt contrastive models GRACE and SUGRL experience
7

---

average performance drops of approximately 5% and 10%, respectively. This result indicates that FiGURe512 learns representations that exhibit high generalization and task- aggossiic capabilities. Another important observation is the effectiveness of RFF projections in improving lower dimen- sional representations. We compared FeGURe at different dimensions, including FiGURe and FiGURe RFF corresponding to learning 32 and 128-dimensional embeddings, respectively, in addi- tion to the baseline representation size of 512 dimensions. Remarkably, even at lower dimensions, FiGURe with REF projections demonstrates competitive performance across datasets, surpassing the 512-dimensional baselines in several cases. This result highlights the effectiveness of RFFprojections in enhancing the quality of lower dimensional representations. Using nower-dimensional embeddings reduces the computation time and makes FiGURe faster than the baselines. The computational efficiency of reducing dimension size becomes more significant with larger datasets, as evidenced in Table 3. On AR XIV- YEAR, a large graph with 169, 343 nodes, 128-dimensional embeddings yield a 1.6x speedup, and 32-dimensional embeddings yield 1.7x speedup. Similar results are observed in OGBN-ARXIV. For further insights into the effectiveness of R RFFprojections, see Section 6.2, and for computational efficiency gains, refer Section 6.4. In Table 2, we include GCN as benchmark for comparison. Remarkably, FiGURe512 remains competitive across most datasets, sometimes even surpassing GCN. This highlights that FiGURe512 can capture task-specific information required by downstream tasks, typically handled by GCN, through unsupervised means. When considering a downstream task, using FiGURes12 embeddings allows for the use of computationally efficient models like Logistic Regression, as opposed to training iesource-intessive end-to-end graph neural networks. There are, however, works such as [6]. [13], and [5], that explore methods speed up end-to-end graph neural network training. In summary, FiGURe offers significant computational efficiency advantages over end-to-end supervised graph neural networks. Performance can potentially improve further by incorporating non-linear models like MLP. For detailed comparisons with other su- pervised methods, please refer 8.4. It's noteworthy that both OGBN-ARXIV and ARXIV- YEAR use the ARXIV citation network but differ in label prediction tasks (subject area and publication year, respectively). FiGURe demonstrates improvements in both cases, showcasing its task-agnostic and multi-task capabilities due to its flexible node representation. This flexibility enables diverse tasks to extract the most relevant information (see Section 5.2), resulting in strong overall performance. Note that in 8.5, we have also performed an ablation study where the depth of the encoder is increased.
6.2 RQ2: RFF Projections on Lower Dimensional Representations
Table 4: Node classification accuracy percentages with and without using Random Fourier Feature projections (on 32 dimensions). higher number means better performance. The performance is improved by using RFF in almost all cases, indicating the usefulness of this transformation
RFF
CORA
CITESEER
SQUIRREL
CHAMELEON
DGI
81.65 (1.90)
65.62 (2.39)
31.60 (2.
45.48 (3.02)
81.49 (1.96)
66.50 (2.44)
38.19 (1.52)
56.01 (2.66)
MVGRL
X
81.03 (1.29)
72.38 (1.68)
37.20 (1.22)
49.65 (2.08)
80.48 (1.71)
72.54 (1.89)
39.53 (1.04)
56.73 (2.52)
SUGRL
X
65.35 (2.41)
42.84 (2.57)
31.62 (1.47)
43,20 (1.79)
70.06 (1 (1.24)
47.03 (3.02)
38.50 (2.19)
51.01 (2.26)
GRACE
76.84 (1.09)
58.40 (3.05)
38,20 (1.38)
53.25 (1.58)
79.15 (1.44)
63.66 (2.96)
51.56 (1.39)
67.39 (2.23)
FiGURes2
82.88 (1.42)
70.32 (1.98)
39.38 (1.35)
53.27 (2.40)
82.56 (0.87)
71.25 (2.20)
48.89 (1.55)
65.66 (2.52)
In this section, we analyse the performance of unsupervised basclines using 32-dimensional embed- dings with and without RFF projections (see Table 4). Despite extensive hyperparameter tuning, we could not replicate the results reported by SUGRL, so we present the best results we obtained. Two noteworthy observations emerge from these tables. Firstly, is evident that lower dimensional embeddings can yield meaningful and linearly separable representations when combined with simple RFF projections. Utilising RFF projections enhances performance in almost all cases, highlight- ing the value captured by MI-based methods even with lower-dimensional embeddings. Secondly, Fi consistently achieves superior or comparable performance to the baselines, even in lower dimensions. Notably, this includes SUGRL, purported to excel in such settings. However, there is 2-
8

---

3% performance gap between GRACE and our method for the SQUIRREL and CHAMELEON datasets. While GRACE handles heterophily well at lower dimensions, its performance deteriorates with homophilic graphs, unlike FiGURe RPF which captures lower frequency information effectively. Additionally, our method exhibits computational efficiency advantages for specific datasets in lower dimensions. Please refer to 8.6: for discussions with regards to the RFF algorithm, 8.7: for analysing the RFF behaviour and community structure, 8.8: for experiments using other random projection methods, 8.9 and 8. 8.10: for ablation studies with regards to projecting to higher dimensional spaces via RFF, and 8. I: for issues related to including RFF in training. Overall, these findings highlight the potential of RFF projections in extracting useful information from lower dimensional embeddings and reaffirm the competitiveness of FiGURe over the baselines.
6.3 RQ3: Sharing Weights Across Filter Specific Encoders
Table 5: A comparison of the performance on the downstream node classification task using indepen- dently trained encoders and weight sharing across encoders is shown. The reported metric is accuracy. In both cases, the embeddings are combined using the method described in 5.2
CORA 86.92 (1.10) %
CITESEER
SQUIRREL 50.52 (1.51) %
CHAMELEON
INDEPENDENT
75.03 (1.75) %
66.86 (1.85) %
SHARED
87.00 (1.24) 74.77 (2.00) %
52.23 (1.19) %
68.55 (1.87) %
Our method proposes to reduce the computational load by sharing the encoder weights across all filters. It stands to reason whether sharing these weights causes any degradation in performance. We present the results with shared and independent encoders across the filters in Table 5 to verify this. We hypothesize that, sharing encoder weights embeds diverse filter representations in a common space, improving suitability for combined representation learning. This enhances features for downstream tasks, in some cases boosting performance. Experimental results confirm that shared weights do not significantly reduce performance; sometimes, they even enhance highlighting shared encoders' effectiveness in reducing computational load without sacrificing performance.
6.4 RQ4: Computational Efficiency
Table 6: Mean epoch time (in milliseconds) averaged across 20 trials with different hyperparameters. lower number means the method is faster. Even though our method is slower 512 dimensions. using 128 and 32 dimensional embeddings significantly reduces the mean epoch time. Using RFF as described in 6.2 we are able to prevent the performance drops experienced by DGI and MYGRL.
DGI
MVGRL
FiGURe012
FiGURe
FiGUReN
CORA
38.53 (0.77)
75.29 (0.56)
114.38 (0.51)
20.10 (0.46)
11.54 (0.34)
CITESEER
52.98 (1.15)
102.41 (0.99)
156.24 (0.56)
30.30 (0.60)
17.16 (0.51)
SQUIRREL
87.06 (2.07)
168.24 (2.08)
257.65 (0.76)
47.72 (1.40)
23.52 (1.14)
CHAMELEON
33.08 (0.49)
64.71 (1.05)
98.36 (0.64)
18.56 (0.39)
11.63 (0.48)
To assess the computational efficiency of the different methods, we analyzed the computation time and summarized the results in Table 6. The key metric used in this analysis is the mean epoch time: the average time taken to complete one epoch of training. We compared our method with other MI based methods such as DGI and MVGRL. Due to the increase in the number of augmentation views, there is an expected increase in computation time from DGI to MVGRL to FiGURe. However, as demonstrated in 6.2, using RFF projections allows us to achieve competitive performance even at lower dimensions. Therefore, we also included comparisons with our method at 128 and 32 dimensions in the table. It is evident from the results that our method, both at 128 and 32 dimensions, exhibits faster computation times compared to both DGI and MVGRL, which rely on higher- dimensional representations to achieve good performance. This result indicates that FiGURe is computationally efficient due to its ability to work with lower-dimensional representations. During training, our method, FiGUR RFF REF is ~ 3x faster than DGI and 6x times faster than MVGRL. Despite the faster computation, FiGUR RFF also exhibits an average performance improvement of around 2% across the datasets over all methods considered our experiments. Please refer to 8.12 and 8.13 for further discussions.
32
9

---

6.5 RQ5: Experiments on Other Filter Banks
Table 7: Accuracy percentages for various filter banks in conjunction with FiGURe. Specifically, F BERNNET and Fll BERNNET refer to the F BERNNET filter bank with K set to 3 and 11, respectively. Simi- larly, F CHEBNET and F CHEENET represent the F Femener filter bank with K set to 3 and 11, respectively.
CORA
CITESEER
SQUIRREL
CHAMELEON
F BERNNET
85.13 (1.26)
73.38 (1.81)
37.07 (1.29)
53.95 (2.78)
F FEERNNET
86.62 (1.59)
73.97 (1.43)
43.48 (3.80)
62.13 (3.66)
F CHEANEr
83.84 (1.36)
71.92 (2.29)
40.23 (1.58)
60.61 (2.03)
F CHERNET
76.14 (6.80)
59.89 (8.94)
52.46 (1.10)
67.37 (1.60)
EGPRONN 87.00 (1.24)
74.77 (2.00)
52.23 (1.19)
68.55 (1.87)
To showcase the versatility of our proposed framework, we conducted an experiment using Bernstein and Chebyshev filters, detailed in Table 7. The results indicate that using FGPRGNN leads to better performance than BERNNET and CHEBNET filters. We believe this is happening is due to the latent characteristies of the dataset. [17, 28] have shown that datasets like CHAMELEON and SQUIRREL need frequency response functions that give more prominence to the tail-end spectrum. GPRGNN filters are more amenable these needs, as demonstrated in [28]. However, different datasets may require other frequency response shapes, where BERNNET and CHEBNEY filters may excel, and give better performance. For instance, F BERNNET may better approximate comb filters, as their basis gives uniform prominence to the entire spectrum. Our framework is designed to accommodate any filter bank, catering diverse dataset needs. Further discussions are in 8.14.
6.6 RQ6: Combining information from different filters (in FGPRGNN)
To analyze how FiGURe combines representations from different filters, we present alpha coefficients for the pighest-perfomming split in Table 8, utilizing F FGPRGNN on: CORA, CITESEER, SQUIRREL, and CHAMELEON (coefficients may vary for different splits within the same dataset) [28]. GPRGNN fil- ters adapt well to heterophilic datasets like CHA MELEOE and SQUIRREL, emphasizing spectral tails with sigmificant weightage on the A filter. In contrast, homophilic datasets require low-pass filters, achieved by assigning higher weightage the and A filters. Achieving these filter shapes with other methods like BERNNET and CHEBNET is possible but more challenging for the model to learn. We believe that this is why GPRGNN filters consistently outperform other filter banks on the datasets we ve examined. However, it's important to note that some datasets may benefit from different response shapes, where BERNNET and CHEBNET filters might be more suitable. This table validates our hypothesis about the efficacy of GPRGNN filter coefficients, with A dominating in homophilic datasets (CORA, CITESEER) and A in heterophilic datasets (CHAMELEON SQUIRREL..
Table 8: We present the alpha coefficients obtained from the best-performing split utilizing GPRGNN filters for CORA, CITESEER, SQUIRREL, and CHAMELEON datasets
I
A
A
A
CORA
18.2
0
0
35.95
CITESEER
0
0
0
0.48
SQUIRREL
0
0
15.3
0
CHAMELEON
0
0
8.93
0.1
7 Conclusion and Future Work
Our work demonstrates the benefits of enhancing contrastive learning methods with filter views and learning filter-specifi representations cater to diverse tasks from homophily to heterophily. We have effectively alleviated computational and storage burdens by sharing the encoder across these filters and focusing on cow-dimensional embeddings that utilize high-dimensional projections, technique inspired by random feature maps developed for kernel approximations. Future directions involve expanding the analysis from [2] to graph contrastive learning and investigating linear separability in lower dimensions, which could strengthen the connection the random feature maps approach.
10

---

References
[1] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperpa- rameter optimization framework. In International conference on knowledge discovery & data mining (KDD), pages 2623-2631, 2019.
[2] S. Arora, H. Khandeparkar, M. Khodak, O. Plevrakis, and N. Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In International Conference Machine Learning (ICML), 2019.
[3] K. Atarashi, S. Maji, and S. Oyama. Random feature maps for the itemset kemel. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 3199-3200000 3206, 2019.
[4] D. Bo, X. Wang, C. Shi, and H.-W. Shen. Beyond low-frequency information in graph con- volutional networks. In Association for the Advancement of Artificial Intelligence (AAAI), 2021.
[5] A. Bojchevski, J. Gasteiger, B. Perozzi, A. Kapoor, M. Blais, B. Rózemberczki, M. Lukasik, and S. Günnemann. Scaling graph neural networks with approximate pagerank. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, KDD '20, page 2464-2473, New York, NY, USA, 2020. Association for Computing Machinery.
[6] J. Chen, T. Ma, and C. Xiao. FastGCN: Fast learning with graph convolutional networks via importance sampling. In International Conference on Learning Representations, 2018
[7] E. Chien, J. Peng, P. Li, and O. Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations (ICLR), 2021.
[8] K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins,
J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger, L. J. Colwell, and A. Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021.
[9] M. D. Donsker and S. R. S. Varadhan. Asymptotic evaluation of certain markov process expectations for large time. In Communications on Pure and Applied Mathematics, 1975.
[10] C. Dugas. Y. Bengio, F. Bélisle, C. Nadeau, and R. Garcia. Incorporating second-order functional knowledge for better option pricing. In Neural Information Processing Systems (NeurIPS), 2000.
[11] I. T. Falk, C. Ciliberto. and massimiliano pontil. Implicit kernel meta-leaming using kernel integral forms. In The 38th Conference on Uncertainty in Artificial Intelligence, 2022.
[12] K. Fang, F. Liu, X. Huang, and J. Yang. End-to-end end kernel learning via generative random fourier features. Pattern Recognition, 134:109057, 2023.
[13] Frasca, E, Rossi, D. Eynard, B. Chamberlain, M. Bronstein, and F. Monti. SIGN: Scalable Inception Graph Neural Networks. In ICML Workshop on Graph Representation Learning and Beyond, 2020.
[14] A. Grover and Leskovee. node2vec: Scalable feature learning for networks. In International Conference on Knowledge Discovery and Data Mining (KDD), 2016.
[15] K. Hassani and A. H. Khasahmadi. Contrastive multi-view representation learning on graphs. International Conference on Machine Learning (ICML), 2020.
[16] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human- performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
[17] M. He, Z. Wei, Z. Huang, and H. Xu. Bernnet: Learning arbitrary graph spectral filters via bernstein approximation. In Neural Information Processing Systems (NeuriPS), 2022.
[18] R. D. Hjelm, A. Fedorov, S. Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations (ICLR), 2019.
K. Grewal, P. Bachman, A. Trischler, and
11

---

[19] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Neural Information Processing Systems (NeurIPS), 2020.
[20] P. Kar and H. Karnick. Random feature maps for dot product kernels. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2012.
[21] P. Kar and H. Karnick. Random feature maps for dot product kernels. In N. D. Lawrence and M. A. Girolami, editors, Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2012, La Palma, Canary Islands, Spain, April 21-23, 2012, volume 22 of JMLR Proceedings, pages 583-591. JMLR.org. 2012.
[22] D. Kim and A. Oh. How to find your friendly neighborhood: Graph attention design with self-supervision. In International Conference on Learning Representations (ICLR), 2021.
[23] G. Kimeldorf and G. Wahba. Some results on tchebycheffian spline functions. Journal of Mathematical Analysis and Applications, 1971.
[24] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
[25] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
[26] Z. Li, J.-F. Ton, D. Oglic, and D. Sejdinovic. Towards unified analysis of random fourier features. Journal of Machine Learning Research (JMLR), 2021.
[27] D. Lim, F. M. Hohne, X. Li, S. L. Huang, V. Gupta, O. P. Bhalerao, and S.N.. Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. In Neural Information Processing Systems (NeurIPS), 2021.
[28] V. Lingam, C. Ekbote, M. Sharma, R. Ragesh, A. Iyer, and S. Sellamanickam. A pisce-wise polynomial filtering approach for graph neural networks. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), 2022.
[29] R. Linsker. Self-organization in a perceptual network. Computer, 1988.
[30] J. Mercer. Functions of positive and negative type, and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of Mathematical or Physical Character, 1909.
[31] Y. Mo, L. Peng, Xu, X. Shi, and Zhu. Simple unsupervised graph representation learning. In Association for the Advancement of Artificial Intelligence (AAAI), 2022.
[32] S. Nowozin, B. Cseke, and R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Neurul Information Processing Systems (NeurIPS), 2016.
[33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems (NeurIPS), 2019.
[34] H. Pei, B. Wei, K. C.-C Chang, Y. Lei, and B. Yang. Geom-gen: Geometric graph convolutional networks. In International Conference Learning Representations (ICLR), 2020.
[35] H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. Smith, and L. Kong. Random feature attention. In International Conference on Learning Representations, 2021.
[36] J. Penninntonn, F. X.X.X X. Yu, and S. Kumar. Spherical random features for polynomial kernels. In Neural Information Processing Systems (NeurIPS), 2015.
[37] B. Perozzi, R. Al-Rfou, and S. Skiena. Deep Walk. In International Conference on Knowledge Discovery and Data Mining (KDD), 2014.
12

---

[38] O. Platonov, D. Kuznedelev, M. Diskin, A. Babenko, and L. Prokhorenkova, A critical look at the evaluation of GNNs under heterophily: Are we really making progress? In International Conference Learning Representations (ICIR), 2023.
[39] Z. Qin, W. Sun, H. Deng, D. Li, Y. Wei, B. Lv, J. Yan, L. Kong, and Y. Zhong. cosformer: Rethinking softmax in attention. In International Conference on Learning Representations. 2022.
[40] A. Rahimi and B. Recht. Random features for large-scale kernel machines. In Neural Informa- lion Processing Systems (NeurIPS), 2007.
[41] A. Rahimi and B. Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Neural Information Processing Systems (NeurIPS), 2008.
[42] D. I. Shuman, K. Narang, Frossard, A. Ortega, and P. Vandergheynst. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE Signal Processing Magazine, 2013.
[43] N. Tremblay, P. Gonçalves, and P. Borgnat. Design of graph filters and filterbanks. Cooperative and Graph Signal Processing, 2017.
[44] Veličković, W. Fedus, W. L. Hamilton, Liò, Y. Bengio, and D. Hjelm. Deep graph infomax. In International Conference on Learning Representations (ICLR), 2019.
[45] Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang. Deep Graph Contrastive Representation Learning. In /CML Workshop on Graph Representation Learning and Beyond, 2020.
13

---

8
Supplementary Material
Contents
Introduction
2 Related Work
3 Problem Setting
Preliminaries
4.1 Filter Banks
4.2
Random Feature Maps for Kernel Approximations
5
Proposed Approach
5.1
Unsupervised Representation Learning
5.2
Supervised Representation Learning
6
Experimental Results
6.1
RQI: FiGURe versus SOTA Methods
6.2
RQ2: RFF Projections on Lower Dimensional Representations
6.3
RQ3: Sharing Weights Across Filter Specific Encoders
6.4
RQ4: Computational Efficiency
6.5
RQ5: Experiments on Other Filter Banks
10
6.6
RQ6: Combining information from different filters (in FGRRGNN)
10
7
Conclusion and Future Work
10
8
Supplementary Material
14
8.1
Reproducibility
8.2
Datasets
8.3
Training Details
16
8.3.1
Unsupervised Training
16
8.3.2
Supervised Training
16
8.3.3
Negative Sampling for the Identity Filter
17
8.4
Comparison with other Supervised Methods
17
8.5
Increasing the depth of the Encoder
17
8.6
RFF Projections
18
8.7
Visualising RFF Behavior and Community Structure
18
8.8
Other Random Projections
19
8.9
RFF ablation study at 128 dimensions
20
8.10
RFF ablation study at 512 dimensions
20
8.11
Issues with including RFF in training
20
14

---

8.12
Computational Comparisons with other Other Unsupervised Methods
21
8.13
Time and storage cost
8.14
Choice of Filter Banks
22
8.1
Reproducibility
We strive to ensure the reproducibility of our research findings. To facilitate this, we provide the details of our experimental setup, including dataset sources, preprocessing steps, hyperparameters, and model configurations. We also make our code and the datasets used, publicly available at https://gi thubb... thub. com/microsoff figur, enabling researchers reproduce our results and
build upon our work. We would like to emphasize that our code is built on top of the existing MVGRL codebase. For the datasets used our evaluation, we provide references to their original sources and any specific data splits that we employed. This allows others obtain the same datasets and perform their own analyses using consistent data. Additionally, we specify the versions of libraries and frameworks used in experiments, in Section 8.3, and in the REQUIREMENTS file and the README file, in the codebase, enabling others to set up a compatible environment. We document any specific seed values randomization procedures that may affect the results. By providing these details and resources, we aim to promote transparency and reproducibility in scientific research. We encourage fellow researchers reach out if they have any questions need further clarification on our methods or results.
8.2 Datasets
Homophilic Datasets: We evaluated model (as well as baselines) three homophilic datasets: CORA, CITESEER, and PUBMED as borrowed from [34]. All three are citation networks, where each node represents research paper and the links represent citations. Pubmed consists of medical research papers. The task to predict the category of the research paper. We follow the same dataset setup mentioned in [34] to create 10 random splits for each of these datasets.
Heterophilic Datasets: our evaluation, we included four heterophilic datasets: CHAMELEON. SQUIRREL, ROMAN-EMPIRE and MINESWEEPER. For CHIAMELENN and SQUIRREL, nodes represent Wikipedia web pages and edges capture mutual links between pages. We utilized the ten random splits provided in [34], where 48%, 32%, and 20% of the nodes were allocated for the train, validation. and test sets, respectively. In ROMAN-EMPIRE EMPIRE each node corresponds to a word the Roman Empire Wikipedia article. Two words connected with an edge if either these words follow each other in the text, or they are connected the dependency tree of the sentence. The syntactic role of the word/node defines its class label. The MINESWEEPHS graph is regular 100x 100 grid where each node is connected to eight neighboring nodes, and the features are on-hot encoded representations of the number of neighboring mines. The task to predict which nodes are mines. For both ROMAN-EMPIRE and MINESWEEPER, we used the ten random splits provided in [38].
Large Datasets: We also evaluate method on two large datasets OGBN-ARXIV (from [19]) and ARXIV- YEAR (from [27]). Both these datasets are from the arxiv citation network. In OGBN- ARXIV, the task is predict the category of the research paper, and in ARXIV- YEAR the task is to predict the year of publication. We use the publicly available splits for OGBN-ARXII [22] and follow the same dataset setup mentioned in [27] generate 5 random splits for ARXIV- YEAR. Note that OGBN-ARXIV a homophilic dataset while ARXIV- YEAR is heterophilic datasets.
The detailed dataset statistics can be found in Table 9.
Table Dataset Statistics. The table provides information on the following dataset characteristics: number of nodes, number of edges, feature dimension, number of classes, as well as the count of nodes used for training, validation, and testing.
HEVEROPHILIC DATASETS
HOMOPHILIC DATASETS
PROPERTIES
SQUIRREL
CHAMELEON
ROXAN-EMPIRE
MINESWEEPER
ARXIV-YEAR
OGBN-ARXIV
CITESEER
PUBMED
CORA
WNODES
X201
2271
27762
(000)
169343
169343
3327
19717
2708
222134
1335586
12431
108365
13264
3703
#CLASSES
5
40
6
#TRAIN
2496
1092
[133]
5000
84671
909:1
1596
9463
1192
#VAL
1664
729
5665
2500
42335
24799
1065
6310
796
#TEST
1841
45
3666
2500
42537
48603
666
3044
15

---

8.3 Training Details
We conducted all experiments on a machine equipped with an Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz processor, 440GB RAM, and a Tesla-P100 GPU with 16GB memory. The experiments were executed using Python 3.9.12 12 and PyTorch 1.13.0 [33]. To optimize the hyperparameter search, we employed Optuna [1]. We utilized the Adam optimizer [24] for the optimization process.
8.3.1 Unsupervised Training
We conducted hyperparameter tuning for all unsupervised methods using 20 Optuna trials. The hyperparameter ranges and settings for each method are follows:
DEEPWALK: We set the learning rate to 0.01, number of epochs to and the varied the random walk length over {8, 9, 10, 11, 12]. Additionally, we varied the context window size over {3,4, 4,54 and the negative size (number of negative samples per positive sample) over (4,5,6].
NODE2 For Node2 Vec, we set the learning rate to 0.01 and number of epochs to 100. We varied the number of walks over {5, 10, 15) and the walk length over (40, 50, 60). The p (return parameter) value was chosen from {0.1, 0.25, 0.5, 1 and y (in-out parameter) value was chosen from {3, 5,J.
DGI: DGI [44] proposes self-supervised learning framework for graph representation learning by maximizing the mutual information between local and global structural context of nodes, enabling unsupervised feature extraction in graph neural networks. We relied on the authors' code? and the prescribed hyperparameter ranges specific to the DGI model, for our experiments.
MVGRL: MVGRL [15] proposes a method for learning unsupervised node representations by leveraging two views of the graph data, the graph diffusion view and adjacency graph view. We relied on the authors' code? and the prescribed hyperparameter ranges specific to the MVGRL model, for our experiments.
GRACE: GRACE [45] proposes a technique where two different perspectives of the graph are created through corruption, and the learning process involves maximizing the consistency between the node representations obtained from these two views. We relied on the authors' code* and the prescribed hyperparameter ranges specific to the GRACE model, for our experiments.
SUGRL [31] proposes a technique for learning unsupervised representations which capture node proximity, while also utilising node feature information. We relied on the authors' code and the prescribed hyperparameter ranges specific to the SUGRL model, for our experiments.
SUGRL:
FiGURe: We followed the setting of the MVGRL model, setting the batch size to 2 and number of GCN layers to I. We further tuned the learning rate over {0.00001, 0.0001, 0.001, 0.01, 0.1} and the sample size (number of nodes selected per batch) over (1500, 1750, 2000, 2250). except for the large graphs, for which we set the sample size to 5,000.
We maintained consistency across all methods by employing identical parameters for maximum epochs and early stopping criteria. Specifically, we set the maximum number epochs to 30, 000 and utilized an early stopping patience of 20 epochs, with the exception of large datasets, where we extended the patience to 500 epochs.
In each case, we selected the hyperparameters that resulted in the lowest unsupervised training loss.
8.3.2 Supervised Training
For all unsupervised methods, including the baselines and our method, we perform post-training supervised evaluation using logistic regression with 60 Optuna trials. We set the maximum number of epochs to 10000 and select the epoch and hyperparameters that yield the best validation accuracy. The learning rate swept over the range {0.00001, 0.0001, 0.001, 0.0015, 0.01, 0.015, 0.1, 0.5, 1, 2). 10 0.5.5, 1,33.
and the weight decay is varied over {10 10 10 3, 10
http://thuub.com/etaaa////t git git
http::://gitaab..co////ttttt//// git
16

---

FiGURe: Along with the hyperparameters described above. following the approach described in [17], we also tune the combination coefficients (a; 's) with separate learning rate. This separate learning rate is swept over the range {0.00001, 0.0001, 0.001, 0.00015 0.01, 0.015, 0.1, 0.5, 1, 1, In addition, we have a coefficient for masking the incoming embeddings from each filter, which varied between O and 1. Furthermore, these coefficients are passed through an activation layer, and we have two options: 'none" and 'exp'. When 'none' selected, the coefficients are used directly, while 'exp' indicates that they are passed through an exponential function before being used.
FiGURe with RFF: For the experiments involving Random Fourier Features (RFF), we use the same hyperparameter ranges as mentioned above. However, we also tune the gamma pa- rameter which is specific RFF projections. The gamma parameter is tuned within the range {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.2)
8.3.3 Negative Sampling for the Identity Filter
In implementation of F FGRRNN FEERNNET. follow specifie procedure for handling the filters during training and evaluation. For all filters except the identity filter (I), we employ the negative sampling approach described in Section 6. However, the identity filter is treated differently. During training, we exclude the identity filter and only include it during evaluation.
During negative sampling, the generation of the negative anchor involves shuffling the node features, followed by premultiplying the shuffled node feature matrix with the filter matrix and computing the mean. On the other hand, for the positive anchor, the same procedure is applied without shuffling the node features. This approach encourages the model to learn meaningful patterns and relationships in the data when the filter matrix is not the identity matrix.
The decision to exclude the identity filter during training is based on the observation that it presents a special case where the positive and negative anchors become the same. As result, the model would optimize and minimize the same quantity, potentially leading to trivial solutions. To prevent this, we exclude the identity filter during training.
By excluding the identity filter during training, we ensure that the model focuses on the other filters in F FGRRGN or F BErNNEr to capture and leverage the diverse information present in the graph. Including the identity filter only during evaluation allows us to evaluate its contribution the final performance of the model. This approach helps prevent the model from learning trivial solutions and ensures that it learns meaningful representations by leveraging the other filters.
8.4 Comparison with other Supervised Methods
Table 10 presents a comparison with common supervised baselines. Specifically, we choose models for comparison, representing three different kinds of supervised methods, standard aggregation models (GCN), spectral filter-based models (GPRGNN) and smart-aggregattion models (H₂GCN). There are two key observations from this table. Firstly, FiGURe,12 is competitive with the supervised basclines, lagging behind only by a few percentage points in some cases. This suggests that much of the information that is required by the downstream tasks, captured by the supervised models, can be made available through unsupervised methods like FiGURe which uses filter banks. is important to note that in FiGURe we only utilize logistic regression while evaluating the downstream task. This is much more efficient that training a graph neural network end to end. Additionally it is possible that further gains may be obtained by utilizing non-linear model like an MLP. Furthermore, as indicated by 10, we can gain further computational efficiency by utilizing lower dimensional representations like 32 and 128 (with RFF), and still not compromise significantly on the performance. Overall FiGURe manages to remain competitive despite not having access to tpec-fpc labels and is computationally efficient as well.
8.5 Increasing the depth of the Encoder
We present an analysis Table 11 featuring an increased number of encoder layers. In this context, "encoder layers" can be interpreted in two ways: firstly, as deeper GCN which entails aggregating information from multiple-hop neighborhoods into the node; and secondly, as single-hop GCN with more extensive network for feature transformation. We provide performance results for both scenar-
17

---

Table 10: Contains node classification accuracy percentages on heterophilic and homophilic datasets.
GCN, GPRGNN and H₂GCN are supervised methods. FiGURe RFF and FiGURe RFF refer to FiGURe trained with 32 and 128 dimensional representations, respectively, and then projected using RFF. The remaining models are trained at 512 dimensions. Higher numbers indicate better performance.
HETEROPHLLIC DAFASETS
HOMOPHILIC DATASETS
SQUIRREL
CHAMELEON
ROMAN EMPIRE
MINESWEEPER
ARXIV YEAR
OGDN ARXIV
CITESEER
PUBMED
GCN
(2.15
(1.52
73.69 (0.74)
39.75 (0.52)
40,02 (0.26)
09.37 (0.00)
$7.30 (0.93)
70.47 (1.34)
88.41 (0.46)
GPRGNN
62.59 (2.04)
64.83
68
87.77(1.33)
89.06
H&GUN
(2.90)
58,40 (2.77)
60.11 (0.52)
39.71 (0.31)
29099 (0.10)
GOM
(7.35
77.07 (1,64)
89.59 (0.33)
FIGUR
65.66 (1.59)
67.67 (0.77)
(5.28(00.71)
41.30
66.58
$2.56 (0.87)
84.18
FIGURAN
48.78 (2.48)
66.03 (2.19)
68. 10 09)
85.16 (0.55)
41 41.94 (0. 15)
691 11 (0.00)
86, (1.14
73.34 (1.91)
85.41 52)
FIGURe
68
70:990.52)
85.58
42.26
(1.00)
ios, encompassing two and three layers each. noteworthy that the single-layer GCN achieves equal or superior performance compared to all other configurations.
Table 11:
Analyzing the impact of altering the number of encoder layers on downstream accuracy. it evident that all instances, the single-layer GCN outperforms the others.
CORA
CITESEER
SQUIRREL
CHAMELEON
Layer GCN Layer GCN
87.00 (1.24)
74.77 (2.00)
52.23 (1.19)
68.55 (1.87)
86.62 (1.43)
73.62 (1.46)
43.80 (1.57)
53.53 (2.13)
Layer GCN
84.40 (1.84)
72.52 (2.09)
42.79 (1.12)
61.73 (2.25)
GCN+ 2 Layer MLP
85.73 (1.03)
70.21 (2.30)
49.91 (2.68)
68.18 (1.76)
GCN+ 3 Layer MLP
84.99 (1.43)
71.39 (2.32)
45.85 (3.33)
64.19 (1.43)
8.6 RFF Projections
As shown in Section 6.2 and in Section 6.4, RFF projections are a computationally efficient way to achieve training by preserving the latent class behavior present in lower dimensional embeddings, by projecting them into higher dimensional linearly separable space. The natural question that comes up is how do we compute these RFF projections? We provide an algorithm to compute the RFF projections in this section, in algorithm 1. Note that this follows [40].
Algorithm 1 Random Fourier Feature Computation
Require: Input data X E IR xxd target dimension D, kernel bandwidth y Random Fourier Features Z E R
Ensure:
1: Initialize random weight matrix W E RRdx with Gaussian distribution
2: Initialize random bias vector E R.D uniformly from [0, 2m)
3: Compute scaled input X' = 7 +
4: Compute random Fourier features Z = V cos
5: return Z
8.7 Visualising RFF Behavior and Community Structure
As shown in prior sections, FiGURe improves on both computational efficiency well perfor- mance by utilising RFF projections. In this section, we aim to gain insights into the behavior of RFF projections and comprehend their underlying operations through series of simple visualizations.
Figure 3 offers insights into the structure of the embeddings for the CORA dataset across different dimensions. Remarkably, even at lower dimensions (e.g., 32 dimensions), clear class structures are discernible, indicating that the embeddings capture meaningful information related to the class labels. Furthermore, when employing RFF to project the embeddings into higher dimensions, these distinct class structures are still preserved. This suggests that the role of RFF not to introduce new information, but rather enhance the suitability of diwer-dimensional embeddings
t-SNE Plots:
18

---

for linear classifiers while maintaining the underlying class-related information. Notably, even at 512 dimensions, the class structures remain distinguishable. However, worth noting that the class-specifie embeddings appear to be more tightly clustered and less dispersed compared to the 32-dimensional embeddings or the projected 52-dimensional embeddings. This suggests that learning a 512-dimensional embedding differs inherently from learning a 32-dimensional embedding and subsequently projecting it into higher dimensions.
1992 car Deri 22
SHO ana Diee
19RC crel Den 512
Ori
Daark
Dhirt
(a) FiGURes2 (without RFF) the CORA dataset
(b) FiGURE (with RFF) on the CORA dataset
(c) FiGURes12 (without RFF) on the CORA dataset
Figure 3: The figures present t-SNE plots for the CORA dataset. These plots showcase the embeddings generated by the F3 filter, which corresponds to A in the case of FiGURe. The t-SNE plots are generated at different embedding dimensions, providing insights into the distribution and clustering of the embeddings for each dataset.
Figure 4 offers insights into the correlation patterns within the embeddings gener- ated from the SQUIRREI. dataset across different dimensions. In lower dimensions, the embeddings exhibit high correlation with each other, which can be attributed the presence of mixture of topics or latent classes within the dataset. However, when the embeddings are projected to higher dimensions using RFF, the correlation is reduced, and block diagonal matrix emerges. This block diagonal structure indicates the presence of distinct classes or communities within the dataset. Even at 512 dimensions, more refined block diagonal structure can be observed compared to the correlation matrix of the 32-dimensional embeddings. Furthermore, it is noteworthy that the correlation of the projected embeddings can be regarded as a sparser version of the correlation observed in the 512-dimensional embeddings.
Correlation Plots:
Halla Irguin 1C
(2p
CANMAAM MUS IA VE (SVN 312
130
20
305
100
20
300
2013
(a) FiGUReş2 (without RFF) on the SQUIRREL dataset
(b) FIGURe RFF (with RFF) on the SQUIRREL dataset
(c) FiGURes12 (without RFF) on the SQUIRREL dataset
The figures display the normalized correlation plots for the SQUIRREL dataset. These plots illustrate the normalized correlation values between embeddings generated by the F3 filter. In the case of FiGURe, this filter corresponds to the square of the adjacency matrix (A²). The normalized correlation provides a measure of similarity or agreement between the embeddings obtained using the F3 filter for different embedding dimensions. These plots can help analyze the consistency or variation of embeddings across different dimensions and datasets.
Figure 4:
8.8 Other Random Projections
We conducted an analysis exploring the impact of utilizing random projections beyond RFF in Table 12. Our findings reveal that RFF outperforms alternative random projection methods.
19

---

Table 12: Accuracy (%) comparison different random projections with RFF. Notably, RFF consis- tently outperforms the other random projection methods.
Features
CORA
CITESEER
SQUIRREL
CHAMELEON
Polynomial (d=2) [21]
81.35 (2.11)
69.81 (2.14)
38,57 (1.56)
53.88 (1.94)
Polynomial (d=10) [21]
80.44 (1.56)
68.71 (1.67)
38.08 (1.10)
55.20 (1.97)
Exp [21]
80.42 (2.14)
68.81 (1.34)
38.06 (1.61)
55,50 (2.11)
ANOVA [3]
83.26 (0.78)
70.09 (2.44)
40.77 (1.46)
56.01 (1.86)
RFF
87.00 (1.24)
74.77 (2.00)
52.23 (1.19)
68.55 (1.87)
8.9 RFF ablation study at 128 dimensions
We replicate the analysis conducted in 6.2 for 128 dimensions, as outlined in Table 13. It is apparent that our approach consistently outperforms or matches most other methods when operating at 32 dimensions across various datasets. This trend persists even after projecting the learned embeddings to higher dimensions through RFF. Our experiment reaffirms the observations detailed in 6.2.
Table 13: Node classification accuracy percentages with and without using Random Fourier Feature projections (on 128 dimensions). A higher number means better performance. The performance is improved by using RFF in almost all cases, indicating the usefulness of this transformation.
RFF
CORA
CITESEER
SQUIRREL
CHAMELEON
DGI
X
84.99 (1.36)
72.22 (2.50)
34.22 (1.47)
49.82 (2.96)
V
84.17 (2.11)
72.65 (1.52)
37.97 (1.41)
$7.72 (2.03)
MVGRL
X
85.31 (1.66)
73.42 (1.63)
36.92 (1.04)
55.20 (1.70)
V
84.61 (1.74)
72.81 (2.13)
38.73 (1.22)
57.81 (1.80)
SUGRL
X
71.49 (1.15)
63.85 (2.27)
38.04 (1.17)
53.03 (1.73)
V
71.40 (1.40)
63.06 (2.22)
43.24 (1.63)
57.04 (1.78)
GRACE
X
80.87 (1.49)
62.52 (3.57)
41.25 (1.32)
63.14 (1.89)
79.70 (1.91)
64.47 (2.12)
52.29 (1.81)
68.90 (2.05)
FiGURe126
X
84.73 (1.13)
73.07 (1.13)
41.06 (1.51)
59.08 (3.36)
86.14 (1.13)
73.34 (1.91)
48.78 (2.48)
66.03 (2.19)
8.10
RFF ablation study at 512 dimensions
We present the results obtained by applying RFF to 512 dimensions, as shown in Table 14. It is evident that there is minimal performance improvement when using RFF on embeddings that are already of sufficient dimensionality.
Table 14: FiGURe 512 dimensions
CORA
CITESEER
SQUIRREL
CHAMELEON
RFF@5IL
86.84 (0.98)
74.40 (1.30)
51.86 (1.87)
68.60 (1.57)
8.11 Issues with including RFF in training
In the context of incorporating Random Feature techniques like RFF into machine learning models, some challenges have arisen the training process. In the Transformer attention module, these features can exhibit large variances in zero kernel score regions, leading to unstable training [8]. Additionally, increasing the sampling rate in methods like Performer and Random Feature Attention (RFA) [35] can also introduce instability, as highlighted in [39]. Another area where the use of random features with deep networks has been explored is kernel learning. There are of course, computational and convergence challenges that have been observed with these methods as well [11][12]. FiGURe avoids these problems by excluding the random feature generation from training altogether. Parameters like gamma, involved in the random feature construction, are treated as hyperparameters and search is performed over them. Kernel learning, however, an important area of research, and integrating RFF in the training loop would be an interesting extension of this work.
20

---

8.12 Computational Comparisons with other Other Unsupervised Methods
Table 15: Mean epoch time (in milliseconds) averaged across 20 trials with different hyperparameters. A lower number means the method is faster. Even though our method is slower 512 dimensions, using 128 and 32 dimensional embeddings sigmificantlyy reduces the mean epoch time. Using RFF as described in 6.2 we able to prevent the performance drops experienced by SUGRL and GRACE
SUGRL
GRACE
FiGURes12
FiGURef2s
FiGURe
CORA
15.92 (4.10)
51.19 (6.8)
114.38 (0.51)
20.10 (0.46)
11.54 (0.34)
CITESEER
24.37 (4.92)
77.16 (7.2)
156.24 (0.56)
30.30 (0.60)
17.16 (0.51)
SQUIRREL
33.63 (6.94)
355.2 (67.34)
257.65 (0.76)
47.72 (1.40)
23.52 (1.14)
CHAMELEON
16.91 (5.90)
85.05 (14.1)
98.36 (0.64)
18.56 (0.39)
11.63 (0.48)
In Section 6.4, we compared the computational time of FiGURe with MVGRL and DGI, as all three methods fall under the category of unsupervised methods that preform contrastive learning with representations of the entire graph. However, there is another class of methods, such as SUGRL and GRACE, that contrast against other nodes without the need for graph representation computation. Consequently, these methods exhibit higher computational efficiency. Hence, as show in Table 15 upon initial inspection, it appears that SUGRL (at 512 dimensions) exhibits the highest computational efficiency, even outperforming FiGURE efficiency, the significant drop in performance across datasets (as discussed in Section 6.1) renders it less favorable for consideration. In fact, FiGURe RPF offers computational cost savings compared to SUGRL, while also achieving significantly better downstream classification accuracy. Turning to GRACE, it demonstrates greater computational efficiency than FiGURe512 for low to medium-sized graphs. However, as the graph size increases, due random node feature level masking and edge level masking, the computational requirements of GRACE substantially increase (as evidenced by the results on SQUIRREL.. Therefore, for larger graphs with more than approximately 5,000 nodes, FiGURe proves to be more computationally efficient than GRACE (even at 512 dimensions). Furthermore, considering the performance improvements exhibited by FiGURe, it is evident that FiGURe (combined with RFF projections) emerges as the preferred method for unsupervised contrastive learning in graph data.
RFF However, despite its computational 128
32
8.13 Time and storage cost
In Table 16, present comparison of training time and storage cost between our method and DGI. To ensure fairness in the evaluation, we maintained consistent settings across all baselines. The maximum number of epochs was uniformly set to 30,000, with an early stopping patience of 20 epochs, except for larger datasets where we extended the patience to 500 epochs. It's important to note that due varying hyperparameter tuning among the baselines, normalizing the number of epochs posed some challenges. Nevertheless, we ensured that a sufficiently high number of epochs were employed to facilitate convergence. Our approach builds upon DGI and exhibits linear scaling of training time with the number of filters used. Currently, our model employs three filters for training, resulting training time three times longer than that of DGI. Specifically for OGBN-ARXIV, we provide information on the number of epochs, mean epoch time, and total training time. Additionally, we report the storage cost of representations generated by these methods. Please keep in mind that the linear relationship in training time between DGI and FiGURe is not apparent here, as it factors in considerations such as batching and sampling time.
Table 16: We provide the number of epochs, mean epoch time, and total training time for OGBN- ARXIV. We also report the storage cost of the representations from these methods.
Model Dims
Num Epochs
Mean Epoch Time
Total time
Storage
DGI / 512
3945
0.77s
50.75mins
330.75MB
FiGURe/ / 512
4801
0.92s
73.62mins
1.32GB
FiGURe/ / 128
4180
0.74s
51.55mins
330.75MB
FiGURe/ 32
3863
0.72s
46.3 46.36mins
82.69MB
21

---

8.14 Choice of Filter Banks
In Section 4.1, we explore the flexibility of FeGURe to accommodate various filter banks. When making choice, it is crucial to examine the intrinsic properties of the filters contained within different filter banks. We pick two filter banks F BERNNEr and F GPRGNN and provide an overview of the filters contained in the filter banks. We use these two filter banks as examples to illustrate what should one be looking for, while choosing filter bank.
Bernstein Polynomials: Figure 5 illustrates that as the number of Bernstein Basis increases, the focus on different parts of the eigenspectrum also undergoes changes. With an increase in polynomial order, two notable effects can be observed. Firstly, the number of filters increases, enabling each filter focus on more fine-grained eigenvalues. This expanded set of polynomial filters allows for more detailed examination of the eigenspectrum. Secondly, if we examine the first and last Bernstein polynomials, we observe an outward shift in their shape. This shift results the enhancement of specific fine-grained part at the ends of the spectrum. These observations demonstrate that Bernstein polynomials offer the capability to selectively target and enhance specifie regions of interest within the eigenspectrum
Standard Basis: Figure 5 reveals two key observations. Firstly, polynomial order of 2, the standard basis exhibit focus at the ends of the spectrum, in contrast to the behavior of Bernstein polynomials, which tend concentrate more on the middle of the eigenspectrum. This discrepancy highlights the distinct characteristics and emphasis of different polynomial bases in capturing different parts of the eigenspectrum. Secondly, as the number of polynomials increases (in contrast to Bernstein polynomials), the lower order polynomials remain relatively unchanged. Instead, additional polynomials are introduced, offering more fine- ann-ggrined focus at the ends of the spectrum. This expansion of polynomials allows for a more detailed exploration of specific regions of interest within the the ends of eigenspectrum.
In the context of filter banks, previous studies [28, 7] have demonstrated that certain datasets, such as SQUIRREL and CHIAMELEN,, benefit from frequency response functions that enhance the tail ends of the eigenspectrum. This observation suggests that the standard basis, which naturally focuses on the ends of the spectrum, may outperform Bernstein basis functions at lower orders. However, as the order of the Bernstein basis increases, as discussed in 4.1, there is a notable improvement in performance. This can be attributed to the increased focus of Bernstein basis functions on specific regions, particularly the ends of the spectrum. As a result, higher-order Bernstein filters exhibit enhanced capability in capturing important information in those regions. is worth noting that the choice between EGPRGNN and F BERNNET depends on the specific requirements of the downstream task. the task necessitates a stronger focus on the middle of the spectrum or requires a band-pass or comb-like frequency response, F BERNNET is likely to outperform F FGPRGNN. Thus, the selection of the appropriate filter bank should be based on the desired emphasis on different parts of the eigenspectrum. Regarding the performance comparison between F RErNNET and F FGPRGNN. it is plausible that as we increase the order of the Bernstein basis, the performance could potentially match that of F FGRRNNN. However, further investigation and experimentation required to determine the specific conditions and orders at which this convergence in performance occurs.
22

---

Demsen Polynomets
(a) Five Bernstein Basis
(b) Five Standard Basis
Demssten Potyaamets
Standars Noyyomias
Eporatas
(c) Seven Bernstein Basis
(d) Seven Standard Basis
Borssien relynamebs
Standors Naymmmis
Ny
Eponatao
(c) Nine Bernstein Basis
(f) Nine Standard Basis
Bersasen Polmurncb
Standars Naymomial
(g) Eleven Bernstein Basis
(h) Eleven Standard Basis
Figure 5: The figures contain the Bernstein basis as well as standard basis for different degrees. The x-axis of the figures represents the eigenvalues of the Laplacian matrix, while the y-axis represents the magnitude of the polynomials. It is important to note that while plotting the standard polynomials, they are computed with respect to the Laplacian matrix (L,) rather than the adjacency matrix. As result, the eigenvalues lie between [0, 2]. On the other hand, the Bernstein polynomials are typically defined for the normalised Laplacian matrix, and therefore there is no change in the eigenvalue range (the eigenvalues of the normalised Laplacian matrix typically range from 0 to 2). By using the Laplacian matrix as the basis for plotting the polynomials, we can observe the behavior and magnitude of the polynomials at different eigenvalues, providing insights into their spectral properties and frequency response characteristics.
23


Reviews:
[{"review_id": "bE2vW9soF1", "reviewer": "Reviewer_qpib", "summary": "The paper proposes a few approaches to improve the contrastive learning framework of the unsupervised graph representation learning (UGRL) problem. Building on top of the prior works in supervised GRL and UGRL areas (e.g., filter bank construction, etc.), the authors argued that 1) filter-based augmentations (essentially treating filter banks as \"additional views\" in contrastive learning schemes) is able to provide useful representations across cases that require high-freq and low-freq components; and 2) we can leverage a lot of techniques long known in the ML/DL community like RFF to efficiently reduce the computational complexity of these latent representations in UGRL.", "questions": "1. What is the effect of the number of encoder layers used in this case? The authors used one (as was in DGI), but there have also been works that used more.\n2. Could the authors expand Table 5 to include RFF + {other baseline methods}?\n3. Are the numbers provided in Table 1 and 2 the results of the authors' reproduction, or the original papers? Some of these numbers were generally different from what was reported in the original & other papers (e.g., MVGRL, GRACE).\n4. Could the authors elaborate more on the corruption function $C$ (especially how it creates $\\tilde{F}$)?\n\n---------\n\nUPDATE: I have read the rebuttals from the authors and appreciate the additional experiments & explanations. As the authors acknowledge, the benefit of RFF is not unique to FiGUREe (which some other reviewers also pointed out), and I think it should be elaborated on and compared to more clearly in the paper's revision (so is the novelty issue). I'd like to maintain the current rating, but I think the paper's writing can be improved more on these axis to avoid confusion.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper proposed a relatively novel approach of leveraging different filter banks as additional views in a contrastive graph representation learning problem. The method basically builds upon the MI maximization scheme proposed in methods like DGI but extends the approach to also include filter bank $F=\\{F_1, \\dots, F_k\\}$ such that each filter's encoder output's MI with the input data is maximized. The empirical analysis reveals that the approach is superior (and likely compatible with?) to the previous UGRL approaches that didn't use filter-bank-related augmentations; and that RFF provides a reasonable boost to low-dimensionality representation learning in graphs. Specifically:\n\n1. The paper provides a reasonable empirical analysis of the approach on a diverse set of heterophilic and homophilic datasets, demonstrating improvements in both settings. \n2. Numerous ablative studies were made, including on low and high dimensionalities, encoder weight sharing, efficiency, and filter bank selection.\n3. The idea of including multiple filters in the contrastive learning framework itself makes sense to me and is a novel addition to the UGRL literature.", "weaknesses": "A few weaknesses of the paper:\n\n1. The main idea of the approach (maximizing mutual information of encoder output and input; applying discriminator to the patch-readout pair), which is described in Sec. 5.1, is still mainly built up on the discussion of the DGI paper. Although the inclusion of the filter bank discussion is new (which follows from other recent work like BernNet, GPRGNN, and ChebNet), this still limits the novelty of the approach itself.\n2. The contribution of RFF projections, as shown in Sec. 6.2, is largely orthogonal to the FiGURe approach itself. I wonder whether it's the best idea to include it in this paper whose title is on FiGURe and filter augmentations. For example, the efficiency analysis in Sec. 6.4 clearly is lacking the comparison to DGI/MVGRL + RFF, and is thus unfair. Moreover, as the authors suggested in the supplementary materials, the RFF projection they used was based on a Gaussian kernel. Whether and how different projections (e.g., Laplacian) make a difference is unknown; the gradient instability of using RFFs in the architecture (which is an important problem in RFFs' usage in Transformers) is not discussed, etc. I personally feel there is a lot of value to the analysis of RFF itself.\n3. I'm less certain whether \"shared encoder weight\" is a contribution (as listed on line 48) of this paper. It seems more like a remedy to the computational burden added by the FiGURe approach, but is not required otherwise?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we would like to extend our sincere gratitude for your comprehensive and constructive feedback on our paper. We appreciate your recognition of the strengths in our work. Specifically, we are pleased that you found value in our novel approach of leveraging different filter banks as additional views in the contrastive graph representation learning problem. Your acknowledgment of the benefits of this method, including its empirical analysis across a diverse set of heterophilic and homophilic datasets, and its potential novelty addition to the literature is encouraging.\n\n# Comment Response\n1. **Novelty:** We acknowledge that our approach in Section 5.1 is largely built upon the foundational ideas from the DGI paper and that prior works have incorporated various filter banks. However, prior filter bank methods jointly learn the filter specific representations and the combination coefficients. The novel insight that we are proposing in this paper is that we can separately learn these two parts and **bridge the gap with supervised methods**. This enabled us to **propose a simple extension** to DGI which easily incorporates these filter banks. Extending other methods to use filter banks is non-trivial and we plan to explore these in our future work. We will make these aspects more clear in our paper.\n2. **Inclusion of RFF:** Our goal was to ensure that augmenting these additional filter banks does not make the approach impractical by significantly increasing the training time. In Section 6.4, we wished to demonstrate that with RFF we are able to make the training time practical enough. Your feedback on the inclusion of RFF projections and its relation to the FiGURe approach is noted. We realize the potential for separate deep dives into the contributions of RFF and FiGURe and will consider refining our content accordingly. Also, here is a table with performance numbers with few different projections.\n| Features                    | Cora         | Citeseer     | Squirrel     | Chameleon    |\n|-----------------------------|--------------|--------------|--------------|--------------|\n| Polynomial (d=2) [A]  | 81.35 (2.11) | 69.81 (2.14) | 38.57 (1.56) | 53.88 (1.94) |\n| Polynomial (d=10) [A] | 80.44 (1.56) | 68.71 (1.67) | 38.08 (1.10) | 55.20 (1.97) |\n| Exp [A]             | 80.42 (2.14) | 68.81 (1.34) | 38.06 (1.61) | 55.50 (2.11) |\n| ANOVA [B]                | 83.26 (0.78) | 70.09 (2.44) | 40.77 (1.46) | 56.01 (1.86) |\n| RFF | 87.00 (1.24) | 74.77 (2.00) | 52.23 (1.19) | 68.55 (1.87) |\n\nWe will add more details about the training setup and results on all dataset to the paper as suggested. We will also add the discussion on the gradient instability of using RFFs in the architecture.\n\n3. **Effect of Encoder Layers:** Refer to global response.\n4. **Numerical Differences:** Refer to global response.\n5. **Clarity of Corruption Function:** Refer to global response.\n\n* [A] Random Feature Maps for Dot Product Kernels. AISTATS 2012.\n* [B] Random Feature Maps for the Itemset Kernel. AAAI 2019.\n"}, {"review_id": "W9eGvWnin9", "reviewer": "Reviewer_Ck44", "summary": "The paper proposes a contrastive learning model for learning node embeddings on a graph, with two technical innovations: First, the authors propose a new augmentation scheme during contrastive learning. Secondly, the authors re-map high dimensional embeddings into lower dimensional space using random Fourier features. The authors claim improvements attributed to both technical innovations on a variety of datasets.", "questions": "- l. 178: What is the corruption function?\n- Eq. 3: I think the equation is off and misses two minus signs, once in front of the loss term, and one in front of the negative examples. I.e., shouldnt the general form be $ - \\left(\\sum \\mathbb E \\log D_\\omega(.) + \\sum \\mathbb E \\log (1 - D_\\omega(.)) \\right)$ ?\n- What is the rationale for only running experiments on all datasets in Table 1, vs. for all the tables? Also, could you outline the choice of these particular datasets for your study? For instance, SUGRL also evaluates on Ogbn-arxiv, Ogbn-mag, Ogbn-products.\n- Could you more clearly outline your contribution w.r.t. the representation learning part? The loss formulation as such seems to be established, still a substantial amount of space in section 5 is distributed to discussing the loss and its derivation. My impression is that mainly the composition and augmentation of the positive pair distinguishes this part from previous work. Could you confirm/discuss this further?\n- Does the FIGURe model in Table 1 use RFF, or not? If it does not use RFF as suggested by the caption, then the result in Table 1 would contradict the result in Table 3. Could you clarify?\n- Are there any assumptions underlying your proposed augmentation scheme with respect to the properties of the datasets? I.e., are the classification tasks in which a representation obtained using FIGURe can be expected to perform worse than other techniques due to the augmentation design?", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Presentation if the paper is mostly clear, figures are well readable, and results are clearly presented.", "weaknesses": "- Although the word \"augmentation\" is very present in both the paper title and abstract (6 occurences), it only appears once in the paper. It is not fully clear how the abstract claims map onto the technique in the paper, beyond the preliminary info in Section 4.1. The terms should be used more consistently here. It is somewhat inferable from 4.1 what the \"augmentations\" are, but this needs to be clarified in Section 5.\n- Presentation in section 5 is a bit unclear to me. The algorithm proposes two innovations (filter augmentations and random Fourier features), but the experiment section is not grouped accordingly. I would propose to split the content in 5.1. and 5.2. differently to aid the understanding of the different algorithmic components and their implementation.\n- l. 183 claims \"Maximizing JS is equiv. to reducing BCE\" --- can you add a suitable reference for this claim?\n- **Baselines**: I checked a few of the reported baselines numbers. For instance, [SUGRL (Table 1)](https://ojs.aaai.org/index.php/AAAI/article/view/20748) reports $83.5 \\pm .5$ on Cora vs. $81.21 \\pm 2.07$ here, $73.0 \\pm 0.4$ on CiteSeer vs. $67.5 \\pm 1.62$ here. Also some other baseline numbers seem to be off. Could you comment on possible discrepancies, and whether the code you are running for your experiment can reproduce the numbers reported in the literature? Some of the differences I mentioned above are similar in scale to the ones you use to claim the effectiveness of your method.\n- From Table 3 (which I find convincing), it seems like RFF substantially improves many of the baseline methods. It is not really clear to me whether the reported gains are mostly due to the proposed augmentation, or mainly due to the RFF. A clean ablation experiement, ideally on all the datasets would help. Specifically, is the method SOTA also *without* RFF, i.e., due to the different representation learning approach? This is not addressed by the experiments currently.\n- In general, the experimental results seem limited. See my clarification questions below.\n\n**Minor:**\n\n- The choice of colors in Table 1 is probably not well readable by readers with colorblindness. \n- The word \"significant\" is used multiple times in the text, but no statistical tests were run. I propose to either add statistics to your tables (my recommendation), or drop/replace the term.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to review our paper. We truly appreciate your insightful feedback. We are glad to know that you found the presentation of our paper mostly clear, with readable figures and clearly presented results. \n\n# Comment Response\n\n1. **Explaining Augmentation:** We acknowledge the inconsistency in the usage of the term \"augmentation\" and will strive to enhance clarity by using the term consistently and providing a more detailed explanation in Section 5.\n\n2. **Numerical Differences:** Refer to global response.\n\n2. **Presentation/RFF Gains/Filter Gains:** In Table 2, FiGURe refers to the model with 512 dimensional embeddings without RFF, whereas FiGUR$e_{32}$ and FiGUR$e_{128}$ refer to models with lower dimensional embeddings that utilize RFF. No single baseline does consistently well across all heterophilic and homophilic datasets. Comparing FiGURe with other methods shows that using only filter augmentation is consistently either competitive or doing better than existing methods. Note that all the baseline methods are utilizing the same number of dimensions as FiGURe. FiGUR$e_{32}$ and FiGUR$e_{128}$ show the value of RFF to improve performance and be competitive even when learning lower dimensional embeddings. So the way to read the gains from the individual innovation parts is: 1] Filter augmentation provides consistent performance across all datasets which is comparable or better than all baselines and 2] RFF allows for efficient training via lower dimensions while providing similar gains as training with large dimensional embeddings. We will add more clarification and rearrange the tables to get the message across better.\n\n3. **Max JS implies Min BCE Reference:** We realized that this statement is imprecise. Maximizing the JS MI estimator is a hard problem but it can be approximatelly optimized as a BCE loss following the works in [A,B,C]\n\n4. **Choice of Colors:** Thank you for making us aware of this. We will fix this in the final version of the paper.\n\n5. **Clarity of Corruption Function:** Refer to global response.\n\n6. **Equation Discrepancy:** Refer to global response.\n\n7. **Choice of Dataset:** SUGRL primarily focused on homophilic datasets, hence we selected a few datasets with varying levels of homophily [D], to demonstrate the capability of our method. Our ablation studies showed results on two homophilic and two heterophilic datasets, demonstrating some of the properties we want to showcase, with regards to our method. We also show results on ArXiv-Year and OGBN-ArXiv to show that we can scale up to 150k+ nodes. Please refer to Table - 8 in the supplementary material. We plan to conduct additional experiments on ogbn-mag/products which have 1M+ nodes.\n\n8. **Contribution Summary:** One of the main contributions of this work is the notion of filter augmentations. Prior works have incorporated various filter banks in the supervised setting, and they jointly learn the filter specific representations and the combination coefficients. The novel insight that we are proposing in this paper is that we can separately learn these two parts and **bridge the gap with supervised methods**. This enabled us to propose a **simple extension to DGI** which easily incorporates these filter banks. However, augmenting more filters can increase the training cost. To save on compute, we also show that reducing the size of the learnt embeddings and then projecting them using RFF allows one to achieve similar performance with lower dimensional embeddings.\n\n9. **Assumptions and threats to model:** Our model is based on previous filter-based supervised models that have been proposed in the literature. It has been shown in previous works [E] that heterophilic datasets like Chameleon and Squirrel require filters that focus on the tail ends of the spectrum, leading to a dumbell/parabolic frequency response function. In the case of the GPRGNN filters the model is able to learn this response function easily, by giving a high weightage to the $A^{2}$ filter. Similarly, homophilic datasets require low-pass filters, which the GPRGNN filters can easily produce by giving higher weightage to the A and/or $A^3$ filters. We show some of the learnt coefficients in the table below for the two homophilic and heterophilic datasets. \n\n|           | I    | $A$ | $A^{2}$ | $A^{3}$  |\n|-----------|------|---|------|-------|\n| Cora      | 18.2 | 0 | 0    | 35.95 |\n| Citeseer  | 0    | 0 | 0    | 0.48  |\n| Squirrel  | 0    | 0 | 15.3 | 0     |\n| Chameleon | 0    | 0 | 8.93 | 0.1   |\n\nHowever, the coefficients required to create these shapes using other filter banks such as BernNet and ChebNet, although possible, are not so simple and the model has a hard time learning them. We believe that this is the reason why GPRGNN filters give better performance on the datasets that we have considered. However, we would like to point out that there may be datasets where other frequency response shapes are required, and in these cases, it is possible that other filter banks like BernNet/ChebNet filters are more suited to the problem. \n\n\nAdditionally, we would like to point out that DGI allowed us to easily extend and incorporate filter banks, while other methods like MVGRL, GRACE and SUGRL, provide some non-trivial challenges to incorporate them. However, we do believe that it would be possible, and would like to include this as a part of future research.\n\n* [A] Learning deep representations by mutual information estimation and maximization, ICLR 2019\n* [B] Learning Independent Features with Adversarial Nets for Non-linear ICA, ICML 2017 Workshop on Implicit Models\n* [C] f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization, NeurIPS 2016\n* [D] Geom-GCN: Geometric Graph Convolutional Networks, ICLR 2020\n* [E] A Piece-wise Polynomial Filtering Approach for Graph Neural Networks, ECML PKDD 2022"}, {"review_id": "AVChPHlBG0", "reviewer": "Reviewer_wrwF", "summary": "This work proposes a model and contrastive learrning method for acquiring a comprehensive specturm of graph representation by employing filters of various levels. They are adaptively aggregated with learnable weights for downstream supervision tasks. As a result, this approach performs well on both homophilic and heterophilic graphs. Additionally, by utilizing a random featrue map for kernel approximation, it demonstrates effectiveness with low computational cost.  They employ a shared encoder for multiple filters, which not only reduces computational load compared to using independent filters for each, but also improves performane in some cases. Experimental results exhibit performance improvement on average. ", "questions": "In Section 6.5, the authors conducted ablation studies for filters, and it was observed that GPRGNN performed better in all cases. However, it would be informative to explore if there are any other options that demonstrate superior performance. This could involve examining alternative filter configurations or considering different methods altogether. Providing such insights would enhance the comprehensiveness of the analysis and strengthen the paper's contribution. Furthermore, analyzing the learned weight $\\alpha$ or  TSNE plots of learned representations with different filters would also be interesting.", "limitations": "-", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "They conducted experiments on data with multiple characteristics, demonstrating that their approach works for both homophilic and heterophilic graphs. They proposed various methods for computational efficiency and provide evidence of the effectiveness through experiments. The contribution of their work is clearly described through abtract and introduction sections.", "weaknesses": "This paper requires some improvements in terms of presentation, as it contains inconsistent notation (such as size of tilde and hat in the line 147 and 149) and deferred explanation for newly introduced symbols that is difficult to follow and numerous mathematical errors. The presence of these mathematical errors prompted a desire to check the implementation, which unfortunately was not provided by the authors. If these issues are addressed in the rebuttal and the necessary improvements are made, it would be considered for increasing score. It is necessary to clarify the notation and provide better explanations to ensure that readers can follow the content more easily.\n\nFor example, in Section 5.1, the notation $D_w$ is introduced for the first time on line 149, while the explanation for it appears much later, specifically on line 166.\n\nAdditionally, there seem to be errors in the equations. In Equation 1, the correct formulation of the MI estimator should be:\n\n$E_{P_i}[-sp(-T)] - E_{P_i \\times \\tilde{P}_i}[sp(T)]$.  (details are omitted here. Just refer the ‘-‘ sign) \n(7th slide in [this lecture](http://people.ee.duke.edu/~lcarin/Jiachang3.20.2020.pdf))\n\nThis formulation intuitively maximizes MI, as the first term increases the similarity between local and global features obtained from the same graph, while the second term decreases the similarity between local and global features obtained from different graphs. Similar errors can be observed in Equation 3, where minimizing the given objective $\\mathcal{L}_{F_i}$ seems to result in learning that decreases the similarity between local and global representations from the same graph, particularly in the first term. Moreover, in the second term of Equation 3, only the corrupted one $(\\tilde{X}_i, \\tilde{F}_i)$ is considered for expectation and $h_g^{F_i}$ is not covered, which, leads to ambiguous representation. Furthermore, in Section 6, $\\sigma$ on lines 227, 233, and in Equation (6) refer to different functions, causing confusion.\n\nOverall, it is crucial to address these issues related to notation, mathematical accuracy, and clarity throughout the paper to enhance its quality and ensure a better understanding for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for dedicating time to thoroughly review our paper and for providing comprehensive feedback on our work. We appreciate your recognition of the strengths of our research. It is encouraging to note that you found value in our experiments across data with multiple characteristics, demonstrating the versatility of our approach in handling both homophilic and heterophilic graphs. We are also glad that you acknowledged the clarity of our contributions in the abstract and introduction sections, and our methods proposed for computational efficiency. We genuinely acknowledge the concerns you have raised regarding the presentation, inconsistencies in notation, and mathematical errors. \n\n1. **Writing improvements:** We apologize for the notational inconsistencies. We thank the reviewer for bringing this to light. We have made the requested changes so that the paper is consistent in its notations. Specifically:\n     1. The size of the tilde has been changed to be consistent with the hats in line 147 and 149.\n     2. Notation $D_{w}$ is now correctly introduced before its use in line 149.\n     3. The MI estimator in equation 1 has now been corrected $\\mathcal{I}^{\\textrm{JSD}}\\_{i, \\theta, \\omega}(\\cdot, \\cdot):= \\mathbb{E}\\_{\\mathbb{P}\\_{i}}[-\\text{sp}(-T\\_{\\theta, \\omega}(\\cdot, \\cdot)]- \\\\ \\mathbb{E}\\_{\\mathbb{P}\\_{i} \\times \\mathbb{\\widetilde{P}}\\_{i}}[\\text{sp}(T\\_{\\theta, \\omega}(\\cdot,  \\cdot)]$\n     4. Equation 3 has been fixed: $\\mathcal{L}\\_{F\\_i} = -\\frac{1}{N + M}\\left( \\sum_{j=1}^{N}\\mathbb{E}[\\text{log}(D_{\\omega}(h_{j}^{F\\_{i}}, h_{g}^{F\\_{i}}))] + \\sum\\_{j=1}^{M}\\mathbb{E}[\\text{log}(1 - D\\_{\\omega}(\\widetilde{h\\_{j}^{F\\_{i}}}, h\\_{g}^{F_{i}}))] \\right)$\n     5. We have mitigated the confusion about the sigma variable, by using psi to represent PReLU and sigma to represent sigmoid\n     6. Additionally, we rewrote some parts to reduce the confusion\n\n2. **Code Access:** We would like to point out that the anonymized link to the code is present in the supplementary material in line number 471, in section 8.1.\n\n3. **Additional Analysis**: We appreciate your suggestion to broaden our exploration and consider other options as well. In addition to the BernNet filters reported in Section 6.5 of the paper, we report the performance of the Chebyshev filters used in ChebNet [A]. We report two configurations, 3 filters and 11 filters, similar to the analysis done with BernNet filters. As can be observed from the table, the GPRGNN filters outperform this too. It has been shown in previous works [B] that heterophilic datasets like Chameleon and Squirrel require filters that focus on the tail ends of the spectrum, leading to a dumbell/parabolic frequency response function. In the case of the GPRGNN filters the model is able to learn this response function easily, by giving a high weightage to the $A^{2}$ filter. Similarly, homophilic datasets require low-pass filters, which the GPRGNN filters can easily produce by giving higher weightage to the A and $A^3$ filters. However the coefficients required to create these shape using other filter banks such as BernNet and ChebNet, although possible, are not so simple and the model has a hard time learning them. We believe that this is the reason why GPRGNN filters give better performance on the datasets that we have considered. However, we would like to point out that there may be datasets where other frequency response shapes are required, and in these cases it is possible that BernNet and ChebNet filters are more suited to the problem. \\\\\nAdditionally, we report the alpha coefficients of the best performing split using GPRGNN filters, on Cora, Citeseer, Squirrel and Chameleon (The coefficients will be slightly different for different splits in the same dataset). The observations from this table support our hypothesis about the coefficients of the GPRGNN filters. Specifically, it can be observed that the homophilic datasests Cora and Citeseer are $A^{3}$ dominant, leading the creation of a low pass filters. In contrast, for the heterophilic datasets Chameleon and Squirrel, the $A^{2}$ filter is dominant, creating a parabolic filter response function. We will add TSNE plots for different filters and filter-banks to the final version of the paper.\n* [A]: Convolutional Neural Networks on Graphs with Chebyshev Approximation, Revisited. NeurIPS 2022.\n* [B]:  A Piece-wise Polynomial Filtering Approach for Graph Neural Networks, ECML PKDD 2022\n\n\n"}, {"review_id": "aDiRIKg2pe", "reviewer": "Reviewer_xu4A", "summary": "The proposer proposed a contrastive learning method for graphs. The goal is to learn representations for nodes in an unsupervised way, by maximizing mutual informations between the local representation and global representation of a graph, training a single encoder to learn node representations. The method proposes to use multiple different filters to learn representations with different emphasis on the graph structure (as seen through the spectral representation), and then combine those representations on downstream tasks, by learning the mixing coefficients in addition to a linear classifier.\n\nThe proposed method, FIGURe, being slower than the other methods it is compared to, when learning 512 representations, the authors also explore ways to produce more compact representations, 32 and 128 in width, using RFF to recover some of the lost accuracy on downstream tasks.", "questions": "Here are a bunch of details that I think need clarification in the experimental sections:\n* How long is the method trained compared to the baselines ? Does the number of filter influence the training time (linearly?).\n* How many filters does the method used compare to the baselines ? If it uses more, should we normalize those to make sure comparisons are fair ?\n* How are the baselines evaluated ? Are those also using the learnable alpha mixing coefficients ? Are those baselines using the same number of filters ? If not, should we normalize those to make sure the evaluations are fair ?\n* What happens if we use RFF on the 512 dimensions representations ? In particular, on the baselines ?\n* What happens if we do not use RFF on the 32 dimensions and 128 dimensions representations ?\n\nIn several sections, the authors talk about the cost of having high number of dimensions and the cost of having several encoder instead of one shared encoder, would it be possible to put numbers on these costs in terms of time / storage ? It would help convey the need for the dimensionality reduction and RFF tricks.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The results show improvement compared to other self-supervised approaches. Although FIGURe does not get the first place on every single benchmark, it performs very consistency across datasets, leading to the highest average performance.\n* The versions of FIGURe with smaller representations are leading to good results as well, underlying the soundness of the approach followed by the authors to reduce the dimension of representation and then recovering part of the lost accuracy.\n* Overall, the paper is well written, and easy to follow. The preliminary section is concise and yet very useful.", "weaknesses": "* The study of the authors is limited to 1-layer GCN. It would be interesting to know how the method compares to others as the GNN architecture gets bigger (in particular deeper).\n* The authors justify the need for learning representations of smaller sizes due to the prohibitive cost of contrastive learning, but this part lacks justification in my view: the mean epoch time does indeed increase compared to baselines (+50%), but we are missing details in the paper related to how many epochs it is trained on (in absolute and compared to others methods) - this would help justify the need for the lower dimensionality version and the RFF tricks used afterwards\n* The authors present the shared encoder as an efficiency approach, but it seems from Table 4 that it is also increasing performance.\n* In general, I would appreciate having more details on the experimental section, and in particular on the training recipe and the evaluation recipes - some of these questions will be asked in the \"Questions\" sections of this review - as this would help understanding if the comparison are fair (and help reproducing results)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First and foremost, we would like to express our gratitude for the time and effort you dedicated to reviewing our paper, providing us with constructive feedback. We're glad to learn that you found our paper well-written, with a concise and useful preliminary section. We highly appreciate your positive remarks about FIGURe's consistent performance across datasets and its demonstrated soundness in our approach, especially with reduced dimension representations. The acknowledgment of FIGURe's improvement over other self-supervised approaches further underscores the value of our contribution.\n\n1. **Multi Layer Encoder:** Refer to global response.\n\n2. **Time and Storage Cost**: For all the baselines, we used the same number of maximum epochs and the same early stopping criteria. The maximum number of epochs was set to 30,000 and the early stop patience was set to 20 epochs, except for the large datasets, for which the patience was set to 500 epochs. Due to the fact that each baseline was tuned over multiple hyperparameters, each having a different convergence criterion, it would be difficult to normalize the epochs. However, we made sure that we set the epochs to a large value such that the models generally converged to a solution. We build upon DGI. The training time of our model scales linearly with the number of filters. Since our model currently uses three filters for training, our training time is three times larger than that of DGI. We provide the number of epochs, mean epoch time, and total training time for ogbn-arxiv. We also report the storage cost of the representations from these methods. Please note that that the linear time relationship between DGI and FiGURe is not visible here due to the inclusion of batching and sampling time.\n\n| Model / Dims| Num Epochs | Mean Epoch Time | Total time | Storage |\n|-----------|--------------------------|-------------------------------|--------------------------|--------------------------|\n| DGI / 512       | 3945   | 0.77s                         |      50.75mins           | 330.75 MB\n| FiGURe / 512       | 4801                     | 0.92s                         | 73.62mins                | 1.32GB|\n| FiGURe / 128       | 4180                     | 0.74s                         | 51.55mins                | 330.75 MB|\n| FiGURe / 32        | 3863                     | 0.72s                         | 46.36mins               | 82.69MB|\n\nWe see both the training time and storage saved by utilizing lower dimensions along with RFF.\n\n3. **Shared Encoder Improvements:** We hypothesize that weight sharing among encoders results in embedding different filter representations in a shared subspace, thereby enhancing their suitability for learning a combined representation. This ultimately leads to improved features for downstream tasks and, in some cases, results in performance improvements. We will add this point in the final version of the paper.\n\n4. **Normalize Comparisons with Baselines:** \n     1. Normalized number of parameters: One way to normalize the models is to ensure that they have roughly equal number of parameters. In the FiGURe model with shared encoder, the number of parameters used are exactly the same as the number of parameters used in DGI and GRACE.  MVGRL and SUGRL use two encoder models, hence roughly have twice the number of parameters. Hence, the FiGURe model uses equal or fewer parameters.\n     2. Normalized filter inputs: Another way to normalize the inputs is to provide the same filter augmentations to all the baselines. However, there are non-trivial challenges in incorporating filter banks in the other baselines. Hence, we chose to incorporate them in DGI. We do believe that it should be possible to incorporate them in other baseline models as well and it is left for future research.\n\n5. **RFF on 512 dimensions**: We report the result of using RFF on 512 dimensions. We notice that there is not much improvement in performance, when you already have sufficiently large dimensionsal embeddings.\n| Cora         | Citeseer     | Squirrel     | Chameleon    |\n|--------------|--------------|--------------|--------------|\n  86.84 (0.98) | 74.40 (1.30) | 51.86 (1.87) | 68.60 (1.57) |\n\n6. **With and without RFF**:\nIn Table -3, we have shown how our method performs with 32 dimensions with and without RFF. One can observe that our method is better than or comparable to most other methods at 32 dimensions across datasets. This trend continues even after projecting the learnt embeddings via RFF to larger dimensions. Additionally, we report the same table but with 128 dimensions below. Similar observations can be made here as well.\n\n|         | RFF | cora        | citeseer    | squirrel    | chameleon   |\n|---------|------------|--------------|--------------|--------------|--------------|\n| DGI     | $\\times$   | 84.99 (1.36) | 72.22 (2.50) | 34.22 (1.47) | 49.82 (2.96) |\n|        | $\\checkmark$ | 84.17 (2.11) | 72.65 (1.52) | 37.97 (1.41) | 57.72 (2.03) |\n| Hassani | $\\times$   | 85.31 (1.66) | 73.42 (1.63) | 36.92 (1.04) | 55.20 (1.70) |\n|        | $\\checkmark$ | 84.61 (1.74) | 72.81 (2.13) | 38.73 (1.22) | 57.81 (1.80) |\n| SUGRL   | $\\times$   | 71.49 (1.15) | 63.85 (2.27) | 38.04 (1.17) | 53.03 (1.73) |\n|        | $\\checkmark$ | 71.40 (1.40) | 63.06 (2.22) | 43.24 (1.63) | 57.04 (1.78) |\n| GRACE   | $\\times$   | 80.87 (1.49) | 62.52 (3.57) | 41.25 (1.32) | 63.14 (1.89) |\n|        | $\\checkmark$ | 79.70 (1.91) | 64.47 (2.12) | 52.29 (1.81) | 68.90 (2.05) |\n| FiGURe| $\\times$   | 84.73 (1.13) | 73.07 (1.13) | 41.06 (1.51) | 59.08 (3.36) |\n|        | $\\checkmark$ | 86.14 (1.13) | 73.34 (1.91) | 48.78 (2.48) | 66.03 (2.19) |\n"}]

Author rebuttal:
We sincerely thank the reviewers for their valuable feedback. Reviewer qpib's recognition of our novel approach leveraging filter banks is appreciated. Reviewer Ck44's positive remarks on clear presentation and readable figures are noted. Reviewer wrwF's acknowledgment of our approach's versatility and clarity in contributions is encouraging. Reviewer xu4A's appreciation of our paper's quality, consistent performance of FIGURe, and improvement over other methods highlights the strength of our work.

We have addressed specific concerns that the reviewers have raised as responses to the individual reviews. Here we address some of the common concerns that were raised by two or more reviewers.

1. **Equation discrepancy**: Equation 3 has been fixed: $\mathcal{L}\_{F\_i} = -\frac{1}{N + M}\left( \sum\_{j=1}^{N}\mathbb{E}[\text{log}(D\_{\omega}(h\_{j}^{F\_{i}}, h\_{g}^{F\_{i}}))] + \sum\_{j=1}^{M}\mathbb{E}[\text{log}(1 - D\_{\omega}(\widetilde{h\_{j}^{F\_{i}}}, h\_{g}^{F\_{i}}))] \right)$

2. **Numerical differences:** There are some differences in the numbers reported in the original papers of the baseline methods and our reproduction. The dataset splits used are different from the ones used in the original papers. Specifically, the splits we use have been used before in papers like [A,B,C]. The reason we chose these splits is two-fold. Firstly, there are ten splits provided in these papers (except for the large datasets), as opposed to the single split versions that the baseline methods are working with. We report all our results as averages over all these 10 splits. Secondly, there are certain inconsistencies in some of the splits being used in the baselines. For example, for cora, the dataset split used by [D] is different from the one used utilised in the original [E] paper, which is the split they use for reporting their baseline performance numbers. For more details on this, issue \#2 on the MVGRL GitHub repository can be referred to. For all the baselines, we ran the code provided by the authors of those works along with extensive hyperparameter tuning. Specifically, we separate the hyperparameter tuning of the unsupervised part and supervised part, and do not use any information from the supervised part while tuning the hyperparameters of the unsupervised part.


3. **Encoder layers:** We include an analysis with more encoder layers. There are two interpretations of 'encoder layers' in this case. First, a deeper GCN, which implies the aggregation of multiple hop neighborhood information into the node. Second, a single hop GCN with a deeper network to transform the features. We report the performance of both these cases, with two and three layers each. It can be observed that the single layer GCN performs equal or better than all other cases. 

| Accuracies            | Cora | Citeseer |  Squirrel | Chameleon |
|-------------|------------------|---------------------|---------------------|----------------------|
| 1 Layer GCN | 87.00 (1.24)     | 74.77 (2.00)        | 52.23 (1.19)        | 68.55 (1.87)         |
| 2 Layer GCN | 86.62 (1.43)     | 73.62 (1.46)        | 43.80 (1.57)        | 53.53 (2.13)         |
| 3 Layer GCN | 84.40 (1.84)     | 72.52 (2.09)        | 42.79 (1.12)        | 61.73 (2.25)         |
| GCN + 2 Layer MLP | 85.73 (1.03)     | 70.21 (2.30)        | 49.91 (2.68)        | 68.18 (1.76)         |
| GCN + 3 Layer MLP | 84.99 (1.43)     | 71.39 (2.32)        | 45.85 (3.33)        | 64.19 (1.43)         |


4. **Clarity of Corruption Function**: The corruption function takes nodes and their associated features ($\mathbf{X}$) as input, along with a graph ($\mathbf{F}_{i}$) representing these nodes. The function maintains the structure of the graph, but it rearranges the rows of $\mathbf{X}$. This rearrangement effectively shuffles the node features, ensuring that each node's features differ from the original input data. When generating embeddings using a Graph Convolutional Network (GCN), the node embeddings are combined, resulting in a corrupted representation of the nodes.


* [A] Geom-GCN: Geometric Graph Convolutional Networks, ICLR 2020
* [B] A Piece-wise Polynomial Filtering Approach for Graph Neural Networks, ECML PKDD 2022
* [C] A critical look at the evaluation of GNNs under heterophily: are we really making progress? ICLR 2023
* [D] Contrastive Multi-View Representation Learning on Graphs. ICML 2020
* [E] Semi-Supervised Classification with Graph Convolutional Networks. ICLR 2017



Meta-review: Paper Summary

The paper proposes an approach to learning representations of nodes using contrastive learning. The authors show a new augmentation scheme that is particularly effective and also show a low-dimensional feature mapping scheme using Random Fourier Features that improves performance.


Reviews & Justification

The paper received mostly positive reviews. One reviewer raised concerns about code and notational issues which the authors have addressed satisfactorily. Multiple reviewers raised concerns about the applicability of the work to multi-layer GNNs. The authors have addressed this concern in their response. The AC encourages the authors to add these experiments to the final paper as well. Finally, the authors also provided an in-depth experimental analysis of RFF dimensions, the alpha coefficients, and examined a different filter bank.
Decision: Accept (poster)