Title: FiGURe: Simple and Efficient Unsupervised Node Representations with Filter Augmentations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Neural Networks

Unsupervised Representation Learning

Graph Filters
Abstract: Unsupervised node representations learnt using contrastive learning-based methods have shown good performance on downstream tasks. However, these methods rely on augmentations that mimic low-pass filters, limiting their performance on tasks requiring different eigen-spectrum parts. This paper presents a simple filter-based augmentation method to capture different parts of the eigen-spectrum. We show significant improvements using these augmentations. Further, we show that sharing the same weights across these different filter augmentations is possible, reducing the computational load. In addition, previous works have shown that good performance on downstream tasks requires high dimensional representations. Working with high dimensions increases the computations, especially when multiple augmentations are involved. We mitigate this problem and recover good performance through lower dimensional embeddings using simple random Fourier feature projections. Our method, FiGURe, achieves an average gain of up to 4.4\%, compared to the state-of-the-art unsupervised models, across all datasets in consideration, both homophilic and heterophilic. Our code can be found at: https://github.com/Microsoft/figure.

FiGURe: Simple and Efficient Unsupervised Node Representations with Filter Augmentations

Abstract

Unsupervised node representations learnt using contrastive learning-based methods have shown good performance on downstream tasks. However, these methods rely on augmentations that mimic low-pass filters, limiting their performance on tasks requiring different eigen-spectrum parts. This paper presents a simple filter-based augmentation method to capture different parts of the eigen-spectrum. We show significant improvements using these augmentations. Further, we show that sharing the same weights across these different filter augmentations is possible, reducing the computational load. In addition, previous works have shown that good performance on downstream tasks requires high dimensional representations. Working with high dimensions increases the computations, especially when multiple augmentations are involved. We mitigate this problem and recover good performance through lower dimensional embeddings using simple random Fourier feature projections. Our method, FiGURe, achieves an average gain of up to 4.4%, compared to the state-of-the-art unsupervised models, across all datasets in consideration, both homophilic and heterophilic. Our code can be found at: https://github.com/ microsoft/figure .

1 Introduction

Contrastive learning is a powerful method for unsupervised graph representation learning, achieving notable success in various applications [44, 15]. However, these evaluations typically focus on tasks exhibiting homophily, where task labels strongly correlate with the graph's structure. An existing edge suggests the connected nodes likely share similar labels in these scenarios. However, these representations often struggle when dealing with heterophilic tasks, where edges tend to connect nodes with different labels. Several papers [7, 17, 4, 28] have tackled the problem of heterophily by leveraging information from both low and high-frequency components. However, these methods operate in the semi-supervised setting, and the extension of these ideas in unsupervised learning still needs to be explored. Inspired by the insights in these papers, we propose a simple method incorporating these principles.

Our approach introduces filter banks as additional views and learns separate representations for each filter bank. However, this approach faces two main challenges: Firstly, storing representations from each view can become prohibitively expensive for large graphs; secondly, contrastive learning

methods typically demand high-dimensional representations, which increase both the computational cost of training and the storage burden. We employ a shared encoder for all filter banks to tackle the first challenge. Our results confirm that a shared encoder performs on par with independent encoders for each filter bank. This strategy enables us to reconstruct filter-specific representations as needed, drastically reducing the storage requirement. For the second challenge, we train our models with low-dimensional embeddings. Then, we use random Fourier feature projection [40] to lift these low-dimensional embeddings into a higher-dimensional space. Kernel tricks [23] were typically used in classical machine learning to project low-dimensional representation to high dimensions where the labels can become linearly separable. However, constructing and leveraging the kernels in large dataset scenarios could be expensive. To avoid this issue, several papers [40, 41, 20, 36, 26] proposed to approximate the map associated with the kernel. For our scenario, we use the map associated with Gaussian kernel [40]. We empirically demonstrate that using such a simple approach preserves high performance for downstream tasks, even in the contrastive learning setting. Consequently, our solution offers a more efficient approach to unsupervised graph representation learning in computation and storage, especially concerning heterophilic tasks. The proposed method exhibits simplicity not only in the augmentation of filters but also in its ability to learn and capture information in a low-dimensional space, while still benefiting from the advantages of large-dimensional embeddings through Random Fourier Feature projections.

Our contributions in this work are, 1] We propose a simple scheme of using filter banks for learning representations that can cater to both heterophily and homophily tasks, 2] We address the computational and storage burden associated with this simple strategy by sharing the encoder across these various filter views, 3] By learning a low-dimensional representation and later projecting it to high dimensions using random Fourier Features, we further reduce the burden, 4] We study the performance of our approach on four homophilic and seven heterophilic datasets. Our method, FiGURe, achieves an average gain of up to 4.4%, compared to the state-of-the-art unsupervised models, across all datasets in consideration, both homophilic and heterophilic. Notably, even without access to task-specific labels, FiGURe performs competitively with supervised methods like GCN [25].

2 Related Work

Several unsupervised representation learning methods have been proposed in prior literature. Random walk-based methods like Node2Vec [14] and DeepWalk [37] preserve node proximity but tend to neglect structural information and node features. Contrastive methods, such as DEEP GRAPH INFOMAX (DGI) [44], maximize the mutual information (MI) between local and global representations while minimizing the MI between corrupted representations. Methods like MVGRL [15] and GRACE [45] expand on this, by integrating additional views into the MI maximization objective. However, most of these methods focus on the low frequency components, overlooking critical insights from other parts. Semi-supervised methods like GPRGNN [7], BERNNET [17], and PPGNN [28] address this by exploring the entire eigenspectrum, but these concepts are yet to be applied in the unsupervised domain. This work proposes the use of a filter bank to capture information across the full eigenspectrum while sharing an encoder across filters. Given the high-dimensional representation demand of contrastive learning methods, we propose using Random Fourier Features (RFF) to project lower-dimensional embeddings into higher-dimensional spaces, reducing computational load without sacrificing performance. The ensuing sections define our problem, describe filter banks and random feature maps, and explain our model and experimental results.

3 Problem Setting

In the domain of unsupervised representation learning, our focus lies on graph data, denoted as G = ( V , E ) , where V is the set of vertices and E the set of edges ( E ⊆ V × V ). We associate an adjacency matrix with G , referred to as A : A ∈ { 0 , 1 } n × n , where n = |V| corresponds to the number of nodes. Let X ∈ R n × d be the feature matrix. We use A I to represent A + I with I is the identity matrix, while D A I signifies the degree matrix of A I . We also define A n as D -1 / 2 A I A I D -1 / 2 A I . No additional information is provided during training. The goal is to learn a parameterized encoder, E θ : R n × n × R n × d ↦→ R n × d ′ , where d ′ ≪ d . This encoder produces a set of node representations E θ ( X , A n ) = { h 1 , h 2 , ..., h n } where each h i ∈ R d ′ represents a rich representation for node i . The

subsequent section will provide preliminary details about filter banks and random feature maps before we discuss the specifics of the proposed approach.

4 Preliminaries

Our approach relies on filter banks and random feature maps. In this section, we briefly introduce these components, paving the way for a detailed explanation of our approach.

4.1 Filter Banks

Graph Fourier Transform (GFT) forms the basis of Graph Neural Networks (GNNs). A GFT is defined using a reference operator R which admits a spectral decomposition. Traditionally, in the case of GNNs, this reference operator has been the symmetric normalized laplacian L n = I -A n or the A n as simplified in [25]. A graph filter is an operator that acts independently on the entire eigenspace of a diagonalisable and symmetric reference operator R , by modulating their corresponding eigenvalues. [43, 42]. Thus, a graph filter H is defined via the graph filter function g ( . ) operating on the reference operator as H = g ( R ) = U g ( Λ ) U T . Here, Λ = diag ([ λ 1 , λ 2 , ..., λ n ]) , where λ i denotes the eigenvalues of the reference operator. We describe a filter bank as a set of filters, denoted as F = { F 1 , F 2 , ..., F K } . Both GPRGNN [7] and BERNNET [17] employ filter banks, comprising of polynomial filters, and amalgamate the representations from each filter bank to enhance the performance across heterophilic datasets. GPRGNN uses a filter bank defined as F GPRGNN = { I , A n , ..., A n K -1 } , while F BERNNET = { B 0 , B 1 , ..., B K -1 } characterizes the filter bank utilized by BERNNET. Here, B i = 1 2 K -1 ( K -1 i ) (2 I -L n ) K -i -1 ( L n ) i . Each filter in these banks highlights different parts of the eigenspectrum. By tuning the combination on downstream tasks, it offers the choice to select and leverage the right spectrum to enhance performance. Notably, unlike traditional GNNs, which primarily emphasize low-frequency components, higher frequency components have proved useful for heterophily [4, 7, 17, 28]. Consequently, a vital takeaway is that for comprehensive representations, we must aggregate information from different parts of the eigenspectrum and fine-tune it for specific downstream tasks .

4.2 Random Feature Maps for Kernel Approximations

Before the emergence of deep learning models, the kernel trick was instrumental in learning non-linear models. A kernel function, k : R d × R d ↦→ R , accepts two input features and returns a real-valued score. Given a positive-definite kernel, Mercer's Theorem [30] assures the existence of a feature map ϕ ( · ) , such that k ( x, y ) = ⟨ ϕ ( x ) , ϕ ( y ) ⟩ . Leveraging the kernel trick, researchers combined Mercer's theorem with the representer theorem [23], enabling the construction of non-linear models that remain linear in k . These models created directly using k instead of the potentially complex ϕ , outperformed traditional linear models. The implicit maps linked with these kernels projected the features into a significantly high-dimensional space, where targets were presumed to be linearly separable. However, computational challenges arose when dealing with large datasets. Addressing these issues, subsequent works [41, 20, 36, 40] introduced approximations of the map associated with individual kernels through random projections into higher-dimensional spaces ( ϕ ′ ( . ) ). This approach ensures that ⟨ ϕ ′ ( x ) , ϕ ′ ( y ) ⟩ ≈ k ( x, y ) . These random feature maps are inexpensive to compute and affirm that simple projections to higher-dimensional spaces can achieve linear separability. The critical insight is that computationally efficient random feature maps, such as Random Fourier features (RFF) [40], exist. These maps project lower-dimensional representations into higher dimensions, enhancing their adaptability for downstream tasks. .

5 Proposed Approach

The following section delineates the process of unsupervised representation learning. Then, we detail the use of filter bank representations in downstream tasks with random feature maps.

5.1 Unsupervised Representation Learning

Our method FiGURe ( Fi lter-based G raph U nsupervised Re presentation Learning) builds on concepts introduced in [18, 44], extending the maximization of mutual information between node and global

filter representations for each filter in the filter bank F = { F 1 , F 2 , ... F K } . In this approach, we employ filter-based augmentations, treating filter banks as "additional views" within the context of contrastive learning schemes. In the traditional approach, alternative baselines like DGI have employed a single filter in the GPRGNN filter bank. Additionally, MVGRL attempted to use the diffusion kernel; nevertheless, they only learned a single representation per node. We believe that this approach is insufficient for accommodating a wide range of downstream tasks. We construct an encoder for each filter to maximize the mutual information between the input data and encoder output. For the i th filter, we learn an encoder, E θ : X i →X ′ i , denoted by learnable parameters θ . In this context, X i represents a set of examples, where each example [ ̂ X ij , ̂ F ij ] ∈ X i consists of a filter F i , its corresponding nodes and node features, drawn from an empirical probability distribution P i . That is, P i captures the joint distribution of a filter F i , its corresponding nodes and node features ( [ X , F i ] ). Note that [ ̂ X ij , ̂ F ij ] denote nodes, node features and edges sampled from the i th filter ( [ X , F i ] ) basis the probability distribution P i . Note that ̂ X ij ∈ R N ′ × d and ̂ F ij ∈ R N ′ × N ′ . X ′ i defines the set of representations learnt by the encoder on utilizing feature information as well as topological information from the samples, sampled from the joint distribution P i . The goal, aligned with [29, 18, 44], is to identify θ that maximizes mutual information between [ X , F i ] and E θ ( X , F i ) , or I i ([ X , F i ] , E θ ( X , F i )) . While exact mutual information (MI) computation is infeasible due to unavailable exact data and learned representations distributions, we can estimate the MI using the Jensen-Shannon MI estimator [9, 32], defined as:

Here, T ω : X i × X ′ i → R represents a discriminator function with learnable parameters ω . Note that [ ˜ X ij , ˜ F ij ] is an input sampled from ˜ P i , which denotes a distribution over the corrupted input data (more details given below). The function sp(.) corresponds to the softplus function [10]. Additionally, T θ,ω ([ h ij ] 1 , [ h ij ] 2 ) = D w ◦ ( R ([ h ij ] 1 ) , [ h ij ] 2 ) , where R denotes the readout function responsible for summarizing all node representations by aggregating and distilling information into a global filter representation. We introduce a learnable discriminator D ω , where D ω ( ., . ) represents the joint probability score between the global representation and the node-specific patch representation. Note that [ h ij ] 1 , denotes representations obtained after passing samples sampled from the original distribution P i , to the encoder, and [ h ij ] 2 , denotes representations obtained after passing samples sampled from the original distribution P i or samples sampled from the corrupted distribution ˜ P i to the encoder. Intuitively, Eq. 1 implies that we would want to maximize the mutual information between the local (patch) representations of the nodes and their corresponding global graph

FilterBanks

Encoding

Shared

SharedEncoder

F1=I

WeightMatrix

Node

o(1x0)

Embeddings

H1

Nodeswith

self-edges

Mutual Information

Maximization

XO

H2

H2

Filtered Graph

Embeddings

Node

Readout

Graph'

Embedding

A

：

maxMI(hf2,hf2)

Hk

o(FkX0)

Readout

Hk

Embeddings

Node

Graph"g

Embedding

i

Filtered Graph

maxMI(hFk,hFk)

representation, while minimising the mutual information between a global graph representation and the local representations of corrupted input data. Note that: [ ˜ X ij , ˜ F ij ] denotes a corrupted version of input features and given filter. More details with regards to this will be given below. In our approach, we first obtain node representations by feeding the filter-specific topology and associated node features into the encoder: H ij = E θ ( X ij , F ij ) = { h F ij 1 , h F ij 2 , ..., h F ij N ′ } . Note that H ij has the following dimensions R N ′ × d ′ .To obtain global representations, we employ a readout function R : R N ′ × d ′ → R d ′ , which combines and distills information into a global representation h F ij g = R ( H ij ) = R ( E θ ( X ij , F ij )) . Instead of directly maximizing the mutual information between the local and global representations, we maximize D ω ( ., . ) . This joint score should be

higher when considering global and local representations obtained from the same filter, as opposed to the joint score between the global representation from one filter and the local representation from corrupted ( X ij , F ij ) . To generate negative samples for contrastive learning, we employ a corruption function C : R N ′ × d × R N ′ × N ′ → R N ′ × d × R N ′ × N ′ , which yields corrupted samples denoted as [ ˜ X ij , ˜ F ij ] = C ( X ij , F ij ) . The designed corruption function generates data decorrelated with the input data. Note that [ ˜ X ij , ˜ F ij ] denote nodes, node features and edges sampled from the corrupted version of i th filter, basis the probability distribution ˜ P i . The corruption function can be designed basis the task at hand. We employ a simple corruption function, whose details are present in the experimental section. Let the corrupted node representations be as follows:

˜ H ij = E θ ( C ( X ij , F ij )) = E θ ( ˜ X ij , ˜ F ij ) = { ˜ h F ij 1 , ˜ h F ij 2 , ..., ˜ h F ij N ′ } . In order to learn representations across all filters in the filter bank, we aim to maximise the average estimate of mutual information (MI) across all filters, considering K filters, defined by I F .

Maximising the Jenson-Shannon MI estimator can be approximately optimized by reducing the binary cross entropy loss defined between positive samples (sampled from the P i ) and the negative samples (sampled from ˜ P i ). Therefore, for each filter, the loss for is defined as follows:

Note that for Eq. 3, the global representation ( h F ij g ) is generated by R ( E θ ( X ij , F ij )) . The local representations ( ˜ h F ij k ∀ k ) are constructed by passing the sampled graph and features through a corruption function (see ˜ H ij ). Therefore to learn meaningful representations across all filters the following objective is minimised:

Managing the computational cost and storage demands for large graphs with distinct node representations for each filter poses a challenge, especially when contrastive learning methods require high dimensions. To address this, we employ parameter sharing, inspired by studies like [7] and [17]. This approach involves sharing the encoder's parameters θ and the discriminator's parameters ω across all filters. Instead of storing dense filter-specific node representations, we only store the shared encoder's parameters and the first-hop neighborhood information for each node per filter, reducing storage requirements. To obtain embeddings for downstream tasks, we reconstruct filter-specific representations using a simple one-layer GNN. This on-demand reconstruction significantly reduces computational and storage needs associated with individual node representations. Fig 1 illustrates such a simple encoder's mutual information-based learning process. To address the second challenge, we initially train our models to produce low-dimensional embeddings that capture latent classes, as discussed in [2]. These embeddings, while informative, lack linear separability. To enhance separability, we project these low-dimensional embeddings into a higher-dimensional space using random Fourier feature (RFF) projections, inspired by kernel methods (see Section 4.2). This approach improves the linear separability of latent classes, as confirmed by our experimental results in Section 6.2, demonstrating the retention of latent class information in these embeddings.

5.2 Supervised Representation Learning

After obtaining representations for each filter post the reconstruction of the node representations, learning an aggregation mechanism to combine information from representations that capture different parts of the eigenspectrum for the given task is necessary. We follow learning schemes from [7, 17, 28],

FilterBanks

F=I

Trained

Encodefrom

g(1x0)

Node

Embeddings

H1

FourierFeatures

viaRandom

Projectionto

HigerDimension

H'

Projection

Φ'(hF1)

self-edges

Nodes with

F

o(F2X0)

H2

cos(wThF1 +t)

Projection

H2

a1

Embedding

Fusion

Z

Class-

probabilities

Filtered Graph

cos(wThF2+t)

Φ'(hF2)

i

Classifier

Node

Fk

o(FkXθ)

Hk

*....

Projection

Hk

ak

Embeddings

Φ'(hFk)

Filtered Graph

learning a weighted combination of filter-specific representations. The combined representations for downstream tasks, considering K filters from filter bank F , are as follows:

The parameters α i 's are learnable. Additionally, the function ϕ ′ ( . ) represents either the RFF projection or an identity transformation, depending on whether E θ ( X , F i ) is low-dimensional or not. A classifier model (e.g. logistic regression) consumes these embeddings, where we train both the α i 's and the weights of the classifier. Fig 2 illustrates this process. Notably, semi-supervised methods like [28, 7, 17] differ as they learn both encoder and coefficients from labeled data, while our method pre-trains the encoder and learns task-specific combinations of filter-specific representations.

6 Experimental Results

Training Details : We define a single-layer graph convolutional network (GCN) with shared weights ( Θ ) across all filters in the filter bank ( F ) as our encoder. Therefore, the encoder can be expressed as follows: E θ ( X , F i ) = ψ ( F i X Θ) . It is important to note that F i represents a normalized filter with self-loops, which ensures that its eigenvalues are within the range of [0, 2]. The non-linearity function ψ refers to the parametric rectified linear unit [16]. As we work with a single graph, we obtain the positive samples by sampling nodes from the graph. Using these sampled nodes, we construct a new adjacency list that only includes the edges between these sampled nodes in filter F i . On the other hand, the corruption function C operates on the same sampled nodes. However, it randomly shuffles the node features instead of perturbing the adjacency list. In essence, this involves permuting the node features while maintaining a consistent graph structure. This action introduces a form of corruption, altering each node's feature to differ from its original representation in the data. Similar to [44], we employ a straightforward readout function that involves averaging the representations across all nodes for a specific filter F i : R ( H i ) = σ ( 1 N ∑ N j =0 h F i j ) where σ denotes the sigmoid non-linearity. We utilize a bilinear scoring function D ω ( ., . ) , whose parameters are also shared across all filters, where D ω ( h F i j , h F i g ) = σ ( h F i T j W h F i g ). We learn the encoder and discriminator parameters by optimising Eq. 4. While we could use various filter banks, we specifically employ the filter bank corresponding to GPRGNN ( F GPRGNN) for all our experiments. However, we conduct an ablation study (see 6.5) comparing F GPRGNN with F BERNNET . Additional training details are available in 8.3.

Weconducted a series of comprehensive experiments to evaluate the effectiveness and competitiveness of our proposed model compared to SOTA models and methods. These experiments address the following research questions: [RQ1] How does FiGURe, perform compared to SOTA unsupervised models? [RQ2] Can we perform satisfactorily even with lower dimensional representations using projections such as RFF? [RQ3] Does shared encoder decrease performance? [RQ4] What is the computational efficiency gained by using lower dimensional representations compared to methods that rely on higher dimensional representations? [RQ5] Can alternative filter banks be employed to

recover good quality representations? [RQ6] How does FiGURe combine information from different filters (in F GPRGNN) in a task-dependent manner?

Datasets and Setup: We evaluated our model on a diverse set of real-world datasets, which include both heterophilic and homophilic networks, to assess its effectiveness. Similar to previous works, we utilized the node classification task as a proxy to evaluate the quality of the learned representations. Please refer to 8.2 for detailed information about the benchmark datasets.

Baselines: In our comparison against baselines, we considered common unsupervised approaches, such as DEEPWALK and NODE2VEC, and state-of-the-art mutual information-based methods, namely DGI, MVGRL, GRACE, and SUGRL. We also include the performance numbers of the widely used GCN for reference. Unless stated otherwise, we use a 512-dimensional representation size for all reported results, following prior research. For additional details please refer to 8.3 and 8.4.

6.1 RQ1: FiGURe versus SOTA Methods

We analyzed the results in Table 1 and made important observations. Across homophilic and heterophilic datasets, F i GUR e 512 consistently outperforms several SOTA unsupervised models, except in a few cases where it achieves comparable performance. Even on the large-scale datasets ARXIVYEAR and OGBN-ARXIV FiGURe 512 performs well, demonstrating the scalability of our method. Two baseline methods MVGRL and GRACE run into memory issues on the larger datasets and are accordingly reported OOM in the table. We want to emphasize the rightmost column of the table, which shows the average percentage gain in performance across all datasets. This metric compares the improvement that FiGURe 512 provides over each baseline model for each dataset and averages these improvements. This metric highlights that FiGURe 512 performs consistently well across diverse datasets. No other baseline model achieves the same consistent performance across all datasets as FiGURe 512 . Even the recent state-of-the-art contrastive models GRACE and SUGRL experience

average performance drops of approximately 5 %and 10 %, respectively. This result indicates that FiGURe 512 learns representations that exhibit high generalization and task-agnostic capabilities. Another important observation is the effectiveness of RFF projections in improving lower dimensional representations. We compared FiGURe at different dimensions, including FiGURe RFF 32 and FiGURe RFF 128 , corresponding to learning 32 and 128 -dimensional embeddings, respectively, in addition to the baseline representation size of 512 dimensions. Remarkably, even at lower dimensions, FiGURe with RFF projections demonstrates competitive performance across datasets, surpassing the 512 -dimensional baselines in several cases. This result highlights the effectiveness of RFFprojections in enhancing the quality of lower dimensional representations. Using lower-dimensional embeddings reduces the computation time and makes FiGURe faster than the baselines. The computational efficiency of reducing dimension size becomes more significant with larger datasets, as evidenced in Table 3. On ARXIV-YEAR, a large graph with 169 , 343 nodes, 128-dimensional embeddings yield a 1.6x speedup, and 32-dimensional embeddings yield a 1.7x speedup. Similar results are observed in OGBN-ARXIV. For further insights into the effectiveness of RFFprojections, see Section 6.2, and for computational efficiency gains, refer to Section 6.4. In Table 2, we include GCN as a benchmark for comparison. Remarkably, FiGURe 512 remains competitive across most datasets, sometimes even surpassing GCN. This highlights that FiGURe 512 can capture task-specific information required by downstream tasks, typically handled by GCN, through unsupervised means. When considering a downstream task, using FiGURe 512 embeddings allows for the use of computationally efficient models like Logistic Regression, as opposed to training resource-intensive end-to-end graph neural networks. There are, however, works such as [6], [13], and [5], that explore methods to speed up end-to-end graph neural network training. In summary, FiGURe offers significant computational efficiency advantages over end-to-end supervised graph neural networks. Performance can potentially improve further by incorporating non-linear models like MLP. For detailed comparisons with other supervised methods, please refer to 8.4. It's noteworthy that both OGBN-ARXIV and ARXIV-YEAR use the ARXIV citation network but differ in label prediction tasks (subject area and publication year, respectively). FiGURe demonstrates improvements in both cases, showcasing its task-agnostic and multi-task capabilities due to its flexible node representation. This flexibility enables diverse tasks to extract the most relevant information (see Section 5.2), resulting in strong overall performance. Note that in 8.5, we have also performed an ablation study where the depth of the encoder is increased.

6.2 RQ2: RFF Projections on Lower Dimensional Representations

In this section, we analyse the performance of unsupervised baselines using 32 -dimensional embeddings with and without RFF projections (see Table 4). Despite extensive hyperparameter tuning, we could not replicate the results reported by SUGRL, so we present the best results we obtained. Two noteworthy observations emerge from these tables. Firstly, it is evident that lower dimensional embeddings can yield meaningful and linearly separable representations when combined with simple RFF projections. Utilising RFF projections enhances performance in almost all cases, highlighting the value captured by MI-based methods even with lower-dimensional embeddings. Secondly, FiGURe RFF 32 consistently achieves superior or comparable performance to the baselines, even in lower dimensions. Notably, this includes SUGRL, purported to excel in such settings. However, there is a 2 -

3 %performance gap between GRACE and our method for the SQUIRREL and CHAMELEON datasets. While GRACE handles heterophily well at lower dimensions, its performance deteriorates with homophilic graphs, unlike FiGURe RFF 32 which captures lower frequency information effectively. Additionally, our method exhibits computational efficiency advantages for specific datasets in lower dimensions. Please refer to 8.6: for discussions with regards to the RFF algorithm, 8.7: for analysing the RFF behaviour and community structure, 8.8: for experiments using other random projection methods, 8.9 and 8.10: for ablation studies with regards to projecting to higher dimensional spaces via RFF, and 8.11: for issues related to including RFF in training. Overall, these findings highlight the potential of RFF projections in extracting useful information from lower dimensional embeddings and reaffirm the competitiveness of FiGURe over the baselines.

6.3 RQ3: Sharing Weights Across Filter Specific Encoders

Our method proposes to reduce the computational load by sharing the encoder weights across all filters. It stands to reason whether sharing these weights causes any degradation in performance. We present the results with shared and independent encoders across the filters in Table 5 to verify this. We hypothesize that, sharing encoder weights embeds diverse filter representations in a common space, improving suitability for combined representation learning. This enhances features for downstream tasks, in some cases boosting performance. Experimental results confirm that shared weights do not significantly reduce performance; sometimes, they even enhance it, highlighting shared encoders' effectiveness in reducing computational load without sacrificing performance.

6.4 RQ4: Computational Efficiency

To assess the computational efficiency of the different methods, we analyzed the computation time and summarized the results in Table 6. The key metric used in this analysis is the mean epoch time: the average time taken to complete one epoch of training. We compared our method with other MI based methods such as DGI and MVGRL. Due to the increase in the number of augmentation views, there is an expected increase in computation time from DGI to MVGRL to FiGURe. However, as demonstrated in 6.2, using RFF projections allows us to achieve competitive performance even at lower dimensions. Therefore, we also included comparisons with our method at 128 and 32 dimensions in the table. It is evident from the results that our method, both at 128 and 32 dimensions, exhibits faster computation times compared to both DGI and MVGRL, which rely on higherdimensional representations to achieve good performance. This result indicates that FiGURe is computationally efficient due to its ability to work with lower-dimensional representations. During training, our method, F i GUR e RFF 32 , is ∼ 3 x faster than DGI and ∼ 6 x times faster than MVGRL. Despite the faster computation, F i GUR e RFF 32 also exhibits an average performance improvement of around 2 %across the datasets over all methods considered in our experiments. Please refer to 8.12 and 8.13 for further discussions.

6.5 RQ5: Experiments on Other Filter Banks

To showcase the versatility of our proposed framework, we conducted an experiment using Bernstein and Chebyshev filters, as detailed in Table 7. The results indicate that using F GPRGNN leads to better performance than BERNNET and CHEBNET filters. We believe this is happening is due to the latent characteristics of the dataset. [17, 28] have shown that datasets like CHAMELEON and SQUIRREL need frequency response functions that give more prominence to the tail-end spectrum. GPRGNN filters are more amenable to these needs, as demonstrated in [28]. However, different datasets may require other frequency response shapes, where BERNNET and CHEBNET filters may excel, and give better performance. For instance, F BERNNET may better approximate comb filters, as their basis gives uniform prominence to the entire spectrum. Our framework is designed to accommodate any filter bank, catering to diverse dataset needs. Further discussions are in 8.14.

6.6 RQ6: Combining information from different filters (in F GPRGNN)

To analyze how FiGURe combines representations from different filters, we present alpha coefficients for the highest-performing split in Table 8, utilizing F GPRGNN on: CORA, CITESEER, SQUIRREL, and CHAMELEON (coefficients may vary for different splits within the same dataset) [28]. GPRGNN filters adapt well to heterophilic datasets like CHAMELEON and SQUIRREL, emphasizing spectral tails with significant weightage on the A 2 filter. In contrast, homophilic datasets require low-pass filters, achieved by assigning higher weightage to the A and A 3 filters. Achieving these filter shapes with other methods like BERNNET and CHEBNET is possible but more challenging for the model to learn. We believe that this is why GPRGNN filters consistently outperform other filter banks on the datasets we've examined. However, it's important to note that some datasets may benefit from different response shapes, where BERNNET and CHEBNET filters might be more suitable. This table validates our hypothesis about the efficacy of GPRGNN filter coefficients, with A 3 dominating in homophilic datasets (CORA , CITESEER) and A 2 in heterophilic datasets (CHAMELEON , SQUIRREL).

7 Conclusion and Future Work

Our work demonstrates the benefits of enhancing contrastive learning methods with filter views and learning filter-specific representations to cater to diverse tasks from homophily to heterophily. We have effectively alleviated computational and storage burdens by sharing the encoder across these filters and focusing on low-dimensional embeddings that utilize high-dimensional projections, a technique inspired by random feature maps developed for kernel approximations. Future directions involve expanding the analysis from [2] to graph contrastive learning and investigating linear separability in lower dimensions, which could strengthen the connection to the random feature maps approach.

References

T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In International conference on knowledge discovery & data mining (KDD) , pages 2623-2631, 2019.

S. Arora, H. Khandeparkar, M. Khodak, O. Plevrakis, and N. Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning (ICML) , 2019.

K. Atarashi, S. Maji, and S. Oyama. Random feature maps for the itemset kernel. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 3199-3206, 2019.

D. Bo, X. Wang, C. Shi, and H.-W. Shen. Beyond low-frequency information in graph convolutional networks. In Association for the Advancement of Artificial Intelligence (AAAI) , 2021.

A. Bojchevski, J. Gasteiger, B. Perozzi, A. Kapoor, M. Blais, B. Rózemberczki, M. Lukasik, and S. Günnemann. Scaling graph neural networks with approximate pagerank. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , KDD '20, page 2464-2473, New York, NY, USA, 2020. Association for Computing Machinery.

J. Chen, T. Ma, and C. Xiao. FastGCN: Fast learning with graph convolutional networks via importance sampling. In International Conference on Learning Representations , 2018.

E. Chien, J. Peng, P. Li, and O. Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations (ICLR) , 2021.

K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger, L. J. Colwell, and A. Weller. Rethinking attention with performers. In International Conference on Learning Representations , 2021.

M. D. Donsker and S. R. S. Varadhan. Asymptotic evaluation of certain markov process expectations for large time. In Communications on Pure and Applied Mathematics , 1975.

C. Dugas, Y. Bengio, F. Bélisle, C. Nadeau, and R. Garcia. Incorporating second-order functional knowledge for better option pricing. In Neural Information Processing Systems (NeurIPS) , 2000.

J. I. T. Falk, C. Ciliberto, and massimiliano pontil. Implicit kernel meta-learning using kernel integral forms. In The 38th Conference on Uncertainty in Artificial Intelligence , 2022.

K. Fang, F. Liu, X. Huang, and J. Yang. End-to-end kernel learning via generative random fourier features. Pattern Recognition , 134:109057, 2023.

F. Frasca, E. Rossi, D. Eynard, B. Chamberlain, M. Bronstein, and F. Monti. SIGN: Scalable Inception Graph Neural Networks. In ICML Workshop on Graph Representation Learning and Beyond , 2020.

A. Grover and J. Leskovec. node2vec: Scalable feature learning for networks. In International Conference on Knowledge Discovery and Data Mining (KDD) , 2016.

K. Hassani and A. H. Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning (ICML) , 2020.

K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV) , 2015.

M. He, Z. Wei, Z. Huang, and H. Xu. Bernnet: Learning arbitrary graph spectral filters via bernstein approximation. In Neural Information Processing Systems (NeurIPS) , 2022.

R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations (ICLR) , 2019.

W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Neural Information Processing Systems (NeurIPS) , 2020.

P. Kar and H. Karnick. Random feature maps for dot product kernels. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2012.

P. Kar and H. Karnick. Random feature maps for dot product kernels. In N. D. Lawrence and M. A. Girolami, editors, Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2012, La Palma, Canary Islands, Spain, April 21-23, 2012 , volume 22 of JMLR Proceedings , pages 583-591. JMLR.org, 2012.

D. Kim and A. Oh. How to find your friendly neighborhood: Graph attention design with self-supervision. In International Conference on Learning Representations (ICLR) , 2021.

G. Kimeldorf and G. Wahba. Some results on tchebycheffian spline functions. Journal of Mathematical Analysis and Applications , 1971.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015.

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR) , 2017.

Z. Li, J.-F. Ton, D. Oglic, and D. Sejdinovic. Towards a unified analysis of random fourier features. Journal of Machine Learning Research (JMLR) , 2021.

D. Lim, F. M. Hohne, X. Li, S. L. Huang, V. Gupta, O. P. Bhalerao, and S.-N. Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. In Neural Information Processing Systems (NeurIPS) , 2021.

V. Lingam, C. Ekbote, M. Sharma, R. Ragesh, A. Iyer, and S. Sellamanickam. A piece-wise polynomial filtering approach for graph neural networks. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD) , 2022.

R. Linsker. Self-organization in a perceptual network. Computer , 1988.

J. Mercer. Functions of positive and negative type, and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character , 1909.

Y. Mo, L. Peng, J. Xu, X. Shi, and X. Zhu. Simple unsupervised graph representation learning. In Association for the Advancement of Artificial Intelligence (AAAI) , 2022.

S. Nowozin, B. Cseke, and R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Neural Information Processing Systems (NeurIPS) , 2016.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems (NeurIPS) , 2019.

H. Pei, B. Wei, K. C.-C. Chang, Y. Lei, and B. Yang. Geom-gcn: Geometric graph convolutional networks. In International Conference on Learning Representations (ICLR) , 2020.

H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. Smith, and L. Kong. Random feature attention. In International Conference on Learning Representations , 2021.

J. Pennington, F. X. X. Yu, and S. Kumar. Spherical random features for polynomial kernels. In Neural Information Processing Systems (NeurIPS) , 2015.

B. Perozzi, R. Al-Rfou, and S. Skiena. DeepWalk. In International Conference on Knowledge Discovery and Data Mining (KDD) , 2014.

O. Platonov, D. Kuznedelev, M. Diskin, A. Babenko, and L. Prokhorenkova. A critical look at the evaluation of GNNs under heterophily: Are we really making progress? In International Conference on Learning Representations (ICLR) , 2023.

Z. Qin, W. Sun, H. Deng, D. Li, Y. Wei, B. Lv, J. Yan, L. Kong, and Y. Zhong. cosformer: Rethinking softmax in attention. In International Conference on Learning Representations , 2022.

A. Rahimi and B. Recht. Random features for large-scale kernel machines. In Neural Information Processing Systems (NeurIPS) , 2007.

A. Rahimi and B. Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Neural Information Processing Systems (NeurIPS) , 2008.

D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE Signal Processing Magazine , 2013.

N. Tremblay, P. Gonçalves, and P. Borgnat. Design of graph filters and filterbanks. Cooperative and Graph Signal Processing , 2017.

P. Veliˇ ckovi´ c, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm. Deep graph infomax. In International Conference on Learning Representations (ICLR) , 2019.

Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang. Deep Graph Contrastive Representation Learning. In ICML Workshop on Graph Representation Learning and Beyond , 2020.

8 Supplementary Material

Contents

8.1 Reproducibility

We strive to ensure the reproducibility of our research findings. To facilitate this, we provide the details of our experimental setup, including dataset sources, preprocessing steps, hyperparameters, and model configurations. We also make our code and the datasets used, publicly available at https://github.com/microsoft/figure , enabling researchers to reproduce our results and build upon our work. We would like to emphasize that our code is built on top of the existing MVGRL codebase. For the datasets used in our evaluation, we provide references to their original sources and any specific data splits that we employed. This allows others to obtain the same datasets and perform their own analyses using consistent data. Additionally, we specify the versions of libraries and frameworks used in our experiments, in Section 8.3, and in the REQUIREMENTS file and the README file, in the codebase, enabling others to set up a compatible environment. We document any specific seed values or randomization procedures that may affect the results. By providing these details and resources, we aim to promote transparency and reproducibility in scientific research. We encourage fellow researchers to reach out to us if they have any questions or need further clarification on our methods or results.

8.2 Datasets

Homophilic Datasets: We evaluated our model (as well as baselines) on three homophilic datasets: CORA, CITESEER, and PUBMED as borrowed from [34]. All three are citation networks, where each node represents a research paper and the links represent citations. Pubmed consists of medical research papers. The task is to predict the category of the research paper. We follow the same dataset setup mentioned in [34] to create 10 random splits for each of these datasets.

Heterophilic Datasets: In our evaluation, we included four heterophilic datasets: CHAMELEON, SQUIRREL, ROMAN-EMPIRE, and MINESWEEPER. For CHAMELEON and SQUIRREL, nodes represent Wikipedia web pages and edges capture mutual links between pages. We utilized the ten random splits provided in [34], where 48 %, 32 %, and 20 %of the nodes were allocated for the train, validation, and test sets, respectively. In ROMAN-EMPIRE each node corresponds to a word in the Roman Empire Wikipedia article. Two words are connected with an edge if either these words follow each other in the text, or they are connected in the dependency tree of the sentence. The syntactic role of the word/node defines its class label. The MINESWEEPER graph is a regular 100x100 grid where each node is connected to eight neighboring nodes, and the features are on-hot encoded representations of the number of neighboring mines. The task is to predict which nodes are mines. For both ROMAN-EMPIRE and MINESWEEPER, we used the ten random splits provided in [38].

Large Datasets: We also evaluate our method on two large datasets OGBN-ARXIV (from [19]) and ARXIV-YEAR (from [27]). Both these datasets are from the arxiv citation network. In OGBNARXIV, the task is to predict the category of the research paper, and in ARXIV-YEAR the task is to predict the year of publication. We use the publicly available splits for OGBN-ARXIV [22] and follow the same dataset setup mentioned in [27] to generate 5 random splits for ARXIV-YEAR. Note that OGBN-ARXIV is a homophilic dataset while ARXIV-YEAR is a heterophilic datasets.

The detailed dataset statistics can be found in Table 9.

8.3 Training Details

We conducted all experiments on a machine equipped with an Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz processor, 440GB RAM, and a Tesla-P100 GPU with 16GB of memory. The experiments were executed using Python 3.9.12 and PyTorch 1.13.0 [33]. To optimize the hyperparameter search, we employed Optuna [1]. We utilized the Adam optimizer [24] for the optimization process.

8.3.1 Unsupervised Training

We conducted hyperparameter tuning for all unsupervised methods using 20 Optuna trials. The hyperparameter ranges and settings for each method are as follows:

DEEPWALK: We set the learning rate to 0 . 01 , number of epochs to 20 and the varied the random walk length over { 8 , 9 , 10 , 11 , 12 } . Additionally, we varied the context window size over { 3 , 4 , 5 } and the negative size (number of negative samples per positive sample) over { 4 , 5 , 6 } .

NODE2VEC: For Node2Vec, we set the learning rate to 0 . 01 and number of epochs to 100 . We varied the number of walks over { 5 , 10 , 15 } and the walk length over { 40 , 50 , 60 } . The p (return parameter) value was chosen from { 0 . 1 , 0 . 25 , 0 . 5 , 1 } and q (in-out parameter) value was chosen from { 3 , 4 , 5 } .

DGI: DGI [44] proposes a self-supervised learning framework for graph representation learning by maximizing the mutual information between local and global structural context of nodes, enabling unsupervised feature extraction in graph neural networks. We relied on the authors' code 2 and the prescribed hyperparameter ranges specific to the DGI model, for our experiments.

MVGRL: MVGRL [15] proposes a method for learning unsupervised node representations by leveraging two views of the graph data, the graph diffusion view and adjacency graph view. We relied on the authors' code 3 and the prescribed hyperparameter ranges specific to the MVGRL model, for our experiments.

GRACE: GRACE [45] proposes a technique where two different perspectives of the graph are created through corruption, and the learning process involves maximizing the consistency between the node representations obtained from these two views. We relied on the authors' code 4 and the prescribed hyperparameter ranges specific to the GRACE model, for our experiments.

SUGRL: SUGRL [31] proposes a technique for learning unsupervised representations which capture node proximity, while also utilising node feature information. We relied on the authors' code 5 and the prescribed hyperparameter ranges specific to the SUGRL model, for our experiments.

FiGURe: We followed the setting of the MVGRL model, setting the batch size to 2 and number of GCN layers to 1. We further tuned the learning rate over { 0 . 00001 , 0 . 0001 , 0 . 001 , 0 . 01 , 0 . 1 } and the sample size (number of nodes selected per batch) over { 1500 , 1750 , 2000 , 2250 } , except for the large graphs, for which we set the sample size to 5 , 000 .

We maintained consistency across all methods by employing identical parameters for maximum epochs and early stopping criteria. Specifically, we set the maximum number of epochs to 30 , 000 and utilized an early stopping patience of 20 epochs, with the exception of large datasets, where we extended the patience to 500 epochs.

In each case, we selected the hyperparameters that resulted in the lowest unsupervised training loss.

8.3.2 Supervised Training

For all unsupervised methods, including the baselines and our method, we perform post-training supervised evaluation using logistic regression with 60 Optuna trials. We set the maximum number of epochs to 10000 and select the epoch and hyperparameters that yield the best validation accuracy. The learning rate is swept over the range { 0 . 00001 , 0 . 0001 , 0 . 001 , 0 . 0015 , 0 . 01 , 0 . 015 , 0 . 1 , 0 . 5 , 1 , 2 } , and the weight decay is varied over { 10 -5 , 10 -4 , 10 -3 , 10 -2 , 10 -1 , 0 , 0 . 5 , 1 , 3 } .

FiGURe: Along with the hyperparameters described above, following the approach described in [17], we also tune the combination coefficients ( α i 's) with a separate learning rate. This separate learning rate is swept over the range { 0 . 00001 , 0 . 0001 , 0 . 001 , 0 . 0015 , 0 . 01 , 0 . 015 , 0 . 1 , 0 . 5 , 1 , 2 } . In addition, we have a coefficient for masking the incoming embeddings from each filter, which is varied between 0 and 1 . Furthermore, these coefficients are passed through an activation layer, and we have two options: 'none' and 'exp'. When 'none' is selected, the coefficients are used directly, while 'exp' indicates that they are passed through an exponential function before being used.

FiGURe with RFF: For the experiments involving Random Fourier Features (RFF), we use the same hyperparameter ranges as mentioned above. However, we also tune the gamma parameter which is specific to RFF projections. The gamma parameter is tuned within the range { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 , 1 . 0 , 1 . 1 , 1 . 2 } .

8.3.3 Negative Sampling for the Identity Filter

In our implementation of F GPRGNN or F BERNNET , we follow a specific procedure for handling the filters during training and evaluation. For all filters except the identity filter ( I ), we employ the negative sampling approach described in Section 6. However, the identity filter is treated differently. During training, we exclude the identity filter and only include it during evaluation.

During negative sampling, the generation of the negative anchor involves shuffling the node features, followed by premultiplying the shuffled node feature matrix with the filter matrix and computing the mean. On the other hand, for the positive anchor, the same procedure is applied without shuffling the node features. This approach encourages the model to learn meaningful patterns and relationships in the data when the filter matrix is not the identity matrix.

The decision to exclude the identity filter during training is based on the observation that it presents a special case where the positive and negative anchors become the same. As a result, the model would optimize and minimize the same quantity, potentially leading to trivial solutions. To prevent this, we exclude the identity filter during training.

By excluding the identity filter during training, we ensure that the model focuses on the other filters in F GPRGNN or F BERNNET to capture and leverage the diverse information present in the graph. Including the identity filter only during evaluation allows us to evaluate its contribution to the final performance of the model. This approach helps prevent the model from learning trivial solutions and ensures that it learns meaningful representations by leveraging the other filters.

8.4 Comparison with other Supervised Methods

Table 10 presents a comparison with common supervised baselines. Specifically, we choose 3 models for comparison, representing three different kinds of supervised methods, standard aggregation models (GCN), spectral filter-based models (GPRGNN) and smart-aggregation models (H 2 GCN). There are two key observations from this table. Firstly, FiGURe 512 is competitive with the supervised baselines, lagging behind only by a few percentage points in some cases. This suggests that much of the information that is required by the downstream tasks, captured by the supervised models, can be made available through unsupervised methods like FiGURe which uses filter banks. It is important to note that in FiGURe we only utilize logistic regression while evaluating on the downstream task. This is much more efficient that training a graph neural network end to end. Additionally it is possible that further gains may be obtained by utilizing a non-linear model like an MLP. Furthermore, as indicated by 10, we can gain further computational efficiency by utilizing lower dimensional representations like 32 and 128 (with RFF), and still not compromise significantly on the performance. Overall FiGURe manages to remain competitive despite not having access to task-specific labels and is computationally efficient as well.

8.5 Increasing the depth of the Encoder

We present an analysis in Table 11 featuring an increased number of encoder layers. In this context, "encoder layers" can be interpreted in two ways: firstly, as a deeper GCN which entails aggregating information from multiple-hop neighborhoods into the node; and secondly, as a single-hop GCN with a more extensive network for feature transformation. We provide performance results for both scenar-

ios, encompassing two and three layers each. It is noteworthy that the single-layer GCN achieves equal or superior performance compared to all other configurations.

8.6 RFF Projections

As shown in Section 6.2 and in Section 6.4, RFF projections are a computationally efficient way to achieve training by preserving the latent class behavior present in lower dimensional embeddings, by projecting them into a higher dimensional linearly separable space. The natural question that comes up is how do we compute these RFF projections? We provide an algorithm to compute the RFF projections in this section, in algorithm 1. Note that this follows [40].

8.7 Visualising RFF Behavior and Community Structure

As shown in prior sections, FiGURe improves on both computational efficiency as well as performance by utilising RFF projections. In this section, we aim to gain insights into the behavior of RFF projections and comprehend their underlying operations through a series of simple visualizations.

t-SNE Plots: Figure 3 offers insights into the structure of the embeddings for the CORA dataset across different dimensions. Remarkably, even at lower dimensions (e.g., 32 dimensions), clear class structures are discernible, indicating that the embeddings capture meaningful information related to the class labels. Furthermore, when employing RFF to project the embeddings into higher dimensions, these distinct class structures are still preserved. This suggests that the role of RFF is not to introduce new information, but rather to enhance the suitability of lower-dimensional embeddings

for linear classifiers while maintaining the underlying class-related information. Notably, even at 512 dimensions, the class structures remain distinguishable. However, it is worth noting that the class-specific embeddings appear to be more tightly clustered and less dispersed compared to the 32 -dimensional embeddings or the projected 32 -dimensional embeddings. This suggests that learning a 512 -dimensional embedding differs inherently from learning a 32 -dimensional embedding and subsequently projecting it into higher dimensions.

(a) F i GUR e 32 (without RFF) on the CORA dataset

(b) F

i

GUR

e

RFF

32

(with RFF) on

the CORA dataset

(c) F i GUR e 512 (without RFF) on the CORA dataset

t-SNE | cora ∣ Dim: 32

t-SNE RFF | cora ∣ Dim: 32

t-SNE | cora | Dim: 512

s|xe

-20

20

60

40

0

Class-2

Class-0

Class-1

Class-3

Class-4

Class-5

Class-6

sixe

20

80

60

40

20

0

Class-3

Class-0

Class-1

Class-2

Class-4

Class-5

Class-6

sIxe

50

Class-2

Class-4

Class-5

Class-6

Class-0

Class-1

Class-3

-40

=40

-60

60

50

08=

50

0S

80

50

0

S0

50

50

X Axis

X Axis

X Axis

Correlation Plots: Figure 4 offers insights into the correlation patterns within the embeddings generated from the SQUIRREL dataset across different dimensions. In lower dimensions, the embeddings exhibit high correlation with each other, which can be attributed to the presence of a mixture of topics or latent classes within the dataset. However, when the embeddings are projected to higher dimensions using RFF, the correlation is reduced, and a block diagonal matrix emerges. This block diagonal structure indicates the presence of distinct classes or communities within the dataset. Even at 512 dimensions, a more refined block diagonal structure can be observed compared to the correlation matrix of the 32 -dimensional embeddings. Furthermore, it is noteworthy that the correlation of the projected embeddings can be regarded as a sparser version of the correlation observed in the 512 -dimensional embeddings.

(a) F i GUR e 32 (without RFF) on the SQUIRREL dataset

(b) F

i

GUR

e

RFF

32

(with RFF) on

(c) F i GUR e 512 (without RFF) on the SQUIRREL dataset

5000

Correlation Matrix | squirrel | Dim: 32

1.00

5000

Correlation Matrix RFF | squirrel | Dim: 512

1.0

5000

Correlation Matrix | squirrel | Dim: 512

1.0

- 0.95

- 0.8

0.8

4000

4000

4000

- 0.90

0.6

0.6

Axis

>

2000

3000

- 0.80

0.85

Y Axis

2000

3000

- 0.4

- 0.2

Axis

2000

3000

- 0.4

- 0.2

1000

- 0.75

1000

- 0.0

1000

- 0.0

0

0

1000

2000

3000

4000

5000

0

1000

2000

X Axis

3000

4000

5000

0.2

0

1000

2000

X Axis

3000

4000

5000

0.2

8.8 Other Random Projections

We conducted an analysis exploring the impact of utilizing random projections beyond RFF in Table 12. Our findings reveal that RFF outperforms alternative random projection methods.

8.9 RFF ablation study at 128 dimensions

We replicate the analysis conducted in 6.2 for 128 dimensions, as outlined in Table 13. It is apparent that our approach consistently outperforms or matches most other methods when operating at 32 dimensions across various datasets. This trend persists even after projecting the learned embeddings to higher dimensions through RFF. Our experiment reaffirms the observations detailed in 6.2.

8.10 RFF ablation study at 512 dimensions

We present the results obtained by applying RFF to 512 dimensions, as shown in Table 14. It is evident that there is minimal performance improvement when using RFF on embeddings that are already of sufficient dimensionality.

8.11 Issues with including RFF in training

In the context of incorporating Random Feature techniques like RFF into machine learning models, some challenges have arisen in the training process. In the Transformer attention module, these features can exhibit large variances in zero kernel score regions, leading to unstable training [8]. Additionally, increasing the sampling rate in methods like Performer and Random Feature Attention (RFA) [35] can also introduce instability, as highlighted in [39]. Another area where the use of random features with deep networks has been explored is kernel learning. There are of course, computational and convergence challenges that have been observed with these methods as well [11][12]. FiGURe avoids these problems by excluding the random feature generation from training altogether. Parameters like gamma, involved in the random feature construction, are treated as hyperparameters and a search is performed over them. Kernel learning, however, is an important area of research, and integrating RFF in the training loop would be an interesting extension of this work.

8.12 Computational Comparisons with other Other Unsupervised Methods

In Section 6.4, we compared the computational time of FiGURe with MVGRL and DGI, as all three methods fall under the category of unsupervised methods that preform contrastive learning with representations of the entire graph. However, there is another class of methods, such as SUGRL and GRACE, that contrast against other nodes without the need for graph representation computation. Consequently, these methods exhibit higher computational efficiency. Hence, as show in Table 15 upon initial inspection, it appears that SUGRL (at 512 dimensions) exhibits the highest computational efficiency, even outperforming F i GUR e RFF 128 . However, despite its computational efficiency, the significant drop in performance across datasets (as discussed in Section 6.1) renders it less favorable for consideration. In fact, F i GUR e RFF 32 offers computational cost savings compared to SUGRL, while also achieving significantly better downstream classification accuracy. Turning to GRACE, it demonstrates greater computational efficiency than FiGURe 512 for low to medium-sized graphs. However, as the graph size increases, due to random node feature level masking and edge level masking, the computational requirements of GRACE substantially increase (as evidenced by the results on SQUIRREL). Therefore, for larger graphs with more than approximately 5 , 000 nodes, FiGURe proves to be more computationally efficient than GRACE (even at 512 dimensions). Furthermore, considering the performance improvements exhibited by FiGURe, it is evident that FiGURe (combined with RFF projections) emerges as the preferred method for unsupervised contrastive learning in graph data.

8.13 Time and storage cost

In Table 16, we present a comparison of training time and storage cost between our method and DGI. To ensure fairness in the evaluation, we maintained consistent settings across all baselines. The maximum number of epochs was uniformly set to 30 , 000 , with an early stopping patience of 20 epochs, except for larger datasets where we extended the patience to 500 epochs. It's important to note that due to varying hyperparameter tuning among the baselines, normalizing the number of epochs posed some challenges. Nevertheless, we ensured that a sufficiently high number of epochs were employed to facilitate convergence. Our approach builds upon DGI and exhibits linear scaling of training time with the number of filters used. Currently, our model employs three filters for training, resulting in a training time three times longer than that of DGI. Specifically for OGBN-ARXIV, we provide information on the number of epochs, mean epoch time, and total training time. Additionally, we report the storage cost of representations generated by these methods. Please keep in mind that the linear relationship in training time between DGI and FiGURe is not apparent here, as it factors in considerations such as batching and sampling time.

8.14 Choice of Filter Banks

In Section 4.1, we explore the flexibility of FiGURe to accommodate various filter banks. When making a choice, it is crucial to examine the intrinsic properties of the filters contained within different filter banks. We pick two filter banks F BERNNET and F GPRGNN and provide an overview of the filters contained in the filter banks. We use these two filter banks as examples to illustrate what should one be looking for, while choosing a filter bank.

Bernstein Polynomials : Figure 5 illustrates that as the number of Bernstein Basis increases, the focus on different parts of the eigenspectrum also undergoes changes. With an increase in polynomial order, two notable effects can be observed. Firstly, the number of filters increases, enabling each filter to focus on more fine-grained eigenvalues. This expanded set of polynomial filters allows for a more detailed examination of the eigenspectrum. Secondly, if we examine the first and last Bernstein polynomials, we observe an outward shift in their shape. This shift results in the enhancement of a specific fine-grained part at the ends of the spectrum. These observations demonstrate that Bernstein polynomials offer the capability to selectively target and enhance specific regions of interest within the eigenspectrum

Standard Basis : Figure 5 reveals two key observations. Firstly, at a polynomial order of 2 , the standard basis exhibit focus at the ends of the spectrum, in contrast to the behavior of Bernstein polynomials, which tend to concentrate more on the middle of the eigenspectrum. This discrepancy highlights the distinct characteristics and emphasis of different polynomial bases in capturing different parts of the eigenspectrum. Secondly, as the number of polynomials increases (in contrast to Bernstein polynomials), the lower order polynomials remain relatively unchanged. Instead, additional polynomials are introduced, offering a more fine-grained focus at the ends of the spectrum. This expansion of polynomials allows for a more detailed exploration of specific regions of interest within the the ends of eigenspectrum.

In the context of filter banks, previous studies [28, 7] have demonstrated that certain datasets, such as SQUIRREL and CHAMELEON, benefit from frequency response functions that enhance the tail ends of the eigenspectrum. This observation suggests that the standard basis, which naturally focuses on the ends of the spectrum, may outperform Bernstein basis functions at lower orders. However, as the order of the Bernstein basis increases, as discussed in 4.1, there is a notable improvement in performance. This can be attributed to the increased focus of Bernstein basis functions on specific regions, particularly the ends of the spectrum. As a result, higher-order Bernstein filters exhibit enhanced capability in capturing important information in those regions. It is worth noting that the choice between F GPRGNN and F BERNNET depends on the specific requirements of the downstream task. If the task necessitates a stronger focus on the middle of the spectrum or requires a band-pass or comb-like frequency response, F BERNNET is likely to outperform F GPRGNN. Thus, the selection of the appropriate filter bank should be based on the desired emphasis on different parts of the eigenspectrum. Regarding the performance comparison between F BERNNET and F GPRGNN, it is plausible that as we increase the order of the Bernstein basis, the performance could potentially match that of F GPRGNN. However, further investigation and experimentation are required to determine the specific conditions and orders at which this convergence in performance occurs.

(a) Five Bernstein Basis

(b) Five Standard Basis

(c) Seven Bernstein Basis

(d) Seven Standard Basis

(e) Nine Bernstein Basis

(f) Nine Standard Basis

Bernstein Polynomials

Poly-0

Poly-1

Poly-2

0.8

Poly-4

Poly-3

Magnitude

0.6

0.4

0.2

0

0.5

1

1.5

2

Elgenvalues

Bernstein Polynomials

Poly-0

Poly-1

Poly-2

0.8

Poly-3

Poly-4

Poly-5

Magnitude

0.6

Poly-6

0.4

0.2

0

0.5

1

1.5

Eigenvalues

Bernstein Polynomials

Poly-0

Poly-1

Poly-2

8°0

Poly-3

Poly-4

Poly-5

0.6

Poly-6

Poly-7

Poly-8

0.4

0.2

0.5

1

1.5

Elgenvalues

Bernstein Polynomials

Poly-0

Poly-1

Poly-2

0.8

Poly-3

Poly-4

Poly-5

Magnitude

0.6

Poly-6

Poly-7

Poly-8

0.4

Poly-9

Poly-10

0.2

0.5

1

1.5

Eigenvalues

Standard Polynomials

Poly-0

Poly-1

Poly-2

0.5

Poly-3

Poly-4

Magnitude

0.5

-1

0

0.5

1

1.5

Elgenvalues

Standard Polynomials

Poly-0

Poly-1

Poly-2

Poly-3

0.5

Poly-4

Poly-5

Magnitude

Poly-6

0.5

0.5

1

1.5

Eigenvalues

Standard Polynomials

Poly-0

Poly-1

Poly-2

Poly-3

0.5

Poly-4

Poly-5

Poly-6

Poly-7

Poly-8

0.5

0

0.5

1

1.5

Elgenvalues

Standard Polynomials

Poly-0

Poly-1

Poly-2

Poly-3

0.5

Poly-4

Poly-5

Magnitude

Poly-6

Poly-7

Poly-8

Poly-9

Poly-10

0.5

U

0.5

1

1.5

Eigenvalues

(g) Eleven Bernstein Basis

(h) Eleven Standard Basis

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposer proposed a contrastive learning method for graphs. The goal is to learn representations for nodes in an unsupervised way, by maximizing mutual informations between the local representation and global representation of a graph, training a single encoder to learn node representations. The method proposes to use multiple different filters to learn representations with different emphasis on the graph structure (as seen through the spectral representation), and then combine those representations on downstream tasks, by learning the mixing coefficients in addition to a linear classifier.\n\nThe proposed method, FIGURe, being slower than the other methods it is compared to, when learning 512 representations, the authors also explore ways to produce more compact representations, 32 and 128 in width, using RFF to recover some of the lost accuracy on downstream tasks.\n\nSTRENGTHS:\n* The results show improvement compared to other self-supervised approaches. Although FIGURe does not get the first place on every single benchmark, it performs very consistency across datasets, leading to the highest average performance.\n* The versions of FIGURe with smaller representations are leading to good results as well, underlying the soundness of the approach followed by the authors to reduce the dimension of representation and then recovering part of the lost accuracy.\n* Overall, the paper is well written, and easy to follow. The preliminary section is concise and yet very useful.\n\nWEAKNESSES:\n* The study of the authors is limited to 1-layer GCN. It would be interesting to know how the method compares to others as the GNN architecture gets bigger (in particular deeper).\n* The authors justify the need for learning representations of smaller sizes due to the prohibitive cost of contrastive learning, but this part lacks justification in my view: the mean epoch time does indeed increase compared to baselines (+50%), but we are missing details in the paper related to how many epochs it is trained on (in absolute and compared to others methods) - this would help justify the need for the lower dimensionality version and the RFF tricks used afterwards\n* The authors present the shared encoder as an efficiency approach, but it seems from Table 4 that it is also increasing performance.\n* In general, I would appreciate having more details on the experimental section, and in particular on the training recipe and the evaluation recipes - some of these questions will be asked in the \"Questions\" sections of this review - as this would help understanding if the comparison are fair (and help reproducing results)\n\nQUESTIONS:\nHere are a bunch of details that I think need clarification in the experimental sections:\n* How long is the method trained compared to the baselines ? Does the number of filter influence the training time (linearly?).\n* How many filters does the method used compare to the baselines ? If it uses more, should we normalize those to make sure comparisons are fair ?\n* How are the baselines evaluated ? Are those also using the learnable alpha mixing coefficients ? Are those baselines using the same number of filters ? If not, should we normalize those to make sure the evaluations are fair ?\n* What happens if we use RFF on the 512 dimensions representations ? In particular, on the baselines ?\n* What happens if we do not use RFF on the 32 dimensions and 128 dimensions representations ?\n\nIn several sections, the authors talk about the cost of having high number of dimensions and the cost of having several encoder instead of one shared encoder, would it be possible to put numbers on these costs in terms of time / storage ? It would help convey the need for the dimensionality reduction and RFF tricks.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a model and contrastive learrning method for acquiring a comprehensive specturm of graph representation by employing filters of various levels. They are adaptively aggregated with learnable weights for downstream supervision tasks. As a result, this approach performs well on both homophilic and heterophilic graphs. Additionally, by utilizing a random featrue map for kernel approximation, it demonstrates effectiveness with low computational cost.  They employ a shared encoder for multiple filters, which not only reduces computational load compared to using independent filters for each, but also improves performane in some cases. Experimental results exhibit performance improvement on average.\n\nSTRENGTHS:\nThey conducted experiments on data with multiple characteristics, demonstrating that their approach works for both homophilic and heterophilic graphs. They proposed various methods for computational efficiency and provide evidence of the effectiveness through experiments. The contribution of their work is clearly described through abtract and introduction sections.\n\nWEAKNESSES:\nThis paper requires some improvements in terms of presentation, as it contains inconsistent notation (such as size of tilde and hat in the line 147 and 149) and deferred explanation for newly introduced symbols that is difficult to follow and numerous mathematical errors. The presence of these mathematical errors prompted a desire to check the implementation, which unfortunately was not provided by the authors. If these issues are addressed in the rebuttal and the necessary improvements are made, it would be considered for increasing score. It is necessary to clarify the notation and provide better explanations to ensure that readers can follow the content more easily.\n\nFor example, in Section 5.1, the notation $D_w$ is introduced for the first time on line 149, while the explanation for it appears much later, specifically on line 166.\n\nAdditionally, there seem to be errors in the equations. In Equation 1, the correct formulation of the MI estimator should be:\n\n$E_{P_i}[-sp(-T)] - E_{P_i \\times \\tilde{P}_i}[sp(T)]$.  (details are omitted here. Just refer the ‘-‘ sign) \n(7th slide in [this lecture](http://people.ee.duke.edu/~lcarin/Jiachang3.20.2020.pdf))\n\nThis formulation intuitively maximizes MI, as the first term increases the similarity between local and global features obtained from the same graph, while the second term decreases the similarity between local and global features obtained from different graphs. Similar errors can be observed in Equation 3, where minimizing the given objective $\\mathcal{L}_{F_i}$ seems to result in learning that decreases the similarity between local and global representations from the same graph, particularly in the first term. Moreover, in the second term of Equation 3, only the corrupted one $(\\tilde{X}_i, \\tilde{F}_i)$ is considered for expectation and $h_g^{F_i}$ is not covered, which, leads to ambiguous representation. Furthermore, in Section 6, $\\sigma$ on lines 227, 233, and in Equation (6) refer to different functions, causing confusion.\n\nOverall, it is crucial to address these issues related to notation, mathematical accuracy, and clarity throughout the paper to enhance its quality and ensure a better understanding for readers.\n\nQUESTIONS:\nIn Section 6.5, the authors conducted ablation studies for filters, and it was observed that GPRGNN performed better in all cases. However, it would be informative to explore if there are any other options that demonstrate superior performance. This could involve examining alternative filter configurations or considering different methods altogether. Providing such insights would enhance the comprehensiveness of the analysis and strengthen the paper's contribution. Furthermore, analyzing the learned weight $\\alpha$ or  TSNE plots of learned representations with different filters would also be interesting.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a contrastive learning model for learning node embeddings on a graph, with two technical innovations: First, the authors propose a new augmentation scheme during contrastive learning. Secondly, the authors re-map high dimensional embeddings into lower dimensional space using random Fourier features. The authors claim improvements attributed to both technical innovations on a variety of datasets.\n\nSTRENGTHS:\nPresentation if the paper is mostly clear, figures are well readable, and results are clearly presented.\n\nWEAKNESSES:\n- Although the word \"augmentation\" is very present in both the paper title and abstract (6 occurences), it only appears once in the paper. It is not fully clear how the abstract claims map onto the technique in the paper, beyond the preliminary info in Section 4.1. The terms should be used more consistently here. It is somewhat inferable from 4.1 what the \"augmentations\" are, but this needs to be clarified in Section 5.\n- Presentation in section 5 is a bit unclear to me. The algorithm proposes two innovations (filter augmentations and random Fourier features), but the experiment section is not grouped accordingly. I would propose to split the content in 5.1. and 5.2. differently to aid the understanding of the different algorithmic components and their implementation.\n- l. 183 claims \"Maximizing JS is equiv. to reducing BCE\" --- can you add a suitable reference for this claim?\n- **Baselines**: I checked a few of the reported baselines numbers. For instance, [SUGRL (Table 1)](https://ojs.aaai.org/index.php/AAAI/article/view/20748) reports $83.5 \\pm .5$ on Cora vs. $81.21 \\pm 2.07$ here, $73.0 \\pm 0.4$ on CiteSeer vs. $67.5 \\pm 1.62$ here. Also some other baseline numbers seem to be off. Could you comment on possible discrepancies, and whether the code you are running for your experiment can reproduce the numbers reported in the literature? Some of the differences I mentioned above are similar in scale to the ones you use to claim the effectiveness of your method.\n- From Table 3 (which I find convincing), it seems like RFF substantially improves many of the baseline methods. It is not really clear to me whether the reported gains are mostly due to the proposed augmentation, or mainly due to the RFF. A clean ablation experiement, ideally on all the datasets would help. Specifically, is the method SOTA also *without* RFF, i.e., due to the different representation learning approach? This is not addressed by the experiments currently.\n- In general, the experimental results seem limited. See my clarification questions below.\n\n**Minor:**\n\n- The choice of colors in Table 1 is probably not well readable by readers with colorblindness. \n- The word \"significant\" is used multiple times in the text, but no statistical tests were run. I propose to either add statistics to your tables (my recommendation), or drop/replace the term.\n\nQUESTIONS:\n- l. 178: What is the corruption function?\n- Eq. 3: I think the equation is off and misses two minus signs, once in front of the loss term, and one in front of the negative examples. I.e., shouldnt the general form be $ - \\left(\\sum \\mathbb E \\log D_\\omega(.) + \\sum \\mathbb E \\log (1 - D_\\omega(.)) \\right)$ ?\n- What is the rationale for only running experiments on all datasets in Table 1, vs. for all the tables? Also, could you outline the choice of these particular datasets for your study? For instance, SUGRL also evaluates on Ogbn-arxiv, Ogbn-mag, Ogbn-products.\n- Could you more clearly outline your contribution w.r.t. the representation learning part? The loss formulation as such seems to be established, still a substantial amount of space in section 5 is distributed to discussing the loss and its derivation. My impression is that mainly the composition and augmentation of the positive pair distinguishes this part from previous work. Could you confirm/discuss this further?\n- Does the FIGURe model in Table 1 use RFF, or not? If it does not use RFF as suggested by the caption, then the result in Table 1 would contradict the result in Table 3. Could you clarify?\n- Are there any assumptions underlying your proposed augmentation scheme with respect to the properties of the datasets? I.e., are the classification tasks in which a representation obtained using FIGURe can be expected to perform worse than other techniques due to the augmentation design?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a few approaches to improve the contrastive learning framework of the unsupervised graph representation learning (UGRL) problem. Building on top of the prior works in supervised GRL and UGRL areas (e.g., filter bank construction, etc.), the authors argued that 1) filter-based augmentations (essentially treating filter banks as \"additional views\" in contrastive learning schemes) is able to provide useful representations across cases that require high-freq and low-freq components; and 2) we can leverage a lot of techniques long known in the ML/DL community like RFF to efficiently reduce the computational complexity of these latent representations in UGRL.\n\nSTRENGTHS:\nThe paper proposed a relatively novel approach of leveraging different filter banks as additional views in a contrastive graph representation learning problem. The method basically builds upon the MI maximization scheme proposed in methods like DGI but extends the approach to also include filter bank $F=\\{F_1, \\dots, F_k\\}$ such that each filter's encoder output's MI with the input data is maximized. The empirical analysis reveals that the approach is superior (and likely compatible with?) to the previous UGRL approaches that didn't use filter-bank-related augmentations; and that RFF provides a reasonable boost to low-dimensionality representation learning in graphs. Specifically:\n\n1. The paper provides a reasonable empirical analysis of the approach on a diverse set of heterophilic and homophilic datasets, demonstrating improvements in both settings. \n2. Numerous ablative studies were made, including on low and high dimensionalities, encoder weight sharing, efficiency, and filter bank selection.\n3. The idea of including multiple filters in the contrastive learning framework itself makes sense to me and is a novel addition to the UGRL literature.\n\nWEAKNESSES:\nA few weaknesses of the paper:\n\n1. The main idea of the approach (maximizing mutual information of encoder output and input; applying discriminator to the patch-readout pair), which is described in Sec. 5.1, is still mainly built up on the discussion of the DGI paper. Although the inclusion of the filter bank discussion is new (which follows from other recent work like BernNet, GPRGNN, and ChebNet), this still limits the novelty of the approach itself.\n2. The contribution of RFF projections, as shown in Sec. 6.2, is largely orthogonal to the FiGURe approach itself. I wonder whether it's the best idea to include it in this paper whose title is on FiGURe and filter augmentations. For example, the efficiency analysis in Sec. 6.4 clearly is lacking the comparison to DGI/MVGRL + RFF, and is thus unfair. Moreover, as the authors suggested in the supplementary materials, the RFF projection they used was based on a Gaussian kernel. Whether and how different projections (e.g., Laplacian) make a difference is unknown; the gradient instability of using RFFs in the architecture (which is an important problem in RFFs' usage in Transformers) is not discussed, etc. I personally feel there is a lot of value to the analysis of RFF itself.\n3. I'm less certain whether \"shared encoder weight\" is a contribution (as listed on line 48) of this paper. It seems more like a remedy to the computational burden added by the FiGURe approach, but is not required otherwise?\n\nQUESTIONS:\n1. What is the effect of the number of encoder layers used in this case? The authors used one (as was in DGI), but there have also been works that used more.\n2. Could the authors expand Table 5 to include RFF + {other baseline methods}?\n3. Are the numbers provided in Table 1 and 2 the results of the authors' reproduction, or the original papers? Some of these numbers were generally different from what was reported in the original & other papers (e.g., MVGRL, GRACE).\n4. Could the authors elaborate more on the corruption function $C$ (especially how it creates $\\tilde{F}$)?\n\n---------\n\nUPDATE: I have read the rebuttals from the authors and appreciate the additional experiments & explanations. As the authors acknowledge, the benefit of RFF is not unique to FiGUREe (which some other reviewers also pointed out), and I think it should be elaborated on and compared to more clearly in the paper's revision (so is the novelty issue). I'd like to maintain the current rating, but I think the paper's writing can be improved more on these axis to avoid confusion.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-24"}