SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FiGURE: Simple and Efficient Unsupervised Node Representations with Filter Augmentations
Chanakya Ekbote*
Microsoft Research India
chanakyaekbote@gmail.com
Ajinkya Pankaj Deshpande*
ajinkya.deshpande56@gmail.com
Arun Iyer
ariy@microsoft.com
Ramakrishna Bairi
rkbairi@gmail.com
Sundararajan Sellamanickam
ssrajan@microsoft.com
Unsupervised node representations learnt using contrastive learning-based methods have shown good performance on downstream tasks. However, these methods rely on augmentations that mimic low-pass filters, limiting their performance on tasks requiring different eigen-spectrum parts. This paper presents a simple filter-based augmentation method to capture different parts of the eigen-spectrum. We show significant improvements using these augmentations. Further, we show that sharing the same weights across these different filter augmentations is possible, reducing the computational load. In addition, previous works have shown that good performance on downstream tasks requires high dimensional representations. Working with high dimensions increases the computations, especially when multiple augmentations are involved. We mitigate this problem and recover good performance through lower dimensional embeddings using simple random Fourier feature projections. Our method, FiGURE, achieves an average gain of up to  $4.4\%$ , compared to the state-of-the-art unsupervised models, across all datasets in consideration, both homophilic and heterophilic. Our code can be found at: https://github.com/microsoft/figure.
Contrastive learning is a powerful method for unsupervised graph representation learning, achieving notable success in various applications [44, 15]. However, these evaluations typically focus on tasks exhibiting homophily, where task labels strongly correlate with the graph's structure. An existing edge suggests the connected nodes likely share similar labels in these scenarios. However, these representations often struggle when dealing with heterophilic tasks, where edges tend to connect nodes with different labels. Several papers [7, 17, 4, 28] have tackled the problem of heterophily by leveraging information from both low and high-frequency components. However, these methods operate in the semi-supervised setting, and the extension of these ideas in unsupervised learning still needs to be explored. Inspired by the insights in these papers, we propose a simple method incorporating these principles.
Our approach introduces filter banks as additional views and learns separate representations for each filter bank. However, this approach faces two main challenges: Firstly, storing representations from each view can become prohibitively expensive for large graphs; secondly, contrastive learning
*Both authors contributed equally to this work. Work done while the authors were Research Fellows at Microsoft Research India.
methods typically demand high-dimensional representations, which increase both the computational cost of training and the storage burden. We employ a shared encoder for all filter banks to tackle the first challenge. Our results confirm that a shared encoder performs on par with independent encoders for each filter bank. This strategy enables us to reconstruct filter-specific representations as needed, drastically reducing the storage requirement. For the second challenge, we train our models with low-dimensional embeddings. Then, we use random Fourier feature projection [40] to lift these low-dimensional embeddings into a higher-dimensional space. Kernel tricks [23] were typically used in classical machine learning to project low-dimensional representation to high dimensions where the labels can become linearly separable. However, constructing and leveraging the kernels in large dataset scenarios could be expensive. To avoid this issue, several papers [40, 41, 20, 36, 26] proposed to approximate the map associated with the kernel. For our scenario, we use the map associated with Gaussian kernel [40]. We empirically demonstrate that using such a simple approach preserves high performance for downstream tasks, even in the contrastive learning setting. Consequently, our solution offers a more efficient approach to unsupervised graph representation learning in computation and storage, especially concerning heterophilic tasks. The proposed method exhibits simplicity not only in the augmentation of filters but also in its ability to learn and capture information in a low-dimensional space, while still benefiting from the advantages of large-dimensional embeddings through Random Fourier Feature projections.
Our contributions in this work are, 1] We propose a simple scheme of using filter banks for learning representations that can cater to both heterophily and homophily tasks, 2] We address the computational and storage burden associated with this simple strategy by sharing the encoder across these various filter views, 3] By learning a low-dimensional representation and later projecting it to high dimensions using random Fourier Features, we further reduce the burden, 4] We study the performance of our approach on four homophilic and seven heterophilic datasets. Our method, FiGURe, achieves an average gain of up to  $4.4\%$ , compared to the state-of-the-art unsupervised models, across all datasets in consideration, both homophilic and heterophilic. Notably, even without access to task-specific labels, FiGURe performs competitively with supervised methods like GCN [25].
Several unsupervised representation learning methods have been proposed in prior literature. Random walk-based methods like Node2Vec [14] and DeepWalk [37] preserve node proximity but tend to neglect structural information and node features. Contrastive methods, such as DEEP GRAPH INFOMAX (DGI) [44], maximize the mutual information (MI) between local and global representations while minimizing the MI between corrupted representations. Methods like MVGRL [15] and GRACE [45] expand on this, by integrating additional views into the MI maximization objective. However, most of these methods focus on the low frequency components, overlooking critical insights from other parts. Semi-supervised methods like GPRGNN [7], BERNNET [17], and PPGNN [28] address this by exploring the entire eigenspectrum, but these concepts are yet to be applied in the unsupervised domain. This work proposes the use of a filter bank to capture information across the full eigenspectrum while sharing an encoder across filters. Given the high-dimensional representation demand of contrastive learning methods, we propose using Random Fourier Features (RFF) to project lower-dimensional embeddings into higher-dimensional spaces, reducing computational load without sacrificing performance. The ensuing sections define our problem, describe filter banks and random feature maps, and explain our model and experimental results.
In the domain of unsupervised representation learning, our focus lies on graph data, denoted as  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , where  $\mathcal{V}$  is the set of vertices and  $\mathcal{E}$  the set of edges  $(\mathcal{E}\subseteq \mathcal{V}\times \mathcal{V})$ . We associate an adjacency matrix with  $\mathcal{G}$ , referred to as  $\mathbf{A}:\mathbf{A}\in \{0,1\}^{n\times n}$ , where  $n = |\mathcal{V}|$  corresponds to the number of nodes. Let  $\mathbf{X}\in \mathbb{R}^{n\times d}$  be the feature matrix. We use  $\mathbf{A_I}$  to represent  $\mathbf{A} + \mathbf{I}$  with  $\mathbf{I}$  is the identity matrix, while  $\mathbf{D_{AI}}$  signifies the degree matrix of  $\mathbf{A_I}$ . We also define  $\mathbf{A_n}$  as  $\mathbf{D}_{\mathbf{A_I}}^{-1 / 2}\mathbf{A}_ID_{\mathbf{A_I}}^{-1 / 2}$ . No additional information is provided during training. The goal is to learn a parameterized encoder,  $E_{\theta}:\mathbb{R}^{n\times n}\times \mathbb{R}^{n\times d}\mapsto \mathbb{R}^{n\times d'}$ , where  $d^{\prime}\ll d$ . This encoder produces a set of node representations  $E_{\theta}(\mathbf{X},\mathbf{A_n}) = \{h_1,h_2,\dots,h_n\}$  where each  $h_i\in \mathbb{R}^{d'}$  represents a rich representation for node  $i$ . The
subsequent section will provide preliminary details about filter banks and random feature maps before we discuss the specifics of the proposed approach.
4 Preliminaries
Our approach relies on filter banks and random feature maps. In this section, we briefly introduce these components, paving the way for a detailed explanation of our approach.
4.1 Filter Banks
Graph Fourier Transform (GFT) forms the basis of Graph Neural Networks (GNNs). A GFT is defined using a reference operator  $\mathbf{R}$  which admits a spectral decomposition. Traditionally, in the case of GNNs, this reference operator has been the symmetric normalized laplacian  $\mathbf{L}_{\mathbf{n}} = \mathbf{I} - \mathbf{A}_{\mathbf{n}}$  or the  $\mathbf{A}_{\mathbf{n}}$  as simplified in [25]. A graph filter is an operator that acts independently on the entire eigenspace of a diagonalisable and symmetric reference operator  $\mathbf{R}$ , by modulating their corresponding eigenvalues. [43, 42]. Thus, a graph filter  $\mathbf{H}$  is defined via the graph filter function  $g(.)$  operating on the reference operator as  $\bar{\mathbf{H}} = g(\mathbf{R}) = \mathbf{U}g(\boldsymbol {\Lambda})\mathbf{U}^T$ . Here,  $\boldsymbol{\Lambda} = diag([ \lambda_1,\lambda_2,\dots,\lambda_n])$ , where  $\lambda_{i}$  denotes the eigenvalues of the reference operator. We describe a filter bank as a set of filters, denoted as  $\mathbf{F} = \{\mathbf{F}_1,\mathbf{F}_2,\dots,\mathbf{F}_K\}$ . Both GPRGNN [7] and BERNNET [17] employ filter banks, comprising of polynomial filters, and amalgamate the representations from each filter bank to enhance the performance across heterophilic datasets. GPRGNN uses a filter bank defined as  $\mathbf{F}_{\mathrm{GPRGNN}} = \{\mathbf{I},\mathbf{A}_{\mathbf{n}},\dots,\mathbf{A}_{\mathbf{n}}^{K - 1}\}$ , while  $\mathbf{F}_{\mathrm{BERNNET}} = \{\mathbf{B}_0,\mathbf{B}_1,\dots,\mathbf{B}_{K - 1}\}$  characterizes the filter bank utilized by BERNNET. Here,  $\mathbf{B}_i = \frac{1}{2^{K - 1}}\binom {K - 1}{i} (2\mathbf{I} - \mathbf{L}_{\mathbf{n}})^{K - i - 1} (\mathbf{L}_{\mathbf{n}})^i$ . Each filter in these banks highlights different parts of the eigenspectrum. By tuning the combination on downstream tasks, it offers the choice to select and leverage the right spectrum to enhance performance. Notably, unlike traditional GNNs, which primarily emphasize low-frequency components, higher frequency components have proved useful for heterophily [4, 7, 17, 28]. Consequently, a vital takeaway is that for comprehensive representations, we must aggregate information from different parts of the eigenspectrum and fine-tune it for specific downstream tasks.
4.2 Random Feature Maps for Kernel Approximations
Before the emergence of deep learning models, the kernel trick was instrumental in learning non-linear models. A kernel function,  $k: \mathbb{R}^d \times \mathbb{R}^d \mapsto \mathbb{R}$ , accepts two input features and returns a real-valued score. Given a positive-definite kernel, Mercer's Theorem [30] assures the existence of a feature map  $\phi(\cdot)$ , such that  $k(x,y) = \langle \phi(x), \phi(y) \rangle$ . Leveraging the kernel trick, researchers combined Mercer's theorem with the representer theorem [23], enabling the construction of non-linear models that remain linear in  $k$ . These models created directly using  $k$  instead of the potentially complex  $\phi$ , outperformed traditional linear models. The implicit maps linked with these kernels projected the features into a significantly high-dimensional space, where targets were presumed to be linearly separable. However, computational challenges arose when dealing with large datasets. Addressing these issues, subsequent works [41, 20, 36, 40] introduced approximations of the map associated with individual kernels through random projections into higher-dimensional spaces  $(\phi'(.))$ . This approach ensures that  $\langle \phi'(\mathbf{x}), \phi'(\mathbf{y}) \rangle \approx k(x,y)$ . These random feature maps are inexpensive to compute and affirm that simple projections to higher-dimensional spaces can achieve linear separability. The critical insight is that computationally efficient random feature maps, such as Random Fourier features (RFF) [40], exist. These maps project lower-dimensional representations into higher dimensions, enhancing their adaptability for downstream tasks.
5 Proposed Approach
The following section delineates the process of unsupervised representation learning. Then, we detail the use of filter bank representations in downstream tasks with random feature maps.
5.1 Unsupervised Representation Learning
Our method FiGURe (Filter-based Graph Unsupervised Representation Learning) builds on concepts introduced in [18, 44], extending the maximization of mutual information between node and global
filter representations for each filter in the filter bank  $\mathbf{F} = \{\mathbf{F}_1, \mathbf{F}_2, \dots, \mathbf{F}_K\}$ . In this approach, we employ filter-based augmentations, treating filter banks as "additional views" within the context of contrastive learning schemes. In the traditional approach, alternative baselines like DGI have employed a single filter in the GPRGNN filter bank. Additionally, MVGRL attempted to use the diffusion kernel; nevertheless, they only learned a single representation per node. We believe that this approach is insufficient for accommodating a wide range of downstream tasks. We construct an encoder for each filter to maximize the mutual information between the input data and encoder output. For the  $i^{\text{th}}$  filter, we learn an encoder,  $E_\theta : \mathcal{X}_i \to \mathcal{X}_i'$ , denoted by learnable parameters  $\theta$ . In this context,  $\mathcal{X}_i$  represents a set of examples, where each example  $[\widehat{\mathbf{X}_{ij}}, \widehat{\mathbf{F}_{ij}}] \in \mathcal{X}_i$  consists of a filter  $\mathbf{F}_i$ , its corresponding nodes and node features, drawn from an empirical probability distribution  $\mathbb{P}_i$ . That is,  $\mathbb{P}_i$  captures the joint distribution of a filter  $\mathbf{F}_i$ , its corresponding nodes and node features  $([\mathbf{X}, \mathbf{F}_i])$ . Note that  $[\widehat{\mathbf{X}_{ij}}, \widehat{\mathbf{F}_{ij}}]$  denote nodes, node features and edges sampled from the  $i^{th}$  filter  $([\mathbf{X}, \mathbf{F}_i])$  basis the probability distribution  $\mathbb{P}_i$ . Note that  $\widehat{\mathbf{X}_{ij}} \in \mathbb{R}^{N' \times d}$  and  $\widehat{\mathbf{F}_{ij}} \in \mathbb{R}^{N' \times N'}$ .  $\mathcal{X}_i'$  defines the set of representations learnt by the encoder on utilizing feature information as well as topological information from the samples, sampled from the joint distribution  $\mathbb{P}_i$ . The goal, aligned with [29, 18, 44], is to identify  $\theta$  that maximizes mutual information between  $[\mathbf{X}, \mathbf{F}_i]$  and  $E_\theta(\mathbf{X}, \mathbf{F}_i)$ , or  $\mathcal{I}_i([\mathbf{X}, \mathbf{F}_i], E_\theta(\mathbf{X}, \mathbf{F}_i))$ . While exact mutual information (MI) computation is infeasible due to unavailable exact data and learned representations distributions, we can estimate the MI using the Jensen-Shannon MI estimator [9, 32], defined as:
$$
\begin{array}{l} \mathcal {I} _ {i, \theta , \omega} ^ {\mathrm {J S D}} ([ \mathbf {X}, \mathbf {F} _ {i} ], E _ {\theta} (\mathbf {X}, \mathbf {F} _ {i})) := \mathbb {E} _ {\mathbb {P} _ {i}} [ - \operatorname {s p} (- T _ {\theta , \omega} ([ \widehat {\mathbf {X} _ {i j}}, \widehat {\mathbf {F} _ {i j}} ], E _ {\theta} (\widehat {\mathbf {X} _ {i j}}, \widehat {\mathbf {F} _ {i j}})) ] - \tag {1} \\ \mathbb {E} _ {\mathbb {P} _ {i} \times \widetilde {\mathbb {P}} _ {i}} [ \mathbf {s p} (T _ {\theta , \omega} ([ \widehat {\mathbf {X}} _ {i j}, \widehat {\mathbf {F}} _ {i j} ], E _ {\theta} [ \widetilde {\mathbf {X}} _ {i j}, \widetilde {\mathbf {F}} _ {i j} ]) ] \\ \end{array}
$$
Here,  $T_{\omega}:\mathcal{X}_i\times \mathcal{X}'_i\to \mathbb{R}$  represents a discriminator function with learnable parameters  $\omega$  .Note that  $[\widetilde{\mathbf{X}_{ij}},\widetilde{\mathbf{F}_{ij}} ]$  is an input sampled from  $\widetilde{\mathbb{P}}_i$  , which denotes a distribution over the corrupted input data (more details given below). The function sp.) corresponds to the softplus function [10]. Additionally,  $T_{\theta ,\omega}([h_{ij}]_1,[h_{ij}]_2) = D_w\circ (\mathcal{R}([h_{ij}]_1),[h_{ij}]_2)$  where  $\mathcal{R}$  denotes the readout function responsible for summarizing all node representations by aggregating and distilling information into a global filter representation. We introduce a learnable discriminator  $D_{\omega}$  ,where  $D_{\omega}(.,.)$  represents the joint probability score between the global representation and the node-specific patch representation. Note that  $[h_{ij}]_1$  denotes representations obtained after passing samples sampled from the original distribution  $\mathbb{P}_i$  to the encoder, and  $[h_{ij}]_2$  ,denotes representations obtained after passing samples sampled from the original distribution  $\mathbb{P}_i$  or samples sampled from the corrupted distribution  $\widetilde{\mathbb{P}}_i$  to the encoder. Intuitively, Eq. 1 implies that we would want to maximize the mutual information between the local (patch) representations of the nodes and their corresponding global graph
representation, while minimising the mutual information between a global graph representation and the local representations of corrupted input data. Note that:  $[\widetilde{\mathbf{X}}_{ij},\widetilde{\mathbf{F}}_{ij}]$  denotes a corrupted version of input features and given filter. More details with regards to this will be given below. In our approach, we first obtain node representations by feeding the filter-specific topology and associated node features into the encoder:  $\mathbf{H}_{ij} = E_{\theta}(\mathbf{X}_{ij},\mathbf{F}_{ij}) = \{h_1^{\mathbf{F}_{ij}},h_2^{\mathbf{F}_{ij}},\dots,h_{N'}^{\mathbf{F}_{ij}}\}$ . Note that  $\mathbf{H}_{ij}$  has the following dimensions  $\mathbb{R}^{N'\times d'}$ . To obtain global representations, we employ a readout function  $\mathcal{R}:\mathbb{R}^{N'\times d'}\to \mathbb{R}^{d'}$ , which combines and distills information into a global representation  $h_g^{F_{ij}} = \mathcal{R}(\mathbf{H}_{ij}) = \mathcal{R}(E_\theta (\mathbf{X}_{ij},\mathbf{F}_{ij}))$ . Instead of directly maximizing the mutual information between the local and global representations, we maximize  $D_{\omega}(.,.)$ . This joint score should be
Figure 1: Unsupervised learning of node embeddings by maximizing mutual information between node and graph representations over the graphs from the filter bank. Note that the parameter  $\Theta$  is shared across all the filters.
higher when considering global and local representations obtained from the same filter, as opposed to the joint score between the global representation from one filter and the local representation from corrupted  $(\mathbf{X}_{ij},\mathbf{F}_{ij})$ . To generate negative samples for contrastive learning, we employ a corruption function  $\mathcal{C}:\mathbb{R}^{N'\times d}\times \mathbb{R}^{N'\times N'}\to \mathbb{R}^{N'\times d}\times \mathbb{R}^{N'\times N'}$ , which yields corrupted samples denoted as  $[\widetilde{\mathbf{X}}_{ij},\widetilde{\mathbf{F}}_{ij}] = \mathcal{C}(\mathbf{X}_{ij},\mathbf{F}_{ij})$ . The designed corruption function generates data decorrelated with the input data. Note that  $[\widetilde{\mathbf{X}}_{ij},\widetilde{\mathbf{F}}_{ij}]$  denote nodes, node features and edges sampled from the corrupted version of  $i^{th}$  filter, basis the probability distribution  $\widetilde{\mathbb{P}}_i$ . The corruption function can be designed basis the task at hand. We employ a simple corruption function, whose details are present in the experimental section. Let the corrupted node representations be as follows:  $\widetilde{\mathbf{H}}_{ij} = E_{\theta}(\mathcal{C}(\mathbf{X}_{ij},\mathbf{F}_{ij})) = E_{\theta}(\widetilde{\mathbf{X}}_{ij},\widetilde{\mathbf{F}}_{ij}) = \{\widetilde{h_1^{\widetilde{\mathbf{F}}_{ij}}},\widetilde{h_2^{\widetilde{\mathbf{F}}_{ij}}},\dots,\widetilde{h_{N'}^{\widetilde{\mathbf{F}}_{ij}}}\}$ . In order to learn representations across all filters in the filter bank, we aim to maximise the average estimate of mutual information (MI) across all filters, considering  $K$  filters, defined by  $\mathcal{T}_{\mathbf{F}}$ .
$$
\mathcal {I} _ {\mathbf {F}} = \frac {1}{K} \sum_ {i = 1} ^ {K} \mathcal {I} _ {i, \theta , \omega} ^ {J S D} ([ \mathbf {X}, \mathbf {F} _ {i} ], E _ {\theta} (\mathbf {X}, \mathbf {F} _ {i})) \tag {2}
$$
Maximising the Jenson-Shannon MI estimator can be approximately optimized by reducing the binary cross entropy loss defined between positive samples (sampled from the  $\mathbb{P}_i$ ) and the negative samples (sampled from  $\widetilde{\mathbb{P}}_i$ ). Therefore, for each filter, the loss for is defined as follows:
$$
\mathcal {L} _ {\mathbf {F} _ {i 1}} = - \frac {1}{2 N ^ {\prime}} \mathbb {E} _ {\left(\left[ \mathbf {X} _ {i j}, \mathbf {F} _ {i j} \right] \sim \mathbb {P} _ {i}\right)} \left(\sum_ {k = 1} ^ {N ^ {\prime}} \left[ \log \left(D _ {\omega} \left(h _ {k} ^ {\mathbf {F} _ {i j}}, h _ {g} ^ {\mathbf {F} _ {i j}}\right)\right) + \log \left(1 - D _ {\omega} \left(\widetilde {h _ {k} ^ {\mathbf {F} _ {i j}}} , h _ {g} ^ {\mathbf {F} _ {i j}}\right)\right) \right]\right) \tag {3}
$$
Note that for Eq. 3, the global representation  $(h_g^{\mathbf{F}_{ij}})$  is generated by  $\mathcal{R}(E_{\theta}(\mathbf{X}_{ij},\mathbf{F}_{ij}))$ . The local representations  $(h_k^{\widetilde{\mathbf{F}}_{ij}}\forall k)$  are constructed by passing the sampled graph and features through a corruption function (see  $\widetilde{\mathbf{H}}_{ij}$ ). Therefore to learn meaningful representations across all filters the following objective is minimised:
$$
\mathcal {L} = \frac {1}{K} \sum_ {i = 1} ^ {K} \mathcal {L} _ {\mathbf {F} _ {i}} \tag {4}
$$
Managing the computational cost and storage demands for large graphs with distinct node representations for each filter poses a challenge, especially when contrastive learning methods require high dimensions. To address this, we employ parameter sharing, inspired by studies like [7] and [17]. This approach involves sharing the encoder's parameters  $\theta$  and the discriminator's parameters  $\omega$  across all filters. Instead of storing dense filter-specific node representations, we only store the shared encoder's parameters and the first-hop neighborhood information for each node per filter, reducing storage requirements. To obtain embeddings for downstream tasks, we reconstruct filter-specific representations using a simple one-layer GNN. This on-demand reconstruction significantly reduces computational and storage needs associated with individual node representations. Fig 1 illustrates such a simple encoder's mutual information-based learning process. To address the second challenge, we initially train our models to produce low-dimensional embeddings that capture latent classes, as discussed in [2]. These embeddings, while informative, lack linear separability. To enhance separability, we project these low-dimensional embeddings into a higher-dimensional space using random Fourier feature (RFF) projections, inspired by kernel methods (see Section 4.2). This approach improves the linear separability of latent classes, as confirmed by our experimental results in Section 6.2, demonstrating the retention of latent class information in these embeddings.
5.2 Supervised Representation Learning
After obtaining representations for each filter post the reconstruction of the node representations, learning an aggregation mechanism to combine information from representations that capture different parts of the eigenspectrum for the given task is necessary. We follow learning schemes from [7, 17, 28],
Figure 2: Supervised Learning: Using the trained parameter  $\Theta$ , we generate the node embeddings by encoding the filtered graphs that get consumed in the classification task.