SlowGuess/NeurIPS_2023_OCR huggingface dataset:
6.1 RQ1:FiGUResversusSOTA Methods 7
6.2 RQ2: RFF Projections on Lower Dimensional Representations 8
6.3 RQ3:SharingWeightsAcrossFilterSpecificEncoders 9
6.4 RQ4: Computational Efficiency 9
6.5 RQ5: Experiments on Other Filter Banks 10
6.6 RQ6: Combining information from different filters (in  $\mathbf{F}_{\mathrm{GPRGNN}}$  10
7 Conclusion and Future Work 10
8 Supplementary Material 14
8.1 Reproducibility 15
8.2 Datasets 15
8.3 Training Details 16
8.3.1 Unsupervised Training 16
8.3.2 Supervised Training 16
8.3.3 Negative Sampling for the Identity Filter 17
8.4 Comparison with other Supervised Methods 17
8.5 Increasing the depth of the Encoder 17
8.6 RFF Projections 18
8.7 Visualising RFF Behavior and Community Structure 18
8.8 Other Random Projections 19
8.9 RFF ablation study at 128 dimensions 20
8.10 RFF ablation study at 512 dimensions 20
8.11 Issues with including RFF in training 20
8.12 Computational Comparisons with other Other Unsupervised Methods 21
8.13 Time and storage cost 21
8.14 Choice of Filter Banks 22
8.1 Reproducibility
We strive to ensure the reproducibility of our research findings. To facilitate this, we provide the details of our experimental setup, including dataset sources, preprocessing steps, hyperparameters, and model configurations. We also make our code and the datasets used, publicly available at https://github.com/microsoft/figure, enabling researchers to reproduce our results and build upon our work. We would like to emphasize that our code is built on top of the existing MVGRL codebase. For the datasets used in our evaluation, we provide references to their original sources and any specific data splits that we employed. This allows others to obtain the same datasets and perform their own analyses using consistent data. Additionally, we specify the versions of libraries and frameworks used in our experiments, in Section 8.3, and in the REQUIREMENTS file and the README file, in the codebase, enabling others to set up a compatible environment. We document any specific seed values or randomization procedures that may affect the results. By providing these details and resources, we aim to promote transparency and reproducibility in scientific research. We encourage fellow researchers to reach out to us if they have any questions or need further clarification on our methods or results.
8.2 Datasets
Homophilic Datasets: We evaluated our model (as well as baselines) on three homophilic datasets: CORA, CITESEER, and PUBMED as borrowed from [34]. All three are citation networks, where each node represents a research paper and the links represent citations. Pubmed consists of medical research papers. The task is to predict the category of the research paper. We follow the same dataset setup mentioned in [34] to create 10 random splits for each of these datasets.
Heterophilic Datasets: In our evaluation, we included four heterophilic datasets: CHAMELEON, SQUIRREL, ROMAN-EMPIRE, and MINESWEEPER. For CHAMELEON and SQUIRREL, nodes represent Wikipedia web pages and edges capture mutual links between pages. We utilized the ten random splits provided in [34], where  $48\%$ ,  $32\%$ , and  $20\%$  of the nodes were allocated for the train, validation, and test sets, respectively. In ROMAN-EMPIRE each node corresponds to a word in the Roman Empire Wikipedia article. Two words are connected with an edge if either these words follow each other in the text, or they are connected in the dependency tree of the sentence. The syntactic role of the word/node defines its class label. The MINESWEEPER graph is a regular 100x100 grid where each node is connected to eight neighboring nodes, and the features are on-hot encoded representations of the number of neighboring mines. The task is to predict which nodes are mines. For both ROMAN-EMPIRE and MINESWEEPER, we used the ten random splits provided in [38].
Large Datasets: We also evaluate our method on two large datasets OGBN-ARXIV (from [19]) and ARXIV-YEAR (from [27]). Both these datasets are from the arxiv citation network. In OGBN-ARXIV, the task is to predict the category of the research paper, and in ARXIV-YEAR the task is to predict the year of publication. We use the publicly available splits for OGBN-ARXIV [22] and follow the same dataset setup mentioned in [27] to generate 5 random splits for ARXIV-YEAR. Note that OGBN-ARXIV is a homophilic dataset while ARXIV-YEAR is a heterophilic datasets.
The detailed dataset statistics can be found in Table 9.
Table 9: Dataset Statistics. The table provides information on the following dataset characteristics: number of nodes, number of edges, feature dimension, number of classes, as well as the count of nodes used for training, validation, and testing.
PROPERTIES	HETEROPHILIC DATSETS	HOMOPHILIC DATSETS
SQUIRREL	CHAMELEON	ROMAN-EMPIRE	MINESWEEPER	ARXIV-YEAR	OGBN-ARXIV	CITESEER	PUBMED	CORA
#NODES	5201	2277	22662	10000	169343	169343	3327	19717	2708
#EDGES	222134	38328	32927	39402	1166243	1335586	12431	108365	13264
#FEATURES	2089	500	300	7	128	128	3703	500	1433
#CLASSES	5	5	18	2	5	40	6	3	7
#TRAIN	2496	1092	11331	5000	84671	90941	1596	9463	1192
#VAL	1664	729	5665	2500	42335	29799	1065	6310	796
#TEST	1041	456	5666	2500	42337	48603	666	3944	497
8.3 Training Details
We conducted all experiments on a machine equipped with an Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz processor, 440GB RAM, and a Tesla-P100 GPU with 16GB of memory. The experiments were executed using Python 3.9.12 and PyTorch 1.13.0 [33]. To optimize the hyperparameter search, we employed Optuna [1]. We utilized the Adam optimizer [24] for the optimization process.
8.3.1 Unsupervised Training
We conducted hyperparameter tuning for all unsupervised methods using 20 Optuna trials. The hyperparameter ranges and settings for each method are as follows:
DEEPWALK: We set the learning rate to 0.01, number of epochs to 20 and the varied the random walk length over  $\{8,9,10,11,12\}$ . Additionally, we varied the context window size over  $\{3,4,5\}$  and the negative size (number of negative samples per positive sample) over  $\{4,5,6\}$ .
NODE2VEC: For Node2Vec, we set the learning rate to 0.01 and number of epochs to 100. We varied the number of walks over  $\{5, 10, 15\}$  and the walk length over  $\{40, 50, 60\}$ . The  $p$  (return parameter) value was chosen from  $\{0.1, 0.25, 0.5, 1\}$  and  $q$  (in-out parameter) value was chosen from  $\{3, 4, 5\}$ .
DGI: DGI [44] proposes a self-supervised learning framework for graph representation learning by maximizing the mutual information between local and global structural context of nodes, enabling unsupervised feature extraction in graph neural networks. We relied on the authors' code<sup>2</sup> and the prescribed hyperparameter ranges specific to the DGI model, for our experiments.
MVGRL: MVGRL [15] proposes a method for learning unsupervised node representations by leveraging two views of the graph data, the graph diffusion view and adjacency graph view. We relied on the authors' code<sup>3</sup> and the prescribed hyperparameter ranges specific to the MVGRL model, for our experiments.
GRACE: GRACE [45] proposes a technique where two different perspectives of the graph are created through corruption, and the learning process involves maximizing the consistency between the node representations obtained from these two views. We relied on the authors' code<sup>4</sup> and the prescribed hyperparameter ranges specific to the GRACE model, for our experiments.
SUGRL: SUGRL [31] proposes a technique for learning unsupervised representations which capture node proximity, while also utilising node feature information. We relied on the authors' code<sup>5</sup> and the prescribed hyperparameter ranges specific to the SUGRL model, for our experiments.
FiGURE: We followed the setting of the MVGRL model, setting the batch size to 2 and number of GCN layers to 1. We further tuned the learning rate over  $\{0.00001, 0.0001, 0.001, 0.01, 0.1\}$  and the sample size (number of nodes selected per batch) over  $\{1500, 1750, 2000, 2250\}$ , except for the large graphs, for which we set the sample size to 5,000.
We maintained consistency across all methods by employing identical parameters for maximum epochs and early stopping criteria. Specifically, we set the maximum number of epochs to 30,000 and utilized an early stopping patience of 20 epochs, with the exception of large datasets, where we extended the patience to 500 epochs.
In each case, we selected the hyperparameters that resulted in the lowest unsupervised training loss.
8.3.2 Supervised Training
For all unsupervised methods, including the baselines and our method, we perform post-training supervised evaluation using logistic regression with 60 Optuna trials. We set the maximum number of epochs to 10000 and select the epoch and hyperparameters that yield the best validation accuracy. The learning rate is swept over the range  $\{0.00001, 0.0001, 0.001, 0.0015, 0.01, 0.015, 0.1, 0.5, 1, 2\}$ , and the weight decay is varied over  $\{10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}, 0, 0.5, 1, 3\}$ .
$^{2}$ https://github.com/PetarV-/DGI.git
<sup>3</sup>https://github.com/kavehhassani/mvgrl.git
<sup>4</sup>https://github.com/CRIPAC-DIG/GRACE.git
<sup>5</sup>https://github.com/YujieMo/SUGRL.git
FiGURe: Along with the hyperparameters described above, following the approach described in [17], we also tune the combination coefficients  $(\alpha_{i}$  's) with a separate learning rate. This separate learning rate is swept over the range  $\{0.00001, 0.0001, 0.001, 0.0015, 0.01, 0.015, 0.1, 0.5, 1, 2\}$ . In addition, we have a coefficient for masking the incoming embeddings from each filter, which is varied between 0 and 1. Furthermore, these coefficients are passed through an activation layer, and we have two options: 'none' and 'exp'. When 'none' is selected, the coefficients are used directly, while 'exp' indicates that they are passed through an exponential function before being used.
FiGURewith RFF: For the experiments involving Random Fourier Features (RFF), we use the same hyperparameter ranges as mentioned above. However, we also tune the gamma parameter which is specific to RFF projections. The gamma parameter is tuned within the range  $\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,1.1,1.2\}$
8.3.3 Negative Sampling for the Identity Filter
In our implementation of  $\mathbf{F}_{\mathrm{GPRGNN}}$  or  $\mathbf{F}_{\mathrm{BERNNET}}$ , we follow a specific procedure for handling the filters during training and evaluation. For all filters except the identity filter (I), we employ the negative sampling approach described in Section 6. However, the identity filter is treated differently. During training, we exclude the identity filter and only include it during evaluation.
During negative sampling, the generation of the negative anchor involves shuffling the node features, followed by premultiplying the shuffled node feature matrix with the filter matrix and computing the mean. On the other hand, for the positive anchor, the same procedure is applied without shuffling the node features. This approach encourages the model to learn meaningful patterns and relationships in the data when the filter matrix is not the identity matrix.
The decision to exclude the identity filter during training is based on the observation that it presents a special case where the positive and negative anchors become the same. As a result, the model would optimize and minimize the same quantity, potentially leading to trivial solutions. To prevent this, we exclude the identity filter during training.
By excluding the identity filter during training, we ensure that the model focuses on the other filters in  $\mathbf{F}_{\mathrm{GPRGNN}}$  or  $\mathbf{F}_{\mathrm{BERNNET}}$  to capture and leverage the diverse information present in the graph. Including the identity filter only during evaluation allows us to evaluate its contribution to the final performance of the model. This approach helps prevent the model from learning trivial solutions and ensures that it learns meaningful representations by leveraging the other filters.
8.4 Comparison with other Supervised Methods
Table 10 presents a comparison with common supervised baselines. Specifically, we choose 3 models for comparison, representing three different kinds of supervised methods, standard aggregation models (GCN), spectral filter-based models (GPRGNN) and smart-aggregation models (H $_2$ GCN). There are two key observations from this table. Firstly,  $\mathrm{FiGUR e}_{512}$  is competitive with the supervised baselines, lagging behind only by a few percentage points in some cases. This suggests that much of the information that is required by the downstream tasks, captured by the supervised models, can be made available through unsupervised methods like FiGURe which uses filter banks. It is important to note that in FiGURe we only utilize logistic regression while evaluating on the downstream task. This is much more efficient than training a graph neural network end to end. Additionally it is possible that further gains may be obtained by utilizing a non-linear model like an MLP. Furthermore, as indicated by 10, we can gain further computational efficiency by utilizing lower dimensional representations like 32 and 128 (with RFF), and still not compromise significantly on the performance. Overall FiGURe manages to remain competitive despite not having access to task-specific labels and is computationally efficient as well.
8.5 Increasing the depth of the Encoder
We present an analysis in Table 11 featuring an increased number of encoder layers. In this context, "encoder layers" can be interpreted in two ways: firstly, as a deeper GCN which entails aggregating information from multiple-hop neighborhoods into the node; and secondly, as a single-hop GCN with a more extensive network for feature transformation. We provide performance results for both scenar-
Table 10: Contains node classification accuracy percentages on heterophilic and homophilic datasets. GCN, GPRGNN and  $\mathrm{H}_2\mathrm{GCN}$  are supervised methods.  $FiGURe_{32}^{RFF}$  and  $FiGURe_{128}^{RFF}$  refer to FiGURe trained with 32 and 128 dimensional representations, respectively, and then projected using RFF. The remaining models are trained at 512 dimensions. Higher numbers indicate better performance.
HETEROPHILIC DATSETS	HOMOPHILIC DATSETS
SQUIRREL	CHAMELEON	ROMAN-EMPIRE	MINESWEEPER	ARXIV-YEAR	OGBN-ARXIV	CORA	CITESEER	PUBMED
GCN	47.78 (2.13)	62.83 (1.52)	73.69 (0.74)	89.75 (0.52)	46.02 (0.26)	69.37 (0.00)	87.36 (0.91)	76.47 (1.34)	88.41 (0.46)
GPRGNN	46.31 (2.46)	62.59 (2.04)	64.85 (0.27)	86.24 (0.61)	45.07 (0.21)	68.44 (0.00)	87.77 (1.31)	76.84 (1.69)	89.08 (0.39)
H2GCN	37.90 (2.02)	58.40 (2.77)	60.11 (0.52)	89.71 (0.31)	49.09 (0.10)	OOM	87.81 (1.35)	77.07 (1.64)	89.59 (0.33)
FiGUReRFF	48.89 (1.55)	65.66 (2.52)	67.67 (0.77)	85.28 (0.71)	41.30 (0.21)	66.58 (0.00)	82.56 (0.87)	71.25 (2.20)	84.18 (0.53)
FiGUReC128	48.78 (2.48)	66.03 (2.19)	68.10 (1.09)	85.16 (0.58)	41.94 (0.15)	69.11 (0.00)	86.14 (1.13)	73.34 (1.91)	85.41 (0.52)
FiGURe	52.23 (1.19)	68.55 (1.87)	70.99(0.52)	85.58 (0.49)	42.26 (0.20)	69.69 (0.00)	87.00 (1.24)	74.77 (2.00)	88.60 (0.44)
ios, encompassing two and three layers each. It is noteworthy that the single-layer GCN achieves equal or superior performance compared to all other configurations.
Table 11: Analyzing the impact of altering the number of encoder layers on downstream accuracy, it is evident that in all instances, the single-layer GCN outperforms the others.
CORA	CITESEER	SQUIRREL	CHAMELEON
1 Layer GCN	87.00 (1.24)	74.77 (2.00)	52.23 (1.19)	68.55 (1.87)
2 Layer GCN	86.62 (1.43)	73.62 (1.46)	43.80 (1.57)	53.53 (2.13)
3 Layer GCN	84.40 (1.84)	72.52 (2.09)	42.79 (1.12)	61.73 (2.25)
GCN+ 2 Layer MLP	85.73 (1.03)	70.21 (2.30)	49.91 (2.68)	68.18 (1.76)
GCN+ 3 Layer MLP	84.99 (1.43)	71.39 (2.32)	45.85 (3.33)	64.19 (1.43)
8.6 RFF Projections
As shown in Section 6.2 and in Section 6.4, RFF projections are a computationally efficient way to achieve training by preserving the latent class behavior present in lower dimensional embeddings, by projecting them into a higher dimensional linearly separable space. The natural question that comes up is how do we compute these RFF projections? We provide an algorithm to compute the RFF projections in this section, in algorithm 1. Note that this follows [40].
Algorithm 1 Random Fourier Feature Computation
Require: Input data  $X\in \mathbb{R}^{N\times d}$ , target dimension  $D$ , kernel bandwidth  $\gamma$
Ensure: Random Fourier Features  $\bar{Z} \in \mathbb{R}^{N \times D}$
1: Initialize random weight matrix  $W \in \mathbb{R}^{d \times D}$  with Gaussian distribution
2: Initialize random bias vector  $b\in \mathbb{R}^D$  uniformly from  $[0,2\pi ]$
3: Compute scaled input  $X' = \gamma X W + b$
4: Compute random Fourier features  $Z = \sqrt{\frac{2}{D}}\cos(X')$
5: return Z
8.7 Visualising RFF Behavior and Community Structure
As shown in prior sections, FiGURe improves on both computational efficiency as well as performance by utilising RFF projections. In this section, we aim to gain insights into the behavior of RFF projections and comprehend their underlying operations through a series of simple visualizations.
t-SNE Plots: Figure 3 offers insights into the structure of the embeddings for the CORA dataset across different dimensions. Remarkably, even at lower dimensions (e.g., 32 dimensions), clear class structures are discernible, indicating that the embeddings capture meaningful information related to the class labels. Furthermore, when employing RFF to project the embeddings into higher dimensions, these distinct class structures are still preserved. This suggests that the role of RFF is not to introduce new information, but rather to enhance the suitability of lower-dimensional embeddings
for linear classifiers while maintaining the underlying class-related information. Notably, even at 512 dimensions, the class structures remain distinguishable. However, it is worth noting that the class-specific embeddings appear to be more tightly clustered and less dispersed compared to the 32-dimensional embeddings or the projected 32-dimensional embeddings. This suggests that learning a 512-dimensional embedding differs inherently from learning a 32-dimensional embedding and subsequently projecting it into higher dimensions.
(a)  $\text{FiGUR}e_{32}$  (without RFF) on the CORA dataset
(b)  $\text{FiGUR} e_{32}^{\text{RFF}}$  (with RFF) on the CORA dataset
(c)  $FiGUR e_{512}$  (without RFF) on the CORA dataset
Figure 3: The figures present t-SNE plots for the CORA dataset. These plots showcase the embeddings generated by the  $\mathbf{F}_3$  filter, which corresponds to  $\mathbf{A}^2$  in the case of FiGURe. The t-SNE plots are generated at different embedding dimensions, providing insights into the distribution and clustering of the embeddings for each dataset.
Correlation Plots: Figure 4 offers insights into the correlation patterns within the embeddings generated from the SQUIRREL dataset across different dimensions. In lower dimensions, the embeddings exhibit high correlation with each other, which can be attributed to the presence of a mixture of topics or latent classes within the dataset. However, when the embeddings are projected to higher dimensions using RFF, the correlation is reduced, and a block diagonal matrix emerges. This block diagonal structure indicates the presence of distinct classes or communities within the dataset. Even at 512 dimensions, a more refined block diagonal structure can be observed compared to the correlation matrix of the 32-dimensional embeddings. Furthermore, it is noteworthy that the correlation of the projected embeddings can be regarded as a sparser version of the correlation observed in the 512-dimensional embeddings.
(a)  $FiGUR_{32}$  (without RFF) on the SQUIRREL dataset
Figure 4: The figures display the normalized correlation plots for the SQUIRREL dataset. These plots illustrate the normalized correlation values between embeddings generated by the  $\mathbf{F}_3$  filter. In the case of FiGURe, this filter corresponds to the square of the adjacency matrix  $(\mathbf{A}^2)$ . The normalized correlation provides a measure of similarity or agreement between the embeddings obtained using the  $\mathbf{F}_3$  filter for different embedding dimensions. These plots can help analyze the consistency or variation of embeddings across different dimensions and datasets.
(b)  $\text{FiGUR}e_{32}^{\text{RFF}}$  (with RFF) on the SQUIRREL dataset
(c)  $FiGUR e_{512}$  (without RFF) on the SQUIRREL dataset
8.8 Other Random Projections
We conducted an analysis exploring the impact of utilizing random projections beyond RFF in Table 12. Our findings reveal that RFF outperforms alternative random projection methods.
Table 12: Accuracy (%) comparison of different random projections with RFF. Notably, RFF consistently outperforms the other random projection methods.
Features	CORA	CITESEER	SQUIRREL	CHAMELEON
Polynomial (d=2) [21]	81.35 (2.11)	69.81 (2.14)	38.57 (1.56)	53.88 (1.94)
Polynomial (d=10) [21]	80.44 (1.56)	68.71 (1.67)	38.08 (1.10)	55.20 (1.97)
Exp [21]	80.42 (2.14)	68.81 (1.34)	38.06 (1.61)	55.50 (2.11)
ANOVA [3]	83.26 (0.78)	70.09 (2.44)	40.77 (1.46)	56.01 (1.86)
RFF	87.00 (1.24)	74.77 (2.00)	52.23 (1.19)	68.55 (1.87)
8.9 RFF ablation study at 128 dimensions
We replicate the analysis conducted in 6.2 for 128 dimensions, as outlined in Table 13. It is apparent that our approach consistently outperforms or matches most other methods when operating at 32 dimensions across various datasets. This trend persists even after projecting the learned embeddings to higher dimensions through RFF. Our experiment reaffirms the observations detailed in 6.2.
Table 13: Node classification accuracy percentages with and without using Random Fourier Feature projections (on 128 dimensions). A higher number means better performance. The performance is improved by using RFF in almost all cases, indicating the usefulness of this transformation.
RFF	CORA	CITESEER	SQUIRREL	CHAMELEON
DGI	×	84.99 (1.36)	72.22 (2.50)	34.22 (1.47)	49.82 (2.96)
✓	84.17 (2.11)	72.65 (1.52)	37.97 (1.41)	57.72 (2.03)
MVGRL	×	85.31 (1.66)	73.42 (1.63)	36.92 (1.04)	55.20 (1.70)
✓	84.61 (1.74)	72.81 (2.13)	38.73 (1.22)	57.81 (1.80)
SUGRL	×	71.49 (1.15)	63.85 (2.27)	38.04 (1.17)	53.03 (1.73)
✓	71.40 (1.40)	63.06 (2.22)	43.24 (1.63)	57.04 (1.78)
GRACE	×	80.87 (1.49)	62.52 (3.57)	41.25 (1.32)	63.14 (1.89)
✓	79.70 (1.91)	64.47 (2.12)	52.29 (1.81)	68.90 (2.05)
FiGURRe128	×	84.73 (1.13)	73.07 (1.13)	41.06 (1.51)	59.08 (3.36)
✓	86.14 (1.13)	73.34 (1.91)	48.78 (2.48)	66.03 (2.19)
8.10 RFF ablation study at 512 dimensions
We present the results obtained by applying RFF to 512 dimensions, as shown in Table 14. It is evident that there is minimal performance improvement when using RFF on embeddings that are already of sufficient dimensionality.
Table 14:FiGURe at 512 dimensions
CORA	CITESEER	SQUIRREL	CHAMELEON
RFF@512	86.84 (0.98)	74.40 (1.30)	51.86 (1.87)	68.60 (1.57)
8.11 Issues with including RFF in training
In the context of incorporating Random Feature techniques like RFF into machine learning models, some challenges have arisen in the training process. In the Transformer attention module, these features can exhibit large variances in zero kernel score regions, leading to unstable training [8]. Additionally, increasing the sampling rate in methods like Performer and Random Feature Attention (RFA) [35] can also introduce instability, as highlighted in [39]. Another area where the use of random features with deep networks has been explored is kernel learning. There are of course, computational and convergence challenges that have been observed with these methods as well [11][12]. FiGURe avoids these problems by excluding the random feature generation from training altogether. Parameters like gamma, involved in the random feature construction, are treated as hyperparameters and a search is performed over them. Kernel learning, however, is an important area of research, and integrating RFF in the training loop would be an interesting extension of this work.
8.12 Computational Comparisons with other Other Unsupervised Methods