SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 15: Mean epoch time (in milliseconds) averaged across 20 trials with different hyperparameters. A lower number means the method is faster. Even though our method is slower at 512 dimensions, using 128 and 32 dimensional embeddings significantly reduces the mean epoch time. Using RFF as described in 6.2 we are able to prevent the performance drops experienced by SUGRL and GRACE.
SUGRL	GRACE	FiGURe512	FiGUReRFF128	FiGURe32ff
CORA	15.92 (4.10)	51.19 (6.8)	114.38 (0.51)	20.10 (0.46)	11.54 (0.34)
CITESEER	24.37 (4.92)	77.16 (7.2)	156.24 (0.56)	30.30 (0.60)	17.16 (0.51)
SQUIRREL	33.63 (6.94)	355.2 (67.34)	257.65 (0.76)	47.72 (1.40)	23.52 (1.14)
CHAMELEON	16.91 (5.90)	85.05 (14.1)	98.36 (0.64)	18.56 (0.39)	11.63 (0.48)
In Section 6.4, we compared the computational time of FiGUR e with MVGRL and DGI, as all three methods fall under the category of unsupervised methods that preform contrastive learning with representations of the entire graph. However, there is another class of methods, such as SUGRL and GRACE, that contrast against other nodes without the need for graph representation computation. Consequently, these methods exhibit higher computational efficiency. Hence, as show in Table 15 upon initial inspection, it appears that SUGRL (at 512 dimensions) exhibits the highest computational efficiency, even outperforming  $\mathrm{FiGUR}e_{128}^{\mathrm{RFF}}$ . However, despite its computational efficiency, the significant drop in performance across datasets (as discussed in Section 6.1) renders it less favorable for consideration. In fact,  $\mathrm{FiGUR}e_{32}^{\mathrm{RFF}}$  offers computational cost savings compared to SUGRL, while also achieving significantly better downstream classification accuracy. Turning to GRACE, it demonstrates greater computational efficiency than  $\mathrm{FiGUR}e_{512}$  for low to medium-sized graphs. However, as the graph size increases, due to random node feature level masking and edge level masking, the computational requirements of GRACE substantially increase (as evidenced by the results on SQUIRREL). Therefore, for larger graphs with more than approximately 5,000 nodes, FiGUR e proves to be more computationally efficient than GRACE (even at 512 dimensions). Furthermore, considering the performance improvements exhibited by FiGUR e, it is evident that FiGUR e (combined with RFF projections) emerges as the preferred method for unsupervised contrastive learning in graph data.
8.13 Time and storage cost
In Table 16, we present a comparison of training time and storage cost between our method and DGI. To ensure fairness in the evaluation, we maintained consistent settings across all baselines. The maximum number of epochs was uniformly set to 30,000, with an early stopping patience of 20 epochs, except for larger datasets where we extended the patience to 500 epochs. It's important to note that due to varying hyperparameter tuning among the baselines, normalizing the number of epochs posed some challenges. Nevertheless, we ensured that a sufficiently high number of epochs were employed to facilitate convergence. Our approach builds upon DGI and exhibits linear scaling of training time with the number of filters used. Currently, our model employs three filters for training, resulting in a training time three times longer than that of DGI. Specifically for OGBN-ARXIV, we provide information on the number of epochs, mean epoch time, and total training time. Additionally, we report the storage cost of representations generated by these methods. Please keep in mind that the linear relationship in training time between DGI and FiGUR is not apparent here, as it factors in considerations such as batching and sampling time.
Table 16: We provide the number of epochs, mean epoch time, and total training time for OGBN-ARXIV. We also report the storage cost of the representations from these methods.
Model / Dims	Num Epochs	Mean Epoch Time	Total time	Storage
DGI / 512	3945	0.77s	50.75mins	330.75MB
FiGUR e / 512	4801	0.92s	73.62mins	1.32GB
FiGUR e / 128	4180	0.74s	51.55mins	330.75MB
FiGUR e / 32	3863	0.72s	46.36mins	82.69MB
8.14 Choice of Filter Banks
In Section 4.1, we explore the flexibility of FiGURe to accommodate various filter banks. When making a choice, it is crucial to examine the intrinsic properties of the filters contained within different filter banks. We pick two filter banks  $\mathbf{F}_{\mathrm{BERNNET}}$  and  $\mathbf{F}_{\mathrm{GPRGNN}}$  and provide an overview of the filters contained in the filter banks. We use these two filter banks as examples to illustrate what should one be looking for, while choosing a filter bank.
Bernstein Polynomials: Figure 5 illustrates that as the number of Bernstein Basis increases, the focus on different parts of the eigenspectrum also undergoes changes. With an increase in polynomial order, two notable effects can be observed. Firstly, the number of filters increases, enabling each filter to focus on more fine-grained eigenvalues. This expanded set of polynomial filters allows for a more detailed examination of the eigenspectrum. Secondly, if we examine the first and last Bernstein polynomials, we observe an outward shift in their shape. This shift results in the enhancement of a specific fine-grained part at the ends of the spectrum. These observations demonstrate that Bernstein polynomials offer the capability to selectively target and enhance specific regions of interest within the eigenspectrum
Standard Basis: Figure 5 reveals two key observations. Firstly, at a polynomial order of 2, the standard basis exhibit focus at the ends of the spectrum, in contrast to the behavior of Bernstein polynomials, which tend to concentrate more on the middle of the eigenspectrum. This discrepancy highlights the distinct characteristics and emphasis of different polynomial bases in capturing different parts of the eigenspectrum. Secondly, as the number of polynomials increases (in contrast to Bernstein polynomials), the lower order polynomials remain relatively unchanged. Instead, additional polynomials are introduced, offering a more fine-grained focus at the ends of the spectrum. This expansion of polynomials allows for a more detailed exploration of specific regions of interest within the the ends of eigenspectrum.
In the context of filter banks, previous studies [28, 7] have demonstrated that certain datasets, such as SQUIRREL and CHAMELEON, benefit from frequency response functions that enhance the tail ends of the eigenspectrum. This observation suggests that the standard basis, which naturally focuses on the ends of the spectrum, may outperform Bernstein basis functions at lower orders. However, as the order of the Bernstein basis increases, as discussed in 4.1, there is a notable improvement in performance. This can be attributed to the increased focus of Bernstein basis functions on specific regions, particularly the ends of the spectrum. As a result, higher-order Bernstein filters exhibit enhanced capability in capturing important information in those regions. It is worth noting that the choice between  $\mathbf{F}_{\mathrm{GPRGNN}}$  and  $\mathbf{F}_{\mathrm{BERNNET}}$  depends on the specific requirements of the downstream task. If the task necessitates a stronger focus on the middle of the spectrum or requires a band-pass or comb-like frequency response,  $\mathbf{F}_{\mathrm{BERNNET}}$  is likely to outperform  $\mathbf{F}_{\mathrm{GPRGNN}}$ . Thus, the selection of the appropriate filter bank should be based on the desired emphasis on different parts of the eigenspectrum. Regarding the performance comparison between  $\mathbf{F}_{\mathrm{BERNNET}}$  and  $\mathbf{F}_{\mathrm{GPRGNN}}$ , it is plausible that as we increase the order of the Bernstein basis, the performance could potentially match that of  $\mathbf{F}_{\mathrm{GPRGNN}}$ . However, further investigation and experimentation are required to determine the specific conditions and orders at which this convergence in performance occurs.
(a) Five Bernstein Basis
(b) Five Standard Basis
(c) Seven Bernstein Basis
(d) Seven Standard Basis
(e) Nine Bernstein Basis
(f) Nine Standard Basis
(g) Eleven Bernstein Basis
(h) Eleven Standard Basis
Figure 5: The figures contain the Bernstein basis as well as standard basis for different degrees. The x-axis of the figures represents the eigenvalues of the Laplacian matrix, while the y-axis represents the magnitude of the polynomials. It is important to note that while plotting the standard polynomials, they are computed with respect to the Laplacian matrix  $(\mathbf{L}_{\mathbf{n}})$  rather than the adjacency matrix. As a result, the eigenvalues lie between [0, 2]. On the other hand, the Bernstein polynomials are typically defined for the normalised Laplacian matrix, and therefore there is no change in the eigenvalue range (the eigenvalues of the normalised Laplacian matrix typically range from 0 to 2). By using the Laplacian matrix as the basis for plotting the polynomials, we can observe the behavior and magnitude of the polynomials at different eigenvalues, providing insights into their spectral properties and frequency response characteristics.