SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Namyong Park $^{1*}$ , Ryan Rossi $^{2}$ , Xing Wang $^{1}$ , Antoine Simoulin $^{1}$ , Nesreen Ahmed $^{3}$ , Christos Faloutsos $^{4}$
<sup>1</sup>Meta AI <sup>2</sup>Adobe Research <sup>3</sup>Intel Labs <sup>4</sup>Carnegie Mellon University
The choice of a graph learning (GL) model (i.e., a GL algorithm and its hyperparameter settings) has a significant impact on the performance of downstream tasks. However, selecting the right GL model becomes increasingly difficult and time consuming as more and more GL models are developed. Accordingly, it is of great significance and practical value to equip users of GL with the ability to perform a near-instantaneous selection of an effective GL model without manual intervention. Despite the recent attempts to tackle this important problem, there has been no comprehensive benchmark environment to evaluate the performance of GL model selection methods. To bridge this gap, we present GLEMOS in this work, a comprehensive benchmark for instantaneous GL model selection that makes the following contributions. (i) GLEMOS provides extensive benchmark data for fundamental GL tasks, i.e., link prediction and node classification, including the performances of 366 models on 457 graphs on these tasks. (ii) GLEMOS designs multiple evaluation settings, and assesses how effectively representative model selection techniques perform in these different settings. (iii) GLEMOS is designed to be easily extended with new models, new graphs, and new performance records. (iv) Based on the experimental results, we discuss the limitations of existing approaches and highlight future research directions. To promote research on this significant problem, we make the benchmark data and code publicly available at https://namyongpark.github.io/glemos.
Graph learning (GL) methods [43, 48] have achieved great success across multiple domains and applications that involve graph-structured data [4, 10, 17, 21, 28, 29, 31, 34]. At the same time, previous studies [30, 40, 46] have shown that there is no universally good GL model that performs best across
all graphs and graph learning tasks. Therefore, to effectively employ GL given a wide array of available models, it is important to select the right GL model (i.e., a GL algorithm and its hyperparameter settings) that will perform well for the given graph data and GL task.
Ideally, we would want to be able to select the best GL model for the given graph near-instantaneously, that is, without having to train or evaluate different models multiple times on the new graph, since even a few such training and evaluations might take a considerable
Instantaneous Graph Learning Model Selection
Figure 1: Via instantaneous graph learning model selection, the best model can be found without performing computationally expensive model training and evaluations.
*Correspondence: namyongp@meta.com
amount of time and resources (Figure 1). Enabling an instantaneous model selection for a completely new graph involves addressing several technical challenges, which includes modeling how well different GL methods perform on various graphs, and establishing a connection between the new graph and observed graphs, such that the best model for the new graph can be estimated in light of observed model performances on similar graphs.
Problem Formulation. With these considerations, we formally define this important problem, which we call Instantaneous Graph Learning Model Selection, and the related terms as follows.
Graph: Let  $G = (V, E, X, Y)$  be a graph where  $V \subseteq \mathbb{N}$  and  $E = \{(i, j) \mid i, j \in V\}$  denote the sets of nodes and edges, respectively;  $X$  denotes the input features, which can be node features ( $X \in \mathbb{R}^{|V| \times F_N}$ ), edge features ( $X \in \mathbb{R}^{|E| \times F_E}$ ), or a set of both, where  $F_N$  and  $F_E$  denote the dimension of corresponding input features; and  $Y$  denotes node labels ( $Y \in \mathbb{N}^{|V|}$ ) or edge labels ( $Y \in \mathbb{N}^{|E|}$ ). Note that input features  $X$  and labels  $Y$  are considered optional since not all graphs have this information.
Model: A model  $M$  refers to a GL method for the given GL task, such as link prediction, with specific hyperparameter settings. In general, a GL model consists of two components, namely, (graph embedding method, hyperparameters) and (predictor, hyperparameters), where the former produces a vector representation of the graph (e.g., node embeddings) and the latter makes task-specific predictions (e.g., link prediction) given the embeddings. The set  $\mathcal{M}$  of models, from which the model selection is made, is normally heterogeneous, where the configuration of each model is unique in the choice of its two components and their hyperparameter settings.
Performance Matrix: Let  $\mathbf{P} \in \mathbb{R}^{n \times m}$  be a matrix containing observed model performances, where  $P_{ij}$  is the performance (e.g., accuracy) of model  $j$  on graph  $i$ .  $\mathbf{P}$  can be sparse with missing entries.
Problem 1 (Instantaneous Graph Learning Model Selection).
- a training meta-corpus of  $n$  graphs  $\mathcal{G} = \{G_i\}_{i=1}^n$  and  $m$  models  $\mathcal{M} = \{M_j\}_{j=1}^m$  for a GL task (e.g., link prediction and node classification):
(1) performance matrices  $\{\mathbf{P}_k\}_{k = 1}^{\ell}$ , i.e.,  $\ell$  records of  $m$  models' performance on  $n$  graphs
(2) input features of the graphs in  $\pmb{G}$  (if available)
(3) configurations (i.e., a GL method and its hyperparameter settings) of  $m$  models in  $\mathcal{M}$
- an unseen test graph  $G_{\mathrm{test}} \notin \mathcal{G}$
- the best model  $M^{*} \in \mathcal{M}$  for  $G_{\mathrm{test}}$  without training or evaluating any model in  $\mathcal{M}$  on  $G_{\mathrm{test}}$ .
Status Quo and Our Contributions. In recent years, several methods have been developed for an efficient selection of GL models. However, most of them cannot tackle Prob. 1 as they require multiple rounds of model training and evaluations; we review these methods in Sec. 2. Most recently, a subset of Prob. 1 was studied by MetaGL [30], which proposed a GL model selection technique that assumes plain graphs without input features, and operates without utilizing model configurations. A few recent works [5, 32, 46] also provide performances of graph neural networks (GNNs), although they cannot address Prob. 1. While the datasets used in [5, 30, 32, 46] are available, they fall short of being a comprehensive benchmark environment to study this significant problem due to the following reasons.
- Limited GL Task and Data. Focusing on link prediction, MetaGL [30] only provides link prediction performances, and does not support other widely-used tasks, such as node classification, which limits follow-up studies and use of the benchmark for different GL tasks. Also, other related works [5, 32, 46] are limited in terms of the number and diversity of graphs they cover (Table 1).
- Limited Evaluation Settings. Some important evaluation settings were not considered in MetaGL's benchmark, such as out-of-domain and small-to-large settings as we later describe, which can be useful in evaluating the performance of model selection techniques in different practical settings.
- Limited Extensibility. The sets of models and graphs are assumed to be fixed, and it is not easy to extend the benchmark with new graphs and models in a consistent and reproducible manner.
In this work, we address these limitations by developing a comprehensive benchmark for instantaneous graph learning model selection. Overall, the contributions of this work are as follows.
- Extensive Benchmark Data with Multiple GL Tasks. We construct a benchmark dataset that includes the performances of 366 models on 457 graphs over fundamental GL tasks, i.e., link pre
Figure 2: GLEmos provides a comprehensive benchmark environment, covering the steps required to achieve effective instantaneous GL model selection, with multiple options for major building blocks.
diction and node classification, which is by far the largest benchmark for Prob. 1 to our knowledge. The benchmark also provides meta-graph features to capture the structural characteristics of graphs.
- Comprehensive Evaluation Testbeds. We evaluate ten representative methods for Problem 1, including both classical methods and deep learning-based ones, using multiple evaluation settings designed to assess the quality of model selection techniques from practical perspectives.
- Extensible Open Source Benchmark Environment. Our benchmark is designed to be easily extended with new models, new graphs, and new performance records. To promote further research on this significant problem, we make the benchmark environment publicly available.
- Future Research Directions. We discuss the limitations of existing model selection methods, and highlight future research directions towards an instantaneous selection of graph learning models.
After reviewing related work in Section 2, we present the proposed benchmark data and testbeds in Sections 3 to 5. Then we provide experimental results in Section 6, and conclude in Section 7.
2.1 Model Selection
Model selection refers to the process of selecting a learning algorithm and its hyperparameter settings. In this section, we review existing model selection approaches, which we divide into two groups depending on whether they require model evaluations (i.e., performance queries for the new dataset).
Evaluation-Based Model Selection: Most existing approaches to select machine learning models belong to this group, ranging from simple solutions, such as random search [2] and grid search [23], to more advanced and efficient ones that employ techniques such as adaptive resource allocation [22], early stopping [12], and Bayesian optimization [9, 33, 42]. Inspired by these advancements, several model selection methods were recently developed for graph learning (GL) models. To tackle challenges involved with GL model selection, these methods adapt existing ideas to GL models, such as reinforcement learning [11, 20, 50], evolutionary algorithm [3], Bayesian optimization [36], and hypernets [52], as well as developing techniques specific to graph data, e.g., subgraph sampling [36] and graph coarsening [14]. Note that all of the above approaches cannot tackle the instantaneous GL model selection problem (Problem 1) as they rely on multiple model evaluations for performance queries of different combinations of GL methods and hyperparameter settings on the new dataset.
Instantaneous Model Selection: To select the best model without querying model performances on the new dataset, methods in this category typically utilize prior model performances or characteristic features of a dataset (i.e., meta-features). A simple approach [1] finds the globally best model (i.e., the one with the overall best performance over all observed datasets), and thus its model selection is independent of query datasets. This can be refined by narrowing the search scope to similar datasets, where dataset similarities are modeled in the meta-feature space, e.g., using  $k$ -nearest neighbors [27] or clustering [18]. Another line of methods [30, 45, 49] take a different approach, which aims to predict the model performance on the given dataset by learning a function that maps meta-features into estimated model performances. Due to their ability to learn such a function in a data-driven manner, this second group of methods generally outperformed the first group in previous studies [30, 49]. While the above methods are one of the first efforts to achieve instantaneous GL model selection, several open challenges remain to be solved, as we discuss in Section 6.2.
Table 1: Comparison of GLEMOS with previous works providing performances of GNN models.
Benchmark Testbeds	Instantaneous Selection Methods	Meta-Graph Features	Graph Learning Models	# Graph Datasets	Graph Size (max # nodes)	# Data Domains
GNN-Bank-101 [5]	X	X	X	GNNs	12	34k	5
NAS-Bench-Graph [32]	X	X	X	GNNs	9	170k	4
GraphGym [46]	X	X	X	GNNs	32	34k	7
GLEMOS (Ours)	✓	✓	✓	GNNs & Non-GNNs (e.g., node2vec, label prop.)	457	496k	37
Table 2: Summary statistics of the GLEmos benchmark.
Node Classification Task	Link Prediction Task
Total performance evaluations (Section 3) (# model performances on benchmark graphs)	41,856	152,070
Total graphs (Sections 3.2 and 3.3)	128	457
• Num nodes	34–421,961	34–495,957
• Num edges	156–7,045,181	156–7,045,181
• Num node feats	2–61,278	2–61,278
• Num node classes	2–195	N/A
• Num graph domains	25	37
Total GL models (Sections 3.2 and 3.3)	327	350
Total meta-graph features (Section 4)	58–1,074	58–1,074
Total model selection methods (Section 5)	10	10
Total benchmark testbeds (Section 5)	5	5
2.2 Benchmarks for Instantaneous Graph Learning Model Selection
A few recent works [5, 32, 46] address problems related to GL model selection, and provide performances of GNNs on different datasets. However, all of them perform evaluation-based model selection discussed above, which requires multiple rounds of model evaluations given a new dataset.
As the first benchmark for instantaneous GL model selection (Prob. 1), GLEMOS provides more than just a collection of performance records, i.e., (1) benchmark testbeds and (2) existing algorithms (Section 5) for instantaneous model selection, as well as different sets of (3) meta-graph features (Section 4). These features (1)-(3) are not provided by these previous works [5, 32, 46]. Furthermore, GLEMOS provides more comprehensive and diverse performance records than these works in several aspects (e.g., in terms of included GL models and graph data distributions), as summarized in Table 1.
The two major components for instantaneous GL model selection are historical model performances of the GL task of interest (e.g., accuracy for node classification), and meta-graph features to quantify graph similarities. For each component, GLEMOS provides several options to choose from. Once these components are chosen, users select a model selection algorithm, as well as a benchmark testbed to perform evaluation, out of several options available in GLEMOS. Fig. 2 summarizes these steps to use GLEMOS. In the next sections, we describe what GLEMOS provides for these steps.
3 Graph Learning Tasks and Performance Collection
Prior model performances play an essential role in instantaneous GL model selection algorithms, as they can estimate a candidate GL model's performance on the new graph based on its observed performances on similar graphs. GLEMOS provides performance collections for two fundamental graph learning tasks, i.e., node classification and link prediction. Below we discuss how the graphs and models are selected, and describe how model performances are evaluated for each GL task.
3.1 Graphs and Models
Graphs. Our principle of selecting the graphs in GLEMOS is to include diverse graph datasets, in terms of both the size and domain of the graph. The size of selected graphs ranges from a few hundred edges to millions of edges, and the graph set covers various domains, e.g., co-purchase networks, protein networks, citation graphs, and road networks. As listed in Table 1, the resulting graph set outperforms existing data banks in terms of the number and size of graphs, as well as the diversity of data domain. Table 2 shows the summary statistics of graphs, and the graph list is given in Appendix.
Models. Our principle for selecting the models to include in GLEMOS is to cover representative and widely-used GL methods. We include graph neural network methods (e.g., GCN [19], GAT [37], and SGC [41]), random walk-based node embeddings (e.g., node2vec [13]), self-supervised graph representation learning methods (e.g., DGI [38]), and classical methods (e.g., spectral embedding [25]). The resulting model set is more diverse than previous works, which considered GNNs alone (Table 1).
3.2 Node Classification
Graph Set. A subset of the graphs have node labels. Excluding the graphs without node labels, the node classification graph set is comprised of 128 graphs from 25 domains.
Table 3: Graph learning methods and their hyperparameter settings that comprise the model set  $\mathcal{M}$ . NC: Applicable for node classification. LP: Applicable for link prediction.
Method	NC LP Hyperparameter Settings	Count
GCN [19]	✓	✓ act a ∈ {relu, tanh, elu}, dropout d ∈ {0.0, 0.5}, hidden channels h ∈ {16, 64}, num layers ℓ ∈ {1, 2, 3}	30
GraphSAGE [15]	✓	✓ act a ∈ {relu, tanh}, aggr g ∈ {mean, max}, hidden channels h ∈ {16, 64}, jumping knowledge j ∈ {none, last}, num layers ℓ ∈ {1, 2}	24
GAT [37]	✓	✓ concat c ∈ {true, false}, dropout d ∈ {0.0, 0.5}, heads n ∈ {1, 4}, hidden channels h ∈ {16, 64}, num layers ℓ ∈ {1, 2, 3}	40
GIN [44]	✓	✓ eps e ∈ {0.0}, hidden channels h ∈ {16, 64}, num layers ℓ ∈ {1, 2, 3}, train eps t ∈ {true, false}	10
EGC [35]	✓	✓ aggregators a ∈ {[sum], [mean], [symnorm], [min], [max], [var], [std]}, hidden channels h ∈ {16, 64}, num bases b ∈ {4, 8}, num layers ℓ ∈ {2}	28
SGC [41]	✓	✓ bias b ∈ {true, false}, num hops k ∈ {1, 2, 3, 4, 5}	10
ChebNet [8]	✓	✓ Chebyshev filter size k ∈ {1, 2, 3}, hidden channels h ∈ {16, 64}, normalization r ∈ {none, sym, rw}, num layers ℓ ∈ {1, 2}	27
PNA [7]	✓	✓ aggregators a ∈ {[sum], [mean], [max], [var]}, hidden channels h ∈ {16}, num layers ℓ ∈ {1, 2}, scalars s ∈ {[identity], [amplification], [attenuation], [linear]}, towers t ∈ {1}	32
Spectral Emb. [25]	✓	✓ num components h ∈ {16, 64}, tolerance t ∈ {0.1, 0.01, 0.001, 0.0001}	8
GraRep [6]	✓	✓ num components h ∈ {16, 32, 64}, power p ∈ {1, 2}	6
DGI [38]	✓	✓ encoder act a ∈ {prelu, relu, tanh}, hidden channels h ∈ {16, 64}, summary s ∈ {mean, max, min, var}	24
node2vec [13]	✓	✓ context size w ∈ {5, 10}, hidden channels h ∈ {16, 64}, p ∈ {1, 2, 4}, q ∈ {1, 2, 4}, walk length l ∈ {10, 20}	72
Label Prop. [53]	✓	alpha α ∈ {0.99, 0.9, 0.8, 0.7}, num layers ℓ ∈ {1, 2, 3, 4}	16
Jaccard's Coeff. [24]	✓	-	1
Resource Alloc. [51]	✓	-	1
Adamic/Adar [24]	✓	-	1
SEAL [47]	✓	GNN conv c ∈ {GCN, SAGE, GAT}, GNN hidden channels g ∈ {16, 64, 128}, k ∈ {0.6, 0.1}, MLP hidden channels m ∈ {32, 128}, num hops n ∈ {1}	36
Total Count 366
Model Set. Most methods in GLEMOS are applicable for both node classification and link prediction. In addition to these common methods, we also include label propagation [53], which can be used for node classification. The GL models evaluated for node classification and their hyperparameter settings are listed in Table 3. In total, 327 models comprise our model set for node classification.
Performance Collection. For node classification, supervised models are optimized to produce the class distribution. For unsupervised models, we first train them to produce latent node embeddings based on their own objective, and apply a trainable linear transform to transform embeddings into the class distribution. More details on the experimental settings are given in the Appendix. To evaluate performance, we calculate multiple classification metrics, including accuracy, F1 score, average precision, and ROC AUC score. Given the graph set  $\mathcal{G}$  and model set  $\mathcal{M}$  described above, we construct the performance matrix  $\mathbf{P}$  by evaluating every model  $M_j \in \mathcal{M}$  on every graph  $G_i \in \mathcal{G}$ , i.e.,
$$
P _ {i j} = \text {P e r f o r m a n c e (e . g . , a c c u r a c y , a n d R O C A U C) o f m o d e l M} _ {j} \in \mathcal {M} \text {o n g r a p h G} _ {i} \in \mathcal {G}. \tag {1}
$$
Splitting: We generate the train-validation-test node splits with a ratio of  $64\% - 16\% - 20\%$ , respectively, and train each model applying validation-based early stopping. For reproducibility, we release all data splits, such that future model evaluations can be done using the same node splits.
3.3 Link Prediction
Graph Set. As link prediction task does not require node labels for evaluation, we greatly expand the graph set used for node classification by adding 329 more graphs. With these graphs, the link prediction graph set consists of 457 graphs from 37 domains. The full list is given in the Appendix.
Model Set. All models used for node classification are used for link prediction, except for label propagation, which requires node labels. We also add models designed for link prediction, e.g., SEAL [47] and Adamic/Adar [24]. In total, 350 models comprise the link prediction model set(Table 3).
Performance Collection. For link prediction, GL models are optimized to produce latent node embeddings, and we apply a dot product scoring between the two node embeddings, followed by a sigmoid function, to obtain the link probability between the corresponding nodes. We calculate multiple evaluation metrics to measure the link prediction performance, including average precision, ROC AUC score, and NDCG (normalized discounted cumulative gain) [39].
Splitting: We randomly split edges into train-validation-test sets, with a ratio of  $64\% - 16\% - 20\%$ , which form positive edge sets. For positive edges, we randomly select the same amount of negative edges (i.e., nonexistent edges), which form negative edge sets. Again, we release all edge splits.
4 Collection of Meta-Graph Features
Instantaneous model selection algorithms carry over historical model performances on various graphs to estimate how GL models would perform on a new graph. In that process, performance transfer can be done more effectively when we consider graph similarities, such that the performance transfer would be done adaptively based on the similarities between graphs. Structural meta-graph features provide an effective way to that end by summarizing a graph into a fixed-size feature vector in terms of its structural characteristics. GLEMOS provides various meta-graph features, which can capture important graph structural properties. Below we first discuss how GLEMOS generates fixed-length meta-graph features, as depicted in Figure 3, and then describe the structural features included in GLEMOS, which are organized into three sets for convenience.
Feature Generation. GLEMOS produces meta-features in two steps, following an earlier work [30].
- Step 1—Structural Feature Extraction: A structural meta-feature extractor  $\psi_{k}$  is a function that transforms an input graph  $G$  into a vector, which represents a distribution of structural features over nodes or edges. For example, degree and PageRank scores of all nodes correspond to node-level feature distributions, and triangle frequency for each edge corresponds to an
Figure 3: Meta-graph features summarize structural graph characteristics into a fixed-size feature vector.
edge-level feature distribution. In general, we apply a set of such extractors  $\Psi = \{\psi_1,\dots ,\psi_K\}$  to the graph  $G$  obtaining a set  $\Psi (G) = \{\psi_{1}(G),\ldots ,\psi_{K}(G)\}$  of multiple feature distributions.
$\circ$  Step 2—Statistical Feature Summarization: Since the number of nodes or edges in each graph determines the size of the output from the meta-feature extractors  $\psi_{k}(G)$ , those structural feature distributions cannot be directly used to compare graphs with different number of nodes or edges. Step 2 addresses this issue via statistical feature summarization, which applies a set  $\Sigma$  of statistical functions (e.g., mean, entropy, skewness, etc) that summarize feature distributions  $\psi_{k}(G)$  of varying size into fixed-length feature vectors; i.e.,  $\dim(\Sigma(\psi_{k}(G_{i}))) = \dim(\Sigma(\psi_{k}(G_{j})))$  for two graphs  $G_{i}$  and  $G_{j}$ . By combining all  $K$  summaries, graph  $G$ ’s meta-graph feature  $\mathbf{m}$  is obtained to be  $\mathbf{m} = [\Sigma(\psi_{1}(G)); \dots; \Sigma(\psi_{K}(G))]$ . The statistical functions  $\Sigma$  used in GLEMOS are listed in Appendix.