SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Collection of Meta-Graph Features. Different graph features may capture different structural properties. Thus, GLEMOS aims to provide representative and diverse graph features, which have been proven effective in earlier studies, while making it easy to work with any set of features. For the convenience of the users, we group the currently supported features into the following three sets:  $M_{\text{regular}}$  includes widely used features that capture structural characteristics of a graph at both node and graph levels;  $M_{\text{graphlets}}$  considers features based on the frequency of graphlets, as they can provide additional information;  $M_{\text{compact}}$  is intended to use the least space, while providing several important features that capture node-, edge-, and graph-level characteristics. The details of each set are as follows.
$M_{\text{regular}}$ : This set includes 318 meta-graph features. We derive the distribution of node degrees, k-core numbers, PageRank scores, along with the distribution of 3-node paths (wedges) and 3-node cliques (triangles). Given these five distributions, we summarize each using the set of 63 statistical functions  $\Sigma$ , giving us a total of 315 features. We include three additional features based on the density of graph  $G$  and the density of the symmetrized graph, along with the assortativity coefficient.
$M_{\text{graphlets}}$ : This set includes 756 meta-graph features. First, we derive the frequency of all 3 and 4-node graphlet orbits per edge in  $G$ . Next, we summarize each of the 12 graphlet orbit frequency distributions using the set of 63 statistical functions  $\Sigma$ , giving us a total of 756 meta-graph features.
$M_{\text{compact}}$ : This set consists of 58 total meta-graph features, including 9 simple statistics such as number of nodes and edges, density of the graph  $G$ , max vertex degree, average degree, assortativity coefficient, and the maximum k-core number of  $G$ , along with the mean and median k-core number of a vertex in  $G$ . We also include the global clustering coefficient, total triangles, as well as the mean and median number of triangles centered at an edge. We further include the total 4-cliques as well as the mean and median number of 4-cliques centered at an edge. Besides the above 16 features, we also compute the frequency of all 3 and 4-node graphlet orbits per edge, and from these 12 frequency distributions, we derive the mean, median, and max. We also derive the graphlet frequency distribution from the counts of all six 4-node graphlets and include those values directly as features.
Note that the framework is flexible, and users can choose to use any set of features, either a subset of the current features (e.g., to further improve efficiency and use less space), or their superset (e.g., to capture distinct structural characteristics using different features in addressing new tasks).
5 Benchmark Testbeds and Algorithms
5.1 Benchmark Testbeds
GLEmos provides multiple benchmark testbeds (i.e., evaluation settings and tasks) designed to assess model selection performance in different usage scenarios. We describe them in detail below.
Fully-Observed Testbed. In this setup, model selection algorithms are provided with a full performance matrix  $\mathbf{P}$  for the given graph learning task, i.e., without any missing entry in  $\mathbf{P}$ . Accordingly, this testbed measures model selection performance in the most information-rich setting, where all models in the model set  $\mathcal{M}$  have been evaluated on all observed graphs.
Splitting: We apply a stratified 5-fold cross validation, i.e., graphs are split into five folds, which are (approximately) of the same size, and balanced in terms of graph domains, and then as each fold  $(20\%)$  is held out to be used for testing, the other folds  $(80\%)$  are used for model training. Note that graph splits are used to split the performance matrix  $\mathbf{P}$  and meta-graph features  $\mathbf{M}$ .
Sparse Testbed. The performance matrix  $\mathbf{P}$  in this setting is sparse and partially observed, i.e., we may only have a few observations for each graph. This setting is important since it can be costly to add a new model to the benchmark, which requires training and evaluating the model multiple times on the graphs in the benchmark. By dealing with model selection using a sparse  $\mathbf{P}$ , this testbed addresses significant practical considerations, e.g., making it more cost-effective to be able to add new models to the benchmark. Using this testbed, researchers can develop and test specialized algorithms capable of learning from such partially-observed performances. To construct a sparse performance matrix  $\mathbf{P}'$ , we sample uniformly at random  $pm$  values from each row of  $\mathbf{P}$ , where  $p$  is the fraction of values to sample and  $m$  is the total number of models. This graph-wise sampling strategy ensures the same number of observations for each graph, which matches the practical motivation that we have a limited budget per graph. For this benchmark, we use different sparsity levels  $p \in \{0.1, 0.3, 0.5, 0.7, 0.9\}$ .
Splitting: We use the same stratified 5-fold cross validation as in Fully-Observed testbed. Algorithms are trained using a sparse split of  $\mathbf{P}$ , and evaluated with fully-observed performances of the test split.
Out-Of-Domain Testbed. Graphs from a particular domain (e.g., road graphs, social networks, and brain networks) often have similar characteristics to each other. In other words, graphs from a certain domain can be considered as having its own distribution, which makes model selection for graphs from a new domain a challenging task. This testbed evaluates the effectiveness of model selection methods for such an out-of-distribution setting by holding out graphs from a specific network domain, and trying to predict for the held-out domain by learning from graphs from all the other domains.
Splitting: We use a group-based 5-fold cross validation for this testbed such that each domain appears once in the test set across all folds.
Small-To-Large Testbed. Training a GL model can take a lot of time and resources, especially for large-scale graphs. While model selection methods may benefit from having more prior performances, having to obtain performance records for large graphs presents a significant computational bottleneck. The meta-training process can be made significantly faster by enabling model selection algorithms to learn from relatively small graphs to be able to predict for larger graphs. This testbed focuses on this challenging yet practical setting, which evaluates the ability to generalize from small to large graphs.
Splitting: Graphs with less than  $\epsilon$  nodes form a small-graph set used for training. The other graphs with at least  $\epsilon$  nodes form a large-graph set, which is used for evaluation. We evaluate using a threshold value  $\epsilon$  of 10000 for this testbed.
Cross-Task Testbed. The above testbeds operate on the model performances measured for one specific type of GL task. By contrast, in this testbed, model selection methods learn from performances of one GL task (e.g., node classification), and are evaluated by predicting performances of a different GL task (e.g., link prediction). This task present an additional challenge to model the relation between two different, yet related GL tasks, and utilize the learned knowledge for transferable model selection.
Splitting: We first choose the source and target tasks, and split the graphs into the two sets, i.e., the source task set and the target task set. Then the graphs in the source set are used for training, and the graphs in the target set are used for testing.
5.2 Model Selection Algorithms
GLEmos provides state-of-the-art algorithms for instantaneous model selection, which are listed in Table 4. These algorithms are selected such that the benchmark covers representative techniques
Table 4: GLEMOS provides representative algorithms for instantaneous model selection. Algorithm characteristics denote whether they utilize meta-graph features (C1) and observed model performances (C2) for model selection, and whether they are estimizable (i.e., have trainable parameters) (C3).
Characteristics	Algorithm	Random Selection	GB-Avg Perf [49]	GB-Avg Rank [30]	ISAC [18]	AS [27]	Spv. Surro. (S2) [45]	ALORS [26]	NCF [16]	MetaOD [49]	MetaGL [30]
C1. Use meta-features	X	X	X	✓	✓	✓	✓	✓	✓	✓	✓
C2. Use prior performances	X	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
C3. Optimizable	X	X	X	X	X	✓	✓	✓	✓	✓	✓
Table 5: Fully-Observed testbed results for link prediction (top) and node classification (bottom) tasks. Higher  $(\uparrow)$  scores are better. The best result is in bold, and the second best result is underlined.
(a) Link prediction
Perf. Metric	RandSel	GB-Perf	GB-Rank	ISAC	AS	S2	ALORS	NCF	MetaOD	MetaGL
AUC (↑)	0.524	0.735	0.730	0.807	0.864	0.809	0.843	0.728	0.764	0.875
MRR (↑)	0.016	0.087	0.064	0.134	0.371	0.198	0.201	0.073	0.096	0.295
NDCG@1 (↑)	0.813	0.942	0.934	0.944	0.957	0.950	0.961	0.943	0.937	0.969
(b) Node classification
Perf. Metric	RandSel	GB-Perf	GB-Rank	ISAC	AS	S2	ALORS	NCF	MetaOD	MetaGL
AUC (↑)	0.518	0.747	0.744	0.746	0.762	0.772	0.734	0.745	0.581	0.740
MRR (↑)	0.029	0.102	0.124	0.118	0.181	0.110	0.103	0.124	0.041	0.129
NDCG@1 (↑)	0.747	0.865	0.860	0.885	0.892	0.916	0.886	0.883	0.839	0.863
for model selection, in terms of whether they use meta-graph features (C1, Section 4) and prior model performances (C2, Section 3), and whether they are estimizable with trainable parameters (C3). Random Selection (RandSel) is used as a baseline to see how well model selection algorithms perform in comparison to random scoring. Global Best (GB)-AvgPerf and GB-AvgRank select a model that performed globally well on average. In contrast, ISAC [18] and ARGOSMART (AS) [27] perform model selection more locally with respect to the given graph, using meta-features. As GB methods rely only on prior performance, comparisons against them can help with investigating the effectiveness of meta-graph features. Supervised Surrogates (S2) [45], ALORS [26], NCF [16], MetaOD [49], and MetaGL [30] are estimizable algorithms, which learn to estimate model performance by capturing the relation between meta-features and observed performances. In comparison to the simpler, non-optimizable algorithms above, we can investigate the advantages of different optimization components for instantaneous model selection. A more detailed description of each algorithm is given in Appendix.
In this section, we report how model selection methods perform in different testbeds. Based on those observations, we discuss the limitations of existing methods and future research directions.
6.1 Model Selection Performance
Evaluation Protocol. To measure how well model selection methods perform on the testbeds presented in Section 5, we evaluate their top-1 prediction results (i.e., the model predicted to be the best for the query graph) as model selection aims to find the best performing model as accurately as possible. Specifically, top-1 prediction performance is measured in terms of AUC, MAP (mean average precision), and NDCG (normalized discounted cumulative gain), all of which range from zero to one, with larger values indicating a better performance. We apply AUC and MAP by treating the task as a binary classification problem, in which the top-1 model is labeled as one, and all other models are labeled as zero. For NDCG, we report NDCG@1, which evaluates the ranking quality of the top-1 model. We evaluate these metrics multiple times for the data splits each testbed provides, and report the averaged performance. For reproducibility, GLEmos provides the data splits of all testbeds.
Fully-Observed Testbed (Table 5). Comparison between methods where meta-graph features are either used (e.g., AS, MetaGL) or not used (e.g., GB-Perf) shows the benefits of utilizing meta-graph features for GL model selection. While estimizable methods (e.g., NCF, MetaOD) have the additional flexibility to adaptively tune their behavior based on data, they are outperformed by relatively simple methods like ISAC and AS. At the same time, the best results on link prediction in the majority of metrics are achieved by another estimizable method, MetaGL, which shows the promising potential of estimizable framework for model selection. In node classification results, the performance decrease
Table 6: Sparse testbed results for link prediction (top) and node classification (bottom) tasks. Higher  $(\uparrow)$  scores are better. The best result is in bold, and the second best result is underlined.
(a) Link prediction
Perf. Metric	Sparsity	RandSel	GB-Perf	GB-Rank	ISAC	AS	S2	ALORS	NCF	MetaOD	MetaGL
AUC (↑)	10%	0.524	0.733	0.732	0.804	0.829	0.813	0.831	0.735	0.743	0.865
30%	0.524	0.728	0.738	0.798	0.763	0.811	0.827	0.739	0.703	0.871
50%	0.524	0.704	0.730	0.790	0.690	0.839	0.814	0.739	0.669	0.866
70%	0.524	0.708	0.730	0.778	0.618	0.814	0.795	0.757	0.630	0.866
90%	0.524	0.717	0.732	0.720	0.547	0.464	0.687	0.656	0.599	0.811
(b) Node classification
Perf. Metric	Sparsity	RandSel	GB-Perf	GB-Rank	ISAC	AS	S2	ALORS	NCF	MetaOD	MetaGL
AUC (↑)	10%	0.518	0.746	0.744	0.744	0.748	0.766	0.727	0.727	0.575	0.761
30%	0.518	0.743	0.738	0.734	0.680	0.769	0.741	0.735	0.533	0.736
50%	0.518	0.726	0.739	0.687	0.592	0.739	0.730	0.713	0.485	0.709
70%	0.518	0.692	0.738	0.653	0.571	0.684	0.694	0.709	0.483	0.662
90%	0.518	0.626	0.697	0.592	0.535	0.620	0.654	0.660	0.490	0.659
Table 7: Out-Of-Domain testbed results for link prediction (top) and node classification (bottom) tasks. Higher  $(\uparrow)$  scores are better. The best result is in bold, and the second best result is underlined.
(a) Link prediction
Perf. Metric	RandSel	GB-Perf	GB-Rank	ISAC	AS	S2	ALORS	NCF	MetaOD	MetaGL
AUC (↑)	0.517	0.809	0.811	0.850	0.786	0.837	0.820	0.837	0.681	0.871
MRR (↑)	0.018	0.110	0.101	0.125	0.237	0.116	0.109	0.109	0.047	0.148
NDCG@1 (↑)	0.820	0.956	0.954	0.951	0.935	0.953	0.953	0.952	0.918	0.951
(b) Node classification
Perf. Metric	RandSel	GB-Perf	GB-Rank	ISAC	AS	S2	ALORS	NCF	MetaOD	MetaGL
AUC (↑)	0.495	0.726	0.727	0.701	0.684	0.750	0.668	0.741	0.571	0.705
MRR (↑)	0.019	0.074	0.086	0.046	0.060	0.089	0.056	0.066	0.044	0.082
NDCG@1 (↑)	0.722	0.828	0.836	0.848	0.828	0.901	0.796	0.842	0.810	0.848
of estimizable methods are notable (e.g., MetaGL). One potential reason for this is that the graph set for node classification is relatively small compared to the graphs applicable for link prediction, which limits the effectiveness of estimizable algorithms that are more prone to overfitting in such cases.
Sparse Testbed (Table 6). As the sparsity of the performance matrix  $\mathbf{P}$  increases, model selection methods perform increasingly worse. In particular, while AS achieves the best or the second best results in the Fully-Observed testbed, its performance quickly declines as sparsity increases. Since AS performs model selection based on the most similar observed graph, it cannot operate effectively in a highly sparse setting. Global averaging methods (e.g., GB-Perf), or more sophisticated optimization methods show more stable results. Due to the additional requirement for node labels, node classification task in this setup presents the most data sparse, yet practically important regime.
Out-Of-Domain Testbed (Table 7). Graphs in the same or similar domains are often more similar to each other than graphs in different domains. As this testbed requires addressing additional challenges to achieve out-of-distribution generalization, most methods perform worse than in other in-distribution testbeds. For instance, AS, which are sensitive to the availability of observed graphs similar to the query graph, perform worse than in Table 5. On the other hand, estimizable methods show more promising results as they learn to extrapolate into new domains by learning from observed domains.
Small-To-Large Testbed (Table 8). In comparison to the Fully-Observed testbed, the performance decreases overall in this testbed. However, considering that methods learn only from small graphs, model selection for large graphs still performs quite well, often achieving a similar level of performance. Successful methods in this testbed can make the model selection pipeline much more efficient as performance collection for small graphs can be done much more efficiently than for large graphs.
Additional Results. We provide additional results in the Appendix, including the results of the Cross-Task testbed, and results obtained with other meta-graph features, e.g.,  $M_{graphlets}$  and  $M_{compact}$ .
Table 8: Small-To-Large testbed results for link prediction (top) and node classification (bottom) tasks. Higher  $(\uparrow)$  scores are better. The best result is in bold, and the second best result is underlined.
(a) Link prediction
Perf. Metric	RandSel	GB-Perf	GB-Rank	ISAC	AS	S2	ALORS	NCF	MetaOD	MetaGL
AUC (↑)	0.522	0.798	0.797	0.842	0.827	0.767	0.812	0.796	0.667	0.870
MRR (↑)	0.031	0.072	0.061	0.132	0.368	0.074	0.209	0.047	0.075	0.260
NDCG@1 (↑)	0.841	0.958	0.960	0.957	0.951	0.953	0.947	0.956	0.921	0.964
(b) Node classification
Perf. Metric	RandSel	GB-Perf	GB-Rank	ISAC	AS	S2	ALORS	NCF	MetaOD	MetaGL
AUC (↑)	0.508	0.724	0.726	0.711	0.761	0.664	0.701	0.697	0.467	0.736
MRR (↑)	0.011	0.058	0.082	0.109	0.095	0.036	0.042	0.034	0.016	0.071
NDCG@1 (↑)	0.795	0.861	0.896	0.902	0.883	0.855	0.862	0.864	0.830	0.857
6.2 Discussion on Limitations and Future Directions
Limitations. In principle, using GLEMOS to select a GL model to employ for a new graph is based on the assumption that similar graph datasets exist in the benchmark. Therefore, it may not be very effective if the new graph is significantly different from all graphs in the benchmark (e.g., the new graph is from a completely new domain). However, as the benchmark data continue to grow over time, such cases will be increasingly less likely, while model selection performances will likely improve with the addition of more data. Furthermore, while GLEMOS currently supports two fundamental GL tasks, namely, node classification and link prediction, it can be further extended with additional tasks (e.g., graph classification). Incorporating them into GLEMOS is one of the future plans.
Future Directions. Below we list promising research directions to further improve the algorithms as well as the benchmark for instantaneous GL model selection.
- Direction 1: enabling model selection methods to use additional graph data. While existing methods utilize model performances and graph structural information captured by meta-features, they currently do not take other available graph data into account, such as node and edge features, timestamps in the case of dynamic graphs, and node and edge types (e.g., knowledge graphs). These data can be useful for modeling graph similarities, and the benchmark can further be enriched with such additional data.
- Direction 2: developing data augmentation techniques. Adding new performance records to the benchmark can improve the effectiveness of model selection methods. However, it is often computationally expensive to train and evaluate GL models on non-trivial graphs. Data augmentation techniques for GL model performances can be helpful in this data sparse regime, especially for estimizable methods that require a lot of data to learn effectively.
- Direction 3: handling out-of-distribution settings. Existing model selection methods are mainly designed for an in-distribution setup, as they assume that there exist observed graphs similar to a query graph. Thus their performance is suboptimal when a query graph comes from a new distribution. Investigating how to achieve generalization in such an out-of-distribution scenario would be beneficial.
- Direction 4: effective performance collection. When we have a limited budget for performance measurements on new graphs, selecting which pairs of graphs and models to evaluate and include in the benchmark can greatly influence the learning of model selection methods. Thus the ability to find a small set of representative pairs can lead to a fast and effective performance collection. Challenges include how to make such selections from a heterogeneous model set with multiple GL methods.
The choice of a GL model has a significant impact on the performance of downstream tasks. Despite recent efforts to tackle this important problem, there exists no benchmark environment to evaluate the performance of GL model selection methods, and to support the development of new methods. In this work, we develop GLEMOS, the first benchmark environment for instantaneous GL model selection.
- Extensive Benchmark Data. Among others, GLEMOS provides an extensive collection of model performances on fundamental GL tasks, i.e., link prediction and node classification, which is by far the largest and most comprehensive benchmark for Prob. 1 to the best of our knowledge.
- Algorithms and Testbeds. GLEMOS provides representative algorithms for Prob. 1, as well as multiple testbeds to assess model selection performance in practical usage scenarios.
- Extensible Open Source Environment. GLEMOS is designed to be easily extended with new GL models, new graphs, new performance records, and new GL tasks, while allowing reproducibility.
[40] David H. Wolpert and William G. Macready. No free lunch theorems for optimization. IEEE Trans. Evol. Comput., 1(1):67-82, 1997.
[41] Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 6861-6871. PMLR, 2019.
[42] Jia Wu, Xiu-Yun Chen, Hao Zhang, Li-Dong Xiong, Hang Lei, and Si-Hao Deng. Hyperparameter optimization for machine learning models based on bayesian optimization. Journal of Electronic Science and Technology, 17(1):26-40, 2019.
[43] Feng Xia, Ke Sun, Shuo Yu, Abdul Aziz, Liangtian Wan, Shirui Pan, and Huan Liu. Graph learning: A survey. IEEE Trans. Artif. Intell., 2(2):109-127, 2021.
[44] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR. OpenReview.net, 2019.
[45] Lin Xu, Frank Hutter, Jonathan Shen, Holger H Hoos, and Kevin Leyton-Brown. Satzilla2012: Improved algorithm selection based on cost-sensitive classification models. Proceedings of SAT Challenge, pages 57-58, 2012.
[46] Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. In NeurIPS, 2020.
[47] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In NeurIPS, pages 5171-5181, 2018.
[48] Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning on graphs: A survey. IEEE Trans. Knowl. Data Eng., 34(1):249-270, 2022.
[49] Yue Zhao, Ryan A. Rossi, and Leman Akoglu. Automatic unsupervised outlier model selection. In NeurIPS, pages 4489-4502, 2021.
[50] Kaixiong Zhou, Qingquan Song, Xiao Huang, and Xia Hu. Auto-GNN: Neural architecture search of graph neural networks. CoRR, abs/1909.03184, 2019.
[51] Tao Zhou, Linyuan Lu, and Yi-Cheng Zhang. Predicting missing links via local information. The European Physical Journal B, 71:623-630, 2009.
[52] Ronghang Zhu, Zhiqiang Tao, Yaliang Li, and Sheng Li. Automated graph learning via population based self-tuning GCN. In SIGIR, pages 2096-2100. ACM, 2021.
[53] Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. 2002.