SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets While our approach can potentially be applied to any node prediction dataset, for our experiments, we pick the following seven homophilous datasets that are commonly used in the literature: three citation networks, including CoraML, CiteSeer [26, 9, 8, 31], and PubMed [28], two co-authorship graphs — CoauthorPhysics and CoauthorCS [32], and two co-purchase datasets — AmazonPhoto and AmazonComputer [25, 32]. Moreover, we consider OGB-Products, a large-scale dataset from the OGB benchmark. Some of the methods considered in our work are not able to process such a large dataset, so we provide only the analysis of structural shifts on this dataset and do not use it for comparing different methods.
For any distributional shift, we split each graph dataset as follows. The half of nodes with the smallest values of  $\sigma_{i}$  are considered to be ID and split into Train, Valid-In, and Test-In uniformly at
random in proportion  $30\% : 10\% : 10\%$ . The second half contains the remaining OOD nodes and is split into Valid-Out and Test-Out in the ascending order of  $\sigma_{i}$  in proportion  $10\% : 40\%$ . Thus, in our base setup, the ID to OOD split ratio is  $50\% : 50\%$ . We have also conducted experiments with other split ratios that involve smaller sizes of OOD subsets, see Appendix B for the details.
Models In our experiments, we apply the proposed benchmark to evaluate the OOD robustness and uncertainty of various graph models. In particular, we consider the following methods for improving the OOD generalisation in the node classification task:
- ERM is the Empirical Risk Minimization technique that trains a simple GNN model by optimizing a standard classification loss;
- DANN is an instance of Domain Adversarial Network [7] that trains a regular and a domain classifiers to make features indistinguishable across different domains;
- CORAL is the Deep Correlation Alignment [34] technique that encourages the representations of nodes in different domains to be similar;
- EERM is the Explore-to-Extrapolate Risk Minimization [38] method based on graph structure editors that creates virtual environments during training;
- Mixup is an implementation of Mixup from [37] and represents a simple data augmentation technique adapted for the graph learning problems;
- DE represents a Deep Ensemble [21] of graph models, a strong but expensive method for improving the predictive performance;
In context of OOD detection, we consider the following uncertainty estimation methods:
- SE represents a simple GNN model that is used in the ERM method, for which the measure of uncertainty is Softmax Entropy (i.e., the entropy of predictive distribution);
- GPN is an implementation of the Graph Posterior Network [33] method for the node-level uncertainty estimation;
- NatPN is an instance of Natural Posterior Network [3] in which the encoder has the same architecture as in the SE method;
- DE represents a Deep Ensemble [21] of graph models, which allows to separate the knowledge uncertainty that is used for OOD detection;
- GPE and NatPE represent the Bayesian Combinations of GPN and NatPN, an approach to construct an ensemble of Dirichlet models [3].
The training details are described in Appendix F. For experiments with the considered OOD robustness methods, including DANN, CORAL, EERM, and Mixup, we use the experimental framework from the GOOD benchmark, whereas the remaining methods are implemented in our custom experimental framework and can be found in our repository.
Prediction tasks & evaluation metrics To evaluate OOD robustness in the node classification problem, we exploit standard Accuracy. Further, to assess the quality of uncertainty estimates, we treat the OOD detection problem as a binary classification with positive events corresponding to the observations from the OOD subset and use AUROC to measure performance.
5 Empirical study
In this section, we show how the proposed approach to creating distributional shifts can be used for evaluating the robustness and uncertainty of graph models. In particular, we compare the types of distributional shifts introduced above and discuss which of them are more challenging. We also discuss how they affect the predictive performance of OOD robustness methods as well as the ability for OOD detection of uncertainty estimation methods.
4Link to the GOOD benchmark repository
5 Link to our GitHub repository
Table 1: Comparison of structural distributional shifts in terms of OOD robustness and OOD detection. We report the drop in predictive performance of the ERM method measured by Accuracy (left) and the quality of uncertainty estimates of the SE method measured by AUROC (right).
Popularity	Locality	Density
AmazonComputer	-10.01%	-21.95%	-7.91%
AmazonPhoto	-8.54%	-30.93%	-3.58%
CoauthorCS	-3.13%	-1.22%	-4.86%
CoauthorPhysics	-3.42%	-4.75%	-1.41%
CoraML	-3.94%	-14.61%	-17.09%
CiteSeer	-0.02%	-26.51%	-8.39%
PubMed	-3.23%	-5.71%	-0.78%
OGB-Products	-2.86%	-2.83%	-0.12%
Average	-4.39%	-13.56%	-5.52%
Popularity	Locality	Density
AmazonComputer	88.52	86.48	44.24
AmazonPhoto	92.05	93.29	41.08
CoauthorCS	83.25	85.74	50.91
CoauthorPhysics	86.60	87.73	37.50
CoraML	75.67	87.13	81.55
CiteSeer	68.01	89.89	66.90
PubMed	68.60	66.34	58.60
OGB-Products	88.50	88.56	36.00
Average	81.40	85.65	52.10
5.1 Analysis of structural distributional shifts
In this section, we analyze and compare the proposed structural distributional shifts.
OOD robustness To investigate how the proposed shifts affect the predictive performance of graph models, we take the most simple ERM method and report the drop in Accuracy between the ID and OOD test subsets in Table 1 (left). It can be seen that the node classification results on the considered datasets are consistently lower when measured on the OOD part, and this drop can reach tens of percent in some cases. The most significant decrease in performance is observed on the locality-based splits, where it reaches  $14\%$  on average and more than  $30\%$  in the worst case. This fact matches our intuition about how training on local regions of graphs may prevent OOD generalization and create a great challenge for improving OOD robustness. Although the density-based shift does not appear to be as difficult, it is still more challenging than the popularity-based shift, leading to performance drops of  $5.5\%$  on average and  $17\%$  in the worst case.
OOD detection To analyze the ability of graph models to detect distributional shifts by providing higher uncertainty on the shifted inputs, we report the performance of the most simple SE method for each proposed distributional shift and graph dataset in Table 1 (right). It can be seen that the popularity-based and locality-based shifts can be effectively detected by this method, which is proved by the average performance metrics. In particular, the AUROC values may vary from 68 to 92 points on the popularity-based splits and approximately in the same range for the locality-based splits. Regarding the density-based shifts, one can see that the OOD detection performance is almost the same as for random predictions on average. Only for the citation networks the AUROC exceeds 58 points, reaching a peak of 81 points. This is consistent with the previous works showing that standard graph neural networks are unable to count substructures such as triangles. In our case, this leads to graph models failing to detect changes in density measured as the number of triangles around the central node.
Thus, the popularity-based shift appears to be the simplest for OOD detection, while the density-based is the most difficult. This clearly shows how our approach to creating data splits allows one to vary the complexity of distributional shifts using different structural properties as splitting factors.
5.2 Comparison of existing methods
In this section, we compare several existing methods for improving OOD robustness and detecting OOD inputs on the proposed structural shifts. To concisely illustrate the overall performance of the models, we first rank them according to a given performance measure on a particular dataset and then average the results over the datasets. For detailed results of experiments on each graph dataset separately, please refer to Appendix G.
OD robustness For each model, we measure both the absolute values of Accuracy and the drop in this metric between the ID and OOD subsets in Table 2 (left). It can be seen that a simple data augmentation technique Mixup often shows the best performance. Only on the density-based shift, expensive DE outperforms it on average when tested on the OOD subset. This proves that data
Table 2: Comparison of several graph methods for improving the OOD robustness (left) and detecting the OOD inputs by means of uncertainty estimation (right). For each task, we report the method ranks averaged across different graph datasets (lower is better).
Popularity	Locality	Density	Popularity	Locality	Density
ID	OOD	ID	OOD	ID	OOD
ERM	4.0	4.0	3.3	4.1	3.9	4.0	SE	1.4	2.1	4.0
Mixup	1.4	2.1	1.4	2.4	1.9	3.3	GPN	3.3	3.9	4.3
EERM	3.6	3.9	4.4	3.3	5.0	4.3	NatPN	5.3	4.1	2.9
DANN	4.3	4.3	5.0	4.1	3.0	3.6	DE	2.1	1.1	2.7
CORAL	4.7	4.1	4.3	4.7	4.1	3.9	GPE	3.1	4.3	3.4
DE	3.0	2.6	2.6	2.3	3.1	2.0	NatPE	5.7	5.4	3.7
Table 3: Comparison of the proposed architecture modifications that are used in the ERM method for OOD robustness (left) and SE method for OOD detection (right). For each task, we report the win/tie/loss counts across graph datasets for the modified GNN architecture against the base one.
Popularity	Locality	Density	Popularity	Locality	Density
ID	OOD	ID	OOD	ID	OOD
ERM + mod	4/2/1	5/2/0	4/2/1	3/2/2	4/2/1	5/2/0	SE + mod	0/0/7	0/1/6	4/0/3
augmentation techniques are beneficial in practice, as they prevent overfitting to the ID structural patterns and improve the OOD robustness. Regarding other OOD robustness methods, a graph-specific method EERM outperforms DANN and CORAL on the popularity-based and locality-based shifts. However, these domain adaptation methods are superior to EERM on the density-based shifts, providing better predictive performance on average for both ID and OOD subsets.
In conclusion, we reveal that the most sophisticated methods that generate virtual environments and predict the underlying domains for improving the OOD robustness may often be outperformed by simpler methods, such as data augmentation.
OOD detection Comparing the quality of uncertainty estimates in Table 2 (right), one can observe that the methods based on the entropy of predictive distribution usually outperform Dirichlet methods. In particular, a natural distinction of knowledge uncertainty in DE enables it to produce uncertainty estimates that are the most consistent with OOD inputs on average, especially when applied to the locality-based and density-based shifts. In general, GPN and the combination of its instances GPE provide higher OOD detection performance than their counterparts based on NatPN when tested on the popularity-based and locality-based splits.
5.3 Influence of graph architecture improvements
In this section, we consider several adjustments for the base GNN architecture of such methods as ERM, which is used to evaluate OOD robustness, and SE, which provides the uncertainty estimates for OOD detection. In particular, we reduce the number of graph convolutional layers from 3 to 2, replacing the first one with a pre-processing step based on MLP, apply the skip-connections between graph convolutional layers, and replace the GCN [16] graph convolution with SAGE [11].
These changes are aimed at relaxing the restrictions on the information exchange between a central node and its neighbors and providing more independence in processing the node representations across neural layers. Such a modification is expected to help the GNN model to learn structural patterns that could be transferred to the shifted OOD subset more successfully. Further, we investigate how these changes in the model architecture affect the predictive performance and the quality of uncertainty estimates of the corresponding methods when tested on the proposed structural distributional shifts.
For this, we use the win/tie/loss counts that reflect how many times the modified architecture has outperformed, got a statistically insignificant difference, or lost to the corresponding method, respectively. As can be seen from Table 3 (left), the ERM method supplied with the proposed modifications usually outperforms the baseline architecture both on ID and OOD, which is proved by high win counts. However, as can be seen from Table 3 (right), the same modification in the corresponding SE method leads to consistent performance drops, which is reflected in high loss
counts. It means that, when higher predictive performance is reached on the shifted subset, it becomes more difficult to detect the inputs from this subset as OOD since they appear to be less distinguishable by a GNN model in the context of the base node classification problem. Our observation is very similar to what is required from Invariant Learning techniques, which try to produce node representations invariant to different domains or environments. This may serve as evidence that there is a trade-off between the quality of learned representations for solving the target node classification task under structural distributional shift and the ability to separate the nodes from different distributions based on these representations.
In this work, we propose and analyze structural distributional shifts for evaluating robustness and uncertainty in node-level graph problems. Our approach allows one to create realistic, challenging, and diverse distributional shifts for an arbitrary graph dataset. In our experiments, we evaluate the proposed structural shifts and show that they can be quite challenging for existing graph models. We also find that simple models often outperform more sophisticated methods on these challenging shifts. Moreover, by applying various modifications for graph model architectures, we show that there is a trade-off between the quality of learned representations for the target classification task under structural distributional shift and the ability to detect the shift using these representations.
Limitations While our methods of creating structural shifts are motivated by real distributional shifts that arise in practice, they are synthetically generated, whereas, for particular applications, natural distributional shifts would be preferable. However, our goal is to address the situations when such natural shifts are unavailable. Thus, we have chosen an approach universally applied to any dataset. Importantly, graph structure is the only common modality of different graph datasets that can be exploited in the same manner to model diverse and complex distributional shifts.
Broader impact Considering the broader implications of our work, we assume that the proposed approach for evaluating robustness and uncertainty of graph models will support the development of more reliable systems based on machine learning. By testing on the presented structural shifts, it should be easier to detect various biases against under-represented groups that may have a negative impact on the resulting performance and interfere with fair decision-making.
Future work In the future, two key areas can be explored based on our work. Firstly, there is a need to develop principled solutions for improving robustness and uncertainty estimation on the proposed structural shifts. Our new approach can assist in achieving this objective by providing an instrument for testing and evaluating such solutions. Additionally, there is a need to create new graph benchmarks that accurately reflect the properties of data observed in real-world applications. This should involve replacing synthetic shifts with realistic ones. By doing so, we may capture the challenges and complexities that might be faced in practice, thereby enabling the development of more effective and applicable graph models.
[1] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
[2] B. Charpentier, D. Zügner, and S. Gunnemann. Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts. Advances in Neural Information Processing Systems, 33:1356-1367, 2020.
[3] B. Charpentier, O. Borchert, D. Zügner, S. Geisler, and S. Gunnemann. Natural posterior network: Deep bayesian predictive uncertainty for exponential family distributions. In International Conference on Learning Representations, 2022.
[4] Z. Chen, L. Chen, S. Villar, and J. Bruna. Can graph neural networks count substructures? Advances in Neural Information Processing Systems, 33:10383-10395, 2020.
[5] Y. Gal. Uncertainty in deep learning. University of Cambridge, Cambridge, 2016.
A Properties of distributional shifts
This section provides a detailed analysis and comparison of the proposed distributional shifts. For this purpose, we consider three representative real-world datasets AmazonComputer, CoauthorCS, and CoraML, and discuss how different distributional shifts affect the basic properties of data: class balance, degree distribution, and graph distances between nodes within ID and OOD subsets.
(c) Density
Figure 2: Class balance for AmazonComputer dataset across different types of shifts.
(a) Popularity
Figure 3: Class balance for CoauthorCS dataset across different types of shifts.
(a) Popularity
Figure 4: Class balance for CoraML dataset across different types of shifts.
Class balance Class balance directly affects the amount of evidence acquired by the graph processing model and used for estimating uncertainty and making predictions. It is especially important for evaluating Dirichlet-based models which exploit normalizing flows, as their density estimates can become irrelevant due to a significant change in class balance.
In Figures 2-4, one can see that the popularity-based split does not create a notable difference in the class balance between the ID and OOD subsets (for the datasets under consideration). Thus, the more important and less important nodes have, on average, the same probability of belonging to a particular class. More noticeable differences are induced by the density-based split. The locality-based split leads to the most significant changes for some classes. This shows that the split strategies based on the structural locality in graph can be very challenging as they also affect such crucial statistics as class balance.
Degree distribution The node degree distribution is one of the basic structural characteristics of graph that describes the local importance of nodes. Degrees are especially important for such graph
Figure 5: The distribution of node degrees for AmazonComputer dataset across different shifts.
(a) Popularity
Figure 6: The distribution of node degrees for CoauthorCS dataset across different shifts.
(a) Popularity
Figure 7: The distribution of node degrees for CoraML dataset across different shifts.
processing methods as GNNs since they describe how many channels around the considered node are used for message passing and aggregation.
In Figures 5-7, one can see that the most significant change in the degree distribution appears when the ID and OOD subsets are separated based on PageRank: the ID part contains more high-degree nodes. This is expected since PageRank is a graph characteristic measuring node importance (a.k.a. centrality), and node degree is the simplest centrality measure known to be correlated with PageRank. For the locality-based splits, the difference in degree distribution is smaller but still significant since PPR selects nodes by their relative importance for a particular node, so some high-degree nodes can be less important. Finally, for the density-based splits, the degree distribution also changes, but the extent of shift is usually less significant, and for some datasets (e.g., CoauthorCS), the higher-degree nodes are in the OOD subset.
Graph distance distribution The distance between two nodes in a graph is defined as the length of the shortest path between them. Here, we compute such distances between the nodes in the ID or OOD subset within the original graph, i.e., we consider the whole graph when searching for the shortest path. The distribution of distances illustrates which parts of the datasets are more locally concentrated.
In Figures 8-10, one can observe that the locality-based split leads to the most significant changes in distances, making the OOD nodes nearly twice as far from each other as the ID ones. At the same time, the popularity-based split does not lead to such a difference, revealing almost the same distributions on ID and OOD subsets. This means that the popularity bias in a graph does not prevent one from
Figure 8: The distribution of graph distances for AmazonComputer dataset across different shifts.
(a) Popularity
Figure 9: The distribution of graph distances for CoauthorCS dataset across different shifts.
(a) Popularity
Figure 10: The distribution of graph distances for CoraML dataset across different shifts.
covering the less popular periphery nodes since the most popular nodes may be widespread. Similarly, the density-based split does not induce a significant difference in pairwise distances.
B Results on other split ratios
The main part of our empirical study is focused on a  $50\% : 50\%$  split ratio. In this section, we extend our discussion with varying ID to OOD ratios and consider two setups where the fraction of OOD samples is smaller —  $70\% : 30\%$  and  $90\% : 10\%$ .
Let us first revisit our analysis of the proposed structural shifts on a  $70\% : 30\%$  ratio. As can be seen in Table 4, the results in both OOD robustness, which is evaluated in the node classification task, and OOD detection, which is done by means of uncertainty estimation, have not changed much compared to our base setup discussed in the main text — the considered graph models show the average drop in performance of  $3\%$  on the popularity-based shift and  $6\%$  on the density-based shift, while the OOD detection performance remains at nearly 81 and 54 points, respectively. Despite the fact that the graph models now have access to more diverse structures in terms of popularity and local density, making decisions about unimportant and sparsely surrounded nodes remains as difficult as before.
Further, the drop in predictive performance on the locality-based shift appears to be less significant compared to the original setup and reaches only  $5\%$  on average instead of the previous  $14\%$ . At the same time, the OOD detection performance on this structural shift drops to 79 points, whereas it was 85 points in the original setup. These results also match our intuition — as the distance between the ID and OOD nodes decreases, the OOD samples become less distinguishable from the ID ones. This makes the OOD detection performance drop on average, while the gap between ID and OOD metrics in standard node classification disappears.
We also may compare the existing graph models based on the results in Table 5. As can be seen, the ranking of methods remains almost the same as in the original setup. Specifically, the most simple data augmentation technique Mixup often shows the best performance in OOD robustness, while DE provides the second best results on most structural shifts. As for OOD detection, the methods based on the entropy of predictive distribution again outperform the Dirichlet ones, and the uncertainty estimates produced by DE are best correlated with the OOD examples. Thus, our observations regarding the performance of graph models are consistent with those reported in the paper.