HGOE: Hybrid External and Internal Graph Outlier Exposure for Graph Out-of-Distribution Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 13:51:28

Abstract: With the progressive advancements in deep graph learning, out-of-distribution (OOD) detection for graph data has emerged as a critical challenge. While the efficacy of auxiliary datasets in enhancing OOD detection has been extensively studied for image and text data, such approaches have not yet been explored for graph data. Unlike Euclidean data, graph data exhibits greater diversity but lower robustness to perturbations, complicating the integration of outliers. To tackle these challenges, we propose the introduction of \textbf{H}ybrid External and Internal \textbf{G}raph \textbf{O}utlier \textbf{E}xposure (HGOE) to improve graph OOD detection performance. Our framework involves using realistic external graph data from various domains and synthesizing internal outliers within ID subgroups to address the poor robustness and presence of OOD samples within the ID class. Furthermore, we develop a boundary-aware OE loss that adaptively assigns weights to outliers, maximizing the use of high-quality OOD samples while minimizing the impact of low-quality ones. Our proposed HGOE framework is model-agnostic and designed to enhance the effectiveness of existing graph OOD detection models. Experimental results demonstrate that our HGOE framework can significantly improve the performance of existing OOD detection models across all 8 real datasets.

Main paper: 
 introduction: 
 Nowadays, graph-structured data have shown significant success in handling non-Euclidean relationships, prevalent in multimedia systems such as social networks [38, 56] , knowledge graphs [31, 36] , citation networks [6, 62] , These capabilities facilitate advanced applications like scene graph generation [5, 10] , fraud detection [17] , and video captioning [2] , by modeling complex relationships between heterogeneous data types [28, 30, 33, 39, 49, 50, 55, 58, 61] . However, a significant challenge arises from the i.i.d. assumption on which the mainstream graph learning methods depend, i.e., the training and testing graph data are from the same distribution. This assumption often fails in real-world scenarios, particularly in domains where data is complex and lacks sufficient labeling, such as in the case of drug molecules and proteins. For example, a new drug might not be any part of the already annotated data. Consequently, this gives rise to an interesting issue: how to determine whether such a new drug is present in the annotated drug library? This problem, known as Graph Out-Of-Distribution (GOOD) detection, is crucial in advancing the use of graph learning in real-world scenarios.Although several methods for graph OOD detection have been developed [27, 32, 59] , they only utilize ID graph data. When the full data distribution is complicated, merely modeling the ID data might be insufficient to capture the essential clues for OOD detection. In fact, auxiliary public OOD data is often accessible to help the detector discover such clues. Exposing such auxiliary OOD data to the model, also known as Outlier Exposure (OE), is widely studied for image data [9, 16] . However, the application of OE for graph samples has not yet been explored. To bridge this gap, we attempt to investigate the incorporation of OE into graph OOD detection.Unlike image data, introducing outliers to assist in the OOD detection of graph data presents two significant challenges.Firstly, since graphs are abstractions of patterns from the natural world, they exhibit inherent diversity. For instance, various non-Euclidean structures ranging from water molecules to complex social networks can be represented using graphs, yet these graphs can significantly differ in structure and properties. In contrast, images possess a Euclidean structure, which facilitates easier feature transfer. Therefore, utilizing existing graphs as outliers directly is often not effective.Secondly, graph data exhibits less robustness to perturbations, resulting in the possibility of OOD samples existing within the boundary of ID classes. For example, as illustrated in Figure 1 , phenol and benzene differ only by a hydrogen atom and an oxygen atom, yet exhibit vastly different properties; phenol is solid at room temperature, while benzene is liquid. This suggests that slight perturbations could transform ID graph sample into an OOD sample. In contrast, image data typically exhibit a more compact intra-class distribution and more well-separated classes (e.g., dogs and cats), where minor perturbations often do not change their categories. Therefore, merely incorporating some external outliers for exposure could be effective for image data. But considering that outliers may exist even within the same class of graphs, the external outlier exposure remedy is insufficient for graph OOD detection.To address the first challenge, existing studies suggest that introducing diverse outliers closer to the in-distribution data is beneficial. Therefore, given the inherent diversity of graph data, we propose incorporating cross-domain external outliers into training. Moreover, in addition to these external outliers, we aim to synthesize outliers that are nearer to in-distribution data. Regarding the second challenge, since there exist subgroups within a class, and outliers also exist between these subgroups, we consider synthesizing some internal outliers between subgroups to assist in OOD detection. Because these samples are close to In-Distribution samples, if the model can accurately identify these samples, it would further enhance the OOD detection performance.Motivated by this, we propose a general hybrid graph outlier exposure (HGOE) framework for graph OOD detection which integrates both external and internal outliers. The external outliers are easily collected from public database. As for internal graph outliers, we design a graphon-based ID-mixup strategy to simulate the OOD region among subgroups and synthesize OOD samples. Given these outliers, we further propose a boundary-aware OE loss to adaptively learn from true outliers and prevent the unintended bias.In summary, the contributions of this paper are as follows:• We propose a novel hybrid graph outlier exposure framework for graph OOD detection. It simultaneously utilizes external outliers and internal outliers to enhance the diversity. To the best of our knowledge, this is the first trial of outlier exposure in graph-level OOD detection tasks. • To synthesize internal outliers, we design an ID-mixup method that can effectively generate outlier graph samples between ID subgroups based on graphons. • We further introduce a novel boundary-aware loss. By instantiating the HGOE framework with a SOTA detector, we have surpassed the competitors on 8 real-world graph datasets. 
related work: 
 Graph Neural Networks. Graph neural networks (GNNs) have been widely adopted in various deep learning tasks due to their ability to process graph-structured data, which can effectively extract both the structural and attribute information of graphs [12, 20, 44] .GNNs have achieved remarkable results in various deep learning tasks in recent years, such as recommendation systems, natural language processing and computer vision [46, 62] . GNNs can be broadly categorized into two distinct classes, which encompass spectral-based GNNs and spatial-based GNNs [63] .Existing Spectral-based Graph Neural Networks leverage spectral graph theory for graph analysis, offering the advantage of incorporating global graph topology information. However, they also exhibit certain limitations, including high computational complexity, challenges in handling dynamic or heterogeneous graphs, and a deficiency in local perception ability [48] . For instance, Cheb-Net approximates spectral graph convolutions using Chebyshev polynomials of arbitrary order, while GCN simplifies this by employing only the first two Chebyshev polynomials as the graph convolution, effectively creating a fixed low-pass filter [37] .Spatial-based Graph Neural Networks leverage the spatial information of nodes within the graph to perform graph convolutions [19] . These networks are rooted in message-passing mechanisms [20, 53] , wherein each node updates its features based on the features of its neighboring nodes. Notably, Graph Attention Networks (GAT) [45] introduce a self-attention mechanism to compute node neighbor weights, allowing for dynamic and adaptive aggregation within neighborhoods. GAT further enhances model capacity by employing multi-head attention. Meanwhile, GraphSAGE [13] learns how to aggregate node features from various sources using diverse functions like mean, max-pooling, or LSTM, making it suitable for inductive learning tasks that involve new nodes or graphs during testing. Graph Isomorphism Networks (GIN) [53] , another class of graph convolutional network, employ sum and MLP operations to aggregate node features. Additionally, Graph Structural Neural Networks [51] provide a versatile solution for incorporating structural graph properties into the message-passing aggregation scheme of GNNs. Out-of-distribution (OOD) Detection. Existing deep learningbased classification methods often exhibit overconfidence on unseen classes [15] . To address this issue, out-of-distribution (OOD) detection [15] involves the task of distinguishing test samples from distributions different from the seen training data. It comprises post-hoc and fine-tuning approaches [54] . Post-hoc methods [21, 23, 43, 47] leverage the logit space and output scores of models that trained on in-distribution data to classify ID and OOD data. Fine-tuning approaches [9, 16] introduce extra regularization terms during training or incorporate auxiliary training data, referred to as outlier exposure, which can be either real, synthetic, or sampled from the feature space. Outlier exposure has proven effective in enhancing OOD detection performance.However, these methods are typically applied to image or text data. OOD detection in graph data remains an under-explored area. Graph OOD detection [27] aims to determine whether test graphs originate from within the in-distribution relative to the training set or are out-of-distribution. Some studies [32, 59] focus on graph anomaly detection, where the training data comprises both indistribution and anomalous data, positioning it as a subset of graph OOD detection. OCGIN [59] utilizes a GIN as its encoder and capitalizes on an SVDD [60] objective for one-class graph anomaly detection. GLocalKD [32] employs joint random distillation to pinpoint anomalous graphs on both local and global scales. GOOD-D [27] differentiates between ID and OOD graph data through hierarchical contrastive learning and perturbation-free graph data augmentation, having been exposed only to ID data during training. AAGOD [11] introduces a novel learnable amplifier generator, designed to generate graph-specific amplifiers, thereby enhancing the detection of OOD graphs on trained GNNs. GraphDE [22] introduces a generative framework for debiased learning and OOD detection in graph data.Graph Data Augmentation. Graph data augmentation, involving transformations to enrich or enhance information in the given graph, has been extensively explored. Non-learnable augmentation methods [8, 57] achieve this through perturbation or random sampling of edges, nodes, or subgraphs. In contrast, learnable augmentation methods [7] train an Augmenter with learnable parameters using techniques like Decoupled Training, Joint Training, or Bilevel Optimization [8] . Among these methods, some utilize graph mixup [14, 25] to generate new graphs. G-mixup [14] enhances classification performance by generating new graphs through mixup based on estimating graphons of the same class. S-Mixup [25] employs soft alignments for node matching to achieve instance-level graph mixup.As an orthogonal direction to existing graph-level OOD detection approaches, our approach focuses on introducing outlier samples that assist in OOD detection training. Our framework relies on utilizing in-distribution graphons to generate internal outliers within them, rather than directly performing augmentations on the original graph structure. 
preliminaries: 
 In this section, we provide definitions for key terms and concepts utilized throughout this paper.Denote a graph by 𝐺 = (V, E, 𝑿 ), where V denotes the set of nodes, E denotes the set of edges, and 𝑿 ∈ R 𝑛×𝑑 is the node feature matrix. The adjacency matrix is denoted as 𝑨 ∈ {0, 1} 𝑛×𝑛 , where 𝑨 𝑖 𝑗 = 1 indicates a connection between nodes 𝑣 𝑖 and 𝑣 𝑗 and 𝑨 𝑖 𝑗 = 0 otherwise.In this paper, we focus on the unsupervised graph-level OOD detection problem. Specifically, given a set of unlabeled in-distribution graphs D 𝑖𝑛 = {𝐺 𝑖 } 𝑁 𝑖=1 drawn from the distribution P 𝑖𝑛 , the aim of unsupervised graph-level OOD detection is to learn a graph OOD scoring function 𝑓 (•) based on the ID data D 𝑖𝑛 . A higher score 𝑠 = 𝑓 (𝐺) indicates a higher probability to be an OOD graph. The scoring function is evaluated on a test setD 𝑡𝑒𝑠𝑡 = D 𝑖𝑛 𝑡𝑒𝑠𝑡 ∪ D 𝑜𝑜𝑑 𝑡𝑒𝑠𝑡 (D 𝑖𝑛 𝑡𝑒𝑠𝑡 ∩ D 𝑜𝑜𝑑 𝑡𝑒𝑠𝑡 = ∅)where D 𝑖𝑛 𝑡𝑒𝑠𝑡 ∼ P 𝑖𝑛 and D 𝑜𝑜𝑑 𝑡𝑒𝑠𝑡 ∼ P 𝑜𝑜𝑑 . It should be emphasized that graph data sourced from P 𝑖𝑛 and P 𝑜𝑢𝑡 might fall into multiple categories. However, in the unsupervised graph-level OOD task, the model is not provided with any categoryspecific labels.A graphon, denoted by the function W : [0, 1] 2 → [0, 1], is a continuous, bounded, and symmetric function in graph theory, extensively used to describe graph generation [1] . The value W (𝑖, 𝑗) approximates the probability of an edge between nodes 𝑖 and 𝑗 in a specific graph. A graphon can be considered as a function that embodies the characteristics of a class of graphs, allowing for the sampling of graphs from the graphon. These sampled graphs share similar topological features. Distinct graphons represent different graph classes, enabling comparative analysis of their structural features. These graphons are estimated using step function approximations [29] . Graphons are particularly useful in synthesizing new graphs that reflect patterns found in real-world data, offering insights into the underlying structure of complex networks. However, there is no closed-form expression for graphons. Previous studies [14, 52] have employed a two-dimensional step function to estimate graphons, which can be considered a matrix representing the probability of edge existence. In this paper, we denote it as 𝑊 ∈ [0, 1] 𝐷 ×𝐷 , where 𝐷 is the dimensionality of the graphon. This matrix 𝑊 can then be used to generate graphs with a number of nodes less than 𝐷. 
methodology 4.1 overall framework: 
 In general, with a graph OOD score function 𝑓 (𝐺), the basic learning objective for unsupervised graph OOD detection can be described as:minEQUATIONwhere L ood is the loss function.By integrating the hybrid graph outlier exposure (HGOE) procedure, we hope the exposed OOD data have higher OOD scores than ID data. Subsequently, we simultaneously minimize the OOD score of ID data and maximize that of OOD data: minEQUATIONApparently, such an introduction of HGOE is independent of the basic graph OOD model and thus can be applied to most existing models.In deploying the GOE framework, we identify two pivotal challenges: (C1) How to acquire high-quality graph outlier data? (C2) How to design a L GOE to effectively utilize these OE data? To tackle (C1), we introduce both internal and external outliers. To address (C2), we develop a boundary-aware loss function. The overall pipeline of HGOE is illustrated in Figure 2 . To facilitate the generation of internal outliers, we start with the division of graphs into distinct subgroups, wherein each subgroup is defined by the similarity in properties among its constituent graphs. This is followed by the estimation of the graphon for each subgroup. Thereafter, we deploy an ID-mixup technique that combines the graphons from these subgroups to obtain internal outliers, incorporating node feature information from external outliers into this process. The resulted internal and external outliers are then fed into the OOD detector, which is further optimized through the implementation of an innovatively designed boundary-aware loss. In subsequent sections, we will introduce the HGOE framework in detail.Our mixture outlier training strategy incorporates outliers from two sources: real-world external outliers and synthesized internal outliers. External outliers are derived from public graph datasets similar to the training graphs, while internal outliers are generated from ID graphs. External Outliers. In contrast to image data, graph data in the real world display significantly more complex structures, with notable gaps in characteristics between fields like social networks and protein networks. To investigate the role of real-world outliers, we sample from several graph datasets to create a real-world OE dataset. During the training of different ID data, we select graphs from the OE dataset that are consistent with them in feature dimensions, ensuring no overlap between the OE graphs and the training and test sets. This process results in an external outlier dataset, denoted as D 𝑒𝑥𝑡 𝑜𝑒 . Internal Outliers. The quality and diversity of external outliers data are crucial for outlier exposure, but are limited by the scope and quality of the existing auxiliary dataset. Synthesizing additional outliers then emerges as a solution, offering tailored, abundant, and diverse graph data for training. However, the challenge lies in ensuring that the synthesized data resonates with real-world outlier scenarios and does not introduce unintended biases. As discussed in the introduction, there exist internal outliers among the in-distribution graph data, which are distributed near the boundaries of subgroups. To obtain these internal outlier samples, we design an internal outlier generation strategy, which will be described in the next subsection.A straightforward way to generate graphs is to interpolate each pair of samples in the dataset directly. However, unlike Euclidean data such as images, different graphs usually have different sizes (i.e., different numbers of nodes) and have unique topology in Non-Euclidean spaces. Therefore, we propose to mix up the graph generators instead of graph samples themselves to synthesize more realistic graph data.To find internal outliers for input graphs, it is imperative to first divide the graphs into subgroups. For the input ID graphs, we adopt graph contrastive learning techniques (e.g., GraphCL [57] ) for feature extraction, and then perform 𝑘-means clustering to obtain 𝑘 subgroups C 𝑖 , where D 𝑖𝑛 = 𝑘 𝑖=1 C 𝑖 . Since the members in each subgroup often share similar properties, we assume that each subgroup within the ID data is generated by a specific graphon. And we use the widely-used universal singular value thresholding (USVT) method [4] to estimate the graphon 𝑊 𝑖 for each C 𝑖 in D 𝑖𝑛 .For each pair of non-overlapping subgroups C 𝑖 and C 𝑗 in D 𝑖𝑛 , we perform a mixup operation over the corresponding graphons 𝑊 𝑖 and 𝑊 𝑗 as follows:EQUATIONwhere 𝜆 ∈ [0, 1] is the balancing hyperparameter. The mixup result M combines the structural features of subgroups C 𝑖 and C 𝑗 , and can be considered a new graph generator positioned between the two subgroups. By sampling based on this graphon, we could obtain graphs lying in the interpolated regions between these subgroups, preserving the subgroups's topologies.Random Size Based Sampling. The interpolated graphon M ∈ [0, 1] 𝑁 ×𝑁 has the capability to generate infinitely many graphs. However, the naive generated graphs are very likely to have a size around 𝑁 . This limits the diversity of the synthesized outliers, which does not meet our goal in the HGOE framework to synthesize outliers broadly distributed near the original subgroups, with various sizes of graphs. In order to increase the diversity of outliers, we employ a random size based sampling strategy, i.e., we first randomly sample the size of the graph 𝑟 ∈ [2, 𝑁 ] and then generate the graph from the sampled graphon M ′ ∈ [0, 1] 𝑟 ×𝑟 . The existence of an edge between nodes 𝑖 and 𝑗 is determined by sampling from a Bernoulli distribution with the parameter M ′ (𝑖, 𝑗). External Feature Alignment. Merely sampling from the interpolated graphons could only produce pure structure graph outliers with the structure information (denoted as 𝑨 ′ ). Therefore, we further propose to generate node features based on external outlier features. Specifically, we calculate the structural features of both the generated internal outlier structure 𝑨 ′ and the external outliers. And then we construct the node features for the generated internal outliers based on that of the external outliers with the most similar structure features. Here, the structural features 𝑠 diff 𝑖 of the 𝑖-th node in a 𝑑 𝑠 -dimensional space is obtained through a 𝑑 𝑠 -step random walk-based diffusion process on the graph:EQUATIONwhere 𝑻 = 𝑨 ′ 𝑫 -1 denotes the transition matrix for the random walk on the graph, and 𝑫 is the degree matrix of 𝑨 ′ . After computing the structural features of 𝑨 ′ , we compare them with the structural features of external outliers. For each node in the synthesized internal outlier 𝐺 ′ , we assign node features that are most closely aligned with the structural features of nodes in the external outliers. Note that our external feature alignment approach is training-free. By performing structural searches on nodes with existing features, we can effectively assign outlier features to nodes that initially lack features. Through the aforementioned ID-mixup based procedure, we can generate an arbitrary number of internal outliers, denoted as D 𝑖𝑛𝑡 𝑜𝑒 . Compared to real-world outliers, these synthetic outliers are not limited in quantity or source. Furthermore, being located near in-distribution samples, they effectively compensate for the uncontrolled quality issue of external outliers.The introduced outliers further present two critical issues: How can we ensure that the introduced outliers do not fall within the in-distribution area, and how to find those more important outliers? Take the social graph ID data as an example. Intermixing social graphs could still result in a social graph, so the generated samples are not OOD data and should be excluded from the outlier dataset. At the same time, different outliers can have varying levels of importance, with points on the boundary potentially being critical points where a change in properties is about to occur.To solve these problems, we design the following boundaryaware OE Loss ℓ 𝑏𝑎 which is adaptively aware of whether the sample is in ID or OOD space. For an input outlier graph 𝐺 ′ , the loss is calculated as:EQUATIONwhere 𝑠 𝐺 ′ = sigmoid(𝑓 (𝐺 ′ )) is the OOD score scaled by a sigmoid function, 𝑙 and 𝛾 are hyperparameters. Note that 𝜏 is an ID-boundary threshold, which we adaptively set as the smallest 𝑠 among the ID samples:EQUATIONTo better understand Eq. ( 5), we rewrite it in the following form:EQUATIONFrom this formulation, we have the following observations:• When log(𝑠 𝐺 ′ ) > 𝜏, it indicates that 𝐺 ′ is a valid outlier. In this case, the smaller 𝑠 𝐺 ′ is, the closer it is to the indistribution boundary. Apparently, such near-boundary outliers are important in guiding the detector to distinguish OOD samples from ID ones. Therefore, we weigh it by the term (𝑙 -𝑠 𝐺 ′ ) 𝛾 , giving higher weights to these boundary samples. As 𝛾 increases, the model pays more attention to samples nearer to the ID boundary. • When log(𝑠 𝐺 ′ ) < 𝜏, it means that the outlier 𝐺 ′ is very likely invalid and fall into in-distribution space. Blindly treating it as an outlier might be harmful to the learning. Therefore, by minimizing -𝜏 (𝑙 -𝑠 𝐺 ′ ) 𝛾 , the OOD score is reduced, which enhances the recognition ability for in-distribution samples.Then, considering both external and internal outliers, the total graph outlier exposure loss is formulated as:EQUATIONFinally, we can instantiate the overall training objective L as follows:EQUATIONIn practice, our L GOE can be added as a regularization term to most existing OOD detection models, without modifying their network architectures. 
experiments 5.1 experimental setup: 
 5.1.1 Datasets. We select three pairs of datasets from the TU dataset [35] and five pairs from the OGB dataset [18] . Each pair of datasets belongs to the same field and shares similar features, but exhibits distribution shifts between the two datasets in the pair. 90% of the In-Distribution (ID) samples are used for training, while 10% of the ID samples and an equivalent number of OOD samples are used for testing. For more detailed information about these datasets, refer to Table 1 .For external outlier data, we grouped datasets with identical feature counts into a unified external dataset. Based on the feature counts of our selected datasets, as detailed in Table 1 , this organization resulted in two distinct external dataset collections, with node feature numbers being 1 and 9, respectively. In this configuration, for each In-Distribution dataset, we utilized other datasets from the corresponding external dataset collection for training purposes, deliberately excluding both the OOD dataset for testing and the ID dataset itself.• Non-deep Two-step Methods. We use WL [41] graph kernels as feature extractors and employ local outlier factor (LOF) [3] , one-class SVM (OCSVM) [34] , and isolation forest (iF) [26] as detectors to perform OOD detection. • Deep Two-step Methods. The process is similar to the above methods, but the feature extractor is replaced with graph deep self-supervised methods InfoGraph [42] and GraphCL [57] , and the detectors are replaced with isolation forest (iF) and Mahalanobis distance (MD) [40] . Compared to graph kernels, self-supervised methods can extract features of graphs better. • End-to-end Methods. We utilize three popular end-to-end learning methods, including OCGIN [59] , which uses graph neural networks for feature extraction and is optimized with SVDD. GLocalKD [32] uses distillation learning for graph anomaly detection, and GOOD-D [27] employs multi-level contrastive learning for end-to-end OOD detection. For the proposed framework, we instantiate it on the SOTA graph OOD detector baseline GOOD-D [27] . Besides, we also implement two ablated variants to show the impact of internal and external outliers. Specifically, HGOE w/o IO and HGOE w/o EO denote the variants without internal and external outliers, respectively.For HGOE, the ratio of external to internal outliers was set to 1:1, with the total number equal to the number of in-distribution samples seen during training. The hyperparameter 𝜆 was set to 2, and 𝛾 was set to 2. The dimension of the structural features 𝑠 diff 𝑖 was set to 16, and the number of clusters for all datasets was determined to be 3. GraphCL was run for 50 iterations with an embedding dimension of 32, and the probabilities for node dropping, feature masking, and edge removing were all set to 0.1. During ID-mixup, 𝜆 was randomly chosen from the range [0.01, 1]. AUC (Area under the ROC Curve) [24] is used as the performance metric. A higher AUC value indicates better detection performance.We evaluate the performance of HGOE and other competitors, with the results presented in Table 2 . Our findings include:(1) Two-step detection methods do not perform as well as end-toend detection methods, especially where non-deep learning methods are inferior to deep learning-based feature extraction methods. Within self-supervised methods, GraphCL-MD demonstrates the best performance, showing GraphCL's capability in extracting features for graph OOD detection. This also explains why we chose to utilize GraphCL features for clustering.(2) For our HGOE framework, compared to GOOD-D without the use of graph outliers, there was an enhancement in performance across all 7 datasets. On the ENZYMES+PROTEIN dataset, the average performance improve from 60.15 to 64.44. On Tox21+SIDER, it increased from 64.98 to 68.24, and on FreeSolv+ToxCast, the performance increase from 78.79 to 83.36.(3) The performance improvement on IMDB-M+IMDB-B was not substantial because both are social datasets, and the outliers we introduce were from molecular and protein datasets, which are biologically oriented. However, using these as outliers still contributed to enhanced detection on IMDB-M+IMDB-B. This indicates that selecting outliers similar to the ID distribution, as opposed to introducing outliers completely unrelated to the ID data, is more effective.After obtaining the subgroups within the known distribution, we performed a mixup of the estimated graphons and visualized the resulting mixed graphons in the form of heatmaps, as shown in Figure 3 . The graphon in the center represents the result of performing ID-mixup on the two adjacent graphons. It is evident that the graphon after ID-mixup retains the structures of the original graphons, forming a new graph generator. This demonstrates that our ID-mixup can effectively blend the distributions of subgroups to a certain extent, fulfilling our hypothesis.Based on the OOD scores for samples in the test set, we visualize the frequency distribution of OOD scores for both ID and OOD samples using different colors. As shown in Figure 4 , the left column depicts the score distributions without using HGOE, while the right column shows the results after applying the HGOE framework. The less the overlap and the greater the distance between the ID and OOD areas are, the better the corresponding model's performance is. It is observable that our HGOE method has decreased the overlap area between ID and OOD distributions, which also explains the reason for the performance improvement.Visualization of Graph-level Features. We extract features of each individual graph by the OOD detection model. These extracted features are then visualized through t-SNE, as demonstrated in Figure 5 . The representation employed in this study is the graphlevel representation acquired from HGOE. It can be observed that, apart from the visualization result of AIDS+DHFR in 5a, the representations of other OOD graphs are distributed among the ID graphs. This highlights the importance of identifying and analyzing internal outliers that reside within the distribution of ID graphs for more effective and accurate OOD Detection. 2 , HGOE w/o IO and HGOE w/o EO represent the results of using only internal outliers and external outliers, respectively. We observe that using just one of these types yields better results than not using any OE (Outlier Exposure) samples at all. Moreover, in most cases, the performance of using only external outliers is superior to that of using only internal outliers, but inferior to using both types. This indicates that the combination of both types of outliers can more effectively enhance the performance of hybrid graph outlier exposure. To explore the specific role of 𝜏, we implement methods setting 𝜏 according to the maximum, average, and minimum normalized scores. The results, as shown in Figure 3 , indicate that the min strategy for allocating 𝜏 performs the best. However, this strategy negatively affects performance on the IMDB-M+IMDB-B dataset, which we believe is due to its significant difference from the other datasets. In this process, a performance decline was observed in most datasets. This suggests that for ID-mixup, blending two in-distribution samples at varying ratios can increase the diversity of internal outliers, thereby enhancing detection effectiveness. However, an opposite trend of performance increase, rather than decrease, was observed in the CLintox+LIPO dataset pair. This anomaly may be related to the characteristics of these datasets, both being molecular graphs from the OGB and possessing extremely similar average node and edge counts, resulting in closer distributions. If ID-mixup outliers falling within the in-distribution range, it could lead to a greater performance decrease compare to other datasets. So when 𝜆 is set to a middle value, the generated graphon lies between subgroups. This explains why choosing a 𝜆 value range [0.3, 0.7] yields better results.Hyperparameter 𝛾 of Boundary-aware OE Loss. As described in our framework, 𝛾 plays a significant role in the boundary-aware loss. We vary the value of 𝛾 to {0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0}, and as observed in Figure 6 , as 𝛾 increases, there is a performance improvement when 𝛾 ∈ [0, 2]. However, performance starts to decline for some datasets when it exceeds 2 and 2.5. This indicates that within an appropriate range, introducing 𝛾 can effectively weight the samples, and this effect improves with the increase of 𝛾, up to a point where it begins to decrease. This phenomenon validates the rationality and effectiveness of our designed boundaryaware OE loss. 
conclusion: 
 In this work, we investigate the enhancement of OOD detection performance on graph-level data through hybrid graph outlier exposure. We demonstrate that not only external outliers are crucial for graph OOD detection, but internal outliers also play a significant role. Based on this, we propose a carefully designed ID-mixup based method for synthesizing internal outliers by generating OOD samples between in-distribution subgroups and aligning these with external outlier features. Upon obtaining these synthesized internal outliers, we effectively utilize the characteristics of outlier samples by adaptively allocating learning weights to OE samples using a well-designed boundary-aware OE loss. Our HGOE framework provide substantial performance improvements for other graph OOD detection methods. Extensive experiments on 8 real-world datasets show its superior performance.  
 
 Appendix:  


Decision: This paper received three positive scores (ba,ba,ba) from all of reviewers after rebuttal. The authors have resolved the main concerns raised by reviewers. I am happy to recommend to accept this paper. Please carefully revise the final manuscript according to the comments and discussions.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel Hybrid External and Internal Graph Outlier Exposure (HGOE) framework to improve out-of-distribution (OOD) detection performance for graph data using deep graph learning, by incorporating realistic external graph data from various domains and synthesizing internal outliers within in-distribution subgroups, along with a boundary-aware OE loss that intelligently weights outliers to maximize the use of high-quality OOD samples while minimizing the impact of low-quality ones, with the HGOE framework being model-agnostic and designed to enhance existing graph OOD detection models, showing significant performance improvements across 8 real datasets.
strength_weakness: strength_weakness: An ID-mixup method based on graphons to effectively synthesize internal outlier graph samples between in-distribution subgroups.
Introduction of a novel boundary-aware loss function.
Integrating the framework with a state-of-the-art detector, it surpasses competitors across 8 real-world graph datasets, demonstrating superior performance.
limitations: limitations: To facilitate a more comprehensive evaluation, it is suggested that the authors extend the experiments to include the remaining 2 datasets in addition to the current 8, and compare against the GOOD method. Expanding the evaluation scope can provide a more complete picture of the proposed model's capabilities and allow for a fairer comparison.

It would be beneficial to include the publication years of the compared methods in the results tables. This additional information would help readers understand the chronological context and technological advancements of the different approaches.

The authors may consider including the anomaly detection task results from the GOOD method and comparing them with the proposed model. Evaluating on this related task could offer further insights into the strengths and limitations of the proposed approach from an alternative perspective, enriching the analysis.
paper_summary: paper_summary: This paper proposes the HGOE method to address the challenge of OOD detection in graph data. The framework integrates external outliers from various domains and synthesizes internal outliers within in-distribution subgroups to improve graph OOD detection performance. The authors introduce a boundary-aware outlier exposure loss that adaptively assigns weights to outliers, maximizing the use of high-quality OOD samples while minimizing the impact of low-quality ones. Experimental results on eight real-world datasets demonstrate the effectiveness of HGOE.
strength_weakness: strength_weakness: 1. This paper is well-written, and the technical details are well-elaborated.

2. This paper introduces a hybrid approach that combines external and internal outliers for graph OOD detection. The use of graphon-based ID-mixup to generate internal outliers is novel and addresses the challenge of diversity in graph data.

3. Extensive experimental validation on eight real-world datasets consistently shows the effectiveness of the proposed HGOE framework.

4. The experiment includes detailed visualizations, such as t-SNE and score distributions, to illustrate the effectiveness of the HGOE. These visualizations provide a clear understanding of how the proposed HGOE improves OOD detection.
limitations: limitations: 1. The authors need to supplement an algorithmic flow to intuitively show the reader its training process.

2. Lack of theoretical analysis (e.g., time and space complexity) and empirical analysis (e.g., running time) of the computational overhead of the proposed method.

3.  Lack of comparison with other baseline methods in the visualization experiments.

4. How does the hyperparameter $\beta$ affect the HGOE's performance? The reviewer doesn't see the relevant experiments and analyses.

5. Lack of source code of HGOE to guarantee reproducibility.
paper_summary: paper_summary: The paper introduces a novel method named HGOE (Hybrid Graph Outlier Exposure) for improving graph out-of-distribution (OOD) detection. This method integrates external and internal graph data to synthesize outliers, enhancing the detection of OOD instances in graph-structured data. The proposed boundary-aware outlier exposure (OE) loss is designed to optimally use high-quality OOD samples while minimizing the impact of lower-quality ones. The method is demonstrated to be effective across eight real-world datasets, improving the performance of existing OOD detection models.
strength_weakness: strength_weakness: 1. Methodological Rigor in Outlier Synthesis: The ID-mixup strategy for generating internal outliers is particularly innovative. This approach not only enhances the diversity of training data but also tailors the outlier generation process to the specific structure of graph data, maintaining the intrinsic properties of the original data distributions.
2. Enhancement of Existing Models: The integration of the HGOE framework with state-of-the-art (SOTA) OOD detection models and its demonstration of improved performance is a significant strength. It shows that HGOE can be adapted to enhance the capabilities of existing models without requiring extensive modifications to their architectures.
3.  Comprehensive Evaluation Across Diverse Domains: The paper conducts extensive experiments across eight real-world datasets from different domains, showcasing the versatility and adaptability of the HGOE framework. This broad evaluation significantly enhances the paper's impact, demonstrating its applicability in diverse real-world scenarios.
limitations: limitations: 1. Scalability and Efficiency Concerns: The method involves complex processes like graphon estimation and ID-mixup for outlier synthesis. These processes can be computationally intensive, especially for large-scale graphs, but the paper lacks a detailed discussion on scalability and computational efficiency.
2. Potential Bias in Internal Outlier Generation: The method for generating internal outliers could introduce biases, particularly if the graphon mixup does not accurately reflect the real-world transitions between graph subgroups. This could lead to misleading training signals and affect the model’s performance.
3. Ambiguity in Parameter Selection: The method involves several hyperparameters, such as the balancing hyperparameter in the ID-mixup and the parameters in the boundary-aware OE loss. The sensitivity of the model's performance to these parameters is not extensively discussed, which may pose challenges in tuning the model for optimal performance.