Detecting Out-of-Distribution Data with Semi-supervised Graph “Feature" Networks
Authors: Debargha Ganguly
Debayan Gupta
Venue: Submitted to ICLR 2023 (2023)

Abstract: Anomalous and out-of-distribution (OOD) data present a significant challenge to the robustness of decisions taken by deep neural networks, with myriad real-world consequences. State-of-the-art OOD detection techniques use embeddings learned by large pre-trained transformers. We demonstrate that graph structures and topological properties can be leveraged to detect both far-OOD and near-OOD data reliably, simply by characterising each data point (image) as a network of related features (visual concepts). Furthermore, we facilitate human-in-the-loop machine learning by expressing this data to comprise high-level domain-specific concepts. We obtained \textit{97.95\% AUROC} on far-OOD and \textit{98.79\% AUROC} on near-OOD detection tasks based on the LSUN dataset (comparable to the performance of state-of-the-art techniques).

Under review as a conference paper at ICLR 2023
DETECTING OUT-OF-DISTRIBUTION DATA WITH
FEATURE NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Anomalous and duttif-distribution (OOD) data present a significant challenge the robustness of decisions taken by deep neural networks, with myriad real-world consequences. State-of-the-an art OOD detection techniques use embeddings learned by large pre-trained transformers. We demonstrate that graph structures and topo- logical properties can be leveraged to detect both Far-OOD and near-OOD data reliably, simply by characterising each data point (image) network of related features (visual concepts). Furthermore, we facilitate human-in-the-loos machine learning by expressing this data to comprise high-level domain-specific concepts. We obtained 97.95% AUROC on far-OOD and 98.79% AUROC on near-OOD detection tasks based on the LSUN dataset (comparable to the performance of state-of- thettheartt art techniques).
INTRODUCTION
Trustworthy machine learning systems must hand over decisions it is not confident about to human experts. Most machine learning pipelines operate on the assumption data is assumed to be drawn in an IID fashion from the same distribution as the training data. The difficulty of OOD detection relies primarily on how semantically close the outliers are to the inliers. Therefore, based on difficulty [Winkens et al. (2020)] the OOD detection task is split into the following.
closed world. The test
1. Near OOD refers to semantic shifts in the data, such as (SVHN and MNIST). Generally, this more challenging problem solve, and the AUROC hovers around 93 per cent for state-of-the the-hhh-rttt art methods [Fort et al. [
2. Far OOD is a covariate shift, which is less difficult to detect. The AUROC hovers around 99 per cent in the current state of the art (Fort et al. (202T).
Common sense is a very essential yet absent element of AI systems. This crucial ability to judge and understand everyday things amongst most humans is non-trivial problem with machines [Xu al. (202T)). The absence of common sense prevents intelligent systems from understanding changing world (distribution drift), behaving reasonably in unforeseen situations (such as OOD detection), and learning quickly from new experiences (i.e. prior information). Furthermore, is hard to learn, encode and represent this information. This shared and undefined knowledge base in humans is known from extensive exposure open domain data such as basic physical phenomena.
In this paper, we operate under the assumption that common sense can be learnt in patterns of occur- rences, and this knowledge can be learnt in a domain specific manner. Therefore, our strategy relies on creating commonsense service that learns from experience, based on computational models that mimic child cognition towards scenes and reasoning.
Intuition Graphs provide a general language for describing and analysing entities with interactions between them. We want to use the rich relational structures among visual concepts in complex domains to represent commonsense concepts. Our hypothesis is this would lead to better OOD prediction while maintaining justifications humans can understand.
Contributions This work includes the following contributions

---

Under review as a conference paper at ICLR 2023
1. We propose novel semi- human- interprraterr concepts. This relies on representing each data point (image) as a graph of visual features.
-eaiiii-eeaiinn--ie framework that operates on
2. We demonstrate that our technique performs on par with state-of-the-art methods on near and far-OOD tasks based on the LSUN dataset.
2 BACKGROUND AND RELATED WORK
Detecting Out Of Distribution (OOD) points in a relatively lower dimensional space have been extensively used [Pimentel et al. (2014)] in experiments. Conventionally, these methods include density estimation, nearest neighbour-baser algorithms, and clustering analysis. The density esti- mation approach uses probabilistic models to estimate the in-distribution density, while declaring data point out of distribution it is located in a on distance measures between points to find dow-density region. Clustering- based methods rely points (that are further away from the neighbourhood). The primary drawback of these methods has always been their inadequacy in working with high-dimensional data (Theis et al. (2015)] such images.
Issues Over the last years, state-of-the-are results in the OOD detection task have been based on deep neural network-dependent approaches. For example, convolutional neural networks have been used to find bizarre scenes by (Sabokrou et al.] (2018). Furthermore, the techniques presented in Andrews et al. (2016) and Fort al. (2021) depend on an amalgamation of transfer-leaming and representation learning. In sensitive environments, such as clinical settings, generative adversarial networks have been used by [Schlegi et al. (2017). The drawback of a technique that requires adding more layers to a neural network or modifying its layers is that pre-trained neural networks is that neural networks can be overconfident about wrong decisions when employed distribution setting [Hendrycks and Gimpel (2016); Lakshminarayanan et al. (2017); Guo et al. (2177). On the other hand, using large pre-trained transtormer networks does improve performance but relies heavily on the assumption that the embeddings generated by them are infallible.
an out-of-
Scene Graph Generation (SGG) refers to the task of automatically mapping an image or a video into semantic structural scene graph, (Zhu et al. (2022) requiring the accurate labelling of detected objects and their relational structures. Although this is a tricky task, the availability of extensive datasets, such Visual Genome Krishna et al. (2016)), and massive models, such OSCAR (Li iet al. (2020); Zhang et al. [202I] and CelationFormer. (Shit et al. (2022)], has shown impressive results. In our case, since we do not need the relational structures between objects to be human- readable, in the interest of reducing computational overhead we favour object detection networks
over scene graph generation networks.
Commonsense
knowledge graphs (CSKGs) are gaining popularity [llievski et al. (2021): Guan
al.) [2019)] as origins of background knowledge (domain-specific conceptual, syntactic informa- tion) that are conceptualised to help with downstream reasoning tasks such as question answering and planning. In our context, we intend to use these for OOD data detection. For this, we exploit the recent advances in geometric learning. These same methods allow graph neural networks to be used to predict molecule properties and social media conversation characteristics.
PROBLEM SETUP
This paper assesses the problem of differentiating between in-distribution and out-of-distribution image examples on pre-trained neural network. Let us assume that two distributions, Di, and Dout, are drawn from the space X. In dataset Din of (xx" .v") pairs where æ denotes the input feature vector, and yt E y ;= {1,.... K} denotes the class label. Let Dout denote an out-of- distribution dataset of (æ=" y pairs where you E your := {K + I,.... . K + +O.. yout n = G. In our experiments, we sample from the mixture distribution. The conditional probability of drawing from this mixed distribution is P Din for in-distribution and Pxizzi = Dout for dutto--distribution. Our problem setup allows access OOD samples for training.
We are therefore presented with the following challenge: Given an image X drawn from the mixture distribution Px.z- can distinguish whether the image is from in-distribution D.,,"
2

---

Under review as a conference paper at ICLR 2023
4 USING FEATURE NETWORKS AS AN OUT-OF-DISTRIBUTION DETECTOR
This section presents our method for detecting out-of-distribution samples. This detector depends on a three-stage pipeline, the feature finder, the unsupervised projector, and the supervised detector. We describe the details of all three components below.
FEATURE NETWORK
IMAGES
5
14
58
34
PROJECTOR
[I..I.IIIIIII
OOD DETECTOR
[1,011311131111
[2.12.1,333333
FEATURE FINDER
GRAPH EMBEDDINGS
Figure 1: Stages for using Semi-Supervised Feature Networks for OOD detection.
4.1 FEATURE FINDER (Ø .imppt + G
We assume the availability of pre-trained object-detection network that discovers the distinct com- ponents inside an image, draws boxes around them and assigns classes. Next, the proposed objects are pruned by thresholding based on the model's 's confidence (in section [5.4] we assume e=0.2)in the prediction. Finally, the stage generates the graph based either of the following strategies:
Unweighted Graph Each class forms node in the network, but edges are only drawn between the nodes found in the image. If multiple object pairs are found, multiple edges are drawn.
Weighted Graph As before, each class forms node in the network, with edges only drawn between the nodes found in the image. If multiple object pairs are found, multiple edges are drawn with pairwise weight factor assigned to each edge product of the intersection over union (IoU) score (Jaccard Index (J)) and the euclidean distance between the centroids of the two bounding boxes.
[ + llobj, objzll J(ob),, obje) if weighted strategy
G(x) (V, G(a) E G; Fobji,onn2
if unweighted strategy
4.2 PROJECTOR (P + 2)
The previous stage's feature networks (graphs) represent each data point (image). Although the graph substructures, such nodes and sub-graphs, are essential for our task the best depiction of the entire data point and its intricacies can only be captured by the whole graph. Therefore, this stage focuses creating mapping from the structural and topological properties of this graph to a vector space which can be used for downstream analysis. Therefore this method performs whole- graph embedding, creating projection into our "graph-feature" space.
3

---

Under review as a conference paper at ICLR 2023
Table 1: OOD detection tasks based Yu al. 2015
Far-OOD Tasks
Near-OOD Tasks
Bridge Room, Dining Room, Kitchen, Living Room, Restaurant, Bedroom)
(Classroom, Conference
Bedroom vs (Classroom, Conference Room, Dining Room, Kitchen, Living Room, Restaurant); Living Room (Restaurant)
Church Outdoor vs (Classroom, Con- ference Room, Dining Room, Kitchen, Living Room, Restaurant, Bedroom)
Church Outdoor vs (Tower, Bridge), Bridge vs (Tower), Classroom vs (Conference Room, Dining Room, Kitchen, Living room, Restaurant); Kitchen vs (Living Room, Restaurant)
Tower vs (Classroom, Conference Room, Dining Room, Kitchen, Living Room, Restaurant, Bedroom)
Conference Room vs (Dining Room, Kitchen, Liv- ing Room, Restaurant); Dining Room (Kitchen, Restaurant, Living Room);
Graph-kernel-basee methods to perform whole-graph embeddings use handcrafted features such as shortest paths, graphlets etc. This hampers their performance and introduces problems such poor generalisation. This is why, in this stage we favour the usage of neural embedding frameworks that learn representations (Z) and can scale to arbitrary-sizee graphs.
4.3 DETECTOR (0,
Given the embeddings (Z.) learnt from the Graph Neural Network (P), to separate and bind the two distributions away from each other. Since the embeddings no longer have very high dimensions, we can leverage techniques from the rich literature anomalous data detection. A classifier, 9z. may be trained to output whether the data point expressed by Z; is out of distribution or not (g). If learning in an unsupervised manner, this includes density estimation, nearest neighbour, and clustering techniques.
downstream model
5 EXPERIMENTS
5.1
OUT-OF-DISTRIBUTIIN DATASETS
The Large scale Understanding Dataset (LSUNN) has about 120k to 3Mn images of 10 categories (each) such as bedroom, kitchen room, living room etc (Yu et al. (2015)). We consider each of these categories as its domain. We classify pairwise combinations into far-OOD and near-OOD tasks depending the semantic similarity between these domains. Out of the total 45 permutations possible (in table 1), we define pairs as near-OOD when both are outdoors or indoor scenes.
5.2 GRAPH EMBEDDING ALGORITHMS
1. Graph2Vee [arayanan al. (2177) creates Weisfeiler-Lehman tree features for the nodes, based on which a feature co-occurrence matrix developed and analysed to gener- ate representations of the graphs. With minimum count of feature occurrences as run this for Weisfeiler-Lehmar iterations, for ten epochs, with learning rate of 0.025, and downsampling with frequency 10-1 This allows us generate 128-dimensional embedding vector.
2. Wavelet Characteristic [Wang al. (021)] uses characteristic functions of node features with wavelet function weights to describe node neighbourhoods. Once generated, node- level features are combined to create embeddings on the entire graph. We use averaging experiments with r= 1.0, =2.5 characteristic
the pooling function and function evaluations and adjacency matrix powers set to 5. Finally, we generate a 1000- dimensional embedding.
4

---

Under review as a conference paper at ICLR 2023
3. LDP Cai and Wang (2018) relies calculating the histograms of degree profiles, which are joined together to form graph representations. We run our experiments assuming 32 histogram bins and generate 160-dimensional embedding.
4. Feather Graph [Rozemberezki and Sarkar (2020)) method utilises characteristic functions of node features with random walk weights describe node neighbourhoods, which are combined to generate the entire graph's 's embeddings. We use averaging as the pooling function and run experiments with =2.5 25 evaluation points and adjacency ma- trix powers set to 5. This generates a 500-dimensional embedding.
5. Invariant Graph Embeddings [Galland and Lelarge [2019) computes graph-deseriptor using a mixture of spectral and node embedding-basee baaeed features, such as eigenvalues and scattering. We use [10,20] as the number of histogram bins, [10,20] as spectral embedding dimensions, with [3.5] as feature embedding dimensions. dimensional embedding.
This lets us generate 220-
6. GL2Vec [Chen and Koga (2019) generates embeddings while leveraging line graphs and edge features. It relies on decomposing feature co-occurrence matrix of Weisfeiler- Lehman tree features for the nodes. With a minimum count of feature occurrences as we run this for Weisfeiler-Lehman iterations, for ten epochs, with learning rate of 0.025, and downsampling with frequency 10-4 This allows us to generate a 128-dimensional embedding vector.
7. NetLSD Tsitsulin et al. (2018)] generates 250-dimensional embeddings using the heat kernel trace of the normalised Laplacian matrix over time scales. We use 200 eigenvalue approximations, with 250-time scale steps and a time scale interval range between the minimum and maximum of -2.0 and 2.0, respectively.
8. SF de Lara and Pineau [018)] was used to generate 128 22--dimensional embeddings based on the lowest eigenvalues of the normalised laplacian.
9. FGSD VVerma and Zhang 2017) was used generate 200-dimensional embedding using the histogram of Moore- Penrose spectral features of the normalised laplacian. Here, we assumed 200 histogram bins, with histogram range of 20.
5.3 EVALUATION METRICS
We use these metrics to gauge the usefulness of our system in differentiating in and out of distribu- tion images. Again, these are consistent with what's used evaluate performance in OOD detection literature. Fort et al. (202
Liang et al. (2017)
1. Area Under the Receiver Operating Characteristic curve (AUROC) is threshold- independent metric representing the relationship between TPR and FPR [Davis and Goad- rich (, where perfect detector reaches an AUROC score of 100%. This can be thought of the probability that an anomalous example given a higher OOD score than in-distribution example [Fawcett (2006)).
2. AUPR another threshold-independer metric, MMnninn and Schutze 1999;; Saito and Rehmsmeier (2015)] evaluating the area under the PR (graph showing the preci- Sion=TP((TP+FP) and recal--PP/TP+FN,, plotted against each other). is useful when anomalous examples are infrequent, as it considers the base rate of the anomalies. We re- port the AP (average precision) score summarizes a precision-recal curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight.
5.4 RESULTS
In this section, we demonstrate our method's effectiveness in detecting OOD datapoints the L.SU--baee benchmark (in table 1). We use an object detection model trained on Open Images V4 with ImageNet pre-trained Inception Resnet V2 an image feature extractor to extract features. This specific model [Google (2022)] attains an mAP of 0.58 on the OpenImagesv4 test set and can successfully find 600 different classes.
5

---

Under review as a conference paper at ICLR 2023
Due to computational constraints, test 20,000 images in each in-distribution and out-of- distribution classes on each near-OOD and far-OOD benchmark. In addition, 20% of the data is held out for unseen testing. The task for each category was randomly chosen from Table 1. In table 2, we describe the AUROC and average precision seores computed for Far-OOD detection between "Church (Outdoor)" and "Dining room" classes. 'abbl 3 describes the AUROC and average preci- sion scores computed for Near-OOD detection between "Living Room" and "Dining room" classes. For both these tasks, logistic regression and gradient boosting were used 0z.
AUROC
AUPR (AP)
Projection Method
Logistic Reg
Grad Boost
Logistic Reg
Grad
(Accuracy)
Test
Train
Test
Train
Test
Train
Test
Train
Wavelet Characteristic
84.65
84.77
85.50
85.58
82.72
82.88
82.75
82.84
LDP
85.51
85.58
85.50
85.58
82.82
82.86
82.75
82.84
FeatherGraph
84.81
84.99
85.50
85.58
82.81
83.06
82.75
82.84
IGE
83.66
83.11
85.44
85.77
82.54
82.40
83.27
83.48
GL2Vec
84.20
84.30
85.82
87.04
82.33
82.75
84.99
86.40
NetLSD
85.61
85.58
85.50
85.58
83.96
83.94
82.75
82.84
SF
85.45
85.54
85.50
85.58
83.33
83.54
82.75
82.84
FGSD
84.80
85.01
92.84
93.13
84.20
84.49
92.67
92.96
Graph2Vee (weighted)
94.47
94.25
97.95
98.22
94.34
94.15
97.69
98.14
Table 2: Results for Far-OOD detection were computed on the classes "Church Outdoor" and "Din- ing Room" based on table I, from the LSUN dataset.
AUROC
AUPR (AP)
Projection Method
Logistic Reg
Grad Boost
Logistic Reg
GradBoost
(Accuracy)
Test
Train
Test
Train
Test
Train
Test
Train
Wavelet Characteristic
73.20
73.63
72.99
73.51
72.33
72.27
70.54
70.57
LDP
72.99
73.50
72.99
73.51
70.56
70.53
70.54
70.57
FeatherGraph
73.29
73.78
72.99
73.51
71.73
71.78
70.54
70.57
IGE
GL.2Vec
72.64
73.01
81.08
82.92
72.56
72.27
81.05
82.42
NetLSD
72.99
73.54
72.99
73.52
72.06
72.28
70.55
70.58
SF
73.06
73.57
72.99
73.51
72.05
72.06
70.54
70.57
FGSD
73.41
74.13
75.97
77.03
72.95
73.17
74.16
74.76
Graph2Vec (weighted)
93.51
93.46
98.48
98.79
92.73
92.73
98.53
98.78
Table 3: Results for
detection computed the classes "Living Room" and "Dining Room" based on table 1, from the LSUN dataset.
DISCUSSION
In Fort et al. (022) i.e. the current SOTA, an AUROC of 96% has been achieved near- near-OOD OOD detection, while AUROC of 99% has been achieved in OoDDODD detection. Considering that achieved 98.79% AUROC on the near-OOD tasks from the LSUN dataset. and 97.95% AUROC on far-OOD tasks from the LSUN dataset model performance is comparable to the mate--of-the-are

---

Under review as a conference paper at ICLR 2023
Benchmarking and comparing the same tests are difficult since many image classification frame- works (CIFAR-10, 100) contain data points that contain only one feature variable (cars, planes), thereby negating the potency that graphs bring express complex relationships inside data. Papers such as Fort et al. (202T) combine all the classes to create a distribution, such as all digits inside MNIST becoming
distribution. In extension, all digits inside SVHN become another distribu- tion. Therefore, each of our classes inside the LSUN dataset is analogous to an entire distribution and not individual classes, such as cars, buses etc.
Just like words make up document, the concept of visual words making up images has been studied in great detail. Topic models such as pLSA [Hofmann (2013)) worked on modelling co-occurence information under probabilistic framework to discover underlying semantic structures.
Handcrafted Features Our semi-supervised featured networks are generated based on handcrafted features, such as human interpretable descriptors, euclidean distance and the jaccard Index. We have shown performance equivalent
ofate-of-the-art methods in our domain. However, in other domains good feature extraction may not be feasible, an adequately large dataset may not be available. In future work, we plan to explore how these graphs can be built using lesser hand-engineering so that we can achieve better generalisation ability.
Constraints Our framework only requires access to an object detector that can find features of particular interest inside the required dataset and domain. It would also be helpful if this detector could find features that are expected to be seen in out-of-distribution data. Access to the list of features that can be detected could mean that the graphs in G contain nodes for every feature and only draw edges when pairwise relationships are observed., as described in section 4.1
Using auxiliary models for sanity checkkng the model's decisions is conceptually the same boost- ing. This is because we effectively fit successive models to the residuals of previous models. We believe such setup can reduce the risk of failure in production settings.
We observe that large amounts of graph data are essential for the self-supervised graph embedding algorithms to learn discriminative features maximally. When the same experiments were performed with 1000 images each, performance was generally not outstanding. This is possible because when the embedding Z is in high enough dimensions, and there are not enough data points -0, can always fit the training set, but generalisation on the test set is inferior.
fundamental assumption in our framework currently access both the in and -idttii-iitiittiionn data. It should be possible to extend our method not to require access to OOD samples, whether to train auxiliary models or to tune hyperparameters. However, if has not trained on varied amount of data the discriminative ability may be limited, thereby compromising the performance of the OOD detector. In future experiments, we intend to examine methods to extend our framework, including the ability for zero or few-shot OOD detection.
7 CONCLUSION
Deep neural networks can make wrong decisions with high confidence when operating on data was not trained on. Therefore, for the safe deployment of Al systems in the real world, model performance and detection of out-of-distribution data are equally important. To tackle this problem and find OOD data propose a novel semi-supervised geometric-llerming--asee framework that operates on human-interpretable concepts. We demonstrate that our technique performs on par with state-of-the-art methods near and far-OOD tasks. In future work, we intend explore more unsupervised methods to generate feature graphs and test our method on few-shot OOD detection.
REFERENCES
Andrews. J., Tanay, T., Morton, E. J., and Griffin, L. D. (2016). Transfer paareseentation-llamnng for anomaly detection. JMLR.
Cai, C. and Wang, Y. (2018). A simple yet effective baseline for non-attributed graph classification. arXiv preprint arXiv: 1011188110000
Chen, H. and Koga, H. (2019). G12vee: Graph embedding enriched by line graphs with edge fea-
tures. In International Conference on Neural Information Processing, pages 3-14. Springer.
7

---

Under review as a conference paper at ICLR 2023
Davis, J. and Goadrich, M. (2006). The relationship between precision-recal and curves. In Proceedings of the 23rd international conference on Machine learning, pages 233-240.
de Lara, N. and Pineau, E. (2018). A simple baseline algorithm for graph classification. arXiv preprint arXiv:1810.091
Fawcett, T. (2006). An introduction roc analysis. Pattern recognition letters, 27(8):861-877
Fort, S., Ren, J., and Lakshminarayanan, B. (2021). Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems, 34:7068-7081.
Galland, A. and Lelarge, M. (2019). Invariant embedding for graph classification. In ICML 2019 Workshop on Learning and Reasoning with Graph-Structured Data.
Google, T. (2022). Tensorflow hub fasterrenn openimagesv4 inception resnet v2. https:// [t.hub. pwl.de/v/veee hwtp://ooll//aateet.eoooooooo
httpa/eeeeeeooeeeeeeeemmeeeeeees reaprppttttoorreeee__eeeee v2/1
Accessed: 2022-09-29.
Guan, N., Song, D., and Liao, L. (2019). Knowledge graph embedding with concepts. Knowledge- Based Systems, 164:38 44.
Guo, C.,, Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural net- works. In International conference machine learning, pages 1321-1330. PMLR.
Hendrycks, and Gimpel, K. (2016). baseline for detecting misclassified and diiti---ist examples in neural networks. arXiv preprint arXiv: 13..0
Hofmann, T. (2013). Probabilistic latent semantic analysis. arXiv preprint arXiv:1301.6705
Ilievski, F., Szekely, P., and Zhang, B. (2021). Cskg: The commonsense knowledge graph. In European Semantic Web Conference, pages 680-696. Springer.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.JJ,, Shamma, D. A., Bernstein, M., and Fei-Fei, L.. (2016). Visual genome: Connecting language and vision using crowdsourced dense image annotations.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive un- certainty estimation using deep ensembles. Advances neural information processing systems, 30.
Li, X., Yin, X., Li, C., Hu, X., Zhang, P., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., and Gao, J. (2020). Oscar: Object-semantics aligned pre-training for vision-language tasks. ECCV 2020.
Liang, S., Li, Y., and Srikant, R. (2017). Enhancing the reliability of out-of-distribution image detection neural networks. arXiv preprint arXiv:1706.02694 02660.
Manning, C. and Schutze, H. (1999). Foundations of statistical natural language processing. MIT press.
Narayanan, A., Chandramohan, M., Venkatesan, R., Chen, L., Liu, Y., and Jaiswal, S. (2017).
graph2vec: Learning distributed representations of graphs. arXiv preprint 17Xv:1707.005005
Pimentel, M. A., Clifton, D. A., Clifton, L., and Tarassenko, L. (2014). A review of novelty detec- tion. Signal processing, 99:215-249.
Rozemberezki, B. and Sarkar, R. (2020). Characteristic functions on graphs: Birds of a feather, from statistical descriptors to parametric models. In Proceedings the 29th ACM international conference on information & knowledge management, pages 1325-1334.
Sabokrou, M., Fayyaz, M., Fathy, M., Moayed, Z., and Klette, R. (2018). Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes. Computer Vision and Image Understanding, 172:88-97.
8

---

Under review as a conference paper at ICLR 2023
Saito, T. and Rehmsmeier, M. (2015). The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3):c01 18432.
Schlegl, T., Seeböck, P., Waldstein, S. M., Schmidt-Erfurth, U., and Langs, G. (2017). Unsupervised anomaly detection with generative adversarial networks guide marker discovery. In Interna- tional conference on information processing in medical imaging, pages 146-157. Springer.
Shit, S., Koner, R., Wittmann, B., et al. (2022). Relationformer: A unified framework for image-to- graph generation. arXiv preprint arXiv:2203.10202.
Theis, L., Oord, A. V. d., and Bethge, M. (2015). A note on the evaluation of generative models. arXiv preprint arXiv: 15/1.01144..
Tsitsulin, A., Mottin, D., Karras, P., Bronstein, A., and Müller, E. (2018). Netlsd: hearing the shape of a graph. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2347-2356.
Verma, S. and Zhang, Z.-L, (2017). Hunt for the unique, stable, sparse and fast feature learning on graphs. Advances in Neural Information Processing Systems, 30.
Wang, L., Huang, C., Ma, W., Cao, X., and Vosoughi, S. (2021). Graph embedding via diffusion- wavelets- based node feature distribution characterization. In Proceedings of the 30th ACM Inter- national Conference Information & Knowledge Management, pages 1478-3487
Winkens, J., Bunel, R., Roy, A. G., Stanforth, R., Natarajan, V., Ledsam, J. R., MacWilliams, P., Kohli, P., Karthikesalingam, A., Kohl, S., et al. (2020). Contrastive training for improved out-of- distribution detection. arXiv preprint arXiv:2007.05566.
Xu, Y., Zhu, C., Wang, S., Sun, S., Cheng, H., Liu, X., Gao, J., He, P., Zeng, M., and Huang, X. (2021). Human parity on commonsenseqa: Augmenting self-attention with external attention. arXiv preprint arXiv:2112.0325
Yu, F., Zhang, Y., Song, S., Seff, A., and Xiao, J. (2015). Lsun: Construction a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03366
Zhang, P., Li, X., Hu. X., Yang, J., Zhang, L., Wang. L., Choi, Y., and Gao, (2021). Vinvl: Making visual representations matter in vision-language models. CVPR 2021.
Zhu, G., Zhang, L., Jiang, Y., Dang, Y., Hou, H., Shen, P., Feng, M., Zhao, X., Miao, Q. Shah, S.
A. A., and Bennamoun, M. (2022). Scene graph generation: A comprehensive survey.


Reviews:
[{"review_id": "tGaNmuJDrz", "reviewer": "Reviewer_sju8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new pipeline that includes 3 stages for detecting out-of-distribution data. First, a graph is generated for an image by using its output from a pre-trained object-detection network. Secondly, the graph-kernel-based method is used to generate the whole-graph embedding from the graph. Finally, given the embedding from the graph neural network, a classifier is trained to detect the out-of-distribution data.", "summary_of_the_review": "I think the idea of this paper is not novel enough and the experiments are not persuasive.\n\n", "strength_and_weaknesses": "Strength:\n1. The idea of this paper is interesting. I am not sure similar idea was proposed before or not.\n2. This paper got results by using different graph embedding algorithms, it should be valuable for future work.\n3. Based on the DISCUSSION part of this paper, the results of this should not be bad.\n\nWeaknesses:\n1. The method of this paper lack novelty. It combined several well-defined methods to do out-of-distribution detection.\n2. I am not an expert in this domain, not sure if it is fine or not that this paper only shows their own results and didn't compare with other state-of-art methods.\n3. I think this paper is not in good written. It lacks much detail about those methods, like what's the architecture of the detector. What if the pre-trained object detection network doesn't include the class for the out-of-distribution dataset?\n4. This paper lack ablation studies like how the graph affects the results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Dx9Omd0_4u", "reviewer": "Reviewer_qAw3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of Out-of-distribution detection. A pre-trained object-detection network is firstly used to build a semantic graph. Then the graph embedding is further used to detect OOD samples. The experiment is conducted on the far-OOD and near-OOD setting.", "summary_of_the_review": "This paper provides an interesting idea that uses the pre-trained object detector to build semantic graph and then detect OOD samples by the graph embedding. Firstly, the assumption of having an oracle object detector for all in-distribution data is unrealistic. Secondly, the idea is poorly presented by the insufficient details and the experimental results are limited. Overall, the paper is far from ready to be published in this venue. ", "strength_and_weaknesses": "\nStrength: \n\n\t1. The idea of building a semantic map to detect OOD is novel. It points out one promising direction for the interpretable OOD detection, which can tell \"why the sample is an OOD\". \n\n\t2. The comparison of different graph embedding algorithms is comprehensive. \n\nWeakness: \n\n\t1. The proposed method relies on a pretrained object detection network that contains the sufficient semantic information for the in-distribution data. When the semantic of in-distribution data like medical images is not covered by the object detection network (pre-trained on natural image), the built semantic graph can be incomplete or even erroneous. If we use additional annotations to train a sufficiently strong object detection network, the effort will be extremely expensive comparing the existing methods. \n\n\t2. The paper is not polished and not ready to publish, with missing details in related work / experiment / writing. See more in \"Clarity, Quality, Novelty And Reproducibility\". \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4GqqyfqcYZg", "reviewer": "Reviewer_cXz2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper authors propose a mechanism for deriving low-dimensional representations suitable for effective use of established non-parametric and parametric out-of-distribution data detection methods. Specifically they utilize graphs which represent relationships among the objects detected in an image. They claim that this low-dimensional representation mechanism mimics human cognitive processes and is better suited to detect novelty.", "summary_of_the_review": "The paper translates an intuitive representation mechanism into a feature extraction technique. It demonstrates that this technique is highly competitive. However there are a few changes which could further strengthen the paper and make it more accessible to the reader. These include \n* more detailed description of the feature extraction procedure with a real example\n* stronger experimental validation by comparing to other low dimension projection techniques which can be utilized with OOD detection algorithms\n* more rigorous/well referenced and less hand-wavy descriptions of the motivation/intuition\n", "strength_and_weaknesses": "Strengths\n-----------\n\nThe paper clearly describes the intuitions and designs a feature extraction technique based on these. \nIt performs extensive comparisons using a variety of projection techniques to derive embeddings from the graph representation.\n\nWeaknesses\n--------------\nThe paper uses weakly defined terms like \"Common sense\". Defining these concepts in a more rigorous way would help the reader.\n\nThe authors summarize their work in the introduction. However it is not straightforward to relate this summary to the actual implementation.\nIt would be helpful if they better establish this relationship with the rest of the paper\n\n e.g.\n\"... Commonsense service that learns from experience\" \n By common-sense service are they alluding to their object detection models and relationship generator ?\nIs this \"experience\" their training data or they saying that they are creating a continuous learning system, which they aren't in this particular paper.\n\"computational models that mimic child cognition\"\nA strong reference which establishes the relationship with child cognition is necessary here.\nThe idea is intuitive and the authors claims are plausible, however the standard of this conference requires more rigorous demonstration of such claims.\n\nThe paper utilizes the performance reported in a reference to claim it is close to SoTA techniques. Typically such claims are validated by reproduction of the SoTA techniques for controlled comparisons.\n\nOne of the critical assumptions here is the accuracy of the object detection model. Though the authors have clarified that embedding extraction methods are susceptible to generate incorrect representations for out-of-domain data, they haven't provided an analysis of how the object detectors accuracy impacts the current model. This is a critical detail as OOD data affects all models.\n\nThere are quite a few grammatical mistakes in the paper which make it difficult to understand.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1yeZ8W7dpP", "reviewer": "Reviewer_ox4H", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors focus on the OOD detection task, where they consider both near OOD and Far OOD. To this end, they aim to exploit graph structures and topological properties to improve this task. Specifically, they propose a semi-supervised geometric-learning-based framework that operates on human-interpretable concepts. Finally, the authors conduct experiments on the LSUN dataset to verify the performance of the proposed method.", "summary_of_the_review": "This work is not well-prepared and also not conducted in a way of scientific research. So, I recommend a strong reject.", "strength_and_weaknesses": "Strength\n\nThis work is not well-prepared so that I cannot give points of strength.\n\nWeakness\n\n1. This paper writing should be improved. First, the notations in this work are confused, such as \"Near OOD\" and \"Far OOD\". Their notations in the introduction seems different from those in the Table 1. Second, the motivation and the proposed method are not clearly described, so I cannot give a judgement for its novelty. Third, this work totally ignores the literature of OOD detection, which is generally introduced in the survey [1].\n\n2. The empirical results are not convincing. The proposed method is not compared to existing state-of-the-art methods for OOD detection, which can be found in [1]. Instead, the authors compare the proposed method to some graph embedding algorithms, which is confused for me. Moreover, they just conduct experiments on the LSUN dataset, which is not extensive.\n\n\n[1] Yang, Jingkang, et al. \"Generalized out-of-distribution detection: A survey.\" arXiv preprint arXiv:2110.11334 (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject