NINCO (Out-Of-Distribution detection dataset for ImageNet)
2023-06-07

&lt;p&gt;The &lt;strong&gt;NINCO&lt;/strong&gt; (&lt;strong&gt;N&lt;/strong&gt;o &lt;strong&gt;I&lt;/strong&gt;mageNet &lt;strong&gt;C&lt;/strong&gt;lass &lt;strong&gt;O&lt;/strong&gt;bjects) dataset is introduced in the ICML 2023 paper&lt;strong&gt; &lt;/strong&gt;&lt;a href="https://arxiv.org/abs/2306.00826"&gt;In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation&lt;/a&gt;. The images in this dataset are free from objects that belong to any of the 1000 classes of ImageNet-1K (ILSVRC2012), which makes NINCO suitable for evaluating out-of-distribution detection on ImageNet-1K .&lt;/p&gt; &lt;p&gt;The NINCO main dataset consists of 64 OOD classes with a total of 5879 samples. These OOD classes were selected to have no categorical overlap with any classes of ImageNet-1K. Each sample was inspected individually by the authors to not contain ID objects.&lt;/p&gt; &lt;p&gt;Besides NINCO, included are (in the same .tar.gz file) &lt;strong&gt;truly OOD versions of 11 popular OOD datasets&lt;/strong&gt; with in total 2715 OOD samples.&lt;/p&gt; &lt;p&gt;Further included are &lt;strong&gt;17 OOD unit-tests&lt;/strong&gt;, with 400 samples each.&lt;/p&gt; &lt;p&gt;Code for loading and evaluating on each of the three datasets is provided at &lt;strong&gt;&lt;a href="https://github.com/j-cb/NINCO"&gt;https://github.com/j-cb/NINCO&lt;/a&gt;.&lt;/strong&gt;&lt;/p&gt; &lt;p&gt;&nbsp;&lt;/p&gt; &lt;p&gt;When using &lt;strong&gt;NINCO&lt;/strong&gt;, please consider citing (besides the bibtex given below) the following data sources that were used to create NINCO:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Hendrycks et al.: &rdquo;Scaling out-of-distribution detection for real-world settings&rdquo;, ICML, 2022. &nbsp;&lt;/li&gt; &lt;li&gt;Bossard et al.: &rdquo;Food-101 &ndash; mining discriminative components with random forests&rdquo;, ECCV 2014. &nbsp;&lt;/li&gt; &lt;li&gt;Zhou et al.: &rdquo;Places: A 10 million image database for scene recognition&rdquo;, IEEE PAMI 2017. &nbsp;&lt;/li&gt; &lt;li&gt;Huang et al.: &rdquo;Mos: Towards scaling out-of-distribution detection for large semantic space&rdquo;, CVPR 2021. &nbsp;&lt;/li&gt; &lt;li&gt;Li et al.: &rdquo;Caltech 101 (1.0)&rdquo;, 2022.&lt;/li&gt; &lt;li&gt;Ismail et al.: &rdquo;MYNursingHome: A fully-labelled image dataset for indoor object classification.&rdquo;, Data in Brief (V. 32) 2020.&lt;/li&gt; &lt;li&gt;The iNaturalist project: https://www.inaturalist.org/&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;When using &lt;strong&gt;NINCO_popular_datasets_subsamples&lt;/strong&gt;, additionally to the above, please consider citing:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Cimpoi et al.: &rdquo;Describing textures in the wild&rdquo;, CVPR 2014. &nbsp;&lt;/li&gt; &lt;li&gt;Hendrycks et al.: &rdquo;Natural adversarial examples&rdquo;, CVPR 2021. &nbsp;&lt;/li&gt; &lt;li&gt;Wang et al.: &rdquo;Vim: Out-of-distribution with virtual-logit matching&rdquo;, CVPR 2022. &nbsp;&lt;/li&gt; &lt;li&gt;Bendale et al.: &rdquo;Towards Open Set Deep Networks&rdquo;, CVPR 2016. &nbsp;&lt;/li&gt; &lt;li&gt;Vaze et al.: &rdquo;Open-set Recognition: a Good Closed-set Classifier is All You Need?&rdquo;, ICLR 2022. &nbsp;&lt;/li&gt; &lt;li&gt;Wang et al.: &rdquo;Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition.&rdquo; ICML, 2022. &nbsp;&lt;/li&gt; &lt;li&gt;Galil et al.: &ldquo;A framework for benchmarking Class-out-of-distribution detection and its application to ImageNet&rdquo;, ICLR 2023.&lt;br&gt; &nbsp;&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;For citing our paper, we would appreciate using the following bibtex entry (this will be updated once the ICML 2023 proceedings are public):&lt;/p&gt; &lt;p&gt;&lt;br&gt; @inproceedings{&lt;br&gt; &nbsp;&nbsp; &nbsp;bitterwolf2023ninco,&lt;br&gt; &nbsp;&nbsp; &nbsp;title={In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation},&lt;br&gt; &nbsp;&nbsp; &nbsp;author={Julian Bitterwolf and Maximilian Mueller and Matthias Hein},&lt;br&gt; &nbsp;&nbsp; &nbsp;booktitle={ICML},&lt;br&gt; &nbsp;&nbsp; &nbsp;year={2023},&lt;br&gt; &nbsp;&nbsp; &nbsp;url={https://proceedings.mlr.press/v202/bitterwolf23a.html}&lt;br&gt; }&lt;/p&gt;