NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds
Authors: Cynthia Feeney
Sarah Schneider
Panagiotis Lymperopoulos
Liping Liu
Matthias Scheutz
Michael C Hughes
Venue: Accepted by TMLR ()

Abstract: In order for artificial agents to successfully perform tasks in changing environments, they must be able to both detect and adapt to novelty. However, visual novelty detection research often only evaluates on repurposed datasets such as CIFAR-10 originally intended for object classification, where images focus on one distinct, well-centered object. New benchmarks are needed to represent the challenges of navigating the complex scenes of an open world. Our new NovelCraft dataset contains multimodal episodic data of the images and symbolic world-states seen by an agent completing a pogo stick assembly task within a modified Minecraft environment. In some episodes, we insert novel objects of varying size within the complex 3D scene that may impact gameplay. Our visual novelty detection benchmark finds that methods that rank best on popular area-under-the-curve metrics may be outperformed by simpler alternatives when controlling false positives matters most. Further multimodal novelty detection experiments suggest that methods that fuse both visual and symbolic information can improve time until detection as well as overall discrimination. Finally, our evaluation of recent generalized category discovery methods suggests that adapting to new imbalanced categories in complex scenes remains an exciting open problem.

Published in Transactions on Machine Learning Research (03/2023)
NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds
Patrick Feeneyl,
fltricckfeeneyyttuutt..com
Sarah Schneider
Panagiotis Lymperopoulos
Li-Ping Liu'
Matthias Scheutz!
Michael C. Hughes'
michaal Stuftss
Dept. of Computer Science, Tufts University
Center for Vision, Automation and Control, Austrian Institute of Technology
Authors PF and SS both provided lead author contributions
Reviewed OpenReview: https: openrevieeneeeee net/ forum? id=4eL629ziu7
Abstract
In order for artificial agents to successfully perform tasks in changing environments, they must be able to both detect and adapt to novelly. However, visual novelly detection research often only evaluates repurposed datasets such CIFAR-10 originally intended for object classification, where images focus on one distinct, well-centered object. New benchmarks are needed to represent the challenges of navigating the complex scenes of an open world. Our new NovelCraft dataset contains mallimodal episodic data of the images and symbolic world-states seen by an agent completing a pogo stick assembly task within a modified Minecraft environment. In some episodes, we insert novel objects of varying size within the complex 3D scene that may impact gameplay. Our visual novelty detection benchmark finds that methods that rank best on popular area-under-hh-curre metrics may be outperformed by simpler alternatives when controlling false positives matters most. Further multimodal novelty detection experiments suggest that methods that fuse both visual and symbolic information can improve time until detection as well as overall discrimination. Finally, our evaluation of recent generalized category discovery methods suggests that adapting new inbalanced categories in complex scenes remains an exciting open problem.
Gameplay: Sennplay:Fence
Gameplay: Tree
Beacon
Cactus
Glowstone
Pumpkin
Web
Figure 1: Example images for subset of novelties in NovelCraft. See Appendix Fig. A.1 for images of all 53 novelties.
1 Introduction
Recent progress in computer vision (Krizhevsky al., 2017) and vision-informed reinforcement learning (Mnih et al., 2015; Silver el al., 2018) is exciting but focused on tasks like classification video game playing where the agent's goals are narrowly defined in world that does not change. Many real applications require agents that can navigate open worlds that evolve over time (Boult et al., 2019). Designing effective agents for these domains requires the ability lo delect, characterize, and ultimately adapt to novel stimuli.
Opeen-acesss dataset, [CC-BY License] and code [MIT License]: https://tovelCraft. tufts. edu/

---

Published in Transactions on Machine Learning Research (03/2023)
To define novelty, take the point of view of an agent with some prior experience in given environment. We define stimuli as novel to the agent if it considerably differs from the agent's model (Rulf et al., 2021) Some authors refer to similar problems as anomaly detection or out-of-diste of-distrib detection. Novelties are not simply outliers. Detecting a novelty suggests that revising the agent's 's model is necessary, whereas an outlier is a rare event expected (however infrequently) under the existing model.
Work on novelty detection has spanned decades (Markou & Singh, 2003; Pimentel et al., 2014; Ruff al., 2021), with wealth of available paradigms spanning "one-class" boundary learning (Schölkopf et al., 2000), reconstruction error (Abati et al., 2019), probabilistic density modeling (Nalismick et al., 2018), and more. For vision-based novelty detection, recent progress has been dominated by modifications of deep image classifiers (Lakshminarayanee et al., 2017; Liang et al., 2018; Cheng & Vasconcelos, 2021).
Despite this wealth of methods, evaluation practices have seen less attention. Recent visual novelty detection systems are typically evaluated by repurposing object-focused datasets such as Caltoch-256 (Griffin al., 2007) or CIFAR-10 (Krizhevsky, 2009) that were originally intended for supervised classification. A typical evaluation paradigm trains "normal" subset of available classes, then tries to detect "novel" images from other classes unseen during training. We contend that when using object-focused datasets, this paradigm is too optimistic, representing best-case scenarios where the object is relatively large and well-framed within the image. While some efforts have turned toward fine-grained classification latasets like distinguishing novel breeds of dogs (Cheng & Vasconcelos, 2021), these images remain object-focusert To develop agents ready to navigate open worlds, we argue that a scene-focused beuchmark is needed.
We have developed and released comprehensive focused dataset that we call NovelCraft, for bench- marking systems that can detect and adapt to novelty in open worlds. Our target domain is modified
Standard Episodes
Frame
{"player"
"pos": [55, 17, 43].
3 "facing": "WEST",
"yaw": 90,
5 "pitch":
},
7 "entities": {
" "103": {
9 "type": "EntityTrader";
"pos": [46, 17, 43],
}, },
"map": {
145,17,522 {
Gameplay Modification:
Fence around Trees
14 "name": "plastic_chest" "facing"; "north"
16
}.
"47,17,56" {
18
"name": "log",
"axis": "y" "variant": "oak"
20
Item Modification: TNT
22
"59,17, 62"; { "name"; "air",
},
25 }}
Figure 2: Example contents of our new multi-modal NovelCraft dataset. Left: Images from two normal episodes (rows 1-2) and novel episodes (rows 3-4). Detection challenging as only a few images novel episodes actually contain novel objects, here outlined in orange when visible. Right: Example symbolic information available at each frame. At each frame, we record a complete JSON representation the player state and all objects and artificial agents the world (r,z.y position in 3D, orientation, etc.). This includes objects outside the player's view recorded in images.
2

---

Published in Transactions on Machine Learning Research (03/2023)
version of Minecraft that provides a complex, colorful 3D world with pixelated video game aesthetic. Within this world, the agent's s goal is to use carefully selected sequence of actions to turn available resources into a pogo stick. We have recorded the observations of an artificial agent rumning through hundreds of episodes of this pogo stick construction task, each one producing a sequence of images as well as non-visual symbolic information about the world's state (including all objects, not just those visible from agent's current. viewpoint). In some of these episodes, the game has been deliberately modified so that novel types of objects (e.g. TNT explosives, fences, or jukeboxes) are inserted into the world and potentially visible to the agent. Some of these novel objects can impact gameplay (e.g. trees with distinct bark color may yield more resources). Representative images from two "normal" episodes and two "novel" episodes are shown in Fig. 2, visualizing how novel objects vary in placement and size in complex scenes.
Using this data, we define three benchmark tasks. First, we have two flavors of novelty detection: (1) using vision (Sec. 4), and (2) using non-visual symbolic or multimodal information (See. 5). Next, we consider (3) characterization of novelty via a category discovery task (Sec. 6) where the system must find new objects within complex scenes and organize them within an ever-growing system of classes. We benchmark how saate-of-the-are methods perform at each task.
The contributions of this study are:
1. A scene-focused dataset purpose-built for visual novelty detection. Existing evaluations are too object-focused, artificially creating "novelty" by repurposing images designed for object classification. In Sec. 4, we benchmark visual detectors on our new data of complex scenes containing many objects, with novelties often found in the periphery or occluded by other objects when the agent first encounters them.
2. Inclusion of two useful modalities: images and symbolic world-state. In Sec. 5, show how our data supports visual as well as non-visual symbolic processing to detect novelties. Our dataset can be used to prototype multimodal methods, which are thus far under-explored in the literature.
3. Examination of metrics informed by task-specific costs. While we do report widely used area- under-curve scores to assess detectors, we contend that reporting only these metries ignores the need to pick a threshold when the system must produce binary decisions. Threshold selection must be guided by the task-specific costs of false positives versus false negatives in any deployment.
4. Going beyond detection, we assess category discovery. The ability to recognize a previously seen novelty is necessary for an agent to improve with more experience. In Sec. 6, we show how our dataset can be used to inspire improvements in generalized category discovery and share code and pretrained model for generalized category discovery.
Ultimately, we hope our work a step loward a long-lerm goal of agents that fuse vision with other sensors to adapt to open worlds. Our primary motivation not one specific real application (like self-driving cars) but rather a general style of problem-solving that we think will become increasingly common and useful: the integration of vision systems for complex scenes within the overall cognitive architectures of a goal-oriented agent to successfully adapt to novelty in 3D open worlds over time.
Rationale for focus on a virtual world. We specifically selected this Minecraft-like world because it has several key attributes that are helpful toward our long-term goal. First, it requires long-range planning over time toward a clear goal (pogo stick assembly), not just immediate stimuli-driver responses (e.g. avoid an obstacle in the road). Second, gameplay in this world allows interaction with other agents (both beneficial and adversarial). Third, gameplay requires reasoning about a combinatorial action space as well 3D movement. (vertical or horizontal). Our current benchmark and this research field in general are not at the finish line, only a few steps into a longer journcy. We posit that our datasct and evaluation practices are a uscful step forward, and the virtual platform we build upon can grow towards this goal.
2 Related Work on Datasets for Novelties and Anomalies
Object-focused datasets. Datasets showcasing distinct object lypes, such as CIFAR-10, Caltech-256, and ImageNet (Deng et al., 2009), originally intended for supervised classification, are often repurposed to assoss novelty detection under a k-classes-out paradigm (Ruff et al., 2021). This approach continues to dominate vision research related to novelty detection: The Open World Vision workshop at CVPR 2021 used repurposed images from ImageNet in their open-set recognition challenge (Kumar et al., 2021). Some
3

---

Published in Transactions on Machine Learning Research (03/2023)
Dataset
Agent pursuing goal?
Images over time?
Multimodal?
Novel classes
NovelCraft
YES
YES
YES
53
COCO Lin et al. (2014)
No
No
YFS No
91
UBnormal Acsintoae et al. (2022)
No
YES
5
Roomaly Chan et al. (2021)
No
No
No
26
Table 1: Summary of visual datasets focused on complex scenes, not individual objects. (Expanded in Tab. D.1)
recent efforts use synthetic perturbations of ImageNet images to assess robustness (Hendrycks & Dietterich, 2019) and separate novelties based on the level of semantic similarity to the training classes (Vaze al., 2021). Many recent novelty detection methods, such as work by Cheng & Vasconcelos (2021), have yet to be evaluated on scenes, where novelties may not be large, centered or fully visible in the images.
Scene-focused datasets. Some recent work has developed datasets that could be used to assess novelty or anomalies in complex visual scenes. The Common Objects in Context (COCO-2014) dataset (Lin et al., 2014) is a widely-used benchmark for scene understanding containing complex images gathered from Flickr, but was not intended to assess novelty detection or discovery. The UBnormal dataset (Acsintoae et al., 2022) contains fixed perspective videos of complex virtual scenes designed to contain anomalous events. Several efforts look at anomalies in scenes gathered from autonomous driving, such as Fishyscapes (Blum et al., 2021) or Road Anomaly (Lis et al., 2019; Chan et al., 2021).
Table 1 compares these datasets to NovelCraft. An expanded table assessing even more datasets is available in Appendix Table D.1. As seen in these tables, NovelCraft is the only dataset containing egocentric images from a goal-oriented agent exploring a 3D world over time, NovelCraft supports benchmark tasks for both detection and category discovery with 50+ classes, while other datasets offer only a limited number of object classes to be discovered (=10 in UBnormal and Fishyscapes, <30 for Romally) COCO, which does provide more classes, is not built for novelty detection, does not have images over time from an agent pursuing goal, and does not have defined dataset splits for novelty detection category discovery. Finally, NovelCraft offers multimodal data, not just visual data.
NovelCraft is complementary to other scene-focused datasets as some methods may not want to try to overcome the challenge of NovelCraft's 's severe class imbalance or aim for collaboration with an agent instead of a more typical classification setting. To our knowledge there are no ccene-focueed datasets that are class-balanced by default, but datasets larger than NovelCraft. could more easily be split into subsets to obtain class balancing without reducing performance significantly. A class balanced setting can be more practical for applications where the deployment euviromment is more controlled, such as robols deployed in a warehouse. UBnormal (Acsintoae et al., 2022) provides sequences of images for novelty detection that are useful for applications utilizing fixed cameras, such as surveillance imagery processing.
Vision for open worlds. Several works share our focus on an open world, presenting scene- locused datasets with test classes not seen in the training set. Rambhatla et al. (2021) present novelty discovery and localization results from the scene-focused COCO-2014 dataset. Liu et al. (2019) explore open long-tailed recognition of not-so-common object categories representing the "long tail" using repurposed view of the scene-focused Places dataset (Zhou et al., 2018). Lin et al. (2021) introduce the CLEAR benchmark for continual learning on scene-focused internet images from 2004-2014. Other works explore open- spen-world instance segmentation, either on the COCO dataset (Saito et al., 2022) or on images collected from cars on the road (Chan et al., 2021). Compared to these works, our dataset is distinct its multimodality and how images are gathered over time by a navigating agent.
Very recent parallel work has developed the MineDojo benchmark frame- work (Fan et al., 2022), which evaluates MineCraft agents that can learn from vision to perform 32 program- matic tasks like "hunt sheep" or "combat zombic" as well as 32 creative tasks like "build a haunted house", by training on an internet-scale knowledge base of multimedia recordings, tutorials, and discussions. This exlends early work on MineRL by Guss et al. (2019), focused on one "obtain diamond" task. Their evaluations focus on the agent's success rate on new tasks new visual environments. Our evaluations specifically evaluate novelty detection and generalized category discovery, and thus should be seen as complementary to their work. Additionally, our NovelCraft dataset provides symbolic world-state data absent in other datasets.
Related Minecraft datasets.
4

---

Published in Transactions on Machine Learning Research (03/2023)
Industrial datasets. Two recent efforts are designed to enable industrial inspection scenarios. The MVTech Anomaly Detection dataset (Bergmann et al., 2019) depicts textures like carpet or glass deliberately altered with anomalous defects, such as scratches or dents. Huang & Wei (2019) offer a similar dataset with images of circuit board defects. Unlike these works, our target scenario is an agent navigating open worlds over time, not a fixed camera on an assembly line. Further analysis of other related work can be found in App. D.
3
NovelCraft Dataset
3.1 Background: PolyCraft Virtual Environment
The virtual world we build upon (with permission from the creators) is PolyCraft?, a modification of the videogame MineCraft developed by a research team at UT-Dallas (Smaldone et al., 2017; Goss et al., 2023). Polycraft is multi-purpose software with several applications. We use a particular environment, PolyCraft. POGO.
Within the Polycraft POGO environment, an agent is tasked with exploring a 3-dimensional voxel world via sequence of actions such as move, break, or craft. Completing the task requires the agent to execute plan roughly 60 actions long. Moving the agent any distance requires only a single action, so many actions involve the agent interacting with an object or another unvionment-controleee agent. The agent's goal is to create a pogo stick from resources, such as wood and rubber, that must. be gathered from the environment. A solution to the task requires the agent to break trees to acquire wood, craft the wood into planks and sticks, move to a work bench, use the work bench to craft the planks and sticks into a tree tap, place the tree tap adjacent an unbroken tree, use the tree tap to acquire rubber, move to a work bench, and use the work bench to craft rubber, planks, and sticks into pogo stick. In the event of a novel change in the environment, such some trees not producing rubber, the agent must be able to change multiple steps in the solution, such breaking those trees for wood and not using the tree tap on them, to successfully complete the modified task.
At each step, the agent observes both an image and the environment's symbolic state in JSON text format. Example images and symbolic information are shown in Fig. 2. Each image is a 256x256 pixel RGB depiction of the agent's i current. perspective. The JSON text includes positions and names for every object in the environment and every environmentt-ontrolledd agent, as well as the agent's position and state information (what materials it has collected). This representation is compatible with other possible use cascs for agent- based open world novelty detection, where the JSON may represent the environment by listing sensor measurements.
3.2 Data Collection
Num. Classes
Num. Images
Our dataset consists of prerendered observations from an agent pursuing the POGO task, with contents sum- marized in Tab. 2. The data is or- ganized into episodes, an ordered set. of data representing the sequence of steps taken by the agent solving the POGO task. Each episode starts with different initial environment and ends after the agent has completed the task. Each episode is represented by a folder with both an RGB image and a JSON file detailing the state of the environment after each action the agent took. Each image is labeled, indicating if it depicts a non-standard object.
Split Train
Type
Episodes
Object
Gameplay
Raw
Filtered
Normal
165
3
9707
7037
Normal+
2575
0
125636
125636
Novel
0
0
0
Valid.
Normal
21
3
2
1154
873
Novel
50
5
2520
332
Test
Normal
21
3
2
1194
890
Novel
440
43
20988
3530
Table 2: Summary statisties of our NovelCraft dataset. Each episode contains a sequence of images and associated JSONs. Each episode has a class label indicating how the POGO environment was modified. The same 5 Normal classes appear in every split: 2 gameplay classes (slandard and feuce) and 3 modiffed-object classes. Novel classes are not repeated between validation and test. The Normal+ training split is additional data from the standard environment added to Normal from our larger NovelCraft-+ data.
Each episode has a class label indicating how the POGO environment. was modified. We collect images under broad scenarios: the standard enviromment with no modifications, using creator-designed modifications that impact gamoplay, and modifying the environment by inserting new objects that are visible but do not impact
Phtpp://www.pllycrttoor..d
5

---

Published in Transactions on Machine Learning Research (03/2023)
gameplay (standard agent can still solve the task). Only one modification is applied per episode, so there is never more than one novelty type in an episode. Fig. 1 shows visual examples for subset of modifications, with Fig. A.l showing visual examples of all modifications and summary statistics shown in Tab. A.2.
For all collection, we used an autonomous agent following the DIARC cognitive architecture (Scheutz et al., 2019) to solve the pogo-stick assembly task, as described in Muhammad et al. (2021). This agent does not make use of visual images, relying purely on the symbolic world-state available each step.
Standard episodes. Using the standard environment, we collected base dataset of 99 standard episodes. Our agent typically produces 45-56 images per episode (5th-95th percentile).
Gameplay-altering modifications. The UT-Dallas PolyCraft team created several modifications that would impact gameplay. We selected two that would be visually recognizable: inserting fences that block access to trees and inserting tree species with distinct bark that can be harvested more efficiently. Harvesting trees is required to build the pogostick, so both modifications modify agent behavior. We collect 80 fence episodes and 85 tree episodes, with our agent producing 29-72 images per episode (5th-95th percentile range).
Inserted object modifications. From library of possible PolyCraft objects that never appear in the standard task, we selected 51 object types (Tab. A.2) and inserted them into the POGO environment. These objects do not assist or hinder the agent attempting the POGO task, so agent behavior is not changed. We collected at least 10 episodes each, with 32-56 images per episode (5th-95th percentile range).
Extra standard episodes. To facilitate models requiring more data we have released No containing over 2500 episodes and over 100,000 additional frames of standard environment. training data. This significantly increases dataset size but results in exaggerated class imbalance between standard and novel images. We use NovelCraft+ to evaluate visual novelty detection (Sec. 4), where class imbalance is best understood and has the least impact. In Sec. 5 and Sec. 6, the original smaller NovelCraft is used for symbolic and multimodal novelty detection and generalized category discovery.
N:
Filtering to identify non-standard images. Within a modified episode, only a few images may actually show non-standard objects. Rendering semantic segmentation maps within the game engine would have been significant engineering challenge. Therefore we use the locations of agents and objects in the JSON to reconstruct the agent's view within the 3D rendering software Blender (Blender Foundation, 2018). Not all camera parameters were available, so a grid search was used to maximize mean intersection over union for non-standard objects in manually annotated images, achieving 93.0%. For each frame from a modified environment, we can thus render somantic segmentation maps highlighting non-standard objects and record the percentage of pixels containing a non-standard object. We then obtain representative frames from modified episodes by filtering out images with less than 1% non-standard content. This mitigates noise from the rendering, while maintaining challenging examples where non-standard objects are small but clear to a human. Per-frame percentage labels and code for this filtering step are provided in our data release.
Splitting classes into normal/novel and episodes into train train/valid// /ttst.. We consider 5 total class labels as normal. The "Standard" environment with no modifications, the "Gameplay:Fenee gameplay- altering modification, and the "Item:Anvil", "Item:Sand", and KteemCCoaBBlock inserted object modifications. This enables methods that repurpose classifiers of multiple normal classes to perform novelty detection. These classes were chosen for containing a high number of images depicting novelty to ensure a reasonable number of images in the training, van,dation, and test sets. Choosing classes with fewer images depicting novelty would likely decrease model performance due to a smaller training set.
Each episode of the normal classes is assigned to either training, validation, or test. To compensate for differing episode sizes, we selected a split that best matches an 8:1:1 ratio of filtered images assigned to train/vailn //aa/vaaaaiiotteeeeeeeeeeeeeeeee out of 100 possible random splits. We randomly picked additional inserted object classes to represent novelty exclusively in the validation set. Finally, all remaining inserted object classes and gameplay classes represent novel classes exclusively in the test set.
In most past work, novelty anomaly detectors are informed by only normal examples, with no examples labeled novel in training or validation sets. In contrast, our validation set provides examples from 5 novel classes as well as normal ones. We do this because in practice, often there are some available examples of
6

---

Published in Transactions on Machine Learning Research (03/2023)
novelties. To not use any such examples to inform models is a missed opportunity. This is similar to the validation set construction of Liang et al. (2018), but provides more challenging novelties (closer appearance to normal). All validation novelties are distinct from test set novelty to avoid leakage. Future users could create other splits as desired to better match their own intended use cases (e.g. no noveltics in validation).
4 Visual Novelty Detection
Here, we describe how we use NovelCraft+ to formulate and evaluate the task of visual novelty detec- tion (Sec. 4.1), which methods we consider (Sec. 4.2), and the results of our experiments (Sec. 4.3). Appendix Sec. E provides experiments assessing how training data size impacts results (NovelCraft+ vs. NovelCraft).
4.1 Task Description and Evaluation Plan
The goal of our visual novelty detection task is to develop methods that can distinguish whether an image is novel or not. The predefined training set defines 5 normal classes. The goal on test data is to assess if a given image contains content that is not in one of the normal classes.
Performance metrics. Detectors are evaluated using receiver-operating characteristic (ROC) curves and precision recall (PR) curves. These visualize tradeoffs across possible thresholds that could be applied to the decision score. The ROC curve assesses tradeoffs between true positive rate (TPR, aka recall) and true negative rate (TNR), while PR compares TPR versus positive predictive value (PPV, aka precision). PR curves provide information about imbalanced tasks like novelty detection that ROC curves alone cannot (Saito & Rehmsmeier, 2015). To summarize performance, we report the area under both curves. Higher numbers mcan better detectors. We bold the best score and any others within 1.0 percentage point.
Measuring the area under ROC and PR curves is insufficient for applications that require automated binary decisions of novel or not. Instead of averaging over many thresholds, one specific threshold must be chosen, ideally in a way that respects the task-specific costs of possible mistakes. We consider two different cost. regimes when reporting metrics like TPR, TNR, and PPV. First, we prioritize avoiding false negatives, by selecting a threshold achieving 95% TPR. Liang et al. (2018) also evaluated OOD detectors in this regime. Second, we avoid too many false positives among all alerts, via a threshold enforcing 80% precision (PPV). As we later show in Tab. 3, the relative ranking of different detectors can change depending on which regime we select. Picking a method based on area- sele--undr--hh-cuee metrics alone might lead to muder-perrormanee in the applied regime of interest. This critical distinction may be under appreciated in current evaluation practice.
Frequency of novelty. The relative frequency of the positive class (novel) matters for metrics such precision. In a typical novel episode, roughly 25% of frames depict novelty. Yet it is difficult to properly score images where very few pixels (0.01-1%) show the novel object. To handle this, we only score images in the filterd subset of the lest set, where we are more confident in the label. Then, for metrics where frequencies matter, we take a weighted average with 75% weight on normal images and 25% on novel images.
4.2 Methods for Visual Novelty Detection
We consider several detection methods representing variety of approaches. We follow Cheng & Vasconcelos (2021) in finetuning a common pretrained VGG-16 network architecture (Simonyan & Zisserman, 2015) for each method utilizing deep classifiers. Each method can access the predefined validation set to tune hyperparameters. For classifiers, grid search maximizes validation accuracy on the normal classes. Novelty detectors then select to maximize validation AUROC. Details for reproducibility (esp. settings for training and hyperparameter grid search) are in the Appendix.
We first consider NDCC (Cheng & Vasconcelos, 2021) as state-of-the-art way to adapl the training objective of deep classifier to handle novelty. NDCC learns class-specific Gaussians in the pemultimate-layer feature space, which are used to calculate Mahalanobis distances for an input image at test time. The minimum distance is used as the novelty score, which assumes novel images are embedded farther away from the closest. Gaussian than normal images.
We also try detectors that repurpose standard deep classifiers. First, a Deep Ensemble (Lakshminarayanan et al., 2017) finetunes 5 distinct VGG-16 networks on normal data, then takes the max of averaged softmax outputs the novelty score. Second, ODIN (Outtof-Distriibttion detector for Neural Networks) (Liang
7

---

Published in Transactions on Machine Learning Research (03/2023)
1.-
ODIN+ (AUPIC 36. 34%)
Ensembles (AUPRC (2.08%)
One Class SVM (AUPRC 29.82%)
0.8
(Patch) (AUIPC 43. 43.60%)
True Positive Rate
0.6
NOCC+ (AUROC 74.52%)
70.45%))
Ensemble+ (AUROC 55.41%)
0.2
Class SVM+ 58.37%)
AE+ (Patch) (AUROC 68. 87%)
0.4
1.0
0.0
0.2
0.4
0.6
0.8
False Positive Rate
(a) Receiver Operating Characteristic (ROC)
(b) Precision Recall curves
Figure 3: Visual novelty detection tradeoff curves (higher is better), evaluated per-frame on NovelCraft test images.
The + symbol method's legend name denotes that our larger NovelCraft+ dataset was used training.
at TPR
95%
al PPV 80%
Method
AUROC AUPRC
TNR
PPV
TNR TPR
NDCC+
74.52
47.60
9.32
25.89
99.89
2.55
ODIN+
70.45
38.34
0.00
25.00
100.00
0.11
Deep Ensemble+
55.41
62.08
0.00
25.00
99.89
2.86
One-Class SVM+
58.37
29.82
9.66
25.96
100.00
0.00
+
68.87
43.60
22.02
28.88
99.78 2.72
Table Visual novelty detection performance metries (percentages, higher better), evaluated per-frame on lest. images. NovelCraft+ used for all results here (denoted with +). NDCC performs best in AUROC but is outperformed by Deep Ensemble in terms of AUPRC, and further outperformed by the Autoencoder in the TPR 95% regime.
et al., 2018) creates an adversarial perturbation (Goodfellow et al., 2015), feeds that through the model, and takes the max of the tenperature-scaled softmax output as the score.
Next, we consider a denoising Autoencoder (AE) trained without labels encode normal image then reconstruct it accurately. We borrow encoder and decoder architectures from Abati et al. (2019). At test time, the reconstruction error is used as novelty score, which assumees novel classes will be reconstructed less accurately than the normal classes (Richter & Roy, 2017). Our dataset has large images (256x256), so we consider reconstructions that process 32x32 patches (whole image AEs performed worse). After reconstructing each patch, we aggregate the novelty scores via an average over all patches.
We finally try a One-Class Support Vector Machine (OC-SVM) (Schãlkopf et al., 2000). We use the penultimate layer of finetuned VGG-16 features and pick an RBF kernel with tuned lengthscale. This model was fit subset of 10,000 images due to the method scaling poorly with dataset size.
4.3 Results and Analysis of Visual Novelty Detection Experiments
Tradcoff curves. Using our new dataset, we evaluated all detection methods described in Sec. 4.2 and report their ROC and PR curve in Fig. 3. NDCC clearly outperforms other methods in AUROC overall, but is beaten by ODIN and Autoencoder at high FPR. Deep Ensemble and the Autoencoder perform well at very low FPR but are surpassed by NDCC above FPR 5%. The One-Class SVM outperforms Deep Ensemble above FPR 30%. The same trends are seen in the precision recall curves which better visualize the effects of novel classes composing only 25% of the lest. set. Deep Ensemble and Autoencoder reach higher precision than others at low recall values. NDCC surpasses the Autoencoder at 20% recall and remains the best performing method until beaten by ODIN 90% recall. These results suggest that Deep Ensembles or Autoencoders may be preferred in applications requiring high precision, despite NDCC and ODIN generally being considered state of the art.
8

---

Published in Transactions on Machine Learning Research (03/2023)
When false negatives matter most. In Table 3, when enforcing TPR of 95%, the Autoencoder performs best followed by the One-Class SVM and NDCC. However, there is still clearly room for significant improvement. as false positives outnumber both true negatives and true positives.
When false positives matter most. In Table 3, when enforcing precision of 80%, we find that NDCC, the Autoencoder and Deep Ensemble perform similarly well with respect to their TNR. TPR values are suprisingly low in the 80% precision regime. This suggests an opportunity for models specialized for low false positive regimes (Rath & Hughes, 2022).
5 Symbolic and Multimodal Novelty Detection
While Sec. 4 focused exclusively on visual processing, we will now compare vision-only models to symbolic models that consume the JSON world-state available at each frame, as well as multimodal models that combine visual and symbolic input. To make this evaluation fair and interesting, here we must focus exclusively on gameplay-altering novelties only, so that the symbolic detection task is non-trivial. In this case, it makes more sense to evaluate novelty detection at the episode-level (aggregating across frames observed over time) rather than at the frame-level. Understanding episode-level performance is valuable for assessing agents acting in the same world over time, as we can highlight key properties such as time-delay until correct detection.
5.1 Task Description and Evaluation Plan
Focus on gameplay modification. In our dataset, most modified enviromments introduce novel objects into the game world that would be trivial to detect symbolically as never-before-seen key in the JSON description of the world. However, the gameplay novelties by design cause changes in the observable behavior of both agent and environment over time, which good agent could detect. We thus focus on these gameplay novelties in this task, avoiding trivial casiness by masking any non-standard JSON keys. Because our numerous inserted-object novelties are not appropriate for this task, only a few gameplay novelties remain available. We thus use only the standard episodes from the base NovelCraft dataset for training and validation to maximize diversity in this task's test set, which contains standard episodes and all available gamoplay novelty episodes. In addition to the Fence and Tree novelties, we add new Supplier and Thief gameplay novelties, which are exclusive to this task due to only being labeled at the episode level instead of at the frame level. These novelties introduce artificial agents that respectively help and hinder the player, and were released after our initial experiments in Sec. 4 were completed.
Episode-level predictions. In our proposed benchmark task, models have access to both the RGB image and the JSON symbolic state at each frame in an episode (vision-only methods will ignore the JSON, and symbolic-only methods will ignore the image). The required output is one novelty score for each episode. For simplicity, all models predict novelty scores for indivicual frames, then to score an episode we take the maximum score over all frames. Other strategies for episodo-level modcling can be tried in future work.
Performance metrics. We use the same performance metrics as in Sec. 4.1, evaluated on episodes instead of single frames. While novelty may be easier to detect after experiencing all frames in an episode, failing to detect novelties until the very end of an episode compromises an agent's 's ability to adapt or take advantage of novelty. We thus also report average delection delay: the number contiguous frames from the start before a true novelty is detected, averaged across all novelty-containing episodes.
5.2 Methods for Multimodal Novelty Detection
Symbolic approach. We reason that jointly modeling the agent's 's behavior and its environment is useful, as both may be affected by gameplay novelty. We transform the JSON into a multivariate time-series where each frame is represented by vector x, of 24 numeric and 2 categorical variables selected from JSON entries for the player and the world. We model the time series via an K-th order autoregressive model, trained to minimize predictive error over the normal-only training set (Choi et al., 2021; Blázquez- García et al., 2021). At test time, for cach frame, the model predicts the current value &, from fixed context of the K previous observed frames. The novelty score at frame i is then computed as the error between predicted and observed values: ||& - xi The architecture of our symbolic detector is a simple multi-layer perceptron (MLP), with two autoregrossive context sizes: K E (2, 5). Hyperparaneters such as number of layers, hidden dimensions and drop-out rate are tuned for each K to minimize validation error. Reprodueíblity details are in the Appendix.
9

---

Published in Transactions on Machine Learning Research (03/2023)
at
TPR 95%
at PPV 95%
Model
AUROC AUPRC
TNR
PPV
Delay
TNR
TPR
Delay
Visual (AE Patch)
81.62
95.61
31.34
88.72
5.32
80.60
66.47
14.41
Symbolic (2 step)
76.12
94.18
4.48
84.96
10.39
98.51
5.04
33.24
Symbolic (5 step)
81.51
95.28
11.94
85.97
10.19
98.51
5.04
32.89
Multimodal (2 step)
80.46
94.65
34.33
89.15
7.44
98.51
5.04
33.14
Multimodal (5 step)
85.17
95.83
34.33
89.15
6.93
77.61
75.07
12.72
Table 4: Episodic novelty detection performance (higher is better for all but delay) on gameplay-only test episodes
(Sec. 5). Multimodal models perform better than their components with similar delay visual models. These per-episode results are not directly comparable with per-frame eveluation in Tab. 3.
Multimodal approach. Our multimodal model is a simple ensemble of the visual and symbolic base detectors. For each base detector, we normalize scores to range from 0 and 1 on the validation set. At test time, the multimodal detector reports the sum of the two normalized scores from visual and symbolic. We chose a patch- based autoencoder for the vision component because it has good performance (top 3 ranking or better in every metric of Tab. 3) and does not require more than one training class (unlike NDCC or ODIN).
5.3 Results and Analysis: Symbolic and Multimodal Novelty Detection
Tab. 4 reports the performance of visual, symbolic, and multimodal detectors. Overall, this episode-level evaluation intuitively produces higher TPR, TNR, and PPV scores than previous frame-level analysis, due to less sensitivity to frame-level noise. Looking at symbolic performance, 5-frame contexts yield significantly better performance than 2- frame contexts on ROC based metrics, suggesting medium-rany range temporal modeling matters for our benchmark. Finally, the multimodal ensemble niccly blends the best of both visual and symbolic models to improve performance with delay similar to the visual model across both cost regimes. Future methods have plenty of room to improve (human performance would be close to 100% AUROC).
6
Generalized Category Discovery
6.1 Task Description and Evaluation Plan
Both generulized category discovery (GCD) (Vaze et al., 2022) and novel category discovery (NCD) (Zhao & Han, 2021; Han et al., 2020) consider the problem of assigning labels to a set of unlabeled data by utilizing a set of related labeled data. NCD assumes this unlabeled data contains only new classes not seen in the labeled data, while GCD allows labeled classes and new classes to appear in the unlabeled set.. Unlike novelty detection, both the labeled and unlabeled data are available during training. During testing, clustering accuracy is used to evaluate the labels assigned to the unlabeled set. We additionally report clustering accuracy on the subset of classes seen in the labeled set as well as the new classes.
To try GCD on our base NovelCraft data, we utilize our training split as the labeled set and our validation split as the unlabeled set. This results in 5 classes in the labeled set and 10 classes in the unlabeled set, which includes new examples the labeled set classes. In our evaluation we assume prior knowledge of the number of new classes in the unlabeled set, but future evaluations may utilize the mothod in Vaze et al. (2022) select the number of new classes. We leave GCD assessment on our larger NovelCrafi++ to future work, methods for GCD under severe class imbalance have not been well explored to our knowledge.
6.2 Methods for Category Discovery
GCD SSKM. Following Vaze et al. (2022), use ViT-B-16 (Dosovitskly et al., 2020) vision transformer pretrained with DINO (Caron et al., 2021) self-supervision on unlabeled ImageNet then fine-tuned with the GCD contrastive loss on NovelCraft. The contrastive loss proposed by Vaze et al. (2022) has an unlabeled term and a labeled lerm, with the unlabeled term applied to all data and the labeled term applied only to labeled data. The unlabeled term encourages two random augmentations of the same image to be mapped close to cach other in embedding space, penalizing being close to different images in the same mini-batch. The labeled term encourages each pair of images that share a class to be mapped close to each other in embedding space, penalizing being close to images from other classes.
10

---

Published in Transactions on Machine Learning Research (03/2023)
sampling is utilized on the labeled set to account for class imbalance. The learned features from this model are then fed into semi-supervised k-means (SSKM) (Vaze et al., 2022), which learns cluster assignments for all examples that respect known labels from the labeled set.
DINO SSKM baseline. To examine the effectiveness of GCD's fine-tuning, we compare to a semi-supervised k-means baseline that uses pretrained DINO features. We chose this instead of unsupervised k-means as Vaze et al. (2022), so that both methods utilize the labeled set examples.
Semi-supervised Gaussian mixture. To examine the effectiveness of SSKM's clustering, we compare to semi-supervised Gaussian mixture model (SSGMM). SSGMM is new extension of Gaussian mixture models that applies the semi-supervision method of SSKM (Vaze et al., 2022) to learn cluster assignments that respect known labels from the labeled set. SSGMM should be more flexible as it allows soft assignment (multiple clusters can take partial responsibility for a given data example), can learn the frequencies of each cluster, and can learn the appropriate scaling of a cluster-speeific spherical covariance. In contrast, k-means assumes hard assignment (each data example matched to exactly one cluster), assumes uniform cluster frequencies, and represents the limiting behavior when all clusters share single spherical covariance whose scale diminishes to zero. We use modified version of expectation maximization (Dempster et al., 1977) to fit this GMM in semi-supervised fashion. Semi-supervision is introduced by modifying the expectation step so that each labeled example is assigned to the correct cluster corresponding to its label as in SSKM, with no responsibility assigned to incorrect clusters.
6.3 Results and Analysis
GS
373
0
37
3
19
16
1
39
Fig. 4 provides quantitative results from our generalized category discovery experiments. The fcatures learned during GCD training clearly improve clustering accuracy, with most of the increases coming from the labeled set. The choice of semi-supervised clustering method has little effect after GCD training, but SSGMM improves labeled accuracy over SSKM using pretrained DINO features.
G1
13
265
38
0
36
300
0
0
0
9
0
0
Thue label
13
0
0
0
0
12
0
0
0
200
17
0
52
150
Comparing to results on classification datasets Vaze et al. (2022), NovelCraft is found to be far more challenging than CIFAR-10 (Krizhevsky, 2009), the only other dataset tested with comparable class splits. Comparable loss in accuracy between labeled and new classes is only seen in experiments with hundreds of classes on finegrained classification datasets.
65
0
4
0
3
17
46
0
0
0
2
0
0
2
-50
18
16
0
45
0
10
G5
61
13
16
17
Predicted label
Clustering Accuracy
Method
All
Labeled
New
The confusion matrix reveals two ways in which class imbalance and scene-focused images, not present in pre- viously tested classification datascts, significantly reduces accuracy. In the labeled classes, see that the small labeled item classes (like T3) are clustered with the more frequent new item classes (15), despite labels informing the semi-supervised k-means step. Additionally, many of the new classes are clustered with the standard class. The lack of confusion with the labelod gameplay novelty class, which has similar frequency to the standard class, suggests opportunity for learned features to distinguish novelty from the standard class specifically.
GCD SSKM
72.3
83.9
41.9
GCD SSGMM
71.5
83.7
39.1
DINO SSKM
30.5
32.4
25.7
DINO SSGMM
34.8
38.3
25.5
Figure 4: Visual category discovery results. Top: Confusion matrix for GCD SSKM. The labeled set includes gameplay classes (GS, G1) and 3 item modi- fications (I1, 12, 13). Remaining classes are new item modifications. Bottom: Table of clustering accuracy (percentages) for all classes, labeled classes, and new classes (only in the unlabeled set). We report the mean accuracy across 5 clustering fits from random initializations, using the same neural representations.
Improving GCD to address class imbalance in unlabeled data will be challenging. Balancing the labeled examples used by sunirvuper k-means risks increasing the splitting of more frequent classes, such as the predicted 13 and I4 classes consisting of subset of the most frequent classes. Separating new classes from the standard class likely requires specialized modifications to the loss, as all unlabeled data resemble standard class images to some extent.
11

---

Published in Transactions on Machine Learning Research (03/2023)
7 Discussion and Outlook
We hope our NovelCraft datasel inspires new methods development on novelty delection and characterization, enabling agents to explore open worlds.
Limitations. Our benchmark focuses on one specific open world. We wish to avoid overclaiming about generality (Raji et al., 2021). Many other open worlds are possible. The kinds of visual properties we call novel emphasize object types. We could instead have varied counts, sizes, spatial relationships, or co-occurrance patterns between objocts. We include only few gameplay modifications duc to the cost of creating these. More diverse gameplay novelties that are not trivial to detect symbolically would be worth future investment. Our multimodal experiments are limited to our specific choice for JSON data. JSON can encode less precise sensory imputs which are worth further exploration.
Future Directions. We can foresee several promising lines of work building on our NovelCraft data:
Multimodal novelty detection. A recent review suggests that multimodal novelty detection is "a largely unexplored research area" " (Pang et al., 2022). Our dataset provides two complementary modalities, symbolic and visual, that could both be leveraged by future methods.
Continual learning. Our new dataset is naturally episodic in nature. Future work could continual learning paradigm (Lin et al., 2021) instead of "train once, then deploy", which is far more realistic for open-world learming. Our discovery task in Sec. 6 could be promising first step.
From vision to action. Our dataset is collected by a real artificial agent interacting in a virtual world. Future work could use visual novelty processing to inform how the agent selects actions to solve its task. Active perception especially promising: novel objects may or may not look novel from all perspectives; an agent may need actively obtain distinct perspectives to determine new object's properties and how it could be used in service of the agent's goal.
Broader Impact Statement
We have designed our dataset with the goal of positive benefit to society. We think novelty detection and characterization can help build agents that recognize when a model not suitable (and might even be harmful) and adapt accordingly. We hope that, our findings on the difficulties of adapting models from research settings to applications encourages users to always inspect model's potential impacts. We recognize that there is a potential for misuse, as with all classifiers and especially visual oncs, if the classifier is used to discriminate by gender, ethnicity, and other protected categories. Ultimately, we hope that the potential of novelty detection to preempt accidental misuse will outweigh the impacts of purposeful misuse of these techniques.
Acknowledgments
This work was primarily funded by the Defense Advanced Research Projects Agency (DARPA), a research and development agency of the United States Department of Defense, as part of the Science of Artificial Intelligence and Learning for Open-world Novelty (SAIL-ON) program under grant W911NF-200220000 Author SS reports funding from the Austrian Institute of Technology via the Decision Making and Cognitive Control of Complex Systems program.
We thank the members of the PolyCraft team at UT Dallas, especially Dr. Eric Kildebeck, Stephen Goss, and Dr. Walter Voit, for creating an interesting platform for novelty research, approving the release of images for this dataset, and providing support throughout the DARPA SAIL-ON program.
At Tufts University, we thank Evan Krause and Ravenna Thielstrom for their work in developing the DIARC agent for Polycraft.
12

---

Published in Transactions on Machine Learning Research (03/2023)
References
Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Latent space auloregression for novelty detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 481-490, 2019.
Andra Acsintoae, Andrei Florescu, Mariana-Iuliana Georgescu, Tudor Mare, Paul Sumedrea, Radu Tudor lonescu, Fahad Shahbaz Khan, and Mubarak Shah. UBnormal: New Benchmark for Supervised Open-Set. Video Anomaly Detection. In IEEE Computer Vision and Pattern Recognàtion (CVPR). arXiv, 2022. URL
http://arxiv.oor abslaaas..././//aaaaaeaaaaaaaaat 08644.
Jerone Andrews, Thomas Tanay, Edward Morton, and Lewis Griffin. Transfer representatioo--lemming for anomaly detection. JMLR, 2016.
Christoph Berger, Magdalini Paschali, Ben Glocker, and Konstantinos Kamnitsas. Confidence-based out-of- distribution detection: a comparative study and analysis. In Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis, pp. 122 132.
Springer, 2021.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. MVTec AD - A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection. In 2019 IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 9584 9592, Long Beach, CA, USA, 2019. IEEE.
Ane Carcca, Angel Conde, Usue Mori, and Jose A Lozano. A review on outlicr/anoman detection in time series data. ACM Computing Surveys (CSUR), 54(3):1-33, 2021.
Blender Poundation. Blender a 3d modelling and rendering package, 2018. URL http://www www.:blenie blender. org..
Hermann Blum, Paul-Edouard Sarlin, Juan Nicto, Roland Siegwart, and Cesar Cadena. The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation. International Journal of Computer Vision, 129(11):3119 3135, 2021.
T. E. Boult, S. Cruz, A. R. Dhamija, M. Gunther, J. Henrydoss, and W. J. Scheirer. Learning and the Unknown: Surveying Steps toward Open World Recognition. Proceedings of the AAAT Conference on Artificial Intelligence, (3(00))9801-988)) 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Robin Chan, Kraysztof Lis, Svenja Uhlemeyer, Hermann Blum, Sina Honari, Roland Siegwart, et al. SegmeettMelfYouCan A Benchmark for Anomaly Segmentation. In Proceedings of the Newral Information Processing Systems Track on Datasets and Benchmarks, 2021.
Jiacheng Cheng and Nuno Vasconcclos. Learning Deep Classifiers Consistent with Finc- Eine-Grained Novelty Detection. In 2021 IEEE/CVI Conference on Compuler Vision and Pallern Recognition (CVPR), pp- 1664-1673, Nashville, TN, USA, 2021. IEEE.
Kukjin Choi, Jihu Yi, Changhwa Park, and Sungroh Yoon. Deep learning for anomaly detection in time-series data: Review, analysis, and guidelines. IEFE Access, 2021.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1):1-22, 1977.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2009.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. Xiv preprint ar 2xxo:2001. 19999, 2020.
13

---

Published in Transactions on Machine Learning Research (03/2023)
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, et al. MINEDOJO: Building Open-Ended Embodied Agents with Internet-Scale Knowledge. In Advances in Newral Information Processing Systems Datasets and Benchmarks Track, 2022. URL j8I8PX.
net/pdf?id=rcso
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Represenlations, 2015.
Stephen A Goss, Robert J Steininger, Dhruv Narayanan, Daniel V Olivença, Yutong Sun, Peng Qiu, Jim Amato, Eberhard O Voit, Walter E Voit, and Eric J Kildebeck. Polycraft World AI Lab (PAL): An extensible platform for evaluating artificial intelligence agents. arXiv preprint arXiv:2301. 11891, 2023.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. Technical Report CNS-TR-2007,001, California Institute of Technology, 2007.
William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint ar arxiv:107. 13440, 2019.
K Han, SA Rebuffi, S Ehrhardt, A Vedaldi, and A Zisserman. Automatically discovering and learning new visual categories with ranking statistics. In Proceedings of the 8th Intennational Conference on Learning Representations, ICLR 2020. Schloss Bngstuul-Leibniz-Zeene
für Informatik, 2020.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903. 12261, 2019.
Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmiclinski. The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXic:1805.006% 2018.
Weibo Huang and Peng Wei. A peb dataset for defects detection and classification. arXiv preprint arXiv:1901008044 2019.
Dioderik Kingma and Jimmy Ba. Adam: A mothod for stochastic optimization. International Conference Learning Representations, 12 2014.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto, 2009. URL https://www.r cs. toronto.
pd.
Alex Krizhevsky, llya. Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6))84-90, 2017.
Pavel Kumar, Amubhav, Abhinav Shrivastava, and Shu Kong. User Guide for Open World Vision Challenge. Technical report, 2021. URL https://www. cs. cmu. www.ccc......e..eeemm..e..e www.ee//shhuukppn-oooodddddd--ee
html.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncortainty estimation using deep ensembles. Advances in newral information processing systems, 2017.
Weixin Li, Vijay Mahadevan, and Nuno Vasconcelos. Anomaly Detection and Localization in Crowded Scenes. IEEE Transactions on Pottern Analysis and Machine Intelligence, 36(1), 2014.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing The Reliability of Neural Networks. Tn International Conference on Learning Representations, 2018. URL https: //openreview inrperrrveee.net/ooruuriiteeee net/forum?id==HGGG
Image Detection
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, et al. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision (ECCV), 2014.
Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The CLEAR Benchmark: Continual LEArning on Real-World Imagery. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. URL https://taaassstt--bbnntt http:////...ttttooooooooooooeoen
www.eeedinnn.nnuiipp.. wctpppcee//21/////////pp ///le//////////////// pdf.
14

---

Published in Transactions on Machine Learning Research (03/2023)
Krzysztof Lis, Krishna Kanth Nakka, Pascal Fua, and Mathien Salzmann. Detecting the Unexpected via Image Resynthesis. In 2019 IEEE/CVF International Conference Computer Vision (TCCV), pp. 2019.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2537-2546, 2019.
Markos Markou and Sameer Singh. Novelty detection: A revieww part 1: Statistical approaches. Signal Processing, 83(1))24481 2497, 2003.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Venoss, Marc G. Bellemare, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
Faizan Muhammad, Vasanth Sarathy, Gyan Tatiya, Shivam Goel, Saurav Gyawali, Mateo Guaman, Jivko Sinapov, and Matthias Scheutz. A Novelty-Centric Agent Architecture for Changing Worlds. In Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 9, 2021.
Eric Nalisnick, Akíhiro Matsukawa, Yee whye Teh, Dilan Gorur, and Balaji Takshminarayanan. Do Deep Generative Models Know What They Don't Know? In International Conference on Learning Representations, 2018. URL https: https://openrevieu het/forum?iddii
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep Learning for Anomaly Detection: A Review. ACM Computing Surveys, 54(2):1-38, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Newral Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL lea--eeennnng--iee pdf.
www..rs....rr.p....... cc/ /
9015-pytorce an-
peytorch-an-immeaattte imperative-style- htihl---iii---rrrroaaaa performance
Luis Patino, Tahir Nawaz, Tom Cane, and James Ferryman. PETS 2017: Dataset and Challenge. In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)), 2017.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Marco A.F. Pimentel, David A. Clifton, Lei Clifton, and Lionel Tarassenko. A review of novelty detection. Signal Processing, 99:215 249, 2014.
Tnioluwa Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. AT and the everything in the whole wide world benchmark. In Thirty-fifth Conference on Newral Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreviet net/ forum?id= forum??de=-OlxppbbEE..
Bharathkumar Ramachandra, Michael J. Jones, and Ranga Raju Vatsavai. A Survey of Single-Scene Video Anomaly Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2223-22)) -2122, 2022.
Sai Saketh Rambhatla, Rama Chellappa, and Abhinav Shrivastava. The Pursuit of Knowledge: Discovering and Localizing Novel Categories using Dual Memory. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9133 9143, Montreal, QC, Canada, 2021. IEEE.
Prectish Rath and Michael C Hughos. Optimizing Early Warning Classifiers to Control Falsc Alarms via a Minimum Precision Constraint. In Artificial Intelligence and Statistics (AISTATS), 2022. URL
https://proceedings. mlr. ptwps///5.//rra22ee.... html..
15

---

Published in Transactions on Machine Learning Research (03/2023)
Charles Richter and Nicholas Roy. Safe visual navigation via deep learning and novelty detection. Robotics: Science and Systems XIII, 2017.
L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller. A Unifying Review of Deep and Shallow Anomaly Detection. Proceedings of the IEEE, pp- 1-40, 2021.
Kuniaki Saito, Ping Hu, Trevor Darrell, and Kate Saenko. Learning to detect every thing in an open world. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviu, Israel, October 23-27, 2022, Proceedings, Part XXIV, pp. 268 284. Springer, 2022.
Takaya Saito and Mare Rehmsmeier. The Precision- Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLoS ONE, 10(3), 2015.
Matthias Scheutz, Thomas Williams, Evan Krause, Bradley Oosterveld, Vasanth Sarathy, Tyler Frasca, et al. An Overview of the Distributed Integrated Cogmition Affect and Reflection DIARC Architecture. In Cognitive Architectures, Intelligent Systerms, Control and Automation: Science and Engineering, pp. 165-193. Springer International Publishing, 2019.
Bernhard Schölkopf, Robert C Williamssn, Alex J. Smola, John Shawe-Taylor, and John C. Platt. Support Vector Method for Novelty Detection. In S. A. Solla, T. K. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems, pp. 582 588. MIT Press, 2000. URL http: //tp:er/pppp.a//e/ee nips. cc/paper/ 1723- wwtppporr-rettto--eedoo--o--oo-o detection pff..
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362 (6419):1140 1444, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Ronald A. Smaldone, Christina M. Thompson, Monica Evans, and Walter Voit. Teaching science through vidco games. Nature Chemistry, 9(2):97-102, 2017.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set
classifier is all you nced. In International Conference on Learning Representations, 2021.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In IEEE Conference on Compuler Vision and Pollern Recognition, 2022.
Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. Advances in Neural Informalion Processing Systems, 34:22982 22994, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 Million Image Database for Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6): 1452-1464, 2018.
16

---

Published in Transactions on Machine Learning Research (03/2023)
Appendix Contents
Dataset Documentation
17
B Dataset Nutrition Label
20
C Training and Model Details
21
D Further Comparisons to Related Work
23
E Further Analysis of NovelCraft:: Extra Train Set of Standard Images
25
A Dataset Documentation
A.1 Links to Data and Code
Dataset Website: http:://novelrrrff.csss.. edu
Data Download: http::/:ttt....ooaacaaaaaa/aaaat
Dataset Code:
Methods Code: tree/TMLR
GCD Datasets Code:
A.2
Licensing
The dataset is released under a CC-BY license and the associated code is released under MIT License.
A.3
Hosting and Maintenance
The dataset will be hosted by Tufts University through their cloud service provider cox.com. Tufts is planning on using this service indefinitely, but the dataset will be transferred and links updated if a provider change does occur.
Additionally, the Tufts University Computer Science Department will host website with documentation, dataset download links, and backups of all externally hosted files. This is supported by the department's 's dedicated IT team, which ensures server maintenance and backups. This website and server will be maintained indefinitely.
A.4 Statistics and Examples
See Fig. A.1 on the next page for an example image from every possible novelty type.
See Table A.2 for statistics summarizing the number of filtered episodes and images available for every single novelty type.
17

---

Published in Transactions on Machine Learning Research (03/2023)
Comeplay Fence
Cameplay Tree
Anvil
Beacon
Bed
Bookshelf
BrewingStand
Cactus
Cake
Cauldron
Clay
CoalBlock
Cobblestone
Deadbush
Dispenser
Dropper
EmeraldBBockk Enchanting Table Table
Glowstone
Gravel
Hopper
Ice
IronBlock
Jukebox
LapisBlock
Lever
Mycelium
Netherrack
Noteblock
Obsidian
Piston
Planks
Prismarine
Pumpkin
QuartzBlock
Reeds
Sand
Sandstone
SeaLantern
Slime
Snow
SoulSand
Sponge
Stone
Stonebrick
Tallgrass
Tht
Torch
Vine
Waterlily
Web
Wheat
Wool
Figure A.1: Example image every novelty NovelCraft.
18

---

Published in Transactions on Machine Learning Research (03/2023)
Modification Name
Modified Episodes
Total Filtered Frames
Gameplay:Fent Fence
3581
Gameplay: Tree
619
Item:Anvil
10
97
Item:Beacon
10
55
Ttem:Bed
10
81
Item:Bedrock
10
420
Ieem:Bookshllf
10
77
htmmBrewingstandd
91
Ttem:Cactus
89
Item:Cake
10
66
Nem:Cauldron
68
Item:Clay
9
68
Item:CoalBlock
8
109
Nem::obblestonee
9
72
hem:Deedbush
33
Mem::Dsspnnser
85
Item:Dropper
57
Teem:FFmerallBBlld
82
Item::nchanting Table
10
84
Ilowstlo
8
45
Item:Grave
10
59
Ttem:Hopper
10
77
Item:Ice
75
Item:IronBlock
80
Ttem:Jukebox
10
54
Item:LapisBlock
79
Item:Lever
10
61
Item:Mycelium
10
98
Item:Netherrack
79
Item:Noteblock
71
Ttem:Obsidian
63
Item:Piston
10
83
Item:Planks
10
91
Irem:Prismarine
8
74
Ttem:Pumpkin
10
64
llem:QuartzBlock
9
72
Item:Reeds
32
Ttem:Sand
116
Item:Sandstone
53
Ttem:SeaT.nter
71
Item:Slime
8
56
Item:Snow
10
93
Ited:SoulSand
90
Item:Sponge
54
Item:Stone
43
Item:Stonebrick
56
Taem:aallgsass
76
Item:TNN
8
52
Item:Torch
10
84
item:Vine
9
73
Mte:WWaerrllyy
7
18
Web
61
Item: Wheat
19
Wool
10 Table A.2: Count available episodes and frames available after filtering so at least 1% pixels show novelty.
79
19

---

Published in Transactions on Machine Learning Research (03/2023)
B Dataset Nutrition Label
Dataset nutrition label for NovelCraft dataset, following the suggested format from Holland al. (2018).
B.1 Metadata
Filename NovelCraft
File Format zip
URL https://novelcraft..caa tufts. edu/
Domain video game
Keywords novelty detection, anomaly detection, generalized category discovery, multimodal, computer vision, open world
Type image (PNG), symbolic (JSON)
Dataset Size 4.5 GB
Missing 0%
License CC BY 4.0
Release Date June 2022
Description Scene-focused, multimodal, episodic data of the images and symbolic world-states seen by an agent completing pogo-stick assembly task within a video game world. Classes consist of episodes with novel objects inserted. A subset of these novel objects can impact gameplay and agent behavior. Novelty objects can vary in size, position, and occlusion within the images.
B.2 Provenance
Environment Source The Polycraft mod was created by Ronald A. Smaldone, Christina M. Thompson, Monica Evans, and Walter Voit and is available at:
http:://www.ooyyrrfffoorlz com/
The environments used are available at:
2.0
Additional details are provided in Goss et al. (2023).
Agent Source The DIARC architecture was produced by Matthias Scheutz, Thomas Williams, Evan Krause, Bradley Oosterveld, Vasanth Sarathy, and Tyler Frasca. The Polycraft agent was produced by Evan Krause and Ravenna Thielstrom.
Dataset Authors NovelCraft was produced by Patrick Feeney, Sarah Schneider, Panagiotis Lymperopoulos, Liping Liu, Matthias Scheutz, and Michael C. Hughos and is available at:
https://novelcrftee ca. tufts. edu/
B.3 Variables
ID is composed of the episode type, episode ID, and frame ID. The first two parts of the ID are specified by string and integer folder names. The frame ID specified by the integer file name.
Image is 256 by 256 pixel RGB image associated with ID.
JSON is symbolic representation of the environment's 's state. This includes "player" data, "entities' other than the player, and 'map' data detailing the position and properties of objects.
Target provided with the dataset code as "targots.csv' and lists the percentage of novel pixels for cach image. Images with target values less than 1% are discarded while others are given the appropriate run type label.
20

---

Published in Transactions on Machine Learning Research (03/2023)
Splits is provided with the dataset code 'splits.csv" and lists the splits for the training set classes, divided by episode. Which classes are assigned to novel validation and novel test are listed in 'data const.py'.
C Training and Model Details
C.1 Visual Novelty Detection
Except for NDCC and One-Class SVM, all methods are implemented in Py Torch (Paszke et al., 2019) and are trained and tested
GPU (NVIDIA TITAN X Pascal). NDCC was trained and tested NVIDIA RTX 2080 GPU, but note that fewer training epochs is the primary reason for reduced training time. The One-Class SVM is implemented using the SciKit-Learn software package (Pedregosa et al., 2011) and is trained and tested on a CPU. See Table C.1 for training times.
Training Time
Method
NovelCraft
NovelCraft
NDCC
Od 00h 29m 18s
Od 08h 33m 358
VGG Backbone
1d 04h 51m 50s
19d 10h 23m 40s
ODIN
1d 04h 51m 50s
19d 10h 23m 40s
Doep Ensemble
5d 23h 25m
25s
96d 10h 13m 15s
One- Class SVM 1d 09h 28m 25s
21d 07h 21m 05s
Autoencoder
Od 20h 01m 53s
8d 01h 41m 24s
Table C.1: Training time for the implemented Visual novelty detection methods for NovelCraft and NovelCraft Results denoted with include the training time for one VGG-16 model. Results denoted with include the time needed train 5 separate VGG-16 models, though note this could be parallelized if sufficient hardware available.
C.1.1 VGG-16 Backbone
A VGG-16 (Simonyan & Zisserman, 2015) is used as a backbone for deep ensemble and ODIN and as feature extractor for the one-class SVM.
Architecture. The VGG-16 model of the PyTorch framework (Paszke al., 2019) pretrained on ImageNet (Deng et al., 2009) is adapted and fine tuned to classify the normal classes. The pretrained classification head is replaced with randomly initialized 5 output fully connected layer classication head.
Training Details. We optimize the parameters using the cross-entropy loss function and the Adam optimizer (Kingma & Ba, 2014). The "Minecraft item bar' is removed by cropping 22 pixel rows at the bottom of the images. The image intensity values are first scaled to [o, 1] and then normalized by mean values 0.485, 0.456, 0.406 and standard deviations 0.229, 0.224, 0.225. Mean accuracy on the NovelCraft training set 98.0%.
Hyperparameter Tuning. We tried batch sizes of 16, 32 and 64 images for the VGG-16 classifier fine tuning and found batch size 16 to perform best in terms of per-normal-class accuracy the validation set. The VGG-16 classifier is trained for 1000 epochs. The learning rate is set to 10-5, all other parameters of the Adam optimizer are set to the default values of PoTorc (0 weight decay and running average coefficients $1,2 = (0.9, 0.999)). Mean accuracy on the validation set is 96.9%.
C.1.2 Deep Ensemble
Architecture. The deep ensemble consists of 5 distinct VGG-16 classifiers The architecture of each VGG-16 classifier models structured as described in C.I.I, just using different random initializations in the classification head.
C.1.3 NDCC
Architecture and Training Details. The architecture and training follow the work of Cheng & Vasconcelos (2021), with the following changes to hyperparameters. Accuracy the NovelCraft+ training set is 90.5%.
21

---

Published in Transactions on Machine Learning Research (03/2023)
Hyperparameter Tuning. Each set of hyperparameters used in the supplement of Cheng & Vasconcelos (2021) was tested, with the Stanford Dogs hyperparameters found to work best. To fine-tune further, the Stanford Dogs hyperparameters were modified with all learning rates multiplied by 0.1 or 0.01. The modification with learning rates multiplied by 0.1 was found to perform the best on the validation set. Accuracy on the validation set is 78.5%.
C.1.4 ODIN
Architecture. The architecture of the VGG-16 classifier used for Odin is described in C.I.I.
Hyperparameter Tuning. The temperature scaling parameter T and perturbation magnitude e are set based on grid scarch. T is scarched for in [1, 2, 5, 10, 20, 50, 100, 200, 500, 1000]. is scarched for in a lincarly spaced interval [0, 0.004] using steps of 0.0002. The perturbation magnilude leading to highest AUROC on the validation set is 0, T set to 1000.
C.1.5 Autoencoder
Architecture. The encoder and decoder are from Abati al. (2019).
Training Details.The parameters of the autoencoder optimized by minimizing the mean squared error (MSE) between the input data and its reconstruction using the Adam optimizer (Kingma & Ba, 2014). The "Minecraft item bar' is removed by cropping 22 pixel rows at the bottom of the images. For the patch-based autoencoder, patches of size 32 x 32 pixels are sampled randomly from the cropped images and normalized between and 1. For the full-image-baser autoencoder, the lowermost 22 pixel rows are padded and the resulting images normalized between 0 and Gaussian noise with standard deviation of 40 added to the input data. Mean reconstruction error on the training set is 0.165.
Hyperparameter Tuning. We trained autoencoders with latent representation dimension 50, 100 and 200 and batch sizes 32, 64 and 128. The dimension of the latent representation of the results given is 100 and the batch size is set to 128. The autoencoder is trained for 8000 epochs with a learning rate of 10-3, all other paramcters of the optimizer are set to the PyTorch default values (0 weight decay and running average coefficients 31.2 = (0.9, 0.999)). Mean reconstruction error on the validation set 0.245.
C.1.6 One-Class Support Vector Machine
Architecture one-class support vector machine is implemented based the OneClassSVM class of the SciKit-Learn software package (Pedregosa al., 2011). A radial basis function (RBF) kernel is used.
Training Details. The Clo-Class SVM is trained on fcature vectors extracted by the fine tuned VGG-16 classifier with its classification head removed (see C.1.1 for archilecture and training details). Each feature vector has dimension 4096 and is normalized between 0 and
Hyperparameter Tuning. We performed a grid search for nu and gamma of the Ono-Clas SVM. A suitable value for nu, which represents an upper bound on the fraction of training errors and a lower bound the fraction of support vectors, is searched in [0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9]. The inverse of the length scale of the RBF kernel gammaa is searched in 9 values logarithmically spaced from 10-5 to 1. The chosen values performing best in terms of AUROC on the validation set are nu = 0.8, gamma = 10 for NoveICraft and gamma = 0.00177827941 for NovelCraft+t. The default values the SciKit-Learn software package are chosen for all other parameters. The One-Class SVM fit is computed on all normal images for NovelCraft. For the NovelCraft+ experiments, the fit is computed on a subset of 10000 images of the normal training data.
C.2 Symbolic Novelty Detection
A Arciteettue. As described in the main paper, the predictive models for symbolic novelty detection consist. of multi-layer perceptrons. We use one-hot encoding for categorical variables.
JSON Feature Extraction. To represent the game state as vector, we extract 22 numerical variables which correspond to the inventory count. for each of the possible objects that the agent interact with in
22

---

Published in Transactions on Machine Learning Research (03/2023)
the standard environment. Additionally extract categorical variables corresponding to the object the agent is currently holding and the object in-front of the agent.
Training Details. We train the predictive models using the Adam optimizer with a learning rate of 10-4 for 50 epochs with early stopping. We use default parameters weight decay and $1,2 = (0.9, 0.999) in the optimizer. For numerical features, the loss function optimized is MSE whereas for categorical features we optimize Categorical Cross-Entropy.
Hyperparameter Tuning. For predictive MLP models fine-tune the number of hidden layers, and single value for all hidden dimensions. We perform grid search over 1 to 3 hidden layers with step size with hidden dimensions varying between 16 to 64 with step size 16. We perform model selection based on the validation predictive loss. For all three settings of the context size, the final hyperparameters are 2 hidden layers with hidden dimension of 64. We also use a default dropout rate of 0.2 for all models and ReLU activations in all layers except the last.
Anomaly Scores. To calculate anomaly scores for single time-step we take the maximum over the predictive loss of all variables in the vector.
C.3 Generalized Category Discovery
Architecture. Following Vaze et al. (2022), we use a ViT-B-16 (Dosovitskiy et al., 2020) vision transformer pretrained with DINO (Caron et al., 2021) self-supervision on unlabeled ImageNet then fine-tuned on NovelCraft.
Training Details. Fine-Luning done with the GCD costrastive loss with supervised loss weight 0.35. Training consists of 200 epochs of SGD optimization with a cosine annealed learning rate ending at the initial learning rate times 0.01.
Hyperparameter Tuning. Supervised loss weights of 0 and 1 were tested to verify that the combination of losses was increasing performance. Initial learning rates 1, 0.1, 0.01, and 0.001 were tested with 0.1 found to best minimize training loss, reaching 1.21 mean loss.
D Further Comparisons to Related Work
D.1 Further Discussion and Analysis of Related Datasets.
Surveillance-focused datasets. Work on anomaly detection in complex scenes has often been motivated by surveillance applications (Ramachandra et al., 2022), such as the UCSD Anomaly Detection dataset (Li et al., 2014). The yearly PETS challenges (Patino et al., 2017) have offered open datasets with anomalies such as abandoned baggage. Unlike these fixed camera datasets, our dataset uses a dynamic, egocentric camera and enjoys fewer ethical concerns than surveillance.
Comparison to scene-focused datasets. In Tab. D.1 (next page), we provide a comprehensive comparison of available datasets focused on scene-focused visual tasks that seem especially relevant for povelty/anomaly detection. We thank anonymous reviewers for bringing some of these to our attention.
To recap the major takeaway messages from this table, we suggest that our NovelCraft is the only one containing egocentric images from a goal-oriented agent exploring a 3D world over time. NovelCraft supports benchmark tasks for both detection and category discovery with 40+ classes, while UBnormal, Fishyscapes, and RoadAnomaly datasets offer only a limited number of object classes to be discovered (<10). Finally, NovelCraft. offers mullimodal dala, not just visual data.
23

---

Datasct
Motivation
Built
Agent pursu- ing goal?
Im- ages over time?
Multi- modal?
Future potential for vision-
Num object classes for deeection/(lisoovee
Num images (train/val/tee) val//est)
Published in Transactions on Machine Learning Research (03 (203)
for
novelty
informed actions
anomaly?
NovelCraft.
Agent building pogostick Minecraft-like 3D open world
YES
YES
YES
YES
YES
51 inserted items + gameplay items
7037 / 1205 4420
> >130k extra
NovelCraft++
Places
Scene recognition: photos covering nearly all possible scenes on Earth
No
No
No
No
No
Object/itooe labels unavailable
> 1.8 million
[url] Zhou et al. (2018)
COCO
Object detection and instance segmentation in complex scenes from Flickr
No
No
No
YES captions
No
91 objects
40504 40775 images
82783
[url] Lin al, (2014)
Example analysis:
(Rambhatla et al., 2021) object types
UBnormal
Anomaly detection in artificial videos of complex scenes
YES
No
YES
No
No
116087 / 28175 /
[url] Acsintoae et al.
92640 images from 543 videos
(2022)
Fishyscapes
Semantic segmentation of anomaly in urban driving scenes
YES
No
No
No
Not easily; would need
10 object types
2975 / 30 1000
[url] Blum
(2021)
driving
simulator No
RoadAnomaly '19 [url] Lis et al. (2019)
Semantic segmentation of anomalies in street scenes
YES
No
No
No
60 images
RomdAnomaly '21
Semantic segmentation of real road hazards
YES
No
No
No
No
26 object types
100 images
[url] Chan et al. (2021)
RoadObstacle '21
Semantic segmentation of real hazards road
YES
No
No
No
No
31 object types
327 images
[url] Chan al. (2021)
Table D.1: Landscape of available datasets where images focus complex scenes, not individual objects.

---

Published in Transactions on Machine Learning Research (03/2023)
D.2 Performance Comparion Across Datasets for Visual Novelty Detection
See Table D.2 for visual novelty detection results on other datasets. Our benchmark more challenging than many existing visual novelty detection evaluations, with only the experiments by Cheng & Vasconcelos (2021) on CUB-200-2010 and Abati et al. (2019) on CIFARIO getting similar AUROC metrics. We also note that varying model backbones, evaluation methodologies, datasets, and the use of only AUROC as metric hinders model comparison. We hope to enable future comparisons between models by providing a dataset with defined visual novelty evaluation methodology, evaluations of a variety of methods with the same backbone, and more metrics for detailed model comparisons.
Method
Backbone Model VGG-16
In/Out Distribution Datasct. NovelCraft Stanford Dogs
AUROC
NDCC
80.6
NDCC
VGG-16
92.3
Cheng & Vasconcelos
GounderTType-211
94.0
CUB-200-2010
77.5
Caltech-250
89.5
ODIN
VGG-16
NovelCraft
65.5
ODIN
Dense-BC
CIFAR100/Tiny TinyImageeeee crop
94.5
Liang et al.
CIFAR100/ Twn.:/00///////ameNeeeee resize
85.5
CIFAR100/ISSUN crop
96.6
CIFARI00/LSUN resize
86.0
Deep Ensemble
VGG-16
NovelCraft.
58.9
Deep Ensemble
WideResNet 28x10
CIFARI0/SVHN
96.0
Berger et al.
CheXpert
70.4
OCSVM
VGG-16 VGG-F
NovelCraft
57.7
OCSVM
MNIST
76.6
Andrews al.
X-RAY
98.7
OCSVM
PCA- CCA-Whitening
MNIST
95.1
Abati et al.
CIFAR10
58.6
Autoencoder
LSA Denoising
NovelCraft
57.5
Auloencoder
LSA Denoising
MNIST
94.2
Abati et al.
CIFAR10
59.0
LSA Variational
MNIST
96.9
CIFAR10
58.6
Table D.2: Visual novelty detection using the tested methods on other datasets. distribution dataset describes the training dataset and out of distribution dataset describes the dataset used for novel examples. Subsets of the dataset are used when only a single dataset is listed. Evaluation procedures vary so directly comparing results without further context may be misleading, Refer to the cited papers for more details their specific evaluation.
E
Further Analysis of NovelCraft+:: Extra Train Set of Standard Images
E.1 Supplementary Data Description
NovelCraft+ consists of supplemental training data for NovelCraft. NovelCraft++ is available through the dataset. website or direct download (link redacted). It contains 125,636 additional frames of data from the standard environment. NovelCraft+ is most useful for experiments needing severe class imbalance or that only want to use additional data from the standard environment for training.
E.2 Results for Visual Novelty Detection on NovelCraft+
Fig. E.2 and Tab. E.1 compare the visual novelty detection methods trained on NovelCraftwith and without the addition of NovelCraft+. The One-Class SVM fit. was computed on a subset of 10,000 images for NovelCraft+ due to the method scaling poorly with dataset size. The patch- bascd autoencoder, the One Class SVM and Odin gain in novelty detection performance when trained with NovelCraft+ terms of both AUROC and AUPRC. For the deep ensemble novelty detection, AUROC decreased when trained on Nove while the AUPRC increases. NDCC trained with NovelCraft performs slightly worse in both
25

---

Published in Transactions on Machine Learning Research (03/2023)
at TPR 95%
at PPV 80%
Method
AUROC
AUPRC
TNR
PPV
TNR
TPR
NDCC
80.58
49.82
47.30
37.56
99.78
2.75
NDCC+
74.52
47.60
9.32
25.89
99.89
2.55
ODIN
65.54
30.35
40.56
34.77
100.00
0.14
ODIN+
70.45
38.34
0.00
25.00
100.00
0.11
Deep Ensemble
58.89
41.36
0.00
25.00
99.55
5.86
Deep Ensemble-t
55.41
62.08
0.00
25.00
99.89
2.86
One-Class SVM
57.67
27.81
0.00
25.00
100.00
0.00
One-Class SVM+
58.37
29.82
9.66
25.96
100.00
0.00
Autoencoder (patch)
57.53
38.90
9.66
25.97
99.55
5.41
Autoencoder++ (patch)
68.87
43.6
22.02
28.88
99.78
2.72
Table E.1: Visual novelty detection performance metrics (percentages, higher better), evaluated per-frame on test images. For the given methods denoted with NovelCraft+ was used for training. While One-Class SVM and autoencoder perform better in terms of AUROC, AUPRC, TNR and precision in the 95% TPR regime, NDCC performs slightly worse in all given metrics except for its TNR in the 80% precision regime. Deep Ensemble increases in AUPRC and TNR, but decreases in AUROC and TPR.
AUROC and AUPRC metrics despite class-balanced sampling during training and longer training times due increased epoch size. We hypothesize that the decreased performance results from class-balanced sampling insufficiently addressing the severe class imbalance or significantly larger batch sizes being necessary to effectively learn from the larger dataset.
Future work with NovelCraft-+ could examine methods to address the severe class imbalance or explore one-class novelty detection methods.
26

---

Published in Transactions on Machine Learning Research (03/2023)
HATKC 490000
18ca1
(a) ROC Curves NDCC
(b) PRC NDCC
16.
CON 13
022
0UN SANES 5S S0N1
0.5
14
04
0.1
(c) ROC Curves Odin
(d) PRC Odin
CR-
10023
(e) ROC Curves Deep Ensemble
(f) PRC Deep Ensemble
0-2
Dee-lia SUM STAPNI INBICC OlA Ricoootive
(g) ROC Curves One- Class SVM
(hh PRC One- One-Class SVM
10
Phich HUPE 39095)
62-
64
04
(i) ROC Curves Autoencoder (patch)
(i) PRC Autoencoder (patch)
Figure E.2: Novelty detection tradeoff curves (higher is better) on NovelCraft test data when trained with and without Nove
27


Reviews:
[{"review_id": "Ju1aEFIif4", "reviewer": "Reviewer_GBwG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The paper targets a useful dataset niche, where there are non-trivial modifications performed in a dataset, but where there are still consistent behaviors and an expectation of transfer. Keeping it to a simulated virtual environment is a limitation but they address the drawback themselves early and effectively.\n   1. However, I found it slightly too difficult to understand the scope and details of the modifications performed. In effect, Figure A.1 is necessary to understand what happened, as well as Table A.2, but these are both in the Appendix.\n   2. There are not enough details about the “original” pogo stick construction task, and how complex it is. Obviously one would like to see videos, which hopefully a website can remedy, but the paper sometimes brushes over information as if they ought to be clear to the reader.\n2. The number of modifications is small, and as it stands it is hard to know if one should expect them to be easily learnable.\n   1. 2 gameplay modifications are performed, but there is no indication of their exact effect on behaviors with statistics or measurements.\n   2. 52 objects are introduced, but again it is hard to tell if these should easily be discriminable, or if they share too many visual features to pose issues for humans to recognize (for example). There is no indication of their effect on the agent’s behavior.\n   3. The dataset is small, judging by Table A.2. We’re talking around 50-100 frames for most objects. This is slightly confusing, given it should be trivial to run the data collection for longer and increase these numbers dramatically?\n   4. There are no human baselines on the proposed benchmarks, which make it hard to assess their difficulty and validity.\n3. The section on “Filtering to identify non-standard images” needs more unpacking.\n   1. Why did you have to use Blender to assess object visibility?\n   2. Don’t you have access to ground-truth information and the game engine state to basically assess this yourself? If not, then I am not convinced that PolyCraft was the best choice?\n4. Sections 4, 5 and 6 cover quite a lot of ground on assessing existing models on several tasks built upon the dataset.\n   1. The analysis performed is good, going beyond just single metric accuracy with the ROC & PR curves.\n   2. Unfortunately, as mentioned above, I feel like the balance is a bit off, and it is unclear if the paper ought to be primarily about the dataset or this model assessment. I would personally have spent more time on the dataset (explaining decisions, showcasing behaviors, statistics, etc).\n   3. Importantly, the results of most assessed methods are very poor on the Test sets (judging from Figure 2, also true for Table 3 and Figure 3 to an extent).\n      1. This is actually quite interesting if true, given this demonstrates that the benchmark is hard and that current methods fail on it.\n   4. However, there is no information about the performance on the Train and Validation sets, so right now this is confounded and it might just be that no model learns to classify the “classes” appropriately enough.\n      1. If performance is also poor on train/valid, then this means the modifications aren’t discriminable enough and diminish the interest of the dataset quite drastically. Again, without human baselines it is hard to assess the usefulness of the dataset.\n   5. There is no discussion of how the “normal” classes were selected, and why one should expect them to form a valid novelty detection task.\n   6. I had to check in the Appendix to see that you were fine-tuning the VGG model on your tasks, I feel like this should be flagged in the main text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new dataset, based on a Minecraft-like engine, to specifically assess the novelty detection abilities of vision models. They collect trajectories (images and groundtruth symbolic game state) of an agent solving a construction task. They then modify the environment to include game modifications (fences around trees or different tree types) and novel objects (52 total), and collect the result as their novelty detection dataset (splitting some of these modifications and novel objects into train/validation/test sets).\nFinally using their data, they assess existing models on their novelty detection performance (using either visual information only or combinations of visual and symbolic) and generalized category discovery abilities.\n\nOverall, the dataset covers an interesting niche for researchers interested in models’ abilities to handle novel objects in simulated environments, but it might be too limited in size and scope to reach its full potential (especially compared to MineDOJO and MineRL). I also feel that the model assessment part of the paper was slightly overemphasized, and I would have preferred the paper to spend more time discussing the dataset and their proposed benchmarks in more detail.\n\nGiven the “TMLR two questions” for submissions, I would answer yes to the 2nd one but will need more details to be assured on the 1st one.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide all Train and Validation curves/final performance for the Visual Novelty Detection, Multimodal Novelty Detection and Generalized Category Discovery tasks.\n2. Clarify why the dataset couldn’t be made larger.\n3. Clarify the “Filtering to identify non-standard images” section in 3.2.\n4. Clarify on the choice of “normal” classes and if this is expected to affect results if they were modified.\n5. [if shared by other reviewers] rebalance the paper to cover the dataset in more details in the main text", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GEiaxpPvvN", "reviewer": "Reviewer_q2q8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is very well-written and easy to follow (some small exceptions; see the next section)\n- The proposed benchmark and related design choices are for the most part well-motivated (some exceptions; see the next section)\n- Interesting analyses of metrics and how relative rankings of methods can differ based on application-specific priorities, and of the effect of class imbalance on performance for the category discovery task. \n\nWeaknesses:\n- Some methodological choices seem inconsistent and should be better motivated\n- The GCD results include only one model (and an ablation of that model). Would be nice to have a more thorough empirical investigation there too.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents NovelCraft: a benchmark for studying novelty detection and discovery. Unlike previous benchmarks, the images considered here are frames taken from episodes of a trained artificial agent solving a task in minecraft, and are consequently substantially different from images in e.g. cifar-10, that features an object at the center of the image. In addition, this benchmark has a multi-modal nature, where aside from visual information, they have symbolic information too, in the form of JSON strings containing information about the state of the world. Finally, its sequential nature is interesting, which is exploited by the symbolic models but not the vision models currently, allowing for interesting future work. The authors contribute a study of existing methods for novelty detection and discovery in different settings (they consider visual-only novelty detection, symbolic-only and multi-modal approaches, as well as the related problem of generalized category discovery) and in particular discuss the nuances specific to the metrics, most notably that simply looking at AUROC and AUPRC doesn’t surface interesting trade-offs between models at particular points that might matter most for some downstream applications (e.g. ensuring at least 95% precision).\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Regarding the relationship to previous scene-focused datasets, it would be good to also discuss what are the limitations of NovelCraft compared to previous ones? In which scenario(s) (e.g problem setting, motivation or modeling approach) should someone pick a previous one over NovelCraft?\n\n- Give a precise definition of ‘episode’\n\n- In Section 3.2, the authors state the number of episodes collected for ‘standard episodes’, ‘inserted object modifications’ and ‘novelcraft+ extra standard episodes’ but not for ‘gameplay altering modifications’. Would be useful to know this information. Especially since this category has the highest variance in terms of the number of images per episode, perhaps more episodes are needed?\n\n- Is there a benefit to ever using the term positive predictive value (PPV) instead of just using precision? If I understand correctly the two are exactly synonyms, so that would help readability. \n\n- For the symbolic and multi-modal novelty detection task, why are only standard episodes available for validation, as opposed to ones with novelty? Section 3 says that some novel classes are held-out for validation - but why isn’t this the case in this specific task? Please clarify and if possible unify the methodology used throughout the paper.\n\n- Similarly, it wasn’t clear to me why there is a discrepancy between visual novelty detection and its symbolic and multimodal counterpart in terms of one being frame-based and the other episode-based. Is there some fundamental reason that this has to be the case? I couldn’t tell from reading the paper. Again, please clarify and unify if possible.\n\n- For the multimodal approach, I was surprised that the Autoencoder was chosen as the vision base detector. As the authors noted, it’s not the highest-ranked out of the vision baselines from the previous section. Is requiring more than one training class a no-go in this scenario? It wasn’t clear to me why. It also isn’t clear to me why e.g. NDCC in principle requires more than one training class? Couldn’t we estimate the single training class with a (single) Gaussian and use that Mahalanobis distance to get the novelty score?\n\n- In Section 6.2, describe the GCD contrastive loss mentioned. \n\n- For Section 6.2, consider evaluating additional models or baselines aside from GCD SSKM and DINO SSKM which is an ablation of the former, if I understood correctly\n\n- It would also strengthen the paper to experiment with using pretrained models for the rest of the paper’s experiments, aside from just GCD\n\n- Minor: unlabeled vs unlabelled - Both are fine but should be consistent. I’ve seen both versions in the paper.\n\n\nAdditional discussion points\n=====================\n\n- Is it perhaps too optimistic to give as input the full state of the world (in the symbolic information), including items that the agent wouldn’t observe? Why is this design decision made? Is it hard to narrow down only the aspects that are observable by the agent? This way the visual and symbolic information would be about the same ‘parts’ of the world. Maybe that can be a different ‘level of difficulty’. Though I do see from the empirical results that there is certainly room to improve and the task is by no means ‘solved’.\n\n- Can the dataset be ‘biased’ depending on the agent chosen for navigation to generate the episodes/image frames (i.e. different agents would generate different datasets with different relative difficulty) - or is this not a concern when we are specifically interested in novelty detection problems?\n", "broader_impact_concerns": "I don't have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pGFMcURCBo", "reviewer": "Reviewer_bgxd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have a number of comments, though in the interest of transparency I should point out to the authors that I am not an expert on novelty detection, and largely unfamiliar with the literature. Hopefully this information will help in addressing and understanding (the focus of) my concerns:\n\n* The motivation for this type of this dataset is largely lacking. In several places it is claimed that \"existing evaluations are too\nobject-focused\", and yet it is not made clear why this is a problem? Similarly, compared to datasets in the related work on \"Vision for open worlds\", it is stated that \"Compared to these works, our dataset is distinct in its multimodality and how images are gathered over time by a navigating agent.\", yet it is not clear why this is an important distinction. The same holds true for related work on \"Industrial datasets\", where it is stated that \"Unlike these works, our target scenario is an agent navigating open worlds over time, not a fixed camera on an assembly line\" without further clarification. I suspect that the authors will want to argue that their dataset, being open-world, egocentric, and scene-centric, better resemble how humans perceive the world. And yet, the proposed datasets consists of artificial renderings of a video game, compared to real-world visual scenes, which is a step back. Is this not an important distinction also?\n\n* Relatedly, no comparison is provided to other available datasets to highlight why NovelCraft is preferential, eg. in terms of the insights that can be derived. For example, Figure 2 reports novelty detection trade-off curves for several baselines on NovelCraft are shown. However, each of these baselines can also be evaluated on what are described as object-focused novelty detection tasks. Thus it is important to compare how these methods compare on these different datasets and point out differences. Perhaps the proposed NovelCraft is more difficult overall, or perhaps a different ordering among the considered baseline is observed when using NovelCraft? In other words, one should provide concrete evidence (or good arguments) why the community should start evaluating on NovelCraft as opposed to datasets that are currently available. I should note that the visual category discovery experiments provide a bit of evidence in this direction already, but it is missing for novelty detection from video frames, which is what I consider to be the \"main track\".\n\n* What makes a dataset \"open-world\" and what not was quite unclear to me. Arguably, the agent that is used for recording the videos in NovelCraft is following reasonably fixed script, similarly to how video recordings of a self-driving car (eg. Waymo Open dataset) tend to follow a clear path. Thus aren't both instances of \"open world\" datasets in this case? How does \"open-worldness\" relate to the task of novelty detection, which is what this dataset is designed for?\n\n* Relatedly, It is written that \"We specifically selected this Minecraft-like world because it has several key attributes that are helpful toward our long-term goal.\", after which long-range planning over time, interaction with other agents, and combinatorial action spaces are listed as such attributes. Why exactly are these attributes important for novelty-detection? What insights can be obtained because of NovelCraft having these properties compared to prior datasets?\n\n* The multi-modal aspect of NovelCraft, i.e. the symbolic counter-part to video frames in the form of json files, appears somewhat adhoc to me. If the motivation of NovelCraft to more closely resemble how humans perceive and act in the visual world, there where does this form of data come in? A secondary modality in the form of written dialogue between agents or auditory information would have made more sense. Is it likely that progress on video-frames + json (even when converted to time-series) using NovelCraft will entail progress on multimodal novelty detection in human-like settings?\n\n* \"Examination of metrics informed by task-specific costs\" is listed as a core contribution of this work, yet in practice this appears to amount to simply reporting results thresholded at 95% TPR; and at 95% PPV. Thus, I would not consider this a core contribution of the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contribution of this paper is the NovelCraft dataset, an open-world dataset composed of egocentric videos of an artificial agent building a pogo stick in a MineCraft-like environment. In some episodes, the world state has been modified such that novel types of objects  can be seen, and hence the primary goal of the dataset is to facilitate novelty detection. In addition to pixel-based video frames, the content of each frame can also accessed in the form of a json file, which acts as a second modality. Finally, the dataset can also be used for category discovery.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "My comments all focus on adjustments that would strengthen the work substantially in my view.\n\nCritical to securing my recommendation are my comments about the lack of comparison to other datasets in terms of baseline performance. From the provided benchmark data it is not clear to me whether this dataset is useful and interesting to the novelty detection community, which is critical when proposing a new benchmark. Less critical are my comments on the symbolic aspect of the proposed dataset.\n\nIn general, I think that if the main benchmark of novelty detection / category discovery from video frames should be the focus on this contribution, and I would encourage the authors to strengthen this part substantially.\n\n", "broader_impact_concerns": "Broader impact section is provided.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposed Novelcraft dataset presents a new scene-based benchmark for novelty detection that differs fairly substantially from object-centric benchmarks. One interesting highlight is that the relative performance of state-of-the-art approaches changes in this domain, and simple auto-encoders end up being quite robust. The multi-modal aspect has potential, but I'm not sure whether a JSON file of the state is necessarily the way to go as it's unclear if a method built on this could be easily ported to other kinds of modalities.

Regardless, after providing new experiments and further clarifications, the reviewers mostly believe that the claims are sufficiently supported and that there would be interest from some members of the community. One reviewer was still ambivalent about the motivation - how the agent-centric view of this dataset specifically interacts with the novelty detection task. Some further discussion of this in the final draft would be helpful.
Decision: Accept (poster)