A Unified Evaluation Framework for Novelty Detection and Accommodation in NLP with an Instantiation in Authorship Attribution
Year: 2023

Abstract:
State-of-the-art natural language processing models have been shown to achieve remarkable performance in ‘closed-world’ settings where all the labels in the evaluation set are known at training time. However, in real-world settings, ‘novel’ instances that do not belong to any known class are often observed. This renders the ability to deal with novelties crucial. To initiate a systematic research in this important area of ‘dealing with novelties’, we introduce NoveltyTask, a multi-stage task to evaluate a system’s performance on pipelined novelty ‘detection’ and ‘accommodation’ tasks. We provide mathematical formulation of NoveltyTask and instantiate it with the authorship attribution task that pertains to identifying the correct author of a given text. We use Amazon reviews corpus and compile a large dataset (consisting of 250k instances across 200 authors/labels) for NoveltyTask. We conduct comprehensive experiments and explore several baseline methods for the task. Our results show that the methods achieve considerably low performance making the task challenging and leaving sufficient room for improvement. Finally, we believe our work will encourage research in this underexplored area of dealing with novelties, an important step en route to developing robust systems.

heading: 1 Introduction
text: Recent advancements in Natural Language Processing (NLP) have led to the development of several pre-trained large-scale language models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2020b), and ELECTRA (Clark et al., 2020). These models have been shown to achieve remarkable performance in closed-world settings where all the labels in the evaluation set are known at training time. However, in real-world settings, this assumption is often violated as instances that do not belong to any known label (‘novel’ instances) are also
∗Equal Contribution, Contact email: hgupta35@asu.edu
observed. This renders the ability to deal with novelties crucial in order to develop robust systems for real-world applications.
The topic of novelty is getting increased attention in the broad AI research (Boult et al., 2020; Li et al., 2021b; Rambhatla et al., 2021). Also, in NLP, the ‘novelty detection’ task in which novel instances need to be identified is being explored (Ghosal et al., 2018; Ma et al., 2021); related problems such as anomaly detection (Chalapathy and Chawla, 2019), out-of-domain detection, and openset recognition (Hendrycks and Gimpel, 2017; Hendrycks et al., 2020; Ovadia et al., 2019) are also being studied. In addition to the task of ‘detection’, dealing with novelties also requires ‘accommodation’ that pertains to learning from the correctly detected novelties. Despite having practical significance, this aspect of dealing with novelties has remained underexplored. Furthermore, dealing with novelties is a crucial step in numerous other practical applications such as concept learning, continual learning, and domain adaptation.
To initiate systematic research in this area of ‘dealing with novelties’, we formulate a multi-stage task called NoveltyTask. Initially, a dataset consisting of examples of a set of labels (referred to as ‘known labels’) is provided for training and then sequential evaluation is conducted in two stages: Novelty Detection and Novelty Accommodation. Both these stages include distinct unseen evaluation instances belonging to both ‘known labels’ (labels present in the training dataset) and ‘novel labels’ (labels not present in the training dataset).
In the first evaluation stage i.e. the novelty detection stage, the system needs to either identify an instance as novel or classify it to one of the ‘K’ known labels. This is the same as the (K + 1) class classification problem (where K corresponds to the number of known labels) used in standard anomaly/OOD detection tasks. This evaluation stage is followed by a feedback phase in which the
1794
ground truth label of the novel instances (from the detection stage) that get correctly reported as novel is revealed. Essentially, in the feedback phase, the system gets some examples of the novel labels (from the evaluation instances of the detection stage) that it correctly identified as novel.
In addition to the initially provided training examples of the K known labels, the system can leverage these new examples of the novel labels for the next evaluation stage, the novelty accommodation stage. This stage also has evaluation instances from both the known and the novel labels (distinct and mutually exclusive from the detection stage); however, in this stage, the system needs to identify the true label of the evaluation instances, i.e. it’s a (K +N) class classification problem where N corresponds to the number of novel labels. We summarize this multi-stage task in Figure 1. We note that NoveltyTask is a controlled task/framework for evaluating a system’s ability to deal with novelties and not a method to improve its ability.
It is intuitive that the ability to deal with the novelties should be directly correlated with the ability to detect the novelties; our two-stage pipelined formulation of NoveltyTask allows achieving this desiderata as higher accuracy in correctly detecting the novelties will result in more feedback i.e. more examples of the novel labels that will eventually help in achieving higher performance in the accommodation stage. However, in the detection stage, the system needs to balance the trade-off between reporting instances as novel and classifying them
to the known labels. Consider a trivial system that simply flags all the evaluation instances of the detection stage as novel in order to get the maximum feedback; such a system will get the true groundtruth label (novel label) of all the novel instances present in the detection stage and will eventually perform better in the accommodation stage but it would have to sacrifice its classification accuracy in the detection stage (especially on instances of the known labels). We address several such concerns in formulating the performance metrics for NoveltyTask (Section 3).
In this work, we instantiate NoveltyTask with authorship attribution task in which each author represents a label and the task is to identify the correct author of a given unseen text. However, we note that the formulation of NoveltyTask is general and applicable to all tasks. We leverage product reviews from Amazon corpus (McAuley et al., 2015; He and McAuley, 2016) for the attribution task. We explore several baseline methods for both detection and accommodation tasks (Section 4).
In summary, our contributions are as follows: 1. We define a unified task for ‘dealing with nov-
elties’ consisting of both novelty detection and novelty accommodation. 2. We provide a controlled evaluation framework with its mathematical formulation. 3. We instantiate NoveltyTask with the Authorship Attribution task. 4. We study the performance of several baseline methods for NoveltyTask.

heading: 2 Background and Related Work
text: In this section, we first discuss the related work on novelty/OOD/anomaly detection tasks and then detail the authorship attribution task.

heading: 2.1 Novelty/OOD/Anomaly Detection
text: Novelty Detection and its related tasks such as outof-distribution detection, selective prediction, and anomaly detection have attracted a lot of research attention from both computer vision (Fort et al., 2021; Esmaeilpour et al., 2022; Sun et al., 2021a; Lu et al., 2022; Liu et al., 2020a; Perera et al., 2020; Whitehead et al., 2022) and language (Qin et al., 2020; Venkataram, 2018; Yang et al., 2022; Varshney et al., 2022b; Kamath et al., 2020; Varshney et al., 2022c) research communities. OOD detection for text classification is an active area of research in NLP. Qin et al. (2020) follow a pairwise matching paradigm and calculate the probability of a pair of samples belonging to the same class. Yang et al. (2022) investigate how to detect open classes efficiently under domain shift. Ai et al. (2022) propose a contrastive learning paradigm, a technique that brings similar samples close and pushes dissimilar samples apart in the vector representation space. Yilmaz and Toraman (2022) propose a method for detecting out-of-scope utterances utilizing the confidence score for a given utterance.

heading: 2.2 Authorship Attribution
text: Authorship attribution task (AA) pertains to identifying the correct author of a given text. AA has been studied for short texts (Aborisade and Anwar, 2018a) such as tweets as well as long texts such as court judgments (Sari et al., 2018). Traditional approaches for AA explore techniques based on n-grams, word embeddings, and stylometric features such as the use of punctuation, average word length, sentence length, and number of upper cases (Sari et al., 2018; Aborisade and Anwar, 2018b; Soler-Company and Wanner, 2017). Transformerbased models have been shown to outperform the traditional methods on this task (Fabien et al., 2020; Tyo et al., 2021; Custódio and Paraboni, 2019).

heading: 3 NoveltyTask
text: NoveltyTask is a two-stage pipelined framework to evaluate a system’s ability to deal with novelties. In this task, examples of a set of labels (referred to as known labels) are made available for initial training. The system is sequentially evaluated in two
stages: novelty detection and novelty accommodation. Both these stages consist of distinct unseen evaluation instances belonging to both ‘known’ and ‘novel’ labels. We define a label as novel if it is not one of the known labels provided for initial training and all instances belonging to the novel labels are referred to as novel instances. We summarize this multi-stage task in Figure 1. In this section, we provide a mathematical formulation of NoveltyTask, detail its performance metrics, and describe the baseline methods.

heading: 3.1 Formulation
text: 3.1.1 Initial Training (DT ) Consider a dataset DT of (x, y) pairs where x denotes the input instance and y ∈ {1, 2, ...,K} denotes the class label. We refer to this label set of K classes as ‘known labels.’ In NoveltyTask, the classification dataset DT is provided for initial training. Then, the trained system is evaluated in the novelty detection stage as described in the next subsection.
3.1.2 Novelty Detection (EvalDet) The evaluation dataset of this stage (EvalDet) consists of unseen instances of both known and novel labels, i.e., EvalDet includes instances from K∪N labels where N corresponds to the number of novel labels not seen in the initial training dataset DT . Here, the system needs to do a (K+1) class classification, i.e., for each instance, it can either output one of the K known classes or report it as novel (not belonging to any known class) by outputting the (0)th class. This is followed by the feedback phase described in 3.1.3.
3.1.3 Feedback Phase (DF ) For each instance of the EvalDet dataset, we use an indicator function ‘f ’ whose value is 1 if the instance is novel ( i.e. not from the K known labels) and 0 otherwise:
f(x) = 1[x ̸∈ {1, 2, ...,K}]
In the feedback phase, we reveal the ground truth label of those novel instances (from EvalDet) that the system correctly reports as novel, i.e., feedback results in a dataset (DF ) which is a subset of the novel instances of EvalDet where f(x) is 1 and the system’s prediction on x is the (0)th class.
DF =    ∈ EvalDet (x, y), f(x) = 1
pred(x) = (0)thclass
Essentially, DF is a dataset that consists of examples of the novel labels. The system can incorporate the feedback by leveraging DF in addition to the initial training dataset DT (refer to Section 3.4 for novelty accommodation methods) to adapt itself for the next evaluation stage, which is the novelty accommodation stage.
3.1.4 Novelty Accommodation (EvalAcc) The system incorporates the feedback and is evaluated in the novelty accommodation stage on the EvalAcc dataset. Like the detection stage dataset, EvalAcc also includes instances of both K known and N novel labels (mutually exclusive from EvalDet i.e. EvalDet∩EvalAcc = ∅). However, in this stage, the system needs to identify the true label of all the evaluation instances (including those belonging to the novel labels) i.e. the task for the system is to do a (K +N) class classification instead of a (K + 1) class classification. Here, N corresponds to the number of novel labels. Essentially, in the feedback phase, the system gets some examples of the novel labels, and it needs to leverage them along with DT to classify the evaluation instances correctly.
Note that the feedback data DF may or may not contain examples of all the N novel classes as it totally depends on the system’s ability to correctly detect novelties in the detection stage. The inability to detect instances of all the novel classes will accordingly impact the system’s performance in the accommodation stage. Next, we describe the performance metrics for both the stages.

heading: 3.2 Performance Evaluation
text: Novelty Detection: For the novelty detection stage, we use F1 score over all classes to evaluate the performance of the system. We also calculate the F1 score for the known classes (F1Known) and for the novel instances (F1Novel) to evaluate the fine-grained performances.
Let {C1, · · · , CK} be the set of known classes and C0 be the class corresponding to the novel instances, we calculate the micro F1 score using:
F1 = 2× P × R P + R ,
where P and R are precision and recall values. Similarly, the F1 scores over known classes
(F1known) and novel class (F1Novel) are computed. Note that all the above measures are threshold de-
pendent i.e. the system needs to select a confidence
threshold (based on which it classifies instances on which it fails to surpass that threshold as novel) and its performance measures depend on that. This is not a fair performance metric as its performance heavily depends on the number of novelties present in the evaluation dataset (EvalDet). To comprehensively evaluate a system, we use a thresholdindependent performance metric in which we compute these precision, recall, and F1 values for a range of reported novelties. To achieve this, we order the evaluation instances of EvalDet based on the system’s prediction score (calculated using various techniques described in the next subsection) and take the least confident instances as reported novelties (for each number in the range of reported novelties). Then, we plot a curve for these performance measures and aggregate the values (AUC) to calculate the overall performance of the method (refer to Figure 2). This evaluation methodology (similar to the OOD detection method) makes the performance measurement comprehensive and also accounts for the number of novelties present in the evaluation dataset.
Novelty Accommodation: In this stage, the task for the system is to do (K+N) class classification instead of (K + 1) class classification. The system leverages the feedback (DF ) (which is contingent on the number of reported novelties) to adapt it for the task, and its performance also depends on that. Following the methodology described for the detection stage, we evaluate the system’s performance over a range of reported novelties and hence over a range of feedback. Specifically, we find the feedback dataset DF for a range of reported novelties and for each individual feedback, we incorporate it into the system and then evaluate its prediction performance on the (K +N) classification task. Similar to the detection stage, we plot a curve (across a range of reported novelties) and calculate its area under the curve value to quantify the overall performance of novelty accommodation.

heading: 3.3 Methods for Novelty Detection
text: As described in the previous subsection, we calculate the system’s performance on a range of reported novelties. To achieve this, we order the evaluation instances of EvalDet based on the system’s prediction confidence score (calculated using various techniques described in this subsection) and take the least confident instances as reported novelties (for each number in the range of reported nov-
elties). This implies that the performance depends on the system’s method of computing this prediction score. We explore the following methods of computing this score for the evaluation instances:

heading: Maximum Softmax Probability (MaxProb):
text: Usually, the last layer of models has a softmax activation function that gives the probability distribution P (y) over all possible answer candidates Y . For the classification tasks, Y corresponds to the set of labels. Hendrycks and Gimpel (2017) introduced a simple method that uses the maximum softmax probability across all answer candidates as the confidence estimator i.e. the prediction confidence score corresponds to maxy∈Y P (y). In this method, we order the evaluation instances of EvalDet based on this confidence measure, and for each value in the range of reported novelties, we report those instances as novel on which the model is least confident. For the remaining instances, we output the label (out of K classes) having the maximum softmax probability.
Euclidean Distance (EuclidDist) : In this approach, we consider each sample as a point in Kdimensional space. For each sample, the probabilities from the K class classifier are chosen as coordinates in the space. We then calculate Euclidean distances between each sample and the entire distribution. The points furthest away from the distribution are classified as novel instances.
The Euclidean distance is given by d =√∑K i=1 (xi − xmu)2 where xmu is the distribution of all the samples.
Mahalanobis Distance (MahDist): This approach is similar to the previous approach with the only difference that Mahalanobis distance is used to compute the distance between the sample and the distribution.
The Mahalanobis distance (Ghorbani, 2019) between xi and xj is given by ∆2 = (xi − xj)⊤Σ−1(xi−xj), where Σ is a d× d covariance matrix. ∆2 is equivalent to the squared Euclidean distance between yi and yj , where y is a linearly transformed version of x.
Mean (CompMean): For each sample, the mean of K-1 classes is computed. The class with the highest probability is left out. The mean is later subtracted from 1. The resultant score for all the samples is sorted in descending order. The last Y elements are classified as Novel instances.

heading: Learning Placeholders Algorithm (Placeholder):
text: Zhou et al. (2021a) propose a Placeholder algorithm for increasing the separation between clusters of samples in different classes. It addresses the challenge of open-set recognition by increasing the distance between class clusters and shrinking the classification boundary, allowing the classifier to classify samples as novel that fall outside these clusters. It demonstrates the effectiveness of the Placeholders algorithm through experiments and comparison with other state-of-the-art open-set recognition methods.

heading: Few Shot Open set Recognition (Few Shot OSR):
text: Jeong et al. (2021) presents a method for recognizing novel classes with few examples available for each class. It uses prototypes to represent each class and a similarity function to compare new examples to these prototypes, allowing for the effective recognition of novel classes. The paper includes experiments on multiple datasets and compares the method’s performance to other state-ofthe-art few-shot open-set recognition methods.
We further detail these methods in Appendix B. We note that other OOD/anomaly detection methods can also be explored here. However, we study only a limited set of methods since the focus of this work is on formulating and exploring NoveltyTask.

heading: 3.4 Methods for Novelty Accommodation:
text: After the detection stage, the system gets feedback i.e. examples of novel labels (DF ). We explore the following methods of leveraging this feedback:
Retrain using DT and DF : DT consists of examples of known labels, and DF consists of examples of novel labels. In this approach, we train a new model (K +N) classifier by combining data instances of DT and DF .
Further Fine-tune using DF : In this method, we first train a model on DT with extra dummy labels, i.e., we train a model having more than K logits. This allows modifying the same model to learn to output the novel labels. To incorporate the feedback, the model initially trained on DT with dummy labels is further fine-tuned using DF .

heading: Further Fine-tune using DT (sampled) and DF :
text: Here, we follow the same strategy as the previous method, but instead of further fine-tuning only on DF , we further fine-tuning using both DT (downsampled) and DF . This is done to reduce catas-
trophic forgetting (Carpenter and Grossberg, 1988) of the known labels.

heading: 4 Experiments and Results

heading: 4.1 Experimental Setup
text: Configurations: We use Amazon reviews (McAuley et al., 2015; He and McAuley, 2016) for the authorship attribution task. In this task, each author corresponds to a class. We compile a dataset consisting of 250k instances across 200 authors and use it for NoveltyTask. We define experimental settings using a set of configuration parameters; for the base setting, we use the following values: • Number of Known Classes (K): 100 • Training Data DT Class Balanced: True • # Instances Per Known Label in DT : 500 • Number of Novel Classes (N): 100 • # Instances Per Class in EvalDet: 100 • # Instances Per Class in EvalAcc: 500
In the above setting, the total number of evaluation instances in EvalDet is 20k out of which 10k are novel. In this work, we also study other settings by varying the values of these parameters.
Models: We run all our experiments using the BERT-base model (Devlin et al., 2019). For classification, we add a linear layer on top of BERT representation and train the model with a standard learning rate ranging in {1−5}e−5. All experiments are done with Nvidia V100 16GB GPUs.

heading: 4.2 Results

heading: 4.2.1 Novelty Detection
text: Figure 2 shows the novelty detection performance on the base setting (EvalDet has 20k instances out of which 10k are novel) i.e. overall Precision, Recall, and F1 achieved by various methods across
the range of reported novelties on EvalDet. Specifically, each point on the curve represents the P, R, or F1 when its corresponding method reports the specified number of novelties (x-axis value) out of all instances in EvalDet.
MaxProb achieves the best overall performance: From the plots, it can be observed that MaxProb achieves the highest AUC value and hence the best overall performance. This result supports the prior finding that complex methods fail to consistently outperform the simiple MaxProb method (Varshney et al., 2022b; Azizmalayeri and Rohban, 2022).
Performance Analysis of MaxProb: To further study the performance of MaxProb in detail, we show its P, R, and F1 curves for Known, Novel, and Overall data in Figure 3. As expected, the precision on Known classes tends to increase as more novelties get reported. This is because the system predicts the known classes only for those instances on which it is most confident (highest MaxProb). Similarly, the precision on novel instances tends to decrease as more and more novelties get reported. The overall precision on the (K+1) classes tends to increase with the increase in the number of reported novelties. We provide a detailed performance analysis on the known classes, novel classes, and overall data in Appendix C.

heading: 4.2.2 Novelty Accommodation
text: Figure 4 shows the novelty accommodation performance on the base setting (EvalAcc has 100k instances uniformly split across 200 classes - 100 known and 100 novel) i.e. Overall F1 achieved by systems trained by leveraging the feedback (using different accommodation methods (a, b, and c)) resulting from different detection methods across the range of reported novelties. Note that for a value of reported novelty, each detection method results
in a different feedback dataset and hence will have a different accommodation performance. We show the Precision and Recall curves in the Appendix.
Retraining w/ DT + DF : We note that MaxProb and MahDist turned out to be the best detection methods. This implies that their corresponding feedback dataset would contain more examples of the novel labels. This further reflects in the novelty accommodation performance as incorporating the feedback of these methods results in the best overall accommodation performance using the retraining method.
Catastrophic Forgetting Increases in further fine-tuning with DF : As previously mentioned, this method leads to catastrophic forgetting of the known classes resulting in low overall F1 performance. We demonstrate this trend in Figure 5. Furthermore, with the increase in the number of novelties reported, the extent of catastrophic forgetting also increases.

heading: Further fine-tuning with Sampled DT and DF
text: improves performance: This method not only mitigates catastrophic forgetting but also results in a slight improvement over the retraining method. For sampling, we use the maximum number of correctly detected instances of a class in DF as the
threshold for sampling instances of known labels from DT . Furthermore, this method is more training efficient than the retraining method as the number of training instances is significantly lower in this method and yet it achieves better performance.

heading: 4.3 Analysis
text: Distribution of Instances over classes in the Feedback dataset: We show the distribution of instances over all the classes (novel) in the feedback dataset DF when the number of reported novelties is 10k in Figure 6. It can be observed from the his-
togram that for all the novel classes, novel instances between 55 and 95 are correctly detected. For majority of the classes, 76-85 instances are detected. This further shows that the detection method is not biased towards or against any set of novel classes in identifying novel instances.
Trend of class level performance in the accommodation stage vs the number of instances in the feedback dataset: In Figure 7, we show a scatter plot of accommodation F1 performance achieved by each class vs the number of its instances in the feedback dataset. The plot is for the MaxProb detection method when 10k novelties are detected and retrain with DT , and DF accommodation method is used. From the trend, it can be inferred that with the increase in the number of instances, the performance generally tends to increase.
Comparing Performance of Known and Novel Classes in the Accommodation Stage: In Figure 8, we compare the performance of the system (in
the accommodation stage) on Known and Novel classes. It clearly shows that the system finds it challenging to adapt itself to the novel classes. This can be partly attributed to the availability of limited number of training examples of novel classes. This also provides opportunities for developing better accommodation techniques that can overcome this limitation.

heading: 4.4 Other Configuration Settings
text: In this work, we also study NoveltyTask for different settings (different configuration parameters defined in 4.1). We observe findings and trends similar to the base setting. We provide detailed results and discussion in the Appendix.

heading: 5 Conclusion and Discussion
text: To initiate a systematic research in the important yet underexplored area of ‘dealing with novelties’, we introduce NoveltyTask, a multi-stage task to evaluate a system’s performance on pipelined novelty ‘detection’ and ‘accommodation’ tasks. We provided mathematical formulation of NoveltyTask and instantiated it with the authorship attribution task. To this end, we also compiled a large dataset (consisting of 250k instances across 200 authors/labels) from Amazon reviews corpus. We conducted comprehensive experiments and explored several baseline methods for both detection and accommodation tasks.
Looking forward, we believe that our work opens up several avenues for interesting research avenues in this space, such as improving performance of detecting the novel instances and leveraging the feedback in a way that helps the system adapt with just a few examples of the novel labels.

heading: Limitations
text: Though the formulation of the task allows exploring several different settings (by varying the configuration parameters), in this work, we investigated only the label-balanced setting. Exploring the labelimbalanced setting is another very interesting research direction, and we leave that for future work. Another limitation was the limited exploration of novelty detection methods, as a number of methods have been proposed in the recent times. However, we study only a limited set of methods since the focus of this work is on formulating and exploring NoveltyTask. Lastly, we note that NoveltyTask is a controlled task/framework for evaluating a system’s ability to deal with novelties and not a method to improve its ability.

heading: Acknowledgement
text: We thank the anonymous reviewers for their insightful feedback. This research was supported by DARPA SAIL-ON program.

heading: Appendix

heading: A Other Related Work

heading: A.1 Novelty/OOD/Anomaly Detection
text: A.1.1 Vision Novelty/OOD/Anomaly detection is an active area of research in computer vision (Fort et al., 2021; Esmaeilpour et al., 2022; Sun et al., 2021a; Lu et al., 2022; Liu et al., 2020a; Sun et al., 2020; Perera et al., 2020). Datasets such as CIFAR 10 and 100 (Krizhevsky, 2009) are typically used to evaluate the efficacy of various detection methods.
Fort et al. (2021) demonstrated that pre-training of transformer-based models using large datasets is fairly robust in detecting near-OOD instances using few examples. Esmaeilpour et al. (2022) proposed to detect OOD instances using pairwise similarity score. They generate synthetic unseen examples and use their closed-set classifier to compute pairwise similarity. Wei et al. (2021) use open-set samples with dynamic, noisy labels and assign random labels to open-set examples, and use them for developing a system for OOD detection. Sun et al. (2021b) analyze activation functions of the penultimate layer of pretrained models and rectify the activations to an upper limit for OOD detection.

heading: A.1.2 Language
text: Zhou et al. (2021b) propose to add an additional classifier in addition to a closed domain classifier for getting a class-specific threshold of known and unknown classes. They generate data placeholders to mimic open set categories. Venkataram (2018) use an ensemble-based approach and replace the softmax layer with an OpenMAX layer. The hypothesis is that the closest (most similar) class to any known class is an unknown one. This allows the classifier to be trained, enforcing the most probable class to be the ground truth class and the runner-up class to be the background class for all source data.
Zhou et al. (2021c) employ a contrastive learning framework for unsupervised OOD detection, which is composed of a contrastive loss and an OOD scoring function. The contrastive loss increases the discrepancy of the representations of instances from different classes in the task, while the OOD scoring function maps the representations of instances to OOD detection scores.
Xu et al. (2019) propose Learning to Accept Classes (L2AC) method based on meta-learning and does not require re-training the model when new classes are added. L2AC works by maintaining a set of seen classes and comparing new data points to the nearest example from each seen class.
Detection approaches are also used in selective prediction (Varshney et al., 2022b; Kamath et al., 2020; Xin et al., 2021; Varshney and Baral, 2023) and cascading techniques (Varshney and Baral, 2022; Varshney et al., 2022a; Li et al., 2021a) where under-confident predictions are detected to avoid incorrect predictions.

heading: A.2 Authorship Attribution
text: BERT (and its different variants like BertAA, RoBERTa) based, Siamese-based, and ensemblebased approaches have been used for authorship attribution. Tyo et al. (2021) propose an approach that uses a pretrained BERT model in a siamese configuration for audio-visual classification. They experiment with using triplet loss, contrastive loss, and a modified version of contrastive loss and compare the results. Bagdon (2021) combine the results of a n-gram-based logistic regression classifier with a transformer model based on RoBERTa (Liu et al., 2020b) via a SVM meta-classifier. Altakrori et al. (2021) explore a new evaluation setting topic confusion task. The topic distribution is controlled
by making it dependent on the author, switching the topic-author pairs between training and testing. This setup allows for measuring the degree to which certain features are influenced by the topic, as opposed to the author’s identity. Other works include (Barlas and Stamatatos, 2020; Fabien et al., 2020; Wang and Iwaihara, 2021). N-grams, word embeddings, and other stylometric features have been used as input feature vectors for the task (Caballero et al., 2021; Boenninghoff et al., 2019; Li et al., 2022; Lagutina et al., 2021, 2020; Lagutina and Lagutina, 2021).

heading: B Novelty Detection Algorithms
text: Learning placeholders: The Placeholders algorithm consists of two main components: "Learning Classifier Placeholders" and "Learning Data Placeholders". "Learning Classifier Placeholders" involves adding a set of weights called classifier placeholders to the linear classifier layer at the end of the network. This modified classifier function denoted as f(x) = [WT (x), wT (x)], where w represents the weights of the additional k+1 class, is trained using a modified loss function that encourages the classifier to predict the k+1 class as the second most likely class for every sample. This loss function helps the classifier learn an embedding function such that the k+1 class is always the closest class to each class cluster boundary. In addition to the k+1 class, the Placeholders algorithm includes a tunable number (C) of additional classifiers to make decisions’ boundaries smoother. The final classifier function is, therefore, f(x) = [WT (x),maxk = 1, ..., CwkT (x)], meaning that the closest open-set region in the embedding space is taken into consideration. "Learning Data Placeholders" involves tightening the decision boundaries around the known-class clusters in the embedding space through a process called manifold mixup. This involves creating "unknown" class data from known class data and using an additional loss function to penalize classifying this new data as any of the known classes. Manifold mixup works by interpolating the embeddings of two samples from closed-set classes to create an embedding for a new sample, which is considered to belong to an unknown class. After training the model using both classifier and data placeholders, the Placeholders algorithm includes a final calibration step in which an additional bias is added to the open-set logits. This bias is tuned using a vali-
dation set of closed-set samples such that 95% of all closed-set samples are classified as known. The combination of these two components and final calibration allows the Placeholders algorithm to train a classifier to identify novel samples even when only trained on closed-set data.
Few Shot Open set Recognition using MetaLearning : In the paper "Few-Shot Open-Set Recognition using Meta-Learning", the authors propose a method for few-shot open-set recognition using meta-learning. The main idea is to train a meta-learner that can recognize new classes given a few examples of each class.
The meta-learner consists of a feature extractor network and a linear classifier. The feature extractor network is responsible for learning an embedding function that maps samples from different classes into a common embedding space. The goal is to learn an embedding function that clusters samples from the same class together while separating samples from different classes by a large margin.
To train the meta-learner, the authors use a metalearning loss function that encourages the embedding function to learn a "smooth" embedding space. This loss function consists of two terms: a classification loss and a separation loss.
During training, the meta-learner is presented with a small number of examples from each new class and is required to classify these examples correctly. The meta-learner is trained to optimize the meta-learning loss function, which encourages the embedding function to learn a smooth embedding space where samples from different classes are well separated.
After training, the meta-learner can be used to classify new samples by first projecting them into the embedding space using the feature extractor network, and then using the linear classifier to assign them to the appropriate class. The final classifier is able to generalize to new classes not seen during training, as it has learned to recognize the underlying structure of the embedding space.

heading: C Results
text: Hyperparameters of the model: Hidden layer dropout probability of 0.15, input Sequence length of 512 tokens, batch size of 32, and standard learning rate ranging in {1−5}e−5.

heading: C.1 Other Dataset definitions:
text: We define two other datasets by varying the following parameters:
Detection_200 setting (Dataset 2) is defined as:
• Number of Known Classes (K): 100 • Training Data DT Class Balanced: True • # Instances Per Known Label in DT : 500 • Number of Novel Classes (N): 100 • # Instances Per Class in EvalDet: 200 • # Instances Per Class in EvalAcc: 500
Detection_500 setting (Dataset 3) is defined as:
• Number of Known Classes (K): 100 • Training Data DT Class Balanced: True • # Instances Per Known Label in DT : 500 • Number of Novel Classes (N): 100 • # Instances Per Class in EvalDet: 500 • # Instances Per Class in EvalAcc: 500

heading: C.2 Novelty Accommodation Stage
text: Table 1, 2, and 3 show the results of all six novelty detection methods across all three accommodation settings on the first dataset whose results are described in details in the main paper.
Similar results are obtained for Detection_200 setting and Detection_500 setting as well. Novelty Accommodation results for Detection_200 setting are present in table 4, 5, and 6 and in table 7, 8, and 9 for Detection_500 setting.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We have Limitations Section at the end of the paper after Conclusion

heading: A2. Did you discuss any potential risks of your work?
text: Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Introduction

heading: 7 A4. Have you used AI writing assistants when working on this paper?
text: Left blank.
B 3 Did you use or create scientific artifacts? References
3 B1. Did you cite the creators of artifacts you used? We use the publicly available standard NLP datasets in this work with proper citations and references.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We use the publicly available standard NLP datasets in this work with proper citations and references.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We do not collect any data for this research and repurpose the standard publicly available NLP datasets

heading: 7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We do not collect any data for this research and repurpose the standard publicly available NLP datasets

heading: 7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? We do not collect any data for this research and repurpose the standard publicly available NLP datasets
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Exploring the labelimbalanced setting is another very interesting research direction, and we leave that for future work. Another limitation was the limited exploration of novelty detection methods, as a number of methods have been proposed in the recent times. However, we study only a limited set of methods since the focus of this work is on formulating and exploring NoveltyTask. Lastly, we note that NoveltyTask is a controlled task/framework for evaluating a system’s ability to deal with novelties and not a method to improve its ability