Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large Language Models
Year: 2023

Abstract:
In many task settings, text classification models are likely to encounter examples from novel classes on which they cannot predict correctly. Selective prediction, in which models abstain on low-confidence examples, provides a possible solution, but existing models are often overly confident on unseen classes. To remedy this overconfidence, we introduce Contrastive Novelty-Augmented Learning (CoNAL), a twostep method that generates OOD examples representative of novel classes, then trains to decrease confidence on them. First, we generate OOD examples by prompting a large language model twice: we prompt it to enumerate relevant novel classes, then generate examples from each novel class matching the task format. Second, we train a classifier with a novel contrastive objective that encourages lower confidence on generated OOD examples than training examples. When trained with CoNAL, classifiers improve in their ability to detect and abstain on novel class examples over prior methods by an average of 2.3% in terms of accuracy under the accuracy-coverage curve (AUAC) and 5.5% AUROC across 4 NLP datasets, with no cost to in-distribution accuracy.1

heading: 1 Introduction
text: Recent progress in NLP has led to text classification models that are accurate not only in-distribution (ID), but also on some out-ofdistribution (OOD) data (Arora et al., 2021). Nonetheless, some categories of real-world distribution shift still pose serious challenges. For instance, in open-set label shift (Garg et al., 2022), the test data includes examples from novel classes not present in the training data, making it impossible for a standard classifier to predict correctly (Scheirer et al., 2013). Moreover, novel class examples can be difficult to detect with conventional OOD detection methods, as they typically bear
1Code is available at github.com/albertkx/CoNAL.
a strong surface resemblance to training examples (Ţifrea et al., 2021). In this paper, we frame open-set label shift as a selective prediction problem (El-Yaniv and Wiener, 2010; Geifman and ElYaniv, 2017) that we call open-set selective classification (OSSC). OSSC requires text classifiers to predict correctly on closed-set examples while abstaining on novel class examples.
To perform well on OSSC, a classifier must have lower confidence on novel class examples than closed-set examples by learning features which differentiate novel classes from closed-set classes (Perera et al., 2020). In order to supervise this representation learning, it is useful to identify what examples from novel classes might look like. Prior work has explored automatically generating OOD images by adding random perturbations to ID examples (Setlur et al., 2022). Text inputs, however, are composed of discrete tokens, and modifying even a single token can unpredictably alter the meaning of a sentence. We seek an automatic generation method that addresses these limitations, leveraging the generative ability of large language models (LLMs) like GPT-3 (Brown et al., 2020). LLMs are a desirable source for novelty, as their generation is informed by a broad corpus of examples seen during pretraining, allowing them to reliably generate from classes outside a dataset.
We present Contrastive Novelty-Augmented Learning (CoNAL), a method to improve the OSSC ability of a classifier by automatically generating OOD examples, then training to abstain on them. To generate a diverse set of OOD examples that anticipate different potential test-time shifts, we introduce Novelty Prompting, a method that augments a source dataset with novel class examples generated by a LLM. We first perform label generation, prompting our LLM to extend the closed-set labels with novel labels. We then prompt the LLM to generate new examples conditioned on each novel label to form a large set of probable novel examples.
11778
Finally, we propose a contrastive confidence loss (CCL) for training, which encourages both high accuracy on the ID training set and lower relative confidence on the generated novel examples. We show that CCL outperforms stricter losses like Outlier Exposure (Hendrycks et al., 2019), which can adversely affect ID accuracy. Our full pipeline is shown in Figure 1. Our method can be viewed as a form of “partial” knowledge distillation: we leverage an LLM “teacher model” to improve novelty detection performance without altering the student model’s ID classification ability.
We evaluate CoNAL against state-of-the-art OOD detection baselines across 14 splits of 4 datasets—AGNews (Zhang et al., 2015), TREC10 (Li and Roth, 2002), TACRED (Zhang et al., 2017), and Emotion (Saravia et al., 2018)—finding that it improves both OOD detection and OSSC, by an average of 5.5% AUROC and 2.3% in terms of area under the accuracy-coverage curve (AUAC) over the best prior method. These improvements come at no cost to ID accuracy, demonstrating that it is possible to distill novelty detection alone without affecting predictive power. Finally, we analyze the settings in which CoNAL can improve OSSC performance. In the data dimension, scale is often optional: with as few as 1000 generated examples, our method outperforms vanilla training on all 4 datasets. LLM size has a larger effect on performance: on some datasets only a sufficiently large model can generate useful examples.

heading: 2 Problem Setting

heading: 2.1 Open-Set Selective Classification
text: In standard classification, an optimal model f should predict the ground-truth label y of an in-
put example x from a closed set of known labels YID. However, under a more realistic open-set setting, some test examples are drawn from unknown novel classes YOOD. Without a priori knowledge of YOOD, a standard discriminative classifier will never correctly classify a novel example. Instead, an optimal open-set selective classifier f should predict y when y ∈ YID, and abstain otherwise.
For a probabilistic model pθ(y | x) and associated confidence metric, the prediction is given by f(x) = (ŷ, c), where ŷ = argmaxy∈YID pθ(y | x) and c denotes the model’s confidence. When used as a selective classifier with threshold γ, f predicts ŷ when c > γ and abstains otherwise (Geifman and El-Yaniv, 2017). This differs from OOD detection (Hendrycks and Gimpel, 2017) in that f must abstain on both novel examples and its own errors and must attain high ID accuracy.

heading: 2.2 Evaluation Protocol
text: We holistically measure selective classification performance with the area under the accuracycoverage curve (AUAC). The accuracy-coverage curve plots accuracy as a function of the fraction of examples on which the model predicts (i.e., coverage) as the confidence threshold γ varies. For accuracy computation, we treat predictions on all novel class examples as incorrect. AUAC measures the combined ability of a model in ID classification accuracy, ID calibration, and OOD detection.
Though we deviate from prior work and report AUAC, to demonstrate that CoNAL is still effective at OOD detection, we also compute the Area under the ROC (AUROC). AUROC measures a model’s ability to detect when a test example x is of a novel class (y ∈ YOOD). Higher is better: 50% AUROC
is random, and 100% is perfect.

heading: 3 Method: CoNAL
text: Here we describe Contrastive Novelty-Augmented Learning, a method for automatically improving OSSC. At a high level, we generate novel examples and then train our model to be less confident on generated novel examples than closed-set examples. We first describe desiderata for useful novelty, then introduce a two-phased novel example generation method, Novelty Prompting, and finally introduce a contrastive confidence loss for classifier training. We illustrate the method in Figure 1.

heading: 3.1 Novelty Prompting
text: Desiderata of Novelty Generation Inspired by previous work which utilize known, representative OOD data to train selective prediction and OOD detection models (Kamath et al., 2020; Hendrycks et al., 2019), we focus on creating an generated “novel set” that is representative of potential label shifts at test time. The “novel set” must be (1) plausible, meaning that it should bear a surface resemblance to the training data, e.g., we should create news examples for a news dataset, and (2) semantically novel, meaning that these examples should be from new classes. In other words, an example is novel if it demonstrates a semantic shift (Arora et al., 2021), but shares non-semantic features with examples in the training set. For example, selecting data from an entirely separate dataset, as is done in Hendrycks et al. (2019), violates plausibility. Meanwhile simply editing surface features or recombining examples as is done in mixup (Zhang et al., 2018) might induce a distribution shift but would not result in semantic novelty.
To satisfy these desiderata, we propose a twostage generation method called Novelty Prompting (NP). To encourage semantic novelty, we first generate novel labels given a dataset’s extant labels. We then show existing examples to a language model (to encourage plausibility) and ask it to generate a new example conditioned on one of the new labels. Figure 1 shows both prompt formats.
Label Generation. Though prompting with large autoregressive language models (LLMs) like GPT3 has typically been explored in the context of few and zero-shot learning to perform standard NLP tasks (Brown et al., 2020), we find that LLMs are also capable of “expanding” a set of topically related concepts that might realistically co-occur via
sequence continuation. We leverage this capability to generate novel labels. We prompt the largest GPT-3 model available (Davinci) with a task-specific instruction and the concatenation of the normalized known (YID) labels.2 Taking the union over continuations of one or more novel labels N times, we obtain a diverse “novel label set.” We combine multiple completions because in preliminary experiments, we observed that single completions tend to overgenerate labels from a narrow subcategory of classes. To remedy concerns about data leakage due to dataset examples of the true unknown class possibly appearing in LLM pretraining, we remove instances of the gold novel label(s) from this set. In practice, predicting the true novel test-time labels is both permissible and desirable, so our experimental setup likely underestimates our method’s performance.
Finally, we filter out generated labels that are closely related to ID labels. For example, if joy appears in the ID labels, we remove synonyms like happiness. We use a large online thesaurus3 to remove synonyms from the final novel label set. We analyze the impact of filtering in Appendix A.10.
Example Generation. To generate each novel example, we randomly sample a novel label from our set and prompt a LLM (we use GPT-J4) to generate an example of that label. We prime this model with one random sampled label-example pair from each ID class in the training dataset in the prompt, resulting in 3-6 in-context examples, varying based on the dataset. Providing these context pairs ensures that our generation is plausible: the model is encouraged to generate a specific style of text. We perform this generation procedure repeatedly to form a novel example set. We show the prompt we use for this step in Appendix A.3, and several generated label-example pairs in Figure 2.

heading: 3.2 Contrastive Confidence Loss Training
text: Our second contribution is an improved loss function for training models to have lower confidence on OOD examples than ID examples. Prior work have used the Outlier Exposure (OE; Hendrycks
2Though this requires some human intervention, it both (1) satisfies the true zero-shot nature of test-time label shift as it requires no knowledge of the unknown labels and (2) requires minimal effort, typically only involving converting an abbreviation label such as LOC into Location.
3https://moby-thesaurus.org/ 4We evaluate GPT-3 for label generation but not example
generation, as the latter would require many more API calls.
et al., 2019) objective, which encourages the model f to output a uniform probability distribution over closed-set classes when given a novel example x. OE can be successfully applied to train models on OOD data gathered from a different dataset (e.g., Wikitext), as there is very little risk of this data overlapping with ID data. In contrast, we automatically generate plausible novel examples, which runs the risk that some novel examples will be indistribution. Since OE encourages models to have the lowest possible confidence on novel examples, it can hurt predictive accuracy when some examples x resemble closed-set examples. Instead, we seek a solution which treats outliers flexibly.
We propose a novel contrastive confidence loss (CCL) that encourages models to be less confident on OOD examples than ID examples. This is a less strict objective as models can achieve minimum loss without predicting a perfectly uniform distribution for the generated novel examples. For an input x, let pθ(y | x) be the model’s predicted distribution over YID. Let cθ(x) = maxy∈YID pθ(y | x), the Maximum Softmax Probability (MaxProb; Hendrycks and Gimpel, 2017), which we use as our confidence metric. Finally, let ℓ denote the cross-entropy loss with a one-hot target vector, and DID and DOOD denote the training set and novel set respectively. We define CCL as follows:
L(θ) = E(xid,yid)∼DID [ℓ(pθ(y | xid), yid)]+ λExid∼DID,xood∼DOOD [max(0, cθ(xood)− cθ(xid))].
That is, we penalize the confidence of novel examples which have higher confidence than any closed-set example. While this still induces our model to learn lower confidence on novel examples, it simultaneously permits our model to learn that some novel examples should have lower confidence than others, rather than learn minimal confidence on all members of the generated novel set. In practice, we obtain an unbiased estimate of the second term by sampling a batch of n ID and n OOD examples at each step and computing the second term pairwise between each of the n2 ID-OOD example pairs. We arbitrarily choose λ = 1.0, weighting the two terms of the objective equally.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: We construct artificial dataset splits from 4 popular NLP classification datasets by holding out one or
more labels from training and moving all examples of that label to the test split, removing classes that are too small to yield statistical significance in our evaluations. Specifically, we use a question intent detection dataset, TREC-10 (Li and Roth, 2002) and construct 5 splits. We also use two popular topic classification datasets, AGNews (Zhang et al., 2015), a news classification dataset, and Emotion (Saravia et al., 2018), a tweet classification dataset. We construct 4 splits for each. Finally, we use TACRED (Zhang et al., 2017), a strongly classimbalanced sentence relation-classification dataset with 41 possible relations. We construct a single split where we hold out the 35 smallest classes. Appendix A.8 contains further dataset details. Results for each dataset are averaged across all splits.

heading: 4.2 Experimental Details
text: For Novelty Prompting, we perform label generation using the best available GPT-3 model, GPT-3 Davinci (Brown et al., 2020) and example generation with a smaller GPT-J 6B model (Komatsuzaki, 2021). For the novel set, we perform 5 label generation iterations, then generate 100,000 examples (after filtering). We train BERT-base classifiers with CCL for 5000 steps and batch size n = 40. On TACRED, we permit only generations containing exactly two entities, one a subject and the other an object, filtering out roughly 90% of generations, as this is a hard constraint for relation extraction. We detail datasets in Appendices A.8 and A.9.

heading: 4.3 Baselines
text: We evaluate our method against baselines from prior work, CCL baselines with other novel sets, and Outlier Exposure (Hendrycks et al., 2019). Though two methods (kFolden and Constrative) can address arbitrary distribution shifts, we evaluate them here only on the open-set shift setting. For all methods, we train a BERT-base model and use hyperparameters from the original papers unless otherwise specified. Of the baselines, only CCL and Outlier Exposure use explicit novel sets. Vanilla. We evaluate vanilla cross-entropy loss training, calculating confidence using MaxProb. kFolden. We evaluate kFolden (Li et al., 2021), a method that trains an ensemble of k individual classifiers, each trained on k−1 labels. The average of the ensemble probability distributions is used for confidence computation. Contrastive. We evaluate Contrastive OOD Detection (Zhou et al., 2021), which uses a contrastive
objective to induce training examples of different classes to be distant and of the same class to be near. This sparsifies the embedding space, ensuring that most OOD examples are far from feature representations of ID samples. We use the supervised constrastive loss and the Mahalanobis distance metric for confidence computation, finding that this setup performed the best on our evaluation. CCL + Zero/Few-Shot Data Augmentation. To measure the impact of explicitly prompting for novel labels, we generate with an identical pretrained GPT-J model, but prompt with only an instruction and one (or zero) ID training example from each class (See Appendix A.3 for the specific prompt format). Essentially, we perform example generation identically but skip label generation entirely. We perform CCL training and MaxProb inference. While some resultant generations will be useful, we expect that many will not be semantically novel, resulting in strictly worse performance. CCL + Wikitext. To measure whether plausibility of examples impacts their usefulness for CCL, we use an entirely different dataset, Wikitext-103, as our novel set. Though these examples represent a distribution shift, they do not accurately reflect the open-set shift the classifier will encounter. Outlier Exposure + Novelty Prompting. We pair our novel set with Outlier Exposure (OE; Hendrycks et al., 2019) as described in Section 3.2 and compute confidence with MaxProb.

heading: 5 Results

heading: 5.1 OSSC Results
text: CoNAL outperforms prior work. We report comparisons of CoNAL against baselines in Table 1. Broadly, we find that while baselines like kFolden and Contrastive training struggle to consistently outperform vanilla training (e.g., on TACRED), CoNAL improves selective classification over vanilla across all datasets. We outperform the best prior method (Contrastive) by 2.3% AUAC, and on three of four datasets, our method significantly outperforms all prior methods. Furthermore, we outperform kFolden by 3.6% AUAC despite its ensemble totaling many times the capacity of our single classifier. CoNAL also results in zero or little accuracy drop (less than 0.2 points) for all datasets. In Appendix A.4, we show full ID accuracy results for all datasets.
Other choices of novel set for CCL training can still be beneficial. Prompting with only a task-relevant instruction (zero-shot) generates sufficiently useful novel examples to slightly outperform the vanilla baseline by 1.5% AUAC. Using Wikitext as our novel set performs roughly on par with zero-shot generation: though Wikitext examples are less noisy than generations, they also tend to be less dataset-relevant. Few-shot generation, which generates more plausible examples, is outperforms all prior methods, but performs worse than Novelty Prompting on 3 of 4 datasets.
To further test the importance of novel set selection, we compare with two oracle methods. In the Gold Data setting, we use CCL with held out data of the gold novel test class(es) as a strict upper bound for both label and example generation. In the Gold Label setting, we eliminate the label generation step, performing example generation using the gold label alone. This setting is overly optimistic as we cannot know what new labels will appear at test-time.5 CCL in the Gold Label setting slightly outperforms CoNAL, but using gold novel data can achieve much stronger OSSC.
Training loss choice matters for generated data. Although OE training with Novelty Prompting data improves OOD detection over vanilla, it sharply decreases accuracy on TREC-10 (96.6% → 71.3%) and on average by 0.6% on the other three datasets (see Appendix A.4). In contrast, we find that CCL training maintains accuracy on all settings (see Appendix A.4), as it does not enforce a uniform probability distribution on all novel set examples. CCL with both zero- and few-shot generation outperforms all prior methods, and our full CoNAL method significantly outperforms prior methods on all but one dataset. OE exhibits this issue only with generated data: when the novel set is instead sampled from held-out gold OOD data OE outperforms CCL in AUAC and AUROC, suffering only a small accuracy drop (an average of 0.4%).
We attribute this behavior to generation noise: some generated examples are similar to ID examples, and thus greatly affect the model’s ID predictions when training with OE. To verify this hypothesis, we conduct an experiment where we train classifiers with synthetic novel sets formed by noising heldout OOD data with various amounts of heldout
5In practice, expert knowledge of the novelty we expect to see at test-time is sometimes available, and as shown in our results, can be leveraged for better performance.
ID data. In Figure 3, we show that as the simulated ID noise ratio increases, OE training hurts accuracy whereas CCL models retain accuracy.
Smaller datasets suffer more. The ID accuracy drop is most salient on TREC-10 because it is by far the smallest dataset we consider, making it easy for generation noise to overwhelm signal from the train set. We conduct two experiments to show that TREC-10 is not unique, but instead exemplifies an inherent pitfall of OE. First, to test whether OE noise sensitivity applies on other datasets, we consider a smaller training set from another dataset. In the first experiment of Appendix A.12, we subsample AGNews training sets to smaller sizes. As the training set becomes smaller, the ID accuracy gap between OE and Vanilla training increases to more than 35%. Meanwhile the ID accuracy gap between CCL and Vanilla is less than 10% even at small training set sizes. Our finding here is that TREC-10 is not unique — OE can suffer from gen-
eration noise on other datasets when the training set size is not large enough.
Second, we show that this ID accuracy drop increases as the novel set size grows, i.e., the larger the novel set, the more noise the classifier sees in OE training and the worse its ID predictive ability. The second experiment in Appendix A.12 shows that when the TREC-10 novel set (100K examples) is much larger than the training set (2.8K examples), accuracy decreases drastically. We generally find a negative correlation between the novel set size and ID accuracy with OE training. In contrast, CCL maintains ID accuracy at all novel set sizes.
CCL improves ID-OOD separability. In Appendix A.13, we show that CCL is effective at OOD detection because it improves the confidencebased separability of ID and OOD examples.

heading: 5.2 OOD Detection Results
text: To confirm that CoNAL improves a classifier’s ability to disambiguate novel class examples, we compare CoNAL against the same baselines on OOD detection in Table 2. We find similar improvements, outperforming the best prior method (kFolden) by 5.5% AUROC. We interpret this result in Appendix A.13, showing that CoNAL improves ID/OOD separability. Unlike other datasets, TACRED exhibits strong OOD overconfidence: all baselines except kFolden yield worse-than-random OOD detection (below 50% AUROC). We hypothesize that this could be due to models incorrectly assuming that an NER tag pair seen at training time in only a single class could not belong to a novel relation. OOD detection on TACRED remains a challenging goal for future work, as the strong performance of CCL training with gold heldout data indicates significant remaining headroom. In fact, on all three other datasets, models achieve greater
than 90% AUROC when trained with gold heldout data. While OE results in better AUROC performance on AGNews, ID accuracy also decreases.

heading: 5.3 Performance Analysis
text: Label Generator Model We investigate whether a smaller, open-source model can suffice as the label generator. Specifically, we replace the label generator with GPT-J and use 100 label generation iterations. We find that GPT-J performs on-par with GPT-3 on 3 of 4 datasets in all metrics, except on AGNews where it performs within 1 point AUAC. We provide full details in Appendix A.5.
Example Generator Size. Since model scale often affects prompting performance (Sanh et al., 2021; Brown et al., 2020), we compare generator models ranging in size from 125M parameters to 6B parameters. For each, we generate 100K examples, and compare CoNAL results in Figure 4. All generators improve over the Vanilla baseline.
GPT2-Neo 125M is competitive with GPT2-Large despite being roughly 5x smaller, suggesting that its larger pretraining corpus (the Pile) aids generation ability. Novel generation is easier on simpler tasks: on Emotion, where labels (or synonyms) can appear directly in the example, inter-generator differences are small. We posit that even larger generators such as GPT-3 could yield better performance on abstract tasks. In Appendix A.7, we analyze the quality of generated examples.
Other Classifier Models. We investigate the generalizability of CoNAL to two other classifier architectures, RoBERTa (Liu et al., 2019) and DeBERTav3 (He et al., 2021), of both base and large sizes, with results in Figure 5. Averaged over datasets, CoNAL improves AUAC for all classifiers, though these improvements are most apparent with the smaller base models. Larger classifiers are better at OSSC: vanilla RoBERTa-large improves over BERT-base by 2.8% AUAC. Vanilla RoBERTabase slightly outperforms vanilla BERT-base, but after CoNAL training, the two perform on-par, suggesting that learning from generated examples can make up for BERT’s smaller pretraining corpus.
Generation Quota. Since large-scale LLM prompting is costly, we analyze the performance tradeoff of shrinking the generation quota, the number of novel examples that we can generate. In Figure 6, we show that on some datasets, using orders of magnitude smaller novel sets can still improve selective prediction. For example, 1000 generations is sufficient to improve AUAC across all datasets, and for most datasets we require far fewer. In cases where a low quota is sufficient, CoNAL is nearly as efficient as vanilla training.
Generation Analysis. To evaluate the remaining errors in OOD generation, we perform two types of manual analysis on Novelty Prompting (NP). First, we categorize the labels generated by NP after filtering, finding that 70%+ of GPT-3 generated labels are novel on all datasets except TREC-10, where only 40% are novel, and the vast majority of the others are valid closed-set labels. This highlights one source of generation noise in our pipeline. Second, we categorize the examples generated by NP and a strong baseline method, Few-shot data augmentation (FS). Specifically, for each of the 4 splits of AGNews, we annotate 100 NP and 100 FS examples. On average, 41% of NP generations come from novel classes, compared to only 26% of FS
generations, explaining CoNAL’s stronger performance over CCL + Few-Shot. We provide further analysis in Appendix A.7. Our method performs well despite the high fraction (50.5%) of closed-set examples generated in NP, showing that CCL is robust to noise in the example generation process.

heading: 6 Related Work

heading: 6.1 Identifying OOD Data
text: OOD Detection. Prior work on OOD detection uses models to detect test examples that come from a new distribution (Hendrycks and Gimpel, 2017). Many of these introduce new training objectives, e.g., with a contrastive objective (Winkens et al., 2020; Sehwag et al., 2021; Zhou et al., 2021). When the nature of the distribution shift is known, the model can directly be trained to be uncertain on known OOD examples (Dhamija et al., 2018; Hendrycks et al., 2019). We draw on the success of these known-shift methods, but eliminate the need for known OOD data by using generative models.
Other works on OOD detection have explored alternative modeling paradigms. Ensembles of neural networks can yield useful confidence estimates (Ţifrea et al., 2021; Li et al., 2021; Lakshminarayanan et al., 2017), as can simple methods like deep nearest-neighbors (Sun et al., 2022; Bergman et al., 2020). Further performance improvements can be achieved by modifying the confidence metric. Podolskiy et al. (2021) find that Mahalanobis distance better exploits the geometry of the learned embedding space, explaining strong performance achieved by replacing probability-based scoring mechanisms (Lee et al., 2018; Ren et al., 2021). We show that standard models are sufficient: MaxProb scoring with a standard classifier can perform well when given proper OOD demonstrations.
OOD Selective Prediction. Selective prediction work focuses on a different paradigm altogether, fusing abstention (detection) with prediction (ElYaniv and Wiener, 2010; Geifman and El-Yaniv, 2017). External calibrators popularized by Kamath et al. (2020) have become popular as a selective prediction framework (Zhang et al., 2021; Ye and Durrett, 2021; Varshney et al., 2022). However, calibrators are typically smaller than classifier models (Tajwar et al., 2021); we instead update the higher-capacity classifier model to better leverage of our large set of generated outliers.

heading: 6.2 Open-Set Classification
text: Open-set classification is well-explored in the image classification space, as tasks like CIFAR-100 tend towards large label spaces (Scheirer et al., 2013; Geng et al., 2021). Some methods for detecting open-set examples build on the classifier, e.g., by classifying over the model’s activations (Bendale and Boult, 2016) or adding an additional reconstruction model (Oza and Patel, 2019). Our work is most closely related to methods that generate near-OOD examples and regularize confidence on them (Ge et al., 2017; Du et al., 2022; Kong et al., 2020; Vernekar et al., 2019; Möller et al., 2021; Setlur et al., 2022). However, methods like perturbation and embedding space sampling align poorly with the discrete nature of text, prompting us to investigate powerful generative language models. Esmaeilpour et al. (2022) is closely related to our work in that they also generate novel labels, but directly use these labels as input to a classifier.
Open-set classification for text has been less explored. Early works built upon the k-way, 1-vsrest paradigm of SVMs, classifying an example as “novel” if all k scores fall below a threshold (Fei and Liu, 2016; Shu et al., 2017; Doan and Kalita, 2017). Some works explore similar methods as prior vision work, but focus on the intent detection setting, as task-oriented dialogue models should abstain on unknown intents (Zeng et al., 2021; Zheng et al., 2020; Lin and Xu, 2019). To the best of our knowledge, we are the first work to generate novel examples for open-set text classification.

heading: 6.3 Data Augmentation
text: Finally, our generation method, Novelty Prompting, relates to prior work in using pretrained language models for data augmentation. Kumar et al. (2021) proposes directly conditioning on class labels to
generate relevant class examples, which forms a component of our prompting approach. AnabyTavor et al. (2020) finetunes a class-conditional generator on a given dataset to yield more relevant generations, though we consider prompting instead of finetuning as a method to prime for relevance.

heading: 7 Discussion and Future Work
text: In this work, we introduce CoNAL, a method for generating novel examples which simulate openset shift and training to abstain on them. Through extensive experiments, we demonstrate that by presenting generated examples to a classifier, we can significantly improve its ability to abstain on examples from novel classes against state-of-the-art baselines. Our work provides a generalizable framework for improving OSSC and OOD detection: in fact, we show through CCL training’s strong performance with gold data that there remains headroom for novel example generation. Additionally, CoNAL is modular, as it provides additional supervision signal but does not alter the classifier’s architecture. It thus remains extensible with other training objectives or classification metrics. Finally, automatically diagnosing dataset issues and improving them is an important step towards making NLP safer and easier to apply. CoNAL allows practitioners to deal with noise introduced by LLM-generated data and apply these generated datasets in settings like open-set selective classification. The success of our method indicates that LLMs can be used to improve datasets with minimal human intervention. Given interest in the emergent capabilities of LLMs, we hope that future work on classification in the presence of distribution shifts can better leverage large language models to both directly identify shifts and improve the abstention ability of smaller classifiers.

heading: Limitations
text: Despite the fact that we demonstrate strong OSSC performance with low generation quotas in Appendix 5.3, CoNAL still is slightly more computationally expensive than vanilla training. It also requires access to a pretrained LLM with which to generate novel examples. To achieve optimal performance, usage of the OpenAI API is required, which poses some concerns around transparency, as details around GPT-3 training and data are not publicly released. Finally, performance varies across datasets, suggesting that types of outliers that are unexpected to LLMs might still confuse a CoNALtrained model.

heading: Acknowledgments
text: We would like to thank Wang Zhu, Yuchen Lin, Muhao Chen, Wenxuan Zhou, and members of the Allegro and INK labs at USC for their valuable feedback. We also thank Johnny Wei for discussions on statistical testing.

heading: A Appendix

heading: A.1 Computational Budget
text: As we did not closely track the total amount of computational resources used, we provide our best estimate. All experiments were completed on 12GB 1080Ti and 48GB RTX8000 GPUs. We did not perform any hyperparameter search.
We note that computation time for CoNAL is divided into two components, example generation and classifier training. We provide two examples of the computational budget for generation. When using GPT-3 as a label generator, label generation costs several cents using the text-davinci-002 endpoint, and example generation with GPT-J 6B takes several hours on a 48GB RTX8000 GPU. We show in Appendix 5.3 that we can generate many fewer examples and still achieve strong performance, in which case generation would require orders of magnitude less time.
Classifier training for a bert-base-cased model takes approximately 30 minutes on a 12GB 1080 Ti GPU. For reference, vanilla training takes about half this time, as CoNAL must compute losses for a pair of batches at each training step.

heading: A.2 Code Release
text: Code for replicating all experiments is released on Github at github.com/albertkx/CoNAL under an MIT license.

heading: A.3 Prompt Format
text: We use the same format for label generation for all datasets, shown in Figure 7, but customize the instruction for each dataset, as shown in Figure 8.
For example generation, we prompt with an example sampled from each class and a random novel label. We use the same instruction for all datasets. An example prompt is shown in Figure 9.
Few-shot prompting is done with a task-specific instruction, but does not include labels, as shown in Figure 10. Zero-shot prompting is done with the task-specific instruction only.

heading: A.4 Full Accuracy Results
text: CoNAL improves AUAC on all datasets without any cost to ID accuracy, as shown in Figure 11. We show full ID accuracy results in Table 3. CCL training maintains accuracy across all datasets, while OE training decreases accuracy on 3 of 4 datasets, with a very sharp drop on TREC-10. In Appendix A.12, we show thorough analyses on
two datasets that this steep accuracy drop is not an anomaly: when paired with generated data, OE training is sensitive to the sizes of the novel set and training set, and can signficantly hurt ID accuracy when the novel set is much larger than the training set. Additionally, despite improving selective prediction performance, training with gold held-out data curiously hurts accuracy on TACRED.
A.5 CoNAL performs well without GPT-3
In our main experiments, we use GPT-3 as the label generator and GPT-J 6B as the example generator. In Section 5.3, we show that smaller models can be used as example generators. Here we investigate whether a smaller, open-source language model can be used as a label generator. In Table 12, we show that GPT-J 6B also performs well at label generation. We empirically observe that GPT-J generates shorter and noisier completions, requiring us to increase the number of model calls from 5 to 100 and filter out all labels containing punctuation marks. After applying these tweaks, we find that the difference between GPT-J and GPT-3 label generation in AUAC is small on 3 of 4 datasets, and differs by only 0.7 on AGNews, suggesting that CoNAL with GPT-J only can still work well.

heading: A.6 Generation Examples
text: We show examples of the generations from Novelty Prompting for AGNews in Table 4. Recall that we do not allow the gold novel label to be generated to hedge against data leakage from LLM pretraining. However, we observe that our generator is still capable of producing relevant examples to the gold novel label due to signal from similar novel labels. Despite many generations not being directly relevant to the gold novel label, we observe that the generated novel labels are sufficiently distinct from
the closed-set labels that most generated examples still provide useful “novelty” supervision signal to the classifier.

heading: A.7 Novelty Prompting Error Analysis
text: Though CoNAL improves OSSC ability on all datasets, we still find headroom between Novelty Prompting generated data and gold OOD data (92.3 → 94.8) in Table 1. To understand the remaining failure modes of Novelty Prompting, we manually inspect the generated labels and examples from our method. Broadly, we seek to attribute “generation noise,” or the frequency with which the purported novel sets which we generate instead contain closed-set class examples.
First, we manually annotate GPT-3 generated labels from all dataset splits, categorizing a label into “implausible” if it does conform to the dataset’s format, “closed-set” (ID) if it is synonymous with a class seen in training, and “novel” (OOD) if it describes a class distinct from all closed-set classes. In Figure 13, we perform this analysis for all four datasets. Across all datasets, less than 15% of generations are implausible, suggesting that the model is usually able to generate reasonable additional labels given only 3-6 ID classes. We also observe that while on 3 of 4 datasets less than 15% of generated classes are closed-set, on TREC-10 more than half of generated labels are closed-set. One reason for this label generation noise is that the TREC-10 labels are very broad (e.g., “entity” describes questions about any subcategory of an entity, including all objects and events), so while a generated label might differ in definition, it could still overlap with or fall into a subcategory of a closed-set class.
Second, we manually annotate GPT-J generated examples to understand whether example generation is a source of generation noise. In Figure 14,
we annotate 100 examples of each split of AGNews for both Few-shot data augmentation and Novelty Prompting. We observe that Novelty Prompting generates novel class examples more frequently across 3 of 4 splits. Both methods generate implausible (e.g., agrammatical, non-news) examples rarely, as ID demonstrations sufficiently prime the model to generate text in the style of news. Additionally, under Novelty Prompting, we find that the fraction of novel class examples (41.3%) is much lower than the fraction of novel labels generated (81.7%), suggesting that GPT-J can easily adhere to the dataset format, but struggles to extrapolate to the novel label. Future work should thus focus on better specifying the example generation step to leverage the generated labels.

heading: A.8 Dataset Split Details
text: TREC-10: We remove the Abbreviation class as it is too small to yield statistically significant metrics in our task setting, leaving 5 remaining classes.
Emotion (Saravia et al., 2018): We remove two small classes, love and surprise, leaving 4 remaining classes.
TACRED (Zhang et al., 2017): We process the data for training following Joshi et al. (2019). This dataset is particularly challenging due to its classimbalanced nature. We evaluate a single split where we keep the 6 largest classes as ID data, and hold out the other 35. This is the largest class, and thus results in approximately 80% of examples being OOD at test time.

heading: A.9 TACRED Processing Details
text: We perform label normalization, removing underscores and prefixes, e.g., converting per : employee_of into employee of. This both helps the label generator model understand our label space and generate more relevant novel labels and ensures that generated novel labels are well-formatted for downstream example generation.
For examples, we normalize the Subject and Object token tags into a standard English equivalent containing the subject or object indicator and the NER tag, e.g., [subject : person]. To ensure that generated examples satisfy the task format, we filter out examples that do not contain exactly one subject and one object (many generations contain partial or malformed indicator/NER spans). Finally, we denormalize tags back into original model input tokens.

heading: A.10 Label Filtering
text: After label generation, we perform synonym filtering to reduce occurrences of ID synonyms. We find this step to have a large impact on datasets for which labels are common English words which appear in our thesaurus, and less where label names are more abstract. For example, for Emotion and TREC-10 , where dataset names are words such as “fear” or “human,” filtering removes 21% and 20% of generated labels respectively. Meanwhile on both AGNews and TACRED, label filtering removes only 2% of labels. In the case of AGNews, news genre overlaps are not easily captured by syn-
onyms, and even after normalization, many TACRED labels such as “employee of” do not appear in our thesaurus.
A.11 Label Smoothing performs poorly. We evaluate label smoothing (LS) (Müller et al., 2019) as an additional baseline for improving OSSC, which mirrors vanilla training but alters the one-hot target vector to a “smoother” version, incentivizing uncertainty. Label smoothing has been shown to be effective in domain shift detection (Kong et al., 2020). We use label smoothing factor α = 0.1 and calculate confidence with MaxProb. In Table 5, we show that label smoothing performs poorly in our setting. While it does not affect classifiers’ ID accuracy, it significantly decreases AUROC on all but one dataset (TREC-10), where it still remains worse than CoNAL and all of our data generation baselines.
A.12 Outlier Exposure is sensitive to generated data
In the setting where Outlier Exposure is originally evaluated, access to some known OOD data (e.g., Wikitext) is assumed (Hendrycks et al., 2019). However in our setting, where we generate a potential novel set, there is no guarantee that the generated examples are indeed OOD. For example, we show in Appendix A.7 that less than 50% of NP generations for AGNews come from novel classes. Without this guarantee, more generated data is not always better when training with OE. One risk of using more generated novel data is that the model will see a large number of ID examples in the novel
set relative to in the training set. We conduct two experiments to analyze the impact of novel set size relative to training set size.
First, we vary the novel set size relative to the training set size. In Figure 15, we train with novel sets on TREC-10 from size 0 to 100K using both OE and CCL. We observe that training with OE hurts accuracy and AUROC when the novel set is larger than 100 examples, whereas CCL continues to improve as the novel set size grows, and maintains accuracy for all novel set sizes. As the novel set becomes larger than the size of the training set (to the right of the dashed line), both OOD detection AUROC and ID accuracy quickly decrease. This result suggests as the ID noise the classifier
sees in OE training outsizes the training set, its ID predictive ability worsens.
Of the datasets in our experiments, TREC-10 is by far the smallest, with only about 2800 training examples per split. To determine whether OE is also sensitive to the size of the ID set, we subsample the AGNews dataset into smaller training sets and perform OE and CCL training with 100K-sized novel sets. We compare the results against Vanilla training with the same ID sets in Figure 16. Although reducing the training set size decreases the ID accuracy even for vanilla training, CCL training achieves similar accuracy for all subsampling sizes. We do observe that a sub-10% accuracy margin appears between vanilla and CCL at extremely small
training set sizes, though this margin disappears at 1000 or more training examples. OE, meanwhile, decreases ID accuracy by as much as 35% when the dataset is subsampled to 30 examples, and 25%+ at 300 examples. OE-trained classifiers are also worse OOD detectors given limited training data: they underperform vanilla classifiers for all training sets smaller than 3000 examples. Finally, we find that OE does yield better OOD detectors than CCL for sufficiently large AGNews training sets. This expands on our findings in Table 2, suggesting that when there is access to a large amount of training data, in this case 10000 examples are more, OE can learn from noisy novel sets (though ID accuracy still decreases). Our results indicate that TREC-10 is not alone: As training set size becomes smaller, the ID classes becomes less well-specified, and ID examples present in the novel set induce the model to make incorrect predictions (and poor confidence estimates) on true ID test examples.

heading: A.13 CoNAL and Separability
text: To understand why CoNAL improves AUROC, we compare the confidence profiles of a vanilla finetuned classifier against those of a CoNAL trained classifier. Specifically, in Figure 17, we select 50 random ID examples and 50 random OOD examples from each dataset split and compute MaxProb confidences. We find that CoNAL decreases confidence on OOD examples, though not to the same extent on all examples. In datasets like TREC-10 and Emotion where CoNAL achieves stronger AUROC gains, the decrease in OOD confidence is more pronounced. Though ID test examples also decrease in confidence on all dataset splits, this decrease is less pronounced and is likely due to the confidence contrastive objective term incentivizing the model’s confidence distributions to be generally less peaked.
The shifts reflected in the confidence distributions directly impact the separability of OOD and ID examples. On the Vanilla model confidence axis,
it is difficult to identify a threshold above which most examples are ID and below which most examples are OOD. Given CoNAL confidences, OOD and ID examples are more separable. This visual separability is reflected in the OOD Detection AUROC metric.
To demonstrate the strictness of the OE objective, we plot the confidences of the same examples without (Vanilla) and with OE training in Figure 18. First, we observe that the vast majority of OOD examples have similar confidence after OE training, as they are all pushed towards minimum confidence (maximum entropy). Second, we observe that OE affects the confidence of ID test examples, decreasing the confidence of some examples lower than that of OOD test examples.

heading: A.14 Measuring Data Leakage in Generation
text: In our experiments, we explicitly forbid the gold novel class from being generated, such that the LLM is disincentivized from generating gold novel examples if the dataset has been seen in pretraining. However, it remains possible that if the LLM had seen the task data in pretraining, it could replicate parts of or an entire example from the dataset in generations. Unfortunately, as we do not have access to GPT-3 pretraining data, we cannot determine whether or not this is indeed a risk. Instead, we probe whether this is a possiblity via an n-gram overlap metric comparing the similarity between our generated examples and the test set.
Specifically, we measure the average fraction of n-grams in a generation that also appear in the test set, which we interpret as the maximal frequency that the LLM could have copied test data via pretraining leakage. For comparison, we compute the same metric between the test set and heldout novel class data. In this case, examples are sampled from exactly the same distribution and thus expected to exhibit some n-gram overlap due to shared background features. We use this value as a baseline: generation n-gram overlap should be
similar to or lower than heldout n-gram overlap. We find in Table 6 that the n-gram overlap of our novelty prompted generations is lower across all datasets and values of n than of the heldout set, indicating that the performance of CoNAL should not be attributed to example data leakage.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations. After section 7, on page 9
3 A2. Did you discuss any potential risks of your work? Limitations. After section 7, on page 9
3 A3. Do the abstract and introduction summarize the paper’s main claims? Page 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Code described in Appendix A.2
3 B1. Did you cite the creators of artifacts you used? Section 4.1, Section 4.2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Appendix A.2
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix A.2
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? Section 4.2, Section 5, Appendix A.1, A.9, A.10, A.11
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4.2, Appendix A.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.2, Appendix A.1, A.9, A.10, A.11
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Table 1. Table 2. Table 3. Table 5. Figure 6. Figure 12. We report the average over multiple seeds on all tables, and report standard error in subscript.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future work should thus focus on better specifying the example generation step to leverage the generated labels