AD-NLP: A Benchmark for Anomaly Detection in Natural Language Processing
Authors: Matei Bejan
Andrei Manolache
Marius Popescu
Venue: EMNLP 2023 Main (2023)
Keywords: NLP
anomaly detection
deep learning
machine learning
dataset
benchmark

Abstract: Deep learning models have reignited the interest in Anomaly Detection research in recent years. Methods for Anomaly Detection in text have shown strong empirical results on ad-hoc anomaly setups that are usually made by downsampling some classes of a labeled dataset. This can lead to reproducibility issues and models that are biased toward detecting particular anomalies while failing to recognize them in more sophisticated scenarios. In the present work, we provide a unified benchmark for detecting various types of anomalies, focusing on problems that can be naturally formulated as Anomaly Detection in text, ranging from syntax to stylistics. In this way, we are hoping to facilitate research in Text Anomaly Detection. We also evaluate and analyze two strong shallow baselines, as well as two of the current state-of-the-art neural approaches, providing insights into the knowledge the neural models are learning when performing the anomaly detection task. We provide the code for evaluation, downloading, and preprocessing the dataset at https://github.com/mateibejan1/ad-nlp/.

AD-NLP: A Benchmark for Anomaly Detection in Natural Language Processing
Matei Bejan"
Andrei Manolache"
University of Bucharest matei, bejanes................ee unibuc.ro
Bitdefender
University of Stuttgart amanolache@bi Edefender.comm com
Marius Popescu
University of Bucharest
pnteescummmiis@gaalall com
Abstract
been interested in the Anomaly Detection (AD) problem for several decades (Knorr and Ng, 1997;
Deep learning models have reignited the in- terest in Anomaly Detection research in re- cent years. Methods for Anomaly Detection in text have shown strong empirical results on ad-hoe anomaly setups that are usually made by downsampling some classes of labeled dataset. This can lead reproducibility is- sues and models that are biased toward de- lecting particular anomalies while failing to recognize them in more sophisticated scenar- ios. In the present work, we provide a uni- fied benchmark for detecting various types of anomalies, focusing problems that can be naturally formulated as Anomaly Detection text, ranging from syntax to stylistics. In this way. we hoping to facilitate research in Text Anomaly Detection. We also evalu- ate and analyze two strong shallow baselines, well two of the current state-of-the-arr neural approaches, providing insights into the knowledge the neural models are learning when performing the anomaly detection task. We provide the code for evaluation, download- ing, and preprocessing the dataset https: //gi thub. com com/mateibejjan /ad-nlp/.
Chandola et al., 2009; Aggarwal and Reddy, 2014), some of the first formal mentions of "discordant observations" going back to the 19th century (Edge- worth, 1887). Both general anomaly detectors (Schölkopf et al., 1999; Liu et al., 2008; Manevitz and Yousef, 2002; Ruff et al.. 2018) and narrow- scoped ones (Wang et al., 2019c,b; Ruff ct 2019; Manolache et al., 2021) have shown promising re- sults in various domains, such network moni- toring (Stolfo al., 2000; Sharafaldin et al., 2018: Radford al., 2018), time series (Braei and Wag- ner, 2020), computer vision (Wang et al., 2019c; Yu et al., 2020), and natural language processing (Ruff et al., 2019; Manolache et al., 2021).
The advent of deep learning methods for detect- ing anomalies amplified the interest in the field, producing a great variety of models such ones based on generative networks (Wang et al., 2019b; Zhou and Paffemroth, 2017), self-supervised Iearn- ing (Wang et al., 2019c; Manolache et al., 2021), or one-class classification (Ruff et al., 2018). Al- though the field has seen a sprout of activity, most of the introduced datasets for Anomaly Detection being specifically crafted for Computer Vision, Intrusion Detection Systems (Sharafaldin al., 2018) or network traffic (Stolfo et al., 2000; Rad- ford et al., 2018). Approaches in NLP are usu- ally benchmarked on ad-hoc setups, typically by making use of an annotated dataset and downsam- pling some classes to produce outliers (Wang et al., 2019c; Ruff et al., 2019; Manolache et al., 2021). This does not represent impediment in com- puter vision, due to the large number and variety of annotated datasets. natural language process- ing, however, deciding what an anomaly is and what it is not is a delicate matter. This problem is rooted in the intrinsic complexity of the text: the
1 Introduction
An anomaly, sometimes referred to as an outlier, discordant, or novelty, can be intuitively described as an observation that appears to be inconsistent with the remainder of that set of data (Ord, 1996) to the degree that it arouses suspicion (Hawkins, 1980). Such an observation can be described as be- ing atypical, irregular, erroneous, simply strange (Ruff al., 2020). Anomalies are relative to the data distribution hand and can be perceived as having characteristics that are not definitory to the vast majority of the population. Researchers have
'These authors contributed equally. *Corresponding author.

---

form the language of text takes is influenced not only by the style and lexicon of the author but also by the literary or historical era it was written in and its purpose (e.g., newspaper article, novel, let- ter, satire, etc.). Thus, anomalies can be defined through multitude of factors, ranging from the concrete syntax, style, and grammar, to the more abstract semantics, metaphorical meaning, and con- textual topic. A notable aspect of certain linguistic phenomena that can be effectively analyzed with Anomaly Detection (AD) is the challenge posed by the lack of clear negative examples. For instance, while there are countless texts that were not written by Shakespeare, determining which ones serve as a good representation of non-Shakespearean texts can be tricky. In similar vein, identifying typical non-scientific text can also be challenging.
In recent years, there has been a focus on Com- puler Vision, especially on vidco anomaly detec- tion that uses such benchmarks as Avenue (Lu et al., 2013), ShanghaiTech (Zhang et al., 2016) and UCSD Ped (Mahadevan et al., 2010).
In NLP we note TAP-DLND 2.0 (Tirthankar Ghosal, 2022), a document-level novelty classification dataset, which focuses on semantic and syntactic novelty and tests multiple baselines on a singular task. All datasets of the TAP benchmark are in the newswire domain, we wish to offer larger variety in terms of anomaly types. On top of this, TAP-DLND 2.0 focuses on detecting the degree of novelty of paraphrased or plagiarized text with respect one or multiple ground truth texts, while we target out-of-distribution samples compared to an overarching distribution.
We introduce AD-NLP: an anomaly detection setup for Natural Language Processing which can be used to benchmark AD systems on syntactic anomalies, semantic anomalies, pragmatic anoma- lies and stylistic anomalies, by aggregating differ- ent tasks and treating them as general anomaly detection task, such as: sentence accepiability de- tection, topic detection, metaphor detection and autorship detection. Furthermore, we experiment with two strong classical baselines and two recent deep methods for detecting anomalies text and make observations on both the quantitative and qualitative results of the models.
Additionally, outrof-distribution detection setups (Arora et al., 2021) can be used construct ad-hoc outliers in the validation or test data, albeit these Distribution-shiftee samples are artificial.
2.2 NLP Datasets
Anomaly detection can be viewed as particular case of the one-class classification setting. Classi- fication datasets can be ideal for AD tasks due to their ease of being adapted for one-class unsuper- vised classification (Ruff et al., 2019; Manolache et al., 2021). Various NLP classification bench- marks are widely used, their purpose varying and including news topic detection tasks (Sam Dobbins, 1987; Lang, Ken and Rennie, Jason, 2008; Zhang et al., 2015), sentiment analysis (Maas et al., 201 Socher et al., 2013; He and McAuley, 2016) or authorship verification (Bevendorff et al., 2020).
In the following, we will use the terms "anomaly" and "outlier" interchangeably to refer to the same concept of divergent observation with respect to the overall data distribution.
The paper is organized as follows: Section 2 cov- ers related work. In Section 3, we outline our task definition, data collection approach, and datasets. Sections 4 and 5 delve into the models we used and our experimental assessment. Finally, we wrap up with our conclusions in Section 6.
Recently, more general NLP datasets that con- tain multi-task challenges have emerged due to the desire to obtain general NLP models. decaNLP (McCann al., 2018) provides general frame- work for multitask learning as question answering and is proposing ten tasks together with a leader- board to facilitate research in areas such as multi- task learning, transfer learning, and general pur- pose question answering. GLUE (Wang et al., 2018) is multi-task benchmark and analysis plat- form for Natural Language Understanding. GLUE provides suite of nine sentences or sentence-pair NLU tasks, an evaluation leaderboard, and so- called "diagnostic evaluation dataset". The perfor- mance on the GLUE benchmark has surpassed the
2 Related Work
2.1 AD Datasets
Most of the AD benchmarks were historically used in the domain of Anomaly-based based Network Intru- sion Detection Systems (A-NIDS). Some of these datasets are synthetic, such as KDD99 (Stolfo et al., 2000), CIC-IDS (Sharafaldin et al., 2018) and the LANL. log dataset (Turcotte et al., 2018), while others are obtained using honeypots, like the Kyoto IDS dataset (Song et al., 2011).

---

Table Data statistics for our benchmark. By the "Several" domain we denote a blend of Politics, Science, Fiction, Academia, and News. The Avg #Words column denotes each dataset's average number of words per sample.
Corpus
Train
Test
Avg #Words
Domain
Anomaly
20Newsgroups
10.996
8.819
92
News
Semantic
AGNews
120.000
7.600
31
News
Semantic
COLA
8.551
1.043
8
Linguistics
Syntactic
VUA
8.485
3.637
15
Several
Pragmatic
Song Genres
15.120
3.780
249
Music
Stylistic
Gutenberg Categories
5.000
1.000
618
Several
Semantic
Gutenberg Authors
4.765
1,000
619
Fiction+Polities
Stylistic
level of non-human experts in just twelve months, thus leading to the release of an updated variant called SuperGLUE (Wang et al., 2019a).
and validation code can be obtained through our GitHub repository.
3.2 Data Collection
3 Data
Our data acquisition methodology was designed with the clear goal of providing a large diversity of what we can define as anomalies. This has been done so the data mimics the in-the-wild distribution of classes well possible. Our benchmark con- sists of four already datasets: 20News- groups (Lang, Ken and Rennie, Jason, 2008), AG News (Zhang al., 2015), CoLA (Warstadt et al., 2018) and VUA (Steen et al., 2010), as well as novel datasets: Song Genres, Gutenberg Cate- gories, and Gutenberg Authors. Table provides an overview of the data through examples of outliers and inliers.
3.1
Task Definition
Problem Setting Our benchmark is aimed at ad- dressing broad spectrum of anomaly detection scenarios. In this regard, we concentrate three crucial elements for our benchmark: diversity in the domain of natural language where anomalies occur, diversity within each of these domains (see Table 2), covering various settings that might arise within the same NLP domain, and diversity in terms of sample counts. The final aspect explores the vari- ation in both train and test sample counts across datasets, well as the variation in sample numbers for each class within each dataset.
3.3 Available Datasets
Dataset Properties Firstly, our benchmark cov- ers four outlier classes, as can be seen in Table 2. On top of this, it also aims at delivering intra-class variety by supplying multiple datasets for our con- tent category. We believe this is important so to not lock an anomaly class specific instance of outlier distribution. As an example, content anoma- lies can uppear in news data through a minority of articles with diverging topic, whereas when comes to music genres, lyricism, or fiction writing, the outliers can present cluster of multiple similar subjects which are different from the ones of the majority class. Some dataset statistics are avail- able in Table 1, with more detailed information in benchmark's s datasheet Similarly, training
We selected set of representative existing datasets due to their prior utilization in literature (Ruff et al., 2019; Manolache et al., 2021). The 20Newsgroups and AGNews datasets have been frequently used in various experimental setups, hence, we included them ensure comprehensiveness. Additionally, we incorporated COLA for syntactic anomaly de- tection and VUA for metaphor detection.
20Newsgroups. The 20Newsgroups dataset (Lang, Ken and Rennie, Jason, 2008) amounts to almost 20,000 news documents clustered into twenty groups, each of which corresponds to a different topic. Some newsgroups are closely related, such homp.sys.bm..p..arddeee and womp.syy.mac.hardwwe while others are highly unconnected, namely, misc.forsale and soc.veligion.christin
httpss: wwwptttub.co//aateettieeeee http:////mateelliiaaaaaaman///ba main/ hitn/daatsheet.m

---

Table 2: The outlier classes contained our benchmark and a succinct definition of their data type.
Domain
Description
Syntactic
A pronounced divergence in the arrangement of words and phrases.
Semantic
A difference in the subject or content appearing throughout the sample.
Pragmatic
The presence of metaphors or figures of speech in the sample.
Stylistic
A distinctive manner of expression, including but not limited to: repetition of verbs phrases, of stopwords or punctuation.
We have extracted six classes from the initial twenty: computer, recreation, science, miscella- neous, politics, and religion, as done in (Ruff et al., 2019). Each category is represented by range of 577-2.856 training samples and 382-1.909 valida- tion samples. Despite the relatively small size of the dataset, its classical relevance for NLP tasks prompted us include in our analysis.
quence, the words annotated as metaphors have been prefixed with the "M_" string in the origi- nal annotation setting. To transform this initial problem setup of text segmentation into one of anomaly detection and for the data to comply with our methodology, removed the word-level an- notations and instead labeled the whole sentences as containing a metaphor or not.
AG News. The AG News dataset (Zhang et al., 2015) encompasses 496.835 categorized news arti- cles from over 2.000 news sources. This topic clas- sification corpus was gathered from multiple news sources for over year. Out of all the classes, we have chosen the four largest classes in the dataset: business, sci, sports, and world. We are using the full 30.000 training samples per class, selecting only the title and description fields.
3.4 Newly-Proposed Datasets
We introduce three new datasets: Song Genres, Gutenberg Categories, and Gutenberg Authors. The latter two were extracted from the Project Gutenberg website 2. We scraped the entire web- site and parsed all bookshelves, which stored the book texts, their authors, titles, and the category in which Project Gutenberg placed them. We an- notated the books for said category. The result is a corpus of over 15.000 literary texts?, along with their authors, titles, and titles and bookshelves (a term that Gutenberg maintainers use for categories). We then filtered this dataset to produce Gutenberg Categories and Gutenberg Authors.
CoLA. CoLA (Warstadt et al.. 2018) consists of corpus of 10.657 sentences from 23 linguis- tics publications, books, and articles, annotated as being grammatically acceptable (in other words, grammatically correct) or not. Each sample is sequence of words annotated with whether it is grammatical English sentence. The public version of this dataset used contains 8.551 sentences belonging to the training set and 1.043 sentences as part of the development set. As the is not publicly available, we have used the development set as a de-facto test set for our work.
Song Genres. The Song Lyrics is dataset com- posed of four sources and consists of over 290.000 multilingual song lyrics and their respective gen- res. The initial data was forwarded from the 2018 Textract Hackathon This was enhanced with data collected from three other datasets from Kaggle: 150K Lyrics Labeled with Spotify Valence, dataset lyrics musics, and AZLyrics song lyrics.
VUA. VUA (Steen et al., 2010) consists of 117 fragments sampled across four genres from the British National Corpus: Academic, News, Con- versation, and Fiction and contain word-level all content-word metaphors annotations. The train set contains 12.122 lines or sentences and is the only publicly available subset of VUA. Under these cir- cumstances, we applied an 80-20 train-test split on this solely open data subset and produced 8.485 train samples and 3.637 test samples.
To deal with the lack of labels, we have built a labeling system using the spotipy library. which uses the Spotify API to retrieve an Artist's genre. The Spotify API returns a list of genres for one artist, so we consider the mode of that list to be the
Ptt //wp:///www.. gutenberg. .oogg//
www..kaggle..o//aaeeieeeeeeeee com//nateibeejn/
15000- gutenberg-books
"https://ww.kaaglle htw.:kkagggee.co//nattiieiiaaaa
The data is annotated according to the MIPVU procedure described by its authors. As a conse-
htttps: /tww://wwwwww// http://wwwwpww.t.t.t/cttt rtctteehhh.o///// tt/pextractt200

---

Table 3: Data samples for each anomaly domain included in our benchmark. Apart from Pragmatic and Syntactic, used only chunk of the sample. We did not mention the inlier class in this table, the majority class is constructed by clustering all non-outlier classes from the dataset as inliers.
Domain
Outlier Class
Outlier Sample
Inlier Sample
Semantic
computers
Apple has patented their implementation of regions, which presumably includes the internal data structure.
I am looking for an inex- pensive motorcycle, nothing fancy, have be able to do all maintenance my self.
Syntactic
Gramatically Unacceptable
They caused him to become angry by making him.
Bill coughed his way out of the restaurant.
Pragmatic
Metaphor
Mr Franklin went there at the end of the 1970s, after the col- lapse of Keyser Ullman.
It would be criticism if was doing it impoverish myself.
Stylistic
Arthur Conan Doyle
You may place considerable confidence in Mr. Holmes, sir, said the police agent loftily.
Mars, I scarcely need remind the reader, revolves around the sun at a mean distance of 1 140,000,000 miles.
dominant genre of the lyrics of said artist. Addi- tionally, used the langdetect library to label the lyries with a language automatically. In total, the lyries come in 34 languages. Please note that we have only used the lyrics as our training data, with their respective genres as labels, leaving aside the corresponding artist, year, or song name. We've applied this procedure for all the data apart from the original 2018 Textract data.
ily distinguishable, while others, such as Biology and Botany or Science and Harvard might exhibit significant overlap. We specifically included the CIA category to offer a class that stands distinctly separable from the rest in the text distribution.
We have selected 500 train samples from each class for the train set, and 100 samples per class for the test set. The samples have been extracted from multiple authors for each category, as offer wider distribution of styles, syntax, and grammar. Gutenberg Authors. The Gutenberg Authors dataset represents our second subset generated from the project Gutenberg data. It comprises the texts respective to 10 authors: Arthur Conan Doyle, Charles Dickens, Charles Darwin, Mark Twain, Edgar Allan Poe, Walter Scott, United States Cen- tral Intelligence Agency, H. G. Wells, L. M. Mont- gomery, and Agatha Christie. Again, we aimed at providing different levels of complexity through- out our data, the reason for which we included authors whose novels are within the same genre, namely Doyle and Christie, as well as those writing about the same historical events, such as Twain and Dickens, and female au- thors which supposedly share a common sensibility, meaning Montgomery and Christie. We've 'v'e added the CIA and Darwin classes with the same purpose as for Gutenberg Categories.
From the original dataset, we have curated our subsequent Song Genres subset, which encom- passes nine genres: Pop, Hip-Hop, Electronic, In- die, Rock, Metal, Country, Folk, and Jazz. Song Genres is designed to present anomaly setup where crucial data aspects (e.g., melody, rhythm, etc.) are obscured or absent. This accentuates the necessity of discerning subtle text variations in songs to distinguish between different groups. Through this, we aim to foster the advancement of more robust models for AD in NLP.
Gutenberg Categories. The initial subset de- rived from the original Gutenberg data is termed as the Gutenberg Categories dataset. It com- prises texts corresponding to 10 categories from the Gutenberg project website: Biology, Botany, United States Central Intelligence Agency, Canada, Detective, Historical, Mystery, Science, Children's, s, and Harvard. It's important to note that the cate- gorics are not inherently distinct by nature. Some, like CIA and Children's. s, are expected to be eas-
similar historical
We have sampled between 400 and 500 train text chunks for each author and 100 test samples.

---

The samples have been extracted through multiple books for each author we consider for this experi- ment. This has been done to avoid the possibility of the event in which a particular anomaly class might be locked into a repetitive word or phrasing, e.g., character names (Sherlock Holmes, Huckleberry Finn, etc.), places (London, Washington, etc.) or simply by the sample length.
ing dataset. The Isolation Forest technique is an- other outlier detection approach, drawing inspira- tion from the Random Forest model, as described in (Liu et al., 2008). In an n-dimensional space, inliers typically form denser clusters, whereas out- liers tend to be more dispersed.
To optimize the performance of traditional meth- ods, we conducted a comprehensive hyperparame- ter tuning. For the OC-SVM, we explored different kernels, namely rbf, polynomial, and linear, and as- sessed a range of V values: V E 0.05, 0.1, 0.2, 0.5. For the Isolation Forest, we evaluated various num- bers of estimators, specifically 64, 100, 128, 256, For both models, we compared the effectiveness of two embedding methods: FastText and Glo Ve, each with an embedding size of 300.
4 Models
4.1 Experimental Methodology
Our methodology consists of creating multiple data splits for each dataset within our benchmark, run- ning our models on all splits, and finally aggre- gating the results. By data split, we refer to a separation of classes into one inlier class and a cluster of classes that are considered to be outliers in this setup. To achieve this, we iterate through all the classes for every dataset and, at each iter- ation, choose one of them as the inlier, while the rest are treated as outliers by the models. Through this, we achieve an important objective: we unfold an exhaustive series of experiments over all possi- ble combinations of outliers and inliers, outlining which of the former are the most prominent and which are the hardest detect. We ran hyperpa- rameter search for the two classical models on each split, thus finding the best parameters for detecting each outlier choice. The deep models were trained with a limited set of hyperparameters, as can be seen in Subsections 4.2 and 4.3.
4.3 Neural approaches
CVDD. Context Vector Data Description (CVDD) (Ruff et al., 2019) is a method that takes advan- tage of pre-trained word embeddings to perform AD on text. CVDD jointly learns the so-called "context vectors" and a multi head self-attention mechanism that projects the word representations near these "context vectors" by minimizing the co- sine distance between them.
CVDD allows the disentanglement of the con- text vectors such that they can provide more inter- pretable results and penalize non-orthogonal con- text vectors. The resulting projection function and context vectors act like a clustering method and cluster centroids, respectively. Anomalies are de- tected based on the mean sample distance from the sequence projection to the context vectors. We only search for the optimal number of context vec- tors like in (Ruff et al., 2019) (c E {3,5, 10)) and report the best performing models.
Evaluation Metries. We use AUROC in Table 4, as well as AUPR-In, and AUPR-Out in Table and Supplementary Tables 7 and 8. AUROC (Area Under the Receiver Operating Characteristic) is the area under the curve where the false positive rate is on the X-axis and the true positive rate on the Y-axis. AUPR-In (Area Under the Precision-Recat Recall for Inliers) is the area under the curve where the recall is on the X-axis and the precision is on the Y- axis. AUPR-OU Out (Area Under the Precision-Resd Recall for Outliers) has the same definition as AUPR-In but is computed on inverted labels.
DATE. Detecting Anomalies in Text using ELECTRA (DATE) an approach that uses self- supervision for training Transformer networks us- ing two pretext tasks tailored for detecting anoma- lies - Replaced Token Detection (RTD) (Clark et al., 2020) and Replaced Mask Detection (RMD). The method uses generator to sample masked tokens and discriminator to identify the replaced to- kens and the masking patterns. The generator can be any distribution over the vocabulary. The dis- criminator
4.2 Classical approaches
The SVM classifier is versatile model adapt- able for outlier detection tasks, known as the One Class Support Vector Machine, as detailed in (Schölkopf et al., 1999). The OC-SVM aims to leamn from an inlier class, designating test sam- ples as anomalies when they deviate from the train-
BERT (Devlin et al., 2018) model trained from scratch. The Replaced Token Detec- tion head ouputs a token-wiin anomaly score which

---

Table 4: Best AUROC scores for each model and each split. We also provide the mean and standard deviations between the splits for all datasets apart from COLA and VUA, where single split inlier class.
Inlier Class
iForest
OCSVM
CVDD
DATE
comp
75.6
76.8
51.9
92.1
20 News
rec
68.0
74.1
55.8
83.4
sci
66.7
73.5
49.2
69.7
misc
60.8
64.0
59.6
86.0
pol
53.0
60.9
63.2
81.9
rel
74.8
78.7
53.8
86.1
mean + std
66.5 7.8
71.3 6.6
55.6 + 4.7
83.2 + 6.8
AG News
business
72.9
83.9
64.0
90.0
sci
75.6
80.6
70.7
84.0
sports
78.3
89.1
79.2
95.9
world
83.8
86.0
65.8
90.1
mean + std
77.6 4.0
84.9 3.1
69.9 + 5.9
90.0 + 4.2
Indie
51.5
45.6
47.1
53.7
Pop
43.1
43.6
47.2
57.6
Metal
55.1
43.4
46.9
51.2
Song Genres
Hip-Hop
63.3
68.0
66.8
54.0
Electronic
47.8
47.2
44.6
54.0
Country
57.8
57.7
54.6
66.7
Folk
48.9
50.2
48.9
52.0
R&B
57.6
56.6
55.5
70.1
Rock
44.7
45.8
43.5
54.8
Jazz
53.3
53.0
45.7
70.2
mean + std
52.3 7.8
51.1 + 7.4
50.1 + 6.7
58.0
+ 7.2
Detective
70.9
72.2
67.5
82.5
Gutenberg Categories
Botany
45.9
42.8
46.9
74.2
CIA
88.1
94.8
85.6
99.9
Mystery
73.3
69.3
55.4
85.4
Biology
83.5
82.3
79.5
81.9
Children's s
56.9
58.8
65.0
79.7
Harvard
70.3
79.3
66.6
82.1
Canada
45.5
39.7
49.0
53.3
Science
70.4
67.2
55.4
68.0
Historical
77.3
73.3
75.5
66.7
mean + std
68.2 + 13.8
68.0 + 16.2
64.6 + 12,3
77.4 + 12.0
Dickens
83.5
85.1
90.0
35.1
A.C. Doyle
53.6
41.8
55.4
34.7
Gutenberg Authors
M. Twain
85,8
82.8
78.8
93.7
C. Darwin
74.0
75.4
82.9
99.8
W. Scott
87.1
82.1
91.0
11.0
A. Christie
93.4
91.2
92.5
55.2
E.A. Poe
71.9
72.7
78.6
48.7
CIA
87.9
94.7
83.0
100
L.M. Montgomery
78.3
84.9
94.1
61.5
H.G. Wells
53.7
59.8
37.9
11.8
mean 1 std
76.9 1 13.2
77.0 + 15.0
78.4
1 17.2
55.2 1. 32.0
COLA
(Acceptable)
49.9
49.2
53.3
57.2
VUA
0 (Non-Metaphor)
75.3
76.2
76.6
51.1

---

Table 5: CVDD's ability to cluster semantically related words from the Hip-Hop Music Genre subset. The model effectively identifies variations in verb and pronoun usage, recognizes foreign language terms, and associates specific contexts with obscene terms. We highlight second context as it is the most meaningful one when detecting anomalies.
Context
Context 2
Context
Context
Context 5
'm
livin
to
the
que
turnin
can of mas
're
shakin
could in
waitin
make 's porque
somebody
walkin
will that sabe
myself'
keepin
would
desde
everybody
slippin
pray this boca
obscenityl
lickin
try
surrounds
obscenity2
standin
come
descends
encontrar
Table 6: The Area under the Precision-Recal curve for all of the datasets, averaged over subsets. highlight the best scores in bold. Abbreviations: 20NG 20Newsgroups, AGN AGNews, SG Song Genres, GC Gutenberg Categories, GA Gutenberg Authors, CA COLA, VA VUA.
the search methodology mentioned Sec. 4. The models were trained on an inlier class, and every other class was considered anomalous at test time. in Tables 4
The results of the experiments and 6, calculated averages over all splits for each dataset. We can observe that DATE has achieved the best AUROC scores on six out of our seven datasets, with a total average score 71.11%, fol- lowed by CVDD, which scored the best on VUA. The average AUROC scores are almost equal across the three models: OC-SVM 64.9%, IsoForest 63.2%, CVDD 64.3%. Interestingly, every other model is close 50% AUROC on COLA. Results on 20Newsgroups and AGNews are similar across all splits; thus, it makes sense to consider either split as the inlier.
Metric
iForest
20NG
AUPR-In
90.8
89.6
133
41.43 11.
AUPR-Out
32.6
96.0
AGV
=
76.9
AUPR-Out
70.4
57.5
46.1
95.9
SG
AUPR-In
90.6
90.7
90.3
12.8
AUPR Out
12.3 7.0
11.7
10.2
92.3
In
94.0
94.1 4:433
93.9
36.8
AUPR-Out
25.1 21.0
22.6 16.3
19.4 11.2
96.3 2:2
AUPR-In
96.3 25
96.4
96.3
48.5 à: 33.5
AUPR Out 39.4 + 25.3
36.8 + 21.1
37.9 = 17.7
97.2 3.35
AUPR-In 31.8 :: N/A
31.6 N/A
34.2 N/A
36.4 H: N/A
3 AUPR-Out
68.0 =: N/A
66.9 NIA
69.9 NA
73.5 1: N/A
The Area under the Precision-Reccd Curve (AUPR) results are summarized in Table 6, with detailed results for cach split being available in the Appendix in Tables 7 and 8, where we provide exhaustive table with the metries for every split and for each dataset.
AUPR-In 77.2 :: NAA
78.1 NIA
76.1 = NA
55.7 1 NAA
$
AUPR-Out 73.3 = NA
74.8 NIA
75.0 NAA
48.4 + NAA
can also be used as an explainability mechanism. The anomalies are then detected by averaging the PLRRD score over the entire sequence.
5.2 Qualitative results
We use the same experimental setup and hyper- parameters in (Manolache al., 2021) for the 20Newsgroups and AG News datasets, and the AG News setup for the other datasets.
We provide a sample of qualitative results in Figure 1, where we show how DATE can detect semantic nuances for the Botany subset of Gutenberg Cate- gories, and in the Supplementary Figure 2, where show how DATE can detect stylistic changes when correctly profiling an author. The model is suspicious of certain punctuation and word usage. When scoring an inlier entry, DATE seems to think that some stopword usage is anomalous; this could indicate a change of stylistic approach when an
5 Experiments
5.1 Quantitative results
We test the performance of all models on all the datasets within our benchmark and report the scores using the best hyperparameters following

---

Botany Detected Inlier
like song lyrics, are harder to detect and separate. We have also disclosed our results on various mod- els and found out that the neural models react well to domain-speciff words, author idiosyncrasies and punctuation as being anomalous or not. We hope that the proposed benchmark and tools will facilitate research in Text Anomaly Detection.
thinty flowers crossed plants crossed with #alen from other
crossed plants of same generation the #lwwen #etty capsule wes thus produced contained on an average, 4, 3 seeds; whilst thirty flowers on the sell fe #art #alis ##ed plants le #arti #ells #sed with the Hepol ##lln from the same flower, produced twenty- - three capsule eNs. each #licon #*tain #eing 4. 3 seeds. thus the average number of seeds the crossed ##cap #esul #ites capsule ##ts as ah #Rund ##red of the crossed seeds weighed [SEP]
that the sell #arti #elis atted
Bo Detected Outlier
Limitations & Further Work
it would have been an old look for child of #ht ##wo ##l ##vo, and sara crewe was only seven. the fact dreaming and thinking things and could not herself errem fiter any time when she had not been thinking things about grown up eape #eople and the world they belonged to. she folt as if she had lived along.. long time. this moment she was remembering the voyage she had just made from #ebo semba #iyy with her father, captain brewe. she thinking of the big seship of the las #scars passing [SEP]
however, that she #ewas always
Some of our more simple basclines managed to outperform more sophisticated anomaly detection methods in some scenarios. For instance, we did observe that on datasets such COLA and VUA, both CVDD and DATE obtain weak results as an example, in Tables 7 and of the Supplemen- tary Material, we can observe that the OC-SVM and the Isolation Forest outperform DATE on the metaphor detection task from VUA. Moreover, we can observe that for every dataset in the benchmark, there are instances where the Isolation Forest and the OC-SVM outperform the more sophisticated CVDD and DATE methods. Therefore, we believe that it's very important be able to analyze the limitations and inductive biases on a wide range of scenarios while developing an Anomaly Detec- tion methodology. One way of accomplishing this would be studying AD-NLP at an even more gran- ular level. For example. one could determine vari- ous linguistic properties of texts written by Edgar Allan Poe, aiming to discover the reasons behind the poor performance of DATE compared to the other authors, as can be observed in Table 4. Fur- ther linguistic analysis would benefit the quality of AD-NLP, and we leave this undertaking for further work.
Figure I: We provide qualitative examples for outliers and inliers detected by DATE for the Botany subset of the Guttemberg Categories dataset. stronger red highlight indicates greater anomaly score. We can that, in the case of the inlier sample, there is no strong indication of abnormalities, since the contents relate to Botany. For the outlier sample, words such as "ship" and "cars" " are detected as being anomalous.
author is writing texts in registers.
We show in Table how CVDD creates clusters of semantically similar words when trained on the Hop-Hop Music Genre subset. The context vectors act as topic centroids. CVDD cannot only distin- guish between colloquial usage of verbs, pronouns, and foreign languages but also associates unrepro- ducible words (e.g., obscene, insulting, etc.) with certain contexts.
6 Conclusions
We introduce AD-NLP, benchmark for anomaly detection in text over an extensive assortment of outlier scenarios, covering syntactic, semantic, pragmatic, and stylistic language anomalies. Addi- tionally, we introduce three new datasets as part of AD-NLP: Song Genres, Gutenberg Categories, and Gutenberg Authors. Song Genres provides a com- plex setting in which part of information about the data has been obscured, enforcing a distinct focus on the subtle differences between texts, while the two datasets derived from the Gutenberg data meant provide variety on multiple levels: syn- tax, style, genre, and literary movement. We find that anomalies that depend solely on semantic or stylistic aspects of the text are casier to recognize, whereas those that only partially depend on the text,
Benchmark updates. We commit to enhancing AD-NLP with future datasets that would expand the intra or doter-domain variety, either by adding new datasets to AD-NLP or scraping, aggregating, and labeling new data off the web ourselves. We also commit to making the dataset more accessi- ble through multiple hosting services, as well updating our GitHub repository.
Acknowledgments
AM acknowledges the International Max Planck Research School for Intelligent Systems (IMPRS- IS) for the support provided.

---

References
Lang, Ken and Rennie, Jason. 20 newsgroups dataset, 2008. URL http::/peoplee.cmt csail, mi.....it.
C. Aggarwal and C. K. Reddy, editors. Data Clus- tering: Algorithms and Applications. CRC Press, 2014. ISBN 978-1-46-655821-2. URL http:/ /www. crcpress. wtt./com//oooe httm///roouu//isb////o /561:s///12//9911166666822222
F. T. Liu, K. M. Ting, and Z.-H. Zhou. Isolation for- est. In Proceedings of the 2008 Eighth IEEE Inter- national Conference on Data Mining, ICDM '08, page 413-422, USA, 2008. IEEE Computer Society. ISBN 9780769535022,, doi: 10. 10910/.....00088.... 17. URL https: //doii org/ ///g 1189/ICDM 200... 17.
U. Arora, W. Huang, and H. He. Types of out-of- distribution texts and how to detect them. CoRR, (bs/2109.06827, 2021. URL https:://axxi...... abs/2109. 06827.
C. Lu, J. Shi, and J. Jia. Abnormal event detection at 150 fps in matlab. In 2013 IEEE International Conference on Computer Vision, pages 2720-2727, 2013. doi: 10. 109/CCCV.2013.338.
J. Bevendorff, B. Ghanem, A. Giachanou, M. Keste- mont, E. Manjavacas, I. Markov, M. Mayerl, M. Pot- thast, F. Rangel Pardo. P. Rosso, G. Specht, E. Sta- matatos, B. Stein, M. Wiegmann, and E. Zangerle. Overview of PAN 2020: Authorship Verification, Celebrity Profiling. Profiling Fake News Spreaders on Twitter, and Style Change Detection, pages 372- doi:
A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sen- timent analysis. In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150. Portland, Oregon, USA, June 2011. Asso- ciation for Computational Linguistics. URL http: //www. wwwweb httpg/anhhoooo//1111100
383. 09 2020. ISBN 978-3-030-58218-0 10.1007/978- 1-788----0--8------1 00280-8221-72555
M. Braei and S. Wagner. Anomaly detection in univari- ate time-series: A survey the tf-th-f-he--rt.... 04 2020.
V. Mahadevan, W. Li, V. Bhalodia, and N. Vasconcelos. Anomaly detection in crowded scenes. In 2010 IEEE Computer Society Conference on Computer Vision and Patten Recognition, pages 1975-1981, 2010. doi: 211100//VPR.2200.555333
V. Chandola, A. Banerjee, and V. Kumar. Anomaly detection: survey. ACM Comput. Sure, 41 (3), July 2009. ISSN 0360-0300. doi: 10.1145/ 1544888..54188. URL http:://doi.orge/ 1145/1541 880. 1541882.
L. M. Manevitz and M. Yousef. One-class syms for document classification. J. Mach. Learn, Res., 2: 139-154, Mar. 2002. ISSN 1532-4435.
K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning. Electra: Pre-trate Pre-training text encoders as discriminators rather than generators. ArXiv, abs/2003. 10555, 2020.
A. Manolache, F. Brad, and E. Burceanu. DATE: Detecting anomalies in text via self-supervision of transformers. Proceedings of the 2021 Confer- ence of the North American Chapter of the Associ- ation for Computaaional Linguistics: Human Lan- guage Technologies, pages 267-277, Online, June 2021. Association for Computational Linguistics. doi: 18.16653/ 1/2021 .nacc-mmin.22 URL. ht //aclanthology org/2021..... wwacl-mmin.n2
J. Devlin, M. Chang. K. Lee. and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL httt:://ar//t wwtpx/ivvoogg///aeee org/abs/1814 04880.
the
F. Y. Edgeworth. Xli. on discordant observations. Philosophical Magazine Series 1, 23:364-375, 1887. URL https: http:://app....mattteeeeeee weappsssaaaiiccholarr...o CorpusID: 120568135
B. McCamn, N. Keskar, Xiong, and R. Socher. The natural language decathlon: Multitask learning as question answering. 06 2018.
D. M. Hawkins. Idenrification of outliers, volume 11. Springer, 1980.
K. Ord. Outliers in statistical data V. Bar- nett and T. Lewis, 1994. 3rd edition, (John Wiley & Sons, Chichester), 584 pp.. [UK pound]55.00, 5.00 ISBN 0-471-93094-6. International Journal of Forecasting, 12(1):175-117, March 1996. URL https: //ideas. repec. hwtpg///eeeeenneeeee v12y1996i1pp 1116611p115-1111 16. html.
R. He and J. McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th In- ternational Conference on World Wide Web, WWW '16, page 507-517, Republic and Canton of Geneva, CHE, 2016. International World Wide Web Confer- ences Steering Committee. ISBN 978 1450341433. doi: 10. 1441144452272422288800777 URL //doi. org/18 1////0.110 1et.11122/22221111 210.///22222222228883..8 2883837.
B. Radford, L. Apolonio. A. Trias, and J. Simpson. Network traffic anomaly detection using recurrent neural networks. 03 2018.
E. M. Knorr and R. Ng. A unified notion of outliers: Properties and computation. In Proceedings of the Third International Conference Knowledge Dis- covery and Data Mining, KDD'97, page 219-222. AAAI Press, 1997.
L. Ruff, R. A. Vandermeulen. N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, F. Müller, and M. Kloft. Deep one-class classification. In Proceedings of the 35th International Conference on Machine Learning. volume 80, pages 4393-4402, 2018

---

L. Ruff. Y. Zemlyanskiy, R. Vandermeulen, T. Schnake, one-class
A. Wang. A. Singh. J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP@ EMNLP, 2018.
and M. Kloft. Self-attentive, classification for unsupervised anomaly detection on text. In Proceedings of the 57th Conference of the As- sociation for Computational Linguistics, pages 4061- 4071, jul 2019.
A. Wang, Y. Pruksachatkum, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy. and S. Bowman. Su- perglue: A stickier benchmark for general-purpose language understanding systems. 05 2019a.
L. Ruff, J. Kauffmann, R. A. Vandermeulen, G. Mon- tavon, W. Samek, M. Kloft, T. G. Dietterich, and K. Muller. A unifying review of deep and shallow anomaly detection. ArXiv, abs/2009. 17732, 2020.
H. Wang, M. Li. F. Ma, S.-L. Huang. and L. Zhang. Unsupervised anomaly detection via generative ad- versarial networks: Poster abstract. In Proceedings of the 18th International Conference on Informa- tion Processing in Sensor Networks, IPSN '19, page 313-314, New York, NY, USA, 109b. Association for Computing Machinery. ISBN 9781450362849. doi: 10. (0.1144533002506331266600 URL https://doi. org/10. 1145/3302566 3605
S. W. Sam Dobbins. Mike Topliss Reuters-21577 dataset, 1987. URL http://kdd. http://ddd.ssccccieeeee edu/ aatabases//euterr255333rrrerrrrr
html.
B. Schölkopf, R. Williamson, A. Smola, J. Shawe- Taylor, and J. Platt. Support vector method for nov- city detection. volume 12. pages 582-588, 01 1999.
I. Sharafaldin, A. Habibi Lashkari, and A. Ghorbani. Toward generating new intrusion detection dataset and intrusion traffic characterization. pages 108-1 16, 01 2018. doi: (0.520000066888000000
S. Wang. Y. Zeng. X. Liu, E. Zhu, J. Yin, C. Xu, and Kloft. Effective end-to- and-to-end unsupervised outlier detection via inlier priority of discriminative network. In NeurlPS, 2019c.
R. Socher, A. Perelygin, J. Wu, J. Chuang, C. Manning, A. Ng. and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. EMNLP, 1631:1631-1642, 01 2013.
A. Warstadt, A. Singh, and S. R. Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805. 124.7 2018.
G. Yu. S. Wang. Z. Cai, E. Zhu, C. Xu, J. Yin. and M. Kloft. Cloze Test Helps: Effective Video Anomaly Detection via Learning to Complete Video Events, page 583-591. Association for Com- puting Machinery. New York, NY, USA, 2020. İSBN 978 9581499379885. URL htt.tttss//do.orgg// 1145/3394171.35 34139773.
J. Song, H. Takakura, Y. Okabe, M. Eto, D. Inoue, and K. Nakao. Statistical analysis of honeypot data and building of kyoto 2006+ dataset for nids eval- uation. In Proceedings of the First Workshop on Building Analysis Datasets and Gathering Experi- ence Returns for Security, BADGERS '11, page 29-36, New York, NY. USA, 2011. Association for Computing Machinery. ISBN 9781450307680. doi: 10. 09911455/978622.1988676 URL h/tps://doi. org/18. 1145/1978672.
X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1. NIPS' 15, page 649-657, Cambridge, MA, USA, 2015. MIT Press.
G. Steen, L. Dorst, J. Herrmann, A. Kaal, T. Krennmayr, and T. Pasma. A method for linguistic melaphor identification: From MIP to MIPVU. 06 2010. doi: 10.1075/celcr.14.
Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma. Single- image crowd counting via multi-column convolu- tional neural network. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 189-597,77 2016. doi: 10. N0.1109/CVPR.2066
S. Stolfo. W. Fan, W. Lee. A. Prodromidis, and P. Chan. Cost-based modeling for fraud and intrusion detec- tion: results from the jam project. In Proceedings DARPA Information Survivability Conference and Exposition. DISCEX'00 00, volume 2, pages 130-144 144 vol.2, 2000, doi: 10. 119////SEEEEEOO. 25008
C. Zhou and R. C. Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining, KDD '17, page 665-674, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874. doi: 10. 10.1145/309998333099888 URL httpp:://do 1145388.144/3897983.3. 3098822
T. B. A. E. P. B. Tirthankar Ghosal, Tanik Saikh. Novelty Detection: A Perspective from Natural Language Processing. Computational Linguistics JOurnal, April 2022. URL. httpp:://diree mit. -ttptt--DDeeeettoo---eeeeeeeeeee Frw.etiaiiie-ro--naturrra.ll.
M. J. M. Turcotte, A. D. Kent, and C. Hash. Unified Host and Network Data Set. chap- ter Chapter 1, pages 1-22. World Scien- tific, nov 2018. doi: 10.1142/978178634555
001. URL https://ww// //www wor wwwldscientiiic.c doi/abs 000. 1142/9781786345566

---

8 Supplementary Materials
8.1 More quantitative results
In Tables 7 and 8 we present the Area Under the Precision-Recale curve (AUPR) results for all our models and datasets. This provides clear mea- of how each model performed across different datasets.
Table 8: Best AUPR-Out scores for each model and for each split.
Table 7: Best AUPR-In scores for each model and for each split.
Inlier Class
iFurest
OCSVM
CVDD
DATE
Inlier Class
iForest
OCSVM
CVDD
DATE
comp
38.7
43.2
27.6
92.1
45.1
57.3
29.6
83.4
20 News
94.5
94.6
78.8
92.1
38.6
53.2
23,4
69.7
20 News
rec sci
82.9
84.6
81.6
83.4
misc
31.2
32.3
25.5
86.0
86.5
88.7
94.7
69.7
pol
$1.9
misc
84.1
85.8
88.9
86.0
35.1
45.2
20.9
86.1
53.0
%6.1
90.4
81.9
rel
74.8
94.6
75.7
86.1
AG News
business
49.8
67.9
38.5
96.1
sci
52.2
61.1
46.4
93.5
AG News
business
83.7
92.6
82.2
74.8
sports
61.4
79.3
58.5
98.5
89.0
91.5
86.1
62.4
world
66.9
73.5
40.8
95.5
sports
89.7
94.7
90.2
88.8
Indie
10.6
9.1
9.1
91.6
world
93.2
93.7
83.1
81.9
Pop
8.9
9.0
10.2
92.5
Indlie
89.9
89.1
88.8
10.5
Metal
12.1
11.0
9.7
89.8
Pop
87.4
86.4
87.7
Song Genres
Hip-Hop
22.5
31.9
15.9
91.6
Metal
91.1
90.8
89.8
10.3
Electronic
9.3
9.1
8.5
90,9
Song Genres
Hip-Hop
93.3
94.1
94.8
9.3
Country
12.4
12.1
10.9
95.0
Electronic
89.7
89.7
88.5
11.5
Folk
9.7
10.0
9.2
91.1
Country
92.9
93.1
92.5
15.1
R&B
12.1
11.2
11.0
94.8
Folk
90.1
90.4
89.9
10.0
Rock
888
9.5
8.2
91.0
R&B
92.4
92.4
92.4
18.6
Jazz
10.5
10.1
8.6
95.0
Rock
88.3
88.4
87.8
11.3
Detective
22.3
23.0
17.5
97.1
Jazz
91.6
91.3
89.7
19.4
Gutenberg Categories
Botany
Detective
95.0
95.4
94.9
38.0
64.9
90.0
45.22
100.0
Gutenberg Calegories
Bolany
87.2
86.8
89.0
59.2
Mystery
11.7
98.2
CIA
98.0
98.8
98.0
99.9
Biology
31.3
27.7
30.0
97.6
Mystery
96.2
95.5
92.6
32.6
Children's
12.6
16.6
97.2
Biology
97.6
97.5
96.8
28.1
Harvard
18.1
26.1
13.4
97.6
Children's
93.1
93.5
94.2
30.0
Canada
9.0
10.6
90.2
Harvard
95.5
97.0
95.4
36.5
Science
17.3
15.5
10.6
95.2
Canada
85.4
84.7
88.5
11.6
Historical
22.1
19.0
25.4
95.5
Science
95.7
95.1
93.1
Charles Dickens
37.4
33.8
45.6
98.5
Historical
96.9
96.4
96.6
13.6
A.C. Doyle
10.9
89.3
C. Dickens
97.6
98.0
98.8
35.1
VUA COLA Gutenberg Authors
M. Twain
44.2
30.4
29.2
99.3
A.C. Doyle
93.2
90.7
92.9
7.2
Darwin
29.9
33.6
45.2
99.9
Gulenberg Authors
M. Twain
98.0
97.6
96.7
61.5
W. Scott
39.5
28.4
48.5
94.6
C. Darwin
95.3
95.7
96.4
98.8
Christic
74.8
69.7
48.0
99.2
W. Scott
98.3
97.6
98.9
11.0
29.0
35.8
31.9
97.9
A. Christie
98.9
98.5
99.1
55.2
CIA
65.8
91.5
47.2
100
E.A. Poe
94.6
94.7
96.9
48.7
L.M. Monigomery
27.6
64.3
99.2
CIA
97.5
98.3
97.4
100
H.G. Wells
12.4
L.M. Montgomery
97.0
97.8
99.3
55.7
H.G. Wells
92.9
93.3
85.8
11.8
(Acceptable)
68.0
66.9
69.9
73.5
VGA COLA
(Acceptable)
31.8
31.6
34,2
36.4
(Non-Metaphor)
73.3
74.8
75.0
48.4
(Non-Mctaphor)
77.2
78.1
76.1
55.7

---

9 More qualitative examples
this section, we provide additional qualitative examples to for DATE in Figure 2. These exam- ples are derived from the Gutenberg dataset, focus- ing specifically on three distinct subsets: Biology. works of Agatha Christie, and CIA-related texts. The aim is showcase how DATE identifies and highlights anomalous patterns within different au- thors and styles of text.
Biology True Positi
these esta recu #elt #tties determine all the others. creature that feels and
#ares stomach carry food in food requires digest plants, which not and ##mo save have need of a stomach but have roots instead thus the arimal functions of leeling and moving determine the character eathe organs of the second order, the organs of digest #eion, these int #ahei #er turn prior to the organs of circulation, which are means #4to the end #est Hon eebu eutin (SEP)
instruments #eide
Agata Christie True Negative
turned to whitaker almai #Rnad #Ak brows upon the
statienics the Hegre #eat suropean armies was rouse #ed from by
the breakfast gong breakfast there was no talk anything but war hught
was as exciled #has cal thunder #ay weather, and the small boys
wanted information #tab #doul fagg the russian and the serbian
dispute, the ##ag page webster dictionary had consulted newspapers
abnormal #ely
mr. bilt #eling Il eatrin of supplying #etri Baviel [SEP)
CIA True Negative
how much more the eacross #9ed plants plants the seventh generation #ese were raised here #Roo #efor ##c with the following result: :-- table 2/ 8, lo #eomo #eca pu #erp #eure #ea seventh generation) ). eight #hs of plants inches: gotumn: number ( name ) pot of #eum #ēn prossed plants ol #aum ##n 3 self fe
Figure 2: Qualitative examples of DATE on the Guten- berg data. From top to bottom: Biology True Positive sample, Agatha Christie True Negative sample, and CIA True Negative sample. The darker the colour of the highlight, the more greater the anomaly score for the word.


Reviews:
[{"review_id": "4MiUWVr24i", "reviewer": "Reviewer_SUCF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We will try to address their questions:\n\nR1: Without available links to anonymized code repositories or a complete description of hyperparameters and other experimental settings, it is not possible for me to verify the reproducibility of the results.\n\nA1: Thank you for pointing this out, we will add the anonymized code to the camera-ready version of the paper, but we also provide it here: https://anonymous.4open.science/r/ad-nlp-B01C/README.md. \n\n---\n\nR2: The motivation for creating a diverse anomaly detection benchmark is reasonable, but could probably be justified a bit better. Why are current evaluation methods insufficient? What specific limitations do they have? For example, explicitly discuss the limitations of current ad-hoc evaluation setups and the benefits of a more comprehensive suite.\n\nA2: Our purpose for this benchmark was to propose NLP several setups that provide a natural anomaly detection problem, not an artificially constructed one (e.g. classic text categorization). For example, we propose the following setup for the COLA dataset: we consider a non-grammatically sound sentences as anomalies. In the case of VUA, we consider the metaphoric use of words being the non-regular of employing said words in speech or writing. In the case of Gutenberg Authors, we consider that authorship is a natural problem of anomaly detection: in our setting we consider one author as the main class and we want to distinguish the rest of the texts from his style, said texts being an anomaly from the point of view of his writing style.\n\n---\n\nR3: The 3 new datasets created have probably not been introduced as well as they could have been. Reading that section, I saw some details about the construction of the dataset, but very little motivation about why the creation of these particular datasets were necessary and how they are more helpful vs other existing datasets in the public domain. More motivation and less details in this section would have been helpful.\n\nA3: We introduced Gutenberg Authors as an authorship anomaly detection problem, which is still an open problem in NLP. Gutenberg Categories, by comparison, presents more similarities with the other topic detection problems (e.g. 20Newsgroups, AG News), but offers a higher degree of diversity. Same goes for the Song Lyrics dataset. Our purpose with the latter two was to introduce diversity and complexity to the topic anomaly detection field.\n\n---\n\nR4: The analysis and discussion of model performance on different anomaly types could do with some more explorations. In its current state, it is borderline.\n\n\nA4: We offer more detailed quantitative results (AUPR-In , AUPR-Out scores) in the Appendix, as well as a couple of qualitative analysis plots of the deep learning approaches. We will add more comments on the results we provided as part of the Appendix.\n\n---\n\nR5: In the first page, after the name of the paper, it says \"Anonymous ACL submission\" which should be \"Anonymous EMNLP submission\" but I am not sure if that makes any difference?\n\nA5: Thank you for pointing this out! We started writing the paper with an ACL template format. At the end we changed to the EMNLP format and forgot to change the subtitle of the paper to “Anonymous EMNLP submission”. We will do so for the camera-ready version.\n\n---\n\nR6: Overall, there are some issues with writing style in the paper that I feel can be resolved with another round of proofreading.\n\nA6: Thank you! We will do another round of proof-reading for the camera-ready version.\n\nWe once again thank the reviewer for their insightful feedback and we hope that all of their raised concerns have been addressed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, the goal of having a unified benchmark for anomaly detection in NLP is an important one. The authors highlight some existing challenges and then take an important step towards solving the problem with their proposed benchmark.\n\nSome existing challenges highlighted in the paper and the corresponding solutions proposed:\n- Deciding \"what is an anomaly\" is challenging. (To overcome this they propose 4 anomaly classes - syntactic, semantic, pragmatic and stylistic)\n- Very narrow domain of previous datasets and benchmarks (To overcome this, they propose a suite of datasets covering various topics and genres of content).\n- The benchmark also overcomes the problem of not having a standard approach for evaluating different methods.", "reasons_to_reject": "Here are some of my personal opinions about the paper. These reflect things that I would have liked to see but may not necessarily be a strong reason to accept or reject a paper.\n\n- Without available links to anonymized code repositories or a complete description of hyperparameters and other experimental settings, it is not possible for me to verify the reproducibility of the results.\n- The motivation for creating a diverse anomaly detection benchmark is reasonable, but could probably be justified a bit better. Why are current evaluation methods insufficient? What specific limitations do they have? For example, explicitly discuss the limitations of current ad-hoc evaluation setups and the benefits of a more comprehensive suite.\n- The 3 new datasets created have probably not been introduced as well as they could have been. Reading that section, I saw some details about the construction of the dataset, but very little motivation about why the creation of these particular datasets were necessary and how they are more helpful vs other existing datasets in the public domain. More motivation and less details in this section would have been helpful.\n- The analysis and discussion of model performance on different anomaly types could do with some more explorations. In its current state, it is borderline.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The main contributions of the paper are as follows:\n- This paper presents AD-NLP, a benchmark dataset for evaluating anomaly detection methods in natural language processing. \n- The benchmark covers 4 anomaly classes - syntactic, semantic, pragmatic, and stylistic. (Table 1)\n- The datasets included in the benchmark cover 4 existing datasets that are used in previous literature - 20Newsgroups, AGNews, COLA and VUA. \n- The authors further augment this with 3 new proposed datasets based on Project Gutenberg and Song Lyrics data from Textract. \n- Then, on the created dataset, 4 methods are evaluated, 2 non-neural (One Class Support Vector Machine, Isolation Forest) and 2 neural (Context Vector Data Description (CVDD), Detecting Anomalies in Text using 452 ELECTRA (DATE)).\n- For evaluation, the primary metric considered is AUROC (Area Under the Receiver Operating Characteristic curve). DATE and CVDD usually have the best performance according to this metric across the datasets in the benchmark. (Table 4)", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- This might just be me, but I felt that there was a bit of space wastage in terms of the available 8 pages of the main paper, which could probably have been better utilized to motivate the approach, datasets and analysis more.\n- Anonymous GitHub links could have been provided to help with the reproducibility of the results.\n- In the first page, after the name of the paper, it says \"Anonymous ACL submission\" which should be \"Anonymous EMNLP submission\" but I am not sure if that makes any difference?\n- Overall, there are some issues with writing style in the paper that I feel can be resolved with another round of proofreading."}}, {"review_id": "zBRAenCKaH", "reviewer": "Reviewer_vnEh", "summary": "", "questions": "1. Can you consider more AD approaches, why not only selecting isoforest, oc-svm, cvdd or date?  Can you explain why these methods fit the AD NLP tasks? \n\n2. The evaluation metric, AUROC is used, how about other metrics, e.g., F1 score, can you explain why do you prefer to use AUROC only?  \n\n3. Not sure if this work is the first AD benchmark for NLP, can you explain the difference between this work and existing baselines?\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We will try to address their questions:\n\nQ1: The topic is very wide but the context seems to be weak to claim such a topic. The datasets, either from existing work or newly created collections are still not so strong to cover the major AD tasks and applications.\n\nA1: Our purpose for this benchmark was to propose NLP several setups that provide a natural anomaly detection problem, not an artificially constructed one (e.g. classic text categorization). For example, we propose the following setup for the COLA dataset: we consider a non-grammatically sound sentences as anomalies. In the case of VUA, we consider the metaphoric use of words being the non-regular of employing said words in speech or writing. In the case of Gutenberg Authors, we consider that authorship is a natural problem of anomaly detection: in our setting we consider one author as the main class and we want to distinguish the rest of the texts from his style, said texts being an anomaly from the point of view of his writing style.\n\n---\n\nQ2: In addition, the evaluation metrics are not comprehensive to compare the pros/cons of these tasks. The AD methods are very limited, only four methods are provided. From the results, we observe that most of the best performance is from the DATE method. Not sure how strong about the other baselines, e.g., isoforest, oc-svm, which are very classific AD approaches.\n\nA2: While we agree with the reviewer that the evaluation metrics were limited, we decided to use AUROC as we consider it the best metric for unbalanced datasets. We also provide extensive AUPR-In and AUPR-Out scores in our Appendix, alongside qualitative figures for CVDD and DATE.\n\nOur methodology is limited due to the fact that anomaly detection in text is not a widely approached subject, thus unexplored. We believe this is an important area of NLP and hope that our benchmark helps the community develop a stronger methodology towards this area or research. The limited analysis is due to the novelty of the problem: we try to categorise possible types of anomalies not apriori (artificially constructed \"toy\" dataset on which all models would work), but aposteriori, by running the models and reviewing their performance on each class.\n\n---\n\nQ3: Not sure if this work is the first AD benchmark for NLP, can you explain the difference between this work and existing baselines?\n\nA3: Regarding existing baselines, we've explored the subject in the \"Related Work\" section of our paper, where we have reported our findings: several frameworks and benchmarks for multitask learning in NLP, but none focused specifically on anomaly detection. \n\nAs stated in the answer for the reviewer's first point, our purpose to provide NLP problems that contain a natural anomaly detection setup. Apart from the pre-existing dataset (20NG, AGNews, COLA, VUA), we introduced 3 novel anomaly detection datasets: Gutenberg Authors, Gutengerg Categories and Song Lyrics. \n\nWe introduced Gutenberg Authors as an authorship anomaly detection problem, which is still an open problem in NLP. Gutenberg Categories, by comparison, presents more similarities with the other topic detection problems (e.g. 20Newsgroups, AG News), but offers a higher degree of diversity. Same goes for the Song Lyrics dataset. Our purpose with the latter two was to introduce diversity and complexity to the topic anomaly detection field.\n\nWe once again thank the reviewer for their insightful feedback and we hope that all of their raised concerns have been addressed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic is critical to NLP and few studies were provided before. The contribution, specifically.  dataset collection needs much effort.  Broader audience will be interested in this AD benchmark. ", "reasons_to_reject": "The topic is very wide but the context seems to be weak to claim such a topic.  The datasets, either from existing work or newly created collections are still not so strong to cover the major AD tasks and applications. \n\nIn addition, the evaluation metrics are not comprehensive to compare the pros/cons of these tasks.  The AD methods are very limited, only four methods are provided. From the results, we observe that most of the best performance is from the DATE method. Not sure how strong about the other baselines, e.g., isoforest, oc-svm, which are very classific AD approaches. ", "questions_to_authors": "1. Can you consider more AD approaches, why not only selecting isoforest, oc-svm, cvdd or date?  Can you explain why these methods fit the AD NLP tasks? \n\n2. The evaluation metric, AUROC is used, how about other metrics, e.g., F1 score, can you explain why do you prefer to use AUROC only?  \n\n3. Not sure if this work is the first AD benchmark for NLP, can you explain the difference between this work and existing baselines?\n", "paper_topic_and_main_contributions": "This paper provides a unified benchmark for anomaly detection in NLP tasks, ranging from syntax to stylistics.  They also build comprehensive datasets, and evaluation metrics and offer several methods for performing anomaly detection. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No ", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well-written and easy to follow. No major concern about the writting. ", "Justification_for_Ethical_Concerns": "No "}, "extra_text": {}}, {"review_id": "zcTls66Q7S", "reviewer": "Reviewer_s43E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We will try to address their questions:\n\nQ1: The authors conducted benchmarking on two classical methods, OCSVM and IForest, which are typically considered unsupervised methods. However, as pointed out by Aggarwal 2017, a common mistake in outlier detection benchmarking arises when algorithms depend on user-defined parameters. It is often observed that the algorithms are repeatedly run to select the best parameter, optimizing the ROC AUC score. Unfortunately, such an approach is not acceptable in outlier detection since it effectively incorporates knowledge of the outlier labels in parameter selection, thereby deviating from the unsupervised nature of the task. In the context of outlier detection, the proper way to benchmark a pair of algorithms is by evaluating both over a \"reasonable\" range of parameters and comparing their performance using a central estimator derived from the resulting runs. Notably, the benchmarking work in anomaly detection usually adheres to this principle. For instance, Han et al. 2022 employed default hyperparameters in their study.\n\nA1: While we agree with the reviewer that hyperparameter selection for unsupervised anomaly detection can lead to an overly-optimistic anomaly metric, we would like to argue that it is acceptable for the classical methods (isolation forest, one-class svm) in this case, since our purpose was highlighting the best possible results for them.\n\nNevertheless, for the neural methods, we don't search for hyperparameters and we use a single set of hyperparameters for all datasets (as mentioned in Sec. 4.1, lines 400-401). Moreover, the neural methods obtain the best results in most of our experiments, therefore we believe that keeping the most optimistic results for the classical baselines is an acceptable approach in this scenatio.\n\nWe thank the reviewer for raising this concern, we will update our manuscript with a detailed explanation regarding our hyperparameter selection decisions.\n\n--- \n\nQ2: The claim made about grammatically unacceptable phrases (COLA dataset) based on the AUROC score 49.9% is inaccurate . An AUROC score of 50% is equivalent to random guessing, and a score below 50% indicates that the model's performance is worse than random. Therefore, the claim that grammatically unacceptable phrases represent an anomaly based on this low AUROC score is not supported. In fact, it suggests that the models are incapable of effectively distinguishing between the classes, and grammatically unacceptable phrases are not being detected as anomalies.\n\nA2: While we agree that the results on COLA are just slightly better than random guessing (53.3 AUROC for CVDD), we note that the competing methods have worse than random performance on the dataset, and we believe that having syntactic anomalies for the linguistics domain can be an interesting problem for further work which can employ better inductive biases for this specific scenario.\n\n---\n\nQ3: For classical methods, it is essential to specify the text representation techniques used in anomaly detection. The way text data is represented significantly impacts the anomaly score calculation and, ultimately, the performance of the anomaly detection method. Providing details about the text representation techniques will contribute to a better understanding of the anomaly detection process for each type of anomaly.\n\nA3: We thank the reviewer for highlighting this issue. For the classical methods, we have used either FastText or GloVe as a feature extraction backbone. Due to space limitations we omitted to report that, but we will update the paper accordingly or we will include this information in the Appendix for the camera-ready version.\n\n---\n\nQ4: The anomaly ratio in the dataset may have a significant impact on the performance of anomaly detection methods. It is crucial to specify the anomaly ratio for each corpus and each split to provide a comprehensive understanding of the experimental setup and the results obtained.\n\nA4: We have followed the Anomaly Detection setup from [1], [2], in which we pick a class as our inlier dataset, while all of the other classes are treated as outliers. This, indeed, makes the resulting test sets imbalanced. However, we would like to note that our primary metric is AUROC (as opposite to the AUPR metric reported in the Supplementary materials), which is robust to dataset imbalances. Nevertheless, we will report the inlier-outlier ratio for the camera-ready version of the paper.\n\nWe once again thank the reviewer for their insightful feedback and we hope that all of their raised concerns have been addressed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written. The topic aligns well with the conference theme and the track, and it is evident that considerable effort has been invested in the research. The article covers text anomalies at various linguistic levels, including syntax, semantics, pragmatics, and style. This inclusive approach provides a comprehensive understanding of anomaly detection in natural language processing.\nIn addition to widely used corpora like AGNews and 20Newsgroups, the authors have curated three new datasets:\nGutenberg Categories, Song Genres, and Gutenberg Authors. The incorporation of these datasets showcases the authors' commitment\nto enhancing the benchmark's diversity and complexity.", "reasons_to_reject": "The paper presents a few methodological weaknesses that should be addressed.\n\n1. The authors conducted benchmarking on two classical methods, OCSVM and IForest, which are typically considered unsupervised methods. However, as pointed out by Aggarwal 2017, a common mistake in outlier detection benchmarking arises when algorithms depend on user-defined parameters. It is often observed that the algorithms are repeatedly run to select the best parameter, optimizing the ROC AUC score. Unfortunately, such an approach is not acceptable in outlier detection since it effectively incorporates knowledge of the outlier\nlabels in parameter selection, thereby deviating from the unsupervised nature of the task. In the context of outlier detection, the proper way\nto benchmark a pair of algorithms is by evaluating both over a \"reasonable\" range of parameters and comparing their performance\nusing a central estimator derived from the resulting runs. Notably, the benchmarking work in anomaly detection usually adheres to this\nprinciple. For instance, Han et al. 2022 employed default hyperparameters in their study.\n\nAggarwal, C. C. 2017. An introduction to outlier analysis (pp. 134.Springer International Publishing.\nHan, S., Hu, X., Huang, H., Jiang, M., & Zhao, Y. 2022. Adbench: Anomaly detection benchmark. Advances in Neural Information Processing Systems, 35, 3214232159.\n\n2) The claim made about grammatically unacceptable phrases (COLA dataset) based on the AUROC score 49.9% is inaccurate . An AUROC score of 50% is equivalent to random guessing, and a score below 50%  indicates that the model's performance is worse than random. Therefore, the claim that grammatically unacceptable phrases represent an anomaly based on this low AUROC score is not\nsupported. In fact, it suggests that the models are incapable of effectively distinguishing between the classes, and grammatically\nunacceptable phrases are not being detected as anomalies.\n\nMoreover, additional information need to be clarified before publication:\n\n3) For classical methods, it is essential to specify the text representation techniques used in anomaly detection. The way text data is represented significantly impacts the anomaly score calculation and, ultimately, the performance of the anomaly detection method. Providing details about the text representation techniques will contribute to a better understanding of\nthe anomaly detection process for each type of anomaly.\n\n4) The anomaly ratio in the dataset may have a significant impact on the performance of anomaly detection methods. It is crucial\nto specify the anomaly ratio for each corpus and each split to provide a comprehensive understanding of the experimental setup and the\nresults obtained.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The article presents a benchmarking study for anomaly detection in text data. It covers text anomalies at various linguistic levels (syntactic,\nsemantic, pragmatic, and stylistic) and explores different complexity levels of anomalies. The authors curated three new datasets in addition to the existing corpus. Classical and deep learning methods are evaluated on the benchmark.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oO8HNMxO0T", "reviewer": "Reviewer_UgrG", "summary": "", "questions": "Can we summarize the relationship between the characteristics of a method and which type of task it is applicable to, based on existing research results?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We will try to address their questions:\n\nQ1: About effectiveness. For the mentioned four aspects, there are four datasets following the same aspect \"semantic\", and each other aspect has a dataset. So, is it really enough to evaluate the effectiveness of a newly-proposed model? Especially for your newly-proposed dataset about Stylistic?\n\nA1: Our purpose for this benchmark was to propose NLP several setups that provide a natural anomaly detection problem, not an artificially constructed one (e.g. classic text categorization). For example, we propose the following setup for the COLA dataset: we consider a non-grammatically sound sentences as anomalies. In the case of VUA, we consider the metaphoric use of words being the non-regular of employing said words in speech or writing. In the case of Gutenberg Authors, we consider that authorship is a natural problem of anomaly detection: in our setting we consider one author as the main class and we want to distinguish the rest of the texts from his style, said texts being an anomaly from the point of view of his writing style.\n\n\n\n---\n\nQ2: I don't know whether the scale of these datasets is sufficient to form a benchmark. For me, I don't think it's enough, but I will I will draw on the views of other reviewers.\n\nA2: Regarding the scale of the datasets, it reflects the natural distribution of the data in the wild (online sources). The problems are inherently complex due to fact that there is a scarcity of samples to start from.\n\n---\n\nQ3: Inconsistent statements: Semantic in Table 1 but Content in Table 2\n\nA3: Thank you, we will fix this in the camera-ready version.\n\n---\n\nQ4: Not strictly controlling the number of pages\n\nA4: Thank you, we will reorganize the paper such that it strictly follows the size guideline for the camera-ready version.\n\n---\n\nR5: Can we summarize the relationship between the characteristics of a method and which type of task it is applicable to, based on existing research results?\n\nA5: During our experiments and analysis we observed no correlation between the performance of a model and the type of the task.\n\nWe once again thank the reviewer for their insightful feedback and we hope that all of their raised concerns have been addressed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "New benchmark and its datasets.", "reasons_to_reject": "- About effectiveness. For the mentioned four aspects, there are four datasets following the same aspect \"semantic\", and each other aspect has a dataset. So, is it really enough to evaluate the effectiveness of a newly-proposed model? Especially for your newly-proposed dataset about Stylistic?\n- I don't know whether the scale of these datasets is sufficient to form a benchmark. For me, I don't think it's enough, but I will I will draw on the views of other reviewers.\n\nAlso,\n- Inconsistent statements: Semantic in Table 1 but Content in Table 2\n- Not strictly controlling the number of pages", "questions_to_authors": "Can we summarize the relationship between the characteristics of a method and which type of task it is applicable to, based on existing research results?", "paper_topic_and_main_contributions": "This paper established a benchmark on anomaly detection in natural language processing, based on 4 existing datasets and 3 newly proposed ones. The authors introduce that the anomaly detection in NLP involve 4 aspects: Syntactic, Semantic, Pragmatic and Stylistic. Also, they evaluate four baselines on these 7 datasets.\n\nOverall, I acknowledge the authors' contributions. \nThey established four new datasets.\nEspecially, there is one dataset that may be the first benchmark about Stylistic in NLP.\nFor these datasets, they made lots of experiments.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new benchmark for evaluating anomaly detection methods in natural language processing, called AD-NLP. This benchmark includes 4 existing datasets and adds 3 new datasets. All the data are organized following a classification of anomality (syntactic, semantic, pragmatic, and stylistic). Moreover different methods are evaluated on this benchmark. The benchmark will be made available.

It would be great to better present the results in the paper instead of in appendix. Also more explanation and information are need at some point (choice of hyperparameters for the NN methods and representations methods).
Decision: Accept-Main