SlowGuess/NAACL_2021_OCR huggingface dataset:
Abl.	Method	Variation	AUROC(%)
CVDD	best	83.1 ± 4.4
OCSVM	best	84.0 ± 5.0
ELECTRA	adapted for AD	84.6 ± 4.5
DATE	(Ours)	90.0 ± 4.2
A.	Anomaly score	MP	72.4 ± 3.7
NE	73.1 ± 3.9
B.	Generator	small	89.3 ± 4.2
large	89.8 ± 4.4
C.	Loss func	RTD only	89.4 ± 4.4
RMD only	85.9 ± 4.1
D.	Masking patterns	5 masks	87.5 ± 4.5
10 masks	89.2 ± 4.3
25 masks	89.8 ± 4.3
100 masks	89.8 ± 4.3
E.	Mask percent	15%	89.5 ± 4.1
25%	89.5 ± 4.1
Table 1: Ablation study. We show results for the competition and report ablation experiments which are only one change away from our best DATE configuration: A.  $PL_{RTD}$ ; B. Rand C. RTD + RMD; D. 50 masks; E.  $50\%$ . For the ELECTRA line, we use: A.  $PL_{RTD}$ ; B. Rand; C. RTD only; D. Unlimited; E.  $15\%$ . A. The Anomaly Score used over classification probabilities shows that  $PL_{RTD}$  (used in DATE) is the best in predicting anomalies, meaning that our self-supervised classification task is well defined, with few ambiguous samples; B. A learned Generator does not justify its training cost; C. RMD Loss proved to be complementary with RTD Loss, their combination (in DATE) increasing the score and stabilizes the training; D+E.
inator. PL is an ideal score if the self-supervised task manages to build and learn well separated classes. The way we formulate our mask prediction task enables a very good class separation, as theoretically proved in detail in the supplementary material A. Therefore,  $PL_{RTD}$  proves to be significantly better in detecting the anomalies compared with MP and NE metrics, which try to compensate for ambiguous samples.
B. Generator performance. We tested the importance of having a learned generator, by using a one-layer Transformer with hidden size 16 (small) or 64 (large). The random generator proved to be better than both parameterized generators.
C. Loss function. For the final loss, we combined RTD (which sanctions the prediction per token)
$^{2}$ http://qwone.com/~jason/20Newsgroups/ $^{3}$ http://groups.di.unipi.it/~gulli/AG Corpus_of_news_articles.html
with our RMD (which enforces the detection of the mask applied on the entire sequence). We also train our model with RTD or RMD only, obtaining weaker results. This proves that combining losses with supervisions at different scales (locally: token-level and globally: sequence-level) improves AD performance. Moreover, when using only the RTD loss, the training can be very unstable (AUROC score peaks in the early stages, followed by a steep decrease). With the combined loss, the AUROC is only stationary or increases with time.
D. Masking patterns. The mask patterns are the root of our task formulation, hiding a part of the input tokens and asking the discriminator to classify them. As experimentally shown, having more mask patterns is better, encouraging increased expressiveness in the embeddings. Too many masks on the other hand can make the task too difficult for the discriminator and our ablation shows that having more masks does not add any benefit after a point. We validate the percentage of masked tokens in E. Mask percent ablation.
4.3 Comparison with other AD methods
We compare our method against classical AD baselines like Isolation Forest (Liu et al., 2008) and existing state-of-the-art OneClassSVMs (Schölkopf et al., 2001b) and CVDD (Ruff et al., 2019). We outperform all previously reported performances on all 20Newsgroups splits by a large margin:  $13.5\%$  over the best reported CVDD and  $11.7\%$  over the best OCSVM, as shown in Tab. 2. In contrast, DATE uses the same set of hyper-parameters for a dataset, for all splits. For a proper comparison, we keep the same experimental setup as the one introduced in (Ruff et al., 2019).
Isolation Forest. We apply it over fastText or Glove embeddings, varying the number of estimators (64, 100, 128, 256), and choosing the best model per split. In the unsupervised AD setup, we manually set the percent of outliers in the train set.
OCSVM. We use the One-Class SVM model implemented in the CVDD work  $\ddagger$ . For each split, we choose the best configuration (fastText vs Glove, rbf vs linear kernel,  $\nu \in [0.05, 0.1, 0.2, 0.5]$ ).
CVDD. This model (Ruff et al., 2019) is the current state-of-the-art solution for AD on text. For each split, we chose the best column out of all reported context sizes  $(r)$ . The scores reported using the  $c^*$  context vector depends on the ground
Inlier class	IsoForest best	OCSVM best	CVDD best	DATE (Ours)
20 News	comp	66.1	78.0	74.0	92.1
rec	59.4	70.0	60.6	83.4
sci	57.8	64.2	58.2	69.7
misc	62.4	62.1	75.7	86.0
pol	65.3	76.1	71.5	81.9
rel	71.4	78.9	78.1	86.1
AG News	business	79.6	79.9	84.0‡	90.0
sci	76.9	80.7	79.0‡	84.0
sports	84.7	92.4	89.9‡	95.9
world	73.2	83.2	79.6‡	90.1
Table 2: Semi-supervised performance (AUROC%). We test on the 20Newsgroups and AG News datasets, by comparing DATE against several strong baselines and state-of-the-art solutions (with multiple variations, choosing the best score per split as detailed in Sec. 4.3): IsoForest, OCSVM, and CVDD. We largely outperform competitors with an average improvement of  $13.5\%$  on 20Newsgroups and  $6.9\%$  on AG News compared with the next best solution. Note that DATE uses the same set of hyper-parameters per dataset.
truth and it only reveals "the potential of contextual anomaly detection", as the authors mention.
4.4 Unsupervised AD
We further analyse how our algorithm works in a fully unsupervised scenario, namely when the training set contains some anomalous samples (which we treat as normal ones). By definition, the quantity of anomalous events in the training set is significantly lower than the normal ones. In this experiment, we show how our algorithm performance is influenced by the percentage of anomalies in training data. Our method proves to be extremely robust, surpassing state-of-the-art, which is a semi-supervised solution, trained over a clean dataset (with  $0\%$  anomalies), even at  $10\%$  contamination, with  $+0.9\%$  in AUROC (see Fig. 3). By achieving an outstanding performance in the unsupervised setting, we make unsupervised AD in text competitive against other semi-supervised methods. The reported scores are the mean over all AG News splits. We compare against the same methods presented in Sec. 4.3.
$^{\ddagger}$ Experiments done using the CVDD published code https://github.com/lukasruff/CVDD-PyTorch.
Figure 3: Unsupervised AD. We test the performance of our method when training on impure data, which contains anomalies in various percentages:  $0\% -15\%$ . The performance slowly decreases when we increase the anomaly percentage, but even at  $10\%$  contamination, it is still better than state-of-the-art results on self-supervised anomaly detection in text (Ruff et al., 2019), which trains on  $0\%$  anomalous data, proving the robustness of our method. Experiments were done on all AG News splits.
Inlier	Label	Pred	Sample (BERT tokens)
Sports	Outlier (World)	Outlier	jail ##ing democrat china politically motivated af ##p af ### p hong kong democrats accused china jail ###ing one members trump ###ed prostitution charges bid disgrace political movement beijing feud ###ing seven years
Sci	Outlier (World)	Outlier	panama flooding kills nine people least nine people seven children died flooding capital banana authorities say least people still missing heavy rainfall caused rivers break banks
Business	Inlier	Inlier	motorola cut jobs new york reuters telecommunications equipment maker motorola inc hr ###ef http www investor reuters com full ###qu ###ote as ###p ###x tick ###er mo ###t target stocks quick ###in ###fo full ###qu ###ote mo ###t said tuesday would cut jobs take related charges million focus wireless business
We show in Fig. 4 how DATE performs in identifying anomalies in several examples. Each token is colored based on its PL score.
Separating anomalies. We show how our anomaly score (PL) is distributed among normal vs abnormal samples. For visualization, we chose two splits from AG News and report the scores from the beginning of the training to the end. We see in Fig. 5 that, even though at the beginning, the outliers' distribution of scores fully overlaps with
Figure 4: Qualitative examples. Lower scores are shown in a more intense red, and point to anomalies. In the  $1^{st}$  example, words from politics are flagged as anomalous for sports. In the  $2^{nd}$  one, words describing natural events are outliers for technology. In the  $3^{rd}$  row, while few words have higher anomaly potential for the business domain, most of them are appropriate.
Figure 5: Normalized histogram for anomaly score. We see how the anomaly score (PL) distribution varies among inliers and outliers, from the beginning of the training  $(1^{st}$  column) to the end  $(2^{nd}$  column), where the two become well separated, with relatively low interference between classes. Note that a better separation is correlated with high performance  $(1^{st}$  line split has  $95.9\%$  AUROC, while the  $2^{nd}$  has only  $90.1\%$ .
the inliers, at the end of training the two are well separated, proving the effectiveness of our method.
We propose DATE, a model for tackling Anomaly Detection in Text, and formulate an innovative self-supervised task, based on masking parts of the initial input and predicting which mask pattern was used. After masking, a generator reconstructs the initially masked tokens and the discriminator predicts which mask was used. We optimize a loss composed of both token and sequence-level parts, taking advantage of powerful supervision, coming from two independent pathways, which stabilizes learning and improves AD performance. For computing the anomaly score, we alleviate the burden of aggregating predictions from multiple transformations by introducing an efficient variant of the Pseudo Label score, which is applied per token, only on the original input. We show that this score separates very well the abnormal entries from normal ones, leading DATE to outperform state-of-the-art results on all AD splits from 20Newsgroups and AG News datasets, by a large margin, both in the semi-supervised and unsupervised AD settings.
Acknowledgments. This work has been supported in part by UEFISCDI, under Project PN-III-P2-2.1-PTE-2019-0532.
Charu C Aggarwal. 2015. Outlier analysis. In Data mining, pages 237-263. Springer.
Lalu Banoth, Mnr Teja, M Saicharan, and Naveen Chandra. 2017. A survey of data mining and machine learning methods for cyber security intrusion detection. International Journal of Research, 4:406-412.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In EMNLP/IJCNLP.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3(null):993-1022.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3):1-58.
Jinghui Chen, Saket Sathe, Charu C. Aggarwal, and Deepak S. Turaga. 2017. Outlier detection with autoencoder ensembles. In SDM.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pretraining text encoders as discriminators rather than generators. ArXiv, abs/2003.10555.
Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. Bertje: A dutch bert model. ArXiv, abs/1912.09582.
L. Deng. 2012. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29:141-142.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
J. Dorronsoro, Francisco Ginel, C. Sanchez, and C. Cruz. 1997. Neural fraud detection in credit card operations. IEEE transactions on neural networks, 84:827-34.
Izhak Golan and Ran El-Yaniv. 2018. Deep anomaly detection using geometric transformations. CoRR, abs/1805.10917.
Douglas M Hawkins. 1980. Identification of outliers, volume 11. Springer.
Simon Hawkins, Hongxing He, Graham J. Williams, and Rohan A. Baxter. 2002. Outlier detection using replicator neural networks. In Proceedings of the 4th International Conference on Data Warehousing and Knowledge Discovery, DaWaK 2000, page 170-180, Berlin, Heidelberg. Springer-Verlag.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers.
Klaus Kammerer, Burkhard Hoppenstedt, Rüdiger Pryss, Steffen Stökler, Johannes Allgaier, and M. Reichert. 2019. Anomaly detections for manufacturing systems based on sensor data—insights into two challenging real-world production settings. Sensors (Basel, Switzerland), 19.
Ramakrishnan Kannan, Hyenkyun Woo, Charu C. Aggarwal, and Haesun Park. 2017. Outlier detection for text data: An extended version. CoRR, abs/1701.01325.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, D. Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008. Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining, pages 413-422. IEEE.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR 2019.
Amogh Mahapatra, Nisheeth Srivastava, and Jaideep Srivastava. 2012. Contextual anomaly detection in text data. Algorithms, 5(4):469-489.
L. Manevitz and M. Yousef. 2001. One-class svms for document classification. J. Mach. Learn. Res., 2:139-154.
L. Manevitz and M. Yousef. 2007. One-class document classification via neural networks. Neurocomputing, 70:1466-1481.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, 'Eric de la Clergerie, Djamé Seddah, and Benoit Sagot. 2020. Camembert: a tasty french language model. ArXiv, abs/1911.03894.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Lukas Ruff, Yury Zemlyanskiy, Robert Vandermeulen, Thomas Schnake, and Marius Kloft. 2019. Self-attentive, multi-context one-class classification for unsupervised anomaly detection on text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4061–4071, Florence, Italy. Association for Computational Linguistics.
Mayu Sakurada and Takehisa Yairi. 2014. Anomaly detection using autoencoders with nonlinear dimensionality reduction. In Proceedings of the MLSDA 2014 2nd Workshop on Machine Learning for Sensory Data Analysis, MLSDA'14, page 4-11, New York, NY, USA. Association for Computing Machinery.
Thomas Schlegl, Philipp Seebock, Sebastian M. Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. 2017. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In IPMI.
B. Scholkopf, John C. Platt, J. Shawe-Taylor, A. Smola, and R. Williamson. 2001a. Estimating the support of a high-dimensional distribution. Neural Computation, 13:1443-1471.
Bernhard Scholkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. 2001b. Estimating the support of a high-dimensional distribution. Neural computation, 13(7):1443-1471.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.
S. Wang, Y. Zeng, Xinwang Liu, En Zhu, Jianping Yin, Chuanfu Xu, and M. Kloft. 2019. Effective end-to-end unsupervised outlier detection via inlier priority of discriminative network. In NeurIPS.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NeurIPS.
A Disjoint patterns analysis
We start from two observations regarding the performance of DATE, our Anomaly Detection algorithm. First, a discriminative task performs better if the classes are well separated (Deng, 2012) and there is a low probability for confusions. Second, the PL score for anomalies achieves best performance when the probability distribution for its input is clearly separated. Intuitively, for three classes, PL([0.9, 0.05, 0.05]) is better than PL([0.5, 0.3, 0.2]) because it allows PL to give either near 1 score if the class is correct, either near 0 score if it is not, avoiding the zone in the middle where we depend on a well chosen threshold.
Since the separation between the mask patterns greatly influences our final performance, we next analyze our AD task from the mask pattern generation point of view. Ideally, we want to have a sense of how disjoint our randomly sampled patterns are and make an informed choice for the pattern generation hyper-parameters.
First, we start by computing an upper bound for the probability of having two patterns with at least  $p$  common masked points. We have  $\binom{S}{M}$  patterns, where  $S$  is the sequence length and  $M$  is the number of masked tokens. We fix the first  $p$  positions that we want to mask in any pattern. Considering those fixed masks, the probability of having a sequence with  $M$  masked tokens, with  $p$  tokens in the first positions is  $r$ :
$$
r = \frac {\binom {S - p} {M - p}}{\binom {S} {M}}. \tag {8}
$$
Next, the probability that two sequences mask the first  $p$  tokens is  $r^2$ . But we can choose those two positions in a  $\binom{S}{p}$  ways. So the probability that any two sequences have at least  $p$  common masked tokens is lower than  $UB_2$ :
$$
U B _ {2} = \binom {S} {p} r ^ {2} \tag {9}
$$
Next, out of our generated patterns, we sample  $N$  masks, so the probability becomes less than the
upper bound  $UB_{N}$
$$
\begin{array}{l} U B _ {N} = \binom {N} {2} U B _ {2} = \binom {N} {2} \binom {S} {p} r ^ {2} \\ = \binom {N} {2} \binom {S} {p} \left(\frac {\binom {S - p} {M - p}}{\binom {S} {M}}\right) ^ {2}. \tag {10} \\ \end{array}
$$
In our experiments, the sequence length is  $S = 128$  and we chose the number of masked tokens to be between  $15\%$  and  $50\%$  ( $M$  between 19 and 64). We consider that two patterns are disjoint when they have less than  $p$  masked tokens in common, for  $N$  sampled patterns.
The probability that any two patterns collide (have more than  $p$  masked tokens in common) is very low. We compute several values for its upper bound:  $UB_{N = 100,p = 12} = 5e - 4$ ,  $UB_{N = 100,p = 15} = 1e - 9$ ,  $UB_{N = 10,p = 15} = 1e - 11$ ,  $UB_{N = 10,p = 13} = 1e - 7$ .
In conclusion, for our specific setup, the probability for two masks to largely overlap (large  $p$  compared with  $S$ ) is extremely small, ensuring us a good performance in the discriminator. We take advantage of this property of our pretext task by combining the discriminator output probabilities with the PL score.
B Model implementation
We add next more details on the implementation of the modules: from the ablation experiments in Tab. 1, Generator (small): 1 Transformer layer, with 4 self-attention heads, token and positional embeddings of size 128, hidden layer of size 16, feedforward layer of sizes 1024 and 16; Generator (large): 1 Transformer layer, with 4 self-attention heads, token and positional embeddings of size 128, hidden layer of size 64, feedforward layer of sizes 1024 and 64; As empirical experiments showed us, we choose a random Generator (samples were drawn from a uniform distribution over the vocabulary) in our final model. Discriminator: 4 Transformer layers, each with 4 self-attention heads, hidden layers of size 256, feedforward layers of sizes of 1024 and 256, 128-dimensional token and positional embeddings, which are tied with the generator. For other unspecified hyper-parameters we use the ones in ELECTRA-Small model. Prediction Heads: both heads have 2 linear layers separated by a non-linearity, ending in a classification. Loss weights: We set the RTD  $\lambda$  weight to 50 as in (Clark et al., 2020), and the RMD  $\mu$  weight to 100.
Inlier	Label	Pred	Sample (BERT tokens)
Sports	Outlier (Business)	Outlier	airways watch bankruptcy court judge stephen mitchell hear arguments today asking rec ##ons ##ider four month percent pay cut imposed many union # #ized workers last month
Sci	Outlier (World)	Outlier	bush defend ##s decision invade iraq president bush went skeptical hall world leaders tuesday mount vigorous defense war iraq telling united nations iraqi people
Business	Outlier (Sci)	Outlier	nokia nec test new multimedia sub # #sy ##ste # #m two high tech communications players completed first phase series tests show next generation data communications infrastructure works
World	Outlier (Sports)	Outlier	man plan athens four years ago sydney gymnast #s gone medal free olympics first time years federation president bob cola # #ross # #i sitting table explaining turn #around already begun women moved sixth fourth world one year men sixth fifth
Sports	Inlier	Inlier	dolphins steelers play sunday night reuters reuters miami dolphins pittsburgh steelers play scheduled game sunday night [SEP]
Sci	Inlier	Inlier	microsoft launch new search engine software giant microsoft corp decided release beta versions updated ms # #n search engine earlier today company hopes compete leading search engines google yahoo [SEP]
Figure 6: More qualitative examples.
Subset	business	sci	sports	world
AUPR-in	74.8	62.4	88.8	81.9
AUPR-out	96.1	93.5	98.5	95.5
Table 3: We report AUPR metric for AG News splits, on inliers and outliers since this is a more relevant metric for unbalanced classes (which is the case for all splits in text AD, as explained in Anomalies setup).
C More qualitative and quantitative Results
In Fig. 6 we show more qualitative results, trained on different inliers. To encourage further more detailed comparisons, we report the AUPR metric on AG News for inliers and outliers (see Tab. 3). When all the other metrics are almost saturated, we notice that AUPR-in better captures the performance on a certain split.