SlowGuess/ACL_2020_OCR huggingface dataset:
Nastaran Babanejad, Ameeta Agrawal, Aijun An, Manos Papagelis
York University, Toronto, Canada
{nasba, ameeta, aan, papaggel}@eecs.yorku.ca
Affective tasks such as sentiment analysis, emotion classification and sarcasm detection have been popular in recent years due to abundance of user-generated data, accurate computational linguistic models, and broad range of relevant applications in various domains. At the same time, many studies have highlighted the importance of text preprocessing, as an integral step to any natural language processing prediction model and downstream task. While preprocessing in affective systems is well-studied, preprocessing in word vector based models applied to affective systems, is not. To address this limitation, we conduct a comprehensive analysis of the role of preprocessing techniques in affective analysis based on word vector models. Our analysis is the first of its kind and provides useful insights of the importance of each preprocessing technique when applied at the training phase, commonly ignored in pretrained word vector models, and/or at the downstream task phase.
Affective tasks such as sentiment analysis, emotion classification and sarcasm detection have enjoyed great popularity in recent years. This success can be largely attributed to the fundamental and straightforward nature of the methods employed, the availability of vast amounts of user-generated natural language data, and the wide range of useful applications, spanning from hate speech detection to monitoring the sentiment of financial markets and news recommendation (Djuric et al., 2015; Babanejad et al., 2019). Most early models of affect analysis employed pretrained word embeddings that have been obtained under the assumption of the distributional hypothesis (Mikolov et al., 2013; Devlin et al., 2018). The distributional hypothesis suggests that two words occurring frequently in
similar linguistic contexts tend to be more semantically similar, and therefore should be represented closer to one another in the embedding space. However, while such embeddings are useful for several natural language processing (NLP) downstream tasks, they are known to be less suitable for affective tasks in particular (Tang et al., 2014; Agrawal et al., 2018). Although some authors claim that there is a need for post-processing word embeddings for affective tasks, others find that off-the-shelf vectors are very powerful for affective lexicon learning (Lison and Kutuzov, 2017). For example, word2vec (Mikolov et al., 2013) estimates the pair of words 'happy' and 'sad' to be more similar than the pair of words 'happy' and 'joy', which is counterintuitive, and might affect the accuracy performance of the models that depend on it.
To address the limitations of traditional word embeddings, several techniques have been proposed, including task-specific fine-tuning (Devlin et al., 2018), retrofitting (Faruqui et al., 2014), representing emotion with vectors using a multi-task training framework (Xu et al., 2018) and generating affective word embeddings (Felbo et al., 2017), to name a few. Other attempts to overcome the limitation of word vectors include optimization of hyperparameters (Levy et al., 2015), as well as fine-tuned preprocessing strategies tailored to different NLP tasks. While these strategies have demonstrated evidence of improving the accuracy performance in tasks such as word similarity, word analogy, and others (Lison and Kutuzov, 2017), their effect in affective tasks has not received considerable attention and remains less explored. Our work is motivated by the observation that preprocessing factors such as stemming, stopwords removal and many others make up an integral part of nearly every improved text classification model, and affective systems in particular (Danisman and Alpkocak, 2008; Patil and Patil, 2013). However, little work has been
5799
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5799-5810 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Framework of applying preprocessing in different stages in affective systems; (a) Pre, (b) Post.
done towards understanding the role of preprocessing techniques applied to word embeddings in different stages of affective systems. To address this limitation, the overarching goal of this research, is to perform an extensive and systematic assessment of the effect of a range of linguistic preprocessing factors pertaining to three affective tasks, including sentiment analysis, emotion classification and sarcasm detection. Towards that end, we systematically analyze the effectiveness of applying preprocessing to large training corpora before learning word embeddings, an approach that has largely been overlooked by the community. We investigate the following research questions: (i) what is the effect of integrating preprocessing techniques earlier into word embedding models, instead of later on in a downstream classification models? (ii) which preprocessing techniques yield the most benefit in affective tasks? (iii) does preprocessing of word embeddings provide any improvement over state-of-the-art pretrained word embeddings? and if yes, how much?
Figure 1 illustrates the difference between a) preprocessing word embeddings pipeline (Pre) vs. b) preprocessing classification dataset pipeline (Post), where preprocessing techniques in (a) are applied to the training corpus of the model and in (b) only to the classification dataset. In brief, the main contributions of our work are as follows:
- We conduct a comprehensive analysis of the role of preprocessing techniques in affective tasks (including sentiment analysis, emotion classification and sarcasm detection), employing different models, over nine datasets;
- We perform a comparative analysis of the accuracy performance of word vector models when preprocessing is applied at the training phase (training data) and/or at the downstream task phase (classification dataset). Interestingly, we obtain best results when preprocessing is applied only to the training corpus or when it is applied to both the training corpus
and the classification dataset of interest.
- We evaluate the performance of our best preprocessed word vector model against state-of-the-art pretrained word embedding models;
- We make source code and data publicly available to encourage reproducibility of results<sup>1</sup>.
The rest of the paper is organized as follows: Section 2 presents an overview of the related work. Section 3 elaborates on the preprocessing techniques employed in the evaluation of models. Section 4 describes the experimental evaluation framework. In Section 5 a comprehensive analysis of the results is provided. Section 6 concludes the paper with key insights of the research.
In this section, we present an overview of related work on preprocessing classification datasets and preprocessing word embeddings, and how our work aims to bridge the gap between those efforts.
2.1 Preprocessing Classification Datasets
Preprocessing is a vital step in text mining and therefore, evaluation of preprocessing techniques has long been a part of many affective systems. Saif et al. (2014) indicated that, despite its popular use in Twitter sentiment analysis, the use of precompiled stopwords has a negative impact on the classification performance. Angiani et al. (2016) analyzed various preprocessing methods such as stopwords removal, stemming, negation, emoticons, and so on, and found stemming to be most effective for the task of sentiment analysis. Similarly, Symeonidis et al. (2018) found that lemmatization increases accuracy. Jianqiang and Xiaolin (2017) observed that removing stopwords, numbers, and URLs can reduce noise but does not affect performance, whereas replacing negation and expanding acronyms can improve the classification accuracy.
1https://github.com/NastaranBa/preprocessing-for-word-representation
Preprocessing techniques such as punctuation and negation (Rose et al., 2018) or post-tagging and negation (Seal et al., 2020) make up a common component of many emotion classification models (Kim et al., 2018; Patil and Patil, 2013). One of the earliest works (Danisman and Alpkocak, 2008) preserved emotion words and negative verbs during stopwords removal, replaced punctuation with descriptive new words, replaced negative short forms with long forms, and concatenated negative words with emotion words to create new words (e.g., not happy  $\rightarrow$  NOThappy). Although stemming may remove the emotional meaning from some words, it has been shown to improve classification accuracy (Danisman and Alpkocak, 2008; Agrawal and An, 2012). Negations have also been found beneficial, whereas considering intensifiers and diminishers did not lead to any improvements (Strohm, 2017).
Pecar et al. (2018) also highlight the importance of preprocessing when using user-generated content, with emoticons processing being the most effective. Along the same lines, while Gratian and Haid (2018) found pos-tags to be useful, Boiy et al. (2007) ignored pos-tagging because of its effect of reducing the classification accuracy
The aforementioned works describe preprocessing techniques as applied directly to evaluation datasets in affective systems. In contrast, we examine the effectiveness of directly incorporating these known effective preprocessing techniques further "upstream" into the training corpus of word embeddings, which are widely used across a number of downstream tasks.
2.2 Preprocessing Word Embeddings
Through a series of extensive experiments, particularly those related to context window size and dimensionality, (Levy et al., 2015) indicate that seemingly minor variations can have a large impact on the success of word representation methods in similarity and analogy tasks, stressing the need for more analysis of often ignored preprocessing settings. Lison and Kutuzov (2017) also present a systematic analysis of context windows based on a set of four hyperparameters, including window position and stopwords removal, where the right window was found to be better than left for English similarity task, and stopwords removal substantially benefited analogy task but not similarity.
A general space of hyperparameters and preprocessing factors such as context window size (Her
shcovich et al., 2019; Melamud et al., 2016), dimensionality (Melamud et al., 2016), syntactic dependencies (Levy and Goldberg, 2014; Vulić et al., 2020) and their effect on NLP tasks including word similarity (Hershcovich et al., 2019), tagging, parsing, relatedness, and entailment (Hashimoto et al., 2017) and biomedical (Chiu et al., 2016) has been studied extensively in the literature. The main conclusion of these studies, however, is that these factors are heavily task-specific. Therefore, in this work we explore preprocessing factors of generating word embeddings specifically tailored to affective tasks, which have received little attention.
A recent study investigated the role of tokenizing, lemmatizing, lowercasing and multiword grouping (Camacho-Collados and Pilehvar, 2018) as applied to sentiment analysis and found simple tokenization to be generally adequate. In the task of emotion classification, Mulki et al. (2018) examined the role of four preprocessing techniques as applied to a vector space model based on tfidf trained on a small corpus of tweets, and found stemming, lemmatization and emoji tagging to be the most effective factors.
Distinct from prior works, we examine a much larger suite of preprocessing factors grounded in insights derived from numerous affective systems, trained over two different corpora, using three different word embedding models. We evaluate the effect of the preprocessed word embeddings in three distinct affective tasks including sentiment analysis, emotion classification and sarcasm detection.
3 Preprocessing in Affective Systems
This section describes the preprocessing factors applied to the training corpus that is then used to generate word representations and the order of the preprocessing factors which we need to follow when applying on the corpus.
3.1 Preprocessing Factors
Basic: A group of common text preprocessing applied at the very beginning, such as removing html tags, removing numbers, and lowercasing. This step removes all common punctuation from text, such as “@%*=(/ +” using the NLTK regextokenizer2.
Spellcheck (spell): A case can be made for either correcting misspellings and typos or leaving
$^{2}$ https://www.nltk.org/_modules/nltk-tokenize/regexp.html
them as is assuming they represent natural language text and its associated complexities. In this step, we identify words that may have been misspelled and correct them<sup>3</sup>. As unambiguous spell corrections are not very common and in most cases we have multiple options for correction, we built our own custom dictionary to suggest a replacement by parsing the ukWac corpora<sup>4</sup> to retrieve a word-frequency list. A misspelled word that has multiple replacements is replaced with the suggested word that has the maximum frequency in the corpora.
Negation (neg): Negation is a mechanism that transforms a positive argument into its inverse rejection (Benamara et al., 2012). Specifically in the task of affective analysis, negation plays a critical role as negation words can affect the word or sentence polarity causing the polarity to invert in many cases. Our negation procedure is as follows:
(i) Compilation of an antonym dictionary: The first stage involves compiling an antonym dictionary using the WordNet corpus (Miller, 1995). For every synset, there are three possibilities: finding no antonym, one antonym or multiple antonyms. The first two cases are trivial (unambiguous replacements). In the case of the third option (ambiguous replacement), which represents the most common case, amongst the many choices, we consider the antonym with the maximum frequency in the ukWac corpus, as described in the previous section and finally the antonym of a word is picked at random from one of its senses in our antonym dictionary.
(ii) Negation handler: Next, we identify the negation words in tokenized text<sup>5</sup>. If a negation word is found, the token following it (i.e., negated word) is extracted and its antonym looked up in the antonym dictionary. If an antonym is found, the negation word and the negated word are replaced with it.
For example, let the sentence "I am not happy today" in its tokenized form ['I', 'am', 'not', 'happy', 'today']. First, we identify any negation words (i.e., 'not') and their corresponding negated words (i.e., 'happy'). Then, we look up the antonym of 'happy' in the antonym dictionary (i.e., 'sad') and replace the phrase 'not happy' with the word 'sad', resulting in a new sentence "I am sad today".
Parts-of-Speech (pos): Four parts-of-speech
classes, namely nouns, verbs, adjectives and adverbs have been shown to be more informative with regards to affect than the other classes. Thus, using the NLTK pos-tagger, for each sentence in the corpus we retain only the words belonging to one of these four classes, i.e.,  $\mathrm{NN^{*}}$ ,  $\mathrm{JJ^{*}}$ ,  $\mathrm{VB^{*}}$ , and  $\mathrm{RB^{*}}$ .
Stopwords (stop): Stopwords are generally the most common words in a language typically filtered out before classification tasks. Therefore, we remove all the stopwords using the NLTK library.
Stemming (stem): Stemming, which reduces a word to its root form, is an essential preprocessing technique in NLP tasks. We use NLTK Snowball stemmer for stemming our training corpus.
3.2 Order of Preprocessing Factors
While some preprocessing techniques can be applied independently of each other (e.g., removing stopwords and removing punctuation), others need a more careful consideration of the sequence in which they are applied in order to obtain a more stable result. For instance, post-tagging should be applied before stemming in order for the tagger to work well, or negation should be performed prior to removing stopwords. To this end, we consider the following ordering when combining all the aforementioned preprocessing factors: spellchecking, negation handling, pos classes, removing stopwords, and stemming.
4 Experimental Evaluation Framework
4.1 Training Corpora
Table 1 summarizes the details of our two training corpora with regards to their vocabulary and corpus sizes after applying various preprocessing settings. For some preprocessing such as POS (pos) and stopwords removal (stop), without any significant loss in vocabulary as indicated by the  $\%$  ratio of preprocessed to basic, the corpus size reduces dramatically, in some cases more than  $50\%$ , a nontrivial implication with regards to training time.
News: This corpus consists of 142,546 articles from 15 American publications, spanning from 2013 to early  $2018^{6}$ .
Wikipedia: Comparatively a much larger corpus than the News, this corpus consists of 23,046,187 articles from Wikipedia<sup>7</sup>.
<sup>6</sup>https://www.kaggle.com/snapcrack/all-the-news
<sup>7</sup>https://www.kaggle.com/jkkphys/english-wikipedia-articles-20170820-sqlite
$^{3}$ https://pypi.org/project/pyspellchecker/  
 $^{4}$ https://www.sketchengine.eu/ukwac-british-english-corpus/  
 $^{5}$ https://pypi.org/project/negspacy/
5802
Corpus	Processing	Vocab	Corpus
size	%	size	%
News	Basic	155K	100	123.2M	100
spell	149K	96	123.2M	100
stem	137K	88	123.2M	100
punc	147K	95	111.0M	90
neg	152K	98	90.7M	73
stop	150K	97	75.6M	61
pos	154K	99	70.7M	57
All - punc	151K	97	93.7M	76
All - pos	140K	90	90.5M	73
All - stop	150K	97	75.3M	61
All	110K	71	55.2M	49
All - stem	110K	71	58.1M	47
All - spell	110K	71	56.4M	46
All - neg	110K	71	54.3M	44
Wikipedia	Basic	5.1M	100	8.1B	100
All - punc	4.9M	96	7.2B	89
All - pos	4.8M	94	7.0B	86
All - stop	4.9M	96	6.8B	84
All - stem	4.3M	84	6.4B	79
All - spell	4.6M	90	6.1B	75
All	4.6M	90	5.6B	69
All - neg	4.6M	90	5.0B	62
Table 1: Details of training corpora
Dataset	Genre	Task	Total
IMDB	reviews	sentiment	50,000
SemEval	tweets	sentiment	14,157
Airline	tweets	sentiment	11,541
ISEAR	narratives	emotions	5,477
Alm	fairy tales	emotions	1,206
SSEC	tweets	emotions	1,017
Onion	headlines	sarcasm	28,619
IAC	response	sarcasm	3,260
Reddit	comments	sarcasm	1,010,826
Table 2: Details of evaluation datasets
4.2 Word Embedding Models
We obtain our preprocessed word representations through three models: (i) CBOw (Continuous Bag-of-Words), (ii) Skip-gram: While CBOw takes the context of each word as the input and tries to predict the word corresponding to the context, skip-gram reverses the use of target and context words, where the target word is fed at the input and the output layer of the neural network is replicated multiple times to accommodate the chosen number of context words (Mikolov et al., 2013). We train both the models on both the training corpora using min count of 5 for News and 100 for Wikipedia with window sizes of 5 and 10, respectively, setting dimensionality to 300.
(iii) BERT (Bidirectional Encoder Representations from Transformers): BERT is an unsupervised method of pretraining contextualized language representations (Devlin et al., 2018). We train the model using BERT large uncased archi
tecture (24-layer, 1024-hidden, 16-heads, 340M parameters) with same setting for parameters as the original paper.
We train each of the three models (CBOW, Skipgram and BERT) 8 times using 16 TPUs (64 TPU chips), Tensorflow 1.15, 1TB memory on Google Cloud and two 32 GPUs cluster of V100/RTX 2080 Ti, 1TB memory using Microsoft CNTK parallelization algorithm $^{8}$  on Amazon server. For a large model such as BERT, it takes upto 4-5 days for each run of the training.
4.3 Evaluation Datasets
We conduct our evaluation on three tasks, namely sentiment analysis, emotion classification and sarcasm detection. Table 2 presents the details of our evaluation datasets, and some illustrative examples of text are shown in Table 3.
Sentiment Analysis: This popular task involves classifying text as positive or negative, and we use the following three datasets for evaluation: (i) IMDB: This dataset<sup>9</sup> includes 50,000 movie reviews for sentiment analysis, consisting of 25,000 negative and 25,000 positive reviews Maas et al. (2011). (ii) Semeval 2016: This sentiment analysis in Twitter dataset<sup>10</sup> consists of 14,157 tweets where 10,076 of them are positive and 4,081 negative Nakov et al. (2016). (iii) Airlines: This sentiment analysis dataset<sup>11</sup> consists of 11,541 tweets about six U.S. airlines from February 2015, with 9,178 tweets labeled as positive and 2,363 negative.
Emotion Classification: A multiclass classification task, this involves classifying text into a number of emotion categories such as happy, sad, and so on. The following datasets are used in our evaluation: (i) SSEC: The Stance Sentiment Emotion Corpus Schuff et al. (2017) is the re-annotation of the SemEval 2016 Twitter stance and sentiment corpus Mohammad et al. (2017) with emotion labels including anger, joy, sadness, fear, surprise. [12]. (ii) ISEAR: This dataset contains narratives of personal experiences evoking emotions Wallbott and Scherer (1986). We use a subset of the data consisting of five categories: sadness, anger, disgust, fear, joy. (iii) Alm: This dataset contains sentences
$^{8}$ https://docs.microsoft.com/en-us/cognitive-toolkit/multiple-gpus-and-machines
<sup>9</sup>http://ai.stanford.edu/amaas/data/sentiment/
10http://alt.qcri.org/semeal2016/task4/index.php
<sup>11</sup>https://www.kaggle.com/crowdflower/twitter-airlinesentiment
12SSEC: http://www.romanklinger.de/ssec/
5803
Text	Label	Dataset
·I must admit that this is one of the worst movies I’ve ever seen. I thought Dennis Hopper had a little more taste than to appear in this kind of yeeecchh... [truncated]	negative	IMDB
·everything was fine until you lost my bag.	negative	Airline
·At work, when an elderly man complained unjustifiably about me and distrusted me.	anger	ISEAR
·The ladies danced and clapped their hands for joy.	happy	Alm
·if this heat is killing me i don’t wanna know what the poor polar bears are going through	sadness	SSEC
·ford develops new suv that runs purely on gasoline	sarcastic	Onion
·Been saying that ever since the first time I heard about creationism	not-sarcastic	IAC
·Remember, it’s never a girl’s fault, it’s always the man’s fault.	sarcastic	Reddit
Table 3: Examples of text instances in the evaluation datasets
from fairy tales marked with one of five emotion categories: angry-disgusted, fearful, happy, sad and surprised Cecilia and Ovesdotter (2008).
Sarcasm Detection: Detecting sarcasm from text, a challenging task due to the sophisticated nature of sarcasm, involves labeling text as sarcastic or not. We use the following three datasets: (i) Onion: This news headlines dataset  $^{13}$  collected sarcastic versions of current events from The Onion and non-sarcastic news headlines from HuffPost Misra and Arora (2019), resulting in a total 28,619 records. (ii) IAC: A subset of the Internet Argument Corpus Oraby et al. (2016), this dataset contains response utterances annotated for sarcasm. We extract 3260 instances from the general sarcasm type. $^{14}$ . (iii) Reddit: Self-Annotated Reddit Corpus  $(\mathrm{SARC})^{15}$  is a collection of Reddit posts where sarcasm is labeled by the author in contrast to other datasets where the data is typically labeled by independent annotators Khodak et al. (2017).
4.4 Classification Setup
For classification, we employ the LSTM model as it works well with sequential data such as text. For binary classification, such as sentiment analysis and sarcasm detection, the loss function used is the binary cross-entropy along with sigmoid activation:
$$
\xi = - \frac {1}{N} \sum_ {i = 1} ^ {N} y _ {i} l o g (p (y _ {i})) + (1 - y _ {i}) l o g (1 - p (y _ {i}))
$$
where  $y$  is the binary representation of true label,  $p(y)$  is the predicted probability, and  $i$  denotes the  $i^{\mathrm{th}}$  training sample.
For multiclass emotion classification, the loss function used is categorical cross-entropy loss over a batch of  $N$  instances and  $k$  classes, along with softmax activation:
$$
\xi = - \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {k} y _ {i j} \log (p (y _ {i j}))
$$
where  $p(y)$  is the predicted probability distribution,  $p(y_{ij})\in [0,1]$