SlowGuess/ACL_2020_OCR huggingface dataset:
Antonis Maronikolakis $^{1*}$  Danae Sánchez Villegas $^{2*}$  Daniel Preotić-Pietro $^{3}$  Nikolaos Aletras $^{2}$
<sup>1</sup> Center for Information and Language Processing, LMU Munich, Germany
$^{2}$  Computer Science Department, University of Sheffield, UK  $^{3}$  Bloomberg
antmarakis@cis.lmu.de,{dsanchezvillegas1, n.aletras}@sheffield.ac.uk
dpreotiucpie@bloomberg.net
Parody is a figurative device used to imitate an entity for comedic or critical purposes and represents a widespread phenomenon in social media through many popular parody accounts. In this paper, we present the first computational study of parody. We introduce a new publicly available data set of tweets from real politicians and their corresponding parody accounts. We run a battery of supervised machine learning models for automatically detecting parody tweets with an emphasis on robustness by testing on tweets from accounts unseen in training, across different genders and across countries. Our results show that political parody tweets can be predicted with an accuracy up to  $90\%$ . Finally, we identify the markers of parody through a linguistic analysis. Beyond research in linguistics and political communication, accurately and automatically detecting parody is important to improving fact checking for journalists and analytics such as sentiment analysis through filtering out parodical utterances.<sup>1</sup>
Parody is a figurative device which is used to imitate and ridicule a particular target (Rose, 1993) and has been studied in linguistics as a figurative trope distinct to irony and satire (Kreuz and Roberts, 1993; Rossen-Knill and Henry, 1997). Traditional forms of parody include editorial cartoons, sketches or articles pretending to have been authored by the parodied person.2 A new form
of parody recently emerged in social media, and Twitter in particular, through accounts that impersonate public figures. Highfield (2016) defines parody accounts acting as: a known, real person, for obviously comedic purposes. There should be no risk of mistaking their tweets for their subject's actual views; these accounts play with stereotypes of these figures or juxtapose their public image with a very different, behind-closed-doors persona.
A very popular type of parody is political parody which plays an important role in public speech by offering irreverent interpretations of political personas (Hariman, 2008). Table 1 shows examples of very popular (over 50k followers) and active (thousands of tweets sent) political parody accounts on Twitter. Sample tweets show how the style and topic of parody tweets are similar to those from the real accounts, which may pose issues to automatic classification.
While closely related figurative devices such as irony and sarcasm have been extensively studied in computational linguistics (Wallace, 2015; Joshi et al., 2017), parody yet to be explored using computational methods. In this paper, we aim to bridge this gap and conduct, for the first time, a systematic study of political parody as a figurative device in social media. To this end, we make the following contributions:
1. A novel classification task where we seek to automatically classify real and parody tweets. For this task, we create a new large-scale publicly available data set containing a total of 131,666 English tweets from 184 parody accounts and corresponding real accounts of politicians from the US, UK and other countries (Section 3);
2. Experiments with feature- and neural-based machine learning models for parody detection, which achieve high predictive accuracy of up to  $89.7\%$  F1. These are focused on the robust
$\dagger$  Work was done while at the University of Sheffield.
Data is available here: https://archive.org/details/parity_data_ac120
2The 'Kapou Opa' column by K. Maniatis parodying Greek popular persons was a source of inspiration for this work - https://www.oneman.gr/originals/to-imerologio-karantinas-tou-dimitri-kouts oumpa/
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4373-4384 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ness of classification, with test data from: a) users; b) genders; c) locations; unseen in training (Section 5);
3. Linguistic analysis of the markers of parody tweets and of the model errors (Section 6).
We argue that understanding the expression and use of parody in natural language and automatically identifying it are important to applications in computational social science and beyond. Parody tweets can often be misinterpreted as facts even though Twitter only allows parody accounts if they are explicitly marked as  $\text{parody}^3$  and the poster does not have the intention to mislead. For example, the Speaker of the US House of Representatives, Nancy Pelosi, falsely cited a Michael Flynn parody tweet;[4] and many users were fooled by a Donald Trump parody tweet about 'Dow Joans'.[5] Thus, accurate parody classification methods can be useful in downstream NLP applications such as automatic fact checking (Vlachos and Riedel, 2014) and rumour verification (Karmakharm et al., 2019), sentiment analysis (Pang et al., 2008) or nowcasting voting intention (Tumasjan et al., 2010; Lampos et al., 2013; Tsakalidis et al., 2018).
Beyond NLP, parody detection can be used in: (i) political communication, to study and understand the effects of political parody in the public speech on a large scale (Hariman, 2008; Highfield, 2016); (ii) linguistics, to identify characteristics of figurative language (Rose, 1993; Kreuz and Roberts, 1993; Rossen-Knill and Henry, 1997); (iii) network science, to identify the adoption and diffusion mechanisms of parody (Vosoughi et al., 2018).
Parody in Linguistics Parody is an artistic form and literary genre that dates back to Aristophanes in ancient Greece who parodied argumentation styles in Frogs. Verbal parody was studied in linguistics as a figurative trope distinct to irony and satire (Kreuz and Roberts, 1993; Rossen-Knill and Henry, 1997) and researchers long debated its definition and theoretic distinctions to other types of humor (Grice et al., 1975; Sperber, 1984; Wilson, 2006; Dynel, 2014). In general, verbal parody
involves a highly situated, intentional, and conventional speech act (Rossen-Knill and Henry, 1997) composed of both a negative evaluation and a form of pretense or echoic mention (Sperber, 1984; Wilson, 2006; Dynel, 2014) through which an entity is mimicked or imitated with the goal of criticizing it to a comedic effect. Thus, imitative composition for amusing purpose is an an inherent characteristic of parody (Franke, 1971). The parodist intentionally re-presents the object of the parody and flaunts this re-presentation (Rossen-Knill and Henry, 1997).
Parody on Social Media Parody is considered an integral part of Twitter (Vis, 2013) and previous studies on parody in social media focused on analysing how these accounts contribute to topical discussions (Highfield, 2016) and the relationship between identity, impersonation and authenticity (Page, 2014). Public relation studies showed that parody accounts impact organisations during crises while they can become a threat to their reputation (Wan et al., 2015).
Satire Most related to parody, satire has been tangentially studied as one of several prediction targets in NLP in the context of identifying disinformation (McHardy et al., 2019; de Morais et al., 2019). (Rashkin et al., 2017) compare the language of real news with that of satire, hoaxes, and propaganda to identify linguistic features of unreliable text. They demonstrate how stylistic characteristics can help to decide the text's veracity. The study of parody is therefore relevant to this topic, as satire and parodies are classified by some as a type of disinformation with 'no intention to cause harm but has potential to fool' (Wardle and Derakhshan, 2018).
Irony and Sarcasm There is a rich body of work in NLP on identifying irony and sarcasm as a classification task (Wallace, 2015; Joshi et al., 2017). Van Hee et al. (2018) organized two open shared tasks. The first aims to automatically classify tweets as ironic or not, and the second is on identifying the type of irony expressed in tweets. However, the definition of irony is usually 'a trope whose actual meaning differs from what is literally enunciated' (Van Hee et al., 2018), following the Gricean belief that the hallmark of irony is to communicate the opposite of the literal meaning (Wilson, 2006), violating the first maxim of Quality (Grice et al., 1975). In this
3Both the profile description and account name need to mention this-https://help.twitter.com/en/rules-and-policies/parody-account-policy4https://tinyurl.com/ybbrh74g5https://tinyurl.com/s34dwgm
Account type	Twitter Handle	Sample tweet
Real	@realDonaldTrump	The Republican Party, and me, had a GREAT day yesterday with respect to the phony Impeachment Hoax, & yet, when I got home to the White House & checked out the news coverage on much of television, you would have no idea they were reporting on the same event. FAKE & CORRUPT NEWS!
Parody	@realDonaldTrFan	Lies! Kampala Harris says my crimes are committed in plane site! Shes lying! My crimes are ALWAYS hidden! ALWAYS!!
Real	@BorisJohnson	Our NHS will never be on the table for any trade negotiations. Were investing more than ever before - and when we leave the EU, we will introduce an Australian style, points-based immigration system so the NHS can plan for the future.
Parody	@BorisJohnson_MP	People seem to be ignoring the many advantages of selling off the NHS, like the fact that hospitals will be far more spacious once poor people can’t afford to use them.
Table 1: Two examples of Twitter accounts of politicians and their corresponding parody account with a sample tweet from each.
sense, irony is treated in NLP in a similar way as sarcasm (González-Ibanez et al., 2011; Khattri et al., 2015; Joshi et al., 2017). In addition to the words in the utterance, further using the user and pragmatic context is known to be informative for irony or sarcasm detection in NLP (Bamman and Smith, 2015; Wallace, 2015). For instance, Oprea and Magdy (2019) make use of user embeddings for textual sarcasm detection. In the design of our data splits, we aim to limit the contribution of this aspects from the results.
Relation to other NLP Tasks The pretense aspect of parody relates our task to a few other NLP tasks. In authorship attribution, the goal is to predict the author of a given text (Stamatatos, 2009; Juola et al., 2008; Koppel et al., 2009). However, there is no intent for the authors to imitate the style of others and most differences between authors are in the topics they write about, which we aim to limit by focusing on political parody. Further, in our setups, no tweets from an author are in both training and testing to limit the impact of terms specific to a particular person.
Pastiche detection (Dinu et al., 2012) aims to distinguish between an original text and a text written by someone aiming to imitate the style of the original author with the goal of impersonating. Most similar in experimental setup to our task, Preoticiuc-Pietro and Devlin Marier (2019) aim to distinguish between tweets published from the same account by different types of users: politicians or their staff. While both pastiches and staff writers aim to present similar content with similar style to the original authors, the texts lack the humorous component specific of parodies.
A large body of related NLP work has ex
plored the inference of user characteristics. Past research studied predicting the type of a Twitter account, most frequently between individual or organizational, using linguistic features (De Choudhury et al., 2012; McCorriston et al., 2015; Mac Kim et al., 2017). A broad literature has been devoted to predicting personal traits from language use on Twitter, such as gender (Burger et al., 2011), age (Nguyen et al., 2011), geolocation (Cheng et al., 2010), political preference (Volkova et al., 2014; Preoticiuc-Pietro et al., 2017), income (Preoticiuc-Pietro et al., 2015; Aletras and Chamberlain, 2018), impact (Lampos et al., 2014), socio-economic status (Lampos et al., 2016), race (Preoticiuc-Pietro and Ungar, 2018) or personality (Schwartz et al., 2013; Preoticiuc-Pietro et al., 2016).
3 Task & Data
We define parody detection in social media as a binary classification task performed at the social media post level. Given a post  $T$ , defined as a sequence of tokens  $T = \{t_1, \dots, t_n\}$ , the aim is to label  $T$  either as parody or genuine. Note that one could use social network information but this is out of the paper's scope as we only focus on parody as a linguistic device.
We create a new publicly available data set to study this task, as no other data set is available. We perform our analysis on a set of users from the same domain (politics) to limit variations caused by topic. We first identify real and parody accounts of politicians on Twitter posting in English from the United States of America (US), the United Kingdom (UK) and other accounts posting in English from the rest of the world. We opted to use
Twitter because it is arguably the most popular platform for politicians to interact with the public or with other politicians (Parmelee and Bichard, 2011). For example,  $67\%$  of prospective parliamentary candidates for the 2019 UK general election have an active Twitter account.6 Twitter also allows to maintain parody accounts, subject to adding explicit markers in both the user bio and handle such as parody, fake.7 Finally, we label tweets as parody or real, depending on the type of account they were posted from. We highlight that we are not using user description or handle names in prediction, as this would make the task trivial.
3.1 Collecting Real and Parody Politician Accounts
We first query the public Twitter API using the following terms: {parody, #parody, parody account, fake, #fake, fake account, not real} to retrieve candidate parody accounts according to Twitter's policy. From that set, we exclude any accounts matching fan or commentary in their bio or account name since these are likely to be not posting parodical content. We also exclude private and deactivated accounts and accounts with a majority of non-English tweets.
After collecting this initial set of parody candidates, the authors of the paper manually inspected up to the first ten original tweets from each candidate to identify whether an account is a parody or not following the definition of a public figure parody account from Highfield (2016) (see Section 1), further filtering out non-parody accounts. We keep a single parody account in case of multiple parody accounts about the same person. Finally, for each remaining account, the authors manually identified the corresponding real politician account to collect pairs of real and parody.
Following the process above, we were able to identify parody accounts of 103 unique people, with 81 having a corresponding real account. The authors also identified the binary gender and location (country) of the accounts using publicly available records. This resulted in  $21.6\%$  female accounts (women parliamentarians percentages as of 2017:  $19\%$  US,  $30\%$  UK,  $28.8\%$  OECD average).<sup>8</sup>
Person
Train	Dev	Test	Total	Avg. tokens (Train)
Real	51,460	6,164	8,086	65,710	23.33
Parody	51,706	6,164	8,086	65,956	20.15
All	103,166	12,328	16,172	131,666	22.55
Table 2: Data set statistics with the person split.
The majority of the politicians are located in the US  $(44.5\%)$  followed by the UK  $(26.7\%)$  while  $28.8\%$  are from the rest of the world (e.g. Germany, Canada, India, Russia).
3.2 Collecting Real and Parody Tweets
We collect all of the available original tweets, excluding retweets and quoted tweets, from all the parody and real politician accounts. We further balance the number of tweets in a real - parody account pair in order for our experiments and linguistic analysis not to be driven by a few prolific users or by imbalances in the tweet ratio for a specific pair. We keep a ratio of maximum  $\pm 20\%$  between the real and parody tweets per pair by keeping all tweets from the less prolific account and randomly down-sampling from the more prolific one. Subsequently, for the parody accounts with no corresponding real account, we sample a number of tweets equal to the median number of tweets for the real accounts. Finally, we label tweets as parody or real, depending on the type of account they come from. In total, the data set contains 131,666 tweets, with 65,710 real and 65,956 parody.
3.3 Data Splits
To test that automatically predicting political parody is robust and generalizes to held-out situations not included in the training data, we create the following three data splits for running experiments:
Person Split We first split the data by adding all tweets from each real - parody account pair to a single split, either train, development or test. To obtain a fairly balanced data set without pairs of accounts with a large number of tweets dominating any splits, we compute the mean between real and parody tweets for each account, and stratify them, with pairs of proportionally distributed means across the train, development, and test sets (see Table 2).
<sup>9</sup>Up to maximum 3200 tweets/account according to Twitter API restrictions.
$^{6}$ https://www.mpsontwitter.co.uk/  
 $^{7}$ https://help.twitter.com/en/rules-and-d-policies/parody-account-policy  
 $^{8}$ https://data.eocd.org/inequality/women-in-politics.htm
Gender
Trained on	Real	Parody	Total
Female	Train	10,081	11,036	21,117
Dev	302	230	532
Test (Male)	55,327	54,690	110,017
Male	Train	51,048	50,184	101,232
Dev	4,279	4,506	8,785
Test (Female)	10,383	11,266	21,649
Table 3: Data set statistics with the gender split (Male, Female).
Location
Trained on	Real	Parody	Total
US & RoW	Train	47,018	45,005	92,023
Dev	1,030	2,190	3,220
Test (UK)	17,662	18,761	36,423
UK & RoW	Train	33,687	35,371	69,058
Dev	1,030	1,274	2,304
Test (US)	30,993	29,311	60,304
US & UK	Train	43,211	42,597	85,808
Dev	5,444	5,475	10,919
Test (RoW)	17,055	17,884	34,939
Table 4: Data set statistics with the location split (US, UK, Rest of the World–RoW).
Gender Split We also split the data by the gender of the politicians into training, development and test, obtaining two versions of the data: (i) one with female accounts in train/dev and male in test; and (ii) male accounts in train/dev and female in test (see Table 3).
Location split Finally, we split the data based on the location of the politicians. We group the accounts in three groups of locations: US, UK and the rest of the world (RoW). We obtain three different splits, where each group makes up the test set and the other two groups make up the train and development set (see Table 4).
3.4 Text Preprocessing
We preprocess text by lower-casing, replacing all URLs and anonymizing all mentions of usernames with placeholder token. We preserve emoticons and punctuation marks and replace tokens that appear in less than five tweets with a special 'unknown' token. We tokenize text using DLATK (Schwartz et al., 2017), a Twitter-aware tokenizer.
4 Predictive Models
We experiment with a series of approaches to classification of parody tweets, ranging from linear models, neural network architectures and pretrained contextual embedding models. Hyperparameter selection is included in the Appendix.
4.1 Linear Baselines
LR-BOW As a first baseline, we use a logistic regression with standard bag-of-words (LR-BOW) representation of the tweets.
LR-BOW+POS We extend LR-BOW using syntactic information from Part-Of-Speech (POS) tags. We first tag all tweets in our data using the NLTK tagger and then we extract bag-of-words features where each unigram consists of a token with its associated POS tag.
4.2 BiLSTM-Att
The first neural model is a bidirectional Long-Short Term Memory (LSTM) network (Hochreiter and Schmidhuber, 1997) with a self-attention mechanism (BiLSTM-Att; Zhou et al. (2016)). Tokens  $t_i$  in a given tweet  $T = \{t_1, \dots, t_n\}$  are mapped to embeddings and passed through a bidirectional LSTM. A single tweet representation  $(h)$  is computed as the sum of the resulting contextualized vector representations  $(\sum_{i} a_i h_i)$  where  $a_i$  is the self-attention score in timestep  $i$ . The tweet representation  $(h)$  is subsequently passed to the output layer using a sigmoid activation function.
4.3 ULMFit
The Universal Language Model Fine-tuning (ULMFit) is a method for efficient transfer learning (Howard and Ruder, 2018). The key intuition is to train a text encoder on a language modelling task (i.e. predicting the next token in a sequence) where data is abundant, then fine-tune it on a target task where data is more limited. During fine-tuning, ULMFit uses gradual layer unfreezing to avoid catastrophic forgetting. We experiment with using AWD-LSTM (Merit et al., 2018) as the base text encoder pretrained on the Wikitext 103 data set and we fine-tune it on our own parody classification task. For this purpose, after the AWS-LSTM layers, we add a fully-connected layer with a ReLU activation function followed by an output layer with a sigmoid activation function. Before each of these two additional layers, we perform batch normalization.
4.4 BERT and RoBERTa
Bidirectional Encoder Representations from Transformers (BERT) is a language model based on transformer networks (Vaswani et al., 2017) pre-trained on large corpora (Devlin et al., 2019). The model makes use of multiple multi-head attention layers to learn bidirectional embeddings for input tokens. It is trained for masked language modelling, where a fraction of the input tokens in a given sequence are masked and the task is to predict a masked word given its context. BERT uses wordpieces which are passed through an embedding layer and get summed together with positional and segment embeddings. The former introduce positional information to the attention layers, while the latter contain information about the location of a segment. Similar to ULMFit, we fine-tune the BERT-base model for predicting parody tweets by adding an output dense layer for binary classification and feeding it with the 'classification' token.
We further experiment with RoBERTa (Liu et al., 2019), which is an extension of BERT trained on more data and different hyperparameters. RoBERTa has been showed to improve performance in various benchmarks compared to the original BERT (Liu et al., 2019).
4.5 XLNet
XLNet is another pre-trained neural language model based on transformer networks (Yang et al., 2019). XLNet is similar to BERT in its structure, but is trained on a permutated (instead of masked) language modelling task. During training, sentence words are permuted and the model predicts a word given the shuffled context. We also adapt XLNet for predicting parody, similar to BERT and ULMFit.
4.6 Model Hyperparameters
We optimize all model parameters on the development set for each data split (see Section 3).
Linear models For the LR-BOW, we use n-grams with  $n = (1,2)$ ,  $n \in \{(1,1),(1,2),(1,3)\}$  weighted by TF.IDF. For the LR-BOW+POS, we use TF with POS n-grams where  $n = (1,3)$ . For both baselines we use L2 regularization.
BiLSTM-Att We use 200-dimensional GloVe embeddings (Pennington et al., 2014) pre-trained on Twitter data. The maximum sequence length
is set to 50 covering  $95\%$  of the tweets in the training set. The LSTM size is  $h = 300$  where  $h \in \{50,100,300\}$  with dropout  $d = 0.5$  where  $d \in \{2,5\}$ . We use Adam (Kingma and Ba, 2014) with default learning rate, minimizing the binary cross-entropy using a batch size of 64 over 10 epochs with early stopping.