SlowGuess/ACL_2020_OCR huggingface dataset:
ULMFit We first update only the AWD-LSTM weights with a learning rate  $l = 2\mathrm{e} - 3$  for one epoch where  $l \in \{1\mathrm{e} - 3, 2\mathrm{e} - 3, 4\mathrm{e} - 3\}$  for language modeling. Then, we update both the AWD-LSTM and embedding weights for one more epoch, using a learning rate of  $l = 2\mathrm{e} - 5$  where  $l \in \{1\mathrm{e} - 4, 2\mathrm{e} - 5, 5\mathrm{e} - 5\}$ . The size of the intermediate fully-connected layer (after AWD-LSTM and before the output) is set by default to 50. Both in the intermediate and output layers we use default dropout of 0.08 and 0.1 respectively from Howard and Ruder (2018).
BERT and RoBERTa For BERT, we used the base model (12 layers and 110M total parameters) trained on lowercase English. We fine-tune it for 1 epoch with a learning rate  $l = 5\mathrm{e} - 5$  where  $l \in \{2\mathrm{e} - 5,3\mathrm{e} - 5,5\mathrm{e} - 5\}$  as recommended in Devlin et al. (2019) with a batch size of 128. For RoBERTa, we use the same fine-tuning parameters as BERT.
XLNet We use the same parameters as BERT except for the learning rate, which we set at  $l = 4\mathrm{e} - 5$  where  $l \in \{2\mathrm{e} - 5, 4\mathrm{e} - 5, 5\mathrm{e} - 5\}$ .
This section contains the experimental results obtained on all three different data splits proposed in Section 3. We evaluate our methods (Section 4) using several metrics, including accuracy, precision, recall, macro F1 score, and Area under the ROC (AUC). We report results over three runs using different random seeds and we report the average and standard deviation.
5.1 Person Split
Table 5 presents the results for the parody prediction models with the data split by person. We observe the architectures using pre-trained text encoders (i.e. ULMFit, BERT, RoBERTa and XLNet) outperform both neural (BiLSTM-Att) and feature-based (LR-BOW and LR-BOW+POS) by a large margin across metrics with transformer architectures (BERT, RoBERTa and XLNet) performing best. The highest scoring model,
Person
Model	Acc	P	R	F1	AUC
LR-BOW	73.95 ±0.00	70.08 ± 0.01	83.53 ±0.02	76.19 ±0.00	73.96 ±0.00
LR-BOW+POS	74.33 ±0.00	71.34 ±0.00	81.19 ±0.00	75.95 ±0.00	74.34 ±0.00
BiLSTM-Att	79.92 ±0.01	81.63 ±0.01	77.11 ±0.03	79.29 ±0.02	79.91 ±0.01
ULMFit	81.11 ±0.38	75.57 ±2.03	84.97 ±0.87	81.05 ±0.42	81.10 ±0.38
BERT	87.65 ±0.29	87.63 ±0.58	87.67 ±0.40	87.65 ±0.18	87.65 ±0.32
RoBERTa	90.01 ±0.35	90.90 ±0.55	88.45 ±0.22	89.66 ±0.33	90.05 ±0.29
XLNet	86.45 ±0.41	88.24 ±0.52	85.18 ±0.40	86.68 ±0.37	86.45 ±0.36
RoBERTa, classifies accounts (parody and real) with an accuracy of 90, which is more than  $8\%$  greater than the best non-transformer model (the ULMFit method). RoBERTa also outperforms the Logistic Regression baselines (LR-BOW and LR-BOW+POS) by more than 16 in accuracy and 13 in F1 score. Furthermore, it is the only model to score higher than 90 on precision.
5.2 Gender Split
Table 6 shows the F1-scores obtained when training on the gender splits, i.e. training on male and testing on female accounts and vice versa. We first observe that models trained on the male set are in general more accurate than models trained on the female set, with the sole exception of ULMFit. This is probably due to the fact that the data set is imbalanced towards men as shown in Table 3 (see also Section 3). We also do not observe a dramatic performance drop compared to the mixed-gender model on the person split (see Table 5). Again, RoBERTa is the most accurate model when trained in both splits, obtaining an F1-score of 87.11 and 84.87 for the male and female data respectively. The transformer-based architectures are again the best performing models overall, but the difference between them and the feature-based methods is smaller than it was on the person split.
5.3 Location Split
Table 7 shows the F1-scores obtained training our models on the location splits: (i) train/dev on UK and RoW, test on US; (ii) train/dev on US and RoW, test on UK; and (iii) train/dev on US and UK, test on RoW. In general, the best results are obtained by training on the US & UK split, while results of the models trained on the RoW & US,
Table 5: Accuracy (Acc), Precision (P), Recall (R), F1-Score (F1) and ROC-AUC for parody prediction splitting by person (± std. dev.). Best results are in bold.
Gender
Model	M→F	F→M
LR-BOW	78.89	76.63
LR-BOW+POS	78.74	76.74
BiLSTM-Att	77.00	77.11
ULMFit	81.20	82.53
BERT	85.85	84.40
RoBERTa	87.11	84.87
XLNet	85.69	84.16
Table 6: F1-scores for parody prediction splitting by gender (Male-M, Female-F). Best results are in bold.
Location
Model	+ →	+ →	+ →
LR-BOW	78.58	78.27	77.97
LR-BOW+POS	78.27	77.88	78.08
BiLSTM-Att	80.29	77.59	73.19
ULMFit	83.47	81.55	81.55
BERT	86.69	83.78	83.12
RoBERTa	87.70	85.10	85.99
XLNet	85.32	85.17	85.32
Table 7: F1-scores for parody prediction splitting by location. Best results are in bold.
and RoW & UK splits are similar. The model with the best performance trained on US & UK, and RoW & UK splits is RoBERTa with F1 scores of 87.70 and 85.99 respectively. XLNet performs slightly better than RoBERTa when trained on RoW & US data split.
Through experiments over three different data splits, we show that all models predict parody tweets consistently above random, even if tested
on people unseen in training. In general, we observe that the pre-trained contextual embedding based models perform best, with an average of around 10 F1 better than the linear methods. From these methods, we find that RoBERTa outperforms the other methods by a small, but consistent margin, similar to past research (Liu et al., 2019). Further, we see that the predictions are robust to any location or gender specific differences, as the performance on held-out locations and genders are close to when splitting by person with a maximum of  $< 5$  F1 drop, also impacted by training on less data (e.g. female users). This highlights the fact that our models capture information beyond topics or features specific to any person, gender or location and can potentially identify stylistic differences between parody and real tweets.
We finally perform an analysis based on our novel data set to uncover the peculiarities of political parody and understand the limits of the predictive models.
6.1 Linguistic Feature Analysis
We first analyze the linguistic features specific of real and parody tweets. For this purpose, we use the method introduced in (Schwartz et al., 2013) and used in several other analyses of user traits (Preotciuc-Pietro et al., 2017) or speech acts (Preotciuc-Pietro et al., 2019). We thus rank the feature sets described in Section 4 using univariate Pearson correlation (note that for the analysis we use POS tags instead of POS n-grams). Features are normalized to sum up to unit for each tweet. Then, for each feature, we compute correlations independently between its distribution across posts and the label of the post (parody or not).
Table 8 presents the top unigrams and part-of-speech features correlated with real and parody tweets. We first note that the top features related to either parody or genuine tweets are function words or related to style, as opposed to the topic. This enforces that the make-up of the data set or any of its categories are not impacted by topic choice and parody detection is mostly a stylistic difference. The only exception are a few hashtags related to parody accounts (e.g. #imwithme), but on a closer inspection, all of these are related to tweets from a single parody account and are thus not useful in prediction by any setup, as tweets containing these
Real	Parody
Feature	r	Feature	r
Unigrams
our	0.140	i	0.181
in	0.131	?	0.156
and	0.129	<mention>	0.145
:	0.118	me	0.136
&	0.114	not	0.106
today	0.105	like	0.097
to	0.105	my	0.095
of	0.098	dude	0.094
the	0.091	don’t	0.090
at	0.087	i’m	0.087
lhl	0.086	just	0.083
great	0.085	know	0.081
with	0.084	#feeltheburp	0.078
de	0.079	you	0.076
meeting	0.078	#callmedick	0.075
for	0.077	#imwithme	0.073
across	0.073	”	0.073
families	0.073	#visionzero	0.069
on	0.070	if	0.069
country	0.067	have	0.067
POS (Unigrams and Bigrams)
NN IN	0.1600	RB	0.1749
IN	0.1507	PRP	0.1546
CC	0.1309	RB VB	0.1271
IN JJ	0.1210	VBP	0.1206
NNS IN	0.1165	VBP RB	0.1123
NN CC	0.1114	.	0.1114
IN NN	0.1048	NNP NNP	0.1094
NN TO	0.1030	NN NNP	0.1057
NNS TO	0.1013	WRB	0.0925
TO	0.1001	VBP PRP	0.0904
CC JJ	0.0972	IN PRP	0.0890
IN DT	0.0941	NN VBP	0.0863
: JJ	0.0875	RB .	0.0854
NNS	0.0855	NNP	0.0837
: NN	0.0827	JJ VBP	0.0813
Table 8: Feature correlations with parody and real tweets, sorted by Pearson correlation (r). All correlations are significant at  $p < .01$ , two-tailed t-test.
will only appear in either the train or test set.
The top features related to either category of tweets are pronouns ('our' for genuine tweets, 'i' for parody tweets). In general, we observe that parody tweets are much more personal and include possessives ('me', 'my', 'i', "i'm", PRP) or second person pronouns ('you'). This indicates that parodies are more personal and direct, which is
also supported by use of more @-mentions and quotation marks. The real politician tweets are more impersonal and the use of ‘our’ indicates a desire to include the reader in the conversation.
The real politician tweets include more stopwords (e.g. prepositions, conjunctions, determiners), which indicate that these tweets are more well formed. Conversely, the parody tweets include more contractions ("don't", "i'm"), hinting to a less formal style ('dude'). Politician tweets frequently use their account to promote events they participate in or are relevant to the day-to-day schedule of a politician, as hinted by several prepositions ('at', 'on') and words ('meeting', 'today') (Preoticiuc-Pietro and Devlin Marier, 2019). For example, this is a tweet of the U.S. Senator from Connecticut, Chris Murphy:
Rudy Giuliani is in Ukraine today, meeting with Ukrainian leaders on behalf of the President of the United States, representing the President's re-election campaign.[...]
Through part-of-speech patterns, we observe that parody accounts are more likely to use verbs in the present singular (VBZ, VBP). This hints that parody tweets explicitly try to mimic direct quotes from the parodied politician in first person and using present tense verbs, while actual politician tweets are more impersonal. Adverbs (RB) are used predominantly in parodies and a common sequence in parody tweets is adverbs followed by verbs (RB VB) which can be used to emphasize actions or relevant events. For example, the following is a tweet of a parody account (@Queen_Europe) of Angela Merkel:
I mean, the Brexit Express literally appears to be going backwards but OK <url>
6.2 Error Analysis
Finally, we perform an error analysis to examine the behavior of our best performing model (RoBERTa) and identify potential limitations of the current approaches. The first example is a tweet by the former US president Barack Obama which was classified as parody while it is in fact a real tweet:
Summer's almost over, Senate Leaders. #doyour-job <url>
Similarly, the next tweet was posted by the real account of the Virginia governor, Ralph Northam:
At this point, the list of Virginians Ed Gillespie *hasn't* sold out is shorter than the folks he has. <url>
Both of the tweets above contain humorous elements and come off as confrontational, aimed at someone else which is more prevalent in parody. We hypothesize that the model picked up this information to classify these tweets as parody. From the previous analyses, we noticed that tweets by real politicians often convey information in a more neutral or impersonal way. On the other hand, the following tweet was posted by a Mitt Romney parody account and was classified as real:
It's up to you, America: do you want a repeat of the last four years, or four years staggeringly worse than the last four years?
This parody tweet, even though it is more opinionated, is more similar in style to a slogan or campaign speech and is therefore missclassified. Lastly, the following is a tweet from former President Obama that was misclassified as parody:
It's the #GimmeFive challenge, presidential style.<url>
The reason behind is that there are politicians, such as Barack Obama, who often write in an informal manner and this may cause the models to misclassify this kind of tweets.
We presented the first study of parody using methods from computational linguistics and machine learning, a related but distinct linguistic phenomenon to irony and sarcasm. Focusing on political parody in social media, we introduced a freely available large-scale data set containing a total of 131,666 English tweets from 184 real and corresponding parody accounts. We defined parody prediction as a new binary classification task at a tweet level and evaluated a battery of feature-based and neural models achieving high predictive accuracy of up to  $89.7\%$  F1 on tweets from people unseen in training.
In the future, we plan to study more in depth the stylistic and figurative devices used for parody, extend the data set beyond the political case study and explore human behavior regarding parody, including how this is detected and diffused through social media.
We thank Bekah Hampson for providing early input and helping with the data annotation. NA is supported by ESRC grant ES/T012714/1 and an Amazon AWS Cloud Credits for Research Award.
Nikolaos Aletras and Benjamin Paul Chamberlain. 2018. Predicting Twitter user socioeconomic attributes with network and language information. In Proceedings of the 29th on Hypertext and Social Media, pages 20-24.
David Bamman and Noah A Smith. 2015. Contextualized Sarcasm Detection on Twitter. In Ninth International AAAI Conference on Web and Social Media, ICWSM, pages 574-577.
John D Burger, John Henderson, George Kim, and Guido Zarrella. 2011. Discriminating gender on Twitter. In Proceedings of the conference on empirical methods in natural language processing, pages 1301-1309. Association for Computational Linguistics.
Zhiyuan Cheng, James Caverlee, and Kyumin Lee. 2010. You are where you tweet: a content-based approach to geo-locating Twitter users. In Proceedings of the 19th ACM international conference on Information and knowledge management, pages 759-768.
Munmun De Choudhury, Nicholas Diakopoulos, and Mor Naaman. 2012. Unfolding the event landscape on Twitter: Classification and exploration of user categories. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW, pages 241-244.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Liviu P. Dinu, Vlad Niculae, and Maria-Octavia Sulea. 2012. Pastiche detection based on stopwords rankings. Exposing Impersonators of a Romanian writer. In Proceedings of the Workshop on Computational Approaches to Deception Detection, pages 72-77, Avignon, France. Association for Computational Linguistics.
Marta Dynel. 2014. Isn't it ironic? Defining the scope of humorous irony. Humor, 27(4):619-639.
Herbert Franke. 1971. A note on parody in Chinese traditional literature. Oriens Extremus, 18(2):237-251.
Roberto González-Ibanez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581-586.
H Paul Grice, Peter Cole, Jerry Morgan, et al. 1975. Logic and conversation. 1975, pages 41-58.
Robert Hariman. 2008. Political Parody and Public Culture. Quarterly Journal of Speech, 94(3):247-272.
Tim Highfield. 2016. News via Voldemort: Parody accounts in topical discussions on Twitter. New Media & Society, 18(9):2028-2045.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339.
Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50(5):73.
Patrick Juola et al. 2008. Authorship attribution. Foundations and Trends in Information Retrieval, 1(3):233-334.
Twin Karmakharm, Nikolaos Aletras, and Kalina Bontcheva. 2019. Journalist-in-the-loop: Continuous learning as a service for rumour analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 115-120.
Anupam Khatri, Aditya Joshi, Pushpak Bhattacharyya, and Mark Carman. 2015. Your sentiment precedes you: Using an author's historical tweets to predict sarcasm. In Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 25-30.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Moshe Koppel, Jonathan Schler, and Shlomo Argamon. 2009. Computational methods in authorship attribution. Journal of the Association for Information Science and Technology, 60(1):9-26.
Roger J. Kreuz and Richard M. Roberts. 1993. On satire and parody: The importance of being ironic. Metaphor and Symbolic Activity, 8(2):97-109.
Vasileios Lampos, Nikolaos Aletras, Jens K Geyti, Bin Zou, and Ingemar J Cox. 2016. Inferring the socioeconomic status of social media users based on behaviour and language. In ECIR, pages 689-695.
Vasileios Lampos, Nikolaos Aletras, Daniel Preoiciuc-Pietro, and Trevor Cohn. 2014. Predicting and characterising user impact on Twitter. In 14th conference of the European chapter of the Association for
Computational Linguistics 2014, EACL 2014, pages 405-413.
Vasileios Lampos, Daniel Preoticiuc-Pietro, and Trevor Cohn. 2013. A user-centric model of voting intention from social media. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 993-1003, Sofia, Bulgaria. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Sunghwan Mac Kim, Qiongkai Xu, Lizhen Qu, Stephen Wan, and Cecile Paris. 2017. Demographic Inference on Twitter using Recursive Neural Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 471-477.
James McCorriston, David Jurgens, and Derek Ruths. 2015. Organizations are users too: Characterizing and detecting the presence of organizations on Twitter. ICWSM, pages 650-653.
Robert McHardy, Heike Adel, and Roman Klinger. 2019. Adversarial training for satire detection: Controlling for confounding variables. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 660-665.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations.
Janaina Ignacio de Morais, Hugo Queiroz Abonizio, Gabriel Marques Tavares, André Azevedo da Fonseca, and Sylvio Barbon Jr. 2019. Deciding among fake, satirical, objective and legitimate news: A multi-label classification system. In Proceedings of the XV Brazilian Symposium on Information Systems, page 22.
Dong Nguyen, Noah A Smith, and Carolyn P Rose. 2011. Author age prediction from text using linear regression. In Proceedings of the 5th ACL-HLT workshop on language technology for cultural heritage, social sciences, and humanities, pages 115-123. Association for Computational Linguistics.
Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2854-2859.
Ruth Page. 2014. Hoaxes, hacking and humour: analysing impersonated identity on social network sites, pages 46-64. Palgrave Macmillan UK.
Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2):1-135.
John H Parmelee and Shannon L Bichard. 2011. Politics and the Twitter revolution: How tweets influence the relationship between political leaders and the public. Lexington Books.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Daniel Preoticiuc-Pietro, Jordan Carpenter, Salvatore Giorgi, and Lyle Ungar. 2016. Studying the dark triad of personality through Twitter behavior. In Proceedings of the 25th ACM international on conference on information and knowledge management, pages 761-770.
Daniel Preotjiuc-Pietro and Rita Devlin Marier. 2019. Analyzing linguistic differences between owner and staff attributed tweets. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2848-2853.
Daniel Preoticiuc-Pietro, Mihaela Gaman, and Nikolaos Aletras. 2019. Automatically identifying complaints in social media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5008-5019.
Daniel Preoticiuc-Pietro, Ye Liu, Daniel Hopkins, and Lyle Ungar. 2017. Beyond binary labels: political ideology prediction of Twitter users. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 729-740.
Daniel Preotciuc-Pietro and Lyle Ungar. 2018. User-level race and ethnicity predictors from Twitter text. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1534-1545.
Daniel Preotjiuc-Pietro, Svitlana Volkova, Vasileios Lampos, Yoram Bachrach, and Nikolaos Aletras. 2015. Studying user income through language, behaviour and affect in social media. PloS one, 10(9).
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931-2937.
Margaret A Rose. 1993. Parody: ancient, modern and post-modern. Cambridge University Press.
Deborah F. Rossen-Knill and Richard Henry. 1997. The pragmatics of verbal parody. Journal of Pragmatics, 27(6):719-752.