SlowGuess/ACL_2020_OCR huggingface dataset:
System	Sarcasticness	Creativity	Humor	Grammaticality
State-of-the-art (Mishra et al., 2019)	1.63	1.60	1.50	1.46
Human Generated	3.57	3.16	3.18	3.98
Reversal of Valence (RV)	3.00	2.80	2.72	4.29
No Reversal of Valence (NoRV)	1.79	2.28	2.09	3.91
No Semantic Incongruity (NSI)	3.04	2.99	2.90	3.68
Full Model (FM)	3.23*	3.24	3.08*	3.69
Table 2: Average scores for generated sarcasm from all systems as judged by the Turkers. The scale ranges from 1 (not at all) to 5 (very). For creativity and grammaticality, our models are comparable to human annotation and significantly better than the state-of-the-art ( $p < 0.001$ ). For sarcasticity and humor, the full model is ranked 2nd by a small margin against the human generated message (denoted by *).
Aspect	FM vs Human	FM vs MTS2019
win%	lose%	win%	lose%
Sarcasticness	34.0	55.3	90.0	6.0
Creativity	48.0	36.0	95.3	4.0
Humor	40.6	48.0	90.0	4.0
Grammaticality	26.6	56.6	98.0	1.3
Table 3: Pairwise comparison between the full model (FM) and human generated sarcasm, and between the full model (FM) and the state-of-the-art model in Mishra et al. (2019). Win % (lose %) is the percentage of the FM gets a higher (lower) average score compared to the other method for the 150 human-rated sentences. The rest are ties.
attempted the HITs (inter-annotator agreement of 0.59, 0.53, 0.47 and 0.66 for the tasks on creativity, sarcasticness, humour and grammaticality, respectively using Spearman's correlation coefficient).
6.1 Quantitative Scores
Table 2 presents the scores for the above mentioned metrics of different systems averaged over 150 test utterances. Our full model as well as the variations that ablated some components improve over the state-of-the-art (Mishra et al., 2019) on all the criteria. The ablation in Table 2 shows that our full model is superior to individual modules in terms of sarcasticity, creativity and humor. For grammaticality, we observe that the Turkers scored shorter sentences higher (e.g., RV), which also explains why NoRV model received a higher score than the full model. NoRV otherwise performed worse than all the other variations.
In terms of creativity, our full model attains the highest average scores over all the other models including sarcastic utterances composed by humans. For grammaticality, the reversal of valence model is the best, even better than human gener
Figure 3: Pie chart comparing the success rate of all the variations of our model.
ated ones. The performance of the full model is the second best in terms of the sarcasticity and humor, only slightly worse than human-generated sarcasm, showing the effectiveness of our approach that captures various factors of sarcasm.
6.2 Pairwise game between Full Model, State-of-the-art and Humans
Table 3 displays the pairwise comparisons between the full model (FM) and human generated sarcasm, and FM and Mishra et al. (2019), respectively. Given a pair of inputs, we decide win/lose/tie by comparing the average scores (over three Turkers) of both outputs. We see that FM dominates Mishra et al. (2019) on all the metrics and human-generated sarcasm on the creativity metric. For sarcasticity, although humans are better, the FM model still has a  $34\%$  winning rate.
We focus our ablation study on the metric of sarcasticness, as we consider this as the main criterion for the success of generating sarcasm. As shown in Figure 3, our best model (FM) outperforms individ
7982
Non Sarcastic	System	Sarcasm	S	C	H	G
I inherited unfavorable genes from my mother.	FM	I inherited great genes from my mother. Ugly goes down to the bone.	5.0	4.0	3.6	3.6
RV	I inherited great genes from my mother.	3.0	2.6	2.0	2.3
NoRV	Ugly goes down to the bone.	3.0	2.6	3.0	4.0
NSI	I inherited great genes from my mother. She makes me feel dowdy and ugly.	2.6	3.6	3.0	4.0
MTS2019	Butch tagging bullies apc seymour good temper good mentor.	1.3	1.0	1.3	2.0
Human	Great I inherited all of my mother's GOOD genes	2.3	4.3	2.0	2.6
It is not fun to date a drug addict.	FM	It is fun to date a drug addict. Spent the night in a police cell after his arrest.	4.3	5.0	4.6	5.0
RV	It is fun to date a drug addict.	5.0	2.3	2.0	4.6
NoRV	Spent the night in a police cell after his arrest.	1.0	1.0	2.0	2.6
NSI	It is fun to date a drug addict. The feds completely screwed up the arrest.	3.3	4.3	2.0	2.6
MTS2019	Butch is a powerful addict in gente he is an optimist great fun.	2.6	2.0	1.0	1.3
Human	Dating a drug addict .. Wouldn't that be fun.	3.0	1.6	2.6	4.0
I hate getting sick from fast food.	FM	I love getting sick from fast food. Stomach ache is just an additional side effect.	3.3	3.6	5.0	3.6
RV	I love getting sick from fast food.	3.3	2.6	3.6	5.0
NoRV	Stomach ache is just an additional side effect.	1.3	2.6	3.6	3.3
NSI	I love getting sick from fast food. I ate too much and got a terrible stomach ache.	2.3	3.3	4.3	5.0
MTS2019	I hate love sick to ikes sword lowest **** giving stains giving stains on printers making pound accidents work bikinis in	1.0	1.3	1.3	1.0
Human	Shout out to the mcdonalds for giving me bad food and making me sick right before work in two hours.	4.0	4.3	4.0	4.3
Burnt popcorn is gross.	FM	Burnt popcorn is lovely. The smell made me want to vomit.	4.6	3.0	3.3	5.0
RV	Burnt popcorn is lovely.	4.0	2.0	3.6	5.0
NoRV	The smell made me want to vomit.	1.0	2.0	3.6	4.6
NSI	Burnt popcorn is lovely. Hold the bag in case I vomit.	4.3	2.3	4.3	5.0
MTS2019	reggae burnt popcorn lol .	2.3	1.3	2.0	1.0
Human	Gotta love the smell of burnt microwave popcorn.	3.3	3.3	4.0	4.0
Table 4: Examples of generated outputs from different systems. S, C, H, G represent Sarcasticness, Creativity, Humor and Grammaticality. Text in bolded black represents the commonsense word/phrase obtained from COMET given the non-sarcastic utterance.
ual ablation modules. We filtered out 60 examples from the 150 with no ties. The ablation component employing just Reversal of Valence is second best for sarcasticness according to Figure 3.
Further, to understand the extent to which ranking the retrieved sentence based on the degree of incongruity helped generate better sarcasm, we took the outputs from FM and NSI for comparisons. Out of the 150 utterances, 119 times there was no tie. Our best model (FM) wins  $66\%$  of the time while the NSI model wins  $34\%$  of the cases.
Table 4 demonstrates several generation outputs from different modules associated with human ratings for different criteria. We notice that often one of our modules generate better sarcasm than humans. For instance, for the first and the second example in Table 4, all of FM, RV and NSI are better than human generated sarcasm. In general, the generations from the FM model are more humorous, which is also an useful criterion to evaluate sarcasm besides sarcasticity (Skalicky and Crossley, 2018).
We also observe that Turkers consistently rated generations from the FM model more sarcastic than
7983
the NSI model suggesting that there is a correlation between human scores of sarcasticness and incongruity. To support this observation, we took the contradiction scores from the RoBERTa model for both best ranked retrieved sentences (FM) and the randomly selected retrieved sentences (NSI). We then computed a correlation between the sarcasticness scores given by the humans and the automatic contradiction scores for both the best ranked retrieved sentences (FM) and the randomly selected retrieved sentences (NSI). For FM model we obtain a higher Pearson correlation coefficient compared to NSI suggesting the important role of incongruency for sarcasm.
While our best model combining different sarcasm factors does outperform the system with individual factors, there are sometimes exceptions. We notice, in few cases, the simple reversal of valence (RV) strategy is enough to generate sarcasm. For instance, for the literal input "It is not fun to date a drug addict", just removing the negation word leads to a full score on sarcasticness without the additional commonsense module. Future work would include building a model that can decide whether just the RV strategy is sufficient or if we need to add additional commonsense context to it.
Although incorporating incongruity ranking is useful, there are several cases when a randomly retrieved message may obtain better sarcasticness score. Table 5 presents such an example. Even though the retrieved message "Please stop whirling me round; it makes me feel sick." scores lower than "The very thought of it makes me feel sick.", in terms of incongruity with respect to "I love being put in the hospital for dehydration", the former received a higher sarcasticity score that suggests the incongruity scores obtained from NLI are not perfect.
The ordering of the commonsense context and the valence reversed sentence is predetermined in our generation. Specifically, we always append the retrieved commonsense context after the valence reversed output. Changing the order can sometimes make the sarcasm better and more humorous. The reason for our current ordering choice is that we always treat the valence reversed version as hypothesis and the commonsense retrieved sentence as premise for the NLI model. We attempted reversing the order in preliminary experiments but
NSI	I love being put in the hospital for dehydration. Please stop whirling me round; it makes me feel sick.
FM	I love being put in the hospital for dehydration. The very thought of it makes me feel sick.
Table 5: Sarcastic Generation from (FM) and (NSI) where NSI scores higher for sacrasticness
received poor scores from the entailment model. In future, we would like to generate more diverse sarcasm that are not tied to a fixed pattern.
Finally, the generations are dependent on COMET and thus the quality will be governed by the accuracy of the COMET model.
We address the problem of unsupervised sarcasm generation that models several sarcasm factors including reversal of valence and semantic incongruity with the context. The key contribution of our approach is the modeling of commonsense knowledge in a retrieve-and-edit generation framework. A human-based evaluation based on four criteria shows that our generation approach significantly outperforms a state-of-the-art model. Compared with human generated sarcasm, our model shows promise particularly for creativity, humor and sarcasticity, but less for grammaticality. A bigger challenge in sarcasm generation and more generally, creative text generation, is to capture the difference between creativity (novel but well-formed material) and nonsense (ill-formed material). Language models conflate the two, so developing methods that are nuanced enough to recognize this difference is key to future progress.
This work was supported in part by the MCS program under Cooperative Agreement N66001-19-2-4032 and the CwC program under Contract W911NF-15-1-0543 with the US Defense Advanced Research Projects Agency (DARPA). The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. The authors would like to thank Christopher Hidey, John Kropf, Anusha Bala and Christopher Robert Kedzie for useful discussions. The authors also thank members of PLUSLab at the University Of Southern California and the anonymous reviewers for helpful comments.
7984
Salvatore Attardo. 2000. Irony markers and functions: Towards a goal-oriented theory of irony and its processing. Rask, 12(1):3-20.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Florence, Italy. Association for Computational Linguistics.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642.
Christian Burgers, Margot Van Mulken, and Peter Jan Schellens. 2012. Verbal irony differences in usage across written genres. Journal of Language and Social Psychology, 31(3):290-310.
Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In *In Proceedings of the Fourteenth Conference on Computational Natural Language Learning*. CoNLL.
Andrea Esuli and Fabrizio Sebastiani. 2006. Sentiwordnet: A publicly available lexical resource for opinion mining. In LREC, volume 6, pages 417-422. Citeseer.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2017. Style transfer in text: Exploration and evaluation. In Proceedings of AAAI.
Aniruddha Ghosh and Tony Veale. 2017. Magnets for sarcasm: Making sarcasm detection timely, contextual and very personal. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 482-491, Copenhagen, Denmark. Association for Computational Linguistics.
Debanjan Ghosh, Alexander R Fabbri, and Smaranda Muresan. 2018. Sarcasm analysis using conversation context. Computational Linguistics, 44(4):755-792.
Debanjan Ghosh, Weiwei Guo, and Smaranda Muresan. 2015. Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words. In proceedings of the 2015 conference on empirical methods in natural language processing, pages 1003-1012.
Debanjan Ghosh, Elena Musi, Kartikeya Upasani, and Smaranda Muresan. 2020. Interpreting verbal irony: Linguistic strategies and the connection to the type of semantic incongruity. Proceedings of the Society for Computation in Linguistics, 3(9):76-87.
Debanjan Ghosh, Alexander Richard Fabbri, and Smaranda Muresan. 2017. The role of conversation context for sarcasm detection in online interactions. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 186-196, Saarbrücken, Germany. Association for Computational Linguistics.
Raymond W Gibbs. 1986. On the psycholinguistics of sarcasm. Journal of Experimental Psychology: General, 115(1):3.
Roberto González-Ibanez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT '11, pages 581-586, Stroudsburg, PA, USA. Association for Computational Linguistics.
Li Huang, Francesca Gino, and Adam Galinsky. 2015. The highest form of intelligence: Sarcasm increases creativity for both expressers and recipients. Organizational Behavior and Human Decision Processes, 131.
Aditya Joshi, Anoop Kunchukuttan, Pushpak Bhattacharyya, and Mark James Carman. 2015a. Sarcasbot: An open-source sarcasm-generation module for chatbots. In WISDOM Workshop at KDD.
Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015b. Harnessing context incongruity for sarcasm detection. In ACL, pages 757-762.
Roger J Kreuz and Kristen E Link. 2002. Asymmetries in the use of verbal irony. Journal of Language and Social Psychology, 21(2):127-143.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
Abhijit Mishra, Tarun Tater, and Karthik Sankaranayanan. 2019. A modular architecture for unsupervised sarcasm generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6146-6155, Hong Kong, China. Association for Computational Linguistics.
7985
Smaranda Muresan, Roberto Gonzalez-Ibanez, Debanjan Ghosh, and Nina Wacholder. 2016. Identification of nonliteral language in social media: A case study on sarcasm. Journal of the Association for Information Science and Technology, 67(11):2725-2737.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Lotem Peled and Roi Reichart. 2017. Sarcasm SIGN: Interpreting sarcasm with sentiment based monolingual machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1690-1700, Vancouver, Canada. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. In URL https://s3-us-west-2. amazonaws.com/openai-assets/researchcovers/languageeunsupervised/language understanding paper.pdf.
Stefanie Regel. 2009. The comprehension of figurative language: electrophysiological evidence on the processing of irony. Ph.D. thesis, Max Planck Institute for Human Cognitive and Brain Sciences Leipzig.
Ellen Riloff, Ashequl Qadir, Prafulla Survey, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704-714.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pages 6830-6841.
Stephen Skalicky and Scott Crossley. 2018. Linguistic features of sarcasm and metaphor production quality. In Proceedings of the Workshop on Figurative Language Processing, pages 7-16.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. We usually dont like going to the dentist: Using common sense to detect irony on twitter. Computational Linguistics, 44(4):793-832.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019. Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 156-165.