SlowGuess/NAACL_2022_OCR huggingface dataset:
<sup>11</sup>If all candidates are filtered out by their paraphrase scores (i.e. they all have the zero final scores), we will select the one with the highest hyperbole score among all candidates.
System	BLEU	BERTScore
R1	2.02	0.229
R3	33.25	0.520
BART	33.57	0.596
MOVER	39.43	0.624
w/o para score	39.22	0.604
w/o hypo ranker	34.83	0.610
COPY	51.69	0.711
Table 3: Automatic evaluation results on the test set of HYPO dataset.
MOVER VS.	Flu.	Hypo.	Crea.	Rel.
W%	L%	W%	L%	W%	L%	W%	L%
R1	79.7	1.7	52.4	47.6	33.9	66.1	94.2	4.3
R3	35.8	11.3	52.5	36.1	50.0	38.5	52.6	29.8
BART	26.2	19.7	67.7	11.3	61.0	10.2	49.2	31.7
HUMAN	22.0	18.6	16.7	81.8	14.3	84.3	46.8	37.1
(Shao et al., 2019). We evaluate the generation results from the 71 testing literal sentences. Each pair of texts (ours vs. a baseline / human reference) is given preference (win, lose or tie) by five people with proficiency in English. We use a set of four criteria adapted from Chakrabarty et al. (2021) to evaluate the generated outputs: 1) Fluency (Flu.): Which sentence is more fluent and grammatical? 2) Hyperbolicity (Hypo.): Which sentence is more hyperbolic? 3) Creativity (Crea.): Which sentence is more creative? 4) Relevance (Rel.): Which sentence is more relevant to the input literal sentence?
Automatic Evaluation Table 3 shows the automatic evaluation results of our system compared to different baselines. MOVER outperforms all three baselines on these two metrics. However, BLEU and BERTScore are far not comprehensive evaluation measures for our hyperbole generation task, since there are only a few modifications from literal to hyperbole and thus there is a lot of overlap between the generated sentence and the source sentence. Even a naive system (COPY in Table 3) that simply returns the literal input verbatim as output (Krishna et al., 2020) can achieve the highest performance. As a result, automatic metrics are not
Table 4: Pairwise human comparison between MOVER and other baseline systems. Win[W]% (Lose[L]%) is the percentage of MOVER considered better (worse) than a baseline system. The rest are ties.
System	Sentence	F.	H.	C.	R.
LITERAL	Being out of fashion is very bad.	-	-	-	-
MOVER	Being out of fashion is sheer hell.	-	-	-	-
R1	Their music will never go out of fashion.	T	W	L	W
R3	Being out of fashion is richly bad.	T	W	W	T
BART	Being out of fashion is very difficult.	T	W	W	T
HUMAN	Better be out of the world than out of the fashion.	W	W	L	W
Table 5: Pairwise evaluation results (Win[W], Lose[L], Tie[T]) in terms of Fluency, Hyperbolicity, Creativity and Relevance between MOVER and the generated outputs of the baseline systems. Changed text spans are in italic. More examples are given in Appendix B.
suitable for evaluating models that tend to copy input as output.
Human Evaluation The inter-annotator agreement of raw human evaluation results in terms of Fleiss' kappa (Fleiss, 1971) is 0.212, which indicates fair agreement (Landis and Koch, 1977). We take a conservative approach and only consider items with an absolute majority label, i.e., at least three of the five labelers choose the same preference (win/lose/tie). On average there are 61 (86%) items left for each baseline-criteria pair that satisfy this requirement. On this subset of items, Fleiss' Kappa increases to 0.278 (fair agreement). This degree of agreement is acceptable compared to other sentence revision tasks (e.g., 0.322 by Tan and Lee (2014) and 0.263 by Afrin and Litman (2018)) since it is hard to discern the subtle changing effect caused by local revision.
The annotation results in Table 4 are the absolute majority vote (majority  $> = 3$ ) from the 5 annotators for each item. The results show that our model mostly outperforms  $(\mathrm{Win}\% > \mathrm{Lose}\%)$  other baselines in the four metrics, except for creativity on R1. Because R1 directly retrieves human written hyperboles from HYPO-XL and is not strict about the relevance, it has the advantage of being more creative naturally. An example of this is shown in Table 5. Our model achieves a balance between generating hyperbolic output while preserving content, indicating the effectiveness of the "overgenerate-and-rank" mechanism. It is also worth noting that in terms of hyperbolicity, MOVER even performs better than human for  $16.7\%$  of the test cases. Table 5 shows a case where MOVER is rated higher than human.
Generated Hyperbole s	hypo(s)	para(s)	score(s)
You have ravished me away by a power I cannot resist.	0.962	0.954	0.962
You have ravished me away by a power I find unyielding to resist.	0.960	0.959	0.960
You have ravished me alive by a power I find difficult to resist.	0.954	0.931	0.954
You have driven me away by a power I find difficult to resist.	0.858	0.914	0.858
You have ravished me away with a beauty I find difficult to resist.	0.958	0.778	0.000
Table 6: Intermediate results of the input literal sentence "You have ravished me away by a power I find difficult to resist" after the over-generation steps (Section 3.2). Their ranking scores (Section 3.3) are displayed in the second to the fourth columns. Generated hyperbolic text spans are in italic.
Case Study Table 5 shows a group of generated examples from different systems. MOVER changes the phrase "very bad" in the original input to an extreme expression "sheer hell", which captures the sentiment polarity of the original sentence while providing a hyperbolic effect. R1 retrieves a hyperbolic but irrelevant sentence. R3 replaces the word "very" with "richly", which is not coherent to the context, although the word "richly" may introduce some hyperbolic effects. BART just generates a literal sentence, which seems to be a simple paraphrase. Although human reference provides a valid hyperbolic paraphrase, the annotators prefer our version in terms of fluency, hyperbolicity and relevance. Since our system makes fewer edits to the input than to the human reference, we are more likely to win in fluency and relevance. Also, the generated hyperbolic span "sheer hell" presents a more extreme exaggeration than "out of the world" according to the human annotators.
Table 6 shows the over-generation results for a literal input, with their hyperbole and paraphrase scores. On the one hand, our system can generate different hyperbolic versions, like the generated words "cannot", "unyielding", and "alive". This is reasonable since there might be multiple hyperbolic paraphrases for a single sentence. It is only for comparison with other baselines that we have to use the ranker to keep only one output, which inevitably undermines the strength of our approach. On the other hand, our ranker filters out the sentence if the infilled text violates the original meaning, which can be seen in the last row of Table 6. In this way, we gain explicit control over hyperbolicity and relevance through a scoring function, and endow MOVER with more explainability.
Despite the interesting results, we also observe the following types of errors in the generated outputs:
- The output is a paraphrase instead of hyperbole: "My aim is very certain"  $\rightarrow$  "My aim is very clear".
The degree of exaggeration is not enough: "The news has been exaggerated"  $\rightarrow$  "The news has been greatly exaggerated".
- The output is not meaningful: "I'd love to hang out every day"  $\rightarrow$  "I'd love to live every day". We believe that incorporating more commonsense knowledge and generating freeform hyperboles beyond word- or phrase-level substitutions are promising for future improvement.
Ablation Study We investigate the impact of removing partial or all information during the ranking stage. The results are shown in Table 3. Specifically, if we rank multiple generated outputs by only hyperbole score (w/o para score), or randomly select one as the output (w/o hypo ranker), the performance will become worse. Note that we do not report the ablation result for ranking only by paraphrase score (w/o hypo score), because it has the same problem with COPY: a generated sentence that directly copies the input will result in the highest paraphrase score and thus be selected as the final output.
Furthermore, we note that the experiments on R3 and BART also serve as ablation studies for the text infilling model in Section 3.2 as they substitute the fine-tuned BART with a retrieve-and-replace method and a non-fine-tuned BART, respectively.
Hyperbole Corpus Troiano et al. (2018) built the HYPO dataset consisting of 709 hyperbolic sentences with human-written paraphrases and lexically overlapping non-hyperbolic counterparts. Kong et al. (2020) also built a Chinese hyperbole
dataset with 2680 hyperboles. Our HYPO-L and HYPO-XL are substantially larger than HYPO and we hope they can facilitate computational research on hyperbole detection and generation.
Figurative Language Generation As a figure of speech, hyperbole generation is related to the general task of figurative language generation. Previous studies have tackled the generation of metaphor (Yu and Wan, 2019; Stowe et al., 2020; Chakrabarty et al., 2021; Stowe et al., 2021), simile (Chakrabarty et al., 2020b; Zhang et al., 2021), idiom (Zhou et al., 2021), pun (Yu et al., 2018; Luo et al., 2019b; He et al., 2019; Yu et al., 2020), and sarcasm (Chakrabarty et al., 2020a). HypoGen (Tian et al., 2021) is a concurrent work with ours on hyperbole generation. However, we share a different point of view and the two methods are not directly comparable. They tackle the generation of clause-level hyperboles and frame it as a sentence completion task, while we focus on the word-level or phrase-level ones and frame it as a sentence editing task. In addition, their collected hyperboles and generated outputs are limited to the "so...that" pattern while we do not posit constraints on sentence patterns.
Unsupervised Text Style Transfer Recent advances on unsupervised text style transfer (Hu et al., 2017; Subramanian et al., 2018; Luo et al., 2019a; Zeng et al., 2020) focus on transferring from one text attribute to another without parallel data. Jin et al. (2020) classify existing methods into three main branches: disentanglement, prototype editing, and pseudo-parallel corpus construction. We argue that hyperbole generation is different from text style transfer. First, it is unclear whether "literal" and "hyperbolic" can be treated as "styles", especially the former one. Because "literal" sentences do not have any specific characteristics at all, there are no attribute markers (Li et al., 2018) in the input sentences, and thus many text style transfer methods based on prototype editing cannot work. Second, the hyperbolic span can be lexically separable from, yet strongly dependent on, the context (Section 3.1). On the contrary, disentanglement-based approaches for text style transfer aim to separate content and style via latent representation learning. Third, MOVER could also be used for constructing pseudo-parallel corpus of literal-hyperbole pairs given enough literal sentences as input, which is beyond the scope of this work.
Unsupervised Paraphrase Generation Unsupervised paraphrase generation models (Wieting et al., 2017; Zhang et al., 2019a; Roy and Grangier, 2019; Huang and Chang, 2021) do not require paraphrase pairs for training. Although hyperbole generation also needs content preservation and lacks parallel training data, it is still different from paraphrase generation because we need to create a balance between paraphrasing and exaggerating. We further note that the task of metaphor generation (Chakrabarty et al., 2021), which replaces a verb (e.g., "The scream filled the night"  $\rightarrow$  "The scream pierced the night"), is also independent of paraphrase generation.
We tackle the challenging task of figurative language generation: hyperbole generation from literal sentences. We build the first large-scale hyperbole corpus HYPO-XL and propose an unsupervised approach MOVER for generating hyperbole in a controllable way. The results of automatic and human evaluation show that our model is successful in generating hyperbole. The proposed generation pipeline has better interpretability and flexibility compared to the potential end-to-end methods. In future, we plan to apply our "mask-overgeneraterank" approach to the generation of other figurative languages, such as metaphor and irony.
7 Ethical Consideration
The HYPO-XL dataset is collected from a public website Sentencedict.com, and we have asked the website owners permission to use their data for research purposes. There is no explicit detail that leaks a user's personal information including name, health, racial or ethnic origin, religious affiliation or beliefs, sexual orientation, etc.
Our proposed method MOVER utilizes the pretrained language model, which may inherit the bias in the massive training data. It is possible that MOVER is used for malicious purposes, since it does not explicitly filter input sentences with toxicity, bias or offensiveness. Therefore, the output generated by MOVER could potentially be harmful to certain groups or individuals. It is important that interested parties carefully address those biases before applying the model to real-world situations.
This work was supported by National Science Foundation of China (No. 62161160339), National Key R&D Program of China (No.2018YFB1005100), State Key Laboratory of Media Convergence Production Technology and Systems and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We appreciate the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.
Tazin Afrin and Diane Litman. 2018. Annotation and classification of sentence-level revision improvement. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 240-246, New Orleans, Louisiana. Association for Computational Linguistics.
Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. 2020a. R^3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7976-7986, Online. Association for Computational Linguistics.
Tuhin Chakrabarty, Smaranda Muresan, and Nanyun Peng. 2020b. Generating similes effortlessly like a pro: A style transfer approach for simile generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6455-6469, Online. Association for Computational Linguistics.
Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, and Nanyun Peng. 2021. MERMAID: metaphor generation with symbolism and discriminative decoding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 4250-4261. Association for Computational Linguistics.
Claudia Claridge. 2010. Hyperbole in English: A corpus-based study of exaggeration. Cambridge University Press.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
He He, Nanyun Peng, and Percy Liang. 2019. Pun generation with surprise. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1734-1744.
Michael Heilman and Noah A Smith. 2009. Question generation via overgenerating transformations and ranking. Technical report, Carnegie-Mellon Univ Pittsburgh pa language technologies instT.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In International Conference on Machine Learning, pages 1587-1596. PMLR.
Kuan-Hao Huang and Kai-Wei Chang. 2021. Generating syntactically controlled paraphrases without using annotated parallel pairs. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1022-1033.
Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2020. Deep learning for text style transfer: A survey. arXiv preprint arXiv:2011.00416.
Li Kong, Chuanyi Li, Jidong Ge, Bin Luo, and Vincent Ng. 2020. Identifying exaggerated language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7024-7034, Online. Association for Computational Linguistics.
Roger J. Kreuz and Richard M. Roberts. 1993. The empirical study of figurative language in literature. Poetics, 22(1):151-169.
Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737-762, Online. Association for Computational Linguistics.
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159-174.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Xu Sun, and Zhifang Sui. 2019a. A dual reinforcement learning framework for unsupervised text style transfer. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5116-5122. ijcai.org.
Fuli Luo, Shunyao Li, Pengcheng Yang, Lei Li, Baobao Chang, Zhifang Sui, and Xu Sun. 2019b. Pun-gan: Generative adversarial network for pun generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3388-3393.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
Nikola I Nikolov, Eric Malmi, Curtis Northcutt, and Loreto Parisi. 2020. Rapformer: Conditional rap lyrics generation with denoising autoencoders. In Proceedings of the 13th International Conference on Natural Language Generation, pages 360-373.
Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252, Copenhagen, Denmark. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of
the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Aurko Roy and David Grangier. 2019. Unsupervised paraphrasing without translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6033-6039.
Zhihong Shao, Minlie Huang, Jiangtao Wen, Wenfei Xu, and Xiaoyan Zhu. 2019. Long and diverse text generation with planning-based hierarchical variational model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3257-3268, Hong Kong, China. Association for Computational Linguistics.
Kevin Stowe, Tuhin Chakrabarty, Nanyun Peng, Smaranda Muresan, and Iryna Gurevych. 2021. Metaphor generation with conceptual mappings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 6724-6736. Association for Computational Linguistics.
Kevin Stowe, Leonardo Ribeiro, and Iryna Gurevych. 2020. Metaphoric paraphrase generation. arXiv preprint arXiv:2002.12854.
Sandeep Subramanian, Guillaume Lample, Eric Michael Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2018. Multiple-attribute text style transfer. arXiv preprint arXiv:1811.00552.
Chenhao Tan and Lillian Lee. 2014. A corpus of sentence-level revisions in academic writing: A step towards understanding statement strength in communication. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 403-408, Baltimore, Maryland. Association for Computational Linguistics.
Yufei Tian, Arvind krishna Sridhar, and Nanyun Peng. 2021. HypoGen: Hyperbole generation with commonsense and counterfactual knowledge. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1583-1593, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Enrica Troiano, Carlo Strapparava, Gözde Özbal, and Serra Sinem Tekiroğlu. 2018. A computational exploration of exaggeration. In Proceedings of the