SlowGuess/NAACL_2022_OCR huggingface dataset:
Yunxiang Zhang, Xiaojun Wan
{yx.zhang,wanxiaojun}@pku.edu.cn
Despite being a common figure of speech, hyperbole is under-researched in Figurative Language Processing. In this paper, we tackle the challenging task of hyperbole generation to transfer a literal sentence into its hyperbolic paraphrase. To address the lack of available hyperbolic sentences, we construct HYPO-XL, the first large-scale English hyperbole corpus containing 17,862 hyperbolic sentences in a non-trivial way. Based on our corpus, we propose an unsupervised method for hyperbole generation that does not require parallel literal-hyperbole pairs. During training, we fine-tune BART (Lewis et al., 2020) to infill masked hyperbolic spans of sentences from HYPO-XL. During inference, we mask part of an input literal sentence and over-generate multiple possible hyperbolic versions. Then a BERT-based ranker selects the best candidate by hyperbolicity and paraphrase quality. Automatic and human evaluation results show that our model is effective at generating hyperbolic paraphrase sentences and outperforms several baseline systems.
Hyperbole is a figure of speech that deliberately exaggerates a claim or statement to show emphasis or express emotions. If a referent has a feature X, a hyperbole exceeds the credible limits of fact in the given context and presents it as having more of that X than warranted by reality (Claridge, 2010). Take the following example, "I won't wait for you: it took you centuries to get dressed." It over-blows the time for someone to get dressed with a single word "centuries" and thus creates a heightened effect. From a syntactic point of view, Claridge (2010) classifies hyperbole into word-level, phrase-level and clause-level types, and conclude that the former two types are more common in English. Although hyperbole is considered as the second most frequent figurative device (Kreuz and Roberts,
1993), it has received less empirical attention in the NLP community. Recently Tian et al. (2021) addressed the generation of clause-level hyperbole. In this paper, we instead focus on word-level and phrase-level hyperbole, which can be unified as span-level hyperbole.
To tackle the hyperbole generation problem we need to address three main challenges:
- The lack of training data that either consists of large-scale hyperbolic sentences or literal-hyperbole pairs, which are necessary to train an unsupervised or supervised model.
- The tendency of generative language models to produce literal text rather than hyperboles.
Trade-off between content preservation and hyperbolic effect of the generated sentences.
In order to address the above challenges, we propose MOVER (Mask, OVEr-generate and Rank), an unsupervised approach to generating hyperbolic paraphrase from literal input. Our approach does not require parallel data for training, thus alleviating the issue of scarce data. Still, we need a non-parallel corpus containing as much hyperbolic sentences as possible. To this end, we first build a large-scale English hyperbole corpus HYPO-XL in a weakly supervised way.
Based on the intuition that the hyperbolic effect of a sentence is realized by a single word or phrase within it, we introduce a sub-task of hyperbolic span extraction. We identify several possible n-grams of a hyperbolic sentence that can cause the hyperbolic bent with syntactic and semantic features. We apply this masking approach to sentences in HYPO-XL and teach a pretrained seq2seq transformer, BART (Lewis et al., 2020), to infill the words in missing hyperbolic spans. This increases the probability of generating hyperbolic texts instead of literal ones. During inference, given a single literal sentence, our system provides multiple masked versions for inputs to BART and gen
Human Language Technologies, pages 6018 - 6030
erates potential hyperbolic sentences accordingly. To select the best one for output, we leverage a BERT-based ranker to achieve a satisfying tradeoff between hyperbolicity and paraphrase quality. Our contributions are three-fold:
- We construct the first large-scale hyperbole corpus HYPO-XL in a non-trivial way. The corpus is publicly available, $^{1}$  contributing to the Figurative Language Processing (FLP) community by facilitating the development of computational study of hyperbole.
- We propose an unsupervised approach for hyperbole generation that falls into the "overgenerate-and-rank" paradigm (Heilman and Smith, 2009).
- We benchmark our system against several baselines and we compare their performances by pair-wise manual evaluations to demonstrate the effectiveness of our approach.
2 HYPO-XL: Hyperbole Corpus Collection
The availability of large-scale corpora can facilitate the development of figurative language generation with pretrained models, as is shown by Chakrabarty et al. (2020b) on simile generation and Chakrabarty et al. (2021) on metaphor generation. However, datasets for hyperbole are scarce. Troiano et al. (2018) built an English corpus HYPO containing 709 triplets [hypo, para, non_hypo], where hypo refers to a hyperbolic sentence, para denotes the literal paraphrase of hypo and non_hypo means a non-hyperbolic sentence that contains the same hyperbolic word or phrase as hypo but with a literal connotation. The size of this dataset is too small to train a deep learning model for hyperbole detection and generation. To tackle the lack of hyperbole data, we propose to enlarge the hyperbolic sentences of HYPO in a weakly supervised way and build a large-scale English corpus of 17,862 hyperbolic sentences, namely HYPO-XL. We would like to point out that this is a non-parallel corpus containing only hyperbolic sentences without their paraphrase counterparts, because our hyperbole generation approach (Section 3) does not require parallel training data.
The creation of HYPO-XL consists of two steps:
1. We first train a BERT-based binary classifier
on HYPO and retrieve possible hyperbolic sentences from an online corpus.
2. We manually label a subset of the retrieved sentences, denoted HYPO-L, and retrain our hyperbole detection model to identify hyperbolic sentences from the same retrieval corpus with higher confidence.
2.1 Automatic Hyperbole Detection
Hyperbole detection is a supervised binary classification problem where we predict whether a sentence is hyperbolic or not (Kong et al., 2020). We fine-tune a BERT-base model (Devlin et al., 2019) on the hyperbole detection dataset HYPO (Troiano et al., 2018). In experiment, we randomly split the data into 567  $(80\%)$  hyperbolic sentences, with their literal counterparts (para and non_hypo) as negative samples, in training set and 71  $(10\%)$  in development set and 71  $(10\%)$  in test set. Our model achieves an accuracy of  $80\%$  on the test set, which is much better than the highest reported accuracy  $(72\%)$  of traditional algorithms in Troiano et al. (2018).
Once we obtain this BERT-based hyperbole detection model, the next step is to retrieve hyperbolic sentences from a corpus. Following Chakrabarty et al. (2020a), we use Sentencedict.com, an online sentence dictionary as the retrieval corpus. We remove duplicate and incomplete sentences (without initial capital) in the corpus, resulting in a collection of 767,531 sentences. Then we identify 93,297  $(12.2\%)$  sentences predicted positive by our model as pseudo-hyperbolic.
2.2 HYPO-L: Human Annotation of Pseudo-labeled Data
Due to the small size of training set, pseudo-labeled data tend to have lower confidence score (i.e., the prediction probability). To improve the precision of our model, $^{3}$  we further fine-tune it with our human-annotated data, namely HYPO-L. We randomly sample 5,000 examples from the 93,297 positive predictions and invite students with proficiency in English to label them as hyperbolic or not. For each sentence, two annotators provide their judgements. We only keep items with unanimous judgments (i.e. both of the two annotators mark the sentence as hyperbolic or non-hyperbolic) to ensure the reliability
$^{2}$ https://sentencedict.com/
<sup>3</sup>Given the massive hyperboles in the "wild" (i.e., the retrieval corpus) we do not pursue recalling more hyperboles at the risk of hurting precision (Zhang et al., 2021).
1Code and data are available at https://github. com/yunx-z/MOVER.
Dataset	# Hypo.	# Non.	# Para.	# Total
HYPO	709	698	709	2,116
HYPO-L	1,007	2,219	-	3,226
HYPO-XL	17,862	-	-	17,862
Table 1: Comparison of different hyperbole datasets (corpora) in terms of hyperbolic (Hypo.), non-hyperbolic (Non.) and paraphrase (Para.) sentences.
Measurement	Value
% Non-hypo	6%
# Avg hypo span tokens	2.23
% Long hypo spans (> 1 token)	37%
# Distinct hypo spans	85
# Distinct POS-ngrams of hypo spans	39
Table 2: Statistics of 100 random samples from HYPO-XL, of which 6 are actually non-hyperboles ("Non-hypo"). The statistics of hyperbolic text spans ("hypo span") are calculated for the rest 94 real hyperboles.
of annotated data. In this way, 3,226 (64.5%) out of 5,000 annotations are left in HYPO-L. This percentage of unanimous judgments (i.e., raw agreement, RA) is comparable to  $58.5\%$  in the creation of HYPO (Troiano et al., 2018). To be specific, HYPO-L consists of 1,007 (31.2%) hyperbolic sentences (positive samples) and 2,219 (68.8%) literal ones (negative samples).
We continue to train the previous HYPO-finetuned BERT on HYPO-L and the test accuracy is  $80\%$ ,<sup>4</sup> which we consider as an acceptable metric for hyperbole detection. Finally we apply the BERT-based detection model to the retrieval corpus again and retain sentences whose prediction probabilities for positive class exceed a certain threshold.<sup>5</sup> This results in HYPO-XL, a large-scale corpus of 17,862  $(2.3\%)$  hyperbolic sentences. We provide a brief comparison of HYPO, HYPO-L and HYPO-XL in Table 1 to further clarify the data collection process.
2.3 Corpus Analysis
Since HYPO-XL is built in a weakly supervised way with only a few human labeled data samples, we conduct a quality analysis to investigate how many sentences in the corpus are actually hyper
bolic. We randomly sample 100 instances from HYPO-XL and manually label them as hyperbole or non-hyperbole. Only six sentences are not hyperbole. This precision of  $94\%$  is on par with  $92\%$  on another figurative language corpus of simile (Zhang et al., 2021). Actually we can tolerate a bit noise in the corpus since the primary goal of HYPO-XL is to facilitate hyperbole generation instead of detection, and a small proportion of non-hyperbole sentences as input will not harm our proposed method. Table 2 shows the statistics of hyperbolic text spans (defined in Section 3.1) for the rest 94 real hyperboles. We also provide additional analyses in Appendix A.
3 Hyperbole Generation
We propose an unsupervised approach to generate hyperbolic paraphrase from a literal sentence with BART (Lewis et al., 2020) such that we do not require parallel literal-hyperbole pairs.<sup>7</sup> An overview of our hyperbole generation pipeline is shown in Figure 1. It consists of two steps during training:
1. Mask. Given a hyperbolic sentence from HYPO-XL, we identify multiple text spans that can possibly produce the hyperbolic meaning of a sentence, based on two features (POS n-gram and unexpectedness score). For each identified text span, we replace it with the <mask> token to remove hyperbolic attribute of the input.  $N$  text spans will result in  $N$  masked inputs, respectively.
2. Infill. We fine-tune BART to fill the masked spans of input sentences. The model learns to generate hyperbolic words or phrases that are pertinent to the context.
During inference, there are three steps:
1. Mask. Given a literal sentence, we apply POS-gram-only masking to produce multiple input sentences.
2. Over-generate. BART generates one sentence from a masked input, resulting in multiple candidates.
3. Rank. Candidates are ranked by their hyperbolicity and relevance to the source literal sentence. The one with highest score is selected as the final output.
4We separate  $10\%$  data of HYPO-L for development and another  $10\%$  for testing.
Based on manual inspection of predicted results, we set the threshold as 0.8 to trade-off between precision and recall.
<sup>6</sup>We further explain the reason in Section 3.2
<sup>7</sup>We note that training our model still relies on instances from a specialized hyperbole corpus.
Figure 1: Overview of our approach to unsupervised hyperbole generation. Literal sentences are in yellow boxes, masked sentences are in blue boxes and hyperbolic sentences are in green boxes.
We dub our hyperbole generation system MOVER (Mask, OVEr-generate and Rank). We apply masking technique to map both the hyperbolic (training input) and literal (test input) sentences into a same "space" where the masked sentence can be transformed into hyperbole by BART. It falls into the "overgenerate-and-rank" paradigm (Heilman and Smith, 2009) since many candidates are available after the generation step. The remainder of this section details the three main modules: hyperbolic span masking (Section 3.1), BART-based span infilling (Section 3.2) and the hyperbole ranker (Section 3.3).
3.1 Mask: Hyperbolic Span Masking
We make a simple observation that the hyperbolic effect of a sentence is commonly localized to a single word or a phrase, which is also supported by a corpus-based linguistic study on hyperbole (Claridge, 2010). For example, the word marathon in "My evening jog with Bill turned into a marathon" overstates the jogging distance and causes the sentence to be hyperbolic. This inspires us to leverage the "delete-and-generate" strategy (Li et al., 2018) for hyperbole generation. Concretely, a literal sentence can be transformed into its hyperbolic counterpart via hyperbolic span extraction and replacement. We propose to extract hyperbolic spans based on POS n-gram (syntactic) and unexpectedness (semantic) features.
POS N-gram We extract POS n-gram patterns of hyperbole from the training set of HYPO dataset and obtain 262 distinct POS n-grams. As a motivating example, the following three hyperbolic spans, "faster than light", "sweeter than honey", "whiter than snow", share the same POS n-gram of "JJR+IN+NN".
Unexpectedness Hyperbolic spans are less coherent with the literal contexts and thus their vector representations are distant from the context vectors. Troiano et al. (2018) have verified this intuition with the unexpectedness metric. They define the unexpectedness score  $U_{s}$  of a sentence  $s$  with the token sequence  $\{x_0, x_1, \dots, x_N\}$  as the average cosine distance among all of its word pairs.
$$
U _ {s} = \underset {i, j \in [ 0, N ], i \neq j} {\text {a v e r a g e}} \left(\operatorname {c o s i n e} _ {-} \operatorname {d i s t a n c e} \left(v _ {i}, v _ {j}\right)\right) \tag {1}
$$
where  $v_{i}$  denotes the word embedding vector of token  $x_{i}$ . Similarly, we define the unexpectedness score  $U_{n}$  of an n-gram  $\{x_{k},x_{k + 1},\dots,x_{k + n - 1}\}$  in a sentence  $s$  as the average cosine distance among word pairs that consist of one word inside the n-gram and the other outside.
$$
U _ {n} = \underset { \begin{array}{c} i \in [ k, k + n - 1 ] \\ j \in [ 0, k - 1 ] \cup [ k + n, N ] \end{array} } {\text {a v e r a g e}} (\operatorname {c o s i n e \_ d i s t a n c e} (v _ {i}, v _ {j})) \tag {2}
$$
The hyperbolic spans are not explicitly provided in the HYPO dataset, so we take the maximum word overlap between hypo and non_hypo (Section 2) as the hyperbolic spans.
Figure 2: A visualization of the cosine distance matrix of the hyperbolic sentence "I've drowned myself trying to help you".
Text spans with higher unexpectedness scores tend to be hyperbolic. Figure 2 illustrates the cosine distance of the word pairs in the sentence "I've drowned myself trying to help you". The words in the span "drowned myself" are distant from other words in terms of word embedding similarity.
For the masking step during training, we extract all text spans in the original input hyperbolic sentences that match one of the hyperbolic POS n-grams. Then we rank them by their unexpectedness scores and choose top-3 items as the masked spans. For the masking step during inference, we simply mask all the spans that match hyperbolic POS n-grams, since the span unexpectedness score is not applicable to a literal input. We evaluate the accuracy of our hyperbolic span masking approach on the development set of the HYPO dataset. The proportion of exact match (EM) (Rajpurkar et al., 2016) between our top-3 masked spans with the human-labeled spans is  $86\%$ , which shows that our simple method based on the above-mentioned hand-crafted features is effective for the task of hyperbolic span extraction.
3.2 Over-generate: Hyperbolic Text Infilling with BART
In order to generate hyperbolic and coherent text from the masked span, we leverage the text span infilling ability of BART (Lewis et al., 2020), a pretrained sequence2sequence model with a denois
ing autoencoder and an autoregressive autodecoder. During its pretraining, it learns to reconstruct the corrupted noised text. One of the noising transformations is random span masking, which teaches BART to predict the multiple tokens missing from a span. During our training process, we fine-tune BART by treating the masked hyperbolic sentence as the encoder source and the original one as the decoder target. This can change the probability distribution when decoding tokens and increase the chance of generating a hyperbolic, rather than literal, text span conditioned on the context. During inference, BART fills the masked span of a literal sentence with possible hyperbolic words.
Note that if the masked span of an input sentence is actually not hyperbolic, then fine-tuning on this example will just enhance the reconstruction ability of BART, which will not exert negative effects on hyperbole generation. This can give rise to our tolerance for non-hyperbolic sentences in the training corpus (Section 2.3) and non-hyperbolic masked span (Section 3.1).
3.3 Rank: Hyperbole Ranker
Recall that for each literal input during inference, we apply POS-ngram-based masking, produce different masked versions of the sentence, and generate multiple output candidates. Obviously, not all masking spans are suitable for infilling hyperbolic words due to the noise of masking. To select the best candidate for final output, we introduce a hyperbole ranker which sorts candidate sentences by their degree of hyperbolicity and relevance to the source inputs. For hyperbolicity evaluation, we leverage the BERT-based hyperbole detection model fine-tuned on HYPO and HYPO-L (Section 2.2) to assign a hyperbole score (i.e., prediction probability) for each candidate. For the evaluation of content preservation, we train a pairwise model to predict whether hyperbolic sentence A is a paraphrase of a literal sentence B. To this end, we use the distilled RoBERTa-base model checkpoint<sup>10</sup> pretrained on large scale paraphrase data provided by Sentence-Transformer (Reimers and Gurevych, 2019). It calculates the cosine similarity between the literal input and the candidate as the paraphrase score. We fine-tune the checkpoint on the training set of the HYPO dataset, where we treat the pairs of hypo and para as positive examples, and
<sup>9</sup>This means that at least 2/3 of the identified spans should not be hyperbolic, but this will not harm the training of our hyperbole generation model, which is explained in Section 3.2
10https://huggingface. co/sentence-transformers/ paraphrase-distilroberta-base-v1
pairs of hypo and non_hypo as negative examples (Section 2). The accuracy on test set is  $93\%$ .
Now that we obtain the hyperbole score  $\mathrm{hypo}(c)$  and the paraphrase score  $\operatorname{para}(c)$  for candidate  $c$ , we propose an intuitive scoring function score  $(\cdot)$  as below:
$$
\operatorname {s c o r e} (s) = \left\{ \begin{array}{l l} \operatorname {h y p o} (s) & \operatorname {p a r a} (s) \in (\gamma , 1 - \epsilon) \\ 0 & \text {e l s e} \end{array} \right. \tag {3}
$$
Here we filter out a candidate if its paraphrase score is lower than a specific threshold  $\gamma$  or it is almost the same as the original input (i.e., the paraphrase score is extremely close to 1). For diversity purposes, we do not allow our system to simply copy the literal input as its output. We then rank the remaining candidates according to their hyperbole score and select the best one as the final output.[11]
There are no existing models applied to the task of word- or phrase-level hyperbole generation. To compare the quality of the generated hyperboles, we benchmark our MOVER system against three baseline systems adapted from related tasks.
4.1 Baseline Systems
Retrieve (R1) Following Nikolov et al. (2020), we implement a simple information retrieval baseline, which retrieves the closest hyperbolic sentence as the output (i.e., the highest cosine similarity) from HYPO-XL, using the hyperbole paraphrase detection model para(·) in Section 3.3. The outputs of this baseline system should be hyperbolic yet have limited relevance to the input.
Retrieve, Replace and Rank (R3) We first retrieve the top-5 most similar sentences from HYPOXL like the R1 baseline. Then we apply hyperbolic span extraction in Section 3.1 to find 3 text spans for each retrieved sentence. We replace the text spans in a literal input sentence with retrieved hyperbolic spans if two spans share the same POS n-gram. Since this replacement method may result in multiple modified sentences, we select the best one with the hyperbole ranker in Section 3.3. If there are no matched text spans, we fall back to R1 baseline and return the most similar retrieved sentence verbatim. In fact, this baseline substitutes the
BART generation model in MOVER system with a simpler retrieval approach, which can demonstrate the hyperbole generation ability of BART.
BART Inspired by Chakrabarty et al. (2020b), we replace the text infilling model in Section 3.2 with a non-fintuned off-the-shelf BART, $^{12}$  because BART has already been pretrained to predict tokens from a masked span.
We use 16,075 (90%) samples in HYPO-XL for training our MOVER system and the rest 1,787 sentences for validation. For POS Tagging in Section 3.1, we use Stanford CoreNLP (Manning et al., 2014). For the word embedding, we use 840B 300-dimension version of GloVe vectors (Pennington et al., 2014). For BART in Section 3.2, we use the BART-base checkpoint instead of BART-large due to limited computing resources and leverage the implementation by Huggingface (Wolf et al., 2020). We fine-tune pretrained BART for 16 epochs. For the parameters of the hyperbole ranker in Section 3.3, we set  $\gamma = 0.8$  and  $\epsilon = 0.001$  by manual inspection of the ranking results on the development set of the HYPO dataset.
4.3 Evaluation Criteria
Automatic Evaluation BLEU (Papineni et al., 2002) reflects the lexical overlap between the generated and the ground-truth text. BERTScore (Zhang et al., 2020) computes the similarity using contextual embeddings. These are common metrics for text generation. We use the 71 literal sentences (para) in the test set of HYPO dataset as test inputs and their corresponding hyperbolic sentences (hypo) as gold references. We report the BLEU and BERTScore metrics for generated sentences compared against human written hyperboles.
Human Evaluation Automated metrics are not reliable on their own to evaluate methods to generate figurative language (Novikova et al., 2017) so we also conduct pair-wise comparisons manually
<sup>12</sup>We also tried to fine-tune BART on the 567 literal-hyperbole pairs from the training set of HYPO dataset in an end-to-end supervised fashion, but the model just copy the input for all instances (same as COPY in Table 3) and is unable to generate meaningful output due to small amount of training data. Besides, we test the performance of a BART-based paraphrase generation model, which is BART finetuned on QQP (Wang et al., 2018) and PAWS (Zhang et al., 2019b) datasets. We still find that  $50\%$  of the outputs from the paraphrase model just copy the input. Therefore, we do not consider these two BART-based systems hereafter.