SlowGuess/NAACL_2021_OCR huggingface dataset:
evaluate on four dimensions for 900 utterances, we have a total of 3600 evaluations. Each criteria was rated on a likert scale from 1 (not at all) to 5 (very). Each group of utterances was rated by three separate Turkers, resulted in 42, 48, 44 and 53 Turkers for the four evaluation tasks respectively. We pay them at a rate of $15 per hour.
Based on the semantic similarity metric shown in column 1 of Table 4, our system MERMAID is better in preserving the meaning of the input than the other baselines. As mentioned, we calculate BLEU-2 and BERTScore between system outputs and human references. MERMAID is better than the other baselines according to BERTScore. In terms of BLEU-2, MERMAID is second best.
Table 5 shows the average scores for the human evaluation on four metaphor quality criteria for MERMAID, the baselines, and human written metaphors on the test set. The inter-annotator agreements computed using Krippendorff's alpha for Creativity, Meaning, Fluency and Metaphoricity are 0.44, 0.42, 0.68, 0.52 respectively. The results demonstrate that MERMAID is significantly better
Literal	System	Metaphor	Flu	Mea	Crea	Meta
The scream filled the night	HUMAN1	The scream pierced the night	4.3	5.0	3.7	4.0
HUMAN2	The scream covered the night	2.7	4.0	3.0	3.0
LEXREP	The scream held the night	1.7	3.7	2.0	1.7
META_M	The scream opened the night	1.0	1.0	1.0	1.0
BART	The scream filled the night	2.3	1.0	2.3	1.0
MERMAID	The scream pierced the night	4.3	5.0	3.7	4.0
The wildfire spread through the forest at an amazing speed	HUMAN1	The wildfire ravaged through the forest at an amazing speed	4.7	4.3	4.0	3.0
HUMAN2	The wildfire leapt through the forest at an amazing speed	3.7	3.0	5.0	3.7
LEXREP	The wildfire saw through the forest at an amazing speed	1.3	1.0	2.7	3.3
META_M	The wildfire grows through the forest at an amazing speed	3.7	2.7	2.7	4.0
BART	The wildfire swept through the forest at an amazing speed	4.0	3.7	4.7	4.0
MERMAID	The wildfire danced through the forest at an amazing speed	3.0	4.0	4.0	3.7
My heart beats when he walks in the room	HUMAN1	My heart skips when he walks in the room	4.7	5.0	4.0	4.3
HUMAN2	My heart sings when he walks in the room	5.0	4.3	3.7	3.3
LEXREP	My heart made when he walks in the room	1.0	1.0	1.0	1.0
META_M	My heart came when he walks in the room	1.7	1.0	1.3	1.3
BART	My heart sings when he walks in the room	5.0	4.3	3.7	3.7
MERMAID	My heart jumps when he walks in the room	4.7	4.7	4.3	4.0
After a glass of wine, he relaxed up a bit	HUMAN1	After a glass of wine, he loosened up a bit	4.7	5.0	5.0	4.0
HUMAN2	After a glass of wine, he unfurled up a bit	2.0	5.0	2.0	3.7
LEXREP	After a glass of wine, he followed up a bit	3.7	1.0	2.7	1.7
META_M	After a glass of he touched up a bit	1.3	1.0	1.7	2.0
BART	After a glass of wine, he dried up a bit	2.7	1.0	2.3	2.0
MERMAID	After a glass of wine, he loosened up a bit	4.3	5.0	5.0	3.7
The tax cut will help the economy	HUMAN1	The tax cut will uplift the economy	4.7	5.0	4.7	4.0
HUMAN2	The tax cut will fertilize the economy	4.0	4.3	4.3	3.7
LEXREP	The tax cut will bring the economy	1.7	3.0	2.7	1.7
META_M	The tax cut will prevent the economy	1.7	1.0	2.0	1.0
BART	The tax cut will strengthen the economy	5.0	5.0	4.3	3.7
MERMAID	The tax cut will stimulate the economy	5.0	4.7	3.7	4.0
I tried to resolve things over between them	HUMAN1	I tried to tide things over between them	4.3	3.0	3.7	4.3
HUMAN2	I tried to patch things over between them	4.7	4.7	5.0	2.0
LEXREP	I tried to push things over between them	3.3	1.0	2.3	2.0
META_M	I tried to make things over between them	4.0	1.0	2.7	2.7
BART	I tried to put things over between them	4.7	2.0	3.0	2.7
MERMAID	I tried to smooth things over between them	4.7	4.7	5.0	4.0
Table 6: Examples of generated outputs from different systems (with human written metaphors as references). We show average scores (over three annotators) on a 1-5 scale with 1 denotes the worst and 5 be the best. The italics texts in the literal column represent the verb while those in Metaphor column represents the generated metaphorical verb. Boldface indicates the best results.
than the baselines on all four criteria  $(p < .001$  according to approximate randomization test).
Table 6 presents several generation outputs from different systems along with human judgements on individual criteria. We observe that incorporating a discriminator often guides our model to generate better metaphors than the already strong baseline using BART. Finally, incorporating symbolic meaning in data creation step helps our model to maintain the same meaning as the input.
6 Task Based Evaluation
Metaphors are frequently used by creative writing practitioners, in particular poets, to embellish their work. We posit that MERMAID can be used to edit literal sentences in poems to further enhance creativity. To test this hypothesis, we first crawl origi
Figure 4: Percentage of Preference of Original Quatrains vs Quatrains rewritten by MERMAID
nal poems submitted by authors from the sub-reddit OCPOETRY. The poems are of variable lengths, so to ensure parity we break them into Quatrains (four sentence stanza). We randomly sample 50 such Quatrains containing at least one sentence with a
And the hills have a shimmer of light between,
And the valleys are covered with misty veils,
And ..........,
......
And the hills have a shimmer of light between,
And the valleys are wrapped with misty veils,
And ..........,
Leaves on a maple, burst red with the shorter days;
Falling to the ground.
......
Leaves on a maple, burgeoning red with the shorter days;
Falling to the ground.
......
Table 7: Example Quatrains from reddit where MERMAID rewrites a sentence containing a literal verb to make it metaphorical.
literal verb in it. We use our metaphor detection model (Section 2.1) to detect literal verbs.
We then select a sentence containing a literal verb from each Quatrain and use MERMAID to re-write it so that the resulting output is metaphorical. We ignore common verbs like is, was, are, were, have, had. If there are more than one sentence in Quatrain with literal verbs, we choose the sentence with a literal verb that has the highest probability for being literal. For sentences with multiple literal verbs, we choose the verb with highest literal probability.
Our goal is to see if re-written poems are qualitatively better than the original forms. To do this, we hire Turkers from Amazon Mechanical Turk and present them with hits where the task is to choose the better version between the original Quatron and the re-written version. 15 Turkers were recruited for the task. Each Quatron was evaluated by 3 distinct Turkers. Table 7 shows metaphorical transformations by a MERMAID Figure 4 shows that poems rewritten by MERMAID were considered better by the Turkers.
Most researchers focused on identification and interpretation of metaphor, while metaphor generation is relatively under-studied.
7.1 Metaphor Detection
For metaphor detection, researchers focused on variety of features, including unigrams, imageability, sensory features, WordNet, bag-of-words features (Klebanov et al., 2014; Tsvetkov et al., 2014; Shutova et al., 2016; Tekiroglu et al., 2015; Hovy et al., 2013; Koper and im Walde, 2016).
With advent of deep learning approaches, Gao et al. (2018) used BiLSTM models based on GloVe (Pennington et al., 2014) and ELMo word vectors (Peters et al., 2018) to detect metaphoric verbs. Inspired by the linguistic theories, MIP (Semino et al., 2007; Steen, 2010) and SPV (Wilks, 1975, 1978), Mao et al. (2019) proposed two detection models consisting of BiLSTM with attention mechanisms that relied on GloVe and ELMo embeddings. Recent work on metaphor detection have also used pretrained language models (Su et al., 2020; Gong et al., 2020). While we focus on metaphor generation, we use (Devlin et al., 2018) to detect metaphoric verbs to create parallel data and (Liu et al., 2019) to rescore our generated hypothesis during decoding.
7.2 Metaphor Generation
Some early works made contributions to use template and heuristic-based methods (Abe et al., 2006; Terai and Nakagawa, 2010) to generate "A is like B" sentences, more popularly referred to as similes. Chakrabarty et al. (2020) concentrated on simile generation, applying seq2seq model to paraphrase a literal sentence into a simile. Other attempts learned from the mappings of different domains and generated conceptual metaphors of pattern "A is B" (Hervás et al., 2007; Mason, 2004; Gero and Chilton, 2019). These works paid attention to the relationship between nouns and concepts to create elementary figurative expressions.
Recent metaphor generation works focus mainly on verbs. Yu and Wan (2019) proposed an unsupervised metaphor extraction method, and developed a neural generation model to generate metaphorical sentences from literal-metaphorical verb pairs. They however do not focus on literal to metaphorical sentence transfer, but generate a sentence given a metaphorical fit word. The closest to our work is that of Stowe et al. (2020), who focus on building a seq2seq model, using a special mask token to mask the metaphorical verbs as input, and the original metaphorical sentences as output. However, this model faces challenges in transferring the literal sentences to metaphorical ones, while maintaining the same meaning. We, on the contrary, focus on maintaining the same meaning through parallel data creation focusing on symbolism. Additionally, we incorporate a metaphor detection model as a discriminator to improve decoding during generation.
We show how to transform literal sentences to metaphorical ones. We propose a novel way of creating parallel corpora and an approach for generating metaphors that benefits from transfer learning and discriminative decoding. Human and automatic evaluations show that our best model is successful at generating metaphors. We further show that leveraging symbolic meanings helps us learn better abstract representations and better preservation of the denotative meaning of the input. Future directions include learning diverse conceptual metaphoric mapping using our parallel data and constraining our metaphoric generations based on particular mapping.
9 Ethics
Second, although we use language models trained on data collected from the Web, which have been shown to have issues with bias and abusive language (Sheng et al., 2019; Wallace et al., 2019), the inductive bias of our models should limit inadvertent negative impacts. Unlike model variants such as GPT, BART is a conditional language model, which provides more control of the generated output. Furthermore, we specifically encode writing style from a poetic corpus in our models and train on parallel data in the direction of literal to metaphorical style. Open-sourcing this technology will help to generate metaphoric text assisting creative writing practitioners or non-native language speakers to improve their writing. We do not envision any dual-use that can cause harm for the use of our the metaphor generation system.
This work was supported in part by the MCS program under Cooperative Agreement N66001-19-2-4032, and the CwC program under Contract W911NF-15-1-0543 with the US Defense Advanced Research Projects Agency (DARPA). The
views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. The authors would like to thank the members of PLUS-Lab at the University of California Los Angeles and University of Southern California and the anonymous reviewers for helpful comments.
Keiga Abe, Kayo Sakamoto, and Masanori Nakagawa. 2006. A computational model of the metaphor generation process. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 28.
Lawrence W Barsalou et al. 1999. Perceptual symbol systems. Behavioral and brain sciences, 22(4):577-660.
Beata Beigman Klebanov, Chee Wee (Ben) Leong, and Michael Flor. 2018. A corpus of non-native written English annotated for metaphor. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 86-91, New Orleans, Louisiana. Association for Computational Linguistics.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Florence, Italy. Association for Computational Linguistics.
Tuhin Chakrabarty, Smaranda Muresan, and Nanyun Peng. 2020. Generating similes effortlessly like a pro: A style transfer approach for simile generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6455-6469, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833.
Ge Gao, Eunsol Choi, Yejin Choi, and Luke Zettlemoyer. 2018. Neural metaphor detection in context. arXiv preprint arXiv:1808.09653.
Katy Ilonka Gero and Lydia B Chilton. 2019. Metaphoria: An algorithmic companion for metaphor creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1-12.
Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. 2020. Content planning for neural story generation with aristotelian rescoring. arXiv preprint arXiv:2009.09870.
Hongyu Gong, Kshitij Gupta, Akriti Jain, and Suma Bhat. 2020. Illinimet: Illinois system for metaphor detection with contextual and linguistic information. In Proceedings of the Second Workshop on Figurative Language Processing, pages 146-153.
Raquel Hervás, Rui P Costa, Hugo Costa, Pablo Gervás, and Francisco C Pereira. 2007. Enrichment of automatically generated texts using metaphor. In Mexican International Conference on Artificial Intelligence, pages 944-954. Springer.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. arXiv preprint arXiv:1805.06087.
Dirk Hovy, Shashank Srivastava, Sujay Kumar Jauhar, Mrinmaya Sachan, Kartik Goyal, Huying Li, Whitney Sanders, and Eduard Hovy. 2013. Identifying metaphorical word use with tree kernels. In Proceedings of the First Workshop on Metaphor in NLP, pages 52-57.
Arthur M Jacobs. 2018. The Gutenberg english poetry corpus: exemplary quantitative narrative analyses. Frontiers in Digital Humanities, 5:5.
Beata Beigman Klebanov, Ben Leong, Michael Heilman, and Michael Flor. 2014. Different texts, same metaphors: Unigrams and beyond. In Proceedings of the Second Workshop on Metaphor in NLP, pages 11-17.
Maximilian Köper and Sabine Schulte im Walde. 2016. Distinguishing literal and non-literal usage of german particle verbs. In Proceedings of the 2016 conference of the north American chapter of the association for computational linguistics: Human language technologies, pages 353-362.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Rui Mao, Chenghua Lin, and Frank Guerin. 2019. End-to-end sequential metaphor identification inspired by linguistic theories. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3888-3898.
James H Martin. 2006. A corpus-based analysis of context effects on metaphor comprehension.
Zachary J Mason. 2004. Cormet: a computational, corpus-based conventional metaphor extraction system. Computational linguistics, 30(1):23-44.
Saif Mohammad, Ekaterina Shutova, and Peter Turney. 2016. Metaphor as a medium for emotion: An empirical study. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 23-33, Berlin, Germany. Association for Computational Linguistics.
Jekaterina Novikova, Ondrej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252, Copenhagen, Denmark. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
Elena Semino et al. 2007. Mip: a method for identifying metaphorically used words in discourse. Metaphor and symbol, 22(1):1-39.
Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3398-3403.
Ekaterina Shutova, Douwe Kiela, and Jean Maillard. 2016. Black holes and white rabbits: Metaphor identification with visual features. In Proceedings of the 2016 Conference of the North American Chapter of
the Association for Computational Linguistics: Human Language Technologies, pages 160-170.
Ekaterina Shutova, Lin Sun, and Anna Korhonen. 2010. Metaphor identification using verb and noun clustering. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1002-1010.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence.
Gerard Steen. 2010. A method for linguistic metaphor identification: From MIP to MIPVU, volume 14. John Benjamins Publishing.
Kevin Stowe, Sarah Moeller, Laura Michaelis, and Martha Palmer. 2019. Linguistic analysis improves neural metaphor detection. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 362-371.
Kevin Stowe and Martha Palmer. 2018. Leveraging syntactic constructions for metaphor identification. In Proceedings of the Workshop on Figurative Language Processing, pages 17-26.
Kevin Stowe, Leonardo Ribeiro, and Iryna Gurevych. 2020. Metaphoric paraphrase generation. arXiv preprint arXiv:2002.12854.
Chuandong Su, Fumiyo Fukumoto, Xiaoxi Huang, Jiyi Li, Rongbo Wang, and Zhiqun Chen. 2020. Deepmet: A reading comprehension paradigm for token-level metaphor detection. In Proceedings of the Second Workshop on Figurative Language Processing, pages 30-39.
Serra Sinem Tekiroglu, Gözde Özbal, and Carlo Strapparava. 2015. Exploring sensorial features for metaphor identification. In Proceedings of the Third Workshop on Metaphor in NLP, pages 31-39.
Asuka Terai and Masanori Nakagawa. 2010. A computational system of metaphor generation with evaluation mechanism. In International Conference on Artificial Neural Networks, pages 142-147. Springer.
Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 248-258.
Tony Veale, Ekaterina Shutova, and Beata Beigman Klebanov. 2016. Metaphor: A computational perspective. Synthesis Lectures on Human Language Technologies, 9(1):1-160.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125.
Yorick Wilks. 1975. A preferential, pattern-seeking, semantics for natural language inference. Artificial intelligence, 6(1):53-74.
Yorick Wilks. 1978. Making preferences more active. Artificial intelligence, 11(3):197-223.
Zhiwei Yu and Xiaojun Wan. 2019. How to avoid sentences spelling boring? towards a neural approach to unsupervised metaphor generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 861-871, Minneapolis, Minnesota. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
For retrieving commonsense symbolism of the sentences, we use the pre-trained COMET model and retrieve top 5 candidates for each input.
1. No of Parameters: For metaphor detection at token level we use BERT-base-cased model (110M). For generation we use the BART large checkpoint (400M parameters) and use the implementation by FAIRSEQ (Ott et al., 2019)  $^{5}$ . For discriminative decoding we use RoBERTa large model (355M)
2. No of Epochs: For metaphor detection at token level for parallel data creation we fine-tune it for 3 epochs. We fine-tune pre-trained BART for 70 epochs for MERMAID model and save best model based on validation perplexity. For discriminator we fine-tune RoBERTa-large model for 10 epoch and save the checkpoint for best validation accuracy
3. Training Time: For metaphor detection training time is 40 minutes.Our training time is 280 minutes for BART. For discriminator we train it for 60 minutes
4. Hardware Configuration: We use 4 RTX 2080 GPU
5. Training Hyper parameters: We use the same parameters mentioned in the github repo
$^{4}$ https://github.com/atcbosselut/comet-commonsense $^{5}$ https://github.com/pytorch/fairseq/tree/master/examples/bart
where BART was fine-tuned for CNN-DM summarization task with the exception of MAX-TOKENS (size of each mini-batch, in terms of the number of tokens.) being 1024 for us. For discriminator finetuning of roberta we use same parameters as RTE task
6. Decoding Strategy & Hyper Parameters: For decoding we generate metaphors from our models using a top-k random sampling scheme (Fan et al., 2018). At each timestep, the model generates the probability of each word in the vocabulary being the likely next word. We randomly sample from the  $k = 5$  most likely candidates from this distribution.
$^{6}$ https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md