SlowGuess/NAACL_2022_OCR huggingface dataset:
Three paper authors labelled completions generated by GPT-3 Davinci as either correct, incorrect, or literal. In some cases, there were valid interpretations that were not the same as the answer given by crowdworkers, which were also marked correct. If the model simply restated the metaphor with no interpretation, the completion was marked as literal. Because some metaphors are ambiguous when presented without context, those examples
were not counted. The inter-rater reliability was moderate due to differing standards for correctness (Krippendorff's  $\alpha = 0.5567$ ). The majority vote was taken between annotators' judgments.
GPT-3 Davinci's accuracy, counting literalized metaphors as incorrect, was  $50.8\%$ . Not counting literalized metaphors, accuracy was  $63.9\%$ . In  $37.7\%$  of cases, GPT-3 generated contradictory completions among the 4 completions. There was at least one correct completion for  $78.1\%$  of the metaphors, but only  $19.3\%$  of metaphors had all completions correct. Examples of annotated generations can be found in Appendix G.
6 Performance and Error Analysis
With these results in mind, we examine what kinds of errors models make, and what factors make the task difficult. This is covered in § 6. We find that autoregressive models rely on the predicted probability of each answer by itself to predict the answer, and that this holds true for all models, before and after training. We find that models have difficulty in interpreting "sarcastic" metaphors, and sometimes inexplicably interpret very simple metaphors wrong. We also examine error typology according to the commonsense typology of § 3.2 and find that models improve significantly on object, visual and social commonsense when trained, but not on cultural commonsense.
6.1 Reliance on Probability of Answers
We find that models often rely solely on the predicted probability of answers  $y_{1}$  and  $y_{2}$  to make their final predictions, regardless of the context. This led models to make the same prediction for the paired sentences in many cases. Figure 3 and Table 5 show that this trend improves with finetuning, and that GPT-3 is best able to disentangle the probability of  $y_{i}$  and the probability of  $P(y_{i}|x_{i})$ , but all three models show a heavy tendency to predict based on the relative probability of an answer alone.
We hypothesize that this may be one reason why BERT and RoBERTa achieve the best finetuned performance; they use a contrastive finetuning strategy providing both the correct and incorrect options as input to the model. On the other hand, the GPT models were finetuned with only the correct option, making the comparison unfair. One way to finetune GPT models contrastively is to include both options into a cleverly engineered prompt, but we
Model	r	p
Untrained
GPT-2	0.8128	6.700 × 10-136
GPT-neo	0.7891	6.075 × 10-123
GPT-3	0.7392	4.329 × 10-100
Trained
GPT-2	0.6765	6.700 × 10-78
GPT-neo	0.6689	1.456 × 10-75
GPT-3	0.4157	2.598 × 10-25
Table 5: Spearman  $r$ -values and p-values between  $P(y_{i}|x_{i})$  and  $P(y_{i})$
leave this as a direction for future work.
Figure 3: Models over-rely on probability of the answer to do their predictions.  $y$ -axis is probability of the first interpretation (answer) given metaphor while  $x$ -axis is log odds of the first interpretation.
6.2 Other Factors Influencing Correctness
We also examined the influence of several other factors on correctness. The point-biserial correlation between length of the context phrase and the binary correctness value was -0.1544 with a p-value of  $1.50 \times 10^{-7}$ , indicating that longer phrases are harder to interpret correctly. The point-biserial correlation between answer probability and binary correctness was 0.1840, with a p-value of  $3.50 \times 10^{-10}$ , indicating that examples where the answer was already more probable were more likely to be an-
swered correctly, in line with our findings that models tended to predict the answer that was already more plausible alone.
Furthermore, we conducted an analysis on subjects, objects, and relations as defined in § 3.1. We examined accuracy by part of speech patterns in each part of the metaphor, as well as by wordnet hypernyms present in each part of the metaphor. This is detailed in Appendix E and Appendix F (Fellbaum, 1998). We used NLTK for POS tagging (Loper and Bird, 2002).
6.3 Qualitative Analysis of Error Trends
Common Sense Knowledge We first examine the error tendencies by the type of common sense knowledge described in § 3.2. Table 6 summarizes accuracies for these types of commonsense questions compared to humans.
Model	Obj	Vis	Soc	Cul
Untrained
GPT-2	52.17	52.07	55.38	58.42
GPT-neo	56.38	55.62	56.01	62.10
GPT-3 Curie	75.00	71.00	72.47	78.42
Trained
GPT-2	53.57	51.48	57.91	57.37
GPT-neo	70.15	72.78	68.67	70.00
GPT-3 Curie	87.50	84.62	83.86	83.16
BERT	87.37	92.31	84.18	77.37
RoBERTa	91.20	94.08	89.56	83.68
Human	95.41	96.45	93.99	90.00
Table 6: The performance of models across different commonsense categories, in terms of accuracy on examples annotated with that category (\%). The strongest category of each model is highlighted.
We find that both humans and trained models tend to find object commonsense and visual commonsense metaphors easier to interpret. We find that as models improve, most of the performance gain comes from the object, visual and social commonsense categories. Interestingly, the untrained models do quite well on cultural examples, but do not improve much on the culture category when trained. This makes sense, as the cultural examples tend to be quite disparate, so training would not help as much with other examples.
Sarcastic Metaphors For both humans and LMs, many of the errors are "sarcastic" metaphors, such as saying "the girl was as bubbly as still water" to mean "the girl was bland", rather than "the girl was vivacious". These sentences can be difficult if
the model or human focuses on simple word association (bubbly with vivacious) without reading the entire sentence to understand the sarcasm.
Inexplicable Errors We examined the errors made by GPT-3 Curie (trained) and found that there was little overlap with mistakes made by humans. Of the 64 human errors, 13 were also errors made by GPT-3. GPT-3 made many more "obvious" errors, such as predicting "The ball is a big red sun" to mean "the ball is small" rather than "the ball is big and red" This is in contrast to the sentences in which humans made errors, which often contained rare vocabulary or unfamiliar cultural references.
7.1 Figurative Language Identification
Most existing work focuses on identifying figurative language at the word level. The VU Amsterdam Metaphor Corpus (VUA) is the largest available corpus of metaphorical language, annotated by humans (Steen et al., 2010). Two shared tasks on metaphor identification have been run (Leong et al., 2018, 2020). Both have utilized the VUA corpus, and the latter also introduced the TOEFL corpus, sampled from essays written by non-native English speakers (Leong et al., 2020; Beigman Klebanov et al., 2018). Most participants in the shared tasks used neural models, notably BERT, RoBERTa, and Bi-LSTMs (Leong et al., 2020; Bizzoni and Ghanimifard, 2018; Gao et al., 2018; Pramanick et al., 2018). These models are generally improved when augmented with semantic data, such as concreteness, and multimodal information.
Another line of work focuses on probing models to determine the extent of metaphor recognition. For instance, BERT assigns higher pseudolog-likelihood scores to metaphors than nonsense expressions, and its contextualized representations show some signs of contextualizing the object domain (Pedinotti et al., 2021). Another study uses linear probes trained on layers of BERT to predict whether a word usage is literal or nonliteral, and finds that this can be done effectively, especially using middle layers as a representation (Aghazadeh et al., 2022),
Despite the utility of these tasks and datasets, they have drawbacks. Most of the metaphor use is conventional, so this task does not capture novel metaphors well. The word-level annotation also does not lend itself well to capturing extended con
ceptual metaphors. Finally, metaphor interpretation may be a more difficult, although less studied, task.
7.2 Figurative Language Interpretation
Recent studies mostly focus on metaphor paraphrases, either through identification (Bizzoni and Lappin, 2018) or generation (Shutova, 2010; Su et al., 2017; Mao et al., 2018). However, there has not been as much work done on interpretation as on detection, and framing metaphor interpretation as a paraphrase task may not capture the emergent meaning of metaphors, such as the intended emotion, or the interaction of subject, relation and object in the metaphor (Tong et al., 2021; Mohammad et al., 2016).
Other work has focused on interpreting figurative language in narratives in context, based on plausible continuations of figurative language such as idioms and similes from stories (Chakrabarty et al., 2021a) or dialogues (Jhamtani et al., 2021). This represents a promising direction, and our work focuses on expanding our understanding of LMs' ability to interpret non-conventionalized metaphors.
7.3 Other Figurative Language Datasets
We note that there are several other challenging NLI datasets available which contain figurative language, including the DNC corpus, and the RTE dataset (Poliak et al., 2018; Chakrabarty et al., 2021b). Other datasets, such as RiddleSense, explicitly test models through difficult commonsense inference, involving figurative language (Lin et al., 2021).
Our work is distinguished by the Winograd schema format, as this format provides a better guard against the possibility that models have simply memorized common word associations that occur in figurative language. Additionally, we specifically instructed crowdworkers to be creative, and this resulted in longer figurative phrases which require more detailed commonsense knowledge. It is likely that a fair number of these figurative phrases have never appeared in any training corpus. However, our figurative phrases also differ from riddles, as they are not supposed to be difficult to reason about, given that the source, relation and object are properly contextualized.
7.4 Human Language Processing
Humans typically do not have any more difficulty processing metaphorical statements in context compared to literal statements (Fussell and Moss, 2008;
Glucksberg, 2003). This may be because certain words serve as a dual reference, which is to say they refer simultaneously to a physical referent and an abstract superordinate category (Glucksberg, 2003). For instance, "shark" may refer to literal sharks, as well as anything that is considered vicious, leading to utterances such as "that lawyer is a shark".
Metaphorical language processing has also been studied in second-language learners, in the case of idioms. In most cases, the meaning of an unfamiliar idiom is inferred from the context or from word association (Cooper, 1999; Carston and Wearing, 2011; Wolff and Gentner, 2000).
As LMs excel at word-association based tasks, this is an encouraging finding. However, there is still a gap between LM and human performance even in our task, in which one answer is obviously wrong when the input is correctly understood.
We take into account that these results are for conventionalized figurative language and that some of the more creative phrases in this dataset may take a longer time to process for humans as well. This is especially true for non-native English speakers. However, the high human accuracy on this task with half the participants being non-native English speakers suggests that this was not a major barrier.
We present a Winograd-like benchmark task to test the ability of LMs to reason about figurative language, based on large-scale collection of creative metaphors written by humans. We find a large gap between LM zero-shot and human performance on this dataset, but show that models can be fine-tuned to perform well on this particular task.
We hope that this work will encourage further study of nonliteral reasoning in LMs, especially in few-shot settings. Given that metaphorical reasoning may play a role in problem-solving and linguistic creativity, the development of models, training methods, or datasets that enable metaphorical reasoning may improve models' abilities to reason creatively and draw analogies between situations that may appear to be different on the surface. One avenue we hope to investigate is multimodal metaphors, as this dataset currently includes only text-based metaphors. Nonliteral expressions also remain understudied cross-linguistically, but further work on identifying and interpreting metaphors in other languages may also improve the abilities of multilingual models.
9.1 Potential Risks
Figurative language has the potential to be used in a harmful way, especially against minority and historically disadvantaged groups. Such language is often emotionally charged or used to insult others, so we took care to remove any examples that were potentially offensive, especially toward protected groups. We acknowledge that this was based on our own judgment, and generically insulting language (for instance, a metaphor that implies that someone is ugly) was not removed because it was not insulting toward any particular individual.
All examples from Fig-QA are also in English, as it is the language that all authors speak, and this was a preliminary dataset, being the first of its type that the authors have worked on. However, figurative language is not just important in English, and we leave investigation of figurative language in other languages as future work.
9.2 Terms of Use of Artefacts Used
Additional datasets we used were the Winogrande dataset, SNLI, MNLI, FEVER-NLI and ANLI. Winogrande is licensed under the Apache 2.0 license, which allows modification and distribution, fitting our use case. SNLI is licensed under a Creative Commons Attribution ShareAlike 4.0 International license, which allows us to share and adapt the work as long as we give attribution. The majority of MNLI is licensed under OANC, which allows free use. The fiction section of this dataset consists mostly of works in the public domain, but several stories are licensed: Seven Swords is available under a Creative Commons Share-Alike 3.0 Unported License, while Living History and Password Incorrect are available under Creative Commons Attribution 3.0 Unported Licenses. These licenses allow sharing and adaptation with attribution. FEVER-NLI is licensed under an MIT license, which also allows modification, distribution, and reuse. ANLI is licensed under Creative Commons Attribution-NonCommercial 4.0 International, which also allows sharing and reuse as long as we give attribution.
Models used were GPT-2, GPT-neo, GPT-3, BERT and RoBERTa. GPT-2 and GPT-neo are licensed under an MIT license, which does not place any restrictions on its use. BERT is licensed under an Apache License 2.0, which allows modification and distribution. RoBERTa is licensed under
a GNU General Public License v2.0. This fits our use case as we are only running and studying the model. GPT-3 is licensed by Microsoft, and we used the public API to receive output.
9.3 Computational Infrastructure and Computing Budget
To run our computational experiments, we had access to a compute cluster, but minimal compute is needed to run the experiments in this paper. We generally did not use more than 2 GPUs at a time. The only models that required GPU parallelism were the GPT-neo models. An estimated 20 GPU hours are required.
Our computing budget was roughly 100 USD. We also used roughly 20 USD on credits for the GPT-3 API.
We thank Pengfei Liu, Lyuyang Hu, and Chih-Hao Wang for helping us set up the leaderboard for this dataset on Explainaboard. We also thank Pengfei Liu for helping run GPT-3, Danish Pruthi for guidance on setting up the MTurk task, and all participants who contributed to the human benchmark. Lastly, we thank all the Turkers who contributed metaphors to the dataset.
This work was supported in part by a CMU Presidential Fellowship and National Science Foundation Award No. 1761548.
Ehsan Aghazadeh, Mohsen Fayyaz, and Yadollah Yaghoobzadeh. 2022. Metaphors in pre-trained language models: Probing and generalization across datasets and languages.
Beata Beigman Klebanov, Chee Wee (Ben) Leong, and Michael Flor. 2018. A corpus of non-native written English annotated for metaphor. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 86-91, New Orleans, Louisiana. Association for Computational Linguistics.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online. Association for Computational Linguistics.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr
Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735, Online. Association for Computational Linguistics.
Yuri Bizzoni and Mehdi Ghanimifard. 2018. Bigrams and BiLSTMs two neural networks for sequential metaphor detection. In Proceedings of the Workshop on Figurative Language Processing, pages 91-101, New Orleans, Louisiana. Association for Computational Linguistics.
Yuri Bizzoni and Shalom Lappin. 2018. Predicting human metaphor paraphrase judgments with deep neural networks. In Proceedings of the Workshop on Figurative Language Processing, pages 45-55, New Orleans, Louisiana. Association for Computational Linguistics.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Robyn Carston and Catherine Wearing. 2011. Metaphor, hyperbole and simile: A pragmatic approach. Language and Cognition, 3(2):283-312.
Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 2021a. It's not rocket science: Interpreting figurative language in narratives. ArXiv, abs/2109.00087.
Tuhin Chakrabarty, Debanjan Ghosh, Adam Poliak, and Smaranda Muresan. 2021b. Figurative language in recognizing textual entailment. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 3354-3361, Online. Association for Computational Linguistics.
Thomas C. Cooper. 1999. Processing of idioms by 12 learners of english. TESOL Quarterly, 33:233-262.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
Yanai Elazar, Hongming Zhang, Yoav Goldberg, and Dan Roth. 2021. Back to square one: Artifact detection, training and commonsense disentanglement in the winograd schema.
Gilles Fauconnier and Mark Turner. 2003. Conceptual blending, form and meaning.
Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books.
Susan Fussell and Mallie Moss. 2008. Figurative language in emotional communication.
Ge Gao, Eunsol Choi, Yejin Choi, and Luke Zettlemoyer. 2018. Neural metaphor detection in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 607-613, Brussels, Belgium. Association for Computational Linguistics.
D Gentner and B Bowdle. 2008. Metaphor as structure-mapping. In *The Cambridge handbook of metaphor and thought*, pages 109–128. Cambridge University Press.
Kahlil Gibran. 1926. Sand and Foam; a book of aphorisms. A.A Knopf.
Sam Glucksberg. 2003. The psycholinguistics of metaphor. Trends in cognitive sciences, 7:92-96.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Taylor Berg-Kirkpatrick. 2021. Investigating robustness of dialog models to popular figurative language constructs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7476-7485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
G. Lakoff and M. Johnson. 1981. Metaphors we Live By. University of Chicago Press.
Chee Wee (Ben) Leong, Beata Beigman Klebanov, Chris Hamill, Egon Stemle, Rituja Ubale, and Xi-anyang Chen. 2020. A report on the 2020 VUA and TOEFL metaphor detection shared task. In Proceedings of the Second Workshop on Figurative Language Processing, pages 18-29, Online. Association for Computational Linguistics.
Chee Wee (Ben) Leong, Beata Beigman Klebanov, and Ekaterina Shutova. 2018. A report on the 2018 VUA metaphor detection shared task. In Proceedings of the Workshop on Figurative Language Processing, pages 56-66, New Orleans, Louisiana. Association for Computational Linguistics.
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR'12, page 552-561. AAAI Press.
Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. 2021. Riddlesense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. CoRR, cs.CL/0205028.
Rui Mao, Chenghua Lin, and Frank Guerin. 2018. Word embedding and WordNet based metaphor identification and interpretation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1222-1231, Melbourne, Australia. Association for Computational Linguistics.
J. S Mio and A. N Katz. 1996. Metaphor: Implications and Applications. Psychology Press.
Saif Mohammad, Ekaterina Shutova, and Peter Turney. 2016. Metaphor as a medium for emotion: An empirical study. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 23-33, Berlin, Germany. Association for Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Paolo Pedinotti, Eliana Di Palma, Ludovica Cerini, and Alessandro Lenci. 2021. A howling success or a working sea? testing what BERT knows about metaphors. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 192-204, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 67-81, Brussels, Belgium. Association for Computational Linguistics.