SlowGuess/NAACL_2022_OCR huggingface dataset:
Emmy Liu, Chenxuan Cui, Kenneth Zheng, Graham Neubig
{mengyan3,cxcui,kzheng2,gneubig}@cs.cmu.edu
Figurative and metaphorical language are commonplace in discourse, and figurative expressions play an important role in communication and cognition. However, figurative language has been a relatively under-studied area in NLP, and it remains an open question to what extent modern language models can interpret nonliteral phrases. To address this question, we introduce Fig-QA, a Winograd-style nonliteral language understanding task consisting of correctly interpreting paired figurative phrases with divergent meanings. We evaluate the performance of several state-of-the-art language models on this task, and find that although language models achieve performance significantly over chance, they still fall short of human performance, particularly in zero- or few-shot settings. This suggests that further work is needed to improve the nonliteral reasoning capabilities of language models.
All our words are but crumbs that fall down from the feast of the mind (Gibran, 1926). When humans read such a metaphorical phrase, how do they interpret it? Conceptual metaphors structure our everyday language and are used to map everyday physical experiences and emotions onto abstract concepts (Lakoff and Johnson, 1981). They allow us to communicate complex ideas, to emphasize emotions, and to make humorous statements (Fussell and Moss, 2008). However, despite relating words in a way that differs from their accepted definition, these phrases are readily interpreted by human listeners, and are common in discourse (Shutova, 2011), occurring on average every three sentences (Mio and Katz, 1996; Fussell and Moss, 2008)
The ability to interpret figurative language has been viewed as a bottleneck in natural language un
derstanding, but it has not been studied as widely as literal language (Shutova, 2011; Tong et al., 2021). Figurative language often relies on shared commonsense or cultural knowledge, and in some cases may be difficult to solve using language statistics. This presents a challenge to language models (LMs), as strong LMs trained only on text may not be able to make sense of the physical world, nor the social or cultural knowledge that language is grounded in (Bender and Koller, 2020; Bisk et al., 2020).
Most previous work on figurative language focuses on metaphor detection, where a model is trained to identify the existence of metaphors in text (Tsvetkov et al., 2014; Stowe and Palmer, 2018; Leong et al., 2020), with datasets consisting mostly of conventionalized metaphors and idioms in wide use. However, identifying these common metaphors that already appear often in language may be an easy task for LMs, and not fully test their ability to interpret figurative language. The little work that exists on metaphor interpretation frames it as a task linking metaphorical phrases to literal rewordings, either through paraphrase detection (Bizzoni and Lappin, 2018) or paraphrase generation (Shutova, 2010; Su et al., 2017; Mao et al., 2018) (details in § 7) Another line of work probes for metaphorical understanding in LMs, but this is similar to the metaphor detection task, in that the LM is not actually asked to choose an interpretation for the metaphor (Pedinotti et al., 2021; Aghazadeh et al., 2022). While interesting, this work does not take into account the fact that metaphors are rich with different implications that may vary depending on the context.
In this work, we ask whether or not LMs can correctly make inferences regarding creative, relatively novel metaphors generated by humans. This task is harder for two reasons: (1) inference is harder than identification or paraphrasing, as it requires understanding the underlying semantics, and (2) the metaphors in our dataset are novel cre
$^{1}$ Code and data are available at https://github.com/nightingal3/Fig-QA, and a leaderboard can be found at https://explainaboardInspiredco.ai/leaderboards?dataset=fig_qa
Human Language Technologies, pages 4437 - 4452
ations, and many may not appear even once in the LMs' training data. We propose a minimal task inspired by the Winograd schema (Levesque et al., 2012), where LMs are tasked with choosing the entailed phrase from two opposite metaphorical phrases. An example of a paired sentence is "Her commitment is as sturdy as (plywood/oak)". The correct answer would be either "She was (committed/uncommitted)". This can also be seen as an entailment task, where input  $x$  is the premise, and the output  $y$  is the hypothesis.
We crowdsourced a benchmark Fig-QA, consisting of 10,256 such metaphors and implications ( $\S 2$ ), which can be used to evaluate the nonliteral reasoning abilities of LMs or for more broad studies of figurative language in general (we provide preliminary analyses in  $\S 3$ ). Through extensive experiments over strong pre-trained LMs ( $\S 4$ ), we find that although they can be fine-tuned to do reasonably well, their few-shot performance falls significantly short of human performance ( $\S 5$ ). An in-depth analysis ( $\S 6$ ) uncovers several insights: (1) LMs do not make use of the metaphorical context well, instead relying on the predicted probability of interpretations alone, (2) the task of associating a metaphor with an interpretation is more difficult than the reverse, (3) even strong models such as GPT-3 make inexplicable errors that are not well-aligned with human ones, indicating that further work is needed to properly model nonliteral language.
2 Dataset Creation and Validation
2.1 Crowdsourcing Task
We crowdsourced data from workers on Amazon Mechanical Turk ( details in Appendix A). Workers were asked to generate paired metaphors with different meanings, as well as literal implications of the two metaphors in context. We instructed workers to try to generate rare or creative metaphors, namely "metaphors that would not appear often in text on the internet, books, social media, or news sites, but that can still be easily understood by people." Workers were given examples of valid pairs that fit the format, and examples of invalid ones to discourage errors. Some examples of generated pairs are displayed in Table 1.
In order to help workers, we employ the randomness as genesis and narrow limits of change principles of Cognitive Load Theory (Sweller, 2006). To add soft constraints, we generate 3 different random words to be shown to each batch of workers. However, workers were not required to use these words, as we wanted to encourage maximal diversity. In order to ensure that the random words were metaphorically rich, we selected them based on metaphorical frames in Lakoff and Johnson (1981).
2.2 Data Validation
The dataset was manually validated by three authors of this paper. Each author covered roughly one-third, evenly split between training, validation, and test. Examples were excluded if they (a) did not make sense given the figurative expression, (b) had grammar or spelling errors that rendered them unintelligible, or (c) did not follow the format of the task. Examples of excluded samples are included in Appendix B. We collected 13,324 sentences and interpretations from the crowdsourcing task, and 10,256 sentences remained after filtering.
2.3 Final Dataset
The release version of our dataset contains the named data splits in Table 2. The medium train, dev, and test splits were generated from partitioning the first stage of data collected. The large train split additionally contains all the new examples from the second collection stage, and the small train split is a small random sample.
3 Figurative Language Typologies
In this sample, we perform an analysis of the collected data to demonstrate its trends and categorize examples for further error analysis. Specifically, we examine (a) subjects, objects, and relations, and (b) types of common-sense knowledge needed to interpret the metaphor.
3.1 Figurative Language Structure
We note that most metaphors and similes can be characterized by three components,  $(S,R,O)$  where  $S$  is a subject,  $R$  is a relation, and  $O$  is an object. For instance, "Her commitment is as sturdy as plywood" can be written (Her commitment, sturdy, plywood). Interpretation involves inferring an attribute of the subject by extracting a relational attribute from the object (Fauconnier and Turner, 2003). In a simile,  $R$  is explicit, whereas it is usually implicit in a metaphor. The most common
2The opposing meanings help to avoid ambiguity in the correct answer, make the task intuitive for human annotators, and help prevent annotation artifacts that have plagued other NLI datasets (Gururangan et al., 2018).
Paired sentences	Possible answers
The pilot flew like a ballet dancer
The pilot flew like a modern dancer	The pilot flew in a (restrained way | creative way)
The pilot flew in a (restrained way | creative way)
The meteor was as bright as New York City
The meteor was as bright as coal	The meteor was (very bright | not bright at all)
The meteor was (very bright | not bright at all)
The atom is like a solar system
The atom is like a cloud	Electrons (orbit the nucleus | are in probability densities)
Electrons (orbit the nucleus | are in probability densities)
He hustles like rent was due three days ago
He hustles like he's a billionaire's son.	He (hustles hardcore. | doesn't hustle at all.)
He (hustles hardcore | doesn't hustle at all)
Life is as easy as kindergarten for a high school senior
Life is as easy as kindergarten for a newborn	Life is (basic | beyond comprehension)
Life is (basic | beyond comprehension)
Table 1: Example sentences from the dataset
S	Train M	L	Dev	Test
200	1,458	8,016	1,094	1,146
Table 2: Examples in each data split
Figure 1: Visualization of 25 most frequent subjects, relations, and objects in the medium train set.
subjects, relations, and objects in the medium train dataset are shown in Figure 1. These were obtained by first segmenting the phrases with syntactic patterns constructed from observation, followed by lemmatization and removal of punctuation and determiners "the", "an", "a" and "that". There are 441 unique subjects, 646 unique relations, and 1,198 unique objects in the medium training set.
3.2 Common-sense Knowledge Types
Next, we examined the test set to determine the types of commonsense knowledge needed to interpret metaphors. Through thematic analysis, we devised 4 categories based on commonsense knowledge, which are not mutually exclusive: common-sense object knowledge, visual metaphors, common-sense social understanding, and cultural knowledge. The same 3 paper authors annotated the test set for these categories, with annotators responsible for separate categories.
Common-sense object knowledge consisted of metaphors that made reference to properties of common objects and animals, such as volume, height or mass of objects, or properties of materials.  $68.35\%$  of the test-set was found to require common-sense object knowledge.
Visual metaphors were a subset of commonsense object metaphors, primarily relying on the visual modality, including attributes such as brightness or colour. Some visual metaphors also sketched a vivid visual scene. These examples comprised  $14.73\%$  of the test set.
Common-sense social understanding examples required knowing about how humans would react in different circumstances, or required knowing about human emotions. These examples comprised  $27.55\%$  of the test set.
Cultural metaphors required knowing cultural traditions, works of art/artefacts, or religion. Due to crowdworkers being recruited from the US, these are centered around US culture. These examples comprised  $16.56\%$  of the test set.
4 Baseline Models and Evaluation
4.1 Auto-regressive Language Models
Auto-regressive LMs generate a probability distribution of the next token given all preceding tokens.
Type of knowledge required	Paired sentences
Common-sense (objects)	The new mattress is just as comfortable as sleeping on a (cloud/rocks outside)
Visual	The professor's argument had the clarity of a (crystal glass/marine fog)
Common-sense (social)	She is as embarrassed as a kid that (forgot homework/got an A)
Cultural	The construction was as disastrous as the (1981 musical Cats/The 2019 film based on the musical Cats)
Table 3: Metaphor types based on types of knowledge required (not mutually exclusive)
As such, we can directly compute the probability of a sentence by multiplying the conditional probability of each token at every time step.
$$
\tilde {P} (w _ {1} \dots w _ {N}) = p (w _ {1}) \prod_ {i = 2} ^ {N} p (w _ {i} | w _ {1} \dots w _ {i - 1})
$$
The ability to directly extract probabilities enables the zero-shot reasoning of these LMs. For a pair of metaphorical expressions  $x_{1}$  and  $x_{2}$  with two corresponding interpretations  $y_{1}$  and  $y_{2}$ , we feed in the concatenation of the metaphor and the interpretation to the pretrained model without finetuning. We define "forward" and "backward" probabilities assigned to interpretations and figurative language expressions, respectively. For the forward probability, for figurative phrase  $x_{i}$  and correct answer  $y_{i}$ , we take
$$
P (y _ {i} | x _ {i}) = \frac {P (x _ {i} , y _ {i})}{P (x _ {i} , y _ {i}) + P (x _ {i} , y _ {j})}
$$
since there are only two answer options. From this, we can calculate accuracy when we taking the indicator of  $P(y_{i}|x_{i}) > 0.5$ . Similarly for the backward probability (predicting phrase based on answer), we take
$$
P (x _ {i} | y _ {i}) = \frac {P (x _ {i} , y _ {i})}{P (x _ {i} , y _ {i}) + P (x _ {j} , y _ {i})}
$$
with analogous backward accuracy.
We examine three state-of-the-art large transformer-based LMs of this category: GPT-2 (with 117M parameters, trained on 40GB of text), GPT-neo (with 1.3B parameters, trained on 800GB of text) and GPT-3 (4 variants between 350M and 175B parameters, trained on 45TB on text) (Radford et al., 2019; Black et al.,
2021; Brown et al., 2020). We also examine the performance of these models after finetuning on the training data. GPT-2 and GPT-neo were trained with a batch size of 8, with early stopping on the medium dataset with a patience of 1 epoch, and a minimal hyperparameter search was done with learning rates 1e-5 to 5e-5. GPT-3 was trained with the default parameters of the GPT-3 finetuning API.
4.2 Masked Language Models
We also evaluate the performance of masked LMs on this task. Unlike auto-regressive LMs, masked LMs cannot directly output the probability of a sentence, so it is not possible to directly test the zero-shot performance of these models. Instead, we test the transfer performance by first finetuning them in two ways: on WinoGrande, which is also a binary choice task based on commonsense reasoning, and on several NLI datasets, including SNLI, MNLI, FEVER-NLI and ANLI (Nie et al., 2020; Sakaguchi et al., 2020). The input to the model trained on WINOGRANDE is formatted as [CLS] [metaphor] [SEP] [answer1] [SEP] [answer2], and we use an extra linear layer on the [CLS] token embedding to perform the classification. In addition to the transfer performance, we also use contrastive finetuning by feeding in each metaphor along with both answer choices, and training the model with our dataset to classify which answer is correct. For the NLI model, we examine accuracy using all three labels the model was originally trained with (entailment, neutral, and contradiction), as well as using a forced binary choice paradigm in which the logits for the contradiction label are subtracted from the logits for the entailment label, and the higher "entailment score" is the ending the model predicts. We treat these two conditions as the analog of "zero-shot" for these models.
We examine two masked LMs that are commonly used as baselines on many NLP tasks: BERT (Devlin et al., 2019), a transformer-based LM jointly trained on the masked LM and next sen
3In actuality, we use the length-normalized probability that a model assigns to a sentence as a heuristic for the total probability, to minimize the effect that the length of a sentence has on the decision (though this is not the probability of the sequence in a strict sense):  $P(w_{1}\dots w_{N}) = \exp \left(-\frac{1}{N}\log \tilde{P} (w_{1}\dots w_{N})\right)$ . Initial experimentation showed marginal differences in accuracy when using these two methods, so we used normalized probabilities by default.
tence prediction objectives, and RoBERTa (Liu et al., 2019), an improved variant of BERT which consistently outperforms BERT across most tasks. We use the large variant of both models (350M parameters). BERT and RoBERTa were finetuned on the medium dataset for 8 epochs with batch size 8, following the setting in (Sakaguchi et al., 2020). A hyperparameter search was done with learning rates 5e-6 to 2e-5. Both BERT and RoBERTa were used for the Winogrande experiments, while only RoBERTa was used for the NLI experiment.
4.3 Forced-choice Paradigm
Due to the inherent creativity of metaphors, there may be different interpretations of the same metaphor. For instance, in Table 1, the example "he hustles like he's a billionaire's son" could also be interpreted in other ways, for instance "he uses his father's contacts and social privileges to make money". In a structural-mapping context, the forced choice between two answers constrains the possible meaning of the metaphor to be along one axis (Gentnder and Bowdle, 2008). In this case, it would be whether or not he is required to work hard.
Of course, many of these metaphors have other valid interpretations. In the "billionaire's son" example, another valid axis of interpretation could be the manner in which he works. For instance, the alternative pair could be "he hustles like he's a (billionaire's son | single mother working three jobs)" with answers "he (uses his contacts and social privileges to make money | works extremely long hours with multiple ventures to make money)". It is possible that LMs could come up with other valid interpretations that are not the ones originally intended, motivating us to also look at generation performance in section § 5.2.
4.4 Human Performance
To estimate the expected human performance on this task, we ran a benchmark on the test set with 10 human volunteers who were not authors of the paper. The human annotators were not shown any training examples, so this would be equivalent to the zero-shot setting for models. Participants ranged from 20 to 29 years old, and there were 5 male and 5 female participants. 5 each were native- and non-native English speakers respectively. Participants were mainly graduate student volunteers.
We shuffled the test set and split it into 10 partitions of  $\approx 115$  examples for each annotator. The
Model	Zero-shot	Tuned (L)	Tuned (XL)
GPT-2	53.93	54.80	62.65
GPT-neo 1.3B	56.89	69.98	72.00
GPT-3 Ada	59.08	69.17	73.56
GPT-3 Babbage	62.91	73.97	77.31
GPT-3 Curie	65.35	79.04	81.94
GPT-3 Davinci	68.41	-	-
BERT	58.14	83.16	85.69
RoBERTa	66.184	89.22	90.32
Human	94.42	-	-
Human (confident)	95.39	-	-
Table 4: Zero-shot and finetuned test accuracies  $(\%)$ , finetuned is averaged across 5 seeds. Dev set accuracies can be found on the leaderboard under the "validation" split.
examples were presented with pairs shuffled and separated, in order to create a better comparison with model performance.
Due to differences in vocabulary or cultural background, we instructed participants to mark examples where they weren't confident, such as those that contained words or cultural references they didn't understand.
5.1 Inference Results
The first question is whether strong LMs can interpret metaphors at all when presented with two opposing meanings, in zero-shot or supervised settings. These results are presented in Table 4. The results for masked language models are higher than those for autoregressive language models, and fine-tuning significantly improves performance for all models.
Zero-shot Performance For the zero-shot setting, we examine the test accuracy based on zero-shot forward probabilities for the GPT models, and the pseudo "zero-shot" transfer performance for BERT and RoBERTa using models pretrained on the WinoGrande task (Sakaguchi et al., 2020). As shown, the GPT-3 models outperform the GPT-2 and GPT-neo models. Among the GPT-3 models, there is a clear correlation between model size and performance, with the largest model (GPT-3 Davinci) achieving the highest zero-shot test accuracy. BERT and RoBERTa achieve accuracies
4This is the accuracy score when transferring from Winogrande. Pretrained NLI results were 50.47 when using original labels (entailment/contradiction/neutral), and 66.32 when forcing a binary decision.
within the range of GPT-3 models. While our models mostly perform much better than chance in the zero-shot setting, there is still a large gap of 26 percentage points between our best model and human level performance.
Fine-tuned Performance For the fine-tuned setting, all listed models are fine-tuned on the small dataset split. GPT models were trained with language modeling loss, whereas BERT and RoBERTa are trained with contrastive loss. We did not evaluate fine-tuning of GPT-3 Davinci due to budget. Overall, fine-tuning improved accuracy significantly for all models, with GPT-3 models uniformly improving by about 13 percentage points, and BERT/RoBERTa improving by about 25 points. Our best model after fine-tuning is RoBERTa, which reaches within  $5\%$  of our human performance.
Prompting We also experiment with prompting methods. Firstly, we use a simple suffix prompting method, where we simply append the phrase "that is to say" between the metaphor and the interpretation, which we hypothesized may "explain" to the LM that the previous statement is figurative. We also evaluate the effectiveness of the examples method, by appending  $k$  random correct metaphor/interpretation pairs before the actual pair we are testing. The results of these experiments can be seen in Figure 2. We found that the suffix method provided a small (1-2%) improvement over the baseline, while the example method was generally ineffective.
Backward accuracies Note the accuracies reported in this section are for the forward direction, and the backward direction is reported in Appendix C. Backward accuracies are lower, with GPT-3 Curie for example having a  $7\%$  reduction in accuracy in the zero-shot case. This suggests that selecting a metaphorical expression to match a literal phrase is more challenging than the reverse for LMs.
Paired Evaluation Because our dataset is formatted as a Winograd schema, we can take advantage of group scoring to evaluate models more stringently (Elazar et al., 2021). We found that performance for autoregressive models plummeted under this evaluation scheme, while masked language models also suffered in accuracy. The human scores were least affected. Details are in Ap
pendix D. This is most likely related to the phenomenon found in  $\S 6.1$ .
Figure 2: Comparison of prompting methods with autoregressive models
5.2 Generation Results
Next, we examine if models can generate sensible interpretations for metaphors. Given the difficulty of evaluating text generation, compounded by the difficulty of figurative language, we opted for manual evaluation of one tenth of the test dataset using generations of the strongest auto-regressive model: GPT-3 Davinci ( $\approx$ 175B parameters).
The metaphor was given as input to the model, and 4 completions were generated for each metaphor, with a maximum length of 100 tokens. Completions were also truncated to the first sentence, as initial experiments showed that contradictory statements (e.g. "he was talented. But he was not very talented") were often generated across subsequent sentences. Suffix prompting was also used because of the lack of context, with "That is to say," appended to each metaphor. Only the first sentence of the output was evaluated. The temperature parameter was determined through grid search through values [0.2, 0.4, 0.6, 0.8, 1] on a small separate set of metaphors. A human annotator inspected the generated completions and found that a temperature of 0.4 produced the most correct results.