SlowGuess/NAACL_2021_OCR huggingface dataset:
Minjin Choi $^{1}$ , Sunkyung Lee $^{1}$ , Eunseong Choi $^{1}$ , Heesoo Park $^{2}$ , Junhyuk Lee $^{1}$ , Dongwon Lee $^{3}$  and Jongwuk Lee $^{1}$
$^{1}$ Sungkyunkwan University, Republic of Korea,  $^{2}$ Bering Lab, Republic of Korea,
<sup>3</sup>The Pennsylvania State University, United States
1{zxcvxd,skl027,eunseong,ljhjoon00,jongwuklee}@skku.edu
$^{2}$ heesoo.park@beringlab.com  $^{3}$ dongwon@psu.edu
Automated metaphor detection is a challenging task to identify the metaphorical expression of words in a sentence. To tackle this problem, we adopt pre-trained contextualized models, e.g., BERT and RoBERTa. To this end, we propose a novel metaphor detection model, namely metaphor-aware late interaction over BERT (MelBERT). Our model not only leverages contextualized word representation but also benefits from linguistic metaphor identification theories to detect whether the target word is metaphorical. Our empirical results demonstrate that MelBERT outperforms several strong baselines on four benchmark datasets, i.e., VUA-18, VUA-20, MOH-X, and TroFi.
As the conceptual and cognitive mapping of words, a metaphor is a common language expression representing other concepts rather than taking literal meanings of words in context (Lakoff and Johnson, 1980; Lagerwerf and Meijers, 2008). For instance, in the sentence "hope is on the horizon," the word "horizon" does not literally mean the line at the earth's surface. It is a metaphorical expression to describe a positive situation. Therefore, the meaning of "horizon" is context-specific and different from its literal definition.
As the metaphor plays a key role in cognitive and communicative functions, it is essential to understand contextualized and unusual meanings of words (e.g., metaphor, metonymy, and personification) in various natural language processing (NLP) tasks, e.g., machine translation (Shi et al., 2014), sentiment analysis (Cambria et al., 2017), and dialogue systems (Dybala and Sayama, 2012). A lot of existing studies have developed various computational models to recognize metaphorical words in a sentence.
Automated metaphor detection aims at identifying metaphorical expressions using computational
models. Existing studies can be categorized into three pillars. First, feature-based models employ various hand-crafted features (Shutova et al., 2010; Turney et al., 2011; Shutova and Sun, 2013; Broadwell et al., 2013; Tsvetkov et al., 2014; Bulat et al., 2017). Although simple and intuitive, they are highly sensitive to the quality of a corpus. Second, some studies (Wu et al., 2018; Gao et al., 2018; Mao et al., 2019) utilize recurrent neural networks (RNNs), which are suitable for analyzing the sequential structure of words. However, they are limited to understanding the diverse meanings of words in context. Lastly, the pre-trained contextualized models, e.g., BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), have been used for detecting metaphors (Chen et al., 2020; Gong et al., 2020; Su et al., 2020). Owing to the powerful representation capacity, such models have been successful for addressing various NLP tasks (Wang et al., 2019) and document ranking in IR (Mitra and Craswell, 2018).
Based on such an advancement, we utilize a contextualized model using two metaphor identification theories, i.e., Metaphor Identification Procedure (MIP) (Pragglejaz Group, 2007; Steen et al., 2010) and Selectional Preference Violation (SPV) (Wilks, 1975, 1978). For MIP, a metaphorical word is recognized if the literal meaning of a word is different from its contextual meaning (Haagsma and Bjerva, 2016). For instance, in the sentence "Don't twist my words", the contextual meaning of "twist" is "to distort the intended meaning", different from its literal meaning, "to form into a bent, curling, or distorted shape." For SPV, a metaphorical word is identified if the target word is unusual in the context of its surrounding words. That is, "twist" is metaphorical because it is unusual in the context of "words." Although the key ideas of the two strategies are similar, they have different procedures for detecting metaphorical words and their contexts in the sentence.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1763-1773 June 6-11, 2021. ©2021 Association for Computational Linguistics
To this end, we propose a novel metaphor detection model using metaphorical identification theories over the pre-trained contextualized model, namely metaphor-aware late interaction over BERT (MelBERT). MelBERT deals with a classification task to identify whether a target word in a sentence is metaphorical or not. As depicted in Figure 2, MelBERT is based on a siamese architecture that takes two sentences as input. The first sentence is a sentence  $S$  with a target word  $w_{t}$  and the second sentence is a target word  $w_{t}$  itself. MelBERT independently encodes  $S$  and  $w_{t}$  into each embedding vector, which avoids unnecessary interactions between  $S$  and  $w_{t}$ . Inspired by MIP, MelBERT then employs the contextualized and isolated representations of  $w_{t}$  to distinguish between the contextual and literal meaning of  $w_{t}$ . To utilize SPV, MelBERT employs the sentence embedding vector and the contextualized target word embedding vector. MelBERT identifies how much the surrounding words mismatch from the target word. Lastly, MelBERT combines two metaphor identification strategies to predict if a target word is metaphorical or not. Each metaphor identification theory is non-trivial for capturing complicated and vague metaphorical words. To overcome these limitations, we incorporate two linguistic theories into a pre-trained contextualized model and utilize several linguistic features such as POS features.
To summarize, MelBERT has two key advantages. First, MelBERT effectively employs the contextualized representation to understand various aspects of words in context. Because MelBERT is particularly based on a late interaction over contextualized models, it can prevent unnecessary interactions between two inputs and effectively distinguish the contextualized meaning and the isolated meaning of a word. Second, MelBERT utilizes two metaphor identification theories to detect whether the target word is metaphorical. Experimental results show that MelBERT consistently outperforms state-of-the-art metaphor detection models in terms of F1-score on several benchmark datasets, such as VUA-18, VUA-20, and VUA-Verb datasets.
2.1 Metaphor Detection
Feature-based approach. Various linguistic features are used to understand metaphorical expressions. Representative hand-engineered features include word abstractness and concreteness (Tur
ney et al., 2011), word imageability (Broadwell et al., 2013), semantic supersenses (Tsvetkov et al., 2014), and property norms (Bulat et al., 2017). However, they have difficulties handling rare usages of metaphors because the features rely on manually annotated resources. To address this problem, sparse distributional features (Shutova et al., 2010; Shutova and Sun, 2013) and dense word embeddings (Shutova et al., 2016; Rei et al., 2017), i.e., Word2Vec (Mikolov et al., 2013), are used as better linguistic features. For details, refer to the survey (Veale et al., 2016).
RNN-based approach. Several studies proposed neural metaphor detection models using recurrent neural networks (RNNs). (Wu et al., 2018) adopts a bidirectional-LSTM (BiLSTM) (Graves and Schmidhuber, 2005) and a convolutional neural network (CNN) using Word2Vec (Mikolov et al., 2013) as text features in addition to part-of-speech (POS) and word clustering information as linguistic features. (Gao et al., 2018) employs BiLSTM as an encoder using GloVe (Pennington et al., 2014) and ELMo (Peters et al., 2018) as text input representation. (Mao et al., 2019) makes use of the metaphor identification theory on top of the architecture of (Gao et al., 2018). Despite their success, the shallow neural networks (e.g., BiLSTM and CNN) have limitations on representing various aspects of words in context.
Contextualization-based approach. Recent studies utilize pre-trained contextualized language models, e.g., BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), for metaphor detection. Because the pre-trained model can encode rich semantic and contextual information, it is useful for detecting metaphors with fine-tuning training. DeepMet (Su et al., 2020) utilizes RoBERTa with various linguistic features, i.e., global text context, local text context, and POS features. IlliniMet (Gong et al., 2020) combines RoBERTa with linguistic information obtained from external resources. (Chen et al., 2020) formulates the multitask learning problem for both metaphor detection, and (Leong et al., 2020) reports the results of these models in the VUA 2020 shared task.
2.2 Semantic Matching over BERT
The key idea of neural semantic matching is that neural models encode a query-document pair into two embedding vectors and compute a relevance score between the query and the document (Mi
tra and Craswell, 2018). The simple approach is to feed a query-document pair to BERT (Devlin et al., 2019) and compute a relevance score, where the query and the document are fully interacted (Nogueira et al., 2019; Dai and Callan, 2020). In contrast, SBERT (Reimers and Gurevych, 2019), TwinBERT (Lu et al., 2020), and ColBERT (Khattab and Zaharia, 2020) adopt late interaction architectures using siamese BERT, where the query and the document are encoded independently. Our work is based on the late interaction architecture. In other words, the sentence with the target word and the target word is encoded separately to represent contextualized and isolated meanings of the target word.
3 MelBERT
In this section, we propose a novel metaphor detection model over a pre-trained contextualized model. To design our model, we consider two metaphor detection tasks. Given a sentence  $S = \{w_{1},\dots ,w_{n}\}$  with  $n$  words and a target word  $w_{t}\in S$ , the classification task predicts the metaphoricity (i.e., metaphorical or literal) of  $w_{t}$ . Given a sentence  $S$ , the sequence labeling predicts the metaphoricity of each word  $w_{t}$  ( $1\leq t\leq n$ ) in  $S$ .
We aim at developing a metaphor detection model for the classification task. Our model returns a binary output, i.e., 1 if the target word  $w_{t}$  in  $S$  is metaphorical or 0 otherwise. By sequentially changing the target word  $w_{t}$ , our model can be generalized to classify the metaphoricity of each word in a sentence, as in sequence labeling.
The pre-trained language models, e.g., BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), usually take two sentences as input and return output to predict the relevance between two input sentences. We adopt RoBERTa as the contextualized backbone model because RoBERTa is known to outperform BERT (Liu et al., 2019). To design a metaphor detection model, we treat one input sentence as a single word (or a phrase).
As depicted in Figure 1, there are two paradigms for representing the interaction between two input sentences: all-to-all interaction and late interaction, as discussed in the document ranking problem (Khattab and Zaharia, 2020). While all-to-all interaction takes two input sentences together as an input, late interaction encodes two sentences
(a) All-to-all interaction
(b) Late interaction
Figure 1: Two interaction paradigms over a contextualized model.
separately over a siamese architecture. Given a sentence  $S$  and a target word  $w_{t}$ , all-to-all interaction can capture all possible interactions within and across  $w_{t}$  and  $S$ , which incurs high computational cost. Moreover, when some interactions across  $w_{t}$  and  $S$  are useless, it may learn noisy information. In contrast, because late interaction encodes  $w_{t}$  and  $S$  independently, it naturally avoids unnecessary intervention across  $w_{t}$  and  $S$ . The sentence embedding vector also can be easily reused in computing the interaction with the target word. In other words, the cost of encoding the sentence vector can be amortized for that of encoding different target words.
Because our goal is to identify whether the contextualized meaning of the target word  $w_{t}$  is different from its isolated meaning, we adopt the late interaction paradigm for metaphor detection. Our model encodes a sentence  $S$  with a target word and a target word  $w_{t}$  into embedding vectors, respectively, and computes the metaphoricity score of the target word. (In Section 4, it is found that our model using late interaction outperforms a baseline model using all-to-all interaction.)
We propose a novel metaphor detection model, namely, metaphor-aware late interaction over BERT (MelBERT) using metaphor identification theories, i.e., Metaphor Identification Procedure (MIP) (Pragglejaz Group, 2007; Steen et al., 2010) and Selectional Preference Violation (SPV) (Wilks, 1975, 1978). Figure 2 illustrates the overall architecture of MelBERT, which consists of three components: a sentence encoder  $Enc(S)$ , a target word encoder  $Enc(w_t)$ , and a late interaction mechanism to compute a score.
We first explain the input layer for two encoders  $Enc(S)$  and  $Enc(w_{t})$ . Each word in the sentence is converted to tokens using an improved implementation of byte-pair encoding (BPE) (Radford
Figure 2: Model architecture of MelBERT. When a target word  $w_{t}$  is split into multiple tokens by BPE, the average pooling is used for the target word.
et al., 2019). As shown in the original BERT, the position embedding is used to represent the position of tokens. The segment embedding is used to distinguish target tokens (denoted as [TAR]) and their local context (denoted as [LOC]). When the sentence is represented as a composite sentence, the local context indicates a clause including target tokens. For simplicity, we represent the local context using comma separator (,) in the sentence. Besides, we add a special classification token [CLS] before the first token and a segment separation token [SEP] after the last token. To make use of the POS feature of the target word, we append the POS tag for the target word after [SEP], as used in (Su et al., 2020). The input representation is finally computed by the element-wise addition of token, position embedding, and segment embedding. For  $Enc(w_{t})$ , the target word is converted to the tokens using BPE, but position and segment embedding are not used.
Given a sentence  $S = \{w_{1},\dots ,w_{n}\}$ ,  $Enc(S)$  encodes each word into a set of contextualized embedding vectors,  $\{\mathbf{v}_S,\mathbf{v}_{S,1},\ldots ,\mathbf{v}_{S,n}\}$  using the transformer encoder (Vaswani et al., 2017), where  $\mathbf{v}_S$  is the embedding vector corresponding to the [CLS] token and  $\mathbf{v}_{S,i}$  is the  $i$ -th embedding vector for  $w_{i}$  in  $S$ . Similarly,  $Enc(w_{t})$  encodes a target word  $w_{t}$  into  $\mathbf{v}_{t}$  without context.
$$
\begin{array}{c} \mathbf {v} _ {S}, \mathbf {v} _ {S, 1}, \dots , \mathbf {v} _ {S, n} = \\ E n c \left(^ {\prime \prime} [ C L S ], w _ {1}, \dots , w _ {n}, [ S E P ] ^ {\prime \prime}\right) \end{array} \tag {1}
$$
$$
\mathbf {v} _ {t} = \operatorname {E n c} \left(" [ C L S ], w _ {t}, [ S E P ] ^ {\prime \prime}\right) \tag {2}
$$
While  $\mathbf{v}_S$  reflects the interaction across all words in  $S$ ,  $\mathbf{v}_{S,t}$  considers the interaction between  $w_{t}$  and
other words in  $S$ . Therefore,  $\mathbf{v}_{S,t}$  and  $\mathbf{v}_t$  can be interpreted as different meanings for  $w_{t}$ , i.e.,  $\mathbf{v}_{S,t}$  is contextualized representation of  $w_{t}$  and  $\mathbf{v}_t$  is isolated representation of  $w_{t}$ .
Then, we utilize two metaphor identification theories using contextualized embedding vectors.
MelBERT using MIP. The basic idea of MIP is that a metaphorical word is identified by the gap between the contextual and literal meaning of a word. To incorporate MIP into MelBERT, we employ two embedding vectors  $\mathbf{v}_{S,t}$  and  $\mathbf{v}_t$ , representing a contextualized embedding vector and an isolated embedding vector for  $w_{t}$ , respectively. Using these vectors, we identify the semantic gap for the target word in context and isolation.
MelBERT using SPV. The idea of SPV is that a metaphorical word is identified by the semantic difference from its surrounding words. Unlike MIP, we only utilize the sentence encoder. Given a target word  $w_{t}$  in  $S$ , our key assumption is that  $\mathbf{v}_{S}$  and  $\mathbf{v}_{S,t}$  show a semantic gap if  $w_{t}$  is metaphorical. Although  $\mathbf{v}_{S}$  and  $\mathbf{v}_{S,t}$  are contextualized, the meanings of the two vectors are different;  $\mathbf{v}_{S}$  represents the interaction across all pair-wise words in  $S$ , but  $\mathbf{v}_{S,t}$  represents the interaction between  $w_{t}$  and other words in  $S$ . In this sense, when  $w_{t}$  is metaphorical,  $\mathbf{v}_{S,t}$  can be different from  $\mathbf{v}_{S}$  by the surrounding words of  $w_{t}$ .
Late interaction over MelBERT. Using the two strategies, MelBERT predicts whether a target word  $w_{t} \in S$  is metaphorical or not. We can compute a hidden vector  $h_{MIP}$  by concatenating  $\mathbf{v}_{S,t}$  and  $\mathbf{v}_{t}$  for MIP.
$$
h _ {M I P} = f \left(\left[ \mathbf {v} _ {S, t}; \mathbf {v} _ {t} \right]\right), \tag {3}
$$
where  $h_{MIP} \in \mathbb{R}^{h \times 1}$  and  $f(\cdot)$  is a function for the MLP layer to learn the gap between two vectors  $\mathbf{v}_{S,t}$  and  $\mathbf{v}_t$ .
We can also compute a hidden vector  $h_{SPV}$  using  $\mathbf{v}_S$  and  $\mathbf{v}_{S,t}$  for SPV.
$$
h _ {S P V} = g ([ \mathbf {v} _ {S}; \mathbf {v} _ {S, t} ]), \tag {4}
$$
where  $h_{SPV} \in \mathbb{R}^{h \times 1}$  and  $g(\cdot)$  is a function for the MLP layer to learn the semantic difference between  $\mathbf{v}_S$  and  $\mathbf{v}_{S,t}$ .
We combine two hidden vectors  $h_{MIP}$  and  $h_{SPV}$  to compute a prediction score:
$$
\hat {y} = \sigma \left(W ^ {\top} \left[ h _ {M I P}; h _ {S P V} \right] + b\right), \tag {5}
$$
where  $\sigma (\cdot)$  is the sigmoid function,  $W\in \mathbb{R}^{2h\times 1}$  is the parameter, and  $b$  is a bias. To learn MelBERT, finally, we use the cross-entropy loss function for binary classification as follows:
$$
\mathcal {L} = \sum_ {i = 1} ^ {N} y _ {i} \log \hat {y} _ {i} + (1 - y _ {i}) \log (1 - \hat {y} _ {i}), \tag {6}
$$
where  $N$  is the number of samples in the training set.  $y_{i}$  and  $\hat{y}_i$  are the true and predicted labels for the  $i$ -th sample in the training set.
In this section, we first present the experimental setup, then report empirical results by comparing our model against strong baselines.
Datasets. We use four well-known public English datasets. First, the VU Amsterdam Metaphor Corpus (VUA) has been released in metaphor detection shared tasks in 2018 and 2020. We use two versions of VUA datasets, called VUA-18 (Leong et al., 2018) and VUA-20 (Leong et al., 2020), where VUA-20 is the extension of VUA-18. Let  $\mathrm{VUA - 18}_{tr}$ ,  $\mathrm{VUA - 18}_{dev}$ ,  $\mathrm{VUA - 18}_{te}$  denote the training, validation, and test datasets, split from VUA-18.  $\mathrm{VUA - 20}_{tr}$  includes  $\mathrm{VUA - 18}_{tr}$  and  $\mathrm{VUA - 18}_{dev}$ .  $\mathrm{VUA - 20}_{te}$  also includes  $\mathrm{VUA - 18}_{te}$ , and  $\mathrm{VUA - Verb}_{te}$  is a subset of  $\mathrm{VUA - 18}_{te}$  and  $\mathrm{VUA - 20}_{te}$ .
Because most of the tokens in a sentence are literal words in VUA-18, VUA-20 selectively chooses the tokens in the training and testing datasets.  $\mathrm{VUA - 18}_{te}$  consists of four genres, including news, academic, fiction, and conversation. It can also be categorized into different POS tags, such as verb, noun, adjective, and adverb. Additionally, we employ MOH-X (Mohammad et al., 2016)
Dataset	#tokens	%M	#Sent	Sent len
VUA-18tr	116,622	11.2	6,323	18.4
VUA-18dev	38,628	11.6	1,550	24.9
VUA-18te	50,175	12.4	2,694	18.6
VUA-20tr	160,154	12.0	12,109	15
VUA-20te	22,196	17.9	3,698	15.5
VUA-Verbte	5,873	30	2,694	18.6
MOH-X	647	48.7	647	8
TroFi	3,737	43.5	3,737	28.3
Table 1: Detailed statistics on benchmark datasets. #tokens is the number of tokens, %M is the percentage of metaphorical words, #Sent is the number of sentences, and Sent len is the average length of sentences.
and TroFi (Birke and Sarkar, 2006) for testing purposes only. MOH-X is a verb metaphor detection dataset with the sentences from WordNet and TroFi is also a verb metaphor detection dataset, including sentences from the 1987-89 Wall Street Journal Corpus Release 1. The sizes of these datasets are relatively smaller than those of VUA datasets, and they have metaphorical words of more than  $40\%$ , while VUA-18 and VUA-20 datasets have about  $10\%$  of metaphorical words. While MOH-X and TroFi only annotate verbs as metaphorical words, the VUA dataset annotates all POS tags as metaphorical words. In this sense, we believe that the VUA dataset is more appropriate for training and testing models. Table 1 summarizes detailed statistics on the benchmark datasets.
Baselines. We compare our models with several strong baselines, including RNN-based and contextualization-based models.
- RNN_ELMo and RNN_BERT (Gao et al., 2018): They employ the concatenation of the pre-trained ELMo/BERT and the GloVe (Pennington et al., 2014) embedding vectors as an input, and use BiLSTM as a backbone model. Note that they use contextualized models only for input vector representation.
- RNN_HG and RNN_MHCA (Mao et al., 2019): They incorporate MIP and SPV into RNN_ELMo (Gao et al., 2018). RNN_HG compares an input embedding vector (literal) with its hidden state (contextual) through BiLSTM. RNN_MHCA utilizes multi-head attention to capture the contextual feature within the window size.
- RoBERTa_BASE: It is a simple adoption of RoBERTa for metaphor detection. It takes a
target word and a sentence as two input sentences and computes a prediction score. It can be viewed as a metaphor detection model over an all-to-all interaction architecture.
- RoBERTa_SEQ (Leong et al., 2020): It takes one single sentence as an input, and a target word is marked as the input embedding token and predicts the metaphoricity of the target word using the embedding vector of the target word. This architecture is used as the BERT-based baseline in the VUA 2020 shared task.
- DeepMet (Su et al., 2020): It is the winning model in the VUA 2020 shared task. It also utilizes RoBERTa as a backbone model and incorporates it with various linguistic features, such as global context, local context, POS tags, and fine-grained POS tags.
Evaluation protocol. Because the ratio of metaphorical words is relatively small, we adopt three metrics, e.g., precision, recall, and F1-score, denoted by Prec, Rec, and F1. MOH-X and TroFi datasets are too smaller than VUA datasets. Thus, we only used them as the test datasets; metaphor detection models are only trained in VUA datasets, and zero-shot transfer is conducted to evaluate the effectiveness of model generalization.
Implementation details. For four baselines, we used the same hyperparameter settings $^1$  in (Gao et al., 2018; Mao et al., 2019; Su et al., 2020). For DeepMet $^2$ , we evaluated it with/without bagging technique. While DeepMet (Su et al., 2020) exploits two optimization techniques, bagging and ensemble, we only used a bagging technique for MelBERT and DeepMet. It is because we want to evaluate the effectiveness of model designs. The performance difference for DeepMet between the original paper and ours thus comes from the usage of the ensemble method. For contextualized models, we used a pre-trained RoBERTa $^3$  with 12 layers, 12 attention heads in each layer, and 768 dimensions of the hidden state. For contextualized baselines, we set the same hyperparameters with MelBERT, which were tuned on VUA-18 $_{dev}$  based on F1-score. The batch size and max sequence length were set as 32 and 150. For training, the number of epochs was three with Adam optimizer.