SlowGuess/NAACL_2021_OCR huggingface dataset:
Dataset	Model	Prec	Rec	F1
VUA-18	RNN_ELMo	71.6	73.6	72.6
RNN_BERT	71.5	71.9	71.7
RNN_HG	71.8	76.3	74.0
RNN_MHCA	73.0	75.7	74.3
RoBERTa_BASE	79.4	75.0	77.1
RoBERTa_SEQ	80.4	74.9	77.5
DeepMet	82.0	71.3	76.3
MelBERT	80.1	76.9	78.5*
DeepMet-CV	77.5	80.2	78.8
MelBERT-CV	78.9	80.7	79.8*
VUA-Verb	RNN_ELMo	68.2	71.3	69.7
RNN_BERT	66.7	71.5	69.0
RNN_HG	69.3	72.3	70.8
RNN_MHCA	66.3	75.2	70.5
RoBERTa_BASE	76.9	72.8	74.7
RoBERTa_SEQ	79.2	69.8	74.2
DeepMet	79.5	70.8	74.9
MelBERT	78.7	72.9	75.7
DeepMet-CV	76.2	78.3	77.2
MelBERT-CV	75.5	78.7	77.1
Table 2: Performance comparison of MelBERT with baselines on VUA-18 and VUA-Verb (best is in bold and second best is in italic underlined). Let -CV denote the bagging technique for its base model (best is in bold-italic). * denotes  $p < 0.05$  for a two-tailed t-test with the best competing model.
Dataset	Model	Prec	Rec	F1
VUA-20	RoBERTa_BASE	74.9	68.0	71.2
RoBERTa_SEQ	76.9	66.7	71.4
DeepMet	76.7	65.9	70.9
MelBERT	76.4	68.6	72.3*
DeepMet-CV	73.8	73.2	73.5
MelBERT-CV	74.1	73.7	73.9
Table 3: Performance comparison of MelBERT with baselines on VUA-20 (best is in bold and second best is in italic underlined). Let -CV denote the bagging technique for its base model (best is in bold-italic). * denotes  $p < 0.05$  for a two-tailed t-test with the best competing model.
We increased the learning rate from 0 to 3e-5 during the first two epochs and then linearly decreased it during the last epoch. We set the dropout ratio as 0.2. All experimental results were averaged over five runs with different random seeds. We conducted all experiments on a desktop with 2 NVidia TITAN RTX, 256 GB memory, and 2 Intel Xeon Processor E5-2695 v4 (2.10 GHz, 45M cache). We implemented our model using PyTorch. All the source code is available at our website<sup>4</sup>.
<sup>1</sup>https://github.com/RuiMao1988/Sequential-Metaphor-Identification
$^{2}$ https://github.com/YU-NLPLab/DeepMet
<sup>3</sup>https://huggingface.co/roberta-base
4 https://github.com/jin530/MelBERT
Genre	Model	Prec	Rec	F1
Academic	RNN_ELMo	78.2	80.2	79.2
RNN_BERT	76.7	76.0	76.4
RNN_HG	76.5	83.0	79.6
RNN_MHCA	79.6	80.0	79.8
RoBERTa_BASE	88.1	79.5	83.6
RoBERTa_SEQ	86.0	77.3	81.4
DeepMet	88.4	74.7	81.0
MelBERT	85.3	82.5	83.9
Conversation	RNN_ELMo	64.9	63.1	64.0
RNN_BERT	64.7	64.2	64.4
RNN_HG	63.6	72.5	67.8
RNN_MHCA	64.0	71.1	67.4
RoBERTa_BASE	70.3	69.0	69.6
RoBERTaSEQ	70.5	69.8	70.1
DeepMet	71.6	71.1	71.4
MelBERT	70.1	71.7	70.9
Fiction	RNN_ELMo	61.4	69.1	65.1
RNN_BERT	66.5	68.6	67.5
RNN_HG	61.8	74.5	67.5
RNN_MHCA	64.8	70.9	67.7
RoBERTa_BASE	74.3	72.1	73.2
RoBERTaSEQ	73.9	72.7	73.3
DeepMet	76.1	70.1	73.0
MelBERT	74.0	76.8	75.4
News	RNN_ELMo	72.7	71.2	71.9
RNN_BERT	71.2	72.5	71.8
RNN_HG	71.6	76.8	74.1
RNN_MHCA	74.8	75.3	75.0
RoBERTa_BASE	83.5	71.8	77.2
RoBERTaSEQ	82.2	74.1	77.9
DeepMet	84.1	67.6	75.0
MelBERT	81.0	73.7	77.2
Overall results. Tables 2 and 3 report the comparison results of MelBERT against other baselines using RNNs and contextualized models on VUA-18, VUA-20, and VUA-Verb. It is found that MelBERT is consistently better than strong baselines in terms of F1-score. MelBERT outperforms (F1 = 78.5, 75.7, and 72.3) DeepMet (Su et al., 2020) with  $2.8\%$ ,  $1.0\%$ , and  $1.9\%$  performance gains on the three datasets. MelBERT also outperforms contextualized baseline models (i.e., RoBERTa_BASE and RoBERTa_SEQ), up to  $1.2 - 1.5\%$  gains on the three datasets, indicating that MelBERT effectively utilizes metaphorical identification theories.
When combining MelBERT and DeepMet with the bagging technique, both models (i.e., MelBERT-CV and DeepMet-CV) show better performance than their original models by aggregating multiple models trained with 10-fold cross-validation process as used in (Su et al., 2020). MelBERT
Table 4: Model performance of different genres in VUA-18 (best is in bold and second best is in italic underlined).
POS	Model	Prec	Rec	F1
Verb	RNN_ELMo	68.1	71.9	69.9
RNN_BERT	67.1	72.1	69.5
RNN_HG	66.4	75.5	70.7
RNN_MHCA	66.0	76.0	70.7
RoBERTa_BASE	77.0	72.1	74.5
RoBERTaSEQ	74.4	75.1	74.8
DeepMet	78.8	68.5	73.3
MelBERT	74.2	75.9	75.1
Adjective	RNN_ELMo	56.1	60.6	58.3
RNN_BERT	58.1	51.6	54.7
RNN_HG	59.2	65.6	62.2
RNN_MHCA	61.4	61.7	61.6
RoBERTa_BASE	71.7	59.0	64.7
RoBERTaSEQ	72.0	57.1	63.7
DeepMet	79.0	52.9	63.3
MelBERT	69.4	60.1	64.4
Adverb	RNN_ELMo	67.2	53.7	59.7
RNN_BERT	64.8	61.1	62.9
RNN_HG	61.0	66.8	63.8
RNN_MHCA	66.1	60.7	63.2
RoBERTa_BASE	78.2	69.3	73.5
RoBERTaSEQ	77.6	63.9	70.1
DeepMet	79.4	66.4	72.3
MelBERT	80.2	69.7	74.6
Noun	RNN_ELMo	59.9	60.8	60.4
RNN_BERT	63.3	56.8	59.9
RNN_HG	60.3	66.8	63.4
RNN_MHCA	69.1	58.2	63.2
RoBERTa_BASE	77.5	60.4	67.9
RoBERTaSEQ	76.5	59.0	66.6
DeepMet	76.5	57.1	65.4
MelBERT	75.4	66.5	70.7
Table 5: Model performance of different POS tags in VUA-18 (best is in bold and second best is in italic underlined).
CV still shows better performance for all metrics than DeepMet-CV in VUA-18 and VUA-20. Also, MelBERT-CV (Recall = 73.7) significantly improves the original MelBERT (Recall = 68.6) in terms of recall. It implies that MelBERT-CV can capture various metaphorical expressions by combining multiple models.
Besides, it is found that contextualization-based models show better performance than RNN-based models in VUA-18 and VUA-Verb. While RNN-based models show  $71 - 74\%$  F1-score, contextualization-based models show  $76 - 78\%$  F1-score on VUA-18. It is revealed that RNN-based models are limited in capturing various aspects of words in context. Compared to RNN_ELMo and RNN_BERT, it also indicates that utilizing contextualization-based models as backbone models can have a better effect than simply utilizing it as an extra input embedding vector in (Gao et al., 2018; Mao et al., 2019).
Dataset	Model	Prec	Rec	F1
MOH-X	RoBERTa_BASE	77.4	80.1	78.4
RoBERTa_SEQ	80.6	77.7	78.7
DeepMet	79.9	76.5	77.9
MelBERT	79.3	79.7	79.2
TroFi	RoBERTa_BASE	54.6	74.3	62.9
RoBERTa_SEQ	53.6	70.1	60.7
DeepMet	53.7	72.9	61.7
MelBERT	53.4	74.1	62.0
VUA-18 breakdown analysis. Table 4 reports the comparison results for four genres in the VUA-18 dataset. MelBERT still shows better than or comparable to all competitive models in both breakdown datasets. Compared to RNN-based models, MelBERT achieves substantial improvements, as high as  $4.9\%$  (Academic),  $4.4\%$  (Conversation),  $10.2\%$  (Fiction), and  $2.8\%$  (News) in terms of F1-score. Particularly, they show the lowest accuracy because Conversation and Fiction have more complicated or rare expressions than other genres. For example, Conversation contains colloquial expressions or fragmented sentences such as "ah", "cos", "yeah" and Fiction often contains the names of fictional characters such as "Tepilit", "Laibon" which do not appear in other genres. Nonetheless, MelBERT shows comparable or the best performance in all genres. For Academic and Fiction, MelBERT particularly outperforms all the models in terms of F1-score.
Table 5 reports the comparison result for four POS tags in the VUA-18 dataset. For all POS tags, MelBERT consistently shows the best performance in terms of the F1-score. Compared to RNN-based models, MelBERT achieves as much as  $5.9\%$  (Verb),  $3.4\%$  (Adjective),  $14.5\%$  (Adverb), and  $10.3\%$  (Noun) gains in terms of F1-score. For all POS tags, MelBERT also outperforms DeepMet. It means that MelBERT using metaphorical identification theories can achieve consistent improvements regardless of POS tags of target words.
Zero-shot transfer on MOH-X and TroFi. We evaluate a zero-shot learning transfer across different datasets, where the models are trained with the VUA-20 training dataset, and MOH-X and TroFi are used as test datasets. Although it is a challenging task, it is useful for evaluating the gener
Table 6: Performance comparison of MelBERT with baselines over two datasets. Note that the models are trained on VUA-20, and these datasets are only used as the test datasets (best is in bold and second best is in italic underlined).
Model	VUA-18	VUA-20
Prec	Rec	F1	Prec	Rec	F1
MelBERT	80.1	76.9	78.5	76.4	68.6	72.3
(-) MIP	77.8	75.8	76.7	74.7	67.8	71.1
(-) SPV	79.5	76.3	77.9	74.9	68.6	71.7
Table 7: Effect of different metaphorical identification theories on VUA-18 and VUA-20. (-) MIP and (-) SPV indicate MelBERT without MIP and SPV, respectively (best is in bold and second best is in italic underlined).
alization power of trained models. Table 6 reports the comparison results of MelBERT against other contextualization-based models. For the MOH-X dataset, MelBERT  $(\mathrm{F1} = 79.2)$  shows the best performance in terms of F1-score with  $0.6 - 1.6\%$  performance gains. It indicates that MelBERT is an effective generalization model. For the TroFi dataset, the overall performance of all the models is much lower than MOH-X. It is because the average length of the sentences in the TroFi dataset is much longer and sentences are more complicated than those in MOH-X. Also, note that we trained DeepMet with the VUA-20 training dataset for evaluating a zero-shot transfer, while (Su et al., 2020) reported the results for DeepMet trained and tested with the MOH-X and TroFi datasets. While the performance gap between models is much small in terms of precision, MelBERT is better than DeepMet in terms of recall. It means that MelBERT can capture complicated metaphorical expressions than DeepMet.
Ablation study of MelBERT. Table 7 compares the effectiveness of metaphor identification theories. It is found that MelBERT using both strategies consistently shows the best performance. Also, MelBERT without SPV shows better performance than MelBERT without MIP, indicating that MelBERT using late interaction is more effective for capturing the difference between contextualized and isolated meanings of target words. Nonetheless, MelBERT shows the best performance by synergizing both metaphor identification strategies.
Error analysis. Table 8 reports qualitative evaluation results of MelBERT. Based on the original annotation guideline<sup>5</sup>, we analyze several failure cases of MelBERT. For MelBERT without MIP, it is difficult to find common words with multiple meanings, e.g., go and feel. Also, when a sentence includes multiple metaphorical words, it mostly fails to detect metaphorical words. In this case,
<sup>5</sup>http://www.vismet.org/metcor/documentation/home.html
(✓)MIP	(✓)SPV	MelBERT	Sentence
✓	✓	Manchester is not alone.
✓	✓	That's an old trick.
✓	✓	Oh you rotten old pig, you've been sick.
✓	✓	"Are the twins trash?"
✓	✓	I know, what is going on!
✓	✓	So who's covering tomorrow?
✓	✓	Do you feel better now?
✓	✓	The day thrift turned into a nightmare.
Way of the World: Farming notes
So many places Barry are going down
Sensitivity, though, is not enough.
Table 8: Examples of incorrect samples for MelBERT on VUA-20. The metaphorical words in the sentence are in red italicized.  $\checkmark$  marks correct model prediction.
the surrounding words of a target word are not a cue to detect metaphors using SPV. Meanwhile, MelBERT without SPV has a failure case if target words are metaphorical for personification. That is, using MIP only, the target word can be closely interpreted by its literal meaning. As the most difficult case, MelBERT often fails to identify metaphorical words for borderline or implicit metaphors, e.g., Way of the World is poetic.
In this work, we proposed a novel metaphor detection model, namely, metaphor-aware late interaction over BERT (MelBERT), marrying pre-trained contextualized models with metaphor identification theories. To our best knowledge, this is the first work that takes full advantage of both contextualized models and metaphor identification theories. Comprehensive experimental results demonstrated that MelBERT achieves state-of-the-art performance on several datasets.
This work was supported by the National Research Foundation of Korea (NRF) (NRF2018R1A5A1060031). Also, this work was supported by the Institute of Information & communications Technology Planning & evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00421, AI Graduate School Support Program and No.2019-0-01590, High
Potential Individuals Global Training Program). The work of Dongwon Lee was in part supported by NSF awards #1742702, #1820609, #1909702, #1915801, and #1934782.
Julia Birke and Anoop Sarkar. 2006. A clustering approach for nearly unsupervised recognition of nonliteral language. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL).
George Aaron Broadwell, Umit Boz, Ignacio Cases, Tomek Strzalkowski, Laurie Feldman, Sarah M. Taylor, Samira Shaikh, Ting Liu, Kit Cho, and Nick Webb. 2013. Using imageability and topic chaining to locate metaphors in linguistic corpora. In Proceedings of the International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction (SBP), pages 102-110.
Luana Bulat, Stephen Clark, and Ekaterina Shutova. 2017. Modelling metaphor with attribute-based semantics. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 523-528.
Erik Cambria, Soujanya Poria, Alexander F. Gelbukh, and Mike Thelwall. 2017. Sentiment analysis is a big suitcase. IEEE Intell. Syst., 32(6):74-80.
Xianyang Chen, Chee Wee Leong, Michael Flor, and Beata Beigman Klebanov. 2020. Go Figure! multitask transformer-based architecture for metaphor detection using idioms: ETS team in 2020 metaphor shared task. In The Second Workshop on Figurative Language Processing (Fig-Lang@ACL), pages 235-243.
Zhuyun Dai and Jamie Callan. 2020. Context-aware term weighting for first stage passage retrieval. In International Conference on Research and Development in Information Retrieval (SIGIR), pages 1533-1536.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171-4186.
Pawel Dybala and Kohichi Sayama. 2012. Humor, emotions and communication: Human-like issues of human-computer interactions. In Annual Meeting of the Cognitive Science Society (CogSci).
Ge Gao, Eunsol Choi, Yejin Choi, and Luke Zettlemoyer. 2018. Neural metaphor detection in context. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 607-613.
Hongyu Gong, Kshitij Gupta, Akriti Jain, and Suma Bhat. 2020. IlliniMet: Illinois system for metaphor detection with contextual and linguistic information. In The Second Workshop on Figurative Language Processing (Fig-Lang@ACL), pages 146-153.
Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. *Neural Networks*, 18(5-6):602-610.
Hessel Haagsma and Johannes Bjerva. 2016. Detecting novel metaphor using selectional preference information. In Proceedings of the Fourth Workshop on Metaphor in NLP.
Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In International Conference on Research and Development in Information Retrieval (SIGIR), pages 39-48.
Luuk Lagerwerf and Anoe Meijers. 2008. Openness in metaphorical and straightforward advertisements: Appreciation effects. Journal of Advertising, 37(2):19-30.
George Lakoff and Mark Johnson. 1980. Metaphors we live by. University of Chicago Press.
Chee Wee Leong, Beata Beigman Klebanov, Chris Hamill, Egon Stemle, Rituja Ubale, and Xianyang Chen. 2020. A report on the 2020 VUA and TOEFL metaphor detection shared task. In The Second Workshop on Figurative Language Processing (FigLang@ACL 2020), pages 18-29.
Chee Wee Leong, Beata Beigman Klebanov, and Ekaterina Shutova. 2018. A report on the 2018 VUA metaphor detection shared task. In *The Workshop on Figurative Language Processing* (FigLang@NAACL-HLT), pages 56-66.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020. Twinbert: Distilling knowledge to twin-structured compressed BERT models for large-scale retrieval. In International Conference on Information and Knowledge Management (CIKM), pages 2645-2652.
Rui Mao, Chenghua Lin, and Frank Guerin. 2019. End-to-end sequential metaphor identification inspired by linguistic theories. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3888-3898.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the Conference on Neural Information Processing Systems (NIPS), pages 3111-3119.
Bhaskar Mitra and Nick Craswell. 2018. An introduction to neural information retrieval. Found. Trends Inf. Retr., 13(1):1-126.
Saif Mohammad, Ekaterina Shutova, and Peter D. Turney. 2016. Metaphor as a medium for emotion: An empirical study. In Proceedings of the Joint Conference on Lexical and Computational Semantics (*SEM@ACL).
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. CoRR, abs/1904.08375.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2227-2237.
Pragglejaz Group. 2007. Mip: A method for identifying metaphorically used words in discourse. Metaphor and Symbol, 22(1):1-39.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI.
Marek Rei, Luana Bulat, Douwe Kiela, and Ekaterina Shutova. 2017. Grasping the finer point: A supervised similarity network for metaphor detection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1537-1546.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In The Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3980-3990.
Chunqi Shi, Toru Ishida, and Donghui Lin. 2014. Translation agent: A new metaphor for machine translation. New Gener. Comput., 32(2):163-186.
Ekaterina Shutova, Douwe Kiela, and Jean Maillard. 2016. Black holes and white rabbits: Metaphor identification with visual features. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 160-170.
Ekaterina Shutova and Lin Sun. 2013. Unsupervised metaphor identification using hierarchical graph factorization clustering. In Proceeding of the Conference of the North American Chapter of the Asso
ciation of Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 978-988.
Ekaterina Shutova, Lin Sun, and Anna Korhonen. 2010. Metaphor identification using verb and noun clustering. In Proceedings of International Conference on Computational Linguistics (CoLING), pages 1002-1010.
Gerard J Steen, Aletta G Dorst, J Berenike Herrman, Anna Kaal, Tina Krennmayr, and Trijntje Pasma. 2010. A method for linguistic metaphor identification. From MIP to MIPVU. John Benjamin.
Chuandong Su, Fumiyo Fukumoto, Xiaoxi Huang, Jiyi Li, Rongbo Wang, and Zhiqun Chen. 2020. DeepMet: A reading comprehension paradigm for token-level metaphor detection. In The Second Workshop on Figurative Language Processing (FigLang@ACL), pages 30-39.
Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proceedings of the 52th Annual Meeting of the Association for Computational Linguistics (ACL), pages 248-258.
Peter D. Turney, Yair Neuman, Dan Assaf, and Yohai Cohen. 2011. Literal and metaphorical sense identification through concrete and abstract context. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 680-690.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), pages 5998-6008.
Tony Veale, Ekaterina Shutova, and Beata Beigman  
Klebanov. 2016. Metaphor: A Computational Perspective. Synthesis Lectures on Human Language  
Technologies. Morgan & Claypool Publishers.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR).
Yorick Wilks. 1975. A preferential, pattern-seeking, semantics for natural language inference. Artif. Intell., 6(1):53-74.
Yorick Wilks. 1978. Making preferences more active. Artif. Intell., 11(3):197-223.
Chuhan Wu, Fangzhao Wu, Yubo Chen, Sixing Wu, Zhigang Yuan, , and Yongfeng Huang. 2018. Neural metaphor detecting with cnn-lstm model. In Proceedings of the Workshop on Figurative Language Processing, pages 110-114.