ANALOGICAL - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models
Year: 2023

Abstract:
Over the past decade, analogies, in the form of word-level analogies, have played a significant role as an intrinsic measure of evaluating the quality of word embedding methods such as word2vec. Modern large language models (LLMs), however, are primarily evaluated on extrinsic measures based on benchmarks such as GLUE and SuperGLUE, and there are only a few investigations on whether LLMs can draw analogies between long texts. In this paper, we present ANALOGICAL, a new benchmark to intrinsically evaluate LLMs across a taxonomy of analogies of long text with six levels of complexity – (i) word, (ii) word vs. sentence, (iii) syntactic, (iv) negation, (v) entailment, and (vi) metaphor. Using thirteen datasets and three different distance measures, we evaluate the abilities of eight LLMs in identifying analogical pairs in the semantic vector space. Our evaluation finds that it is increasingly challenging for LLMs to identify analogies when going up the analogy taxonomy. 1 Introducing ANALOGIAL a Benchmark for Analogy The ability of humans to perceive a situation in one context as similar to that in a different context is known as analogy-making. It is considered to be a central component of human cognition and learning. Analogy-making has received attention from a broad audience, including cognitive scientists (Gentner and Markman, 1997; Holyoak et al., 2001), linguists (Itkonen, 2005), and educators (Richland and Simms, 2015) during the last several decades. Current neural network-based word embeddings are primarily influenced by the distributional hypothesis "You shall know a word by the company it keeps" (Firth, 1957). Corresponding author † Work does not relate to position at Amazon. -200 -150 -100 -50 0 50 100 -150 -130 -110 -90 -70 -50 -30 -10 10 A member of my team will execute your orders with immense precision. One of our member will carry out your instructions minutely. Read for Slate's take on Jackson's findings. Slate had an opinion on Jackson's findings. I'm upset that my walkman broke and now I have to turn the stereo up really loud My walkman broke so I'm upset now I just have to turn the stereo up real loud Figure 1: Expected vector space embeddings of three analogical sentence pairs from a hypothetical LLM that captures sentence analogies accurately. During 2013-2017, less complex, word-level analogies played a central role in intrinsically evaluating the quality of word embedding methods, such as word2vec (Mikolov et al., 2013a), GloVe (Pennington et al., 2014), and fastText (Bojanowski et al., 2017). Different types of textual analogies can be identified, such as word analogies (Gladkova et al., 2016a), proportional analogies (Mikolov et al., 2013a), and long-text analogies (Ichien et al., 2020). The techniques to create word embeddings have progressed from categorical (i.e., one-hot, bagof-words) to continuous contextualized techniques exemplified by LLMs such as BERT (Devlin et al., 2018) and T5 (Raffel et al., 2022). However, only a few investigations have been done on the capabilities of LLMs to draw analogies between long text (Czinczoll et al., 2022). For example embeddings of sentences ‘I can speak two languages.’ and ‘I am bilingual.’ should be closeby in vector space and ‘I like chocolate.’ and ‘I do not like chocolate.’ should not be close-by. Performance evaluations of modern LLM are driven mainly by extrinsic measures based on benchmarks such as GLUE (Wang et al., 2018), and Super-

heading: Future Work