ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base (2024)

Abstract: Analogical reasoning is a fundamental cognitive ability of humans. However, current language models (LMs) still struggle to achieve human-like performance in analogical reasoning tasks due to a lack of resources for model training. In this work, we address this gap by proposing ANALOGYKB, a million-scale analogy knowledge base (KB) derived from existing knowledge graphs (KGs). ANALOGYKB identifies two types of analogies from the KGs: 1) analogies of the same relations, which can be directly extracted from the KGs, and 2) analogies of analogous relations, which are identified with a selection and filtering pipeline enabled by large language models (LLMs), followed by minor human efforts for data quality control. Evaluations on a series of datasets of two analogical reasoning tasks (analogy recognition and generation) demonstrate that ANALOGYKB successfully enables both smaller LMs and LLMs to gain better analogical reasoning capabilities. Resources of this paper can be found at https://github.com/siyuyuan/ analogykb.

heading: 1 Introduction
text: Making analogies requires identifying and mapping a familiar domain (i.e., source domain) to a less familiar domain (i.e., target domain) (Hofstadter and Sander, 2013). As shown in Figure 1, utilizing the analogy of the solar system can facilitate comprehension of the complex structure of atoms. Analogical reasoning is an important aspect of the cognitive intelligence of humans, allowing us to quickly adapt our knowledge to new domains (Hofstadter, 2001; Ding et al., 2023), make decisions (Hansen-Estruch et al., 2022), and solve problems (Yasunaga et al., 2023). As a result, the topic of analogy has been drawing significant research attention in the community.
∗Equal contribution. †Work done during ByteDance AI Lab. ‡Corresponding authors.
1
Earth
Sun
Newton
Gravity Force
Electron
Nucleus
Faraday
Electric Force
Has Has

heading: Orbit OrbitDiscover Discover
text: Discovered Analogies

heading: Knowledge Graphs

heading: Solar System Atom Structure
text: However, resources for analogical reasoning are rather limited in scale (Mikolov et al., 2013b; Gladkova et al., 2016; Chen et al., 2022), which usually consist of only hundreds or thousands of data samples. As a result, these datasets do not support effective training of language models to gain analogical reasoning abilities. Although large language models (LLMs) can make some reasonable analogies without requiring gradient update, their performance still lies behind humans (Bhavya et al., 2022; Jiayang et al., 2023). Therefore, larger-scale data sources are needed to facilitate the research in this area. With richer analogies, we can train specialized analogy-making models and retrieve high-quality examples to assist LLMs. Therefore, the research question is: How to acquire largescale analogies at a moderate cost?
An analogy is determined by the relational structure (Bartha, 2013), e.g., A:B::C:D (i.e., A is to B as C is to D), where the relation between A and B is analogous to the relation between C and D. The concepts A, B, C, and D can be entities and events. As shown in Figure 1, the “solar system” and an “atom” share a similar structure, allowing us to quickly grasp the relation between an “electron”
1249
and a “nucleus” in concepts of their source domain counterparts. Such relational structure can be derived from the triplet knowledge, e.g. (electron, orbit, nucleus) and (earth, orbit, sun), in knowledge graphs (KGs) (Wu et al., 2012). Therefore, such structure knowledge can be utilized and reorganized to create new analogy knowledge, supporting large-scale knowledge acquisition.
In this work, we aim to build a knowledge base (KB) for storing analogies derived from existing KGs to improve analogical reasoning. However, due to the complicated relational structures, discovering analogies from KGs is not a trivial task. Although two pairs of concepts with the same relation can form a valid analogy (e.g., lion, isA, animal and apple, isA, fruit), interesting and diverse analogies are implicit in the KGs, with more complex relations. Concepts under two distinct but similar relations in KGs can also form a reasonable analogy (Hesse, 1959). For example, chief executive officer and head of state can both be abstracted into a meta relation (Hesse, 1959; Gentner and Maravilla, 2017), i.e., head of organization. Therefore, they are analogous relations under a meta relation. It is important to generalize the finding of implicit analogies beyond the same relations within KGs.
We present ANALOGYKB, which is a largescale analogy KB. We use Wikidata (Vrandečić and Krötzsch, 2014) and ConceptNet (Speer et al., 2017) as our seed KGs and discover two types of analogies from these KGs: analogies of 1) same relations and 2) analogous relations. Analogies of the same relations can be directly extracted from existing KGs. In contrast, analogies of analogous relations are more implicit, requiring the finding of relation pairs from the KGs that can form valid analogies. However, it is costly to manually select analogous relation pairs. Therefore, we use InstructGPT003 (Ouyang et al., 2022), a LLM of great capabilities in NLP tasks, for finding and deciding the analogical semantics of relations. To eliminate the noise from the outputs of InstructGPT003 (§ 3.5), we devise two filtering rules based on 1) the symmetry of analogy and 2) meta relation summarization, which generalizes two relations into a more abstract meta relation. Then, we manually review the filtered results to further ensure data quality.
Our ANALOGYKB comprises over 1 million analogies with 943 relations, including 103 analogous relations. Smaller LMs trained on ANAL-
OGYKB gain significant improvements over the previous methods, even rivaling human performance on some analogy recognition tasks. Furthermore, we prove that ANALOGYKB can endow both smaller LMs and LLMs with satisfactory analogymaking capabilities. Our contributions are summarized as follows:
• To the best of our knowledge, we are the first to construct an analogy KB (ANALOGYKB) with a million scale and diverse relational structures.
• We propose a novel framework with LLMs to discover more interesting and implicit analogies of analogous relations;
• We conduct extensive experiments to evaluate the effectiveness of ANALOGYKB, which significantly improves the analogical reasoning performance of both smaller LMs and LLMs.

heading: 2 Related Work
text: Analogy Acquisition Early studies mainly acquire analogy knowledge via linguists (Turney et al., 2003; Boteanu and Chernova, 2015), which is costly and inefficient. Recent studies consider exploiting relations in KGs to build analogies (Speer et al., 2008; Allen and Hospedales, 2019; Ulčar et al., 2020), which can be divided into two lines of work: 1) Acquiring from commonsense KGs, which leverages semantic and morphological relations from WordNet (Miller, 1995), ConceptNet (Speer et al., 2017), etc. However, some of these datasets are large-scale but of poor quality (Li et al., 2018, 2020), while others are of high quality but limited in size (Mikolov et al., 2013b; Gladkova et al., 2016). 2) Acquiring from encyclopedia KGs (Si and Carlson, 2017; Zhang et al., 2022; Ilievski et al., 2022), which utilizes the relations from DBpedia (Auer et al., 2007) and Wikidata (Vrandečić and Krötzsch, 2014), but their empirical experiments are relatively small in size.
Analogical Reasoning Analogical reasoning aims to identify a relational structure between two domains (Bartha, 2013; Chen et al., 2022). Previous work adopts the word analogy task to investigate the analogical reasoning capability of LMs (Mikolov et al., 2013a,b; Levy and Goldberg, 2014; Gladkova et al., 2016; Schluter, 2018; Fournier et al., 2020; Ushio et al., 2021). Recent work demonstrates that LLMs can generate some
1
reasonable abstract (Mitchell, 2021; Hu et al., 2023; Webb et al., 2023) and natural language-based analogies (Bhavya et al., 2022; Wijesiriwardene et al., 2023; Jiayang et al., 2023) but still lay behind humans in some cases, and smaller LMs struggle to learn analogical reasoning ability due to a lack of training data.
Knowledge Base Construction Knowledge base (KB) consists of structured knowledge to support various applications. The approaches to constructing KBs can be divided into three categories: 1) Manual construction (Miller, 1995; Speer et al., 2017), which creates the KBs with specialized knowledge written by experts, and thus is laborintensive; 2) Automatic construction (Wu et al., 2012; Martinez-Rodriguez et al., 2018), which leverages models to extract knowledge from unstructured corpora, may lead to low data quality; 3) Semi-automatic construction (Dalvi Mishra et al., 2017; Romero and Razniewski, 2020), which involves manual curation and annotation. Our work is based on automatic approaches with LLMs only requiring small-scale human checking efforts.

heading: 3 ANALOGYKB Construction
text: This section details the framework for building ANALOGYKB. We first define the schema of ANALOGYKB (§ 3.1). Then, we collect relations with concept pairs from existing KGs (§ 3.2, Step 1) and directly obtain analogies of the same relations from KGs (§ 3.3, Step 2). We propose adopting LLMs (Ouyang et al., 2022) followed by minor human efforts to acquire analogies of analogous relations (§ 3.4, Step 3).

heading: 3.1 Schema for Analogies in ANALOGYKB
text: This paper focuses on the analogy formed as A:B::C:D, where concepts as A, B, C and D can be entities or events. The concept pair A:B is analogous to C:D based on an underlying relational structure. Since ANALOGYKB is built on existing KGs, we define two types of that relational structure based on KG semantics: 1) analogies of the same relation and 2) analogies of analogous relations. Data in ANALOGYKB is organized as in Figure 2, where each relation R contains subjectobject concept pairs s : o. Within each relation, analogies of the same relation can be naturally formed, e.g., “Up is to Down as High is to Low”. Also, the concept pairs between two relations can form analogies, as long as the relation pair have analogous structures (Hesse, 1959). For example, “Tim Cook is to Apple as Joe Biden is to USA”, where R2 (CEO) is analogous to R3 (head of state). Therefore, ANALOGYKB only has to store concept pairs of each relation and analogous relation pairs, from which analogies can be easily derived. We list the definitions of each terminology with examples in Appendix A.1 for better understanding.

heading: 3.2 Source Data Collection
text: We choose the two most-used KGs, i.e., ConceptNet and Wikidata consisting of high-quality concept pairs with relations, as our data sources. For ConceptNet, we select the concept pairs with weights bigger than 2.0 to improve the data quality and collect 100,000 concept pairs with 27 relations. Due to the vast amount of Wikidata, we randomly sample 5 million concepts with 813 relations from Wikidata, resulting in 20 million concept pairs.

heading: 3.3 Acquiring Analogies of the Same Relation
text: We can directly utilize the concept pairs in the KGs to generate analogies of the same relations. An important perspective is that humans usually draw upon familiar domains and situations to better understand unfamiliar ones. To make our analogy KB more applicable to real-world scenarios, we rank the concept pairs according to their popularity scores, reflected by pageview times (in Wikidata) and concept weights (in ConceptNet).

heading: 3.4 Acquiring Analogies of Analogous Relations
text: As defined in § 3.1, analogies of analogous relations consist of two concept pairs with analogous
relations R1 and R2. However, it is difficult to automatically check whether R1 and R2 are analogous and manual annotation is costly. Recently, LLMs (Ouyang et al., 2022; OpenAI, 2022) have shown their remarkable few-shot learning abilities with in-context learning. Given a task prompt describing the task and several examples, LLMs can do the task well without training. Therefore, we propose to exploit LLMs (e.g., InstructGPT003) to acquire analogies of analogous relations.
Finding Candidate Relation Pairs We collect 840 relations, leading to a potential amount of ( 840 2 ) relation pairs. The relations that are semantically similar to each other can form an analogy (Hesse, 1959). For each relation, we first narrow down the candidate set from the 840 relations to the 20-most similar ones. Specifically, we use InstructGPT embeddings (text-embedding-ada-002) to convert the relations into embeddings and calculate the cosine similarity between them. By identifying the top 20 relations with the highest similarity as candidate relations for the query relation, the search space is significantly reduced for filtering analo-
gous relations.
Predicting Analogous Relation Pairs While the search space is reduced, manual annotation remains cost-prohibitive (840 × 20). Thus, we continue to adopt InstructGPT003 to predict analogous relation pairs. An example in Table 1 (I) shows the acquisition of analogous relation pairs. Given examples and the query (“chief executive officer”), InstructGPT003 selects the relations “head of state” and “head of organization” from the candidates to form analogies. Finally, InstructGPT003 obtains 284 relation pairs. However, we find that InstructGPT003 struggles to filter out similar but wrong relations that cannot form analogies with queries, e.g., “operator” for “chief executive officer”, which requires further filtering.

heading: Filtering for High-quality Relation Pairs In
text: the examination process of 284 acquired relation pairs, we further implement two automatic filtering rules before conducting manual filtering to reduce human labor:
1. Rule 1: if two relations can form an analogy, InstructGPT003 should simultaneously select R1 for R2 and R2 for R1. 2. Rule 2 (Hesse, 1959): The second rule is using a more abstract meta relation to decide if two relations can form an analogy.
The rationale behind the Rule 2 is that if two relations are analogous, then they can be generalized into a more abstract meta relation. For example, in Table 1 (II), written by and composed by are analogous since they can be induced to a meta relation created by. To acquire meta relations, we prompt InstructGPT003 with a task prompt with some examples, as shown in Table 1 (II). If InstructGPT003 returns “None”, we discard this case.
After filtering, 103 relation pairs remain. To further improve data quality, we adopt a third filtering by recruiting two volunteers to manually examine the remaining results, including deleting relation pairs that fail to form analogies or adding previously unchosen relation pairs that can form analogies from candidates. Finally, we sort the concept pairs by pageview (Wikidata) and weight (ConceptNet).

heading: 3.5 Analysis of ANALOGYKB
text: As shown in Table 2, ANALOGYKB is massive, consisting of over 1 million concept pairs and 943
relations, which can form even more pairs of analogies. Since ANALOGYKB provides a more comprehensive range of relations than previous datasets, it allows users to select their preferred analogies within each relation (pair). To evaluate the quality of ANALOGYKB, we randomly sample 200 analogies from each data type, i.e., two concept pairs of the same or analogous relations, in the form of A:B::C:D. The data is annotated by two annotators with Fleiss’s κ = 0.86 (Fleiss et al., 1981). Results show that ANALOGYKB is of high quality. Even for analogies of analogous relations, analogies are still of over 95% accuracy.
We further compare ANALOGYKB with the resources related to the analogy, as reported in Table 3. We find that ANALOGYKB is much larger than previous data sources, with more analogies and relations. To better present the fabric of ANALOGYKB, we present the distribution of the categories of concepts covered in ANALOGYKB in Figure 3. The categories are obtained from the hypernym of concepts from Probase (Wu et al., 2012). We find that ANALOGYKB exhibits high diversity.

heading: Method # Total # Correct
text: Are the filtering techniques for analogous relations useful? We evaluate the usefulness of the filtering components, i.e., symmetry (Rule 1) and meta relation summarization (Rule 2), and manual correction. We also adopt ChatGPT (OpenAI, 2022) as an ablated variant. We record the total number of analogous relation pairs output by models (# Total) and then employ annotators to report the number of correct ones out of them (# Correct). In this process, the annotators need to review these relation pairs but no need to correct them. Each pair is examined by two annotators with Fleiss’s κ = 0.86. The results in Table 4 show that: 1) InstructGPT003 is superior to ChatGPT but it still cannot filter out similar but wrong relation pairs, indicating the need for further filtering; 2) We find the rule-based filtering technique to be rather effective, as there are not many manual corrections based on human annotations. This overcomes the
labor-intensiveness of traditional KB construction methods and reveals the potential of this approach to be extended to the construction of other KBs.

heading: 4 ANALOGYKB Evaluation

heading: 4.1 Analogy Recognition Evaluation
text: Analogy recognition task aims to recognize the most analogous candidate to the query, formulated as multiple-choice question-answering.
Can models trained on ANALOGYKB acquire better analogy recognition abilities? We adopt six analogy benchmarks, i.e., E-KAR (Chen et al., 2022), BATS (Gladkova et al., 2016), UNIT 2 and UNIT 4 (Boteanu and Chernova, 2015), Google (Mikolov et al., 2013b) and SAT (Turney et al., 2003) for evaluation. Compared to BATS and Google, E-KAR, UNIT 2, UNIT 4, and SAT contain more abstract and complex analogies and thus more difficult for humans.
For the backbone model, we use the RoBERTaLarge (Liu et al., 2019) and randomly sample 10,000 data points from ANALOGYKB to train the model in a multiple-choice question-answer format. We first train the model on the data from ANALOGYKB and then further fine-tune it on benchmarks.1 For baselines, we adopt pre-trained word embeddings (Ushio et al., 2021; Ouyang et al., 2022), pre-trained sentence embeddings (Reimers
1Detailed information on the benchmarks is shown in Appendix B and the construction of ANALOGYKB sample data is shown in Appendix C.1.
and Gurevych, 2019; Gao et al., 2021), pre-trained language models (Raffel et al., 2022; Devlin et al., 2019; Liu et al., 2019; He et al., 2023). To rule out the confounder in ANALOGYKB, we also add knowledge-enhanced models, ERNIE (Zhang et al., 2019) and LUKE (Yamada et al., 2020) which contain the relational knowledge between entities. Moreover, we also randomly sampled 10,000 data points from the ANALOGYKB without checking and filtering, i.e., + ANALOGYKB (w/o check), to prove the necessity of filtering. After human examination, nearly about 63% of data points do not form analogies. Previous benchmarks, except E-KAR, do not have a training set. Thus, we fine-tune LMs on their small development set.2
The results presented in Table 5 show that: 1) The performance of sentence embeddings is inferior to word embeddings. The rationale is that such word analogy is based on relational rather than semantic similarity between two sentences. Therefore, taking the difference between two word embeddings is a more reasonable yet still problematic approach for finding word analogies. 2) Incorporating entity knowledge cannot improve model performance on analogy recognition; 3) The training data without checking brings noise and even degrades model performance, further emphasizing the importance of high-quality data in ANALOGYKB. 4) Training models on ANALOGYKB can signifi-
2Details about the baselines and training process are shown in Appendix C.2 and C.3. The statistical significant test is shown in Appendix C.5
cantly improve the model performance on analogy recognition by a large margin.
How much do analogies of analogous relations in ANALOGYKB contribute to performance? We create two ablated variants from ANALOGYKB to train the models: 1) Analogies of the same relations, denoted as Datasame: we randomly sampled 10,000 data of the same relations as an ablated variant. 2) Pseudo analogies, denoted as Datapseudo: for each data point, we randomly sample 5 concept pairs from the ANALOGYKB and choose one as the query, one as the answer, and the remaining three as distractions. This makes sure that ANALOGYKB indeed imposes analogical reasoning ability on the model rather than simply data augmentation. We adopt two settings: only train RoBERTa-Large on 10,000 data (i.e., Pre-trained) and first train RoBERTa-Large on 10,000 data and then fine-tune it on the specific benchmarks (i.e., Fine-tuned).
The results in Figure 4 show that: 1) Analogies of analogous relations in ANALOGYKB are rather important for models to comprehend analogies with more abstract and complex relations. 2) Training models on randomly constructed analogy-style data even drags down model performance, further emphasizing the importance of ANALOGYKB.3
How do data sizes and model sizes affect performance? We use T5-Large as the base model to examine the effects of training data size on model
3We also compare the data from different KB sources, which is shown in Appendix C.4.
performance. We first train the model on data from ANALOGYKB, and fine-tune it on E-KAR. As illustrated in Figure 5(a), increasing the amount of training data from ANALOGYKB improves model performance. Figure 5(b) shows the results of different-sized T5 models on 10,000 data points from ANALOGYKB. We find that the larger models get less of a performance gain from E-KAR, indicating that they learn more from ANALOGYKB and can better generalize to E-KAR.

heading: 4.2 Analogy Generation Evaluation
text: This task can be formulated as a text generation task: completing the D given A, B, C to form a plausible analogy A is to B as C is to D. Analogy generation is of more practical use, since the generation of familiar analogies could be helpful to
comprehend the source problem.
Does ANALOGYKB support analogy generation? To answer this question, we investigate two settings: For smaller LMs, we randomly sample 1 million data points from ANALOGYKB. Then we fine-tune T5-Large on ANALOGYKB (named AnalogyT5) to compare vanilla T5. For LLMs, we convert the query and analogies from ANALOGYKB into InstructGPT embeddings, retrieve the top-8 most similar analogies based on cosine embedding similarity, and use them as examples in the prompt. We test models on 100 test data sampled from three challenging benchmarks, which are not found in the training set.4 Each generation is evaluated by three annotators with Fleiss’s κ = 0.93. The results in Table 6 show that, in both pre-training and in-context learning, ANALOGYKB enables better analogy generation, and the analogies of analogous relations prove significantly valuable to the performance of models.
Does ANALOGYKB help LMs generalize to outof-domain analogies? Despite its high coverage of common concepts (§ 3.5), ANALOGYKB contains few analogies related to metaphor and science which are not common in the KGs and thus out-
4Detailed information on the training process and the results on six benchmarks are shown in Appendix D.1 and D.3. We also conduct the impact of data and model sizes and case studies for further analysis in Appendix D.2 and D.4.
of-domain. To examine whether ANALOGYKB can generalize the ability of LMs to reason about these analogies, we test AnalogyT5 on the SCAN dataset (Czinczoll et al., 2022), which has 449 analogies of metaphor and science domains. For smaller LMs, we follow the original experimental setup and compare the models trained on ANALOGYKB (see Appendix D.5 for details). For LLMs, we retrieve the top-8 most similar analogies from ANALOGYKB as examples, in contrast to zeroshot settings, retrieving from BATS and E-KAR. The results shown in Table 7 reveal that 1) For smaller LMs, training on BATS even worsens performance on SCAN. However, training on E-KAR with complex analogies can indeed improve the model performance on SCAN. 2) Compared to EKAR, ANALOGYKB can further help both LLMs and smaller models generalize to out-of-domain analogies.
Can ANALOGYKB better support analogy explanation for LLMs? Analogy explanation needs LLMs to provide a reasonable explanation for a given analogy, which more closely simulates the process of human reasoning and knowledge explanation. In this setting, we first retrieve top-8 most similar analogies based on cosine embedding similarity. Then, we ask GPT-4 to generate explanations for the analogies given relations and use them as examples in the prompt. We test InstructGPT003 and ChatGPT on 100 data samples from E-KAR, and employ two annotators to judge whether the explanations are correct with Fleiss’s κ = 0.97). The results in Figure 6 are consistent with Table 6, demonstrating that ANALOGYKB can facilitate better analogy explanation for LLMs, and the analogies of analogous relations are significantly valuable for performance.

heading: 5 Conclusion
text: In this paper, we introduce ANALOGYKB, a million-scale analogy KB to improve model performance in analogical reasoning tasks. We identify two types of analogies in existing KGs, i.e., analogies of the same and analogous relations, and utilize LLMs with minor human examinations to find them. ANALOGYKB demonstrates its great value in assisting both smaller LMs and LLMs with the resolution of analogy recognition and generation tasks, especially with analogies of analogous relations in ANALOGYKB.

heading: Limitations
text: First, this paper only considers analogies involving one or two relations and primarily concentrates on analogies in the form of “A is to B as C is to D”. However, analogies may involve the combination of multiple relations of multiple entities or even events. For example, an engineer can learn the eye cross-section by taking the analogy of the camera structure. Here, the analogy involves multiple entities and relations in the two systems (camera and eye): Aperture should be analogous to pupil since both are channels for light to enter and black paint should be analogous to choroid since both absorb light to prevent it from bouncing and reflecting.
Second, our ANALOGYKB is constructed using data from Wikidata and ConceptNet, which do not include analogies in other domains such as the scientific domain. For example, it would be challenging for LMs trained on ANALOGYKB to reason about an analogy such as Protein synthesis in a cell is like a factory assembly line as it would require a deep understanding of biological and industrial processes, which is not well-covered in our data sources. Also, ANALOGYKB is stored in the form of tuples, but in practice, some analogy situations may not be easily converted to this format. Future research should address how to bridge this gap.
Due to the limited computational resources, we only use a subset of ANALOGYKB. Assuming unlimited computational resources, the far-stretching goal of this project is to enable the discovery of new, better analogies for applications such as explanation (e.g., science popularization), text polishing, and case-based reasoning. So, with the full scale of the data, we can train a specialized open-source large language model (e.g., Llama 2) in such related tasks with data from ANALOGYKB so that these models can discover novel analogies
and understand new concepts and knowledge with analogical reasoning ability.

heading: Ethics Statement
text: We hereby acknowledge that all authors of this work are aware of the provided ACL Code of Ethics and honor the code of conduct.
Use of Human Annotations The annotations of relation pairs in ANALOGYKB are implemented by annotators recruited by our institution. The construction team remains anonymous to the authors, and the annotation quality is ensured by using a double-check strategy as described in Section 3. We ensure that the privacy rights of all annotators are respected throughout the annotation process. All annotators are compensated above the local minimum wage and consent to the use of ANALOGYKB for research purposes, as described in our paper. The annotation details are shown in Appendix A.2.
Risks The database is sourced from publicly available sources, Wikidata and ConceptNet. However, we cannot guarantee that it is free of socially harmful or toxic language. Additionally, analogy evaluation relies on commonsense, and different individuals with diverse backgrounds may have varying perspectives.

heading: Acknowledgement
text: We thank the anonymous reviewers for their valuable comments. This work is supported by the Chinese NSF Major Research Plan (No.92270121), Shanghai Science and Technology Innovation Action Plan (No.21511100401) and the National Natural Science Foundation of China (No. 62102095). The computations in this research were performed using the CFFF platform of Fudan University.

heading: A Details of ANALOGYKB

heading: A.1 Terminology Definition in ANALOGYKB
text: To better understand the schema for analogies in ANALOGYKB, we list the terminologies in Table 9.

heading: A.2 Crowd-sourcing Details
text: We have recruited a team of two undergraduates. We pay each annotator $8/h, exceeding the local minimum wage. The screenshots of the instructions and annotation interface are shown in Figure 8.

heading: B Benchmark
text: We compare our methods with baselines and human performance in 6 different benchmarks. An example of these benchmarks is given in Table 8 For benchmarks without training sets, we only finetune models on their validation sets.
• E-KAR (Chen et al., 2022): a Explainable Knowledge-intensive Analogical Reasoning benchmark sourced from the publicly available Civil Service Examinations (CSE) of China, which contains linguistic, commonsense, encyclopedic, and cultural (e.g., idiom and historical) knowledge. This dataset contains 870 training data, 119 validation data, and 262 test data. The SOTA model on this benchmark is proposed by Chen et al. (2022).
• BATS (Gladkova et al., 2016): is Bigger Analogy Test Set containing more than 1,000 analogies. The analogies can be divided into four categories: lexicographic, encyclopedic, derivational and inflectional morphology. This dataset contains 199 validation data and 1799 test data. The SOTA model on this benchmark is proposed by Ushio et al. (2021).
• UNIT 2 (Boteanu and Chernova, 2015): a benchmark using word analogy problems from an educational resource. This dataset contains 24 validation data and 228 test data. The SOTA model on this benchmark is proposed by Ushio et al. (2021).
• UNIT 4 (Boteanu and Chernova, 2015): this benchmark also comes from an educational resource but is harder than U2. This dataset contains 48 validation data and 432 test data. The SOTA model on this benchmark is proposed by Ushio et al. (2021)
• Google (Mikolov et al., 2013b): a benchmark for intrinsic evaluation of word embeddings proposed by Google, which contains semantic and morphological relations. This dataset consists of 50 validation data and 500 test data. The SOTA model on this benchmark is proposed by Chen et al. (2022)
• SAT (Turney et al., 2003): a benchmark constructed from SAT exams in the US college admission test consisting of 374 word analogy problems. This dataset contains 37 validation
data and 337 test data. The SOTA model on this benchmark is proposed by Ushio et al. (2021).
As shown in Table 10, We list the overlap rates of ANALOGYKB with other analogy datasets. The overlap rates are calculated as (Data in ANALOGYKB Data in Other Datasets) / (Data in Other Datasets). Specifically, one data sample, i.e., "A is to B as C is to D" can be changed into two tuples (A, R1, B) and (C, R2, D), where R1 and R2 can be exactly the same or analogous. If both tuples are present in ANALOGYKB, the overlap rate for this data instance is considered greater than 0. The results indicate that ANALOGYKB contains a portion of the data from other analogy benchmarks, exhibiting high coverage. However, after our checking, we confirm that the training data sampled from ANALOGYKB, which is used to train LMs, does not contain the test data from other analogy benchmarks. This confirms the absence of data leakage, underscoring that LMs on ANALOGYKB can significantly improve the model performance on analogy recognition and generation tasks.

heading: C Analogy Recognition Task

heading: C.1 Data Construction
text: To pre-train RoBERTa-Large on ANALOGYKB, we randomly sample 5,000 analogies of the same relation and 5,000 analogies of analogous relations from ANALOGYKB and formulate them into the multiple-choice question-answering format. Specifically, for each instance, we randomly sample a concept pair from ANALOGYKB as a query and select another concept pair from the analogous relation as the answer. Then, we randomly sample 3 concept pairs from the relations that can not be analogous to the relation of the query as distractions.
We also randomly sample 10,000 data of the same relations as an ablated variant to show the effectiveness of analogies of analogous relations (denoted as Datasame). The construction method
is similar, except that the query and answer are derived from the same relation. Additionally, we randomly sample 10,000 data points from ANALOGYKB and construct analogy-style data (denoted as Datapseudo). Specifically, we randomly sample 50,000 concept pairs without considering analogous relations from ANALOGYKB as the data pool. For each data point, we randomly sample 5 concept pairs from the data pool and choose one as the query, one as the answer, and the remaining three as distractions.

heading: C.2 Details of Baselines

heading: Word Embedding and Sentence Embedding
text: For the method of pre-trained word embeddings, we follow the method proposed by Ushio et al. (2021). And represent word pairs by taking the difference between their embeddings. Then, we choose the answer candidate with the highest cosine similarity to the query in terms of this vector difference. For the method of sentence embedding, we convert query A:B to "A is to B" and choose the answer candidate ("C is to D") with the highest cosine similarity to the query.

heading: C.3 Training Process
text: To pre-train language models on the sample data from ANALOGYKB, we follow the code from Huggingface 5. Since previous benchmarks, except E-KAR, do not have a training set, we fine-tune LMs on their small development set (about 300 samples). To achieve hyperparameter search, we maximize performance on the development set of E-KAR (119 data samples) as a compromise. The training settings are: batch size = 64, learning rate = 3e-5, dropout rate = 0.1 and training epoch = 10.

heading: C.4 Comparison with Different KB Sources
text: We also create two ablated variants to train the models to evaluate the necessity of ConceptNet and Wikidata: 1) Analogies from ConceptNet, denoted as Datacon: we randomly sampled 10,000 (the same size as before) data of the relations only in ConceptNet as an ablated variant. 2) Analogies from Wikidata, denoted as Datawiki: we randomly sampled 10,000 data of the relations only in Wikidata as an ablated variant. The results in Figure 7 show that ANALOGYKB can combine the commonsense knowledge of ConceptNet and the entity knowledge of Wikidata and thus exhibits superior
5https://huggingface.co/docs/transformers/ tasks/multiple_choice
performance in improving the analogy-making ability of models compared to utilizing a single data source.

heading: C.5 Significant Test
text: For the results in Table 5, we demonstrate that the random sampling of data does not greatly impact the accuracy through the statistical significance test. Specifically, we sample the training data of ANALOGYKB twice with different random seeds and run our method on these benchmarks in Table 5. Then,
we implement a t-test on the two results with a 0.05 significance level. The result is not significant (pvalue: 0.208), and thus we can not reject the null hypothesis (H0: r1 − r2 = 0, where ri=(Acc. of E-KAR, Acc. of BATS, Acc. of UNIT 2, Acc. of UNIT 4, Acc. of Google, Acc. of SAT)); Furthermore, we fix the training data of ANALOGYKB and run our method on the benchmarks in Table 5 twice with different random seeds. The result is insignificant (p-value: 0.250), and thus, we can not reject the null hypothesis (H0: r1 − r2 = 0).
For the results in Figure 4, we conduct a statistical significance test on Data and Datasame. We
average the accuracy of the two settings and implement a t-test with a 0.05 significance level. The null hypothesis H0 is r1 − r2 = 0, and the H1 is r1 − r2 > 0, where r1 and r2 are the lists of benchmarks’ average accuracy of Data and Datasame in Pre-trained and Fine-tuned settings. The result is significant (p-value: 0.012), and we can reject the null hypothesis H0. Thus, we can conclude that analogies of analogous relations in ANALOGYKB are rather important for models in the analogy recognition task.

heading: D Analogy Generation Task

heading: D.1 Training Process
text: To construct the training data, we convert A:B::C:D to “A is to B as C is to D” and let T5-Large generate the concept D given the input text “A is to B as C is to”. The training settings are: batch size = 32, learning rate = 3e-5, dropout rate = 0.1 and training epoch = 20.
D.2 The impact of data sizes and model sizes
For the analogy generation task, we have examined the effects of training data size and model size on model performance. The results in Fugure 12 and Fugure 13 show that: 1) By incorporating a larger volume of data from ANALOGYKB, we observe a gradual improvement in model performance, revealing the essential role of ANALOGYKB. 2) Only larger models with enough training data can boost the ability to generate reasonable analogies.

heading: D.3 Results on Six Benchmarks in Analogy Generation Tasks
text: We expanded the experiments in Table 6 to six analogy benchmark tasks. The results in Table 14 indicate that compared to analogies with simple and same relations, ANALOGYKB is more crucial for
models to understand analogies with more abstract and complex relations, such as E-KAR, UNIT 4, and SAT.

heading: D.4 Case Study
text: We are curious whether LMs trained on ANALOGYKB can generalize to novel analogies. After manual inspection, we observe from Table 15 that, AnalogyT5 can generate a reasonable concept D for the input. AnalogyT5 also generates reasonable analogies of analogous relations, such as “triangle” is to “area” as “cube” is to “volume”. However, analogies about adjectives are more error-prone, possibly due to the paucity of adjectives in ANALOGYKB. We also discover that training on ANALOGYKB enables LMs to generate reasonable analogies by changing concept B while holding fixed A (i.e., electron) and C (i.e., earth).

heading: D.5 Out-of-domain Analogy
text: Dataset SCAN (Czinczoll et al., 2022) is an analogy dataset consisting of 449 analogy instances clustered into 65 full-concept mappings. The overlap rate of ANALOGYKB with SCAN is only 2.67%. An example mapping in SCAN is shown in Table 16. Unlike the previous analogy dataset, SCAN mainly contains metaphorical and scientific analogies, which are abstract and thus rarely appear in the corpus and are difficult for LMs. In addition, each concept in SCAN only has one token and SCAN is not confined to the word analogy task due to its full-concept mappings.
Baseline The original paper evaluates the analogical capabilities of GPT-2 and BERT on the SCAN dataset. The authors convert the analogy instance to “If A is like B, then C is like D”, and force the models to predict the last token of the sentence. For GPT-2, the model needs to generate the last token given the input text “If A is like B, then C is like”. For BERT, the authors first mask D as “If A is like B, then C is like [MASK]” and let the model predict word D.
In addition, the authors fine-tune the LMs on the 1,500-sized set of BATS (i.e., + BATS) and investigate whether the models learn about analogical reasoning in general after training on BATS. We follow this setting and randomly sample 1,500 data from ANALOGYKB and fine-tune the LMs on the sample data (i.e., + ANALOGYKB). To prove the necessity of analogies of analogous relations, we randomly sample 1,500 analogies of the same relations as
an ablated variant (i.e., + ANALOGYKBsame). We also added LMs trained on the 800 data points of E-KAR (i.e., + E-KAR).
We further explore the performance of LLMs on the SCAN dataset. Specifically, we also adopt the prompt in Table 11 to let LLMs generate the word D. Since each concept in SCAN has only one token, we can obtain the top 5 results from InstructGPT through the OpenAI API.
Evaluation Metrics Following Czinczoll et al. (2022), we report accuracy, recall@5 and the mean reciprocal rank (MRR) to compare the performance of models. To reduce computing, we only consider
the MRR of the first token of the target word among the top 10 predicted tokens. The RR of a label is 0 if it is not in the top 10 tokens.
Training Process The training settings of GPT-2 and BERT are: batch size = 128, learning rate = 3e-5, dropout rate = 0.1 and training epoch = 10.

heading: Future Work
text: Future research should address how to bridge this gap. Due to the limited computational resources, we only use a subset of ANALOGYKB. Assuming unlimited computational resources, the far-stretching goal of this project is to enable the discovery of new, better analogies for applications such as explanation (e. g. , science popularization), text polishing, and case-based reasoning. So, with the full scale of the data, we can train a specialized open-source large language model (e. g. , Llama 2) in such related tasks with data from ANALOGYKB so that these models can discover novel analogies
and understand new concepts and knowledge with analogical reasoning ability