Solving Hard Analogy Questions with Relation Embedding Chains
Authors: Nitesh Kumar
Steven Schockaert
Venue: EMNLP 2023 Main (2023)
Keywords: relation embedding
analogy questions
ConceptNet
knowledge graphs

Abstract: Modelling how concepts are related is a central topic in Lexical Semantics. A common strategy is to rely on knowledge graphs (KGs) such as ConceptNet, and to model the relation between two concepts as a set of paths. However, KGs are limited to a fixed set of relation types, and they are incomplete and often noisy. Another strategy is to distill relation embeddings from a fine-tuned language model. However, this  is less suitable for words that are only indirectly related and it does not readily allow us to incorporate structured domain knowledge.  In this paper, we aim to combine the best of both worlds. We model relations as paths but associate their edges with relation embeddings. The paths are obtained by first identifying suitable intermediate words and then selecting those words for which informative relation embeddings can be obtained. We empirically show that our proposed representations are useful for solving hard analogy questions.

Solving Hard Analogy Questions with Relation Embedding Chains
Nitesh Kumar and Steven Schockaert
Cardiff NLP, School of Computer Science and Informatics Cardiff University, United Kingdom (kumarn8, schockaerts (ectaaerrrtt@eerdffff )ecardiff
ac.
Abstract
There is also tradition in Natural Language Pro- cessing (NLP) to model relations as embeddings. For instance, DIRT (Lin and Pantel, 2001) and LRA (Turney, 2005) are early examples of meth- ods which used vectors to represent the relation between two words. More recently, relation em- beddings have been obtained using fine-tuned lan- guage models. For instance, RelBERT (Ushio al., 2021a) has achieved state-of-the-ar results on sev- eral analogy datasets using fine-tuned RoBERTa- large model (Liu et al., 2019). The strengths and weaknesses of relation embeddings are comple- mentary to those of KGs: relation embeddings lack transparency and cannot easily incorporate external knowledge, but they are capable of capturing subtle differences, making it possible encode relational knowledge in way that is considerably richer than what can be achieved with KGs. final drawback of relation embeddings that they are best suited for concept pairs that have a clear and dircet rela- tionship. For instance, they can encode the relation between umbrella and rain, but are less suitable for encoding the relation between umbrella and cloudy (e.g. if it is cloudy, there is a chance of rain, which means that might take umbrella).
Modelling how concepts are related is a central topic Lexical Semantics. common strat- egy is to rely on knowledge graphs (KGs) such as ConceptNet, and to model the relation be- tween two concepts as of paths. However, KGs are limited to a fixed set of relation types, and they are incomplete and often noisy. An- other strategy is to distill relation embeddings from a fine-tuned language model. However, this is less suitable for words that are only indi- rectly related and it does not readily allow to incorporate structured domain knowledge. In this paper, we aim combine the best of both worlds. We model relations as paths but asso- ciate their edges with relation embeddings. The paths obtained by first identifying suitable intermediate words and then selecting those words for which informative relation embed- dings can be obtained. We empirically show that our proposed representations are useful for solving hard analogy questions."
1 Introduction
Many applications rely on Knowledge graphs (KGs) to model the relationship between concepts. For instance, KGs have been used to characterise how an answer candidate is related to the concepts that appear in a question (Yasunaga et al., 2021; 1; Zhang et al., 2022b) and to help interpret visual scenes (Gu et al., 2019). In such applications, rela- tions are modelled as KG paths, which has two key advantages: (i) we can easily inject domain -seccfic knowledge and (ii) the interpretable nature of KG paths offers a degree of transparency. But KGs also have important drawbacks. They use a fixed set of relation types, which may not be fine-grained enough. KGs are also inevitably incomplete and, especially in the case of commonsense KGs such ConceptNet (Speer et al., 2017), noisy.
In this paper, propose hybrid approach which combines the advantages of KGs with those of relation embeddings. The idea is conceptually straightforward. We represent relations as paths, where nodes correspond to concepts, but rather than associating edges with discrete types, we label them with relation embeddings. We will refer to such paths as relation embedding chains. Clearly, this approach allows us to model indirect relation- ships, while keeping the flexibility of embeddings, as well as some of the interpretability of KG paths.
We are still faced with the challenge of select- ing suitable paths. KGs are insufficient for this purpose, given their noisy and incomplete nature. Our solution relies on the following idea: to de- cide whether a + x - is a suitable path for
'Source code to reproduce our experimental results and the model checkpoints are available in the fol- lowing repository: /ttps://githult com/no tishrooyal solivingtardknalooyyuasttito

---

modelling the relationship between and b, what matters is whether we have access to an informa- tive relation embedding for the word pairs (a, a)) and (xr b). Motivated by this, we first develop a classifier to predict whether a given RelBERT em- bedding is informative or not. We then generate possible relational paths, using ConceptNet as well standard word embeddings, and filter these paths based on the informativeness classifier.
(2019) highlighted the noisy nature of many Con- ceptNet paths. To address this issuc, they trained a system to predict path quality based on crowd- sourced judgments of naturalness. More recently, ConceptNet paths have been used to provide ex- ternal knowledge to NLP systems, for instance in question answering systems (Lin et al., 2019; Ya- sunaga al., 2021; Zhang et al., 2022b; Jiang al., 2022; Sun et al., 2022). In such cases, Graph Neural Network is typically used to process the paths, and the resulting representations are then in- tegrated with those obtained by a language model. An important distinction with our work is that we focus on unsupervised settings. KG paths are es- pecially helpful for question answering over KGs. However, most approaches rely matching chains of discrete relation types (Das et al., 2022; Zhang et al., 2022a). Chains of relation embeddings, as we study in this paper, make it possible to model finer relationships.
While relation embedding chains are expressive, we sometimes need simpler representation, espe- cially in unsupervised settings. We therefore also study how the information captured by relation embedding chains can be summarised as single vector, without relying on task task-specific supervision.
To evaluate the usefulness of relation embedding chains, we focus on word analogy questions. Given a query word pair (e.g. word:language) and a set of candidate word pairs, the task is to select the most analogous candidate (e.g. music).. We show that relation embedding chains are well-suited for answering hard analogy questions.
3 Scoring RelBERT Embeddings
Given word pair (a, b). we write rab E Rd for the corresponding RelBERT embedding. While ReIBERT can provide a relation embedding for any pair of words, not all these embeddings are equally informative. In particular, we would like to dis- tinguish embeddings rab which encode a specific relationship from embeddings that rather reflect the lack of any (known) relationship between a and b. Unfortunately, the relation embeddings of different unrelated word pairs are typically not similar.
2 Related Work
Modelling analogies has been a long-standing topic of interest in NLP (Turney, 2005, 2012). Recent years have seen a significant increase in interest in this topic, fuelled by the success of large lan- guage models (LLMs). For instance, Bhavya et al. (2022) used LLMs for generating explanations of scientific concepts involving analogies, while Sul- tan and Shahaf (2022) used LLMs for identifying analogies between procedural texts. However, most work has focused on modelling analogies between word pairs (Ushio et al., 2021b,a; Chen et al., 2022: Czinczoll et al., 2022a; Li et al., 2023; Yuan et al., 2023), which is also the setting we consider in this paper. We focus in particular on RelBERT (Ushio et al., 2021a), which is the state-of-the-art on sev- eral benchmarks. RelBERT is RoBERTa-large model that was fine-tuned the relational simi- larity dataset from SemEval 2012 Task (Jurgens et al., 2012). Given a word pair, RelBERT com- putes& a relation embedding by feeding that word pair as input to the fine-tuned RoBER model and averaging the output embeddings.
To address this, we want to train a classifier to predict whether a given relation embedding rab is informative or not. However, this requires access to set of related word pairs Pos and a set of unre- lated word pairs Neg. The set Neg can simply be constructed by choosing random word pairs. While we may occasionally end up with word pairs that are actually related in some way, the probability that this is the case for randomly chosen words is sufficiently low for this not to be problematic. However, constructing the set Pos is less straightfor- ward, since we are not looking for words that have particular relation, but rather for words that have any kind of (clear) relationship. If the examples are not sufficiently diverse, then our classifier will sim- ply detect whether rab corresponds to one of the relations that were considered during training. One of the few relevant large-scale resources we have at our disposal is ConceptNet. However, during
The use of KG paths for modelling relations be- tween concepts has also been extensively studied. For instance, Boteanu and Chernova (2015) pro- posed the use of ConcepiNet paths for explaining why two word pairs are analogous. Zhou et al.

---

initial experiments, we quickly found ConceptNet be too noisy for this puppose. We therefore in- stead relied on GPT-42 to generate diverse set of around 1 1,,00 positive examples. The prompt we used to obtain a sufficiently diverse set of examples is shown in the Appendix D.
veluction
entice
indacement
CREE
Mundichnenn
perenade
cajol:
To create set of negative examples, of the same size, we corrupted the positive examples. We then trained logistic regression classifier on the re- sulting training set. Throughout this paper, we will write inf(rab) for the probability that rab ex- presses an informative relationship, according to this classifier. We will refer to this value as the informativeness of the embedding rab-
fattery
ConceptNet Link
persuasion
Mivinz Link Prediated
Wokdnng
Scuamlic Suaoillng Link
Figure 1: For the concept pair (blandishment, coax), the integration of missing link prediction and semantic smoothing strategies identifies seduction, inducement, enficement, persuade, flattery, persuasion, and wooing as intermediate concepts. The concepts entice and ca- jole used for smoothing not considered intermediate concepts; they merely help in inducing additional in- termediate concepts. If only ConceptNet were used determine intermediate concepts, flattery and persuade would be the only ones identified.
4 Connecting Concepts
The relationship between two concepts a and is sometimes easiest to explain using an intermedi- ate concept x. For instance, umbrella is related to cloudy because (i) an umbrella protects against rain and (ii) rain is only possible if it is cloudy. A natural strategy for identifying such intermediate concepts is to find paths of length two connecting the target words a and in ConceptNet. However, the coverage of ConceptNet is limited, and many relevant intermediate concepts might not be found in this way. One possible solution would be to consider a sequence of intermediate concepts, and model the relation between a and in terms of path + æ1 ... In + b. However, longer ConceptNet paths are often too noisy be use- ful. Moreover, in practice single intermediate concept is usually sufficient, so rather than consid- ering longer paths, we propose two strategies to find additional links between concepts based on word embeddings. They are illustrated in Figure 1.
finds words that are related in different ways. For instance, some links that are not present in Con- ceptNet and have been added using this approach include: (dog, owners), (cashier, grocery store), (helium, noble gases), (drug trafficking, illegal). and (disinfectant, sterilization). Such links clearly enrich ConceptNet in non-trivial way.
Semantic Smoothing For the second strategy. we only consider the top-5 neighbours, noting that these are often either synonyms or morphological variations of the same word (e.g. cloud and cloudy). Specifically, rather than only considering r as an intermediate concept if æ is connected to both and b, we now consider as an intermediate concept as as it is connected neighbours of a (or a itself) and one of the 5 nearest neighbours of b (or b itself). We will refer to this second strategy as semantic smoothing.
of the nearest
Missing Link Prediction We add missing links based on the informativeness classifier from Sec- tion 3. Specifically, we first use standard word embedding to find the top-k most similar words to o, with e.g. k 500. For each of these words y. we add a link between a and y inf(ray) > 0.75. We add missing links from in the same way. We will refer to this strategy as missing link prediction. Note that this strategy not only finds synonyms or morphological variations of words but also often
5 Condensing Relation Embedding Chains
The strategy from Section 4 allows us to identify intermediate concepts 21. Zn that can help to de- scribe the relation between two given words and b. Each intermediate concept z; corresponds to chain raxyi 'x of two relation embeddings. These relation embedding chains can encode relationships in fine- ine-grained way, but in practice we often need a more compact representation. We therefore train a simple model to summarise a set of relation em-
Phttps://opena /ttp://///peecccceeem wateonaa..cc/rreeee com//research/ott
'Since we focused on unsupervised settings in this paper. we cannot tune this value. Compared to a threshold of 0.5, the choice of 0.75 refiects the idea that we want to be selective: if in doubt it seems hetter not to add the link.

---

bedding chains {Taxıi Exib. Taxa Txnb) for given concept pair (a, b) as single vector. Our model has the following form:
Solving Analogy Questions
Analogy questions involve query pair (a, b) and number of candidate answers (x1, U1),,,,,, (Ty, 9k The aim is to select the candidate (2r, M6) whose relationship is most similar to that of the query pair. When using RelBERT, we simply choose the pair (I', yi) for which cos(Tab. Fxy;) is maximal.
Sab - ( Fxib), (Taxx (xab))
The function Φ Rd x Rd Rm intuitively doing a kind of relational composition: it aims to predict relation embedding for the pair (a, b) from the relation embeddings Fax; and Fxyb- These pre- dicted embeddings are then combined using pool- ing function ʃ Ih x x mh >> IR Finally, the resulting vector is mapped back to d-dimensional embedding using the decoder 1/ R. -> Rdd
Identifying Hard Analogy Questions Our key hypothesis is that the informativeness of the Rel- BERT vectors, as predicted by the classifier from Section 3, can tell us whether using ReIBERT vec- tors is a reliable strategy for a particular analogy question. The lower the informativeness of rab or "xyr with (a, b) the query and (æ;; Ui) the cho- answer candidate, the less confident can be about the chosen answer. We define our confidence in ReIBERT's prediction follows:
For the decoder vi, we simply use a linear layer. For the pooling function f. we experimented with sum-pooling and max-pooling. We found the result to be largely insensitive to this choice. Throughout this paper, we will therefore fix as summation. Finally, the composition function & is implemented follows:
conf(a, (,, b, v) = min(inf(rabs) inf(rr)) (3)
with (a, b) the query pair and (z, y) the candidate selected by RelBERT. In cases where the confi- dence is low, we aim to improve the prediction by taking advantage of relation embedding chains.
(Frxx 1xb) GeLU(A(rax rxb) b) (1)
where we write for vector concatenation, A is matrix, and b is bias vector.
Condensed Relation Chain Comparison We can straightforwardly solve analogy questions us- ing condensed relation chains. In particular, we use the model from Section 5 obtain relation embeddings Sxy for the query and the candidate pairs. We then proceed in the same way as with ReIBERT, choosing the answer candidate (x,, Mi) for which cos Sxyy) is maximal.
Training Our focus is on solving analogy ques- tions, which is an unsupervised problem. In the absence of speciipe training data, we train the model to predict the RelBERT embedding Tab, us- ing the following loss:
L =
cos (Sab; rab)
Direct Relation Chain Comparison We can also solve analogy questions by using relation chains directly. Let Ci....C Cu be the intermediate concepts for the query pair (a, b) and let 21,-., 2n be the intermediate concepts for an answer can- didate (z, y). Then we evaluate the compatibility comp(a, b, x, v) of this candidate as follows:
(a.b)
where Sab is the vector predicted from the relation embedding chains, and rab is the RelBERT embed- ding. The sum in (2) ranges over concept pairs (a, b) from ConceptNet for which the informative- ness (ra) is sufficiently high. This is important ensure that the model is not trained on a noisy su- pervision signal. Specifically, we only considered concept pairs (a, b) for which inf(rab) > 0.75.
a
Σ máx sim (inc Tebb Txay Iz,y) 5=1
Note that while we train the model on concept pairs (a, b) which have an informative RelBERT embedding, our aim is use this model for pairs for which this is not the case. We thus rely on the assumption that composition model which is trained on word pairs with informative RelBERT vectors will generalise in a meaningful way to word pairs for which this is not the case.
The idea is that when (a, b) and (I,, y) are analo- gous, for most relation chains rac; Fe,b connecting and b there should exist similar relation chain Tx Tzyy connecting x and y. The similarity be- tween relation chains can be evaluated as follows:
simj (Tac; Te,b: T'xey ( min min(cos(ran (cos(racç: Tx2y); cos cos(be,b, (osyb,,
(4)

---

In other words, two relation chains are similar if their first component is similar and their second component also similar. Our default configu- ration uses this approach. However, this may be too strict. For instance, suppose that Te and Fzzy capture the located-at relation, while Fac; captures part-of and T'x, captures is-a. Then the relation chains Facyi Te,b and Exaji T2y both essentially en- code that is located considered be similar according to (4). To allow such relation chains to be identified as being simi- lar, we will also consider the following alternative:
same repository. Finally, we include the English version of E-KAR* (Chen al., 2022). E-KAR contains questions questions involving word pairs and questions involving word triples. We only con- sidered the former for our experiments. We use accuracy as the evaluation metric.
There are some important differences between these datasets. For instance, Google and BATS focus morphological, encyclopedic, and in the case of BATS, lexical relations. SCAN focuses on scientific and creative analogies, requiring models to link relationships from one domain (e.g. the solar system) to relationships to a completely different domain (e.g. atoms). The other datasets focus on abstract conceptual relations, but they cover range of difficulty levels (for humans): SAT is obtained from college entrance tests; U2 is aimed at chil- dren from primary and secondary school; U4 has difficult levels ranging from college entrance tests to graduate school admission tests; E-KAR was derived from Chinese civil service exams.
but they would not be
simz(Taes Fe, Exayi Feyy) = -co(($(ó(Tac)) (eab)), th (T(Txx Tejy)))
(5)
with & and t the composition function and de- coder of the model for condensing relation chains. Finally, we will also consider a baseline strategy which ignores the order of the relation embeddings:
sima (Face; Tegb; Exaji Tejy)
Methods The RelBERT repository on Hugging- face contains number of different ReIBERT vari- ants. We have used the model trained using Noise Contrastive Estimation on SemEval 2012 Task 2 (Jurgens et al., 2012), as this variant outperforms the original variant from Ushio et al. (2021a).
= (rac + rc,b, Xxzz + r2zyy)
7 Experiments
We empirically analyse our proposed strategies for solving hard analogy questions. We are specifi- cally interested in the following research questions. (i) How suitable is the confidence degree (3) proxy for estimating the difficulty of an analogy question? (ii) How suitable are relation embedding chains for answering difficult analogy questions? (iii What are the best ways for learning and using these relation embedding chains?
For our implementation of missing link predic- tion we combined the top-250 neighbours (in terms of cosine similarity) according to the ConceptNet Numberbatch word embedding (Speer al., 2017) with the top-250 neighbours according to GloVe (Pennington et al., 2014). For semantic smoothing we used to top-5 neighbours from GloVe.
In terms of baselines, our primary emphasis is on comparing the proposed methods with RelBERT, which is the state-of-the-art relation embedding model?. To provide additional context, we have also obtained results with GPT-4. 8 Following ear- lier work on using LLMs for solving analogy ques- tions (Yuan et al., 2023), we ask the model to gen- crate explanations (Wei et al., 2022) and include a few solved questions as part of the prompt. The
7.1 Experimental Setup
Datasets We evaluate models on number of different analogy question datasets. First, use the 5 datasets that were used by Ushio et al. (2021b): the SAT dataset proposed by Turney et al. (2003); the U2 and U4 datasets which were ob- tained from an educational website; and reformu- lations of the Google (Mikolov et al., 2013) and BATS (Gladkova et al., 2016) datasets into the multiple-choice analogy question format. We ob- tained these datasets from the RelBERT reposi- t. We also use reformulation of SCAN (Cz- inezoll et al., 2022b) into the multiple-choice anal- ogy question format, which is available from the
Availaabl at https::/ekar--leaddbbbar github.thhb.t.. io co/relbert/ 6 'Available from https: / relbert-robertar taoberttllargeee nce- - semeval2012-ee 8-400
"While Yuan et al. (2023) have reported higher results on some datasets using another fine-tuned RoBERTa model, their method does not learn relation embeddings but solves the task as a multiple-choice question answering problem. Moreover. their model was fine-tuned on the validation split of each benchmark, which makes the results incomparable. DDeaall of our prompt can be found in the appendix.
'https: http://hggggaaaaa ace. wwwcc...o/aaaaaeee wwt../aatateeseeeeerrrr..e amalogy_question questions

---

SAT
U2
BATS
Conf
RelB
Cond
Dir
GPT4
ReIB
Cond
Dir
GPT4
RelB
Cond
Dir
GPT4
RelB
Cond
Dir
GPT4
0.25)
40.7
51.9
37.0
59.3
17.6
35.3
52.9
76.5
26.9
46.2
46.2
38.5
32.1
46.7
55.5
90.5
0.255, 0.5)
51.4
48.6
60.0
54,3
38
33.3
47.6
42.9
43.8
43.8
50.0
62.5
66.3
72.9
68.7
94.0
0.5. 0.5.0.75)
78.0
71.0
67.0
65.0
65.9
70.5
61.4
70.5
59.3
52.3
48.8
61.6
68.0
69.5
67.0
87.3
0.755, 1.0]
80.6
77.7
66.9
78.3
78.1
67.8
59.6
76.0
71.0
68.0
57.7
75.0
89.9
84.8
80.1
91.0
Google
SCAN
E-KAR
Avg
Conf
RelB
Cond
Dir
GPT4
RelB
Cond
Dir
GPT4
RelB
Cond
Dir
GPT4
RelB
Cond
Dir
GPT4
0.255.
42.0
$2.0
66.0
98.0
23.2
22.9
18.9
23.5
35.0
40.0
35.0
30.0
31.1
42.1
44.5
59.5
0.255, 0.5)
50.0
50.0
57.1
100.0
26.7
28.3
27.4
20.4
42.3
34,6
19.2
50.0
45.5
44.5
47.2
60.6
0.750.75)
54.1
51.4
64.9
100.0
32.2
32.5
30.9
22.1
53,5
55.8
55.8
48.8
58.7
57.6
56.5
65.0
0..75, 1.0]
93.0
88.1
84.9
98.9
33.6
35.5
35.0
31.4
52.3
50.8
41.5
60.0
71.2
67.5
60.8
72.9
Results on analogy questions different difficulty levels, across datasets (accuracy). The best performing model per dataset and difficulty level is shown in bold.
Table I:
SAT
U2
U4
BA
GO
SC
EK
four hybrid methods. The idea of these methods is to rely on RelBERT for the easy questions and on either Cond or Direct for the hard questions. For instance, Cond uses Cond for questions with difficulty levels below 0.5 (as estimated us- ing (3)) and ReIBERT for the others. A number of conclusions can be drawn.
GPT-4
70.3
71.9
68.8
90.8
99.0
23.5
51.3
ChatGPT!
65.8
68.9
70.8
90.6
97.8
InstructGPT 003
48.9
59.0
57.9
82.8
96.7
ReIBERT
73.6
67.5
63.0
80.9
81.4
27.3
48.7
Condensed
70.6
62.7
60.9
79.2
78.6
27.9
48.1
Direct
63.8
58.3
54.4
75.7
79.2
25.7
40.9
Conde0.25
74,5
68.9
64,1
82.0
82,4
27.2
49.4
Cond
74.2
68.4
64.1
82.7
82.4
27.6
48.1
Confidence scores faithfully predict question dif-
Direct.a.25
73.3
70.2
64,1
82.7
83,8
25.6
48.7
Direct.co.o
74.2
71.1
64.8
82.9
84.4
25.7
44.8
ficulty We can see that the performance of Rel- BERT is closely aligned with the predicted diffi- culty level. On average, across all datasets, the ac- curacy ranges from 31.1% % for the hardest questions to 71 71.2% for the easiest questions. This pattern can be observed for all datasets. Moreover, the pre- dicted difficulty level is also aligned with the perfor- mance of the other models. For instance, GPT-4 on average also performs best on the questions with high confidence values, especially for SAT, U4, E-KAR and SCAN. For Google and BATS, GPT- 4 performs well throughout. This suggests that the confidence score (3) is successful in predict- ing intrinsic question difficulty, rather than merely predicting where RelBERT is likely to fail.
Table 2: Overall results of the different models (accu- racy). Results with f were taken from ( (Yuan et al., 2023).
results of GPT-4 should be interpreted with some caveats, however. For instance, it is well-known that the results of LLMs can be highly sensitive to the choice of the prompt, so it is likely that better results are possible. Moreover, it is possible that GPT-4 has seen some of the datasets during train- ing, which could lead inflated results. Besides GPT-4, we also include the results that were ob- tained by Yuan et al. (2023) using ChatGPT and InstructGPToos with chain-of-tee -ffthughtt prompting.
Relation embedding chains are helpful for hard questions Focusing on the performance for the hardest questions, with confidence levels in [0.0,0.25), we can see that Condensed outperforms RelBERT on all datasets, with the exception of SCAN (where the results are close). Direct outper- forms in most datasets well, but not in SCAN and SAT. For the questions with a difficult range in [0.25,0.50), Direct still outperforms ReIBERT in most cases, with E-KAR now the only excep- tion. For the questions with confidence level in the range (0.50,0.75) the performance of RelBERT is
7.2 Results
Table shows our main results, focusing on four methods: RelBERT (RelB), condensed relation em- bedding chain comparison (Cond), direct relation chain comparison (Dir) and GPT-4. The results are broken down based on our confidence in Rel- BERT's prediction, as computed by (3). The over- all results per dataset are summarised Table In this table, we also report the LLM results obtained by Yuan et al. (2023). We furthermore consider

---

SAT
U2
U4
BA
GO
SC
EK
Direct Relation Chain Comparison Table 3 compares our default configuration (Direct), which uses sim; to measure the similarity between rela- tion embedding chains, with a number of variants. First, the rows labelled simy and simy show the im- derstand the nature of relation embedding chains.
Direct
63.8
58.3
54.4
75.7
79.2
25.7
40.9
sim2
67.4
60,5
56.9
80.3
79.8
29.1
42.9
sims
59.3
54.8
53.0
66.9
70.8
24.4
42.2
CN + ss
57.6
58.3
55.1
67.9
66.4
22.3
39.0
CN mlp
58.8
64.5
59.3
77.4
78.0
22.8
44.8
CN only
56.7
57.5
57.4
71.2
62.0
21.4
41.6
mlp only
60.8
57.0
54.9
74.9
75.4
22.9
42.2
pact of replacing (4) by, respectively, (5) and (6) for measuring the similarity. The model based on simg combines elements of the direct and condensed re- lation chain comparisons. Accordingly, we see that its performance is typically in between that of these two methods. However, for BATS, Google and SCAN it actually outperforms both. The com- paratively poor results for sim3 show that the order of the relation embeddings matters.
CN types
40.4
42.1
40.5
56.3
50.4
15.3
37.7
Table 3: Analysis of direct relation chain comparison.
SAT
U2
U4
BA
GO
SC
EK
Condensed
70.6
62.7
60.9
79.2
78.6
27.9
48.1
CN + ss
69.4
63.2
60.4
76.9
74.6
27.3
48.1
CN + mlp
67.1
64.5
60.9
79.7
81.4
25.4
48.1
CN only
68.8
59.2
60.4
74.0
71.2
24.4
48.1
Next, the table compares a number of variations in how the relation embedding chains themselves constructed. As we discussed in Section 4, our main method combines ConceptNet with two aug- mentation strategies: semantic smoothing (ss) and missing link prediction (mlp). The rows labelled CN + ss and CN + mlp show the impact of only using one of these augmentation strategies. Note that our default configuration for Direct is CN + mlp + Furthermore, the row labelled CN only shows the results when we only use ConceptNet paths, without applying either of the two augmen- tation strategies. Finally, the row labelled mlp only shows the results of variant which only relies on the links predicted by the missing link prediction strategy, without using ConceptNet at all. As can be seen, the links from ConceptNet and those pre- dicted using mlp lead to the best performance. In fact, the CN + mlp variant achieves the best re- sults in several cases. Interestingly, mip on its own already performs quite well, which shows that re- lation chains can be constructed even without the help of an external KG.
mlp only
65.9
67.1
59.3
78.0
80.2
25.1
48.1
Table 4: Analysis of condensed relation chain compari- son.
generally quite similar to that of Condensed and Direct. Finally, for the easiest questions, RelBERT is clearly better, with the exception of SCAN. In ac- cordance with these observations, in Table 2 we can see that the hybrid models generally outperform RelBERT, except for SCAN and E-KAR where their performance is similar. For SAT, three of the hybrid models outperform the state-of-the-ar re- sult of ReLBERT. Finally, comparing Condensed and Direct there is no clear winner, with the former performing better for the easiest questions and the latter performing better for the hardest ones.
GPT-4 performs best overall It is also notable how similar the GPT-4 results are to the ChatGPT results obtained by Yuan et al. (2023). However, GPT-4 and ChatGPT do not provide relation em- beddings and can thus not replace models such as RelBERT in many applications (e.g. for retrieval tasks). In Table 1, we can also see that Condensed and Direct can sometimes outperform GPT-4 on the hardest questions, namely for U4 and E-KAR. In- terestingly, these are also the benchmarks with the highest intended difficulty level for humans. More- over, for SCAN, which requires making suitable abstractions, GPT-4 generally underperforms.
Finally, we also evaluate a variant which does not rely on relation embeddings (CN types). In this case, we only use ConceptNet paths for find- ing intermediate concepts. We then compute the compatibility comp(a, a, b, x, y) as follows:
Σ máx [racc = -xzz Aro = Tzyyl
7.3 Analysis
Here Tzy is the ConceptNet relation type of the edge that connects x and y, and 1[a] is if the condition a is satisfied and 0 otherwise. As can be seen, this variant performs poorly. The contrast
We now provide further analysis about the direct and condensed relation chain comparison methods. We also include a qualitative analysis to better un-

---

between this method and CN only reflects the im- pact of (i) the noisy nature of ConceptNet and (ii) the fact that the fixed relation types are often less informative than relation embeddings.
the word pairs involved have clear and direct re- lationship. The bottom half of the table similarly shows examples of questions that were predicted to be hard. These examples are hard for differ- ent reasons. Some examples are challenging be- cause they involve clear but indirect relationship (e.g. shopping:bank card). Others involve rather abstract relations (e.g. aloof.connectedd. The ex- ample from Google reflects the fact that RelBERT was not trained on named entities.
Condensed Relation Chain Comparison Table 4 compares our default approach based on con- densed relation embedding chains (Condensed) with some variants. Note that our default configura- tion for Condensed is CN + mlp + ss. Specifically, as in Table 3, we analyse the impact of the semantic smoothing (ss) and missing link prediction (mlp) strategies. We can broadly observe the same pat- tems; e.g. again find that CN mlp achieves the best results in several cases and that it is possible to obtain competitive results without using a KG.
8 Conclusions
Relations between concepts are typically modelled either as KG paths or as relation embeddings. this paper, we have proposed a hybrid approach, which uses paths whose edges correspond the re- lation embeddings. This is useful to model rela- tionships between concepts that are only indirectly related, and more generally for concept pairs whose relation embedding is unreliable. Our approach cru- cially relies on classifier which can predict the informativeness of RelBERT embedding. This classifier used (i) to identify analogy questions where we should not rely on RelBERT embeddings; (ii) to select reliable RelBERT vectors for training model for condensing relation embedding chains; and (iii) to identify informative paths beyond those in ConceptNet. We have relied on GPT-4 to gener- ate training data for the informativeness classifier, which allowed us to address the lack of suitable existing datasets.
Qualitative Analysis Table shows some ex- amples of informativeness scores for word pairs from ConceptNet. As these examples illustrate, concept pairs with high scores consistently have a clear relationship. Those with the lowest scores either do not have any obvious relationship, or they have a relationship which is not captured by their ReIBERT embedding. As an example of this lat- ter case, the table contains pairs that are linked by the "sounds like" (chad: chat) and "rhymes with" relation (time fine).
Table shows examples where ReIBERT made an incorrect prediction while Direct picked the right answer. In each case, we also show the most influential intermediate concept for the query and find the interme- diate concept e for the query (a, b) and the inter- mediate concept for the answer (zz y) for which sim (Fac; reb, F'xx Iny) ) maximal. These exam- ples illustrate some of the different roles that in- termediate concepts can play. For instance, in the
answer pairs. Specifically,
Limitations
Our evaluation has focused on solving analogy questions, which are useful because they allow for a direct evaluation of different approaches for modelling relationships. While analogies play an important role in certain downstream tasks, we sentially regard this as in intrinsic evaluation task. It thus remains a matter for future work anal- yse the usefulness of relation embedding chains in downstream tasks (e.g. retrieving relevant cases for case-based reasoning methods). From a practical point of view, using relation embedding chains is more involved than directly using RelBERT embed- dings. To address this, the augmentation strategies can be applied offline, by creating graph of re- lated concepts. While the corresponding RelBERT embeddings can also be computed offline, storing millions of relation embeddings takes up a large amount of space. In certain applications, it may
example from SAT, the intermediate worth con- demmable makes it possible to characterise the pair reprehensible:ccomdemm in terms of near-synonym peppreeensibleeccmdemmeee phological variation (condemiaalee:conden) example from U4 illustrates a case where the close similarity between two terms (vernacular:regionaa) is more casily recognised by RelBERT as a compo- sition of two antonyms (vernacular:national and antiional:regonall The example from SCAN illus- trates case where the word pairs involved are only indirectly related.
and kind of mor- The
The top half of Table 6 shows examples of ques- tions that were predicted to be easy. As can be seen,

---

Query
RelBERT
Correct
Interm. query
Interm. answer
SAT
reprehensible: condemn
depraved admire surplus squander budget: austere
estimable praise eulogy praise fluctuation irregular thinking mistake
condemnable seduction national
praiseworthy
U2
blandishmentt:
homage
U4
vernacular: regional
normal
SCAN
processing: bug
memorize mistake hieroglyph cloth
computing
working
BATS
package: parcel
flower: blossom
courier
grower
Table 5: Examples analogy questions which are incorrectly answered by RelBERT, while being correctly
answered by Direct. For each question, we also show the intermediate concept for the query pair (a,b) and the intermediate concept 2 for the answer pair (z, y) for which sim; (Tac; Teb, Txy is maximal.
Dataset
Query
Candidates
SAT
abbreviation:word
abridge:book, laminate: layer, inhibitidea, expedite:mail, invoke:deity
U2
sewing:craft
gasoline:fuel, salt:food, fatc:science, sunrisctart
U4
tragmatic:ppactiien
trivial:negligible irritating:leassing. conaciousfaltterng opaque
EASY
BATS
market.
sofazcouch, murder:clothes, lazy:help, shirt:hutton
Google
hand:hands
horse:horses, swimming:swe dollars:goats dollarmouse
SCAN
earth:air
sun:space, planct:orbit, planct:sun, planet:elliptical, planet:space, planct-gravity. fish:fin, sheep.dog, locust:cicada, cattleggrass
E-KAR
bird: bings
SAT
shallowsdepth
apathetic:caring, salty:ocean, hhouht, lurid:shock, pious:faith
U2
conneccede
dellberraeeacciddentall rigid:frrm, ethereal:fleetingg logical:calculating
HARD
U4
cestidioos:particus
fanatical:endhusiastte odible:delicious, opppant. manipulatiermusteruu
BATS
shirt:button
tonne:kilogram, pie:tripod, fridge:appliance, sonata:door
Google
Yerevan:Armenia
Zagreb:Croos Croatia, Azerbaijan Denmark.. Dushanbe:Tume boy:girl
SCAN
money:effective
timezefficient,, schedule:quick, time:schedule, time:quick, time:slow,
shopping:bank card E-KAR
cooking:natural gas, driving:steering wheel, rumning:sncakers, travel:navigator
Table 6: Top: examples of analogy questions with confidence in [0.75, 1.0). i.e. questions predicted to be easy.
Bottom: examples of analogy questions with a confidence in [0, 0.25), i.e. questions predicted to be hard. In each case, the correct answer shown in bold.
Pair
Score
guage models: A case study of instructgpt. arXiv preprint arxiv:22
horse pony
0.99
intermarriag: intramarriage
0.99
Adrian Botcanu and Sonia Chernova. 2015. Solving and explaining analogy questions using semantic net- the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015. Austin, Texas, USA, pages 1460-1466. AAAI Press.
foresight: hindsight
0.96
addressahle: unaddressahle
0.95
balloon: popped
0.52
works. In Proceedings
call text
0.50
at one time: individually
0.49
fate : choice
0.47
farmer: : slicker
0.10
moment: ages
0.09
Jiangjie Chen, Rui Xu, Ziquan Fu, Wei Shi, Zhongqiao Li, Xinbo Zhang, Changzhi Sun, Lei Li. Yanghua Xiao, and Hao Zhou. 2022. E-KAR: benchmark for rationalizing natural language analogical reason- ing. In Findings of the Association for Computa- rional Linguistics: ACI. 2022, pages 3941-3955, Dublin, Ireland. Association for Computational Lin- guistics.
time: fine
0.01
chad chat
0.01
Table 7: Examples of the predicted informativeness scores for concept pairs from ConceptNet.
thus be preferable to compute the required Rel- BERT embeddings only when they are needed.
Tamara Czinczoll, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2022a. Scientific and creative analogies in pretrained language mod- els. In Findings the Association for Computational Linguistics: EMNLP 2022, pages 2094-21000 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Acknowledgments This work was supported by EPSRC grants EP/V025961/1 and EP/W003309/1.
References
Tamara Czinczoll, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2022b. Scientifie and creative analogies in pretrained language models. CoRR, abs/221 1. 15268.
Bhavya Bhavya, Jinjun Xiong, Chengxiang Zhai. 2022. Analogy generation by prompting large lan-

---

Rajarshi Das, Ameya Godbole, Ankita Naik, Elliot Tower, Manzil Zaheer, Hannaneh Hajishirzi, Robin Jia, and Andrew McCallum. 2022. Knowledge base question answering by case-based reasoning over subgraphs. In International conference on machine leurning, pages 4777-4793, PMLR.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/ 1907.100711999
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746-751, Atlanta, Georgia. Association for Computational Linguistics.
Anna Gladkova, Aleksandr Drozd, and Satoshi Mat- suoka. 2016. Analogy-hased detection of morpholog- ical and semantic relations with word embeddings: what works and what doesn't t. In Proceedings of the NAACL Student Research Workshop, pages 8-15, San Diego California, Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguisties.
Jiuxiang Gu. Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. 2019. Scene graph genera- tion with external knowledge and image reconstruc- tion. In IEEE Conference on Computer Vision and Patlern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2079, pages 1969-1978. Computer Vision Foundation/ / IEBE.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty-Firs AAAI Conference Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 4444-4451. AAAT Press.
Jinhao Jiang, Kun Zhou, Ji-Rong Wen, and Xin Zhao. 2022. great truths are always simple a rather simple knowledge encoder for enhancing the com- monsense reasoning capacity of pre-trained models. In Findings of the Association for Computational Linguistics: MMACL2022, pages 7730-1741. Seattle, United States. Association for Computational Lin- guistics.
Oren Sultan and Dafna Shahaf. 2022. Life a circus and we are the clowns: Automatically finding analo- gies between situations and processes. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3547-3562, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
David Jurgens, Saif Mohammad, Peter Turney, and Keith Holyoak. 2012. SemEval-2012 task 2: Measur- ing degrees of relational similarity. In #SEM 2012: The First Joint Conference on Lexical and Compu- tational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 356- 364, Montréal, Canada. Association for Computa- tional Linguisties.
Yueqing Sun, Qi Shi, Le Qi, and Yu Zhang. 2022. JointLK: Joint reasoning with language models and knowledge graphs for commonsense question answer- ing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 5049-5060, Seattle, United States. Asso- ciation for Computational Linguistics.
LearninTExpress. 2002. 50) Word Analogies Questions and Answers, edition. Delmar Cengage Learning.
Shuyi Li, Shaojuan Wu, Xiaowang Zhang, and Zhiyong Feng. 2023. An analogical reasoning method based on multi-task learning with relational clustering. In Companion Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, 7X, USA, 30 April 2023- 4 May 2023, pages 144 147. ACM.
Peter D. Turney. 2005. Measuring semantic similarity by latent relational analysis. In IJCAI-05, Proceed- ings of the Nineteenth International Joint Conference on Artificial Intelligence, Edinburgh, Scotland, UK, July 30- August 2005, pages I 1136-1141. Profes- sional Book Center.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. KagNet: Knowledge-awee graph net- works for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP, pages 2829-2839, Hong Kong, China. Association for Computational Linguistics.
Peter D. Turney. 2012. Domain and function: A dual- space model of semantic relations and compositions. J. Artif. Intell. Res., 44:533-585.
Peter D. Turney, Michael L. Littman, Jeffrey Bigham, and Victor Shnayder. 2003. Combining independent modules in lexical multiple-choice problems. In Re- cent Advances in Natural Language Processing III, Selected Papers from RANLP 2003, Borovets, Bul- garia, volume 260 of Current Issues in Linguistic Theory (CILT), pages 101-110. John Benjamins, Am- sterdam/ hirdaam/hilldelphia..
Dekang Lin and Patrick Pantel. 2001. Discovery of inference rules for question- answerin.......igg.. Nat. Lang. Eng. 744):343-360.

---

Asahi Ushio, Jose Camacho- Schockaert. 2021a. Distilling relation embeddings from pretrained language models. In Proceedings of the 2021 Conference Empirical Methods in Natu- ral Language Processing, pages 9044-9062, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Camacho-Collados, Collados, and Steven
Semantic relationship: This includes syn- onyms, antonyms, hyponyms, hypernyms. meronyms, holonyms, etc.
Function or purpose, e.g., key and lock.
Manner, way, or style: Concepts that describe the way in which another concept is accom- plished, e.g., limp and walk.
Asahi Ushio, Luis Espinosa Anke, Steven Schockaert, and Jose Camacho-Colladeo 2021b. BERT is to NL.P what AlexNet is to CV: Can pre-trained lan- guage models identify analogies? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 3609-3624, Online. Association for Computational Linguistics.
Symbol or representation: Concepts where one represents or symbolizes the other, e.g., dove and peace.
Degrees of intensity: Concepts that describe different degrees of intensity of particular situation, e.g., shower and monsoon.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NeurIPS.
Cause and effect: Concepts where causes the other results from the other, e.g., fire and smoke.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Perey Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 535-546, Online. Association for Computational Linguisties.
Sequence or hierarchy: Concepts that follow each other in a sequence or are organized hi- erarchically, e.g., manager and employee.
In order to identify large list such relation types, we asked GPT-4:
Siyu Yuan, Jiangjie Chen, Changzhi Sun, Jiaqing Liang, Yanghua Xiao, and Deging Yang. 2023, ANALO- GYKB: unlocking analogical reasoning language models with A million-scale knowledge base. CoRR, abs/2305.05994.
Let A and B be two concepts in natural language. When can we say that is related to B, and when is A not related to B?
Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022a. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 5773 5784.
We identified approximately 100 relationships through this approach, and then again used GPT-4 to generate examples for each of these relationships, using prompts of the following form:
way two concepts be related is
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, and Jure Leskovec. 2022b. Greaselm: Graph The Tenth International Conference on Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Tools and materials: Concepts where one tool material used to the other, e.g., "paintbrush" 'paint.'
interact
soning enhanced language models.
Now, generate 100 high-quelity examples such concept pairs.
Yilun Zhou, Steven Schockaert, and Julie Shah. 2019. Predicting conceptnet path quality using crowd- sourced assessments of naturalness. In The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pages 2460-2471. ACM.
We prompted the model once for each relation type to maintain a balanced number of examples for each relationship. The generated examples were utilized to train our informativeness classifier.
A Generating Examples of Related Concept Pairs with GPT-4
B Additional Training Details
Two concepts can be related in various ways, in- cluding, but not limited to, the following:
To train the composition model, we used 200,000 concept pairs from ConceptNet (version 5.6.0) with

---

Dataset
#Quest.
#Cand.
of the datasets, where the difficulty is measured in terms of the confidence score (3) as before.
SAT
337
5.0
U2
228
4.3
U4
432
4.2
D Solving Analogy Questions using GPT-4
BATS
1799
4.0
Google
500
4.0
SCAN
1616
74.5
To solve analogy questions with GPT-4, we used the following prompt, which was inspired by (LearningExpress, 2002).
E-KAR 154
Table 8: Overview of the considered datasets, showing for each dataset the number of questions in the test set (#Quest.) and the average number of options per question (#Cand.).
Many standardized tests, including high school entrance exams, the SATs, civil service exams, the GREs, and others, questions test both logic and reasoning skills and word knowledge. These questions test takers to identify relationships between pairs of words.
analogy
Conf
SAT
U2
U4
BA
GO
SC
EK
0.00, 0.25)
27
137
652
20
0.25, 0.5)
35
21
166
427
26
00.5, 0.75)
100
44
197
37
317
43
In order to solve analogy questions, you must first have a clear understanding of the words' definitions and then use that understanding to determine how the words are related. The key to solving an analogy question is to precisely describe the relationship between the pair of words and then apply the same relationship to determine which word completes the analogy. Most analogy questions rely on your ability to deduce the correct relationship between words and to draw logical conclusions about the possible answer choices.
[0.75, 1.0
175
146
272
1299
371
220
65
Table 9: Number of analogy questions for each of the difficult ranges.
informativeness scores greater than 0.75. The con- cepts belonged to the English language only. We did not consider ConceptNet relations "/r/NotCa- pableOf", "/n/NotDDesires", and "///NotHasProp- erty" for determining intermediate concepts. Our default augmentation strategy, i.e., CN+MLP+sS, identified approximately 50 intermediate concepts for each pair of concepts on average. We used Num- berbatch version 19.08 and Glove model glove- wiki-gie gitei-igiwwwor00000 in our augmentation strategy. The concept pairs for which the strategy could not determine any path were not considered for training. 10% of the selected concept pairs were utilized for validation. We trained a function Φ 1102 1004 + R", where the valida- tion set was used to choose the optimal dimen- sion n of the latent space. We observed in par- ticular that high-dimensions latent spaces outper- formed dimeesiameiionall ones, where we found n 81, 920 to be optimal. The Adam optimizer used with learning rate of 0.0025, and the important to
The relationships that are found in analogy questions fall into several general types.
1) Part to Whole. In this type of question, pair of words consists of a part and whole. For example, spoke wheel. A spoke is part of wheel.
2) Type and Category. These questions pairs of words in which one word is specific type in general category. For example, orange citrus. An orange is type of citrus.
3) Degree of Intensity. These questions test your ability discern among pairs of words. example, shower monsaon. shower light rainfall and monsoon is heavy rainfall.
of meaning
4) Function. These questions pair words that are related through function. For example, hammer build. hammer is to build.
model was trained for 10 epochs. note that we did not use validation splits from the analogy datasets to tune these hyperparameters, as we consider the unsupervised setting.
5) Manner.
This type analogy describes the manner, way, or style by which an action is accomplished. For example, shamble : walk. Shamble means to walk in an awkward manner.
Details about the Analogy Question Datasets
6) Symbol or representation. These questions pair words in which one word is the symbol of the other. For example, dove peace. A dove is a symbol of peace.
Some basic statistics about the considered datasets are shown in Table Table 9 shows how many questions there are in each difficulty range, for each
7) Action and significance. In this type of analogy one word describes an action and

---

Conf
RelB
Cond
Dir
GPT4
Conf
RelB
Cond
Dir
GPT4
0.00 0.25)
26.3
30.0
29.1
39,9
0.00 0.25)
43.9
45.5
48.6
72.5
0.255, 0.5)
39.6
41.4
41.2
46.5
0.225, 0.5)
39.3
39.9
41.3
$1.6
00.5..775. 0.75)
53.0
52.2
50.2
54.5
0.75)
41.5
45.6
41.9
52.4
[0.75, 1.0
81.2
76.9
71.4
82.8
[0.75, 1.0
67.3
64.5
60.4
68.2
Table 10: Results for the main experiments, micro- averaged across all datasets.
Table 11: Micro-averaged results, broken down accord- ing the confidence predicted by the informativeness classifier that was trained on Conceptnet.
the other word indicates the significance of the action. For example, sorrow. To signifies sorrow
ries the same weight, regardless of the dataset it appears in. The main conclusions remain the as the ones we could draw based on the macro- average. One difference that the spread in per- formance between the easiest and the hardest ques- tions is even wider for the micro-averaged results. For comparison, in Table 1 we also show the micro-averaged result, but in this case, we break down the results based informativeness classi- fier that was trained on examples from ConceptNet as positive examples. In contrast, for our main experiments, this informativeness classifier was trained on examples we obtained from GPT-4. As can be seen in Table 11, the informativeness classi- fier based on ConceptNet is far less successful in identifying easy and hard questions. For instance, GPT-4 actually achieves the best results the ques- tions that are predicted to be hardest.
Analogy questions also be used test word knowledge and factual content. Word knowledge questions
generally pairs of synonyms pairs of antonyms. Factual content questions demand certain level of general knowledge, and cannot deduced from the relationship alone.
word pair, your word pair from choices that analogously most similar. Also, give an explanation. The explanation should precise. will show some examples then you will have to do yourself.
choose
Query = ['banana', 'peel' ']; Choices = [[ [['egg', JJ, l'carrot', 'uproot J ['apple', "coe'', I, ['bread', 'slice'], ['corn'. "husk' ']]
Answer:
choice number 4; Explanation: banana has peel that can be removed, and corn has husk that can be removed.
Query = ['birds', 'wings']; Choices
[[ [t'moose',, 'antlers'], [ ['camel', 'hump 'J, [ ('spider', 'legs']. ['alligator', 'tail'] [ ['cat', "whisers' [J
Answer: choice number 2; Explanation: Birds have wings, and spiders have legs.
Query = L'berate', 'criticize'J, Choices [[' [['goad', 'urge'l, ['accuse', 'apologize'], ['regret', 'remember ['betray', 'follow]], [ ['evaluate', 'praise' ']]
Answer: choice number Explanation: To berate to criticize, and to goad is to urge.
Now, answer the follawing questions:
E Additional Results
Table reports the macro-averaged accuracy, as summary of the performance of the different methods across all datasets. In this macro average, each dataset carries the same weight. To comple- ment this result, Table 10 instead shows the micro- averaged result, where each analogy question car-


Reviews:
[{"review_id": "SnW69JTdMK", "reviewer": "Reviewer_zL7T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We now address the concern raised by the reviewer. \n\nQ1: Besides GPT-4, the proposed method is only compared with RelBERT on hard analogy questions. How about other models, and other tasks, such as relation classification.\n\nA1:  We chose to compare our method with RelBERT because it is the state-of-the-art model for the datasets we considered. Moreover, our specific focus is on showing the benefit of relation embedding chains, and since we use RelBERT for modelling such chains, comparing with RelBERT allows us to directly evaluate this aspect. \n\nWe did not consider relation classification because existing benchmarks focus on rather coarse-grained relations (where the benefit of using RelBERT embeddings is more limited), and because, by construction, they only involve word pairs that have a direct relationship.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It introduced a training model to learn relation embeddings by predicting the similarity of RelBERT embedding with relation embedding chains. Experimental results on hard analogy question shows the benefit of this representation method.", "reasons_to_reject": "Besides GPT-4, the proposed method is only compared with RelBERT on hard analogy questions. How about other models, and other tasks, such as relation classification. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a concept relation representation method, which views relations as paths, and labels them with relation embedding, which become relation embedding chains.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5YltnMwG0q", "reviewer": "Reviewer_yTYV", "summary": "", "questions": "I do not have specific questions for the authors.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the motivating comments. We now address the concern raised by the reviewer. \n\nQ1: The impact of learning a good relation embedding is rather limited in the current research environment.\n\nA1: We interpret \"current research environment\" to refer to large language models (LLMs) such as GPT-4. While it is clear that GPT-4 performs very well on analogy questions, our experiments also show that in some cases we can still get better results using relation embedding chains. More fundamentally, learning representations using smaller/efficient models remains important in various applications. For instance, relation embeddings have an obvious role to play in applications that currently rely on (commonsense) knowledge graphs, such as recommendation (where existing models cannot simply be replaced by an LLM). Representations also matter for information retrieval, and we believe that relation embeddings have an important role to play in this area as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of using word chains to represent open relations is very novel and interesting.  \n\nThe experiment is extensive and persuasive. \n", "reasons_to_reject": "The impact of learning a good relation embedding is rather limited in the current research environment.  However, this is still a very good idea worth the community to know, especially for the people who are working on constructing and leveraging knowledge graphs. \n", "questions_to_authors": "I do not have specific questions for the authors.", "paper_topic_and_main_contributions": "The paper first discusses the pros and cons of using vectors, instead of a fixed set of relation types, to represent the relations between entities. \n\nIn this paper, a new approach is proposed to combine the good perspectives of using vectors and fixed relation types. The authors propose to use relation embedding chains. \n\nThe proposed method is built on the RelBERT, where the relation embedding is first computed from any pair of words. Then they train a classifier for related/unrelated concepts based on the samples generated from the large language models, namely GPT4. A trained classifier can be used to give the informativeness of the embedding. \n\n\nThe generated relation embedding tasks can be used for answering hard analogy questions. The proposed method achieved state-of-the-art performance on the analogy questions. \n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kHBz2grECn", "reviewer": "Reviewer_S1H7", "summary": "", "questions": "- Which similarity measures (sim1, sim2 or sim3) is used for the results in Table1 for Direct method? \n- The third baseline in equation 6 for measuring similarity between two relation chains is not motivated. Why did you decide to add the two relation embeddings? Why not concatenate? Subtracting? \n- Table 3 did not show the result of the full proposed model (CN +ss+mlp). ", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We now address the concerns raised by the reviewer one by one as below.\n\nQ1: As the first step of missing link prediction is to find the semantically similar words y to a concept a, then the link is added if the relation embedding between an and y is informative. This step of adding relational links between concepts is questionable as it focuses on words that are semantically similar based on their attributes from static word embeddings. By doing so, we don't expect to get words that are related to a by some relations instead we got synonyms and semantically similar neighbours.\n\nA1: When using static word embeddings, some of the neighbours we find are indeed synonyms or morphological variations, but we also often find words that are related in different ways. Note that we check for informative relation embeddings among the top 500 neighbors, a number we find sufficient to go beyond just synonyms and morphological variations. For instance, some links that are not present in ConceptNet and have been added using this approach include the following:\n\n(“dog”, “owners”): Reflects the relationship that dogs can have owners.\n(\"cashier\", \"grocery store\"): Cashiers have a key role in grocery stores.\n(\"helium\", \"noble gases\"): Helium belongs to the category of noble gases.\n(\"drug trafficking\", \"illegal\"): Drug trafficking is illegal.\n(“disinfectant”, “sterilization”): Disinfectants are used for sterilization.\n\nSuch links clearly enrich ConceptNet in a non-trivial way. To make this clearer in the paper, we will add a qualitative analysis about the kinds of semantically related words that are found by our strategy.\n\n\nQ2: The proposed method is weak in solving easy analogy questions. However, we expect a method that works well on hard cases to also perform well on easy cases. Does that mean for easy questions the relation between an and b are direct and the missing link prediction injects noisy paths?\n\nA2: Easy questions indeed typically require modelling direct relationships, which can be modelled well using RelBERT relation embeddings (see Table 6 for examples of easy and hard questions). In fact, the proposed methods are quite robust in the presence of noisy paths. The main limitation is rather that no suitable intermediate concepts may be found in some cases. In other words, the effectiveness of the method mostly depends on the quality of the most informative path.\n\nIt is indeed quite intuitive that we should only try to model relationships using intermediate concepts (i.e. using relation embedding chains) in cases where the relationship cannot be modelled directly. One of our most significant findings is that we can identify those cases using the proposed informativeness classifier. As our results in Table 2 show, this allows hybrid methods such as Cond_{<0.25} to perform well on both easy and hard questions.\n\n\nQ3: Which similarity measures (sim1, sim2 or sim3) is used for the results in Table1 for Direct method?\n\nA3: We used “sim1” for the Direct method. We will clarify this in the paper.\n\n\nQ4: The third baseline in equation 6 for measuring similarity between two relation chains is not motivated. Why did you decide to add the two relation embeddings? Why not concatenate? Subtracting?\n\nA4: We introduced the third baseline, referred to as \"sim3,\" to specifically analyse the importance of modeling the order of relations in a chain. Unlike \"sim1\" and \"sim2,\" \"sim3\" does not take into account the order of relations. We find that, as a result, it performs poorly in comparison.\n\n\nQ5: Table 3 did not show the result of the full proposed model (CN +ss+mlp).\n\nA5: Our default configuration, which is labeled 'Direct' in the table, is CN + ss + mlp. We will clarify this in the paper.\n\n\nQ6: Regarding typos and presentation improvements.\n\nA6: We are grateful for the suggestions and will address them in the paper. \n\nIndeed, \"y\" is a typo; we meant 'a -> x1 -> x2 -> ... -> b.'\n\nAs suggested, we will improve the 'missing link prediction' section for greater clarity. Regarding the question of whether 'y' must be connected to both 'a' and 'b': the answer is no, it may not be. Specifically, a path 'a -> y -> b' exists if 'a -> y' is the predicted link and 'y -> b' is not a predicted link but exists in ConceptNet.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The novelty of using chains of relation embeddings to solve analogy questions. \n- Extensive experiments that include analogy questions benchmarks and multiple baselines. \n- The incorporation of relation embeddings and relational paths between a pair of concept to solve analogy questions is novel, as most related work focus on the two concepts in the given query ignoring the intermediate concepts that could lead to the targeted relation in a compositional way. \n- Interesting analysis of the main results in Table 1 (with highlighted summaries). ", "reasons_to_reject": "- As the first step of missing link prediction is to find the semantically similar words y to a concept a, then the link is added if the relation embedding between an and y is informative. This step of adding relational links between concepts is questionable as it focuses on words that are semantically similar based on their attributes from static word embeddings. By doing so, we don't expect to get words that are related to a by some relations instead we got synonyms and semantically similar neighbours. \n- The proposed method is weak in solving easy analogy questions. However, we expect a method that works well on hard cases to also perform well on easy cases. Does that mean for easy questions the relation between an and b are direct and the missing link prediction injects noisy paths? ", "questions_to_authors": "- Which similarity measures (sim1, sim2 or sim3) is used for the results in Table1 for Direct method? \n- The third baseline in equation 6 for measuring similarity between two relation chains is not motivated. Why did you decide to add the two relation embeddings? Why not concatenate? Subtracting? \n- Table 3 did not show the result of the full proposed model (CN +ss+mlp). ", "paper_topic_and_main_contributions": "This paper proposes a method for solving analogy questions by considering paths between concepts and relation embeddings distilled from large language models. To do so, they first learn a binary classifier that predicts a confidence score for a relation embedding RelBERT to use it as a threshold for predicting links between concepts. To find a path between two concepts, they propose to use word similarities in addition to the links provided in a structured knowledge graph (e.g., ConceptNet). This is done by first finding the top-k nearest neighbour for a concept in a given pair using static word embeddings, then choosing those words with informative relation embeddings (high confidence score) with the concept. Experimental results on multiple datasets show that the proposed method performs well for hard analogy questions in which the confidence score for the answers using the RelBERT embedding is low.  ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Rewrite the abstract to clarify some important concepts such as informative relation embeddings? hard analogy questions? \n- Similarly, the last part of the introduction section is not well-motivated as you have to briefly explain your proposed approach/techniques. In addition, the analogy question (hard ones) are not explained. how do you classify a question to be hard or not? As this terminology is also used in the title, it is important to clarify it better in early sections.  \n- For better analysis, show the answers for the RelBERT and the proposed method for the shown easy/hard questions in Table 6. \n- Clearly state the setting for the hybrid models shown in Table 2. \n- It is better to explain the notations of Table 3 in its caption for quick access.\n- The missing link prediction section from line 209 needs clarification. How do you add links from b to? Do you mean y must be connected to both a and b? \n- In line 201, do you mean a-> x1 -> x2-> …->b not y? "}}]

Author rebuttal:

Meta-review: This paper presents an application of relation embeddings as a method for finding relation chains between concepts in the ConceptNet knowledge graph, implemented also for finding analogies, the task on which the method is evaluated.

The reviewers agree that the paper presents a novel method in a clear way, with some expressing excitement at its elegant nature and impressive results over the benchmarks. All concerns were addressed by the author responses, which were acknowledged by the reviewers. If accepted, I recommend the authors evaluate the benchmarks on at least one more competing method as proposed by reviewer zL7T, and add a short discussion featuring examples of nearest-neighbor words to alleviate the first concern raised by reviewer S1H7.

As a postscript, although I am not an official reviewer and have not carefully read the paper, but on a quick skim I noticed that there is no reference to literature on multi-hop relations and models, e.g. https://aclanthology.org/2020.acl-main.412/ (and much preceding work in ACL and non-ACL venues). This seems fairly relevant, and I hope the authors can look into this literature as well.
Decision: Accept-Main