Self-Supervised Rule Learning to Link Text Segments to Relational Elements of Structured Knowledge
Authors: Shajith Ikbal
Udit Sharma
Hima Karanam
Sumit Neelam
Ronny Luss
Dheeraj Sreedhar
Pavan Kapanipathi
Naweed Khan
Kyle Erwin
Ndivhuwo Makondo
Ibrahim Abdelaziz
Achille Fokoue
Alexander G. Gray
Maxwell Crouse
Subhajit Chaudhury
Chitra K Subramanian
Venue: EMNLP 2023 Findings (2023)
Keywords: self-supervised learning
rule learning
relation linking
question answering
TL;DR: A self-supervised rule learning approach for relation linking

Abstract: We present a neuro-symbolic approach to self-learn rules that serve as interpretable knowledge to perform relation linking in knowledge base question answering systems. These rules define natural language text predicates as a weighted mixture of knowledge base paths. The weights learned during training effectively serve the mapping needed to perform relation linking. We use popular masked training strategy to self-learn the rules. A key distinguishing aspect of our work is that the masked training operate over logical forms of the sentence instead of their natural language text form. This offers opportunity to extract extended context information from the structured knowledge source and use that to build robust and human readable rules. We evaluate accuracy and usefulness of such learned rules by utilizing them for prediction of missing kinship relation in CLUTRR dataset and relation linking in a KBQA system using SWQ-WD dataset. Results demonstrate the effectiveness of our approach - its generalizability, interpretability and ability to achieve an average performance gain of 17% on CLUTRR dataset.

Self-Supervised Rule Learning to Link Text Segments to Relational Elements of Structured Knowledge
Shajith Ikbal", Udit Sharma"', Hima Karanam" Sumit Neelami, Ronny Lusst, Dheeraj Sreedhar', Pavan Kapanipathi", Naweed Khan Kyle Erwin Ndivhuwo Makondo Ibrahim Abdelazizi, Achille Fokoue', Alexander Gray", Maxwell Crouse', Subhajit Chaudhury', Chitra K Subramanian
IBM Research
Abstract
Text
Kate Winslet shared the screen with Leonardo DiCaprio
We present a neuro-symbolic approach self- learn rules that serve as interpretable knowl- edge to perform relation linking in knowledge base question answering systems. These rules define natural language text predicates as weighted mixture of knowledge base paths. The weights learned during training effectively serve the mapping needed to perform relation linking. We use popular masked training strat- egy to self-learn the rules. A key distinguishing aspect our work is that the masked train- ing operate over logical forms of the sentence instead of their natural language text form. This offers opportunity to extract extended con- text information from the structured knowledge source and use that to build robust and hu- man readable rules. We evaluate accuracy and usefulness of such learned rules by utilizing them for prediction of missing kinship relation in CLUTRR dataset and relation linking in KBQA system using SWQ-WD dataset. Re- sults demonstrate the effectiveness of our ap- proach- its generalizability, interpretability and ability to achieve an average performance gain 17% on CLUTRR dataset.
Logical Statement (triple)
share-screen
('Kate Winslet', 'Leonardo DiCaprio')
(Prdianil
(Surce stchy
Masked logical statement
??
Kate Winslet', 'Leonardo DiCaprio')
wd:Q202765
wd:Q38111
KB (Wikidata) Entity
KB (Wikidata) Entity
Figure Illustration of logical statement of NL text, masked logical statement, and the grounding (i.e. link- ing) of unmasked elements of the statement to knowl- edge base (Wikidata).
reasoning capabilities of symbolic systems. For ex- ample, by accurately mapping predicate component of the NL text shown in Figure 1, i.e., share-screen to its corresponding relational elements within KB, we should be able to successfully utilize KBQA sys- tems answer questions like Whether Kate Winslet shared screen with Leonardo DiCaprio? and Who all have shared screen with Kate Winsler? How- ever, learning these mappings is challenging for various reasons: lexical variations, modeling defi- ciencies to handle complex linking involving mul- tiple KB predicates?, and limited training data to build models in supervised fashion.
1 Introduction
Relation linking is key step in many Natural Lan- guage Processing (NLP) tasks including Semantic Parsing, Triple Extraction, and Knowledge Base Question Answering (KBQA). Its goal to ac- curately map predicate components of a natural language (NL) text segment onto their correspond- ing predicate (i.e., relational) elements within a knowledge base (KB). Such mapping would en- able question answering systems to exploit the deep
Various approaches have been tried in the past work, including graph-based approaches (Pan al.,
Provided entity mentions in the NL text also appro- priately mapped to their corresponding entity elements in the KB. through entity linking (Wu al., 2019). For example, in Figure entity mention Kate Winslet NL text is mapped corresponding KB (Wikidata) entity wd:0200765.
*On several occasions, there no single relational elc- ment but multiple connected relational elements within the KB that actually correspond to a NL text predicate. These connected relational elements constitute a path within the KB. For example, two connected relational elements as in Fig- ure (i.c.. one-hop path involving a reverse KB relation edge cast-member and a forward KB relation edge cast-member) correspond to NL text predicate share-screen. Note that formal expression of this KB path is given in Figure 2, i.c., cast-member 2 (x, 2) & cast-member((. 11).
IIM Research, Bangalore/Gurugraa, India.
IIBM Research, J. Watson Research Center, Yorktown Heights, New York, United States.
'IBM Research, Johannesburg, South Africa.
"Correspondence c-mails: shajmoha @ in. ih.ihm.com,
udit.sharman @ in com, hkaranam @ in com

---

Rule: share-screen E w1 P161+)
getting masked. Given that the three elements of triple play different semantic roles, the masked training need to be adapted to suit the role played by the element being masked. As will be seen later in the paper, such differences reflect mainly on the loss computation during training.
cast member " (x, z) cast member (z, y) Description of the rule: has shared screen with if is a cast member of movie z where ys also cast member;
Figure 2: Illustration of human readable rule for the example shown in Figure 1 and the corresponding ex- tended knowledge base context shown in Figure 3. Note that inverse knowledge base predicate (i.e., reverse knowledge hase relation) in the rule is denoted by su- perscript as described in Section 3.2.
The main contributions of our work are:
1. The first self-supervised rule learning ap- proach that uses an adaptation of masked lan- guage objective for logical forms extracted from text. We have applied this to relation linking, a prominent task for understanding natural language.
2019; Dubey et al., 2018), linguistic features based approaches (Sakor et al., 2019a,b; Lin et al., 2020), and recently neural approaches that achieve state- of-the-art performance (Rossiello et al., 2021: Mi- hindukulasooriya et al., 2020; Yu et al.. 2017). However, these approaches have the following drawbacks: they are (a) data hungry requiring huge amount of supervised data for training, (b) non-interpretable by design, and (c) non-trivial to adapt and generalize to new domains or knowledge graphs. In this paper, we propose a neuro-symbent symbolic approach that formulates relation linking as a rule learning problem in the space of structured knowl- edge sources. Our approach tries to learn human readable rules of the form illustrated in Figure 2 and use them to perform relation linking, thus re- sulting in explainable relation linking model. In contrast to the existing approaches that build link- ing models in a supervised fashion, we adopt a self-supervised learning approach that offer ben- efits: does not require annotating large amounts of training data, and offers generalization across domains.
2.
The output of our model is completely inter- pretable; the set of rules obtained to perform relation linking can explain the predictions of the model. An example of a learnt rule for the SWQ dataset shown in Figure 2.
3. We evaluate our approach on two datasets CLUTRR and SWQ demonstrating (a) Gener- alizability over two different knowledge bases as context on the datasets, (b) Effectiveness of relation linking where results show signifi- cant gains (17% gain in average performance) for CLUTRR across different test sets along with superior systematic generalization and competitive results SWQ, (c) Interpretabil- ity, by showing qualitative results of the learnt rules.
Related Work
Relation linking has been important for various NLP tasks such as semantic parsing, knowledge graph induction (Gardent et al., 2017; Chen et al.. 2021; Rossiello et al., 2022; Lin et al., 2020) and knowledge base question answering (Rossiello et al., 2021; Kapanipathi al., 2020; Neelam al., 2022). Prior to the surge of generative models, re- lation linking was addressed either by graph traver- sal based (Pan et al., 2019; Dubey et al., 2018) or by linguistic-features based methodologies (Sakor et al., 2019a,b; Lin et al., 2020). Several learning based approaches to relation linking have been pro- posed (Mihinduukulasooriyaa et al., 2020; Yu et al., 2017; Bornea et al., 2021). Most recent approaches to relation linking have focused on generative mod- els (Rossiello et al., 2021). These approaches are data hungry and non-interpretable. In contrast, our work is a self-supervised rule learning based ap- proach for relation linking. The learnt rules are
For self-supervision popularly used self-supervised training strategy for large language models. However, in contrast to the use of NL text as in the large language models (LLMs), masked training in our approach operates on logical statements the sentences. In this work, we use triples² extracted from semantically parsed output of the sentences as the logical statements. An example sentence in its NL form and its corre- sponding triple form (logical statement) are shown in Figure 1. The masking procedure when applied on a triple may end up masking any of the three el- ements, i.e., subject, object, or predicate. Figure shows an illustration of the predicate component
masked training,
TTripe format: predicate(source entity, destination entity), where source entity and destination entity are also alternatively referred to as subject and object, respectively.

---

human readable and they can be learnt for different knowledge bases as context for different domains.
3 Our Approach
Our approach to relation linking is a rule-learning framework where rules learned in self-supervised fashion used to map from 'NL text segments' to the 'elements of structured knowledge'. For self-supervised rule learning, we use masked train- ing. A distinctive aspect of the masked training our approach that is applied over the logical form of the input text. When logical form is not readily available, we first convert NL text into its logical form using a semantic parser, before ap- plying masked training. In this section, first we briefly describe the conversion of imput text to logi- cal statements, and then describe the proposed rule model and its self-supervised training.
Rule learning, specifically in terms of detect- ing relations has also been popular for knowledge base completion (KBC) task. KBC problems are addressed in two different ways in the literature. Rule based KBC learns explainable rules that can be used to predict the missing links in the knowl- edge base (Sen et al., 1; Qu et al., 2020; Yang et al., 201 7a; Sadeghian et al., 2019; Purgal et al., 2022; Rocktäschel and Riedel, 2017). These rules are learned for predicates in the knowledge base in terms of other predicates and paths in the knowl- edge base. Our work uses the rule model and some the ideas from the rule based KBC to build self supervised explainable rules for linking text ele- ments to knowledge base relations. There have also been several learning based methods that use vector embeddings of the entities and relationships and then use them predict any missing relation- ships (Nickel al., 2015; Yang et al., 2014; Wang et al., 2014; Lin et al., 2015; Trouillon al., 2016; Sun et al., 2018; 8: Zhang et al., 2019); they have the same drawbacks as neural relation linking ap- proaches. There are many recent works that use semi-ssuperviieddnssuprvvissss learning for rules and knowledge induction (Pryzant et al., 2022; Bha- gavatula et al., 2023; Lang and Poon, 2021; Zhu and Li).
3.1 Logical Statements
In this work, we triples extracted from sen- tences their logical form*. Note that logical forms are expressed as predicate(source entity, des- tination entity), as illustrated Figure 1. To extract such logical forms from NL text, we use the seman- tic parsing approach described in (Neelam et al., 2022), which performs an Abstract Mcaning Rep- resentation (AMR) (Banarescu et al., 2013) based X-expression extraction. For simple sentences, X- expressions can be converted directly into triples². Figure 1 shows an illustration of NL text and its corresponding logical form. Different elements of the triple represent distinct segments of the NL text. Since we use an AMR based approach to derive A- expressions, predicate components of the triple are typically a propbank predicate. Relation linking is expected to map this predicate component onto the corresponding knowledge-bert base elements.
Transformers (Vaswani al., 2017) have given rise to wide range of masked pre-training mod- els that try to learn latent representation of words. Masked language modeling has gained popular- ity in building large pre-trained models that learn word representations that can easily be adapted to a variety of tasks. Masked language training has re- sulted in building various model architectures like encoder only models which learn vector repre- sentation of tokens using masked training (Peters al., 2018; Devlin et al., 2019; Liu al., 2019; Clark et al., 2020). Several Encode-decoder LLMs are built using similar training strategy (Song et al., 2019; Lewis et al., 2019; Soltan et al., 2022; Raffel et al., 2019). Recently decoder only models built using similar masked language training are gaining popularity (Radford and Narasimhan, 2018; Rad- ford et al., 2019; Brown et al., 2020: Chowdhery al., 2022: Zhang al., 2022; Scao al., 2022), In this paper we take inspiration from masked lan- guage training on text and use the ideas to perform similar masking on logic statements to learn rules.
3.2 Rule Model
Towards the goal of relation linking, our rule model is formulated as a function that maps a weighted mixture of 'knowledge base predicates' onto 'predicate component of the NI text'. This enables interpretable prediction of the set of knowledge base predicates that the text predicate should get linked to. Figure 2 gives an illustration of a simple rule.
Let us assume triple pred d) extracted from NL text, where is the source entity, d is the des- tination entity and pred, is the text predicate. The
"We use triples and logical form interchangeably. *We also emplay heuristics to handle the AMR and X- expression computation failures and errors.

---

Q202765 IKate Winslet)
Q38111
pred(()
(Leonardo DiCaprio)
wdt:P161
wd1:
f()
(cast member)
Q44578
(cast member)
Titanic movie)
Figure 3: Illustration of one-hop path between enti- ties Kate Winslet and Leonardo DiCaprio via entity Titanic movie in Wikidata (through reverse relation edge cast-member connecting Kate Winsler to Titanic movie and forward relation edge cast-member con- necting Titanic movie to Leonardo DiCaprio). This also be viewed as an extended context extracted from knowledge base through the unmasked elements of the masked logical statement in Figure
w
W2
w
W
path,(} (.) path2 (.)
Cath
pathy (.)
Figure Neural model implementing each rule of the rule model defined in Equation (1).
which is denoted as cast member in the corre- sponding path expression given in Figure 2. All the knowledge base paths of various lengths, i.., from O-hop paths until k-hop paths put together, consti- tute the overall of paths considered in our rule learning framework. Let us assume there are N such paths total. Given that the knowledge base paths pathy are defined in terms of the knowledge base predicates, our rule model in (1) effectively defines mapping between the 'NL text predicates' and the 'knowledge base predicates', During learn- ing, want to estimate weights w so that rule model in (1) can be used for relation linking.
goal is to ground pred; its corresponding knowl- edge base predicates using relation linking model. Our proposed rule model to achieve this is defined as:
N
pred((s,)) =1 f Σ wy path;(s,d)
(1)
for i (1....,M) where (() denotes a function such as Sigmoid and path, denotes a path in the knowledge base that passes through one or more knowledge base predicate edges. (sth,(s, d) can take values 0 1 depending upon presence or absence of path path; connecting source entity s and destination entity d. Depending upon the length of the path, i.e., number predicate edges and the intermediate entities, paths could be cat- egorized as k-hop, where & denotes the count of intermediate entities in the path. An example of 1- hhhop path between entities in Wikidata is shown in Figure 3. If there are L unique knowledge base predicates, i.e., (pr), < L, the of 0, 1 and 2- 2-hop paths are defined as follows:
A neural implementation of each rule (1) is given in Figure 4. Note that there one such net- work each for every rule in (1), and all these net- works are trained together during learning. Note that the parameters of our rule model, denoted by W, is matrix of learnable weights:
W W isisM, 15iEN
where each row corresponds to the set of path weights of specific rule. In the next section, we describe masked training used to estimate these weights in self-supervised manner.
O-hop paths = (pi(s, d)), IS1SL
1-hop paths = (m(s))) x): :): (m(
3.3 Self-Supervised Rule Learning
1 1 L, 1 < L
Masked training in large language model training involves masking certain parts of the input text and having the model predict those masked parts by treating the rest of the (unmasked) text as its context. The model parameters are adjusted during training effectively model the context so can act proxy for the masked part. our approach masked training is performed on the logical form of the input text as detailed below.
h-hop paths (pr *) pm,(, (x,+)) (m( 1 1 l < L, 12 m L, 1 L
where denote any possible entity. Note that paths always originate from source entity and end at destination entity. Also note that, the constituent knowledge base predicates in path could also be inverse (reverse), denoted by superscript -1, For ex- ample, path in Figure 3 contains an inverse relation,

---

3.3.1 Masked Training on Logical Forms
Logical statements are composed of distinct ele- ments playing distinct semantic roles. For example, triples are composed of predicate, a source entity (also called subject) and a destination entity (also called object). The masking procedure could mask any of these elements:
patha
?? d)
pathg d
pathg
Figure 5: Illustration of knowledge base paths to be used as context when predicate in the triple masked.
1. Predicate masking predicate component of the triple is masked
2. Source entity masking source entity compo- nent of the triple is masked
above, knowledge base paths associated with the unmasked elements of the triple are used as the context for the prediction. However, the set of such paths differ based on which element masked and which elements are unmasked and could be linked to the knowledge base. Training loss functions are defined to improve prediction accuracy over the training iterations, as deseribed below:
3. Destination entity masking- destination entity element of the triple masked
Figure 1 gives an illustration of predicate mask- ing. Similar to the large language model training, masked training for rule learning also aims to ad- just the parameters of the rule model so that it can accurately predict the masked elements from the unmasked context.
a) Predicate Masking: As illustrated in Figure 5, when the predicate is masked, unmasked source and destination entities are linked to the correspond- ing elements in the knowledge base through entity linking. Then the set of all paths that connect these linked entities are computed and used context. Let j E denote the set of indices of knowledge base paths that connect the linked source and desti- nation entities. These paths when applied to all the rules in (1), corresponding to all the text predicates p.)d,(), 1 S < M, would result in scores for each text predicate as below:
However, the distinctive aspect of our approach is in the context computation. For the context, in addition the unmasked elements of the logical statement, we also use paths within the knowledge base that are associated to those unmasked ele- ments. A set of such associated paths are obtained by grounding the unmasked elements of the logi- cal statement to the knowledge base. For example, given Wikidata the knowledge base, we first entity linking (Wu al., 2019) to link the un- masked source and destination entities of the triple to corresponding Wikidata entities, as illustrated in Figure! Then the Wikidata paths originating from unmasked source entity and ending at unmasked destination entity are computed and used as the con- text. Figure 3 gives an illustration of Wikidata path, through entity node corresponding to Titanic movie, for the unmasked entities of Figure 1. Note that such use of knowledge base paths as the con- text help establish the mapping between the NL text predicates and the knowledge base predicates, which indeed is the goal of our relation linking model. Next we describe rule model parameter es- timation procedure for different masking scenarios.
Σ ny (st jEM
=
isis M.
For accurate prediction of the masked predicate by the model, the score should be highest (1.0) for the text predicate being masked and should be the lowest (0.0) for all others. Accordingly, let t, s < M denote the target scores, where the score corresponding to the index of the masked predicate is 1.0 and the rest of the scores are 0.0. We compare the estimated scores against these target scores, to compute the training loss that could be used further to update the model weights through stochastic gradient descent (SGD). In actual, we use a cross- entropy training loss computed as below:
3.3.2 Rule Model Estimation
Learning procedures for the 3 masking scenar- ios differ mainly in the way training loss is com- puted. all masking scenarios we use rule mod- els to predict the masked element. As discussed
loss
tg log(si)
1=1

---

is masked, unmasked source entity linked to the knowledge base and the of all paths originat- ing from the linked source entity are computed as context, resulting in the associated candidate desti- nation entities. Then scores for all candidate desti- nation entities are computed in similar fashion to further compute the training loss.
path,
$g
path2
pred,
d)
paths
pathzı
S5
pathi
path
S3
3.4 Inference
Figure 6: Illustration of knowledge base paths to be used as context when source entity in the triple is masked.
Note that, once trained, the right hand side of the rule model in Equation (1) directly gives human readable information of the relation linking. high- lighting the interretabiiityleepaaaabbbii aspect of our model. Each rule gives information of the knowledge base paths (that turn are composed of the knowledge base predicates) that the text pred- icate should get linked to, with path weights de- noting the linking confidence. During inference, for example while using our rule model to perform QA, the information being asked in the NL question would correspond to the missing element in the cor- responding logical form. Hence the goal is esti- mate candidates for that missing element and score them using our rule model. The scoring procedure is similar to that for the masked elements during masked training as described in Section 3.3.2. Top- K scoring candidates are chosen as the answers estimates of the rule model.
b) Source Entity Masking: As illustrated in Fig- ure 6 when the source entity masked, unmasked destination entity is linked to the corresponding ele- ment in the knowledge base, through entity linking. Then the set of all paths that end at the linked des- tination entity are computed for use as the context. Note that all these paths together have a set of asso- ciated origination entities that could potentially be considered the prediction for the masked source entity. Let S {si, 82; ...., 8p) denote the set of all such candidate source entities. Let Eg correspond to the set of indices of all paths that originate from the candidate source entity Se and end at the un- masked linked destination entity. These paths when applied to a specific rule in (1) that corresponds to the unmasked text predicate pred,, would give score for 8, below:
3.5 Scaling
Σ wa (se,), ISESE JEE,
Se
The number of paths N is exponential in the num- ber of predicates I. that exist in the knowledge base. As a result, solving problem (1) becomes pro- hibitive with large N and hence we explore meth- ods for reducing the search space of paths. In other words, we learn a set of paths P with [P| << N that contains the optimal paths, allowing us to re- place the summation in problem in Equation (1) to a summation over P, and thereby reducing the computation required to find the solution.
A list of such scores computed for each can- didate source entity in S. Note that among these scores the one corresponding to candidate entity that same as the masked source entity should be highest (1.0) and the score for the rest should be the lowest (0.0). Accordingly, target scores are te: 1 1 e < E, where the score for the index of the masked entity is set to 1.0 and the rest are set to 0.0. We then compute training loss by compar- ing the estimated scores against the target scores. A cross-entropy training loss computed as below is used to update the rule model weights through stochastic gradient descent (SGD):
We make use of a Chain of Mixtures (CoM) model, also known as Neural Logic Programming (Yang al., 2017b) 7b) edge-based rules (Sen al., 2022), to learn P. In CoM, a rule is represented by a conjunction mixtures of predicates. Specifi- cally, rather than define a weight per path, for cach of M predicates we define a weight per predicate per hop to get rules of the form:
loss ΣE to log(se) eES
predCoM (s, d) =g(f /EM w w..o
c) Destination Entity Masking: Destination en- tity masking is similar to that of the source entity masking, except that when the destination entity
(2)
f (IM, 1 (11,

---

1 (E (122s((2: (12.
of length up to 10, while the train set has clauses of length up to 4. ii) Model robustness which tests how well model performs in the presence of noisy facts. It consists of three kinds of noise: a) Supporting facts: It adds alternative equivalent paths between source and target entities. b) Irrelevant facts: It adds path which originates on the path between the source and target entity but ends at an entity not in that path. c) Disconnected facts: It adds a path which is completely disconnected from the path between the source and target entity.
for i E {l,..., where now wI. is weight for rule (i.e., for the ht predicate) for predicate j on the hop k of the rule being learned, and g(. is function approximating conjunction such as a product. Equation (2) is example 2-hop rule where ry and r2 can take the values of any entities in the knowledge base. The number of weights learned in a CoM rule is M (K +1) where K is the number of hops in the rule, which greatly reduces the number of weights learned in problem (1).
We considered graph stories with only clause length 2 for training rule models in a self- supervised fashion. This results in multiple one- hop rules along with their confidence scores. These rules can be used to handle stories of clause length 2 at the time of inference. For stories with clause length greater than 2. we combine relations within the path from source target entity using Algo- rithm 1 to compute the path scores. A n-length path R1,... R,, from source entity to target entity can be reduced by invoking infer(1, n). It utilises two sub- modules predict(R,, R₂) and score(R,, R₃) which returns prediction and confidence scores for merging R, and R, using one-hop rule model.
While CoM requires learning much fewer parameters than our rule model (1), it is less interpretable because it is not known which paths are actually relevant to the rule. CoM rules suggest that any path that can be obtained from predicates with positive weights in the hops is possible. For example, suppose there are 8 predicates in the knowledge base and the set of strictly positive weights in a rule of the form (2)
is (wo.1 W0,3, " wf.1- "f.2 2222; 42 =2.s)- Then such chain rule only allows rules of the form (1) A (2() A e)) where j E (1,3, 1). & E (1,2). 1 E 82. with predicates, there are a total of 8 x 8 x = 512 possible paths, but the CoM reduces this to 3 x 22x x 3- = 18 possible paths. Given these possible paths, a more interpretable rule of the form Equation (1) can be learned more efficiently since the summation can be taken over 18 paths instead of N 512 paths.
Algorithm 1: Inference procedure using one-hop rule model
Output: :Ry: Prediction for path R. ...R,, S.y Prediction score for path R, ...,,
Function (i.
if ==== then
return R.,
else
Rsj <- null, Sij 0 for k < i - to by do
4 Experimental Setup
R Su- +- in fnr l 1)
4.1 Datasets
Rej. Sez + (k
Sy < Sie_ &Sk, "SS, *score(Rxx-1 Ray)
We evaluate our approach on two datasets as de- scribed below that involve utilization of two distinet types of knowledge sources.
R predict(Rse Rkz)
if S S.y then
Rτ + Riy, Sig + 8/3
end return Rıj, S.,
a) CLUTRR (Sinha et al., 2019) is a benchmark dataset for evaluating the reasoning capabilities of a Natural Language Understanding (NLU) system, where the goal is to infer kinship relations between source and target entities based on short stories. It contains both graph and textual representations of these storics. Each story consists of clauses (chains of relations) of length ranging from 2 to 10. The benchmark tests the model's ability for: i) Systematic Generalization which tests how well the model generalizes beyond the combination of rules seen during training. The test set consists of clauses
end
b) SWQ-WD (Diefenbach et al., 2017) is a Ques- tion Answering (QA) dataset with simple questions built to evaluate the QA accuracy of KBQA sys- tems that use Wikidata as the knowledge source. This dataset fits our relation linking evaluation goal because it consists of simple questions with single text predicates, hence it is possible to perform a focused evaluation of the relation linking task, re- ducing the impact of uncertainties of other compo- nents of KBQA systems such as semantic parsing

---

CLUTRR:
son: (son and brother, 0.95), (daughter and brother, 0.95), (wife and son, 0.96), (husband and son, 0.96)
granddaughter: (son and daughter, 0.97), (daughter and daughter, 0.96),
(granddaughter and sister, 0.92), (grandson and sister, 0.96),
(wife and granddaughter, 0.94), (husband and granddaughter, 0.90) }
niece:
(brother and daughter, 0.98), (sister and daughter, 0.91) }
SWQ-WD:
book-write: { (author, 0.88), (notable work 0.87)
film-language: (... (original language of film or TV show, 0.97), (language of work or name, 0.83),
(cast member and writing language, 0.97), (screenwriter and writing language, 0.95),
(film editor and languages spoken, written or signed, 0.87),
(director and languages spoken, written or signed, 0.83),
(follows and original language of film TV show, 0.96),
(based and language of work or name, 0.70),
Figure 7: Sample learned rules for both CLUTRR and SWQ- WD datasets. Note that each rule is represented by a set of tuples, with each tuple containing a knowledge base path and its weight value. These corresponds to the knowledge base paths and the corresponding weights in rule model (I). Note that inverse knowledge base predicate (i.e., reverse knowledge base relations) in the rules are denoted by superscript as described in Section 3.2.
son: (son, 0.97), (husband, 2.49), (daughter, 0.70), (wife, 2.30)) and {(son, 0.81), (brother, 2.86) } * granddaughter: (son, 1.26), (grandson, 2.50), (daughter, 1.20), (granddaughter, 4.22))
and ((daughter, 1.22), (granddaughter, 5.09) }
. niece: { (brother, 1.69), (sister, 1.43)] and ((daughter, 1.72) }
Figure 8: Sample learned rules for CLUTRR using Chain of Mixtures.
and entity linking. SWQ-WD has 13894 train set questions and 5622 test set questions. We use train set self-learn rules and report relation linking accuracy on the set. Answer prediction using our rule model described in Section 3.4.
4.3 Code Release
The code and the experimental setup used train and evaluate our self-supervised rule learn- ing based approach for relation linking avail- able at https//gghhuubcooo//////s/sssss/ rule-Icarning.
4.2 Baselines and Metries
For both the datasets, we compared our method against the best available baseline as per our knowledge. For CLUTRR, we compared against GAT (Veličković et al., 2017), a neural graph rep- resentation framework that learns representations through self-attention over the neighbourhood. For SWQ, we compared against SYGMA (Neelam et al., 2022), modular and generalizable pipeline based approach consisting of KB- KB-specific and KB- agnostic modules for KBQA.
Results
Figure 7 shows (in upper part) sam- ple 1-hop rules learned from CLUTRR dataset. Note that these rules are of form as in Equation (1), For CLUTRR, there were 22 predicates and so each 1-hop rule was learned over N 22 x 22 484 possible paths. Figure shows corresponding CoM rules for the CLUTRR dataset. For example, the CoM rule for son implies possible 1-hop rules that start with either son, husband, daughter, or wife and end with either son or brother, offering 4 x 2 = 8 possible rules for son. Rules of the form (1) could thus be rather learned over only these possible rules and capture the same interpretable rules for son seen in the CLUTRR part of Figure 7. Overall, rules for the 22 different relations search over 484 paths each (10648 total possible rules); CoM reduced that search to 1 118 to-
a) CLUTRR:
For CLUTRR experiments, where relation linking performance is directly evaluated, we use a simple accuracy metric of accuracy success_cce poccesss_cout__eee wwl.c/otttttaal.t /totttttaalaaa /oota_ccouut.. count. For SWQ experi- ments, where the relation linking performance is not directly evaluated but through its influence on the question answering (QA) accuracy, we use the metric of Recall which a fraction of correct an- swers that the system is able to estimate.

---

tal possible rules with perfect coverage (i.e., CoM results include all optimal rules as subset) for all but 2 relations.
Training
Testing
GAT
Our
approach
Clean
Clean
Supporting
0.24
Figure 9 compares the Systematic general- ization performance of our approach against GAT (Veličković et al., 2017) on CLUTRR dataset with test set clause length ranging from 2 to 10. The results suggest that the proposed approach trained with stories of clause length generalizes better in comparison GAT trained with stories of clause length k = 2, 3 and stories of clause length 3, Table compares the model robustness of our approach against GAT on CLUTRR dataset for dif- fering noise scenarios during training and inference. The results suggest that the proposed approach out- performs GAT for most scenarios, achieving average absolute accuracv improvement of 17%.
Inrlevant
0.51
Disconnected
Supporting
Clean
0.92
0.89
Supporting
0.98
0.97
Irrelevant
0.5
0.94
Disconnected
0.92
0.91
Irrelevant
Clean
0.92
0.94
Supporting
0.77
0.94
Irrelevant
0.93
0.96
Disconnected
0.85
0.88
Disconnected
Clean
0.75
0.91
Supporting
0.78
0.93
Irrelevant
0.56
0.91
Disconnected
0.96
0.93
Average
0.77
0.94
Table I: Model robustness on CLUTRR Accuracies for various training and inference noise conditions.
Method
Recall
SYGMA
55.0
Our approach
53.7
SYGMA (with GT entities)
68.0
Our approach (with GT entities)
70.1
Table 2: QA performance on SWQ.
Out approach))=3)
GAl((e22330) 2,14
used, our approach performs marginally better than SYGMA (as shown in the last two rows). Note that when ground truth entities are used, QA per- formance is more reflective of the relation linking performance, thus asserting the effectiveness of our relation linking model.
Length
Figure 9: Systematic generalization CLUTRR.
b) SWQ-WD: Figure shows (in lower part) sam- ple 0/1-hop rules learned from SWQ dataset, of form in Equation (1). Interestingly, for text pred- icate film-language it is able to learn rules that associate with the language spoken/writter by the director or screen writer or cast members of the film. Table 2 compares QA accuracy of our ap- proach against SYGMA (Neelam et al., 2022). Our approach achieves comparable QA performance (as given in the first two lines), in spite of SYGMA using supervised training for its relation linking module whereas our approach self-supervised approach (not using any labelled data for training). Note that rule learning is prone to inaccurate en- tity linking Thus, when ground truth entities are
Conclusions
In this paper, we presented a novel masked train- ing based self rule learning approach for relation linking. The proposed approach takes inspiration from the masked training of large language models and rule learning in knowledge base completion to create rule learning model that results in a relation linking model that is interpretable and gen- eralizable to any knowledge base and domain. Our evaluation on two datasets CLUTRR (rule learning for kinship relations) and SWQ (relation linking for Wikidata) shows that our model is able to pro- duce high quality rules for relation linking that generalize across domains. Our next steps scale to learn from large amount of data, extend the framework beyond triples. We see lot of potential for our rule learning framework beyond relation linking, like general knowledge acquisition.
'Note that inaccuracies both in AMR parsing and entity linking could lead to erroneous logical statements. However, SWQ sentences are simple and hence their AMR parsing is straight forward, with SMATCH score of 83.0% (Neelam et al., 2022). Thus, incorrect logical statements for SWQ are largely due to entity linking emrrr.. In case of CLUTRR, GAT haseline used the readily available ground truth logical forms of the sentences that we also used for fair comparison.

---

7 Limitations
Jacob Devlin, Ming-Wor Beng-Weii Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre- training of deep bidirectional transformers for language understand- ing. ArXiv, abs/ Abss/880..0485.
Proposed approach is an initial attempt at tak- ing masked training to logic statements and using knowledge graph context. Current approach described is only taking single triple to learn re- lationship linking problem for KBQA, but overall approach has potential to extend to larger logic statements. Extending the work to handle logic statements with more than one triple is future direction that we are looking into. In the current approach we assumed semantic parses to be per- feet and available, but in reality they can be noisy resulting in noisy rule model. Handling noise com- ing from the parsing is another main extension we would like to look into as part of our future work.
Dennis Diefenbach, Thomas Pellissier Tanon, Ka- mal Deep Singh, and Pierre Maret. 2017. Question answering benchmarks for wikidata. In International Workshop on the Semantic Web.
Mohnish Dubey. Debayan Banerjee, Debanjan Chaud- huri, and Jens Lehmann. 2018. Earl: Joint entity and relation linking for question answering over knowl- edge graphs. ArXiv, abs//801 103825....
Claire Gardent, Anastasia Shimorina, Shashi Narayan. and Laura serez-Beltrachini. 2017. Creating training corpora for nlg micro-planners. In Annual Meeting of the Association for Computational Linguistics.
Pavan Kapanipathi, 1. Abdelaziz. Srinivas Ravishankar, Salim Roukos, Alexander G. Gray. Ramón Fernández Astudillo, Maria Chang, Cristina Comello, Saswati Dana, Achille Fokoue, Dinesh Garg, A. Gliozzo, Sairam Gurajada, Hima P. Karanam, Naweed Khan, Dinesh Khandelwal, Young suk Lee, Yunyao Li, Fran- cois P. S. Luus. Ndivhuwo Makondo, Nandana Mi- hindukulasooriya, Tahira Naseem, Sumit Neelam. Lucian Popa, Revanth Reddy Gangi Reddy, Ryan Riegel, Gactano Rossiello, Udit Sharma, G. P. Shri- vatsa Bhargav, and Mo Yu. 2020. Leveraging abstract meaning representation for knowledge hase question answering. In Findings.
References
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013, Abstract meaning representation for sembanking. In LAW
Chandra Bhagavatula. Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Lianhui Qin, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, and Yejin Choi. 2023. 12d2: Inductive knowledge distil- lation with neurologic and self-imitation.
Hunter Lang and Hoifung Poon. 2021. Self-supervised self-supervision by combining deep learning and probabilistic logic. Proceedings of the AAAI Con- ference on Artificial Intelligence, 35(6):4978-499)
Mihaela A. Bornea, Ramón Fernández Astudillo, Tahira Naseem, Nandana Mihindukulasooorya,, I. Abdelaziz, Pavan Kapanipathi, Radu Florian, and Salim Roukos. 2021. Learning to transpile amr into spargl. ArXiv, abs/21 12.07877.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019, Bart: Denoising eqqeenc-to-sequenee pre-training for nat- ural language generation, translation, and compre- hension. In Anmual Meeting of the Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, and et.al. 2020. Language models are few-shot learners. ArXiv, 141/2505.14165.
Xueling Lin, Haoyang Li, Hao Xin, Zijian Li, and Lei Chen. 2020. Kbpearl: A knowledge base population system supported by joint entity and relation linking. Proc. VLDB Endow., 13:1035-1044
Yubo Chen, Yunqi Zhang, Changran Hu, and Yongfeng Huang. 2021. Jointly extracting explicit and implicit relational triples with reasoning pattern enhanced binary pointer network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5694-5705 Online. Association for Computational Linguistics.
Yankai Lin. Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation em- beddings for knowledge graph completion. In AAAI Conference on Artificial Intelligence.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, and et.al. 2022. Palm: Scaling language modeling with pathways. ArXiv, abs/2204.00311.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, abs/1907.11692.
Kevin Clark, Minh-Thang Luong, Quoe V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. ArXiv, abs/2003. 10555.
Nandana Mihindukulasooriyy. Gactano Rossiello, Pa- van Kapanipathi, I. Abdelaziz, Srinivas Ravishankar, Mo Yu, A. Gliozzo, Salim Roukos, and Alexander G.

---

Gray. 2020. Leveraging semantic parsing for rela- tion linking over knowledge bases. In International Workshop the Semantic Web.
Gactano Rossiello, Nandana Milinnukulasooriiya, I. Ab- delaziz, Mihaela A. Bornea, A. Gliozzo, Tahira Naseem, and Pavan Kapanipathi. 2021. Generative relation linking for question answering over knowl- edge bases. ArXiv, abs/2108.07337.
Sumit Neelam, Udit Sharma, Hima P. Karanam, Sha- jith Ikbal, Pavan Kapanipathi, I. Abdelaziz. Nandana Mihindukulasooriya, Young-Suk Lee, Santosh K. Sri- vastava, Cezar Pendus, Saswati Dana, Dinesh Garg, Achille Fokoue. G. P. Shrivatsa Bhargav, Dinesh Khandelwal, Srinivas Ravishankar, Sairam Gurajada, Maria Chang. Rosario A. Uceda-Sosa, Salim Roukos, Alexander G. Gray, Guilherme Lima, Ryan Riegel, Francois P. S. Luus, and L. V. Subramaniam. 2022. Sygma: A system for generalizable and modular question answering over knowledge bases. In Con- ference on Empirical Methods in Natural Language Processigg
Ali Reza Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. 2019. Drum: End-to-end differentiable rule mining on knowledge graphs. ArXiv, abs/191 htss/1911.000 .00055.
Ahmad Sakor, Isaiah Onando Mulang, Kuldeep Singh, Saeedeh Shekarpour, Maria-Estae Vidal, Jens Lehmann, and S. Auer. 2019a. Old is gold: Lin- guistic driven approach for entity and relation linking of short text. In North American Chapter of the As- sociation for Computational Linguistics.
Maximilian Nickel, Lorenzo Rosasco, and Tomaso A. Poggio. 2015. Holographic embeddings of knowl- edge graphs. ArXiv, 0tls/1510.0499..
Ahmad Sakor, Kuldeep Singh, Anery Patel, and Maria- Esther Vidal. 2019b. Falcon 2.0: An entity and rela- tion linking tool over wikidata. Proceedings of the 29th ACM International Conference on Information & Knowledge Management.
7. Pan, Mei Zhang, Kuldeep Singh, F. V. Harmelen, Jinguang Gu, and Zhi Zhang. 2019. Entity enabled relation linking. In International Workshop on the Semantic Web.
Teven Le Scao, Angela Fan, Christopher Akiki, Elizabeth-Jane Pavlick, Suzana Ili'e, Daniel Hess- low, and ct.al. 2022. Bloom: A 176b-parameter open-access multilingual language model. ArXiv, abs/221 1.05100.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zeulemoyer. 2018. Deep contextualized word repre- sentations. In North American Chapter of the Associ- ation for Computational Linguistics.
P. Sen, B. W. R. de Carvalho, I. Abdelaziz, P. Ka- panipathi, S. Roukos, and A. Gray. 2022. Logical neural networks for knowledge hase completion with embeddings rules. EMNLP.
Reid Pryzant, Ziyi Yang. Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Automatic rule induction for interpretable semi-supervised learning.
Prithviraj Sen, Breno W. Carvalho, Ryan Riegel, and Alexander G. Gray. 2021. Neuro-symbolic induc- tive logic programming with logical neural networks. ArXiv, abs/2112.03324.
Stanislaw J. Purgal, David M. Cerna, and C. Kaliszyk. 2022. Differentiable inductive logic programming in high-dimensional space. ArXiv, abs/2208.0666 066522..
Meng Qu Junkun Chen, Louis-Pascal Xhonneux, Yoshua Bengio, and Jian Tang. 2020. Rnnlogic: Learning logic rules for reasoning on knowledge graphs. ArXiv, abs/2010.04029.
Koustuv Sinha, Shagun Sodhani, Jin Dong. Joelle Pineau, and William Hamilton. 2019. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint 198.07790.
Alec Radford and Karthik Narasimhan. 2018. Im- proving language understanding by generative pre- training.
Saleh Soltan, Shankar Ananthakrishnan, Jack G. M. FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith S. Peris, and et.al. 2022. Alexatm 20b: Few- shot learning using large-scale multilingual seq2seq model. ArXiv, abs/2208. 014488.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre- training for language generation. In International Conference on Machine Learning.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Kather- ine Lec. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits transfer learning with a unified text-to-text transformer. ArXiv, abs/ 10. 10683.
Zhiqing Sun, Zhihong Deng. Yun Nie, and Jian Tang. 2018. Rotate: Knowledge graph embedding by relational rotation in complex space. ArXiv, abs/1902.10197.
Tim Rocktäschel and Sebastian Riedel. 2017, End-to- end differentiable proving. ArXiv, abs/1705. 11040.
Gaetano Rossiello, Faisal Chowdhury, Nandana Mi- hindukulasooriya, Owen Cornee, and Alfio Gliozzo. 2022. Knowgl: Knowledge generation and linking from text. arXiv preprint arXiv: 13952.
Théo Trouillon, Johannes Welbl, Sehastian Riedel, Eric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Interna- tional Conference on Machine Learning.

---

Ashish Vaswani, Noam M. Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10003.
Zhen Wang. Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014, Knowledge graph embedding by trans- lating hyperplanes. In AAAI Conference Artifi- cial Intelligence.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebas- tian Riedel, and Luke Zettlemoyer. 2019. Scalable zero-shot entity linking with dense entity retrieval. Conference on Empirical Methods Natural Lan- guage Processing.
Bishan Yang, Wen tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference knowledge bases. CoRR, abs/1412..577
Fan Yang. Zhilin Yang, and William W. Cohen. 2017a. Differentiable learning of logical rules for knowledge base reasoning. NIPS.
Fan Yang. Zhilin Yang, and William W. Cohen. 2017b. Differentiable learning of logical rules for knowledge base reasoning. In NeurIPS.
Mo Yu, Wenpeng Yin. Kazi Saidul Hasan, Cicero Nogueira dos Santos, Bing Xiang. and Bowen Zhou. 2017. Improved neural relation detection for knowledge base question answering. In Annual Meeting of the Association for Computational Linguistics.
Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019. Quaternion knowledge graph embeddings. In Neural Information Processing Systems.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, and et.al. 2022. Opt: Open pre-trained transformer language models. ArXiv, abs/2205.01068.
Ronghang Zhu and Sheng Li. superviieri based Semantic Aligmment for Unsupervised Domain Adap- tation, pages 1-9.


Reviews:
[{"review_id": "o96NGhUxFF", "reviewer": "Reviewer_d5MH", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments. We will improve the presentation issues raised by the reviewer in the final version.\n\n**Response to 'Reasons To Reject':**\n\n\n**1. The baselines GAT and Sygma are cited but not suffiiciently described in the paper.**\n\nWe have given references to both the works and will include more details in the revision.\n\n**2. Abstract Meaning Representation is used to extract logical forms from natural statements. How was the the quality of the extracted forms? How much does it affect the final performance?**\n\nAMR is used for parsing the SWQ dataset that has simple sentences with subject, object, predicate kind of structure. For these sentences, getting the logical form using AMR was straight forward and the SMATCH score for a small test set was approximately 83.0. For the CLUTRR dataset, the ground truth logical forms are readily available with the data and we used those for rule learning similar to the GAT baseline which used the same logical forms.\n\n**3. For both the datsets, only one baseline is presented.**\n\nWe took the best baseline available as per our knowledge and compared our method with those baselines.\n\n**4. Sufficient details for reproducibility are not provided. Will the code be made public?**\n\nWe will open source the code and add a git link to the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) First paper to use mask training on logical forms of text instead of natural text. \n2) Proposes a mechanism for self supervised learning of rules for realtion linking, which is useful in the absense of labeled data.\n3) The learned rules are human interpretable as shown in the paper. \n4) Perfromance on QA is comparable to a spervised learning baseline.", "reasons_to_reject": "1) The baselines GAT and Sygma are cited but not suffiiciently described in the paper. \n2) Abstract Meaning Representation is used to extract logial forms from natural statements. How was the the quality of the extracted forms?\nHow much does it affect the final perfromance? \n3) For both the datsets, only one baseline is presented. \n4) Sufficient details for reproducibility are not provided. Will the code be made public?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a masked training strategy (similar to MLM) over logical forms of sentence to learn rules for relation linking in knowledge bases. The logical forms are of type \"predicate(source entity,destination entity)\", and one out of predicate/source/destination is masked. \nThe rule model for relation linking is formulated as a function that maps a weighted mixture of ‘knowledge base predicates’ onto the ‘predicate components of the text'. The rule model is trained in a self supervised manner using the masked training, and the learned rules are human interpretable. \nWhen using this model for inference during QA, depending on the information being asked in the question, the missing element of the logical form of the question is predicted, similar to masked element prediction. \nThe effectiveness of this method is shown on  CLUTRR dataset (missing kinship relation) and for relation linking in knowledge base question answering system (SWQ-WD).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "abstract - \"Results demonstrate the effectiveness of our approach\" - this line should give more details."}, "extra_text": {}}, {"review_id": "9vVTIcJoNr", "reviewer": "Reviewer_CrYn", "summary": "", "questions": "A: How accurate is the system (semantic parser) that generates logical forms from the sentences? How does its performance affect the presented methodology?\nB: Section 3.3, is `path_j(sad)` a binary variable (line 224) or the length of the path between the entities?\nC: In Section 3.2.2 b), if `Ee` is the set of all paths that end in d, it appears that the masked source is the only remaining source candidate, potentially making the task trivial. Can you clarify this aspect and explain how the approach addresses such cases?\nD: Can the model predict cases where \"no relationship\" exists? If not, how are these cases handled in the relation linking process?\nE: Could you elaborate on how the context is used during training time?\n", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to 'Reasons To Reject':**\n\nWe thank the reviewer for the detailed comments. We will include more examples to improve overall readability of the paper. Also details on the semantic parser and how logical forms are derived will be added in the final version. We will add clear definitions for paths/rules/relation linking task definitions in the paper. Regarding reproducibility, we will open source the code and add a git link to the final version.\n\n**path**:  path_i(s,d) is nothing but a set of edges between s and d of which by traversing one can reach d from s.  There can be multiple paths that connect two entities in the graph. When we refer to a path_i, we are referring to one of those paths. \n\n**rule** : A rule is nothing but an argmax over weighted paths. Depending on the entities participating in the rules, different paths may be valid and the one with highest score among them is used to predict the masked entity/relation.\n\n**relation linking** : relation linking in general is used in knowledge base context where for a given textual form of the relation in the natural language question, you are expected to predict the corresponding knowledge base relation.  Ex: Leonardo Dicaprio acted in Titanic  then the relation linking goal is to map the text predicate acted in to the P161 (cast member) relationship in wikidata.\n\n**Response to 'Questions For The Authors':**\n\n**A: How accurate is the system (semantic parser) that generates logical forms from the sentences? How does its performance affect the presented methodology?**\n\nAMR is used for parsing the SWQ dataset which has simple sentences with a subject, object, predicate kind of structure. For these sentences, getting the logical form using AMR was straightforward and the SMATCH score for a small test set was approximately 83.0. For the CLUTRR dataset, the ground truth logical forms are readily available with the data and we used those for rule learning similar to the GAT baseline which also uses the same logical forms.\n\n**B: Section 3.3, is path_j(sad) a binary variable (line 224) or the length of the path between the entities?**\n\n Yes, it is. This means that the model in equation (1) learns a weighted combination of indicators of whether a path exists between s and d using the jth path of predicates. \n\n **C: In Section 3.2.2 b), if Ee is the set of all paths that end in d, it appears that the masked source is the only remaining source candidate, potentially making the task trivial. Can you clarify this aspect and explain how the approach addresses such cases?**\n\nE_e is the set of all paths that start at some entity s_e and end at d (not the set of all paths that end at d). Suppose there are 10 entities in our knowledge base, and then there would be 10 different sets E_e. For each of these sets (and corresponding possible source entities), a score is computed as in line 371 to use for the training loss in line 381. Note that there can be multiple paths in each E_e, denoted by the two paths from s_3 to d in Figure 6. Suppose castmember(?, Leonardo Dicaprio) is the query. Then castmember can map to a rule that maps multiple paths between destination entity Leonardo Dicaprio through the castmember rule learned during training, resulting in scoring many source entities corresponding to each path and selecting the one with highest score. Please note that the predicate here is not the direct knowledge base predicate which is a single relation, but rather the text predicate which has weighted paths in our rule model. \n\n **D: Can the model predict cases where \"no relationship\" exists? If not, how are these cases handled in the relation linking process?**\n\nThis would be answered by the predicate masking task used for training in 3.3.2 (a). Specifically, all potential relationships, i.e., predicates, can be scored as in line 341, and no relationship would be indicated by all low scores (or under some defined threshold). As the model is data driven, such a threshold would have to be tuned.\n\n**E: Could you elaborate on how the context is used during training time?**\n\nWe refer to context in the paper as the unmasked parts of text samples. Thus, the context can be either a pair of source and destination entities (i.e., in predicate masking), a destination entity and predicate (i.e., in source entity masking), or a source entity and predicate (i.e., in destination entity masking). Then either of the three masking procedures can be done using either of the three corresponding loss functions detailed in 3.2.2. For example, if the context is a pair of source and destination entities, then the loss function for this particular sample will be of the form in line 352 where s_i in line 341 makes use of weights corresponding to the ith predicate. Considering CLUTRR, suppose predicate i in a sample is son(s,d) and let the context in a sample have s=’Adam’ and d =’Jim’.  Then a subset of the paths from those in the top row of Figure 7 might lead to verifying son(Adam, Jim) and this subset corresponds to set E  in the score computed in line 341. Then for this sample the corresponding weights would get updated through the loss in line 352. \n\n**Equation 1, What is N?**\n\nN is the number of overall paths of various lengths (including 0-hop paths through k-hop paths) that can be extracted from a given knowledge base. This is described in the text written after equation (1) in lines 246-250. We will add a comment right after equation (1) to be clear.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents a novel approach to relation linking, modeling it as a self-supervised rule learning task. This is a fresh perspective on the problem and can offer new insights and benefits compared to traditional neural methods.\n- The paper explicitly identifies the shortcomings of neural approaches in relation linking, such as data hunger, lack of interpretability, and limited generalization to other domains. By proposing a self-supervised rule learning approach, the paper aims to overcome these limitations.\n- The paper claims that the self-supervised rule learning approach enables better generalization and requires fewer annotated data compared to other methods. \n- The approach offers interpretable results by generating rules that define natural language text predicates as weighted mixtures of knowledge base paths. This can provide valuable insights into how the relation linking decisions are made and contribute to the transparency of the model.\n", "reasons_to_reject": "- The methodology shows a high level of intricacy and needs a more lucid exposition. Multiple readings may be required to grasp the technical details, which can hinder the paper's accessibility and understanding for the broader community.\n- While the paper claims to offer interpretable results, it lacks an example describing what the learned rules represent. Providing concrete examples would be crucial to demonstrate the interpretability of the proposed approach.\n- The paper does not provide a clear explanation of how the logical forms from the sentences are generated, and the accuracy of this process remains unknown. Understanding the process of generating logical forms is vital to assess the reliability of the proposed approach.\n- The paper needs to provide a clearer explanation of the relationship between paths, rules, and relation linking. A more coherent description would enhance the readers' understanding of the proposed approach\n\nIn general, the presentation is insufficient to properly convey the merits of the paper.\n", "questions_to_authors": "A: How accurate is the system (semantic parser) that generates logical forms from the sentences? How does its performance affect the presented methodology?\nB: Section 3.3, is `path_j(sad)` a binary variable (line 224) or the length of the path between the entities?\nC: In Section 3.2.2 b), if `Ee` is the set of all paths that end in d, it appears that the masked source is the only remaining source candidate, potentially making the task trivial. Can you clarify this aspect and explain how the approach addresses such cases?\nD: Can the model predict cases where \"no relationship\" exists? If not, how are these cases handled in the relation linking process?\nE: Could you elaborate on how the context is used during training time?\n", "paper_topic_and_main_contributions": "The paper proposes a novel approach called \"Self-Supervised Rule Learning to Link Text Segments to Relational Elements of Structured Knowledge\" to address the shortcomings of neural approaches in relation linking for knowledge base question answering systems. The main problems tackled are data hunger, lack of interpretability, and limited generalization to other domains.\n\nThe proposed approach models relation linking as a self-supervised rule learning task. It leverages the masked training strategy commonly used in large language models but operates on logical statements derived from the sentences rather than raw text inputs. By doing so, the approach extracts extended context information from the structured knowledge source, enabling the construction of more robust and interpretable rules. These rules define natural language text predicates as weighted mixtures of knowledge base paths, and the learned weights effectively map text segments to the corresponding relational elements in the knowledge base.\n\nMain Contributions:\n1. Rule Learning for Relation Linking: The paper introduces relation linking as a rule learning task in a self-supervised setting. This novel approach enables better generalization across different domains and reduces the need for extensive annotated data compared to traditional neural methods.\n2. Interpretable Relation Linking: Unlike neural network-based approaches, the proposed method offers interpretability based on the learned rules. This provides valuable insights into how the relation linking decisions are made.\n3. Comprehensive Evaluation: The paper includes a thorough evaluation of the proposed approach on two datasets, CLUTRR (for rule learning related to kinship relations) and SWQ (for relation linking in Wikidata). The results demonstrate the effectiveness of the approach in producing high-quality rules that generalize well across domains.\n", "excitement": 4, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "# Style\n\n- Paragraph starting in line 206 needs better explanation.\n- Equation 1, What is `N`?\n- Include a formal definition of relation linking"}, "extra_text": {}}, {"review_id": "NuzRbfLZOZ", "reviewer": "Reviewer_aFLf", "summary": "", "questions": "- the advantages of transforming link text segments into relational elements of structured knowledge are not sufficiently detailed: \n1) using the authors' solution how much can the quality of a QA system be increased? \n2) Have the authors made experiments where to use their solution in a wider context? (such as a QA-type system) If so, how much has the quality of the previous solution increased? What are the advantages of using the solution proposed by them in this paper in such a system?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for detailed comments. Please find answers to the questions raised in the review below\n\n**Response to 'Reason To Reject':**\n\nDue to space constraints, we could not add more examples. In the revision, we can add more examples to improve the overall readability of the paper using the additional page of content allowed in the final version. We will address the typos in the final version. Regarding reproducibility, we will open source the code and add a git link to the final version.\n\n\n**Response to 'Questions For The Authors':**\n\n**the advantages of transforming link text segments into relational elements of structured knowledge are not sufficiently detailed:**\n\nThe role of relation linking in QA is well studied and there are many works [e.g., SYGMA] which show its importance in QA system.  We have also described a body of work in the first paragraph of the related work section that motivates relation linking. One of the critical components of any KBQA system is relation linking along with the entity linking. Entity linking accuracy is generally in the high 90s leaving relation linking a critical component that has the maximum effect on the overall QA. The SYGMA paper along with other relation linking papers provide ample evidence as to how critical the relation linking task is. We can add numbers from the references that correlate relation linking performance and overall QA performance to show the value of relation linking.\n\n**1 & 2. using the authors' solution how much can the quality of a QA system be increased? & Have the authors made experiments where to use their solution in a wider context? (such as a QA-type system) If so, how much has the quality of the previous solution increased? What are the advantages of using the solution proposed by them in this paper in such a system?**\n\n Table 2 gives the recall performance for QA on SWQ compared using our rule learning approach for relation linking. The key advantages of our relation linking with self supervised rule learning is that we do not require any supervised data and that we get human readable interpretable rules that can help in explaining the predictions in practical applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- good and actual references from important conferences and journals\n- the formalization of the problem is well done\n- the examples help the reader to understand better what is the problem and how the authors deal with it\n- the quality of the obtained results which outperforms GAT demonstrates the robustness of the proposed method ", "reasons_to_reject": "- in subsections from section 3 the authors need to add more examples to help the reader to understand how the proposal method works \n- there are also some questions without answers in the paper (see below section)", "questions_to_authors": "- the advantages of transforming link text segments into relational elements of structured knowledge are not sufficiently detailed: \n1) using the authors' solution how much can the quality of a QA system be increased? \n2) Have the authors made experiments where to use their solution in a wider context? (such as a QA-type system) If so, how much has the quality of the previous solution increased? What are the advantages of using the solution proposed by them in this paper in such a system?", "paper_topic_and_main_contributions": "In this paper, the authors present an approach to self-learn rules that serve as interpretable knowledge to perform relation linking in knowledge base QA systems.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 452: \"l ∈ {2,4,8}.\" => \"l ∈ {2,4,8},\"\nLine 511: \"13894\" => \"13,894\"\nLine 512: \"5622\" => \"5,622\" \nLine 531: \"10648\" => \"10,648\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors describe a novel approach (similar to masked language model) to self-learn rules that serve as interpretable knowledge to perform relation linking in knowledge base Question Answering. 
The paper has some merits and the proposed solution is demonstrated to be effective on 2 datasets.
The main issue of the writing: the proposed method is not clearly described and the paper does not provide enough examples. This lack of clarity does not enable the reader to have a full understanding and obfuscate the quality of the work.
Decision: Accept-Findings