LogicDP: Creating Labels for Graph Data via Inductive Logic Programming
Authors: Yuan Yang
Faramarz Fekri
James Clayton Kerce
Ali Payani
Venue: ICLR 2023 poster (2023)
Keywords: Data Programming
Graph Reasoning
Inductive Logic Programming

Abstract: Graph data, such as scene graphs and knowledge graphs, see wide use in AI systems. In real-world and large applications graph data are usually incomplete, motivating graph reasoning models for missing-fact or missing-relationship inference. While these models can achieve state-of-the-art performance, they require a large amount of training data.

Recent years have witnessed the rising interest in label creation with data programming (DP) methods, which aim to generate training labels from heuristic labeling functions. However, existing methods typically focus on unstructured data and are not optimized for graphs. In this work, we propose LogicDP, a data programming framework for graph data. Unlike existing DP methods, (1) LogicDP utilizes the inductive logic programming (ILP) technique and automatically discovers the labeling functions from the graph data; (2) LogicDP employs a budget-aware framework to iteratively refine the functions by querying an oracle, which significantly reduces the human efforts in function creations. Experiments show that LogicDP achieves better data efficiency in both scene graph and knowledge graph reasoning tasks.

Published as a conference paper at ICLR 2023
LOGICDP: CREATING LABELS FOR GRAPH DATA VIA INDUCTIVE LOGIC PROGRAMMING
Yuan Yang', Faramarz Fekri', James C. Kerce' & Ali Payani"
Georgia Institute of Technology', Cisco? [yyang7548, faramarz. fekri@ece, blayton.kerceee Jgatech. ppayani@ciscn
com com
ABSTRACT
Graph data, such as scene graphs and knowledge graphs, see wide use in Al systems. In real-world and large applications graph data are usually incomplete, motivating graph reasoning models for missing-fact or missing-relationnhipp inference. While these models can achieve state-of-the-ar performance, they require a large amount of training data.
Recent years have witnessed the rising interest in label creation with data program- ming (DP) methods, which aim to generate training labels from heuristic labeling functions. However, existing methods typically focus on unstructured data and are not optimized for graphs. In this work, we propose LOGICDP, a data programming framework for graph data. Unlike existing DP methods, (1) LOGICDP utilizes the inductive logic programming (ILP) technique and automatically discovers the labeling functions from the graph data; (2) LOGICDP employs budget-awart framework to iteratively refine the functions by querying an oracle, which signif- icantly reduces the human efforts function creations. Experiments show that LOGICDP achieves better data efficiency in both scene graph and knowledge graph reasoning tasks.
INTRODUCTION
Graph data are widely used in many applications structured representations for complex information. In the visual domain, scene graph can be used to represent the semantic information of an image. As shown in Figure la, each node in the graph corresponds to localized entity (e.g., T1; 22, and I3) and each edge represents the semantic relations between a pair entities (c.g., (æ3, Has, Similarly, knowledge graphs (KGs) represent real-world facts with entities and the relations that connect them. Standard KGs such as Freebase Toutanova & Chen (2015) and WordNet (Bordes et al., 2013) consist of facts that describe commonsense knowledge and have played important roles in many applications (Yang et al., 2017; Sun et al., 2019; Mitchell et al., 2018; Yang et al., 2022).
Graph datasets are usually incomplete and new facts be inferred from the existing facts. For example, in Figure a, the class label of I3 is missing, and one can infer that it is a Car r because 43 Has a Wheel and Window This process is referred as graph reasoning. A large body of research has been proposed to address this task. For scene graph reasoning, methods such as iterative message passing (Xu al., 2017). LSTM (Zellers et al., 2018) and GNN (Yang al., 2018) are proposed. And for KG reasoning, a variety of graph embedding methods (Bordes et al., 2013; Sun al., 2019) are proposed. These graph reasoning methods rely on standard supervised training, where the model is fed with fixed set of data that are curated beforehand. Such training leads to a state-of-the-ar performance when a large amount of data is available. However, this approach shown to be suboptimal with respect to data efficiency (Misra et al., 2018; Shen et al., 2019), and creating a sufficiently large training dataset manually can be expensive.
In this work, we approach this problem by adopting the data programming (DP) paradigm (Ratner et al., 2016), which aims at creating a large high- labeled dataset from a set of labeling functions that generate noisy, incomplete, and conflicting labels. To this end, we propose LOGICDP, a DP framework that creates training data for graph reasoning models. Compared to the existing domain-general DP frameworks (Ratner et al., 2016; 2017), LOGICDP utilizes the inductive logic

---

Published as a conference paper at ICLR 2023
Bookstrap set
Associated Predicates
Whedl
Window
LOOKDDP
Wheel
Firstrorerr logic rule
Ground
Rasz, Wheelly
Wilddowwz
Scane graph
iDtr
Hoss
IDtry
Hem
Inside
Vihrol
that has window
sumar Ping
wheels
IDcr
Oin
Inside
Has
Window
Ground
Person
logic corresponds pattern the graph
CarlXi
HeslX.Y)
WheellY) |HeeXX
Windowi
Oracle
(a)
(b)
Figure 1: (a) An image can be represented by scene graph and logic rule can be used to represent the shaded pattern the graph. (b) Learning and refining a labeling function for the unary predicate Car.
programming technique and can automatically generate labeling functions in the form of first-order logic rules from small labeled set; LOGICDP also employs a budget-aware framework that refines the functions through weak supervision from oracle.
The LOGICDP framework is flexible and agnostic to the choice of the graph reasoning model and the ILP method. In experiments, evaluate LOGICDP on scene graph and KG datasets together with several strong generic weakly-supervised methods, which suggests that LOGICDP scales well with large graph datasets and generalizes better than the baselines. We also showcase the training with human oracles and discuss its potential as novel human coman-in-thyn in-the-loop learning paradigm.
2 RELATED WORK
Data Programming. LOGICDP is related to the recent advances in data-centric Al and data pro- gramming (Ratner al., 2016; 2017; Varma & Ré, 2018), which aims at new paradigm for training and evaluating ML models with weak supervision. Existing frameworks such as Snorkel (Ramer al., 2017) show great potential in this direction but are also limited in applications where expert labeling functions are difficult or expensive to construct, and they do not utilize the rich semantic information in graphs automate the process. In LOGICDP, we incorporate the II.P technique and automatically generate the labeling functions from a small set of graph data.
Graph reasoning and inductive logic programming. Graph reasoning is the fundamental task performed graph data. It can be addressed by many approaches. For example, graph embedding methods (Bordes et al., 2013; Sun et al., 2019), GNN-based methods (Yang et al., 2018; Zhang al., 2020) and various deep models (Xu et al., 2017; Zellers et al., 2018). While these data-driven models achieve state-of state-of-the-ar the-art performance, they require many samples to train. On the other hand, graph reasoning can be solved by finding a multi-hop path in the graph that predicts the missing facts (Guu et al., 2015; Lao & Cohen, 2010; Lin et al., 2015; Gardner & Mitchell, 2015; Das et al., 2016). Specifically, inductive logic programming (ILP) methods (Galárraga et al., 2015; Evans & Grefenstette, 2018; Payani & Fekri, 2019; Campero et al., 2018; Yang & Song, 2020; Yang et al., 2017) learns to predict missing facts by searching for such paths and representing them logic rules. Compared to the previous methods, ILP methods are more data-efficiens and offer better interpretability. In this work, we investigate using logic rules learned by ILP methods labeling functions and generating triples to train data-driven graph reasoning model.
3 PRELIMINARIES AND PROBLEM STATEMENT
We consider graphs such as scene graphs and KGs that consist of set of facts in the form of head- predicate-tail triples. Here, we the scene graph in Figure 1a running example. Formally,
2

---

Published as a conference paper at ICLR 2023
define such graph as 9 = P. P. T),, where X' denotes the set of entities in the graph and in the case of Figure la, X' = 22, P denotes the set of predicates and example predicates P E P in Figure la are Has, Inside, Wheel and Car; finally, T denotes the set of triples in this graph, where each triple t consists of two entities and a predicate T = [t == (a, P. P,x'). J E X P E P).. commonly used in KGs, which correspond to relations between two entities. For example in Figure la, Has is a binary predicate and triples involving it are {13,, (aas,Ti) (23, Has, ...)...; they effectively represent the edges in G. On the other hand, (2) wnary predicates are not typically used in KGs but are common in scene graphs, which correspond to the class label or attributes of single entity. In Figure 1a, this can be Wheel and Wi ndow, and triples of this type are { ( Wheel, .1), (22, Windww,we .22,, ...J (we duplicate the same entity for unary predicate for notation consistency); they effectively represent the class labels of the entities.
There two types of predicates: (1) binary predicates
Graph reasoning. The fundamental task of graph reasoning is to infer missing facts from the existing ones. These missing facts are referred to as queries and take the form of (zz, ?, x²). In Figure 1a, query can be (xx, ?, dz) which asks for the missing class labels of Tz. In this work, we are interested in approaching this task by training graph reasoning model. Formally, let fo (t) be graph reasoning model parameterized by 6, which infers the score of how likely a triple exist in the graph, and let T be a set of training triples, one trains the model by minimizing the following margin-based objective
[(0:T)= 1 - Σ 1 1- fo(t) + fo(t',
RET PEN())
where (t) denotes the random triples generated by corrupting the entities x, x' in the triple (z. P. P.a"). Note that this different from classical objectives such as cross-entropy and is widely used in graph reasoning tasks because there are usually no explicit negative triples in the graph (Refer to details in (Guu et al., 2015; Bordes et al., 2013)). Our goal is to generate enough triples such that the model fe is sufficiently trained via (1).
Data programming for graph reasoning. Acquiring a large number of training triples manually is expensive. We adopt the data programming (DP) paradigm and propose to generate triples from a set of labeling functions. Similar to the reasoning model for these functions infer the predicate given query, but they are noisy and likely to conflict with each other and one needs aggregate the outputs to generate the triples. Additionally, existing DP methods (Ratner et al., 2017; 2016) rely on domain experts to manually create good labeling functions. We observe that graph data contain rich semantic information and propose to automatically learn the labeling functions from the graph. Once the functions are learned from the data, one also needs a principled framework to incorporate human weak supervision (e.g., expert confidence scores on the functions) and refine the functions.
In summary, we specify the three challenges that need to be addressed to generate high-quality training triples for graph reasoning: (1) Function Generation: how to automatically generate labeling functions? (2) Function refinement: how to incorporate human weak supervision and refine the labeling functions? (3) Label aggregation: how to aggregate the labeling function outputs into the training triples usable for (I)? We address these challenges section 4.
PROPOSED METHOD
4.1 FUNCTION GENERATION
The first challenge we address is to automatically learn labeling functions from graphs. We assume a small bootstrap set Tinst is given, from which we will generalize the functions. This realistic setting because many graph datasets such as Visual Genome (VG) (Krishna et al., 2016) have incomplete graphs with sparse connections (i.e., facts), while the data insufficient for supervised training for fe (especially for the rare predicates), one can use them to construct functions with good heuristics. For example, in Figure 1b, a few scene graphs associated with predicates Car, Wheel, etc. are given the bootstrap set, and we can generalize heuristic that "'Anything that has Wheel and Wi Window is a Car", which can be later used as a function to generate triples for Car.
To generalize functions from the bootstrap set Tinits need (T1) function representation that is interpretable as they need to be refined by humans later in the process, and (T2) a data-efficient approach to generate functions because Tiuit is small.
3

---

Published as a conference paper at ICLR 2023
Logic rules as labeling functions. The first-order logic rule severs as a principled representation for (T1). One common logic rule family used for graphs is the chain-like Horn rules:
R: P(x,, x²) t- (() =1) A (21, A
P(T) (2T-1,2))
(2)
rule R of this family consists of two parts: the head, which is P(x, x'), and the body which is p(U) (ar, 21) A (2,, 22)... A P(T) ((T) (27-1, 2")). In first-order logic, binary predicate can be treated as function P: X' x X + {0, 1). In other words, it predicts True or Fal se given pair of entities (and similarly for unary predicates), The rule body is the conjunction (i.e., logical And) of a set of predicate functions, and it is True if all of them are True. Therefore, the logic rule R is effectively a binary labeling function R X' x X + {0, 1), which encodes the statement that "P(x,z') is True if the body (x, 21) A..A A P(T) (2T-1, x') is True". Note that one can express other graph patterns with more complex rules. For example, the logic rule in Figure la corresponds to a tree consisting of two paths. For notation simplicity, we focus on the chain-like rules for the remaining contents and one can extend it to complex rules with different ILP methods.
Rule learning via Inductive logic programming. A chain-like rule R can be learned from the bootstrap set Timit efficiently via inductive logic programming (ILP). Many ILP-based methods formalize this problem as a search problem, where given a query (r, ?, x²), one searches for a path in the graph from to x' that entails the missing predicate. In particular, a chain-like rule of the
p()) p(r) x' in the graph. Therefore, learning the logic rule is equivalent to finding the path from graph. In general, learning explicit paths in the graph requires fewer data and ILP methods are significantly more data-efficient (Yang & Song, 2020).
form (2) corresponds to a path x
(T2) is a self-contained ILP problem: given graph with bootstrap triples 9 {X, P, Timith J. one learns a set of logic rules for each P E P. This can be solved with any off-the-shelff ILP methods. Here, we briefly introduce the underlying methodology and leave the details in Appendix A.
In LOGICDP, we follow the differentiable backward-chaining IIP formalism, which learns the rule by training model that searches for the optimal path in the graph. This is done by performing a [X] be the adjacency matrix of graph. For a graph G with (P] = K, there are K matrices M = {M,, Mx) J. Let Uny v'z be the one-hot
graph random walk with soft attention. Let M E {0, 1|
vectors of entity x'; suppose we choose path p(t) how likely the path exists for x. x' as
p(t)
x', we can compute the score of
T
II M ( 1
score(U2, v'.) = (T) vyg.
where ( M is the adjacency matrix of the predicate used in (t)th step. One can verify that the jth element u/ is the count of unique paths from x to j (Guu et al., 2015). Therefore, searching for the chain-like rule is equivalent to searching for sequence of matrix multiplications M [= 1, 1.... T.
This hard-search problem can be relaxed into learning the weighted sums of all
possible paths
score(vz, score(U,,1,,a, B) = 1 Σ (t)
II
(2)
M
(4)
1
where a = (a (..,),... "..,,...." (T) is the path attention vector, and 800 = [8(0 ..., 2 T is the matrix attention vector at (f)th step: they are generated from differentiable models such as RNN and transformer (Yang et al., 2017; Yang & Song, 2020).
Finally, since the model is differentiable via attention, one can train the model to maximize (q) for triples in Tins and extract the rules by sampling from the attention multinomials. We leave the training details in Appendix A because, as mentioned above, LOGICDP is agnostic to the ILP model and it is a self-contained task. For the remaining contents, we will assume the training is complete and set of logic rules Reand is generated for each predicate P E P.
4.2 FUNCTION REFINEMENT
We have now obtained a set of logic rules in the form of Eq.(2) via ILP, which generalizes predictive heuristics (e.g., Figure 1b) from small bootstrap set Teand- However, these rules are potentially noisy
4

---

Published as a conference paper at ICLR 2023
Algorithm 1: Training graph reasoning model with LOGICDP
Input: Graph G, bootstrap set Timit. budget B, threshold ?
Init: Reasoning model fo. belief set T = Timit, refined rules R. 0
Reand < Apply II.P
for i 1,... B do
R f arg ma Rand COV( (R), s.t. (t) >
À < Obtain oracle score for R
R + R u {A, [}} Reand f- Reand/ {R)
Mabe RR) Construct labeling model with Eq.(7)
T - Update belief set with Eq)
repeat
Sample triples from T
Train So by updating 0 with (q.
until Convergence;
end
and subject to spurious correlations in the data, which are harmful to generating labels. Therefore, one must refine the rules via human weak supervision. Note that this process can be very expensive in some data programming frameworks (Ratner et al., 2017) because the expert needs to manually create the functions, whereas in LOGICDP, the expert only needs to refine them.
Budget-aware function refinement. An effective refinement process should minimize human effort. We propose budget-aware framework that iteratively refines the rule by querying oracle and obtaining a confidence score. Let be the budget (the maximum number of interactions with the oracle) and Reand be the initial set of rules obtained from ILP for predicate P (one can extend it to sets of all predicates readily). The goal is to obtain a refined rule subset R. = {A) R E Reand) 3=1 3=1 with X E [0, 1] be the oracle assigned confidence score for rule Rg.
B
With a limited budget B, one should pick rules that balance between coverage: the number of triples that can be generated from the rule, and support: the number of generated triples that are correctly labeled. We quantify them through the belief set: the set of triples generated by the refined rules during the refinement process. Formally, we denote Ti as the belief set at i-th iteration and we have
To Timit, T Tinit U ffr P 2') ,'',''
arg max Masell ((x, P,,')| (x, x²) E x2 (5)
PEP
where Masel (introduced in section 4.3) is the model that generates aggregated score of triple (x, P. x') from R, and p is the inferred label with the largest score. In other words, T, is the set of triples outputted by the labeling model at i-th iteration. It is first initialized into the bootstrap set Timet then for every iteration i 1...B, we first update the labeling model Masel and then reconstruct the belief set by inferring the labels for all pairs of entities and combining the inferred triples with Tinit-
Now we define the support and coverage of rule R. Let T {<x, R[z, 2') 1, (z, x') e X²2 be the set of triples generated by rule R. Then, we define the true positive subset of T at i-th iteration T. T n Ti, which is the intersection of the generated triples and the belief set. Finally, we compute the support and coverage as
supp R) = IT. R.ttt///
cov (t) = |T
(6)
In other words, supp( R) is the ratio of the true positives in R and cov is the size of T divided by the total number of possible entity pairs.
For each iteration, we measure the support and coverage for each rule in Reand and send the best one to the oracle for weak supervision (i.e., A). This is done by (1) filtering out rules with support below a threshold y; and (2) choosing the rule with the largest coverage and sending it to the oracle (Algorithm 1). In the experiments, we demonstrate the noise introduced by the incorrect labels can be alleviated by the aggregation model introduced in section 4.3. A similar strategy is also used in previous work (Ramee et al., 2017; Varma & Ré, 2018).
5

---

Published as a conference paper at ICLR 2023
4.3 LABEL AGGREGATION
Once we obtain the refined rule set R, the last task is to aggregate the noisy and conflicting labels generated by different rules and produce the final labels for training the reasoning model f. In previous work (Ratner et al., 2017), this process is formalized with generative model, which can be expensive for graph data as involves probabilistic inference. In LOGICDP, we approach this task by utilizing the posterior regularization (PR) (Ganchev et al., 2010) technique as it is principled framework that can efficiently incorporate weak supervision and previous work (Guo et al., 2018; Hu al., 2016) has shown is effective for alleviating the noise introduced by the imperfect logic rules.
Formally, let R. = {(A Ry) } be the set of refined rules and confidence scores of predicate P (one can extend it to sets of all predicates readily). The PR technique treats not as independent functions but as a set of constraints imposed on the posterior of the reasoning model fa. This leads to a constrained optimization problem (Appendix B). Solving the problem yields a labeling model
B
Miahel (4] fo exp
11 (-(((RR((T,))))) t: (E, exp -CE
where C' is the weight of the exponential term. The score of Maael is proportional to the product of fo(t) and an exponential term. This term is effectively log-linews model that sums over the scores of rules in R weighted by As. Maael balances between the supervision from R and the inference from for with large C, the model generates scores that close to that from R. and vice versa.
With M we train the reasoning model fo by constructing and sampling from the belief set with (5). We summarize the overall pipeline in Algorithm 1. Note that this training process partly contains self-learning: So is trained on the belief set which is in turn partly generated by So via (7). This is intended as it is shown to lead smooth learning process (Hu et al., 2016).
EXPERIMENTS
In the experiments, we demonstrate the following properties of LOGICDP: (P1) the budget-aware framework empowers LOGICDP to generate high-quality triples leading to better data efficiency than generic weakly- reakly-supervised methods; (P2) LOGICDP is robust against the incorrect labels; and (P3) the logic rule is interpretable, allowing humans to supervise the model efficiently.
5.1 DATASETS
Knowledge graphs. We evaluate LOGICDP on two KG datasets. FB15K-237 is a subset of the Free- base dataset (Toutanova & Chen, 2015) which contains general knowledge facts. WN18RR (Dettmers al., 2018) is the subset of WordNet18 which contains relations between English words. We evaluate the DP on standard graph reasoning on binary predicates with train/ wain//tlid/test splits provided addition to it, we remove a large portion of triples from the train split, effectively reducing the size of the bootstrap set Tiriit, so that we can inspect how the size of Timit can influence the performance. And the removed triples are used as the unlabeled sets for baseline methods such as active learning, where the ground-truth labels are hidden. (Statistics in Table 4).
in (Yang et al., 2017).
Visual Genome. We evaluate LOGICDP on the Visual Genome (VG) dataset provided (Krishna et al., 2016) which contains scene graph for each image. The original dataset is highly noisy (Zellers et al., 2018). We instead use a processed version provided in (Hudson & Manning, 2019). We further process the relational data by filtering out the relation types that occur less than 1500 times. The remaining set consists of 31 types of relations. For each scene graph, we filter out the isolated nodes that are not connected by those relations. For this benchmark, the model is tasked to predict the class label of an entity in the image. To do so, we rank the classes by their frequencies: triples belonging to the top 50 classes together with all the binary predicate triples are treated as background knowledge sets
and given to the model; triples of the next 70 classes are split into bootstrapp/vliidtts//mm (triples of the rest of the classes are discarded as they are too sparse for evaluation). The bootstrap set Tenst for each of the 70 classes contains 15 random triples, and the valid, test, and unlabeled sets are split with a 10%/80% /10% ratio on the remaining triples.

---

Published as a conference paper at ICLR 2023
Table 1: Training the TransE and the RotatE models with 5%, 10% and 20% of the bootstrap set Tanit on two knowledge bases. LOGICDP achieves the best scores (P1) and has similar performance as the case trained with full ground-truie truth data, i.e. the ideal* case.
TranE
ResuE
Methods
FB11K-237
WNTIRR
FHI5K-231
WNINRR
XIRR
1015
2016
10%
206
10%
20%
10%
20%
20%
5%
10%
30%
5%
10%%
20%
5%
1095
AI-RINI
0:31
0.21
0.32
16.1
16-7
16
0.76
0.29
0.31
98.4
97.8
96.1
0.34
0.25
0.27
57.1
9.5
41.3
0.22
08.22
0.50
24/7
14.0
45.5
MaxMan
0.22
0.22
0.23
36.1
37.2
36.5
0.29
0.31
0.35
29.5
35.1
45.9
0.24
0.28
0.29
38.5
432
47.
8.31
0.32
8.57
33.2
42.2
51.2
Labe
0.22
0.23
0.24
35,9
38,5
40.2
0.30
0.32
0.35
31.2
28.3
43.5
0.25
0.28
0.29
38.8
438
46.9
0.31
0.33
0.37
348
43.9
50.8
Lomec1p
0.37
0.36
0.39
4M
43.8
46.5
0.38
1143
0,43
16.7
89A
50.5
01311
0.32
11.33
01.3
51.7
52,0
0.41
0.63
[1,1
69.4
F2.9
53.7
0.29
46.5
0:45
51.3
0,34
52.6
0.46
55.2
5.2 EXPERIMENTAL SETTINGS
Oracle. We evaluate LOGICDP with two oracle settings: (1) synthetic. Experiments in section 5.3 (P1) is conducted with a synthetic oracle. We simulate human weak supervision by computing supp( R) on a separate held-out set and sending it back as A. (2) Human. To better demonstrate the fixed set of questions with human oracles who are asked to judge the quality of the rule based on commonsense.
interpretability of LOGICDP (P3), we also evaluate LOGICDP
Graph reasoning models. For FB15K and WN18RR, we use TransE (Bordes et al., 2013) and RotatE (Sun et al., 2019) as the reasoning models as they are widely used methods for KG reasoning. For VG, the generated triples are effectively class labels of the scene entities; the task reduces to multi-class classification task, therefore, we use a standard MLP model which takes in the RCNN feature (provided in (Hudson & Manning, 2019)) of the entity and outputs its class label.
ILP model. LOGICDP is agnostic to the choice of ILP model. We choose the ILP model based on the type of benchmark dataset. For FB15K and WNISRR, the predicate and entity spaces are moderate, we use the For VG, the predicate and entity spaces are significantly larger, we use the backward-chaining method NLII (Yang & Song, 2020) as it is more scalable than the forward-chainin ones. In general, LOGICDP is insensitive to the choice of ILP methods (Appendix F) and performs similarly across different configurations.
method dNL-ILP (Payani & Fekri, 2019) for rule learning.
Baselines. LOGICDP utilizes a budget-aware framework for supervision where we train the model with bootstrap set Tinit and limited budget B. This setting is comparable to those generic seuklyy-upervvied methods. We demonstrate (P1) by comparing against 4 generic weakly-supervine methods: (1) AL-RND: naive method that trains fo with ground-truth triples; (2) AL-MaxEnt: the active learning method with maximum entropy criterion with budget size as B: (3) LabelProp: a semi-supervised method that infers labels using label propagation algorithm (Zhu & Ghahramani, 2002); (4) MAML (Finn et al., 2017): a meta-learning framework that learns the optimal parameter initialization (this is only evaluated on VG there is no balanced set of sub- graphs in72 FB15K and WN18RR). Apart from Timit. methods (1))3)(4) are given B randomly sampled ground-truth triples, and method (2) can acquire B ground-truth triples from the unlabeled sets.
Parameters and evaluation. We set hyper- over the validation set. For FBISK and WN1SRR, we evaluate the mean reciprocal rank (MRR) and hits@ 10. For the scene graph dataset, We evaluate the Recall @ I (R@) 1) score. Experiments are conducted on a machine with 17-8700K and one GTX1080ti. Implementation available here!
hyper-paramette parameters y = 0.7, and C = 20 by running a grid search
5.3 DATA EFFICIENCY
Knowledge graphs. We evaluate the methods by changing the size of the Timis while fixing B We show the results with 5%, 10% and 20% of the original Tinst in Table 1. We also show the performance when the model trained with full ground-truth data, i.e. the ideal* # case. LOGICDP outperforms all baselines in both two benchmarks. Note that the gap is large for WN18RR because many triples can be inferred with similar logic rules. LOGICDP also achieves similar performance as that in the ideal* case with 10% of the data.
Visual Genome. We evaluate the methods by varying the budget from I to 500 while fixing [Triel 15. We show the results in Table 2 and example logic rules in Figure 5, Similar to Table 1,
7

---

Published as a conference paper at ICLR 2023
"A maror sammching that can be 15
"A door somathing that below sgg
Table 2: Training the MLP with varied budget sizes on VG dataset. LOGICDP achieves the best R @ I (P1) and 80% performance as the case trained with full ground- truth data (ideal* R case) with B = 10.
human wai
20
syrthetic
10
10
ed esskurttoo
0.68
0.35
0.66
0.35
R@I of MI.P on Visual Genome
7A train something that nears window
"An sem ss something chair can have
Method w/ MLP
Budget
18
human
15
human symthetc
1
5
10
50
200
500
10
10
AL-RND
0.47
0.47
0.5
0.52
0.57
0.6
AL-MaxEnt
0.47
0.47
0,51
0.52
0.58
0.61
LabelProp
0.49
0.51
0.51
0.57
0.59
0.59
0.88
0:35
PB1
0.66
0.35
MAML
0.48
0.48
0.5
0.54
0.58
0.6
Figure 2: Human evaluation score distribu- tions of 4 questions. The red bar corresponds the score of the synthetic oracle.
LOGICDP
0.56
0.57
0.58
0.6
0.62
0.63
ideal*
0.71
LOGICDP achieves the best R@ 1. The improvement moving from budget to 10 is small because most triples can be generated with less than 3 rules and additional rules typically cover rare triples that only occur less than 50 times for each class. As verification, 80% of the ideal* performance with B- = 10, meaning most of the correct samples are covered in the belief set. Note that ILP generates at most [Tenir] number of rules (i.e. one unique rule per sample), thus for B >> 15. LOGICDP simply fills in the belief set with random triples just as that in AL-RND, in which case, the performance improves at a rate similar to other baselines.
see that LOGICDP achieves
Technically, the budget size of LOGICDP and other methods are not comparable as a logic rule carries more information than a single triple. However, the main motivation of LOGICDP is to train the model efficiently with questions that generalize to many triples. In this way, being able to carry more information in a single query is by itself an advantage over the traditional paradigm. In section 5.5, we further quantify this difference by comparing the efforts spent on evaluating a rule vs. a triple.
5.4 ANALYSIS ON LABELING ERRORS
Imperfect logic rules introduce noisy labels into the belief set. We analyze the robustness of LOGICDP against these errors (P2) on VG dataset with B = 5. In Figure 3, we show the ratio of false positives (FP), false negatives (FN), and true positives (TP) for each of the 70 classes with respect to the size of their ground-truth triples.
Ideally, if one learns perfect of rules for class P. then the TP ratio should be 100% and FP and FN be zero. However, a noisy rule can introduce (1) FP, where triples of other classes are incorrectly classified as P. Introducing FP deteriorates the belief set and is harmful to the training. Here, figure suggests that the FP ratio is small across all classes and the total FP ratio is 6.7%, suggesting the rules are of high support. On the other hand, there are (2) FN, where ground-truth triples class P are not covered by any rules and thus fall outside of the belief set. We found that logic rules are generally of high coverage for frequent classes and less for infrequent ones. This suggests that these classes contain more diverse graph patterns and the rules learned from small bootstrap set cannot cover all of them. This aligns with the fact that VG contains massive scene graphs of various structures. Unlike FP, FN does not deteriorate the performance and one can readily improve the coverage by increasing the size of the bootstrap set.
5.5 WEAK SUPERVISION VIA INTERPRETABLE LOGIC RULES
As one of the key motivations, argue that human supervision via logic rules is data-efficient because humans can evaluate rules directly with commonsense (P3). To see this, we compare two types of supervision paradigms: (1) sample-wise labeling, where humans supervise the model by labeling individual samples; and (2) rule-wise labeling, where humans supervise the model by evaluating the rules (i.e., the function refinement introduced in section 4.2).
8

---

Published as a conference paper at ICLR 2023
Auto-labeling Quality
Question evallatiio
125%
False Pos.
False Neg.
True Pos.
0.8
100%
0.5
75%
0.4
50%
0.2
human synthetic
30
35
25%
# of questions
Figure 4: P@1 1 scores of the 35 logic questions sorted in descending order with respect to the human score.
Figure The class- ratio of true positive, false negative, and false positive triples.
For this study, we invite 25 graduate students to go through these two paradigms on VG dataset and record the "wall clock time taken" for completing the task (Details in Appendix E). For sample-wise labeling, the participant is shown 50 images randomly sampled from the target classes. Target objects are highlighted with the bounding boxes and the participant is asked to mark true false on whether or not the image patch belongs to the target class. For rule-wise labeling, We collect 35 logic rules and convert them into natural language questions (Figure 6). For each question, the participants choose from a 4-point scale that reflects how they think the rule is generally true or not; we interpret this as the intuitive support score RR).(R).
We summarize the average labeling time of two types of supervision in Table 3. Rule-wise labeling takes 8.3s which is 2x longer than the sample-wise labeling, but this time is amortized as the rule applies to many samples, leading to amortized time of 0.03s, which is 100x faster than the sample-wise counterpart.
Table 3: Average/amortized time taken in evaluating the individual samples and the logic rules.
Eval. type
Avg. time (s)
Std. (s)
Synthetic oracle human oracle. We compare the scores of two oracles of the 35 questions in Figure 4. The 4-point scores of the human oracle converted to {1, 0.66, 0.33, 0} respectively and are averaged across par- ticipants for each question. Overall, the human evaluations tend to agree with that of the oracle: we fit the scores with linear trend line (in light blue) and is close to the score curve of the oracle. However, there are also outliers: Figure 2 showcases the score distributions of four questions, where the two oracles agrre on the first row but disagree on the second row. This due to the inherent bias in the dataset: in real-life, a Cha r does not always have arms, but in VG, the Arm is highly correlated with Cha i i in the dataset, Icading to counter-inten inntter-nttuiive statements. Addressing this issue involves altering the original dataset and we leave this for future investigation.
Sample Rule
3.7
1.8
8.3
3.1
Amortized
0.03
Additionally, Figure 2 also suggests that human evaluations are generally consistent for different participants. This observation is verified in Appendix E, where we found the standard deviations of the scores are small. This aligns with (P3) that human commonsense is a reliable source of weak supervision for tasks such image classification. Nevertheless, for domains that involve graphs of complex reasoning or procedures, such as theorem proving and molecular synthesis, human commonsense will not be effective. In future work, investigate extending the framework to these domains.
CONCLUSION
In this work, we propose LOGICDP, a data programming framework optimized for creating labels for graph data. The framework capable of automatically learning labeling functions in the form of logic rules via ILP and refining them with budget-aware interactive framework. Experiments show that LOGICDP is more data-efficient than generic supely-s methods and supports efficient human-in-ther -nt---ooo supervision.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported in part by sponsored research award by Cisco Research.
REFERENCES
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pp. 2787-2795, 2013.
Andres Campero, Aldo Pareja, Tim Klinger, Josh Tenenbaum, and Sebastian Riedel. Logical rule induction and theory learning using neural theorem proving, arXiv preprint arXiv: 02193, 2018.
Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew McCallum. Chains of reasoning over entities, relations, and text using recurrent neural networks. arXiv preprint arXiv: 1607.01426, 2016.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. Thirty-second AAAI conference on artificial intelligence, 2018.
Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61:1-64, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volie Volume 70, pp. 1 1135-1 JMLR. org, 2017.
Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M Suchanek. Fast rule mining in ontological knowledge bases with amie+. The VLDB Journal-The International Journal Very Large Data Bases, 24(6):707-730 2015.
Kuzman Ganchev, Joao Graça, Jennifer Gillenwater, and Ben Taskar. Posterior regularization for structured latent variable models. The Journal of Machine Learning Research, 1 11:2001-2049, 2010.
Matt Gardner and Tom Mitchell. Efficient and expressive knowledge base completion using subgraph feature extraction. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1488-1498, 2015.
Shu Guo, Quan Wang, Lihong Wang, Bin Wang, and Li Guo. Knowledge graph embedding with iterative guidance from soft rules. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Kelvin Guu, John Miller, and Percy Liang. Traversing knowledge graphs in vector space. arXiv preprint arxiv: 1066:1506.00 106.01500.001944 2015.
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logic rules. arXiv preprint arXiv: 06318, 2016.
Drew A Hudson and Christopher D Manning, Gqa: A new dataset for real-world visual reasoning and compositional question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6700-6709, 2019.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bersstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 2016.
URL hwtpt:ttp//.o / http///axiii// rxiv, wwwgorgg/ooo..... abs/1 200.6602333333 07332.
Ni Lao and William W Cohen. Relational retrieval using a combination path-constrained random walks. Machine learning, (1(1):33-67, 2010.
10

---

Published as a conference paper at ICLR 2023
Yankai Lin, Zhiyuan Liu, Huanbo Luan, Maosong Sun, Siwei Rao, and Song Liu. Modeling relation paths for representation learning of knowledge bases. arXiv preprint 150.00397,, 2015.
Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens Van Der Maaten. Learning by asking questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11-20, 2018.
Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bo Yang, Justin Betteridge, An- drew Carlson, B Dalvi, Matt Gardner, Bryan Kisiel, et al. Never-ending learning. Communications of the ACM, 61(5): 115, 2018.
Ali Payani and Faramarz Fekri. Inductive logic programming via differentiable deep neural logic networks. arXiv preprint 196.v:190..03523, 2019.
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré, Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, 11(3):269-288) 2017.
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29, 2016.
Tingke Shen, Amlan Kar, and Sanja Fidler. Learning to caption images through a lifetime by asking questions. In Proceedings of the IEEE International Conference on Computer Vision, pp- 10393-10402, 2019.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation complex space. arXiv preprint /902 10197, 2019.
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop Continuous Vector Space Models and their Compositionaliiy, pp- 57-66, 2015.
Paroma Varma and Christopher Ré. Snuba: automating weak supervision to label training data. Proceedings of the VLDB Endowment, 12(3):223-2366 2018
Danfei Xu, Yuke Zhu, Christopher Choy, and Li Fei-Fei. Scene graph generation by iterative mes- sage passing. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pp. 5410-5419, 2017.
Fan Yang. Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. In Advances Neural Information Processing Systems, pp. 2319-2328, 2017.
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruy Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In Proceedings of the European conference on computer vision (ECCV), pp. 670-685, 2018.
Yuan Yang and Le Song. Learn to explain efficiently via neural logic inductive learning, In International Conference on Learning Representations, 2020. URL https: /tttpp:://pppreeee /ptp///eeeeeee.ee//mmmemee net/
Yuan Yang, James C Kerce, and Faramarz Fekri. Logicdef: An interpretable defense framework against adversarial examples via inductive scene graph reasoning. In AAAI, 2022.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5831-5840, 2018.
Yuyu Zhang, Xinshi Chen, Yuan Yang, Arun Ramamurthy, Bo Li, Yuan Qi, and Le Song. Efficient probabilistic logic reasoning with graph neural networks. arXiv preprint arXiv:2001.11885 2020.
Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propaga- tion. 2002.
11

---

Published as a conference paper at ICLR 2023
A LOGIC RULES AND INDUCTIVE LOGIC PROGRAMMING
first-order logic (FOL) rules consists of (i) a set of predicates defined in P. (ii) a set of logical variables such as X and Y, and (iii) logical operations {A, V, -}. For example,
GrandPatherOf(X. X') <Fathezof((Xx))) (x)))
(8)
involves predicates GrandFatherof, FatherOf and MotherOfs Components such as FatherOf Y) are called atoms which correspond to the predicates that apply to the logical variables. Each atom can be seen lambda function with its logical variables input. This function can be evaluated by instantiating the logical variables such as X into the object in X.. For example, let X (Amy, Bob, Charlie), we evaluate FatherOf (Bob/ X, Amy/ /Y) by instantiating X and Y into Bob and Amy respectively. This yields 1 (i.e. True) if "Bob is the father of Amy". The outputs of all atoms are combined using logical operations (A, V, }} and the imply operation p + q is equivalent to p V -q. Thus, when all variables are instantiated, the rule will produce an output as the specified combinations of those from the atoms. By using the logical variables, the rule encodes the "lifted" knowledge that does not depend on the specific data. Such representation is beneficial because (i) the rules are highly interpretable and can be translated into natural language for human assessment, and (ii) the rules guarantee to generalize to many examples.
In section 4.1, we learn a set of rules Reand from bootstrap set Tinit via ILP. Given query triple (ar, ?, ,²), ILP seeks to learn logic rule that captures graph pattern within which the query triple is
True. This can be formulated as finding a relational path p(t) p(r) x' which starts from one node and ends the other node x', where p(t) PP L 1,... T. Finding such path is equivalent to learning a chain- like entailment rules (Yang et al., 2017) as
R: P(x, x²) + (x)) 21) A (21, 22)... A p(l) (2T-1,
where the knowledge is encoded "if the path exists, then P(xr, is True". Numerically, we have R A x X' + {0, }. that is R(x, 2²) = 1 if the query is True. Note that if the path does not exist, the rule outputs R(x,, )
abstain. Due to the nature of logical implication "T"", "abstain" simply means the rule does not apply instead of logical False. In other words, the rule I can be used infer positive triples but not negative ones. Since many of the KG completion models are optimized with margin loss (I) and thus do not rely on the negative triples for training, we omit the learning of the negative rules. This chain-like rule is also applicable to unary predicate P(X) by instantiating X' into localized entity (see details in (Yang & Song, 2020)).
To compute the path efficiently, one encodes graph into a set of adjacency matrices. Each predicate P' can be represented M (0, 1)1œ|*|œ1, where may 1 indicates (I,, 2zz) exists in the graph. In the case of unary predicate, the matrix is diagonal such that my = 1 indicates (I;, P, x exists in the KG. Here, we duplicate the same variable for notation consistency. Recall that vg is the one-hot vector with dimensionality [X'.. We can represent each hop in the relational path as matrix multiplication, such that the (4)th hop of the reasoning along the path computed as
(t) =
11T (
Up
where M (1) denotes the adjacency matrix of the predicate used in th hop. One can verify that the jthh element u in (tt) is the count of unique paths from x to j (Guu et al., 2015), After (T') steps of reasoning, compute the score of P(x,, x²) being True
score(u.: o D II M 2/
(10)
Finding the chain-like rule is equivalent to finding sequence of matrix multiplication M / 1..., T. such that (q,) is maximized for all the triples in Tinst. This hard search problem can be relaxed into learning the weighted sums of all possible paths
T
C'
K
u,laa,, 8) = T Σ (1)
=
II
(()) k
M
(11)
1=
12

---

Published as a conference paper at ICLR 2023
where [a 1..,, ( ( is the path attention vector, and ((0) [8 xx ]T is the matrix attention vector at (1)th step, and we denote all the attentions as (a,, 13.. Therefore, for each triple P(z,z") one can find the optimal logic rule by maximizing Eq.(1 with respect to (a. 8). To extract the actual discrete rule, one takes the ArgMax on (a, 3), which returns the index of the max value. The optimal rule length is 7 ArgMax(a) and the (4)th matrix is M (0) ArgMax(8(()) which corresponds to the predicate p
These attentions can be generated differentiably by deep models. For example, NeuralLP (Yang et al., 2017) 7) uses an RNN controller to generate the sequence of attention vectors with æ as the initial input. And NLIL (Yang & Song, 2020) generates attentions with a stacked Transformer module with predicate embeddings as the input. Let ILF be the differentiable ILP model, we optimize paramter such that
arg mnax
Σ
score( v/
(12)
(M P. Tenet
where 3) ((U, P, 1g) The LOGICDP framework is agnostic to the actual choice of the ILP model, one can also apply forward-chaining methods such as ØILP (Evans & Grefenstette, 2018) or dNL-ILP (Payani & Fekri, 2019) to achieve similar outcomes, Finally, we collect all rules learned from Tinat as our rule set Reand-
B POSTERIOR REGULARIZATION
In section 4.3, apply the posterior regularization (Ganchev et al., 2010) on the rule set and construct Mabel as a teacher model that aggregates the joint inferences the rules R. Formally, a teacher model is defined as the solution convex optimization problem
B
min KL (Mabbal (t/R) Ilfe (t)) + C Masrie
Σ
(13)
1=1
s.l. A1 EM [ x )]) < j j= j= B,
(14)
where & = EI..... 0 are the slack variables and C' is the regularization coefficient controlling the weight of the penalty term. Intuitively, we want the teacher to reflect the belief of the logic rules. This is done by putting the inequality constraint (14). On the other hand, while rule constraints are all satisfied, we want the teacher to stay close to fo which the reasoning model we seek to train, such that it creates smooth optimization surface that makes it easy for the fo learn from the teacher. This is done by minimizing the KL divergence in Eq.(13) between Maaell and fe. This problem is solved analytically
B A5(1 -c 5=1
Mishel (t|R) x (t)
R)(o,,')) t:= (r, P,x'),
(15)
C DATASET STATISTICS
We summarize the statistics of the three datasets Table 4.
Table 4: Statistics of FB15K-237, WN18RR and Visual Genome datasets.
KG
# facts
# entities
# predicates
FB15K
272K
15K
237
WNISRR
93K
41K
11
VG
1.9M
1.4M
2100
D EXAMPLE LOGIC RULES
We show example logic rules learned from the Visual Genome dataset in Figure 5.
13

---

Published as a conference paper at ICLR 2023
Shelf
Person
Pole
Bike
Flag
Food
Book
Person
Bike
Pe Ridingg x)
BookiXi
ShelflY)
Food
Person(Y)
FlagiX)
P)le
Onix, Y
Eating X)
Onn(x, YJ
Pillow
Player
Roof
Ocean
Ball
bed
Undeegion
Boat
Building
Bailx
Player(Y)
X
R
BedxX
PillowlY
BoattX)
Oceanky)
^ Holding X)
A Has
AOnY,
A AIn(X, Y
Figure Showcases of logic rules learned from the scene graphs.
E
LOGICDP WITH HUMAN ORACLE
mirror something that the wall
Yes, always
Yes, usually
Not quite right, but sometimes yes
No
A mirror is something that can be on the motorcycle
Yes, always
Yes, usually
quite right, sometimee yes
No
glass something that can the
allays
Yes, usually
Not quite right, but sometimes
Figure 6: Screenshot of the question forms
The screenshot of the question forms used in the human oracle study is shown in Figure 6. This study approved by the institute IRB. The document will be made available after the review phase for anonymous purposes.
14

---

Published as a conference paper at ICLR 2023
1.0
0.4
02
0.0
Figure Mean and std of human evaluation scores of the 35 questions collected in section 5.5.
Human score distribution. We show the mean and std of human evaluation scores in Figure 7 for the 35 questions. We find the confidence scores generally consistent for all participants, which suggests that the logic rules can be evaluated with general human commonsense. However, note that LOGICDP does not require consistent score distribution function: each rule R obtains only single confidence score A.
F ADDITIONAL ABLATION STUDIES
Table 5: Performance of LOGICDP with three different ILP methods on the FB15K and VG datasets: NeuralLP (Yang al., 2017), NLIL (Yang & Song, 2020), and dNL-ILP (Payani & Fekri, 2019).
FB 1K3-
VG
ILP method
MRR
Hits 10
R@
NeuralLP
0.26
38.9
0.55
NLIL
0.28
42.9
0.57
dNL-ILP 0.28
43.8
0.54
LOGICDP is agnostic to the choice of ILP methods. In the experiments, found LOGICDP performs similarly with both dNL-ILP (Payani & Fekri, 2019) and NLIL (Yang & Song, 2020) methods, indicating LogicDP is also insensitive the choice of ILP methods. To further validate this observation, we show LOGICDP performance on FB15K and VG datasets with three ILP methods.
We the same budget and bootstrap set size as in section 5.3. For FB FB15K, set [Tana 10% and B = 5; for VG, Timii = 15 and B = 5. The results are shown in Table 5, which suggests that LOGICDP is not sensitive to the choice of the II.P methods.
15


Reviews:
[{"review_id": "uy3Q50eIhr1", "reviewer": "Reviewer_M5YG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper introduces an ILP based method for adding labels tp unlabelled graph The ussfdulnesss of rge algorithm is supported by an extensive set of experiments,SS\n\n", "summary_of_the_review": "I really liked the basic idea of the paper. Unlabeled data is a big problem,and it is nice to see progress in the area/\nThe experimentas are welll designod;  I usually feel a bit iffy about human input, but I suppose there is a real need here.\n\nI am not focused on graph learning, so I may ne missing smething. but in general I liked the paperr.\n", "strength_and_weaknesses": "The main strength of the paper is uding iLP in an unusual directiom.\nThe factt that the authord use sampling of the original datasets raises the question of scalability.\nI felt it wss sometimes a bit unclear what was ols and what was new.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "sFOChOIDbH", "reviewer": "Reviewer_nzQK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a pipeline approach to supplement labels for graph data. Firstly, it learns some logical rules from existing labeled data. Secondly, it refines the learnt set of logical rules by asking an oracle (such as human being) to assign confidence scores to a fixed number of selected rules. Finally, it estimates labels (the predicates on entities) through the refined set of soft rules and a self-learnt reasoning model. Experimental results on three benchmark datasets demonstrate the proposed approaches outperform four baseline weakly-supervised methods.", "summary_of_the_review": "Although the paper proposes a new method for adding labels to graph data and provides evaluation results to demonstrate its effectiveness, the originality is marginal considering that the proposal is only a simple application of the data programming framework. Besides, there remain some issues on whether oracles can assign reasonable confidence scores to logical rules and on whether the empirical comparison with other baseline methods is fair enough.", "strength_and_weaknesses": "Strengths:\n\n(1) The paper proposes a new method for adding labels to unlabeled graph data, which will be a good contribution to the field of semi-supervised learning.\n\n(2) The advantages of the proposed method are demonstrated with certain choices of the pipeline steps.\n\nWeaknesses:\n\n(1) The experiments are somewhat unfair. The oracle of the proposed approach assigns confidence scores to B logical rules, while the oracles of all other compared methods handle B ground-truth triples. Considering that logical rules are more informative than triples, the superiority of the proposed methods may possibly be due to more information obtained from oracles.\n\n(2) It is unclear whether human beings can assign reasonable confidence scores to logical rules. The experiments do not confirm this. Although the evaluation cost (in time) is reported, there is no evidence to show that different human beings can assign consistent confidence scores to logical rules.\n\n(3) The ultimate performance of the proposal heavily depends on the pipeline steps, including which method is used to learn logical rules and how accurately the oracle assigns confidence scores to selected rules. No principles or guidelines are provided to design a reasonable pipeline step.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "binSu3TB8f", "reviewer": "Reviewer_FEgy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an integrated prediction and labeling framework for graph data based on learning logical labeling rules which are scored by an oracle. Candidate labeling rules are learned via Inductive Logic Programming (ILP), and new labeled triples are identified using the potentially noisy rules via posterior regularization. Experimental evaluation show LogicDP achieves good performance with fewer original labeled data points by using the labeling rules, and that, for human oracles, labeling rules for these tasks is much more time-efficient than labeling individual instances.", "summary_of_the_review": "I find the core contribution to be strong, although there is some room for improvement on clarity aspects.", "strength_and_weaknesses": "The core idea is a compelling, natural, and creative extension of existing research directions. The technical execution of the idea seems sound, and the experimental evaluations show some good advantages of the approach.\n\nI found some clarity challenges in both the method description and the evaluation. To some extent this is probably an inherent challenge to the \"model within a model\" approach and the varied techniques (logic rules, neural nets, continuous relaxations, posterior regularizations) employed here. It might be helpful to have a running concrete example, or more diagrams with a concrete example for different steps in the process. Besides this I had a number of smaller stumbling blocks from a clarity perspective, documented below.\n\nBaselines: why not compare against a \"Data Programming\" method like Snorkel or Snuba? It seems strange that only AL-MaxEnt is also doing a budget-based approach (although I guess it makes sense to give the other methods B extra randomly labeled triples).\n\nGiven the \"self-training\" in the approach, are there potentially catastrophic failure modes if poor initial rules or triples are chosen?\n\nThe \"outliers\" in the synthetic-vs-human oracle due to bias in the dataset seems like a valuable observation and good direction for future work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FptwgX6zSP5", "reviewer": "Reviewer_1tSH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a weakly supervised learning approach to generate labels for unlabeled triples on graphs. Overall, the proposed approach could be regarded as a combination of inductive logic programming and active learning. The ILP module learns a set of (noisy) logic rules to perform relation completion in the graph data; while the active learning module combines rule coverage and true positive rate into a scoring function to rank the rules and generates queries to ask the oracles to make annotations/revisions.  The combination of ILP and active learning benefits from first-order logic's interpretability and expressive power, which makes human-involved model revision possible, and the revision of first-order logic rules is much more efficient than example-level label revision.", "summary_of_the_review": "This paper combines ILP and active learning to solve the edge completion task in graph reasoning. The technical novelty of this paper is limited since it is a combination of two mature systems for solving the task, therefore I think this work will attract more audience in data-mining-related venues.", "strength_and_weaknesses": "### Strength:\n- The proposed approach uses ILP to learn human-understandable rules to perform the edge completion task, making human-aided model revision possible.\n- The first-order representation reduces human effort in label revision in data programming.\n- The proposed approach is straightforward and practical.\n\n###  Weakness:\n- Edge completion in knowledge graph by rule induction is a commonly used technique.\n- In order to perform rule learning with ILP, users need to prepare a set of primitive predicates and corresponding knowledge base, which increases the difficulty in deploying your method.\n- The presentation of this paper could be improved, the introduction to ILP is not enough and the terminology in this paper is confusing. For example, Eq. 5 defines an immediate consequence operator (which is usually denoted as $\\mathrm{T_P}$ in logic programming literature) $\\mathcal{T}_i$, however, the $i$ does not appear in the RHS of the equation, or maybe it should be $\\mathcal{T}_1$?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster