Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 
This paper introduces an ILP based method for adding labels tp unlabelled graph The ussfdulnesss of rge algorithm is supported by an extensive set of experiments,SS


strength_and_weaknesses: The main strength of the paper is uding iLP in an unusual directiom.
The factt that the authord use sampling of the original datasets raises the question of scalability.
I felt it wss sometimes a bit unclear what was ols and what was new.
clarity,_quality,_novelty_and_reproducibility: The paper is very cleat (ok, examples would help)

`That suggest high quality,

Most of the woks in ,ethods seem tobuild open prior work, but the key  is novel (to the best of knwldge).

Repro: ok
summary_of_the_review: I really liked the basic idea of the paper. Unlabeled data is a big problem,and it is nice to see progress in the area/
The experimentas are welll designod;  I usually feel a bit iffy about human input, but I suppose there is a real need here.

I am not focused on graph learning, so I may ne missing smething. but in general I liked the paperr.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a pipeline approach to supplement labels for graph data. Firstly, it learns some logical rules from existing labeled data. Secondly, it refines the learnt set of logical rules by asking an oracle (such as human being) to assign confidence scores to a fixed number of selected rules. Finally, it estimates labels (the predicates on entities) through the refined set of soft rules and a self-learnt reasoning model. Experimental results on three benchmark datasets demonstrate the proposed approaches outperform four baseline weakly-supervised methods.
strength_and_weaknesses: Strengths:

(1) The paper proposes a new method for adding labels to unlabeled graph data, which will be a good contribution to the field of semi-supervised learning.

(2) The advantages of the proposed method are demonstrated with certain choices of the pipeline steps.

Weaknesses:

(1) The experiments are somewhat unfair. The oracle of the proposed approach assigns confidence scores to B logical rules, while the oracles of all other compared methods handle B ground-truth triples. Considering that logical rules are more informative than triples, the superiority of the proposed methods may possibly be due to more information obtained from oracles.

(2) It is unclear whether human beings can assign reasonable confidence scores to logical rules. The experiments do not confirm this. Although the evaluation cost (in time) is reported, there is no evidence to show that different human beings can assign consistent confidence scores to logical rules.

(3) The ultimate performance of the proposal heavily depends on the pipeline steps, including which method is used to learn logical rules and how accurately the oracle assigns confidence scores to selected rules. No principles or guidelines are provided to design a reasonable pipeline step.

clarity,_quality,_novelty_and_reproducibility: The quality and clarity of the paper is good. The descriptions about the proposed approach and experiments are generally clear, except a lack of principles or guidelines on method choices in pipeline steps. The originality is marginal by considering that the proposed pipeline approach is rather straightforward and it is merely a simple application of the data programming (Ratner et al., 2016) framework. No source code or data is provided through the supplemental material.
summary_of_the_review: Although the paper proposes a new method for adding labels to graph data and provides evaluation results to demonstrate its effectiveness, the originality is marginal considering that the proposal is only a simple application of the data programming framework. Besides, there remain some issues on whether oracles can assign reasonable confidence scores to logical rules and on whether the empirical comparison with other baseline methods is fair enough.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a weakly supervised learning approach to generate labels for unlabeled triples on graphs. Overall, the proposed approach could be regarded as a combination of inductive logic programming and active learning. The ILP module learns a set of (noisy) logic rules to perform relation completion in the graph data; while the active learning module combines rule coverage and true positive rate into a scoring function to rank the rules and generates queries to ask the oracles to make annotations/revisions.  The combination of ILP and active learning benefits from first-order logic's interpretability and expressive power, which makes human-involved model revision possible, and the revision of first-order logic rules is much more efficient than example-level label revision.
strength_and_weaknesses: ### Strength:
- The proposed approach uses ILP to learn human-understandable rules to perform the edge completion task, making human-aided model revision possible.
- The first-order representation reduces human effort in label revision in data programming.
- The proposed approach is straightforward and practical.

###  Weakness:
- Edge completion in knowledge graph by rule induction is a commonly used technique.
- In order to perform rule learning with ILP, users need to prepare a set of primitive predicates and corresponding knowledge base, which increases the difficulty in deploying your method.
- The presentation of this paper could be improved, the introduction to ILP is not enough and the terminology in this paper is confusing. For example, Eq. 5 defines an immediate consequence operator (which is usually denoted as $\mathrm{T_P}$ in logic programming literature) $\mathcal{T}_i$, however, the $i$ does not appear in the RHS of the equation, or maybe it should be $\mathcal{T}_1$?
clarity,_quality,_novelty_and_reproducibility: The clarity and presentation quality of this paper could be improved. This work is practical and reproducible but has limited novelty.
summary_of_the_review: This paper combines ILP and active learning to solve the edge completion task in graph reasoning. The technical novelty of this paper is limited since it is a combination of two mature systems for solving the task, therefore I think this work will attract more audience in data-mining-related venues.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an integrated prediction and labeling framework for graph data based on learning logical labeling rules which are scored by an oracle. Candidate labeling rules are learned via Inductive Logic Programming (ILP), and new labeled triples are identified using the potentially noisy rules via posterior regularization. Experimental evaluation show LogicDP achieves good performance with fewer original labeled data points by using the labeling rules, and that, for human oracles, labeling rules for these tasks is much more time-efficient than labeling individual instances.
strength_and_weaknesses: The core idea is a compelling, natural, and creative extension of existing research directions. The technical execution of the idea seems sound, and the experimental evaluations show some good advantages of the approach.

I found some clarity challenges in both the method description and the evaluation. To some extent this is probably an inherent challenge to the "model within a model" approach and the varied techniques (logic rules, neural nets, continuous relaxations, posterior regularizations) employed here. It might be helpful to have a running concrete example, or more diagrams with a concrete example for different steps in the process. Besides this I had a number of smaller stumbling blocks from a clarity perspective, documented below.

Baselines: why not compare against a "Data Programming" method like Snorkel or Snuba? It seems strange that only AL-MaxEnt is also doing a budget-based approach (although I guess it makes sense to give the other methods B extra randomly labeled triples).

Given the "self-training" in the approach, are there potentially catastrophic failure modes if poor initial rules or triples are chosen?

The "outliers" in the synthetic-vs-human oracle due to bias in the dataset seems like a valuable observation and good direction for future work.
clarity,_quality,_novelty_and_reproducibility: The model is well-positioned in relation to existing works. Using ILP for DP is (as far as I know) a novel and exciting idea. The technical execution and evaluation seem solid, modulo my clarity challenges.

Logic rules as labeling functions: "...which encodes the statement that P(x,x') is True if and only if the body ... is True" - I don't think this is correct, the Horn-like rule has a simple implication (not a bidirectional iff)?

Baselines: "LogicDP utilizes a budge-aware framework" - typo/misspelling of "budget".

Figure 3: it would be helpful to have an in-place x-axis label here.

4.2 Function refinement:
-probably it is variation in terminology across sub-fields, but I mentally mapped supp() to be the "precision" of the rule R.
-also, are the predicates typed here? If not, |X|^2 is always the same for all R and is ignored in practice for cov(), right?

Algorithm 1: for the belief set update based on Eqn 5, are we adding _exactly_ one triple <x,P,x'> each outer loop? The notation
here got a little unclear for me. If so, why not add the Top N triples under M(t|R)?

5.4 Analysis on labeling errors:
-"6.7% of the ground-truth triples are FP" - this doesn't make any sense to me?
-"indicates that LogicDP learns high-quality functions" - is it accurate to characterize the learned rules as "high precision, low/medium recall"?
-in Figure 3, what explains the persistently high proportion of False Negative? Are there "difficult" triples to cover with rules? If so, are there any common or interesting properties of these?

5.5 Weak supervision via interpretable logic rules:
-"...humans can evaluate rules directly with commonsense." - I would suggest to qualify this statement, arguably it is true for some domains (eg, wheels on cars) but probably not others (eg, binding sites on proteins).
-The paper is not misleading here, but I think it would be fair to emphasize that of course the human experimenters did the work of converting the logic rules into natural language statements for the human labeling oracles - automating this could be called out as some future work.

Figure 3 and 4: I had to flip back and forth between the text and these images a lot to figure out exactly what was going on. I'm not sure if there is an easier or clearer way to visualize the relationships here, but as-is I think these figures put some heavy lifting on the reader.

Appendix D: I guess Figure 5 is the content, but the section could have some text indicating this.
summary_of_the_review: I find the core contribution to be strong, although there is some room for improvement on clarity aspects.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes data programming for graph data with ILP-learned labeling rules.
Any human in the loop evaluation is difficult (can humans do this, how much information is provided, how to make baselines equal, etc.) but since this is not an HCI conference, this paper does an adequate and fair job.
The main strength of the paper is the proposed methodology which is interesting and fresh. The proposal is significantly different from classic data programming, and leverages core strengths of ILP.
The meta-reviewer agrees with the authors that interacting through rules is the main strength, rather than a weakness of the proposed method. In general the meta-reviewer found the author response to be very strong.
justification_for_why_not_higher_score: Not the broadest reviewer support.
justification_for_why_not_lower_score: There are no major flaws identified in the reviews that are not adequately addressed in the author response. What remains is a subjective evaluation of how important the style of work is.
Decision: Accept: poster