Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the problem of knowledge base (KB) reasoning and propose to learn new concepts via clustering vectorized (propositionalized) entities and use the resulted entity-concept membership information as data augmentation for neural link prediction models.Two methods of finding such concepts and entity-concept memberships are proposed, one based on hard clustering of entity embeddings, and one based on an EM-style joint learning with the base neural link prediction model. Experiments are mainly conducted on FB15K-237 and WN18RR, which shows that the proposed methods can improve over ComplEx and DistMult, especially when sparsity is high.

Strengths:

- The paper is well-written and pretty easy to follow
- The proposed concept formation idea is interesting and plausible
- Some of the experiment designs are well thought-out and the results seem supportive

Weaknesses:

- I have reservations about the whole discussion around the concepts in existing knowledge graphs/bases. Concepts (or types/classes) are a central part of many KBs/KGs. I find it surprising that for the lack of quantitative intrinsic evaluation of clustering quality, it's claimed that "to the best of our knowledge there exists no large KG for which such information is available". Many KGs have a type system. For example, Freebase, of which FB15K-237 is a subset, has a strong type system. There's polymorphism and one entity could belong to multiple types, which is inconsistent with the hard partition assumption of the proposed methods (that is a strong assumption that could be easily violated), but I think one could at least use the Notable_Types in Freebase to do a reasonable large-scale quantitative evaluation of clustering quality. I'm not sure if the Freebase types and entity-type memberships are retained in FB15K-237 or are pruned in the downsampling process. If pruned, it will be interesting to see a comparison with the original Freebase type information added back to FB15K-237. Note that it's okay if the comparison is not favorable to the proposed method, because such type information is not always available for any KG, but simply omitting its existence is not okay.

- The improvement over baselines, while statistically significant, is somewhat marginal, and it's not clear whether the same improvement would translate to more advanced base models proposed more recently.

- It's not entirely clear to me why directly using embeddings learned by a neural link prediction model for clustering doesn't work well, and why using the method from Das et al [2020] would work instead. The provided explanation ("This could be due to the latent concept information which we explicitly introduce via augmentation being already captured by the neural link predictor embeddings") is hand-wavy. 
review_title: Well-written paper with interesting idea but the evaluation could be strengthened

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper aims to improve the performance of neural link predictors on sparse KGs, by augmenting KGs with clusters of concepts. It proposes two versions of augmentations: (1) ConFormA which inserts the clustered concept, and (2) ConformAE which jointly learn the concepts and embeddings through an EM scheme. The authors show that the augmentation helps improve performance of two baselines. on different datasets.

Strength:

S1. The research topic is interesting.

S2. The paper has clear motivations for each section. I especially like Q1-Q4 set at the beginning of the experiment.

S3. The paper explains its results nicely, with hypotheses/explanations for all the phenomena.

S4. The paper provides detailed appendices with additional experiment setups and results which are important for reproducibility.

Weaknesses / questions:

W1. The quality of clusters. 

From the method description, It looks like the augmentation quality is greatly impacted by the clustering quality, and so I would appreciate it if the authors can offer more detailed analyses on the clustering results. For example, would the clustering also work less stably for more sparse concepts? Are their ambiguous entity that can belong to more than one concept which may cause the hard clustering to backfire?

I understand it might be difficult to do scaled quantitative analysis, but on top of the (nice) qualitative tables in the paper, I wish to at least see some more rigorous manual inspection. For example, if we manually inspect 20 concepts, in the top-10/ last-10 clusters, what's the ratio of correctly clustered entities? Are the sparse entities correctly clustered? Are there some quality difference between concepts from ConFormA and ConFormAE?

W2. Augmentation method.

I'm curious why the authors only choose to add ISA relationships. If the concept C for an entity E1 is indeed correct, would it work better/worse if the we also extended links between (E1,E2) to (C, E2)?

W3. (minor) the paper content can be better organized.

e.g. Table 1/2 are referred in the text after Table 3, and so it would also be nice if they occur closer to when they are explained in the text.
review_title: Well-written paper, but more analyses on clustering quality would be nice

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose two algorithm for unsupervised concept learning in knowledge graphs: ConFormA and ConFormAE, where the latter is iteratively optimizing the concepts. In their experiments, the authors investigate the semantic meaningfulness of the concepts (qualitatively), the effect on link prediction performance, the impact in low-data regimes, and the generalization ability for relation types of different frequency.

Strengths:
- The research questions are interesting and the experimental design is solid.

Weaknesses:
- The main weakness of the paper is that both contributions (1. task of unsupervised concept formation, 2. algorithms based on entity clustering and KG augmentation) are not novel but have been investigated before, e.g., in Gad-Elrab et al. (ISWC 2020, https://openreview.net/forum?id=wbVVXSziQFQ).
- In general, the related work section of the paper seems to be outdated as the latest cited paper in that section is from 2018.
- The meaningfulness of the concepts is evaluated only qualitatively by manually looking at some examples. This leaves room for questions about their actual quality. Although a large quantitative evaluation might not be possible due to missing gold data, there are other possibilities, such as user studies or manual annotations of a subset of clusters.

Other comments:
- Although the authors say that clustering KG embeddings did not lead to improvements, it would be valuable to add a comparison to this approach to the paper.
- Do the authors have an explanation why the joint training (i.e., ConFormAE) did not clearly outperform ConFormA? This is a bit unexpected. Could one reason be that the E and M steps are only loosely connected (in the form of the augmented graph but not in the form of the used embeddings for instance)? In general, I see only a loose connection of Algorithm 2 to "Expectation-Maximization" as the M-step optimizes a parameter set that is not used in the E-step.
- It is not clear how the number of clusters was chosen (neither the range nor how it was optimized) and which number was used in which experiments. Since it can be expected that this choice has a large impact on the performance, this would be very important to clarify.
- The introduction should also cite TransE as this was a link prediction approach clearly before the one that was cited.
- I don't follow the first reason why ConFormA should improve link prediction performance. Can the authors elaborate on that? Which knowledge does ConFormA add that is not implicitly contained in the embeddings for link prediction already? (This is even given as one reason for not using embeddings for clustering)
review_title: Contributions not exactly novel

Meta-review: This paper investigates the learning of novel concepts in KGs via unsupervised entity clustering used for the augmentation of edge prediction models. While reviewers had some questions as to the strength of the experiments (thoroughness of baselines, strength of results above baselines), the novelty of methods proposed, and explanations for results, overall reviewers found the paper well-written and interesting.
Decision: Accept