Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presented a jointly learning framework based on GAN for tackling both knowledge graph completion and noise detection simultaneously. Existing works only deal with each of task independently and did not investigate the benefits of coping with both tasks together. The paper is well motivated. In order to achieve them, the paper presented a GAN framework in order to train a noising KG embedding as well the generator and discriminator. The key connections between two parts are through the confidence of a noise triple and generation of the negative sample triples. The whole framework looks quite interesting and promising. The experimental results are provided to validate the effectiveness of the proposed model. 

There are two key concerns about this paper:

1) It is well known that both GAN and RL are hard to train, not to mention combining them together to joint train in order to deal with data indifferenceability issue of discrete triple generation. Are the results easy to reproduce?

2) Choosing 10% triples as positive training examples seems very ad-hoc. Have you studied the sensitivity of the number of percentage of triples as positive training examples on the system performance?

3) I don't know too much about the methods from knowledge graph noise detection so maybe one baseline - CKRL is enough for representing state-of-the-arts. However, for knowledge graph completion task, TransE is most simple baseline and they are rich state-of-the-art methods in this line such as [1]. It is not convincing to show the advantages of the proposed NoiGAN without such comparisons. 

[1]  “RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space.” ICLR'19. 



review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: This paper proposes a GAN-oriented framework for training robust-to-noise neural link predictors. My main concern is that CKRL is the only baseline -- ignoring years of prior works in this space (see e.g. [1, 2]).
Furthermore, [2] shows that two of the three datasets used by the authors suffer from test triple leakage in the training set.

Finally, the considered datasets do not really test for the presence of noise - authors may want to check out e.g. https://arxiv.org/abs/1812.00279 (there are several works in this space, all of which were systematically ignored by this paper).

Finally, authors claim neural link predictors were never used for denoising, but actually [3] use them to learn a prior distribution over triples in a probabilistic DB setting.


[1] https://arxiv.org/abs/1806.07297
[2] https://arxiv.org/abs/1707.01476
[3] https://ai.google/research/pubs/pub45634
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes to provide a novel noise-aware knowledge graph embedding (NoiGAN) by combining KG completion and noise detection through the GANs framework. More specifically, NoiGAN repeatedly utilizes a GAN model to 1) approximate the confidence score for facts identifying reliable data (discriminator) and 2) generate more challenging negative samples (generator). Then, it uses this confidence score and negative samples to train a more accurate link prediction model. The authors validate the proposed model through several experiments.

This paper reads well and the results appear sound. I personally find the idea of incorporating confidence score into a link prediction model to achieve a more accurate model very interesting. Furthermore, the provided experiments support their intuition and arguments outperforming considered baselines.

As for the drawbacks, I find the baselines considered in this work outdated missing many SOTA and related works in link prediction and noise detection [1,2, 3, 4, 5]. Further, I believe this work needs more experimental results and an ablation study capturing different aspects of the presented method. My concerns are as follows:

•	Considering the existing reverse relation issue in FB15K and WN18, I suggest conducting the experiments on the FB15K-237 and WN18RR from [6] instead. 
•	I suggest considering more recent link prediction models as baselines.
•	I am wondering if the only difference between NoiGAN and KBGAN [7] is incorporating the confidence score in the link prediction loss?
•	Considering the fact that NoiGAN repeatedly retrains GAN and link prediction model, I suggest providing a comparison of computational complexity.
•	I am wondering if NoiGAN can only work with pre-knowledge of noisy triples in KG? If not, why didn’t you report NoiGAN performance with 0% noise in Table 3?
•	I find utilizing few examples to evaluate the power of discriminator in distinguishing noisy triples (Table 4) not satisfactory at all. I suggest experimenting with more data and providing the per-relation breakdown performance of the discriminator.

On overall, although I find the proposed model quite novel and interesting, the paper needs more experimental results to validate the idea.
 
[1] Pinter, Yuval, and Jacob Eisenstein. "Predicting Semantic Relations using Global Graph Properties".
[2] Nathani, Deepak, et al. "Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs". 
[3] Balažević, Ivana, Carl Allen, and Timothy M. Hospedales. "TuckER: Tensor Factorization for Knowledge Graph Completion".
[4] Sun, Zhiqing, et al. "Rotate: Knowledge graph embedding by relational rotation in complex space".
[5] Pezeshkpour, Pouya, Yifan Tian, and Sameer Singh. "Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications".
[6] Dettmers, Tim, et al. "Convolutional 2d knowledge graph embeddings.", AAAI-18.
[7] Liwei Cai and William Yang Wang. “Kbgan: Adversarial learning for knowledge graph embeddings”. 

review_title: Official Blind Review #1

Meta-review: This paper proposes a noise-aware knowledge graph embedding (NoiGAN) by combining KG completion and noise detection through the GANs framework. The reviewers find that the idea is interesting, but the comparison to SOTA is largely missing. The paper can be improved by addressing the reviewer comments. 
Decision: Reject