Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Authors did an extensive experimental study over neural link prediction architectures that was never done before, in such a systematic way, by other works in this space. Their findings suggest that some hyperparameters, such as the loss being used, can provide substantial improvements to some models, and can be the reason of the significant improvements in neural link prediction accuracy the community observed in recent months.

This is a really interesting paper, and can really shine some light on what was going on in neural link prediction over recent years. It also provides a great overview of the field -- in terms of architectures, loss functions, regularizers, sampling strategies, data augmentation strategies etc. -- that is really needed right now in the field.

One concern I have is that the hyperparameter tuning strategy is not really described -- authors just say something along the lines of "we use av.dev", but for those unfamiliar with this specific hyperparameter optimiser this does not provide much information (e.g. what is a Sobol sequence? I had to look it up).
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary
========
The paper conducts a thorough analysis of existing models for constructing knowledge graph embeddings. It focuses on attempting to remove confounding aspects of model features and training regime, in order to better assess the merits of KGE models. The paper describes the reimplementation of five different KGE models, re-trained with a common training framework which conducts hyperparameter exploration. The results show surprising insights, e.g., demonstrating that a system from 2011, despite being the earliest of the KGE models analyzed, demonstrates competitive results over a more recent (2017) published model.

Overall Comments
===============
The paper, and the described software release specifically, represent a solid contribution to the area of knowledge graph embeddings. I agree with the basic premise of this paper’s analysis: in order to accelerate research in a maturing field (like knowledge graphs), it is important to be able to properly compare with older systems, removing artifacts that are due to general improvements in training and optimization techniques, from modeling specific changes. The report of the strong results from the RESCAL system, along with others, drive the point through. Furthermore, the paper is well-written and easy to follow, and should become a good reference for future works on KGEs.

Detailed comments
===============
Below are some detailed comments about specific parts of the paper, in order of importance:

1. The paper mentions disregarding “monolithic” models in the current analysis, primarily due to the expensive training of these models. It may, however, be the case that the future state-of-the-art models will be larger and slower to train (and, perhaps, of the monolithic type). Are there any limitations to the proposed experimental framework that would prevent running monolithic/large models?

2. Regarding the item above, if one were to look at the training curves for the exploration of the current 5 KGE models, is it possible that verify winning hyperparameter configurations earlier than the full training is complete. In my experience, it is often the case that with fewer than 1/10th steps of full training (well before convergence), it is possible to compare model configurations (relatively). For example, “Population-base training” (https://arxiv.org/abs/1711.09846, https://arxiv.org/abs/1902.01894) is one framework where fewer training steps are used to quickly learn good hyperparameter configurations. I’m wondering whether the KGE hyperparameter exploration training curves display similar early trends. Could a shortened training procedure produce sufficient information for learning good parameters, and potentially deal with larger/slower models?  In addition: would adopting population-based training be applicable to the proposed framework?

3. In Section 3.2, “Limitations”, there is a surprising comment that performance can be improved with further hyperparameter tuning. It is not clear how the authors found the configurations that produced the improved results. It would be helpful to clarify why the hyperparameter exploration proposed in the paper did not discover these improved configurations. Were the improved configurations outside of the range of considered values? Or would the exploration require more points to find the improved configuration?

4. In Section 3.3 “Best configurations (quasi-random search)”, specifically Table 3, the paper presents an ablation of independent hyperparameters, over the best configuration for each of the 5 models. This is a very interesting section. One further suggestion, however, is whether the paper could include the performance of each of the models on the _average_ best configuration. Although the paper describes losses for switching individual parameters to their second best values, it is unlikely that the losses are cumulative. So, for example, if we can take the average/majority best value for each parameter (embedding size = 512, batch size = 1024, training type = 1vsall, loss = CE, etc.), and collect results for that configuration. I think it would be interesting to know the difference between a model trained on a “collectively known good” set of parameters vs. a model and task specific tuned set of parameters.

5. In Section 2, “Evaluation”, HITS@k is not formally defined. Unfortunately, I have encountered slight variants of this metrics (e.g: (1) given a SINGLE correct label, HITS@k is the average rate of the label being present in the top k scored results, or (2) given ALL possible correct labels, HITS@k is the percentage of correct labels present within the top k scored results, etc.). It would be nice to precisely describe HITS@k in this work.

6. Caption for Table 2 does not contain a description for the “Recent” super-column.

7. In Section 3.3 “Best configuration (quasi-random search)” Space missing at “... Tables 6 and 7(in …”, between 7 and (.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper presents an experimental study about some KGE methods. It argues that papers often propose changes in several different dimensions, such as model, loss, training, regularizer, etc., at once without providing a sufficient investigation about the individual components' contributions. The experimental study considers two datasets (FB15k-237 and WNRR) and five different models (RESCAL, TransE, DistMult, ComplEx, ConvE). The models were selected using a quasi-random hyperparameter search, followed by a short Bayesian optimization phase to fine-tune the parameters. The performance of the best models found during this hyperparameter search are compared to first published results for the same model, as well as to a small selection of recent papers. To analyse the influence of single hyperparameters, the best found configuration is compared to the best configuration which does not use this specific value for the given hyperparameter.

Overall, the paper adresses an important problem, as papers about new KGE methods often lack a clear separation of the individual changes' contribution. The experimental results show that older, simpler can compete with recently proposed models when trained properly. The intra-model comparison lacks statistical rigorousity, yet hints a few directions to further explore.

The experiments are based on a quasi-random hyperparameter search. While it is necessary for efficient exploration of larger search spaces [1], and should be the standard methodology for hyperparameter search of a new method, the interpretability of the comparison of two runs suffers. For the comparison between the trained models and previously published results, the sample size might be sufficient to draw the conclusions. However, the intra-model comparison, e.g. in Figure 2, are now comparing subsets of the runs which only comprise approx. (200/6) runs. Furthermore, the influence of random initialization is not accounted for. Another place where this can be witnessed is Table 3. Here, for some ablations, e.g. TransE + Reciprocal, no reduction is given. If I understood it correctly, this is due to not having a configuration which uses TransE and reciprocal relations. Also for the other ablations, it is unclear how statistically significant the reduction is.


Further Comments:
1. Please add the best published results for a specific model-dataset combination to table 2.
2. Do the plots in Figure 1 include the runs which were stopped after 50 epochs due to insufficient MRR?
3. Could you elaborate on the combination of KvsAll and CE?
4. The combination of subject and object triple scores has for instance been used in SimplE [2].


[1] Bergstra, James, and Yoshua Bengio. "Random search for hyper-parameter optimization." Journal of Machine Learning Research 13.Feb (2012): 281-305.
[2] Kazemi, Seyed Mehran, and David Poole. "Simple embedding for link prediction in knowledge graphs." Advances in Neural Information Processing Systems. 2018.

review_title: Official Blind Review #3

Meta-review: The authors analyze knowledge graph embedding models for multi-relational link predictions. Three reviewers like the work and recommend acceptance. The paper further received several positive comments from the public. This is solid work and should be accepted.
Decision: Accept (Poster)