Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a thorough empirical investigation of the performance of knowledge graph embedding models on a variety of graph structure prediction tasks, including link prediction, relation prediction, domain prediction, and neighborhood prediction. The results show that these models do not perform well on the tasks except link prediction. The authors go on to propose and test multi-task learning to solve this problem. Multi-task learning can also help with KG downstream tasks like entity classification and regression.
strength_and_weaknesses: This paper presents some "interesting" findings regarding KG embedding models and applications. It would motivate us to reconsider the suitability and generalizability of KG embedding models. However, I have the following concerns.

First, as a KG embedding expert, the finding does not surprise me that a model trained on the link prediction task performs poorly on other structure prediction tasks. There is a significant gap between the training and evaluation tasks in this experiment. It is also not surprising that multi-task learning can improve the model's performance on some structure prediction tasks, because, just as stated by the authors in Section 4.2, "these tasks have been introduced as auxiliary training objectives". So, I think this paper is not so attractive to the KG embedding researchers, although I appreciate the authors' hard work. It lacks in-depth analysis of the reasons behind the performance.

Second, the selected four KG embedding models in the empirical study, i.e., TransE, DistMult, ComplEx, and RotatE, are old. I think these four models could not adequately represent the large and fast-growing family of KG embedding models. In Section 4.1, the authors state that ComplEx is the SOTA factorization model. But in my view, the SOTA factorization-based KG embedding model is TuckER [1]. Also, some recent GNN-based [2] or Transformer-based KG embedding models are ignored in this study. So, a natural question is, whether the recent KG embedding models still suffer from the claimed issue in this paper.

[1] Ivana Balazevic, Carl Allen, and Timothy Hospedales. 2019. TuckER: Tensor Factorization for Knowledge Graph Completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5185–5194, Hong Kong, China. Association for Computational Linguistics.

[2] Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal A. C. Xhonneux, Jian Tang: Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. NeurIPS 2021: 29476-29490

[3] Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, and Yangfeng Ji. 2021. HittER: Hierarchical Transformers for Knowledge Graph Embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10395–10407, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
clarity,_quality,_novelty_and_reproducibility: Overall, the work is of acceptable quality and originality. But, there is still room for improvement in writing:

* "Knowledge graph embeddings (KGE) models provide low-dimensional representations of the entities and relations in a knowledge graph (KG)." -> Knowledge graph embedding (KGE) models provide low-dimensional representations of entities and relations.

* "..., STD training approach performs poorly on all graph-structure tasks" -> ..., the STD training approach performs poorly on all graph-structure tasks

* "The performance for these tasks is significantly increased using MTT training." -> The performance of these tasks is significantly improved by using MTT training.

summary_of_the_review: Overall, this paper sheds some light on both KG embedding models and their applications. However, most of the findings, in my opinion, are common knowledge in the related field. Some recent representative KG embedding models and in-depth analysis of the reasons for the performance are lacking in the empirical study. So, I think the contribution of the paper is not enough.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to explore the suitability of various Knowledge Graph Embeddings (KGE) for graph structure prediction tasks such as entity and relation neighborhood prediction, type/range of a given relation, and relation prediction. This text also investigates the performance of these models on several downstream tasks. Building upon these two investigations, the authors then propose a Multi-Task Training (MTT) objective to potentially improve the performance on graph structure prediction tasks and downstream applications without further task-specific training. 
strength_and_weaknesses: Strengths

+ Motivation for the work is well-founded in literature and concretely defined in the work. 
+ Authors introduce several new basic graph structure prediction tasks (Neighborhood and Domain prediction). They also evaluate existing models on these tasks showing the poor generalizability of current KGE methods to such tasks.
+ Further, this work proposes an MTT objective that includes the well-known Link and Relation Prediction tasks in conjunction with other graph structure prediction tasks introduced in this work.
+ Evaluation shows that MTT training does help improve the existing KGE model's performance on graph structure prediction tasks and downstream tasks. 

Weaknesses / Questions

+ Authors should provide an ablation study in order to better understand the role of each task introduced in MTT. 
+ In my opinion, this method loses the Efficiency vs Performance tradeoff. It takes 2x-4x (worst) more time to train but seems to marginal gain in performance on downstream tasks.
+ It would have been interesting to see the results of KE-GCN on basic graph structure prediction tasks as well (treating these tasks as downstream and training on them). 
+ Instead of YAGO3-10, why didn't the authors use one of the [OGB](https://ogb.stanford.edu/docs/linkprop/) datasets for large-scale experiments? I would like to know if this is a conscious design choice by the authors.
+ "We also use the so-obtained wildcard embeddings for prediction in the same fashion; e.g., we set $$s(s, r^{'}, *) = s(s, r^{'}, any_{O})$$". I am not sure which prediction the authors are talking about here. 
+ Please provide the code used for all the experiments. 
clarity,_quality,_novelty_and_reproducibility: This paper is well written and the motivation of this paper is clear. While there are some minor weaknesses in the work, I feel that the research problem is well-founded and the authors have done a good job overall. 
Authors use a well-known KGE framework for implementing the experiments in this paper but do not provide their code, so reproducibility takes a hit. 

summary_of_the_review: Overall, I feel this is a good piece of work with a well-known and well-founded research problem. Authors have done extensive evaluation (with some minor weaknesses) to explore the use of Multi-Task Training objectives to improve the generalizability of Knowledge Graph Embeddings methods. To that extent, I vote to accept the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a multi-task training strategy for improving the performance of the knowledge graph embedding (KGE) model in downstream tasks, e.g., entity classification or regression. In addition to the conventional link prediction training of KGE, the proposed model is pre-trained with 4 extra tasks, e.g., relation prediction, entity neighboring, etc. The experiments on FB15K-237, YAGO3-10, and WRNN show the multi-task trained model achieves better MRR on the pretraining tasks but lags behind LP than the single training (as expected). The performance on downstream tasks are marginally better for certain KGE model, e.g., RotatE on FB15K-237. Overall, the performance of graph-structure prediction tasks (e.g., NBE) is not stunning as dedicatedly trained. The performance of downstream tasks is also limited.
strength_and_weaknesses: Strength:
1.	Some observations are interesting, e.g., KGE performs poorly on graph-structure prediction tasks other than link prediction. 
2.	The paper is clearly written. 

Weakness:
1.	The results are not stunning. The multi-task training does not lead to consistently better performance on all graph-structure prediction tasks. The performance improvement on downstream tasks is not effective. 
2.	Some experimental settings are not rigorously designed. E.g., MTT should test on unseen tasks. The experimental results could be further discussed or explained, i.e., the variation of model behavior under different datasets/settings.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the representation is good. However, the model behavior should be further discussed or explained.
This paper contributes little technical novelty. Albeit some observations are interesting, the proposed method does not show enough effectiveness. 


summary_of_the_review: 1.	The results of applying multi-task training are not stunning at all. The model trained on the single link prediction task (STD) still has the best performance on the link prediction task (Table 3). And the model trained on extra tasks (MTT) no doubt performs well on the extra tasks. I would suggest authors train models ablative, i.e., for the unseen test tasks. In addition, the performance improvements on downstream tasks are also limited. E.g., On the FB15K-237 dataset, the STD strategy excels the proposed MTT for nearly all settings of EC (except for RotatE). In this regard, the effectiveness of the proposed multi-task training strategy is nor promising.

2.	The domain and range predictions seem not widely used tasks for KGE. Are there any reasons for choosing these two tasks? E.g., do they have any potential for benefiting downstream applications?

3.	As the SOTA performances of KGC are achieved by pretrained LM-based models, such as KG-BERT (Yao et al., 2019), KGT5 (Saxena et al., ACL’22), and KG-S2S (Chen, et al., COLING’22). The applicability of the proposed MTT strategy should be further discussed.

4.	It is not clear how the wildcard entities (MTT part) approach the max-aggregation (MTR part).  The max-aggregation search for the best relation over entire entity set, while the wildcard entities make use of an abstract "wildcard" embedding to get the result. Although using the wildcard entity would be easier as it reduces the computational cost for searching, but I believe the two methods The methodologies for the two methods are conceptionally different and may have different results.

5.	Why STD has better results than the dedicated MTT on NBE task (0.21 v.s. 0.152, Table 3) needs further explanation. 

Minor comments:
(1) Authors should give the description of abbreviation STD (i.e., the model trained with single task, i.e., LP.) when it is first introduced in Section 4.1.
(2) .. replace wildcards by their the corresponding .. -> remove the (page 5)
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper makes one key observation that the KGE models trained only for link prediction are unsuitable for other related tasks. So, the paper proposes some other tasks and trains embeddings in multi-task framework, to show overall improvement in all tasks. 

All reviewers appreciate the key insight. However, none of the reviewers feel that the paper has any wow effect. The downstream path from the observation of training multi-task is expected and obvious. And the results in many cases show marginal improvements. This reduces excitement on the paper.

There is also the added issue that the authors have very substantially revised the experimental section and added new experiments, etc. However, I feel that the additions to the original paper should be limited, and if one makes substantial changes then the revised version should go through another round of full review. In this paper's case, unfortunately, none of the reviewers responded to the improvements, and so we cannot be sure how to judge the revised paper.

In light of that, I suggest that the paper be rejected for this iteration. One, because the results are not entirely surprising, which limits the overall contribution of the work. And two, because the new changes probably need another round of review.

I also have some thoughts on 1-2 of the points raised by reviewers. Note that these are my individual thoughts, and should be used as just one other person's opinion (may helpful in preparing your next draft).
1) did you use a good set of models to start with? I'd personally say that the use of ComplEx and RotatE makes your models near state of the art. I checked the Tucker paper and noted that Complex's old results are used there and better implementations are not used. You may consider justifying your choice of models by the following two references (worth reading). Plus, you may limit your study to "tensor factorization" style models to avoid bringing some of the more recent models, like neural bellman ford and rnnlogic etc. into consideration.
- https://madoc.bib.uni-mannheim.de/54954/1/you_can_teach_an_old_dog_new_tricks_on_training_knowledge_graph_embeddings.pdf
- https://arxiv.org/abs/2005.00804

2) you say in your response that additional improvements may be possible with more study. I believe the onus is on you to convince the reviewers... maybe it makes sense to achieve more than marginal improvements, and increase the strength of the contributions before resubmitting the paper.

 
justification_for_why_not_higher_score: overall contributions are felt limited, unsurprising, so no reviewer championed the paper
Decision: Reject