Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a temporal KG-enhanced language model. 

The idea is to combine pretrained temporal knowledge embeddings with the language model as input tokens, and the language model will take temporal-aware embeddings as input. 
The authors propose masking the predicates, entities, or subwords for pretraining the model. 
The authors convert three KGs into the pretraining corpus to find relevant text with temporal KG knowledge. Experiments show that the pretrained model has improved performance on link prediction for the three temporal KGs.
strength_and_weaknesses: Strength: 

The paper fills in the gap on using temporal KG to enhance the language model. The results look solid and bring consistent improvement in link prediction.

Weakness: 
1. The method in the paper lacks novelty. It basically reformulates the previous KG+LM methods to the temporal KG+LM setting. Actually, the method (and even the figures) looks a lot like the KEPLER paper (Wang et al., 2019), with the KG embeddings replaced by tKG embeddings, and the transE loss replaced by the corresponding tKG loss.
2. The method is only tested on link prediction tasks. I would expect this model to work in more diverse settings (i.e., any task that LM can be used for). It would be great if the model could work for settings like question answering, event extraction, etc. I'm also interested in how the tKG-pretraining affects general NLU performance like on GLUE.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to understand. As mentioned above, I think it lacks novelty. Code and data are included with the submission.

A question: What are b_i and b_j in (1)? I did not find the definition in the paper.
summary_of_the_review: In all, I feel the paper focuses on an important area, but the novelty and contribution are not enough for a conference like ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to use contextualized language representation to enhance the temporal knowledge graph embedding. It combines tKGE losses with knowledge-text-prediction loss during training. As such, the proposed method can be combined with arbitrary tKGE methods. Experimental results showed significant improvements in various temporal knowledge graph completion tasks.
strength_and_weaknesses: Strength: This paper proposed a technique to enhance the temporal knowledge graph embeddings by contextualized language models. The motivation is reasonable and experiment results have demonstrated decent improvements for the temporal KG completion tasks.

Weakness: 
1. Enhancing knowledge graph embeddings by contextualized language model embedding itself has been widely explored, even for enhancing temporal knowledge graph embedding [1]. 

2. It is unclear how the transformer encoder (BERT) combines with the tKG model in Figure 2. From section 3.1, we know the quadruple representation can be obtained by the BERT encoder. However, the introduction of DyERNIE is confusing. Since the embedding of DyERNIE is derived from a static embedding and a velocity vector, how does this equation link to the notations in section 3.1? The title of section 3.1 is ‘Embedding layer’, whereas the final representation of BERT should be dynamic contextualized embedding since the BERT encoder is updated by the KTP (MLM-like) loss. In figure 3, the time expression 2019.08.02 is an input to the transformer, but it seems to be treated differently compared to the other words. The notation tau mentioned in figure 3 was not explained elsewhere, whereas tau is related to the most important temporal aspect of this model. 


[1] Mavromatis, Costas, et al. "Tempoqr: temporal question reasoning over knowledge graphs." Proceedings of AAAI. 2022.


clarity,_quality,_novelty_and_reproducibility: Question: as mentioned in the section 3 introduction, no textual data is used during inference time, does it mean that the knowledge triples still need to pass through the transformer encoder? Or only the tKG model is used during inference. 


In terms of novelty, incorporating contextualized embedding into the temporal knowledge graph completion task has been proposed. The main modification of ECOLA is to use an additional knowledge-text prediction task to jointly optimize the pre-trained language models. However, it is also possible that the performance gain is due to the encoder’s capability from the pre-trained language model. It would be beneficial if the authors can show the experiments without KTP loss and give more insights into how the KTP task benefits the temporal knowledge graph completion task.

summary_of_the_review: In summary, this paper proposed a framework that enhances the temporal knowledge graph embedding with pre-trained language models and jointly optimizes the TKGC task with knowledge-text prediction loss (KTP). The proposed model has demonstrated decent experimental improvements in the temporal knowledge graph completion tasks.  However, this paper lacks sufficient details on the model structure. It is not clear how the pre-trained transformer model combines with tKG model. 


Besides, the analysis section of the paper can also be improved. For example, in the qualitative analysis part, the examples only showed that ECOLA-DE benefited from the extra relevant textual training data. Whereas this finding is somewhat superficial, and it is not directly related to the proposed KTP loss. Further analysis can be conducted on the underlying reasons behind ECOLA’s strong empirical performance.

==============================
Updates on Nov 30. 

I appreciate the clarification of the authors, which help me understand some parts better. The authors also mentioned that they will update their paper accordingly. Therefore, I made an adjustment to my recommendation score. Overall, the paper's technique contribution is not significant enough, though the proposed method for combining the two sources improved the performance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper mainly addresses the problem of learning the representations of temporal knowledge graphs (tKGs) which contain time-evolving relations between the subject and object. To alleviate the sparsity of the tKG and learn better embeddings, the authors propose to extend the MLM objective so that during training, the knowledge quadruples are paired with corresponding texts and some of the tokens or entities are masked. The model is trained to predict the mask tokens and the mask entities or relationships. This way the model can build latent connections between two embeddings spaces.
strength_and_weaknesses: Strength:

1. The paper is well-written and easy to follow. The motivation to consider time-evolving relations in KGs is clear. Leveraging the rich textual information to enhance the embeddings of knowledge graph is interesting.

2. Propose a simple yet effective method to improve the representations of tKGs and the experiment results show the ECOLA can improve embeddings from various embedding methods and achieve large improvement.

Weakness:

1. Novelty is the main concern. The main method ECOLA extends the MLM objective to predict both masked tokens and entities or relationships. The Model Variants are merely combinations of ECOLA with different previous knowledge embedding methods.

2. Not clear what benefits the language model gain after the joint training. The paper can be stronger if the author can show that the proposed ECOLA can benefit the language model as well.
clarity,_quality,_novelty_and_reproducibility: 1. The paper is clear and well-written.
2. The quality is good.
3. The empirical contribution is high as it significantly improves the performance of previous embedding methods.
4. The proposed ECOLA which jointly models the embedding space of text and knowledge graph has some aspects existing in previous work.
summary_of_the_review: Overall the paper is clear and well-motivated. It has strong empirical results and the results can support the claim that the proposed method is model-agnostic. Some analysis is conducted and provides some insights. Some of the remaining research questions need to be answered (see questions below).

Question:
1. What do you mean by “we solve the temporal alignment challenge by using tKG quadruples as an implicit measure. ” Can you be more specific with the term “implicit measure”?

2. Based on the results we can see that the ECOLA mainly leverages the language model to improve the KG embeddings. I wonder if the performance can be further improved if larger language models are leveraged since BERT base is relatively small.

3. Does the quality of the text significantly affect the model’s performance? If in another domain there is no high-quality paired text, can the proposed method still improve the KG embeddings?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - This paper studies temporal knowledge graph (tKG) embeddings and proposes a new method that leverages textual knowledge/language models to overcome the sparsity of KGs. The authors identify two challenges: aligning temporally relevant textual information with KG entities, and incorporating temporal information into joint text-KG modeling. The paper then proposes the ECOLA method that addresses these two challenges: it creates new datasets that align tKG and text using using tKG quadruples, and uses a knowledge-text prediction (KTP) task to combine textual knowledge and temporal knowledge quadruples from tKG. The proposed method outperforms existing tKG embedding methods that do not use textual information. 
strength_and_weaknesses: Strength
- The paper is generally well-written and easy to understand 
- The problem studied is well-motivated. Static KGs are often studied in existing works, but a lot of knowledge is inherently time-dependent, motivating temporal KG representations. The use of textual information for temporal KG is an important direction for coping with the sparsity of tKGs.
- The paper identifies clear technical challenges (alignment of text and KG tuples, and modeling of temporal information), and proposes reasonable solutions to them.
- Experiments are solid, conducted on three datasets, and show that the proposed method provides significant improvement in performance.

Weakness / suggestion for improvement
- The authors mention that at inference time, the method does not use textual descriptions for test quadruples, but it still achieves improved performance. I wonder if the authors could provide a bit more discussions about the intuition behind this? - e.g. Is this because the test entities were seen during training and the text information used during training already provided extra knowledge for these test entities? A related question is, would the proposed method work if an unseen entity is given at test time? I think it would be great if the authors could add these discussions to the paper.

Question
- Did the author also consider masking and predicting the temporal information in the KTP task? If not, might that be an interesting thing to try?

Typo
- Page 2 at “a novel knoweldge-text prediction (KTP) task”: “knoweldge” -> “knowledge”
clarity,_quality,_novelty_and_reproducibility: - Clarity - The writing and presentation are clear 
- Quality - The paper is well-executed
- Novelty - I think the novelty is sufficient. The method is a relatively intuitive extension of text-KG models to the temporal setting, but there is a sufficient contribution in designing a method such that it actually improves performance on temporal KG tasks. 
- Reproducibility - the authors promise to release data and code, which is great.
summary_of_the_review: Overall, the paper studies an important problem and develops a method that works well. There is some room for improvements, but I did not find critical weaknesses. Overall I think this paper is above acceptance threshold.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper presents a method for improving temporal knowledge graph embedding by using pre-trained language models and optimizing the TKGC task with a knowledge-text prediction loss. The proposed approach showed promising results in temporal knowledge graph completion tasks.

Strengths:
- All reviewers agree that the motivation of the method and the experimental results (improvements) are convincing.
- All reviewers agree that the paper is clearly written. Unclear parts were clarified after rebuttals.

Weaknesses:
- Most reviewers are concerned with the novelty; that is, the proposed method is not too different from previous methods (e.g. KG+LM).
justification_for_why_not_higher_score: This is clearly a borderline paper that does not have strong supports nor oppositions. While the paper's motivation and experimental results are solid, as most reviewers are concerned, the novelty of the approach is rather limited; the paper can be viewed as making changes to existing methods that can improve their performance in the given task. While these achievements should not be undervalued, considering the characteristics of ICLR and its audience, the paper's weaknesses seem to outweigh its strengths in this venue and might not have enough interested audience.
Decision: Reject