Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper aims to extract relational facts from pre-trained language models. Instead of tuning the prompts at the input layer of the language model, the paper investigates fine-tuning the model on a set of training facts (<subject, relation, object> triples) for each relation. The results show that fine-tuning the pre-trained language model can achieve higher accuracy on the LAMA benchmark compared to prompt-tuning approaches. The paper also studies the transfer learning capabilities of the pre-trained language models for knowledge extraction and provides interesting insights.

Strengths:
* The paper studies extracting knowledge from pre-trained language models which are trained on unlabeled text. The problem seems promising with the development of modern large-scale language models and it fits well with the scope of AKBC.
* The paper is well written and easy to understand.
* The empirical results on LAMA are strong and significantly outperform previous approaches.
* Transfer learning experiments lead interesting discussions and insights.

Weaknesses:
* The main weakness of this paper is that the goal of this paper is unclear to me. The approach is only evaluated on the LAMA benchmark, but LAMA is originally to probe how much knowledge is stored in the pre-trained language models, where the parameters of the models are frozen. In this paper, the model parameters are updated. If the paper targets to better estimate how much knowledge is stored in the language models, it lacks discussion on whether the results can reflect how much knowledge the model has stored in parameters (because the parameters have been updated). If the paper aims to build a neural knowledge base, the scope of the LAMA benchmark can be limited -- it only contains single-token objects and 41 relations.
* The paper lacks qualitative analyses of the experiments. It would be insightful if the paper can include some concrete examples when showing the experimental results. For example, in transfer learning experiments’ group (c), the “omitted” performance can be much lower than the original BERT. Some examples might help understand what’s happening there.

review_title: The paper fine-tunes the model directly (instead of performing prompt tuning) and shows strong results on the LAMA benchmark.

final_score: rating: 7: Good paper, accept
review_content: This paper presents the BERTirple model for the cloze-style fact extraction task. Instead of complex additional prompt tuning methods, they show that an adaptive fine-tuning of the pre-trained language model can outperform the best state-of-the-art baselines on the LAMA and LAMA-UHN probes. They propose to use the triples of a knowledge graph in the form of masked sentences to alleviate the difficulties of manually designing prompts. This new data representation leads to better performance, where the proposed model outperforms the previous manually created, mining-based and learning-based prompts. Finally, the experiments show that by omitting some relations during training, the performance doesn't degrade, which validates the transfer learning capabilities of the proposed model.

Strengths:
- The paper is mainly well-written, and the empirical results are promising.
- Their approach doesn’t need to tune the prompts and the idea of using the triples as masked sentences is simple and efficient.
- They show that the proposed model requires fewer training data to outperform the previous SOTA model.
- The discussion about transfer learning experiments reported in the paper is compressive and informative.

Weaknesses:

- In Figure (1), it's better to repeat the experiments (for each sample size) at least 3 times and report the mean and variance of each experiment. Since the paper claims that “by using significantly fewer training facts, BERTriple outperforms all baselines”, the mean and variance of P@1 are required to see the stability of the model.
- Regarding the training speed, authors mention that “existing techniques usually need a significant amount of training data in the form of existing knowledge graph triples and a large amount of training time to optimize prompts using complex additional models” also “ with fewer training data required and without the need for an additional model. Consequently, the fine-tuning is very fast.” Since the training speed is another contribution of the paper, it’s much better to quantify the training speed (and/or fine-tuning) and compare the training time between mining-based, learning-based, and BERTirple.
- It’s a little hard to follow the reported results in Table 2 without the name of each relation. Please add each relation’s name (label) in the final version. 




review_title: Adaptive fine-tuning of pre-trained masked language models for knowledge extraction using masked triple prompts

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a simple but effective method for the task of "fact retrieval" from pretrained language models. The method requires a training set of  (s,r,o ) triples, which must have human-readable labels, such as wkikdata triples. The triples are treated as pseudo-natural language sentences e.g.  (Dante, born in ,Florence) -> "Dante born in Florence", and the pre-trained language model is given additional MLM finetuning on these training data. 
(they also investigate a setting which uses the LAMA templates rather than just directly using triples). 

The fine-tuned model is then queried for a fact by providing the subject and relation, and replacing the object with a [mask] token, and then extracting the highest scoring predicted token as the object prediction, similar to cloze-question answering, à la LAMA.

The authors evaluate their approach on LAMA, and show improved P@1 scores, comparing to prompting approaches - both the simple manual approach from the original LAMA probe, and more recent approaches that try to optimize the prompt. 
The authors then assess how much training data is required for their approach to be effective, and conclude that about 50 examples per relation is sufficient for the approach to work well.

They also assess to what extent finetuning on triples can improve predictions for unseen relations. They do this by holding out a relation, fine-tuning a model on all the other relations ,and then evaluating the LAMA performance for that relation, and observe that some relations improve relative to the original LAMA probe zero-shot scores, but most are either around the same performance as the original LAMA probe scores or perform very poorly.

The authors conclude with a thoughtful discussion.   
 
This work resembles recent work in "soft prompting", in the sense that explicit prompts are avoided and fine-tuning is used instead, e..g https://arxiv.org/pdf/2101.00190.pdf, https://arxiv.org/abs/2104.08691 and  https://arxiv.org/abs/2103.10385 - the latter paper also experimenting with LAMA and achieving almost the same P@1 for bert-base. Unlike these works however, this model fine-tunes the whole language model, rather than learning only "continuous prompt". 

The paper feels similar in spirit to COMET, which also trains a transformer essentially directly on facts from a knowlegebase, although in the case of COMET, its data from commonsense KBs.

Strengths

* the paper is very clear, easy to read and well laid out. The ideas are well-motivated and argued The related work section is a particularly good summary of the important relevant work.
* A directly-finetuned baseline like this is important for contextualizing and understanding other works on this dataset.
* The experiments are thoughtful and well executed
* I particularly like the discussion and future work section, acknowledging the strengths and limitations of this work.
* this approach eliminates the need for manually-written prompts for relations, which is a great strength
* the training process is fast and computationally fairly cheap, and the number of examples needed for the method to work well are quite small
* the authors use both LAMA and its harder subset LAMA-UHN

Weaknesses

* unlike prompting work, this directly fine-tunes the model, which specializes it, and reduces its ability for zero-shot tasks - this is quite a fundamental difference. Whereas other approaches try to fit prompts/queries to the model, this approach changes the model itself - as such, it is unclear wether this is appropriate for a probing dataset like LAMA. The authors are aware, and touch on this in their discussion. 
* Only one base model (BERT) is used, and only one dataset is experimented on (LAMA, and its subset LAMA-UHN) - the paper could be improved it it assessed how effective the method was for other models, such as GPT-style models or encoder-decoder models like T5 or BART
* Whilst this approach doesn't require manually written prompt templates for relations, it probably does rely on human-readable relation labels to work well in the low data regime - e..g "born in" rather than a human unreadable label  (e.g. P19) "Q23243". An interesting experiment would be to see how important these are, and wether the method would work with unique but non-human-descriptive relation labels. 
 
Overall, I think this is a well-presented, well written paper giving useful contribution/baseline methods for people interested in fact-retrieval, although i have qualms about treating LAMA as as "task", to be trained an optimized for, rather than a "probe", used for analyzing knowledge already present in LMs.

review_title: Simple but effective method for the task of "fact retrieval" from pre-trained language models, despite modifying the model directly.

Meta-review: This paper proposes a simple method for extracting relational information out of language models: fine-tune the model to fill in masked out entity names, without any prompting (and hence without any prompt tuning). This outperforms baselines that do have to rely on prompt tuning. The reviewers agreed that this paper is strong and well-written.
Decision: Accept