Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: By probing pre-trained language models with prompts, the paper presents a novel and practical framework for automatically constructing knowledge graphs. The framework's input consists of relation definitions and a small set of seed entity pairs. For each relation, the framework generates and then paraphrases prompts. The extracted entity pairs of each relation are then evaluated using the proposed consistency score. Finally, it reranks these pairs and outputs the top-K results. Experiments show that the resulting KG is diverse and accurate. The proposed automatic prompt generation method is also effective. The resulting KG can serve as a symbolic interpretation
of pre-trained language models.
strength_and_weaknesses: Pros:

* The work's most significant strength is its practical pipeline for automatically constructing knowledge graphs (KGs) from pre-trained language models (LMs). Previous research on "LMs as KGs" or "knowledge probing from LMs" has primarily focused on prompt generation and does not cover the entire KG construction pipeline.

Cons:

* One potential shortcoming of the work is that its technical contribution may not be sufficient for a research paper.

* The paper also has a few minor flaws, and the writing could be improved, as listed below:

Minor issues:

* The multi-token issue in entity names. The authors state in Sect. 3.2 that "Each entity consists of one or more tokens". How to resolve the multi-token issue in the likelihood computation? Or, for example, how to compute P_{LM}(t|p,h) if the name of entity t has more than one tokens? In Appx. A.5, the authors state that "we enumerate the number of masks for every entity". Is there a maximum value for such a number? Is it a hyper-parameter? 

* Sect. 3.2 says that "The use of the minimum individual log-likelihoods allows us to apply rich thresholding and pruning strategies (e.g., thresholding on log PLM(h|p) for proposing the head entities), as described more in the appendix." However, it seems that there is no related content in the appendix. Or, is it Appx. A.5?

* It would be preferable to include a subsection in Section 4.2 that simply introduces the baselines used in the experiment.

* In Sect. 4.2, the authors state that "we restrict every entity to appear no more than 10 times to improve the diversity of generated knowledge". This, in my opinion, is not reasonable because it would result in a sparse KG. Table 2 also demonstrates this problem.

Typos:

* "Those systems, however, often involves a complex set of components" -> Those systems, however, often involve a complex set of components.


clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clear, but the writing could be improved. The authors promise to release the code as well as the resulting KG.
summary_of_the_review: Overall, I think the paper has both advantages (a practical KG construction pipeline) and disadvantages (some small issues). It can be further improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a pre-trained LMs based KG construction model aiming at identifying new entities for a single query relation. The proposed model is a bootstrapping strategy that starts from a query prompt with a few seed entity pairs and iteratively generates new pairs by means of new prompts generated by a GPT-based paraphrase model. Further, an LM-based reweight function is employed to weigh the new prompts and new pairs. The experiments conducted on crowdsourcing-based evaluation and case study to show the model is competitive to SOTA baselines. 
strength_and_weaknesses: Strength:
1. The problem of automatically constructing KG from pre-trained LMs is well-motivated.
2. It is interesting to combine the iterative KG construction into the current pre-trained LMs fashion.

Weakness:
1. The framework of iterative KG construction (i.e., bootstrapping) is not novel.
2. Some important technical details, e.g., how to generate multi-token entities and how to reduce the candidate set, are missing.
3. The experiment is not comprehensive. There is no objective evaluation.

clarity,_quality,_novelty_and_reproducibility: Clarity: Some important technical details are missing, e.g., efficiently generating candidate entities. Please refer to the Summary of the review Q3.

Quality: The model is only evaluated by crowdsourcing, with no objective evaluation. The effectiveness is not significant compared with SOTA baselines.

Novelty: The idea is generally interesting, the authors bring the old wisdom of iterative KG construction into the pre-trained LM fashion. But the core technical contribution is just an LM-based reweighting. Methodological nvoelty is a shortage.

Reproducibility: Unable to evaluate, since the technical details are unclear.

summary_of_the_review: 1. The proposed framework is a kind of bootstrapping-based knowledge graph construction approach, which, reminisce of the Hearst patterns for hypernym relation. My major concern is how to effectively handle the semantic drift, i.e., the semantics of new patterns and instance harvest may change drastically along with iteration or paraphrasing. But authors did not discuss this problem in-depth or provide a method with a plausible bound. What we have in the model is the LM-based paraphrasing and weighting, however, the key point is, LM does not necessarily guarantee the semantics.

2. For the diversity of paraphrase (second paragraph of Section 3.1), it is not clear (1) why to use a textual measurement (i.e., edit distance) to ensure diversity of paraphrases (second paragraph of Section 3.1). It is a bit weird that the whole paper is deep learning based, except for the paraphrase generation, which adopts an old-fashioned textual similarity ED, which, is obviously out of the deep learning fashion. (2) how this ED constraint works. As ED works for a text pair, do all generated prompt pairs need to keep a certain distance?

3. Some important technical details are missing:
  (1) Efficient generation candidate entity pair generation of Section 3.2. Appendix A.5 has a few sentences talking about "mask-filling" but I cannot find the details of how to generate multi-token entities and how to use "thresholding and pruning" to reduce the candidate set.
  (2) For computing Eq.1, it is not clear (a) how to generate the query text. Is the query text formed by inserting a specific entity (e.g., insert h into p for P(h|p))? If so, what is the difference between Plm(t|p,h) and Plm(h,t|p)? (b) how to compute the probability of Plm(h,t|p), Plm(h|p), and Plm(t|p,h). Is the probability computed by the generative probability of the LM?


4. In addition to the manual evaluation (crowdsourcing rating), the objective evaluation based on evaluation datasets should be given. For example, the authors can select datasets adopted by relevant tasks, e.g., knowledge graph completion, such as WRNN, FB15K-237, etc. for evaluation.

5. The baseline methods should be introduced in a few sentences. 

6. The layout of Table 2 should be fixed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors proposed a method to extract knowledge graph relations from a pretrained LM through automatic prompt creation and leveraging the pretrained LM to score the candidate entity pairs. Compared to previous works that usually rely on human annotated data or existing massive KGs, the authors' approach requires only the minimal definition of relations as inputs.
strength_and_weaknesses: Strength:
The proposed idea is simple and seems straightforward to implement. The proposed method needs minimal human input (only an initial prompt and 5 candidate entity pairs for each relation). It effectively leverages GPT3 to rephrase and generate better alternative prompts used for extracting new entity pairs. Overall, the paper is well written with detailed examples to illustrate the main idea.

Weaknesses:
Although the model shows better results than the baseline (COMET), it still lacks more analysis in the evaluation to convince the effectiveness of the proposed model. 

First, the result in Table 2 shows the precision of the prediction but not the recall. For example, the recall can be estimated on the known set of factual relations. The P-R curves in Figures 3 and 4 seem to partly answer this question. However, looking at the 0.8 level of precision, the recall is only less than 0.2, which raises another question about the overall effectiveness of the extraction method.

The baseline WebChild in Table 2 seems to extract a much higher number of tuples at 4.6M, while the proposed model ROBERTANET only extract 6741 tuples. It can be WebChild having much higher coverage, but this difference was not discussed in the paper.

Second, the quality of the extraction heavily depends on the compatibility scoring function (Eq. 1). It would be better to evaluate more on the design choice of this formula and explain why the proposed formula is optional for this purpose.

Third, Looking at the extracted relations, it looks like some relations are more non-trivial than others. It would be useful to analyze more on the factual relations (e.g., the relations between named entities such as 'ceo_of', 'invent') because these relations will be more useful for downstream applications like QnA. The LAMA dataset has more of these relations so authors can extract and analyze more case studies from this dataset.

Another suggestion, in the GPT3 prompt: "paraphrase: {sentence}" used to extend the prompt set, would it be better to include example of all 5 seed entity pairs into the prompt to help eliminating the ambiguity?

clarity,_quality,_novelty_and_reproducibility: N/A
summary_of_the_review: Overall, the idea proposed in this work is simple, easy to implement and it shows promising results compared to the baselines. However, the evaluation currently lacks analysis about the coverage and quality of the extracted relations.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a method to extract knowledge from language models such as BERT. Their approach establishes a set of paraphrase prompts representing a relation. Once this set of prompts is found, they search the space of entity pairs that fit this prompt to relational tuples representing knowledge from these models. This method enables relational knowledge to be extracted from language models at scale.


**Strengths:**
Reviewers all thought that the idea of extracting knowledge graphs from language models was neat.

**Weaknesses:**
In general, the reviewers felt the method lacked conceptual novelty and weren’t convinced by the effectiveness of the proposed method given the low evaluation results. Regarding novelty, I’m inclined to disagree with the reviewers as I do think there’s novelty to the idea. While the method is not completely unique (as the work of West et al., 2022 also looks at few-shot knowledge generation from LMs), it has enough differences. 

Regarding evaluations, though, I’m inclined to agree with the sentiment of the reviewers that the evaluations fall short of convincing of the value of the task. While the authors respond to this criticism that their method achieved better results than the baselines they set up, these methods are designed for different task instantiations of the “generate knowledge from LMs" problem. Consequently, even though the BertNet method improves over these methods, it’s not clear they validate the task instantiation in the first place for automatic KG construction (as opposed to other methods for automatic knowledge construction, such as retrieval). I think the authors need evaluations that validate their task instantiation, rather than assuming the premise that KGs should be generated from LMs the "BertNet" way the first place.

justification_for_why_not_higher_score: See meta-review
Decision: Reject