Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper compares pretrained LLMs' question-answer results for CSQA dataset and ConceptNet. The study is on comparing the QA performance between CSQA dataset and QA problems extracted from ConceptNet through a relevancy heuristic procedure.   The authors attempt to interpret the average precision metrices comparison of the two related QA datasets as conceptual consistency. 

strength_and_weaknesses: Strength
	• An attempt to evaluate LLMs  through a metaphor of conceptual consistency (it seems that the authors want to evaluate a form of reliability or trust by looking at what LLMs' "knows" but no rigorous definitions are given)
	• The definition of conceptual consistency itself is well-defined in the form 


Weakness
	• It seems that the authors want to evaluate a form of reliability or trust by looking at what LLMs' "knows" but no rigorous definitions are given. 
	• It is not clear that whether Conceptual consistency between LLMs' CSQA and the QA question-answers extracted from ConceptNet really reveals "what LLMs know". There are heuristic syntactic procedures to extract QA background facts from ConceptNet for CSQA question-answer pairs. The influence of heuristic procedures needs to be formally characterized and bounded for the "what LLMs know" conclusion to be rigorously valid.  There is a logical gap between "what LLMs know" and the "conceptual consistency definition" as it is now. 

clarity,_quality,_novelty_and_reproducibility: The writing is mostly clear. In terms of technical quality, there are missing links between what the paper attempt to conclude on and the indicator/proxy measures which this paper investigates for LLMs.  The idea to investigate what LLMs knows or how to determine their trustfulness regarding given tasks is inspiring but not complete yet.

summary_of_the_review: This paper investigates into an important problem of LLMs what LLMs knows or how to determine their trustfulness regarding given tasks are inspiring but not complete yet.  This paper attempts to understand what LLMs know by looking at the conceptual consistency between QA datasets and the underlying knowledge graph. However,  it is not clear that whether Conceptual consistency between LLMs' CSQA and the QA question-answers extracted from ConceptNet really reveals "what LLMs know" thus reveals whether LLMs are trustworthy for given tasks. 

Details
	• Page 4, please investigate formal means to evaluate whether the extracting background facts are truly background facts corresponding to the queries or answers. This might be a formal means of relevancy beyond syntactic or linguistic heuristics but more in terms of semantics. 
	• Page 5, please quantify the influence of prompting engineering in relevance to the conceptual consistency performance and the variants of how the pretrained LLMs was obtained (e.g. training data)
	• Page 6, models: Please provide more training dataset details of the tested LLMs. In particular, the training datasets and distribution of topics (or meanings in general) might highly correlate with their performance over the problem sets created from the QA pairs or concept tuples from ConceptNet --- informally the coincidence and difference of the two distribution of meanings in LLMs' training data and ConceptNet.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper describes a method to evaluate how LLMs learn about knowledge. 
The model seems to be based on the following intuition. Given a task item, a set of background facts is extracted. If the LLM can solve the background facts, then it can solve the task item. This is a clever idea.
strength_and_weaknesses: Strength
- an interesting method to explore real capabilities of pre-trained language models

Weaknesses
- many details of the method are not clear
- it is difficult to asses the quality of results
- the paper is not well organized
clarity,_quality,_novelty_and_reproducibility: The construction of the cornerstone of the theory is really fuzzy. Section 3.1. seems to be a very important spot in this paper. However, it is not clear at all how background knowledge is created from the initial (Q,S,A) triple. This seems to be related to facts F=(c^1,r,c^2), but it is not clear how. Moreover, there are other symbols f in a set B, which is not clear what they are (pag.4)
summary_of_the_review: The paper hides important content. Yet, it should be written more clearly In order to let the reader reach this content
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper attempts to measure the conceptual consistency of the current large language models (LLMs) in question answering tasks of the sort exemplified in the CommonsenseQA (CSQA) dataset. Specifically, the authors focused on the consistency between an LLM's answers to binary (yes/no) questions regarding the existence of relations between relevant concepts and the the answer to multiple-choice questions from the CSQA dataset. The authors claim the finding that the current open LLMs including OPT, GPT-3 (EleutherAI), and T0 have low conceptual consistency (in the range of 0.15 to 0.55 average precision), but the consistency score increases with model size.
strength_and_weaknesses: Strengths of the paper:
1. The formulation conceptual consistency of LLM as a binary prediction problem for answers to the anchor CSQA questions based on answers to background-knowledge questions.
2. The author made the observation that the LLMs show a strong bias towards certain answers to binary questions (preferring "Yes" to "No") and may prevent a reliable extraction of background knowledge. The authors invented the novel approach that combines 1) a method for creating negative examples from the ConceptNet graph, 2) using different wording in the prompt when probing background knowledge, and 3) relying on a majority voting approach. The authors quantified the improvement of background-knowledge probing of this novel approach compared with a single-prompt baseline.

Weaknesses
1. While the formulation of the conceptual consistency is practical and clear, it only covers a narrow aspect of self-consistency that determines whether "theory of mind" (in the author's wording) can be applied to LLMs. Due to the operational definition of the conceptual consistency, the method only captures the model's knowledge with respect to the existence of a relatively small number of (14) relations. In addition, the answers of the LLMs capture only the binary existence of a relation, with no attention paid to other information, such as whether a certain relation's existence is not clearly binary and may dependent on other factors.
2. The way in which the authors selected background-knowledge questions from the ConceptNet is not justified clearly enough. In particular, the authors used a maximum path length of 1 in the knowledge graph, which essentially limits the background-knowledge questions to concepts that are present only in the anchor question itself. However, it is conceivable and likely that answering a certain anchor question depends on knowledge beyond the concepts that are present in the anchor questions. Take the example that author gave in the introduction, the fact that "The peak of a mountain almost always reaches above the tree line" relates to concepts such as "tundra", "tree line", and "height" that are not present in plain text form in the question's text. Therefore the authors should have included a more careful analysis of how varying the maximum path length in the ConceptNet affected the consistency scores. It should be possible to at least experiment with small values of maximum path length to stay within practical computational budget. 
3. The hand-engineered meta-prompts used to probe background knowledge is not sufficiently justified. It has been shown previously that model fine-tuning and soft prompt tuning (https://arxiv.org/abs/2104.08691) out perform hand engineering of text prompt. It is likely that the LLMs can be more reliably probed for the background knowledge by fine-tuning or prompt-tuning based on a small number of (O(100)) unambiguous and basic examples. Alternatively, few-shot prompting can also be explored.
4. The paper lacks some important technical details. The most important ones are how the sampling of the LLMs were performed, e.g., what temperature was used and whether greedy sampling or beam search was involved.
5. A remarks in the Results section are not well supported.  Specifically, the authors claim that the LLMs show a wider range in the their background knowledge than their question-answering accuracy (Figure 4). This claimed isn't supported by quantitative or statistical analysis and hence seems anecdotal. It also doesn't seem to be true even based on the average curves alone in Figure 4.
clarity,_quality,_novelty_and_reproducibility: Apart from the weakness cited above, the paper's writing is sufficiently clear and easy to follow. The quality and reproducibility of the work is hard to judge due to a lack of shared (open source) code. As discussed above, this paper does have novelty in terms of the formulation of the conceptual consistency and the way in which the author probe what background knowledge an LLM possesses, although a lot is left to be desired in the detailed technical implementation of the ideas.
summary_of_the_review: In summary, this paper attempts to address an important and interesting question of how self-consistent LLMs are in their answers to knowledge-based commonsense questions. It contains the novel ideas of formulating the conceptual consistency problem as a binary prediction problem based on probing relevant background knowledge through binary questions synthesized from ConceptNet. It also proposes an interesting approach to synthesize negative questions and using majority voting to gauge the model's background knowledge more reliably. However, the paper did not carefully implement the knowledge measurement, leaving out a thorough exploration of path length on ConceptNet and alternatives ways of eliciting model outputs.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Although current large-scale language models are already being used very successfully to complete many NLP tasks, sometimes with surprisingly good results, they are criticised for not understanding language but only imitating the data on which they have been trained. In order to understand where the limitations of these language models lie and how they can perhaps be overcome, it is all the more important to be able to interpret and explain the decisions of these neural networks against the background of what knowledge they can actually draw on. The authors therefore propose a model to check whether the relevant background knowledge of a language model corresponds to its ability to perform a particular task. To do this, they design a measure of conceptual consistency that relates background conceptual knowledge extracted from the language model to performance on a QA task. The paper examines conceptual consistency for language models of different sizes, finding  that although background knowledge increases with model size, it does not increase as much as consistency or performance on the task. The authors therefore assume that correctly answering questions about the background knowledge of a language model requires less skill than the QA task and would be at a lower level in a hierarchy.
strength_and_weaknesses: The basic idea of measuring consistency in a language model is well founded and comprehensible. The argumentation of the authors and the structure of the experiments can by and large be followed very well. It becomes clear that specific conceptual consistency in large generative language models has not yet been comprehensively considered, and findings in this area could provide clues as to how models might arrive at a better understanding of language. The results of the measurements carried out are convincingly presented and discussed. The paper is overall well written with few negligible typos. 

Although the extraction of background knowledge from the language model and the generation of the background questions is described in great detail, it might be possible to describe more clearly how the concepts to the anchor questions of the QA dataset are selected: The set of concepts is "all the [...]words and phrases that appear in any part 
of the anchor query". Since the anchor question (Q,S,A) also includes the incorrect answers, I assume that only concepts from Q and A (not S) are considered? If the mapping of a phrase to the concept of the knowledge base is done by a 50% words match, could this possibly lead to errors?
Since the results of the conceptual consistency and the accuracy of the QA task are very similar (Fig. 3/Fig. 4(b)), although different things are measured, it might be interesting to show the consistency on the basis of one a different task also.

minor comments: 
•	order of figures 6, 5 (p. 8)
•	typo punctuation "perform better at this Now conceptual" (p. 6)
•	cite parentheses "BERT Devlin et al." (p. 2)
•	references: all authors shown (p. 12!)

clarity,_quality,_novelty_and_reproducibility: The paper is well written and almost error free, it addresses a current problem and is fit very well into the literature. While many works have a focus on constructive consistency, the present study, with a looser definition of consistency, aims at a more general relationship between background knowledge and "understanding" of language models. The proposed  measurement of conceptual consistency can be used on a wide variery of different language models, downstream tasks and datasets. The results should be easily reproducible, since the datasets and models used are public and the large language models have not been further trained or finetuned. 
summary_of_the_review: The authors propose a measure for examining language models that they can justify well and show its value in a sample study. Interesting (although perhaps not particularly surprising) results are also obtained by looking at the individual components, the background knowledge of the model in comparison with its performance on the downstream task. The results of this and further studies could provide clues to a taxonomy of skills in language models.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper describes a new metric for accessing how well language models can understand concepts. The authors use this metric to analyze the performance of language models on common sense reasoning tasks. A main finding is that most language models do not exhibit strong conceptual consistency. 

Strengths:
- Relevant problem space
- Interesting formulation of conceptual consistency as a classification problem

Weaknesses:
- Doubts about whether the conceptual consistency metric used in the paper indeed gives more insights into what language models know
- Somewhat limited novelty of the method 
- Coverage of only a relatively small number of relations (14) and no description of why not all (36) relations of ConceptNet were used
- Lack of clarity in the paper for background knowledge inclusion (improved through revision) and some technical details such as sampling strategies and tuning approaches
justification_for_why_not_higher_score: While the paper addresses a very relevant question, the reviewers express doubts about whether the used metric actually gives us insights. The experiments are also limited and, more importantly, it is not clear why this is the case, e.g. why not testing on all relations on ConceptNet.
Decision: Reject