Unpacking Large Language Models with Conceptual Consistency
Authors: Pritish Sahu
Michael Cogswell
Yunye Gong
Ajay Divakaran
Venue: Submitted to ICLR 2023 (2023)
Keywords: Conceptual Consistency
Theory of Mind
Zero Shot Prompting
Large Language Models
Semantic Consistency
Unsupervised Question Answering
Background Knowledge Extraction

Abstract: If a Large Language Model (LLM) answers “yes” to the question “Are moun-
tains tall?” then does it know what a mountain is? Can you rely on it respond-
ing correctly or incorrectly to other questions about mountains? The success of
Large Language Models (LLMs) indicates they are increasingly able to answer
queries like these accurately, but that ability does not necessarily imply a general
understanding of concepts relevant to the anchor query. We propose conceptual
consistency to measure a LLM’s understanding of relevant concepts. This novel
metric measures how well a model can be characterized by finding out how con-
sistent its responses to queries about conceptually relevant background knowledge
are. To compute it we extract background knowledge by traversing paths between
concepts in a knowledge base and then try to predict the model’s response to the
anchor query from the background knowledge. We investigate the performance
of current LLMs in a commonsense reasoning setting using the CSQA dataset
and the ConceptNet knowledge base. While conceptual consistency, like other
metrics, does increase with the scale of the LLM used, we find that popular mod-
els do not necessarily have high conceptual consistency. Our analysis also shows
significant variation in conceptual consistency across different kinds of relations,
concepts, and prompts. This serves as a step toward building models that humans
can apply a theory of mind to, and thus interact with intuitively.

Under review as a conference paper at ICLR 2023
UNPACKING LARGE LANGUAGE MODELS WITH CON- CEPTUAL CONSISTENCY
Anonymous authors Paper under double-blind review
ABSTRACT
If a Large Language Model (LLM) answers "yes" the question "Are moun- tains tall?" then does it know what a mountain is? Can you rely on it respond- ing correctly or incorrectly other questions about mountains? The success of Large Language Models (LLMs) indicates they are increasingly able to answer queries like these accurately, but that ability does not necessarily imply general understanding of concepts relevant to the anchor query. We propose conceptual consistency to measure LLM's understanding of relevant concepts. This novel metric measures how well model can be characterized by finding out how con- sistent its responses to queries about conceptually relevant background knowledge are. To compute it we extract background knowledge by traversing paths between concepts a knowledge base and then try to predict the model's response to the anchor query from the background knowledge. We investigate the performance current LLMs in a commonsense reasoning setting using the CSQA dataset and the ConceptNet knowledge base. While conceptual consistency, like other metrics, does increase with the scale of the LLM used, we find that popular mod- els do not necessarily have high conceptual consistency. Our analysis also shows significant variation in conceptual consistency across different kinds of relations, concepts, and prompts. This serves step toward building models that humans can apply a theory of mind to, and thus interact with intuitively.
INTRODUCTION
Large Language Models (LLMs) have had many exciting recent successes. These include high performance and even emergent capabilities using just zero or few-shot prompting (Brown et al., 2020; Wei et al., 2022a), but overall performance is still low compared to humans on wide range of tasks for even the largest models (Srivastava et al., 2022), and our understanding these models work is still limited. A popular explanation of low performance and inconsistencies is that LLMs are simply learning to mimic the data used to train them, and this basic pattern recognition limits generalizability, in the case of LLMS exposing the limits of any understanding (Zhang et al., 2022a; Bender & Koller, 2020). We would use a similar line of reasoning to guess how a LLM would answer the following question: "Can GPT-3 see?"
If it performed well on examples from the same distribution we would say it is likely to get it right or vice-versa if performed poorly on those examples. Though valid, this explanation in- complete because it is completely agnostic to the specific content of the statement. We would ap- ply the exact same reasoning and come the same conclusion for similar statements about say blood banks or mock trials, as long as they were from the same distribution (in this example, the CSQA2 dataset (Talmor et al., 2021)). This is in contrast to our day to day life, where our Theory of Mind allows to understand other agents (people) by attributing beliefs, intentions, and desires to them (Premack & Woodruff, 1978) in a way that allows us to usefully predict their behavior (Ra- binowitz et al., 2018; Dennett, 1991). Beliefs are most relevant here, and should be conceptual in order to best support human understanding (Yeh et al., 2021). Ideally we would also be able to apply this kind of understanding LLMs, predicting that the model is more likely correct about GPT-3's sight it knows about GPT-3 generally than if it does not. This would be a conceptual model of the L.M that allows to predict its behavior.

---

Under review as a conference paper at ICLR 2023
Irrelevant Background:
Relevant Background:
Target Task:
Conceptually Consistent
Can GPT-3 see?
Is the sky blue?
Was GPT-3 built by OpenAI?
Model
correct
wrong
wrong
Model 2
correct
correct
wrong
X
Model 3
correct
correct
correct
Figure I: Example target question with relevant and irrelevant background knowledge. A model is conceptually consistent when its knowledge of relevant background information sharing concepts with the target- is consistent with its ability to answer questions correctly.
We want to build models for which that kind of understanding is possible, in this work we take a step toward that goal by modeling the conceptual knowledge of LLM and predicting when the LLM will be correct from that model. Our conceptual model is based on LLM's answers to ques- tions about background knowledge relevant to a particular anchor task (e.g.. question answering), which we assume be reasonable though imperfect measurement of what the LLM can be said to "know." From this and a measurement of question answering performance we compute concep- tual consistency (Figure 1), quantifying whether a model's knowledge of relevant background consistent with its ability to perform the anchor task. Unlike standard approaches evaluation this approach relies on example specific context.
Defining background knowledge is key to our approach because we need it to be relevant enough to establish meaningful consistency. Given the target query "Can GPT-3 see?" a relevant background query might be "Was GPT-3 built by OpenAI?" while "Is the sky blue?" would be irrelevant query. Instead requiring a background fact to logically support the target query in some way, we say background fact relevant if it can tell us something about how typical human language user would respond. Given ground truth response to the target with human response Y and respective responses YK and YK to a potentially relevant background fact we define relevance using a conditional probability. If P(Y = Y|YK relevant because it is not independent of the target and gives us information about whether the speaker will be right or wrong. Knowing GPT-3 was built by OpenAl makes more likely that someone will also know GPT-3 cannot see because it is true and involves relevant concepts. While knowing the color of the sky laudable, there's 's no conceptual overlap with GPT-3's ability to see. In this paper, we focus on this kind of conceptual relevance.
YK) * P(Y Y) then the background fact is
After extracting background knowledge we use prompting to measure how a given LLM handles the background knowledge and how it performs at the anchor task. For this we study three varieties of generative language models across multiple scales up to 66 billion parameters and use a majority vote style prompting procedure to maximize the robustness of our approach to linguistic variations.
Our core contributions are
We extract conceptually relevant background knowledge with respect examples from an anchor task and map them onto background knowledge questions.
We use novel majority vote style zero-shot prompting procedure applied to generative language models to measure LLM performance.
We measure conceptual consistency, focusing on generative language models and showing that consistency is low though it does increase with model size up 66 billion parameters.
We report conceptual patterns in model behavior that fall out of our analysis.
2 RELATED WORKS
Much work has been devoted to studying the limits of large language models beginning with BERT Devlin al. (2019). Typical evaluation of language models measures performance on datasets of labeled examples across a range of tasks, such as those that constitute the amalgamated BIG Bench benchmark (Srivastava et al., 2022). In general they explained simply mimicking the data they were trained with. Low level critiques question the ability of LLMs to reason log- ically (Zhang al., 2022a) or pass adapted psychological tests of language in humans (Ettinger,
2

---

Under review as a conference paper at ICLR 2023
2020). High level critiques question the ability of LLMs to understand anything at all (Bender & Koller, 2020), though one alternative avoids this by defining meaning based on conceptual role (Pi- antadosi & Hill, 2022).
Consistency Most relevant here is the large literature that studies failures in consistency of these models. A common approach is to verify expectation about how the outputs of two different prompts should be related. For example, the prompts "Is the rose red?" and "What color is the rose?" with the expectation that answering "no" to the first question is inconsistent with answering "red" to the second (Ribeiro et al., 2019). CheckList (Ribeiro et al., 2020) offers a taxonomy that can be applied these approaches. Invariance based consistency considers two versions of the input (often paraphrases) where the output is expected to be the same for both versions (Kassner & Schütze, 2020; Elazar et al., 2021; Ray al., 2019; Jang et al., 2021; Ribeiro et al., 2020). Directional expectation consistency expects specific change in the output based on how the inputs are constructed (Ribeiro et al., 2019; Wu et al., 2021; Kassner & Schütze, 2020; Jang et al., 2021; Ribeiro et al., 2020) e.g.. that one output should be the negation of the other. These works generally find that the models they study are often inconsistent, with varying degrees for various kinds of consisteney. Many also improve their consistency measure by fine-tuning the studied models for it.
These kinds of consistencies are constructive rather than conceptual. They establish and improve the degree to which language models can be said have beliefs (Hase et al., 2021), but in order for beliefs to support a theory of mind they should be actionable in that they should support predictions about another agent's 's behavior (Rabinowitz et al., 2018; Dennett, 1991). That LLM has logically consistent outputs does not imply it will be right, but a theory of mind can provide evidence for or against correctness. This looser definition of consistency also makes the idea more general. In contrast, previous work, especially for more complex definitions of consistency, has relied on domain specific and expensive annotated datasets to measure consistency.
Furthermore, in this work we focus on large generative language models as opposed to the masked language models where most prior work on consistency focused. Some of the greatest success has come from scaling generative LLMs (Hoffmann et al., 2022; Wei et al., 2022a), however most ex- isting work on consistency focuses on smaller scale models (mostly masked style language models) because large scale generative models were not openly available until recently (Zhang et al., 2022b), and also because fine-tuning the larger models is prohibitively expensive, making it infeasible as a way to improve consistency. It may be that even though performance increases with scale consis- tency does not, so in this work we study model scale to the degree possible with open source models and limited compute budget.
Explanations and Interpretability Explaining neural net decisions has been studied for a long time, but recent work has shifted focus towards concept based explanations (Yeh al., 2021). For example, derivatives can be used to quantify conceptual influence in neural nets (Kim et al., 2018) and model the causal importance of concepts (Goyal al., 2019). Another line of work links theory of mind to explanations (Chandrasekaran et al., 2017; Shvo et al., 2020). We use concepts to explain LLMs from a theory of mind perspective.
PROPOSED METHOD
To measure conceptual consistency we need to measure background knowledge and QA perfor- mance then predict the latter from the former. First we describe extraction background knowledge in the form of questions with known answers, second we describe how we use prompting to answer both background and anchor questions, and finally we describe the conceptual consistency metric which correlates the two by predicting one from the other.
3.1 BACKGROUND KNOWLEDGE EXTRACTION
Here we focus on question answering (QA) problems and assume a knowledge base of content relevant to the specific QA task is available. Examples in our QA dataset consist of question Q with corresponding choices S = {.i,..., s J. one of which is the correct answer A. These choices can be single words or short phrases. We call (Q, S, A) the anchor query because our first task is to find conceptually relevant background knowledge with respect to that information.
3

---

Under review as a conference paper at ICLR 2023
The background knowledge for given anchor corresponds to list of facts in knowledge base. We assume each fact F = (c r, e²) is represented by two concepts c' and c², and a relation r' between those concepts. Our task is to extract set B {fi..... fiai} of facts conceptually relevant to the anchor and then map those facts onto questions that can be asked to roughly determine whether the LLM knows the background knowledge.
Extracting Background Facts We conceive of the knowledge base as a graph that connects con- cepts nodes and relations as edges. To extract list of concepts from given text snippet we employ basic tokenization then remove stop words and keep only nouns, verbs, adjectives, and adverbs, following the pipeline from Ma et al. (2019). We construct the set of concepts C' all con- cepts that appear in any part of the anchor query (Q.S, A) (including incorrect answers) and have overlap (> 50% of words match) with concept word or phrase from the knowledge base. For two different concepts c²,c² E C' we consider all tuples from all paths length L or less which connect those concepts in the knowledge base, forming a cloud of relational information which constitutes the background knowledge for the anchor given by the selected knowledge base.
In practice this list of background knowledge tuples is too large, so we need to restrict it to a more manageable yet still relevant list. We do this by setting the maximum path length I. to 1, essentially looking for concepts which appear in the anchor and are directly related to each other in the knowl- edge base. This case follows Ma et al. (2019), where they were interested in extracting knowledge tuples to be inserted into neuro-sy neuro-symbolic models.
Background Questions In order to measure how well a LLM already knows the content captured by these tuples we automatically translate them into natural language questions. Consider a fact tuple (c², r²). We substitute its three variables into natural language template designed for the specific relation r. For example, the tuple (book, used for, school) would be translated into "Are book used for school?" Note that because the tuple exists in the knowledge base we know the correct answer to be some version of "yes". These templates are included in Table 2 of the appendix.
Relevance This construction is likely to result in facts that are more relevant than irrelevant be- cause each background query will share at least one concept with the target. Consider a human's 's response Y to target query given their response Yx to background query extracted by this pro- cedure. By construction our background query shares at least one concept with the target query. Since we assume that answers to questions are reflective of knowledge, answering the background query correctly indicates some knowledge of the background concepts, so it also indicates some knowledge of at least one of the concepts in the target query. As a result we expect the difference [P(Y Ý| Yk)= - P(Y
Y) for human language user to be positive for most back- ground queries. We do not claim that all background queries extracted will be highly relevant or that all background queries put together will always be enough to determine the LLM's response confidently. Nevertheless, we do think our background knowledge should be predictive of target responses. Future work could try to increase the relevance of extracted background knowledge.
Negative Background Knowledge In all tuples so far the relation r really does exist between el and c², so the correct answer is always "yes." Language models are often biased towards certain outputs and in this case we found a "yes" bias to be particularly strong in some models, especially the smaller versions of OPT (Figure 6a). As result those smaller models can outperform the larger ones even when they understand the background knowledge less, just because they are prone to saying "yes" to everything. We fix this by extracting negative background knowledge tuples - to which the correct answer is some version of "no" to mirror the positive ones.
We frame the problem in a pairwise fashion: given a positive tuple (ct, r,, r?) the negative tuple generation task is to find an alternative 2² for which the correct answer is "no." The pairwise nature ensures that we measure background knowledge in a balanced fashion to best address the "yes" (or "no"") bias issue. As depicted in Figure 2, we form set of choices C2 that includes every concept ē in the knowledge base that meets criteria:
1. ē does not form valid tuple (c', r),)),
2. č is not cyclic (i.c., not equal to to c²), and
3. ē is in the English dictionary.
4

---

Under review as a conference paper at ICLR 2023
Positive Concepts
Negative Concepts
(do complete "River At Location x")
(do not complete "River At Location X")
Planet Earth
Wilderness
Lawlessness
Wisconsin
Continent
Doorknob
Longhai
Chronicle
Canyon
Town
Valle
Country
Potato
Slant
Coup
No Path from
At Location
Concept "River" and Relation "AtLocat
River
River
Facts from knowledge bases are true by virtue of being in the knowledge base. We mine negative background facts by finding concept pairs without an edge ensure our extracted background knowledge is balanced.
Figure 2:
Our final choice for e² is single sample from the uniform distribution over C The amount of concepts in the English dictionary is quite large, hence we curate the distribution space of C after applying criteria and 2 keep most frequently used concepts. We make this choice in advance, treating background tuples as a dataset, so that even if a positive background tuple is used for multiple anchor queries multiple choices of c² for the same cl and r then we sample e² independently for each of those choices. The final set of background tuples for an anchor query includes all positive tuples that match that anchor along with all negative tuples that paired with the positives. Examples of extracted negative background facts can be found in the appendix Table
always paired with the same negative tuple. However, if there are
3.2 ANSWERING BACKGROUND AND ANCHOR QUESTIONS WITH PROMPTING
Now need to actually extract answer given background question anchor question from the model. We could simply provide the resulting text as input to the language model, see what most likely word it generates next, and take the result as the LM's answer. However, in experiments mainly with the OPT models we found this process to be highly noisy, with inconsistent perfor- mance across models and relations. This is probably due to inconsistencies in the models them- selves (Flazar et al., 2021), but also the simple nature of our templating procedure which doesn't 't always get the grammar correct. We are interested in measuring the language model's conceptual consistency, not linguistic consistency, so we implemented majority-vote vote-style procedure to make our prompting procedure more robust.
Instead of using.just just one question and one answer we consider many variations on the same question and many potential answers, taking the combination assigned the highest likelihood by the model its predicted answer to the question. To vary the question presentation substitute the question generated from our templates into 6 meta-prompts, which are minor variations on how to ask the question reported in Table To vary answer presentation negative words as potential answers including ((Yes, No), (True, False), (Right, Wrong), (Correct, Incorrect), (Positive, Negative), (Pass, Fail), (On, Off).. This results for each query, and we evaluate the model's likelihood of each potential answer word. Note that no sampling required because we only use single answer words. The final answer to question is positive if the input with the highest likelihood used a positive answer word, and otherwise the answer negative.
chose list of 14 positive and
14 84 model inputs
We found this variation to be essential for achieving some consistency across linguistic variations when experimenting with the OPT language models (Zhang et al., 2022b). Voting over multiple variations is similar to the strategies adopted in number of recent approaches. In Shwartz et al. (2020) clarifying statements and answers to questions are chosen by taking a maximum over like- lihood scores of different possibilities, but they do not this to account for linguistic variation in prompts. Similarly, in Elazar et al. (2021) knowledge base tuples are completed by picking a choice from restricted candidate set; here it is not used control linguistic variation in order to study another variable, but used to study linguistic variation itself. More recently majority vote has been used evaluate generative models when generating longer snippets of text (Wang al., 2022; Wei
5

---

Under review as a conference paper at ICLR 2023
Table 1: Meta-prompts utilized for question generation.
Meta-Prompts
1. <queation>?
squestions this true?
3. Answer this question as a<label a>> or <<label.br b>.. Question; <questian>"
4. Each is a question and Answer is of adlabel.a or "<label.b>' Question: <question>? Answer:
5. Pick "<label_a>> <<labblb>>,
Question: <questions ond lAnswer:
6. Question: <question>? Answer:
et al., 2022b). These works use the approach to account for linguistic variation, but in the model outputs (more than just a few words) rather than in the inputs (i.e., the prompts). In comparison to all those studies, our majority-vote-style procedure is used to account for linguistic variation in the prompts themselves.
3.3 MEASURING CONCEPTUAL CONSISTENCY
Now we are almost ready to measure conceptual consistency. Till now we have extracted answers A for the bth background knowledge question Q'=² for the ith example in the anchor task dataset. The anchor questions and answers are Q4 and Á'x- We translate these questions and answers into scores that measure how well the model knows the background knowledge and how well it performs at the anchor task. These background and task scores are defined respectively for each anchor example using accuracy
SB = (BoEP, [|A == Ab + ESEN, [[LA == A'y'l)) /2
S = [[A' = A'All
where Ag and AA are the correct answers, N₁ is the set of negative background tuples for anchor example i, P, is the
positive background tuples for anchor example and [[-]] is the indicator function. Note that the background score weights negative and positive tuples evenly.
Finally we compute the conceptual consistency of model on given dataset and knowledge base by predicting the task score from the background score and reporting average precision
CC = AP,
(3)
where AP()) measures the average precision of the 5 predictor. Intuitively, this score will be high when the model answered more background knowledge questions correctly, so predicting that the model will perform better when knows relevant background knowledge.
4 EXPERIMENT SETUP
Dataset and Knowledge Base We conduct zero-shot evaluation on the CommonsenseQA (CSQA) (Talmor al., 2019) task. CommonsenseQA covers broad range of common sense concepts, with each entry containing a question and 5 answer choices. We choose ConceptNet as our knowledge base because Commonsense(AA is derived from it, although other knowledge bases would also be valid design choices. We use a subset of 14 out of the 36 available relations most rele- vant to common sense, including "antonym", "at location", "capable of", "causes", "desires", "form of". "has a", "is a", "made of", "part of". "related to",, "similar to", "synonym" and "used for". Our experiments used the development set, since test set answers are not publicly available. No model training hyperparameter tuning was applied to our pre-trained LLMs. Most of our development effort went into creating stable prompts.
Models We probe for conceptual consistency using three publicly available model fami- lies, including OPT-(350M, 1.3B, 13B, 30B, 66B} (Zhang et al., 2022b). GPT(EleetherAAD> (125M, 2.7B, 6B) (Black et al., 2022) and TO-(3B, 11B} (Sanh et al., 2022). The OPT and GPT(EllutherAI) models were chosen because they are some of the largest publicly available models and have a range of checkpoints over sizes. We use TO because was tuned specifically for zero- shot prompting setting and achieved competitive results in zero-shot generalization. The prompts and general approach remain the same for models, though we did most the development using OPT models.

---

Under review as a conference paper at ICLR 2023
5 RESULTS
Here we report the conceptual consistency of the models we study, analyze the individual back- ground and task components of that score, show qualitatively how performance on relations and concepts varies, and analyze bias related to our prompting approach.
Conceptual Consistency We compute the conceptual consistency (Equation 3) for each LLM and report the results in Figure 3. In absolute terms average precision on the lower end, showing significant conceptual inconsistency. A model's knowledge of background informa- tion is somewhat predictive of its question answering correctness. Our results show that con- ceptual consistency generally grows with the scale of the model across all model classes, indicating that bigger models are not just more accurate, but also more consistent. A notable exception is between OPT-30B and OPT-66B, where there is improve- ment from increased size. Both achieve the highest consistency among all mod- els. but this one data point suggests there may be diminishing returns. OPT mod- els perform best, with GPT models close
Conceptual Consistency
(Average Precision)
0.55
0.50
0.40
0.30
GPT(EICUtherA))
behind and TO demonstrating much worse performance. This indicates that OPT and GPT are much more conceptually consis- tent than TO. The difference could be due to the decoder only nature of OPT and GPT. It could also reflect degree of forgetting in TO due to supervised fine- tuning, but this is doubtful because of the many datasets TO was trained on was prompts are not designed specifically for TO.
0.20
TO
0.15
10a
10
10a
10
Parameters
Figure 3: Conceptual Consistency of Language Mod-
els. This measures our ability to predict whether lan- guage model will be correct from its knowledge of rel- evant background information.
CSQA. However, may also reflect the fact that our
Background Knowledge
GPT(BleucherA)
Question Answering
55
(Avg. Accuracy)
T0
65
Accuracy
40
60
55
30
25
OFT
50
GFTiElsutherAl
45
15
Parameters
Parameters
(a) Background knowledge performance: how good language models are verifying whether our extracted background facts are truc/falsc. Averaged over 14 lations.
(b) Anchor Task (CommonsenseQA question answer-
ing) performance measured by zero-shot prompt- ing approach.
Figure 4: Aggregated background performance and anchor task performance.
Background and Task Performance We also measure the components of conceptual consistency in Figure 4. For background knowledge we compute Equation 1 averaged per relation and then averaged over all 14 relations. This is reported with 95% confidence interval in Figure 4a. There is an imbalance in how often these relations occur in the data, so this approach weights relations equally just for the purpose assesing background knowledge. For task knowledge we report the anchor score (Equation 2) in Figure 4b, which is CSQA accuracy in these experiments. In both cases our prompting approach is able to extract correct answers from our LLMs. The trend both cases is again an increase in performance with model size. This expected in both cases, but it is interesting to note that the range of performance smaller for background knowledge, suggesting that increasing size helps task knowledge more than background knowledge, though the opposite is

---

Under review as a conference paper at ICLR 2023
80
Accuracy
Accuracy
60
60
40
Negative Tupe
Poal
luple
Negative Tiple
BK
Posiive Tuple
Wajority Prompt Style
Nodel
Singls Proppt Style
GPT
TO
10
10'
10
1011
10a
10'
10th
101l
Parameters
Parameters
(a) OPT models with our majority style prompting and with a single prompt.
(b) GPT and Bias
Figure 6: Background knowledge performance for negative and positive facts.
true for TO. Intuitively question answering is more complex skill that builds on the less complex background knowledge completion skill. From that perspective these results are also an evidence of a skill hierarchy like Bloom's Taxonomy (Sahu et al., 2021), where simpler skills are learned before more complex ones. As for conceptual consistency, OPT models have higher performance compared to GPT and TO models, and this is likely due the same reasons discussed previously. It however notable that the gap between TO and the other models is much smaller for background knowledge. Also, both task and background knowledge, though it may not be significant.
marginal increase in performance between OPT-30B and OPT-66B for
Background Consistency
1.0
Next we ask where back- ground knowledge and con- sistency diverge. In Fig-
Re ation
atfocation
capabloof
Conceptual Consistency Precision)
0.8
causes
ure we plot background score versus conceptual con- sistency for relations that seemed to be more like out- liers. Smaller models tend to be further from the diago-
desires
Small models consistent but inoeledgaable.
simil arto Modal
1.38
(Average
OPT 138
308
66B
nal, either being inconsistent but knowledgeable or consis- tent without knowing much. Relations also behave differ- ently. Small models don't know the "desires" relation very well, but they are some- what conceptually consistent about it, to the point that even though large models get to know the relation better they do get more consistent about it. In the reverse direction, all model scales know roughly the same amount about "causes" background information, but the larger models much more conceptually consistent about it.
Smalier models inowledgeabl but not consistent
0.0
20
40
60
60
100
Background Knowledge (Avg, Accuracy)
Figure 5: Scatter plot of Background Knowledge vs Conceptual Consistency.
Prompting Bias In Figure 6 we analyze our majority-vote-style prompting and prompting biases found our LLMs. Each figure reports the background score (Equation restricted either the set of negative background facts or positive background facts, so accuracy reduces to the percentage of times the model said a version of "no" or "yes," respectively. Figure reports this metric for the majority-vote-stylee prompting we used (Majority Prompt Style) and previous version where we tried find a single best meta-prompt and answer pair (Single Prompt Style). In our single prompt experiments we found that the two smaller OPT models had a strong "yes" bias shown by the red dashed line in Figure 6a. We introduced negative background knowledge, which detected this prob- lem (green dashed line), but also found that majority-vote- vote-y-- style prompting helped ameliorate the issue (solid lines). Even with majority-vote-stye prompting both TO and GPT((leutherA.) still display significant "yes" (TO) and "no" (GPT) bias at smaller scales.
8

---

Under review as a conference paper at ICLR 2023
Compaptual Condidancy
0.5
CFT 2:28
OPT 14
CFT
OFF 1250
GFT
0.3
S4 Symtom
Rolaedd
Pator
Relation Types
(a) By Relation
Conceptual Consitaricy
OPT 1.38
CFT
Nverege
CFT 641
GFT 1234
0.2
0.2
0.3
H20z
Peoddo
budling
School
Mert
Store
Toom
P39d
100
Concept Names
(b) By Concept
Figure 7: Conceptual consistency computed for different subsets of CSQA examples partitioned according to whether the background facts for an example contain the relation or concept.
Concept and Relation Level Consistency We analyze the conceptual consistency at the level of relations and concepts Figure Figure 7a shows consistency for all relations. In case Figure 7b, we picked 14 most occurring concepts with minimum occurrence count of 28. Though the overall trend is that larger models know more background knowledge, there are variations which concepts and relations model is consistent about. In some cases you don't 't need larger model, or least larger models can have sub-par performance. For example, our largest models (OPT-66B and OPT- 30B) are outperformed by smaller versions of OPT, GPT(EleutherAD) and even TO the "Similar To" relation. Less extreme differences occur for other relations like "Made Of", and again lesser differences occur for concepts like "Play". This sensitivity of relations indicates we design per relation, could be a factor. Another observation is that the increase in performance with size is robust for GPT and TO, but that trend is more often violated for the OPT models. In general, trends in concept consistency are more robust than trends in relation consistency.
prompts, which
CONCLUSIONS
In this paper we built a kind of theory of mind about LLMs to study their conceptual consistency, whether their knowledge of relevant background information is consistent with their ability to an- swer questions correctly. For a question we extracted background knowledge from knowledge base of related concepts and used prompting to measure whether popular open source LLMs knew that background information. We also measured their ability to answer common sense questions correctly. This set us up to measure conceptual consistency by predicting correctness from our background knowledge measure. We found that LLMs have moderate amount of conceptual con- sistency, and that it increases with scale. We also found that while knowledge of background infor- mation increases with model scale it does not increase nearly as much correctness conceptual consistency, indicating that models size has a larger impact on difficult tasks than simpler ones and providing evidence of a hierarchy of related skills. In the future would like to study whether this measure of consistency can be used to guide how humans understand LLMs. We also want to focus further the hierarchy of skills by creating dataset that tests multiple levels of comprehension.

---

Under review as a conference paper at ICLR 2023
AUTHOR CONTRIBUTIONS
ACKNOWLEDGMENTS
REFERENCES
Emily M. Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and understand- ing in the age of data. In ACL, 2020.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He. Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler. Usvsn Sai Prashanth, Shiv- anshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang. and Samuel Weinbach. GPT-NeoX- 20B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 Workshop on Challenges & Perspectives in Creating Large Language Models, pp. 95-136, virlual+Dublin, May 2022. Association for Computational Linguistics. doi: 10.18653/w1/2022. bigscience-1.9 URL htsps::t//. hat//aaaaaatholoooo
org bigscience--i.......ee..
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020.
Arjun Chandrasekaran, Deshraj Yadav, Prithvijit Chattopadhyay, Viraj Prabhu, and Devi Parikh. It takes two to tango: Towards theory of ai's mind. ArXiv, abs/1704.00717, 2017.
Daniel C. Dennett. Two contrasts: Folk craft vs folk science and belief vs opinion. John D. Greenwood (ed.), The Future of Folk Psychology, pp. 135-148. Cambridge University Press, 1991.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod- els. Transactions of the Association for Computational Linguistics, 9:1012-1031, 2021.
Allyson Ettinger. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48, 2020.
Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). ArXiv, abs/ abs/1997.07715. 2019.
Peter Hase, Mona T. Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov. Mohit Bansal, and Srini Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. ArXiv, abs/211 13654, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Train- ing compute-optimal large language models. ArXiv, abs/2203. 155566, 2022.
Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. Accurate yet inconsistent? consis- tency analysis on language understanding models. ArXiv, 0tas/2108.06666 2021.
Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In ACI., 2020.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In ICML. 2018
10

---

Under review as a conference paper at ICLR 2023
Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, and Alessandro Oltramari. Towards gen- eralizable neuro-symbolic systems for commonsense question answering. ArXiv, abs/1910.14087, 2019.
Steven T. Piantadosi and Felix Hill. Meaning without reference in large language models. ArXiv, 0oss2208.02955 2022.
David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, -:51,-526, 526, 1978.
Neil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew M. Botvinick. Machine theory mind. ICML, 2018.
Arijit Ray, Karan Sikka, Ajay Divakaran, Stefan Lee, and Giedrius Burachas. Sunny and dark outside?! improving answer consistency in vqa through entailed question generation. ArXiv, abs/1909.04696 2019.
Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. Are red roses red? evaluating consistency of question-answering models. In ACL, 2019.
Marco Tulio Ribeiro, Tongshuang Sherry Wu, Carlos Guestrin, and Sameer Singh. Beyond aceu- racy: Behavioral testing of nlp models with checklist. In ACI., 2020.
Pritish Sahu, Michael Cogswell, Ajay Divakaran, and Sara Rutherford-Quach, Comprehension based question answering using bloom's 's taxonomy. In Proceedings of the 6th Workshop on Rep- resentation Learning for NLP (RepL4NLP-202)) pp. 20-28, Online, August 2021. Association for Computational Linguisties.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang A. Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Can- wen Xu, Urmish Thakker, Shanya Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Stella Rose Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. Mul- titask prompted training enables zero-shot task generalization. ArXiv, abs/21 10.08207, 2022.
Maayan Shvo, Toryn Q. Klassen, and Sheila A. Mcllraith. Towards the role of theory of mind in explanation, Explainable, Transparent Autonomous Agents and Multi-Agent Systems, 12175:75- 93, 2020.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsupervised commonsense question answering with self-talk. ArXiv, abs/2004.05483, 2020.
Aarohi Srivastava, Abhinav Rastogi, Abhishek B Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexan- der W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Annasaheb Rahane, Anantharaman S. Iyer, Anders Jo- han Andreassen, Andrea Santilli, Andreas Stuhlmuller, Andrew M. Dai, Andrew D. La, An- drew Kyle Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakacs, Bridget R. Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bo- janowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekei, Bill Yuchen Lin, Blake Stephen Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, C'esar Ferri Ram'irez, Chandan Singh. Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Tatiana Ramirez, Clara Rivera, Clemen- cia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Daniel H Gar- rette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy,
11

---

Under review as a conference paper at ICLR 2023
Daniel Gonz' alez, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Do- han, D. Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen. Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dim- itri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth P. Donoway, Ellie Pavlick, Emanuele Rodolà, Emma FC Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fan Xia, Fatemeh Siar. Fernando Mart inez-Plumed, Francesca Happ'e, e, François Chollet, Frieda Rong. Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-L opez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Han Sol Kim, Hannah Rashkin, Hanna Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hubert Wong, Ian Aik- Soon Ng, Isaac Noble, Jaap Jumelet. Jack Geissinger, John Kermion, Jacob Hilton, Jachoon Lee, Jaime Fernández Fisac, J. Brooker Simon, James Koppel, James Zheng, James Zou, Jan Koco" n, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Narain Sohl-Dickstein, Jason Phang, Ja- son Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jenni Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel. Jesujoba Oluwadara Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Jane W Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jorg Frohberg. Jos Rozen, José Hernández-
Joseph Boudeman. Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ig- natyeva. Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Ochieng' Omondi, Kory Wal- lace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang. Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando, Louis-Philippe Morency, Luca Moschella, Luca Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Col'on, Luke Metz, Lutfi Kerem cSenel, Maarten Bosma, Maarten Sap. Maartje ter Hoeve, Madotto Andrea, Maheen Saleem Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, M Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew Leavitt, Matthias Hagen, M M'aty' as Schu- bert, Medina Baitemirova, Melissa Arnaud, Melvin Andrew McElrath, Michael A. Yee, Michael Cohen, Mi Gu, Michael I. Ivanitskiy, Michael Starritt, Michael Strube, Michal Swkedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Monica Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, T MukundVarma, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas S. Roberts, Nicholas Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pas- cale Fung, Paul Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Pe- ter W. Chang, Peter Eckersley, Phu Mon Htut, Pi-Bei Hwang. P. Milkowski, Piyush S. Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, QING LYU, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ram' on Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang. Sahib J. Singh, Saif M. Mohammad, Sajant Anand. Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Sam Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebas- tian Schuster, Sepideh Sadeghi, Shadi S. Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo hwan Lee, Spencer Bradley Torene, Sriharsha Hatwar, Stanis- las Dehaene, Stefan Divic, Stefano Ermon, Stella Rose Biderman, Stephanie C. Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swa- roop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq A. Ali, Tatsuo Hashimoto, Te-Lin Wu, Theo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang. Tiberius Nkinyili, Timo Schick, T. N. Kornev, Timothy Telleen-Lawton Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler O. Shultz, Uri Shaham, Vedant Misra, Vera Dem- berg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, W Vossen, Xi- ang Ren, Xiaoyu F Tong, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yang
12

---

Under review as a conference paper at ICLR 2023
Song, Yasaman Bahri, Ji Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Hou, Yushi Bai, Zachary Seid, Zhao Xinran, Zhuoye Zhao, Zi Fu Wang. Zijie J. Wang. Zirui Wang, Ziyi Wu, Sahib Singh, and Uni Shaham. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ArXiv, abs/2206.046155 2022.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4149-4158, 2019.
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandrasekhar Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. Commonsensequ 2.0: Exposing the limits of ai through gamification. In NeurIPS Datasets and Benchmarks, 2021.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ArXiv, abs/2203. 11711, 2022.
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsumori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. ArXiv, 0is/2206.07688 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11990 2022b.
Tongshuang Sherry Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. Polyjuice: Gener- ating counterfactuals for explaining, evaluating, and improving models. In ACL, 2021.
Chih-Kuan Yeh, Been Kim, and Pradeep Ravikumar. Human-centered concept explanations for neural networks. Neuro-Symbolic Artificial Intelligence, 2021.
Honghua Zhang, Liunian Harold Li, Tao Meng, Kai- Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. ArXiv, abs/2205. 1502, 2022a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov. Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. ArXiv, abs/2205.01 01068, 2022b.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
We prepared different prompt templates for the picked 14 relations used in our experiments shown in Table 2.
Table 2: All the relations with its corresponding prompt style with a sample example.
Relation
Prompt Style
Sample Instance
Is A
Is ct c2?
Is security department?
Has A
Does et has a c
does house has a basement?
Antonym
Is ct an antonym of c
Is clever an antonym of dull?
Cause
Does ct cause c²?
does fencing cause small cuts?
Desires
Does a c² desires c²?
does dog desires affection?
Form Of
Is a form of c²?
Is partying a form of of party?
Made Of
Is the cl made of c??
Is the car made of metal?
Part Of
Is c¹ part of c
Is book part of library? Is doctor related to illness? Is ridiculous similar to silly? Is reply synonym of answer?
Related To Similar To Synonym
Is related to c??
Is c similar c²?
Is c synonym of ???
Used For
Are et used for c²?
Are clothes used for wearing?
At Location Capable Of
Is ct at location c??
Is door at location library?
Is c capable of c??
Is child capable of form opinions?
Table 3, we showcase few instances of the negative background facts created from the process described in subsection 3.1.
Table 3: Examples of negative background knowledge task for each relation.
Relation
Negativv Background Facts
I
IIs drill clamp?, Is ocean a shume'? space a micrometer?]
Has
[does mammals has watch?, does has unicycle?, does has uncle?]
Antonym
IIs wash an antonym of detached"', bad an antonym of nighidress?, shade an antonym improvement")] company". does standing (dees a person desires schizophrenia?. does a children desires unemployed?, does tree desires criticism?)
Cause
idees going into coma
shrinking", does doing bousework cause mermaid!)
Desires
Form Of
Ils recycled form burned". Is bath
ferm of interrupted", Is storing form of bleeding!!
Made
[ss the occan made uncomfortablcness. Is car made rain?. Is the plants made of bieycle?]
Part Of
[Is gulf part of round?, Is shower part of anger?, Is bell part of congress?]
Related To
Iss class related to comstarch?? Is room related to jordan?. cable related to tilemaking?)
Similar To
[Is lie similar to botany?, ? Is ridiculous similar aspirin?. is distant similar physiology? 7]
Synonym
heart synonym volition?, station synonym subordination?.
part synonym undermine?]
Used For
Aree hair used for council?. Are theatre used for desk?" Are disk used for pain!!
[Is monkey
fuzzzall?? piano Is computer capable of pillow?, 2, Is a band capable of overmodolation" ?, Is a tiger caguble of fireman!!
macaroni?, Is location gauging"]
Capable Of
14


Reviews:
[{"review_id": "LXbZAKhTzDT", "reviewer": "Reviewer_w16h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attempts to measure the conceptual consistency of the current large language models (LLMs) in question answering tasks of the sort exemplified in the CommonsenseQA (CSQA) dataset. Specifically, the authors focused on the consistency between an LLM's answers to binary (yes/no) questions regarding the existence of relations between relevant concepts and the the answer to multiple-choice questions from the CSQA dataset. The authors claim the finding that the current open LLMs including OPT, GPT-3 (EleutherAI), and T0 have low conceptual consistency (in the range of 0.15 to 0.55 average precision), but the consistency score increases with model size.", "summary_of_the_review": "In summary, this paper attempts to address an important and interesting question of how self-consistent LLMs are in their answers to knowledge-based commonsense questions. It contains the novel ideas of formulating the conceptual consistency problem as a binary prediction problem based on probing relevant background knowledge through binary questions synthesized from ConceptNet. It also proposes an interesting approach to synthesize negative questions and using majority voting to gauge the model's background knowledge more reliably. However, the paper did not carefully implement the knowledge measurement, leaving out a thorough exploration of path length on ConceptNet and alternatives ways of eliciting model outputs.", "strength_and_weaknesses": "Strengths of the paper:\n1. The formulation conceptual consistency of LLM as a binary prediction problem for answers to the anchor CSQA questions based on answers to background-knowledge questions.\n2. The author made the observation that the LLMs show a strong bias towards certain answers to binary questions (preferring \"Yes\" to \"No\") and may prevent a reliable extraction of background knowledge. The authors invented the novel approach that combines 1) a method for creating negative examples from the ConceptNet graph, 2) using different wording in the prompt when probing background knowledge, and 3) relying on a majority voting approach. The authors quantified the improvement of background-knowledge probing of this novel approach compared with a single-prompt baseline.\n\nWeaknesses\n1. While the formulation of the conceptual consistency is practical and clear, it only covers a narrow aspect of self-consistency that determines whether \"theory of mind\" (in the author's wording) can be applied to LLMs. Due to the operational definition of the conceptual consistency, the method only captures the model's knowledge with respect to the existence of a relatively small number of (14) relations. In addition, the answers of the LLMs capture only the binary existence of a relation, with no attention paid to other information, such as whether a certain relation's existence is not clearly binary and may dependent on other factors.\n2. The way in which the authors selected background-knowledge questions from the ConceptNet is not justified clearly enough. In particular, the authors used a maximum path length of 1 in the knowledge graph, which essentially limits the background-knowledge questions to concepts that are present only in the anchor question itself. However, it is conceivable and likely that answering a certain anchor question depends on knowledge beyond the concepts that are present in the anchor questions. Take the example that author gave in the introduction, the fact that \"The peak of a mountain almost always reaches above the tree line\" relates to concepts such as \"tundra\", \"tree line\", and \"height\" that are not present in plain text form in the question's text. Therefore the authors should have included a more careful analysis of how varying the maximum path length in the ConceptNet affected the consistency scores. It should be possible to at least experiment with small values of maximum path length to stay within practical computational budget. \n3. The hand-engineered meta-prompts used to probe background knowledge is not sufficiently justified. It has been shown previously that model fine-tuning and soft prompt tuning (https://arxiv.org/abs/2104.08691) out perform hand engineering of text prompt. It is likely that the LLMs can be more reliably probed for the background knowledge by fine-tuning or prompt-tuning based on a small number of (O(100)) unambiguous and basic examples. Alternatively, few-shot prompting can also be explored.\n4. The paper lacks some important technical details. The most important ones are how the sampling of the LLMs were performed, e.g., what temperature was used and whether greedy sampling or beam search was involved.\n5. A remarks in the Results section are not well supported.  Specifically, the authors claim that the LLMs show a wider range in the their background knowledge than their question-answering accuracy (Figure 4). This claimed isn't supported by quantitative or statistical analysis and hence seems anecdotal. It also doesn't seem to be true even based on the average curves alone in Figure 4.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Fb3mDA4FP_U", "reviewer": "Reviewer_nKxz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Although current large-scale language models are already being used very successfully to complete many NLP tasks, sometimes with surprisingly good results, they are criticised for not understanding language but only imitating the data on which they have been trained. In order to understand where the limitations of these language models lie and how they can perhaps be overcome, it is all the more important to be able to interpret and explain the decisions of these neural networks against the background of what knowledge they can actually draw on. The authors therefore propose a model to check whether the relevant background knowledge of a language model corresponds to its ability to perform a particular task. To do this, they design a measure of conceptual consistency that relates background conceptual knowledge extracted from the language model to performance on a QA task. The paper examines conceptual consistency for language models of different sizes, finding  that although background knowledge increases with model size, it does not increase as much as consistency or performance on the task. The authors therefore assume that correctly answering questions about the background knowledge of a language model requires less skill than the QA task and would be at a lower level in a hierarchy.", "summary_of_the_review": "The authors propose a measure for examining language models that they can justify well and show its value in a sample study. Interesting (although perhaps not particularly surprising) results are also obtained by looking at the individual components, the background knowledge of the model in comparison with its performance on the downstream task. The results of this and further studies could provide clues to a taxonomy of skills in language models.", "strength_and_weaknesses": "The basic idea of measuring consistency in a language model is well founded and comprehensible. The argumentation of the authors and the structure of the experiments can by and large be followed very well. It becomes clear that specific conceptual consistency in large generative language models has not yet been comprehensively considered, and findings in this area could provide clues as to how models might arrive at a better understanding of language. The results of the measurements carried out are convincingly presented and discussed. The paper is overall well written with few negligible typos. \n\nAlthough the extraction of background knowledge from the language model and the generation of the background questions is described in great detail, it might be possible to describe more clearly how the concepts to the anchor questions of the QA dataset are selected: The set of concepts is \"all the [...]words and phrases that appear in any part \nof the anchor query\". Since the anchor question (Q,S,A) also includes the incorrect answers, I assume that only concepts from Q and A (not S) are considered? If the mapping of a phrase to the concept of the knowledge base is done by a 50% words match, could this possibly lead to errors?\nSince the results of the conceptual consistency and the accuracy of the QA task are very similar (Fig. 3/Fig. 4(b)), although different things are measured, it might be interesting to show the consistency on the basis of one a different task also.\n\nminor comments: \n•\torder of figures 6, 5 (p. 8)\n•\ttypo punctuation \"perform better at this Now conceptual\" (p. 6)\n•\tcite parentheses \"BERT Devlin et al.\" (p. 2)\n•\treferences: all authors shown (p. 12!)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "54_NLPGiSn", "reviewer": "Reviewer_dh2f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper compares pretrained LLMs' question-answer results for CSQA dataset and ConceptNet. The study is on comparing the QA performance between CSQA dataset and QA problems extracted from ConceptNet through a relevancy heuristic procedure.   The authors attempt to interpret the average precision metrices comparison of the two related QA datasets as conceptual consistency. \n", "summary_of_the_review": "This paper investigates into an important problem of LLMs what LLMs knows or how to determine their trustfulness regarding given tasks are inspiring but not complete yet.  This paper attempts to understand what LLMs know by looking at the conceptual consistency between QA datasets and the underlying knowledge graph. However,  it is not clear that whether Conceptual consistency between LLMs' CSQA and the QA question-answers extracted from ConceptNet really reveals \"what LLMs know\" thus reveals whether LLMs are trustworthy for given tasks. \n\nDetails\n\t• Page 4, please investigate formal means to evaluate whether the extracting background facts are truly background facts corresponding to the queries or answers. This might be a formal means of relevancy beyond syntactic or linguistic heuristics but more in terms of semantics. \n\t• Page 5, please quantify the influence of prompting engineering in relevance to the conceptual consistency performance and the variants of how the pretrained LLMs was obtained (e.g. training data)\n\t• Page 6, models: Please provide more training dataset details of the tested LLMs. In particular, the training datasets and distribution of topics (or meanings in general) might highly correlate with their performance over the problem sets created from the QA pairs or concept tuples from ConceptNet --- informally the coincidence and difference of the two distribution of meanings in LLMs' training data and ConceptNet. \n", "strength_and_weaknesses": "Strength\n\t• An attempt to evaluate LLMs  through a metaphor of conceptual consistency (it seems that the authors want to evaluate a form of reliability or trust by looking at what LLMs' \"knows\" but no rigorous definitions are given)\n\t• The definition of conceptual consistency itself is well-defined in the form \n\n\nWeakness\n\t• It seems that the authors want to evaluate a form of reliability or trust by looking at what LLMs' \"knows\" but no rigorous definitions are given. \n\t• It is not clear that whether Conceptual consistency between LLMs' CSQA and the QA question-answers extracted from ConceptNet really reveals \"what LLMs know\". There are heuristic syntactic procedures to extract QA background facts from ConceptNet for CSQA question-answer pairs. The influence of heuristic procedures needs to be formally characterized and bounded for the \"what LLMs know\" conclusion to be rigorously valid.  There is a logical gap between \"what LLMs know\" and the \"conceptual consistency definition\" as it is now. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "308tzm8tAD", "reviewer": "Reviewer_yTJs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a method to evaluate how LLMs learn about knowledge. \nThe model seems to be based on the following intuition. Given a task item, a set of background facts is extracted. If the LLM can solve the background facts, then it can solve the task item. This is a clever idea.", "summary_of_the_review": "The paper hides important content. Yet, it should be written more clearly In order to let the reader reach this content", "strength_and_weaknesses": "Strength\n- an interesting method to explore real capabilities of pre-trained language models\n\nWeaknesses\n- many details of the method are not clear\n- it is difficult to asses the quality of results\n- the paper is not well organized", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject