Reasoning about concepts with LLMs: Inconsistencies abound
Authors: Rosario Uceda Sosa
Karthikeyan Natesan Ramamurthy
Maria Chang
Moninder Singh
Venue: COLM (2024)
Keywords: KG reasoning in LLMs, LLM consistency, Synthetic data generation for LLM evaluation, RAG, prompt engineering,
TL;DR: Through systematic testing of simple knowledge graphs, we've discovered plenty of inconsistent reasoning  in LLMs. Here we discuss the test results and ways to make the LLMs more robust.

Abstract: The ability to summarize and organize knowledge into abstract concepts is key to learning and reasoning. Many industrial applications rely on the consistent and systematic use of concepts, especially when dealing with decision-critical knowledge. However, we demonstrate that, when methodically questioned, large language models (LLMs) often display and demonstrate significant inconsistencies in their knowledge.

Computationally, the basic aspects of the conceptualization of a given domain can be represented as Is-A hierarchies in a knowledge graph (KG) or ontology, together with a few properties or axioms that enable straightforward reasoning. We show that even simple ontologies can be used to reveal conceptual inconsistencies across several LLMs. We also propose strategies that domain experts can use to evaluate and improve the coverage of key domain concepts in LLMs of various sizes. In particular, we have been able to significantly enhance the performance of LLMs of various sizes with openly available weights using simple knowledge-graph (KG) based prompting strategies.

Published as a conference paper at COLM 2024
Reasoning about concepts with LLMs: Inconsistencies abound
Rosario Uceda Sosa, Karthikeyan Natesan Ramamurthy, Maria Chang & Moninder Singh
IBM Research
Yorktown Heights, NY 10598, USA
{rosariou, knatesa @us. ibm. com, maria. chang@ibm.ccmm. con ibm. com
Abstract
The ability to summarize and organize knowledge into abstract concepts is key to learning and reasoning. Many industrial applications rely the consistent and systematic use of concepts, especially when dealing with decision-critical knowledge. However, demonstrate that, when method- ically questioned, large language models (LLMs) often show significant inconsistencies in their knowledge.
Computationally, the basic aspects of the conceptualization of a given do- main can be represented as Is-A A hierarchies in knowledge graph (KG) ontology, together with a few properties or axioms that enable straightfor- ward reasoning. We show that even simple ontologies can be used to reveal conceptual inconsistencies across several LLMs. We also propose strategies that domain experts can use to evaluate and improve the coverage of key domain concepts in LLMs of various sizes. In particular, we have been able to significantly enhance the performance of these LLMs with openly available weights, using simple KG-based prompting strategies.
Introduction
Conceptualization is key cognitive ability that enables abstract thinking. Through concepts we communicate and learn complex knowledge, generalizing from instances and applying learned principles new situations. Conceptualization sits the base symbolic reasoning and allows to plan ahead and innovate beyond our physical experience.
For example, children can easily conceptualize 'chair' to the point of identifying new instances of chairs they haven't ' seen before. Furthermore, when they learn 'armchair' they innately understand is a type of chair (Is-A hierarchy) and that whatever principles we apply to 'chair' also apply to its sub-concept 'armchair'. In summary, children innately learn not only the concepts themselves, but their associated Is-A hierarchies and how to reason about them consislently.
Such consistent use of conceptualization is critical several industrial applications where LLMs are used. Take, for example, customer-facing chatbot in a property and casualty insurance company. It has to be dependable in its knowledge of, say, vehicle types (Kout- somitropoulos & Kalou, 2017): if 'vehicle' is defined as an 'insurable object' that covered according to 'policy', the LLM should consistently assert that 'cruiser motorcycle', 'van' or 'scooter' are vehicles but 'child's tricycle' is not considered vehicle for the purpose of an insurance policy. Any inconsistency in identifying other sub-concepts (related by the IsA or subConceplOf relation) of 'vehicle' in this context could lead to lack of trust in the system and downstream harm to users.
this consistent use of and reasoning about a concept hierarchy by LLMs that we want to evaluate and discuss. Provided that an LLM has already some knowledge about concepts (and sub-concepts) in a given domain, we ask, is this knowledge dependably shown when answering direct questions? Can we correct any inconsistencies with additional context? Can we leverage the LLM's knowledge consistently in simple reasoning tasks, for example, concluding that cafe racer' and 'naked bike' are both types of motorcycles and that all

---

Published as a conference paper at COLM 2024
Extract
Knowledge Base
concept Netarchy
A. Create test cases to test properties
Concept as a type All pediatric surgeons are surgoons.
Strict subconcept- There are surgeons who are not pediatric sungsons.
Extraction of concept hierarchy to be tested
Transitivity A pediatri surgeon surgeon and surgean is medical specialist. Hence a pediatric surgeon is also medical specialist
using:
Icently Inconsiotenciis
Property inheritance An orthopedic pediatric surgeon works in the field of chidhood disease since pedistricann works the field of chichoodd disease.
2)
Lo
Language model
Create
under realistic scenarios/potlices
Pollcy: Surgaons are not allowed to work more than four days week
Improve inconsistencies in propertiss context
every specialist days a week?
Test and correct inconsistencies in language models
Generate test cases
Figure 1:
Our proposed approach to test and correct the inconsistencies in an LLM's knowledge of concept hierarchies and in its application realistic scenarios.
properties of motorcycle (like having maximum capacity for two passengers) apply to both of them, as well as all other sub-concepts of 'motorcycle'?
We propose systematic and automatic method to test and correct inconsistencies, as outlined in Figure 1. It consists of three step process: (1) we automatically extract concept hierarchy to be tested from a knowledge base; (2) we generate -also automatically- various test cases to sieve inconsistencies via direct questions (Figure 1A) that leverage reasoning about these concepts (Figure 1B) and (3) we test the language model to identify inconsistencies and reduce them using generated additional context.
We believe that the relation between KGs and LLMs is at the heart of a neuro-symbolic approach to AI. KGs provide structured, factual information in an algorithmic, traceable way, while LLMs offer advanced natural language understanding and generation. As interest adapting LLMs to specialized domain vocabularies is growing (Zhang et al., 2023; Shen et al., 2024), the integration these complementary technologies holds the potential for creating more accurate and reliable AI systems, specially in applications requiring both precise information and sophisticated language capabilities (AlKhamissi et al., 2022). That why, in general, are interested in the various aspects of the integration of both KGs and LLMs, including knowledge validation.
It is also worth noting that, at this point, it is not necessary to consider more complex relations or reasoning patterns, since we already found plenty of inconsistencies at the most basic levels of concept reasoning in all studied LLMs. Our goal is not produce a thorough vetting of a concept graph in an LLM, but to create a systematic baseline for the fine-grained evaluation of inconsistencies. Understanding why these happen and how fix them in practice promises to be fertile area of research.
Our main contributions are: (1) We devise methods for using ontologies assess the consistency and coverage of conceptualization in LLMs this is done by creating test cases based on the knowledge graphs (KGs) or ontologies an automated manner, (2) we demonstrate that several well-known LLMs with openly available weights demonstrate many inconsistencies in their knowledge, even with very rudimentary, small ontologies, and (3) we show that using simple prompting approaches can reduce these inconsistencies and improve the coverage of domain concepts in several LLMs with openly available weights.
Our paper is structured as follows. We start with a working definition of conceptualization (Section 2), then show how to extract sample ontology from Wikidata for our evaluation
2

---

Published as a conference paper at COLM 2024
(Section 3). We define the inconsistencies we look for in LLMs (Section 4), and discuss use case where we test the consistency in reasoning performed by LLMs for this ontology (Section 5). The results of our evaluation are discussed in (Section 6), followed by related work (Section 7) and conclusions/ future directions (Section 8). Additional experimental details and results for one more domain (finance) are presented in the appendix. The datasets needed to reproduce our results along with prompts that we use are included in the supplemental material.
A working definition of conceptualization for KGs
We define concept C as set of its instances. For example, 'medical specialist' describes all the people whose professional occupation is a medical specialty. Subconcepts like 'surgeon' or 'pediatrician'' represent subsets of medical specialists. A subConcep!Of (also known as IsA) hierarchy of concepts is the simplest incarnation of an ontology, where every node represents concept and the directed edges represent the subConceptOf relationship. This directed graph reflects 'mental picture' of the domain that users would expect to be stable and consistent.
Here, we consider the key computational properties of conceptualization shown below:
Concept as type. If A a subConceptOf B then every instance of is an instance of B. Paraphrasing, every A is also B, or an A is type of B. E.g. all pediatric surgeons are surgeons.
Strict subconcept property. When the subConceptOf relation is strict, there are instances of that are not instances of A. E.g., there are surgeons who are not pediatric surgeons.
Transitive property. The relation subConceptOf is transitive. L.e., if A is a sub- ConceptOf B, and B is subConceptOf of D, then A is subConceptOf of D. E.g., Given that a pediatric surgeon is a surgeon and surgeon is medical specialist, a reasonable user would infer that a pediatric surgeon also a medical specialist. In particular, when we apply the transitive property to a concept graph, we are effectively adding implicit edges those already explicit the graph, like the edge between pediatric surgeon and medical specialist. The resulting, augmented set edges is usually called the deductive closure of the graph.
Subconcept property inheritance. Every property that B's have, A's also have. E.g., if we assert that 'medical specialists must be board certified', we would also expect that surgeons and pediatric surgeons need be board certified.
There are other properties that apply to conceptualizations (like reflexivity, or specific semantics of concept properties), but we consider that the properties above sum up the behavior that most users would expect when reasoning about concepts, well as rules and constraints, which also require the use of subConceptOf hierarchies.
Even though this is an informal discussion about concepts and how most people would reason about them, we must remark that in the first property above we are equating the set theoretical definition of concept (i.e., the set of its instances) with type theory (a concept is also a type). Most people won't have trouble understanding the context which the term "surgeon' is used, and we expect that an LLM would do likewise.
3 A Wikidata-based sample ontology
We start with small ontology fragment automatically extracted from Wikidata (Vrandežić, 2012; Erxleben et al., 2014; Vrandečié & Krötzsch, 2014) from a set of seed concepts. Wikidata is well known KG, whose content is agreed upon by thousands of contributors. We chose a common vocabulary, namely, medical specialties and specialists ('surgeon', 'pediatrician', etc.) as shown in Figure 2. For the sake of conciseness, we have not drawn all of the subConceptOf relations between the medical specialties, but they are part of the underlying deductive closure of the KG.
3

---

Published as a conference paper at COLM 2024
We tried this small vocabulary with openly available LLMs, and we found that all of them answered correctly questions about the edges in over 90% of our dataset question clusters, designed to test the edges and paths of the knowledge graph (see Section 6). This fact alone demonstrates that the LLMs 'knew' of the graph vocabulary. Also, it is worth noting that our results are not domain specific, we have obtained similar ones in other domains. In Appendix we show simple example from personal finances.
subConcept
medical specialist
medicine
hasFieldofOccupatton
orthopedics
orthopedian
surgeon
surgery
pediatrician
pediatrics
orthopodic surgery
orthopedic sungeon
podiatric surgeon
pediatric surgery
orthopadic pediatric surgeon
orthopedic pediatric surgery
Figure 2: A concept hierarchy snapshot: medical specialists and their specialities.
To extract the ontology, we provided two seed concepts, the main entity, 'medical specialist' (Q8332438), and one sample property to test the property inheritance, 'medical specialty' (Q930752). We get graph with 3130 nodes (medical specialists and their occupations).
From this initial graph, it is straightforward instances (P31) and subclasses (I279).. Given that the differences between both of them are arbitrary higher order graph like Wikidata (where entities can be instances of a class and yet have instances themselves), we subsume them into the subConceptOf relation.
extract the segment that includes their
worth repeating that reasoning about these concepts means being able to assert not just the explicit edges, but also their deductive closure, i.e., the virtual, implied edges by the subConceptOf relation as defined above.
Logical and ontological inconsistencies and question clusters
The simplest form of logical contradiction is to both assert and deny the exact same fact, e-g., "a cardiologist is a medical specialist" and "a cardiologist is not a medical specialist". Other common form of logical inconsistency consists in asserting fact and denying one of its (more or less immediate) consequences with respect to a given set of axioms or properties. For us, this set is described in Section 2 above. This definition aligns with the formal definition of consistency discussed in Nguyen (2008)- "in knowledge-baser systems the notion consistency of knowledge often understood situation in which knowledge base does not contain contradictions."
Starting with the KG, we automatically generate a set (or cluster) of queries with a uniform yes/no expected answer. The questions in each cluster map to statements that must collec- tively be true or false (depending on how the cluster is designed). A set of mixed answers reveals an inconsistency with respect to the conceptualization properties above, which can be checked automatically and renders the cluster inconsistent.
However, there is the possibility that an LLM answers 'no' for entire cluster for which the expected answer is 'yes'. This could be due to knowledge missing in the LLM. We call these incomplete clusters instead of inconsistent clusters (where answers and 'no'). As we see in Section 6 the former are exceedingly rare (or non existent in several LLMs), meaning the LLMs tested 'know' these concepts.
mix of both 'yes'
While it is not possible to algorithmically sieve all the knowledge in LLMs, even using standard heuristics that exist to determine sets of unsatisfiable statements (statements
4

---

Published as a conference paper at COLM 2024
which cannot possibly be all true at the same time and thus reveal inconsistency), our approach allows an end user to define mission critical concept hierarchies and test them to ensure consistent responses for them. These graphs are small and test the key properties of concepts, as the general problem of iddntifying minimal unsatisfiable sets in KBs (equivalent to inconsistent clusters) is NP-hard (McAreavey et al., 2014; Gernert, 2005; Pan & Zhang, 2007).
Before we describe in detail these question clusters, we emphasize that all LLMs tested have responded correctly to some of their individual questions in over 98% of the clusters (that is, we have very few incomplete clusters), even with a simple prompt. This means that the LLMs can process both the vocabulary as well as the linguistic forms shown here.
Each of the cluster types defined below test one or more of the conceptualization properties defined in Section 2.
Edge clusters We check the first two properties in Section 2 with edge clusters.
The first type of edge cluster is called positive edge cluster. As the name indicates, it refers to an edge that exists in the KG. We use various expressions in order to test the LLM linguistic flexibility and robustness. Take, for example, the edge (surgeon, subconceptOf, medical specialist). The corresponding positive edge cluster consists of the following questions:
Is a surgeon medical specialist?
Is a surgeon type of medical specialist?
Is every surgeon a medical specialist?
Is a surgeon also a medical specialist?
Obviously, the expected answer to all of these is 'yes'.
If an LLM answers all these questions in the negative, is possible that it hasn't been trained or doesn't know this particular edge (i.e., it's an incomplete cluster). However, if say, all questions except the third question are answered 'yes', there is obviously an inconsistency in the LLM knowledge. If every surgeon is NOT a medical specialist, it cannot be that surgeon IS a medical specialist or that a surgeon is type of medical specialist. That is, these answers imply an unsatisfiable set of statements. These questions are very simple and, in theory, would possible to increase the variations in questions in each of the clusters, but it would only likely make the model responses more inconsistent according to our definition, so we consider that the performance we report here an optimistic estimate.
used to test the strict subconcept property above, when concept is strictly contained in its parent B, meaning that there are instances of that are not instances of A. For example, for the inverse of the positive edge cluster above, subset of the questions we ask would be:
Inverse edge clusters
Is every medical specialist a surgeon?
Is a medical specialist type of surgeon?
All these questions be generated automatically by comparing the instances (P31) of both A and B and checking there is no 'same as' property (P460) between them (which is exceedingly rare, anyway).
The third type of edge cluster, the negative edge cluster tests the first set theoretic property of conceptualization, but for non existent (false) edges. These 'false' edges are automatically built from the hierarchy by selecting random nodes not related by the subConceptOf rela- tion. For example, (cardiologist, subConceptOf, dermatologist) or (surgeon, subConceptOr, hypnotherapist. The questions are linguistically formulated as in the positive edge clusters. In this case, subset of the questions we have would look like:
Is a surgeon hypnotherapist?
Is surgeon type of hypnotherapist?
5

---

Published as a conference paper at COLM 2024
It is important to note that some of the questions where the LLM disagrees with the ground truth answer in our dataset may be technically correct. An LLM may object to one of these particular linguistic forms and may make well reasoned argument for its answer. For example, when asking "is an orthopedic pediatric surgeon an infection control physician?" the language model mrixtral-8x7b-insste offers an explanation for non-committal answer: is possible that an orthopedic pediatric surgeon may work in the field of infection control, however this not their primary field of occupation, which is orthopedic surgery and pediatric surgery". This answer is technically correct, but not consistent with the answers to the majority of similar questions such as, "is an orthopedic pediatric surgeon a infectious disease physician ?" or 'is an orthopedic pediatric surgeon hepatologist?", and dozens of other similar questions that this LLM answered with simple 'no'. Still, given that we are not testing the LLM knowledge, but its consistency, we still have to mark this answer -when compared to the majority of similar answers- inconsistent.
in this case), instead of a 'yes' 'no' answer,
Path clusters This second type of cluster tests the transitivity of the subConceptOf relation relatioon described in section by querying sequence of edges in a given path. Using the same linguistic forms as before, we ask about the deductive closure of path (the curved arrows in fig 3). In our sample graph there are 4 such paths. Two of these are, [orthopedics pediatric surgeon, pediatric surgeon, surgeon, medical specialist], shown, and [orthopedics pediatric surgeon, orthopedics surgeon, orthopedian, medical specialist). The edges in the paths are queried as shown in the case of the edge clusters.
subConceptOf
orthopedic pediatric surgeor
pediatricc surgeon
surgeon
medical specielist
hasFieleOlDccupattio
subConceptor deductive closure
orthopedic pediatric surgery
pediatric su surgery
surgery
medicine
Figure 3: Deductive closure between orthopedic pediatric surgeon and medical specialist.
Property hierarchy clusters The last type of cluster tests the fourth property of concep- tualization, subconcept property inheritance. This is a core feature of conceptualization that affords abstract reasoning. For example, consider the questions below:
is the field of occupation of surgeon surgery?
is an orthopedic pediatric surgeon surgeon ?
is the field of occupation of a orthopedic pediatric surgeon surgery?
If the field of occupation of a surgeon is surgery, and orthopedic pediatric surgeon is surgeon, we would expect that the field of occupation of an orthopedic pediatric surgeon is also surgery. Of course, a more specific answer is that orthopedic pediatric surgery is the occupation of an orthopedic pediatric surgeon, but the fact remains that all of the models tested answer the above cluster in the affirmative in the majority of cases. Again, it is the matter of consistency that concerns this study. Again, the edges in this subgraph are tested using linguistic forms in edge clusters.
Demonstrative use case
Why is it important to ensure that an LLM can consistently answer seemingly simple questions about the edges of a given KG? Imagine set of policies, rules or processes that health care network or an insurance company wants define and use in an Al application. Take, for example:

---

Published as a conference paper at COLM 2024
1. "Only pediatric surgeons can perform surgery on patients younger than 18 years old."
2. "Only surgeons are required to work no more than four days per week."
Not only the policy designers would expect define and manage these rules using abstract concepts, but the users of the application would expect to query these policies using more specific vocabulary related their case. Consequently, the application should be able to understand whether pediatric surgeon or pediatrician satisty either policy.
We have created a small dataset of 10 scenarios with simply worded policies that apply to the medical specialists in our sample knowledge graph (included in our supplemental materials). Each scenario tested with two types of questions. The first one is "Does the policy apply to every (specialist)?" where (specialist) is substituted by one of the 7 terms in our sample graph ("pediatrician', 'surgeon', 'orthopedic surgeon' and so on). The second type of question mimics the policies above, using the same type of term substitution. The queries corresponding to the policies above are:
1. Is every (specialist) allowed to treat or operate on patients younger than 18 years old?
2. Is every (specialist) allowed to work more than four days per week?
Knowledge of our sample graph, the equivalent implicit knowledge, is required to answer these straightforward questions correctly, which shows that concept hierarchies lie at the base of this type of industrial applications. However, we see in Table 1 many of the LLMs with openly available weights get many individual answers wrong, even though they also get some answers right, It worth noting that there are no 'incomplete' scenarios (where every individual question in a cluster is incorrect) here. So, we ask ourselves again, what happened? lack of specific knowledge (one particular edge or node) or lack of overall consistency in the knowledge? Why do the LLMs fail to answer correctly in some cases and not in others? Can we pinpoint the specific holes in the knowledge so they can be corrected?
To dig deeper into these questions, we need to generate dataset to test systematically the knowledge graph directly, as we have discussed in Section
Table 1: Evaluation of 10 policy-based scenarios (14 questions per scenario).
% incorrect individual scenarios (10) answers (140)
% inconsistent
LLM name
google/flan-t5-xl (Chung et al., 2022)
65.71
100
google/flan-t5-xxl (Chung et al., 2022)
24.28
90
google/flan-ul2 (Tay et al., 2023)
15
70
metaa-laama/llaaa----3B-oo (Touvron et al., 2023)
22.8
80
meta-llama/llmaa---00b--hh (Touvron et al., 2023)
15
60
tiiuae/falconn11800 (Almazrouei et al., 2023)
15
60
mistrabiimmistra/---iisttt---- (Jiang et al., 2023)
13.57
60
mistraliimmiitaalkk//---nttttttt (Mistral.Al, 2023a)
13.57
40
thebooke/mixxttalk//b--------- AMis 2023b)
35
100
Evaluation and coverage improvement
The three types of clusters described above are designed to highlight the inconsistencies of the LLM knowledge. We automatically extract them from the topology of the test KG above, producing 119 clusters, with 96 edge clusters (the high number is due to the fact that have negative and inverse edge clusters representing edges NOT in the graph). More details are provided in Appendix A, and additional results for a different domain ontology are presented in Appendix B.
7

---

Published as a conference paper at COLM 2024
We test this graph in openly available models (see Table 1 for model information) using a simple prompt with 11 sample questions from the medical domain. These models are hosted in our own organization's infrastructure. The prompt used is provided in the supplementary materials. We ask for yes/ no answers which can be automatically tallied. A 'yes' answer means that, for every possible instance, the question the affirmative. Otherwise, the answer should be 'no', as it doesn't hold for the concept (i.e., all its instances). The results are displayed in Table 2. For conciseness' sake, we have added all the edge clusters together. A few facts worth noting. First, we notice in the leftmost column that there are very few incomplete edges (where all the individual responses in cluster are wrong). This means that out of the 96 edge clusters, the vast majority of them are known the LLMs. In fact, some LLMs have no incomplete edges. Second, we notice that the notion of property inheritance is the most challenging, since all of the models fail over 36% of the time.
always be answered in
Table 2: Evaluation results by model using simple prompting strategy.
% incomp.
% incons. % incons.
% incons.
% all
LLM name
edges
edges (96)
paths
property inherit. (11)
incons.
(96)
(12)
(119)
google/flan-t5-xl
4.17
40.62
16.66
36.36
41.18
google/flan-t5..
1.04
35.41
16.66
36.36
34.45
google/ 2fgnn//fann-22
4.17
26.04
33.33
54.54
32.77
1eta-llama/llama-2133333
13.54
16.66
36.36
15.97
meta-llama/lllmm-22777
3.13
22.91
16.66
45.45
26.89
tiuuae/falcon-1880b
17.7
16.66
36.36
19.33
mistrabi/nnisttal----insuuuu-
4.16
25
36.36
9,24
mistelai/mmixtaa-kk/Binmmuuuuse
2.08
22.91
16.6
36.36
25.21
thebookk/miixttalk///----11-1112
1.04
32.29
16.66
36.36
31.93
Table 3: Evaluation results by model with context-augmented prompts
incomp. edges (96)
% incons.
% incons.
% incons.
% all
% improve.
LLM name
edges (96)
paths
property
incons.
(all
(12)
inherit. (11)
(119)
incons.)
flan-t5-xl
1.04
10.41
25
27.27
14.29
26.89
flan-t5-xxl
1.04
10.41
9.24
25,21
flan-ul2
1.04
12.5
27.27
13.45
19.33
-lab--t-
0
7.29
9.09
6.72
9.24
-lab--t-
2.08
10.41
0
9.09
10.92
15.97
falcon-180b
1.04
13.54
0
11.76
7.56
mistral-7b-insttuut---00
0
6.25
5.04
4.20
mixtral-8k7h-insstuut--o11
0
9.37
27.27
10.08
15.13
mixtral-88x7--v0-1-pttt
0
9.375
0
27.27
10.08
21.85
Next, we look to enhance the performance of the initial prompt by adding to the queries context with the propositionalization of the knowledge that was missed by all the models, ie., we use the same context for all the models. This context is computed automatically, as our underlying dataset (included in supplemental file) maps the cluster questions into their corresponding assertions. For example, 'ss every orthopedic surgeon surgeon?' is mapped to 'every orthopedic surgeon a surgeon'. This allows us generate the context for queries on second test. This 'wholesale' approach to context augmentation yields roughly the same improvements as if we tailored the context to each individual model.
With this simple prompt augmentation strategy, we obtain a sizable performance enhance- ment as shown in Table The rightmost column reflects this performance enhancement in the clusters, showing that now points of inconsistency have been reduced up to one third. It worth noting that even leveraging this explicit knowledge doesn't eliminate inconsistency altogether.
8

---

Published as a conference paper at COLM 2024
7 Related Work
Seminal work by Petroni et al. (2019) demonstrated that a language model could learn relational knowledge (i.e. facts would expect to be found in knowledge base) during pre-training. This raised the possibility that language models could serve as approximations for knowledge bases right out of the box. However, Elazar et al. (2021) used paraphrased querying to show that such knowledge could not elicited consistently /relially This led to the development of frameworks for measuring inconsistency in language models (Jang et al., 2021; Laban et al., 2023; Sahu et al., 2022) as well as novel training setups with consistency- based loss (Elazar et al., 2021). The consistency issues found in LLMs have been identified as one of the key areas of future work needed to enhance LLMs so they share the same strengths -and consistency- as KBs (AlKhamissi et al., 2022).
Large language models have recently been shown to exhibit abilities akin to 'reasoning' when prompted in certain ways. For example, chain-of-thee -f-thogght prompting Wei et al. (2022)) gets models to provide explicit steps it took to arrive at an answer. Nevertheless, it is not clear whether it actually demonstrates that the LLMs are actually reasoning Wei et al. (2022); Kojima et al. (2023). Wang et al. (2023) explores the consistency of LLM results via chain-of-thought and studies ways of making such results more consistent. A nice survey the current state of knowledge in reasoning in LLMs is provided by Huang & Chang (2023). Other work has looked what LLMs actually know Yin et al. (2023); Srivastava et al. (2023); Sun et al. (2023) and have shown that LLMs exhibit are very weak in this regard, with performance sometimes barely surpassing random guessing Srivastava et al. (2023).
Improving consistency and factual correctness of language models is related to ongoing work that aims to integrate external knowledge into LLMs, either from unstructured sources like retrieved documents or from structured knowledge bases (Feng al., 2023; Yang et al., 2024). Approaches may be applied at different stages of the model lifecycle (Pan et al., 2024): KGs may be used in pre-training (Yasunaga et al., 2022), tuning (Zhang et al., 2024; Cheng et al., 2023) or information from KGs can be incorporated directly into the prompt (Andrus et al., 2022; Fatemi et al., 2024).
Our proposed approach differs from the above related works in that we perform analysis of consistency of knowledge of LLMs with respect to small and targeted KG by automatically generating test cases in the shape of query clusters. Our clusters can act as building blocks of satisfiability -or unsatisfiability,, so we can identify small portions of knowledge to edit or evaluate. Our generated query clusters measure the knowledge consistency of simple edges, both positive (e.g. 'true') and negative (e.g. 'false'), as well as the consistency of paths and property inheritance reasoning, in contrast to other approaches which do not discriminate the semantics of relations or take into account the deductive closure of the graph (Rajan et al., 2024).
Also, we do not require an externally annotated dataset, such as a QA benchmark. We also perform targeted editing of the LLM's knowledge using prompting. This is because in industrial applications, the domain expert requires consistency in a relatively small fragment of a specialized KG. For example, in a general KG, a bicycle is objectively type of vehicle, but in our introductory insurance example, bicycles are typically not covered by vehicle insurance and so they cannot be considered vehicles per the insurance contract. This means that domain experts may need to edit the knowledge. While KG reasoning and editing in general may be useful like in GraphRAG or (Luo et al., 2024), we explore more targeted editing that can be systematically tested and verified to gain the trust of the domain experts and other relevant stakeholders.
http://wwww...t..o./o/oe//e/e/ea marrative-priiate--aata/

---

Published as a conference paper at COLM 2024
8 Conclusions and future work
Consistent conceptualization, especially when addressing mission critical data, is key in industrial applications. We have shown that inconsistencies creep LLMs even when using common vocabulary and even after prompting the system with targeted content.
There are some natural future directions that emerge from these insights. The first looks to identify fine-grained knowledge issues and systematically evaluate an LLM for them. This may be done by mapping the knowledge from a KG to richer, linguistically more challenging queries that users may realistically pose to the LLM or using train-of-thought factoring of the user query into simpler queries, like the ones we produce, may help in this mapping.
The second line or research is to allow for questions that require non-committal answers and thereby handle ambiguous contexts. For example, the question 'does pediatric surgeon always work with children?' may have a correct 'maybe' answer, as pediatric surgeons also work with teenagers. Part of establishing trust in the LLM to ensure that ambiguous queries are properly, and consistently, dealt with. We are currently working to address these complex scenarios in various domains, like Software, Natural Disasters, Music Genres, Academic Disciplines, Occupations, etc.
Acknowledgements
The authors thank Kush Varshney for his advice and support.
References
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. A review on language models as knowledge bases, 2022.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan- dra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023. URL https: /aagss:/g/ggggggaaa/ co/tiiuae/falcon- 180B.
Berkeley R Andrus, Yeganeh Nasiri, Shilong Cui, Benjamin Cullen, and Nancy Fulda. Enhanced story comprehension for large language models through dynamic document- based knowledge graphs. Proceedings of the AAAI Conference on Artificial Intelligence, 36 (10):10446-10444, Jun. 2022. doi: 10.1609 16099/aai.v36ii00 36610.22... URL https: //jo:://oj.... www.//oss.aaa.a.oooooaaaaa.....a http://ppp//attaattttleeeee/////
Siyuan Cheng, Ningyu Zhang, Bozhong Tian, Xi Chen, Qingbing Liu, and Huajun Chen. Editing language model-based knowledge graph embeddings, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan
Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros,
Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-fineture language models, 2022. URL https: /tthg//gggggggggee www.uggingaae..... wtt.faacc.ccoooggeeee co/goggle/ flan-t5-xl. Also https:/ htt///ggggfac...cocc// / google/ google//lan-t--xll. flxx--t-xxl.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and Improving Consistency in Pretrained Language Models. Transactions of the Association for Computational Linguistics, 9:1012-1031, 12 2021. ISSN 2307-387X. doi: 10.1162/ tacl 00410. URL https: /ttp///oioooo// org 1162/ tacl 00410.
10

---

Published as a conference paper at COLM 2024
Fredo Erxleben, Michael Günther, Markus Krötzsch, Julian Mendez, and Denny Vrandečié. Introducing Wikidata to the Linked Data Web. In International Semantic Web Conference, pp. pp. 50-65. Springer, 2014.
Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. Talk like graph: Encoding graphs for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: //tpn//eee./ee.eee....... hwt.//ee///teeee www.//orum/meeeee I=IuXR1CCrSS CCrSi.
Zhangyin Feng, Weitao Ma, Weijiang Yu, Lei Huang, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting liu. Trends in integration of knowledge and large language models: A survey and taxonomy of methods, benchmarks, and applications, 2023.
Dieter Gernert. Implicit, incoherent and inconsistent knowledge. Frontier Perspectives, 14: 26-33 26-31, 03 2005.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey, 2023.
Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. Accurate, yet inconsistent?
consistency analysis on language understanding models, 2021.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://huggingface co/misttalaaiMMstraaa http:///ggggggaeeee////////aa/aa
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023.
Dimitrios A Koutsomitropoulos and Aikaterini Kalou. A standards-based ontology and 2017 support for big data analytics in the insurance industry. Ict Express, 3(2):57-61, 2017.
Philippe Laban, Wojeiech Kryšcinski, Divyansh Agarwal, Alexander R. Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. Lims as factual reasoners: Insights from existing benchmarks and beyond, 2023.
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning, 2024.
Kevin McAreavey, Weiru Liu, and Paul Miller. Computational approaches to finding and measuring inconsistency in arbitrary knowledge bases. International Journal of Approximate Reasoming, 5569)11659-1693, 2014. ISSN 0888-613X. doi: https:/ //oioor// 2.11016/.jarr22044.6..0000 URL https: http:///wwwwwwccceeeeeeeeeeeee wtt.///cccccccceeeeeecc ccmm///ciiene///tiiiee/eiiiiiii S088861 3X1 4001029.
Mistral.AI. Blog post: Mixtral of experts - a high quality sparse mixture-of-expertss, 2023a. URL UtL.ttttss//ugggggaaaee.oo//// http:://wwngnniaa.......sssssrrr -anaal--tttt--ntttttt//// Instruct-v0.1.
Mistral.AI. Theeboke/mixtrrl/xxbww..... 2023b. The
URL
Ngoc Thanh Nguyen. Advanced Methods for Inconsistent Knowledge Management. 01 2008.
ISBN 978-1-846288888-3. doi: 10.1007/978-1-8848
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge saaohsssgegraphs:: A roadmap. IEEE Transactions on Knowledge and Data Engineering, pp. 1-20, 2024. ISSN 2326-3865. doi: 10.1109/ /t// 20.4.3202......000...
URL http://dx dow..or//..... ww///////////////// 2024. 3352100.
Zhenghua Pan and Shengli Zhang. Five kinds of contradictory relations and opposite relations in inconsistent knowledge. Fourth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD 2007), volume 4, pp. 761-7661666 2007.
11

---

Published as a conference paper at COLM 2024
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases?, 2019.
Sai Sathiesh Rajan, Ezekiel Soremekun, and Sudipta Chattopadhyay. Knowledge-based consistency testing of large language models, 2024. URL /tRL.tttss///rrrvoorr/a///// 12830.
Pritish Sahu, Michael Cogswell, Yunye Gong, and Ajay Divakaran. Unpacking large language models with conceptual consistency, 2022.
Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicolo Fusi. Tag-Ilm: Repurposing perpoos llms for specialized domains, 2024.
Aarohi Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023.
Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowledgeable are large language models (Ilm)? a.k.a. will ilms replace knowledge graphs?, 2023.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. U12: Unifying language learning paradigms, 2023.
URL http:/hugginggaaeee ctl coo/golle/flanuuii
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https:// huggingface htt/metaallmmaaaaaaaaa 2-13b-chat-hhf Also https:/ https://gihuub.co/ htt///gtttubbommmeee Hlama/llama/ /aonaa//lbb///h /et//mmii//ii Swwp://OOE............
Denny Vrandečić. Wikidata: A new platform for collaborative data collection. In Proceedings of the 21st Int. Conf. on world wide web, pp. 1063-1064, 2012.
Denny Vrandečić and Markus Krötzsch. Wikidata: A free collaborative knowledgebase. Commun. ACM, 57(10)778-85, 2014. ISSN 0001-0782.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022.
Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. Give us the facts: En- hancing large language models with knowledge graphs for fact-aware language modeling, 2024.
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Man- ning, Percy Liang, and Jure Leskovec. Deep bidirectional language-knowledge graph pretraining, 2022.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they dont know?, 2023.
Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengiie Ren, Shu Wu, and Zhumin Chen. Knowledge graph enhanced large language model editing, 2024.
Zheng Zhang, Chen Zheng, Da Tang, Ke Sun, Yukun Ma, Yingtong Bu, Xun Zhou, and Liang Zhao. Balancing specialized and general skills in 1lms: The impact of modern tuning and data strategy, 2023.
12

---

Published as a conference paper at COLM 2024
A Cluster dataset construction
Starting with the seed concept' 'orthopedic pediatric surgeon', we automatically generate data set that comprises 109 clusters, with total of 584 questions, which include 4 different linguistic forms per query, so we have approximately 146 semantically different queries (some of the property clusters have 2 questions only per medical specialty). The size of the dataset is as follows:
positive edge clusters.
66 negative edge clusters. The number of these can be adjusted with a parameter. Obviously, in small hierarchy, looking for unrelated nodes becomes harder if the top number is higher.
15 inverse edge clusters.
12 path clusters.
11 property inheritance clusters.
Each cluster, regardless of type is made up of the following:
Expected answer. 'yes' or 'no'.
Source. This the source node in the directed graph
Target. The target node of an edge or a path cluster.
Questions. These are generated from fixed linguistic patterns for subConceptOf and for property edges. For example: "is a orthopedic pediatric surgeon medical spe- cialist ?", "is orthopedic pediatric surgeon type of medical specialist ?", "is every orthopedic pediatric surgeon a medical specialist ?" and "is a orthopedic pediatric surgeon also medical specialist ?" for an edge cluster with source 'orthopedic pediatric surgeon'.
The corresponding statements to the questions above: "a orthopedic pediatric surgeon is medical specialist", "a orthopedic pediatric surgeon is type medical specialist", "every orthopedic pediatric surgeon is medical specialist" and " orthopedic pediatric surgeon is also medical specialist". These statements are used to create augmented context to improve the consistency of the LLMs.
Statements.
Our full json dataset is provided in the supplementary file.
B The finance domain
To prove how pervasive the inconsistencies in LLMs are, we tried several domains: govern- ment agencies, corporate occupations and finance. In this later case, we created a simple graph with just one path, so we can test the first three properties of conceptualization.
subConceptOf
Home equity loan
Mortgage loan
credit
loan
Financial product
Figure 4: Finance domain: home equity loan path
This dataset has 75 edges and only 4 hierarchy clusters. The results below may not be statistically significant but we include them here because even in this Hello World example,
13

---

Published as a conference paper at COLM 2024
get inconsistencies in similar percentages above. Also, it is interesting that even after specifically adding the context in the prompt, we don't necessarily improve the performance in all models with respect to this simple path.
Finally, the fact that after prompting some models degrade slightly in performance (probably without statistically significance), indicates that only prompting may not be the only answer.
% incomp. clusters (84)
% incons. clusters (84)
% incomp. clusters (84)
incons clusters (84)
%improve incons.)
LLM name
google/llan-45-x
2.38
25
9.33
15.47
google/flan-t5xxl
google/fann-l12
1.19
25
0
10.86
15.47
mete-llamalllaa-2213818
1.19
5.95
2.38
9.33
-2.38
meta-liamalllmma-22 2-70lllcmma------hatthh
1.19
6.95
2.38
10.06
-3.57
liluae/falcon-180b
10.71
2.38
-3.57
mistelaiimisraa/---nnsruuu-2222
1.19
1.19
1.19
6.66
-5.95
1.19
4,76
0
1.33
-2.88
hhebookeimkkrralkxx--00--...gg
1.19
Simple prompt
Prompt augmented by context
Figure 5: Finance domain: eval with simple prompt and with context
C
Ethics statement
Our datasets were created by ourselves using publicly available wikidata ontologies. The content of our knowledge graphs is common knowledge and we do not involve any human subjects for data generation or validation.
One of the key motivations of our proposed approach is to enable users calibrate trust in LLMs and improve the consistency LLMS in specific domains to make them more trustworthy. We believe that exhaustive testing using methods such ours is necessary in any high-stakes application. potential issue in using exhaustive testing methods such as what propose is that lot of inference calls need to be made to LLM and this increases their power consumption. However, this is mitigated by the fact that this needs to be done only for the domains of application in which the LLM is used. This testing will also reduce downstream harms for users that may happen due to inconsistent knowledge in the models.
D Reproducibility statement
We provide the dataset that we generated in the supplementary material. This can be used to test any model in the domains that we presented in the paper. We also discuss the methodology by which we created the dataset in sufficient detail in the paper. Any knowledgeable reader can use similar methodology to test their own model in domain of interest.
14


Reviews:
[{"review_id": "IjLCIap7Uj", "reviewer": "Reviewer_WH6c", "summary": "This paper presents an approach that leveraging ontologies and knowledge graphs to evaluate the conceptual knowledge understanding capabilities in several publicly available LLMs. The paper also presents simple prompting strategies to improve the domain concept knowledge in LLMs being evaluated.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to review our paper. We count on this insightful feedback in order to improve our work.\n\n>Dealing with more complex relationships for domain understanding:\n\nWe completely agree that domain understanding requires us to deal with many more complex relationships. In fact, when we began exploring this topic, we asked models questions about more complex relations, such as those used to describe events, processes, and other kinds of dynamic knowledge. However, for the purposes of this paper, we decided to start with concepts and relations that were simple and universal enough that there could be no doubt that most LLMs would have been exposed to them during training. Additionally, we wanted to establish a foundational set of axioms that can help us in the automatic generation of datasets. Had we chosen more complex/rich/specialized/esoteric domains, we may have hidden the issues already obvious with these very basic axioms for conceptualization.  It turns out that even with these very simple graphs and vocabulary, inconsistencies are already glaring.  \n\nAs part of our future research, we will be working to generalize our approach to many common domains used in industrial ontologies with more, richer relations and we are looking at how to interpret and flag the inconsistencies in these more complex, interrelated concepts. We can state this rationale more explicitly in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents an interesting approach to assess the consistency and coverage of conceptualization in LLMs by automatically creating test cases based leveraging ontologies and knowledge graphs. Using these automated tests, the authors are demonstrate that several publicly available LLMs show inconsistent domain knowledge. The authors apply simple prompting approaches which allow the LLMs to respond with better understanding of certain medical domain concepts.\n\nThe paper is well written and easy to understand. The experimental design and evaluation setting is decently thorough.", "reasons_to_reject": "The paper presents an approach to evaluate several LLMs (lack of) understanding of medical domain concepts using simple IS-A hierarchical knowledge. However, domain understanding requires us to deal other more complex relationships including part-of, cause/effect, inference, entailment, etc. This paper would be better accepted by the conference audience if such evaluations were presented in the paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vd3dX8LLbQ", "reviewer": "Reviewer_71gk", "summary": "This paper studies the capability of LLM to reason with conceptual knowledge. It aims at testing LLM performance by evaluating conceptual consistency and the ability of handling logical inconsistencies. It proposes enhancing LLMs through knowledge integration.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your review and helpful feedback. Your input is greatly appreciated and will enhance the quality of our paper.\n\n> Definition of inconsistency\n\nWe define an inconsistency as a set of statements which violates one or more axioms which, in our case, are defined in section 2. For example, ‘a pediatric surgeon is a surgeon’, ‘a surgeon is a medical specialist’ and ‘a pediatric surgeon is NOT a medical specialist’ is an inconsistent set of statements with respect to the transitive property.  \n\nWe have designed our question clusters so that they can automatically reveal an inconsistency with respect to a given axiom when we have both expected and unexpected answers. \n\nWe will amend the paper to give specific examples of inconsistencies with respect to each of the axioms in our section 2. \n\n> Comparison to ontology completion\n\nIn our view, ontology completion methods are intended to solve a slightly different problem than the one we explore. While ontology completion methods seek to remedy gaps in knowledge from an ontology, our method seeks to discover gaps and inconsistencies in the latent knowledge of LLMs. Specifically, we use the ontology to systematically generate questions about concepts and properties that can be asked of an LLM in different ways. When the LLM responses to these questions are inconsistent, it indicates that the underlying LLM knowledge is inconsistent. If we have misunderstood your point, we can hopefully better understand it during the discussion period.\n\n> Would it be possible to consider reducing inconsistency and improving the coverage of domain concepts as NLI tasks and fine-tune an LLM accordingly?\n\n Yes, it should be possible and, in fact, our datasets map the cluster questions to their corresponding knowledge statements (i.e. premises). In this paper, we wanted to define the foundational -and pragmatic- axioms for conceptualization so we could generate these datasets automatically. However, more work is needed to define the right set of entailments when knowledge is missing or inconsistent. NLI is a very interesting task to consider in this regard, although care must be taken to avoid the same inconsistent behavior from LLMs that we observe in our experiments (since state of the art NLI systems are also built upon LLMs).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and illustrated by figures.\n- The experiments and the use case show the potential of the proposed approach.", "reasons_to_reject": "- Missing formal definition of inconsistency. \n- Missing comparison with approaches of ontology completion (instead of adding plausible rules, removing plausible inconsistency)", "questions_to_authors": "Would it be possible to consider reducing inconsistency and improving the coverage of domain concepts as NLI tasks and fine-tune an LLM accordingly?", "extra_scores": {}}, {"review_id": "vvPSYZdjOE", "reviewer": "Reviewer_AJmS", "summary": "This paper devised methods that utilize ontologies to assess the consistency and coverage of conceptualization in Large Language Models (LLMs). By automatically generating test cases from knowledge graphs or ontologies, they effectively evaluate the models. Moreover, they have found that many prominent LLMs with publicly available weights exhibit inconsistencies in their knowledge, even when confronted with fundamental, small ontologies. However, through the application of straightforward prompting strategies, they have shown that it is possible to reduce these inconsistencies and improve the coverage of domain concepts across various LLMs with publicly accessible weights.\n\nStrengths:\n\n1.A more detailed analysis of the hierarchical confusion across multiple large language models has been conducted;\n2.The paper is well-crafted, featuring lucid clarity and a straightforward logical flow that is effortless to follow.\nWeaknesses:\n1.The author focuses solely on hierarchical confusion while lacking necessary discussions on more comprehensive issues such as mixing of domains, inaccurate Generalizations, and contradictory information.\n2.In terms of details, the analysis of the results in Table 2 and Table 3 is relatively rough. It just lists the results of different large language models on the issue without carefully analyzing the possible causes and influencing factors.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your helpful feedback. Your input is highly appreciated and will guide the rewrites of our paper. \n\n> Lacking a more comprehensive discussion of complex, realistic domains as well as inaccurate generalization and contradictions: \n\nWe wanted to create a simple, foundational -and automatic- baseline that we can build upon in future work. While we agree there is more work ahead of us to address the nuances of incomplete and inconsistent knowledge in more complex, realistic domains, we didn’t want to obscure the glaring issues we have already found when dealing with simple concepts and basic conceptualization axioms (in section 2). Our initial goal was to test many different LLMs of various sizes on a common, simple set of concepts and a single relation and to show that, even in this limited case, all LLMs failed to deliver consistency one way or another, even though they were obviously able to process most of the semantic entailments of a concept (with respect to our axioms). \n\n>Careful analysis of possible causes and influencing factors\n\nit is clear more work is needed to come up with general enough answers. As we point out in the next response, we did look into a possible correlation between the model size (number of parameters) and inconsistencies, and while there was a negative correlation, it wasn't strong. Lines of future research include (1) Using larger datasets in several domains to give us an accurate picture of which features of an LLM affect consistency.  (2) An analysis of where the inconsistencies happen (how high in the KG hierarchy) and the interrelation between the different axioms. However,  overanalyzing a small graph, may lead to faulty conclusions. We can add these points in our discussion and future work.\n\n> Is there a connection between the inconsistency of conceptual hierarchy and the parameter size of large language models? \n\nWe looked into this. There was a negative correlation between model size and volume of inconsistencies (correlation -0.36, p-value 0.046) across 32 models. However, a larger analysis across more domains would be needed to support a general conclusion. We are looking into other factors, like the topology of the network and layers or even the data they are being trained on. There is much future work to be done in this area and we believe our work sets a solid foundation for this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.A more detailed analysis of the hierarchical confusion across multiple large language models has been conducted;\n2.The paper is well-crafted, featuring lucid clarity and a straightforward logical flow that is effortless to follow.", "reasons_to_reject": "1.The author focuses solely on hierarchical confusion while lacking necessary discussions on more comprehensive issues such as mixing of domains, inaccurate Generalizations, and contradictory information.\n2.In terms of details, the analysis of the results in Table 2 and Table 3 is relatively rough. It just lists the results of different large language models on the issue without carefully analyzing the possible causes and influencing factors.", "questions_to_authors": "Is there a connection between the inconsistency of conceptual hierarchy and the parameter size of large language models? In other words, the larger the model parameter size, the lower the possibility of inconsistency in the conceptual hierarchy will be significantly reduced?", "extra_scores": {}}, {"review_id": "YaLLrQN4WT", "reviewer": "Reviewer_Gt8x", "summary": "This paper seeks to address a well-known issue in LLMs: inconsistency in knowledge. The strategy is to use ontologies to extract Is-A relations and then study them in LLMs, with a focus on applicability to real-world tasks. They find that models have a lot of knowledge, but that they are inconsistent in this knowledge. They show improvements by using KB-based prompting.\n\nThis paper addresses an important topic, is clearly written, and has a promising and interesting approach. I would be happy to see it published with some revisions, but I have a few comments and suggestions.\n\nFirst, a strength is that the description of ontologies (transitivity, etc) is very clear and accessible. This would I think be valuable for the audience. But, perhaps as a trade-off, I found Section 6 very rushed, with most of the important details about the key prompting method in the Appendix. I would recommend putting more detail in the main text, perhaps with an illustrative figure for the prompting method. Some of the relevant detail is in Fig 1, but it would be good to focus on exactly the prompting part.\n\nSecond, I was somewhat surprised by the discussion of not counting cases where the LLM gives mealymouthed answers like “it is possible that an orthopedic pediatric surgeon may work in the field of infection control, however this is not their primary field of occupation, which is orthopedic surgery and pediatric surgery”. This is increasingly a strategy imposed on LLMs. It’s perhaps fair (although I’m not sure) to count these as inconsistent vis a vis yes/no. But, if a lot of the apparent inconsistency comes from such examples, it would be good to know. The scenario where an LLM hedges like that in one case and simply says “no” in another feels very distinct from a yes/no asymmetry. I would address this explicitly in the main text.\n\nA ubiquitous question: I was left wondering what, if anything, should we take away from the differences in model performance. It’s nice to test a lot of models, but are there hypotheses as to if/why some models might be better than others on this particular task.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and for raising some thought-provoking questions regarding the ‘hedged’ LLMs. We're looking forward to addressing these points as we refine our paper.\n\n>More detail on the prompting method.\n\nGiven that we will be given an extra page to elaborate, we will provide samples of our prompts and discuss several strategies we have used in prompting. \n\n>The counting of hedged LLM answers as inconsistent makes me wonder if the inconsistency is overstated.\n\nWe have been curious about the hedged answers as well. The issue is that the LLMs were not consistent about them either. When asked with slightly different language, or referring to different edges in the graph the LLM was categoric in their answers. Out of 4600 questions that were supposed to be answered ‘no’ (across all models), these nuanced answers happened only 17 times. Even if we only count the 2700 questions about negative inverse edges (“is every surgeon a pediatric surgeon?\", “is ‘surgeon’ a subtype of ‘pediatric surgeon’?”), where the system would be reasonably expected to say ‘that’s not always the case’, LLMs (all of them) didn’t answer consistently in all incarnations of the question. Hence, we considered the 17 ‘hedged answers’ as outliers for the purposes of this paper. \n\nThis is not to say that a larger, more complex dataset would reveal some insight on these questions. And we may want to provide ‘automatic’ answers in our dataset (‘not always’, ‘not necessarily’ and so on) so we can take them into account. We’ll look into this as part of our future work. \n\n >Are cafe racer and naked bike real examples?\n\nAll the concepts mentioned in the paper are ‘real’, in the sense that are described (and curated by several writers) in Wikidata. \n\nCAFE RACER (https://www.wikidata.org/wiki/Q1025508) \n\nNAKED MOTORCYCLE, (synonym of) NAKED BIKE ( https://www.wikidata.org/wiki/Q1964027)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This tackles an important topic in a clear way: making LLMs more trustworthy and consistent in the way that KBs are while retaining strengths of LMs.\n- It’s clearly written and might help introduce some of the ontology-based thinking to the LLM audience in a helpful way. I found the literature reviews useful for understanding the place in the literature.\n- The method seems to work and show meaningful improvements, and it seems scalable.", "reasons_to_reject": "- Not really a reason to reject, but I’d recommend more detail on the prompting method.\n\n- The counting of hedged LLM answers as inconsistent makes me wonder if the inconsistency is overstated.", "questions_to_authors": "I take it cafe racer and naked bike are real examples? I wasn’t familiar with them and so was unsure if they were supposed to be “novel concepts”. \n\nI was briefly thrown by the term “field of occupation”, which doesn’t seem idiomatic to me.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper shows an automatic way to use ontologies to create test cases that test the ability of LLMs to understand concepts. The authors report many conceptual inconsistencies and propose prompting methods for reducing their effect. All reviewers agree the paper is written clearly, has a clear message that is demonstrated convincingly and should be accepted.
Decision: Accept