Towards Concept-Aware Large Language Models
Authors: Chen Shani
Jilles Vreeken
Dafna Shahaf
Venue: EMNLP 2023 Findings (2023)
Keywords: Concepts
Pretrained Large Language Models
TL;DR: Exploring to what extent do pretrained LLMs grasp human concepts and how to enhance their understanding of concepts with and without training.

Abstract: Concepts play a pivotal role in various human cognitive functions, including learning, reasoning and communication. However, there is very little work on endowing machines with the ability to form and reason with concepts. In particular, state-of-the-art large language models (LLMs) work at the level of tokens, not concepts.

In this work, we analyze how well contemporary LLMs capture human concepts and their structure. We then discuss ways to develop concept-aware LLMs, taking place at different stages of the pipeline.
We sketch a method for pretraining LLMs using concepts, and also explore the simpler approach that uses the output of existing LLMs. Despite its simplicity, our proof-of-concept is shown to better match human intuition, as well as improve the robustness of predictions. These preliminary results underscore the promise of concept-aware LLMs.

Towards Concept-Aware Large Language Models
Chen Shani", Jilles Vreeken', Dafna Shahaf
The Hebrew University of Jerusalem, Israel CISPA Helmholtz Center for Information Security, Germany
8 (chenxshani, dshahaf)@cs .ej.eeesss.............. ac. il vreeken@cispa de
Abstract
Thus, they are of interest to Al researchers wishing to endow machines with such abilities.
Concepts play a pivotal role in various human cognitive functions, including learning, reason- ing and communication. However, there is very little work on endowing machines with the abil- ity form and reason with concepts. In par- ticular, state-of-the-an large language models (LLMs) work the level of tokens, not con- cepts.
The representation of concepts has been studied in NLP, ML, and knowledge representation (Fuma- galli and Ferrario, 2019; Davis and Marcus, 2015; Gardenfors, 2014: Speer al., 2017), where they often view concepts as fixed, shallow structures representing some set of entities. For example, the work on Chen et al. (2020) concepts are flat sets of context-independent entities. However, recent studies suggest concepts are more flexible and dy- namic (Gabora et al., 2008); unfortunately. Al still struggles with accounting for the creative, context- sensitive manner in which people employ concepts.
In this work, analyze how well contempo- rary LLMs capture human concepts and their structure. We then discuss ways to develop concept-aware LLMs, taking place at different stages of the pipeline. We sketch a method for pretraining LLMs using concepts, and also ex- plore the simpler approach that uses the output cxisting LLMs. Despite its simplicity, our proof-of-coneepii is shown better match hu- man intuition, well as improve the robustness predictions. These preliminary results under- score the promise of concept-aware LLMs.
In this work we focus on adding concepts to large language models (LLMs). Recently, LLMs (Yang et al., 2019; Raffel et al., 2020; Thoppilan et al., 2022; Scao et al., 2022; Zhang et al., 2023; Bubeck et al., 2023) gained immense popularity, achieving SOTA results across the board. However, they all work at the level of tokens, not concepts.
1 Introduction
This is problematic for even the most fundamen- tal LLM task perplexity- -baed text completion. Ranking by string probability is distorted by sur- face form competition: different tokens compete with each other, even if they represent the same concept ("mother" and "mom") (Holtzman et al., 2021). In other words, the probability mass of a concept distributed across many different tokens, distorting the ranking.
Concepts are the glue that holds our mental model of the world together. It is hard to see how any intelligent agent could do without them. While there is no agrced-upon definition of concepts, one can think of them as the mental representations that enable us to identify objects and events be- longing certain categories, communicate about them, and comprehend new situations in terms of previous ones: when we encounter a new situa- tion (e.g., restaurant), we draw inferences about using concepts we have already formed ("menu", "waiter")..
The problem runs deeper than mere synonyms. For example, consider the sentence "I can't get home for the holidays because of the [MASK]." The completions "snow", "blizzard", "weather", and "slippery roads" are not synonyms per se, but they correspond to the same concept bad weather leading to hazardous driving conditions and we believe that an LLM should treat them as such.
Concepts can be concrete ("soup") abstract ("tasty"). They can also be complex, c.g., "good winter beach destinations". While there is lively debate on their exact nature, researchers agree con- cepts play a pivotal role in various cognitive skills such as reasoning, categorization, learning, planning, and decision-mann makiing (Murphy, 2004).
We stress that concepts are context-dependent; for example, while "snow" and "blizzard" are sim- ilar in the context of the sentence above, they

---

are very different for the sentence "1 love eating [MASK] cones."! Thus, we cannot rely on knowl- edge bases (such WordNet (Miller, 1995)) for gen- erating static, context- concepts to be used for training LLMs.
To automatically extract TypeOf-relations we extracted the direct hypernyms and hyponyms of the 100 ETs' first sense using WordNet. This results in data of the following form (ET in bold): (foowwear) <- shoe +- {anklet, baby shoe, moccasin, oxford, sandal, running shoe, (canine, domestic animal) dog (new- foundland, hunting dog. dalmatian, corgi, ...) On average, each ET had 9.5 hyponyms and 1.1 hypernyss. Note that WordNet is noisy, and some- times the first sense is not the intended one. We added a human baseline to show the data is of sat- isfactory quality.
We take the first step towards concept-aware LLMs, exploring the following questions:
RQ1: How well do LLMs capture concepts?
RQ2: How well do LLMs match human organi-
zation of concepts?
RQ3: How can enhance an LLM in terms of concepts, with or without retraining?
We first show that contemporary LLMs capture human concepts some extent, but they are still far from humans (RQ1). We then find that LLMs violate many of the principles of human concept organization, exhibiting some inconsistency (RQ2). Lastly, we explore RQ3 from two different an- gles: first, we sketch method to pretrain concept- aware LLMs. Next, we implement proof-of- concept model-agnostic method to shift any off- the- shelf pretrained LLM from token- concept-level with no further training. Our method improves both the ranking and robustness of the underlying LLM.
LLM probing. We test the concept-Type0ff hi- erarchy of four representative LLMs: BERT-base- uncased, T5-large, GPT-davinci-003 (Legacy), and GPT-4.3
Since GPT-based models support question- answering, we use binary questions: "Is <ET> a type of <hypernyms?", and "Is <hyponym> a type of <ET>?"4
BERT and T5 are not optimized for question- answering; thus, we query them follows: <<ET> is type of [MASK]." and "<hyponym> is type of [MASK]," and search for the desired hypernym and ET respectively along the top-k completions. The LLM is correct only the answer is within the top-k completions.
While we present here merely a promising proof- offconncpt,, our underlying objective of endowing LLMs with concepts holds tremendous promise for the next-generation of LLMss. We hope our work will spur further research, paving new roads into this exciting new territory.
As sanity check, added a human bascline using random sample of 100 Type0f (ET, hyper- nym) and 100 TypeOf (hyponym, ET) questions used for querying the GPT models. To balance a bit the answer distribution, We added 20 negative examples per question type (of the form: "Is <ET> type of <another ET's hypernym>"", "Is <hy- ponym> a type of <different ET>?"). The negative examples were not part of our analysis and were included to avoid annotators seeing only positive examples. We used six members of research group for the annotation. Their mean response vari- ance is 0.18 for the hyponyms and 0.07 for the hypernyms, showing they are calibrated.
2 RQ1: How well do LLMs capture concepts?
this section, we explore to what extent LLMs grasp concepts. While concepts can be abstract and complex, here we focus on concrete and simple ones, since these are the basic building blocks of human concepts (Varela et al., 2017).
Dataset. To probe LLM ubilitics, everyday things (ET's) dataset (Gu al.. 2022), containing natural and man-made items everyone should be familiar with (egg, dog, elevator, table, etc.).
the 100
Perhaps the most basic relation between con- cepts is TypeOf called IsA). This relation governs the hierarchical organization of concepts humans possess. For example, we all know that sandal is type of shoe (denoted TypeOf shoe)).
"We ran the GPT-4 experiments on October 2023.
*We queried both GPT models on 200 random combina- tions of "Is <FT a type of <ET2>?" ("Is dishwasher a type tent?"). returned "yes" 0(!) times.
*We are aware that this task is potentially harder than GPT's. However, testing GPT as it were masked LLM would likely lead to sub-optimal performance. We hence prefer to explore two different scenarios, increasing the ro- bustness of the analysis, rather than artificially leveling the playing tield.
Snow cones are shaved-ice desserts. Blizzard cones are appurently beak-shaped face masks from the 1930s.

---

Mean Hypemnym Concept Retrieval for 100 ETs
Mean Hyponym Concept
Retrieval for 100 ETs
1.0
1.0
Mean Retrieval
0.8
Mean Retrieval
0.8
BERT
CPT-3.5
GPT-4
0.4
Human
0.4
15
0.2
0.2
GP1-3.5
Human
10
20
30
40
50
10
20
30
40
50
K
K
Figure 1: [Higher means better] LLMs concept retrieval function of For each ET, we measure how well it retrieves in its top-k completions the ET's hypernyms (left plot) and hyponyms (right plot). Since GPT and humans answer yes/no questions, their performance does not change as a function of K.
Mean Hypernym Asymmetry Preservation ETs
Mean Hyponym Asymmetry Preservation for 100 ETs
1.0
1.0
Mean Preservation
Mean Preservation
0.6
0.6
0.4
0.4
0.2
GPT-3.5
GPI-3.3
GPT-4
GPT-4
0.0
0.0
10
50
Figure 2: [Higher means better] LLMs' asymmetry preservation as a function of K. For each ET, we measure how well it preserves asymmetry by not returning the relevant item its top-k completions (measured only using Type0f relations the LLMS correctly retrieved in RQI). We probe both for hypernyms (left plot) and for hyponyms (right plot). Since GPT supports cesstion-answering its performance does not change as a function K.
Results. We plot the accuracy (or concept retrieval, all examples
ples of the human Type0f hierarchy of concepts: asymmetry, transitivity, and property inheritance.
positive), as function of for all four LLMs and humans (Figure 1). The re- sults indicate that LLMs capture concepts to some extent, but is far from perfect and human base- line. Models achieved (at k=50) 30%-83% retrieval for the hypernyms (human baseline is 88%) and 43%-60% for the hyponyms (human baseline is 96%). Interestingly. the GPT-based models seem grasp general concepts better (hypernym level), whereas BERT and T5 perform better more specific concepts (hyponym level).
Asymmetry
Type0f(A, B) - Type0f A)
The human TypeOf relation between concepts is asymmetric; if sandals are shoes, shoes are not sandals. To measure how well an LLM preserves asymmetry we take all the TypeOf relations it iden- tified in RQ1 and query the other direction. If an LLM correctly retrieved TypeOf (sandal, shoe), we now query for TypeOf (shoe, sandal) using the same querying methods and LLMs. the LLM did not retrieve this other direction - successfully preserved the asymmetry principle.
3 RQ2: How well do LLMs match human organization of concepts?
this section, we explore concept organization. We focus on three agreed-upon organization princi-
In Figure 2 we see that all four LLMs preserve

---

asymmetry better for hyponyms compared hyper- nyms (stronger trend for BERT and T5). Overall, LLMs do preserve asymmetry, but they are still far from a complete understanding of directionality.
R(hyper) 3 R(ET)
R(hyper) 4 R(hypo)
R(hyper)
LLM
R(hypo) R(ET)
BERT
0.85
0.80
0.92
T5
0.56
0.68
0.93
Transitivity
GPT-3.5
0.73
0.72
0.85
(ypeOf(A, B) A (B, C) TypeOf(A, C)
GPT-4
0.73
0.73
0.85
Table 1: [Higher means better] Property inheritance: when the LLM knows that property holds for hyper- nym of an ET, check how often it knows that R holds for the ET (left), for its hyponyms (middle) and for its hyponyms given that knows R(ET) holds (right).
TypeOf is transitive relation; if TypeOf (shoes, footwear) and TypeOf shoes), then (sadaas, sure whether LLMs preserve transitivity, we collect for each LLM all the pairs it correctly identified both (EyT hypernym), TypeOf ET)). We then query the LLM for the relation between the hyponym and the hypernym directly (skipping the middle level of the ET). We use the queries "Is <hyponym> a type of <hypernym>?" (for GPT), and "<hyponym>> (BERT and T5; we then search for the hypernym along the top-k=50 completions).
(sandals, footwear). To mea-
To test for LLMs' property inheritance, we first enrich our dataset with attributes of the hypernyms. We used Quasimodo, commonsense knowledge base that focuses on salient properties that are typ- ically associated with certain objects or concepts (Romero et al., 2019). It consists of (object, predi- cate, subject) triplets, e.g., "(footwear, affect, skele- tal system)", "(footwear, has property, manufac- tured)". To ensure high-quality properties, set the triplet saliency score threshold to 0.9. We col- lected 771 triplets for 37 ETs. The mean number of triplets per ET is 20.84 (STD=26.14),
type of [MASK]"
Figure 3 depicts the results. All models exhibit some level of transitivity, with GPT performing best, followed closely by BERT (starting around k=15). Interestingly, GPT-3.5 performs slightly better than GPT-4.
We use the first paraphrase of wordtune? to turn triplets into sentences and mask the object: "(footwear, affect, skeletal system)" + "The skele- tal system is affected by footwear."
Mean Transitivity
Preservation for 100 ETs
1.0
"The
[MASK] is affected by footwear.".
Mean Preservation
0.8
For each sentence, we query the LLMS to see if they believe it to be true (k=50 for BERT and T5), For each LLM, if it believes a hypernym triplet holds, also query it by replacing the hypernym with the ET (Table left column) and with its hy- ponyms (middle column).
0.6
0.4
BERT
0.2
3.5
GPT-4
We modified the sentences to questions for GPT by adding "?" at the end of the paraphrased sen- tence, and taking the first proposed paraphrase ("Does footwear affect the skeletal system?").
10
20
30
40
50
K
Figure 3: [Higher means better] Transitivity func- tion of K for Type0f relations correctly retrieved.
Overall, all models show some level of property inheritance (Table 1). Interestingly, all models are more likely to transfer relation from the hypernym to its hyponym if they correctly transferred it from the hypernym to its ET (middle vs. right columns), showing some robustness in LLMs' understanding of transitivity. Also worth noting BERT seems to preserve property inheritance better than the rest of
Property inheritance
(A, A R(B) - R(A)
If a property is true for a concept, it should hold for concepts below it in the Type0f hicrarchy. For example, if all footwear is manufactured, then all sandals are manufactured as well."
birds can fly, but we have yet to discover a flying penguin. Pht //wp://www. wordtune, com/
Note this has exceptions, c.g., penguins are birds and

---

0 Imput
Wikipedia categories
2 Filttring loss
Augmentations
8 = "At the party we had cake and [MASK]]
C
desserts Types food
TIS (C2)) &TS
, the party we had cake and 5Frozen desserte ca cream."
L TSS. [c LTS)
Dairy products
S = the party we had cake and KFrozen desserts, Tynes foods cream
Frozen food
T|. (G,, C.M) (8()
T: "ice cream"
Worid--siiinee
T1S. [c2))> ((S))
= tha party had and #Darr products.
Types of food, Daiyy products ice cream."
TS. [Gg. C.,, C.)) S
S. = the party we had cake and <Dairy products.
2TS. (C.. Cg)) < (2(T)S)
Dairy productas ice cream."
Figure 4: Overview of our sketch for method to pretrain concept-aware LLMs. The model is trained to choose the categories that will best assist in predicting the missing token(s).
the models.
out categories that are too narrow ("Independence Day (US) foods") or too wide ("World cuisine"), resulting in set of candidate concepts C.
To conclude RQ2, GPT models seem consis- tently better than BERT/T5 for asymmetry and tran- sitivity (for inheritance property BERT takes the lead), but all models do violate principles of human organization and exhibit some inconsistencies.
Next, iterate over all possible subset com- binations of C (or perhaps sample from them, if the set is too big) and interleave them with the text using a special token, resulting in augmented sen- tences {S] such as "At the party we had cake and <Dairy food, Frozen desserts> ice cream". We then mask T in all augmented sentences (SS) and filter out sentences that do not reduce the prediction loss over T, compared to the con-augmented masked sentence S. That is, we only keep the original sentences augmented with subsets of concepts that help predict the missing text T.
4 RQ3: How can we enhance an LLM in terms of concepts, with or without retraining?
In the previous sections, we have shown that LLMs' understanding of concepts is still limited. As we noted earlier, concepts play a pivotal role in var- ious human cognitive skills, including reasoning. categorization, planning, and decision-making: we strongly believe that equipping LLMs with notion of concepts can significantly expand their capabili- ties, bringing them closer to human performance. Endowing LLMs with concepts can take many forms, and may occur at different stages (pretrain- ing, fine-tuning, post-processing). We now lay the ground for future studies on concept-aware L.MMs.
We suggest to train an LLM over the augmented. non-masked sentences {S corresponding to con- cepts that do reduce the loss, resulting in model that teaches itself when and which concept(s) to prediet, and how to best incorporate the pre- dicted concept(s) into future token prediction.
The above suggestion is merely a sketch of the approach, and there are many possible variants. We can treat the problem of concept prediction as a probabilistic multi-class classification (over closed set of categories) or free text generation; can even take an approach similar to Toolformer (Schick al., 2023) and use external tools.
We start by proposing a sketch of a method for training concept-aware LLMs. We then take the simpler, proof-of-concepl approach of building con- cepts on top of the output of existing LLMs.
Approach 1: Rethinking LLM training
We envision training a model to choose subset from a closed set of concepts that will assist the most in complcting a missing sequence of text. Potential concepts could come from Wikipedia cat- egories, or perhaps some knowledge graph.
The sketch is inspired by the recent success of approaches that augment LLMs with additional information during pretraining (Aghajanyan et al., 2021; Schick et al., 2022; Izacard et al., 2022), and in particular by Toolformer (Schick al., 2023), that adds additional information only if is helpful.
See our suggested approach in Figure 4. Given input sentence, we sample span of text T that can be linked Wikipedia article (similar to Wu al. (2019)), such "ice cream",, and mask it. We denote by the masked sentence. We compile a subset of Wikipedia categories that the missing text T belongs to and their parent categories, filtering
Approach 2: Building on top of existing LLMs
We consider Approach 1 to be promising, but im- plementing it requires significant resources. Rather than rethinking LLM architecture and training, we now consider the other end of the spectrum, build-

---

Input
Input phrases
LM completions
went the parant teacher conference with my [MASK]."
went the teacher conference with my [MASK]]
[parents, mother, friends, mom. dad, father, teacher. family, sister. teachers)
As parent-leacher conference attandee, went with [MASK]."
[prrents, instimets, mother, dad, father, mom, get, family, heert, classes]
Output
mom, mother, mum, dad, father
parents, parent, families, family. folks
children, child, kids
sons, boys, daughters, girls
niece
brothers, brother, siblings, sister, sisters, twin
uncle, uncles, aunt, cousin, cousins
Figure 5: Overview our algorithm for extracting concepts from pretrained LLMs. We augment the input sentence by paraphrasing and predict the top & completions for each paraphrase. Next, filter (strikethrough) and perform agglomerative clustering using the LLM's token-contextual embeddings (centroid in bold). We assign new weights to each node the dendrogram (darker ranked higher, sorted according to weight).
and unlikely tokens
ing concept-aware LLM on top of existing LLMs, without any further training or external data.
"dad", followed closely by a concept containing "parents" and "family". Next concepts refer to chil- dren and other family members. As go higher, concepts become more general; the top node in the figure roughly corresponds "family member".
Previous works showed improved results of pre- trained LLMs without further training on tasks such word sense disambiguation, factualness and con- sistency (Levine al., 2020; Liu al., 2022). We believe our post-hoe method could similarly en- hance downstream tasks.
4.1 Algorithm
Figure 5 depicts our implementation. We give succinet overview, for details see the Appendix.
Concept-completion task. We start by testing ideas the fundamental LLM task of text completion (fill-mask.. Given a masked sentence S and an LLM, our goal is return ranked list concepts .... Cy. Each concept C; a non- empty set of tokens T. Ideally, concepts and their ranking should correspond to human intuition.
Augmentation. To augment the input sentence So we first retrieve the LLM top-k completions. We replace the "[MASKJ" token with the first com- pletion that is not stopword or sub-word and paraphrase using wordtune.? We then re-mask the input sentence So and mask its paraphrases [S,, SM-1.M-)).. resulting in M masked sentences.
Figure illustrates our main idea. In short, given a masked sentence Sy ("I went to the parent teacher conference with my [MASK]."), we retrieve the LLM top completions, paraphrasing Sn as an aug- mentation technique to increase robustness. To form concepts, we perform agglomerative cluster- ing using the LLM contextual embeddings.
Top-k completions retrieval. We retrieve the top-k (k=100) completions for each sentence in (Sp, S...SM 1). We count how often completions appear and remove infrequent ones? We extract the contextual embeddings (the token embedding from the last hidden layer using the masked in- put sentence Sp with the corresponding token completion). We use the contextual embedding, different tokens may belong to the same concept or not, depending on the context. The fact that the LLM's 's embedding yields meaningful clusters hints it somewhat captures concepts, which is
For clarity of presentation, Figure 5 shows only segment of the dendrogram, rather than going all the way to singletons. Nodes in the dendrogram have scores based on their tokens' weights and fre- quency in paraphrases (darker means higher). The bottom layer is sorted according to score. The first (bold) token in each node is the cluster centroid.
$ *Our code can found Setps://githhe com/ chenxshani http///addd--ocep-tt-aerro "Soe details in the Appendix.
In this example, the most likely concept (left) contains tokens such as "mom", "mother" and

---

line with our findings from RQI.
Mean Accumulated Score as a Function of K
Clustering & Ranking. We reduce the dimension- ality using PCA and t-SNE? and use agglomerative clustering to cluster the completions into concepts. We use agglomerative clustering as different thresh- similar to
Mean Accumulated Score
0.94
SERT
0.92
0.90
0.88
olds yield different concept-granulant the flexibility of concepts in humans. Each clus- ter is assigned with a weight that corresponds to: 1) the token with the maximal soft-max score, to avoid problems related to surface form competi- tion, and 2) the token with the maximal number of repetitions across augmentations' top-100 comple- tions, to increase robustness (a token that repeated frequently is probably very relevant).
0.86
0.84
0.82
0.80
2
4
6 8
Figure 6: [Higher means better] Concept-BERTTs and BERT's mean score at k=10. K denotes the number of clusters for Concept-BEBl and the number tokens for BERT. Our method's mean score at k=1 is 95% whereas BERT's is 84%. Concept-BERL BERT consistently better than BERT.
4.2 Evaluation
To evaluate our method, we focus on fill-mask task (completing a masked sequence of text).
Experiments. We use the ProtoQA dataset, consist- ing of questions regarding prototypical situations (c.g., "Name something you are likely making if you buy milk, eggs, sugar and cream.") (Boratko et al., 2020). We believe this setting relevant for our use case, as there are usually multiple relevant answers. To make the input similar to the language LLMs are usually trained on, we manually changed the questions to first-person statements ("I bought milk, eggs, sugar and cream to make [MASK].") We used 63 sentences to set our and an additional 100 sentences for evaluation." the most popular
weather, snow]. While the concept is coherent (i.e., cold weather conditions), some tokens are less- natural completions without their cluster-context (e.g., "temperature").
4.2.2 Ranking quality
We evaluate the quality of ranking by annotating all completions in the top- 10 concept- BERT clusters and top- 10 BERT tokens for all 100 input sentences. Three crowdworkers received the masked sentence and a possible completion. and were asked to clas- sify the completion as either: likely (score=1)) possible but unlikely (score=0.5)/does not make sense (score=0). See qualifications, compensation, and instructions in the Appendix. Note this evalua- tion cannot be automated, as wish to see if our concept-aware modification aligns the LLM's out- put with humans. completion's 's aggregated score its mean score across the three annotators (mean- variance across annotations=0.117 For concepts, we average all tokens in the cluster. Our score at k=1 is 95 % whereas BERT's is 84% (see Figure 6). Moreover, for all k values, concept-BERTT is consistently better than BERT.
We used fill-mask model (Devlin et al., 2018).0 We com- pute both BERT's top 100 completions and our manipulation, which we call concept-BERT, on top of BERT's output, resulting in ranked concepts.
4.2.1 Cluster quality
We measure the semantic coherence of clusters using the cosine similarity of word2vec's token em- bedding (first ten clusters for all sentences), The mean within-cluster similarity is 0.41, whereas the mean inter-cluster similarity is 0.12. For refer- ence, BERT's top-ten completion similarity 0.22. Hence, our clusters are coherent and distinct.
4.2.3 Completions in dispute
A closer examination highlights the distinction approach and
We now focus on completions for which BERT and concept-BERT disagree one predicts the comple- tion is likely, while the other does not (and vice versa). We believe these are the most interesting regions to evaluate our manipulation on. See exam- ples of disputed completions in Table 2.
between the pext-tooen-prediceeee ours. Consider the sentence "I can't get home for the holidays because of the [MASK]." and its cluster: (blizzard, cold, temperature, snowfall,
10 Most common according to Hugging Face: https:// pelinen aaaliinnnttttglllee tag=filllllleaa mask.
huggingface htto//odelspppp wppplliepppppppppepeepppppiepepp
We treat the middle 15% of the ranked lists as

---

Input sentence
Completion
BERT
Concept-BERT
I bought a fake [MASK] from street vendor.
jersey
0.08
0.79
When retired I started [MASK].
cycling
0.06
0.77
shudder
0.04
Whenever suffer from cold always [MASK].
rise
0.93
0.24
When to the beach use [MASK] to protect myself from the sun.
sticks
0.91
0.28
soap
0.74
0.03
always take my [MASKJ with me to the gym. squeezed myself into the [MASK].
laptop
0.76
0.29
sand
0.71
0.23
Table Examples completions which the weight BERT and concept-BERI assign are notably different. Our manipulation increases the relative rank of appropriate completions and decreases the rank of inappropriate ones. Relative rank calculation: (1 completion rank)/ A where K=100 for BERT and k=number outputted clusters for concept-BEER BERT. Color coding: red=low score, orange=intermediatee, green=high.
Scenario Concept-BERT
Mean score
Norm. score
crease when going down the ranked list. While concept-BERT does have negative correlation, BERT is actually positively correlated (meaning, its top-ranked completions are average worse than the bottom-ranked ones). Both curves have significant correlation (p-values<00.05), whereas BERT's weaker (coefficient 0.54 versus 0.91). We stress this is not random sample, but rather the disputed completions (and buffer). Thus, our concept-aware manipulation reveals appropriate completions and removes inappropriate ones, with respect to the original LLM.
0.84
0.304
BERT +
Buffer 0.74
Concept-BERT
0.66
-0.142
1ERT
Table 3: (Higher means better] Mean scores and normal- ized (using the buffer) scores of the three scenarios in the dispute evaluation. Tokens concept-BERT ranked probable while BERT as improbable (first row) are an- notated significantly better than both buffer (middle row) and the tokens BERT ranked high and concept- BERT low (bottom).
Lastly, we also analyze the mean accuracy of the disputed completions as function of how strict the threshold for "in dispute" is. BERT's accu- racy decreases much more sharply compared to concept-BERT (> 10% versus < 2%), hinting our manipulation increases robustness (Figure A.9),
buffer and output tokens that are above the buffer according to one model and below according the other. This way, we identified 282 disputed tokens. addition, we annotated completions that both models ranked in the middle 15% (buffer). Volun- teer computer science graduate students annotated 585 completions using the same setup in $4.2.2 (282 disputed completions and 303 buffer com- pletions). Each completion was annotated by two students (mean-variance across annotations=0.1).
To conclude, our simple implementation led overall coherent and distinct concept-clusters with meaningful ranking that improve the original rank- ing in a robust manner. We see promising indica- tions that similar techniques could enhance LLM robustness.
We divide the annotated completions into three groups and compute their mean annotation scores. As some sentences have more good completions than others, also normalize the mean score per sentence by subtracting the sentence buffer's mean score. Table 3 shows that when the models dis- agree, concept-BERT more often correct (also refer to Figure A.7 in the Appendix)
Related Work
There is little work on endowing LLMs with a notion of concepts. The work that closest to ours in spirit is SenseBERT (Levine et al., 2020), which shifts BERT from the form- to the sense-level by injecting WordNet token information, improving word sense disambiguation. Unlike SenseBERT, suggest learning the categories in context- dependent manner (by using the categories that best assist in predicting the missing text), rather than their static, context-independen method.
Next, compute the accumulated mean accu- racy completions as function of rank given by each of the models (Figure A.8). We expect a negative correlation since the quality should de-

---

Works on knowledge base embeddings learn rep- resentations from knowledge graphs (Bordes et al., 2013), but they are also limited to simple concepts and KB relations, with no external context.
the underlying model.
The notion of concepts has been extensively stud- ied in psychology. and we believe many of the ques- tions posed there could inspire directions for future work. For example, Prototype theory states there is graded degree of belonging to concept (Rosch, 1975), which we have not taken into account here. Other works tackled the problem of complex con- cepts and the composition of concepts. For exam- ple, Rosch (1975) showed that concepts can be combined ("tall" + "man") in a context-dependent way (e.g., the height for a man to be considered tall is not the same as for a building). For this direction, commonsense sources such as the distribution-over quantities dataset (Elazar et al., 2019) might prove helpful.
Aharoni and Goldberg (2020) showed that LLMs capture the domain sentence belongs to. While domains can also be thought of as concepts, we after a different (and dynamic) granularity.
ConceptX is an interpretability framework to analyze how latent concepts are encoded in repre- sentations learned within pretrained LLMs (Sajjad et al., 2022). They found that roughly 50% of the encoded concepts adhere to their suite of human- defined linguistic concepts, which aligns with our results from RQI. However, their method fails to address complex, muuti--aceted concepts.
Hanna and Mareček (2021) specifically explored the extent to which BERT captures hypernymy. exploring different prompts and evaluation metrics. We note that their dataset was somewhat biased towards Australian and New Zealand culture, and also contained multiple errors.
While this only preliminary work, we believe that concept-aware LLMs hold immense promise for the next gext-eeenration of LLMs, and could benefit many downstream tasks around learning, planning, and reasoning, One place where we believe this up- proach will be particularly useful is whenever there is disambiguity (speech recognition, word sense, spell check, etc.). Consider rare words that need to be identified by automatic speech recognition: by working at concept-level, these words could be clustered together with completions that are likely given the rest of the sentence but do not sound similar to the audio, thus increasing the acoustic model's certainty score. We hope draw the com- munity's attention to this exciting new research area of concept-aware LLMs.
6 Conclusions & Future Work
In this work explored the possibility of concept- aware LLMs, inspired by the importance of con- cepts in human cognition. Today's s LLMs all work at the level tokens, not concepts; this is problem- atic, as different tokens from the same underlying concept compete for the probability mass. Instead, we envision L.MMs that first zero in on a likely concept, and only then pick the best surface form among the candidates.
Limitations
We started by analyzing what extent con- temporary pretrained LLMs capture concepts and match the human organization of concepts. We showed that LLMs do capture concepts to some (far-from-perfecct) extent, while sometimes violat- ing the human concept-orgaanization principles of asymmetry, transitivity, and property inheritance.
In RQI & RQ2, for BERT and T5 we rely on ex- act string matching. This entails that if the LLM retrieved a synonym of the right answer, we do not detect it as a successful retrieval.
Another limitation is the usage of noisy automa- tion. We rely on WordNet's first word-sense for the 100 ETs, which might not be the original intent, Quasimodo might also introduce wrong relations (even when using a strict threshold, as we did). Moreover, Wordtune's 's paraphrasing is another pos- sible source of noise. We note that throughout the process we consistently sampled to verify the quality of the automatic parts, and found them sat- isfactory.
Next, we explored directions for augmenting LLMs with concepts from two different angles. We first sketched our vision for training concept- aware LLMs. Next, we presented a model-agnostic method to shift any off-the-shed shelf pretrained LLM from the token- to the concept-level, without fine- tuning adding any external information. We showed that our coRcept-BERT (built on top of BERT) outputs a ranked list of concepts which are relatively coherent and distinct, better match human intuition, and are more robust compared to
As for RQ3, our method heavily relies on the input LLM, and thus might preserve some the LLM's biases. One might try to overcome these

---

biases, e.g., by injecting external knowledge. Another limitation of our method the usage of not just the LLM completions, but also their embeddings. This does not let us apply our method to LLMs that expose only their completion output (e.g., accessible via API).
Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12711
Xiaojun Chen, Shengbin Jia, and Yang Xiang. 2020. A review: Knowledge reasoning over knowledge graph. Expert Systems with Applications, 141:1 12948.
Lastly, as we run the LLM several times and postprocess (paraphrasing extraction, dimensional- ity reduction, clustering, etc.), computation of our proof-of-concept model is somewhat slower than that of the underlying LLM. We note, however, that many of those operations are easily parallelizable.
Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 58(9)992 103.
Jacob Devlin, Ming-Wai Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint 18..v18100004005.
Acknowledgements
We thank the reviewers for their insightful com- ments. This work was supported by the Euro- pean Research Council (ERC) under the European Union's Horizon 2020 research and innovation pro- gramme (grant no. 852686, SIAM).
Yanai Elazar, Abhijit Mahabal, Deepak Ramachandran, Tania Bedrax- Weiss, and Dan Roth. 2019. How large are lions? inducing distributions over quantitative attributes. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 3973-3983
Mattia Fumagalli and Roberta Ferrario. 2019, Repre- sentation of concepts in ai: Towards a telcological explanation. In JOWO.
In memory of the more than one thousand victims of the horrific massacre carried out by Hamas terrorists on October 7th, 2023.
Liane Gabora, Eleanor Rosch, and Diederik Aerts. 2008. Toward an ecological theory of concepts. Ecological Psychology, 20(1):84-111) 116.
Peter Gardenfors. 2014. The geometry of meaning: Se- mantics based on conceptual spaces. MIT press.
References
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu. Gargi Ghosh, and Luke Zettlemoye. 2021, Htlm: Hyper-text pre-training and prompting of language models. arXiv preprint arXiv:2107.06955.
Yuling Gu, Bhavana Dalvi Mishra, and Peter Clark. 2022. Do language models have coherent men- models of everyday things? arXiv preprint arXiv:2212.1002,
Michael Hanna and David Mareček. 2021. Analyzing bert's knowledge of hypernymy via prompting. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 275-282.
Roce Aharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7747- 7763.
Ari Holızaan, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competi- tion: Why the highest probability answer isn' always right. Proceedings of the 202/ Conference on Em- pirical Methods in Natural Language Processing. pages 7038-7051
Michael Boratko, Xiang Li, Tim O' O'Gorma, Rajarshi Das, Dan Le, and Andrew McCallum. 2020. Pro- toQA: A question answering dataset for prototypi- cal common-sense reasoning. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 1122-1136. Online. Association for Computational Linguistics.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick. Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03294
Antoine Bordes, Nicolas Usunier, Alberto Gareia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. Advances in neural information pro- cessing systems, 26.
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020. Sensebert: Driv- ing some sense into bert. In Proceedingy of the 58th
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar,

---

Annual Meeting of the Association for Computational Linguistics, pages 4656-4661
LLM
Mean retrieval
# ETs
# Relations
BERT
0.79
15
169
T5
0.47
18
141
Ruibo Liu, Guoqing Zheng, Shashank Gupta, Rad- hika Gaonkar, Chongyang Gao, Soroush Vosoughi, Milad Shokouhi, and Ahmed Hassan Awadallah. 2022. Knowledge infused decoding. arXiv preprint arXiv:2204003084
GPT-3.5
0.88
57
493
GPT-4
0.83
56
562
Table 4: Mean retrieval the transitivity property for the four LLMss. For each pair where the L.I.M correctly retrieved that "<ET> <hypemmmmm and "<hy- ponym> TypeOf<ET>", we queried to see it will also retrieve that "<hyponyms> TypeDf <hpeemmmmymmmm"""" Higher means better transitivity abilities.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, (8(11):39-41
Gregory Murphy, 2004. The big book of concepts. MIT press.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-5551
Francisco Varela, Evan Thompson, Eleanor Rosch, and Jon Kabat-Zinn. 2017. The embodied mind. (No Title).
Ledell Wu, Fabio Petroni, Martin Josifoski. Sebastian Riedel, and Luke Zettlemoyer. 2019. Scalable zero- shot entity linking with dense entity retrieval. arXiv preprint arXiv: 19100881
Julien Romero, Simon Razniewski, Koninika Pal, Jeff Z. Pan, Archit Sakhadeo, and Gerhard Weikum. 2019. Commonsense properties from query logs and ques- tion answering forums. In Proceedings of the 28th
ACM International Conference on Information and Knowledge Management, pages 1411-1420.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019, Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems. 32.
Eleanor Rosch. 1975. Cognitive representations of se- mantic categories. Journal of experimental psychol- ogy: General, 10993)1192.
Chaoning Zhang, Chenshuang Zhang, Sheng Zheng, Yu Qiao, Chenghao Li, Mengchun Zhang, Sumit Ku- Dam, Chu Myaet Thwal, Ye Lin Tun, Le Luang Huy, al. 2023. complete survey on generative (aige): Is chatgpt from gpt-4 to gpt-5 all you need? arXiv preprint arXiv:2303.111: 11717.
Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Firoj Alam, Abdul Rafac Khan, and Jia Xu. 2022. Analyzing encoded concepts in transformer language models. arXiv preprint 2006::2206.1 13289.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05/00
RQ2: Transitivity
Table 4 depicts the mean retrieval of the transitivity property for the three LLMs.
Timo Schick, Jane Dwivedi-YV Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves use tools. arXiv preprint arXiv:2302.0476.
RQ2: Property inheritance
BERT's coverage is rather low, leaving us with 29 ETs and 709 triplets to consider. T5 covered 52 ETs with 262 triplets. Both GPT models' coverage is 51 ETs and 748 triplets.
Timo Schick, Jane Dwivedi- Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebas- tian Riedel. 2022. Peer: A collaborative language model. arXiv preprint Xiv::22008. 1663.
RQ3: implementation details
LM. We used BERRT-bas--nnaased with the default parameters. We performed no training.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Concepinet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings the AAAI confer- ence on artificial intelligence, volume 31.
Augmentation. For this phase we first replaced the missing token with the LLM's most probable completion that contains more than three letters and is not stop-word (using the stopwords list from nitk. nork.co package). We inserted So AT21's Wordtune paraphrasing model using the default parameters:
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint 2201 08239.

---

requests, post(
Location is one of CA, GB, US (for English speak- ers)). We also used custom qualification using five example sentences and completions. Annota- tors were allowed to make one error order to qualify. We paid annotators S0.02 per completion. O Overall, we had 39 unique annotators.
"htpps : // api ai21 com/ studio/ vl vl/experimental/ rewritee,
headers=("Authentiaaall
<ai21 -e--rrivteetookene
json=(" S_0,
"intent": "o::""geee "general"))
Full instructions:
You will be presented with a sentence containing missing word and candidate word to fill-in the blank. Your role is to determine for each comple- tion whether is likely / possible but not likely does not make sense all. Note! If completion not grammatically correct ("I enjoy *raining* instead of "rain") that is fine, we do care about grammar here. But the sentence + completion is not a full sentence ("I enjoy *doing*") that NOT fine, as the sentence is meaningless.
And extracted the text suggestions from the out- put JSON file. We then searched for the original completion and masked all sentences. Sentences in which were unable automatically find the word were dropped.
Top-k completions retrieval. We used = 100 for each masked sentence. pletion that did not appear in at least half our augmentations. Note: another possible implemen- tation would be function of unique completions.
drop each com-
Example 1
Clustering & Ranking, As latent space represen- tation of contextual token, we extract the LLM's token embedding for this token from the last hid- den layer with the input sentence So. We reduce the dimensionality of the embeddings from 768 to 100 using PCA (scikit learn implementation, __ commponents=100, svd_ cod_ssover=''lllll 'full') and from 100 to 10 using t-SNE (scikit learn implementa- tion, n components==10, init='pcat 'pea', perplexity=10, method='exacr"
Sentence: went to the parent teacher conference with my
Completion: parent
Desired response: Likely
Example 2
Sentence: went to the parent teacher conference with my
Completion: schedule
Desired response: Does not make sense Example 3
We cluster the embeddings after the dimen- sionality reduction using agglomerative cluster- ing using the distance metric cosine similar- ity, linkage='linkage, distance threshold=0.45, n_ clusterrs=Noee and conpute_ddisaaccs=TTuu distances=True (scikit learn implementation)
Sentence: went to the parent teacher conference with my
Completion: grandfather
Desired response:
Possible but unlikely
Explanation: While this is not the common nario, it is still possible.
To rank the clusters, we used the following for- mula:
Example 4
(maaruoogeewwiighht weight(C:) = wagg (weight(t) VE E C.)
Sentence: went to the parent teacher conference with my
+(1 a). (repe AE C:)
Completion: mothers
where a 0.7.
Desired response: Likely OR Possible but unlikely Completions in dispute. We recruited vol- unteers, all are graduate students from the com- puter science department (same instructions as the Amazon Mechanical Turk experiment, see in- structions above). Each student annotated about 150 completions (cutoff at the end of the sen- tence). Each completion was annotated by two students. Mean-variance across annotation=0.1. showing fairly good quality of annotations (pos- sible responses=(0, 0.5, 1}). Students reported the task to take about 15 minutes to complete.
RQ3: Human annotators
For both annotation tasks (computer science gradu- ate students and Amazon Mechanical Turk), anno- tators were presented with sentence and possi- ble completion and asked "Do you think this completion makes sense?" Possible responses are: (likely, possible but unlikely, does not make sense), Precision at We used Amazon Mechanical Turk with the following qualifications: (HIT Approval Rate 98, Number of HITs Approved > 5000,

---

Probabiliyy and Accuracy of the LMs' Completions
Concept-BERT's BERT-S'T' Probability
0,8
0.8
0.4
0.4
0.2
0.0
0.0
BERT'S Probability
Figure 7: Heat-map the disputed completions (higher means better). The y-axis represents concept-BERT's completion relative rank. The x-axis represents BERT's s relative rank. The top-left part higher scores compared to the middle (buffer) and the bottom-right part. Meaning, our manipulation ranked high appropriate completions and low inappropriate ones.
map received
Precision as Function of Threshold
Accumulated Accuracy as Function of Normalized Rank
0.85
our
Mean Precision
0.80
Accuracy
0.70
0.7
0.6
0.65
34
36
38
40
Threshold (%)
0.5
Figure 9: Mean accuracy of the disputed completions for both models as a function of how strict the threshold for disagreement BERT's accuracy decreases sharply compared to concept-BERT, suggesting our manipula- tion increases robustness.
20
100
Normalized Rank
Figure 8: Completions' accumulated mean accuracy a function of rank given by each of the models. Both curves have significant correlation (p-value < 0.05), whereas BERT's correlation is weaker (correlation coef- ficient 0.54 versus 0.91). Interestingly, while concept- BERT has negative correlation, as expected since the rank's 's quality should decrease, BERT's correlation is positive. stress this is not random sample, rather the disputed completions (and the buffer). Thus, this again strengthens our claim, that our manipulation helps reveal appropriate completions and remove in- appropriate ones, with respect to the original LLM.
RQ3: Figures


Reviews:
[{"review_id": "ePBs0ewxG2", "reviewer": "Reviewer_HKKb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We are glad you find our research questions and motivation interesting and timely, and our experimental results interesting. \n\n* “The motivation and central questions of the paper talks about concepts at a very broad scale but the actual implementation is limited to hyponyms and hypernyms”\n\nConcepts are indeed a very broad and rich topic that can be tackled from many different perspectives. We started exploring this topic at the most fundamental level – the way concepts are hierarchically organized. This aspect has been extensively studied in cognitive science, and thus provides a good starting point for our exploration: we start by assessing the common, agreed-upon organization principles of asymmetry, transitivity, and property inheritance. \n\n* Approach 1, “go against the fundamentals of language modeling since such concepts don’t explicitly occur as special tokens in language utterances”, “a sketch not backed by concrete empirical evidence”: \n\nWe wish to clarify that Approach 1 uses regular natural language text, without any special tokens for concepts; the model’s task is to find concepts that will help it complete the masked text.\n\nWe are inspired by the Toolformer paper, which was trained to decide which API calls to use and when (given standard text input, without special tokens) and how to best incorporate the results of the calls into token prediction. Toolformer showed improved performance across various downstream tasks compared to GPT-based models. It demonstrated that LLMs can be taught (in a self-supervised manner) when and how to use tools (in our case, concepts) to enhance performance.\n\nAnother reason we believe this approach is promising is that many papers have shown empirical evidence that adding intermediate tasks can actually improve performance on the final task (most famously, the chain-of-thought paradigm).\n\n* “It is not immediately clear to me what the benefit would be to shift existing LLMs from token-level to concept-level by following the second approach presented in the paper”\n\nWe base our proof-of-concept on the most fundamental language-model task – text completion, and demonstrate the potential of concepts. (As this is the theme track, we focused more on the high-level idea of concept-aware LLMs and less on downstream tasks.)\n\nWe acknowledge there are many downstream tasks to explore in follow-up work. Intuitively, one place where we believe this approach will be particularly useful is whenever there is _disambiguity_ (speech recognition, word sense, spell check…). Consider rare words that need to be identified by automatic speech recognition: by working at concept-level, these words could be clustered together with completions that are likely given the rest of the sentence but do not sound similar to the audio, thus increasing the acoustic model’s certainty score.  \n\nThanks again for your comments and questions. We believe they will considerably improve the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research questions and motivation of the paper is interesting and timely.\n2. Some of the results presented in the paper are interesting (for eg the difference in performance of GPT on hypernym vs hyponym).\n3. The paper is well-written and well-presented.", "reasons_to_reject": "1. The motivation and central questions of the paper talks about concepts at a very broad scale but the actual implementation is limited to hyponyms and hypernyms.\n2. The approach 1 of rethinking LLMs training does not make a lot of sense to me. It would go against the fundamentals of “language modeling” since such concepts don’t explicitly occur as special tokens in language utterances. It is also only a sketch not backed by concrete empirical evidence so I am skeptical about its usefulness.\n3. It is not immediately clear to me what the benefit would be to shift existing LLMs from token-level to concept-level by following the second approach presented in the paper. The authors do not demonstrate how such a “concept-aware” LLM would be better at language understanding or reasoning than normal LLMs.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a study on how well LLMs capture human concepts and their organization. The authors perform this analysis by focusing on the “type of” relation which pertains to hyponyms and hypernyms. Their results indicate that while LLMs do indeed capture concepts to some extent, they are still far from the human baseline. The authors also propose approaches to enhance LLMs in terms of concepts.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ov9n7Fmg4c", "reviewer": "Reviewer_YpEe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We are glad you found our exploration innovative and the proposed approach interesting and novel, with possible practical applications. \n\n* “[T]he paper doesn't validate this claim adequately as it doesn't test its hypotheses on state-of-the-art LLMs like ChatGPT, GPT-4, or Llama.”\n\nThere are countless LLMs to try, and we chose several representatives from different model families. Specifically, we used GPT-davinci-003, which is a sibling of ChatGPT (both based on GPT-3.5). \nAs for GPT-4, it was made publicly available via OpenAI’s API on June 6th (https://openai.com/blog/gpt-4-api-general-availability) – less than 2.5 weeks before the submission deadline for EMNLP. We would gladly add results for it to the manuscript.\n\n(We note that the recent GPT models do not provide information regarding their training data and architecture, which makes it difficult for the scientific community to evaluate, except as a black box. Having said that, we acknowledge that these models are SOTA, and thus we chose a representative from this family.)\n \n\n* LM vs. LLM, alignment with the theme track:\n\nIn line with the spirit of the theme track (and in particular, the question “How do LLMs capture world knowledge?” and the invitation for all kinds of contributions, including position papers), we presented here an approach for developing concept-aware LLMs as well as a proof-of-concept. Our main goal is to raise awareness and spur new research around concept-aware LLMs; this work was _not_ meant to be a comprehensive analysis of all contemporary LLMs. \n\nWe chose BERT for our proof-of-concept as it is (still today) the most popular model on HuggingFace for the fundamental fill-mask task. Our pipeline supports any model that can return a ranked list of completions, their embeddings and perplexity scores (and indeed, we have already achieved promising initial results for T5 that we could add to the paper); GPT does not provide the access we need, and thus we have not included it in the comparative study. \n\nWe note it is possible to extend our framework to black-box models (e.g., by sampling multiple completions and rethinking the aggregation function), but we leave this for future work.\n \n(In Table 1 that was just a typo. Thanks for pointing it out.)\n\n* “The paper lacks a comprehensive analysis or understanding of the practical challenges and potential privacy concerns of obtaining and using concept-level data for training large language models” \n\nWe will gladly expand on practical challenges (we did discuss the main practical challenges for approach 2 in the limitations section). As for privacy concerns, we believe our approach does not add any major concerns on top of those already raised by using LLMs, as we are not using any private data. In particular, approach 1 uses publicly available data such as Wikipedia categories or knowledge graphs. Approach 2 uses the LLM’s embeddings. The only place we can see private information leaking might be the automatic paraphrasing (if the paraphrasing tool was trained on data such as personal emails), but this is completely orthogonal to our algorithm; we will add a note about this to the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides an innovative exploration of the interaction between large language models (LLMs) and human concepts, opening up a relatively unexplored area of research.\n\n- The paper proposes an interesting approach for developing concept-aware LLMs, which involves pretraining these models using concepts, a novel shift from token-based to concept-based processing.\n\n- The proof-of-concept model developed in the paper manages to enhance the ranking and robustness of existing LLMs without the need for additional training, suggesting practical applications of the research.", "reasons_to_reject": "- Despite claiming that large language models (LLMs) do not align well with human concepts, the paper doesn't validate this claim adequately as it doesn't test its hypotheses on state-of-the-art LLMs like ChatGPT, GPT-4, or Llama. \n\n- The paper seems to be initially developed with a focus on language models (LMs), with some cosmetic changes made to tailor it for LLM theme track. Evidence for this includes the naming of the model column as \"LM\" instead of \"LLM\" in Table 1. Moreover, the comparative study between BERT and concept-BERT in Tables 2 and 3 doesn't convincingly demonstrate the concept's application in the context of state-of-the-art LLMs. The experimental design, overall, appears more suited for masked language models like BERT, which raises doubts about the paper's alignment with the LLM theme. \n\n- The paper lacks a comprehensive analysis or understanding of the practical challenges and potential privacy concerns of obtaining and using concept-level data for training large language models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper engages with the interaction between large language models (LLMs) and human concepts. The authors conducted an analysis of contemporary LMs, examining how well these models capture and organize human concepts. They uncovered inconsistencies in these models and identified areas where LMs diverge from human organization of concepts. Moving from analysis to application, they proposed an approach for developing concept-aware LLMs that involves pre-training LLMs with concepts. This represents a shift from the traditional token-based processing to a concept-based approach. They presented a proof-of-concept model which, despite its simplicity, was able to enhance the ranking and robustness of LLMs without requiring additional training. Finally, the authors highlighted potential avenues for future research, suggesting that integrating concept-awareness in LLMs could improve numerous downstream tasks such as learning, planning, and reasoning. This work thus offers a fresh perspective on the intersection of LLMs and human concepts.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UXe7RgLfwp", "reviewer": "Reviewer_YmXf", "summary": "", "questions": "A. RQ2: Did you collect human judgments for asymmetry, transitivity, and property inheritance? Even though asymmetry should theoretically give 1.0, for example, perhaps for completeness the authors should include the corresponding human baseline (this comment did not affect my score)\n\nB. l385 Which parameters are being finetuned? In l304 it is written \"without any further training\".", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable insights. We are glad you found our research question timely, important, and well-framed, our experiments well-designed, and our paper clear and enjoyable to read. \n\nWe thank you for pointing out the missing reference, and we will add it to the final version. We will also fix all the style issues.\n\n* “experiments on GPT and T5 should also be included (perhaps in an Appendix).”\n\nOur method requires a model that returns perplexities and conditional embeddings for the completions. Recent GPT implementations only offer partial access to this information. We acknowledge this limitation in the manuscript. We note it is possible to extend our framework to black-box models (e.g., by sampling multiple completions and rethinking the aggregation function), but we leave this for future work.\n \nWe have not included T5 because it is an autoregressive model (the masked text has to be at the end of the input), but we have in fact experimented with T5-3b (only on sentences obeying the input requirement), and we have preliminary (promising) results that we will be happy to add to the Appendix.\n\n* Line 319 (“In practice, one directly uses LLM word embeddings or finetunes them on some downstream task. As the conceptBERT uses a frozen BERT model, however, the practical downstream application is unclear.”)\n\nWe envision two main ways for incorporating concepts in downstream tasks. \n(1) We could fine-tune BERT on the downstream task and then build conceptBERT on top of the fine-tuned model (Approach 2). We posit the fine-tuning process will allow our manipulation to better identify concepts relevant for the downstream task. \n(2) Alternatively, Approach 1 suggests incorporating concepts into the training process itself; while we had pre-training in mind, this approach is also applicable to fine-tuning. \n\n\n* “[T]he inference latency is higher for conceptBERT. To what extent is the inference latency unwieldy?”\n\nRoughly speaking, this is our inference process for conceptBERT: \n1) BERT prediction (~latency of the original inference) \n2) Paraphrase generation (we use word-tune, but can be implemented locally to reduce the latency to milliseconds)\n3) BERT prediction on N sentences (embarrassingly parallel; ~latency of the original inference) \n4) Sparse completion removal (milliseconds)\n5) Dimensionality reduction and clustering (depend on the algorithm and compute power, but because of step 4 they are typically done on only tens/few hundreds of data points) \n\n2+5 are our current bottlenecks, in terms of latency. For our proof-of-concept we did not optimize these steps for latency, and we believe it can be greatly improved. We will add a discussion of factors affecting latency to the manuscript. \n\n\n* Line 385 (“Which parameters are being finetuned?”)\n\nWe apologize, fine-tuning was not the correct term here. The parameters tuned are not BERT’s. We meant to write that we perform a hyperparameter search, for setting hyperparameters such as how many completions to retrieve, the function/threshold for sparsity, which scoring functions to use for the clustering, etc. (see Appendix “RQ3: implementation details”, and also in the repository we will release upon acceptance). We will fix this in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research question (whether LLMs process concepts like humans) is timely, and I believe, important. The behavioral experiments, which are similar to those in Hanna and Mareček (2021), are straightforward and well-designed. The results make concrete the intuition that \"LLMs capture concepts, but not well enough\". \n2. The paper is clear, enjoyable to read, and well-written (with some exceptions, see Weaknesses). I especially liked the broader framing on concepts and human cognition in the Introduction and Conclusions.", "reasons_to_reject": "I have three reservations about the paper, one minor related to prior work (1), and then two regarding the engineering contribution. In particular, notes 2-3 would make the difference between a 3 and a 4/5 for Soundness.\n\n1. Missing reference which uses almost the same prompting method to analyze hypernymy in BERT: [Analyzing BERT’s Knowledge of Hypernymy via Prompting](https://aclanthology.org/2021.blackboxnlp-1.20) (Hanna & Mareček, BlackboxNLP 2021)\n2. l319 In practice, one directly uses LLM word embeddings or finetunes them on some downstream task. As the conceptBERT uses a frozen BERT model, however, the practical downstream application is unclear. In particular, the paraphrase procedure seems quite inefficient if the downstream application is similar to the one used for evaluation (ProtoQA), as the inference latency is higher for conceptBERT. To what extent is the inference latency unwieldy? These factors should be discussed in the paper. \n3. Section 4.3: For completeness, experiments on GPT and T5 should also be included (perhaps in an Appendix).", "questions_to_authors": "A. RQ2: Did you collect human judgments for asymmetry, transitivity, and property inheritance? Even though asymmetry should theoretically give 1.0, for example, perhaps for completeness the authors should include the corresponding human baseline (this comment did not affect my score)\n\nB. l385 Which parameters are being finetuned? In l304 it is written \"without any further training\".", "paper_topic_and_main_contributions": "Whether LLMs process concepts the same ways humans do is an underexplored question. Current state-of-the-art LLMs are trained to predict the next or masked _token_, which doesn't obviously induce concept-awareness-- probability mass of a concept will be spread over many tokens. The authors explore concept-awareness in pre-trained LLMs by (1) showing they capture concepts to some extent, but (2) often violate human organization of concepts such as transitivity. Then, the authors propose two mechanisms, one a pre-training sketch akin to Toolformer (Schick et al. 2023), and one off-the-shelf, where (3) the off-the-shelf procedure improves the concept-awareness of the LLM.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. [Analyzing BERT’s Knowledge of Hypernymy via Prompting](https://aclanthology.org/2021.blackboxnlp-1.20) (Hanna & Mareček, BlackboxNLP 2021) \n\n2. [Concept Understanding in Large Language Models: An Empirical Study](https://openreview.net/forum?id=losgEaOWIL7) (Liao et al., ICLR 2023)\n\n3. [Better Language Model with Hypernym Class Prediction](https://aclanthology.org/2022.acl-long.96) (Bai et al., ACL 2022)", "Typos_Grammar_Style_and_Presentation_Improvements": "### Major\nI think the paper is well-organized and readable as-is. An alternative (this did not impact my score) could be to reduce and move the pre-training engineering sketch to later in the paper as it is more speculative, perhaps in Future Work or Discussion. Instead, this could be replaced with extended experiments on GPT and T5. \n\n### Low-hanging fruit\n1. Figure titles are often cut off at the top.\n2. l145 Should $K$ be $k$?\n3. l130 Should add: it is correct only if the answer is within the top-k completions. (Up to this point it is unclear)"}}]

Author rebuttal:

Meta-review: Scores were mixed: 3,2,4 (soundness) and 3,3,4 (excitement). The authors addressed some concerns, leading to increased scores.

Some of the key strengths and weaknesses identified by the reviewers included:

Strengths:

- The research question and motivation were considered timely and interesting (R1, R2, R3)
- Reviewers found some of the results interesting (R1) and the paper well-written (R1, R2)
- paper proposes a new approach for developing concept-aware LLMs by pretraining using concepts (R2)
- includes promising proof-of-concept results (R2)
- behavioral experiments well-designed (R2)

Weaknesses

- despite very general motivation, the paper focuses on hyponyms and hypernyms (R1)
- concerns about how realistic a shift to concept-level LLMs would really be (R1)
- makes claims about inabilities of LLMs without testing on SOTA LLMs (R2) [a similar concern from R3 was addressed in the author response]

I did not find the soundness-related concerns by the second review (YpEe) to be fully convincing. That review lists three reasons to reject: lack of SOTA LLMs, perceived mismatch with the theme track, and lack of discussion of privacy concerns. Only the first concern appears to be potentially a major soundness concern, but the authors pushed back, noting i.a. that the paper includes GPT davinci-003, which can allay some of this concern.

In conclusion, the reviews suggest that the paper is reasonably sound and was perceived as moderately-to-strongly exciting by the reviewers.
Decision: Accept-Findings