Conceptual structure coheres in human cognition but not in large language models
Authors: Siddharth Suresh
Kushin Mukherjee
Xizheng Yu
Wei-Chun Huang
Lisa Padua
Timothy T. Rogers
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Cognitive Science
Semantic Norms
Human conceptual structure
AI conceptual structure

Abstract: Neural network models of language have long been used as a tool for developing hypotheses about conceptual representation in the mind and brain. For many years, such use involved extracting vector-space representations of words and using distances among these to predict or understand human behavior in various semantic tasks. In contemporary language models, however, it is possible to interrogate the latent structure of conceptual representations using methods nearly identical to those commonly used with human participants. The current work uses three common techniques borrowed from cognitive psychology to estimate and compare lexical-semantic structure in both humans and a well-known large language model, the DaVinci variant of GPT-3. In humans, we show that conceptual structure is robust to differences in culture, language, and method of estimation. Structures estimated from the LLM behavior, while individually fairly consistent with those estimated from human behavior, depend much more upon the particular task used to generate behavior responses–responses generated by the very same model in the three tasks yield estimates of conceptual structure that cohere less with one another than do human structure estimates. The results suggest one important way that knowledge inhering in contemporary LLMs can differ from human cognition.

Conceptual structure coheres in human cognition but not in large language models
Siddharth Suresh', Kushin Mukherjee', Xizheng Yu', Wei-Chun Huang Lisa Padua² and Timothy T. Rogers! University of Wisconsin-Madiionn, Albbany State University www.srtwwssseeewiiss....
Albany
Abstract
prediction of upcoming words in sentences-early models exhibited properties that upended received wisdom about what language and how works. They acquired internal representations that blended syntactic and semantic information, rather than keeping these separate as classic lgycho-liguisstes required. They handled grammatical dependencies, not by constructing syntactic structure trees, but by learning and exploiting temporal patterns in language. Perhaps most surprisingly, they illustrated that statistical structure latent natural language could go a long way toward explaining how we acquire knowledge of semantic similarity relations among words. Because words with similar meanings tend to be encountered in similar linguistic contexts (Firth, 1957; Osgood, 1952; Harris, 1954), models that exploit contextual similarity when representing words come express semantic relations between them. This enterprise of learning by predicting has persisted into the modern era of Transformer based language models (Devlin et al., 2018; Brown et al., 2020a).
Neural network models of language have long been used as a tool for developing hypotheses about conceptual representation the mind and brain. For many years, such use involved extracting vector-space representations of words and using distances among these predict understand human behavior in various semantic tasks. Contemporary large language models (LLMs), however, make it possible to interrogate the latent structure conceptual representations using experimental methods nearly identical to those commonly used with human participants. The current work utilizes three common techniques borrowed from cognitive psychology to estimate and compare the structure of concepts humans and suite of LLMs. In humans, we show that conceptual structure is robust to differences in culture. language, and method of estimation. Structures estimated from LLM behavior, while individually fairly consistent with those estimated from human behavior, vary much the particular task used to generate responses- across tasks, estimates of conceptual structure from the very same model cohere less with one another than do human structure estimates.
depending upon
Though early work was limited the nature and complexity of the language corpora used to train models (Elman, 1991; McClelland et al., 1990), these ideas spurred variety of computational approaches that could be applied to large corpora of written text. Approaches such as latent semantic analysis (Deerwester
These results highlight important difference between contemporary LLMs and human cognition, with implications for understanding some fundamental limitations of contemporary machine language.
al., 1990) and skip- gram models (Mikolov et al., 2013a; Bojanowski et al., 2017), for instance, learn vector-space representations of words from overlap in their linguistic contexts, which turn out to capture a variety of semantic relationships amongst words. including some that are highly abstract (Grand et al., 2022; Elman, 2004; Lupyan and Lewis, 2019).
1 Introduction
Since Elman's pioneering work (Elman, 1990) showcasing the ability of neural networks to capture many aspects of human language processing (Rumelhart et al., 1986), such models have provided useful tool, and sometimes gadfly, for developing hypotheses about the cognitive and neural mechanisms that support language. When trained on a task that seems almost absurdly simplistic-conie continuous,, sequential
In all of this work, lexical-semantic repre- sentations are cast as static points in high- dimensional vector space, either computed directly from estimates of word co-occurrence large

---

text corpora (Deerwester al., 1990; Burgess, 1998). or instantiated as the learned activation patterns arising neural network model trained on such corpora. To evaluate whether a given approach expresses semantic structure similar to that discerned by human participants, the experimenter typically compares the similarities between word vectors learned by a model to decisions behaviors exhibited by participants in semantic tasks. For instance, LSA models were tested on synonym-judgment tasks drawn from a common standardized test of English language comprehension by comparing the cosine distance between the vectors corresponding to a target word and each of several option words, and having the model "choose" the option with the smallest distance (Landauer et al., 1998). The model was deemed successful because the choice computed in this way often aligned with the choices of native English speakers. Such procedure was not just a useful way for assessing whether model representations are human-like-it was just about the only way to do so for this class of models.
law and medicine (Newton and Xiromeriti, 2023; Choi et al., 2023; Kung et al., 2023) produce working Python code from a general description of the function (OpenAI, Year of the webpage, e.g., 2023), generate coherent explanations for a variety of phenomena, and answer factual questions with remarkable accuracy ! Even in the realm of failures, work has shown that the kinds of reasoning problems LLMs struggle with are often the same ones that humans tend to find difficult (Dasgupta et al., 2022). In short, if solely evaluated based on their generated text, such models appear to show several hallmarks of conceptual abilities that until recently were uniquely human.
These innovations allow cognitive scientists, for the first time, measure and evaluate conceptual structure in a non-human system using precisely the same nattral-language-baasa methods that we use to study human participants. Large language models can receive written instructions followed by series of stimuli and generate interpretable, natural-language responses for each. The responses generated can be recorded and analyzed in precisely the same manner responses generated by humans, and the results of such analyses can then be compared within and between humans and LLMs, as a means of understanding whether and how these intelligences differ.
In the era of large language models such as Open Al's GPT3 (Brown et al., 2020a), Meta's LLaMa family (Touvron et al., 2023; Taori et al., 2023), Google's FLAN (Wei et al., 2021), and many others (Zhang et al., 2022; Chowdhery et al.. 2022: Hoffmann et al., 2022; Du et al., 2022), this has changed. Such models are many orders of magnitude larger than classical connectionist approaches, employ a range of architectural and training innovations, and are optimized on truly vast quantities of data-but but nevertheless they operate on principles not dissimilar to those that Elman and others pioneered. That is, they exploit patterns of word co-occurrence in natural language to learn distributed, context-sensitive representations of linguistic meaning at multiple levels, and from these representations they generate probabilistic predictions about likely upcoming words. Current models generate plausible and grammatically well-formed responses, created by iteratively predicting what words are likely to come next and sampling from this distribution using various strategies (Wei et al., 2022; Wang et al., 2022; Yao al., 2023). So plausible is the text that recent iterations like ChatGPT (Ouyang et al.. 2022) can write essays sufficiently well to earn a A in an undergraduate setting (Elkins and Chun, 2020), pass many text-based licensing exams in
The current paper uses this approach to understand similarities and differences in the way that lexical semantic representations are structured in humans vs LLMs, focusing on remarkable aspect of human concepts-specifically their robustness. As Rosch showed many years ago (Rosch. 1975, 1973). the same conceptual relations underlie behavior in a variety of tasks,
from naming and categorization to fcature-listing to similarity judgments to sorting. Similar conceptual relations can be observed across distinct languages and cultures (Thompson et al., 2020). Robustness is important because it allows for shared understanding and communication across cultures, over time, and through generations: Homer still speaks to us despite the astonishing differences between his world and ours, because many of the concepts that organized his world cohere with those that organize ours. Our goal was to assess whether conceptual structure
Whiie it is likely that GPT-3 has been trained on examples of many of these exams and tasks, that it can efficiently retrieve this knowledge when queried with natural language is nonetheless worth noting.

---

in contemporary LLMs is also coherent when evaluated using methods comparable to those employed with human participants, or whether human and LLM "mind" differ in this important regard.
have used models generate lists of features and estimated semantic structure from feature overlap (Hansen and Hebart, 2022: Suresh et al., 2023; Mukherjee et al., 2023), or have asked models to produce explicit pairwise similarity ratings (Marjich et al., 2023). While such work often compares aspects of machine and human behaviors, to our knowledge prior study has evaluated the coherence of elicited structures across tasks within given model, or between structures elicited from humans and machines using the same set of tasks.
To answer this question, we first measured the robustness of conceptual structure in humans by comparing estimates of such structure for a controlled set concepts using three distinct behavioral methods feature-listing, pairwise similarity ratings, and triadic similarity judgements across two distinct groups Dutch and North American - differing in culture and language. We then conducted the same behavioral experiments on LLMs, and evaluated (a) the degree to which estimated conceptual relations in the LLM accord with those observed in humans, and (b) whether humans and LLMs differ in the apparent robustness of such structure. We further compared the structures estimated from the LLM's overt patterns of behavior to those encoded in its internal representations, and also to semantic vectors extracted from other common models in machine learning. In addition to simply demonstrating how methods from cognitive psychology can be used to better understand machine intelligence, the results point to an important difference between current state of the art LLMs and human conceptual representations.
Measuring Human Conceptual Structure
For both human and LLM experiments, we focused on a subset of 30 concepts (as shown in Table I) taken from a large feature-norming study conducted KU Leuven (De Deyne et al., 2008).
The items were drawn from two broad categories- tools and hatplsssaaaphibian--sseeeeeeeee because they span the living/nonliving divide and also possess internal conceptual structure. Additionally this dataset has been widely used and validated in the cognitive sciences.
To measure the robustness of conceptual structure in humans, we estimated similarities amongst the 30 items using 3 different tasks: (1) semantic feature listing and verification data collected from Dutch-speaking Belgian population in the early 2000s, (2) triadic similarity- matching conducted in English in the US in 2022, and (3) Likert-scale pairwise similarity judgments collected in English in the US in 2023. The resulting datasets thus differ from each other (1) the task used (feature generation vs triadic similarity judgments pairwise similarity ratings), (2) the language of instruction and production (Dutch English), and (3) the population from which the participants were recruited (Belgian students in early 2000's vs American MTurk workers 2022/2023). The central question was how similar the resulting estimated structures are to one another, a metric we call structure coherence. If estimated conceptual similarities vary substantially with language, culture, or estimation method, the structural coherence between groups/methods will be relatively low; if such estimates are robust these factors, it will be high. The comparison then provides baseline against which to compare structural coherence in the LLM.
2 Related work
addition to many of the studies highlighted in the previous section, here we note prior efforts to model human semantics using NLP models. Many recent papers have evaluated ways in which LLMs are and are not humanlike in their patterns of behavior when performing tasks similar to those used in peychology-enouge that, despite the relative youth of the technology, there has been a recent review summarising how LLMs can be used psychology(Demszky et al., 2023), along with work highlighting cases where L.MSs can replicate classical findings in the social psychology literature (Dillion et al., 2023). Looking further back, several authors have evaluated the semantic structure of learned word embeddings in static- vector spaces (Mikolov et al., 2013b; Marjieh et al.. 2022b), while others have examined the semantic structure of more fine-grained text descriptions of concepts in language models capable of embedding sequences (Marjieh et al., 2022a). A few studies

---

3.1 Methods
which Euclidean distances between two words capture the probability that they will be selected as "most similar" relative to some arbitrary third word. Like feature-listing, triplet judgment studies can be conducted completely verbally, and so can be simulated using large language models. Participants were 18 Amazon Mechanical Turk workers recruited using CloudResearch. Each participant provided informed consent in compliance with our Institutional IRB and was compensated for their time.
3.1.1 Feature listing study
Data were taken from the Leuven feature- listing norms(De Deyne et al., 2008). In an initial generation phase, this study asked 1003 participants to list 10 semantic features for 6-10 different stimulus words which were were one of 295 (129 animals and 166 artifacts) concrete object concepts. The set of features produced across all items were tabulated into a 2600d feature vector. In a second verification phase, four independent raters considered each concept-feature pair and evaluated whether the feature was true of the concept. The final dataset thus contained a C (concept) by F (feature) matrix whose entries indicate how many of the four raters judged concept C to have feature F. Note that this endeavour required the raters to judge hundreds of thousands of concept-property pairs.
Stimuli were English translations of the 30 item names listed above, half reptiles and half tools.
Procedure. On each trial, participants viewed a target word displayed above two option words, and were instructed choose via button press which of the two option words was most similar the target in its meaning. Each participant completed 200 trials, with the triplct on each trial sampled randomly with uniform probability from the space of all possible triplets. The study yielded a total of 3600 judgments, order of magnitude larger than the minimal needed to estimate an accurate 3D embedding from random sampling according to estimates of sample complexity in this task (Jamieson et al., 2015). Nincty percent of the judgments were used to find a 3D embedding in which pairwise Euclidean distances amongst words minimize the crowd-kernel triplet loss on the training set (Tamuz et al., 2011). The resulting embedding was then tested by assessing its accuracy in predicting human judgments the held-out ten percent of data. The final embeddings predicted human decisions held-out triplets with 75% accuracy, which matched the mean level of inter-subject agreement on this task.
From the full set of items, we selected 15 tools and 15 reptiles for use in this study (as shown in Table 1). We chose these categories because they express both broad, superordinate distinctions (living/nonliving)) as well as finer-grat finer-grained internal structure (e.g. snakes lizards vs crocodiles).
The raw feature vectors were binarized by converting all non-zero entries to I, with the rationale that given feature is potentially true of a concept if at least one rater judged it to be so. We then estimated the conceptual similarity relations amongst all pairs of items by taking the cosine distance between their binarized feature vectors, and reduced the space three dimensions classical multidimensional scaling (Kruskal and Wish, 1978). The resulting embedding expresses conceptual similarity amongst 30 concrete objects, estimated via semantic feature listing and verification, in a study conducted in Dutch on large group of students living in Belgium in the early 2010s.
3.1.3 Pairwise similarity study
Our final estimate of conceptual structure relied on participants making similarity ratings between pairs of concepts from the set of 30 items using a standard point Likert scale. Unlike the previous two methods which implicitly arrive at a measure of similarity between concepts, this approach elicits explicil numerical ratings of pairwise similarity. To account for the diversity in ratings between people, we had multiple participants rate the similarity between each concept pair in dataset, with each participant seeing each pair in different randomized order.
3.1.2 Triadic comparison study
As a second estimate of conceptual structure amongst the same 30 items, we conducted triadic comparison or triplet judgment task in which participants must decide which of two option words is more similar in meaning to third reference word. From many such judgments, ordinal embedding techniques (Jamieson et al., 2015; Hebart et al., 2022: Hornsby and Love, 2020) can be used to situate words within a dow-dimensional space in
Participants were 10 MTurk workers recruited using CloudResearch. Each participant provided

---

Tasks
Concepts
Feature Listing
Triadic Comparison
Pairwise Similarity
Tools
Reptiles
List the properties Cobras
Which of bottom two words more similar to the word at the top?
On of how similar is Gecko Cobra?
Axe
Cobra
Nail
Human Participants
Knife
Gecko
Cobra: has scales, venomous, cold-blooded...
Gecko-Cobra Simlarity:
Cobra
Saw
Lizard
Spanner Toad
Gecko
Spanner
Chisel...
Calman...
Concepts from Ruts al. (2004)
GPT-8
(text-davinc-002)
The three tasks used to estimate conceptual structure both LLMs and Humans. The exact prompts used in our experiments with Humans and LLMs are shown in Table 4
Figure I:
informed consent in compliance with our Institutional IRB and was compensated for their time.
the Procrustes correlation pairwise between the dissimilarity matrices for the 30 concepts derived from the three tasks (Gower, 1975). This metric, analogous to 2. indicates the extent to which variations in pairwise distances from matrix are reflected in the other. The metric yielded substantial values of 0.96, 0.84, and 0.72 when comparing representations from the feature-listing task to the triplet-judgenan listing task to the pairwise comparison task, and the triplet task to the pairwise comparison task, respectively. All these values were significantly better than chance (p < 0.001), suggesting that in each comparison, distances in space accounted for 96%, 84%, and 72% of the variation observed in the other.". Thus despite differences in language, task, and cultures, the three estimates of conceptual structure were well- allgned,,,,,, suggesting that human conceptual representations of concrete objects are remarkably robust. We next consider whether the same is true of large language models.
Stimuli were each of the 435 ((2)) possible pains of the 30 tool and reptile concepts introduced in the earlier sections.
Procedure. On each trial the experiment, participants were presented with a question of the form 'How similar are these two things? (concept I and (concept 2)' and were provided with Likert scale below the question with the options 1: Extremely dissimilar, 2: Very dissimilar, 3: Likely dissimilar, 4: Neutral, Likely similar, Very similar, Extremely similar. On each trial (concept 1) and (concept 2] were randomly sampled from the set of 435 possible pairs of concepts and each participant completed 435 ratings trials rating each of the possible pairs.
task, the feature-
3.2 Results
We found that the inter-rater reliability within the feature-listing and pairwise rating tasks were quite high (r= .11 and r= .98 respectively). We could not compute similar metric in straightforward manner for the triadic judgement study because each participant was provided with unique set triplets. Figure 2 top row shows hierarchical cluster plots of the semantic embeddings from feature lists (left), the triadic comparison task (middle), and judgemeett task (right). Both embeddings strongly differentiate the living from nonliving items, and show comparatively little differentiation of subtypes within each category (though such subtypes are clearly apparent amongst the feature-listing embeddings). To assess the structural coherence among the three different embedding spaces, we computed the square of
Measuring Machine Conceptual Structure
In this section, we consider whether one of the most performant LLMs, OpenAl's GPT 3, expresses coherence the structural organization concepts when tested using the same methods used the human behavioral experiments. Using the OpenAl API, we conducted the feature listing and verification task, triadic comparison task, and the pairwise similarity rating task GPT 3. Given the recent deluge of open-source LLMs, also tested FLAN-T5 XXL, and FLAN-U2 triadic comparison and pairwise ratings tasks see how they perform relative to larger closed models. Finally for completeness we also tested
the

---

Feature listing
Triadic comparisons
Pairwise ratings
Humans
GPT-3 (002)
FLAN-XXL
Figure 2: Organization of conceptual representations estimated from Humans, GPT-3, and FLAN-XXL -XLL using feature listing, triadic comparisons, and pairwise ratings. Grouping of concepts is hased on hierarchical clustering of representations, Tools are shown in cooler colors and reptiles are in warmer colors.
the similarity between embeddings extracted from GPT 3, word2vec, and the language component of CLIP. While word2vec embeddings are staple of NLP research, relatively fewer works have explored the structure of the language models that are jointly trained in the CLIP procedure.
produced different responses from repetitions of the same query. For each concept We repeated the process five times and tabulated all responses across these runs for each item. The responses were transcribed into features by breaking down phrases sentence into constituent predicates; for instance, a response such as tiger is covered stripes" " was transcribed as "has stripes." Where phrases included modifiers, these were transcribed separately; for instance, phrase like "has long neck" was transcribed as two features, "has neck" and "has long neck." Finally, alternate wordings for the same property were treated denoting a single feature; for instance, "its claws are sharp" and "has razor-sharp claws" would be coded as the two features "has claws" and "has sharp claws." We did not, however, collapse synonyms or otherwise reduce the feature set. This exercise generated a total of 580 unique features from the 30 items.
After computing the similarity structure between concepts expressed by the NLP methods outlined above, we considered (a) how well these estimates aligned with structures estimated from human behaviors within each task, and (b) the structural coherence between the embeddings estimated via different methods from LLM behavior.
4.1 Methods
4.1.1
Feature listing simulations
To simulate the fcature-generation phase of the Leuven study, We queried GPT-3 with the prompt "List the features of a [concept]" and recorded the responses (see Table 3). The model was queried with temperature of 0.7, meaning that responses were somewhat stochastic so that the model
To simulate the feature verification phase of the Leuven study, we then asked GPT to decide, for each concept and feature F, whether the

---

concept possessed the feature. For instance, to assess whether the model "thinks" that alligators are ectothermic, we probed with the prompt "In one word, Yes/No: Are alligators ectothermic?" (temperature 0). Note that this procedure requires the LLM to answer probes for every possible concept/feature pair-for instance, does an alligator have wheels? Does car have heart? etc. These responses were used to flesh out the original feature-listing matrix: every cell where the LLM affirmed that concept had feature F was filled with a I, and cells where the LLM responded "no" were filled with zeros. We refer to the resulting matrix as the verified feature matrix. Before the feature verification process, the concept by feature matrix was exceedingly sparse, containing 786 I's (associations) and 16614 O's (no associations). After the verification process, the concept by feature matrix contained 7845 I's and 9555 O's. Finally, we computed pairwise cosine distances between all items based on the verified feature vectors, and used classical multidimensional scaling to reduce these to three- dimensional embeddings, exactly comparable to the human study.
for Flan-XXL to see a failure case). The verified feature lists additionally yield within-domain structure similar to that observed in human lists, with all items relatively similar to one another, and with some subcategory structure apparent (e.g. turtle/tortoise, snake, crocodile). within- domain structure estimated from triplet judgments, in contrast, looks very different.
These qualitative observations are borne out by the squared Procrustes correlations between different embedding spaces, shown in Figure 3. Similarities expressed in the space estimated from LLM verified feature lists capture 89% of the variance in distances estimated from human feature lists, 78% of the variance in those estimated from human triplet judgments, and 89% of the variance estimated from human pairwise similarities. Similarities expressed in the space estimated from LLM triplet judgments account for 82% of the variance in distances estimated from human feature lists, 80% of the variance those estimated from human triplet judgments, and 69% of the variance estimated from human pairwise similarities. Finally similarities expressed in the space estimated from LLM Pairwise comparisons account for 32% of the variance in distances estimated from human feature lists, 14% of the variance in those estimated from human triplet judgments, and 51% of the variance estimated from human pairwise similarities. Similarities estimated from LLM pairwise comparisons, contrast to those estimated from LLM feature lists and LLM triplets, account for less than half the variance in embeddings generated from human judgment. More interestingly, they account for less than half the variance in the embeddings generated from the LLM verified feature lists and LLM triplet judgements. Unlike the human embeddings, conceptual structures estimated from different behaviors in the very same model do not cohere very well with each other.
4.1.2 Triadic comparison simulations
To simulate triplet judgment, we used the prompt shown in Figure 1 for each triplet, using the exact same set of triplets employed across all participants in the human study. We recorded the model's response for each triplet (see Table 2) and from these data fit a 3D embedding using the same algorithm and settings as the human data. The resulting embedding predicted GPT- 3 judgements for the held-out triplets at 78 % accuracy, comparable to that observed in the human participants.
4.1.3 Pairwise similarity simulations
To simulate the pairwise similarity task, we used the prompt shown in Table 4 for all the possible pairs of concepts (435 (2))..
Figure 4 also shows the squared Procrustes correlations for semantic embeddings generated via several other approaches including (a) the raw (unverified) feature lists produced by GPT-3, (b) the word embedding vectors extracted from GPT-3's internal hidden unit activation patterns, (c) word embeddings from the popular word2vec approach, and (d) embeddings extracted from a CLIP model trained to connect images with their natural language descriptions. None of these
4.2 Results
Hierarchical cluster plots for embeddings generated from the LLM's feature lists, triadic judgements, and pairwise judgments
shown in the second and third rows of Figure 2, immediately below the corresponding plots from human data. Most approaches reliably separate living and nonliving things (although see the pairwise representations

---

Tiilet Festure Listing
Humans
GPT-3 (002)
FLAN-XXL
10
0.96
0.84
1
0.72
0.47
06
0.96
0.72
0.72
1
0.25
NA-
00
Pairise
0.84
0.72
0.47
0.25
1
camparsers FLAA-.
wee the
Trilett
Phinwise
ConturiLissiig
Tippet
Painuiie
Feature
Triplt
Raiwise
Figure 3: Matrices showing conceptual coherence across testing methods for Humans, GPT-3, and FLAN-XXL. The value in each cell corresponds to the squared Procustes pairwise correlation.
1.0
Human Feature Listing
Feature Ust
Parvise
there might ways to optimize human- machiic conceptual alignment using in-context learning (Brown et al., 2020b; Chan et al., 2022) or specialized prompting strategies (Wei et al., 2022) to the extent that the community wants to deploy LLMs to interact with humans in natural language and use them as cognitive models, it is crucial characterize how stable the models' concepts are without the use of specialized prompting.
0.8
Enteedding
0.
0.0
Human Triadic Comparison
1.0
0.8-
0.2
Conclusion
0.0
Human Pairwise Similarity
In this study, we compared the conceptual structures of humans and LLMs using three cognitive tasks: a semantic feature-listing task, a triplet similarity judgement task, and pairwise rating task. Our results showed that the conceptual representations generated from human judgments, despite being estimated from quite different tasks, in different languages, across different cultures, were remarkably coherent: similarities captured in one space accounted for 96% of the variance in the other. This suggests that the conceptual structures underlying human semantic cognition are remarkably robust to differences in language, cultural background, and the nature of the task at hand.
1.0
04
02
GPT-3 (092) ber
0.0
FLAN-OXLL (re)
GPT (002) (umver)
GPT-3 (002) Viglel
FLAN-XXL Tripeat
FLAN FLAN-UL2 Teplot
GPT-3 (002) Patietse
FLAN-UL Painsiue
GPT-3 (003) Painwise
GPT a embedding
Woni2Vo
CUP
Figure Human-mann machine alignment using human semantic representations estimated from feature listing (top), triadic comparisons (middle), and pairwise ratings (bottom). Heights of bars correspond to squared Procrustes correlations.
In contrast, embeddings obtained from analo- gous behaviors in LLMs differed depending upon on the task. While embeddings estimated from verified feature lists aligned moderately well with those estimated from human feature norms, those estimated from triplet judgments or from the raw (unverified) feature lists did not, nor did the two embedding spaces from the LLM cohere well with each other. Embedding spaces extracted directly from model hidden representations or from other common neural network techniques did not fare better: in most comparisons, distances captured by one model derivee embedding space accounted
approaches accord with human-based embeddings well as do the embeddings estimated from the the various
LLM verified-feature lists, structures particularly coherent with one another. No pair of LLM-estimated embeddings shows the degree of coherence observed between the estimates derived from human judgments.
In general, while LLMs vary in their degree of human alignment with respect to conceptual structure depending on the probing technique, the critical finding is that they are not coherent within- themselves across probing techniques. While

---

for, at best, half the variance in any other. The sole exception was the space estimated from LLM- verified feature vectors, which cohered modestly well with embeddings taken directly from the GPT- 3 embeddings obtained using the triplet task (72% of the variance) and the hidden layer (66% of variance)5.
ill-suited for understanding human conceptual representation.
Limitations
While there are benefits to studying the coherence of a constrained set of concepts, as we have done here, human semantic knowledge is vast and diverse and covers many domains beyond tools and reptiles. While it reasonable to conduct our experiments on 30 concepts split across these domains both due resource limitations and limit the concept categories to those that are largely familiar to most people, a larger scale study on larger concept sets (Hebart et al., 2022; Devereux et al., 2014; McRae et al., 2005) might reveal different degree of coherence in conceptual structure across probing methods in LLMs.
While recent advances in prompting techniques including chain-of-thought prompting (Wei et al., 2022), ceff-consistence (Wang et al., 2022), and tree-of- tre-of-thoughts (Yao et al., 2023) have been shown improve performance in tasks with veridical solutions such as mathematical reasoning and knowledge retrieval, we highlight here through both direct and indirect tasks that the underlying conceptual structure learned by LLMs is brittle. We implemented chain-of- thouhht reasoning for some models and found that this led to LLM model representations being more aligned with human conceptual structure (Fig 6). However, the conceptual coherence within a model only increased for only some of the models but still nothing comparable to human conceptual robustness.
When conducting LLM simulations, didn't employ any prompting technique like tree-of- thought(Yas (Yao al., 2023), self-consistencyyWaar et al., 2022), etc.. While we think that the purpose of this work to highlight the fragility of the conceptual 'core' of models as measured by incoherent representations across tasks, it remains possible that representations might cohere to a greater extent using these techniques and might align closer with human representations.
Together these results suggest an important difference between human cognition and current LLM models. Neuro-commen neerrocomputationall models of human semantic memory suggest that behavior across many different tasks is undergirded by common conceptual "core" that is relatively insulated from variations arising from different contexts or tasks (Rogers al., 2004; Jackson et al., 2021). In contrast, representations of word meanings in large language models depend essentially upon the broader linguistic context, Indeed, in transformer architectures like GPT- 3. each word vector is computed a weighted average of vectors from surrounding text, it is unclear whether any word possesses meaning outside or independent of context. Because this is so, the latent structures organizing its overt behaviors may vary considerably depending upon the particular way the model's behavior is probed. That is, the LLM may not have coherent conceptual "core" driving its behaviors, and for this reason, may organize its internal representations quite differently with changes to the task instruction or prompt. Sonsevttsy of this kind is precisely what grants such models their notable ability to simulate natural natural-sceming language, but this same capacity may render them
Finally, Human semantic knowledge is the product of several sources information including visual, tactile, and auditory properties the concept. While LLMs can implicitly acquire knowledge about these modalities via the corpora they are trained on, they nevertheless bereft of much of the knowledge that humans exposed to that might help them organize concepts into a more coherent structure. In this view, difference in the degree in conceptual coherence between LLMs and humans should not be surprising.
7 Code
Here's the link to the repository containing the code and data we used to run our experiments.
References
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135-146.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind

---

Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020a. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rxiv:1810004805.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020b. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray, 2023. Can ai language models replace human participants? Trends in Cognitive Sciences.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang. and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). pages 320-335.
Curt Burgess. 1998. From simple associations the building blocks of language: Modeling meaning in memory with the hal model. Behavior Research Methods, Instruments, & Computers, 30(2))188-19))
Stephanic CY Chan, Ishita Dasgupta, Junkyung Kim, Dharshan Kumaran, Andrew K Lampinen, and Felix Hill. 2022. Transformers generalize differently from information stored in context vs in weights. arXiv preprint arXiv:2210.0567.
Katherine Elkins and Jon Chun. 2020. Can gpt-3 pass writer's turing test? Journal of Cultural Analyties. 5(2).
Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179-211.
Jonathan H Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarez. 2023, Chatgpt goes to law school. Available SSRN.
Jeffrey L Elman. 2004. An alternative view of the mental lexicon. Trends in cognitive sciences, 8(7):3013366.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
JL Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning, 7(2-3):195-225
John Firth. 1957. A synopsis of linguistic theory, 1930- 1955. Studies in linguistic amalysis, pages 10-32.
Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James I. McClelland, and Felix Hill. 2022. Language models show human-like content effects reasoning. arXiv preprint urXiv:2207.0705.
John C Gower. 1975. Generalized procrustes analysis. Psychometrika, 40:33-51.
Gabriel Grand, Idan Asher Blank, Francisco Pereira, and Evelina Fedorenko. 2022. Semantic projection recovers rich human knowledge of multiple object features from word embeddings. Nature human behaviour, 6(7):755 987.
Simon De Deyne, Steven Verheyen, Fef Ameel, Wolf Vanpaemel, Matthew J Dry, Wouter Voorspoels. and Gert Storms. 2008 Exemplar by feature applicability matrices and other dutch normative data for semantic concepts. Behavior research methods, 40:1030- 1048.
Hannes Hansen and Martin N Hebart. 2022. Semantic features of object concepts generated with gpt-3, arXiv preprint arXiv:2202.03753.
Scott Deerwester, Susan Dumais, George Furnas, Thomas K Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391- 407.
Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146-162.
Martin N Hebart, Oliver Contier, Lina Teichmann, Adam Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vazini-Pashkam. and Chris Baker. 2022. Things-data: A multimodal collection of large-scale datasets for investigating object representations brain and behavior. bioRxiv. pages 2022-07.
Dorottya Demszky, Diyi Yang, David Yeager, Christopher Bryan, Margarett Clapper, Susannah Chandhok, Johannes C Eichstaedt, Cameron Hecht, Jeremy Jamieson, Meghann Johnson, et al. 2023. Using large language models in psychology. Nature Reviews Psychology, pages 1-14.
Jordan Hoffmann, Sebastian Borgcaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford. Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. 2022. Training compute- optimal large language models. arXiv preprint arXiv:2203.1 15556.
Barry Devereux, Lorraine K Tyler, Jeroen Geertzen, and Billi Randall. 2014. The centre for speech, language and the brain (esIb) concept property norms. Behavior research methods, 46:1 19911127.

---

Adam N Hornsby and Bradley C Love. 2020. How decisions and the desire for coherency shape subjective preferences over time. Cognition, 200:104244.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation word representations in vector space. In International conference on machine learning, pages 1188-1196.
Rebecca L Jackson, Timothy T Rogers, and Matthew A Lambon Ralph. 2021. Reverse-engincerigg the cortical architecture for controlled semantic cognition. Nature human behaviour, 566):774-776.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, pages 1-11-3119.
Kevin G Jamieson, Lalit Jain, Chris Fernandez, Nicholas Glattard, and Robert D Nowak. 2015. Next: A system for real-world development, evaluation, and application of active learning. In NIPS, pages 2656-2664. Citeseer.
Kushin Mukherjee, Siddharth Suresh, and Timothy T Rogers. 2023. Human-machine cooperation for semantic feature listing. arXiv preprint arXiv:2304.0500
Philip Mark Newton and Maira Xiromeriti. 2023. Chatgpt performance meq exams in higher education. a pragmatic scoping review.
Joseph B Kruskal and Myron Wish. 1978. Multidimen- sional scaling, volume II. Sage.
Tiffany H Kung. Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz- Candido, James Maningo, et al. 2023. Performance of chatgpt usmle: Potential for ai-assisted medical education using large language models. PLoS digital health, 2(2):e0000198
OpenAl. Year of the webpage, e.g., 2023. GPT-4: A powerful language model. Accessed on: Date of your access, e.g., September 27, 2023.
Charles Osgood. 1952. The nature and measurement of meaning. Psychological bulletin, 49(3):197.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll I. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
Thomas Landauer, Peter W Foltz, and Darrell Laham. 1998. An introduction to latent semantic analysis. Discourse processes, 25(2-3):59--844
Gary Lupyan and Molly Lewis. 2019. From words-as- mappings to words-as-cues: The role of language in semantic knowledge. Language, Cognition and Neuroscience, 14110):131-1337.
Timothy T Rogers, James I. McClelland, et al. 2004. Semantic cognition: A distrruld processing approach. MIT press.
distributed
Raja Marjich, Ilia Sucholutsky, Theodore R Sumers, Nori Jacoby, and Thomas Griffiths. 2022a. Predicting human similarity judgments using large language models. arXiv preprint 047228.
Eleanor Rosch. 1975.
Cognitive representations
of semantic categories. Journal of experimental psychology: General, 104(3):192.
Eleanor H Rosch. 1973. On the internal structure of perceptual and semantic categories. Cognitive development and acquisition of language, pages 1- 144. Elsevier.
Raja Marjich, Ilia Sucholutsky, Pol van Rijn, Nori Jacoby, and Thomas L Griffiths. 2023. What language reveals about perception: Distilling psychophysical knowledge from large language models. arXiv preprint ar 2x1i::2302.01308.
David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. 1986. Learning representations by back- propagating errors. nature, (3366)8))5333-33
Raja Marjich, Pol van Rijn, Ili Sucholutsky, Theodore R Sumers, Harin Lec, Thomas L Griffiths, and Nori Jacoby. 2022b. Words are all you need? capturing human sensory similarity with textual descriptors. arXiv preprint arXiv:2206.04105.
Siddharth Suresh, Kushin Mukherjee, and Timothy T Rogers. 2023. Semantic feature verification in flan-15. arXiv preprint 2.44::33000059/
Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir, and Adam Tauman Kalai. 2011. Adaptively learning the crowd kernel. arXiv preprint 1rXiv: 1033.
JL McClelland, DE Rumelhart, and the PDP Research Group. 1990. The development of distributed representations for words. Explorations in parallel distributed procexsing: a handbook of models, programs, and exercises, pages 77-109
Rohan Taori, Ishaan Gulrajani. Tianyi Zhang, Yann Dubois. Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori Hashimoto. 2023. Alpaca: A strong, replicable instruction-folllwiii Stanford Center for Research on Foundation Models. hupp::/erffm stanford. edu/2023/0 3/alppcc himl, 3(6)77
Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior research methods, 37(4))547.
model.

---

Bill Thompson, Seán Roberts, and Gary Lupyan. 2020. Cultural influences on word meanings revealed through large-scale semantic alignment. Nature Human Behaviour, 1020):102--0388
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 13971.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 22Xiv:2003.11171.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint 2rXhv:21090016
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain thought prompting elicits reasoning large language models. arXiv preprint arXiv:2201. 1903.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao. and Karthik Narasimhan. 2023, Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint ar35v:2305. 10607.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint 22xiv:2005.0106

---

A
Appendix
Table Concepts used in the experiment
Concepts
Categories
15 Reptiles
Turtle
Alligator
Lizard
Tortoise
Cobra
Snake
Blindworm
Gecko
Boa python
Toad
Crocodile
Chameleon
Caiman
Salamander
Dinosaur
15*Tools
Hammer
Screwdriver
Grinding disc
Vacuum cleaner
Spanner
Lawn mower
Axe
Saw
Knife
Nail
Chisel
Shovel
Anvil
Oilcan
Paint brush
Head
Option
Option 2
FLAN-T5-XXL FLAN-UL2
davinci-002
davinci-O0s
Shovel
Alligator
Spanner
Alligator
Alligator
Spanner
Spanner
Anvil
Caiman
Tortoise
Tortoise
Caiman
Caiman
Caiman
Nail
Boa python
Snake
Snake
Snake
Boa Python
Boa python
Paint brush
Chisel
Toad
Chisel
Toad
Chisel
Chisel
Shovel
Caiman
Crocodile
Crocodile
Dangerous
Crocodile
Crocodile
Table 2: Example responses to triplet judgement task by different models.

---

Concepts
Prompt
Features
Alligutor
List all features of an Alligator.
have tail, can stay underwater, have tough skin
Anvil
List all features of an Anvil.
have wings, have hummer, have hole...
Axe
List all features of an Axe.
have blade, can be used for chopping wood, are a tool ...
Blindworm
List all features of Blindworm.
have no legs, can smell with tongue, are small...
Boa python
List all features of Boa python.
have pits, can be green, are dimorphic...
Caiman
List all features of Caiman.
have short body,can swim, are reptile ...
Chamelcon
List all features of Chamelcon.
have tongue, can change color, are a good swimmer. have a blade, can cut material, are hand held... ...
Chisel
List all features of Chisel.
Cobra
List all features of Cobra.
have long body, can inject venom, are carnivoruus.. have leeth, can breathe air, swimmer... ...
Croendile
List all features of Crocodile.
Dinosaur
List all features of a Dinosaur.
have claws, can lay eggs, are reptiles... ...
Gecko
List all features of Gecko.
have tail, can stick to surfaces, are nocturnal..
Grinding disk
List all features of Grinding disk.
have diameter, have handle, can be used for pounding nails, are tool have blade, can cut things, are sharp...
be used for grinding. abrusive... ...
Hammer
List all features of Hummer.
Knife
List all features of a Knifc.
Lawn Mower
List all features of Lawn Mower.
have engine. mulch, are powered by gas
Lizard
List all features of Lizard.
have legs,can climb, are carnivores ...
Nail
List all features of a Nail.
have covering, have point, are metal
Oil can
List all features of an Oil can,
have a spout, can pour, are used to hold oil... have handle, be used for painting, are tool have tail, can a variety of colors, are an amphibian have handle, can be used to cut lumber, ure tool ...
Paint brush
List all features of Paint brush.
Sulamander
List all features of Salamander.
Saw
List all features of a Saw.
Screwdriver
List all features of Screwdriver.
have handle, have tip, have cap... ...
Shovel
List all features of Shovel.
have point. have blade, have handle...
Snake
List all features of Snake,
have long body, can he dangerous to humans, are camivorous.... ...
Spanner
List all features of Spanner.
have different tips, can grip holt, are tool...
Toad
List all features of Tood.
have glands, can jump, good jumpers...
Tortoise
List all features of Tortoise.
have shell, can live long, are cold blooded...
Turtle
List all features of a Turtle.
have shell, can swim, are reptiles ...
Vacuum
List all features of Vacuum.
have nozzle, can suck up dirt, have filter..
Table Examples of features produced by GPT-3.

---

Human norms
Human triplet
0.96
0.72
0.78
0.8
0.2
0.8
0.14
0.77
0.078
0.0021
0.13
0.17
0.33
0.57
0.12
Human Pairwise
(00-
0.89
0.78
0.88
0.72
0.47
0.72
0.38
0.93
0.32
0.022
0.42
0.48
0.65
0.28
1.0
GPT-3(002) triplet
GPT-3(002) Pairwise
0.32
0.51
0.47
0.25
0.25
0.64
0.31
0.099
0.66
0.633
0..6
0.27
0.4
GPT-3(002) triplet-
0.39
GPT-3(003) Pairwise
0.24
0.14
0.42
0.38
0.2
0.64
0.2
0.4
0.098
0.081
0.55
0.55
0.59
0.26
0.39
FLAN-XXL(ver) 0.00
0.77
6.89
0.91
0.21
0.73
0.66
FLAN-XXL Triplet-c.0e
0.078
0.14
0.12
0.12
0.32
0.098
0.12
0.059
0.11
00.098
0.1
00.08
0.069
FLAN-UL2 Triplet
0.059
-02
FLAN-UL2 Pairwise
0.25
0.49
0.13
0.74
0.42
GPT-3(002 0.29
0.51
0.21
0.55
3.54
0.098
0.79
0.47
GPT embedding
0.74
0.44
Word2Vec
0.63
0.57
0.61
0.6
0.57
0:27
0.57
0.26
0.085
0.033
0.24
0.23
0.43
0.19
CLIP
0.2
0:33
0.20
0.2
0.2
0.39
0.30
0.069
0.054
0.42
0.42
0.64
0.19
Human nomss
Human triplet
Human Pairwise-
TPT-31002) triplet
GPT-3 Palrwise
GPT-310022 triplet
GPT-3(003) FLAN-XXL[ver]
FLAN-XXL Triplet
FLAN-ULZ Triplet
GPT embedding
Word2Vec
GPT-3(002)(vee
6PT:3/002)unven)
FLAN-UL2
Figure 5: A correlation matrix showing square of the Procrustes correlations between semantic representations
estimated from different models across different tasks.

---

Human
3.94
Human triplet
3.96
0.72
0.91
0.14
0.97
0.074
0.85
0.38
1.0
Human Pairwise
0.84
0.77
0:24
0:77
0.8
GPT3(002) COT Triplet-
0.6
GPT3(002) COT Pairwise
0.24
0.14
0.45
0.37
0.16
0.61
0.17
0.51
GPT3(003) COT Triplet-
GPT3(003) COT Painwise
0.16
0.074
0.37
0.093
0.61
0.098
0.14
0.57
FLAN-UL2 Triplet-
FLAN-UL2 COT Painwise
0.3
0.18
0.54
0.22
D:11
0.21
0.57
0.27
Human norms
Human triplet.
Human Poirwise-
GPT3(002) COT Pairwise
GPT31003 COT Triplet
GPT3(003) COT Pairwise
FLAN-UL2 COT Triplet
FLAN-UL2 COT Pairwise
GPT3/002) COT Tripl
Figure correlation matrix showing square the Procrustes correlations between semantic representations
estimated from different models across different tasks using only Chain of Thought prompting.

---

Humans
LLMs
Feature generation
This bundle contains up 10 sheets with word written top of the page. We would like you to write down preferably 10 features underneath the word. Try to give different sorts of features, such as, for example, physical or perceptual features (what it looks like, how it smells, how it tastes, ...), functional features (what it is used for, when and where it is used, ...), background information (where it comes from, some historical facts, ...).
List all the properties of (conceptl)
etc. (De Deyne al., 2008))
Feature verification
The participants were instructed to judge, for every feature-cxemplar pair, whether the feature characterizes the exemplar, and to write down or O in the corresponding matrix entry. (De
In one word, Yes/No : Are [concept]) ) { |property| (or) In one word, Yes/No: Do (concept1) ) have (propertyl) )/
Deyne et al., 2008)
Triplet task
Which of the bottom two words more similar to the word at the top?
Answer using only one word (con- cept1] or (concept2) and not [anchor). Which is more similar in meaning to {anchor)?"
Pairwise similarity
On scale of 1-7, how similar is [conceptl] to (concept2)?
Answer with only one number from 1 to 7, considering 1 as 'extremely dissimilar's 'very dissimilar', 3 as 'likely dissimilar', as 'neutral', as 'likely similar', as 'very similar', and as 'extremely similar': How similar is {conceptl] | and (concept2)?
Table 4: Prompts used across the three tasks for humans and LLMs


Reviews:
[{"review_id": "7iAMfSH3tO", "reviewer": "Reviewer_DWHV", "summary": "", "questions": "- The diversity of tasks and populations is a strength, but why were these particular datasets chosen (convenience?) Clarifying the motivation here may be helpful, but this is a minor point", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer taking the time to read our paper and for their positive comments! We address the concerns raised below - \n\n>Ad hoc choices are, by necessity, used to adapt the human-studies for use with the large language models (e.g. how features were transcribed, how feature verification was implemented etc.) The work might be more impactful if combined with a robustness analysis of the importance of specific analytic choices (this is a small point; time is finite.)\n\nWe thank the reviewer for raising this pertinent point. Firstly, our main aim was to implement all tasks (feature listing, feature verification, triplet judgements, pairwise judgments) in the LLMs as similarly as they were conducted in human participants to provide a fair comparison. For LLMs and humans, we asked the agent to list all the features that they knew to be true for a given concept. \nThese features were all compiled into a single concept X feature matrix, and the feature verification task was also conducted in a parallel manner between LLMs and humans. \nHere are the specific prompts that were used with Humans and LLMs for the various tasks.\n| | Humans | LLMs |\n|- |-|-|  \n| Feature generation | This bundle contains up to 10 sheets with a word written on top of the page.  We would like you to write down preferably 10 features underneath the word.  Try to give different sorts of features, such as, for example, physical or perceptual features (what it looks like, how it smells, how it tastes, ...), functional features (what it is used for, when and where it is used, ...), background information (where it comes from, some historical facts, ...), etc. ((De Deyne et al., 2008)) | List all the properties of {concept1}|\n| Feature verification | The participants were instructed to judge, for every feature-exemplar pair, whether the feature characterizes the exemplar, and to write down a 1 or a 0 in the corresponding matrix entry. (De Deyne et al., 2008)) | In one word, Yes/No : Are {concept1} {property1}  (or) In one word, Yes/No : Do {concept1} have {property1}/ |\n| Triplet task| Which of the bottom two words is more similar to the word at the top? | Answer using only one word - {concept1} or {concept2} and not {anchor}. Which is more similar in meaning to {anchor}?\"|\n| Pairwise similarity| On a scale of 1-7, how similar is a {concept1} to a {concept2}? | Answer with only one number from 1 to 7, considering 1 as 'extremely dissimilar', 2 as 'very dissimilar', 3 as 'likely dissimilar', 4 as 'neutral', 5 as 'likely similar', 6 as 'very similar', and 7 as 'extremely similar': How similar is {concept1} and {concept2}? |\n\nHowever, the reviewer raises a fair point — LLMs are not humans and might be more or less performant with slightly different prompting strategies. This would, in our view, support the central thesis of the paper that conceptual structure does not cohere in LLMs. But to address the reviewer’s concerns, we implemented chain-of-thought reasoning [1], a now well-known prompting strategy shown to improve LLM reasoning capabilities. We found that this led to slightly higher conceptual coherence for only some of the models but still nothing comparable to human conceptual robustness. We will include these results in the supplementary materials.\n\nReferences:\n[1] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824-24837.\n\n>The diversity of tasks and populations is a strength, but why were these particular datasets chosen (convenience?) Clarifying the motivation here may be helpful, but this is a minor point.\n\nThank you for this question! Briefly, the Leuven semantic norms dataset is the only dataset, to our knowledge, that includes a feature verification phase after feature generation. Amongst the subset of concepts we included, they were chosen to span semantic domains (living and nonliving) and levels of granularity. Future studies can work on expanding our approach to other semantic norm datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting application of methods from cognitive psychology and research on concepts to large language models\n- Use of multiple human datasets spanning tasks, languages, and participant groups is a strong way to demonstrate similarity of human conceptual structure against variable experimental settings.\n- Direct comparison between LLM and human data is valuable\n- Testing multiple LLMs helps to support generalizability of findings", "reasons_to_reject": "- Ad hoc choices are, by necessity, used to adapt the human-studies for use with the large language models (e.g. how features were transcribed, how feature verification was implemented etc.) The work might be more impactful if combined with a robustness analysis of the importance of specific analytic choices (this is a small point; time is finite.) ", "questions_to_authors": "- The diversity of tasks and populations is a strength, but why were these particular datasets chosen (convenience?) Clarifying the motivation here may be helpful, but this is a minor point", "paper_topic_and_main_contributions": "This paper probes the conceptual representations in a class of large language model using cognitive psychology methods designed to assess the similarity between concepts; results are compared to outcomes from the same methods applied to human participants. Results showing some similarities, but striking differences, are relevant for work pushing forward on the strengths and weaknesses of large language models as models for human cognition.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YP6ug1v1ar", "reviewer": "Reviewer_qZ7X", "summary": "", "questions": "A. Have you tried running the Dutch experiments with GPT-3? It is unclear exactly what languages the model has been exposed to, but it might be an additional point of comparison to see how consistent measures derived from the same model in other languages are (note, I understand that this isn’t possible for some of the other models investigated, so I understand why it isn’t included in the paper). ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read our paper and for their encouraging comments. Below, we address the concerns raised in a point-by-point manner.\n\n>With respect to framing, the introduction of the paper asserts a few claims about the abilities of models like Chat GPT that are unsupported textually and/or difficult to substantiate (and thus prone to unintended misreadings). For example, lines 132-136 list a variety of abilities of ChatGPT, including the claim that the model “can write essays sufficient to earn a B in a typical college class”. No citation is given for these claims. Additionally, it is claimed that the model demonstrates “hallmarks of conceptual abilities that until recently were uniquely human”. This can be read as asserting that models have these conceptual abilities, which is at-issue in the paper. Finally, models are deemed intelligent, which is not substantiated. Again, these are minor.\n\nThank you for prompting us to be more careful in the claims we make and in our choice of wording. Here, we note some of the references that would help support our claims. The claims range from documented news articles - ChatGPT scoring Bs in essays: https://www.wbur.org/news/2023/07/26/harvard-student-chat-gpt-experiment-maya-bodnick— to further evidences from earlier last year documenting the ability of off-the-shelf chatGPT to pass multiple-choice exams [1], medical licencing exams [2], and the exams relating to the law [3]. Of course, this is because chatGPT might be trained on corpora containing materials from these exams, but it nevertheless remains unchanged that within these domains its behavior is impressive. We would add these references in addition to the caveat regarding the training data as mentioned above, if this paper were to be accepted.\n\nRegarding our claim that chatGPT `demonstrates “hallmarks of conceptual abilities that until recently were uniquely human”`, we completely agree that this sentence could be misleading. Our point was that on the surface, it appears as though these models have conceptual abilities if we were to simply judge the model based on the generated text for most normal conversation scenarios. However, as we demonstrate with our experiments in this paper, this is not the case and GPT-like models do not in fact have a conceptual core.\n\nTo our knowledge, we do not refer to the models as ‘intelligent’ anywhere. If the reviewer is referring to the sentence on line 156 ‘as a means of understanding whether and how these intelligences differ’, we use the term intelligence liberally to refer to any system that is capable of generating output given a query. We can relax this and refer to the GPT models as ‘agent’ or just be direct and say the ‘difference between humans and Transformer models’, if the reviewer believes it to be appropriate.\n\n\n>With respect to model comparisons, there is an implicit move to treat different models (sometimes quite different, e.g., word2vec vs. GPT-3.5) as analogous to different human participants (or populations). It is possible, that only certain models produce conceptual structures which are comparable, and thus, this comparison is problematic. The results in the paper about the lack of consistency across tasks shows that even for a single model, they behave in non-human-like ways. Nonetheless, it could be beneficial to note this in the paper.\n\nHere, we do not intend to treat different models as different populations or participants. Instead, we attempt to see which (if any) model can capture the behavioral patterns expressed by a population of human participants. As the reviewer notes, some models better capture conceptual structure than others (GPT-3 is better than FLAN is better than CLIP), but GPT-3 feature listing and GPT-3 triplets, for example, are not as coherent with each other as human feature listing is with human triplets. We take this as evidence that models lack conceptual coherence. Ideally, we agree that it would be a powerful result if, say, GPT3 triplet features strongly cohered with FLAN-XXL triplet features, but we only present the data here for completeness rather than to make a claim about cohere between models.\n\n> Have you tried running the Dutch experiments with GPT-3? It is unclear exactly what languages the model has been exposed to, but it might be an additional point of comparison to see how consistent measures derived from the same model in other languages are (note, I understand that this isn’t possible for some of the other models investigated, so I understand why it isn’t included in the paper).\n\nWe think that this is an excellent suggestion and thank the reviewer for it. LLMs seem to have multilingual capabilities[4] and we believe that future work should strive to run precisely the same experiments in different languages on the LLMs as well as construct new human benchmark datasets to validate the model results. It would be interesting to see if conceptual structure coheres more or less for some languages vs. others (e.g., high vs. low resource languages, same vs. different family of languages). The authors of this paper do not speak Dutch unfortunately so it is out of scope for us to run this study in the short term, but if the paper is accepted we will aim to note the strong need for such experiments in the Conclusions section.\n\n\nReferences:\n\n[1] Newton, P. M., & Xiromeriti, M. (2023). ChatGPT Performance on MCQ Exams in Higher Education. A Pragmatic Scoping Review.\n[2]Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., ... & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS digital health, 2(2), e0000198.\n[3]Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. (2023). Chatgpt goes to law school. Available at SSRN.\n[4]Armengol-Estapé, J., Bonet, O. D. G., & Melero, M. (2021). On the Multilingual Capabilities of Very Large-Scale English Language Models. arXiv preprint arXiv:2108.13349.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is great and enjoyable to read. The argument is clear, the experiments are well described and motivated, and the results are engaging. The work highlights the brittleness of conceptual knowledge in neural models, which will (hopefully) spur researchers to explore new modeling techniques to bridge the gap between humans and state-of-the-art models. ", "reasons_to_reject": "There are two (minor) issues in the paper: (i) a framing problem around the (actual) capacities of current models, and (ii) sparse discussion of the validity of the model comparisons. These are not reasons to reject, but there is no natural place in the review form to put these, so I include them here. Ultimately, I believe these two issues are easily addressable in revisions and that the paper should be accepted. \n\nWith respect to framing, the introduction of the paper asserts a few claims about the abilities of models like Chat GPT that are unsupported textually and/or difficult to substantiate (and thus prone to unintended misreadings). For example, lines 132-136 list a variety of abilities of ChatGPT, including the claim that the model “can write essays sufficient to earn a B in a typical college class”. No citation is given for these claims. Additionally, it is claimed that the model demonstrates “hallmarks of conceptual abilities that until recently were uniquely human”. This can be read as asserting that models have these conceptual abilities, which is at-issue in the paper. Finally, models are deemed intelligent, which is not substantiated. Again, these are minor. \n\nWith respect to model comparisons, there is an implicit move to treat different models (sometimes quite different, e.g., word2vec vs. GPT-3.5) as analogous to different human participants (or populations). It is possible, that only certain models produce conceptual structures which are comparable, and thus, this comparison is problematic. The results in the paper about the lack of consistency across tasks shows that even for a single model, they behave in non-human-like ways. Nonetheless, it could be beneficial to note this in the paper. \n", "questions_to_authors": "A. Have you tried running the Dutch experiments with GPT-3? It is unclear exactly what languages the model has been exposed to, but it might be an additional point of comparison to see how consistent measures derived from the same model in other languages are (note, I understand that this isn’t possible for some of the other models investigated, so I understand why it isn’t included in the paper). ", "paper_topic_and_main_contributions": "This work compared conceptual structure in humans with large language models. At issue was how coherent conceptual structure was across different tasks (and for humans different languages). Three paradigms were employed: (i) a feature listing task, (ii) a triadic similarity-matching task, and (iii) a Likert-scale pairwise similarity task. While human responses are consistent across tasks, languages, and populations, neural languages models are not. \n\nThe main contribution of this work is a careful investigation of the conceptual knowledge of a range of neural models. If the materials from this paper are released publicly upon publication, the paper also contributes a paradigm for evaluating models along with some human baseline data. \n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 279: norms( --> norms (\n\nLine 601: variance)5. Presumably 5 is a typo\n\nLine 663: Finally, Human --> Finally, human\n"}, "extra_text": {}}, {"review_id": "omIRjj4lLA", "reviewer": "Reviewer_P15m", "summary": "", "questions": "Why do you map feature spaces to three dimensions? Unless I'm missing something, all evaluations would work in higher dimensional spaces as well, and this dimensionality reduction may get rid of signal. Could this be justified better?\n\nIt seems like participants did not flinch, but a screwdriver may be a drink and the translation will inevitably get rid of some polysemy while potentially introducing some which was not there. Did you handle this? If so, how? If not, can you comment on how this may impact your results? Participants saw all stimuli and may have understood polysemous terms in the intended meaning because of surrounding concepts, but LLMs may not since, if I understand correctly, you presented each prompt separately, making it impossible for the model to use preceding trials as context. In a similar fashion, \n\nWhen you sample 90% of the responses for training and 10 for testing, how was the sampling performed? fully random? stratified on something? You mention 75% accuracy in out-of-sample prediction: could you provide a stupid baseline for comparison to better appreciate the figure?\n\nI'm slightly concerned but mostly interested in possible order effects in the human data for pairwise similarity ratings. Given the very closed domain and the repetition of concepts in different pairs, is it possible that the way people interpreted the Likert scale changed as they saw more pairs? to exemplify: if I see the pair boa-crocodile first, I may choose a 6, indicating they are very similar. However, if I then see the pair crocodile-caiman I may also give a 6 but if the two trials are presented in the opposite order crocodile-boa would look less similar and probably elicit a 5. Considering that all raters rated all pairs in different order, I think it would be informative to investigate whether the order played a role. I don't have strong intuitions about how this analysis should be carried out, but I'd appreciate seeing it in an appendix to better situate the results. \n\nHow does the verification phase work? Does GPT-3 hallucinate in front of crocodiles having blades or not? did you check that the verification worked or did you trust it as such?\n\n\n-------------------------------------------------------------\nThanks for the thorough and insightful answers!", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and for the thoughtful critiques. Below we respond to them in a point-by-point manner-  \n\n>I have only one real possible reason why I may not fully trust the results: human participants did the triplet similarity responding to a prompt asking to decide which of two words is more similar to the target, with no further qualifiers. By my understanding, the model was prompted with the same text. Is there a chance that some models did not interpret similar as semantically similar but rather, e.g., similar from a word from point of view? would results differ if the prompt was more specific about meaning relations? I agree that if we have to change the task specification for a model to perform well whereas humans are flexible enough to fill in the ambiguity and do the task it's not really good for the models, but Turner's are skilled at taking experiments whereas LLMs may not. The same concern applies to the pairwise similarity: can it be confirmed that the models interpret the 'similar' in the prompt as pertaining to lexical semantics?\n\nWe agree that it is hard to understand if the model is paying attention to the semantic similarity or the word similarity. In our triplet and pairwise task, we asked the model to make a judgment based on the ‘meaning’ of the words as shown by the prompt table in response to reviewer DWHV. To make this more concrete, we ran a new set of experiments (on triplet judgment and pairwise similarity )where we asked Flan-xxl and gpt-3(002) to make a judgment based on the ‘semantic meaning’ of the words. We find that in some cases, it decreases coherence but in some, it increases coherence. In the cases where the model coherence increases, it is not close to the coherence observed in humans.\n\n>The sample size, both in terms of number of concepts and semantic categories, as well as in terms of raters sampled is limited. I agree the choice of tools and reptiles captures something interesting, but I'm more worried about showing all stimuli to only a handful of raters, which may bring in very specific biases. Could you show what the agreement between raters on a same task is, maybe in an appendix?\n\nWe computed the inter-rater reliability for the feature listing tasks and the pairwise tasks and found them to be 0.81, 0.98 respectively.\n\n\n\n\n>Why do you map feature spaces to three dimensions? Unless I'm missing something, all evaluations would work in higher dimensional spaces as well, and this dimensionality reduction may get rid of signal. Could this be justified better?\n\nThank you for prompting us to clarify our analysis decisions. When fitting triplet embeddings, we often want to fit an embedding that is compact yet expressive. A 3D embedding for our data predicted 77% of the held-out data which is comparable rates with other studies in this domain. Since we chose to conduct a Procrustes correlation analysis, all the embedding dimensions had to be the same, hence the choice of 3 (the smallest dimensionality). \nWe agree that all evaluations would work in higher dimensional spaces as well. We re-did the analysis in 30d (since the distance matrix for 30 concepts is 30*30), and the Procrustes correlations were slightly higher. However, it didn’t change any of the conclusions of the paper and we observe a similar trend as before.\n\n\n>It seems like participants did not flinch, but a screwdriver may be a drink and the translation will inevitably get rid of some polysemy while potentially introducing some which was not there. Did you handle this? If so, how? If not, can you comment on how this may impact your results? Participants saw all stimuli and may have understood polysemous terms in the intended meaning because of surrounding concepts, but LLMs may not since, if I understand correctly, you presented each prompt separately, making it impossible for the model to use preceding trials as context. \n\nThank you for this thought-provoking comment! We posit that if humans are greatly sensitive to polysemy in the concepts used, this would lead to less coherent representations across testing modalities (since, e.g., the features listed are for the tool screwdriver, but in the triplet task and similarity task participants are free to interpret whichever meaning they like). But this is not the case, and representations are quite coherent in humans. It is only in models where the polysemous information carried by a word might be negatively impacting coherence, another point of difference between humans and overly context sensitive LLMs. In terms of participants seeing all the stimuli and the models only seeing individual stimuli, we randomized the order in which stimuli were presented such that if there were any order effects, they would be mitigated when averaging across participants. \nWe agree with the point that participants have memory and can use preceding trials as context whereas LLMs cannot. Future studies can explore how these results change with in-context learning[1] by providing few-shot examples in the prompt.\n\nReferences:\n[1]​​Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., ... & Sui, Z. (2022). A survey for in-context learning. arXiv preprint arXiv:2301.00234.\n\n\n>When you sample 90% of the responses for training and 10 for testing, how was the sampling performed? fully random? stratified on something?\n\nSampling was performed randomly with uniform probability and without replacement as specified in prior work [1].\n\nReferences:\n[1] Sievert, S., Nowak, R., & Rogers, T. (2023). Efficiently Learning Relative Similarity Embeddings with Crowdsourcing. Journal of Open Source Software, 8(84), 4517.\n\n\n>You mention 75% accuracy in out-of-sample prediction: could you provide a stupid baseline for comparison to better appreciate the figure?\n\nChance in this case would be 50%, since in a given triplet one has a 50/50 chance of choosing the ‘correct’ response.\n\n\n>I'm slightly concerned but mostly interested in possible order effects in the human data for pairwise similarity ratings. Given the very closed domain and the repetition of concepts in different pairs, is it possible that the way people interpreted the Likert scale changed as they saw more pairs? to exemplify: if I see the pair boa-crocodile first, I may choose a 6, indicating they are very similar. However, if I then see the pair crocodile-caiman I may also give a 6 but if the two trials are presented in the opposite order crocodile-boa would look less similar and probably elicit a 5. Considering that all raters rated all pairs in different order, I think it would be informative to investigate whether the order played a role. I don't have strong intuitions about how this analysis should be carried out, but I'd appreciate seeing it in an appendix to better situate the results.\n\nThank you for allowing us to explain the rationale behind our experimental design in further detail. Order effects do exist and are prevalent in many psychological studies. Here, our guard against this is via randomization — that is, randomly shuffling the order in which pairs/triplets were presented to participants and averaging data across participants eliminates any reliable effect of order. As noted in a previous response, if any order effects were to ‘seep through’ in our human studies, this would actually cause us to underestimate the degree of conceptual coherence across tasks. We computed the inter-rater reliability for the feature listing tasks and the pairwise tasks and found them to be 0.81, 0.98 respectively.\n\n>How does the verification phase work? Does GPT-3 hallucinate in front of crocodiles having blades or not? did you check that the verification worked or did you trust it as such?\n\nWe describe the prompting strategy for feature verification in response to reviewer DWHV. \nThere is no ground truth to check if the process worked. If a model hallucinates and states that crocodiles have blades, we note this as the response chosen by the model. We analyzed the feature verification of Flan-XXL in prior work and developed[1] a method to increase the accuracy of feature verification [2].\n\n>I was surprised not see the work by Lenci, Sahlgren and colleagues on the analysis of LLMs' embeddings discussed in the current work, as it also tests several models on a host of semantic tasks and reports interesting patterns about the degree to which underlying embedding spaces cohere or not. https://arxiv.org/abs/2105.09825\n\nWe apologize for missing this very pertinent work and thank the reviewer for bringing it to our attention! We will plan to add this reference and a sentence summarizing the findings in our related work section in our revision if the paper were to be accepted.\n\nReferences:\n[1] Suresh, S., Mukherjee, K., & Rogers, T. T. (2023). Semantic Feature Verification in FLAN-T5. arXiv preprint arXiv:2304.05591.\n[2]Mukherjee, K., Suresh, S., & Rogers, T. T. (2023). Human-machine cooperation for semantic feature listing. arXiv preprint arXiv:2304.05012.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is sound, timely, accessible, presents a well-rounded comparison in terms of tasks and models, the way data were obtained from participants adds to making this paper interesting as it effectively taps into possible sources of variation. Results are interesting and highlight interesting differences between humans and models which should be further investigated.", "reasons_to_reject": "I have only one real possible reason why I may not fully trust the results: human participants did the triplet similarity responding to a prompt asking to decide which of two words is more similar to the target, with no further qualifiers. By my understanding, the model was prompted with the same text. Is there a chance that some models did not interpret similar as semantically similar but rather, e.g., similar from a word from point of view? would results differ if the prompt was more specific about meaning relations? I agree that if we have to change the task specification for a model to perform well whereas humans are flexible enough to fill in the ambiguity and do the task it's not really good for the models, but Turner's are skilled at taking experiments whereas LLMs may not. The same concern applies to the pairwise similarity: can it be confirmed that the models interpret the 'similar' in the prompt as pertaining to lexical semantics?\n\nThe sample size, both in terms of number of concepts and semantic categories, as well as in terms of raters sampled is limited. I agree the choice of tools and reptiles captures something interesting, but I'm more worried about showing all stimuli to only a handful of raters, which may bring in very specific biases. Could you show what the agreement between raters on a same task is, maybe in an appendix?\n\n-----------------------------------------\nI think agreement puts many concerns to rest, thanks!", "questions_to_authors": "Why do you map feature spaces to three dimensions? Unless I'm missing something, all evaluations would work in higher dimensional spaces as well, and this dimensionality reduction may get rid of signal. Could this be justified better?\n\nIt seems like participants did not flinch, but a screwdriver may be a drink and the translation will inevitably get rid of some polysemy while potentially introducing some which was not there. Did you handle this? If so, how? If not, can you comment on how this may impact your results? Participants saw all stimuli and may have understood polysemous terms in the intended meaning because of surrounding concepts, but LLMs may not since, if I understand correctly, you presented each prompt separately, making it impossible for the model to use preceding trials as context. In a similar fashion, \n\nWhen you sample 90% of the responses for training and 10 for testing, how was the sampling performed? fully random? stratified on something? You mention 75% accuracy in out-of-sample prediction: could you provide a stupid baseline for comparison to better appreciate the figure?\n\nI'm slightly concerned but mostly interested in possible order effects in the human data for pairwise similarity ratings. Given the very closed domain and the repetition of concepts in different pairs, is it possible that the way people interpreted the Likert scale changed as they saw more pairs? to exemplify: if I see the pair boa-crocodile first, I may choose a 6, indicating they are very similar. However, if I then see the pair crocodile-caiman I may also give a 6 but if the two trials are presented in the opposite order crocodile-boa would look less similar and probably elicit a 5. Considering that all raters rated all pairs in different order, I think it would be informative to investigate whether the order played a role. I don't have strong intuitions about how this analysis should be carried out, but I'd appreciate seeing it in an appendix to better situate the results. \n\nHow does the verification phase work? Does GPT-3 hallucinate in front of crocodiles having blades or not? did you check that the verification worked or did you trust it as such?\n\n\n-------------------------------------------------------------\nThanks for the thorough and insightful answers!", "paper_topic_and_main_contributions": "The paper presents an assessment of the robustness of conceptual representations in humans and LLMs. Humans and models are probed on the same three tasks (feature listing, triplet similarity and explicit similarity ratings), allowing to assess how coherent underlying conceptual spaces are. For humans, different tasks are administered to different populations in terms of culture and language (Dutch and English), adding to the credibility of the high robustness exhibited by human-derived conceptual spaces. The work also goes beyond probing LLMs by prompting and analyses underlying embedding spaces, also including word2vec and CLIP, which offers a first idea of how including visual information in the training regime affects derived representations.\n\nResults show a much stronger coherence in human conceptual spaces, where one can be used to predict another even when the population who provided the ratings differed. For LLMs, figures vary a lot: feature verification and triplet similarity performed by GPT3 offer a good correlation with human tasks (judging from Figure 5, all numbers are above .8 - but explicit similarity ratings on Likert scales do a poor job. FLAN-XXL (ver) also does a decent job at capturing human conceptual spaces, but other tasks performed by the same model do a poor job with respect to the same FLAN XXL (ver) and humans, underscoring the point made by the author that model-based spaces are often brittle and task dependent unlike humans'. Other models lag behind considerably. \n\n----------------------------------------------------------------\nI appreciate all the responses. I haven't increased my 4 on Excitement to 5 as I don't think the paper is transformative; I have raised the score on reproducibility considering the extra details provided in the rebuttals. That said, I do think this paper should feature in the conference.\n\nThe work is methodologically solid and insightful, offering a well-rounded evaluation of consistency in conceptual spaces in terms of tasks, variation in human populations, breadth scope of models probed, procedure, and presentation of results. The number of concepts and the sample size of raters are a bit concerning, as I will expand below, but all in all I think this paper should have a place in the main conference.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "I was surprised not see the work by Lenci, Sahlgren and colleagues on the analysis of LLMs' embeddings discussed in the current work, as it also tests several models on a host of semantic tasks and reports interesting patterns about the degree to which underlying embedding spaces cohere or not. \nhttps://arxiv.org/abs/2105.09825", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is very well written and clear in presenting results. I only find Figure 4 sub-optimal and somewhat hard to read at a glance, but at the same time I don't have any good advice on how to improve it. A few typos I'm sure the authors will catch upon reading the paper again (We capitalised in the middle of sentences, a couple of reduplications, nothing major).\n\nIn Figure 5 one row is duplicated (as evidenced by correlations of 1 in off-diagonal cells) and it's not fully clear what the difference between GPT-3 (002) pairwise and GPT-3 (003)-pairwise is."}}]

Author rebuttal:

Meta-review: This work compares conceptual structure in humans and language models. The research question is how consistent is the conceptual strcuture across different tasks for humans vs for models. The findings are that conceptual structure is more consistent across people than it is across models.

Overall, the reviewers unanimously agree that the work is sound and the contributions are valuable and would be exciting to the EMNLP audience. Specifically, the reviewers praise the comprehensive human experimental conditions spanning different datasets, languages, tasks, the clarity of the arguments, and the timely nature of the comparison between human behavior and large language models. Reviewers also point out a small number of opportunities for improvement, including motivating experimental choices made in the human experiments and an addition of an agreement analysis among the human annotators, which the authors responded to and reported to be high. This information should be included in a revision.
Decision: Accept-Main