Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding
Authors: Bram Van Dijk
Tom Kouwenhoven
Marco Spruit
Max Johannes van Duijn
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Critical Analysis
Interpretation
Explanation
Philosophy
Understanding
Pragmatism
NLP
TL;DR: This paper critically analyzes recurring points in critiques of LLMs, and provides a philosophical pragmatic perspective on language understanding and intentionality

Abstract: Current Large Language Models (LLMs) are unparalleled in their ability to generate grammatically correct, fluent text. LLMs are appearing rapidly, and debates on LLM capacities have taken off, but reflection is lagging behind. Thus, in this position paper, we first zoom in on the debate and critically assess three points recurring in critiques of LLM capacities: i) that LLMs only parrot statistical patterns in the training data; ii) that LLMs master formal but not functional language competence; and iii) that language learning in LLMs cannot inform human language learning. Drawing on empirical and theoretical arguments, we show that these points need more nuance. Second, we outline a pragmatic perspective on the issue of `real' understanding and intentionality in LLMs. Understanding and intentionality pertain to unobservable mental states we attribute to other humans because they have pragmatic value: they allow us to abstract away from complex underlying mechanics and predict behaviour effectively. We reflect on the circumstances under which it would make sense for humans to similarly attribute mental states to LLMs, thereby outlining a pragmatic philosophical context for LLMs as an increasingly prominent technology in society.

Large Language Models: The Need for Nuance in Current Debates and Pragmatic Perspective on Understanding
Bram van Dijk', Tom Kouwenhoven', Marco Spruit",, and Max van Duijn'
'Leiden Institute of Advanced Computer Science
Leiden University Medical Centre
(b.m (b.m.a. van. van.dij,, t. kouwenhoven, .pruit, m.j. v..va...ull duijj @liacs. leidenuniv.nn
input syntactically in hierarchical form, and they also learn semantic relations (Rogers et al., 2021), unparalleted their ability to generate gram- matically correct, fluent text. L.MS are ap- pearing rapidly, and debates on LLM capacities have taken off, but reflection is lagging behind. Thus, in this position paper, we first zoom in on the debate and critically assess three points recurring in critiques of LLM capacities: that LLMs only parrot statistical patterns in the training data; that LLMs master formal but not functional language competence; and iii) that language learning in LLMs cannot inform human language learning. Drawing on empir- ical and theoretical arguments, we show that these points need more nuance. Second, we outline pragmatic perspective the issue of 'real' understanding and intentionality in LLMs. Understanding and intentionality pertain to un- observable mental states we attribuee to other humans because they have pragmatic value: they allow to abstract away from complex underlying mechanics and predict hehaviour ef- fectively. We reflect on the circumstances under which it would make larly attribute mental states to LLMs, thereby outlining pragmatic philosophical context for LLMs an increasingly prominent technology in society.
Abstract
Current Large Language Models (LLMs) are
which are useful features in summarising, question- answering, and translating text. Examples of recent LLMs are LLaMA (Touvron et al., 2023), GPT-3 & 4 (Brown et al., 2020; OpenAI, 2023), and PaL.M (Chowdhery et al., 2022).
LLMs have sparked a lot of debate, inside and outside academia, around the question what their successes and failures say about linguistic capac- ities in Al systems, but also in humans. In the first part of this paper, we scrutinize three key points recurring in arguments by experts criti- cal of LLM capacities (e.g. Bender and Koller, 2020; Bender et al., 2021; Browning and LeCun, 2022; Marcus, 2022; Shanahan, 2022; Mitchell and Krakauer, 2022; Bisk et al., 2020; Floridi, 2023; Mahowald et al., 2023). although there are also experts who are more optimistic on this matter (e.g. Sahlgren and Carlsson, 2021; Agüera y Ar- cas, 2022a,b; Berger and Packard, 2022; Cerullo, 2022; Sejnowski, 2022: Piantadosi and Hill, 2022; Piantadosi, 2023). These points are:
for humans simi-
1. that all LLMs do is predict next words;
2. that LLMs can only master formal as opposed functional language competence;
that language learning in L.LMs cannot inform human language learning.
1 Introduction
In the first part of this paper, we aim nuance these points and show that they are hard maintain the face of empirical work on LLMs and theoretical arguments. In the second part, this leads us to de- velop a pragmatist perspective on LLMs, for which we draw on work by Daniel Dennett, Richard Rorty, and others. 'Real' language understanding and in- tentionality consist of attributions of unobservable mental states, that humans make on the basis of observable behaviour. We do because this has pragmatic value: it simplifies complex underlying biophysical processes and allows us to predict fu- ture behaviour. Instead of asking whether LLMs
The performance of Large Language Models (LLMs) has recently reached high levels (see e.g. Bommasani et al., 2021; Mahowald et al., 2023). LLMs are deep neural networks with a Transformer architecture (Vaswani et al., 2017), trained to pre- dict masked words from context, using massive text datasets. During training. LLMs leam represent
"Some LLMs additionally benefit from further fine-tuning, e.g. reinforcement learning from human feedback (Christiano et al., 2017). Since evidence is emerging that most of LLMs' capabilities are learned during pre-training (Gudibande et al., 2023; Ye et al., 2023; 7hou et al., 2023a), we abstract away from this aspect in this paper.

---

have 'real' understanding and intentionality, we ask under what cireumstances regarding LLM be- haviour and their role in society, it is reasonable for humans to make mental models of LLMs that include capacities like understanding and intention- ality.
model weights or patterns of neuronal activation. An example regarding syntax is that LLMs are able to hierarchically represent input (Hewitt and Man- ning, 2019; Manning et al., 2020; Rogers et al., 2021; Mahowald et al., 2023). That is, they are capable of internally parsing the example into syn- tactic chunks, such that the prepositional phrase ('of The Netherlands") provides information about the noun phrase ("The capital city'), which con- stitutes a clue to the answer ("Amsterdam"). In addition, regarding semanties, the vector represen- tations of words that neural networks induce are shown to be contex--ensstive and rich enough to capture conceptual relations in line with human judgements (Reif et al., 2019; Grand et al., 2022; Piantadosi and Hill, 2022). Moreover, in our exam- ple, "The Netherlands' and 'Amsterdam' are likely geometrically related in the vector space of a LLM, which provides a further clue regarding the answer.
In sum, aim is to contribute to more re- alistic framework for understanding LLMs within academia and beyond, which is better grounded in empirical and philosophical work. Since LLMs impact on research and society likely increases in the future, properly understanding them is key. Still, although we defuse various critiques of LLMs in this paper, it is not our purpose to advocate their deployment without ongoing reflection on their im- plications. Examples include their environmen- tal impact (Bender et al., 2021), biases (Lucy and Bamman, 2021), problems for educators (Sparrow, 2022), and ethical issues in adding human feed- back (Perrigo, 2023), but these issues go beyond the scope of this paper.
Our syntactic and semantic examples here are not necessarily the way LLMs represent the rele- vant linguistic information; is not trivial to extract representations from LLMs (Rogers et al., 2021). The point is rather that LLMs are capable of fur- ther representing input in various ways that are not reducible to either their training data or objec- tive (Piantadosi and Hill, 2022). While they are not explicitly trained to represent input hierarchi- cally, or represent semantic relations, such proper- ties emerge while becoming better their relatively simple stochastic training objective (Manning et al.. 2020). On second thought, this should not be too surprising, given that there is a lot of linguistic in- formation 'hidden' in the web text used to train LLMs, which LLMs (partially) reconstruct. Note that this argument does not require LLMs to 'really' understand or know their inputs representations.
2 Key points in debates on LLMs
this section, we qualify three key points from the debate regarding LLM capacities by drawing on theoretical and empirical work.
2.1 All LLMs can do is next word prediction
Shanahan (2022) claims that whenever we prompt a LLM, for example with "The capital city of the Netherlands is ...,, actually ask 'Given the sta- tistical patterns you learned from dataset Y during training, what word is most likely follow the pro- vided sequence?', where the answer is likely 'Ams- terdam'. According to this assumption, this is all there is to it; we should not speak about the model's potential topographical knowledge, nor should we say that the model understands the question in any way comparable to how humans understand it. We similar claims in Bender et al. (2021); Mar- cus (2022); Chomsky (2023); Floridi (2023), and in weaker form in Mitchell and Krakauer (2022).
The assumption that LLMs only echo sta- tistical regularities is important to qualify. So- called 'underclaiming' (downplaying what LLMs do and Iearn), resonates more broadly in the aca- demic sphere, which could hinder studying how LLMs work in detail (Bowman, 2022b), and ex- ploring whether they are useful for studying ques- tions about human language usage. The assump- tion likely stems from the idea that probabilistic modelling of language may successfully simulate or approximate linguistic facts (i.e. generate co- herent language), but does this via such a different route that it cannot provide any further insight into human language (Norvig, 2012; Piantadosi, 2023).
Although it is true that the good performance of LLMs on many tasks stems from a simple train- ing objective, which is predicting masked words from context, we argue that this point overlooks the complex ways in which LLMs are able represent information. During training, LLMs induce vari- ous semantic and syntactic features that the model uses internally to represent the input in a manner that can be extracted, for example, by analysing

---

Although it may seem on the surface that LLMs are just language simulation machines, this overlooks all the linguistic complexity that is stored in the weights that are updated during training, with word prediction providing powerful supervision signal. Beyond the potential impact of "underclaiming" in academic debates (Bowman, 2022a), this assump- tion could reinforce simplistic views of what LLMs are and why they are useful in society at large.
Mind/ToM) humans do, two vital functional capac- ities, benchmarks inevitably rely on presenting a particular (social) situation using linguistic prompts (e.g. Collins et al., 2022; Creswell et al., 2022; Sap et al., 2022; Binz and Schulz, 2023; Borji, 2023; Kosinski, 2023; Ullman, 2023; van Duijn et al., 2023). Implicitly or explicitly, such works draw on the assumption that in LLMs, there must be dis- tinction between thought as internal symbolie sys- tem representing abstract relations, and language as a mapping between these representations and their outward expression in text. Although this is not unreasonable to think, given that LLMs have many emergent capacities for which they were not explicitly trained (Section 2) currently we do not know how much formal linguistic information LLMs leverage when performing such tasks. LLM output could, for example, be the result of specific semantic or syntactic relations with the input, while seems unlikely that human would approach such tasks in the same way. Thus, in assessing thought in LLMs, language and thought are con- founded.
2.2
LLMs can master formal aspects of language, but not its function
The distinction between formal and functional lan- guage competence we draw on here stems from Ma- howald et al. (2023): formal language competence concerns employing information about linguistic rules and patterns in producing coherent output, whereas functional language competence draws on further cognitive capacities, such as formal reason- ing, intentional reasoning, and situation modelling. The authors paraphrase this difference as the dif- ference between being good at language and be- ing good at thought; in their view, LLMs master language but not thought. They motivate this dis- tinction with the finding that the two competences recruit independent brain cireuits, and discuss per- sons with aphasia as a concrete example: they can have limited formal linguistic competence, yet still be able to compose music, solve logic puzzles, rea- son about other persons' mental states, thus, lever- age thought independently of language. Whether one buys into its neural grounding or not, the dis- tinction between formal and formal language com- petence as such is useful one make, and we see similar oppositions in Bender and Koller (2020), where the distinction is made between L.Mss mas- tery of linguistic form as opposed to extra- lstra-linguistic meaning, and in Bisk et al. (2020); Browning and LeCun (2022); Floridi (2023).
This issue has an analogy in testing thought in children. When for instance testing ToM, confound- ing factors are always present, as the myriad tests of ToM that exist and the different modalities they solicit (vision, speech, text) illustrate (Quesque and Rossetti, 2020). General language and memory abilities of children are typically controlled with ad- ditional tasks (Milligan et al., 2007); few tests exist that rely on language alone (Beaudoin et al., 2020). Still, many seemingly superficial aspects shape per- formance on such tests, such as how questions are phrased (Siegal and Beattic, 1991; Beaudoin et al.. 2020). The influence of superficial linguistic arte- facts of tests can be controlled some extent when conducting ToM tests with LLMs, for example, pre- venting memorization by rewriting tests such that they are not in the training data (e.g. Shapira et al., 2023; Kosinski, 2023; van Duijn et al., 2023), but this is only the beginning of disentangling language and thought in LLM output.
2.2.1 Disentangling language and thought Here we do not claim that L.MSs have thought that can be mcaningfully separated from language, as we are agnostic on this matter, but we question some of the methods currently used to disclose thought.
Moreover, disentangling language and thought is difficult, because for many cognitive test we have an idea of what the test operationalises, but not when they are used in the context of LLMs. For the ToM context, one hallmark test is the 'unexpected contents' test (Perner et al., 1987), where a Smar- ties box with unexpected contents (c.g. a pencil) is shown to a child. The child is asked what a friend,
First of all, whereas persons with aphasia can be tested their capacity to employ thought in
way that is clearly independent of language (e.g. composing music), for LLMs this is not possi- ble. For example, for the common sense reason- ing and intentional reasoning ka. Theory of

---

unfamiliar with the box, would think its contents are, thereby asking to manage two conflicting beliefs: the false belief imputed to the friend, and its own belief about the box' contents. This conflict management, as instantiation of ToM ability, is ar- guably what the test operationalises, and something humans know from subjective experience, which makes it casier to understand what was measured when such tests are used on humans. Yet, this is far less clear when using such tests on LLMs.
sophisticated measures, and aim for more nuance in the interpretations of results, before we can deny (or affirm) thought in LLMs. Instead of focusing merely on (average) success or failure, knowing that a LLM has, for example, 51% confidence in wrong answer is already more informative than just knowing the LLM erred. Fortunately, work on more nuanced evaluations of thought in LLMs is emerging (e.g. Collins et al., 2022; Binz and Schulz, 2023). Alternatively, we could evaluate the model's intermediate reasoning steps in solving a complex reasoning task, besides only the answer, when em- ploying Cfain-of- Thought prompting (Wei et al., 2022). Evidence is emerging that in the context of ToM tests, more sophisticated prompting improves performance (Moghaddam and Honey, 2023).
2.2.2 Thought is a continuum
Here we assume, for the sake of argument, that we can separate thought from language ability in LLMs with cognitive tests (which questioned in Section 2.2.1). We further reflect how various tests are currently being used to deny or affirm thought in LLMs. Again, we do not argue that LLMs have or lack thought here, but rather that we should suspend conclusions, based on the issues raised below.
From a methodological perspective, testing thought in LLMs and humans differs a lot because the entities at issue differ a lot. Yet, we can improve the comparability of testing, A single output of a LLM single test item likely does not yicld good estimate of its capacities; in testing humans, we typically ask multiple humans do the same item. Thus, we could for example initialize LLMs with slightly higher temperature values multiple times on the same test item, to get a fuller view on what LLMs can, and to obtain a larger sample of responses which statistical tests are possible. Although we know that low temperatures settings make models deterministic, not much evaluation of slightly higher temperature settings in relation to performance has been done, with the exception of Moghaddam and Honey (2023). addition, we know from work on knowledge extraction LLMs, that paraphrasing a particular input improves model performance in retrieving knowledge and relations (Jiang et al., 2020). Paraphrasing test items is not only way to increase model performance, but could also provide way increase our confidence in our estimates of thought capacity as indicated by LLM performance on multiple paraphrased items.
Quite some cognitive tests currently employed with LLMs make assumptions about thought that need qualification. Many are designed that a LLM can only fail or succeed on them (e.g. (part of) tests employed by Sap et al., 2022; Borji, 2023; Bubeck et al., 2023; Kosinski, 2023; Shapira et al., 2023; Ullman, 2023). Yet. thought capac- ity is better understood as a continuum (Beaudoin et al., 2020; Sahlgren and Carlsson, 2021). That is, thought capacity is unequally distributed in hu- mans; people who excel in logic, may still be horri- ble composers, or struggle to recognise other per- sons' intentional states.
In addition, we are typically much more lenient towards failure in exercising thought. In daily con- texts, humans are generally susceptible to a host of misplaced heuristics and formal errors (Haselton et al., 2015; Dasgupta et al., 2022), but we gen- erally do not conclude from this that humans do not have unique thought capacity. Sahlgren and Carlsson 12021) make similar claim for language understanding: different language users are good at different things, at different times, in different situ- ations. Thus, it is unsurprising that disagreement exists among NLP-scholars about proper opera- tionalisations of various tasks in Natural Language Understanding, such as Natural Language Infer- ence (Subramonian et al., 2023), disagreement that can also be anticipated for other cognitive tests.
Lastly, from more general perspective, failure of LLM a cognitive test does not imply that the system does not have the mappings required to do the test; the tests could also be less suited retrieve them (Bommasani et al., 2021).
2.3 Language acquisition in LLMs cannot inform human language acquisition
To make our evaluations of thought in LLMs as compelling as possible, we can employ more
Bisk et al. (2020) argue that, since children can- not acquire a language by merely listening to the

---

radio, it is likewise wrong to expect that LLMs can acquire language by purely ingesting text from the internet; similar claims are offered Bender and Koller (2020); Chomsky (2023). This point relies on the presumed poverty of "extralinguistic" information in the train data of LLMs. In language acquisition, children draw not only on linguistic imput but also sense perception (e.g. seeing and touching the world), motor experience (e.g. mov- ing objects), and interaction with caretakers (e.g. feedback). Also, children receive far less language input compared to LLMs (Warstadt and Bowman, 2022). Thus, if language acquisition in children and LLMs is so different from the start, it seems language acquisition in LLMs cannot inform lan- guage acquisition in humans.
distributional information in text. This illustrates another key role LLMs can play in language ac- quisition: as Warstadt and Bowman (2022) note, LLMs in ablation studies can provide 'proof of concept' whether target linguistic knowledge (e.g. verb-subject agreement in triply embedded clauses) is learnable in an ablated environment (e.g. with- out triply embedded clauses in the training data). Such studies are helpful in identifying sufficient conditions for obtaining specific linguistic knowl- edge in language acquisition. Thus, the examples mentioned above echo the broader point made by various scholars that we should see LLMs as dis- tributional learners that show what linguistic phe- nomena are in principle learnable from statistical information in text (Contreras Kallens et al., 2023; Evanson et al., 2023; Wilcox et al., 2023).
2.3.1 LLMs as useful distributional models
Warstadt and Bowman (2022) note that in lan- guage acquisition contexts, LLMs need to be less advantaged to humans in one key aspect: the amount of training data. That is, they need to be made more ecologically valid. The latter is indeed important, and fortunately, work is emerging which shows that it is possible to train LLMs with more realistic amounts of data, that at the same time per- form equally well in predicting human neural and behavioural patterns as models trained with large datasets (e.g. Hosseini et al., 2022: Wilcox et al.. 2023). Yet, it is equally remarkable that LLMs have become so successful, despite being very disadvan- taged as well (no multimodal input, no feedback in learning, no sensorimotor input). We argue that it is not obvious how should weigh such disadvan- tages and advantages in LLMs' language learning. LLMs make wrong assumptions about language acquisition in key respects, but all scientific models do this (Box, 1979; Buayen, 2008), while this does not render such models useless: they can provide lower bound on what linguistic phenomena are learnable in principle from distributional informa- tion.
LLMs and children evidently differ a lot. Yet, as Sahlgren and Carlsson (2021) formulate, LLMs are theoretically and practically our current 'best bet' for machines acquire language understand- ing, given the empirical work documenting LLM proficiency in many language tasks (see e.g. Bom- masani al., 2021; Wei al., 2023). Like any sci- entific model they are wrong in some respects, but they seem our current best distributional models to study specific aspects of language acquisition.
For example, Chang and Bergen (2022) use BERT and GPT-2 as distributional agents that ex- clusively learn from word co-occurrence statistics. They employ a.o. word frequency, lexical class and word length, as known predictors of word acqui- sition in children, and predict word acquisition in LLMs and children to gauge the extent to which these known effects in children can be accounted for by statistical learning mechanisms. Chang and Bergen (2022) show that language acquisition in LLMs and children differs in key respects (LLMs are more frequency-driven), but are also similar (learning in both takes longer for words embedded in longer utterances). As the authors note, distri- butional models can be used in similar fashion to explore the extent to which acquisition of seman- tics or syntax in children can be accounted for by statistical learning.
2.3.2 How poor is training data?
Here we discuss the assumption mentioned in Sec- tion 2.3 that data used to train LLMs lacks ex- tralinguistic information required in language ac- quisition, by considering what extralinguistic in- formation LLMs learn to represent during training. Since humans use language to do variety of things (Sahlgren and Carlsson, 2021), 1), such as providing explanations, describing all sorts of objects and processes, entertaining and convincing others, it is
Cevoli et al. (2023) provide another example by unravelling lexical ambiguity with BERT. The authors show that psychological theories positing complex mechanisms for representing ambiguity, are not necessary to explain how such representa- tions are acquired, since they can be decoded from

---

natural to assume that LLMs are able to recover some of the knowledge about c.g. properties of objects in the world, communicative intents, and users' mental states. Recent work shows that LLMs are able to represent conceptual schemes for worlds (c.g. for direction) they have never observed (Pa- tel and Pavlick, 2022). thus it seems that LLMs have sufficiently rich conceptual structure de- code at least some of the extralinguistic information present text, as surrogate grounding. Similarly, Abdou al. (2021) show that internal represen- tations LLMs show a topology of colours that correspond to human perceptual topology. In addi- tion, evidence emerges that LLMs are able to repre- sent communicative intents behind texts (Andreas, 2022), and the ways LLMs represent semantic fea- tures of various object concepts aligns with humans (Hansen and Hebart, 2022). Moreover, studies in which LLMs are trained and tested on synthetic tasks, provide an even stricter scenario for testing whether LLMs are able to decode emergent prop- erties from simple input. LLMs trained on simple input such as lists of player moves in board game, prove able to recover emergent properties such as game rules, valid future moves, and board states (Li al., 2022). For additional examples of ex- tralinguistic grounding, see Bowman (2023).
4. Given the increasing prominence of LLMs, interacting with them in terms of mental state attri- bution will likely become more common, yet lacks a comprehensive theory:
5. This practice is fully explainable from a prag- matist perspective, although in different commu- nities, such as the scientific community, different pragmatist values may play a role, that makes this practice less acceptable for this community.
3.1 Invoking 'real' understanding
Various scholars have claimed that LLMs are inca- pable of 'really' understanding language and using intentionality like humans (e.g. Bender and Koller, 2020; Bishop, 2021; Browning and LeCun, 2022; Floridi, 2023; Mahowald et al., 2023). Indeed, it seems that the critiques of LLMs as autocomplete systems that do not know how language functions in the world, or as language learners that cannot learn by drawing on such functions, implicitly in- voke this claim. Still, in such critical works it is sel- dom made explicit what 'real' understanding or in- tentionality amounts to. These works often revolve around John Searle's 'Chinese Room' thought ex- periment (Searle, 1980). We illustrate this with the following quote from Bender and Koller (2020):
3 A pragmatic philosophy of LLMs
"This means we must be extra careful devising evaluations for machine un- derstanding, as Searle (1980) elaborates with his Chinese Room experiment: he develops the metaphor of a "system" in which a person who does not speak Chi- nese answers Chinese questions by con- sulting library of Chinese books accord- ing predefined rules. From the outside, the system seems like it "understands" Chinese, although in reality no actual un- derstanding happens anywhere inside the system." " (p. 5188)
This section sketches a more general, pragmatist philosophical context for LLMs. Although LLMs are prominent academia and society, philosophi- cal reflection is lagging behind. This is lamentable, given that LLMs and the way they are deployed raise pressing philosophical questions. Here we develop pragmatist view on LLMs with the fol- lowing claims that we will motivated with reference to philosophical pragmatism:
1. All three key points from the debate about capacities of LLMs discussed above ultimately re- volve around the issue of 'real' understanding and intentionality, but fail to address what that means;
Once we try to explain what 'real' under- standing and intentionality are, we find that these (and mental states more generally) are not acces- sible in others we interact with, irrespective of whether they are humans or other kinds of systems; 3. Attributing mental states others has foremost pragmatic value, in that they help us to abstract underlying complexity away, predict be- haviour, and obtain goals the world;
The argument presented in the quote is that, for any system, being able to deliver the expected out- put on a range of inputs is insufficient for having 'actual understanding', where it is important to note that the perspective of anyone interacting with the system is 'from the outside'. The point of this thought experiment that, although the idea of 'real' understanding is implied, it is not explained, which makes the argument incomplete.

---

3.2 Explaining 'real' understanding
the behaviour intelligible and allows us predict further behaviour, e.g. waving the bus driver, Similarly, attributing the set of mental states that constitutes 'real' language understanding to other humans, makes their behaviour intelligible, and smooths our social interactions. This pragmatic
The Chinese room argument appeals situation where we would grant that the system understands Chinese: if the human in the system understands Chinese. That is, this human would need to have a set of mental states involving knowledge, be- liefs, and intentions, such that in producing out- put, the human does not draw on predefined rules, but rather on its knowledge of Chinese, beliefs about the desired output, and further communica- tive intent. This would constitute an example of what is meant by human having 'real' understand- ing. Nonetheless, this explanation cannot save the thought experiment as presented above, since makes no difference for anyone interacting with the system if we would replace the rulc-abiding human with the human as full mental agent, since from the outside, there's only the system's 's behaviour to observe, which does not change.
perspective is closely related to the idea that men- tal states are key concepts for humans that have strong social justification (Rorty, 2009); they help community to achieve its goals in the world, and that is all the justification we need to use them. Ex- actly because attributing mental states to others has such clear pragmatic value, we have sufficient rea- son to take them seriously. From this perspective, it is counterproductive to adopt behaviourist (i.e. denying their importance or existence) or essential- ist (i.e. accepting them only if there is evidence that they are 'real') attitude towards mental states.
3.4 Pragmatic value and LLMs
This distinction between what observable, c.g. behaviour, and what is inaccessible or unobserv- able, e.g. mental states, is a distinction known in the philosophy of science (see e.g. Van Fraassen, 1980; Churchland, 1985; Fodor, 1987), but it is also at work in the empirical domain. For example, Rabinowitz et al. (2018) note with reference to Dennett (1991), we make mental models of others' internal states that are inaccessible from the out- side, and that make 'little to no reference' to the underlying mechanisms of the agent that produces the observed behaviour. Our point here is that we are always confined to observable behaviours of other agents, regardless of whether they are hu- mans machines. Whenever we claim that 'real' understanding and intentionality is lacking in some other agent, we make a claim about states that are in principle inaccessible from the outside.
We can make a similar claim for LLMs, even though humans and LLMs differ. With regard to ob- servable behaviour, humans can deploy more subtle and multimodal observable behaviours compared to LLMs, like tone of voice, facial expressions, gestures, even unconsciously. So the observable behaviour that underlies our mental models of hu- mans is arguably much richer, which gives us more details to work with when attributing mental states to others. At the same time, we should acknowl- edge that the way we interact with LLMs is strik- ingly different from the way we interacted with artificially intelligent systems before. Their lan- guage output is grammatically correct, fluent, and critically, increasingly well adapted context, user, and input. This is starting to challenge assumptions about what to be human and what it is to be a machine, and what it is to communicate as a hu- man with an intelligent system that communicates in many ways like human would do (Guzman and Lewis, 2020). The increasing sophistication of in- teraction has led to humans viewing such systems as distinct, social entities, and consequence, humans triggered even more attribute mental states to such systems (see e.g. Guzman and Lewis, 2020; Stuart and Kneer, 2021).
3.3 Pragmatic value
We are nevertheless fully entitled to make 'mental models' of other humans, that is, attribute to them believes, desires, and intentions, because this is useful in everyday interaction: it has clear prag- matic value. This point is perhaps best known in the form worked out by Daniel Dennett as the 'in- tentional stance': by attributing mental states to other humans, we abstract away from their under- lying biophysical complexity, while still having a ground for anticipating future behaviour (Dennett, 1989). If we see a person running towards a bus stop, attributing the desire to catch the bus makes
In the context of LLMs, attributing mental states to LLMs has often been addressed as oversensitive anthropomorphisation, with our mental models be- ing illusions 'in the eye of the beholder' (Bender et al., 2021). Such critiques overlook that making

---

mental models of intelligent systems can have clear pragmatic value, in that they abstract away from the underlying complexities of LLMs, and at the same time help us to predict and explain their behaviour, and achieve goals in the world. Obviously there are complex systems for which making mental mod- els make less sense, e.g. for a Mars rover, where our goal of landing it on Mars is better served by physical models. On the other hand, our interaction with LLMs as complex systems, as we show with examples below, is often best served by attribut- ing mental states to them 'as if" they were socially intelligent in the way we think other humans are.
Given the increasing prominence of LLMs in socicty, we can expect that making mental models of LLMs will become more common.2 We can already see some examples where LLMs are specif- ically used to impersonate individuals, for example, helpdesk service agents (Brynjolfsson et al., 2023), influencers (Lorenz, 2023), deceased beloved ones (Pearcy, 2023), virtual friends (Marr, 2023), and personal assistants (Chen, 2023). These may strike one as rather worrying examples, but such devel- opments could have pragmatic value for humans in that LLMs can give them a sense of relationship or consolation. This is not to say that we advocate such deployment of LLMs, as they have many un- addressed ethical implications, but rather that there are conditions imaginable, in which mental state attribution to LLMs is explainable, justified, and has pragmatic value. Here we rather want stress that the larger role LLMs (in whatever future form) will likely play in society, demands a theory of our interactions with them that does not simplify our behaviour to 'anthromorphisation..
Our mental model about what an LLM 'knows' or 'wants', can allow us, among other things, to communicate our requests succinctly ('Do you know how to do X' in prompting), explain errors ("The system confuses X for Y"), formulate next step in interaction ('The system now expects input X'), or gauge reliability of output ('How strong is your belief that X?') And this need not apply only to our own interaction with LLMs, but is also relevant for explaining LLM behaviour to other humans. LLMs optimized for dialogue, (e.g. Chat- GPT, PaLM2-chat), increasingly enable this form of interaction that involves mental state language.
3.5 Pragmatic value and science
We want to emphasise that pragmatic values can be different in different communities, since they may have different goals in the world. In scien- tific community that attempts describe/explain LLMs and their purported cognitive capacities in more detail than is typically required in daily life, researchers may balk at attributing mental states to LLMs. Yet, they should not do so because LLMs do not have any 'real' understanding and intention- ality, as we saw that this claim misses the point. Mental states are not intended literal accounts of the underlying complexity of humans or machines, and the subjective experience associated with 'real' understanding is not something we can access from the outside, and therefore deny outright in other entities.
This development should not surprise us, given that language is a tool that has evolved for commu- nicating and manipulating mental states to achieve goals in the world (Clark, 1996; Tomasello, 2003), for example resolving conflict and working to- gether. Similarly, in child development, language competence and the ability to reason about mental states strongly overlap (for an overview see Milli- gan et al., 2007). Furthermore, scholars argue that children in learning word meanings (for example for verbs of perception like 'to look') do not just learn abstract sign-object mappings (that interlocu- tor X literally perceives object Y). but foremost their pragmatic effects, which is for children typi- cally directing an interlocutor's attention to various concrete objects (Enfield, 2023; San Roque and Schieffelin, 2019), and such forms of joint atten- tion are precursor to ToM (Tomasello et al., 1995). In similar vein, current research that focuses on ways to have LLMs 'reflect' on their 'confidence' in their assertions, or on uncertainty in their input, can be understood in terms of the pragmatic value this has for L.M users, that in the normal world also deal with uncertainty in information (Zhou et al., 2023b; Kadavath et al., 2022).
A better reason, grounded in the pragmatist per- spective we offer here, seems to be that mental models made in everyday interaction may allow us to explain and predict behaviour, but lack other pragmatic values critical in the scientific commu- nity. If mental states are to play role in a scientific description or explanation of LLMs, then they must, for example, also cohere with other currently ac- cepted theories/models; offer an elegant or simple
Note that our account is not intended to be normative, in that we are not claiming that humans should make mental models of LLMs

---

explanation, have large scope, etc. (Van Fraassen, 1980). Such values are pragmatic, because they do not primarily depend on the relation a theory has with the observable world; there is, for example, no reason to think that the world must be elegant or simple because our theories are, or that phenomena in the world cohere because our theories cohere.
(Ma et al., 2023).
A related point is that, although pragmatic count of mental states in humans abstracts away from their complex underlying biophysical corre- lates, pragmatism does not entail that there no point in trying to disclose such correlates scien- tifically, with the aim of opening the black box. A biophysical account of mental states may have pragmatic values for community of scientists (see Section 3.5), and also broader pragmatic value for society in that it can help to, for example, treat dysfunctional mental states better. This biophysical account resides at a different level of explanation, and does not necessarily conflict with pragmatic accounts of the mental in general. In the case of LLMs, the pragmatist has similarly no principal issues with trying to find out what patterns of (artifi- cial) neuronal activation are correlated with mental state content in LLM input and output.
The upshot is that attributing mental states to LLMs may not cohere well with empirical work on mental states other fields that map them to patterns of neuronal activity in the brain, for which neurons in LLMs currently constitute at best only a loose analogy. Or it may not cohere with more theo- retical work that holds that the possibility of mental states in machines entails category mistake (as introduced by Ryle (1950)), as mental states are properties of beings such as humans, which fun- damentally differ from machines. By considering pragmatic values at play in different communities, we are able to explain why, on a general level, at- tributing mental models to LLMs can be explain- able and justifiable. but at the same time could be less acceptable in the scientific community that has different pragmatic values.
5 Conclusion
The goal of this paper was to provide further reflec- tion LLMs in two ways. First, scrutinised three key points surfacing in recurring critiques LLMs, and found that on empirical and theoretical grounds, these points need more nuance. Our con- clusions are that LLMs are more than exploiters of statistical patterns; that we need better measures for evaluating thought competence in LLMs before we can draw conclusions; and that LLMs have a role to play in language acquisition, as our current best distributional models.
4 Discussion
Although in Section 3 we discuss LLM capacities and mental states mostly at fundamental level, arguments also relevant for engineers work- ing on concrete systems that employ LLMs. Such systems will always require reflection on under- standing and mental states in humans and machines, which our pragmatic outlook can inform. Our argu- ments are agnostic about the explicit taxonomies and frameworks of the mental, which engineers may develop and employ in such systems, as it is the system's 's behaviour that the pragmatist is typ- ically most interested in, and it can be realised in various ways. In the design of LLMs, no such tax- onomies or frameworks exist (Kosinski, 2023; Trott et al., 2023), but possible that systems that do have them manifest equally complex behaviour. In a similar vein, we can imagine training scenarios that include visual (or other multi modtl) input proxy for grounding denotations of words in the world, which would also make LLM behaviour more sophisticated, disambiguating input is ar- guably simpler with an additional information chan- nel. Evidence is emerging that enriching LLMSs with vision modules as surrogate grounding allows such models to learn new words more efficiently
Second, we provided philosophical context for LLMs from pragmatist perspective. An un- resolved question underlying various critiques of LLMs, is whether they have something like 'real' language understanding and intentionality. We ar- gued that whether we attribute unobservable mental states to other entities, including the set that would constitute 'real' language understanding and inten- tionality, depends on how much pragmatic value this has to us, not whether mental states are actual properties of the entities at issue.
LLMs (in whatever future form) will become more prominent in the years to come. We hope to have contributed to a better understanding of what LLMs can(not) do, as well as to philosoph- ically informed understanding our interaction with LLMs that is more than a story of mere an- thropomorphisation.

---

Limitations
References
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. 2021. Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 109-132, Online. Association for Computational Lin- guistics.
In this paper, we addressed LLMs as the set of large Transformer-based neural networks that are trained with cloze tasks, using large text datasets. Still, there is some variation in this set, LLMs can have different sizes, different architectures, train- ing datasets, methods for further fine-tuning, and so on. Up this point, is typically the case that larger LLMs trained with more data obtain the best performance on a variety of tasks, which also makes that such larger LLMs are overrepresented in evaluations of general LLM capacities. Ope- nAI's flagship models like GPT-3 and ChatGPT are LLMs that frequently recur in tests (although the work of c.g. Shapira et al. (2023) is exception).
Blaise Agtiea Arcas 2022a. Artificial neural net- works are making strides towards consciousness. Ac- cessed on: 2023-01-28.
Blaise Agüera y Arcas. 2022b. Do Large Language Models Understand Us? Daedalus, 151(2): 183-177.
Jacob Andreas. 2022. Language models as agent mod- els. In Findings of the Association for Compulational Linguisties: EMNLP 2022, pages 5769-5779, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
In addition, new models are appearing at a fast pace, such as LLaMA (Touvron et al., 2023), Fal- con (Penedo et al., 2023), and P2LM2 (Anil et al., 2023). It remains to be seen how these new models fare on various tests, such as those for cognition, but they are fairly similar regarding their neural network architecture, training data, and training ob- jective. the same time, signs are emerging that OpenAI's flagship models may be slowly deterio- rating with respect to their performance on writing code and doing basic math (Chen et al., 2023).
Rohan Anil, Andrew M Dai. Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey. Zhifeng Chen, al. 2023, Palm 2 technical report. arXiv preprint arXiv:2355.10403.
H. Baayen. 2008 Analyzing linguistic data: A prac- rical intrdduction to statistics using R. Cambridge University Press.
All these developments together challenge the idea that there something like 'the' LLM, which is a simplification we made in this paper that is not doing complete justice to the large zoo of LLMs that currently exists. In addition, the continuing updates they are undergoing to make them derail less quickly, safer, less bias-driven, more efficient, and so on, also imply that they are moving target in many discussions. These fast developments may also limit the import of the arguments into the more distant future, as is hard to foresce for example developments in different neural architcctures and training regimes.
Cindy Beaudoin, Élizabel Leblanc. Charlotte Gagner, and Miriam H. Beauchamp. 2020. Systematic Re- view and Inventory of Theory of Mind Measures for Young Children. Frontiers in Psychology, 10.
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Mod- els Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, pages
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5185-5198, Online. Association for Computational Linguistics.
Acknowledgements
This research was not possible without collab- oration with dr. Max van Duijn's research project 'A Telling Story' (with project number U..eei.1911..555. which is financed by the Dutch Research Council (NWO). We thank Li Kloostra for helpful comments on earlier versions of this paper. Lastly, the authors thank three anonymous reviewers for their constructive feedback.
Jonah Berger and Grant Packard. 2022. Using natural language processing to understand people and culure. American Psychologist, 77(4):525.
Marcel Binz and Eric Schulz. 2023. Using cognitive psychology understand GPT-3. Proceedings of the National Academy of Sciences, 12066)x22185231200
Mark J. Bishop. 2021. Artificial intelligence is stupid and causal reasoning will not fix it. Frontiers Psychology, 11:6003.

---

Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nis- nevich, Nicolas Pinto, and Joseph Turian. 2020. Ex- perience Grounds Language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718 18-8735, Online. Association for Computational Linguistics.
Samuel R. Bowman. 2023. Eight things know about large language models. arXiv preprint 2rXiv:2304.00612.
George E. Box. 1979, All models are wrong, but some are useful. Robustmess in Statistics, 209(1979)549.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quiney Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard. Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Kr- ishna, Rohith Kuditipudi, Ananya Kumar, Faisal Lad- hak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchan- dani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Pa- padimitriou, Joon Sung Park, Chris Picch, Eva Porte- lance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the Opportuni- ties and Risks of Foundation Models. arXiv preprint arXiv:2108.07255.
Jacob Browning and Yann LeCun. 2022. AT and The Limits Of Language. Accessed on: 2023-01-28.
Erik Brynjolfsson. Danielle Li, and Lindsey R Raymond. 2023. Generative AT at work. Technical report, Na- tional Bureau of Economic Research.
Sébastien Bubeck, Varun Chandrasckaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint w:Xiv:2303.1271..
Michael Cerullo. 2022. In Defense of Blake Lemoine and the Possibility of Machine Sentience in Lamda.
Benedetta Cevoli, Chris Watkins, Yang Gao, and Kath- leen Rastle. 2023. Shades of meaning: Uncover- ing the geometry of ambiguous word representa- tions through contextualised language models. arXiv preprint arXiv:2304. 13597.
Tyler A. Chang and Benjamin K. Bergen. 2022. Word acquisition in neural language models. Transactions of the Association for Computational Linguistics. 10:1-16.
Ali Borji. 2023. A Categorical Archive of ChatGPT Failures. arXiv preprint arXiv:2302.03::
Brian X. Chen. 2023. How ChatGPT and Bard Per- formed as My Executive Assistants. Accessed on: 2023-10-15.
Samuel Bowman. 2022a. The dangers of underclaim- ing: Reasons for caution when reporting how NLP systems fail. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7484-7499, Dublin, Ireland. Association for Computational I. Linguisties.
Lingjao Chen, Matei Zaharia, and James Zou, 2023, How is ChatGPT's behavior changing over time? arXiv preprint arXiv:2307.099: .00009.
Noam Chomsky. 2023. The False Promise of Chat-GPT. Accessed on: 2023-01-30.
Samuel R. Bowman. 2022b. The Dangers of Under- claiming: Reasons for Caution When Reporting How NLP Systems Fail. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 7484-7499. Dublin, Ireland. Association for Computational Lin- guistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint orXiv:2204.02311.

---

Paul F. Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Sys- lems, volume 30. Curran Associates, Inc.
Gabriel Grand, Idan Asher Blank, Francisco Pereira, and Evelina Fedorenko. 2022. Semantic projection recovers rich human knowledge of multiple object features from word embeddings. Nature human be- haviour, 6(7):975-997.
Paul M. Churchland. 1985. The ontological status of observables: in praise of the superempirical virtues. Images of science, pages 35-47.
Arnay Gudibandee Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprictary llms.
Herbert H. Clark. 1996. Using Language. Cambridge University Press.
Andrea L. Guzman and Seth C. Lewis. 2020. Artificial intelligence and communication: A human- machine communication research agenda. New Media & Soci- ery, 22(1):00-86
Katherine M Collins, Catherine Wong. Jiahai Feng, Megan Wei, and Josh Tenenbaum. 2022. Structured, flexible, and robust: benchmarking and improving large language models towards more human-like be- havior in out-of-distribution reasoning tasks. In Pro- ceedings of the Annual Meeting of the Cognitive Sci- ence Society, volume 44.
Hannes Hansen and Martin N Hebart. 2022. Seman- tic features of object concepts generated with gpt-3, arXiv preprint arXi::2202.037.
Martie G. Haselton, Daniel Nettle, and Paul Andrrssss 2015. The evolution of cognitive bias. The handbook of evolutionary psychology, pages 724-746.
Pablo Contreras Kallens, Ross Deans Kristensen- McLachlan, and Morten H. Christiansen. 2023. Large language models demonstrate the potential of statistical learning in language. Cognitive Science, 13355..
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4129-4 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Antonia Creswell, Murray Shanahan, and Trina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.0977 027122
Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2022. Lan- guage models show human-like content effects on reasoning.
Fighbal A. Hosseini, Martin Schrimpf, Yian Zhang, Samuel Bowman, Noga Zaslavsky, and Evelina Fe- dorenko. 2022. Artificial neural network language models align neurally and behaviorally with humans even after a developinentally realistic amount of train- ing. bioRxiv.
Daniel Dennett. 1989. The Intentional Stance. MIT Press.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, 8:423-438.
Daniel C. Dennett. 1991. Two Contrasts: Folk Craft vs. Folk Science and Belief vs. Opinion. In John D. Greenwood, editor, The Future of Folk Psychology: Intentionality and Cognitive Science, pages 135 48.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson. Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage. Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort. Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Ka- mal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language Models (Mostly) Know What They Know. arXiv preprint arXivv2207.05221.
Nick J. Enfield. 2023. Linguistic concepts are self- generating choice architectures. Philosophical Trans- actions of the Royal Society B, 378(1870):20210332
Linnea Evanson, Yair Lakretz, and Jean Rémi King. 2023. Language acquisition: do children and lan- guage models follow similar learning stages? In Findings of the Association for Computational Lin- guistics: ACI. 2023, pages 12205-12218, Toronto, Canada. Association for Computational Linguisties.
Luciano Floridi. 2023. AI as Agency without Intel- ligence: On ChatGPT, large language models, and other generative models. Philosophy & Technology. 36(1):15.
Michal Kosinski. 2023. Theory of mind may have spon- taneously emerged in large language models. arXiv preprint arXiv:2302.02083
Jerry A. Fodor. 1987. Psychosemantics: The problem of meaning in the philosophy of mind, volume MIT press.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2022. Emergent world representations: Exploring a

---

sequence model trained on synthetic task. arXiv preprint urXiv:2210.33366
Aimee Pearcy, 2023. How ChatGPT and Bard Per- formed as My Executive Assistants. Accessed on:
Taylor Lorenz. 2023. An influencer's s Al clone will be your girlfriend for $1 a minute. Accessed on: 2023- 05-30.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei. and Julien Launay. 2023. The RefinedWeb dataset for Falcon L.M:: outperforming curated corpora with web data, and web data only, arXiv preprint arXiv:2306.01116.
Lucy and David Bamman. 2021. Gender and rep- resentation bias in GPT-3 generated stories. In Pro- ceedings of the Third Workshop Narrative Un- derstanding, pages 18,55, Virtual, Association for Computational Linguistics.
Josef Perner, Susan R Leekam, and Heinz Wimmer, 1987. sirreeycerr-ods"" difficulty with false belief: The case for conceptual deficit, British journal of developmental psychology, 5(2):125-137.
Ziqiao Ma, Jiayi Pan, and Joyce Chai. 2023. World- to-words: Grounded open vocabulary acquisition through fast mapping in vision-language models. In
Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Papers), pages 524-544, Toronto, Canada. As- sociation for Computational Linguistics.
Billy Perrigo. 2023. Exclusive: OpenAl Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic. Accessed on: 2023-01-25.
Kyle Mahowald, Anna A Ivanova, Idan Blank, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fe- dorenko, 2023. Dissociating language and thought in large language models: cognitive perspective. arXiv preprint arXiv:2301.064m
Steven Piantadosi and Felix Hill. 2022. Meaning with- out reference in large language models. In NeurIPS 2022 Workshop Neuro Causal and Symbolic Al (nCSI).
Steven T. Piantadosi. 2023. Modern language mod- refute chomsky's approach language. Ling- bucz/001180.
Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, Omer Levy. 2020. Emer- gent linguistic structure in artificial neural networks trained by self- supervisoo.. Proceedings of the Na- tional Academy of Sciences, 317(46):300463333 30054..
François Quesque and Yves Rossetti. 2020. What do theory-of-mind tasks actually measure? Theory and practice. Perspectives Psychological Science, 15(2):384-396.
Gary F. Marcus. 2022. Nonsense Stilts. Accessed on: 2023-01-30.
Neil Rabinowitz, Frank Perbet, Francis Song. Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. 2018. Machine theory of mind. In Proceedings of the 35th International Conference on Machine Learn- ing. volume 80 of Proceedings of Machine Learning Research, pages 4218 4188-427..2 PMLR.
Bernard Marr. 2023. Artificial Intimacy: How Gen- crative AI Can Now Create Your Dream Girlfriend. Accessed on: 2023-10-10.
Karen Milligan, Janet Wilde Astington, and Lisa Ain Dack. 2007. Language and Theory of Mind: Meta- Analysis of the Relation Between Language Ability and False-Belicf Understanding. Child Development, 78(2):622 646.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, Been Kim. 2019. Visualizing and measuring the geometry of bert. Advances in Neural Information Processing Systems, 32.
Melanie Mitchell and David C Krakauer. 2022. The Debate Over Understanding in Al's Large Language Models. arXiv preprini arXiv:2210.1 1396.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2021. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Asso- ciation Computational Linguisties, 8:842 866.
Shima Rahimi Moghaddam and Christopher Honey. 2023. Boosting theory-of-nen -ind performance in large language models via prompting.
Peter Norvig. 2012. Colorless green ideas leam furi- ously: Chomsky and the two cultures of statistical learning. Significance, (4):300.3.
Richard Rorty. 2009. Philosophy and the Mirror of Nature. Princeton University Press.
Gilbert Ryle. 1950. "The Concept of Mind". The Uni- versity of Chicago Press.
OpenAL 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774.
Magnus Sahlgren and Fredrik Carlsson. 2021. The Sin- gleton Fallacy: Why Current Critiques of Language Models Miss the Point. Frontiers in Artificial Intelli- gence, 4:682578.
Roma Patel and Ellie Pavlick. 2022. Mapping Language Models Grounded Conceptual Spaces. In Interna- tional Conference on Learning Representations.

---

Lila San Roque and Bambi B Schieffelin. 2019, Per- ception verbs in context: Perspectives from Kaluli (Bosavi) child-caregiver interaction. Laura Speed, C. O'eara,, Lila San Roque, and Asifa Majid (eds.) Perception Metaphors, pages 347-368.
Sean Trott, Cameron Jones, Tyler Chang, James Michaelov, and Benjamin Bergen. 2023. Do Large Language Models know what humans know? Cogni- rive Science, x7(7)ee1330.
Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399
Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022. Neural tfeory-of-minee social intelligence in large LMs. In Proceedings of
the limits of
the 2022 Conference on Empirical Methods Nat- ural Language Processing, pages 3762-3780, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Max van Duijn, Bram M.A. van Dijk, Tom Kouwenhoven, Werner de Valk, Marco Spruit, and Peter van der Putten. 2023. Theory of Mind in Large Language Models: Examining Performance of 11 Sfate-of-the-Ar models vs. Children Aged 7-10 on Advanced Tests. Proceedings of 27th Confer- ence Computational Natural Language Learningn (CoNNL).
John R. Searle. 1980. Minds, brains, and programs. Behavioral and brain sciences, 3(3):417-424.
Terrence Sejnowski. 2022. Large language mod- els and the reverse turing test. arXiv preprint vnXiv:2207.14388
Bas Van Fraassen. 1980. The scientific image. Ox-
ford University Press.
Murray Shanahan. 2022. Talking About Large Lan- guage Models. arXiv preprint arXiv:2212.03551
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention Is All You Need. Advances in neural information process- ing systems, 30.
Natalie Shapira. Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap. and Vered Shwartz. 2023. Clever Hans or Neu- ral Theory of Mind? Stress Testing Social Reasoning in Large Language Models.
Alex Warstadt and Samuel R Bowman. 2022. What Artificial Neural Networks Can Tell Us About Human Language Acquisition. In Algebraic Structures in Natural Language, pages 17-60. CRC Press.
Michael Siegal and Karen Beattie. 1991. Where to look first for children's ' knowledge of false beliefs. Cognition, 38(1):1-12.
Chengwei Wei, Yun-Cheng Wang, Bin Wang, and C- C Jay Kuo. 2023. An overview on language models: Recent developments and outlook. arXiv preprint 005559230..05759.
Jeff Sparrow. 2022. 'Full-on robot writing': : the artificial intelligence challenge facing universities. Accessed 2023-01-25.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. CoRR, abs/2201.11900
Michael T. Stuart and Markus Kneer. 2021. Guilty artificial minds: Folk attributions mens rea and culpability to artificially intelligent agents. Proceed- ings of the ACM Camman-Computn Interaction, 5(CSCW2):1-27,
Ethan Gotlieb Wilcox. Richard Futrell, and Roger Levy. 2023. Using computational models to test syntactic learnability. Linguistic Inquiry, pages 1-44.
Arjun Subramonian, Xingdi Yuan, Hal Daumé III, and Su Lin Blodgett. 2023. It Takes Two to Tango: Nav- igating Conceptualizations of NLP Tasks and Mea- surements of Performance. In Findings of the Asxo- ciation for Computational Linguistics: ACL 2023, pages 3234-3279, Toronto, Canada. Association for Computational Linguistics.
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, 7ekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong. Yang Shen, et al. 2023. A comprehensive capability analysis of gpt-3 and 3.5 series models. arXiv preprint arXiv:2303. 10420..
Michael Tomasello. 2003. The key is social cognition. Language in mind: Advances in the study of language and thought, pages 44-57.
Chunting Zhou, Pengfei Liu, Puxin Xu, Stini Iyer. Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer. and Omer Levy. 2023a. Lima: Less is more for alignment.
Michael Tomasello et al. 1995. Joint attention as social cognition. Joint attention: Its origins and role in development, 103130: 103-130.
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023b. Navigating the Grey Area: Expressions of Overconfidence and Uncertainty Language Mod- els.
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Aune Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.15 139771.


Reviews:
[{"review_id": "bYrmHR6Q9g", "reviewer": "Reviewer_QBMH", "summary": "", "questions": "Question A: Does the position paper implies that the selection and definition of mental states for LLMs hold little significance as long as pragmatic value exists?\n\nQuestion B: Does the position paper implies that researchers can only form theories about how LLM blackbox works, and confirming theories about LLM blackbox workings become extremely challenging, akin to forming theories for human biological processes?\n\nQuestion C: Can this perspective be extended to future foundation models, such as large vision-language models, that could also be interpreted as agents with a language channel? Why?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**First of all, thank you very much for your time. We are grateful for your positive remarks and the thoughtful and constructive comments. Please find our responses below: suggested changes to the final draft are marked with bullets.**\n\n\n## Weaknesses\n---\n### 1\nOur paper is indeed focused on how we can approach LLM capacities and mental states at a fundamental level. Still, we believe that the pragmatic approach is also relevant for e.g. engineers and users. Interactive systems that show complex behaviour will always ask reflection on the relation between human and machine cognition. Users can use our arguments to explain why e.g. attributing mental states to such systems is defendable. And, because a pragmatic account is mainly focused on the practical value of a system’s complex behaviour, it is liberal on the specific taxonomies/frameworks of the mental employed in realizing behaviour. An engineer can make sense of system behaviour with pragmatist arguments, *even* when no such explicit taxonomy/framework in creating the system is involved (which seems largely the case for LLMs). \n\n- We will explain in the discussion section our conviction that discussion at the fundamental level, will ultimately benefit theorists and practitioners.\n\n### 2\nPragmatic accounts of mental states are in no way intended as the only attempt to ‘open the black box’ of LLMs (or humans). Accounts of mental states can operate at different levels of analysis and explanation, similar to how neurobiology may identify different correlates for specific mental states compared to social psychology, yet these need not exclude each other (although their relation is likely complex). Regarding (1), it is not the case that ‘anything goes’ in a pragmatic account, as different explanations have different pragmatic values depending on in which context they are used. A scientific context may require an explanation to meet additional pragmatic concerns, compared to an everyday social context (section 3.5).  \n\n- We will make the different levels of analysis explicit in the discussion, possibly with reference to David Marr’s (1982) account of levels. \n\nMarr, D. (1982), Vision: A Computational Approach, San Francisco, Freeman & Co.\n\n### 3\nWe agree that a broader discussion of ToM could be an interesting extension for this paper; as we could discuss more extensively e.g. i) why people attribute mental states to others at all (what are its benefits in the context of evolution, lines 670-678), and ii) what further benefits ToM has (e.g. advanced reasoning about mental states such as embedded mindstates, in addition to benefits mentioned at lines 651-684). \n\n- If this matches with your idea of explainability, we will include these points in section 3.4. \n\n## Questions\n---\n### A\nThis question is mostly addressed in the discussion of Weakness 2: for the pragmatist there is no objection towards further study of correlates of mental states in LLMs (or humans), although this is indeed a challenging task. Nor would the pragmatist object to creating different types of systems (symbolic, RL-based, language models, etc.) that attempt to conceptualize or model intentionality. They operate at different levels of analysis, which will be made clearer in the final draft of the paper.\n\n### B\nPragmatism does not object against forming and testing theories about how LLMs work, yet it offers a workable approach to go by while the extremely challenging task of mapping out the mechanisms of LLMs’ ability for e.g. modelling intentions of users is ongoing. \n\n### C\nVery interesting; we think the short answer is yes, for the same reasons set out in 3.1 and 3.2 of the paper. Visual information could additionally help for example to disambiguate information by ‘grounding’ (albeit as a proxy) distinctions common in the verbal realm with an additional information channel, which likely boosts pragmatic value of the system’s output. \n\n- We will add reflection on the applicability of our approach to future models in our discussion.\n\n---\n**We hope that this addresses your points. Thank you also for the pointers to additional relevant literature; we will carefully check the papers and consider them for the final draft.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strength 1: This position paper is very fluent and beautifully-written, and presents a delightful read. It masterfully addresses timely issues around LLMs, providing a well-rounded representation of the current state of the field, while also giving due recognition to the main counter-arguments.\n\nStrength 2: The authors offer a philosophical perspective centered in pragmatic values, which potentially helps to dispel unwarranted accusations directed at LLMs. Moreover, this perspective aims to redirect the focus of debates to the pragmatic value it brings to the human users, rather than definition of “real understanding”.\n", "reasons_to_reject": "Weakness 1: Lack of connection to concrete action items to build NLP applications with LLMs. The definition of mental states in this work focuses on the latent nature, rather than a concrete taxonomy or a theoretic abstraction framework. While helping researchers to shape a scientific philosophical perspective of language understanding in LLMs, the connection to concrete, grounded applications is relatively weak. The audience would largely be academic researchers with a focus on AI as a science rather than practitioners and researchers who are interested in AI as engineering.\n\nWeakness 2: The pragmatic account of LLMs offers a perspective to simplify and abstract the actual underlying blackbox, and understand their behaviors. On the other hand, it introduces two additional concerns: (1) the selection and definition of mental states for LLMs hold little significance as long as pragmatic value exists, and (2) researchers can only form theories about how LLM blackbox works, and confirm theories about LLM blackbox workings become extremely challenging, akin to forming theories for human biological processes.\n\nWeakness 3: The definition of mental states in this work seems to overwhelmingly favor intentionality. As theory of mind (ToM) is essentially the ability to ascribe mental states, I believe this paper can benefit from a slightly extended discussion on ToM and explainability.\n", "questions_to_authors": "Question A: Does the position paper implies that the selection and definition of mental states for LLMs hold little significance as long as pragmatic value exists?\n\nQuestion B: Does the position paper implies that researchers can only form theories about how LLM blackbox works, and confirming theories about LLM blackbox workings become extremely challenging, akin to forming theories for human biological processes?\n\nQuestion C: Can this perspective be extended to future foundation models, such as large vision-language models, that could also be interpreted as agents with a language channel? Why?\n", "paper_topic_and_main_contributions": "Upon the recent debate on the understanding ability of large language models (LLMs), this position paper presents a critical analysis of three prominent criticisms levied against them. These criticisms assert that LLMs are limited in their capacity to grasp concepts beyond distributional semantics, lack functional language competence, and do not aid in human language acquisition. The authors emphasize the need for nuance in the discussion of these issues. In the latter section, the authors offer a philosophical perspective grounded in pragmatic considerations, which serves to justify situations where attributing mental states to LLMs becomes relevant and holds pragmatic significance. By delving into these discussions, the paper seeks to contribute to the community by dispelling unfounded allegations against LLMs and shifting the focus of debates from the ontological nature of understanding to the practical value it entails.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {"Missing_References": "**General Ideas:**\n\nIt Takes Two to Tango: Towards Theory of AI's Mind. Arjun Chandrasekaran et al.\n\n**Evidence that LMs learn more than the next word:**\n\n[1] Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. Kenneth Li et al. ICLR. 2023\n\n[2] Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs. Peter Hase et al. 2023. EACL.\n\n**Computational inquiry of LMs as human language learners:**\n\n[1] Using Computational Models to Test Syntactic Learnability. Ethan Gotlieb Wilcox et al. 2023. Linguistic Inquiry.\n\n[2] Language Acquisition: Do Children and Language Models Follow Similar Learning Stages? Linnea Evanson et al. 2023. ACL.\n\n[3] World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models. Ziqiao Ma et al. 2023. ACL.\n"}}, {"review_id": "mx2GDJPAn0", "reviewer": "Reviewer_6wK9", "summary": "", "questions": "Additional Comments: \n\nA. The paper states “[m]ental states are not intended as literal accounts of the underlying complexity of humans”. This is given no supporting citation, and I do not think this is a universally agreed upon point. I would imagine many theorists actually attribute mental states to humans. \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**First of all, thank you very much for your time. We are grateful for your positive remarks and the thoughtful and constructive comments. Please find our responses below: suggested changes to the final draft are marked with bullets.**\n\n## Reasons to Reject\n---\n### 1\nWe intend our account not to be normative, but aim to explain why humans can attribute mental states to LLMs, and why it can be rational and productive to do so in certain contexts. This is close to Daniel Dennett’s ‘intentional stance’, as explained in section 3.3. We observe that current LLMs, especially those that are instruction-tuned (e.g. GPT-4, PaLM2), prompt users to engage with them as they would with other humans, incl. queries that pertain to mental states. We do not argue that users *ought* to see such systems as having beliefs and desires, although this behaviour *is* justifiable: it may enable efficient communication or perhaps even a sense of relationship (lines 651-700), thus, have pragmatic value. Although we mention example applications of LLMs (lines 685-700) in society, in which interaction with LLMs using mental state language seems natural, and mention that there are critical ethical concerns, we feel an in-depth discussion of such topics is beyond the scope of the current paper.\n\n- We will review and disambiguate section 3.4 to clarify that our approach is not intended to be normative.\n\n### 2\nThe pragmatist focuses primarily on a system’s (complex) behaviour, and on whether our attribution of mental states helps us achieve our goals (e.g. efficient communication, or perhaps a sense of relationship as mentioned in lines 685-700). This is not because a system might be intelligent, but because interaction with it is best served by assuming that it is. While we argue that this is the case for LLMs, it is not for planets: our interaction with their behaviour is better served by physical models, given our goals such as landing a Mars rover, thus in this case physical models would have more pragmatic value.\n\n- This point will be made more explicit in the final draft, to avoid confusion.\n\n### 3\nIn section 2.2.2., the points we raise about human thought are not primarily intended to claim that LLMs actually have or do not have thought (line 315), but rather to call for some reflection about how testing thought in LLMs is currently done. \n\nIf we understand you correctly, you see (i) as the idea that all people have mental states, and (ii) as a form of (commonsense) reasoning. The pragmatist agrees with (i), as it has a strong social justification (lines 611-616) to think that others have mental states. The point regarding (ii) is that the robustness of human reasoning varies with context (lines 300-307), and capacities for subtypes of thought (e.g. social reasoning) may differ among people, yet we are typically more nuanced in drawing conclusions on thought in humans; we e.g. may critically reflect on our tests first, attempt to test skills via other modalities, and so on. We call for the same caution in testing LLMs, but do not aim to assert or deny thought in LLMs with these points.\n\n- We agree that our final draft will benefit from making this point clearer in section 2.2.2. \n\n## Questions\n---\n### A\nOne example of AI researchers who employ the same pragmatic idea of mental states as we do, are Rabinowitz et al. (2018). From their introduction: \n\n*\"What does it actually mean to “understand” another agent? As humans, we face this challenge every day, as we engage with other humans whose latent characteristics, latent states, and computational processes are almost entirely inaccessible [...] A salient feature of these “understandings” of other agents is that they make little to no reference to the agents’ true underlying structure. A prominent argument from cognitive psychology is that our social reasoning instead relies on *high-level models* of other agents.\"* [Our italics]\n\nThe idea of high-level models stems from Gopnik & Wellman (1992) (quoted by Rabinowitz and colleagues), scholars from developmental psychology, who make a related point about how children develop models of other humans' mental states as *theories* intended to explain and predict behaviour, rather than literal accounts. Regarding language understanding specifically, pragmatism as stance within the philosophy of language, in which language understanding or intentionality is primarily assessed through observable behaviour, is discussed in the review by Bommasani et al. (2022). Note that for a pragmatist, attributing mental states to others is perfectly justifiable if it has pragmatic value (e.g. allows us to explain and predict behaviour), which will be often the case.\n\n- Bommasani et al. (2021). On the Opportunities and Risks of Foundational Models. arXiv preprint, arXiv:2108.07258. \n- Gopnik, A. and Wellman, H. M. (1992). Why the child’s theory of mind really is a theory. Mind & Language, 7(1-2): 145–171. \n- Rabinowitz, N., Perbet, F., Song, F., Zhang, C., Eslami, S.M.A. &amp; Botvinick, M.. (2018). Machine Theory of Mind. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research. 80:4218-4227 \n\n---\n**We hope that this addresses your points.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is thought-provoking and articulates an interesting philosophy for understanding neural models. I expect the paper to engender discussion in the field (at least the part of the field interested in how to conceptualize models). Further it is well-written and bridges the gap between NLP and philosophy in ways that are rare in the field right now.  ", "reasons_to_reject": "While I enjoyed this paper, there are three things that the paper should consider (assuming that I have correctly understood the paper. If not, I will update my comments). \n\nFirst, the aim of the paper is a bit unclear. Namely, whether the paper is outlining why people may be led to attribute mental states to LLMs and an articulation of when they should. I believe my confusion stems from statements like “we reflect on the circumstance under which it would make sense for humans to similarly attribute mental to LLMs” and “we ask [when] it is justifiable for humans to make mental models of LLMs”. I read this as either stating conditions that motivate people to make these claims or stating conditions when people ought to make such claims. While I accept the work as possibly accounting for why people make such claims, the work lacks discussion of whether people ought to. Similar issues concern the “pragmatic” aims of treating language models as having mental states with queries (on lines 663 and 667) like “Do you know how to do X” and “How strong is your belief that X”. Under the first interpretation (i.e., mental states are attributed), theses queries seem reasonable. People treat the model as if it has these capacities, so they naturally query it for these states (as they do with other humans). Under the second interpretation (i.e., models ought to be deemed as having mental states), more work should be done to justify why these queries are good. For example, do current models actually have the capacity to respond to queries like this in useful ways? More generally, it actually may be harmful for people to associate mental states to LLMs, as this may lead them to be prone to misinformation, for example. The paper would benefit from engaging with the societal conditions that language models are embedded in. \n\nSecond, one argument (as I understand it) in the paper lacks some justification. I read part of the work as asserting something like i) LLMs are intelligent systems, ii) humans best understand intelligent systems by ascribing to them mental states, and iii) therefore LLMs should be deemed as having mental states. The question the field faces, in some part, is whether LLMs are actually intelligent systems, so this first point depends heavily on how one understands (limited) existing empirical evidence. If we read these points as an account for the reasons laypersons assign mental states to LLMs, then as above, some justification is needed for why this actually helps people understand LLMs. Cars and planetary movements are complex, but I don’t think the best account of those is to assign them mental states. Presumably, other factors contribute to when we should do this (e.g., things being “social entities” as on line 647, though again LLMs aren’t necessarily social entities). The work, then, would be to justify why LLMs should be taken as having those abilities. I see this as also dove tailing with point 3 on page 6 -- “attributing mental states…help[s] us to abstract underlying complexity away”. However, we don’t understand all complex things as mental, so why treat LLMs this way? \n\nThird, section 2.2.2 seems to me to be taken as given when the points of it are at issue in the ongoing debates of the field. I think my confusion stems for brining two sense of thought: i) thought as a human capacity for certain types of mental states, and ii) instances of thinking in a colloquial sense (e.g., good at solving riddles). It is stated on lines 297-298, that “thought capacity is unequally distributed in humans”. However, all humans possess thought of type (i) and I take this claim to be of type (ii). Why should differences in things people are good at thinking about be a justification for thinking LLMs have thought? \n", "questions_to_authors": "Additional Comments: \n\nA. The paper states “[m]ental states are not intended as literal accounts of the underlying complexity of humans”. This is given no supporting citation, and I do not think this is a universally agreed upon point. I would imagine many theorists actually attribute mental states to humans. \n", "paper_topic_and_main_contributions": "This work is a position piece arguing for how one should proceed to understand whether neural models understand. In particular, the paper first discusses three main critiques of the capacity of large language models: i) language models as parrots, ii) language models as learning formal and not functional aspects of language, and iii) language models as uninformative w.r.t. language acquisition. The main contribution of this work is a sustained discussion of a particular world view with which to interpret models. ", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "J0VNiAf2EZ", "reviewer": "Reviewer_rDe7", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**First of all, thank you very much for your time. We are grateful for your positive remarks and the thoughtful and constructive comments. Please find our responses below: suggested changes to the final draft are marked with bullets.**\n\n## Points to improve\n---\n### Regarding Section 2.3 / what LLMs can contribute to language acquisition research:\n\nResearch shows that the unparalleled capacity of LLMs to produce coherent text relies at least in part on their capacity to learn complex linguistic features during training (lines 287-388). While it is obvious that there are numerous differences between LLMs and human learners, we argue that their case is still informative for studying specific aspects of language acquisition, such as the question to what degree lexical understanding and semantic ambiguity in children can be theoretically accounted for by statistical learning mechanisms (lines 392-413 and 414-431). LLMs are wrong in key respects (like all models), still they are very advanced distributional learners that can help us find out what linguistic phenomena can *in principle* be accounted for by information extracted from word contexts.\n\n### Section 2.2.1 / disentangling language and thought:\nThe confusion (“the paragraphs seem to contradict themselves”) could follow from our phrasing at the beginning of this section (215-219). We do not want to argue for or against thought in LLMs in this section, but argue that the way we usually try to separate language from thought capacity in humans, is not straightforwardly applicable for LLMs. Lines 215-219 serve as clarification (but is perhaps not clear enough) that we question the *methods* for testing thought in LLMs, but do not intend to use our points as arguments for or against thought capacity (as separate from language) in LLMs.\n\n- We will clarify our aim of this section in lines 215-219.\n\n### Regarding potential outdatedness of BERT references:\nGiven the fast pace of the field right now, it is a good point that this paper requires recalibration as more LLMs come to the fore (e.g. LLaMA2, Falcon-Instruct). We have carefully checked our arguments and references for relevance to recent developments at the time of writing the paper (April-June 2023), and will include an additional round of updates in the final draft. Still, research about the specific linguistic and cognitive capacities of recent models has yet to emerge. In the discussion mentioned in the introduction (lines 44-59), various reserchers voice (theoretical) arguments concerning BERT (e.g. Bender et al. (2021)), but also more recent models like GPT-3, LaMDA (e.g. Bender et al. (2021), Marcus (2022)) and Chat-GPT (e.g. Piantadosi (2023) and Shanahan (2022)). \n\n- We will include an outlook on recent models in the discussion section in the final draft.\n\n---\n- Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. \n- Gary F. Marcus. (2022). Nonsense on Stilts. Online resource. https://garymarcus.substack.com/p/nonsense-on-stilts Accessed on: 2023-01-30. \n- Murray Shanahan. (2022). Talking About Large Language Models. arXiv preprint arXiv:2212.03551. \n- Steven T. Piantadosi. (2023). Modern language models refute Chomsky’s approach to language. Online resource. https://lingbuzz.net/lingbuzz/007180. Accessed on: 2023-04-15.\n---\n**We hope that this addresses your points.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper makes some convincing arguments. It convincingly argues that the claim that “all LLMs can do is next word prediction” is too simplistic and that underclaiming LLMs capabilities is not useful in academic debates. The paper also brings up an interesting point by saying that when we claim that humans have mental models, we generally also only do so from an outsider perspective. Lastly, the emphasis on more sophisticated LLM evaluation is also very important and the point that thought is a continuum is also very interesting.", "reasons_to_reject": "A couple of the arguments made in the paper could be made more convincing: The section about language acquisition in LLMs vs. humans (Section 2.3) is confusing and it is unclear what argument is being made about language acquisition. It is also unclear what the message is in Section 2.2.1: the paragraphs seem to contradict themselves.\nAdditionally, many of the citations (especially in the introduction) are bertology papers and it is unclear how their findings can be applied to more recently published LLMs.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is a position paper (with some aspects of a survey paper). It presents two main arguments. First it argues that more nuance is needed in response to claims that LLMs simply regurgitate statistical patterns. Secondly, it offers a new perspective on the definition of “understanding” and why attributing mental states to LLMs could be useful in terms of pragmatic value. To form these arguments, this paper surveys a multitude of papers ranging from the analysis of LLMs to philosophy of science papers.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers gave scores of 4,3,3 (soundness) and 4,4,4 (excitement).

Strengths and weaknesses including the following were prominent:

Strengths:

- well written (R1) and thought-provoking (R2, R3)

Weaknesses

- lack of connection to concrete implications for NLP applications using LLMs (R1)
- questions were raised about the philosophical underpinnings and justification of some claims (R1, R2, R3)
Decision: Accept-Main