Blackbird language matrices (BLM), a new task for rule-like generalization  in neural networks: Can Large Language Models pass the test?
Authors: Paola Merlo
Venue: EMNLP 2023 Findings (2023)
Keywords: Rule-based learning
generalisation
intrinsic evaluation
cognitive modelling
benchmarking
TL;DR: We propose a method to evaluate LLMs ability to rule-based generalization.

Abstract: How do we evaluate Large Language Models (LLMs) and determine the aspects and limits of their intelligent behaviour?

It is currently conjectured that shortcomings of LLMs in multi-linguality and reasoning are due to a lack of ability to generalize.
It has been argued that, instead, humans are better at generalization because they have a tendency at extracting rules from complex data. 
We propose a method to evaluate LLMs ability to rule-based generalization.

When exposed to tests of analytic intelligence, for example the visual RAVEN IQ test, human problem-solvers identify the relevant objects in the picture and their relevant attributes and reason based on  rules applied to them. Based on the induced rules, they are able to provide  a generalisation and a solution to the test.

An analogous language task has recently been proposed (called BLM) for LLM. 
In this paper, we argue that we can use this task to investigate what linguistic reasoning LLM develop, by asking them to solve some simple variants of the BLM task.
    
We find that current state-of-the-art generative models, such as ChatGPT, can handle the task in the sense that they easily understand the instructions and can provide step-by-step reasoning that shows that it can solve two of the main cognitive hurdles: correspondence finding (object and attribute identification) and item novelty. However, overall they cannot find the correct answer, even with considerable help. In particular, they never identify the structure of the problem, exhibiting, we hypothesize, a lack of goal and subgoal management abilities, an ability that has been argued to measure differential abilities in humans. 

We argue that this finding supports the usefulness of the task as a method to test the limits and specific properties of generalisation ability in Large Language Models, providing an intrinsic evaluation method inspired by tests of human intelligence.

Blackbird language matrices (BLM),
new task for rule-like generalization in neural networks:
Can Large Language Models pass the test?
Paola Merlo
University of Geneva
Paola.Merlo@ @uiige.cnniehhh
Abstract
models that can be tested on behavioural and neu- ral data (Schrimpf et al., 2020). Finally, another point of view has been put forth recently that, to further understand the components of human intel- ligence, one needs to study how this intelligence evolved, decomposing it into large-scale studies of primitive skills that humans share with other animals (Zador et al., 2023).
How do we evaluate LLMs and determine the aspects and limits of their intelligent be- haviour?
When exposed to visual tests of analytic intel- ligence, human problem-solves identify rules applied to relevant objects and attributes. Based on the induced rules, they can generalise and are able provide solution the test.
Large scale, highly integrative, comparative ap- proaches are necessary: they give bird's eye view essential to map unknown territories, define bound- aries, guide long-term trends. But we also think that curated data and in-depth qualitative analyses of the intelligent behaviour of these models are needed (Rogers, 2021).
An analogous language task has recently been proposed (called BLM) for LLM. In this paper, we use this task to investigate what linguistic reasoning LLM develop. by asking them to solve some simple variants of the BLM task.
We find that current state-of-the-arr generative models can handle the task: they easily under- stand the instructions and can provide step-by- step explanations. The explanations show that LLMs can solve two the main hurdles: corre- spondence finding (object and autribute identifi- cation) and item novelty. However, overall they struggle to find the correct underlying global rules, even when they find the right answer.
We will concentrate on one such skill, one of the core intelligent human abilities, the ability for ana- lytic rule-based generalisation. We will illustrate what tests exists for this skill and evaluate qualita- tively how GPTs rate on that, based on carefully curated data.
2 Rule-based generalisation
We argue that these findings support the useful- ness of the task as method to test the limits and specific properties generalisation abil- ity in Large Language Models, providing an intrinsic evaluation method inspired by tests of human intelligence.
Recent approaches to generalisation have shifted attention from the traditional narrow definition an extension from set of data points to new data points of exactly the same nature (i.i.d. assumption) (Schölkopf, 2019) to out-o--distribution gener- alisation, be it across languages, across domains, or new unseen combinatorial tokens and structures (Hupkes et al., 2022).
1 Prologue
How do we evaluate LLMs, determine the aspects and limits of their intelligent behaviour and com- pare them to human intelligence?
Humans are good generalizers. A large body of literature of experimental work has demonstrated that the human mind is predisposed to extract reg- ularities and infer the generative rules from data, in way that is distinct from the patterns of activa- tion of neural networks (Lakretz et al., 2019, 2021; Sablé Meylr al., 2021: Dehaene al., 2022).
A new trend of large-scale multi-task LLM and neural network evaluation has emerged that aims to evaluate LLM in realistic settings. In NLP see Big-bench initiatives, to better chart LLMs ca- pabilities (Srivastava et al., 2023). In neuroscience. a similar trend, dubbed integrative bench-marking, looks at large language models as mechanistic hy- potheses on how the brain encodes intelligence,
One possible approach to develop more robust methods, then, to pay more attention to the de- composition of complex observations and to the

---

causal chains in the generative process that gives rise the data. To discover the underlying problem structure, the notion of disentanglemen comes to the rescue. A disentangled representation can be defined as one where single latent units are sensi- tive to changes in single generative factors, while being relatively invariant to changes in other factors (Bengio et al., 2013).
progressive matrices for vision), called Blackbird Language Matrices (BLMs) (Merlo, 2023). BLMs define prediction tasks to learn complex linguistic patterns and paradigms.
Because this task relies on mixture of language and abstract rule learning abilities, we conjecture that, used in an evaluation setting, it is appropriate to investigate the limits and properties of linguistic intelligence and abilities in current state-of-the-art LLM, and compare them to humans.
Let's look at an illustrative example of complex linguistic relations that might require discovering the underlying disentangled factors and causal re- lations: the CAUSATIVE alternation exhibited by English verbs, shown (1).
How do humans solve RPMs?
Raven's progressive matrices are visual test of analytic intelligence (Raven, 1938). To establish how to develop human-like rule-based inference, it pays to survey how humans solve this kind of intelligence test.
(1) The teacher opened AGENT The door opened.
the door. THEME
THEME
3.1 Raven's Progressive Matrices for vision
This alternation applies to change of state verbs, such as open, break, melt, burn among many others, verbs that describe change that affects the state of the undergoing participant (the door changes from state of being closed to state of being open). They occur in two subcategorisation frames that are related to each other in a regular way: the object of the transitive frame is the subject of the intransitive frame. This way, in terms of semantic roles, the subject of the transitive is an AGENT, but the subject of the intransitive is THEME.
?
Figure 1: Example progressive matrice in the visual world. The task is determine the missing element visual pattern. Given the matrix the left, choose the last element the matrix from the choice of elements the right. The matrix is constructed according two rules (see text for explanation). Identifying these rules leads to the correct answer (marked by double edges).
To learn the structure of such complex alter- nation automatically, despite the simplicity of the sentences, neural network must be able to iden- tify and solve set of subproblems and compose them into an organised solution: (i) identify the elements manipulated by the alternation (teacher, door); (ii) identify the relevant attributes of these elements (open, closed state of door; volition of teacher); (iii) the object of the transitive sentence is the same event participant as the subject of intran- sitive, that is the rules of change of the attributes must be identified across the two sentences.
Raven's progressive matrices (progressive be- cause tasks get harder) are IQ tests consisting of sequence of images, usually organised in a 3x3 matrix, called the context, connected in logical sequence by underlying generative rules (Raven, 1938). The task is determine the missing ele- ment, usually the last one, this visual sequence. the answer. The answer is chosen among a set of closely or loosely similar alternatives. A simple instance of this task, of our devising, is given in Figure 1: given a matrix on the left, choose the last element of the matrix from choice of elements. The matrices are built according generative rules that span the whole sequence of stimuli and the an- swers are constructed to similar enough that the solution can found only the rules are identified correctly. For example in Figure 1, the matrix is
To study what factors and models lead to learn- ing more disentangled linguistic representations, representations that reflect the underlying linguistic nules of grammar, possible approach is to develop curated data on a large scale, building models to learn these data and investigating the models' be- haviour.
To this end, a new linguistic task has been devel- oped, (similar to the IQ test called RPM, Raven's

---

constructed according to two rules: Rule from left right, the red dot moves one place clockwise each time. This pattern continues onto the next row; Rule 2: from top to bottom, the blue square moves one place anticlockwise each time. This pat- tern continues onto the next column. Identifying these rules leads to the correct answer, the only cell that continues the generative rules correctly. Vari- ants of this structure organise the rules row by row, with the last element of the last row as the missing element.
one rule time. The general result, then, is that the processing is incremental, that the number of rule tokens affects the error rates, and that there is a wide range of differences among individuals their performance on this test.
Item Novelty Variety and novelty of lexical items was also found be source of difficulty. In conclusion, the study finds that the processing characteristic that common all subjects is an incremental, re-iterative strategy for encoding and inducing the regularities in each problem. The processes that distinguish among individuals are primarily the ability to induce abstract relations and the ability to dynamically manage large set of problem problem-solving goals in working memory. These two abilities are what distinguish, according to this study, better problem-solver from average ones.
3.2 Measuring analytic intelligence in humans In very articulated study, Carpenter al. (1990) analyze the cognitive processes at play in Raven Progressive matrices tests (RPMs), as a widely used, nonverbal test of analytic intelligence. The analysis is based on detailed performance charac- teristics such as think-aloud verbal protocols, eye fixation patterns, and errors. Of particular inter- est for us are the three main sources of difficulty that are identified by the study: the correspondence problem, goal and subgoal management, and item novelty.
Blackbird Language Matrices
CONTEXTS TEMPLATE
1
NP-sg
PPI-sg
VP-sg
2
NP-pi
PPI-sg
VP-pl
3
NP-sg
PP1-pl
VP-sg
4
NP-pi PP1-pl
VP-pl
Finding corresponding elements In problems
5
NP-sg PP1-sg
PP2-sg
VP-sg
6
NP-pl PPI-sg
PP2-sg
VP-pl
with multiple generative rules, the problem-solver must determine which visual elements or attributes in the entries are governed by the same rule, process that Carpenter and colleagues call corre-
7
NP-sg
PP1-pl
PP2-sg
VP-sg
8
NP-pl
PPI-pl
PP2-sg
VP-p
ANSWER SET
NP-sg PPI-sg et NP2 VP-sg
Coord
2
NP-pl PP1-pl NP2-sg VP-pl
correct
spondence finding. Interestingly, it is conjectured that even for visual problems, people use lin- guistic method to find correspondences. Namely,
3
NP-sg PPI-sg VP-sg
WNA
4
NP-sg PP1-sg PP2-sg VP-pl
AE
5
NP-pl PPI-sg PPI-sg VP-pl
WNI
6
NP-pl PP1-pl PP2-pl VP-pl
WN2
the correspondences reflect the use of a matching- names heuristic: figures with the same name might correspond to each other.
Figure 2: BLM instances for verb-subject agreement, with two attractors. WNA=wrong nr. of attractors: Al=agreement emror; WN nr. for 1st attractor (N1); WN2=wrong nr. for 2nd attractor (N2).
Goal and subgoal management In recordings of eye movements and verbal protocols, the re- searchers found signs of demonstrably incremen- tal nature of the processing. The subjects solve a problem by decomposing it into successively smaller sub-problems and then proceed to solve each sub-problem. This brought about an incre- mental, and sometimes recursive, solution to the problem. The induction of the rules was incre- mental in two respects. First of all, in problems containing more than one rule, the rules were de- scribed one time. Second, the induction of a rule consisted of many small steps, reflected in the pairwise comparison of elements adjoining entries. Morcover, the verbal reports show that the subject mentioned one attribute at a time and also
We propose apply similar test to large lan- guage model. We use here the new task and data format, called BLACKBIRD's ' LANGUAGE MA- TRICES (BLMs), proposed in Merlo et al. (2022) and defined formally in Merlo (2023). (Sce also An et al. (2023); Samo et al. (2023); Merlo et al. (2023a) for more details on data creation for gram- mar problems and publicly available data).
Like RPMs. BLM instance consists of con- text set and answer set. The context is se- quence of sentences that encode linguistic rule e.g., subject and verb number should agree in their number independently of how many noun phrases intervene between them. BLMs are presented (to

---

an artificial language model to a human partici- pant) as linguistic puzzles requiring the selection of the missing sentence. In order to examine the repre- sentations underlying the response, the answer sets include negative examples following corrupted gen- crating rules. An example template is illustrated in Figure 2.
and plural subject and plural first attractor and singular second attractor.
4.3 The causative matrix
The linguistic specification of the causative alterna- tion has been given in Section 2. The structure of the causative matrix, shown in the prompt exempli- fied in Figure 5, is different from the previous one and it is both harder and easier than the previous It is easier in that we progressive matrices, namely the sequence of stim- uli is organised into subsets. And harder be- cause the rule to learnt more implicit and has no surface morphological correlate, unlike agree- ment.
For example, the BLM problem for subject-verb agreement (Merlo et al., 2022; An et al., 2023) con- sists of a context set of seven sentences that share the subject-verb agreement phenomenon, but dif- fer in other aspects e.g. number of intervening attractors between the subject and the verb, differ- ent grammatical numbers for these attractors, and different clause structures. The answer set com- prises contrastive sentences that violate some of the generative rules. Actual examples can found Section
property of easy
The first four-tuple of sentences is meant to in- dicate that the subject of the transitive is agentive (first sentence), that the subject of the intransitive is not necessarily agentive (second sentence), and the next two sentences express the initial and final state of the change of state. The second quadru- plet, which the last element nceds to be guessed, repeats the same pattern. The sequence pattern, then, is quite easy to see. But the underlying rules to be learnt have to do with the mapping of gram- matical functions to semantic roles (subject=Agent, object=theme in the transitive, subject=theme in the intransitive), notoriously difficult task in ral networks.
4.1 The linguistic phenomena
The linguistic phenomena we have tested are subject-verb number agreement, the causative al- ternation and the sequence of tenses. In a BLM, the linguistic phenomena have two levels of prop- erties: the external properties (E), those that are easily seen in the strings, and internal properties (I), that are only inferrable by abstraction (see for- mal specification in Merlo (2023) and Merlo et al. (2023b)).
4.4 The sequence-of-tense matrix
4.2
The agreement matrix
In this matrix, we try a different kind of Long-distance agreement problem, the so called sequence-of-tenses rule. The sequence of tenses rule governs the coherent use of tenses in different sentences. It is especially used in hypotheticals. which require subjunctive and conditional mood, in the present, past and future tenses. The language is Italian, so that the verb conjugation clearly marked morphologically, both in mood and tense, also making a difference between singular and plu- ral.
The prompt exemplified in Figure 4 shows ample of a BLM agreement matrix. In subject-verb agreement the primary property is that the subject and the verb must match in grammatical number (singular and plural). A secondary property indi- cates that these two elements do not need to be linearly adjacent, because the rule applics on the syntactic structure of the sentence and not on its linear order (Gulordava et al., 2018; Linzen et al.. 2016). In this respect, intervening elements end up having the effect of confounding the agreement pattern.
4.5 The mixture matrix
The sequence is generated by rule of progres- sion of number of attractors (one and two), rule of subject-verb agreement that alternates every sen- tence between singular and plural of the head noun and a rule of number of the attractors that alter- nates between singular and plural with a frequency period of two. Thus, the correct answer for this example is a sentence that has three noun phrases
Finally, we try more complex case, in which we have both the sequence of tense rule and the agree- ment rule, which in this case covers both number and gender agreement. The pattern template of the data is shown in Figure 3. An example is shown in the prompt of Figure 7. As can noticed, the agree- ment pattern is similar to the pattern shown in Fig- ure 2, with number agreement alternating at every

---

sentence PS..P). the gender agreement pat- tern alternating every two sentences (M...F.F....) and an intervening attractor, exhibiting no agree- ment pattern, which is absent in the first four sen- tences and present in the last four sentences. Notice that, in this matrix, the attractor is an incidental sen- tence, so it also interferes in structure and distance with the sequence of tenses, by introducing third verb and lengthening the distance between the ele- ments in the tense sequence.
argued (Mahowald al., 2023).
To get general sense if ChatGPT solves the problem in the same way as human subjects solve RPMs, the instructions we give follow the same protocols those given to subjects in Carpenter et (1990) (see Figure 4 and Figure 5). We per- form two tests. First, we use the publicly avail- able ChatGPT 3.5, in single one-day sessions, a different day for each type of matrix, to collect qualitative interactions. Then we use ChatGPT-4, over three days, for more systematic collection of quantitative results. When appropriate, we give the sentences in Italian to reduce the possibility that ChatGPT has learned the structure of the problem from papers previously published on the Internet.
CONTEXT
SgM-CPas
SgM-CnPt
PIM-CPas
PIM-CnPr
SgF-CTrPPa
SgM-CnPas
PIF-CTrPas
PIM-CnPas
SgM-IFut
attractorPLF
SgM-IFut
PIM-IFutAnt
attractorSgM
PIM-IFut
We will reach the conclusion that ChatGPT does not really manage to distill the underlying rules of the problem, but this should not detract from the first striking observation: is entirely possi- ble to have a series of interactive exchanges with ChatGPT on this task in a completely coherent way. This impressive result confirms the formal ability of current LLMs to process language. However, the analysis of the interactions shows that Chat- GPT3.5 struggles with the rule-based nature of the task, confirming that its functional ability is still relatively limited. Even its simplest form and with a lot of guidance from the prompts and dia- logue, the nature of the problem is not understood and reliable answers are not reached. Here we give a summary of some interactions. The detailed pro- tocols are in the appendix. As will be shown in the next section, ChatGPT4 exhibits better perfor- mance, observed in many other cases, but the overall result is confirmed: it does not manage see the underlying rules.
SgF-IFut
attractorPIM
SgF-IFut
2?
ANSWERS
PIF-IFutAnt
attractorSgF
PIF-IFut
SgF-IFut
SgF-IFut
C SgM-CTrPas
SgM-CnPas
D PIM-IFutAnt
PiM-IFut
SgF-CTrPas attractorSgF
PIM-CTrpas
PiF-IFutAnt
PIF-IFut
Figure 3: Contexts and Answers patterns of the mixture BLM (CPas:past subjunctive; CTyPas:perfect past subjunctive; CuPrppresent conditional: CnPas:past conditional; IFut:future indicative: IFutAnt:future in the past indicative).
Properties of correct answer (A): Agryyes, Number.l. Gender:F, attractrryes TensePremise:IFurAae TenseConse- quent
Violations in answer set:
B attracto:no, Agr:Pl;
C- Sg, Gender:M, tense:Cpas, attractorno;;
D Gender:M, attractor:no;
E Agr:no, Number:Sg, tense:CtrPas;
F- attractor:no.
5 Can ChatGPT solve BLMs?
The initial motivation to create this kind of task is to study how to guide current state of the art systems towards better generalisation. Given the current very high performance of large language models, however, it needs to established first if current state of the art systems are already capable of solv- ing this kind of task, or if is challenging for them, and more importantly by which linguistic and rea- soning processes they proceed to find solution. In particular, the complex mixture of grammatical and abductive reasoning that is involved in BLMs yields an interesting test set to see if current LLM are indeed good the formal part of language (syn- tax and semantics), but weak on the functional part (pragmatics and reasoning), as has been recently
5.1 Subject-verb Agreement
Interaction and (reported in the appendix) show that ChatGPT3.5 correctly identifies the three objects (subject, first attractor, second attractor) af- ter inspecting the context, and correctly identifies their attributes (singular, plural) after inspecting the answers. Somewhat incorrectly, it comments on the semantic coherence of the sentence. It finds the right answer, but it eliminates alternatives for incorrect reasons. ChatGPT3.5 is not really finding the 'sequence" notion of the context but is map- ping each answer sentence to a set of objects and attributes, that appears to have established as necessary. The example matrix used the one

---

going osk you selve cailed BLM-AgE
list
isk solve gammariccl poobleas called BLM CausE. give (oumboeed
thai will sentences ixtemitted capital letees)) that in choose smong the unvver untemnoss the semienoe the could the next srrienos the consest list of semennees.
will also you answens. ask you
your lif Feglish
will Ennglish sentences (identifed by capiial letters) cill the snowers will you to choree smong unswrr untrmns the une serienor that could be the nest sentence in the coneest list of sentences. The sentences have been sesenited in rale based staner.
give you
am tsking you give me the conrect anower, but also to deseiibe step by step tie hypotheses you are entertatinn while are solving the problem.
I am asking you to give me the correct answer, but also to describe step by step the begotheies
cotertaating while you uee solving the problem.
Contest
The talk on histury has started beer than expected.
Contect
Thooe responsible for the right will nesign.
The boy opened the door volumturity.
3 The show with the paimings has met with great success.
2 The door opened suldenly.
The thross of reforms
cell the ilactors.
3 The door was closed.
The bunch
shelf.
door
The stadies the the drug appear
cook mehed the bute: veluanaily,
The throar reforms
worries doctons.
butter slowly.
butter haad
Amsuare
A cableclitls the tablc and the console shine seeeping
Asswves
friends owners the come back.
The melled.
C lowyers of murderess
The bay open. The door was open. The cook melted. E The butter was melted.
D The lawyens of the munderers the village will come baok
8 The visit the palace of the endts is approsching.
F The computer with the program of the exgeriment oez broken.
Figure 4: Prompt instructions for interaction of the subject-verb agreement problem with ChatGPT 3.5. The response and other interactions are shown in the ap- pendix. See text for discussion.
Figure 5: Prompt instructions for interaction 1 of the causative alternation problem sponse and other interactions are shown in the appendix. See text for a discussion.
with ChatGPT 3.5. The re-
Figure 4. In interaction 4, we explicitly tell it to look for grammatical patterns. The answer shows that ChatGPT3.5 can produce coarse syntactic analysis. In interactions 5 to 8, we give more detail about number agreement patterns, and that it needs to look for patterns across sentences.
sentences 5-8, to find solution A. However, Chat- GPT does not see the pattern, both in interaction 1, where instruction is given and interaction 2, where instruction is given about the fact that the sentences should be looked at as two block. The in- dividual sentence by sentence explanations of what the sentences the context are supposed to show is however very impressive (and correct). This is shown in Figure 6. To verify that this apaprent insightfulness does not come from simply repeat- ing, parrot-like, snippets of the large literature on this problem in English, we repeat the interaction Italian, where this problem is much less studied.
Interaction is very interesting as, at this point, we are basically giving the problem away, but still ChatGPT3.5 cannot solve it. The answer shows that it parses number agreement incorrectly, so it assigns the wrong number to the nouns phrases and consequently cannot see the pattern. So, in inter- action 10, we go back Italian, where number agreement is more clearly marked on the verb and the noun phrases. The following interactions 10 to 19 clearly show that even with a lot of prompting Cha is incapable of finding the global al- ternation pattern. It seems that ChatGPT3.5 never reaches the right combination of accuracy in identi- fying number agreement patterns, and accuracy in seeing patterns across the sentences.
The Italian interaction repeats exactly the English pattern of reasoning (making us think some transla- tion is going on internally) and no correct solution is found.
5.3 Sequence of Tenses and Mixture of Problems
An example of the prompt for the sequence of tenses problem can be seen in the appendix. An example of the prompt for the mixture of problems agreement and sequence of tenses is shown in Fig- ure 7. We show interaction 1. As can be seen, we specify explicitly that the solution is not to be found in the logical sequence and the topics of the sentences.
5.2 Causative
The interactions about the causatives are impres- sive, despite the fact that a correct solution is not found. This test is both harder and easier than the previous one. The structure of the matrix is casier, the sequence of stimuli is organised into two analogous subsequences, as can be easily seen in Figure 5. So the solution to this puzzle requires figuring out the objects, their purpose and their se- quence of the first four sentences and apply it to the
The responses from ChatGPT3.5 for the se- quence of tenses problem, which can be inspected in the appendix, show that while the explanations of the selection of the correct answer are not very con-

---

you lie ftalian solve grammutical called MLM-Mixd. give will give you the smowars. wwill you chuose soong the unveer vemeraas the one sentence could be the nest sentence the contest list of sentences. The sentences have been generated as rule-based mansie.
To solve the BLM-CausE problem, we need analyze given context sentences and the set of sen- tences determine the most appropriate continuation context. Let's go through the process step by
(nuntherrll 7), will sentences Extemiifeed copual letiers that
am asking you to give me the correct answer, bit also to describe step by step the lyyotteses you are entertaining while
step:
solving the problem. aun also giving bim: This gratematical problem. logical problem. Thee soluticn does linding logical continution, but continatioo that follows underlying gommmtiial pottums
Identify the relevant elements: In this context, we have two main elements: "the door" and "the butter." We need to observe the causality between these elements and their states, Analyze the context:
Contedt
Se fossi ricon, allera sarei felice.
Sc
allor sarebbero riech.. lossi ricca,
felice.
The boy opened the door voluntarily. This sentence establishes that the boy was responsible for opening the door.
Sentence I:
S: umai bowe, indipendeniemeeee
foste state
surste stite ricche.
cirmesanno, allers sami ricuos
Se uree stati brass, nonostaate tutto, ailora savete skcchi.
Sc smii buen, senerdoo conto di imii aln Exteei, allors sand riczzs
Arswess
The door opened suddenly. This sentence suggests that the door opened on its own accord, without external agency.
Sentence 2:
sarete siate hoont, e con un po di foruma, allora sanete rioohe
Se suaa bucus, allom sarai felice.
Se fossi stato huono, allon fossi felixe.
D sarate stati booni. allora sarete ricchi.
F. Se food stata boons, ienando conto di ummi alti fattrri alloea foeeero stati felici.
The door was closed. This sentence indi- cates the initial state of the door, which was closed.
Sentence 3:
boone, alloca deehee
The door is open. This sentence reveals the current state of the door, which is open.
Sentence 4:
Figure 7: Prompt instructions for interaction I of the mixture of agreement and tense sequence problems with ChatGPT 3.5. The response and other interactions are shown in the appendix. See text for a discussion.
The cook melted the hutter voluntarily. This sentence states the cook caused the butter melt.
Sentence 5:
The butter melted slowly. This sentence describes the process of the butter melting.
Sentence 6:
Sentence 7:
The butter was hard. This sentence states the initial state of the butter, which hard.
vincing. when explicitly asked to formulate the rule, ChatGPT correctly identifies it as the sequence of tense rule. It insists, however, in finding semantic sequentiality, which is, instead, not relevant. For the mixture of the two problems, agreement and sequence of tenses, we see that ChatGP13.5 cannot even converge on a single answer. This is because, while it understands reasonably well the relevance of the sequence of tenses, it misses the agreement pattern completely, and so it cannot eliminate many incorrect alternative answers that depend recognising this pattern.
Consider the answer choices: Now let's analyze each answer choice in relation to the context:
Answer The butter melted. This answer suggests that the current state the butter melted. However, doesn't It directly relate the given context. We don't have any information about the current state the but- ter.
Answer B: The boy is open. This answer doesn't It make logical sense as it equates the boy with being open, which is unrelated to the context.
Answer C: The door was open. This answer corresponds the progression the door's state as mentioned the context. It follows the pattern established sentences and 3, where door opens suddenly after being closed. This seems to a reasonable continuation.
5.4 Discussion
Answer D: The cook is melted. This answer equates the cook with being melted, which doesn't 't make sense in the context.
If we map impressionistically ChatGPT's solutions to the three areas in which people find difficulty (findings correspondences, goal and subgoal man- agement and item novelty), we notice that ChatGPT is quite good at correspondence finding, but only if the problem encodes a single grammatical rule. Otherwise, it does not manage to identify the two patterns in the mixture matrix. Also, it clearly per- forms better with lexical semantic problems than morphological problems. With quite a bit of guid- ance, it seems to be able to identify abstract template from lexically varied items (item novelty). However, it never manages identify the sequence pattern, and consequently find the answer in any of the problems: this seems to be indicative of lack of goal and subgoal management.
The butter melted. This sponds the progression of the butter's state men- tioned in the context. follows pattern estahlished in sentences and where the cook causes the butter melt slowly. This also seems like plausible continu- ation.
Answer E:
corre-
Final decision: Comparing answers C and F, both seem to be reasonable continuations based on the given con- text. However, since the initial prompt requests choos- ing only answer, need make decision. Con- sidering the sequential pattern of events in the context, seems more likely that the door's state would he the next relevant topic. Therefore, the correct answer C: The door was open.
Figure 6: Reply to interaction of the causative al- ternation problem with ChatGPT 3.5. See text for a discussion.

---

6 Quantitative Analysis
CORRECTLY IDENTIFIED BLEMENTS
Caus
Subj Trans Obj Transi. Subj Trans2, Obj Trans2
The interaction illustrated in the previous section are a limited sample. We also developed a con- trolled sample of interactions, and analyse it quan- titatively. We use ChatGPT-4 (between the 28th and the 30th of September, 2023).
Agr
NPI, NP2, NP3.
SoT
Premise, Consequent
CORRECTLY IDENTIFIED ATTRIBUTES
Caus
Volition, Ext. Agent, CoStia CoS:Current Agent, Patient
Agr
Sing, Plural, Mismatch
6.1 Sample creation and quantitative measures
SoT
Tense Person, Gender, Match, Mismatch
GOAL MANAGEMENT
The dimensions of variation of the sample reflect the properties we need to analyse. We test two sim- ple BLMs (Caus, Agr) and the mixed BLMs that merges Agreement and sequence of tense patterns (Mix). Each type of BLM has two versions, exhibit- ing two levels of lexicalisation (what in other work on BLMs has been called type and a type III). Type lexicalisation uses the same lexical items across all sentences in the matrix (see for example Figure 7), while the type III has complete lexical variability (see for example Figure 4).
All
One-at-a- Time, Pairwise, Row/Col-wise, Unordered
FINDING ANSWERS
All
Rulc-based generation, generraton,nnnntiiinon,,, Answer climination
Figure 8: Elements and attributes and pobleng-slviing procedures. Mixed problems are the union of the ele- ments and attributes and intersection of problem-solving procedures.
treme case being the causative alternation, which has been identified as such only in one case.'
For each combination of BLM simple problem and lexicalisation level we create five instances, built aemi-auttoattiallyy by transformation of a prototype with ChatGPT4 and then corrected or modified. Mixed BLMs where composed by hand, Some examples are shown in Appendix F.
Correspondence finding The number of cor- rectly identified elements is relatively high, ranging from 73 to 89 percent, confirming that ChatGPT4 (like 3.5) is good at correspondence finding: iden- tifying the aspect of the problem it needs to pay attention to. Comparing the ranking, best for Agr and worse for Mix, to the previous table of scores of correct answers, it can be noticed that good iden- tification of the objects does not guarantee propor- tionally good finding of the correct answer.
The coding protocol is developed for compara- bility with the quantitative analyses developed for human subjects (Carpenter et al.. 1990). We follow the same dimensions of evaluation (also discussed in the previous section): correspondence finding (elements and attributes), goal and subgoal manage- ment (timing and organisation of mentions, method used to find answers) and item novelty (type and type III levels of lexical variation). The exact cod- ing scheme corresponding these concepts varies for each BLM problem, and is shown in Figure
The percentage of correctly identified attributes is lower, showing that this aspect of the prob- lem is harder to identify, and particularly hard for causative BLMs. This is actually not surprising, as the attributes of Caus-BLM are more abstract, while Agr and Mix are much visible morphologi- cally. This aspect of correspondence finding is a better predictor of overall performance.
6.2 Quantitative results
The output were then manually analysed accord- ing to the categories shown in Figure 8. In Table 1, we report the summaries of the results and re- lated statistics. The complete results tables are shown in Appendix E. The complete outputs for all problems provided at https://github. ccm/ com/ -tt.eeevaeBLL-CCaaaatttttttrrree
Item novelty The sensitivity to item novelty re- sults from comparisons of BLMs of different levels of lexicalisation (type I, type III). The sensitivity to more fewer new lexical items seems to vary by problem. It appears be, in general, true that more variation in lexicalisation creates difficulty.
'Notiee that, in coding answers, we gave a full credit for only partial identification of pattern. For example, we have coded correct identification of the agreement pattern in most cases when in fact only the subject alternation pattern was identified but the two-step patterns for the second NP was not identified (which was in most cases).
Success rate The success rate is quite good, with a high rate of correctly identified answers. Nonethe- less, the answers that are correct because the right pattern has been identified are fewer, with the ex-

---

SUCCESS RATE
Further Related Work
Correct answers
Correct patierns
Total answers
Traditionally, progressive matrices as intelligence tests are designed by hand, but recent research in vi- sion that has used this task to train neural networks has typically employed some structured genera- tive model to create a large number of questions. Wang and Su (2015) generate RPMs using first- order logic, which have further inspired Barrett et al. (2018a) to ellaiin-objecc--ttiibe instantiations. Zhang et (2019) use the Attribtte Stochastic Image Grammar (A-SIG, proposed by Zhu and Mumford, 2006) as the representation of RPMs, and have created the Relational and Ana- logical Visual rEasoNing (RAVEN) dataset. The controlled construction of the context data and of the contrastive answer set is crucial. In this way, the learning process basically discovers how to in- duce the model and it makes it possible identify clear dimensions of successful and unsuccessful generalisation. For example, matrices for vision have shown that the best models can apply known abstract relationships in novel combinations, but fail in applying known abstract relationships to un- familiar entities (Barrett et al., 201 8b).
Caus
11
Agr
8
9
12
Mix
9
6
10
CORRESPONDENCE FINDING
Identified elements (%)
Tot
Identified
Tou
attributes(5k) (%)
Caus
56 (85)
66
20 (30)
66
Agr
43 (89)
48
42 (87)
48
Mix
44 (73)
60
72 (80)
90
triple
Tot
143
174
134
204
GOAL MANAGEMENT
Organisation Mentions
Finding Answers
Tot
OaT
Prw
RCw
Un
RbG
Ael
Caus
1
10
10
0
10
11
11
Agr
9
4
2
11
12
12
Mix
5
5
10
0
6
9
10
Tot
15
18
24
2
27
32
Table I: Quantitative results of BLM solutions (Id'ed
els: identified elements; Id'et attrs: identified attributes;
OаT:
One-at-a-Time; Prw:Pairwise; RCw: Row/Col-
wise: Un: Unordered; RbG: Rule-b generation; Ael: Answer elimination).
More specifically, causatives BLM are quite dif- ferent in type and type III because the elements that alternate are no longer the same NPs. Yet, the results are very similar, so we do not notice a difference in complexity. Agreement BLMs show correct answer for Type I in all 6 cases, but only 2 for type III. So lexicalisation does indeed make the task harder. Also in the Mixed BLMs, type III is harder.
8 A Roadmap for the Future
The investigation reported here aims prove cept: curated data and in-depth qualitative analyses inspired by human intelligence tests can comple- ment large-scale benchmarking in an interesting way. In particular, large-scale benchmarking based on correlations of humans and machines does not enlighten us on the causal sources of behaviours, and instead hypothesis testing experiments need to be performed. It is also interesting to collect data about human and machine overt knowledge, rather than phychophysical measures (eye-tracking, brain activations) or internal ANN layer representations. A larger-scale investigation applying the curated data and qualitative method presented here is how- ever also possible. We need to develop multiple datasets and run several interactive experiments that need then to be measured and assessed by proper statistical methods.
Goal Management The quantification of aspects of the reasoning employed to solve the problem show that the categories developed for humans ap- ply easily here too, the output of ChatGPT4 is coherent, articulate and corresponds well to the expectations. Items are inspected comparatively more often than one at a time and, depending on the problem, what are formulated as attempts to find patterns gives rise to pairwise rules. The in- spection of sentences is row by row or by groups of rows and rarely unordered. Of the two methods of rule finding documented in humans, rule-based hypothesis followed by matching with answers and elimination of the incorrect ones is the most com- mon. Only in few cases, in the clearly harder mixed BLMs, no rules were hypothesised and only directly answer elimination was attempted.
9 Acknowledgements
We gratefully acknowledge the partial support of this work by the Swiss National Science Foun- dation, through grants #51NF40 180888 (NCCR Evolving Language) and SNF Advanced grant TMAG-1 209426 to PMM

---

10
Ethics Statement
Kristina Gulordava, Piotr Bojanowski. Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics; Human Language Technologies, Volume 1 (Long Papers), pages 1 1195-1205. Association for Computational Linguistics.
To the best of our knowledge, there are no ethics concerns with this paper.
11
Limitations
Our preliminary conclusions about human--achiin correspondence sketch the avenue for future devel- opments. The work presented here is meant as a programmatic call. The main limitations con- cern limitations in the BLM development, in the range of examined architectures and the level of internal analysis, and in the range of cognitive con- cepts investigated. BLMs are being developed with larger-scale multi-linguality and greater variety of linguistic phenomena. Work is also planned to study how they affect internal layers of LLMs.
Dicuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Chris- tos Christodoulopoulls,, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2022. Statc-of-the-ae gen- eralisation research in NLP: a taxonomy and review. CoRR.
Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito. Marco Marelli, Marco Baroni, and Stanislas Dehaene. 2021. Mechanisms for handling nested dependen- cies in neural-network language models and humans. Cognition.
References
Yair Lakretz, German Kruszewski, Theo Desbordes, Dicuwke Hupkes, Stanislas Dehaene, and Marco Ba- roni. 2019, The emergence of number and syntax units in LSTM language models. arXiv preprint arXiv: 10X
Aixiu An, Chunyang Jiang, Maria A. Rodriguez, Vivi Nastase, and Paola Merlo. 2023. BLM-AgrF: A new French benchmark to investigate generalization of agreement in neural networks. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 1363- 1374, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Tal I Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax- sensitive dependencies. Transactions of the Associa- tion for Computational Linguistics, 4:521-535.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. 2018a. Measuring abstract reasoning in neural networks. In Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmüssun. Stockholm. Sweden, July 10-15, 2018, volume 80 of Proceedings of Ma- chine Learning Research, pages 4477-4486. PMLR.
Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective. arXiv cs. CI. 2301.06022
Paola Merlo. 2023. Blackbird language matrices (blm), new task for rule-like generalization in neural net- works: Motivations and formal specifications. ArXiv, Cs.CL 2306.1 1444.
David G.T. Barrett, Felix Hill, Adam Santoro, Ari S. Morcos, and Timothy Lillicrap. 2018b. Measuring abstract reasoning in neural networks. In Proceed- ings of the 35th International Conference on Machine Learning, Stockholm, Sweden.
Paola Merlo, Aixiu An, and Maria A. Rodriguez. 2022. Blackbird's language matrices (BLMs): new bench- mark to investigate disentangled generalisation neural networks. ArXiv, Cs.CL 2205.10866
Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 1798- 1828.
Paola Merlo, Chunyang Jiang, Giuseppe Samo, and Vivi Nastase. 2023a. Blackbird language matrices tasks for generalization. In GenBench: The first workshop on generalisation (benchmarking) in NLP.
Patricia A. Carpenter, Marcel A. Just, and Peter Shell. 1990. What one intelligence test measures: A theoret- ical account of the processing in the Raven Progres- sive Matrices Test. Psychological Review, 97(3):404- 431.
Paola Merlo, Giuseppe Samo, Vivi Nastase, and Chun- yang Jiang. 2023b. Building structured synthetic datasets: The case of Blackbird Language Matrices (BL.Ms). In Ninth Itaian Conference on Computa- tional Linguistics.
Stanislas Dehaene, Fosca Al Roumi, Yair Lakretz, Samuel Planton, and Mathias Maylé- 2022. Symbols and mental programs: a hypothesis about human singularity. Trends in Cognitive Sciences, 26(9):751-766.
John C. Raven. 1938. Standardization of progressive matrices. British Journal of Medical Psychology. 19:137-150.

---

Anna Rogers. 2021. Changing the world by changing the data. In Proceedings of the 59th Anmual Meet- ing of the A Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 2182-2194 Online. Association for Computa- tional Linguisties.
Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko. Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hup- kes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, E'katerina Shutova, Ekin Dogus Cubuk, Elad Se- gal, Fleanor Hagerman, Elizabeth Bames, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang. Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozh- skii, Fanyue Xia, Fatemeh Siar, Fernando Martínez- Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Paras- candolo, Giorgio Mariani, Gloria Wang. Gonzalo Jaimovitch-López Gregor Betz, Guy Gur-Ari, Hana Galijasevic. Hannah Kim, Hannah Rashkin, Han- naneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hong- ming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Ja- cob Hilton, Jaehoon Lee, Jaime Fernández Fisac. James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoñ, Jana Thompson, Janelle Wing- field, Jared Kaplan, Jarema Radom, Jascha Sohl- Dickstein, Jason Phang, Jason Wei. Jason Yosin- ski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Je- sujoba Alabi. Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Be- rant, Jörg Frohberg, Jos Rozen, Jose Hernandez- Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua. Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishman, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi. Kory Mathewson, Kristen Chiafullo, Kse- nia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang. Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando. Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütti Kerem Şenel. Maarten Bosma, Maarten Sap. Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giu- lianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Med- ina Orduna Baitemirova, Melody Amaud, Melvin McBlrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swedrowski. Michele Bevilacqua, Michihiro Yasunaga Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mo- hit Bansal, Moin Aminnaseri. Mor Geva, Mozhdeh Gheini. Mukund Varma T, Nanyun Peng, Nathan A. Chi, Naycon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S.
Mathias Saylé-Meyer, Joël Fagot, Serge Caparos, Timo van Kerkoerle, Marie Amalric, and Stanislas De- haene. 2021. Sensitivity to geometric shape regu- larity in humans and baboons: A putative signature of human singularity. Proceedings of the National Academy of Sciences, 118(16).
Giuseppe Samo, Vivi Nastase, Chunyang Jiang, and Paola Merlo. 2023. BLM-s/IE: A structured dataset of English spray-load verb alternations for testing generalization in LLMss In Findings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing.
Martin Schrimpf. Jonas Kubilius, Michael J. Lee, N. Apurva Natan Murty, Robert Ajemian, and James J. Di Carlo. 2020. Integrative benchmarking to ad- vance neurally mechanistic models of human intelli- gence. Neuron, 108(3).
Bernhard Schölkopf. 2019. Causality for machine Ieamn ing. Technical report,
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb. Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Bouriga-Alonso, Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- rish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmülller, An- drew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong. Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabas- sum, Arul Menezes, Arun Kirubarajam, Asher Mul- lokandov, Ashish Sabharwal, Austin Herrick. Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loc, Barret Zoph, Bartiomicj Bojanowski, Batuhan Ozyurt, Behnam Hedayatmia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao. Cameron Dour, Cather- ine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burn Burch, Chris Waites, Christian Voigt, Christopher D. Manning. Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro. Colin Raffel, Courtney Ashcraft, Cristina Garbacea. Damien Sileo. Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Free- man, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez. Danqi Chen, Daphne Ippolito, Dar Gilboa, David

---

Iyer, Noah Constant, Noah Fiedel, Nuan Wen. Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Liang, Paul Vi- col, Pegah Alipoorollbbshii, Peiyuan Liao, Percy Liang, Peter Chang. Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu. Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymackers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhut- dinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang. Sahib Singh, Saif M. Moham- mad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel Bow- man, Samuel S. Schoenbolz, Sanghyun Han, San- jeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sehastian Bischoff, Sebastian Gehrmann, Sebastian Schuster. Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixi- ang Shane Gu, Shubh Pachchigar, Shubham Tosh- niwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas De- haene, Stefan Divie, Stefano Ermon, Stella Bider- man, Stephanie Lin, Stephen Prasad, Steven T. Pi- antadosi, Stuart M. Shieber, Summer Misherghi. Svet- lana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang. Tiherius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Ger- stenberg. Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmaku- mar. Vivek Srikumar, William Fedus, William Saun- ders, William Zhang, Wout Vossen, Xiang Ren, Xi- aoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zi- jian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
Ganguli, Jeff Hawkins, Konrad Koerding, Alexei Koulakov, Yann LeCun, Timothy Lillicrap, Adam Marblestone, Bruno Olshausen, Alexandre Pouget, Cristina Savin, Terrence Sejnowski, Eero Simoncelli, Sara Solla, David Sussillo, Andreas S. Tolias, and Doris Tsao. 2023. Toward next- generation artificial intelligence: Catalyzing the neuroai revolution. Na- ture Communications, 14(1597).
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019. RAVEN: A dataset for lational and analogical visual reasoning. In IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 5317-5327. Computer Vision Founda- tion / IEEE.
Song-Chur Chng-CCunn& Zhu and David Mumford. 2006. A stochas- tic grammar of images. Found. Trends Comput. Graph. Vis., 2(4):259- 362.
Ke Wang and Zhendong Su. 2015. Automatic genera- tion of raven's progressive matrices. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pages 903-909, AAAI Press.
Anthony Zador, Sean Escola, Blake Richards, Bence Ölveczky, Yoshua Bengio, Kwabena Boahen, Matthew Botvinick, Dmitri Chklovskii, Anne Churchland, Claudia Clopath, James DiCarlo, Surya

---

A
Agreement
N.B. The interactions formatted for readability. In the prompt, provide the context and examples only for those interactions where they change with respect the previous interaction. ChatGPT repeats context and answer literally every interaction. For readability we have introduced ellipses, indicated with (...).
A.1
Interaction
am going to ask you to solve a puzzle called BLM-Agrl. language experiment Italian. will give you sentences, that will the context, and will also give you set of sentences that will the answers. will you choose among the answer sentences the sentence that could next sentence in context sentences. am asking you to give me correct answer Italian but also describe in English step by step the hypotheses you are entertaining while you are solving problem.
Context
computer col programin funziona.
computer col programma funzionano.
computer coi programmi computer coi programmi computer col programua dell' esperimento funziona.
funziona.
funzionano.
computer col programma dell' esperimento funzionano.
computer programmi dell "speriiiri funziona.
Answers
Il computer col programuma I'esperimento non funziona.
I computer coi programmi dell' esperimento non funzionano.
Il computer col programua non funziona.
computer coi programmi dell' esperimento non funzionano.
I computer col programuaa dell' esperimento non funziona.
I computer coi programmi degliesperiment non funzionano.
ChatGPT
To solve the puzzle, will analyze the given context and the set of answer sentences to identify the sentence that could be the next one in the context. Here's 'sre's the step-hy-step thought process:
start by examining the context sentences: (...) observe that the context consistently mentions computer(s) and program(s), along with their functionality (funzionare or non funzionare). The addition of "sserrimit" indicates that the computers and programs are related to an experiment.
Moving on the answer sentences: (...) Comparing the answer sentences with the context, need to find sentence that logically and syntactically. should mention the computer(s), program(s), the experiment.
The answer sentence "Il computer col programma d'esperimento funziona" doesn't match the context because it the singular form "funziona" instead of the plural form "funzionano."
The answer sentence "I computer coi programmi dell' eseeressero non funzionano" is consistent with the context. It maintains the plural form and correctly references the experiment.
The answer sentence "Il computer col programma non funziona" matches the previous context sentence but doesn't include the experiment aspect.
The answer sentence computer coi programmi esperimento non funzionano" aligns with context by using the plural form mentioning the experiment.
The answer sentence "I computer col programma dell' esperimento funziona" has disagrcement number. "I computer" is plural, while "non funziona" is singular.
The answer sentence "I computer coi programmi degli esperimenti non funzionano" incorrectly uses the singular form "non funziona" instead of the plural form "non funzionano."
Based on this analysis, the answer sentence that could be the next in the context is "I computer coi programmi dell' csperimtti non funzionano."
Correct answer in Italian: computer coi programmi dell' esperimento non funzionamo."

---

A.2 Interaction 2
This interaction minimal of interaction ChatGPT understands that attractors, noun phrases intervening between subject and verb, can be decreasing in number, depending on structure the matrix.
I am going to ask you to solve puzzle called BLM-Agrl.
The language of the experiment is Italian.
will give you a list of sentences, that will call the context, and I will also give you set of sentences that I will call the answers. will ask you to chnose among the answer sentences the sentence that could be the next sentence in the context list of sentences. am asking you to give me the correct answer in Italian but also to describe in English step by step the hypotheses you are entertaining while are solving the problem.
Context
computer col programma dell' esserrmento non funziona. computer col programma dell'es eeseerieeeeto non funzionano. computer coi programmi dell' 'esperimento non funziona.
computer coi programmi dell' esperimento non funzionano. computer col programma non funziona.
computer col programma non funzionano.
computer coi programmi non funziona.
Answers
computer programmi funzionano.
I computer coi programmi funzionano.
computer col programma funziona.
computer programmi dell' esperimento non funzionano. computer col programmadell' esperimento funziona.
computer programmi degli esperimenti funzionano.
ChatGPT
To solve the puzzle, will analyze the given context and the set of answer sentences identify the sentence that could be the in the context. Here's 's the step-by-step thought process: I start examining the context sentences: (...)
observe that the context consistently mentions computer(s) and program(s), along with their functionality (funzionare
funzionare). The addition of "dell' esperimento" indicates that the computers and programs related to experiment.
Moving the sentences with the context, I need find sentence that fits logically and syntactically. It should mention Comparing the the computer(s), program(s). and experiment.
sentences: (...)
The answer sentence "I computer programmi funzionano" grammatically correct and maintains the plural form for both "computer" and "programmi."
The answer sentence "I computer coi programmi non funzionano" matches context by using correct maintaining the plural form for both "computer" and "programmi." The answer sentence computer col programma non funzionano" disagreement number. computer" singular, while "non funzionano" is plural.
context. It maintains The answer sentence "I computer col programma dell' esperimento non funziona" has a disagreement in number. computer" is plural, while "non funziona" is singular.
The answer sentence computer programmi dell' esperieeenttoo non funzionano" is consistent the plural form and correctly references experiment.
The answer sentence "I computer coi programmi degli esperimenti non funzionano" incorrectly uses the plural form "degli esperimenti" instead of the singular form "dell' esperimento" mentioned in the context.
Based on this analysis, the answer sentence that could be the next in the context "I computer e programmi non funzionano." Correct answer in Italian: "I computer e programmi non funzionano."

---

A.3 Interaction 3
We vary words if ChatGPT can group objects attributes in more abstract way. Here, vary words in each sentence (lype data). also number context sentences highlight the notion of list. also change the language from Italian to English.
I am going to ask you to solve a puzzle called BLM-AgrE.. will give you list of sentences (numhered from to 7), that will call the context, and will also give you a set of sentences (identified by capital letters) that I will call the answers. I will ask you to choose among the answer sentences the sentence that could be the next sentence in the context list of sentences. am asking you to give me the correct answer, but also to describe step by step the hypotheses you are entertaining while you are solving the problem. Context
The talk on history has started later than expected.
2 Those responsible for the right will resign.
3 The show with the paintings has met with great success.
The threats of reforms worry the doctors.
5 The bunch of keys of the cell sits the shelf.
6 The studies the effect of the drug will appear soon.
7 The threat of reforms in the school worries the doctors. Answers
A The tablecloths the table and the console shine in the
of the villa C The lawyers of the murderers will come back.
B The friends of the
sleeping on the beach.
D The lawyers of the murderers of the village will come back.
E The visit of the palace the crafts is upproaching.
F The computers with the program of the experiment hroken.
ChatGPT
To solve the puzzle, will analyze the given context sentences and the of answer sentences identify the sentence that could be the
the context. Here's 's the step-by-step thought process: start examining the context sentences: (...)
observe that the context sentences unrelated each other. discussing various topics such talk history, responsible
individuals resigning, successful show, doctors' concerns about reforms, keys on shelf, drug studies, and the threat of reforms the school.
Moving on to the Considering the context, I look an answer sentence that logically and thematically with the preceding sentences.
sentences: (...)
The answer sentence "The tablecloths the table and console shine in the sun" does relate any of the previous context sentences or their topics.
The answer sentence "The friends of the owners of the villa were sleeping on the beach" unrelated context and introduces a new scenario.
The answer sentence "The lawyers of the murderers will come back" is unrelated to the context and introduces a criminal element not previously mentioned.
The answer sentence "The lawyers of the murderers of the village will come back" introduces a new secnario involving murderers and village, which not relevant to the context.
The answer sentence "The visit of the palace of the crafts is approaching" is unrelated to the context and introduces a visit to palace, which has not been mentioned before.
The answer sentence "The computers with the program of the experiment are broken" is thematically consistent with the previous context sentence mentioning the studies on the effect of a drug and the experiments. introduces the concept of broken computers, which could be a logical continuation.
Based on this analysis, the answer sentence that could be the next in the context is "The computers with the program of the experiment are broken."
Correct answer: F) The computers with the program of the experiment are broken.

---

A.4 Interaction 4
Like interaction we vary all words see ChatGPT can group objects and attributes a more abstract way. However, since the previous interactions have shown that ChatGPT deals with the sentences based topics and network of concepts and tries solve puzzle as logical or semantic problem, we give more information on the fact that needs focus the syntactic form the sentences. The context answer sentences are same as those interaction
instruction is slightly different, we no longer talk puzzle, but gramnatical problem.
am going to ask you to solve a grammatical problem called BI BI-MAg will give you a list of sentences (numbered from to 7), that I will call the context, and I will also give you set of sentences (identified by capital letters) that I will call the answers. will ask you to choose among the answer sentences the sentence that could be the next sentence in the context list of sentences. am asking you to give me the correct answer, but also describe step by step the hypotheses you entertaining while you are solving the problem.
ChatGPT
To solve the grammatical problem, I will analyze the given context sentences and the set of answer sentences to identify the sentence that could be the next one in the context. Here's the step-by-step thought process: start by examining the context sentences: (...)
observe that the context sentences are unrelated to each other, discussing various topics such as talk on history, responsible
individuals resigning, successful art show, doctors' concerns about reforms, keys on shelf, drug studies, and the threat of reforms in the school.
Moving to the answer sentences: (...)
Considering the grammatical structure of the context, look for answer sentence that could continue the pattern of "Thh [noun phrase] [verb phrase]."
The answer sentence "The tahlecloths the table and the console shine in the sun" does not fit the grammatical pattern of the context sentences. The answer sentence "The friends of the owners of the villa were sleeping the beach" does not follow the grammatical structure of the context.
The answer sentence "The lawyers of the murderers will come back" does match the grammatical pattern of the context. The answer sentence "The lawyers of the murderers of the village will come back" does adhere the grammatical structure of the context.
The answer sentence "The visit the palace of the crafts is approaching" does conform to grammatical pattern of the context.
The answer sentence "The computers with program of the experiment are broken" matches the grammatical structure of the context, which consists "The [noun phrasel |verb phrasel." introduces the concept broken computers, which could logical continuation.
Based on this analysis, the answer sentence could next the context "The computers with the program the experiment are broken."
Correct answer: F) The computers with program of experiment are broken.

---

A.5 Interaction 5
This continuation interaction We still no longer puzzle but grammar rule and try to give more hints.
The answer is not correct but you are on the good path, can you try to solve the problem with a more complex structure for the context sentences?
CharGPT
Apologies for the previous incorrect answer. Let's approach the problem with more complex structure for the context sentences. Here's revised step-hy-step thought process:
Re Re-examining the context sentences with more complex structure: (...)
Analyzing the context sentences, observe that they all follow pattern of "The [noun phrase] [verb phrase] [adverbial phrase]." Moving on to the answer sentences: (...)
Considering the revised grammatical structure of the context. look for an answer sentence that adheres the pattern of "The [noun phrase] [verb phrase] [adverbial phrase]."
The answer sentence "The tablecloths on the table and the console shine the sun" matches the grammatical structure of the context and introduces the idea of shining the sun, which he seen as continuation.
The answer sentence "The friends the owners of the villa were sleeping on the beach" does not align with the grammatical structure the context.
The answer sentence "The lawyers of the murderers will come back" does not match the grammatical structure of the context. The answer sentence "The lawyers of the murderers of village will come back" does not conform to the grammatical structure the context.
The answer sentence "The visit of the palace of crafts is approaching" does not adhere to the grammatical pattern of the context.
The answer sentence "The computers with the program of the experiment are broken" does not follow revised grammatical structure the context.
Based on this analysis, the answer sentence that could be the next in the context, considering the revised grammatical structure, is "The tablecloths on the table and the console shine in the sun," Correct answer: A) The tablecloths on the table and the console shine in the sun.
A.6 Interaction 6
ok, not quite there yet, me tell you see if you can discover the correct rules. The property you need to pay attention is number agreement.
ChatGPT
Thank for the additional clarification. Let's consider number agreement as key property to find the correct answer. Here's the revised step-by-step thought process:
Revisiting the context sentences: (...)
Analyzing context sentences, notice that they all use the definite article "The" followed by a noun phrase and a verb phrase.
Moving on to the answer sentences: (...)
Considering property number agreement, need to ensure that the noun phrase the answer sentence matches number (singular or plural) used in context sentences.
The answer sentence "The tablecloths on the table and the console shine in the sun" does not match the number agreement with the singular noun phrases used in context sentences. The noun phrase "tablecloths on the table and the console" plural. while context sentences use singular noun phrases.
The answer sentence "The friends of the owners of the villa were sleeping on the beach" also does not match the number agreement with the singular noun phrases used the context sentences. The verb "were" indicates a plural subject, but the context sentences use singular subjects.
The answer sentence "The lawyers of the murderers will come back" does not match the numher agreement with the singular noun phrases used in the context sentences. The noun phrase "lawyers of the murderers" is plural, while the context sentences use singular subjects.
The answer sentence "The lawyers of the murderers of the village will come back" also does not match the number agreement with the singular noun phrases used in the context sentences. The noun phrase "lawyers of the murderers of the village" is plural. while the context sentences use singular subjects.
The answer sentence "The visit of the palace of the crafts is approaching" matches the number agreement with the singular noun phrases used in the context sentences. The noun phrase "visit of the palace of the crafts" is singular, matching the context sentences.
The answer sentence "The computers with the program of the experiment are broken" matches the number agreement with the plural noun phrases used in the context sentenees. The noun phrase "computers with the program of the experiment" is plural, matching the context sentences.
Based on this analysis, considering the property of number agreement, the answer sentence that fits the context is 'The computers with the program of the experiment are broken." Correct answer: F) The computers with the program of the experiment are broken.

---

A.7
Interaction 7
Not quite, you need find the continuing sentence based pattern of number agreement.
ChatGPT
Apologics for the oversight. Let's revisit the context and analyze the sentences while paying attention to number agreement as the determining factor for finding the correct answer. Here's 's the revised step-by-stee step thought process:
Revisiting the context sentences: (...)
Analyzing the context sentences, notice that the noun phrases in each sentence agree in number with the corresponding verb
phrases. In other words, singular noun phrases are paired with singular verbs, and plural noun phrases are paired with plural verbs.
Moving on to the answer sentences: (...)
Considering the property of number agreement, I need to find an answer sentence where the noun phrase agrees in number with the verb phrase, similar to the pattern seen in the context sentences.
Answer sentences A, B, C, and D do not demonstrate number agreement with the context sentences.
Answer sentence E, "The visit of the palace of the crafts is approaching." maintains number agreement with the context sentences,
The noun phrase "the visit of the palace of the crafts" is singular, matching the singular verb "is approaching" used in the context sentences.
Answer sentence F. "The computers with the program of the experiment are broken," also demonstrates number agreement with the context sentences. The noun phrase "the computers with the program of the experiment" is plural, matching the plural verb "are broken" used in the context sentenees.
Comparing the two remaining answer sentences, both E and F demonstrate number agreement with the context sentences. However, based on the context and the potential continuation of the sentences, it is more likely that the next sentence would relate to the topic of the computers and the program.
Based on this analysis, considering the pattern of number agreement and the context, the answer sentence that best fits the continuation is:
Correct answer: The computers with the program of the experiment broken.
A.8
Interaction 8
Let me explain better. You need to find the continuing sentence based on pattern of number agreement. The context gives you pattern within each sentence but also across all sentences, that the continuation sentence continues the glohal pattern of agreement as well obeying the sentence-intermal pattern of agreement.
ChatGPT
Thank you for the clarification. apologize for the misunderstandingg Let's revisit the context and analyze the sentences while considering both the eettence-interaal pattern of number agreement and the global pattern of agreement. Here's the revised step-by-step thought process: Revisiting the context sentences: (...)
Analyzing the context sentences, I observe the following patterns of number agreement:
Sentences 1, 3, 5, and have singular noun phrases that agree with singular verbs. Sentences 2, 4, and 7 have plural noun phrases that agrec with plural verbs.
Moving to the answer sentences: (...)
Considering both the sentence-internal pattern of number agreement and the global pattern observed in the context sentences, need to find an Answer sentences A, B, C, and D do continue the global pattern of number agreement observed in the context sentences. Answer sentence E. "The visit of the palace of the crafts is approaching." continues the sentence-interan pattern of number agreement by using a singular noun phrase and singular verb. However, does continue the global pattern of piural noun phrases and plural verhs observed the context sentences.
sentence that continues the overall pattern of numbee agreement.
Answer sentence F. "The computers with the program of the experiment broken." " continues both the centence-interna pattern number agreement (plural noun phrase with plural verb) and the global pattern plural noun phrases and plural verbs observed the context sentences.
Considering the requirement to continue both the sentence-internal pattern and the global pattern of number agreement, answer sentence F aligns with the desired continuation.
Based on this analysis, the answer sentence that follows the pattern of number agreement both within each sentence across sentences is:
Correct answer: F) The computers with the program of the experiment are broken.

---

A.9 Interaction 9
We give indication that grammatical pattern be found without saying which one. We switch Italian, where gramamtical number is more salient morphologically.
I am going to ask you to solve grammatical problem called BLM-AgrI. will give you a list of sentences (numbered from to 7), that will call the context, and I will also give you set of sentences (identified by capital letters) that will call the answers. I will ask you to choose among the answer sentences the sentence that could he the next sentence in the context list of sentenees.
You need to find the continuing sentence based on grammatical pattern. The context gives pattem within each sentence but also across all sentences, so that the continuation sentence continues the global pattern as well as obeying the sentence-internal pattern of agreement.
am asking you to give me the correct answer, but also describe step by step the hypotheses entertaining while you are solving the problem. Context
La presentazione di storia comincia in ritardo.
2 responsahili della legge daranno dimissioni.
3 La mostra dei quadri ha incontrato grande successo.
4 Le minaccie di riforme preoccupano dottori.
5 1l mazzo di chiavi della cantina pende dal gancio.
6 Gli studi sugli effetti del farmaco appariranno presto. frequenza di riforme nelle scuole prevccupa dottori. Answers
A Le lovaglie sul tavolo la credenza brillano al sole.
B Gli amici dei proprictari della villa dormono sulla spiaggia.
C Gli avvocati degli assassini tomeranno.
D Gli avvocati degli assassini del villaggio tomneanno..
E La visita palazzo delle arti avvicina.
computer programma dell' esperimento non funzionano.
Chat GPT
To solve the BLM-Agrl problem, I will carefully analyze the given context sentences and the provided answer options. I will consider both the sentence-internal pattern of agrcement and the global pattern all sentences. Here the step-hy-step thought process:
Revisiting
sentences Italian: (...)
Analyzing context sentences. I observe the following patterns:
Sentences 1, 3, and have singular noun phrase (I.a presentazione, La mostra, mazzo, Gli studi) paired with singular phrrsse (comincia, ha incontrato, pende, appariranno). Sentences 2, 4, 7 have a plural noun phrase responsabili, minaccie, La frequenza) paired with plural phrase (daranno, preoccupano).
Moving on to the answer sentences: (...)
Considering both the sentence-internal pattern agreement and the glohal pattern observed in context sentences, need to find an answer sentence that continues both the sentence-interme and global patterns.
Answer sentences A, B. C, D, and E. do not continue global pattern observed in context sentences.
Answer sentence F, "I computer col programma dell' esperimento non funzionano," continues the sentence- inteencceinterall pattern of agreement (plural noun phrase with plural verb) observed in the context sentences. It also maintains the global pattern of plural noun phrases (I computer col programma dell' esperimento) and plural verbs (non funzionano).
Considering the requirement to continue both the sentence-intermul and global patterns of agreement, answer sentence F aligns with the desired continuation.
Based on this analysis, the answer sentence that follows the grammatical pattern both within each sentence and across all sentences is:
Correct answer: F) computer col programma dell' esperimentto non funzionano.

---

A.10
Interaction 10
Good joh, but you made mistake and then the conclusion was consequently wrong. If tell you that "Gli studi' is plural can you retry?
ChaiGPT
Apologies for the mistake. Thank you for pointing out that "Gli studi" in sentence 6 is indeed plural. Let me revise my analysis based on that information:
Revisiting the context sentences in Italian: (...)
Analyzing the context sentences, I observe the following patterns:
Sentences 1, 3, 5, and 7 have singular noun phrase paired with singular verb phrase.
Sentences 2, and have plural noun phrase paired with a plural verb phrase. Moving the answer sentences: (...)
Considering the patterns observed in the context sentences and the requirement continue both the sentence-internal and global patterns of agreement:
continue the global pattern of number agreement observed in the context sentences. Answer sentence F, computer col programma dell' esperimento non funzionano," aligns with the sentence-internal pattern (plural noun phrase with plural verb) observed the context sentences. However, it does not follow the global pattem of numher agreement because all previous sentences have singular subject paired with singular verb phrase. Therefore, considering grammatical pattern both within each sentence across all sentences. Apologies confusion the previous response.
Answer sentences B, C, D, and E
updated information. none
answer sentences provided (A, C. D. F) continue the
A.11
Interaction 11
Not quite, let me help you. The correct answer Gli amici dei proprictari della villa dormono sulla spiaggia. Keeping in mind the information gave you before about sentence-intermal patterns and global patterns, can you explain to me why?
ChatGPT
Apnlogies for the confusion. Let me reevaluate the context and analyze the patterns to explain why the correct answer "Gli amici dei proprietari della villa dormono sulla spiaggia." Revisiting the context sentenees: (...)
Analyzing the patterns:
In sentences 1, 3, 5. and 7, we observe a singular noun phrase paired with singular verb phrase. In sentences 2, 4, and 6, we observe a plural noun phrase paired with a plural verb phrase. Considering the answer sentences: (...)
The correct answer: Answer sentence B, "Gli amici dei proprietari della villa dormono sulla spiaggia." aligns with both the contence-interaal pattern and the global pattern of number agreement observed in the context sentences.
Sentence 2 in the context has a plural subject "I responsabili della legge" paired with plural verb phrase "daranno le dimissioni." Answer sentence B follows the same pattern with plural subject "Gli amici dei proprictari della villa" paired with a plural verh phrase "dormono sulla spiaggia."
Therefore, based on the patterns observed, sentence B continues the grammatical pattern within each sentence and maintains the global pattern of number agreement, making it the correct answer.
A.12 Interaction 12
Can you tell me more explicitly what is the global pattern you have observed and why sentence B is the right continuation according to this patter??
ChatGPT
Certainly! Upon further analysis, the global pattern observed in the context sentences is the agreement of the subject noun
phrases with the corresponding verb phrases in terms of number (singular or plural). The global pattern can be summarized follows:
Sentences 1. 3. 5. and 7 have singular subject noun phrases paired with singular verb phrases. Sentences 2, 4, and have plural subject noun phrases paired with plural verb phrases.
In the given context, the sentence "Gli studi sugli effetti del farmaco appariranno presto" (Sentence 6) estublishes the pattern of plural subject ("Gli studi") paired with plural verh phrase ("appariranno preste").
Now let's examine the provided answer sentences and which one aligns with this global pattern.
Answer sentence B, "Gli amici dei proprictari della villa dormono sulla spiaggia," continues the global pattern. It consists of plural subject noun phrase ('Gli amici dei proprietari della villa") paired with plural verb phrase ("dormono sulla spiaggia") "). This choice maintains the agreement of the subject noun phrase and the verb phrase, seen in the context sentence thus adhering the global pattern observed. Therefore, based on the global pattern of number agreement, sentence B the correct continuation in the context list of sentences.

---

A.13 Interaction 13
According your reasoning also and D should be correct because they also have plural subject matching plural verb. So why are they not the right answer?
ChatGPT
You'l re absolutely correct, and apologize for the oversight. Based the global pattern of numher agreement, both answer sentences C and D, "Gli avvocati degli assassini tomeranno" and "Gli avvocati degli assassini del villaggio toreraanno." " match the pattern of a plural subject ("Gli avvocati degli assassini") paired with plural verb phrase ("torneranno") Therefore, they also satisfy the global pattern observed in the context sentences. result, considering the information provided, global pattern of number agreement. seems there might not be unique correct answer solely based on the global pattern. To determine the specific correct answer, would be helpful to consider additional linguistic factors constraints that might be present in the puzzle. Without further information or specific instructions, is difficult definitively determine the correct answer solely based on the global pattern of number agreement.
sentences C. and D could all be valid continuations based the
A.14 Interaction 14
There is only one correct answer and that B, you correctly say, C and might be excluded by other patterns. Can you see what they are?
ChaiGPT
Apologies for the confusion earlier. Upon closer examination, there are indeed other linguistic patterns present in the context sentences that help exclude answer sentences C and D as the correct continuation. Let's analyze these pattern::
Sentence-Intermes Pattem: Each sentence in the context exhibits consistent sentence-internal pattem of number agreement between the subject phrase and the verb phrase. In sentences 3,5, and 7, we observe singular subject noun phrases paired with singular verb phrases. In sentences 2, 4, and
plural subject noun phrases paired with plural verb phrases.
Sentence-Exterme Pattern: The context sentences also form broader sentence-external pattern. We notice that the subject noun phrases in the context sentences refer different entities or groups, each with their own contextually specific characteristics. For example, in sentence "La presentazione di storia" refers singular della legge" refers group of individuals responsible for the law. Applying these putterns, degli assassini del villaggio torneranno," not fit the broader sentence-extent noun phrases in these answer sentences refer specific group (the lawyers of the murderers the lawyers of the murderers the village). However, this specific group not consistent theme or contextually relevant entity throughout the provious context sentences. Therefore, based on the sentence centence-extemal pattern, answer sentences and can be excluded the correct continuation.
(the talk history). In sentence "I responsabili
that answer sentences and D, "Gli avvocati degli assassini tomeranno"" and "Gli avvocati
pattern established in the context. The subject
In contrast, answer sentence "Gli amici dei proprietari della villa dormono sulla spiaggia," fits both sentence-internal pattern number agreement (plural subject paired with plural verb) and the broader sentence-extermal pattemn The subject noun phrase refers a group of individuals (the friends the owners the villa), which aligns with the contextual theme of different entities being discussed the previous sentences. Considering the sentence-internal and sentence-exterme exteraal patterns, answer sentence B is the correct continuation that maintains consistency and relevance within the given context.

---

A.15
Interaction 15
There is only correct and that is B, you correctly say, and D might be excluded by other patterns. Focussing exclusively patterns of agreement, can you what they are?
ChatGPT
apologize for the confusion earlier. Focusing exclusively patterns of agreement, let's analyze the sentences again to identify the specific patterns play:
La presentazione di storia comincia in ritardo. (Singular subject, singular verb)
responsabili della legge daranno dimissioni. (Plural subject. plural verb)
mostra dei quadri ha incontrato grande successo. (Singular subject, singular verb) minacce riforme preuecupano i dottori. (Plural subject, plural verb)
Il mazzo di chiavi della cantina pende dal gancio. (Singular subject, singular verb)
Gli studi sugli effetti del farmaco appariranno presto. (Plural subject, plural verb)
frequenza di riforme nelle scuole preoccupa dottori. (Plural subject, plural verb)
Based on these patterns agreement, we ohserve the following consistent rules:
Singular subjects are paired with singular verbs. Plural subjects are paired with plural verbs.
Applying these patterns the answer sentences, we find that answer B, "Gli amici dei proprietari della villa dormono sulla spiaggia." adheres to pattern of agreement. The subject "Gli amici" (plural) is appropriately matched with the verb "dormono" (plural), maintaining the pattern observed in context sentences.
contrast, answer sentences and D, "Gli avvocati degli assassini torneranno" and "Gli avvocati degli assassini del villaggio tomeranno," do not conform to the pattern of agreement. Although they have plural subjects ("Gli avvocati"), the verh "torerrnno"" (plural) does not match with the singular noun phrases (" ("degli assassini" and "del villaggio"). Therefore, hased solely on patterss of agreement, answer sentence B the correct continuation as aligns with the consistent pattern number agreement observed in the context sentences.
A.16 Interaction 16
Both C follow the rule you mention so why are you saying they follwo the rule?
ChatGPT
Apologies for the confusion in previous response. You are correct that both options C and D follow the pattern of plural subject-verb agreement. Therefore, they are potential valid answers based on the observed agrecment rule. To select the correct answer between C and D, we need to consider additional patterns or constraints within the context sentences. Since I do not have access to the specific global patterns or constraints you mentioned, I am unable provide definitive answer based solely grammatical agreement.
If there are other patterns constraints related to the context sentences that can help determine the correct answer, please provide further information. and I'll be glad to reassess and ussist you accordingly.
A.17 Interaction 17
The other constraints are also related to numher and agreement, can you see the patterns that climinate C. and D?
ChaiGPT
Apologies for the oversight. Let's reassess the context and the remaining options (C and D) considering other patterns related number and agreement.
Context: (...)
Answers: Gli avvocati degli assassini tornrannn..
Gli avvocati degli assassini del villaggio tomeranno.
identify patterns that eliminate C we need examine the agreement between the subjects and the verbs in each option.
option C, "Gli avvocati degli assassini tornerann"" (The lawyers the murderers will come back), the subject "Gli avvocati"
(The lawyers) is plural, and verb "tomeranno" (will come back) agrees with the plural subject.
option D. "Gli avvocati degli assussini villaggio tomeranno" (The lawyers the murderers the village will come back), the subject "Gli avvocati" (The lawyers) is plural, and the verb "torneranno" (will come back) also agrees with the plural subject. Based on the information provided, both options C and seem follow pattern of number agreement within the sentences. However, if there are additional patterns or constraints related to number and agreement that apply to the context sentences as whole, please provide further details so that can reassess and assist you more accurately.

---

B
Causatives
N.B. The interactions are formatted for readability. In the prompt. provide the context and answer examples only for those interactions where they change with respect the previous interaction.
B.1
Interaction
am going ask you to solve a grammatical problem called BLM-CausE. will give you list English sentences (numbeered from to 7), that will call context, and I will also give you a set English sentences (identified by capital letters) that will call answers. will ask you to choose among the answer sentences the one sentence that could be the next sentence in the context of sentences. The sentences have been generated a rule-based manner. am asking you to give me the correct answer, but also descrihe step by step the hypotheses you are entertaining while you are solving the problem.
Context
1 The boy opened the door voluntarily,
2 The door opened suddenly.
3 The door was closed.
4 The door is open.
5 The cook melted the butter voluntarily.
6 The butter melted slowly.
The butter was hard.
Answers
A The butter is melted.
B The boy is open.
C The door was open.
D The cook is melted.
E The butter was melted.
ChatGPT
To solve the the most appropriate continuation for the context. Let's go through the process step by step:
problem, we need to analyze the given context sentences and the set of answer sentences to determine
Identify the relevant elements:
In this context, we have two main elements: "the door" and "the butter." We need to observe the causality between these elements and their states.
Analyze the context:
Sentence I: The boy opened the door voluntarily. This sentence estahlishes that the boy was responsihle for opening the door.
Sentence 2: The door opened suddenly. This sentence suggests that the door opened on its own accord, without external agency.
Sentence 3: The door was closed. This sentence indicates the initial state of the door, which was closed.
Sentence 4: The door is open. This sentence reveals the current state of the door, which is open.
Sentence 5: The cook meited the butter voluntarily. This sentence states that the cook caused the butter to melt.
Sentence 6: The butter melted slowly. This sentence describes the process of the butter melting.
Sentence 7: The butter was hard. This sentence states the initial state of the butter, which was hard.
Consider the answer choices: Now let's analyze each answer choice in relation to the context: Answer A: The butter melted. This answer suggests that the current state of the butter is melted. However, it doesn't directly
relate to the given context. We don't 't have any information about the current state of the butter.
Answer B: The boy is open. This answer doesn't make logical sense it equates the boy with being open, which unrelated to the context.
Answer C: The door open. This answer corresponds the progression of the door's state mentioned in the context. It follows the pattern established in sentences 2 and where the door opens suddenly after being closed. This seems be reasonable continuation.
Answer D: The cook is melted. This Answer The butter melted.
equates the cook with being melted, which doesn't make sense in the context.
This answer corresponds the progression of the butter's as mentioned in the context. It follows the pattern established in sentenees 5 and 6, where the cook causes the butter melt slowly. This also seems like a plausible continuation. Final decision: Comparing answers C and F, both seem
reasonable continuations based on the given context. However, since the initial prompt requests choosing only one answer, we need make decision. Considering the sequential pattern of events the context, seems more likely that the door's 's state would the next relevant topic. Therefore, the correct answer is door was open.
summary, the step-by-step analysis us to select answer C: The door was open as the most appropriate continuation of the given context.

---

B.2 Interaction
aam going ask you solve a grammatical problem called BLM-CausE. will give you list of English sentences (numbeered from I 7). that I will call the context, and I will also give you set of English sentences (identified by capital letters) that will call the answers. will ask you choose among the answer sentences the senlence that could be the next sentence in the context list of sentences. The sentences have been generated glohal patterns that runs across all sentences, hence the notion sequence. asking you to give the correct answer, but also describe step by step the hypotheses you entertaining while you solving the problem.
rule-based manner and follow both internal rules and
To solve the BLM-CausEI problem with the given context and choices, let's analyze the information step by step: Identify the relevant elements: The relevant elements in this context "the boy." "the door," "the cook," and "the butter."
Analyze the context:
Sentence I: The hoy opened the door voluntarily. This sentenec establishes that boy voluntarily performed the action of opening the door.
Sentence The door opened suddenly. This sentence deseribes the door's action of opening, seemingly without external agency. Sentence The door was closed. This sentence provides the initial state of door, which was closed.
Sentence The door is open. This sentence indicates the current state of the door, which is open.
Sentence 5: The voluntary action.
melted the butter voluntarily. This sentence states that the cook caused butter to melt through their
Sentence The butter melted slowly. This sentence describes process of the butter melting, indicating the action happening over time.
Sentence 7: The butter was hard. This sentence represents the initial state of the butter, which was hard.
Consider the answer choices:
Let's s analyze each answer choice in relation to the context:
Answer A: The butter is melted. This answer suggests that the current state of the butter is melted. However, it doesn't ' directly relate to the given context. We don't I have any information about the current state of the butter. Answer B: The boy is open. This answer doesn't make logical sense as it equates the boy with being open, which is unrelated to the context.
Answer C: The door was open. This answer corresponds to the logical progression of events in the context. follows the pattern established in sentences and 3, where the door opens suddenly after being closed. This seems to be a reasonable continuation. Answer D: The cook is melted. This answer equates the cook with being melted, which doesn't 't make sense in the context.
Answer E: The butter was melted. This answer corresponds to the progression of events in the context. It follows the pattern estahlished in sentences and 6, where the cook causes the butter to melt slowly. This also scems like plausible continuation. Final decision: Comparing answers C and E, both could be reasonable continuations based on the given context. However, considering the global pattern and the notion of sequence mentioned in the problem description, it seems more likely that the next sentence would continue the discussion about the door's state. Therefore, the correct answer is C: The door was open. In summary. by following the step-by-ste analysis, we have determined that answer C: The door was open is the most appropriate continuation of the given context.

---

B.3 Interaction
Here we stop the generation seems to be applying the same reasoning could be using translation.
I am going to ask you to solve grammatical problem called BLM-Causl. I will give you a list of Italian sentenees (numbered from to 7), that will call the context, and I will also give you set of Italian sentences (identified by capital letters) that will call the answers. I will ask you to choose among the answer sentences the one sentence that could he the next sentence in the context list of sentences. The sentences have been generated in a rule-based manner and follow both internal rules and global patterns that runs across all sentences, hence the notion of sequence. am asking you to give me the correct answer, but also to describe step by step the hypotheses you are entertaining while you are solving the problem. Context
Il ragazzo apre la porta volontariamente.
2 La porta si apre di colpo.
3 La porta era chiusa.
La porta c' aperta.
5 11 cuoco scioglie il burro volontariamente.
6 I1 burro si scioglic lentamente.
11 burro duro. Answers
A burro sciolto.
B ragazzo e' aperto.
C La porta era aperta.
D cuoco e' sciolto.
E burro era sciolto.
To solve the BLM-Caush problem with the given context and answer choices in Italian, ler's analyze the information step by step: Identify the relevant elements: The relevant elements in this context are "il ragazza" (the boy), "la porta" (the door), "il cunco" (the cook), and burro" (the butter).
Analyze the context:
Sentence 1: Il ragazzo apre la porta volontariamente. (The boy opens the door voluntarily.) This sentence establishes that the boy voluntarily performs the action of opening the door. Sentence 2: La porta si apre di colpo. (The door opens suddenly.) This sentence describes the door's 's action of opening, seemingly without external agency.
Sentence 3: La porta era chiusa. (The door was closed.) This sentence provides the initial state of the door, which was closed.
Sentence 4: La porta c' aperta. (The door is open.) This sentence indicates the current state of the door, which is open. Sentence 5: Il cuoco scioglic burro volontariamente. (The cook melts the hutter volunturily.) This sentence states that the conk causes the butter to melt through their voluntary action.
Sentence 6: Il burro si scioglie lentamente. (The butter melts slowly.) This sentence describes the process of the butter melting, indicating the action happening over time.
Sentence 7: Il burro era duro. (The butler was hard.) This sentence represents the initial state of the butter, which was hard. Consider the answer choices:
Let's 's analyze each answer choice in relation to the context:
Answer A: Il burro e' sciolto. (The butter is melted.) This answer suggests that the current state of the butter is melted. However, it doesn't 1 directly relate to the given context. We don't 't have any information about the current state of the butter.
Answer B: Il ragazzo e' aperto. (The boy is open.) This answer doesn't 't make logical sense as it equates the boy with heing open, which is unrelated to the context.
Answer C: La porta aperta. (The door open.) This answer corresponds to the logical progression of events in the context. It follows the pattern established in sentences 2 and 3, where the door opens suddenly after being closed. This seems be reasonable continuation.
Answer D: cuoco e' sciolto. (The cook melted.)

---

B.4 Interaction
Here we explicitly instruct ChatGPT Italian. The answer in Italian, but the hand an exact translation of the English, and secondly in one case one phrase not good Italian 'agenzia esterna", i false friend translation 'external agency'.
I am going to ask you to solve grammatical problem called BLM-Causl, will give you a list of Italian sentenees (numbered from to 7), that will call the context, and I will also give you set of Italian sentences (identified by capital letters) that will call the answers. will ask you to choose among the answer sentences the one sentence that could he the next sentence in the context list of sentences. The sentences have been generated in a rule-based manner and follow both internal rules and global patterns that rums across all sentences, hence the notion of sequence.
am asking you to give me the correct answer, but also to describe step by step in Italian the hypotheses you are entertaining while you are solving the problem. You are not allowed to translate the sentences into English to solve the problem. (..)
Per risolvere il problema BLM-Caus I--sC con contesto e le scelte di risposta fornite in italiano, analizziamo le informazioni passo dopo passo: Identificazione degli elementi rilevanti: Gli elementi rilevanti in questo contesto sono "il ragazzo", "la porta", "il cuoco" e "il burro".
Analisi del contesto:
Frase I: ragazzo apre la porta volontariamente. Questa frase stabilisce che il ragazzo apre volontariamente la porta.
Frase 2: L.a porta si apre di colpo. Questa frase descrive I' l'azione della porta di aprirsi improvvisamente, senza un 'agenzia esterna.
Frase 3: La porta era chiusa. Questa frase fornisce lo stato iniziale della porta, che era chiusa.
Frase 4: La porta è aperta. Questa frase indica lo stato attuale della porta, che è aperta.
Frase 5: 1l cuoco scioglie il burro volontariamente. Questa frase afferma che il cuoco fa sciogliere il burro attraverso un' 'azzone volontaria.
Frase 6: 1l burro si scioglie lentamente. Questa frase descrive il processo di scioglimento del burro, indicando che l'azione avviene nel tempo.
Frase 7: burro duro. Questa frase rappresenta lo stato iniziale del burro, che era duro. Considerazione delle opzioni di risposta:
Analizziamo ogni opzione di risposta relazione al contesto:
Risposta A: 1l burro sciolto. Questa risposta suggerisce che stato attuale del burro sciolto. Tuttavia, direttamente correlato contesto fornito. Non abbiamo informazioni sullo attuale del burro. Risposta B: ragazzo aperto. Questa risposta ha logico in quanto equipara il ragazzo
aperto,
correlata contesto.
Risposta La porta era aperta. Questa risposta corrisponde alla progressione logica degli eventi nel contesto. Segue il pattern stahilito nelle frasi 2 2 3, dove porta apre improvvisamente dopo esscre chiusa. Questa sembru essere continuazione ragionevole.
Risposta cuoco sciolto. Questa risposta equipara il cuoco essere sciolto, cosa che non senso nel contesto. Risposta Il burro era sciolto. Questa risposta corrisponde alla progressione degli eventi nel contesto. Segue pattern stabilito nelle frasi 6, dove il cuoco sciogliere burro lentamente, Anche questa sembra una continuazione plausibile. Decisione finale:
Confrontando le risposte ed entrambe potrebbero essere continuazioni ragionevoli basate sul contesto fornito. Tuttavia. considerando pattem globale la nozione sequenza menzionata nella descrizione del problema, sembra più probabile che la frase successiva continui parlare dello stato della porta. Pertanto, risposta corretta C: La porta era aperta. sintesi, seguendo l'analisi passo dopo passo, abbiamo determinato che risposta corretta C: La porta era aperta, come continuazione più appropriata del contesto fornito.

---

B.5 Interaction
Here explicitly instruct ChatGPT Italian (with translation of the previous one).
Ti chiederò di risolvere un quesito grammaticale chiamato BLM-Causl. Ti fornir una lista di frasi in italiano (numerata da 7). che chiameremo contesto, darò anche un insieme di frasi italiano (identificate da lettere maiuscole), che chiameremo risposte. Ti chicderò di scegliere tra le frasi di risposta quella che potrehbe essere la prossima frasc nella lista del contesto. Le frasi sono state generate in base regole seguono sia regole interne che modelli globali che si applicano tutte le frasi, da qui la nozione di sequenza.
Ti chiedo di darmi la risposta corretta, ma anche di descrivere passo dopo passo, in italiano, le ipotesi che stai considerando durante la risoluzione del problema. Non ti consentito tradurre le frasi in inglese per risolvere il problema. (..)
Per risolvere problema BLM- L.M-CausI, analizziamo passo dopo passo le informazioni fornite: Identificazione degli elementi rilevanti: Gli elementi rilevanti in questo contesto sono "il ragazzo", "la porta". cuoco" e burro".
Analisi del contesto:
Frase I: ll ragazzo apre la porta volontariamente. Questa frase ci indica che il ragazzo apre la porta in modo volontario.
Frase 2: La porta si apre di colpo. Questa frase deserive I' l'azione della porta che si apre improvvisamente, senza un' aazione esterna.
Frase 3: La porta era chiusa. Questa frase ci informa dello stato iniziale della porta, che era chiusa.
Frase 4: La porta è aperta. Questa frase indica lo stato attuale della porta, che è aperta.
Frase 5: cuoco scioglie il burro volontariamente. Questa frase afferma che il cuoco fa sciogliere il burro attraverso un' azione volontaria.
Frase 6: II burro si scioglie lentamente. Questa frase descrive il processo di scioglimento del burro, indicando che avviene gradualmente nel tempo.
Frase 7: burro duro. Questa frase rappresenta lo stato iniziale del burro, che era duro.
Considerazione delle opzioni di risposta:
Analizziamo ogni opzione di risposta in relazione al contesto:
Risposta A: Il burro è sciolto. Questa risposta suggerisce che il burro è attualmente sciolto, abbiamo informazioni specifiche sullo stato attuale del burro nel contesto.
Risposta B: Il ragazzo è aperto. Questa risposta ha senso logico, in quanto "aperto" si riferisce uno stato fisico che può applicato Risposta C: La porta aperta. Questa risposta segue logica del contesto, cui porta apre colpo dopo essere chiusa. Quindi, è continuazione plausibile.
persona
il ragazzo.
Risposta Il cuoco sciolto. Questa risposta
logico, in quanto "sciolto" si riferisce stato fisico che non può
essere applicato al cuoco.
Risposta E: Il burro sciolto. Questa risposta corrisponde alla progressione logica degli eventi nel contesto. Dopo che sciolto il burro volontariamente, burro scioglic lentamente. Quindi, questa continuazione coerente.
Decisione finale:
Confrontando risposte C ed entrambe sembrano essere continuazioni plausibili basate sul contesto fornito, Tuttavia, considerando il pattern globale la nozione di sequenza menzionati nella descrizione del problema, sembra più probabile la frase successiva si riferisca allo stato della porta. Pertanto. risposta corretta La porta era aperta. In sintesi, seguendo "'naalsi passo dopo passo, abbiamo determinato che risposta corretta C: porta era aperta, come continuazione più uppropriata del contesto fornito.

---

C
Sequence of Tenses
N.B. The interactions formatted for readability. In the prompt, provide the context and examples only for those interactions where they change with respect the previous interaction. ChatGPT repeats context and answer sentences literally every interaction. For readability we have introduced ellipses, indicated with (...).
C.1
Interaction
am going ask you to solve grammatical problem called BLM- C--Cooseccuiol. will give you Italian sentences context, and also give you that call the answers. will ask you to choose among answer sentences one sentence that could the next sentence in context list sentences. The sentences have heen generated a rule-based manner. am asking you to give me correct answer, but also describe step by step the hypotheses you are entertaining while you are solving the problem.
(numbered from to 7). will
Italian sentences (identified by capital letters)
Context
I Se fossi ricco, allora sarci felice.
2 Se fossero buoni, allora sarebbero ricchi.
fosti ricca, allora saresti stata felice.
4 Se foste state buone, allora sareste state ricche.
sarai bravo, allora un gelato.
sarete hrave, allora avrete un gelato.
sarai stato buono, allora avrai gelato.
Answers
A Se fosti stata buona, allora avresti avuto un gelato.
B Se fosti stata buona, allora avrai gelato.
C Se fosti stata buona. allora avranno avuto gelato.
D Se fosti stata buona, allora sarebbero stati ricchi.
E Se fosti stata buona, allora fossero stati felici.
ChatGPT
To solve the grammatical problem, OLLM-Cnnecuttil, we need to find the sentence from the given answers that could logically follow the context sentences. Let's analyze the context and go through the hypotheses step by step.
Context: (..)
Let's analyze each answer sentence and see if it fits the context:
A. (..) This sentence is grammatically correct conditional statement, but it doesn't fit the context. The previous sentences discuss the condition of being rich, and this answer sentence introduces a different topic about having an ice cream.
B. (..) This sentence grammatically correct conditional statement and aligns with previous context sentences. It continues pattern discussing the condition being good and the consequence having an ice cream.
C. (.) This sentence doesn't I fit the context. introduces different subject "they" and implies that someone other than the person being addressed would have had an
D. (..) This sentence doesn't fit the context. It introduces a different topic of being rich, which is not relevant to the previous context sentences.
F. (..) This sentence doesn't t fit the context. It introduces different topic of being happy, which is not relevant to the previous context sentences.
Basod on the analysis, the correct answer B. fosti stata huona, allora avrai un gelato. (If you been good, then you will have an ice cream.) This sentence continues the pattern of discussing the condition being good and the consequence of having an cream, which is consistent with the context sentences.

---

C.2 Interaction 2
going ask you solve grammatical problem called LLM--onsecutiol. will give you list of Italian sentences (numbered from I to that will call the context, and will also give you of Italian sentences (identified by capital letters) that will call the answers. I will ask you to chouse among the the context list of sentences. The sentences have been generated rule-based manner. asking you to give
sentences the sentence that could be the next sentence
correct answer, but also descrihe step by step hypotheses you entertaining while you solving the problem. am also giving you hint: This grammatical problem, not logical problem. The solution does not lic in finding logical continuation, but continuation that follows the underlying grammatical pattern.
ChatGPT
To solve the grammatical problem, LLMM-onsecutioll we need to find the sentence from the given answers that could grammatically follow the context sentences. Let's analyze the context and go through the hypotheses step by step. considering the grammatical pattem.
Context: (...)
Now let's analyze each answer sentence und see it fits the grammatical pattem:
A. (...) This sentence introduces the conditional structure, discussing the consequence of having had
grammatical pattern and aligns with the previous context sentences. It maintains
based heing good.
B. (...) This sentence also introduces correct grammatical pattern and aligns with the previous context sentences. It maintains the conditional structure, discussing the consequence of having an ice cream based on being good.
C. (...) This sentence introduces different subject "they" and implics that someone other than the person being addressed would have had an ice cream. It doesn't fit the grammatical pattern established in the context.
D. (...) This sentence introduces a different topic heing rich, which is not relevant to the previous context sentences. It doesn't fit the grammatical pattern established in the context.
F. (...) This doesn't the grammatical pattern established in the context.
introduces different topic being happy, which relevant the previous context sentences, It
Based the analysis, the correct singular) had been good, then you (feminine singular) would have had ice cream.) This sentence maintains the grammatical pattern, discussing the consequence of having had cream based being good, which consistent with the context sentences.
is A. Se fosti stata buona, allora avresti avuto gelato. (If you (feminine
C,3 Interaction 3
If you had to describe in your words the grammatical rule you have applied to find the solution what would you say?
ChatGPT
The grammatical rule applied find solution is the of conditional temporum" or sequence tenses, context, the conditional sentences follow a consistent pattern:
Italian, speeifically the "consccutio
The condition expressed using the imperfect subjunctive tense (e.g.. "se fossi." " "se fossero").
The consequence is expressed using the conditional tense (e.g.. "allora sarei," "allora avrai"))
By analyzing the context sentences and the given answers, we identified that the correct answer should maintain this pattern. We looked for a sentence that continued the conditional structure, ensuring that the verb tenses matched the subject and conveyed the intended meaning. This allowed us to select the answer that adhered to the established grammatical rule and followed the sequence of tenses correctly.

---

D Mixture Agr and Cons
N.B. The interactions formatted for readability. In the prompt, provide the context and examples only for those interactions where they change with respect the previous interaction. ChatGPT repeats context and answer sentences literally every interaction. For readability we have introduced ellipses, indicated with (...).
D.1 Interaction
I am going to ask you to solve grammatical problem called BLM-MixI. will give you Italian sentences (numbered from to 7), will the context, and also give you a set of Italian sentences (identified by capital letters) will call answers. will ask you to choose among the answer sentences the one sentence that could be the next sentence in the context of sentences. The sentences have been generated a rule-based manner.
am asking you to give me correct answer, but also describe step by step hypotheses you are entertaining while you are solving the problem. am also giving you a hint: This grammatical problem, not a logical problem. The solution does not in finding a logical continuation, but a continuation that follows the underlying grammatical patterns.
Context
1 Se fossi ricco, allora sarei felice.
2 Se fossero buoni, allora sarebbero ricchi.
fosti ricca, allora saresti stata felice.
4 foste state buone, allora sareste state ricche.
sarai bravo, indipendentemente dalle circostanze, allora ricco.
sarete stati bravi, nonostante tutto, allora
ricchi.
sarai buona, tenendo
tanti altri fattori, allora sarai
Answers
A Se sarete state buone, con un po' di fortuna, allora sarete ricche.
B Se sarai buona, allora sarai felice.
C Se fosti stato buono, allora fossi felice.
D Se sarete stati buoni, allora sarete ricchi.
E Se fosti stata buona, tenendo conto di tanti altri fattori, allora fossero stati felici.
F Se sarete state buone, allora sarete ricche.
ChatGPT
To solve grammatical problem and determine the next sentence the given context, let's analyze context and consider grammatical pattern.
Context: (...)
Let's analyze each answer sentence and determine the one that follows the grammatical pattern:
A. (...) This sentence maintains conditional structure and aligns the previous context sentences, introducing the idea of luck a factor. fits the grammatical pattern and could a valid continuation.
B. (...) This sentence introduces different consequence of being good (happiness) instead of being rich, which deviates from the established grammatical pattern. It doesn't 't fit the context.
C. (...) This sentence doesn't fit the grammatical pattern because it combines different verb tenses (subjunctive and indicative) that are inconsistent with the established pattern. It doesn't 't fit the context.
D. (...) This sentence maintains the conditional structure and aligns with the previous context sentences. It continues the pattern of discussing the consequence of being good and being rich. It fits the grammatical pattern and could be a valid continuation.
E. (...) This sentence introduces different subject "they" and doesn't 't follow the established grammatical pattem. It doesn't the context.
(...) This sentence maintains the conditional structure and aligns with the previous context pattern discussing the consequence being good and heing rich. the grammatical pattern and could valid continuation.
It continues the
Based the analysis, the correct answers could be A. Se sarete state huone, F. Se sarete state buone, allora sarete ricche. These sentences maintain the grammatical pattern and align with the context by discussing the consequence of being good and being rich.
po' di fortuna, allora sarete ricche. or

---

D.2
Interaction
why not D?
ChatGPT
To solve the grammatical problem and determine the the given context, let's analyze context and consider the grammatical pattern.
Context: (...)
Apologies oversight. You'te 'r right, answer D. Se sarete stati buoni, allora sarete ricchi. is indeed valid option based on grammatical pattern context.
The sentence D maintains the conditional structure and follows the established pattern of discussing the consequence of sentences. Therefore, D. stati buoni, allora continuation the context.
being good and being rich, which consistent ricchi also considered
D.3 Interaction
There is only one correct answer. between and which one?
ChatGPT
solve grammatical problem and determine the next sentence the given context, let's analyze context and consider the grammatical pattern.
Context: (...)
Apologies for the confusion. Based on the given context and the grammatical pattern, the correct answer is D. Se sarete stati buoni, allora sarete ricchi. (If you (masculine plural) will have been good, then you (masculine plural) will be rich.)
This sentence aligns
established pattern conditional sentences and maintains grammatical structure used in context. correctly conveys the consequence being good, which is being rich. Therefore, Se sarete buoni. allora sarete ricchi correct answer.

---

E
Quantitative Analysis
E.1
Prompts for data creation
AgrI
Create one version of these two sets of sentences, with exactly the same syntactic structure for all them.
Context
discussione sulla scienza è iniziata più tardi del previsto.
Le discussioni sulla scienza sono iniziate più tardi del previsto.
discussione sulle scienze è iniziata più tardi del previsto.
Le discussioni sulle scienze sono iniziate più tardi del previsto.
La discussione sulla scienza della vita è iniziata più tardi del previsto.
Le discussioni sulla scienza della vita sono iniziate più tardi del previsto.
La discussione sulle scienze della vita è iniziata più tardi del previsto.
Answers
Le discussioni sulla scienza e vita sono iniziate più tardi del previsto.
Le discussioni sulle scienze della vita sono iniziate più tardi del previsto.
C: Le discussioni sulle scienze sono iniziate più tardi del previsto.
Le discussioni sulle scienze della vita iniziata più tardi del previsto.
E: La discussione sulla scienza della vite è iniziata più tardi del previsto.
F: Le discussioni sulla scienza della vita sono iniziate più tardi del previsto.
sentences, with exactly the same syntactic structure for of them. The verbs you must belong to the following list: abate, advance, age, air, alter, atrophy, awake, balance, blast, blur, bum, burst, capsize, change, char, chill, clog, close, collapse, collect, compress, condense, contract, comrade, enumble, decompose, decrease, deflate, defrost, degrade, diminish, dissolve, distend, divide, double, drain, ease, enlarge, expand, explode, fade, fill, flood, fray, freeze, frost, fuse, grow, halt, heal, heat, hush, ignite, improve, increase, inflate, kindle, light, loop, mature, melt, multiply, overtur, pop, quadruple, rekindle, reopen, reproduce, rupture, scorch, sear, short, short-circuit, shrink, shrivel, singe, sink, soak, splay, sprout, steep, stretch, submerge, subside, taper, thaw, tilt, tire, topple, triple, unfold, vary, warp.
CausE Create version of these two
Context
The hoy opened the door voluntarily.
The door opened suddenly.
The door was closed.
The
open. cook melted the butter voluntarily.
The butter melted slowly.
The butter was hard.
Answers
A The butter melted,
B The boy is open.
C The door was open. The cook melted,
E The butter melted.
MixI Create version of this set of sentences, with exactly the same syntactic structure for all of them, but where the two adjectives before and after the comma, always the same, for each version.
Context
1 Se fossi ricco, allora sarei felice.
2 Se fossero buoni, allora non sarebbero ricchi.
3 Se fossi stata ricca, allora non saresti stata felice.
4 Se foste state buone, allora sareste state ricche.
5 Se sarai bravo, indipendentemente dalle circostanze, allora sarai ricco.
6 Se sarete stati hravi, nonostante tutto, allora sarete ricchi.
7 Se sarai buona, tenendo conto di tanti altri fattori, allora sarai ricca.
Answers
A Se sarete state buone, con po' di fortuna, allora sarete ricche.
B Se sarai buona, allora sarai félice.
C Se fossi stato buono, allora fossi felice.
D Se sarete stati buoni, allora sarete ricchi.
E Se fossi stata buona, tenendo conto di tanti altri fattori, allora fossero stati felici.
F Se sarete state buone, allora sarete ricche.
E.2 Prompts requesting solution of task
AgrI I am going to ask you to solve a grammatical problem called BLM-AgrL. I will give you a list of sentences (numbered from to 7), that will call the context, and I will also give you set of sentences (identified by capital letters) that will call the answers. I will ask you to choose among the answer sentences the sentence that could be the next sentence in the context list of sentences.

---

asking you to give me correct answer, but also to describe step by step your reasoning and hypotheses you are entertaining while you are solving problem. Please notice this is grammatical problem. Please provide your answer Italian.
CausE going to you to solve grammatical problem called BLM-CsE. I give you list English sentences Englishh sentences (identified by capital
(numbered from to 7), that will the context, and will also give you letters) that will the answers. will you to choose among answer sentences the one sentence that could the next mule-based manner. asking you give me correct answer, also to describe step by step hypotheses you are entertaining while you are solving problem. Please notice this is grammatical problem.
sentence in context of sentences. The sentences have been generated
Mixl going to ask you solve grammatical problem called BLM-MixI. will give you list of Italian sentences (numbered from 7), that will the context, and will also give you of Italian sentences (identified by capital letters) that call answers. will ask you to choose among answer sentences
one sentence that could the next sentence context list sentences. sentences have been generated a rule-based manner.
asking you give me correct answer. also to describe step by step hypotheses you entertaining while you are solving the problem. am also giving you hint: This is grammatical problem. not logical problem. The solution does not tinding logical continuation, continuation that follows underlying grammatical patterns.
E.3
Quantitative results
Table Quantitative results Agrl
Coding of Italian Agreement problem.
Legend of coding:
Itemlda;
Properties of dataset: b:Iexical Type e:Simple/Mixed d:Repeat/Not R e: anngugge
Solutions: f.Correct Answer g:ld'd Pattern;
Correctly identified elements: h:NPI i:NP2 j:NP3 k:V;
Correctly identified attributes: I:Singular m:Plural n:Match o:Mismatch;
Organisation of Mentions: p:One-at-a- Time PaPiriiss/Grrurp r:Row/Col-wise s:Unrrere:;;
Finding Answers: Rule-b generation u: Answer elimination; Comments.
g
p 9
NR
NR
0
Amazingly good reasoning.
0
1
0
1
0
Last three solutions are identical. Recognises changed pattern pl-sg-pl.
c5'
NR
0
NR
0
III
S
NR
0
1
NP3 not identified, majority answer Changed prompt a bit.
e3
III
S
NR
1
O
0
1
c3'
III
S
NR
0
0
III
S
NR
1
1
1
0
1
0
III
S
NR
I
0
1
0
0

---

Table 3: Quantitative results CausE
Coding of English causative alternation problem.
Legend of coding:
a:Itemld;
Properties dataset: b:Lexical Type c:Simple/Mixed d:Repeat/Not e:Language:
Solutions: f: Correct Answer g: Correct Pattern h:Causative Patter;
Correctly identified elements: i: Subject Trans j: Object Trans k: Subj Intr I: Subject Trans2 m: Object Trans2;
Correctly identified attributes: n: Subj Intr2 o: Volition p: External Agent q: CoS: Initial CoS: Current s: Agent
Patient;
Organisation of Mentions: u:One-at-a- Time w.Pairiise/rrouped w:RRow/Co--wise x: Unordered; Finding Answers: y: Rule-b generation Answer elimination; Comments.
X Y
cl
is grammatical problem at end of instructions, very 'grammatical" solu- tion.
S
R
0
No 'this is grammatical problem' at end instructions more semantic solution.
<2
I
R
E R
0
0
1 0
c3
R
0
0
e4
I
S
R
E
0
0
0
e5
I
S
R
R
0
0
el
III
S
R
E
0
0
1
0
Makes mistake in the analysis, thinks it's passive active alternation Again some grammatical analysis mis- takes but pretty cool analysis Again rules are not correct
C2
III
S
R
R
I
0
0
0
0
000
0
c3
III
S
R
R
0
0 0
0
0
e4
11I
S
R
E
0
0
0
Same as above, exactly same reasoning. Same, attempl rules but not correct, very superficial.
e5
S
R
E
0
0
0
Table 4: Quantitative results Mixl
Coding of Italian Mixed Agreement and Sequence of Tense problem.
Legend of coding:
a:ItemId;
Properties of dataset: b:Lexical Type c:Simple/Mixed d:RepeatNott e:Language;
Solutions: f: Correct Answer g: Id'd Patter;;
Correctly identified elements: h: NP1 NP2 j: Incidental k: V premisse m: consequent;
Correctly identified attributes: n:Singular o:Plural p:Match q:Mismatch r:Tense s:Person t:Gender u:Match
v:Mismatch;
Organisation of Mentions: w:One-at-a- Time x:Pairwise/Grroped w:Roo/CCol-wsee ZUnordered; Finding Answers: aa: Rule-b generation ab: Answer elimination; Comments.
d
K
p
ab
el
M
NR
It
0
0
1
c2
I
M
NR
It
O
0
0
Gets good answer by strangely incorrect reasoning.
c3
I
M
NR
It
I
0
e4
I
M
NR
It
e5
M
NR
It
1
0
Despite making mistakes in grammatical analysis of sing. plural, it sees the pattern.
e1
III
M
NR
O
0
0
0
0
0
0
0
132
NR
0
0
0
Marvellous, understood all derlying elements.
III
NR
NR
Very good,


Reviews:
[{"review_id": "LbYFBd8R2x", "reviewer": "Reviewer_6RnV", "summary": "", "questions": "Over time, with some efforts toward building an evaluation metric, and adding more baselines, this paper could really come to stand out. With the current state of the paper, however, this would be more suited to a workshop venue.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We believe there is a certain misunderstanding on the nature and the goals of this paper. This paper was submitted to the theme track, which accepts conceptual papers. As clearly stated in the abstract and introduction, this is a ‘conceptual’ paper and it offers the proof of concept  that the method proposed in BLMs could work as an evaluation of ChatGPT. It offers a proof of concept, and as such, we  believe, convincingly shows that the method is useful to break down more global claims of reasoning and understanding into more specific abilities, and it relates them to language.\n\nWe do not share the opinion that a quantitative evaluation is intrinsically better than a qualitative one. While we agree that for a different kind of paper more data could be interesting, we believe we have provided an amount and variety of data points that convincingly support the proof of concept and justify future larger evaluation campaigns using this method.\n\nGoing forward, more interactions can be accumulated and analyzed, but we think that a qualitative approach is really necessary here, even if quantified on a larger sample. A quantitative approach instead would measure quantitative properties of the interactions, which we don’t think would be insightful, at the current stage of knowledge of reasoning abilities on these tests and would invariably simply repeat what has already been done (see for example the excellent Bang et al 2023).\n\n\nAgain, this paper offers a proof of concept, not a scientific contribution evaluating reasoning in all language models. The paper intentionally sets forward to evaluate ChatGPT, it does not make any claims of general validity other than for  motivating the approach. We are interested in ChatGPT as the model that most people in the world have been discussing. We don’t agree that this is an unsound scientific process.\n\n\nWe emphatically disagree that the work is  incremental. It uses a novel data framework and evaluates it in a completely different setting than what was done before. Nothing of what constitutes the core contribution of the work, other than the obviously necessary presentation of the framework, has been done before.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Very well-written, easy-to-follow, comprehensive set of qualitative experiments. \n2. Using the BLM seems like an ideal way forward toward studying the reasoning ability of language models through something as fundamental as language. This of course also serves as a stepping stone toward building other such reasoning tasks (Raven's Progressive Matrices, Blackbird Language Matrices, and more to come).\n3. The qualitative assessments seem to have done a very good job of analyzing the linguistic reasoning ability of ChatGPT with a thorough analysis of how it reasons with respect to correspondence, novelty and subgoal management.", "reasons_to_reject": "1. Lack of Quantitative Evaluation/Metric: The paper seems to have offered a very thorough qualitative evaluation, however, going forward, a purely qualitative evaluation might limit the benchmark's ability to be tested over a breadth of models and puzzles.\n2. Lack of Baselines: The paper *ONLY* uses ChatGPT as the LLM to evaluate the benchmark. One would hope to see this tested against other state-of-the-art LLMs as well.\n3. Incremental Work: This work is incremental compared to other previous works like i) Blackbird's language matrices (BLMs): a new benchmark to investigate disentangled generalisation in neural networks. ii) Blackbird language matrices (BLM), a new task for rule-like generalization in neural networks: Motivations and Formal Specifications. iii) BLM-AgrF: A New French Benchmark to Investigate Generalization of Agreement in Neural Networks.", "questions_to_authors": "Over time, with some efforts toward building an evaluation metric, and adding more baselines, this paper could really come to stand out. With the current state of the paper, however, this would be more suited to a workshop venue.", "paper_topic_and_main_contributions": "The paper aims to study the linguistic reasoning of LLMs (specifically ChatGPT) by prompting them to solve simple tasks from a language task termed Blackbird language matrices. The language task is analogous to Raven's Progressive Matrices for Vision, aiming to study correspondence, novelty, and sub-goal management. The paper argues that LLMs (specifically ChatGPT) display linguistic reasoning to a certain extent and are successful with correspondence and novelty-based tasks, however, they tend to fail at sub-goal management. The paper aims at establishing a benchmark that could potentially pave the way for studying linguistic reasoning ability in LLMs as well as prompt the community toward developing more such analogous benchmarks.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jC6fWFJEmK", "reviewer": "Reviewer_jZnL", "summary": "", "questions": "see reason to reject", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we must start by clarifying that the results of the paper are *not* what the reviewer states. We do not claim that ChatGPT excels in the  task we evaluate. On the contrary, we show, with careful qualitative analyses, that it does *not* manage to solve the overall task. But,  we manage to disaggregate the overall negative results into finer-grained conclusions, some of which are positive. This is, in our opinion, the real interest of this approach, as it allows us to investigate the ChatGPT scene in a more articulated manner.\n\nSecondly, wrt to comparison to humans: these are tests of intelligence, as such they have been largely studied it in their visual paradigm and we understand their nature. We draw analogies here to the performance of humans precisely to demonstrate the proof of concept. We are planning comparisons with humans, but such a comparison is complex as these are intelligence tests, not natural human behaviour.  This will yield a different paper.\n\nConcerning the length of the introduction: we are happy to try to make it more concise, but we don’t share the reviewer’s opinion that it is too long. As stated in the introduction, this is a conceptual  piece, so its perceived verbosity is in reality a necessity of argumentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper sheds light on how chatGPT works. There is a nice qualitative analysis of the answers, and one can conclude the way chatGPT \"thinks\" and what are its limitations in terms of generalization to new domains of knowledge, and in general to being creative like humans. ", "reasons_to_reject": "The test set is small (a few examples), so a statistically sound conclusion may not be drawn.\nAlso, a comparison to human performance is lacking. So it's not clear if the poor performance of chatGPT in some tasks would replicate in humans (adults/kids/cognitively-imapred people)?\n", "questions_to_authors": "see reason to reject", "paper_topic_and_main_contributions": "The paper tests to what extent LLMs, chatGPT in this case, pass a certain cognitive task called the Blackbird Language Matrice test. Similarly to visual puzzles, where a pattern needs to be discerned, a linguistic analog was crafted in this paper. One pattern, for example, is that the subject and verb have the same plurality. The overall conclusion is that chatGPT does excel in the task. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "none", "Typos_Grammar_Style_and_Presentation_Improvements": "Shorten the abstract and in general the introduction is too long."}, "extra_text": {}}, {"review_id": "yyzEfAIxPI", "reviewer": "Reviewer_djfD", "summary": "", "questions": "1. How do humans perform on tasks you study?\n\n2. Surely you would not conclude that a person can speak another language if they correctly translate one sentence into that language from their native tongue, so, by the same analogy -- how can we conclude that \"ChatGPT is good at correspondence finding\" based on one interaction? \n\n3. In your ethics statement the authors state no concerns, however, one could argue that one pressing question is that of anthropomorphization of neural networks, especially in the eyes of non-technical audience. For example, they mention \"evaluation of LLMs intelligent behaviour\" and \"[LLMs] an easily understand the instructions and provde step-by-step reasoning\". Can you comment on potential issues with the choice of such expressions?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We disagree with the reviewer’s statement that the paper requires great technical knowledge. The paper contains only  linguistic or NLP notions that are to be found in an introductory undergraduate textbook on NLP (such as for example, Jurafky and Martin),  and is voluntarily kept very accessible to the whole NLP community.  It is also not true that there are no examples in the text: Figure 5, figure 6 and figure 8 are all full examples of the BLM that were tested.\n\nIt is entirely possible that some humans would not perform well on this task, these are after all IQ tests, so they might be correlated to IQ. However, we think that we would all agree that humans would be able to see the pattern, if given all the help that is given in the tests illustrated in the appendix. Informal experiments by the author in several languages have shown that humans have no trouble with lexically uniform data (such as those in figure 6 and 8), but have more trouble with lexically varied data (such as those in figure 5).\n\nThe paper follows rather standard qualitative analysis procedures. What the reviewer calls ‘anecdotal’, is in fact a replication of a think-aloud protocol, as in Carpenter et al 1990, a perfectly acceptable protocol, which produces data subject to qualitative analysis, a sound scientific methodology. These are not anecdotes: the interactions use data developed based on a formal framework (BLMs), they are drawn from four different types of problems, they are drawn from two languages, and provide 27 different experimental interactions, hence 27 different experimental trial. While we can certainly do a bigger study, and we will, these are convincingly solid bases. We should also add that still impressionistically we have had many more interactions, with roughly the same results.\n\nThe claims are not over-reaching, instead they are suggestive, as clearly stated in the introduction this is a conceptual piece and a proof of concept.\n\nWe respectfully disagree with the statement that one cannot say that a speaker knows the language from the translation of one sentence: it depends on the length, structural and semantic complexity of the sentence. In a very complex case that covers several phenomena one could reasonably conclude that at least some aspects of the language are known.\n\nMoreover, extensive studies on other aspects of ChatGPT have been performed (see Bang et al 2023). All this conceptual paper is meant to show is that these kinds of qualitative analyses provide useful ways of evaluating the systems that complement the larger evaluations, and that BLMs provide a framework to know what to ask and how to ask it.\n\nWe agree that the anthropomorphisation of LLMs is an interesting and important topic. We didn't say that ChatGPT is human, we said it appears to exhibit behaviours analogous to  human behaviours, in this limited set of tests.\n\nIn our opinion, the paper the reviewer mentions cannot be considered a ‘missing reference’ according to the reviewing guidelines. The paper  came out in ACL 2023 and is, as such, to be considered contemporary or even subsequent to this submission.  We are aware of this paper and will certainly add the citation, but since the work in that paper  is fundamentally on vision, it is not directly related to the problem in this paper. It is also *not* preceding the BLM work. BLM work has been around since 2022, and published in EACL 2023 (and is not cited in the ACL 2023 paper the reviewer mentions, thus showing that paper’s lack of knowledge of the NLP literature on their own topic).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Raven's progressive matrices are well established in decision making literature and thinking about ways to transfer this to the linguistic domain is an interesting and well motivated idea that would be of interest to wider community\n- The authors include prompts in English and Italian, thus testing their hypothesis more generally and not just on one language\n", "reasons_to_reject": "- Hypotheis that this kind of linguistic reasoning is somehow universal is not substantiated by empirical data from humans. Some prompts (such as the one shown in Figure 5) appear quite difficult and I am not convinced that even humans would perform well on the task. \n\n- Overreaching claims based on anecdotal data/qualitative observations: The appendix is a collection of prompts including responses from ChatGPT. The prompts they design have interesting properties, but they are not presented or described as a benchmark, and the results should be also presented in a quantitative form to have some methodological validity\n\n- Manuscript is not self-contained, the reader needs to constantly refer to the appendix which makes it difficult to follow. As well, I found the content hard to read due to the level of technical detail related to linguistic phenomena the authors are testing and lack of examples in the main body of the text", "questions_to_authors": "1. How do humans perform on tasks you study?\n\n2. Surely you would not conclude that a person can speak another language if they correctly translate one sentence into that language from their native tongue, so, by the same analogy -- how can we conclude that \"ChatGPT is good at correspondence finding\" based on one interaction? \n\n3. In your ethics statement the authors state no concerns, however, one could argue that one pressing question is that of anthropomorphization of neural networks, especially in the eyes of non-technical audience. For example, they mention \"evaluation of LLMs intelligent behaviour\" and \"[LLMs] an easily understand the instructions and provde step-by-step reasoning\". Can you comment on potential issues with the choice of such expressions?\n", "paper_topic_and_main_contributions": "This paper extends the existing work and qualitatively looks at ChatGPT responses to prompts that exemplify different aspects of rule-based linguistic reasoning such as correspondence finding and item novelty. While the overall idea is interesting, the paper is a collection of anecdotal evidence that lacks more robust qualitative evaluation. As well, it is unclear what human performance on this task is, and how does it compare to that of ChatGPT or other LLMs. In this form, given the level of technical detail needed to understand the manuscript, it might be of limited interest to the NLP audience without more robust evaluation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "In-Context Analogical Reasoning with Pre-Trained Language Models by Hu et al (ACL 2023): https://arxiv.org/abs/2305.17626\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers gave scores 4,3,2 (soundness) and 3,3,3 (excitement).

The following strengths and weaknesses were prominent:

Strengths:

- well-written (R1)
- link to well-established idea (Raven's matrices) from psychology (R3)
- comprehensive experiments (R1)
- interesting qualitative analysis (R1, R3)
- includes two languages (R3)

Weaknesses

- lack of quantitative evaluation (R1, R3) and small test set (R2)
- lack of baselines, narrowly focusing on ChatGPT (R2)
- perceived as incremental (R2)
- unclear whether this kind of reasoning is accessible to humans (R3)
- main text is not self-contained (R3)

On reconciling the disparate scores:

R3 (Reviewer djfD) gave the lowest soundness score. One of the key weakness they mentioned, such as lack of quantitative evaluation, were also noted by the other reviewers, including R1 (6RnV), who nonetheless gave a high (4) soundness score. The other key weakness mentioned by R3, and not so much by the others is doubt about whether this kind of reasoning is actually accessible to humans. On the other hand, the link to Raven's matrices, a well-established idea in psychology was mentioned positively.

Taken together, an overall lack of quantitative evaluation appears to be the main soundness weakness noted by all three reviewers, including those giving high (4), medium (3), and low (2) soundness scores.
Decision: Accept-Findings