ReCogLab: a framework testing relational reasoning & cognitive hypotheses on LLMs
Authors: Andrew Liu
Henry Prior
Gargi Balasubramaniam
Rivka Moroshko
Amir Zait
Ilia Labzovsky
Danny Karmon
Ishita Dasgupta
Kim Stachenfeld
Kenneth Marino
Venue: ICLR 2025 Poster (2025)
Keywords: Congitive Science
Large Language Models
Datasets
Evaluation
Relational Reasoning
Primary area: datasets and benchmarks
TL;DR: We build a flexible framework generator for evaluating long-context relation reasoning in LLMs to probe different cognitive effects and biases.

Abstract: A fundamental part of human cognition is the ability to not only recall previous memories, but also reason across them to draw conclusions. In cognitive science and psychology, this is termed relational reasoning and a number of effects and biases have been observed in human cognition. Designing experiments to measure these reasoning effects is effortful and does not transfer easily to analyzing language model reasoning patterns. To make exploring language models on relational reasoning easier, we introduce ReCogLab – a generative framework for constructing reasoning examples. Unlike static datasets, our framework has a number of benefits that help us in our goal of flexible evaluation of LLMs. First, our framework allows us to control the difficulty and context-length of the problem, allowing us to scale with model capability and evaluate LLMs at a variety of scales. Second, the ability to change the configuration of a dataset dynamically allows us to probe models on different aspects and capabilities. Finally, the flexibility of our approach enables the recreation of classic cognitive science experiments and the systematic study of relational reasoning biases in language models. We demonstrate several such experiments and present our findings on a wide variety of open and closed-source language models. We release all data and code at https://github.com/google-deepmind/recoglab.

Published as a conference paper at ICLR 2025
RECOGLAB: A FRAMEWORK TESTING RELATIONAL REASONING & COGNITIVE HYPOTHESES ON LLMs
Andrew Liu', Henry Prior!, Gargi Balasubramaniam! Rivka Moroshko', Amir Zait', Ilia Labzovsky'. Danny Karmon', Ishita Dasgupta', Kim Stachenfeld', and Kenneth Marino!
'Google Deepmind, (ahliu, hprior, gargisb, rikimoroshko, amirza ilabz, dannykarmon, idg, stachenfeld, (marino)@google.comm
ABSTRACT
A fundamental part of human cognition is the ability to not only recall previous memories, but also reason across them to draw conclusions. In cognitive science and psychology, this is termed relational reasoning and a number of effects and biases have been observed in human cognition. Designing experiments mea- sure these reasoning effects is effortful, and does not transfer easily to analyzing language model reasoning patterns. To make exploring language models on re- lational reasoning easier, we introduce ReCogLab a generative framework for constructing reasoning examples. Unlike static datasets, our framework has a number of benefits that help us in our goal of flexible evaluation of LLMs. First, our framework allows us to control the difficulty and context-lengli of the problem, allowing us scale with model capability and evaluate LLMs at variety of scales. dataset dynamically allows
Second, the ability to change the configuration to probe models on different aspects and capabilities. Finally, the flexibility our approach enables the recreation of classic cognitive science experiments and the systematic study of relational reasoning biases in language models. We demonstrate several such experiments and present our findings on a wide vari- ety of open and closed source language models. We release all data and code https///gihbbbbcooo/ogoeeeeeee/p
INTRODUCTION
While recent work on memory in large language models (LLMs) assumes that memory refers to recall of specific piece of information from the past (Li et al.) [024a]] Levy et al. (024)) research into human memory and reasoning has long shown memory to be much more complex. When humans remember events, facts, places, they don't just recall disconnected pieces, they recall associations (Cohen) 1993. Eichenbaum 2004). Thus humans have the remarkable ability recall relationships ( ("relational memory") and draw inferences across related memories ("relational reasoning").
Motivated by the reasoning and memory abilities of language models, prior works have created new reasoning benchmarks by defining new problems and collecting data targeting specific purposes like visual reasoning (Johnson et al.] 2017| Kahou al. 2018 Wang et al.] 2024). math reason-
ing (Lewkowycz et al.) [022 Wang et al. (022)) rule-following (Kazemi et al. 2023 Clark et al.)
2020b), or story problems (Yang al. 2024 Yu al., 2020 resulting in fixed test dataset. These datasets are subequently benchmarked ked against current state-of-the-ar models to gauge capability progression on the specific problem. However, this approach has numerous problems. First, these datasets, like all static datasets are susceptible to increasing model performance. Even as dataset creators try to invent harder problems that are quite difficult for current models, the rapid progress of the field often overtakes the dataset, giving it short shelf-life for useful benchmarking, Second, depending on the design of the benchmark, it can often quite difficult to probe or tease apart what makes the problem difficult for models or to explore different capabilities of the model. For example, if reasoning requires both recalling information and logical deduction, can we easily sepa- rate whether model fails due memory recall poor reasoning? Finally, these datasets are fixed and cannot be used by later authors to study related phenomenon. In the Cognitive Sciences for instance, many researchers want study whether specific effects observed in human reasoning arise

---

Published as a conference paper at ICLR 2025
Comparison
Social Networks
Syllogism
JSON Familles
Numbera
Ertidis
canl with sothcc
timeic
Typss
With bope mower
Beesosing
drimes a ynaagge Paat
and Nir
Audilar
Optans
Colcil
Figure 1: ReCogLab is customizable framework for generating relational reasoning evaluation examples. We show example problems and graph structure for Comparison, Social Networks, Syllogisms and JSON Families reasoning tasks. These tasks require variety of important relational reasoning capabilities such as transitive inference, working memory capacity, and set reasoning. Our framework's 's customization allows to probe language models increasingly difficult classes of relational reasoning problems generated by our test configurations. See Appendix [ for generated examples on each of these tasks.
in LLMs, such as content sensitivity (Lmmpinen et al. 2024). Often researchers will have create specific datasets by hand to probe these effects because fixed benchmarks do not have the flexibility to measure the parameters of interest.
We aim to solve these problems with a generative approach by introducing framework which creates datasets that can be used for different purposes. We introduce ReCogLab (see Figure [ for generating relational reasoning problems over long contexts that can be customized for many purposs.. The key aspect of this dataset is virtually every aspect of the generating process can be controlled via a configuration file specifying parameters. For instance, in all tasks we can change the complexity of the reasoning problem, allowing for the difficulty of the problem to scale up as models become more powerful or scale down for diagnosing poor performing models. The fine control of parameters such as the ordering of statements, the complexity of the language, or the amount of filler context allow us to probe models behaviors along many different dimensions. Finally, because of its configurability, Lab can be adapted for use by both MI. and Cognitive Science community to easily run a wide variety of exploratory experiments.
With ReCogLab, Lab, we perform a number of experiments and probes to demonstrate how our framework can be used to scale model performance, probe and analyze model performance, and recreate Cognitive Science experiments on humans and language models using our framework like ordering and congruency. Using our framework, we can identify several language model performance charac- teristics on different relational reasoning tasks. Our framework allows us to isolate failure breakpoints and provide insight into understanding general relational reasoning capabilities in language models.
In this paper we: (I) introduce ReCogLab, a new flexible dataset framework for scalable evaluation and cognitive inipice- probes of LLMs on relational memory (2) perform experiments showing how our dataset can scale and adapt over time to remain a useful evaluation framework (3) show how our configability can be used to probe and analyize different problems in relational memory (4) study several observed and hypothesized relational reasoning phenomena reported from the cognitive science literature enabled by the procedural generation of structured, text-based prompts. We release the ReCogLab framework and datasets used in this manuscript useful tool for evaluating examining LLMs for relational reasoning.
BACKGROUND
Relational Memory and Reasoning in Cognitive Science. have been studied by psychologists across variety of domains
Relational memory and reasoning
(Behrens et al. 2018 Eichenbaum
2004 Nelli et al. 2023 Kumaran & Maguire 2005] Tolman
1048|/Stupe & Ball
2008
Evans
et al. 1983). common experimental probe is "transitive inference" " (Burt et al. 1911 Piaget
1957): subjects are given data with the form "A>B, B>C, C>D, D>EE" and then queried about relationships that they did not observe directly but can be inferred (e.g. B>D). Different experiments vary parameters of transitive inference to construct meaningful insights into cognitive processes. We draw inspiration from this literature, measuring effects of presentation order (Hotta et al., 2013) and symbolic distance (Moyer & Bayer 1976) with 8 different language models. We also explore how these models' relational reasoning abilities differ in sequential processing versus feature learning settings (Koster al. 2018) through our congruency experiments.
2

---

Published as a conference paper at ICLR 2025
Other experiments have studied humans ability to reason about and sometimes navigate complex relational structures, such as trees, grids, or community graph structures (Mark et al. 2020 Schapiro et al. 2013 Garvert et al. 2017:Lymn & Bassett 2020). These experiments inspired our more richly structured, configurable options in our framework. Relevant work in psychology has also studied syllogistic reasoning (Chater & Oaksford T999p, and connections have been hypothesized that link syllogistic reasoning with reasoning over transitive chains (Guyote & Sternberg, (981). Indeed, a notable failure mode is that humans appear to simplify syllogisms create caster-to-reason-aaee transitive chains (Ceraso & Provitera 1971), which informs our experiments on symbolic distance.
Evaluation of LLMs Early work on reasoning benchmarks for language models includes bAbl (Weston et al. 2016), QA-style synthetic benchmark requiring very simple language un- derstanding and reasoning questions, and Knowledge Base Question Answer (KBQA) |Berant et al. (2013): Lopez al. (2013), ) a system for evaluating multi-hop reasoning on knowledge graphs Like our work, KBQA is heavily inspired by using graphs to assess relational reasoning capabilities.
More recently, datasets such as BigTOM (Gandhi al. 2023), ToMi (Le et al. 2019) and Rule- Takers (Clark et al., (020a) have been developed requiring much more complex reasoning. Other relevant work has investigated logic programming for logical deduction (Li et al. (024c), multihop QA (e.g. (Li et al. 2024b)), and knowledge graph reasoning (e.g. (Luo et al. 2023)).
Other related work includes long-context L.M evaluations Levy et al.) 2024; Bai et al. 2024 Vodrahalli et al.] 2024). Li al. (2024a) in particular includes some of the aspects of relational reasoning but without our dataset's ' customizability. All of these works focus specifically on this question of thoroughly examining the context length of language models. Our dataset is unique in affording a joint inspection of context length in the context of relational reasoning, and affords scaling complexity along other directions with customizabilty along multiple dimensions.
Comparison of LLMs to humans Substantial work has compared reasoning in LLMs humans (Hagendorff et al.; 2024]] Binz & Schulz 2023). employing cognitive science and psychological
methodologies to explore the biases in L.MSs Jones & Steinhardi (2022); [Seals & Shalin (2023); Webb et al. (2023); [Berglund et al. (2023). Some of the cognitive effects that we study here have been studied individually in some of these works such as: consistency of logical reasoning with prior knowledge Lampinen et al. (2024), syllogistic reasoning [isape et al. (2024), premise order in syllogisms Chen al. (2024), and susceptibility irrelevant information (an aspect of capacity experiments in. We similarly adopt experiments from the cognitive science literature as a reference point for understanding LLM behaviors, but in the context of relational memory and reasoning. We
show many of these experiments in Section 4.3
We have proposed framework for automatically generating new templated datasets for relational reasoning problems. Other works have looked to increase the speed of creating evaluations, for instance Dynabench Kiela et al. (202T) and later Dynatask Thrush et al. (2022) which allow for rapid integration of models into dataset creation. However, this still requires human annotation and thus would not be able to automatically change parameters of the dataset instantaneously as our framework does.
Sandbox Evaluation Toolkits
Cognitive scientists have long been interested in evaluating LLMs we do with humans Hernández Orallo (017) and environments such as animal-ai environment Voudouris et al. (2022) have been created. However, this environment is 3D embodied environment, and while it also touches on memory, it does not touch on language or on relational reasoning specifically. Another close comparison is CLEVR Johnson et al. (2017), a dataset of synthetically rendered 3D scenes that can be used to query for simple relational reasoning capabilities from images. Our work extends upon synthetically templates scenes by using generative framework and language to precisely construct more complex relational reasoning examples.
3 RECOGLAB
Unlike dataset which consists of static evaluation examples, the ReCogLab framework a generative system that deterministically creates relational reasoning dataset based on particular input
As well other AI systems, see e.g. Cropper & Dumančić 2022
3

---

Published as a conference paper at ICLR 2025
configuration and seed. Our highly customizable system configuration has many tunable knobs required to support the breadth of experiments needed to explore relational reasoning.
This key difference gives us several advantages when running experiments.
We can design experiments to isolate specific reasoning biases in language models that are well supported in human cognition studies. Not only can we show the existence of these biases, we can also measure their effect on problem-solving.
Effect Isolation:
Safeguards against leaks: With the advent of internet-scale training data, there are high risks of accidentally leaking dataset examples into training data. Not only is this hard to detect, if it does happen it becomes impossible to compare numbers before and after the leak. A generative framework avoids this by providing instructions for re-generating the same configuration of the dataset but with completely different test seed. We show that despite re-seeding a configuration, model performance across different seeds are statistically consistent under different experiment configurations.
Scalable Difficulties: A framework enables the creation of datasets with arbitrary difficulty, allowing for the measurement language model capabilities across various models, from compact device models to large, state-of-the of-the-art models.
3.1 GENERATIVE FRAMEWORK
Examples generated by ReCogLab come from one of several relational reasoning tasks: Comparison, Social Networks, Syllogisms and JSON Families. See Appendix [B.2 for more details and results. We chose these tasks because they can be generated automatically and include spectrum of problems in relational reasoning. Importantly, each of these tasks can be characterized as reasoning about graph problems using language. See Figure for examples from each dataset and the corresponding logical graphs associated with each.
Each example consists of context C multiple factual statements that the model assumes to be true (c.g. "Miranda is related to Steve"). Each of these facts denotes relationship between pairs of entities E,) person or object or other noun in the relationship) which compose edges within graph. Next the question Q which relies on subset of the context C' to answer, requires integrating information across distinct facts unambiguously produce an answer or answers A. We emphasize that ReCogLab is not just a dataset, but a framework for automatically generating infinite datasets and dataset examples. Our framework can control the core difficulty of the graph problem by increasing the number of nodes, the number of relationships, and even the structure of the graph. In addition, we can introduce auxiliary difficulties to the graph problem such as using "flavor text" or scrambling the order of statements in the context. Because we control the data generating process by specifying parameters, we can produce datasets that isolate specific auxiliary and core difficulties to understand how they affect the capabilities of different language models. Please Appendix B. for more details about how each task procedurally generates test examples.
3.2 TASK DOMAINS
Comparison. A comparison a directed edge in an acyclic graph where vertices are entities and edges describe directional comparison between them. Inspired by transitive inference, we construct comparison problems using three attributes that exhibit transitivity: comparison of age, size, or weight of objects.
For comparison reasoning problems, we target specific parameters to understand how language models behave when presented with similar problems in different forms such as: the comparison type, the number of objects, the ordering of comparisons, and the attribute types of directional reasoning.
Social Network. Inspired by Kumaran & Maguire (2005), the next set of tasks we propose requires reasoning about Social Networks. Like comparisons, a social network can be algorithmically represented as graph problem where nodes represent people and edges represent a particular relationship like friendship. Unlike with Comparisons where relationships are directional, Social Network assumes relationships are symmetric and therefore graphs undirected.
social network is good for probing long-context relational reasoning for several reasons. First, social networks can be arbitrarily structured. This makes it possible generate very large graphs
4

---

Published as a conference paper at ICLR 2025
with complex structure. Secondly we can introduce "flavor text" (see Appendix B for more details) generation in every example's 's context to provide a more story-like problem. Finally one could incorporate other kinds of social network edges like teacher-student (directed edge) or even enemies (undirected edge with a different semantic interpretation) to build even more complex reasoning probes.
Syllogisms. Previous work on syllogistic reasoning has shown interesting parallels between humans and language models (Eisape et al. 2024). We extend prior work that has studied ordering and congruence effects on humans and language models to include a capacity dimension. We construct chains of propositions that transitively resolve to single final conclusion and prompt the language model to choose between all possible conclusions with the given subject and predicate.
JSON Families. Inspired by knowledge based question answering (Lan et al. 202T) on structured problems, we also consider how relational reasoning is impacted by the use of programming language input like JSON. We created generating process for document graphs where elements in the JSON are nodes in the graph that connect to the parent object. We use this to model knowledge and generate testable questions like whether two individuals have the same hobbies.
3.3 DATASET CREATION
For each task, we create different configurations that generate datasets used in this paper with fixed seeds to guarantee consistency and reproducibility. We generate sweeps of different configurations targeting evaluations of specific cognitive probe. We generate 50 validation examples and report on 250-1000 test examples. Here the important thing is not the specific number of examples, but the ability tailor skill and difficulty levels to measure cognitive abilities in different L.Mss. We provide technical details in Appendix B and open-source code for generating examples from ReCogLab.
4 EXPERIMENTS
Here we demonstrate the flexibility in benchmarking, probing and experimentation of the ReCogLab framework. We find that our generative approach is not hindered by sampling issues by showing that different seeds generate comparable evaluations in expectation.
Next, we look at how our framework can be used to isolate critical problem parameters on reasoning
performance and the relative effect of increasing problem complexity in Section 4.1 We also investigate whether the exact way in which we ask the task or the reasoning complexity of the language affects performance Section 4.2
Finally, we show the flexibility of our framework reproduce Cognitive Science inspired experiments such as congruency, ordering effects, premise consistency, and indeterminacy in Section
Language Models We evaluate several open-source and closed-sourced models on our suite of generative reasoning benchmarks. our results, we show Google's open-source Gemma-2B. 9B, 27B family in green. The open-source Mixtral models are shown in orange. Gemini Flash and Gemini Pro (closed-source) are shown in blue. OpenAl's closed source GPT-4o is shown in red. Each specific probe involves validating on 50 examples before selecting the best prompt and parserfor test-time evaluation on. Please see Appendix [ for the library of prompts and parsing strategies.
Seeded Sample Variance Analysis When using a framework to evaluate capabilities, the test configuration defines shareable benchmark evaluate multiple models against. However, because a benchmark is constructed by randomly generating examples from a framework given configuration, there exists sample variance benchmark scores. Our key insight that ReCogLab generates examples based on a seed and configuration file in such a way that different seeds produce the same dataset "in expectation."
To understand the behavior of a generative framework's 's sampling variability. we run several "re- seeding" experiments. For different configurations on Social Network, we generated 30 different seeded datasets of 200 examples each. This gives us 30 re-seeded accuracy scores of single configu- ration for every model we benchmark on. Because measuring sample variance is computationally expensive (30 8 * 200 test examples), we restrict the analysis to Gemma-2B, Gemini Flash and
5

---

Published as a conference paper at ICLR 2025
Sced Sample Variance Validasion Set
hogu
Genii Rish
+.20
Gemiri
0.0
+.70
Evood Pa
5
+.20
0.0
1.0
Accurccy
Figure 2: Re-seeded Variance. We show
the distribution of 8 re-seeded config experi- ments in a boxplot and find that regenerated datasets from the same config score similarly. @ and are tasks with and without flavor text enabled and the number of entities used dur- ing generation. This shows that ReCogLab results are reproducible under a new seed.
Figure 3: Capacity vs. Accuracy. We evaluate each language model's long context capacity by increasing the number of entities for each of our tasks. We find in all cases that performance drops, although this drop is less severe in more capable models such GPT-4o and Gemini Pro
Gemini Pro. We measure performance a single chain-of -ftt-hugttthhttttt prompt which has good perfor- mance across all model families we validated on. Our results in Figure FigureBindicaae indicate that performance between different seeds of the same configuration are comparable and form valid hypothesis testing distributions. While it would be rigorous to demonstrate these properties on all evaluations presented in this paper (and future benchmarks configurations too), we posit that these guarantees hold true based on Figure 2 Therefore future evaluations using the same configuration are comparable even with different seeds and we can always reproduce results from a single-seeded experiment.
4.1 SCALING EXPERIMENTS
Capacity. One application of ReCogLab is generating long-context reasoning problems automat- ically for evaluation or training. Context can be added in several different ways such as the total length, number of relations, or the size of irrelevant context. In Figure 3 we evaluate how reasoning ability depends on the number of entities for each task domain by problems iwth different graph sizes. We find, perhaps unsurprisingly. that performance generally degrades as the complexity of the problem increases across all models. The more capable closed-source models (Gemini Flash, Gemini Pro, GPT-4o) perform the best.
We also have other studies of reasoning performance and capacity including slicing by symbolic distance and slicing by the type of irrelevant content being added in subsequent sections.
Symbolic Distance. Symbolic Distance, or the number of relations that separate two entities in a graph, is understood to be a key parameter in relational reasoning performance Moyer & Bayer] 1976). symbolic distance of implies there is exactly a one statement in the input that gives sufficient. information to answer the question, in which case the query reduces to a Needle-in-Haystan probe. In Figure 4 the first row each heatmap corresponds to NIH evaluation on each model which shows minimal degradation across all long context sizes. ReCogLab enables a more robust analysis of symbolic distance.
(NIH)
We find that most models have strong performance when only 10 entities are involved in the graph, regardless of the symbolic distance. However as the graph size grows, models ability to reason about the relationship shrinks to only reliably answering correctly Pro and GPT-4o, we also evaluate up to 70 entities and find that the quality of the prediction stabilizes, in other words adding more entities doesn't 't degrade relational reasoning capabilities as long as the number of traversals required is less than 6. For more detailed commentary and analysis on other tasks see Appendix C.T
symbolic distance of 6. With Gemini
4.2 QUESTION TYPE AND LANGUAGE EFFECTS
Next, we examine the capability for ReCogLab to further tease apart how model performs these problems. We want not only to show the relative performance of different models on a task; we want to understand the effect of question and language degrade different model's capabilities. A key aspect

---

Published as a conference paper at ICLR 2025
Number of Hops. No Flavor Text
Rich
61
A401,
666.1
0.00
61001
6:44
6:06
140%
617
5411
LIK
6
AMS
6:M
LIB
2816-11
6726
M65-0
COR
2000
506.1
5:46
KOK
R.CK
Figure 4: Symbolic Distance and Graph Size. Here we show the each model's 's performance
Social Network task, sliced by the Symbolic Distance and Graph Size. The x-axis denotes the graph size in terms of number of entities and the y-axis denotes the symbolic distance required to answer. Each box reports the accuracy and number of test examples in the test set in the slice. Red indicate higher accuracy on slice. See Appendix for this analysis presented on other tasks.
Gemma 28
Germa 98
Gemma 278
Gemini Flesh
Gemini Pro
GPT 40
Mixtrel 8x78
Mixtral 8x228
Namber of Extllis
Figure 5: Social Network Reasoning. Here we show two tasks and two ablations on all models. The two tasks consists of identifying the number of hops and the exact path of a shortest path in Social Network example, See Section 4.2 for deeper discussion of the results. The x and square ticks benchmarks with and without flavor text respectively.
of ReCogLab the ability to change different aspects of the dataset easily. Figure shows the Social Network problem with various different perturbations.
For each graph problem generated by Social Network, we ask two questions. The first is, given two nodes of the graph, what is the Number Hops of the shortest path from one to the other. The second is a harder version of the problem. what is the Exact Path of the shortest path. For Exact Path, we also include instructions to format list answers as a Python code block since all models demonstrate strong proficiency in generating Python code. We ablate several factors among graph problem such as the use of flavor text, and the question type in Figure 5
Flavor text is a problem augmentation where the presentation of the graph information becomes more complex. This has two effects, increasing the number of input tokens while decreasing the information density per token. We observe in Figure tha in the majority of models, flavor text obstructs reasoning performance. Barring GPT-4o, this difference suggests that language of information has critical role in solving the problem.
We also explored difference between the same context asked with a different question per example. Exact Path is a strictly harder problem than Number Hops, the solution to Exact Path also solves Number Hops. However many language models, Gemini Pro, Gemini Flash, and Mixtral-7x22B show better performance on the harder problem Exact Path. An initial explanation for this behavior lie in the number of decoded tokens which is higher for Exact Path.
Complexity and Filler Text, Another question that comes up when we examine the scaling results of Figure Blis what is actually making the problem difficulty scale. There are two separate things happening. In Social Network, as we add entities/nodes to the graph, we not only make the reasoning problem more difficult because the model has to make more logical deductions, we also increase the
7

---

Published as a conference paper at ICLR 2025
Germe 28
Gemma 9B
Gemma 278
Gemini Flash
Gemini Pro
-- Random lest
Accuracy
Number of Input Tokens
Figure 6: Filler text. Performance of different models using different kinds of filler text to increase context length. Random text fills context with public domain books, entity filler adds distractor sentences about entities but do not effect the answer, and node sweep is base experiment with an increasing number of entities.
G4140 MOSSCCT MOSEOS
Figure 7: Transitive Inference and Context Ordering. We evaluate the effect of context dering on transitive inference performance. Sim- ilar patterns observed in human cognition, when the context is presented in topological or- der, all language models improve in relational reasoning capabilities over random. Even re- verse topological order improves results as the statements are still in some sorted order.
Figure 8: Congruent and Incongruent reason- ing. We find that transitive inference is strongly impacted by real-world priors. Across differ- ent language models, we observe degradation in reasoning performance when presented with logically consistent, but incongruent statements. The Random Name ablation removes any connec- tion between the graph problem and real world priors control.
context length of the problem which makes more difficult for the model to identify the important context.
To further analyze this, we do an experiment where we separate out these two effects. We take the Number Hops task from Social Network and show the effect of adding additional context several of the models. In the first version, we simply increase the number of entities, increasing both the complexity of the problem and increasing the context length. In the second version, add more distractor "flavor filler text" about the entities that are irrelevant to the question. In the final version add completely irrelevant text like lines from public domain novels to create more input context.
In Figure 6 we plot accuracy as a function of the number of input tokens for an even comparison pretty clearly that adding graph
across these different ways of increasing context length. We complexity is by far the hardest for models than either kind of filler. Interestingly, we do see that there a real drop in performance when add any extra tokens, including filler, but the drop off much less steep. We also note that in several of the models such as Gemini Pro, the random filler context degrades performance up certain point before plateauing, suggesting that needle-in needle-in-the-hare th-thhhhhystackk type filler problems have limit to how much they can make problems harder by simply adding irrelevant context.
4.3 COGSCI INSPIRED EXPERIMENTS
Ordering Bias. Transitive inference is possible for humans even when the temporal order differs from the symbolic ordering (e.g. "B > C and A > B"). However, order affects sequence memory in humans (Domjan 2010), and when the temporal order and symbolic order match, performance should in principle improve. Thus, about associations presented in context, and measure whether there was a dependence on presentation order in their context across different domains.
wanted to evaluate models' ability reason accurately
8

---

Published as a conference paper at ICLR 2025
In Figure 7 consider several ordering experiments on Comparison and Syllogisms. In particular, we construct linear chain of directed entities and order the context according to the linear chains' topological order. This means that adjacent sentences always include at least one entity in common.
For Comparison, we can clearly see across nearly all models that randomizing order causes a noticeable degradation. This implies that topologically sorting statements can improve relational reasoning performance on the same underlying problem.
Congruent and incongruent. Despite only being trained on textual data, language models exhibit remarkable knowledge about real world physical attributes of objects. This can be easily demonstrated when asking questions like: Are fire trucky larger than teddy bears?. This prior knowledge is aligned with facts found in the training data and is essential to many applications of language models.
Motivated by the findings of Lampinen et al. (2024). who report that logical reasoning is more accurate when logical premises are congruent with real-world facts for LLMs (as it is for humans), we explore this in the relational reasoning regime. Expanding on their paradigm, we categorize premises based their grounding in real- world knowledge. Congruent relationships are those that are consistent with the real world (e.g "whales are larger than minnows"). Conversely, incongruent relationships contradict our prior knowledge about the world ("minnows larger than whales"). Finally, we consider a third category of relationships involving contrived, nonsensical objects ("3x0Bn are larger than hJp65") that have no meaningful semantics behind the name. Intuitively, congruent relationships are more likely to be consistent with the prior training of the language model and may bias the answering of these questions.
First, we investigate the impact of congruence on language model performance, replicating the observation that congruence contributes positively to performance.
For the Comparison task, to extract real-world knowledge about objects, we curated list of 540 objects and asked Gemini Pro to estimate their size in kilograms and meters. From this, we construct comparison problems that are Congruent and Incongruent. We also generated Random-String version of the comparison problem which replaces all object names with a random string of alphamu- meric characters of length 5 for the contrived, nonsensical bascline.
We find that using Congruent statements outperforms similar comparison problems constructed with Incongruent statements. This behavior is consistent across language models and relational reasoning tasks. Some models even exhibit worse- worse-than-chane than-chanceee (0.5) correctness on Incongruent comparison problems. Random-String performance lands between the two extremes, suggesting positive relationship between reasoning competency and factual coherence.
Identifying Inconsistent Premises. One important capability of a reasoning system identifying when premise is logically inconsistent (Black et al. 1986 Johnson-Lairt et al. 2004). This reasoning task is different from the previous structured reasoning tasks as identifying the set of inconsistent facts requires global reasoning capabilities. Furthermore this question does not incorporate any specific entity name so there's 's no prior on which statements from the context are relevant; and there are no rules on organizing or processing statements that trivialize this task. Therefore all statements must be equally considered before a conclusion of logically consistent can be drawn.
Here, logically consistent directed graph assumes cycles exist. If A B, B C. then you cannot have C > A. This implies that all inconsistent graphs are comparison graphs that contain a cycle. To create inconsistent examples, we start with valid comparison directed acyelic graph (DAG) and randomly sample nodes to add an edge that would induce a cycle and turn the problem inconsistent.
We show the confusion matrix on predictions extracted from each L.LM on the first of Figure Comparison across diagonal elements indicate that larger models perform better on detecting incon- sistent statements. We provide deeper analysis on the prediction calibration issues in Appendix C.2
Indeterminate reasoning. Another important aspect of reasoning is detecting when there's s insufficient information to draw a conclusion. This relates to an ability often ascribed somewhat uniquely to humans: our ability to characterize uncertainty (Courville et al.) 2006). Being able to detect indeterminacy is important for several reasons. First, a reasoning system that fails to identify an indeterminate premise may draw erroneous conclusions that are not supported by the evidence.

---

Published as a conference paper at ICLR 2025
Detecting Statement Consistncy
0.0G7
0.622
5.897
BLLED
0.000
6L083
0.335
Extta
6:313
0.204
GU9R2
6:0P6
6.109
Detecting Indeterminatee Conclusior
6.038
D:0L
0.442
10.02
0.827
0.000
ELUSE
5.00P
0.000
0.155
00.00
S:LAT
8:391
0.000
5.723
6.023
0.300
0.370
0.830
3.042
0.000
0:259
0.892
$,750
0.933
0.000
6:947
6:347
0.609
0.947
D.99
0.033
000
0.109
0.689
0.087
D.072
0.939
0.000
Gurena 20
Guntr
Ourrin
Figure 9: Confusion matrix on meta-reasoning. ReCogLab allows us to probe language models for important meta-reasoning capabilities. The first row shows confusion matrices on classifying consistent and inconsistent premises. We find later generations of models iterations can better identify statements that are inconsistent. We also find a similar trend when inferring whether conclusion indeterminate. These are important meta-reasoning capabilities with critical implications for designing human interaction and autonomous decision-making systems.
Second, recognizing when a conclusion cannot be drawn is important for improving decision making and handling ambiguity.
We start by generating a random DAG tree of comparison problems. Under a DAG tree, querying comparison between two nodes may result in an indeterminate conclusion if the two nodes no longer have a path to each other. In other words, insufficient context was provided to reason about the relationship between the two. This results in the modified answer set consisting of three potential answers (Yes, No. Unknown). For this particular evaluation, we provide instructions to answer whether the context and question are indeterminate before asking a question. While this experiment follows by combining cognitive science experiments about uncertainty estimation and transitive inference, it has not to our knowledge been performed in the cognitive science literature.
We show results of this probe in the second row of confusion matrices in Fig. 9 Similar to the previous probe, the diagonals are correctly classified examples. We also analyze calibration biases in Appendix C.2 and find that many language models disproportionately prefer to answer Yes. This has strong parallels to "acquiescence bias" towards answering affirmative that has been documented in humans in variety of settings (Krosnick 1999..
5 DISCUSSION
We introduce ReCogLab, flexible, generative framework for quantitatively measuring relational reasoning abilities LLMs. We demonstrate the utility of this approach, conducting a number of experiments to benchmark relational reasoning performance across different models and problem complexities. Moreover, we recreate cognitive science experiments that characterize how reasoning performance depends on features such presentation order (Hotta et al.) 2015), symbolic distance (Moyer & Bayer 1976), and congruency with prior experience (Lampinen et al. 2024). We also create a number of novel experiments that measure how models recognize inconsistencies and indeterminacies. an important contribution for model safety. Ultimately, we hope that by accurately measuring the cognitive capabilities of our language models, we can contribute metric for hill- climbing improvements in the field that are inspired by human cognition. Furthermore, we hope that this work will inspire deeper, rigorous probes into reasoning capabilities of both humans and artificial agents, and help provide insight into the success and diagnosing failure modes of reasoning.
Many of the effects we observe in LLMMs mirror effects observed humans, suggesting commonalities in the factors contributing to relational reasoning in LLMMs and humans. While it is instructive to compare and contrast LLM reasoning to that of humans, we should as always be careful about drawing too many conclusions about the similarities between LLMs and human psychology (Shevlin & Halina 2019 Shanahan 2022). Even when there are similarities, there is ambiguity about the explanation: it may derive from similarities in the statisties of experienced data, task objectives, architectures, or learning rules. Alternatively, it could be that two entirely different mechanisms are responsible for the same effect.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: bilingual, multi- task benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- lics (Volume I: Long Papers), pp. 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL
2024 acw-loon--... .72
Timothy E J Behrens, Timothy H Muller, James C R Whittintoo, Shirley Mark, Alon B Baram, Kimberly Stachenfeld, and Zeb Kurth-Nelson. What a cognitive map? organizing knowledge for flexible behavior. Neuron, 100(2):490-509, October 2018.
Jonathan Berant, Andrew K. Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-anse answer pairs. In Conference on Empirical Methods in Natural Language Processing, 2013. URL httpst:::: / appp..i...//////// swwant wwt.manniisccoola...... Dtt.aaaar..or/coorppsDDD.o00....
Lukas Berglund, Meg Tong. Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on" is b" fail learn" b is a". In International Conference on Learning Representations, 2023.
Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3, Proceedings the National Academy of Sciences, (206))e22185231200 2023.
Alison Black, Paul Freeman, and Philip N Johnson-Laird. Plausibility and the comprehension of text., 1986.
Cyril Burt et al. Experimental tests of higher mental processes and their relation to general intelligence. American Psychologist, 1911.
John Ceraso and Angela Provitera. Sources of error in syllogistic reasoning. Cog-
nitive Psychology, 4(4):400-410, 1971. ISSN 0010-0285. doi: htps://doi.org/10
1016/010-0285711990225. URL https://www.scttt http:://wwwwwccee.e.........c rect. ret.eeerrrerrrrett.oooo/sieeeree
atticlee/p1100100225119000
Nick Chater and Mike Oaksford. The probability heuristics model syllogistic reasoning. Cognitive Psychology, 38(2):191-253 1999. ISSN 0010-0285.
doi: https://doi.org/10
1006/cogp. 1998. 0696. URL htt..:/:///ww....ee..eeeeeeeeeee https//sccc/ccceeec///t/tttioooo 011/8001002859890553
Xinyun Chen, Ryan Chi, Xuezhi Wang, and Denny Zhou. Premise order matters reasoning with large language models. In International Conference on Machine Learning. 2024.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over lan- guage. In International Joint Conference on Artificial Intelligence, 2020a. URL https
//api wewaemetttii wwweenmticcccllaa. wtt.ecccccc..a.oooo/oorooopsaap.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers soft reasoners over language. CoRR, abs/2002.05867, 2020b. htttps: hww.:/orgggaaa. httpg/ab//22022 /ttp:p////arri.oooo/////2/22.. 05867
NJ Cohen. Memory, amnesia and the hippocampal system. MIT Press, 1993.
Aaron C Courville, Nathaniel D Daw, and David S Touretzky. Bayesian theories of conditioning in changing world. Trends in cognitive sciences, 10(7):294-300, 2006.
Andrew Cropper and Sebastijan Dumančić. Inductive logic programming at 30: new introduction. Journal of Artificial Intelligence Research, 74:765-850. 2022.
Michael Domjan. The Principles of Learning and Behavior. The Principles of Learning and Behavior. Belmont: Cengage learning, edition, 2010.
Howard Eichenbaum. Hippocampus: cognitive processes and neural representations that underlie declarative memory. Neuron, (111:1099-100 2004.
11

---

Published as a conference paper at ICLR 2025
Tiwalayo Eisape, Mh Tessler, Ishita Dasgupta, Fei Sha. Sjoerd van Steenkiste, and Tal Linzen. A systematic comparison of syllogistic reasoning in humans and language models. In North American Chapter of the Association for Computational Linguistics, volume abs/231 0oos//31.00045,, L00445, 2024. URL https: / app. semant iommanttccccon ccschol ar wwt.oorg/ooooo 264
JSBT Evans, Julie L. Barston, and Paul Pollard. On the conflict between logic and belief in syllogistic reasoning. Memory & cognition, 13))2955-306 1983.
Kanishk Gandhi, Jan-Philipp Franken, Tobias Gerstenberg, and Noah D. Goodman. Understanding social reasoning in language models with language models. In Advances in Neural Information Pro- cessing Systems, volume abs/2306. 15448, 2023. httth::::: // esemanticschle a
brg/CorpusID: 2592622::552257
Mona M Garvert, Raymond Dolan, and Timothy EJ Behrens, A map of abstract relational knowledge in the human panoocampal-entrhiine cortex. eLife, 6xe 17066, apr 2017. ISSN 2050-084X. doi: 10.7554/eLife. 17086. URL httthtts: //do:s:://d.... org/10../.... 2554//eeiffe 17086
Martin J. Guyote and Robert J. Sterbberg. A transitive-chain theory of syllogistic reason- ing. Cognitive Psychology, 13(4):461-525, 1981. ISSN 0010-0285. doi: https://oii.org
10110160010--22$88119000003
https:htss:///tet www.::sceenneee.. rect com/ ccom/sccince... article/ps11001000288TBB
Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanic C. Y. Chan, Andrew Lampinen, Jane X.
Wang. Zeynep Akata, and Eric Schulz. Machine psychology, 2024. URL https: :ettps::/arxe //axxiv/
org/abs/ 2303. 13988
José Hernández-Orallo. The measure of all minds: evaluating natural and artificial intelligence. Cambridge University Press, 2017.
Takashi Hotta, Lyndon A. Jordan, Tomohiro Takeyama, and Masanori Kohda. Order effects in transitive inference: does the presentation order of social information affect transitive infer- ence in social animals? Frontiers Ecology and Evolution, 2015. ISSN 2296-701X. doi: 100.3389/fevv22155.00
URL https: //www. front iersin wwweersiinorrr.. wowg///ooa///ool..... aeoology-ann-eeooouuooooorrttaaa /3t.l//eee....///eeee//e/ 2015 00059
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Johnson-Laird, Paolo Legrenzi, and Vittorio Girotto. How we detect logical inconsistencies. 2004.
Erik Jones and Jacob Steinhardt. Capturing failures of large language models via human cognitive biases. Advances in Neural Information Processing Systems, 35:11785-11799, 2022.
Samira Ebrahimi Kahou, Adam Atkinson, Vincent Michalski, Ákos Kádár, Adam Trischler, and Yoshua Bengio. FigureQA: An annotated figure dataset for visual reasoning, 2018. htRhtt
//openreview. www.////////mmm iid SyunbfbAb
Mchran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. Boardgameqa: A dataset for natural language reasoning with contradictory information. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 39052-390177 Curran Associates, Inc., 2023. URL httppp::/proceeeding...eeeee neuripg....eeus.p. httpccccpppp__i___1 Paper Datasets.
and Bww.BBnnchhaakse pdf
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking bench- marking in nlp. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 41 411044124, 2021.
12

---

Published as a conference paper at ICLR 2025
Raphael Koster, Martin J. Chadwick, Yi Chen, David Berron, Andrea Banino, Emrah Düzel, Demis Hassabis, and Dharshan Kumaran. Big-loop recurrence within the hippocampal system supports integration of information across episodes. Neuron, (0100)01322-3344..0 Sep 2018. ISSN 0896- 6273. doi: 10. 10.1016/j.mo /0to///.eeuro.20000 2018.08.000 [20 8 .0.008.00 00%)
URL httpt:ttt/////e httpp/////ooo.rg//////ee n neuron
Jon A. Krosnick. 1999):537-567, 1999. ISSN 1545-2085. doi: http:///tiioooo/o...o//aommaaaam 144 1.537. URL. https:: //w::///wwwwww http///www..aaaaaa.a.aa//aaaaaaw annual reviews wtwp://www.org/coteet/////rrrrae
Survey research. Annual Review of Psychology, 50(Volume 50,
annurev. . psych. 50.1.537
D Kumaran and E Maguire. Knowing your place: Navigation in spatial and social networks. In Journal of Cognitive Neuroscience, pp. 100-100. MIT PRESS, 2005.
Andrew K Lampinen, Ishita Dasgupta, Stephanie C Y Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James McClelland, and Felix Hill. Language models, like humans, show content effects reasoning tasks. PNAS Nexus, 3(7):233, 07 2024. ISSN 2752-6542. doi: 10. 10.1999//paasccuu//paa2233 URL https: :tap::://dda doi www.////..00...... 1093/ wwt3//passexuuuaaapaaaaa /pgae233
Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. Complex knowledge base question answering: A survey, CoRR, abs/2108.06688, 2021. URL https
7/arxiv. org/ 2100.11011.. 06688
Matthew Le. Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory of mind through question answering. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( (EMNLP-IJCNLP), pp. 5872-5877, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19-11
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Annual Meeting of the Association for Com- putational Linguistics, 2024. URL httpt::/paappi http://///appisssnniiiisssseeeie remanticschon
org/ CorpusID
267897954
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag. Theo Gutman-Solo Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. fttpse__lleesspppeeeeeeeee ciates, Inc., 2022. URL UtL/httss:/pproceeeeii neurips ccpparer Paap:a/aeeeecttieaaaaaaaeeoeeeee htt.///aeeeccceeaetooeeeeeeeeeer
Curran Asso-
.ddf
Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can llms do retrieval and reasoning in million context window? ArXiv, abs/2407.11963. 2024a. URL httth://p///pp
ar. . org/ Ctwpp//ooouuuuua
Yanyang Li, Shuo Liang, Michael Lyu, and Liwei Wang. Making long-context language models better multi-hop reasoners. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pp. 2462-2475. Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.186533w1/2024..22 aclno 135. URL httph: http::::/aallaatthooooooo org.
135
Zhiming Li, Yushi Cao, Xiufeng Xu, Junzhe Jiang, Xu Liu, Yon Shin Teo, Shang- Wei Lin, and Yang Liu. Lims for relational reasoning: How far are we? In Proceedings of the 1st International Workshop on Large Language Models for Code, LLM4Code '24, pp. 119-126, New York, NY. USA, 2024c. Association for Computing Machinery. ISBN 9798400705793. 3643795.3648388 URL
doi: 10.1145/ 1145/3643799 3648387
Vanessa Lopez, Christina Unger, Philipp Cimiano, and Enrico Motta. Evaluating question an- swering over linked data. Journal of Web Semanties, 21:3-13, 2013. ISSN 1570-8268. doi:
13

---

Published as a conference paper at ICLR 2025
http://dd..rr///lll.l.oeoeeeeeee URL httpl:ttts:s///tt https//www....eeeereettttt com/ bcciaaees/arrccee///////27768822 Special Issue on Evaluation Semantic Technologies.
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. In International Conference Learning Representations, volume abs/2310.01061, 2023. URI.https:/ //ppi semant insmantticccoo... httc/CorpuuTD:223605533
Christopher W Lynn and Danielle Bassett. How humans learn and represent networks. Proceedings of the National Academy of Sciences, 117(47):9447-2299) 2020.
Shirley Mark. Rani Moran, Thomas Parr, Steve W. Kennerley, and Timothy E. Behrens. Transferring structural knowledge across cognitive maps in humans and models. Nature Communications, 11 (1):4783, 2020. doi: 00.1038/441467-0222 18254-6. URL /tttpp///doll.orrr000100033 S41 111467-220-88553
Robert S. Moyer and Richard H. Bayer. Mental comparison and the symbolic distance effect. Cognitive Psychology, 8(2):228 246, 1976. doi: 10. 1100000000005566699000000 |
//doi.org/10. htt.://or01..1110000000000000010 0285 (76) 90025-6
Stephanie Nelli, Lukas Braun, Tsvetomira Dumbalska, Andrew Saxe, and Christopher Summerfield. Neural knowledge assembly in humans and neural networks. Neuron, 111(9):10441516644 May 2023.
Jean Piaget. Logic and psychology: Logic and psychology. Basic Books, Oxford, England, 1957.
Anna C Schapiro, Timothy T Rogers, Natalia I Cordova, Nicholas B Turk-Browne and Matthew M Botvinick. Neural representations of events arise from temporal community structure. Nature neuroscience, 16(4):486-492) 2013.
M. Seals and Valerie L. Shalin. Long-form analogies generated by chatgpt lack human-like psycholinguistic properties. In Annual Meeting of the Cognitive Science Society, 2023.
Murray Shanahan. Talking about large language models. arXiv, pp. 1-11, 2022.
Henry Shevlin and Marta Halina. Apply rich psychological terms in ai with care. Nature Machine Intelligence, 1:165-167, 2019.
Edward JN Stupple and Linden J Ball. Belief-logic conflict resolution in syllogistic reasoning: Inspection-ting evidence for a parallel-process model. Thinking Reasoning, 14(2):168-181, 2008.
Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, and Douwe Kiela. Dynatask: A framework for creating dynamic ai benchmark tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 174-181, 2022
Edward C. Tolman. Cognitive maps rats and men. Psychological Review, 55(4): 1948.
ISSN 1939-1471.
doi: 11100700006626 URL
doi doi=10.2 1037 10..1003//00066222 hh066626666
Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, Rohan Anil, Ethan Dyer, Siamak Shakeri, Roopali Vij, Harsh Mehta, Vinay Ramasesh, Quoc Le, Ed Chi, Yifeng Lu, Orhan Firat, Angeliki Lazaridou, Jean- Baptttttttte Lespiau. Nithya Attaluri, and Kate Olszewska. Michelangelo: Long context evaluations beyond haystacks via latent structure queries, 2024. URL https:
wlaaxiv.orrg//o abs wttp://r///b///299....// 12600
Konstantinos Voudouris, Matthew Crosby, Benjamin Beyret, José Gamanndez-Oralle Murray Shana- han, Marta Halina, and Lucy G Cheke. Direct human-ai comparison the animal-ai environment. Frontiers in Psychology, 13:711822 2022.
14

---

Published as a conference paper at ICLR 2025
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with MATH-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024, URL htt.ttttt///ppeeeeeeeeee ew net
Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 199::122-1551, 2023.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. Towards ai ai-complete question an- swering: A set of prerequisite toy tasks. In International Conference on Learning Representations, 2016. URL httpt::t/////m /appp.pp./// www.nticschoooe..eo....o wwtp/oopoo//......../o 31 3178759
Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. Language models as inductive reasoners. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume Long Papers), pp. 209-225, St. Julian's Malta, March 2024. Association for Computational
URL StLhttps::o httLLttpp////aaaaaooooooo//////p www.::2...........e
13/
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: reading comprehension dataset requiring logical reasoning. In International Conference Learning Representations, 2020. URL
attpps:/cppeneet net/ forum? foormm?dd-HJgjtTte-.
15

---

Published as a conference paper at ICLR 2025
A PROMPT TEMPLATE VALIDATION AND MODEL CAPABILITIES
While prompt strategies are an important line of research for exploring language model capabilities, they introduce additional variance to observations of language model's 's true problem-solving capabilities. From a scientific point-of-view, this is problematic; optimizing prompt templates (even if they are O-shot) for specific tasks may yield better hill-climbing results, but they make it harder to transfer our understanding
language model's ' general capabilities to a different task.
To mitigate this issue, treat prompts template and answer parsing hyper-paramann parameters fit validation set first, Because ReCogl Labis generative framework, we can generate always generate validation sets find the best prompt template strategy for each and every language model. This means that every language model is given the opportunity to figuratively "play to their strength."
In experiments we only consider "O-turn" and "O-trained" prompt strategies to preserve the generalization of our results. We also use several prompts that targeted specific sub-datasets or tasks like Comparisons, Syllogism, and Feasibility because their setup requires more explanations. We show the prompts we tested below. [question] refers to both the contextual information and task-specific question combined.
Common Prompts Templates used in all tasks.
[question]
[question]
Answer in only one word.
[question]
Think through your answer then respond at the end with a newline and 'Answer:' with your answer.
[question]
Think through your answer then respond the end with a newline and 'Answer:' with your answer. Use only one word for the answer.
[question]
Let's think step-by-step
Social Network Prompts Templates
You are a language model with advanced cognitive abilities. Your task is to understand and reason about the following social scenario, much like human would. Read the story carefully and answer the questions that follow. [question]
Comparison Prompts Templates
[question]
Answer the above relational reasoning question with Yes or No. Use only one word for the answer.
You are language model being probed for your reasoning abilities. Your task is to carefully think about the following information and answer the question. [question]
Make sure to respond at the end with 'Answer:"
[question]
Answer the above relational reasoning question with Yes or No with a newline and 'Answer:' with your answer. Give your best guess if uncertain. Use only word for the answer.
[question]
Answer the above relational reasoning question with only Yes or No with a newline and 'Answer:'. Give your best guess if uncertain. Use only one word for the answer.
[question]
Answer the above relational reasoning question with Yes, No. Unknown. Use Unknown if the question cannot be answer with the information given. Use only one word for the answer.
16

---

Published as a conference paper at ICLR 2025
We find that after validation, different models from the same family of model classes perform better with different prompts.
This framework of splitting prompts and model capabilities works for more complex patterns of cognition like Chain-of ohoutttth and In-context learning. We also consider sentence parsing but that has much weaker impact performance and is also significantly cheaper test since they re-use the same language model base prediction. We will release the prompt and sentence parsing functions when releasing code.
B FRAMEWORK DETAILS
We discuss further details about ReCogLab.
B. GENERAL OVERVIEW OF FRAMEWORK
Here we explicitly describe how the framework constructs training examples to support different experimental hypotheses. We will make the entire framework codebase and dataset available the public.
Our framework works by:
1. Creating config file for the dataset you want to create. This includes the task and sub-task (see the longer task descriptions in Section B.2) and the task parameters (see Section B.3)
2. The user passes in this config as well as the dataset split and the random seed to the dataset generator class.
3. Based on which task is chosen, the dataset generator for the corresponding task generates the underlying logic the graph problem (e.g. for Social Networks it generates an undirected graph) and generates the context C text, question Q and answer(s) A from templates. The context is made up of a list of relationship R,j between pairs of entities (E,, E,) (each of which is a person or object or other noun in the relationship
be attribute). The templates then fill the text to describe the relationship Ray (often sampling from number of possible templates) and sampling entities (F,, Fly) fill in. The generator also enforces and uses all of the task parameters to do this generation. For details on each of these constructions
the corresponding paragraph of Section B.5
4. The final example is then saved and the generator continues to generate up until the desired dataset size
5. For experiments which "sweep" parameters, the dataset generation of N examples is repeated, changing the value of the sweep parameter each step generate all Cp parameter configurations.
B.2 TASK DESCRIPTION
Comparison
Older-Younger: Constructs comparisons on age between people.
Larger-Smalles Constructs comparisons on size. These can incorporate congru-
ent/incongruent knowledge priors.
Heavier-Lighter: Constructs comparisons on weight. These can incorporate congru- ent/incongruent knowledge priors
Consistency Detection: Instead of asking comparison question, asks whether the statements are logically consistent.
Generate comparison tree and ask whether the statements support drawing a conclusion.
Indeterminate Conclusion:
Social Network
Fastest Message: Given the goal of passing message from person A person B with the fewest hops, who should person B give a message to. Reduces to a logical breadth first search.
17

---

Published as a conference paper at ICLR 2025
Given statement about family, who is the oldest and youngest generation? Similar to Older- Younger.
Oldest/Youngeet Generation:
Syllogism
Given statements describing group membership between different labels, determine how two labels' members intersect.
Set Membership:
JSON Families
Family Size: Calculate how many members in specific family.
Family Member Hobby: Given a hobby, check whether is hobby of specific family member.
Family Size Comparison: Given two families, compare their size.
Hobby Comparison: Given specific individuals in different family, describe lapping hobbies.
Age Comparison: Given two specific individuals in different family, compare age.
B.3
TASK PARAMETERS
network.typp type: tree.
Whether to construct the task using a linear chain or randomly generated
num_entities:
The number of entities used when generating experiment. We use this for
evaluating capacity performance.
entity-tyy -tye: The type of entities construct word problems out of. These consist of names, objects, labels depending on the need of the task.
congruence: If the task supports integrating real world priors to generate congruent or incongruent statements.
nundomize.rclatoons For directional statement, whether to reverse the statement and relation. A > B becomes B AA in the word problem.
do do.reeeree.oomps To evaluate whether the order of the statement in the question matters.
relation.type. For social network, applies labels to the edge which include other kinds of relationships.
num families: Equivalent to num entities, measure of capacity for length (total number of families).
num hom_fammly_mnn members: Equivalent to num_entities, measure of capacity for width (max members within family).
hop length: Distance between two families chosen. If not set, default is sampled randomly between half the number of families overall number of families.
B.4 COMMON TECHNICAL DETAILS
We use NetworkX Python library to construct randomly generated graphs. To seed deterministic randomness, we use Jax's PRNG Keys to ensure that each example isolated from each other while still being fully reproducible given the correct key. We used Jax v0.4.33 randomization implementation since the PRNG key behavior is consistent within a version.
Our experiments generate linear chain graphs random tree graphs. A chain graph is sequence of nodes where each node is connected to the next one in a linear sequence. This allows us to test for ordering effects in the context because answering the question of a linear chain graph is equivalent to traversing the statements sequentially. Other experiments generate more complex graphs like random trees.
We incorporate multiple stages of deterministic randomness, first at initialization to help randomly configure parameters shared across all test examples, and next per-test example generation.
18

---

Published as a conference paper at ICLR 2025
B.5
DOMAIN-SPECIFIC DETAILS
Comparison
We use NetworkX to generate linear chain and random trees depending on the sub-
task.
Comparison sub-tasks consist of asking questions about a bunch entities in three comparative settings, Size, Age, and Weight. Consistency Detection and Indeterminate Conclusion use the Age comparison but ask a modified question.
For evaluating the ordering effects, we use linear chain graph shown in the contextual premise ordering experiments in 7 Here inorder and reverse corresponds to matching the toplogical sequential order of the linear chain. Random simply randomizes the ordering. We also use linear chain for Congruent and Incongruent reasoning as well as the Symbolic Distance probe as they simplify the problem construction.
For indeterminate reasoning and inconsistent detection, we use random trees. This is because creating an inconsistent premise requires making a self-loop which is trivial to identify in a linear chain. It's also impossible to construct unknown infeasible questions with a linear chain because every pair of object has relationship.
Social Network We use NetworkX to generate linear chain graphs for evaluating capacity.
We evaluate on the sub-task of Fastest Message which is a specific question that asks who should [Entity 1laass a message have it arrive to [Entity2]in the fewest hops.
To construct a Fastest Message problem, we generate premises about the nodes in the graph where edges denote friendship and way for a message be passed. We randomly pick two entities and ask how to pass a message. Here the answer can be determined through breadth first search.
Oldest/Youngent sub-task reformulates the social network graph directed problem where nodes indicate relationships between parent and child. The oldest/youngest generation asks which family member is part of the oldest or youngest generation. We can solve and verify the answer using combination of common-ancestor graph algorithms. While we didn't report benchmark performance on this task, we note that oldest/youngest has several interesting, distinguishing properties from other tasks presented in this paper. First it does not incorporate any entity names in the question which might inform the language model how to search the context. It also requires processing every bit of information before being able to verifiably prove the answer.
Syllogism We use wrrttenwritten logic to generate syllogisms with an arbitrary number of propo- sitions. For a given number of propositions, we keep track of the current valid conclusion, and enumerate all new propositions which, along with the current conclusion, generate a new valid conclusion. This is done in depth-first way, and when hit the desired number of propositions, we yield the current chain.
We evaluate the model by showing it all of the propositions in the chain, as well as all syllogism types with the valid conclusion's 's subject and predicate, i.e. "All A are Z," "No A are Z," "Some A are Z," and "Some A not Z."
To study ordering effects, we sort the propositions by constructing a Hamiltonian path between the proposition which contains the subject of the conclusion, and the proposition which contains the predicate of the conclusion.
JSON Families Inspired by knowledge based question answering (Lan al., 2021) on structured problems, we also consider how relational reasoning is impacted by the use of programming language input like ISON. We created generating process for document graphs where elements in the JSON nodes in the graph that connect to the parent object. We use this to model knowledge and generate testable questions like whether two individuals have the same hobbies.
For the JSON Families task, our framework produces structured, nested JSON representations of multiple families. Each family is identified by last name, address, and a set of members. Each member is then identified by their name, age, and hobbies. Table outtlnnes the kind of questions i.e. probes that can be be conducted on this dataset. The question types range from simple fact retrieval (e.g., family size) to complex comparisons (e.g., relative age of members across different families or
19

---

Published as a conference paper at ICLR 2025
shared hobbies). This structured data allows to easily scale the number of families well as the members per family.
During the dataset generation process, we ensure that the entity in the question is uniquely identifiable (e.g. no two families share the same combination of name and address) and if there are multiple possible answers, check against all of them for determining correctness.
Sub-tasks for JSON Families
Question Type
Number of entilies
Question Details
Family Size
Single Family
Finding the size of a given family in the context
Family Member Hobby
Single Family
Checking a given hobby is bobby of specitic member from specific family
Family Size Comparison
Multi Family
Comparing size of two given familics
Family Member Age Compurison
Multi Family
Comparing age two members from given families
Family Member Hobby Comparison
Multi Fumily
Comparing hubbies of two members from two given families
Table I: Family Question Types
B.6 ENTITY SOURCES
In addition to constructing the problem, we also provide a list of pre-defined entities to populate the problem with. We prepare train-val-test splits entity names. For names, we dataset of 258k popular baby-names? We use this for both Social Network and Comparison-Age.
For comparing objects, we collected list of 540 commonly encountered physical objects like "Firetruck" and "Shoebox." We used Gemini-Pro to generate many such candidate objects with mass and size that is consistent and easy to measure. We then asked Gemini-Pe to estimate their size and weight to allow us to construct Congruent statements based on their estimated physical properties.
For incongruent samples we simply reverse the congruent ordering.
We also use random-string configuration for comparison to test the effect of prior knowledge on congruent and incongruent relationships. Specifically we randomly generated alphanumeric ids of length 5 to replace the name of each object in the comparison. This controls for the effect of prior knowledge as entity names like 3Am40 and gj BBx have no semantic priors in the language model.
For Syllogism we prompted an LLM to generate diverse set of plural nouns. The entities are randomly assigned when generating the syllogisms.
For ISON Families, the entities include family names (first, last names), address and details of every member. These are randomly chosen from a set of predefined names, city names, states, postal codes and hobbies. These lists were generated by prompting an LLM to provide a commonly used entities, following which a random combination is chosen when identifying a family a member.
B.7
MISC DETAILS
Heuristic Rejection Sampling Because ReCogLab is data generating process, do not have precise control of the posterior distribution of examples. This issue becomes prevalent if the goal is to slice the posterior distribution understand the behavior
specific subset of generated problems.
One example where we used heuristic rejection sampling is in symbolic distance experiments. Because ReCogLab generates the graph problem first and then testable question, some kinds of testable questions are impossible to generate given a specific graph problem. To make this concrete, we tested the symbolic distance effect on a random tree configuration of social network. However once the configuration of the random tree has been sampled, randomly sampling two entity nodes to construct question will tend to produce low symbolic distance questions. This means that high symbolic distances are hard to find in this data generating process. Another example of a rebalancing we use is to generate comparable numbers of determinate and indeterminate examples since in large graphs, indeterminate examples are easier sample than determinate ones.
Source: http://wwwssspeeeece.t//meeaaaas
20

---

Published as a conference paper at ICLR 2025
To prevent oversampling, use heuristic rejection sampling discard generated examples that have a common property and increase the proportion of rare-pra aar--proerty questions. In particular, we annotate metadata like symbolic distance or the answer associated with each example and record histogram of annotated metadata over examples in the generated dataset. If a newly created example has metadata that is twice common the least common metadata above certain minimum threshold, we discard that example. Because generating evaluation examples is essentially free, we can simply resample an example until it has metadata properties that we care about.
We choose this approach of balancing test examples since it doesn't I require specific implementation, as long as process has a probability of generating an interesting configuration we can upsample it's occurrence in the dataset. Additionally some notions of parameters share common meaning but have wildly different implementation strategies. For instance symbolic distance is trivial to calculate for linear chain configuration, but impossible to control in random tree. Rejection sampling resolves this issue by allowing us to upsample rare examples without needing explicitly define how those rare labels generate.
Flavor Text Flavor text text that adds depth or background to character or relationship. This embellishment of factual statements adds an additional layer of cognitive load. Instead of simply presenting statements as facts, flavor text rewrites them to a statement that implies two entities' relationship with each other. We generate a total of 83 flavor texts using Gemini-Pa and specific instructions.
C ADDITIONAL EXPERIMENTS
C.I SYMBOLIC DISTANCE EFFECT
Symbolic distance refers to the number of relational reasoning steps that join two entities in sequential reasoning task (e.g.. if A > B >>C> the symbolic distance between A and is 3). The symbolic distance effect refers to the observation that more distant comparison judgments are, perhaps surprisingly, easier for humans (except distance 1, where observed pairs can be memorized)
Here we provide further experimental results from Fig. 4 on other tasks. In Fig. 10 and Fig. II we show all tasks and the frontier performance of Symbolic Distance and Graph Size. We can see that there's a ordinality to difficulty on these tasks, as reflected in the main paper flavor text results in harder problems and degrades performance (despite being the same underlying grpah problem). This indicates that relational reasoning is not robust against textual distractions.
The one exception to this trend is GPT-4o which shows consistent performance across the various tasks according to the symbolic distance and graph size. This implies strong robustness to textual distractions and also the ability to perform equally well between variations of the same task. There's also consistent performance between ExactPath and NumHops tasks which indicates robustness to the format of the question. ExactPath is a version of NumHops that requires the exact order of the answer.
We also see that the performance of each model is dictated by the symbolic distance first, and then the number of entities in the graph. This is indicated by all models have strong performance among problems with symbolic distance < and even when the graph size is greater than 40 entities. However smaller graph of only 20 entities does result in being able to solve higher symbolic distance questions.
C.2 META REAS-NNG FPR ANALYSIS
The corresponding figure of results Fig. For ease of referencing we have reproduced the figure
in Fig. 12
C.2.1 DETECTING STATEMENT CONSISTENCY
As mentioned before, the diagonal elements of the confusion matrix represent correctly classified examples. An additional interesting statistic to analyze is the type of classification errors each model makes. This is represented by the off-diagonal elements and is related classifiers' false positive
21

---

Published as a conference paper at ICLR 2025
Number of Hops, No Flvvr Teat
Number of Hops, Flivor Text
Exact Fath, No Filvor Textt
Exact Fath, Flavor Texct
6405
3045,
371
6:50 141
3754
0.10)
BU0K
6.99
500 37%
0:343 64
5.0%%
0000%
3.0
3:0
0.00
Cemtaa
6:19
6339
0.60%
0:000
6.00
6:113:18
50-
3:00
6.36.
0.00
6.00
6.30
0.00
6000.
040%1
0:00
0006.7
0401
03877
8385
3.99
6-16 11
6.80
0456
3937
444
0.0%
0334
8:18
633
Barma 27B
2014
611%
6471
1.94
0181
0110
1.7%
0..%
6.00
008%.11
0.00
0.00%
6.00
0180
60K. 19
9:00
0,00.
3.K
6000
9.00% 9
0003
30
20
20
15
26
0/0%
E-36 181
1.17
6.363.20
6.9%
6:30 181
6144
944 131
6385
2-14
63
519%
6.79
8:31 1
1155
2315
619%
c27
Micraal 8:78
12
cons
0.0%
5.14
6600
0.03
0.0
0000
8:00
6CM
0.00 96
6091 13
0007
0OK 11
0180
1
8.00 11
64004
0003
9.00.
0,001
38-
0,000
20-
0083
0.0%
0100
DA35
6.464
0110
0:19 199
1
1.9
0149%
517
0.0%
622h
0007
1.2% 15
0.99%
5.518
6.377
K11
616%%
3.6K 51
E 55
0221
647
0241
8137 55
6310 45
Midiral Bb228
0.00%
6199
0:300
6000
6.
6.000
0.00
0.56
6
0.004
0.05.11 12
0.000
0.000
0
6135
0200
0.00
0/00
0.51.
0400,
0.377.0
1.50%9
4.221.5
0711.
59
0533.4
0.05
0.000
6376.44
Matral Bx228
9000
0.304
1 56
6.26 37
4000 3
0.442 39
4.14 24
0,503 2
0.535 30
E413 54
0324 27
0.00.
0.000 30
LA0 55
1.54.77
60-
00031.22
1.06
24
6.00
0.90.
0.800
6.00.12
LGW
0099
0000.
2.003
6590.
0.000
BOK
0000
06901
0.00%
20
48
22
12
Figure 10: Symbolic Distance and Graph Size. Here we show the each model's performance on Social Network tasks. The y-axis denotes performance examples sliced by the Symbolic Distance required to answer while the x-axis denotes the number of entities in the problem. Each box reports the accuracy and number of test examples in the test in the slice. Red indicate better performance on a slice.
22

---

Published as a conference paper at ICLR 2025
Number of Hops, No Flavor Text
Number of Hops, Fovor Text
Exact Fath, No Flavor Text
Exact Parh, Fllvor Text
Figure 11: Symbolic Distance and Graph Size. Here we show the each model's performance on 4 Social Network tasks. The y-axis denotes performance on examples sliced by the Symbolic Distance required to answer while the x-axis denotes the number of entities in the problem. Each box reports the accuracy and number of test examples in the test set in the slice, Red indicate better performance on a slice.
Detecting Statement Consistency
Detecting Indeterminatee Conclusione Con
0.355
0.529
BALG
0.824
0.345
0.031
0:997
0.532
6.009
0.085
8.500
0.022
0.458
0.310
6,000
0.030
8.42
6.000
0.554
0:258
0.002
$.70
0453
0.000
6:347
6:477
0.655
0.047
6:033
0.333
2.055
6:100
0.085
0
6.772
0.03
0.10
Guvenu 28
Ouenr Fuch
Ounri Phy
Mitull Au/d
Figure 12: Confusion matrix on meta-reasoning, Reproduced copy of Fig. g
rates (FPR). As an example, the Gemma-2B has a false positive rate of 93.3% when given consistent set of statements.
Because our probes are zero-shot tasks, the models' predictions are inherently uncalibrated. Addi- tionally the decoded predictions are complicated joint distribution of token likelihoods that make impossible to estimate the logits of a specific parsed prediction so we cannot use ranking metrics. This presents an opportunity to investigate each models' implicit reasoning bias. For instance, the GPT-4o predictions have higher FPR for Inconsistent premises than consistent (26.7% vs 2.9%), implying that GPT-4o exhibits a preference for answering consistent. Gemini Flash predictions have the opposite bias, with higher rates of False Positive for Consistent premises that indicate preference to predict inconsistent. Gemini Pro performs the best at zero-shot statement consistency detec- tion and has relatively balanced false positive and false positive rates indicating the best predictive performance.
C.2.2 DETECTING INDETERMINATE CONCLUSIONS
We show results of this probe in the second row of confusion matrices in Fig. 12 Similar to the previous probe, the diagonals are correctly classified examples and the off-diagonal represent misclassified false positives.
For any determinate example we construct, determining if A > B is as difficult determining if B A. However we see that there's 's significant performance gap across all language models in correctly identifying determinate examples. In other words, presenting the same problem with answer Yes, but flipping the question such that the answer is now No, will cause the language model to prefer predicting indeterminacy.
One possible interpretation is that language models prefer to answer Yes. This is further supported when examining the language models bias towards answering Yes on indeterminate examples that are incorrectly reasoned to be determinate. Of note, a similar "acquiescence bias" toward answering Yes has been noted in humans in variety of settings (Krosnick) 1999..
23

---

Published as a conference paper at ICLR 2025
The immediate findings suggest that self-consistency scheme could improve logical deductions and reduce the implicit reasoning beyond instruction tuning improve detection of indeterminate logical premises is important for safety and human interactions.
bias towards affirmative answers. Longer term, pursuing strategies
D ADDITIONAL TEST EXAMPLES
Below we show additional examples generated from our framework for various tasks/sub- lusks/configurations.
D.I COMPARISON
D.1.1 COMPARISON OBJECTS
Question:
Straightener is smaller than Sugar bowl Paperweight is larger than Oyster Folder is smaller than Gemstones Shaving cream is larger than Sand dollar
Eye pin is smaller than Flower arrangement
Sand dollar is larger than Pliers
Oyster is larger than Lego
Sugar bowl is smaller than Watermelon
Drum is larger than Cello
Folder is larger than Flower arrangement
Pill is smaller than Pliers
Ceiling is smaller than Cello
Paperweight is smaller than Pill
Gemstones is smaller than Keyboard Ceiling is larger than Audio interface Straightener is larger than Shaving cream
Keyboard is smaller than Lego Apple is smaller than Audio interface Drum is smaller than Eye pin Pill smaller than Oyster?
Answer(s):
No
Question:
Lego is larger than Leaf
Thermostat is larger than Stage Thermostat is smaller than Trowel Ribbon is smaller than Saxophone Sculpture is larger than Saxophone
Cane is smaller than Coffee mug Playbill is smaller than Rake
Rake is smaller than Ribbon Leaf is larger than Lamp
Banjo is larger than Bandage Paperweight is smaller than Playbill
Gemstones is larger than Coffee mug
Stage larger than Sinker
Banjo is smaller than Cane
Bandage larger than Accordion Trowel is smaller than Water bottle Gemstones is smaller than Lamp Sculpture is smaller than Sinker Paperweight is larger than Lego Is Ribbon smaller than Paperweight?
24

---

Published as a conference paper at ICLR 2025
Answer(s):
No
Question:
Trowel is smaller than Wire cutters Birthday cake is smaller than CD
Mushroom larger than Measuring cup CD is smaller than Cap
Thesaurus is larger than Soda can
Measuring cup larger than Lego
Leaf is larger than Guitar
Mushroom smaller than Power strip Extension cord is larger than Envelope Wire cutters is smaller than Wood Conditioner is smaller than Cup Extension cord is smaller than Guitar Power strip is smaller than Soda can Envelope is larger than Cup
Trowel is larger than Tripod Ashtray is smaller than Birthday cake Leaf is smaller than Lego Tripod is larger than Thesaurus Conditioner larger than Cap Is Trowel larger than Birthday cake? Answer(s):
Yes
D.1.2 COMPARISON PEOPLE
Question:
Silver is younger than Trace Madyson is older than Lorelai
Evertt is younger than Hilma
Rush is older than Petra
Fulah is younger than Fulalie Petra is older than Orlena
Lorelai is older than Leta
Trace is younger than Vernetta
Cooper is older than Chrissie Evertt is older than Eulalie Nigel is younger than Orlena Ceil is younger than Chrissie Madyson is younger than Nigel Hilma is younger than Khalid
Ceil is older than Betty
Silver is older than Rush
Khalid is younger than Leta Allison is younger than Betty Cooper is younger than Eulah Is Nigel younger than Lorelai?
Answer(s):
No
Question:
Julius is older than Jalissa
Shemar is older than Raquel
Shemar is younger than Treyvon
Lyn is younger than Mae
Natasha is older than Mae
25

---

Published as a conference paper at ICLR 2025
Danyel is younger than Demario
Lonzo younger than Lori
Lori is younger than Lyn Jalissa older than Garey Cherrie is older than Carmen Kaleigh is younger than Lonzo Estefani is older than Demario Raquel is older than Phoenix
Cherrie is younger than Danyel Carmen is older than Ayana Treyvon is younger than Vickey Estefani is younger than Garey Natasha is younger than Phoenix Kaleigh is older than Julius
Question: Is Lyn younger than Kaleigh?
Answer(s):
No
Question:
Myah is younger than Ozzie
Antonetta is younger than Anya
Hettie is older than Elmore
Anya is younger than Arizona Marely is older than Marcela
Elmore is older than Donn
Devonta is older than Darryl Hettie is younger than Iver Cedrick is older than Case Ozzie is younger than Thomas Blake is younger than Briana younger than Darryl Iver is younger than Marcela Case is older than Briana Myah is older than Mikeal
Cedrick y
Abraham is younger than Antonetta Devonta is younger than Donn
Mikeal is older than Marely
Blake is older than Arizona Is Myah older than Antonetta?
Answer(s):
Yes
D.2 SOCIAL NETWORKS
Question:
Arjun is always honest with Marcelo, when it's hard.
When Arjun needs to talk, Vida is the first one they call. Casandra is always impressed by Lorena's 's knowledge and intelligence.
Vida is always willing to listen to Julianne's problems.
Roselyn is like family to Ava.
Exie is always willing to listen to Casandra's 's problems.
Lukas and Marcelo's families often have dinner together. Ava is always impressed by Casandra's 's creativity and artistic talents.
Lorena and Vida enjoy discussing books and movies they've ve both seen. Any two friends are able pass along message, which allows messages move from one friend to another. Thus, messages can be passed between two people through friends they have in common. If Vida wants to get message to Marcelo quickly possible, who should Vida give it to? Answer(s):
Arjun
26

---

Published as a conference paper at ICLR 2025
Question:
Austin knows how to make Gunnar laugh, even on a bad day. Barton and Marlana share inside jokes that only they understand. Daisey always respects Creola's opinions, even when they disagree.
Branden is always honest with Tierr,, even when it's hard. Tierra can always count on Paola for a shoulder to cry on. Marlana is always the first person Creola calls with good news. Marvin and Gunnar have inside jokes that no one else understands.
Branden knows all of Barton's 's favorite snacks and surprises them with them sometimes.
Marvin knows how to tease Barton without hurting their feelings.
Any two friends are able to pass along message, which allows messages move from one friend to another. Thus, messages can be passed between two people through friends they have in common. Daisey wants to get a message to Tierra as quickly as possible, who should Daisey give it to? Answer(s):
Creola
Question:
Daijah is always there to listen when Elmore needs to vent about work.
Elmore is always willing listen Charlee's problems.
You can often find Anwar and Finn laughing and chatting away.
When Walker needs to talk, Alvin is the first one they call.
Charlee knows how to calm Anwar down when they 'r stressed or anxious.
Minerva always knows how cheer Elmore up.
Cathi and Anwar have mutual respect for each other's 's personal space and boundaries. Walker and Charlee enjoy trying new hobbies and activities together. Toma and Cathi have deep and meaningful connection.
Any two friends are able to pass along message, which allows messages to move from one friend to another. Thus, messages can be passed between two people through friends they have in common. Cathi wants to get a message to Minerva as quickly as possible, who should Cathi give to? Answer(s): Anwar
D.3
JSON FAMILIES
D.3.1 JSON FAMILIES FAMILY SIZE
Question:
"544 Pine St, Palo Alto, CA 95841", "Members": [ "Name": "Ava". "Age": 13, "Hobbies": [ "cycling", "music", "reading" 1 "Name": "Grace", "Age": 71, "Hobbies": ("running") ]. "Name": "Bob", "Age": 91, "Hobbies": I "writing", "painting" "gardening", "running", "knitting". "cooking" ], "Name": "Diego", "Age": 93, "Hobbies": [ "music", "running", "dancing"'
"Family Name": "Wilson". "Address":
"Family Name": "Wilson", "Address": "695 Divisadero St, Daly City, CA 70635", "Members": "Name": "Frank", "Age": 45, "Hobbies": [ "knitting", "cooking". "cycling" "Name": "Diego", "Age": 67, "Hobbies": "reading". "gardening", "music", "writing", "dancing", "cooking", "traveling") ]. "Name": "Bob", "Age": 89, "Hobbies": "music", "knitting", "gardening", "painting", "writing", "dancing"], 1. "Name": "Liam", "Age": 96, "Hobbies": "knnttinng,, "painting"
"Family Name": "Rodriguez", "Address": "48 Lombard St, San Mateo, CA 35388", "Members": "Name": "Jack" "Age": 91, "Hobbies": "music"" "gardening" J, "Name": "Alice", "Age": 41, "Hobbies": ("gardening", "writing", "running", "cycling", "reading", "dancing". "music". "traveling". "painting"] 1. "Name": "Frank", "Age": 100. "Hobbies": [ ""writing", "gardening". "music"
"Family Name": "Brown". "Address": "959 Market St, San Jose, CA 10946", "Members") [ "Name": "Bob", "Age": 36, "Hobbies": ("running". "music", "painting", "reading", "knitting", ; "writing" 1, "Name": "Bob", "Age": 10. "Hobbies": [ "painting", "traveling". "dancing") "Name": "Kai", "Age": 88, "Hobbies": ["cooking", "writing" ]] "Family Name": "Wilson", "Address": "326 Lombard St, Daly City, CA 99979", "Members": [ "Name": "Muhammad", "Age": 63. "Hobbies": "ccycling", "knitting". "writing") J. "Name": "Emily", "Age": 36. "Hobbies": ("writing", "traveling", "knitting", "reading". "rumning"] How many members are the Brown family living on 959 Market St, San Jose, CA 10946? Answer single number.
27

---

Published as a conference paper at ICLR 2025
Answer(s):
3
D.3.2 JSON FAMILIES FAMILY MEMBER HOBBY
Question:
"Family Name": "Wilson". "Address": "544 Pine St, Palo Alto, CA 95841", "Members": "Name": "Ava". "Age": 13, "Hobbies": I "cycling". "music", "reading" J, "Name": "Grace", "Age":
71. "Hobbies"" "running"] ]. "Name": "Bob". "Age": 91. "Hobbies": "writing". "painting". "gardening". "running". "knitting", "cooking" ]. "Name": "Diego", "Age": 93, "Hobbies": [ "music", "running", "dancing"
"Family Name": "Wilson", " "Address": "695 Divisadero St, Daly City, CA 70635", "Members": "Name": "Frank", "Age": 45, "Hobbies": [ "knittinng, "cooking". "cycling" J. "Name": "Diego", "Age": 67, "Hobbies": [ "reading". "gardening", "music", "writing", "dancing", "cooking". "traveling"] ]. "Name": "Bob", "Age": 89, "Hobbies"; "music", "knitting", "gardening". "painting", "writing". "dancing" J. "Name": "Liam", "Age"" 96, "Hobbies": [ "knitting". "painting" ] "Family Name": "Rodriguez" "Address"; ; "48 Lombard St, San Mateo, CA 35388", "Members": [ "Name": "Jack", "Age": 91. "Hobbies": [ [mmuscc,, "gardening" ] "Name": "Alice", "Age": 41, "Hobbies": ("gardening", "writing", "running". "cycling". "reading". "dancing", "music", "traveling", "painting"), "Name": "Frank", "Frank"."Age" "Age": 100, "Hobbies": [ "writing". "gardening", "music"
"Family Name": "Brown", "Address": "959 Market St, San Jose, CA 10946", "Members": [ "Name": "Bob", "Age": 36, "Hobbies"": '"running", "music", "painting". "reading". "knitting". "writing" 1. "Name": "Bob", "Age": 10, "Hobbies": [ "painting", "traveling". ''dncing"' 1. "Name": "Kai", "Age": 88, "Hobbies": '"cooking" "writing"
1] "Family Name": "Wilson", "Address": "326 Lombard St, Daly City, CA 99979", "Members": [ "Name": "Muhammad", "Age": 63, "Hobbies": [ "cyclling, "knitting", "writing") 1. "Name": "Emily", "Age": 36, "Hobbies": ("writing", "traveling". "knitting", "reading", "running" Is writing a hobby of Jack from the Rodriguez family living on 48 Lombard St, San Mateo, CA 35388? Answer with Yes or No. Answers: Answer(s): No
D.3.3 JSON FAMILIES FAMILY SIZE COMPARISON
Question:
"Family Name": "Wilson", "Address": "544 Pine St, Palo Alto, CA 95841", "Members": [ "Name": "Ava". "Age": 13, "Hobbies": "cycling", "music", "reading" J. "Name": "Grace", "Age": 71, "..obbies"" ("running") ]. "Name": "Bob", "Age": 91, "Hobbies": [ "writing". "painting". "gardening", "running". "knitting", "cookingg" 1. "Name": "Diego", "Age": 93, "Hobbies": [ "music", "running". "dancing"
"Family Name": "Wilson", "Address": "695 Divisadero St, Daly City, CA 70635", "Members": "Name": "Frank", "Age": 45, "Hobbies": [ ("nittingg, "cooking", "cycling" ]. "Name": "Diego", "Age": 67, "Hobbies": (rradiing. "gardening", "music", "writing", "dancing". "cooking", "traveling") ]. "Name": "Bob", "Age": 89, "Hobbies": [ "music", "knitting" "gardening", "painting", "writing", "dancing" J. "Name": "Liam", "Age": 96. "Hobbies": "knitting". "painting" ] "Family Name": "Rodriguez", "Address": "48 Lombard St, San Mateo, CA 35388", "Members": [ "Name": "Jack", "Age": 91, "Hobbies": "mmuic"" "gardening" J. "Name": "Alice", "Age": 41, "Hobbies": ""gardening", "writing", "running", "cycling". "reading", "dancing", "music", "traveling", "painting" J. "Name": "Frank" "Age": 100.
"Hobbies": [ "writing", "gardening", "music" ] "Family Name": "Brown", "Address": "959 Market St. San Jose, CA 10946", "Members": [ "Name": "Bob", "Age": 36, "Hobbies": l"running". "music", "painting", "reading". "knitting", "writing"" " 1. "Name": "Bob", "Age": 10. "Hobbies": "painting", "traveling". ''daccnng') 1. "Name": "Kai". "Age": 88, "Hobbies": "cooking". "writing" ] "Family Name": "Wilson", "Address": "326 Lombard St, Daly City, CA 99979", "Members": [ "Name": "Muhammad". "Age": 63, "Hobbies": "cyccling", "knitting". "writing" "Name": "Emily", "Age": 36, "Hobbies": ["writing", "traveling". "knitting", "reading", "running" Which family is larger, the Wilson family living on 326 Lombard St, Daly City, CA 99979 or the Brown family living 959 Market St, San Jose, CA 10946? Answer with the family name of the larger family. Answer(s):
Brown
28

---

Published as a conference paper at ICLR 2025
D.3.4 JSON FAMILIES FAMILY MEMBER AGE COMPARISON
Question:
"Family Name": "Wilson". "Address": "544 Pine St, Palo Alto, CA 95841", "Members": "Name":
"Ava". "Age": 13, "Hobbies": [ "cycling", "music", "reading" 1, "Name": "Grace", "Age":
71, "..obiess" 'runnning") J. "Name": "Bob", "Age": 91, "Hobbies": [ "writing"" "painting", "gardening". "running". "knitting", "cooking" ]. "Name": "Diego", "Age": 93, "Hobbies": [ "music". "running". "dancing"
"Family Name": "Wilson". "Address": "695 Divisadero St, Daly City, CA 70635", "Members": "Name": "Frank", "Age": 45, "Hobbies": [ ["nittting" "cooking". "cycling" 1. "Name": "Diego", "Age": 67, "Hobbies": [ "reading". "gardening", "music", "writing", "dancing", "cooking", "traveling") 1. "Name": "Bob", "Age": 89, "Hobbies": "music", "knitting", "gardening". "painting", "writing", "dancing" I. "Name": "Liam", "AAe"" 96, "Hobbies": "kkitttnng.. "painting") ] "Family Name": "Rodriguez", "Address": "48 Lombard St, San Mateo, CA 35388", "Members": I "Name": "Jack", "Age": 91, "Hobbies": [ ("music", "gardening" ]. "Name": "Alice", "Age": 41, "Hobbies": [ "ggardening". "writing", "running". "cycling", "reading". "dancing". "music", "traveling" "painting"] 1. "Name": "Frank", "Age": 100. "Hobbies": [ "writing". "gardening", "music" ] "Family Name": "Brown", "Address": '959 Market St, San Jose, CA 10946", "Members": [ "Name": "Bob", "Age": 36, "Hobbies": ""unning". "music", "painting". "reading". "knitting". "writing") J. "Name" "Bob", "Age": 10. "Hobbies": [ "painting". "traveling", "dancing"]] J. "Name": "Kai", "Age": 88, "Hobbies": "ccooking" "writing" ]] "Family Name": "Wilson", "Address": "326 Lombard St, Daly City, CA 99979". "Members": [ "Name": "Muhammad". "Age": 63, "Hobbies": [ "eycling"" "knitting", "writing" 1, "Name": "Emily". "Age": 36, "Hobbies": ("writing", "traveling", "knitting", "reading". "running" Who is older: Muhammad from the Wilson family living on 326 Lombard St, Daly City, CA 99979 or Jack from the Rodriguez family living on 48 Lombard St, San Mateo, CA 35388? If both are the same age, answer with the name that comes first alphabetically. Answer with the name.
Answer(s):
Jack
D.3.5 JSON FAMILIES- FAMILY MEMBER HOBBY COMPARISON
Question:
"Family Name": "Wilson". "Address": "544 Pine St, Palo Alto, CA 95841", "Members": "Name" "Ava", "Age": 13, "Hobbies": [ "cyclling. "music", "reading" J. "Name": "Grace", "Age":
71, "Hobbies": (runnnin""] ]. "Name": "Bob", "Age": 91, "Hobbies"; "writing", "painting", "gardening". "running", "knitting", "'cooking" ]. "Name": "Diego", "Age": 93, "Hobbies": [ "music", "running". "dancing"
"Family Name": "Wilson", "Address": "695 Divisadero St, Daly City, CA 70635", "Members": "Name": "Frank", "Age": 45, "Hobbies": [ '"knitting", "cooking", "cycling" J. "Name": "Diego", "Age": 67, "Hobbies": I "reading". "gardening", "music", "writing", "dancing", "cooking", "traveling"] ]. "Name": "Bob", "Age": 89, "Hobbies": [ "music", "knitting", "gardening", "painting", "writing", "dancing" ]. "Name": "Liam", "Age" 96, "Hobbies": [ "knitting"" "painting" ] "Family Name": "Rodriguez", "Address": ; 748 Lombard St, San Mateo, CA 35388", "Members": [ "Name": "Jack", "Age": 91, "Hobbies": [mmusic", "gardening" }. "Name": "Alice", "Age": 41, "Hobbies": ("gardening". "writing", "nunning". "cycling". "reading". "dancing", "music", "traveling", "painting"), J. "Name": "Frank", "Age": 100, "Hobbies": [ "writing",, "gardening", "music" ] "Family Name": "Brown", "Address": "959 Market St, San Jose, CA 10946", "Members": [ "Name": "Bob", "Age": 36, "Hobbies": ""running". "music". "painting". "reading". "knitting", "writing") 1. "Name": "Bob", "Age": 10. "Hobbies": [ " "painting", "traveling". "dancing") J. "Name": "Kai", "Age": 88, "Hobbies": ''cooking". "writing"
1 "Family Name": "Wilson", "Address"" "326 Lombard St, Daly City, CA 99979", "Members": [ "Name": "Muhammad". "Age": 63, "Hobbies": [ "cycllngg, "knitting", "writing" 1. "Name": "Emily", "Age": 36, "Hobbies": [ ("writing", "traveling", "knitting". "reading", "running" What hobbies do Muhammad from the Wilson family living on 326 Lombard St, Daly City, CA 99979 and Jack from the Rodriguez family living on 48 Lombard St, San Mateo, CA 35388 share? List the hobbies in alphabetical order, separated by commas, or answer N/A if they share no hobbies. Answer(s):
N/A
29

---

Published as a conference paper at ICLR 2025
D.4 SYLLOGISMS
Question:
All accounts actions
No actions are actors
Which of the following is true?
All accounts are actors
No accounts are actors
Some accounts are actors
Some accounts are not actors
Answer(s):
No accounts are actors
Question:
No accounts are actions
All actors are accounts
Which of the following is true?
All actors actions
No actors are actions
Some actors are actions
Some actors are not actions
Answer(s): No actors are actions
Question:
Some accounts are actions
No actors are accounts
Which of the following is true?
All actions are actors
No actions are actors
Some actions actors
Answer(s):
Some actions actors
30


Reviews:
[{"review_id": "tIyRUELiwl", "reviewer": "Reviewer_P5ze", "summary": "This article introduces ReCogLab, a flexibly generated dataset for quantitatively measuring relational memory and reasoning performance in LLMs, and conducts several experiments using the framework. ReCogLab offers a fully customizable and automatically generated dataset, enabling researchers to delve into the nuances of cognitive processes.\n\nThe paper begins by introducing ReCogLab. Examples generated by ReCogLab come from one of four tasks: Comparison, Social Networks, Syllogisms and JSON Families. Each example includes context C, question Q and answer(answers) A. Every aspect of the data generated can be controlled via a configuration file specifying parameters, which allows for the creation of datasets with varying length and complexity. And for each specific cognitive probe, ReCogLab generates 50 validation examples to select the best prompt and 1000 test examples for comprehensive evalution. Notably, the authors decoupled the prompt and parsing hyperparameters from the LM to minimize the impact of prompting on LMs’ performance.\n\nThen, the authors conduct several experiments to benchmark relational reasoning performance across different models and problem complexities and explore how reasoning performance depends on certain features. The article looks at transitive inference, congruency, the symbolic distance effect, identifying logical inconsistencies and indeterminate reasoning. These experiments were all made with little additional effort using the ReCogLab framework by altering the configuration files. The findings reveal that many human cognitive phenomena are mirrored in LLMs.\n\n**Experiment 1: Transitive Inference**\n\nPurpose:\n\nTo evaluate models’ ability to reason accurately about associations presented in context and measure whether there was a dependence on presentation order, and further measure how the ability depends on complexity measures of the reasoning problem. \n\nResults:\n\n1.(Comparison and Syllogisms) Randomizing the order causes noticeable degradation while reversing the presentation order affects performance in many but not all cases.\n\n2.(Syllogisms)The authors speculate that consistent presentation order may bias the model toward approximate solutions.\n\n3.Performance generally degrades as the complexity of the problem increases across all models.\n\n**Experiment 2: Congruent and Incongruent**\n\nBackground:\n\nLogical reasoning is more accurate when logical premises are congruent with real-world facts for LLMs(as it is for humans).\n\nPurpose:\n\nTo investigate the impact of congruence on language model performance.\n\nResults:\n\nUsing congruent statements outperforms similar comparison problems constructed with incongruent statements.\n\n**Experiment 3: Symbolic Distance Effect**\n\nPurpose:\n\nTo validate that LLMs will also show a symbolic distance effect.\n\nResults:\n\nFor models that perform above chance, there is a clear positive symbolic distance effect that starts when the symbolic distance is greater than 15.\n\n**Experiment 4: Identifying Inconsistent Premises**\n\nPurpose:\n\nTo explore whether LLMs have the ability of identifying when a premises is logically inconsistent.\n\nResults:\n\nLarger LMs perform better on detecting inconsistent statements.\n\n**Experiment 5: Indeterminate Reasoning**\n\nPurpose:\n\nTo evaluate models’ ability to understand when there’s insuffient information to draw a conclusion.\n\nProcedure:\n\nThe authors start with comparison problems which contain a fixed label set of Yes or No. Then they modify the comparison problems from a linear chain to a random tree generation while still asking questions about two random nodes, which provides insufficient context.\n\nResults:\n\nThere exists a bias toward incorrectly answering \\textquotedblleft yes\" on logical prompts when it is uncertain and reporting uncertainty when in fact it is unknown.", "questions": "As seen above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "**Flexibility and universality**\n\nReCogLab has significant advantages over the previous dataset in terms of flexibility and universality. Such a dataset would bring many conveniences to research in cognitive sicence and holds significant practical value.\n\n**Good breadth**\n\nThis article demonstrates a good breadth, reprising experiments from multiple aspects of cognitive science, and measuring the ability of multiple LLMs across various dimensions.", "weaknesses": "**Lack of evaluation of dataset quality**\n\nWhen evaluating the quality of a dataset, people commonly employ methods such as statistical analysis, visualization, cross-validation, and expert review. However, in this article, we only see qualitative expressions about the performance of ReCogLab, such as its comprehensiveness and flexibility. And I’m wondering would it be possible to add some quantitative indicators to more strongly demonstrate the quality of ReCogLab or compare ReCogLab with those originally used in the same or similar experiments to illustrate its advantages?\n\n**Lack of detailed experimental procedures**\n\nThe description of the experimental procedures could be more detailed. It did not provide a detailed explanation of how ReCogLab was used in these experiments and how exactly the parameters were set.\n\n**Unclear motivation**\n\nThe authors haven’t provided evidence for their classification (comparison, social networks, syllogism, json families). What’s more, the decision to test all tasks as a whole is questionable, especially considering that previous research has addressed these aspects individually. The necessity of compiling these tasks for collective testing is not immediately clear, and this approach may need further justification.\n\n**Lack of insights**\nThe authors simply test LLM without delving into a more detailed analysis of each aspect, which results in superficial analyses. The authors should perform deeper analysis, such as for the transitive inference, the authors should offer an explanation of why the change of order matters for LLM, for congruent why is it important (whether this phenomenon comes from data or from training algorithm or transformers architecture), ...\n\n**Lack of novelty**\n1. In Experiment 1, the authors' assertion that \"premise order matters\" was previously identified in the article \"Premise Order Matters in Reasoning with Large Language Models\" published on May 28th. In Experiment 2, the earlier paper \"Language Models, like humans, show content effects on reasoning tasks\" demonstrated the importance of congruence in reasoning. Furthermore, there appears to be no significant distinction between relational reasoning and general reasoning. Could the authors compare with those above-mentioned papers to address the novelty of this paper? Furthermore, could the authors talks about the difference between relational reasoning and general reasoning?\n\n 2. Additionally, in Experiment 4, the detection of inconsistency is an old topic, and it comes as no surprise that Large Language Models (LLMs) are capable of performing this task, could the authors demonstrate some new discoveries in their experiments? Moreover, this topic bears a striking resemblance to Knowledge Base Question Answering (KBQA), can the authors provide the difference between the topic of relational reasoning and KBQA? \n\n**Some editorial errors and missing citations**\n\nIntroduction paragraph 1: While recent work...\n\nCitations are missing.\n\n\n\nIntroduction paragraph 2: From this literature...\n\nThe authors didn’t point out what “this literature” refers to.\n\nIntroduction paragraph 3: In this work, we aim to provide an evaluation framework that allows for the systematic evaluation or LLMs on relational memory and investigation of different possible effects(many inspired from the cognitive science literature).\n\nMaybe it means “of”?\n\nBackground paragraph 1: $A > B$, $B > C$, $B > C$, $C > D$, $D > E$\n\nPerhaps “$B > C$” is repeated?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NVmlwqmjWs", "reviewer": "Reviewer_UYz6", "summary": "This paper does several things: \n\n* It introduces ReCogLab, a way of generating data-sets for testing relational understanding in LLMs\n* It evaluates several LLMs on several relational understanding tasks, noting successes and limitations, and speculating on the connection to human reasoning about relations.", "questions": "* It would be good if the authors could provide more details on their framework and how it works. \n\n* It would be good if the authors tested their frameworks on open models in a way that allowed them to draw scientific conclusions about why some models fail and some don't or show the effects they do. \n\n* It would be good if the authors justified their tasks (I'm not saying they aren't good tasks, I'm saying the authors should motivate them further). The introduction and background could use major tightening and also moving up the fact that you tested indeterminate reasoning and inconsistent premises, and also to explain more how this differs from other major works recently looking at relationship reasoning. \n\n* It would be good if the authors connected this better to the specific psychological literature\n\n* It would be good if the authors provided statistical tests to back up their claims.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper has several welcome aspects: \n\n- Generally speaking, relational understanding is a basic part of cognition, so the topic itself is important. \n- Testing LLMs on their basic understanding of relations is useful for our understanding both of these models and of human cognition.\n- The paper includes more than a single example or a single model, allowing us to (in principle) draw more general conclusions\n- The 'stress testing' done by increasing the number of variables/nodes/edges is useful for assessment.", "weaknesses": "While there are many positive aspects to the work, and I think it is generally in the right direction, there are some specifics to the paper that make it difficult to recommend acceptance:\n\nSome general comments: \n\n* While I understand the authors' decision not to provide full details of their framework, the fact that they present it as a major contribution of the paper while at the same time not providing full details of how the framework works and promising to do so only upon publication makes it difficult to assess it. \n\n* While I appreciate the use of several tasks and models, it isn't clear or justified why *these* tasks are specific to relational understanding (and not others), and the use of closed models makes it difficult to assess what is actually going on beyond saying \"these models do it well, those models don't\". I get that many of us are currently assessing models in this way and I do think it is useful to assess the state of the art, but at the moment the assessment of closed-models by mega-corporations falls more into doing QA for them than being able to say something scientific about why and how LLMs fail or succeed on these tasks. \n\n* The connections to human reasoning and psychology are welcome and overall ok, but there are many places where they could be improved (see more details below). Also, I found several of the cases of arguing along the lines of \"the models are doing poorly and people also do poorly so maybe they're the same\" to be speculative at best and not digging into the phenomena, again see more details below. \n\n* There are many cases where statistical claims are made without statistical tests to back them up. While I agree the tests would likely back up the claims I don't think severe of the claims are resting on solid ground at the moment (again, see more details below). \n\nComments in more detail (these are not in order of importance but more in order of reading the paper, so some of these are truly minor and had no real affect on the final assessment, they are provided in case they help the authors tighten the work):\n\n-- Not terribly important but the first paragraph is making a bunch of claims that would ideally be backed up by references. For example, \"While recent work on memory in large language models (LLMs) assumes that memory refers to the recall of a specific piece of information from the past\" -- which recent work?, or, when you say \"When humans remember events, facts, places, etc., they don’t just recall disconnected pieces, they recall the associations\" -- what is backing this? Similar comments apply to much of the second paragraph\n\n-- Given the importance of relational reasoning that the authors emphasize and I agree with, I found it a bit odd that when we do get to references to back it up in humans it is to a 1976 paper on a specific phenomena of symbolic distancing, a 2010 general textbook, and a 2018 neuroscience paper on episodic memory in the hippocampus. This feels like a \"grab bag\" of disconnected things.\n\n---- Side note, related to the above: With all respect to Domjan (2010) I found the frequent reliance on an undergrad psych text-book throughout the paper odd. It would be good to back up your claims with regards to more specific papers in the literature. \n\n-- As mentioned in the general comments: I'm not knocking the specific things you decided to test too hard (syllogisms, comparisons, etc) but it would be good to provide some stronger justification for why *these* specific things and not others. Do they specifically cover a majority of relational reasoning? Are they uniquely interesting in some way? I'm not saying they don't or aren't, but the reader shouldn't have to do that work. Also, while we're here: Given the weight and importance you attach to identifying inconsistent premises later in the paper, they seem to come out of nowhere because they aren't really justified in the introduction and given weight or airtime in the figures and background and text that discusses the other cases you're looking at. \n\n-- I said I wouldn't knock the specific tests too hard, but I think the 'social networks' one deserves even more justification, given that it doesn't seem to actually be about 'relations' or 'social reasoning' but, as the authors themselves point out later, navigation of graphs independently of the 'relations' between people. It doesn't really seem to be about relational understanding in the way that, say, \"X is on Y\". The results would presumably replicate with just \"X is to the right of Y\" or various other room descriptions. So, why go through stuff like \"X is the child of Y, Z never shares T's secrets...\"; it all seems kind of unnecessary, and would probably not really work in humans. The interesting stuff about enemies and non-relations is mentioned by the authors but not explored further. \n\n-- super-duper-minor: For figure 1 it would be nice to have the colors match up with the text. For example, the setup for comparison is \"orange > bottle > tv > dog\", yet in the text dog is red and orange is blue, while in the graph the red node is 'bigger' than the blue node and is separated by only 1 node from it. \n\n-- I get the focus on LLMs and it is good to keep it, but it would be nice to at least acknowledge briefly in the background that other models have tried to model this stuff that don't rely on LLMs. The entire field of Inductive Logic Programming seems relevant, for example. \n\n-- The first paragraph in the background is very disjointed, it makes 3 separate points that don't connect.\n\n-- \"Substantial work has compared reasoning in LLMs to humans\" -- it is odd to follow this sentence up with a reference to Stupple & Ball (2008) or Evans et al. 1983, which took place before 'LLMs' were a term. I'm not sure if the authors were tripped up by the use in the paper of \"parallel processing\" (which doesn't mean what it means in PDP connectionism).\n\n-- I appreciate mentioning relevant LLM papers in the background section but it would be good if the authors further delineate what their work contributes beyond the already substantial contributions of those works, right now it isn't particularly clear what the contrast is. \n\n-- ״because friendship is a symmetric relationship, social networks are undirected graphs״ -- while it is true that many social networks are modeled to a first approximation using a simple undirected graph, I'm sure people in social psychology would balk at the statement that 'friendship is a symmetric relationship'. Also the consequent doesn't follow, social networks describe friendships as well as rivalries, hierarchies, family relations, and many other things besides not captured by a simple representation along these lines. As mentioned above, it is not clear to what degree this task is about 'relational understanding' more than it is about navigating an undirected graph, where the graph could describe spatial relations, concepts, or nonsense. \n\n-- it wasn't entirely clear to me how you check for grounding in a generalizable way. That is, I understand that in this case specifically, you used a different LLM to generate the sizes and weights of things, but if I wanted to ensure that my data generally matches reality for some arbitrary property, this seems like a currently unsolved problems in LLMs so we can't simply say 'oh, we'll use LLMs to make sure it is grounded'. \n\n-- I appreciate the decoupling of prompting, but what are the actual details of how this decoupling happened? Which prompts did you use and why? \n\n-- \"publicly available language models\" doesn't mean \"open\"\n\n-- There are several cases in the paper where the authors make claims that are meant to be backed up with a statistical test. \n\nFor example:\n\" In Fig. 2 we see this experiment for Comparison and Syllogisms. For Comparison, we can clearly see across nearly all models randomizing the order causes noticeable degradation (Gemma 2B and 9B perform near or below chance of 0.5, guessing or giving invalid answers). Reversing the presentation order also often negatively affects performance in many, but not all cases.\"\n\n'We can clearly see' is not a statistical test. Presumably you want to use an ANOVA here or one of a myriad of other ways of assessing this claim. Similarly 'negatively affects performance in many, but not all cases' is a non-exact, non-specific way of putting a result. \n\n* \"For most models we see a curve in which accuracy decreases with symbolic distance for very small symbolic distances, but then improves for symbolic distances >15.\" -- Again, this is highly inexact. What do you mean 'most models', what do you mean 'improves above 15'? What statistical test are you running to say these statements? Are you comparing a linear fit to a quadratic fit, for example? Are you simply fitting a linear line and looking at the rise? Are you doing a piece-wise and finding a transition point at 15, and what is associated value of the parameters for any statistical test you ran?\n\n* \"\"Gemini Pro actually levels out rather than decreasing and GPT-4o only decreasing slightly, suggesting that they are particularly good at needle-in-the haystack style questions\" -- here and elsewhere, while I appreciate the speculation, we see the problem with evaluating these closed models, we simply have no way of knowing what is going on. \n\n-- The authors should probably face head-on the fact that spatial distancing in these models isn't working like it does in people. In people it drops from a distance of 1 to 2, but then steadily rises. For these models it seems (to the degree that one can say anything) more like a u-shaped function. \n\n-- I didn't understand why you tested the effect of capacity in the ranges you did, sometimes it seems to go 0-100, sometimes 0-50, sometimes 1-20, sometimes 5-30. In cases where we see degradation it is reasonable to argue it exists, but in cases where we don't (JSON family, values 1-20) it isn't clear if this is a result of not going beyond the current range. \n\n-- Super minor hair splitting: 'nonsense relations' like 'Benny blexed Kelly' is not 'inconsistent' with the real world in the same way that 'iron is lighter than air'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cdjxmYjP4W", "reviewer": "Reviewer_G6qF", "summary": "ReCogLab provides a series of benchmark tasks for relational memory. Although the tasks themselves do not necessarily seem problematic, the coherence of this series of benchmarks was unclear. Why were these tasks used as opposed to others? The paper would be greatly improved explicitly stating the criteria for selecting these tasks and articulating how these tasks relate to different theoretically distinct aspects of relational reasoning. Building specific hypotheses with respect to different cognitive operations may allow the reader to better understand how to position or understand differences between tasks, especially with the potential for different results. As part of this. the authors could more explicitly articulate specific reasons LLMs (either in general, or specific LLMs) would perform on some tasks compared to others.\n\nIn the background, the authors note that some of these tasks have been used in isolation, but there was limited information about how LLMs did on those tasks, and other than having all together in one set, how these benchmarks were preferred to the ones in the literature. Please provide summaries of previous LLM performances on related tasks. What do the authors see concretely about using this set as opposed to the previous tasks?\n\nIf this is a benchmark, is it necessary to have human data? Do humans perform differently on these tasks, and might that be a reason that the different benchmarks are needed? The authors should consider whether human data would help understand LLM performances on these tasks, or whether human data would provide insights into how to understand differences between the tasks. What results would the authors expect?\n\nIt is unclear why this is relational memory as opposed to relational thinking. What aspects of this make the processes specifically memory related? Is memory necessary for solving these tasks in a way that is different than other LLM tasks (one could make the argument that all LLM tasks require some degree of memory).\n\nIn sum, the weaknesses of this paper come from inadequate framing and positioning of its importance in the larger literature. It is possible with more information that the strengths of the paper would be more clearly understood.", "questions": "1. Why are previous benchmarks inadequate?\n2. Why are these methods preferred?\n3. Why do LLMs succeed or fail on these different benchmarks?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "Examining the capacities of LLMs is important, and it is important to understand the more difficult aspects of potential cognition such as relational reasoning. Establishing benchmarks is an important part of building this research program.", "weaknesses": "Although benchmarks are important, it wasn't clear from the writing why these benchmarks are the ones to be used. A disucssion about previous benchmarks is needed with attention to their limitations. The authors main claim is that there isn't a set of all these benchmarks together, but one should simply stitch together other benchmarks as a set. A discussion of why this is not a good strategy is needed. If these are benchmarks, human data may be needed to understand what good performance is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYvcNHR9km", "reviewer": "Reviewer_P3cs", "summary": "This paper introduces ReCogLab, a sandbox evaluation paradigm for studying memory and reasoning in Large Language Models, drawing inspiration from the cognitive sciences. ReCogLab currently contains a toolkit for building memory and reasoning tasks based on four domains: property comparisons, social networks, hierarchical reasoning (which are called JSON families), and syllogistic reasoning. Each of these domains are formalised as tasks that probe graph-based reasoning. Problems are constructed as (directed) graphs, where each vertex is a person or an object and each edge is some directed relation between those objects. Graphs are built implicitly in ReCogLab and then described only in natural language vignettes. The LLM is then tasked with answering questions about the graphs, also in natural language. For instance, in the Social Networks domain, a graph is implicitly constructed that relates certain members of a social group. The graph is described with sentences such as 'X is friends with Y' and 'Y is friends with Z'. The LLM is then tasked with, for example, describing how a message would be most quickly communicated between two members of the social network, essentially by traversing the edges of the graph. The authors create a testbed of tasks from their four domains drawing inspiration from real experiments done with humans in the cognitive sciences, and study the reasoning and memory capabilities of six large language models. This enables them to study, in LLMs, a number of psychological tendencies that have been observed in humans. Ultimately, they observe many human-like effects emerging in LLMs.", "questions": "1. What is the goal of studying this type of reasoning in LLMs? \n2. How might we use the outputs of ReCogLab to better direct LLM development and deployment?\n3. Did you run any experiments with Chain-of-Thought prompting? It would be interesting to determine whether this had any impact on performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper has several strengths. First, the production of a sandbox testbed, rather than a static, fixed benchmark, is exactly where evaluation should be going in AI and machine learning. ReCogLab is a toolbox for creating nuanced cognitive experiments, rather than a dataset that someone can take off the shelf and apply without much thought. Moreover, the toolbox significantly reduces the likelihood of future LLMs being trained directly on testing data - contamination is something that has significantly hindered the validity of many new and old benchmarks in the literature. Second, the authors identify a series of psychological effects that few have studied directly in LLMs, but which are key markers of behavioural similarity between us and them, allowing us to diagnose the human-likeness of these systems. Finally, the analyses are comprehensive and balanced. The authors are cautious not to prematurely ascribe human like cognitive capabilities to these systems, noting that there is more work to be done to show this. This is a refreshing contrast to much of the overzealous AI hype from other (non-cognitive-science) researchers in the field.", "weaknesses": "There are a number of weaknesses in the paper. First, the authors only evaluate 6 LLMs. The addition of at least some members of the Llama series would lead to the results being more comprehensive of the state-of-the-art. Second, the authors could better review other calls to action for doing cognitive science experiments on artificial intelligence/large language models, as well as other frameworks that offer sandbox evaluation toolkits. I have included a number of relevant citations below. Third, the authors could have played more with chain-of-thought prompting to determine whether this leads to better performance on their tests. Finally, there are some minor linguistic disfluencies. The title is difficult to parse. \"A framework testing relational reasoning, cognitive hypotheses on LLMs\" contains two disjoint clauses that aren't related. This is off-putting to the reader. Furthermore, the abstract needs to be checked for sense. 'Phenomenon' should be plural 'phenomena' and the sentence started 'while some of these have been studied individually...' needs to be reworded as it doesn't make grammatical sense. Lastly, the term 'indeterminancy' should be replaced with 'indeterminacy' in the latter part of the paper.\n\nReferences\n\nHernández-Orallo, J. (2017). The measure of all minds: evaluating natural and artificial intelligence. Cambridge University Press.\nKiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., ... & Williams, A. (2021). Dynabench: Rethinking benchmarking in NLP. arXiv preprint arXiv:2104.14337.\nThrush, T., Tirumala, K., Gupta, A., Bartolo, M., Rodriguez, P., Kane, T., ... & Kiela, D. (2022). Dynatask: A framework for creating dynamic AI benchmark tasks. arXiv preprint arXiv:2204.01906.\nVoudouris, K., ..., Cheke, L. G. (2024) The Animal-AI Environment: A Virtual Laboratory For Comparative Cognition and Artificial Intelligence Research. arXiv preprint arXiv:2312.11414.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)