ReCogLab: a framework testing relational reasoning & cognitive hypotheses on LLMs

Authors: Andrew Liu, Henry Prior, Gargi Balasubramaniam, Rivka Moroshko, Amir Zait, Ilia Labzovsky, Danny Karmon, Ishita Dasgupta, Kim Stachenfeld, Kenneth Marino
Primary area: datasets and benchmarks
Keywords: Congitive Science, Large Language Models, Datasets, Evaluation, Relational Reasoning

Abstract:
A fundamental part of human cognition is the ability to not only recall previous memories, but also reason across them to draw conclusions. In cognitive science and psychology, this is termed relational reasoning and a number of effects and biases have been observed in human cognition. Designing experiments to measure these reasoning effects is effortful and does not transfer easily to analyzing language model reasoning patterns. To make exploring language models on relational reasoning easier, we introduce ReCogLab – a generative framework for constructing reasoning examples. Unlike static datasets, our framework has a number of benefits that help us in our goal of flexible evaluation of LLMs. First, our framework allows us to control the difficulty and context-length of the problem, allowing us to scale with model capability and evaluate LLMs at a variety of scales. Second, the ability to change the configuration of a dataset dynamically allows us to probe models on different aspects and capabilities. Finally, the flexibility of our approach enables the recreation of classic cognitive science experiments and the systematic study of relational reasoning biases in language models. We demonstrate several such experiments and present our findings on a wide variety of open and closed-source language models. We release all data and code at https://github.com/google-deepmind/recoglab.

Decision: accept (poster)
Reviewer scores: 5, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1729512208301

thread_type:
official_review

<thread_object>
creator: reviewer_p3cs

<content>
summary:
This paper introduces ReCogLab, a sandbox evaluation paradigm for studying memory and reasoning in Large Language Models, drawing inspiration from the cognitive sciences. ReCogLab currently contains a toolkit for building memory and reasoning tasks based on four domains: property comparisons, social networks, hierarchical reasoning (which are called JSON families), and syllogistic reasoning. Each of these domains are formalised as tasks that probe graph-based reasoning. Problems are constructed as (directed) graphs, where each vertex is a person or an object and each edge is some directed relation between those objects. Graphs are built implicitly in ReCogLab and then described only in natural language vignettes. The LLM is then tasked with answering questions about the graphs, also in natural language. For instance, in the Social Networks domain, a graph is implicitly constructed that relates certain members of a social group. The graph is described with sentences such as 'X is friends with Y' and 'Y is friends with Z'. The LLM is then tasked with, for example, describing how a message would be most quickly communicated between two members of the social network, essentially by traversing the edges of the graph. The authors create a testbed of tasks from their four domains drawing inspiration from real experiments done with humans in the cognitive sciences, and study the reasoning and memory capabilities of six large language models. This enables them to study, in LLMs, a number of psychological tendencies that have been observed in humans. Ultimately, they observe many human-like effects emerging in LLMs.

strengths:
This paper has several strengths. First, the production of a sandbox testbed, rather than a static, fixed benchmark, is exactly where evaluation should be going in AI and machine learning. ReCogLab is a toolbox for creating nuanced cognitive experiments, rather than a dataset that someone can take off the shelf and apply without much thought. Moreover, the toolbox significantly reduces the likelihood of future LLMs being trained directly on testing data - contamination is something that has significantly hindered the validity of many new and old benchmarks in the literature. Second, the authors identify a series of psychological effects that few have studied directly in LLMs, but which are key markers of behavioural similarity between us and them, allowing us to diagnose the human-likeness of these systems. Finally, the analyses are comprehensive and balanced. The authors are cautious not to prematurely ascribe human like cognitive capabilities to these systems, noting that there is more work to be done to show this. This is a refreshing contrast to much of the overzealous AI hype from other (non-cognitive-science) researchers in the field.

weaknesses:
There are a number of weaknesses in the paper. First, the authors only evaluate 6 LLMs. The addition of at least some members of the Llama series would lead to the results being more comprehensive of the state-of-the-art. Second, the authors could better review other calls to action for doing cognitive science experiments on artificial intelligence/large language models, as well as other frameworks that offer sandbox evaluation toolkits. I have included a number of relevant citations below. Third, the authors could have played more with chain-of-thought prompting to determine whether this leads to better performance on their tests. Finally, there are some minor linguistic disfluencies. The title is difficult to parse. "A framework testing relational reasoning, cognitive hypotheses on LLMs" contains two disjoint clauses that aren't related. This is off-putting to the reader. Furthermore, the abstract needs to be checked for sense. 'Phenomenon' should be plural 'phenomena' and the sentence started 'while some of these have been studied individually...' needs to be reworded as it doesn't make grammatical sense. Lastly, the term 'indeterminancy' should be replaced with 'indeterminacy' in the latter part of the paper.

References

Hernández-Orallo, J. (2017). The measure of all minds: evaluating natural and artificial intelligence. Cambridge University Press.
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., ... & Williams, A. (2021). Dynabench: Rethinking benchmarking in NLP. arXiv preprint arXiv:2104.14337.
Thrush, T., Tirumala, K., Gupta, A., Bartolo, M., Rodriguez, P., Kane, T., ... & Kiela, D. (2022). Dynatask: A framework for creating dynamic AI benchmark tasks. arXiv preprint arXiv:2204.01906.
Voudouris, K., ..., Cheke, L. G. (2024) The Animal-AI Environment: A Virtual Laboratory For Comparative Cognition and Artificial Intelligence Research. arXiv preprint arXiv:2312.11414.

questions:
1. What is the goal of studying this type of reasoning in LLMs? 
2. How might we use the outputs of ReCogLab to better direct LLM development and deployment?
3. Did you run any experiments with Chain-of-Thought prompting? It would be interesting to determine whether this had any impact on performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729987516552

thread_type:
official_review

<thread_object>
creator: reviewer_g6qf

<content>
summary:
ReCogLab provides a series of benchmark tasks for relational memory. Although the tasks themselves do not necessarily seem problematic, the coherence of this series of benchmarks was unclear. Why were these tasks used as opposed to others? The paper would be greatly improved explicitly stating the criteria for selecting these tasks and articulating how these tasks relate to different theoretically distinct aspects of relational reasoning. Building specific hypotheses with respect to different cognitive operations may allow the reader to better understand how to position or understand differences between tasks, especially with the potential for different results. As part of this. the authors could more explicitly articulate specific reasons LLMs (either in general, or specific LLMs) would perform on some tasks compared to others.

In the background, the authors note that some of these tasks have been used in isolation, but there was limited information about how LLMs did on those tasks, and other than having all together in one set, how these benchmarks were preferred to the ones in the literature. Please provide summaries of previous LLM performances on related tasks. What do the authors see concretely about using this set as opposed to the previous tasks?

If this is a benchmark, is it necessary to have human data? Do humans perform differently on these tasks, and might that be a reason that the different benchmarks are needed? The authors should consider whether human data would help understand LLM performances on these tasks, or whether human data would provide insights into how to understand differences between the tasks. What results would the authors expect?

It is unclear why this is relational memory as opposed to relational thinking. What aspects of this make the processes specifically memory related? Is memory necessary for solving these tasks in a way that is different than other LLM tasks (one could make the argument that all LLM tasks require some degree of memory).

In sum, the weaknesses of this paper come from inadequate framing and positioning of its importance in the larger literature. It is possible with more information that the strengths of the paper would be more clearly understood.

strengths:
Examining the capacities of LLMs is important, and it is important to understand the more difficult aspects of potential cognition such as relational reasoning. Establishing benchmarks is an important part of building this research program.

weaknesses:
Although benchmarks are important, it wasn't clear from the writing why these benchmarks are the ones to be used. A disucssion about previous benchmarks is needed with attention to their limitations. The authors main claim is that there isn't a set of all these benchmarks together, but one should simply stitch together other benchmarks as a set. A discussion of why this is not a good strategy is needed. If these are benchmarks, human data may be needed to understand what good performance is.

questions:
1. Why are previous benchmarks inadequate?
2. Why are these methods preferred?
3. Why do LLMs succeed or fail on these different benchmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730215782656

thread_type:
official_review

<thread_object>
creator: reviewer_uyz6

<content>
summary:
This paper does several things: 

* It introduces ReCogLab, a way of generating data-sets for testing relational understanding in LLMs
* It evaluates several LLMs on several relational understanding tasks, noting successes and limitations, and speculating on the connection to human reasoning about relations.

strengths:
The paper has several welcome aspects: 

- Generally speaking, relational understanding is a basic part of cognition, so the topic itself is important. 
- Testing LLMs on their basic understanding of relations is useful for our understanding both of these models and of human cognition.
- The paper includes more than a single example or a single model, allowing us to (in principle) draw more general conclusions
- The 'stress testing' done by increasing the number of variables/nodes/edges is useful for assessment.

weaknesses:
While there are many positive aspects to the work, and I think it is generally in the right direction, there are some specifics to the paper that make it difficult to recommend acceptance:

Some general comments: 

* While I understand the authors' decision not to provide full details of their framework, the fact that they present it as a major contribution of the paper while at the same time not providing full details of how the framework works and promising to do so only upon publication makes it difficult to assess it. 

* While I appreciate the use of several tasks and models, it isn't clear or justified why *these* tasks are specific to relational understanding (and not others), and the use of closed models makes it difficult to assess what is actually going on beyond saying "these models do it well, those models don't". I get that many of us are currently assessing models in this way and I do think it is useful to assess the state of the art, but at the moment the assessment of closed-models by mega-corporations falls more into doing QA for them than being able to say something scientific about why and how LLMs fail or succeed on these tasks. 

* The connections to human reasoning and psychology are welcome and overall ok, but there are many places where they could be improved (see more details below). Also, I found several of the cases of arguing along the lines of "the models are doing poorly and people also do poorly so maybe they're the same" to be speculative at best and not digging into the phenomena, again see more details below. 

* There are many cases where statistical claims are made without statistical tests to back them up. While I agree the tests would likely back up the claims I don't think severe of the claims are resting on solid ground at the moment (again, see more details below). 

Comments in more detail (these are not in order of importance but more in order of reading the paper, so some of these are truly minor and had no real affect on the final assessment, they are provided in case they help the authors tighten the work):

-- Not terribly important but the first paragraph is making a bunch of claims that would ideally be backed up by references. For example, "While recent work on memory in large language models (LLMs) assumes that memory refers to the recall of a specific piece of information from the past" -- which recent work?, or, when you say "When humans remember events, facts, places, etc., they don’t just recall disconnected pieces, they recall the associations" -- what is backing this? Similar comments apply to much of the second paragraph

-- Given the importance of relational reasoning that the authors emphasize and I agree with, I found it a bit odd that when we do get to references to back it up in humans it is to a 1976 paper on a specific phenomena of symbolic distancing, a 2010 general textbook, and a 2018 neuroscience paper on episodic memory in the hippocampus. This feels like a "grab bag" of disconnected things.

---- Side note, related to the above: With all respect to Domjan (2010) I found the frequent reliance on an undergrad psych text-book throughout the paper odd. It would be good to back up your claims with regards to more specific papers in the literature. 

-- As mentioned in the general comments: I'm not knocking the specific things you decided to test too hard (syllogisms, comparisons, etc) but it would be good to provide some stronger justification for why *these* specific things and not others. Do they specifically cover a majority of relational reasoning? Are they uniquely interesting in some way? I'm not saying they don't or aren't, but the reader shouldn't have to do that work. Also, while we're here: Given the weight and importance you attach to identifying inconsistent premises later in the paper, they seem to come out of nowhere because they aren't really justified in the introduction and given weight or airtime in the figures and background and text that discusses the other cases you're looking at. 

-- I said I wouldn't knock the specific tests too hard, but I think the 'social networks' one deserves even more justification, given that it doesn't seem to actually be about 'relations' or 'social reasoning' but, as the authors themselves point out later, navigation of graphs independently of the 'relations' between people. It doesn't really seem to be about relational understanding in the way that, say, "X is on Y". The results would presumably replicate with just "X is to the right of Y" or various other room descriptions. So, why go through stuff like "X is the child of Y, Z never shares T's secrets..."; it all seems kind of unnecessary, and would probably not really work in humans. The interesting stuff about enemies and non-relations is mentioned by the authors but not explored further. 

-- super-duper-minor: For figure 1 it would be nice to have the colors match up with the text. For example, the setup for comparison is "orange > bottle > tv > dog", yet in the text dog is red and orange is blue, while in the graph the red node is 'bigger' than the blue node and is separated by only 1 node from it. 

-- I get the focus on LLMs and it is good to keep it, but it would be nice to at least acknowledge briefly in the background that other models have tried to model this stuff that don't rely on LLMs. The entire field of Inductive Logic Programming seems relevant, for example. 

-- The first paragraph in the background is very disjointed, it makes 3 separate points that don't connect.

-- "Substantial work has compared reasoning in LLMs to humans" -- it is odd to follow this sentence up with a reference to Stupple & Ball (2008) or Evans et al. 1983, which took place before 'LLMs' were a term. I'm not sure if the authors were tripped up by the use in the paper of "parallel processing" (which doesn't mean what it means in PDP connectionism).

-- I appreciate mentioning relevant LLM papers in the background section but it would be good if the authors further delineate what their work contributes beyond the already substantial contributions of those works, right now it isn't particularly clear what the contrast is. 

-- ״because friendship is a symmetric relationship, social networks are undirected graphs״ -- while it is true that many social networks are modeled to a first approximation using a simple undirected graph, I'm sure people in social psychology would balk at the statement that 'friendship is a symmetric relationship'. Also the consequent doesn't follow, social networks describe friendships as well as rivalries, hierarchies, family relations, and many other things besides not captured by a simple representation along these lines. As mentioned above, it is not clear to what degree this task is about 'relational understanding' more than it is about navigating an undirected graph, where the graph could describe spatial relations, concepts, or nonsense. 

-- it wasn't entirely clear to me how you check for grounding in a generalizable way. That is, I understand that in this case specifically, you used a different LLM to generate the sizes and weights of things, but if I wanted to ensure that my data generally matches reality for some arbitrary property, this seems like a currently unsolved problems in LLMs so we can't simply say 'oh, we'll use LLMs to make sure it is grounded'. 

-- I appreciate the decoupling of prompting, but what are the actual details of how this decoupling happened? Which prompts did you use and why? 

-- "publicly available language models" doesn't mean "open"

-- There are several cases in the paper where the authors make claims that are meant to be backed up with a statistical test. 

For example:
" In Fig. 2 we see this experiment for Comparison and Syllogisms. For Comparison, we can clearly see across nearly all models randomizing the order causes noticeable degradation (Gemma 2B and 9B perform near or below chance of 0.5, guessing or giving invalid answers). Reversing the presentation order also often negatively affects performance in many, but not all cases."

'We can clearly see' is not a statistical test. Presumably you want to use an ANOVA here or one of a myriad of other ways of assessing this claim. Similarly 'negatively affects performance in many, but not all cases' is a non-exact, non-specific way of putting a result. 

* "For most models we see a curve in which accuracy decreases with symbolic distance for very small symbolic distances, but then improves for symbolic distances >15." -- Again, this is highly inexact. What do you mean 'most models', what do you mean 'improves above 15'? What statistical test are you running to say these statements? Are you comparing a linear fit to a quadratic fit, for example? Are you simply fitting a linear line and looking at the rise? Are you doing a piece-wise and finding a transition point at 15, and what is associated value of the parameters for any statistical test you ran?

* ""Gemini Pro actually levels out rather than decreasing and GPT-4o only decreasing slightly, suggesting that they are particularly good at needle-in-the haystack style questions" -- here and elsewhere, while I appreciate the speculation, we see the problem with evaluating these closed models, we simply have no way of knowing what is going on. 

-- The authors should probably face head-on the fact that spatial distancing in these models isn't working like it does in people. In people it drops from a distance of 1 to 2, but then steadily rises. For these models it seems (to the degree that one can say anything) more like a u-shaped function. 

-- I didn't understand why you tested the effect of capacity in the ranges you did, sometimes it seems to go 0-100, sometimes 0-50, sometimes 1-20, sometimes 5-30. In cases where we see degradation it is reasonable to argue it exists, but in cases where we don't (JSON family, values 1-20) it isn't clear if this is a result of not going beyond the current range. 

-- Super minor hair splitting: 'nonsense relations' like 'Benny blexed Kelly' is not 'inconsistent' with the real world in the same way that 'iron is lighter than air'.

questions:
* It would be good if the authors could provide more details on their framework and how it works. 

* It would be good if the authors tested their frameworks on open models in a way that allowed them to draw scientific conclusions about why some models fail and some don't or show the effects they do. 

* It would be good if the authors justified their tasks (I'm not saying they aren't good tasks, I'm saying the authors should motivate them further). The introduction and background could use major tightening and also moving up the fact that you tested indeterminate reasoning and inconsistent premises, and also to explain more how this differs from other major works recently looking at relationship reasoning. 

* It would be good if the authors connected this better to the specific psychological literature

* It would be good if the authors provided statistical tests to back up their claims.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730513677828

thread_type:
official_review

<thread_object>
creator: reviewer_p5ze

<content>
summary:
This article introduces ReCogLab, a flexibly generated dataset for quantitatively measuring relational memory and reasoning performance in LLMs, and conducts several experiments using the framework. ReCogLab offers a fully customizable and automatically generated dataset, enabling researchers to delve into the nuances of cognitive processes.

The paper begins by introducing ReCogLab. Examples generated by ReCogLab come from one of four tasks: Comparison, Social Networks, Syllogisms and JSON Families. Each example includes context C, question Q and answer(answers) A. Every aspect of the data generated can be controlled via a configuration file specifying parameters, which allows for the creation of datasets with varying length and complexity. And for each specific cognitive probe, ReCogLab generates 50 validation examples to select the best prompt and 1000 test examples for comprehensive evalution. Notably, the authors decoupled the prompt and parsing hyperparameters from the LM to minimize the impact of prompting on LMs’ performance.

Then, the authors conduct several experiments to benchmark relational reasoning performance across different models and problem complexities and explore how reasoning performance depends on certain features. The article looks at transitive inference, congruency, the symbolic distance effect, identifying logical inconsistencies and indeterminate reasoning. These experiments were all made with little additional effort using the ReCogLab framework by altering the configuration files. The findings reveal that many human cognitive phenomena are mirrored in LLMs.

**Experiment 1: Transitive Inference**

Purpose:

To evaluate models’ ability to reason accurately about associations presented in context and measure whether there was a dependence on presentation order, and further measure how the ability depends on complexity measures of the reasoning problem. 

Results:

1.(Comparison and Syllogisms) Randomizing the order causes noticeable degradation while reversing the presentation order affects performance in many but not all cases.

2.(Syllogisms)The authors speculate that consistent presentation order may bias the model toward approximate solutions.

3.Performance generally degrades as the complexity of the problem increases across all models.

**Experiment 2: Congruent and Incongruent**

Background:

Logical reasoning is more accurate when logical premises are congruent with real-world facts for LLMs(as it is for humans).

Purpose:

To investigate the impact of congruence on language model performance.

Results:

Using congruent statements outperforms similar comparison problems constructed with incongruent statements.

**Experiment 3: Symbolic Distance Effect**

Purpose:

To validate that LLMs will also show a symbolic distance effect.

Results:

For models that perform above chance, there is a clear positive symbolic distance effect that starts when the symbolic distance is greater than 15.

**Experiment 4: Identifying Inconsistent Premises**

Purpose:

To explore whether LLMs have the ability of identifying when a premises is logically inconsistent.

Results:

Larger LMs perform better on detecting inconsistent statements.

**Experiment 5: Indeterminate Reasoning**

Purpose:

To evaluate models’ ability to understand when there’s insuffient information to draw a conclusion.

Procedure:

The authors start with comparison problems which contain a fixed label set of Yes or No. Then they modify the comparison problems from a linear chain to a random tree generation while still asking questions about two random nodes, which provides insufficient context.

Results:

There exists a bias toward incorrectly answering \textquotedblleft yes" on logical prompts when it is uncertain and reporting uncertainty when in fact it is unknown.

strengths:
**Flexibility and universality**

ReCogLab has significant advantages over the previous dataset in terms of flexibility and universality. Such a dataset would bring many conveniences to research in cognitive sicence and holds significant practical value.

**Good breadth**

This article demonstrates a good breadth, reprising experiments from multiple aspects of cognitive science, and measuring the ability of multiple LLMs across various dimensions.

weaknesses:
**Lack of evaluation of dataset quality**

When evaluating the quality of a dataset, people commonly employ methods such as statistical analysis, visualization, cross-validation, and expert review. However, in this article, we only see qualitative expressions about the performance of ReCogLab, such as its comprehensiveness and flexibility. And I’m wondering would it be possible to add some quantitative indicators to more strongly demonstrate the quality of ReCogLab or compare ReCogLab with those originally used in the same or similar experiments to illustrate its advantages?

**Lack of detailed experimental procedures**

The description of the experimental procedures could be more detailed. It did not provide a detailed explanation of how ReCogLab was used in these experiments and how exactly the parameters were set.

**Unclear motivation**

The authors haven’t provided evidence for their classification (comparison, social networks, syllogism, json families). What’s more, the decision to test all tasks as a whole is questionable, especially considering that previous research has addressed these aspects individually. The necessity of compiling these tasks for collective testing is not immediately clear, and this approach may need further justification.

**Lack of insights**
The authors simply test LLM without delving into a more detailed analysis of each aspect, which results in superficial analyses. The authors should perform deeper analysis, such as for the transitive inference, the authors should offer an explanation of why the change of order matters for LLM, for congruent why is it important (whether this phenomenon comes from data or from training algorithm or transformers architecture), ...

**Lack of novelty**
1. In Experiment 1, the authors' assertion that "premise order matters" was previously identified in the article "Premise Order Matters in Reasoning with Large Language Models" published on May 28th. In Experiment 2, the earlier paper "Language Models, like humans, show content effects on reasoning tasks" demonstrated the importance of congruence in reasoning. Furthermore, there appears to be no significant distinction between relational reasoning and general reasoning. Could the authors compare with those above-mentioned papers to address the novelty of this paper? Furthermore, could the authors talks about the difference between relational reasoning and general reasoning?

 2. Additionally, in Experiment 4, the detection of inconsistency is an old topic, and it comes as no surprise that Large Language Models (LLMs) are capable of performing this task, could the authors demonstrate some new discoveries in their experiments? Moreover, this topic bears a striking resemblance to Knowledge Base Question Answering (KBQA), can the authors provide the difference between the topic of relational reasoning and KBQA? 

**Some editorial errors and missing citations**

Introduction paragraph 1: While recent work...

Citations are missing.



Introduction paragraph 2: From this literature...

The authors didn’t point out what “this literature” refers to.

Introduction paragraph 3: In this work, we aim to provide an evaluation framework that allows for the systematic evaluation or LLMs on relational memory and investigation of different possible effects(many inspired from the cognitive science literature).

Maybe it means “of”?

Background paragraph 1: $A > B$, $B > C$, $B > C$, $C > D$, $D > E$

Perhaps “$B > C$” is repeated?

questions:
As seen above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732240956957

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
First, we would like to thank all of the reviewers for their time and helpful comments. They were genuinely really helpful for us in adding references, experiments and clarifications. We have incorporated this feedback into our draft and believe that the paper is better as a result.

We are glad that many of the reviewers found significant value to our framework. 
P5ze says “ReCogLab has significant advantages over the previous dataset in terms of flexibility and universality. Such a dataset would bring many conveniences to research in cognitive science and holds significant practical value.” Similarly, P3cs says our dataset generation approach is “exactly where evaluation should be going in AI and machine learning.” G6qF recognizes the importance of benchmarking aspects of cognition “Examining the capacities of LLMs is important, and it is important to understand the more difficult aspects of potential cognition such as relational reasoning. Establishing benchmarks is an important part of building this research program.” 

Several reviewers are also happy with the breadth of the experiments enabled with our framework and our careful analyses.
P3cs: “Finally, the analyses are comprehensive and balanced. The authors are cautious not to prematurely ascribe human-like cognitive capabilities to these systems, noting that there is more work to be done to show this. This is a refreshing contrast to much of the overzealous AI hype from other (non-cognitive-science) researchers in the field.”
P5ze: This article demonstrates a good breadth, reprising experiments from multiple aspects of cognitive science, and measuring the ability of multiple LLMs across various dimensions.

Next, we would like to respond to some common points of misunderstanding and more clearly restate our motivation and goal for this work. We think that reviewer P3cs stated it quite nicely:
“[T]he production of a sandbox testbed, rather than a static, fixed benchmark, is exactly where evaluation should be going in AI and machine learning. ReCogLab is a toolbox for creating nuanced cognitive experiments, rather than a dataset that someone can take off the shelf and apply without much thought.”
This was exactly our motivation! The goal of ReCogLab was not to introduce a single new, static dataset for evaluating LLMs. The goal was to create a framework which can generate datasets for comparing and evaluating language models along many different dimensions. Imagine that you are a Cognitive Scientist who has a hypothesis about how LLMs work motivated by work in relational reasoning in humans. Or, imagine that you are an LLM user or designer and you want to know how well different LLMs respond in different scenarios. What you want to do is isolate the variable you care about, such as the complexity of the underlying graph or the order in which statements appear. What prior work has done in the past is create single, one-off datasets, often hand-written, to study each particular effect and then validate results on a number of different models. What we are instead trying to do is create a framework where this researcher can instead use our framework and quickly set up this particular experiment and get results fast.

With this motivation in mind, we hope this clarifies a few points for reviewers. P5ze asks what is “the necessity of compiling these tasks for collective testing” and similarly G6qF asks “The authors main claim is that there isn't a set of all these benchmarks together, but one should simply stitch together other benchmarks as a set.” We think this misunderstands what it is that we are doing. Our paper is not a static collection of datasets; it is a framework for generating new datasets. We suspect that the fact that we deliberately generated some datasets and experiments to be similar to prior work’s experiments may have led reviewers to think this, but our motivation here was to show that our framework is able to generate datasets that include all of these prior experiments but also that our framework is able to generate new datasets which prior works did not and could not. So to answer the question “why not just staple together previous datasets” the answer is that if you want to: study the same parameter of interest (e.g. ordering) but also want to simultaneously vary another (e.g. context size) or study that parameter with much longer context windows, the prior datasets made for studying ordering will not work! Or, if there is a parameter you want to vary, a researcher can quickly add a new condition to the generation in a few lines of code and re-generate new datasets studying that new parameter. These things are simply not possible by using prior static datasets. The goal here is to minimize the time the researcher needs to set up the datasets for an experiment and make extremely specific specifications of the data possible, which we believe our framework provides.

(continued)
</content>
</thread_object>

</thread>
<thread>
creation date:
1733153756749

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Today is the last day reviewers can respond. We ask that you kindly acknowledge our responses if you haven’t already and ask any additional questions you need to finalize your review.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a framework, called ReCogLab, for generating synthetic evaluation data that permits careful probing of the various aspects of relational reasoning (in the psychological sense). RecogLab enables flexible testing, revealing that many human-like cognitive effects are also present in LLMs, offering a valuable tool for advancing AI research on human-like reasoning capabilities.  Overall, the reviewers agree that testing LLMs on their basic understanding of relations is useful for our understanding of both AI models and human cognition. However, the reviews also raised downsides. As one reviewer points out, the authors present the generative framework itself as a major contribution without providing much detail on it. While the authors provide some details in the appendix, they are not very detailed. Consider the rejection sampling. How exactly is this done? What does the pseudo-code look like? Are there important hyperparameters?Additionally, one may provide an (anonymous) code repository already for the reviewing process. Regarding details on the dataset, however, the appendix looks very nice. Thanks. Moreover, a discussion with one reviewer actually shows how deep ReCogLab can be anchored in the cognitive literature. This should definitely go into the final version so that even the computer scientist can understand the value of ReCogLab. This is even more important as the most critical review makes the point that the design choices of ReCogLab are not clear. For instance, do humans perform differently on these tasks, and might that be a reason that the different benchmarks are needed? However, due to the discussion, e.g. about the literature, this point seems to be somewhat addressed, and hence the paper should rather be accepted.