Title: Generating Computational Cognitive models using Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language models

cognitive computational models

cognitive science

learning

decision making

neuroscience
Abstract: Computational cognitive models, which formalize theories of cognition, enable researchers to quantify cognitive processes and arbitrate between competing theories by fitting models to behavioral data. Traditionally, these models are handcrafted, which requires significant domain knowledge, coding expertise, and time investment. However, recent advances in machine learning offer solutions to these challenges. In particular, Large Language Models (LLMs) have demonstrated remarkable capabilities for in-context pattern recognition, leveraging knowledge from diverse domains to solve complex problems, and generating executable code that can be used to facilitate the generation of cognitive models. 
Building on this potential, we introduce a pipeline for Guided generation of Computational Cognitive Models (GeCCo). Given task instructions, participant data, and a template function, GeCCo prompts an LLM to propose candidate models, fits proposals to held-out data, and iteratively refines them based on feedback constructed from their predictive performance. We benchmark this approach across four different cognitive domains -- decision making, learning, planning, and memory -- using three open-source LLMs, spanning different model sizes, capacities, and families. On four human behavioral data sets, the LLM generated models that consistently matched or outperformed the best domain-specific models from the cognitive science literature. 
To validate these findings, we performed control experiments that investigated (1) the contribution of the different LLM features (model size, model family, capacities); (2) the causal role of different prompt components; (3) the effect of data contamination; (4) the ability to recover ground truth models from simulated data; and (5) the total explainable variance in human behavior captured by LLM-generated models. 
Taken together, our results suggest that LLMs can rapidly generate cognitive models with conceptually plausible theories that rival -- or even surpass -- the best models from the literature across diverse task domains.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is builds computational cognitive models using LLMs.  It describes the method for generating computational cognitive models called GeCCo and then applies this to 4 main experimental outcomes: \n\t1. Decision Making\n\t2. Learning\n\t3. Planning\n\t4. Working Memory\nIt then evaluates the effects of different control experiments and finishes with a discussion of the model discovery capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n•\tThe proposed method has strong empirical performance across different cognitive tasks and models\n•\tThe proposed method relies only on in-context learning and is tested with open-source models, contributing to its accessibility and reproducibility\n•\tControl experiments, ablation studies, and ground truth recovery analysis add rigor to the results\n•\tGood experiment setup for the 4 classes.  Clearly written with value proposition and experimental setup.\n•\tAblation studies on prompt components was useful and Figure 9 was especially interesting.  Curious as to how variance on these components (e.g. how to formulate each of those portions of the prompts) also affects things.\n\nWeaknesses\n•\tThe paper focuses on evaluating the predictive performance of the LLM-generated models, but it does not assess their scientific value. Do the models offer new cognitive insights or reinforce existing findings? Are the underlying mechanisms theoretically plausible? The paper could benefit from additional qualitative analysis or evaluations by domain experts to help establish the scientific utility and interpretability of the generated models. \n•\tUsing BIC as the only model selection criterion may bias toward simpler models. Including alternative measures (e.g., AIC, cross-validation) would help support the paper’s claims about models’ generalizability. \n•\tThe paper fixes the number of iterations to 10 but does not analyze how model quality evolves across iterations. It would be useful to examine whether performance plateaus or continues improving, and whether fewer iterations can be used.\n•\tWould love more detail in the method section to fully understand the approach with more expanded and detailed explanation.  The current description is just so incredibly information dense, it is difficult to understand some of the exact methodology and algorithm without source code or some more structured framework describing the approach and its specifics.  An appendix of some exemplar models would be helpful for understanding the underlying methods a bit better.\n\nQUESTIONS:\nHow do some of the other popular models perform on these tasks (E.g. models from OpenAI / Anthropic / Google).\n\nI see you performed prompt ablations, but how did you optimize and arrive at your specific prompt form for each section?  \n\nWhat about using a reasoning models without a code template (since those don't rely as much on few shot)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCognitive models, in the sense of this work, are interpretable generative models with a small number of parameters that aim to explain human behavior in an experiment. Here, the authors prompt LLMs to write a Python program to explain the data from a human experiment. The prompting method is quite simple, works for open weights models, and yields interpretable models that have a better fit to human data.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS\n\nThe paper is well-written and the framing is compelling. \n\nThe problem is important: an automated way to speed up model discovery could accelerate science in general and cognitive science in particular.\n\nIn four case studies, the paper shows that the method is effective in finding interpretable models that explain the human data well, sometimes better than the best existing hand-crafted models.\n\nThorough follow-up experiments and ablations isolate the components of the prompting pipeline and the LLMs that make them most effective at this task. \n\nThe fit to human data is measured carefully using BIC.\n\nWEAKNESSES\n\nFrom a methodological perspective, the novelty is somewhat limited. Prior work has shown that LMs can be prompted to construct programs that explain data, especially the Li et al paper on statistical model discovery (cited in this manuscript). So if I understand correctly the main novelty here is the application to cognitive science, but this isn't made fully clear in the introduction. I am happy to be corrected on this and would appreciate author engagement.\n\nFrom the cognitive science perspective, it was hard for me to appreciate what new insight emerged from the automatically constructed models. There are four case studies, each on a different cogntiive domain, and there's not enough background information on any of them to convey to a reader who's not already familiar with those tasks what the LLM-generated cognitive model discovered and how it differs from existing models. Perhaps one of the case study is a slam dunk for the method but this should be made clearer; one option could be to move one or two the case studies to the appendix and focusing on the biggest success stories.\n\nThe tasks and resulting cognitive models are quite simple. I would have appreciated more detail on the synthetic data experiments, which could provide a nice way to determine whether the method can scale to more complex datasets.\n\nQUESTIONS:\nIt is hard to tell if the differences in prompting strategy between the current paper and the Li et al model discovery paper were significant, since no attempt is made to compare the new prompting pipeline to the Li et al one. Could you provide some clarifications on the comparison?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nComputational cognitive models are often hand-crafted by researchers, a process that can be highly time-consuming. In this work, the authors develop a new pipeline (GeCCo) for the automated generation of computational cognitive models. The authors apply GeCCo in 4 domains of cognitive science and assess the role of each component of their pipeline, as well as the base models.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is very well-motivated. The problem the authors take on, and the general method they develop, could have broad implications for cognitive science. The paper definitely hits the mark on originality. \n\nEach step in the proposed pipeline was also generally well-motivated, and I appreciated that the authors conducted rigorous ablations into the role of each component. \n\nWhere the paper struggled, in some respects, is in the actual communication of the experimental findings -- particularly cross-model comparisons. The authors rely heavily on BIC as a measure; however, the actual scale of BIC is not communicated anywhere in the paper. It is clear in many cases that BIC \"goes down\" for the LLM-generated models, but it's hard to understand from the aggregate measures alone whether that is \"significant\" (beyond statistical significance). The error bars were hard to interpret as well (and seemed large in many instances). Is this SEM over seeds (the 5 runs?) or bootstrapped over participant data? \n\nMoreover, in many instances, the BIC is actually not substantially going down? The role of the base models in producing good computational cognitive models is also highly sporadic. I did not walk away with a clear picture on the role of the base model in synthesis. The motivation suggested that the authors did care about base model, and of course -- as in good research! -- the results may be surprising (or negative, into the role of base models, as they seem to be here). However, I would want to see a bit more of an analysis into the role of base model. One possible way the authors could better understand the role of base model is by trying LLaMA 3.1 8b? Or at least one other model of the same class as one of the models they considered here? The choice of models seemed a bit scattered. For instance, we may expect that the results overall are worse when using 8b. But if they are not, then that would throw a bit of a wrench into GeCCo and the results would seem too sporadic; like it just matters that the models \"happen\" upon a good synthesized model. If possible, I'd encourage the authors to try 8b if they have not already. \n\nThere was also limited detail on the way the paired t-tests was computed. Paired over per-participant judgements?\n\nQUESTIONS:\nI would especially like a clearer description of the scale of BIC and error bar computation (as noted above), and other statistical tests run. \n\nThe ablations and Fig 11 indicate that feedback was very important. It'd be great if the authors could dig into this more (I believe the figure is never referenced in the text?) \n\nR1 beta spanned zero in the statistical models -- so can we really say it produced \"better\" models? I'd consider adjusting that sentence (or moving those analyses fully to the Supplement?)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Guided generation of Computational Cognitive Models (GeCCo), a novel pipeline that leverages Large Language Models (LLMs) to automate the discovery of cognitive models. The framework prompts an LLM with task instructions, participant data, and a code template to generate candidate models in the form of Python code. These models are then fitted to data, and their predictive performance (measured by BIC) is used to provide iterative feedback to the LLM, guiding it toward better solutions. The authors test this approach across four distinct cognitive domains (decision-making, learning, planning, and memory), demonstrating that the LLM-generated models often match or outperform state-of-the-art, human-handcrafted models from the literature.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- Significance and Originality: The paper addresses a significant and challenging problem in cognitive science: the time-consuming and expertise-dependent nature of computational model development. The proposed GeCCo framework is an original and innovative approach to this problem, effectively demonstrating how modern AI can be used as a tool for scientific discovery. The work is a compelling example of AI-assisted science and is likely to have a strong impact on the field.\n\n- Quality and Rigor: The methodological quality of the paper is significantly higher than your average \"we prompted some LLMs to do stuff\" papers. The GeCCo pipeline is well-designed, and the iterative feedback loop is a sound mechanism for guiding model discovery. The authors rigorously test their framework across four canonical and diverse cognitive paradigms, showing its versatility. The inclusion of multiple control experiments is a good too. The prompt ablations, data contamination analysis, and ground-truth recovery simulations provide strong evidence for the pipeline's robustness and help isolate the key drivers of its success. The comparisons are made against strong, well-established baseline models from the literature as well as CENTAUR, a foundation model of cognition, which provides an estimate of explainable variance.\n\n- Clarity: The paper is very well-written, clear, and easy to follow. The introduction does an excellent job of motivating the problem and situating the work within the existing literature. The GeCCo framework is explained clearly, and the schematic in Figure 1 provides an intuitive overview of the entire process. Providing the full prompts in the appendix significantly enhances the paper's reproducibility.\n\n\n# Weaknesses and Areas for Improvement\n\nWhile the paper is strong overall, there are several areas where clarification and more nuanced framing would strengthen it.\n\n- Clarity of \"Democratization\" Motive: The paper suggests that a primary motivation is to \"democratize and scale cognitive model development\". This is an interesting goal, but its meaning and scope are not fully clarified. Cognitive modeling is a highly specialized task, and it is unclear who the target audience for this democratization is. This point would benefit from a more concrete explanation of what a \"democratized\" modeling ecosystem would entail and why it is a necessary or desirable goal for the scientific community.\n\n- Distinction Between Models and Theories: The paper occasionally makes strong claims about LLMs broadening the space of possible \"theories\" and generating \"conceptually plausible theories\". While the work clearly demonstrates the ability to generate novel models with high predictive accuracy, a model with a good fit is not the same as a new scientific theory. Models don't need to be correct, they just need to be useful. But the word \"theory\" imply causal mechanisms that must be rigorously tested for correctness. \n\n- Interpretation of Statistical Significance: Some of the key comparisons, while promising, are not statistically conclusive. In the comparison against the CENTAUR foundation model (Table 1), the LLM-generated models do not show a statistically significant performance difference in two of the four domains (Decision-making, p=0.78; Planning, p=0.18). The claim that LLM-generated models \"matched or exceeded\" CENTAUR's performance across domains should be qualified to reflect this.\n\nQUESTIONS:\nQ1: Does achieving a better fit (e.g., lower BIC score) equate to a better scientific theory? How does the GeCCo pipeline ensure that the generated models represent genuinely new, theoretically insightful cognitive hypotheses, rather than just being more complex or cleverly parameterized functions that find statistical regularities in the data without corresponding to a plausible cognitive process?\n\nQ2: The prompt includes a Code template that often contains a baseline model from the literature. How can the authors be certain that the LLM is exploring a diverse model space, rather than being heavily anchored to the provided template? While the ablation study showed this was the 'weakest' predictor of performance, a weak effect is not a null effect.\n\nQ3: The paper presents a data contamination analysis using LogProber and finds no evidence of leakage from the prompts. However, many of the discovered model components are well-established concepts in cognitive science (e.g., forgetting parameters, separate learning rates for positive/negative errors etc). How do the authors disentangle genuine, data-driven model discovery from the LLM simply reconstructing known modeling components from its vast pre-training data, which almost certainly includes the cognitive science literature?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}