From Strategic Narratives to Code-Like Cognitive Models: An LLM-Based Approach in A Sorting Task
Authors: Hanbo Xie
Hua-Dong Xiong
Robert Wilson
Venue: COLM (2024)
Keywords: Large Language Models; Introspections; Cognitive Models; Programming Codes
TL;DR: This paper illustrates how large language models can turn verbal reports into programming codes, serving as cognitive models to predict and interpret human behaviors in a sorting task.

Abstract: One of the goals of Cognitive Science is to understand the cognitive processes underlying human behavior. Traditionally, this goal has been approached by analyzing simple behaviors, such as choices and response times, to try to indirectly infer mental processes. However, a more direct approach is to simply ask people to report their thoughts - for example, by having them Introspect after the fact about the thought processes they used to complete a task. However, the data generated by such verbal reports have been hard to analyze, and whether the reported thoughts are an accurate reflection of the underlying cognitive processes has been difficult to test. Here we take a first stab at addressing these questions by using large language models to analyze verbally reported strategies in a sorting task. In the task, participants sort lists of pictures with unknown orders by pairwise comparison. After completing the task, participants wrote a description of their strategy for completing the task. To test whether these strategy descriptions contained information about people’s actual strategies, we compared their choice behavior with their descriptions of the task. First, we compared the descriptions and choices at the level of strategy, finding that people who used similar sorting algorithms (based on their choices) provided similar verbal descriptions (based on the embeddings of these descriptions in the LLM). Next, we generated code based on their strategy descriptions using GPT-4-Turbo and compared the simulated behaviors from the code to their actual choice behavior, showing that the LLM-generated code predicts choice more accurately than chance other, more stringent, controls. Finally, we also compare the simulated behaviors of generated codes with those from standard algorithms and induct the strategies that this code internally represents. In sum, our study offers a novel approach to modeling human cognitive processes by building code-like cognitive models from introspections, shedding light on the intersection of Artificial Intelligence and Cognitive Sciences.

Published as a conference paper at COLM 2024
From Strategic Narratives to Code-Like Cognitive Models: An LLM-Based Approach in A Sorting Task
Hanbo Xie Huadong Xiongl2 Robert C. W2
Scchoo of Psychology, Georgia Institute of Technology, Atlanta, GA, USA 2Department of Psychology, University of Arizona, Tucson, AZ, USA
[hanboxie199] ,
| @gatech edu, bob wilson@gatech..dd
Abstract
One of the goals of Cognitive Science is to understand the cognitive pro- cesses underlying human behavior. Traditionally, this goal has been ap- proached by analyzing simple behaviors, such as choices and response times, to try to indirectly infer mental processes. However, more direct approach is to simply ask people to report their thoughts for example, by having them Introspect after the fact about the thought processes they used to complete a task. However, the data generated by such verbal reports have been hard to analyze, and whether the reported thoughts are an ac- curate reflection of the underlying cognitive processes has been difficult to test. Here we take a first stab at addressig these questions by using large language models to analyze verbally reported strategies in a sorting task. In the task, participants sort lists of pictures with unknown orders by pairwise comparison. After completing the task, participants wrote a description of their strategy for completing the task. To test whether these strategy descriptions contained information about people's actual strategies, we compared their choice behavior with their descriptions of the task. First, we compared the descriptions and choices at the level of strategy, finding that people who used similar sorting algorithms (based on their choices) provided similar verbal descriptions (based on the embed- dings of these descriptions in the LLM). Next, we generated code based on their strategy descriptions using GPT-4-Turbo and compared the sim- ulated behaviors from the code to their actual choice behavior, showing that the LLM -LM-generated code predicts choice more accurately than chance other, more stringent, controls. Finally, we also compare the simulated behaviors of generated codes with those from standard algorithms and induct the strategies that this code internally represents. In sum, our study offers novel approach to modeling human cognitive processes by build- ing code-like cognitive models from introspections, shedding light on the intersection of Artificial Intelligence and Cognitive Sciences.
1
Introduction
One of the goals of Cognitive Sciences is to understand the cognitive processes underlying human behaviors. Cognitive scientists have long strived for this goal by building computa- tional models that simulate cognitive processes, and have tested these models by comparing their performance to behavior on some task. However, building these models relies heavily on the cognitive scientist's intuition, which subject bias and blind spots, while testing these models is limited by the low dimensionality of the behavioral observations, which usually consist of only choices and response times.
*Corresponding Author

---

Published as a conference paper at COLM 2024
Instead of this behaviorist approach to Cognitive Science, another way to probe human cognition is to simply ask people what they are thinking - that is, to get them to infrospect about how they are performing the task. Such introspection research has a long and checkered history in psychology going back to the 19th century but despite early interest from the likes of (Simon & Ericsson| (1984), has had relatively little impact on modern computational cognitive science. A major reason for this lack of impact has been due to the difficulty of handling verbal data, which requires (labor-intensive) transcription and (often subjective) qualitative coding before it can be analyzed, making it hard to be compatible with traditional behavioral analysis and computational modeling. With the advent of Natural Language Processing and Large Language Models (LLMs), there is potential to link introspection the form of text to computational cognitive models, predicting human behavior and inducting - in a data-drinen manner - the underlying cognitive processes.
In this paper, we bridge the gap between introspection and cognitive models by using LLMs to generate computer code from verbally reported thoughts. By converting introspection to a series of symbolic operations as represented in a program Lipkin et al. (2023), this approach allows us to generate bespoke, data-driven cognitive model for each person and test whether the behavior generated by this model is consistent with the behavior the participant actually exhibited.
An ideal paradigm for accommodating such research should have two features. First, it should not be oversimplified so in the task, there are diverse strategies to achieve the goal and the performance depends on the strategy used. Second, it should be highly structured so that the strategies can be easily formed as programming codes for analysis. Christian & Griffiths (2016) wrote a book Algoritimms to Live By, which introduces a variety of problems that both humans and computers will face, and the algorithms they use to solve these problems. As one of the classical programming problems, sorting tasks is a good fit for our study. [Thompson et al. (0022) conducted a large-scale sorting experiment and collected rich behavioral and strategy descriptions dataset. Therefore, our paper, as an extension of their paper, focuses on this open dataset to investigate how we could generate code-like cognitive models in the sorting task from the strategy descriptions.
Our study leverages the public dataset to develop a pipeline that uses LLM to generate programming codes that could simulate sorting behaviors, based on the strategy description by human participants. We first used text embeddings of strategy descriptions to test their relationship to behaviors and found that the text embeddings are informative to predict strategies discovered from behaviors in the original paper. We then used these descriptions to generate codes that can mimic human sorting behaviors, compared the simulated results with the true behaviors of participants, and found a significantly higher accuracy than the chance level as well as the prior knowledge that LLM can predict. Finally, we inducted those generated codes into known strategies based on their simulated results and found some strategies are highly consistent with those identified from behaviors. In brief, our study shed light on formulating cognitive models from strategy descriptions by generating codes, paving potential path toward further development of cognitive sciences with LLM.
2 Methods
2.1 Task and Dataset
We used the dataset from Thompson et al. (2022), which consists of a sorting task in a social learning context. The whole dataset contains 3,408 participants, and each participant needs to complete 13 trials of the sorting task. In each trial, participants sorted six images, which were not previously known to them, into the correct order by choosing any pair to swap positions. Only pairs that were out of order could be successfully swapped (see Figure IB). When participants think everything is in order, they could click on 'inishh' to end the trial. Then, they will receive feedback about how many comparisons are made and whether the final order is correct or not. The number of comparisons and the outcome will be used to calculate their trial performance. Each participant will need to complete 13 trials of the task, with each task exhibiting random images well as underlying orders. After they complete
2

---

Published as a conference paper at COLM 2024
the whole task, they instructed to write down how they completed the task and what their strategies are, which are used in our analysis.
A
Prompt Content
B
Task
As function:
Hixcen
Initial order and order Input
Action sequences and outcomes output
Only besic swaps are allowed to implement the sorting.
fixed code template justify whether comparison successful.
Strategy Narratives
Incomroo compertoore)
C
D
Example Code (Participant 23)
Example 1
Participant 23, 'Sllcctinn Sort':
ville attempts att.mpttt:
'first selected the 1st image and every time cross chock with other then picked up second image and cross checked with other 4..and so on till image..
for de cargatlencintttta (ineiintttllllttittt)
for 3 ningrct per(inttllsstealll Ch. Lospsttoowttittiin
seggresaltii
sortol flog Palse
attomes mecpettompaa
bookk
AM. pww.peteeets...o corted_fig
break
ootold Dag.
Example
Participant 482, 'Gnome Sort'): 'most efficient split the set into groups and first sort each half individually. Compare each option other two in half, then each set of three will be sorted. Then take the first item the right half and compare it from right to left in the left half until stops. it is now in the right place. Then take the second item from the right half and do the same. Then take the lost item from the second holf and repent ogain. The whole list will be sorted!
Example Code (Participant 23)
len(current mid n // saet wach holf range(2):
GPT-4-0125-
preview
start widd end start rnd if else for: in renge end
Regular Expression
range(i end):
attemot, swap((1,k) 11 success:
Henge
range(edd, nn)
while
attempt Suap((1, 1
Example 3
Errors and Generated codes
If succoss:
True
Participant 1725, 'Backward Selection
Sort':
break
'Start the righthand side (6). Swap with 5, with 6 with with the position. Then repeat with 5 (switch with 4, with 5 with 5 with 1). Keep going until you 're at 1. Note: doesn't 't matter if the icons swap keep going with the pattern. You will end up with 15 comparisons every time!
with 1. That will lock
Example Code (Participant 1725)
fer reagect (1))
action Oe 1)
Sentiag 901 frevent cffistte log. speaeeeee "jaisk', Roee))
Figure 1: A. The Prompt Content to generate codes. B. The task procedure. C. Example strategy descriptions, with their behaviorally assigned strategies. The participants best fitted by the codes in their algorithm. D. Generated codes of the example participants. For exhibition purposes, we only keep the core logic of the codes.
3

---

Published as a conference paper at COLM 2024
2.2 Generating Executable Codes with GPT-4-Turbo with Zero-Shot Training
We use the most recent version of GPT-4-Turbo, GPT-440125-preeiieu to directly generate the codes. To make the codes executable and simulate behaviors efficiently, we restricted the form of the codes in prompts. The principle of restricting the code is to build human-like sorting function, which could directly take the task input and return sorting behavior, as general form of cognitive models. Therefore, first, the generated code has to be function, which accepts the initial order and true order as input. It should produce an action sequence directly to complete the sorting task. Second, the code cannot use any sorting function, look at the true order to get the sorting behavior. It should simulate how the participants do the task, and use pairwise comparisons. Third, to minimize the variances in justifying the task rule, we directly provide a function template that can tell the 'agent' whether pair of comparisons is successful or not. Finally, we also insert the participant's strategic description to ask the LLM to try to generate the function aligned with the description (see Figure JA). The prompts are fixed for each participant but only varying their strategic descriptions.
2.3 Behavior Simulations
After all codes were generated and passed the pre-examination for execution, we ran two types of simulations to generate sorting behaviors for different purposes. First, we retrieve the initial order and true order for each participant and each trial from the public dataset and input them into the codes. The code-like models will produce sorting behavior sequence like what human participants face in the real task. This result was used to evaluate how well the model can predict real human behaviors. We ran the other comprehensive simulation that looped through all possible permutations of the task (for length sequences, there are 720 permutations). We use this result to measure the behavioral outcome of the codes in this task and try to differentiate different strategies indicated by the codes.
To evaluate how accurately the model can predict, for each pair of the predicted and real action sequences, we compare them and count how many actions are predicted correctly.
The accuracy of each predicted sequence is computed by the number of correct predictions divided by the length of the true behavior sequence. To evaluate how algorithmically similar each generated code to each other, we used the comprehensive simulation result and compared each subject to each trial's behavioral sequence. We then averaged each pair of participants' similarity across 720 permuted trials. Specifically, for each pair of behavioral sequences, we define their similarity as the first index the two sequences deviate from each other as the proportion of the longer sequence. In this definition, when two sequences are identical, the similarity is 1. It is formulated as below:
S(A,B) argmin, # by) max( A),
(1)
3 Result
3.1 Text Embeddings of Strategic Narratives Reveal Information of Human Sorting Behaviors
Strategic narratives are summaries by the participant after completing the task. The in- formation underlying the strategic narratives should reveal how participants did the task. The original study modeled the participants' sorting behaviors by RNN and inducted 16 commonly used sorting algorithms by model comparisons. Here, we tested how strategy descriptions relate to these algorithms purely inducted from behaviors.
We input the strategy descriptions of all participants into the embedding model, text-ada-002 OpenAl (2023) and use K-mean Clustering by gathering them into different clusters. We mapped each participant's algorithm discovered from the original study by behavioral data into the cluster. We picked the two most commonly used algorithms (Selection Sort and Gnome Sort) with their forward and backward versions (see the number of participants
4

---

Published as a conference paper at COLM 2024
identified to all other algorithms by behaviors in table and found out that different algorithms distributed differently in these clusters. More interestingly, the same algorithms but with different implementational directions (i.e., 'from left to right' vs. 'from right to left') exhibit very similar distribution patterns across clusters(Fig, 2 Despite their descriptions may be different semantically, the similar distribution suggests that the text embeddings of strategy description can reveal the underlying algorithmic information. We further tested whether the text embedding sufficient to classify the two strategies and we found that with PCA to 200 dimensions (with 89% variances explained), text embeddings could well classify the two strategies with SVM with five-fold poos-validation, obtaining an average test accuracy of 88% 1 2%. This result suggests that strategic narratives can indeed reveal algorithms inferred purely from behaviors and could be potentially informative in building cognitive models. We also expand the test to eight common algorithms (merging forward version and backward version of the same algorithm) and 16 algorithus (without merges), and obtain averaged test accuracy of 37,4% 4% + 4.6%, 36.3% + 4.3%, exhibiting higher performance than chance level.
Gnome Sort
Selection Sort
N 711
996
0.0
0200
Fractions in each cluster
Fractions in each cluster
0315
Forward
6415
0c50
0025
0000
Cluster
Cluster
N 178
N 67
Fractions in each cluster
020
Fractions in each cluster
Backward
835
0.0
030
0.0
Cluster
Cluster
Figure/ 2: Text Embeddings from different algorithms exhibit different distributions in clusters, and the same algorithms with different versions exhibit similar patterns.
3.2 LLM--enerated Codes Surpass Chance and Prior Knowledge in Predicting Human Sorting Behaviors
Since strategy descriptions reveal the behavioral information, we create code-like cognitive models. These models mimic human sorting behavior by prompting 4-TT-4002--rreviee with the strategy descriptions. The generated codes can be executable after extracting and debugging from the LLM responses. For each code, we input the task's specific challenges faced by the corresponding participant to predict their sorting behaviors. We compared the true behavior sequence with the simulated behavior sequence at the element level. The result suggests that the model accurately predicts the actions of participants beyond the chance level, with 78% of participants surpassing the chance level (predicting 1 out of 16 possible actions, 15 possible pairs plus 'finish' action, see Figure 3/)). However, one could argue that the LLM's prior knowledge about sorting algorithms could influence its code generation, enabling it to produce sorting behaviors that align with human actions above chance levels, regardless of the content of the strategic descriptions. Yet, this in influence does not imply
5

---

Published as a conference paper at COLM 2024
random generation nor does it guarantee human-like behavior outcomes. To confirm the LLM makes use of the strategic descriptions to generate codes, we ran a simplified permutation test in which we permute code-participant correspondence to evaluate a null level of predicting human sorting behaviors by randomly generating sorting codes. Our results show that if the LLM does not use strategic descriptions to generate the codes, the mean prediction accuracy for behaviors is 14.8% (95% Cl: [14.3, 15.3]). Therefore, the true mean accuracy is higher and out of the null distribution from purely prior knowledge, with 49% of participants surpassing the prior knowledge. This result suggests that our model performance surpasses the chance and prior knowledge level, indicating its robust performance in predicting human sorting behaviors as cognitive models.
We further take a look at how the prediction accuracy could vary across trials and par- ticipants since we suppose algorithmic information revealed by strategic descriptions is different in these conditions. We first tested whether as the trial went on, the prediction accuracy of the model increased accordingly. The result shows strong correlation between the trial number and the predicting accuracy (see Figure 3B, = 0.82, P < 0.001). This indicates learning effect in their task that better performance occurs in later trials as already indicated by the original research. It might also be because the strategy descriptions happen at the end of the task, and it is more likely for participants to summarize their thoughts from the most recent trial (last trial).
We also test whether the prediction accuracy varies across participants as their task per- formance varies. We first ran an overall correlation across all participants, finding that task performance significantly positively correlated with predicting accuracy (r 0.21, < 0.001). This may indicate that participants with better performance may have clearer expressions about their strategy, and their behaviors are less random which could be eas- ier to be fitted by sorting codes. However, as we observed the plot and found there is mixture of distributions across these two variables (see Figure A1), we plot and calculate the correlation by previously behaviorally assigned algorithms. We still picked the two most commonly used algorithms here to calculate their correlations. As shown in Figure 3C and 3D, both algorithms of participants exhibited significant correlations (Gnome sort: == 0.19, p < 0.001; Selection sort: = 0.31, p < 0.001). However, their distributions of both task performance and prediction accuracy are quite different, suggesting that either the LLM may have inducive biases into some particular sorting algorithms or descriptions from some algorithms are more explicit and comprehensible, but those from others are more implicit and challenging to convey. This variation in semantic transparency among strategies could significantly influence the effectiveness of the LLM's code generation and subsequent behavior prediction.
In sum, the generated codes exhibit robust prediction accuracy above chance and prior knowledge and reveal individual differences and temporal relationships from participants' behaviors, suggesting their potential to serve as predictive and explainable cognitive models.
3.3 Comprehensive Simulated Behaviors in the Task Induct Strategies Underlying LLM-Gennrated Codes
Since these codes can predict the behaviors, and align with the simulated behaviors and strategy descriptions, we are interested in the underlying cognitive processes they represent. In sorting tasks, the most important cognitive process the strategy or algorithm humans use to sort unordered lists. As revealed in the original study, researchers used RNNs to iden- tify 16 standard and commonly used sorting algorithms for most participants. Following their hypothesis, we evaluate the similarity between each generated code and hypothetical algorithms. By running comprehensive simulations with the standard algorithm codes, we compare the simulated behaviors of each generated code to those of the standard algo- rithms. When generated code's ' simulated behaviors exactly match those from a standard algorithm (similarity 1), we attribute the code to that algorithm. We compared our results with the original study, which based its strategy attributions on behaviors. Attributions between codes and behaviors were consistent for 28.2% of participants.

---

Published as a conference paper at COLM 2024
A
B
Subtectbwis Accuracy Distribution
Acouracy Across Trials
420
Partiopent Accuracies
Chance 11/te
350
Accuracy
330
230
0.15
150
0.30
120
0.05
0.0
0.
0.000
Accuracy
1100
C
Gnome Sort
D
Selection Sort
3.50
0.75
Reediction scouncry
Peddcton
0.50
0.25
0:30
0.09
0.25
0.75
Figure 3: A. The prediction accuracy generated codes surpass random chance and prior knowledge by the LLM. B. The generated codes capture learning effects in human behaviors. C, D. The generated codes can capture individual differences in the task. For different algorithms, the LLM has different performance and variability in capturing such individual differences.
Moreover, we used a confusion matrix to compare LLM-inferred strategies with behaviorally assigned strategies, examining the alignment and discrepancies between LLM-generated computational models and algorithms based on actual participant behaviors. The result shows that several strategies well aligned (e.g., Selection Sort, Successive Sweeps) but also indicates the shortcomings of LLM in identifying some other strategies (e.g., Gnome Sort and Backward Gnome Sort) from purely strategy descriptions. To confirm whether this shortcoming in identifying strategies is because the LLM has some inductive biases to some particular algorithms or people from some of the algorithms find hard describe the algorithm clearly, we ran a recovery test that we instruct 2PT--4-0125-Peviii each algorithm this task context, and ran completely same pipeline as above. We found that even for the LLM-generated descriptions, the LLM cannot correctly identify those biased strategies (e.g,, Gnome Sort, see Figure A2). The recovered confusion matrix exhibits very similar pattern to Figure which suggests there is systematic inductive bias in sorting algorithms in the LLM.
to describe
4 Discussion
4.1 Introspections, cognitive processes, and behaviors.
Building precisely predictive and interpretable computational models is crucial to the field of cognitive sciences. Introspections can be informative in probing thought processes rather than purely inferring from behavioral observations. In our study, we showed that text embeddings of strategy descriptions can well predict strategies discovered from behavioral datasets. We not the first ones to observe such effects. Previous research revealed that both self-report ratings and Think- as natural languages can be used to predict human choice behaviors, well as probe the underlying cognitive processes with computational
7

---

Published as a conference paper at COLM 2024
Averaged Similarity Between Behaviorally Assigned Strategies and LLM Inferred Strategies
Uridentified 0.03
0.07
0.00
0.00
0.01 0.01
0.01
Gnome Sort
0.00
0.00
0.00
0.00
0.00
0.01
Selection Sort 0.04
0.43
0.05
0.06
0.00
0.00
0.05
0.08
0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
Insertion
0.8
Bubole Sort 0.05
0.11
0.09
0.23
0.00
0.00
0.13
0.32 0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
True Stratejies (Behowlrra Assignedi
Cumb Sort
Medified Selection Sort
0.02
0.00
0.02
0.6
Shaker Sort 0.04
0.05
0.08
0.12
0.00
0.00
0.19
0.16
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Sweeps-
0.00
0.00
Backward Gnome Sort 0.01
0.02
0.02
0.08
0.01
0.01
0.03
0.04
0.02 0.02
0.94
0.03
0.03
0.04
0.03
0.04
Beciwerd Selection Sort 0.01
0.00
0.01 0.02
0.00
0.00
0.02
0.04 0.02 0.22 0.05 0.06
0.01
0.05
0.04
0.05
Insertion Sort
0.00
0.00
0.00
Beckward Bubble Sort 0.02
0.02
0.03
0.03
0.00
0.00
0.04
0.05
0.04
0.03
0.09
0.30
0.02
0.09
0.09
0.36
Sort
0.00
0.00
Backward Modified Selection Sort 0.02
0.11
0.07 0.02
0.01
0.04
0.07 0.00
0.06 0.01
0.01
0.00
0.01
0.01
Backward Shaker Sort 0.01
0.01
0.01
0.03
0.02
0.01
0.04
0.06
0.03
0.02
0.05
0.05
0.01
0.05
0.11
0.12
Beckward Sweeps-
0.00
0.02
0.0
Grome Sort
Selection Sort
Comb Sort
Modified Selection Sart
Shaker Sort
successive Swees
Betkoaard Selection Sort-
Backward Insection Sart-
Bac corara Bubble Sort
Backwardd Comb Sort
Backward Successive Sworps
Insertion
Bubole
Dackward Madified Selection
Backward Shaker
Beckward
Predicted Strategies ILLM Inferred)
Figure 4: The confusion matrix compares the behavioral similarity of code-generated simulations to standard algorithms, organized by behaviorally identified strategies. It quantifies how closely codes, derived from participants' strategy-based descriptions, align with algorithmic benchmarks in behavioral outcomes.
models and neural networks (Morris et al. 2023 Xie et al. 2023). Our study again highlights the importance of introspection in building computational cognitive models of human behaviors.
4.2 Code-like models and behaviors.
In our study, we generated codes with the LLM based on the strategy descriptions from human participants, which are later used predict behaviors and induct strategies that they use throughout the task. With the code-like models achieving higher accuracy than chance level and surpassing the prior knowledge the LLM can produce, the descriptions do contain information that the LLM could utilize. However, the current model performance still inferior to the purely behavioral RNN model performance Correa et al. (2024); Ellis (2024), indicating that zero-shot learning is not yet sufficient to comprehend behaviors fully. The generated code mostly deterministic and optimal, overlooking human-centric cognitive features like memory, action biases, and randomness. To further encapsulate human-centered behavioral features, program induction could utilized, treating the generated code prior.
8

---

Published as a conference paper at COLM 2024
4.3 LLM, Introspections, Code-like Models, and Behaviors.
Interestingly, we also found that the LLM exhibits strong inductive biases towards some sorting algorithms. This bias affects both strategy attribution and model performance for specific datasets. The relationship between the LLM, introspections, code-like models, and behaviors can be regarded as a hierarchical Bayesian structure, where the domain knowl- edge of LLM serves as the prior when generating codes conditioned on the introspections. Therefore, correcting inductive biases of the LLM is as crucial as fitting the codes to behav- ioral data when generating precise and interpretable code-like models. Ellis (2024) revealed that tuned-prior LLM could outperform fit behaviors. This suggests the importance of teaching the LLM to be good 'scientist' in proposing models (generating codes), rather than serving as a mere cognitive model to predict behaviors.
pre-trained-prio LLM in generating codes that
4.4 Limitations and Future Directions.
Introspections and Task Settings. In the original study, the strategy descriptions are mainly used for communication purposes and are passed on the next generation of participants to learn. This means participants may not fully reveal their behavior, instead telling what they think optimal in the task. Additionally, each participant only provides one paragraph of introspection at the end of the task, which overlooks the dynamics of their behavioral evolution. A higher temporal resolution for probing thoughts, such as at each trial, would better capture behavioral patterns. The lack of demographic information also prevents us from investigating individual differences in task performance and strategy descriptions. These task settings constrain the predictive performance of code-like models, which could be better designed for a separate task.
The Biases of Large Language Models. This language-based program induction task re- quires LLMs to have the integrative capability to understand the instructions and strategy descriptions and generate codes correctly. We attempted use 'code-LLate Ladee-LLMM-334---nntttte hf' for the task but found the model failed to generate executable codes for all participants without human intervention, preventing us from further comparing their actual perfor- mance. We also used the most recent open-source models, 'LLaMA3-8b' and 'LLaMA3-70b', to classify the strategies participants used based on their descriptions. We found our cur- rently used model, 20T-4-0125pprevin exhibits better performance than the other two models (Fig. A3 Despite the superior performance, the current model still exhibits obvious inductive biases toward some algorithms, hindering the model from correctly generating codes for specific groups of participants. One easy way to investigate whether different LLMs exhibit different biases is to use mixture of models to optimize general perfor- mance. More importantly, correcting model biases by fine-tuning or instruction-tuning can help 'teach' models to distinguish different algorithms, benefiting downstream program induction tasks.
Interpretability of Generated Codes. In this paper, we have shown that the generated codes can serve as a form of cognitive model that can predict human sorting behaviors based on their strategy descriptions. However, how the codes represent the cognitive processes is poorly understood. In our analysis, we compare each participant's generated codes standard algorithms, trying to see how language-basee program induction captures human behavior compared to purely behavior-driven models. How the strategy descriptions provide information beyond behavioral observations and the standard algorithm hypothesis space is yet to be investigated. One future avenue could be decomposing the codes into primitives (e.g., operations, index, loop) and instructing LLMs to build purely on these primitives. By analyzing how each generated code organizes these primitives, can understand how codes vary from each other individually, and potentially identify whether biases stem from humans or LLMs among these primitives.

---

Published as a conference paper at COLM 2024
Author Contributions
H.X., H.X., and R.W. conceptualized the project. H.X. and H.X. designed the analysis pipeline. H.X. implemented the analysis and drafted the manuscript. H.X., H.X., and R.W. reviewed and revised the manuscript for submission.
Acknowledgments
We thank Thomas. L. Griffiths, Bill Thompson, and Ted Summers for helpful discussions and providing the datasets for our analysis. This work was funded by SCIALOG Award #29079 from the Research Corporation for Scientific Advancement (to RCW) and the OpenAl Researcher Access Program (to HX). We thank the four anonymous reviewers and area chairs for their constructive feedback. The authors declared no competing interests.
Data & Code Availability
The dataset and the codes for this https:/ /
paper can
be
found at
like cog model.
References
Brian Christian and Tom Griffiths. Algorithms to live by: The computer science of human decisions. Macmillan, 2016.
Carlos G Correa, Thomas L Griffiths, and Nathaniel D Daw. Program-based strategy
induction for reinforcement learning. ar preprint arXtt:2402.16666, 2024.
Kevin Ellis. Human-like few-shot learning via bayesian reasoning over natural language.
Advances in Neural Information Processing Systems, 36, 2024.
Benjamin Lipkin, Lionel Wong, Gabriel Grand, and Joshua B Tenenbaum. Evaluating statistical language models as pragmatic reasoners. arXiv preprint ar xrXiv:2305001020, 2023.
Adam Morris, Ryan W Carlson, Hedy Kober, and Molly Crockett. Introspective access to value-based choice processes. 2023.
OpenAL. OpenAI embedding models, 2023, URL https://platforrm htt.:ppplattoroooopaa...... com/docs/ Accessed: [September 24, 2023J.
Herbert Alexander Simon and K Anders Ericsson. Protocol analysis: Verbal reports as data. 1984.
B Thompson, B Van Opheusden, T Sumers, and TL Griffiths. Complex cognitive algorithms preserved by selective social learning in experimental populations. Science, 376(6588): 95-98, 2022.
Hanbo Xie, Huadong Xiong, and Robert C Wilson. Text2decision: Decoding latent variables in risky decision making from think aloud text. In NeurlPS 2023 Al for Science Workshop, 2023.
A Appendix
A.1 Prompts of Generating Codes from Strategic Narratives
For each participant, we extract their strategic descriptions and insert it into the prompt. The complete prompt is like below:
Create a Python sorting function 'humanlike Like-ilikessoao tite_ssorting which is directly executable and effective for general sorting task, based on strategic approach to sorting list of items. The function should simulate the decision-making process humans might use when sorting, without direct comparison to known "true order'. The function must:
10

---

Published as a conference paper at COLM 2024
1. Accept stimull' as a list of items to be sorted, with order' available only to the internal 'attempt_swap" function for providing feedback on swap actions. 'True_order' represents the correct order of items and is used to simulate feedback on whether a swap action brings items closer to their desired order:
def attempt_swap (current_ true_ true_order, action): index1, index2 = action
if index1 len len (current_ and 0 index2 len Len(current_orderr)
and index1 != index2:
item1, item2 current_order [index1], parrent_ordes [index2]
true true_posl, true_pos2 true ooderr index (item1), true true_ordes index (item2)
if (true_posi > true_pos2 and index1 < index2) or
(true_posi < true_post and index1 > index2):
current_
pearent__rddrree order [index2] = ourrent_order order [indexi], current,
current_order [index2], current_orde order [indexi] return True, ourdent_order return False, current_order
2. Return a log of actions attempted, regardless of the outcome, formatted formatteo as tuples of item indices attempted for swapping, well the outcome of this attempt. A 'finish' action signals the end of the sorting process. This log tracks the sorting actions as well as the outcome of the comparison, simulatinn step-by-step decision- mariiinnnmakingg process.
3. Contain all necessary logic for sorting within the function, making capable of sorting lists of any length using basic operations. The function should not depend on external libraries or Python's 's built-in sorting functions for its core sorting logic. To avoid infinite loop, the function should set hard limit up to 50 comparison attempts and always ended up with 'finis'. All the basic functions should be correctly used (if used).
4. Refrain from using 'true order' for direct comparisons with 'current order' to assess sorting completion. Instead, infer the completion through the sorting process itself, akin to how person might determine they have finished sorting without knowing the exact 'true 'rue_order'. 'true 'rdder can only be used in the 'attempt-suwp function.
Focus on generating executable, valid Python code that effectively sorts 'initial stimuli' into order. The code should first and foremost be functional and capable of performing the sorting task well under general conditions. Once functional validity is ensured, align the code as closely as possible with the strategic description provided, within the bounds of simulating human-like sorting behavior under fair computational constraints.
Please provide only the implemented function, ready for direct execution with 'initial-stimuli and ooder' inputs. Don't 't include any comments, explanations, notes, or examples.
The strategy description (strategy Aeserrition
For best reproducibility, we always fixed the hyper-parameters, with max_token 1024, temperature 0.1, seed 2024, with all other hyper-parameter their default values.
A.2 Representational Similarity Analysis: Aligning Generated Codes, Descriptions, and Simulated Behaviors
To investigate how well the strategic descriptions, generated codes, and simulated behaviors are aligned, we use inter-subject RSA to validate their correlations. The very first step of RSA is to define similarity matrix for each level of datasets. For strategy descriptions, we obtain the text embeddings by inputting the text into the text-ada-002 model by OpenAl (2023). We compute cosine similarity for each pair of participants to obtain the similarity matrix. For the generated codes, we use CodeBERT to obtain the embeddings of codes and similarly compute the inter-subject similarity matrix of code embeddings. For the behavioral similarity matrix, we computed behavioral similarity in the same way as described the
methods section 2.3
11

---

Published as a conference paper at COLM 2024
After three inter-subject similarity matrices were computed, correlated the upper trian- gular, non-diagonal elements of flattened similarity matrices to compare representational structures without redundancy self-comparsssss. To test the correlation, we used the Spearman correlation and permutation test with 2,000 permutations to define whether the correlation is statistically different from the null distribution.
Our results suggest that all three pairs of correlations are out of null distributions (description-code correlation:
0.07, 95%CI behavioral correlation: = -0.01, 95%CI = - 0.0008, 0.0008); code-simulated behavior correlation: 0.18, 95%CI [-0.0008, 0.0008)), ), indicating significant strong aligmments
-0.000,, 0.0008); description-simullted
among strategic descriptions, generated codes, and simulated behavior.
A.3 Behaviorally Assigned Strategies Sample Summary
Algorithm
Regular
Backwards
Selection sort
996
67
Gnome sort
711
178
Shaker sort
71
23
Modified selection sort
63
58
Bubble sort
47
38
Successive sweeps
44
Insertion sort
19
Comb sort Unlabeled
1112
Table 1: Algorithm Samples Discovered from Behaviors. Source from Thompson et al. (2022
A.4 Overall Subject-wise Task Performance and Prediction Accuracy
Overall Correlation
1.00
0.75
Prediction Accuracy
0.50
0.25
0.00
0.00
0.25
0.50
0.75
1.00
Mean Trial Score
Figure A1: The overall correlation between individual task performance and prediction accuracy. The distributions for both task performance and prediction accuracy suggest mixture of distributions the dataset and it is necessary to isolate the algorithms.
12

---

Published as a conference paper at COLM 2024
A.5
Recovery Test in LLM for Code-Generations
Averaged Similarity of Recovery Test
Unidentrlidd
Selection Sort 0.07
0.48
0.07
0.09
0.00
0.00
0.07
0.07
0.00
0.00 0.00
0.00
0.00
0.00
0.00
0.00
0.8
Insertion Sort 0.08
0.15
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Bubole Sort
0.00
0.00
Thue Strategies (Behavorall Assigned)
Camb Sort 0.00
0.00
0.00
0.01
0.00
0.00
0.00
0.02
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Shaker Sort 0.06
0.06
0.14
0.28
0.00
0.00
0.35
0.22
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Successive Sweeps 0.05
0.05
0.12
0.43
0.00
0.00
0.23
0.95
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Backward Gname Sort
0.00
0.4
Beciward Selection Sort 0.01
0.02
0.02
0.10
0.02
0.01
0.03
0.04
0.06
0.22
0.13
0.20
0.01
0.13
0.00
0.30
Sort
Backward Bubrole Sort 0.00
0.00
0.00
0.02
0.00 0.00
0.01
0.01
0.12
0.05
0.28
0.44
0.02
0.28
0.23
0.55
Backward Comb Sort 0.00
0.00
0.00
0.01 0.02 0.01
0.00
0.01
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.01
Backward Modified Selection
0.00
0.11
Backward Shaker Sort 0.01
0.01
0.01
0.03
0.00
0.00
0.03
0.02
0.10
0.05
0.25
0.23
0.02
0.25
0.31
0.19
Sweeps
0.77
0.0
Grome Sort
Selection Sort
Insertion Sort-
Comb Sort
Modiffed Selection Sort
Shakar Sort
Successive Sweeps
Backowerd Selection Sort
borkwaddd Insertion Sort
Escloward Bubble Sort
Bascoarrd tomb Sort
Eacoward Successive Sweess
Bubble
Bockward Croome
Backward Shaker
Becioward
Prodicted Strategies (LLM inforred)
Figure A2: Confusion matrix in recovery test. This recovery test indicates biases towards algorithms similar to those human data, such Gnome Sort and Backward Gnome Sort.
A.6 Strategy Classifications of Several LLMs
13

---

Published as a conference paper at COLM 2024
Accuracy of Model Predictions for Different Algorithms
0.7
mett:llama/MMtaallam------- 3-10-------B----
meta- meta-llama/Meta-Llama 1asamma---8-0- 3-8B-Instrud
0.6
11144-0125-previiee
0.5
Accuracy
0.4
0.3
0.2
0.1
0.0
Unidentified Gnome Sort Selection Sort
Insertion Sort
Bubble Sort
Modified Selection Sort
Comb Sort
Successive Sweeps
Shaker Sort
Backward Gnome Sort
Backward Selection Sort
Backward Insertion Sort
Backward Bubble Sort
Backward Modified Selection Sort
Backward Comb Sort
Backward Successive Sweeps
Backward Shaker Sort
Algorithms
Figure A3: GPT-4-0125-preview generally exhibits better performance than the two most
recent versions of LLaMA.
14


Reviews:
[{"review_id": "elUA5ujtqz", "reviewer": "Reviewer_myFw", "summary": "The paper proposes to use people's post-collected unstructured descriptions of their behavior in a human study (sorting task) to generate a computer code using LLMs. This code can be then run as a simulation and compared to the actual human behavior shown in the task requested, helping to shed light on the accuracy of the post-explanatory self reporting of the participants. Using GPT-4, the attributions between code and behaviors were consistent in 28% of the cases on a study of about 3k participant samples. Inductive biases of GPT-4 towards some more common sorts are shown and discussed.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback on our paper.\n\n**Regarding the use of different LLMs:** Due to limited time and resources, we employed LLaMA-3-70b to address a similar task. We clustered human algorithms based on their descriptions and utilized the standard algorithm clustered by the LLM to predict human behavior. The results were comparably good (mean accuracy: 23.6%). The predictions made by different LLMs were consistent for most individuals (correlation = 0.55). Our findings indicate that using both GPT-4 and LLaMA-3-70b does not significantly improve accuracy, as over 75% of participants' accuracy differences were within 5%.\n\n\n**On the suggestion to try keywords and key phrases:** This is an excellent idea. In our study, we instructed the LLM to generate code in a general format, adaptable to any sorting task length. Even when participants included action sequences in their descriptions, the LLM abstracted these sequences into executable and general codes. This abstraction suggests that the LLM can effectively distill the algorithms from the descriptions.\n\n**Concerning positional statements:** As noted, backward algorithms were less accurately identified and predicted. Our supplementary analysis revealed that 6 of 8 standard algorithms exhibit a bias toward the forward version in predictive performance. However, in the recovery test (Fig. A2, appendix), backward algorithms were better identified with LLM self-generated strategy descriptions. We believe that this bias observed in empirical data likely stems from unequal sample sizes in human data and biases in human descriptions.\n\n**On individual differences in participant backgrounds:** Little information is available beyond basic demographics, as participants were recruited online via Amazon’s Mechanical Turk. It is possible that the highest-performing participants had some education in sorting algorithms, providing clearer descriptions. This might also reflect the “good subject effect,” where participants who perform well on one task tend to perform well on all tasks, likely due to their higher investment in the experiment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper shows quite an innovative quantitative way to evaluate participant testimonies in behavioral studies using LLMs. It is well written and easy to follow.", "reasons_to_reject": "- Only one model and one test task is used in the study - it would be interesting to see how a range of different LLMs performs in this tasks (in relation to the found inductive biases for example; i would assume this can be a matter of LLM training data; with some luck one could find a few open-source LLMs that vary in this respect). \n- i'm not entirely convinced the full strategy description is relevant for the code generation; the authors show that the generation is significantly better than without it, but it would be interesting to check for inductive biases on a small set of keywords, for example if saying something like \"then I select\" can already trigger a full selection sort implementation without further ado.", "questions_to_authors": "- from the paper it sounds like the LLM generating code has some difficulties with positional statements ('left to right'/'right to left') -this could be a broader fundamental issues worth further exploring?\n\n- the prediction performance correlated with task accuracy - who were the participant? Are the higher performing ones more likely to have computer science education and with that default to sorting algorithm description that is more comprehensive for LLM to generate code?\nIf so, how does this generalize to other tasks - the low-performer case where the candidates try to describe their more \"ad hoc\", unstructured behavior can be of more interest in many studies, but sounds like LLMs would not be applicable for that?", "extra_scores": {}}, {"review_id": "OvYFyyOzMV", "reviewer": "Reviewer_fG11", "summary": "This paper uses LLM to translate participants' written strategy into program code and see if the code simulates their actual behavior. Usually, such translation is done by human labor. LLM can reduce costs and enable larger-scale cognitive experiments.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback on our paper. We also hope COLM could be a good place for this paper.\n\n**Addressing the gap between LLM and RNN:** One approach to close the gap between LLM and RNN performance is to consider biases and random deviations from the described strategies. While RNNs easily handle these effects, the LLM-to-algorithm approach tends to ignore them unless explicitly described. A potential solution is for the LLM to output algorithms that generate likelihoods instead of hard choices. This would allow for a principled combination with other biases and suboptimalities inferred from choices. For example, randomness could be accommodated with a softmax parameter. \n\n**Prompt Engineering and Inductive Bias:** We recognize that different prompts can introduce various inductive biases. This is a valuable consideration for future work to potentially close the gap between LLM and RNN performance.\n\n**Regarding the dataset and experimental design:** It is important to note that our initial work used a preexisting dataset from an experiment not designed for this type of analysis. The original study ignored the verbal descriptions, leading to a limited number of trials per subject and after-the-fact strategy summaries. Moving forward, acquiring verbal data in real-time as participants complete the task, for example, via Think Aloud procedures, and including more trials and tasks per subject would enable more detailed analysis and better application of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The purpose of this paper is to establish a technical foundation/verification for a tool to be used in future large-scale cognitive science research. This is a really valid purpose, and an AI/technical conference like COLM is a good place for papers like this. The motivation is clear. The paper is well-written. And the analysis seems detailed.\n\nThe authors found that vanilla LLMs cannot surpass an RNN at the moment.", "reasons_to_reject": "The paper could have explored more LLM strategies/techniques to see whether the gap between LLM and RNN can be closed with fine-tuning or prompt engineering. If not, then it would be nicer to know how other researchers in the field can hope to close the gap. Also, LLM's ability lies with generalization -- a behavioral RNN trained on one task might not generalize to another task. LLM however can. It might be nice to evaluate more than one dataset/task.\n\nI do think this is an interesting paper. I don't know where the bar of COLM should be (since this is the first year). If this paper is submitted to the CogSci conference, it will vote for acceptance. I wish the paper had more technical contributions or explored more on the methodology front.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a1IpBWrfqR", "reviewer": "Reviewer_ZEwv", "summary": "This paper aims to show that there is a parallel between humans and LLMs in solving problems which are solvable with specific sorting algorithms.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate your comments and have addressed the main concerns below.\n\n1. **Purpose of the Study**\n   The reviewer questioned the utility of using LLMs in our research. Our study investigates not the capability of humans and LLMs to solve the sorting problem, but their potential to reconstruct human cognitive processes in algorithm development based on human descriptions in naturalistic languages. This approach allows us to explore how LLMs leverage their inherent knowledge of sorting algorithms and humans’ language to simulate human problem-solving strategies, offering a unique intersection between AI capabilities and cognitive science insights.\n\n2. **Contributions to Cognitive Science**\n   The contribution of our research lies in demonstrating how LLMs can be utilized to model and understand human algorithmic thinking. By analyzing how LLMs reconstruct full algorithms from human descriptions, we gain insights into potential cognitive processes employed by humans, thus providing a novel method to explore human cognitive strategies through AI on a more detailed level.\n\n3. **Response to Specific Question**\n   We do not specify algorithm names; these algorithms are clustered in the previous study. In generations of codes, only human participants’ strategy descriptions are provided.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper investigates the parallel between humans and LLMs.", "reasons_to_reject": "Notably, LLMs are good memorization devices and these devices have been trained on code generated by humans. Then, these machines are really able to build up code of notorious sorting algorithms. The purpose of this study is then really unclear.", "questions_to_authors": "Are you giving the name of the algorithm in the prompt?", "extra_scores": {}}, {"review_id": "ZbX15Uaa1Q", "reviewer": "Reviewer_6wRT", "summary": "This paper highlights the challenge of analyzing introspective free-response data in cogntiive science, and proposes to use LLMs to generate code implementing the strategies people report. It shows that this technique successfully generates code that approximates some aspects of the human behavior, and outperforms various control conditions. Overall, I thought the approach was somewhat novel and interesting. However, what I felt was missing was a sense of what cognitive insight was gained by converting the free responses into code, and more discussion of the limitations and drawbacks of the approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful observations and valuable feedback on our work. We share many of your concerns about this work.\n\n**Comparing generated code with ground-truth algorithms:** We compared participant behavior predictions generated by GPT-4 with ground-truth algorithms from RNNs. The generated codes performed worse (mean accuracy 30.5% vs. 40.4%), but GPT-4 performed better for more subjects. This suggests that GPT-4-generated codes may worsen predictions or that RNN algorithms capture some biases more accurately.\n\nWe used GPT-4 and LLaMA-3-70b to attribute a standard algorithm from 16 given algorithms, then used this prediction as standard approximated codes. Changing models or tasks did not improve accuracy compared to the original GPT-4 codes, directing the idea that RNN-attributed algorithms capture biases better.\n\nData inspection showed participants better predicted by RNNs primarily used the 'selection sort' strategy, involving simple heuristic steps. GPT-4 was sometimes misled by subjects’ descriptions, causing errors. For some, descriptions didn't match actions, leading to incorrect code generation. In other cases, GPT-4 implemented 'selection sort' more efficiently, causing errors. Simple heuristic descriptions, like 'swap each pair of images', also confused GPT-4.\n\nThese findings indicate our prompts need refinement. We'll use advanced prompting techniques to better guide LLM agents in solving this task.\n\n**Fundamental utility and limitations:** We acknowledge that converting cognitive experiences from natural language or behavior into code requires significant abstraction. Currently, we don't expect LLMs to fully automate scientific discovery but to aid in generating hypotheses through code-like cognitive models. Traditional cognitive modeling involves iteratively fitting models to data, which can lead to overfitting and may miss important processes. LLM-generated codes expand the hypothesis space for cognitive models, serving as an initial step and benchmark. While LLMs can't capture all biases, they offer a human-understandable medium that aids model-building. This approach allows us to leverage LLMs' knowledge of sorting algorithms and human language to gain insights into human problem-solving strategies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This is an interesting approach to analyzing human free-response data.\n* The results show that the code generated from the free responses is predictive of their behavior.\n* There is some interesting analysis of the relationship between subject performance and the accuracy of the code predictions, as well as learning effects.", "reasons_to_reject": "* My main concern about this paper is that it's not clear what, cognitively, we learn by converting free responses into code. The paper could be more compelling if it demonstrated some insight that the analysis yields for cognitive science. Just predicting behavior above chance is not sufficient; indeed, as the authors note, an RNN-based approach would predict the behavior better (presumably because it can also account for biases etc.) The authors seem to hint that assigning participants to algorithm buckets is the cognitive insight (\"In sorting tasks, the most important cognitive process is the strategy or algorithm humans use to sort unordered lists.\"), but it is not obvious to me how much we learn from this (especially since it seems fairly weakly correlated with the behavioral sorting, and shows notable biases as the paper notes).\n    - A very basic analysis to start would be to see whether the generated code predicts participant behaviors better than the closest ground-truth algorithm to that code, for example. This is similar to the Fig. 4 analysis, but again, with more of an eye to explaining what the participants actually did and how that links to the self report, rather than something about the code/algorithm itself.\n   - It would be much more compelling if the code-based analyses brought some new kind of insight that could not have been gained from behavior alone, though I am not sure what exactly that might be.\n\n* More fundamentally, the idea of converting the hypothesis space for cognitive experience from the rich flexibility of natural language or behavior into code requires abstracting away many details that might be important to understanding the complexity of human cognition. If we think that code is only an approximation to the underlying messy way cognition actually works (cf. https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(10)00124-5) then we may be missing important details by moving to that level of abstraction. For example, there may be systematic biases, order effects, etc. in the human behavior that don't emerge in their self-report (many cognitive biases don't), but that nevertheless drive key aspects of their behavior. When we analyze humans through code that approximates their behavior, however, we abstract away those biases and effectively treat them as noise in the data. We also tend to restrict ourselves to research tasks (like sorting) that are amenable to code like analyses, rather than understanding rich naturalistic human behavior. Of course, research and analyses often require such simplifying abstractions, so there is certainly potential utility in the method. But I felt that a paper proposing a new method of analysis, that might be adopted by other researchers, should be more explicit about these tradeoffs in its assumptions.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces an approach for using LLMs to analyze the verbal reports that people provide to describe their introspective impression of how they performed cognitive tasks: the authors show that LLMs can be helpful for translating unstructured, introspection-based descriptions into structured code that can be systematically analyzed.

Reviewers pointed out several strengths:
- The proposal is innovative from both the LLM side (it is a novel application of LLMs) and the cognitive science side (it could be helpful in addressing an important question in cognitive science)
- The LLM-generated code is predictive of people’s behavior, suggesting that this approach is capturing something important about human cognition
- The paper is well-written and easy to follow

Limitations noted by reviewers include:
- Only one LLM and task are analyzed; it would enhance the paper to test the generalizability of the approach.
- The utility of this approach to cognitive science is currently mainly speculative. What could really make the paper compelling would be to show some concrete insight that is provided by using LLM-generated code but that would not have been obtained otherwise. That is definitely a lot to ask, and it’s hard to say what this insight would look like, but if it were accomplished it would really strengthen the paper.
- The code that is generated by the LLM reflects to some extent the inductive biases of the LLM (rather than being just determined by the language produced by the participants). However, the authors include some thoughtful discussion of this point - and, more broadly, any approach to interpreting language is guaranteed to rely on inductive biases of whatever does the interpreting, due to the ambiguity of language.

Overall, I believe that this paper is right on the edge - I think it could justifiably be accepted or rejected. Its ideas are interesting and well-executed, but its main utility remains fairly speculative. 

[At least one review was discounted during the decision process due to quality]
Decision: Accept