Evaluating the Robustness of Analogical Reasoning in Large Language Models
Authors: Martha Lewis
Melanie Mitchell
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) have performed well on several reasoning benchmarks, including ones that test analogical reasoning abilities. However, there is debate on the extent to which they are performing general abstract reasoning versus employing shortcuts or other non-robust processes, such as ones that overly rely on similarity to what has been seen in their training data. Here we investigate the robustness of analogy-making abilities previously claimed for LLMs on three of four domains studied by Webb et al. (2023): letter-string analogies, digit matrices, and story analogies. For each of these domains we test humans and GPT models on robustness to variants of the original analogy problems—versions that test the same abstract reasoning abilities but that are likely dissimilar from tasks in the pre-training data. The performance of a system that uses robust abstract reasoning should not decline substantially on these variants.

On simple letter-string analogies, we find that while the performance of humans remains high for two types of variants we tested, the GPT models’ performance declines sharply. This pattern is less pronounced as the complexity of these analogy problems is increased, as both humans and GPT models perform poorly on both the original and variant problems requiring more complex analogies. On digit-matrix problems, we find a similar pattern but only on one out of the two types of variants we tested. Lastly, we assess the robustness of humans and GPT models on story-based analogy problems, finding that, unlike humans, the performance of GPT models are susceptible to answer-order effects, and that GPT models also may be more sensitive than humans to paraphrasing.

This work provides evidence that, despite previously reported successes of LLMs on zero-shot analogical reasoning, these models often lack the robustness of zero-shot human analogy- making, exhibiting brittleness on most of the variations we tested. More generally, this work points to the importance of carefully evaluating AI systems not only for accuracy but also robustness when testing their cognitive capabilities.

Code, data, and results for all experiments is available at https://github.com/marthaflinderslewis/robust-analogy.

Published in Transactions on Machine Learning Research (02/2025)
Evaluating the Robustness of Analogical Reasoning in GPT Models
Martha Lewis
m.a.f.leaiss@wvo.r
ILLC, University of Amsterdam, Amsterdam, The Netherlands Santa Fe Institule, 1399 Hyde Park Road, Santa Fe, NM 87501, USA
Melanic Mitchell
mm@santafe. edu
Santa Fe Institute, 1899 Hyde Park Road, Santa Fe, NM 87501, USA
Reviewed on OpenReview: https: // openreview. net/ formm? id= t5cy5u9uph
Abstract
Large language models (LLMs) have performed well on several reasoning benchmarks, in- cluding ones that test analogical reasoning abilities. However, there debate on the extent to which they are performing general abstract reasoning versus employing shorteuts or other non-robust processes, such ones that overly rely on similarity to what has been seen in their training data. Here investigate the robustness of analogy-making abilities previ- ously claimed for one prominent class of LLMs GPT models three of four domains studied by Webb et al. (2023): letter-string analogies, digit matrices, and story analogies. For each of these domains we test humans and GPT models on robustness to variants of the original analogy problems- versions that test the same abstract reasoning abilitics but that are likely dissimilar from lasks in the pre-training data. The performance of a system that uses robust abstract reasoning should not decline substantially on these variants.
On simple letter-string analogies, we find that while the performance of humans remains high for two types of variants tested, the GPT models' performance declines sharply. This pattern is less pronounced the complexity of these analogy problems is increased, as both humans and GPT models perform poorly on both the original and variant problems requiring more complex analogies. On digit-matrix problems, we find a similar pattern but only on one out of the two types of variants we tested. Lastly, we assess the robustness of humans and GPT models on story-based analogy problems, finding that, unlike humans, the performance of GPT models are susceptible to answer-order effects, and that GPT models also may be more sensitive than humans to paraphrasing.
This work provides evidence that, dospite previously reported successes of GPT models zero-shot analogical reasoning, these models often lack the robustness of zero-shot human analoyy-making, exhibiting brittleness on most of the variations we tested. More generally, this work points to the importance of carefully evaluating Al systems not only for accuracy but also robustness when testing their cognitive capabilities.
Code, data, and results for all experiments is available at https://github. com/ harrhaelindeerseis//ooosss---al
Keywords: Analogy; Reasoning; Large Language Models; Evaluation; Robustness; Coun- terfactual Tasks
1
Introduction
The degree to which pre-trained large language models (LLMs) can reason iosson-deductivel, inductively, ana- logically, otherwise- remains subject of debate in the AT community. Many studies have shown that

---

Published in Transactions on Machine Learning Research (02/2025)
LLMs perform well on certain reasoning benchmarks (Huang & Chang 2022 Wei et al. 2022a bp. However, other studies have questioned the extent. to which these systems are able to reason abstractly, as opposed to relying on shortcuts Branco et al. [202]]; Taghanaki et al. (2024) or other heuristics, including "approxi- mate retricval" from encoded training data [ambhampati 2023) Several groups have shown that LLMs' performance on reasoning lasks degrades, in some cases quile dramatically, on versions of the lasks that
vary from standard benchmarks or that are likely to be rare in the LLMs' training data (Dviri et al. 2023
McCoy et al., 2024a| b Mirzadeh et al. 2024 Razeghi et al. 2022 Srivastava et al. 2024 Wu al. 2023 The lack of robustness to variations on tasks means that the performance of LLMs on worldworld tasks might not be well-predicted from their performance on standard benchmarks.
In this paper, we evaluate the robustness of analogical reasoning in three of OpenAl's GPT models. In particular, we evaluate the robustness of results reported by Webb, Holyoak, and Lu (2023) (hereafter referred to as WHL)), who carried out experiments to assess GPT-3's zero-shot ability to solve analogy problems in four domains: four-term verbal analogies, digit matrices, letter-string analogics, and story analogies. They found that on several lypes of problems GPT-3 matched or exceeded human performance, and concluded that "GPT-3 appears to display [zero shot] emergent ability to reason by analogy."
After repeating the experiments of WHL on letter-string analogies, tigit-matrin problems, and story analo- gies, we assess the robustness of analogical reasoning in both humans and GPT models by testing on variants of the original tasks that are unlikely to be similar to reasoning tasks seen in the models' training data. If an LLM (or human solver) is using robust abstract reasoning procedures, it should perform comparably on both the original tasks and variants; if it is using procedures that rely on shortcuts or similarity to training data, the performance should drop substantially on the variants
For letter-string problems we tested two types of variants using "fictional alphabets (1) in which the positions of some letters are perturbed; and (2) in which letters are replaced by non-letter symbols. On simple problems we find that while human performance remains high across variants, the GPT models' performance declines sharply. This pattern less pronounced as the complexity of these analogy problems is increased, as both humans and GPT models perform poorly on both original and variant problems requiring more complex analogies.
For digit-matrix problems we also tested two types of variants: (1) versions in which the position of the "blank" (missing answer) in the matrix was randomly chosen rather than always being the bottom-right entry; and (2) versions in which digits were replaced by non-digit symbols. For variants of type I we found that while human accuracy did not change from that on the original problems, the GPT models' performance again declines sharply. For variants of type we found that neither the performance of humans or GPT models changed substantially from that on the original problems.
Finally, we evaluate the robustness of humans and GPT models story-analogy problems in two ways: we examine (1) the effects of different ordering of the answer candidates and (2) the effects of paraphrased versions of storics. We find that, unlike humans, the performance of GPT models show strong answer-order effects, and that these models seem more sensitive than humans to paraphrasing effects.
Because the purpose of this paper is to evaluate the robustness of published claims of zero-shot analogical reasoning with simple prompts, we do not experiment with other prompting formats, such as multi-shot chain-of-thought prompting, or with models specifically trained for reasoning capabilities (e.g., OpenAT (2024)), but rather leave such evaluations for future work. However, the zero-shot, simple prompt case is arguably the most interesting one, as noted by WHL: "Of particular interest is the ability of these models to reason about novel problems zero-shot, without any direct training. In human cognition, this capacity is closely tied to an ability to reason by analogy."
This work provides evidence that, despite previously reported successes of GPT models on analogical reasoning, these models often lack the zero-shot robustness of human analogy-making, exhibiting brittleness on most of the variations we tested. More generally, this work points to the importance of carefully evaluating AI systems not only for accuracy but also robustness when testing their cognitive capabilities.
Contributions:
We did experiment on WHL's four-term verbal analogies; the ability to solve such "proportional analogies" hased single words comes easily to GPT models likely due to properties of the word embeddings they learn Chiang et al. 2020)
2

---

Published in Transactions on Machine Learning Research (02/2025)
Data Availability: Code, data, and results for all experiments available at httpe://github..com
2
Letter-String Analogies
2.1
Background
Letter-string analogies were proposed by Hofstadter (1985) an idcalized domain in which processes un- derlying human amalogy-making could be investigated. The following sample problem:
a abed-aber
ijkl
Here, c d + is generate a new string that transforms the target analogously to the source transformation. There is single correct answer to such problems, but there is typically general agreement in how humans answer them. For example, for the problem above, most people answer i k m, and answers that deviate from this tend to do so in particular ways (Mitchell 1993).
b c called the source trunsformation and i k is called the target. The solver's task
In addition to the work of Hofstadler & Mitchell (994 on creating computer models of analogy-making using this domain, letter-string analogies have been used to isolate the neural correlates analogy formation [Long et al. 2013 Gcake & Hansen 2010), and to model higher-order analogy retricval (Dekel al. 2023
WHL compared the ability of GPT-3 with that of humans a dalaset of letter-string analogies, finding that, in most cases, GPT-3's performance exceeded the average performance of the human participants. Here, performance is measured as fraction of correct answers on given set of letter-string problems, where WHL used their intuitions to decide which answer displays abstract analogical reasoning and thus should be considered "correct." In this paper we will use their definition of correctness.
WHL's s dataset consists a set of problem types involving different kinds of transformations and levels of generalization. The following are the six transformation types with sample transformations:
1. Extend Sequence: bbd - a aede
2. Successor: a b c d + aabe
3. Predecessor: bede + aadde
Remove Rodundant Lettor: a + abcd
Fix Alphabetic Sequence: abewe-abed.
Sort: ad c blbe +abede
Each type of transformation can be paired with "zaro-generalization" target (e.g., ij k or with between one and three of the following types of generalizations:
Letter-To-Number a abcd + +abce 1234 4 ?
Grouping: a + +abe ; iijjkkll+ ?
Longer Target: abcd + +abee m ?
Reversed Order: abcd-abee 1kji ?
Interleaved Distractor: abcd +bcd+abbe ixjxkxlx=* ?
6. Larger Interval: b +abee ikmo
3

---

Published in Transactions on Machine Learning Research (02/2025)
The following are examples of (1) 1-generalization problem (Extend Sequence paired with Longer Target); (2) 2-generalization problem (Remove Redundant. Letter paired with Letter-To-Number and Grouping); and (3) a 3-generalization problem (Successor paired with Grouping, Reversed Order, and Interleaved Dis- tractors).
abbe ; ijkl + ?.
abbc-abe ; 1122333344-
3. abc--bb llxkkxjjxiix= ?.
Finally, WHL included a number of problems involving "real-workd" concepts, such as,
bc - a cold cool warm +
In this paper we do not include the "rral-world" concepts in our studies, focusing instead on non-linguistic inputs.
WHL generated numerous problems for each problem type and presented these to GPT-3 (text-davinci-003 as well as 57 UCLA undergraduales. The human participants exhibited a large variance in accuracy, but on average, WHL. found that GPT-3 outperformed the average human performance on most problem types.
2.2 Variants of Letter-String Analogies
To assess the robustness of WHL's results on letter-string analogies, created variants of the same letter- string analogy problem types, and evaluated both humans and three GPT models on these versions.
Fictional Alphabets We created two types of variants using "fictional" alphabets: problems using per- muted alphabets and problems using "alphabets" of symbols rather than letters. To create a permuted alphabet, we reorder a letters, where can be 2, 5, 10, or 20. For each of the four values of n, we generated seven distinct alphabets an (i), i {L.... distinct permuted alphabets. We also generated two symbol alphabets, (1) and (2), cach consisting of 10 non-letter symbols in given order, and seven symbol alphabets (i) i {1...., each consisting of 15 non-letter symbols in given order.
..., 7}, with n randomly chosen letters reordered, yielding x 28
Creating Zero-Gennral Letter-String Problem Variants For the zero-generalizattinn case, for each 0r, (1), we created 10 different letter-string problems for each of WHL's six transformation types. This results 7 10 x 420 zero-generalization analogy problems for each value of n. We added to this 420 letter-string problems using the non non-permuted (n = 0) alphabet, spread evenly over the six transformation types. Figure Ta gives example of a Fix Alphabetic Sequence problem using an alphabet with two letters (e and m) reordered.
For each of the symbol alphabets (1) and 22) we created 10 problems each for the Successor and Predecessor problem types, for a total of 40 distinct non-letter symbol problems. Figure 1b gives an example of a Predocessor problem using a symbol alphabet.
Creating Letter-String Problem Variants With Generalizations To create variants of letter-string problems with generalizations, we used the same alphabets (t), well as the seven longer symbol alpha- bets, (i).
Recall that WHL created problems that paired each transformation type with either one, two, or three generalization types.
To generate L-generalization problems with permuted alphabets, for each a, (i), E {l...., 7}, and for each of the six gencralization types, we created 10 problems, cach with a randomly chosen transformation type. This yields x 6 10 420 distinct 1-generalization permuted-alphaban problems. We added to this 420 distinct I-generalization problems for the original (n 0) alphabet, spread evenly over the six transformation types.
4

---

Published in Transactions on Machine Learning Research (02/2025)
To generate 1-generalization problems with symbol alphabets, for each (1s E {1,...,), and for each of the six generalization types, we created 10 problems, each with randomly chosen transformation type, for a total of 7 x 6 x 10 = 420 distinet I-generalization symbol-alphabet problems.
For two- (three-) generalization problems, for each number of letters permuted and each alphabet, we create problems with two (three) randomly selected generalization types and randomly selected task. We thus create 7 70 490 unique two-generalization problems and the same number of unique sure--geneallzztton problems. Due to the difficulty of two- and three- cirreggenrralization symbol problems for both humans and GPT models, we limited our human and GPT experiments to symbol problems only with zero- and one- generalizations.
2.3 Methods For Experiments On Letter-String Analogies and Variants
Human Study Methods For all human studies, we collected data on Prolific Academic Participants were screened have English as a first language, to currently be living the UK, the USA, Australia, New Zealand, Canada, or Ireland, and to have a 100% approval rate on Prolific.
In order to assess humans' abilitics on the original and variants of letter-string problems, collected data from 263 participants. Each participant was asked to solve 14 letter-string analogy problems drawn from original, permuted, and symbol alphabets, with different numbers of generalizations. The 14 problems given to each participant were sampled from different transformation types.
In addition to the problems, participants were also given two attention-check questions at random points during the experiment, with a warning that if the attention checks were failed, then payment (S7 for the experiment, which was expected take about 20 30 minutes) would be withheld. Figure[Io an example of an attention check. Four of the 263 participants' submissions were rejected due to failed attention checks. As in WHL's studies, as part of the initial instructions, participants were given simple example problem to complete and then were shown the solution.
II Compition
Pregeees
Compatten Prggees
Preblem
Probiem
Jas the to siphabetto guea the piasinppice.
Uhe the bhbhwheeehhbbtteeoo missing piace.
Use the to lowing aiphabet guess the miliing plece
For instustton and is
Complite
Complese
triight J Di C73
Pa - pixi C73
Saand Diaced Desund L73
Orense
One net
(a) Example analogy problem with permuted alphabet.
(b) Example analogy problem with symbol alphabet.
(c) Example attention check.
Figure 1: Example items in human study.
GPT Study Methods We evaluated the performance of three GPT models GPT-3 (1ext-davinci-003, which the model tested by WHL), GPT-3.5 (gpt-3.5-turbo-061)
and GPT44 (gpt-4-0613) on the same problems given to humans. Following WHL, all GPT experiments were done with temperature set to zero. GPT-3 takes in a single prompt, whereas GPT-3.5 and GPT-4 take in a list of messages that define the role of the system, input from a "user" role, and optionally some dialogue with simulated responses from the model given under the role "assistant."
httpp://www.rrllfff..omo//o/
5

---

Published in Transactions on Machine Learning Research (02/2025)
For our experiments on the original letter-string problems used by WHL, our system and user prompts for GPT-3.5 and GPT-4, have the following format:
System: You are able to solve letter-string analogies.
User: Let's try to complete the pattern:\n\nfa b c j
The prompt is identical to the prompt WHL gave to GPT-3; the An character signifies line break to the model.
For our experiments on letter-string analogy variants, we tested three different prompt formats, including one similar to instructions given our human study and one in a zero-shot chain-of-thought setup. The best performance across models was achieved with the prompt format used in Hodel & West (0023
System: You are able to solve letter-string analogies.
User: Use this fictional alphabet: [a u c d wwweffgiijiimooooeeoooooooooooee jikmmmoprrsbbrwwyy 2]. \nLet's try to complete the pattern: n/a u d] (a c e]
Appendix A.1.2 gives the other prompt variants we tested, and Appendix zero-shot chain-of-thought experiments.
A.2.1 gives results from the
Note that in our studies, the "fictional alphabet" part of the prompt and the alphabet listing was included even for problems using the per-permuted (n 0) alphabet.
In this and all other experiments, we tested GPT-3 with concatenation of the system and user prompts. Following WHL, in our experiments all GPT model responses were truncated at the point where closing bracket was generated.
2.4 Replication of WHL's Studies
Our first set of experiments attempted a replication of WHL's experiments testing humans and GPT-3 on letter-string analogics.
Replication: Human Study Results Figure compares the results of our human study with that of WHL on the original letter-string problems, averaged across transformation types for different numbers of generalizations. The participants in our study achieved higher average accuracy than those of WHL on zero- generalization problems, and similar accuracy problems with one more generalizations. The differences zcro-generalization problems may be due to differences in experimental protocols or in the participant pools.
Replication: GPT Study Results Figure 3 shows GPT-3 data from WHL compared with data from our computational experiments with GPT-3, GPT-3.5 and GPT-4, averaged across transformation types for different numbers of generalizations. In all cases our GPT-3 results are similar to those of WHL. GPT-3.5 and GPT-4 show slightly lower accuracy than GPT-3, possibly due to their fine-tuning beyond strict prompt-completion objective.
In summary, our human and GPT-model replication results are generally consistent with those of WHL, although our human study yields higher human performance on zero-generalization problems.
2.5 Results Variants of Letter-String Problems
Counterfactual Comprehension Check On Fictional Alphabets For problems involving permuted alphabets, we follow Wu el al. (2023) by providing counterfactual comprehension checks (CCCs) to test if the models grasp the basics the task proposed. We use two CCCs: firstly, given an alphabet and a sample letter (or symbol) from that alphabet, give the successor of that letter. Sccondly, we use the same format but ask for the predecessor of the letter. We ensure that we do not ask for the successor of the last letter the alphabet or the predecessor of the first. Details of our CCCs given in Appendix A.1.T In summary,

---

Published in Transactions on Machine Learning Research (02/2025)
1.0
WHL Ours
0.8
Accuracy
0.6
0,4
0.2
0.0
Num. Generalizations
Figure 2: Human results WHL's original letter-string task for problems with zero to three generalizations. "WHL," refers to the results of WHL's original human studies, and "Ours" refers to the results of our human studies. Data points give mean accuracy across all transformation types, and bars indicate 95% binomial confidence intervals. Numbers of samples for WHL are 342 for each number of generalizations. Numbers of samples for our data are 276 for zero generalizations, 138 for one, and 92 each for two and three generalizations. Figure is best viewed in color.
1.0
GPT-3 WHL
GPT:3
GPT-3.5
0,8
Accuracy
0.6
0.4
0.2
0.0
3
Num. Generalizations
Figure 3: GPT results on WHL's original letter-string task for problems with zero to three generalizations. "GPT-3 WHL" refers to WHL's results on GPT-3. "GPT-3", "GPT-3.5", and "GPT-4" refer to our results with those models. Data points give mean accuracy across all task types, and bars indicate 95% binomial confidence intervals. Numbers of samples for GPT-3 WHL are 300 for each number generalizations. Numbers of samples for our data are 420 for zero and one generalization, and 490 for each of two and three generalizations. Figure is best viewed in color.

---

Published in Transactions on Machine Learning Research (02/2025)
find that the CCC accuracy is generally quite high, indicating that the models generally grasp what "successor" and "predecessor" mean in each fictional alphabet.
Humans
GPT-3
GPT:3.5
0.8
Accuracy
0.2
Num, Generalizations
Figure 4: Accuracy for humans and GPT models in our studies, versus number of generalizations. Data points indicate mean accuracy across all alphabets and problem types and bars indicate 95% binomial confidence intervals. Number samples for GPT models for zero genoralizations is 2,140, for humans 1,876. Number of samples for GPT-3 for one generalization 2,100, for GPT-3.5 and GPT-4 2560, and for humans is 1,062. Number of samples for GPT models for two and three generalizations is 2,450, and for humans is 504. Note that mean accuracies for and generalizations do not include symbol alphabets, and mean accuracies for GPT-3 L-generalization also do not include symbol alphabets. Figure is best viewed in color.
Zero Generalizations
One Generalization
Two Generalizations
Three Generalizations
0.8
GPT-3
GPT-3.5
GPT4
+*
0.0
2
5
10 20
Symb
Symb
10
20
100
20
Letters Permuteed
Letters Permuted
Letters Parmutee
Num: Permuted
Figure Accuracy of human participants and GPT models versus different alphabet types (number of permuted letters), for different numbers of generalizations. all transformation types for different alphabets, and bars indicate 95% binomial confidence intervals. The number of samples for each human data point is given in Table The mumber of samples for GPT-model data point is as follows. For zero one generalizations, and 0-20 letters permuted, each data point corresponds to 420 samples. For two or three generalizations, 0-20 letters permuted, each data point corresponds to 490 samples. For the symbol alphabets, zero-generalization data points correspond to 40 samples and one- generalization correspond to 420 samples. Figure is best viewed in color.
Data points indicate mean accuracy across
Variants of Letter-String Analogy Problems Figure Aeguressowee shows the mean accuracy and 95% binomial confidence intervals for humans and GPT models across all alphabets and problem lypes. In all cases the average human performance on these problems is higher than that of any the GPT models, and in the case of zero generalizations it is substantially higher. Note that GPT-3 was not available for our experiments on symbol alphabets with one generalization. Note also that due funding limitations, we did not test humans on two- and auree-generalizztion problems for symbol alphabets, for both humans and GPT models, we only report results two- and three-generaliizttion
Results On Human and GPT Experiments

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: Number human samples per data point in Figure
A Alphabets
Num. Generalizations
2
10
20
Symb
0
276
342
336
384
270
268
138
153
156
150
159
300
2
92
102
104
100
106
3
92
102
104
100
106
problems for permuted alphabets. Given the very low accuracies one-generalization problems with symbol alphabets, we expect that the two- and three-generalization accuracies would be equally low or lower.
Figure 5 shows the performance of human participants and GPT modols, averaged across problem types for problems with different alphabets and numbers of generalizations.
For vero-generalization problems, human performance is significantly above the performance of each GPT model, across all types of alphabets, and, most notably, stays relatively constant across alphabet variants. In contrast, the GPT models show more dramatic drop for the alphabet variants, especially in the case of symbol alphabets. This shows that for these simpler problems, humans are robust to variants whereas GPT models are not. This pattern is also present, to a lesser degree, for one- and two-generalization problems, except that human accuracy drops substantially on one-generalizatio symbol problems. For the most difficult-hhree-ee thwr---enneallaaiion----ee accuracy across alphabet variants.
humans and GPT-models maintain relatively constant (poor)
3
Digit Matrices
3.1
Background
WHL proposed digil. matrices novel analogy-making domain inspired by Raven's Progressive Matrices The following is sample digit-matrix problem:
(RPM) Raven 1938
[2]
[3]
[4]
[3]
[4] [5]
[4]
[5]
[]
As in RPMs, the challenge is to recognize patterns across the rows and columns, and to in the blank (C ]) cell.
The original RPM problems were created mamually, and only small number (108) were in the original formulation. Matzen al. (2010) identified number of rules used in creating these problems; WHL used these rules to programmatically generate digit-matrix problems of different problem types:
Constant: the same digit occurs across each row or column. Example:
[2]
[2]
[2]
[5]
[5]
[5]
[6]
[6] []
Distribution of a set of digits is permuted across rows and columns. Example:
[2] [6] [6]
[5] [6] [2]
[6] [2] [
Progression: digits increase decrease by or across rows column. Example:

---

Published in Transactions on Machine Learning Research (02/202))
[2] [3] [4]
[3] [4]
[5]
[4] [6] []
Logic: digits in a particular row or column are a logical combination digits in the other rows or columns. The logical operators used are AND, OR, and XOR. Example (OR- the last column is the union of the two previous columns):
[1] [3] [1 3]
[5]
[6]
[5
[1 5] [6
WHL tested humans and GPT-3 on digit matrices with up to three rules combined (logic problems were tested only with a single rule). The tests were done in two ways: (1) solvers were asked to select an answer from a list of candidate answers; and (2) solvers were asked to generate an answer. Our tests included only the "generate answer" format.
3.2 Variants of Digit-Matrix Problems
To assess the robustness of WHL's results on digit matrices, we created two types of variants on the digit- matrix lask: one which we randomly assign the matrix position of the "blank" (answer element), and the other in which we replace digits with non-numeric symbols. As in our letter-string problems, these variants rely on the same abstract reasoning processes needed to solve the original digit matrices.
Alternate Blank Position Using WHL's generation process, generated digit matrices similar to those of WHL, but with the position of the blank element chosen randomly, instead of always being in the bottom- right position. Figure Ga gives an example.
Symbol Matrices We replaced the digits by non-numerical symbols in each the digit-matrix problems used by WHL in their experiments (excepting the "progression" -type problems, since the symbols have no inherent ordering). Figure 6b gives an example.
3.3 Methods for Experiments on Digit Matrices
Human Study Methods We collected data on Prolific Academic. Participants were screened to have English as a first language, to currently be living in the UK, the USA, Australia, New Zealand, Canada, or Ireland, and have a 100% approval rate on Prolific with five more studies approved.
In order to assess humans' abilitics the original and variants of digit-matrix problems, collected data from 301 participants. Each participant original digit-matrin problems, problems with alternative blank positions, or problems with symbols in place of digits. The 10 problems given to each participant were sampled from different problem types.
asked lo solve 10 matrix problems, all of which were either
In addition to the problems, participants were also given two altention-check questions at random points during the experiment, with a warning that if the attention checks were failed, then payment ($6 for the experiment, which was expected to take less than 20 minutes) would be withheld. Figure Gelgives an example of attention check. Only one of the 301 participants' submissions rejected due to failed attention checks. As in WHI, as part of the initial instructions participants were given a simple example problem to complete and then were shown the solution.
GPT Study Methods We evaluated the performance of GPT-3.5 (gpt-3.5 (1t--3.5-ttrbb-00633) and GPT-4 (gpt- 4-0613 on the same digit matrices given to humans. At the time of our experiments, GPT-3 was not longer available for testing.
We also evaluated two more recent GPT-4 models, GPT-4-1106 and GPT4-0125, but neither surpassed the accuracy of GPT-4-0613, here we only report results for the 0613 version.
10

---

Published in Transactions on Machine Learning Research (02/2025)
Predemm or
Prodfam 2112
Wyloggantheembeee plice
Try gama the tiing place.
Try gaes the maxinng
(a) Problem with alternative blank position.
(b)
Problem with symbols insted of (c) Example attention check.
digits.
Figure 6: Example items in human study.
Following WHL, all GPT experiments were done with lemperature to zero.
We experimented with different prompt formats, and found that the following gave the best performance for both models:
System: You are genius at solving analogy problems.
User: Try to complete the pattern below. Give ONLY the answer as briefly as possible.
|n|6] [6] [n() [9] [9] [] [8]
We give the other prompts we lested in Appendix A.4.1
3.4 Replication of WHL's Studies
Performance Digit
0.0
Accuracy
0.2
0.0
Logic
Probiem Types
Figure 7: Accuracy across problems with different numbers of rules for human participants on original digit- matrix problems. Data points indicate mean accuracy across all task types for each number of rules and bars indicate 95% binomial confidence intervals. The number of samples for WHL's one- and two-rule problems is 258, and for three- and four-rule problems 430. The number of samples for our data are 306 for one-rule problems, 297 for two-rule problems, 502 for threc-rule problems, and 445 for logic problems. Figure is best viewed in color.
Replication: Human Study Results Figure 7 gives the accuracy on the original digit-matrix task for humans in WHL's study (*WHI.") and replication ("Ours"). The participants in study had higher
11

---

Published in Transactions on Machine Learning Research (02/2025)
performance than those in WHL's study, but generally followed the same trend across problems with different numbers of rules.
GPT Performance Digit Matrices
Accuracy
0.4
GPT-3 WHL
GPT-3355
GPT-4
Two Rules
Three Logic
Figure Accuracy across different numbers of generalizations for GPT models on original digit-matris problems. Data points indicate mean accuracy across all task types for each number of rules and bars indicate 95% binomial confidence intervals. The number of samples for WHL's one-rule problems is 176, for two-rule 240, for three-rule 400 and for logic problems 400. For results GPT-3.5 and GPT-4, the number of samples for one-rule problems is 416, for two-rule 600, for 3-rule 1,000, and for logic 900. Figure is best viewed in color.
Replication: GPT Study Results Figure 3 gives the accuracy of on the original digit-matrix task for GPT models in WHL's study ("GPT-3 WHL") and in study ("GPT-3.5" and "GPT-4"). Our results very closely match those of WHL for problems with each mumber of rules.
3.5 Results on Human and GPT Experiments on Variants of Digit-Matris Problems
Counterfactual Comprehension Check On Alternative Blank Positions For the digit-matrix prob- lems with alternate blank positions, tested GPT-3.5 and GPT-4 to make they comprehended the format of the task. For the simple digit-matrix problem
[2]
[2]
[2]
[5]
[5]
[5]
[6]
[6]
[]
tested each each model nine times, where each (independent, zero-temperature) run the blank was in different position. The prompt we used was as follows:
System: You are genius solving analogy problems.
User: The pattern below is incomplete. What is the position of the missing element? \n[6]
[6] [9] [9] [8]
GPT-3.5 correctly identified the location of the blank in 6 out of 9 cases, and GPT-4 correctly identified the location of the blank in 9 out of 9 cases. This suggests that when asked to complete the pattern, GPT-4 will comprehend what the task is, while GPT-3.5 might have some problems comprchending
Results On the Two Types of Variant Problems Figure shows mean accuracy (across all problem types and numbers of generalizations) for humans, GPT-3.5, and GPT-4 on the original digit matrices and the two types of variations tested. Human accuracy stays roughly constant between the original problems
12

---

Published in Transactions on Machine Learning Research (02/2025)
and both variations. Like humans, the accuracy of the GPT models we tested not sensitive to the symbol variants, but unlike humans, the GPT models' accuracy drops dramatically on the problems with alternate blank positions, showing a lack of robustness to this simple variation on the original task.
Accuray
0.4
0.2
Humans
GPT-4
Digits
Alt. Blank
Symbols
Figure 9: Mean accuracy and 95% binomial confidence intervals averaged over all matrix problem types for humans, GPT-3.5, and GPT-4. Numbers of samples for humans are 1,550 for Digits and 1,000 for each of Alt. Blank and Symbols. Numbers of samples for GPT models are 2916 for Digits, 1,466 for Alt. Blank, and 2100 for Symbols. Figure is best viewed in color.
More detailed results for our variant digit-matrix problems, including results for different numbers of rules are given in Appendix A.5
4 Story Analogies
4.1
Background
WHL tested humans, GPT-3, and GPT-4 on a set of 18 story-analogy problems from Gentner et al. (1993 These problems were designed to test how well humans identify analogies involving causal relations between story elements. For each problem, participant model is presented with a source story (Story 1), paired with two other stories (Story A and Story B), one of which is analogous to the source story, meaning that it has the same causal structure as the source story, although actors, objects, and events may differ. The other story has the same actors, objects, and events as the correct-analogy story, but the causal relations between the story elements differ from those in the source story. In WHL, these were termed, respectively, target story."
"Far analogy- correct target story" and "Far analogy-incoreen
WHL presented humans, GPT-3, and GPT-4 prompts that gave Story Story A, and Story B, and asked, "Which of Story A and Story B is better analogy to Story 1? Is the best answer Story A, Story B, or both are equally analogous?" To mitigate ordering biases, for each of the 18 story problems half of the human participants received prompts in which Story A was the correct answer, and for the other half, Story B was the correct answer. GPT-3 and GPT-4 were tested on two versions of each story problem, with opposite ordering of the correct and incorrect answers.
WHL found that while humans perform better on average than GPT-3 and GPT-4 these "far analogies," " both GPT models perform well above the random-guessing baseline of 50% accuracy.
WHL also performed experiments pairs of stories they called "near analogics," in which the source and target simply paraphrases another with small inconsequential changes (or, as termed GGentner et "literally similar stories"). Here we discuss results only the "far analogy" stories.
13

---

Published in Transactions on Machine Learning Research (02/2025)
4.2 Robustness Tests for Story Analogies
Testing Ordering Biases In reviewing the detailed data collected by WHI., we noticed that their experiments, GPT-4'ss accuracy on the 18 story-analogy problems was biased by the order of the candidate answers: when Story A was the correct answer, GPT-4 was 89% accurate (16/18 correct), but when Story B was correct, GPT-4's accuracy decreased to 61% (11/18 correct) To further test the effects of answer order, we performed experiments similar to those of WHL, testing both human participants and GPT-4 (0613) on the same 18 story-analogy problems using both orderings for candidate answers. A human or machine employing robust analog-maaking abilities should not be affected by the order of candidate answers.
Testing With Paraphrased Stories One potential confounding factor in evaluating GPT models using previously published tests such as Gentner al's story-analogy problems is that these tests are likely to have appeared in the models' pre-training data. If so, it is not clear what effect this would have on the model's accuracy, but it does require caution in interpreting the results.
Another polential confounder is the possibility of "shortcuts" in the text that is, features of the candidate answers that can be used to predict the correct answer without requiring a robust analogy-making making capability. In reading the 18 story-problems, we noticed one possible source of shortcuts: in addition to sharing causal structure with the source story, the correct-answer story often seems more similar to the source story at the sentence level than the incorrect-answer story. For example, Figure 10 shows one example source story (Story 1) compared sentence-by-see se-senencc with the correct target story (Story A); Figure [1 shows the same comparison with the incorrect target story (Story B). Note that Story 1 and Story A share the same number of sentences (except for final irrelevant "distracior" sentence which was appended by Genter et al. to each source story), and corresponding sentences tend to be structurally similar. These superficial similarities are not relevant to the abstract causal analogy that is meant to link the two stories. However, in many of the 18 stories such similarities with the source story are present in the correct answer and largely missing from the incorrect answer.
To test this possible source of bias, for each story problem we wrote a paraphrased version of the correct target story that lacked sentence- by-sentence similarity with the source story. For example, Figure L shows the paraphrased version of Story A from Figure 10 In each case we replaced the original correct target story with the paraphrased version, keeping the original souree story and incorrect target story. We then tested humans and GPT-4 on the story-problerns with paraphrased correct largets A human or machine employing robust analogy-making abilities should not be affected by paraphrasing of candidate answers.
4.3 Methods for Studies on Humans and GPT Models on Story Analogies
Human Study Methods As in our other studies, we collected data on Prolific Academic. Participants were screened to have English as first language, to currently be living in the UK the USA, Australia, New Zealand, Canada, or Ireland, and to have 100% approval rate on Prolific with five more studies approved.
We collected data from 245 participants. Each participant was asked to solve six story-analogy problems, where each problem consisted of one source story and two comparison stories. Figure Balgivves an example of the presentation format. The order of the answers poorect/incorreen were either both in the original format used by WHL, or versions with paraphrased correct target stories. The six problems were randomly selected from the set of 18. Participants were paid $6 for the base task and $1 for each of the problems they answered correctly, giving maximum of $12 payment. The task was expected to take approximatcly 30 minutes.
was randomized. Comparison stories
In addition to the six problems, participants were also given two attention-check questions at random points during the experiment, with a warning that if the attention checks were failed, then payment would be withhold. Figure 13b gives an example of an attention check. Five of the 245 participants' submissions
WWHL did not provide data with respect answer ordering for humans or for GPT-3. 6Te original stories and paraphrased versions are available htttps:///ithub htttpp///ggitub...c//aaattaaaaaa htt.cc//caaaaataaaaaaattaaaaaaaa
14

---

Published in Transactions on Machine Learning Research (02/2025)
Story
Story A
pears sitting in window
girl Cindy records
2 He thought to himself, "These pears seem to be rotten.
"These records look really bad," Cindy thought to herself.
Perhaps get some
my prediction correct."
think buy them so can my right."
4 However, the were too high for Jullus.
the records too expensive for her.
And because was hungry he felt too weak to jump up to them.
After thinking over, she saw there was no way she could afford them.
6 Naturally, this rather disappointing.
Needless say, she became somewhat disappointed.
7 He closed his eyes and went to sleep, dreaming about mountains pears.
Figure 10: Senlence-by-senlence comparison of source and correct target. story, from one of the 18 story- analogy problems.
Story
Story B
1 Jullus was mule who discovered several pears sitting window sill.
girl Cindy found some
wanted at
2 He thought himself, "These seem be rotten.
She really wanted them but, needless to say, they were too expensive for her.
3 Perhaps I'll get some and find out if my prediction correct."
After brooding a while, Cindy got over her disappointment.
4 However, the pears were too high for Julius.
"These records probably boring," she thought.
5 And because he was hungry he felt too weak to jump to them.
"So even If could afford them wouldn't listen to them."
6 Naturally, this was rather disappointing.
7 closed his eyes and sleep, dreaming about mountains pears.
Figure 11: centence-by-sentenn comparison of source and incorrect target story, from the same story-analogy problem as in Figure 10
Story
Story
Jullus mule who discovered several pears sitting in window sill.
girl named Cindy glimpsed some records she was curious about record store.
He thought to himselt, "These seem be rotten.
Cindy didn'tt think the records would be fun to listen wanted purchase them see whether they were or not.
Perhaps ('ll get and find if
correct."
Sadiy, she saw there was na she could afford them.
However, the pears were too high for Julius.
And because was hungry he felt too weak to jump up to them.
Naturally, this was rather d'sappointing.
He closed his and went sleep, dreaming about mountains of pears.
Figure 12: sattenco-by-sentome comparison of source and paraphrased correct target story, from the same
story-analogy problem in Figure TO
15

---

Published in Transactions on Machine Learning Research (02/2025)
rejected due to failed attention checks. As in WHL, part of the initial instructions participants were shown template of the format that stories would be presented in.
Comp
Probiem: of
Problem
Story cotra aithernn through the brush toker some
wep hungryy ie dlosed to eyes and ts deep. dreaming about mocrrans pears.
prodectoo comerc loooover:
hemoelt
work tojinppu thehe Notunally this rother droepparning mang
And
both ofthhe, afher
landed, sranngree moough in the madile of cundior wecking pertined
Cormeguenity they plummeeet strught cown
the woundedd becurs
Stery A gigh named cinyly founetsone recrntts the worned au recort dooe she realy warnedd them neecles to may, they nern too sxperaive for her, After broading for dio, cindly get cant desapoo cysond "These recentss are protuully booing she though: So
sudderly Gone dkad and hin the policeman the side the haad The polloman
Story houng Gore
hore from school whan cemen approuched
warted to bog stra gey. iinane other instructins and respend Story At. untermusaally ancee they ware the read the pa iomman becamm noo dinry
could affond lwoutii lon't
Consequartly they off the and when they
Story ie git sarned cindly gli inpood come recortss the was about record stons City diart the recards would
they were Sodly
wanted purchese way
Stary Young Gene was wai lingt horma from school when matt policeman.
iccenan grabhed drowe avacy with they get
"Stoy Corooppenity
Stary Saady analogy
Xeauye
Iusi iqualty anstcppes
Which
anolngg
(a) Example story-analogy problem (here, Story B was paraphrased from the original cor- rect story.
(b) Example story-analogy problem with atten- tion check.
Figure 13: Example items in our human study on story-analogy problems.
GPT Study Methods Following WHL, we evaluated GPT-4 (0613) on cach of the 18 story-analogy problems twice, once with each answer candidate listed first We did not evaluate GPT-3 as it was no longer available at the time of our experiments.
The prompt we used was as follows:
System: You are helpful assistant.
User: Consider the following story:kn'nnSoww 1: [Text of Story 1]\n),nNw consider two
more stories:) nStoryy A: [Text of Story A]/mntnStory B: [Text of Story B)/nccnWich of Story and Story B is a better analogy to Story 1? Is the best answer Story A, Story B, or both are equally analogous?
4.4 Results of Human and GPT Studies of Ordering Effects
Robustness to Answer Order Table 2 gives the accuracies reported by WHL for GPT-4, along with the accuracies we recorded in our GPT-4 and human studies, for story-problem presentations in which the first or second answer was correct, and over all presentations. In our GPT-4 study, we found a similar ordering bias to that seen in WHL's data: the model is correct more often when the correct answer is given first. In contrast, we see no ordering bias for humans.
Robustness to Paraphrasing Our experiments on stories with the correct answer paraphrased were run exactly as our experiments on the original stories. Table 3 gives the accuracies obtained in our GPT-4 and humans studies for both original and paraphrased cases. Both GPT-4 and human performance decreases on the paraphrased storics, suggesting that the superficial similaritics in the original storics we described above may have contributed to the original accuracies for both GPT-4 and for people. GPT-4's performance decreases more than humans' performance paraphrased stories, but with only 18 stories it is difficult to determine if this effect is statistically significant.
We also evaluated two more recent GPT-4 models, GPT-4-1106 and GPT4-0125, but neither surpassed the accuracy of GPT-4 0613, here we only report results for the 0613 version.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: Accuracy on 18 "far-analogy" story problems. The first / second columns give the accuracy for presentations in which the first/ second answer is the correct one. The last colurn gives the accuracy over all story presentations.
Accuracy: Correct Answer First
Accuracy: Correct Answer Second Accuracy: Total
GPT-4 (WHL)
0.89 (16/18)
0.61(11/18)
0.75 (27/36)
GPT-4 (Ours)
1.0 (18/18)
0.72 (13/18)
0.86 /36)
Humans (Ours)
0.78 (292/373)
0.78 (272/347)
0.78 (564/720)
Table 3: Accuracy on the original 18 "far-analogy" story problems and on stories with the correct answer paraphrased.
Accuracy: Original Stories
Accuracy: Stories with Paraphrasing
GPT-4 (Ours)
0.86 (31/36)
0.72 (26/36)
Humans (Ours)
0.78 (564/720)
0.70 (503/720)
5
Related Work
In the last few years there has been considerable work on evaluating the robustness of reasoning in LLMs, with many studies showing that the performance of LLMs on reasoning tasks decreases, sometimes quite
substantially, when tested on variants of tasks that are likely to differ from those seen in the training data
Dziri al. (2023): Hong et al. (2024); Jiang et al. (2024); McCoy et al. (2024a); Mirzadeh et al. (2024)
Mondorf & Plank 20244: Nezhurina et al. (2024] Prabhakar et al. (2024): Srivastava et al. (2024 Wu ct al. 2023
Yan et al. (024).
Related Work On Letter-String Analogies For studying letter-string analogies, the work most closoly related to the work presented here is that of Hodel & West (2023), who tested GPT-3 with two lypes of variations on the letter-string analogy problems used by WHL: ones that include larger intervals between letters, and ones with randomly permuted alphabets. They found that GPT-3 performed substantially worse than humans on both variations for all but one transformation type. similar set of experiments performed by SStevenson et al. (2024) compared several L.MSs with adults and children letter-string analogies and variations, and also found that LLMs performed poorly compared to humans on the variations. Here we experiment with similar, but more systematic variations, and we compare the performance of different GPT models with that of humans on these variations.
small set of
WHL published a response to Hodel West as well as to earlier work by our team Lewis & Mitchell 2024
arguing that the GPT models tested do indeed have an "emergent capacity for analogical reasoning, and that the poor performance of these models on variant problems could be explained, at least in part, by the models' inability to count, rather than lack of analogical reasoning abilities. In particular, WHL claimed that solving problems with permuted alphabets "require[s] that letters be converted into the corresponding indices in the permuted alphabet, a process that depends on the ability to precisely count the items in a list." WHL tested version of GPT-4 that could generate and execute Python code on variants of letter- string analogy problems, using a randomly shuffled alphabet from Hodel & West's paper. They found that this version of GPT-4 generated code to convert letters to their corresponding numerical indices and then computed differences between indices in order to to solve the problem.
For example, given the "fictional" alphabet x y 1k www.nnjgaahggmmooeeeooopp problem
and the analogy
bfzt-bfzz ; pdic ?
GPT-4 would generate code that would (1) assign each letter its numerical position in the alphabet (b is the 6th letter, f is the 7th letter, etc.), (2) translate the problem to the equivalent numerical one:
5678 -788-5679 21 22 23 24 ?
17

---

Published in Transactions on Machine Learning Research (02/2025)
It would then compute the numerical difference between numbers in the first two sequences, and create a numerical sequence that has the same numerical differences with the target sequence. Finally, would be translated back to letters based on their indices in the alphabet.
Perhaps not surprisingly, on problems with this shuffled alphabet the version of GPT-4 with code generation matched or exceeded human accuracy on problem types on which such familiar numerical pattens and counting abilities were useful, such as "Successor," " "Predecessor," " "Extend sequence", as well as on "Remove redundant letter," " but rode-generation was less successful on "Fix alphabetic sequence" and "Sort" problems; translating into numerical patterns were not as useful in those cases. (WHL did not experiment with GPT-4 code generation on symbol alphabet problems.)
We disagree that the letter-string problems with permuted alphabets (or alphabets with non-numeric sym- bols) "require that letters be converted into the corresponding indices." One doesn't have to compute that, say, b is the 6th letter and p is the 21st letter to solve the problem given above. Rather, one just needs to understand general abstract concepts such as successorship and predecessorship, and what these mean in the context of the given "fictional" alphabet. Indeed, testing this general abstract understanding was the point of creating variants of the original task.
Moreover, this notion of counting violates the spirit of the letter-string domain. As was described in Hofs- tadter & Mitchell (1994), the whole point of the letter-string domain was to get at general mechanisms of analogical reasoning, rather than requiring very domain-specifie reasoning, such as counting up the positions of letters in the alphabet, or subtracting the indices of one letter from another. According to Hofstadter & Mitchell, "[P]roblems should not depend on arithmetical facts about letters, such as the fact that 't' comes exactly eleven letters after 'i', or that 'm' and 'n' flank the midpoint of the alphabet...rithmmtiin operations such addition and multiplication play no role in the [letter-string] domain."
Related Work On Story Analogies For studying story analogies, the work most closely related to our own is that of Sourati et al. (2024), who created a benchmark of over 1,000 story-analogy problems similar to the ones from Gentner et al. (1993), and systematically tested different. types of mappings. They found that humans were substantially more accurate on these problems than any of the LLM models they tested, with a particularly large gap between human and LLM performance on "far analogies."
To our knowledge, ours is the first study to have tested the robustness of WHL's results on digit matrices.
6 Conclusions
The purpose of the work described in this paper was to investigate the robustness of claims of emergent zero-shot analogical reasoning in GPT models Webb et al. (2023). In particular, we evaluated GPT models' performance on three of the four domains used by WHL: letter-string analogies, digit matrices, and story analogies, and tested the robustness of these models and of humans on variants of problems in those domains, ones that required the same abstract analogical reasoning but were unlikely to be similar to those seen in the models' pre-training data. In addition, we tested the effects of answer order for the story analogies.
In the letter-string-annlle domain, we evaluated humans and GPT models on two types of variant prob- lems: ones using permuted alphabets and ones using symbol alphabets. On the simplest problems (zero- generalization) humans' performance remained relatively high across variants, whereas GPT models' perfor- mance decreased, particularly on symbol alphabets. This is in spite of the fact that GPT models seemed to comprehend basic relationships in the variant alphabots, as shown by our countorfactual comprehension checks. On more difficult problems (one to three generalizations), this effect was less prominent as both humans and GPT models performed poorly across original and variant problems.
In the digit-matrix domain we also evaluated humans and GPT models on two types of variant problems: ones in which the answer "blank" position was randomly selected, and ones in which we replaced digits with symbols. For the problems with alternate blank positions, the human participants performance was essentially unchanged from that on the original problems, whereas the performance of the GPT models we tested dropped dramatically. For problems with symbols replacing digits, the performance of both humans and GPT models stayed approximately constant with that on the original problems.
18

---

Published in Transactions on Machine Learning Research (02/2025)
On the story-analogy problems tested by WHL, where the solver was asked not to generate answer but to choose one of three possibilities ("Story A is more analogous"; "Story B is more analogous,"; "They are equally analogous") we evaluated humans and GPT-4 on two dimensions of robustness: answer-ordering effects and paraphrasing effects. We found that while humans are not affected by answer order, GPT-4 seems substantially biased by order. The performance of both humans and GPT models seems to decrease with paraphrasing (in which surface parallels between the original story and correct-answer story are minimized). While our results are suggestive, the small number of stories in this experiment (18) makes it difficult to obtain useful statistics on the size of these effects.
Overall, our results provide evidence that, despite previously reported successes of GPT models on zero-shot analogical reasoning, these models in many cases lack the robustness of zero-shot human analogy-making, exhibiting brittleness on most of the variations and biases we tested.
These results support the conclusions of other work showing that LLMs' performance is better, sometimes dramatically, on versions of reasoning lasks that are likely similar to those seen in training data. While they may have some capability for abstract reasoning, LLMs often lack general, robust reasoning abilities, but instead rely on "narrow, non-transferable procedures for task solving" (Wu et al. 2023). Humans have also been found to perform better on tasks involving familiar content (Lampinen et al. 2024). but, as seen in ours and others' results, are often able to overcome their biases and outperform LLMs on abstract analogical reasoning, perhaps due to their abilities for metacognitive deliberation (Thompson 2009), which are largely lacking in current state-of-the-art Al systems (Johnson et al.] 2024). Our results also illustrate the importance of replicating published claims and carefully evaluating Al systems not only for accuracy but also robustness when testing their cognitive capabilities (Frank 2023
Ivanova 2023).
As we discussed in Section 1 our goal in this paper was to evaluate the robustness of published claims of zero-shot analogical reasoning with simple prompts, so we did not test other prompting formats or models specifically trained for chain-of-thought reasoning, but rather leave such evaluations for future work. That being said, we agree with WHL's assessment that using simple zero-shot prompts is the most appropriate way to evaluate LLMs' abilities for analogical reasoning.
This work does not probe into how either humans or GPT form responses to the problems we used in this ovaluation, or analyze in depth the kinds of errors they make (though Appendix A.3 gives a preliminary analysis of human and GPT-model errors for simple letter-string problems). Future work in this area could be to request that both humans and LLMs give justifications for a particular answer, and to examine how aligned are the kinds of errors made by humans and LLMs.
Acknowledgments This material is based in part upon work supported by the National Science Foundation under Grant No. 2139983. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. This work has also been supported by the Templeton World Charity Foundation, Inc. (funder DOI 501100011730) under the grant https://doi..rg:: 11/1/11..22222200 54224/20650 This research project has benefited from the Microsoft Accelerate Foundation Models Research (AFMR) grant program.
References
Ruben Branco, António Branco, Joao Rodrigues, and Joao Silva. Shortcutted commonsense: Data spuriousness in deep learning of commonsense reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1504 1521, 2021.
Hsiao-Yo Yu Chiang, Jose Camacho-Collane and Zachary Pardos. Understanding the source of semantic regularities in word embeddinggs. In Proceedings of the 24th Conference on Compulational Natural Language Learninng, pp. 119-131,220 2020.
Shir Dekel, Bruce Burns, and Micah Goldwater. Leaping across the mental canyon: Higher-order long-distance analogical retrieval. Journal of Cognitive Psychology, 35(8):856 875, 2023.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Blngavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang, Ren, Allyson. Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality, In Proceedings of the Thirth- Annual Conference on Newral Information Processing Systems (NeurIPS), 2023.
19

---

Published in Transactions on Machine Learning Research (02/2025)
Michael C. Frank. Baby steps in evaluating the capacities of large language models. Nalure Reviews Psychology, 2 (8):451-45) 452, 2023.
John G. Geake and Peter C. Hansen. Functional neural correlates of fluid and crystallized analogizing. Neurolmage, 49(4):3489 3497, 2010.
Dedre Gentner, Mary Jo Rattermann, and Kenneth D. Forbus. The roles of similarity in transfer: Separating retrievability from inferential soundness. Cognitive Psychology, 25(4):524- 575, oct 1993. doi: 00.1006/cogp.199) 1013.
Damian Hodel and Jevin West. Response: Emergent analogical reasoning in large language models. arXiv preprial arXiv:2908. 16118, 2023.
Douglas R. Hofstadter. Metamagical Themas: Questing for the Essence of Mind and Pattern, chapter 24. Basic Books, New York, NY, 1985.
Douglas R. Hofstadter and Melanie Mitchell. The Copycat project: A model of mental fluidity and mmalogy-makingg In Keith J. Holyoak and J. A. Barnden (eds.), Advances in Connectionist and Neural Computation Theory, volume 2, pp. 31 112, Ablex, Norwood, NJ, 1994.
Pengfei Hong, Deepanway Ghosal, Navonil Majumder, Somak Aditya, Rada Mihalcea, and Soujanya Poria. Stuck in the quicksand of numeracy, far from AG1 summit: Evaluating LLMs' mathematical competency through ontology- guided perturbations. arXiv preprint or xxXx:2401.09399 2024.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. arx preprinl 2r11::2212.10003, 2022.
Anna A. Ivanova. Running cognitive evaluations on large language models: The do's and the don'ts. arXiv preprint 231.22212001266, 2023.
Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J. Su, Camillo J. Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. arXiv preprint 2r 11050, 2024.
Samuel G. B. Johnson, Amir-Hossein Karimi, Yoshua Bengio, Nick Chater, Tobias Gerstenberg, Kate Larson, Sydney Levine, Melanie Mitchell, lyad Rahwan, Bernhard Schölkopf, et al. Imagining and building wise machines: The centrality of ai metacognition. arXiv preprint arXin:24 11 11.02478 2024.
Subbarao Kambhampati Can LLMs really reason and plan? Communications of the ACM, 2023. https://cacm acm. wrggblogs/blog-ooooooooooooooo cac.226666 canee---aa--- can-llms-aaee mean-lmmm-rellly-eeaaee reaooly-e mand-paanffullexttt
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in Newral Information Processing Systems, 35:22199 22213, 2022.
Andrew K. Lampinen, Ishita Dasgupta, Stephanic C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. Language models, like humans, show content effects on reasoning tasks. PNAS Nerus, 3(7):pgae233, 2024.
Martha Lewis and Melanie Mitchell. Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. arXiu preprint orXiv:2402 08955, 2024.
Changquan Long, Jing Li, Antao Chen, Jiang Qiu, Jie Chen, and Hong Li. Event-related potential responses to letter-string comparison analogies. Experimental Bruin Research, 233:1563 1573, 2015.
Laura E Matzen, Zachary O Benz, Kevin R. Dixon, Jamie Posey, James K Kroger, and Ann E Speed. Recreat- ing Raven's: Software for systematically generating large numbers of Raven-like matrix problems with normed properties. Behavior Research Methods, (2(2):525 541, 2010.
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew D. Hardy, and Thomas L. Griffiths. Embers of autore- gression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121 121(11) (41))=e232210011) 2024a.
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew D. Hardy, and Thomas L. Griffiths. When language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAl ol. arxio preprint arXio:2400.00 70.01792, 2024b.
20

---

Published in Transactions on Machine Learning Research (02/2025)
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oucel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM- symbolic: Understanding the limitations of mathematical reasoning in large language models. arXio preprinl (xXi::240000229, 2024.
Melanie Mitchell, Khalogy-Makingg As Perception: A Computer Model, chapter 5. MIT Press, Cambridge, MA, 1993.
Philipp Mondorf and Barbara Plank. Beyond accuracy: Evaluating the reasoning behavior of large language models: A survey. arxiv preprint arXim:244400180 2024.
Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, and Jenia Jitsev. Alice in Wonderland: Simple tasks showing complete reasoning breakdown in state-oftthe-at large language models. Xr preprint Xrv::240 02061, 2024.
OpenAl. Learning to reason with LLMs, 2024. http:////pppaaa..c////////////ar httpp//aaaa..oom///eee///eeeeeee
Akshara Prabhakar, Thomas L. Griffiths, and R. Thomas McCoy, Deciphering the factors influencing the efficacy of Chain-of Thoughh: Probability, memorization, and noisy reasoning. orXiv preprint arXiv:2/07.01661 2024.
J. C. Raven. Progressive Matrices Test: perceptual test of intelligence: Individual form. H.K. Lewis & Company, 1938.
Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 240-854, 2022.
Zhivar Sourati, Filip llievski, Pia Sommerauer, and Yifan Jiang. ARN: Analogical reasoning on narratives. Transac- tions of the Association for Computational Lánguistics, 12:1063 1086, 2024.
Saurabh Srivastava, Annarose M. B., Anto P. V., Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, and Sooraj Thomas. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint 2tt::2402.114500 2024.
Claire E. Stevenson, Alexandra Pafford, Han L. J. van der Maas, and Melanie Mitchell. Can large language models generalize analogy solving like people can? orXiv preprint orXiv:2411.02348, 2024.
Saeid Asgari Taghanaki, Aliasgahr Khani, and Amir Khasahmadi. MMLU-Pro+: Evaluating higher-order reasoning and shortcut learning in LLMs. arXin preprint arxin:2409. 09227777.002..... 2024.
Valerie Thompson. Dual process theories: A metacognitive perspective. In J. St. B. T. Evans and K. Frankish (eds.). Two Minds: Dual Processes and Beyond, pp. 171 196. Oxford University press, 2009.
Taylor Webb, Keith J. Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526-115) 1541, 2023.
J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E, H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models. Transuc lions on Mackine Learning Research, 2022a.
J. Wei, X. Wang, D. Schuurmans, Bosma, lchter, F. Xia, Chi, Q Le, and D. Zhou. Chain-of-thougll prompting elicits reasoning in large language models. Advances in Newral Information Processing Systems, 35: 24821 24837, 2022b.
Zhaofeng Wn, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint a724::2307.02477, 2023.
Junbing Yan, Chengyu Wang, Jun Huang, and Wei Zhang. Do large language models understand logic or just mimick context? arXiv preprint Xix::24022 12091, 2024.
21

---

Published in Transactions on Machine Learning Research (02/2025)
A Appendix
A.1
Additional Information on Letter-String Experiments
A.1.1
Letter-String Problems: Counterfactual Comprehension Checks
Here we give details of the results of the counterfactual comprehension checks (CCCs) for letter-string-anllog
problems described in Section 2.5 The prompts for these checks have the following format:
System: You are able to solve simple letter-based problems.
User: Use this fictional alphabet: [a c ...]. \nWhat the next letter after a? n'TThe next letter after a is:
System: You are able to solve simple letter-based problems.
User: Use this fictional alphabet: [a u ...... AnWhat is the letter before c? n'The letter before c
is:
For the non-permuted alphabet, each permuted alphabet an (i), and for two symbol alphabets th1o(1) and (2), we performed the Successor and Predecessor CCCs on each letter (or symbol) as described above. Note that we ran Predecessor CCC' after GPT-3 was no longer available, so we only include results for that model on Successor CCCs,
On the non-permuted (n 0) alphabet, all three models scored 100% accuracy on the Successor test, and both GPT- 3.5 and GPT 4.0 scored 100% accuracy the Predecessor test. Table A gives the accuracy for each model on these tests on permuted alphabets (averaged over alphabets with = 2,5, 5, 10, 20) and symbol alphabets. For permuted alphabets, the average accuracies are given for two cases: (1) the sample letter and its successor (or predecessor) are in their original alphabetic position and (2) the sample letter and/or or its successor (or predecessor) are not their original alphabetic position. For example, for the alphabet in Figure [aa, the letter 'f' and its successor 'g' are in the first case bose-both in their original positions- -eeitions-wwheeas whereas the letter 'f' and its predecessor 'm' are in the second case, since 'm' is not in its original position.
Table 4: (a) Accuracy on "Successor" and "Predecessor" counterfactual comprehension checks for different GPT models averaged across alphabets with different numbers of permuted letters (2, 5, 10, and 20), and across symbol alphabets. The first value in each triplet (e.g., 1.0 in the triplet 1.0/0.82/1.0) is the average accuracy over permuted alphabets in cases in which letters and their successors (or predecessors) were in their original positions; the second value is the average accuracy over permuted alphabets in cases which letters and/or their sucessors (or predecessors) were not in their original positions; and the third value is the average accuracy over the symbol alphabets. Note that we ran Predecessor CCCs after GPT-3 was no longer available, so only include results for that model on Succossor CCCs.
Successor
Predecessor -/-/-
GPT-3
L.0/0.821.00 0.99/0.95/0.94
GPT-3.5
1.0/0.75/0.94 1.0/0.97/1.0
GPT-4
1.0/1.0/1.0
We see that accuracy generally quile high, indicating that the models generally grasp what "successor" and "predecessor" mean in each fictional alphabet. However, GPT-3 and GPT-3.5 have lower accuracies on, respectively, giving the successor predecessor in cases of permuted letters.
A.1.2 Letter-String Prompts
For experiments letter-string analogy problems, experimented with four different prompt formats for GPT models. The prompt format described in Section 2.3 resulted in the highest performance for all GPT models; the results reported in Sections 2.4 and 2.5 used this format. We also tested a prompt similar the instructions given to human participants ("Humanlike Prompt"), minimal version of the human prompt (*Minimal Prompt"), and a
zero-shot chain-of-thouuhtt prompt (Kojima al. 2022 These prompts are given below.
Humanlike Prompt:
System: You able to solve letter-string analogies.
22

---

Published in Transactions on Machine Learning Research (02/2025)
User: In this study, you will be presented with series of patterns involving alphanumeric char- acters, logether with an example alphabet. alphabet./u/uNoot AuluNote that the alphabet may be in an unfamiliar order. \u/nEach paltern will have missing piece marked by [ ]Au)uFor each pattern, you will be asked to guess the missing piece. piece.An)nte nnse the given alphabet when guessing the missing piece.)m piece.nnknoon You do not need include the '[] spaces between letters in your response. An nn b h f d ijj k m parstuuvvvv.
Anlula a a] [b b bjinnuufc [?)
Assistant: h h h
User: In this case, the missing piece after 'a' and 'h' is the letter after 'c'
\n' Note that the given alphabet, 'b' the letter
guess the missing piece. An)n(a e fg hijkk nor s b w x z] Note that the alphabet may be in an unfamiliar order. Complete the pattern using this order. Anaala c d] [a u ellnn/niiikkii
User: Use the following alphabet
Minimal Prompt:
System: You are able to solve letter-string analogies.
User:
Use the following alphabet to complete the pattern./n\ufa e pu
s wywxyy y z] An/nNote that the alphabet may be in an unfamiliar order. Complete the pattern using this order. \n/nfaa e d] [a u e cee)m/nff
Zero-Shot Chain-of Thought Prompt:
System: You are able to solve letter-string analogies.
User: Use this fictional alphabet. An/u(a c d e highiji kl molimmmoopppo w w y \a\nLet's try to complete the pattern. Finish your answer with 'The answer is:" followed by the answer. \n)nfa a nl)nn k [NnLet's think step by step.
A.2 Examples of GPT-4 Errors for Letter-String Problems Using Original Prompt
Table gives examples of errors produced by GPT--4 for gero-generalization problem types, using the prompt described
in Section 2.3
Problem Type
Problem
Correct Answer
Given Answer
Successor
vw w ; aled
alce
a alez hijl
Successor
grst=qiso hijk
hijb
Predecessor
Imno-kmne defg ?
cefg
befg
Predecessor
luvs-wuvs klmn ?
ilmn
w wlmn
Add Letter
ijkr-ijkrm 1pqx-
lpqxs
lpqxst.
Add Letter
defg-defgh hijk
hijkb
hijkl
Fix Alphabet
chijk-thijk. :Tghi +
fghij
Pghij
Fix Alphabet
wjklm=ijklm [gghj
[ghij
bedaf
Remove Redundant
eefghi+efghi nnopqr ?
nopqr
nopqrs
Remove Redundant
e pqqxst=pqxst krmnll ?
krmnl
krinl
Sort
omnlp ? Im
ponml
Sort
qrwut +qrwtu hgfij+
fghij
hgfji
Table 5: Examples of incorrect responses from GPT-4 on zero-generalization letter-string problems. All responses are available at htttps://gitte http:////oo//aaaaaaaooaooaaooaaa
A.2.1 Zero-Shot Chain-of- Thought Prompt Results
We evaluated the performance of GPT'335 S.pt-3.5-turb--1225 and GPT- turbo--ttr)--06113 the generalization problems described in Section 2.2 using the zero-shot chain-of-thoughtt prompt given in Section A..1. All experiments were done with temperature set to and models given 400 tokens complete their response.
the time of writing, 1w--3.5-turb--0611
longer available for direct comparison with previous experiments.
23

---

Published in Transactions on Machine Learning Research (02/2025)
GPT-3.5 Original Prompt
GPT-3.5 Zero Shot
Original
GPT- Zero-Shat CoT
0.8
Accuracy
0.5
Accursa
0.4
0.2
10
20
Letters Permuted
Num, Letters Permuted
(a) Accuracy versus number of letters permuted for GPT-3.5 (0613) with our original prompt, and for GPTT3.5 (0125) for the zero-shot chain-of-thought prompt. Numbers of samples are in Figure zero generalizations.
(b) Accuracy versus number of letters permuted for GPT-4 (0613) with our original prompt and the zero- shot chain-of-thoun thoughtt prompt. Numbers of samples are in Figure 5 zero generalizations.
Figure 14: Comparison of zero-shot chain-of-thought prompting with our original prompt. Figure is best viewed in color.
Figure 1 shows the accuracy of the models given the zero-shot chain-of-thouuht prompt function of the number of letters permuted, compared with the accuracies of the models given the original prompt (Section 2.3 As the figure shows, the zero-shot chain-of-thought prompt results in lower accuracy than our original prompt.
A.2.2 Examples of Zero-Shot Chain- -hain-Of-Thought Reasoning for Letter-String Problems
The following are representative correct and incorrect responses from GPT-4 using zero-shot Coll prompting, as described in Section 1.2
Successor (correct response) j jk m jjkln u
(correct answer: u V W y)
Response: The pattern seems be that the second set of letters is the same the first set, except the last letter is one step further in the alphabet.
So, [ 1 m], the corresponding set k u], where one letter after in in the alphabet.
Following this pattern, for [u x], the corresponding set should be ju w y), where y is one letter after in the alphabet.
The answer is: lu v w y).
Successor (incorrect response) rstu-rse : oppr- (correct answer: o p q s)
Response: In the first pair, the letters consecutive the alphabet: r, 8, t, u. In the second pair, the first three letters are the same, but the fourth letter not the next one in the alphabet, it's instead of u.
The same pattern should be applied the second set. The first three letters consecutive in the alphabet: o, P. g, r. The fourth letter should not be the next one in the alphabet (s), but the letter after that (t).
The answer is: o opqt.
Predecessor (correct response) g ghij-r -iij-fhii nopq
(correct answer: q)
The first group of letters, [g h i]. shifts to [T j] in the second group. This means the first letter in the group moving back by letter the alphabet.
Response:
24

---

Published in Transactions on Machine Learning Research (02/2025)
Applying the same rule the third group, [u q]. shift the first letter back by to get [in p q]-
The answer is: [m o ql. Prodecessor (incorrect response) m -lnop cdef- (correct answer: d
Response: The first group of letters [1 m n o] is shifted two places to the left to form the second group [k m n o).
Following the same pattern, |c should be shifted two places to the left to form (a dJ. The answer is: Ja b dJ.
Add Letter (correct response) defg defgh 1 + (correct answer: j k m n)
The first pair of strings are [d I g] and [d I g h]. The second string is the first string with additional letter at the end. The additional letter the next in the given alphabet after the last letter the first. string.
Response:
So, second pair strings, we have [ k m] and we need to find second string. The last letter the first string of the second pair is 'm'. The next letter in the given alphabet after 'm' is 'n'.
So, the second string in the second pair should be k I m] with an additional at the end, which makes [j k m n].
The answer is: j m n].
Add Letter (incorrect response) W XVWX-+VWXX lmno- ? (correct answer: I n p)
Response: In the first set, the pattern is that each subsequent letter is the next in the alphabet. This pattern is continued the second set, so the next letter after 'o' in the second set 'p:.
However, the second part of the pattern is that the second set of letters is one letter longer than the first. So, in the second set, we need add two more letters after 'p' make one letter longer than the first
So, the next two letters after 'p' in the alphabet are 'q' and 'r!
The is: omoopr.
Fix Alphabet (correct response) opqrh soarh-oop +opqrw wluvk (correct answer: w Luvss
Response pattern here seems that the fifth letter in the first group corresponds to the fifth letter the alphabet. In the first group, 'h' is the fifth letter in the alphabet. In the second group, the fifth letter in the alphabet is 'w!
So, if we apply this to the third group, see that the first four letters are 'w', 4t, 'u', !v! The fifth letter in the alphabet following this sequence is is!.
Therefore, the completed pattern would be:
(o p h] pr r w| [w t [wtuss]
The answer is: (wtuvs]
Fix Alphabet (incorrect response) bjdef-bedef efwhi ? (correct answer: e efg
Response First, let's identify the paltern between the first two sets of letters.
In the first set [b j d e f], the second letter 'j' is 8 places ahead of bb' in the alphabet. The other three letters 'd', fe', If' in alphabetical order.
In the second set [b c e fl, the second letter 'c' is 1 place ahead of bb. The other three letters 'd', 'e', 'f' are in alphabetical order in the first set.
So, the pattern seems to be that the second letter in the set a certain number places abead the first letter, and the last three letters remain the same.
25

---

Published in Transactions on Machine Learning Research (02/2025)
Now, let's apply this pattern to the third set.
The first letter is fel. In the first set, 'J' is places ahead of "b', and in the second set, *c" is place ahead of bb: So, the number of places ahead seems be decreasing by 7. If apply this to 'e', the second letter should be places ahead, which is I
The last three letters should be the in the first two sets, which 'd', 'e', 'f.
The answer is: le lelde f
Remove Redundant (correct response) 1 cemoop-llman VWWXyZ- ? (correct answer: w yxyx)
In the first set, m o [1 n pl, the second sequence is the same the first but without the last repeated letter.
So, if we apply the same pattern the second set (v y z), remove the repeated 'w!
The answer is: [v w y
Remove Redundant (incorrect response) deffgh qrstun ? (correct answer: qr t
The pattern seems to he that the first letter-string is shifted two letters to the right compared to the second one. So, if we apply the same pattern to the third letter-string [q r s u u], we need to shift all the letters two positions the left.
'q' becomes 'o' 'r' becomes 'p' 's' becomes 'q' 't' becomes 'r' 'u' becomes 's' 'u' becomes 's'
The answer is: pr
Sort (correct response) ptrsq pprstt fghji- ? (correct answer: fg i
Responsc: First, we need to determine the pattern in the first pair of letter strings. In the first string (p r q], if we rearrange the letters in alphabetical order, get [p qr.t. So, the pattern seems be rearranging the letters in alphabetical order.
Now, let's apply this pattern to the second pair.
The given string is g jjj.. If rearrange these letters alphabetical order, we get (fghij The answer is: h
Sort (incorrect response)
; dgfeh ? (correct answer: d defgh
Response:
First, let's look at the pattern the first set of letters.
The second set of letters is rearrangement of the first set. The pattern appears to be reversal of the first set of letters, with the first letter moved to the end. So, [b c e) becomes (a c e).
Now, let's apply this pattern to the second set of letters (d g fe h).
If we rearrange this set of letters the same pattern as the first set, we get [g feh
So, the answer is: [g f la d].
A.2.3 Letter-String- Ataae--String-Analgyy Problems: More Detailed Results
Here we give more detailed results human and GPT--model performance on zero-generalization letter-string analo- gies. Figure 15 gives breakdown of performance by problem type and by number of letters permuted.
A.3
Preliminary Error Analysis Letter-String Analogies
A crucial aspect of letter-string analogy problems that they do not necessarily have a "correct" answer, although, mentioned above, humans generally agree on what are the "best" rules describing letter-string transformations
26

---

Published in Transactions on Machine Learning Research (02/2025)
Successor
Predecessor
Add Letter
1.0
0.6
Human
GPT-3
0.2
GPT-3.5
GPT-4
0.0
None
10
20
Symb
None
2
5
10
20
Symb
None
2
10
20
Symb
Remove Redundant
Fix Alphabet
Sort
1.0
0.8
0.4
0.2
Htp
0.0
None
2
10
20
Symb
None
5
10
Symb
None
2
10
Symb
Figure 15: Performance of humans and GPT-models zero-generaalization lasks, broken down by problem type and number of letters permuted. Points are mean accuracy across samples and bars give 95% binomial confidence intervals. Numbers of samples for humans for cach problem type are 46 for zero letters permuted, 56 for 2 letters permuted, 45 for 5 letters permuled, 57 for 10 letters permuted, 64 for 20 letters permuted, and 134 for symbolic alphabets. Samples per point for GPT models are 70 for each problem type and number of letters permuted from to 20, and 40 for each Symb problem type (Successor and Predecessor). Figure is best viewed in color.
in this domain. However, there are other rules that can be inferred from a given pair of letter strings. We therefore examined the "incorrect" answers of humans and of GPT-3 and 4 to ascertain whether the kinds of errors made similar.
For both GPT-3 and GPT-4, we randomly selected five incorrect giving sample of approximately 160 incorrect responses per GPT model. This number can be lower there were fewer than incorrect responses for a problem type and alphabet. For humans, we selected 184 incorrect answers.
from each problem type and alphabet,
Problem Type
Source
Target [e fgg h] h [ [ytusw]
Literal Answer
Explanation
Suce
fg h il f hj] b h il le fghi
[e fffgl fe firkk]
Replace last letter with 'j² Replace first letter with 'c'. Remove first letter of sequence. Swap 3rd and 5th letters.
Fix
Rem
gg g j befed [bdeff]
[ m n
Sort
[ w
Table 6: Examples of literal interpretations of rules found by humans.
By manually examining these selections, we identified four broad categories of errors: 1) Alternate rule formation, where the answer given is consistent with alternative rule. For example, if we have source transformation [a b c la e] with target j I], then according to the Successor rule answer [i mj correct. However, the answer i l e] is consistent with the rule "Replace the last letter with 'e!" 2) Incorrect rule use, in which the answer given clearly related to the target letter string, and some kind of rule has been applied, but the rule is inconsistent with the example pair. For example, for [a b c d] (a b e) with k the response (i k m] is given. which the answer given inconsistent with the expected answer, but related to the largel letter string, We could not discern any clear misunderstauding alternate rule
Wrong, in
For example, for b c [a b with [i 1], the response jj k q] is given. 4) Completely Wrong, in which the answer given inconsistent with the expected answer, and unrelated the target letter string. Again, we could not discern any clear misunderstanding or alternaee rule
27

---

Published in Transactions on Machine Learning Research (02/2025)
For example, for [a b d] [a b with [ j I., the respouse [< y is given. Table 7 gives percenlages for each error lype for humans and for each model. We see that in hummans a large percentage (38.59%) of errors stem from in GPT-4 to lesser extent (22%). but much less in GPT-3 (5.81%). We also see difference in the percentage of incorrect rules applied, with GPT and 4 both having over 30% of errors in this category and humans having around 15% of errors in this category. GPT models also have higher percentage in the Wrong category, and for each of the models this category is the largest across the errors they made. Humans have larger percentage of errors the Completely Wrong category than GPT-3 and 4 however. Across these four broad categories GPT-3 and 4 make different patterns of errors than humans.
using alternate rules. This is also
Table 7: % error types across GPT-3, GPT-4, and Human
Error type
Alt rule
Incorrect Rule
Wrong
Completely Wrong
GPT-3
5.81%
30.97%
55.48%
7.74%
GPT-4
22.00%
32.67%
42.67%
2.67%
Human
38.59%
14.67%
34.24%
12.50%
We can further look at the kinds of alternative rules that are used by humans and by GPT, One key type of alternative rule is where 'literal' interpretation of rule is applied, illustrated in Table As well literal rules, humans found alternative rules for the Fix Alphabet problem lype: they would interpret the changed letter as being moved a certain number of steps in the alphabet, and would "equivalent" means position; sometimes it means the identity of the letter. We find that GPT-4 gives the same kind of literal responses that humans do, but does limited number of errors in this category, and almost all are literal responses to Remove Redundant. In summary, within the " "Alernattive Rule" category, the GPT models found literal rules in the same way humans did, but did not find more inventive alternative rules.
equivalent letter in the prompt the
way. Usually
use alternative rules other than literal responses. GPT-3 has
Breaking down the Incorrect Rule calegory, we see more differences belween lumman and GPT behavior. Human responses in this category mostly where one of the rules has been applied in an incorrect situation, for example Add Letter has been applied instead of Successor. GPT-3 errors include adding two letters instead of one; continuing the alphabet; reversing the target; shifting the target; using an unpermuted alphabet instead of the one given; and repeating the target. GPT-4 made these mistakes and also generated responses that were too long. Very few humans made any of these mistakes. Out of the incorrect responses, the types of response made by humans and GPT models are very different.
A.4
Additional Information on Digit-Matrie Experiments
A.4.1
Digit Matrix Prompts
For our experiments on digit-matrix problems we experimented with several different prompts for GPT models. We found that the prompt described in Section 3.3 performed the best for all models; that the prompt used for the results reported Sections 3.4 and 3.5 The other prompts we tested were as follows:
Alternate Prompt 1
System: You are helpful assistant.
User: [2] [2] [25] [5] [66]
Alternate Prompt 2
System: You are a helpful assistant.
User: Let's try to complete the pattern:(n\n(2) [2] [2]] [5] [5]]][[]] [6]
Alternate Prompt 3
System: You helpful assistant.
User: Try to guess the missing piece. Give ONLY the answer with no explanation (,n) [n]nn[2]
[]]](5] (5) (66] [6]
28

---

Published in Transactions on Machine Learning Research (02/2025)
Zero-Shot Chain-of-Thouue Prompt
System: You are a genius at solving analogy problems.
User: Try to complete the paltern below. Fimish your with 'The answer is:' followed by the answer. (a),([] [2] [2]] [5] [5] [6] [ALLet's think step by step.
Zero-Shot Chain-of- Thoouhh Prompt Alternate Blanks
System: You are genius solving analogy problems.
User: Try to fill the gap in the pattern below. Finish your answer with "The answer is:' followed
by the answer. Anin[2] [2] [[]]55] [5] [5] [6] [6] [T]tt'''ss think step by step.
A.5 Digit Matrix Problems: More Detailed Results
Here we give more detailed results of human and GPT-model performance on our variants on digit-matrix problems. Figure 16bfa) gives the performance of our lumman participants on the original digit-matrix problems (blue points) and on the problems with alternate blank positions (orange points), for problems with different numbers of rules, and logic problems. Human performance does not change significantly when alternate blank positions are used. Figure (bb gives the performance of GPT-3.5 and GPT-4 the original problems (blue and green points) and on problems with alternate blank positions (orange and red ploints). can be seen that the performance of both models drops substantially for alternative blank positions in all cases.
Human Performance Digit Matrices and Alt. Blank Matrices
GPT Parformance Digit Matrices and Alt. Blank Matrices
0.6-
Acourscy
0.6
0.4
0.4
0.2
GPT-3.5 Digis
GP 1-3.5 At.. Blank
Human
Dights
Rule
Two Rules
Three Rules
Logic
One
Two Rules
Three
Problem
Problem
(a) Number of samples for human participants for digit matrices in Figure 7 Number of samples for alternate blank matrices for one-rule problems is 208, for two-rule 182, for 3-rule 299 and for logic 311.
(b) Numbers of samples for digit matrices are as in
Number of samples for alternate blank trices for one-rule problems is 216, for two-rule 300, for 3-rule 500 and for logic 450.
Figure N
Figure 16: Accuracy the original digil-matrix problems and those with and alternative blanks positions across different problem types for human participants and GPT models. Data points indicate mean accuracy across all problem types for each number of rules and bars indicate 95% binomial confidence intervals. Numbers of samples are given in individual captions. Figure is best viewed in color.
Figure a7 gives the performance of our human participants on the original digit-matrix problems (blue points) and ones in which symbols replace numbers (orange points). The performance of humans does not show substantial decrease when digits are replaced by symbols, except in the case of three-rule problems. Figure 17(b) shows a similar
effect for GPT models, except in the of logic problems, in which the performance of the CPT models does show a small but significant decrease when digits are replaced by symbols.
A.5.1 Digit Matrices Zero-Shot Chain-of- Thought Prompt Results
We evaluated the performance of GPT-3.5 (0125) and GPT-4 (0613 on the same digit matrices described in section 3.2 For the digit matrix and symbol matrix experiments we used the Zero-Shot Chain-of-Thuuge Prompt and for the
9GPT-3 was longer available at the time of writing, and similarly 1pt-3.5-turbo-06f was longer available direct. comparison with previous experiments.
29

---

Published in Transactions on Machine Learning Research (02/2025)
Performance Digit Symbol Progession)
Performance Digit Symbol
1,0
1.0
0.8
0.8
0.8
Acoursy
Digit
GFT Symb
Haman Dights (No Progression)
GPT4 Dights
Human Symbols
GPT Symbols
0.0
One Ride
Twn Rulls
Three Roles
Logic
One Rode
Tan Rules
Thine Rules
Logic
Problem Type
Problem type
(a) Number of samples for human participants for digit matrices for one-rule problems is 206, for two- rule 147, for three-rule 202 and for logic 441. Number of samples for symbol matrices for one-rule problems is 205, for two-rule 149, for 3-rule 205 and for logic 441.
(b) Number of samples both digit and symbol matrices for one-rule problems is 400, for two-rule 300, for 3-rule and for logic 900.
Figure 17: Accuracy on the digit matrix and symbol matrix tasks across different numbers of rules, plus logic problems, for human participants and GPT models. Data points indicate mean accuracy across all task types for each number of rules and bars indicate 95% binomial confidence intervals. Numbers of samples are given individual captions. Figure is best viewed in color.
alternate blank experiments, we used Zero-Shot Chain Thought Prompt Alternate Blanks given in Appendix A.4.1 All experiments were done with temperature set and models were given 400 tokens to complete their response.
Results reported in Figures 18 and
19
As the figures show, the zero-shot chain-of-thought prompt results in
lower accuracy than our original prompt.
Digits
Blank
Symbols
1.0
0.8
3.8.
0.8
Accirsey
0.6
3.6-
52
GPT ginaial Prop:
CP7-3.5 Zero-shot Col
0.0
3.0
0.0
Logic
Figure 18: GPT 3.5, zero-shot chain-of-thought results: Accuracy and 95% binomial confidence intervals for the digit matrix task for GPT-3.5 (0125) with the original best performing prompt and zero-shot chain-of- thought prompting. Numbers of samples for either prompt are 2916 for Digits, 1,466 for Alt. Blank, and 2100 for Symbols. Figure is best viewed in color.
30

---

Published in Transactions on Machine Learning Research (02/2025)
Digits
Symbols
1.3
1.0
1.0
Accurscy
0.4
0.2
0.-
02-
CP1-44 ginal Prempi
deroshet
Rule
One
Tao Rujes Three
Logic
One
Teeo Three
Prttlem Tipe
Problem
Probiem
Figure 19: GPT-4, zero-shot chain-of-thought results; Accuracy and 95% binomial confidence intervals for the digit matrix task for GPT-4 (0613) with the original best performing prompt and zero-shot chain-of- thought prompting. Numbers of samples for either prompt are 2916 for Digits, 1,466 for Alt. Blank, and 2100 for Symbols. Figure is best viewed in color.
A.6 Examples of Zero-Shot
Reasoning for Digit Matrix Problems
The following representative correct and incorrect responses from GPT-4 using zero-shot Coll prompting, described in Section A.4.1
Digit Problem (correct response):
[6
9]
[8
9]
[7 9]
[7
4]
[6
4]
[8 4]
[8
8]
[7
8]
[
(correct answer:
Response: In the first pair, the second number remains the same while the first number decreases by 1. the second pair, the second number remains the same while the first number increases by 1.
So, following same pattern, the third pair, the second number should remain same while the number should decrease by 1.
The answer is: [6 8J.
Digit Problem (incorrect response):
[6] [6] [6]
[3]
[3] [3]
[1] [1] []
(correct answer: 1)
Responsc:
The pattern seems to decreasing by half each time. The first row has 6, second row has which
is half of 6. So, the third row should have half of 3, which is 1.5.
The answer is: [1.5]
Digit Problem (incorrect response):
[1
5]
[3
8]
[6
0]
[5
8]
[7
0]
[
]
(correct answer:
31

---

Published in Transactions on Machine Learning Research (02/2025)
Response: First, let's look al the first numbers in each pair. They increase by each time: 3, 5, 5, 7, 5, 7, and so on. So, the first number in the next pair should be
Next, let's look at the second numbers in each pair. They follow pattern of 5, 0, 5, 8, 0, and So, the second number in the next pair should be 5.
The is: [7 5]
Alt. Blank Problem (correct response):
6]
]
13
71
[2
1]
(2
6]
[6
71
[3
1]
(correct answer:
First, let's look at the paltern in the first position of each bracket. In the first row, it goes from 3 to 6. In the third row, goes from to 6 to It seems like the numbers are rolating in a cycle. we follow this pattern, the first number in the second row should be 6.
Response:
Now, let's look at the pattern third row, it goes from 6 to 7 to 1 again. The pattern is the same in both rows. If we follow this pattern, the second number in the second row should
second position of each bracket. the finst row, it goes from
the
The answer is: [6 6]
Alt Blank Problem (incorrect responsc):
[0] [9]
[9] [4]
[0]
[4] [0] [9]
(correct answer: 4)
First, let's look at the pattern in the first two numbers of each row. In the first row, we from to 9, which is increase of 9. In the second row, we go from to which is decrease of 5. In the third row, go from 4 to 0, which is decrease of 4.
Response:
Now, let's look the pattern in the last number of each row. In the first row, we don't know the last number yet. In the second row, the last number is O. In the third row, the last number 9.
So, the pattern seems be that the last number of each
the first number of the next
Therefore, the last number of the first row should be the first number of the second row, which is 9.
The is: 9.
Symbol Problem (correct response):
[< [~ X]
[~
[ 0 [<
[<
+]
2~ +] ]
(correct answer: & *)
Response: In the first set, the pattern moves from '&' to '<' to Sw) In the second set, the pattern moves from "J' to & to '<. So, in the third set, the pattern should move from '<? for to '8c..
The answer is: & *1
Symbol Problem (incorrect response):
[$]
[1]
[~]
32

---

Published in Transactions on Machine Learning Research (02/2025)
[$] [1] [~] [$] [I] []
(correct answer: )
First, we can see that the pattern repeats every three symbols. The first symbol is always [8]. the second symbol always [!], and the third symbol alternates between [~] and blank space.
Response:
So, following this pattern, the third symbol in the third sequence should be a blank space.
The is: [8] [!]
33


Reviews:
[{"review_id": "1Oc2gnqdYa", "reviewer": "Reviewer_piBS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThis paper studies an interesting question that has been the source of a lot of debate in the community: to what extent can LLMs reason, and how does their behavior compare to human reasoning. The focus on testing multiple variants of the same task for robustness is laudable. The authors also did several \"sanity-checks\" in each case, testing whether the LLMs understood the task even from the simple prompts that were given. I appreciate the reproduction of Webb et al. results with GPT-3 as a starting point, before moving on to evaluation on the new tasks. Generally, I think the paper is well-written and has a strong motivation, chooses interesting tasks and will provide a good starting point (tasks, human experiment results) for further work on these questions.\n\n# Weaknesses\n\nMy main worry is about the methodology used to evaluate the LLMs. The authors main result is that the findings from Webb et al. break down once we look at simple variants of their task. From what I understand, the authors focus on what prior work calls \"Direct Prediction\", where the LLM outputs its answer without \"thinking step-by-step\" first. I believe the experiments in the paper do show clearly that this is not robust to intuitive variations of the original tasks. But I don't think that this is enough to say that the models cannot reason robustly, since this is just one way of trying to probe for this ability, and it's not the standard way anymore with modern models (chain-of-thought is). So, my main methodological criticism is that the authors should test their claims with zero-shot chain-of-thought instead: if they still hold, I believe the paper's argument will be much strengthened. I'll elaborate on the argument for why below.\n\nIn the introduction, Direct Prediction (DP) is motivated by the quote from WHL \"Of particular interest is the ability of these models to reason about novel problems zero-shot, without any direct training\". However, zero-shot chain-of-thought (prompting the model to reason first, without any task-specific guidance of examples) would still count as \"zero-shot, no direct training\", and it has been shown to elicit reasoning much more effectively than direct prediction. While it's essentially impossible to fully match the methodology between the human experiment and the LLM evaluation, DP seems to be more akin to giving participants a fixed, short time limit on each task, since the LLM can only do one forward pass to compute the answer token. CoT allows the model to think harder in harder instances, or instances that contradict the model's priors more (such as when changing the alphabet, or applying any other of the task generalizations that the authors test). Analogously, humans generally have higher response times in these instances - I don't know if the authors measured this, which could also be interesting to look at.\n\nSo to summarize this, I think that results on CoT would be important to make the claim that \"these models often lack the zero-shot robustness of human analogy-making\", which is the main stated claim of the paper. At the very least, you need to show in some tasks that CoT does not perform much different from DP in analogy making (which would be surprising but interesting, given the body of prior work on CoT observing otherwise) in order to mainly rely on the DP results to state this conclusion.\n\nThe paper also has some inconsistencies in which models are used in each experiment, due to text-davinci-003 having been phased out in early 2024, and some experiments were ran before that and some after (as clarified in Appendix A - though I was confused while reading the paper). It would be thus be impossible to run new CoT experiments on GPT-3 (this is not a fault of this paper, but is an issue in the general field of studying proprietary models in a way that makes replicating results later on impossible once the models are taken down).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper evaluates three models of OpenAI's GPT family (GPT-3, GPT-3.5 Turbo and GPT-4) on a range of analogical reasoning tasks, with the goal of evaluating the robustness of findings from the Webb et al. paper \"\"Emergent analogical reasoning in large language models.\". This previous paper had reported that GPT-3 generally matched our surpassed human participants in these abstract analogical reasoning tasks. This paper takes 3 such classes of tasks from Webb et al: letter-string analogies (completing sequences of letters following a pattern), digit matrices (completing a 2D matrix) and story analogies (associate similar causal relations between story elements). These tasks are augmented with several variants in order to probe the robustness of the models' reasoning capabilities. The findings can be summarized as: the paper largely replicates results with GPT-3 reported in Webb et al, but these results change substantially with the simple variants of the original tasks (e.g., using permuted alphabets), with LLM performance dropping more often (showing a lack of robustness) compared to human performance, which was generally higher. This was not the case in the story analogies, however, where human performance also dropped similarly after paraphrasing (but GPT-4 was more sensitive to answer order).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* [minor, typo] intro: \"and and variants\"\n* [minor, typo] paragraph title: \"Results Oon the Two Types of Variant Problems\"\n* critical: please report experiments with CoT\n* Please also add a few examples of raw model responses in the appendix, especially showing common error types. I appreciated the preliminary error analysis in A5, but think it would be enriched by having a few concrete examples besides just the statistics and descriptions of the errors.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6qUxbevQYW", "reviewer": "Reviewer_SPSG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Important replication study showing that GPT performance on cognitive tests may not be robust to perturbations which humans are insensitive to, such as changing the underlying alphabet in character level tasks.\n2. Shows claims of superhuman analogical reasoning from a highly cited recent paper are questionable\n3. Significant effort into replicating human studies, which proves fruitful as human performance is higher than the original study.\n\n**Weaknesses**\n\nA Only GPT-3, 3.5, 4 are studied. This leads to the following weaknesses:\n\n1. Reproducibility may not be possible: The exact versions/dates of the models are not provided, so it is hard to reproduce these results as models keep changing, and also become unavailable after some time. This already affects their own study, \"GPT-3 was not available for our experiments on symbol alphabets with one generalization\".\n\n2. Claims about \"LLMs\" cannot be made: In numerous occasions in the abstract / intro / conclusion, the paper generalises observations about GPT models studied to LLMs as a whole. This generalization cannot be made without studying other LLMs from other providers, including open-source ones. \n\nB Concerns about experiment design:\n\n1. Task descriptions for variants of letter-string analogies are unclear and written in abstract terms.\n2. For the digit matrix task, putting the blank in other positions than the last one might make it harder for the model to identify what blank has to be filled. Humans are provided a clean 3x3 grid with a blank, whereas models are provided a linear string, which may not be a fair comparison and contribute to the drop in performance. Providing few shot examples can help fix this. \n3. More generally, it is not clear why models are not allowed to use chain-of-thought, or few-shot-examples to understand the task format. Even humans are allowed some time to think and respond, and it's unfair to compare models given only 1 token of compute. Access to more inference-time compute can be essential to make the model expressive enough to solve certain kinds of reasoning tasks [1]. I understand this sticks to the WHL setup, but since the point of this paper is to argue that setup was flawed, it's worth also looking at ways in which model performance is underreported in that setup, not just overrated for an unbiased evaluation.\n\n**References**\n\n[1] Pfau, Jacob, William Merrill, and Samuel R. Bowman. \"Let's Think Dot by Dot: Hidden Computation in Transformer Language Models.\" arXiv preprint arXiv:2404.15758 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper replicates and re-analyzes the robustness of Webb et al. (2023)'s claims of emergent analogy-making in GPT models. Their human study replication finds higher human performance in the same tasks as Webb et al. For the letter-string analogy task, GPT performance drops drastically when a different alphabet (permuted or symbols) is used, whereas human performance stays similar. For the digit matrix task, GPT performance drops significantly when the blank position is changed, but changing the symbols doesn't affect it, whereas human performance stays similar for both. For the story analogy task, a paraphrased version of the stories is designed which leads to drops in both human and GPT performance, but no statistically significant comparisons can be made.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please add results on recent open-weight models like Llama 3.1 - 3.3, OLMO-2, Qwen 2.5 etc. and other API providers like Gemini Flash 2, Claude 3.5 Sonnet so that general claims about LLMs can be substantiated. This would also make this paper much more interesting to the audience if recent models are still struggling at these analogical reasoning tasks, confirming this testing didn't stop being interesting a year ago.  This would fix weaknesses A1, A2.  \n\n2. Please add examples to demonstrate the types of variants designed for the letter string analogy task. Also breakdown the aggregate results across the different variant types for a better understanding of whether certain types are leading to the performance drops. This would fix weakness B1.\n\n3. Please also add results on few-shot examples, systematic prompting (using DSPy), and chain of thought to accurately report model performance. This would address weakness B2, B3.\n\n4. Finally, it's unclear why human performance is higher in your replication than WHL's study. Could you add a discussion on the potential reasons for this?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t2DjOsdaQJ", "reviewer": "Reviewer_nvRX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is generally well-written, with concepts, background, and setups clearly introduced. It is easy to understand and follow. The results are reasonable, and the analysis of the experimental results is mostly well-grounded, with sufficient discussion of variations in human and GPT results. While the related work is not extremely comprehensive, it is adequate to support the claims made in the paper.\n\nRegarding weaknesses, the main issue lies in the somewhat unsurprising conclusion. Many earlier works have reached the same conclusion using similar approaches (some of which are cited, others are not, though this does not significantly impact the paper). I have personally published and reviewed several similar works. Therefore, the conclusion may not be particularly exciting to the community, even though the evaluation method could be of interest to some. However, since the evaluation process is rigorous and reasonable, I don't think this should be a factor in rejecting the article, in accordance with the TMLR reviewing guidelines.\n\nThere are minor concerns and suggestions, which are detailed in the \"Requested Changes\" section below.\n\nIn general, I am inclined to accept this article.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article explores the abstract reasoning abilities of LLMs through tasks adapted from Webb et al. (2023), including letter-string analogies, digit matrices, and story analogies. The findings reveal that, despite prior claims of LLMs excelling in zero-shot analogical reasoning, they fall short of the robustness demonstrated by humans in zero-shot analogy-making and show vulnerability to task variations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- On page 7, above Figure 3, the authors mention that \"GPT-3.5 and GPT-4 show slightly lower accuracy than GPT-3, possibly due to their fine-tuning beyond a strict prompt-completion objective.\" Does this imply that the results are considered incorrect because the GPT answers are unparsable? Could the authors provide statistics on this issue, such as the ratio of unparsable results, and include case studies to illustrate the problem?\n\n- It is observed that the results are sometimes presented as figures and other times as tables, even though they appear to convey the same information. Why not use a consistent format for all results?\n\n- I suggest using different markers in the figures to represent different models, as the current color distinctions are hard to differentiate in black-and-white prints.\n\n- It would be interesting to see how up-to-date open-source models such as LLaMA 3.2 or Gemma 2 perform on these tasks. I would anticipate that they perform similarly to the models already discussed, but presenting actual results would be beneficial. Additionally, since these models have both text-completion and instruction-tuned versions, the former might not encounter the issue mentioned in the first point.\n\n- According to the template, the appendix should be put after the references.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors requested an extension to provide their rebuttal, which the AE approved. However, the system still requires the reviewers to make their recommendations earlier than the rebuttals were uploaded.
The AE carefully reads all reviewers' comments, the revision, and the authors' rebuttal. The AE believes that the main concerns (e.g. CoT prompting experiments, clarifications on the use of the zero-shot setup or GPT models) have been addressed by the rebuttal, and therefore is supportive of the submission to be accepted by TMLR.
Decision: Accept (poster)