Decoding Stumpers: Large Language Models vs. Human Problem-Solvers
Authors: Alon Goldstein
Miriam Havin
Roi Reichart
Ariel Goldstein
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
Problem-solving abilities
Stumpers
Cognitive abilities
Human performance
Riddles
TL;DR: This study examines Large Language Models' problem-solving capabilities on stumpers, revealing that they outperform humans at solving but lag behind in verifying solutions

Abstract: This paper investigates the problem-solving capabilities of Large Language Models (LLMs) by evaluating their performance on stumpers, unique single-step intuition problems that pose challenges for human solvers but are easily verifiable. We compare the performance of four state-of-the-art LLMs (Davinci-2, Davinci-3, GPT-3.5-Turbo, GPT-4) to human participants. Our findings reveal that the new-generation LLMs excel in solving stumpers and surpass human performance. However, humans exhibit superior skills in verifying solutions to the same problems. This research enhances our understanding of LLMs' cognitive abilities and provides insights for enhancing their problem-solving potential across various domains.

Decoding Stumpers: Large Language Models vs. Human Problem-Solvers
Alon Goldstein'
Miriam Havin²
Roi Reichart
Ariel Goldstein 1245
Corresponding Author: alon @ @xoltar.com
Xoltar Inc
2 Department of Cognitive and Brain Sciences, Hebrew University, Jerusalem, Israel Faculty of Data and Decision Sciences, Technion 4 The Hebrew University Business School, Jerusalem, Israel
5 Google Research
Abstract
cognitive literature System 2 capabilities (Matsuo, 2020; Kojima et al., 2022). The dual-system model of the mind has arguably been the most prevalent model of thought and behavior in psychology, eco- nomies and social science in general (Goldstein and Young, 2022; Evans and Stanovich, 2013; Chaiken and Trope, 1999; Gawronski and Creighton, 2013) especially in addressing systematic limitations of cognitive and artificial systems. In simple terms, System / is associated with effortless, associative processes and is often thought of compatible with neural nets implementation, while System 2 related effortful, serial, and often symbolic processes (Frankish, 2010; Evans, 2003). A mous example where System I's heuristic hinders a solution described in Box 1.
This paper investigates the problem-solving capabilities of Large Language Models (LLMs) by evaluating their performance on stumpers, unique single-step intuition problems that pose challenges for human solvers are easily verifiable. compare the performance of four state-of-the-art LLMs (Davinci-2, Davinci-3, GPT- 3.5- Turbo, GPT-4) to human participants. Our findings reveal that the sew-generation LLMs excel in solving stumpers and surpass human performance. However, humans exhibit superior skills in verifying solutions the same problems. This research enhances our derstanding LLMs' cognitive abilities and provides insights for enhancing their groblem-solvigg potential across various domains'
1 Introduction
Box 1: The bat and the ball
Since their inception, Large Language Models (LLMs) have astonished the scientific community with their ability to tackle complex tasks (Rad- ford et al., 2019; Brown et al., 2020; Devlin et al.. 2018). These emerging capabilities, along with shared principles with human cognition and the brain, have motivated significant efforts to utilize deep language models and, recently, LLMs for ex- plaining neural activity (Tikochinski et al., 2023; Goldstein et al., 2022b,a; Schwartz et al., 2019). predicting human behavior (Goldstein et al., 2022b; Brand et al., 2023), and even providing a theo- retical framework for the human mind (Richards al., 2019; Hasson al., 2020). Recent advance- ments. particularly the ability of LLMs perform tasks requiring different skills such as mathemat- ical calculations, analytical reasoning and use of world knowledge, have led several papers to de- clare that LLMs possess what is termed in the
"A bat and ball cost 1.10 dollars total. The bat costs dollar more than the ball. How much does the ball cost?"
The immediate but incorrect response is to assume that the ball costs 10 cents. How- ever, symbolic-serial approach formula- tion of the problem "r + (x: 1) 1.10" yields the correct solution of 0.05 dollars.
While this type of questions (Cognitive Reflec- tive Test; CRT) are considered hard to solve, as they tend to elicit wrong responses (Frederick, 2005; Toplak al., 2011), people can be primed to solve them correctly by insisting a formalist approach (i.e., applying System instead of System 1: (Alter et al., 2007)). In contrast, problems that require insight (i.e., have neither an intuitive/associative so- lution nor a symbolic one) are hard for humans and often elicit no response (i.e., humans are "stuck"; Bar-Hillel et al., 2018; Bir-Hillel, 2021)).
'The data available at httpps///git thub. wwwo.coo/lllll com/Alon-Go/ Stumpers LLMs.

---

Consider, for example, the riddle in Box 2:
stumper, by asking GPT-3.5-Turbo to change the names and wording of the original data-set. After the generation, we manually approved or edited each stumper to reduce confusion and alternative solutions as much as we could. This process re- sulted in a set of additional 152 stumpers. As the set of new stumpers was not validated with human participants and may differ from the original set, we present the results for the original set in the body of this paper and the detailed results in the appendix. The data also includes correct and incorrect solu- tions, which allowed a comparative analysis of the accuracy and reasoning strategies of the responses given by models and human participants. A dataset sample can be found in Appendix A.
Box 2: Blood relatives
"Alex is Bobbie's s blood relative, and Bob- bie is Charlie's blood relative, but Alex is not a blood relative of Charlie. How come?".
Answer: Alex and Charlie can be related to Bobbie from different sides of the family, e.g., they could be his parents, uncles, etc.
This riddle typically challenges human intuition (System 1) because humans tend consider Alex, Bobbie, and Charlic blood relatives. However, a symbolic solution (System 2) is typically also not available to humans who try to solve it, as there is no clear algorithm to follow to reach a solution. Facing this question, humans seem to be anchored (or stuck) in the framing according to which the three men are blood relatives and cannot escape it to generate alternative framing of the problem that would yield effective explanations of the situation (Bar-Hille Hillel, 2021).
Models and Experiments
The study involved four language models: Davinei- 2, Davinci-3, GPT-3.5-turbo, and GPT-4. Addition- ally, 81 human participants (F=48%; ages 20-54, m=28.52, sd=7.73) were recruited via an online survey participation platform (Prolific). When solving each stumper, both humans and mod- els were presented with a prompt. To normalize the answers across conditions, models, and par- ticipants, all prompts started with a standardized definition correct answer to a riddle:
The above-mentioned question is an example of a stumper. A stumper is a one-step intuition riddle, the solution to which is typically so elusive that it does not come to mind, at least initially leaving the responder stumped. Stumpers do not fall within the System 1 or System 2 frameworks but are re- lated to creative thinking (Bar-Hillel et al., 2019). Importantly, once presented with a solution, people can easily classify it as right or wrong a simple system-2 2 task. In this paper, we demonstrate that recent LLMs (e.g., GPT-3.5 and GPT-4) outper- form humans in solving stumpers but lag behind humans in classifying solutions as right wrong.
An answer to a riddle is correct only if it is consistent with all the riddle's clues, sensical, specific, logical, and fitting with the context of the riddle.
See prompts examples in Appendix C.
3.1 Answer Generation
2 Task
To avoid learning, each participant was presented with only one stumper and was asked to answer it or type "IDK" if they did not know the answer. Each model, in each prompt, was presented either with only one stumper ("naïve response") or with two other pairs of riddles and their ground-truth answer ("prompted response").
A stumper is single-step intuition riddle in which misleading cue sparks visual semantic repre- sentation that blocks the solution from coming to mind. Unlike other riddles, there no need for fur- ther computation or thinking, and once the obstacle is removed, the answer is clear. See examples in Appendix A.
3.2 Answer Verification
The dataset used for analysis consists of all 76 stumpers curated in (Bar-Hillel, 2021). Each stumper is a textual description of a narrative or scenario that requires a unique solution. To en- hance the number of stumpers beyond this exhaus- tive list, we generated two similar riddles for each
After answering the riddle or typing "IDK", human participants were presented with the two possible solutions and were asked to choose the correct one. The models were presented with the same choice without their previous response in the prompt.

---

Answer Generation
3.3 Answer Verification Modell' response To further compare, the models were given verifi- cation problem where their own answers replaced one of the answers. For riddles to which the model knew (/did not know) the answer, their response used instead of the correct (/incorrect) ground truth.
0.6
Natve
Pronae
0.51
Peccent Carrect
0.3
0.4
0.33
0.32
4 Results and Observations
11.3
0.228
0.26
We replaced participants who reported knowing their riddle finding the answer online. Two aul- thors evaluated the responses unanimously, with only one response being disagreed upon, leading to its exclusion.
1:2
Davind1-2
Davinel
G-I-3--Tunb
GPT4
Figure performance of the different models. Human performance presented in the dashed line.
LLMs proficiency at solving stumpers Our re- sults are provided in Figure 1. Solving a stumper by chance has virtually zero probability, given the infinitesimal likelihood of randomly arriving the correct solution among countless potential answers. Human participants in our sample have accurately solved 15% of the stumpers, replicating the 35% accuracy found (Bar-Hillel, 2021).
correct answer from the ground-truth. The models did not show different performances (F(3,3)=2.43, p=0.24], nor did the models' previous success in solving the stumper [F(1, 3) 5.25, p 0.106]. For the Model's responses (Figure 3), three- way ANOVA was conducted, with model type, the correctness of the response, and the prompt type. Here, too, no effects were found for the model type [F(3,10)11228 p>0.33] or the prompt 10(1, p>0.7]. The model's previous success has significantly improved performance [F(1,10)=24.97, p=0.0005].
Improved performance of the advanced models A two-way ANOVA was conducted to examine the effects of the models and the prompting: The chat models (GPT-3.5- Turbo and GPT-4; m=57.8% for the original stumper, 43.4% for the enhanced data- set) have performed significantly better than the GPT-3 model (Davinci-2 and Davinci-3; m=29.6%% for the original stumper, 22.5% for the enhanced data-set) (F(1,4)=686, p=0.00001]. The main effect Prompt was not significant [F(1, 4) = 0.023, p 0.887J. but the interaction was [F(11, 4) = 30.82, = 0.0055, indicating that the prompt has positive impact on the performance of the GPT-3 models and negative impact the performance of the chat models. See results for the original data-set in figure 1. See results for the enhanced set in Appendix
Answer Verification; Ground Truth
1 Succeeded failed
11
0.8
Percent Correct
0.64
0.6
0.38
0.35
0.410.41
0.390.39
0.22
0.2
Answer Verification The answer verification task was tested once with the ground-truth answers (Figure 2) and once with the model's responses vs. ground truth (Figure 3). Different scores are reported for correct and incorrect responses.
Davine-22
3
GPT-3 5. Turbo
GPT4
Homans
Figure Accuracy of the models and humans choos- ing the correct answer out of two alternatives.
While humans performed perfectly verifica- tion when knowing generate the answer (100%) and above chance even when they failed generate (63.8%) most models performed below the chance level (m=41% %). two-way ANOVA was con- ducted to compare the models' ability to choose the
Discussion and Conclusions
This study compared the stumper-solving abilitics of LLMs and humans. We found that while the

---

Answer Verification: Model's response
swers. Humans, however, demonstrated higher proficiency in recognizing the correct answer, even when they were unable to solve the problem ini- tially (Figure 2). This suggests that humans possess a verification capability, considered a System 2 pro- cess, which has not yet fully emerged in LLMs. Interestingly, the Davinci-3 model showed good performance in recognizing correct answers cu- rated by the authors (70%; Figure 2) and by itself (85 ;:; Figure 3).
I Succeeded, Naive I
failed, Naive
I Suceecded, Prongt 1 fasled, Prompt
0.85
Percent Correct
0.8
0.75
0.7
0.06
0.6
0.52
0.54
0.48
0.47
0.4
0.38
0..1
0.29
0.3
0.24
1.2
20
0.2
The overall pattern of results suggests that GPT- 3 aligns better with human capabilities, as its answer-verificatoon capabilities are better than its solving capabilities. This pattern stands contrast to GPT-3.5 and GPT-4, which solve stumpers better than they verify their solutions. This finding indi- cates that for disciplines interested in using LLMs to model human behavior or cognition, Davinci-3 is likely a more suitable model to employ. This is in line with (Hagendorff and Fabi, 2023), which shows how GPT-3 (but not GPT-4 and GPT-3.5) exhibits similar biases and errors as demonstrated by humans. Another reason to consider Davinci-3 over GPT-4 and GPT-3.5 in modeling human be- havior is the fact that the results acquired here, as well as the psychological literature, suggest that it is easier for humans classify correct response than generates it (Pintrich, 2002), a pattern of re- sult similar to Davinei-3 and not congruent with GPT-4 and GPT-3.5 performance. This closely related to the literature showing that recognition is considered easier than recall, as the former requires only identifying the presence of familiar informa- tion, whereas the latter demands retrieving specific information from memory without external cues (Roediger III and Karpicke, 2006).
Davinek
Davinet-
GPT-35-Turbo
GPT-
Figure 3: models' accuracy in choosing a solution be- tween their previous response and the ground truth.
LLMs are better than humans at solving stumpers, their answer-verification abilities are limited and fall far from human performance. These findings provide valuable insights into the capabilities and limitations of LLMs, their relationship with human cognition, and the potential for utilizing LLMs as a framework for cognitive capacities.
The results showed that L.LMs, specifically the LLMs used in this study, demonstrated proficiency in solving stumpers that are obstructed by mislead- ing representations. The models correctly solved 26%-58% of the stumpers, outperforming human participants and surpassing the chance level (Fig- ure 1). This suggests that LLMs possess the skills required for solving these types of questions.
The study revealed an improvement in perfor- mance for more advanced models. The chat mod- els, GPT-3.5-Turbo and GPT-4, which are fine- tuned with human feedback during training, outper- formed the GPT-3 models (Davinci-2 and Davinci- 3). This indicates that advancements in model training contribute to enhanced problem-solving abilities. Prompting the models with additional pairs of riddles and their ground-truth answers had a positive impact on the performance of the GPT- 3 models, further emphasizing the importance of context and prior knowledge in solving stumpers.
The challenge of answer verification is closely related to the problem of text classification, which has been found be challenging for LLMs (Sun et al., 2023). There is a significant discrepancy between the abilities to generate a correct answer and to classify a correct response. This has im- portant implications for estimating LLM capabili- ties, as many NLP benchmarks are designed based on the model's 's ability to classify correct answers (Rajpurkar et al., 2016; Wang et al., 2018; Da- gan et al., 2005: Reddy et al., 2019; Clark et al., 2019). One possible implication is the necessity of including interaction- based measures (Collins et al., 2023), based on continuous human-LLM in- teraction, when evaluating LLMs. Like in oral ex-
Despite their ability to generate correct answers, the models fell short in the task of answer verifi- cation compared human participants (Figure 2). To further stress this problem, we have asked the models to compare their correct responses against a false response (Figure 3, full bars), demonstrat- ing their inconsistency and inability to verify an-

---

ams, the opportunity to react to the models' output in tailored follow-up questions allows the evalua- tor deeper probing into the models' capabilities (Gharibyan, 2005; Davis and Karunathilake, 2005). Furthermore, the findings from this study can inform the development of benchmark tasks for evaluating the intelligence and human-like behav- ior of LLMs. Stumpers provide a challenging do- main that tests problem-solving, associative capac- itics, and verification skills. By designing more comprehensive benchmarks and evaluating LLMs' performance on these tasks, we can gain a better understanding of their cognitive capabilities and identify areas for improvement.
References
Adam L Alter. Daniel M Oppenheimer. Nicholas Ep- ley, and Rebecca N Eyre. 2007. Overcoming in- tuition: metacognitive difficulty activates analytic reasoning. Journal of experimental psychology: Gen- eral, 136(4):569.
Maya Bar-Hillel. 2021. Stumpers: an annotated com- pendium. Thinking & Reasoning, 2664):33--566
Maya Bar-Hillel, Tom Noah, and Shane Frederick. 2018. Learning psychology from riddles: The case of stumpers. Judgment and Decision Making. 13(1):1
Maya Hillel, Tom Noah, and Shane Frederick. 2019. Solving stumpers, crt crat: Are the abilities re- lated? Judgment and Decision Making, 14(5):620~ 623.
In conclusion, this study investigated the abil- ity of large language models (LLMs) to solve stumpers, challenging riddles characterized by elu- sive solutions. Our findings demonstrate that LLMs, especially the advanced models GPT-3.5- Turbo and GPT-4, exhibit a remarkable proficiency in solving stumpers, surpassing human perfor- mance in some cases. These results highlight the potential of LLMs as powerful problem-solvine tools and provide insights into the cognitive pro- cesses involved in solving complex puzzles. Our analysis also uncovered that the human ability to verify solutions has not been fully developed yet in LLMs. Future research can build upon these findings to explore the role of context, expand the variety of stumpers, and investigate the generaliz- ability of LLMs in different domains, contributing to developing more robust and human-like artificial intelligence.
James Brand, Ayelet Israeli, and Donald Ngwe. 2023. Using gpt for market research. Available at SSRN 4395751.
Tom Brown, Bemjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Shelly Chaiken and Yaacov Trope. 1999, Dual-process theories in social psychology. Guilford Press.
Christopher Clark, Kenton Lee, Ming-Wo Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL. pages 1922-29332
Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong. Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, et al. 2023. Evaluating language models for mathematics through interactions. arXiv preprint arXiv:2306.016 01694.
6 Limitations
Our study on stumpers and large language models (LLMs) has several limitations to consider. Firstly, the limited number of 76 validated stumper, even with two more versions of each in the enhanced dataset, potentially restricting the representative- ness and generalizability of our findings. Sec- ondly, we focused exclusively on OpenAl models, limiting the scope of comparison with other lan- guage models. Thirdly, subjective judgment was involved in evaluating correctness, leading to po- tential variations in interpretations. Lastly, prompt engineering techniques were underutilized, poten- tially limiting the models' problem-solvin poten- tial. Future research should address these limita- tions for more robust and comprehensive insights into LLMs' problem-solving abilities.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges. Evalu- ating Predictive Uncertainty, Visual Object Classifi- cation, And Recognising Textual Entailment, pages 177-190.
Margery H Davis and Indika Karunathilake. 2005. The place of the oral examination in today's 's assessment systems. Medical teacher, 2744):294-297.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and
Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint a.0.v:1810.04805.
Jonathan St BT Evans. 2003. In two minds: dual- process accounts of reasoning. Trends cognitive sciences, 7 (10):454-459

---

Jonathan BT Evans and Keith E Stanovich. 2013. Duull-pocess, theories of higher cognition: Advanc- ing the debate. Perspectives on psychological sci- ence, 8(3):223-241.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 05250.
Keith Frankish. 2010.
Dual-process and dual- system theories reasoning. Philosophy Compass, 5(10):914-92) 926.
Siva Reddy, Danqi Chen, and Christopher Manning. 2019. Coqa: A conversational question answering challenge. Trunsactions of the Association for Com- putational Linguistics, 7:249 266.
Shane Frederick. 2005. Cognitive reflection and de- cision making, Journal of Economic perspectives, 1944
Blake Richards, Timothy Lillicrapp, Philippe Beau- doin, Yoshua Bengio, Rafal Bogacz, Amelia Chris- tensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et 2019. A deep learning framework for neuroscience. Nature neuro- science, 22 1:71111770
Bertram Gawronski and Laura A Creighton. 2013. Dual process theories 14, The Oxford handbook of social cognition, page 282.
Hasmik Gharibyan. 2005. Assessing students' knowl- edge: oral exams vs. written tests. ACM SIGCSE Bulletin, 37(3):143-147,
Henry L Roediger III and Jeffrey D Karpicke. 2006. Test-enhanced learning: Taking memory tests im- proves long-term retention. Psychological science, 17(3):249-2555
Alon Goldstein and Benjamin Young. 2022. The unconscious mind. In Mind, Cognition, and Neuro- science, pages 143. Routledge.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. 2019. Inducing brain-relevant bias natural language pro- cessing models. Advances in neural information pro- cessing systems, 32.
Ariel Goldstein, Avigail Dabush, Bobbi Aubrey, Mari- Schain, Samuel A Nastase, Zaid Zada, Eric Ham, Zhuoqiao Hong, Amir Feder, Harshvardhan Gazula, et al. 2022a. Brain embeddings with shared geome- try artificial contextual embeddings, a code for representing language the human brain. BioRxiv, pages 2022-03.
Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shang- wei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text classification via large language models. arXiv preprint 205.072305.08377.
Refael Tikochinski, Ariel Goldstein, Yaara Yeshurun, Uri Hasson, and Roi Reichart. 2023. Perspective changes in human listeners are aligned with the con- textual transformation of the word embedding space. Cerebral Cortex, page bhad082.
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nas- tase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. 2022b. Shared computational principles for language processing in humans and deep language models. Na- ture neuroscience, 25(3):36--380.
Maggie E Toplak, Richard F West, and Keith E Stanovich. 2011. The cognitive reflection test as predictor of performance on andd tasks, Memory & cognition. 39(7):125--1289.
Thilo Hagendorff' and Sarah Fabi. 2023. Human-lite like intuitive behavior and reasoning biases emerged in language models-and disappeared in gpt-4. arXiv
preprint arXiv:2306.07760
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding.
Uri Hasson, Samuel A Nastase, and Ariel Goldstein. 2020. Direct fit to nature: An evolutionary perspec- tive on biological and artificial neural networks. Neu- ron, 105 434.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205.11916
Yutaka Matsuo. 2020. Special features of deep learning and symbol emergence, New Generation Computing, 38:5-6.
Paul R Pintrich. 2002. The role of metacognitive knowl- edge in learning, teaching. and assessing. Theory into practice, 41 41(4):219-225.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 18))9

---

Appendix
B Enhanced data-set results
A Stumpers examples
Figure Perfomrnace scores the different models on the enhanced data-set. The dashed line indicates human performance in the original set.
1 A father and son were involved in traffic accident. The father was killed, and the son rushed to hospital.
Answer Verification
The surgeon walked into the operating room, and upon sceing the severely wounded buy cried out: "OMG, is my son!".
0.8
Succeeded
0.72
folled
0.65
0.67
How could this be true?
0.6
Percent Correct
0.51
0.51
0.52
0.5
Answer: The surgeon is the boy's mother
0.4
A very tall man was holding up a wine decanter way above his head.
2
0..1
0.2
0.12
He let go of it, and it dropped to the carpet he was standing on. Explain briefly how not a single drop of wine was spilled.
Davinel-2
Davinei-3
GPT-35 5-
GPT-4
Origmal Datnset
Answer: The decanter was empty
Figure 5: Accuracy of the models in choosing the cor- rect answer out of two alternatives.
Farmer Joe eats two fresh eggs from his own farm for breakfast every day. Yet there are no chickens his farm. Where does Farmer Joe get his eggs?
3
Answer: Famer Joe does not eat chicken eggs, but different animal's egg, such as ducks.
4
Marcy went from one bank of river to the one 20 meters across.
There bridges the river.
Marcy had no equipment, no devices, no special clothing, and she can't even swim.
She relied on her own body only -ndd none of got wet!
Explain briefly how she managed this.
Answer:
The river was dry.

---

C Prompt examples
Answer Generation, Prompt, GPT-3 models answer to riddle is correct only if it consistent with all the riddle's s clues, sensical, specific, logical, and fitting with the context of the riddle.
Answer Generation, Naive, GPT-3 models An answer to a riddle is correct only if it is consistent with all the riddle's clues, sensical, specific, logical, and fitting with the context of the riddle.
Riddle:
Long after the screen of Kim's smart phone had cracked. It was still functioning just fine. Before he could replace it, the phone accidentally fell into the family's s swimming pool. It was retrieved almost at once, but alas - the phone was dead. Yet water had penetrated the cracked screen, so all the critical components remained completely dry. Explain briefly why the phone was dead.
Riddle:
A father and son were involved in a traffic accident. The father was killed, and the son was rushed to hospital. The surgeon walked into the operating room, and upon seeing the severely wounded boy cried out: "OMG, it is my son!". How could this be true?
Answer:
Answer:
the pool was empty
Riddle:
Polly bought beautiful parrot. The seller guaranteed that the bird repeats everything it hears. However, try as Polly might to teach it, her squawking parrot never repeated single word. The seller did not lie. Explain briefly.
Answer: The parrot was deaf
Riddle:
An accountant says: "That attorney is my brother", and that is true they really do have the same parents. Yet the attorney denies having any brothers and that is also true! How is that possible?
Answer:

---

Answer Generation, naive, Chat models modell: 3.5-turbo',
Answer Generation, prompt, Chat models { ('model': 'gpt-4',
messages':
'messages' [
role' 'system',
('role': 'system',
'content': "An answer to a riddle is correct only if it is consistent with all the riddle's clues, sensical, specific, logical, and fitting with the context of the riddle.", 'role': 'user', 'content' 'Riddle: \nIwo Italians are sharing a pizza. The older Italian is the brother of the younger Italian. But the younger Italian is not the brother of the older Italian. Explain briefly.
'content'; "An answer to riddle is correct only if it is consistent with all the riddle's clues, sensical, specific, logical, and fitting with the context the riddle."), ('role': 'user', 'cotent': 'Riddle: AnCindy recycles everything: paper, glass, metal, plastic, etc. She also brings her still useable stuff (books, housewares, etc. to the donation bins at the recycle center. Recently, she brought bag full of large (2 liter) bottles to the recycling center. The volunteer on duty could barely lift it. Explain briefly.
'role' 'assistant',
ccottnnt: 'Answer: An H,
temperature': 0.0, frequency_penalty" 1.0,
"presence_penalty" 0.5, 'n': 1, max max_tokens': 120)
\n\nAnswer: bottles were full. They were donations to the plant. \nholmm=\nniddde: bought used car from his neighbor next door. The neighbor claimed that the car got 35 miles per gallon. Fred, an excellent driver, could only get about half that much, in spite of driving the same roads as his neighbor. Explain in few words. in'nAnswer:Aofhe seller was -yinginllnlnn-biRiddedeel Russians were standing in line. The taller one was the brother of the shorter one, but the shorter one was not the brother of the taller one. Explain in few words how that is possible. "), ('role': 'assistant', 'content': 'Answer:'n' An' },, 'temperature'' 0.0, prequercyypenaltyy': 1.0, prrssence_eenatty' 0.5, 'n': 1, 120)

---

Answer Verification, GPT-3 models answer to a riddle is correct only if it is consistent with all the riddle's s clues, be sensical, specific, logical, and fitting with the context of the riddle.
Answer Verification, Chat models
('model': "gpt 3.5-turb'',
messages': [
{ rorl'' 'system',
'content'; "An answer to riddle is correct only if it is consistent with all the riddle's clues, be sensical, specific, logical, and fitting with the context the riddle."), ('role': 'user', content'; "Riddle: \nAlex is Bobbie's s blood relative, and Bobbie is Charlie's s blood relative, but Alex is not blood relative of Charlie. How come? \n\nAnswers: \n1. Alex and Charlie could be Bobby's parents, making them both Bobby's s blood relatives but not each other's sn 2.Alex is Bobbie's s parent, and Bobbie is Charlie's parent, but Alex is not a parent of Charlie. \n\nWhich of these answers is correct?"), ('role' 'assistant', ccontent: 'The correct answer is number '11, "temperature': 0.0, frequency_penalltyy: 1.0, "presence_penalty': 0.5, 'n': 1, maxx_toeens'; 20)
Riddle:
Farmer Joe eats two fresh eggs from his own farm for breakfast every day. Yet there are no chickens on his farm. Where does Farmer Joe get his eggs?
Answers:
1. Famer Joe do not eat chicken eggs, but a different animal's egg, such as ducks.
2. Farmer Joe gets his eggs from the grocery store.
Which of these answers is correct?


Reviews:
[{"review_id": "BQe6VkK7d4", "reviewer": "Reviewer_2ch8", "summary": "", "questions": "A. How would you define a stumper problem? How did you choose 76 stumper problems? What procedure did you use to choose the stumper problems?\nB. Unlike other problems from professional exams such as LSAT bar test and SAT, stumper problems are hard to be standardized. Is there any strategy that you can standardize the stumper problems? Also, is there any way to make sure the abilities of the human solvers are similar? \nC. In your study, human subjects are involved. Please, provide IRB information.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful evaluation of our paper and your valuable comments. Your insights have provided us with an opportunity to address critical concerns and enhance the robustness and credibility of our research. We are committed to addressing each of your points and ensuring that the revised paper is more comprehensive and well-grounded.\n\nReasons to Reject:\nYour concerns about the sample size and the potential difficulty in generalizing our results are duly noted. We would like to point out that we have used an exhaustive list of all available stumpers (found in Bar Hillel 2021).\nThe problem with human’s lack of consistency is a general issue  in the psychological literature. In all behavioral experiments we treat two sources of variances (or inconsistencies), individual differences (the real difference between different participants) and noise (the fact that the participants are not necessarily consistent with themselves). In order to establish the reliability of our results we compared the human performance we acquired with the results previously attained. We mention it in the text stating: “All models correctly solved more than 25% of the stumpers, compared to human accuracy of 35% in (Bar-Hillel, 2021),  and 38.15% in our sample.” [142-144]. This shows in the group level the human performance is relatively stable.  We will note that our results replicate previous research and fall within the lines of human behavior experimental design.\n\nQuestions for the Authors:\nA. How would you define a stumper problem? How did you choose 76 stumper problems? What procedure did you use to choose the stumper problems?\n\nWe appreciate the reviewer's request for a clearer explanation of stumpers and their selection in our study. “A stumper is a riddle, the solution to which is typically so elusive that it does not come to mind, at least initially - leaving the responder stumped… [stumpers] work by eliciting a representation of the situation described in the narrative, which then blocks the solution ” (Bar Hillel, 2021). In other words, stumpers are a specific type of riddle that poses two key characteristics. Firstly, they present a notable challenge, often leaving individuals stuck and unable to find any solution, even if incorrect. Secondly, their solutions are remarkably straightforward - once revealed, they are immediately recognized by the solver. These unique attributes make stumpers an intriguing subject of study as they reveal the capacity of humans and language models to not merely interpret language but to reinterpret it in a way that resolves a problem. We will certainly provide the above mentioned definition in the revised paper. The 76 stumper problems used in our study were a comprehensive list taken from Bar-Hillel’s (2021) paper listing all researched verbal stumpers. \n\nB. Unlike other problems from professional exams such as LSAT bar test and SAT, stumper problems are hard to be standardized. Is there any strategy that you can standardize the stumper problems? Also, is there any way to make sure the abilities of the human solvers are similar?\n\nStandardizing stumper problems indeed presents challenges due to their propensity to induce cognitive blocks. To achieve a level of standardization, we provided a well-defined criterion for what constitutes a correct response to a stumper (lines 109-112: “An answer to a riddle is correct only if it is consistent with all the riddle’s clues, sensical, specific, logical, and fitting with the context of the riddle”). This criterion was communicated to both the models and to the human participants and the judges of the answers, ensuring a common basis for evaluation. Concerning the abilities of human solvers, while it is challenging to ensure complete similarity, we judged each response by the same definition, allowing multiple different responses to be correct. The stumpers were scored using 2 independent judges with above 95% agreement, ensuring the scoring is as standardized as human behavior can be.\n\nC. In your study, human subjects are involved. Please, provide IRB information.\n\nWe highly value the ethical aspects of our study involving human participants. Our research was conducted with strict adherence to ethical guidelines, and since no personal-identifiable information was collected, nor any harm was done to participants it was eligible for Institutional Review Board (IRB) exemption. Throughout the study, we upheld the principles of participant privacy, informed consent, and ethical integrity. Given the opportunity, we will address this issue in the extra page allowed in the final version.\n\nOnce again, we genuinely appreciate your feedback, which has been instrumental in identifying areas for improvement. Your insights will contribute significantly to the enhancement of our paper, and we are dedicated to incorporating your suggestions to produce a more comprehensive and credible final version.\n\n\nBest regards,\nSubmission668 authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This short paper contributes to the NLP community by adding new stumper datasets and by comparing the performances of human solver and LLMs empirically. ", "reasons_to_reject": "However, this short paper may have too small samples to generalize the results. Additionally, unlike LLMs, human solvers are hard to be similar. There should be a way to control human solvers’ abilities. ", "questions_to_authors": "A. How would you define a stumper problem? How did you choose 76 stumper problems? What procedure did you use to choose the stumper problems?\nB. Unlike other problems from professional exams such as LSAT bar test and SAT, stumper problems are hard to be standardized. Is there any strategy that you can standardize the stumper problems? Also, is there any way to make sure the abilities of the human solvers are similar? \nC. In your study, human subjects are involved. Please, provide IRB information.\n", "paper_topic_and_main_contributions": "This short paper entitled “Decoding Stumpers: Large Language Models vs. Human Problem Solvers” has a potential to improve the ability of LLMs and can contribute to NLP engineering as well as new dataset addition. Identifying a stumper itself is already a contribution. Additionally, the authors found out that the current LLMs are excellent solvers for stumpers in comparison with human solvers. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "IRB information is required."}, "extra_text": {}}, {"review_id": "zq5V9oc8uJ", "reviewer": "Reviewer_zMZm", "summary": "", "questions": "Thanks for your hard work. To make it clear what you're saying in this paper, please answer these three questions.\n\nQuestion A. I think the paper does not clearly states what a stumper is. Though I understood it as a riddle that has some tricky things that makes people fail to answer it, but it seems yet vague to me. Could you give me an operational or formal definition of it?\n\nQuestion B. Why do you use the notion of System 1 and 2 (I assume that you're inspired by Daniel Kahneman)? The relationship between these two systems and your stumper task is not clearly stated in the paper.\n\nThese are questions about some logical gaps in the discussion.\n\nQuestion C. To clearly state one condition is better than the other after ANOVA, we need to run post-hoc tests. The ANOVA just says that the distribution of two conditions are differ (not the direction of which thing is larger). Could you report the corresponding results?\n\nQuestion D. I think two statements lines 071-073 and lines 222-224 are in conflict. Here, you said that stumpers both do not fall within two systems but related to system 2. Please correct my view if I'm wrong.\n\nQuestion E. I think the reported result are irrelevant to lines 266-273. I cannot see any relationship between \"interaction-based measures\" and generation / verification procedure. Could you explain further?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer zMZm,\n\nWe sincerely appreciate your thoughtful and detailed review of our paper. Your feedback is invaluable in enhancing the quality and clarity of our work. We have carefully considered each of your comments and suggestions, and we would like to address them individually.\n\nFirst, we would like to acknowledge your concern about the paper's organization. We apologize for any confusion caused by the current structure and assure you that we will reorganize the paper to provide a clearer narrative flow. Your feedback underscores the importance of delivering our methodology, findings, and discussions in a well-structured manner that facilitates a cohesive understanding of our research.\n\nAs for the dataset size, we would like to clarify that the 76 stumpers used in our study represent an exhaustive list taken from the Bar-Hillel 2021 paper, the definitive paper about stumpers. While we acknowledge the limitation of the dataset size, the generation of new stumpers is a complex and non-trivial task, a challenge that many psychologists have grappled with. Also, such newly created stumpers would, by definition, have a different status than the original 76. We hope that if the NLP community becomes interested in this type of stimuli it  will encourage a multidisciplinary effort to curate a bigger dataset.  \n\nQuestion A: We appreciate the reviewer's request for a clearer explanation of stumpers and their selection in our study. “A stumper is a riddle, the solution to which is typically so elusive that it does not come to mind, at least initially - leaving the responder stumped… [stumpers] work by eliciting a representation of the situation described in the narrative, which then blocks the solution ” (Bar Hillel, 2021). In other words, stumpers are a specific type of riddle that poses two key characteristics. Firstly, they present a notable challenge, often leaving individuals stuck and unable to find any solution, even if incorrect. Secondly, their solutions are remarkably straightforward - once revealed, they are immediately recognized by the solver.  These unique attributes make stumpers an intriguing subject of study as they reveal the capacity of humans and language models to not merely interpret language but to reinterpret it in a way that resolves a problem. We will certainly provide a more explicit explanation of this definition in the revised paper.\n\nQuestion B: Your observation about the relationship between System 1 and System 2 and the stumper task is accurate. We recognize that we didn't adequately establish the connection between the systems and our hypotheses within the paper. The dual-system model of the mind has been the most prevalent model of thought and behavior in psychology, economics and social science in general (Goldstein & Young, 2022; Sherman, Gawronski & Trope, 2014), especially in addressing systematic limitations of cognitive and artificial systems (Kahneman, 2013).\nOur main point in mentioning the dual systems model is that the stumper task represents a distinctive challenge that doesn't fit neatly into the System 1 and 2 dichotomy. Particularly, stumpers are a good example of a failure of both systems, with System 1 failing to provide a swift response, even if incorrect, and System 2 lacking a clear procedure for a solution. To tackle stumpers, a third process emerges, involving reinterpreting the problem's description to enable a solution. This unique contribution of the stumper dataset showcases the limitations of the existing framework, which is why we chose to focus on it. We will address this motivation more comprehensively in the final version, given the opportunity.\n\nQuestion C: We apologize for not including post-hoc test results after ANOVA. We will ensure that the corresponding post-hoc test outcomes are reported in the final version of the paper.\n\nQuestion D: Your observation regarding the apparent conflict between lines 071-073 and lines 222-224 is astute. In lines 071-073, we highlight that stumpers do not neatly fall within either System 1 or System 2, as we further elaborate in response to question B. On the other hand, lines 222-224 pertain to the recognition of potential solutions by humans and not to generating them. This distinction implies that while generating responses may not align with System 1 or 2, recognizing solutions could be associated with System 2-like processes. We will clarify this distinction in the revised manuscript. Thanks very much for highlighting this.\n\nQuestion E: In the paper, we demonstrate that models can provide suitable answers but often struggle in making accurate judgments, while humans excel at recognizing solutions. Lines 266-273 hint at the potential for interaction-based measures to improve this situation.\n\nWe greatly appreciate your suggestions regarding missing references, typos, grammar, style, and presentation improvements. We will certainly incorporate your recommendations to enhance the clarity and coherence of our paper. Moreover, we will address the issues you highlighted in the Appendix and strive to eliminate any equivocations or repetitions.\n\nOnce again, thank you for your diligent review. Your insights have significantly contributed to refining our work, and we are committed to addressing all the points you've raised to enhance the quality and rigor of our paper.\n\nBest regards,\nSubmission668 Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Propose a new way to analyze LLM's linguistic or common-sense-related ability.\n- Demonstration of such ways and identifying some limitations of current LLMs", "reasons_to_reject": "- The paper is not well written: the authors should re-organize the paper to clearly deliver what they did. There are several equivocations about stumpers or riddles, which prohibits understanding of this paper.\n- The datasize of 76 is somewhat small, though the dataset is designed for testing purposes and I know that the authors already stated this point. To reflect diverse linguistic variations or situations, I think the task requires at least 200 stumpers.\n- The discussion has some logical gaps. Please refer to the questions sections for the logical gaps.", "questions_to_authors": "Thanks for your hard work. To make it clear what you're saying in this paper, please answer these three questions.\n\nQuestion A. I think the paper does not clearly states what a stumper is. Though I understood it as a riddle that has some tricky things that makes people fail to answer it, but it seems yet vague to me. Could you give me an operational or formal definition of it?\n\nQuestion B. Why do you use the notion of System 1 and 2 (I assume that you're inspired by Daniel Kahneman)? The relationship between these two systems and your stumper task is not clearly stated in the paper.\n\nThese are questions about some logical gaps in the discussion.\n\nQuestion C. To clearly state one condition is better than the other after ANOVA, we need to run post-hoc tests. The ANOVA just says that the distribution of two conditions are differ (not the direction of which thing is larger). Could you report the corresponding results?\n\nQuestion D. I think two statements lines 071-073 and lines 222-224 are in conflict. Here, you said that stumpers both do not fall within two systems but related to system 2. Please correct my view if I'm wrong.\n\nQuestion E. I think the reported result are irrelevant to lines 266-273. I cannot see any relationship between \"interaction-based measures\" and generation / verification procedure. Could you explain further?", "paper_topic_and_main_contributions": "This paper proposes a new task, solving stumpers. Here, the authors say that stumpers are sort of riddle that has misleading cue which sparks a certain visual or semantic representation. This paper proposed a new dataset with 76 problems and evaluated performance of LLMs on solving such stumpers. As a result, this paper suggested that LLMs exhibits better ability of solving stumpers but worse ability of identifying right answers for the stumpers compared to human.\n\nMain contributions:\n- Proposing a new task, solving stumpers and provide a small dataset to test LLMs\n- Demonstrate two ways to analyze LLM's performance on stumpers: Answer Generation and Answer Verification.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I think you should cite Daniel Kahneman's book \"Thinking, Fast and Slow\" or equivalent paper where two systems come from."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Some of the required things to understand this paper stated in the Appendix. Please state some example classes of stumpers in Section 2 and (if possible) provide a short-version of prompt in Section 3.\n- Distinction between Section 3.2 and 3.3 makes me confused. Isn't 3.3 are subset of 3.2?\n- There are several equivocations and repetitions in the paper, such as 'stumper'. Please adjust them."}}, {"review_id": "aJn4ixGiOV", "reviewer": "Reviewer_WvWv", "summary": "", "questions": "A. What is the evaluation metric for the experiments? The authors mentioned 'two authors evaluated the responses unanimously' (line 138), but how? and based on what metric? The stumpers seem to be open-ended and can have many different possible answers. \n\nB. No details of the dataset. It is possible (some/all) the models have already seen these stumper questions in their training corpus, and the models remember the answers instead of performing 'thinking'. Can authors provide justifications?\n\nC. In section 3.2, it seems the authors just changed the original open-ended QA task into a binary-choice task. Why the change of task format becomes answer verification? \n\nD. In section 3.2, the authors mentioned 'humans were presented with the two possible solutions'. Where do these alternative solutions come from? These alternatives may be easy to distinguish and are not suitable for evaluation. \n\nE. In section 3.3, why the design of the answer verification task for the models is different from the one for humans? Please provide justifications. \n\nF. In Figure 2, Davinci-3's success rate is abnormally high compared with the other models and its answer generation performance in Figure 1. Did the authors further analyze the reasons? If so, please provide justifications. (This question is more related to the concern of novelties, it seems the authors just 'presented' the results without actually analyzing the experiment scientifically)\n\nG. For asking each human subject to answer one single question in the dataset, the authors mentioned 'to avoid learning' in line 114. It seems unreasonable. Are the authors suggesting humans can learn how to answer new stumpers by seeing any other stumpers? If so, this hurts the claim that the task need 'creative thinking' and 'hard for humans to solve' (line 75). Please provide justifications for this design. \n\nH. The results in Figures 2&3 show Davinci-3 outperforms GPT-3.5 and GPT-4 in the binary-choice task, but the authors claim GPT-3.5 and GPT-4 'outperformed the GPT-3 models' (line 204). This is contradictory. Please provide justifications.\n\nI. The authors claimed that few-shot prompting improves GPT-3's performance, thus justifying 'the importance of context and prior knowledge in solving stumpers'. However, few-shot prompting leads to a performance drop on GTP-3.5 and GPT-4, which is contradictory. Please provide justifications.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer WvWv,\n\nWe sincerely appreciate your thorough review of our paper and your insightful feedback. Your detailed observations and questions provide us with valuable opportunities to improve the clarity, rigor, and depth of our research. We are committed to addressing each of your concerns to enhance the quality and contribution of our work.\n\nReasons to Reject:\n\n(1). The novelty in this paper does not meet a top-venue standard.\n\nWe understand your perspective regarding the novelty of our paper. While the core of our work lies in the evaluation of Large Language Models (LLMs) on stumpers, we recognize that the contribution might appear limited. However, we believe the unique nature of stumpers and the insights we draw from LLMs' performance offer valuable implications for understanding cognitive processes and the capabilities of LLMs. We will reemphasize these points to ensure the significance of our work is better articulated.\n\n(2). The experiments are not well-designed and organized.\n\nWe apologize for the lack of details in our paper regarding prompt design and dataset. Prompt design is indeed crucial for obtaining reliable results from LLMs. We will provide a more comprehensive explanation of the prompt design and justification for our choice in the final version, given the opportunity. The prompts used in this study were the same instructions given to the human participants. Other, less intuitive prompts, were tested after the original one, and yielded similar results (including the use of chain-of-thought). Due to the lack of space in this short paper, and since prompt engineering is not the focus of this project, we did not provide further details about how we built the prompts, and we will amend it in the final version.\n \nAs for the dataset, we acknowledge that more information is required to assess its quality and validity. In the extra page of the final version, we will include details about how the dataset was created and the process of collecting ground-truth annotations. Furthermore, we will make the entire dataset, including scoring, publicly available to address transparency concerns.\nWe also appreciate your concern about the human study design. The constraint of asking each participant to solve only one question was imposed in order to mitigate the potential bias from shared mechanisms among stumpers. Moreover, past studies in the psychological literature have always limited each participant to one stumper for this reason, and we naturally wanted to be consistent with the literature. We acknowledge the potential reliability issues you've raised and will consider this aspect more carefully in our future research.\n\nAs for the answer verification task, we apologize for the misuse of terminology. You are correct that a verification task should involve reasoning to verify the correctness of provided answers. We will correct this and ensure that our terminology aligns with the intended task. We are currently considering using “recognition” or “comparison”, and would greatly appreciate your feedback on it.\n\n(3). The paper structure is not well-organized.\n\nYour feedback on the paper's structure is duly noted. We agree that the current organization may lead to confusion and hinder the paper's readability. We will streamline the structure, ensuring that relevant content is allocated the appropriate space while maintaining a cohesive narrative that highlights the novelty of our findings. Specifically, we will add our reasoning of using the dual-systems model, as prevalent in social-sciences when modeling thinking process,  in the introduction while keeping it as short as possible, limit our discussion, and extend the details provided for the experiment details and discoveries.\n\nQuestions for the Authors:\n\nA. What is the evaluation metric for the experiments?\n\nThe evaluation metric for our experiments involves comparing the responses generated by LLMs and human participants to the ground-truth answers and computing the fraction of correct responses. In the case of answer generation, a \"correct response\" is defined as any answer that adheres to the given criteria for correctness stated in the paper (lines 109-112 “An answer to a riddle is correct only if it is consistent with all the riddle’s clues, sensical, specific, logical, and fitting with the context of the riddle”). In the answer verification task (that will be renamed), participants and models are presented with pairs of possible answers, one correct and one incorrect. The performance metric is based on their ability to choose the correct answer from these options.\n\nB. No details of the dataset.\n\nWe apologize for the lack of details about the dataset. As previously mentioned, the dataset is an exhaustive list of stumpers curated by Bar-Hillel (2021) with possible correct and incorrect responses provided by the authors. We will provide a comprehensive explanation of the dataset's creation process and the methods used to ensure its quality and validity. \n\nAs to the possibility that the LLM was exposed to the dataset during training, we addressed it in two ways. First, we asked the LLM to generate each of the stumpers in the dataset given their first sentence and observed poor results. Then, we also noticed, as reported in the paper, that LLMs do not perform well on recognizing the correct answer to a stumper when asked to choose between two answers. This is an indication that even if the model was exposed to the dataset during training, it was not able to memorize it properly. Finally, we note that the stumpers dataset was published in 2021 so it is possible that it was not included in the GPT training set. We thank the reviewer for this important comment and given the opportunity we will address it in the extra page allowed in the final version of the paper.\n\nC. In section 3.2, why the change of task format becomes answer verification?\n\nAgain, your comment about the inaccurate naming of this process was precise, and we will amend it. The task change from answering (or generating a response to) the stumper to judging the answers’ correctness, was introduced to highlight the model’s lack of “understanding”. Stumpers, by their definition, become very trivial once the solution is given (i.e. it is easy to recognize the correct answer once it is presented). Judging the truthfulness of a solution is a very simple task humans can do almost at ceiling level, yet the models could not. When asking the models to rule “correct” or “incorrect” on answers, each model had their bias (either to mostly agree or disagree). For this reason, we used the comparison task instead of asking for agreement. \n\nD. In section 3.2, the authors mentioned 'humans were presented with the two possible solutions'.\n\nThe alternative solutions were crafted by the authors to be (a) semantically relevant to the stumpers while (b) being incorrect according to the criteria for a correct response. Both the correct and the incorrect answers were presented to both human participants and LLMs to ensure a fair comparison. We will add to the discussion the need to systematically test different distractors in this task in order to ensure the results on different effort levels of the task.\n\nE. In section 3.3, why the design of the answer verification task for the models is different from the one for humans?\n\nThe unequal design of the answer verification task was motivated by the difference in the capabilities of humans and models. While models can generate responses independently, human participants remember recent experiences, which make an exact design impossible. This design choice was made in order to address this difference and ensure a more accurate comparison.\n\nF. In Figure 2, Davinci-3's success rate is abnormally high compared to the other models.\n\nWe appreciate your observation. While we did notice this anomaly, which was persistent across prompts, further analysis of this particular pattern falls out of the scope of this short paper. It's an interesting avenue for future research to delve into the reasons behind this disparity in performance among the models.\n\nG. For asking each human subject to answer one single question in the dataset...\n\nYour concern regarding the human study design is valid. Some stumpers share common mechanisms, which can introduce a potential bias when participants are exposed to multiple similar problems. Like many other psychological tests (e.g., the bat-and-the-ball, optical illusions), seeing the solution for one may bias (in favor or against) finding the solutions in the next. This tendency does not mean that the tasks are easy or that they do not call for creativity, it just points out that some cues can be (too) informative and should be avoided for the fairness of the comparison. Therefore, and following established psychological practices, we opted for a design where each participant encountered a single stumper to avoid potential learning effects.\n\nH. The results in Figures 2&3 show Davinci-3 outperforms GPT-3.5 and GPT-4 in the binary-choice task...\n\nWe apologize for any confusion. Your observation is correct, and we will clarify the text to accurately reflect the model performances based on the specific tasks.\n\nI. The authors claimed that few-shot prompting improves GPT-3's performance...\n\nWe appreciate your insight into this matter. The performance variation introduced by few-shot prompting across different models indeed requires a more thorough reporting. We will acknowledge this behavior more explicitly in the revised version.\n\nOnce again, we sincerely thank you for your thoughtful feedback and questions. Your insights have significantly enriched our understanding of potential improvements, and we are fully committed to addressing these concerns in the revised version of our paper.\n\nBest regards,\nSubmission668 authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The analysis of LLM's ability for solving stumpers seems to be an interesting research direction, but I would emphasize my concerns in Reasons To Reject.", "reasons_to_reject": "I have three primary comments regarding this paper (ranked by importance)\n\n(1). The novelty in this paper does not meet a top-venue standard.\n- Essentially, the authors took an existing dataset, used OpenAI's API to run some evaluations, recruited humans to do the same task, and plotted some diagrams. Instinctively, the contribution in the paper seems to be just a section of a regular top-venue paper or a benchmarking section for the dataset paper. \n- The experiments are too simple and not well-designed (see (2)), leading to (3), where the paper is filled with less-relevant content and discussions but with minor novel discoveries and contributions. \n\n(2). The experiments are not well-designed and organized.\n- No details of the design of the prompts. As NLP experts, the authors need to justify the design of the prompts to make the best use of LLMs for solving the stumpers. \n- No details of the dataset (the authors only mentioned the dataset size in the limitation section!). The quality of the dataset is questionable, especially for the ground-truth annotations. By looking at the samples in Appendix A and B, it seems each stumper question can have a lot of different reasonable answers, but the dataset only has one answer for each data. (see question B/D below). In addition, missing a lot of details for the experiment design (see question A/E below)\n- The human study design is questionable. The human performance should not be evaluated by asking each human subject to answer one question in the dataset, which can cause reliability issues with the evaluation result. (see question G below)\n- The answer verification section is not a verification task. Instead, the authors just converted the original QA task into a binary-choice task. (see question C below) For a verification task, I expect to see the authors ask models and humans to provide reasoning to verify whether their provided answers are correct or not. \n\n(3). The paper structure is not well-organized. \n- Authors spend a whole page in a short paper to describe the difference between 'stumpers' and system1&2 (which are not directly related to the experiment in this paper) and another full page to provide discussions, while only using half a page to describe the experiment details and another half page to report the experiment discoveries (exclude the spaces for the figures). Too much text to describe less-relevant content and hypothesis makes the whole paper read like a blog post instead of a scientific paper, leading to a lack of novelty. \n- The details of the dataset are missing (By only reading the paper, I have no idea of this dataset's quality, validity, and stats), leading to concerns about the dataset quality. E.g., how the ground-truth is collected? Can it faithfully reflect the performance of LLMs and humans?\n- Many experiment design decisions are missing supporting evidence (see questions below), which leads to a potentially over-claiming issue (see (1)). \n\nBecause of the rebuttal word limit, I don't expect the authors to be able to address all my questions and concerns, but I'd like to use the questions as suggestions for the authors to improve their work.", "questions_to_authors": "A. What is the evaluation metric for the experiments? The authors mentioned 'two authors evaluated the responses unanimously' (line 138), but how? and based on what metric? The stumpers seem to be open-ended and can have many different possible answers. \n\nB. No details of the dataset. It is possible (some/all) the models have already seen these stumper questions in their training corpus, and the models remember the answers instead of performing 'thinking'. Can authors provide justifications?\n\nC. In section 3.2, it seems the authors just changed the original open-ended QA task into a binary-choice task. Why the change of task format becomes answer verification? \n\nD. In section 3.2, the authors mentioned 'humans were presented with the two possible solutions'. Where do these alternative solutions come from? These alternatives may be easy to distinguish and are not suitable for evaluation. \n\nE. In section 3.3, why the design of the answer verification task for the models is different from the one for humans? Please provide justifications. \n\nF. In Figure 2, Davinci-3's success rate is abnormally high compared with the other models and its answer generation performance in Figure 1. Did the authors further analyze the reasons? If so, please provide justifications. (This question is more related to the concern of novelties, it seems the authors just 'presented' the results without actually analyzing the experiment scientifically)\n\nG. For asking each human subject to answer one single question in the dataset, the authors mentioned 'to avoid learning' in line 114. It seems unreasonable. Are the authors suggesting humans can learn how to answer new stumpers by seeing any other stumpers? If so, this hurts the claim that the task need 'creative thinking' and 'hard for humans to solve' (line 75). Please provide justifications for this design. \n\nH. The results in Figures 2&3 show Davinci-3 outperforms GPT-3.5 and GPT-4 in the binary-choice task, but the authors claim GPT-3.5 and GPT-4 'outperformed the GPT-3 models' (line 204). This is contradictory. Please provide justifications.\n\nI. The authors claimed that few-shot prompting improves GPT-3's performance, thus justifying 'the importance of context and prior knowledge in solving stumpers'. However, few-shot prompting leads to a performance drop on GTP-3.5 and GPT-4, which is contradictory. Please provide justifications.", "paper_topic_and_main_contributions": "The paper attempts to experiment with a few LLMs on a dataset of cognitive stumpers and evaluate the performance between LLMs and humans. \nThe experiments consist of two parts: answer generation and answer verification. \nThe authors claim that LLMs' task-solving ability outperforms humans while humans exhibit better verification capabilities.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I did not look into the missing references because of the major concerns in this paper (see Reasons To Reject and Questions For The Authors)"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I did not look into the typos because of the major concerns in this paper (see Reasons To Reject and Questions For The Authors).\n\nI would like to emphasize that the writing of this paper is not well-organized. For a short paper, I expect to see all the key justifications, experiment design decisions, results, and important outcomes in the paper, not 2 pages of intro and discussions. If the authors are spending a lot of content for discussions, the authors should provide supporting evidence from the experiments to support the 'hypothesis' in discussions."}}]

Author rebuttal:

Meta-review: The paper investigates the problem-solving capabilities of Large Language Models (LLMs) by evaluating their performance on stumpers, unique problems that pose challenges for human solvers but are easily verifiable. The paper compares the performance of four state-of-the-art LLMs (Davinci-2, Davinci-3, GPT-3.5-Turbo, GPT-4) to human participants. The paper finds that the new-generation LLMs excel in solving stumpers and surpass human performance. However, humans exhibit superior skills in verifying solutions to the same problems. The paper enhances our understanding of LLMs’ cognitive abilities and provides insights for enhancing their problem-solving potential across various domains.

Pros:
The paper addresses an interesting and novel research question: how well can LLMs solve stumpers compared to humans?
The paper uses a large and diverse set of stumpers from various sources and domains, covering logic, math, language, trivia, and creativity.
The paper employs rigorous experimental methods and statistical analyses to compare the performance of LLMs and humans on both solving and verifying stumpers.
The paper discusses the implications of the findings for the design and evaluation of LLMs, as well as the ethical and social issues raised by their problem-solving abilities.

Cons:
The paper does not provide a clear definition or formalization of what constitutes a stumper, or how to measure its difficulty or uniqueness.
Decision: Accept-Findings