TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles

## Abstract

Abstract As the application of Large Language Models (LLMs) expands, the demand for reliable evaluations increases. Existing LLM evaluation benchmarks primarily rely on static datasets, making it challenging to assess model performance in dynamic interactions with users. Moreover, these benchmarks often depend on specific background knowledge, complicating the measurement of a model’s logical reasoning capabilities. Other dynamic evaluation methods based on strong models or manual efforts may introduce biases and incur high costs and time demands, hindering large-scale application. To address these issues, we propose TurtleBench. TurtleBench collects real user guesses from our online Turtle Soup Puzzle 1 1 1 Turtle Soup Puzzles, or yes/no puzzles, involve uncovering a bottom story behind a surface story through guesses answered with ”yes” or ”no.” (Sloane, 2016 ; Wikipedia, 2023 ) platform that we developed. This approach allows for the relatively dynamic generation of evaluation datasets, mitigating the risk of model cheating while aligning assessments more closely with genuine user needs for reasoning capabilities, thus enhancing the reliability of evaluations. TurtleBench includes 1,532 user guesses along with the correctness of guesses after annotation. Using this dataset, we thoroughly evaluated nine of the most advanced LLMs available today. Notably, the OpenAI o1 series models did not achieve leading results in these evaluations. We propose several hypotheses for further research, such as “the latent reasoning of o1 utilizes trivial Chain-of-Thought (CoT) techniques” and “increasing CoT length not only provides reasoning benefits but also incurs noise costs.” The TurtleBench data and evaluation code are available at https://github.com/mazzzystar/TurtleBench . Note: The dataset mentioned in this paper may contain some elements of horror; please view selectively.

## 1 Introduction

As the capabilities of Large Language Models (LLMs) continue to improve, they are increasingly being applied across various scenarios such as e-commerce, healthcare, and daily conversations (Li et al., ; Chen et al., ; Yang et al., ). In these real-world contexts, LLMs must address a wide range of user inquiries and provide logically coherent responses. However, the unpredictability of user questions complicates the scenarios that models face, raising the bar for the reasoning capabilities of LLMs. Thus, evaluating these models’ reasoning abilities is of significant importance (Liang et al., ).

Figure: Figure 1: TurtleBench Construction (For Chinese version, refer to Fig. 14)
Refer to caption: x1.png

However, current model evaluation practices are plagued by issues such as fraud and data contamination (Zhou et al., ). On one hand, we call for integrity and fairness in evaluation efforts; on the other hand, the inherent limitations of many existing benchmarks cannot be overlooked. For instance, benchmarks like MMLU (Hendrycks et al., ) and ARC (Clark et al., ), which consist of single-turn static questions based on common sense and academic knowledge, contain many memorization-based items. This evaluation method primarily assesses the model’s memory capacity, making it difficult to accurately measure its language comprehension and logical reasoning skills. Furthermore, since the test sets in these benchmarks are static, they may become contaminated, compromising the reliability of the evaluation results. In contrast, MT-Bench (Zheng et al., ) is a multi-turn dialogue evaluation benchmark where models must respond to preset questions and answer follow-up inquiries. However, the open-dialogue approach introduces new challenges, as it does not provide clear standard answers, making the quality of model responses reliant on evaluations from strong models like GPT-4 (Achiam et al., ). Using GPT-4 as a judge may introduce bias, with lower scores for certain models while being more lenient towards ChatGPT. Moreover, this evaluation method has its limitations, as it cannot assess models stronger than GPT-4. A better alternative is the Chatbot Arena (Chiang et al., ), which selects better-performing models through votes from real users. This method is straightforward and has a higher credibility. However, for a new model to obtain reliable scores, it must undergo extensive public testing to gather substantial user feedback, making its scores credible.

To address these limitations, we propose TurtleBench, a reliable benchmark for assessing LLM reasoning capabilities. We designed and launched an online Turtle Soup game. As shown in the left half of Fig. 1, we present the surface and bottom story to the model, allowing it to determine the correctness of user guesses. This Turtle Soup game encapsulates nearly all the information needed for reasoning, enabling the LLM to make judgments based on contextual information (the surface and bottom story). This design ensures that the evaluation focuses primarily on reasoning capabilities rather than knowledge recall, thereby enhancing the reliability of LLM evaluations. By collecting user guesses inputted during the Turtle Soup game and conducting detailed multi-turn manual annotations, we constructed a bilingual dataset in Chinese and English. Compared to existing benchmarks for evaluating LLM reasoning capabilities, TurtleBench has three main advantages:

- •
No additional background knowledge required. All information needed for reasoning evaluation in TurtleBench is contained within the task itself, limiting the assessment to the model’s reasoning capabilities without relying on external knowledge bases, thus avoiding unfair evaluations arising from differences in knowledge bases among models.
- •
Objective and quantifiable results. In the assessment of multi-turn dialogue benchmarks, the output of the model is a piece of text, making it challenging to quantify model performance. TurtleBench quantifies the model’s reasoning ability through clear ground truth (Correct/Incorrect), eliminating interference from subjective factors.
- •
Dynamic data reduces the risk of cheating. Existing static benchmark datasets may be manipulated by some models during training to boost scores, whereas TurtleBench ensures dynamic updates of evaluation data through continuously collected new guesses from users, reducing the likelihood of models gaming fixed datasets for score inflation.

Additionally, we systematically evaluated the performance of nine LLMs on TurtleBench. When assessing the OpenAI o1 models, we identified several directions for future enhancements in large reasoning models (Valmeekam et al., ), including the incorporation of more complex reasoning topologies in latent Chain-of-Thought (CoT) processes and dynamically selecting reasoning needs for questions to mitigate the influence of noise tokens in reasoning.

## 2 TurtleBench

In this section, we describe the details of collecting real user guess data for TurtleBench, including data preprocessing and annotation, and present summary statistics of the dataset. The process of dataset creation is illustrated in Fig. 1.

### 2.1 Data Collection

We designed and launched a Turtle Soup Puzzle game(^2^22Refer to Appendix C for screenshots of the game.) specifically to collect user guesses for TurtleBench. Specifically, we first gathered 1,500 common Turtle Soup stories from the internet and filtered them down to 32 ethical and logically challenging stories to serve as the source for the Turtle Soup Puzzle platform. Users are assigned a story during the game and make guesses based solely on the available surface story. We used Claude-3.5-Sonnet (Anthropic, ) as the judge to determine whether players’ guesses are Correct, Incorrect, or Unknown. Users have eight opportunities to guess, and the answer is revealed immediately upon a correct guess or exhaustion of attempts. As participation in the game increased, we noted a significant piece of user feedback: “LLM Judgement Is Not Reasonable!” This negatively impacted the gaming experience, highlighting the need for TurtleBench.

Within two weeks of the platform’s launch, over 4,000 users posed more than 26,000 guesses, which we parsed from logs and saved as our raw dataset.

### 2.2 Data Pre-processing

During the data preprocessing stage, we first removed duplicates from the 26,000 collected entries; for example, “Is the Turtle Soup poisonous?” and “Is the soup he drank poisonous?” essentially pose the same question. Next, we eliminated questions that could not be answered with Correct, Incorrect, or Unknown, such as, “How old is the man this year?” Finally, we excluded ambiguous questions. For example, in the story “The Best Friend” (refer to Fig. 10), the guess “Did he do something to his wife’s best friend” contains the word “something,” which could refer to anything, making it ambiguous. Through these preprocessing steps, we could initially enhance the quality of the dataset.

In the annotation phase, we initially categorized entries into three classes: Correct, Incorrect, and Unknown. However, during the annotation process, we found it challenging to distinguish between the labels for ”Incorrect” and ”Unknown” in many cases. For instance, in the story “The Turtle Soup” (see Fig. 9), both responses to the guess “The turtle is kept by a man” could be reasonable. To ensure evaluation stability, we categorized “Unknown” responses as “Incorrect,” resulting in a final classification of two categories: Correct and Incorrect. Ultimately, from the original 26,000 entries, we annotated 4,448 guesses. We conducted preliminary tests across all LLMs and filtered out simple questions that all models answered correctly. On the remaining 1,699 entries, we performed a secondary confirmation of annotations. We ultimately obtained a dataset of 1,532 accurately annotated entries.

### 2.3 Data Statistics

From the collection of 26,000 real user guess data, we ultimately annotated 1,532 entries. We recorded the number of guesses for each Turtle Soup story, as shown in Fig. 2. Additionally, we provide some more detailed examples of the dataset in Appendix B.

Figure: Figure 2: Number of User Guesses in Each Story (For Chinese version, refer to Fig. 15)
Refer to caption: x2.png

## 3 Experiments

### 3.1 Setup

#### Models

We evaluated nine top LLMs on the TurtleBench, covering both open-source and closed-source models, as shown in Table 1. The o1 series models (o1-preview and o1-mini) (OpenAI, ) and GPT-4o (OpenAI, ) represent OpenAI’s current state-of-the-art models. The Claude series models developed by Anthropic were assessed, specifically the advanced Claude-3.5-Sonnet (Anthropic, ). Llama models (Dubey et al., ) are open-sourced by Meta, and we conducted experiments on Llama-3.1-405B and Llama-3.1-70B. Additionally, we evaluated popular models recently developed by Chinese institutions, including Moonshot-v1-8k (MoonShot-AI, ), DeepSeek-v2.5 (DeepSeek-AI, ), and Qwen-2-72B (Qwen-Team, ). For all closed-source models, we used their official APIs; for all open-source models, we utilized the Model-as-a-Service Provider, SiliconFlow’s API (^3^33[https://siliconflow.cn/](https://siliconflow.cn/)).

#### Settings

When evaluating LLMs on the TurtleBench dataset, we ensured parameter settings were consistent whenever possible. We set the temperature of all LLMs to 0 and top_p to 0.9. Furthermore, we uniformly employed two prompt templates: 0-shot and 2-shot templates. Complete prompt templates can be found in Appendix A. It is important to note that OpenAI’s o1 series models (o1-preview and o1-mini) currently do not support custom parameters (^4^44[https://platform.openai.com/docs/guides/reasoning/how-reasoning-works](https://platform.openai.com/docs/guides/reasoning/how-reasoning-works)), so we maintained the default settings. Additionally, to save on API costs, we only evaluated the o1 series models in the 0-shot setting; for a related cost analysis, see Appendix E.

**Table 1: Evaluated LLMs**
| Model | Checkpoint Name | #Parameters | Publisher |
| --- | --- | --- | --- |
| OpenAI o1-preview | o1-preview-2024-09-12 | undisclosed | OpenAI |
| OpenAI o1-mini | o1-mini-2024-09-12 | undisclosed | OpenAI |
| GPT-4o | gpt-4o-2024-08-06 | undisclosed | OpenAI |
| Claude-3.5-Sonnet | claude-3-5-sonnet-20240620 | undisclosed | Anthropic |
| Llama-3.1-405B | Meta-Llama-3.1-405B-Instruct | 405B | Meta |
| Llama-3.1-70B | Meta-Llama-3.1-70B-Instruct | 70B | Meta |
| Moonshot-v1-8k | moonshot-v1-8k | undisclosed | MoonShot AI |
| DeepSeek-V2.5 | DeepSeek-V2.5 | 236B | DeepSeek |
| Qwen-2-72B | Qwen2-72B-Instruct | 72B | Alibaba |

### 3.2 Main Results

Table 6 and Table 3 present the evaluation results for 0-shot and 2-shot settings, respectively. We report the average accuracy per story, overall accuracy across all test cases, and F1 Score. These experimental results clearly illustrate performance differences among the models. Notably, Claude-3.5-Sonnet and GPT-4o outperform other models significantly, both achieving overall accuracy exceeding 87%. However, the performance of OpenAI’s latest o1 series models was underwhelming, with o1-preview ranking third and o1-mini lagging nearly 14% behind GPT-4o. More discussion on the performance of the o1 models can be found in Section 3.4. Following them were Qwen-2-72B, Moonshot-v1-8k, and Llama-3.1-405B, with decreasing performance, while Deepseek-v2.5 and Llama-3.1-70B ranked the lowest. We found that a larger number of parameters in different model series does not necessarily correlate with better performance compared to models with fewer parameters. For instance, Qwen-2-72B outperformed both Llama-3.1-405B and the 236B parameter Deepseek-V2.5 model.

**Table 2: Zero-Shot Evaluation Results(^6^66Ordered by Overall Accuracy. Same as below.)**
| Model | Story-Level Avg. Acc. | Overall Acc. $\uparrow$ | F1 Score |
| --- | --- | --- | --- |
| GPT-4o | 88.05% | 87.66% | 0.8501 |
| Claude-3.5-Sonnet | 87.63% | 87.53% | 0.8436 |
| OpenAI o1-preview | 84.65% | 84.40% | 0.8071 |
| Qwen-2-72B | 83.62% | 82.90% | 0.7741 |
| Moonshot-v1-8k | 82.80% | 82.05% | 0.7619 |
| Llama-3.1-405B | 82.39% | 81.79% | 0.8114 |
| Deepseek-V2.5 | 80.48% | 79.77% | 0.7368 |
| Llama-3.1-70B | 79.44% | 78.33% | 0.7340 |
| OpenAI o1-mini | 73.66% | 73.69% | 0.6480 |

**Table 3: Two-Shot Evaluation Results**
| Model | Story-Level Avg. Acc. | Overall Acc. $\uparrow$ | F1 Score |
| --- | --- | --- | --- |
| Claude-3.5-Sonnet | 90.00% | 89.49% | 0.8729 |
| GPT-4o | 87.89% | 87.92% | 0.8521 |
| Qwen-2-72B | 85.85% | 85.12% | 0.8152 |
| Moonshot-v1-8k | 84.71% | 84.07% | 0.8039 |
| Llama-3.1-405B | 82.20% | 81.72% | 0.8061 |
| Deepseek-V2.5 | 81.70% | 80.68% | 0.7723 |
| Llama-3.1-70B | 79.52% | 79.37% | 0.7713 |

To analyze whether there are significant differences among stories, especially those that are particularly challenging and may lead to discrepancies in accuracy, we calculated the average accuracy on each story in the 0-shot evaluation. This average accuracy was computed by story and overall accuracy, as shown in Fig. 3. We found that the overall average accuracy calculated by story differs from the overall accuracy by only 0.01%, indicating that most stories have a comparable level of difficulty, demonstrating the stability of this evaluation. However, there are individual stories, such as A Painting (see Fig. 11), that are more challenging, but since the number of samples for these stories is relatively small, their impact on the overall results is limited.

Figure: Figure 3: Story-Level Zero-Shot Evaluation Results (For Chinese version, refer to Fig. 16)
Refer to caption: x3.png

Furthermore, to explore the benefits of few-shot prompting on model performance, we compared the results of 0-shot and 2-shot evaluations, as shown in Table 4. We found that across all models, performance under 2-shot prompting improved compared to 0-shot. Specifically, the accuracy of Claude-3.5-Sonnet, Qwen-2-72B, and Moonshot-v1-8k increased by approximately 2%, while Deepseek-V2.5 and Llama-3.1-70B saw an increase of about 1%. The performance of Llama-3.1-405B slightly decreased under 2-shot, but the difference is not significant.

**Table 4: Comparation between 0-shot and 2-shot Evaluations.**
| Model | 0-shot Overall Acc. | 2-shot Overall Acc.$\uparrow$ | Diff. |
| --- | --- | --- | --- |
| Claude-3.5-Sonnet | 87.53% | 89.49% | 1.96% |
| GPT-4o | 87.66% | 87.92% | 0.26% |
| Qwen-2-72B | 82.90% | 85.12% | 2.22% |
| Moonshot-v1-8k | 82.05% | 84.07% | 2.02% |
| Llama-3.1-405B | 81.79% | 81.72% | -0.07% |
| Deepseek-V2.5 | 79.77% | 80.68% | 0.91% |
| Llama-3.1-70B | 78.33% | 79.37% | 1.04% |

### 3.3 Evaluation in English

TurtleBench is an evaluation benchmark in the Chinese context. To explore the performance of models on TurtleBench across multiple contexts, we translated the current 1532 samples from Chinese into English using Claude-3.5-Sonnet. The translated samples and labels were manually reviewed. We present the new results of the 0-shot and 2-shot evaluations in Tables 5 and  6, respectively. Notably, Claude-3.5-Sonnet and Llama-3.1-405B ranked first in the 0-shot and 2-shot evaluations, respectively. It is worth mentioning that GPT-4o and Deepseek-V2.5 significantly outperformed their 0-shot performance in the 2-shot evaluation. On the English dataset, OpenAI’s o1 series models still lag behind, and we analyze and speculate on this phenomenon in Section 3.4.

**Table 5: Zero-Shot Evaluation Results on the Translated Dataset**
| Model | Story-Level Avg. Acc. | Overall Acc. $\uparrow$ | F1 Score |
| --- | --- | --- | --- |
| Llama-3.1-405B | 87.87% | 86.95% | 0.8445 |
| Claude-3.5-Sonnet | 85.22% | 84.27% | 0.7935 |
| OpenAI o1-preview | 82.41% | 82.90% | 0.7838 |
| Qwen-2-72B | 82.25% | 81.92% | 0.7682 |
| Llama-3.1-70B | 82.49% | 81.53% | 0.7851 |
| Moonshot-v1-8k | 81.76% | 81.33% | 0.7671 |
| GPT-4o | 79.48% | 79.57% | 0.7050 |
| OpenAI o1-mini | 75.60% | 75.13% | 0.6752 |
| Deepseek-V2.5 | 68.47% | 68.41% | 0.4450 |

**Table 6: Two-Shot Evaluation Results on the Translated Dataset**
| Model | Story-Level Avg. Acc. | Overall Acc. $\uparrow$ | F1 Score |
| --- | --- | --- | --- |
| Claude-3.5-Sonnet | 86.27% | 85.18% | 0.8021 |
| Llama-3.1-405B | 85.59% | 84.79% | 0.8198 |
| GPT-4o | 83.04% | 83.03% | 0.7658 |
| Qwen-2-72B | 83.38% | 83.03% | 0.7943 |
| Moonshot-v1-8k | 82.36% | 81.72% | 0.7836 |
| Llama-3.1-70B | 80.96% | 80.42% | 0.7774 |
| Deepseek-V2.5 | 77.69% | 76.37% | 0.6610 |

### 3.4 Why the OpenAI o1 Models Perform Poorly

The OpenAI o1 series models use latent CoT to significantly enhance reasoning performance, yet they perform poorly on our dataset. Here, we provide some analysis and explanations. We extracted 65 guesses from the Chinese version of the TurtleBench dataset that were correctly answered by the other seven models, excluding the o1 series. Using a prompt similar to the previous zero-shot evaluation (which includes a request for judgement reasoning: Fig. 7), we queried o1-preview to obtain both the model’s judgment on a guess and its reasoning. This reasoning is key to analyzing where the model goes wrong.

Figure: Figure 4: Completion Token Lengths for Wrong and Right Judgments of o1-preview
Refer to caption: x4.png

Firstly, the new judgments are quite interesting. Among the guesses, 29 were re-evaluated as correct by o1-preview. Unlike other models, the default temperature for OpenAI’s o1 model is 1.0 and cannot be adjusted, which is a source of response inconsistency. This may also indicate that the o1 model likely does not implicitly employ non-linear CoT strategies like Monte Carlo Tree Search (MCTS) (Zhao et al., ) or Self-Consistency (Wang et al., ). Trivial CoT methods inevitably lead to single-point reasoning failures and are hard to self-correct, suggesting that the model’s reasoning consistency has significant room for improvement.

Secondly, from the model’s own reasoning, it tends to focus too much on details. For example, in the story ”The Elevator” (for the complete story, see Fig. 8), one of the user’s guesses was ”I don’t like going to school.” The bottom story mentions, ”On Monday morning, urged by my mother, I absent-mindedly enter the elevator to go to school…” However, o1-preview fixated on a small detail, the word ”absent-mindedly” (^7^77The full response from o1-preview is: ”Correct. Because the story mentions that I absent-mindedly walked into the elevator at my mother’s urging, indicating that I was less inclined to go to school.”), leading it to confirm that the user’s guess of ”I don’t like going to school” was correct. Inferring ”I don’t like going to school” from ”absent-mindedly” is a classic reasoning error caused by the paradox of induction.

Finally, we observed another important phenomenon. Among the 65 new guesses we tested, we recorded the number of completion tokens for each output, which can reflect the computational load of the model’s latent CoT to some extent. We separately counted the number of completion tokens for wrong and right judgments, as shown in Fig. 4. It can be observed that for wrong judgments, the model often generates more completion tokens. Therefore, we hypothesize that more tokens for reasoning do not necessarily lead to better outcomes; excess tokens might introduce noise, potentially damaging reasoning performance for certain tasks (Sprague et al., ; Liang et al., ).

## 4 Related Work

In real-world scenarios, the language understanding and reasoning capabilities of LLMs face increasingly complex and diverse challenges, making reliable evaluations of LLMs a critical issue. Although many benchmarks have been proposed, the reliability of existing evaluation methods still faces several challenges (Yu et al., ).

For example, benchmarks that evaluate models’ commonsense reasoning abilities often use static datasets and multiple-choice questions. These include MMLU (Hendrycks et al., ), ARC (Clark et al., ), HellaSwag (Zellers et al., ), AGIEval (Zhong et al., ), PIQA (Bisk et al., ), and GSM8K (Cobbe et al., ). However, static datasets pose a risk of data leakage, where models can overfit the test data to improve evaluation results (Zhou et al., ). Furthermore, these benchmarks often heavily rely on background knowledge, making it difficult to disentangle the model’s logical reasoning capabilities from the evaluation.

At the same time, some studies have proposed multi-turn dynamic interactive evaluation benchmarks, such as MT-Bench (Zheng et al., ), BotChat (Duan et al., ), and AgentBench (Liu et al., ). These benchmarks typically do not have definitive correct answers, often relying on powerful models like GPT-4 as judges. However, this evaluation method can lead to instability and unreliability due to biases in the judge models, and it also tends to incur high costs.

To address the issues in these benchmarks, some studies have proposed real-time benchmarks based on human interaction, such as Chatbot Arena (Chiang et al., ) and FlagEval (BAAI, ). While these methods are more credible, new models often require longer test periods to obtain reliable scores, leading to high time costs.

We believe that a reliable evaluation benchmark should align with the real-world application needs and focus on the performance and practicality of LLMs in real scenarios (Han et al., ). Moreover, the evaluation dataset should be capable of real-time updates to prevent models from cheating by memorizing test data. Therefore, this paper proposes TurtleBench, an evaluation benchmark with a continuously updating dataset, offering concise and easily quantifiable evaluation results, ensuring reliability while meeting the real-world needs of users.

## 5 Conclusion

We propose a new reliable evaluation benchmark, TurtleBench, specifically designed to assess LLMs’ reasoning and understanding abilities in real-world scenarios. Our evaluation framework collects 32 turtle soup stories and releases a turtle soup game in which LLMs serve as judges. Through this game, we can collect user query data in real-time and dynamically update the evaluation dataset, thereby avoiding distortion in evaluation results caused by data contamination and enhancing the credibility of the assessment. We evaluated nine of the currently most popular top LLMs, and the results show that closed-source models represented by GPT-4o and Claude-3.5-Sonnet still exhibit state-of-the-art overall performance, while the latent reasoning techniques of o1 still have room for improvement. In future research, we will continue to explore methods to enhance the reliability of LLM evaluations to obtain more authentic assessment results, facilitating the application of LLMs in real-world scenarios.

## Appendix A Prompt Templates

Figure: Figure 5: Prompt Template for 0-Shot Evaluation

Figure: Figure 6: Prompt Template for 2-shot Evaluation

Figure: Figure 7: Prompt Template for 0-Shot Evaluation with Request for Judgement Reasoning

## Appendix B Examples in the TurtleBench Dataset

Figure: Figure 8: Story “The Elevator” and Relevant Guesses

Figure: Figure 9: Story “The Turtle Soup Story” and Relevant Guesses

Figure: Figure 10: Story “The Best Friend” and Relevant Guesses

Figure: Figure 11: Story “A Painting” and Relevant Guesses

## Appendix C Screenshots of the Turtle Soup Puzzle Platform

Figure: Figure 12: Screenshot
Refer to caption: extracted/5898689/figures/screenshot.jpeg

Figure: Figure 13: Screenshot (In Chinese)
Refer to caption: extracted/5898689/figures/screenshot_zh.jpeg

## Appendix D Figures in Chinese

Figure: Figure 14: TurtleBench Construction (For English version, refer to Fig. 1)
Refer to caption: x5.png

Figure: Figure 15: Number of User Guesses in the TurtleBench dataset (For English version, refer to Fig. 2).
Refer to caption: x6.png

Figure: Figure 16: Story-Level Zero-Shot Evaluation Results (For English version, refer to Fig. 3)
Refer to caption: x7.png

## Appendix E Cost Analysis

As shown in Table 7, we present the pricing for each LLM, the total number of tokens consumed by each model, and our overall expenditure.

**Table 7: Pricing, Token Usage and Cost for Each LLM**
| Model | Cost per 1M Input Tokens | Cost per 1M Output Tokens | Token Usage | Total Cost |
| --- | --- | --- | --- | --- |
| OpenAI o1-Preview | $15.00 | $60.00 | 6324152 | $290.10 |
| OpenAI o1-mini | $3.00 | $12.00 | 4758673 | $41.94 |
| GPT-4o | $5.00 | $15.00 | 4526769 | $11.51 |
| Claude-3.5-Sonnet | $3.00 | $15.00 | 5808712 | $21.70 |
| Llama-3.1-405B | ¥21.00 | ¥21.00 | 4694779 | ¥98.59 |
| Llama-3.1-70B | ¥4.13 | ¥4.13 | 4694654 | ¥19.39 |
| Deepseek-V2.5 | ¥1.33 | ¥1.33 | 4411584 | ¥5.87 |
| Qwen-2-72B | ¥4.13 | ¥4.13 | 4316888 | ¥17.83 |

It is important to note that the two models in the OpenAI o1 series were evaluated using only 0-shot evaluation on the Chinese version of TurtleBench, with a total of 1,532 evaluation items. Other models underwent both 0-shot and 2-shot evaluation on the Chinese and English versions of TurtleBench, totaling $1,532*4=6,128$ evaluation items. Based on the information above, we can roughly calculate the unit cost per model for a single guess, as shown in Table 8.

**Table 8: Unit Costs**
| Model | Cost per 1K Guesses |
| --- | --- |
| OpenAI o1-Preview | $189.36 |
| OpenAI o1-mini | $27.38 |
| GPT-4o | $1.87 |
| Claude-3.5-Sonnet | $3.54 |
| Llama-3.1-405B | ¥16.09 |
| Llama-3.1-70B | ¥3.16 |
| Deepseek-V2.5 | ¥0.96 |
| Qwen-2-72B | ¥2.91 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: The acquisition of TurtleBench involves significant manual labor, such as selecting 32 logically challenging stories from a pool of over 1,500 Turtle Soup stories and narrowing down 1,699 entries from 26,000 collected user guesses to form the dataset. Besides some basic automated filtering, like removing duplicates, extensive manual work is required. Who is responsible for this manual effort? What criteria determine which stories are considered logically challenging and which user guesses should be retained? Additionally, ensuring the accuracy of user guess annotations would also require considerable manual work, yet the paper lacks a detailed description of this process.
soundness: 2
strengths: 1. The idea is innovative. Using the game logic of Turtle Soup to test the reasoning abilities of LLMs is a unique and intuitively reasonable approach. Given the surface and bottom stories, the LLM should, in theory, possess a deep understanding and logical comprehension of the entire narrative to provide correct answers to user guesses.
2. A thoughtful analysis is provided on counterintuitive findings. The authors highlight that although the OpenAI o1 series models use latent Chain-of-Thought (CoT) techniques to enhance reasoning performance, they still perform poorly on TurtleBench. They propose a possible explanation: the reasoning of OpenAI o1 models may focus excessively on details. This insight is both illuminating and inspiring for future efforts to improve the reasoning abilities of LLMs.
summary: The paper introduces TurtleBench, a novel evaluation benchmark designed to assess the reasoning capabilities of LLMs using data collected from an online game called Turtle Soup Puzzles. Traditional benchmarks for evaluating LLMs often rely on static datasets or manual evaluations, which can introduce bias, inefficiencies, and limitations in measuring logical reasoning. TurtleBench addresses these challenges by offering a dynamic and realistic evaluation approach. Specifically, each Turtle Soup story features a surface story and a bottom story. The authors gathered user guesses in the form of questions and annotated them. These annotated guesses were subsequently used to evaluate the reasoning abilities of LLMs, based on the context provided by the surface and bottom stories.
weaknesses: 1. One of the main contributions of TurtleBench, as claimed by the authors, is its ability to address the problem of static evaluation benchmarks, which are prone to contamination and compromise the reliability of evaluation results. They argue that TurtleBench is dynamic because the data is collected from real user guesses from online platform. However, once the data is collected and filtered, it becomes static and no longer evolves, making it susceptible to contamination. The process of preparing TurtleBench data involves extensive manual effort for checking and filtering, resulting in a dataset that, once finalized, remains static.
2. Limited size and scope. TurtleBench ended up with only 32 stories, which is an unconvincing number. Additionally, it is unclear whether these 32 stories represent different categories, what specific content each story covers, or how they can comprehensively test the reasoning abilities of LLMs. The article does not address these concerns, so while the evaluation approach is quite interesting, there isn't enough evidence to support its comprehensiveness.
3. Limited analysis. The paper raised an interesting point: the OpenAI o1 series, which is well-known for its reasoning capabilities, performs worse than other LLMs on TurtleBench. While the authors suggest possible reasons, such as the o1 models focusing excessively on details, they do not explore these explanations further. Additionally, there is no convincing ablation study to support their hypotheses.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: On line 131, the authors say they used Claude-3.5-sonnet for all real-world games. Could this bias the data in favor of Claude (who tops the benchmarks)? What was the reason the authors chose Claude as the sole judge in this process?

On line 135, the authors state that users' feedback that the LLM judgment is not reasonable highlights the need for TurtleBench, but is not clear to me why this is the case? Can the authors please elaborate on this point.

On lines 152-153, the authors state that it was challenging to distinguish between “Incorrect” and “Unknown” cases which raises the question: is this benchmark as “Objective and quantifiable” (line 103) as suggested in the intro? How could this be shown with greater conviction?

On line 143: the authors say the prompts were deduplicated semantically, but do not explain how. What method was used to deduplicate prompts? 

On lines 154-159, the authors explain that the ground truth “golden answers” were annotated. However, it is not quite clear what the exact process was here. Who did the annotations? Did the annotators construct the labels with or without seeing Claude’s original label? How was the second confirmation done?

On line 201, the authors say they used temperature 0 and top_p 0.9. Why does top_p need to be set when temperature is already set to 0, which should be greedy decoding?

On line 312, the authors say Claude-3.5-Sonnet was used to translate all the queries to English. Does this pose any sort of advantage for or against Sonnet given it is one of the models being evaluated? Why was only Claude chosen for this task?

On line 356, the authors hypothesize that o1’s restriction to temperature 1 is a possible reason for poor performance. Why not test GPT-4o on temperature 1 to compare degradation (if any)? This would isolate the delta to just be model type rather than temperature and model type.

On line 366-374, the authors make an observation that o1 focuses too much on details, which causes errors. However, only one such example is provided. Can the authors provide additional examples to show a genuine pattern of failures?

On line 381-384, the authors suggest that excess tokens might introduce noise, damaging performance based on the results in Figure 4. However, the authors do not consider the confounding effect that more difficult queries may lead the model to produce more reasoning (and still fail). Could the authors show an experiment where this effect is controlled for? For example, comparing correct and incorrect responses generated for the same query. Or potentially, subtracting the average length from all other models on a given question, to find the gain over average tokens used for the given query.

On lines 412-414, the authors suggest that TurtleBench has “concise and easily quantifiable evaluation results” while “meeting the real-world needs of users”. Can the authors explain or show how “concise and easily quantifiable” answers correlate real-world tasks? Does TurtleBench correlate with real-world performance (say does it correlate with Chatbot Arena rankings, or any other determined standard for what is “real-world”)?

Note there are backwards quotations on lines: 53, 153, 368, 370, 371, 373, 374, and 377.
soundness: 2
strengths: Originality:
The authors present an interesting and novel framework to collect data from users via LLM-judged games. This feels like a strong foundation for expansion— games are a natural environment to study and collect data on human-llm interactions, where the human is incentivized to act intentionally toward a goal (winning the game). The Turtle Soup Puzzle game designed by the authors is clearly engaging to users, judging by the amount of data collected and the number of unique users. The author's “game” approach to data collection is an interesting way to scale real-world data collection. There is potential to expand frameworks like this in future research.
summary: This paper presents TurtleBench, and LLM benchmark utilizing lateral thinking puzzles and yes/no labels. Specifically, the authors construct a website for real users to play “Turtle Soup Puzzles” where users try and guess a “bottom story” based on the given “top story”. Then they use an LLM (Claude-3.5-Sonnet) to give feedback to the user if their guess is correct, incorrect, or unknown. Through this, they crowdsource over 26,000 user queries regarding the 32 Turtle Soupe Puzzles. To construct the evaluation dataset, the authors filter the collected data for duplicates/ambiguities. Then the authors annotate the correct/incorrect labels and filter queries that are trivial, yielding 1.5k queries with an associated ground truth label. The authors than use these (puzzle, query, label) triplets to evaluate different LLMs, looking at the accuracy in which they can output the ground truth label. The authors evaluate a number of popular and high performing models with both 0-shot and 2-shot prompting schemes. Interestingly, they find OpenAI’s recent SOTA models, o1-preview and o1-mini, to perform relatively poorly. The authors investigate this phenomenon, and hypothesize that more reasoning tokens may actually damage reasoning.
weaknesses: In their introduction, the authors criticize static benchmarks like MMLU for lack of logical reasoning, and contamination. However, it is unclear how the author’s proposed benchmark mitigates these problems. Mainly, despite crowdsourced user queries, the authors still must hand annotate each correctness label (which took 2 weeks according to line 66 in the main figure). It is doubtful that this process is continuously replicable, such that a new fresh set of labeled queries can be released in frequent intervals so that contamination is avoided. As such, it seems the author’s implementation is not substantially different that a regular static benchmark other than being based on lateral thinking puzzles and crowdsourcing queries (but not labels). Therefore, this benchmark still contains all the weaknesses the authors outlined for existing static benchmark implementations.

Moreover, the claims (lines 412-414, 418-419) of addressing the balance between real-worldness and verifiability seem unsupported.  Lateral thinking puzzle judgments are an interesting aspect to test LLMs on, but are not real-world in themselves. Alternatively, the authors have not shown that excelling at this task is correlated to excelling at some other open-ended task that is much more applicable but hard to judge, e.g. general reasoning capability, or more human preferred. TurtleBench is based entirely on an LLM’s ability to judge queries on 32 different lateral thinking puzzles. To make the claims on real-world application supported, the paper should explore how this connects to performance on actual real-world scenarios in depth. For more details, please see the questions section for specifics.

Additionally, the section analyzing o1’s performance (section 3.4) needs more supporting experimentation. Additional questions and recommendations are in the questions section.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How did you address potential biases arising from the user demographics?

2. How do you ensure that the models evaluated, especially closed-source ones like GPT-4o, have not been exposed to the Turtle Soup stories?

3. did you observe any notable differences in model performance or reasoning patterns compared to the Chinese dataset beyond the reported accuracy scores?

4. How did you address variations in model capabilities and API constraints to ensure a fair and consistent evaluation across all models?

5. Have you analyzed what makes certain stories more challenging for the models?

6. did you explore factors such as training data diversity, model architecture,etc for why that larger models do not always outperform smaller ones.
soundness: 2
strengths: 1. Novel benchmark creation: The field needs more evals, and more high-quality evals. The paper is a good step in the right direction.

2. Dynamic collection of data: By collecting data from an online platform where users interact with LLMs in real-time, the benchmark reduces the risk of data contamination. This is extremely important. also very useful to avoid overfitting on benchmark.

3. Authors give hypotheses related to latent reasoning and Chain-of-Thought (CoT) techniques causing underperformance.

4. By translating the dataset into English and re-evaluating the models, there is multilingual evaluation.

5. Making the dataset and evaluation code publicly available is another plus point.
summary: The paper introduces TurtleBench, an evaluation benchmark designed to assess the reasoning (and understanding) capabilities of LLMs.The benchmark has 1,532 user guesses with annotated correctness. The results are that while GPT4o and Claude3.5 Sonet out-perform other models, the o1 series models underperform.
weaknesses: 1. The paper lacks detailed information on how the user guesses were annotated for correctness.

2. Since the data comes from users interacting with the Turtle Soup Puzzle platform, there may be inherent biases based on the user demographics and the nature of the puzzles.

3. While accuracy and F1 scores are reported, the paper does not provide enough detail on how these metrics are calculated

4. The results are presented without statistical analysis to determine if the differences in model performances are significant.

5. The OpenAI o1 models could not be adjusted for temperature and top-p settings

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 2
questions: - Is my understanding of how you evaluate models correct? I.e. that a model is evaluated by asking it to be a judge of incorrect/correct given a surface and a bottom story.
- I suggest that you re-run your statistics to be more robust.
- I suggest that you investigate the o1 responses for potential refusals given the topics of your game setting. E.g. o1 may refuse to discuss cannibalism, even in a narrative setting.
- I suggest that you include related works on interactive benchmarks with objective resolution criteria.
- I suggest that you reorganize your arguments for why your benchmark may avoid the (correctly identified) issues with evaluations you identify in the introduction.
- I suggest you clarify what exactly a model is evaluated for. I had to go to the appendix to complete my understanding of this, as the body text was somewhat ambiguous.
soundness: 1
strengths: - Human data basis: Building an evaluation benchmark from insights provided by users in a naturalistic game setting is interesting and valuable. Deriving data from the same setting means that there is a good baseline of data quality, avoiding issues that LLM-generated benchmarks often encounter.
- Multilingual: The benchmark is in both Chinese and English which covers multiple major languages and expands the scope of what the benchmark may cover in the future (the evaluation was only on the English translation).
- Transparency: The appendix provides valuable insight into the prompts and the code repository is easy to replicate in only a few steps that are well documented.
summary: The paper introduces TurtleBench. TurtleBench is a benchmark consisting of 1,532 incorrect or correct user guesses to 32 different "Turtle Soup" stories, stories that have a "Surface" story (a story shown to the user) and a "Bottom" story (a story underlying the surface story that provides the reasons for why the surface story happens). The benchmark evaluates an LLM's logical reasoning ability by asking them to judge whether a user guess is correct or incorrect given both the surface and bottom stories but evaluating only on the surface story. The benchmark evaluates whether an LLM is a good judge of guesses to the story, indicating its ability for logical coherence between the surface and the bottom story. The benchmark is originally in Chinese and stem from 26,000 entries filtered down and translated to English. The paper also speculates why o1 models are worse than expected at this task.
weaknesses: - Experimental design: The prompt a model under evaluation receives instructs the model to answer correct or incorrect based only on the surface story, violating the premise of the benchmark and incentivizing the model to disregard the bottom story which should be the basis for logical coherence.
- Statistics: The authors claim that wrong responses from o1 have more tokens than correct responses. However, they lack a statistical test (such as significance testing under a mixed effects linear model) and using statistical heuristics for signifiance on their boxplot visualization in Figure 4 shows that this is false, as the confidence intervals overlap nearly 100%.
- Data quality: Figure 11 shows an example story "A Painting". The surface story includes irrelevant but seemingly important information for inferring the bottom story. In this case "my scalp tingled" is irrelevant to inferring the bottom story from a user's perspective, hence the data generation process may be contaminated by the difficulty of the task itself. This may also be by design to lead someone astray, but this seems to detract from the benchmark.
- Claims of objective results (minor): Similarly, in Figure 9, one of the user entries are "This soup tastes different from the human flesh he ate before" where the fact that "he ate [human flesh] before" is not in the surface story at all, suggesting that the user had separate information or they had memorized the story from a previous session. Additionally, in Figure 11, "I feel scared" is a somewhat subjective entry, though the story would imply this.
- Memorization: The authors claim that their benchmark "[reduces] the likelihood of models gaming fixed datasets for score inflation" through their interactive design. However, the stories themselves may be memorized, as evidenced above by the users themselves. The data is indeed new so this can be seen as solving this, but the stories themselves may be memorized.
- Dataset filtering: The authors filter for "something" to remove ambiguous entries, but there are many ways that "something" may be used in an unambiguous context.
- Interactivity: The authors claim it is an interactive benchmark but there is no interactivity from the model's side, only from the user side. The model's are queried in exactly the same way you would for a static benchmark.
- Related work: The related work section before the conclusion misses key papers introducing interactive benchmarks with objective resolution criteria, such as capture-the-flag challenges in Cybench and other papers. 
- Conclusions: Due to the above issues, the conclusions of model performance are dubious. Additionally, the assumptions made about the design of o1 do not hold. The authors assume o1 is composed simply of CoT and excludes self-consistency functionality, however if you observe the performance difference between CoT prompting of GPT-4o and extrapolate to similar inference costs as o1, you see a significant and large performance gap between o1 and simple CoT, implying that o1 has more functionality embedded in its architecture than just CoT. I can provide sources for this statement and a series of papers that may be involved in the creation of o1.
- Translation: How the translation is done isn't expanded on but this may be a critical bottleneck to the English benchmark's data quality.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Can the authors address my concerns above?

Also, given that the LLM judge can achieve high judging accuracy, would it be interesting to test how good the LLMs can "play" this kind of games? That is, for a fixed LLM judge, we can use it to see which LLM can get the correct surface from the bottom using the least number of queries? It could be a nice way to test the lateral thinking ability of LLMs.
soundness: 3
strengths: The paper proposes a very interesting benchmark and I can see a very straightforward application for LLMs to serve as the "host" for some of these lateral thinking games. From the paper, it also seems that the LLMs are quite capable of judging the correctness of the questions that is aligned with human annotations.
summary: This paper proposes a meta-evaluation benchmark for evaluating LLMs judging Yes / No questions from lateral thinking puzzles. To construct such benchmark, human participants are recruited for writing Yes / No questions and manual annotations are performed on the correctness of these questions. With this benchmark, the paper benchmarks 9 frontier models on this dataset and finds that OpenAI o1 series models are not the best models for this benchmark.
weaknesses: There are a few concerns I have for this benchmark

1. The task seems already quite saturated by decent LLMs. There is already a filtering for many easy to judge questions. "Ultimately, from the original 26,000 entries, we annotated 4,448 guesses. We conducted preliminary tests across all LLMs and filtered out simple questions that all models answered correctly. On the remaining 1,699 entries, ..." Even so, the accuracy on this benchmark can be around 80%, which means the unfiltered accuracy should be more than 90%. The remaining questions are also very likely to be ambiguous to judge. Given how easy this benchmark could be, I am concerned about the adoption of it.

2. The paper claims "Other dynamic evaluation methods based on strong models or manual efforts may introduce biases and incur high costs and time demands, hindering large-scale application." in Abstract and "Therefore, this paper proposes TurtleBench, an evaluation benchmark with a continuously updating dataset, offering concise and easily quantifiable evaluation results, ..." in Related Work. I don't see how the dataset can be continuously updated without (high) human annotation cost. Also once the benchmark is released, it can be found in the training corpus of LLMs which makes evaluation on future collected data appear to be easier for LLMs.

3. There is also usually some violent or horrific content associated with these lateral thinking problems. I am also slightly concerned about ethics.