WILT: A Multi-turn, Memorization-Robust Inductive Logic Benchmark for LLMs

Authors: eryk  banatt
jonathan  cheng
tiffany  hwu
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-17 02:04:54

Abstract: While large language models (LLMs) have shown impressive capabilities across a wide range of domains, they still encounter significant challenges in reasoning tasks that require gathering evidence over multiple turns and drawing logical conclusions from this evidence. Despite the multi-turn nature of many real-world LLM use cases, most existing benchmarks rely on carefully curated single-turn tests, which often blur the line between memorization and genuine reasoning. To address this, we introduce the $\textbf{Wason Inductive Logic Test (WILT)}$, a simple yet challenging multi-turn reasoning benchmark designed to resist memorization. WILT is inspired by the Wason 2-4-6 task, where participants must infer a basic boolean function involving three variables (e.g., $x < y < z$) by proposing test cases (such as $(2, 4, 6)$). In WILT, each test starts from a clean slate, with only the initial instructions provided, preventing models from relying on pre-learned responses. Our findings reveal that LLMs struggle with this task, with the best-performing model achieving only 28% accuracy, highlighting a significant gap in LLM performance on complex multi-turn reasoning tasks.

Main paper: 
 introduction: 
 Large Language Models (LLMs) powered by the transformer architecture [36] have enabled a new computing paradigm powered by natural language. LLMs have seen an increased ubiquitousness in day-to-day life beyond the machine learning research space, where they help many people across the world with common tasks. These models interact with users through multi-turn conversations, a capability added to next-token-prediction models via instruction-tuning [19] and alignment posttraining phases [25] .Measuring the performance of LLMs has been challenging for the research community. Publicly available benchmarks are subject to Goodhart's law [12] or implicit overfitting [27] , and difficult benchmarks often keep a held-out, publicly unavailable test set to accurately evaluate models [5] . In addition, the vast majority of benchmarks are suites of single-turn tests, which differ substantially from the common day-to-day use of LLMs [15] [16] .The reasoning capability of LLMs, particularly in multi-turn scenarios, is of substantial interest. A commonly reported failure pattern for LLMs is the "doom loop", where the model repeatedly responds with a near-identical message to one of its earlier messages, providing minimal utility. While some benchmarks have emerged to attempt to measure this multi-turn propensity of repetition [18] , none so far have done so for multi-step inductive reasoning.In this work, we make the following contributions:1. We introduce the Wason Inductive Logic Test (WILT), a multi-turn inductive logic benchmark that is robust to memorization. We show that frontier LLMs struggle significantly on this task. 2. We further evaluate the reasoning capabilities of the LLMs by framing the task as an exploration of hypothesis space. The LLMs show marked differences in the rate of hypothesis space reduction, novelty of responses, and complexity. 
wilt: 
 The Wason Inductive Logic Test (WILT) is a benchmark for LLM reasoning inspired by the Wason 2-4-6 task [41] . Models begin with the instruction that they must uncover the hidden rule, and may pose up to 30 test cases of that rule. For example, they can pose the tuple (2, 4, 6) and the test will respond with "True, 29 Attempts Remaining."All hidden rules take three numbers and return a boolean. These rules are simple and non-stochastic, so there is no additional value to posing the same test multiple times. Valid inputs include any float or integer that can be typed in three or fewer characters, excluding signs and the decimal point (e.g.-999, 1. 23, 5) . The hidden rules are written as Python lambda functions. After a maximum of thirty tries (or any turn before then), the model may make one attempt to guess the function, after which the test will terminate. The model must return a Python lambda function that is the same as or equivalent 1to the hidden rule in order to receive full points.WILT is conceptually simple, but very challenging. Humans can identify simple rules fairly easily despite the infinitely large hypothesis space, the unbounded difficulty of a hidden function, and the impossibility of verifying the correctness of your response. For example, consider the canonical Wason task rule of x < y < z. This rule has very high overlap with the much more arbitrary rule (x < y < z) ∧ (x ̸ = 12). The WILT benchmark therefore tests a few high-value behaviors of interest:1. Multi-Turn Capability: Participants that fall into "doom loops" are punished by virtue of having less useful information with which to infer the hidden rule. 2. Hypothesis Space Reduction: Participants are rewarded for proposing test cases that effectively narrow down the possible rules, despite that hypothesis space being infinitely large. 3. Susceptibility to Confirmation Bias: Participants who are more prone to "confirming their hypothesis" rather than seeking falsification will perform poorly upon this task. We release two test suites: a lite split, with 10 very easy tests and a canonical full split with 50 moderately difficult tests. Future work will extend this to include a procedurally generated split for robustness to overfitting. We find that the lite split quickly produces a roughly similar ordering to the full split, but we report results upon the full split for the remainder of this work. Please see the appendix for further details. The test and associated code can be found at github.com/riotgames/wilt.We calculate the string length of the rule expression. The longer the length, the more complex it is likely to be. As longer outputs tend to be arbitrarily preferred by automatic evaluators [9] , it is particularly important to measure the brevity of the response for cases in which simplicity is desired. 3. Set Inclusion: We generate a grid of integer-float tuples and apply them to guessed and actual rules to generate sets of tuples returning "True". If the set of the guessed rule is a subset or superset of the actual rule, we then calculate their set size ratio. A ratio of 1 is ideal, > 1 suggests a less complex guess, and < 1 a more complex one.Appendix A.3 shows the complexity metrics of the LLMs. Most LLMs with high accuracy have long response lengths and many operators, with some exceptions. 
related work: 
 Compared to other reasoning benchmarks, WILT stands out as both highly multi-turn focused and unusually robust to memorization. In contrast to other benchmarks, WILT requires models to interact with an environment by proposing their own test cases to uncover a hidden function without relying on pre-provided examples. This setup reduces the risk of overfitting, as each test begins with the same initial instructions, and the model must generate and interpret its own data.There are a wide variety of reasoning benchmarks used to evaluate large language models. Some very notable among these are MATH [16] , GSM8K [6] , CommonsenseQA [30] , StrategyQA [11] , BIG-BENCH [29] , SciBench [39] , SVAMP [26] , ARC-AGI [5] , MMLU [15] , GPQA [28] , and HumanEval [4] . These benchmarks are the standard for measuring LLM reasoning capabilities, but are overwhelmingly carefully chosen single-turn problems which aim to meaningfully separate the performance of different models on reasoning-like outputs such as math, code, or logic puzzles. However, these benchmarks are subject to train-on-test leakage, even if efforts are made to decontaminate the dataset [45] , and the majority are explicitly single-turn tests. Our benchmark directly measures the model's ability to navigate multi-turn scenarios and does not require careful hiding of a test set to prevent misleading results.With respect to reasoning about simple functions, a benchmark that stands out as similar to ours is CRUXEval [14] , which assembles a list of 800 simple Python functions and input-output pairs, evaluating language models on their ability to predict input from output and output from input. Our work could be seen as a multi-turn, more difficult extension of this work -one where the function is replaced with a black box, where helpful and informative input-output pairs are not provided but instead need to be searched for by the language model, and where the objective is to infer the hidden function rather than the input or output.There are a handful of multi-turn benchmarks used to evaluate LLMs. PlanBench [34] is one prominent benchmark that attempts to measure the ability of LLMs to navigate planning problems. This is a class of problems that is solved easily by classical planning algorithms such as STRIPS [10] , and like our benchmark poses a significant challenge to LLMs. PlanBench is a primarily multi-step, single-turn benchmark with a multi-turn component (i.e. replanning based on unexpected events), which contrasts with our benchmark's more direct multi-turn focus. This can be observed in the o1 models performing comparatively well on PlanBench [35] , since scaling inference time compute within a single turn would be expected to improve performance substantially. Similarly, BIG-BENCH [29] includes a "20 Questions" task, which pairs two language models together to guess a word by posing questions about it. Like ours, this involves an iterated reduction of hypothesis space over multiple turns, but unlike ours relies on potentially subjective or unreliable LLM responses to queries about the concept.Closest to ours are MINT [40] and Aidan-bench [18] , which have more direct multi-turn focus. MINT repurposes existing single-turn benchmarks by allowing models to use tools before answering. While the range of tasks in MINT is therefore quite large, strong models can still solve these tasks in few (or one) turns, and the unmodified prompts remain subject to test set leakage. Aidan-bench measures the cosine similarity between multi-turn responses. This represents a more pure measurement of the doom loop phenomenon. In our benchmark, rather than directly measuring the doom loops, we are instead measuring how often those doom loops lead to failures of reasoning. We see similar performances in our benchmark compared to Aidan-bench (e.g. Mistral Large), but with an ordering more tied to capabilities (e.g. Sonnet's strong results, see Table 1 ).Hypothesis space representation is a commonly used framing in inductive logic tasks for LLMs. In [38] , the authors show a technique called hypothesis search where the model will propose hypotheses in natural language and then implement these hypotheses as Python programs. This technique was shown to improve performance on ARC-AGI [5] , but a similar approach could be used along with chain-of-thought [42] for WILT as well. 
results: 
 Our results for this test can be found in Table 1 . We show that despite the test's relative simplicity, most models struggle substantially both to propose good tests and to infer a rule based on available evidence. As in the original Wason 2-4-6 task, we find a common failure mode to be confirmation bias -a participant will identify a plausible hypothesis and continue to propose tests that attempt to confirm it. Stronger models will more explicitly attempt to falsify these hypotheses instead.In Table 1 , we include a column approximately correct, measuring the number of rules in which the model was able to correctly identify some critical behavior of the rule, but returned a rule with failing edge cases. For example, guessing (x < y < z) instead of (x ≤ y ≤ z) is approximately correct. We include this column to highlight models that are more willing to guess immediately instead of uncovering edge cases by default (e.g. Llama 3.1 405B). In these cases, we could see potentially improved performance through more targeted prompting techniques.We find that LLMs (particularly smaller ones) will frequently repeat tests they have already used, sometimes dozens of times. We therefore also provide a column repeats which counts the total proposed tests which were already tested for that rule. Further discussion on test novelty can be found in Appendix A.4. Additionally, further experiments analyzing what parts of the task models are strong or weak at can be found in Appendix A.2. We find that some models are produce very useful test cases (e.g. chatgpt-4o-latest) whereas other models are strong at guessing the rule (e.g. o1-mini).To compare the LLMs' ability to efficiently reason about the task, we estimate how effectively each model reduces the hypothesis space [38] . At best, an LLM should always propose a triplet that eliminates as many untested hypotheses as possible. At worst, a model repeatedly proposes a triplet confirming previously covered hypotheses. For example, an LLM that has already guessed (2, 4, 6) retreads much of the same hypothesis space by guessing (4, 6, 8) rather than (0.01, -1, 100).To represent that hypothesis space, we randomly generate a large number of lambdas that encompass a wide range of potential hypotheses. For example, we randomly generate lambdas having to do with: ordering (x < y < z), equality (x = y ̸ = z), arithmetic relations (x + y = z), parity (x ≤ 10, y ≤ 5, z ≤ 100), etc. When an LLM proposes a triplet, we cross off any lambdas that do not match the observed behavior of the hidden rule. Figure 1 illustrates the rate at which different models reduce the hypothesis space over successive turns. Models with worse reasoning spend more attempts to clear less of the hypothesis space.Figure 1 : Models can succeed upon this task by reducing the hypothesis space quickly or providing useful tests for many turns. We show that models with strong reasoning capabilities can narrow the space quickly, but weaker multi-turn capability harms their ability to get value out of later tests.To determine how well the models employ Occam's Razor, we explore different metrics to gauge whether the models find the simplest rule that covers the examples. From existing Bayesian models of cognition [32, 33] , the size principle uses hypothesis size as a measure of simplicity. In these Bayesian models, hypothesis size is calculated as the number of values that match the hypothesis. Calculating hypothesis size in this manner is only possible when the test values are within a limited countable range. In our case, the possible test values are infinite, requiring some alternative metrics to gauge hypothesis size. We use three metrics:1. Number of Operators: We count the number of operators used in the rule expression. 
discussion: 
 In our experiments, we show that LLMs struggle substantially with this task. Specifically, their propensity to repeat test cases, propose useless test cases, and guess very unlikely rules harms their performance on this task substantially. The varying performance in a multi-turn setting represents a previously underappreciated dimension of measuring reasoning capability in LLMs. There is much work in language modeling for code-based agents [7] [43] and LLM-driven unit testing [44] , and the difficulty of LLMs to explore edge cases effectively has substantial implications on those applications.With this work, we aim to provide a benchmark that measures a model's capacity for exploring an environment and reasoning based on its own decisions across multiple turns. We believe that this paradigm offers a more direct measurement of reasoning capability compared to other benchmarks. By focusing specifically on how LLMs handle multi-turn reasoning, we better understand their most common real-world applications.A.1 Experimental DetailsModels are accessed when possible with default configurations and settings provided by their respective proprietary APIs (i.e. Anthropic, OpenAI, Mistral, DeepSeek, and Vertex APIs). Llama 3.1 models are run via the Amazon Bedrock API. Gemma 2 9b and Llama 3 70B are run via the Groq API [1] .All models begin from the same initial prompt.You are an AI tasked with discovering a hidden rule that takes three numbers as input and returns a boolean. These numbers can be any float or integer number that can be typed in 3 or fewer digits, not including signs and decimal characters (e.g. -11.3 is a valid value). In each response, you will provide your step-by-step reasoning for your future plans on this task, and then you will take one of two actions:1. Provide a single test case to observe the output of a rule (e.g."Test Case: ```(1, 2, 3)```" which corresponds to x=1, y=2, z=3. You may only propose one test case at a time. 2. Provide a final "guess" in the form of a Python lambda function (e.g."Final Guess: ```lambda x,y,z: x < y < z```"). You only get to select this action one time, so don't use it unless you are confident or are forced to.You will have up to 30 attempts at test cases, and you will have exactly one attempt at guessing the final function. Make sure all of the tests you provide are unique from other ones you have already tried -the functions are all deterministic, and will be the same every time. You should prioritize getting the rule as correct as possible, since an incorrect answer will get no points. You should therefore not guess the function unless you are quite confident, or if you are very close to running out of attempts.You must return your responses in the format laid out above at the very bottom of your message. For example, if you want to submit a test case, you must conclude with the string "Test Case: ```(x,y,z)```", where x,y,z are replaced with your guesses. If you want to submit a final guess, you must conclude with the string "Final Guess: ```<function>```" where <function> is replaced with a Python lambda function. Do not include any comments or additional text on the same lines as these two things.Make sure to include your reasoning for your tests -what you are testing for, why you selected that test, etc.Responses by the models are pulled out via regular expressions matching the formatting in the prompt. We find that after controlling for various formatting eccentricities (Python blocks, markdown, bold characters, etc) that all listed models are capable of providing test cases in this format.To verify two provided lambda functions are equivalent, we generate a large number of test cases and ensure the provided rules match on all outputs. Specifically, we create three sets of cases:1. Integer Grid Cases -We construct a 40x40x40 grid of integer triplets from -20 to 20, inclusive, leading to 64,000 triplet cases.2. Random Uniform Cases -We construct a list of 10,000 uniformly random float triplets from -200 to 200, inclusive.3. Special Cases -We hand-design a small set of test cases to ensure all hidden rules in the full split are adequately tested.We mark a rule as incorrect if any test cases generated above show different behavior between the hidden rule and the guessed rule, and mark it correct otherwise.To succeed at the WILT task, models must succeed at both gathering evidence (hypothesis reduction) and drawing logical conclusions from evidence (function inversion). To distinguish a model's ability to do one or the other, we perform an experiment where models attempt to guess the rule using tests from another model. Rather than asking the model to gather evidence, we directly provide it all the reasoning-stripped3 input-output pairs generated by another model for the same rule, and ask the model to guess the rule in a single turn. Without the original reasoning and subsequent observation before and after each test case, we expect most models to underperform relative to the full test even when provided their own cases. Likewise, we expect models stronger at single-turn to perform better in this experiment relative to other models subject to the same evidence. Our results can be found in Figure 2 .This reveals some notably varied capabilities among the top performing models. While Claude Sonnet 3.5 was the narrowly highest performing model on the full test, this experiment reveals important context for why that may be. We see that it performs better than most other models subject to the same evidence, but proposes test cases that are generally slightly less useful for other models without the attached justifications. Likewise, without its own reasoning for each case, Sonnet's performance degraded substantially more than other models in the same setting, suggesting a larger component of its success was its reasoning, compared to the test cases alone.o1-mini shows highly superior single-turn capability in this test, but notably performs relatively less well when provided its own tests rather than the tests of another high-performing model. When paired with cases from chatgpt-4o-latest, it successfully guessed 19 of the 50 rules, far surpassing the best-performing single model in the full test.Despite having many repeated tests and messages which were generally similar to each other (see Tables 1 and 3 ), we see that Mistral Large performs well with other models' tests and provides a corpus of tests useful to other models. We note its comparable performance to chatgpt-4o-latest both along the rows (model's performance with other model's test cases) and columns (performance of other models with the model's test cases), reinforcing its strong performance in the full test.Critically, we show that models have non-identical strengths and weaknesses, and that success on the full WILT task depends on strong performance on a few key metrics of interest. Even without the attached reasoning for test cases, composing the test case generation of one model and the function inversion of another model very often outperforms using a single strong model for both subtasks. This has some notable implications for future LLM applications: in [37] it was shown that several language models coordinated by an aggregator LLM could outperform strong single models. Future work could explore coordinating models for both single-turn and multi-turn oriented tasks, potentially leading to improved performance.Figure 2 : Models have varying success when using test cases proposed by other models. o1-mini stands out as having much stronger single-turn reasoning in this experiment, but it performs poorly with its own tests. Table 2 shows the complexity of each model by the three aforementioned metrics.In Figure 3 we show the set inclusion ratios in the case where a model is provided another model's test cases. That is, we show whether an error in the final guess of a model is likely to be smaller / less than one (e.g. x < y < z instead of x ≤ y ≤ z), or larger / greater than one (e.g. x > 0 instead of x > 0 ∧ x < 5). This seems more test-case dependent than other complexity benchmarks, where tests provided by certain models seem to lead to smaller hypotheses.Figure 4 shows the length of the string used to guess the rule by each model, which is comparatively more consistent for a model. We find this to be fairly consistent across settings, with most models hovering near 45. Figure 5 shows the median number of operators used by a model when given another model's test cases, as in Section A.2. This can be used to estimate the model's bias toward guessing a simpler rule. o1-preview, for example, tends to use fewer operators than the other models subject to the same evidence. This also highlights potential discrepancies in the complexity resulting from a model's test cases and the complexity of its final guesses. When models use test cases generated by DeepSeek Chat v2.5, they tend to use fewer operators, likely because the test cases are fully encompassed by simple rules like lambda x,y,z: False. Conversely, when given other models' tests, DeepSeek Chat v2.5 responds with a high level of complexity compared to other models. Its guesses often overfit a complicated rule to the test cases (e.g., it guesses a rule of lambda x, y, z: y == x or y == z or y == (x + z) / 2 when given the following true test cases for even numbers: (2.0, 2.0, 2.0), (2.0, 2.0, 4.0), (-2.0, 4.0, 6.0), (0.0, 2.0, 4.0), (2.2, 4.0, 6.0), (2.0, 4.0, 6.0) .)Test case novelty is an interesting second order metric for success upon the WILT task. Broadly speaking, models that reuse fewer tests are rewarded with more information for which to solve the task. Models that very rarely re-propose a test tend to perform very well upon WILT, but the inverse is not necessarily true -models that loop tests often still arrive at the right answer.Repeated tests are useful for bifurcating the types of failures on WILT -one being the doom loop phenomenon, and the other being reasoning capability conditioned upon some available evidence. One hypothesis for the observed behavior is that certain models are primarily oriented towards single-turn scenarios, and that one type of failure need not imply the other. DeepSeek Chat v2.5, for example, demonstrates strong initial hypothesis space reduction compared to other models, which allows it notably better performance on WILT compared to other models with similar repeat counts (e.g. open-mistral-nemo). Strong single-turn performance and deductive reasoning capabilities can help salvage performance from a model that demonstrates difficulty with multi-turn inductive logic.Following Aidan-bench [18] , we provide Table 3 , containing additional novelty metrics. These include:1. Average novelty -which reports the average cosine similarity between each message's gpt-3 embeddings and the closest previous message within the same test. 2. Average minimum novelty -which reports the average minimum cosine similarity between each message's gpt-3 embeddings and the closest previous message within the same test.These capture an additional dimension of "test novelty" and "message novelty", where models may propose the same tests for different reasons, or repeat previously generated messages verbatim. We bold results which are best within the class of high performing models. We note that models that propose fewer tests before guessing (e.g. o1-preview, o1-mini) should see lower values for all of these compared to models that tend to use many tests before guessing (e.g. mistral-large) even for otherwise equally performing models. We also show the average novelty by turn in Figure 6 , which captures the model's novelty scores across turns. Table 4 contains selected results for the lite split. We show this much easier split produces a similar ordering, suggesting that the bulk of the separation in our benchmark lies in the easier tests. x < y < z 3x ≥ y ≥ z 4x ≤ y ≤ z 5x = y = z 6x ̸ = y ∧ y ̸ = z ∧ x ̸ = z 7x < 0 ∧ y < 0 ∧ z < 0 8x + y = z 9x • y = z 10x < y ∧ y > zB.1 Doom Loop Without Reasoning Failure Table 7 shows an example of a model (Deepseek-Chat-v2.5) entering a doom loop during the test proposal phase, but where that doom loop does not constitute a reasoning failure. Reasoning has been removed for brevity.Table 8 shows an example of o1-mini getting a very difficult test case (co-primality) approximately correct, failing only because it adds an additional arbitrary constraint upon the magnitude of the values despite no such constraint existing. Reasoning has been removed for brevity.Table 9 shows an example of o1-preview failing a relatively easy test case (x ≥ y ≥ z) due to a confirmation bias error. The model uses only 9 test cases and correctly identifies that the rule returns true when all three are equal, but submits five test cases confirming that and none exploring other rules which are true when three items are equal. Reasoning has been removed for brevity. x > y > z 2x < y < z 3x ≥ y ≥ z 4x ≤ y ≤ z 5x < z < y 6x ≤ z ≤ y7 z < x < y 8 z ≤ x ≤ y 9 x = y = z 10 x ̸ = y ∧ y ̸ = z ∧ x ̸ = z 11 x < 0 ∧ y < 0 ∧ z < 0 12 x > 0 ∧ y > 0 ∧ z > 0 13x mod 2 = 0 ∧ y mod 2 = 0 ∧ z mod 2 = 0 14x mod 2 ̸ = 0 ∧ y mod 2 ̸ = 0 ∧ z mod 2 ̸ = 0 Medium Tests 15 x + y = z 16 x • y = z 17 x + z = y 18 x • z = y 19 y + z = x 20 y • z = x 21 max(x, y, z) = x 22 max(x, y, z) = y 23 max(x, y, z) = z 24 min(x, y, z) = x 25 min(x, y, z) = y 26 min(x, y, z) = z 27 x + y + z = 0 28 x • y • z = 0 29 (x + y + z) mod 2 = 0 30 (x + y + z) mod 2 = 1 31 (x • y • z) mod 2 = 0 32 (x • y • z) mod 2 = 1 33 (x + y)/2 = z 34 -5 ≤ x,= ⌊|x|⌋ ∧ ⌊|y|⌋ 2 = ⌊|y|⌋ ∧ ⌊|z|⌋ 2 = ⌊|z|⌋ 48 0 < x mod 1 ∧ 0 < y mod 1 ∧ 0 < z mod 1 49 0 < x mod 1 < y mod 1 < z mod 1 < 1 50 x < y < z ∧ 0 < z -x ≤ 1Final Guess: lambda x, y, z: all(isinstance(n, int) and n ̸ = 0 and -3 ≤ n ≤ 3 for n in [x, y, z]) and math.gcd(x, y) == 1 and math.gcd(x, z) == 1 and math.gcd(y, z) == 1 Sorry, that's not the correct rule. Table 9 : o1-preview fails early on an easy test case despite having a wealth of remaining tests, because it latched on to an early behavior and sought to confirm it rather than falsify it. 38th Conference on Neural Information Processing Systems (NeurIPS 2024).For example, x -y = z is equivalent to y + z = xWe bold results for this column only if the models are also high performing. For example, ClaudeHaiku uses fewer guesses than o1-preview, but this is because it is failing.We strip reasoning to avoid conflating confirmation bias in the attached reasoning, rather than just the accumulated evidence. 
 
 Appendix:  



Reviews: review: title: title: Review
main_review: main_review: #### Summary:
The paper introduces WILT, a multi-turn reasoning benchmark inspired by the Wason 2-4-6 task, designed to test inductive reasoning capabilities of LLMs. The benchmark focuses on multi-turn interactions where models must infer a hidden boolean function by proposing test cases. WILT aims to be robust against memorization and emphasizes inductive and deductive reasoning, hypothesis space reduction, and susceptibility to confirmation bias. The paper demonstrates that existing LLMs struggle with this task and provides empirical results comparing the performance of several models, highlighting significant differences in reasoning abilities across them.

#### Strengths:
1. **Interesting Problem**: The paper tackles a novel problem of multi-turn inductive reasoning that hasn't been extensively studied in previous benchmarks. By modeling the benchmark after the classic Wason 2-4-6 task, it introduces a challenging reasoning scenario that is particularly difficult for LLMs, pushing the boundaries of reasoning in AI systems.
  
2. **Sufficient Number of Models for Evaluation**: The authors evaluate a wide range of models, from Claude and GPT-4 variants to Mistral and Gemini models, providing a comprehensive analysis of their performance across different reasoning metrics such as hypothesis space reduction and response complexity.

#### Weaknesses:
1. **Lack of Insights**: While the paper does a good job of presenting the empirical results, it falls short in providing deeper insights or theoretical analyses of why models struggle with WILT. A more detailed discussion on the nature of the failures and what specific aspects of model architectures or training regimes contribute to their performance deficits would significantly strengthen the paper's contributions.

2. **Absence of Proven Prompting Methods**: The paper doesn't incorporate or experiment with prompting techniques such as chain-of-thought (CoT) reasoning, which has been shown in prior works to improve reasoning tasks in LLMs. Including these methods in the evaluations would have provided a more thorough assessment of LLM reasoning capabilities and potentially enhanced model performance on WILT.
title: title: a significant step forward in the design of benchmarks for evaluating the reasoning capabilities of LLMs in multi-turn scenarios.
main_review: main_review: The paper introduces the Wason Inductive Logic Test (WILT), a new benchmark for large language models (LLMs) focused on multi-turn, memorization-resistant inductive logic. This benchmark, inspired by the classical Wason 2-4-6 task, challenges LLMs to infer a hidden boolean function through up to thirty test cases. The models begin with no initial clues and must propose tests to discover the underlying rule, encoded as a Python lambda function. This setup aims to evaluate and enhance LLMs' capabilities in multi-turn reasoning and hypothesis testing, areas where current models often struggle.

Strengths:

1.	WILT is designed to be robust against memorization, requiring models to actively participate in a reasoning process rather than recalling fixed responses, addressing a critical limitation in current LLM evaluations.

2.	The benchmark assesses multiple aspects of cognitive reasoning, including the application of Occam's Razor, avoidance of repetitive loops, and efficient navigation of large hypothesis spaces.

3.	By focusing on multi-turn interactions, WILT mirrors realistic use-cases of LLMs, making its findings applicable to practical settings where LLMs must understand and manipulate complex, evolving scenarios.

Weaknesses:

1.	The benchmark's reliance on synthetic, logical constructs might not adequately represent the variety of real-world logic and reasoning challenges, potentially biasing the evaluation towards tasks that are well-suited to current AI capabilities rather than pushing the boundaries of what LLMs can achieve.

2.	The use of approximations and subjective judgments in determining whether a model's response is "approximately correct" introduces ambiguity into the assessment process, which could affect the benchmark's reliability and the consistency of its outcomes across different studies.
title: title: Review
main_review: main_review: This paper proposes a new benchmark called the Wason inductive logic test (WILT) for testing the multi-turn reasoning ability of large language models.  Intuitively, this can be seen as an interactive inductive logic programming task that a language model can feed a test case to the environment and get back a response saying yes or no, at the final stage the model will guess the hidden rule of this logic test. 

This task is very interesting because it tests the ability of the language model to do induction. The result of this benchmark is also significant, we can see that almost none of the current LLM can do this task well. 

Overall, I think this is a nice work and strongly recommend to accept it, but I still have one question: 

About *Occam's Razor*: It is quite natural that there are multiple rules that can satisfy the same observations in a limited number of tries, so in this case, what if the LLM returns a plausible explanation that satisfies all the test cases they have quried, but still not 100% same with the underlying ground truth? How can we measure which rule is simpler?
title: title: Official Review of Submission9 by Reviewer YxGM
main_review: main_review: **Summary:** The authors propose a new benchmark dataset: WILT. Each task in this dataset is a Boolean algebra expression and the model needs to infer what this expression is by querying some oracle. Models’ responses are then rated through a host of complexity metrics: set inclusion, response length, number of operations. The authors demonstrate that WILT is difficult for existing LLMs and claim that it is inherently susceptible for memorization or overfitting. 

**Quality:** 3 good. 

**Clarity:** 3 good. From the writing itself, this reviewer can understand the main ideas of the paper, the experimental setup as well as the motivation of the paper.

**Originality:** 3 good. Though there are existing interactive benchmarks for reasoning (such as twenty questions from BIG bench), this task is relatively novel.

**Significance:** 3 good. The idea proposed by the authors is a general one. This reviewer can easily see this setup being generalized to more complex tasks and settings.

**Pros:**
- The behaviors of interest that the authors identify are cogent. They capture many fundamental processes of inductive reasoning. 
- Though the size of the dataset is small, this reviewer can easily imagine how this can be fixed by simply crafting more lambda expressions. 
- The authors benchmark their dataset against a wide range of the newest models providing solid experimental evidence that this benchmark is useful. 

**Cons:**
- Though this reviewer understands the page length is limited, a review of the related literature is lacking. Specifically, there are little comparisons with existing interactive reasoning benchmarks. In this reviewer’s opinion, the setup here is analogous to a hybrid of tool use benchmarks (where the LLM understand how to craft useful queries to the oracle) [1] and interactive reasoning benchmarks (understand how to update its world view given new information) [2]. It would be nice to discuss this more. 

[1] https://arxiv.org/pdf/2310.03128

[2] https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/twenty_questions