InductionBench: LLMs Fail in the Simplest Complexity Class
Authors: Wenyue Hua, Tyler Wong, Fei Sun, Liangming Pan, Adam Jardine, William Yang Wang
Accepted at: acl25

InductionBench: LLMs Fail in the Simplest Complexity Class

Wenyue Hua1 Tyler Wong1 Sun Fei1
Liangming Pan3 Adam Jardine4 William Yang Wang1*
1University of California, Santa Barbara, 2Independent Researcher
3University of Arizona, 4Rutgers University, New Brunswick

Abstract

Large language models (LLMs) have shown remarkable improvements in reasoning and many existing benchmarks have been addressed by models such as o1 and o3 either fully or partially. However, a majority of these benchmarks emphasize deductive reasoning, including mathematical and coding tasks in which rules such as mathematical axioms or programming syntax are clearly defined, based on which LLMs can plan and apply these rules to arrive at a solution. In contrast, inductive reasoning, where one infers the underlying rules from observed data, remains less explored. Such inductive processes lie at the heart of scientific discovery, as they enable researchers to extract general principles from empirical observations. To assess whether LLMs possess this capacity, we introduce InductionBench, a new benchmark designed to evaluate the inductive reasoning ability of LLMs. Our experimental findings reveal that even the most advanced modelw available struggle to master the simplest complexity classes within the subregular hierarchy of functions, highlighting a notable deficiency in current LLMs’ inductive reasoning capabilities. Code and data are available https://github.com/wenyueh/inductive_reasoning_benchmark.

1 Introduction

The remarkable progress of large language models (LLMs) in recent years has yielded substantial improvements in their reasoning capabilities. This progress is most evident in benchmarks involving complex mathematics (Cobbe et al., 2021; Hendrycks et al., 2021) and coding tasks (Jain et al., 2024; Jimenez et al., 2023; Chen et al., 2021; Fan et al., 2023). Beyond these domains, researchers have also explored the logical reasoning abilities of LLMs from various angles, including propositional logic (Zhu et al., 2023), first-order logic (Han et al., 2022; Parmar et al., 2024), and propositional logic under different contexts (Hua et al., 2024).

![Diagram comparing deductive and inductive reasoning](page_1016_370_388_246.png)

Figure 1: Deductive vs. Inductive Reasoning

Despite significant progress in model capabilities, existing benchmarks predominantly focus on deductive reasoning, largely overlooking inductive reasoning. The former requires applying explicitly defined premises to derive valid conclusions, whereas the latter requires inferring the underlying principles, rules, or patterns from observations (Hawthorne, 2004). Both forms of reasoning are essential; inductive reasoning, in particular, is critical in domains such as scientific discovery where researchers seek to characterize natural laws based on empirical data (Grünwald, 2007; Hansen and Yu, 2001) that captures complex phenomena. Figure 1 illustrates the differences between inductive and deductive reasoning.

In this paper, we address this gap by introducing InductionBench, a rigorous benchmark designed to assess LLMs’ inductive reasoning abilities by testing whether they can infer a string-to-string transformation from a finite set of input–output pairs. A model must hypothesize the underlying relationship between inputs and outputs based on a finite set of examples and then extrapolate those rules to unseen strings. The process of discovering the underlying function from limited data reflects the core principles of inductive reasoning.

Our benchmark is grounded in the subregular
hierarchy (Rogers and Pullum, 2011; Truthe, 2018; Graf, 2022; Jäger and Rogers, 2012; Heinz, 2018) (see Figure 2) for string-to-string mappings (Mohri, 1997), focusing on input transformations restricted to regular functions. By systematically increasing task complexity across multiple classes in the sub-regular hierarchy, we gain detailed insights into how effectively LLMs detect, hypothesize, and generalize underlying rules from theoretically sufficient datapoints.

We evaluate multiple state-of-the-art LLMs to understand LLM’s inductive reasoning ability and identify factors that increase the difficulty of inductive reasoning tasks for LLMs, such as the length of the minimum-length description, the number of datapoints, and in-context examples. Through extensive experiments, we find that even advanced models such as o3-mini struggle with basic inductive tasks, highlighting a significant shortcoming in the current generation of LLMs. More detailed findings are presented in Section 5.

2 Related Work

Deductive Reasoning. One major branch of reasoning benchmarks centers on deductive inference, where models apply established premises to derive specific conclusions. Notable examples include ReClor (Yu et al., 2020), which evaluates the ability to solve logical reasoning questions resembling those found in standardized tests, and various logic-based benchmarks of increasing complexity, from propositional logic to first-order logic (Han et al., 2022; Parmar et al., 2024; Zhu et al., 2023; Hua et al., 2024). These tasks typically require handling structured logical relationships with minimal ambiguity in how premises lead to conclusions.

Another type of reasoning benchmarks is mathematical problem solving, including elementary arithmetic to advanced competition-level questions. Hendrycks et al. (2021) test both computational skills and the sequential reasoning steps involved in mathematics. Cobbe et al. (2021) covers a broad spectrum of topics, including geometry and higher-level problem solving. However, most standard mathematics problem-solving tasks can be framed as deductive reasoning, as they involve applying established axioms, definitions, and theorems in a logically valid sequence to derive a conclusion.

Inductive Reasoning. Despite the diversity of existing benchmarks, inductive reasoning, where models hypothesize and generalize patterns from examples without pre-specified rules, remains comparatively underexplored. Current evaluations of inductive skills have largely been limited to small-scale symbolic regression, artificial language translation, and concept learning (Liu et al., 2024b; Lake et al., 2019; Qiu et al., 2023), which, although important in real-world scenarios, often lack three key elements: (1) an explicit analysis of the inherent difficulty of the task (2) a guarantee that the provided input–output dataset can identify the target function (3) a mechanism to evaluate whether models can identify the “best possible hypothesis” under Occam’s Razor (Blumer et al., 1987; Baker, 2007) principle, i.e., a description with minimal length (Hansen and Yu, 2001; Grünwald, 2007).

Our Contribution. To address these shortcomings, we introduce a new benchmark targeting on inductive reasoning skills. Building on subregular hierarchy and corresponding polynomial time and data learnability guarantees, our benchmark, InductionBench, tests how effectively LLMs infer underlying transformation functions from finite datapoints. We also measure the degree to which models produce minimal, non-redundant hypotheses, providing a lens into their ability of generalization. Through a fine-grained, gradually increasing level of complexity, our evaluations reveal how current LLMs cope with the growing search space. There are several advantages of our benchmark: (1) Automated Evaluation: Because the data is derived from well-defined functions, one can directly compare the model’s output with the known ground-truth function, eliminating the need for expensive human annotations. (2) Dynamic Data Generation: The dataset is produced randomly based on specific function classes, allowing periodic “refreshes” to prevent models from relying on memorized examples. (3) Rigorous Assessment of Hypothesis Space: As the function is well-defined,

![Subregular hierarchy in string-to-string maps](page_186_120_527_312.png)

Figure 2: Subregular hierarchy in string-to-string maps
one can control the size of the hypothesis space with precision. This control enables a rigorous and systematic evaluation of LLM performance from a theoretically grounded perspective.

3 Computational Complexity in Inductive Reasoning

InductionBench uses string-to-string transformation/functions as a proxy to study inductive reasoning, which has established computational complexity hierarchy (Roche and Schabes, 1997; Engelfriet and Hoogeboom, 2001). We focus on the subregular hierarchy, the hierarchy under regular functions. Though with limited expressive power, our experiments show that these classes already present substantial challenges for LLMs.

Specifically, we limit our attention to three classes of deterministic regular functions—Left Output-Strictly-Local (L-OSL), Right Output-Strictly-Local (R-OSL), and Input-Strictly-Local (ISL), whose positions in the subregular hierarchy are illustrated in Figure 2 (Heinz, 2018). These classes represent the lowest-complexity tier for string-to-string mappings within the subregular hierarchy. They are proper subclasses of subsequential function class and, more broadly, of weakly-deterministic class and non-deterministic class, which are themselves subsets of the regular function classes. Although we do not elaborate on the complete regular function hierarchy here, it is important to note that the ISL, L-OSL, and R-OSL classes are among the simplest in this framework.

Strictly local functions can be seen as operating with a fixed amount of look-ahead, similar to Markov processes. They are provably learnable in polynomial time from polynomially sized samples (Chandee et al., 2014; De La Higuera, 1997; Chandee et al., 2015; Jardine et al., 2014). Moreover, prior work has shown that an algorithm exists to learn the unique (up to isomorphism) smallest subsequential finite-state transducer that represents such ISL, L-OSL, R-OSL functions (Satta and Henderson, 1997; Arasu et al., 2009). This property allows us to evaluate not only whether LLMs can discover the correct patterns but also whether they can identify the simplest or most concise representation consistent with the data.

3.1 Preliminary

Before providing the definitions of the three function classes, we first introduce the fundamental mathematical notations and formal definitions underpinning our discussion of string-to-string transformations and their properties.

Let \( \Sigma \) be a finite alphabet. We denote by \( \Sigma^* \) the set of all finite strings over \( \Sigma \), and by \( \Sigma^{<n} \) the set of all strings over \( \Sigma \) of length at most \( n \). The empty string is denoted as \( \lambda \). The set of prefixes of a string \( w \) is denoted as \( \mathrm{PREFIX}(w) \), defined as \( \{ p \in \Sigma^* \mid \exists s \in \Sigma^* . s.t. w = ps \} \), and the set of suffixes of \( w \) denoted as \( \mathrm{SUFFIX}(w) \), defined as \( \{ s \in \Sigma^* \mid \exists p \in \Sigma^* . s.t. w = ps \} \). The longest common prefix of a set of strings \( S \) is denoted as \( \mathrm{LCP}(S) \), defined as

\[
p \in \cap_{w \in S} \mathrm{PREFIX}(w) \text{ such as } \forall p' \in \cap_{w \in S} \mathrm{PREFIX}(w), |p'| < |p|.
\]

For any function \( f : \Sigma^* \to \Gamma^* \) and \( w \in \Sigma^* \), let the tails of \( w \) with respect to \( f \) be defined as

\[
\mathrm{TAILS}_f(w) = \{ (y, v) \mid f(wy) = uv \text{ and } u = \mathrm{LCP}(f(w\Sigma^*)) \}.
\]

Intuitively, \( \mathrm{TAILS}_f(w) \) collects all possible continuations \( (y, v) \) by appending \( y \) to \( w \). It summarizes how \( f \) might extend beyond the partial input \( w \). The total number of distinct tails across all strings in \( \Sigma^* \) provides a measure of how many different non-trivial local transformation \( f \) encodes.

3.2 Function Class Definition

Based on the concepts outlined above, we define the three function classes.

Definition 1 (ISL) A function \( f \) is ISL if there is a \( k \) such that for all \( u_1, u_2 \in \Sigma^* \), if \( \mathrm{SUFFIX}^{k-1}(u_1) = \mathrm{SUFFIX}^{k-1}(u_2) \), then \( \mathrm{TAILS}_f(u_1) = \mathrm{TAILS}_f(u_2) \).

![A diagram showing the ISL definition, with input and output strings and a marked segment labeled k-1.](page_1016_1342_377_120.png)

Figure 3: ISL definition

In simpler terms, this means that the output at each position in the string depends only on the preceding \( k - 1 \) characters of the input, making the transformation Markovian with respect to the input. Figure 3 illustrates this definition. Below is a simple example:

Example 3.1 Suppose a function \( f : \{a, b\}^* \to \{a, b\}^* \) rewrites each \( b \) to \( a \) only if it appears after
the input substring ba. In this scenario, we have \( k = 3 \), and there are two distinct tails:

\[
\text{TAILS}_f(w) = \{(\lambda, \lambda), (b, a), (bb, ab), (ab, ab)... \}
\]
\[
\forall w \in \Sigma^* \text{ such that } ba \in \text{SUFF}(w)
\]

and

\[
\text{TAILS}_f(w') = \{(\lambda, \lambda), (a, a), (bb, bb), (ab, ab)... \}
\]
\[
\forall w' \in \Sigma^* \text{ such that } ba \notin \text{SUFF}(w')
\]

These tails indicate how the function’s behavior shifts depending on whether the immediate context ends in \( ba \). Such context-dependent tails also highlights that ISL functions can be effectively characterized or represented by local input constraints.

Definition 2 (L-OSL) A function \( f \) is L-OSL if there is a \( k \) such that for all \( u_1, u_2 \in \Sigma^* \), if \( \text{SUFF}^{k-1}(f(u_1)) = \text{SUFF}^{k-1}(f(u_2)) \), then \( \text{TAILS}_f(u_1) = \text{TAILS}_f(u_2) \).

![L-OSL definition](page_682_1047_591_120.png)

Figure 4: L-OSL definition

In other words, the output at each position in the transformed string depends only on the preceding \( k - 1 \) characters of the output itself, rather than on the input. This property can be understood as a form of Markovian process on the output. Below is a simple example:

Example 3.2 Suppose a function \( f \) rewrites each \( b \) to \( \lambda \) only if it appears after the output substring \( ba \). In this scenario, we have \( k = 3 \), and there are two distinct tails:

\[
\text{TAILS}_f(w) = \{(\lambda, \lambda), (a, a), (b, \lambda),
(bb, \lambda), (ab, ab), (ba, a)... \}
\]
\[
\forall w \in \Sigma^* \text{ such that } ba \in \text{SUFF}(f(w))
\]

and

\[
\text{TAILS}_f(w) = \{(\lambda, \lambda), (a, a), (b, b),
(bb, bb), (ab, ab), (ba, ba)... \}
\]
\[
\forall w \in \Sigma^* \text{ such that } ba \notin \text{SUFF}(f(w))
\]

While L-OSL depends preceding output symbols to the “left”, R-OSL functions depends on a limited number of future output symbols to the “right”.

Conceptually, one can view R-OSL as analogous to L-OSL, except that the input is processed in reverse order. Although both belong to the broader OSL paradigm, they are incomparable classes: each can express transformations the other cannot. The formal definition of R-OSL follows:

Definition 3 (R-OSL) A function \( f \) is R-OSL if there is a \( k \) such that for all \( u_1, u_2 \in \Sigma^* \), if \( \text{SUFF}^{k-1}(f(u_1^{-1})) = \text{SUFF}^{k-1}(f(u_2^{-1})) \), then \( \text{TAILS}_f(u_1^{-1}) = \text{TAILS}_f(u_2^{-1}) \).

Intuitively, this class of functions can be viewed as a rightward Markovian process on the output. Each output symbol is determined not by the preceding symbols as in L-OSL but by the next \( k - 1 \) symbols that will appear in the output.

The three classes, ISL, L-OSL, and R-OSL, are each deterministic and exhibit Markovian behavior, yet remain pairwise incomparable within the broader subregular hierarchy. In this work, we further restrict our attention to functions that involve substitution which replaces one character with another and deletion which maps a character to the empty string \( \lambda \).

3.3 Learnability

The three function classes are identifiable in polynomial time using a polynomially sized characteristic sample (Chandlee et al., 2014, 2015). In other words, there exists a polynomial-time algorithm that, given sufficient data for a target function \( f \), can produce a representation \( \tau \) that satisfies \( f(w) = \tau(w) \) for every \( w \in \Sigma^* \). In other words, once sufficient data is presented, one can reliably recover a function equivalent to \( f \) on all possible inputs. This learnability property underpins the value of these classes as testbeds for inductive reasoning, since the data requirement remains polynomial and successful inference is theoretically guaranteed.

We formalize “sufficient data” as the minimal set of input–output pairs needed to learn a \( k \)-strictly local function \( f \), which is known as characteristic sample. Adapting the original definition\(^1\) for clarity (Chandlee et al., 2014, 2015), we define:

\(^1\) simplified from original definition
Definition 4 (Characteristic Sample) For a given k-ISL \( f \), the characteristic sample \( S \) is defined as \( \{ (w, w') \mid w \in \Sigma^{\leq k} \land f(w) = w' \} \). For a given k-OSL \( f \), the characteristic sample \( S \) is defined as \( \{ (w, w') \mid w' \in \Sigma^{\leq k} \land f(w) = w' \} \).

If a provided dataset contains such characteristic sample, a learning algorithm can reconstruct a representation of \( f \) that matches its behavior on every string in \( \Sigma \). Accordingly, in the context of LLMs, we expect that providing this dataset as in-context examples should enable the model to induce the underlying string-to-string mapping.

For example, the characteristic sample with the transformation function being an ISL-3 function as described in Example 3.2: \( f : \{a, b\}^* \to \{a, b\}^* \) rewrites each \( b \) to \( a \) only if it appears after the input substring \( ba \) is: \( \{(\lambda, \lambda), (a, a), (b, b), (aa, aa), (ab, ab), (ba, ba), (bb, bb), (aaa, aaa), (aab, aab), (aba, aba), (abb, abb), (baa, baa), (bab, baa), (bba, bba), (bbb, bbb)\} \).

3.4 Unique Function Representation

Beyond verifying that a model can accurately discover a function from data, we also investigate how succinctly the model describes its inferred rules. This aspect is of both theoretical and practical interest: a minimal or most concise representation not only offers interpretability advantages but can also reflect the model’s capacity for truly generalizable, rather than merely enumerative, learning.

One function can be represented or written in a non-unique way. For instance, consider an ISL function \( f_1 \) with \( k = 2 \) over \( \Sigma = \{a, b\} \) that maps the input character \( a \) to \( b \) when it comes after \( b \), that rewrites each \( a \) to \( b \) only if the preceding character is \( b \), while leaving other substrings unchanged. One concise description is:

\[
f_1(w) =
\begin{cases}
f_1(w_1)ba^{-1}f_1(aw_2), & \text{if } w_1 \text{ ends with } b \text{ and } w=w_1 \\
aw_2 \text{ for some } w_1, w_2 \in \Sigma^* \\
w, & \text{otherwise}
\end{cases}
\]

An alternative yet more verbose description of the same function might redundantly enumerate multiple cases:

\[
f'_1(w) =
\begin{cases}
f'_1(w_1)ba^{-1}f'_1(aw_2), & \text{if } w_1 \text{ ends with } ab \text{ and } w = w_1aw_2 \text{ for some } w_1, w_2 \in \Sigma^* \\
f'_1(w_1)ba^{-1}f'_1(aw_2) & \text{if } w_1 \text{ ends with } bb \text{ and } w = w_1aw_2 \text{ for some } w_1, w_2 \in \Sigma^* \\
w, & \text{otherwise}
\end{cases}
\]

Although these two representations encode the same function, the second contains repetitive conditions and fails to emphasize that the output of \( f_1 \) depends solely on the single preceding character instead of the penultimate character.

Because these functions admit a unique minimal representation (up to isomorphism) (Chandee et al., 2014; Oncina and Garcia, 1991), we can directly compare the function produced by an LLM to the ground-truth minimal form. In doing so, we evaluate whether the model not only discovers the correct transformation but also simplifies it to the most parsimonious description possible—an essential indicator of robust inductive reasoning.

3.5 Rule-based Representation

To streamline the generation and parsing of function representations, we employ a simplified notation wherein each transformation is written as “condition \( \circ \) target character \( \rightarrow \) output of the target character” (Bird and Ellison, 1994). In this notation, the condition represents the minimal substring needed to trigger a transformation, while any input substring not matching this condition remains unchanged. For instance, in the earlier example, this approach permits a concise notation \( b \circ a \rightarrow b \), indicating that the input \( a \) is mapped to \( b \) when it comes after \( b \); otherwise, the input string remains unaltered. This concise, rule-based format simplifies both the model’s output generation (by reducing complex functional descriptions) and our subsequent evaluation, as the applicable transformations can be easily parseable and verified.

To demonstrate the simplicity of rule-based representation: given an ISL function \( f_2 \) with \( k = 2 \), the input \( a \) becomes \( b \) when it comes after \( b \) and two consecutive \( a \)s will be reduced to one single \( a \).
The minimal function representation is as below:

\[
f_2(w) = \begin{cases}
f_2(w_1)ba^{-1}f_2(aw_2), & \text{if } w_1 \text{ ends with } b \text{ and } w=w_1aw_2 \text{ for some } w_1, w_2 \in \Sigma^* \\
f_2(w_1)a^{-1}f_2(aw_2), & \text{if } w_1 \text{ ends with } a \text{ and } w=w_1aw_2 \text{ for some } w_1, w_2 \in \Sigma^* \\
w, & \text{otherwise}
\end{cases}
\]

In the simplified rule-based format, it can be written as: \(a \circ a \rightarrow \lambda, b \circ a \rightarrow b\). In summary, \(f_1\) can be minimally expressed with a single rule, \(f_2\) requires two rules.

4 Benchmark Construction

In this section, we detail how our benchmark is constructed from the previously defined function classes. Each datapoint \((\mathcal{D}, f)\) in the benchmark is a pair of dataset \(\mathcal{D}\) and function \(f\) where \(\mathcal{D}\) is a set of input-output pairs generated by \(f\).

Each of ISL, L-OSL, and R-OSL classes can be further subdivided into incremental levels of complexity, determined by three key parameters: (1) the context window size \(k\) (2) the vocabulary size \(|\Sigma|\) (3) the minimal representation length of the function, i.e. the minimal set of rules corresponding to the function. Given \(k\) and \(|\Sigma|\), the search space is \(2^{|\Sigma|^k}\); given the number of rules \(n\) additionally, the search space is \(n^{\binom{|\Sigma|^k}{n}}\). To rigorously evaluate LLMs’ inductive capabilities, we systematically vary these parameters across ISL, L-OSL, and R-OSL function classes.

In addition, we examine how performance changes with different numbers of input–output pairs in the prompt. Although having the characteristic sample present should theoretically guarantee recoverability of the underlying function, our empirical results indicate that the overall number of examples strongly affects performance. While extra data can provide richer information, it also increases context length considerably and heightens processing demands (Li et al., 2024). By varying the number of provided datapoints, we further investigate the extent to which the model engages in genuine reasoning and how robust its inductive abilities remain under changing input sizes.

Function Generation To systematically create benchmark instances, we first randomly generate functions \(f\) based on the three parameters: \(k, |\Sigma|\), and the number of minimal rules describing \(f\) by generating the set of rules that can describe \(f\). While multiple representations of varying length can describe the same function, each function has a unique minimal representation (up to isomorphism). During function generation, we therefore ensure that each function is expressed by a minimal, non-redundant rule set. Formally, if a \(f\) is represented by a set of rules \(R_f = \{r_1, r_2, ..., r_n\}\) where each \(r_i\) has the form of \(c_i \circ u_i \rightarrow v_i\) (with \(c_i\) as the condition substring, \(u_i\) the target character, and \(v_i\) the transformed output for \(u_i\)), there are several constraints may be applied to functions belonging to the three classes.

Definition 5 (General Consistency) Given \(f\) represented by a set of rules \(R_f : \forall r_i, r_j \in R_f, c_i \circ u_i \notin \mathrm{SUFF}(c_j \circ u_j)\) and \(c_j \circ u_j \notin \mathrm{SUFF}(c_i \circ u_i)\).

General Consistency ensures that the rules do not contradict one another or become redundant when conditions overlap. For instance, a function whose rule-based representation of \(r_1 : a \circ b \rightarrow a\) and \(r_2 : aa \circ b \rightarrow a\) is redundant, as the scenarios where \(r_1\) is applied is a superset of the scenarios where \(r_2\) is applied. For another instance, there does not exist a deterministic function that can be described by \(r_1 : a \circ b \rightarrow a\) and \(r_2 : aa \circ b \rightarrow \lambda\). Generating rule-based representations for ISL functions needs only satisfy this constraint.

Definition 6 (OSL Non-Redundancy Guarantee)
Given \(f\) represented by a set of rules \(R : \forall r_i \in R_f, \neg \exists s'_i \in \{s_i | s_i \in c_i\}\) such that \(\exists r_j \in R_f\) such that \(s'_i = c_j \circ u_j\), unless \(\exists r_k \in R\) such that \(c_k \circ v_k = s'_i\).

Constraint 2 is specific to the two OSL function classes because we need to make sure that all output conditions in the rule actually surface somewhere in the outputs of some datapoints. If the output condition \(c\) never actually surface as the output, the rule will never be put into effect. Thereby the above rule basically requires that condition part of all rules can surface, either because it will never be modified by some other rule, or it emerges on the surface because of the application of other rule. For instance, a function represented by rules \(r_1 : aa \circ b \rightarrow a, r_2 : a \circ a \rightarrow c\) is redundant because \(r_1\) will never be applied because the string \(aa\) will never surface as output and thus it will never be put into effect; For another instance, a function represented by \(r_1 : aa \circ b \rightarrow a, r_2 : a \circ a \rightarrow c, r_3 : a \circ d \rightarrow a\) is
non redundant because even though into \( aa \) string will be modified into \( ac \), but \( aa \) will surface in some datapoint because \( ad \) will be modified into \( aa \) and thus \( r_1 \) will be able to be applied.

Generating the functions following the two constraints, we ensure that the generated function representation is minimal, non-reducible guarantees a clear measure of complexity. One additional requirement is imposed to ensure each function indeed requires a look-ahead of size \( k \). Specifically:

Definition 7 (k-Complexity Guarantee) Given \( f \) whose designated context window \( k = k_1 \), \( \exists r' \in R \) such that \( c' \circ u' \rightarrow v' \) such that \( |c' \circ u'| = k_1 \).

This condition guarantees that the function is genuinely \( k \)-strictly local (for ISL or OSL), rather than being representable with a smaller window size. Consequently, the functions we generate faithfully reflect the intended complexity level.

After generating the function \( f \), we generate the characteristic sample of input-output pairs. For instance, given a function \( f \) with \( k = 2 \) and \( \Sigma = \{a, b\} \), the characteristic sample is \( \{(a, f(a)), (b, f(b)), (ab, f(ab)), (aa, f(aa)), (bb, f(bb)), (ba, f(ba))\} \), a small set whose size is 6. By expanding this sample set, we can explore whether providing more than the minimal necessary examples aids or hinders the model’s performance to infer the underlying function.

To evaluate how effectively an LLM can induce the underlying function, we include in the prompt (1) the function class, (2) context window \( k \), (3) the alphabet \( \Sigma \) which are information that guarantee learnability of the function. Then given the sample dataset, we request LLMs to produce a minimal rule-based description that reproduces the provided sample set, revealing whether it can discover and optimally represent the underlying transformation.

5 Main Experiment

Experiment Setting We evaluate using zero-shot chain-of-thought prompting on six SOTA LLMs, including Llama-3.3-70b (Dubey et al., 2024) with FP8 quantization, Llama-3.1-405b with FP8 quantization, GPT-4o (Hurst et al., 2024), DeepSeek-V3 (Liu et al., 2024a), o1-mini(Jaech et al., 2024), and o3-mini. For all models, we evaluate with all settings including \( k \in \{2, 3, 4\}, |\Sigma| \in \{2, 3, 4\} \), number of rules \( \in \{2, 3, 4\} \), and sample set size to be 1, 2, 3, 4 times larger than the characteristic sample. For each setting, we randomly generate 10 functions \( f \) and corresponding input-output sample \( \mathcal{D} \) to calculate the result. As o1-mini and o3-mini perform much better than other models, in addition, we evaluate on two more complex settings with \( k \in \{4, 5\}, |\Sigma| = 5 \).

Evaluation Metrics For each experiment setting, we leverage three metrics to evaluate performance: Precision, Recall, Compatibility. Let \( R \) be the unique ground-truth rule set of minimal length for function \( f \), \( P \) be the predicted rule set generated by LLM, \( \mathcal{D} \) be the provided sample set in the context on which we evaluate the correctness of \( P \).

Precision measures how many of the predicted rules are correct relative to all rules the model generated: \( \frac{|R \cap P|}{|P|} \). captures the proportion of the model’s rules that align exactly with the ground-truth rules. A higher precision indicates fewer unnecessary/redundant rules.

Recall measures what fraction of the ground-truth rules the model successfully recovered: \( \frac{|R \cap P|}{|R|} \). A higher recall reflects the model’s ability to cover all aspects of the correct transformation.

Compatibility measures whether applying the predicted rule set \( P \) to each input in the sample set \( \mathcal{D} \) yields the correct output:

\[
\text{Compatibility}(P, \mathcal{D}) = \begin{cases}
1 & \text{if } \forall (x_i, y_i) \in \mathcal{D}, \\ & P(x_i) = y_i \\
0 & \text{otherwise}
\end{cases}
\]

Compatibility is the most fundamental measure, as it verifies whether the generated function accurately reproduces all observed input–output pairs in \( \mathcal{D} \). A trivial way to achieve perfect compatibility is to include every pair \( (x_i, y_i[-1]) \in \mathcal{D} \) as an independent rule; however, doing so results in very low precision, indicating a failure to capture the underlying generalizable structure of the function. Note that all results presented are expressed as percentages.

5.1 Model Performance Comparison

Table 1 showcases model performance under particularly challenging conditions: \( k = 4, |\Sigma| = 4 \) with 3 rules and sample size twice that of the minimal size of characteristic sample (detailed analysis on the impact of sample size is presented in Section 5.2). As seen in the table, compatibility scores collapse to 0 for all models except o3-mini, which also achieves relatively modest compatibility overall. This pattern highlights the difficulty that current
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">ISL</th>
    <th colspan="3">L-OSL</th>
    <th colspan="3">R-OSL</th>
  </tr>
  <tr>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
  </tr>
  <tr>
    <td>Llama-3.3 70B</td>
    <td>10.00</td>
    <td>5.32</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>6.33</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>10.83</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Llama-3.1 405B</td>
    <td>10.00</td>
    <td>3.75</td>
    <td>0.00</td>
    <td>6.67</td>
    <td>1.10</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>1.85</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>10.00</td>
    <td>2.67</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>3.82</td>
    <td>0.00</td>
    <td>16.67</td>
    <td>6.73</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>13.33</td>
    <td>2.46</td>
    <td>0.00</td>
    <td>23.33</td>
    <td>2.73</td>
    <td>0.00</td>
    <td>3.33</td>
    <td>0.25</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>36.67</td>
    <td>22.09</td>
    <td>0.00</td>
    <td>43.33</td>
    <td>32.12</td>
    <td>0.00</td>
    <td>26.67</td>
    <td>17.58</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td><b>73.33</b></td>
    <td><b>59.58</b></td>
    <td><b>10.00</b></td>
    <td><b>66.67</b></td>
    <td><b>69.17</b></td>
    <td><b>10.00</b></td>
    <td><b>63.33</b></td>
    <td><b>62.00</b></td>
    <td><b>30.00</b></td>
  </tr>
</table>

Table 1: Zero-shot CoT benchmark result with \( k = 4 \), \( |\Sigma| = 4 \), number of rules = 3, sample size = 2

![Bar charts showing ISL results for five models: Llama3.3-70b, Llama3.1-405b, GPT-4o, DeepSeek-V3, o1-mini, across different vocabulary sizes and context window sizes](page_324_624_1092_1012.png)

Figure 6: ISL results for five models: Llama3.3-70b, Llama3.1-405b, GPT-4o, DeepSeek-V3, o1-mini

LLMs face when required to track slightly broader contexts window even under very limited vocabulary size = 4. Full results are presented in Tables 3, 4, 5 in Appendix. Table 2 further reports the performance of o1-mini and o3-mini under slightly more challenging settings. Although both models generally exhibit non-trivial recall and precision, their compatibility scores consistently remain at or near zero. It is important to note that ISL, L-OSL, and R-OSL are the simplest function classes within the subregular hierarchy of string-to-string mappings. Thus, despite the strong performance of state-of-the-art models on benchmarks in coding (Jain et al., 2024), mathematics (Mirzadeh et al., 2024), and knowledge-intensive tasks (Wang et al., 2024), they falter on this elementary inductive reasoning task.

5.2 Impact of Different Factors

Figure 6 shows how five models (Llama3.3-70B, Llama3.1-405B, GPT-4o, DeepSeek-V3, and o1-mini) perform on various ISL tasks, organized by three key parameters: the context window size \( k \), the vocabulary size \( |\Sigma| \), and the minimal number of rules required to describe the function. The top row of panels presents recall, the middle row presents precision, and the bottom row presents compatibility. We did not include o3-mini here because its performance is way stronger than all other five models and thereby in order to show the
Figure 7: Impact of Few-shot Examples under different k.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Settings</th>
    <th colspan="3">k = 4</th>
    <th colspan="3">k = 5</th>
  </tr>
  <tr>
    <th>R</th>
    <th>P</th>
    <th>C</th>
    <th>R</th>
    <th>P</th>
    <th>C</th>
  </tr>
  <tr>
    <td colspan="8"><b>ISL</b></td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 4</td>
    <td>25.00</td>
    <td>12.21</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>9.10</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 5</td>
    <td>36.00</td>
    <td>40.41</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>3.14</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 4</td>
    <td>37.50</td>
    <td>49.83</td>
    <td>0.00</td>
    <td>27.50</td>
    <td>30.75</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 5</td>
    <td>42.00</td>
    <td>58.67</td>
    <td>0.00</td>
    <td>20.00</td>
    <td>38.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="8"><b>L-OSL</b></td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 4</td>
    <td>32.50</td>
    <td>37.34</td>
    <td>0.00</td>
    <td>15.00</td>
    <td>29.33</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 5</td>
    <td>28.00</td>
    <td>23.17</td>
    <td>0.00</td>
    <td>8.00</td>
    <td>4.61</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 4</td>
    <td>37.50</td>
    <td>58.93</td>
    <td>0.00</td>
    <td>22.50</td>
    <td>39.26</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 5</td>
    <td>48.00</td>
    <td>71.38</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>23.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="8"><b>R-OSL</b></td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 4</td>
    <td>17.50</td>
    <td>22.33</td>
    <td>0.00</td>
    <td>12.50</td>
    <td>7.63</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 5</td>
    <td>16.00</td>
    <td>15.42</td>
    <td>0.00</td>
    <td>18.00</td>
    <td>22.82</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 4</td>
    <td>45.00</td>
    <td>43.76</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>50.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 5</td>
    <td>38.00</td>
    <td>55.17</td>
    <td>0.00</td>
    <td>14.00</td>
    <td>36.17</td>
    <td>0.00</td>
  </tr>
</table>

Table 2: o1-mini and o3-mini results (R = Recall, P = Precision, C = Compatibility) on harder setting with \( k \in \{4, 5\}, |\Sigma| = 5 \), sample size = 2

impact of various factors, we omit this model for better visual clarity. Based on these figures, the impacts of \( k, |\Sigma| \), and number of rules are clear:

Impact of \( k \). Across all models, moving from \( k = 2 \) to \( k = 4 \) markedly reduces recall, precision, and compatibility. This trend underscores how increasing the context window increases the complexity of the underlying ISL functions and making it more challenging for current LLMs to induce patterns. Longer look-ahead requires the model to track additional input context, which can overload its capacity to induce reliable rules.

Impact of \( |\Sigma| \). In contrast, enlarging the vocabulary from \( |\Sigma| = 2 \) to \( |\Sigma| = 4 \) does not consistently degrade performance to the same degree as increasing \( k \). While some models exhibit slight declines in recall or precision with a larger alphabet, these effects are neither as uniform nor as pronounced as those induced by a bigger Markov window. It suggests that the breadth of symbol variation matters less than the depth of sequential dependencies.

Impact of the Number of Rules. Notably, the number of minimal rules can substantially affect compatibility. When \( k = 2 \) and \( |\Sigma| = 2 \), a comparatively small search space, changing the number of rules does not drastically alter compatibility. However, under more demanding scenarios where \( k \in \{3, 4\} \), the data indicate that adding rules can cause compatibility to plummet. In many cases, having just one rule still yields nontrivial compatibility, whereas introducing a second or third rule often overwhelms the models, resulting in compatibility scores near 0.

Impact of Examples. We further examined whether few-shot prompting could enhance model performance. In our experiments with Llama-3.3 70B, we varied the number of in-context examples (1-shot, 2-shot, and 3-shot) to determine their effect on the model’s ability to induce the correct function representation. The results indicate that when both the vocabulary size and the context window \( k \) are small, adding more examples improves performance across the evaluated metrics. However, as the complexity increases with larger values of \( k \) and vocabulary sizes, the benefits of additional few-shot examples become negligible. While few-shot learning is beneficial for simpler settings, its efficacy diminishes in more complex inductive tasks. Full experiment results are presented in Tables 6, 7, and 8. More detailed analysis are presented in Appendix A.

6 Conclusion

We introduced InductionBench, a benchmark designed to evaluate the inductive reasoning abilities of LLM using subregular function classes that are provably learnable in polynomial time and data. By systematically varying the Markov window size \( k \), vocabulary size \( |\Sigma| \), and the minimal number of rules, our experiments reveal that even SOTA models struggle with inferring underlying transformations in the simplest complexity class in string-to-string transformation. These findings highlight a critical caveat in inductive reasoning ability, underscoring the need for further research.
Limitations

While our benchmark offers a rigorous, theoretically grounded approach to evaluating inductive reasoning in LLMs, it is subject to three notable constraints:

Synthetic Rather Than Real-World Data. All tasks and evaluations rely on functions generated from carefully controlled parameters rather than naturally occurring texts or real-world datasets. Although this design enables precise measurement of inductive capabilities, it may not fully capture the complexity of practical language use, where ambiguous contexts, noisy inputs, and domain-specific factors can further challenge inference.

Restricted Access to the o1 Model. Our investigation into the o1 family of models is hindered by limited availability and computational resources. As a result, certain aspects of o1’s inductive behavior may remain unexamined, and a more exhaustive exploration of variations or fine-tuning strategies for o1 could further illuminate its performance.

Existence of Provably Correct Algorithms. A potential concern is that the benchmark could be trivially “hacked” by implementing known polynomial-time learning algorithms for the classes we study, rather than testing genuine inductive reasoning. However, we deliberately select function classes with established provable learnability precisely so that performance can be measured against a well-understood theoretical baseline. To address the possibility of artificially inflating scores, we provide an additional variant of the benchmark that omits explicit information on the function class (ISL, L-OSL, or R-OSL) and the parameter \( k \). Furthermore, we introduce an alternate class on another version of the benchmark, Input-Output Strictly Local (IOSL), for which no known algorithm can reliably learn the underlying functions from finite samples. By doing so, we aim to evaluate whether LLMs can truly demonstrate inductive reasoning skills—even in the absence of a well-documented learning procedure—thereby mitigating concerns that performance gains merely reflect a “hack” rather than robust inference.

References

Arvind Arasu, Surajit Chaudhuri, and Raghav Kaushik. 2009. Learning string transformations from examples. Proceedings of the VLDB Endowment, 2(1):514–525.

Alan Baker. 2007. Occam’s razor in science: a case study from biogeography. Biology & Philosophy, 22(2):193–215.

Steven Bird and T Mark Ellison. 1994. One-level phonology: Autosegmental representations and rules as finite automata. Computational Linguistics, 20(1):55–90.

Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. 1987. Occam’s razor. Information processing letters, 24(6):377–380.

Jane Chandlee, Rémi Eyraud, and Jeffrey Heinz. 2014. Learning strictly local subsequential functions. Transactions of the Association for Computational Linguistics, 2:491–504.

Jane Chandlee, Rémi Eyraud, and Jeffrey Heinz. 2015. Output strictly local functions. In 14th Meeting on the Mathematics of Language, pages 112–125.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Colin De La Higuera. 1997. Characteristic sets for polynomial grammatical inference. Machine Learning, 27:125–138.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Joost Engelfriet and Hendrik Jan Hoogeboom. 2001. Mso definable string transductions and two-way finite-state transducers. ACM Transactions on Computational Logic (TOCL), 2(2):216–254.

Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, and Yongfeng Zhang. 2023. Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes. arXiv preprint arXiv:2312.14890.

Thomas Graf. 2022. Diving deeper into subregular syntax. Theoretical Linguistics, 48(3-4):245–278.

Peter D Grünwald. 2007. The minimum description length principle. MIT press.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, et al. 2022. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840.

Mark H Hansen and Bin Yu. 2001. Model selection and the principle of minimum description length. Journal of the american statistical association, 96(454):746–774.

James Hawthorne. 2004. Inductive logic.

Jeffrey Heinz. 2018. The computational nature of phonological generalizations. Phonological typology, phonetics and phonology, 23:126–195.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.

Wenyue Hua, Kaijie Zhu, Lingyao Li, Lizhou Fan, Shuhang Lin, Mingyu Jin, Haochen Xue, Zelong Li, JinDong Wang, and Yongfeng Zhang. 2024. Disentangling logic: The role of context in large language model reasoning capabilities. arXiv preprint arXiv:2406.02787.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai ol system card. arXiv preprint arXiv:2412.16720.

Gerhard Jäger and James Rogers. 2012. Formal language theory: refining the chomsky hierarchy. Philosophical Transactions of the Royal Society B: Biological Sciences, 367(1598):1956–1970.

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.

Adam Jardine, Jane Chandlee, Rémi Eyraud, and Jeffrey Heinz. 2014. Very efficient learning of structured classes of subsequential functions from positive data. In International Conference on Grammatical Inference, pages 94–108. PMLR.

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ohir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.

Brenden M Lake, Tal Linzen, and Marco Baroni. 2019. Human few-shot learning of compositional instructions. arXiv preprint arXiv:1901.04587.

Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhui Chen. 2024. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Emmy Liu, Graham Neubig, and Jacob Andreas. 2024b. An incomplete loop: Deductive, inductive, and abductive learning in large language models. arXiv preprint arXiv:2404.03028.

Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229.

Mehryar Mohri. 1997. Finite-state transducers in language and speech processing. Computational linguistics, 23(2):269–311.

José Oncina and Pedro García. 1991. Inductive learning of subsequential functions. Univ. Politécnica de Valencia, Tech. Rep. DSIC II, 31.

Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13679–13707.

Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, et al. 2023. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. arXiv preprint arXiv:2310.08559.

Emmanuel Roche and Yves Schabes. 1997. Finite-state language processing, volume 115. MIT press Cambridge, MA.

James Rogers and Geoffrey K Pullum. 2011. Aural pattern recognition experiments and the subregular hierarchy. Journal of Logic, Language and Information, 20:329–342.

Giorgio Satta and John Henderson. 1997. String transformation learning. In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics, pages 444–451.
Bianca Truthe. 2018. Hierarchy of subregular language families.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574.

Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset requiring logical reasoning. arXiv preprint arXiv:2002.04326.

Kaijie Zhu, Jiaoo Chen, Jindong Wang, Neil Zhenqiang Gong, Dify Yang, and Xing Xie. 2023. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations.
A More Analysis

Robustness  We assess the stability of inductive reasoning by varying the number of input–output pairs provided to the model. The x-axis represent \( \frac{|D|}{|S|} \) where S is the minimal set of examples needed to guarantee learnability of the underlying function. The hypothesis is that if the model were performing genuine logical or inductive reasoning, we would expect performance to remain stable or even improve as more data points become available, since these points should further clarify the underlying function. Figure 8 illustrates how average compatibility decreases steeply as the number of provided input–output examples increases. This drop suggests that the LLM’s reasoning process is not robustly inductive: rather than refining its hypothesis with additional data, the model appears to become confused or overwhelmed, leading to poorer overall performance. Consequently, these findings highlight the limited robustness of current LLMs’ inductive reasoning, particularly in scenarios where increasing the available data should theoretically facilitate, rather than hinder, function inference.

Moreover, to isolate the influence of context length from the effect of adding genuinely new data, we conduct an additional experiment in which we simply extending the context size by repeating the minimal characteristic sample without introducing novel input–output pairs. Comparing Figures 8 and 9 reveals that while compatibility does diminish with increased context length (e.g., at a multiple of 2), the decline is relatively small when scaling further to multiples of 3, 4, or 5. By contrast, when truly new datapoints are added (and not just repeated), compatibility plummets nearly to zero for multiples of 4 and 5. These results confirm that the primary driver of performance degradation is the inclusion of additional, distinct datapoints rather than simply lengthening the context.

Error Type Analysis  We further examined the specific types of errors made by LLMs when their predicted functions failed to match the ground-truth dataset. At a high level, we distinguish between missing rules (leading to low recall) and wrong rules (leading to low precision).

Missing Rules: These refer to ground-truth rules that do not appear in the model’s predicted rule set. We classify missing rules into three subtypes:

1. Too General. Although a certain ground-truth rule \( r : c \circ u \rightarrow v \) was missed, there exists a corresponding predicted rule \( r' : c' \circ u' \rightarrow v' \) that over-generalizes. Specifically, the condition \( c' \) is a proper suffix of \( c \), causing \( r' \) to apply more broadly than intended.

2. Too Specific. The opposite of the above: a predicted rule condition \( c' \) is a proper extension of \( c \), thus applying too narrowly and failing to match some instances that should have been captured by the ground-truth rule.

3. Completely Missed. No predicted rule overgeneralizes or under-generalizes the ground-truth rule; in other words, this pattern is simply absent from the predicted rule set altogether.

Wrong Rules: These refer to rules present in the model’s predicted set that do not exist in the ground truth. We categorize such rules into four types:

1. Too General. The rule \( r' : c' \circ u' \rightarrow v' \) is overly broad, applying in contexts where the ground truth does not. This typically arises when \( c' \) is a proper suffix of some genuine condition \( c \) and thus fails to capture necessary constraints.

2. Too Specific. The rule narrowly addresses only a subset of the intended patterns (e.g., by employing a condition \( c' \) that is an extension of the legitimate condition \( c \)), thereby missing broader contexts that should have matched.

3. Correct Condition but Wrong Transformation. Here, the predicted rule accurately identifies the correct condition \( c' \) and target input character \( u' \), but the transformation \( v' \) is incorrect.

4. Completely Wrong. None of the above criteria apply: the rule’s condition and transformation are both inconsistent with the ground truth, indicating a fundamental misunderstanding.

We present a breakdown of error types for three models in Figure10: Llama3.3-70B, o1-mini, and o3-mini. Among missing rules, the most common issue is completely missed, where the model fails to identify the relevant pattern at all. The second most frequent error is too general, suggesting that the predicted condition is shorter than needed, thereby overgeneralizing the intended behavior. In contrast, too specific errors in this category are relatively rare. Among wrong rules, the majority are completely wrong, followed by a notable fraction of too general. Although there is also a
Figure 8: Impact of Sample Size based with Different Sample Size Multiples.

Figure 9: Impact of Context Length based with Different Sample Size Multiples.

Figure 10: Error Type Analysis
non-negligible number of too specific errors, these tend to occur when a single ground-truth rule (e.g., \( ab \circ c \rightarrow b \) is replaced by multiple subcases (\( e.g., aab \circ c \rightarrow b, bab \circ c \rightarrow b, cab \circ c \rightarrow b \)), indicating the model has uncovered individual instances but failed to unify them into a concise representation. Finally, correct condition but wrong transformation occurs relatively infrequently, implying that once the model infers the correct condition pattern, it typically produces the correct transformation.

B  Summary of findings

Overall, our experiments reveal four main insights into the inductive reasoning performance of current LLMs:

• Context window size \( k \) dominates complexity: Increasing \( k \) from 2 to 4 significantly degrades recall, precision, and compatibility, underscoring how longer look-ahead windows intensify the complexity of ISL functions.

• Number of Rules increases difficulty under large hypothesis space: The number of minimal rules required can drastically lower compatibility in more challenging settings with large \( k \) adn \( |\Sigma| \), indicating that managing multiple interacting rules overwhelms many models.

• Few-shot examples do not help much: Few-shot examples help in simpler configurations but yield diminishing returns as \( k \) adn \( |\Sigma| \) grows—suggesting that, past a certain complexity threshold, additional examples do not compensate for the model’s limited inductive capacity.

• Current LLMs are very unrobust: Providing more novel data should theoretically clarify function patterns, yet performance often plummets, reflecting a fragility in inductive reasoning.

• Error analysis shows that missing rules are most frequently “completely missed” or “too general,” while wrong rules often end up “completely wrong” or again “too general.” Only a small fraction are “too specific” or feature a “correct condition but wrong transformation,” indicating that once models identify the right condition, they typically produce the correct transformation.

Taken together, these findings highlight fundamental limits in current LLMs’ inductive reasoning. Even state-of-the-art models often fail as complexity grows, or when confronted with more data than their inductive mechanisms appear able to systematically absorb.

C  Full Result
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Settings</th>
    <th colspan="3">k = 2</th>
    <th colspan="3">k = 3</th>
    <th colspan="3">k = 4</th>
  </tr>
  <tr>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
  </tr>
  <tr>
    <td colspan="10"><b>vocab size = 2</b></td>
  </tr>
  <tr>
    <td>Llama-3.3 70B</td>
    <td>rules = 1</td>
    <td>60.00</td>
    <td>55.00</td>
    <td>60.00</td>
    <td>30.00</td>
    <td>23.33</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 2</td>
    <td>60.00</td>
    <td>65.00</td>
    <td>50.00</td>
    <td>45.00</td>
    <td>60.00</td>
    <td>30.00</td>
    <td>15.00</td>
    <td>8.25</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>53.33</td>
    <td>68.33</td>
    <td>20.00</td>
    <td>30.00</td>
    <td>46.67</td>
    <td>10.00</td>
    <td>16.67</td>
    <td>8.54</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>30.00</td>
    <td>25.00</td>
    <td>30.00</td>
    <td>50.00</td>
    <td>35.00</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>15.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>Llama-3.1 405B</td>
    <td>rules = 2</td>
    <td>55.00</td>
    <td>50.00</td>
    <td>40.00</td>
    <td>10.00</td>
    <td>6.67</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>7.50</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>56.67</td>
    <td>44.17</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>9.50</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>90.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>32.50</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>12.33</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>rules = 2</td>
    <td>80.00</td>
    <td>60.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>19.89</td>
    <td>10.00</td>
    <td>15.00</td>
    <td>3.82</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>70.00</td>
    <td>54.83</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>26.15</td>
    <td>0.00</td>
    <td>33.33</td>
    <td>10.61</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>60.00</td>
    <td>43.33</td>
    <td>40.00</td>
    <td>50.00</td>
    <td>22.00</td>
    <td>40.00</td>
    <td>10.00</td>
    <td>2.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>rules = 2</td>
    <td>60.00</td>
    <td>37.50</td>
    <td>50.00</td>
    <td>35.00</td>
    <td>18.43</td>
    <td>20.00</td>
    <td>15.00</td>
    <td>5.63</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>73.33</td>
    <td>68.33</td>
    <td>60.00</td>
    <td>36.67</td>
    <td>19.30</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>2.50</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>50.00</td>
    <td>45.00</td>
    <td>50.00</td>
    <td>70.00</td>
    <td>45.83</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>16.67</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 2</td>
    <td>75.00</td>
    <td>75.00</td>
    <td>75.00</td>
    <td>70.00</td>
    <td>60.00</td>
    <td>40.00</td>
    <td>50.00</td>
    <td>28.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>66.67</td>
    <td>61.67</td>
    <td>60.00</td>
    <td>43.33</td>
    <td>34.00</td>
    <td>0.00</td>
    <td>40.00</td>
    <td>38.52</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>80.00</td>
    <td>58.33</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 2</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.67</td>
    <td>90.00</td>
    <td>85.00</td>
    <td>80.00</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>100.00</td>
    <td>97.50</td>
    <td>100.00</td>
    <td>83.33</td>
    <td>71.83</td>
    <td>60.00</td>
    <td>70.00</td>
    <td>63.81</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td colspan="10"><b>vocab size = 3</b></td>
  </tr>
  <tr>
    <td>Llama-3.3 70B</td>
    <td>rules = 1</td>
    <td>70.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>8.33</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 2</td>
    <td>85.00</td>
    <td>83.33</td>
    <td>60.00</td>
    <td>10.00</td>
    <td>7.50</td>
    <td>0.00</td>
    <td>5.00</td>
    <td>2.50</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>66.67</td>
    <td>74.17</td>
    <td>20.00</td>
    <td>33.33</td>
    <td>35.36</td>
    <td>0.00</td>
    <td>6.67</td>
    <td>3.43</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>Llama-3.1 405B</td>
    <td>rules = 2</td>
    <td>45.00</td>
    <td>32.58</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>7.50</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>50.00</td>
    <td>38.45</td>
    <td>20.00</td>
    <td>6.67</td>
    <td>3.10</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>5.27</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>70.00</td>
    <td>65.00</td>
    <td>60.00</td>
    <td>70.00</td>
    <td>45.00</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>13.93</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>rules = 2</td>
    <td>80.00</td>
    <td>56.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>13.23</td>
    <td>0.00</td>
    <td>25.00</td>
    <td>1.89</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>80.00</td>
    <td>60.76</td>
    <td>50.00</td>
    <td>56.67</td>
    <td>21.64</td>
    <td>0.00</td>
    <td>40.00</td>
    <td>5.88</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>50.00</td>
    <td>33.33</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>18.33</td>
    <td>40.00</td>
    <td>20.00</td>
    <td>4.17</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>rules = 2</td>
    <td>60.00</td>
    <td>40.42</td>
    <td>30.00</td>
    <td>25.00</td>
    <td>6.00</td>
    <td>0.00</td>
    <td>30.00</td>
    <td>6.39</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>66.67</td>
    <td>64.33</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>9.61</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>1.72</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>50.00</td>
    <td>43.33</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 2</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>80.00</td>
    <td>40.00</td>
    <td>25.11</td>
    <td>10.00</td>
    <td>55.00</td>
    <td>24.82</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>80.00</td>
    <td>77.33</td>
    <td>60.00</td>
    <td>63.33</td>
    <td>36.25</td>
    <td>10.00</td>
    <td>30.00</td>
    <td>20.44</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>95.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>78.33</td>
    <td>70.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 2</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>95.00</td>
    <td>91.67</td>
    <td>80.00</td>
    <td>75.00</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td><b>96.67</b></td>
    <td><b>97.50</b></td>
    <td><b>80.00</b></td>
    <td><b>93.33</b></td>
    <td><b>91.67</b></td>
    <td><b>90.00</b></td>
    <td><b>83.33</b></td>
    <td><b>85.17</b></td>
    <td><b>50.00</b></td>
  </tr>
  <tr>
    <td colspan="10"><b>vocab size = 4</b></td>
  </tr>
  <tr>
    <td>Llama-3.3 70B</td>
    <td>rules = 1</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 2</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>15.00</td>
    <td>11.67</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>53.33</td>
    <td>68.33</td>
    <td>20.00</td>
    <td>6.67</td>
    <td>5.00</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>3.32</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>40.00</td>
    <td>35.00</td>
    <td>30.00</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>Llama-3.1 405B</td>
    <td>rules = 2</td>
    <td>75.00</td>
    <td>52.33</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>5.83</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>40.00</td>
    <td>34.33</td>
    <td>10.00</td>
    <td>13.33</td>
    <td>1.54</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>3.75</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>80.00</td>
    <td>52.50</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>17.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>14.58</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>rules = 2</td>
    <td>85.00</td>
    <td>57.15</td>
    <td>50.00</td>
    <td>65.00</td>
    <td>18.66</td>
    <td>20.00</td>
    <td>45.00</td>
    <td>5.30</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>76.67</td>
    <td>63.12</td>
    <td>40.00</td>
    <td>50.00</td>
    <td>16.05</td>
    <td>10.00</td>
    <td>13.33</td>
    <td>2.46</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>80.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>50.00</td>
    <td>16.67</td>
    <td>20.00</td>
    <td>50.00</td>
    <td>21.67</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>rules = 2</td>
    <td>45.00</td>
    <td>29.00</td>
    <td>10.00</td>
    <td>45.00</td>
    <td>12.62</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>56.67</td>
    <td>38.62</td>
    <td>0.00</td>
    <td>33.33</td>
    <td>20.60</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>2.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>80.00</td>
    <td>70.00</td>
    <td>80.00</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>40.00</td>
    <td>15.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 2</td>
    <td>75.00</td>
    <td>63.33</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>29.93</td>
    <td>10.00</td>
    <td>35.00</td>
    <td>35.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td><b>93.33</b></td>
    <td><b>91.67</b></td>
    <td><b>80.00</b></td>
    <td><b>46.67</b></td>
    <td><b>37.72</b></td>
    <td><b>10.00</b></td>
    <td><b>36.67</b></td>
    <td><b>22.09</b></td>
    <td><b>0.00</b></td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>95.00</td>
    <td>100.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 2</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>95.00</td>
    <td>91.67</td>
    <td>80.00</td>
    <td>75.00</td>
    <td>67.67</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td><b>96.67</b></td>
    <td><b>95.00</b></td>
    <td><b>90.00</b></td>
    <td><b>93.33</b></td>
    <td><b>93.33</b></td>
    <td><b>80.00</b></td>
    <td><b>73.33</b></td>
    <td><b>59.58</b></td>
    <td><b>10.00</b></td>
  </tr>
</table>

Table 3: Input Strictly Local with sample size = 2
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Settings</th>
    <th colspan="3">k = 2</th>
    <th colspan="3">k = 3</th>
    <th colspan="3">k = 4</th>
  </tr>
  <tr>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
  </tr>
  <tr>
    <td colspan="10"><b>vocab size = 2</b></td>
  </tr>
  <tr>
    <td>Llama-3.3 70B</td>
    <td>rules = 1</td>
    <td>50.00</td>
    <td>45.00</td>
    <td>50.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 2</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>8.33</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>10.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>56.67</td>
    <td>65.00</td>
    <td>0.00</td>
    <td>6.67</td>
    <td>8.33</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>12.83</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>70.00</td>
    <td>45.83</td>
    <td>70.00</td>
    <td>30.00</td>
    <td>9.33</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>1.67</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>Llama-3.1 405B</td>
    <td>rules = 2</td>
    <td>50.00</td>
    <td>33.33</td>
    <td>10.00</td>
    <td>25.00</td>
    <td>11.39</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>3.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>63.33</td>
    <td>53.83</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>6.67</td>
    <td>0.00</td>
    <td>6.67</td>
    <td>5.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>30.00</td>
    <td>12.50</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>10.83</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>rules = 2</td>
    <td>75.00</td>
    <td>63.17</td>
    <td>60.00</td>
    <td>20.00</td>
    <td>7.42</td>
    <td>0.00</td>
    <td>15.00</td>
    <td>6.35</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>66.67</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>30.00</td>
    <td>19.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>4.16</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>100.00</td>
    <td>75.00</td>
    <td>70.00</td>
    <td>50.00</td>
    <td>32.50</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>12.78</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>rules = 2</td>
    <td>60.00</td>
    <td>44.17</td>
    <td>30.00</td>
    <td>10.00</td>
    <td>15.00</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>11.94</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>83.33</td>
    <td>77.67</td>
    <td>50.00</td>
    <td>20.00</td>
    <td>12.92</td>
    <td>0.00</td>
    <td>23.33</td>
    <td>13.97</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>70.00</td>
    <td>55.00</td>
    <td>40.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 2</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>65.00</td>
    <td>53.83</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>90.00</td>
    <td>82.50</td>
    <td>50.00</td>
    <td>66.67</td>
    <td>60.67</td>
    <td>20.00</td>
    <td>50.00</td>
    <td>54.22</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 2</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>95.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>85.00</td>
    <td>78.33</td>
    <td>70.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>86.67</td>
    <td>85.00</td>
    <td>80.00</td>
    <td>56.67</td>
    <td>50.33</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td colspan="10"><b>vocab size = 3</b></td>
  </tr>
  <tr>
    <td>Llama-3.3 70B</td>
    <td>rules = 1</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>20.00</td>
    <td>12.50</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>13.33</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 2</td>
    <td>35.00</td>
    <td>33.67</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>6.93</td>
    <td>10.00</td>
    <td>25.00</td>
    <td>15.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>40.00</td>
    <td>65.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>18.33</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>2.78</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>60.00</td>
    <td>45.00</td>
    <td>40.00</td>
    <td>10.00</td>
    <td>3.33</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>1.11</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Llama-3.1 405B</td>
    <td>rules = 2</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>0.00</td>
    <td>15.00</td>
    <td>13.33</td>
    <td>0.00</td>
    <td>5.00</td>
    <td>0.53</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>66.67</td>
    <td>57.83</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>8.39</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>2.96</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>40.00</td>
    <td>27.50</td>
    <td>40.00</td>
    <td>20.00</td>
    <td>8.33</td>
    <td>20.00</td>
    <td>40.00</td>
    <td>11.00</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>rules = 2</td>
    <td>55.00</td>
    <td>46.50</td>
    <td>10.00</td>
    <td>45.00</td>
    <td>25.67</td>
    <td>0.00</td>
    <td>30.00</td>
    <td>6.50</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>10.00</td>
    <td>33.33</td>
    <td>15.95</td>
    <td>0.00</td>
    <td>20.00</td>
    <td>6.21</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>80.00</td>
    <td>70.00</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>22.00</td>
    <td>40.00</td>
    <td>50.00</td>
    <td>16.11</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>rules = 2</td>
    <td>90.00</td>
    <td>60.32</td>
    <td>60.00</td>
    <td>70.00</td>
    <td>13.82</td>
    <td>20.00</td>
    <td>30.00</td>
    <td>5.04</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>66.67</td>
    <td>53.50</td>
    <td>40.00</td>
    <td>23.33</td>
    <td>16.42</td>
    <td>0.00</td>
    <td>30.00</td>
    <td>7.54</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>100.00</td>
    <td>95.00</td>
    <td>90.00</td>
    <td>80.00</td>
    <td>63.33</td>
    <td>70.00</td>
    <td>50.00</td>
    <td>17.50</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 2</td>
    <td>90.00</td>
    <td>83.33</td>
    <td>80.00</td>
    <td>70.00</td>
    <td>49.42</td>
    <td>40.00</td>
    <td>35.00</td>
    <td>29.52</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>96.67</td>
    <td>96.00</td>
    <td>90.00</td>
    <td>70.00</td>
    <td>56.15</td>
    <td>30.00</td>
    <td>50.00</td>
    <td>33.58</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 2</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>90.00</td>
    <td>75.15</td>
    <td>70.00</td>
    <td>80.00</td>
    <td>72.50</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>96.67</td>
    <td>94.17</td>
    <td>90.00</td>
    <td>96.67</td>
    <td>87.50</td>
    <td>90.00</td>
    <td>63.33</td>
    <td>68.43</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td colspan="10"><b>vocab size = 4</b></td>
  </tr>
  <tr>
    <td>Llama-3.3 70B</td>
    <td>rules = 1</td>
    <td>50.00</td>
    <td>29.00</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>13.33</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 2</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>15.96</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>50.00</td>
    <td>52.50</td>
    <td>20.00</td>
    <td>6.67</td>
    <td>6.00</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>8.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>60.00</td>
    <td>34.50</td>
    <td>30.00</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>Llama-3.1 405B</td>
    <td>rules = 2</td>
    <td>50.00</td>
    <td>29.00</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>3.13</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>2.90</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>43.33</td>
    <td>30.73</td>
    <td>20.00</td>
    <td>16.67</td>
    <td>5.83</td>
    <td>0.00</td>
    <td>6.67</td>
    <td>1.10</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>40.00</td>
    <td>35.00</td>
    <td>30.00</td>
    <td>60.00</td>
    <td>25.33</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>12.50</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>rules = 2</td>
    <td>75.00</td>
    <td>45.50</td>
    <td>30.00</td>
    <td>55.00</td>
    <td>20.47</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>9.44</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>70.00</td>
    <td>48.22</td>
    <td>20.00</td>
    <td>33.33</td>
    <td>10.77</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>3.82</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>100.00</td>
    <td>82.50</td>
    <td>80.00</td>
    <td>50.00</td>
    <td>23.67</td>
    <td>30.00</td>
    <td>40.00</td>
    <td>16.11</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>rules = 2</td>
    <td>70.00</td>
    <td>50.67</td>
    <td>30.00</td>
    <td>25.00</td>
    <td>8.01</td>
    <td>0.00</td>
    <td>15.00</td>
    <td>3.24</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>60.00</td>
    <td>48.36</td>
    <td>30.00</td>
    <td>50.00</td>
    <td>12.81</td>
    <td>20.00</td>
    <td>23.33</td>
    <td>2.73</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>90.00</td>
    <td>85.00</td>
    <td>90.00</td>
    <td>30.00</td>
    <td>38.33</td>
    <td>30.00</td>
    <td>40.00</td>
    <td>28.33</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 2</td>
    <td>100.00</td>
    <td>93.33</td>
    <td>80.00</td>
    <td>60.00</td>
    <td>36.92</td>
    <td>20.00</td>
    <td>50.00</td>
    <td>31.92</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>80.00</td>
    <td>72.25</td>
    <td>60.00</td>
    <td>70.00</td>
    <td>43.04</td>
    <td>10.00</td>
    <td>43.33</td>
    <td>32.12</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 2</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>85.00</td>
    <td>81.67</td>
    <td>70.00</td>
    <td>45.00</td>
    <td>58.33</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>76.67</td>
    <td>76.67</td>
    <td>70.00</td>
    <td>66.67</td>
    <td>69.17</td>
    <td>10.00</td>
  </tr>
</table>

Table 4: Left Output Strictly Local with sample size = 2
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Settings</th>
    <th colspan="3">k = 2</th>
    <th colspan="3">k = 3</th>
    <th colspan="3">k = 4</th>
  </tr>
  <tr>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
  </tr>
  <tr>
    <td colspan="10"><b>vocab size = 2</b></td>
  </tr>
  <tr>
    <td>Llama-3.3 70B</td>
    <td>rules = 1</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 2</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>15.00</td>
    <td>18.33</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>18.67</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>63.33</td>
    <td>76.67</td>
    <td>30.00</td>
    <td>23.33</td>
    <td>24.17</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>7.83</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>20.00</td>
    <td>8.33</td>
    <td>0.00</td>
    <td>40.00</td>
    <td>18.33</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Llama-3.1 405B</td>
    <td>rules = 2</td>
    <td>60.00</td>
    <td>51.67</td>
    <td>40.00</td>
    <td>25.00</td>
    <td>12.50</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>6.25</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>63.33</td>
    <td>54.72</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>14.33</td>
    <td>10.00</td>
    <td>6.67</td>
    <td>4.50</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>rules = 2</td>
    <td>70.00</td>
    <td>61.67</td>
    <td>60.00</td>
    <td>45.00</td>
    <td>16.93</td>
    <td>10.00</td>
    <td>30.00</td>
    <td>15.83</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>83.33</td>
    <td>71.83</td>
    <td>30.00</td>
    <td>43.33</td>
    <td>24.10</td>
    <td>0.00</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>60.00</td>
    <td>58.33</td>
    <td>20.00</td>
    <td>40.00</td>
    <td>14.17</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>7.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>rules = 2</td>
    <td>75.00</td>
    <td>51.67</td>
    <td>40.00</td>
    <td>35.00</td>
    <td>18.33</td>
    <td>0.00</td>
    <td>25.00</td>
    <td>13.00</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>86.67</td>
    <td>72.56</td>
    <td>50.00</td>
    <td>40.00</td>
    <td>27.00</td>
    <td>0.00</td>
    <td>23.33</td>
    <td>13.68</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>50.00</td>
    <td>38.33</td>
    <td>40.00</td>
    <td>20.00</td>
    <td>5.83</td>
    <td>0.00</td>
    <td>20.00</td>
    <td>11.67</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 2</td>
    <td>55.00</td>
    <td>38.33</td>
    <td>30.00</td>
    <td>35.00</td>
    <td>17.50</td>
    <td>0.00</td>
    <td>15.00</td>
    <td>8.70</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>46.67</td>
    <td>46.67</td>
    <td>10.00</td>
    <td>33.33</td>
    <td>24.17</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>4.41</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>100.00</td>
    <td>95.00</td>
    <td>90.00</td>
    <td>70.00</td>
    <td>50.00</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 2</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>90.00</td>
    <td>85.00</td>
    <td>60.00</td>
    <td>45.00</td>
    <td>41.67</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>96.67</td>
    <td>92.67</td>
    <td>80.00</td>
    <td>86.67</td>
    <td>78.33</td>
    <td>50.00</td>
    <td>46.67</td>
    <td>46.67</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>Llama-3.3 70B</td>
    <td>rules = 1</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>25.00</td>
    <td>30.00</td>
    <td>7.26</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 2</td>
    <td>30.00</td>
    <td>60.00</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>10.93</td>
    <td>0.00</td>
    <td>25.00</td>
    <td>19.17</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>30.00</td>
    <td>45.00</td>
    <td>0.00</td>
    <td>26.67</td>
    <td>22.67</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>4.58</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Llama-3.1 405B</td>
    <td>rules = 1</td>
    <td>20.00</td>
    <td>11.43</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>0.00</td>
    <td>20.00</td>
    <td>4.17</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 2</td>
    <td>30.00</td>
    <td>20.83</td>
    <td>10.00</td>
    <td>15.00</td>
    <td>4.17</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>3.41</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>16.67</td>
    <td>9.11</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>4.75</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>40.00</td>
    <td>50.00</td>
    <td>27.50</td>
    <td>20.00</td>
    <td>30.00</td>
    <td>8.82</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>rules = 2</td>
    <td>60.00</td>
    <td>41.67</td>
    <td>30.00</td>
    <td>50.00</td>
    <td>27.50</td>
    <td>20.00</td>
    <td>40.00</td>
    <td>9.42</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>66.67</td>
    <td>57.83</td>
    <td>30.00</td>
    <td>40.00</td>
    <td>21.88</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>6.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>rules = 2</td>
    <td>80.00</td>
    <td>63.19</td>
    <td>50.00</td>
    <td>55.00</td>
    <td>26.67</td>
    <td>20.00</td>
    <td>40.00</td>
    <td>10.10</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>70.00</td>
    <td>42.95</td>
    <td>50.00</td>
    <td>23.33</td>
    <td>15.28</td>
    <td>0.00</td>
    <td>20.00</td>
    <td>3.56</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>20.00</td>
    <td>13.33</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>13.33</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 2</td>
    <td>15.00</td>
    <td>15.00</td>
    <td>10.00</td>
    <td>40.00</td>
    <td>25.00</td>
    <td>0.00</td>
    <td>30.00</td>
    <td>19.50</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>53.33</td>
    <td>45.83</td>
    <td>20.00</td>
    <td>30.00</td>
    <td>17.63</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>8.43</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>100.00</td>
    <td>95.00</td>
    <td>100.00</td>
    <td>50.00</td>
    <td>40.00</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 2</td>
    <td>90.00</td>
    <td>83.33</td>
    <td>70.00</td>
    <td>80.00</td>
    <td>61.67</td>
    <td>30.00</td>
    <td>60.00</td>
    <td>59.50</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>83.33</td>
    <td>64.11</td>
    <td>50.00</td>
    <td>43.33</td>
    <td>41.83</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td colspan="10"><b>vocab size = 4</b></td>
  </tr>
  <tr>
    <td>Llama-3.3 70B</td>
    <td>rules = 1</td>
    <td>40.00</td>
    <td>25.00</td>
    <td>30.00</td>
    <td>40.00</td>
    <td>23.33</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 2</td>
    <td>45.00</td>
    <td>47.33</td>
    <td>10.00</td>
    <td>50.00</td>
    <td>45.83</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>1.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>33.33</td>
    <td>39.50</td>
    <td>10.00</td>
    <td>30.00</td>
    <td>23.85</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>10.83</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Llama-3.1 405B</td>
    <td>rules = 2</td>
    <td>15.00</td>
    <td>12.00</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>13.33</td>
    <td>0.00</td>
    <td>5.00</td>
    <td>0.26</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>33.33</td>
    <td>22.67</td>
    <td>0.00</td>
    <td>3.33</td>
    <td>16.67</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>1.85</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>70.00</td>
    <td>49.17</td>
    <td>50.00</td>
    <td>60.00</td>
    <td>19.50</td>
    <td>50.00</td>
    <td>23.10</td>
    <td>10.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>rules = 2</td>
    <td>80.00</td>
    <td>78.10</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>16.02</td>
    <td>0.00</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>66.67</td>
    <td>43.00</td>
    <td>0.00</td>
    <td>20.00</td>
    <td>11.02</td>
    <td>0.00</td>
    <td>16.67</td>
    <td>6.73</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>rules = 2</td>
    <td>70.00</td>
    <td>59.17</td>
    <td>30.00</td>
    <td>45.00</td>
    <td>27.79</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>0.88</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>73.33</td>
    <td>60.17</td>
    <td>40.00</td>
    <td>43.33</td>
    <td>13.45</td>
    <td>0.00</td>
    <td>3.33</td>
    <td>0.25</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>70.00</td>
    <td>53.33</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>18.33</td>
    <td>10.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>rules = 2</td>
    <td>70.00</td>
    <td>66.67</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>47.50</td>
    <td>20.00</td>
    <td>40.00</td>
    <td>34.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>90.00</td>
    <td>79.17</td>
    <td>50.00</td>
    <td>23.33</td>
    <td>23.33</td>
    <td>0.00</td>
    <td>26.67</td>
    <td>17.58</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 1</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>100.00</td>
    <td>93.33</td>
    <td>90.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td>rules = 2</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>80.00</td>
    <td>75.00</td>
    <td>60.00</td>
    <td>70.00</td>
    <td>69.17</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td></td>
    <td>rules = 3</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>76.67</td>
    <td>78.33</td>
    <td>50.00</td>
    <td>63.33</td>
    <td>62.00</td>
    <td>30.00</td>
  </tr>
</table>

Table 5: Right Output Strictly Local with sample size = 2
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Settings</th>
    <th colspan="3">k = 2</th>
    <th colspan="3">k = 3</th>
    <th colspan="3">k = 4</th>
  </tr>
  <tr>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
  </tr>
  <tr>
    <td colspan="11"><b>vocab size = 2</b></td>
  </tr>
  <tr>
    <td rowspan="4">rules = 1</td>
    <td>0-shot</td>
    <td>60.00</td>
    <td>55.00</td>
    <td>60.00</td>
    <td>30.00</td>
    <td>23.33</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>35.00</td>
    <td>40.00</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>50.00</td>
    <td>60.00</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>60.00</td>
    <td>55.00</td>
    <td>60.00</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 2</td>
    <td>0-shot</td>
    <td>60.00</td>
    <td>65.00</td>
    <td>50.00</td>
    <td>45.00</td>
    <td>60.00</td>
    <td>30.00</td>
    <td>15.00</td>
    <td>8.25</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>65.00</td>
    <td>70.00</td>
    <td>60.00</td>
    <td>40.00</td>
    <td>53.00</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>18.33</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>85.00</td>
    <td>85.00</td>
    <td>80.00</td>
    <td>45.00</td>
    <td>56.67</td>
    <td>20.00</td>
    <td>25.00</td>
    <td>23.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>60.00</td>
    <td>65.00</td>
    <td>40.00</td>
    <td>35.00</td>
    <td>36.67</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 3</td>
    <td>0-shot</td>
    <td>53.33</td>
    <td>68.33</td>
    <td>20.00</td>
    <td>30.00</td>
    <td>46.67</td>
    <td>10.00</td>
    <td>16.67</td>
    <td>8.54</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>76.67</td>
    <td>83.33</td>
    <td>60.00</td>
    <td>43.33</td>
    <td>57.67</td>
    <td>0.00</td>
    <td>20.00</td>
    <td>18.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>86.67</td>
    <td>86.67</td>
    <td>60.00</td>
    <td>26.67</td>
    <td>28.33</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>16.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>90.00</td>
    <td>93.33</td>
    <td>70.00</td>
    <td>46.67</td>
    <td>52.50</td>
    <td>20.00</td>
    <td>16.67</td>
    <td>21.17</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="11"><b>vocab size = 3</b></td>
  </tr>
  <tr>
    <td rowspan="4">rules = 1</td>
    <td>0-shot</td>
    <td>70.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>8.33</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>35.00</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>18.33</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 2</td>
    <td>0-shot</td>
    <td>85.00</td>
    <td>83.33</td>
    <td>60.00</td>
    <td>10.00</td>
    <td>7.50</td>
    <td>0.00</td>
    <td>5.00</td>
    <td>2.50</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>90.00</td>
    <td>95.00</td>
    <td>80.00</td>
    <td>10.00</td>
    <td>13.33</td>
    <td>0.00</td>
    <td>5.00</td>
    <td>2.50</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>65.00</td>
    <td>65.00</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>28.33</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>2.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>65.00</td>
    <td>75.00</td>
    <td>30.00</td>
    <td>5.00</td>
    <td>3.33</td>
    <td>0.00</td>
    <td>5.00</td>
    <td>2.50</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 3</td>
    <td>0-shot</td>
    <td>66.67</td>
    <td>74.17</td>
    <td>20.00</td>
    <td>33.33</td>
    <td>35.36</td>
    <td>0.00</td>
    <td>6.67</td>
    <td>3.43</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>70.00</td>
    <td>69.17</td>
    <td>40.00</td>
    <td>20.00</td>
    <td>22.50</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>13.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>76.67</td>
    <td>76.67</td>
    <td>50.00</td>
    <td>33.33</td>
    <td>43.33</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>13.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>60.00</td>
    <td>60.83</td>
    <td>10.00</td>
    <td>23.33</td>
    <td>31.67</td>
    <td>0.00</td>
    <td>16.67</td>
    <td>19.76</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="11"><b>vocab size = 4</b></td>
  </tr>
  <tr>
    <td rowspan="4">rules = 1</td>
    <td>0-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>50.00</td>
    <td>31.67</td>
    <td>50.00</td>
    <td>20.00</td>
    <td>13.33</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>70.00</td>
    <td>57.00</td>
    <td>60.00</td>
    <td>30.00</td>
    <td>25.00</td>
    <td>30.00</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>20.00</td>
    <td>15.00</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 2</td>
    <td>0-shot</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>15.00</td>
    <td>11.67</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>30.00</td>
    <td>15.00</td>
    <td>23.33</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>80.00</td>
    <td>90.00</td>
    <td>60.00</td>
    <td>15.00</td>
    <td>12.50</td>
    <td>0.00</td>
    <td>15.00</td>
    <td>10.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>15.00</td>
    <td>18.33</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 3</td>
    <td>0-shot</td>
    <td>53.33</td>
    <td>68.33</td>
    <td>20.00</td>
    <td>6.67</td>
    <td>5.00</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>5.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>66.67</td>
    <td>70.83</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>47.50</td>
    <td>0.00</td>
    <td>3.00</td>
    <td>5.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>66.67</td>
    <td>73.33</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>55.00</td>
    <td>0.00</td>
    <td>3.33</td>
    <td>3.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>60.00</td>
    <td>71.67</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>39.24</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
</table>

Table 6: Input Strictly Local with sample size = 2 with few-shot example
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Settings</th>
    <th colspan="3">k = 2</th>
    <th colspan="3">k = 3</th>
    <th colspan="3">k = 4</th>
  </tr>
  <tr>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
  </tr>
  <tr>
    <td colspan="11"><b>vocab size = 2</b></td>
  </tr>
  <tr>
    <td rowspan="4">rules = 1</td>
    <td>0-shot</td>
    <td>50.00</td>
    <td>45.00</td>
    <td>50.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>40.00</td>
    <td>33.33</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>5.00</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>80.00</td>
    <td>75.00</td>
    <td>70.00</td>
    <td>30.00</td>
    <td>25.00</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>13.33</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>20.00</td>
    <td>15.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>15.00</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td colspan="11">---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ----------------------</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 2</td>
    <td>0-shot</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>10.00</td>
    <td>8.33</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>10.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>80.00</td>
    <td>85.00</td>
    <td>70.00</td>
    <td>30.00</td>
    <td>30.83</td>
    <td>10.00</td>
    <td>30.00</td>
    <td>19.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>85.00</td>
    <td>85.00</td>
    <td>80.00</td>
    <td>20.00</td>
    <td>21.67</td>
    <td>10.00</td>
    <td>25.00</td>
    <td>27.90</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>75.00</td>
    <td>80.00</td>
    <td>60.00</td>
    <td>25.00</td>
    <td>20.83</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>20.83</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="11">---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ----------------------</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 3</td>
    <td>0-shot</td>
    <td>56.67</td>
    <td>65.00</td>
    <td>0.00</td>
    <td>6.67</td>
    <td>8.33</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>12.83</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>40.00</td>
    <td>42.00</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>11.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>80.00</td>
    <td>75.00</td>
    <td>70.00</td>
    <td>33.33</td>
    <td>48.33</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>28.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>33.33</td>
    <td>39.17</td>
    <td>0.00</td>
    <td>16.67</td>
    <td>26.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="11"><b>vocab size = 3</b></td>
  </tr>
  <tr>
    <td rowspan="4">rules = 1</td>
    <td>0-shot</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>20.00</td>
    <td>12.50</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>13.33</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>40.00</td>
    <td>23.33</td>
    <td>40.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>30.00</td>
    <td>23.33</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>80.00</td>
    <td>75.00</td>
    <td>80.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td colspan="11">---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ----------------------</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 2</td>
    <td>0-shot</td>
    <td>35.00</td>
    <td>33.67</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>6.93</td>
    <td>10.00</td>
    <td>25.00</td>
    <td>15.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>80.00</td>
    <td>76.67</td>
    <td>80.00</td>
    <td>30.00</td>
    <td>30.33</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>15.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>50.00</td>
    <td>55.00</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>38.33</td>
    <td>0.00</td>
    <td>25.00</td>
    <td>21.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>20.00</td>
    <td>30.00</td>
    <td>0.00</td>
    <td>20.00</td>
    <td>35.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="11">---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ----------------------</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 3</td>
    <td>0-shot</td>
    <td>40.00</td>
    <td>65.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>18.33</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>2.78</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>70.00</td>
    <td>66.67</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>24.83</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>20.30</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>83.33</td>
    <td>90.00</td>
    <td>60.00</td>
    <td>33.33</td>
    <td>40.83</td>
    <td>0.00</td>
    <td>30.00</td>
    <td>43.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>70.00</td>
    <td>78.33</td>
    <td>30.00</td>
    <td>23.33</td>
    <td>44.50</td>
    <td>0.00</td>
    <td>16.67</td>
    <td>18.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="11"><b>vocab size = 4</b></td>
  </tr>
  <tr>
    <td rowspan="4">rules = 1</td>
    <td>0-shot</td>
    <td>50.00</td>
    <td>29.00</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>13.33</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>20.00</td>
    <td>15.00</td>
    <td>20.00</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>30.00</td>
    <td>25.00</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td colspan="11">---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ----------------------</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 2</td>
    <td>0-shot</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>15.96</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>55.00</td>
    <td>56.67</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>28.33</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>55.00</td>
    <td>50.00</td>
    <td>20.00</td>
    <td>35.00</td>
    <td>55.00</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>10.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>0.00</td>
    <td>30.00</td>
    <td>33.33</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="11">---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ----------------------</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 3</td>
    <td>0-shot</td>
    <td>50.00</td>
    <td>52.50</td>
    <td>20.00</td>
    <td>6.67</td>
    <td>6.00</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>6.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>56.67</td>
    <td>68.33</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>36.25</td>
    <td>0.00</td>
    <td>16.7</td>
    <td>22.83</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>66.67</td>
    <td>67.50</td>
    <td>50.00</td>
    <td>16.67</td>
    <td>26.67</td>
    <td>0.00</td>
    <td>6.67</td>
    <td>15.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>3.33</td>
    <td>3.33</td>
    <td>0.00</td>
    <td>23.33</td>
    <td>40.83</td>
    <td>0.00</td>
    <td>3.33</td>
    <td>3.33</td>
    <td>0.00</td>
  </tr>
</table>

Table 7: Left Output Strictly Local with sample size = 2 with few-shot example
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Settings</th>
    <th colspan="3">k = 2</th>
    <th colspan="3">k = 3</th>
    <th colspan="3">k = 4</th>
  </tr>
  <tr>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
    <th>recall</th>
    <th>precision</th>
    <th>compatibility</th>
  </tr>
  <tr>
    <td colspan="11"><b>vocab size = 2</b></td>
  </tr>
  <tr>
    <td rowspan="4">rules = 1</td>
    <td>0-shot</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>40.00</td>
    <td>50.00</td>
    <td>30.00</td>
    <td>18.33</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>40.00</td>
    <td>35.00</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>25.00</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>40.00</td>
    <td>50.00</td>
    <td>10.00</td>
    <td>2.50</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 2</td>
    <td>0-shot</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>15.00</td>
    <td>18.33</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>18.67</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>40.00</td>
    <td>42.50</td>
    <td>20.00</td>
    <td>30.00</td>
    <td>33.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>70.00</td>
    <td>80.00</td>
    <td>60.00</td>
    <td>45.00</td>
    <td>43.33</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>27.50</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>90.00</td>
    <td>45.00</td>
    <td>41.67</td>
    <td>30.00</td>
    <td>15.00</td>
    <td>15.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 3</td>
    <td>0-shot</td>
    <td>63.33</td>
    <td>76.67</td>
    <td>30.00</td>
    <td>23.33</td>
    <td>24.17</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>7.83</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>40.00</td>
    <td>47.83</td>
    <td>0.00</td>
    <td>20.00</td>
    <td>24.17</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>48.33</td>
    <td>20.00</td>
    <td>16.67</td>
    <td>18.33</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>43.33</td>
    <td>46.67</td>
    <td>10.00</td>
    <td>16.67</td>
    <td>25.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="11"><b>vocab size = 3</b></td>
  </tr>
  <tr>
    <td rowspan="4">rules = 1</td>
    <td>0-shot</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>25.00</td>
    <td>30.00</td>
    <td>30.00</td>
    <td>7.26</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>70.00</td>
    <td>65.00</td>
    <td>60.00</td>
    <td>40.00</td>
    <td>35.00</td>
    <td>40.00</td>
    <td>20.00</td>
    <td>15.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>80.00</td>
    <td>75.00</td>
    <td>70.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>30.00</td>
    <td>18.33</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>70.00</td>
    <td>65.00</td>
    <td>60.00</td>
    <td>40.00</td>
    <td>35.00</td>
    <td>40.00</td>
    <td>20.00</td>
    <td>15.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 2</td>
    <td>0-shot</td>
    <td>30.00</td>
    <td>60.00</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>10.93</td>
    <td>0.00</td>
    <td>25.00</td>
    <td>19.17</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>65.00</td>
    <td>70.00</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>45.00</td>
    <td>10.00</td>
    <td>15.00</td>
    <td>11.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>60.00</td>
    <td>70.00</td>
    <td>40.00</td>
    <td>50.00</td>
    <td>44.17</td>
    <td>10.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>65.00</td>
    <td>70.00</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>45.00</td>
    <td>10.00</td>
    <td>15.00</td>
    <td>11.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 3</td>
    <td>0-shot</td>
    <td>30.00</td>
    <td>45.00</td>
    <td>0.00</td>
    <td>26.67</td>
    <td>22.67</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>4.58</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>80.00</td>
    <td>80.83</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>37.59</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>11.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>73.33</td>
    <td>71.67</td>
    <td>30.00</td>
    <td>26.67</td>
    <td>34.50</td>
    <td>0.00</td>
    <td>33.33</td>
    <td>47.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>80.00</td>
    <td>80.83</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>37.60</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>11.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="11"><b>vocab size = 4</b></td>
  </tr>
  <tr>
    <td rowspan="4">rules = 1</td>
    <td>0-shot</td>
    <td>40.00</td>
    <td>25.00</td>
    <td>30.00</td>
    <td>40.00</td>
    <td>23.33</td>
    <td>20.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>60.00</td>
    <td>50.00</td>
    <td>31.67</td>
    <td>50.00</td>
    <td>10.00</td>
    <td>10.00</td>
    <td>10.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>80.00</td>
    <td>68.33</td>
    <td>80.00</td>
    <td>60.00</td>
    <td>33.33</td>
    <td>30.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>70.00</td>
    <td>40.00</td>
    <td>19.50</td>
    <td>40.00</td>
    <td>20.00</td>
    <td>20.00</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 2</td>
    <td>0-shot</td>
    <td>45.00</td>
    <td>47.33</td>
    <td>10.00</td>
    <td>50.00</td>
    <td>45.83</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>1.67</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>80.00</td>
    <td>70.00</td>
    <td>40.00</td>
    <td>45.00</td>
    <td>61.67</td>
    <td>10.00</td>
    <td>5.00</td>
    <td>10.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>65.00</td>
    <td>65.00</td>
    <td>40.00</td>
    <td>45.00</td>
    <td>52.50</td>
    <td>20.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>75.00</td>
    <td>75.00</td>
    <td>70.00</td>
    <td>45.00</td>
    <td>48.33</td>
    <td>20.00</td>
    <td>15.00</td>
    <td>30.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td rowspan="4">rules = 3</td>
    <td>0-shot</td>
    <td>33.33</td>
    <td>39.50</td>
    <td>10.00</td>
    <td>30.00</td>
    <td>23.85</td>
    <td>0.00</td>
    <td>10.00</td>
    <td>10.83</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>66.67</td>
    <td>71.83</td>
    <td>30.00</td>
    <td>23.33</td>
    <td>30.83</td>
    <td>0.00</td>
    <td>3.33</td>
    <td>2.50</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>83.33</td>
    <td>86.50</td>
    <td>50.00</td>
    <td>26.67</td>
    <td>37.50</td>
    <td>0.00</td>
    <td>13.33</td>
    <td>28.83</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>76.67</td>
    <td>80.00</td>
    <td>50.00</td>
    <td>40.00</td>
    <td>51.67</td>
    <td>10.00</td>
    <td>13.33</td>
    <td>20.00</td>
    <td>0.00</td>
  </tr>
</table>

Table 8: Right Output Strictly Local with sample size = 2 with few-shot example

Reviewer paper summaries: This paper introduces InductionBench, a benchmark aims to evaluate the inductive reasoning abilities of LLM using subregular function classes that are provably learnable in polynomial time and data. It first proposes several function-based ways to automatically construct a benchmark, then evaluate many recent LLMs on this benchmark. Evaluation results prove that existing LLMs lack inductive reasoning abilities on the proposed benchmark.

This paper introduces InductionBench, a benchmark designed to evaluate the inductive reasoning capabilities of large language models (LLMs). While deductive reasoning has been extensively explored in benchmarks for mathematical and logical reasoning, inductive reasoning—where a model must infer underlying rules from observed data—remains underexplored. The benchmark is grounded in the subregular hierarchy and evaluates models based on their ability to infer string-to-string transformations from limited input-output pairs. The paper demonstrates that even state-of-the-art LLMs struggle with basic inductive tasks, suggesting a significant limitation in current models' reasoning capabilities. The authors conduct extensive experiments, systematically varying parameters such as context window size, vocabulary size, and the minimal number of rules required for function representation. They introduce three key metrics—recall, precision, and compatibility—to assess model performance and find that, despite improvements in many areas, modern LLMs fail to generalize effectively in inductive reasoning tasks.
Summary of strengths: 1. The evaluated models are comprehensive.
2. The paper is written in a clear way.

1. **Novelty and Scope**: The paper addresses an underexplored aspect of LLM capabilities, specifically their inductive reasoning skills, which is crucial for tasks involving scientific discovery and real-world inference.
2. **Theoretically Grounded Benchmark**: The use of subregular functions ensures that tasks are well-defined, learnable in polynomial time, and systematically scalable in complexity.
3. **Rigorous Experimental Design**: The authors evaluate multiple state-of-the-art LLMs under controlled conditions, systematically varying task complexity and input conditions.
4. **Quantitative and Qualitative Analysis**: The study provides a well-structured evaluation of model performance, breaking down errors into categories that highlight key weaknesses in current models.
5. **Clear Implications for Future Research**: The findings highlight critical gaps in LLMs’ reasoning abilities, motivating further research into improving models' generalization and learning strategies.
Summary of weaknesses: 1. Although the function generation method is interesting, the constructed benchmark is similar to the one in "Instruction Induction: From Few Examples to Natural Language Task Descriptions", while the latter one includes simple transformation functions to build their benchmark. The authors should discuss their work with this one.
2. More fine-grained analysis can be added. For example, while all the models except o3-mini gets 0 compatibility score in Table 1, are their results similar?

1. **Limited Real-World Relevance**: The benchmark is synthetic and may not fully capture the complexities of real-world inductive reasoning, where ambiguity and noise play significant roles.
2. **Benchmark “Hackability”**: The authors acknowledge that the benchmark could be solved via explicit polynomial-time learning algorithms rather than genuine inductive reasoning, which may limit its ability to test models in a truly open-ended setting.
3. **Lack of Theoretical Explanation for Failure**: While the paper thoroughly documents LLM failures, it does not delve deeply into why these failures occur from a model architecture or training perspective.
4. **Narrow Evaluation Scope**: The study only examines a subset of inductive reasoning tasks (subregular functions), potentially overlooking other forms of inductive reasoning (e.g., causal inference, abstraction learning).
5. **Failure to Address Scaling Trends**: The paper does not provide insight into whether larger models with more training data would overcome these limitations, leaving the open question of whether this deficiency is fundamental or an artifact of current training methods.
Comments, suggestions and typos: 1. Could you list some examples of the actual generated tasks?

1. **Clarity of Task Description**: While the paper provides formal definitions, it would benefit from additional intuitive explanations and real-world analogies to help readers unfamiliar with formal language theory.
2. **Benchmark Expansion**: Including additional types of inductive reasoning tasks beyond string-to-string transformations would make the benchmark more comprehensive.
3. **Comparisons with Human Performance**: Adding a baseline comparison with human performance on these tasks could help contextualize model weaknesses.
4. **Typographical Errors**:
   - Page 2, Line 22: "modelw" should be "models"
   - Page 8, Line 622: "provably learnable" -> consider rephrasing for clarity.
Overall assessment scores: 3.5

2.5
Metareview score: 3
Metareview: The authors introduce a new benchmark that targets the assessment of LLMs inductive reasoning properties. Building on the subregular hierarchy and its corresponding polynomial guarantees for time and data learnability, the benchmark assesses how effectively LLMs infer underlying transformation functions from finite datapoints. It also measures the extent to which the models generate minimal, non-redundant hypotheses, thereby providing insights into their generalization capabilities.
Summary of reasons to publish: - Convincing theoretical framing with a supporting empirical analysis.
- Aligned to the stated contributions.