IDEA: Enhancing the Rule Learning Ability of Large Language Model Agent through Induction, Deduction, and Abduction

## Abstract

Abstract While large language models (LLMs) have been thoroughly evaluated for deductive and inductive reasoning, their proficiency in abductive reasoning and holistic rule learning in interactive environments remains less explored. We introduce RULEARN , a novel benchmark specifically designed to assess the rule-learning abilities of LLM agents in interactive settings. In RULEARN , agents strategically interact with simulated environments to gather observations, discern patterns, and solve complex problems. To enhance the rule-learning capabilities for LLM agents, we propose IDEA , a novel reasoning framework that integrates the process of I nduction, DE duction, and A bduction. The IDEA agent generates initial hypotheses from limited observations through abduction, devises plans to validate these hypotheses or leverages them to solve problems via deduction, and refines previous hypotheses using patterns identified from new observations through induction, dynamically establishing and applying rules that mimic human rule-learning behaviors. Our evaluation of the IDEA framework, which involves five representative LLMs, demonstrates significant improvements over the baseline. Furthermore, within this framework, our comparison with 50 human participants reveals notable discrepancies in rule-learning behaviors. LLM agents tend to generate plausible initial hypotheses but struggle to refine them through interaction. Conversely, humans, despite sometimes overlooking initial details, excel at incorporating feedback and continuously improving their hypotheses. We believe our benchmark, RULEARN , will serve as a valuable and challenging resource, and that the IDEA framework will provide crucial insights for the development of LLM agents capable of human-like rule learning in real-world scenarios. We will release our code and data upon acceptance of the paper.

## 1 Introduction

Figure: Figure 1: The reasoning cycle of rule learning encompasses abduction, deduction, and induction.
Refer to caption: x1.png

One major pillar of human intelligence is the ability to discern rules and apply them. We identify patterns, formulate hypotheses, and refine them by interacting with the environment. This exploratory process traditionally involves three stages: abduction, deduction, and induction.
According to Charles Peirce’s definition (Frankfurt, ; Peirce, ), the rule-learning loop typically begins with an explanatory hypothesis that arises from abduction. This is followed by iterative experiments guided by the hypothesis, known as deduction, which leads to the modification and refinement of the hypothesis through induction (As shown in Figure 1). Whether the hypothesis emerges from rigorous logical reasoning or a more creative process is debated, but the cyclic nature of this reasoning is pivotal in advancing our understanding and problem-solving capabilities in various disciplines. This establishes the foundation of how humans learn the rules of the world and apply them to effect change.

For such cyclic reasoning to function effectively, it requires an interactive foundation. In the real world, the observations we reason from are gathered through interaction. Then, through alternating cycles of refining hypothesis and interaction, we iteratively collect evidence and refine our hypotheses, drawing ever closer to the truth. The dynamic interplay between interaction and different stages of the reasoning loop within an interactive environment is the key to humans’ reasoning effectiveness. However, recent studies focusing on reasoning often lack this dynamic and interdependent nature, as abduction, induction, and deduction are tested one at a time in non-interactive environments (Bowen et al., ; Wang et al., ; Saparov et al., ; Liu et al., ).

To address this gap, we introduce RULEARN, a benchmark designed to evaluate the rule-learning abilities of LLM agents in interactive environments. RULEARN consists of puzzles with hidden rules set in a text-based environment, where agents begin exploration without prior knowledge of these rules. The agent’s objective is to solve the puzzle by interacting with the environment. At each step, the agent selects an action from a predefined set, executes it, and gathers observations to gain insights into the environment. Successfully solving the puzzles requires the agent to strategically select actions, efficiently gather pattern-revealing observations, and accurately reason from them to infer the hidden rules. RULEARN presents substantial challenges, as agents must rely entirely on observations generated by their chosen actions to discern rules. If these observations fail to reveal a clear pattern, the agent is likely to fail.

Figure: Figure 2: A simplified puzzle in the RULEARN benchmark and the IDEA agent’s workflow (in real puzzles, agents have fewer initial observations and more complex rules). The agent generates an initial hypothesis through abduction, develops an exploration plan via deduction, and refines its hypothesis using induction. For example, the IDEA agent first hypothesizes that the password is the number of the blue paintings, tests this by entering code 003, and adjusts its strategy based on the results.
Refer to caption: x2.png

We design three types of environments within RULEARN to evaluate the rule-learning ability in different scenarios: (1) The Function Operator: The agent is tasked with determining the coefficients of mathematical functions defined by hidden expressions. Agents can assign various values to the input variables and observe the outputs, using this information to hypothesize the function’s form. The challenge lies in efficiently selecting input values that reveal the underlying structure and accurately computing the coefficients based on limited observations. (2) The Escape Room: The agent is tasked with deciphering the passcode to exit an escape room. A hidden rule determines how the objects in the room infer the passcode digits. Agents interact with these objects to gather clues and input passcodes into the door. The door provides hints indicating which digits are correct. Based on these hints, agents formulate hypotheses and infer the relationship between the objects and the passcode. (3) The Reactor: The agent is tasked with synthesizing target strings using a reactor with a hidden string-combining rule. Agents need to experiment with different inputs and analyze outputs to deduce the reactor’s transformation rule and achieve the desired outcome. Detailed puzzle definitions are provided in § 3.

To tackle the challenge in RULEARN, we introduce IDEA, a novel reasoning framework that integrates the process of Induction, DEduction, and Abduction. The IDEA agent employs these reasoning processes iteratively to explore the environments, learn rules, and achieve goals. In the abduction phase, the IDEA agent generates an initial hypothesis from limited observations. During the deduction phase, the IDEA agent creates and executes plans to meet objectives or test its hypothesis. In the induction phase, the IDEA agent refines its hypothesis based on new observations, enhancing their accuracy and robustness. This iterative cycle enables the LLM agent to continually improve the learned rules through environmental feedback. An overview of how the IDEA agent solves puzzles in RULEARN is shown in Figure 2.

We evaluat the IDEA framework using five popular LLMs—GPT-3.5-Turbo, GPT-4o, Gemma-7B, Llama3-8B and Llama3-70B. The experimental results demonstrate that IDEA significantly improves rule-learning proficiency and puzzle-solving performance of the LLM agent within the RULEARN benchmark. The IDEA agent increases success rates by approximately 10% compared to the baseline LLM agent. Specifically, the action of the IDEA agent aims to validate or leverage current hypothesis, providing important guidance in an environment with hidden rules. Without the guidance of the hypothesis, the LLM agent will always choose the most direct actions attempting to quickly solve the puzzle. Such actions, taken without understanding the hidden rules, are akin to navigating without a compass—efforts are made, but they often lead to ineffective or misguided outcomes. Based on our experiments, the IDEA agent achieves 47.6% fewer repeated actions, yielding more diverse observations and finally solving the puzzles with a better understanding of the hidden rule. These improvements suggest that IDEA enables LLMs to better adapt and learn rules in unfamiliar environments, narrowing the gap between human and AI rule-learning abilities.

Within the IDEA framework, we conduct a study to compare the rule-learning behaviors and performances between LLMs and 50 human participants solving puzzles.
Our findings reveal that, despite the improvements brought by the IDEA framework, LLMs still face the following challenges when compared to human participants:
(1) LLM agents continue to struggle with efficiently exploring unfamiliar environments. The new observations gathered usually do not provide enough information to help reveal the underlying rules, moreover, LLM agents tend to repeat previous actions that do not yield new information. (2) LLMs fall short in deducting valid plans to verify current hypotheses and guide future exploration. (3) LLMs fall short to correct previous hypotheses when they contradict new observations and are less capable of refining a hypothesis to make it more robust and encompassing all observations. These findings provide valuable insights for the development of LLM agents capable of human-like rule learning in real-world scenarios.

## 2 Related Works

### 2.1 LLM agent

Agents driven by large language models equipped with memory, tool use, planning, and reasoning capabilities have shown significant advancements in understanding complex tasks and achieving notable performance (Wang et al., ). Techniques like memory summarization (Chen et al., ; Zhou et al., ; Wang et al., ) and retrieval (Andreas, ; Park et al., ; Zhong et al., ) assist LLM agents in filtering relevant information based on the current context (Zhang et al., ). Additionally, the integration of tool use enables LLM agents to perform better in knowledge-grounded tasks. For instance, some studies (Nakano et al., ; Lu et al., ; Shi et al., ) have enabled LLM agents to access up-to-date information from external databases to provide correct answers, while others (Schick et al., ) have facilitated the use of various tools like calculators to enhance solution accuracy. Planning modules (Yuan et al., ; Shen et al., ) allow these agents to decompose complex tasks into simpler components, achieving significant improvements. Moreover, employing search algorithms (Yao et al., ; Besta et al., ) helps LLM agents reach solutions in fewer steps. Building on this foundation, we are interested in whether LLM agents can derive new knowledge–learning rules through interacting with the environment, rather than merely answering questions. Recent studies have begun investigating different types of reasoning in LLMs, including abduction, deduction, and induction (Bowen et al., ; Wang et al., ; Saparov et al., ; Cheng et al., ; Yang et al., ). However, existing research mostly focuses on isolated aspects of the rule-learning process. Consequently, the comprehensive rule-learning abilities of LLMs in interactive environments remain largely underexplored, a topic that will be further examined in Section 2.2.

### 2.2 Rule Learning

Rule learning—involving evidence gathering, pattern identification, hypothesis formulation, and validation—is gaining focus in recent research. Yang et al. () and Zhu et al. () improved question-answering systems by formulating optimal response rules, while Shi et al. () advanced event prediction by enabling LLMs to hypothesize causes of previous events. Some works also evaluate LLMs rule-learning abilities (Liu et al., ). However, existing studies adopt static, non-interactive settings for rule learning, where LLMs are provided with complete information to formulate rules. This approach has two significant limitations: it contrasts with real-world rule learning, which involves actively and strategically gathering evidence, and it restricts agents from conducting experiments to validate the rules they learn. In real-world scenarios, the process of gathering observations, validating hypotheses, and refining hypotheses is iterative. These simplified, static settings fail to adequately assess an agent’s capabilities in real, dynamic environments. Even in studies with interactive environments (Xu et al., ; Montes et al., ), information gathering, rule generation, and application are treated as independent objectives or are incomplete. In reality, these processes are dynamic and interdependent: generated hypotheses guide actions, and new observations refine those hypotheses. In contrast, RULEARN provides a realistic setting that allows LLMs to engage in dynamic, interdependent rule-learning processes, closely mirroring real-world scenarios. To effectively navigate and learn within this environment, IDEA equips LLM agents with the ability to emulate human reasoning by managing these interdependent processes.

## 3 The RULEARN Benchmark

We develop three puzzle sets, the Function Operator, the Escape Room, and the Reactor, each comprising 20 puzzles associated with environments of varying complexity.

The Function Operator. This puzzle type simulates scenarios where systemic theories or established knowledge (in this case, mathematics) is applicable. The agent interacts with a set of univariate multi-term equations involving integer parameters from [0,9] and elementary functions of the variable $x$, selected from $f(x)\in\{x^{0},x^{1},x^{2},\sin(x),\frac{1}{x},|x|,-x\}$. The agent is provided with the number of functions, the presence of specific parameters in each function (the exact numerical values of these parameters are unknown and represented by letters), and the types of elementary functions involved in the current puzzle. The goal of the agent is to deduce the values of these parameters. For example, in one puzzle, the ground truth is $\mathbf{F_{1}}(x)=a\sin(x)+b\times\frac{1}{x}$, $\mathbf{F_{2}}(x)=ax^{2}$ where $a=3$ and $b=2$. The agent knows the following information: There are three elementary functions in this puzzle $\{\sin(x),\frac{1}{x},x^{2}\}$, there are two functions $\mathbf{F_{1}}(x)$ and $\mathbf{F_{2}}(x)$, $\mathbf{F_{1}}(x)$ has 2 terms and parameters $a,b$ in it, and $\mathbf{F_{2}}(x)$ has 1 term and one parameter $a$. To solve the puzzle, the agent must interact with the environment through a defined action space: selecting a function and assigning values to $x$, then observing the resultant output. For example, assigning values 1 and 2 to $\mathbf{F_{2}}$ reveals a quadratic increase in output, indicating the presence of $x^{2}$ in $\mathbf{F_{2}}$. Similarly, assigning a value of 1 to $\mathbf{F_{1}}$ results in a floating-point output, rather than an integer, suggesting the inclusion of trigonometric components, confirming that $\sin(x)$ is a component of $\mathbf{F_{1}}$. Each puzzle features a different number of functions, unknown parameters, and elementary functions in use.

The Escape Room. This environment simulates scenarios where no established knowledge is applicable. We create a fictitious environment: an agent is in an escape room–an art gallery, tasked with deciphering a password to unlock a code-secured door and escape. The password is a 3-digit number where each digit represents the count of paintings of a specific type—watercolor, oil, or acrylic—that share a given color. The agent can explore the environment and investigate paintings, receiving brief descriptions such as This is an acrylic painting of a green jungle, providing information about the painting’s type and color. The agent only knows that the password is a 3-digit number and receives a hint to focus on a specific color relevant to the password. After forming an initial hypothesis and inputting the password into the door, the door provides hints indicating which digits are correct. Based on these hints, the agent refines its hypotheses and infers the relationship between the painting attributes and the passcode. To prevent the problem from being solved merely by brute force, the specific color associated with the password changes after every three failed attempts. Each puzzle presents a varying number of paintings; paintings in the same room as the agent are directly visible, while those in different rooms are not, requiring the agent to take actions such as moving to another gallery to reveal all the paintings.

The Reactor. This environment type also simulates scenarios where no established knowledge is applicable. However, unlike in the Escape Room where agents only input passwords to test hypotheses, in the Reactor, agents are challenged to design more fine-grained experiments to uncover hidden rules. Specifically, the agent’s task is to synthesize target strings using a reactor governed by a hidden string-combining rule. These strings are represented by sequences of alphabetic letters, such as A, B, AABB, and CAB. The reactor permits the agent to input two strings, initiating a reaction that produces a new string for use in subsequent experiments. The agent’s objective is to decipher the specific rules that govern string synthesis by methodically testing different string combinations, with the ultimate goal of synthesizing the target string using the discovered rules. We have designed four types of rules, detailed in Table 1.

The detailed statistics for each puzzle type and example puzzles are provided in Appendix A.2 and A.5, respectively. Note that the RULEARN benchmark is designed to simulate real-world interactions and is adaptable to complex text environments with diverse rules. To achieve this, we do not restrict the representation of rules to a specific formal language, as doing so would limit realism and complexity. Moreover, predefining the rule format would provide agents with prior knowledge, contradicting real-world conditions. Therefore, we enable the LLM agents to use natural language to describe rules in RULEARN, promoting generalizability and future extensions of the benchmark.

**Table 1: The reacting rules in the Reactor Puzzle. All letters are functionally equivalent and exhibit no special behaviors. Identical symbols represent the same letter, while different symbols denote different letters. Each puzzle operates under one specific rule. The Middle Insertion rule inserts the shorter string into the longer string; if the length of the longer string is odd, the shorter string is inserted just to the right of the center. If both strings are of equal length, the second string is inserted into the middle of the first string. The Prefix Replacement rule retains the prefix of the longer string and concatenates it with the shorter string, dropping the tail of the longer string results in two output strings. There are two special cases where the strings are simply concatenated in order.**
| Rule Description | Example Reaction 1 | Example Reaction 2 | Special Case 1 | Special Case 2 |
| --- | --- | --- | --- | --- |
| Simple Concatenation | AB + C = ABC | AB + CDE = ABCDE | — | — |
| Reverse Concatenation | AB + C = CAB | AB + CDE = CDEAB | — | — |
| Middle Insertion | AB + C = ACB | AB + CDE = CDABE | A + B = AB | — |
| Prefix Replacement | AB + C = AC + B | AB + CDE = CAB + DE | AB + CD = ABCD | AA + A = AAA |

## 4 The IDEA Agent

We introduce IDEA, a novel reasoning framework that integrates the process of Induction, DEduction, and Abduction to learn rules in interactive environments.
The IDEA agent consists of the following components: Goal(G), Action Space($\mathbb{A}$), Memory($\mathbb{M}$), Hypothesis(H), and Plan(P), which are elaborated in Table 2.

**Algorithm 1 IDEA Agent Rule-learning Loop**
| The IDEA Agent<br>Component | Definition |
| --- | --- |
| Goal(G) | Goal of the agent<br>in the current puzzle. |
| Action Space($\mathbb{A}$) | Set of actions the<br>agent can take,<br>including<br>abductive action,<br>deductive action,<br>inductive action,<br>as well as the set<br>of interactive actions<br>defined by the puzzle. |
| Memory($\mathbb{M}$) | Set of natural language<br>strings to record<br>all interaction results<br>till the current step. |
| Plan(P) | Generated plans to<br>guide future actions. |

Upon beginning to explore a puzzle, we initialize the agent memory with an initial observation of the environment. The agent’s goal is initialized with the objective of the puzzle, e.g., synthesize a target string for a Reactor puzzle. The agent’s action space is initialized as the set of interactive actions defined by the puzzle, such as choosing two strings and running the Reactor, as well as establish the initial hypothesis (abductive action), devises a plan to validate or leverage hypothesis (deductive action), and refining the current hypothesis (inductive action).

The IDEA agent begins with an abductive action to generate an initial hypothesis, followed by a deduction step to create a new plan. Based on this plan, the agent interacts with the environment. Upon receiving feedback from the environment as a new observation, the agent may take an inductive action to refine the hypothesis or perform another interaction with the environment. Deductive action is invoked to adjust the plan every time the hypothesis changes. This reasoning loop continues until the puzzle is solved or a maximum number of steps is reached. After each step, the results are appended to the agent’s memory, including interaction outcomes and any modifications to the hypothesis or plan. We provide a simplified algorithm demonstrating how the IDEA agent operates in Algorithm 1. Specifically, at each step, we prompt the LLM to reflect on the information recorded in the IDEA agent’s components to make decisions and take actions. We employ the chain-of-thought (COT) reasoning (Wei et al., ) for all prompts. Detailed prompts for each type of action are available in Appendix A.4.
More detailed implementation of the agent can be found in Appendix A.3.1.
Similar to real-life scenarios, when agents solve tasks in RULEARN puzzles, they do not know the outcomes in advance. Consequently, it is challenging to decide when to refine or change their hypothesis and plans, as well as what interactive actions to take to gather pattern-revealing observations. A detailed example of the IDEA agent solving the Reactor Puzzle is provided in Figure 3.

### 4.1 Experiment Settings

To evaluate the effectiveness of IDEA, we respectively initialize it with three popular open-source LLMs, including Gemma-7B (Team et al., ), Llama3-8B, and Llama3-70B (Dubey et al., ), and two closed-source LLMs, GPT-3.5-Turbo (Dubey et al., ) and GPT-4o (^1^11https://openai.com/index/gpt-4o-system-card/). We compare the IDEA agent against the following two variants:

- •
Baseline Agent: The Baseline agent lacks the reasoning loop of abduction, deduction, and induction and does not generate hypotheses or plans. At each step, it selects an interactive action solely based on its current memories and the goal.
- •
Oracle-rule Agent: Even if the agent could successfully learn the correct rule, applying the learned rule to solve the puzzle is non-trivial. The Oracle-rule agent serves as a special control group to establish the Oracle performance with the ground-truth rule provided at the beginning. Specifically: 1) For the Function Operator puzzles, agents are given the exact forms of the functions. Their task is to derive the values of the coefficients. 2) For the Escape Room puzzles, agents are provided with how the password is constructed from the objects in the room. Their task is to derive the password using the provided rule. 3) For the Reactor puzzles, the reaction rule is given to the agents in natural language accompanied with examples. The agents only need to synthesize the target strings using the provided rules.

Each variant is evaluated on all three puzzle types (totaling 60 puzzles), with five trials per puzzle, resulting in 300 tests per variant. In our initial experiments, we found that using a higher temperature enhanced the agent’s creativity, allowing it to generate more diverse and higher-quality hypotheses that aided in discovering the underlying rules. Therefore, we set the temperature to $1.0$. We apply the COT for the prompting of each step for all agents.
Additionally, since we observed that the success rate does not improve after 15 interactive steps, we set the maximum interaction step count to 15. The agent is considered to have failed a puzzle if it doesn’t solve it within 15 interactive steps.

Figure: Figure 3: An example of the IDEA agent solving a Reactor puzzle. At each step, the agent must choose whether to interact with the environment or adjust its hypothesis and plan based on current observations. If observed facts contradict the existing hypothesis, the agent is expected to refine its hypothesis. The refined hypothesis and plan will then guide subsequent exploration.
Refer to caption: x3.png

### 4.2 Human participants

To compare the behavior and performance of humans with that of LLMs in the abduction, deduction, and induction reasoning processes, we recruited 50 participants and randomly selected 30 puzzles (10 from each of the three puzzle types), assigning each participant three puzzles—one from each type. Each puzzle was attempted by five different individuals, ensuring no prior knowledge of the underlying rules and maintaining an unbiased sample across puzzle types. Details about Institutional Review Board (IRB) approval and participant recruitment are provided in §7. The human participants are instructed to operate under the same reasoning procedure as IDEA.
Failure to solve a puzzle within 15 steps resulted in marking the attempt as unsuccessful, ensuring a fair comparison.

### 4.3 Main Results

We calculated the Wilson Confidence Interval (CI) (Brown et al., ) for puzzle solving success rate across different variants. The detailed results are displayed in Table 3.

**Table 3: Puzzle Success Rate (95% CI). The table displays the success rate for each setting, with each pair of numbers in parentheses representing the lower and upper bounds of the 95% confidence interval for the success rate. The human results are based on a randomly selected 50% subsample of the puzzles. We can see that across all LLMs, IDEA achieves consistent significant improvements except for the Reactor puzzles. We use Boldface to highlight the performance comparisons between Baseline and IDEA Agents with GPT-4o.**
| Setup | LLMs | All Types (%) | Function Operator (%) | Escape Room (%) | Reactor (%) |
| --- | --- | --- | --- | --- | --- |
| Oracle-rule Agent | Gemma-7B | (0.71, 3.84) | (0.0, 3.7) | (1.57, 9.84) | (0.18, 5.45) |
| Llama3-8B | (7.09, 13.92) | (2.78, 12.48) | (12.51, 27.78) | (2.15, 11.18) |  |
| Llama3-70B | (33.33, 44.29) | (28.18, 46.78) | (48.21, 67.2) | (14.17, 29.98) |  |
| GPT-3.5-Turbo | (3.83, 9.28) | (0.55, 7.0) | (4.11, 15.0) | (4.11, 15.0) |  |
| GPT-4o | (47.35, 58.57) | (35.61, 54.76) | (72.22, 87.49) | (24.56, 42.69) |  |
| Baseline Agent | Gemma-7B | (0.34, 2.9) | (0.0, 3.7) | (0.18, 5.45) | (0.55, 7.0) |
| Llama3-8B | (2.55, 7.27) | (0.55, 7.0) | (4.81, 16.23) | (0.55, 7.0) |  |
| Llama3-70B | (18.6, 28.09) | (22.78, 40.63) | (12.51, 27.78) | (12.51, 27.78) |  |
| GPT-3.5-Turbo | (3.57, 8.89) | (5.52, 17.44) | (1.03, 8.45) | (1.57, 9.84) |  |
| GPT-4o | (19.21, 28.79) | (18.6, 28.09) | (14.17, 29.98) | (14.17, 29.98) |  |
| IDEA Agent (Ours) | Gemma-7B | (0.06, 1.86) | (0.18, 5.45) | (0.0, 3.7) | (0.0, 3.7) |
| Llama3-8B | (1.36, 5.17) | (0.18, 5.45) | (3.43, 13.75) | (0.0, 3.7) |  |
| Llama3-70B | (25.41, 35.76) | (34.67, 53.77) | (21.89, 39.58) | (10.89, 25.55) |  |
| GPT-3.5-Turbo | (10.53, 18.38) | (22.78, 40.63) | (2.78, 12.48) | (2.15, 11.18) |  |
| GPT-4o | (30.14, 40.9) | (30.94, 49.8) | (32.8, 51.79) | (16.69, 33.23) |  |
|  | Human | (55.38, 70.62) | (52.15, 77.56) | (42.31, 68.84) | (54.19, 79.24) |

For the Oracle-rule agent, GPT-4o exhibits superior deductive capabilities, outperforming other models with an approximate 50% success rate across all three puzzle types. Specifically, in the Escape Room puzzles, agents achieve up to an 80% success rate by simply following the provided rule to count the number of paintings of specific colors and types. However, in the Function Operator and Reactor puzzles, merely knowing the rule is not sufficient for success, applying the rule to solve the puzzle is more challenging. Agents must also devise effective plans to apply these rules to reach a solution. This additional planning requirement results in a comparatively lower success rate in these puzzles.

For the Baseline agent, where they are not provided with the underlying rules and solely rely on historical observations to make interactive actions, GPT-4o continues to lead in performance. Notably, Llama3-70B narrows the performance gap, especially in the Function Operator puzzles. Across models, the success rates drop by about half compared to the Oracle-rule agent. This significant decrease highlights the challenge of rule learning and indicates that current LLM agents struggle to learn rules in unfamiliar environments without explicit guidance.

The IDEA Agent significantly boosts success rates. Our proposed IDEA framework leads to approximate 10% increases in success rates for Llama3-70B, GPT-3.5-Turbo, and GPT-4o compared to the Baseline agent. This improvement demonstrates that incorporating a reasoning loop of abduction, deduction, and induction substantially enhances the LLM rule-learning performance in unfamiliar environments. IDEA enables the LLMs to generate hypotheses, plan actions, and refine their understanding based on new observations, which is crucial for rule learning. However, smaller models like Llama3-8B and Gemma-7B experience a slight decline in performance when applying IDEA. This decline is likely due to the increased complexity and context management demands introduced by the IDEA framework, which pose challenges for small models with limited capacity.

LLM agents still fall far behind humans. In the Escape Room puzzle, where the primary challenge is to discover the rule, the Oracle-rule agents excel because once the rule is identified, applying the rule is simple. However, in other types of puzzles, human participants significantly outperform all LLM agents, including the Oracle-rule agents, even without knowing the rules beforehand. This notable disparity highlights that LLM agents lag substantially behind humans, not only in learning new rules but also in effectively applying known rules to solve problems.

### 4.4 Analysis

Figure: Figure 4: Comparison of the cumulative number of puzzles solved at each interaction step. The IDEA agent significantly increases the solving speed compared to the Baseline agent.
Refer to caption: x4.png

IDEA boosts puzzle-solving speed. Figure 4 illustrates the cumulative number of puzzles solved at each interaction step for the Baseline agent, the IDEA agent, and human participants. The slopes of the lines represent the rate at which puzzles are solved per step. Compared to the Baseline agent, the IDEA agent exhibits a steeper slope, indicating that the integration of abductive, deductive, and inductive reasoning enhances the agent’s efficiency in exploring the environment and learning the underlying rules, especially during the early stages.

When focusing on human participants, we observe that they solve fewer puzzles in the initial steps compared to the LLM agents. However, as interactions continue, the number of puzzles solved by humans increases rapidly, eventually surpassing that of all LLM agents. In contrast, LLM agents solved 88.76% of the puzzles within the first 10 steps. Beyond this point, additional interactions contribute less to their success rate. This pattern suggests that humans have a superior ability to learn continuously from interactive environments, effectively improving their performance over time. If we did not limit the puzzles to 15 steps, we anticipate that the success rate of human participants would be even higher. We will explore this observation further in Section 5.

IDEA reduces repetitive actions. LLM agents frequently repeat previous actions instead of exploring new ones. This behavior is highly inefficient in our controlled puzzle environments, where each interaction yields deterministic results, and repeating the same action generally does not provide new information. An exception is the Escape Room puzzle, where entering the same password could result in different outcomes after the rule changes following three failed attempts.

We calculate the average number of repeated actions performed while solving each puzzle, with detailed statistics presented in Table LABEL:tab:repeat_count in Appendix A.2. We observe that most LLMs commonly repeat actions in the Baseline agent. Although models like GPT-4o and Llama3-70B exhibit this issue to a lesser degree, even these advanced models sometimes repeat actions that fail to provide further insights.
The IDEA agent effectively reduces this tendency by explicitly generating plans during the deduction phase. By outlining a clear plan, the IDEA agent can better assess whether the current observations are sufficient or if further specific evidence is needed to reveal the underlying rule. For example, in the Escape Room puzzle, the IDEA agent avoids unnecessary attempts at entering passwords when the evidence gathered is sufficient to determine the correct code (see Figure 11 in Appendix A.4).

IDEA relies on the reasoning ability of underlying LLMs.
The effectiveness of IDEA depends on the underlying LLMs’ ability to reason from hypotheses and observations. Particularly, if an agent generates a false hypothesis and fails to properly refine it, being guided by this incorrect hypothesis can lead the agent to perform even worse than the baseline. During our experiments, we observed that current LLMs tend to hallucinate, especially in the Escape Room puzzles and more severely in the Reactor puzzles. This results in smaller performance improvements compared to those seen with the Function Operator puzzles. This is likely because such fictitious scenarios are not extensively represented in LLM training data. Moreover, LLMs struggle to recognize letter-level patterns, and their reasoning capabilities still require significant enhancement. Examples of hallucination can be seen in Appendix A.4.4).

## 5 Human evaluation

We employ three computer science graduate students (our co-authors) to evaluate all the hypotheses and plans generated by both the IDEA agent and human participants during the abduction, deduction, and induction stages within a randomly selected 50% subsample of the puzzles.

Abduction stage. During the abduction stage, agents formulate an initial hypothesis based on initial observations. Since some puzzles are simple, agents can potentially guess the ground truth rule at this stage. Figure 5(a) shows that LLMs such as GPT-4o have approximately a 30% success rate in correctly identifying the rule during abduction. Notably, humans exhibit the lowest success rate in this stage. We believe this is because humans often lack the patience to thoroughly review all textual information initially and may overlook certain details. In contrast, LLMs meticulously process every word of the prompt and leverage their comprehensive pre-training data to generate plausible hypotheses. Additionally, humans tend not to formalize hypotheses when operating under uncertainty.

Deduction stage. After establishing an initial hypothesis—or each time the agent refines a hypothesis—the agent derives a plan to either validate this hypothesis or use it to solve the puzzle. As shown in Figure 5(b), humans generally outperform LLMs in creating high-quality plans. However, in the Function Operator puzzles, Llama3-70B and GPT-4o perform better than humans, suggesting that these models may have superior mathematical intuition compared to our human participants, who were undergraduate students from computer science-related majors.

Induction stage. Figure 5(c) illustrates the proportion of inductions where the refined hypothesis improved over the previous one, which we define as the effective induction rate. Humans outperform LLMs by a significant margin in this process, with 40% of their refined hypotheses being better than the previous ones. Surprisingly, GPT-4o performs worse than Llama3-70B and GPT-3.5-Turbo. We found that GPT-4o tends to hallucinate, especially in Escape Room puzzles, which negatively impacts its overall effective induction rate. In contrast, Llama3-70B almost never engages in induction activities within Reactor puzzles, as it fails to recognize when its hypothesized rules contradict its observations (see Appendix A.4.4). Despite these issues, both Llama3-70B and GPT-4o exhibit performance closer to humans during the induction process compared to other models in Function Operator puzzles. This relative success is believed to be due to the extensive training of LLMs with mathematical content, which ensures stable and superior performance in these types of puzzles. In contrast, Escape Room and Reactor puzzles, which are less likely to appear extensively in training datasets, see poorer and more variable performance from the models.

Average interactions needed for effective induction. Figure 5(d) shows that humans require fewer interactions—approximately four on average—to effectively refine their hypotheses, compared to LLMs. While LLMs can process initial information thoroughly and generate plausible hypotheses, they face challenges in refining these hypotheses based on new observations during interaction with the environment (see Figure 7 in Appendix A.1). This limitation suggests that LLMs may struggle to learn from new observations and incorporate feedback to continuously improve their hypotheses and problem-solving strategies. This gap may become more pronounced when agents are faced with larger action spaces and environments governed by more complex rules.

Figure: Figure 5: Human Evaluation Results. Bars represent measured values per model and puzzle type; the absence of a bar indicates zero or unavailable data. Plot (a): Abduction Correct Rate—the frequency of correctly guessing the rule during abduction. Plot (b): Effective Deduction Rate—the rate at which deduction plans effectively validate hypotheses or solve puzzles. Plot (c): Effective Induction Rate—the proportion of inductions where the refined hypothesis improved over the previous one. Plot (d): Average Actions per Effective Induction—the average number of interactive actions needed for an effective induction.
Refer to caption: x5.png

## 6 Conclusion

In this work, we introduce RULEARN, a benchmark designed to evaluate LLM agents’ rule-learning abilities in interactive environments. We propose IDEA, an agent framework that mimics human reasoning through abduction, deduction, and induction. Comprehensive experiments involving five prominent LLMs and human participants reveal that while IDEA significantly improves the rule learning ability of LLM agents, there is still a large gap between LLM and humans particularly in refining hypotheses and adapting strategies. Despite these advancements from the IDEA framework, LLMs still face challenges in generating valid hypotheses and avoiding repetitive actions in complex scenarios. Our findings underscore the need for further development of LLMs that can emulate human cognitive processes more effectively in explorations of novel environments. RULEARN provides a foundational resource for future research aimed at closing these gaps.

## 7 Ethics Statement

Our work aims to benefit the broader research community by introducing RULEARN, a benchmark for evaluating the rule-learning abilities of LLM agents and proposing the IDEA agent framework. All data in RULEARN contains no personal or sensitive information, ensuring respect for privacy and ethical standards. This project is approved by our Institutional Review Board (IRB). Human participants are recruited through emails from our university’s computer science and engineering department. All participants were adults over 18 years old and provided informed consent. The data collected from these participants were de-identified and consented for release for research purposes. Participants were compensated $15 each for one hour of their time. We ensured that all content presented during evaluations was free from offensive or inappropriate material. For human evaluations of all the hypotheses and plans generated by LLM agents and human participants, three computer science graduate students (our co-authors) conducted the evaluation. We are committed to the ethical use of our benchmark and agent framework, and upon acceptance of this paper, we will release our code and data to encourage open collaboration and advancement in the field.

## Appendix A Appendix

### A.1 Figures

Figure: Figure 6: The scaled performance radar plot shows varying performances across different puzzle types. GPT-4o leads, followed by Llama 70B, GPT-3.5, Llama 8B, and Gemma 7B.
Refer to caption: x6.png

Figure: Figure 7: Although agents continuously refine their hypotheses toward the ground truth rule, identifying the exact rule remains a challenging task. According to our evaluation, humans have a 43.3% success rate in finding the ground truth rule, with 13% of these discoveries occurring during the abduction stage and 30% during the induction stage. In contrast, LLM agents exhibit a different pattern. They successfully identify the ground truth rule in approximately 30% of puzzles, with nearly all of these discoveries occurring during the abduction stage and only 3% achieved through interaction and induction. This highlights a significant limitation of current LLM agents, as they lack the ability to learn effectively from interactions. Consequently, the rule-learning patterns of LLM agents differ markedly from those of humans.
Refer to caption: x7.png

### A.2 Tables

**Table 4: Function operator puzzle distribution**
| Puzzle ID | No. of Functions | No. of Unknown Paramters | No. of Elementary Functions |
| --- | --- | --- | --- |
| 1–5 | 1 | 1 | 1 |
| 6–10 | 2 | 2 | 2 |
| 11–15 | 2 | 2 | 3 |
| 16–20 | 3 | 3 | 4 |

**Table 5: Escape room puzzle distribution, the number of paintings ranges from 3 to 13, increasing by one for each subsequent puzzle ID within each group. The first group (1–10) presents all paintings initially, whereas the second group (11–20) requires agent actions to reveal all paintings.**
| Puzzle ID | Number of Paintings | Paintings Visibility |
| --- | --- | --- |
| 1–10 | 3–13 | All paintings visible at start |
| 11–20 | 3–13 | Requires actions to view all paintings |

**Table 6: Reactor puzzle distribution, five unique target strings need to be synthesized (ACB, CCADD, CADEA, ABCDEF, FEADE) are uniformly distributed across each rule category.**
| Puzzle ID | Reactor Rule |
| --- | --- |
| 1–5 | Simple Concatenation |
| 6–10 | Reverse Concatenation |
| 11–15 | Middle Insertion |
| 16–20 | Prefix Replacement |

**Table 7: Average Number of Repeated Actions Per Puzzle: Repeating actions is a common pattern among LLM agents during rule-learning tasks. Even sophisticated models like GPT-4o often exhibit reduced action duplication when exploring environments using the IDEA agent. The implementation of this agent has been shown to mitigate this tendency across all models. However, Gemma-7B frequently generates nonsensical actions that are not recognized as duplicates. Consequently, a duplication rate of 0.02 does not necessarily indicate that Gemma-7B effectively avoids repeating historical actions.**
| Setup | Model | All Puzzles | Function Operator | Escape Room | Reactor Puzzles |
| --- | --- | --- | --- | --- | --- |
| Deduction Only | Gemma-7B | 8.28 | 3.75 | 11.49 | 9.59 |
| Llama-8B | 2.39 | 3.4 | 1.2 | 2.56 |  |
| Llama-70B | 2.38 | 2.91 | 0.44 | 3.8 |  |
| GPT-3.5-Turbo | 6.18 | 8.79 | 5.26 | 4.5 |  |
| GPT-4o | 1.75 | 1.55 | 0.34 | 3.35 |  |
| Baseline | Gemma-7B | 7.27 | 0.61 | 11.73 | 9.46 |
| Llama-8B | 3.05 | 3.38 | 1.99 | 3.77 |  |
| Llama-70B | 2.47 | 1.81 | 0.92 | 4.68 |  |
| GPT-3.5-Turbo | 5.97 | 7.16 | 2.64 | 8.12 |  |
| GPT-4o | 2.03 | 1.5 | 0.11 | 4.48 |  |
| IDEA | Gemma-7B | 2.74 | 6.33 | 1.88 | 0.02 |
| Llama-8B | 1.83 | 2.29 | 1.64 | 1.56 |  |
| Llama-70B | 1.01 | 0.45 | 0.39 | 2.18 |  |
| GPT-3.5-Turbo | 4.49 | 4.14 | 2.4 | 6.93 |  |
| GPT-4o | 1.12 | 0.52 | 0.15 | 2.68 |  |
|  | Human | 0.76 | 0.46 | 1.6 | 0.22 |

### A.3 IDEA agent detail

#### A.3.1 Environment Entities

- $\bullet$
Agent: Represents the entity focused on rule-learning and problem-solving, comprising the following components:
*
Goal $(\mathbf{G})$: The objective of the agent, articulated in natural language.
*
Buffer Memories $(\mathbb{\tilde{M}}:=\{\mathbf{\tilde{m}_{1},\tilde{m}_{2},\dots,\tilde{m}_{n}
}\})$: This temporary storage holds all newly generated information during the agent’s exploration, including actions taken, outcomes of each action, and observations. This is where the most recent activities are initially recorded.
*
Memories $(\mathbb{M}:=\{\mathbf{m_{1},m_{2},\dots,m_{n}}\})$: This permanent memory stores all significant observations and facts from the beginning of the task. When the agent forms new assumptions and plans, the contents of the Buffer Memories are evaluated; non-essential details like are discarded, while important facts and observations are transferred to the permanent Memories. This architecture ensures that each time the agent revises its hypotheses, it can clearly distinguish which observations occurred under the new assumptions and plan.
*
Hypotheses $(\mathbf{H})$: The current hypotheses formulated by the agent to explain all the observations, are expressed in natural language.
*
Plan $(\mathbf{P})$: The current strategy devised by the agent to either test the correctness of the existing hypotheses or to leverage these hypotheses to achieve the goal, also represented in natural language.
*
Action Space $(\mathbb{A})$: A set of potential actions available to the agent, determined by its current hypotheses and plan. The Action Space is dynamic and can change in response to interactions with the environment. For example, after investigating a fridge, the agent gains the additional option to open the fridge and inspect its contents.
- $\bullet$
Objects $(\mathbb{O})$: Represents all interactive entities within the environment that provide the agent with a means to interact with the world. A single object in this set is denoted as $\mathbf{O}$.
*
Description $(\mathbf{D_{o}})$: A concise description of the object, detailing its nature and potential uses, presented in natural language.
*
Predefined interactive actions $(\mathbf{O_{A}}:=\{\mathbf{\tilde{a}_{1},\tilde{a}_{2},\dots,\tilde{a}_{n}}\})$: A set of actions that are predefined for each object. Each action is described in natural language, explaining its purpose. Additionally, each action is associated with a coded function that processes the agent’s inputs and produces an effect, potentially altering the environment based on these inputs.

#### A.3.2 Interactive Functions

- $\bullet$
Perceptual Action:= $\mathbf{\hat{a}}(\mathbf{O})$: An action automatically added to the agent’s action space for all objects within the same scope. Upon perceiving an object, the agent gains the ability to interact more detailedly with it, adding its interactive actions to the $\mathbf{S}$.
- $\bullet$
Interactive Action:= $\mathbf{\tilde{a}(D_{o},G,H,P,I},\mathbb{\tilde{M},M})$: A predefinec action that triggers a pre-coded effect based on the agent’s input $\mathbf{I}$. For example, in using a reactor, the agent decides the materials and their order of addition, and the reactor processes these inputs based on pre-coded rules to synthesize new materials.
- $\bullet$
Abductive Action:= $\mathbf{\bar{a}(G},\mathbb{\tilde{M}})$: An action based on initial observations, allowing the agent to formulate the first hypotheses and generate a new plan.
- $\bullet$
Inductive Action:= $\mathbf{\dot{a}(G,H,P},\mathbb{\tilde{M},M})$: An action based on the current observations, goals, prior hypotheses, and previous plans, allowing the agent to refine hypotheses and generate new plans.
- $\bullet$
Deductive Action := $\mathbf{\ddot{a}(G,H,}\mathbb{A,M,\tilde{M}}\mathbf{)}$: An action based on the current memories, hypothesis, and action space that generates a plan to either validate the current hypothesis or leverage it to solve problems.
- $\bullet$
Action select:=$F_{a}(\mathbf{G,H,P,}\mathbb{\tilde{M},M,A})\rightarrow\mathbf{a}$: A function where the agent selects an action from the action space, considering all gathered information.

With the definitions and entities described above, we can formalize our interactive, rule-learning process. The sequence begins with the agent selecting an action from the available action space. The agent then decides on an input based on the selected action. Once the action is executed, the environment responds by providing feedback to the agent. The outcome of this action results in changes to $\mathbb{\tilde{M},M,S},\mathbf{H,P}$ and $\mathbb{O}$, making the environment dynamic as the exploration process progresses. These changes reflect the agent’s interactions and adaptations to the evolving conditions within the environment.

#### A.3.3 Pseudocode of interactive rule learning procedure

Figure: Algorithm 2 Agent rule-learning procedure

### A.4 Prompt example

#### A.4.1 Function Operator Puzzles

Figure: Figure 8: Prompt of Function Operator Puzzles, Action select and Deduction.
Refer to caption: x8.png

Figure: Figure 9: Prompt of Function Operator Puzzles, Interactive input and Induction.
Refer to caption: x9.png

Figure: Figure 10: An example of Function Operator Puzzles is provided where actions marked in green are interactive actions, while the rest are perceptual actions through which the agent reads and perceives necessary environmental information.
Refer to caption: x10.png

Figure: Figure 11: In the provided example, despite extensive exploration, the agent continues to assign multiple values to the function. In contrast, the IDEA agent hypothesizes that the function is simple, possessing only a single constant parameter. Consequently, this agent efficiently solves the puzzle by assigning just one value to the function.
Refer to caption: x11.png

#### A.4.2 Escape Room Puzzles

Figure: Figure 12: Prompt of Escape Room puzzles, Action select and Abduction.
Refer to caption: x12.png

Figure: Figure 13: Prompt of Escape Room puzzles, Interactive Input and Induction.
Refer to caption: x13.png

Figure: Figure 14: An example of Escape Room is provided where actions marked in green are interactive actions. The remaining actions, which are perceptual, allow the agent to read and gather necessary environmental information. In the given example, although the agent successfully guesses the correct rule behind the observations, it fails to adhere to its plan and assumptions. When inputting the password, the attempts do not align with the planned strategy, and it also makes repeated attempts (repeated 101 twice) that yield no useful results.
Refer to caption: x14.png

#### A.4.3 Reactor Puzzles

Figure: Figure 15: Prompt of Reactor Puzzles, Action select and Abduction
Refer to caption: x15.png

Figure: Figure 16: Prompt of Reactor puzzles, Interactive input and Induction.
Refer to caption: x16.png

Figure: Figure 17: An example of Reactor Puzzles is provided where actions marked in green are interactive actions. The remaining actions are perceptual, allowing the agent to read and gather necessary environmental information. In the given example, the agent eventually realizes the flaws in its initial hypothesis and generates a correct one. However, the agent reaches the step limit before it can implement the solution, failing.
Refer to caption: x17.png

#### A.4.4 Hullicination examples

Figure: Figure 18: Example of GPT-4o agent failing to leverage rule towards the result.
Refer to caption: x18.png

Figure: Figure 19: Example of hallucination by GPT-4o when refining previous hypothesis.
Refer to caption: x19.png

Figure: Figure 20: Example of Llama-3 70B agent failing to detect a contradiction in experimental results.
Refer to caption: x20.png

Figure: Figure 21: Example of hallucination by Llama-3 70B when generating a password.
Refer to caption: x21.png

### A.5 Puzzle Examples

Figure: Figure 22: Function operator puzzle No. 2.
Refer to caption: x22.png

Figure: Figure 23: Function operator puzzle No. 17.
Refer to caption: x23.png

Figure: Figure 24: Escape room puzzle No. 3
Refer to caption: x24.png

Figure: Figure 25: Escape room puzzle No. 13
Refer to caption: x25.png

Figure: Figure 26: Reactor puzzle No. 8 and No. 14
Refer to caption: x26.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - How do you ensure the feedback from the environment is accurate in the feedback stage? How are environment feedback given (i.e., are they static pre-coded rule-based feedback, or are they dynamically provided feedback by another LM)?
- Could you be more specific about how the benchmark is created? How do you verify the quality and diversity of the benchmark?
- Have you tested the statistical significance of the human vs. agent comparison?
- Could you provide more details of how human participants are instructed about the task, and how they're trained at the beginning of the study? It'll be very helpful to provide more details of the exact human experiment setup, by providing concrete details of the procedure of the study.
- Might you have suggestions of how to improve the agentic framework for this task as future directions, based on the insights of this current study?
soundness: 3
strengths: - The RULEARN benchmark is a novel benchmark for agentic rule-learning abilities of LLMs, complementary to previous resources for assessing LLM induction abilities.
- The IDEA framework shows promises in enabling more enhanced rule-learning ability for LLM-based agents.
- The human study provides very interesting insights comparing human and LM rule induction abilities. It's especially interesting to see humans are not very good at formulating initial hypotheses.
summary: This works proposes RULEARN, a benchmark consists of interactive puzzles that are designed to assess the rulelearning abilities of LLM agents in interactive settings. It also proposes IDEA, an agent that dynamically establishing and applying rules that mimic human rule-learning behaviors. It has the following three steps: (1) generates initial hypotheses from limited observations through abduction, (2) devises plans to validate these hypotheses or leverages them to solve problems via deduction, (3) refines previous hypotheses using patterns identified from new observations through induction. With the novel testbed and the rule-learning framework, this paper also compares LLM agent rule-learning abilities with that of humans. Such study shows several interesting insights: (1) LLM agents continue to struggle with efficiently exploring unfamiliar environments. (2) LLMs fall short in deducting valid plans to verify current hypotheses
and guide future exploration. (3) LLMs fall short to correct previous hypotheses when they contradict new observations and are less capable of refining a hypothesis to make it more robust.
weaknesses: - There lacks details for how the puzzles in the benchmark are curated. Also, there's not discussion around the quality and diversity of the benchmark.
- The size of the benchmark is relatively small compared to usual benchmarks.
- The paper lacks important details of the human study, e.g., how are human participants are instructed about the task, or how are participants enforced to follow the abduction-deduction-induction framework. These details are critical for accessing the rigor of the human studies.
- Minor typo "evaluat" at L117
- Missing critical citation that thoroughly tested iterative rule refinement ability of LMs: https://arxiv.org/pdf/2310.08559

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: What is the difference between the Abduction stage and Induction stage? It seems like these two stage are both about generate / refine a hypothesis based on observations.
soundness: 2
strengths: 1. The authors construct an interactive environment containing three types of task, each has 20 puzzles.
2. The authors propose a three-stage framework to improve LLMs' rule-learning ability.
3. The authors conduct experiments on five popular LLMs and compare the results with human's abilities.
summary: This paper constructs a benchmark, named RULEARN, to evaluate the rule-learning abilities of LLM agents in interactive settings and proposes a reasoning framework, named IDEA, which integrates the process of induction, deduction, and abduction. The authors evaluate five representative LLMs.
RULEARN contains three types of tasks: The Function Operator, The Escape Room, and The Reactor.
These tasks are all about reasoning some hidden rules.
IDEA first generate a hypothesis and then generates a plan based on the hypothesis, interacts with the environment to valid the hypothesis and then refine the initial hypothesis based on observations.
weaknesses: 1. The small number of questions in the benchmark may limit its applicability.
2. Improving the output of large language models (LLMs) based on the results of interactions with observations and the environment has become a common approach, as seen in methods like ReACT (https://arxiv.org/pdf/2210.03629) and Reflexion (https://arxiv.org/pdf/2303.11366). The main idea of the IDEA framework is quite similar to these works.
3. The experiment lacks some baseline comparisons, such as using search methods or observation-based reflection approaches.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. How is the Baseline agent implemented, how is the IDEA agent different from the baseline agent? It would be good to give a prompt example that highlights the key differences between the proposed IDEA agent and the baseline agent.
2. Why use the Wilson confidence interval instead of simply the success rate as the evaluation metric, since it's less intuitive to understand the performance of different methods?
soundness: 2
strengths: - The paper proposed a benchmark to assess LLM agents' rule-learning ability in an interactive environment, which is different from existing reasoning benchmarks where there are no interactive information-seeking actions required.
- The paper proposed an LLM agent, IDEA, which achieves higher performance than the baseline reasoning method in the proposed benchmark.
- The presentation of the paper is clear and easy to follow.
summary: The paper proposed an interactive reasoning benchmark RULEARN, which focuses on evaluating LLM agents' rule-learning and planning abilities in an interactive environment. The benchmark contains three types of puzzles "function operator", where the agent needs to determine the parameters and element functions of a set of functions by trying different input values and observe output values; "escape room", where the agent needs to gather information in a textual environment and guess a code; "reactor" where the agent needs to figure out what a string operator do and come up with a plan to reach a specific string using the operator.

The paper also proposed a rule-learning agent, IDEA, which can perform abduction, induction, and deduction iteratively to explore an interactive environment, learn rules, and achieve goals.
weaknesses: While the idea of evaluating the reasoning ability of LLM agents in an interactive environment where information-seeking is necessary is interesting, there are several key drawbacks of the paper:
1. The size of the proposed benchmark is very small, with only twenty puzzles per problem. This will make the evaluation very noisy and also eliminate the possibility of any training on these tasks, which significantly limits the application of the benchmark.
2. The proposed IDEA agent needs prompt tuning for each specific problem, which limits its generalizability. Considering the proposed benchmark has limited test cases, the reviewer is worried that the good performance of the IDEA agent is a result of prompt tuning overfitting to the test problems.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - I’m curious about the connection between this paper and active learning. The interactive action defined in IDEA is similar to adding new examples. Some studies related to this line of work, e.g. [2], are not discussed in the paper. It would be helpful to discuss these connections and differences.

[2] Wasu Top Piriyakulkij, Cassidy Langenfeld, Tuan Anh Le, Kevin Ellis. Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoning.
soundness: 3
strengths: - The paper provides a benchmark to evaluate various reasoning capabilities of LLMs in interactive settings. The proposed tasks, although mainly focused on puzzles, are interesting and serve the evaluation purpose. The results provide insights into LLMs' rule-learning capabilities in real-world scenarios.
- The experiments are comprehensive, covering a wide range of puzzle sets and five LLMs. Human experiments are also helpful. 
- The paper includes various quantitative and qualitative analyses of the experimental results. Comparisons between human and model performance provide useful insights.
- The paper is clearly written and easy to follow.
summary: This paper proposes a benchmark (RULEARN) and a framework (IDEA) to evaluate the rule-learning capabilities of LLM agents in interactive settings. The proposed framework, IDEA, integrates induction, deduction, and abduction, where the LLM agent forms initial hypotheses, devises plans to validate these hypotheses, and refines previous hypotheses. The IDEA agent is evaluated on RULEARN across three domains: the Function Operator, the Escape Room, and the Reactor. Experiments show that IDEA agents significantly outperform baselines but still lag behind human participants.
weaknesses: - The naming of the Oracle-rule agent feels somewhat misleading to me. IMO, it is not really an oracle, as the LLM is not provided with the ground truth rule. Instead, the agent is simply solving an easier task with additional information about the rule. Therefore, this might not serve as a fair or useful upper bound, as the task is actually different. Based on L311-312, “Even if the agent could successfully learn the correct rule, applying the learned rule to solve the puzzle is non-trivial,” I assume the authors meant that the “meta-rule” is provided, but the agent still needs to determine the exact instantiation. I think this should be clarified further to avoid confusion.
- It would be interesting if the authors could include a breakdown of performance based on puzzle difficulty for different models. Difficulty could be measured using the statistics defined in Appendix A2, e.g., for the Function Operator, it could be the number of functions, etc. This could help readers understand how the performance of different models changes when varying difficulty levels.
- There are many studies evaluating LMs’ performance on text-based games, e.g. [1] and many follow-up works. In most of these settings, the LM agent needs to determine the goal and solve the game. What’s the difference between this paper and that line of work? Is the key difference that the rule is more fine-grained, or that the IDEA agent is explicitly evaluated on induction, abduction, and deduction? It would be beneficial if the authors could further discuss and clarify this.

[1] Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, Adam Trischler. TextWorld: A Learning Environment for Text-based Games.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: It would be helpful if the capabilities the benchmark is testing can be clearly stated, e.g. see the priors stated in the ARC challenge https://arcprize.org/arc. More insights on the human problem solving strategies used by the human participants would be useful since they appear not to do well in abduction but are more successful at the end.
soundness: 3
strengths: The benchmark introduced in this paper is reasonably timely, pushing towards open language description of problems that require interactive problem solving. The proposed framework of abduction, followed by induction and deduction is reasonable and showed improved performance over the baseline. Some interesting observations are made in the human studies, e.g. that humans do not do well initially and appear not to do much abduction. This can lead to interesting followup work. Overall, I think that this is useful work that is competently done and can help advance the area.
summary: The paper introduces a benchmark for assessing the rule learning abilities of LLMs in an interactive environment. The paper also proposes a reasoning framework that does abduction, induction, and deduction in order to do well in such interactive environments. The framework outperforms the baseline, although has not reached human performance in the tasks.
weaknesses: Overall, the benchmark is okay for exploring interactively solving problems described in natural language. But it is not highly compelling. It is not as targetted towards human priors like the ARC challenge which targets objectness, goal-directness, number and counting, and basic geometry and topology. It is also not clearly useful like the Behaviour-1K benchmark. The size of the benchmark is also not large, so it is useful for exploring the problem with more benchmarks required for verifying the capabilities tested.