Title: Active Reasoning in an Open-World Environment
Venue: NeurIPS 2023
Accepted: yes
Keywords: Visual Reasoning

Abductive Reasoning

Active Reasoning
Abstract: Recent advances in vision-language learning have achieved notable success on *complete-information* question-answering datasets through the integration of extensive world knowledge. Yet, most models operate *passively*, responding to questions based on pre-stored knowledge. In stark contrast, humans possess the ability to *actively* explore, accumulate, and reason using both newfound and existing information to tackle *incomplete-information* questions. In response to this gap, we introduce **Conan**, an interactive open-world environment devised for the assessment of *active reasoning*. **Conan** facilitates active exploration and promotes multi-round abductive inference, reminiscent of rich, open-world settings like Minecraft. Diverging from previous works that lean primarily on single-round deduction via instruction following, **Conan** compels agents to actively interact with their surroundings, amalgamating new evidence with prior knowledge to elucidate events from incomplete observations. Our analysis on \bench underscores the shortcomings of contemporary state-of-the-art models in active exploration and understanding complex scenarios. Additionally, we explore *Abduction from Deduction*, where agents harness Bayesian rules to recast the challenge of abduction as a deductive process. Through **Conan**, we aim to galvanize advancements in active reasoning and set the stage for the next generation of artificial intelligence agents adept at dynamically engaging in environments.

Active Reasoning in an Open-World Environment

Abstract

Recent advances in vision-language learning have achieved notable success on complete-information question-answering datasets through the integration of extensive world knowledge. Yet, most models operate passively , responding to questions based on pre-stored knowledge. In stark contrast, humans possess the ability to actively explore, accumulate, and reason using both newfound and existing information to tackle incomplete-information questions. In response to this gap, we introduce 🔍 Conan , an interactive open-world environment devised for the assessment of active reasoning . 🔍 Conan facilitates active exploration and promotes multi-round abductive inference, reminiscent of rich, open-world settings like Minecraft. Diverging from previous works that lean primarily on single-round deduction via instruction following, 🔍 Conan compels agents to actively interact with their surroundings, amalgamating new evidence with prior knowledge to elucidate events from incomplete observations. Our analysis on 🔍 Conan underscores the shortcomings of contemporary state-of-the-art models in active exploration and understanding complex scenarios. Additionally, we explore Abduction from Deduction , where agents harness Bayesian rules to recast the challenge of abduction as a deductive process. Through 🔍 Conan , we aim to galvanize advancements in active reasoning and set the stage for the next generation of AI agents adept at dynamically engaging in environments.

1 Introduction

Active interaction with the environment is fundamental to human understanding of the world around us. Both neural and behavioral studies indicate that through active engagement with their surroundings, humans garner critical insights and foster a profound understanding of complex phenomena (Goodale and Milner, 1992; Rizzolatti et al., 1997; Rieber, 1996). When confronted with partial or ambiguous data, our innate response is to seek supplementary evidence, hypothesize, and put forth possible explanations, sometimes even reevaluating initial assumptions (Yuan et al., 2022). This iterative process persists until a satisfactory resolution emerges.

The process of formulating theories based on observations and prior knowledge is classically termed as abductive reasoning or simply, abduction (Peirce, 1965; Douven, 2021). A topic of enduring

Survival-driven:

Get

Goal-driven:

Actions

Move_left

Move_right

Cut_tree

Mine_coal

Cook_beef

Get_lava

…

Planner

Parser

Navigate

Collect

Make

●

Goal

What was the criminal's goal?

To get the diamond.

●

Intent

Why did the criminal cut

these trees?

To make a wood pickaxe.

●

Survival

What happened between

the criminal and the zombie?

The criminal fought and

killed the zombie despite

being injured.

Key Frames

Explorer

Observer

…

Question Answering

(a)

(b)

(c)

Traces

detective

vandal

interest among psychologists, abduction is perceived as a cornerstone of human cognitive processes. Historical and contemporary studies have delved into its cognitive mechanisms (Josephson and Josephson, 1996; Thagard, 1988; Peirce, 1965), practical applications (Hobbs et al., 1993; Shank, 1998), and ties to scientific thinking and decision-making (Hanson, 1965; Gigerenzer and Gaissmaier, 2011; Zhang et al., 2021a). With growing momentum in the machine learning sphere, recent years have witnessed the advent of dedicated benchmarks and models accentuating abductive reasoning (Bhagavatula et al., 2019; Kayser et al., 2021; Hessel et al., 2022; Liang et al., 2022).

However, the bulk of prior work in this domain relies heavily on a single-round, passive questionanswering paradigm that offers complete information. This setup often sees an agent simply responding to queries, leveraging vast pre-trained knowledge, as evidenced by the latest strides in language and vision-language learning. Recent progress in the field has notably already improved performance in such complete-information information question-answering. Contrarily, humans demonstrate a far more nuanced approach when navigating abductive scenarios with incomplete data (Edmonds et al., 2018). We actively engage, explore, gather, and reason, drawing from both new information and prior knowledge. Our iterative approach allows for continuous refinement based on newly acquired evidence (Oaksford and Chater, 1994; Bramley et al., 2017; Edmonds et al., 2019, 2020).

To capture the dynamic and exploratory essence of abductive reasoning-termed herein as active reasoning -we present 🔍 Conan , a new open-world environment tailored for abductive reasoning. Standing head and shoulders above traditional single-round passive reasoning benchmarks, 🔍 Conan boasts an open-world arena, urging agents to actively probe surroundings and engage in multi-round abductive inferences, all while leveraging in-situ collected evidence alongside pre-existing knowledge.

At its core, 🔍 Conan is conceived as a detective game, transmuted into a question-answering challenge. Here, the detective is tasked with a query and an 'incident scene' riddled with traces left by a vandal . Given the initial paucity of conclusive information, the detective must embark on an in-depth exploration of the scene. As the inquiry progresses, the detective has the opportunity to actively scout its environment, continually reshaping and honing its hypotheses, especially when new revelations potentially contradict the prior hypothesis. Furthermore, we meticulously craft questions within 🔍 Conan to span various levels of abstraction, from localized intentions (Intent) to overarching objectives (Goal) and survival states (Survival).

To probe the proficiency of active reasoning, we evaluate state-of-the-art Reinforcement Learning (RL) and multimodal question-answering models on 🔍 Conan . Our observations highlight an intriguing dichotomy: while these cutting-edge models exhibit prowess in addressing low-level, short-term tasks, they struggle with multi-round environmental interactions and high-level abductive reasoning.

A plausible root of this challenge could be the absence of structurally represented knowledge. Predicated predominantly on associative training, these agents are versed in correlating traces with responses without genuinely internalizing holistic world models. In sharp contrast, humans seamlessly navigate abductive reasoning by forecasting potential trajectories leading to a perceived outcome. This intricate dance gradually transmutes from abductive to deductive reasoning, where humans harness their innate understanding of causality to deduce and mirror observed patterns. In our pursuit to mirror this quintessential human trait, we integrate Abduction from Deduction (AfD) into 🔍 Conan via a Bayesian approach. Experimental results underscore the efficacy of AfD, indicating a substantial avenue for bolstering agent adeptness in 🔍 Conan .

To sum up, our work makes the following three contributions:

We usher in the novel domain of active reasoning , underscoring the indispensable roles of active exploration and iterative inference in abductive reasoning. This paradigm shift transforms traditional single-round passive question-answering paradigms into a more immersive format, compelling agents to actively engage with the environment to procure pivotal evidence.

We introduce 🔍 Conan , a new environment tailored to evaluate the abductive reasoning ability of current machine learning models within dynamic settings. 🔍 Conan surpasses its predecessors that hinge on step-by-step deductive reasoning, revealing the limitations of present-day models.

We formulate a new learning method for abduction, AfD, grounded in Bayesian principles. This framework elegantly reformulates abduction into deduction, proving instrumental in navigating the complex active reasoning challenges posed by 🔍 Conan .

2 Related Work

Machine Abductive Reasoning Abductive reasoning, foundational to human cognition, is crucial for scientific exploration, decision-making, and problem-solving (Peirce, 1965; Magnani, 2011). In the Artificial Intelligence (AI) landscape, there is a rich history of efforts to equip machines with this ability, where they use prior knowledge and sparse observations to hypothesize amidst uncertainty (Josephson and Josephson, 1996; Xu et al., 2023). Key developments span logic-based abduction (Kakas et al., 1992; Poole, 1993) and hybrid neural-symbolic methods (Rocktäschel and Riedel, 2017; Zhang et al., 2021b; Li et al., 2022, 2023). With computational progress, Large Language Models (LLMs) have effectively addressed several challenges through text generation, exhibiting outstanding performance (Brown et al., 2020; OpenAI, 2023; Thoppilan et al., 2022). Modern research usually frames abductive reasoning within natural language understanding (Bhagavatula et al., 2019) or multimodal vision-language integration (Hessel et al., 2022; Liang et al., 2022). However, there is still a notable gap: many benchmarks lean heavily on deduction, sidelining abduction's interactive essence. Our work addresses this gap, emphasizing the core of active reasoning in abductive contexts.

Embodied Question Answering Embodied question answering enhances traditional Visual Question Answering (VQA) by placing agents in interactive environments (Johnson et al., 2017; Das et al., 2018; Gordon et al., 2018; Yu et al., 2019). In 🔍 Conan , agents actively explore to gather data, preparing them to solve abductive questions based on partial information. Unlike standard embodied question-answering frameworks (Das et al., 2018; Gordon et al., 2018; Yu et al., 2019), where questions become simple instructions for agents, 🔍 Conan introduces complexity: (i) its questions, rooted in high-level intent and goals, resist simple decomposition into a series of actions; (ii) agents in 🔍 Conan act as detectives, constantly hypothesizing from observations and prior knowledge, and iterating their strategies in light of new data. For a comprehensive comparison of 🔍 Conan with other benchmarks, see Tab. 1.

3 The 🔍 Conan Environment

🔍 Conan is crafted as an interactive question-answering environment aimed at evaluating a machine's active abductive reasoning capacity, as depicted in Fig. 1. Building on the foundation of the Crafter (Hafner, 2021), 🔍 Conan evolves into a detective game featuring two agents: the vandal and the detective . The gameplay kickstarts with the vandal undertaking a randomly designated task, leaving behind traces for the detective to unravel. Subsequently, given these traces, pertinent queries are generated. Finally, the detective is spawned in the environment, tasked with navigating these traces and actively probing the environment, all to derive answers through abductive reasoning.

3.1 Basic Components

Playground Originating from the Crafter playground, 🔍 Conan operates within a 64 ˆ 64 grid matrix. Agents navigate this space with a localized 9 ˆ 9 grid field of view centered on their current position. Once the detective is created in the environment, all traces left behind by the vandal persist, serving as clues for the detective to unravel. While pivotal studies (Johnson et al., 2016; Fan et al., 2022; Cai et al., 2023; Wang et al., 2023) address perception in 3D Minecraft settings using foundational models, our emphasis is on honing active abductive reasoning. To this end, we transition from a 3D visual perception to a 2D plane, ensuring a harmonious blend of reduced visual complexity and retaining rich interactivity (Xie et al., 2021).

Items and Actions 🔍 Conan offers an extensive assortment of interactive items: food, materials, mobs, and tools, each tied to specific actions, as illustrated in Fig. 2. It furnishes 26 unique actions to foster agent-environment en-

Collect

Make

Defeat

Survive

發

铁人

gagement. Certain actions leave traces, and together, the items and their mechanics provide a rich set of affordances for agents in the playground. This knowledge about item operations and traces aids the detective in comprehending the incident scene. Advancing from its predecessor, the original Crafter, 🔍 Conan now boasts over 30 achievements, a significant rise of over 50%. It features 32 distinct traces covering all agent actions such as crafting, collecting, defeating, eating, drinking, and incurring injuries. This enhancement enables the design of 60 varied abductive reasoning tasks within the scene. For an in-depth overview of the playground, refer to Appx. A.

Vandal Each 🔍 Conan map starts with the initialization of a vandal . This agent is driven by two primary aims: executing a specific task and preserving its existence within the environment. It is noteworthy that external threats might terminate the vandal prematurely. Traces left in the aftermath of the vandal 's activities form the question foundation for the detective , with every trace potentially birthing several questions. For a detailed overview, see Sec. 3.2. We model the vandal as optimal: when given a random task and the full map, it strategically delineates a sequence of subgoals based on the task dependency graph, all while ensuring its survival. In scenarios with multiple viable paths to an objective, uniform sampling comes into play. This sampling, supported by a probabilistic parser, presents varied strategies for task completion. Hence, the detective must delve deeper to distinguish the actual sequence of events from possible decoys. The execution of the vandal 's individual actions, as per the planned subgoal sequence, is steered by a collection of pre-established policies.

Detective After generating questions from a given trace, a detective is spawned to answer them. Traces left by the vandal span multiple steps and are only partially observable within the detective 's 9 ˆ 9 grid field of view. This requires the detective to actively interact with the environment and gather evidence to answer the questions.

Though both detective and vandal share the same action space, the detective boasts a unique capability. It not only navigates and interacts like the vandal , but can also generate its own traces during its investigation. These overlaid traces from the detective enhance the environment's depth and complexity. This setup pushes the agent to actively derive conclusions from its dynamic interactions. Importantly, the detective is invulnerable; its focus lies squarely on problem-solving, eliminating concerns about survival or evasion. This design emphasizes active exploration and reasoning, ensuring 🔍 Conan 's primary goal remains addressing complex reasoning tasks and answering visual scenerelated questions.

3.2 Questions and Choices

🔍 Conan is designed to assess the abductive reasoning capability of machine models through a diverse set of questions varying in difficulty and abstraction. These questions fall into three primary categories: Intent (local intent), Goal (global goal), and Survival (agent's survival status change). We approach evaluation as a multi-choice question-answering task. Each question offers four choices, with only one being correct. Questions and choices derive from predefined templates, as showcased in Tab. 2. For a more detailed explanation, see Appx. B.1.

Intent questions target the vandal 's immediate objectives or intentions during its task. To decipher these traces, agents must deduce the vandal 's underlying intent or subgoals. Solving these questions necessitates a learning model's comprehension of the local context.

Goal questions probe the vandal 's overarching objectives, extending beyond immediate intents. They necessitate grasping the wider context of a task or action sequence. Such questions query the vandal 's ultimate aims, demanding a learning model to reason within the broader context of the traces.

Survival questions address the wider investigative scope, posing added challenges to the detective . Centered on the vandal 's survival status changes during tasks ( e.g ., collecting food for sustenance), they lead to deviations from the optimal action plan. While not tied to a task's primary objective, these questions require a deeper grasp of the present context, often necessitating reasoning around potential scenarios or alternate results.

Compared with the prevalent VQA setup, wherein questions are based on factual information that is readily obtainable from the input, 🔍 Conan questions cannot be deciphered given only the initial information, necessitating further exploration in the scene. Unlike standard embodied question answering, 🔍 Conan questions cannot be directly parsed as modular primitives; they demand abductive reasoning, drawing from both new observation and former knowledge to hypothesize, validate, and revise. For benchmarking purposes, 🔍 Conan produced a corpus comprising 100,000 questions. These were derived from 10,000 unique scenes, generated via the Crafter's scene generator, with each scene stemming from a task executed by a vandal . This resulted in an average generation of 10 questions per scene.

Image

Key-frame

Extractor

Encoder

Vision-Language Model

Text Encoder

Text Encoder

Question: What's the

criminal's intention?

Exploration Track

Reward

Observation

Incident Scene

Choice A

Key Traces

...

Explorer

Conan

Action

!

Linear

Linear

Linear

Linear

Choice B:

To get the diamond.

Explorer

4 The Detective Pipeline

🔍 Conan casts the abductive reasoning challenge as a detective game, necessitating a detective to efficiently explore and gather information from the environment to deduce plausible explanations ( i.e ., answers) for the given question. This process involves taking into account the temporal dependencies and incompleteness of the traces. To tackle these challenges encountered in 🔍 Conan , we devise a detective pipeline, as depicted in Fig. 3.

Building on previous work that utilizes hierarchical models for task decomposition (Gordon et al., 2018; Das et al., 2018; Wijmans et al., 2019), our pipeline is structured into two primary phases: an exploration phase for trace collection, followed by an abductive reasoning phase. Initially, interaction with the playground is carried out to collect relevant visual information, which is subsequently leveraged in the reasoning phase to infer answers to the posed questions.

Computationally, our pipeline first employs RL agents as explorers (see Sec. 4.1) that learn an exploration policy based on the traces and the question, thereby rendering it goal-oriented. Next, given the question, we recruit vision-language models (see Sec. 4.3) to predict the answer based on the observation. A key-frame extractor (see Sec. 4.2) is inserted into the two phases to selectively identify relevant frames for abduction. The individual components undergo separate training procedures.

4.1 Explorer for Trace Gathering

The primary responsibility of an explorer is to efficiently collect information pertinent to the provided question. Initially, masks are employed to encode questions by highlighting relevant grids. Subsequently, the explorer takes in both the observation and the target question as input and outputs the action probability.

We use a reward function that incentivizes the agent to scout for clues and traces relevant to the given question. Additionally, a penalty term is incorporated to discourage unnecessary actions and inefficient searching, thereby promoting a more targeted exploration strategy.

Specifically, the agent is rewarded with ` 1 when a trace first appears within its local view, or ` 2 when the trace bears a close association with the question. A substantial reward of ` 100 is conferred upon the agent if it successfully uncovers all traces left by the vandal . Concurrently, the agent incurs a penalty of ´ 0 . 1 for every timestep elapsed, with an additional penalty of ´ 1 imposed for executing operating actions.

We evaluate multiple well-regarded RL frameworks as our explorer, including Deep Q-Network (DQN) (Mnih et al., 2015), Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), and Recurrent Proximal Policy Optimization (RecurrentPPO) (Schulman et al., 2017). The StableBaselines3 library (Raffin et al., 2021) is employed for all implementations.

4.2 Key-Frame Extractor

Given that the frames gathered by the explorer tend to be excessively lengthy and redundant, a keyframe extractor is utilized to sift through and select informative frames containing crucial evidence for the detective . We adopt a prevalent selection strategy employed in video understanding (Arnab et al., 2021). Specifically, frames within the temporal bounds determined by the detection of the first and last traces are retained, from which k frames are uniformly sampled. This design is intended to tailor the input with the constrained context window size to downstream vision-language models.

4.3 Vision-Language Models for Abductive Reasoning

We employ a multi-choice question-answering paradigm akin to the one used in Ding et al. (2021). Specifically, the model is presented with a question, its corresponding exploration frame sequence, and each potential answer choice, subsequently generating a score for each choice. The model is trained with a categorical cross-entropy loss. During inference, the choice with the highest score is considered the answer. We evaluate several well-established multimodal models; these models are known for their efficacy in processing both visual and textual data. Additional details on model implementation can be found in Appx. D.1.

Vanilla-Trans The first baseline method leverages a vanilla transformer encoder to fuse observation and textual inputs. Specifically,the raw symbolic map from 🔍 Conan serves as the visual feature, while CLIP's text encoder (Radford et al., 2021) is employed to encode the textual input.

FrozenBiLM FrozenBiLM (Yang et al., 2022), a state-of-the-art model for video question answering, combines visual input with frozen bidirectional language models, trained on web-scraped multimodal data. The approach integrates a frozen language model and a frozen vision encoder with light trainable visual projection modules. FrozenBiLM is tested with BERT-Large (Kenton and Toutanova, 2019) and DeBERTa-v3 (He et al., 2022) as the language model within our questionanswering system, utilizing the symbolic map from 🔍 Conan for visual input.

Flamingo-Mini Flamingo (Alayrac et al., 2022) is a family of vision-language models adept at rapid adaptation to novel tasks with minimal annotated examples. These models can handle sequences of visual and textual data, seamlessly accommodating interleaved images or videos as input. We finetune an open-sourced Flamingo-Mini model with frozen OPT-125M (Zhang et al., 2022), utilizing the symbolic map from 🔍 Conan for visual input.

4.4 Abduction from Deduction (AfD)

The adage ' Set a thief to catch a thief ' suggests the use of someone with a similar background or expertise to apprehend a wrongdoer: the best vandal catchers are vandals. This notion resonates with the core principle of Abduction from Deduction (AfD): for a skillful detective to abduce what a vandal does, it needs an in-depth grasp of vandals' modus operandi , motivations, and decisionmaking process. Translating the implication to a mathematical language, we articulate the problem of abductive reasoning based on evidence and knowledge from known deductive transitions. It can also be seen as an extension of inverse planning (Baker et al., 2007, 2009; Baker and Tenenbaum, 2014). Formally, let g denote the goal of the vandal , O the detective 's observation, and S the playground states post the vandal 's actions. We then have:

where we assume the independence of g w.r.t. O given S , as the goal ought to be clear given the states. Leveraging Bayesian rules, we further observe that

assuming a uniform prior over g and known deterministic environment transitions. Eq. (2) asserts that P p g | S q is proportional to a goal-conditioned forward action policy, where s i , a i Ñ s i ` 1 .

Intuitively, Eqs. (1) and (2) can be understood as follows: to abduce the vandal 's goal from observation, it is imperative to first reconstruct the actual states traversed by the vandal and subsequently ascertain the most plausible goal that, if pursued forward, would result in those states; see Eq. (1). Eq. (2) can be interpreted as a form of deduction , being contingent on transition knowledge derived from a forward action policy. Hence the name Abduction from Deduction (AfD).

In practice, two approaches emerge for implementing P p g | S q based on Eq. (2). The first entails iterating over all g and utilizing a learned or predefined π p¨q to score a lengthy sequence of states. Conversely, the second approach embraces a data-driven strategy, wherein one arbitrarily selects g , samples S from π p¨q , and learns a model of P p g | S q using the p g, S q pairs. The former approach proves time-intensive during inference due to the combinatorial temporal space and expansive goal space, thereby compelling us towards the latter approach. For implementation, we train P p S | O q independently as a Dirac delta function of δ p f p O qq and P p g | S q from sampled pairs from π p¨q employed in task execution in the vandal . The derived goal features, along with the question, are fed into the model for answer prediction. Please refer to Appx. F for additional details.

5 Experiments

5.1 Experimental Setup

Exploration The explorer is trained using DQN, TRPO, and RecurrentPPO for 10 8 steps, with a buffer size of 10 7 and a batch size of 512 . In the case of DQN, training is conducted with ϵ ' 0 . 96 . Each episode is capped at a maximum of 500 steps for the explorer. A curriculum is employed to encourage long-term exploration whilst maintaining a balance with local search: initial training is carried out with traces from long-horizon tasks like 'get the diamond,' compelling the agent to venture farther from its starting point. Subsequently, the agent undergoes further finetuning across the entire dataset. Such a curriculum design prevents a sole focus on local discovery. For downstream reasoning models, k ' 30 keyframes are extracted by the key-frame extractor.

Abductive Inference Our reasoning models are tested under three different settings: Standard, Ideal Explorer, and AfD. In the Standard setting, models undergo training and testing based on the explorer's exploration. The Ideal Explorer setting sees models leveraging on an optimal exploration policy-visible to the ground-truth vandal 's trajectory, albeit imperfect, it facilitates the agent in gathering sufficient evidence for reasoning. This scenario can be conceived as a measure of the reasoning model's aptitude for passive reasoning given complete information. Under the AfD setting, models are trained and used as delineated in Sec. 4.4. All models are trained utilizing 8 NVIDIA GeForce RTX 3090 GPUs. For further training specifics, please refer to Appx. D.2.

5.2 Results and Analysis

Fig. 4 shows the learning curves of various RL agents during exploration. TRPO and RecurrentPPO manifest similar performance in terms of rewards following a substantial number of steps, markedly surpassing the DQN explorer. Additionally, we probe the impact of augmenting the maximum number of exploration steps to 5 , 000 on performance. The data suggests a marginal performance uplift. Nonetheless, we acknowledge that such a performance increment is at the expense of substantially longer exploration time and a notable surge in the accrual of unrelated information. Consequently, we select TRPOwith a maximum of 500 steps per episode as our standard RL explorer.

Quantitative results on 🔍 Conan are depicted in Tab. 3; both the standard and AfD results reported employ TRPO as the explorer. In the

0

2e7

4e7

6e7

8e7

1e8

Step

-150

-100

-50

0

50

100

Reward

TRPO-500

RecurrentPPO-500

DQN-500

RecurrentPPO-5000

DQN-5000

standard setting, we discern that while models exhibit some aptitude in tackling low-level Intent questions, they struggle with higher-level questions pertaining to Goal and Survival. Among the models, Flamingo-Mini ascends to the pinnacle with an accuracy of 66 . 3% . FrozenBiLM models also perform relatively well. Notably, the DeBERTa variant slightly outperforms BERT, insinuating that a robust language backbone can improve general comprehension. Contrarily, the Vanilla-Trans model languishes across all tasks, achieving merely random-level performance.

With the Ideal Explorer, we notice a clear performance boost across all tasks, particularly in the Goal and Survival questions. These results allude to the potential bottlenecking of models' abductive reasoning capability due to the insufficient information collected, underscoring the significance of effective exploration. An adept explorer can significantly aid in the accrual of useful information, informatively pursuing a hypothesis to scrutinize evidence, swiftly self-correcting upon encountering conflicting evidence, and reasonably re-planning. The findings also hint sufficient room for the RL explorer to improve. Remarkably, the Vanilla-Trans exhibits the greatest increase, insinuating that, in comparison to other baseline models, it is markedly vulnerable to insufficient evidence.

For AfD results, nearly all multimodal models exhibit performance on par with end-to-end supervisedly trained models. Remarkably, FrozenBiLM models even surpass the performance observed in standard settings. The persisting failure of Vanilla-Trans can be ascribed to its weakness in reasoning amidst incomplete observations due to the significant disparity between the familiar complete state S and incomplete observation O . Examining task-specific results, a notable performance uplift in the Survival task models is discernible for almost all models relative to the standard setting, albeit sharing the same observation. These results intimate that the inclusion of deductive information sensitizes the detective to vandal 's concerns during task execution. Nevertheless, the exhibited performance in long-term planning remains weak, reinforcing the pressing need for a better exploration policy. Critically, these models continue to find short-term intent questions to be most easily answered.

5.3 Further Discussion

Additional Experiments We further experiment in the absence of visual inputs, serving as a negative control baseline, resulting in random performance across all settings; see Appx. E. This random-level performance underscores the severe constraints imposed on the agent without visual information. The TRPO explorer shows a noticeable improvement over the ones without visual inputs, suggesting that even minimal exploration is preferable to none. Nonetheless, the performance remains relatively modest. On the other hand, the Ideal Explorer demonstrates markedly superior performance, attesting to the substantial benefits its capacity to accrue perfect trace evidence renders to the downstream reasoning task. This accentuates the imperative of effective exploration.

Error Analysis We extend an error analysis for the 'goal' split, probing the reasoning model across a spectrum of tasks. Table 4 compares two groups: reasoning based on the Ideal explorer and the TRPO explorer. The findings underscore that proficient exploration, i.e ., the heuristic Ideal explorer who recovers the vandal's trajectory, is sufficient for satisfactory performance. However, to fully harness the potential, a more adept reasoner is requisite, one capable of deciphering the vandal's hidden states from observed traces. For instance, the act of felling trees could signify a need for either wood or food (apples), and discerning the intent solely from traces of felled trees presents a challenge. When it comes to 'trace-relevant' frames or 'keyframes,' the Ideal explorer could ostensibly furnish all trace-relevant frames. However, the concept of keyframes remains nebulous. Within the video understanding domain, a formidable challenge lies in the extraction of 'keyframes.' This is a post-hoc concept that eludes straightforward acquisition upfront. A prevailing approach, aimed at augmenting efficiency (diminishing context length in Transformer), entails truncating it via every k-th frame.

Joint Reasoning The collective enhancement of both exploration and reasoning elements emerges as quintessential, given its mirroring of human-like intelligence. For instance, by providing feedback, the reasoner can steer the explorer towards actions that are potentially more insightful and likely to produce pertinent traces. Nonetheless, practical implementation encounters significant hurdles. Assigning credit to exploratory decisions bearing long-term implications can be intricate, particularly when the outcomes of exploratory actions become evident after a substantial time lapse, thereby muddying the causal relationship between the decisions and their ultimate effect on reasoning and answering questions. This accentuates the mutual reliance between exploration and reasoningadvancement in one facet demands progression in the other, introducing a bilateral dependency that complicates optimization. The reasoning component alone demands hefty training and computational resources, especially when utilizing large language models. The demand for formidable computational power renders the simultaneous optimization of exploration and reasoning exceedingly daunting. Collectively, this approach is also widely adopted (Gordon et al., 2018; Lei et al., 2018; Koˇ cisk` y et al., 2018). Consequently, we navigate along this trajectory, projecting that future endeavors on 🔍 Conan should prioritize reasoning above exploration.

To summarize, the engagement of a proficient explorer substantially enhances abductive reasoning, particularly in higher-level tasks such as goal-oriented and survival-centric inquiries. This underlines the criticality of exploration as a precursor to tackling abductive reasoning tasks in the presence of incomplete information. Furthermore, the achievement of the AfD hint at the potential for models to harness world knowledge, especially transition knowledge pertaining to tasks and traces, to transform abductive reasoning into deductive simulation. We posit that the presented approach resonates more with human-like reasoning, edging us closer to the core of human intelligence.

6 Conclusion

In this paper, we introduce 🔍 Conan , a benchmark tailored to evaluate and assess models' active reasoning ability in addressing incomplete-information questions in an interactive environment. 🔍 Conan sets itself apart from existing abductive reasoning benchmarks by incorporating an openworld playground facilitating active exploration. It differentiates itself from prevailing embodied question-answering benchmarks by introducing the demanding abductive process in question answering, necessitating multi-round abductive inference based on gathered evidence. Moreover, we propose a new learning paradigm, Abduction from Deduction (AfD), that turns the problem of abduction to deduction, exploiting the problem structure through Bayesian principles. Benchmarking the efficacy of contemporary machine learning models on 🔍 Conan , we elucidate the model limitations in interacting with the environment that leads to failure in higher-level, longer-term abductive reasoning.

Limitations and Future Work In general, we notice two significant limitations from the experimental results. For one, the explorer does not supply particularly relevant information for the reasoning model. In the human abductive reasoning process, exploration and reasoning should be closely intertwined, with an agent using the current hypothesis to guide exploration and improve its understanding. However, due to long-range exploration and complex vision-language reasoning, we only applied the conventional visual question-answering method and did not fully integrate these two processes. For another, learning naive question-answer mapping shall be sub-optimal. By leveraging the problem structure, AfD has shown improved performance on a particular set of problems. Nevertheless, the current AfD formulation is still rudimentary. We believe an in-depth understanding of the structure and well-crafted implementation could further boost performance.

Acknowledgement The authors would like to thank Ms. Zhen Chen (BIGAI) for designing the figures, and NVIDIA for their generous support of GPUs and hardware. M.X., G.J., W.L., C.Z., and Y.Z. are supported in part by the National Key R&D Program of China (2022ZD0114900), M.X. and W.L. are supported in part by the NSFC (62172043), and Y.Z. is in part by the Beijing Nova Program.

References

Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y ., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022). Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS) . 7

Arnab, A., Dehghani, M., Heigold, G., Sun, C., Luˇ ci´ c, M., and Schmid, C. (2021). Vivit: A video vision transformer. In International Conference on Computer Vision (ICCV) . 7

Baker, C. L., Saxe, R., and Tenenbaum, J. B. (2009). Action understanding as inverse planning. Cognition , 113(3):329-349. 7

Baker, C. L. and Tenenbaum, J. B. (2014). Modeling human plan recognition using bayesian theory of mind. In Plan, activity, and intent recognition: Theory and practice , volume 7, pages 177-204. Morgan Kaufmann Waltham. 7

Baker, C. L., Tenenbaum, J. B., and Saxe, R. R. (2007). Goal inference as inverse planning. In Annual Meeting of the Cognitive Science Society (CogSci) . 7

Bhagavatula, C., Le Bras, R., Malaviya, C., Sakaguchi, K., Holtzman, A., Rashkin, H., Downey, D., Yih, W.-t., and Choi, Y. (2019). Abductive commonsense reasoning. In International Conference on Learning Representations (ICLR) . 2, 3, 4

Bramley, N. R., Dayan, P., Griffiths, T. L., and Lagnado, D. A. (2017). Formalizing neurath's ship: Approximate algorithms for online causal learning. Psychological Review , 124(3):301. 2

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS) . 3

Cai, S., Wang, Z., Ma, X., Liu, A., and Liang, Y. (2023). Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. In Conference on Computer Vision and Pattern Recognition (CVPR) . 4

Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., and Batra, D. (2018). Embodied question answering. In Conference on Computer Vision and Pattern Recognition (CVPR) . 3, 4, 6

Ding, D., Hill, F., Santoro, A., Reynolds, M., and Botvinick, M. (2021). Attention over learned object embeddings enables complex visual reasoning. In Advances in Neural Information Processing Systems (NeurIPS) . 7

Douven, I. (2021). Abduction. In Zalta, E. N., editor, The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Summer 2021 edition. 1

Edmonds, M., Kubricht, James, F., Summers, C., Zhu, Y., Rothrock, B., Zhu, S.-C., and Lu, H. (2018). Human causal transfer: Challenges for deep reinforcement learning. In Annual Meeting of the Cognitive Science Society (CogSci) . 2

Edmonds, M., Ma, X., Qi, S., Zhu, Y., Lu, H., and Zhu, S.-C. (2020). Theory-based causal transfer: Integrating instance-level induction and abstract-level structure learning. In AAAI Conference on Artificial Intelligence (AAAI) . 2

Edmonds, M., Qi, S., Zhu, Y., Kubricht, J., Zhu, S.-C., and Lu, H. (2019). Decomposing human causal learning: Bottom-up associative learning and top-down schema reasoning. In Annual Meeting of the Cognitive Science Society (CogSci) . 2

Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y ., and Anandkumar, A. (2022). Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Advances in Neural Information Processing Systems (NeurIPS) . 4

Gigerenzer, G. and Gaissmaier, W. (2011). Heuristic decision making. Annual Review of Psychology , 62:451-482. 2

Goodale, M. A. and Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in Neurosciences , 15(1):20-25. 1

Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., and Farhadi, A. (2018). Iqa: Visual question answering in interactive environments. In Conference on Computer Vision and Pattern Recognition (CVPR) . 3, 4, 6, 10

Hafner, D. (2021). Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations (ICLR) . 3

Hanson, N. R. (1965). Patterns of discovery: An inquiry into the conceptual foundations of science . CUP Archive. 2

He, P., Gao, J., and Chen, W. (2022). Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In International Conference on Learning Representations (ICLR) . 7

Hessel, J., Hwang, J. D., Park, J. S., Zellers, R., Bhagavatula, C., Rohrbach, A., Saenko, K., and Choi, Y . (2022). The abduction of sherlock holmes: A dataset for visual abductive reasoning. In European Conference on Computer Vision (ECCV) . 2, 3, 4

Hobbs, J. R., Stickel, M. E., Appelt, D. E., and Martin, P. (1993). Interpretation as abduction. Artificial intelligence , 63(1-2):69-142. 2

Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation , 9(8):1735-1780. A4

Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. (2017). Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR) . 3, 4

Johnson, M., Hofmann, K., Hutton, T., and Bignell, D. (2016). The malmo platform for artificial intelligence experimentation. In International Joint Conference on Artificial Intelligence (IJCAI) . 4

Josephson, J. R. and Josephson, S. G. (1996). Abductive inference: Computation, philosophy, technology . Cambridge University Press. 2, 3

Kakas, A. C., Kowalski, R. A., and Toni, F. (1992). Abductive logic programming. Journal of Logic and Computation , 2(6):719-770. 3

Kayser, M., Camburu, O.-M., Salewski, L., Emde, C., Do, V., Akata, Z., and Lukasiewicz, T. (2021). e-vil: A dataset and benchmark for natural language explanations in vision-language tasks. In Conference on Computer Vision and Pattern Recognition (CVPR) . 2

Kenton, J. D. M.-W. C. and Toutanova, L. K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT . 7

Koˇ cisk` y, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., and Grefenstette, E. (2018). The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics (TACL) , 6:317-328. 10

Lei, J., Yu, L., Bansal, M., and Berg, T. L. (2018). Tvqa: Localized, compositional video question answering. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP) . 10, A5

Li, L., Chen, Y.-C., Cheng, Y., Gan, Z., Yu, L., and Liu, J. (2020). Hero: Hierarchical encoder for video+ language omni-representation pre-training. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP) . A5

Li, Q., Huang, S., Hong, Y., Zhu, Y., Wu, Y. N., and Zhu, S.-C. (2023). A minimalist dataset for systematic generalization of perception, syntax, and semantics. In International Conference on Learning Representations (ICLR) . 3

Li, Q., Zhu, Y., Liang, Y., Wu, Y. N., Zhu, S.-C., and Huang, S. (2022). Neural-symbolic recursive machine for systematic generalization. arXiv preprint arXiv:2210.01603 . 3

Liang, C., Wang, W., Zhou, T., and Yang, Y. (2022). Visual abductive reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR) . 2, 3, 4

Magnani, L. (2011). magnani2011abduction . Springer Science & Business Media. 3

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. nature , 518(7540):529-533. 6

Oaksford, M. and Chater, N. (1994). A rational analysis of the selection task as optimal data selection. Psychological Review , 101(4):608. 2

OpenAI (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774 . 3

Peirce, C. S. (1965). Collected papers of Charles Sanders Peirce , volume 5. Harvard University Press. 1, 2, 3

Poole, D. (1993). Probabilistic horn abduction and bayesian networks. Artificial Intelligence , 64(1):81-129. 3

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) . 7

Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. (2021). Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research (JMLR) , 22(1):1234812355. 6

Rieber, L. P. (1996). Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Educational Technology Research and Development , 44(2):43-58. 1

Rizzolatti, G., Fadiga, L., Fogassi, L., and Gallese, V . (1997). The space around us. Science , 277(5323):190-191. 1

Rocktäschel, T. and Riedel, S. (2017). End-to-end differentiable proving. In Advances in Neural Information Processing Systems (NeurIPS) . 3

Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI) . A6

Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015). Trust region policy optimization. In International Conference on Machine Learning (ICML) . 6

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 . 6

Shank, G. (1998). The extraordinary ordinary powers of abductive reasoning. Theory & Psychology , 8(6):841860. 2

Thagard, P. (1988). Computational philosophy of science . MIT press. 2

Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022). Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 . 3

Wang, Z., Cai, S., Liu, A., Ma, X., and Liang, Y. (2023). Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560 . 4

Wijmans, E., Datta, S., Maksymets, O., Das, A., Gkioxari, G., Lee, S., Essa, I., Parikh, D., and Batra, D. (2019). Embodied question answering in photorealistic environments with point cloud perception. In Conference on Computer Vision and Pattern Recognition (CVPR) . 6

Xie, S., Ma, X., Yu, P., Zhu, Y ., Wu, Y . N., and Zhu, S.-C. (2021). Halma: Humanlike abstraction learning meets affordance in rapid problem solving. In ICLR Workshop on Generalization beyond the training distribution in brains and machines . 4

Xu, M., Jiang, G., Liang, W., Zhang, C., and Zhu, Y. (2023). Interactive visual reasoning under uncertainty. In Advances in Neural Information Processing Systems (NeurIPS) . 3

Yang, A., Miech, A., Sivic, J., Laptev, I., and Schmid, C. (2022). Zero-shot video question answering via frozen bidirectional language models. In Advances in Neural Information Processing Systems (NeurIPS) . 7

Yu, L., Chen, X., Gkioxari, G., Bansal, M., Berg, T. L., and Batra, D. (2019). Multi-target embodied question answering. In Conference on Computer Vision and Pattern Recognition (CVPR) . 3

Yuan, L., Gao, X., Zheng, Z., Edmonds, M., Wu, Y. N., Rossano, F., Lu, H., Zhu, Y., and Zhu, S.-C. (2022). In situ bidirectional human-robot value alignment. Science Robotics , 7(68). 1

Zhang, C., Jia, B., Edmonds, M., Zhu, S.-C., and Zhu, Y. (2021a). Acre: Abstract causal reasoning beyond covariation. In Conference on Computer Vision and Pattern Recognition (CVPR) . 2

Zhang, C., Jia, B., Zhu, S.-C., and Zhu, Y. (2021b). Abstract spatial-temporal reasoning via probabilistic abduction and execution. In Conference on Computer Vision and Pattern Recognition (CVPR) . 3

Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022). Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 . 7

A 🔍 Conan Playground

🔍 Conan 's playground is a computationally efficient 2D open-world environment with diverse items and rich tasks. The most distinctive feature of 🔍 Conan 's playground over the original Crafter environment is that agents in 🔍 Conan leave diverse traces when interacting with the environment. These traces serve as the foundation for abductive reasoning; the detective has to effectively connect the traces to figure out what the vandal has done.

A.1 Items and Traces

Land Based on Crafter, there are three types of terrains that agents can walk on: sand , grass , and path . Sand and grass are soft surfaces where agents leave directional footprints after walking on them (see Fig. A1 first 2 rows in Columns 2 and 3 for examples). If a grid is left with more than one footprint, the footprints will become melded (Fig. A1 Column 4 in first 2 rows). Agents' actions will also leave traces on the terrain, e.g ., water on the ground (Fig. A1 Column 5 first 2 rows). If an agent gets injured, blood will be shed on the ground (Fig. A1 Column 6 first 2 rows).

Creatures There are four creatures in the playground: plant , cow , zombie and skeleton . plant grows from sapling to ripe plant. Cow randomly wander on the ground, whereas zombie and skeleton (monsters in general) will target agents in sight: zombie chases agents and skeleton shoots arrow at agents. Agents can fight with creatures and kill them. These actions will leave monster bodies on the ground.

Tools Agents can make tools on the table . There are 7 tools in total: bucket , wood_sword , wood_pickaxe , stone_sword , stone_pickaxe , iron_sword , and iron_pickaxe . These tools can be made using different materials and used for certain tasks. Both swords and pickaxes can be used to fight with creatures, but only pickaxes can be used in mining. Buckets can be used to collect water and lava.

Actions 🔍 Conan 's playground enables agents to interact with objects, non-playable characters, and even other agents in the playground. Agents can cut tree to get apple and wood , as well as collect sapling and grow plant (Fig. A1 Row 3). They can also mine with different tools to get stone , coal , iron , and diamond . Using these materials, agents can make bed for sleep, furnace for keeping monsters away and grilling food, table for making tools, etc . Of note, these items should be placed in an empty grid to use and they can be destroyed by monsters.

A.2 Achievements and Tasks

There are 60 tasks and 39 achievements in 🔍 Conan 's playground. We list all achievements in Tab. A1. Tasks are composed achievements. We select 60 nontrivial and meaningful tasks from all compositions in 🔍 Conan as the final task set.

A.3 Observation and Action

🔍 Conan offers both pixel representation and symbolic representation for training agents. For pixel representation, the environment returns a 900 ˆ 900 RGB image each time step for the detective 's

sand

sand_left

sand_up

sand_unknown

sand_water

sand_blood

grass

grass_left

grass_up

grass_unknown

grass_water

grass_blood

tree

apple_tree

tree_cut

plant

young_plant

ripe_plant

cow

dead_cow

dead_zombie

dead_skeleton

stone

stone_left

iron

iron_left

diamond

dia_left

bed

bed_left

furnace

fur_left

table

table_left

apple

beef

steak

bucket

water_bkt

lava_bkt

wood_sword

wood_pickaxe

stone_sword

stone_pickaxe

iron_sword

iron_pickaxe

arrow_left

fence

bow

player

sleep_player

dead_player

water

path

health

drink

energy

鼎

9 ˆ 9 local view. For symbolic representation, the environment returns a 9 ˆ 9 tensor, with each entry an index representing one of 50 grid types, covering materials, resources, objects, creatures, and etc . The agent is always at the center of the observation.

🔍 Conan affords a larger action space. See Tab. A2 for a detailed list of actions.

B 🔍 Conan Questions

B.1 Question Generation

Questions in 🔍 Conan are generated based on vandal 's task-finishing process. To generate a question, (1) we initialize a playground and put the vandal in it; (2) the vandal is randomly assigned a task; (3) the vandal tries to finish the task with the help of the pre-build parser and planner, and generates logs along the way; (4) a question is generated based on a certain part of the log. We randomly select a template from the template pool and fill placeholders with related objects in it. The answer is also parsed from the log. Other choices are sampled based on the question and the context to avoid unrelated choices that can be easily excluded.

B.2 Question Templates

Tab. A3 lists all the templates we use for generating questions.

B.3 Dataset Statistics

See Tab. A4 and Tab. A5 for details.

C Explorer

The Explorer in the detective is an RL agent. The agent receives an observation of a r 64 , 64 , 2 s tensor. This tensor combines the 9 ˆ 9 symbolic local view of the detective and a 64 ˆ 64 question mask.

The local view is zero-padded to 64 ˆ 64 . This ensures the agent knows its relative position on the map. Additionally, the mask is generated based on the question, with the area related to the question unmasked. The mask serves as the goal of the exploration policy.

All the RL baselines are trained for 10 8 steps. See more details below. Unless specified otherwise, parameters are set as default in Stable Baselines.

C.1 Model Details

DQN The DQN baseline is trained using a γ value of 0.98, a τ value of 1, a learning rate of 0.0001, a buffer size of 10 7 , and a batch size of 512. We leverage an MLP policy with two layers of 64 neurons each. The model is updated every 10 steps.

TRPO The TRPO baseline updates its policy with a special KL-divergence constraint on the distance between the new and old policies. We also leverage an MLP policy for TRPO, where the same multi-layer perceptron is used for both policy and value prediction.

RecurrentPPO The ReucrrentPPO baseline uses long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) as the recurrent policy. The LSTM layers' weights are initialized with standard Gaussian. We reset LSTM states at the end of the episode. The LSTMs for both the actor and the critic have the same architecture, with two LSTM layers of 256 neurons each.

C.2 Training Details

Explorers are firstly trained on long-horizon tasks as explained in the main text. These long-horizon tasks include 'get diamond,' 'get lava,' 'get water,' 'make iron sword,' 'make iron pickaxe' and 'eat steak.' These tasks can be further broken down into over 20 subtasks and have an average episode length of more than 200 steps. We generate 10,000 unique scenes with traces given these tasks and train explorers on them for 10 8 steps. Then the explores are fine-tuned on all tasks in 🔍 Conan for 10 7 steps.

We also show the frame rate per second (FPS) for different RL baselines during training in Fig. A2. As can be seen from the figure, DQN exhibits the highest training efficiency, reaching an FPS exceeding 3000. TRPO maintains a stable FPS of 2000. On the contrary, RecurrentPPO operates significantly slower, requiring over 96 hours to complete training with 128 subproc environments, whereas TRPO accomplishes the task in just 14 hours.

D VL Reasoning

In this section, we describe the experimental details for the Vision-Language (VL) models used in the paper.

D.1 Model Details

Vanilla-Trans For Vanilla-Trans, the visual features together with the text features are concatenated in the format of [frame_1, frame_2, ..., frame_n, question, choice_1, choice_2, ..., choice_4] . Visual features, if from the symbolic observation, are directly passed into the model. Otherwise, we utilize CLIP's pre-trained image encoder

0

2e7

4e7

6e7

8e7

1e8

Step

0

1000

2000

3000

4000

5000

FPS

TRPO-500

RecurrentPPO-500

DQN-500

RecurrentPPO-5000

DQN-5000

(ViT-B/16) to extract features from pixel input. Text features are calculated using the text encoder of CLIP. These input features are then passed through a 6-layer Transformer model with an MLP head for classification.

FrozenBiLM We adopt the cross-modal FrozenBiLM for 🔍 Conan , drawing inspiration from models used in Multiple-choice VideoQA benchmarks such as How2QA (Li et al., 2020) and TVQA (Lei et al., 2018) 1 . 🔍 Conan can be formulated as a multiple-choice VideoQA problem given the fixed explorer. We concatenate all of the observation frames as the video input. The questions and choices are converted into the following format: ['{question} Is it {choice_1}?,' ..., '{question} Is it {choice_4}?'] . We then evaluate the probabilities of the model producing 'Yes' and 'No'. The visual features are processed in the same way as in VanillaTrans and then forwarded for visual-text projection. We utilize BERT-Large and DeBERTa as our frozen language backbones in this work; however, other general language models are applicable as well.

Flamingo-Mini Our Flamingo-Mini baseline is based on an open-source implementation of the Flamingo model 2 , as the original Flamingo model's pre-trained weights are not accessible. FlamingoMini is built upon OPT-125M and CLIP pre-trained ViT-L/14 model. We also formulate 🔍 Conan as a multiple-choice problem for Flamingo-Mini. The questions and choices are converted into the following format: ['Question: {question} Answer: {choice_1},' ..., 'Question: {question} Answer: {choice_4}'] . Each question-choice pair is fed into the model and then a binary classifier head is used on Flamingo's last layer output to predict the final answer.

D.2 Training Details

Vanilla-Trans was trained for 100 epochs, with a batch size of 128 . FrozenBiLM models were trained for 50 epochs, with a masking probability (for the MLM objective) of 0 . 15 , a batch size of 32 , a learning rate of 3 ˆ 10 ´ 4 , a gradient clipping max norm of 0 . 1 , and Adam as the optimizer ( β 1 ' 0 . 9 , β 2 ' 0 . 95 , ϵ ' 1 ˆ 10 ´ 8 ). Flamingo-Mini was trained for 100 epochs, with a learning rate of 5 ˆ 10 ´ 5 , a batch size of 8 , and also Adam as the optimizer ( β 1 ' 0 . 9 , β 2 ' 0 . 999 , ϵ ' 1 ˆ 10 ´ 8 ).

E Additional Experiments

E.1 Negative Control Baselines

We compare our VL reasoning results on the trained explorers with those on empty visual inputs as a negative control baseline. The results are shown in Tab. A6.

The results show that using empty visual inputs yields random performance across all settings. Besides, it also shows that the training QA pairs are unbiased. The TRPO explorer achieves higher performance, which suggests that the exploration strategy learned by TRPO helps gather some informative evidence for the reasoning process. The Ideal explorer is an oracle-like exploration policy that has access to perfect trace evidence and temporal information. It provides the most comprehensive information about the environment. This highlights the importance of effective exploration in improving reasoning performance. However, it does not mean that reasoning is less important, as even with the Ideal explorer, the model still could not achieve satisfactory performance. Based on all results, collecting informative evidence seems to be more important in the overall objective.

F Abduction from Deduction (AfD)

As mentioned in Sec. 4.4, we adopt a data-driven strategy to learn a model of P p g | S q and simultaneously answer the questions. To be more specific, we train the detective agent self-supervisedly. The detective is randomly assigned with one of all possible tasks. It then finishes the task by following the action policy π p¨q . Note that we assume the detective 's π p¨q is the same as the vandal 's in order to best implement the idea of AfD. Based on the task execution process, questions are generated. Since our ultimate goal is to have our models answer 🔍 Conan 's questions, we do not explicitly construct P p g | S q , but rather consider the question-answer process as the g . We then train P p g | S q , where S is the detective 's observation during the task execution, and the label can be derived from the assigned tasks together with the π p¨q .

Besides P p g | S q , we still need to learn a model of P p S | O q , which, intuitively, can be understood as inferring the true state of the environment from partial observation. In our experiment, we tried two ways to model P p S | O q . One approach is to directly train a model using multi-frame observations to predict the states. We employed a UNet (Ronneberger et al., 2015) and a multi-layer CNN as the network. However, this method did not work effectively. Reasoning based on the reconstructed states only achieved performance at a random level. The second approach, which was finally used to report performance, aligned the hidden feature spaces from true states and observations. When training P p g | S q , we added a head before the VL models, converting the input S into a 4096-dimensional vector. Then we trained a head on O with the same structure, minimizing the difference between features from O and features from S .

G 🔍 Conan Task Demo

7

7

7

To better illustrate the core components in 🔍 Conan , We take the playground shown in Fig. 1 as an example. In this scenario, the assigned task is 'get diamond' (Fig. A3 shows the task dependency). As shown in Fig. A4, once the vandal completes the task, it leaves behind traces in the playground. The vandal ends at the bottom of the figure. The detective then enters the playground, starting at the beginning of the traces. In this case, traces encompass footprints and remnants left after certain actions. Note that footprints cannot be left on sand or stone, and different footprints may overlap. The vandal will collect objects crafted on a table, making them invisible.

Let's suppose the detective's exploration begins by following footprints (note the context window size is 9 ˆ 9).

Firstly we can see some cut trees. As the footprints are not seriously overlapped and mostly one-directional, we can deduce the vandal did not return. After seeing the tool-making table, with the only resources being wood, we could say that the vandal could only make wooden tools, not stone swords or iron pickaxes, further restricting possible actions the vandal took.

Note that this is already critical reasoning in 🔍 Conan .

Moving on, we note that footprints become missing on the sand surface. However, we note broken stones and coals. Therefore, the wooden tool to break stones and coals shall be a wooden pickaxe. So the agent should have made a wooden pickaxe on the table earlier. Despite the fact that the tool has been collected, we could still figure that out.

Following the reemerged footprints, we note blood and a zombie body on the ground, suggesting the vandal should have had a fight.

Searching on, we find the broken diamond. As an iron pickaxe is the only tool to collect diamonds. The vandal must have built an iron pickaxe with iron and coal in the furnace. With no other footprints around, we can safely conclude our search.

.9

detective

vandal'straces

vandal

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Conan, a new benchmark for evaluating active reasoning in an embodied/open-world environment where an agent acts as a detective and must answer questions about the actions and intentions with the traces of another agent (a vandal). The experiments benchmark a RL + vision-and-language baseline and a more structured model that uses Bayesian inference on the Conan task.\n\nSTRENGTHS:\n- ambitious benchmark trying to tackle an exciting and interesting problem that highlights a limitation in the current focus of the community: lack of benchmarks testing reasoning/question answering with active information gathering\n- problem and setting are well-motivated\n- interesting setup for the active reasoning problem with a vandal/detective\n\nWEAKNESSES:\n- many important details that are not clear. One of the main things that needs more clarity is what the format of the benchmark is - it would be very helpful to have more examples and earlier in the paper of what the traces look like / what context is available to models, or maybe an example of a series of reasoning steps a model (or human) would go through to answer a question in Conan. See questions below for specific points of confusion\n- I wasn't totally convinced/clear that the questions in Conan are focused on testing abductive reasoning. Although eventually we would want to solve this with end-to-end methods like the RL baselines presented here, I feel like some heuristic exploration baselines would be important to understand what it takes to solve the benchmark and contextualize the scores of the RL agents (what scores do these agents get?):\n\t- the ideal explorer (specifically wondering: is the ideal explorer the ideal policy?)\n\t- an agent that explores the whole 64x64 grid (how much better does the ideal explorer do compared to this?)\n\t- using an oracle extractor that extracts the trace-relevant frames instead of the every-k key-frame extractor\n\t- instead of using RecurrentPPO, some heuristic extractor that allows agents to condition on key frames in the past (e.g. the frames with traces that are closely associated with the question)\n- Related to previous point, the benchmark also conflates *memory* and *exploration* with the abductive reasoning challenges -- both of these are already huge challenges for RL/current models. I worry that these may make the benchmark too difficult, and additionally not actually test for the reasoning abilities that it claims to test. Overall, I just don't have a good understanding of what solving the benchmark actually entails - if we have agents with good memory + exploration, is the reasoning involved actually quite simple? I think heuristic baselines/agents with privileged information would help the case here, as well as simplifying aspects of the task that are orthogonal to abductive reasoning\n\nQUESTIONS:\n- table 1: what's the difference between embodied and open-world?\n- l146: what does a trace look like? are objects left behind, does the state change, etc.?. l151: when you say \"32 distinctive traces\" do you actually mean \"32 object interactions\" and traces can consists of multiple object interactions?\n- l153: \"60 abductive reasoning tasks\" - does this mean 60 question templates? \n- l164-168: what's an example of a path + alternative and why exactly does this require the detective to gather additional information? \n- why is the detective allowed to take the same actions as the vandal instead of just exploring the scene and collecting information? (feels like potentially unnecessary complexity)\n- l226-227: \"masks are first employed to encode questions...\" - what does this sentence mean?\n- How does the explorer condition on the question (e.g. some sentence embedding of the question or something else?)\n- Is there a step limit on the explorer's episodes?\n- l174-177: this made it sound like the detective and the agent were separate. My understanding is that the agent *is* the detective?\n- l255: what is the representation of symbolic map?\n- l307: to clarify, does the ideal explorer get the frames of the ground truth vandal trajectory, and then we extract every 30 frames? how much of the gap in performance is due to the keyframe extractor dropping important frames vs. the inaccuracy of the reasoning model?\n- sec 4.4: does this assume access to the vandal policy pi? Is that realistic?\n- l345: why would vanilla-trans be more susceptible to insufficient evidence? Isn't a more likely explanation that it's not pretrained to do question answering, in contrast to the other models?\n- AfD doesn't improve on standard (small improvements at best / mixed results across the board) - doesn't this suggest that this kind of reasoning is not super important in the benchmark?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper introduces Conan, an interactive open-world environment for evaluating the active reasoning abilities of agents. In Conan, agents need to answer questions by actively seeking for evidence and acquiring new knowledge in a setting of incomplete information.
Conan is formulated as a detective game. First, during the initialization of the game, a vandal agent completes a task, leaving traces in the environment. Next, questions are generated for a detective agent to answer. The detective has to answer these questions by actively interacting with the environment to reconstruct the actions of the vandal.

STRENGTHS:
* The environment differs from existing benchmarks for its active reasoning and interactive multi-round setting
* The paper is very well written and easy to follow
* Conan is an interesting environment that could spur further research on active reasoning in interactive contexts

WEAKNESSES:
* It is not clear how challenging the task is and how good the performance of the models reported in the paper is. A human study would help the reader understand this critical point.
* A qualitative analysis of the games and the behavior of the different agents would also be needed in the main paper to understand when the agents make mistakes and to what extent the tasks are solvable
* As mentioned in the paper, it looks like the explorer does not supply particularly relevant information to the reasoning model, substantiating the intuition that the bottleneck is there, rather than in the reasoning abilities of the model

QUESTIONS:
Is a human study or an error analysis feasible?

LIMITATIONS:
The authors described the limitations of the paper
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a benchmark for \"active reasoning\" titled Conan, where, instead of passively answering questions from provided context, an agent must interactively explore its environment to discover information. The authors differentiate such a task from so-called \"passive reasoning\" tasks such as video-language understanding where the visual input is directly fed into the model. Here, a model itself is responsible for exploring and attaining input that will help a model answer a question.\n\nConan is implemented in a Minecraft-like gridworld where a \"detective\" agent must identify activities that a rule-based, goal-oriented \"vandal\" agent completed, by using traces of the vandals' behavior in the environment. E.g. if a vandal's goal was to make a wooden pickaxe, it may have cut some trees down, and the task for the model will be to answer why the vandal decided to make a wooden pickaxe.\n\nTwo alternative methods are adopted to approach this task: one which uses an explorer agent, trained with an exploration reward, to generate evidence that is fed into a standard VLM, and another, \"Abduction from Deduction\", where one directly learns to predict the goal of the vandal from an inferred state trajectory.\n\nOverall this is an interesting paper, and I think the dataset and benchmark will be valuable for the community. The models tested for this environment are relatively simple, not really making full use of the \"joint exploration and reasoning\" abilities that Conan purportedly tests, however. Moreover I do have some outstanding questions, and some issues with the experiments (specifically a missing baseline) that prevent me from assigning a higher score. However I'm open to changing my score after the author response and discussion period.\n\nSTRENGTHS:\n- To my knowledge, the \"active reasoning\" component of Conan is an important area in vision/lang/RL, and is certainly underexplored in current embodied QA settings (although it is probably implicitly present in instruction following benchmarks and work like SayCan).\n- The dataset seems high quality and should be a useful contribution for the field.\n- Fairly sound experimental evaluation, comparing a variety of RL explorer agents with a variety of vision-language models.\n\nWEAKNESSES:\n- **Missing negative control baselines.** The experimental results should compare evidence gathered by the trained explorer agents to a weaker negative baseline, e.g. a random untrained explorer policy or no explorer input at all. This is needed to convincingly show that the explorer agents are actually learning to explore in a way that is more beneficial than chance, at least for the downstream QA task.\n    - In general I'm definitely more interested in to what extent performance is bottlenecked by good exploration on this task, rather than fixing the explorer evidence and trying subtly-differently-trained large VLMs.\n- As mentioned by the in the conclusion (L376-377), one of the key promises of Conan is that strongest performance on this benchmark should intuitively be achieved by models that jointly learn to reason and explore, but right now a decoupled two-stage process is adopted—first training an exploration model with an oracle \"find trace\" reward, extracting relevant keyframes, then training a VLM on top of the frozen policy. As a result, models trained for this task (for now) look like typical embodied QA models doing passive reasoning (just with the extra step of training an explorer to get the vision input). Of course, it's not strictly necessary for a benchmark paper to include an strong novel model for the benchmark, but without it we aren't able to evaluate whether Conan performance numbers have **headroom** for more sophisticated approaches to joint exploration and reasoning. (It seems like this should be the case, but we can't really be sure).\n- Some more outstanding questions (see next section)\n\nQUESTIONS:\n- The idea of answering questions via \"traces\" left behind in the environment is a little hard to grasp and could use more explaining. There are some example traces in Figure 1b but these are small figures and aren't explained that much. The authors could spend some more time walking through example \"traces\" and more fully convince readers that the traces indeed provide answers to some of the more subtle questions in the benchmark (e.g. how do traces indicate that the vandals goal is to \"get the diamond\", couldn't the vandal have alternative goals like get iron/coal?).\n\t- It might be valuable to have an \"oracle\" setting to demonstrate that with perfect trace evidence and enough training models can 100% this task—or else explain why such an oracle doesn't exist (e.g. what causes the lagging performance even for the \"Ideal Explorer\" setting in Table 3? Is it an amount of training data issue? Are there still limitations as to the quality of the trajectory attained even by the ideal explorer, and is there an even more suitable oracle for this?)\n\t - Related to this, can authors further explain L308 ideal explorer: \"visible to the ground-truth vandal’s trajectory\"? What does \"visible to\" mean—shouldn't the ideal explorer literally *be* the ground-truth vandal's trajectory?\n- More details could be provided on the specific kinds of generalization splits tested here. At test time, is there a subset of abductive reasoning tasks held out that the model has never seen? Or does the model see the same questions it has seen during training, just in unseen environments? Do the test environments differ systematically from those during train (or is it possible to induce such a )\n\t- Evaluating (and supporting) such compositional/systematic generalization splits would greatly increase the appeal of the benchmark.\n - (Related to the above) Do authors have an intuition as to whether test-time performance drop is more due to the VLM's inability to generalize, or the explorer's inability to generalize and generate good evidence for unseen enviroments (if test does indeed have unseen environments)? Is it possible to disentangle the two using different evaluation splits in the environment?\n - In Figure 4, does a score of near 100 as attained by eg TRPO indicate that the explorer indeeds find all traces (+100) reward all of the time? If not, is there an explicit measure of that?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the gap in handling incomplete-information questions, this paper introduces an interactive open-world environment called \"Conan.\" The purpose of Conan is to motivate and evaluate agents' active reasoning ability by requiring them to explore, gather evidence, and combine knowledge to solve complex scenarios. The ultimate goal is to improve AI agents' ability to interact with and learn from the world.\n\nSTRENGTHS:\nThis paper introduces a new interactive open-world environment called \"Conan.\" The purpose of Conan is to motivate and evaluate agents' active reasoning ability by requiring them to explore, gather evidence, and combine knowledge to solve complex scenarios. The ultimate goal is to improve AI agents' ability to interact with and learn from the world.\n\nWEAKNESSES:\nThe author claims that the detective spawned in the environment needs to answer all questions through exploration, connecting key frames, and reaching conclusions. Yet, some questions presented in the Figures/Tables can be answered without exploring the environment. The process of answering questions through exploration requires more clarity and elaboration.\n\nRegarding the reward function used in the task, the author mentions that it incentivizes the agent to search for clues and traces related to the given question. However, the paper lacks detailed information about how the reward is designed and the specific motivation behind it.\n\nThe writing appears to be cumbersome and illogical, making it difficult to grasp the actual meaning of the task. The purpose of introducing this new task is not clearly stated, leaving readers with questions about its significance and potential applications.\n\nIn conclusion, while the paper introduces an intriguing concept of an interactive open-world environment for AI agents, there are aspects that require further clarification and refinement. Addressing the limitations and providing more detailed explanations about the task and reward design would enhance the overall understanding and impact of this research.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Conan, an interactive environment as a benchmark to evaluate agent’s active abductive reasoning abilities to answer questions in an incomplete (or partial) information scenario. Because of partial information, the model requires further exploration in the scene to answer the questions which is posed as a detective game. The paper also proposes the Abduction from Deduction (AfD) approach which relies on Bayesian statistics. The experimental evaluation showcases the strengths and weaknesses of various machine learning models in different settings.\n\nSTRENGTHS:\n- The paper is well-motivated and clearly written in most parts. \n- The authors promise the availability of corresponding code and a detailed description of hyperparameters which would be essential for reproducibility if open-sourced. \n- The paper introduces the AfD approach using Bayesian rules which is interesting to the research community. \n- Relevant baselines have been selected to comprehensively evaluate various RL and multi-modal models.\n\nWEAKNESSES:\n- The proposed environment looks like a toy benchmark with limited real-life applications.\n- Dataset statistics is missing in the work which might expose data biases related to goal, intents or survival questions.  \n- The question templates are also limited and it is not clear if the model learns to take advantage of any data biases to answer the questions. It is also not clear if the questions are asked in a particular order. \n- This is framed as a multi-choice QA task with limited generalizability. \n- The proposed approach lacks a close integration between exploration and reasoning processes where the training and evaluation for different models is independent.\n\nQUESTIONS:\n- Can you elaborate on how this research can be improved to integrate exploration and reasoning processes in future work?\n- What was the main motivation behind maintaining the detective set invincible, ie. not maintaining the survival status on line 178? \n- On line 363, the authors mention that the proposed approach is akin to “human-like reasoning”. It is not clear as to how solving this toy task with an independent explorer and reasoning model brings us closer to “human intelligence”\n\nSuggestions: \n- It would help to make the best results in bold in Table 3.\n\nLIMITATIONS:\nSee weaknesses for more information about the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}