Active Reasoning in an Open-World Environment
Authors: Manjie Xu
Guangyuan Jiang
Wei Liang
Chi Zhang
Yixin Zhu
Venue: NeurIPS 2023 poster (2023)
Keywords: Visual Reasoning
Abductive Reasoning
Active Reasoning
TL;DR: we unveil Conan, an innovative interactive platform designed to bridge the gap in active reasoning capabilities between humans and AI, challenging current models to engage in an open-world environment for multi-round abductive reasoning.

Abstract: Recent advances in vision-language learning have achieved notable success on *complete-information* question-answering datasets through the integration of extensive world knowledge. Yet, most models operate *passively*, responding to questions based on pre-stored knowledge. In stark contrast, humans possess the ability to *actively* explore, accumulate, and reason using both newfound and existing information to tackle *incomplete-information* questions. In response to this gap, we introduce **Conan**, an interactive open-world environment devised for the assessment of *active reasoning*. **Conan** facilitates active exploration and promotes multi-round abductive inference, reminiscent of rich, open-world settings like Minecraft. Diverging from previous works that lean primarily on single-round deduction via instruction following, **Conan** compels agents to actively interact with their surroundings, amalgamating new evidence with prior knowledge to elucidate events from incomplete observations. Our analysis on \bench underscores the shortcomings of contemporary state-of-the-art models in active exploration and understanding complex scenarios. Additionally, we explore *Abduction from Deduction*, where agents harness Bayesian rules to recast the challenge of abduction as a deductive process. Through **Conan**, we aim to galvanize advancements in active reasoning and set the stage for the next generation of artificial intelligence agents adept at dynamically engaging in environments.

Active Reasoning in an Open- World Environment
Manjie Xu
Guangyuan Jiang?
manjietsu@bit. edu.cn
jgy@stu. pku. edu.
Wei Liang 1,3,5
Chi Zhang 4.B3
Yixin Zhu 2. 2.
Liangwei@bit. edu.
zhangchi@bigai.at
rixinnzzu@ppp edu,
School of Computer Science & Technology, Beijing Institute of Technology 2 Institute for AI, Peking University
3 Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China
National Key Laboratory of General Artificial Intelligence, BIGAI
Abstract
Recent advances in vision-language learning have achieved notable success on guestion-answering datasets through the integration of exten- sive world knowledge. Yet, most models operate passively, responding to questions based on pre-stored knowledge. In stark contrast, humans possess the ability to actively explore, accumulate, and reason using both newfound and existing in- formation tackle eecomplete-iffommatinn questions. In response to this gap, we introduce Q Conan, an interactive open-world environment devised for the assessment of active reasoning. Q Conan facilitates active exploration and pro- motes mulli-round abductive inference, reminiscent of rich, open-world settings like Minecraft. Diverging from previous works that lean primarily single-round deduction via instruction following, Q Conan compels agents to actively interact with their surroundings, amalgamating new evidence with prior knowledge eluci- date events from incomplete observations. Our analysis on Q Conan underscores the shortcomings of contemporary state-of-the-arr models in active exploration and understanding complex scenarios. Additionally, we explore Abduction from De- duction, where agents harness Bayesian rules recast the challenge of abduction as deductive process. Through Q Conan, we aim to galvanize advancements in active reasoning and set the stage for the next generation of AI agents adept dynamically engaging in environments.
Introduction
Active interaction with the environment is fundamental to human understanding of the world around us. Both neural and behavioral studies indicate that through active engagement with their surroundings, humans garner critical insights and foster a profound understanding of complex phenomena (Goodale and Milner, 1992; Rizzolatti al., 1997; Rieber, 1996). When confronted with partial or ambiguous data, our innate response is to seek supplementary evidence, hypothesize, and put forth possible explanations. sometimes even reevaluating initial assumptions (Yuan et al., 2022). This iterative process persists until satisfactory resolution emerges.
The process of formulating theories based observations and prior knowledge is classically termed as abductive reasoning or simply, abduction (Peirce, 1965; Douven, 2021). 1). A topic of enduring
Work done while M. Xu was an intern at Peking University.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Task Initializer
The Playground
Question Answering
Goal driven: Get
Intert Why did the criminal cut these trees?
Survieal-drivem:
vandal
To make & wood pickase.
Porser
Planner
criminal's goal?
Navigate
To get
Survicet What happemed between the criminal and the sombie?
Actions Traces
Move left
The criminal lought and
killed the zomble despitc being intured.
right
Key Framos
Mine__co coal
Cook, hoof
Kill zombie
Get lava
detective
(b)
(c)
Figure 1: An example of Q Conan, an open-world environment for active reasoning. (a) Q Conan initialization. A vandal is randomly assigned task from the task space while keeping alive. A probabilistic parser, utilizing knowledge graph, selects sequence of subgoals to fulfill the main objective. This decision then conveyed to planner which, turn, invokes heuristic policies to execute atomic actions. Some these actions leave discernible traces within the environment. (b) Q Conan playground with traces. (c) Q Conan questions. Here, a detective is spawned and is tasked with answering queries. does so by actively exploring the environment, connecting keyframes, and reaching conclusions.
interest among psychologists, abduction is perceived as a cornerstone of human cognitive processes. Historical and contemporary studies have delved into its cognitive mechanisms (Josephson and Josephson, 1996; Thagard, 1988; Peirce, 1965), practical applications (Hobbs al., 1993; Shank, 1998), and ties to scientific thinking and decision-making (Hanson, 1965; Gigerenzer and Gaissmaier, 2011: Zhang et al., a)) With growing momentum in the machine learning sphere, recent years have witnessed the advent of dedicated benchmarks and models accentuating abductive reasoning (Bhagavatula al., 2019; Kayser al., 2021; Hessel al., 2022; Liang et al., 2022).
However, the bulk of prior work in this domain relies heavily on a single-round, passive question- answering paradigm that offers complete information. This setup often ing to queries, leveraging vast pre-trained knowledge, as evidenced by the latest strides language and vision-language learning. Recent progress in the field has notably already improved performance information question-ans answering. Contrarily, humans demonstrate a far more nuanced approach when navigating abductive scenarios with incomplete data (Edmonds et al., 2018). We actively engage, explore, gather, and reason, drawing from both new information and prior knowledge. Our iterative approach allows for continuous refinement based newly acquired evidence (Oaksford and Chater, 1994; Bramley et al., 2017; Edmonds et al., 2019, 2020).
agent simply respond-
in such complete-infomaale
To capture the dynamic and exploratory essence of abductive reasoning-a we present Q Conan, a new open-world environment tailored for abductive reasoning. Standing head and shoulders above traditional single-round passive reasoning benchmarks, Q Conan boasts an open-world arena, urging agents to actively probe surroundings and engage in multi-round abductive inferences, all while leveraging in-situ collected evidence alongside pre-existing knowledge.
termedd herein active
At its core, Q Conan is conceived as a detective game, transmuted into a question-answeriin challenge. Here, the detective is tasked with a query and an "incident scene" riddled with traces left by a vandal. Given the initial paucity of conclusive information, the detective must embark on an in-depth exploration of the scene. As the inquiry progresses, the detective has the opportunity to actively scout its environment, continually reshaping and honing its hypotheses, especially when new revelations potentially contradict the prior hypothesis. Furthermore, meticulously craft questions within Q Conan to span various levels of abstraction, from localized intentions (Intent) to overarching objectives (Goal) and survival states (Survival).
To probe the proficiency of active reasoning, we evaluate state-of-the-art Reinforcement Learning (RL) and multimodal question-answering models on Q. Conan. Our observations highlight intriguing dichotomy: while these cutting-edge models exhibit prowess in addressing low-level, short-term tasks, they struggle with multi-round environmental interactions and high-level abductive reasoning.

---

A plausible root of this challenge could be the absence of structurally represented knowledge. Predicated predominantly on associative training, these agents are versed in correlating traces with responses without genuinely internalizing holistic world models. In sharp contrast, humans seamlessly navigate abductive reasoning by forecasting potential trajectories leading to a perceived outcome. This intricate dance gradually transmutes from abductive to deductive reasoning, where humans harness their innate understanding causality to deduce and mirror observed patterns. In our pursuit to mirror this quintessential human trait, we integrate Abduction from Deduction (AID) into Q Conan via Bayesian approach. Experimental results underscore the efficacy of AfD, indicating substantial avenue for bolstering agent adeptness in Q Conan.
To sum up, our work makes the following three contributions:
We usher in the novel domain of active reasoning. underscoring the indispensable roles of active exploration and iterative inference in abductive reasoning. This paradigm shift transforms traditional single-round passive question-anse -answennnnn paradigms into a more immersive format, compelling agents to actively engage with the environment to procure pivotal evidence.
We introduce Q Conan, a new environment tailored to evaluate the abductive reasoning ability of current machine learning models within dynamic settings. Q, Conan surpasses its predecessors that hinge on step-by-step deductive reasoning, revealing the limitations of present-day models.
learning method for abduction, AfD, grounded in Bayesian principles. This framework elegantly reformulates abduction into deduction, proving instrumental in navigating the complex active reasoning challenges posed by Q Conan.
We formulate
2 Related Work
Machine Abductive Reasoning Abductive reasoning, foundational human cognition, is crucial for scientific exploration, decision-making, and problem-solving (Peirce, 1965; Magnani, 201 1). In the Artificial Intelligence (AI) landscape, there rich history efforts to equip machines with this ability, where they use prior knowledge and sparse observations to hypothesize amidst uncertainty (Josephson and Josephson, 1996; Xu et al., 2023). Key developments span logic-based abduction (Kakas et al., 1992; Poole, 1993) and hybrid neural-symbolic methods (Rocktäschel and Riedel, 2017; Zhang et al., 2021b; Li et al., 2022, 2023). With computational progress, Large Language Models (LLMs) have effectively addressed several challenges through text generation, exhibiting outstanding performance (Brown et al., 2020; OpenAl, 2023; Thoppilan et al., 2022). Modern research usually frames abductive reasoning within natural language understanding (Bhagavatula et al., 2019) or multimodal lasion-language integration (Hessel et al., 2022; Liang et al., 2022). However, there is still notable gap: many benchmarks lean heavily deduction, sidelining abduction's 's interactive essence. Our work addresses this gap, emphasizing the core of active reasoning in abductive contexts.
Embodied Question Answering Embodied question answering enhances traditional Visual Ques- tion Answering (VQA) by placing agents in interactive environments (Johnson et al., 2017; Das et al., 2018; Gordon et al., 2018; Yu et al., 2019). In Q Conan, agents actively explore to gather data, preparing them to solve abductive questions based on partial information. Unlike standard embodied question-ans where questions become simple instructions for agents, Q Conan introduces complexity: (i) its questions, rooted in high-level intent and goals, resist simple decomposition into series of actions; (ii) agents in Q Conan act as detectives, constantly hypothesizing from observations and prior knowledge, and iterating their strategies in light of new data. For comprehensive comparison of Q Conan with other benchmarks, see Tab. 1.
frameworks (Das et al., 2018; Gordon et al., 2018; Yu et al., 2019),
3 The Q Conan Environment
Q. Conan is crafted as an interactive cresston-annserrigg environment aimed at evaluating a ma- chine's s active abductive reasoning capacity, as depicted in Fig. Building on the foundation of the Crafter (Hafner, 2021), Q Conan evolves into a detective game featuring two agents: the vandal and the detective. The gameplay kickstarts with the vandal undertaking a randomly designated task, leaving behind traces for the delective to unravel. Subsequently, given these traces, pertinent queries are generated. Finally, the detective spawned in the environment, tasked with navigating these traces and actively probing the environment, all to derive answers through abductive reasoning.
3

---

Table Comparison between Q Conan and related visual reasoning benchmarks. Conan is unique for its active reasoning and interactive multi-round setting on abductive reasoning tasks.
Benchmark
Format
Multimodal
Interactive
Multi-round
Abductive
CLEVR (Johnson et al., 2017)
image
X
X
IQA (Gordon et al., 2018)
embodied
X
X
EmboliedQA (Das et al., 2018)
embooiee
/
X
X
(Bh(Baagaaattlla et al., 2019)
language
X
X
X
V
VAR (Liang al., 2022)
video
X
V
Sherlock (Hessel al., 2022)
image
X
V
Q Conan (Ours)
open-world
/
3.1 Basic Components
Playground Originating from the Crafter playground, Q Conan operates within 64 x 64 grid matrix. Agents navigate this space with a lo- calized 9 9 grid field of view centered on their current position. Once the detective is created in the environment, all traces left behind by the vandal persist, serving as clues for the detective unravel. While pivotal studies (Johnson et al., 2016; Fan et al., 2022; Cai et al., 2023; Wang al., 2023) address perception in 3D Minecraft settings using foundational models, our empha- sis is on honing active abductive reasoning. To this end, we transition from 3D visual percep- tion to a plane, ensuring a harmonious blend of reduced visual complexity and retaining rich interactivity (Xic et al., 2021).
Survive
Collect
Make
Defeat
Items and Actions Q Conan offers an ex- tensive assortment of interactive items: food, materials, mobs, and tools, each tied to specific actions, as illustrated in Fig. 2. It furnishes 26 unique actions to foster agent-environt eeveromenir en- gagement. Certain actions leave traces, and together, the items and their mechanics provide a rich set of affordances for agents in the playground. This knowledge about item operations and traces aids the detective comprehending the incident scene. Advancing from its predecessor, the original Crafter, Q Conan now boasts over 30 achievements, significant rise of over 50%. It features 32 distinct traces covering all agent actions such as crafting, collecting, defeating, cating, drinking, and incurring injuries. This enhancement enables the design of 60 varied abductive reasoning tasks within the scene. For an in-depth overview the playground, refer to Appx. A.
Figure 2: Part of the task dependency graph. Starting from root note, any path forms multi-step task for an agent to interact with the environment.
Vandal Each Q Conan map starts with the initialization of a vandal. This agent is driven by two primary aims: executing specific task and preserving its existence within the environment. It is noteworthy that external threats might terminate the vandal prematurely. Traces left in the aftermath of the vandal's activities form the question foundation for the detective, with every trace potentially birthing several questions. For a detailed overview, see Sec. 3.2. We model the vandal as optimal: when given a random task and the full map, strategically delineates a sequence of subgoals based on the task dependency graph, all while ensuring its survival. In scenarios with multiple viable paths to an objective, uniform sampling comes into play. This sampling, supported by probabilistic parser, presents varied strategies for task completion. Hence, the detective must delve deeper to distinguish the actual sequence of events from possible decoys. The execution of the vandal's 's individual actions, per the planned subgoal sequence, is steered by collection of pre-established policies.
Detective After generating questions from a given trace, detective spawned to answer them. Traces left by the vandal span multiple steps and are only partially observable within the detective's s x 9 grid field of view. This requires the detective to actively interact with the environment and gather evidence to answer the questions.

---

Though both detective and vandal share the same action space, the detective boasts unique capability. not only navigates and interacts like the vandal, but can also generate its own traces during its investigation. These overlaid traces from the defective enhance the environment's s depth and complexity. This setup pushes the agent to actively derive conclusions from its dynamic interactions. Importantly, the detective is invulnerable; its focus lies squarely problem-solving eliminating concerns about survival or evasion. This design emphasizes active exploration and reasoning, ensuring Q Conan's 's primary goal remains addressing complex reasoning tasks and answering visual scene- related questions.
3.2 Questions and Choices
Q Conan is designed to assess the abductive reasoning capability of machine models through a diverse set of questions varying in difficulty and abstraction. These questions fall into three primary categories: Intent (local intent). Goal (global goal), and Survival (agent's 's survival status change). We approach evaluation
multi-choice guestion-answerrs with only one being correct. Questions and choices derive from predefined templates, as showcased Tab. 2. For a more detailed explanation, see Appx. B.I.
task. Each question offers four choices,
Table 2: Examples of three categories of questions in Q Conan created from predefined templates.
Type
Questions
What did the vandal make on this table?
A: wood sword; B: wood pickaxe; C: iron sword; D: stone sword;
Intent
Why did the vandal cut a tree here?
A: make table; B: make wood sword; C: make finance; D: collect apple:
What was the vandal's 's primary objective in this scenario?
A: get diamond; B: defeat zombie; C: collect apple; D: make iron sword;
Goal
What was the desired outcome of the task performed by the vandal?
A: make steak; B: make table; C: defeat skeleton; D: collect lava;
Why did the vandal die in this situation?
A: lack of water; B: lack of food; C: hurt by monster; D: hurt by lava;
Survival
What could the vandal have done differently to avoid negative outcome?
A: avoid monsters; B: get sleep; C: get food; D: get water;
Intent questions target the vandal's immediate objectives or intentions during its task. To decipher these traces, agents must deduce the vandal's underlying intent subgoals. Solving these questions necessitates a learning model's comprehension of the local context.
Goal questions probe the vandal's s overarching objectives, extending beyond immediate intents. They necessitate grasping the wider context of a task or action sequence. Such questions query the vandal's ultimate aims, demanding learning model to reason within the broader context of the traces.
Survival questions address the wider investigative scope, posing added challenges to the detective. Centered on the vandal's ' survival status changes during tasks (e.g., collecting food for sustenance), they lead to deviations from the optimal action plan. While not tied to a task's primary objective, these questions require a deeper grasp of the present context, often necessitating reasoning around potential scenarios or alternate results.
Compared with the prevalent VQA setup. wherein questions are based on factual information that readily obtainable from the input, Q Conan questions cannot be deciphered given only the initial information, necessitating further exploration in the scene. Unlike standard embodied question answering. Q Conan questions cannot be directly parsed as modular primitives; they demand abductive reasoning, drawing from both new observation and former knowledge to hypothesize, validate, and revise. For benchmarking purposes, Q Conan produced a corpus comprising 100,000 questions. These were derived from 10,000 unique scenes, generated via the Crafter's 's scene generator, with each scene stemming from a task executed by vandal. This resulted in an average generation of 10 questions per scene.
5

---

Explorer
Key-frame Extractor
Image Encoder
Deploration Trank
Key Traces
Incident Seene
Vision-Languee Language Model
Text Eneuder
Text Encoder
Linear
Länear
Reward
Explorer
Question: What's the criminal's intention?
Choice
ag
Conan
Action
Choice A
To get the diamond.
Playground
Figure An illustration of detective pipeline Q Conan. An explorer is first trained to gather traces accordance with the given question. Given a question and the incident scene, the detective calls the explorer subroutine to gather evidence. Next, the exploration sequence undergoes key-frama, extraction, processed by visual encoder, subsequently feeding into vision- sainuala model for answer selection.
4 The Detective Pipeline
Q Conan casts the abductive reasoning challenge as a detective game, necessitating a detective to efficiently explore and gather information from the environment deduce plausible explanations (i.e.. answers) for the given question. This process involves taking into account the temporal dependencies and incompleteness of the traces. tackle these challenges encountered in Q Conan, we devise detective pipeline, as depicted in Fig. 3.
Building on previous work that utilizes hierarchical models for task decomposition (Gordon et al., 2018; Das et al., 2018; Wijmans et al., 2019), our pipeline is structured into two primary phases; an exploration phase for trace collection, followed by an abductive reasoning phase. Initially, interaction with the playground is carried out to collect relevant visual information, which is subsequently leveraged in the reasoning phase infer answers to the posed questions.
Computationally, our pipeline first employs RL agents as explorers (see Sec. 4.1) that learn an exploration policy based the traces and the question, thereby rendering goal-oriented. Next, given the question, we recruit vision-language models (see Sec. 4.3) to predict the answer based on the observation. A key-frame extractor (see Sec. 4.2) is inserted into the two phases selectively identify relevant frames for abduction. The individual components undergo separate training procedures.
4.1 Explorer for Trace Gathering
The primary responsibility of an explorer is to efficiently collect information pertinent to the provided question. Initially, masks are employed to encode questions by highlighting relevant grids. Subse- quently, the explorer takes in both the observation and the target question as input and outputs the action probability.
We use a reward function that incentivizes the agent to scout for clues and traces relevant to the given question. Additionally, penalty term incorporated to discourage unnecessary actions and inefficient searching, thereby promoting more targeted exploration strategy.
Specifically, the agent is rewarded with +1 when trace first appears within its local view, +2 when the trace bears a close association with the question. A substantial reward of + 100 conferred upon the agent if it successfully uncovers all traces left by the vandal. Concurrently, the agent incurs a penalty of -0.1 for every timestep elapsed, with an additional penalty of - imposed for executing operating actions.
evaluate multiple well-regarded RL frameworks as our explorer, including Deep Q-Network (DQN) (Mnih et al., 2015), Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), and Recurrent Proximal Policy Optimization (RecurrentPPO) (Schulman al., 2017). The Stable- Baselines3 library (Raffin al., 2021) is employed for all implementations.

---

4.2 Key-Frame Extractor
Given that the frames gathered by the explorer tend to be excessively lengthy and redundant, key- frame extractor is utilized to sift through and select informative frames containing crucial evidence for the detective. We adopt a prevalent selection strategy employed in video understanding (Amab et al., 2021). Specifically, frames within the temporal bounds determined by the detection of the first and last traces are retained, from which & frames are uniformly sampled. This design is intended to tailor the input with the constrained context window size to downstream vision-language models.
4.3 Vision-Language Models for Abductive Reasoning
We employ multi-choice question-answerre paradigm akin the one used in Ding al. (2021). Specifically, the model is presented with a question, its corresponding exploration frame sequence, and each potential answer choice, subsequently generating score for each choice. The model is trained with a categorical cross-entropy loss. During inference, the choice with the highest score is considered the answer. We evaluate several well-established multimodal models; these models are known for their efficacy in processing both visual and textual data. Additional details on model implementation can be found in Appx. D.I.
Vanilla-Trans Trans The first baseline method leverages a vanilla transformer encoder to fuse observation and textual inputs. Specifically,the raw symbolic map from Q Conan serves the visual feature, while CLIP's text encoder (Radford et al., 2021) is employed to encode the textual input.
FrozenBiLM FrozenBiLM (Yang et al., 2022), a hate-of-the-art model for video question an- swering, combines visual input with frozen bidirectional language models, trained on web-scraped multimodal data. The approach integrates a frozen language model and frozen vision encoder with light trainable visual projection modules. FrozenBiLM is tested with BERT-Large (Kenton and Toutanova, 2019) and DeBERTa-v3 (He et al., 2022) as the language model within question- answering system, utilizing the symbolic map from Q Conan for visual input.
Flamingo-Mini Flamingo (Alayrac et al., 2022) is a family of vision-languse language models adept at rapid adaptation to novel tasks with minimal annotated examples. These models can handle sequences of visual and textual data, seamlessly accommodating interleaved images or videos as input. We finetune an open-sourced Flamingo-Mini model with frozen OPT-125M (Zhang et al., 2022), utilizing the symbolic map from Q Conan for visual input.
4.4 Abduction from Deduction (AfD)
The adage "Set a thief catch thief" suggests the of someone with a similar background or expertise to apprehend a wrongdoer: the best vandal catchers are vandals. This notion resonates with the core principle of Abduction from Deduction (AfD): for a skillful detective to abduce what a vandal does, it needs an in-depth grasp of vandals' modus operandi, motivations, and decision- making process. Translating the implication to mathematical language, we articulate the problem of abductive reasoning based on evidence and knowledge from known deductive transitions. It can also be seen as an extension of inverse planning (Baker al., 2007, 2009; Baker and Tenenbaum, 2014). Formally, let g denote the goal of the vandal, O the detective's ' observation, and S the playground states post the vandal's actions. We then have:
P(g O)
E P(S)0)
[P(o [(O)) = F. [P(g ()].
(1)
P(S
where we assume the independence of g w.r.t. O given S, as the goal ought to be clear given the states. Leveraging Bayesian rules, we further observe that
P(g S) < P(S g) < <(a; s;.g),
(2)
assuming uniform prior over & and known deterministic environment transitions. Eq. (2) asserts that P(g S) is proportional to a goal-conditioned forward action policy, where $;, a; - 8i+1-
Intuitively, Eqs. (1) and (2) can be understood as follows: to abduce the vandal's goal from observa- tion, it is imperative to first reconstruct the actual states traversed by the vandal and subsequently ascertain the most plausible goal that, if pursued forward, would result in those states; see Eq. (1). Eq. (2) can be interpreted as a form of deduction, being contingent on transition knowledge derived from forward action policy. Hence the name Abduction from Deduction (AID).
7

---

In practice, two approaches emerge for implementing P(g S) based on Eq. (2). The first entails iterating over all g and utilizzig learned or predefined π (-) to score a lengthy sequence of states. Conversely, the second approach embraces data-driven strategy, wherein one arbitrarily selects samples S from (T), and learns a model of P(g ()) using the (g. S) pairs. The former approach proves time-intensive during inference due to the combinatorial temporal space and expansive goal space, thereby compelling us towards the latter approach. For implementation, we train P(S [O] independently Dirac delta function of (((()) and P(g S) from sampled pairs from x(.) employed in task execution in the vandal. The derived goal features, along with the question, are fed into the model for answer prediction. Please refer to Appx. F for additional details.
5 Experiments
5.1 Experimental Setup
Exploration The explorer is trained using DQN, TRPO, and RecurrentPPO for 10th steps, with buffer size of 10? and a batch size of 512. the case of DQN, training is conducted with E 0.96. Each episode is capped at a maximum of 500 steps for the explorer. A curriculum is employed to encourage long-term exploration whilst maintaining balance with local search: initial training is carried out with traces from long- Long-horizon tasks like "get the diamond," compelling the agent to venture farther from its starting point. Subsequently, the agent undergoes further finetuning across the entire dataset. Such curriculum design prevents sole focus local discovery. For downstream reasoning models, 30 keyframes are extracted by the key-frame extractor.
Abductive Inference Our reasoning models are tested under three different settings: Standard, Ideal Explorer, and AfD. In the Standard setting, models undergo training and testing based on the explorer's exploration. The Ideal Explorer setting sees models leveraging on optimal exploration policy-visible to the ground-truth vandal's trajectory, albeit imperfect, facilitates the agent in gathering sufficient evidence for reasoning. This scenario can be conceived reasoning model's s aptitude for passive reasoning given complete information. Under the AfD setting, models are trained and used as delineated in Sec. 4.4. All models are trained utilizing 8 NVIDIA GeForce RTX 3090 GPUs. For further training specifics, please refer to Appx. D.2.
measure of the
5.2 Results and Analysis
Fig. shows the learning curves of vari- ous RL agents during exploration. TRPO and RecurrentPPO manifest similar performance in terms of rewards following a substantial number of steps, markedly surpassing the DQN explorer. Additionally, we probe the impact of augment- ing the maximum number of exploration steps to 5, 000 on performance. The data suggests marginal performance uplift. Nonetheless, we acknowledge that such performance increment at the expense of substantially longer explo- ration time and notable surge in the accrual of unrelated information. Consequently, we select TRPO with maximum of 500 steps per episode as our standard RL explorer.
Reward
TRIPO $30
Tocumartt 300
20N 503
TRINO
407
B67
Step
Figure 4: Learning curves of various RL explorers. The suffix n denotes the maximum number steps per episode during exploration. Results show that (i) TRPO and RecurrentPPO markedly outperform DQN in perfor- mance, and (ii) longer episodes marginally contribute the performance at the expense of longer exploration time and the accrual of unrelated information.
Quantitative results on Q Conan are depicted in Tab. 3; both the standard and AID results reported employ TRPO as the explorer. In the standard setting, we discern that while models exhibit some aptitude in tackling low-level Intent questions, they struggle with higher-level questions pertaining to Goal and Survival. Among the models, Flamingo- Mini ascends to the pinnacle with an accuracy of 66.3%. FrozenBiLMM models also perform relatively well. Notably, the DeBERTa variant slightly outperforms BERT, insinuating that a robust language backbone can improve general comprehension. Contrarily, the Vanilla- Vanilla-Trans model languishes across all tasks, achieving merely random-level performance. perfoommacce....

---

Table 3: Performance of abductive reasoning models Conan. We report the question-answering accuracy (%) across various settings, with the overall accuracy averages over question categories. F-BiLM refers to the FroZenBiLM model. denotes Intent, G denotes Goal, S denotes Survival, and O denotes Overall. Results exhibiting the top individual performance are highlighted in bold, while models with the superior overall performance shaded in gray.
Standard
Idcal Explorer
AfD
8
0
G
S
Vanilla-Trans
32.9
25.0
24.5
28.8
64.0
78.4
58.1
66.1
24.8
23.3
24.5
24.3
F-BiLM-BER?
72.6
44.4
54.4
61.0
87.5
59.5
61.5
74.0
82.8
42.9
55,5
66.0
B-ETRa
82.9
43.1
52.2
65.3
87.7
71.8
63.9
77.8
82.9
41.9
53.8
65.4
Flamingo-Mial
86.2
43.3
49.5
66.3
85.8
47.8
56.6
69.0
84,9
42.5
52.2
66.1
With the Ideal Explorer, we notice a clear performance boost across all tasks, particularly in the Goal and Survival questions. These results allude to the potential bottlenecking of models' abductive reasoning capability due to the insufficient information collected, underscoring the significance of effective exploration. An adept explorer can significantly aid in the accrual of useful information, informatively pursuing a hypothesis to scrutinize evidence, swiftly self-correcting upon encountering conflicting evidence, and reasonably re-planning. The findings also hint sufficient room for the RL explorer to improve. Remarkably, the Vanilla-Trane exhibits the greatest increase, insinuating that, in comparison other baseline models, it is markedly vulnerable insufficient evidence.
For AfD results, nearly all multimodal models exhibit performance on par with end-to-end supervis- edly trained models. Remarkably, FrozenBiLM models surpass the performance observed in standard settings. The persisting failure Vanilla-Trame can be ascribed to its weakness in reasoning amidst incomplete observations due to the significant disparity between the familiar complete state S and incomplete observation O. Examining task-specific results, a notable performance uplift in the Survival task models discernible for almost all models relative the standard setting, albeit sharing the same observation. These results intimate that the inclusion of deductive information sensitizes the detective to vandal's concerns during task execution. Nevertheless, the exhibited performance in long-term planning remains weak, reinforcing the pressing need for better exploration policy, Critically, these models continue find short-term intent questions to be most easily answered.
5.3 Further Discussion
Additional Experiments We further experiment in the absence of visual inputs, serving as negative control baseline, resulting in random performance across all settings; see Appx. E. This random-level performance underscores the severe constraints imposed on the agent without visual information. The TRPO explorer shows noticeable improvement over the ones without visual inputs, suggesting that even minimal exploration is preferable to none. Nonetheless, the performance remains relatively modest. On the other hand, the Ideal Explorer demonstrates markedly superior performance, attesting to the substantial benefits its capacity to accrue perfect trace evidence renders to the downstream reasoning task. This accentuates the imperative of effeective exploration.
Table 4: Error analysis on Q Conan. We examine the accuracy of BrozenBil MoozenBiLM-DeBBBB M-DeBERTa across various tasks, comparing two explorer groups: reasoning based on the TRPO explorer and the Ideal explorer (in gray).
get drink
de feat .ow
get _appleppeee_____ 35.7
dafeat etale 46.59
make__ ron pf akaxe 56.52 52.17
47.06
43.90
100.00
85.37
78.57
82,95
place bed bad
make steak 46.15
make stonn botone_piicceeeee
gez get_ccel cccllcll
make wtw.ssttteeeeeeeeeeeeeeeee sword 37.50 4.17
43.90
48.48
50.00
87.80
50.00
39.39
45.45
get_Lren
get_wäter
sret_stooe 36.84
make. _ron__soree
place_ Furnace
place_furnace
28.57
45.95
$6.25
44,
46.43
$4.05
47.37
28.12
83.95
ges amond 40.62
p1 aase_tablle
ges_wwod 36.00
wood pi pickaxe wood_sword
39.36
40.00
$0.00
84.38
91.49
96.00
$5.00
64.29
make bbdd 47.83
get._ Lave 50.00
make, make_buucked buccket
gee__bef $3.85
defeat. leteat__oommae zomble
35.29
$2.50
39.13
66.67
73.53
42.31
75.00

---

Error Analysis We extend an error analysis for the "goal" split, probing the reasoning model across a spectrum of tasks. Table compares two groups: reasoning based on the Ideal explorer and the TRPO explorer. The findings underscore that proficient exploration, i.e., the heuristic Ideal explorer who recovers the vandal's 's trajectory, is sufficient for satisfactory performance. However, to fully harness the potential, a more adept reasoner is requisite, one capable of deciphering the vandal's hidden states from observed traces. For instance, the act of felling trees could signify a need for either wood or food (apples), and discerning the intent solely from traces of felled trees presents challenge. When it comes to "trace-relevant" frames or "keyframes," " the Ideal explorer could ostensibly furnish all trace-relevae relevant frames. However, the concept of keyframes remains nebulous. Within the video understanding domain, a formidable challenge lies in the extraction of "keyframes." " This is a post-hoc concept that cludes straightforward acquisition upfront. A prevailing approach, aimed at augmenting efficiency (diminishing context length in Transformer), entails truncating it via every k-th frame.
Joint Reasoning The collective enhancement of both exploration and reasoning elements emerges as quintessential, given its mirroring of human-like intelligence. For instance, by providing feedback, the reasoner can steer the explorer towards actions that are potentially more insightful and likely to produce pertinent traces. Nonetheless, practical implementation encounters significant hurdles. Assigning credit to exploratory decisions bearing long-term implications can be intricate, particularly when the outcomes of exploratory actions become evident after a substantial time lapse, thereby muddying the causal relationship between the decisions and their ultimate effect reasoning and answering questions. This accentuates the mutual reliance between exploration and reasoning advancement in one facet demands progression in the other, introducing bilateral dependency that complicates optimization. The reasoning component alone demands hefty training and computational resources, especially when utilizing large language models. The demand for formidable computational power renders the simultaneous optimization of exploration and reasoning exceedingly daunting. Collectively, this approach is also widely adopted (Gordon et al., 2018; Lei et al., 2018; Kočiský et al., 2018). Consequently, we navigate along this trajectory, projecting that future endeavors on Q Conan should prioritize reasoning above exploration.
To summarize, the engagement of a proficient explorer substantially enhances abductive reasoning. particularly in higher-level tasks such goal-oriented and survival-centrie inquiries. This underlines the criticality of exploration as a precursor to tackling abductive reasoning tasks in the presence of incomplete information. Furthermore, the achievement of the AfD hint at the potential for models to harness world knowledge, especially transition knowledge pertaining to tasks and traces, to transform abductive reasoning into deductive simulation. We posit that the presented approach resonates more with human-like reasoning, edging us closer to the core of human intelligence.
6 Conclusion
In this paper, we introduce Q Conan, a benchmark tailored to evaluate and assess models' active reasoning ability in addressing incomplete-ing incomplee-iinfommatoon informatioooonnnn questions in an interactive environment. Q Conan sets itself apart from existing abductive reasoning benchmarks by incorporating an open- world playground facilitating active exploration. It differentiates itself from prevailing embodied aasseiinn benchmarks by introducing the demanding abductive process question answer- ing, necessitating multi-round abductive inference based on gathered evidence. Moreover, we propose a new learning paradigm, Abduction from Deduction (AfD), that turns the problem of abduction to deduction, exploiting the problem structure through Bayesian principles. Benchmarking the efficacy of contemporary machine learning models on Q Conan, we elucidate the model limitations in interacting with the environment that leads to failure in higher-level, longer-term abductive reasoning.
Limitations and Future Work In general, we notice two significant limitations from the ex- perimental results. For one, the explorer does not supply particularly relevant information for the reasoning model. In the human abductive reasoning process, exploration and reasoning should be closely intertwined, with an agent using the current hypothesis to guide exploration and improve its understanding. However, due to long-range exploration and complex vision- language reasoning, we only applied the conventional visual processes. For another, learning naive question-anse mapping shall be sub-optimal. By leveraging the problem structure, AfD has shown improved performance on particular set of problems. Never- theless, the current AfD formulation is still rudimentary. We believe an in-depth understanding of the structure and well-crafics implementation could further boost performance.
-anstiinnnnswrinng method and did not fully integrate these two
10

---

The authors would like to thank Ms. Zhen Chen (BIGAI) for designing the figures, and NVIDIA for their generous support of GPUs and hardware. M.X., G.J,, W.L., C.Z., and Y.Z. are supported in part by the National Key R&D Program of China (20227D01 14900), M.X. and W.L., are supported in part by the NSFC (62172043), and Y.Z. is in part by the Beijing Nova Program.
Acknowledgement
References
Alayrac, J.-B., Donahue, J., Luc, P., Micch, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022). Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS).
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C. (2021). Vivit: video vision transformer. In International Conference on Computer Vision (ICCV). 7
Baker, C. L.. Saxe. R., and Tenenbaum, J. B. (2009). Action understanding as inverse planning. Cognition, 113(3):329-34. 7
Baker, C. L. and Tenenbaum, J. B. (2014). Modeling human plan recognition using bayesian theory of mind. In Plan, activity, and intent recognition: Theory and practice, volume pages 177-204. Morgan Kaufmann Waltham.
Baker, C. L., Tenenbaum, B., and Saxe, R. R. (2007). Goal inference as inverse planning. In Annual Meeting of the Cognitive Science Society (CogSci). 7
Bhagavatula, C., Le Bras, R., Malaviya, C., Sakaguchi, K., Holtzman, A., Rashkin, H., Downey, D., Yih, W.-t,, and Choi, Y. (2019). Abductive commonsense reasoning. In International Conference on Learning Representations (ICLR). 3,4,
Bramley, N. R., Dayan, P., Griffiths, L.. and Lagnado. D. (2017). Formalizing neurath's 's ship: Approximate algorithms for online causal learning. Psychological Review, 124(3)3301, 2
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS). 3
Cai, S., Wang, 7., Ma, X., Liu, A., and Liang, Y. (2023). Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. In Conference on Computer Vision and Pattern Recognition (CVPR).
Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., and Batra, D. (2018).. Embodied question answering. In Conference on Computer Vision and Pottern Recognition (CVPR). 3, 6,
Ding, D., Hill, F., Santoro, A., Reynolds, M., and Botvinick, M. (2021). Attention over learned object embeddings cnables complex visual reasoning. In Advances Neural Information Processing Systems (NeurIPS).
Douven, I. (2021). Abduction. Zalta, E. N., editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2021 edition.
Edmonds, M., Kubricht, James, F., Summers, C., Zhu, Y., Rothrock, B., 7hu, S.-C., and Lu, H. (2018). Human causal transfer: Challenges for deep reinforeement learning. In Annual Meeting of the Cognitive Science Society (CogSci).
Edmonds, M., Ma, X., Qi. S., Zhu, Y., Lu, H., and Zhu, S.-C. (2020). Theory-based based causal transfer: Integrating instance-level induction and abstract-level structure learning. In AAAI Conference on Artificial Intelligence (AAAT). 2
Edmonds, M., Qi, S., Zhu, Y. Kubricht, J., Zhu, S.-C., and Lu, H. (2019). Decomposing human causal learning: Bottom-up associative learning and top-down schema reasoning. In Annual Meeting of the Cognitive Science Society (CogSci).
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H.. Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. (2022). Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neurud Information Processing Systems (NeurIPS). 4
Gigerenzer, G. and Gaissmaier, W. (2011). Heuristic decision making. Annual Review of Psychology, 62:451-44 4822.
2

---

Goodale, M. A. and Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in Neurosciencex, 15(1):20-25.
Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., and Farhadi, A. (2018). lqa: Visual question
answering interactive environments. Conference on Computer Vision and Pattern Recognition (CVPR). 3, 4, 6, 10
Hafner, D. (2021), Benchmarking the spectrum of agent capabilities. International Conference on Learning Representations (ICLR).
Hanson, N. R. (1965). Patterns of discovery: An inquiry into the conceptual foundations of science. CUP
Archive. 2
He, P.. Gao, J. and Chen, W. (2022). Debertav3: Improving deberta using electra-style pre-training with gradient-disentanlled embedding sharing. In International Conference Learning Representations (ICLR).
Hessel, J., Hwang, J. D., Park, S., Zellers, R., Rhagavatula, C., Rohrbach, A., Saenko, K., and Choi, Y. (2022). The abduction sherlock holmes: A dataset for visual abductive reasoning. European Conference on Computer Vision (ECCV). 2, 3,4
Hobbs, J. R., Stickel, M. E., Appelt, D. E., and Martin, P. (1993). Interpretation as abduction. Artificial intelligence, 63(1-2):69 142. 2
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9()):1335 1735-178-7 A4
Johnson, J., Hariharan, B., Van Der Maaten, L., Foi-Fei, L., Lawrence Zitmick, C., and Girshick, R. (2017). Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Conference Computer Vision and Pattern Recognition (CVPR). 3,
Johnson, M., Hofmann, K., Hutton, T., and Bignell, D. (2016). The malmo platform for artificial intelligence experimentation. International Joint Conference on Artificial Intelligence (IJCAI).
Josephson, J. and Josephson, S. G. (1996). Abductive inference: Computation, philosophy, technology. Cambridge University Press. 2,
Kakas, A. C., Kowalski, R. A., and Toni, F. (1992). Abductive logic programming. Journal of Logic and Computation, 2(6):719-77.
Kayser, M., Camburu, O.-.., Salewski, L., Emde, C., Do, V., Akata, Z., and Lukasiewicz, T. (2021). e-vil: A dataset and benchmark for natural language explanations in vision-language tasks. In Conference on Computer Vision and Pattern Recognition (CVPR).
Kenton, J. M.-.W C. and Toutanova, L. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT.
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, M., Melis, G., and Grefenstette, (2018). The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics (TACL), 6:317-328. 10
Lei, J., Yu, L., Bansal, M., and Berg, T. L. (2018). Tvqa: Localized, compositional video question answering. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP). 10, AS
Li, L.. Chen, Y.-... Cheng, Y., Gan, Z., Yu, L., and Liu, J. (2020). Hero: Hierarchical encoder for videot language omni- renneeeeprsnntttionnn pre-training. In Annual Conference Empirical Methods in Natural Language Processing (EMNLP). A5
Li, Q., Huang, S., Hong, Y., Zhu, Y., Wu, N., and Zhu, S.-C (2023), minimalist dataset for systematic generalization of perception, syntax, and semantics. International Conference on Learning Representations (ICLR). 3
Li, Q.. Zhu, Y., Liang, Y., Wu, N., Zhu, S.-C., and Huang, S. (2022). Neural-symholic recursive machine for systematic generalization. arXiv preprint arXiv:2210.01603, 3
Liang C., Wang, W., Zhou, T., and Yang. Y. (2022). Visual abductive reasoning. In Conference Computer Vision and Pottern Recognition (CVPR). 2, 3.4
Magnani, L. (2011). magnani201 labduction. Springer Science & Business Media.
12

---

Mnih. V. Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J. Bellemare, M. G., Graves, A., Riedmiller, M.. Fidjeland, A. K. Ostrovski, G., al. (2015). Human-level control through deep reinforcement learning. nature, 518(7540):529-333
Oaksford, M. and Chater, N. (1994), A rational analysis of the selection task as optimal data selection. Psychological Review, (0)6666 2
OpenAl (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 3
Peirce, C. S. (1965). Collected papers of Charles Sanders Peirce, volume 5. Harvard University Press, 1,
Poole, D. (1993). Prohabilistic horn abduction and bayesian networks. Artificial Intelligence 64(1):81-129.
Radford, A., Kim, W. Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML). 7
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. (2021). Stable-basclines:: Reli- able reinforcement learning implementations. Journal of Machine Learning Research (JMLR), 22:1):12348- 12355. 6
Rieber, I.. P. (1996). Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Educational Technology Research and Development. 44(2):43-58 58.
Rizzolatti, G., Fadiga, L.,, Fogassi, L., and Gallese, V. (1997). The space around us. Science, 277(5323))190- 191.
Rocktäschel, T. and Riedel, (2017). End-to-end differentiable proving. Advances in Neural Information Processing Systems (NeurIPS). 3
Ronneherger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Asiisted Intervention (MICCAI). A6
Schulman, J., Levine. S., Abbeel, P., Jordan, M., and Moritz, P. (2015). Trust region policy optimization. In International Conference on Machine Learning (ICML). 6
Schulman, J. Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:/707.060 1707.06347 6
Shank, G. (1998). The extraordinary ordinary powers of abductive reasoning. Theory & Psychology, 8(6):411 860.
Thagard, (1988). Computational philosophy science. MIT press. 2
Thoppilan, R., De Freitas, D., Hall, J.. Shazeer, N., Kulshreshtha, A., Cheng. H.-T.. Jin, A.. Bos. T., Baker. L.. Du, Y., al. (2022). Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239
Wang, Z., Cai, S., Liu, A., Ma. X., and Liang, Y. (2023). Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint a0X.v:2230..01560.
Wijmans, E., Datta, S., Maksymets, O., Das, A., Gkioxari, G., Lee, S., Essa, I., Parikh, D., and Batra, D. (2019). Embodied question answering photorealistic environments with point cloud perception. In Conference on Computer Vision and Pattern Recognition (CVPR).
Xie. S., Ma, X., Yu, P., Zhu, Y., Wu, Y. N., and Zhu, S.-C. (2021). Halma: Humanlike abstraction learning meets affordance rapid problem solving. In ICLR Workshop Generalization beyond the training distribution in brains and machines.
Xu, M., Jiang, G., Liang, W., Zhang, C., and 7hu, Y. (2023). Interactive visual reasoning under uncertainty. In Advonces Neural Information Processing Systems (NeurIPS).
Yang, A., Miech, A., Sivic, J., Laptev, I., and Schmid, C. (2022). Zero-shot video question answering via frozen bidirectional language models. Advances Neural Information Processing Systems (NeurIPS).
Yu, L.. Chen. X., Gkioxari, G., Bansal, M., Berg. T. L.. and Batra, D. (2019). Multi-target embodied question answering. Conference on Computer Vision and Pattern Recognition (CVPR). 3
Yuan, L., Gao. X., Zheng. Z., Edmonds. M., Wu, Y. N., Rossano, F, Lu, H., Zhu, Y., and Zhu, S.-C. (2022). In situ bidirectional human robot value alignment. Science Robotics, 7668). 1
13

---

Zhang, C.. Jia, B., Edmonds, M., Zhu. S.-C., and Zhu, Y. (2021a). Acre: Abstract causal reasoning beyond covariation. Conference Computer Vision and Pattern Recognition (CVPR). 2
Zhang, C.. Jia, B., Zhu, S.-C., and Zhu, Y. (2021b). Abstract spatial-temporal reasoning via probabilistic
abduction and execution. In Conference Computer Vision and Pottern Recognition (CVPR). 3
Zhang, S., Roller, S., Goyal, N., Artetxe, M.. Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.
(2022). Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00m6 000000 7
14

---

A
Q Conan Playground
Q Conan's playground computationally efficient 2D open-world environment with diverse items and rich tasks. The most distinctive feature of Q Conan's 's playground over the original Crafter environment that agents in Q Conan leave diverse traces when interacting with the environment. These traces serve as the foundation for abductive reasoning; the detective has to effectively connect traces to figure out what the vandal has done.
A.1 Items and Traces
Land Based Crafter, there are three types of terrains that agents can walk sand, grass, and path. Sand and grass are soft surfaces where agents leave directional footprints after walking on them (see Fig. A1 first 2 rows in Columns 2 and for examples). If grid is left with more than footprint, the footprints will become melded (Fig. A1 Column 4 in first rows). Agents' actions will also leave traces the terrain, e.g., water the ground (Fig. Al Column 5 first rows). If an agent gets injured, blood will be shed on the ground (Fig. A1 Column 6 first 2 rows).
There are four creatures in the playground: plant, cow, zombie and skeleton. plant grows from sapling to ripe plant. Cow randomly wander on the ground, whereas zombie and skeleton (monsters in general) will target agents in sight: zombie chases agents and skeleton shoots arrow at agents. Agents can fight with creatures and kill them. These actions will leave monster bodies on the ground.
Creatures
Tools Agents can make tools on the table. There are tools in total: bucker, wood_sword. wood pickodeiceeeeee stone_sword stone pickaxe_ iron sword, and iron pickaxx. These tools can be made using different materials and used certain tasks. Both swords and pickaxes can be used fight with creatures, but only pickaxes can be used in mining. Buckets can be used to collect water and lava.
Actions Q Conan's playground enables agents to interact with objects, non-playable characters, and even other agents in the playground. Agents can cut free get apple and wood, well as collect sapling and grow plant (Fig. Al Row 3). They can also mine with different tools to get stone, coal, iron, and diamond. Using these materials, agents can make bed for aleep.furnace for keeping monsters away and grilling food, table for making tools, etc. Of note, these items should be placed in an empty grid use and they can be destroyed by monsters.
A.2 Achievements and Tasks
There are 60 tasks and 39 achievements in Q Conan's playground. We list all achievements in Tab. A1. Tasks are composed achievements. We select nontrivial and meaningful tasks from all compositions in Q Conan as the final task set.
Table Al: Achievements in Q Conan.
Type
Achievements
drink_wacan
eat_apple sleep_on_bed
eat_beet Make_up
eat_steak
Survive
sleep
eat_gri1Ied__ppp.
"rsnn__wate__fro_biicco
est_plant
collect_wood
collect_apple
collect_wad _ater
cellect_stone
Collect
sollest_iron
collect_diamont
collect_bed
collect_roal
collest_water
collest_lava
colloat__apiiee
callact_plaan
make_steak
make_grilled_appl
buck_b make_stong__s aword_ place_table
nake_fence
Make
make make_wood_nng
make_izon_swoze
wood_aword
make_wod__iicke
make_iron_pickaxe
make_stona_pa pickaee cickaxe
place
furnace
place_pplatt
Defeat
defeas_.oo
defeat_zzmbie
defeat_ skeleten
A.3
Observation and Action
Q Conan offers both pixel representation and symbolic representation for training agents. For pixel representation, the environment returns 900 900 RGB image each time step for the detective's
A1

---

sand
sand_
sand up
sand unknown
sand_water water
sand_blood
grass
grass left
grass up
grass unknown
grass water
grass blood
free
apple tree
tree cut
plant
young. plant
ripe plant
cow
dead_cow
zombic
dead_ domb__
skeleton
lend_sseletoo
stone
stone_let
iron
iron left
diamond
dia left
bed
bed_left
furnace
fur_left
table
tablc_left
apple
beef
steak
bucket
water_bk bkt
lava_ba bkt
wood_sword
pick_pic
stone_sword
stone_ stone_pickaxe pickaxe
iron_sword
iron cin__ickaxe
arrow left
fence
bow
player
sleep_ player
dead player
water
path
lava
health
drink
energy
Figure Al: Items and related traces Q Conan.
A2

---

9 x 9 local view. For symbolic representation, the environment returns a 9 x 9 tensor, with each entry an index representing one of 50 grid types, covering materials, resources, objects, creatures, and etc. The agent is always at the center of the observation.
Q Conan affords larger action space. See Tab. A2 for detailed list of actions.
Table A2: Actions in Q Conan.
Action
Details
Noop
Do nothing.
Move Left
Move left the grid is walkable.
Move Right
Move right if the grid is walkable.
Move Up
Move if the grid is walkable.
Move Down
Move down if the grid is walkable.
Do
Collect materials or fight with monsters. Use tools possible. Sleep to restore energy. Sleep on bed can restore energy faster; Place stone if the grid is not occupied. Should have stone, Place table if the grid is not occupied. Should have table. occupied. Should have furnace. Place plant if the grid is grass. Should have sapling. Place a bed the grid is not occupied. Should have bed. Nearby table. Should have wood.
Sleep
Place Stone
Place Table
Place Furnace
Place furnace if the grid is
Place Plant
Place Bed
Make Wood Pickaxe
Make Stone Pickaxe
Nearby table. Should have wood, stone. Nearby table, furnace. Should have wood, coal, iron. Nearby table. Should have wood.
Make Iron Pickaxe
Make Wood Sword
Make Stone Sword
Ncarby table. Should have wood, stonc.
Make Iron Sword
Ncarby table, furnace. Should have wood, coal, iron.
Make Bucket
Nearby table. Should have wood, stone.
Make Steak
Nearby table, furnace. Should have beef. Restore 2 health. Should have apple. Restore 4 health. Should have beef.
Eat Apple
Eat Beef
Eat Steak
Restore 6 health. Should have steak. Collect water to bucket. Should have empty bucket.
Collect Water
Collect Lava
Collect lava to bucket. Should have empty bucket.
Drink
Drink water. Drink water from water bucket if not near the water.
B
Conan Questions
B.1 Question Generation
Questions in Q Conan are generated based on vandal's task-finishing process. To generate a question,
(1) we initialize a playground and put the vandal in it; (2) the vandal is randomly assigned a task;
(3) the vandal tries to finish the task with the help of the pre-build parser and planner, and generates logs along the way; (4) question is generated based on a certain part of the log. We randomly select a template from the template pool and fill placeholders with related objects in The answer is also parsed from the log. Other choices are sampled based on the question and the context to avoid unrelated choices that can be easily excluded.
B.2 Question Templates
Tab. A3 lists all the templates we use for generating questions.
B.3 Dataset Statistics
See Tab. A4 and Tab. A5 for details.
C
Explorer
The Explorer in the detective is an RI. agent. The agent receives an observation of [64, 64,2] 2] tensor. This tensor combines the 9 x 9 symbolic local view of the detective and 64 x 64 question mask.
A3

---

Table A3: Question templates in Q Conan. is the placeholder.
Type
Templates
What was the vanda''s objective in these area?
What was the vandal's current intent? What did the vandal do before this step? Why did the vordal make this table?
What did the vandal do after this step?
What did the vandal make on this table?
What item did the vandal most likely craft using the table?
Why did the vandal make the U
What action did the I1 perform immediately?
What was the [l used for?
Intent
What did the vandal make on this furnace?
Why did the vandal make this furnace?
What item did the vandal most likely craft using the furnace?
Why was tree cut?
What was the intended use for the wood?
How was the tree cut?
What was the purpose mining U?
Why was the I| mincd?
What was the intended use for []? What did the vandal use defcat the []?
How did the vandal defeat the []?
Why did the vordal defcat the []?
What was the vanda's final goal?
What was this vandal trying to achieve?
Goal
What did the vandal want to achieve?
What the vandal's survival intent for doing []?
why did the vandal collect/make
What the vandal's goal for survival currently?
Did the vandal die? Why?
Survival
Why did the vandal die during the task?
How did the vondal die?
What was the vandal trying to do when died?
What can the vandal do avoid death?
what helped keep the vandal away from hungry?
what food did the vandal cat?
Table A4:
Dataset split and choice distribution.
Category
Train
Test
Val
Choice A
Choice B
Choice C
Choice
Intent
71162
9152
8822
24.99 .9%
25.20%
24.89%
24.93%
Goal
8000
1000
1000
24.89%
25.08%
24.87%
25.16%
Suvival
7365
1560
1596
25.13%
24.95%
24.95%
24.97%
Table A5: Task distribution.
Tuak
defest_cco
make_ www.etteeepppeeeee _pe_ppe
place_bes 8.44
Blace_EErcnae 8.23
Percentage
2.47
3.49
2,52
2.87
get Iava
defeat, skeleton
neke__ron_ swo.d
get cocl 2
get beef 27
get 1r._dla
get. stottt 2,77
2.22
87
2.44
2.39
Yake, busket 5.11
get Exon 2.44
Hoter 2.2
nake, nake_Iron pickaks
nake, bed 221
iake steak 281
mooe_wood .ooord 2.53
2,95
delmat_ zorble 7.8
nucceett ciee swoodd
place table 5.34
get ret_
make wttpe__ood___pcec pictase
2,63
2.67
2,63
The local view is zero-padded to 64 x 64. This ensures the agent knows its relative position on the map. Additionally, the mask generated based on the question, with the area related to the question unmasked. The mask serves as the goal of the exploration policy.
All the RL baselines are trained for 1 steps. See more details below. Unless specified otherwise, parameters are set as default in Stable Baselines.
C.1
Model Details
DQN
The DQN baseline is trained using a value of 0.98, a value of 1, learning rate of 0.0001, buffer size of 10", and a batch size of 512. We leverage an MLP policy with two layers of 64 neurons each. The model is updated every 10 steps.
TRPO The TRPO baseline updates its policy with a special KL-divergence constraint the distance between the new and old policies. We also leverage an MLP policy for TRPO, where the same multi-layer layer perceptron is used for both policy and value prediction.
RecurrentPPO The ReucrrentPPO baseline uses long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) as the recurrent policy. The LSTM layers' weights are initialized with standard Gaussian. We reset I.STM states the end of the episode. The L.STMs for both the actor and the critic have the same architecture, with two LSTM layers of 256 neurons each.
A4

---

C.2 Training Details
Explorers are firstly trained on long-horizon tasks as explained in the main text. These long-horizon tasks include "get diamond," "get lava," "get water," "make iron sword," "make iron pickaxe" and "eat steak." These tasks can be further broken down into over 20 subtasks and have an average episode length of more than 200 steps. We generate 10,000 unique scenes with traces given these tasks and train explorers on them for 10 steps. Then the explores are fine-tuned on all tasks in Q Conan for 10" steps.
We also show the frame rate per second (FPS) for different RL. baselines during training in Fig. A2. As can be seen from the figure, DQN exhibits the highest training efficiency, reaching an FPS exceeding 3000. TRPO maintains a stable FPS of 2000. On the contrary, RecurrentPPO operates significantly slower, requiring over 96 hours to complete training with 128 subproc environments, whereas TRPO accomplishes the task just 14 hours.
D VL Reasoning
5000
In this section, we describe the experimental details for the Vision-Language (VL. models used in the paper.
4000
3000
D.1 Model Details
2000
TAPO:
Vanilla-Trans For Vanilla-Trans, the visual features together with the text features are concatenated in the for- mat of [frame_] 1,, frame 2, frame_n, n, question, choice_l, ce_,,
1000
Recame-thre
RecumentFRO 5002
DON 2000
0
2e7
Step
Be7
1e8
Figure A2: Frame per second (FPS) curves of eral RL explorers in training. Results show that DQN and TRPO are significantly faster than RecurrentPPO.
Visual features, if from the symbolic observation, are directly passed into the model. Otherwise, we utilize CLIP's pre-trained image encoder (ViT-B/16) to extract features from pixel input. Text features are calculated using the text encoder of CLIP. These input features are then passed through 6-layer Transformer model with MLP head for classification.
choice_4].
FroZenBiLM We adopt the cross-modal FrozenBiLM for Q Conan, drawing inspiration from models used in Multiple-choice VideoQA benchmarks such as How2QA (Li et al., 2020) and TVQA (Lei et al., 2018)'. . Q Conan can be formulated the fixed explorer. We concatenate all of the observation frames as the video input. The questions and choices are converted into the following format: [* quest ionnstion] Is it {
multiple-choice VideoQA problem given
(choice_] ce_]] ? "(question) Is it [choice_5 | } We then evaluate the probabilities of the model producing "Yes" and "No". The visual features are processed in the same way as in Vanilla- Trans and then forwarded for visual-text projection. utilize BERT-Large and DeBERTa as our frozen language backbones in this work; however, other general language models are applicable as well.
Flamingo-Mini Our Tlamingo-Ming baseline is based on an open-source implementation of the Flamingo model?, as the original Flamingo model's pre-trained weights are not accessible. Flamingo- Mini is built upon OPT-125M and CLIP pre-trained ViT-L/14 model. We also formulate Q Conan as a multiple-choice problem for Flamingo-Minn Mini. The questions and choices are converted into the follow- ing format: ("Question: (question) Answer: {choice @coocce_]]
"Quest on (qusstion) Answer: (choice_d) __)"]. Each question-choice pair is fed into the model and then binary classifier head is used on Flamingo's 's last layer output to predict the final answer.
Ihttps://gi http://hhh.....mmmaaata///rrrrer IM
A5

---

D.2 Training Details
Vanilla-Tram was trained for 100 epochs, with a batch size of 128. FroZenBiLM models were trained for 50 epochs, with a masking probability (for the MLM objective) of 0.15, a batch size of 32, a learning rate of 10 a gradient clipping max norm of 0.1, and Adam as the optimizer (31 0.9, $2 0.95, 1 x 10 Flamingo-Mini was trained for 100 epochs, with a learning rate of x 10-6, a batch size of 8, and also Adam as the optimizer (31 0.9, 82 0.999, x 10-8).
E
Additional Experiments
E.1 Negative Control Baselines
We compare our VI. reasoning results on the trained explorers with those on empty visual inputs as a negative control baseline. The results are shown in Tab. A6.
Table A6: VL Reasoning models' performance explorers compared with empty visual inputs.
Vanilla- Trans
F-BiLM-BERRT F-BiLM--eBERTa Flamingo-mini
Empty visual inputs
26.4
25.5
25.9
22.9
TRPO explorer
25.0
44.4
43.1
43.3
Ideal explorer
78.4
59.5
71.8
47.8
The results show that using empty visual inputs yields random performance across all settings. Besides, it also shows that the training QA pairs are unbiased. The TRPO explorer achieves higher performance, which suggests that the exploration strategy learned by TRPO helps gather some informative evidence for the reasoning process. The Ideal explorer is an oracle-like exploration policy that has access to perfect trace evidence and temporal information. It provides the most comprchensive information about the environment. This highlights the importance of effective exploration in improving reasoning performance. However, does not mean that reasoning less important, as even with the Ideal explorer, the model still could not achieve satisfactory performance. Based on all results, collecting informative evidence seems to be more important in the overall objective.
F
Abduction from Deduction (AfD)
As mentioned in Sec. 4.4, we adopt data-driven strategy learn model of P(g S) and simulta- neously answer the questions. To be more specific, we train the detective agent self-superviseelly The detective is randomly assigned with one of possible tasks. It then finishes the task by following the action policy (-). Note that we assume the detective's 's x(.) is the same the vandal's is in order to best implement the idea of AfD. Based on the task execution process, questions are generated. Since our ultimate goal is to have our models answer Q Conan's 's questions. we do not explicitly construct P(g S), but rather consider the question-answe process as the g. We then train P(g S), where S is the detective's 's observation during the task execution, and the label can be derived from the assigned tasks together with the (-)).
Besides p(g S), we still need to learn a model of P(S | O), which, intuitively, can be understood as inferring the true state of the environment from partial observation. In our experiment, we tried two ways to model P(S O). One approach is to directly train model using multi-frame observations to predict the states. We employed UNet (Ronneherger et al., 2015) and multi-layer CNN as the network. However, this method did not work effectively. Reasoning based on the reconstructed states only achieved performance at random level. The second approach, which was finally used to report performance, aligned the hidden feature spaces from true states and observations. When training P(g S), we added a head before the VL models, converting the input into a 4096-dimensional vector. Then we trained head on O with the same structure, minimizing the difference between features from O and features from S.
G Q Conan Task Demo
A6

---

To better illustrate the core components in Q Conan, We take the playground shown in Fig. 1 as an example. In this scenario, the assigned task is "get diamond" (Fig. A3 shows the task dependency). As shown in Fig. A4, once the vandal completes the task, leaves behind traces in the playground. The vandal ends at the bottom of the figure. The detective then enters the playground, starting at the beginning of the traces. In this case, traces encompass footprints and remnants left after certain actions. Note that footprints cannot be left on sand or stone, and different footprints may overlap. The vandal will collect objects crafted on a table, making them invisible.
Let's suppose the detective's exploration begins by following footprints (note the context window size is 9))).
some trees. As the footprints are seriously overlapped and mostly one-directional, we can deduce the vandal did not return. After seeing the moll-making table, with the only resources being wood, we could say that the vandal could only make wooden tools, not stone swords or iron pickaxes, further restricting possible actions the vandal took.
Firstly
Note that this is already critical reasoning in Q Conan.
Figure A3: The task
Moving on, we note that footprints become missing on the sand surface. However, we note broken stones and coals. Therefore, the wooden tool break stones and coals shall be wooden pickaxe, So the agent should have made wooden pickaxe on the table earlier. Despite the fact that the tool has been collected, we could still figure that out.
struc- ture of "get diamond".
Following the reemerged footprints, we note blood and a zombie body on the ground, suggesting the vandal should have had a fight.
Searching on, we find the broken diamond. As an iron pickaxe is the only tool to collect diamonds. The vandal must have built an iron pickaxe with iron and coal in the furnace. With no other footprints around, we can safely conclude our search.
vanda's ESOSS
Figure A4: A demostration of core components in Q Conan. We show how detective can do reasoning based on the task structure and traces left in the playground. Zoom for more details.
A7


Reviews:
[{"review_id": "CLml4TGeG4", "reviewer": "Reviewer_u9zC", "summary": "This paper introduces Conan, an interactive environment as a benchmark to evaluate agent’s active abductive reasoning abilities to answer questions in an incomplete (or partial) information scenario. Because of partial information, the model requires further exploration in the scene to answer the questions which is posed as a detective game. The paper also proposes the Abduction from Deduction (AfD) approach which relies on Bayesian statistics. The experimental evaluation showcases the strengths and weaknesses of various machine learning models in different settings.", "questions": "- Can you elaborate on how this research can be improved to integrate exploration and reasoning processes in future work?\n- What was the main motivation behind maintaining the detective set invincible, ie. not maintaining the survival status on line 178? \n- On line 363, the authors mention that the proposed approach is akin to “human-like reasoning”. It is not clear as to how solving this toy task with an independent explorer and reasoning model brings us closer to “human intelligence”\n\nSuggestions: \n- It would help to make the best results in bold in Table 3. \n\n\n\n\n\n", "limitations": "See weaknesses for more information about the limitations\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-motivated and clearly written in most parts. \n- The authors promise the availability of corresponding code and a detailed description of hyperparameters which would be essential for reproducibility if open-sourced. \n- The paper introduces the AfD approach using Bayesian rules which is interesting to the research community. \n- Relevant baselines have been selected to comprehensively evaluate various RL and multi-modal models. \n", "weaknesses": "- The proposed environment looks like a toy benchmark with limited real-life applications.\n- Dataset statistics is missing in the work which might expose data biases related to goal, intents or survival questions.  \n- The question templates are also limited and it is not clear if the model learns to take advantage of any data biases to answer the questions. It is also not clear if the questions are asked in a particular order. \n- This is framed as a multi-choice QA task with limited generalizability. \n- The proposed approach lacks a close integration between exploration and reasoning processes where the training and evaluation for different models is independent. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you very much for your thoughtful review! We would like to discuss your concerns here:\n> \"look like a toy benchmark with limited real-life applications\"\n\nIt is true that Conan is synthetic and may appear simplistic at first glance. However, the primary goal of Conan is not to present real-world complexity but rather to introduce the active reasoning setting to the community. Direct applications may not be straightforward; however, we believe the autonomocy in a successful Conan solver would be utterly desirable in our never-ending pursuit of AI. For example, we have many real world scenearios where we need informed search to draw conclusion, say new material discovery (like LK-99), causual discovery, or new drug synthesis. Though Conan is more about common sense, the general problem should be very similar. Besides, \"toy\" as it may seem, Conan, as Reviewer yGBc puts it, is already quite significant and non-trivial for AI systems to master. From our perspective, the environment already covers all the critical aspects in active reasoning. The toy-ish nature simplifies researchers' efforts to control and explore various aspects of the problem when attempting to address the difficulties without concerning about the real-world complexities, as the research history on CLEVR[1], CLEVRER[2] and EQA[3] shows (despite them being simpler than Conan). Successfully solving these tasks should be the grounds for developing more general and adaptable AI models in the future when we envision autonomous machine and human-machine collaboration. We believe that treating Conan as \"toy\" might overlook the true challenges and benefits it offers.\n\n> \"Dataset statistics is missing\"\n\nThanks for the reminder. Please see dataset statistics below: \n|Category|Train|Test|Val|A|B|C|D|\n| -|-|-|-|-|-|-|-|\n|Intent|71162|9152|8822|24.99%|25.20%|24.89%|24.93%|\n|Goal|8000|1000|1000|24.89%|25.08%|24.87%|25.16% |\n|Suvival|7365|1560|1596|25.13%|24.95%|24.95%|24.97% |\n\n|Tasks|get_drink|defeat_cow|get_apple|make_stone_pickaxe|place_bed| place_furnace|defeat_zombie|make_stone_sword |\n|-|-|-|-|-|-|-|-|-|\n|Percentage|2.47|8.49|2.52|2.87|8.44 | 8.23 |7.8|2.65|\n|**get_lava**|**defeat_skeleton**|**make_iron_sword**|**get_coal**|**get_beef**|**get_diamond**|**get_stone**|**place_table**|**get_wood**|\n|2.72 |8.7|2.64|2.42|2.7|2.39|2.67|8.24|2.67|\n|**make_bucket**|**get_iron**|**get_water**|**make_iron_pickaxe**|**make_bed**|**make_steak**|**make_wood_sword**|**make_wood_pickaxe**||\n|3.11|2.44|2.2|2.95|2.71|2.81|2.53|2.63||\n\nWe will release more detailed dataset statistics together with the benchmark.\n\n> \"The question templates are also limited\"  \n> \"if take advantage of any data biases / questions are asked in a particular order\"\n\nAlmost all VQA tasks do suffer from this issue as question templates are always limited. Our approach involved carefully crafting and curating a diverse set of question templates that cover various aspects of the task. Questions are not asked in a particular order. The choices are also randomly sampled from a pool and shuffled. Experiments have shown that without any visual input, the models only achieved random-level performance (see response to Reviewer nDEB), showing that templates do not introduce biases.\n\n> \"framed as a multi-choice QA task with limited generalizability.\"\n\nConan can be designed as an open QA task. However, we design Conan as a multi-choice QA task for easier and clearer evaluation.\n\n> \"the main motivation behind maintaining the detective set invincible\"\n\nThe decision was driven by the design goal of focusing on active exploration and reasoning rather than adding to the challenge by survival. In this way, we can ensure that the main emphasis remains on solving complex reasoning tasks and answering questions related to the visual scenes provided. Besides, the task will be too hard if the detective may be dead (but it can be if you change the hyperparam).\n\n> \"integrate exploration and reasoning processes\"  \n> \"how an independent explorer and reasoning model brings us closer to “human intelligence”\n\nWe acknowledge this limitation and appreciate your interest for improvement. Ideally, we should jointly optimize both exploration and reasoning components. As you said, it's more human-like and intelligent, for instance, the reasoner can provide feedback to the explorer, guiding it towards more informative exploratory actions that are likely to yield relevant traces. \nHowever, there are significant obstacles in practice (we tried but failed). Attributing credit to exploratory decisions that have long-term consequences can be complex. When exploration actions yield results much later, it becomes difficult for the model to understand the causal relationship between the exploratory decisions and their eventual impact on reasoning and answering questions. This highlights the interdependence between the exploration and reasoning processes. Improving one aspect requires advancements in the other, creating mutual dependency that complicates the optimization process. The reasoning component itself requires significant training and computational resources, especially when based on large language models. The need for substantial computational power makes the joint optimization of exploration and reasoning even more challenging. \nFrom a community perspective, it's also a common practice [3-5]. So we adopt this route. We believe future work on Conan shall prioritize reasoning over exploration: not to say that exploration is less important, but guided by a reasoning engine. Similar to some forms of (soft) tree search, we hope that a reasoner would propose possible subgoal and the explorer would explore the environment conditioned on the subgoal, and the entire process may be integrated in a MCTS-based learning framework like AlphaGo. \n\n[1] Clevr\n[2] Clevrer\n[3] Embodied question answering  \n[4] Embodied question answering in photorealistic environments with point cloud perception  \n[5] Iqa\n"}, {"review_id": "r9wgoTIlzz", "reviewer": "Reviewer_i6nS", "summary": "To address the gap in handling incomplete-information questions, this paper introduces an interactive open-world environment called \"Conan.\" The purpose of Conan is to motivate and evaluate agents' active reasoning ability by requiring them to explore, gather evidence, and combine knowledge to solve complex scenarios. The ultimate goal is to improve AI agents' ability to interact with and learn from the world.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper introduces a new interactive open-world environment called \"Conan.\" The purpose of Conan is to motivate and evaluate agents' active reasoning ability by requiring them to explore, gather evidence, and combine knowledge to solve complex scenarios. The ultimate goal is to improve AI agents' ability to interact with and learn from the world.", "weaknesses": "The author claims that the detective spawned in the environment needs to answer all questions through exploration, connecting key frames, and reaching conclusions. Yet, some questions presented in the Figures/Tables can be answered without exploring the environment. The process of answering questions through exploration requires more clarity and elaboration.\n\nRegarding the reward function used in the task, the author mentions that it incentivizes the agent to search for clues and traces related to the given question. However, the paper lacks detailed information about how the reward is designed and the specific motivation behind it.\n\nThe writing appears to be cumbersome and illogical, making it difficult to grasp the actual meaning of the task. The purpose of introducing this new task is not clearly stated, leaving readers with questions about its significance and potential applications.\n\nIn conclusion, while the paper introduces an intriguing concept of an interactive open-world environment for AI agents, there are aspects that require further clarification and refinement. Addressing the limitations and providing more detailed explanations about the task and reward design would enhance the overall understanding and impact of this research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you very much for your review and positive rating!\n> \"Some questions presented in the Figures/Tables can be answered without exploring the environment. \"\n\nNo. All of the questions have multiple possible answers and need to be inferred from traces in the environment. From the detective's view, he cannot directly \"see\" what happened in the environment. For example, when asked \"Why did the vandal cut a tree? \", the detective needs to find traces before and after (especially after) to see whether the vandal needs some woods to make tools or just want to collect apples to eat. If the reviewer believes some questions could be answered without exploration, please point it out in the discussion phase and we are happy to help further address the concern.\n\n> \"However, the paper lacks detailed information about how the reward is designed and the specific motivation behind it.\"\n\nWe apologize for the ambiguity. As mentioned in Sec 4.1,  the agent will receives a positive reward of 1 when a trace first appears within its local view or 2 when the trace is closely associated with the question. The agent receives a reward of 100 if it successfully finds all traces left by the vandal. Additionally, the agent incurs a penalty of -0.1 for every timestep it takes. By rewarding the agent for successfully identifying and following the traces, we foster behavior that mimics real-world detective work, where finding clues is a fundamental aspect of solving mysteries.\n\n> \"The writing appears to be cumbersome and illogical, making it difficult to grasp the actual meaning of the task. The purpose of introducing this new task is not clearly stated, leaving readers with questions about its significance and potential applications.\"\n\nThank you for your feedback on the writing. We apologize for any confusion. We will make the necessary improvements to address these concerns and provide a more concise and logical presentation. In a nutshell, the task is designed for capturing humans' active and exploratory nature in abductive reasoning in the real world and providing agents with an open-world environment which encourages active exploration and multi-round abductive inference based on in-situ gathered evidence and existing knowledge. The task goes beyond existing single-round passive tasks and contributes to the broader pursuit of building more intelligent and human-like AI systems.\n\n> \"providing more detailed explanations about the task and reward design\"\n\nThanks for your kind suggestion. We kindly refer you to the attached file for a simple demonstration of traces and reasoning steps. We are also considering adding more comprehensive examples in the supp. Thanks again for your suggestion!"}, {"review_id": "gHLvynwIKM", "reviewer": "Reviewer_nDEB", "summary": "This paper proposes a benchmark for \"active reasoning\" titled Conan, where, instead of passively answering questions from provided context, an agent must interactively explore its environment to discover information. The authors differentiate such a task from so-called \"passive reasoning\" tasks such as video-language understanding where the visual input is directly fed into the model. Here, a model itself is responsible for exploring and attaining input that will help a model answer a question.\n\nConan is implemented in a Minecraft-like gridworld where a \"detective\" agent must identify activities that a rule-based, goal-oriented \"vandal\" agent completed, by using traces of the vandals' behavior in the environment. E.g. if a vandal's goal was to make a wooden pickaxe, it may have cut some trees down, and the task for the model will be to answer why the vandal decided to make a wooden pickaxe.\n\nTwo alternative methods are adopted to approach this task: one which uses an explorer agent, trained with an exploration reward, to generate evidence that is fed into a standard VLM, and another, \"Abduction from Deduction\", where one directly learns to predict the goal of the vandal from an inferred state trajectory.\n\nOverall this is an interesting paper, and I think the dataset and benchmark will be valuable for the community. The models tested for this environment are relatively simple, not really making full use of the \"joint exploration and reasoning\" abilities that Conan purportedly tests, however. Moreover I do have some outstanding questions, and some issues with the experiments (specifically a missing baseline) that prevent me from assigning a higher score. However I'm open to changing my score after the author response and discussion period.", "questions": "- The idea of answering questions via \"traces\" left behind in the environment is a little hard to grasp and could use more explaining. There are some example traces in Figure 1b but these are small figures and aren't explained that much. The authors could spend some more time walking through example \"traces\" and more fully convince readers that the traces indeed provide answers to some of the more subtle questions in the benchmark (e.g. how do traces indicate that the vandals goal is to \"get the diamond\", couldn't the vandal have alternative goals like get iron/coal?).\n\t- It might be valuable to have an \"oracle\" setting to demonstrate that with perfect trace evidence and enough training models can 100% this task—or else explain why such an oracle doesn't exist (e.g. what causes the lagging performance even for the \"Ideal Explorer\" setting in Table 3? Is it an amount of training data issue? Are there still limitations as to the quality of the trajectory attained even by the ideal explorer, and is there an even more suitable oracle for this?)\n\t - Related to this, can authors further explain L308 ideal explorer: \"visible to the ground-truth vandal’s trajectory\"? What does \"visible to\" mean—shouldn't the ideal explorer literally *be* the ground-truth vandal's trajectory?\n- More details could be provided on the specific kinds of generalization splits tested here. At test time, is there a subset of abductive reasoning tasks held out that the model has never seen? Or does the model see the same questions it has seen during training, just in unseen environments? Do the test environments differ systematically from those during train (or is it possible to induce such a )\n\t- Evaluating (and supporting) such compositional/systematic generalization splits would greatly increase the appeal of the benchmark.\n - (Related to the above) Do authors have an intuition as to whether test-time performance drop is more due to the VLM's inability to generalize, or the explorer's inability to generalize and generate good evidence for unseen enviroments (if test does indeed have unseen environments)? Is it possible to disentangle the two using different evaluation splits in the environment?\n - In Figure 4, does a score of near 100 as attained by eg TRPO indicate that the explorer indeeds find all traces (+100) reward all of the time? If not, is there an explicit measure of that?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- To my knowledge, the \"active reasoning\" component of Conan is an important area in vision/lang/RL, and is certainly underexplored in current embodied QA settings (although it is probably implicitly present in instruction following benchmarks and work like SayCan).\n- The dataset seems high quality and should be a useful contribution for the field.\n- Fairly sound experimental evaluation, comparing a variety of RL explorer agents with a variety of vision-language models.", "weaknesses": "- **Missing negative control baselines.** The experimental results should compare evidence gathered by the trained explorer agents to a weaker negative baseline, e.g. a random untrained explorer policy or no explorer input at all. This is needed to convincingly show that the explorer agents are actually learning to explore in a way that is more beneficial than chance, at least for the downstream QA task.\n    - In general I'm definitely more interested in to what extent performance is bottlenecked by good exploration on this task, rather than fixing the explorer evidence and trying subtly-differently-trained large VLMs.\n- As mentioned by the in the conclusion (L376-377), one of the key promises of Conan is that strongest performance on this benchmark should intuitively be achieved by models that jointly learn to reason and explore, but right now a decoupled two-stage process is adopted—first training an exploration model with an oracle \"find trace\" reward, extracting relevant keyframes, then training a VLM on top of the frozen policy. As a result, models trained for this task (for now) look like typical embodied QA models doing passive reasoning (just with the extra step of training an explorer to get the vision input). Of course, it's not strictly necessary for a benchmark paper to include an strong novel model for the benchmark, but without it we aren't able to evaluate whether Conan performance numbers have **headroom** for more sophisticated approaches to joint exploration and reasoning. (It seems like this should be the case, but we can't really be sure).\n- Some more outstanding questions (see next section)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,  \nThank you very much for your thoughtful and detailed review. \n> \"Missing negative control baselines.\"  \n\nThank you for your insightful feedback. We have tried our model with empty visual inputs as a negative control baseline as your suggestion and we show the results on \"goal\" split here:\n|| Vanilla-Trans | F-BiLM-BERT | F-BiLM-DeBERTa | Flamingo-mini |\n|-| -|-| -|- |\n| Empty visual inputs | 26.4| 25.5| 25.9| 22.9|\n| Trpo explorer | 25.0 | 44.4| 43.1 | 43.3 |\n| Ideal explorer| 78.4| 59.5| 71.8| 47.8|\n\nThe results show that using empty visual inputs yields random performance across all settings. Besides, it also shows that the training QA pairs are unbiased. The TRPO explorer achieves higher performance, which suggests that the exploration strategy learned by TRPO helps gather some informative evidence for the reasoning process. The ideal explorer is an oracle-like exploration policy that has access to perfect trace evidence and temporal information. It provides the most comprehensive information about the environment. This highlights the importance of effective exploration in improving reasoning performance.  \nHowever, it does not mean that reasoning is less important, as even with the ideal explorer, the model still could not achieve satisfactory performance. Based on all results, collecting informative evidence seems to be more important in the overall objective. \n\n> \"a decoupled two-stage process is adopted\"  \n> \"more sophisticated approaches to joint exploration and reasoning\"\n\nDue to the word limit, we kindly refer you to our response to Reviewer u9zC for a detailed discussion.\n\n> \"answering questions via \"traces\" left is a little hard to grasp\"\n> \"some example traces in Figure 1b aren't explained that much\"\n\nThanks for your kind suggestion. We kindly refer you to the attached file for a simple demonstration of traces and reasoning steps. We are also considering adding more comprehensive examples in the supp. Thanks again for your suggestion!\n\n> \"traces indicate that the vandals goal is to \"get the diamond\" or other alternative goals\"\n\nFor instance, if traces indicate that the vandal made a pickaxe, its intent could be to get iron or diamond according to the task dependency graph. As a result, the detective should continue to find more distinctive traces that can help exclude the wrong possibility, say checking the mine area.\n\n> “an \"oracle\" setting to demonstrate that with perfect trace evidence and enough training models can 100% this task”  \n> \"limitations of the ideal explorer traces\"  \n> \"L308 visible to the ground-truth vandal’s trajectory”  \n\nThe ideal explorer provides the vandal's trajectory, while it cannot directly observe the actual vandal states when an action is taken, but only the traces the action left. This can introduce confounding factors, leading to potential confusion, necessitating the reasoning component in connecting the traces and figuring out the states. That's why we design Conan: it is important not only to explore the traces, but also to reason the hidden states. Agents must engage in abductive reasoning based on the traces they observe to reconstruct the entire story, which is indeed a challenging task. It could be a data problem if one takes on a data-driven perspective. The ideal explorer is good enough, so in this case we tend to believe it is the reasoner to blame.  \n\n> \"Is there a subset of abductive reasoning tasks held out that the model has never seen? Or does the model see the same questions it has seen during training, just in unseen environments? Do the test environments differ systematically from those during train?\"\n> \"Evaluating (and supporting) such compositional/systematic generalization splits\"\n\nThe test environments are disjoint from those during training, though they are sampled from the same distribution. But test questions are from the same distribution as training and could be similar. As the results show, even on IID questions, models still fare worse than expected. Therefore, as the first attemtp in active reasoning, we primarily focus on the current setting, and would like to see joint community efforts when it is time to introduce other generalization in the problem.\n\n> \"Performance drop due to the VLM's inability to generalize, or the explorer's inability to generalize and generate good evidence for unseen environments? Possible to disentangle them?\"\n\nWe tend to believe the answer is complicated, but the intuition is that incomplete information plays the more critical role. Evidence above and in the paper suggests that compared to the reasoning component, the exploration component may be more important in order to achieve \"OK\" performance. But to reach perfection, the juice in the long tail can only be squeezed out by a good reasoner. Our current setting (test environments are unseen during training) shows that with the ideal explorer, performance could be greatly improved, meaning that the VLM could generalize but the explorer couldn't.\n\n> \"Does a score of near 100 indicate that the explorer indeeds find all traces? Explicit measure?\"\n\nNot exactly. As mentioned in section 4.1, the agent receives a step reward when it discovers more traces, which is intentionally designed to avoid sparse rewards. Additionally, if the agent successfully finds all traces left by the vandal, it will receive another 100 reward. Since there can be different environments and the vandal can leave long traces (note that there are steps with a reward of 2), the total step reward may vary slightly. We check statistics to explain this issue. For the TRPO explorer, it manages to find all traces in only about 1% of the environments. The ideal explorer achieves an average reward of 123.12 in 10000 envs. Treating it as the lower bound, the TRPO explorer is still imperfect. Careful analysis shows that TRPO only manages to find all traces in 1% of the environments.\n\n[1] Kurt Spencer. Noise!, 2014. URL https://github.com/KdotJPG/OpenSimplex2\n"}, {"review_id": "KsXrNzAdZX", "reviewer": "Reviewer_2rwT", "summary": "The paper introduces Conan, an interactive open-world environment for evaluating the active reasoning abilities of agents. In Conan, agents need to answer questions by actively seeking for evidence and acquiring new knowledge in a setting of incomplete information.\nConan is formulated as a detective game. First, during the initialization of the game, a vandal agent completes a task, leaving traces in the environment. Next, questions are generated for a detective agent to answer. The detective has to answer these questions by actively interacting with the environment to reconstruct the actions of the vandal.", "questions": "Is a human study or an error analysis feasible?", "limitations": "The authors described the limitations of the paper", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The environment differs from existing benchmarks for its active reasoning and interactive multi-round setting\n* The paper is very well written and easy to follow\n* Conan is an interesting environment that could spur further research on active reasoning in interactive contexts", "weaknesses": "* It is not clear how challenging the task is and how good the performance of the models reported in the paper is. A human study would help the reader understand this critical point.\n* A qualitative analysis of the games and the behavior of the different agents would also be needed in the main paper to understand when the agents make mistakes and to what extent the tasks are solvable\n* As mentioned in the paper, it looks like the explorer does not supply particularly relevant information to the reasoning model, substantiating the intuition that the bottleneck is there, rather than in the reasoning abilities of the model", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer:\n\nThank you very much for your thoughtful and detailed review. We are pleased that you found this work interesting and unique from existing benchmarks. \n\n> \"It is not clear how challenging the task is and how good the performance of the models reported in the paper is. A human study would help the reader understand this critical point.\"  \n> \"Is a human study or an error analysis feasible?\"\n\nWe have now uploaded a PDF file detailing the problem solving process in case you miss the challenges presented in the work. We kindly refer you to it for detailed explanation. Indeed, conducting a human study would be valuable. However, playing this particular game requires strong prior knowledge and familiarity with the game mechanics. Training is not easy, and it's hard to distinguish what leads to humans' failure in this task. As a result, we are still conducting the human study and would update the results in the discussion phase when we have preliminary results.  \nBelow we present the error analysis for the \"goal\" split. We examine the accuracy of Frozenbilm-deberta on various tasks, comparing two groups: reasoning based on the ideal explorer and the TRPO explorer.  \nAs mentioned in the paper, the ideal explorer is relatively oracle-like, as it not only gathers all the traces left by the vandal but also captures temporal information. Results show that in most cases the model answers more questions correctly on Ideal explorer than on TRPO explorer. Reasoning on ideal explorer fails when the questions cannot be directly \"seen\" from the environment and not found afterwards. For example, detectives can see a table but can not see what was made. On the other hand, reasoning on TRPO explorer struggles with long-term traces, indicating the fact that it can not cover all traces during exploration. Results from the reward also validate this (see response to Reviewer nDEB). We are going to add more error analysis into the supp. Thanks again for your suggestion!\n\n| **Tasks**   | get_drink       | defeat_cow  | get_apple         | make_stone_pickaxe | place_bed   | place_furnace   | defeat_zombie     | make_stone_sword |\n| ----------- | --------------- | ---------------- | ----------------- | ------------------ | ----------- | --------------- | ----------------- | ---------------- |\n| **Trpo**    | 47.06           | 43.90            | 35.7              | 48.48              | 43.90       | 44.44           | 52.50             | 37.50            |\n| **Ideal**   | 100.00          | 85.37            | 78.57             | 39.39              | 87.80       | 83.95           | 75.00             | 4.17             |\n| **get_lava**    | **defeat_skeleton** | **make_iron_sword**  | **get_coal**          | **get_beef**           | **get_diamond** | **get_stone**       | **place_table**       | **get_wood**         |\n| 50.00       | 46.59           | 56.25            | 50.00             | 53.85              | 40.62       | 36.84           | 39.36             | 36.00            |\n| 66.67       | 82.95           | 28.12            | 45.45             | 42.31              | 84.38       | 47.37           | 91.49             | 96.00            |\n| **make_bucket** | **get_iron**        | **get_water**        | **make_iron_pickaxe** | **make_bed**           | **make_steak**  | **make_wood_sword** | **make_wood_pickaxe** |                  |\n| 35.29       | 28.57           | 45.95            | 56.52             | 47.83              | 46.15       | 50.00           | 40.00             |                  |\n| 73.53       | 46.43           | 54.05            | 52.17             | 39.13              | 50.00       | 64.29           | 55.00             |                  |\n\n> \"A qualitative analysis of the games and the behavior of the different agents would also be needed in the main paper to understand when the agents make mistakes and to what extent the tasks are solvable.  \n> \"As mentioned in the paper, it looks like the explorer does not supply particularly relevant information to the reasoning model, substantiating the intuition that the bottleneck is there, rather than in the reasoning abilities of the model.\"\n\nThanks for your suggestion. We do consider adding more analysis and discussion about different agents and their results. We have included some qualitative analysis in Sec 5.2. Besides, some extra experiments has been conducted, see response to Reviewer nDEB for more information.  \nTo make it short, in the case of empty visual inputs, the reasoning performance is quite low, indicating that the agent is severely limited by the lack of visual information. The TRPO explorer shows a noticeable improvement compared to empty visual inputs, indicating that some exploration is better than none. However, the performance is still relatively low. The ideal explorer achieves significantly better performance, indicating that its ability to gather perfect trace evidence greatly benefits the downstream reasoning task. This highlights the importance of effective exploration.  \nHowever, this is not to say the reasoning part can be considered solved. As can be seen in Table 3, even with an ideal explorer, the reasoner still could not well answer the questions. Also see response to Reviewer nDEB for a detailed discussion.\n"}, {"review_id": "HjOMtERhD7", "reviewer": "Reviewer_yGBc", "summary": "This paper introduces Conan, a new benchmark for evaluating active reasoning in an embodied/open-world environment where an agent acts as a detective and must answer questions about the actions and intentions with the traces of another agent (a vandal). The experiments benchmark a RL + vision-and-language baseline and a more structured model that uses Bayesian inference on the Conan task.", "questions": "- table 1: what's the difference between embodied and open-world?\n- l146: what does a trace look like? are objects left behind, does the state change, etc.?. l151: when you say \"32 distinctive traces\" do you actually mean \"32 object interactions\" and traces can consists of multiple object interactions?\n- l153: \"60 abductive reasoning tasks\" - does this mean 60 question templates? \n- l164-168: what's an example of a path + alternative and why exactly does this require the detective to gather additional information? \n- why is the detective allowed to take the same actions as the vandal instead of just exploring the scene and collecting information? (feels like potentially unnecessary complexity)\n- l226-227: \"masks are first employed to encode questions...\" - what does this sentence mean?\n- How does the explorer condition on the question (e.g. some sentence embedding of the question or something else?)\n- Is there a step limit on the explorer's episodes?\n- l174-177: this made it sound like the detective and the agent were separate. My understanding is that the agent *is* the detective?\n- l255: what is the representation of symbolic map?\n- l307: to clarify, does the ideal explorer get the frames of the ground truth vandal trajectory, and then we extract every 30 frames? how much of the gap in performance is due to the keyframe extractor dropping important frames vs. the inaccuracy of the reasoning model?\n- sec 4.4: does this assume access to the vandal policy pi? Is that realistic?\n- l345: why would vanilla-trans be more susceptible to insufficient evidence? Isn't a more likely explanation that it's not pretrained to do question answering, in contrast to the other models?\n- AfD doesn't improve on standard (small improvements at best / mixed results across the board) - doesn't this suggest that this kind of reasoning is not super important in the benchmark?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- ambitious benchmark trying to tackle an exciting and interesting problem that highlights a limitation in the current focus of the community: lack of benchmarks testing reasoning/question answering with active information gathering\n- problem and setting are well-motivated\n- interesting setup for the active reasoning problem with a vandal/detective", "weaknesses": "- many important details that are not clear. One of the main things that needs more clarity is what the format of the benchmark is - it would be very helpful to have more examples and earlier in the paper of what the traces look like / what context is available to models, or maybe an example of a series of reasoning steps a model (or human) would go through to answer a question in Conan. See questions below for specific points of confusion\n- I wasn't totally convinced/clear that the questions in Conan are focused on testing abductive reasoning. Although eventually we would want to solve this with end-to-end methods like the RL baselines presented here, I feel like some heuristic exploration baselines would be important to understand what it takes to solve the benchmark and contextualize the scores of the RL agents (what scores do these agents get?):\n\t- the ideal explorer (specifically wondering: is the ideal explorer the ideal policy?)\n\t- an agent that explores the whole 64x64 grid (how much better does the ideal explorer do compared to this?)\n\t- using an oracle extractor that extracts the trace-relevant frames instead of the every-k key-frame extractor\n\t- instead of using RecurrentPPO, some heuristic extractor that allows agents to condition on key frames in the past (e.g. the frames with traces that are closely associated with the question)\n- Related to previous point, the benchmark also conflates *memory* and *exploration* with the abductive reasoning challenges -- both of these are already huge challenges for RL/current models. I worry that these may make the benchmark too difficult, and additionally not actually test for the reasoning abilities that it claims to test. Overall, I just don't have a good understanding of what solving the benchmark actually entails - if we have agents with good memory + exploration, is the reasoning involved actually quite simple? I think heuristic baselines/agents with privileged information would help the case here, as well as simplifying aspects of the task that are orthogonal to abductive reasoning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer:\n\n> \"have more examples about traces and reasoning steps\"\n\nSee pdf attached.\n\n> \"Are the questions in Conan focused on testing abductive reasoning? Some heuristic exploration baselines\"\n\nGood exploration, ie, the heuristic ideal explorer recovers the vandal's trajectory, is enough for \"OK\" performance, but to sqeeuze out the juice, one needs much better reasoner that can figure out the hidden states from of the vandal, from observed traces (see response to nDEB). Eg, cutting trees could both mean the vandal is in need of wood or food (apples) and from only traces of cut trees, it is not easy to figure it out. For \"trace-relevant\" frames or \"key frames\", the ideal explorer could already be understood as providing all trace-relevant frames. The notion of key frames is ill-defined. As in the video understanding community, one major challenge is how to extract \"key frames\". This is a post-hoc concept that can not be easily obtained beforehand. The common wisdom, and to improve efficiency (reduce context length in Transformer) is to trunck it via every k frame. For the exhaustive search agent, we show the results with an agent with full 64x64 grid observation here: 25.6(Vanilla-Trans), 64.6(F-BiLM-BERT), 69.1(F-BiLM-DeBERTa), 67.9(Flamingo-mini). The noisy sequence, with unoptimal temporal order (unlike the ideal explorere), makes the results slightly worse.\n\n> \"conflates memory and exploration with abductive reasoning\"\n\nMemory and exploration are other terms to describe the challenge. However, they are not all. Indeed, the agent needs to have a good memory, propose good possibilities and explore to check the possibility's validity. However, memory and exploration can only provide observation rather than states, as the difference between observation and state in POMDP, which remains challenge still. The agent needs to have reasoning-guided exploration, rather than exhaustive exploration, to quickly find frames that most likely lead to guess, and reason about the underlying hidden states based on the evidence gathered, as explained in the cut-tree example above and the examples in the PDF. Besides, experimentally, we note that with the ideal explorer, the reasoning is still not perfect.\n\n> \"embodied vs open-world?\"\n\nThe difference is subtle. However, in our work, we'd like to emphasize the diverse and rich environment the agent could interact with, while simplifying actuator control, thus the wording.\n\n> \"l146: what does a trace look like? l151: \"32 distinctive traces\"\n\nSee pdf attached and supp. \"32 distinctive traces\" means that there are a total of 32 distinctive kinds of traces that can be left in the environment (see Figure A1).\n\n> \"60 abductive reasoning tasks\"\n\nNo. Abductive reasoning tasks refer to tasks that Conan environment can support.\n\n> \"l164-168: example of a path + alternative and why...?\"\n\nAlso see the attached pdf.\n\n> \"why detective same actions\"\n\nIn the ideal case, the detective only needs to navigate. However, in some cases, the detective needs additional actions, such as meeting blocked roads with stones and needing to perform actions to make tools to break stones. Surely masking other actions will make it easier.\n\n> \"l226-227: masks are first employed to encode questions...How does the explorer condition on the question?\"\n\nMask is used to guide the model's attention to the relevant parts of the playground observation. In questions like \"Why does the vandal cut the tree\", we need a pointer to what \"tree\" actually refers to. So we use a mask the same size of the map to highlight which tree the question refers to, and extract both the linguistic features and the visual features from the masked map as question encoding.\n\n> \"step limit on the explorer?\"\n\nYes. 500. Discussed in line 326-327.\n\n> \"l174-177: My understanding is that the agent is the detective?\"\n\nYes.\n\n> \"l255: the representation of symbolic map?\"\n\nIt is a 2D array with dimensions [64, 64], where each number in the array represents the type of object in the position.\n\n> \"l307: the ideal explorer get the frames of the ground truth vandal trajectory, and extract every 30 frames? Performance gap due to the keyframe extractor dropping vs. the inaccuracy of the reasoning model?\"\n\nIt's that the final downsampled length is 30 frames. That corresponds to every 6-9 frame extraction. The context window size is 9x9, which means that no information is lost in this process.\n\n> \"sec 4.4: access to the vandal policy pi?\"\n\nWe did assume the detective shares the same policy with the vandal. It's not so realistic; however, it is also reasonable to assume to some extent that we share the same world model: we all know how to make swords or break the stones. But we admit that it's better to train the detective agent, say in traditional RL methods in the forward way to distill the world knowledge. The work explores this direction with a simple assumption and certainly has room for improvement.\n\n> \"l345: vanilla-trans susceptible to insufficient evidence ... it's not pretrained to do question answering?\"\n\nA very reasonable explanation. Its failure can be attributed to its weakness in reasoning based on incomplete observations. Will pretraining on other vqa tasks help solve Conan? We are willing to explore this possibility in the future work.\n\n> \"AfD doesn't improve on standard\"\n\nWhile the final performance is only silightly better. But compared with other methods, AfD is not directly supervised with relationship from traces and questions and answers like others. Instead, AfD is trained with P(S|O) model (state prediction from observation) and a policy model as in vandal. The final results are obtained from an explicit inference process. Considering this fact, AfD and other directly supervised models reaching similar performance and even slightly better should be considered significant results.\n\n[1] Sherlock  \n[2] The scientist as child  \n[3] Detecting blickets  \n[4] Causal learning mechanisms in very young children  "}]

Author rebuttal:
To all reviewers:  
We are sincerely appreciative and grateful for the time each of you have spent reading our work and giving useful, thoughtful and constructive feedback. The feedback is substantial and quite helpful for improving our paper. In particular, we would like to thank reviewers for acknowledging our work to be well-motivated (Reviewer u9zC, i6nS, nDEB, yGBc), high quality and should be a useful contribution (Reviewer nDEB), an interesting environment that could spur further research on active reasoning in interactive contexts (Reviewer 2rwT) and an ambitious benchmark trying to tackle an exciting and interesting problem (Reviewer yGBc). We are very glad to see that there is a general interest in such a topic and do hope that this work serves as a valuable contribution to the community.

Meta-review: This work received primarily positive reviews from the reviewers. Several questions were raised during the review period, including about baselines, evaluation setup, and claims made in the paper. The authors provided a rebuttal that addressed many of these concerns effectively. The rebuttal contained several key pieces of information, including additional experiments, which provide additional quantitative support for the benchmark and a better understanding of what capabilities might be required to solve it. The authors are strongly encouraged to incorporate the information from the rebuttal into the final version for completeness.
Decision: Accept (poster)