SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Manjie  $\mathbf{X}\mathbf{u}^{1,\dagger}$
manjietsu@bit.edu.cn
Guangyuan Jiang
jgy@stu.pku.edu.cn
Wei Liang  $^{1,3,\boxtimes}$
liangwei@bit.edu.cn
Chi Zhang
zhangchi@bigai.ai
Yixin Zhu2,
yixin.zhu@pku.edu.cn
<sup>1</sup> School of Computer Science & Technology, Beijing Institute of Technology
$^{2}$  Institute for AI, Peking University
<sup>3</sup> Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China
$^{4}$  National Key Laboratory of General Artificial Intelligence, BIGAI
https://sites.google.com/view/conan-active-reasoning
Recent advances in vision-language learning have achieved notable success on complete-information question-answering datasets through the integration of extensive world knowledge. Yet, most models operate passively, responding to questions based on pre-stored knowledge. In stark contrast, humans possess the ability to actively explore, accumulate, and reason using both newfound and existing information to tackle incomplete-information questions. In response to this gap, we introduce  $\mathbb{Q}$  Conan, an interactive open-world environment devised for the assessment of active reasoning.  $\mathbb{Q}$  Conan facilitates active exploration and promotes multi-round abductive inference, reminiscent of rich, open-world settings like Minecraft. Diverging from previous works that lean primarily on single-round deduction via instruction following,  $\mathbb{Q}$  Conan compels agents to actively interact with their surroundings, amalgamating new evidence with prior knowledge to elucidate events from incomplete observations. Our analysis on  $\mathbb{Q}$  Conan underscores the shortcomings of contemporary state-of-the-art models in active exploration and understanding complex scenarios. Additionally, we explore Abduction from Deduction, where agents harness Bayesian rules to recast the challenge of abduction as a deductive process. Through  $\mathbb{Q}$  Conan, we aim to galvanize advancements in active reasoning and set the stage for the next generation of AI agents adept at dynamically engaging in environments.
Active interaction with the environment is fundamental to human understanding of the world around us. Both neural and behavioral studies indicate that through active engagement with their surroundings, humans garner critical insights and foster a profound understanding of complex phenomena (Goodale and Milner, 1992; Rizzolatti et al., 1997; Rieber, 1996). When confronted with partial or ambiguous data, our innate response is to seek supplementary evidence, hypothesize, and put forth possible explanations, sometimes even reevaluating initial assumptions (Yuan et al., 2022). This iterative process persists until a satisfactory resolution emerges.
The process of formulating theories based on observations and prior knowledge is classically termed as abductive reasoning or simply, abduction (Peirce, 1965; Douven, 2021). A topic of enduring
$^{\dagger}$ Work done while M. Xu was an intern at Peking University.
Figure 1: An example of  $\mathcal{Q}$  Conan, an open-world environment for active reasoning. (a)  $\mathcal{Q}$  Conan initialization. A vandal is randomly assigned a task from the task space while keeping alive. A probabilistic parser, utilizing a knowledge graph, selects a sequence of subgoals to fulfill the main objective. This decision is then conveyed to a planner which, in turn, invokes heuristic policies to execute atomic actions. Some of these actions leave discernible traces within the environment. (b)  $\mathcal{Q}$  Conan playground with traces. (c)  $\mathcal{Q}$  Conan questions. Here, a detective is spawned and is tasked with answering queries. It does so by actively exploring the environment, connecting keyframes, and reaching conclusions.
interest among psychologists, abduction is perceived as a cornerstone of human cognitive processes. Historical and contemporary studies have delved into its cognitive mechanisms (Josephson and Josephson, 1996; Thagard, 1988; Peirce, 1965), practical applications (Hobbs et al., 1993; Shank, 1998), and ties to scientific thinking and decision-making (Hanson, 1965; Gigerenzer and Gaissmaier, 2011; Zhang et al., 2021a). With growing momentum in the machine learning sphere, recent years have witnessed the advent of dedicated benchmarks and models accentuating abductive reasoning (Bhagavatula et al., 2019; Kayser et al., 2021; Hessel et al., 2022; Liang et al., 2022).
However, the bulk of prior work in this domain relies heavily on a single-round, passive question-answering paradigm that offers complete information. This setup often sees an agent simply responding to queries, leveraging vast pre-trained knowledge, as evidenced by the latest strides in language and vision-language learning. Recent progress in the field has notably already improved performance in such complete-information information question-answering. Contrarily, humans demonstrate a far more nuanced approach when navigating abductive scenarios with incomplete data (Edmonds et al., 2018). We actively engage, explore, gather, and reason, drawing from both new information and prior knowledge. Our iterative approach allows for continuous refinement based on newly acquired evidence (Oaksford and Chater, 1994; Bramley et al., 2017; Edmonds et al., 2019, 2020).
To capture the dynamic and exploratory essence of abductive reasoning—termed herein as active reasoning—we present  $\mathcal{Q}$  Conan, a new open-world environment tailored for abductive reasoning. Standing head and shoulders above traditional single-round passive reasoning benchmarks,  $\mathcal{Q}$  Conan boasts an open-world arena, urging agents to actively probe surroundings and engage in multi-round abductive inferences, all while leveraging in-situ collected evidence alongside pre-existing knowledge.
At its core,  $\mathbb{Q}$  Conan is conceived as a detective game, transmuted into a question-answering challenge. Here, the detective is tasked with a query and an "incident scene" riddled with traces left by a vandalism. Given the initial paucity of conclusive information, the detective must embark on an in-depth exploration of the scene. As the inquiry progresses, the detective has the opportunity to actively scout its environment, continually reshaping and honing its hypotheses, especially when new revelations potentially contradict the prior hypothesis. Furthermore, we meticulously craft questions within  $\mathbb{Q}$  Conan to span various levels of abstraction, from localized intentions (Intent) to overarching objectives (Goal) and survival states (Survival).
To probe the proficiency of active reasoning, we evaluate state-of-the-art Reinforcement Learning (RL) and multimodal question-answering models on  $\mathbb{Q}$  Conan. Our observations highlight an intriguing dichotomy: while these cutting-edge models exhibit prowess in addressing low-level, short-term tasks, they struggle with multi-round environmental interactions and high-level abductive reasoning.
A plausible root of this challenge could be the absence of structurally represented knowledge. Predicated predominantly on associative training, these agents are versed in correlating traces with responses without genuinely internalizing holistic world models. In sharp contrast, humans seamlessly navigate abductive reasoning by forecasting potential trajectories leading to a perceived outcome. This intricate dance gradually transmutes from abductive to deductive reasoning, where humans harness their innate understanding of causality to deduce and mirror observed patterns. In our pursuit to mirror this quintessential human trait, we integrate Abduction from Deduction (AfD) into  $\mathbb{Q}$  Conan via a Bayesian approach. Experimental results underscore the efficacy of AfD, indicating a substantial avenue for bolstering agent adeptness in  $\mathbb{Q}$  Conan.
To sum up, our work makes the following three contributions:
- We usher in the novel domain of active reasoning, underscoring the indispensable roles of active exploration and iterative inference in abductive reasoning. This paradigm shift transforms traditional single-round passive question-answering paradigms into a more immersive format, compelling agents to actively engage with the environment to procure pivotal evidence.
- We introduce  $\mathcal{Q}$  Conan, a new environment tailored to evaluate the abductive reasoning ability of current machine learning models within dynamic settings.  $\mathcal{Q}$  Conan surpasses its predecessors that hinge on step-by-step deductive reasoning, revealing the limitations of present-day models.
- We formulate a new learning method for abduction, AfD, grounded in Bayesian principles. This framework elegantly reformulates abduction into deduction, proving instrumental in navigating the complex active reasoning challenges posed by  $\mathcal{Q}$  Conan.
Machine Abductive Reasoning Abductive reasoning, foundational to human cognition, is crucial for scientific exploration, decision-making, and problem-solving (Peirce, 1965; Magnani, 2011). In the Artificial Intelligence (AI) landscape, there is a rich history of efforts to equip machines with this ability, where they use prior knowledge and sparse observations to hypothesize amidst uncertainty (Josephson and Josephson, 1996; Xu et al., 2023). Key developments span logic-based abduction (Kakas et al., 1992; Poole, 1993) and hybrid neural-symbolic methods (Roktäschel and Riedel, 2017; Zhang et al., 2021b; Li et al., 2022, 2023). With computational progress, Large Language Models (LLMs) have effectively addressed several challenges through text generation, exhibiting outstanding performance (Brown et al., 2020; OpenAI, 2023; Thoppilan et al., 2022). Modern research usually frames abductive reasoning within natural language understanding (Bhagavatula et al., 2019) or multimodal vision-language integration (Hessel et al., 2022; Liang et al., 2022). However, there is still a notable gap: many benchmarks lean heavily on deduction, sidelining abduction's interactive essence. Our work addresses this gap, emphasizing the core of active reasoning in abductive contexts.
Embodied Question Answering Embodied question answering enhances traditional Visual Question Answering (VQA) by placing agents in interactive environments (Johnson et al., 2017; Das et al., 2018; Gordon et al., 2018; Yu et al., 2019). In  $\mathbb{Q}$  Conan, agents actively explore to gather data, preparing them to solve abductive questions based on partial information. Unlike standard embodied question-answering frameworks (Das et al., 2018; Gordon et al., 2018; Yu et al., 2019), where questions become simple instructions for agents,  $\mathbb{Q}$  Conan introduces complexity: (i) its questions, rooted in high-level intent and goals, resist simple decomposition into a series of actions; (ii) agents in  $\mathbb{Q}$  Conan act as detectives, constantly hypothesizing from observations and prior knowledge, and iterating their strategies in light of new data. For a comprehensive comparison of  $\mathbb{Q}$  Conan with other benchmarks, see Tab. 1.
3 The Conan Environment
$\mathbb{Q}$  Conan is crafted as an interactive question-answering environment aimed at evaluating a machine's active abductive reasoning capacity, as depicted in Fig. 1. Building on the foundation of the Crafter (Hafner, 2021),  $\mathbb{Q}$  Conan evolves into a detective game featuring two agents: the vandal and the detective. The gameplay kickstarts with the vandal undertaking a randomly designated task, leaving behind traces for the detective to unravel. Subsequently, given these traces, pertinent queries are generated. Finally, the detective is spawned in the environment, tasked with navigating these traces and actively probing the environment, all to derive answers through abductive reasoning.
Table 1: Comparison between  $\mathrm{Q}$  Conan and related visual reasoning benchmarks.  $\mathrm{Q}$  Conan is unique for its active reasoning and interactive multi-round setting on abductive reasoning tasks.
Benchmark	Format	Multimodal	Interactive	Multi-round	Abductive
CLEVR (Johnson et al., 2017)	image	✓	X	X	X
IQA (Gordon et al., 2018)	embodied	✓	✓	X	X
EmbodiedQA (Das et al., 2018)	embodied	✓	✓	X	X
AR&T(Bhagavatula et al., 2019)	language	X	X	X	✓
VAR (Liang et al., 2022)	video	✓	X	X	✓
Sherlock (Hessel et al., 2022)	image	✓	X	X	✓
Conan (Ours)	open-world	✓	✓	✓	✓
3.1 Basic Components
Playground Originating from the Crafter playground,  $\mathbb{Q}$  Conan operates within a  $64\times 64$  grid matrix. Agents navigate this space with a localized  $9\times 9$  grid field of view centered on their current position. Once the detective is created in the environment, all traces left behind by the vandal persist, serving as clues for the detective to unravel. While pivotal studies (Johnson et al., 2016; Fan et al., 2022; Cai et al., 2023; Wang et al., 2023) address perception in 3D Minecraft settings using foundational models, our emphasis is on honing active abductive reasoning. To this end, we transition from a 3D visual perception to a 2D plane, ensuring a harmonious blend of reduced visual complexity and retaining rich interactivity (Xie et al., 2021).
Items and Actions Conan offers an extensive assortment of interactive items: food, materials, mobs, and tools, each tied to specific actions, as illustrated in Fig. 2. It furnishes 26 unique actions to foster agent-environment en
Figure 2: Part of the task dependency graph. Starting from the root note, any path forms a multi-step task for an agent to interact with the environment.
gagement. Certain actions leave traces, and together, the items and their mechanics provide a rich set of affordances for agents in the playground. This knowledge about item operations and traces aids the detective in comprehending the incident scene. Advancing from its predecessor, the original Crafter,  $\mathbb{Q}$  Conan now boasts over 30 achievements, a significant rise of over  $50\%$ . It features 32 distinct traces covering all agent actions such as crafting, collecting, defeating, eating, drinking, and incurring injuries. This enhancement enables the design of 60 varied abductive reasoning tasks within the scene. For an in-depth overview of the playground, refer to Appx. A.
Vandal Each Conan map starts with the initialization of a vandal. This agent is driven by two primary aims: executing a specific task and preserving its existence within the environment. It is noteworthy that external threats might terminate the vandal prematurely. Traces left in the aftermath of the vandal's activities form the question foundation for the detective, with every trace potentially birthing several questions. For a detailed overview, see Sec. 3.2. We model the vandal as optimal: when given a random task and the full map, it strategically delineates a sequence of subgoals based on the task dependency graph, all while ensuring its survival. In scenarios with multiple viable paths to an objective, uniform sampling comes into play. This sampling, supported by a probabilistic parser, presents varied strategies for task completion. Hence, the detective must delve deeper to distinguish the actual sequence of events from possible decoys. The execution of the vandal's individual actions, as per the planned subgoal sequence, is steered by a collection of pre-established policies.
Detective After generating questions from a given trace, a detective is spawned to answer them. Traces left by the vandal span multiple steps and are only partially observable within the detective's  $9 \times 9$  grid field of view. This requires the detective to actively interact with the environment and gather evidence to answer the questions.
Though both detective and vandal share the same action space, the detective boasts a unique capability. It not only navigates and interacts like the vandal, but can also generate its own traces during its investigation. These overlaid traces from the detective enhance the environment's depth and complexity. This setup pushes the agent to actively derive conclusions from its dynamic interactions. Importantly, the detective is invulnerable; its focus lies squarely on problem-solving, eliminating concerns about survival or evasion. This design emphasizes active exploration and reasoning, ensuring Conan's primary goal remains addressing complex reasoning tasks and answering visual scene-related questions.
3.2 Questions and Choices
Conan is designed to assess the abductive reasoning capability of machine models through a diverse set of questions varying in difficulty and abstraction. These questions fall into three primary categories: Intent (local intent), Goal (global goal), and Survival (agent's survival status change). We approach evaluation as a multi-choice question-answering task. Each question offers four choices, with only one being correct. Questions and choices derive from predefined templates, as showcased in Tab. 2. For a more detailed explanation, see Appx. B.1.
Table 2: Examples of three categories of questions in  $\mathbb{Q}$  Conan created from predefined templates.
Type	Questions
Intent	What did the vandal make on this table?A: wood sword; B: wood pickaxe; C: iron sword; D: stone sword;Why did the vandal cut a tree here?A: make table; B: make wood sword; C: make finance; D: collect apple;
Goal	What was the vandal's primary objective in this scenario?A: get diamond; B: defeat zombie; C: collect apple; D: make iron sword;What was the desired outcome of the task performed by the vandal?A: make steak; B: make table; C: defeat skeleton; D: collect lava;
Survival	Why did the vandal die in this situation?A: lack of water; B: lack of food; C: hurt by monster; D: hurt by lava;What could the vandal have done differently to avoid a negative outcome?A: avoid monsters; B: get sleep; C: get food; D: get water;
Intent questions target the *vandal*'s immediate objectives or intentions during its task. To decipher these traces, agents must deduce the *vandal*'s underlying intent or subgoals. Solving these questions necessitates a learning model's comprehension of the local context.
Goal questions probe the *vandal*'s overarching objectives, extending beyond immediate intents. They necessitate grasping the wider context of a task or action sequence. Such questions query the *vandal*'s ultimate aims, demanding a learning model to reason within the broader context of the traces.
Survival questions address the wider investigative scope, posing added challenges to the detective. Centered on the vandal's survival status changes during tasks (e.g., collecting food for sustenance), they lead to deviations from the optimal action plan. While not tied to a task's primary objective, these questions require a deeper grasp of the present context, often necessitating reasoning around potential scenarios or alternate results.
Compared with the prevalent VQA setup, wherein questions are based on factual information that is readily obtainable from the input,  $\mathbb{Q}$  Conan questions cannot be deciphered given only the initial information, necessitating further exploration in the scene. Unlike standard embodied question answering,  $\mathbb{Q}$  Conan questions cannot be directly parsed as modular primitives; they demand abductive reasoning, drawing from both new observation and former knowledge to hypothesize, validate, and revise. For benchmarking purposes,  $\mathbb{Q}$  Conan produced a corpus comprising 100,000 questions. These were derived from 10,000 unique scenes, generated via the Crafter's scene generator, with each scene stemming from a task executed by a vandal. This resulted in an average generation of 10 questions per scene.
Figure 3: An illustration of the detective pipeline for  $\mathbb{Q}$  Conan. An RL explorer is first trained to gather traces in accordance with the given question. Given a question and the incident scene, the detective calls the explorer subroutine to gather evidence. Next, the exploration sequence undergoes key-frame extraction, processed by a visual encoder, subsequently feeding into a vision-language model for answer selection.
4 The Detective Pipeline
Conan casts the abductive reasoning challenge as a detective game, necessitating a detective to efficiently explore and gather information from the environment to deduce plausible explanations (i.e., answers) for the given question. This process involves taking into account the temporal dependencies and incompleteness of the traces. To tackle these challenges encountered in Conan, we devise a detective pipeline, as depicted in Fig. 3.
Building on previous work that utilizes hierarchical models for task decomposition (Gordon et al., 2018; Das et al., 2018; Wijmans et al., 2019), our pipeline is structured into two primary phases: an exploration phase for trace collection, followed by an abductive reasoning phase. Initially, interaction with the playground is carried out to collect relevant visual information, which is subsequently leveraged in the reasoning phase to infer answers to the posed questions.
Computationally, our pipeline first employs RL agents as explorers (see Sec. 4.1) that learn an exploration policy based on the traces and the question, thereby rendering it goal-oriented. Next, given the question, we recruit vision-language models (see Sec. 4.3) to predict the answer based on the observation. A key-frame extractor (see Sec. 4.2) is inserted into the two phases to selectively identify relevant frames for abduction. The individual components undergo separate training procedures.
4.1 Explorer for Trace Gathering
The primary responsibility of an explorer is to efficiently collect information pertinent to the provided question. Initially, masks are employed to encode questions by highlighting relevant grids. Subsequently, the explorer takes in both the observation and the target question as input and outputs the action probability.
We use a reward function that incentivizes the agent to scout for clues and traces relevant to the given question. Additionally, a penalty term is incorporated to discourage unnecessary actions and inefficient searching, thereby promoting a more targeted exploration strategy.
Specifically, the agent is rewarded with  $+1$  when a trace first appears within its local view, or  $+2$  when the trace bears a close association with the question. A substantial reward of  $+100$  is conferred upon the agent if it successfully uncovers all traces left by the vandal. Concurrently, the agent incurs a penalty of  $-0.1$  for every timestep elapsed, with an additional penalty of  $-1$  imposed for executing operating actions.
We evaluate multiple well-regarded RL frameworks as our explorer, including Deep Q-Network (DQN) (Mnih et al., 2015), Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), and Recurrent Proximal Policy Optimization (RecurrentPPO) (Schulman et al., 2017). The Stable-Baselines3 library (Raffin et al., 2021) is employed for all implementations.
4.2 Key-Frame Extractor
Given that the frames gathered by the explorer tend to be excessively lengthy and redundant, a key-frame extractor is utilized to sift through and select informative frames containing crucial evidence for the detective. We adopt a prevalent selection strategy employed in video understanding (Arnab et al., 2021). Specifically, frames within the temporal bounds determined by the detection of the first and last traces are retained, from which  $k$  frames are uniformly sampled. This design is intended to tailor the input with the constrained context window size to downstream vision-language models.
4.3 Vision-Language Models for Abductive Reasoning
We employ a multi-choice question-answering paradigm akin to the one used in Ding et al. (2021). Specifically, the model is presented with a question, its corresponding exploration frame sequence, and each potential answer choice, subsequently generating a score for each choice. The model is trained with a categorical cross-entropy loss. During inference, the choice with the highest score is considered the answer. We evaluate several well-established multimodal models; these models are known for their efficacy in processing both visual and textual data. Additional details on model implementation can be found in Appx. D.1.