SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Vanilla-Trans The first baseline method leverages a vanilla transformer encoder to fuse observation and textual inputs. Specifically, the raw symbolic map from  $\mathbb{Q}$  Conan serves as the visual feature, while CLIP's text encoder (Radford et al., 2021) is employed to encode the textual input.
FrozenBiLM FrozenBiLM (Yang et al., 2022), a state-of-the-art model for video question answering, combines visual input with frozen bidirectional language models, trained on web-scraped multimodal data. The approach integrates a frozen language model and a frozen vision encoder with light trainable visual projection modules. FrozenBiLM is tested with BERT-Large (Kenton and Toutanova, 2019) and DeBERTa-v3 (He et al., 2022) as the language model within our question-answering system, utilizing the symbolic map from  $\mathcal{Q}$  Conan for visual input.
Flamingo-Mini Flamingo (Alayrac et al., 2022) is a family of vision-language models adept at rapid adaptation to novel tasks with minimal annotated examples. These models can handle sequences of visual and textual data, seamlessly accommodating interleaved images or videos as input. We finetune an open-sourced Flamingo-Mini model with frozen OPT-125M (Zhang et al., 2022), utilizing the symbolic map from  $\mathbb{Q}$  Conan for visual input.
4.4 Abduction from Deduction (AfD)
The adage "Set a thief to catch a thief" suggests the use of someone with a similar background or expertise to apprehend a wrongdoer: the best vandal catchers are vendors. This notion resonates with the core principle of Abduction from Deduction (AfD): for a skillful detective to abduce what a vandal does, it needs an in-depth grasp of vendors' modus operandi, motivations, and decision-making process. Translating the implication to a mathematical language, we articulate the problem of abductive reasoning based on evidence and knowledge from known deductive transitions. It can also be seen as an extension of inverse planning (Baker et al., 2007, 2009; Baker and Tenenbaum, 2014). Formally, let  $g$  denote the goal of the vandal,  $O$  the detective's observation, and  $S$  the playground states post the vandal's actions. We then have:
$$
P (g \mid O) = \underset {P (S \mid O)} {\mathbb {E}} [ P (g \mid S, O) ] = \underset {P (S \mid O)} {\mathbb {E}} [ P (g \mid S) ], \tag {1}
$$
where we assume the independence of  $g$  w.r.t.  $O$  given  $S$ , as the goal ought to be clear given the states. Leveraging Bayesian rules, we further observe that
$$
P (g \mid S) \propto P (S \mid g) \propto \prod_ {i} \pi \left(a _ {i} \mid s _ {i}, g\right), \tag {2}
$$
assuming a uniform prior over  $g$  and known deterministic environment transitions. Eq. (2) asserts that  $P(g \mid S)$  is proportional to a goal-conditioned forward action policy, where  $s_i, a_i \to s_{i+1}$ .
Intuitively, Eqs. (1) and (2) can be understood as follows: to abduce the vandal's goal from observation, it is imperative to first reconstruct the actual states traversed by the vandal and subsequently ascertain the most plausible goal that, if pursued forward, would result in those states; see Eq. (1). Eq. (2) can be interpreted as a form of deduction, being contingent on transition knowledge derived from a forward action policy. Hence the name Abduction from Deduction (AfD).
In practice, two approaches emerge for implementing  $P(g \mid S)$  based on Eq. (2). The first entails iterating over all  $g$  and utilizing a learned or predefined  $\pi(\cdot)$  to score a lengthy sequence of states. Conversely, the second approach embraces a data-driven strategy, wherein one arbitrarily selects  $g$ , samples  $S$  from  $\pi(\cdot)$ , and learns a model of  $P(g \mid S)$  using the  $(g, S)$  pairs. The former approach proves time-intensive during inference due to the combinatorial temporal space and expansive goal space, thereby compelling us towards the latter approach. For implementation, we train  $P(S \mid O)$  independently as a Dirac delta function of  $\delta(f(O))$  and  $P(g \mid S)$  from sampled pairs from  $\pi(\cdot)$  employed in task execution in the vandal. The derived goal features, along with the question, are fed into the model for answer prediction. Please refer to Appx. F for additional details.
Exploration The explorer is trained using DQN, TRPO, and RecurrentPPO for  $10^{8}$  steps, with a buffer size of  $10^{7}$  and a batch size of 512. In the case of DQN, training is conducted with  $\epsilon = 0.96$ . Each episode is capped at a maximum of 500 steps for the explorer. A curriculum is employed to encourage long-term exploration whilst maintaining a balance with local search: initial training is carried out with traces from long-horizon tasks like "get the diamond," compelling the agent to venture farther from its starting point. Subsequently, the agent undergoes further finetuning across the entire dataset. Such a curriculum design prevents a sole focus on local discovery. For downstream reasoning models,  $k = 30$  keyframes are extracted by the key-frame extractor.
Abductive Inference Our reasoning models are tested under three different settings: Standard, Ideal Explorer, and AfD. In the Standard setting, models undergo training and testing based on the explorer's exploration. The Ideal Explorer setting sees models leveraging on an optimal exploration policy—visible to the ground-truth *vandal*'s trajectory, albeit imperfect, it facilitates the agent in gathering sufficient evidence for reasoning. This scenario can be conceived as a measure of the reasoning model's aptitude for passive reasoning given complete information. Under the AfD setting, models are trained and used as delineated in Sec. 4.4. All models are trained utilizing 8 NVIDIA GeForce RTX 3090 GPUs. For further training specifics, please refer to Appx. D.2.
5.2 Results and Analysis
Fig. 4 shows the learning curves of various RL agents during exploration. TRPO and RecurrentPPO manifest similar performance in terms of rewards following a substantial number of steps, markedly surpassing the DQN explorer. Additionally, we probe the impact of augmenting the maximum number of exploration steps to 5,000 on performance. The data suggests a marginal performance uplift. Nonetheless, we acknowledge that such a performance increment is at the expense of substantially longer exploration time and a notable surge in the accrual of unrelated information. Consequently, we select TRPO with a maximum of 500 steps per episode as our standard RL explorer.
Quantitative results on  $\mathbb{Q}$  Conan are depicted in Tab. 3; both the standard and AfD results reported employ TRPO as the explorer. In the
standard setting, we discern that while models exhibit some aptitude in tackling low-level Intent questions, they struggle with higher-level questions pertaining to Goal and Survival. Among the models, Flamingo-Mini ascends to the pinnacle with an accuracy of  $66.3\%$ . FrozenBiLM models also perform relatively well. Notably, the DeBERTa variant slightly outperforms BERT, insinuating that a robust language backbone can improve general comprehension. Contrarily, the Vanilla-Trans model languishes across all tasks, achieving merely random-level performance.
Figure 4: Learning curves of various RL explorers. The suffix  $n$  denotes the maximum number of steps per episode during exploration. Results show that (i) TRPO and RecurrentPPO markedly outperform DQN in performance, and (ii) longer episodes marginally contribute to the performance at the expense of longer exploration time and the accrual of unrelated information.
Table 3: Performance of abductive reasoning models on Conan. We report the question-answering accuracy  $(\%)$  across various settings, with the overall accuracy averages over all question categories. F-BiLM refers to the FrozenBiLM model. I denotes Intent, G denotes Goal, S denotes Survival, and O denotes Overall. Results exhibiting the top individual performance are highlighted in bold, while models with the superior overall performance are shaded in gray.
Standard	Ideal Explorer	AfD
I	G	S	O	I	G	S	O	I	G	S	O
Vanilla-Trans	32.9	25.0	24.5	28.8	64.0	78.4	58.1	66.1	24.8	23.3	24.5	24.3
F-BiLM-BERT	72.6	44.4	54.4	61.0	87.5	59.5	61.5	74.0	82.8	42.9	55.5	66.0
F-BiLM-DeBERTa	82.9	43.1	52.2	65.3	87.7	71.8	63.9	77.8	82.9	41.9	53.8	65.4
Flamingo-Mini	86.2	43.3	49.5	66.3	85.8	47.8	56.6	69.0	84.9	42.5	52.2	66.1
With the Ideal Explorer, we notice a clear performance boost across all tasks, particularly in the Goal and Survival questions. These results allude to the potential bottlenecking of models' abductive reasoning capability due to the insufficient information collected, underscoring the significance of effective exploration. An adept explorer can significantly aid in the accrual of useful information, informatively pursuing a hypothesis to scrutinize evidence, swiftly self-correcting upon encountering conflicting evidence, and reasonably re-planning. The findings also hint sufficient room for the RL explorer to improve. Remarkably, the Vanilla-Trans exhibits the greatest increase, insinuating that, in comparison to other baseline models, it is markedly vulnerable to insufficient evidence.
For AfD results, nearly all multimodal models exhibit performance on par with end-to-end supervisedly trained models. Remarkably, FrozenBiLM models even surpass the performance observed in standard settings. The persisting failure of Vanilla-Trans can be ascribed to its weakness in reasoning amidst incomplete observations due to the significant disparity between the familiar complete state  $S$  and incomplete observation  $O$ . Examining task-specific results, a notable performance uplift in the Survival task models is discernible for almost all models relative to the standard setting, albeit sharing the same observation. These results intimate that the inclusion of deductive information sensitizes the detective to vandal's concerns during task execution. Nevertheless, the exhibited performance in long-term planning remains weak, reinforcing the pressing need for a better exploration policy. Critically, these models continue to find short-term intent questions to be most easily answered.
5.3 Further Discussion
Additional Experiments We further experiment in the absence of visual inputs, serving as a negative control baseline, resulting in random performance across all settings; see Appx. E. This random-level performance underscores the severe constraints imposed on the agent without visual information. The TRPO explorer shows a noticeable improvement over the ones without visual inputs, suggesting that even minimal exploration is preferable to none. Nonetheless, the performance remains relatively modest. On the other hand, the Ideal Explorer demonstrates markedly superior performance, attesting to the substantial benefits its capacity to accrue perfect trace evidence renders to the downstream reasoning task. This accentuates the imperative of effective exploration.
Table 4: Error analysis on Conan. We examine the accuracy of FrozenBiLM-DeBERTa across various tasks, comparing two explorer groups: reasoning based on the TRPO explorer and the Ideal explorer (in gray).
get_drink	defeat_cow	get.apple	defeat}skeleton	make_iron_pickaxe
47.06	43.90	35.7	46.59	56.52
100.00	85.37	78.57	82.95	52.17
place.bed	make_steak	make_stone_pickaxe	get_coal	make_stone_sword
43.90	46.15	48.48	50.00	37.50
87.80	50.00	39.39	45.45	4.17
get_iron	get_water	get_stone	make_iron_sword	place_furnace
28.57	45.95	36.84	56.25	44.44
46.43	54.05	47.37	28.12	83.95
get_diamond	place_table	get_wood	make_wood_pickaxe	make_wood_sword
40.62	39.36	36.00	40.00	50.00
84.38	91.49	96.00	55.00	64.29
make.bed	get_lava	make_bucket	get_beef	defeat_zombie
47.83	50.00	35.29	53.85	52.50
39.13	66.67	73.53	42.31	75.00
Error Analysis We extend an error analysis for the "goal" split, probing the reasoning model across a spectrum of tasks. Table 4 compares two groups: reasoning based on the Ideal explorer and the TRPO explorer. The findings underscore that proficient exploration, i.e., the heuristic Ideal explorer who recovers the vandal's trajectory, is sufficient for satisfactory performance. However, to fully harness the potential, a more adept reasoner is requisite, one capable of deciphering the vandal's hidden states from observed traces. For instance, the act of felling trees could signify a need for either wood or food (apples), and discerning the intent solely from traces of felled trees presents a challenge. When it comes to "trace-relevant" frames or "keyframes," the Ideal explorer could ostensibly furnish all trace-relevant frames. However, the concept of keyframes remains nebulous. Within the video understanding domain, a formidable challenge lies in the extraction of "keyframes." This is a post-hoc concept that eludes straightforward acquisition upfront. A prevailing approach, aimed at augmenting efficiency (diminishing context length in Transformer), entails truncating it via every k-th frame.
Joint Reasoning The collective enhancement of both exploration and reasoning elements emerges as quintessential, given its mirroring of human-like intelligence. For instance, by providing feedback, the reasoner can steer the explorer towards actions that are potentially more insightful and likely to produce pertinent traces. Nonetheless, practical implementation encounters significant hurdles. Assigning credit to exploratory decisions bearing long-term implications can be intricate, particularly when the outcomes of exploratory actions become evident after a substantial time lapse, thereby muddying the causal relationship between the decisions and their ultimate effect on reasoning and answering questions. This accentuates the mutual reliance between exploration and reasoning—advancement in one facet demands progression in the other, introducing a bilateral dependency that complicates optimization. The reasoning component alone demands hefty training and computational resources, especially when utilizing large language models. The demand for formidable computational power renders the simultaneous optimization of exploration and reasoning exceedingly daunting. Collectively, this approach is also widely adopted (Gordon et al., 2018; Lei et al., 2018; Kocisky et al., 2018). Consequently, we navigate along this trajectory, projecting that future endeavors on Conan should prioritize reasoning above exploration.
To summarize, the engagement of a proficient explorer substantially enhances abductive reasoning, particularly in higher-level tasks such as goal-oriented and survival-centric inquiries. This underlines the criticality of exploration as a precursor to tackling abductive reasoning tasks in the presence of incomplete information. Furthermore, the achievement of the AfD hint at the potential for models to harness world knowledge, especially transition knowledge pertaining to tasks and traces, to transform abductive reasoning into deductive simulation. We posit that the presented approach resonates more with human-like reasoning, edging us closer to the core of human intelligence.
In this paper, we introduce  $\mathcal{Q}$  Conan, a benchmark tailored to evaluate and assess models' active reasoning ability in addressing incomplete-information questions in an interactive environment.  $\mathcal{Q}$  Conan sets itself apart from existing abductive reasoning benchmarks by incorporating an open-world playground facilitating active exploration. It differentiates itself from prevailing embodied question-answering benchmarks by introducing the demanding abductive process in question answering, necessitating multi-round abductive inference based on gathered evidence. Moreover, we propose a new learning paradigm, Abduction from Deduction (AfD), that turns the problem of abduction to deduction, exploiting the problem structure through Bayesian principles. Benchmarking the efficacy of contemporary machine learning models on  $\mathcal{Q}$  Conan, we elucidate the model limitations in interacting with the environment that leads to failure in higher-level, longer-term abductive reasoning.
Limitations and Future Work In general, we notice two significant limitations from the experimental results. For one, the explorer does not supply particularly relevant information for the reasoning model. In the human abductive reasoning process, exploration and reasoning should be closely intertwined, with an agent using the current hypothesis to guide exploration and improve its understanding. However, due to long-range exploration and complex vision-language reasoning, we only applied the conventional visual question-answering method and did not fully integrate these two processes. For another, learning naive question-answer mapping shall be sub-optimal. By leveraging the problem structure, AfD has shown improved performance on a particular set of problems. Nevertheless, the current AfD formulation is still rudimentary. We believe an in-depth understanding of the structure and well-crafted implementation could further boost performance.
Acknowledgement The authors would like to thank Ms. Zhen Chen (BIGAI) for designing the figures, and NVIDIA for their generous support of GPUs and hardware. M.X., G.J., W.L., C.Z., and Y.Z. are supported in part by the National Key R&D Program of China (2022ZD0114900), M.X. and W.L. are supported in part by the NSFC (62172043), and Y.Z. is in part by the Beijing Nova Program.
Zhang, C., Jia, B., Edmonds, M., Zhu, S.-C., and Zhu, Y. (2021a). Acre: Abstract causal reasoning beyond covariation. In Conference on Computer Vision and Pattern Recognition (CVPR). 2
Zhang, C., Jia, B., Zhu, S.-C., and Zhu, Y. (2021b). Abstract spatial-temporal reasoning via probabilistic abduction and execution. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022). Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. 7
A Conan Playground
Conan's playground is a computationally efficient 2D open-world environment with diverse items and rich tasks. The most distinctive feature of Conan's playground over the original Crafter environment is that agents in Conan leave diverse traces when interacting with the environment. These traces serve as the foundation for abductive reasoning; the detective has to effectively connect the traces to figure out what the vandal has done.
A.1 Items and Traces
Land Based on Crafter, there are three types of terrains that agents can walk on: sand, grass, and path. Sand and grass are soft surfaces where agents leave directional footprints after walking on them (see Fig. A1 first 2 rows in Columns 2 and 3 for examples). If a grid is left with more than one footprint, the footprints will become melded (Fig. A1 Column 4 in first 2 rows). Agents' actions will also leave traces on the terrain, e.g., water on the ground (Fig. A1 Column 5 first 2 rows). If an agent gets injured, blood will be shed on the ground (Fig. A1 Column 6 first 2 rows).
**Creatures** There are four creatures in the playground: plant, cow, zombie and skeleton. plant grows from sapling to ripe plant. Cow randomly wander on the ground, whereas zombie and skeleton (monsters in general) will target agents in sight: zombie chases agents and skeleton shoots arrow at agents. Agents can fight with creatures and kill them. These actions will leave monster bodies on the ground.
Tools Agents can make tools on the table. There are 7 tools in total: bucket, wood_sword, wood_pickaxe, stone_sword, stone_pickaxe, iron_sword, and iron_pickaxe. These tools can be made using different materials and used for certain tasks. Both swords and pickaxes can be used to fight with creatures, but only pickaxes can be used in mining. Buckets can be used to collect water and lava.
Actions Conan's playground enables agents to interact with objects, non-playable characters, and even other agents in the playground. Agents can cut tree to get apple and wood, as well as collect sapling and grow plant (Fig. A1 Row 3). They can also mine with different tools to get stone, coal, iron, and diamond. Using these materials, agents can make bed for sleep, furnace for keeping monsters away and grilling food, table for making tools, etc. Of note, these items should be placed in an empty grid to use and they can be destroyed by monsters.
A.2 Achievements and Tasks
There are 60 tasks and 39 achievements in  $\mathbb{Q}$  Conan's playground. We list all achievements in Tab. A1. Tasks are composed achievements. We select 60 nontrivial and meaningful tasks from all compositions in  $\mathbb{Q}$  Conan as the final task set.
Table A1: Achievements in Conan.
Type	Achievements
Survive	drink_water sleep drink_water_from_bucket	eat.apple sleep_onBED eat_plant	eat_beef wake_up	eat_steak eat_grilled.apple
Collect	collect_wood collect_iron collect_water	collect.apple collect_diamond collect_lava	collect_water collect_beef collect_sapling	collect_stone collect_coal collect_plant
Make	make_steak make_wood_sword make_iron_sword place_furnace	make_grilled.apple make_wood_pickaxe make_iron_pickaxe place_plant	make_bucket make_stone_sword place_table	make_fence make_stone_pickaxe place bed
Defeat	defeat_cow	defeat_zombie	defeat}skeleton
A.3 Observation and Action
Conan offers both pixel representation and symbolic representation for training agents. For pixel representation, the environment returns a  $900 \times 900$  RGB image each time step for the detective's
sand_left
sand_up
sand_unknown
sand_water
sand_blood
grass_left
grass_up
grass_unknown
grass_water
grass_blood
apple_tree
tree_cut
young_plant
ripe_plant
COW
stone
dead_cow
stone_left
dead_zombie
skeleton
diamond
dead_skeleton
iron_left
fur_left
dia_left
table_left
bedleft
beef
lava_bkt
apple
wood_sword
water_bkt
iron_pickaxe
wood_pickaxe
fence
stone_sword
bow
stone_pickaxe
player
iron_sword
sleep_player
arrow left
water
path
Figure A1: Items and related traces in Conan.
dead_player
energy
$9 \times 9$  local view. For symbolic representation, the environment returns a  $9 \times 9$  tensor, with each entry an index representing one of 50 grid types, covering materials, resources, objects, creatures, and etc. The agent is always at the center of the observation.
Conan affords a larger action space. See Tab. A2 for a detailed list of actions.
Table A2:Actions in Conan.
Action	Details
Noop	Do nothing.
Move Left	Move left if the grid is walkable.
Move Right	Move right if the grid is walkable.
Move Up	Move up if the grid is walkable.
Move Down	Move down if the grid is walkable.
Do	Collect materials or fight with monsters. Use tools if possible.
Sleep	Sleep to restore energy. Sleep on bed can restore energy faster;
Place Stone	Place a stone if the grid is not occupied. Should have a stone.
Place Table	Place a table if the grid is not occupied. Should have a table.
Place Furnace	Place a furnace if the grid is not occupied. Should have furnace.
Place Plant	Place a plant if the grid is grass. Should have sapling.
Place Bed	Place a bed if the grid is not occupied. Should have bed.
Make Wood Pickaxe	Nearby table. Should have wood.
Make Stone Pickaxe	Nearby table. Should have wood, stone.
Make Iron Pickaxe	Nearby table, furnace. Should have wood, coal, iron.
Make Wood Sword	Nearby table. Should have wood.
Make Stone Sword	Nearby table. Should have wood, stone.
Make Iron Sword	Nearby table, furnace. Should have wood, coal, iron.
Make Bucket	Nearby table. Should have wood, stone.
Make Steak	Nearby table, furnace. Should have beef.
Eat Apple	Restore 2 health. Should have apple.
Eat Beef	Restore 4 health. Should have beef.
Eat Steak	Restore 6 health. Should have steak.
Collect Water	Collect water to bucket. Should have empty bucket.
Collect Lava	Collect lava to bucket. Should have empty bucket.
Drink	Drink water. Drink water from water bucket if not near the water.
B Conan Questions
B.1 Question Generation