SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Manjie  $\mathbf{X}\mathbf{u}^{1,\star ,\dagger}$
Guangyuan Jiang $^{2,\star}$
Chi Zhang4,
* M. Xu and G. Jiang contributed equally. ☑ corresponding authors
3 Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China
https://sites.google.com/view/ivre
One of the fundamental cognitive abilities of humans is to quickly resolve uncertainty by generating hypotheses and testing them via active trials. Encountering a novel phenomenon accompanied by ambiguous cause-effect relationships, humans make hypotheses against data, conduct inferences from observation, test their theory via experimentation, and correct the proposition if inconsistency arises. These iterative processes persist until the underlying mechanism becomes clear. In this work, we devise the IVRE (pronounced as ivory) environment for evaluating artificial agents' reasoning ability under uncertainty. IVRE is an interactive environment featuring rich scenarios centered around Bicket detection. Agents in IVRE are placed into environments with various ambiguous action-effect pairs and asked to determine each object's role. They are encouraged to propose effective and efficient experiments to validate their hypotheses based on observations and actively gather new information. The game ends when all uncertainties are resolved or the maximum number of trials is consumed. By evaluating modern artificial agents in IVRE, we notice a clear failure of today's learning methods compared to humans. Such inefficacy in interactive reasoning ability under uncertainty calls for future research in building human-like intelligence.
Situated in a room, you rarely have a clear idea of what specific factor caused a sudden lights-out. You might begin to check the light switch, the main circuit breaker, or the light bulb itself. With a series of experiments, you can finally realize the actual cause. This is a canonical example of reasoning and resolving uncertainty through interaction: when encountering a novel scenario, humans typically lack sufficient information and knowledge to arrive at a definitive conclusion based solely on the initial observation. Instead, we formulate hypotheses, subject them to testing, and utilize newly gathered data to address the preceding uncertainty in our reasoning process (Halpern, 2017).
Navigating uncertainty through reasoning is a distinctive feature of human intellect. This journey, from Hume's explorations of causation (Hume, 1896) to contemporary scientific breakthroughs, illustrates humanity's reliance on formulating hypotheses to actively probe and gather fresh evidence, thereby diminishing ambiguity and carving certainties out of the realms of the unknown (White, 1990; Shanks, 1985). In the current landscape, the field of machine learning has witnessed remarkable advancements, spanning domains from comprehending natural language to deciphering visual stimuli. Yet, the aspiration to emulate human-like reasoning within machines remains an unfulfilled odyssey. The existing capabilities of artificial systems notably diverge from the human, even infantile, cognitive processes used to interpret our surroundings, deduce causal connections, and pioneer scientific discoveries (Gopnik, 1996).
In this work, we study the problem of visual reasoning under uncertainty, which tasks an agent to design new experiments that test hypotheses and discern each variable's causal role. In pursuit of this goal, we introduce the Interactive Visual Reasoning (IVRE) (pronounced as ivory) environment as an interactive testbed. IVRE is grounded on the Bicket detection setup (Gopnik and Sobel, 2000; Sobel et al., 2004; Sobel and Kirkham, 2006; Zhang et al., 2021a), initially designed to evaluate children's induction ability via passive observation and active trials. In a series of experiments, children were presented with a Bicket machine, which has a very intuitive working mechanism: whenever a Bicket is put on top of it, the device becomes activated, lighting up and playing music. Participants were shown a series of experiments to demonstrate the Bicketness of a set of objects. They were then encouraged to engage in exploratory play with the objects and the machine to determine the Bicketness of each object. Critically, during the exploratory process, children generated and validated different hypotheses un
til they were confident about the properties of each object (Gopnik and Sobel, 2000; Sobel and Kirkham, 2006; Sobel et al., 2004; Walker and Gopnik, 2014) and can even rationally infer causes of failed actions when they are as young as 16-month old (Gweon and Schulz, 2011). Fig. 1 shows an illustrative example of how a participant resolves uncertainty when unraveling how the Blicket machine works and which object is a Blicket.
IVRE is built following a similar interactive setting in the original Bicket experiments. Specifically, we are inspired by recent work in building synthetic reasoning benchmarks (Johnson et al., 2017; Edmonds et al., 2018; Zhang et al., 2019a; Yi et al., 2019; Girdhar and Ramanan, 2019; Zhang et al., 2021a; Xie et al., 2021; Li et al., 2022, 2023; Jiang et al., 2023; Xu et al., 2023) and adopt the CLEVR (Johnson et al., 2017) universe in creating the interactive environment. Following the recent work of ACRE (Zhang et al., 2021a) for causal reasoning, we borrow the object appearances and the Bicket machine setup. In particular, an agent is presented with a few observations of objects on Bicket machines that are either activated or not (referred to as context) at the beginning of each IVRE episode. Information for identifying which subset of objects are Bickets is incomplete from the context only, thereby introducing uncertainty. To determine the Bicketness of each object, an agent is tasked to propose trial experiments that could be carried out in each of the upcoming time steps (referred to as trials), and in the meantime, updating its belief over which object is an actual Bicket. The correctness of its belief at each step serves as the motivating signal for the agent, who additionally receives a constant penalty for every unsuccessful trial to encourage efficiency.
Serving as a testbed, IVRE evaluate interactive reasoning under uncertainty of today's state-of-the-art artificial agents (Schmidhuber, 2015; Lillicrap et al., 2015; Fujimoto et al., 2018; Mnih et al., 2015; Sutton and Barto, 2018; OpenAI, 2023). Not only do we benchmark Reinforcement Learning (RL) algorithms with visual input from the rendering engine, but also with the ground-truth
Figure 1: Children resolve uncertainty through exploratory play with the Bicket machine and objects. In the beginning, the child is uncertain about which object can activate the machine from the initial context panel that contains an active machine. S/he procedurally designs new experiments to test hypotheses. When a new trial indicates that the machine remains activated when only the yellow cylinder is present, the child knows that the cylinder alone can activate the machine, confirming its Bicketness. However, the red cube requires an additional test for verification.
Table 1: Comparison between IVRE and other related visual reasoning benchmarks in terms of tasks (classification, visual question answering, and game), sizes (number of scenarios), and input formats. IVRE introduces a spatial-temporal-causal reasoning task, which allows intervention and belief update. It aims at few-shot uncertainty resolution with fast experimentation and reasoning.
Benchmarks	Task	Size	Format	Temporal	Interactive	Uncertainty	Few-shot
CLEVR (Johnson et al., 2017)	vqa	100k	image	✘	✘	✘	✘
CLEVRER (Yi et al., 2019)	vqa	20k	video	✓	✘	✘	✘
CATER (Girdhar and Ramanan, 2019)	cls	5.5k	video	✓	✘	✘	✘
CURI (Vedantam et al., 2021)	cls	990k	image	✓	✘	✓	✓
ACRE (Zhang et al., 2021a)	cls	30k	image	✓	✘	✓	✓
Alchemy (Wang et al., 2021)	game	-	image/symbol	✓	✓	✓	✘
IVRE (Ours)	game	-	image/symbol	✓	✓	✓	✓
symbolic representation of the environment, including some additional study with Large Language Models (LLMs). We note that the environment is challenging enough for agents with even the symbolic representation, and visual complexity poses additional challenges. Further comparing the performance of the heuristic algorithms and that of the RL agents, the prominent failure of today's artificial agents in resolving uncertainty becomes even more evident, calling for future investigation into building intelligence that can learn and reason like people (Lake et al., 2017; Zhu et al., 2020).
To sum up, our work makes the following contributions:
- We present the IVRE platform, a unique environment tailored for assessing the proficiency of artificial agents in dynamically resolving uncertainty through interaction. What sets IVRE apart is the dual challenges it imposes: demanding both logical reasoning and the creation of effective strategies to mitigate uncertainty.
- The IVRE setup was meticulously designed to maintain a balance between perceptual simplicity and a rich array of visual elements and situational tasks. This environment ushers in a novel paradigm of interactive reasoning in the face of uncertainty, compelling agents to engage directly with their conjectures by formulating and executing new experimental trials.
- Utilizing the IVRE framework, we evaluated a spectrum of agents on their ability to navigate the complex problem of interactive reasoning under uncertain conditions. Additionally, our human studies confirmed the human aptitude for managing such tasks. Our observations underscore that (i). visual complexity is not the core difficulty in this task, which echos our design principle to minimize visual complexity, (ii). the art of uncertainty reduction within IVRE hinges on advanced reasoning skills and the strategic implementation of active trials, and (iii). contemporary learning agents are yet to master uncertainty reduction through interactive methods.
Visual Reasoning A range of visual reasoning and vision-language understanding tasks has been proposed recently. On the basis of a series of Visual Question Answering (VQA) benchmarks (Antol et al., 2015; Krishna et al., 2017; Tapaswi et al., 2016; Zhu et al., 2016), Johnson et al. (2017) use synthetic images depicting simple 3D shapes to scrutinize a suite of VQA models and discover their potential weaknesses. From a causal and physical reasoning perspective, the video dataset of CLEVRER was introduced by Yi et al. (2019) to investigate the performance of state-of-the-art (SOTA) models on learning complex spatial-temporal-causal structures from interacting objects in a scene. At a more abstract level, model performance in human Intelligence Quotient (IQ) tests has been studied; Barrett et al. (2018) and Zhang et al. (2019a) proposed datasets inspired by the Raven's Progressive Matrices (RPM) (Carpenter et al., 1990; Raven and Court, 1938), featuring reasoning on the hidden spatial-temporal transformation from a limited number of context panels. Approaches for this abstract reasoning task range from the neural end towards neuro-symbolism over the years (Santoro et al., 2017; Hill et al., 2018; Zhang et al., 2019b, 2021b; Wang et al., 2019; Spratley et al., 2020; Zheng et al., 2019; Wu et al., 2020). Inspired by Bicket detection and the problem of causal induction (Gopnik and Sobel, 2000; Gopnik et al., 2001), the ACRE dataset (Zhang et al., 2021a) was presented as a way to systematically evaluate current vision systems' capability in causal induction. It is worth noting a visual reasoning method based on object-centric representation and self-attention
Figure 2: A simple example episode in the IVRE environment. At the beginning of the episode, an agent is given a set of context panels (4 in an actual instance) as an introduction to the problem. Next, the agent is motivated to determine which object is a Blicket by proposing new experiments to validate its hypothesis and update its belief. The agent will receive a high reward if all Blickets have been found out.
(Ding et al., 2021) has obtained notable performance on various visual reasoning domains, including CLEVRER (Yi et al., 2019), CATER (Girdhar and Ramanan, 2019), and ACRE (Zhang et al., 2021a), with the help of unsupervised object segmentation algorithms (Burgess et al., 2019). The inclusion of embodied versions in visual reasoning tasks also marks a significant advancement in developing AI systems that can reason within intricate and realistic environments. Some other works (Beattie et al., 2016; Wayne et al., 2018) incorporate tasks designed to evaluate the reasoning capabilities of RL agents. Hill et al. (2020) demonstrates that RL agents have the ability to conceptualize new ideas within 3D settings.
While the proposed IVRE environment is based on the Bicket setup and visually similar to the ACRE dataset, the new interactive environment emphasizes a drastically different problem: apart from figuring out the hidden causal factors, the agent is also tasked with making maximally meaningful exploration from an initial setup of high uncertainty and aggregating the collected information to update its belief and guide its next decision.
Few-Shot Reasoning Several notable works have contributed to advancing agents' reasoning abilities in few-shot scenarios, where agents learn from a small number of examples or instances to solve problems. Popular works include Omniglot (Lake et al., 2015, 2019) and RAVEN (Zhang et al., 2019a). Additionally, other studies have explored the topic of uncertainty in few-shot settings. In particular, Vedantam et al. (2021) introduces a dataset that highlights compositional reasoning under uncertainty; Zhang et al. (2021a) proposes the task of causal induction, which requires a model to determine if a factor resulted in a subsequent effect; Jiang et al. (2023) introduces a benchmark to assess how machines resolve referential uncertainty when learning new words. Our environment goes beyond simple passive reasoning by allowing fast trials to reduce uncertainty.
IVRE differs from previous works by introducing uncertainty into visual reasoning. Tab. 1 compares IVRE with existing dataset benchmarks. As an interactive environment, IVRE enables agents to gain information actively from the environment to test and revise their hypotheses, fundamentally distinctive from classic visual reasoning tasks such as CLEVR. The causal structure in IVRE is rich and easy for intervention, going beyond the traditional paradigm of passive observation and reasoning. In addition, IVRE also focuses on active exploration and efficiency in reasoning. Notably, while adopting a similar setting with the Bicket experiment in ACRE, IVRE is more than interactive ACRE: by abstracting out the perceptual complexity, IVRE places emphasis on the under-explored topic of uncertainty resolution. An agent needs to induce the hidden relations based on observation and, more importantly, propose interventional trials to collect new information efficiently to test its hypothesis and disentangle confounders in complex phenomena.
3 The IVRE Environment
We build the proposed IVRE under the OpenAI Gym framework (Brockman et al., 2016). As a visual-based interactive platform, IVRE aims at fostering an agent to understand the visual information collected and disentangle the causal factors underlying the observation by actively proposing new experiments to demystify the phenomenon, validate its hypothesis, and correct its belief. Following earlier works, the visual domain of IVRE is consistent with the CLEVR (Johnson et al., 2017) and ACRE (Zhang et al., 2021a) universe, where a Bicket machine sits on a tabletop. Lying upon the Bicket machine are objects with potential Bicketness, which can be inferred from the observation of the machine's activation pattern across frames. The objects' attributes vary in
shape (cube, sphere, or cylinder), material (metal or rubber), and color (gray, red, blue, green, brown, cyan, purple, or yellow). We signal activation of the Blicket machine by lighting it up.
An agent in IVRE is tasked with determining which objects are Bickets. At the start of each episode, the agent is presented with several initial observations of various object combinations (henceforth referred to as context). The context alone is insufficient to solve Bicketness for all objects. Hence, in each following step (henceforth referred to as trials), the agent proposes a new experiment of a specific object combination and updates its belief of Bicketness based on the outcome of experiments.
An episode will be terminated if the agent works out the Bicketness of all objects or consumes all  $T = 10$  time steps. The agent is, therefore, rewarded at each step based on the correctness of its belief, and as a way to encourage efficient exploration, penalized every step it fails the problem. Please refer to Fig. 2 for a simplified example of an episode in the IVRE environment.
In the following, we discuss the designs in detail.
Context To instantiate a problem at each episode, we first sample 9 unique shape-material-color combinations from the pool to create the objects, the Blicketness of which is for the agent to solve. Next, we randomly assign  $n$  objects ( $1 \leqslant n \leqslant 4$ ) to be Blickets. To build a context panel, we randomly pick  $m$  objects ( $1 \leqslant m \leqslant 4$ ) out of the 9 and place them onto a Blicket machine. The status of the Blicket machine can be determined by checking if the  $m$  sampled objects contain a Blicket. We repeat the sampling process 4 times to create a set of context panels as the agent's initial observation.
Trial After observing the context panels, the agent forms an initial belief of Bicketness over all the objects. At each time step  $t$  that follows, the agent observes the outcome of the previous experiment, updates its own belief about the Bickets, and, if uncertainty about object(s) remains, proposes a new experiment to test. There is no limit to the number of objects when proposing trials. This process resembles the active hypothesis testing process and is crucial for discerning related variables.
Observation Space We consider two forms of observation for IVRE: a symbolic version and a pixel version. For the symbol-input version, we use a binary vector to describe the state of the scene, where the first 9 entries represent if the object of interest is present or not and the last entry if the Blicket machine is on or off. For the pixel-input version, we feed the scene description into Blender EEVEE engine (Blender Online Community, 2016) to render images in real-time. For efficiency, we render images of shape  $160 \times 120$ . Notably, the pixel version adds more confounding variables to the problem, whereas a well-defined symbolic version drastically simplifies it. For the pixel-input version, we hypothesize that an agent could handle uncertainty from many possible levels of abstraction; they could be defined on an attribute basis (e.g., color, shape, material), an object basis, or on a group basis. We indicate the total number of Blickets for agents in both environments to improve the naive try-one-by-one strategy.
Action Space IVRE action space is composed of two sub-components: the trial and the belief. The trial component represents the objects selected to perform experiments on in the next trial step for resolving the uncertainty. The belief component denotes the agent's belief of Bicketness after analyzing all experimental results up to the current time step. For agents in IVRE, we soften the binary sub-spaces into continuous values in [0, 1], interpreting each value in the trial sub-space as the probability of selecting that object and that in the belief sub-space as the probability of being a Bicket.
Reward The reward is based on the correctness of its belief and the efficiency of its trial. If the agent figures out the Bicketness for every object within the maximum  $T$  time steps, it will receive a constant reward of 20. For every step that fails the guess, a penalty of -1 will be sent. We also introduce an auxiliary reward based on the partial correctness of its belief. Specifically, at each time step, we first calculate an oracle Bicketness belief based on all the experimental results the agent has observed via search. Next, we use the negative Jensen-Shannon Distance (JSD) between the current and oracle beliefs as the motivating signal. Note that as negative JSD is bounded by  $[-1,0]$ , the reward an agent can receive ranges from -20 (the agent completely fails at each step and consumes all time) to 20 (the agent instantly solves the problem after observing the context).
4 Benchmarking IVRE
We detail two groups of models for the proposed IVRE environment: (i). heuristic methods and (ii). RL methods under the Partially Observable Markov Decision Process (POMDP) formulation. Additionally, we collect human performance with a web-based IVRE environment.
4.1 Heuristic Methods
We consider seven heuristic agents running on the symbol-input version of the IVRE environment.
Random Agent The simple random agent only randomly samples belief and the next trial from a uniform distribution without processing any observation. As a result, neither does the agent generate reasonable belief nor propose any meaningful trials during the interaction.
Bayes Agent Numerous studies have shown that children and adults propose hypotheses to explain causal relationships using Bayesian models (Lucas and Griffiths, 2010; Gopnik, 1996; Gopnik and Sobel, 2000; Gopnik et al., 2001). Therefore, we propose to use a Bayes agent for evaluation as well. Specifically, we implement a Naive Bayes classifier based on the Bernoulli distribution. The classifier predicts the Bicketness for each object via the Bayes' rule using the observation collected up to this time step as training data. The agent then proposes a random trial to gather more information.
Naive Agent Cook et al. (2011) reveal that, to learn a latent causal structure, children without formal science education tend to perform actions that isolate relevant variables in a naive strategy. Following this empirical observation, we implement a naive agent whose trial experiment only contains one object. In the next round, the agent updates its belief for the Bicketness of the object with certainty, and randomly selects an object that has not been tested. Though feasible and certain, this policy results in low efficiency in trials.
NOTEARS NOTEARS (Zheng et al., 2018, 2020) is a score-based continuous optimization algorithm that aims at structure learning of directed acyclic graphs. It can also be used for deriving causal relations (Zhu et al., 2019). Following Zhang et al. (2021a), the causal relation learning process in IVRE can be formulated as an optimization problem and thus learned by NOTEARS. In our NOTEARS implementation, we use a naive policy to propose trials and a nonlinear NOTEARS using MLP to calculate the belief.
Search-based Random Agent We also propose a search-based random agent. The agent is non-parametric in the sense that the agent assumes knowledge of the ground-truth disjunctive causal overhypothesis and, at each time step, searches for all possible Bicket assignments that are consistent with observation up to now. The agent then computes the frequency of the appearance of an object in the set of possible Bicket assignments and randomly selects a set of objects to test.
Search-based Naive Agent The search-based naive agent follows the same design as the search-based random agent in that the prior belief probability is computed in the same way. This naive version uses the probability distribution to select objects for the next round. Specifically, we apply the naive strategy and only select the object with the highest uncertainty to test.
LLMs We also test contemporary LLMs on symbol-input IVRE with GPT-3.5 (gpt-3.5-turbo) (Brown et al., 2020; Ouyang et al., 2022) and GPT-4 (gpt-4-0314) (OpenAI, 2023). As studied in previous research, LLMs demonstrate the ability to understand and reason with the context in which they operate. Specifically, we tame the LLMs using a specific template in a multi-round question-answering format. Please refer to Appx. D for additional details.
4.2 Reinforcement Learning Methods