Interactive Visual Reasoning under Uncertainty
Authors: Manjie Xu
Guangyuan Jiang
Wei Liang
Chi Zhang
Yixin Zhu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: visual reasoning
uncertainty
few-shot
interactive
TL;DR: We introduce the IVRE environment, simulating complex uncertainty scenarios to assess artificial agents' interactive reasoning skills, suggesting a need for advanced research towards achieving human-like intelligence in uncertainty resolution.

Abstract: One of the fundamental cognitive abilities of humans is to quickly resolve uncertainty by generating hypotheses and testing them via active trials. Encountering a novel phenomenon accompanied by ambiguous cause-effect relationships, humans make hypotheses against data, conduct inferences from observation, test their theory via experimentation, and correct the proposition if inconsistency arises. These iterative processes persist until the underlying mechanism becomes clear. In this work, we devise the  **IVRE** (pronounced as *"ivory"*) environment for evaluating artificial agents' reasoning ability under uncertainty. **IVRE** is an interactive environment featuring rich scenarios centered around *Blicket* detection. Agents in **IVRE** are placed into environments with various ambiguous action-effect pairs and asked to determine each object's role. They are encouraged to propose effective and efficient experiments to validate their hypotheses based on observations and actively gather new information. The game ends when all uncertainties are resolved or the maximum number of trials is consumed. By evaluating modern artificial agents in **IVRE**, we notice a clear failure of today's learning methods compared to humans. Such inefficacy in interactive reasoning ability under uncertainty calls for future research in building human-like intelligence.

Interactive Visual Reasoning under Uncertainty
Manjie Xu 1
Guangyuan Jiang
manjietsubbit. edu. cn
jey@stu.pku.. edu. cn
Wei Liang 1,3,5
Chi Zhang 4.B3
Yixin Zhu 2. 2.
siangweibbit.en.
zhangchi@bigai.ad
rixin..zu@phee edu
M. Xu and G. Jiang contributed equally. corresponding authors School of Computer Science & Technology, Beijing Institute of Technology
2 Institute for AI, Peking University
3 Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China
4 National Key Laboratory of General Artificial Intelligence, BIGAI
https://sites. https://sites..oogle.cco
Abstract
One of the fundamental cognitive abilities of humans is quickly resolve uncer- tainty by generating hypotheses and testing them via active trials. Encountering a novel phenomenon accompanied by anbiguous cause-effect relationships, humans make hypotheses against data, conduct inferences from observation, test their the- ory via experimentation, and correct the proposition inconsistency arises. These iterative processes persist until the underlying mechanism becomes clear. In this work, we devise the + IVRE (pronounced as ivory) environment for evaluating artificial agents' reasoning ability under uncertainty. Ahe IVRE is interactive environment featuring rich scenarios centered around Blicket detection. Agents IVRE are placed into environments with various ambiguous action-effect pairs and asked to determine each object's 's role. They are encouraged to propose effective and efficient experiments to validate their hypotheses based observations and actively gather new information. The game ends when all uncertainties are resolved the maximum number of trials is consumed. By evaluating modern artificial agents in t IVRE, we notice a clear failure of today's ' learning methods compared to humans. Such inefficacy in interactive reasoning ability under uncertainty calls for future research in building human-like intelligence.
Introduction
Situated in room, you rarely have clear idea of what specific factor caused sudden lights-out. You might begin to check the light switch, the main circuit breaker, or the light bulb itself. With series of experiments, you can finally realize the actual cause. This is canonical example of reasoning and resolving uncertainty through interaction: when encountering novel scenario, humans typically lack sufficient information and knowledge arrive at definitive conclusion based solely the initial observation. Instead, we formulate hypotheses, subject them to testing, and utilize newly gathered data to address the preceding uncertainty in our reasoning process (Halpern, 2017).
Work done while M. Xu was an intern at Peking University.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Navigating uncertainty through reasoning is distinctive feature of human intellect. This journey, from Hume's explorations of causation (Hume, 1896) contemporary scientific breakthroughs. illustrates humanity's is reliance on formulating hypotheses to actively probe and gather fresh evidence, thereby diminishing ambiguity and carving certainties out of the realms of the unknown (White, 1990; Shanks, 1985). In the current landscape, the field of machine learning has witnessed remarkable advancements, spanning domains from comprehending natural language to deciphering visual stimuli. Yet, the aspiration to emulate human-like reasoning within machines remains an unfulfilled odyssey. The existing capabilities of artificial systems notably diverge from the human, even infantile, cognitive processes used to interpret surroundings, deduce causal connections, and pioneer scientific discoveries (Gopnik, 1996).
In this work, we study the problem of visual rea- soning under uncertainty, which tasks an agent to design new experiments that test hypotheses and discern each variable's causal role. In pur- suit of this goal, we introduce the Interactive Vi- sual Reasoning (4ta IVRE) (pronounced as ivory) environment as interactive testbed. a IVRE is grounded on the Blicket detection setup (Gop- nik and Sobel, 2000; Sobel et al., 2004; Sobel and Kirkham, 2006; Zhang et al., 2021a), ini- tially designed to evaluate children's 's induction ability via passive observation and active tri- als. In series of experiments, children were presented with a Blicket machine, which has a very intuitive working mechanism: whenever Blicket is put on top of it, the device becomes activated, lighting up and playing music. Par- ticipants were shown a series of experiments to demonstrate the Blicketness of set of objects. They were then encouraged to engage in ex- ploratory play with the objects and the machine to determine the Blicketness of each object. Crit- ically, during the exploratory process, children generated and validated different hypotheses un- til they were confident about the properties of each object (Gopnik and Sobel, 2000; Sobel and Kirkham, 2006; Sobel et al., 2004; Walker and Gopnik, 2014) and can even rationally infer causes of failed actions when they are as young as 16-month old (Gweon and Schulz, 201 1). Fig. shows an illustrative example of how participant resolves uncertainty when unraveling how the Blicket machine works and which object is a Blicket.
I know it!
Figure I: Children resolve uncertainty through ex- ploratory play with the Blicket machine and objects. In the beginning. the child is uncertain about which ob- ject can activate the machine from the initial context panel that contains an active machine. S/he procedurally designs new experiments to test hypotheses. When new trial indicates that the machine remains activated when only the yellow cylinder is present, the child knows that the cylinder alone can activate the machine, confirming its Blicketness. However, the red cube requires an addi- tional test for verification.
IVRE is built following a similar interactive setting in the original Blicket experiments. Specifically, we are inspired by recent work in building synthetic reasoning benchmarks (Johnson et al., 2017; 77; Edmonds et al., 2018; Zhang et al., 2019a; Yi et al., 2019; Girdhar and Ramanan, 2019; Zhang et al., 2021a; Xie et al., 2021; Li et al., 2022, 2023; Jiang et al., 2023; Xu et al., 2023) and adopt the CLEVR (Johnson et al., 2017) universe in creating the interactive environment. Following the recent work of ACRE (Zhang et al., 2021a) for causal reasoning, we borrow the object appearances and the Blicket machine setup. In particular, an agent is presented with a few observations of objects on Blicket machines that are either activated or not (referred to as context) at the beginning of each t IVRE episode. Information for identifying which subset of objects are Blickets is incomplete from the context only, thereby introducing uncertainty. To determine the Blicketness of cach object, an agent is tasked to propose trial experiments that could be carried out in each of the upcoming time steps (referred to as trials), and in the meantime, updating its belief over which object is an actual Blicket. The correctness of its belief at each step serves the motivating signal for the agent, who additionally receives a constant penalty for every unsuccessful trial to encourage efficiency.
Serving as testbed, A IVRE evaluate interactive reasoning under uncertainty of today's state-of- the-art artificial agents (Schmidhuber, 2015; Lillicrap et al., 2015; Fujimoto et al., 2018: Mnih et al., 2015; Sutton and Barto, 2018; OpenAL. 2023). Not only do we benchmark Reinforcement Learning (RL) algorithms with visual input from the rendering engine, but also with the ground-trutt -ruth
2

---

Table 1: Comparison between IVRE and other related visual reasoning benchmarks in terms of tasks (classification, visual question answering. and game), sizes (number scenarios), and input formats, the IVRE introduces a spatial-temporal-caaeee reasoning task, which allows intervention and belief update. It aims at sho-shot uncertainty resolution with fast experimentation and reasoning,
Benchmarks
Task Size
Format
Temporal Interactive Uncertainty Few-shot
CLEVR (Johnson ul., 2017)
vqu
100k
image
X
x
x
x
CATER (Girdhar and Ramanan, 2019)
CURI (Vedantam et al., 2021)
CLEVRER (Yi et al., 2019)
vqa
cls
990k
5.5k
20k
image
video
video
/
X
X
x
/
x
V
x
xx
ACRE (Zhang et al., 2021a)
cls 30k
image
X
/
J
Alchemy (Wang et al., 2021)
game
image/symbol
J
J
X
IVRE (Ours)
game
image/symbol
V
V
symbolic representation of the environment, including some additional study with Large Language Models (LLMs). We note that the environment is challenging enough for agents with even the symbolic representation, and visual complexity poses additional challenges. Further comparing the performance of the heuristic algorithms and that of the RL. agents, the prominent failure of today's artificial agents in resolving uncertainty becomes even more evident, calling for future investigation into building intelligence that can learn and reason like people (Lake et al., 2017; Zhu et al., 2020).
To sum up, our work makes the following contributions:
We present the d IVRE platform, unique environment tailored for assessing the proficiency of artificial agents in dynamically resolving uncertainty through interaction. What sets a IVRE apart is the dual challenges it imposes: demanding both logical reasoning and the creation of effective strategies to mitigate uncertainty.
The IVRE setup was meticulously designed to maintain a balance between perceptual simplicity and rich array of visual elements and situational tasks. This environment ushers in novel paradigm of interactive reasoning in the face of uncertainty, compelling agents to engage directly with their conjectures by formulating and executing new experimental trials.
Utilizing the t IVRE framework, we evaluated a spectrum of agents on their ability to navigate the complex problem of interactive reasoning under uncertain conditions. Additionally, our human studies confirmed the human aptitude for managing such tasks. Our observations underscore that (i). visual complexity is not the core difficulty in this task, which echos our design principle minimize visual complexity, (ii). the art of uncertainty reduction within t IVRE hinges on advanced reasoning skills and the strategic implementation of active trials, and (iii). contemporary learning agents are yet to master uncertainty reduction through interactive methods.
2 Related Work
Visual Reasoning A range of visual reasoning and vision-language understanding tasks has been proposed recently. On the basis of a series of Visual Question Answering (VQA) benchmarks (Antol et al., 2015; Krishna et al., 2017; Tapaswi et al., 2016; Zhu et al., 2016), Johnson et al. (2017) use synthetic images depicting simple 3D shapes to scrutinize a suite of VQA models and discover their potential weaknesses. From a causal and physical reasoning perspective, the video dataset of CLEVRER was introduced by Yi al. (2019) to investigate the performance of state-oftthe art (SOTA) models on learning complex spatial-eemporalccasss structures from interacting objects scene. At a more abstract level, model performance in human Intelligence Quotient (IQ) tests has been studied; Barrett et al. (2018) and Zhang et al. (2019a) proposed datasets inspired by the Raven's 's Progressive Matrices (RPM) (Carpenter et al., 1990; Raven and Court, 1938), featuring reasoning on the hidden apatial-tempore transformation from limited number of context panels. Approaches for this abstract reasoning task range from the neural end towards neuro- symoo-smmbolism over the years (Santoro et al., 2017; Hill et al., 2018; Zhang et al., 2019b, 2021b; Wang et al., 2019; Spratley et al., 2020; Zheng et al., 2019; Wu al., 2020), Inspired by Blicket detection and the problem of causal induction (Gopnik and Sobel, 2000; Gopnik et al., 2001), the ACRE dataset (Zhang et al., 2021a) was presented as a way to systematically evaluate current vision systems' capability in causal induction. It is worth noting a visual reasoning method based on object-centric representation and self att-nttention
3

---

Context
Trial
Episode
Figure 2: A simple example episode in the A IVRE environment. At the beginning of the episode, an agent is given a set of context panels (4 in an actual instance) as an introduction to the problem. Next, the agent is motivated to determine which object is a Blicket by proposing new experiments to validate its hypothesis and update its belief. The agent will receive high reward if all Blickets have been found out.
(Ding et al., 2021) has obtained notable performance on various visual reasoning domains, including CLEVRER (Yi al., 2019). CATER (Girdhar and Ramanan, 2019), and ACRE (Zhang et al., 2021a), with the help of unsupervised object segmentation algorithms (Burgess et al., 2019). The inclusion of embodied versions in visual reasoning tasks also marks significant advancement in developing Al systems that can reason within intricate and realistic environments. Some other works (Beattie et al., 2016; Wayne et al., 2018) incorporate tasks designed evaluate the reasoning capabilities of RL agents. Hill et al. (2020) demonstrates that RL agents have the ability to conceptualize new ideas within 3D settings.
While the proposed s IVRE environment based on the Blicket setup and visually similar to the ACRE dataset, the new interactive environment emphasizes drastically different problem: apart from figuring out the hidden causal factors, the agent is also tasked with making maximally meaningful exploration from an initial setup of high uncertainty and aggregating the collected information to update its belief and guide its next decision.
Few-Shot Reasoning Several notable works have contributed to advancing agents' reasoning abilities in few-shot scenarios, where agents learn from small number of examples or instances to solve problems. Popular works include Omniglot (Lake et al., 2015, 2019) and RAVEN (Zhang et al., 2019a). Additionally, other studies have explored the topic of uncertainty few-shot settings, In particular, Vedantam et al. (2021) introduces a dataset that highlights compositional reasoning under uncertainty; Zhang et al. (2021a) Ia) proposes the task of causal induction, which requires a model to determine if factor resulted in subsequent effect; Jiang et al. (2023) introduces a benchmark to assess how machines resolve referential uncertainty when learning new words. Our environment goes beyond simple passive reasoning by allowing fast trials to reduce uncertainty.
IVRE differs from previous works by introducing uncertainty into visual reasoning. Tab. com- pares d IVRE with existing dataset benchmarks. As an interactive environment, + IVRE enables agents to gain information actively from the environment to test and revise their hypotheses, fun- damentally distinctive from classic visual reasoning tasks such CLEVR. The causal structure IVRE is rich and easy for intervention, going beyond the traditional paradigm of passive ob- servation and reasoning. In addition, IVRE also focuses on active exploration and efficiency in reasoning. Notably, while adopting a similar setting with the Blicket experiment in ACRE, IVRE is more than interactive ACRE: by abstracting out the perceptual complexity, IVRE places emphasis on the under-explored topic of uncertainty resolution. An agent needs to induce the hidden relations based observation and, more importantly, propose interventional trials to collect new information efficiently to test its hypothesis and disentangle confounders in complex phenomena.
3 The A IVRE Environment
We build the proposed IVRE under the OpenAl Gym framework (Brockman et al., 2016). As a visual-based interactive platform, tte IVRE aims at fostering agent to understand the visual information collected and disentangle the causal factors underlying the observation by actively proposing new experiments to demystify the phenomenon, validate its hypothesis, and correct its belief. Following earlier works, the visual domain of IVRE is consistent with the CLEVR (Johnson et al., 2017) and ACRE (Zhang et al., 202 1a) universe, where Blicker machine sits on a tabletop. Lying upon the Blicket machine are objects with potential Blicketness, which can inferred from the observation of the machine's activation pattern across frames. The objects' attributes vary in

---

shape (cube, sphere, or cylinder), material (metal or rubber), and color (gray, red, blue, green, brown, cyan, purple, or yellow). We signal activation of the Blicket machine by lighting up.
An agent in TVRE is tasked with determining which objects are Blickets. At the start of each episode, the agent is presented with several initial observations of various object combinations (henceforth referred to as context). The context alone is insufficient to solve Blicketness for all objects. Hence, in each following step (henceforth referred to as trials), the agent proposes new experiment of a specific object combination and updates its belief of Blicketness based on the outcome of experiments.
An episode will be terminated the agent works out the Blicketness of all objects or consumes all T = 10 time steps. The agent is, therefore, rewarded each step based on the correctness of its belief, and as a way to encourage efficient exploration, penalized every step it fails the problem. Please refer to Fig. for simplified example of an episode in the IVRE environment.
In the following, we discuss the designs in detail.
Context To instantiate a problem at each episode, we first sample 9 unique shape-material-colo combinations from the pool create the objects, the Blicketness of which is for the agent to solve. Next, we randomly assign n objects (1 $ n & 4) to be Blickets. To build a context panel, we randomly pick m objects (1 < m < 4) out of the and place them onto a Blicket machine. The status of the Blicket machine can be determined by checking if the m sampled objects contain a Blicket. We repeat the sampling process times to create a set of context panels as the agent's initial observation.
Trial After observing the context panels, the agent forms an initial belief of Blicketness over all the objects. At each time step & that follows, the agent observes the outcome of the previous experiment, updates its own belief about the Blickets, and, if uncertainty about object(s) remains, proposes a new experiment test. There is no limit to the number of objects when proposing trials. This process resembles the active hypothesis testing process and is crucial for discerning related variables.
Observation Space We consider two forms of observation for IVRE: symbolic version and pixel version. For the symbol-input version, we use binary vector to describe the state of the scene, where the first 9 entries represent if the object of interest is present not and the last entry if the Blicket machine is on or off. For the pixel-input version, we feed the scene description into Blender EEVEE engine (Blender Online Community, 2016) to render images in real-time. For efficiency, we render images of shape 160 x 120. Notably, the pixel version adds more confounding variables to the problem, whereas a well-defined symbolic version drastically simplifies it. For the pixel-input version, we hypothesize that an agent could handle uncertainty from many possible levels of abstraction; they could be defined on an attribute basis (e.g., color, shape, material), an object basis, or on a group basis. We indicate the total number of Blickets for agents in both environments to improve the naive try-one-by-ond one strategy.
Action Space IVRE action space is composed of two sub-components: the trial and the belief. The trial component represents the objects selected to perform experiments on in the next trial step for resolving the uncertainty. The belief component denotes the agent's 's belief of Blicketness after analyzing all experimental results up to the current time step. For agents in t I IVRE, we soften the binary sub-spaces into continuous values in [0, IJ, interpreting each value in the trial sub-space as the probability of selecting that object and that in the belief sub-space as the probability of being a Blicket.
Reward The reward is based on the correctness of its belief and the efficiency of its trial. If the agent figures out the Blicketness for every object within the maximum T time steps, it will receive constant reward of 20. For every step that fails the guess, a penalty of -1 will be sent. We also introduce an auxiliary reward based on the partial correctness of its helief. Specifically, at each time step, we first calculate an oracle Blicketness belief based on all the experimental results the agent has observed via search. Next, we use the negative and oracle beliefs as the motivating signal. Note that as negative JSD is bounded by 11 0], the reward an agent can receive ranges from 20 (the agent completely fails at each step and consumes all time) 20 (the agent instantly solves the problem after observing the context).
Distance (JSD) between the current
5

---

4 Benchmarking tha IVRE
We detail two groups of models for the proposed da TVRE environment: (i). heuristic methods and (ii). RI. methods under the Partially Observable Markov Decision Process (POMDP) formulation. Additionally, we collect human performance with a web-based TVRE environment.
4.1 Heuristic Methods
We consider seven heuristic agents running on the symbol-input version of the IVRE environment.
Random Agent The simple random agent only randomly samples belief and the next trial from a uniform distribution without processing any observation. As result, neither does the agent generate reasonable belief nor propose any meaningful trials during the interaction.
Bayes Agent Numerous studies have shown that children and adults propose hypotheses to explain causal relationships using Bayesian models (Lucas and Griffiths, 2010; Gopnik, 1996; Gopnik and Sobel, 2000; Gopnik et al., 2001). Therefore, we propose to use a Bayes agent for evaluation as well. Specifically, we implement a Naive Bayes classifier based on the Bernoulli distribution. The classifier predicts the Blicketness for each object via the Bayes' rule using the observation collected up to this time step as training data. The agent then proposes a random trial to gather more information.
Naive Agent Cook et al. (2011) reveal that, to learn latent causal structure, children without formal science education tend to perform actions that isolate relevant variables in a naive strategy. Following this empirical observation, we implement naive agent whose trial experiment only contains one object. In the next round, the agent updates its belief for the Blicketness of the object with certainty, and randomly selects an object that has not been tested. Though feasible and certain, this policy results in low efficiency in trials.
NOTEARS NOTEARS (Zheng et al., 2018, 2020) is a score-based continuous optimization algo- rithm that aims structure learning of directed acyclic graphs. can also be used for deriving causal relations (Zhu et al., 2019). Following Zhang et al. (2021a), the causal relation learning process in d IVRE can be formulated as an optimization problem and thus learned by NOTEARS. In our NOTEARS implementation. we use naive policy to propose trials and a nonlinear NOTEARS using MLP to calculate the belief.
Search-based Random Agent We also propose a search-based random agent. The agent is non- parametric in the sense that the agent assumes knowledge of the ground-truth disjunctive causal overhypothesis and, at each time step, searches for all possible Blicket assignments that are consistent with observation up now. The agent then computes the frequency of the appearance of an object in the set of possible Blicket assignments and randomly selects a set of objects to test.
Search-based Naive Agent The search-based naive agent follows the same design the search- based random agent in that the prior belief probability is computed in the same way. This naive version uses the probability distribution to select objects for the next round. Specifically, apply the naive strategy and only select the object with the highest uncertainty to test.
LLMs We also test contemporary LLMS on symbol-inpur (gpt-3, 5-turbo) (Brown et al., 2020; Ouyang et al., 2022) and GPT-4 (gpt-4-0314 (OpenAI, 2023). As studied in previous research, LLMMs demonstrate the ability to understand and reason with the context in which they operate. Specifically, we tame the LLMs using a specific template in multi-round question-answering format. Please refer to Appx. D for additional details.
TVRE with GPT-3.5
4.2 Reinforcement Learning Methods
The IVRE environment can be modeled as a POMDP for RL training. Formally, the POMDP problem a tuple (S. A, R, Ω, O,y), where S, A, T, R, are the state space, the action space, the transition probability. the reward function, and the observation space, respectively. Note that the state space covers the ground-truth belief up to each time step l. The action space, the observation space, and the reward function have been discussed in Sec. 3. O is the observation generator for each state-action pair (s, a) and 7 is the discount factor (set to 0.99). As mentioned in Sec. 3, we consider two versions of the observation space: the symbol-input version and the pixcl-input version. For the

---

Triil
Belief
Value
Trai
Bselee
Value
Trai
Delied
Valte
Trail
Beliet
Valse
Policy Natwork
Value Network
Policy Natwork
Value Netwank
Policy Network
Value Network
Poliy Network
Value Natwrik
Feed-formard
Recurrent
CNN
CNN Encodder
CNN Enooder
CNN Encoder
Agent
Enooder
Shared
Environment
Figure 3: The general RL architecture for benchmarking IVRE. The architecture follows the actor actor-critic design; both the policy and value functions are represented with neural networks. We use shared CNN encoder extract visual features for the pixel-input version of the environment depicted here. The symbol-input version differs from the pixel version in providing binary vector description processed by ML...
POMDP formulation of the RL algorithms, we use recurrent agent for learning, the underlying state is not directly observable and has to be inferred from the history of the observation.
The POMDP could also be transformed into an Markov Decision Process (MDP) if we know the underlying state each time step. Therefore, also consider feed-forward architecture for this formulation: if one regards the belief sub-space from the agent's 's output as the true state space, we can use the belief from the previous time step and the new observation propose new trial and update the belief. Note that despite a feed-forward architecture being used, the entire process is still recurrent the belief state's dependency is traced back the history observation. See Fig. 3 for graphical illustration of the general RL architecture only depict the pixel-ing inputt version, where the agent architecture a CNN encoder module for visual perception and a feed-forward module that uses the agent's ' belief as an approximate.
use for benchmarking & IVRE. Here we
As the action space has been softened, we consider popular continuous RL methods for evaluation. Specifically, we benchmark Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015), Twin Delayed Deep Deterministic Policy Gradient (TD-3) (Fujimoto al., 2018), and Proximal Policy Optimization (PPO) (Schulman et al., 2017). We also test an agent that incorporates NOTEARS (Zheng al., 2018, 2020) as its reasoning component. Please refer Appx. B for details.
DDPG DDPG is designed based on the successful Deep Q-Learning (Mnih et al., 2015) and two Multi-
extended the continuous action domain. For the symbol-input DDPG model, Layer Perceptron (MLP) with three hidden layers of width 512 and ReLU activation as the actor and critic network, respectively. For the pixel-input version, pre-trained ResNet-118 (He et al., 2016) to process image panels and concatenate it with the agent's belief from the previous time step input to the actor and critic network. We add an additional MILP and an LSTM layer with 384 units to process the raw history observation the recurrent agent.
ImageNet (Deng al., 2009)
TD-3 Compared DDPG, TD-3 notices the problem of the overestimated value function in the actor-critic setting and suggests a new mechanism in mitigating this effect. In our TD-3 implemen- tation, a backbone similar to DDPG: three MLP with three hidden layers of width 512 and ReLU activation are used for the actor and two critic networks, respectively. A similar adaptation in the DDPG setup is made for the recurrent agent in the TD-3 implementation.
PPO RL tasks, PPO is widely used as an online learning baseline. based on policy optimiza- tion methods and generally has trust-region methods' stability and reliability. Similar to the two RL. agents mentioned above, we adopted the same three-layer MI.P for its backbone.
4.3 Human Baseline
We recruited 54 participants from Peking University to take part in the study, which was conducted through a web-based t IVRE platform (see Fig. A8). Each participant was compensated with course credits upon the completion of an episode. The episodes were randomly allocated to the participants, who were then tasked with solving the IVRE challenge maximum steps, without any time restrictions.

---

Table 2: Left: Performance of heuristic models the symbol-input version of IVRE enviromment. Right: Performance RL models humans on ta IVRE environment. report two evaluation metrics: the problem-solving accuracy, denoted as Acc, and the total reward, denoted as R. The context column records results when the agent only observes the context panels, whereas the episode column after the entire episode (FF: feed-forwate Re: recurrent, V: pixel-inpupe
Context
Episode
Context
Episode
Model
Model
Acc
R
Acc
R
Acc
R
Acc
R
Random
0.86%
-5.42
1.87%
-14.14
DDPG-FF
22.47%
-0.17
32.47%
-3.70
Bayes
15.60%
-2.98
43.03%
-3.78
DDPG-Re
13.55%
-2.03
46.03%
-0.29
Naive
3,50%
-3.91
43.62%
-1.69
TD-3-Re
12.57%
-2.40
36.83%
-2.71
NOTEARS
9.10%
-4.66
12.70%
-13.02
TD-3-FF
21.91%
-0.42
30.05%
-4.48
Scarch-Naive
1.51%
-3.68
83.80%
9.39
PPO
6.87%
-3.87
28.56%
-5.85
Search-Random
1.80%
-3.62
34.15%
-1.87
DDPG-V
0.35%
-5.02
0.72%
-13.40
GPT-3.5
3%
-5.91
11%
-13.99
TD-3-V
0.27%
-5.04
0.31%
-13.51
GPT-4
10%
-3.36
26%
-7.88
Human
33.33%
5.01
98.15%
12,70
5
Experiment
5.1 Experimental Setup
We run experiments on the th IVRE environment with the agents and their aforementioned variants. For evaluation metrics, report the agent's 's average reward together with problem-solvin accuracy over 10* random test episodes (except LLMs 10²). The average reward measures how the model performs in terms of reasoning and trial efficiency, while the problem-solving accuracy is computed by counting the number of episodes where an agent correctly figures out Blicketness for all objects. Apart from the final results after finishing entire episode, also evaluate how the agent performs after observing the initial context panels only: metric on how the model understands the problem without any trials.
5.2 Performance of Agents
Heuristic Agents Tab. 2 shows the performance of the heuristic agents the proposed ta IVRE environment. In general, we note that the more information collected, the better the agents resolve the uncertainty: the low accuracy after the context panels also verifies that additional trials are necessary for solving the entire problem. The Random agent fails in this task unsurprisingly, while the Search- Naive agent reaches more closely to the human performance. The comparison among the random agent, the naive agent, and their search-based variants indicates that both the reasoning component and the exploration strategy are beneficial for a symbolic approach. Without the reasoning component, the naive agent does not build reliable belief which object is more likely be Blicket; without the cxploration strategy, an agent only randomly collects new experiments, doing little help in demystifying the phenomenon. LLM agents, equipped with priors learned from large-scale text corpora, exhibit a notable level of reasoning ability to reduce uncertainty, yet still far from human performance.
Symbol-Input RL Agents Tab. summarises the performance of symbol-input RL agents. RL agents performed significantly better than the Random agent, where DDPG-Re gets the highest reward. Recurrent agents generally perform better than feed-forward agents, indicating that history observation and trials play role in generating valid hypotheses and proposing new trials. One interesting observation from the performance of these RL agents is that they generally propose very inefficient trials: the final reward decreases time goes by. However, it becomes more likely for an agent to stumble on a solution by chance.
Pixel-Input RL Agents We consider DDPG and TD-3 with the pixel-input experiments. As shown Tab. 2, all the models tested in the pixel-input version t IVRE catastrophically fail with random- level problem-solving accuracy and total reward. Surprisingly, while their rewards are slightly higher than the random agent's, their problem-solving accuracy even worse, let alone the conspicuously large gap from the performance under the symbol-input circumstance. We carefully checked the output of these agents and found that belief and trial predicted by agents oscillated around 0.5, indicating that they had difficulty learning to understand the a IVRE environment from pixel-level
8

---

131
93
Sxpt
Stops
Skip
Figure 4: Left and Middle: Distribution of the steps agent takes to successfully solve an L IVRE episode. Left: Search-based Naive agent. Middle: DDPG-RO Re agent. Right: Box plot of the number of objects the DDPG-Re agent proposes at each time step.
input. Our results suggest that current models should be further improved in representation learning help interactive reasoning under uncertainty.
Comparison Comparing experimental results in heuristic agents, symbol-input RL agents, and pixel-input RL agents, we note that uncertainty reduction challenging not only in reasoning but also in proposing valid trials. Our heuristic results reveal that the lack of either capability can lead to failure in tasks like a IVRE, which are quite common in the real world. For almost all of the RL-learned policies. they have difficulty effectively proposing meaningful experiments to validate and update their belief. The heuristic method with the best performance designed based on human prior and equipped with a fixed and naive policy, far from the talent shown by human beings when performing this task. Even LLMs with vast prior knowledge still fall short.
5.3 Analysis
The naive trial policy achieves significantly better results compared to the random trial policy with oracle belief. Several factors contribute to the performance of the Search-Naive agent. First, the agent operates under the assumption that the Blicker machine functions as an OR machine (the disjointive causal overhypothesis), meaning will activate if at least one object is a Blicket. Second, although not entirely accurate, the agent uses correlation correct outcomes in certain situations. Third, the agent tests for "Blicketness" one object at a time, which, while not the most efficient approach, helps to isolate other confounding variables. These findings suggest that solving the IVRE problem requires more than random testing; it calls for thoughtful selection of trials.
proxy for causality, which may yield partially
Moreover, the visual complexity adds additional challenges to the problem for agents, whereas humans are used to reasoning and refining their hypotheses from visual stimuli. Looking into the failure in the pixel-input agents, we hypothesize that causal representation learning could mitigate its learning inefficacy: not only do we need features to distinguish between different experiments and those that support an in-depth understanding of the environment.
Analyzing the patterns and outcomes of human participants, we observe that they exhibit strong capabilities in reasoning and exploration. They employ a versatile approach to experimentation and demonstrate effective reasoning even within constrained scenarios. For further insights, refer to the Appx. E detailed set of examples provided.
Distribution of Steps The symbolic version of a IVRE is considered diagnostic tool and an upper bound for pixel a IVRE. With pixel-input RL agents reaching only random-level performance, we analyze the best heuristic model (i.e., Search-based Naive agent) and the best symbol-input RI. model ei.e., DDPG-Re). Fig. shows the distribution of steps the agents take to successfully solve problem and the number of objects proposed at each time step in the 10 random test episodes. We also plot the number of objects the DDPG-Re agent proposes at each time step. The search-based method makes reasonable trials with the naive strategy at each step, and as the information collected amounts, more episodes are solved. RL agents also have learned specific exploration strategies to reduce uncertainty, although not as perfect as humans: more flexible and diverse actions are observed Steps 5-7, and more episodes are solved in these steps. On the other hand, DDPG-Re agent might have learned data bias as it directly solves certain number of episodes without interaction. For its trial policy, the DDPG-Re agent consistently selects around 3 to objects to test at earlier steps in the process and picks fewer objects towards the end. Combining the analysis, find the agent in

---

the early steps learns to use mixed strategy even less effectively than the naive trial, showing very limited ability actively reasoning under uncertainty.
Conclusion and Discussion
In this work, introduce IVRE, interactive testbed for evaluating artificial agents' reasoning ability under uncertainty. Inspired by the theoretical proposition and the empirical observation of infants, the newly introduced t IVRE mimics the classic Blicket detection experiment but intentionally simplifies the sensorimotor control by abstracting it out into a discrete space of object selection. t IVRE's design not only requires the agent to effectively update its belief based on the information collected so far but also necessitates the capability to come up with maximally efficient new experiments to disentangle confounding factors.
Measuring performance and concluding this manuscript is not the end but rather the beginning of our pursuit of an intelligent agent who can learn and think like people when handling uncertainty during the interaction. With today's agents catastrophically failing in this problem, how do humans, very young children, successfully resolve uncertainty in the world around them without specific training? What role does training from nurture play in the process? And how to incubate an artificial agent with this ability through interaction with the environment? With many questions unanswered, we hope this preliminary work will motivate further research.
Societal Implication We have not identified any negative societal implications arising from the proposed benchmark. On the contrary, the act of uncertainty resolution within IVRE necessitates robust reasoning capabilities, contingent on effective intervention strategies. It is noteworthy that contemporary learning agents still struggle identifying interconnected variables through interactive engagement. We believe A development of agents exhibiting human-level intelligence.
TVRE has the capacity to make positive contribution towards the
Limitations and Future Work To highlight reasoning and uncertainty reduction, we design IVRE with synthetic instead of real-world scenarios. Given the challenges associated with visual perception and action in the real world, additional research is required to investigate how agents can effectively address uncertainty within more complex environment. IVRE also employs relatively simple causal structures to create uncertainty. This calls for further research to study how machines and humans can actively resolve uncertainties from different levels various causal structures.
Acknowledgement We thank Liangru Xiang for helpful discussions, Ms. Zhen Chen (BIGAT) for designing the figures, and NVIDIA for their generous support of GPUs and hardware. M.X., G.J. W.L., C.Z., and Y.Z. are supported in part by the National Key R&D Program of China (2022ZD01 14900). M.X. and W.L. are supported in part by the NSFC (62172043), and Y.Z. is part by the Beijing Nova Program.
10

---

References
Antol, S., Agrawal, A., Lu. J.. Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. (2015). VQA: Visual question answering. In International Conference on Computer Vision (ICCV). 3
Barrett, D., Hill, F., Santoro, A., Morcos, A., and Lillicrap, T. (2018). Measuring abstract reasoning in neural networks. In International Conference on Machine Learning (ICML).
Beattic, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés,
V., Sadik, A., et al. (2016). Deepmind lab, arXiv preprint arXiv:1612.03800
Blender Online Community (2016). Blender-a 3d modelling and rendering package. 5, A2
Brockman, G., Cheung, V., Pettersson, L., Schncider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAl
Gym. arXiv preprint arXiv:
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Burgess, C. P. Matthey, I.., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A. (2019). MONet: Unsupervised scene decomposition and representation. arXiv preprint a 1901. 4
Carpenter, P. A., Just, M. A., and Shell, P. (1990). What one intelligence test measures: a theorctical account of the processing in the raven progressive matrices test. Psychological Review, 97(3):404.
Cook, C., Goodman, N. D., and Schulz, L. E. (2011). Where science starts: Spontancous experiments in preschoolers' exploratory play. Cognition, 1203))341-349, 6
Deng, J., Dong, W., Socher, R., Li, L.JJ., Li, K., and Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR). 7
Ding, D., Hill, F., Santoro, A., Reynolds, M., and Botvinick, M. (2021). Attention over learned object embeddings enables complex visual reasoning. In Advances Neural Information Processing Systems (NeurlPS). 4
Edmonds, M., Kubricht, J., Summers, C., Zhu, Y., Rothrock, B., 7hu, S.-C., and Lu, H. (2018). Human causal transfer: Challenges for deep reinforcement learning. In Annual Meeting of the Cognitive Science Society (CogSci).
Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML). 2,7
Gebru, T., Morgenstern, J.. Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., and Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12))8699) 99. A3
Girdhar, R. and Ramanan, D. (2019). CATER: A diagnostic dataset for compositional actions & temporal reasoning. In International Conference on Learning Representations (ICLR). 2, 3,
Gopnik, A. (1996). The scientist child. Philosophy of Science, 63(4):485-514. 2,
Gopnik, A. and Sobel, D. M. (2000). Detecting blickets: How young children use information about novel causal powers in categorization and induction. Child Development, (115):1205-222 3,
Gopnik, A.. Sobel, D. M., Schulz, L. E., and Glymour, C. (2001). Causal learning mechanisms in very young children: two-, three-, and four-year-olds infer causal relations from patterns of variation and covariation. Developmental Psychology. 37(5):220. 3,
Gweon, H. and Schulz, L. (2011). 16-month-olds rationally infer causes of failed actions. Science, 332(6037):1524-122
Halpern, J. Y. (2017). Reasoning about uncertainty. MIT press.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR). A2
Hill. F., Santoro. A., Barrttt, D.,, Morcos, A., and Lillicrap. T. (2018). Learning to make analogies by contrasting abstract relational structure. In International Conference Learning Representations (ICLR). 3
Hill, F. Tieleman, O., von Glehn, T., Wong, N., Merzic, H., and Clark, S. (2020). Grounded language learning fast and slow. In International Conference on Learning Representations (ICLR).

---

Hume, D. (1896). A treatise of human nature. Clarendon Press. 2
Jiang. G., Xu, M., Xin, S., Liang. Peng. Y., Zhang, C., and Zhu, Y. (2023). MEWL: Few-shot multimodal word learning with referential uncertainty. In International Conference on Machine Learning (ICML). 2,
Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitmick, C., and Girshick, R. (2017). CLEVR: A diagnostic datasct for compositional language and elementary visual reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR). 2, 3,4
Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv: 112.6944..6698...
A1
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J.. Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017). Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV). 123(1):32-73.3
Lake, B. M., Salakhutdinov, R., and Tenenbaum, B. (2015), Hluman-leve concept learning through probabilistic program induction. Science, 350(6266); 1332-13333 4
Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. (2019). The omniglot challenge: 3-year progress report. Current Opinion in Behuviorul Sciences, 29:97-104. 4
Lake, B. M., Ullman. T. D.. Tenenbaum, J. B., and Gershman, S. J. (2017). Building machines that learn and think like people. Behavioral and bruin sciences, 40. 3
Li, Q.. Huang, S., Hong, Y., Zhu, Y., Wu, Y. N., and Zhu, S.-C. (2023). A minimalist dataset for systematic generalization of perception, syntax, and semantics. In International Conference on Learning Representations (ICLR). 2
Li, Q. Zhu, Y., Liang, Y., Wu, N., Zhu, S.-C., and Huang, S. (2022). Neural-symbolic recursive machine for systematic generalization. arXiv preprint arXiv:2210.01603. 2
Lillicrap, P., Hunt, J. J.. Pritzel, A., Heess. N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015).
Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02977 2,7
Lucas, C. G. and Griffiths, T. L. (2010). Learning the form of causal relationships using hierarchical bayesian models. Cognitive Science, 34(1):113-144
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540):29-5333 2,
OpenAl (2023). GPT-4 technical report. 2,
Ouyang. L.,, Wu, J., Jiang. X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NewrIPS), 35:27730-27744, 6
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., Veto,, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017). Automatic differentiation in pytorch. In NeurlPS Autodiff Workshop. Al
Raven, J. C. and Court, J. (1938). Raven's progressive matrices. Western Psychological Services Los Angeles, CA. 3
Santoro, A., Raposo, D., Barrett D. G., Malinowski, M., Pascanu, R., Battaglia, P., and Lillicrap. T. (2017).
A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems (NeurIPS). 3
Schmidhuber, J. (2015), Deep learning in neural networks: An overview. Neural Networks, 61:85-117. 2
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint /rX7,:/707.066
Shanks, D. R. (1985). Hume on the perception of causality. Hume Studies, 11(1):94-108.
Sobel. D. M. and Kirkham, N. Z. (2006). Blickets and babies: the development of causal reasoning in toddlers and infants. Developmental Psychology, 42(6):1103.
Sobel, D. M.. Tenenbaum, J. B.. and Gopnik, A. (2004). Children's causal inferences from indirect evidence:
Backwards blocking and bayesian reasoning in preschoolers. Cognitive Ccience, 28(3):303- 2
12

---

Spratley, S., Ehinger, K., and Miller, T. (2020). A closer look at generalisation in raven. In European Conference on Computer Vision (ECCV).
Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press. 2
Tapaswi, M., Zhu, Y., Stiefelhhag, R., Torralba, A., Urtasun, R., and Fidler, S. (2016). Movieqa: Understanding
stories in movies through question-answering. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
Vedantam, R., Szlam, A., Nickel, M., Morcos, A., and Lake, B. M. (2021). CURI: A benchmark for productive concept learning under uncertainty. In International Conference on Machine Learning, pages 10519-10529. PMLR. 3,
Walker, C. M. and Gopnik, A. (2014). Toddlers infer higher-order relational principles in causal learning. Psychological Science, 25(1):161-16. 2
Wang. D., Jamnik, M., and Lio. P. (2019). Abstract diagrammatic reasoning with multiplex graph networks. In International Conference on Learning Representations (ICLR).
Wang X., King, M., Porcel, N. M., Surth-Nelson,, Z., Zhu, T., Deck, C., Choy, P., Cassin, M., Reynolds, M., Song, H. F., et al. (2021). Alchemy: A benchmark and analysis toolkit for meta-reinforremmnt learning agents. In Advances in Neural Information Processing Systems (NeurlPS). 3
Wayne, G., Hung, C.-C., Amos, D., Mirza, M., Ahuja. A., J. Z.. Santoro, A., et al. (2018). Unsupervised predictive memory in goal-directed agent. arXiv preprint arXiv: 10760. 4
A., Rae, J., Mirowski, P., Leibo,
Weng, J. Chen, H., Yan, D., You, K., Dubureq, A., Zhang, M., Su, Y., Su, H., and Zhu, J. (2022). Tianshou: A highly modularized deep reinforcement learning library. The Journal of Machine Learning Research, 23(1):12775-1223 12280.. Al
White, P. A. (1990). Ideas about causation in philosophy and psychology. Psychological bulletin, 108(1):3. 2
Wu, Y., Dong, H., Grosse, R., and Ba, (2020). The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning. arXiv preprint arXiv:2007.04212
Xic, S., Ma, X., Yu, P., Zhu, Y., Wu, Y. N., and Zhu, S.-C. (2021). HALMA: Humanlike abstraction learning meets affordance in rapid problem solving. In ICLR Workshop on Generalization beyond the training distribution in bruins and machines. 2
Xu, M.. Jiang, G., Liang, W., Zhang, C., and Zhu, Y. (2023). Conan: Active reasoning in an open-we open-world environment. In Advances in Neural Information Processing Systems (NeurIPS). 2
Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A., and Tenenbaum, (2019). CLEVRER: Collision events for video representation and reasoning. In International Conference on Learning Representations (ICLR). 2, 3,4 4
Zhang, C., Gao, F., Jia, B., Zhu, Y., and Zhu, S.-C. (2019a). Raven: A dataset for relational and analogical visual
reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR). 2, 3, 4
Zhang, C., Jia, B., Edmonds, M., Zhu, S.-C., and Zhu, (2021a). ACRE: Abstract causal reasoning beyond
covariation. In Conference on Computer Vision and Pattern Recognition (CVPR). 2, 3, 4, 6
Zhang, C., Jia, B., Gao, F., Zhu, Y., Lu, H., and Zhu, S.-C. (2019b), Learning perceptual inference by contrasting.
Advances Neural Information Processing Systems (NeurIPS). 3
Zhang, C., Jia, B., Zhu, S.-C., and Zhu, Y. (2021b). Abstract spatial-temporal reasoning via probabilistic abduction and exccution. In Conference on Computer Vision and Pattern Recognition (CVPR).
Zheng, K., Zha, Z.-J., and Wei, W. (2019). Abstract reasoning with distracting features. Advances in Neural Information Processing Systems (NeurIPS). 3
Zheng, X., Aragam, B., Ravikumar, P., and Xing. E. P. (2018). Dags with tears: continuous optimization for structure learning. In Advances in Neural Information Processing Systems (NeurIPS). 6,
Zheng, X., Dan. C., Aragam. B., Ravikumar, P., and Xing, E. (2020). Learning sparse nonparametric dags. In International Conference on Artificial Intelligence and Statistics (AISTATS). 6,7
Zhu, S. Ng. I., and Chen, Z. (2019). Causal discovery with reinforcement learning. In International Conference on Learning Representations (ICLR).
13

---

Zhu, Y.. Gao, T., Fan, L., Huang. S., Edmonds, M., Liu, H., Gao, F., Zhang, C.. Qi. S., Wu, Y. N., et al. (2020). Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common Engineering. 603):10-345.2 3
Zhu, Y. Groth, O., Bermstein, M., and Fei-Fei. L. (2016). Visual7w: Grounded question answering images. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
14

---

A Ablation with Fewer Objects
Table AI: Performance of models the symbolic version of aa IVRE with fewer objects and blickets. We conduct experiments where most blickets are selected from unique objects ta IVRE. Results show that reducing the number of objects in IVRE would simplify the environment.
Model
Random
Bayes
Naive
Search-Naive
DDPG-FF
TD-3-FF
Acc 7.61
71.92
94.56
100.00
54.53
90.08
88.64
R
-12.11
4.73
10.82
14.32
3.17
11.96
11.53
Striking balance between simplicity and complexity is crucial to maintaining the t IVRE's ability to assess agents' reasoning abilities effectively. We conduct experiments where at most blickets selected from 7 different objects in a IVRE. The results are shown in Tab. Al. As demonstrated, reducing the number of objects in ea IVRE would simplify the environment. However, this approach could inadvertently lead shorteuts, random naive trials might efficiently address much of the uncertainty.
B
Agent Details
B.1
RL Agent Details
All RL models are implemented in PyTorch (Paszke et al., 2017) under the helper library of Tianshou (Weng et al., 2022). When training the ML. and the LSTT backhones, we use the Adam optimizer (Kingma and Ba, 2014) with learning rate of 3 x 10-4 The DDPG and the TD>33 agent share the same hyper-parameters. parameters: we set the exploration noise to 0.1 and the target network's 's soft update to 0.005. Other parameters have been set using the default parameters from the original work. All RL models were trained for 10" steps during training. The best model from training was saved and used during the evaluation phase. All the experiments reported herein were NVIDIA GeForce RTX 3090 or Tesla V100 graphics cards.
C Model Details
C.1 Symbol-Input Backhone
Tab. A2 shows the feed-forward architecture we use for the RL agents in the symbol-input version of the environment, where the parameter of the Linear layer denotes the size of its output and m is set equal to the size of the action space. Of note, the DDPG agent uses actor network and critic network, while the TD-3 agent uses one actor network and two critic networks. The actor network and the critic network share the same architecture. Note that the feed-forward architecture takes in both the current observation and the belief from the previous time step. Our online learning baseline PPO shares the same three-layer MLP for its backbone.
The recurrent architecture of the RL agent recruits the network components listed in Tab. A3, where we explicitly use single-layer LSTM with a hidden size of 512 units.
Table A2: Feed-forward architecture RL agents.
Table A3: Recurrent architecture of RL agents.
Layer
Units
Layer
Units
Linear 512
LSTM 512
ReLU /
Linear 512
Linear 512
ReLU /
ReLU /
Linear m
Linear 512
ReLU /
Linear m
A1

---

C.2 Pixel-Input Backbone
The pixel-input model leverages pre-trained ResNet-18 (He et al., 2016) for extracting image features. Specifically, we resize each image rendered from the Blender EEVEE engine (Blender Online Community, 2016) into 224 224 and feed it through the backbone, The features are then passed through the same actor network and the critic network listed in Tabs. A2 and A3.
D LLMs Experiments
We use the OpenAl Chat Completions API for our LLMs experiments; see examples in Tab. A4.
Table A4: An example dialogue generated between GPT and our d IVRE (the User).
Role
Content
User
"There nine objects: 1, 2, 3, 5, 6, 7,8, 9 that be put on machine. Up to four of all the objects have a unique property that can activate the machine.
Your goal find out all the objects that can activate the machine in rounds. All the following questions in the format of '1, 2, and 3."
User
"Round O:
are on the machine, and the machine not activated.
Question: Which objects can activate the machine?" "Answer: From 1-9, objects that can activate the machine are" "2 3, 4, 6,
Assistant
User
Your answer wrong."
User
"Round 1:
3 on the machine, and the machine is activated. Question: Which objects can activate the machine? Answer: From 1-9, objects that can activate the machine are" "2. 4, 5, 6, 7,"
Assistant
User
"Yorr answer wrong."
User
"Round 2:
1, 6,6,8 8 are the machine, and the machine is not activated.
Question: Which objects can activate the machine?
Answer: From 1-9, objects that can activate the machine
Assistant
"2, 3,4, 7."
User
"Your answer is wrong."
User
"Round 3:
3, 4, 7, 9 are the machine, and the machine is activated.
Question: Which objects can activate the machine?
Answer: From 1-9, objects that can activate the machine are
Assistant
"3, 4, 7, and 9."
User
"Your answer is wrong."
User
"Round 4:
None are on the machine, and the machine is not activated. Question: Which objects can activate the machine?
Answer: From 1-9, objects that can activate the machine are
Assistant
"Please provide the objects on the machine in Round for me to give..."
User
"Now design your own trial, you can put some of the objects on the machine to validate your hypothesis. Question: Which objects will you put on the machine? Answer: From 1-9, I will put" "2, 5, and 6 on the machine."
Assistant
User
"Your answer is wrong."
User
"Round 5:
2, 5, 6 are on the machine, and the machine is not activated.
Question: Which objects can activate the machine?
Answer: From 1-9, objects that can activate the machine are
A2

---

Assistant User
"3, 4, 7, and 9."
"Now design your own trial. you can put some of the objects on the machine validate your hypothesis. Question: Which objects will you put on the machine? Answer: From 1-9, I will put
Assistant
"3 and 4 on the machine."
User
Your answer is wrong."
User
Round 6:
3, 4 are on the machine, and the machine not activated.
Question: Which objects can activate the machine?
Answer: From 1-9, objects that can activate the machine
Assistant
"7 and 9."
User
"Now design your own trial, you can put some of the objects on the machine validate your hypothesis. Question: Which objects will you put on the machine? Answer: From 1-9, I will put" "7 on the machine."
Assistant
User
"Your answer wrong."
User
'Round" 7:
7 the machine, and the machine is not activated.
Question: Which objects can activate the machine?
Answer: From 1-9, objects that can activate the machine
Assistant
"9."
E
Case Study
We give examples of interaction replay from different agents in IVRE in Figs. Al A7. In each case, the first two figures show available objects in episode, where the first one contains Blickets and the second one non-Blickets. For the naive agent, it always proposes trial composed of single object (see Fig, A1), and from its result of activation, the agent deterministically assigns Blicketness to an object. A search-base naive agent proposes trials based on the confidence every object. Figs. A3 and A4 show trials from such an agent. In Fig. A3, the agent is quite sure of the non-Blicketness of the purple and green metal cube after the context panels, so tests other objects. However, it cannot figure out the Blicketness of the green metal ball through all of the contexts, leading to its failure.
The DDPG agent, however, has unique behavior. In Fig. A6, all the Blicket machines in initial contexts are activated, so the agent has no chance to generate correct belief from the context only, except guessing. However, the DDPG agent makes repetitive trials, making harder to gain effective information solve the problem. In follow-up experiments, the DDPG agent also solves tasks when most objects non-Blickets But it often fails when there more Blickets, which means it cannot handle situations with high correlation.
Humans show powerful abilities in both reasoning and exploration. Fig. A7 shows an example where a successful trial is achieved. We can see that flexible trial policy is adopted, and humans can reason effectively given limited contexts.
F
Data Documentation
We follow the datasheet proposed in Gebru et al. (2021) for documenting our proposed benchmark:
1. Motivation
(a) For what purpose was the benchmark created? The benchmark was created as an environment for evaluating artificial agents' reasoning ability under uncertainty, aa IVRE an interactive environment featuring rich scenarios centered around Blicket detection. Agents in A IVRE are placed into environments with various ambiguous action-effect pairs and asked to figure out each object's role. Agents encouraged to propose effective and efficient experiments validate their
A3

---

hypotheses based on observations and gather more information. The game ends when all uncertainties are resolved the maximum number of trials consumed.
(b) Who created the dataset and on behalf of which entity?
This dataset was created by Manjie Xu, Guangyuan Jiang, Wei Liang, Chi Zhang and Yixin Zhu. They are from Beijing Institute of Technology (Manjie Xu, Wei Liang), Peking University (Guangyuan Jiang, Yixin Zhu) and National Key Laboratory of General Artificial Intelligence, BIGAI (Chi Zhang).
(c) Who funded the creation of the dataset?
M.X., G.J., W.L., C.Z., and Y.Z. are supported in part by the National Key R&D Program of China (2022ZD01 14900), M.X. and W.L. supported in part by the NSFC (62172043), and Y.Z. is in part by the Beijing Nova Program.
(d) Any other Comments?
None.
2. Composition
(a) What do the instances that comprise the benchmark represent? The benchmark contains episodes in which agents are tasked to figure out which objects are Blickets.
(b) How many instances are there in total?
N/A. Each episode in t IVRE is randomly sampled and can have infinite tasks.
(c) Does the dataset contain all possible instances or a sample (not necessarily random) of instances from larger set?
TVRE contains all possible instances.
(d) What data does each instance consist of?
In each episode, an agent is presented with several initial observations of various object combinations (referred to as context). The context alone is insufficient to solve Blicketness for all objects. Hence, in each following step (referred to as trials), the agent proposes
experiment of specific object combination and updates its belief of Blicketness based on the outcome of experiments.
(e) Is there label or target associated with each instance? Yes.
(f) Is any information missing from individual instances?
No.
g Are relationships between individual instances made explicit?
Yes.
(h) Are there recommended data splits?
No.
ii) Are there any errors, sources of noise, redundancies in the benchmark?
No.
Is the benchmark self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? Self-contained.
(k) Does the benchmark contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)?
No.
()) Does the benchmark contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
No.
(m) Does the benchmark relate people?
No.
(n) Does the benchmark identify any subpopulations (e.g., by age, gender)?
No.
(0) Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e,, combination with other data) from the dataset?
No.
A4

---

(p) Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? No.
(q) Any other comments?
None.
3. Collection Process
(a) How was the data associated with each instance acquired? We render them using Blender.
(b) What mechanisms or procedures were used to collect the data (e-g., hardware apparatus sensor, manual human curation, software program, software API)?
In each episode, Ah IVRE will randomly sample objects and blickets from a pool.
(c) If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities? N/A.
(d) Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g.. how much were crowdworkers paid)?
Manjie Xu and Guangyuan Jiang wrote the generation code.
(e) Over what timeframe was the data collected? N/A.
(f)
Were any ethical review processes conducted (e.g., by an institutional review board)?
The dataset raises no ethical concerns.
g Does the dataset relate to people?
No.
Did you collect the data from the individuals question directly, or obtain via third parties or other sources (e.g., websites)?
N/A.
(i) Were the individuals in question notified about the data collection? N/A.
(j) Did the individuals in question consent to the collection and use of their data? N/A.
(k) If consent obtained, were the consenting individuals provided with mecha- nism to revoke their consent in the future or for certain uses?
N/A.
Has an analysis of the potential impact the dataset and its use on data subjects (e.g., data protection impact analysis) been conducted? Yes.
(m)
Any other comments?
None.
4. Preprocessing, Cleaning and Labeling
(a) Was any preprocesing/cceaainggaabae of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?
N/A.
(b) Was the "raw" data saved in addition to the preproceseelcceemec//hbeeeel data (e.g., to support unanticipated future uses)? N/A.
(c) Is the software used to preprocess/cean//abel the instances available? N/A.
(d) Any other comments?
None.
A5

---

5. Uses
(a) Has the dataset been used for any tasks already?
No, the dataset newly proposed by us.
(b) Is there repository that links to any or all papers systems that the dataset?
Yes, we provide the link to all related information on our project website.
(c) What (other) tasks could the dataset be used for? This dataset could be used for other reserach topics like causal discovery, causal reasoning and active learning.
(d) Is there anything about the composition of the dataset or the way it was collected and preppoceseed/leenndlaabee that might impact future uses? N/A.
(c)
Are there tasks for which the dataset should not be used?
N/A.
(f) Any other comments?
None.
6. Distribution
(a) Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
No.
(b) How will the dataset be distributed (e.g., tarball on website, API, GitHub)? I VRE could be accessed on project website.
(c)
When will the dataset be distributed?
VRE has already been released.
(d) Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
We release our benchmark under CC BY-NC license.
(e) Have any third parties imposed IP-based or other restrictions on the data associ- ated with the instances? No.
(f) Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?
No.
(g) Any other comments? None.
7. Maintenance
(a) Who is supportigghnsttng/mmntiiiii the dataset? Manjie Xu and Guangyuan Jiang are maintaining.
(b) How the owner/ccurator/manaeer of the dataset be contacted (e.g., email address)? man wwe.jstiitttt..mm edu. en, jgy@stu. pwtpssu..pu.... edu.
(c) there an erratum?
Future erratum will be released through the website.
(d) Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances")?
Yes.
(e) If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals question told that their data would be retained for fixed period of time and then deleted)?
N/A. The dataset does not relate to people.
(f) Will older versions of the dataset continue to be suppotee/nnseedmainniieee? Yes.
A6

---

(g) If others want to extend/aaggment/build on/contribute to the dataset, is there mechanism for them do so?
Yes. We have released the source code well as a licence on our project website. Future developments are welcome.
(h) Any other comments?
None.
A7

---

Blicket
Non-Blicket
Initial Contexts:
Naive Agent Trials:
Figure Al: An example that solved by the naive agent.
Blicket
Non-Blicket
Initial Contexts:
Nalve Agent Trials:
Figure A2: example where the naive agent fails.
Blicket
Non- Non-Blicker Blicket
Initial Contexts:
Search-Naive Agent Trials:
Figure A3: An example that solved by the search-naive agent.
Blicket
Non-Blicket
Initial Contexts:
Search-Naive Agent Trials:
Figure A4: An example where the search- naive agent fails.
A8

---

Blicket
Non-Blicket
Initial Contexts:
DDPG Agent Trials:
Figure A5: An example that is solved by the DDPG agent.
Blicket
Non-Blicket
Initial Contexts:
DDPG Agent Trials:
Figure A6: An example where the DDPG agent fails.
Blicket
Non-Blicket
Initial Contexts:
Human Trials:
Figure A7: An example that is solved by a human participant.
A9

---

Gamepage
New Game
Current Panel: 4/10
History Panel
Reward/Total Reward: 0/0
Context
Trial
Trial
Your current belief. (which objects can aclivale the panel?)
Guess!
Figure A8: UI design for the web-based + IVRE environment.
A10


Reviews:
[{"review_id": "57QL8UIXKs", "reviewer": "Reviewer_hE8F", "summary": "", "questions": "", "limitations": "The only limitation in terms of social impact is the lack of details on the human study.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The proposed benchmark is easy to understand and well described. The experiments are quite comprehensive. This benchmark is a useful and relevant demonstration that our current learning methods are insufficient for capturing the nuances of causal learning.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "mibV90po7Z", "reviewer": "Reviewer_svXQ", "summary": "", "questions": "", "limitations": "Adequately discussed", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The environment seems like a good contribution. I could see people using it in future work.\n- The paper is well-written.\n- The paper is situated well with respect to prior work\n- The baselines and experiments are comprehensive, and the analysis is useful", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "cdaxmMDkkd", "reviewer": "Reviewer_KESZ", "summary": "", "questions": "", "limitations": "Yes. The limitations section is comprehensive.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "-\tThe authors intelligently adapt the CLEVR environment to produce a benchmark for vision-centric ambiguity resolution.\n-\tThe paper includes a strong experiments section that considers a wide range of approaches. It is helpful that human performance is included in this section. The inclusion of GPT is interesting.\n-\tThe metrics used for the evaluation are well-chosen and informative.\n-\tTwo versions of the task are presented: A symbolic version and a pixel version. This flexibility allows for the visual reasoning portion of the task to be abstracted to observe reasoning ability alone, which is helpful given the complexity of the overall task.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "O9v624o1cG", "reviewer": "Reviewer_7bBX", "summary": "", "questions": "", "limitations": "I believe authors correctly identified the main limitation and future work - i.e. introducing more visually complex environment, and further real-world scenarios. As mentioned above visual part of pixel-input models seems to be critical and more investigation into that would be great to see.\n\nSocietal impact is not discussed, which I belive is a requirement of the conference. It would be good if authors could add that.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper proposes a new task that emphasises an interesting problem for the community (research in the field of reasoning, including visual reasoning). The interesting factor is introducing uncertainty to the reasoning process forcing an agent to formulate and test hypotheses in the given scenario. I believe this to be an interesting problem for the community being relevant to various exploratory scenarios.\n\nThis work builds the environment upon CLEVR [1] in terms of visual representation, making it easily reproducible.\n\nA rich benchmark of various models is introduced. I believe that a good range and selection of models are tested. I appreciate benchmarking on basic, heuristic methods (random, naive, etc.) as a baseline, as well as a comparison to current LLMs and RL models, followed by a human benchmark. The results provided prove that compositional reasoning introduced as in this benchmark is indeed a challenging task for most of the current models (yet not completely unsolvable).", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "cGXcavHYZv", "reviewer": "Reviewer_pTW5", "summary": "", "questions": "", "limitations": "- L98 authors mention the simplicity but not lack of sensorimotor control. However, the action space of the environment is merely probability simplex over objects, which is totally different from the common definition of motor control in reinforcement learning or robotics. Therefore, it is not faithful to to call it \"sensorimotor control\" at all.\n\n- More technical aspects about the proposed environment are encouraged to include. For example, since running RL requires significant data collection, what is the throughput for both symbol-based and pixel-baed inputs?", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper is well written.\n- Code and document are provided on the website.\n- The idea of incorporating visual reasoning into a sequential decision-making process is novel.\n- Evaluation is comprehensive and informative.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors have proposed an interactive environment titled IVRE featuring scenarios involving *Blicket* detection. The primary motivation is to provide a resource for evaluating the capability of agents to reason under uncertainty, which is a natural objective towards developing agents with humanlike intelligence. The reviewers have expressed that the paper is well-written, and the clarity and empirical rigor has further improved after the authors have incorporated the suggestions of the reviewers and the discussions during the rebuttal phase. The webpage is well documented and provides relevant details. I can see this being a resource that is useful for the community. 

The code/doc and sample still point to GDrive links; I encourage the authors to heed the reviewers' suggestions and move to hosting the code on Github or equivalent, which would further help in the adoption of IVRE. The authors have expressed that they will do so, and I am taking this in good faith. 

Given the favorable response of the reviewers, the updated and improved discussion and additional experiments from the authors, and the overall promising utility of this work to the community, I recommend Accepting this paper.
Decision: Accept (Poster)