SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The IVRE environment can be modeled as a POMDP for RL training. Formally, the POMDP problem is a tuple  $(S,A,T,R,\Omega,O,\gamma)$ , where  $S$ ,  $A$ ,  $T$ ,  $R$ ,  $\Omega$  are the state space, the action space, the transition probability, the reward function, and the observation space, respectively. Note that the state space covers the ground-truth belief up to each time step  $t$ . The action space, the observation space, and the reward function have been discussed in Sec. 3.  $O$  is the observation generator for each state-action pair  $(s,a)$  and  $\gamma$  is the discount factor (set to 0.99). As mentioned in Sec. 3, we consider two versions of the observation space: the symbol-input version and the pixel-input version. For the
Figure 3: The general RL architecture for benchmarking IVRE. The architecture follows the actor-critic design; both the policy and value functions are represented with neural networks. We use a shared CNN encoder to extract visual features for the pixel-input version of the environment depicted here. The symbol-input version differs from the pixel version in providing a binary vector description processed by an MLP.
POMDP formulation of the RL algorithms, we use a recurrent agent for learning, as the underlying state is not directly observable and has to be inferred from the history of the observation.
The POMDP could also be transformed into an Markov Decision Process (MDP) if we know the underlying state at each time step. Therefore, we also consider a feed-forward architecture for this formulation: if one regards the belief sub-space from the agent's output as the true state space, we can use the belief from the previous time step and the new observation to propose a new trial and update the belief. Note that despite a feed-forward architecture being used, the entire process is still recurrent as the belief state's dependency is traced back to the history observation. See Fig. 3 for a graphical illustration of the general RL architecture we use for benchmarking IVRE. Here we only depict the pixel-input version, where the agent architecture is a CNN encoder module for visual perception and a feed-forward module that uses the agent's belief as an approximate.
As the action space has been softened, we consider popular continuous RL methods for evaluation. Specifically, we benchmark Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015), Twin Delayed Deep Deterministic Policy Gradient (TD-3) (Fujimoto et al., 2018), and Proximal Policy Optimization (PPO) (Schulman et al., 2017). We also test an agent that incorporates NOTEARS (Zheng et al., 2018, 2020) as its reasoning component. Please refer to Appx. B for details.
DDPG DDPG is designed based on the successful Deep Q-Learning (Mnih et al., 2015) and extended to the continuous action domain. For the symbol-input DDPG model, we use two MultiLayer Perceptron (MLP) with three hidden layers of width 512 and ReLU activation as the actor and critic network, respectively. For the pixel-input version, we use an ImageNet (Deng et al., 2009) pre-trained ResNet-18 (He et al., 2016) to process image panels and concatenate it with the agent's belief from the previous time step as input to the actor and critic network. We add an additional MLP and an LSTM layer with 384 units to process the raw history observation in the recurrent agent.
TD-3 Compared to DDPG, TD-3 notices the problem of the overestimated value function in the actor-critic setting and suggests a new mechanism in mitigating this effect. In our TD-3 implementation, we use a backbone similar to DDPG: three MLP with three hidden layers of width 512 and ReLU activation are used for the actor and two critic networks, respectively. A similar adaptation in the DDPG setup is made for the recurrent agent in the TD-3 implementation.
PPO In RL tasks, PPO is widely used as an online learning baseline. It is based on policy optimization methods and generally has trust-region methods' stability and reliability. Similar to the two RL agents mentioned above, we adopted the same three-layer MLP for its backbone.
4.3 Human Baseline
We recruited 54 participants from Peking University to take part in the study, which was conducted through a web-based IVRE platform (see Fig. A8). Each participant was compensated with course credits upon the completion of an episode. The episodes were randomly allocated to the participants, who were then tasked with solving the IVRE challenge in a maximum of 10 steps, without any time restrictions.
Table 2: Left: Performance of heuristic models on the symbol-input version of IVRE environment. Right: Performance of RL models and humans on the IVRE environment. We report two evaluation metrics: the problem-solving accuracy, denoted as Acc, and the total reward, denoted as  $R$ . The context column records results when the agent only observes the context panels, whereas the episode column after the entire episode (FF: feed-forward, Re: recurrent, V: pixel-input).
Model	Context	Episode	Model	Context	Episode
Acc	R	Acc	R	Acc	R	Acc	R
Random	0.86%	-5.42	1.87%	-14.14	DDPG-FF	22.47%	-0.17	32.47%	-3.70
Bayes	15.60%	-2.98	43.03%	-3.78	DDPG-Re	13.55%	-2.03	46.03%	-0.29
Naive	3.50%	-3.91	43.62%	-1.69	TD-3-Re	12.57%	-2.40	36.83%	-2.71
NOTEARS	9.10%	-4.66	12.70%	-13.02	TD-3-FF	21.91%	-0.42	30.05%	-4.48
Search-Naive	1.51%	-3.68	83.80%	9.39	PPO	6.87%	-3.87	28.56%	-5.85
Search-Random	1.80%	-3.62	34.15%	-1.87	DDPG-V	0.35%	-5.02	0.72%	-13.40
GPT-3.5	3%	-5.91	11%	-13.39	TD-3-V	0.27%	-5.04	0.31%	-13.51
GPT-4	10%	-3.36	26%	-7.88	Human	33.33%	5.01	98.15%	12.70
We run experiments on the IVRE environment with the agents and their aforementioned variants. For evaluation metrics, we report the agent's average reward together with problem-solving accuracy over  $10^{4}$  random test episodes (except LLMs  $10^{2}$ ). The average reward measures how the model performs in terms of reasoning and trial efficiency, while the problem-solving accuracy is computed by counting the number of episodes where an agent correctly figures out Bicketness for all objects. Apart from the final results after finishing an entire episode, we also evaluate how the agent performs after observing the initial context panels only: a metric on how the model understands the problem without any trials.
5.2 Performance of Agents
Heuristic Agents Tab. 2 shows the performance of the heuristic agents in the proposed IVRE environment. In general, we note that the more information collected, the better the agents resolve the uncertainty: the low accuracy after the context panels also verifies that additional trials are necessary for solving the entire problem. The Random agent fails in this task unsurprisingly, while the Search-Naive agent reaches more closely to the human performance. The comparison among the random agent, the naive agent, and their search-based variants indicates that both the reasoning component and the exploration strategy are beneficial for a symbolic approach. Without the reasoning component, the naive agent does not build a reliable belief over which object is more likely to be a Blicket; without the exploration strategy, an agent only randomly collects new experiments, doing little help in demystifying the phenomenon. LLM agents, equipped with priors learned from large-scale text corpora, exhibit a notable level of reasoning ability to reduce uncertainty, yet still far from human performance.
Symbol-Input RL Agents Tab. 2 summarises the performance of symbol-input RL agents. RL agents performed significantly better than the Random agent, where DDPG-Re gets the highest reward. Recurrent agents generally perform better than feed-forward agents, indicating that history observation and trials play a role in generating valid hypotheses and proposing new trials. One interesting observation from the performance of these RL agents is that they generally propose very inefficient trials: the final reward decreases as time goes by. However, it becomes more likely for an agent to stumble on a solution by chance.
Pixel-Input RL Agents We consider DDPG and TD-3 with the pixel-input experiments. As shown in Tab. 2, all the models tested in the pixel-input version of IVRE catastrophically fail with random-level problem-solving accuracy and total reward. Surprisingly, while their rewards are slightly higher than the random agent's, their problem-solving accuracy is even worse, let alone the conspicuously large gap from the performance under the symbol-input circumstance. We carefully checked the output of these agents and found that belief and trial predicted by agents oscillated around 0.5, indicating that they had difficulty learning to understand the IVRE environment from pixel-level
Figure 4: Left and Middle: Distribution of the steps an agent takes to successfully solve an IVRE episode. Left: Search-based Naive agent. Middle: DDPG-Re agent. Right: Box plot of the number of objects the DDPG-Re agent proposes at each time step.
input. Our results suggest that current models should be further improved in representation learning to help interactive reasoning under uncertainty.
Comparison Comparing experimental results in heuristic agents, symbol-input RL agents, and pixel-input RL agents, we note that uncertainty reduction is challenging not only in reasoning but also in proposing valid trials. Our heuristic results reveal that the lack of either capability can lead to failure in tasks like IVRE, which are quite common in the real world. For almost all of the RL-learned policies, they have difficulty effectively proposing meaningful experiments to validate and update their belief. The heuristic method with the best performance is designed based on human prior and equipped with a fixed and naive policy, far from the talent shown by human beings when performing this task. Even LLMs with vast prior knowledge still fall short.
The naive trial policy achieves significantly better results compared to the random trial policy with oracle belief. Several factors contribute to the performance of the Search-Naive agent. First, the agent operates under the assumption that the Bicket machine functions as an OR machine (the disjointive causal overhypothesis), meaning it will activate if at least one object is a Bicket. Second, although not entirely accurate, the agent uses correlation as a proxy for causality, which may yield partially correct outcomes in certain situations. Third, the agent tests for "Blicketness" one object at a time, which, while not the most efficient approach, helps to isolate other confounding variables. These findings suggest that solving the IVRE problem requires more than random testing; it calls for thoughtful selection of trials.
Moreover, the visual complexity adds additional challenges to the problem for agents, whereas humans are used to reasoning and refining their hypotheses from visual stimuli. Looking into the failure in the pixel-input agents, we hypothesize that causal representation learning could mitigate its learning inefficacy: not only do we need features to distinguish between different experiments and those that support an in-depth understanding of the environment.
Analyzing the patterns and outcomes of human participants, we observe that they exhibit strong capabilities in reasoning and exploration. They employ a versatile approach to experimentation and demonstrate effective reasoning even within constrained scenarios. For further insights, refer to the Appx. E detailed set of examples provided.
Distribution of Steps The symbolic version of IVRE is considered a diagnostic tool and an upper bound for pixel IVRE. With pixel-input RL agents reaching only random-level performance, we analyze the best heuristic model (i.e., Search-based Naive agent) and the best symbol-input RL model (i.e., DDPG-Re). Fig. 4 shows the distribution of steps the agents take to successfully solve a problem and the number of objects proposed at each time step in the  $10^{4}$  random test episodes. We also plot the number of objects the DDPG-Re agent proposes at each time step. The search-based method makes reasonable trials with the naive strategy at each step, and as the information collected amounts, more episodes are solved. RL agents also have learned specific exploration strategies to reduce uncertainty, although not as perfect as humans: more flexible and diverse actions are observed in Steps 5-7, and more episodes are solved in these steps. On the other hand, DDPG-Re agent might have learned data bias as it directly solves a certain number of episodes without interaction. For its trial policy, the DDPG-Re agent consistently selects around 3 to 5 objects to test at earlier steps in the process and picks fewer objects towards the end. Combining the analysis, we find the agent in
the early steps learn to use a mixed strategy even less effectively than the naive trial, showing very limited ability in actively reasoning under uncertainty.
In this work, we introduce IVRE, an interactive testbed for evaluating artificial agents' reasoning ability under uncertainty. Inspired by the theoretical proposition and the empirical observation of infants, the newly introduced IVRE mimics the classic Bicket detection experiment but intentionally simplifies the sensorimotor control by abstracting it out into a discrete space of object selection. IVRE's design not only requires the agent to effectively update its belief based on the information collected so far but also necessitates the capability to come up with maximally efficient new experiments to disentangle confounding factors.
Measuring performance and concluding this manuscript is not the end but rather the beginning of our pursuit of an intelligent agent who can learn and think like people when handling uncertainty during the interaction. With today's agents catastrophically failing in this problem, how do humans, even very young children, successfully resolve uncertainty in the world around them without specific training? What role does training from nurture play in the process? And how to incubate an artificial agent with this ability through interaction with the environment? With many questions unanswered, we hope this preliminary work will motivate further research.
Societal Implication We have not identified any negative societal implications arising from the proposed benchmark. On the contrary, the act of uncertainty resolution within IVRE necessitates robust reasoning capabilities, contingent on effective intervention strategies. It is noteworthy that contemporary learning agents still struggle in identifying interconnected variables through interactive engagement. We believe IVRE has the capacity to make a positive contribution towards the development of agents exhibiting human-level intelligence.
Limitations and Future Work To highlight reasoning and uncertainty reduction, we design IVRE with synthetic instead of real-world scenarios. Given the challenges associated with visual perception and action in the real world, additional research is required to investigate how agents can effectively address uncertainty within a more complex environment. IVRE also employs relatively simple causal structures to create uncertainty. This calls for further research to study how machines and humans can actively resolve uncertainties from different levels in various causal structures.
Acknowledgement We thank Liangru Xiang for helpful discussions, Ms. Zhen Chen (BIGAI) for designing the figures, and NVIDIA for their generous support of GPUs and hardware. M.X., G.J., W.L., C.Z., and Y.Z. are supported in part by the National Key R&D Program of China (2022ZD0114900), M.X. and W.L. are supported in part by the NSFC (62172043), and Y.Z. is in part by the Beijing Nova Program.
Zhu, Y., Gao, T., Fan, L., Huang, S., Edmonds, M., Liu, H., Gao, F., Zhang, C., Qi, S., Wu, Y. N., et al. (2020). Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense. Engineering, 6(3):310-345. 3
Zhu, Y., Groth, O., Bernstein, M., and Fei-Fei, L. (2016). Visual7w: Grounded question answering in images. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
A Ablation with Fewer Objects
Table A1: Performance of models on the symbolic version of IVRE with fewer objects and blickets. We conduct experiments where at most 3 blickets are selected from 7 unique objects in IVRE. Results show that reducing the number of objects in IVRE would simplify the environment.
Model	Random	Bayes	Naive	Search-Naive	Search-Random	DDPG-FF	TD-3-FF
Acc	7.61	71.92	94.56	100.00	54.53	90.08	88.64
R	-12.81	4.73	10.82	14.32	3.17	11.96	11.53
Striking a balance between simplicity and complexity is crucial to maintaining the IVRE's ability to assess agents' reasoning abilities effectively. We conduct experiments where at most 3 bickets are selected from 7 different objects in IVRE. The results are shown in Tab. A1. As demonstrated, reducing the number of objects in IVRE would simplify the environment. However, this approach could inadvertently lead to shortcuts, as random or naive trials might efficiently address much of the uncertainty.
B Agent Details
B.1 RL Agent Details
All RL models are implemented in PyTorch (Paszke et al., 2017) under the helper library of Tianshou (Weng et al., 2022). When training the MLP and the LSTM backbones, we use the Adam optimizer (Kingma and Ba, 2014) with a learning rate of  $3 \times 10^{-4}$ . The DDPG and the TD-3 agent share the same hyper-parameters: we set the exploration noise to 0.1 and the target network's soft update to 0.005. Other parameters have been set using the default parameters from the original work. All RL models were trained for  $10^{7}$  steps during training. The best model from training was saved and used during the evaluation phase. All the experiments reported herein were run on NVIDIA GeForce RTX 3090 or Tesla V100 graphics cards.
C Model Details
C.1 Symbol-Input Backbone
Tab. A2 shows the feed-forward architecture we use for the RL agents in the symbol-input version of the environment, where the parameter of the Linear layer denotes the size of its output and  $m$  is set equal to the size of the action space. Of note, the DDPG agent uses an actor network and a critic network, while the TD-3 agent uses one actor network and two critic networks. The actor network and the critic network share the same architecture. Note that the feed-forward architecture takes in both the current observation and the belief from the previous time step. Our online learning baseline PPO shares the same three-layer MLP for its backbone.
The recurrent architecture of the RL agent recruits the network components listed in Tab. A3, where we explicitly use a single-layer LSTM with a hidden size of 512 units.
Table A2: Feed-forward architecture of RL agents.
Layer	Units
Linear	512
ReLU	/
Linear	512
ReLU	/
Linear	512
ReLU	/
Linear	m
Table A3: Recurrent architecture of RL agents.
Layer	Units
LSTM	512
Linear	512
ReLU	/
Linear	m
C.2 Pixel-Input Backbone
The pixel-input model leverages a pre-trained ResNet-18 (He et al., 2016) for extracting image features. Specifically, we resize each image rendered from the Blender EEVEE engine (Blender Online Community, 2016) into  $224 \times 224$  and feed it through the backbone. The features are then passed through the same actor network and the critic network listed in Tabs. A2 and A3.
D LLMs Experiments
We use the OpenAI Chat Completions API for our LLMs experiments; see examples in Tab. A4.
Table A4: An example dialogue generated between GPT and our IVRE (the User).
Role	Content
User	"There are nine objects: 1, 2, 3, 4, 5, 6, 7, 8, 9 that can be put on a machine.
Up to four of all the objects have a unique property that can activate the machine.
Your goal is to find out all the objects that can activate the machine in ten rounds.
All the following questions in the format of '1, 2, and 3.'"
User	"Round 0:
1 are on the machine, and the machine is not activated.
Question: Which objects can activate the machine?"
"Answer: From 1-9, objects that can activate the machine are"
Assistant	"2, 3, 4, 5, 6, "
User	"Your answer is wrong."
User	"Round 1:
3 are on the machine, and the machine is not activated.
Question: Which objects can activate the machine?
Answer: From 1-9, objects that can activate the machine are"
Assistant	"2, 4, 5, 6, 7, "
User	"Your answer is wrong."
User	"Round 2:
1, 6, 8 are on the machine, and the machine is not activated.
Question: Which objects can activate the machine?
Answer: From 1-9, objects that can activate the machine are"
Assistant	"2, 3, 4, 5, 7, "
User	"Your answer is wrong."
User	"Round 3:
3, 4, 7, 9 are on the machine, and the machine is activated.
Question: Which objects can activate the machine?
Answer: From 1-9, objects that can activate the machine are"
Assistant	"3, 4, 7, and 9."
User	"Your answer is wrong."
User	"Round 4:
None are on the machine, and the machine is not activated.
Question: Which objects can activate the machine?
Answer: From 1-9, objects that can activate the machine are"
Assistant	"Please provide the objects on the machine in Round 4 for me to give..."
User	"Now design your own trial, you can put some of the objects on the machine to validate your hypothesis. Question: Which objects will you put on the machine?
Answer: From 1-9, I will put "
Assistant	"2, 5, and 6 on the machine."
User	"Your answer is wrong."
User	"Round 5:
2, 5, 6 are on the machine, and the machine is not activated.
Question: Which objects can activate the machine?
Answer: From 1-9, objects that can activate the machine are"
Assistant	"3, 4, 7, and 9."
User	"Now design your own trial, you can put some of the objects on the machine to validate your hypothesis. Question: Which objects will you put on the machine? Answer: From 1-9, I will put "
Assistant	"3 and 4 on the machine."
User	"Your answer is wrong."
User	"Round 6: 3, 4 are on the machine, and the machine is not activated. Question: Which objects can activate the machine? Answer: From 1-9, objects that can activate the machine are "
Assistant	"7 and 9."
User	"Now design your own trial, you can put some of the objects on the machine to validate your hypothesis. Question: Which objects will you put on the machine? Answer: From 1-9, I will put "
Assistant	"7 on the machine."
User	"Your answer is wrong."
User	"Round 7: 7 are on the machine, and the machine is not activated. Question: Which objects can activate the machine? Answer: From 1-9, objects that can activate the machine are "
Assistant	"9."
E Case Study
We give examples of interaction replay from different agents in IVRE in Figs. A1 to A7. In each case, the first two figures show available objects in an episode, where the first one contains Blickets and the second one non-Blickets. For the naive agent, it always proposes a trial composed of a single object (see Fig. A1), and from its result of activation, the agent deterministically assigns Bicketness to an object. A search-based naive agent proposes trials based on the confidence of every object. Figs. A3 and A4 show trials from such an agent. In Fig. A3, the agent is quite sure of the non-Bicketness of the purple and green metal cube after the context panels, so it tests other objects. However, it cannot figure out the Bicketness of the green metal ball through all of the contexts, leading to its failure.
The DDPG agent, however, has unique behavior. In Fig. A6, all the Bicket machines in initial contexts are activated, so the agent has no chance to generate correct belief from the context only, except guessing. However, the DDPG agent makes repetitive trials, making it harder to gain effective information to solve the problem. In follow-up experiments, the DDPG agent also solves tasks when most objects are non-Blickets. But it often fails when there are more Blickets, which means it cannot handle situations with high correlation.
Humans show powerful abilities in both reasoning and exploration. Fig. A7 shows an example where a successful trial is achieved. We can see that a flexible trial policy is adopted, and humans can reason effectively given limited contexts.
F Data Documentation
We follow the datasheet proposed in Gebru et al. (2021) for documenting our proposed benchmark:
(a) For what purpose was the benchmark created?