SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The benchmark was created as an environment for evaluating artificial agents' reasoning ability under uncertainty. IVRE is an interactive environment featuring rich scenarios centered around Blicket detection. Agents in IVRE are placed into environments with various ambiguous action-effect pairs and asked to figure out each object's role. Agents are encouraged to propose effective and efficient experiments to validate their
hypotheses based on observations and gather more information. The game ends when all uncertainties are resolved or the maximum number of trials is consumed.
This dataset was created by Manjie Xu, Guangyuan Jiang, Wei Liang, Chi Zhang and Yixin Zhu. They are from Beijing Institute of Technology (Manjie Xu, Wei Liang), Peking University (Guangyuan Jiang, Yixin Zhu) and National Key Laboratory of General Artificial Intelligence, BIGAI (Chi Zhang).
M.X., G.J., W.L., C.Z., and Y.Z. are supported in part by the National Key R&D Program of China (2022ZD0114900), M.X. and W.L. are supported in part by the NSFC (62172043), and Y.Z. is in part by the Beijing Nova Program.
(d) Any other Comments?
(a) What do the instances that comprise the benchmark represent?
The benchmark contains episodes in which agents are tasked to figure out which objects are Bickets.
N/A. Each episode in IVRE is randomly sampled and can have infinite tasks.
IVRE contains all possible instances.
In each episode, an agent is presented with several initial observations of various object combinations (referred to as context). The context alone is insufficient to solve Bicketness for all objects. Hence, in each following step (referred to as trials), the agent proposes a new experiment of a specific object combination and updates its belief of Bicketness based on the outcome of experiments.
(i) Are there any errors, sources of noise, or redundancies in the benchmark?
(j) Is the benchmark self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?
Self-contained.
(k) Does the benchmark contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)?
(1) Does the benchmark contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
(m) Does the benchmark relate to people?
(n) Does the benchmark identify any subpopulations (e.g., by age, gender)?
(p) Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? No.
(q) Any other comments? None.
(a) How was the data associated with each instance acquired? We render them using Blender.
(b) What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? In each episode, IVRE will randomly sample objects and bickets from a pool.
(c) If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? N/A.
(d) Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? Manjie Xu and Guangyuan Jiang wrote the generation code.
(e) Over what timeframe was the data collected? N/A.
(f) Were any ethical review processes conducted (e.g., by an institutional review board)? The dataset raises no ethical concerns.
(g) Does the dataset relate to people? No.
(h) Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? N/A.
(i) Were the individuals in question notified about the data collection? N/A.
(j) Did the individuals in question consent to the collection and use of their data? N/A.
(k) If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? N/A.
(1) Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? Yes.
(m) Any other comments? None.
(a) Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? N/A.
(b) Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (e.g., to support unanticipated future uses)? N/A.
(c) Is the software used to preprocess/clean/label the instances available? N/A.
(d) Any other comments? None.
This dataset could be used for other reserach topics like causal discovery, causal reasoning and active learning.
(d) Is there anything about the composition of the dataset or the way it was collected
and preprocessed/cleaned/labeled that might impact future uses?
IVRE could be accessed on our project website.
IVRE has already been released.
We release our benchmark under CC BY-NC  ${}^{1}$  license.
Manjie Xu and Guangyuan Jiang are maintaining.
manjietsu@bit.edu.cn, jgy@stu.pku.edu.cn
(c) Is there an erratum?
Future erratum will be released through the website.
(d) Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?
N/A. The dataset does not relate to people.
<https://creativecommons.org/licenses/by-nc/4.0/>
Yes. We have released the source code as well as a licence on our project website. Future developments are welcome.
Bicket
Non-Bicket
Initial Contexts:
Naive Agent Trials:
Figure A1: An example that is solved by the naive agent.
Figure A2: An example where the naive agent fails.
Search-Naive Agent Trials:
Figure A3: An example that is solved by the search-naive agent.
Figure A4: An example where the search-naive agent fails.
DDPG Agent Trials:
Figure A5: An example that is solved by the DDPG agent.
Figure A6: An example where the DDPG agent fails.
Human Trials:
Figure A7: An example that is solved by a human participant.
Current Panel: 4/10
Your current belief: (which objects can activate the panel?)
Figure A8: UI design for the web-based IVRE environment.
Gamepage