Title: Can Language Models Serve as Text-Based World Simulators?
Benchmark: BYTESIZED32-State-Prediction
Summary: This paper introduces a new benchmark, containing a dataset of text game state transitions and accompanying game tasks. They use this to directly quantify how well LLMs can serve as text-based world simulators. They test GPT-4 on this dataset and find that, despite its impressive performance, it is still an unreliable world simulator without further innovations.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon:  abilities of LLMs to directly simulate virtual environments
Phenomenon defined by authors: No
Phenomenon definition: The phenomenon is not defined with details. I understood it as the ability to simulate the world.
Definition scope: Comprehensive
Definition integrity: Authors' description is unclear
Definition integrity detail: Not applicable
Phenomenon contested: Not defined
Task definition: In this task, LLMs serve as world simulators in text-based virtual environments, in which an agent receives observations and proposes actions in natural language in order to complete certain objectives.  
Task item definition: Each task is a human-authored text games that each simulate a different scientific or commonsense reasoning concept. Each item is the game state
(s_t, r_t, d_t) as well as its intermediate state s_act (t+1) at each time step t as a JSON object. 
Task source: Modified from another benchmark (e.g. translation into another language), The dataset is derived from the open BYTESIZED32 corpus.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 76,369 virtual text environment state transitions
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Grounding / 
New taxonomy: General Capability / Grounding