Title: LOTA-BENCH: BENCHMARKING LANGUAGE-ORIENTED TASK PLANNERS FOR EMBODIED AGENTS
Benchmark: LoTa-Bench
Summary: LoTa-Bench is a benchmark for task planning for home-service agents. It proposes a quantitative and automated evaluation framework for language-based agents to complete different home-making tasks like placing an apple in a micro-wave.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: embodied task planning
Phenomenon defined by authors: No
Phenomenon definition: The ability to create high-level plans for an action sequence resulting in a specified goal state in an embodied home-making context.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: The task is to obtain a specified home-making goal (e.g., put the plate and forks in the dishwasher) based on interacting with a simulator. The end state is evaluted.
Task item definition: A simulator and high-level instructions chosen from one of the overall task types (e.g., `Put groceries`).
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task source detail: The crowd-sourced component is for translating one of the sub benchmarks to natural language instructions. 
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 308
Dataset size notes: 943
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Validation
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Success rate
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Planning
New taxonomy: General Capability / Planning