Title: Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents
Benchmark: Mobile-Bench
Summary: Mobile-Bench is a novel benchmark for evaluating LLM agents' capabilities in mobile device interactions. It  creates a more realistic environment for benchmarking that combines API and UI operations, evaluates multi-app coordination, and introduces more nuanced evaluation metrics.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Mobile phone agent interaction capabilities
Phenomenon defined by authors: Yes
Phenomenon definition: agent interactions in "a mobile phone environment that includes a platform supporting both API and UI interactions, and a corresponding dataset with multi-APP tasks."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task requires an LLM agent to accomplish mobile phone operations based on a prompt instruction - this involves  UI elements, API calls, and different apps.
Task item definition: a user query (instruction), target application(s), and CheckPoints (expected execution path)
Task definition detail: Tasks have increasing complexity: SAST (single-app-single-task), SAMT (single-app-multi-task), and MAMT (multi-app-multi-task)
Task source: Real task examples (e.g. GitHub issues), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 832
Dataset metadata available: Yes
Dataset metadata detail: Task comlexity (SAST/SAMT/MAMT), API calls, CheckPoints
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Extended interaction (e.g. conversation, calling an API and processing the response), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), checkpoint coverage
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: by task complexity and checkpoints
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Analysis of different application categories to ensure comprehensive coverage, and ablation studies to validate the importance of APIs and planning/thought components  in the agent's performance.
Taxonomy: Agents / Web
New taxonomy: General Application / Web Agents