Title: MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: agent, mllm, memory, nlp

Abstract: Current mobile GUI agent benchmarks systematically fail to assess memory capabilities, with only 5.2-11.8\% memory-related tasks and no cross-episode learning evaluation. We introduce \textbf{MemGUI-Bench}, the first comprehensive benchmark designed to assess both short-term and long-term memory in mobile GUI agents. Our contributions include: (1) a systematic memory taxonomy with analysis of 11 prominent agents; (2) 128 tasks across 26 applications where 89.8\% challenge memory through cross-temporal and cross-spatial information retention; (3) \textbf{MemGUI-Eval}, an automated evaluation pipeline with novel "Progressive Scrutiny" and 8 hierarchical metrics for memory fidelity and learning effectiveness; and (4) comprehensive assessment revealing significant memory deficits across all evaluated agents. Our experiments expose 4-10× performance gaps between memory-intensive and standard tasks, demonstrate the potential of explicit long-term memory mechanisms, and identify 7 distinct failure modes through systematic analysis. MemGUI-Bench establishes crucial empirical baselines for developing more capable and human-like GUI agents. Code and results: \url{https://anonymous.4open.science/r/MemGUI-Bench-Anonymous}.