DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile
  Phone Agents
Authors: name: Kai Shi, name: Jun Yang, name: Ni Yang, name: Binqiang Pan, name: Qingsong Xie, name: Chao Zhang, name: Zhenyu Yang, name: Tianhuang Su, name: Haonan Lu
Organization: OPPO

Mobile Phone Agents (MPAs) have emerged as a promising research direction due
to their broad applicability across diverse scenarios. While Multimodal Large
Language Models (MLLMs) serve as the foundation for MPAs, their effectiveness
in handling multiple mobile phone tasks simultaneously remains limited.
Although multitask supervised fine-tuning (SFT) is widely adopted for multitask
learning, existing approaches struggle to determine optimal training data
compositions for peak performance. To address this challenge, we propose DaMo
(Data Mixture Optimizer) - a novel solution employing a trainable network that
predicts optimal data mixtures by forecasting downstream task performance for
any given dataset ratio. To support comprehensive evaluation, we introduce
PhoneAgentBench, the first specialized benchmark to evaluate MLLMs on
multimodal mobile phone tasks, comprising 1235 QA pairs spanning diverse
real-world industrial mobile application scenarios. Demonstrating strong
predictive capability (R^2=0.81) in small-scale pilot experiments, DaMo
efficiently extrapolates optimal data mixing configurations. Our results show
DaMo achieves a 3.38% performance improvement on PhoneAgentBench compared to
alternative methods. Furthermore, extensive experiments across established
benchmarks including BFCL-v3, MME-Reasoning, MME-Perception, and OCRBench
reveal DaMo's superior generalization, outperforming other approaches by 2.57%
in terms of average score. When used solely for MLLM optimization on the
BFCL-v3 task, DaMo improves the metrics by 12.47% than other methods. Notably,
DaMo maintains robust scalability, preserving its effectiveness when applied to
other model architectures. The code and dataset are available at
https://github.com/OPPO-Mente-Lab/DaMo.git