Title: CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Mobile Agents

Large Language Models

Collaboration

UI Exposure Reduction
Abstract: Mobile agents rely on Large Language Models (LLMs) to plan and execute tasks on smartphone user interfaces (UIs). While cloud-based LLMs achieve high task accuracy, they require uploading the full UI state at every step, exposing unnecessary and often irrelevant information. In contrast, local LLMs avoid UI uploads but suffer from limited capacity, resulting in lower task success rates. We propose $\textbf{CORE}$, a $\textbf{CO}$llaborative framework that combines the strengths of cloud and local LLMs to $\textbf{R}$educe UI $\textbf{E}$xposure, while maintaining task accuracy for mobile agents. CORE comprises three key components: (1) $\textbf{Layout-aware block partitioning}$, which groups semantically related UI elements based on the XML screen hierarchy; (2) $\textbf{Co-planning}$, where local and cloud LLMs collaboratively identify the current sub-task; and (3) $\textbf{Co-decision-making}$, where the local LLM ranks relevant UI blocks, and the cloud LLM selects specific UI elements within the top-ranked block. CORE further introduces a multi-round accumulation mechanism to mitigate local misjudgment or limited context. Experiments across diverse mobile apps and tasks show that CORE reduces UI exposure by up to 55.6\% while maintaining task success rates slightly below cloud-only agents, effectively mitigating unnecessary privacy exposure to the cloud. The code is available at https://github.com/Entropy-Fighter/CORE.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CORE, a collaborative framework that combines cloud and local LLMs to reduce UI exposure in mobile agents while maintaining task accuracy. The approach leverages layout-aware block partitioning, co-planning, and co-decision-making to selectively upload only task-relevant UI elements, achieving significant exposure reduction (up to 55.6%) with minimal impact on success rates. Experimental results on public datasets demonstrate the effectiveness of the method compared to both cloud-only and local-only baselines. The work is well-motivated, technically sound, and addresses an important privacy concern in mobile agent systems.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- This paper addresses an important problem: how to enable effective collaboration between local LLMs and cloud LLMs while minimizing privacy exposure.\n- The proposed processing pipeline is reasonable and effectively tackles the identified challenge.\n- The experimental results support the authors’ claims.\n\n\n### Weaknesses\n- The main issue with this paper lies in its experimental setting, which differs from most mainstream approaches. The paper primarily relies on XML-based input, whereas many modern mobile applications no longer provide XML. As a result, image-based processing would be more meaningful. Approaches involving visual-based segmentation or ranking may be more practical.\n-  In addition, there is a lack of detail. I recommend the authors include a more comprehensive case study, as it is otherwise difficult to understand the exact form of the sub-tasks.\n- The evaluation metrics in Table 1 are limited. The authors should not only consider the usage cost of the cloud LLM, but also account for the overall system cost (e.g., local latency), as shown in Figure 4 (Left). Therefore, I suggest the authors consider optimizing the “Rank blocks by current sub-task” step using a dedicated ranking model.\n\nQUESTIONS:\n- I suggest that the authors keep the local LLM consistent in Figure 5 (right).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CORE, a collaborative framework combining local and cloud large language models (LLMs) to automate mobile tasks while limiting unnecessary UI data exposure to the cloud. CORE partitions the UI into meaningful blocks, uses the local model to filter and rank them, and lets the cloud model make final decisions within top-ranked blocks. Experiments show CORE cuts UI exposure by over 50% while keeping high task success rates.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe idea proposed in this paper to address ambiguous instructions in GUI automation is interesting, and some practical attempts have also been made. \n\n\nWeaknesses:\n\n1. Although the motivation of privacy protection proposed in this paper is reasonable, the presented approach does not actually address this challenge. Using local/cloud models in combination can reduce some redundant information being uploaded to the cloud, but it cannot prevent the exposure of sensitive information. For example, in the scenario described in the Introduction, even if only the visual information of the search bar is uploaded to the cloud-based model, the search bar often contains recommendation content and may reveal user preferences to the cloud model. The “sufficient UI content” necessary for decision-making inevitably carries private information, which may contradict the paper’s claim of privacy protection.\n\n2. The technical contribution of this paper is also limited. The multi-agent framework design is rather naive and has already been attempted and validated in many previous works [1,2].\n\n3. I have doubts about the practical significance of the Reduction Rate metric. This metric only represents the quantitative reduction compared to uploading the entire UI to the cloud, but (1) the cloud model’s reasoning efficiency is unlikely to improve significantly because of it, and (2) it does not guarantee that the selected UI elements are free of sensitive information.\n\n4. The paper lacks performance evaluation on the commonly used dataset AndroidWorld [3].\n\n[1]. Wang J, Xu H, Jia H, et al. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration[J]. arXiv preprint arXiv:2406.01014, 2024.\n\n[2]. Agashe S, Han J, Gan S, et al. Agent s: An open agentic framework that uses computers like a human[J]. arXiv preprint arXiv:2410.08164, 2024.\n\n[3] Rawles C, Clinckemaillie S, Chang Y, et al. Androidworld: A dynamic benchmarking environment for autonomous agents[J]. arXiv preprint arXiv:2405.14573, 2024.\n\nQUESTIONS:\n1. I have concerns about the method for selecting necessary UI elements to upload to the cloud, as there is no specific approach designed to ensure that the uploaded information is non-sensitive. Perhaps the authors could provide concrete quantitative or qualitative criteria to demonstrate that these UI elements do not contain private information.\n\n2. I suggest including at least one case study in the main text to qualitatively demonstrate the effectiveness of the proposed method.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes CORE, a framework to addresses the problem of excess UI data being sent to cloud LLMs by mobile agents. The authors provide empirical results to show that CORE can reduce the number of UI elements uploaded by over 50% while keeping task success rates within ~5% of a full-access cloud agent.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1)\tThe paper focuses on an important and timely research topic, namely, the LLM agents for smartphone operation.\n2)\tThe paper is easy to follow.\n3)\tThe GUI exposure problem is a novel problem to solve.\n\nWeaknesses: \n1)\tThe practical value of the proposed method is insufficiently justified. The paper’s primary motivation is reducing data exposure. However, the experimental results (Table 1 and Figure 4) show that CORE incurs higher latency and lower accuracy than the cloud-only baselines. To strengthen the case for CORE, the authors should demonstrate that limiting GUI exposure yields tangible privacy or usability gains. For example, they could: 1) include a user study measuring perceived comfort or trust when sensitive fields are hidden; or 2) include quantitative privacy-risk analysis (e.g., metrics on sensitive information leakage).\n2)\tThe novelty of the CORE framework is incremental. While CORE’s hierarchical local/cloud architecture is well motivated, similar two-stage frameworks have appeared recently (e.g., [1, 2, 3]). The manuscript needs to clearly articulate what distinguishes CORE’s “co-planning/co-decision” mechanism from prior work.\n[1] Dai G, Jiang S, Cao T, et al. Advancing mobile gui agents: A verifier-driven approach to practical deployment[J]. arXiv preprint arXiv:2503.15937, 2025.\n[2] Christianos F, Papoudakis G, Coste T, et al. Lightweight neural app control[J]. arXiv preprint arXiv:2410.17883, 2024.\n[3] Mei K, Zhu X, Xu W, et al. Aios: Llm agent operating system[J]. arXiv preprint arXiv:2403.16971, 2024.\n\nQUESTIONS:\n1.\tHave you considered running a small-scale user study to measure whether hiding non-essential UI blocks actually increases users’ comfort or trust in the agent?\n2.\tCan you provide a more concrete analysis of what sensitive information is omitted by CORE?\n3.\tAside from the prompt templates, what are the differences between your LLM models collaboration pipeline, which comprises Co-planning and Co-decision-making, and those employed in prior works [1,2,3] ?\n\n[1] Dai G, Jiang S, Cao T, et al. Advancing mobile gui agents: A verifier-driven approach to practical deployment[J]. arXiv preprint arXiv:2503.15937, 2025.\n[2] Christianos F, Papoudakis G, Coste T, et al. Lightweight neural app control[J]. arXiv preprint arXiv:2410.17883, 2024.\n[3] Mei K, Zhu X, Xu W, et al. Aios: Llm agent operating system[J]. arXiv preprint arXiv:2403.16971, 2024.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents CORE, a framework that blends cloud and local large language models (LLMs) to streamline task automation on mobile devices while cutting down on unnecessary UI data sent to the cloud. By using XML-based layout-aware block partitioning, collaborative planning, and decision-making, CORE reduces UI exposure by up to 55.6% while keeping task success rates close to cloud-only models. Tests on DroidTask and AndroidLab datasets show it maintains strong performance—within 5% of GPT-4o—while significantly outperforming local-only models by up to 46.85%.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper tackles a real privacy concern in mobile agents by reducing UI data uploads, a practical issue given smartphones’ sensitive nature. The proposed CORE framework smartly leverages both local and cloud LLMs to balance privacy and performance, which feels like a meaningful step forward.\n- Experiments are thorough, covering two datasets (DroidTask and AndroidLab) with diverse apps and tasks, and results are compelling—55.6% UI reduction with minimal accuracy loss is impressive. The ablation study nicely pinpoints the value of each component, like co-planning and multi-round accumulation.\n- The writing is engaging and clear, with figures like the CORE pipeline and XML partitioning visuals making the technical details digestible. The problem formulation is precise, grounding the work in a well-defined optimization goal.\n- The idea of using XML structure for block partitioning is clever, preserving the UI’s logical design in a way that feels intuitive and practical for mobile interfaces.\n\nWeaknesses:\n- The evaluation is solid but could dig deeper into edge cases, like apps with highly dynamic UIs or tasks needing multiple steps across screens. It’s unclear how CORE handles these trickier scenarios, which could limit its real-world robustness.\n- The paper glosses over the computational cost of local LLM processing. While cloud token usage is discussed, the added latency from local models on resource-constrained devices feels underexplored, especially for consumer-grade phones.\n- The reliance on XML-based UI representations might not generalize to screenshot-based or multimodal agents, which are increasingly common. This narrows the framework’s immediate applicability.\n\nQUESTIONS:\n1. How does CORE perform on tasks requiring navigation across multiple screens or highly dynamic UIs, like social media apps? Testing these could strengthen claims of robustness, and strong results here would boost my confidence in its general applicability.\n2. Can you quantify the local LLM’s computational burden, especially on low-end devices? A breakdown of resource demands (e.g., memory, CPU) would clarify feasibility and could address my concerns about practical deployment.\n3. Why focus solely on XML-based UIs? Exploring compatibility with screenshot-based or multimodal approaches could broaden CORE’s relevance. Showing adaptability to other formats would elevate the paper’s impact.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}