Lightweight Neural App Control
Authors: Filippos Christianos
Georgios Papoudakis
Thomas Coste
Jianye HAO
Jun Wang
Kun Shao
Venue: ICLR 2025 Spotlight (2025)
Keywords: vision-language model
multi-modal
android control
app agent
Primary area: foundation or frontier models, including LLMs

Abstract: This paper introduces a novel mobile phone control architecture, Lightweight Multi-modal App Control (LiMAC), for efficient interactions and control across various Android apps. LiMAC  takes as input a textual goal and a sequence of past mobile observations, such as screenshots and corresponding UI trees, to generate precise actions. To address the computational constraints inherent to smartphones, we introduce a small Action Transformer (AcT) integrated with a fine-tuned vision-language model (VLM) for real-time decision-making and task execution.  We evaluate LiMAC on two open-source mobile control datasets, demonstrating the superior performance of our small-form-factor approach against fine-tuned versions of open-source VLMs, such as Florence2 and Qwen2-VL. It also significantly outperforms prompt engineering baselines utilising closed-source foundation models like GPT-4o. More specifically, LiMAC increases the overall action accuracy by up to 19% compared to fine-tuned VLMs, and up to 42% compared to prompt-engineering baselines.

Published as a conference paper at ICLR 2025
LIGHTWEIGHT NEURAL APP CONTROL
Filippos Christianos", Georgios Papoudakis", Thomas Coste", Jianye , Jun Wang', Kun Shao
Huuawei Noah's Ark Lab, TTanjin University, 'A Centre, University College London filippos. christianos @aaoossshwwwi..eemmm com, georgios papoudaki sonedaaiss@uhweer ww1@hhawwi.comm com thomas. wwwtteoutteeeeeeoo com, hanjeiaayeeuweei com, jun wangêcs ucl ac. shaokun28huawei.com
ABSTRACT
This paper introduces a novel mobile phone control architecture, Lightweight Multi-modal App Control (LiMAC), for efficient interactions and control across various Android apps. LiMAC takes input textual goal and sequence of past mobile observations, such as screenshots and corresponding UI trees, generate precise actions. To address the computational constraints inherent to smartphones, we introduce a small Action Transformer (AcT) integrated with fine-tuned vision-language model (VLM) for real-time decision madisiionmaking and task execution. We evaluate LiMAC on two open-source mobile control datasets, demonstrating the superior performance of our falll-for--fctorr approach against fine-tuned versions of open-source VLMs, such as Florence2 and Qwen2-VL. It also significantly outperforms prompt engineering baselines utilising closed-source foundation models like GPT-4o. More specifically, LiMAC increases the overall action accuracy by up to 19% compared fine-tuned VLMs, and up 42% compared to prompt-enginecring baselines.
INTRODUCTION
Smartphone application agents, commonly known as app agents, are expanding the potential applica- tions of artificial intelligence to smartphones and other mobile devices. Such agents could allow users to accomplish range of tasks, from scheduling appointments and sending messages to purchasing items and booking flights, with minimal effort. Fundamentally, app agents observe user instructions and progressively interact with the smartphone's 's user interface-by clicking, scrolling, inputting text, etc.-to accomplish the task. However, due to the limited computational resources smartphones, these agents must be optimised for efficiency, employing lightweight models with minimal memory usage and fast processing speeds.
Recent advancements have leveraged foundation models to develop app agents that understand natural language instructions and execute complex user commands within the smartphone's 's interface (e.g.. Rawles et al. 2024 Bai al., 2024 Wang et al.) 2024b a). While foundation models enable sophisticated capabilities, relying on them for every action introduces significant drawbacks. Their substantial size and computational complexity make them resource-intensive and impractical for constant use on mobile devices. Alternatively, querying server-hosted foundation models, such as GPT-4o or Gemini, for each task can be prohibitively expensive due the operational costs of running large models, making this approach impractical for everyday applications. For example, ofate-of-the-art GPT-4o-based app agent (e.g., [Rawles al., (024) may require one two minutes to run and cost approximately $1.00 .00 per task on average, based on tasks from the evaluated datasets.
To address these limitations, we propose a gated architecture that combines a lightweight transformer network with small fine-tuned VI.M. The task description and the smartphone state are first processed by a compact model (~500 million parameters) which effectively handles most actions. For actions that require natural language understanding, such composing text message querying a search engine, a VLM invoked to generate the necessary text. This hybrid approach reduces
First authors with equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
computational demands and improves responsiveness, resulting in significantly faster execution times -0 times faster, down to 3 seconds per task on average and improved accuracy.
In the proposed architecture (Lightweight Multi-modal App Control, or LiMAC), the initial processing stage is managed by an Action Transformer (AcT), and is primarily responsible for determining the type of action required to fulfil a user's command. AcT first predicts the action type, such as clicking, inputting text, or scrolling, based on the current state of the smartphone's 's interface and the task description. For most action types, such clicks and scrolls, AcT autonomously executes the task. For predicting the targets of the cl ilic ck action, we employ a contrastive objective between the outputs of AcT and the embeddings of each user interface (UI) element. The specific approaches for predicting action types and handling click actions are detailed in Sections 3.3 and 3.5 respectively.
However, when the action type predicted by AcT is input a deeper prior knowledge and understanding of natural language nuances, LiMAC passes the selected action type and user's goal to a fine-tuned VLM to generate the appropriate textual content. This division of labour allows AcT to handle straightforward interactions while leveraging the VLM's advanced capabilities for more complex text generation tasks, ensuring that the system remains both resource-efficient and capable of sophisticated responses. The process of integrating and fine-tuning the VLM in the app agent domain is detailed in Section 3.4
or open- app, which necessitate
In summary, the primary contributions of this work are as follows:
We propose LiMAC, an architecture for app agents that balances efficiency and natural language understanding by combining a lightweight transformer with a fine-tuned VLM.
We also introduce AcT, submodule of LiMAC, which is designed to efficiently predict action types and UI element interactions, featuring novel contrastive objective for click prediction.
We fine-tune and evaluate two open-source vision-language models (VLMs) specifically for handling text-based actions. Our fine-tuned VLMs achieve performance comparable to or exceeding GPT-4o methods while only having 2B parameters less.
We present experimental results demonstrating that LiMAC improves both task execution time and accuracy up to 30 times faster and 40 % higher accuracy-compared to GPT-4o-based and fine-tuned VLM app agents.
TECHNICAL PRELIMINARIES
2.1 PROBLEM FORMULATION
We model phone interaction as a sequential decision-making process. Each task consists of a given goal g that should be completed during an episode. At each timestep t of the episode, the phone's internal state is denoted by 8t. while or represents an observation of this state, including screen captures and UI element trees. The set of visible UI elements on the screen at timestep is defined as L, with or,; representing the i-th UI element at timestep / where i E L. Each UI element is represented by three different components: the image that corresponds to the UI element that we denote or the text that corresponds to the UI element o,3 and the related attributes of the UI element, such as whether it is clickable or not, that we denote as alt Therefore, the representation of each UI element can be written as:
img txt attr o Ot 1,55 [[,]]
The agent interacts with the phone through actions, denoted as a at timestep t. Each action is char- acterised by two components: its type a.Ye= E Avre (e.g., cl i llick, scroll- -down, input-text) and its specifications at axpe E Assee The specifications vary based on the action type: for clicks, a appe might represent the targeted UI element; for typing actions, it would contain the text to be input. Thus, an action can be represented as the tuple a, = (a ) This formulation allows for a flexible representation of diverse actions while maintaining a consistent structure.
In this work, the main goal is to learn a model that will maximise action prediction accuracy, which corresponds correctly predicting both the action type well as the action specifications. To achieve this, we train AcT, which predicts aype If the predicted action type is cl i ck, AcT also predicts
2

---

Published as a conference paper at ICLR 2025
the 0e in the form of UI element targets. We focus click targets because they among the most difficult and common actions to predict, and AcT's architecture easily accommodates predicting them with contrastive learning approach (see Section 3.5) For actions that require natural language specifications (e.g., i input -text), we use a VLM fine-tuned on the same dataset.
2.2 SEQUENCE MODELLING WITH TRANSFORMERS
Transformers /Vaswani al. 2017) have demonstrated exceptional effectiveness in modelling and generating sequential data across wide range of domains. They excel in various sequence modelling tasks, including those related to language, video processing, and decision-making (Chen et al. 2021) Regardless of the specific application, transformers begin by converting the input into a sequence of vectors. For text, this involves tokenising the input, with each token represented by an embedding vector. In the case of images, the input is typically divided into patches, where each patch is similarly represented by vector, analogous the tokenisation process in text. These embeddings, which map tokens patches to vectors, can either be learned during the model's training or sourced from pre-trained models (e.g., [Devlin et al. 2018). The embeddings are fed through several multi- head self-attention layers, which are designed capture dependencies and contextual relationships between different embeddings in the imput sequence. These self-attention mechanisms allow the model to focus relevant parts of the sequence when processing each embedding, enabling it to handle long-range dependencies more effectively. After passing through multiple layers, each consisting of self-attention and feed-forward components, the final activations from the transformer's last hidden layer are passed through a linear (fully connected) layer. This layer is typically tasked with mapping the learned representations to the output space, whether for classification, prediction, or another specific task. The entire model is trained andtt--nd, with backpropagation adjusting both the attenttention layers and the final linear layer to optimise performance on the desired task.
THE LIGHTWEIGHT MULTI-MODAL APP CONTROL FRAMEWORK
Our methodology processes the user's goal y and the phone's state at time I, utilising AcT, to determine the action type aPPe If the predicted action type is either i input or open-app, then 9. and 0t are passed to fine-tuned VLM, which is responsible for determining the specific action at For actions involving clicks, AcT handles the prediction directly but employs a different training objective that contrasts UI element embeddings to determine the most likely interaction target. Accordingly, this section is divided into three parts: predicting the action type, predicting specific actions for text input and app launching, and predicting clicks using our novel approach for interaction with UI elements. The full architecture of LiMAC is presented below.
We refer to method lightweight because it uses fewer parameters average during inference than baselines and, as we will show in Section 4.3 has faster inference speeds. The AcT module only has 520M parameters and the additional VLM component is called for less than 15% of actions in our datasets. LiMAC also selects actions more efficiently than a single VLM, as AcT does not require auto-regressive generation. While our approach has higher memory footprint than solely using VLMs, due to loading both the AcT module and VLM, its low parameter count remains within the capacity of modern devices (Li al. 2024b; Laskaridis et al. 2024).
3.1 MODEL INPUTS
AcT, the model responsible for predicting the action type (and later the click target, as seen in Sec- tion 3.5).. is built on top of typical transformer architecture. However, unlike standard transformers. where tokens represent words or characters, our "tokens" are pretrained embeddings that are mapped to the hidden dimension of the transformer. These tokens represent three key components: the user's goal g. the UI elements on the phone's 's screen ot, and the possible actions. By using these pretrained embeddings as input, we allow the model to effectively capture the relationships between the user's intent, the current state of the interface, and the set of available actions. We encode each key component (UI elements, actions, and goal) into embeddings that can be processed by the transformer. Below, we describe the encoding process for each type of input.
Goal: We encode the user's textual goal g using a sentence encoder, resulting in the embedding cg fralg). This embedding captures the user's intent and serves as the first token to the transformer.
3

---

Published as a conference paper at ICLR 2025
phone timestep:
CLIP
61
61-1,
Jnss
Cp-L.
611
encoding
TYW
CLIP
prediction
Ciy
0d
BERT
eol
onc-hot encading
tppe
Figure Illustration AcT. separate encoding each element into a vector ēt,i by using pretrained embedding models. The embeddings are then fed into the sequence of transformer æt along with the previous timesteps in that episode. The prediction of the transformer is decoded to produce the next action which consists of a; type and a; spec
The observed representation of each UI element or, at time is transformed into a vector et,; through several embedding functions. First, the text component is encoded using a fea( (a,,))))) and the image is encoded using a fine-tuned CLIP img
Elements:
sentence encoder (c.g., BERT)
ext
tt
visual encoder (Radford et al. 2021 Ct, fumg (o (o)))) Additionally, any other attributes (e.g.. clickable, editable, nested) are encoded into atr
S'ate (o.tr) The final embedding for each UI element is the concatenation of these vectors, e,, [eaatr ext. (.f We fine-tune CLIP using the standard contrastive learning objective (Radford et al.) (222)) using the screenshot of the observations and the related UI trees allow adapting app control datasets. We also add a positional encoding E IRd to represent the order or nesting of UI elements: c = ctt Pi. This process is illustrated in Figure 1 To adapt the visual encoder fing to our task, we fine-tune using our dataset by minimising the InfoNCE loss (Oord et al. 2018), aligning image and text representations of UI elements. Similar methods representing each UI element as an embedding for transformer have been suggested by Li et al. (2020); Rawles et al. (2023). with the key distinction that our approach additionally fine-tunes the vision encoder to better adapt for app control tasks.
Actions: Each action is represented using two embeddings: the action type embedding which is mapped to its corresponding learnable embedding e and, for actions requiring a specification (e.g., the target of cl click action), the specification embedding esee Depending the action type, the action specification embedding is computed differently (e.g., sentence embedding for the textual action, learnable embeddings mapped the UI element's 's id for click targets, special token for empty specifications). Each action contributes two tokens to the transformer's input sequence, clearly separating action types from their parameters.
Positional Embeddings: To represent temporal information, we also add learnable positional encoding Pt for all the embeddings timestep.
3.2 CONSTRUCTING THE INPUT SEQUENCE
After generating the goal, UI elements, and action embeddings, we organise them into a sequence representing the entire episode. Each episode the dataset is encoded sequence of embeddings which is fed into the transformer. The sequence starts with the goal embedding cg. followed by the UI element embeddings % at timestep 0. Once all UI elements are encoded, special end marker and is added. The action type e type and specification eg embeddings for timestep are then appended. This process repeats for each subsequent timestep: encoding UI elements, appending and and adding the action embeddings. For an episode with H timesteps, the final sequence is:
[egi 60,0 - C0 cod, type spec. ...;; CH-10: ...; -
end, type spec
36 36 1--1e-1
4

---

Published as a conference paper at ICLR 2025
Input
LiMAC
Goal: "Order new Nike shoes"
VLM
Tree
ut elemenn stemant: 2-. UT
AcT
Final Action
Final Action
Type
selection
see Pigure
Figure The architecture of LIMAC. The history of observations cscervations-cctions for, az-1, 0c-1.. and goal are processed to vector x and passed to AcT. The image observation or with the bounding boxes and the goal are passed inputs to the VLM. The VLM is only called if action that requires text completion is selected, based on the action type output of AcT. The action is finally selected based on the protocol described in Sections 3.3 3.5
During training, the full sequence is fed into the transformer. For inference timestep the sequence up to the f-th observation is processed, with the hidden state he (up to end) used to predict the action.
3.3 ACTION TYPE PREDICTION
In our pipeline, the prediction of the next action begins with determining the action type. Predicting the action type dude can be framed as a classification problem, where we identify a combined eleven distinct action types (see Appendix AA), of which a subset are seen in individual datasets used in this work. These action types represent various possible interactions, such as c llick, open-app, or other essential commands. We implement the action type prediction with a specialised head. The action type head, denoted as fiype, transforms the final hidden state h2 of the transformer (after the end token) into probability distribution over the possible action types, p(a (h,) = Siyre(lu (lt). The learning objective for this task to minimise the cross-entropy loss between the predicted and actual action types. Given dataset D, the cross-entropy loss for action type prediction is defined as:
scroll-dowe
input-tex:
Lype - E EED [log(p(a>> h))]
(2)
Here, / represents the transformer's output corresponding to the final hidden state before action prediction, averaged over all observations in the dataset. This loss function ensures that the model is trained to correctly classify the action type based on the sequence of embeddings from previous steps.
3.4 LEVERAGING FINE-TUNED VLMs FOR TEXT GENERATION IN ACTION EXECUTION
As described in the previous section, our agent first predicts the action type. Among the eleven action types, two specifically require textual specifications: the specification is the text to be entered into a text box, and ii) the open-app action, where the specification the name of the application be opened. For these actions, we rely on fine-tuning a VLM using an app control dataset. The dataset provides action data in dictionary-like format, such as: ""action- type": "open- oppen-ap"" "'appnaam name" "Chrome ), with one key corresponding to the action type and another to the action specification. The VLM is trained generate the correct sequence tokens that corresponds to the successful completion of each action, optimising for the likelihood of generating the proper tokens based on the observation at each timestep.
the input-text action, where
During inference, after predicting the action type, AcT guides the VLM to start its response with this action type. For instance, if AcT predicts input the action type, the VLM is forced to
5

---

Published as a conference paper at ICLR 2025
begin its response with the token pattern: ("action-type" :inp--tttp"""ippt-teettee "input-text" "text": The model then completes the specification, producing a, qus the textual content needed for the action. The full action selection pipeline is presented in Figure
3.5 EFFICIENT CLICK TARGETING USING CONTRASTIVE OBJECTIVES WITH AcT
Having covered how action specifications are generated for textual actions, we now turn to the case of click ck actions, where the specification is the UI element interact with. To predict the correct UI element for click action, we employ a contrastive learning approach that operates over the entire episode, using cosine similarity and a learnable temperature parameter. Since the number of UI elements varies across timesteps and episodes, contrastive method is better suited than classification, which can suffer from class imbalance and limitations when handling more UI elements test episodes than seen during training. Let h: be the transformer's last hidden state up to embedding expe and furget be an affine transformation that projects the hidden states to an embedding space. Simultaneously, the hidden states of the transformer corresponding to the UI element embeddings, denoted as h. are also projected into the same embedding space:
80 furger (h!P)) and pd furgea (h"))
Assuming the embedding space lies in IRd, the query embedding qe has dimensions 1 x D, while the matrix p") representing all UI elements, has dimensions K x D, where is the total number of UI elements in the episode. The goal is to train the model such that q. aligns closely with the correct UI element's embedding at timestep I, using cosine similarity as the alignment measure. To achieve this, we adopt contrastive training techniques with the InfoNCE loss (Oord et al. 2018). We first compute the similarity matrix between the query embedding s. qyee and all UI element embeddings, scaling the similarity by learnable parameter 7 (e.g., Radford et al. 2021). The scaled cosine similarity matrix is defined as:
qp
S
dgll lpll.
where ||p||+ is the L2 norm of each row of p. For simplicity, we drop the superscripts in this equation. The InfoNCE loss for UI element selection across the episode is computed as:
log exp(S+)
Lelem
E
(5)
Li=1 exp(S.)
Here, S+ is the scaled similarity between the transformer's output and the correct UI element for the cl i ck action, and S; represents the similarity between the output and all other UI elements. During inference, for each action requiring target element, the UI element with the highest similarity is selected. This contrastive approach enables AcT to effectively learn which UI elements to interact with during click action by treating all other UI elements the episode as negative examples. The use of cosine similarity focuses on the directional alignment of the embeddings, while the learnable temperature adjusts the sharpness the similarity distribution during training, allowing for more flexible and precise UI element selection.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets: Our experiments focus on two open-source mobile phone control datasets, AndroidCon- trol (Li et al. 2024a) and Android-in-the- Wild (AitW) (Rawles et al.) 2023). Both contain extensive human demonstrations of mobile phone navigation across a wide variety of tasks. In AndroidControl,, every episode is defined by a specific goal, accompanied by sequence of observations and actions. Each observation includes a screenshot from the phone and its corresponding UI tree. Conversely, observations in AitW lack the UI tree. As result, it is necessary to extract the UI tree using OCR system that identifies all the UI elements and provides a brief description of each. More details on the goal format, observation space, and action space for each dataset can be found in Appendix A
GPT-4o Baselines: We compare our approach against four prompt-engineering baselines that use GPT-4 to generate actions in the evaluation dataset. First, we evaluate two baselines proposed by

---

Published as a conference paper at ICLR 2025
Table Comparison of models in terms of average inference time and overall accuracy on the AitW and AndroidControl datasets. The table presents the size of each model, the average inference time (in seconds, lower better), and the overall accuracy (higher is better) for both datasets.
Avg Inf. Time (s)
Overall f
Model
Size +
AitW
AndCtrl
SeeActchoice
unk
9.81
37.7
29.9
SeeActann
unk
9.76
42.5
35.5
T3A
unk
4.87
26.9
53.1
M3A
unk
10.64
35.6
57.5
Florence2
820M
0.50
70.8
57.0
LiMAC with Florence2 (ours)
+520M
0.34
72.2
63.1
Qwen2-VL
2B
3.03
51.0
52.2
LiMAC with Qwen2-VL. VL (ours)
+520M
0.63
70.9
62.5
Rawles et al. (2024): the text-based T3A and the multi-modal M3A. In T3A, the observation is represented as list of UI elements, while M3A includes screenshots of the observation. Additionally, we evaluate two variants of the SecAct agent (Zheng et al. 2024), adapted for mobile app control tasks (Rawles et al. 2024). Specifically, we assess two SeeAct variants: SeeAcfchokc and SeeActann which use the UI tree text and screenshots of the observations, respectively, to determine the correct action. More details about the prompt engineering baselines are presented in Appendix B
Vision Language Models (VLMs): We fine-ture tune two VLMs for our experiments. The first, Flo- renee2 Xiao et al., 2024), is an 220M-parameter VLM that takes as input an annotated screenshot with numbered bounding boxes, along with the task goal in natural language. Florence2 is trained to maximise the log-likeling likeliool of the correct action tokens from dataset. Similarly, we fine-tune Qwen2-VL VL (Bai et al. 2023). 2B-parameter VLM, using LoRA adapters (Hu et al. 2021). Qwen2- VL follows the same pipeline as Florence2, taking the annotated screenshot and goal as imputs, with supervision provided by the correct action. In most of our experiments, these fine-tuned VLMs tested in conjunction with ACT (forming LiMAC).
4.2 EVALUATION PIPELINE
We evaluate on the test set of two datascts, using the same process for all models, with only the observation format and model calling differing. For each timestep, we call the model with the relevant observation format generate action. VLMs are trained to return actions in specific format, while pre-trained models use a detailed prompt with the observation, as in Rawles et al. (2024). One can calculate strict accuracy by directly comparing returned actions the ground truth. However, in this work we relax this metric for a more practical assessment, where a UI element is deemed correct if its bounding box is within the target element, as described by Li al. (2024a). For i input actions, correctness is determined by a Jaccard index score of at least 0.5, reflecting the functional equivalence of similar inputs in search bars. We report the relaxed accuracy metrics in Tables I and
In Section 4.5 we also evaluate models' action-type and click-target accuracy. Action-type accuracy reflects how well the model predicts the correct type for an action, regardless of the specifications such the text content target element. Click-target accuracy measures how accurately the model predicts the correct target for cl i ck actions when the action type known. Computing the click- target accuracy requires rerunning full evaluation over the dataset, where the output of the model constrained to predict the click action and specify the target element. It should be noted that higher overall accuracy can still be achieved with lower action-type and/or click-target accuracy. This because -taret accuracy is calculated separately, and because not all action types are equally advantageous for overall accuracy. Indeed, as defined in Section 2.1 action is represented as a, (o spoc), where both 2 and spec must be predicted correctly for successful timestep. Actions which always have null a like wait, are easier to predict correctly than those which have complicated spec that may be incorrectly predicted, like input-text
7

---

Published as a conference paper at ICLR 2025
Table Performance comparison of various model configurations using different combinations of modules across the AitW and AndroidControl datasets. Using LiMAC but integrating AcT with baseline methods improves accuracy and reduces inference time (and cost). Not all pairings are shown here for conciseness, the full list can be found in Table
Modules Used
Avg Inf., Time (s)
Overall
Framework
Type
Click
Text
AitW
AndCtrl
T3A only
T3A
T3A
T3A
4.87
26.9
53.1
LiMAC (ours)
AcT
T3A
T3A
4.03
42.7
65.4
LiMAC (ours)
AcT
AcT
T3A
1.04
69.8
63.2
M3A only
M3A
M3A
M3A
10.64
35.6
57.5
LiMAC (ours)
AcT
M3A
M3A
8.40
52.6
66.8
LiMAC (ours)
AcT
AcT
M3A
1.87
70.0
62.5
Florence only
Florence2
Florence2
Florence2
0.50
70.8
57.0
LiMAC (ours)
AcT
Florence2
Florence2
0.72
71.6
61.1
LiMAC (ours)
AcT
AcT
Florence2
0.34
72.2
63.1
Qwen only
Qwen2-VL
Qwen2- VL
Qwen2-V VL
3.03
51.0
52.2
LiMAC (ours)
AcT
Qwen2-VL VL
Qwen2- L-L
2.64
55.7
59.1
LiMAC (ours)
AcT
AcT
Qwen2-VL.
0.63
70.9
62.5
LiMAC (ours)
AcT
M3A
T3A
7.57
52.4
67.4
4.3 MEASURING END-TO-END ACCURACY
In this section, we present the total action accuracy of our method, as well as the baselines. Table present the accuracy for action prediction in AndroidControl and AitW, respectively. In both AitW and AndroidControl, we observe that LiMAC consistently outperforms Florence2, Qwen2- VL, and GPT-4o-based baselines with respect to the action prediction accuracy, demonstrating superior generalisation to the held-out test set. The overall improvement of LiMAC in the accuracy compared AndroidControl can be attributed to the closer alignment between the training and test sets, the test set includes the same set of instructions but applied to mobile devices with varying characteristics, such screen size and Android version. Additionally, we observe a significant performance drop in text-based baselines like T3A and image-text--aas models like M3A and SeeAct. The absence of original UI trees the AitW dataset can explain this decline. Since UI trees must be extracted from images using an annotation tool, inaccuracies are often introduced, which diminishes the performance of models that rely on text-based output conditioning. This underscores a key advantage of LiMAC, which remains robust even when UI trees are imprecise completely missing (as seen in Table 4 with minimal impact on overall performance.
4.4 COMBINING DIFFERENT MODULES
LiMAC a modular architecture that enables the integration of different modules for tasks such as predicting the action type, identifying the target element in clich ck actions, and generating text for open-app and input-text. type and the target element for cl i ck actions. However, alternative modules can be employed for these predictions as well. Table we present combinations of different models, excluding SeeAct due to its low overall accuracy, and compare their performance across two datascts.
this architecture, we primarily use AcT to predict both the action
In the AndroidControl dataset, we observe that using M3A for predicting the target elements in click actions improves performance over using AcT alone. This demonstrates that GPT-4o is highly effective at identifying the correct target element when the prompt specifies that the action cl click. This of courses comes the cost of calling GPT-4o, which significantly increases the inference time. The highest overall accuracy is achieved when LiMAC is used to predict the action type, M3A is applied for target element prediction, and T3A is used for text generation. In the AitW dataset,
8

---

Published as a conference paper at ICLR 2025
Table Action-type. click-target, and accuracies across module combinations on the Ait and AndroidControl datasets. LiMAC achieves the best action-type accuracy in both datasets and the best click-target accuracy in AitW, while our fine-tuned Florence2 excels text prediction.
Modules Used
Action Type
Click Target
Text
Framowork
Type
Click
Text
AitW
AndCul
AitW
AndCtrl
AitW
AndCurl
SeeAct only
SeeAclgtoie
SeeAcfchocc
SeeAckctocc
67.1
66.8
36.9
48.5
69.4
67.1
SccAct only
SocActam
SecActun
SocActun
68.2
66.8
44.7
55.7
66.0
61.8
T3A only
T3A
A3A
T3A
56.2
67.7
33.5
71.1
66.5
78.4
M3A only
M3A
M3A
M3A
63.8
69.8
48.3
77.1
67.3
74.3
Qwen only
Owen2-VL
Qwen2-Ve
Qwen2-Ve
81.7
70.7
53.2
55.2
70.5
75.7
LiMAC (ours)
AcT
Qwen2-VI.
Qwen2-VI.
86.9
82.3
53.2
55.2
70.5
75.7
LiMAC (ours)
AcT
AcT
Qwen2-VT.
86.9
82.3
77.4
65.4
70.5
75.7
Florence only
Florenoc2 ACT
Florenee2
Florence2
86.4
79.6
76.2
62.0
84.2
77.5
LiMAC (ours)
Florence2
Florenc22
86.9
82.3
76.2
62.0
84.2
77.5
LiMAC (ours)
AcT
AcT
Florence2
86.9
82.3
77.4
65.4
84.2
77.5
Table Evaluation of three ablated versions of LiMAC using different types of input, on Android- Control. For actions that require text completion, we use the fine-tuned Florence2.
Size
Action Type
Click Target
Overall
LiMAC LiMAC (no CLIP FT)
520M
82.3
65.4
63.1
520M
81.9
62.3
60.0
LiMAC (no img)
433M
82.4
54.9
56.0
LiMAC (no txt)
410M
83.2
65.7
63.0
LiMAC combined with Florence for text generation yields the highest accuracy. This outcome is expected, as both M3A and T3A show significantly lower accuracy in this dataset (see Table
4.5 ABLATION STUDIES
Table presents the action-type, click-target, and text accuracies for various module combinations across the two datasets. The results show that LiMAC, particularly the AcT, achieves the best performance in action-type prediction. the AndroidControl dataset, M3A and T3A perform well in click-target and text prediction but struggle with action-type accuracy, and they underperform in the automatically annotated AitW dataset. Overall, AcT within LiMAC excels at click-target predictions while being significantly smaller. Finally, our Florence fine-ture excels at text prediction, significantly outperforming GPT-4o baselines in AitW and remaining competitive in AndroidControl.
Lastly, present three ablation studies further explore AcT design choices. A feature of AcT its ability to process each UI element as a distinct embedding within the transformer, created by concatenating the image, text, and attribute embeddings of the corresponding UI element. To assess the impact of the image and text modalities, as well as the CLIP fine-tuning on LiMAC's performance, we compare it three ablated versions: one that excludes the image component, another that omits the UI text in the embedding process, and one that uses the original CLIP for encoding the image embeddings instead of the fine-tuned version. The evaluation metries for these comparisons in the AndroidControl dataset and using Florence2 for text completion are shown in Table 4 The results demonstrate that removing image embeddings significantly reduces accuracy across metries, highlighting the crucial role of visual information in AcT. In contrast, omitting the text embeddings has only slight effect on performance, suggesting that AcT can function effectively using only screenshots of observations without accessing the UI tree. Additionally, observe that fine-tuning CLIP (see Section 3.1) is an important factor in improving the overall accuracy of LiMAC.
These findings underscore the importance of visual features and the benefits of fine-tume fine-tuning pre-trae rre-trained models like CLIP in our context. The minimal impact of removing text embeddings indicates that LiMAC is robust even when textual information is limited unavailable, which is adyantageous in

---

Published as a conference paper at ICLR 2025
scenarios where UI trees are inaccessible or incomplete. Future work could explore integrating other modalities or further optimising the embedding process to enhance performance.
5 RELATED WORK ON APP CONTROL
Though graphical user interface (GUI) control mainly started with web-based datasets and foundation
model agents (Shi et al. 2017: Liu et al. 2018 Yao et al. 2022a; Deng et al.. 2023 Furuta et al. 2023 Gur et al. 2023 Zheng et al. 2024). there has recently been a significant focus on mobile phone control. This can be seen both by the rapid development of Android navigation datasets, environments,
and benchmarks (Rawles et al. 2023 2024 Li et al.) 2024a Chen et al. 2024), and of mobile control
2023 Hong et al. 2024 Rawles et al.) 2024; Li et al., 2024aa] Bai et al. 2024 Wang et al. (202c)). Though many agents are published with
agents (Yang al. 2023 Wang
2222bb] Wen al.
their own specific evaluation data, popular datasets such as Android-in-the. thil (Rawles et al.) 2023) or AndroidControl (Li et al. 2024a) are often used as benchmarks. Agents developed for this task
can be divided into two clear input types: text-based, using UI accessibility tree or XMI. information to describe the screen, or image-based. Image-based agents require vision models, which are capable of directly processing image inputs, and are usually backed by VLMs. On the other hand, text-based agents are backed by classical LLMs. Image-based agents also often take combination of text and image as input to the model. Many mobile control agents propose intricate prompting methods backed by off- the-shelf, often proprietary. LLMs such GPT-4 (Rawles et al.) [0224] Yang et al., [023 Wang
let al. (2024]la a) WWen et al.) 2023] Zheng et al. 2024). Although this requires little to no training, be both slow and expensive. Moreover, these models cannot be further tailored and trained for specific tasks. As such, another approach is to build mobile control agents around fine-tuned of foundation models on Android control datasets such as AitW or AndroidControl. Firstly, both AitW and AndroidControl present results for fine-tuned LLM on their dataset, alongside the dataset itself. For example, [Li et al. (024a) train various PaL.M2 (Anil et al. (023) models on their dataset. However, these models are proprietary and supposedly quite large, with the base PaLM 2 model reported to have over 300B parameters. CogAgent (Hong et al. 2024) also performs fine-tuning on an 18r-large VLM. Bai et al. (2024) propose a different approach, called DigiRL, using RL to train their 1.3B VLM. This achieves strong performance but has limitations such as gathering cost and simulation difficulty, leading to the model only being adept on small subset of AitW.
6 CONCLUSION
In summary, we propose LiMAC, lightweight framework designed to address app control tasks. LiMAC extracts UI elements from each phone screenshot and encodes them using specialised vision and text modules. These UI element encodings are then passed embeddings to AcT, which predicts the type and specifications of the next action. AcT focuses on two key aspects of actions: the action type and the target element when the predicted action is click. For actions requiring text generation, LiMAC uses a fine-tuned VLM to ensure successful completion. We compare LiMAC against six baselines supported by state-of of-the-ar art foundation models and evaluate them on two open-source source datasets. Our results show that LiMAC can outperform the baselines while requiring significantly fewer computational time for both training and inference. This demonstrates that LiMAC is capable of handling task completion on devices with limited computational capabilities.
One of the main limitations of the proposed method is the limited training data. LiMAC is trained on just 13K and 18K episodes for AndroidControl and AitW, respectively. The absence of any pretraining further hinders the model's ability to improve performance on more complex tasks. In the future, we aim to enhance the model's performance by incorporating online learning techniques, such as reinforcement learning. After the initial training stage presented in this work, LiMAC could interact with an Android emulator to generate additional data. By using suitable reward function, or even leveraging GPT-4 to evaluate the generated trajectories and assign rewards (Bai et al. 2024), could fine-tune LiMAC to improve the completion rate of tasks. An important focus for future work will be to develop error handling and recovery mechanisms to enable high success rates and robustness in online interactions. Another area of future research could address the safety of such models when handling sensitive data, such as credit card information and personal identifiers. It is essential design foundation models with robust security protocols to protect against data breaches, especially when interacting with mobile phones containing sensitive information.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (Grant Nos. 62422605, 92370132).
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training wiil device-control agents with autonomous reinforcement learning. arXiv preprint arXiv:2406. 1896, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 2023.
Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Kaiwen Zhou, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye Hao, Jun Wang, and Kun Shao. Spa-bench: A comprehensive benchmark for smartphone agent evaluation. 2024.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 2021.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2023.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Hiroki Furuta, Kuang-Huud Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetueed foundation models. arXiv preprint arXiv:2305.1 / 17844, 2023.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint urXiv:2307.12855 2023.
Wenyi Hong, Weihan Wang, Qingsong Lx, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14281-14290, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.09685 2021.
Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, and Hamed Haddadi. Melting point: Mobile evaluation of language transformers. 2024.
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv preprint 03X6::2406.0367, 2024a.
Xiang Li. Zhenyan Lu, Dongqi Cai, Xiao Ma. and Mengwei Xu. Large language models on mobile devices: Measurements, analysis, and insights. In Proceedings of the Workshop on Edge and Mobile Foundation Models, pp. 1-6, 2024b.
11

---

Published as a conference paper at ICLR 2025
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instruc- tions to mobile ui action sequences. In 58th Anmual Meeting of the Association for Computational Linguistics, 2020.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using warkflow-guieed exploration. In International Conference on Learning Representations (ICLR), 2018.
Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiiv:80...3777 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, 2021.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36, 2023.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Camphell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405. 14573, 2024.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pp. 3135-3144. PMLR, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Junyang Wang. Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Zhang, Fei Huang. and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint wxXiv2246.010014 2024a.
Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint ar ar11v:2401.16/58. 2024b.
Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asynchronous distributed reinforcement learning framework for on-device control agents. 2024c.
Hao Wen, Yuanchun Li. Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference Mobile Computing and Networking, pp. 543-557, 2023.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing unified representation for variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 1818-8829, 2024.
Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint 1311 2023.
12

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, arXiv preprint arXiv:2210.03629, 2022b.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is generalist web agent, if grounded. arXiv preprint 4rXiv:2401.01614, 2024.
13

---

the provided Published as a conference paper at ICLR 2025
A DATASET FORMAT
We use AndroidControl and AitW dataset. While we use the full AndroidControl dataset, for AitW we only select a few episodes for each unique instruction, due to the sheer size of the dataset and the repetitive nature of its instructions. The dataset is divided into five categories of tasks, of which use the 'GoogleApps', 'Install', and "WebShopping' splits, since the other two contain single-step and Q&A tasks. We process the episodic data present in these datasets into refined, step- sip-wis,, datapoints which can be used for training and evaluation. Each datapoint is composed of the high-level goal for the task, observation of the current screen, and the correct action. Details are given below.
Goal: The goal is always a raw text string describing the high-level instruction for the episode to which the datapoint belongs.
The exact form of the observation depends on the type of model it is used for. Text- based approaches such T3A need a textual observation. For AndroidControl, accessibility UI trees, which we further process into a list of UI elements containing information such the element type, description, and attributes (clickable, editable, selected, etc..... Like in Li et al. (2024a), we filter these to retain only important elements, namely those that contain text or have critical attributes. For AitW, OCR representations of text and icons are given in the dataset, but no comprehensive UI trees are provided. Therefore, to obtain the final representation, each element must be identified and converted using similar procedure to that for AndroidControl. Vision models such as Qwen2-VL and Florence2 will expect an image-based observation. This observation will consist of the current phone screenshot along with an overlay of the UI element bounding boxes and their index. Finally, some models, such as M3A and ours, use mixture of observations, both text and image-based. In particular, our model expects a text-based list of UI elements similar to the one described above, as well list of cropped images. The list of cropped images corresponds each of the UI elements in the text-basee observation and is used by our model as described in Section 3.1
Observation:
Action: Action grounding is crucial part of mobile phone control, as in previous works (Zheng iet al. 2024 Yang et al. 2023 Wang et al. 2024b a) and both the datasets we use, we define fixed action space, seen in T Of this action space, open-app, wait, and long-press do not feature in Ai while navi aevigate-home does not feature in AndroidControl. Information for most actions sourced directly from the datasets, with only the action name at times varying. The only exceptions to this are the cl lckk and 1 long-press actions, which require target element, rather than x-y coordinates. For these, we select the best matching candidate from the observation list of UI elements. The action takes specific JSON format we expect the models to match to facilitate parsing, which simply a dictionary with the action type and an action specification (see Table An example would be: ("action-type
"open-app" "app-name" " "Chrome") ".
Table 5: Agent action space, along with the relevant datasets.
Action type
Action specification
AitW
AndCtrl
open-app
<app-name>> starget-elente
X
click
long-pres
<target eengee-elle ellemet> <text>
X
input-text
scrolllfupp/omm//tf////tttttt
navigate-home
navigate-back
wait
B PROMPT ENGINEERING BASELINES
We evaluate four prompt engineering methods leveraging GPT-4o to generate actions. First, we assess two baselines proposed by Rawles et al. (2024) text-based method (T3A) and a multi- modal approach (M3A). In both methods, GPT-4o generates a summary of the previous timestep by reflecting on prior actions, the current observation, and previous observations and actions. GPT-4o
14

---

Published as a conference paper at ICLR 2025
then generates proposed action in a ReAct-like (Yao al. 2022b) fashion using detailed prompt that includes task guidelines, action space descriptions, previously generated summaries, and the current observation. In T3A, the observation is represented list of UI elements, while in M3A, also includes two screenshots: one of the original image and another with UI element bounding boxes.
The final two prompt-engineering baselines are SeeActahoice and SeeActan (Zheng et al. 2024). In both methods, GPT-4o is prompted with the current task and screenshot from the observation to generate a high-level description of the proposed action. This proposal is then passed to GPT-4o for determining the final action, including both the action type and its specifications in the appropriate format. In SecActchoice, multiple-choice list of textual UI element choices is appended to the prompt to allow GPT-4o to predict the action specifications, such as the target element in c iick actions. In SeeActam. the observation's s screenshot is annotated with bounding boxes and labels for each UI element. We base our implementation off the SeeAct agent in Rawles et al. (2024), which adapted to app control tasks.
C IMPLEMENTATION DETAILS
AcT is compact transformer based on GPT-2 architecture. The transformer consists of 24 layers and 16 heads per layer. The hidden dimension of the transformer 1024. We apply a dropout rate of 0.3 (Srivastava al., 2014) during training across all layers. The AdamW optimiser (Loshchilov et al.) 2017) is used in all experiments, with learning rate of 3 x 10 specifically for AcT. The functions Juye and furget are implemented as two-layer fully connected networks, each with hidden size of 4096 and a dropout rate of 0.3. We use a batch size of with gradient accumulation being set to 32.
We fine-ture tune Florence2 for 10 epochs, starting with an initial learning rate of 10-6, which is gradually reduced to zero during training. The batch size is set to 2. with gradient accumulation configured to 8. 64, beginning with an initial learning
For Qwen2-VL. we employ LoRA with a dimensionality rate of 10-1, also gradually decreasing to zero throughout training. The batch size for Qwen2-VL. is 1, with gradient accumulation similarly set to We fine-tuned Qwen2-VL VL for epochs.
D ADDITIONAL STUDIES
This section presents additional evaluation results for LiMAC.
D.1 EXTENDED SUCCESS RATE TABLE
In Table 6 we provide a full set of evaluation metrics for the baseline models, as well as for various combinations of LiMAC with other methods. These combinations are used to predict the target element in click actions or generate text for specific actions, such as open. app and input-text In all the experiments involving LiMAC, AcT employed to predict the action type, while different combinations of methods are used to predict the action specifications, such as the target element or text generation. This approach allows us to isolate the impact of each combination on performance while maintaining consistent action type prediction. This table extends the results already presented Tables to providing a more in-depth understanding of the performance across a range of metrics. This additional breakdown offers clearer understanding of how LiMAC performs when integrated with other methods, offering insights into the strengths and potential trade-off's of each combination in different scenarios.
D.2 CONFUSION MATRIX
Figure 3 shows the confusion matrix for action type prediction using LiMAC on the AndroidControl dataset. The results indicate that actions like open-ape and input text are generally easier to predict compared to other actions. One of the most frequently mispredicted actions is wait which is unsurprising given that it can be challenging, even for humans, to determine when this action is required. Additionally, actions such as 1 long-press and swipe in any direction are often misclassified, likely due to their relatively low occurrence in the training dataset compared to other actions.
15

---

Published as a conference paper at ICLR 2025
Table 6: Comprehensive table of accuracy results for different modules, All rows which have AcT for the action type module fall under our LiMAC framework.
Modules Used
Action Type
Click Target
Text
Total
Type
Click
Text
ATTW
AndCtr
AITW
AndCtr
AITW
AndCtr
AITW
AndCtr
AcT
AcT
Florenec2
86.9
82.3
77.4
65.4
84.2
77.5
72.2
63.1
ACT
Florence2
Florence2
86.9
82.3
76.2
62.0
84.2
77.5
71.6
61.1
ACT
ACT
Qwen2-Ve VL
86.9
82.3
77.4
65.4
70.5
75.7
70.9
62.5
ACT
Qwen2-VL
Qwen2-VL
86.9
82.3
53.2
55.2
70.5
75.7
55.7
59.1
ACT
AcT
T3A
81.7
77.6
65.4
66.5
78.4
69.8
63.2
AcT
T3A
T3A
85.3
81.7
33.5
71.1
66.5
78.4
42.7
65.4
ACT
M3A
T3A
85.3
81.7
48.3
77.1
66.5
78.4
52.4
67.4
AcT
ACT
M3A
85.3
81.7
77.6
65.4
67.3
74.3
70.0
62.5
AcT
T3A
M3A
85.3
81.7
33.5
71.1
67.3
74.3
43.0
64.7
ACT
M3A
85.3
81.7
48.3
77.1
67.3
52.6
66.8
AcT
AcT
SecActehoice
85.3
81.7
77.6
69.4
70.5
62.0
ACT
SeeAcTahice
SeeActaacue
85.3
81.7
36.9
48.5
69.4
67.1
45.7
$3.7
ACT
AcT
SeeAcla
85.3
81.7
77.6
65.4
66.0
61.8
70.0
61.1
ACT
SecActan
SecActy
85.3
81.7
44.7
55.7
66.0
61.8
49.2
6166
Florence2
Florence2
Florence2
86.4
79.6
76.2
62.0
84.2
70.8
57.0
Qwen2-VL
Qwen2-VL
Qwen2-VL
81.7
70.7
53.2
55.2
70.5
$1.0
52.2
T3A
T3A
T3A
56.2
67.7
33.5
71.1
66.5
78.4
26.9
53.1
T3A
M3A
T3A
56.2
67.7
48.3
77.1
66.5
78.4
30.9
55,2
M3A
T3A
T3A
63.8
69.8
33.5
71.1
66.5
78.4
27.0
$3.5
M3A
M3A
T3A
63.8
69.8
48.3
77.1
66.5
35.8
57.7
SeeAclanic
Ace
Acechlee
67.1
66.8
48.5
38.9
SecAclan
SeeActan
SeeActun
68.2
66.8
44.7
55.7
66.0
61.8
34.3
45.7
Confusion Matrix Action Type Predictions LiMAC
Walt
160
75
1750
81
33
Long
1500
Open
Actual Action hype
Input Text
5
14
265
1000
Navg. Back
32
73
750
Scroll Down
12
75
2
307
Scroll Up
21
Swipe
Right-
Watt
Open App
Input Text
Back-
Scroil Down.
Swipe Left
Swipe Right
Scroll
Long
Navg.
Predicted Action Type
Figure 3: Confusion matrix for action type selection for LiMAC in AndroidControl.
D.3 FAILURE ANALYSIS
We also examine the failure patterns of LiMAC, using Florence2 as the VLM, across the two datasets studied. Figure 4 displays the frequency of these failures, categorised by the type of failure in predict- ing either the action type or the action specifications. Specifically, within the action specifications, failures occur in areas: incorrect prediction of the click target and inaccurate generation of input
16

---

Published as a conference paper at ICLR 2025
text by the VLM. In both datasets, the most common type of failure is misclassification of the action type, closely followed by failures in predicting the click target. These findings underscore the key challenges that research app control should address.
Frequency of Failure Types
1000
AndroidControl
Frequency
600
400
Action Type
Click Target Type of Failure
Input Text
Figure Relative frequency of different types of action prediction errors in the two datasets
D.4 UI ELEMENTS SCALABILITY
In this section, we assess how the number UI elements in an observation impacts the success and failure rates of action prediction. Figure displays the number of successful and unsuccessful action predictions made by LiMAC, categorised by the number of UI elements. The results are grouped into bins of ten the x-axis. The number of UI elements extends up 150 in AitW and up to 290 in AndroidControl. However, for clarity, only bins containing more than five samples are included in the figures. Overall, the data suggests that the rate of failed action predictions increases slightly the number of UI elements grows. This trend is expected since accurately predicting the target of el ck actions becomes more challenging with more UI elements present.
AitW Predictions
AndroidControl Predictions
Cornect
1200
600
Incorrect
Number of Predictions
2000
5.00
400
200
200
100
100
119
100
110
120
130
140
Number Ui Elements
Number UIl Elements
Figure 5: Number of successful and failed prediction of actions with respect to the number of UI elements in the observation, for the two datasets.
17

---

Published as a conference paper at ICLR 2025
E
CASE STUDIES
Some sample episodes from AndroidConttol, including agent predictions, are provided in Figures 6 and These are provided for illustration purposes, as well as to further explain 'relaxed' accuracies and an example failure. Figure 6 presents both an instance of relaxed target element in the third timestep and failed input text action in the final timestep. Figure 7 shows a relaxed input-text action in the fourth timestep and an otherwise successful episode. Further details are provided in the figure captions.
wann to fighttrrmm Detroit Vegas business passengers Expedia departing Octoter and returning October 16, 2023 because organising fimilyy vacation Vegas.
Dorrect
Correct action: type"
Correct "turget wwwwrnnn
Correct action: 'waa Twerrf))
Correct action [bakknaype"
Correct actios Lypee toper
Partion hyper sppr, name" Expectir)
Predieted action: [anton_oye:: looon app.. upo_namat" Espacia)
Predidted actiore (action Jypel clat. bangel 17)
Predicted actiinn (action Jype'
Preskcted actions fact Noct inst Detrof))
Predicted actions
Predicted action: (bestor, 1yper "rrpat (wet (Deerol?)
Dectton, Jypa target Nomentt 27)
Taagar,
Figure 6: Relaxed target element in yellow (timestep 3) and failed action in red (final timestep). The target element of the cl click in timestep 3 is considered correct under our relaxed accuracy because its bounding box is almost identical the correct element, and clicking either would have the same effect (opening the text bar). In the final timestep, the agent inputs text 'Detroit' rather than "Las Vegas', clear confusion between the origin and destination of the trip stated in the goal, leading to an incorrect prediction.
Goal: 7d like togett new three scater sofa for Christmas becouse old one broke, so Il seek for't the molustyybuping
[bctton_yspe"
[aptio_lype' wall)
(becton
jumi: Insustayyuuy
target swementt
Tust: 3 seater solas]
sdementt
Eed of epiteds
Preditted action
Fredicted actions [bettin, byw" weer)
Predicted actionn sype
Prodicted actions
Prodicted artion (bctko_ope""
hype" open_aapt. app_rams: Induayyuuyrg)
Jype tapur Joot
Figure 7: Relaxed input-text in yellow (timestep 4) and overall successful episode. Timestep is considered correct under our relaxed input -extt textual component because it is simply the singular form of the correct text, leading Jaccard index greater than 0.5 and presumably the same search results. The episode terminates successfully, with all timesteps being considered correct under our evaluation metrics.
18


Reviews:
[{"review_id": "ltYxHJW2O1", "reviewer": "Reviewer_mpP2", "summary": "The paper introduces Lightweight Multi-modal App Control (LiMAC), a framework designed for efficient mobile app control by combining a small Action Transformer (AcT) and a fine-tuned vision-language model (VLM). LiMAC processes user goals and smartphone states, making real-time decisions through AcT for action types like clicking or scrolling. For complex tasks requiring text input, it leverages the VLM to generate appropriate content. Evaluation of LiMAC on AndroidControl and Android-in-the-Wild datasets, LiMAC shows superior accuracy and speed over traditional VLMs and foundation models like GPT-4o, achieving up to 42% higher action accuracy and reducing task completion time by 30-fold. This approach emphasizes efficient model use on resource-limited devices, while future improvements may incorporate reinforcement learning for enhanced performance.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- LiMAC combines a small Action Transformer (AcT) with a fine-tuned vision-language model (VLM). This hybrid approach is tailored to the computational constraints of mobile devices, achieving efficient and accurate control without relying on large, resource-intensive models. The AcT independently handles common actions, while the VLM is selectively employed for complex natural language tasks, optimizing both resource usage and response time.\n- LiMAC’s modular structure supports the integration of different models for specialized tasks, such as using AcT for action type prediction and click targeting, while optionally substituting modules for specific needs (e.g., Qwen2-VL for text generation tasks). This flexibility enables developers to adapt LiMAC easily for varied app control tasks, optimizing specific components without overhauling the entire architecture​.\n- The AcT component employs a contrastive objective to identify UI elements for click actions, using cosine similarity and a learnable temperature parameter. This approach is advantageous in handling class imbalance and dynamically varying UI elements across episodes. The use of contrastive learning allows AcT to focus on directional alignment in the embedding space, facilitating precise UI element targeting even in dense or complex layouts", "weaknesses": "-  Although the paper evaluates LiMAC on two datasets, both datasets are relatively specific to Android applications, potentially limiting the generalizability of results to other operating systems (e.g., iOS) or app control tasks with distinct interface designs.\n- The paper does not provide an extensive scalability analysis of LiMAC’s architecture as task complexity or the number of available UI elements increases, which may impact its robustness in more complex or densely populated app environments.\n- LiMAC operates within a fixed action space, which could restrict its adaptability to applications requiring more dynamic or unconventional actions not included in its current design. This might hinder its flexibility in expanding to novel app interaction scenarios.\n- The evaluation is conducted on simulated datasets without testing LiMAC’s performance on actual mobile devices. This limits understanding of its practical usability, particularly concerning latency, responsiveness, and potential challenges from hardware constraints and real-world conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AqEd9n5Ho0", "reviewer": "Reviewer_Wgue", "summary": "This paper introduces LiMAC (Lightweight Multi-modal App Control), a new architecture designed for mobile app control that combines a lightweight transformer, AcT, with a fine-tuned vision-language model (VLM). The standout feature here is the gated architecture, which smartly assigns tasks: AcT handles basic interactions like clicks and scrolls, while the VLM is called upon only for text-based actions that require deeper language comprehension. The approach yields substantial improvements in both inference speed and accuracy on two mobile action datasets.", "questions": "See my suggestions in the weakness section, here are some of my questions:\n\n1. How does the model handle dynamic UI elements or applications with frequently changing interfaces? Do you need to retrain a model for each software update?\n\n2. What is the impact of the VLM size on the overall performance? Could one larger VLM learn to solve all the tasks?\n\n3. How does the system handle errors or recovery from incorrect actions? Or safegaurding it from performing unwanted actions (for example send out user's passwords to someone else).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Novel design.** The authors designed a lightweight module to predict the type of actions to be taken, and execute simple actions with this light-weight module directly. Leaving the VLM to solve complex tasks that involve text generation. This leads to both performance speed-up and better accuracy.\n\n2. **Thorough evaluations.** I like how the authors compared using AcT/VLM for different tasks, clearly demonstrating the performance gain by adopting the current design, which makes sense to me.\n\n3. **Good writing.** The paper is easy to follow.", "weaknesses": "1. **Limited Dataset and Tasks.** The authors used two datasets of relatively small size, this paper could benefit from larger-scale experiments and maybe real-world user studies.\n\n2. Due to the limited data size, the proposed model may have additional difficulties in solving difficult tasks (which is where the mobile AI is needed to most, from my opinion). More studies/analysis on failure mode could make this paper better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lLHkfKW1CS", "reviewer": "Reviewer_j9ZA", "summary": "The paper presents a new framework, called LiMAC, for lightweight and efficient Android navigation. Combining text and image embeddings, an action transformer predicts which of ten actions needs to be taken, and, depending on this output, a fine-tuned VLM may be queried to help with more complex tasks, such as text input. A major advantage of this setup is its significantly reduced inference time requirements compared to models such as SeeAct, while attaining higher overall performance on the Android-in-the-Wild and AndroidControl datasets. The authors apply ablation studies to demonstrate the usefulness of CLIP fine-tuning and the image embeddings, as well as the robustness of their setup to the absence of UI tree descriptions. In all, LiMAC is just a 520M-parameter addition, allowing it to run efficiently directly on Android devices.", "questions": "Please see the weaknesses section, parts 1–4.\n\n5. You mention that LiMAC \"address[es] the computational constraints inherent to smartphone\". Have you tried running it on a smartphone? If so, how well did it go?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is incredibly clear and well-written. I am not an expert on Android or UI agents, but it was obvious what the contribution was, and why it was important. The small number of parameters in LiMAC make it clear how this advances the ability of users to run advanced Android control models directly on devices. The way the information is encoded is also very thoroughly described. It is somewhat difficult for me to evaluate novelty, but, assuming the related works section does not have any glaring omissions, is a novel approach to solving this problem. The figures are very clear and well-made, and efficiently convey the achitectural design to the reader. Finally, the ablation experiments are well-run, and justify the use of the multiple submodules, as well as demonstrating robustness to a lack of a UI hierarchy description, implying that this method may be applicable out-of-the-box to completely novel Android environments.", "weaknesses": "I am not an expert in this area, so it is difficult for me to point out major weaknesses—the paper clearly states a contribution, and is very self-contained. However, there are several minor things that were unclear to me, where the paper might benefit from more detail:\n\n1. The paper mentions that positional encodings are used to represent the nesting of UI elements. How is this done, exactly?\n2. It would be good to know what the \"ten distinct action types\" are—it seems like the same few examples are given at several points in the paper, and only a few are focused on. Are the rest omitted because they're not very interesting and \"just work\", or are very similar to each other, or something else?\n3. In Table 1, is Florence2 fine-tuned or the base model? (I know that in LiMAC with Florence2, Florence2 would be fine-tuned, but it's not fully clear whether this is the case for Florence2 alone).\n4. How, exactly, is \"end-to-end accuracy\" calculated? Why would predicting \"wait\"s rather than \"input-text\"s increase overall accuracy? I understand that you try to explain this on lines 369–403, but a clear definition of the accuracy calculation would make this paragraph make more sense.\n\nMinor issues:\n- On line 118, \"that treat\" should probably be \"that they treat\"\n- On line 142, it's a bit weird that the full architecture link links back to the current section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oznHwyZaq9", "reviewer": "Reviewer_skyy", "summary": "This work introduces LiMAC, which is an architecture for training smaller neural networks that could fit on-device for UI control. LiMAC processes UI elements as embeddings, uses a contrasting learning approach for click actions, a gated architecture for selectively invoking a fine-tuning VLM to generate text content, and it shows ablations of the architecture.\n\nThe work evaluates several different variances of LiMAC and compares them to multiple baselines, including those using large, proprietary models. In addition to comparing accuracy, they also compare inference time, which is important for on-device applications.", "questions": "Please see weaknesses.\n\n## Comments\n(You do not need to respond to these; they are intended to be helpful)\n\nIn future work, you can…\n* Do actual on-device implementation. I suspect it may be non-trivial\n* You could report mobile-specific performance metrics (battery impact, memory usage, etc.)\n* Analyze real-world latency measurements on real phone, which would be compelling", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work tackles an under-explored problem of considering compute/inference costs for UI control agents. To get practical agents, the community needs this type of work.\nThe performance of the LiMAC agent is strong compared to the baselines in the paper. They apply InfoNCE loss in an interesting way for screen understanding and automation. The work includes a large number of baselines, which is appreciated. The authors take off-the-shelve VLMs and fine-tune them for device control, as part of the baselines and main results. And the work include ablation analyses of their results.\nThe authors explain their methodology well and do rigorous evaluation.", "weaknesses": "The results are not that much better compared to the baseline of fine-tuning an off-the-shelf model, particularly for Florence2 as shown in Table 1 (70.8 vs 72.2 for AitW and 57 vs. 63.1 for AndCnrl). It's also hard to tease out if the improved performance is from architectural breakthroughs or rather just from adding more parameters by incorporating the LiMAC network.\n\nOn a related note, the claims of the superiority of the proposed architecture would be strengthened with online evaluation by using model-based evaluation on AitW as in DigiRL: https://arxiv.org/abs/2406.11896 or using an online benchmark (e.g., AndroidWorld https://arxiv.org/abs/2405.14573).\n\nThe architecture of LiMAC is not particularly novel. While the contrastive loss is interesting, the other parts such as representing UIs using embeddings of UI elements is not novel (past examples: Mapping Natural Language Instructions to Mobile UI Action Sequences: https://arxiv.org/pdf/2307.10088, Android in the Wild: A Large-Scale Dataset for\nAndroid Device Control https://arxiv.org/pdf/2307.10088). While the gated architecture with the VLM is a sensible engineering decision, it is not novel from a research perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)