OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds
Authors: Longrong Yang
Zhixiong Zeng
Yufeng Zhong
Huang Jing
Liming Zheng
Lei Chen
Haibo Qiu
Zequn Qin
Lin Ma
Xi Li
Venue: ICLR 2026 Poster (2026)
Keywords: generalist agent; GUI agent; embodied agent; MoE
Primary area: foundation or frontier models, including LLMs

Abstract: Multimodal large language models are progressively advancing toward multimodal agents that can proactively execute tasks. Existing research on multimodal agents primarily targets either GUI or embodied scenarios, corresponding to interactions within 2D virtual world and 3D physical world, respectively. However, many real-world tasks inherently require agents to interleave interactions across both types of environments. We initially mix GUI and embodied data to train models, but find performance degradation caused by data conflicts. Further analysis reveals that GUI and embodied data exhibit synergy at shallow layers but conflict at deep layers, resembling the cerebrum-cerebellum mechanism in the human brain. To this end, we introduce a high-performance generalist agent, OmniActor, designed from both structural and data perspectives. First, we propose Layer-heterogeneous MoE that separates parameters at deep layers to eliminate conflict, while sharing parameters at shallow layers to leverage synergy. This design enables OmniActor to outperform agents trained solely on GUI or embodied data in their respective tasks. Furthermore, we unify the action spaces of GUI and embodied tasks and collect large-scale datasets from diverse sources for training. This substantially enhances the performance of OmniActor across various scenarios, especially in GUI tasks.

Published as a conference paper at ICLR 2026
OMNIACTOR: A GENERALIST GUI AND EMBODIED AGENT FOR 2D D WORLDS
Longrong Yang'. Zhixiong Zeng 2 Yufeng Zhong", Jing Huang", Liming Zheng?, Lei Chen', Haibo Qiu Zequn Qin Lin Ma Xi Li
*College of Computer Science and Technology, Zhejiang University
Meituan Technology
Schooo of Software Technology, Zhejiang University
ABSTRACT
Multimodal large language models are progressively advancing toward multi- modal agents that can proactively execute tasks. Existing research on multimodal agents primarily targets either GUI embodied scenarios, corresponding to in- teractions within 2D virtual world and 3D physical world, respectively. However, many real-world tasks inherently require agents to interleave interactions across both types of environments. We initially mix GUI and embodied data to train models, but find performance degradation caused by data conflicts. Further anal- ysis reveals that GUI and embodied data exhibit synergy at shallow layers but conflict at deep layers, resembling the cerebrum-ceerbbellum mechanism in the human brain. To this end, we introduce niActor, designed from both structural and data perspectives. First, we propose Layer-heteroggenous MoE that separates parameters at deep layers to eliminate conflict, while sharing parameters at shallow layers leverage synergy. This de- sign enables OmniActor to outperform agents trained solely on GUI or embodied data in their respective tasks. Furthermore, we unify the action spaces of GUI and embodied tasks and collect large-scale datasets from diverse sources for train- ing. This substantially enhances the performance of OmniActor across various scenarios, especially in GUI tasks.
generalist agent, Om-
INTRODUCTION
Foundation models have demonstrated remarkable capabilities in both language and vision under- standing tasks. In particular, Multimodal Large Language Models (MLLMs) (Liu et al. 2023b 2024 Wang et al. 2024 Bai et al., 2025) Team et al., 2025). iee. multimodal foundation models trained massive image-text corpora, have achieved strong performance in image-text understand- ing benchmarks. As an extension of MLLMs, multimodal agents have been successfully applied to graph user interface (GUT) control (Wu al. 2025b Xu al. 2025 Qin al., (025) and embodied
intelligence (Shah et al. 2023 Ha et al. 2023 Kim et al. 2025) and are increasingly adopted in
industrial applications.
Humans naturally perform actions across diverse environments, such as shopping in 2D digital world or interacting with entities in 3D physical world. This raises a natural question: can an agent act in both 21 and 3/) environments like humans, thereby realizing generalist agent? This goal is non-trivial due to the inherent differences between 2D and 3D worlds. GUI and embodied tasks serve typical tasks for evaluating agent abilities in 2D and 3D worlds, respectively. Several preliminary attempts have been made to unify GUI and embodied actions within a single agent, For instance, Magma (Yang al. 2025a) represents GUI and embodied actions set-of-mark and trace-of ofiee-of-mna of-mark, and uses vidcos to enrich action supervision. GEA (Szot et al., (025) introduces continuous multi-embodiment tokenizer for mixed action prediction, combined with online reinforcement learning phase to further improve performance.
"Corresponding authors. Prooeector leader.

---

Published as a conference paper at ICLR 2026
LIBERO-N
Gryyeyy
74.5
77.1
69.5
63.4
60.8
600
60
50.5
Socccss Rale
-50
40
30
30
20
OomiArtto-GUII
OmmiAci--AARGUT
OmniAckr
OmniAcue-EA OmmiAcnn-EARGU
OuniAdte
Meethds
Methods
(a) GUI lasks in 2D digital world
(b) Embodied tasks in 3D physical world
Figure 1: Performance comparison. AndroidControl and GUI Odyssey are evaluation benchmarks for GUI tasks. LIBERO-90 is evaluation benchmark for embodied tasks. OmniActor-Gul denotes the agent trained solely on GUI data. OmniActor-EAA denotes the agent trained solely on embodied data. SamiiActor-EA&GGG is trained jointly on both data sources. OmniActor, equipped with the proposed Layer-heterogeneity MoE, effectively leverages synergy while mitigating conflict, leading to substantial improvements.
When unifying data from distinct environments into one model, fundamental issue is conflict and symergy across these data. Our initial experiments reveal that naively mixing GUT and embodied data leads to degraded performance due to conflict, as shown in Figure [ We attribute the conflict to the significant action differences: GUI actions are typically rext-described operations (e.g. "click"), whereas embodied actions involve continuous 6-DoF end-effector displacements. On the other hand, GUI and embodied tasks should also exhibit synergy, as they share similar task structure (environ- ments, instructions, and actions). The comprehension of environments and instructions can mutually reinforce each other across embodied and GUI tasks. Unfortunately, existing generalist agents lack explicit mechanisms to manage such conflict and synergy, resulting in suboptimal performance.
To this end, we propose a novel generalist agent, termed OmniActor. We not only propose Layer- heterogencity MoE to leverage the synergy between GUI and embodied data while mitigating their conflict, but also collect and unify large-scale data to train a niigh-performance unified agent. Specifi- cally, (i) Later-eettoyyeneitty MoE. By examining the parameter update directions across different data, we observe that the directions from GUI and embodied data are more consistent in shallow lay- than in deep layers. This phenomenon is analogous to the cerebrum-cerebelluu mechanism in humans: the cerebrum, being closer to the sensory input, undertakes a comprehensive understand- ing of environments and instructions, whereas the cerebellum, being closer the motor output, is responsible for executing diverse actions. Thus, we share parameters in shallow layers to exploit the synergy between GUI and embodied data, while separating parameters in deeper layers avoid conflicts caused by divergent action requirements. As shown in Figure[]] this design substantially improves the agent performance.
(ii) Large-scale GUI and embodied data. We curate large- scale training data from multiple sources, including GUI data from AS-Atlas Wu et al. (2025b), Uground Gou et al. (2025), Aguvis Xu et al. (2025), ) and Aria-Ul Yang et al. (2025b), as well as embodied data from LIBERO Liu et al. (2023a). All data are unified into a standardized format, where each sample contains a system prompt, an image, a task instruction, and an output action. We employ text tokenizer for GUI actions and a specialized embodied tokenizer for embodied actions, both mapped into a shared vocabulary. By unifying the data format and action space, we enable training on large-scale GUT and embodied data, thereby enhancing the agent across diverse tasks.
In summary, our contributions are as follows:
We propose the Layer-thterogeneity MoE, which shares parameters in the shallow layers to exploit the synergy between GUI and embodied data, while separating parameters the deep layers mitigate the conflict arising from action discrepancies.
2

---

Published as a conference paper at ICLR 2026
We expand and unify GUI and embodied data by standardizing data formats and action spaces. This enables training with large-scale datasets from both domains, substantially enhancing the performance of agents on GUI and embodied tasks.
We extend the capabilities of MLLMs build generalist agent, OmniActor. OmniActor significantly outperforms existing generalist agents in both GUI and embodied tasks, and even surpasses oftte-of-the-arr GUI embodied agents.
2 RELATED WORKS
2.1 GUI AGENT
Early GUI agents (Deng et al. 2023 Gur et al. 2024 Lai ct al. 2024 He et al. 2024 Yang et al., 2023) rely on HTML or Tree data describe GUI elements through textual descriptions. These methods depended on the structured representation of web page elements, enabling agents to lo- cate targets based on tags, attributes, or text content. However, these methods need meticulous pre-processing designs, limiting their generalization under different scenarios. With the rise of mul- timodal large language models (MLLMs), an increasing number of pure vision-based methods have been proposed. Leveraging the strong visual capabilities of MILM, these methods eliminate the need for manually designing data pre-processing schemes for each scenario, and thus have signif- icant advantages over HTML--eppndenn GUI agents in terms of generalization. Early pure visual GUT agents (Hong et al. 2024 Zhang & Zhang 2024 ZZhang et al. 2024, Wu et al. 2025a 2024 2025a) focus using MLLMS process GUI screenshots understand GUI components, replac- ing HTML-dependent GUI component understanding. Furthermore, researchers find that scaling data crucial for enhancing GUI agents. thus proposing the use of synthetic data or video data (Yang et al. 20255b Xie et al.) 20255 Wu et al.) 2025b [Sun et al., (025). For instance, Aria-UP UI (Yang et al.) 2025b) introduces an extensible data synthesis pipeline for generating grounding instruction sam- ples, which are used to train MLLMs specifically for GUI grounding. OS-Atlas (Wu et al.) (025b) releases the first multi-platform GUI data synthesis toolkit, supporting the automatic synthesis of cross-platform GUI grounding data while resolving action naming conflicts.
Recently, researchers have aimed to make GUI agents more similar to human-like agents (Gou et al. 2025 Xu et al. 2025 Lu et al. 2025 Qin et al. 2025). For example, UGround (Gou et al. 2025 proposes to execute actions only using human- like keyboard and mouse operations, demon- strating the feasibility of human-like GUI agents. Aguvis (Xu et al. 2025) unifies different GUI action spaces and divides training into grounding and planning, enhancing the planning capabili- ties of agents after they have acquired strong GUI grounding abilities. GUI-Odyssey (Lu et al.) 2025) proposes a large-scale cross-application training and evaluation dataset, allowing agents to interleavely use multiple applications to perform tasks instead of being limited to single appli- cation. UI-TARS (Qin et al., 2025) can perform human-like interactions, which achieves accurate perception of GUI elements by collecting large amount of GUI screenshots and enhances agent capabilities through multiple reasoning modes. Ultron ZettonnZeng aaa et al. (2025) achieves the state-of-the- art performance through system-level design and engineering optimization. MobileDreamerr [Cao al.] (2026) enhances long-horizon performance for mobile GUI agents by integrating an effi- cient textual-sketch world model and rollout imagination to enable future-aware decision-making. GUIRoboTron-Sppee Speech Han et al. 2025b a) empowers agents with speech recognition capabilities.
2.2
EMBODIED AGENT
In the field of robotics, benefiting from the rapid development of MLLMs, there has emerged new trend of leveraging the strong generalization ability of MLLMs to enable them to act as "brains" for controlling robots to perform tasks. ATM (Wen al. 2023) uses pre-trained models to predict trajec- tories in videos, providing action guidance for agents. MUTEX (Shah et al. 2023) learns strategies from multiple modalities, which can understand instructions in six modalities or their combina- tions. ACT (Zhao et al.) (023)) proposes low-cost system that enables robots to complete multiple high-diffiullty tasks through Supervised Fine-Tuning (SFT) and a Transformer based action chunk- ing algorithm. Distill-D (Ha al. 2023) efficiently generates embodied data with language labels using Large Language Models (LLMs) and sampling planners. MalL (Jia et al. (0244) proposes new SFT architecture based Mamba, which reduces overfitting and enhance generalization.
3

---

Published as a conference paper at ICLR 2026
Prise (Zheng et al. 2024a) regards temporal action abstraction sequence compression problem, and combines continuous action quantization with byte pair encoding to learn action abstractions. MDT (Reuss al. 2024 addresses data annotation issues through potential goal-conditional state representations and can handle long-term manipulation tasks with sparsely annotated data. Then, RoboFlamingo (Li et al., 2024b). Octo (Team et al. (024), and OpenVL.A (Kim al., 2025)
vision-lagguage-action models built on open-source MLLMs, which can be applied to robot control through fine-tuning. GR-2 (Cheang et al., (024) pre-trains agents on massive internet video data and fine-tune agents on embodied trajectory data. Some recent works Yan et al. (2025): ZZhong et al. (2025); Zheng et al. (2025b) focus on unifying data from different sources or using cheap data to improve the performance of embodied agent.
2.3 GENERALIST AGENT
Researchers have recently begun to study generalist agents, aiming to develop a foundational model that serves multimodal agent tasks in both digital and physical worlds, capable of accomplishing various agent tasks ranging from GUI navigation to robotic manipulation. Magma (Yang et al. 2025a) unifies GUT and embodied tasks into State of Motion (SoM) and Trajectory of Motion (ToM), and augments action trajectory data with video data. GEA (Szot et al. (225)) utilizes a Continuous Multi-Embodintt Tokenizer to perform hybrid action prediction for embodied intelligence, games, GUI control, planning, and navigation, and sets Supervised Fine- Fine-Tuning and Online Reinforcement Learning. NaviMaster (Luo et al. 2025) holds that both GUI and embodied tasks can be formulated as Markov decision processes, thus enabling the generation of trajectories for GUI and embodied tasks in a generalist form. Based on this, employs reinforcement learning and distance-aware reward to enhance the generalization of the agent.
3 METHODOLOGY
3.1 OVERVIEW
Generalist agent: A multimodal generalist agent should be capable of interacting with both 2D virtual world and 3D real world. We define multimodal agent π,, whose inputs include the visual observation 7 of the environment, task description T, and historical information H in text form. Its output is of actions A, defined
A= T (1,
(ai, ,dy.
where / E {gui, robot} indicates whether the n-th action token on is GUI token or an embodied token. This formulation is applicable to different types of tasks:
Embodied tasks in 3D real world: For instance, the embodied task T may be "put the black bowl at the back on the plate". The output includes efegree-of-freeddeee (6-DoF) displacement (pos pos pos,, rot,, roly, rotz) of the end-effector, and an additional di- mension indicating whether the gripper is open or closed. (pos,. pos,, y pos_) denote the spatial position of the robotic arm, and (rotz, roly, rot.) denote its rotation angles. Each dimension is typically normalized to the range [-1, 1].
GUI tasks in 2D virtual world: For example, the GUI task T may be "book a hotel". The output includes both language tokens representing the semantic type of the action (e.g.. "tap" or "click") and the position (1r, y) where the action is applied.
Given an observation image, task instruction, and historical information, a multimodal generalist agent should predict either GUT actions or embodied actions depending on the environment.
Our method: To train generalist agent, OmniActor, that leverages the synergy between GUI and embodied tasks while mitigating conflicts, we introduce improvements in both data and structure: (i) Data. As shown in Figure 2, we unify GUI and embodied data into consistent format where each sample includes a system prompt, an image, a task instruction, and an action description. Em- bodied actions, normalized to [-1, 1], are discretized into K intervals, each corresponding to token ID, thereby converting embodied actions into tokens. (ii) Structure. We propose layer-heterope heterogeneity MoE. As illustrated in Figure shallow layers are shared to exploit the synergy between GUI and
4

---

Published as a conference paper at ICLR 2026
Task Instruction
Action
Action: scarch right search for the YOOX app.
<image> Please generate next move accarding to the UI scrcenshot, insinaction and previous actions.
Openation: sauutoggiillicc(y-O..k.. ym0.977)
Instruction: Go to the Google Play Store, install the YOOX shopping app, and open
app.
Text tokenizer
actions:
Step I: Tap on the Play Store ioun open il.
Unifed
Please generate natural language deseription of action 'Action'' and the specific exccution action in OOeeaatonn
Action Space
Embodied tokenizer
Task Instruction
Action
<Image> What action should the roboc take to put the black bowl the back on the plate?
Action: 0.4488771442794885 -00050000022022224 an 00.7789714888898677 0.O, -0.146785721 1822311,,,,
0.0803571411922742,
Figure 2: Data format. We unify GUI tasks in 2D digital world and embodied tasks in 3D real-world environments Each sample includes a system prompt, an image task instruction, and an action description. GUI actions are typically textual, and we use the MLLM tokenizer as a GUI tokenizer to convert these actions into tokens. Embodied actions usually described by 6-DoF displacement of the end effector plus gripper control signal, which are tokenized using an embodied tokenizer. Both GUT and embodied actions share the same vocabulary, forming unified action space for prediction.
embodied data, while deep layers are separated to avoid conflicts arising from action differences. cerebrum-ccrrbelllm This design is analogous to the cerebrum cerebellum mechanism: shallow layers (like the cere- brum) process information comprehensively, whereas deep layers (like the cerehellum) generate task-specific actions. Further details of these modules are provided in the subsequent sections.
3.2 DATA PROCESSING
Unification of Data Format: We adopt Qwen2- VL (Wang al. 2024 and Qwen2.5-VL (Bai
et al. 2025) as our base MILMs. To ensure compatibility with Qwen2-VL. and Qwen2.5-VI. in Lla-
malactory (Zheng al. 2024b), convert all GUI and embodied data into the ShareGPT format, as illustrated in Figure Each sample consists of system prompt, an image, a task instruction, and an action. The system prompt specifies the action space of the agent, while the image represents the current environment. Images are tokenized via the image encoder of the base MLLM, whereas system prompts, task instructions, and GUI actions are tokenized using the text tokenizer of the base MLLM. Embodied actions, however, require dedicated tokenization process.
Tokenization of embodied actions: We represent embodied actions discrete tokens for pro- cessing in this paper. An embodied action is defined as 6-DoF displacement of the end effector, (pos,, pos, y. pos_, rot,r, rot,, rot,), together with one additional dimension indicating whether the gripper is open closed. All dimensions cretize this interval into bins and assign token IDs to each bin by selecting the K least frequent token IDs from the vocabulary of the base MILM. For instance, as shown in Figure embodied action [0.043, -0.075, -0.579, 0.0, -0. 147, -0.080, 1.0] is converted into the token sequence [151510, 151500, 151482, 151515, 151515, 151516, 151642].
normalized to lie within [-1, 1]. We uniformly dis-
Through these two steps, we standardize and tokenize both GUI and embodied data, enabling effec- tive training of the base MI.MM to build unified generalist agent.
5

---

Published as a conference paper at ICLR 2026
Action Space
GUI Actin
Embodied A ction
FFN
FFN
Multi modal Self A ttention
QKV
QKV
GUI expert
Embodied expert
FFN
Shared Blocks Layer 1-7
Multi-modal Self A ttention
QKV
Instruction: Unline Booking.com com and Instagrum to plan an ilinerary for viewing Northern Lights in Iceland and socure your uccommodations.
Text Tokenizer
Image Encoder
Figure 3:
The proposed pipeline. The proposed Layer-heterogeneity MoE shares parameters in shallow layers to leverage the synergy between GUI and embodied data, and separates parameters in deep layers to eliminate conflicts between GUT and embodied data caused by action differences.
3.3 LAYER- HETEROGENEITY MoE
GUI and embodied data exhibit both synergy and conflict, which, from the perspective of model optimization, are reflected in the directions of parameter updates. Specifically, "synergy" refers to similar parameter update directions arising from GUI and embodied data, whereas "conflict" refers to divergent update directions between the two. From this viewpoint, if GUI and embodied data are synergistic for certain parameters, these parameters should be shared; otherwise, parameters should be separated. Motivated by this principle, we analyze the parameter update directions between GUI and embodied data and observe that the similarity of optimization directions is much higher in shallow layers compared to deep layers. We hypothesize that this phenomenon occurs because the understanding of environments and instructions is synergistic across GUI and embodied tasks, while the differences between GUI actions and embodied actions necessitate separate parameters. Thus, we propose the lnyer-heterogeneity MoE, designed to leverage the synergy between GUI and embodied data while mitigating their conflict, as illustrated in Figure
Specifically, assuming the layer count of the LLM is L, we set a layer depth threshold T to dis- tinguish shared layers and separated layers. (i) For shallow layers I, 1, < T), as shown in Figure 3 we share parameters to process both GUI and embodied data, for utilizing the synergy between GUI and embodied data:
X MSA(LN((x-1)) X (1........
X FFN(LN((X ()) x' (I,...;))
where X2 indicates the hidden representation in the layer L, MSA indicates the Multi-head Self- Attention, FFN indicates the Feed- Fordard Network, and LN indicates the Layer Normalization.
(ii) For deep layers la (T < la < L), as shown in Figure 3 we separate the attention and FFN. One set of parameters (e.g., FFN FFNgui) gui) is used to process GUI data, and another set of parameters (e.g., FFN,.)) is used to process embodied data:
gui (Lngii Xx-1,
if X i is GUI data if X is embodied data
x
LE (T 1, (T....., L} (4)
MSA (LNNoo(X(-1)) + Xe-1:

---

Published as a conference paper at ICLR 2026
FFNgut gui (LN gui (x,) + X
if x is GUI data
X
FoN (LN rob (x))) + X
if x/ is embodied data
E [T
L}
(5)
where MSAgui (FFN guis LN gui) and oS (FFN LN denote expert parameters for GUI and embodied tasks, respectively. Moreover, as illustrated in Figure distinct prediction heads lgui and ay used for GUT and embodied actions. Although both action types are expressed within unified vocabulary, their action spaces differ substantially; thus, separating heads is crucial to avoid conflicts. During inference, we assume the task type of the imput sample (GUI or embodied) is known, and the corresponding branch is selected to process the data.
Through the layer-heterogeneity MoE design with shallow sharing and deep separation, we harness the synergy between GUI and embodied data while eliminating conflict, thereby enabling a high- performance generalist agent capable of operating across both 2D and 3D worlds.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Training data: Due to the diversity and complexity of GUI components, we firstly train the base MLLM GUI grounding data (a combination of open-source datasets OS-Atlas (Wu et al. (025b), Uground (Gou et al., 2025), Aguvis (Xu et al.] 2025). and Aria-UI Yang et al. 2025b). similar to ScaleTrack (Huang al., 2025)) ). This stage is not applied to embodied tasks, the components in embodied scenarios are relatively simple. After the grounding stage, we train the model using trajectory data enable to perform tasks an agent. For GUI tasks, we follow ScaleTrack (Huang iet al. 2025) and select the Aguvis dataset (Xu et al. 2025), whereas for embodied tasks, we utilize the popular LIBERO dataset (Liu al. 2023a). Our GUI and robot trajectory data contain 706K and 669K samples respectively, with a ratio close to 1:1. During training, embodied actions consist of a series of continuous points, which are substantially different from typical MILLM outputs. Thus, it is necessary to resample the embodied data to ensure sufficient learning of embodied tasks (Zawalski al. 2025, Szot et al. 2025). We resample the embodied data 5 times, resulting an approximate GUI-to-embodied data ratio of 1:5. The total data size is around 3.4M samples for grounding training and 4. IM samples for trajectory training.
Evaluation benchmarks: We select two datasets as the GUI evaluation benchmark: (i) Android Control et al.) 2024a). Android Control is designed to evaluate GUI agents on mobile platforms,
which consists of AndroidConntrolLLww and AndroidControl-Hinn High. (ii) GUI-Odyssey (Lu et al. 2025) provides a comprehensive dataset for evaluating cross-application GUT agents. Then, we select LIBERO (Liu al., 2023a) as the embodied evaluation benchmark, which includes total of 90 evaluation tasks covering three different scenarios: kitchen, living room, and study room. More details please refer the supplementary material.
Comparison: We conduct a comprehensive comparison with GUI, embodied, and generalist agents.
(i) In-house GUI agents, including Claude 3.5 (Hu al. 2024). GPT-4o (Hurst al., 2024), and
UI-TARS-7B (Qin et al. 2025). Open-sourced GUI agents, including SeeClick (Cheng et al. 2024). Aria-UP UI (Yang et al. 2025b), AS-Atlas-7B (Wu et al. 2025b), Aguvis-7B (Xu et al. 2025 and
Scale Brack-r (Huang et al. 2025).
(ii) Embodied agents, mainly for robotic arms, including MalL (Jia al. 2024). PRISE Zheng
et al. 2024a), MUTEX (Shah et al., 2023). ACT (Zhao et al. 2023). DistilIDD (Ha et al. 2023).
MDT Reeuss et al. 2024), Vpe (Kim et al.
2024). To (Black et al., 2024). Spatial (Qu
et al. 2025). and UniACT (Zheng et al. 2025a).
(i11) Generalist agents, including Magma Yang et al. 2025a GEA (Szot et al.
2025). and Navi-
Master (Luo et al. 2025). Magma (Yang et al. (225aa) does not report results on LIBERO, Android- Control, and GUI Odyssey, but provides codes and the pre-trained model. Thus, we finetune the pre-trained model on LIBERO, AndroidControl, and GUI Odyssey report the performance. GEA and NaviMaster do not provide codes and pre-trained models.
Our method: (i) OmniActor-
Use GUI data train Qwen2-VL. (Wang et al. 2024 (i) OmniActor-BB EA: Use embodied data to train Qwen2 BoLn2-VL (Wang et al. 2024). (M0) OmniActor: Use GUT and embodied data to train Qwen2-VL. (Wang al. 2024). Parameters in shallow layers

---

Published as a conference paper at ICLR 2026
Table 1: Comparison between different agents on embodied (robot control) tasks and GUI navigation tasks. We report the success rate on each task. "-" indicates that the agent is unable to perform the task or its performance on the task has not been reported.
Embodied Tasks LIBERO-90
GUI Tasks
Agents
Source
AndroidCContrllLLLwww
AndroidControl-Hipn GUT Odyssey
Claude
In-house
19.4
12.5
3.1
GPT-4o
In-house
19.4
20.8
3.3
UI-TARS-7B
In-house
90.8
72.5
87.0
SceClick
ACL'24
75.0
59.1
53.9
Aria-UI
ACL'25
67.3
10.2
36.5
OS-Atlas-7B
ICLR'25
85.2
71.2
62.0
Aguvis-7B
ICML'25
80.5
61.5
63.8
ScaleTrack-7B
arXiv'25
77.9
65.3
MUTEX
CoRL'23
53.0
Distill-D
CoRL'23
49.9
ACT
RSS'23
46.6
MalL
CoRL'24
60.3
PRISE
ICML'24
54.4
MDT
RSS'24 24
67.2
VLe
CoRL'24
73.5
arxiv'24
87.3
SpatialVLA
RSS'25
46.2
UniACT
CVPR'25
73.0
Magma
CVPR'25
34.7
52.1
32.7
51.0
GEA
CVPR'25
57.3
NaviMaster
arXiv 25
68.9
54.0
OmniActor-GUI
88.4
74.5
OnniActor-FA OmniActor
63.4
69.5
87.5
77.1
66.0
shared. Parameters in deep layers are separated as the MoE, where GUI data is processed by one set of experts, and embodied data is processed by another set of experts.
4.2 MAIN RESULTS
As shown in Table 1 have several findings: (i) Existing generalist agents usually have rel- atively low performance than GUI agents on GUI tasks and embodied agents on embodied tasks, although they can finish the tasks simultaneously. (it) OmniAAtor-GUI achieves the compara- ble performance with existing GUI agents on GUI tasks. OmniActor-EA. achieves the comparable performance with existing embodied agents on embodied tasks. (iit) On embodied tasks, OmniAc- tor achieves 6.1% higher task success rate than OmniActor -AA On GUI tasks, OmniActor achieves the average 1.3% higher task success rate than OcniActor- GUI. Especially, OmniActor has a signifi- cantly higher task success rate on AndroidConntol-HHgg and GUI Odyssey, indicating the superiority on long-chain trajectory prediction.
4.3 STUDY
Study about parameter sharing and separation: To leverage the synergy between GUI and em- bodied data while mitigating their conflicts, we investigate strategies for parameter sharing and separation. We conduct the following experiments: (i) OmniAetor-EAAGGin
Directly mix GUI
and embodied data to train Qwen2-VL (Wang et 2024). (ii) OmniActor hard: Building on (i), modify the model to MoB structure, where GUI data is processed by one set of experts and embodied data by another. The parameters of attention heads, FFNs, and the classification head are fully separated. (idl) OmniActor: Different from (ii), parameters in shallow layers shared, and parameters deep layers are separated. (iv) OmniActor router: Building on (iti), a router is incor- porated prior each MoE layer, which takes token features input and outputs the probabilities
8

---

Published as a conference paper at ICLR 2026
Table Comparison between different parameter sharing and separation strategies. We report the success rate on each task.
Embodied Tasks
GUI Tasks
Models
Avg
LIBERO-90
AndroidComtrol-Lne
AndroidControl High
GUI Odyssey
OmniActor-GUT OmniActor- EA
88.4
74.5
63.0
63.4
Omni OmmiActor-FAA&Go
50.5
86.3
71.0
60.8
67.2
OcniActor hard
59.5
85.6
75.2
63.9
71.1
OmniActor
69.5
87.5
77.1
66.0
75.0
OmniActor router
64.0
86.0
72.6
66.
72.2
0011
mk an B
0.90
0.11
Layer
4.82
(a) Parameter upulat similurity statistic for FEN
(b) Purameter update similurity statistic for attention
Figure 4: Statistic on parameter update similarity. Shallow layers exhibit significantly higher pa- rameter update similarities than deep layers, motivating our design of the Layer-heterogeneity MoE.
of assigning the token to different experts. The expert with the highest probability is selected as the active expert for the token.
As shown in Table 2 we have some important findings: (i) Directly mixing GUI and embodied data degrades model performance due to conflicts between the two tasks. (ii) OmniActor hard separates almost all parameters for GUI and embodied tasks. Compared mixed training, improves per- formance by eliminating confficts, but cannot exploit the synergy between GUI and embodied data. (iii) OmniActor outperforms separate training by leveraging the synergy between GUI and embod- ied data while simultaneously resolving conflict. (iv) OmniActor outperforms OmniActor router, which verifies that improvements stem from task-specific partitioning rather than MoE capacity.
We analyze the parameter optimization direc- tions for GUI and embodied tasks to answer two key questions: Which parameters can be shared and which should be separated? Parameters be shared when the update directions from GUT and embodied data are consistent; otherwise, they should be separated.
Statistic about parameter optimization directions:
How to identify shareable parameters? We propose a novel metric, parameter update similarity, which estimates the consistency between GUI and embodied tasks for each parameter. For instance, for the gate projection in the first FFN, we compute the parameter difference before and after training with GUI data, denoted as dgui, and similarly compute the difference after training with embodied data, denoted as drobot- The cosine similarity between dgui and drobot defines the parameter update similarity for the gate proj. A high similarity indicates that the parameter can be shared; otherwise, it should be separated. When the parameter update similarity high, the parameter should be shared; otherwise, the parameter should be separated. We conduct this analysis for FFNs (gate proj, up proj, and down proj) across layers Figure 4(a) and for attention (k proj. q proj, v proj, and proj) in Figure bb) Based on these statistics, shallow-layer parameters shared, while deep-layer parameters are separated. Accordingly, we empirically set the layer depth threshold T = 8, which distinguishes shared from separated layers.
[
Generalization verification: We analyze the impact training agents on different base MLLMs. Specifically, replace the base MILLM with Qwen2.5-VL. 7B (Bai al, 2025), adjust the training

---

Published as a conference paper at ICLR 2026
Table 3: Study about different base MLLMs. We evaluate different base MLLMs verify the generalization of the proposed strategy. We report the success rate on each task.
Embodied Tasks
GUI Tasks
Base MILMS
Avg
LIBERO-90
AndroidConttrl-LLww
AndroidControl Hoigh GUI Odyssey
Qwen2-VL. 7B
69.5
87.5
77.1
66.0
75.0
Qwen2.5-Ve VL 7B
65.2
87.8
83.2
81.0
79.3
data to match its input format, and migrate the Layer-heterogeneity MoE to this model. Experi- mental results are summarized in Table 3 When training the generalist agent on the more powerful Qwen2.5-VL 7B, the average success rate increases by 4.3% compared to training on Qwen2-VL 7B, and the average performance is close to the state-of-the in-house GUI agent UI-TARS-7B for GUI tasks. This verifies the generalization of OmniActor across different base MLLMs.
5 CONCLUSION
Our research aims to construct high-performance generalist agent from both data and structural perspectives. Data scaling is particularly crucial for agent performance; therefore, we unify the ac- tion spaces of GUI and embodied tasks, and collect a large amount of GUI and embodied data for training the agent. Furthermore, ied tasks. By separating deep-layer parameters and sharing shallow-laye parameters, we leverage the synergies between tasks and eliminate conflicts. Extensive experiments demonstrate the effec- tiveness of the generalist agent OmniActor across different scenarios. Especially in GUI scenarios, OmniActor even outperforms many of the latest state-of-the-art GUI agents.
identify the conflicts and synergies between GUI and embod-
Acknowledgements. This work was supported in part by the National Science Foundation for Distinguished Young Scholars under Grant 62225605, Zhejiang Provincial Natural Science Founda- tion of China under Grant LD24F020016, Ningbo Science and Technology Special Projects under Grant No. 20257028, and the Fundamental Research Funds for the Central Universities.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.1399 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. piO: A vision- lasion-lnguaae-action flow model for general robot control. arXiv preprint 4411422410.24164, 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06811 2022.
Yilin Cao, Yufeng Zhong, Zhixiong Zeng, Liming Zheng, Jing Huang. Haibo Qiu, Peng Shi, Wenji Mao, and Wan Guanglu. Mobiledreamer: Generative sketch world model for gui agent. arXiv preprint 2r0iv:2601. 2000:20000.0055. 2026.
Chi-Lam Cheang, Guangzeng Chen, Ya Jing. Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language-accto action model with web- scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024.
Kanzhi Cheng. Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint 24X.::2401.10935.. 2024.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-288 14, 2023.
10

---

Published as a conference paper at ICLR 2026
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference Learning Representations, 2025. URL https:
7 //openreview. net. net./for d=kxnoqaiss
Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Alcksandra Faust. A real-world webagent with planning, long context understanding, and pro- gram synthesis. In ICLR, 2024.
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guider robot
skill acquisition. In Conference on Robot Learning, pp. 3766-3777. PMLR, 2023.
Wenkang Han, Zhixiong Zeng, Jing Huang, Shu Jiang, Liming Zheng, Longrong Yang. Haibo Qiu, Chang Yao, Jingyuan Chen, and Lin Ma. Guirobotron-speechh: Towards automated gui agents based on speech instructions. arXiv preprint arXiv:2506. 1115000011122.. 2025a.
Wenkang Han, Zhixiong Zeng, Jing Huang, Shu Jiang, Liming Zheng, Longrong Yang, Haibo Qiu, Chang Yao, Jingyuan Chen, and Lin Ma. Uitron-speech: Towards automated gui agents based on speech instructions. arXiv preprint arXiv:2506. / 11277, 2025b.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
2024.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14281-14290,, 2024.
Siyuan Hu, Mingyu Ouyang, Difei Gao, and Mike Zheng Shou. The dawn of gui agent: A prelimi- nary case study with claude 3.5 computer use. arXiv preprint arXiv:2411.10332 2024.
Jing Huang, Zhixiong Zeng, Wenkang Han, Yufeng Zhong, Liming Zheng, Shuai Fu, Jingyuan Chen, and Lin Ma. Scaletrack: Scaling and back-tracking automated gui agents. arXiv preprint arXiv:2505.004/1 004100 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Xiaogang Jia, Qian Wang, Atalay Donat, Bowen Xing, Ge Li, Hongyi Zhou, Onur Celik, Denis Blessing, Rudolf Lioutikov, and Gerhard Neumann. Mail: Improving imitation learning with mamba. arXiv preprint arXiv:2406.08224 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.0922 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Pannag R Sanketi, Quan Vuong, et al. Openvla: An open-source vision-language-action model. In Conference on Robot Learning, pp. 2679-2713, PMLR, 2025.
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5295-5306, 2024.
Wei Li, William E Bishop, Alice Li. Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyam- agundlu, and Oriana Riva. On the effects data scale on ui control agents. Advances in Neural Information Processing Systems, 37:92130 00022130-900 92154, 2024a.

---

Published as a conference paper at ICLR 2026
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. lasgual foundation models as effective robot imitators. In ICLR, 2024b.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776-44791, 2023a.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, (6:34492-349166 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26296-26300 2024.
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang. Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. In International Conference on Computer Vision, 2025.
Zhihao Luo, Wentao Yan abd Jingyu Gong, Min Wang, Zhizhong Zhang, Xuhong Wang, Yuan Xie, and Xin Tan. Navimaster: Learning a unified policy for gui and embodied navigation tasks. arXiv preprint ar Ar5iv:2508. 02046, 2025.
Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12322 2025.
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding. Zhigang Wang. Yuan Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial representations for visual- language-action model. arXiv preprint urXiv:2501.15830 2025.
Moritz Reuss, Ömer Erdinç Yagmurlu, Fabian Wenzel, and Rudolf Lioutikov. Multimodal diffu- sion transformer: Learning versatile behavior from multimodal goals. In Robotics: Science and Systems, 2024.
Rutav Shah, Roberto Martín-Martin, and Yuke Zhu. Mutex: Learning unified policies from multi- modal task specifications. In 7th Annual Conference on Robot Learning, 2023.
Yuchen Sun, Shanhui Zhao, Tao Yu, Hao Wen, Samith Va, Mengwei Xu, Yuanchun Li, and Chongyang Zhang. Gui-xplore: Empowering generalizable gui agents with one exploration. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19477-19486, 2025.
Andrew Szot, Bogdan Mazoure, Omar Attia, Aleksei Timofeev, Harsh Agrawal, Devon Hjelm, Zhe Gan, Zsolt Kira, and Alexander Toshev. From multimodal iims to generalist embodied agents: Methods and lessons. In Proceedings the Computer Vision and Pattern Recognition Conference, pp. 10644-10655, 2025.
Meituan LongCat Team, Bairui Wang, Bin Xiao, Bo Zhang, Bolin Rong, Borun Chen, Chang Wan, Chao Zhang, Chen Huang, Chen Chen, et al. Longcat-flash-ome technical report. arXiv preprint arXiv:2511.00277 2025.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv: 12213, 2024.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing lasoun- model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024.
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy leaming. arXiv preprint 10000224. 2023.
12

---

Published as a conference paper at ICLR 2026
Tao Wu, Xuewei Li, Zhongang Qi, Di Hu, Xintao Wang, Ying Shan, and Xi Li. Spherediffusion: Spherical geometry-awe aware distortion resilient diffusion model. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 38. pp. 6126-6134 2024.
Tao Wu, Yibo Jiang, Yehao Lu, Zhizhong Wang. Zeyi Huang, Zequn Qin, and Xi Li. Multicrafter: figh-fidelity multi-subject generation via disentangled attention and identity awwre preference alignment. arXiv preprint arXiv:2509.21953 2025a.
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang. Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et Os-atlas: Foundation action model for generalist gui agents. In The Thirteenth International Conference on Learning Representations, 2025b.
Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, and Liqiang Nie. Gui-explorer: Autonomous exploration and mining of transition-awan -aware knowledge for gui agent. In Anmual Meeting of the Association for Computational Linguistics (ACL), 2025.
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong- Aguvis: Unified pure vision agents for autonomous gui interaction. In International Conference on Machine Learning, 2025.
Feng Yan, Fanfan Liu, Yiyang Huang. Zechao Guan, Liming Zheng, Yufeng Zhong, Chengjian Feng, and Lin Ma. Robotron-mani: All-in-one multimodal large model for robotic manipula- tion. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 13707- 13718, 2025.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. 14441, 2023.
Jianwei Yang, Reuben Tan, Qianhui Wa, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: foundation model for multimodal ai agents. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14203-14214, 2025a.
Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria- UI: Visual grounding for GUI instructions. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 22418 -22433, 2025b. URL 2025. findings 111121
org/
Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thoughh reasoning. In Conference on Robot Learning, pp. 3157- 3181. PMLR, 2025.
Zhixiong Zeng, Jing Huang, Liming Zheng, Wenkang Han, Yufeng Zhong, Lei Chen, Longrong Yang, Yingjie Chu, Yuzhi He, and Lin Ma. Uitron: Foundational gui agent with advanced per- ception and planning. arXiv preprint (XXXx:2508.21661 2025.
Jiwen Zhang, Jihao Wu, Teng Yihua, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-actioon-hhuuutt for gui agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 2016-120311220 2024.
Zhuosheng Zhang and Aston Zhang. You only look screens: Multimodal chain-of-action agents. In Findings of the Association for Computational Linguistics ACL 2024, pp. 3132-3199, 2024.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine graineri bimanual manipulation with low-cost hardware. In ICML Workshop on New Frontiers in Learning. Control, and Dynamical Systems, 2023.
Jinliang Zheng, Jianxiong Li, Dongxiu Liu, Yinan Zheng, Zhihao Wang. Zhonghong Ou, Yu Liu, Jingjing Liu, Ya-Qin Zhang, and Xiamyuan Zhan. Universal actions for enhanced embodied foun- dation models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 22508-22519 2025a.
13

---

Published as a conference paper at ICLR 2026
Liming Zheng, Feng Yan, Fanfan Liu, Chengjian Feng, Yufeng Zhong, Yiyang Huang, and Lin Ma. Dataplatter: Boosting robotic manipulation generalization with minimal costly data. arXiv preprint WrXi:22503.19566 2025b.
Ruijie Zheng, Ching-An Cheng, Hal Daumé III au2, Furong Huang, and Andrey Kolobov. Prise: Learning temporal action abstractions as a sequence compression problem, 2024a.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024b. Association for Computational Linguis- tics.
als 2403. 13372
Yufeng Zhong, Chengjian Feng, Feng Yan, Fanfan Liu, Liming Zheng, and Lin Ma. Robotrom-nav: A unified framework for embodied navigation integrating perception, planning, and prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 64 6416-6425, 2025.
14

---

Published as a conference paper at ICLR 2026
6
SUPPLEMENTARY MATERIAL
In our supplementary material, we provide the following details and experiments:
6.
We provide more details about hyper-parameters and evaluation benchmarks.
6.2
We provide more details about prompts and representative samples.
We provide experimental results on
robots.
We provide the study about the layer depth threshold T, statistical mean of perfor- mance, and different disentangling ways.
6.4
We discuss the in-group parameter update similarity and the parameter update simi- lariy the head.
6.3
6.6
We provide case study.
6. We provide a visualization study about synergy and conflict phenomenon.
6.8 We provide details about the use of Large Language Models for this paper.
6.1 DETAILS ABOUT HYPER- PARAMETES AND EVALUATION
Training hyper-parameters: We use the AdamW optimizer with learning rate of le-5 and cosine learning rate scheduler with warm-up ratio of 0.03. We set the global batch size to 128 in the grounding training phase and 64 in the trajectory training phase, and adopt the DeepSpeed ZERO3-style parallel strategy. To align with GUI data, resize all x56x256 images from LIBERO to 448 x44.. The embodied tokenizer uniformly discretize [-1, 1] into K intervals, where K=256. The layer depth threshold T to distinguish shared and separated layers is set to 8.
Evaluation benchmarks: We select two datasets as the GUI evaluation benchmark: (1) Android Control (Li et al.) (2024)). Android Control is designed to evaluate GUI agents mobile platforms. consists of two types of sub-tasks: AndroidControll-
requires the agent to predict specific action types and parameters (e.g., coordinates) each step based an image, task instruction, and a
natural language description of actions. MndroidControl-Hige requires the agent independently plan and sequentially predict action types and parameters when only given images and task instructions. (ii) GUI-Odyssey (Lu et al. 2025) provides a comprehensive dataset for evaluating cross- cross-applin aposscappiia GUI agents. Similar to AndroidConttro-Higgt it only provides images and task instructions, and it requires the agent to perform complex cross-application operations. Following previous studies, we randomly sample 800-step actions to create a subset for evaluation.
We select LIBERO (Liu et al. 2023a) as the embodied evaluation benchmark, which includes total of evaluation tasks covering three different scenarios: kitchen, living room, and study room. For each task, we test 20 unseen start and goal configurations. In each test, the agent first receives the task instruction and the initial scene (image), then outputs an action. The action and the scene are fed into the end effector to generates new scene (image). The task instruction and the scene are then fed into the agent for predicting the next action. When the task is successfully executed, the end effector returns success signal, and the evaluation proceeds to the next task. The maximum steps allowed for task execution is 400.
6.2 DETAILS ABOUT SAMPLES
In this subsection, we detail the prompts we use in this work and present representative GUI and embodied samples for further clarification.
We use different prompts for GUT and embodied tasks. Specifically, the system prompt for GUI tasks is "You are GUI agent. You are given task and screenshot of the screen. You need to perform a series of pyautogui actions to complete the task.", and that for embodied tasks is "You are an agent that can see, talk and act. You are given task and an image. You need to perform a series of actions to complete the task.".
Prompts:
The instruction format for GUI tasks is "Please generate the next move according to the UI screen- shot, instruction and previous actions. \nAn Instruction: (overall [overall_goai] \n An Previous ac- tions: (previous, act ions) AnAn Please generate the natural language description of the cur-
15

---

Published as a conference paper at ICLR 2026
Please geneente the nest acccedlngg UI instruction joevious acsion.
plut oppt
order soguire the finestsoumd possible, want recording quality Sound Recorder app 96 kips, 10.1 kitz.
Instruction:
the
Step Click on
Menu loon at the left
oomer of the
Presious actions None
Please gencente the natural language desctiption of the curvent action in cution
and the specilic execution actica Openailot:
Actions Click the settings
Action Click on the Menu icon et tse left
Operation: praatogul click00-0.... y-0 3325)
corher the screen
Openation pysutogui.ciooo www.ccccccccccc... y-t
Step 1
Step
Previous actioss:
Previous actions Stap 1: Click the
Step Click on the Metu
loon the left oomer of the
comor
Stop
Step 2: Click on the settings optin.
settings option
eutty
Step 3: Click on the Recording quality opton.
Action: Recorting quility optica
Action: Select the kips, 46.1 kHH quality
Operation: proutogul clickix-0.3 46224)29)
Openaiion: syuubgui.clickts-d -lt.ccllhhoo004981 y=0.5048)
Step 3
Step 4
Figure 5: A representative GUI sample. Each GUI sample consists of several steps. Each step includes an image (environment observation), instruction, history (previous actions), and action.
<imagio>
What should
<inooge> should robot
What should sobot
Wot should robot
take cabluet?
drover
cabieet?
cabiint?
caliaut?
In 5308571 19646
JO 064422571
34011167000000050000
03482 40836806494 -0.75,
[0. -3 7388228911
42947614118
IL0091071
0064228571 4172777742
0L003214228
LUU7288711 00000357.
00046442666 1.00
0.04 135881399777
Step 0
Step 20
Stop 40
Step GD
Figure 6: A representative embodied sample. Each embodied sample consists of several steps, with each step including image (environment observation), instruction, and action.
rent action in "Action and the specific execution action in 'Operation '. For embodied tasks, the instruction format is "What action should the robot take to (overall_goal]
Each sample has different (overall_goai) and serevious_actionn ure 5) for the GUI sample, the (overall_gooe possible, want to increase the recording quality in the Sound Recorder Plus app 96 kbps, 44.1 kHz.", " and the (previous aattatti ions) serves as the agent's 'sss history and grows as the number of steps increases, As shown in Figure for the embodied sample, the (overal [goal] is "close
As shown in Fig- "In order to acquire the finest sound quality
16

---

Published as a conference paper at ICLR 2026
Instruation: Whdt adtion should the robol take to place ribor bluckerty inlo lop drawer?
Step
Step
Step
Step
Step
Figure 7: An example about real-world robot data.
100%
80%
82%
85%
62%
65%
60%
505
50%
45%
30%
10%
Average
move redbull can near sponge
open top drawer
place green jalapeno chip bag into top drawer
pick orange
OmniA EA
OmniActor
Figure 8: The model performance on real-world robot data. We evaluate OmniActor-EA and Om- niActor on four tasks: "move redbull can near sponge", "open top drawer", "place green jalapeno chip bag into top drawer", "pick orange".
the top drawer of the cabinet". We maintain consistency in the system prompts and instruction formats during both training and inference.
In addition, we conduct experiments based on LlamaFactory (Zheng et 2024b) and largely follow the practices in LlamaFactory (Zheng et al. 2024b).
Representative GUI and Embodied Samples: shown in Figure 5 each GUT sample consists of several steps. Each step includes an image (environment observation), instruction, history (previous actions), and action. Similarly, as shown Figure 6 each embodied sample consists of several steps, with each step including an image (environment observation), instruction, and action. During the evaluation of GUI tasks, the model predicts an action given an image, instruction, and history; for embodied task evaluation, the model predicts action given image and instruction.
6.3 REAL-WORLD ROBOT EXPERIMENTS
In this subsection, we validate the generalization ability of OmniActor in 3D real-world scenarios through weal-world robot experiments. Specifically, to preserve the structure of the training data, we collect approximately 700K real-world robotic trajectory data from RT-1 (Brohan et al.) 2022), which is comparable scale to the LIBERO trajectory data. Then, we convert the format these data into the embodied data format described in the main text (as shown in Figure 6). which is used to replace the LIBERO data for training both OmniActor-Ei EA and OmniActor. Subsequently, evaluate the performance of the trained models four unseen tasks: "move redbull sponge", "open top drawer". "place green jalapeno chip bag into top drawer". "pick orange". We present example of the real-world robot
perform 20 trials per task. As shown in Figure
17

---

Published as a conference paper at ICLR 2026
Table 4: Study about the layer depth threshold T. We report the success rate on each task.
Embodied Tasks
GUI Tasks
Avg
LIBERO-90
AndroidConttrl-Loww
AndroidConttollHiggg GUT Odyssey
7=4
63.0
84.5
74.1
65.4
71.8
7=8
69.5
87.5
77.1
66.0
75.0
T=12
63.0
87.7
75.8
67.7
73.6
T=20
61.5
71.1
67.6
72.0
Table 5: Study about the statistical mean and standard deviation of the model performance. We report the success rate on each task.
AndroidConttol-Low
AndroidCContro-HHggh
GUI Odyssey
OctiActor-GUI
88.82 + 1.27
75.20 + 0.78
63.77 + 0.32
OmniActor
88.98 1 1.53
77.30 1 0.85
65.67 1: 0.31
Table Study about different disentangling ways. We report the success rate on each task.
Embodied Tasks LIBERO-90
GUI Tasks
Avg
AndroidControl-Low
AndroidControl- High
GUI Odyssey
OmniActor 69.5
87.5
77.1
66.0
75.0
OnniiAtoo-aattenton
68.0
86.5
76.7
63.7
73.7
OmniActor-FFR OmniActor-han
66.0
84.6
74.0
63.2
72.0
61.0
86.0
75.7
62.6
71.3
data, illustrating the motion the robotic arm 3D real-world environment. Figure compares the performance of OmniActor-EA and OmniActor, along with environmental examples of the four evaluation tasks. The experimental results demonstrate that OmniActor achieves success rate ex- ceeding 60%, and its performance outperforms OmniActor-F thereby validating the effectiveness of the Layer-heterogeneous MoE under real-world robot data.
6.4 FURTHER STUDY
Layer depth threshold T: The layer depth threshold 7' is tunable hyper-parametert We discuss different selections of T Table 4 and find that the performance optimal when T=8.
Statistical mean of performance: We conduct five independent training runs for OmniActor-Gui and OmniActor respectively, and calculate their statistical mean and standard deviation. As shown in Table 5 the performance gain is statistically reliable.
Different disentangling ways: We further explore different ways of disentangling the GUI and embodied knowledge in the T OmniActor separate the parameters of attention, FFNs, and prediction heads. OmmiActor-attentioo shares attention while seperates OmniActor-FFN shares FFNs and OmniActor-head shares the prediction head. We find that sharing attention shows the least performance degradation, while sharing FFNs orheads leads to significant performance drops. This may be attributed to that FFNs account for large proportion of the total parameters, and heads are crucial for prediction.
remaining parameters.
6.5 PARAMETER UPDATE SIMILARITY
In-group parameter update similarity: We have randomly divided the GUI training data into two in-groups, named GUI-1 and GUI-2 respectively. GUI-1 and GUI-2 are roughly equal size, each accounting for 50% of the total GUI training data. Then, we train the model using GUI-1 and GUI-2 separately. Similarly, the parameter changes induced by GUI-1 and GUI-2 are denoted as dgai.1 and dgui,2 respectively. Then, calculate the cosine similarity between dgui,1 and dgui,2 as the parameter update similariry. As shown in Figure 9 we find that the parameter update similarity across different layers is consistently high, and there
trend of "higher similarity in shallow
18

---

Published as a conference paper at ICLR 2026
0.3
D up dlow
parameteer update update similarit
0.25
022
0.15
11
0.05
2
4
5
6
8
9
10
11
12
13
15
16
28
Layer
Figure 9: In-group parameter update similarity for FFN. There is no trend of "higher similarity in
shallow layers and lower similarity in deep layers".
0.01
gale
00.01
-0.02
parnmetrr znellrry
0LOUG
-0.03
-004
0L00G
-0.05
-0.08
00.09
Head
-0.002
Liger
(a) Parameter update similarity statistic for FFNS
(b) Statistic for prediction head
Figure 10: Parameter update similarity for FFNs and the prediction head. Due to the action space difference, the parameter update similarity on the prediction head significantly low.
layers and lower similarity in deep layers". This further verifies the necessity of designing layer- heterogeneity MoE for GUI and embodied data.
Parameter update similarity on the head:
The main text of our paper does not present the pa- rameter update similarity of the prediction head, and this section supplements this data point. The calculation process of the parameter update similarity on the prediction head consistent with that on other parameters. As shown in Figure 10, the parameter update similarity the prediction head is significantly low, indicating a severe conflitt between GUI and embodied data on the prediction head, which results from the significant action space difference of the two types of data.
6.6
CASE STUDY
We further analyze conflicts and synergies between GUI and embodied tasks. As shown in Figure II Figure 12 and Figure 13 We provide the related case study.
The transfer from GUI to embodied tasks: GUI data enhance the model ability to understand key terms such as "top" in the instructions of embodied tasks, so helping the model distinguish similar objects different positions. We present two tasks in Figure 11. task is "close the top drawer
19

---

Published as a conference paper at ICLR 2026
Task: close the top drawer of the cabinet
Task open the top drawer of the cabinet
90% success rate OmniA ctor- EA
90% success rate OmniA ctor
40% success rate OmniA ctor-EA
70% success rate OmniA ctor
Figure 11: The case study about the transfer from GUI to embodied tasks. GUI data enhance the model ability to understand key terms such as "top" or "bottom" the instructions of embodied tasks, thereby helping the model distinguish similar objects at different positions.
Clock
2:04 PM
Ground Truth: ("action lype": "ogen app". app. "Weather")
Ground Truth: ("action type": "wait")
Von Noy
Prediction gemerated by OmniA etor-G UU: ction:
Incorreci prediction generated by OmnlA ctor-GUl:A A
Open the weather Operaiio:
Open Weatber app. Operation:
mobile.open app(app
mobile.open, app(app, name Weather")
me= Weather')
Eassauass
Prediction generated
Correct prediction genemted by OmnlA A ction: Open the Weather app.
OmniAAtorr ction: Open the Weather app.
Operation: mobile-open, app(app_ngg me- WWatterr).
Operalion: mobile walt(seconds=33)
Figure 12: The case study about the transfer from embodied to GUI tasks. Embodied data enhance the model ability to recognize detailed visual cues for GUT tasks.
of the cabinet", where the target region ("open drawer") has significant features, so the success rates of both OmniActor-EA. and OmniActor exceed 90%. The other task is "open the top drawer of the cabiner", where the model needs to identify the target drawer among three similar drawers. Thus, success rate of OmniAAttr-FA only 40%, while that of OmniActor reaches 70%.
The transfer from embodied to GUT tasks: Embodied data enhance the model ability to recog- nize detailed visual cues for GUI tasks. We observe phenomenon: without embodied data, the model may repeatedly output similar GUI actions. We suspect this is because the model fails to effectively capture detailed visual cues to generate new actions. As shown in Figure 12 the screen has changed; however, AmmiAActo-GGU) still outputs the action open_apee app. This may be because OmniActor-Gri GUI fails to capture the visual cue "Please wait..." on the screen. In contrast, OmniActor correctly output the action moble wait.
20

---

Published as a conference paper at ICLR 2026
Type Error
Grounding Error
Instruction: By mistake, saved contact with the name Shreya, so delete it from my drupe ap.
Instruction: In the Furlenco App. Search for Queen bed the Buy sectio.
Sh
Ground Truth:
Ground Truth: ("action type": 'click", "x" 0.931, "y" 0.41).
Shreya Bit.tooggrraacccoooco
(" ('action, type": "click", 'x": 0.66, "y": 0.098).
Incorrect prediction generated by O ctor-
Incorrect prediction generated by miniA ctor- EA&GUL UI: ction: Go to the previous page. Operation: mobile.back().
E Action: Click
FURNITURE
the Bedroom Icon. Operation:
23932923
@yautoguiclickxx003338e y=0.8775).
Correct prediction
generated by mmiA ctor:
Correct prediction generated by mmni ctor: A ction: Click the search icon the top the
Action: Click the right arow icon at the bottom right comer of the screen.
RCCENTTS
hu Rastory
0 Opertion:
Operation:
cruutoguiclickkxx00..... y333).3)
pyautoguiclickós (ww.ccccciii.eeeeee (0.06662, y=0.0975).
Figure 13: The case study about the performance degeneration brought from data conflict. We present two examples representing two typical types of errors in GUI tasks.
Layer
Layer
Layer
Layer
Figure 14: Visualization about synergy and conflict. We analyze the similarity relationships between GUI and embodied features in different layers. Darker colors indicate higher feature similarity, while lighter colors indicate lower feature similarity.
OmniActor: Compared with OmniActor, OmmiActor-EA&GUU per- forms worse on almost all tasks due to the severe conflict between GUI and embodied data. As shown in Figure 13] we present two examples representing two typical types of errors in GUI tasks: in the first example, OmmiiActorEAA&CHE OmniActor makes a correct prediction; in the second example, OmniActor- AerrAccto--&GGG) predicts the correct action type but fails to identify the accurate click position (a grounding error), whereas Om- Actco achieves correct prediction.
AmnniActo-EEA&CGG
incorrectly predicts the action type (a type error), while
6.7 VISUALIZATION STUDY
We conduct a more in-depth analysis of shallow and deep features. Specifically, we randomly select a subset of GUT data and use OmniActor-Gis to extract intermediate-layer features fgui E RCx where Ngui indicates GUI sample count. Similarly, we randomly select a subset of embodied data and use OmniActor-EA to extract intermediate-layer features fronee E R.N where Nrobot indi- cates GUI sample count. We then calculate the similarity matrix between fewi and freba- We focus analysis the layers (6, 13, 20, 27). In other words, generate four similarity matrices. By visualizing these similarity matrices in Figure 14 we observe that the similarity of shallow features significantly higher than that of deep features. The mean similarity is 0.4206. 0.3018, 0.2000, and
21

---

Published as a conference paper at ICLR 2026
0.1134 for layer 6, 13, 20, and 27, respectively. This phenomenon further indicates that GUI and embodied tasks exhibit synergy at the shallow layers while showing conflict at the deep layers.
6.8 DETAILS ABOUT THE USE OF LARGE LANGUAGE MODELS
We use Large Language Models (LLMs) to aid polish writing. In detail, firstly write all the content to complete the initial draft. After finishing the draft, we use large language model to polish the sentences, mainly for the Methodology and Experiments sections, e.g., "Training data" in Section 4.1. The used prompt is "Please polish the following text and return LaTex--tyle text:".
22


Reviews:
[{"review_id": "fY693yMJCH", "reviewer": "Reviewer_kuDc", "summary": "This paper introduces OmniActor: a unified vision-language-action agent for GUI and Robotic applications.\n\nThere are two main contributions in this paper:\n\n1- Solving the common data conflict problem for generalist agents by observing that the data conflict leads to gradient updates that are consistent in shallow layers, but divergent at deeper layer. The solution suggested to this is by sharing the parameters in the shallow layers but specializing the experts (weights) for the deeper layer closer to the action decision.\n\n2- Unifying the data and action formats across both GUI and Robotics domains.\n\nThe paper has multiple evaluations on various GUI and Robotic benchmarks, and using multiple base models, showing significant uplift and the ability to generalize.", "questions": "- The separation/specialization algorithm used here is manual and depends heavily on similarity statistics (i.e. check the instance type gui/robotics, then route to the correct set of weights). Do you think that this can be automated within the neural architecture itself (i.e. make it a part of the learning process)?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper makes a very insightful observation about the data conflict problem, which is an important challenge for generalist agents.\n\n- The change suggested to solve the data conflict problem is rather simple and easy to implement, and the results show significant improvement.\n\n- The paper is very well written and illustrated. \n\n- The ablation studies and changing the base model evals are very well done. Makes me trust the results more.", "weaknesses": "- The suggested separation method seems arbitrary in a few design decisions (e.g. how to decide on the T parameter for shared layers properly, and the granularity of the separation/specialization: why separate GUI and Robotic only but not Mobile and Computer Use)\n\n- I have doubts about how this separation/specialization method would scale if we have lots of domains not just two.\n\n- The paper does not use any kind of RL to improve the reasoning like other similar systems. Would be interesting to see how RL can be combined with this finding to produce even better numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q0wTswpt1y", "reviewer": "Reviewer_HVrX", "summary": "This paper introduces a unified multimodal agent that can perform both 2D GUI and 3D embodied tasks. The authors find that naive joint training causes conflicts between modalities, so they propose a Layer-Heterogeneity Mixture-of-Experts architecture that shares shallow layers and separates deep layers. Trained on large-scale standardized datasets, OmniActor achieves state-of-the-art or superior performance across both GUI and embodied benchmarks, demonstrating effective generalization between virtual and physical environments.", "questions": "1. Why did the authors choose to fuse 2D and 3D scenes specifically using robot manipulation and GUI grounding tasks? Does this choice lead to excessive data heterogeneity?\n2. Could this design cause a cross-domain data imbalance issue, and if so, how is it addressed?\n3. In the dataset, are the GUI and robot data quantities balanced? Should more diverse robot datasets be incorporated?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper presents a clear insight into the conflict–synergy relationship between GUI and embodied data, supported by empirical analysis. \n2. The proposed Layer-Heterogeneity MoE is biologically inspired, effectively balancing parameter sharing and specialization. \n3. The authors demonstrate strong experimental results, outperforming both domain-specific and prior generalist models across multiple benchmarks.", "weaknesses": "1. The text on the left side of Figure 3 is too small to read clearly.\n2. Please verify whether LIBERO-90 should be the training dataset. Typically, LIBERO-10 is used as the test set.\n3. Table 1 could be improved by including some recent works on LIBERO published in 2025.\n4. I suggest that the authors include real-robot experiments to demonstrate the algorithm’s generalization capability in the 3D physical world.\nI5. It would be valuable to add backbone ablation studies, such as experiments using Qwen-3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UZKV9YJXTD", "reviewer": "Reviewer_h7cP", "summary": "This paper studies unified 2D GUI and 3D embodied control with a single MLLM, showing naïve joint training hurts because optimization is synergistic in shallow layers but conflicting in deeper ones. It introduces a layer-heterogeneous MoE that shares shallow blocks but splits deep experts with task-type routing to reconcile domains. It yields consistent gains compared with baselines.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper is well-motivated, and the proposed method makes sense.\n\n* It shows improvements over both training separate single-domain agents and naïvely mixing all data into one model.", "weaknesses": "* The gains on benchmarks are modest and may fall within variance (e.g., 77.1 vs. 74.5 on GUI tasks).\n\n* In section 4.3, the authors propose a metric called \"parameter update similarity\". Although they show differences in GUI and Embodied groups, more evidence about in-group metric is needed to further support the claim. If in-group metric shows similar trend then the claim may be incorrect.\n\n* Table 2 should include a MoE variant without explicit expert separation to test whether improvements stem from MoE capacity rather than task-specific partitioning.\n\n\n* The appendix should provide fuller experimental details (prompts, representative training samples, and evaluation samples) to support reproducibility.\n\n* Another limitation of this work is that the scope is quite limited with only offline GUI control and one simulated robot task. Including online GUI evaluation and real-robot experiments (e.g., as in Magma) would broaden the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ji1KsDqJVe", "reviewer": "Reviewer_YycE", "summary": "This paper explores the synergy between training generalist agents on UI and embodied action data. OmniActor is a generalist agent across embodied and UI tasks, leveraging a mixture of expert architecture for deep layers to separate the UI and embodied knowledge. Empirical results demonstrate OmniActor outperforming training with only one data source and prior UI or embodiment-specific baselines.", "questions": "1. How does performance vary with different choices of K? \n\n1. How does OmniActor handle varying resolutions between GUI and embodiment data? Other GUI benchmarks, like AndroidWorld, have high resolution images (2400x1080) which are necessary to perceive and interact with all the GUI elements. On the other hand, Libero images are 256x256. L676 says all the images are resized to 448x448, but how would the OmniActor framework train with high-resolution GUI images as well?\n\n1. Why is does OmniActor trained with only the GUI data achieve such higher performance than prior works across generalist and GUI agents?\n\n1. Why is the transfer from GUI to embodied tasks higher than vice versa? Is it because there is more GUI data?\n\n1. Why use distinct prediction heads for GUI and embodied actions, since the vocabulary of these action spaces is distinct? Since the action space is distinct, the token embeddings do not overlap, so there is no difference in splitting versus keeping the embeddings the same.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses the important problem of generalist models between UI and embodiment data. Both domains already have limited amounts of interactive data. Therefore, it's important to leverage the synergies from both to improve performance.\n\n1. The results in Table 1 show that OmniActor outperforms baselines in UI control tasks and closely matches specialized embodiment-specific baselines in embodied tasks.\n\n1. Through the proposed parameter update similarity metric, the paper illustrates how the parameter update diverges for deeper layers between the two data modalities in Figure 4. This motivates the intuitive solution of an MoE for deeper layers.", "weaknesses": "1. The paper does not sufficiently explore the ways of disentangling the UI and embodiment knowledge, and only explores one MoE approach. For example, is it possible to share only the FFN or attention modules as well, or is it necessary to separate both? This is a key contribution of the work and thus should be sufficiently analyzed.\n\n1. The paper does not analyze how to select the \"deep layer threshold\" K value (L468). Likewise, the paper does not analyze how robust the training is to the choice of K. This value is a key hyperparameter that could have a large impact on performance since it addresses the core problem of conflicting optimization directions. It is not clear how to select K based on the visualization in Figure 4 alone. The parameter update similarity appears similar for deeper layers as well in Figures 4a and 4b.\n\n1. The paper does not compare to other VLA models in embodied tasks, such as OpenVLA or pi0. These VLA models are more directly comparable to OmniActor since they are also based on a VLM, are more recent, and perform better on the Libero benchmark.\n\n1. For embodied tasks, the paper only evaluates on the Libero benchmark. Likewise, only data from Libero is used for training. Only evaluating on a single embodied benchmark is contrary to the claim that OmniActor is a \"generalist agent\". A generalist agent should be evaluated on multiple embodiment benchmarks (for example, CALVIN or SimplerEnv).\n\n1. OmniActor achieves strong performance relative to baselines largely because of the starting OmniActor framework rather than a strong cross-domain transfer. The OmniActor, only trained on single-domain data in GUI or embodied tasks, already outperforms most baselines in Table 1.\n\n1. The paper does not analyze which capabilities are transferring between GUI and embodied domains, just that there is broad transfer happening on the benchmarks. Which tasks in Libero and the GUI tasks is the embodiment data improving performance in? This is important for understanding why this cross-domain data is helpful. Likewise, it is important to study for which individual subtasks or behaviors the cross-domain data hurts performance in with the default training to better understand where OmniActor helps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)