Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization
Authors: Zhitao He
Zijun Liu
Peng Li
Yi R. Fung
Ming Yan
Ji Zhang
Fei Huang
Yang Liu
Venue: COLM 2025 (2025)
Keywords: Large language model; Multi-agent learning; Reinforcement learning; UI agent
TL;DR: Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization

Abstract: LLM-based agents have made significant advancements in interactive environments, such as mobile operations and web browsing, and other domains beyond computer using. Current multi-agent systems universally excel in performance, compared to single agents, but struggle with generalization across environments due to predefined roles and inadequate strategies for generalizing language agents. The challenge of achieving both strong performance and good generalization has hindered the progress of multi-agent systems for interactive environments. To address these issues, we propose **CollabUIAgents**, a multi-agent reinforcement learning framework with a novel multi-agent credit re-assignment (CR) strategy, assigning process rewards with LLMs rather than environment-specific rewards and learning with synthesized preference data, in order to foster generalizable, collaborative behaviors among the role-free agents' policies. 
Empirical results show that our framework improves both performance and cross-environment generalizability of multi-agent systems. Moreover, our 7B-parameter system achieves results on par with or exceed strong closed-source models, and the LLM that guides the CR. We also provide insights in using granular CR rewards effectively for environment generalization, and accommodating trained LLMs in multi-agent systems. Our work is available at https://github.com/THUNLP-MT/CollabUIAgents.

Published as a conference paper at COLM 2025
Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization
Zhitao Hel.3tt Zijun Liu?*, Peng Li Yi R. (May) Fung", Ming Yan*, Ji Zhang* Fei Huang*, Yang Liu
Institute for AT Industry Research (AIR), Tsinghua University, Beijing, China Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China HHong Kong University of Science and Technology, Hong Kong, China *Tongyi Lab, Alibaba Group zhitao. hew.onnect.......mmmmmmmm .uteeectt..t............ hk, 2j-1iu24@mmils. www.gtua..ndeeece. cn
Abstract
LLM-based agents have made significant advancements in interactive envi- ronments, such as mobile operations and web browsing, and other domains peroonooopa performance, compared to single agents, but struggle with generalization across environments due to predefined roles and inadequate strategies for generalizing language agents. The challenge of achieving both strong per- formance and good generalization has hindered the progress of multi-agent systems for interactive environments. To address these issues, we propose CollabUIAgents a multi-agent reinforcement learning framework with a novel multi-agent credit re-assignment (CR) strategy, assigning process rewards with LLMs rather than cetironment-specjffe rewards and learning with synthesized preference data, in order to foster generalizable, collaborative behaviors among the role-free agents' policies. Empirical results show that our framework improves both performance and cross-environment gen- eralizability of multi-agent systems. Moreover, our 7B-parameter system achieves results par with or exceed strong closed-source models, and the LLM that guides the CR. We also provide insights in using granular CR rewards effectively for environment generalization, and accommo- dating trained LLMs in multi-agent systems. Our work is available at https:// //gi thub. Httm//HHNNL-MT//COllAUAAAAA
beyond computer using. Current multi-agent systems universally excel in
1 Introduction
Autonomous agents have made substantial progress in interactive environments, such as mobile operations and web browsing, by leveraging large language models (LLMs). These agents hold immense potential not only to automate repetitive tasks but also to enhance decision-making and streamline complex workflows. As a result, they can free up human resources for higher-level problem-solving and innovation. The increasing interest in developing such agents is evident in the growing body of work on, for instance, mobile (Rawles et al., 2023; 2025; Zhang et al., 2024b; Deng et al., 2024a; Wang et al., 2024c), web browsing (Shi et al., 2017; Liu et al., 2018a; Yao et al., 2022; Zhou et al., 2024b; Deng et al., 2023; 2024b), and computer using environments (Xie et al., 2024; Sun et al., 2024), and LLM-based agents targeting on these tasks, including single-agent (Yan et al., 2023; Wang et al., 2024b; Hong et al., 2024b; Cheng et al., 2024a; Hu et al., 2024; Zhang al., 2025) and multi-agent systems (Wang et al., 2024a; Zhou et al., 2023; Zhang et al., 2024c).
However, current efforts in language agent learning still face challenges to balance both performance and generalizability across interactive environments. (1) Single-agent learning
learninngg
'Equal contribuation. $Corresponding authors: Peng Li, Yang Liu. Work done at Institute for AT Industry Research (AIR), Tsinghua University.

---

Published as a conference paper at COLM 2025
SingeeAent Leariog
(b) Mulli- Learinng
Training Environment
don'tt thisk Maybe should back presious app Action [NAVIGATE BACK]
Training Environment
Based knowledge the cnvitemment to upan Calendar add e sitedale... ...
Actor: (OPEN APP 3)
wiggee 1 use Calmmtor schedule maansgeeeent
Fneed open Calender
3pe Calendar
Action: TOPPN
Action: (OPEN APP 31
Baaod the knznnddggge of the onvtromentt I need open Calander add aschstule....
I don'tt think ss. The is Celercke cureer home Action INAVIGATE LETT]
Target Environment
Target Environment
Action: [OPEN Action:[oPEN___
APP
Lagree on ubservatin
"Thre App (R CRICLI gugg
X
Zzed to open Calendkar
page empty
Acaon: [OPEN 3PP
Action:
INAVIGATE LEFT]
Figure 1: Illustration of (a) single-agent and (b) multi-agent learning for environment generalization. In single-agent learning, the agent could encounter obstacles when target environments are different from the training one, while in multi-agent learning, collabora- tion between agents might enable effective decision-making in both environments.
methods (Chen et al., 2023; Gur et al., 2024; Furuta et al., 2024; Bai et al., 2024) heavily relies on in-domain supervised data or rewarding signals improve environment-specifne performance, which restricts its generalization across environments, such as transitioning between web environments using HTML and mobile environments using Android automa- tor. (2) Despite being trained on vast amounts of data from diverse domains, single agents based on open-source LLMs (Zeng et al., 2024; Zhang et al., 2024a; Yin et al., 2024; Liu et al., 2025a) demonstrate only moderate generalization capabilities and lag behind closed-source models. (3) Although multi-agent learning methods (Qiao et al., 2024; Liang et al., 2024) have better performance, existing ones are often constrained by rigid role assignments and lack dedicated designs for generalization, which limits their adaptability to unseen environments, e.g,, an agent designed to retrieve documents for question answering is not feasible to handle mobile operations. In short, it is still unclear how to achieve strong performance and good generalization in interactive environments at the same time.
In this work, we introduce reinforcement learning framework for language multi-agent systems, named CollabUIAgents designed to address the challenges above in real-world interactive environments. The methodology is inspired by the success of classic multi- agent reinforcement learning (MARL) to stimulate collaboration (Ma & Luo, 2022). Beyond previous work (Tran et al., 2025), collaboration among langugae agents may also be beneficial for generalization across environments, as illustrated in Figure 1. Compared to existing methods using LLMs in credit assignment for non-language multi-agent settings (Qu et al., 2025; Lin et al., 2025) (Appendix A), further considerations are made to enrich sparse outcome rewards, enhance adaptation across language environments, and enable multi- agent training for language agents. We propose a novel multi-agent credit re-assignment (CR) strategy, assigning process rewards without using encironment-speciff outcome rewards, but with the world knowledge embedded in LLMs, and learning with synthesized preference data, aiming to foster generalizable, collaborative behaviors.
The core of the framework is to be completely powered by MARL-driven policy learning. rather than fixed role prompts. Specially, an agentic fine-tuned model, as the base LllAgent, is used to initialize a multi-agent system, where each agent has its own policy. After rolling out actions from the policy multi-agent system, the critic agent allocates process rewards at both the agent and conversation round levels based on its comprehension of the environment state and agent behaviors. This approach not only enables finer-grained rewards without training numerous value estimators for agents, but also expands data scales by restoring gen- eralizable behaviors from failed trajectories according to the training environment. To avoid misleading the agents with incorrect CR, policies optimized with preference data synthe- sized by the adversarial agent, to ensure guiding them with correct preference signals. After preference optimization (Rafailov et al., 2023), the multi-agent system is updated in both
2

---

Published as a conference paper at COLM 2025
model parameters and edges of the communication structure. Empirically, we show that with the CR strategy, (1) preference optimization benefits performance and generalization; (2) edge updates is crucial to orchestrate trained LLMs in multi-agent systems.
CollabUlAgents is capable of cross-environment user interface (UI) interaction, supporting both mobile and web environments. Experimental results demonstrate that the trained multi- agent system achieves superior performance compared to existing agentic learning methods and the strong closed-source model Gemini 1.5 Pro (Gemini Team Google, 2024), with Qwen2 7B (Yang et al., 2024) as the base model. The system also achieves performance comparable to the guidance LLM used in the CR, GPT-4 (OpenAI, 2024), in training environments, and even better in an unseen environment. Especially, CollabUIAgents demonstrates effectiveness in largely gapped generalization from mobile to web environments, still comparable to GPT-4.
In summary, our contributions are as follows:
We propose a language MARL framework CollabULAgents with a novel CR strategy, to achieve both strong performance and generalization in interactive environments.
Empirically, we provide insights into the effectiveness of using CR rewards for environ- ment generalization, and the adaptation of trained LLMs in multi-agent systems.
Extensive experiments show that CollabUlAgents surpasses the performance of strong baselines and shows competitiveness, comparable to the guidance LLM of CR in both trained and target environments, even under cross-enviromment generalization.
2 Formulation and Notations
We treat interactive tasks as sequential decision-making process with single-agent or multi-agent systems in the dynamic environments, Agentic systems make decisions based on the current environment state and accumulated interaction history.
Task Formulation Let S be the set of all possible states of given interactive environment, where each s E S represents a specific configuration of the UI and hidden states of the environment at given time step. There is an initial state sp and terminal state s". The action space of an agentic system G is denoted as A, where a E A could represent an action, e-g., clicking buttons, typing, scrolling through content. The environment evolves according to transition function T :.::):
$1+1
T(s,, a:),, St, S/+1 E 5.
(1)
where sy is the state at time step and a; is the action taken by the agent system at that step. The task ends when reaching a terminal state or exceeding the maximum step Tmax- From the state st, the observation 0; is derived in the system holds a policy πi and accordingly selects actions based on shared current observation 0t, the history of past interactions H₁-1 = (so, Ag, ..., St-1, 4-11,, 1), and the message for agent πú from other agents at conversation round denoted C. Specifically, C/ is omitted for single agents:
formatted descriptions in language. Each agent
d = Mi (o. a 1,
(2)
where |9] is the number agents in the system. And a; is determined by an aggregation function fags (which is identity for single agents (19) 1)):
a: fagg
(fa/'] 1, [9]; = 1,
where m is the number of conversion rounds. The task goal is to maximize the outcome reward from the environment over sequence of interactions.
Interactive Environment The observation and action space in interactive environments enormous. Specifically, for the mobile operation environments, which offer an interface that allows agents to receive observations and perform actions on mobile devices, the obser- vation space may include high-resolution screenshots and UT tree from Android automater.
3

---

Published as a conference paper at COLM 2025
The action space mirrors human interactions, featuring gestures (such tapping, long- pressing, and swiping), typing, and navigation buttons (e.g,, home and back). Complete actions listed in Table 6. For web browsing environments, the observation space may include task description, simplified HTML, and current location. The HTML. offers the model both structural and content details of the page, while the current location information allows it to understand its position on the webpage. Consistent with previous work (Lai et al., 2024), we use unified web browsing action space in both of the aforementioned environments. The actions include hover, select, click, etc. Complete actions are in Table 7.
Outcome Reward Function The outcome reward R {0,1} is defined in the environ- ment based on static rules. Static rules are predefined to check whether agents arrive in a successful terminal state inherent to the given task query. Specifically, experiments, Andriod and MobileMiniWoB-+ (Rawles et al., 2025) feature online environments and according state annotations, and Mind2Web (Deng et al., 2023) and AutoWebBench (Lai et al., 2024) use offline trajectories and verifiers. Let the terminal step be t",
the following
[1, if 6p- =
R. - (0, otherwise
(4)
Thus, the outcome reward is sparse, as only the terminal state s* gives out positive rewards, posing a challenge to traditional RL approaches.
3 CollabUlAgents Framework
The CollabUlAAents framework is designed to achieve both high performance and gen- eralizability in interactive environments. It optimizes language multi-agent systems without predefined roles. As shown in Fig- ure 2, the base model undergoes agentic fine-tuning (detailed in Appendix C) and then the MARL process. This section elab- orates the multi-agent system architecture and the multi-agent learning process with multi-agent credit re-assignment.
Task
Envirenment Se
Step
Obs
No Reward
Action
Agents (Il..
Reward Mateis
3. Credit
Action Matris,
Re Re-Assigement
Critic Agent
Moxul
Reand Actions Rolling Out
3.1 Multi-Agent System Architecture
4. Preference Optimization
FIM
System Update
The architecture of the multi-agent system (G) in CollabUIAgents is consistent with previous work (Zhuge et al., 2024a; Liu et al., 2024), which consists of [g] agents, each represented by a policy z; that communicate with each other through message network Eg. As shown in Figure 2, the network is directed acyclic graph (DAG), where messages are passed from 7i to Ní, there an edge pointing from Ti to Ti Specifically, the message comes from the output of TH, The framework re- mains the compatibility for more complex architectures, which is left for future work. We instead use DAGs for simplicity.
Group Initialization
Adversarial Agent
Multi-Agent Reinforcementt earning
и
Agentie Fine-Tunine
Curriculum Learning
Dataa Collection
Base Model
Base UlAggnt
The CollabUlAgents framework with credit re-assignment. The critic agent assess the environment state and the action matrix get granular rewards. Agents are optimized with synthesized preference data to learn rewarded behaviors.
Figure 2:
The agents operate in topological order, and starting from the source to the sink node, allowing each agent to aggregate all responses from its predecessors to form CVi in Equa- tion 2. We define the round of conversation m. In each conversation round, all agents output an action and messages once along the DAG, and each agent receives outputs from itself at last round besides from predecessors, i.e, we keep a contextual memory

---

Published as a conference paper at COLM 2025
of the size equal to 1. The proper size of contextual memories enhances the diversity of imnrrdduc decision making and avoids introducing too long contexts during multi-agent communi- cations. According to Equation 2, at the time step the system produce an action matrix
A, (a))), = 1,.., nij = 1, m.. Then, majority voting is used to decide the final action:
a: fagg (A.) argmax
5
where condition is the indicator function. The agents are all required to output an intermedi- ate decision and collaborate towards a common objective, which allows them to function with the same base model for better efficiency, and operate heterogeneously due to different conversation messages.
3.2 Language Multi-Agent RL
As shown in Figure 2, the essential stage of CollabUlIAeetts is MARL, after agentic fine- tuning the base model into a base UlAgent (Appendix C). We introduce a novel multi-agent Credit Re-Assignment (CR) strategy for better MARL. This approach utilizes the critic agent to provide fine-grained rewards at both the agent and conversation round levels, enabling agents to learn effectively from general knowledge of L.LMs rather than the sparse environment-specff reward, and improve collaboration by preference optimization with synthesized data from the adversarial agent. We also introduce an Edge Update trick to better accommodate trained LLMs in the multi-agent system.
Credit Re- Ke-Assignment Traditional MARL systems use value decomposition for credit assignment (Ma & Luo, 2022), and recent work also uses multiple value estimators to alleviate reward sparcity in interactive environments (Bai et al., 2024). However, training critic models for each agent based on outcome rewards is computationally expensive and may hinder generalization. Instead, using an LLM-based critic agent, represented by Teritie, we provide process rewards in finer granularity for each agent in each conversation round. The action matrix A, is assessed by the critic agent based on the observed environment state or, interaction history H and the task query 9, generating reward matrix R,
-r,)),
i L, 1,,., ], 1,.., 11, where p {0,1) denotes the intermediate reward for a!l.
R, Taritic (o,, A1,,) 0.51,
(6)
where T E [0, [0,1]**0 | In our experiments, use strong 40t-40-2024-08- 06 to guide the CR process for better validity of the assigned rewards. However, the judgement accuracy would not be perfect when the complexity of the environment increases. To overcome the potential incorrect CR results, we introduce an adversarial agent Tadv to synthesize preference data for policy optimization substituting value-based learning.
Specially, Tadv generates low-quality responses ai" paired with a.!
Mr a
-aadv (or, H, 1. ...)).)) if .). =1.
For better clarity, the adversarial agent is only used for generating inferior actions and responses to help learning, instead of applying attacks (Pinto et al., 2017; Bukharin et al., 2023; Yuan al., 2023; Lee al., 2025). Here is the rationale of the design: (1) The critic agent provides a more detailed reward signal for each agent without training multiple value estimators; (2) The critic agent's assessment is based on general pretrained LLMs, which by practice could expand the data scale by restoring intermediate decisions from failed trajectories according to the training environment, and thus may enhance performance; (3) Although errors in CR are inevitable, the synthesized preference data can still provide
agents with meaningful preference signals. For example, even if ay is not the optimal
action, it is still better than a. it. knowledge of the LLM might be misjudged. We argue that the situation is similar when value estimator is poorly learned, where the policy might be optimized to a wrong direction, but our approach could keep rolling out new trajectories based on these actions without
Indeed, environment-specifi, actions beyond general
5

---

Published as a conference paper at COLM 2025
unlearning them. The qualitative study shown in Appendix B demonstrates CR results could be valid without the outcome reward.
MARL with Edge Updates Different from classic MARL settings, agents in CollabUIA- gents could communicate and the message network should also be rolled out in the op- timization. To alleviate the overhead of learning the optimal network from enormous combinations of edges, we introduce an edge update trick, that randomly update edges to form DAG message network, independently from model parameter updates. Through this process, we encourage agents to learn the awareness of multi-agent collaboration and adapt to diverse message networks rather than being rigid in locally optimal DAG pattern. As shown in Figure 2, the edge update is functioned before rolling out actions from the policy system coordinate agents randomly sampled communication graph. Empirical evidence for its effectiveness is shown in Section 4.3. The overall learning objective for each agent n. is formulated as:
LMARL (7+,)
-Egg Exg E
Tiux
logo o (B
log
0t, Al) To Tog
(a
log Tee (a) / loc, Ai) ))
log गT (a 10,, A()
--r (8)
log Tret (a) 0s, Hi)
where A = (H I, c² .q). 02 are the updating parameters of agent π7, and ref; is the original model used as reference policy, Kig) is a fully connected graph of [ nodes, E/3 represents DAG subgraph randomly sampled from K g). &(),&)) is the preference dataset sampled with agents in the message network Egg, o is the sigmoid function, & is the hyper-parameter, and TT, Tef are the base model and reference model. This objective encourages the policy it to assign higher probabilities to preferred actions aj' compared to adversarial actions b/- The policy is updated online and off-policy, similar to previous work (Bai et al., 2024).
3.3
Cross-Environment Adaptation
One of the key strengths of the CollabUlAgents framework is its ability to generalize across different interactive environments, such as from mobile operations to web browsing envi- ronments. The framework supports two approaches for adaptation.
Direct Transfer In scenarios where the new environment shares similarities with the training environment, agents can be deployed directly without additional training. For example, agents trained in mobile UI environments can directly apply their knowledge to web environments, leveraging the knowledge of common interaction patterns and UT elements. The multi-agent setup and according MARL stage are keys decrease error rates through enhancing generalizable collaborative behaviors in agents as expected. The effectiveness is shown in Section 4.2 (applying CollabUlAgentSmaable to web environments).
Continual MARL When the new environment presents significant differences or the high- est success rates are required, agents can undergo further training using MARL with the CR strategy in new environments. This continual learning approach allows agents to refine their policies without stashing the knowledge of previous environments, showing substantial per- formance increase without re-trainig the agent system (Section 4.2, CollabUTAgents
Experiment
4.1 Experimental Settings
Environments We conduct experiments in both mobile operation and web browsing
environments. For the mobile environments, we use AndroidWorld (Rawles et al., 2025) for training and MobileMimiWoB++ (Rawles et al., 2025) for testing: (1) AndroidWorld has 116 programmatic tasks across 20 real-world apps, such as Chrome, Markor, and Pro Expense. (2) MobileMiniWoB++ is derived from MiniWoB++ (Shi et al., 2017), which is web-based

---

Published as a conference paper at COLM 2025
Method
I'araams/Agentt
Input
SR AndroidWorld SRMMiniWoB++ Gennralization
Agenis based Closed-Sounce LLMs
M3A (GPT-4)
N/A
A
Text
30.6
59.7
M3A (GPT-4)
N/A
Multimodal
25,4
67.7
SeeAct (GPT-4)
N/A
Multimodal
15.5
66.1
M3A (Gemini 1,5 Pro)
N/A
Text
19.4
57.4
M3A (Gemini 1.5 Pro)
N/A
Multimodal
22.8
40.3
Agents based on Open- 111 Ms
Qwen2
7B/ 1A
Text
6.2
12.9
Qwen2
7B/ 7B/4A
Text
4.2
15.2
Qwen2
2B/1A
Multimodal
Owen2 VL
2B/4A
Multimodal
0.0
11.5
Qwen2.5 VL
3B/1A
Multimodal
10.1
18.7
Qwen2.5 VL
3B/4A
Multimodal
12.6
21.3
InfiGUIAgent (Qwen2 VL)
2B/1A
Multimodal
9.1
15.6
5.6
DigiRL (Qwen2.5 VL)
3B/1A
Multimodal
22.3
35.2
16.5
DigiRL (Qwen2.5 VL)
3B/4A
Multimodal
20.1
20.0
Our Methods
UlAgent (Qwen2)
7B/1A
Text
18.9
48.4
35.5
UlAgent (Qwen2)
7B/4A
21.4
40.3
UlAgent (Qwen2)
7B/6A
Text
18.9
41.5
CollabUIAgentsmann wwe22)
7B/4A /4A
Text
29.3
61.2
48.3
CollabU (Qwen2)
7B/6A
Text
32.7
59.7
46.9
Table 1: Experimental results mobile operation environments. Success rates (SR) in AndoridWorld and MobileMiniWoBB+ (MMiniWoB++)) are listed. AGeneralization indicates the performance gap between the base model and agent learning methods based on the model in MobileMiniWoB++: "7B/4A" denotes a four-agent system upon 7B model.
benchmark. MobileMiniWoB++ shares the same observation space AndroidWorld and supports 92 tasks from MiniWoB++. We use the success rate (SR) as an evaluation metric. For the web environments, we leverage Mind2Web (Deng al., 2023) for training and AutoWebBench (Lai et al., 2024) for testing: (1) Mind2Web features over 2,000 open-ended tasks sourced from 137 websites in 31 different domains. (2) ButnWe a bilingual benchmark featuring approximately 10,000 traces, from mainstream Chinese and English websites, providing a diverse dataset for web browsing. We use the step-success rate (SSR) as the evaluation metric. For agent learning methods on open-source LLMs, we use the performance gap between the base model and the trained model in unseen environments, AGeneralization/ to indicate generalizability for each agent learning method.
Evaluated Methods We compare our framework against the following methods under their original settings: (1) M3A (Rawles et al., 2023) is prompt-based multimodal agent, which combines ReAct- (Yao et al., 2023) and Reflexion-style (Shinn et al., 2023) prompting to interpret user instructions and screen content, then update its decisions. (2) SeeAct (Zheng et al., 2024) is a prompt-based navigation agent originally designed for GPT-4V to perform actions with visual input and textual choices. (3) InfiGUUAgent (Liu et al., 2025a) fine-tunes a generalist mobile operator model with multimodal input. (4) DigiRL (Bai et al., 2024) is an off-policy RL algorithm for single agents trained on the same dataset as CollabUIAgents, based on Qwen2.5 VL 3B (Wang et al., 2024d). (5) SeeClick (Cheng et al., 2024b) is fine-tuned visual GUT agent that automates tasks relying on screenshots and employs GUT grounding. We leverage Qwen2 7B as our base model and evaluate the following systems derived from the model: (1) Base Model directly calls the general instruction-tuned model to interact with the environment without fine-tuning. (2) Base UIAgent is the base model that has undergone agentic fine-tuning in AndroidWorld. (3) CollabUTAgentsmmohl
our framework trained AndroidWorld with 4, 3. (4) CollabUIAgents web builds upon CollabULAgents with continue MARL on the training set of Mind2Web, which is autonomously collected with the pipeline in Appendix C and D.
4.2 Main Results
Effectiveness in Mobile Environments In this section, explore the effectiveness of our proposed method in trained and unseen mobile operation environments. Experimental results in mobile environments are shown in Table 1. The performance of proprietary
7

---

Published as a conference paper at COLM 2025
Method
TParams/AAgentt
Input
Mind
Autheeeeeeencc Aennrallzatton
Agenis based Closad-Source
LLMs
GPT-3.5-Turbo
N/A
Text
17.4
10.7
GPT4
N/A
Text
30.9
37.8
Claude2
N/A
Text
10.5
Agents bused on Open Source LLMs
Qwen2
7B/1A
Text
7.4
8.5
LLaMA2
7B/1A
Text
2.9
LLaMA2
70B/1A
Text
10.6
SFI (Qwen VL)
9.6B/1A
Multimodal
10.1
SeeClick (Qwen VL)
9.6B/IA
Multimodal
20.9
Our Methods
UlAgent (Qwen2)
7B/1A
Text
11.9
12.8
4.3
UlAgent (Qwen2)
7B/4A
Text
13.2
14.0
CollabUlAgenttaan (Qwen2)
7B/4A
Text
16.2
17.7
CollabUlAgenttan SagibbuAggttnn--ee genttsn-wweeb (Qwen2)
7B/4A
Text
30.7
34.7
26.2
Table 2: Experimental results web browsing environments. Average step success rates (SSR) in Mind2Web and AutoWebBench are reported. "SFT" denotes the base model supervised fine-tuned in Mind2Web. AGeneralization indicates the gap between the base model and agent learning methods based on the model in AutoWebBench. "7B/ 7B/4A" denotes four-agent system upon 7B-parameter model.
models face instability with input format, e.g., text-only multi-modal input. With ap- propriate agentic fine-tuning, open-source models like Qwen2 7B could gain significant performance improvement, and even higher within multi-agent systems ("Base UlAgent"). CollabUUAgents achieves the best results among systems based on open-source LLMs. Remarkably, it outperforms Gemini 1.5 Pro in both environments and achieves performance comparable to or better than GPT-4, due to its instability. These outcomes demonstrate the effectivenese eness of our framework and provide evidence for the validity of the CR strategy. On the other hand, InfiGUIAgent uses a large amount of training data but falls behind other methods; DigiRL shows good performance in the training environment, but falls short in generalization compared to CollabUIAgents In addition, increasing the number agents leads further improvement the training environment but does not improve as much in unseen environments, showing a trade-off. This identifies the importance maintaining the same number of agents in both the MARL and the deployment stages for the best effectiveness and generalization.
which improves greatly unseen tasks.
Takeaway 1: The critic agent could generate valid process rewards guide the policy multi-agent system learning during CR.
Generalization from Mobile to Web Environments In this section, examine the cross-environment generalization capabilities of our proposed method. Results for web environments are presented in Table 2. Directly transferring CollabUIAgentsmaaii obtained from the AndroidWorld environment yields substantial performance improvement over Base UlAgent vanilla Qwen2; however, the absolute gains remain modest and slightly lags behind fine-tuned SeeClick in the training environment. CollabUIAgentsmmaln with continue MARL on data from Mind2Web, collected with the pipeline depicted Ap- pendix C, could significantly improve the generalization on web browsing environments. CollabUTAgentsmm >web achieves results comparable to GPT-4, without training on large scaled web data. It is also noteworthy that we do not require human-annotated data for the Mind2Web environment, which significant advantage in transferring the agent sys- tem to new environments. Also, generalization performance (AGeneralization) in the unseen environment indicates that our method generalizes well in diverse web browsing tasks.
Takeaway 2: Continue MARL significantly improves the language multi-agent sys- tem for cross-environment generalization.

---

Published as a conference paper at COLM 2025
4.3
Ablation Study
The results of the ablation study are presented in Table including replacing preference optimization with rejective fine-tuning, removing the CR, and removing edge updates in MARL. The empirical findings highlight the following key insights:
(1) Further training of the Base UlAgent with trajectory data using either rejec- tive SFT ["CollabUIAgentsmoole w/ PO + RFT") or DPO whole trajectories ("CollabUTAgentsmmolll w /o CR") improves performance, with DPO showing superior results. The primary distinction between these methods is that SFT can only learn from correct actions, while DPO can learn from both correct and incorrect actions. (2) CollabUTAgentSmaable introduces credit re-assignment, providing more granular feed- back that facilitates exploration of the large action space at each step. The synthesized preference data also helps generalization through preference optimization, compared to CollabUlAgents w/o o CR which only uses the outcome reward R. This boosts both performance and generalizability, yielding the best overall results.
Takeaway 3: Preference optimization with synthesized data enhances performance and generalization with CR-generated rewards.
(3) Combining multiple agents based on a vanilla base model using random edges leads to modest im- provements ("Base UTAgent" (n = 4)), and the similar trend exists for DigiRL. in Table 1, underscoring the importance of proper agen- tic fine-tuning and MARL with online multi-agent trajectories. (4) A compari- son between systems with and without edge updates ("CollabUIAgentsmmmn "w/o edge update") demonstrates that the edge update trick contributes to further accommodating language agents in complex multi-agent systems with communications.
Method
SR AndroidWorld
MMMinWWoB++
Single-Agent Systems
Qwen2
6.2
12.9
UTAgent (LLaMA2) UTAgent (Qwen2)
15.1
43.7
18.9
UTAgentael -ritii (Qwen2)
10.7
19.5
Multi-Agent Systems (n = 4)
Qwen2
8.6
16.1
UlAgent (Qwen2)
21.4
53.2
UTAgent (Qwen2)
12.5
CollabUlAggettsmabbll w/ PO RFT w/o w/o Edge Update
29.3
61.2
23.2
54.8
25.0
56.4
27.6
58.1
CollabUlAgentsm
26.7
58.1
Table 3: Ablation study. Success Rates (SR) in the An- dorid and MobileMinnWoo++ (MMiniWoB+++) envi- ronments are reported. All methods are based on Qwen2 7B. "PO" denotes preference optimization, and "PO - RFT" means performing rejective fine-tuning based on CR rewards, i.e, filtering unrewarded data.
Takeaway Edge updates during MARL help accommodate language agents in the multi-agent system.
(5) After cross-environment reinforcement learning on the web, CollabUlAgentsm exhibits impressive autonomous adaptability in the new environment, with only minor performance fluctuations the original mobile environment, thereby validating the stability of our method.
5 Related Work
Agents Interactive Environments Before the advent of LLMs, agents relied tradi- tional RL to perform interactions such as clicking and typing (Liu et al., 2018b; Humphreys et al., 2022). However, recent advancements have shifted towards leveraging foundation

---

Published as a conference paper at COLM 2025
models with in-context learning fine-tuning across various interfaces, including mo- bile (Wang et al., 2023; Hong et al., 2024b), web (Lai et al., 2024; Deng et al., 2024b), and computer using environments (Xu et al., 2024a; Wu et al., 2024c). Recently, there are emerg- ing methods designing process rewards (He et al., 2024a; Pan et al., 2024; Xu et al., 2024b; Wu et al., 2024b; He et al., 2025; Xu et al., 2025; Liu et al., 2025b), synthetic data (Yuan et al., 2024; Qin et al., 2025) and language RL (Bai et al., 2024) for better performing single agents.
Interactive Environments for Agents To effectively evaluate language agents, is es- sential to create environments that replicate real-world conditions and deliver accurate rewards (Rawles et al., 2023; Deng et al., 2023). MiniWoB++ (Shi et al., 2017) is lightweight framework that features small, synthetic HTML pages with parameterized tasks. We- bArena (Zhou et al., 2024a) and its visual counterpart, VisualWeb (Koh et al., 2024), simulate websites spanning up to distinct domains, while WorkArena (Drouin et al., 2024) focuses on enterprise software. For more specialized environments, WebShop (Yao al., 2022) simulates an e-commerce platform for online shopping.
Prompt-Based Multi-agent Learning Collaboration among multiple LLM agents has shown effective for various tasks (He et al., 2023; Hong et al., 2024a; Wu et al., 2024a; He et al., 2024c; Jin et al., 2024; Qian et al., 2024; He et al., 2024b; Wang et al., 2025). However, employing a static architecture without team optimization may restrict the performance and generalization. Chen et al. (2024) selects a fixed number of agents from a set of man- ual prompt candidates via an additional LLM during each round of discussion. Zhuge et al. (2024b) unify language agent systems by describing them as optimizable computa- tional graphs and develop optimization methods for nodes and edges, enabling automatic improvements of agent prompts and inter-agent orchestration.
Conclusion
In this paper, we introduce CollabUlAgents, novel multi-agent reinforcement learning framework aimed at addressing the challenge of balancing strong performance and general- ization interactive environments. The framework employs a credit re-assignment (CR) strategy that utilizes world knowledge embedded in LLMs to assign process rewards, and optimize policies with synthesized preference data. Through extensive experimentation, the proposed framework not only surpasses existing methods in terms of performance metrics but also exhibits exceptional generalization capabilities. Notably, achieves results exceed, those of closed-source models
that are comparable to, and in some cases when deployed in previously unseen environments. Overall, CollabUTAgents presents promising solution to the limitations of current agent learning methods by offering more flexible, data-efficient, and generalizable approach for real-world applications.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (No. 62276152), and funding from Wuxi Research Institute of Applied Technologies, Tsinghua University under Grant 20242001120.
References
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. DigiRL: Training in-the-wild device-control agents with autonomous reinforcement learn- ing. In The Thirfy-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
Yoshua Bengio, Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pp. 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 10.1145/1555 1553//4553.... 11533380. 1553380.
URL htttp:://doi.oor/1/0/ wtt..o////1..11/55533/3777777771 1155/553374
10

---

Published as a conference paper at COLM 2025
Alexander Bukharin, Yan Li, Yue Yu, Qingru Zhang, Zhehui Chen, Simiao Zuo, Chao Zhang, Songan Zhang, and Tuo Zhao. Robust multi-agent reinforcement leamiigg via adversarial regularization: Theoretical foundation and stable algorithms. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httph::/t///oo. /ttp///pppeeeeeeeeeeeeeeeeeeeeee net/forummid= FmZVRe0gg8.
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint ar arXit:2310.05915, 2023.
URL https://aaxxi www.:///g/bb//////// 251000555.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, 2024. URL /ttps:///penrevveww.. aerpenreiie..net/orrmmmt www.eeeeeoommimeeomomoooo diEEgggDyg..
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 9313- 9332, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653 w0w.022.ccloooo..555555 URL https:://claathholoon org/ acl acl-long. 505/.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUT grounding for advanced visual GUT agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Comprutttional Linguistics (Volume 1: Long Papers), pp. 9313- 9332, Bangkok, Thaillnd, August 2024b. Association for Computational Linguistics. doi: 10.18653 v1 /2l.//2222loloogg.......
URL https:
www.ethhooogy.ooooo.a.oo org/ acl-long. 505.
Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Liujianfeng Liujianfeng, Ang Li, Jian Luan, Bin Wang, Rui Yan, and Shuo Shang. Mobile-bench: An evaluation bench- mark for LLM-based mobile agents. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 8813-8831, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/v v1 /2024 aol--loooo.7888 URL https://acllanthol..g org/2024. acl mocl-loog.4777 478//
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards generalist agent for the web. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Proceedings of Advances in Neural Information Processing Systems 36 (NeurIPS 2023) Datasets and Benchmarks Track, volume 36, pp. 28091-28114. Curran Associates, Inc., 2023. URL https://pproceedings.oo neurips. cc/paper tttees/pape//e223/// org.://006...eeee111188822666660
and Benchmarks.. .ddff
Yang Deng, Xuan Zhang, Wenxuan Zhang, Yifei Yuan, See-Kiong Ng, and Tat-Seng Chua. On the multi-turn instruction following for conversational web agents. In Lun-Wer Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1995-881, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/v1/ org. 2024. Long-loog..444 477//
2024.acl-long-44 477. URL https://aclanttoll/
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks? In Forty-first International Conference on Machine Learning, 2024. URL https: htttepee//penrevieww.. id-BRfqYrikdo.
Hiroki Furuta, Kuang-HHun Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixi- ang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-iinetune
11

---

Published as a conference paper at COLM 2025
foundation models. In The Twelfth International Conference Learning Representations, 2024. URL https://openrevieeee wwwpr/eewwwwwwweefrrroeeeee net/forumiidee F=meiiiio
Gemini Team Google. Gemini 1.5:
Unlocking multimodal understanding across millions of tokens of context. arXiv preprint ar arXo::2003.05530 2024. URL https://arxiv www..oggg/aa.... /rg/abs/ 05530.
Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. The Twelfth International Conference on Learning Representations, 2024. URL https: //tperee//ppeeeiiewww net/forum?? wwwommmiid:g00 for.mm?d-ggJttrumvvg
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhen- zhong Lan, and Dong Yu. WebVoyager: Building an end-to-end web agent with large multimodal models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6864-6890, Bangkok, Thailand, August 2024a. Associa- tion for Computational Linguistics. doi: 10.18653/v /v653/vl// wlw.....clooooog..... URL https: //aclanthology.orgg org/ acl long.
Zhitao He, Pengfei Cao, Yubo Chen, Kang Liu, Ruopeng Li, Mengshu Sun, and Jun Zhao. Lego: A multi-agent collaborative framework with role-playing and iterative feedback for causality explanation generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9142-9163 2023. URL https: https://aclantholl /aa/aaaaaaaantyyyy... org/2023. findings-emnlp 313.
Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. Agentscourt: Building judicial decision-making agents with court debate simulation and legal knowledge augmentation, 2024b. URL
wwtp:///////b//..////////// 02959.
Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. Simucourt: Building judicial decision-making agents with real-world judgement documents. arXiv e-prints, pp. arXiv-2403, 2024c. URL https://arxivv http:///aaxxvoooo//mmm//// 24t.//222.....11..
Zhitao He, Sandeep Polisetty, Zhiyuan Fan, Yuchen Huang, Shujin Wu, and Yi R. Fung. Mmboundary: Advancing milm knowledge boundary awareness through reasoning step wrgg 23224.
confidence calibration, 2025. URL https://arxxee http:///aaxiv..ooabb//////a
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openrewieee mett/foruumiddvVVtttAGGG
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. CogAgent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, Pp. 14281-14290, 2024b.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: /ttps://openrewwee http://pppppeeeeetttppteeee///et net/ forum?id=nZeWKeeFya.
Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, Yuhuai Li, Shengze Xu, Shawn Wang, Xinchen Xu, Shuofei Qiao, Kun Kuang, Tieyong Zeng, Liang Wang, Jiwei Li, Yuchen Eleanor Jiang, Wangchunshu Zhou, Guoyin Wang, Keting Yin, Zhou Zhao, Hongxia Yang, Fan Wu, Shengyu Zhang, and Fei Wu. Os agents: A survey computing devices use. Preprints, December 2024. doi: 10. 10.20944 prtp:///ppppnt2222122222. 222 v2. URL htttps///off org/10 p00 2294 2294. v1.
mllm-based agents for general
12

---

Published as a conference paper at COLM 2025
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. In International Conference on Machine Learning, pp-
PMLR, 2022.
Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models. ariiv preprint arXiv:2406.10800 2024. URL https://arxiv. org/pdf /200/ 1890.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. VisualWebArenat Evalu- ating multimodal agents on realistic visual web tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 881-905, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 00.18653vv1/0224.e0 httpp://aaee https://aclaanthllog/ org/ acl -ong. 50.
URL
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. AutoWebGLM: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp. 5295-5306, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704901. doi: 10. 1145/36375288.6771
URL nttps://don org/10. 1145/3637528888 3671620.
Sunwoo Lee, Jaebak Hwang, Yonghyeon Jo, and Seungyul Han. Wolfpack adversarial attack for robust multi-agent reinforcement learning. In Forty-second International Conference on Machine Learning, 2025, URL /ttttps://pperrvvvie wttp:/wwwwwweeooom/////////////e
Xuechen Liang, Meiling Tao, Yinghui Xia, Tianyu Shi, Jun Wang, and JingSong Yang. Cmat: A multi-agent collaboration tuning framework for enhancing small language models. ar Xiv preprint 24t.::400.00663. 2024. URL https:://arxins www.aaawwww....ooa//eeeeeeeooore wwwp///////////// abs/2404 010.0....66.......
Muhan Lin, Shuyang Shi, Yue Guo, Vaishnav Tadiparthi, Behdad Chalaki, Ehsan Moradi Pari, Simon Stepputtis, Woojun Kim, Joseph Campbell, and Katia Sycara. Speaking the language of teamwork: Lim-guided credit assignment in multi-agent reinforcement learning. Xivev preprint ar Xix:22502.03723 2025. URL https: http://v...ro//ab///22..... 037233.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang. Reinforcement leamnin on web interfaces using workflow-guided exploration. In International Conference on Learning Representations, 2018a. 8a. URL
Thomas F Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. Learning design semantics for mobile apps. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, pp. 569-579, 2018b.
Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection. arviv preprint arXiv:2501.0457;, 2025a. URL https://arxiv.org htt.:/orggg//////eeeee 04575.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic Loweee agent network for task-oriented agent collaboration. In First Conference on Language Modeling, 2024. URL https: //tte:e//oppereviiew wwwpeeewwwe.wneeeeeeeee net//orumPid=xIIINN XX99
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling, 2025b. URL https:
Yanhao Ma and Jie Luo. Value-decomposition multi-agent proximal policy optimization. In Proceedings of the 2022 China Automation Congress (CAC), pp. 3460-3464, 2022. doi: 10.1109/ Www.0///ACC/5.....0 10054763.
13

---

Published as a conference paper at COLM 2025
OpenAl. Gpt-4 technical report. arXio preprint arXii:2303.08777, 2024. URL https://arxiv.. org/abs/2303.comm 230/00
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. In First Conference on Language Modeling, 2024.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Cupta. Robust adversarial reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2817-2826. PMLR, 06-11 Aug 2017. URL https: https://proceediiggg mlr press/ v70/pinto17a 7 html.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative agents for software development. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15174-15186, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v /v1 11/86/////2224.acllllrgg.g 810. URL https: / /ttaaaaaaanntooooooooo org wrg/2024..... 20t.2222..cclllog acl-long. 810/.
Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Jiang, Chengfei Lv, and Huajun Chen. AutoAct: Automatic agent learning from scratch for QA via
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volune Long Papers), pp. 3003-3021, Bangkok, Thailand, August 2024. Association for Computational 2024.
Linguistics. doi: 10.18653/v1. 11/2024.acl-long 165. URL 1el-long.165/
Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, and Furu Wei. Scaling laws of synthetic data for language models, 2025. URL 19551.
https://arxiv.. http:///axii..or/bb/////// wrtp
Yun Qu, Yuhang Jiang, Boyuan Wang, Yixiu Mao, Cheems Wang, Chang Liu, and Xiangyang Ji. Latent reward: LIm-empowered credit assignment in episodic reinforcement learning. ar Xiv preprint arXiv:2412.11120, 2025. URL https: //axxiv org/ aa////b//2//11 11120.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https::/opemreview net wett/formm?? HPSHAAIIKa
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lilli- crap. Androidinthewild: A large-scale dataset for android device control. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Proceedings of Advances in Neural Information Processing Systems 36 (NeurIPS 2023) Datasets and Benchmarks Track, volume 36, PP- 59708-59728. Curran Associates, Inc., 2023. URL https://ppoocedinss.eeeree newrips. cc/paper samperrfiiesse bboe488000e0003300900200-0-----0 bet.a//eeeeeeeeetteeee..ee.eeeer Bww../nnncaaass.... pdf.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E Bishop, Wei Li, Folawiyo Campbell-Ajala Daniel Kenji Toyama, Robert James Berry, Divya Tyamagundlu, Timothy P Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview. net
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pp. PML.R 2017.
14

---

Published as a conference paper at COLM 2025
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL Forum?id=vAEllfcckW
net/
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, and Zhiyong Wu. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arviv preprint afXiv:2412.19723, 2024. URL https://arxiv.orr abs/2412. 19723.
Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O'Sullivan, and Hoang D. Nguyen. Multi-agent collaboration mechanisms: A survey of llms. arviv preprint arXiv:2501.0633 2025. URL httppp://aaaxv..rrr/b///////
Bryan Wang, Gang Li, and Yang Li. Enabling conversational interaction with mobile UI using large language models. In Proceedings of the 2023 CHI Conference Human Factors in Computing Systems, pp. 1-17,2022
Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024a. URL https: //op://////eeevewww www.:eetfforumme mi O0nBMR1kc8.
Junyang Wang, Haiyang Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agentt Autonomous multi-modal mobile device agent with visual perception. Xrviv preprint arXio:2401.16158, 2024b. URL https://arxx..or//ass//// 16158.
Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. MobileAAenttBenee An efficient and user-friendly benchmark for mobile Ilm agents. arxiv preprint 7XXi:24466088144 2024c. URL https://arxiv.orr abs/2406.081
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhanc- ing vision-language model's perception of the world at any resolution. arXiv preprint 1rXi9:2409.12191, 2024d. URL https://arxix org/ 2b59 12191.
Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint ar Xix::2011.1133. 2025. URL https://arxiiv http:://aaxxivo.oaaabbbs
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi- agent conversations. First Conference on Language Modeling, 2024a. URL https:// openreview www.eetofooouumiddAAAaa hNKS.
Shujin Wu, Yi R. Fung, Sha Li, Yixin Wan, Kai-Wei Chang, and Heng Ji. Macaroon: Training vision-language models to be your engaged partners, 2024b. URL https://axvvi... org/ abs/2406. 14137.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng. Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. OS-copilot: Towards generalist computer agents with self-imppoveement In Proceedings of the ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024c. URL https://openrewee http://w.eee/froommmddd///rrrrl
Tianboo Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan
15

---

Published as a conference paper at COLM 2025
Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Bench- marking multimodal agents for open-ended tasks in real computer environments. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint. In Forty-first International Conference on Machine Learning, 2024. URL https::/opeeneeiieee net/ forumridd c1ccm1KAAGrrrr.
Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, et al. CRAB: Cross-environmennt agent benchmark for multimodal language model agents. ariio preprint ar arXiv:2407.0151, 2024a.
URL
Yao Xu,Shizhii Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Guang Liu, Kang Liu, and Jun Zhao. Generate-on-graphh Treat 1lm as both agent and kg in incomplete knowledge graph question answering, 2024b. URL https: /ttp:ss//////vv./////eee org/ grgg/bbs/244/ 11044..44444 14741.
Yao Xu, Shizhu He, Jiabei Chen, Zeng Xiangrong, Bingning Wang, Guang Liu, Jun Zhao, and Kang Liu. Llasa: Large language and structured data assistant, 2025. URL https: //arxiv.org wrgg/bs///////1 14460.
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. GPT-4V in wonderland: Large multi- modal models for zero-shot smartphone GUI navigation. arviv preprint arXiv:2311.07562, 2023. URL https: //axiiv brrg/bbs/20 2311.1111 07562.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. arXiv preprint ar 240..1167.11 2024. URL https: https://arxiv.ort hwwp.xiiorg/bbb/40001 10671.
Shunyu Yao, Howard Chen, John Yang, and Karthik R Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: //opnr/////eeevewww http://ppnrreiee.eeeeerrrrrrm/// net/forum?iddRKKnufF
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https: hetp:://pppp/iiieeee /ett forum??d=NE.wluuYU
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 12380-12405 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 /v w1/2222..cl-long.. acl-long. 670/.
URL
wrrg//rg////0 2004..
Lei Yuan, Ziqian Zhang, Ke Xue, Hao Yin, Feng Chen, Cong Guan, Lihe Li, Chao Qian, and Yang Yu. Robust multi-agent coordination via evolutionary generation of auxiliary adver- sarial attackers. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Infelligence and Whirly-Fifit Conference Innovative Applications of Artificial Intelligence and Thirleenth
16

---

Published as a conference paper at COLM 2025
Symposium Educational Advances in Artificial Intelligence, AAA113/1AAA131AAA3/3 /AA//23 AAAI Press, 2023. ISBN 978-1-57735-880-0. doi: 10.1609/ aaii.v37: URL
https://doi org/oog///// /0. 609////ai..... w3w.1311110 1077100.44 26388.
Lifan Yuan, Yangyi Chen, Xingyao Wang. Yi R. Fung, Hao Peng, and Heng Ji. Craft: Customizing mlms by creating and retrieving from specialized toolsets, 2024. URL https: wwaaxii.orggabs//23302 2ttpo//2399.1144 17428.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. AgentTuning: Enabling generalized agent abilities for LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 3053-3077, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653. /v1 221//222..fmmg 20044findiigg--cc.115 URL https://acclatthooog. org/ 2024. findings- acl. 181/
Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Large language model-brained gui agents: A survey. arxiv preprint ar Xiv:2411.18279, 2025. URL /taps: org/abs/225 abs/2411. 18279.
Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liangwei Yang, Yihao Feng, Zuxin Liu, et al. Agentohana: Design unified data and training pipeline for effective agent learning. arXiv preprint ar Xio::442..15006 2024a. URL https:/ https://arxiv: http///aixi..rrab/////// 2402. 15506.
Jiwen Zhang, Jihao Wu, Teng Yihua, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, wwl.---ctcoonououghhhhh for GUT agents. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 12016-12031, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10. 10.18653/v1 /1 2024 www.diinnnennniill 702. URL https: //t::://accaa http:///llaaaaoooyyooyoo org 2024 findings- emnlp. 702/-
and Duyu Tang. Android in the zoo:
Yao Zhang, Zijian Ma, Yumpu Ma, Zhen Han, Yu Wu, and Volker Tresp. WebPilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration. ar Xiv preprint ar Xiv::2248 159788 2024c. URL https: / /aaxxiv..viv/////a
/ss//bb//448.. 15978.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a generalist web agent, if grounded. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 61349-61383 PMLR, 21-27 Jul 2024. URL https: http::://proceedingg. www.mmrrpprerre .pres// v235/zheng24e. html.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In Proceedings of The Tweifth International Conference on Learning Representations (ICLR 2024), 2024a. URL https: //openreview. net.eeww.ee///eeeeeeeeeeeeeeeeee wwt. /ooo
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024b. URL hwtph:://///oo http::///ppeeeeeeeeeee www.wew.eeefffrreeee fet/frrmm?id= oKn9c6ytLx.
Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Xiangru Tang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An open-source framework for autonomous language agents. arXiv preprint 078777 07870.
ar 230907700, 2023. URL http://awww..org//////...////
17

---

Published as a conference paper at COLM 2025
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. GPTSwarm: Language agents as optimizable graphs. In Ruslan Salakhutdinov,, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 62743 62767. wpppss/v23//huppppeee
PMLR, 21-27 Jul 2024a.
URL https:: petps::/proccedingg:
html.
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. GPTSwarm: Language agents as optimizable graphs. Forty-first International Conference on Machine Learning, 2024b. URL https: https: /opprreeeeee net/forum? id=uTC9AFXIng.
18

---

Published as a conference paper at COLM 2025
A Comparisons with Non-Language Multi-Agent Learning Methods
To situate CollabUIAgents among recent studies that also employ LLMs for multi-agent credit assignment, we contrast with the most closely related approaches of Lin al. (2025); Qu et al. (2025). Table 4 summarizes the key differences.
Feature
CollabUIAgents
LCA (Lin et al., 2025)
LaRe (Qu et al., 2025)
Primary focus
Perf. & gen. of language agents across interactive environments Mobile / web dependent language tasks Fine-grained process rewards & synthesized DPO preference
Sparse-team credit assignment
Episodic credit assignment (redun- dancy/ambigg
Core application
GridWorld, Pistonball
MuJoCo, MPE
Assigned reward
Dense potential-based rewards from LLM rankings
Proxy rewards via latent encoder-decoder
Generalization goal
X
X
Interactive env.
X
Dynamic MARL communication
X
X
Table 4: Comparison with other LLM-based credit-assignment methods for non-language agents.
Specifically Designed for Language Agents & Interactive Environments: Unlike LCA and LaRe-boke devised for classical MARL control domains our framework is purpose- built for language agents acting in complex environments (e.g. mobile navigation, web browsing) that combine large observation spaces with natural-language actions and agent communication.
LLM-based process reward assignment: We innovatively employ an LLM as Critic Agent to assign fine-grained "process rewards". The assigned step-level and agent-level process rewards reflect the semantic utility of each action; these rewards are then distilled into synthesized preference data for DPO, moving beyond sparse environmental signals.
Emphasis on Enooo-Enviro Generalization: Our CR strategy is designed to lever- age the general world knowledge of LLMs. Combined with continual MARL and mech- anisms like communication structure (edge) updates, this significantly enhances the generalization capabilities of language multi-agent systems unseen environments.
This targeted design delivers superior performance and generalisation for language-driven multi-agent systems, establishing CollabUIAgents as distinct advancement over contem- poraneous aasit- methods.
B Case Studies on Mobile Operation Environments
Figure 3 shows an example of task execution steps in the AndroidWorld environment during the rolling out phase, where agents in the multi-agent system collectively accomplish the task within the shortest path. We also demonstrate that the reward from the CR process is correctly identified for each action (only part of rolled out actions shown).
19

---

Published as a conference paper at COLM 2025
Running lask: FilesMoveFilo
Running FolesMoveFile goal "Move file holiday_ photosjjg from Podcasts within gphone x86 64 storage the DCIM within the sdik, gphone x86 storage the ndroid filesystem."
Film
UI element
elemmnt 8
UI element
action type=/open app', app name=Files'
action type=tlicck', Index=1
action type=/click', index=0
action type=tclcck, Index =24
action type='scroll", directiondown'
UI detient
Decision-maken
Indlo=44
UT element 16
demennt
Incles=O
Indlax
mom
Nadignti
lack:
Agnnt
Action
Reward
action typo=Tongg.ee press
action type-
action type= inde=4
indexc=1
index:
COM
Peluads
Ui elmment 3
U obomennt 12
action type=/lickk
action, (ysooo__yy==clikkk
action, type=/click'
action type='status',
index=3
index=12
index=8
goal status=/completr
Figure 3: An example of task execution steps of CollabUIAgentsmodie which is trained on Qwen2 7B and facilitates 4 agents engaging rounds of conversation. "action_tyyp represents the action taken, and "index" represents the index of the UI element. The positions of the relevant elements on the interface are marked red.
20

---

Published as a conference paper at COLM 2025
C Agentic Fine-Tuning in CollabUlAgents
C.1 Methodology
The agentic fine-tuning process of the CollabUIAgents framework focuses on adapting base models to new environments through curriculum-based single-agent training (Bengio et al., 2009). The training data is synthesized automatically with a multi-agent data synthesis pipeline and consists of progressively complex instruction sets in three levels, designed to help agents build a strong foundation of environmental knowledge. The UI agent generates responses to synthesize queries faithfully, the adversarial agent generates negative samples, and the critic agent grades process rewards.
Curriculum Structure
The training data is divided into three categories, collected in
Figure 4:
(1) Basic Environmental Knowledge:
This data segment includes identifying UI elements and understanding their properties. We categorize basic knowledge into two types: Ul Understanding (coarse-grained): This refers to broad understanding of the layout and information contained in the UI, such as identifying the purpose of the interface. UI Element Recognition dine-grained) Since UI typically contains large number of densely packed interface, the agent needs to be able to distinguish between different types of elements, such as buttons, input fields, and drop-down menus, and understand the associated actions. We develop series of queries accordingly in Appendix 1.4.1, and randomly select UI elements and the layout to assemble queries for the UI Agent.
(2) Simple Instruction Knowledge: The agents are tasked with performing basic interactions, such as clicking or typing, in response to simple instruc- tions. Specifically, given the complete action actiensppac, space, we prompt the UI agent to generate possible instructions related to a random UT element, and their cor- responding responses. For example, in Figure 4, the UlAgent was prompted to generate an instruction for element 9 ("selecting the M4a format") and then generates the corresponding response to interact with it. By learning this type of knowledge, the agent lays the foundation for completing a complex sequential mecision-making process.
Basic Environmental Knowledge
Question: What is the purpose the curent
Sining breel lock
Response: This interfice for setting the recording fontar...
UI understanding
Question: What action is sesociated with element 97
Response: Element is clickable and allows the user to change the
Element Recognition
Instruction Knowledge
Instruction: select M4a fonuee.
Response: recunding correctly....
Promp? L.M
Process Preference Knowledge
Ageat,
Taslc Recood an audio clip using Audio Recorder upp and save
Action history: Siep1: Open Audio Recerder app, Stop2.
Current UI description: Elementi: Element2: Element3:
(3) Process Preference Knowledge: Real-world interactive tasks is quite difficult, and even the most advanced large language model, GPT-4, shows a low task completion rate (30%) in the mobile environment Android- World (Rawles al., 2025). Training a model solely on scarce successful tra- jectories still inevitably results in er- rors. Therefore, as illustrated below Figure 4, we introduce the adversar- ial agent against the UI agent, and the critic agent to score all actions, obtaining process preference data with step-level rewards. By learning from process preference data, the agent can better distinguish between correct and incorrect actions during the process, ultimately improving task completion rates. The distribution of the collected data can be found in Appendix D.4.2.
Respunse: I need click "Apply" ... Action: ("tlick,, "index": 3)
Adversartal Response: I should select" Way "to ensure hight
quality Action: ("elick", "index":
Adversarial Agent
Figure 4: Our multi-agent autonomous data syn- thesis pipeline. Given a task, the pipeline can au- tonomously collect data from each step covering basic environmental knowledge, simple instruc- tion knowledge, and process preference knowl- edge in interactive environments.
21

---

Published as a conference paper at COLM 2025
The base model is first trained using supervised fine-tuning (SFT) on the basic environmental knowledge and the simple instruction knowledge, progressively. The learning objective is:
SFF = - (sa))) [log T (als))
where D represents the dataset of state-action pairs. Following SFT, the base model are further optimized using direct preference optimization (DPO) on the process preference knowledge:
DDO Essat )~P logo Blog
(a [s) Tet (ee (a -Blog
AT(aa((s Nee (aer(a+(s)) )
(10)
where P is the preference-labeled dataset, a a denote positive and adversarial actions, the sigmoid function, & is the hyper-parameter, and Te, Tref are the base model and reference model.
C.2 Ablation Study of Agentic Fine-tuning
In this stage, we develop an automated data synthesis method to gather basic environmental knowledge, simple instruction knowledge, and process preference knowledge from the dynamic mobile environment, Worl..
Method
SR AndroidWorld
SR
Qwen2
6.2
12.9
Basic knowledge SFT
12.1
22.5
Instruction SFT
15.1
35.8
Process DPO
18.9
48.4
Table 5: Ablation study of the agentic fine-tuning process mobile operation environ- ments. Success Rates (SR) in the AndroidWorld and MobileMiniWoB+++ (MMiniWoB+++) environments are reported. All methods are based on Qwen2 7B.
Based on the upper section of Table 5, we derive the following conclusions: (1) Incorporating basic environmental knowledge data substantially improves the base model's comprehen- sion of dynamic mobile environments, achieving a absolute performance gain of 5.9% in AndroidWorld and 9.6% in MobileMiniWWoB++ ("+ Basic knowledge SFT"). ). It is noteworthy that the collected UT page information excludes app-specific details of MobileMiniWooB+t yet training with general knowledge from AndroidWorld enables the model to generalize effectively to new apps and tasks. (2) Simple instruction knowledge data is crafted to guide the agent in interacting with the environment using actions from the specified action space. Our experiments demonstrate that incorporating instruction data further enhances the base model's s ability to complete simple tasks within UI environments ("+ Instruction SFT"). (3) A key advantage of our proposed method is its ability to learn from incorrect actions using process preference knowledge data. Experimental results confirm that this addition significantly boosts performance ("+ Process DPO"). The improvement more pronounced in the MobileMiniWWo++ environment, which we attribute to the simplicity of its tasks. Fewer steps are required to complete these tasks, leading to greater performance gains.
D Experiment Details
D.1 Action Space in the Environments
Table and show the action spaces of agents in mobile and web environments, respectively,
D.2 Detailed Results in Web Browsing Environments
Detailed experimental results in web browsing environments are shown in Table 8 and Table 9, corresponding to the results in Table 2.
22

---

Published as a conference paper at COLM 2025
Action
Description
CLICK
Tap once on the element
DOUBLE.TAP
Quickly tap the element twice
SCROLL
Slide the screen to view more content
SWIPE
Quick swipe across the screen
INPUT TEXT
Type text into the element
NAVIGATE. HOME
Return to the home screen
NAVIGATE BACK
Go back to the previous screen
KEYBOARD ENTER
Press the enter key
OPEN APP
Launch an app
STATUS
Check task status
WAIT
Pause briefly
LONG_PRESS
Tap and hold on the element
ANSWER
Give response Undefined action
UNKNOWN
Table 6: The action space in mobile environment.
Action
Description
CLICK HOVER
Click at an element
Hover
element
SELECT
Select option in an element
TYPE STRING
Type to an element
SCROLL PAGE
Scroll up or down of the page
GO JUMP_TO SWITCH.TAD JNEUTI
Go forward or backward of the page
Jump to URL Switch to i-th tab
Notify user to interact Stop with answer
FINISH
Table 7: The action space in web environment.
D.3 Method Implementation
D.3.1 Baselines
(1) M3A (Rawles et al., 2023) is prompt-based multimodal agent, which combines ReAct- (Yao et al., 2023) and Reflexion-style (Shinn et al., 2023) prompting to interpret user in- structions and screen content, then update its decisions. (2) SeeAct (Zheng al., 2024) is prompt-based navigation agent originally designed for GPT-4V to perform actions with visual input and textual choices. (3) InfiGUiAgent (Liu et al., 2025a) used with the temper- ature set to 0.1, and other settings remain default. (4) DigiRL (Bai et al., 2024) is reproduced (Wang et al.,
on the same dataset as CollabülAgentss based on 2024d). Due to limited computational resource and the high costs of RL training with multi- ple value estimators, a larger scaled base model could not be applied. (5) SeeClick (Cheng et al., 2024b) is a fine-tuned visual GUI agent that automates tasks relying on screenshots and employs GUI grounding. For those baselines based on closed-source LLMs, we conduct experiments using prompts and settings consistent with Rawles et al. (2025), with model data up to August 2024.
D.3.2 CollabllAAeents Framework
In the agentic fine-tuning process for data collection, we employ GPT-40-2024408-00 (OpenAI, 2024) as the UI agent. We set the temperature of different agents to 0.1 to ensure the accuracy of their responses. For the backbone model in the policy multi-agent system, we utilize Qwen2 7B (Yang et al., 2024). Detailed process is depicted in Appendix D.4. In this phase, the model undergoes both supervised learning and off-line preference learning with DPO. Details of the experimental data statistics can be found in Appendix D.4.2. During the supervised fine-tuning phase, the model's context length is set to 8,192, with a learning rate
23

---

Published as a conference paper at COLM 2025
System
#Params/#Agents
Input
Cross- Task Cross-Webste Cross-Domain Avg.
Agents based on
LLMs
GPT-3.5-Turbo
N/A
Text
17.4
16.2
18.6
17.4
GPT-
N/A
Text
36.2
30.1
26.4
30.9
Agents hased on SpennSSurce M.MMs
Qwen-VL*
9.6B/1A 1A
Multimodal
12.6
8.0
10.2
SeeClick*
9.68/1A
Multimodal
23.7
18.8
20.2
20.9
Qwen2
7B/1A
Text
8.6
6.3
7.5
7.4
Our Methods
Base UlAgent
7B/1A
Text
13.4
10.6
11.8
11.9
Base UlAgent
7B/4A
Text
15.7
12.9
13.2
CollabUTAgentsm.aae
7B/4A
Text
19.2
13.8
15.5
16.2
CollabUTAgentS
7B/4A
Text
34.5
32.7
25.1
30.7
Table 8: Step Success Rates (SSR) in the Mind2Web environment. iidicates the system fine-tunes its base model on the corresponding training set of the environment.
English
Chinese
System
#PParams#tAgents
Avg.
Cross-Tast Cross-Domain
Cross-Task Cross-Domain
Agents based Closed-Source LLMs
GPT-3.5-Turbo
N/A
12.1
6.4
13.5
10.8
10.7
GPT-4
N/A A
38.6
39.7
36.7
36.3
37.8
Claude2
N/A
13.2
8.1
13.0
7.9
10.5
Agents based on Open-Souree LLMs
LLaMA2
7B/1A
3.3
2.5
LLaMA2
70B/1A
8.3
10.6
Qwen2
7B/1A AA
8.6
8.1
8.5
Our Methods
Base UlAgent
7B/1A
12.0
13.3
12.7
13.4
12.8
Base UlAgent
7B/4A
13.7
14.5
15.0
13.9
14.0
CollabUlA CollabUIAgenttmmdbiiie
7B/4A
18.6
17.7
19.1
15.6
17.7
CollabUlAgeetts
7B/4A
34.3
36.9
35.3
32.5
34.7
Table 9: Step Success Rates (SSR) of agent systems on different LLMs in the AutoWebBench environment.
of 1e-4 and training conducted over 3 epochs. During the preference optimization process, we adapt LoRA fine-tuning (Hu et al., 2022), and the model's 's context length is capped at 8,500, with a learning rate of 5e-6. We use bf16 precision, and the training is also carried out for 3 epochs.
In the multi-agent reinforcement learning process, the critic agent is played by GPT-4o- 2024-08-06, with its temperature also set to 0.1, providing reward feedback to the rolled out actions, the MARL phase, the rolling out temperatures of different UI agents are set to 0.1, 0.3, 0.5, and 0.8, respectively. This variation encourages diverse responses and stimulates different behaviors. We use iterative-DPO (Xiong et al., 2024) as the online reinforcement algorithm, and the multi-agent system is updated for each epoch. We train the multi-agent system within the AndroidWorld environment, where the environment seed is randomly generated. Continue MARL on Mind2Web uses instruction datasets autonomously collected with the pipeline in Appendix C. During testing, we maintain consistency with previous work (Rawles al., 2025) by setting the seed to 30. All our experiments are conducted on 8 A100 80GB GPUs. The model's context length th is capped at 8,500, with learning rate of 5e-6. We use bf16 precision, and the training carried out for 3 epochs where the off-policy rolling out is conducted at the beginning of each epoch.
D.4 Data Collection for Agentic Fine-Tuning
We collect data from the AndroidWorld environment for agentic fine-tuning the base model into the base UlAgent. In this section, provide details of the data collection process, including the questions list, the quantity of the collected data, and the prompts for data collection.
24

---

Published as a conference paper at COLM 2025
D.4.1 Questions
The questions used for UI basic environmental knowledge generation are shown in Table 10.
Type
Question
What the purpose of the current UI?
2. What does the current
UI Understanding
Ul aim to achieve?
3. Summarize the cur-
rent interface one para- graph.
What is the function of UI element X?
2. What information does
UI element X provide?
Element Recognition
What happens when click the Ul element X?
4. What action is associ-
ated with UT element X?
Table 10: Questions for basic environmental knowledge generation.
D.4.2 Statistics of the Collected Data
The quantity of the collected data is shown Table 11.
Data Type
Number
Basic Environmental Data
88,513
Simple Instruction Data
18,041
Process Preference Data
3,440
Table 11: Quantity of the collected data.
D.4.3 Prompts
Prompts for data collection process in the agentic fine-tuning are shown in Figure 5, 6, and 8.
D.5 Prompts for Different Agents in the CR
Prompts for different agents in the CR are shown in Figure 8, and 10.
D.6 Prompts for CollabUlAgents Framework
Prompts for the multi-agent system trained in CollabUIAgents on mobile operation environ- ments and web browsing environments shown in Figure 11 and Figure 12, respectively, within the ReAct (Yao et al., 2023) style.
25

---

Published as a conference paper at COLM 2025
You are an agent who can operate an A Anrrid phone on behalf user.
Here is a list of descriptions for some UI elements on the current screem:
{ui elements description]
Please answer the following questions for all the UI elements above.
Questions = ('
"W hat is the purpose of the current UI?'
'Summarize the current interface in one paragraph.'
'W hat does the current. UI aim to achieve?
Please format, your response as follows:
"( [["Question": "What the purpose of the current UI?", "Answer": "....." "}3' "{"Question": "Summarize the current interface in one paragraph.", "Answwer:"..... "}} '( (""Question": "What does the current UI aim to achieve?", "Answer"."......"""
Your response:
Figure 5: The UI understanding prompt template.
You are an agent who can operate an A Android phone on behalf a user.
Here is a list of descriptions for some UI elements on the current screen:
{ui elements description]
Please answer the following questions for all the UI elements above.
Questions = (
'What is the function UI element
'Wha hhtt information does UI element X provide ?'
'W hat happens when click the UI element
'What action is associated with UI element X
Please format your response as follows:
"{""uestion"": "W hat is the function of UI element X?", "Answrr" ........II "H
"[["Question": "W hat information does UI element X provide?", "Answer"."...... "}}
"{""Qeestoon": "W hat happens when click the UI element X?", "A "Answer" "".....""""" "[""Questoon"; "W hat action is associated with UI element X?". "A "swerr".. ........""!"
Your response:
Figure 6: The element recognition prompt template.
26

---

Published as a conference paper at COLM 2025
You agent who operate an A inroodd phone behalf of user.
Here is a list of descriptions for some UI elements on the current screen:
{ui elements deseriptiociii
The action space the agent: (action space}
General guidance: {general gutrrncee
Please propose diverse simple instructions (one-step tasks) as many as possible based on the agentl's action space and the current UI elements above in the following format: (contains at least one but more than two \'completel' actions and more than one \'answer'' action)'
'( ("Instruction": "......, "Response": "Reason: Action: {1 ("action type":...3)'""...
For example:
(""Instruction"; "I need to start recording audio", "Response": "Reason: The recording settings are all configured, I need to click ppypll to apply the current settings and start recording. Action: {{ <(actioon type": "click", :
"("Instruction": "I want to select the M4a format for recording.", ". "Response": "Reason: The recording format has been set correctly. Action: {("action type": "status", "goal status": "complete")) "}3"
'Y our response:
Figure 7: The instruction knowledge prompt template.
27

---

Published as a conference paper at COLM 2025
The current user goal/request. is: {goal}
Here is a history of what you have done so far: {history)
Here a list of descriptions for some UI elements on the current screen:
{ui elements description}
General guidance: (general guuddncce
Now you need to role- play very clumsy agent that can only output incorrect answer (if you have no choice, you can make up wrong action and reason) from the above list in the JSNN format, following the reason why you do that.
Your answer should look like:
'Reason: ...tioio:: { 3
Your answer:
Figure 8: The prompt template for the adversarial agent.
28

---

Published as a conference paper at COLM 2025
You are super- innelggittelligeett agent who can expertly operate an Android phone on behalf of a user
Now, you need to act as a critic, evaluating the actions taken by other A Android agents. These agents receive user tasks and current A ndroid interface information and then take the next step.
Your evaluation should be between [0,1]. score close 0 means the agent's 's action is useless or incorrect in achieving the user's task, a score close to 0.5 means you are uncertain
whether the agent's decision useful for achieving the user's task, and a score close to 1 means the agent's action is useful correct in achieving the user's task.
The current user goal/request is: {goal}
Here is a history of what have done so far: (history)
Here a list of descriptions for some UI elements on the current screen:
{ui elements description]
General guidance: (general guidance)
Here the next actions different agents would like to take: (agents actions)
Please output each agent's score in the correct JSON format, following the reason why you think the agents' actions and reasons are correct or not, and ensuring that their actions are necessary and not redundant for achieving the user's goals when you give your scores.
Figure 9: The prompt template for the critic agent.
29

---

Published as a conference paper at COLM 2025
Your answer should look like:
'Reason: The goal is {{user ).. Score: {{ {("agent id": score...
Here are some demonstrations of evaluations:
1. Reason: The goal is A gent and A gent 2 attempt to scroll down find additional options. This is a logical step given that no explicit save button is visible and the app might
have additional options accessible through scrolling. Agent decides to click the "Settings" button in hopes that it might lead to menu with a save option. However, this seems less directly connected to saving the recording as the Settings menu is generally for configuration rather than saving recordings. Score: ("aagent 0": 0.9, "agent_ 1": 0.1, "agent_22 2": 0.9) 2. Reason: The goal is ... All agents (Agent 0, 1, and 2) have chosen to input the desired name "xxx.m4a" into the text field, However, user did not specify a name. This is the incorrect next:
step, .corc: {!"agent 0": 0.2, "agent, 1": 0.2, "agent, 2": 0.2)
3. Reason: The goal is Historical information shows that the agent has taken the same action multiple times. I am unsure if taking the same action again is reasonable. Score: ("agent 0": 0.5, "agent 1": 0.5, "agent 2": 0.5)).
Now output each agent's score.
Your answer must in the format:
'Reason: The goal is {(user ) S.o.core: (("agent id": score, "agent id": score, "agent_in id": score)]
Your Evaluation:
Figure 10: The few-shot prompt template for the critic agent.
30

---

Published as a conference paper at COLM 2025
The current user goal/request is: {goal}
Here is a history of what you have done far: (history)
Here a list of descriptions for some UI elements on the current scroen:
{ui elements descriptiis))
General Guidance: {general guidance)
Now output an action from the above list in the correct.SONN format following the reason
why you do that. Your answer should look like:
'Reason: ....Actio: (("action (""action_type":...] type":...))
Your answer:
Figure 11: The prompt template for mobile operation.
<html [html content)
</html
You are a helpful assistant that can assist with web navigation tasks. You are given a simplified html webpage and task description. Your goal is to complete the task. You use the provided functions below to interact with the current webpage.
#Provided functions: (action_space)
#Previous commands: {previous commands}
#Window tabs: (exist window tabs with pointer to current, tab)
#Current viewport (pages): (current_ositiinn) / (max _ize)
#Task: (task description)
You should output one command to interact to the current webpage. You should add a brief comment to your command to explain your reasoning and thinking process.
Figure 12: The prompt template for web browsing.
31


Reviews:
[{"review_id": "fiNrvr8cf7", "reviewer": "Reviewer_9rgY", "summary": "This paper explores learning in multi-agent systems using small language models, aiming to achieve strong generalization performance without relying on predefined roles for each agent. Specifically, the authors apply MARL, where each agent has its own policy, and introduce a critic agent that assigns rewards and updates value estimates based on the rollout of actions. The experiments test whether this approach can enable UI dialogue in diverse environments, and the authors report that their 7B-model achieves performance comparable to much larger open models.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work presents a highly interesting attempt to improve a known weakness in multi-agent systems, the difficulty of generalizing beyond the specialized behavior of individual agents.\n\nThe experimental comparisons seem reasonable when the paper is read as a proposal for applying MARL to language-based agents. The experiments themselves are comprehensive.", "reasons_to_reject": "There is room to improve clarity in how the core ideas of the proposed method relate to the specific problems being addressed. Methodologically, the work could be interpreted as an extension of reward/credit assignment approaches in multi-agent learning (e.g., Qu et al., 2025; Lin et al., 2025). However, as a whole, the paper proposes a solution tailored explicitly to applying MARL to language agents. I suggest providing a more transparent and precise explanation of the scope of the paper’s contribution in this regard.\n\nTables 1 and 2 are complex and challenging to interpret, as they attempt to compare a wide range of experimental conditions. While space limitations are understandable, I recommend streamlining the tables to focus more clearly on key comparisons relevant to the central points of discussion.", "questions_to_authors": "I found the performance gain in the CollabUIAgents setting (mobile to m→web) particularly striking regarding the experimental results. While this is presumably an intended outcome, it would be helpful if the authors could provide more analysis or discussion of which types of cases benefited most from the proposed approach.\n\nThe case studies in the Appendix are informative. However, please indicate which model/configuration each case corresponds to. It would also strengthen the paper to include case studies demonstrating generalization, if such results are claimed.", "extra_scores": {}}, {"review_id": "FT48V5Ee9I", "reviewer": "Reviewer_wGPK", "summary": "The paper presents a MARL (multi-agent RL) approach for an environment with sparse rewards and where an LLM is able to provide critic feedback. Empirical results show that the multi-agent system does well compared to strong single agent systems and has an ability to generalise.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed system clearly improves upon the single agent systems, as demonstrated by results on public benchmarks reported in tables 1 and 2.\nThe ability to introduce new LLM agents is a good property. \nThe structure of the paper is good, and the takeaways outline clearly the claims of the paper nicely.", "reasons_to_reject": "It's not clear (or I have missed) important details about what LLM is used for the reward model. \n\nThere is no comparison against multi-agent systems. The proposed MARL system is only compared against strong but single agent systems. I don't know the MARL literature well, but I believe a comparison of multi-v-single is misleading, and a multi-v-multi comparison would help demonstrate whether this proposed multi agent approach is an incremental improvement. \n\nThere's a lack of detail on how the environment provides the binary feedback. Says \"based on static rules\" on line 127. Presumably this is information available in the public benchmarks themselves? It would be helpful to note this if so, and/or say a little more about what success means in the environment and how this is measured.", "questions_to_authors": "Were the number of agents experimented with? It would be interesting to see results on different numbers of agents, rather than just the n=4 and m=3 setting. How robust is this? Why was this specific setting chosen?", "extra_scores": {}}, {"review_id": "8GXfHfGkTn", "reviewer": "Reviewer_sVoJ", "summary": "This paper employs multi-language-agents systems to solve problems in mobile and web environments. To mitigate the sparse reward issue, the paper designs an LLM-based credit re-assignment technique. Experiments show that this framework achieves comparable results with advanced closed-source large models like GPT-4 based on the small model Qwen2.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The application of MARL to mobile/web automation is relatively under-explored and presents practical value.\n2. The proposed credit re-assignment technique could potentially benefit other cooperative multi-agent scenarios if properly validated.", "reasons_to_reject": "1. **Questionable Multi-Agent Contribution**:\n    - The \"multi-agent\" framework appears functionally equivalent to a single-agent system with role-specific prompts and voting. The agents lack true coordination.\n    - The choice of agent roles/numbers relies heavily on human priors, with no justification or ablation studies.\n2. **Methodological Issues**:\n    - The so-called \"adversary\" does not engage in meaningful opposition; it merely generates inferior actions as negative samples. This may confuse with robust RL and MARL literature [1,2,3,4].\n    - The core learning part (Eq (8)) is not clearly explained, with multiple symbols undefined (Although I found some of them in Appendix B.1).\n    - Edge updates are described as \"random\" without theoretical or empirical justification, raising doubts about its role.\n3. **Clarity and Rigor**:\n    - Key components (e.g., Eq. 8, \"conversation rounds\") are ambiguously defined.\n    - Section 3.3 is superficial, stating obvious facts about fine-tuning without linking to the proposed method.\n    - Typographical errors (e.g., line 319, faces → face) undermine professionalism.\n4. **Lack of Validation**:\n    - The method’s generality is claimed but untested. No experiments on simpler tasks (e.g., grid-worlds) validate its core mechanisms before complex mobile/web deployments.\n\nReferences\n\n[1] Pinto et al. Robust adversarial reinforcement learning. 2018.\n\n[2] Bukharin et al. Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms. 2023.\n\n[3] Yuan et al. Robust Multi-agent Coordination via Evolutionary Generation of Auxiliary Adversarial Attackers. 2023.\n\n[4] Lee et al. Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning. 2025.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "29goZciG7O", "reviewer": "Reviewer_Mpaq", "summary": "The paper presents CollabUIAgents, a multi-agent reinforcement learning algorithm which aims to better train agents in collaborative environments. They achieve this through 3 main techniques, mainly credit reassignment, which uses an LLM to provide fine-grained rewards for steps in collaboration rather than a simple 1 or 0 final reward. In addition, they use preference optimization with adversarial outputs, and provide the \"edge update\" trick.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper pushes in an extremely important direction and shows a solid proof of concept for many good ideas. They show very strong performance with a small model. Their training setup seems solid, and overall I do believe in the core concept being presented.", "reasons_to_reject": "Reaching gpt-4 performance is impressive, but honestly a bit suspicious considering that 4o is in fact also the critic model. I'm not sure if these are slightly different GPT versions, but it suggests to me that this may essentially be distilling the critic into the agents. This isn't necessarily a bad thing, but it significantly changes the framing of the method and raises other questions (can we achieve the same performance through directly training on gpt-4 outputs?). I'm also concerned that this detail isn't mentioned until appendix section c.3.2. This should definitely be mentioned in the main text.\n\nI'm also not convinced by the novelty in the method. Training with AI feedback is well established, so the main novelty is applying this to multi-agent environments. However, there are not enough ablations for this - what happens as we vary the critic? Can we use the model to critique itself? Can we get better results from pure distillation if we have access to a critic like gpt-4o?", "questions_to_authors": "- Do you think that a policy trained on critic written rewards can exceed the capabilities of the critic? If so, can you provide experimental evidence or use a wider array of critics? \n- How does this compare to directly distilling gpt-4o?\n- How do you know if a prompt is good for the llm critic? Do you experiment with different critic prompts?\n- Can you add one or two more training runs which vary the base model to show that other base policies see similar gain or how they end up landing vs. Gpt4? Especially larger models?\n- Do you have a sense of noise in performance? I'm especially curious about table 3, where I feel some differences may be within noise range. \n- Could you explain the \"edge update\" trick in more detail and with more ablations? I don't fully understand what it entails from what is written or figure 2. The paper mentions that edges are \"randomly\" updated - how much does this matter, is probabilily < 1  optimal or is it for efficiency?\n- Can you provide sample outputs from the learned policies and the corresponding critic outputs? I want to see how the agents evolve - what they learn, and especially if they differ from gpt-4.", "extra_scores": {}}]

Author rebuttal:

Meta-review: While multi-agent systems outperform single agents in interactive tasks (e.g., web browsing), their generalization across environments remains limited by rigid role definitions. This paper proposes CollabUIAgents, a reinforcement learning framework that uses LLM-guided credit re-assignment (CR) and synthetic preference data to develop role-free, generalizable policies, and achieves performance comparable to closed-source 7B models. The reviewers provided valuable feedback, which the authors have made great efforts to address. One reviewer maintained their original low score without providing specific justification following the rebuttal, so I have discounted this evaluation in my assessment. Given the authors' rebuttal and other reviewers' supportive scores, I recommend acceptance.
Decision: Accept