Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark
Authors: Bingchen Miao
Yang Wu
Minghe Gao
Qifan Yu
Wendong Bu
Wenqiao Zhang
Yunfei Li
Siliang Tang
Tat-Seng Chua
Juncheng Li
Venue: ICML 2025 poster (2025)
Keywords: Virtual Agent; Digital Agent; Reward Model
Primary area: applications->computer_vision
TL;DR: We propose Similar, a step-wise, multi-dimensional generalist reward model, to improve Virtual Agents by providing process-focused multi-faceted assessment training and inference-time scaling signals, supported by a novel benchmark called SRM.

Abstract: The development of Generalist Virtual Agents (GVAs) has shown significant promise in autonomous task execution. However, current training paradigms face critical limitations, including reliance on outcome supervision and labor-intensive human annotations. To address these challenges, we propose **Similar**, a **s**tep-w**i**se **m**ult**i**-dimensiona**l** gener**a**list **r**eward model, which offers fine-grained signals for agent training and can choose better actions for inference-time scaling. Specifically, we begin by systematically defining five dimensions for evaluating agent actions. Building on this framework, we design an MCTS-P algorithm to automatically collect and annotate step-wise, five-dimensional agent execution data. Using this data, we train **Similar** with our crafted Triple-M strategy. Furthermore, we introduce the first benchmark in the virtual agent domain for step-wise, multi-dimensional reward model training and evaluation, named ***SRM***. This benchmark consists of two components: ***SRMTrain***, which serves as the training set for **Similar**, and ***SRMEval***, a manually selected test set for evaluating the reward model. Experimental results demonstrate that **Similar**, through its step-wise, multi-dimensional assessment and synergistic gain, provides GVAs with effective intermediate signals during both training and inference-time scaling. The code is available at [https://github.com/antgroup/Similar](https://github.com/antgroup/Similar).

Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark
Bingchen Miao 12
Yang Wu
Minghe Gao
Qifan Yu
Wendong Bu 12 Juncheng Li 1.0
Wenqiao Zhang'
Yunfei Li
Siliang Tang' Tat-Seng Chua
Abstract
(MLLMs (Li ct al. 2023a b Pan et al.
2025 2024b)) pro-
cess multimodal inputs (UI elements (Zhang et al. 2024a), text (Shen al. 2023), visuals Yaan et 2023)) to navigate digital enviromments, performing tasks and generating out- puts that manipulate interfaces provide responses. The training of GVAs relies on outcome-based rewards from human- annotated trajectories, where task completion serves as the primary supervision signal (He et al. 2024).
The development of Generalist Virtual Agents (GVAs) has shown significant promise in au- tonomous task execution. However, current training paradigms face critical limitations, in- cluding reliance on outcome supervision and labor-intensive human annotations. To address these challenges, we propose Similar, a step- wise qulii-dimmesiooal generalist reward model, which offers fine- fine-grained signals for agent train- ing and can choose better actions for inference- time scaling. Specifically, we begin by systemati- cally defining five dimensions for evaluating agent actions. Building on this framework, we design an MCTS-P algorithm automatically collect and annotate step-wise, five-dimensional agent exe- cution data. Using this data, train Simil aar with our crafted Triple-M strategy. Furthermore, we introduce the first benchmark in the virtual agent domain for step-wise, multi-dimensional reward model training and evaluation, named SRM. This benchmark consists of two compo- nents: SRMTrain, which serves as the training set for Similar, and SRMEval, a manually se- lected test set for evaluating the reward model. Experimental results demonstrate that Simi aar, through its step-wise, multi-dimensional assess- ment and synergistic gain, provides GVAs with effective intermediate signals during both training and inference-time scaling. The code is available in https://gitaub.commmagrroo/////p
However, this paradigm with the outcome reward for GVAs has significant limitations. 1) Lack of -unti-ddimnnioonll fine-grained process supervision: Existing methods typ- ically focus on global task success or the final state of the task, overlooking intermediate steps in execution Yuu et al., 2024a). This oversight makes it impossible to pinpoint failures in unsuccessful trajectories or identify errors in successful ones, resulting in inefficient learning and reason- ing processes (Ucsato et al., 2022 Lightman et al., 2023 Gao al. 2025). In contrast, Process Reward Model (PRM) offers better alternative by providing fine-grained supervision signals to guide agent behavior. 2) Reliance on human-annotated trajectories with reward signals: Domain experts need to meticulously annotate trajectories consisting of dozens of steps with accurate outcome-basee rewards to train GVAs (He et al. 2024). Furthermore, ob- taining step-wise fine- geeigrained process- based rewards makes the process labor-intensive, consucnn,,,,n, and nearly in- feasible at scale (Deng al. 2023 Burns al. 2022 3) Difficulty in scaling inference-tine time. Recent outstanding work has demonstrated that inference-time scaling can sig- nificantly enhance agent performance (DeepSeek-Al et al.
process-based
2025 Wu et al. 2024b). However, relying on result-based training with extensive human annotation limits the abil- ity to handle complex tasks by selecting the best action step-by-step assessments. (Snell et al.) 2024 Zelikman et al. 2022). Therefore, our focus has shifted to break- ing result-oriented manual immotation-deeenden training methods through step-wise automatic reward model.
1. Introduction
Generalist Virtual Agents (GVAs (Gao et al. 2024 Bu et al. powered by Multimodal Large Language Models
2025)) )
'Equal contribution 'Zhejiang University, Hangzhou, China AAnt Group. Hangzhou, China National University of Singa- pore, Kent Ridge, Singapore. Correspondence Juncheng Li <junchengli@@ zju.edu.cn>.
To address these challenges, we propose Similar. a step- wise qulti-dimensional generalist reward model. pro- vides fine- ine-grained supervision signals for agent training and inference-time scaling, enabling automated, multi-faceted assessment without relying labor-intensive human an-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Blep-Wise, Multi-Dimennional, and Generalist Reward Model with Benchmark for Virtual Agents
Traditional
VS.
Ours
Task
WORK
PB
AA
MD
PLATFORMS
SIzE
Hello! Please
Though the final result
WEBARENA
X
X
X
WEB
812
help me... us
is wrong, many steps
WERUALWERARENA
K
K
K
WER
910
10000
WORKARENA
K
X
X
WEB, WINDOWS
19.9K
Agent
are correct, we can
ANDROID WORLD
K
X
ANDROID
116
The final result wrong, think this ttrajectory is incorrect,
Trajectory
assess steps from five dimensions. Step 1:
WEYVGR
K
V
K
WEB
300
OSWORLD
X
V
X
LINUX, WINDOWS
369
Step
Helpfulness is
GUI ODYSSEY
V
K
ANDROID
7.7K
MOBILE-AGENT
X
X
ANDROID
30
Step 2: vf
OMNIACT
X
WEB, WINDOWS
9.8K
... ...
Step X
SRM (OURS)
/
/
/ WINDOWS, ANDROID
WEB, LINUX,
110k
(a)
(b)
(a) Traditional coarse- gease--gaindd outcome- outcome-based labor-intensive paradigm vs. Our fine- fine-grained process- based autonomous paradigm. (b) Comparison of benchmark platforms. Previous works focused on virtual agent benchmarks, while ours is the first benchmark specifically for virtual agent reward models. The PB, AA, and MD in the list represent Process-based, Automatic Annotation, and Multi-Dimension, respectively.
Figure 1:
notations. Specifically, 1) we introduce step-wise, multi- dimensional assessment system for GVA actions, defining five key dimensions of process supervision signals: Help- fulness, Odds of Success, Efficiency, Task Relevance, and Coherence. These dimensions are designed to minimize overlap while collectively providing a comprehensive as- sessment of each action's quality. 2) Then we design an MCTS-P algorithm to automatically collect and annotate tens of thousands of step-wise actions based on the five dimensions. This approach is applied across four distinct environment domains: Web, Android, Linux, and Windows. Unlike existing methods that rely on labor-intensive human annotations, this automated framework ensures scalability across diverse environments and generates a unified, fine- grained dataset that captures universal reasoning patterns, significantly reducing the cost and time required for data collection. 3) Finally, using this dataset, we employ a Triple- M (multi-step, multi-objective, and multi-modal) strategy to train a reward model. This strategy integrates multi- ple dimensions of assessment and generates a synergistic gain by combining the strengths of five dimensions. As illustrated in (i) traditional methods focus solely on outcomes, require significant manual effort, and are coarsc-grained, nutcome-based. and labor-intensive. In con- trast, our approach enables Simi lar to perform step- wei-wise, multi-dimensional automatic assessment of agent trajecto- ries, making it -reigrdied,, process-based and autonomous. Building on this, Similar provides fine- -raiind rewards for GVA during the training phase, while seamlessly guid- ing GVA and robustly optimizing performance by scaling inference-time during the inference phase.
SRMTrain (78k) for training Similar and the curated SRMEval (32k) for evaluating reward models.
Our reward model, Similas can enhance the learning and reasoning of GVAs. For training, it serves as re- ward model in reinforcement learning framework, guiding GVAs to optimize its behavior based on action quality. By providing fine-grate mne-grained feedback, it effectively guides the agents' learning process and enhances their performance. For inference-time scaling, can be integrated with search algorithms such Monte Carlo Tree Search (MCTS) to leverage reward signals for filtering candidate actions, and improve model performance (DeepSeek-Al et al., 2025 Zang et al. 2025). By selecting actions that are more likely to complete the task, it enhances accuracy and reduces time.
Extensive experiments demonstrate the superiority of our approach: 1) Effectiveness of step-wise, multi-dimensional data: Using our collected data for reward modeling. S-milar-RL-Llama achieves a 13.2% improvement over the baseline (.uma-3.2-11B-Viae benchmark, demonstrating the effectiveness of our auto- mated framework in enabling fine-grained assessment of GVA actions. 2) Synergistic gain from the Triplc-M strat- egy: The Triple-M strategy integrates multiple dimen- sions by leveraging the strengths of five dimensions, en- abling Similar--TM-Llama to achieve an Avg score of 61.2 on SRMEval, significantly outperforming Similar- RL-Llama (53.9, 13.5% improvement). This highlights the synergistic gain of our training strategy. 3) Effective guidance in training and inference: Similar provides fine-grained, multi-dimensional feedback during training and integrates with search algorithms like MCTS to scale inference-time during inference to improve reasoning accu- racy. Its strong performance across multiple benchmarks underscores its versatility and practical applicability.
model on the SRMEval
Since reward models crucial for GVAs, and prior search has not focused on evaluating reward models, we propose SRM, the first benchmark in the GVA domain for step-wise, multi-dimensional reward model training and evaluation. [ illustrates that it consists of 110k au- tomatically annotated data points, divided into the scalable
Our contributions can be summarized as follows:
We define five dimensions for step-wise GVA assess-
2

---

A Step-Wiss, -isp,,i,, Multi-Dimennional, and Generalist Reward Model with Benchmark for Virtual Agents
ment and an MCTS-P algorithm collect fine-grained, cross-platform reward model data annotations.
annotation is expensive. To address this, methods like step- level Q-value (Zhai et al. 2024) and ReST-MCTS' (Zhang) et al. 2024b) have explored MCTS to automate data collec- tion, achieving significant gains. Building on these insights, our work introduces a step-wise, multi-dimensional sys- tem leveraging MCTS to collect fine- aeee-graind annotations, enabling robust reward model to guide GVAs.
Based on these data, we propose Triple-M strategy to train a reward model, called S similar, integrating multiple dimensions and generating synergistic gains for robust, fine- fine-grained feedback.
Moreover, we introduce SRMEval, a multi-step. multi- dimensional, and pllti-pl benchmark for evalu- ating reward models, which is a set of SRM to advance research in reward model performance assessment.
3. Method
In this section, present the pipeline for training our pro- posed Similar model. The SRM benchmark will be intro- duced in Section ! As shown in Figure 2 to evaluate agent steps multi-dimensionally. we first define five-dimension process supervision (Section 3.I). Next, we introduce an MCTS-P algorithm to automatically collect step-wise anno- tations (Section 3.2). Finally, we design the Triple-M strat- egy to train Similar, achieving synergistic gains across five dimensions (Section 3.3
Experiments demonstrate that our approach, through step-wise multi-dimensional assessment, provides GVAs with superior intermediate signals during both training and inference-time scaling.
2. Related Work
2.1. Fine-Tuning Virtual Agent
3.1. Five- Diveenimnsio Process Supervision Framework
Fine-tuning Virtual Agents traditionally relies on human- annotated datasets, which are labor-intensive and time- consuming (Wang et al., (023). Methods such as imitation learning (Humphreys et al. 2022) and reinforcement learn-
To assess the quality of an agent's steps, we systematically define diveeiimesiooal process supervision framework. Given task complexity and interdependencies, single met- ric is insufficient for assessing step quality (Zhai et al., 2024). Our framework addresses this limitation by cov- ering the multi- nature of step assessment. The first three dimensions HHeesflnsssllflllnssssss Odds of Success, and Ef- ficiency are computed automatically, while the remain- ing two-Task Relevance and Coherence-are assessed us- ing MLLMs. These dimensions are independent and inter- pretable, ensuring broad applicability across tasks.
ing (Branavan et al.] 2009 2010) have been employed to fine-tune agents based on curated expert trajectories or out- come rewards, but these approaches often suffer from com- pounding errors and limited exploration (Christianos et al.]
Xi et al. 2024 [Song et al. 2024). Recent advance- ments, such as reject sampling fine-tuning (RFT) YYaan ct al.
2023
2023) and direct policy optimization (DPO) (Rafailov et al. 2023), have sought to reduce reliance on human an- notations by leveraging both successful and failure trajecto- ries
The current step is denoted as S;, where i is the step index. The three automatic metrics are derived through MCTS simulations (Luo et al. 2024). For S,, we simu- late N subsequent trajectories until termination condition is met (i.e., the agent completes the task or reaches the maximum step length). We define the basic reward r; as:
(Lai et al. 2024| Zhang et al. 2024c). However, these methods face significant challenges, including the lack of process supervision and reliance on human-ana annottaed data, which limit their scalability and adaptability (Xu et al. 2021 He et al.) 2024 Rawles et al., 2024b)) In contrast, our work addresses these limitations by introducing novel training paradigm that leverages multi-dimensional process supervi- sion and automated annotation to enhance the learning and reasoning capabilities of GVAs.
1, 3a,, E A, aij = a* . N, where a" repre- ri - 0, otherwise
sents the ground truth, aij denotes the final action of the j-th trajectory in step i, and A is the set of all actions. The following sections detail how each dimension assesses St.
2.2. Reward Models for Virtual Agent
Helpfulness (H). It quantifies whether a given step con- tributes positively or negatively to task completion, assign- ing values inversely proportional to the trajectory length. This dimension is designed to assess the impact of each step on the overall task trajectory. Steps that facilitate task completion are considered helpful, while those that hinder progress are assigned negative values. For example, each step in 3-step successful trajectory is worth 1 1/3, while steps hindering progress (those failing to lead to success) re- ceive corresponding negative values. And in two successful trajectories of the same task, the steps in the trajectory with fewer steps will have higher Helpfulness value.
Reward Models (RMs) are critical for guiding virtual agents by evaluating action quality (Zhai et al. 2024 Zhou et al. 2024a). While Outcome Reward Models (ORMs) focus on task success (Yu et al. 2024a| 2023a), Process Reward Models (PRMs) provide feedback on intermediate steps, offering an evaluation of agent performance in complex reasoning tasks (Uesato al. 2022). Recent studies show that PRMs outperform ORMs in tasks like math reason- ing, where process supervision is essential (Lightman et al.) 2023; Li et al., 2023ch. However, generating high-quality process supervision data remains challenging, as human
3

---

A Step- Wisc, Multi-Dimennional, and Generalist Reward Model with Benchmark for Virtual Agents
Five-dimension Step-wise Assessment
MCTS-P Automatic Annotation
Triple-M Strategy RM Training
observation
5-dimension Score
MCTS-P
Prompt
Response
Google
Helpfuliness
Sn
Oolia of Surceess
0.32
Token Embedding
Efficlency
Tosk Hello! Please help me....
S insue dria as (H0.22
$1,2 Niqa as-a.s oxt
Last Hidden State
0,45
CLe CLa
Experts
$111 Hwo.2 osead po.4 tos Oil
$1.2 med 05-0. Ex0.54 70vt oo
S1. me.22 coed of Exd.25 roup Ceg
Expert H
Regression Loyer
Task Relevance
Buuan Lopee Ealan
Expert os
Coherence
@Expert
action View the match result,
5
$3, Ahdee
$33 H.0.s 05-8.41
Expert TR
Similar
Expert C
Automatic Derivation
MILM--based
Expert
Assessment
#SWord android world ...
Figure 2: Similas lar model training pipeline. First, we systematically define five dimensions to describe the quality of an agent's s step. Next, we propose an MCTS-P algorithm to automatically collect annotated step-wise data. Finally, we design the Triple-M strategy train the Similar model, which can guide the agent during both the training and inference phases.
by evaluating the proportion of successful paths among simulated results from given step.
instruction
steps
Start
TR=1
The formula for Odds of Success is defined as:
S Enter 'neakerr'' and search.
steps rest
C=1
N
Maiy aa)) a²)
oS, =
B
E-1/3
N
current step Sz: Select pair of sneakers, priced at $90.
1 step
where I(-) is the indicator function.
rest
Efficiency (E). It evaluates whether given step is opera- tionally efficient in terms of resource consumption, such as time or computational effort. A fundamental assumption is that fewer steps equate to higher efficiency, as shorter paths imply lower resource usage. Steps that reduce the total number of steps required to complete a task are considered efficient, as they enable the agent to accomplish the task more quickly and with fewer resources.
S3.1
S1.2
$3,3
OS~2/3
final state Symnat
H=1//3
Figure 3: An example describing the 5 dimensions. A single step by agent relates to 5 task clements: instruction, last step, next step, final state, and number of steps. Our 5 dimensions align with these: H (final state), OS (next step), E (number of steps), TR (instruction), and C (last step).
The Efficiency metric is calculated as the following formula:
L Len,
E;
The Helpfulness can be calculated as the following formula:
leno
1 - AC
and Len(St.) represents the number of steps remaining to complete the task after executing action aijj-
where Len, =
H M i+1
(2r: 1).
where AC; max(AC,-11 + 0), otherwise' which
0,
Task Relevance (TR). It assesses whether a step is related to the task instruction. Some steps may be telk-relevant but still fail (e.g., recording incorrect notes), while others may be irrelevant yet contribute to success (e.g., clicking on a blank screen). These distinctions cannot be captured through automated calculations. However, MLLMs with advanced image understanding can evaluate this dimension. Task Relevance is binary, with values {0, 1}.
a mathematical placeholder to recursively track cumula- tive Helpfulness scores during MCTS rollouts. And M is the total number of reasoning steps.
Odds of Success (OS). It measures the probability that given step will lead to the successful completion of the task. This dimension is crucial for identifying steps that are more likely to result in a successful outcome. Steps with higher values are more likely to lead to success, while those with lower values are less likely to succeed. Conversely, incorrect steps lead to failure. The Odds of Success calculated
Coherence (C). It: measures the continuity and logical flow between consecutive steps. Some operations, although task- relevant, efficient, and likely to lead to success, may lack coherence with the previous step. For example, task
4

---

A Slep-Wisc, Multi-Dimennional, and Generalist Reward Model with Benchmark for Virtual Agents
such as "Query the Lakers' game result and record it in a Note," opening a browser and a Note simultaneously may lack coherence compared to directly searching for the game result after opening the browser. Coherence is also evaluated using MILMS and is a binary classification dimension, with possible values of {0, 1}.
Algorithm 1 MCTS-P Algorithm
Input:
Initial state
1:
Create root node Sp with state so while within compatational budget do
3:
S: TreePolicy
4:
A - (efauttPolicy(((5)) # Simulate a random playout to catimate
RackuptS., A) BBckkpoopggtte the value update parent nodes
6: end while
return (BBesChiildSo, 0))
The prompts of two MLLM-based assessment dimensions To better understand the five
function TreePolicy(5)
detailed in Appendix A
10:
while is nonterminal do
dimensions, use Figure 5 Each agent step relates to five task elements: instruction, last step, next step, final state, and step count. In the figure, step S2 is assessed by these dimensions. The task requires 3 steps: Si, S2, and S3.2. with all sharing an H value of 3. Among the next steps from S2, S3.2 and S3.3 are correct, yielding an OS value of 3- The value is 3. calculated as 2-1, Since S2 aligns with S1 and the instruction, TR and C values are 1.
11:
if S not fully expanded then
12:
return Expand(S)
// Expand the tree by adding new child node
13:
else
14:
+ BeatChild C)
15:
end if
16:
end while
17:
return
18:
end function
19:
20:
function BestChiddss e)
21: v(S) = H(S) + OS(S) + E(S) TR(S) + C(S)
22: return arg max cf 8 R(3)) oV m(3")
(119')
21a
3.2. Automatic Generalist Dataset Collecting
23:
end fimetion
MCTS (Monte Carlo Tree Search) is a heuristic search algo- rithm used in preiision-mainng combining random sampling and tree-based search to find the optimal option. Based on its advantages, such as its scalability and efficient exploration- exploitation balance (Luo et al. 2024 Wang et al. (2024c), we propose a modified version, MCTS-P, to automatically collect annotated data. MCTS-P P leverages the five dimen- sions introduced in Section 3.1 to comprehensively assess each step taken by the virtual agent.
Output: Action
remaining paths using the evaluation methods provided by the four benchmark environments. The obtained trajectories are selected as the final dataset for training and evaluation.
3.3. Triple-M Strategy for RM Training
Traditional reward modeling tasks typically rely on human- annotated data (Wang et al.. 2024a), whereas our approach generates step-wise annotations across multiple dimensions. To address the challenge of integrating Multi-step, Multi- dimensional, and Multi-modal data, we propose novel Triple-M strategy tailored for virtual agents.
In MCTS-P, the five-dimensional scores are used the basis for node selection and backpropagation. Specifically, the algorithm computes weighted sum of the five dimensions to obtain a composite score for each step. This composite score serves as the value Vr, for each node Sr in the search tree. The tree structure in MCTS-P is similar to traditional MCTS, with each node Sky storing the action aigj visit count ni.j. and value va.- The detailed pseudo-code for MCTS-P is provided in Algorithm
Our Triple-M strategy leverages a pre-trained decoder-only MILLM as the backbone feature extractor Sa, divided into two stages. The first stage trains a regression layer for five-dimensional score prediction. For each input sequence x + (where x represents the prompt and y represents the response), we extract the last hidden state h E Rd with d-dimensional features and map it to a five-dimensional reward score through a linear regression layer W E R
To build a comprehensive and generalist dataset for train- ing and testing reward models, we collect a large number of task trajectories from agents across four different plat- forms: Web, Android, Linux, and Windows. Using the MCTS-P algorithm, we perform automatic data annotation to collect process supervision signals. The annotation pro- cess involves the following steps: 1) For each node Si.j in the search tree Tg, we calculate the minimum number of steps M required to reach a correct answer. 2) During the expansion phase, the algorithm simulates N possible outcomes for each step to obtain the basic reward r;. 3) Based on M and rı. we compute the three automatically calculated dimensions: Helpfulness, Odds of Success, and Efficiency. 4) We then use a MLLM (GPT-40 (Hurst et al.) 2024)) to evaluate the Task Relevance and Coherence of each step. 5) We prune all incomplete branches (those that do not reach a final answer) and verify the correctness of the
The model optimized using a regression loss:
LRG
min E EnR.ED | h - 2 6.W
where E R is the ground-truth reward vector, and 'D is the training dataset.
In the second stage, we train a gating network to dynam- ically balance the five-dimensional scores, addressing the multi-objective optimization problem. We introduce a prompt-aware gating network Jo. implemented multi-layer perceptron (MLP). This network dynamically adjusts the model's focus based on the input prompt x. The gating network computes non-negative coefficients w E R for the five reward dimensions. These coefficients are de- rived from the last hidden state corresponding to the prompt xr and normalized via a softmax function.
shallow
5

---

A Step- Wisp, Multi-Dimenniioal, and Generalist Reward Model with Benchmark for Virtual Agents
Step
Trajec-
Table I: Performance comparison of common MLLMs, Somilar-Rue and Similar- TM on SRMEval.
Instruction
Observation
Type
Candidate Action Pair
Idx
tory
In Simple Calendar Pro, delete all the events.
Input
REWARD MODEL
H
0S
E
TR
C
Tor
TRAI
Avo
Step1: Click
Click Calender.
"Simple Calendar
Search.
GPT-4-TURB
44.7
46.3
44.8
48.7
42.3
46.5
44.5
46.6
Pro".
GPT-40
49,9
50.1
47.9
51.4
43.8
51.1
49.8
51.4
INTERNVL-2.5
38.9
43.1
44.3
41.4
41.8
40.7
39.0
40.9
Figure 4: A data point of SRMEval.
QWEN2-VL
45.7
42.1
41.7
44.5
42.1
43.2
41.6
42.9
+ SIMILAR-RL
52.4
49.6
48.3
50.2
47.9
51.0
45.4
49.2
The gating network is trained using the Bradley-Terry (BT) loss (Bradley& & Terry] [952) function, which aligns the model's 's predictions with human preferences. The BT loss
SINILAR TM
60.5
57.8
56.6
59.7
56.2
58.4
53.9
57.3
LLAMA-3.2-V
48.2
47.6
47.1
51.1
42.6
49.5
44.5
47.6
+ SIMILAR RL
55.1
52.1
52.7
55.3
47.8
54.6
49.5
53.9
SINTLAR-TM
63.8
60.5
59.2
62.7
56.8
61.4
58.7
61.2
is formulated as:
observation screenshot, step index, trajectory, evaluation type, and candidate action pair. The evaluation types in- clude our proposed five key dimensions- Helpiuuusss (H), Odds of Success (OS), Efficiency (E), Task Relevance (TR), and Coherence (C)-as well as a total dimension that inte- grates the five dimensions (Tot, weighted sum of the five dimensions) and a trajectory-level dimension (Traj, average Tot score of all steps in trajectory). More visualization cases of SRMEval are detailed in Appendix
exp( Recwsen)
LBT min EE log exp( Reboson) + exp( Brejected)
where Rehosen and Reejected represent the preference scores for the chosen and rejected responses. During training, only the gating network parameters are updated, while the backbone network and regression layer remain frozen.
Finally, the scalarized reward R of the trained Simi lar model is computed as ( rr. Through this design, our Similar model can not only output five- dimensional scores but also a comprehensive score that balances the five dimensions, just like an all-dround expert combining the capabilities of six experts.
New Task and Evaluation Metric. Based on SRMEval, we proposed new task for reward models in the virtual agent domain: Selecting the better action from candidate action pair at step in a specific dimension. The evaluation metric is Accuracy, measuring the reward model's s ability to select the better action. Accuracy is calculated under each evaluation type. For clarity, we use abbreviations such as H to represent each metric in our experiments.
4. SRM Benchmark
We introduce the SRM benchmark, built from multi-modal, multi-dimensional, and multi-platform annotated data.
5. Experiments
Data Collection. We used GPT-4o- 1 120 (Hurst et al. 2024) as the agent to collect agent action trajectories across four
5.1. Experimental Setup
benchmarks Beeteaaark--ebbeerrareeee WebArena (WA) (Zhou al., 2024b), Vi- sualWebArena (VWA) (Koh et al. 2024), Android World (AW) (Rawles et al.] 2024a). and OS World (OW) (Xie et al. 2024). Since these environments do not provide dedicated training and test sets, to ensure fairness and prevent data leakage, we rigorously used 70% data provided by these benchmarks for agent trajectory data collection, while the remaining 30% were reserved for evaluation experiments to ensure no data overlap between SRMTrain and evalua- tion sets. Ultimately, we collected 10k agent trajectories by generating multiple distinct actions per step through task- specific prompt injection and stochastic exploration. And we constructed 110k preference pairs for the SRM bench- mark based on the scores of the designed dimensions. We also sampled some data for human experts to verify the accuracy of the pairs, as detailed in Appendix C
Baselines. We selected two baseline methods: 1) Qwen2- VL-7B- (Wang et al. 2024b) and Llama-3.2-11B. Vision-Instruct were directly used reward models, with prompts provided (detailed in in Appendix A to score agent steps. 2) Simi aarrll--r---eerrn and Similar-Ril -L-mmmma whose backbones match the aforementioned models, were trained using reinforcement learning (Arulkumaran et al., 2017) on our SRMTrain dataset score agent steps.
To benchmark against these baselines, we introduce Simi -ar--a-----eren and Similar-TMM-Lana. which are trained on the SRMTrain dataset using the Triple-M strategy with Sive2-VVL-BB-Innn Vision-Instruct as backbones, respectively.
and L.ama-3.2-11B.
Evaluation Benchmarks. We first tested the preference alignment capability of Similar on our SRMEval, com- pared with OPTT-4o-120. GPT-4-Turbo, and VL-2.5- 8B. Additionally, we evaluated our model's effectiveness as reward model for virtual agents during both the training and inference phases. 1) Training Phase. Using WebArena and Android World as benchmarks, we employed our model and other reward models to annotate GPT--4-colleetee data
Dataset Construction. We carefully selected 32k data points for manual annotation as the test set SRMEval, while the remaining 78k data points were used as the training set SRMTrain to train the Simi lar model. The test data tasks are distinct from those in the training data. As shown in Figure each data point SRMEval includes instruction,
6

---

A Step- Wisc, Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
Train
Inference
oction
0.8
oction
Enter "Los Angeles Clipers" search.
Open
H 0.56 0S 0.76
Open
Browser
c+1 Totol 0.75
opp.
=R=1 C=1 Total 0.17
Similar
Similar
oction
oction
= 0S 0.88 0
Click
0 =1=1
+R Total
Success
search
Training
the search
box.
Enter "Los Angeles Lakers" and search
step wise multi- dimension data
Figure case of Simi lar provides guidance GVA training and inference.
Table 2: Task Success Rates (SR) on Android World and WebArena in training setting.
Table 3: Task Success Rates (SR) on Android World and OSWorld in inference setting.
REWARD MODEL
ANDROID WORLD
WEBARENA SR
REWARD MODEL
ANDROID WORLD
OSWORLD
SR
SR
SR
GPT-4-TURMS GPT-40
24.1
11.2
GPT-4-TURBO
24.1
8.4
12.7
GPT-4-TURBO
QWEN2-VL
24.9
GPT-4-TURBO
S-SNILAR-RL
25.9
10.5
UGROUNDD
32.4
19.6
GPT-4-TURBO
+ SINIIAAR-TM
28.3
13.4
UGROUND
QWEN2-VL
4PR-O
11AMA-3.2.V
25.3
8.8
UGROOUND
SIMILAR RI.
33.1
26.5
GPT-4-TURBO
+ SINTNAR-RL RL
26.5
10.8
EGGROUND
SSMELLR-TM
GPT-4-TURDO
30.4
13.9
UGROUND
LLAMA-3.2-V 2-V
33.0
23.4
UGROUND
SIMELLARRR RL
33.8
29.6
GPT-40
25.4
10.8
UGROUND
SIMELAR TM
34.6
36.7
GP
QWEN2-VE VL
26.0
11.3
GPT-4n
+ SINTTAR--R
27.1
12.9
GPT-40
SINNILAR-MM
32.9
14.3
OS-ATLAS
QWEN2-VL
30.8
20.8
GPT-40
LLAMA-3,2-V
26.2
11.7
OS-ATLAS
SIMELAR-RL
32.1
25.9
GPT-40
+ SINITAARRRL
29.6
OS ATLAS
SIMELAR- TM
34.2
34.5
GPT-40
SINILAR--MM TM
34.6
16.5
0S ATLAS
LLAMA-332VV
31.3
22.4
+BIMELAR-RL
OS-ATLAS
30.4
14.3
OS ATLAS
SIMELAR TM
34.9
35.6
OS-ATLAS
QWEN2-VL
30.9
14.8
OS-ATLAS
SSNIILAR-RE
32.0
from these environments, generating preference data. This preference data then used perform DPO training the open-source agents OS-Atlas (Wu et al. 2024c) and UGround (Gou et al.) 2024), validating our model's 's ability to guide agents in the training phase. 2) Inference Phase. With Android World and OSWorld as benchmarks, we used OS-Atlas as the open-source agent and GPT-4o-1 120 and GPT-4-Turbo the closed-source agents. During inference, Similar and other reward models evaluated the agent's simulated N actions, providing rewards and updating the states of nodes in MCTS. Notably, 30% of the examples partitioned from these benchmarks mentioned carlier were used as the evaluation data.
OS-ATLAS
SINILAR-TM
34.5
16.4
OS-ATLAS
LLAMA-3.2-V
31.5
14.8
0S OS-ATLAS
RL.
32.9
15.7
OS-ATIAS
+ SINTEAR-THI TM
35.4
17.8
hensive and accurate assessment of GVA action quality. 2) Synergistic gain from the Triple-M strategy: Similar- TM-Llama achieved an Avg score of 61.2, significantly outperforming BiiillarRR--Laann with 53.9 (f 13.5%). And it achieved higher scores across all dimensions, with improvements such as H increasing from 48.2 to 63.8 (t 32.3%) and E increasing from 47.1 to 59.2 (f 25.6%). The Simi aarl-arr----we model showed similar perfor- mance. This highlights the effectiveness of our Triple-M strategy, leveraging the complementary expertise of each component to achieve synergistic gain. The experiments demonstrate our model's 's ability to align preferences.
5.2. Effective Alignment of Preference
We first report the performance the models on SRMEval in Table [ The main findings are follows: 1) Effective- ness of step-wise, multi mintiiddmmessonnll, cross-platfone data: Using our collected data for reward modeling, Similam lar- RL-Llama achieved an Avg score of 53.9, remarkably outperforming the bascline Llama-3.2-1 2 Vision-Instruct with 47.6 (t 13.2 13.2%) and surpassing closed-source models GPT-4o (51.4) and GPT-4-Turbo (46.6). It demonstrates that training reward models with our data enables fine- grained, based evaluation, providing more compre-
5.3. Similar for RL Training
We used GPT-4o and multiple reward models to annotate reward data across benchmark environments. The annotated data was then used train the final agent via DPO. The results, shown in Table demonstrate that model sig- nificantly improves agent learning: 1) The Simi lar-RL model derived through Reward Modeling the SRMTrain dataset outperforms the baseline. When using OS-Atlas as the agent, Simi lar-R Lar-RL- -ee-RL-LLaa RL-Llama achieves improvements of

---

A Step-Wist Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
Table 4: Abaltion study (inference experiments), Similar in table represents Meaillar-T---amme
36
Correlation dimensions
044
H
1.0
0.38
0.47
0.25
0.24
SimiorTH Uama
122
Ul
1.0
0.26
DIMENSION
SUCCESS RATE
30
Llama-s N-liin
Dimerslons
E 0.47
0.33
0.25
0.21
MODEL
H
B
C
WA
18
28
TR
0.28
0.26
0.25
1.0
0.42
BACKBONE
30.4
20.6
2.24
/
32.5
26.1
+0S
31.9
24.7
N
05
Dimeriiinns
TR
+E
31.6
23.3
(a)
(b)
+TR
31.1
21.6
+C
30.9
21.0
Figure 6: (a) Inference-time scaling research. The agent is
+OS.B
32.7
27.5
GPT-4o. (b) Correlation research of five dimensions.
+H,E
33.1
29.8
+H,OS
33.4
31.4
method is generalizable and effectively enhances the virtual agent's inference ability.
+TR.C
<
31.5
22.5
+
34.3
35.9
+OS.E,TR.C
<
33.1
33.9
We further demonstrate that our model is essential for seal- ing the inference- capabilities of agents by varying the number of child nodes N in MCTS. As shown in Fig- ure 6 (a): 1) When N S 8, agent performance improves. However, when N 8, performance plateaus or declines, likely due to limitations in the agent model, as simulating more actions fails to identify viable paths. 2) Simila -ar-RL and Simi -ammilarriii TM outperform other settings, with Similar- RL surpassing MLLM- baseed reward models and Simi lar- TMM exceeding Similar-Ril These results demonstrate the superiority of our models while highlighting the challenges of scaling inference time in agent systems.
H.E.TR.CC
33.9
35.7
H..O..R..C
34.2
36.5
H,,OS,ECC
34.7
37.2
+H,OS,E.TR
35.1
37.7
SINIDAR
V <
35.4
38.2
10.5% (30.4 + 33.6) and 7.3% (31.3- - 33.6) over the original OS-Atlas model and the setting using Llama-3.2V as the reward model, respectively, on Android World. On WebArena, the improvements are 35.6% (20.2 - 27.4) and 22.3% (22.4- - 27.4), respectively. 2) The Similar- TM model performed best. With OS-Atlas, Similar- TM-Llama achieved improvements of 3.8% (33.6 - 34.9) and 29. 29.9%(27.4 35.6) on Android World and WebArena, respectively, compared to Simi Similar-RR-il -L-Laaaa 3) When using UGround as the agent or adopting Qwen2-VL as the baseline reward model, comparable performance can be ob- served. The consistent performance improvements across different models and environments demonstrate that our method enhances virtual agents' learning capabilities.
5.5. Indepth Analysis
Ablation study. The results, shown in Table 4 show that models with partial-dimensional rewards underperformed compared to Similar. For example, on Android World, models excluding H, OS, E, TR, and C rewards showed de- clines of 6.9%, 4.4%, 3.5%, 2.0%, and 0.8%, respectively, with similar trends on WebArena. Analysis reveals that the H dimension has the most significant impact, as Helpfulness captures a step's 's contribution to task completion. The OS dimension follows closely, reflecting the influence of the current step on the next step. The C dimension has the least impact, as agent actions are often inherently coherent and contextually aligned. These results confirm that fine-grained rewards outperform coarse- -ranined ones and that our five dimensions comprehensively assess agent actions. More comprehensive results can be found in Appendix D
5.4. Simi lar for Inference-Timee Scaling
During inference, we used various reward models evalu- ate the agent's N simulated actions, providing rewards and updating MCTS node states. Table 3 shows that our model effectively guides the agent: 1) Consistent with the train- ing setup, the Similar-RI model outperformed both the original agent without a reward model and the setting using MLLM as the reward model. With GPT-4o, Similar- RL-Llama achieved improvements of 16.5% (25.4 29.6) and 12.9% (26.2 29.6) on Android World for these two settings, respectively. A similar performance is observed OSWorld. 2) The Simi lar- model performed best. With GPT-4o, Similar- TM-Llama achieved improve- ments of 16.8% (29.6 + 34.6) and 25.9% (13.1 + 16.5) on Android World and OSWorld, respectively, compared to Similax-RRL-Llma. 3) When employing GPT---Turb, GPT-4o, or OS-Atlas the agent, or when using Qwen2- VL as the baseline reward model, we consistently observe similar model performance. It can be concluded that our
Case Study. To demonstrate the role of our model in train- ing and inference, we included visual cases, as shown in Figure During training, Similar annotates the agent's ss trajectory with multi-dimmessonal scores, used for DPO training. During inference, the agent simulates multiple actions for single step, and Simi lar evaluates these ac- tions. In the figure, our model assigns high scores to action 2 at the third step, with a total score of 0.94, while action 1 re- ceives lower scores. Therefore, action 2, the highest-scoring action, is easily selected for the current step. More case studies are detailed in Appendix D
8

---

A Siep,Wis, Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
Correlation Study. The Pearson correlation coefficients among the five dimensions are calculated to analyze their in- dependence, as shown in Figure (b). The results show that while some correlation exists among the five dimensions, the values are all below 0.47, indicating independence.
The Triple-M strategy further enhances the RL training pro- cess by integrating multiple dimensions of assessment and leveraging the strengths of different experts. This synergis- tic approach not only improves the accuracy of the reward signals but also ensures that the agent learns robust policies that generalize well across diverse tasks and environments. As a result, Similaz significantly reduces the reliance on labor-intensive human annotations, making RL training more scalable and cost-effective.
6. Conclusion
In this work, we introduce a novel reward model-based paradigm for training GVAs. Our reward model, Simi 1 lar, provides step, multi-dimensional feedback during GVAs' training and inference, enabling fine-grained as- sessment. Additionally, we build the first reward model evaluation benchmark called SRM. Extensive experiments demonstrate our model's superior performance on SRMEval and its effectiveness in guiding GVAs across diverse tasks. Acknowledgment. This work was supported by the NSFC (62272411), )). the Fundamental Research Funds for the Cen- tral Universities (226-2025-04 2025-00017) the Key R&D Projects in Zhejiang Province (No. 2024C01 106, 2025C01030), Ningbo Yongjiang Talent Introduction Pr0gramme(2024A- 401-G),the Zhejiang NSF (LRG25F020001), Ant Group.
Impact on Inference- Scaling
Recent outstanding work has demonstrated that inference- time scaling can significantly enhance agent perfor- mance (DeepSeek-Al et al. 2025] Wu et al.) 2024b). Similar significantly enhances the inference-time capa- bilities of GVAs by integrating with search algorithms like MCTS. During inference, our model provides fine-grained, multi-dimensional rewards to evaluate and filter candidate actions, ensuring the agent selects the most promising paths. This not only improves task completion accuracy but also reduces computational overhead, making it highly efficient for real-time applications.
Impact Statement
Experiments demonstrate that our model consistently out- performs baseline models and MLLM based reward settings across diverse environments, such Android World and OSWorld. The Similar-TM model, leveraging the Triple- M strategy, achieves the best performance, highlighting the synergistic gains from integrating multiple dimensions of assessment. Furthermore, our model effectively scales inference-time computations, with performance improve- ments observed when expanding the number of child nodes in MCTS, though performance plateaus beyond a certain threshold due to inherent agent limitations.
Limitations and Future Investigation
While Similar demonstrates significant advancements in the training and inference of GVAs, several limitations remain. First, the current framework relies heavily the quality of the automatically annotated data generated by the MCTS-P algorithm. Although this approach reduces human annotation efforts, it may introduce biases or inaccuracies in the reward signals, particularly in complex or ambigu- ous task scenarios. Future work should focus on improving the robustness of the automatic annotation process, poten- tially by incorporating more sophisticated error-correction mechanisms hybrid human-At AI annotation strategies.
Impact on Data Cleaning
Simi lar also has significant implications for data clean- ing in the context of GVA training. The model's ability to provide fine-grained, multi-dimensional feedback allows for the precise identification and systematic removal of low- quality or noisy data points. This is particularly useful in large-scale datasets where manual inspection is impracti- cal. By filtering out irrelevant or incoherent actions and prioritizing consistent, lask-aligned examples, our model ensures that the training data is of high quality, leading to more robust and reliable agent performance.
Moreover, the scalability of Simi immilar across diverse envi- romments and tasks is promising but not yet fully explored. While the current experiments cover four major platforms (Web, Android, Linux, and Windows), the model's per- formance in more niche or specialized domains remains untested. Future investigations should extend the evaluation to broader range of environments, including those with less structured or more dynamic interfaces, to ensure the generalizability of the approach.
Impact on RL Training
Moreover, the automatic annotation process introduced by Simi reduces the need for human intervention in data cleaning, further enhancing the scalability of GVA train- ing. This is especially beneficial in domains where data is abundant but of varying quality, such web navigation or mobile app interaction. By improving the quality of the training data, our model contributes to the overall efficiency and effectiveness of GVA training pipelines.
Similar has a transformative impact on reinforcement learning (RL) training for GVAs. By providing fine- mie-grained, multi-diimensionl feedback, Simi lar enables more effi- cient and effective learning compared to traditional outcome- based reward models. The step-wise, multi-dimensional assessment allows the agent to identify and correct errors at intermediate stages, leading to faster convergence and improved task performance.
9

---

Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y., Zhang, Y. Shi, Y., Xiong, Y., He, Y., Piao, Y., Wang. Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Ou, Y., Wang. Y., Gong, A Step-Wisc, Multi-Dimennional, and Generalist Reward Model with Benchmark for Virtual Agents
References
F. Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Ding, H., Xin, H., Gao, H., Qu, H., Li, H., Guo, J., Li, J., Wang. J., Chen, J., Yuan, J., Qiu, J., Li, J., Cai, J. L., Ni, J.. Liang, J., Chen, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang. l.,, Zhang, L., Zhao, L.,, Wang, L.,. Zhang, L., Xu, L., Xia, L., Zhang, M., Zhang, M., Tang, M., Li, M., Wang. M., Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q. Chen, Q. Du, Q. Ge, R., Zhang, R., Pan, R., Wang. R., Chen, R. J., Jin, R. L., Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S. Wang, S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S. Wu, S., Ye, S., Yun. T., Pei, T., Sun, T., Wang, T., Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W. Yu, W., Zhang, W., Xiao, W. L.,, An, W., Liu, X., Wang, X., Chen, X., Nic, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang. X., Li, X., Su, X., Lin, X., Li, X. Q., Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song, X., Zhou, X., Wang, X., Shan, X., Li, Y. K., Wang, Y. Q. Wei, Y. X., Zhang,
Arulkumaran, K., Deisenroth, M. P.. Brundage, M., and Bharath, A. A. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38, 2017. doi: 10.1 11100/9M222211..27555552
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biomeirika, 39:324, 1952. URL httph:ttt///appt sema
nticscholaze org/
Branavan, S., Chen, H., Zettlemoyer, L., and Barzilay, R. Reinforcement learning for mapping instructions to actions. In Su. K.-Y., Su, J., Wiebe, J., and Li, H. (eds.), Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th Interna- tional Joint Conference on Natural Language Processing of the AFNLP, pp. 82-90, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL. https: http://aaaaaaahhccoo hwtp://oooyoro/ooopooo 10107
Branavan, S., Zettlemoyer, L., and Barzilay, R. Reading between the lines: Learning to map high-level instruc- tions to commands. In Hajič, J., Carberry, S., Clark, S., and Nivre, J. (eds.), Proceedings of the 48th Annual Meeting of the Association for Computational Linguis- tics, pp. 1268-1277, Uppsala, Sweden, July 2010. As- sociation for Computational Linguistics. URL 1/aclanthology. ott.ooy010///001111111
Y. Zou, Y., He, Y., Xiong. Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao,
Z., Ma, Z., Yan, Z., Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z Song, Z., Pan, Z., Huang, Xu, 7.., Zhang, Z. and Zhang, Z. Deepseck-rl: Incentivizing reasoning capability in llms via reinforcement learning. 2025. URL https:// www.v.owg.......o......... httpg///aa///...... 12950....99..
Bu. W., Wu, Y., Yu, Q., Gao, M., Miao, B., Zhang, Z., Pan, K., Li, Y., Li, M., Ji, W., Li, J., Tang, S., and Zhuang, Y. What limits virtual agent application? omnibench: A scalable multi- milei-dimensional benchmark for essential virtual agent capabilities, 2025. URL hRthttts://axx v. v.org/abs abs 2506. 08933
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards generalist agent for the web. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 28091-281144 Curran Associates, Inc., 2023. URL https///pcceeddnng.eeuripp.o..op ff f 0aa001119882222110000028 Papaoo----Papee---ttttteeeeee nd Benchharks.... pdf
Burns, A., Arsan, D., Agrawal, S., Kumar, R., Saenko, K., and Plummer, B. A. A dataset for interactive vision language navigation with unknown command feasibility. In European Conference on Computer Vision (ECCV), 2022.
Christianos, F., Papoudakis, G., Zimmer, M., Coste, T., Wu, Z., Chen, J., Khandelwal, K., Doran, J., Feng, X., Liu, J. Xiong, Z., Luo, Y., Hao, J., Shao, K., Bou-Ammar, H., and Wang, J. Pangu-agent: A fine-tunable generalist agent with structured reasoning, 2023. URL https: //arxiv. wwwxivvoorg/ae abs/ 2alp.//211...1 14878
Fei, H., Wu, S., Zhang, H., Chua, T.-S., and Yan, S. Vit- ron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing. 2024.
Gao, M.. Bu, W., Miao, B., Wu, Y., Li, Y., J., Tang, S., Wu, Q., Zhuang, Y., and Wang, M. Generalist virtual agents: A survey on autonomous agents across digital platforms, 2024.
DeepSeek-AI, Guo, D., Yang. D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F,, Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F, Dai, F., Luo,
2411.10943
Gao, M., Liu, X., Yue, Z., Wu, Y., Chen, S., Li, J., Tang, S., Wu, F., Chua, T.S.., and Zhuang, Y. Benchmarking multimodal cot reward model stepwise by visual program,
10

---

A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
2025. URL httts:://axiv..ott
2504.0
Li, J., Tang, S., Zhu, L.., Zhang, W., Yang. Y., Chua, T.-S., Wu, F., and Zhuang. Y. Variational cross-graph reasoning and adaptive structured semantics learning for composi- tional temporal grounding IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10))1 2601-112677, 2023b. doi: 10.1100TPAM..22333374...
6606
Gou, B., Wang, R., Zheng, B., Xie, Y., Chang, C., Shu,
Y. Sun, H., and Su, Y. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05245 2024. URL https: //axxiv.ore abs/ /t.b///4100 2424..524.33
Li, Y., Lin, Z., Zhang. S., Fu, Q.. Chen, B., Lou, J.-G., and Chen, W. Making language models better reasoners with step-aware verifier. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61 Annual Meet- ing of the Association for Computational Linguistics (Vol- ume I: Long Papers), pp. 5315-5333, Toronto, Canada, July 2023c. Association for Computational Linguistics. doi: 10.118663/111202233ell1o0o 291. URL https ac
He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint vrXiv:2401.13919 2024.
Humphreys, P. C., Raposo, D., Pohlen, T., Thornton, G., Chhaparia, R., Muldal, A., Abramson, J., Georgiev, P., Santoro, A., and Lillicrap, T. A data-driven approach for learning to control computers. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9466-9482. PMLR, 17- mlrr
//acllanthology.orggg
long 291/
Lightman, H., Kosaraju. V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.200555 2023.
Luo, L., Liu, Y., Liu, R., Phatale, S., Guo, M., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L.., Sun, J., and Rastogi, A. Improve mathematical reasoning in language models by automated process supervision, 2024. URL https //arxiv.orq//
23 Jul 2022. URL/Bcttss://ppreeeeeei press press/vi62/1 Trres//116/hummhreeys2222 html
Hurst, A., Lerer, A., Goucher, A. P.. Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card, arXiv preprint arXiv:2410.21276, 2024.
06592
Miao, B., Zhang, W., Li, J., Tang. S., Li, Z., Shi, H., Xiao, J., and Zhuang, Y. Radar: Robust two-stage modality- incomplete industrial anomaly detection, 2024. URL https://arxiv.cm wwtp://gg/abb///////// Bttp:://aaxxi..rrr//bb//.....
Koh, J. Y., Lo, R., Jang, I.., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. Visualwebarena: Evaluating multimodal agents on re- alistic visual web tasks. arXiv preprint 13X1422401.13649 2024.
01737]
Pan, K., Fan, Z., Li, J., Yu, Q., Fei, H., Tang, S., Hong. R., Zhang, H., and Sun, Q. Towards unified multimodal edit- ing with enhanced knowledge collaboration. Advances in Neural Information Processing Systems, 37:1 10290- 1 110014 14, 2024a.
Lai, X., Tian, Z., Chen, Y., Yang, S., Peng, X., and Jia, J. Step-dpo: Step-wise preference optimization for long- chain reasoning of Ilms. (xXi::2406.18622 2024.
Pan, K., Tang, S., Li, J., Fan, Z., Chow, W., Yan, S., Chua, T.-S., Zhuang, Y., and Zhang, H. Auto-encoding morph-tokens for multimodal 1lm. arXiv preprint arXiv:2405.01926 2024b.
Li, J., Wang, X., Tang, S., Shi, H., Wu, F., Zhuang, Y., and Wang, W. Y. Unsupervised reinforcement learning of transferable meta-skills for embodied navigation. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12123 12132, 2020.
Pan, K. Lin, W., Yue, Z., Ao, T., Jia, L., Zhao, W., Li, J., Tang, S., and Zhang, H. Generative multimodal pre- training with discrete diffusion timestep tokens. arXiv preprint arXiv:2504. |4666, 2025.
Li, J., He, X., Wei, L.. Qian, L., Zhu, L., Xie. L., Zhuang, Y., Tian, Q. and Tang, S. Fine-grained semantically aligned vision- language pre-training, Advances in neural information processing systems, 35:7290-7303. 2022.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimiza- tion: Your language model is secretly reward model. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53728 53741. Curran Associates, Inc., 2023. URL https //proceedingss neuripssps... caper files
Li, J., Pan, K., Ge, 7.,, Gao, M., Ji, W., Zhang, W., Chua, T.-S., Tang, S., Zhang, H., and Zhuang, Y. Fine-tuning multimodal llms to follow zero-shot demonstrative in- structions. In The Twelfth International Conference on Learning Representations, 2023a.

---

A Slep-Wise, Multi-Dimeensonal,, and Generalist Reward Model with Benchmark for Virtual Agents
e paaper/202/fflee/485556666663e32
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z.,, Tang, J., Chen, X., Lin, Y., Zhao, W. X., Wei, Z., and Wen, J.-R. A survey on large language model based autonomous agents, 2023.
Rawles, C., Clinckemailiie, S., Chang, Y., Waltz, J., Lau, G., Fair, M., Li, A., Bishop, W., Li, W., Campbell-Ajala, F., Toyama, D., Berry, R., Tyamagundlu, D., Lillicrap, T., and Riva, O. Androidworld: A dynamic benchmarking environment for autonomous agents, 2024a. URL http
Wang. P., Bai, S., Tan, S., Wang, S., Fan, Z.., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang. K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing rision-language model's perception of the world at any resolution. arXiv preprinl arXiv:2409. 12/91, 2024b.
s: s://arx /t//arxiv.or wttp:////bbbbb///...b./ 14573
Rawles, C., Li, A., Rodriguez, D., Riva, O., and Lillicrap, T. Android in the wild: large-scale dataset for android device control. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024b. Curran Asso- ciates Inc.
Wang. Z., Li, Y.. Wu, Y., Luo, L.,, Hou, L., Yu, H., and Shang, J. Multi-step problem solving through a verifier: An empirical analysis on model-induces process super- vision. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. Findings of the Association for Computational Lin- guistics: EMNLP 2024, pp. 7309-7319, Miami, Florida, USA, November 2024c. Association for Computational Linguistics. doi: 10. 10011/66/////20444ffnnnnngooooo/
Shen, Y., Song, K., Tan, X., D., Lu, W., and Zhuang, Y. Hugginggpt: Solving AT tasks with chatgpt and its friends in hugging face. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. /ttL:tttp////peeee cc
ndings Saings-emnlpp www.m.nnpp..... 4297
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.S.. NExT- GPT: Any-to-any multimodal LLM. In Proceedings of the International Conference Machine Learning, pp. 2024a.
Abstrt Conference.html
Wu, S., Peng, Z., Du, X., Zheng, T., Liu, M., Wu, J., Ma, J., Li, Y., Yang. J., Zhou, W., al. A comparative study on reasoning patterns of openai's ol model. arXiv preprint mxXiv:2410.33633 2024b.
Snell, C., Lee, J. Xu, K., and Kumar, A. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://aaxx
beerg//aa v. v.org/abs/ 2408. 03314
Wu, Z., Wu, Z., Xu, F., Wang. Y., Sun, Q., Jia, C., Cheng, K., Ding, Z., Chen, L., Liang, P., et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint 24X:v:2410.232118 2024c.
Song, Y., Yin, D., Yue, X., Huang, J., Li, S., and Lin, B. Y. Trial and error: Exploration-based trajectory optimization of LLM agents. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7584-7600, Bangkok, Thailand, August 2024. Association for Computational Linguistics. 409. URL https: 7/ac pacclantholoo www.laancoooooy..rg//22222 wae.:aaallllnonnnn...eeemae long. 4097
Xi, Z.,, Ding, Y., Chen, W., Hong, B., Guo, H., Wang, J., Yang, D., Liao, C., Guo, X., He, W., Gao, S., Chen, L., Zheng, R., Zou, Y., Gui, T., Zhang, Q., Qiu, X., Huang, X., Wu, 7., and Jiang, Y.GG Agentgym: Evolv- ing large language model-based agents across diverse environments, 2024.
doi: 10. 100.18655//1112222.alllloooo
Xie, T., Zhang, D.,, Chen, J., Li, X.,, Zhao, S., Cao, R., Hua, T.J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024.
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, Soly- ing math word problems with process- and outcome- based feedback, 2022. URL g/abs 14275
Xu, N., Masling, S., Du, M., Campagna, G., Heck, L., Landay, J., and Lam, M. Grounding open-domain instruc- tions to automate web support tasks. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani- D., Belt- agy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Conference of
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable preferences via multi-objective reward mod- eling and mixturr--ffeepprrs. In The 2024 Conference on Empirical Methods in Natural Language Processing, 2024a.
12

---

A Step-Wiet Wisc, Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies, pp. 1022-1032, Online, June 2021. Association for Com- putational Linguistics. doi: 10.18653/v1/2000 10.18655//1122221mmmmaa maacl- n.80. URL /tttps://acho http://cllaanttooolooooorr.// naacl main.80
Zhai, Y., Yang, T., Xu, K., Dawei, F,, Yang, C., Ding, B., and Wang, H. Enhancing decision-making for Ilm agents via step-level q-value models, 2024. URL https://
orxiv.org/aal absp///b///// /bwpg///bb//4..... 2409. 09345
Zhang, C., Li, L.,, He, S., Zhang, X., Qiao, B., Qin, S., Ma, M., Kang, Y., Lin, Q.. Rajmohan, S., Zhang, D., and Zhang, Q. Ufo: A ui-focused agent for windows os interaction, 2024a. URLhttps: /ttppss:///xxv...o org/ab 5 s/2402.0 07939
Yan, A., Yang, Z., Zhu, W., Lin, K., Li, L., Wang. J., Yang, J., Zhong, Y. McAuley, J., Gao, J., al. Gpt-4v in won- derland: Large multimodal models for zero-shot smart- phone gui navigation. arXiv preprint arXiv:2311.07562, 2023.
Zhang, D., Zhoubian, S., Hu, Z.,, Yue, Y., Dong, Y., and Tang, J. Rest-mets*: Lim self- self-training via process reward guided tree search. arXiv preprint 2024b.
Yu, F., Gao, A., and Wang, B. OVM, outcome-supervised value models for plamning in mathematical reasoning. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 858-875, Mexico City, Mexico, June 2024a. Association for Computational Linguistics. doi: 10.186 53/v 53/v1/2024 55. htttthtt////cl
Zhang, X., Du, C., Pang, T. Liu, Q. Gao, W., and Lin, M. Chain of preference optimization: Improv- ing chain-of-thought reasoning in llms. arXiv preprint arXiv:2406 24t.::400.09966, 2024c.
nt othology.org /ww../2ggg2222..2// 2024. findi ndi ngs wwacc.........mm........mmam...a 55/
Zhou, R., Yang. Y., Wen, M., Wen, Y., Wang, W., Xi, C., Xu, G., Yu, Y., and Zhang, W. TRAD: Enhancing Ilm agents with step-wise thought retrieval and aligned decision. In Proceedings of the 47th International ACM SIGIR Con- ference on Research and Development in Information Retrieval (SIGIR), 2024a.
Yu, L., Jiang, W. Shi, H., Yu, J., Liu, Z. Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023a.
Yu, Q. Li, J., Wu, Y., Tang, S., Ji, W., and Zhuang, Y. Visually- proomppte language model for fine- grainer scene graph generation in open world. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 21560-21571, October 2023b.
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al. Webarena: A realistic web environment for building autonomous agents. ICLR, 2024b.
Yu, Q., Li, J., Wei, L.., Pang, L., Ye, W., Qin, B., Tang. S., Tian, Q., and Zhuang, Y. Hallucidoctor: Mitigating hal- lucinatory toxicity in visual instruction data. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12944-12953, June 2024b.
Yuan, Z.,, Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning mathe- matical reasoning with large language models, 2023.
Zang, Y., Dong, X., Zhang, P., Cao, Y., Liu, Z., Ding, S., Wu, S., Ma, Y., Duan, H., Zhang, W., Chen, K., Lin, D., and Wang, J. petrrnlmxxccoooerr2..55555
A simple yet effective multi-modal reward model, 2025. URL atttp:://arxia org/ abs/ 2501. 12368
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Bootstrapping reasoning with reasoning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and
Oh, A. (.ddan).. in Neural Information Processing Systems, volume 35, pp. 15476-15488, 2022. URL http
s: et//ppoceedinng..eerpppscc.oooop
es /as/pppe//022//11//3333312222220 41 1755faal2e955-Ppee--Offfeeeeeee
13

---

Appendix
This is the Appendix for the paper "Boosting Virtual Agent Learning and Reasoning: A Step-wise, Multi-dimensional, and Generalist Reward Model with Benchmark".
agent, the two ACTION.X and ACTIONLY predicted by the agent, and the current action step number STEP_IDX. Your GOAL is help me complete step-wise evaluation, that is, evaluate the quality of the Agent's s ACTION in a specitic dimension. Choose the better action (ACTION_X or ACTIONLY) based on the given (EVALUATION DIMENSION). Output tyrrr and the reason if ACTION is better, or "'' and the reason if ACTION.Y is better. Do not output respenses like 'two actions are similar''.
Overview
In this supplementary material we present:
Detailed prompts for testing general MLLMs on SRMEval and for building SRM are provided in Section
Pseudocode for the Similar pipeline, including key al- gorithmic steps, is described in Section B
(Word Meaning)
Human evaluation details and human acceptance of 5- dimension data from SRM are illustrated in Section C
1. INSTRUCTION: refers to the command of human users to the Agent, which is the specific content that the Agent needs to complete the task on specific platform, that is, the ultimate GOAL of the Agent. 2. OBSERVATION: refers to the specific information of the current platform that an agent can observe on the platform where the task needs to be completed, which is the envi ronment which the our task,
Case studies demonstrating the applications similar are presented in Section D
Comprehensive ablation experiments conducted and analyzed in Section
Additional visualizations of SRMEval, showcasing task performance, are provided in Section
agent currently located. observations presented in the form of images, known screenshots. 3 TRAJECTORY: refers to the action prediction made by an agent in the past to complete the INSTRUCTION, which records all actions taken by the agent from the first step to the current step. If this is the first step, then the trajectory is empty.
A. Detailed Prompt Design
This section provides comprehensive overview of the prompt designs used in experiments. We detail the prompts for testing general MILMs on SRMEval and the prompts for building the SRM model, highlighting their structure and purpose.
A.1. Prompt for testing general MILMs on SRMEval
This subsection describes the prompts used to evaluate gen- eral MILMs on the SRMEval benchmark.
4 A.ATTIN: refers to the predicted operation of the Agent in the current state to complete the INSTRUCTION in the current step. This operation generally refers to simple action command, such as "CLICK'' 'TYPE'' etc. Note that ACTION is the result predicted the agent after observing the current OBSERVATION, and the Agent often cannet complete the task in one step.
Prompt for SRMEval (main part)
You are an expert in ivaluating the performance of Virtual Agent.
The Virtual Agent is designed to help human user complete specified tasks (such as app usage, web navigation, web content Q&A, etc.) on various platform applications (such websites, mobile devices, operation systens, etc.) based en given instructions. Given the user's INSTRUCTION, the OBSERVATION of current platforms, the action TRAJECTORY of the
5. STEP_IDX: refers to the sequence number of the Agent executing the current ACTION to complete the INSTRUCTION.
14

---

A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
Here is the evaluation dimension part of the prompts.
Efficiency.
Helpfulness.
Prompt for SRMEval (Efficiency)
Prompt for SRMEval (Helpfulness)
3. [EFFICIENCY]
3.1 Meaning: It indi cates whether this step is efficient in completing the task. We calculate this metric as the difference between 'the number of steps required to complete the final task after the current step' and 'the number of steps required to complete the final task after the previous step', divided by 'the total number of steps required to complets ete the task'. This indicates the degree of efficiency improvement in completing tasks after the current step is executec.
1. [HELPFULNESS]
1.1 Meaning: It indicates the degree to which this step contributes to the completion of the final task. There are geod and bad contributions, the correct steps will give positive contribution, and the wrong steps will give negative contribution.
1.2 Design motivation: Different steps contribute di diferrenlly to the completion of the final task, with good steps helping to accomplish the task and bad steps hi hindering it. Good steps should be rewarded positively, while bad steps shculd be punished negatively. If each step is correct and the total number of steps is 5, then the contribution of each step can be considered as 1/5, meaning that each step completes 1/5 the final task. If more steps needed from the current step and the current step is incorrect, then the centribution of the current step is -1/4, indicating that it hinders 1/4 of the final task progress.
3.2 Design motivation: basic assumption is that the fewer steps the Agent operates, the nore efficient it is, because the consumption of these paths (time consumption, hardware consumption. can be considered to be the least and the efficiency the highest. Therefore, if the operation can reduce the number of steps required to complete the task as whole, then it can be considered that the opcraticn of this step is very efficient. For example, after the previous step, it takes steps to complete the task, but after the current step, only takes 4 steps to complete the task. The difference of 7-4-3 is the efficiency improvement of the current step in completing ing the final task.
step
Odds of Success.
Prompt for SRMEval (Odds of Success)
2. [ODDS SUCCESS]
2.1 Keaning: It indicates the potential of the step to complete the task, which
is the probability of a step reaching the completion of the task. 2.2 Design motivation: The more correct steps lead a higher probability of success the final task, and the more incorrect steps lead to a higher probability of failure in the final task. Different steps have different potential to complete the task. If one step of the agent is to follow the Instructions to complete the task, then this step generally has high potential. We can derive the probability of step leading success from the paths generated by that step, which serves as the potential for that step to complete the task which is crucial for evaliating.
Task Relevance.
Prompt for SRMEval (Task Relevance)
4. [TASK RELEVANCE]
4.1 Meaning: It indicates is whether the operation of the Agent is related to achieving the INSTRUCTION.
4.2 Design notivation: Some operational steps may prevent the task from being completed, but they are related to the task (for example, we need to ask the agent to take notes, and the agent takes notes, which related to the task, but the recorded note content is incorrect, indicating that this step). Some operational steps may be
incorrect
15

---

A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
meaningless, but they can still lead task completion (such as clicking blank screen without generating any response, which is unrelated to the task, but the agent's s aubsequent actions can still result in task success). Therefore, an indicator is needed to identify whether the current step of operation is related to the task.
expressed as the average total score of all steps in the trajectory.
A.2. Prompt for building SRM
This subsection outlines the prompts designed for construct- ing the SRM model.
Prompt for building SRM
4.3 Range of values after mapping: {0, 1}. The larger the value, the greater the correlation between the step and the task.
You virtual agent.
The Virtual Agent is designed to help a human user complete specified tasks (such app usage, web navigation, web content Q&A, etc.) on various platform applications (such as websites, mobile devices, operation systems, etc.) based on given instructions.
Coherence.
Prompt for SRMEval (Coherence)
[COHERENCE]
You will predict the next action based on the following content [INSTRUCTION], [OBSERVATION], DRESSONSTEPS!!
5.1 Keaning: It represents the compactness and coherence between the current step and the previous step.
1. [INSTRUCTION]: It is your ultimate GOAL, and all your actions are almed at completing this task.
5.2 besign motivation although task-related, not inefficient, and highly likely to lead to success, lack coherence with the previous step. For example, the task is to "query the Lakers' game results and record them in the Note'', The Agent operations are follows: a Open the browser; b. Open Note; c. Create new notes; d. Search for Lakers games; e. Query the results the competition; 2. Record the results of the competition in your notes. be found that the operations of and b 1ack coherence, and it is more in line with haman preferences to directly search for competition results after opening the browser instead of simultaneously opening Note.
Some operati ons,
2. [OBSERVATION] an image, which is the screenshot of the platform (such as a computer screen).
is an observation of
STEPS] [ They are the trajectory of the actiens you performed to complete the instruction, from which you can understand how you thought in order to complete the instruction. If it is empty, it means it is currently the first step.
3. [REASON
the past
B. Similar Pipeline Pseudocode
The rapid advancement of MLLMs (Li et al. 2020 2022 Yu
et al. 20236 2024b Pan et al. [202a]] Wu et al. 2024a Fei
et al. 2024 text, vision, and other modalities, has enabled the develop-
Miao et al. 2024), which excel at integrating
5.3 Range of values after mapping: {0, 1}. The larger the value, the greater the coherence of the step.
ment GVAs (Gao et al. 2024 Zhang et al. 2024a Shen et 2023). which also inspires to study Reward Models for GVAs with its Benchmark.
Total dimension and Trajectory-level dimension.
In this section, we present the pipeline pseudocode for train- ing our proposed Similax model. The pipeline consists of three main components: I) five-dimensional process super- vision framework to evaluate agent steps, 2) an automatic generalist dataset collecting process, and 3) a Triple-M strat- egy for reward model training.
Prompt for SRMEval (Total and Trajectory-level)
6. [TOTAL]
Meaning: Integrated decision-making based on the dimensions mentioned earlier.
7. [TRAJECTORY]
B.1. Five-Dimensiona Process Supervision Framework
Meaning: Represents the quality of
The fve-dimensional process supervision framework sys- tematically evaluates the quality of an agent's 's steps using
the entire trajectory, which can be
16

---

A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
Algorithm 2 Tive-Dimensional Process Supervision
Algorithm 4 Triple-M Strategy for Reward Model Training
1: Input: Step St. ground truth a", number of simulations N
1: Input: Training dataset D. pre-trained MLLM fo, gating network
2: Output: Five-dimensional scores (H,, OS., E., R., Ct)
2: Oulput: Trained reward model I
3: Compute Helpfulness (H):
3: Stage Regression Layer Training
4: H, MM
--AC (1 2r:)
4: for each batch (2t y. D do
5: AC, = =max( -C; H2.0) 0)
Extract hidden state A = fa @ y)
6: Compute Odds Success (08):
6: Compute predicted scores = W h
7: OS, = E (or.y=a")
7: Update O, W using L.R. || ml
8:
end for
8: Compute Efficiency (E):
9:
Stage 2: Gating Network Training
9: E, Leng-1 - Lene
10:
for each batch (ar, Mchorss Besisce)) E D
11: Compute coefficients
(f)
10: Len; = (vg(Len(Ss.))
11: Compute Task Relevance (TR) and Coherence (C):
12:
Compute preference scores Ratoaen w Tehean
12: TR:
EvaluteE SS. instruction)
13: Compute preference scotes Ragjened expt Ressen)
Tegeced
13: C, = MILLM.Evallatee (S., S,-1)
14:
Update & using Lor = log expt/ddbseen tesptM
14: return (H,, OS,, F,, TR,
15:
end for
16: return Trained reward model R (o(So(0))TT
Algorithm 3 Automatic Generalist Dataset Collecting
rounds of iterative "labec-reveew-feedbeee""" cycles to clar- ify ambiguities of annotation (e.g., the complexity of Ul interaction tasks). Only after achieving 95% accuracy on validation samples did formal annotation begin. 2) Formal Amnotation: Each test sample in SRMEval was indepen- dently labeled by three annotators and three checkers. The final data in test set required > 99% accuracy.
1: Input: Task instruction g. platforms P (Web, Android, Linux, Winlows)
2: Output: Annolated dataset D
3: Initializze empty dataset D
4: for each platform E P do
5: Initialize MCTS-P tree Ty lusk q on platform p
6:
for each node S.,, T,
7:
Calculate minimum steps Ad to reach a correct answer
8:
Simulate trajoctories to compute basic reward ri
9:
Compute OS,, E, using formulas from Algoritim
10:
Evaluate TRR, and Ct using MLLM (e.g.. GPT-4) node St., leads complete trajectory then
To validate the quality of our five-dimensional assessment data and ensure alignment with human preferences, we domly sampled batch of data from the SRM Benchmark. The sample size varied across dimensions due to differences in score distributions. This stems from their fundamental design - for instance, Task Relevance and Coherence are binary values, which naturally yield fewer possible prefer- ence pairs. Human annotators were then asked to select the better action from candidate action pairs in the sampled data, based specific evaluation types. If the annotator considers sample correct, mark it as 1; otherwise, mark it as 0, and calculate Accuracy as Human Acceptance.
11:
12:
Verily correctness using plaffor-sppcilie evaluation methods Add annotated step (Si.s OS; E,, TR,, to D
13:
14: end
15:
end for
16:
end for
17:
Prune incomplete branches from 2.g
18:
return Annotated dataset D
five distinct dimensions: Helpfulness (H), Odds of Success (OS), Efficiency (E), Task Relevance (TR), and Coherence (C). The following pseudocode outlines the computation of these dimensions for given step S,, providing a compre- hensive assessment of step quality.
B.2. Automatic Generalist Dataset Collecting
The results, as shown in Table 5 demonstrate that the hu- man acceptance rate for all five dimensions exceeds 78.8%, strongly indicating the superiority of our designed annota- tion dimensions and the high quality of the collected data.
The automatic generalist dataset collecting process lever- ages the MCTS-P algorithm to collect annotated step-wise data across multiple platforms, including Web, Android, Linux, and Windows.
The evaluation process was further enhanced by incorpo- rating a rigorous double-blind annotation protocol, where neither the annotators nor the analysts were aware of the origin or automated scores of the candidate actions.
B.3. Triple-M Strategy for Reward Model Training
The Triple-M strategy integrates multi-step, multi- dimensional, and multi-modal data for reward model train- ing, ensuring high flexibility, robustness, and adaptability. The following pseudocode outlines the two-stage training process, which includes regression layer optimization and dynamic gating network adjustment.
Table 5: Sample size and human acceptance rate for each dimension in SRM.
Dimension
Sample Size
Human Acceptance
C. Human Evaluation Details and Human Acceptance of SRM
Helpfulness
6000
87.9%
Odds of Success
2000
78.8%
To ensure high-quality annotations, we collaborated with a professional commercial data labeling team. The process included: 1) Training Phase: Annotators underwent three
Efficiency
6000
82.6%
Task Relevance
1000
84.7%
Coherence
2000
93.5%
17

---

A Step-Wisc, Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
D. Case Studies on Similar Applications
D.2. Odds of Success
D.I. Helpfulness
Case of Odds of Success
Case of Helpfulness
Input:
Input:


In Simple Calendar Pro, delete all the calendar events 2023-10-27. 


 Create timer with hours, 16 minutes, and 35 seconds. Do not start the timer.
[OBS]


[OBS]
DD. C0 O2.
[/OBS]
[TRAJ]
[/OBS]
Step 1: Click Search.
[TRAJ]
Step Click Calendar.
Step 1: Click Search.
Step 3: Scroll down.
Step 2: Click Clock.
Step Long press 27 Friday.
Step 3: Click 2. [/TRAJ]
Step 5: Click More options. [/TRAJ]
Output:
Output:
[ACTION_X) X]
click Backspace 2.
[ACTION_X] Navigate back. [/ACTION_X] [SCORE_X]
!/ACTION_X) [SCORE_X]
H: 0.72 [/SCORE_X]
OS: 0.75 I/SCORE
[ACTION_Y Y]
Click 1. [/ACTION_Y) [SCORE_Y] Н: -0.37
[ACTION_Y]
Click Import events from .ics file. [/ACTION_y] [SCORE_Y] OS: 0.13
[/SCORE_Y]
l/SCORE_V) Y]
In this case, the task is to set a timer for minutes and 35 seconds. According to the reasoning steps, the previous action clicking "2", which does not match the required time. The current step should involve deleting the incor- rect input immediately. ACTION X (Backspace) is correct and more helpful, while ACTION_Y (Click 1) further hin- ders task completion and is clearly harmful. Therefore, ACTION.X receives higher Helpfulness score.
In this case, the task is to delete events on specific day in the Calendar. The previous step, clicking "More Options", aimed to locate the delete button, but the current observation shows no delete option is available. Therefore, the correct action is to navigate back and search for the delete button elsewhere. ACTION X (Navigate back) is the appropriate choice, while ACTION. Y (Import file) is clearly incorrect. Thus, ACTION X receives a higher Odds of Success score.
18

---

A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
D.3. Efficiency
meal.
Servings: 2
Case of Efficiency
Prep Time: 10 mins
Ingredicnts: As desired
Input:
Directicns: Cook pasta, toss with Alfredo sauce and grilled chicken. Tcp with Parmesan and spices.

 In Simple Calendar Pro, delete all the events.


2. Quinoa Salad with Vegetables Description: Quick and easy for busy days.
[OBS]
Servings: 8
Prep Time: 30
Ingredients: To your liking Directions: Mix quinoa, diced veggies, feta, and lemon olive eil dressing. Add spices for flavor.
3. Butternut Squash Soup
Description: A healthy, delicicus choice.
Servings:
Prep Time: 45 mins
Ingredients: Per taste
[/OBS]
Directions: Saute onions and garlic, acd squash and broth. Puree and season with nutmeg, salt, and pepper. Substitute as needed. 

[TRAJ]
step 1: Click Search. [/TRAJ]
Output:
[OBS]
[ACTION_X]
Click Calender.
[/ACTION []
[SCCRE_X] X]
0.88 [/SCORE_X]
[ACTICN_Y] Y]
Input text 'Simple Calendar Pro!
[/ACTION_Y] [SCORR_Y] Y]
#: 6. 1/SCORE Y
[/OBS]
In this case, since the search interface displays past search history, including "Calendar", ACTION_X (Click "Calen- dar") is more efficient in the Efficiency dimension. However, ACTION_] Y is also correct approach, so its Efficiency score remains relatively high.
[TRAJ]
Step 1: Click Search.
Step Input text Broccoli.
Step Click Broccali.
Step 4: Click New Recipe.
D.4. Task Relevance
Step Input text Chicken Alfredo Pasta.
Case of Task Relevance
Input:
Step 6: Input text A delicious and healthy choice for any time the day.


Add these recipes to the Broccoli app:
Step 7: Input text 2 servings.
Chicken Alfredo Pasta Description: A healthy, delicious
Step Input text 10 mins.
19

---

A Slep-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
[/OBS]
Step 9: Input text desired.
[TRAJ]
step 10: Scroll down.
Step 1: Click menuitem ' Aue622 Grocery Gourmet Food' hasPopup:
step 11: Input text Cook fettuccine pasta, toss with Alfredo sauce and grilled chicken strips. Serve with sprinkle of Parmesan cheese. Try adding pinch of your favorite spices for extra flavor.
Step 2: Click link 'Pantry Staples 4891 item ). [/TRAJ]
Output:
[ACTION X]
step 12: Click SAVE.
click link 'Herbe, Spices \ Scasonings ( 707 item y/
[/TRA]]
Output:
l/ACTION_X] [SCORE_X]
[ACTICN_] X
C:
Click Cook. [/ACTION_]] [SCORE_X]
[/SCORE_X]
|ACTION
TR: [/SCORE_X] X]
Click menuitem \ue622 Grocery & Gourmet Food' hasPopup: menu.
[/ACTION_Y] [SCORE_Y]
[ACTICN_Y)
Navigate back.
l/ACTION_Y
(/SCORE_Y] Y]
[SCORF_Y]
TR:
[/SCORF_Y]
In this case, the task is to find relevant products in the "Herbs, Spices & Seasonings" category. The previous step in- volved the agent entering link, and the current step should logically advance the task. Clearly, ACTION X (clicking the "Herbs, Spices & Seasonings" link) is coherent and logical continuation, while ACTION_Y (repeating the ac- tion from step 1) is unintelligible and incoherent. Therefore, ACTION X receives a higher Coherence score.
In this case, the task is to create a recipe. The first 12 steps have completed the recipe creation process. In the Broccoli app. the next action should be directly click "Cook" (i.e., ACTION X), which is highly relevant the instruction. In contrast, "Navigate back" (i.e., ACTION Y) is not directly related to the instruction. Therefore, ACTION_X receives higher Task Relevance score.
E. Comprehensive Ablation Experiments
D.5. Coherence
The extended ablation study, presented in Table 6 pro- vides more comprehensive analysis of the impact of each dimension in the Similar model. The results confirm the trends observed in the main experiments and offer ad- ditional insights into the contributions of the five dimen- sions- Helps-eelssl (H), Odds of Success (OS), Efficiency (E), Task Relevance (TR), and Coherence (C) across three benchmarks: Android World, WebArena, and OSWorld.
Case of Coherence
Input:
(INST)
Add this exact product my shopping cart. think it is the "Herbs, Spices Seasoni ngs" category. |/INST]
[OBS]
E.1. Impact of Individual Dimensions
The results demonstrate that the Helpfulness (H) dimension has the most significant impact on performance, consistent with the findings in the main experiments. For example, adding H alone improves the success rate on Android World from 30.7% to 32.5%, on WebArena from 20.6% to 26. 16.1% 1%, and on OSWlll from 14.6% to 15.8%. This aligns with our hypothesis that the quality of a step is primarily reflected in its contribution to task completion, which H effectively
Geldayy Rod
20

---

A Step-Wisc, Multi-Dimensiona and Generalist Reward Model with Benchmark for Virtual Agents
Table 6: Ablation study (inference experiments). Similar in table represents Lar-ill------aam
DIMENSION
SUCCESS RATE
MODEL OS
E TR
C
ANDROID WORLD
WEBARENA
OSWORLD
BACKBONE
30.4
20.6
14.3
+HI
32.5
26.1
15.8
+OS
31,9
24.7
15.4
+E
31.6
23.3
15.2
+TR
31.1
21.6
14.9
+C
30.9
21.0
14.8
+H.OS
33.4
31.4
16.7
+H,E
33.1
29.8
16.5
+H,TR
32.8
28.5
16.3
+1CC
32.6
28.2
16.2
OSSE
32.7
27.5
16.3
OS,TR
32.3
26.8
16.0
OSSC
32.1
26.5
15.9
E,TR
31.8
25.9
15.7
E,C
31.7
25.7
15.6
+R.CC
31.5
22.5
15.1
H,OS,
34.3
35.9
17.2
T,
34,0
34.5
17.0
H,OS,C
33,8
34.2
16.9
H,E.T
33,5
33.8
16.7
H.E,C
33,4
33.6
16.6
H,,RR,C
33.2
33.1
16.5
OS,,
32,9
32.8
16.4
O,SE
32.8
32.7
16.3
OS,,RRCC
32.6
32.5
16.2
+E,TR,C
32.4
32.3
16.1
H,OS,E,TR
35.1
37.7
17.8
E,OS,E,C
34.7
37.2
17.6
H.OS.TR.C
34.2
36.5
17.3
H,E,TR,C
33.9
35.7
17.1
OS,FETR,C
33.1
33.9
16.9
SIMILAR
35.4
38.2
17.8
captures. The Odds of Success (OS) dimension follows, with improvements of 31.9%, 24.7%, and 15.4% the respective benchmarks, indicating its importance in guiding the agent. The Efficiency (E) dimension also contributes positively, though to lesser extent, while Task Relevance (TR) and Coherence (C) show more modest improvements, consistent with their secondary roles.
yields even higher success rates (34.3%, 35.9%, and 17.2%). demonstrating the cumulative benefits of integrating comple- mentary dimensions. These results reinforce the importance of fine- fineggrined rewards over coarse-grained ones, as models with partial-dimensional rewards consistently underperform compared to the full Similar model.
F. More Visualizations of SRMEyal
E.2. Combined Impact of Multiple Dimensions
As depicted in Table 7 we present additional visualizations of SRMEval. From the data in the table, we can more clearly understand the content SRMEval, which is the first benchmark in the virtual agent domain designed for step-wise, dilei-dimensional,, and platt-- evaluation of reward models. It comprehensively tests the ability of reward models assess the quality of agent actions, well as the degree of preference alignment.
The extended results further highlight the synergistic effects of combining multiple dimensions. For instance, the combi- nation H and OS achieves success rates of 33.4%. 31.4%, and 16.7% on Android World, WebArena, and OSWorld, respectively, outperforming models with only one of these dimensions. Similarly, the combination of H, OS, and E
21

---

A Heep,Wie,, Multi-Dimensional, and Generalist Reward Model with Benchmark for Virtual Agents
Table 7: Cases of SRMEval.
Instruction
Observation
Step Idx
Trajectory
Type
Candidate Action Pair
Input text
In Simple Calendar Pro. delete all the events.
Step Click Search.
Click Calender.
"Simple Calendar Pro".
2
E
Step 1: Click Search.
Step 2: Click Calendar.
In Simple Calendar Pro. delete the calendar events on 2023-10 3023-10-27.
Step 3: Scroll down.
Click Import from an.ics file.
Navigate back.
6
OS
Step 4: Long press 27 Friday.
Step S: Click More options.
Step 1: Click menuitem
Add this exact product to my shopping cart. I think it is in the "Herbs, Spices & Seasonings" category.
"Grocery & Gourmet Food"
Click menuitem "Grocery & Gourmet Food" hasPopup:
Click link "Herbs, Spices Seasonings( 707 item )"
3
hasPopup: menu.
Step Click link "Pantry Staples( 4891 item )"
menu.
Step Click menuitem "Grocery & Gourmet Food" hasPopup: menu.
Click menuitem
Add a exact product my shopping cart.
"Grocery & Gourmet
2
H
Scroll down.
Food"
hasPopup:
menu.
Step 1: Scroll down.
Hover menuitem "Patio, Lawn & Garden" hasPopup: menu.
Step 2: Click
Can you add the red flower seeds with around 4 stars to my cart?
link "Page 2".
Click link "Page Next".
4
Step 3: Click
TR
link "Plants, Seeds & Bulbs( 59
item )"
Step Coordinates
Click on the address and search bar Type the URL access search engine settings.
Click on the
Can you make Bing the main search thingy when I look stuff up on the internet?
for "Customize Chromium". Step Click
"Account Settings" button coordinates for "Account Settings".
Total
on "Customize Chromium".
22


Reviews:
[{"review_id": "68zBOLO0F2", "reviewer": "Reviewer_sojt", "summary": "Traditional training methods for virtual agents depend on outcome supervision and costly human annotations, limiting their scalability. The authors propose Similar, a step-wise multi-dimensional reward model that refines agent training and improves inference-time decision-making. They define five key dimensions for evaluating agent actions and introduce MCTS-P, an automated algorithm for collecting and labeling execution data. Using this data, Similar is trained with the Triple-M strategy and the authors also introduce SRM, the first benchmark designed for step-wise, multi-dimensional reward model training and evaluation, consisting of SRMTrain for training and SRMEval for testing. Experiments demonstrate that Similar provides more granular feedback, enhancing both training efficiency and real-time decision-making for virtual agents.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Claim on no manual annotations.**  \n**A1:** We appreciate to **clarify a misunderstanding**: our claim of \"no manual annotations\" means while utilizing the benchmarks' evaluation scripts (for dimension calculations), our method **annotates step-wise multi-dimensional data without any additional human effort**. **1)** In fact, these scripts are all readily available and involve only simple result-based judgments - neither complex nor labor-intensive, just basic API calls. The human effort required to write these scripts is very limited. Notably, nearly all related works use these scripts. **2)** In contrast, manually annotating the 110k step-wise multi-dimensional data provided by our automated annotation framework would require substantial human effort. Our experiments show even trained annotators can only process ~**25 step-wise annotations/hour**.\n\nFurther, we invited researchers to reproduce these evaluation scripts. We compared their completion time against our estimated time for manually annotating our 110k data:\n\n|Work|Task|Size|Labor Effort|\n|-|-|-|-|\n|WebArena|Scripts Coding|N/A|2.5h|\n|AndroidWorld|Scripts Coding|N/A|2h|\n|OSWorld|Scripts Coding|N/A|4h|\n|**Similar (w/o autonomous annotation)**|Step-wise Multi-dimensional Data Annotation|110k|**4400h**|\n|**Similar**|Step-wise Multi-dimensional Data Annotation|110k|**0h**|\n\nThe table shows the coding time for evaluation scripts is negligible (relative proportion **0.19%**) compared to the manual time saved by our approach.\n\n**Q2: Evaluation in Tables 2 and 3.**  \n**A2:** We rigorously ensured **no data overlap** between training (*SRMTrain*) and evaluation (4 benchmarks) sets by randomly splitting the original test tasks from WebArena(WA), AndroidWorld(AW), and OSWorld into **30% for our training data collection** and **70% for evaluation** (as detailed in Section 4).\n\nWe further validate the fairness of our configuration through **OOD experiments** (in our reply to Reviewer **Evp4's Q3**), demonstrating our model's robust performance on out-of-distribution data.\n\n**Q3: Total number of trajectories.**  \n**A3:** Thank you for your question. We achieved 10k trajectories sampling by **generating multiple distinct actions per step** through task-specific prompt injection and stochastic exploration. For a 5-step task with 5 action variations per step, this theoretically yields $5^5=3125$ unique trajectories (before deduplication). In this way, we easily generated 10k trajectories.\n\n**Q4: Reliability of the test set.**  \n**A4:** To ensure high-quality annotations, we collaborated with a professional commercial data labeling team. The process included: **1) Training Phase**: Annotators underwent three rounds of iterative \"label-review-feedback\" cycles to clarify ambiguities of annotation (e.g., the complexity of UI interaction tasks). Only after achieving >95% accuracy on validation samples did formal annotation begin. **2) Formal Annotation**: Each test sample in *SRMEval* was independently labeled by three annotators and three checkers. The final data in test set required >99% accuracy.\n\nBesides, we find that high-performing RMs on *SRMEval* also boost agent performance across environments. This finding additionally validates the reliability of the test set (*SRMEval*), motivating the following comparative experiments:\n\n|RMs|SRMEval Acc (%)|SR Improvement on AW (%)|SR Improvement on WA (%)|\n|-|-|-|-|\n|Qwen2-VL|10.3|0.4|0.6|\n|Qwen2-VL + Similar-RL|25.7|1.7|5.7|\n|Qwen2-VL + Similar-3M|46.5|3.8|14.3|\n\nAs shown in the table, achieving better performance on *SRMEval* often leads to greater improvement in agent performance.\n\n**Q5: Effect of the five reward dimensions.**  \n**A5:** Thank you for your suggestions regarding the 5-dimension weighting scheme. As shown in Tab. 4, Section 5.5 and Appendix E's ablation study, we found the 5 dimensions' relative importance follows approximately H > OS > E > TR > C. We find these dimensions show similar importance patterns in healthcare, education, and other fields[1].\n\nAligning with your insightful observation, we actually avoided using equal weights and instead **adopted the weighted scoring scheme** presented in Section 3.2: **$5×H + 3×OS + 3×E + TR + C$**, with weights based on empirical analysis. As you nicely suggested, we conducted additional experiments comparing different weighting schemes:\n\n|Method|Android World SR (%)|WebArena SR (%)|\n|-|-|-|\n|Similar + Equal Weights|33.9|35.3|\n|Similar + Dynamic Weights[2]|34.8|**38.4**|\n|**Similar (Fixed Weights)**|**35.4**|38.2|\n\nThe table demonstrates that the dynamic weights strategy also achieves significant improvements, we will explore optimal weighting in future work.\n\n[1] Hattie J. Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. 2008.\n\n[2] Aminul H, et al. Adaptive Weight Assignment Scheme For Multi-task Learning. 2023.", "claims_and_evidence": "From the performance perspective, the argument is well-supported by their proposed process reward modeling methods. However, weaknesses remain in the experimental design and the fairness of evaluation settings, which will be discussed in the following sections.", "methods_and_evaluation_criteria": "The proposed methods and evaluation benchmarks are well-designed, achieving top performance on both the in-house annotated SRMEval and widely used datasets such as OSWorld and WebArena.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "While the research direction is promising with significant potential impact, several weaknesses in the methodology and experimental design should be addressed:  \n\n**Claim on no manual annotations**: The paper asserts that the method requires no manual annotations. However, pruning trajectories and verifying correctness depend on human-annotated evaluation scripts from (Visual)WebArena, AndroidWorld, and OSWorld, which involve substantial human effort. This contradicts the core claim that the process is fully automated.  \n\n**Evaluation in Tables 2 and 3**: The training data appears to be sampled from WebArena, AndroidWorld, and OSWorld, which are benchmarks originally designed for testing. This raises concerns about potential overlap between SRMEval’s training data and downstream evaluation tasks. The authors should clarify the sampling process to ensure fair evaluation.  \n\n**Total number of trajectories**: Given that (Visual)WebArena, AndroidWorld, and OSWorld contain only a few hundred tasks each, it is unclear how the authors sampled 10K trajectories from these benchmarks. Further explanation is needed to validate the dataset scale.  \n\n**Reliability of the test set**: Although human annotators create the test set, their accuracy remains uncertain. Determining the correct UI interactions can be challenging, especially for unfamiliar websites and apps. A more detailed discussion on monitoring annotation quality and potential errors would strengthen the evaluation.", "relation_to_broader_scientific_literature": "Compared to previous works, this paper introduces a novel perspective on process reward modeling by incorporating multiple dimensions, which involves direct and meaningful connection to actual agent performance.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "**Effect of the five reward dimensions**: It would be insightful to analyze whether some reward dimensions influence performance more than others. For instance, should helpfulness weigh more than coherence? Instead of a pure sum, a weighted sum approach could better reflect their relative impact.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ElElaDQLRa", "reviewer": "Reviewer_d34U", "summary": "Previous multimodal LLM-based virtual agents usually require human annotations, multi-dimensional fine-grained process supervision, and scaling inference time. The paper proposes a new step-wise, multi-dimensional generalist reward model to offer fine-grained signals for agent training and can choose better actions for inference-time scaling. Specifically, the paper first provides five dimensions for agent action evaluation, including helpfulness, odds of success, efficiency, task relevance, and coherence. The first three dimensions can be computed automatically, and the other two can be evaluated by MLLMs.  The paper then designs a Monte Carlo Tree Search-Plus to collect excutation data. Finally, the paper employs a multi-step, multi-objective, and multi-modal strategy to train a reward model. Moreover, the paper proposes a new virtual agent benchmark for step-wise, multi-dimensional reward model training and evaluation. The SRM benchmark is based on GPT-4o-1120's agent action on WebArena (WA), VisualWebArena (VWA), Android World (AW), and OSWorld (OW). The paper selects 32k data for manual annotation and as a test set. The paper also proposed a new task focusing on the candidate action of the MLLM agent. The paper tests two baseline models, Qwen2VL and Llama-3.2-Vision-Instruct. The paper analyzes the alignment of preference, RL-training using the proposed dataset, and Inference-time scaling. The paper also includes an ablation study, a case study, and a correlation study.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Benefit of *SRMEval* as a benchmark.**  \n**A1:** Thank you for suggestions on more comparisons with other benchmarks. Section 1 of our paper proposes that ***SRMEval*** is **the first multi-step, multi-dimensional, and multi-platform benchmark** specifically for **evaluating reward models(RMs) in virtual agents**, featuring 32k professionally annotated preference pairs. Our Section 5.2 experiments further demonstrate its challenging nature. We summarize and compare the benefits of our RM:\n\n|Benchmark|Target|Step-wise|Multi-Dimension|Platform|Size|\n|-|-|-|-|-|-|\n|RewardBench[1]|General RM|✗|✗|N/A|3k|\n|RMB[2]|General RM|✗|✗|N/A|18k|\n|WebArena| Virtual Agent|✗|✗|Web|812|\n|OSWorld|Virtual Agent|✗|✗|Linux, Windows|369|\n|***SRMEval***|**RM for Virtual Agent**|**✓**|**✓**|**Web, Android, Linux, Windows**|**32k**|\n\nThrough experimentation, we find these benchmarks cannot evaluate RMs for Virtual Agent, which constitutes *SRMEval*'s benefit. We find that **high-performing RMs on *SRMEval* also boost agent performance across environments**, which also indicates the benefit of *SRMEval*. Relevant experiments are in our reply to Reviewer **sojt's Q4**.\n\n[1] Nathan L, et al. RewardBench: Evaluating Reward Models for Language Modeling. 2024.\n\n[2] Enyu Z, et al. RMB: Comprehensively Benchmarking Reward Models in LLM Alignment. 2024.\n\n**Q2: AC and M.**  \n**A2:** We are sorry for not clearly explaining the definition of our dimensions. And we appreciate the opportunity to clarify these terms:\n- **$ AC $**:  Purely an intermediate computational variable with no standalone semantic meaning. It serves as a mathematical placeholder to recursively track cumulative *Helpfulness* scores during MCTS rollouts.\n- **$M$**: Already defined in Section 3.1 under *Helpfulness*:  \"*M* is the total number of reasoning steps\". \n\nFor a detailed explanation of *Helpfulness*, please refer to our reply to Reviewer **Evp4's Q4**.\n\n**Q3: Human evaluation details.**  \n**A3:** Thank you for suggestions on more human evaluation details. The **human evaluation details** on how to train annotators and ensure data quality can be found in our response to Reviewer **sojt's Q4**.\n\nThe **Human Acceptance in Appendix C** is that, we randomly sampled data from *SRM* and asked trained annotators to verify these data. If the annotator considers a sample is correct, mark it as 1; otherwise, mark it as 0, and calculate Accuracy as Human Acceptance.\n\n**Q4: Sample size variation.**  \n**A4:** The varying sample sizes occur because the **five dimensions inherently contain different amounts of data**. This stems from their fundamental design - for instance, *Task Relevance* and *Coherence* are binary values, which naturally yield fewer possible preference pairs.\n\nThe distribution of dimensions in *SRM* is roughly as follows:\n|Dimension|Size|\n|-|-|\n|Helpfulness|25k|\n|Odds of Success|15k|\n|Efficiency|25k|\n|Task Relevance|5k|\n|Coherence|10k|\n|Total|10k|\n|Traj|20k|\n\nWe think dimensions with small amounts of data indicate weak discriminability and are easily trained for prediction.\n\n**Q5: Why is our model better than GPT-4o in *SRMEval*?**  \n**A5:** We appreciate the opportunity to clarify \"why our model performs better than GPT-4o\": \n\n- **GPT-4o** was only used to provide reliable labels for binary *Task Relevance* and *Coherence* dimensions in SRM construction. For *SRMEval* testing, **GPT-4o served as a zero-shot reward model**. It failed to accurately evaluate the three formula-based dimensions (*Helpfulness/Odds of Success/Efficiency*), causing unreliable predictions.\n- Conversely, our **dedicated reward model (Llama 3.2-V + SimilarRL)** was trained on 78k multi-dimensional preference pairs using our Triple-M strategy. The model employs a purpose-built architecture with gating and regression layers for precise prediction of the benchmark's action pairs.\n\n**Q6: Pretrained MLLM and backbone.**  \n**A6:** We used **Qwen2-VL-7B-Instruct** and **Llama-3.2-11B-Vision-Instruct** as frozen backbones (mentioned in Section 5.1) which are also the pretrained MLLMs.\n\n**Q7: Lack of qualitative analysis?**  \n**A7:** We appreciate this suggestion. But in fact, we have provided qualitative analyses in: **1) Fig. 5**: Illustrates how Similar guides the agent during both the training and testing phases. **2) Appendix D**: Includes 6 visible detailed cases.\n\nAs you nicely suggested, we add textual descriptions of **failure cases qualitative analysis**, which we will continue to explore and improve our dimension weighting strategy in the future: \n- ***Efficiency* Misjudgment**: Prioritized shortcut clicks over explicit menu navigation, even when shortcuts were contextually invalid. \n- ***Coherence* Over-penalizing**: In a \"file backup\" task, our model incorrectly penalized opening a text editor and file explorer simultaneously, though both were necessary. \n\n**Q8: Writing issues.**  \n**A8:** We thank you for the suggestions and will address these writing issues in the next version.", "claims_and_evidence": "1. The paper claims that Triple-M can provide robust fine-grained feedback for the model. Tables 1, 2, and 3 show that the baseline adding triple-M can improve the performance. \n2. The paper also introduces SRM-eval and SRM-train. However, in the experiment, the paper fails to show the benefit of using SRV-eval instead of traditional metrics in section 4. Including similar benchmarks for comparison will strengthen the argument by demonstrating that previous benchmarks are insufficient for fully evaluating the model's capabilities.", "methods_and_evaluation_criteria": "1. The proposed five dimensions for step-wise assessment are not clear. What is the AC in line 207? What is the M here? The paper seems to use human evaluation to judge the quality of the proposed assessment data. However, none of the human evaluation details are provided in Appendix C. Additionally, why are the sample sizes for human evaluation not the same? Given that the dataset SRM is constructed with GPT-4o, why does the knowledge distillation model,  Llama 3.2-V + SimilarRL, perform better than the original GPT-4o?\n2. The paper uses Triple-M Strategy for Reward Model Training. What is the pretrained decoder-only MLLM uised here? is it the backbone of training data?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The paper conducts relatively comprehensive experiments to measure the performance of the proposed method. The paper evaluates the alignment preference in the training stage to analyze the effectiveness of the proposed dataset and training method. However, the whole paragraph seems to only draw conclusions from Table 1, with two findings supported by Table statistics. Adding additional qualitative analysis can strengthen this paragraph. Similarly, for section 5.3, the authors can remove the contents already shown in the paper and move some of the analysis to the appendix here.", "relation_to_broader_scientific_literature": "The paper proposes a new interesting multi-dimensional model with a five-dimensional reward to provide better feedback. The paper also adopts the multi-step, multi-objective, and multi-modal to train the model. The paper finally release a new dataset called SRM.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Pros:\n* The visualization, examples, and tables are clear to readers and easy to understand. \n\nCons:\n* The paper is not well written and contains a lot of abbreviations without proper definitions. The abstract contains too many abbreviations, such as MCTS-P and Triple-M, which are not defined until the introduction section. The MCTS-P is really confusing since only the full name of MCTS exists in the paper. The paper should not assume all the readers will check the paper carefully. Additionally, the paper sometimes uses Triple-M and sometimes uses X+3M, which is very confusing.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UTMwrxBbVs", "reviewer": "Reviewer_Evp4", "summary": "The paper proposes Similar, a Step-wise, Multi-dimensional Generalist Reward Model, designed to improve the training and inference of Generalist Virtual Agents (GVAs). Similar addresses limitations in outcome-based reward models by introducing a process-based system that provides fine-grained supervision signals. The authors define five dimensions to assess agent actions: Helpfulness, Odds of Success, Efficiency, Task Relevance, and Coherence. They develop an MCTS-P algorithm to automatically annotate agent trajectories across four environments (Web, Android, Linux, and Windows) and train Similar using the Triple-M strategy (multi-step, multi-objective, and multi-modal). Extensive experiments show Similar significantly boosts GVA learning and inference-time scaling.\n\n## update after rebuttal\n\nThank you to the authors for the detailed response. I am satisfied with the rebuttal and would like to maintain my score after the response.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Compared with more process reward models (PRMs) and outcome reward models (ORMs).**  \n**A1:** Thank you for suggestions on more comparisons with more RMs, To our knowledge, our **Similar** is the first step-wise, multi-dimensional, cross-platform PRM for *virtual agents (VA)*. **PAVs[1]** is a mathematical reasoning PRM that uses advantages as a single-dimensional reward. **Math-shepherd[2]** is a PRM for mathematical reasoning that uses soft estimation as a single-dimensional reward. Through experimentation, we found **these RMs are not applicable** as RMs for our proposed multi-dimensional, cross-platform VA tasks.\n\nFollowing your nice advice, we try to apply these RMs in VA tasks in following comparative experiments:\n|RM|Android World(AW) SR (%)|WebArena(WA) SR (%)|\n|-|-|-|\n|General ORM[3]|31.0|21.4|\n|Math-Shepherd[2]|31.5|22.9|\n|PAVs[1]|31.3|23.8|\n|**Similar**|**35.4**|**38.2**|\n\nThe table shows that our model outperforms other RMs, likely due to its multi-dimensional scoring designed for VA tasks. In contrast, these RMs only provide either outcome rewards or single-dimensional process rewards, making them unsuitable for VA tasks.\n\n[1] Setlur A, et al. Rewarding progress: Scaling automated process verifiers for llm reasoning. 2024.\n\n[2] Wang P, et al. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. 2023.\n\n[3] Hunter L, et al. Let's Verify Step by Step. 2023.\n\n**Q2: Significance of gating network.**  \n**A2:** The gating network (GN) **balances the 5 dimensions, addressing multi-objective optimization challenges**. For example, *Task Relevance* may dominate in instruction-following tasks, while *Efficiency* matters more in time-sensitive scenarios. The GN is not the focus of our research, but we will consider it in the future. The ablation experiments we have added show that removing the GN degrades performance, validating its necessity:\n- **Similar (w/o GN)**: AW SR = 33.7%, WA SR = 34.1%\n- **Similar (with GN)**: AW SR = **35.4%**, WA SR = **38.2%**\n\n**Q3: 4 benchmarks and more OOD experiments.**  \n**A3:** Thank you for your suggestions on more OOD experiments. 4 benchmarks (WebArena, VisualWebArena, Android World, OSWorld) mentioned in our paper are **widely recognized** and cover the major **real-world online interactive environments** of Web, Android, Linux, and Windows, as well as real-world scenarios such as web navigation and app usage. Thus, we believe that these benchmarks can already reflect the situation of real scenes.\n\nFurther, as you nicely suggested, we supplement 3 **OOD tests**: Game of 24 (a mathematical puzzle unrelated to virtual agents), ScienceWorld (embodied science experiments), and 30 real-world Android tasks we designed (e.g., ordering bubble tea). Experimental results are shown below:\n|Benchmark|Platform|Baseline SR (%)|Similar SR (%)|\n|-|-|-|-|\n|Game of 24|Virtual Game|1.0|**3.0**|\n|ScienceWorld[4]|Embodied Science|unseen: 18.7|**unseen: 20.1**|\n|Real Android Device (designed by ourselves)|Physical Phone|16.7|**46.7**|\n\nThese results confirm our model’s scalability beyond the original benchmarks. We will include full details in the next version.\n\n[4] Ruoyao W, et al. ScienceWorld: Is your Agent Smarter than a 5th Grader? 2022.\n\n**Q4: *Helpfulness*.**  \n**A4:** We are sorry for not clearly explaining this dimension's definition. The ***Helpfulness* quantifies each step's contribution to task completion** by assigning values inversely proportional to the trajectory length. For example, each step in a 3-step successful trajectory is worth 1/3, while steps hindering progress (those failing to lead to success) receive corresponding negative values. To account for the compensatory relationship between helpful and harmful steps, we introduce an intermediate accumulator (AC) to track each step's cumulative influence throughout the trajectory.\n\n**Q5: Other inference-time scaling methods.**  \n**A5:** Our model can seamlessly integrate with methods like majority voting and beam search. For example, we apply beam search as follows:  1) At each step, Similar scores k=5 candidate actions. 2) The top k=2 paths are expanded further, guided by our rewards. Additional experiments demonstrate this flexibility:\n\n- **Similar + Beam Search (k=5)**: AW SR = 34.1%, WA SR = 36.1%\n- **Similar + Majority Voting (N=10)**: AW SR = 34.3%, WA SR = 35.6%\n\n**Q6: Computational costs in data annotation and training.**  \n**A6:**\n\n**Annotation**: The 5 dimensions can directly derive from a single MCTS process. While MCTS introduces computational costs, we reduced them via pruning and limiting the number of rollouts (e.g., N=8).\n- Generating 10k trajectories required ~15,000 GPT-4o API calls, and the total annotation time is ~80 hours\n- Parallel MCTS rollouts with pruning reduced per-task latency by ~30%\n\n**Training**: The model is trained on preference pairs using lightweight regression with Bradley-Terry loss.\n- Training Similar on 78k preference pairs took ~100 GPU hours", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "There is not theoretical claim in this submission.", "experimental_designs_or_analyses": "Yes. The experiments are conducted in multiple GVA scenarios with detailed ablations.", "relation_to_broader_scientific_literature": "The paper connects well to literature on reward models, GVA training, and MCTS-based data collection.", "essential_references_not_discussed": "I think that prior works[1,2] also explore process reward models, which should be discussed.\n\n[1] Setlur A, Nagpal C, Fisch A, et al. Rewarding progress: Scaling automated process verifiers for llm reasoning[J]. arXiv preprint arXiv:2410.08146, 2024.\n[2] Wang P, Li L, Shao Z, et al. Math-shepherd: Verify and reinforce llms step-by-step without human annotations[J]. arXiv preprint arXiv:2312.08935, 2023.", "other_strengths_and_weaknesses": "Strengths:\n--\n\n1. The paper is well-structured and the idea is well-motivated.\n2. Implementation details and prompt designs are explained in clear and exhaustive detail.\n3. The experimental results are impressive.\n\n\nWeaknesses:\n--\n\n1. I do not understand the significance of the gating network, and there is no ablation study to evaluate it.\n2. As mentioned in the manuscript, the proposed method is trained and evaluated on only four benchmarks, making it difficult to assess its scalability to real-world scenarios. Evaluating the trained model on OOD or real-world tasks would be highly beneficial.\n3. I am confused about the definition of the Helpfulness metric. Detailed explanations and examples would be greatly appreciated.\n4. There is no comparison with the outcome reward model or other works focused on the process reward model, which should be added or the rationale discussed.", "other_comments_or_suggestions": "Minor: Highlighting relative improvements over the backbone may make them clearer in the tables.", "questions_for_authors": "1. How about the computational costs about the data annotation and training.\n2. Could Similar be combined with other inference-time scaling methods, like majority voting and beam search.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for training multimodal virtual agents, using reward models around 5 dimensions for training and evaluation. The approach is unique and somewhat timely given the difficulty of both training and evaluating virtual agents, but more importantly permits using LLMs in various ways to provide a variety of signal to the virtual agent. The benchmarks are sound and cover both well-known problems as well as OOD generalization.

There were various concerns about clarity which have been well resolved in the discussion. I generally found the questions from reviewers to be unconfused, meaning the core ideas of the paper were clear and there were no significant issues on support for the central message. All in all, I found the responses to be sufficient to address a large majority of the concerns.

I therefore recommend the paper is accepted to the conference.
Decision: Accept (poster)