MUG: Interactive Multimodal Grounding on User Interfaces
Authors: Tao Li
Gang Li
Jingjie Zheng
Purple Wang
Yang Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multimodal Grounding
UI
Mobile
Interaction

Abstract: We present MUG, a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen. Prior works modeled multimodal UI grounding in one round: the user gives a command and the agent responds to the command. Yet, in a realistic scenario, a user command can be ambiguous when the target action is inherently difficult to articulate in natural language. MUG allows multiple rounds of interactions such that upon seeing the agent responses, the user can give further commands for the agent to refine or even correct its actions. Such interaction is critical for improving grounding performances in real-world use cases. To investigate the problem, we create a new dataset that consists of 77,820 sequences of human user-agent interaction on mobile interfaces in which 20% involves multiple rounds of interactions. To establish our benchmark, we experiment with a range of modeling variants and evaluation strategies, including both offline and online evaluation—the online strategy consists of both human evaluation and automatic with simulators. Our experiments show that allowing iterative interaction significantly improves the absolute task completion by 18% over the entire test dataset and 31% over the challenging subset. Our results lay the foundation for further investigation of the problem.

Under review as a conference paper at ICLR 2023
MUG: INTERACTIVE MULTIMODAL GROUNDING ON USER INTERFACES
Anonymous authors Paper under double-blind review
ABSTRACT
We present MUG, novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface sereen. Prior works modeled multimodal UI grounding in one round: the user gives a command and the agent responds to the command. Yet, in realistic scenario, a user command can be ambiguous when the target action is inherently difficult to articulate in natural language. MUG allows multiple rounds of interactions such that upon seeing the agent responses, the user give further commands for the agent to refine even correct its actions. Such interaction is critical for improving grounding per- formances in real-world dataset that consists of 77, 820 sequences of human user-agent interaction on mo- bile interfaces in which 20% involves multiple rounds of interactions. To establish our benchmark, we experiment with a range of modeling variants and evaluation strategies, including both offline and online evaluation-Ate online strategy con- sists of both human evaluation and automatic with simulators. Our experiments show that allowing iterative interaction significantly improves the absolute task completion by 18% over the entire test set and 31 % over the challenging subset. Our results lay the foundation for further investigation of the problem.
cases. To investigate the problem,
create new
INTRODUCTION
Natural language understanding on graphical user interfaces (GUIs) is crucial for realizing human- computer interaction and assisting scenarios that have accessibility difficulties (Sarsenbayeval 2018)) Specifically, interpreting user commands into executable actions has drawn increasing interests as it manifests rich research problems including multimodal modeling and natural language ground-
ing (e.g.. Li ct al. 2017: Gur et al. 2019; He et al., 2020 Li et al. 2020a| 2021 Prior works often consider Ol grounding in a single-pass fashion where the model predicts actions with given instruction without looking backward to refine prediction. However, in a realistic scenario, user instructions be ambiguous inaccurate especially when the target action is difficult or incon- venient to articulate. Reasoning in such cases is inherently iterative. Therefore, is important and beneficial to incorporate interaction for resilient grounding (Suhr et al. 2019 Chandu et al. 2021)
In this paper, we investigate interactive grounding on GUTs, which aligns multimodal input to ac- tionable objects of screen. We focus on single-screen interaction which is the building block of UI reasoning. Specifically, we introduce the MUG (Multi-turn UI Grounding) task in which the user iteratively guides the agent select a desired UI object (see Fig. DD. With a given UI and target object, the user instructs the agent via natural language, ranging from casual intent to more descrip- tive commands. The agent infers which UI object intended by the user and and highlights it. If the agent is correct, the user can confirm the selection and the grounding is completed. Otherwise, the user issues further guidance, e.g., "Click the one below". to the agent to refine its selection. We col- lecte the MUG dataset from live interaction sessions between pairs of human annotators one acts as the user and the other as the agent. Our dataset has 77, 820 examples, each records the transaction history in session. Specially, 20% of the dataset challenging ones their human commands need multiple rounds to ground, even for human agents.
To establish the benchmark, we experiment with range of variants to model the dynamics between the two roles. While the main goal of the task is to develop agent models for grounding, we also develop the models for online instruction simulation. We build our models upon a Transformer-

---

Under review as a conference paper at ICLR 2023
bar yalbos
pelice
right
Meerage
Gocer sotee
Selpoond
Talppuud
Shan
Pestages
Onalinete Port comnertee
(a) Example one.
(b) Example two.
Figure 1: Two illustrative examples of the MUG task. There are two turns in each of these examples. Interac- tions happen within single screen. User commands shown above the screens. The target object is bounded Agent choices are marked with
based encoder-decoder architecture (Li al. 2021), and experiment with various learning methods, including traditional sequence modeling and reinforcement learning. To fully examine the model performances, evaluate the agent model with spectrum of evaluation strategies, including both offline and online evaluations. For the online evaluation, we employ both automatic and human evaluations, which include interactions between the agent and the (either human or the user model) and offer a comprehensive probe into model understanding. Our experiments show that incorporating interaction substantially improves UI grounding task completion by 18% on the en- tire dataset and 31% on the challenging set, both in absolute scales. Furthermore, our robustness measurements suggest MUG, while being seemingly easy single-screen task, is actually difficult since neural agents sometimes struggle to correct themselves, resulting in repeated wrong selections across multiple turns. This suggests large rooms for future improvement grounding agents.
In summary, key contributions are:
1. We introduce MUG, a novel interactive vingun-language task that focuses multi-turn language grounding on graphical UI screen, which is a challenging task that is meant to improve language grounding in realistic Uls.
2. We create a rich dataset that includes 77,820 examples recorded from live sessions between pairs of human users and agents. And 20% of the data are challenging for both human annotators and neural agents.
3. We experiment with range of model variants and evaluation strategics, showing that iter- ative interaction significantly improves grounding accuracy by 18% and 31% on the entire and challenging test sets respectively, with automatic assistance from models. Our work lays a solid foundation for future investigations on collaborative grounding.
2 BACKGROUND
modeling has long history of research (c.g.. Winograd 1972; Barnard & Forsyth
2001 Lavrenko et al. 2003 Plummer et al. 20155) Yu et al. 2016). One important area focuses on
grounding objects in images where the natural language is used as additional input (Chen et al.
2017
Yu et al. 2016 2018 Fukui et al. 2016 Deng et al. 2021
Interactive Multimodal Grounding Prior works have formulated grounding as a multi-step rea-
soning task, e.g., navigation via multiple steps of grounding (c.g., "u.,Kuu al. 2020 Gur et al. 2019). Our work differs by focusing on agent's 's ability to self-correct in synchronized turns of interaction
'The dataset and code for reproducing experiments http::/giitbbbccommoooooddaaaamm
2

---

Under review as a conference paper at ICLR 2023
on UI screen. It is also conceptually linked to repeated reference game (Hawkins et al. 2020 except we use a different form of communication (language-action instead of dialogue (language- language). Our task leverages iteratively refined instructions creased instruction utility over multi-step actions (Effenberger et al. (021). Our work models both the user and the agent, and let them communicate online. This is different from single-sided model- ings (Suhr et al. 2019; Kojima et al. 20221) Our observation that interaction improves grounding is
atomic action instead of the in-
also in line with dialogue-basee works (e.g.. Haber et al. 2019 Takmaz et al. 2020).
UT Grounding Grounding UI objects involves automatic completion of actions on web or mobile also an important ac- cessibility task for users who are situationally impaired when they are occupied by real-world tasks at hand (Sarsenbayeval 2018). Compared to grounding on natural images, these tasks usually take well-specified user commands and aim to select the object that best matches the command. The UI image is often encoded via ResNet (He et al. 2016) or ViT (Dosovitskiy et al.
interfaces (e.g. Pasupat et at. 2018 [i et al. 2020a He et al. 2020).
20200. The structure
and text features of UI are often encoded by Transformer model (Vaswani et al. 2017). Fusing mul-
timodal information is widely handled by cross-attention (e.g. He et al. 2020 Li et al.
2021 Bai
al. 202T
We adopt these neural components in our benchmark.
Mobile UI Datasets Many grounding tasks, while covering multiple screens, remain one-pass reasoning, such as PIXELHELP (Li al., 2020a) and MOTIF (Burns al. 2022). Prior works (e.g., Todi et al. (021)) model sequence of (action, state) pairs via reinforcement learning (RL). In contrast, MUG focuses on correcting a single action on one screen. Tab. summarizes key differences among other Mobile UI datasets. Importantly, MUG is a challenging task as it enables corrective interaction in synchronized turn between user and agent.
Data
Screen multi multi multi single single
Instruction
Natural Instruction
Interactive Correction
RicO (Deka et al. 2017)
FIXEHHELP
al. 2020a)
MOTIF (Burns el 2022
RICOSCA (L. al.
2020a
X
REFEXP BBa et al.
2021)
MUG (Ours)
single
Table Comparison to prior mobile UT Datasets.
3 TASK FORMULATIONS
As grounding task, MUG involves two participants: a user and an agent. Our formulation includes both roles to provide a holistic view of interactive grounding. The user's goal is instruct, via natural language, the agent to select the desired object on the UI screen S. The unique aspect of MUG that it allows the user to guide the agent iteratively identify the target action by issuing series of commands, each in response to the agent's 's prior inference.
We separate such user-agent interaction into turns. At turn f, the interaction consists of:
0 user command,
0r agent action.
where the user first instructs the agent with command ce, and the agent responds with a suggestion of action ar Here a, is essentially the index of object. The task is completed when ae = g.
3.1 AGENT TASK
In MUG, the action space for the agent consists of a set of UI objects to click on the interface, e.g.. in Fig Intuitively, we would want the agent to take the desired action y as early as possible, i.e., as few turns as possible. Thus, at turn t, the agent models
Pe Pa cjo,j (
(I)
3

---

Under review as a conference paper at ICLR 2023
where 9 denotes the agent parameters, This iterative grounding early stops once a, 9 reaches a maximum number turns allowed.
3.2
USER TASK
The user's role is to provide guidance to the agent through iteratively refined instructions. In contrast to one-pass prediction tasks (e.g. Pasupat et al. 2018 He al., (2000) where the agent makes one- shot guess, a MUG user issues fo commands that are dependent of prior instructions Cjo,t-1] and agent actions djo which is formalized the following:
Po(ce[S, 9, C[o,t-1): (10.-1))
where & denotes the user. Here, the user model is aware of the target object g.
Interplay between User and Agent The agent task (Eq. is the pivot of MUG. The user task (Fq.2) aims to guide agent towards task completion, which potentially includes online training. In benchmark, we let the user and agent play together. Although automatic evaluation is not as realistic as human evaluation, it offers fast, low-cost, and reproducible environment. This setting also allows us to study various questions surrounding the interplay between the two, e.g., whether an automatic user can assist an agent? and whether agent errors would confuse the user?
4 DATASET CREATION
As there is no available dataset for model training and evaluation, we developed an interactive la- beling interface to collect data for MUG. Our data collection involves two human annotaters to play the roles of the user and the agent respectively in live session. The user and the agent have two separate views, running on different machines (Appx. A). Both views share the same UI screen and message box showing instruction history. Our task embodies the eyes-on, hands-free situation for mobile interaction where the user is required to only use language for the task, and the machine re- sponds its prediction by highlighting. The user can commit the action if the prediction is confirmed. During a session, only the user can see the target; and the the message box is read-only to the agent thus no language-bast base dialogue would take place.
4.1 ANNOTATION WORKFLOW
We use the UI corpus, mobile UI screenshots and view hierarchies, from RICO (Deka et 2017) and auxiliary object features from the CLAY dataset (Li et al., 2022). Each session starts with randomly sampled UI object (e.g., button), from the visible view hierarchy, as the target object User annotators are encouraged to articulate their initial command (cp) casually or goal-oriented.
We consider such design cover the realistic scenarios discussed in Sec. T and potentially free users from composing long and precise instructions.
In the agent view, all clickable objects on the UI screen are revealed with their bounding boxes highlighted, which show what objects the agent can select, without indicating which one is the target g. The current agent selection reflected on both the user and the agent's view. The session continues to the user's turn if the agent selection does not match g. In follow-up turns, the user is not allowed to repeat a command issued in previous turns, and likewise the agent is not allowed to select previously chosen object. Upon the agent selection matching the target object in the user view the task is completed. Each session allows up to 5 turns and we filter out those unfinished.
4.2 RESULTS& & ANALYSIS
We collected 77,820 examples based on 31,265 unique screens from 7,132 apps (see details in Table 2). We split the dataset into the training, development, and test sets. We use app-wise split (Li et al. 2020b) to avoid potential leaking across sets. As shown in Table 2 the three splits have a similar lar distribution regarding the number of turns in each example.
In general, users tend to provide short instructions (~ 4 words). Across the dataset, annotators completed ~ 80% examples in one turn ( i.e., one instruction and one selection ). The rest 20%
4

---

Under review as a conference paper at ICLR 2023
Stutistics of examples
Distribution of Turns (%)
Split
Apps
Screens
Interactions
Avg. #Turns
Avg. #Token/Turn
2
4
5
Train
6,039
26,090
65,235
1.24
4.26
78.91
18.31
2.37
0.35
0.06
Dev
544
2,625
6,377
1.23
4.18
79.99
17.77
1.91
0.27
0.06
Test
549
2,550
6,208
1.23
4.18
80.20
16.82
2.55
0.40
0.03
All
7.132
31,265
77,820
1.24
4.25
79.10
18.15
2.35
0.35
0.06
Table 2: Dataset statistics.
dataset is considered to be more challenging, with the majority taking 3 turns. We call this 20% as the Challenging subset. In Sec. 6 we will show that examples requiring more turns for human agents are also more difficult for grounding models. We refer readers Appx. forr examples.
We should note that while the 20% multi-turn ratio seems low percentage but it leads to large impact in practice. Firstly, lines up an upper bound for agent ground accuracy (i.e., 80%). More- over, real-world grounding sessions can also span multiple screens. If we apply the same 20% rate here, the probability for a multi-screen session to require correction much more significant. For instance, 67% with 5 screens. Lastly, we will see in Sec. that neural agents could also struggle instructions that are trivial for human to exccute.
In Table we categorize 200 Challenging examples from the development split. We found follow- up commands are mainly to make spatial adjustments (50% + 10%) or to add extra information.
Percentage
Attribution
Example
50%
Adjusting relative position the layout.
the value before the text. show me channels. - click icon. not reward but collect the bottom. to books. show me books logo.
31%
Providing more information the target.
10%
Adjusting direction/position the screen.
3%
Rephrasing the instruction.
Table 3: Major categories for the second from 200 examples in the development split.
GROUNDING MODELS
We aim to have general architecture for the UI domain and explore its variants model multi-turn interaction. Our agent model is based on transformer encoder-decoder network, inspired by (Li al. 202T). The encoder comprehends the objects on the screen (Sec. 5.1 and the decoder predicts a target action based on the UI and interaction history (Sec. 5.2)
5.1
UI ENCODER
Our encoder processes the interface S. Each S consists of two modalities information, i.e., a screenshot [s and view hierarchy features & (Deka al. [007]] Li et al. (002). The concrete list of d is in Appx.
D The output is an encoding o for each object indexed by k, similar to (e.g.. Li
al. 2020a
He al. 2020
Li al. 2020b):
C = ResNet(Ts)
(3)
Tene ((RO( (()))
(4)
For the image, we use a pre-trained ResNet-50 (He al. 2016) which is fine-tuned with other mod- ules. The resulted &g (grid size of h x w) is then mapped to object level by region-of-interen. (ROI) pooling (Ren et al., 2015). Finally, we fuse multimodal features for each object by transformer encoder Te The output stands for sequence of object encodings which are -agnnttioon.nossticc..
5.2 GROUNDING DECODER
We use a causal transformer Tase to predict from interaction history. The architecture is similar to (Li al. 2021), except multi-tum interaction input, e.g. Clo.j and ajo,r-1]- The output of Taee

---

Under review as a conference paper at ICLR 2023
vector 2 that summarizes prior interaction up ce:
(5) where d, denotes object index, either from model prediction or human selection. The specific input to Eq. will be subject to modeling variants in Sec 6.1 For classification, we use linear layer f to score the k-th object:
2 Tee (U, co. v cr, 2 c.).)
0r arg max [(z)(("))
EXPERIMENTS
The goal of our experiments is to explore training and evaluation methods for MUG and establish benchmark. Simply matching the initial instruction and object text insufficient as text features are often incomplete. For instance, the validation split has 46% objects missing text, and deterministic classifier using METEOR (Banerjee & Lavie 2005) has only 21% Fl. this section, we present setups for the neural architecture discussed in Sec. 5 In Sec. 6.1 explore multiple variants for the agent. For automatic evaluation, we present simple and effective heuristics- based user model and neural version in Sec. 6.22. Finally, we show extensive F1 results in Sec. Sec.66.4| and 6.5 robustness in 6666, ablations in 6.7 and 6.8 and error analysis in 6.9 We refer readers to Appx. for hyperparameters, Appx. H for sample predictions, and Appx. G for extended discussion.
To avoid confusion, we thereafter use at to refer to the selection predicted by the agent model at turn while o, to the human agent's selection. Similarly, refer ef instruction generated by the user model while to the one from the human user.
6.1 AGENT MODELS
Our agent models use the Tene and Tike (in Sec. 5) as a backbone, denoted as 9. Recall that Tenc processes S while Taee processes interaction. Here, we discuss different handlings of Ta
Single or Multi-turn Model The first factor we investigate is how allowing multiple turns helps grounding. For each example, we feed the entire interaction history as input to the agent model and supervise agent selection on the very last turn T:
Po(a- = g/8, cjorr] 90,r-11)
We can further reduce the input be (S, cg) only, making single-turn model. To evaluate single- turn model with multi-turn examples, we simply concatenate all into one instruction.
Instruction-only Model To understand how helps grounding by taking into account of previous actions of the agent in the multi-turn model (Eq. 7) we introduce the command-only baseline, which ignores agent actions (selections) in the interaction history:
Po g g(
(8)
Imitation Model Instead of supervising the agent only at the last turn, model the entire action sequence as an imitation model:
II, Pa(a) q0.4) "po.-11)
qo.4):
This variant investigates whether the supervision of the intermediate actions helps.
Offline RI. Lastly, because each turn the agent action affects how the user responds, MUG can be formulated as RL problem where the user and the UI constitute the environment. We use the
Decision Transformer (Chen et al. (02I)) for offline RL, by inserting extra learnable return tokens we to the Tace before each action: Tace Co, co, Wo, 8 Ct, w.). The model is:
II, Pa ar co..j. Wo.d] ajo.-11)
(10)
Possible discrete return tokens are {1, 3, 4] where on the last turn. During testing, we fol- low Chen et al. (202) to force the current turn have return and adjust prior returns.

---

Under review as a conference paper at ICLR 2023
Challenging
All
Model
Flo
Fl,
Fl,
Fl,
Fla
Fla
F1,
F1
Fly
Fl,
avgas
Single
26.8
44.7
45.6
45.7
45.7
46.1
56.9
60.5
60.7
60.7
60.7
60.3o.s
Ins-only
25.2
49.7
52.1
52.2
52.2
53.5
58.5
63.4
63.8
63.8
63.9
64.06.5
Multi
25.2
54.2
57.2
57.4
57.4
59.9
58.6
64.3
64,9
64,9
64,9
65.1o.2
Imitation
23.5
59.6
59.6
59.6
59.4..5
56.6
63.1
63.7
63.7
64.0p.8
Offline RI.
24.2
55.4
58.1
58.2
58.2
58. 11.1
58.0
64.2
64.7
64.8
64.8
65.1o.s
Table 4: Offline agent Fl the Flo4 from model trained with seed and avgas is Fl2 of
6.2
USER MODELS
Here, we design a simple and effective heuristics- -based user model, and then develop a neural ver- sion. To show automatic online evaluation is promising direction for MUG, we also conducted human evaluation on a shared set of 500 examples from the test split (Sec. 6.7
Heuristics-based Model We observe that, when the selection a' is incorrect, we can deterministi- cally devise a follow-up instruction by using a template as below:
Not the a/. click the y to/on the dir.
This template is to be instantiated on view hierarchy features (in Appx. D). Compared to human follow-ups, heuristic ones are more specific and longer, such as:
Not the icon, click the action notifications the top right of the screen.
Not the text, click the input search to the slight right and below of your choice.
Neural Instruction Model
We extend the Multi agent architecture to model follow-up commands: Polc c S g, C[o,a-1], ((j,t-1))
(11)
which has decoder Tdee v9, co. , CI...., - at turn t. For training, we teacher-force at each
turn 0). We found that using heuristics prompt greatly boosts development CIDEr /Vedantam
et al. 2015 to from 70 to 78. For inference, we use greedy decoding with a maximum length of 12.
6.3 METRICS
We focus on evaluating the agent model it is the pivot task of MUG. Intuitively, would want the agent to take the desired action with as few turns as possible. That is,
F1, = Σ, P(a, = g C(o,r): (0o,r1))
(12)
where, in practice, we compute F1, with early stop turns to avoid double counting. Clearly, an agent with high Fl and a lower value of is better than an agent that requires more turns for the same accuracy. With limited to 0, the task is reduced a one-pass grounding task.
In an extreme case, we consider an agent with high Fl, but flat changes in Fl, > to be problematic, since it questions the agent's understanding about the interface. For more comprehensive testing, we also use a simple robustness metric for prediction changes across turns:
r p(()) # T)
(13)
which is the percentage of examples that have duplicate actions within T valid turns. We observe that user models, unlike human annotators, sometimes give the same instruction across turns, especially when the agent repeats the same error. Therefore, we only count agent selections that are associated with unique instructions in each example, and ignore other turss.
6.4 OFFLINE RESULTS
Tab. presents offline results on the test set, over the Challenging (see Sec. 4 and the All sets. During inference, we use instructions from the human user and actions from the human agent for turss in between and ask an agent model to predict at each turn. Doing so requires agent models to correct human agent actions, instead of the model's 's Clearly, the models that take into account interaction history outperform those use none or partially. While the Ins-only and the Imitation models perform closely on the All set, they bear larger margins the Challenging and online tests.
7

---

Under review as a conference paper at ICLR 2023
Heuristics
Neural
Model
Fla
Fli
Fla
Fl
Fl,
avgel
Fla
Fl
Fl,
Fls
F1,
avguc
Challenging
Single
26.8
39.8
43.3
44.6
44.6
44.
26.8
41.7
43.9
44.6
45.2
44.9..
Ins-only
25.2
47.4
51.7
52.9
53.5
52.91.4
25.2
43.4
46,5
48.2
48.5
49.1
Multi
25.2
47.8
50.9
51.7
52.4
54.3
25.2
43.9
47.4
48.9
49,4
50.01.1
Imitation
23.5
39.8
43.3
46.8
48.1
55.20.4
23.5
44.1
51,4
55.5
57.6
57.71.s
Offline RI.
24.2
47.6
52.7
54.1
54.6
54.61.2
24.2
44.6
49.4
51.3
52.0
53.41.3
Single
56,9
65.2
67.4
68.1
68.1
68.70.8
56.9
65.0
66.5
67.0
67.4
67.1u.s
Ins-only
58.5
70.9
72.9
73.6
74.0
73.50.4
58.5
67.8
69.9
70.9
71.3
70.9b.a
Multi
58.6
71.7
72.9
73.3
73.6
74.20.5
58.6
67.9
69.8
70.6
70.8
71.1
Imitation
56.6
69.1
72.4
73.5
74.60.5
56.6
68.7
72.6
74.4
75.5
75.40.s
Offline RI.
58.0
71.6
74.0
74.7
75.0
74.60.6
58.0
68.4
71.2
72.2
72.7
73.30.5
Table 5: Online agent F1 on the test set. Fl from model trained with seed and avgad is F1,, of
6.5
ONLINE RESULTS
Tab. presents online test scores. In general, models that are supervised by action sequences (i.e., Imitation and Offline RL) perform better. Both heuristics-based and neural user models are able to guide agents towards task completion. Comparing Single's 's Fl, and Imitation's 's Fl4, we that properly using interaction boosts task completion by 18 and 31 on the Challenging and All test sets.
The average FL, show that heuristics-based user works better, except that the Imitation collaborates better with the neural user. This might be attributed to the neural user is trained to mimic human command patterns which can be ambiguous and short, while heuristics are more precise while being artificial. This also implies that a large room for further improvement to the user modeling.
Overall, we can see interactive grounding is a challenging task, even on a single screen. The agent modeling involves robust multimodal understanding to self-correct. The user modeling requires controlled language generation, which still an open problem. The best task completion rate on the Challenging subset is only ~ 55%, suggesting large room for future improvements.
6.6 AGENT ROBUSTNESS
We take a deeper look at agent behavior in Tab. 6 We observe that agents with higher F1 tend to be more robust (lower r). The best agent model (Imitation) repeats the same mistake for only 16.8% on the All test set. However, if we ignore those examples finished in turn i.e., 7 >> columns, the repeating rate rises the Neural user, has a mixed robustness impact on the Imitation and Offline RL agents. On weaker agents (the first rows), the Heuristics user leads to more salient robustness. These observations suggest improving agent F1 has a more direct and positive impact robustness.
40%. The Heuristics user, while generally improves agent F1 more than
Challenging
All
Challenging (T>1)
(TS1)
Heuristics
Neural
Heuristics
Neural
Heuristics
Neural
Heuristics
Neural
Single
44.4a.
44.9,
25.8
26.90.3
60.3o.
61.0i.
59.1o.a
61.70.5
Ins-only
37.91.4
40.5..0
21.2a.4
23.4o.5
51.41.
55.0n.
51.2o.s
56.40.9
Multi
38.31.3
41.3
21.3p.3
23.8o.5
51.51.8
$5.61.4
51.3o.s
57.9..
Imitation
31.0,
28,3,
17.6..3
16.8.
40.7
37.2.s
40.7o.s
38.9,
Offline RL
36.41.
35.50
19.9a.a
20.5o.5
48.61.
47.4.
48.0o.:
49.5g.s
Table 6: Agent I' 1. on the test split. Results are from 5 random runs. Smaller I' means more robust.
6.7 AUTOMATIC EVALUATION V.S. HUMAN EVALUATION
To show automatic online test is a promising surrogate for human-in- numan-in-the-loog evaluation, we com- pare Single with Mult with group of human annotators (acting the user) (Tab. 77.. We ask the user annotators to follow the same annotation interface and guideline in Sec. 4 and Tet them to use the trained agent model to ground their commands. That is, human plays the user role and trained
We choose these two models as pilot study since they perform consistently different in all our metries.
8

---

Under review as a conference paper at ICLR 2023
agent model plays the agent role. This setting maximally mimics realistic situation where human user guides the agent to locate a target solely using language commands. The results (Tab. (7) are generally consistent with those from the automatic evaluation (Tab. [5) We should also note that such human study is not meant to reflect every minor differences in automatic evaluations.
Model
Flo
F1,
F12
Fla
F14
r4
Single Multi
50.0
56.4
58.2
58.4
59.4
42.6
60.4
62.2
62.6
39.4
Table Human- teee-ii--tthteeee the-loop evaluation on examples from the All test set. Models are trained with seed
6.8 ABLATION ON HEURISTICS
To show agent improves from follow-up instructions effectively, instead of overfitting potential arti- facts the dataset, we report our ablation studies in Tab. Specifically, we focus on the heuristics- based user since it offers well-controlled instruction generation. We underperform by ~ 14% and repeating the initial instruction is even worse. The scores also suggest that randomly instantiated instructions are less effective in guiding the agent.
that random heuristics
Multi
Flo
Fli
Fl:
Fla
Fl.
aVgad
r
Heuristies
25.2
47.8
50.9
51.7
52.4
40.0 35.1
Random (5 runs)
25.2
32.7
34.3
34.7
35.6p
$1.61.5
Repeat co
25.2
29.3
30.9
31.6
32.0
Table Ablation of instructions using heuristics-basse based user model for the Multi agent on the Challenging test set. The Multi is trained with seed 1. Random: randomly instantiated heuristics for ceso across 5 seeds.
6.9 ERROR ANALYSIS
manually analyze errors from the best agent (Imitation). In Tab. 9 we inspect 30 failed devel- opment examples (i.e., unfinished after 5 turns) that are subject to the Neural user. Due to the role interplay, we also count problematic commands. We observe that the user model sometimes issues repetitive or uninformative instructions starting from the 3rd turn, leading the agent to the same wrong selection. This might be caused by the data sparsity for examples with > 3 turns.
Agent
User
Incapabilitiis text
icon
layout pos/dir wrong ea stale ex
#Exumple
27
Table 9: Major error categories of the Imitation model on 30 failed development examples (150 turns). stale ex: repetitive/uiinfommavve instruction. Model is trained with random seed 1.
7 CONCLUSIONS
In this paper, we presented MUG, a novel and challenging task for multimodal grounding on UL. MUG requires a grounding agent being able to correct its own prediction, and allows user to guide the agent via natural language instructions. We contribute new dataset for the task, investigate various modeling options for the agent and developed multiple evaluation strategies including two user models
to enable automatic online testing. We found that interaction greatly improves grounding accuracy in the UI domain. Our experiments and analyses also suggest large room for grounding performances, tigation. Our work contributes to the general effort of multimodal language understanding and its robustness by enabling synchronized multi-turn interactivity.
seemingly easy single screen task, which calls for future inves-

---

Under review as a conference paper at ICLR 2023
REFERENCES
Chongyang Bai, Xiaoxue Zang. Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al. Uibert: Learning generie multimodal representations for ui understanding. In IJCAI, 2021.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on In- frinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp- 65-72, Ann Arhor, Michigan, June 2005. Association for Computational Linguistics. URL
org.
Kobus Barnard and David Forsyth. Learning the semanties of words and pictures. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pp. 408-415.
IEEE, 2001.
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plum- mer. A dataset for interactive vision language navigation with unknown command feasibility. In European Conference on Computer Vision (ECCV), 2022.
Khyathi Raghavi Chandu, Yonatan Bisk, and Alan W Black. Grounding 'grounding' in NLP. In Findings of the Association for Computational Linguisties: ACL-IJCNLP 2021, pp. 4283- 4305, Online, August 2021. Association for Computational Linguisties. doi: 10. 10.18653/v1/221. andings-acll.75..
(tLhhtts::// aclanchology.orrg/
acl. 375
Kan Chen, Rama Kovvuri, and Ram Nevatia. Query-guided regression network with context policy for phrase grounding. In Proceedings of the IEEE International Conference Computer Vision, pp. 924-832,20 2017.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 1584-15097, 2021.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibsehman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design ap- plications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 845-885880 854, 2017.
Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Hougiang Li. Transvg: End- to-end visual grounding with transformers. CoRR, abs/2104.08541, 2021. URL https://
arxiv.org/abe abs/ 220... 08541
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 16 words: Transformers for image recognition scale. September 2020.
Anna Effenberger, Rhia Singh, Eva Yan, Alane Suhr, and Yoav Artzi Analysis of language change in collaborative instruction following. In Findings the Association for Computational Linguistics:
EMNLP 2021, pp. 2803-2811, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 12228653/112222111111 findings-emnlp.22. URL https::/
ac acllanthllogy.og// wwwp:/looy.oog///// findingg. emn 239
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. CoRR, abs/1606.01 847, 2016. httphttp::////// /ttp:v/aaavvvvv/vv///a/ wwwp:vvvvoorrooooooooooo wwtp:///ggbabb.bb...o..o.a
01847
Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani- Tur. Learning to navigate the web. In International Conference Learning Representations, 2019.
Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernán- dez. The PhotoBook dataset: Building common ground through Assulll-grounnde dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1895-1910, Florence, Italy, July 2019. Association for Computational Linguistics. doi:
10. 10.18653/v1/P19. 184. URL L
https: pettts:///acla. aaacaaantooo hot.aaaooooooooooo/1111111111111
10

---

Under review as a conference paper at ICLR 2023
Robert D Hawkins, Michael C Frank, and Noah D Goodman. Characterizing the dynamics of learning in repeated reference games. Cognitive science, 44(6)=11284, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schu- biner, Ruby Lee, Jindong Chen, and Blaise Agüera y Arcas. ActionBert: Leveraging user actions for semantic understanding of user interfaces. December 2020.
Noriyuki Kojima, Alane Suhr, and Yoav Artzi. Continual learning for grounded instruction gen- eration by observing human following behavior. Transactions of the Association for Com- putational Linguistics, 9:1303-1319, 2021. doi: 10.1162/tacl URL https://
aclanthology wrg. 17
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-
room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing
(EMNLP), pp. 4392-4412, Online, November 2020. Association for Computational Linguis- org/2020. baattp::/aclanttaee
tics. doi: 1121866///12222.mmmmmmmoo
emr.pp wwi.n.in... 356
Victor Lavrenko, Raghavan Manmatha, and Jiwoon Jeon. A model for learning the semanties of pictures. Advances in neural information processing systems, 16, 2003.
Gang Li, Gilles Baechler, Manuel Tragut, and Yang Li. Learning to denoise raw mobile UI layouts for improving datasets at scale. CoRR, abs/2201. 1000000 2022. URL htt.tttss///arii...tt
abs / 22012211112 04100
Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. Sugilite: Creating multimodal smartphone automation by demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, pp. 6038 6049, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450346559. doi: 10. 14453025453330225888 URL https:
7/doi. Becilorgg/11 10. 30155/02545533 3025483
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8198-8210, 2020a.
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP),, pp. 5495- 5510, Online, November 2020b. Association for Computational Linguistics. doi: 18.15v53/v1 wwt.0emllp-miin..o
org/ 2020. emnlp main. 443
Yang Li, Gang Li, Xin Zhou, Mostafa Dehghami, and Alexey Gritsenko. Vut: Versatile transformer for multi-modal multi-task user interface modeling. arXiv preprint 1rXiv:21 2021.
Panupong Pasupat, Tian-Shun Jiang, Evan Liu, Kelvin Guu, and Percy Liang. Mapping natu- ral language commands to web elements. In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 4970-4976. Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 1DD18153/1/DI8-155440 URL
1540
Bryan A Plummer; Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting Region-to-Pinn to-Phrs correspondences for richer Image- to-Sentence models. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2641-2649, December 2015,
Shaoqing Ren, Kaiming He. Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. CoRR, abs/1506.00497, 2015. URL http: 01497
11

---

Under review as a conference paper at ICLR 2023
Zhanna Sarsenbayeva. Situational impairments during mobile interaction. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, pp. 498-5098 2018.
Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. Executing instructions in situated collaborative interactions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing GEMNLP-IJCNLA pp. 2119- 2130, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 1/D18665//1/DD11122 URL htR.htttss//ccaaactooooo...oo wttp://gggyooo/1111111100
Ece Takmaz, Mario Giulianelli, Sandro Pezzelle, Arabella Sinclair, and Raquel Fernández. Re- fer, Reuse, Reduce: Generating Subsequent References Visual and Conversational Contexts. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Process- ing (EMNLP), pp. 4350 4350-4368, Online, November 2020. Association for Computational Linguis- a anthelogy.oo org/2020.
tics. doi: 10.11665///20000mmllplp/
emn p- 1..mann..555 353
Kashyap Todi, Gilles Bailly, Luis Leiva, and Antti Oulasvirta. Adapting user interfaces with model- based reinforcement learning. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-13, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-basse image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4566-4575, 2015.
Terry Winograd. Shrdlu: system for dialog. 1972.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision ECCV 2016, pp. 69-85. Springer International Publishing, 2016.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mat- tnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1307-1315, 2018.
A LABELING INTERFACE
Fig. 2 presents the user and agent views in our data collection interface. In the user view, the user can send commands the message box, instruct the agent select the target object as highlighted by red bounding box on the UI screen. On the agent's 's view, the agent annotator can respond the user request by performing object selection on the UI screen, which has all the clickble objects highlighted. But there is no indication of the target object the agent annotator has to guess from the user instruction. The agent is not allowed to text back to the user. The agent's 's current selection is reflected on the UI screen so the user understands how to further instruct the agent. The annotation task designed based on the eyes-on hands-free situation of mobile interaction.
B DETAILS OF THE LABELING TASK
The labelers of the task were native English speakers and had experience using mobile phones. They were trained with few pilot tasks to get familiar with the task, during which we also improved the labeling interface and the guidelines based on labelers' feedback. The dataset completed by 30 labelers batches. The labeling quality was monitored by sampling examples from each batch for manual examination.
12

---

Under review as a conference paper at ICLR 2023
Your role io Instrcctins the other partipant so that they solect the widget on the screen thar's highighttdd and to you green.
agent
Settings
cltic
Shake To Skip
acciive when playping
Slide Right To fleuun
bess boot ertualion oon was changed.
Oripriy Spectrum
Enable Look Screen
Inwert Lock Screen Oulor
Lockscreen
Dipply aloum su beskgroud pifthe
selcting
dimple notruction to connet agent response.
(a) The user the target object (boxed in red) and the agent selection in the previous round (boxed yellow). The issue commands in the message box.
You AGENT the Your rate select the cornec widget screnn based the Intructions given you by the other partidpent.
sammlle inatrution for the
Settings
Io ci checkbox.
Vivatto
Dayday
Daplay Altur Lotboooreed
bociggount
Submit the selection curentty highiighed.
(b) The agent sees the user commands, and all the available candidates (clickable objects) on the screen, which are all boxed in red, and the current selection boxed in yellow.
Figure 2: MUG annotation interfaces consist user view and an agent view.
C LANGUGAE DIVERSITY
The word-level vocabulary in the training set consists of 13, 794 unique words. Fig. shows the distribution the 50 most frequent words in the training split with certain non-content words (c.g., is, of. comma) filtered out.
13

---

Under review as a conference paper at ICLR 2023
0.3
0.52
0.1
8:00
Figure 3: Distribution of top 50 words in MUG training split.
D VIEW HIERARCHY FEATURES
Tab. lists the complete view hierarchy features we used. We unify each feature into real-valued vector. These view hierarchy features first represented with trainable embeddings, and then encoded by the transformer model (Sec. 6.1). For text attributes (e.g., text), we max-pool their non- contextualized token embeddings, which are randomly initialized and trained. For discrete-valued attributes (e.g., type), we use trainable vector for each possible value. The ordering of objects in transformer input follows the pre-order traversal in the view hierarchy (which is tree structure). We then combine the vision representations of individual UI objects via ROI pooling over ResNet featuremap of the encoded screenshot image, and view hierarchy encoding to form a multimodal representation of each UI object for the downstream computation of the model.
We consider these view hierarchy features to be auxiliary. There is often a huge gap between what command the user would issue based on what they mation is for the UI. As we discussed in Sec. 6 about 46% of UI objects do not have a text label, and the would need to come up with their own language description about the object, which is why the text matching baseline fails. Even when there are text descriptions, they are not necessarily what the user would articulate since user command can be abstract. Fundamentally, the internal representation of the UI is often inaccessible or uninterpretable to the user, thus calling for the help of multimodal modeling and interaction modeling.
the UI, and what the underlying infor-
Feature
Example
bounding box Icaf type
[xmin, xmax, ymin, ymax]
true/false
button/checkbox....
clickable
true/false
text
email address/passcode
resource id dom
login_icon
[pre/post-order index]
Table 10: Features & used for visual structure.
E HYPERPARAMETERS & TRAINING
For all our agent models, we use the same configurations, which are grid- searched based on models' offline validation performances. Our hyperparameters are chosen from the best offline develop- ment Fl scores. For the number of self-attention modules, we grid-searched {1, 2, 4, 6), which resulted in 2 hidden layers for the user interface Transformer encoder and 6 hidden layers for the grounding decoder. Each self attentttnnion module uses 8-head self and pecoder-decodee attention with 256 hidden size. The dropout rate for attention and MLP layers is 0.1, which is grid-searched in (0.1, 0.2, 0.5). For learning rate, we grid-searched from {1e-3, 3e-4, 1e-4, 3c-5, le-5). and use 3e-4 with linear warmup with cosine annealing for the first 10k steps. All the models trained to
14

---

Under review as a conference paper at ICLR 2023
100k steps with batch size of 128 on a 32-core Google Cloud TPUv3. Models evaluated every 1k steps and the version with the best development offline Fl4 is saved. The training time for our agent model is around hours.
Our neural user model has the same crid-searched configuration as the agent, i.c., 2 encoder layers, 6 decoder layers, 0.1 for dropout, and the same warmup scheduling. The best learning rate is 1c-4. Different from the agent model, we found the neural user model's 's development CIDEr score quickly drops after 6k steps, possibly due overfitting and data sparsity, thus its training early-stops there.
F EXAMPLES IN THE MUG DATASET
We present some examples from the MUG dataset in Fig. [annd Each example contains instruc- tions and selections from human user and agent annotators.
G LIMITATIONS
English-only Dataset While non-English examples exists, we acknowledge that MUG mostly con- sists of English UI. Other languages do exist the dataset, but consists small portion. Specif- ically, our instructions are English-only. Future extensions to our work should address or alleviate this issue.
Platforn-specific Interfaces Our interfaces, since coming from RICO, only consist of Android screens. In practice, it is also difficult to obtain non-Android interfaces. We acknowledge this is an application limitation. And the bias from the top and bottom banner of Android could make trained model brittle in other domains.
Going beyond Single Screen We aim establish the task and report baseline performances for future work. The interaction in MUG happens within the same user interface. A natural extension would be extending the task to span over sequence of interfaces. Indeed, the task would become more challenging, and potentially require large offline training data and reliable online simulation.
Better User Model The current best neural instruction generation we use has a CIDEr 78.0 on the validation set. We acknowledge there is space for further improvement. Note that neural instructions are trained on multi-turn examples in MUG, which amounts to 20% of the training data. It suggests external resources could be useful for improving user model performances.
Interaction Dynamics between User and Agent It would be helpful to study how/why the agent sometime repeatedly makes incorrect actions in Tab. 6 such as whether repeated mistakes are due to the lack of language utility/diversity in user instruction or the lack of understanding in the agent.
H PREDICTION EXAMPLES
Here, we demonstrate predictions from the Imitation model. Fig. demmonstrtts successfully solved examples following the instructions generated by the Heuristic user model, while failed ones are in Fig. 7 Similarly, Fig. demoosttaas solved ones following the instructions generated by the Neural user model, and failed ones are Fig.
15

---

Under review as a conference paper at ICLR 2023
AOTINATE
almus Fabie cdl den De ana ediorrye
whos dee buck
CeDroeee telrag
Oetraeess Srting
Ouroneeees Beli
of
J0 200
Notoeee 500n Spotie vemus
Mozeriat 200g cgyptam Waran
Austrollan coult appoovs
Austrellen count rigrowe
coant
coult
fond
Pfoer feed 234.2m
Cansona iffeciing
Domancon
Ieled vile Seary
Irkkdd aknn teury
EI 4
Figure 4: MUG examples 1-4, Instructions are at top of each turn. Agent selection is in and target is in D
16

---

Under review as a conference paper at ICLR 2023
Free Wu
Fole Numbor
Seonee
oode
Srint code
numberty
of
Ooct
pulataga
AUS0 Quet nntfebeedee
2V Tuery
1ord
LOWE'S
LOWE'S
Lowers!
Text (SM)))
Text (SM)))
messages
Log Matineeery
Log oe Aath
Log (n librry
buf 201y
Del 200y
Covire
Dertre
Figure 5: MUG examples 5-8. Instructions are top of each Agent selection is in and target is in
17

---

Under review as a conference paper at ICLR 2023
30
07
DI ET
Dostt
Ger bndd
Sipelian
Fxpreise eagl place del
plan umter
derq.sin
# 60AnEEE BRAGE
iTuner
iTuner
Podicaea
Preferences
Preferences
turter
LEARD ATRT ST
HEAR HEARST
Heart Spa
Snargancy Optioss
Papurt lipe
Emageno Sphare
Rejor Tg
gaicy Oplion
Sakey
Securty
Dartey Vio
Scaurty Cooor
Salre
Security
Figure 6: Completed examples by the Imitation agent following the instructions generated by Heuristic user.
18

---

Under review as a conference paper at ICLR 2023
Select Dectricty Board
Select Your Bectridt
Your Elecricity Beard
Select Elcctictty Seard
Select Dlcctittty
(P/R1)
Sigthar
Righhart
Denn
ESOS Rogthen De
Reghhe
1029 Numano Sala
1025 Namawa Deta
9955 Taruuta Dele
KEOS Yamun Dolk
1000 Yemame Doll
Dengiton Eeericty Supdy
Bangeto Eacticty Supaly
Zargake Elcttityy Bapel
Eurgalon Dlccticly Supply
Burgaton Destrity Gupple (MRROOU)
(H600011)
Absi
Compary (SC001)
Company SED0P)
Comperty (SDDONL
Compuny (BSD0N)
Company (B0pany
CESC
CF 2 393
Cu zurt taillgn
CINCP 1280
CUSCE poin
CINCP sorm 1280
cunce 2ard
Poudees
Fuesday
Fussday
Tuoddy
uusdaay
Tuesday
Polits Sanoll 210
Polro Extrel 354
Parra Exnol 264
Poiro Bar el 254
Pii Exnoal 210
D111
II
0TH
Werso
Wirso
Wanesher
Wamiaker
1001
64
Orpale
Oargle
17
pertyng
Figure 7: Failed examples by the Imitation agent following the instructions generated by the Heuristic user.
19

---

Under review as a conference paper at ICLR 2023
60000
O
1yp
O
100
100
Protect
Prinec
Protec
BEEBB CORBEEE
:Tuner
:Tuner
Preferences
2
T 1:0
0,
MiCroten
# D D
Ewiyg Denerit
Divenioss
Bau Peir Dood
Salifrre Dan
Drodel
Figure 8: Completed examples by the Imitation agent following the instructions generated by the Neural user.
20

---

Under review as a conference paper at ICLR 2023
22.002-
25,000
DR 20.000
26,00,
940 BOROO
TIMB DO00
IVO MECS
ACCIANT BETTINSS
ACCIANT BETIINSS
M
ABOUT
M
M
M
YOUMAET
DIETS
TASE ERETS
Opariis
in
in
in
in
M
O
Foin Color
Fart Color
Farm Color
Forn Color
Form Color
Souch
07 30
07 30
07 30
07 30
07 30
0
Graluhy
firok-y
Pulmersi
Pegp
Beatrs
Li teaart
April
Skenn Mitendif
Davn Mitena
1i Cana Stew
Done tittowe
Npleeerees
Senon Sueprttome
E) Planse Suopageree
Figure 9: Failed examples by the Imitation agent following the instructions generated by the Neural user.
21


Reviews:
[{"review_id": "pIhRYs2AFmf", "reviewer": "Reviewer_t7tT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In  this paper,  it proposed a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen.  The authors collected a new large dataset for interactions between human and agent, in which 20% are more than 1 round. The paper also experimented with different modeling variants and evaluation strategies to build different benchmarks on different statistics. \n", "summary_of_the_review": "This paper clearly presents and verifies the contributions including the dataset created and the experimental benchmarks.  As it mainly focuses on the interactions and grounding. It might be better to submit to nlp related conferences or dialog related conferences.", "strength_and_weaknesses": "The main contribution of this paper is to create a rich dataset which contains the human agent interactions for language grounding on a GUI screen. The paper also built some benchmarks based on different model variants and evaluation strategies to show the interaction  significantly improves grounding accuracy.\nSome questions:\n1. Besides the Mobile UI grounding task, there are also some tasks in interactive grounding, for example in the HRI field, what’s the difference between GUI Grounding and other fields’ interactive grounding task?\n2. What’s the difference between online results and offline results?\n3. It’s not clear how you evaluate the human-in-the-loop setting?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gKSdMJi_Ij", "reviewer": "Reviewer_YoBY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents present MUG, a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen, essentially bridging multimodal applications with user interfaces in HCI research. Their main novelty is an interactive task as opposed to existing static interaction tasks in user interfaces, and multiple rounds of interactions enables the user to give further commands for the agent to refine or correct its actions. This paper presents a dataset of 77,820 sequences of human user-agent interaction on mobile interfaces to study this problem and experiment several models and evaluation strategies, finding that iterative interaction improves task completion accuracy but also presents directions for future work.", "summary_of_the_review": "There can be more comparisons and evaluation of actual multimodal methods, including better motivation and definition of multimodal grounding in this task.", "strength_and_weaknesses": "Strengths:\n1. The paper is well motivated and described. The ideas are clear and experiments are on several large multimodal datasets.\n2. The paper is largely clear with clear figures and exposition. It was a joy reading this paper.\n3. The paper makes important contributions to the data collection and model evaluation, with important insights for future work.\n\nWeaknesses:\n1. There can be more comparisons and evaluation of actual multimodal methods - right now all the tested methods are largely on interactive learning/online and offline RL, but the paper is motivated as a multimodal paper. See https://arxiv.org/abs/2107.07502 for an example of a recent multimodal benchmark and a suite of tested multimodal fusion models with key takeaways for future work, and see https://arxiv.org/abs/2209.03430 for more references in the field of multimodal grounding and the key methods there.\n2. There can also be a better motivation and definition of multimodal grounding in this task - is grounding simply finding the right region of interest as a bounding box? How does grounding change across interaction? How important is grounding to the specific object versus whole UI-level representation? What are the effects of the amount of data required for grounding? Many questions should be formalized or motivated here. Given that this seems to be the main selling point of the dataset there needs much more analysis on how current grounding (or more generally alignment) methods work on this task and what future innovations are needed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_2jXvRJG9HM", "reviewer": "Reviewer_FLkv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a novel benchmark dataset for language grounding on user interfaces, notably exploring the ability to handle *sequences* or multiple turns of instructions/actions between a human user and virtual assistant. This dataset is of remarkable size, consisting of 77K unique language utterances, grounded in 31K individual mobile app “screens” belonging to a set of 7K unique mobile applications. In addition to the dataset itself, this work proposes a straightforward baseline with a set of meaningful ablations that characterize the difficulty and open challenges of working on this task.\n\nThe dataset is collected by pairing two human annotators together, with one human acting as the “speaker” and the other as the “follower” — similar to many other multi-turn data collection procedures in the NLP literature. In addition to assembling this dataset, the work further splits the data into different splits based on the “challenge” level of the interactions in question — where “challenge” is a function of taking more than 1 turn to get to the correct “answer” (agent interaction on the screen). That means that ~80% of the dataset falls into the single-turn instruction following paradigm.\n\nThe proposed approach pairs an object-oriented visual tokenizer with a Transformer decoder that eats the history of state/actions and outputs the a “context” vector that is used to score each object to interact with (e.g., as a ranking problem). The ablations of the approach show that single-turn instruction following is not enough, and that modeling the entire context, and ability to make decisions in sequence (e.g., via imitation learning) is critical to performing well on this dataset. \n", "summary_of_the_review": "This is a decent dataset paper, proposing an interesting resource for language grounding on user interfaces. However, I don’t believe the paper as written delivers on its promises; I think the language diversity is lacking, and the multi-turn nature of the dataset is a bit oversold.\n\nI would really love to see an ablation experiment that just maps key phrases (bigrams, trigrams) to corresponding UI components to see if the “difficulty” of this dataset is actually multi-turn grounding, or just having the right mapping of words to components on the interface —> if it’s the latter, then it feels like any pretrained model with some amount of knowledge of UI/web components (e.g., WebGPT) would make this dataset trivial.\n", "strength_and_weaknesses": "I believe this to be a well constructed dataset, with applications for assistive technology, and a nice playground for studying deeper questions of multimodal language grounding. \n\nHowever, I do believe this work has several weaknesses; while the paper focuses on the multi-turn aspect of their data collection, only 20% of the collected interactions actually require “revisions” or “corrections” from the human — this seems rather low for a dataset that hopes to spawn future work in multi-turn interactions. Furthermore, the type and diversity of language that human users give is rather limited; as noted in the work, most utterances are only ~4 words long; it’s not clear the data is rich enough to enable multi-turn grounding research, or if the difficulty is just in mapping “key phrases” to UI components — it would have been nice to see this ablation.\n\nFinally, the choice of Offline RL as an ablation seems a bit unnatural; given that we know the “right” UI component to select in each interaction, treating this as a multi-step MDP doesn’t feel appropriate; instead, it’s almost as if something like DAgger would be the better baseline —> this would also be strictly better than the imitation learning baseline — I would love to see how much this improves on the existing results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KFcSV3RfJ6", "reviewer": "Reviewer_tBFu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a new database for training a multimodal agent in a simple grounding task.  The user is faced with a screen and wishes to select a radio button or checkbox by voice.  If the agent gets it wrong, the user can attempt to correct the agent by a followup command such as \"click the checkbox below\".  The data was collected using a wizard of oz paradigm in which the human user sees a screen display and the randomly sampled UI object to be selected.  The user then issues an appropriate written (?) command.  The human agent sees the same screen with all clickable objects marked and the user's input.  The agent then selects the most likely target and the process repeats.  All data is derived from an existing data set called RICO, first published in 2017. \n\nIn addition to the data, the authors provide benchmark models and experimental results.  The benchmark model consists of a resnet encoder for the screen which is then plugged into a transformer along with the view hierarchy features. The resulting embedding is then input to an decoder along with the previous turns of the \"conversation\".  The agent is tested using both a heuristic and neural trained user model.\n", "summary_of_the_review": "There is little that is original in this paper and I found the motivation for the work unconvincing.  If the goal is to empower a UI agent to respond to natural language commands as well as direct clicks, then why would you complicate matters by conditioning with a resnet encoded screen shot?  If the agent rendered the display, it knows exactly where everything is so why not simply create a list of the available UI objects and their spatial relationship.", "strength_and_weaknesses": "The authors will make both the dataset and the code available.  This dataset will be a useful addition to those working on this type of interaction and more generally for those interested in multi-modal processing systems.   However, the outputs appear to be limited to clicking buttons and check boxes.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject