Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a novel benchmark dataset for language grounding on user interfaces, notably exploring the ability to handle *sequences* or multiple turns of instructions/actions between a human user and virtual assistant. This dataset is of remarkable size, consisting of 77K unique language utterances, grounded in 31K individual mobile app “screens” belonging to a set of 7K unique mobile applications. In addition to the dataset itself, this work proposes a straightforward baseline with a set of meaningful ablations that characterize the difficulty and open challenges of working on this task.

The dataset is collected by pairing two human annotators together, with one human acting as the “speaker” and the other as the “follower” — similar to many other multi-turn data collection procedures in the NLP literature. In addition to assembling this dataset, the work further splits the data into different splits based on the “challenge” level of the interactions in question — where “challenge” is a function of taking more than 1 turn to get to the correct “answer” (agent interaction on the screen). That means that ~80% of the dataset falls into the single-turn instruction following paradigm.

The proposed approach pairs an object-oriented visual tokenizer with a Transformer decoder that eats the history of state/actions and outputs the a “context” vector that is used to score each object to interact with (e.g., as a ranking problem). The ablations of the approach show that single-turn instruction following is not enough, and that modeling the entire context, and ability to make decisions in sequence (e.g., via imitation learning) is critical to performing well on this dataset. 

strength_and_weaknesses: I believe this to be a well constructed dataset, with applications for assistive technology, and a nice playground for studying deeper questions of multimodal language grounding. 

However, I do believe this work has several weaknesses; while the paper focuses on the multi-turn aspect of their data collection, only 20% of the collected interactions actually require “revisions” or “corrections” from the human — this seems rather low for a dataset that hopes to spawn future work in multi-turn interactions. Furthermore, the type and diversity of language that human users give is rather limited; as noted in the work, most utterances are only ~4 words long; it’s not clear the data is rich enough to enable multi-turn grounding research, or if the difficulty is just in mapping “key phrases” to UI components — it would have been nice to see this ablation.

Finally, the choice of Offline RL as an ablation seems a bit unnatural; given that we know the “right” UI component to select in each interaction, treating this as a multi-step MDP doesn’t feel appropriate; instead, it’s almost as if something like DAgger would be the better baseline —> this would also be strictly better than the imitation learning baseline — I would love to see how much this improves on the existing results.

clarity,_quality,_novelty_and_reproducibility: In general this paper was a joy to read, with very digestible takeaways. The novelty of this dataset with respect to prior work is clear (though perhaps a bit understated), and I’m excited to see this dataset and baselines released as mentioned in the work!

summary_of_the_review: This is a decent dataset paper, proposing an interesting resource for language grounding on user interfaces. However, I don’t believe the paper as written delivers on its promises; I think the language diversity is lacking, and the multi-turn nature of the dataset is a bit oversold.

I would really love to see an ablation experiment that just maps key phrases (bigrams, trigrams) to corresponding UI components to see if the “difficulty” of this dataset is actually multi-turn grounding, or just having the right mapping of words to components on the interface —> if it’s the latter, then it feels like any pretrained model with some amount of knowledge of UI/web components (e.g., WebGPT) would make this dataset trivial.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents present MUG, a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen, essentially bridging multimodal applications with user interfaces in HCI research. Their main novelty is an interactive task as opposed to existing static interaction tasks in user interfaces, and multiple rounds of interactions enables the user to give further commands for the agent to refine or correct its actions. This paper presents a dataset of 77,820 sequences of human user-agent interaction on mobile interfaces to study this problem and experiment several models and evaluation strategies, finding that iterative interaction improves task completion accuracy but also presents directions for future work.
strength_and_weaknesses: Strengths:
1. The paper is well motivated and described. The ideas are clear and experiments are on several large multimodal datasets.
2. The paper is largely clear with clear figures and exposition. It was a joy reading this paper.
3. The paper makes important contributions to the data collection and model evaluation, with important insights for future work.

Weaknesses:
1. There can be more comparisons and evaluation of actual multimodal methods - right now all the tested methods are largely on interactive learning/online and offline RL, but the paper is motivated as a multimodal paper. See https://arxiv.org/abs/2107.07502 for an example of a recent multimodal benchmark and a suite of tested multimodal fusion models with key takeaways for future work, and see https://arxiv.org/abs/2209.03430 for more references in the field of multimodal grounding and the key methods there.
2. There can also be a better motivation and definition of multimodal grounding in this task - is grounding simply finding the right region of interest as a bounding box? How does grounding change across interaction? How important is grounding to the specific object versus whole UI-level representation? What are the effects of the amount of data required for grounding? Many questions should be formalized or motivated here. Given that this seems to be the main selling point of the dataset there needs much more analysis on how current grounding (or more generally alignment) methods work on this task and what future innovations are needed.
clarity,_quality,_novelty_and_reproducibility: The paper is largely clear with clear figures and exposition. It was a joy reading this paper. The paper makes important contributions to the data collection and model evaluation, with important insights for future work.
summary_of_the_review: There can be more comparisons and evaluation of actual multimodal methods, including better motivation and definition of multimodal grounding in this task.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In  this paper,  it proposed a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen.  The authors collected a new large dataset for interactions between human and agent, in which 20% are more than 1 round. The paper also experimented with different modeling variants and evaluation strategies to build different benchmarks on different statistics. 

strength_and_weaknesses: The main contribution of this paper is to create a rich dataset which contains the human agent interactions for language grounding on a GUI screen. The paper also built some benchmarks based on different model variants and evaluation strategies to show the interaction  significantly improves grounding accuracy.
Some questions:
1. Besides the Mobile UI grounding task, there are also some tasks in interactive grounding, for example in the HRI field, what’s the difference between GUI Grounding and other fields’ interactive grounding task?
2. What’s the difference between online results and offline results?
3. It’s not clear how you evaluate the human-in-the-loop setting?

clarity,_quality,_novelty_and_reproducibility: The paper proposed to create a large dataset for the interactive GUI Grounding task. It clearly described how the dataset is collected, and also built different model variants and metrics to investigate the relative statistics of the task.

summary_of_the_review: This paper clearly presents and verifies the contributions including the dataset created and the experimental benchmarks.  As it mainly focuses on the interactions and grounding. It might be better to submit to nlp related conferences or dialog related conferences.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper describes a new database for training a multimodal agent in a simple grounding task.  The user is faced with a screen and wishes to select a radio button or checkbox by voice.  If the agent gets it wrong, the user can attempt to correct the agent by a followup command such as "click the checkbox below".  The data was collected using a wizard of oz paradigm in which the human user sees a screen display and the randomly sampled UI object to be selected.  The user then issues an appropriate written (?) command.  The human agent sees the same screen with all clickable objects marked and the user's input.  The agent then selects the most likely target and the process repeats.  All data is derived from an existing data set called RICO, first published in 2017. 

In addition to the data, the authors provide benchmark models and experimental results.  The benchmark model consists of a resnet encoder for the screen which is then plugged into a transformer along with the view hierarchy features. The resulting embedding is then input to an decoder along with the previous turns of the "conversation".  The agent is tested using both a heuristic and neural trained user model.

strength_and_weaknesses: The authors will make both the dataset and the code available.  This dataset will be a useful addition to those working on this type of interaction and more generally for those interested in multi-modal processing systems.   However, the outputs appear to be limited to clicking buttons and check boxes.
clarity,_quality,_novelty_and_reproducibility: The paper is generally well-presented although important information is glossed over.  In particular, I wasn't sure how the view hierarchy features were encoded and what role the screenshot played.   It also seemed to me that a simple hand-crafted baseline could have done very well on this task.
summary_of_the_review: There is little that is original in this paper and I found the motivation for the work unconvincing.  If the goal is to empower a UI agent to respond to natural language commands as well as direct clicks, then why would you complicate matters by conditioning with a resnet encoded screen shot?  If the agent rendered the display, it knows exactly where everything is so why not simply create a list of the available UI objects and their spatial relationship.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:

This paper presents a new dataset and a benchmank for language grounding on user interface, where a user and an agent collaboratively on an interface screen.  This dataset is quite large in which 20% are more than one round. For data collection, two humans are working together using a wizard of oz paradigm to complete certain tasks.  Experiment are conducted with multiple models and evaluation strategies, finding that iterative interaction improves task completion accuracy but also presents directions for future work.


Strength:

The paper is well motivated and described. The main contribution is the dataset which contains the human agent interactions grounding on a user interface screen, as well as multiple baseline models showing the interaction can truely improves grounding accuracy.   Both the dataset and the codes will be released publicly.

 
Weaknesses:

The task is simple: just click buttons and check boxes. About 80% of the interactions are single turn and most sentences are only 4 words. Comparisons with similar work are needed.  The selection of baseline models could be improved.
justification_for_why_not_higher_score: The task is simple: just click buttons and check boxes. About 80% of the interactions are single turn and most sentences are only 4 words. Comparisons with similar work are needed.  The selection of baseline models could be improved.
justification_for_why_not_lower_score: NA
Decision: Reject