Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A multi-modal multi-task model is proposed for UI understanding. It has an Image-Structure Transformer and Question-Answer Transformer to encode/decode image, structure and language information. The method is evaluated thoroughly on 5 different data sets and achieves good results.
main_review: Strengths
+ Formulated a multi-modal multi-task learning for graphical user interfaces
+ A new two-tower Transformer architecture is designed

Weakness
- By jointly trained on 5 tasks, the model is on par or slight improves over models trained on individual data sets. This does not match the observation on other image-language data joint training where significant improvement is achieved.
- The 5 tasks are closely related, and it is hard to see how the model can generalize to a different task, such as generating/editing new UI layout
- The main architecture of the model is new, while all the building blocks are not. The technical contribution is somewhat lacking.
summary_of_the_review: The paper demonstrated an interesting problem addressed by a standard transformer based model. The potential of generalization to more tasks is not clear, and the improvement over single task models falls below expectation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a multi-modal Transformer for multi-task modeling of user interfaces. It is potentially worthy to investigate how to incorporate multiple UI related tasks into a model. The authors hence design a so-called Versatile UI Transformer model which involves three modals inputs to handle five unique tasks. 
main_review: In summary, their main works can be summarized as followed:
(1) The authors formulate multi-modal multi-task learning for graphical user interfaces and design a VUT model.
(2) The authors create a so-called Language Grounding dataset for language grounding task.
This topic is interesting and it is pioneer to incorporate muti-task learning in graphical user interfaces. The manuscript is easy to follow. And the technical contribution is significant. The experimental results conducted on 5 distinct UI tasks demonstrate its superior performance over some state-of-the-art methods.


The reviewer suggests borderline. The weakness of this paper is as follows.
(1)	Tables 5,6,7 and 8 compare the multi-task learning with the single task counterpart. However, their results show that the multi-task learning cannot significantly facilitate the studied tasks. For example, in the Table 5, the setting of “Screen Summarization alone” performs better than the setting “all 5 tasks”. This is contrary to the original intention of this paper. More explanations should be given.
(2)	There are some typos in this manuscript, e.g., “We experiment with VUT on 5 distinct UI task”. 

summary_of_the_review: I am a researcher interested in UI and published many paper of UI on top conferences.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an architecture for graphical user interfaces which involve multi-modal inputs (UI screenshots, Hierarchy structures, Natural Language) and multi-task learning (UI Object Detection, Widget Captioning, Screen summarization, Language grounding, and Tappability).The proposed architecture consists of seperate transformer blocks to encode image and text modalities. The two transformer blocks attends to each other to produce multi-modal outputs, which is then used for downstream tasks. The authors that with the proposed architecture, training on all tasks simultaneously is better than training on individual task alone.
main_review: **Strengths:**

1. Overall, the formulation of the problem and various design choices (global positional encoding, focus map, pointer head for the grounding task) make sense. The authors also seem to have carefully designed their experiments. For instance, ensuring that comparisons are fair (same number of parameters, etc) for the object detection task.
2. Given the nature of the problem statement (with multiple tasks, inputs and outputs), the authors have done a good job in explaining each of them properly. Overall, I thought the paper was easy to read and understand. 
3. The authors provide several reasonable insights that might be relevant to the user interface modeling community — using object detection as part of multi-task learning instead of standalone pre-training task, design choices to create a single unified architecture for all the tasks, multi-task learning outperforming single-task learning etc. 

**Weaknesses**

1. I understand that a single model is helpful for multiple UI tasks, but I wonder if this approach is scalable beyond the 5 tasks and 5 modalities mentioned. The authors comment that the model architecture is designed to remain stable for a growing set of tasks, but this seems to be under the assumption that the input and output modalities would remain constant. A discussion on how to add new tasks to the same framework might help, or a discussion on why the current framework is enough.
2. One of the important motivations of multi-modal multi-task learning mentioned was to achieve better or on-par performance with a single model (and supposedly fewer computations) which is crucial for devices with limited computing resources. But a direct head to head comparison for the computational cost of a multi-task model and individual models is not provided. Are there any overheads/disadvantages because of multi-task learning (Like a larger model size, inference time for individual tasks etc)? The model sizes are varied across experiments to achieve on par performance which makes the comparison of the computational cost not so obvious. Such a comparison would highlight the advantages of the multi-task model and would be helpful for the relevant audience.
3. I also believe that the downstream tasks are also somewhat similar (language command grounding, tappability, UI object detection, UI summarization, widget captioning). It is therefore not surprising that multi-task learning should help these tasks. 
4. There is no comparison provided with any baseline for 2/5 tasks (Language grounding, Tappability)
5. The novelty in the paper is somewhat lacking. The techniques used in the paper (multi-branch transformer, pointing mechanism, cross-modal attention, global positional encodings, etc) have been shown to work in the past for image-text tasks [1, 2]. The paper is taking all the lessons from past works and applying it to a new domain. 

**Clarifications:**

1. The training procedure mentioned in section 5.2.2 talks about joint training but the procedure followed for training for individual tasks or a subset of tasks is not described in detail. Were the same hyperparameters used for all configurations? 

**Updates after rebuttal period** 
The authors addressed some of the concerns -- showing inference time, model size and a discussion about training details and hyperparameters in the appendix. However, I am not convinced that the paper presents new insights that are relevant for the broader ICLR community. Concerns around novelty and the multi-task setup was also raised by another reviewer (e7Hg). Responses to reviewer e7Hg also aren't convincing. If the tasks are not similar, and the learning objectives are not aligned, then the motivation for multi-task learning is solely for reducing memory footprint and computational cost. I believe that this contribution isn't enough for me to recommend acceptance.   Thus, I am going to stick to my original rating. 

[1] R. Hu, A. Singh, T. Darrell, M. Rohrbach, *Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA*. in CVPR, 2020

[2] J. Lu, V. Goswami, M. Rohrbach, D. Parikh, S. Lee, 12-in-1: Multi-Task Vision and Language Representation Learning. in CVPR, 2020
summary_of_the_review: Overall, the paper showed how multi-task learning can improve performance on several downstream UI modeling tasks. These insights might be relevant to the UI modeling community. The authors use several popular ideas in the field of multi-modal, multi-task learning and showed it's effectiveness on multiple downstream tasks. The paper is well written, and the ideas are clearly presented but it lacks in technical novelty. Additionally, the paper lacks in a few other areas -- comparison to single-task models with respect to computational cost, inference time etc, the downstream tasks are similar but the performance gains aren't significant. Due to these reasons, I think the paper is marginally below the acceptance threshold.
review_title: null

Meta-review: This paper proposes a neural architecture for tasks involving user interfaces. Tasks involve detecting objects on screen, writing captions about UI components, attribute recognition, etc. The reviewers for this submission found the proposed model to be reasonable and effective. They also found the paper to be well written and easy to understand. However, they did have one major concern, before and after rebuttal: While the model and design choices were reasonable, they questioned if the insights gained from this paper were of interest and relevance to the broader vision community. They also had other concerns/suggestions including adding inference costs and adding more detail, which were addressed in the rebuttal. Another concern was the fact that multi-task did not provide large gains over single task. I agree with the authors in this regard. I think the goal here was to produce a multi-task model that attained at least parity with respect to single task, because a single model would provide large benefits when running on a device, and hence I think this concern was well addressed.

My takeaway from the paper, reviews and discussion, however, continues to focus on the major concern of the reviewers. I think this paper would have benefited from answering at least one (if not more) of the following questions to the reader: (1) Why should the broader community work on this task ? (2) If this task is of limited interest, are there instead, aspects of this task that serve as a useful testbed for multimodal research ? (3) If the task and testbed are not directly applicable, are there new techniques developed in this paper that are broadly applicable to other problems or domains ?

Unfortunately, I think that this paper does not presently address either of these questions strongly to the reader. The paper proposes a method for their task, but readers who aren't directly interested in that end task may find this submission less interesting, in terms of insights for their own work. Given the above, I encourage the authors to address this concern and resubmit. I recommend rejection.
Decision: Reject