Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model that incorporates bounding box queries in the visual encoder to extract region-specific features for UI modeling tasks. With the proposed framework, no view hierarchy would be required during the above modeling process, which thus alleviates the potential concerns of missing object text or bounding box misalignment.

strength_and_weaknesses: Strength
1. The idea of incorporating bounding box queries to obtain region-specific features provides a unique way to extract fine-grained features.
2. Compared with existing approaches, impressive performance on finetuning experiments for three mobile UI understanding tasks were achieved.

Weaknesses:
1. Overall, the novelty of this work is limited. The model architecture is similar to Flamingo (Alayrac et al., 2022). The only difference is that the inputs of Flamingo are visual tokens and learned latent queries, while SPOTLIGHT takes visual tokens and bounding box queries.
2. The main idea of this work is that, the bounding box queries have capacities to learn how to represent specific image regions, while jointly taking into account of the screen context. However, there is no further analysis, explanation, or qualitative/quantitative supports for this claim.
3. The experiment comparisons are potentially unfair. The improved results reported by SPOTLIGHT might come from large-scale pretraining, overwhelming parameters, and the uses of more powerful models (i.e., ViT and T5). All the baseline models (e.g. Widget Caption (Li et al., 2020b) and Screen2Words (Wang et al., 2021)) are composed of ResNet and vanilla Transformer with 128 dimensions. 
4. Missing comparisons to SOTAs in multi-task (Table 4) and few-shot learning tasks (Table 5).
5. In Section1, the paper states its second contribution: “we develop a method for creating a large-scale pretraining dataset from automatically collected mobile screens.” However, this newly dataset is not discussed or presented later.
6. In Sect. 5.3, the authors note that SPOTLIGHT outperforms previous models. However, in Table 4, there are no other experiments to support this claim.







clarity,_quality,_novelty_and_reproducibility: Please see the comments above.
summary_of_the_review: This paper proposes a model that incorporates bounding box queries in the visual encoder to extract region-specific features for UI modeling tasks. With the proposed framework, no view hierarchy would be required during the above modeling process, which thus alleviates the potential concerns of missing object text or bounding box misalignment. However, the paper is not well presented (with insufficient technical contributions and details, and missing or over-claimed experimental results). I would reconsider my ratings if the above issues can be properly addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: For the mobile UI understanding tasks, the authors achieve advanced performance on multi-task learning and few-shot learning by pretraining the vision-language model on the proposed 2.69M dataset. This pretraining-finetune framework is easily scalable to other UI modeling tasks and not needs the view hierarchy as auxiliary information. Meanwhile, the authors proposed a pipeline for automatically creating the large-scale UI understanding dataset.
strength_and_weaknesses: Strength:
The authors attempted to study UI understanding tasks by multi-task learning with pretrained multi-modal models, which is an independent, innovative and laborious work. The experiment results prove the generalization and validity of this method. Meanwhile, The author also proposes a large-scale UI modeling pretraining dataset. 

Weakness:
1. Although the motivation is innovative in constructing the pretraining model for the UI modeling field, the overall pretraining pipeline may lack appropriate innovation and some aspects are similar to Flamingo, e.g., the vision-language model architecture, the evaluation method on the multi-task and few-shot learning, and the computational pipeline of Region Summarizer is similar to the Perceiver Resampler of Flamingo. 
2. The illustration of the pretraining dataset is too brief. It is recommended to add more statistical analysis and construction details, e.g., The text and content description are human-annotated or not?
3. Considering the readability, it is recommended to add the introduction of test datasets about downstream tasks. Perhaps the authors can add these in the supplementary material rather than suggesting the readers read the previous literature.
4. The ablation study is limited and maybe authors can consider the following ablation to make the paper clear and complete.
(1)	Region Summarizer:
①	Why choose the bbox coordinates as q rather than the region feature of bbox?
②	What is the result when kv is just the vit_outputs rather than the concatenation of vit-outputs and bbox.
(2)	Pretraining:
①	Can the text input is concatenated by the four text elements of an object?
②	The ViT is freezing as Flamingo or not?  If not, The much screenshots of examples will bring the burden of GPU memory or not?   
5. The 2.69M pre-training data as extra knowledge may obscure the fairness of the method compared with the baseline method. Maybe author can try to further finetune the comparison methods combined with single-modal encoders of the pretraining model? This is just a suggestion out of curiosity and not influence the final judgment.
6. In the Discussion section, these “early exploration” are supposed to illustrate hypotheses by some experimental data. 
7. The paper of the comparison method “Widget caption” is cited repeatedly.
8. Maybe the authors can consider making the dataset and code public to promote the development of this field. It is just advice and not influence the final judgment.

clarity,_quality,_novelty_and_reproducibility: It is a original work but some details of paper are ambiguous.
summary_of_the_review: It is recommended to add some details and ablation study in paper. Moreover, since there is very little research in this field, I encourage the authors to open source the pretraining dataset,model checkpoints, and code, which will promote the development of this field and attract more researchers to make the field more prosperous.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors introduced a novel vision-language multi modality model (Spotlight) for mobile UI understanding tasks. Specifically this model is designed for tasks for mapping visual components in the UI to natural language text. In the context of UI understanding models, this paper presents 2 major contributions.

1. The authors challenge the recent belief that view-hierarchy based UI understanding model is a hands-down choice for accuracy. They proposed the Spotlight model which only uses visual information at inference time; this can potentially lead to higher accuracy by working around the typical incomplete hierarchical view information for prediction.

2. The authors developed a pretrained model for the UI understanding tasks. They present SoTA accuracy on UI understanding tasks under single-task finetuning, single-task few shot and multi-task finetuning. They show that the pretraining is crucial to the high accuracy in widget captioning, screen summarization, command grounding tasks.
strength_and_weaknesses: Strength

1. The technical effort is solid. The creation of the pretrained model involves the collection and curation of millions of UI screen shots. Such endeavor combined with the novel pretraining objective for UI understanding task enables doubling the CIDEr score from no pretraining efforts on the same model.

2. The empirical results is impressive. It attains SoTA accuracy on the 3 tasks with only visual info input during inference time.

Weakness

The paper is a bit hard to follow and needs modification to enable a self-containing manuscript for readers to understand.

1. There are technical concept/facts missing their intuitive or formal definitions in the related work / background sections. Representative missing information are
     a. What is an intuitive definition of view-hierarchy information which is the primary input to existing models. 
     b. How severe is the information missing problem in view-hierarchy information? Could you slightly quantify it w.r.t. the datasets the author used or collected? This is important to evaluate the claim on the problem of previous view-hierarchy info based models. 

2. The model design is a bit hard to fully understanding with pure text. Overall, I would recommend a symbol based formal model description which could answer the following questions.
    a. In listing 1, it seems the Dense function encodes the coordinate into vectors. Is this encoding the normal way in VIT which just maps the coordinates to the 1D index of the image patch in the patch sequence?
    b. Also why the coordinate is encoded twice via both the Dense function and the SinusoidalEmbed function?
    c. What is the training objective? My rough guess from the relevant description in 4.1 is that it use the self contrastive loss type thing to promote the patch and text description pairs. But it is very unclear to me if this is the case.
clarity,_quality,_novelty_and_reproducibility: Clarity: As discussed in the Strength and Weakness, I would highly recommend heavy re-writing to ensure the best possible impact of the great technical results. Currently it is hard to follow the story and understand some of the technical details.

Quality: The proposed model demonstrated substantial accuracy improvement over previous models. The ablation is solid in showing key contributing factos such as region summarizer and pretraining. I consider the experiment quality as high.

Novelty: Given the authors challenged a common believe on view-hierarchy info based model is a hands-down choice for good accuracy, and demonstrate SOTA accuracy using visual input only model. I would rate the novelty as high.
summary_of_the_review: Overall I vote for a marginally-below-acceptance. I think the empirical results are overall great and the visual input only model is a novel design over the conventional view-hierarchy based models [Based on my limited understanding of UI view-hierarchy]. However I think the writing quality is below the bar for top tier conference submissions. Disclaimer: :) I am experienced in developing large language model and multi modality models. But I am not an expert in the UI understanding domain.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a purely vision+language model for mobile UI understanding. Compared to past works it does not use the webpage view hierarchy and instead relies on visual cues as its input. A new pre-training method enables the authors to pre-train a large powerful model on a large new corpus. Finetuning on downstream tasks improves on previous methods in spite of them using the view hierarchy of the webpage.

Three reviewers reviewed this paper and provided two borderline reject ratings and one borderline positive rating. Their main concerns were: The novelty of the work, given that the model is similar to Flamingo, the experiments being unfair to the baselines since they do not employ pre-training, the poor presentation and missing details. The authors have made several improvements to the paper adding a lot of detail and improving the presentation of the material. I feel that the model figure can be made clearer, but apart from that, I think the presentation is much improved and this issue is resolved. In terms of novelty, the model itself is similar to Flamingo. But I disagree with the reviewers about the overall novelty of this paper beyond the model design. I think the pre-training strategy, although an extension of past work in pre-training for V+L, is still new to the mobile UI space and adds a lot of value. The pre-training dataset which will be made public adds value to the community. The model, although similar to past work, still makes small but meaningful improvements for this task. In terms of impact, I think people working in this space should read this paper and perhaps build on it. This paper may convince the community to move away from view hierarchies and use pixel information for future tasks. Finally, the reviewers complained about the baselines not having access to pre-training. I think that is the point. This paper shows that in spite of not having view hierarchies, pre-training and fine tuning with this model improves on more traditional methods. To make this argument, I think the baselines are appropriate.

Given my above arguments, I am recommending an accept in spite of the limited enthusiasm shown by the reviewers for this work.
justification_for_why_not_higher_score: I am recommending an accept to this paper in spite of borderline reviews since I think the authors have addressed many points brought up in the reviews, and because the paper is interesting and adds value to the community. I am recommending a poster, since the paper provides limited novelty to a broader audience, outside of UI understanding.
Decision: Accept: poster