Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes screenshot parsing as a simple yet effective pre-training task for visually-situated language modeling (e.g., document QA). The model gets rid of the OCR and directly accepts an image (with rich layouts and rendered text) as input. The empirical results show that the pre-training outperforms prior pixel-only methods by a large margin.
strength_and_weaknesses: The paper makes a welcomed and impressive empirical contribution, with several nice discussions on certain design choices. I will note 3 concerning points:

1. OCR or no OCR?

The paper is in line with the trend of replacing domain-specific components with a general pre-trained neural model, which is generally welcomed. However, in domains where OCR is cheap and adequate, I am not sure if it is worthwhile to spend so much compute on replacing the OCR with a pixel-only LM. 

An ideal solution would be an adaptive model which is pre-trained on such general screenshot parsing task but when adapted to a downstream task, could be fine-tuned to accept domain-specific inputs when there are cheap and easy-to-use solutions. 

2. Scaling up the model parameters seems to only bring marginal performance improvement.

Is there any observation about this? Is it because the pre-training task is too simple? It would also be good if the authors could provide evaluations on the pre-training task itself, using some simple metrics such as blue or exact match. 

3. More details on how the pre-training data are extracted and cleaned should be provided. 

The authors mentioned that the data are from C4. Does C4 already provide the cleaned data in HTML? My impression of C4 was that it was primarily a web-text corpus.

Statistics on the pre-training data should also be provided. E.g., what’s the lengths distribution of the output strings?


clarity,_quality,_novelty_and_reproducibility: The paper is clear and makes a novel empirical contribution.
summary_of_the_review: Overall, the paper shows that screen parsing is an effective pre-training task for visually-situated language modeling. I am on the fence about the design choice of using a pure pixel-only transformer but that does not undermine the core argument for screen parsing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a pretrained image-to-text model for visual language understanding, which has a wide range of application in sources such as textbooks with diagrams, webpages with images and tables, and mobile apps with buttons and forms. The model has a simple and general architecture which only takes an image as the input. It is pretrained by parsing masked screenshots of webpages into simplified HTML, which subsumes common pretraining signals. This paper also proposes an integration of language and vision inputs for fine-tuning the model. The model achieves SOTA results in six out of nine tasks across four domains.

strength_and_weaknesses: Strength:

1. Pix2Struct uses a general-purpose pixel-to-text design, which simplifies the model architecture and can be easily applied to multiple domains.
2. The masked screenshot parsing objective is intuitive and effective. The pretraining data is easy to obtain from the web. The warmup stage with an image-to-text curriculum further improves the pretrained model.
3. The proposed fine-tuning strategy seamlessly integrate language and vision inputs by rendering language prompts on the image, without changing the architecture.
4. A single pretrained model achieves strong performance on multiple visual language understanding tasks from diverse domains.

Weakness:

1. The presentation of this paper is easy to follow, but not clear enough. Many important techniques are described at a high level, for example 

    (1) How the proposed variable-resolution input representation is implemented, how to determine the number of patches for width and height, do small and large inputs have the same number of patches?

    (2) How the pretraining data is collected and processed, especially for the warmup stage. 

    (3) How the masked parts are selected?

2. The authors do not mention if the data/code/model will be publicly available. It would be difficult for the community to reproduce or follow if they cannot be released.

3. Some of the designs lack empirical justification. For example, are there alternative ways to fine-tune the model, such as concatenating image and text? Does the location/size/font of where the prompt is rendered affect the performance? 

clarity,_quality,_novelty_and_reproducibility: The idea presented in this paper is novel and the quality of the proposed method is good. 

I have concerns about this paper’s presentation and reproducibility. As mentioned before, many technical details are missing. The paper uses a lot of space to explain the downstream datasets and previous models for them, which seems to me unnecessary. I would suggest to use a figure for each dataset to explain the task and how Pix2Struct handles it, move the details to appendix, and use more space to describe the data processing or modeling of Pix2Struct in more detail. 

This paper can be easily understood, but the model cannot be easily reproduced. I appreciate the authors’ effort for processing the web data and designing the model, which requires a lot of engineering. However, it is hard to follow this research if they cannot be made public.
summary_of_the_review: This paper proposes a simple pretraining framework for visual language understanding. The simplicity of the model, taking only image as the input, enables it to apply to diverse tasks and achieve supreme results. It is an exciting innovation in this area, and I believe it would greatly benefit the community if the data/model can be made public.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents Pix2Struct, a pretrained image-to-text model. The model is pretrained to parse masked screenshots of web pages and the authors show that it can be fine-tuned on multiple tasks containing visually-situated language (VQA, Image captioning, Infographic VQA). The fine-tuned models achieve state-of-the-art results in six tasks, while using only pixel-level inputs. The authors also introduce a variable-resolution input representation to prevent the distortion of input images.

strength_and_weaknesses: Although interesting and very well written, this paper does not make major contributions to the field. 

The first contribution concerning the pre-training strategy is quite interesting. The authors propose a strategy based on web pages screenshots that allows to collect quickly and easily a very large amount of data. Moreover, these data can be quite varied and contain a large variety of elements, which allows to train a model that can work on many tasks. 

The second contribution consists in adding modifications to the transformer inputs to handle variable aspect ratios and resolutions. This strategy, although interesting, brings little gain to the results. Moreover, it does not constitute a major contribution to the paper.

For the experiments, the following comments should be addressed.

1. The idea of simplifying the multi-modalities problem is interesting. However, it could have been nice to compare the method to a standard approach with a modality combination.
2. The input masking is done using crossed-out opaque bounding boxes. There is no justification about this choice. I wonder if this choice has an impact on the results compared to other approaches like removing the boxes or using white bounding boxes.
3. The authors claim that the changes they added to the standard ViT inputs provide major advantages in terms of robustness to extreme aspect ratios and on-the-fly changes to the sequence length. However, except for the results presented on Figure 4, there is no experiments showing these advantages.
4. It is not clear whether the modifications applied to the screenshots and HTML only consist in masking some texts or if other modifications are applied.
5. I would have appreciated to have the inference times of the base and large models.
6. The method of Wang et al (2021a) for the Screen2Words task could be more detailed.
7. It is not said on which task and dataset the ablation study on the inputs resolution is carried on.
8. 13 articles are cited as ArXiv preprints, please update the citations for articles that have been published.
inor comments:

1. In Table 2, it would make the table clearer to add a line with the metrics.
2. In Table 2, the value 160.4 should be highlighted instead of 145.0.
3. The values should be given with the same number of significative numbers:
    - 11.27 in AI2D
    - 145 in TextCaps
    - 40 and 81 in InfographiVQA
4. Figure 3 is a table, it should be renamed Table 3.
5. Typo:
    - Figure 4 caption: "Our variable-resolution inputs prevent**s**..."
6. In the appendix
    - In the fine-tuning section, it is said "Tables 4 and 4", it should be corrected to "Tables 4 and 5". By the way, tables 4 and 5 are quite similar, they could be merged into a single table.
clarity,_quality,_novelty_and_reproducibility:  I have a major concern regarding the applicability and the reproducibility of the results. The base and large models contain many parameters and the pre-training requires very important resources which represent a huge financial and ecological cost. This could be an argument for not using this pre-training method in other applications. Moreover, few organisations or companies have the capacity or even the desire to devote so much money and resources to such pre-training, which makes the results non-reproducible.
summary_of_the_review: Although the proposed method for pre-training a model from web page screenshots is interesting, the conditions of the experiment, with very large amounts of data and mobilizing unreasonable computational resources, do not allow to highlight its interest, its applicability and the reproducibility of the results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:   
This paper proposes a pretrained image-to-text model for visual language understanding. The proposed model, Pix2Struct, is pretrained by learning to parse masked screenshots of web pages into simplified HTML. The model is OCR-free and directly accepts an image as input, which is thought to be computationally heavy but more general. Its fine-tuned model achieves SOTA results in six out of nine tasks across four domains.

Strengths:  
1. The proposed pretraining strategy is intriguing and seems practically useful.  
2. The model achieves good performance on wide range of downstream tasks.

Weaknesses:  
1. Reproduciblity of the method is questionable mainly due to the lack of details in presentation. Also, it is still not certain whether the code is really available.
2. The method requires huge amount of data and computational resource, although this is a common problem for this kind of pretraining.

justification_for_why_not_higher_score: While the all reviewers agree that the proposed vision–language pretraining strategy is quite interesting and thought to be useful, the central concern is the reproducibility issue because the method involves many algorithmic details as well as huge amount of web data and computational resource. The authors replies that the code and pretrained checkpoints are open sourced, but unfortunately, we cannot confirm them at the link added in the paper. Yes, we know it is a dummy for anonymization and maybe the code is really published somewhere, but we cannot check them without actively searching the web and breaking authors' anonymity, which is not allowed. (at least, the authors could have submitted a part of the code with anonymization as a supplementary material)

Related to this, one reviewer raised a concern regarding the presentation of the paper that many important techniques are described at a high level and unclear. It seems that this comment was not addressed very seriously (just saying 'we have open sourced the code') and its not clear how the paper will be improved. Since neither the code nor the better description can be verified, the AC believes that the reproducibility issue still remains. There was no strong support from the reviewers, so the AC would like to recommend rejection accordingly.
Decision: Reject