Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a new way to pre-train VLMs (vision-language-models) using both text generation and image generation as the VLPs (pre-training objectives).

They then ablate the objectives on their model (and show that each objective’s inductive bias is actually helping) and the datasets used for training (smaller more curated datasets being better in general) both their model and a reimplementation of SimVLM.

strength_and_weaknesses: Strengths:
 - All required ablations of the VLPs
 - All required ablations of the dataset sources
 - Comparisons with appropriate re-implementations of baselines.
 - Comparisons are over a varied class of task suites (NLU, Image Understanding, Multimodal Understanding, Image Generation, NL Generation) and actual benchmarks on 27 tasks.

Weaknesses:
 - Not necessarily a weakness: Since the “prefix” of the image is more seen than the suffix, I wonder if this leads to an inductive bias that the image will be less likely to paint the prefix of the image itself. i.e. say if the most important aspect of the image falls in the top-left block of the image. I wonder if the authors considered other ways of masking the image, by either just in-painting (i.e. “missing tiles”) instead of suffix-painting. This could be left for future work.
 - Also not necessarily a weakness: Right now images are treated in a hybrid manner, image input to the model is treated in pixel space (first three blocks of resnet extracts the feature maps), however the suffix that needs to be generated is in the token space (VQGAN tokens). One could have resorted to also taking the whole image as a sequence of VQGAN tokens, projecting a prefix of them to the hidden dimension of the backbone network and that prefix would be the input, while the rest of the suffix became the output. This could be left for future work.
 - Re: SimVLM base size, did the authors contact the SimVLM authors to get the exact specifics of the SimVLM base size and architecture. Although SimVLM_small seems to be replicated exactly, it is also possible that some bugs remain in the reimplementation and SMALL was small enough for these not to matter.

clarity,_quality,_novelty_and_reproducibility: Clarity & Reproducibility:
Certain details either aren’t clear to me, and I couldn’t find them in the paper (apologies if they were there):
 - On the PIM objective, it is not clear how the loss is propagated back (gumbel softmax?), or we’re just doing cross-entropy loss on the VQGAN tokens we need to generate?
 - Page 5, Table 1 - LWD - What is DaVinci 200M dataset?
 - How are the Object-Region Datasets used, i.e. what is the text part of the image example? How is an example formulated?
 - The authors don’t make it clear how “dynamic masking” is done, i.e. what is the probability distribution of the prefix sample? Is it uniformly randomly distributed, or something else, what are the parameters of this distribution? I don’t find this information anywhere.

Quality:
Paper is quite well written and is almost free of typos [1]

Novelty:
New way to pre-train VLMs, although this is novel, this is actually a straightforward extension of the way things are headed. Parti / DallE-1 both have image generation using VQGAN tokens and both PaLI and Parti have decoders.

Miscellaneous:
Neither Parti, not PaLI is mentioned and they seem relevant.


[1] Table 2, Page 7 “and they use the use mid-training”, extra “use” before mid-training.


summary_of_the_review: Leaning towards acceptance, since this seems a good enough empirical contribution and a “proof of existence” of image generation working as a pre-training objective to make the model stronger.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, a generative vision-language model that can perform both image-to-text generation and painting text-to-image generation, called DAVINCI, is proposed. In DAVINCI, prefix language modeling and prefix image modeling are adopted as the training objective, which make the model simple and scalable. On 27 generation/understanding tasks, DAVINCI achieves highly competitive results.
strength_and_weaknesses: Strengths:

1.A simple and unified foundation model for vision-language pre-training is proposed.
2.Compared with previous unified vision-language foundation models and other vision-language models, the proposed model shows highly competitive results.
3.Various ablation studies have confirmed the advantages of the proposed model.

Weaknesses:

1.The authors should conduct more rigorous experiments and comparisons. For example, in Tab. 3, the authors only listed the results of SimVLM (1.8B), OFA and Florence, and claimed that these results should not be directly compared. However, to clearly demonstrate the effectiveness of the proposed method, the authors could change the training settings (e.g., train on the same data of OFA or use a model with the same size as Florence) to make fair comparisons with these models.
clarity,_quality,_novelty_and_reproducibility: Clarity:

The presentation of this paper is excellent.

Quality:

The paper has a good quality in general.

Novelty:

The novelty of this proposed model is moderate.

Reproducibility:

I believe this work can be reproduced easily.

summary_of_the_review: This work is overall a good contribution to the research community. I would recommend acceptance of this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new unified model named Davinci, which is trained with prefix language modeling and prefix image modeling. Davinci is able to do images and text generation as well as other vision and language understanding tasks. The authors carefully benchmark the performance of different vision and language pre-training objectives on different scales of pre-training datasets and achieve competitive performance. 
strength_and_weaknesses: [Strength]

- The proposed method combines prefix language modeling with prefix image modeling, which is a natural and simple extension. 

-  Experiments ablating different objectives, as well as different dataset sources and showing the effectiveness of the different types of pre-training data. 

- Similar to OFA, the proposed method can do text-to-image generation tasks as well as vision and language understanding tasks. 

[Weakness]

- From the introduction, the proposed method is motivated by auto-regressive models like GPT3 and PaLM. GPT3 has few shot abilities, and PaLM can multi-task and even do zero-shot tasks. One key property of these two models is they can use the same set of parameters for many tasks. However, the proposed method still falls into the first pretrain and then finetune formula. 

- The authors claim that VL-T5 and OFA are hard to scale up since it is non-trivial to collect a large number of vision and language datasets for pre-training. I hold a different opinion, the pre-training tasks (mask language modeling and mask image modeling) can be one of the tasks, and thus the model still benefits from a large training corpus. 

- Another big concern of the proposed method is the model size. The paper only shows models with 152M parameters. Thus it is not clear whether the proposed method can scale to a large model. 

- The pre-training datasets also contain supervised datasets such as COCO, refer coco, VG, etc. Given the claim of OFA and VL-T5, the proposed method suffers a similar scaling problem. 

- In Table 3, the author didn't compare OFA and simVLM (1.8B). Although OFA uses some additional tasks, its pre-training corpus is substantially smaller compared to the proposed approach. 

- for prefix language modeling, the author randomly samples the mask ratio,  is it from (0, 1)? or some other span? Table 5 shows the results of the ablation study. I am more interested in the effect of image masking strategies: such as random masking vs. masking at the end. Different ratios of masking etc. However, there is no experiment on that. 
clarity,_quality,_novelty_and_reproducibility: Most of the part of the paper is clear, however, some of the details of the experiment and model setting is missing, 
summary_of_the_review: This paper proposed a new unified model named Davinci, which is trained with prefix language modeling and prefix image modeling. The proposed method is simple and shows some potential for training a unified model. However, there is multiple weakness in the paper that require the authors furhur clarify. (check the weakness section for details).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a vision-and-language transformer that is multi-tasked to both generate an image completion, given language ("Prefix Image Modeling") and to both generate language given images ("Prefix Language Modeling"). The key claim of this paper is that by multitasking these two settings, the resulting model is able to do well on a wider range of vision-and-language tasks.

Despite not being a very large model, (it's BERT-Base sized), it performs favorably to SimVLM and FLAVA (two recent very large-scale approaches) in a finetuned setting, on both language-only and vision-language tasks. An ablation study (Table 5) suggests that these objectives are complementary.
strength_and_weaknesses: Strengths:
* The proposed model does well while being quite simple, which in turn suggests it has potential to be widely adopted (especially considering how many transformer implementations are already very well-optimized). The paper is easy to read as well.
* I appreciate the comparison with a variety of past models covering NLP, Image Classification, and vision-and-language. 

Weaknesses:
* I'd like to see a comparison with this paper if possible CM3 A Causal Masked Multimodal Model of the Internet https://arxiv.org/abs/2201.07520 - because it (like this proposed approach) uses a Discrete VAE token-based approach.
* Most of the tasks specified in the vision-and-language space seem focused on literal recognition of what's in images. I'd be curious as to how this model stacks up on Visual Commonsense Reasoning (visualcommonsense.com) for instance.

Perhaps less important but still important: I'm a fan of this paper overall, but I do wonder why it takes 10 days to train on 1024 V100 GPU's. That's quite a lot for a BERT-Base model -- and in turn, it makes me think that one of the main reasons this paper does so well is because it was just trained with more effective compute than FLAVA or SimVLM. That doesn't mean that this paper is bad by any means, but I would be happy to increase my score if there was some experiment showing e.g. the performance of DaVinci over the course of pretraining -- e.g. as a function of the compute spent, both a) to compare better with prior work, as well as to b) show if that level of pretraining compute was necessary.

Another related question relates to the ablations in Table 5. For PLM only vs. PIM only, vs. the full model -- are the ablations only trained with half the effective compute as the full model? (as they're trained on half as many sequences) I'd be curious as to how they do given double the pretraining compute.
clarity,_quality,_novelty_and_reproducibility: The paper seems clear and well-written. I think the idea is mostly novel, but with significant overlap to CM3 from what I understand, which I'd like the authors to clarify in rebuttal.
summary_of_the_review: I'm leaning positive on this paper but I'd like some of my concerns to be addressed in the author response first :)

---

Update post-review: I think the authors addressed my main concerns, so I'm bumping up my score to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Reviewers mostly liked the paper! Accept! Congrats! 

Reviewers thought that
- "This work is overall a good contribution to the research community." 
-" Leaning towards acceptance, since this seems a good enough empirical contribution and a “proof of existence” of image generation working as a pre-training objective to make the model stronger."

Authors did a good job in convincing the reviewer to increase their score. 
justification_for_why_not_higher_score: Seems interesting but maybe not good enough for spotlight? 
justification_for_why_not_lower_score: Paper is interesting and should be accepted.
Decision: Accept: poster