Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a unified pre-trained model UPGen for both generation and representation learning.
Like DALL-E/BEiTv1-3, the image data are tokenized into dVAE/VQ-VAE tokens, and then is used to learn MLM on mono-modal data or image-text pair data with a random probability.

strength_and_weaknesses: Strength:

  1.This paper proposes a simple idea.
  2.This paper is well written and organized.

Weaknesses:

1.Lock of novelty. In contrast to DALL-E, this paper uses a BERT-like bidirectional encoding during the pretraining phase. And similar to BEiT-3, pretrain on both mono-modal and image-text pair data, and perform mask-and-reconstruct loss.
2.The reviewer thought the result in this paper was not very satisfactory. 

clarity,_quality,_novelty_and_reproducibility: Lack of novelty, and please revise other mistakes carefully.
summary_of_the_review: I prefer to reject this paper since the novelty, please check the weaknesses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a unified self-supervised learning framework for both representation learning and generation. The proposed approach first discretizes an input image into a sequence of tokens using VQGAN and concatenate them with text tokens. It then trains a bidirectional Transformer encoder using the masked token prediction objective. At inference, it generates image or text by iteratively re-masking and predicting the masked tokens for image or text generation tasks. The paper empirically validates the quality of learned visual representations on ImageNet1K classification and the model's generation ability on MS-COCO, both qualitatively and quantitatively.
strength_and_weaknesses: Strength
+ The paper is well-organized and easy to read.
+ The proposed method, UPGen, is the first to unifies pretraining for both representation learning and generation.

Weaknesses

It seems that the technical novelty of the proposed method is somewhat marginal: It throughly follows the training scheme of previous work, MaskGIT except that it uses additional modality, language. It does not have any contributions in terms of learning methods, model architectures and training/evaluation data. I believe that the model should competitive empirical results for publication in ICLR and the authors should provide the following things:
- I wonder why the linear classification result on ImageNet1K are worse than MAE using very similar objective even though the model is trained on much more image data than MAE. Do you have results using ImageNet1K as pretraining image data?
- I can't figure out the model's image-to-text ability just by looking at the results in Table 2. Could you provide apple-to-apple comparison results, in terms of training data or model capacity? You do not have to conduct new experiments. You can use numbers reported in previously published papers.
- The quality of generated outputs of image-to-text generation shown in Figure 6 is not good. Could you provide quantitative results for this task?
- You argue that the proposed method is the first framework that unifies pretraining for both representation learning and generation and you use large-scale paired or unpaired text data for pretraining. You need to provide empirical results on natural language processing tasks, not just image classification results on visual modality.

Minor comments:
- You should fix typos.
- Does the qualitative examples in Figure 6 come from CC12M or MS-COCO?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and already in a good shape, but the technical novelty seems to be marginal. It does not provide code for reproduction.
summary_of_the_review: I am not fully convinced that it is significant enough to warrant acceptance (see the weaknesses above). I will make the decision after seeing the rebuttal.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper combines two tasks - masked image modeling and masked language modeling - in a single framework, and trains a vision-language encoder to solve both tasks. The resulting model shows some capability for image-to-text generation, as well as reasonable representation learning performance.
strength_and_weaknesses: Strengths:
- This paper is one of the first papers that combine masked image modeling with masked language modeling, which is an interesting direction.
- The proposed model shows the capability of image-to-text generation while also achieving reasonable representation learning performance.

Weaknesses:
1. The motivation - current text-to-image methods have drawbacks due to two-stage training - is not very convincing to me, for the following reasons.
    - The proposed method requires a pretrained VQGAN model, so technically it is also a two-stage method, similar as LDM or Parti.
    - The claimed "drawbacks" - extra hyperparameters and complexity - do not seem to be particularly harmful. On the contrary, a pre-trained image tokenizer or pre-trained text encoder might be necessary to ensure good image generation quality.

2. The experiments are insufficient to verify the advantage of UPGen over existing vision-language representation learning and text-to-image generation methods.
    - In Table 1, UPGen with image-text data achieves the same representation learning performance as UPGen with image-only data. This suggests that the additional language supervision is not effectively utilized. Thus UPGen offers no advantage compared to existing MIM methods such as BEIT.
    - In Table 1, the authors hypothesis that the superior performance of CLIP comes from the larger amount of training data. This hypothesis can be easily verified. The authors can pre-train CLIP on the same data as UPGen and make a fair comparison. The same goes for other baselines such as MAE.
    - More downstream vision-language tasks such as image-text retrieval and VQA are necessary to evaluate the vision-language representation learning performance.
    - The performance of UPGen is far worse than existing text-to-image generation methods. The authors attribute this to the smaller amount of pre-training data used. While it might be difficult to scale-up the pre-training data for UPGen, it is much easier to scale-down the pre-training data for existing methods and make a fair comparison.
    - The image captioning performance does not seem good. What is performance if the model is fine-tuned on image captioning datasets?

4. An important question has not been answered in the paper: whether the unification of representation learning and image generation brings benefit to each individual task?
    
clarity,_quality,_novelty_and_reproducibility: The novelty is not significant. The paper is mostly clearly-written with sufficient implementation details.
summary_of_the_review: This paper proposes an interesting idea similar to BEIT-3, which unifies MIM and MLM with a single model. Unfortunately, the experiments are not well-executed. The paper does not demonstrate the advantage of the proposed method over existing ones. Some claims are also not well-supported.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed UPGen, a unified pre-trained model for both representation learning and generation, which is an encoder-only model based on masked Transformer for multimodal tasks. They show the potential of the masked token prediction model which can be directly used to generate images and language by iteratively re-masking and predicting masked tokens.

Following MaskGIT, they use VQGAN as the image tokenizer, which can tokenize image into tokens and generate image by discrete tokens. Then they combine sequence of image tokens and language tokens as input of ViT after several embedding layers. They use masked token prediction objective in both pre-training step and generation step.
And at the stage of iterative cross-modality generation, the author uses a confidence-based sampling mechanism following MaskGit. From experiment, the authors devote to demonstrating the usage of this model in serving as a representation learning model and generative model for image and language.
strength_and_weaknesses: Strengths:

- This paper is the first one completing representation learning and generative tasks by training on a single objective of masked token prediction. This way of text-prompt image generation and image-prompt text generation is simple but elegant.

- This paper narrow the gap between generation and representation and has great potential in other image tokenizer and masked tokens prediction based model. 

- The experiments are relatively rich and the proposed model can be applied to tasks like representation learning, conditional generation tasks between image and text and unconditional image generation simultaneously.

Weaknesses:

- There are some grammatical errors in this paper. For example: (1) typo in page 2, “Multimodal representation learning for language and image” paragraph, “if we connect these two modalities modalities together” -> “if we connect these two modalities together”; (2) in the second paragraph of the introduction, page 1, it says “is trained conditioned on top of”, the word “conditioned” should be modified; (3) “Following the convention of in prior works of image-to-text” -> “Following the convention of in prior works of text-to-image” (4) “and Zˆ = Z ⊙ M denote the sequence of unmasked tokens.”-> “and Zˆ = Z ⊙ M denote the sequence of masked tokens.””

- The authors skip the details of iterative strategy and image token decoding step, which seems just following MaskGIT and VQGAN. The only novelty in the model is the generation objective.

- What’s more, this work does not show the advantage over other generative model. The quality of generated images do not perform SOTA results. And the Linear Classification accuracy on ImageNet 1K does not achieve comparable performance to prior works. In comparison with the two stage, the authors only indicate that one stage models need less hyperparameters and has less complexity in developing models, which do not show the superiority of one stage models adequately. Although the authors strengthen that they train the model on a small dataset, they still need to compare with the prior works on the same dataset scale to prove their model’s performance and transferability.

- The authors point out UPGen learns generalizable representations that transfer to a wide variety of tasks in page 2. But in the paper, the authors only release the comparison of linear classification accuracy between prior works. More downstream task comparisons should be completed.

- The ablation studies should show more results of different loss weight of image and different mask ratio strategy. In this paper, the authors sample a mask ratio uniformly between 0 and 1 every time in paired image-text tokens and for unpaired text tokens we use a fixed mask ratio of 15%, so the expectation of the mask ratio in paired tokens is 50%. The author need to show the mask strategy of uniformly sampling is better than a fixed one. Also, this work applies a weighting of 1.0 for the loss on the image tokens, a weighting of 0.1 on the paired text tokens and a weighting of 0.01 on the unpaired text tokens, but in the ablation study they only show the accuracy of text loss weight in 1.0 and 0.1. So more weight combination should be compared in the study.

- From the experimental results in Table 1, the reviews can see that the performance of the proposed model is inferior to MAE under the similar data size. The cause of the phenomenon is lacked.

- The model does not perform as well as prior methods in the tasks it aimed to address. In the image to text task, the model can only capture important keywords and the generated text is not fluent with meaningless items in. Why?

-  Lack of experiments trained with image only data. For text only data, the additional unpaired data did not improve the representation learning performance as expected

Suggestions:

- The experiments results of the proposed model should be in bold to make it more clear for readers to follow.

- There are many pictures in the main text, such as figure 5: Examples of unconditioned image generation of UPGen. I am wondering if these pictures are necessary. In my opinion, the figures should be the something like tend chart and so on.

- The ablation studies only introduce  a smaller weight for language loss,and failed to consider using a smaller weight for image loss. It only proves the influence of text on image, not the influence of image on text.

- The model it trained on much smaller datasets,  and thus it can't perform as well as the SOTA methods on each downstream tasks. It is recommended to train with same dataset and compare their performance.  

- The authors say the proposed model can reduce the number of hyperparameters need to be tuned and reduce the complexity of the development process of model. I recommend that the authors may increase the conviction of the experiments by experimenting the proposed model and other text-to-image models on the same datasets and comparing the speed and efficiency.

- The comparative methods are not enough in Section 4.2. The authors may increase the conviction of the experiments by experimenting on more datasets.

- Difficulties in adjusting the weights of predicted image tokens and predicted language tokens in the training process, and the paper does not mention how the weights of image loss and text loss in the model are determined, but only mentions qualitatively that the weight of image loss should be greater. Besides, only two values for text loss weight is listed for comparison, and image loss is not mentioned in the context, which is not sufficient. Please clarify this.
clarity,_quality,_novelty_and_reproducibility: Fair
summary_of_the_review: Although this work puts forward a simple and elegant way of generation, it needs more study of this one stage model to show the advantage. And the performance of this model is still far from SOTA, even though the author strengthen that they pre-train this model on a smaller dataset, they still need to show the comparison between the same dataset scale at least.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 1. The paper presents a unified framework based on a vision-language Transformer trained with a masked token prediction for representation learning and generation.
2. The proposed work first time investigates building a model for representation learning and generation., and obtains interesting results.
3. The paper has some degree of scientific novelty, but may not meet ICLR's very high standards.
4. The achievable performance is not satisfactory.
5. The results could not directly answer the main question: whether the unification of representation learning and image generation brings benefit to each individual task? 
justification_for_why_not_higher_score: 1. The achievable performance is not satisfactory.
2. The results could not directly answer the main question: whether the unification of representation learning and image generation brings benefit to each individual task? 
justification_for_why_not_lower_score: 1. The paper presents a unified framework based on a vision-language Transformer trained with a masked token prediction for representation learning and generation.
2. The proposed work first time investigates building a model for representation learning and generation., and obtains interesting results.
Decision: Reject