Connecting representation and generation via masked vision-language transformer
Authors: Xinyang Geng
Lisa Lee
Igor Mordatch
Sergey Levine
Pieter Abbeel
Hao Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Representation Learning
Pre-training
Generative Model
Conditional Generation

Abstract: Recently, there has been great progress in the self-supervised pre-training of multimodal representation models that understand image and language jointly. One particularly popular application of such models is text-to-image generation, which is typically obtained via a two-stage process: in the first stage, a representation model is trained via self-supervised objectives; then in the second stage, a conditional generative decoder is trained on top of the representation to generate natural images. In this work, we aim at bringing representation learning and conditional generation together by unifying the two stages into a single model and training objective. We present UPGen, a unified pre-trained model for both representation learning and generation. UPGen is trained with a simple masked token prediction objective on a flexible mixture of image and language data. We use a pre-trained VQGAN image tokenizer to convert images into discrete tokens, then train a masked token prediction model on both paired image-text datasets and unpaired language datasets, using randomly sampled mask ratios. We show that this masked token prediction model can be directly used to generate images and language by iteratively re-masking and predicting the masked tokens. We demonstrate empirically that UPGen serves as both a good representation learning model and a generative model for both image and language.

Under review as a conference paper at ICLR 2023
CONNECTING REPRESENTATION AND GENERATION VIA MASKED VISION-LANGUAGE TRANSFORMER
Anonymous authors
Paper under double-blind review
ABSTRACT
Recently, there has been great progress in the sule--upe pre-training of mul- timodal representation models that understand image and language jointly. One particularly popular application of such models is text-to-image generation, which is typically obtained via two-stage process: in the first stage, representation model is trained via self-supervised objectives; then in the second stage, a con- ditional generative decoder trained on top of the representation to generate natural images. In this work, we aim at bringing representation learning and conditional generation together by unifying the two stages into a single model and training objective. We present UPGen, unified pre-trained model for both representation learning and generation. UPGen is trained with simple masked token prediction objective use a pre-trained VQGAN image tokenizer to convert images into discrete tokens, then train masked token prediction model on both paired image-text datasets and unpaired language datasets, using randomly sampled mask ratios. We show that this masked token prediction model can be directly used generate images and language by iteratively re- re-masking and predicting the masked tokens. We demonstrate empirically that UPGen serves as both a good representation learning model and a generative model for both image and language.
flexible mixture of image and language data. We
INTRODUCTION
With the rapid improvement of deep learning architecture and accelerator hardware, researchers have made significant progress in self-supervised representation learning from image and text data (Radford et al. 2021 Geng et al. 2022 Mu et al. 2021 Wang et al. (022). Such models are trained to jointly understand language and image data and learn generalizable representations that transfer across image and language modalities, and thus can be applied to wide variety of downstream tasks, One interesting task that has gained popularity recently is rext-to-image generation, where the model is given text prompt and generates image that corresponds to the text prompt's description (Sabaria et al. 2022 Ramesh et al. 2021 Yu et al. 2022]. This task is particularly attractive because it enables human to directly interact with the model and inspect its understanding of language and image, providing great tools for artistic creations.
Driven by the successes of representation learning, text-to-image models have achieved impressive progress (see e.g., Saharia et al. 2022; Ramesh et al. 2021 Yu et al. 2022). However, the drawbacks of these text-to-images models are due the pipeline having two stages. In the first stage, representation model is trained with self-supervised pre-training objectives. In the second stage, diffusion (Saharia et al. 2022; Ramesh et al.) 202T) or autoregressive (Yu et al., (2022) generative model is trained conditioned on top of the (typically frozen) pre-trained representation. Such two- stage training pipeline requires more hyperparameters be tuned, and introduces extra complexity in developing models.
To address the above limitations, we propose UPGen, a simple but effective framework that unifies pre-training for representation and generation. Using a pre-trained VQGAN model (Esser al.) 2021), we convert an image into a sequence of discrete tokens and concatenate the image tokens and language tokens into a single sequence. We then train encoder-only transformer model using simple masked token prediction objective on the concatenated sequence. The masked token prediction objectives produces good representations for downstream tasks. Furthermore,

---

Under review as a conference paper at ICLR 2023
Visual token
Language token
Mask token
Encoder
VQGAN
VQGAN Decoder
tok
tok2
tok3
tok4
tok5
tok6
tok7
Encoder-only Bidirectional Transformer
tok1
tok4
tok6
Lok?
UPGen consists of a pre-trained VQGAN tokenizer that converts images into discrete tokens and an encoder-only transformer that processes the image tokens and language tokens jointly, The concatenated image and language sequence are randomly masked according to uniformly sampled ratio, and the transformer trained predict the masked tokens.
Figure 1:
show that UPGen be directly used for conditional and unconditional generation of images and language without having to train extra components. To achieve this, we apply an iterative refinement strategy by repeatedly re-masking and regenerating the masked tokens, following the approach of MaskGIT (Chang et al. 2022).
In this work, we provide large-scale empirical study of UPGen on a mixture paired image-text datasets and unpaired language datasets. We find that UPGen learns generalizable representations that transfer to a wide variety tasks, including image classification, text-guided image inpainting. and image-to-text generation. We also demonstrate that UPGen can generate high-quality images with and without language conditioning on language prompts. While achieving competitive results, UPGen does not perform well state-of-the-art methods each of the downstream tasks due to being trained on much smaller datasets. However, to the best of our knowledge, UPGen the first model that combines representation learning, image-to unconditioned image-generating into a single model and training objective. Scaling UPGen to larger datasets and model size is left as promising future work.
generation, text-to-image- ti--to--maageggeerattionn and
RELATED WORKS
Self-supervised learning via masked modeling. Ever since the introduction of Transform- ers (Vaswani et al. 2017), self-supervised pre-training has made significant progress in the recent years. A particularly popular style of self-supervised learning is masked modeling, where the input example is partially masked and the model is trained to predict the masked part from the unmasked part. Masked modeling first its success in natural language processing (NLP), with large language models like BERT (Devlin et al. 2018). RoBERTa (Liu et al. 2019). T5 (Raffel et al. 2020) and UL2 (Tay et al.) 2022) that learns highly generalizable representations that transfer well to various of downstream tasks. Inspired by the effectiveness of these NLP models, researchers have taken the masked modeling ideas into computer vision and also met with great success. Vision transform- ers (Dosovitskiy et al.] (020)) introduces the transformer architecture into computer vision and make the natural analogy between language tokens and patches of natural images. Building on this work, BEiT (Bao et al. 2021) proposes to apply the masked token prediction on vision transformers, taking image patches as input and predicting the output of a discrete tokens produced by a VQVAE (Van Den Oord et al. (017) model. Recently, MAE (He et al.) 2022) further removes the need of a discrete image tokenizer by directly predicting the patch pixel output using an encoder-decoder architecture. Our UPGen first tokenizes the image into discrete tokens by using pre-trained VQGAN (Esser et al.
2

---

Under review as a conference paper at ICLR 2023
Image-to-text
Text-to-image
lok.
Tol2
Tok-
toleel
tols
Encoder-only Bidirectional Transformer
Encoder-only Bidirectional Transformer
tok1
tok2
Figure 2: Pre-trained UPGen can be directly used for cross- cross-modality generation: Image-to-text generation (Ieft) and rext-to-image generation (right) by inputting sequence of masked tokens.
2021) model, and then follows the simple masked token prediction objective of BERT. UPGen differs from BETT mainly in the way that it uses the discretized image tokens as both input and output, which facilitates the generation process. In addition, our model is trained with both image and language data, which connects the information from different modalities together.
Multimodal representation learning for language and image. While self-supervised learning on image language data alone has proven great success in extracting useful information and producing generalizable representations, recent works have shown that the such capability can be greatly improved if we connect these two modalities modalities together. Multimodal self-supervised learning models achieves this by jointly learning on paired data of two modalities. One popular approach for multimodal pre-training is contrastive learning. Contrastive learning approaches such as CLIP (Radford et al. (02T), SLIP (Mu et al.) 202T) and FLAVA (Singh et al.) (202T) connects representation between different modalities together by maximizing the similarity between positive example pairs and minimizing the similarities between negative pairs. Such objectives are limited to paired image-text datasets, and extending the pre-training to unpaired data often requires the
introduction of additional objectives. Recently, with the advancement of masked modeling techniques for images, masked modeling has gained a lot of popularity as an alternative objective for contrastive learning due to its simplicity and flexibility. M3AE (Geng et al.) 2022) extends the MAE approach to both image and language by combining image patches and language tokens as a single sequence. BEiTv3 (Wang al., (022) also combines image patches and language tokens, but predict discretized image tokens from a VQVAE model as output. Both of these models uses simple masked token prediction objective. Unlike M3AE and BEiTv3, UPGen uses discretized image tokens for both model input and output in order to perform generation. Moreover, UPGen is trained with uniformly removing the need tune mask ratio for each modality.
sampled mask ratios from
text-to-image generation. saat-to-image generation has been popular field with various approaches being proposed recently. Typically, text-to-image models follow the general pattern of pre-training a language encoder model via self-supervised objective and then training a generative image decoder conditioned on the language representation. Therefore, these models can be characterized by their language encoder type and their generative decoder type. From the language encoder perspective, DALL-E (Ramesh et al., 2021). DALL-E 2 (Ramesh et al. 2022) and Stable Diffusion (Rombach et al. 2021) uses the multimodal CLIP as their language encoder. Imagen (Saharia al., 2022) uses T5 (Raffel et al.) 2020), a pure language model, and Parti (Yu et al. 2022) trains a BERT-like transformer its language encoder. GL.DD (Nichol et al.) (021) trains transformer with the generative model an end-to-end way to process language. From the generative model perspective, most approaches including DALL-E 2, GLIDE, Stable Diffusion and Imagen uses diffusion mod- els (Ho et al. 2020) as its generative decoders. DALL-E and Parti uses autoregressive models in the discretized image token space as the generative decoders. Unlike these prior approaches, our model combines the representation encoder and generative decoder into single transformer model. We show that the UPGen can complete both representation learning and generative tasks by training on single objective of masked token prediction.
3 UNIFYING REPRESENTATION PRE-TRAINING AND GENERATION
In this section introduce method UPGen, which is an encoder-only transformer model that takes the masked sequence of image and language tokens as input and predicts the masked tokens. We summarize the architecture and training objectives in Figure [
3

---

Under review as a conference paper at ICLR 2023
VQGAN-based image tokenization. In order apply the transformers architectures to images which do not exhibit natural sequence structure, prior works typically use one of two approaches: slice the image into patches and treat these patches
pre-trained discrete representation model such as VQVAE (Van Den Oord et al., 2017), DVAF (Ramesh et al. 2021) or VQGAN (Esser et al. (02T) convert the image into sequence of discrete tokens. In this work, choose the latter approach for many practical reasons. Firstly, the discrete representations bring the images closer to language in terms of format, allowing us use the same input processing and cross-entropy loss function for both image tokens and language tokens. Moreover, the discrete-token classification output of our model naturally forms a probability distribution with high degree of freedom, allowing us to directly sample from the predicted token classes during generation without having to rely on extra probability distribution assumptions. For simplicity, we directly use the open- open-source pre-trained model from MaskGIT (Chang et al. 2022). which is a highly effective ConvNet-based VQGAN model trained on the ImageNet IK dataset (Deng et al. 2009p. We keep the VQGAN model fixed throughout our training process.
sequence,
UPGen Transformer architecture. After using the pre-trained VQGAN to convert an image into a sequence of tokens, we concatenate the image tokens and language tokens into a single sequence. For unpaired examples, we simply take the sequence of image or language tokens. We apply the standard trainable word embedding layer obtain a sequence of vectors. In addition to the standard positional embeddings, we also add modality-type embeddings to enable the model to distinguish between two modalities, following the approach of M3AE (Geng et al.. 2022). We feed the sequence of vectors into an encoder-only Transformer network, and apply a final linear projection on the output of the Transformer obtain the classification logits.
Masked token prediction training, We train UPGen with a simple masked token prediction objective. With the combined sequence of image and language tokens, we first randomly sample a mask ratio uniformly between 0 and 1 for each example in the batch, following the approach of MaskGIT (Chang et al., 2022), and then randomly replace the tokens in the combined sequence with a special mask token according to the ratio. We feed the masked sequence into the Transformer, and supervise the output with the ground token only at the masked position, similar to BERT (Devlin et al. 2018).
Specifically, let X = [ denote the latent tokens obtained by processing the image with the VQGAN encoder, where N is the length of the reshaped token matrix, and Y = [yr] denote the text tokens obtained via the text tokenizer, where K is the number of text tokens. During training, we sample subset of tokens and replace them with a special [MASK] token. Let M = {0, 1) ()(
denote a randomly generated mask, we mask the token I; or yj if Mii] = or M[i = 1. Let
Z X, Y denote the sequence of image tokens and text tokens, and Ž Z @ M denote the sequence of unmasked tokens. The training objective is to minimize the negative log-likelihood of the masked tokens:
L L=
E ZED
Σ
logp(z; pa (2; (z)))
log
ViE[I,N M(1]=
where Z E 'D denotes over all image-text sequences. Note that when Z X, Equation I reduces to MaskGIT (Chang et al., 2022) which does image-only masked token prediction. When Z = Y, Equation rrduucss to training a BERT (Devlin et al. 2018) on language tokens with random mask ratio. Since the model is agnostic what Z consists of, i.e.. can be image-text pair, text-only and image-only data, we train UPGen on not only image-text pairs but also text-only data to enable better language understanding.
Iterative mross-modality generation. During inference, since the model has been trained to model any set of unconditional and conditional probabilities, we can sample any subset of tokens per sampling iteration, from the extreme cases of sampling all tokens (independent) versus sampling each token a time (autoregressive). We follow MaskGIT (Chang et al. (202)) to use a confidence-basee sampling mechanism. Specifically, at each iteration, we only keep the predictions that the model is most confident with and discard the rest of predicted tokens and replace them with mask. We repeat this process until all tokens are generated.
We can control the model to condition on image tokens to predict text tokens by inputting a sequence of image tokens followed by sequence of [MASK] tokens. The same can be done for text-to-imant -mage generation. The process is illustrated in Figure 2 For text-to-image generation, we visualize the
tokens generated at intermediate iterations in Figure 3
4

---

Under review as a conference paper at ICLR 2023
A cup of coffee with latte art
A large dog in the park
Figure 3: Visualization of the iterative text-to-image generation process UPGen. The text prompt is shown below the image generation process.
EXPERIMENTS
In this section, we conduct empirical study of UPGen and present the results. Since UPGen unifies representation learning and generation into a single model and training objective, important to evaluate its performance from both perspectives. Hence, in designing the experiments, we aim to answer the following questions: (1) As representation learning model, does UPGen learn generalizable visual representations that transfer well to downstream tasks? (2) As generative model, can UPGen generate high-quality image and text samples?
To answer these questions, pre-train UPGen pairs. In addition to the paired data, we also use a combination unpaired language-only data. We evaluate the representation quality of pre-trained UPGen models image classification tasks, and we evaluate the image and text generation by inspecting the generated samples both qualitatively and quantitatively.
large and diverse dataset of image-language
4.1 UPGEN PRE-TRAINING
Datasets. In pre-training UPGen, a combination of image-text paired datasets and text-only unpaired datasets. For the image-tex paired dataset, we use the publicly available Conceptual Caption 12M (CC12M) (Changpinyo YFCC100M (Thomee et al., 2016) selected according to (Radford et al. 2021). For Conceptual
al. 202I), Redcaps (Desai et al. 2021). and 15M subset of the
Captions 12M, since the dataset is provided in the form of online image URLs,
only able
retrieve 1OM images due to some expired URLs. The combination of these three datasets give us total of 36M image-text pairs.
For the unpaired text-only dataset, we follow the practice of BERT (Devlin et al. 2018 and use mixture of multiple publicly available text corpus. Specifically, we combine the text datasets of Wikipedia (Foundation). CC-News (Hamborg et al. 2017), OpenWebText (Gokaslan & Cohen) and Toronto BookCorpus (Zhu al. 2015).
Model architecture and input processing. UPGen's main model is a bi-directional encoder-only transformer (Vaswani et al.) (207)) network. We adopt the convention of Vision Transformers (Doso vitskiy et al., (2020) and use a ViT-Large sized model. The pre-training takes the combined sequence of image tokens and language tokens as input after applying fixed sinusoidal positional positional embedding and learnable type embedding. For the VQGAN image tokenizer, we use the pre-trained model from MaskGIT (Chang et al. 2022), which is a convnet- based encoder and decoder VQGAN pre-trained on ImageNet. The tokenizer converts 256x image into 256 discrete tokens, where the codebook size 1024. We list the detailed architecture and input processing information in Appendix(M A
Pre-training. For all models, we pre-train for 50 epochs on the combined dataset with cross-entropy loss on the masked tokens. For paired image-text tokens, sample mask ratio uniformly between 0 and every time and for unpaired text tokens we use a fixed mask ratio of 15%. For the default model, apply weighting of 1.0 for the loss the image tokens, weighting of 0.1 the paired text tokens and a weighting of 0.01 on the unpaired text tokens. Detailed training hyperparameter information be found in Appendix A
5

---

Under review as a conference paper at ICLR 2023
A group of animals standing in the
Galaxy colored apple
Black forest cake with many cherries
A nice cheese pizza
A crowd of people standing on top of a beach Figure 4: Examples of text-to-image generation of UPGen. The two images for the each prompt are generated with different random seeds.
A large seagull sitting on post
4.2 REPRESENTATION LEARNING
Following the convention in prior works, we evaluate the representation learning performance of our model by performing linear classification on top of the learned representation on ImageNet K We freeze the weights of the UPGen pre-train and take the embeddings of all the intermediate layers. We concatenate these embeddings and train a linear classification layer top. We summarize the results in Table The results show that UPGen achieves comparable performance to prior works in representation learning, indicating that UPGen can serve simple but strong alternative existing visual representation learning methods. We do note that CLIP outperforms our model significantly, and hypothesize that this is the result of CLIP being trained proprictary dataset 10 times larger than the one we use.
Model
UPGen
UPGen (paired data only)
UPGen (image-only)
Accuracy
70.18
70.18
70.17
Model
CLIP (YFCC100M)
MAE (ImageNet)
M3AE (CC12M)
Accuracy
83.9
73.5
64.1
Table 1: Linear classification accuracy ImageNet 1K validation set. All models based ViT-L with the same number of parameters. In the top row, we present the accuracy of UPGen pre-trained with both paired and unpaired data, with unpaired data only and with image data only. In the second row, we show the performance of visual representation learning methods trained on other datasets. We see that UPGen reaches similar performance other representation learning methods.

---

Under review as a conference paper at ICLR 2023
Figure 5: Examples of unconditioned image generation of UPGen. Even without conditioning text prompts, UPGen is still able to generate images that contain diverse objects.
4.3 IMAGE GENERATION
Qualitative results. We perform qualitative evaluation the image generation of UPGen. We present selected text-to-image generation results in Figure We see UPGen is able to generate mostly coherent images that correspond well the description of the text prompt. It can be observed that these images contain certain unnatural artifacts. We hypothesize that this is largely caused by the specific VQGAN image tokenizer we use, which only trained on ImageNet and therefore is not able to encode objects not included in the ImageNet dataset. We also present unconditioned image generations in Figure 5
We see that even without conditioning on text prompts, UPGen can still generate images that contain natural objects. This further demonstrates the generality of our UPGen training objective.
Zero-Shot FID score MS-COCO Following the convention in prior works of image-to-text generation, we use the FID (Heusel et al. 2017) score on MS-COCO (Lin et al. 2014) validation set quantitative metric of quality of generated images. To perform the evaluation, 30K captions from the annotations MS MS-COCO 2014 validation set and generate images from these captions. We then calculate the FID score against the original images in the validation set. This setting is known as zero-shot FID score since our model is not trained or finetuned on MS-COCO. We present the results in Table 2 While text-to-image generation, we note that our model is trained with a much smaller dataset, and to the best of our knowledge, UPGen is the first model that unifies representation learning and generation (of text and language, both conditioned and unconditioned) into single model and training objective.
sample
model does perform as well as prior methods for
Model
UPGen
DALL-E
StableDifllsion
GLIDE
DALL-E2
Imagen
Parti
FID-30K score
65.25
28
12.63
12.24
10.39
7.27
7.23
Table Zero shot FID-30K score evaluated on the MS-COCO validation set.
4.4 TEXT GENERATION
To evaluate the quality of image-to-text generation of UPGen, we take several images from the MS-COCO validation set and use our model to generate captions for these images. We perform the exact same sampling process as text-to-image generation, starting from the image tokens instead of text tokens as unmasked input. We present the results in Figure 6 While the text is not fluent as those generated by an autoregressive language model, we see that UPGen is able to capture important keywords that describe the content of the image.
4.5 ABLATION STUDIES
Sampling temperature for image generation. Following the practice of MaskGIT, temperature to sample from the predicted logits during the generation process. This temperature hyperparameter has great influence the quality of generated images. To find out the optimal
7

---

Under review as a conference paper at ICLR 2023
Image prompt
Dataset annotation
Model output
this violent volcano going out of the atmosphere will just crupt! an active volcano called the volcano thundering lightning has glow only been abandoned for two weeks. the human sense of sound with its active bodies lies within, if ever in the parts, event. < person >, lightning wallpaper.
the light display of the erupting volcano of stromboli.
this vintage 1950s electric guitar available to elements from unique parts of the body. we will welcome this to your life. are all unique. made of diameter. 3's 6. 5 in gold top, in brass and engraved hardware, with custom hardware. electric guitar, guitar amp, guitar amp
the who vintage yellow guitar 0
Jep
cnv
according to person the houston zoo is one of the most beautiful zoos of america, but the zoo does, if it cannot afford for any terms of the budget, it will cost you too much money. if it is going to saving us, would wonder if visit it.
tower grove park beautiful landscaping throughout the park.
was looking for a walk, but decided to conveniently cross my break when received this lovely, brief breeze perfectly between my legs either going for a stroll, or not to walk in it, whether you should walk the beach place. if you like me, then go out.
north sea beach, the hague, netherlands
the
Figure 6: Examples of image-to-text generation using our method. The images are sampled from the CC12M dataset. Left. The sampled image prompt from CC12M. Middle. Text annotation of the image from CC12M. Right. Model-generated text based on the image prompt.
temperature, we measure the FID) score on MS-COCO validation set of generated images on different temperatures, and present the result in Figure 7 The result suggests an optimal temperature of 20. Qualitatively, we find that having too low temperature makes the generation process more greedy by only sampling the most probable token, which result less diverse images. If the temperature too high, the generation process becomes too chaotic and the generated images are less coherent.
The effect of unpaired text data. While flexibility of UPGen enables the use both paired and unpaired image and text data in training, it is important understand the effects of using additional unpaired text data during pre- pre-training. We conduct an ablation study comparing the performance of UPGen pre-trained with and without unpaired text data. We measure the visual representation quality of these two models via ImageNet linear classification accuracy, and test the quality of text-to-image generation with the FID score MS-COCO validation We present the results in Table 3 We
8

---

Under review as a conference paper at ICLR 2023
Training data
With unpaired text No unpaired text
MS-COCO zero-shot FID-30K score
65.25
67.32
ImageNet linear classification accuracy
70.18
70.18
Table 3: Ablation study of the effects of using additional unpaired language data in pre-training UPGen. We see that while the use of unpaired text data does not improve visual representation quality, it improves the quality of text-to-image generation which requires more complex language understanding.
that the use of unpaired data does not improve the quality visual representations. This result is expected since the ImageNet classification task does not require understand of complex logic and reasoning in natural language, and simply matching keywords sufficient. On the other hand, we observe that the of unpaired text data improves the FID score for text-to-image generation, which a task that requires stronger language understanding.
The effect of loss weights. During the pre-training of UPGen, although we use cross-entropy loss for both predicted image tokens and predicted language tokens, we find that important to balance the weight of these two losses. Specifically, discover that hav- ing a lower weight on the language loss improves the quality of both visual representation and image genera- tion. Following the previous ablation study, we present the effect of the language loss weights in Table 4 We hypothesize that this is caused by the text token predic- tion being much simpler problem than image token prediction. This can also be observed by the training accuracy of text and token predictions. During the fi- nal steps of training, the masked text token prediction accuracy converges to 37% while the masked image token prediction accuracy stay around 15%. Hence, it important to put more weights on the more difficult image token prediction problem.
120
110
100
FID Score
90
70
1
Taken Sampling Temperature
Figure 7: The effect of different sampling temperature on MS-COCO FID score.
Text loss weight of UPGen
1.0
0.1
MS-COCO zero-Shot FID-30K score
66.85
65.25
ImageNet linear classification accuracy
69.13
70.18
Table 4: Ablation study of the text loss weight during pre-training of UPGen. We see that using smaller weight for language loss improves both visual representation and image generation.
5 CONCLUSION AND LIMITATIONS
In this work, we present UPGen, a simple masked Transformer-based method for multimodal tasks, including representation learning, image-to-text generation, and text-to-image generation. We conducted quantitative and qualitative experimental evaluations on UPGen and demonstrate competitive performance on a wide range of tasks. As UPGen scalable and flexible, an interesting future work would be scaling up the amount of image-text, image-only and text-only data. A limitation of UPGen is that incorporating text-only data does not improve visual representation learning, even though training additional language data should enable the model to better leverage modal information. We believe resolving this would further improve the results UPGen, and leave this as future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2 08254, 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: http://github com/ com/google/jax
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11315-11322 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 1 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redeaps: Web-curated image- text data created by the people, for the people. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
betps::/dataseeee http::///daaseeeeebencmmaaaaa/a benchmarks cacpllsppppceeddde www.eeeingg....oooo..p..... wwwp:/g......rips.ooooooo..e.. wwweer/pps..cpppppee//e....
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04885, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesníkov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Wikimedia Foundation. Wikimedia downloads. URL
org
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205.14200 14204, 2022.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus.
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science, pp. 218-223, March 2017. doi: 10.528 4120316..
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pp. 16000-16009, 2022.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX. 2020. URL
httt::/giithnn com/ wwwm//ooo www.lo/ooooooooo.oooooooooooo flax
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhaard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
10

---

Under review as a conference paper at ICLR 2023
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-68551 2020.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint 1907 1692, 2019.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. arXiv preprint arXiv:2 1.Xi::2/12.1270: 12.500 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 12. 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21(140): 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 1821- PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125,, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorcalistic text-to- -maag diffusion models with deep language understanding. arXiv preprint 1448 2022.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. arXiv preprint arXiv:2/ /2. 022422 2021.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint arXiv:2205.051:0 2022.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, (9()):64 -7, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint 2208 10442, 2022.
11

---

Under review as a conference paper at ICLR 2023
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint Akiv:2206.10789) 2022.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV). December 2015.
12

---

Under review as a conference paper at ICLR 2023
A IMPLEMENTATIONS DETAILS
We provide our implementation and training details of UPGen this section.
A.1 IMPLEMENTATION
We implement our UPGen Transformers using Flax (Heek al., 2020) and JAX (Bradbury et al. 2018). Our implementation is largely based on open source codebases, such as the released code for MaskGIT (Chang et al. 2022) and M3AE (Geng et al. 2022). For the VQGAN tokenizer, we directly use the open source ImageNet pre-trained VQGAN from MaskGIT for 256x256 image size.
A.2 HYPERPARAMETERS
In this section we provide the training hyperparameters of UPGen. These configurations largely follow the training hyperparameters of MaskGIT (Chang al. 2022).
Hyperparameter
Value
Model size
ViT-Large
Dropout
0.1
Optimizer
AdamW
Base learning rate
0.005
Weight decay
0.05
Optimizer momentum
81= 0.9, 33 0.95
Batch size
4096
Learning rate schedule
cosine decay
Epochs
50
Warmup epochs
Image data augmentation
Random crop and flip
Paired image loss weight
1.0
Paired text loss weight
0.1
Unpaired text loss weight
0.01
Table Hyperparameters for training UPGen.
A.3 TRAINING AND COMPUTE RESOURCES
We train model Cloud TPUv3 pods on Google Cloud Platform through the TPU Research Cloud program. On TPUv3-512 pod, UPGen using unpaired data takes 80 hours to train and UPGen with only paired data takes 45 hours to train.
13


Reviews:
[{"review_id": "sdknfHFfTg", "reviewer": "Reviewer_s48X", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a unified self-supervised learning framework for both representation learning and generation. The proposed approach first discretizes an input image into a sequence of tokens using VQGAN and concatenate them with text tokens. It then trains a bidirectional Transformer encoder using the masked token prediction objective. At inference, it generates image or text by iteratively re-masking and predicting the masked tokens for image or text generation tasks. The paper empirically validates the quality of learned visual representations on ImageNet1K classification and the model's generation ability on MS-COCO, both qualitatively and quantitatively.", "summary_of_the_review": "I am not fully convinced that it is significant enough to warrant acceptance (see the weaknesses above). I will make the decision after seeing the rebuttal.", "strength_and_weaknesses": "Strength\n+ The paper is well-organized and easy to read.\n+ The proposed method, UPGen, is the first to unifies pretraining for both representation learning and generation.\n\nWeaknesses\n\nIt seems that the technical novelty of the proposed method is somewhat marginal: It throughly follows the training scheme of previous work, MaskGIT except that it uses additional modality, language. It does not have any contributions in terms of learning methods, model architectures and training/evaluation data. I believe that the model should competitive empirical results for publication in ICLR and the authors should provide the following things:\n- I wonder why the linear classification result on ImageNet1K are worse than MAE using very similar objective even though the model is trained on much more image data than MAE. Do you have results using ImageNet1K as pretraining image data?\n- I can't figure out the model's image-to-text ability just by looking at the results in Table 2. Could you provide apple-to-apple comparison results, in terms of training data or model capacity? You do not have to conduct new experiments. You can use numbers reported in previously published papers.\n- The quality of generated outputs of image-to-text generation shown in Figure 6 is not good. Could you provide quantitative results for this task?\n- You argue that the proposed method is the first framework that unifies pretraining for both representation learning and generation and you use large-scale paired or unpaired text data for pretraining. You need to provide empirical results on natural language processing tasks, not just image classification results on visual modality.\n\nMinor comments:\n- You should fix typos.\n- Does the qualitative examples in Figure 6 come from CC12M or MS-COCO?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "I7yGcXF0KB", "reviewer": "Reviewer_DLC8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed UPGen, a unified pre-trained model for both representation learning and generation, which is an encoder-only model based on masked Transformer for multimodal tasks. They show the potential of the masked token prediction model which can be directly used to generate images and language by iteratively re-masking and predicting masked tokens.\n\nFollowing MaskGIT, they use VQGAN as the image tokenizer, which can tokenize image into tokens and generate image by discrete tokens. Then they combine sequence of image tokens and language tokens as input of ViT after several embedding layers. They use masked token prediction objective in both pre-training step and generation step.\nAnd at the stage of iterative cross-modality generation, the author uses a confidence-based sampling mechanism following MaskGit. From experiment, the authors devote to demonstrating the usage of this model in serving as a representation learning model and generative model for image and language.", "summary_of_the_review": "Although this work puts forward a simple and elegant way of generation, it needs more study of this one stage model to show the advantage. And the performance of this model is still far from SOTA, even though the author strengthen that they pre-train this model on a smaller dataset, they still need to show the comparison between the same dataset scale at least.", "strength_and_weaknesses": "Strengths:\n\n- This paper is the first one completing representation learning and generative tasks by training on a single objective of masked token prediction. This way of text-prompt image generation and image-prompt text generation is simple but elegant.\n\n- This paper narrow the gap between generation and representation and has great potential in other image tokenizer and masked tokens prediction based model. \n\n- The experiments are relatively rich and the proposed model can be applied to tasks like representation learning, conditional generation tasks between image and text and unconditional image generation simultaneously.\n\nWeaknesses:\n\n- There are some grammatical errors in this paper. For example: (1) typo in page 2, “Multimodal representation learning for language and image” paragraph, “if we connect these two modalities modalities together” -> “if we connect these two modalities together”; (2) in the second paragraph of the introduction, page 1, it says “is trained conditioned on top of”, the word “conditioned” should be modified; (3) “Following the convention of in prior works of image-to-text” -> “Following the convention of in prior works of text-to-image” (4) “and Zˆ = Z ⊙ M denote the sequence of unmasked tokens.”-> “and Zˆ = Z ⊙ M denote the sequence of masked tokens.””\n\n- The authors skip the details of iterative strategy and image token decoding step, which seems just following MaskGIT and VQGAN. The only novelty in the model is the generation objective.\n\n- What’s more, this work does not show the advantage over other generative model. The quality of generated images do not perform SOTA results. And the Linear Classification accuracy on ImageNet 1K does not achieve comparable performance to prior works. In comparison with the two stage, the authors only indicate that one stage models need less hyperparameters and has less complexity in developing models, which do not show the superiority of one stage models adequately. Although the authors strengthen that they train the model on a small dataset, they still need to compare with the prior works on the same dataset scale to prove their model’s performance and transferability.\n\n- The authors point out UPGen learns generalizable representations that transfer to a wide variety of tasks in page 2. But in the paper, the authors only release the comparison of linear classification accuracy between prior works. More downstream task comparisons should be completed.\n\n- The ablation studies should show more results of different loss weight of image and different mask ratio strategy. In this paper, the authors sample a mask ratio uniformly between 0 and 1 every time in paired image-text tokens and for unpaired text tokens we use a fixed mask ratio of 15%, so the expectation of the mask ratio in paired tokens is 50%. The author need to show the mask strategy of uniformly sampling is better than a fixed one. Also, this work applies a weighting of 1.0 for the loss on the image tokens, a weighting of 0.1 on the paired text tokens and a weighting of 0.01 on the unpaired text tokens, but in the ablation study they only show the accuracy of text loss weight in 1.0 and 0.1. So more weight combination should be compared in the study.\n\n- From the experimental results in Table 1, the reviews can see that the performance of the proposed model is inferior to MAE under the similar data size. The cause of the phenomenon is lacked.\n\n- The model does not perform as well as prior methods in the tasks it aimed to address. In the image to text task, the model can only capture important keywords and the generated text is not fluent with meaningless items in. Why?\n\n-  Lack of experiments trained with image only data. For text only data, the additional unpaired data did not improve the representation learning performance as expected\n\nSuggestions:\n\n- The experiments results of the proposed model should be in bold to make it more clear for readers to follow.\n\n- There are many pictures in the main text, such as figure 5: Examples of unconditioned image generation of UPGen. I am wondering if these pictures are necessary. In my opinion, the figures should be the something like tend chart and so on.\n\n- The ablation studies only introduce  a smaller weight for language loss,and failed to consider using a smaller weight for image loss. It only proves the influence of text on image, not the influence of image on text.\n\n- The model it trained on much smaller datasets,  and thus it can't perform as well as the SOTA methods on each downstream tasks. It is recommended to train with same dataset and compare their performance.  \n\n- The authors say the proposed model can reduce the number of hyperparameters need to be tuned and reduce the complexity of the development process of model. I recommend that the authors may increase the conviction of the experiments by experimenting the proposed model and other text-to-image models on the same datasets and comparing the speed and efficiency.\n\n- The comparative methods are not enough in Section 4.2. The authors may increase the conviction of the experiments by experimenting on more datasets.\n\n- Difficulties in adjusting the weights of predicted image tokens and predicted language tokens in the training process, and the paper does not mention how the weights of image loss and text loss in the model are determined, but only mentions qualitatively that the weight of image loss should be greater. Besides, only two values for text loss weight is listed for comparison, and image loss is not mentioned in the context, which is not sufficient. Please clarify this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CZCO-hfQkWG", "reviewer": "Reviewer_jhwP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines two tasks - masked image modeling and masked language modeling - in a single framework, and trains a vision-language encoder to solve both tasks. The resulting model shows some capability for image-to-text generation, as well as reasonable representation learning performance.", "summary_of_the_review": "This paper proposes an interesting idea similar to BEIT-3, which unifies MIM and MLM with a single model. Unfortunately, the experiments are not well-executed. The paper does not demonstrate the advantage of the proposed method over existing ones. Some claims are also not well-supported.", "strength_and_weaknesses": "Strengths:\n- This paper is one of the first papers that combine masked image modeling with masked language modeling, which is an interesting direction.\n- The proposed model shows the capability of image-to-text generation while also achieving reasonable representation learning performance.\n\nWeaknesses:\n1. The motivation - current text-to-image methods have drawbacks due to two-stage training - is not very convincing to me, for the following reasons.\n    - The proposed method requires a pretrained VQGAN model, so technically it is also a two-stage method, similar as LDM or Parti.\n    - The claimed \"drawbacks\" - extra hyperparameters and complexity - do not seem to be particularly harmful. On the contrary, a pre-trained image tokenizer or pre-trained text encoder might be necessary to ensure good image generation quality.\n\n2. The experiments are insufficient to verify the advantage of UPGen over existing vision-language representation learning and text-to-image generation methods.\n    - In Table 1, UPGen with image-text data achieves the same representation learning performance as UPGen with image-only data. This suggests that the additional language supervision is not effectively utilized. Thus UPGen offers no advantage compared to existing MIM methods such as BEIT.\n    - In Table 1, the authors hypothesis that the superior performance of CLIP comes from the larger amount of training data. This hypothesis can be easily verified. The authors can pre-train CLIP on the same data as UPGen and make a fair comparison. The same goes for other baselines such as MAE.\n    - More downstream vision-language tasks such as image-text retrieval and VQA are necessary to evaluate the vision-language representation learning performance.\n    - The performance of UPGen is far worse than existing text-to-image generation methods. The authors attribute this to the smaller amount of pre-training data used. While it might be difficult to scale-up the pre-training data for UPGen, it is much easier to scale-down the pre-training data for existing methods and make a fair comparison.\n    - The image captioning performance does not seem good. What is performance if the model is fine-tuned on image captioning datasets?\n\n4. An important question has not been answered in the paper: whether the unification of representation learning and image generation brings benefit to each individual task?\n    ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-qzvcwnPem", "reviewer": "Reviewer_woHh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a unified pre-trained model UPGen for both generation and representation learning.\nLike DALL-E/BEiTv1-3, the image data are tokenized into dVAE/VQ-VAE tokens, and then is used to learn MLM on mono-modal data or image-text pair data with a random probability.\n", "summary_of_the_review": "I prefer to reject this paper since the novelty, please check the weaknesses.", "strength_and_weaknesses": "Strength:\n\n  1.This paper proposes a simple idea.\n  2.This paper is well written and organized.\n\nWeaknesses:\n\n1.Lock of novelty. In contrast to DALL-E, this paper uses a BERT-like bidirectional encoding during the pretraining phase. And similar to BEiT-3, pretrain on both mono-modal and image-text pair data, and perform mask-and-reconstruct loss.\n2.The reviewer thought the result in this paper was not very satisfactory. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject