Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency
Authors: Tianhong Li
Sangnie Bhardwaj
Yonglong Tian
Han Zhang
Jarred Barber
Dina Katabi
Guillaume Lajoie
Huiwen Chang
Dilip Krishnan
Venue: ICLR 2024 spotlight (2024)
Keywords: vision-language generative model
cycle consistency
Primary area: generative models

Abstract: Current vision-language generative models rely on expansive corpora of $\textit{paired}$ image-text data to attain optimal performance and generalization capabilities. However, automatically collecting such data (e.g. via large-scale web scraping) leads to low quality and poor image-text correlation, while human annotation is more accurate but requires significant manual effort and expense. We introduce $\textbf{ITIT}$ ($\textbf{I}$n$\textbf{T}$egrating $\textbf{I}$mage $\textbf{T}$ext): an innovative training paradigm grounded in the concept of cycle consistency which allows vision-language training on $\textit{unpaired}$ image and text data. ITIT is comprised of a joint image-text encoder with disjoint image and text decoders that enable bidirectional image-to-text and text-to-image generation in a single framework. During training, ITIT leverages a small set of paired image-text data to ensure its output matches the input reasonably well in both directions. Simultaneously, the model is also trained on much larger datasets containing only images or texts. This is achieved by enforcing cycle consistency between the original unpaired samples and the cycle-generated counterparts. For instance, it generates a caption for a given input image and then uses the caption to create an output image, and enforces similarity between the input and output images. Our experiments show that ITIT with unpaired datasets exhibits similar scaling behavior as using high-quality paired data. We demonstrate image generation and captioning performance on par with state-of-the-art text-to-image and image-to-text models with orders of magnitude fewer (only 3M) paired image-text data. Code will be released at https://github.com/LTH14/itit.

LEVERAGING UNPAIRED DATA FOR VISION-LANGUAGE GENERATIVE MODELS VIA CYCLE CONSISTENCY
Tianhong Li
Sangnie Bhardwaj2 Yonglong Tian
Han Zhang'
Jarred Barber
Dina Katabi
Guillaume Lajoie
Huiwen Chang
Dilip Krishnan
MIT CSAIL
Mila *Google Research
*Google DeepMind *OpenAl
ABSTRACT
Current vision-language generative models rely on expansive corpora of paired image-text data to attain optimal performance and generalization capabilities. However, automatically collecting such data (e.g. via large-scale web scraping) leads to low quality and poor image-text correlation, while human annotation is more accurate but requires significant manual effort and expense. We introduce ITIT (InTegrating Image Text): an innovative training paradigm grounded in the concept of cycle consistency which allows vision-language training on unpaired image and text data. ITIT is comprised of a joint image-text encoder with disjoint image and text decoders that enable bidirectional image-to-text and text-to-ime image generation in a single framework. During training, ITTT leverages a small set of paired image-text data to ensure its output matches the input reasonably well in both directions. Simultaneously, the model is also trained on much larger datasets containing only images texts. This is achieved by enforcing cycle consistency between the original unpaired samples and the cycle-generated counterparts. For instance, it generates caption for given input image and then uses the caption to create an output image, and enforces similarity between the input and output images. Our experiments show that ITIT with unpaired datasets exhibits similar scaling behavior as using high-quality paired data. We demonstrate image gen- eration and captioning performance on par with state-of-the-ar rext-to-imago and image-to-text models with orders of magnitude fewer (only 3M) paired image-text data. Code will be released at https: /tt.tbb....uu//gg///////...bb... www..ooo///.... ht//L/1H14//t
INTRODUCTION
Image-text multimodal training has gained remarkable attention in recent years. Models for text- to-image generation show the impressive ability to synthesize realistic images from textual prompts
(Rombach et 2022 Chang al. [2023] Saharia et al. [202] et. 20222b Ramesh et al. 202
Similarly, image-to to-ge-to-textt text models have demonstrated advanced image comprehension capabilities by
providing precise descriptions of input images (Chen et al. 2023 Wang et al. 2022a Li al. 2022
2023a Alayrae et al.] 2022) Wang et al. (022b). Training these models to exceptional performance demands datasets comprising hundreds of millions billions of paired image-text samples (Schuh- mann et al. 2022). The collection of very large paired datasets comes at a considerable cost as well as concerns of low quality (Jia et al. (02T). On the other hand, diverse and vast unimodal images or texts datasets remain unused in current generative dision-langual training (Raffel et al., 2020: Sun jet al. 2017: Zhai et al. 2022). This raises a natural question: can we leverage unpaired image and text data to facilitate generative vision-language training?
The major problem with using unpaired data during laiigag training the lack of supervi- sion. Generative vision-language training needs paired text to supervise image-to-text captioning, and paired images to supervise text-to-image generation. To solve this problem, we introduce ITIT, a novel training paradigm that significantly reduces the reliance on paired image-text data, favoring in- stead the utilization of unpaired image-only and text-only data. As shown in Figure T the core prin- ciple underlying ITTT is the use of cycle consistency losses between sycle-generater images/texts
"Co-first authors. This work was done when Tianhong Li interned at and Huiwen Chang worked at Google Research. Correspondence Tianhong Li <t anianno 2mi t edux.

---

Text-lmage-Text (T2I2T) Cycle
Reconstruction Loss
Flamingo
lake
T21
12T
Flamingo
lake
Unpaired Text 7
Reconstructed Text f
Synthesized Image
Image-Text-Imann (12T21) Cycle
Reconstruction Loss
12T
A
cat
on
green
grass
T21
Synthesized Text
Unpaired Image! I
Reconstructed Image I
Figure 1: Overview of ITTT. For unpaired data, ITTT first generates the image/text counterpart, and then these generated counterparts reconstruct the original text image.
and their corresponding original inputs to provide supervision for image-only and text-only data during training.
To enable cycle training. first unify image-to-text (I2T) and text-to-image (T2I) generation in the same framework, with bi-directional image-text encoder and disjoint image and text decoders. We tokenize images into discrete visual tokens (Van Den Oord et al. 2017) and combine them with text embeddings from a pre-trained T5 model (Raffel et al. (020)) as input to the joint image-text encoder. For 12T generation, we employ an autoregressive text decoder (Wang et al. 2022a), while
for T2I generation we use non-autoregressive parallel image decoder (Chang et al.
2023). which
an order of magnitude faster than autoregressive image decoders such as Yu et al.
2022b)
During training, ITIT utilizes a small set of paired image-text data to achieve reasonable text-to- image and image-to-ters generation performance, and align concepts in the image and text spaces. Simultaneously, for unpaired image (text) data, ITTT generates corresponding text (image) counter- parts and employs them as inputs to reconstruct the input image (text): this corresponds to a full cycle loss. We consider two kinds of full cycles: T212T (starting with an unpaired text sample); and 12T2I (starting with an unpaired image sample). For both modalities, we employ cross-entropy loss for the reconstruction, which enable simple, unified approach for cycle consistency. ITIT enables us to leverage unpaired image and text data to provide informative supervision signals for training.
We evaluate the performance of ITIT on standard image-to-te textext and text-to- -mage generation bench- marks and demonstrate that, by leveraging unpaired data and cycle consistency, ITIT attains per- formance levels similar non-cycle baseline. However, ITTT uses significantly up to 2 orders of magnitude lower paired data. Furthermore, ITTT scales similarly with unpaired data as the baseline does with equivalent amounts of paired data, while being much more robust to low data quality. We also compare ITIT with state-of-the-ane methods and show that we can achieve comparable per- formance on common text-to-image and image-to-text benchmarks with substantially lesser paired data. Our contributions are summarized as follows:
We introduce framework that unifies sott-to-mmag and image-to-ter to-text generation, and pro- pose ITIT, novel technique that enforces consistency between genereegeeettd images/text and their corresponding originals. This approach allows the training of image-to-text and text-to-image models using unpaired image and text data.
We comprehensively evaluate the proposed ITIT framework and the image-text cycle con- sistency method, and demonstrate that they significantly enhance model performance.
We show that ITIT achieve performance par with state-of-the -frth---rtttt methods com- mon text-to- toxt-to-image and image-to-tex benchmarks with much lesser (~x00xx paired data. When scaling up training data to improve model efficacy, show that add only un-
2

---

paired examples using our framework and achieve similar performance as scaled-up up paired data, without the downsides of significant manual effort and poor pairing quality.
2 LITERATURE REVIEW
Image-to- Text Generation. Various works explore autonomously generating textual descriptions from input images, either training the network with generative loss alone (Wang et al., 2022h Alayrac et al. 2022] Chen et al.] 2023] Li et al.] 2022; 2023a), or combining with contrastive learning (Yu et al. 2022ap. GIT (Wang et al. 2022a) trains model comprising an image encoder and an auto-regressive text decoder using a language modeling loss, the image encoder pre-trained with contrastive loss (Radford et al. 2021). In our work, we adopt similar framework to GIT for our Image-to- (12T) framework, but we initialize our image encoder from scratch.
Text-to-Image Generation. Recent works focus two primary paradigms: diffusion- -based mod- els (Rombach et al. (2022); Dhariwal & Nichol (202T); | Nichol et al.] (2021): [Saharia et al.] (2022); Ramesh et al. (2022); Ruiz et al. (2023); ); and token-based methods. Token- strategies trans-
form raw images into image tokens, and predict these tokens either in an autoregressive manner
(Esser et al. 2021: Ramesh et al., |021 Gafni et al. [022 Yu et al. [021 Ding et al. 2021] Yu
et al. 2022b or in parallel (Chang et al. 2022 et al.) 2023b; Chang et al. 2023). Muse (Chang
al. 2023) demonstrates that token- strategies with parallel decoding can be considerably
faster than diffusion- based or autoregressive generative models. Since this speed advantage facili-
tates text-to-image synthesis during training, we adopt this strategy in our T21 framework.
Unifying Image and Text Generation. COBIT (You et al. (2023) achieves this by employing distinct image and text unicoders, coupled with unified mooss-modal decoder. Additionally, CM3 (Aghajanyan et al. (2022)) and CM3Leon (Yu et al.) (2023)) harness causally masked generative models trained on extensive multi-modal document datasets, and enable the synthesis of both text and images. However, all these works still heavily rely on large-scale paired image-text datasets.
Leveraging Unpaired Data in Generative Vision-Language Training, Early works have tried to use unpaired image and text to train image captioning model in an unsupervised way (Feng et al.) 2019) However, the performance is relatively poor. Recent efforts in incorporating unpaired data into generative vision-language training primarily focus pre-trained image and text encoders (Esser et al., 2021] Roberts et al.) 2019). However, these applications are limited to pre-training and do not encompass the entire generative vision-language training procedure, thus providing only incremental improvements. In some cases, researchers have explored the use of text-only data to improve text decoders (Wang et al.] (2022b), utilizing text-to-text training. However, this only enhances the text decoder and not the image encoder, resulting again in constrained improvements.
Cycle-consistency. The concept of cycle consistency has previously been used provide regular- ization and/or compensate for lack of annotated data. 7ach et al. (2010): et al. (2016); Godard al.] (2016); Zhu al. (2017); Messikommer
al. (2022) explore for computer vision applica- tions such as learning dense correspondence, event detection, depth estimation, and image-to-image translation. Wang et al. 2023) and Goel et al. (2022) introduce regularization on image-text consis- tency during CLIP pre- pra-tiainnng, aiming to learn a better visual representation. Most related to our work is Gorti & Ma (2018), which uses text-image-tex cycle consistency to perform tot-to--mage translation, but the performance is poor. Moreover, none of the previous works has explored the potential of cycle consistency in generative vision-language training using unpaired data.
Our novel approach diverges from preceding vision-language models that heavily rely on either large corpus of paired image-text data, or fine-tuning methods that target only text or image en- coder/decoders separately. For the first time, our method facilitates the utilization of unpaired image and text data during generative vision-language training. This innovation significantly reduces the dependency on paired image-text samples during the training process, which empowers the expan- sion of generative vision- languaee training to nearly boundless text-only and image-only datasets.
3 METHOD
ITIT is the first framework that enables generative lanoun-anguuag training on unpaired image-only and text-only data. It uses simple yet effective architecture: unified image-text encoder and two
3

---

Empty Embedding
12T Training Module
15 Text Embedding
T21 Training Module
Image Encoder
Image Encoder
Image Decoder
Decoder
on
Generated Text Logits
Generated Image Logits
906
Masked Image Tokens
Ground-truth
Masked Image
Figure 2: 12T (left) and T21 (right) training pipelines for paired image and text data.
separate image and text decoders. This design seamlessly enables text-to-image and image-to-text generation in the same framework, which paves the way for coxt-image-text (T2I2T) and image-text- image (I2T2I) cyclic losses. Below, we describe each component of our ITIT architecture and the cycle-consistency training paradigm in detail.
3.1 UNIFIED IMAGE-TER GENERATION FRAMEWORK
Architecture. We first obtain text embedding 7 = [4] from the output of TS encoder (Roberts et al.) 2019) the text. Similarly, images are passed through a pre-trained VQ-tokenizer [Esser et al. 2021) to output image tokens / = [ L and K are the token sequence lengths text and image, respectively. The image tokens
then embedded with an embedding layer and concatenated with the T5 text features T as input to the image-text encoder. Modality-specifff decoders then operate on the encoded image-text features to generate either text or image tokens. The text decoder is autoregressive (Wang et al. 2022a), while the image decoder is parallel (Chang al. 2023). Both encoder and decoders are based on Transformer /Vaswani et al. 2017) layers. A detailed description of the model architecture is included in Appendix B.
Image-to- Text (I2T) Training. As shown in Figure L we input masked image tokens along with
empty text embedding to the image-text encoder. Masking is used to save computation, similar to MAE (He et al. 2022). ). We then use the features generated by the image-text encoder, as well as the ground-truth text tokens prepended with [BOS] (begin-of- )ffeeeence)))) token as the input to text decoder. We use an auto-regressive language modeling (LM) loss to train the encoder and decoder:
Lirr = (I,TT) Σ log pepptt( ... (....,,tt1)))). ft-1)),
(1)
which CE loss with label smoothing 0.1. Here, is the (subset of) unmasked tokens in / and p(ix Jar, 2 is the probability predicted by the encoder- decoder network (the 'logits' layer). D the distribution of paired image-text data. Note that the text decoder employs causal attention similar to GIT (Wang et al. (2022))): each text token only depends on the preceding text tokens and all image features.
be the [BOS] token. Ias
Text-to-Image (T2I) Training. As shown in Figure 2 right panel, we use masked image modeling for image generation, where the training objective reconstruct masked image tokens conditioned on the unmasked image tokens and the paired text features. We denote the binary mask determining use a cross-entropy loss between ground-
which image tokens are masked by = [ truth one-hot image tokens and the output of the image decoder. Specifically,
Cr21 = - Eur
Σ
dogp(ixl4ms T)],
(2)
Vk:m=1
Inference. follow GIT WWag et al., 2022a) for image-to-text inference and Muse (Chang et al., 2023 for text-to-image inference. More details are included in Appendix B.
3.2
TRAINING WITH CYCLE CONSISTENCY
Our cycle consistency training paradigm allows training with image-only and text-only data. The key idea is to first synthesize the corresponding text/image from the image-only text-only data, and then use the synthesized data as input to reconstruct the original image/text. This allows us to apply cycle consistency supervision image-only and text-only data.
Poat-Image-Ter (T212T) Cycle. Our T212T training pipeline is shown in Figure 3 top panel. At each training iteration, first synthesize pseudo paired image tokens I' for input text = [bi]
4

---

Online Pseudo Pair Generation
Cycle Consistency Training
Cross-Entropy
A
dog and
dog
T212T
A
35g
and
T21
Step Grod
24-Step Parallel
T21
argmax
12T
Decoding
I
TM
Entropy
12T
argmax
12T21
Greedy
J
T21
Decoding
Stop Grad
JM
Figure 3: Text-image-text (top) and mrage-text-imarn (bottom) cycle training pipelines for unpaired image and text data. We use pseudo-generated image and text to enable the cycle consistency. Image token masks M are always randomly chosen. The dashed line denotes causal attention. Text tokens prepended with [BOS] token used for auto-regressive language modeling loss.
using our T2I inference pipeline. We then apply random mask M to I', perform reconstruction on Is with the text T using the T2I pipeline, and obtain the reconstructed synthesized image I. This two-step process allows gradients through all 24 steps of parallel decoding, while still training the T21 module. Finally, we randomly mask and use IAs to generate text using the I2T pipeline. The objective of our cycle paradigm is to enforce consistency between this generated text and the original text. Therefore, the T212T cycle-consistency loss can be formulated as follows:
avoid the excessive memory requirements of back poooppropggg
Lr2rT = -ETED
Σ 1=1
log p(ttp to. (1)1))
(3)
This is very similar to the 12T loss in Equation being drawn from the image-texi joint distribution.
1. except that i is synthesized from T instead of
Image- Text- (12T2I) Consistency. Our 12T2I training pipeline is shown in Figure bottom panel. Similar to the T2I2T pipeline, we first synthesize pscudo paired text tokens T for input image tokens using 12T inference pipeline. We then use the 12T training pipeline to prediet fj from 10 ... 1i=_ and Js- As before, this avoids the excessive memory requirements of back- back-propagating gradients through the auto-regressive greedy decoding. We then mask 7, and pass it through the T21 pipeline with the predicted T to reconstruct the masked image tokens. Again, the loss enforces consistency between the reconstructed and the original image tokens using cross-entropy:
L12r21 =-ESED
log log (,,
(4)
Vksm==1
Gradient Estimation. One challenge in our cycle training is that i arg max(p and which are not differentiable. To solve this, we use a straight- through estimation on the predicted logits approximate the gradient. Specifically, we directly copy
& = arg max p(tz M t.. 1)
the gradient on the one-hot prediction to the predicted logits after softmax. We show in section 4.4 that this helps improve both text-to- to-image and image-to-ter performance.
RESULTS
4.1 EXPERIMENT SETUP
Datasets. We use three datasets in experiments: CC3M (harma et al. 2018). WebLI (Chen ct al. (023). and Shutterstock (Shutterstock) (023). CC3M contains 3.3 million high-quality image- text pairs. WebL.I (Web Language Image) contains 111 million images where the image-text pairing

---

No Cycle Loss
Cycle loss (398M Unpaired Data)
130.6
Image Captioning Erero-shol cintri
-100x fower
107.5
-100x fewer image pais
Gercraton IE
24-
Image Caploninn Iineuuing
105.5
23-
102.5
22-
23-
20-
18-
lower Image toxt pairs
295.0
1.2
Image Dats (in
Paired Image (n
Paired Image-Text Data M)
Figure 4: How IITT-H's performance scales with additional paired Shutterstock data. The baseline (T2I+12T) is trained with paired samples only, ITIT is trained with the same number of paired samples, as well as 398M unpaired samples (the full Shutterstock dataset) using cycle loss.
quality is much lower than CC3M. Thus, WebLI is significantly noisier and, as we show, leads to worse performance for 12T. Shutterstock contains 398 million images labeled by human annotators, which incurs significant expense and effort. More dataset details are included in Appendix C.
We use CC3M as our paired dataset, 50% of WebLI images as our unpaired image dataset, and the other 50% of WebLI texts as unpaired text dataset for most of our experiments (Section 43 and Section 4.4) This 50%-50% split ensures that corresponding image-text pairs are not present in our unpaired image and text splits. We use the Shutterstock dataset in Section 4.2 where analyze how ITIT scales w.r.t different number of paired and umpaired data samples.
Training. We set the input image resolution 256x256 be consistent with previous literature. Af- ter passing through the VQGAN tokenizer, the image token sequence length 16x16 (256 tokens). The raw text (maximum length of 64) is tokenized by SentencePiece tokenization (SentencePPiece 2023), and embedded using a pre-trained T5 encoder. These embeddings are then concatenated with the image token embeddings the input to our image-text encoder.
We experiment with ViT-B, ViT-L., and ViT-H size Transformers (Dosovitskiy et al. (2021) for our image-text encoder. We combine the losses Equations [] through with equal weight for training. For results in Section 4.3] we use Adafactor (Shazeer & Stern 2018) to train the model for 1.5M steps with a batch size 2048 (1024 for image-text pairs, 512 for unpaired images, and 512 for unpaired texts). We use cosine learning rate schedule with 5K steps warmup and maximum learning rate x 10-4 For other experiments, we use the exact same training paradigm except that we train the models for 500K steps. More details are included in Appendix B.
Evaluation. We follow the commonly used MS-COCO benchmark and evaluation protocols. For image-captioning, we evaluate both the zero-shot and fine-tuning performance of ITTT on the COCO Karpathy split (Karpathy& & Fei-Fei| 2015) and report the CIDEr score (Vedantam et al. 2015). For -iaage generation, COCO Captions training set and report the Frechet Inception Distance (FID) score (Heusel et al. 2017). CIDEr is the higher the better, and FID is the lower the better.
evaluate ITIT on 30K image-text pairs randomly selected from the
4.2 SCALE WITH DATA
In this section, we comprehensively evaluate ITIT's performance with different amounts of paired and unpaired data on Shutterstock dataset (Shutterstock) 2023) consisting of 398M image-text pairs.
Figure analyses how ITTT's performance scales with paired data. We train a baseline with only paired data, with the sum of the losses in Equation (I) and Equation (2). ITTT is trained with the same paired data as the baseline, and the entire set of 398M images and text present in Shutterstock as unpaired data. More paired data helps both settings, but training with unpaired data significantly improves ITIT's performance over the baseline on both image captioning and text-to-image gen- eration. Remarkably, with only 4M paired data and 398M unpaired data, ITTT achieves similar performance as training with 398M paired data. Note that ITIT does not use any samples not present in the baseline trained with 398M paired data, as all of the samples are from Shutterstock. Therefore

---

Additional Paired Data Only
Additional Unpaired Data with Cycle loss
ShutterStock
CC3M WebLi
100%
21.0
12.6
100%
20-
0.3%
2.1%
20.5-
12.6
18.
30%
30-
12.5
16
% Paired
30%
100%
12.4
100%
19.5
100%
14
Total Data
26-
12.2
100%
100%
100%
12.1
24
0.0.1.2)
(6.0)
396.8 (team
0.0
2.8 (4,0)
396 (398.5)
0
13 (44
111 (144)
0 (3)
11 b4)
111 I110
Additional (Total) Data inn M)
Additional (Total) Data (nn Mi
Additional (Total) Data (In M)
Additional (Tétal) Data (In M)
Figure 5: How ITIT's performance scales with the total amount of data used (x-axis). The base- line (T2I + 12T) in blue trained entirely with increasing amounts paired data. ITIT (orange) is trained with an increasing amount of unpaired data using cycle loss, while keeping the total amount of data equal for both curves. For example, the rightmost point with Shutterstock uses 1.2M image-text pairs and 396.8M unpaired samples (half as unpaired image and half as unpaired text) for ITIT with cycle loss, and 398M image-text pairs for the baseline. Left: Shutterstock data as both paired and unpaired. Right: CC3M as paired data, and varying fractions of WebLI as additional paired/ unpaired data.
ITIT can perform similarly baseline with 100x fewer image-text pairs, significantly reducing the effort and expense for the training of generative vision-language training.
Next, we evaluate how ITIT's performance scales w.r.t. the total amount of data used. We first train model with 1.2M paired image-text Shutterstock data. We then evaluate the effect of training models on adding increasing amounts of additional paired data vs, adding increasing amounts of unpaired data with cycle loss, keeping the total amount of data the same for both. As expected, we see in Figure that performance scales up with additional paired data. Surprisingly, however, additional unpaired data exhibits similar scalability. In fact, we can achieve 19.2 FID and 21.0 CIDEr with only 1.2M paired and 396.8M unpaired examples, which is very competitive with 19.0 FID and 22.2 CIDEr using 398M paired examples only. This experiment thus demonstrates that when scaling up training data, practitioners can rely on only adding unpaired examples using our method and achieve similar performance paired data without the extra manual effort required to collect it.
We repeat the above experiment in a more realistic setting, where the small-scale paired dataset can contain high-quality image-text pairs but a large-scale paired dataset has much lower quality. For this, we use the high-quality CC3M as the paired dataset, and the much larger WebLI as the low-quality unpaired dataset. As before, we start with model trained 3M paired examples (from CC3M), and add additional training data from WebLI in paired (blue) or unpaired (orange) form. As shown in Figure right pair, adding low-quality image-text pairs harms image captioning performance severely for the fully-paired case. However, the ITTT regime is not affected by this low quality and scales similarly as before. This demonstrates that our method robust to low data quality in large datasets, and can in fact be used to achieve significantly better performance in settings when paired data present but of low quality.
4.3 COMPARISON PRIOR WORK
In Table we compare ITIT with state-of-the-art image-to-text and text-to-image models on the commonly used MS-COCO benchmark. As shown, all SOTA methods rely heavily on training on large corpus of paired image-text data. ITIT, however, is trained with only 3M paired examples (CC3M), and an additional 55M unpaired image and text examples each (WebLI). Despite this, it beats many other methods trained on much more data for text-to-image generation (FID). For 12T, beats methods using a comparable amount of data (highlighted in green), and achieves performance competitive with other SOTA methods. We find that the pre-training data (both the mixture and the size) also makes difference to CIDEr score. For example, GIT (Wang et al. 2022a) achieves only 89.0 CIDEr fine-tum men-tuning performance on COCO captions when trained from scratch with 10M image-text pairs, which is far from its reported performance (1 (144.8) when trained with 800M image- text pairs. Our approach is orthogonal to dataset mixture considerations, and we believe that scaling data size and variety will further enhance FID and CIDEr scores. We leave this to future work.
7

---

Table 1: Quantitative comparison with state-of-the-art text-to-image and image-to-text models on MS-COCO. The image-captioning performance is evaluated on the COCO Karpathy split, and the text-to-image image generation FID is evaluated on 30K COCO images. denotes our re-implementation. We highlight in green other models that use comparable amounts of paired data. Note that the GIT (CLIP) model uses a CLIP (Rafford et al. 2021) encoder pre-trained with 400M image-text pairs.
Methods
#params
#paired data
#unpaired data
FID
CIDErt (2s)
CIDErt (ft)
StableDiffusion [Rombach et
2022
800M
400M
12.60
GLIDE (Nichol el al. 2021
5B
250M
12.24
S-cene-Soene (Gaami et al.
35M
11.84
DALL-E2 [Ramesh
650M
10.39
PARTI Yu et b022b
750M
5000M
10.71
Muse-5 (Chan 2023
3B
860M
7.88
Muse Chang el al.
750M
3M
23.7
127
BLIP Li al
2022
446M
129M
136.7
VL
Wang čt al
2022b
1100M
365M T
24.0
134.8
Sim VLMgg. (Wang
~.L4B
1100M
365M T
32.2
143.3
GIT (CLIP) (Wang et al 2022a
681M
800M
144.8
GIT (scratch) ( Wang of al. 2022g
129M
10M
89.0
72/+/27
CoBIT-Buse You el al
2023
626M
5200M
10.35
43.0
135.4
CoBIT-Large
1091M
5200M
9.37
44.8
139.5
CM3Leon Yu al.
7B
340M
4.88
61.6
ITTT-B
221M
3M
SSM 1+55M T
13.4
32.1
103.5
ITTT-L
487M
3M
5SM 1+55M T
12.0
35.1
116.4
ITIT-H
3M
55M 1455M
10.4
38.2
125.3
4.4 ABLATIONS
In Table we ablate the effectiveness of the four components of ITTT: T21, 12T, T212T, and I2T2I. As shown in rows 1-3, combining '21 and I2T training in our framework already improves image captioning performance. This likely because the T2I training alleviates the overfitting problem of 12T training, as shown in GIT (Wagg et al. 2022a).
As before (Figure we can see in row that combining CC3M and WebLI improves text-to-image generation, but harms image captioning performance. This is because of the lower image-text pairing quality WebLI compared to CC3M. The remaining rows demonstrate that the cycle loss alleviates this by using WebLI unpaired data and does not depend its image-text pairing quality. It is thus more generalizable to large-scale image-text datasets.
Next, 5-7 naive baselines for using unpaired image text data during generative vision- language training. We can simply perform text-to-text (T2T) autoregressive training without condi- tioning on images, which has been explored in some prior works (Wang al. (2022)). Similarly, we can perform image-to-image (121) reconstructive training without conditioning on text. Such baselines do improve the performance over not using any paired data (row 3).
We consider an ablation where the gradient of the cycle consistency loss is backpropagated up until the argmax step. Hence, only half of the cycle is trained. In fact, this is equivalent to first synthesizing an image counterpart from unpaired text and then using it as a pseudo image-text pair train the 12T model (similarly for T2I). Rows 8- 10 show that the half-cycle loss achieves much better performance than non-cycle baselines.
Finally, rows 11-14 show the performance of the full cycle ITTT training. Although T212T favors image captioning while 12T21 favors text-to-image generation, they both show significant improve- ment in text-to-image generation and image captioning. Moreover, row demonstrates that such two cycle losses can be combined further improve performance. Additionally, we can that the full cycle loss beats the half-cycle baselines (row 8-10), demonstrating the effectiveness of the gradient estimation step.
Lastly, we find by comparing row 3 and 13 that the cycle consistency loss can slightly improve the performance even without any additional data. We believe this is because it forces better image- text alignment. However, comparing row 13 and 14 shows that the huge improvements in both text-to-image and image-to-text generation mainly stem from the usage of additional unpaired data.
8

---

Table 2: Quantitative comparison between different variants of ITIT on MS-COCO. All experiments use ITIT trained with 500K steps. We take 50% of WebLI data and use the images as our unpaired image data, and the other 50% of WebLI data and use the texts as our unpaired text data.
T21
12T
T212T
12T21
paired data
unpaired text
unpaired image
FID,
CIDErt
Paired data only
x
x
X
CC3M
X XX
15.5
N/A
X
/
X
X
CC3M
X
X
N/A
19.0
X
X
CC3M
15.7
23.5
X
CC3M+WebLI
14.2
20.7
Paired umpaired data, no cycle
T2T
X
CC3M
50% WebLI
X
15.1
26.0
X
121
CC3M
X
50% WebLI
15.9
24.2
T2T
121
CC3M
50% Webl.
50% WehLI
15.6
28.5
Paired + unpaired data, half cycle
Half
X
CC3M
50% WebLI X
14.8
27.6
X
Half
CC3M
50% WebLI
14.7
24.8
Half
Half
CC3M
50% WebLI
50% WebLI
14.5
30.5
Paired umpaired data, full cycle
11
Full
X
CC3M
50% WebLI
14.6
28.4
12
Full
CC3M
50% WebLI
14.6
26.3
13
Full
Full
CC3M
CC3M 50% WebLI
CC3M
15.4
24.4
Full
Full
CC3M
50% WebLI
14.3
31.1
ITIT
No Cycle Loss
blue teddy bear on a bed
blue teddy bear on bed
blue teddy bear on the bed
blue teddy bear on a bed
teddy bear on the bed
soft toy bear sitting bed
Figure 6: Iteratively generating text image text and so on. With ITTT, the generated results are more consistent than the results from model trained without the cycle consistency loss.
4.5
RESULTS
With framework that can perform both image-to-text and text-to-image, we can easily perform sycle-generation, as shown in Figure 6 With ITTT training, the cycle generation often keeps the same semantics as the input text prompt. On the other hand, without the cycle consistency training, the cycle generation misses the "blue" semantics after the first cycle. This demonstrates that our cycle consistency training not only enables integrating unpaired image and text data into generative vision-languee training, but also improves image-text alignment for both image-to-text and text- to-image generation. We include a number of results of image and text generation in Appendix A (Figures through 4).
5 DISCUSSION
We propose ITTT, a novel training scheme that for the first time incorporates unpaired images and text into generative vision-language training. Through extensive ablations, we demonstrate the fectiveness of both the T212T cycle and I2T2I cycle in improving text-to-image and image-to-text generation performance. As a result, ITIT achieves performance competitive with state-of-the-ee art vision-language generative models, but with only million paired image-text samples. Our method can be used even when paired image-text data is present, and is especially helpful when the pairing quality low. Future directions include scaling ITIT larger unpaired image and text data and model sizes, and utilizing more diverse datasets.

---

REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, ct al. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07220 2022.
Jean-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23736, 2022.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transforeer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315- 11325, 2022.
Huiwen Chang. Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-I Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, al. Muse: Text-to-image generation via masked gener- ative transformers. arXiv preprint arXiv:2301.000
2023.
Xi Chen, Xiao Wang. Soravit Changpinyo. AJ Piergiovanni. Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong. Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Scyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language-image model, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 0188780087944 2021.
Ming Ding, Zhuayi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang. et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34:19822-19988 2021.
Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scalc, In Int. Conf. on Learning Representations (ICLR), 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. Unsupervised image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4125-4134, 2019.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In European Conference on Computer Vision, pp. 89-106. Springer, 2022.
Clément Godard, Oisin Mac Aodha, and Gabriel J. Brostow. Unsupervised monocular depth estimation with left-right consistency. CoRR, abs/1609.03367, 2016. URL htllaatp//axiv..ooo org/
03677
Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems, 35:6704- 6719, 2022.
Satya Krishna Gorti and Jeremy Ma. sext-to-image-t--eee translation using cycle consistent adversarial net- works, arXiv preprint 808.v.188..04538. 2018.
Kaiming He. Xinlei Chen, Saining Xie, Yanghao Li. Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, June 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classificr-free diffusion guidance. arXiv preprint arXiv:2207.1259. 2022.
Fric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, arXiv preprint arXiv: 1,1001144, 2016.
10

---

Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and langun- representation learning with noisy text supervision. In International conference on machine learning, pp.
PMLR, 2021.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3137, 2015.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language eagguage--mage pre-training for unified vision- language understanding and generation. In International Conference on Machine Learning, pp. 12888 12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 2023a.
Tianhong Li. Huiwen Chang. Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2142-2152, 2023b.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
Nico Messikommer, Daniel Gehrig, Mathias Gehrig, and Davide Scaramuzza. Bridging the gap between events and frames through unsupervised domain adaptation. IEEE Robotics and Automation Letters, 7(2):5155- 3522, apr 2022. doi: 10. 20.11000/a.0222315555555 3145053
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorcalistic image generation and editing with text- text-guided diffusion models. arXiv preprint 12.iv:2112.1074::: 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel. Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):: (1)5:558555511, 2020.
Aditya Ramesh, Mikhail Paylov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821- 8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Cascy Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612. 06125, 2022.
Adam Roberts, Colin Raffel, Katherine Lee, Michael Matena, Noam Shazeer, Peter J. Liu, Sharan Narang, Wei Li. and Yanqi Zhou. Exploring the limits of transfer learning with a unified text-to-text transformer. Technical report, Google. 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffuusin models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 22500 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479- 36494, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278 27594,, 2022.
SentencePiece. Unsupervised text tokenizer for neural network-based text generation. https:///gittb. com//googlee//eneeeceppieee ece, 2023.
11

---

Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hyper- nymed, image alt-text dataset for automatic image captioning. In Proceedings the 56th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2056-25566500 2565, 2018.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Interna- tional Conference on Machine Learning, pp. 4596-4604, PMLR, 2018.
Shutterstock. Shutterstock. https::/ http::/wustttttsttoooooooooooeee
2023.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep leamnigg era. In Proceedings of the IEEE international conference on computer vision, pp- 843-88 852, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shiens, and Zhigniew Wojna. Rethinking the incep- tion architecture for computer vision. Proceedings of the I.F.F. conference on computer vision and pattern recognition, pp. 2818-2828 8-2626, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances neural infor- mation processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-base image description evaluation. In Proceedings of the IEEE conference computer vision and puttern recognition, pp. 4566- 4575, 2015.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-tex, transformer for vision and language, 2022a.
Tan Wang, Kevin Lin, Linjie Li, Chung- -Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Equivariant similarity for vision-language foundation models. arXiv preprint arXiv:2303. 14465, 2023.
Zirui Wang, Jiahui Yu, Adams Wei Yu. Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision, 2022h.
Haoxuan You, Mandy Guo. Zhecan Wang. Kai-Wei Chang. Jason Baldridge, and Jiahui Yu. Cobit: A con- trastive bi-directional image-text generation model, 2023.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Ja- son Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vegan. arXiv preprint
1r.:02210..0667.. 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Con- trastive captioners are image-text foundation models. arXiv preprint arXiv:2205.0190 001917, 2022a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint 2rXiv:2206. 10789, 2(3):5, 2022b.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller. Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and 2023.
instruction tuning. arXiv preprini 4rXiv:
Christopher Zach, Manfred Klopschitz, and Mare Pollefeys. Disambiguating visual relations using loop con- straints. In 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 1433, 2010. doi: 10. 20.109999VP220105553999...10
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12104 12113,, 2022.
Tinghui Zhou, Philipp Krähenbühl, Mathicu Aubry, Qixing Huang, and Alexci A. Efros. Learning donse correspondence via 3d-guided cycle consistency. CoRR, abs/1604.05383, 2016. URL http://arxiv..
org/abs/1604 05383]
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to- translation using cycle-consistent adversarial networks, 2017.
12

---

6 ADDITIONAL QUALITATIVE RESULTS
Image-to-Test Text and Text-to-Image Generation. Figure 7 Figure and Figure we show the performance of ITIT on text-to-image generation and image captioning. The model uses VIT-H as the backbone and is trained with CC3M as paired data and WebLI as unpaired data for 1.5M steps. As shown in the results, our model can generate realistic images from text prompts, and can also generate accurate captions for the input image.
Cycle Generation. We include more cycle-generation results in Figure 10 With ITIT. the gen- erated results are quite consistent with the original input. Without the cycle consistency loss, the generated text/image can easily miss some key information in the input image/text, causing the cycle-generation results to diverge from the original input. This demonstrates that the proposed cy- cle consistency loss forces the cycle generation be more consistent and improve the input-output alignment for both lext-to-image and image-to-text generation.
7 IMPLEMENTATION DETAILS
In this section, we include our implementation details, including prpeepparamtter, model architec- ture, and training paradigm.
Image Tokenizer and Detokenizer. We use a CNN-based VQGAN encoder encode the 256x256 input images to 16x 16 feature maps. The quantizer then quantizes each pixel of the encoder's ' output feature map using a codebook with 8192 entries. The detokenizer operates on the 16x 16x16 discrete tokens and reconstructs the 256x256 image. Our VQGAN tokenizer and detokenizer are trained on the WebLI dataset with batch size 256.
VIT architecture. After the tokenizer, the image latent sequence length becomes 256. Since we always use a masking ratio larger than 50%, we drop 128 of the masked image tokens from the input. The tokens then embedded with an embedding layer and concatenated with the text embedding from T5-XXL with a length of 64, We then use an image-tex text encoder Transformer with standard ViT architecture Dosovitskiy al. (202T), which consists of stack of Transformer blocks Vaswani ct al. (017), where each block consists of a multi-head self-attention block and an MIP block.
The text decoder is similar to the used in GIT (Wang al. 2022a) which consists of 6 Trans- former blocks with causal attention mask. The attention mask ensures each text token can only attend its previous text tokens and the image tokens.
The image decoder is similar to the one used in MAGE (Li et al. 2023b), which consists of 8 Transformer blocks with self-attention. The input the image decoder is padded with the previously dropped masked image tokens. In this way, we save a lot of computation in the image-text encoder.
Vision-language Training. Please refer Table for our default vision-language training setting. With the online synthesis step, ITIT requires ~22x training time as standard 12T and T2I non-cycle training. Our ViT-H training with 1.5M steps takes 10.9 days on 512 TPUv3.
Gradient scale for 12T loss: Similar to GIT (Wang et al. (2022))), we found that the image-text encoder should receive a smaller gradient than the text decoder. Therefore, we scale down the gradient backpropagated from the text decoder to the image-text encoder by 0.1. In practice, this is simply implemented by 2 0.1z + stopgrad(0.9:)) where 2 denotes the output of the image-text encoder.
Cycle Training. During cycle training, the image-text encoder experiences the forward pass twice. We found that if we back-propagated the gradient back to it twice, the training becomes unstable. Therefore, we stop the gradient after the first forward pass of the image-text encoder, which signifi- cantly stabilize the cycle training.
Gradient Estimation. We use Gumbel softmax (Jang et al. 2016) with strength 1.0 for the gradient estimation in the T212T cycle, and use straight-through softmax for the gradient estimation in the 12T2I cycle. We found that using Gumbel softmax in the T2I2T cycle improves zero-shot CIDEr by 0.3, while using Gumbel softmax in the 12T2I cycle does not bring improvement.
12T Inference. To generate captions for an image, we first extract visual features from the image tokens (and empty text tokens) using the image-text encoder. Then we auto-regressively generate the
13

---

Table 3: Pre-training Setting.
config
value
optimizer
Adafactor (Shazeer Stern 2018)
peak learning rate
le-4
weight decay
0.045
optimizer momentum
B1, 92
0.96 0.96
T2I batch size
512
12T/121 batch size
512
T212T batch size
512
12T2I batch size
512
learning rate schedule
cosine decay Loshchilov & Hutter 2016
warmup steps
5000
training steps
1.5M
gradient clip
3,0
label smoothing (Szegedy al. 2016
0.1
dropout
0.1
image masking ratio min
0.5
image masking ratio max image masking ratio mode image masking ratio std
1.0
(T2I), 0.75 (12T)
0.75
0.25
tokens, conditioned on the visual features and previously generated tokens. Following GIT (Wang al. (022a)), we use beam search to generate the tokens with a beam size set
T2I Inference. To enable classifier-free guidance (Ho & Salimans/ 2022) for generation, when performing 12T training with empty text embedding as input, we also train the image decoder to reconstruct missing tokens. In such cases, the T21 training becomes image-to-image (121) training, which is to reconstruct the original tokens from unmasked tokens.
Similar to Muse (Chang et al. (2023), generate an image from a text prompt. We start with entirely masked image tokens, and concatenate them with the text prompt tokens. At each iteration, the decoder predicts conditional logit la and an unconditional logit Lu for each masked token. The final logits lg are formed by moving away from L, by the guidance scale т: L = (1 7) Tlu- We then sample each token using categorical sampling. After that, the corresponding prediction score of each token plus a noise sampled from random Gumbel distribution multiplied by temperature / is used the "confidence" score indicating the model's belief of each token prediction. We then sample the top-k tokens with the highest predicted probability, and replace the corresponding masked tokens with these sampled predicted tokens. The number of masked tokens to be replaced in each iteration follows cosine function (Chang al. 2022). We use total 24 steps to generate an image. For each model, we sweep temperature and classifier-free guidance scale for the optimal FID. In practice, we find I and 2.0 serve as near-optimal temperature and guidance scale.
parallel decoding with classifier-free guidance to
8 DATASETS
We three datasets in main paper: CC3M, WebLI, and Shutterstock. CC3M contains 3.3M image-text pairs. The raw descriptions are harvested from the alt-text HTML attribute associated with web images and then filtered to form the final dataset. WebLI (Web Language Image) contains 111 million images from the public web with image-associated alt-text labels where the image-text pairing quality is much lower than CC3M, because no filtering applied the alt-text data. Shutterstock contains 398 million images labeled by human annotators, which incurs significant expense and effort.
TRAINING SPEED
show the training speed of different variants of ITIT. All experiments evaluated on a cluster 256 TPUv4, with total batch size equals 2048. As shown in the table, ITIT ~2233 slower than the non-cycle baseline. The full cycle ITIT is slightly slower than the half cycle ITIT as the gradient needs to back propapre before the argmax. Despite doubling the training time, ITTT significantly reduce the need of collecting large-scale paired datasets. We show in paper
In Table 4
14

---

Table Training speed of different variants of ITIT in steps/s. All experiments use ITIT₂ with total batch size of 2048 and are evaluated using cluster of 256 TPUv4.
T21
12T
T212T
12T21
Steps/s
No cycle
x
X
3.41
Holf cycle
2
Half
X
2.41
X
Half
2.19
4
Half
Half
1.64
Full cycle
5
Full
X
2.25
V
X
Full
2.05
V
7
Full
Full
1.47
that ITTT trained with only 4M paired data can achieve a similar performance as baseline trained with 400M paired data, reducing the cost of collecting paired data by 100 times.
10 WARM-UP
Table 5: Warm-up steps. Incorporating cycle consistency loss at 25K steps achieves the best perfor- mance.
Warm-up steps
25K
50K
100K
200K
FID
14.3
14.1
14.3
14.6
14.8
CIDEr
31.1
31.3
31.3
30.9
30.5
At the beginning of the training process, the text-to-image (T2I) and image-to-text (I2T) modules struggle to generate realistic images or texts. Introducing cycle consistency loss during this initial phase would likely introduce unnecessary noise into the training process. To address this, exper- imented with a warm-up training scheme, as shown in Table 5) This approach involves delaying the introduction of eycle consistency loss until the T2I and 12T modules have undergone several steps of training. We evaluated all experiments using an ITTT-B model trained for 500K steps. The re- sults indicate that incorporating cycle consistency loss after 25K steps yields optimal performance. Interestingly, introducing the cycle loss at the very beginning (0 step) does not significantly impair performance. This resilience is likely due to the dominance of paired data the early training stages. As the generation results improve, the impact of cycle consistency loss becomes more pronounced, achieving greater diversity and generalization ability through the utilization of unpaired data.
11 FAILURE CASES
In Figure we present various instances where the ITTT model does not perform as expected. The first notable type of failure involves the generation of text within images. Additionally, the model occasionally produces images with watermarks margins, artifacts that can be traced back to the training data. Another challenge arises when the model attempts to generate multiple objects within single image, often resulting in compromise the generation quality of each individual object.
important to recognize, however, that these shortcomings are not unique to ITTT; they are, in fact, common across most image-to-text generation models. Some of these issues, such as water- marks, are closely tied to the quality of the data utilized in training. Given this context, we believe that ITTT has the potential mitigate these prevalent shortcomings by lessening the dependency
15

---

paired data in vision-language generative training. A promising direction for future research would involve training ITIT with a combination of small but exceptionally high-quality, paired image-text dataset and extensive, high-quality unpaired image and text datasets. This approach could significantly enhance the model's performance by providing a richer and more diverse training environment, potentially overcoming the common failure cases that currently hinder image-to-text generation models.
16

---

Half of a white cake with cholocate on top
A full bathroom with wicker laundry basket
A small white cat large bowl
A larrge passenger
airplane flying through the air
large red bus on the side a city street
motorcycle parked on grass pathway
A fall landscape with house next to lake
plate of food with noodles, carrots and broccoli
Plants sitting near window inside a house
Figure 7: ITIT text-to-image generation results.
17

---

Baseline
person and me on the beach
person preparing food the kitchen
close up o bull
tv in the living room
ITIT
with surfboard
preparing food the
bull the farm
stand
Baseline
homeless sleeps on a park
black and white photograph tower
selection of food from menu
eating watermeion
ITIT
sitting bench
black photo of clock tower
baby birthday cake
Baseline
living room
sleops on
man soiling bananas on the streat
bathroom ideas with shower stall
ITIT
my desk the office
person sleeping on the bed
vendor sells benaanee at market
example trendy bathroom design
Baseline
plane the sky
pizza topped with mozzarella and cherry
skateboarder does trick on skatshoard
woman clances in traditional dress
ITIT
plane the sky
photo of pizza
younn man skateboarding park
girl traditional coetumee
Figure 8: image-to-text generation performance (zero-shot COCO Captions). The baseline is trained on CC3M only. ITIT is trained on CC3M as paired data and WebLI as unpaired data. We obfuscate the faces of people in the images.
18

---

living room with tumitre and flat. screon
man uniform holding bat
slanding umbrolla in the grass
group of paople flynn kites on cloudy day
couple zebra standing other
white sitting top
giraffe standing
sidewalk next to a body
water
BARU
013
cel phone sitting on top newspaper
a woman in blu dress holding
a bethroom with a toilel sink and window
fire hydrant on a city street night
man walking street holding a coll phone
malorcycle parked on the side of road
a person riding board on a
display case with different types vegetables
group todiy bears sitting of table
of boats are docked in
man riding a skateboard side ramp
clock pole front building
Figure 9: ITIT image-to-ter to-tett generation results (fine-tuned on COCO Captions).
19

---

ITIT
pink bicycle leaning against a tree
pink bicycle leaning against tree
pink bicycle leaning against a tree
pink bicycle leaning against tree
pink bike leaning against a tree
No Cycle
Loss
pink bioycle leaning against a tree
pink bicycle leaning against tree
pink bicycle leaning against wall
pink bicycle
vector illustration of a bicycle
ITIT
piece of chocolate cake on yellow plate
slice of chocolate cake on a yellow plate
a slice of chocolate cake yellow plate
a slice of chocolate cake on yellow plate
slice of chocolate cake yellow plate
No Cycle
Loss
piece of chocolate cake on a yellow plate
piece chocolate cake white background
chocolate cake isolated on white background
chocolate cake isolated on a white background
cake on a white background
ITIT
pizza and cup coffee on table
pizza with cup coffee
pizza with cup of coffee
coffee and slice of pizza
coffee and a slice of pizza
No
Cycle
Loss
pizza and cup of coffee table
pizza
pizza from the top spicy pizza
pizza from the top spicy pizza
pieces of pizza white background
Figure 10: More cycle generation results with or without cycle consistency loss.
20

---

DINCTH TOANT THDEH
50 RESIIO ty/AA
210kA SAUT
A bear holding board with 'hello' written it
A colorful slogan
cake with 'happy birthday' it
A t-shirt with slogan on it
(a) Generate text image
ast anota
A plate of food with noodles, carrots and broccoli
A small white cat large bowl
A large red bus on the side of city street
White cake with cholocate top
(b) Watermarks / margins in the dataset
Five cats and five dogs are playing together
Many bicycles on the street
Many fishes in a pool
Many apples and pears
(c) Multiple objects in one image
Figure 11: ITIT failure cases.
21


Reviews:
[{"review_id": "GRdE131AOF", "reviewer": "Reviewer_xcv8", "summary": "This paper proposes to learn from unpaired data for T2I I2T generation by exploiting a cycle consistency constraint. A joint text-image architecture is presented as the unify model for this learning. Then, pseudo examples are generated and cycle consistency is optimized as two sub-steps in one training iteration. The learned model produces good generation quality with limited paired data, such as 4M vs. 398M.", "questions": "(discussed in the weakness section)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The semi-supervised learning process for generative models has not been well studied before.\n2. This paper provides a plausible way of learning from unpaired data with optimized computation cost.\n3. Ablation studies are well presented and support the claim of adopting unpaired data.", "weaknesses": "1. This paper claims that the paired data gap is almost closed in Figure 4 (from 398M to 4M). However, there is still a significant gap in FID and CIDEr according to Figure 1 compared with other methods e.g. CM3Leon, Muse, GIT. Is there any other reason besides different numbers of parameters? Or maybe Shutterstock data is so similar that scaling data does not make a difference, but on more diverse datasets the Figure 4 claim will not hold?\n2. How does ITIT generalize to new/different domains without paired data in that domain (assuming unpaired domain specific text and image available)? One example readers might be interested in could be medical imaging where paired data is much more difficult to obtain.\n3. Unclear compatibility with other types of generative models such as diffusion/flow/consistency models as the I2T stage. How much does the ITIT framework depends on the unified architecture? Does ITIT work with any kind of T2I I2T model?\n4. ITIT essentially denoises training data by learning a better alignment as shown in Figure 6. Does it worth to study other methods that also cleans the data, e.g. simply CLIP filtering with high confidence as pseudo labels, i.e. semi-supervised discriminative learning? This might keep the “blue” information in Figure 6 as well?\n5. Runtime due to the pseudo pair generation stage. The appendix mentioned that the ITIT requires 2x training time compared with T2I and I2T non-cycle training, but how is it the case considering the 24 steps of decoding happening? How much of the time was spent on the online pseudo pair generation stage? Does it require careful batching or implementation for good TPU utilization?\n6. Although scaling data is left to future work, the value of this paper depends a lot on the promise of further scaling unpaired image and text data beyond the combination of all current paired datasets. Are we still able to collect 10x high quality images/texts but not paired? Do we still need more data in this domain? Unfortunately this has not been shown to a good extent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oU9I1UTYn9", "reviewer": "Reviewer_Qb8X", "summary": "This paper suggests novel  ITIT(InTegrating Image Text) method which enables leveraging massive unpaired images or texts. It suggests a concept of cycle consistency by using image-to-text (I2T) generation model and text-to-image (T2I) generation model in a single framework. As a result, the authors verify that ITIT with unpaired data shows comparable performance with paired data and, it achieves state-of-the-art performance with much fewer paired image-text data.", "questions": "1. Could you provide a comparison of the computational overhead involved in ITIT when implemented in full cycle, half cycle, and no-cycle\n2. Are the models involved in this study trained entirely from scratch, or do they utilize weights from pre-existing pre-trained models?\n3. Is it possible to extend the method for using diffusion based model for the T2I?\n4. Given the concerns mentioned in the weakness section about ITIT's susceptibility to failure cases in the early pre-training phase, I wonder:  What are the results when trained with a two-stage approach  (first pre-trained with paired dataset and then use unpaired dataset / paired-unpaired dataset)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation (usage of unpaired image and text) is clear and solid.\n2. The method (ITIT) is a simple but strong:\n     - It introduces the concept of consistency, effectively bridging image-to-text generation models (like BLIP) with text-to-image generation models (like Maze).1\n     - The two-step phase in T2I2T / I2T2I facilitates efficient pre-training.\n3. The experimental result is strong. To the best of my knowledge, this might be the first paper where a method pre-trained on unpaired image-text data demonstrates comparative performance with those pre-trained on paired image-text datasets.\n4. The paper is well-written and figure and tables are well-organized.", "weaknesses": "1. In Table 2, the gap between full cycle and half cycle of ITIT training seems very marginal, which raises doubts about the effectiveness of proposed consistency concept. Namely, it seems that the pseudo-targets (ITIT-half cycle) would be sufficient to leverage unpaired image and texts. In this case, I believe that generating pseudo-targets would be more powerful:  1) it is computationally efficient and 2) it could easily integrate with the recent pre-trained models such as BLIP v2, stable-diffusion, and so on. (since the generation process of pseudo-targets are model-agnostic)\n\n\n2. Given that ITIT employs synthetically generated captions or images during the training process, there is a reasonable expectation that numerous failure cases may occur. This concern is particularly pronounced with datasets like CC3M, which are known to consist of noisy image-text pairs, potentially leading to a higher frequency of failures.\n It would be grateful if authors could share the fair cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VmYMF2E6jB", "reviewer": "Reviewer_ySUK", "summary": "The ITIT model introduces a new training paradigm in vision-language generative models, focusing on cycle consistency to facilitate training on unpaired image and text data. Unlike conventional models reliant on vast, paired image-text datasets, ITIT optimizes performance using a limited set of paired data, supplementing it with larger unpaired datasets, and ensures output accuracy through bidirectional image-to-text and text-to-image generation. Experimental results show that ITIT, even with significantly fewer paired datasets, delivers performance comparable to state-of-the-art models in image generation and captioning.", "questions": "See the three questions in weaknesses \"other issues\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a view of training of image-to-text and text-to-image models using unpaired image and text data. The overall design is based on an existing reasonable idea of \"cycle consistency\".\n\nThe paper evaluates the proposed ITIT framework and the image-text cycle consistency method and demonstrates that it can enhance model performance in image generation and captioning.\n\nThe proposed method is claimed to be scalable to get higher performance improvement.\n\nThe paper writing is clear and easy to follow.", "weaknesses": "Innovation. It seems all innovation comes down to how to reduce memory consumption. Others such as network structures and designs already exist. Specifically, regarding the method of reducing memory consumption, the core idea is that the author found that generating training data from 0 to 1 has too high a memory cost, so they first go from 0 to 1 (without calculating the gradient), then manually return to 0.9, and redo 0.9 to 1. From this point of view, the contribution is somewhat engineering-oriented. \n\nOther issues. There are a few other issues: \n1. The author only mentions a reduction in memory usage, but the time consumption should be quite large, which is a disadvantage. Any justification for this?\n2. The I2T2I method seems a bit unreasonable because the latter half of T2I can already generate many kinds of images, so why must reconstructing the image necessarily be correct? \n3. For visual-language cycle consistency, there are two papers already: 'Equivariant Similarity for Vision-Language Foundation Models' (ICCV 2023), and 'CyCLIP: Cyclic Contrastive Language-Image Pretraining' (NeurIPS 2022). In this submission, there is an image-text encoder, for which it’s very straightforward to use the above two methods as a baseline (in the cycle consistency of embedding). So the reviewer is quite curious about how the cycle of embedding compares to the cycle of generating data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FbyyysxIyC", "reviewer": "Reviewer_NW6v", "summary": "The study introduces a groundbreaking training paradigm called ITIT, designed for vision-language generative models. ITIT capitalizes on cycle consistency to train on unpaired image and text datasets, thereby minimizing reliance on costly and extensive paired datasets. The architecture features a single encoder for both image and text data, along with distinct decoders for each. A minimal amount of paired data is employed for baseline training, while the bulk of the training leverages unpaired data, regulated by cycle consistency to maintain output fidelity. Experimental evaluations reveal that ITIT achieves performance levels similar to existing leading models but with a significantly reduced need for paired data. Furthermore, the study confirms the model's ability to effectively scale with unpaired data. ITIT paves the way for more efficient and economical training of vision-language models.", "questions": "1. Question for the gradient estimation of the cycle training: Referring to the figure 3, T2I is performed twice with a stop gradient operation in the middle for the T2I2T, why not directly perform the T2I once. Similar questions for the  twice I2T in the process of I2T2I.\n\n2. Why ViT is chosen as the joint image-text encoder, instead of using some multimodal architectures. How does ViT extract the text feature?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The introduction of cycle consistency in vision-language generative models is innovative and addresses a significant gap in the literature. This is particularly relevant for scenarios where paired data is scarce or expensive to obtain.\n\n2. ITIT scales well with unpaired data, achieving performance levels similar to non-cycle baselines but with much lower paired data requirements. In addition, the paper provides a thorough evaluation of the proposed ITIT framework, demonstrating its efficacy in both image-to-text and text-to-image generation tasks with comprehensive experiments.\n\n3. The cycle consistency training improves image-text alignment for both image-to-text and text-to-image generation, while the method without the cycle consistency loss fail on the generation consistency to some extent.", "weaknesses": "1. While the paper mentions techniques to reduce computational overhead by one-step back-propagation in the I2T2I cycle, it does not provide quantitative metrics to evaluate the efficiency of these techniques. For the pre-trained data used in the experiments, ITIT still uses 4M paired data and 398M unpaired data to compete with the method that uses 398M paired data. In that case, the overall computational overhead is larger than the traditional paired data training. Can you provide some analysis in comparing the training efficiency between the paired data training and the ITIT.\n\n2. The pseudo-synthesized data pairs could be noisy at the beginning of the training since the generator is not trained yet, I am wondering whether the pseudo-data pairs generated at the beginning will play negatively on the performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)