Data Extrapolation for Text-to-image Generation on Small Datasets

## Abstract

Abstract Text-to-image generation requires large amount of training data to synthesizing high-quality images. For augmenting training data, previous methods rely on data interpolations like cropping, flipping, and mixing up, which fail to introduce new information and yield only marginal improvements. In this paper, we propose a new data augmentation method for text-to-image generation using linear extrapolation.
Specifically, we apply linear extrapolation only on text feature, and new image data are retrieved from the internet by search engines. For the reliability of new text-image pairs, we design two outlier detectors to purify retrieved images. Based on extrapolation, we construct training samples dozens of times larger than the original dataset, resulting in a significant improvement in text-to-image performance. Moreover, we propose a NULL-guidance to refine score estimation, and apply recurrent affine transformation to fuse text information. Our model achieves FID scores of 7.91, 9.52 and 5.00 on the CUB, Oxford and COCO datasets. The code and data will be available on GitHub.

## 1 Introduction

Text-to-image generation aims to synthesize images according to textual descriptions. As the bridge between human language and generative models, text-to-image generation  (Reed et al., ; Ye et al., ; Sauer et al., ; Rombach et al., ; Ramesh et al., )is applied to more and more application domains, such as digital human (Yin & Li, ), image editing (Brack et al., ), and computer-aided design (Liu et al., ).
The diversity of applications leads to a large number of small datasets, where existing data are not sufficient to train high-quality generative models, and generative large models cannot overcome the long-tail effect of diverse applications.

To augment training data, existing methods typically rely on data interpolation techniques such as cropping, flipping, and mixing up images (Zhang et al., ). While these methods leverage human knowledge to create new perspectives on existing images or features, they do not introduce new information and yield only marginal improvements. Additionally, Retrieval-base models (Chen et al., ; Sheynin et al., ; Li et al., ) employs retrieval methods to gather relevant training data from external databases like WikiImages. However, these external databases often contain very few images for specific entries, and their description styles differ significantly from those in text-to-image datasets. Furthermore, VQ-diffusion (Gu et al., ) pre-trains its text-to-image model on the Conceptual Caption dataset with 15 million images, but the resulting improvements are not obvious.

In this paper, we explore data linear extrapolation to augment training data. Linear extrapolation can be risky, as similar text-image pairs may not be nearby in Euclidean space. For information reliability, as depicted in Figure 1, we explore linear extrapolation only on text data, and new image data are retrieved from the internet by search engines. And then outlier detectors are designed to purify retrieved web images. In this way, the reliability of new text-image pairs are guaranteed by search precision and outlier detection.

To detect outliers from web images, we divide outliers into irrelevant and similar ones. For detecting irrelevant outliers, K-means (Lloyd, ) algorithm is used to cluster noisy web images into similar images and outliers. In the image feature space generated by a CLIP encoder (Radford et al., ), similar images will be close to dataset images, while outliers will be far away. Based on this observation, we remove images that differ significantly from dataset images. For detecting similar outliers, each web image is assigned a label by a fine-grained classifier trained on the original dataset. If the label does not match the search keyword, the image is considered as an outlier and removed. For every purified web image, we extrapolate a new text descriptions according to the local manifold of dataset images. Based on extrapolation, we construct training samples dozens of times larger than the original dataset.

Moreover, we propose NULL-condition guidance to refine score estimation for text-to-image generation. Classifier-free guidance (Ho & Salimans, ) uses a dummy label to refine label-conditioned image synthesis. Similarly, in text-to-image generation, such a dummy label can be replaced by a prompt with no new physical meaning. For example, “a picture of bird” provides no information for the CUB dataset “a picture of flower” provides no information for the Oxford dataset). In addition, we apply recurrent affine transformation (RAT) in the diffusion model for handling complex textual information.

The contributions of this paper are summarized as follows:

- •
We propose a new data augmentation method for text-to-image generation using linear extrapolation. Specifically, we apply linear extrapolation only on text feature, and new image data are retrieved from the internet by search engines.
- •
We propose a NULL-condition guidance to refine the score estimation for text-to-image generation. This guidance is also applicable to existing text-to-image models without further training.
- •
We apply recurrent affine transformation in the diffusion model for handling complex textual information.

## 2 Related work

#### GAN-based text-to-image models.

Text-to-image synthesis is a key task within conditional image synthesis (Feng et al., ; Tan et al., ; Peng et al., ; Hou et al., ). The pioneering work of (Reed et al., ) first tackled this task using conditional GANs (Mirza & Osindero, ). To better integrate text information into the synthesis process, DF-GAN (Tao et al., ) introduced a deep fusion method featuring multiple affine layers within a single block. Unlike previous approaches, DF-GAN eliminated the normalization operation without sacrificing performance, thus reducing computational demands and alleviating limitations associated with large batch sizes. Building on DF-GAN, RAT-GAN employed a recurrent neural network to progressively incorporate text information into the synthesized images. GALIP (Tao et al., ) and StyleGAN-T (Sauer et al., ) explore the potential of combining GAN models with transformers for large-scale text-to-image synthesis. However, the aforementioned GAN-based models often struggle to produce high-quality images.

#### Diffusion-based text-to-image models.

Recently, diffusion models (Ho et al., ; Song & Ermon, ; Song et al., ; Hyvärinen, ) have demonstrated impressive generation performance across various tasks. Building on this success, Imagen (Saharia et al., ) and DALL·E 2 (Ramesh et al., ) can synthesize images that are sufficiently realistic for real-world applications. To alleviate computational burdens, they first generate 64×64 images and then upsample them to high-resolution using another diffusion model. Additionally, the Latent Diffusion Model (Rombach et al., ) encodes high-resolution images into low-resolution latent codes, avoiding the exponential computation costs associated with increased resolution. DiT (Peebles & Xie, ) integrated latent diffusion models and transformers to enhance performance on large datasets. VQ-Diffusion (Gu et al., ) pre-train their text-to-image model on the Conceptual Caption dataset, which contains 15 million text-image pairs, and then fine-tune it on smaller datasets like CUB, Oxford, and COCO. Hence, VQ-Diffusion is the work most similar to ours but we use significantly less pre-training data while achieving better results.

#### Data augmentation methods.

Data augmentation increases training data to improve the performance of deep learning applications, from image classification (Krizhevsky et al., ) to speech recognition (Graves et al., ; Amodei et al., ). Common techniques include rotation, translation, cropping, resizing, flipping (LeCun et al., ; Vedaldi & Zisserman, ), and random erasing (Zhong et al., ) to promote visually plausible invariances. Similarly, label smoothing is widely used to boost the robustness and accuracy of trained models (Müller et al., ; Lukasik et al., ). Mixup (Zhang et al., ) involves training a neural network on convex combinations of examples and their labels. However, interpolated samples fail to introduce new information and effectively address data scarcity. Hence, Re-imagen (Chen et al., ; Sheynin et al., ; Li et al., ) retrieval relevant training data from external databases to augment training data.

## 3 Linear Extrapolation for Text-to-image Generation

In this section, we begin by collecting similar images from the internet. Next, we explain how to extrapolate text descriptions. Following that, we use the extrapolated text-image pairs to train a diffusion model with RAT blocks. Finally, we sample images using NULL-condition guidance.

### 3.1 Collecting Similar and Clean Images

Linear extrapolation requires the images to be sufficiently close in semantic space. Hence, we automatically retrieve similar images by searching for their classification labels. However, search engines return both similar images and outliers. To eliminate unwanted outliers, we employ a cluster detector for irrelevant outliers and a classification detector for similar outliers. For the cluster detector, each image is encoded into a vector using the CLIP image encoder. Images retrieved with the same keyword are then clustered using K-means. If the distance from the cluster center to dataset images exceeds a threshold, this cluster is excluded. For the classification detector, we train a fine-grained classification model on the original dataset, which assigns a label to each web image. If the label does not match with the search keyword, corresponding image is then excluded.

### 3.2 Linear Extrapolation on Text Feature Space

Here we introduce how to extrapolates text descriptions for web images. Assuming that web images are sufficiently close to dataset images in semantic space, each web image can be represented by nearest k images:

$$ $\operatorname*{arg\,min}_{W}|\textbf{f }-\textbf{F}\times\textbf{w}|^{2},$ (1) $$

where $\textbf{w}=[w_{1},w_{2},...,w_{k}]$are the reconstruction weights and $\textbf{F}=[\textbf{f}_{1},\textbf{f}_{2},...,\textbf{f}_{k}]$ are the image features of dataset images produced by CLIP image encoder. Since the above equation is a super-determined problem, we solve this coefficient using least squares:

$$ $\textbf{w}=(\textbf{F}^{T}\textbf{F})^{-1}\textbf{F}^{T}\textbf{f}.$ (2) $$

We assume that the image feature space and text feature space share the same local manifold. Hence, the image reconstruction efficient w can be used to compute the text feature of web images:

$$ $\textbf{s}=\textbf{S}\times\textbf{w},$ (3) $$

where $\textbf{S}=[\textbf{s}_{1},\textbf{s}_{2},...,\textbf{s}_{k}]$ is the fake sentence features for nearest k dataset images, and s is the sentence feature for a web image.

### 3.3 Recurrent Diffusion Transformer on Latent Space

Figure: Figure 2: Latent diffusion model with recurrent affine transformation and NULL-guidance for text-to-image synthesis. The RAT blocks are connected by a recurrent neural network to ensure the global assignment of text information.
Refer to caption: x2.png

The training objective of the diffusion model is the squared error loss proposed by DDPM (Ho et al., ):

$$ $L(\theta)=\left\|\epsilon-\epsilon_{\theta}\left(\sqrt{\overline{\bar{\alpha}} _{t}}\mathrm{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon\right)\right\|^{2},$ (4) $$

where $\epsilon\in N(0,1)$ is the score noise injected at every diffusion step, and $\epsilon_{\theta}$ is the predicted noise by a diffusion network consisted of 12 transformer layers. $\overline{\bar{\alpha}}_{t}$ and $\bar{\alpha}_{t}$ are hyper-parameters controlling the speed of diffusion. The work of score mismatching (Ye & Liu, ) shows that predicting the score noise leads to an unbiased estimation.

#### Network architecture.

As depicted in Fig 8, the diffusion network consists of transformer blocks. Recurrent affine transformation is used to enhance the consistency between transformer blocks. To avoid directly mixing text embedding and time embedding, we stack four transformer blocks as a RAT block and text embedding is fed into the top of each RAT block.
Each RAT block applies a channel-wise shifting operation on a image feature map:

$$ $c^{\prime}=c+\beta,$ (5) $$

where $c$ is the image feature vector and $\beta$ is shifting parameters predicted by a one-hidden-layer multi-layer perception (MLP) conditioned on recurrent neural network hidden state $h_{t}$.

In each transformer block, we inject time embedding by a channel-wise scaling operation and a channel-wise shifting operation on $c$. At last, the image feature $c$ is multiplied by a scaling parameter $\alpha$. This process can be formally expressed as:

$$ $c^{\prime}=Transformer((1+\gamma)\cdot c+\beta)\cdot\alpha,$ (6) $$

where $\alpha,\gamma,\beta$ are parameters predicted by two one-hidden-layer MLPs conditioned on time embedding.

When applied to an image feature map composed of $w\times h$ feature vectors, the same affine transformation is repeated for every feature vector.

#### Early stop of fine-tuning.

Extrapolation may produces training data very close to the original dataset, which makes fine-tuning saturate very quickly.
Excessive fine-tuning epochs would forget knowledge gained from the extrapolated data and overfit small datasets. As a result, the training loss of the diffusion model becomes unreliable. Therefore, fine-tuning should be stopped when the FID score begins to increase.

### 3.4 Synthesizing Fake Images

Finally, we introduces how to synthesizing images from scratch. As depicted in Figure 8, the synthesis begins with sampling a random vector $z$ from standard Gaussian distribution. And then, this noise is gradually denoised into an image latent code by the diffusion model. The reverse diffusion iterations are formulated as:

$$ $\mathbf{x}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\frac{1- \alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}\left(\mathbf{x}_{t},t \right)\right)+\sigma_{t}\mathbf{z},$ (7) $$

where, $\alpha_{t}$, $\bar{\alpha}_{t}$ and $\sigma_{t}$ are diffusion hyper-parameters, and $z$ is a random vector sampled from standard Gaussian distribution. At last, we decode image latent codes into images with the pre-trained decoder from Stable Diffusion (Rombach et al., ).

#### NULL guidance.

A sentence with no new information is able to boost text-to-image performance obviously. This guidance is inspired by Classifier-free diffusion guidance (Ho & Salimans, ) which uses a dummy class label to boost label-to-image performance. Similarly, we design CLIP prompt without obvious visual meaning and embed them into the diffusion model. Specifically, we denote the original score estimation based on text description as $\epsilon_{text}$ and score estimation based on null description as $\epsilon_{null}$. Then we mix these two estimations for a more accurate estimation $\epsilon^{\prime}$:

$$ $\epsilon^{\prime}=(\epsilon_{text}-\epsilon_{null})\times\eta+\epsilon_{null},$ (8) $$

where, $\eta$ is the guidance ration controlling the balance of two estimations. When $\eta=1$, NULL Guidance falls back to an ordinary score estimation. Usually, a NULL prompt with the average meaning of the dataset achieve the best performance.

## 4 Experiments

Figure: Figure 3: Qualitative comparison on the CUB and Oxford dataset. The input text descriptions are given in the first row and the corresponding generated images from different methods are shown in the same column. Best view in color and zoom in.
Refer to caption: extracted/5896104/fig/fig_1/real/1.jpg

#### Datasets.

We report results on the popular CUB, Oxford-102, and MS COCO datasets. The CUB dataset includes 200 categories with a total of 11,788 bird images, while the Oxford-102 dataset contains 102 categories with 8,189 flower images. Unlike the approaches taken in Reed et al. (; ), we utilize the entire dataset for both training and testing. Each image is paired with 10 captions. To expand the original datasets, we collect 300,000 bird images and 130,000 flower images. The MS COCO dataset comprises 123,287 images, each with 5 sentence annotations. We use the official training split of COCO for training and the official validation split for testing. During mini-batch selection, a random image view (e.g., crop or flip) is chosen for one of the captions.

#### Web images.

For the CUB and Oxford datasets, we collected 603,484 bird images and 331,602 flower images using search engines, utilizing fine-grained classification labels as search keywords. After removing detected outliers, we retained 399,246 bird images and 132,648 flower images. In the case of the COCO dataset, we gathered 770,059 daily images without applying any outlier detection, as the precise descriptions in COCO allow search engines to retrieve clean images effectively.

#### Training details.

The text encoder is a pre-trained CLIP text encoder with an output of size $512$. The latent encoder and decoder is pre-trained by Stable Diffusion (Rombach et al., ). We have tried to pre-train new latent encoders on extrapolated data but the results are not satisfying.
Adam optimizer is used to optimize the network with base learning rates of 0.0001 and weight decay of 0. The same as RAT-GAN, we used a mini-batch size of 24 to train the model. Most training and testing of our model are conducted on 2 RTX 3090 Ti and the detailed training consumption is listed in Table 3.

#### Evaluation metrics.

We adopt the widely used Inception Score (IS) (Salimans et al., ) and Fréchet Inception Distance (FID) (Heusel et al., ) to quantify the performance. On the MS COCO dataset, an Inception-v3 network pre-trained on the ImageNet dataset is used to compute the KL-divergence between the conditional class distribution (generated images) and the marginal class distribution (real images). The presence of a large IS indicates that the generated images are of high quality. The FID computes the Fréchet Distance between the image feature distributions of the generated and real-world images. The image features are extracted by the same pre-trained Inception v3 network. A lower FID implies the generated images are closer to the real images. We only compare the FID on the COCO dataset. On the CUB and Oxford-102 dataset, pre-trained Inception models are fine-tuned on two fine-grained classification tasks (Zhang et al., ).

There are two conflicts in evaluation methods in previous works. First, some studies report Inception Score (IS) using the ImageNet Inception model, while others use a fine-tuned version. Second, some works evaluate using the entire training data, whereas others use only the test split. To address these inconsistencies, we report IS and FID using both Inception models and employ the same Inception model as DM-GAN for consistency. Additionally, to resolve conflicts related to data splits, we report the FID scores of our model and other re-implemented models using the entire dataset for both training and testing. According to results from RAT-GAN (Ye et al., ), training and testing on the full dataset typically yields the best FID scores. We will also release all evaluation codes on GitHub.

#### Compared models.

We compare our model with recent state-of-the-art methods: StackGAN++ (Zhang et al., ), DM-GAN (Zhu et al., ), DF-GAN (Tao et al., ), DAE-GAN (Ruan et al., ), VQ-diffusion (Gu et al., ), AttnGAN (Xu et al., ),
GALIP (Zhang & Schomaker, ),U-ViT (Bao et al., ), and RAT-GAN (Ye et al., ).

### 4.1 Comparisons with Others

Table: Table 1: Performance of IS and FID of StackGAN++, AttnGAN, SSGAN, DM-GAN, DTGAN, DF-GAN and our method on the CUB, Oxford and MS COCO datasets. The results are taken from the authors’ own papers. The best results are in bold.

#### Quantitative results.

We present results for the CUB dataset of bird images, the Oxford-102 dataset of flower images, and the MS COCO dataset of common objects, as shown in Table 1. On the CUB dataset, our model achieve an IS score of 6.56 and an FID score of 6.36, outperforming all the previous models. For the Oxford dataset, we achieve an IS score of 4.35 and an FID score of 6.36, outperforming all the previous models. On the COCO dataset, our model achieves an FID score of 5.00 that is competitive with previous best result.Compared with VQ-Diffusion, our model uses less training data and achieve much better performance. This comparison reveals that pre-training on large datasets can be inefficient and lead to suboptimal results.
Moreover, results in Table 1 reveal that Inception model pre-trained on ImageNet is less sensitive than fine-tuned on small datasets.
Additionally, the Inception score on the Oxford dataset exceeds that of real images (4.10). Extensive results demonstrate the effectiveness and generalization ability of the proposed data extrapolation method.

#### Qualitative results.

We present qualitative results for the CUB dataset of bird images and the Oxford-102 dataset of flower images. In Figure 3
, we compare the visualization results of DF-GAN, RAT-GAN, and our model. DF-GAN and RAT-GAN are previous state-of-the-art methods for text-to-image synthesis.
On the CUB dataset, with more clear details such as feathers, eyes, and feet, our model clearly outperforms DF-GAN and RAT-GAN. Additionally, the background in our model’s results is more coherent compared to RAT-GAN. On the Oxford dataset, our model exhibits better texture and more relevant colors than the others. With the proposed text extrapolation, RAT block, and null-guidance, our model demonstrates fewer distorted shapes and more relevant content compared to the other two models.

Figure: Figure 4: Qualitative comparison of our model with RAT-GAN on the COCO dataset.
Refer to caption: extracted/5896104/fig/fig_coco/ratgan/1.png

The qualitative results for the COCO dataset are shown in Figure 4. The COCO dataset includes a wide variety of common objects, which makes it particularly susceptible to the long-tail problem (Chen et al., ). With additional training data obtained through extrapolation, our model generates more realistic objects compared to RAT-GAN. However, the collected 770,059 images are still insufficient to cover the entire distribution of images in COCO. As a result, the outputs from COCO are not as realistic as those from the CUB and Oxford datasets.

### 4.2 Ablation Studies

**Table 2: Ablation studies on the CUB dataset. We utilize a NULL-guidance ratio of 1.5 during sampling. The FID score was employed to evaluate generation performance.**
| ID | Component | Extrapolation Quantity(k) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Cluster | Classification | RAT | NULL | 0 | 50 | 100 | 200 | 300 | 400 |  |
| 0 | - | - | - | - | 16.74 | - | - | - | - | 30.78 |
| 1 | ✓ | - | - | - | - | - | - | - | - | 20.67 |
| 2 | - | ✓ | - | - | - | - | - | - | - | 12.45 |
| 3 | ✓ | ✓ | - | - | - | - | - | - | - | 9.87 |
| 4 | ✓ | ✓ | ✓ | - | - | - | - | - | - | 8.76 |
| 5 | ✓ | ✓ | - | ✓ | - | - | - | - | - | 7.65 |
| 6 | ✓ | ✓ | ✓ | ✓ | - | 9.56 | 7.34 | 6.87 | 6.54 | 6.36 |

#### Analysis of outlier detectors.

In Table 2, we present text-to-image results without cluster detector or classification detector. According to ID 0,1 and 2, the FID score without outlier detectors degrade severely because noisy images force the diffusion model to generate irrelevant objects. Although fine-tuning on small datasets could alleviate noise pollution but parameters also forget general knowledge at the same time. According to ID 2 and 3, classification detector performs better than cluster detector because it has utilized fine-grained classification labels.

#### Analysis of extrapolation quantity.

More images generally lead to improved text-to-image results. however, this trend saturates around 100,000 images, after which the improvement in FID becomes less significant with more training samples. This phenomenon aligns with that diffusion models perform much better than GANs on the COCO dataset (84K images) but exhibit similar performance to GANs on the CUB and Oxford datasets( 10K images). Furthermore, with transformers as core building blocks, GALIP performs similarly to previous models on the CUB dataset. This suggests that transformer architectures exacerbate the need for larger training datasets.

**Table 3: Training consumption on the CUB, Oxford and COCO datasets. Fine-tuning is performed on the original dataset until the FID scores increase.**
| Dataset | Device | Original dataset | Extrapolated data | Fine-tuning |
| --- | --- | --- | --- | --- |
| CUB | 2 RTX 3090 Ti | 5 days/1500 epochs | 10 days/100 epochs | 6 hours/50 epochs |
| Oxford | 2 RTX 3090 Ti | 5 days/1500 epochs | 8 days/200 epochs | 6 hours/50 epochs |
| COCO | 2 RTX 4090 | 10 days/125 epochs | 20 days/95 epochs | 7 days/70 epochs |

**Table 4: The impact of various NULL prompts on FID scores in the CUB dataset.**
| NULL Prompts | Guidance Ratio |  |  |
| --- | --- | --- | --- |
| 1.25 | 1.5 | 2.0 |  |
| “Null” | 7.23 | 7.16 | 7.68 |
| “a picture” | 6.89 | 6.54 | 7.14 |
| “no description” | 6.97 | 6.47 | 7.25 |
| “a picture of bird” | 6.46 | 6.36 | 6.86 |
| “a picture of flower” | 9.04 | 10.6 | 11.4 |
| “we don’t know what it is” | 8.98 | 9.35 | 9.94 |

#### Analysis of NULL guidance.

The performance of NULL guidance is influenced by both the NULL prompt and the guidance ratio. The results in Table 4 indicate that a NULL prompt reflecting the average meaning of the dataset achieves the best performance. Additionally, a suitable guidance ratio is crucial for optimal results, and we find that a ratio around 1.5 yields the best performance on the CUB and COCO datasets. However, on the Oxford dataset, NULL guidance improves the Inception Score from 4.10 to 4.35 but degrades the FID score from 9.52 to 11.07.

**Table 5: Ablation studies on the MS COCO dataset. We adopt “A picture” as the NULL prompt.**
| Training data | FID score |  |  |
| --- | --- | --- | --- |
| $\eta=1.0$ | $\eta=1.5$ | $\eta=2.0$ |  |
| COCO | 11.89 | 7.99 | 8.43 |
| Extrapolation | 12.33 | 8.41 | 9.24 |
| COCO-ft | 8.45 | 5.00 | 5.56 |

#### Analysis of text injection.

Text injection is crucial for text-to-image generation. As shown in ID 4 and 5 of Table 2, RAT significantly improves the FID score. Further experiments indicate that directly mixing text feature with time embedding results in an FID score of 25.41, which is much worse than 16.74 achieved by RAT. This suggests that time embedding provides information very different to text embedding. Additionally, incorporating a scaling operator into RAT can lead to model collapse, as information becomes highly compressed in latent space. Consequently, the mean value of the latent code becomes sensitive, and the scaling operation disrupts the information structure.

#### Ablation studies on the MS COCO dataset.

We conduct ablation studies on the MS COCO dataset, as presented in Table 5. The MS COCO dataset differs significantly from the CUB and Oxford datasets in terms of variety and image quantity. Experimental results demonstrate that linear extrapolation and fine-tuning (5.00) outperform the original COCO dataset (7.99). However, unlike CUB and Oxford, fine-tuning on COCO requires much more time, as shown in Table 3. Additionally, we observe that early stopping is unnecessary for fine-tuning on the COCO dataset due to its larger image volume compared to CUB and Oxford.

In Table 6, we compare the pre-training dataset and model parameters with previous models on the MS COCO dataset. The compared models are all pre-trained on external datasets and fine-tuned on MS COCO dataset. Our result outperforms all previous models except for Parti but we use much less pre-training images and parameters than Parti. Moreover, our diffusion model is designed for small datasets and requires very few GPUs for training.

Figure: Figure 5: Randomly generated images from the Oxford dataset. Best view in color and zoom in.
Refer to caption: extracted/5896104/fig/fig_f/variaty/1.png

#### Diversity.

To qualitatively evaluate the diversity of our proposed model, we generate random images conditioned on the same text description and different random noises. In Figure 5, we present 10 images generated from the same text. These images exhibit similar foreground elements while showcasing high diversity in spatial structure, demonstrating that our model effectively controls the image content.

**Table 6: Comparison of pre-training dataset and parameter quantity of different models on the MS COCO dataset. Parameters for text encoder, latent encoder and super resolution are not counted.**
| Model | FID | Type | Pre-training images | #Params |
| --- | --- | --- | --- | --- |
| Parti (Yu et al., ) | 3.22 | Autoregressive | 4.8B | 20B |
| Make-A-Scene (Gafni et al., ) | 7.55 | Autoregressive | 35M | 4B |
| Re-Imagen (Chen et al., ) | 5.25 | Diffusion | 50M | 2.5B |
| VQ-Diffusion (Gu et al., ) | 19.75 | Diffusion | 15M | 370M |
| Ours | 5.00 | Diffusion | 7M | 464M |

## 5 Conclusion and Future work

In this paper, we propose a new data augmentation method for text-to-image generation using linear extrapolation. Specifically, we apply linear extrapolation only on text data, and new image data are retrieved from the internet by search engines. For the reliability of new text-image pairs, we design two outlier detectors to purify retrieved images. Based on extrapolation, we construct training samples dozens of times larger than the original dataset, resulting in a significant improvement in text-to-image performance. Moreover, we propose a NULL-condition guidance to refine the score estimation for text-to-image generation. This guidance is also applicable to existing text-to-image models without further training. In the future, linear extrapolation and NULL-condition guidance could be applied to tasks beyond text-to-image generation.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: - Why did you choose image retrieval as the data augmentation method? What advantages does the proposed approach offer over generating synthetic datasets with generative models (e.g., recent models like Stable Diffusion)? In what aspects is retrieval beneficial?
- Could this approach benefit other downstream tasks, such as classification? Why is the method only applied to text-to-image generation in this paper?
- What does "ID" represent in Table 2? The paper does not provide an explanation. Additionally, ID=6 appears to perform well in Table 2; would increasing this value further improve performance?
- Why was the MS-COCO dataset selected, and what advantage does it demonstrate for the proposed approach? Considering the large scale of COCO (with millions of images), the paper states that “770,059 images are sufficient to cover the entire distribution of images in COCO.” How many images do you think would be necessary? Providing an estimate could improve the practicality of this method.
- Following up on the previous question, is this paper’s method specifically for augmenting smaller datasets, as it is mentioned that it cannot cover the full distribution of COCO? If so, could it be an effective data augmentation method in such cases?
soundness: 1
strengths: - The proposed approach appears effective, as shown in Table 1.
- The paper provides extensive analysis and ablation studies.
summary: This paper proposes linear extrapolation as a novel data augmentation method for text-to-image generation. Specifically, the extrapolation is applied only to the text, and new images are retrieved from the web. Outlier detection techniques are then used to filter out irrelevant data points.
weaknesses: - The paper suffers from poor presentation. The motivation for the proposed method is unclear, and it lacks a smooth narrative explaining why data extrapolation is necessary or beneficial in certain scenarios. Additionally, the reason behind retrieving images from the web is not well-explained. In other words, the rationale behind the overall pipeline is unclear.
- Practical concerns arise regarding the feasibility of this method. How long does it take to retrieve millions of images from the web and filter out outliers? Can this realistically serve as a data augmentation method in practice?
- The qualitative results (particularly in Figure 4) do not appear promising, raising doubts about the effectiveness of the proposed approach. For example, the face of the man is not properly generated, and the prompt specifies “red and green flower,” but no green flower is present in the output.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: **Terms Usage of Extrapolation:** While the paper claims to use extrapolation, the actual technique employed appears closer to interpolation, as it utilizes least square projection. Clarifying this aspect would be appreciated.

**Table 1 Comparison:** It is unclear if Table 1 offers a fair comparison. The other models seem to have been trained on each dataset individually, while this approach alone appears to include an additional web-crawled dataset. Could you provide the detailed settings for the comparison?
soundness: 1
strengths: - The paper is well-written and easy to follow, with clear presentations of each method that facilitate understanding.
- The proposed RAT block and NULL guidance are experimentally very effective. However, they are based on well-established methods, AdaIN and CFG, and the distinctions between these and the proposed approach are not adequately highlighted in either explanations or experiments.
- Synthesizing fake sentence features via projection for unlabeled image datasets without text descriptions is intriguing. However, it would be beneficial to cite DeCap and analyze the differences between this approach and DeCap’s method.

Li et al. "Decap: Decoding clip latents for zero-shot captioning via text-only training." ICLR. 2023.
summary: The paper proposes a method for training text-to-image generation models on small datasets by leveraging web-crawled images. This approach enables the use of image datasets without text descriptions by synthesizing text descriptions based on the available dataset. However, the paper does not adequately explain the necessity of addressing this problem, the comparisons lack fairness, and the proposed RAT block and NULL guidance do not make meaningful contributions.
weaknesses: **Task Justification:** The paper does not clearly explain why a generally well-performing text-to-image generation model needs to be adapted for datasets like CUB, Oxford, and COCO in a text-to-image generation context. It is unclear if the primary focus is to address the image distribution shift (e.g., may be COCO?) or the text description distribution shift (e.g., the CUB dataset, where text is very simple). A clearer justification for this task is needed.

**Method Justification 1:** Is it necessary to train a text-to-image generation model from scratch on small datasets? Numerous well-trained general text-to-image models (e.g., Stable Diffusion, SD-XL, SD3 or 3.5, FLUX.1, DALL-E, etc.) are readily available. If the goal is to shift image distribution for smaller datasets, many researchers simply fine-tune these large-scale, pre-trained models. It is essential to explain why this approach is better than existing fine-tuning methods, such as full fine-tuning or LoRA fine-tuning with varying ranks. 
  - Specifically, it would be helpful if the paper could report performance comparisons for recent text-to-image generation models (e.g., SD, SDXL, SD3) under zero-shot, LoRA-fine-tuned, and full-fine-tuned settings.

**Method Justification 2:** Recently, there has been substantial progress in training with images crawled from the web, where captions are generated using vision-language models (VLMs) like GPT-4 and BLIP [e.g., DALL-E 3]. An analysis of performance differences between synthetic captions from these approaches and the extrapolated captions generated by this method would be beneficial.

**Network Architecture and NULL Guidance:** The proposed network architecture and NULL guidance do not seem to provide particularly novel insights or advantages. When compared to methods frequently used in prior diffusion model research, such as AdaIN and CFG, the added benefits of this approach remain unclear.

**Table 6 Comparison:** the authors highlight that their model uses fewer pre-training images than Parti. However, it may be worth reconsidering this point, as maximizing efficient training on large datasets is generally more important in pre-training. Additionally, Table 6 includes models only up to 2022; models based on Stable Diffusion from 2023 and 2024 should be included for an updated comparison.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: Regarding the 3 contributions mentioned in this paper, I believe:

1. The approach of text extrapolation followed by an online image search seems more of a practical implementation choice than a novel, research-driven idea. While it is likely to yield performance improvements, its impact may not significantly diverge from that of other augmentation methods.

2. The NULL guidance mechanism does not present substantial distinctions from classifier-free guidance (CFG).

3. The inclusion of RAT raises questions, as the comparisons do not incorporate more recent, robust baselines for text processing within text-to-image diffusion models.
soundness: 2
strengths: The method is simple and well-explained.
Experiments cover 3 different datasets.
Ablations are comprehensive.
summary: This paper presents a data augmentation approach for text-to-image generation on small datasets, using linear extrapolation on text features to retrieve relevant images from the internet to augment training data, with outlier detection ensuring quality. Recurrent affine transformations and NULL-guidance are introduced to the diffusion model to improve text-image alignment, boosting generative performance under limited initial data.
weaknesses: 1. Limited Novelty and Impact: The originality and impact of the methods presented are limited. The retrieval-augmented diffusion model itself is not novel. While this work proposes a specific technique for extrapolating text and searching for augmentation images online, such retrieval-based methods are already common in industrial applications. Additionally, the NULL guidance technique appears the same as Classifier-Free Guidance (CFG). Although CFG was initially introduced for label-to-image generation, it is now widely used across models like Stable Diffusion 3. 

2. Limited Evaluation: The evaluation is insufficient for a few reasons. Since the proposed method includes a data collection phase, demonstrating its advantage should involve time complexity analysis. Currently, the benefit of standard fine-tuning on pretrained models remains unclear. Text-to-image generation is typically assessed with additional metrics like CLIP-Score or human evaluations for alignment accuracy, but this work is limited to quality evaluations, overlooking essential alignment-focused measures.

3. Limited Comparison and Improvement: The qualitative results are only compared against GAN-based models, with minimal improvement demonstrated. Given that this method employs a diffusion-based approach, a qualitative comparison with other recent diffusion-based backbones, such as DiT or U-ViT, would be far more relevant and indicative of its potential impact. The use of RAT on diffusion models should be compared with AdaNorm, AdaRMSNorm, cross-attention, and other similar text guidance architectures used in more recent diffusion models.

4. Questionable Baseline Choices: The baseline choices for evaluation raise concerns. The proposed method requires additional data, while most baselines rely solely on static datasets for pretraining and fine-tuning. To truly showcase the method's advantage, a time complexity comparison between the data collection process and a traditional training approach on a similarly sized additional dataset would offer a more meaningful assessment. The

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: - Please address the weaknesses section, specifically the limited novelty.
- Have you tried this approach for finetuning a pre-trained text-image model? if not why not.
- Can you please explain the novelty regarding the NULL guidance? it is unclear to me if it is claimed to be a novelty or is it simply a technical detail.
soundness: 2
strengths: - Augmentation using external source of data.
- Extrapolate text embedding based on k-nearest neighbors of the image.
- Architecture design.
- Ablation validating different components.
summary: The paper proposes an alternative approach to augmentation for text-to-image training. Unlike traditional augmentation techniques, which produce new text-image pairs by modifying the text or images, the proposed method extrapolates text features, retrieves additional images from the internet, and removes outliers.

The paper also introduces novelties in sampling procedures, updates to the well-known classifier-free guidance (CFG), and slight modifications to the transformer architecture. The results show improvements over reference images created using the same datasets.
weaknesses: - The novelty of the method is limited  - using third party images as source and outlier detection, which by itself is not new.  Since this is a more engineering approach, the main novelty is reduced to the pairing of crawled images with text encoding, as well as outlier detection for these images. For me this does not meet the bar of acceptance for ICLR. However, I'm open hearing from the authors and this might affect my final decision.

- The test is done on CUB and MS-COCO, I would like to see a more stress test of this approach. It is unclear when this approach is not valid since the outlier detector relies on having enough paired text-to-image data. I wish to see additional results stressing this approach to more limited scenario.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1、Is the recurrent affine transformation (RAT) mentioned in Line 072 similar to the one used in RAT-GAN? If so, it would be better to cite the relevant work where we first introduce the concept to avoid confusion. If not, it would be better to discuss the difference.

2、It could be better to show more samples to illustrate the diversity. 

3、Since the technique involves collecting images from websites, the authors should discuss the broader impact and potential risks related to privacy and legal issues. And in this work, the authors should clarify the specific source of collected images for training and ensure legal compliance.

4、See Weakness.


**Minor Issue**

Line 032-033: 2022)is

Line 076-083: Use too much “we”. It could be better to involve more flexible sentence form 

In the supplementary material, since there is ample space, there is no need to omit any text descriptions.
soundness: 2
strengths: - This paper is well-written and easy to follow with clear figure to illustrate data linear extrapolation and network architecture.

- Experiments are conducted on three datasets including CUB, Oxford, and COCO datasets and ablation to show the effectiveness of the proposed methods with fair comparisons.
 
- The setting of small-scale text-to-image dataset is interesting.
summary: The authors propose to use data extrapolation for text-to-image generation on small-scale datasets. Moreover, they leverage an interesting NULL-guidance to refine score estimation, such as an abstract generalization, “a picture of bird”. They also apply recurrent affine transformation in the diffusion model to deal with complex text input. Experiments are conducted on CUB, Oxford, and COCO datasets to show the effectiveness of these methods. And code is promised to be released.
weaknesses: **Longer training time:** It seems that the authors use basically double training time to achieve superior performance

**Limited application scenario of proposed data extrapolation:** In my view, text-to-image datasets are abundant. More importantly, the provided text description may be abundant and similar to that in COCO. Hence, as the authors say in Line269, potentially, we may avoid the outlier detection. Moreover, what if we could not train a fine-grained classifier on the original dataset which may contain complex scenario?

**Unreasonable layout:** It seems that in Sec 3, only Sec 3.1 and 3.2 are related to Data Linear Extrapolation while the rest is not. Could the authors clarify how the other subsections relate to the main theme of Data Linear Extrapolation?

**Weak connection between data linear extrapolation and another two contributions:** From the title, it seems that the authors want to highlight the importance of data linear extrapolation. However, they additionally involve another two tricks unrelated to data linear extrapolation without more explanation. The readers may feel that the logic behind is not clear and the motivation is not persuasive. In other words, it may make readers feel that the three contributions are isolated, failing to formulate a complete story. Could the authors provide a clearer explanation of how these three contributions work together or why they were chosen to be combined in this work?

**Lack more suitable comparison:** Intuitively, to demonstrate the effectiveness of data linear extrapolation, the authors should conduct experiments on existing methods such as GAN-based RAT-GAN and diffusion-based U-ViT, using datasets like CUB, Oxford, and MS COCO, as this is a general technique. For example, it seems that using all three tricks only brings slight improvement compared to UViT. Is there still an improvement when we use these tricks on UViT collectively and separately? Same question for other models, like RAT-GAN. Similarly, in Table 2, after using data linear extrapolation (ID 3), the performance is still similar to some models like VQ-Diffusion and GALIP. Though we know that data linear extrapolation is employed on a newly designed architecture, it may still cause confusion.