Data Extrapolation for Text-to-image Generation on Small Datasets

Authors: Senmao Ye
Primary area: generative models
Keywords: Diffuison, Text-to-image，Data augmentation

Abstract:
Text-to-image generation requires large amount of training data to synthesizing high-quality images. For augmenting training data, previous methods rely on data interpolations like cropping, flipping, and mixing up, which fail to introduce new information and yield only marginal improvements. In this paper, we propose a new data augmentation method for text-to-image generation using linear extrapolation. Specifically,  we apply linear extrapolation only on text feature, and new image data are retrieved from the internet by search engines. For the reliability of new text-image pairs, we design two outlier detectors to purify retrieved images. Based on extrapolation, we construct training samples dozens of times larger than the original dataset, resulting in a significant improvement in text-to-image performance. Moreover, we propose a NULL-guidance to refine score estimation, and apply recurrent affine transformation to fuse text information.  Our model achieves FID scores of 7.91, 9.52 and 5.00 on the CUB, Oxford and COCO datasets. The code and data will be available on GitHub.

Decision: reject
Reviewer scores: 3, 3, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1729687015353

thread_type:
official_review

<thread_object>
creator: reviewer_egsf

<content>
summary:
The authors propose to use data extrapolation for text-to-image generation on small-scale datasets. Moreover, they leverage an interesting NULL-guidance to refine score estimation, such as an abstract generalization, “a picture of bird”. They also apply recurrent affine transformation in the diffusion model to deal with complex text input. Experiments are conducted on CUB, Oxford, and COCO datasets to show the effectiveness of these methods. And code is promised to be released.

strengths:
- This paper is well-written and easy to follow with clear figure to illustrate data linear extrapolation and network architecture.

- Experiments are conducted on three datasets including CUB, Oxford, and COCO datasets and ablation to show the effectiveness of the proposed methods with fair comparisons.
 
- The setting of small-scale text-to-image dataset is interesting.

weaknesses:
**Longer training time:** It seems that the authors use basically double training time to achieve superior performance

**Limited application scenario of proposed data extrapolation:** In my view, text-to-image datasets are abundant. More importantly, the provided text description may be abundant and similar to that in COCO. Hence, as the authors say in Line269, potentially, we may avoid the outlier detection. Moreover, what if we could not train a fine-grained classifier on the original dataset which may contain complex scenario?

**Unreasonable layout:** It seems that in Sec 3, only Sec 3.1 and 3.2 are related to Data Linear Extrapolation while the rest is not. Could the authors clarify how the other subsections relate to the main theme of Data Linear Extrapolation?

**Weak connection between data linear extrapolation and another two contributions:** From the title, it seems that the authors want to highlight the importance of data linear extrapolation. However, they additionally involve another two tricks unrelated to data linear extrapolation without more explanation. The readers may feel that the logic behind is not clear and the motivation is not persuasive. In other words, it may make readers feel that the three contributions are isolated, failing to formulate a complete story. Could the authors provide a clearer explanation of how these three contributions work together or why they were chosen to be combined in this work?

**Lack more suitable comparison:** Intuitively, to demonstrate the effectiveness of data linear extrapolation, the authors should conduct experiments on existing methods such as GAN-based RAT-GAN and diffusion-based U-ViT, using datasets like CUB, Oxford, and MS COCO, as this is a general technique. For example, it seems that using all three tricks only brings slight improvement compared to UViT. Is there still an improvement when we use these tricks on UViT collectively and separately? Same question for other models, like RAT-GAN. Similarly, in Table 2, after using data linear extrapolation (ID 3), the performance is still similar to some models like VQ-Diffusion and GALIP. Though we know that data linear extrapolation is employed on a newly designed architecture, it may still cause confusion.

questions:
1、Is the recurrent affine transformation (RAT) mentioned in Line 072 similar to the one used in RAT-GAN? If so, it would be better to cite the relevant work where we first introduce the concept to avoid confusion. If not, it would be better to discuss the difference.

2、It could be better to show more samples to illustrate the diversity. 

3、Since the technique involves collecting images from websites, the authors should discuss the broader impact and potential risks related to privacy and legal issues. And in this work, the authors should clarify the specific source of collected images for training and ensure legal compliance.

4、See Weakness.


**Minor Issue**

Line 032-033: 2022)is

Line 076-083: Use too much “we”. It could be better to involve more flexible sentence form 

In the supplementary material, since there is ample space, there is no need to omit any text descriptions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626259600

thread_type:
official_review

<thread_object>
creator: reviewer_jzmn

<content>
summary:
The paper proposes an alternative approach to augmentation for text-to-image training. Unlike traditional augmentation techniques, which produce new text-image pairs by modifying the text or images, the proposed method extrapolates text features, retrieves additional images from the internet, and removes outliers.

The paper also introduces novelties in sampling procedures, updates to the well-known classifier-free guidance (CFG), and slight modifications to the transformer architecture. The results show improvements over reference images created using the same datasets.

strengths:
- Augmentation using external source of data.
- Extrapolate text embedding based on k-nearest neighbors of the image.
- Architecture design.
- Ablation validating different components.

weaknesses:
- The novelty of the method is limited  - using third party images as source and outlier detection, which by itself is not new.  Since this is a more engineering approach, the main novelty is reduced to the pairing of crawled images with text encoding, as well as outlier detection for these images. For me this does not meet the bar of acceptance for ICLR. However, I'm open hearing from the authors and this might affect my final decision.

- The test is done on CUB and MS-COCO, I would like to see a more stress test of this approach. It is unclear when this approach is not valid since the outlier detector relies on having enough paired text-to-image data. I wish to see additional results stressing this approach to more limited scenario.

questions:
- Please address the weaknesses section, specifically the limited novelty.
- Have you tried this approach for finetuning a pre-trained text-image model? if not why not.
- Can you please explain the novelty regarding the NULL guidance? it is unclear to me if it is claimed to be a novelty or is it simply a technical detail.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681031032

thread_type:
official_review

<thread_object>
creator: reviewer_dunq

<content>
summary:
This paper presents a data augmentation approach for text-to-image generation on small datasets, using linear extrapolation on text features to retrieve relevant images from the internet to augment training data, with outlier detection ensuring quality. Recurrent affine transformations and NULL-guidance are introduced to the diffusion model to improve text-image alignment, boosting generative performance under limited initial data.

strengths:
The method is simple and well-explained.
Experiments cover 3 different datasets.
Ablations are comprehensive.

weaknesses:
1. Limited Novelty and Impact: The originality and impact of the methods presented are limited. The retrieval-augmented diffusion model itself is not novel. While this work proposes a specific technique for extrapolating text and searching for augmentation images online, such retrieval-based methods are already common in industrial applications. Additionally, the NULL guidance technique appears the same as Classifier-Free Guidance (CFG). Although CFG was initially introduced for label-to-image generation, it is now widely used across models like Stable Diffusion 3. 

2. Limited Evaluation: The evaluation is insufficient for a few reasons. Since the proposed method includes a data collection phase, demonstrating its advantage should involve time complexity analysis. Currently, the benefit of standard fine-tuning on pretrained models remains unclear. Text-to-image generation is typically assessed with additional metrics like CLIP-Score or human evaluations for alignment accuracy, but this work is limited to quality evaluations, overlooking essential alignment-focused measures.

3. Limited Comparison and Improvement: The qualitative results are only compared against GAN-based models, with minimal improvement demonstrated. Given that this method employs a diffusion-based approach, a qualitative comparison with other recent diffusion-based backbones, such as DiT or U-ViT, would be far more relevant and indicative of its potential impact. The use of RAT on diffusion models should be compared with AdaNorm, AdaRMSNorm, cross-attention, and other similar text guidance architectures used in more recent diffusion models.

4. Questionable Baseline Choices: The baseline choices for evaluation raise concerns. The proposed method requires additional data, while most baselines rely solely on static datasets for pretraining and fine-tuning. To truly showcase the method's advantage, a time complexity comparison between the data collection process and a traditional training approach on a similarly sized additional dataset would offer a more meaningful assessment. The

questions:
Regarding the 3 contributions mentioned in this paper, I believe:

1. The approach of text extrapolation followed by an online image search seems more of a practical implementation choice than a novel, research-driven idea. While it is likely to yield performance improvements, its impact may not significantly diverge from that of other augmentation methods.

2. The NULL guidance mechanism does not present substantial distinctions from classifier-free guidance (CFG).

3. The inclusion of RAT raises questions, as the comparisons do not incorporate more recent, robust baselines for text processing within text-to-image diffusion models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701365887

thread_type:
official_review

<thread_object>
creator: reviewer_iuy2

<content>
summary:
The paper proposes a method for training text-to-image generation models on small datasets by leveraging web-crawled images. This approach enables the use of image datasets without text descriptions by synthesizing text descriptions based on the available dataset. However, the paper does not adequately explain the necessity of addressing this problem, the comparisons lack fairness, and the proposed RAT block and NULL guidance do not make meaningful contributions.

strengths:
- The paper is well-written and easy to follow, with clear presentations of each method that facilitate understanding.
- The proposed RAT block and NULL guidance are experimentally very effective. However, they are based on well-established methods, AdaIN and CFG, and the distinctions between these and the proposed approach are not adequately highlighted in either explanations or experiments.
- Synthesizing fake sentence features via projection for unlabeled image datasets without text descriptions is intriguing. However, it would be beneficial to cite DeCap and analyze the differences between this approach and DeCap’s method.

Li et al. "Decap: Decoding clip latents for zero-shot captioning via text-only training." ICLR. 2023.

weaknesses:
**Task Justification:** The paper does not clearly explain why a generally well-performing text-to-image generation model needs to be adapted for datasets like CUB, Oxford, and COCO in a text-to-image generation context. It is unclear if the primary focus is to address the image distribution shift (e.g., may be COCO?) or the text description distribution shift (e.g., the CUB dataset, where text is very simple). A clearer justification for this task is needed.

**Method Justification 1:** Is it necessary to train a text-to-image generation model from scratch on small datasets? Numerous well-trained general text-to-image models (e.g., Stable Diffusion, SD-XL, SD3 or 3.5, FLUX.1, DALL-E, etc.) are readily available. If the goal is to shift image distribution for smaller datasets, many researchers simply fine-tune these large-scale, pre-trained models. It is essential to explain why this approach is better than existing fine-tuning methods, such as full fine-tuning or LoRA fine-tuning with varying ranks. 
  - Specifically, it would be helpful if the paper could report performance comparisons for recent text-to-image generation models (e.g., SD, SDXL, SD3) under zero-shot, LoRA-fine-tuned, and full-fine-tuned settings.

**Method Justification 2:** Recently, there has been substantial progress in training with images crawled from the web, where captions are generated using vision-language models (VLMs) like GPT-4 and BLIP [e.g., DALL-E 3]. An analysis of performance differences between synthetic captions from these approaches and the extrapolated captions generated by this method would be beneficial.

**Network Architecture and NULL Guidance:** The proposed network architecture and NULL guidance do not seem to provide particularly novel insights or advantages. When compared to methods frequently used in prior diffusion model research, such as AdaIN and CFG, the added benefits of this approach remain unclear.

**Table 6 Comparison:** the authors highlight that their model uses fewer pre-training images than Parti. However, it may be worth reconsidering this point, as maximizing efficient training on large datasets is generally more important in pre-training. Additionally, Table 6 includes models only up to 2022; models based on Stable Diffusion from 2023 and 2024 should be included for an updated comparison.

questions:
**Terms Usage of Extrapolation:** While the paper claims to use extrapolation, the actual technique employed appears closer to interpolation, as it utilizes least square projection. Clarifying this aspect would be appreciated.

**Table 1 Comparison:** It is unclear if Table 1 offers a fair comparison. The other models seem to have been trained on each dataset individually, while this approach alone appears to include an additional web-crawled dataset. Could you provide the detailed settings for the comparison?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706463497

thread_type:
official_review

<thread_object>
creator: reviewer_3fhw

<content>
summary:
This paper proposes linear extrapolation as a novel data augmentation method for text-to-image generation. Specifically, the extrapolation is applied only to the text, and new images are retrieved from the web. Outlier detection techniques are then used to filter out irrelevant data points.

strengths:
- The proposed approach appears effective, as shown in Table 1.
- The paper provides extensive analysis and ablation studies.

weaknesses:
- The paper suffers from poor presentation. The motivation for the proposed method is unclear, and it lacks a smooth narrative explaining why data extrapolation is necessary or beneficial in certain scenarios. Additionally, the reason behind retrieving images from the web is not well-explained. In other words, the rationale behind the overall pipeline is unclear.
- Practical concerns arise regarding the feasibility of this method. How long does it take to retrieve millions of images from the web and filter out outliers? Can this realistically serve as a data augmentation method in practice?
- The qualitative results (particularly in Figure 4) do not appear promising, raising doubts about the effectiveness of the proposed approach. For example, the face of the man is not properly generated, and the prompt specifies “red and green flower,” but no green flower is present in the output.

questions:
- Why did you choose image retrieval as the data augmentation method? What advantages does the proposed approach offer over generating synthetic datasets with generative models (e.g., recent models like Stable Diffusion)? In what aspects is retrieval beneficial?
- Could this approach benefit other downstream tasks, such as classification? Why is the method only applied to text-to-image generation in this paper?
- What does "ID" represent in Table 2? The paper does not provide an explanation. Additionally, ID=6 appears to perform well in Table 2; would increasing this value further improve performance?
- Why was the MS-COCO dataset selected, and what advantage does it demonstrate for the proposed approach? Considering the large scale of COCO (with millions of images), the paper states that “770,059 images are sufficient to cover the entire distribution of images in COCO.” How many images do you think would be necessary? Providing an estimate could improve the practicality of this method.
- Following up on the previous question, is this paper’s method specifically for augmenting smaller datasets, as it is mentioned that it cannot cover the full distribution of COCO? If so, could it be an effective data augmentation method in such cases?
</content>
</thread_object>

</thread>


Metareview: