CycleAug: Cycle-Consistent Visual Augmentation for Large Multimodal Models

Authors: Zhongrui Gui, Luoxin Ye, Wufei Ma, Zhao-Yang Wang, Ariel Lubonja, Daniel Khashabi, Alan Yuille, Jieneng Chen
Primary area: foundation or frontier models, including LLMs
Keywords: large multimodal models; synthetic data; data augmentation

Abstract:
Training multimodal large language models (MLLMs) requires high-quality image-question-answer (IQA) triplets, which are labour-intensive to curate and often lack diversity. We propose a novel data augmentation framework for visual instruction tuning that efficiently generates diverse synthetic images based on existing IQA anchor triplets. To ensure that the generated images align with their associated QA pairs, we propose CycleAug --- cycle-consistency visual augmentation which involves synthesizing images from text (text $\rightarrow$ image) and then performing a verification step to confirm that the answers derived from the synthetic images match the original answers (image $\rightarrow$ text), ensuring consistency across images, questions, and answers. By combining synthetic images with high-quality real data in the training phase, we demonstrate these synthetic triplets act as an implicit regularization, which improves the robustness of MLLMs and enables analogical reasoning. Extensive experiments show that our approach improves model performance on multiple visual question-answering benchmarks without additional real-world data. This work highlights the potential of leveraging visual foundational models to enhance visual instruction tuning in MLLMs.

Decision: reject
Reviewer scores: 6, 6, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1729067579567

thread_type:
official_review

<thread_object>
creator: reviewer_pwkd

<content>
summary:
This paper proposes a new visual augmentation strategy for improving the MLLM's generalization ability. Specifically, it introduces a cycle-consistency sampling process, which utilizes a three-step process to collect new images. In the first step, it generates images based on the caption/question; then in the second step, it measures the consistency between the new image and the original caption/question. At last, the positive samples will be added to the datasets for training. The authors also exploit implicit Bayesian to interpret the generation process. Experiments are conducted on thirteen benchmarks.

strengths:
1. The paper is easy to follow.

2. The motivation is reasonable.

weaknesses:
1. The novelty is weak. The consistency sampling is directly inspired by the existing cross-modal consistency alignment framework. Although the idea is straightforward, there is no special or novel technical design, weakening the novelty of the proposed method.

2. The implicit Bayesian interpretation is good, however, it is not the main contribution of this paper in my opinion. The authors just did a straightforward thing to generate synthetic data to boost the MLLMs, which is reasonable and easy to understand. Utilizing technique-irrelevant formulations and proof is not necessary.

3. The authors only consider the visual augmentation via the loop of text-image-text. What about generating synthetic texts via the loop of image-text-image to provide new aspects of the same image?

4. Experiments are unconvincing. Firstly, the authors only conduct the experiments on a single MLLM model. At least 3-4 MLLM models should be performed to evaluate the proposed method fairly. Secondly, as shown in Table 1, the proposed method fails to achieve the SOTA on FIVE important benchmarks. Thirdly, the authors should provide a detailed complexity and efficiency analysis of the cycle-consistency visual augmentation process.

5. The authors should provide discussions about the failure cases and their limitations. Not all generated images are positive to the MLLM training. A more in-depth analysis is also required to be provided.

questions:
1. The novelty is weak. The consistency sampling is directly inspired by the existing cross-modal consistency alignment framework. Although the idea is straightforward, there is no special or novel technical design, weakening the novelty of the proposed method.

2. The implicit Bayesian interpretation is good, however, it is not the main contribution of this paper in my opinion. The authors just did a straightforward thing to generate synthetic data to boost the MLLMs, which is reasonable and easy to understand. Utilizing technique-irrelevant formulations and proof is not necessary.

3. The authors only consider the visual augmentation via the loop of text-image-text. What about generating synthetic texts via the loop of image-text-image to provide new aspects of the same image?

4. Experiments are unconvincing. Firstly, the authors only conduct the experiments on a single MLLM model. At least 3-4 MLLM models should be performed to evaluate the proposed method fairly. Secondly, as shown in Table 1, the proposed method fails to achieve the SOTA on FIVE important benchmarks. Thirdly, the authors should provide a detailed complexity and efficiency analysis of the cycle-consistency visual augmentation process.

5. The authors should provide discussions about the failure cases and their limitations. Not all generated images are positive to the MLLM training. A more in-depth analysis is also required to be provided.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730184670476

thread_type:
official_review

<thread_object>
creator: reviewer_ajx2

<content>
summary:
This paper propose CYCLEAUG, a data augmentation framework for MLLM training. CYCLEAUG first generate new images based on the image caption of the source image-text data, then use an MLLM to check the consistency of the generated images with the original image, and filter the inconsistent ones. The authors use the filtered augmented data to train a LLaVA1.5 and show the effectiveness of the generated data in MLLM training.

strengths:
Data augmentation/generation for MLLM training is an important problem. The experiment results prove the effectiveness of the augmented data output by the proposed CYCLEAUG.

weaknesses:
1. The computational cost of the whole pipeline might be too heavy. 
2. The main experiment is not convincing. The authors use LLama3-LLaVA-NeXT-8B for image caption (line 388), and LLaVA-NeXT for for consistency judge (according to the citation in line 308). However, the augmented data are used to train a LLaVA-1.5-7B, which uses a way worse base LLM (Vicuna-7B) and a weaker structure than LLaVA-NeXT. The potential assumption is that there exists a better MLLM of similar scale compared to the MLLM we are going to train. If so, why can't we use the better MLLM directly?

questions:
1. What is the approximate computational cost for the data augmentation by CYCLEAUG in the main experiment?
2. Since the authors use better MLLM in the CYCLEAUG framework, is the performance gain from the CYCLEAUG framework itself, or from distilling knowledge from the better MLLM (LLaVA-NeXT) to the MLLM we are training (LLaVA-1.5-7B)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730254693840

thread_type:
official_review

<thread_object>
creator: reviewer_lt5h

<content>
summary:
This paper propose a novel data augmentation framework for visual instruction tuning.  The framework uses MLLM to generate detailed captions for the original images, and then generate synthetic image based on these captions. Question-answer pairs of the origin images are re-verified on the synthetic image to select the well-aligned synthetic image and question-answer triplets, thus achieving cycle-consistent visual augmentation. Experiments shows the framework can improve model performance on multiple visual question answering benchmarks without additional real data, confirming the effectiveness of the augmentation .

strengths:
(1) This paper propsed a novel data augmentation framework for visual instruction tuning. By leveraging cycle-consistency sampling, the framework can generate diverse synthetic images based on existing IQA anchor triplets.
(2) The proposed method is simple and intuitive. The authors validate their method across various benchmarks, demonstrating the effectiveness of the framework.
(3) The paper is generally well written.

weaknesses:
(1) Performance should be evaluated using more diverse backbones, which can better assess the robustness and effectiveness of the proposed framework.
(2) There are some works that have explored the cycle consistency in the VQA area. The approach presented in [1] involves Cycle-Consistency for Robust Visual Question Answering, wherein the model is trained not only to generate answers to given questions but also to formulate questions conditioned on specific answers. This methodology ensures that the answers predicted for the generated questions are consistent with the true answers corresponding to the original questions. This paper should clarify the relevance to the existing work [1].
[1] (CVPR 2019) Cycle-Consistency for Robust Visual Question Answering

questions:
1. When generating image captions,  integrating the question-answer pair into the prompt  may improve the proportion of consistent triplets in all generated images. I hope the authors can analyze the potential trade-offs of this suggestion, such as its impact on image diversity or whether it introduces other effects into the framework.
2. Related to weakness (1). What is the impact of different MLLM and diffusion models on experimental results? I hope the author can provide additional experiments and analysis on how various MLLM and diffusion models will affect the final performance on the benchmark, and whether non-diffusion generative model architectures have any impact on the effectiveness of the framework.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730455295559

thread_type:
official_review

<thread_object>
creator: reviewer_rjuf

<content>
summary:
This paper presents CYCLEAUG, a data augmentation framework for MLLMs that generates diverse, cycle-consistent synthetic image-question-answer triplets. By ensuring alignment between generated images and their corresponding QA pairs, CYCLEAUG enhances model robustness and generalization without requiring additional real-world data. Experiments verify the effectiveness of the proposed methods.

strengths:
1. The idea of using cycle-consistency for MLLMs for data augmentation is novel.  While data augmentation and synthetic data generation are established fields, CYCLEAUG advances these techniques by ensuring semantic alignment between generated images and their question-answer pairs, effectively addressing quality control and enhancing robustness in synthetic data generation.

2. The paper is well-organized and easy to follow.

3. The proposed method can be seamlessly integrated with existing visual instruction tuning frameworks, and its modular design holds potential for adaptation and extension to a broader range of multimodal tasks.

weaknesses:
1. The author does not provide a detailed discussion on the specific computational costs and resource consumption of the proposed method.  This could pose challenges for actual application environments with limited resources.

2. The paper presents a method for creating diverse synthetic images from multi-turn question-answer dialogues but lacks an in-depth exploration of alternative dialogue generation strategies, which could restrict the analysis of image diversity and relevance, impacting the overall model performance.

3. In addition to the direct sampling method, there are numerous data augmentation techniques applicable to the field of multimodal learning.  Incorporating comparative experiments would aid in a more comprehensive assessment of CYCLEAUG's performance and potentially reveal its advantages.

questions:
Refer to Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730462211538

thread_type:
official_review

<thread_object>
creator: reviewer_zgm1

<content>
summary:
The paper presents a data augmentation framework called CYCLEAUG, which generates multiple images paired with existing QA pairs using Stable Diffusion. Several post-processing mechanisms are employed to ensure consistency between the generated images and the anchor QA pairs. Experimental results demonstrate slight improved performance of MLLMs when incorporating this augmented data for VQA tasks.

strengths:
1. The paper is well written and organized.
2. The method is simple and easy to follow and replicate.

weaknesses:
1. The main issue is that the paper aims to address the limitations of images generated by SD, specifically "homogeneous content" and the "can not handle negation word" (lines 253-257), but the work does not have any concrete efforts to resolve these problems.
2. The method is not well-suited for text-rich VQA, as the current SD model struggles with generating characters.
3. The performance gain is quite limited; for example, in Table 3, the augmented images even have a negative impact on the VQAT dataset.
4. The paper would benefit from reporting the cost (e.g., FLOPS, latency) required to produce these images, as well as the proportion of images filtered out by cycle consistency sampling.

questions:
1. Besides using detailed LLM-generated captions to prompt SD, have you tried generating an image with SD where the starting point is a noisy anchor image? I believe this could also help maintain consistency with the QA pairs.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a data augmentation framework called CYCLEAUG, which generates multiple images paired with existing QA pairs using Stable Diffusion. Several post-processing mechanisms are employed to ensure consistency between the generated images and the anchor QA pairs. CYCLEAUG enhances model robustness and generalization without requiring additional real-world data. Experimental results demonstrate slightly improved performance of MLLMs when incorporating this augmented data for VQA tasks.

Strengths:
+ The paper is well-written and organized.
+ The method is simple and intuitive. 
+ The authors validate their method across various benchmarks, demonstrating the effectiveness of the framework.

Weaknesses:
+ The novelty of the proposed method is incremental.
+ The experimental setting is not reasonable, it uses a larger MLLM to generate training data, and utilizes it to fine-tune a smaller MLLM.
+ More data augmentation techniques on multimodal learning should be compared as baselines.
+ More ablation studies (such as more diverse backbones, training/inference computational cost) are needed.