Enhancing Large Vision Language Models with Self-Training on Image Comprehension

Authors: yihe  deng
pan  lu
fan  yin
ziniu  hu
sheng  shen
quanquan  gu
james  zou
kai-wei  chang
wei  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 00:03:31

Abstract: Large vision language models (LVLMs) integrate large language models (LLMs) with pre-trained vision encoders, thereby activating the perception capability of the model to understand image inputs for different queries and conduct subsequent reasoning. Improving this capability requires high-quality vision-language data, which is costly and labor-intensive to acquire. Self-training approaches have been effective in single-modal settings to alleviate the need for labeled data by leveraging model's own generation. However, effective self-training remains a challenge regarding the unique visual perception and reasoning capability of LVLMs. To address this, we introduce **S**elf-**T**raining on **I**mage **C**omprehension (**STIC**), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference dataset for image descriptions using unlabeled images. Preferred responses are generated through a step-by-step prompt, while dis-preferred responses are generated from either corrupted images or misleading prompts. To further self-improve reasoning on the extracted visual information, we let the model reuse a small portion of existing instruction-tuning data and append its self-generated image descriptions to the prompts. We validate the effectiveness of STIC across seven different benchmarks, demonstrating substantial performance gains of 4.0% on average while using 70% less supervised fine-tuning data than the current method. Further studies dive into various components of STIC and highlight its potential to leverage vast quantities of unlabeled images for self-training.

Main paper: 
 introduction: 
 In recent years, we have witnessed remarkable advancements in large language models (LLMs), such as GPT-4 (OpenAI, 2023a) and the LLaMA family (Touvron et al., 2023a,b) . The increasing importance of processing multimodal inputs, including images and text, has significantly driven progress in vision language models (Radford et al., 2021; Jia et al., 2021b; Goel et al., 2022) . Leveraging the powerful language understanding and generation capabilities of LLMs, researchers have advanced vision language models into large vision language models (LVLMs). This enhancement is achieved by integrating LLMs with image encoders (Radford et al., 2021; Li et al., 2023a) , which were pre-trained on large-scale image-text pairs to ensure alignment between the two domains. For instance, LLaVA (Liu et al., 2023b) integrates a vision encoder from CLIP (Radford et al., 2021) with the LLM Vicuna (Chiang et al., 2023b) , which is further fine-tuned on carefully constructed vision-language instructional datasets to activate the model's perception capability of capturing the vision information according to different queries. This recent development has substantially expanded the requirement for large-scale instruction fine-tuning data for LVLMs (Gao et al., 2023b; Bai et al., 2023; Chen et al., 2023b; Gao et al., 2024; Anthropic, 2024; McKinzie et al., 2024) . While LVLMs have shown promising results, a key challenge lies in the acquisition of high-quality fine-tuning data. Obtaining human-curated content at scale is often prohibitively expensive, especially for multi-modal data. Many recent studies resort to GPT-4V (OpenAI, 2023b) for generating or labeling high-quality vision-language fine-tuning data. However, this approach does not significantly reduce the cost (Liu et al., 2023b; Wu et al., 2024) . For instance, using GPT-4V to generate 6k image descriptions with 1k tokens per output would cost approximately $200. There remains a pressing need for cost-effective methods to gather fine-tuning data to further enhance LVLMs.To tackle the data acquisition bottleneck in multi-modality, we propose Self-Training on Image Comprehension (STIC). Our method is inspired by the recent success of self-training (Chen et al., 2024; Yuan et al., 2024; Fränken et al., 2024; Rosset et al., 2024) in LLMs, which leverages selfgenerated data to improve their downstream performance. However, different from the text-only domain, the unique vision modality of LVLMs introduces new challenges, as LVLMs must understand the input image content before reasoning and responding to any related textual queries about the image. Therefore, the proposed STIC approach is a novel two-stage self-training method that targets both image perception and reasoning over images and texts.The overall framework is summarized in Figure 2 . STIC specifically emphasizes the image comprehension self-training of LVLMs where the model generates its own preference dataset focused on image description. The self-generated dispreferred response is obtained by gathering model responses from either (1) prompts likely to elicit inaccurate responses or (2) corrupted images. The preferred responses are collected via a detailed prompt that guides the model through a step-by-step image description process. Figure 3 shows examples of such generated responses. During fine-tuning, we consider a DPO loss (Rafailov et al., 2023) with an additional regularized term explicitly emphasizing the preferred response. Lastly, we allow the model to self-improve its reasoning ability based on its own extracted image information by reusing a small amount of existing instruction fine-tuning data and appending its self-generated image descriptions to the prompts. We refer to this second stage as description-infused fine-tuning. Notably, our STIC approach does not require pre-labeled information of the images, which contrasts to the recent works that rely on such information for constructing vision-language preference data (Zhou et al., 2024) .To demonstrate the effectiveness of STIC, we conduct extensive experiments on seven vision-language benchmarks, including ScienceQA (Lu et al., 2022) , TextVQA (Singh et al., 2019) , ChartQA (Masry et al., 2022) , LLaVA-Bench (Liu et al., 2023a) , MMBench (Liu et al., 2023c) , MM-Vet (Yu et al., 2023) , and MathVista (Lu et al., 2024) . These benchmarks encompass scientific reasoning, math reasoning, optical character recognition (OCR), and conversation capabilities based on vision inputs, spanning various image sources such as natural, chart, and text-rich images. We employ LLaVA-v1.6 (Liu et al., 2024) as the primary base LVLM for our experiments and unitize 6k images from MSCOCO (Lin et al., 2014) to construct the image description preference data. As depicted in Figure 1 , STIC achieves consistent and significant performance improvements across these benchmarks, with an average accuracy gain of 4.0% over the base LVLM and a notable gain of 6.4% on ScienceQA. We also provide an example of the different responses from the original LVLM and STIC in Figure 1 , where STIC successfully identifies the key visual information and accurately reason with it. These results demonstrate the remarkable effectiveness of our image comprehension self-training approach in enhancing the visual perception capabilities of LVLMs. (1) Suggest and detail practical items or people that could logically inhabit the image's setting.(2) Augment the scene with details of potential events or items.(3) Incorporate elements that, though absent, would seamlessly fit into the context of the picture.Supervised fine-tuning DPO alignment fine-tuning w/ regularizationFigure 2 : Framework overview of STIC, a two-stage self-training algorithm focusing on the image comprehension capability of the LVLMs. In Stage 1, the base LVLM self-constructs its preference dataset for image description using well-designed prompts, poorly-designed prompts, and distorted images. In Stage 2, a small portion of the previously used SFT data is recycled and infused with model-generated image descriptions to further fine-tune the base LVLM.In addition, we explore the benefits of the various components of STIC. First, based on the descriptioninfused fine-tuning stage that enhances the model's reasoning ability with self-generated description, we show that further letting the model describe the image before responding to a query provides further improved reasoning capability. This results in a notable improvement of 2.8% on ScienceQA and 1.1% on average as compared to direct responses to queries (Table 2 ). Moreover, we examine the impact of self-generated dispreferred responses, from either bad prompting or image corruption. By excluding these dispreferred responses and conducting SFT solely with preferred responses, we observed a performance decrease of 2.5% on average across three benchmarks as compared to STIC with the preference data (Table 3 ). This highlights the importance of the negative samples in the self-constructed preference data by STIC. We also assess the scalability of our self-training scheme. By increasing the amount of generated preference data from 6k to 12k, we show an even further improvement of STIC from 1.9% to 3.1% on LLaVA-Bench (Figure 6 ). This result suggests that STIC holds considerable potential for leveraging vast quantities of unlabeled images for self-training, given the immense availability of unlabeled image data. Lastly, our t-SNE visualization analysis shows that the closer the distribution between MSCOCO images, which we use for preference data construction, to images in downstream tasks, the more likely STIC results in higher performance gains (Figure 7 ). The main contributions of this work are summarized as follows:• We propose STIC, a novel two-stage self-training approach for LVLMs that focuses on enhancing their image comprehension capabilities by generating a preference dataset for image description without relying on pre-labeled image information. • Through extensive experiments on seven diverse benchmarks, STIC demonstrates significant performance gains over the base LVLM, achieving an average accuracy gain of 4.0%. • We explore the benefits of various components of STIC, highlighting its potential to leverage vast quantities of unlabeled images for self-training.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The main claims in the abstract and introduction focus on the novel self-training method for image comprehension, description-infused fine-tuning, and the significant performance gains across several benchmarks. These contributions are accurately reflected and supported by the theoretical and experimental results presented in the paper.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Vision language models (VLMs). VLMs (Tan and Bansal, 2019; Li et al., 2019 Li et al., , 2020;; Kim et al., 2021; Wang et al., 2022b; Bao et al., 2022; Wang et al., 2022a; Alayrac et al., 2022; Li et al., 2023b; Chen et al., 2022; Jia et al., 2021a; Shen et al., 2022; Singh et al., 2021) , processing both images and text, are pivotal in a wide range of multimodal understanding and reasoning tasks, capable of generating text or encoding multimodal representations. These models have shown increasing proficiency in visual perception and textual reasoning, and are also capable of following complex instructions (OpenAI, 2023b; Team et al., 2023) . Recent advancements in the field have been propelled by the availability of open-source large language models (LLMs) (Touvron et al., 2023a,b; Jiang et al., 2023) and innovative image encoders (Radford et al., 2021; Li et al., 2022) . For instance, LLaVA (Liu et al., 2023b) combines a vision encoder from CLIP (Radford et al., 2021) with the Vicuna LLM (Chiang et al., 2023b) , and has been further fine-tuned on vision-language instruction- following datasets. The recent development of LVLMs has significantly expanded the scale and diversity of VL instruction-following data, including models such as LLaMA-Adapter-V2 (Gao et al., 2023b) , Qwen-VL (Bai et al., 2023) , InternVL (Chen et al., 2023b) , InstructBLIP (Dai et al., 2024) , SPHINX-X (Gao et al., 2024) , Claude-3 (Anthropic, 2024 ), MM1 (McKinzie et al., 2024 ), and Grok-1.5V (xAI, 2024) . In this work, we focus on enhancing the visual perception and mathmatical reasoning capabilities of LVLMs by efficiently aligning them with purely unsupervised data.Alignment fine-tuning. Subsequent to supervised fine-tuning (SFT), alignment fine-tuning has emerged as a prominent approach to further enhance the performance of LLMs by aligning them with human preferences (Ouyang et al., 2022; Casper et al., 2023) . Early efforts utilized on-policy reinforcement learning (RL) methods, such as proximal policy optimization (PPO) (Schulman et al., 2017) , to train a reward model based on preference data (Bai et al., 2022; Touvron et al., 2023a) .With the notable introduction of direct policy optimization (DPO) (Rafailov et al., 2023) , a new line of research emphasizes direct learning from human preferences without relying on an explicit reward model (Zhao et al., 2023; Azar et al., 2024; Ethayarajh et al., 2024; Zheng et al., 2024) . Another prominent direction is iterative preference fine-tuning, which has proven effective in enhancing model performance by repeatedly optimizing on newly generated preference pairs in each iteration (Adolphs et al., 2023; Xu et al., 2023; Xiong et al., 2023; Pang et al., 2024) . While substantial research has focused on alignment fine-tuning for LLMs, efforts to adapt these techniques for LVLMs have been significantly limited. Initial attempts involve constructing preference datasets using human-labeled data (Sun et al., 2023) or GPT-4 generations for fine-tuning with a DPO loss (Zhou et al., 2024) . Concurrent works (Pi et al., 2024; Zhou et al., 2024) begin to focus on generating preference dataset of LVLMs, while our method distinguishes itself with the unique preference prompt set. Self-training. Traditional self-supervised training schemes (He et al., 2019; Xie et al., 2020; Wei et al., 2020; Zoph et al., 2020; Sohn et al., 2020; Ghiasi et al., 2021; Kang et al., 2023) leverage trained models to generate labels for unlabeled data and incorporate these self-labeled examples into training as a form of data augmentation. These frameworks primarily focus on self-supervised representation learning of vision models. While both classical self-training schemes and our approach share the fundamental goal of effectively utilizing unlabeled data to enhance model performance, our method differs in its focus on vision LLMs, maintaining an LLM as the backbone architecture. Rather than optimizing image representations, our approach aims to generate synthetic data that enables the LLM to produce higher-quality responses to image queries.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations of the proposed method are thoroughly discussed in Appendix D, where the authors address potential challenges and constraints, including domain alignment issues and possible failure cases, providing a clear reflection on the robustness and scope of their approach.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem setting and preliminaries: 
 Notation. We use lower case letters and lower case bold face letters to denote scalars and vectors.We use the symbol p to represent the probability of an LLM's response. And we denote the sequence of tokens generated from the LLM before the t-th token as y <t = [y 1 , . . . , y t-1 ] for t > 1.Generative vision language models. LVLM typically consists of three components: a vision encoder f (•), a projection network g(•), and an LLM p θ parameterized by θ. The model processes an image input e along with a text sequence x = [x 1 , . . . , x n ] as the prompt to generate a corresponding response y = [y 1 , . . . , y m ], where x i and y j represent individual tokens from the vocabulary of the LLM. The image is therefore converted into visual tokens within the language token space by the vision encoder and the projection network, producingv = [v 1 , . . . , v k ] = f • g(e). The response y is then considered as a sample from the conditional probability distribution p θ (•|v, x). As a Markov process, the conditional probability distribution p θ (y|v, x) can be decomposed asp θ (y|v, x) = m j=1p θ (y j |v, x, y <j ).(3.1)Alignment fine-tuning. To improve LLM alignment with human preferences, RL fine-tuning (Bai et al., 2022; Gao et al., 2023a) is typically employed after supervised fine-tuning (SFT). This process involves a reward function r(x, y) for a given sequence pair (x, y). The more preferred response y is expected to result in a higher reward r(x, y), where the corresponding objective is to maximize the following:L(θ) = E x∼D,y∼p θ (•|x) [r(x, y)] -λE x∼D KL p θ (•|x)||p ref (•|x) , (3.2)where x ∼ D is sampled from a given distribution D and the KL regularization term prevents the new model p θ from deviating too much from the reference model p ref , with λ > 0 as the regularization parameter. Training the reward function is challenging in practice, but direct preference optimization (DPO) (Rafailov et al., 2023) simplifies this process using a predefined preference dataset S pref = (x (i) , yw , y(i) l ) i∈[N ], where y(i)w denotes the preferred response and y(i)l denotes the dispreferred response given the same prompt x (i) . The objective function is then formulated asEQUATIONwhere ℓ(t) = log(1 + exp(-t)) is the logistic loss function and θ ref is the reference model.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: Section 3 provides the problem setting and preliminaries, outlining the assumptions and theoretical foundations supporting our proposed approach. Theoretical assumptions are clearly stated and supported by relevant literature. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
our method: stic: 
 In this section, we introduce STIC, a two-stage self-training algorithm designed to enhance image comprehension capabilities. The first stage constructs its own preference dataset and the second stage infuses the used SFT data with self-generated image descriptions for fine-tuning. Figure 2 presents the general framework of STIC. Notably, unlike recent work on fine-tuning algorithms (Sun et al., 2023; Zhou et al., 2024) , STIC enables a base LVLM, such as LLaVA-v1.6 (Liu et al., 2024) , to evolve from self-generated image captions, thus eliminating the need for additional supervised and preference data from human annotators or advanced teacher models. This approach fundamentally enhances image comprehension abilities and can be seamlessly applied to a wide range of vision-language reasoning tasks. We summarize STIC in Algorithms 1 and 2, and detail the process below.Stage 1: Image comprehension self-training. The process begins with a self-constructed preference dataset from the base LVLM, which we aim to improve through fine-tuning. The dataset contains paired preference data for image descriptions:• Preferred response: Model-generated image descriptions derived from well-crafted prompts with explicit reasoning steps. • Dispreferred response: Model-generated descriptions resulting from either (1) corrupted image with low resolution or distorted color, or (2) "bad" prompts that cause the base model to hallucinate and describe elements that may not logically exist in the image. The self-constructed preference dataset is used for the first-stage self-training using DPO (Rafailov et al., 2023) with an additional regularization term to further emphasize the preferred response, controlled by the hyperparameter α. The regularized loss function is as follows:L(θ, θ ref ) = E (x,yw,y l )∼S ℓ λ log p θ (y w |x) p θ ref (y w |x) -λ log p θ (y l |x) p θ ref (y l |x) -α log p θ y w |x . (4.1)The use of an explicit loss term for positive examples can be similarly found in previous studies on contrastive learning (Chen et al., 2021; Chen and He, 2021; Chen et al., 2023a) and more recently in preference fine-tuning (Pang et al., 2024) . Specifically, Chen et al. (2023a) ∼ p θ0 (•|v (i) , x g ). if n < 0.5 then Randomly sample bad prompt x b ∼ P hallu . Generate dispreferred response y b ∼ p θ0 (•|v (i) , x b ). else Corrupt the image input v (i) b = h(v (i) ). Generate dispreferred response y b ∼ p θ0 (•|v (i) b , x). end if Add (x, y g , y b ) to D. end for Update θ 1 = argmin θ∈Θ (x,yg,y b )∈D ℓ λ log p θ (yg|x) p θ 0 (yg|x) -λ log p θ (y b |x) p θ 0 (y b |x) -α log p θ y g |x . Output: θ 1 .vision-language reasoning tasks, confirming that the enhanced visual comprehension ability directly benefits the model performance and its multimodal reasoning ability. Prompt design. Our prompt design for the well-crafted prompt focuses on quality and diversity. We use GPT-4 to generate and sample multiple initial prompts, which are then refined through human filtering. To ensure effectiveness, we test these prompts on MSCOCO samples, verifying their ability to produce well-structured and relevant responses from the model. The bad prompts are sampled from GPT-4 and, in contrast, designed to elicit inaccurate descriptions by setting up a slightly different task (describe objects that would logically exist in the image) for the model. We thus work under the assumption that responses generated from prompts that have differences in human preference lead to responses of the same preference with high probability. The key is that the discrepancy between good and bad prompts should result in pairs of responses that share the same implicit preference with high probability, which is sufficient for effective DPO training.Stage 2: Description-infused fine-tuning. In the second stage, we further fine-tune the self-trained LVLM to leverage self-generated high-quality image descriptions for instruction-following tasks, and thus help ground its reasoning ability on self-generated descriptions. To achieve this, we randomly select a small subset of data from the model's instruction fine-tuning dataset already used during SFT.We then infuse the instructions in this subset with model-generated image descriptions as follows:Image description: {model description} <original instruction>The original ground-truth completions remain unchanged. We then fine-tune the LVLM for one epoch on this small description-infused subset. This fine-tuning step ensures that the model effectively integrates visual information into its responses, thereby enhancing its ability to handle a variety of vision-language reasoning tasks. Describe and Respond. During inference, optionally, we can let the model self-augment its prompt for downstream vision-language reasoning tasks by describing the image before answering. Rather than generating an immediate response, we first elicit an image description, which is then concatenated with the original question to produce a more informed answer.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Section 5 and the Appendix B provide detailed information on the experimental setup, datasets, and evaluation metrics, ensuring reproducibility. Additionally, we commit to releasing the entire codebase, including data generation and fine-tuning procedures, to facilitate replication of our results. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments: 
 In this section, we present the experiment results of STIC across seven visual question answering (VQA) benchmarks. We demonstrate that STIC effectively and substantially improves LVLM's performance across different VQA tasks using a self-constructed preference dataset without labels.Algorithm 2 STIC (Stage 2: description-infused fine-tuning)Input: Instruction-following dataset already used for fine-tuning the target LVLM model:{v (i) , x (i) , y (i) } i∈[m] . Image description prompt set: P = {x (i) des } i∈[M1] . LVLM parameterized by θ 1 after self-training: p θ1 . Let description-infused dataset D des = {}. for i = 1, . . . m do Randomly sample x des ∼ {x (i) des } i∈[M ] . Generate model image description y des ∼ p θt (•|v (i) , x des ). Add [y des , x (i) ], y (i) to D des .end for Update θ = argmin θ∈Θ (x,y)∈Ddes ℓ log p θ y|x .Output: θ.Model and datasets. In experiments, we consider llava-v1.6-mistral-7b (Liu et al., 2023a) as our base model for self-training with model generated preference data. We additionally consider llava-v1.5-7b (Liu et al., 2023a) based on Vicuna-7B (Chiang et al., 2023b) to directly compare with one concurrent baseline POVID (Zhou et al., 2024) . A detailed discussion with POVID can be found in Appendix C.3. We follow the optimization process described in Section 4 for selftraining on image description in Algorithm 1 and description-infused fine-tuning in Algorithm 2 to achieve improved downstream performances. For the self-constructed preference dataset, we gather 6k unlabeled image data randomly sampled from the MSCOCO dataset (Lin et al., 2014) and specifically the train2014 split for its high-quality images popularly used for pre-training and fine-tuning. In the second stage, we randomly subsample 5k used instruction fine-tuning data from LLaVA's SFT data to construct the description-infused fine-tuning data with model-generated image descriptions. Lastly, we use low-rank adaptation (LoRA) fine-tuning (Hu et al., 2021) instead of full fine-tuning for efficient computation. We defer the detailed prompts and corruptions to Appendix B. Evaluation. We consider the widely used benchmarks for LVLM evaluation across different domains including: ScienceQA (Lu et al., 2022) , TextVQA (Singh et al., 2019) , ChartQA (Masry et al., 2022) , LLaVA-Bench (Liu et al., 2023a) , MMBench (Liu et al., 2023c) , MM-Vet (Yu et al., 2023) , and MathVista (Lu et al., 2024) . Specifically, ScienceQA focuses on scientific question answering and MathVista focuses on math reasoning with visual information. TextVQA consists of images with text-rich contents and ChartQA with visual charts. Lastly, LLaVA-Bench, MMBench, and MM-Vet are three recent benchmarks to comprehensively evaluate a model's capabilities in a wide range of tasks and evaluation criteria. We use the evaluation scripts provided by LLaVA (Liu et al., 2023a) to obtain the results for both our base model and after using STIC to ensure a fair comparison.Table 1 : Performance of STIC compared with the original LVLM model across vision-language reasoning tasks. For LLaVA-v1.5 (Vicuna 7B), we directly report the values in the paper of POVID, and "-" indicates an unreported value. We present our main results in Table 1 and detail the benchmark performances of STIC (LLaVA-v1.6 7B) on MMBench and MM-Vet in Figure 4 . In Appendix B, we present detailed results for MMBench in Table 7 and MM-Vet in Table 8 . Our results show a consistent and significant improvement of STIC over the original models (LLaVA-v1.5 and LLaVA-v1.6) across all seven datasets. This improvement is achieved using only self-constructed preference data and a small portion of the model's SFT dataset, which had already been used for fine-tuning the original model.On average, STIC improves LLaVA-v1.5 by 1.7%, increasing from 45.3% to 47.0%, and LLaVA-v1.6 by a notable score of 4.0%, increasing from 54.7% to 58.7%. The improvement is comprehensive, as detailed in Tables 7 and 8 , where STIC consistently enhances performance across all evaluation tasks and targets. Moreover, while STIC improves both LLaVA-v1.5 and LLaVA-v1.6, a more significant improvement is observed in the more advanced model, LLaVA-v1.6. This trend suggests that the extent of self-improvement could be correlated with the model's inherent capabilities. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The paper commits to providing open access to the code, including data generation and fine-tuning procedures. We discuss the open benchmarks used for evaluation in Section 5.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
ablation studies and discussions: 
 In this section, we conduct ablation studies on the key components of STIC to demonstrate their importance and effectiveness. Additionally, we examine the image distribution of our self-training data (MSCOCO) alongside the image distributions of benchmark datasets, revealing a positive correlation between performance gains and similarity in image distributions. Effectiveness of describe-and-respond (DaR) prompting. We assess the significance of the finetuning process in STIC by comparing it to the approach of directly allowing the base LVLM to describe an image and then respond to a query with a self-augmented prompt, which we refer to as the describe-and-respond (DaR) prompting method. As indicated in Table 2 , applying DaR to the base LVLM yields mixed results, with performance improvements on some datasets and degradation on others, resulting in an overall average drop of 2.3%. In contrast, when DaR is combined with the fine-tuning process of STIC, it leads to a further average enhancement of 1.1% and a notable 2.8% on ScienceQA. This demonstrates the synergistic effect of DaR and the fine-tuning process in STIC. Additionally, it is worth noting that STIC achieves a substantial average improvement of 2.8% even without the DaR prompting method, compared to the base LVLM. Table 2 : Test performance of STIC based on llava-v1.6-mistral-7b. We investigate the benefit of DaR as a prompting method toward the base LVLM model as compared to the benefit on STIC. Progression of stages. In Figure 5 , we illustrate the sequential improvement in performance of STIC on ScienceQA. While stage 1 focuses exclusively on enhancing the perception capabilities of the LVLM, it still notably improves performance on downstream VQA tasks. Building on the improved image comprehension achieved in stage 1, stage 2 introduces an enhanced reasoning process that utilizes the model's self-generated image descriptions and results in an even more significant gain. This enhancement further enables the model to self-augment its prompts with Describe and Respond (DaR), resulting in total the substantial performance gains of 6.4% observed. The role of dispreferred samples in STIC. To understand the importance of dispreferred samples in STIC, we conduct an ablation study using llava-v1.6-mistral-7b as the base LVLM. We remove the negative examples from the preference data and only use the positive samples for supervised fine-tuning (SFT), effectively creating an SFT version of STIC. Table 3 shows that omitting the dispreferred samples even leads to a performance drop of 0.6% on LLaVA-Bench, while failing to provide equally significant improvement as STIC with preference data. This highlights the crucial role of negative examples in aligning preferences and enabling the model to distinguish between high-quality and low-quality responses. By leveraging both positive and negative examples, STIC effectively improves the model's performance and generates more preferred outputs. Scaling law of STIC. We explore the scaling law of STIC by expanding the preference data in Stage 1. Using the LLaVA-Bench benchmark as a case study, we scale up the preference data from 6k to 12k MSCOCO images. As depicted in Figure 6 , there is an obvious gain on the LLaVA-Bench from 1.9% to 3.1%. This finding demonstrates that STIC can effectively leverage larger amounts of unlabeled image data and presents a cost-effective solution to the challenge of acquiring high-quality vision-language data. Correlation between image distribution and performance gains.To gain further insight into the effectiveness of STIC across different benchmarks, we conducted a t-SNE visualization analysis comparing the image distributions of MSCOCO, which we used for preference data construction, with those of four benchmarks: Sci-enceQA, TextVQA, MathVista, and ChartQA (Figure 7 ). Our analysis revealed a general trend: the greater the overlap between the MSCOCO image distribution and that of a benchmark, the higher the performance gain achieved by STIC on that benchmark. This observation held true for ScienceQA and TextVQA, which exhibited substantial distributional overlap with MSCOCO and yielded the highest performance gains of 6.4% and 4.9%, respectively. Conversely, MathVista, with its diverse image types and limited overlap with MSCOCO, saw a more modest gain of 2.4%. Interestingly, ChartQA was an outlier, achieving a high gain of 5.1% despite minimal overlap with MSCOCO, suggesting that the improved image comprehension from STIC played a fundamental role in understanding and reasoning about the charts. Detailed per-benchmark visualizations and discussions are provided in Appendix C.2. Diversity in image distribution. Based on the observation on the effect of image distribution in the final performance, we further utilize the Vision Flan dataset (VFLAN † ) for stage 1 image comprehension self-training. This dataset includes images from 191 diverse vision tasks, providing a broader spectrum of image types. We ensured a fair comparison by maintaining the same sample size (randomly sampled 6k images) and present the experimental results in Table 4 . As shown, our approach improves consistently across different datasets, demonstrating its robustness and adaptability.Notably, the increased diversity of VFLAN led to further improvements in STIC, suggesting the potential for even greater enhancement with better sets of unlabeled images. Scalability. To explore STIC's applicability to models with higher representation capacity, we conducted supplementary experiments using LLaVA-v1.6 (Vicuna-13B). Table 5 shows the detailed experiment results. We used the same images for STIC fine-tuning as in our experiments for LLaVA-v1.6 (Mistral-7B) to ensure fairness and the same set of hyperparameters. The improvements observed with LLaVA-v1.6 (Vicuna-13B) demonstrate that STIC is not only effective with smaller models but also scales well with larger or more capable LVLMs.Qualitative example. In Figure 8 This indicates that a significant portion of the plastic produced is not being properly managed and is ending up in the environment or landfills.The difference between the amount of plastic produced and the amount discarded is 302 million metric tonnes. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Section 5 provides details on the experimental setup, Section 6 covers the ablation studies, and Appendix B includes additional experimental details such as data splits, hyperparameters, and the type of optimizer used.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 We introduce Self-Training on Image Comprehension (STIC), a novel self-training approach designed to enhance the image comprehension capabilities of large vision language models (LVLMs). Our method leverages a two-stage self-training process, creating a preference dataset for image descriptions from unlabeled images and refining reasoning abilities through description-infused fine-tuning.Extensive experiments across seven vision-language benchmarks demonstrated significant performance improvements, with an average accuracy gain of 4.0%, while reducing the need for supervised fine-tuning data by 70%. Our findings underscore the potential of STIC to harness vast quantities of unlabeled images, offering a cost-effective solution for advancing LVLMs.The promising results demonstrated by STIC in enhancing the capabilities of 7B LVLMs suggest its potential applicability to larger models, such as those with 40B, and 100B parameters, if computational resources permit.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: On LLaVA-bench, where the evaluations are done with GPT-4, we reported the error bars in Figure 4 . However, most evaluations in this work are done using greedy decoding, resulting in a deterministic process that does not incur error bars. Therefore, the reported values of the baseline method that we compare to similarly did not report error bars. Lastly, due to the computational expense of fine-tuning large language models multiple times, we did not include error bars. However, we acknowledge this limitation and ensure that the reported results are consistent and reliable based on our experimental setup. Guidelines:• The answer NA means that the paper does not include experiments. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Appendix E discusses both the potential positive societal impacts, such as advancements in vision-language understanding and applications in education, and the potential negative societal impacts, including concerns about misuse and ethical considerations.The paper also addresses possible mitigation strategies for negative impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our approach does not involve releasing models or data, so there are no high-risk elements requiring safeguards. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The Self-improving language models. High-quality data, including human-crafted and advanced AI generated content, has been demonstrated to significantly enhance the performance of LLMs on various tasks (Josifoski et al., 2023; Taori et al., 2023; Chiang et al., 2023a; Li et al., 2023c) . Although, acquiring such high-quality data is often prohibitively expensive. To circumvent the costs associated with obtaining human-annotated or expertly curated data, researchers have shifted their focus to leveraging data generated by the target model itself, exploring ways of self-improvement (Chen et al., 2024; Yuan et al., 2024; Fränken et al., 2024; Rosset et al., 2024) . Recent studies have also emphasized the rephrasing capabilities of LLMs, which either enhance their own response quality (Deng et al., 2023; Prasad et al., 2023) or augment synthetic data for self-supervised fine-tuning (Kim et al., 2023) .To the best of our knowledge, our work is the first to explore the potential for self-improvement in LVLMs, specifically focusing on the vision modality and emphasizing the self-improvement of image comprehension capabilities.Perference data construction. We consider randomly sampling from the following "bad" prompts as a means of generating dis-preferred examples.• "Describe the image with imaginative objects that may exist in the scene."• "Enrich the description by adding hypothetical objects or characters that could be part of the scene." • "Suggest and detail practical items or people that could logically inhabit the image's setting."• "Incorporate elements that, though absent, would seamlessly fit into the context of the picture." • "Imagine and describe additional everyday objects or activities taking place just out of frame." • "Augment the scene with details of potential events or items that are plausible."• "Conceive of and detail natural elements, such as weather or animals, that could realistically enter the scene. Make the description affirmative." • "Invent and incorporate details of practical tools, vehicles, or gadgets that could be expected in a similar scenario." Given an input image, with 50% chance, we generate the dis-preferred response using the "bad" prompt. Otherwise, we generate the dis-preferred response with a corrupted image either from color jittering or lower resolution. To generate the preferred response, we use the following step-by-step prompt:• "Please provide a detailed description of the image, focusing on the following. Identify the main subjects (people, animals, objects) in the image and describe what they are doing. Describe the setting of the image. Is it indoors or outdoors? What kind of environment or location does it depict? What mood does the image convey? Are there any specific elements (such as lighting, weather, expressions) that contribute to this atmosphere? Describe the dominant colors and the overall composition. How do these elements affect the image's impact? Point out any details or symbols that might be relevant to understanding the image's meaning or context. If applicable, provide interpretations of what the image might represent or communicate."The prompts used as instructions in DPO are listed below:• "Illustrate the details of the picture."• "Summarize the visual content presented."• "Explain what is depicted in the photograph."• "Outline the key elements captured in the image."• "Detail the composition and subjects within the frame."• "Convey the atmosphere and mood represented in the snapshot."• "Interpret the scene shown in the image."• "Identify and describe the main focal points in the visual."In Stage 2, we prompt the LVLM with simple instructions like "Explain what is depicted in the photograph." to gather the image descriptions for Stage 2 fine-tuning.Task: Explain what is depicted in the photograph.The image depicts a social setting, likely a dinner party or a gathering of some sort. The main subjects in the image are a bottle of St-Germain liqueur and several wine glasses. The bottle is prominently placed in the foreground, with its label facing the viewer, while the wine glasses are scattered around the table, some filled with wine.\n\nThe setting appears to be indoors, possibly a dining room or a similar space, as indicated by the presence of a dining table. The table is covered with a white tablecloth, ......In the image, there's a sense of a social gathering or a celebration. Just out of frame, you might imagine several everyday objects and activities:\n\n1. A person is holding a glass of wine, toasting to a special occasion.\n2. The image depicts a young child, likely a boy, seated at a dining table with a birthday cake in front of him. The child appears to be in the process of blowing out the candles on the cake, which are lit and casting a warm glow on his face.The child is wearing a gray t-shirt and has short, light-colored hair.\n\nThe setting is indoors, with a wooden dining table and chairs visible in the background. The environment suggests a domestic, homey atmosphere, possibly a dining room or a kitchen area.\n\nThe mood of the image is joyful ......The image shows a close-up of a person, likely a child, looking at a cake with candles. The cake appears to be a birthday cake, and the child seems to be focused on it. The background is blurred, but it looks like an indoor setting with warm lighting. The focus is on the child and the cake, making the background indistinct. The image depicts a joyful and festive scene, likely a birthday party. The main subjects in the image are a young child, who appears to be the focus of the celebration, and several other children and adults. The child is seated at a table, engrossed in eating a piece of cake. ...... The setting is indoors, with the table and chairs arranged in a typical party layout. The table is covered with a tablecloth featuring a cartoon character, adding to the festive atmosphere. ...... The mood of the image is one of happiness and anticipation ......The image shows a child's birthday party. The child in the foreground is wearing a striped shirt and appears to be in the process of eating a piece of cake. The cake is green and has a \"Happy Birthday\" message on it, suggesting it's a birthday celebration. The child is using a plastic fork to eat the cake.\n\nIn the background, there are other children and adults, some of whom are also eating cake. ...... The overall atmosphere seems to be one of joy and celebration. Evaluation details. We use the evaluation scripts provided by LLaVA (Liu et al., 2023a ) for all evaluations. It is important to note that the potential new evaluation scripts and prompts used to report the results for LLaVA-v1.6 have not been released at the time of writing this manuscript. This may cause discrepancies in the evaluation results of the original model.Experiments were conducted on NVIDIA RTX A6000 GPU clusters. The entire self-training process for LLaVA v1.5 (7B) and LLaVA v1.6 (7B), using 6k image data and 5k reused instruction fine-tuning data, takes approximately 6 hours on 4 GPUs. The evaluation process of STIC for the benchmarks typically varies from 2 to 8 hours, mainly depending on the test set size. Based on the menu displayed in the image, the price for a funghi pizza is 6,90€, and the price for a prosciutto pizza is 7,90€. Therefore, the total cost for one funghi pizza and one prosciutto pizza would be 14,80€.Based on the menu prices, the total cost for one funghi pizza and one prosciutto pizza would be 14,00€. ScienceQA, TextVQA, MathVista, and ChartQA were chosen because there are at least 1, 000 images in the test set, providing enough data points for analysis.MSCOCO vs ScienceQA. The gain achieved by STIC on ScienceQA was 6.4%, the highest across all seven benchmarks. As evident from Figure 14 (a), the images in ScienceQA have substantial overlap with those in MSCOCO. This suggests that the image comprehension capabilities developed by STIC through self-training on MSCOCO translated effectively to the scientific reasoning tasks in ScienceQA.MSCOCO vs TextVQA. STIC yielded a gain of 4.9% on TextVQA, one of the higher gains observed across the benchmarks. Figure 14 (b) shows a significant overlap between the image distributions of TextVQA and MSCOCO. This indicates that the enhanced image understanding from self-training on MSCOCO proved beneficial for the text-based visual question answering tasks in TextVQA.MSCOCO vs MathVista. On MathVista, STIC achieved a gain of 2.4%, which, while still notable, was lower compared to other benchmarks. As Figure Figure 14 (c) illustrates, the images in MathVista have limited overlap with those in MSCOCO. Moreover, MathVista features diverse image types and mathematical reasoning tasks that pose additional challenges beyond image comprehension. These factors likely contributed to the more modest performance gain.MSCOCO vs ChartQA. STIC attained a high gain of 5.1% on ChartQA, despite the images in ChartQA having minimal overlap with those in MSCOCO, as shown in Figure 14 (d) . This seemingly contradictory result can be explained by considering that the improved image comprehension capabilities developed by STIC play a fundamental role in understanding and reasoning about the charts in ChartQA. Thus, even with limited distributional similarity, the enhanced perception skills proved valuable for this benchmark. We detail the differences between STIC and POVID. In POVID, the dispreferred response is generated either by adding Gaussian noise to the original image or by manually injecting hallucinations into the ground truth completion, using the labeled object information of the images. In contrast, STIC (1) specifically targets the image description task, (2) constructs preference datasets exclusively from unlabeled images using self-generated content for both preferred and dispreferred responses, (3) employs an automatic model generation process without manual injections or modifications, and (4) utilizes only a small portion of SFT data for instruction-following fine-tuning with uniquely infused model descriptions. Lastly, we compare the data types and scales used in POVID and STIC in Figure 15 .Table 9 presents an additional experiments using DaR to demonstrate prompt quality. We compared normal prompts from our main paper (e.g., "Illustrate the details of the picture.") with the hallucination prompts and well-curated prompts used for DPO pair generation. The results show an expected discrepancy in QA performance: hallucination prompts significantly decreased performance, while well-curated prompts maintained a decent performance. We also included results based on a prompt proposed by Llama-3 8B and filtered using the same restrictions. The performance difference between GPT-4 and Llama-3 8B prompts underscores the quality of GPT-4's proposals. Table 9 : Test performance of llava-v1.6-mistral-7b using various prompts with DaR. We evaluate prompt quality using DaR as a prompting method. DaR=None represents the original LVLM model's performance. Normal prompt refers to the simple prompt we used for DaR in our paper. GPT-4's well-curated prompt refers to the prompt we used for preferred response generation, and we include Mistral 7B's curated prompt for additional comparison. 79.1 (+2.1) 42.9 (+0.7) 60.9 (-2.8)While STIC demonstrates significant performance gains across a diverse set of benchmarks, there are still some limitations to be addressed in future work. First, STIC achieves a relatively small accuracy gain on MathVista compared to other benchmarks. This is likely because MathVista features various mathematical reasoning abilities across a wide range of image types, from elementary school to college level problems, which go beyond the scope of image comprehension. In contrast, STIC uses MSCOCO, containing only natural images, for constructing its preference data. To further improve performance on tasks like MathVista, future work could expand the source image types and generate task-specific image description data that better aligns with the target benchmark. Second, while our approach of using good prompts for positive examples and bad prompts or corrupted images for negative examples is straightforward and effective, this preference data may be insufficient for scenarios requiring nuanced understanding of image details. More sophisticated strategies for preference data construction could potentially further boost fine-tuning performance with the DPO loss. Additionally, our current method relies on a two-stage process of first training on the preference data and then performing description-infused fine-tuning. An interesting direction for future work would be to explore integrating these stages into a single end-to-end training process, which could potentially lead to even greater synergies and performance gains.Lastly, while we have demonstrated the scalability of STIC by doubling the amount of preference data, leading to further improvements, we have not yet explored the upper limits of this scaling. It is possible that even larger amounts of self-training data could lead to diminishing returns at some point.Characterizing the scaling behavior of STIC more fully is an important direction for future research. Despite these limitations, we believe STIC represents an important step towards leveraging the vast amounts of unlabeled image data available to enhance the image comprehension capabilities of large vision-language models in a cost-effective manner.The development of STIC, our self-training approach for enhancing the image comprehension capabilities of large vision language models (LVLMs), presents several potential societal impacts. Positively, our method can democratize access to advanced vision-language models by significantly reducing the cost and effort required for fine-tuning, making state-of-the-art LVLMs more accessible to researchers and organizations with limited resources. This can accelerate advancements in healthcare, education, and environmental monitoring, where improved image comprehension can lead to better diagnostic tools, personalized learning experiences, and more effective environmental protection measures. Additionally, by encouraging the reuse and recycling of existing data, STIC aligns with sustainable AI practices, promoting efficient use of computational and data resources. However, there are potential negative societal impacts that must be considered. Enhanced LVLM capabilities could be misused for generating disinformation, creating fake profiles, or conducting unauthorized surveillance, contributing to the spread of misinformation and erosion of public trust. Fairness considerations are crucial, as biased training data may lead to outputs that disproportionately impact specific groups, resulting in unfair treatment or discrimination. Privacy concerns also arise from using self-generated data, particularly if models are trained on sensitive or personal visual content without proper consent. To mitigate these risks, strategies such as gated release of models, robust fairness audits, diverse data inclusion, and enhanced transparency about the technology's limitations and risks are essential to ensure responsible use.† https://huggingface.co/datasets/Vision-Flan/vision-flan_191-task_1k 


Decision: This paper proposes a self-training approach to enhance the image comprehension abilities of VLMs by prompting the model with good and bad prompts to generate a preference datasets. The good prompts guide the model to reason step-by-step while the bad prompts are intentionally misleading or used with corrupted images. In addition, the image descriptions of good prompts are used to augment existing instruction tuning data. Experiments show the effectiveness of STIC across many established VLM benchmarks despite using a small amount of additional unlabeled data.

Many reviewers appreciate the novelty of the STIC approach to construct preference dataset (qFQt), its data efficiency using unlabeled images (s8oQ, qFQt, B7oX) and finally the significant performance gains (B7oX, s8oQ) on common benchmarks. The reviewers are happy with the authors' rebuttal and most of them are leaning to accept the paper with overall scores (6, 6, 6, 4). The WR reviewer's main concern was about the instruction-following ability of the model when good prompts are given. The author had provided a detailed response in their rebuttal and global response A1, which has addressed the concern adequately in my opinion.

Thus, I recommend to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper introduces Self-Training on Image Comprehension (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference dataset for image descriptions using unlabeled images. Preferred responses are generated through a step-by-step prompt, while dis-preferred responses are generated from either corrupted images or misleading prompts. To further self-improve reasoning on the extracted visual information, the model reuses a small portion of existing instruction-tuning data and appends its self-generated image descriptions to the prompts. Improvements in several benchmarks are reported.
strength_weakness: strength_weakness: 1. The experiments are conducted with 7B-level LLava 1.5 and 1.6. The method's scalability remains questionable.
questions: questions: What if we try the proposed method with smaller or bigger models or other ViT families (e.g., EVA-CLIP models)? With a higher representational capacity, will the model benefit more or less from self-training?
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: This paper proposes a two-stage method to enhance Large Vision Language Models (LVLMs) using unlabeled images. In the first stage, well-designed good and bad prompts are used to make the LVLM generate preferred and dis-preferred completions, respectively, conditioned on the unlabeled images (from COCO). Then, direct preference optimization (DPO) is used to fine-tune the LVLM using the generated preferred and dis-preferred completions. In the second stage, the fine-tuned model is used to generate descriptions for images in an instruction-tuning dataset (from LLaVA's data). The generated descriptions are inserted into the instruction-tuning data to fine-tune the model. The fine-tuned model shows significant improvement over the baseline (the LVLM before fine-tuning) on seven VLM benchmarks.
strength_weakness: strength_weakness: - The paper raises some questions unanswered.
  - The effect of the prompt set is less explored in the paper. It seems that the prompts play a crucial role in data construction. It is unclear how the authors designed the well-curated captioning prompt and the hallucination prompt set. Are there any principles behind the design? Especially for the well-curated captioning prompt that generates the preferred data, how do different design choices affect the final model performance in the evaluation?
  - Is the performance gain dependent on the MSCOCO data set? Do other image datasets (such as Flickr30k) work in stage 1? 
- It seems counterintuitive that fine-tuning an LVLM on MSCOCO will help improve its performance on science-related benchmarks like ScienceQA. It will help us better understand the mechanism by comparing the model generations in the benchmarks before and after using the proposed method. 
- The description of describe-and-respond (DaR) prompting is a bit unclear. I could not fully understand the setting.
questions: questions: - What is the difference between the image captioning prompt set $P$ and the well-curated captioning prompt (Algorithm 1)?
- What is the prompt $x$ used in DPO in stage 1?
- What is the model performance if only stage 1 is performed (i.e., without stage 2's SFT)?
limitations: limitations: See Weaknesses.
paper_summary: paper_summary: Summary:
The paper presents STIC, a method to enhance LVLM by reducing the need for labeled data. STIC generates image descriptions using unlabeled images and improves reasoning by reusing existing instruction-tuning data. It demonstrates performance gain across seven benchmarks, showing potential to effectively leverage vast quantities of unlabeled images for self-training.
strength_weakness: strength_weakness: Weakness:
1. Interesting idea about using different prompts to generate both good and bad captions. However, I wonder if there is any method needed to ensure the correctness of the prompts generated by the "step-by-step" prompt strategies. Based on my own observations, VLLM is not good at following instructions, which means the description will be even worse if the prompts are complicated. Have the authors observed similar problems?
questions: questions: Have the authors changed the default latex template? There should be no anonymous submission ID. And it seems that the authors gain more space because they remove the anonymous authors part.
limitations: limitations: Yes
paper_summary: paper_summary: This paper addresses the problem of acquiring high-quality fine-tuning data for large vision language models (LVLMs) with minimum human effort. The paper presents STIC (Self-Training on Image Comprehension). STIC contains two stages: image comprehension self-training and description-infused fine-tuning. LVLM first generates descriptions based on images and corrupted images and regards each sample as preferred and dis-preferred responses, respectively. The base LVLM is trained on the generated preference data leveraging the directed preference optimization (DPO) framework. Next, the LVLM is fine-tuned on instruction following data. Experimental results show that STIC outperforms the compared methods on diverse benchmarks.
strength_weakness: strength_weakness: - W1: The manuscript has limited soundness for several reasons. For example, the authors did not validate the effectiveness of description-infused fine-tuning. In Table 2, much of the performance gain comes from the prompting method (DaR) which is outside STIC.  
- W2: While there is a rich literature on existing self-training algorithms [1,2,3,4,5], the paper only discusses recent self-improvement systems, especially in the context of LLMs. This limited investigation leads readers to question what the technical contribution of STIC is compared with the existing line of semi-supervised learning (or self-training) research.
- W3: The paper experimented with a fairly small amount of unlabeled images (6k and 12k images). What and how much data should be used to maximize the performance has not been thoroughly investigated.


**References**

[1] Self-training with noisy student improves imagenet classification. Xie et al., CVPR 2020.

[2] Rethinking pre-training and self-training. Zoph et al., NeurIPS 2020.

[3] Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Sohn et al., NeurIPS 2020.

[4] Revisiting self-training for neural sequence generation. He et al., ICLR 2020.

[5] The dialog must go on: improving visual dialog via generative self-training. Kang et al., CVPR 2023.
questions: questions: - Q1: In Algorithm 2, why aren’t the model-generated description $\mathbf{y}_{\mathrm{des}}$ used for fine-tuning?
- Q2: What is the motivation of the stage 2 (Description-infused fine-tuning)? It is not directly related to the motivation of the paper (difficulty of obtaining high-quality fine-tuning data).
- Q3: How can we validate the effectiveness of description-infused fine-tuning? Did the authors check the performance of the method which did not use the model description?
- Q4: Table 1 just shows the performance with and without STIC. How much does each stage contribute to performance improvement?
- Q5: Table 2 shows that the describe-and-respond (DaR) prompting method improves overall performance. DaR was not mentioned until the section for experiments. Why did not the authors describe DaR in detail in the method section?
- Q6: Is there any reason why STIC randomly selects unlabeled images? Some self-training algorithms present the methods for selecting unlabeled data to learn.
- Q7: How can we guarantee that all samples generated by the LVLM inputting clean images are preferred responses?

**Typos**

Figure 1: SPIC to STIC
limitations: limitations: The limitations were adequately addressed.

Comments: title: After rebuttal
Thanks for the response
Invitation for discussion
Invitation for discussion
Invitation for discussion
Invitation for discussion
comment: I keep my original score and tend to accept this paper.
Dear reviewer s8oQ,

Thank you again for taking the time to review our paper. We appreciate your acknowledgment of our work's soundness, presentation, and contribution. 

In response to the feedback received, we have included extensive additional experiments as well as detailed clarifications in our rebuttal. While other reviewers have responded positively to our rebuttal, we hope to adequately address your concern as well. We would greatly appreciate your attention to our specific response regarding your question on prompt design, as we believe it addresses your primary concern. It would be great to also give us the opportunity to provide more details and further improve our work. Thank you!
Thank you for replying and providing encouraging feedback on our rebuttal! We are glad that we addressed most of the concerns.
I thank the authors for providing detailed responses to my concerns and questions. I read the responses from the authors and the other reviewers' comments as well, and most of my concerns are addressed. I will raise my initial rating accordingly. Thanks again!
Thank you for replying! We greatly appreciate your positive feedback on our rebuttal.
Thank you for the detailed responses. My questions are addressed. I will raise my rating to 6.
Dear reviewer uhYN, 

Thank you again for your detailed and valuable feedback to this paper. We hope that our responses and clarifications have adequately addressed your questions and concerns. Specifically, 

1. We provided detailed clarifications on the results of Table 2 and explanations on DaR (Global A3). 
2. We added a discussion paragraph on the similarities and differences with the mentioned related works, emphasizing our specific focus. These works will be incorporated into our revision. 
3. We conducted additional experiments scaling up the data to 30k and explained the data efficiency of our method. 
4. We provided clarifications and explanations for each of your specific questions. 

We hope these responses adequately address your concerns. If you have any further questions about our rebuttal, we'd be happy to provide additional information or clarification. Thank you once again for your time and efforts!
Dear reviewer s8oQ, 

Thank you again for your time and feedback to this paper. We hope that our clarifications have adequately addressed your questions. Specifically, we provided detailed explanations into the prompts of our method and further discussion in Global A1. We are following up to inquire if there are any remaining questions. We are more than happy to further discuss and provide clarifications. 

Thank you once again for your time and efforts!
Dear reviewer B7oX, 

Thank you again for your support and constructive comments. We hope that our responses and clarifications have adequately addressed your questions and concerns. 

Specifically, we provided detailed explanations toward the prompt design (Global A1) and DaR (Global A3). We further added an experiment on unlabeled images from a different distribution than MSCOCO, where we observed that a more diverse unlabeled image data can provide better improvement for STIC. Regarding W2 and Q3, we provided specific generation examples before and after STIC, as well as its stage-wise performance on ScienceQA. 

We would like to inquire if there are any questions about our rebuttal, for which we're happy to provide additional information and further clarifications. Thank you once again for your time and efforts on this paper!
Dear reviewer qFQt, 

Thank you again for your strong support and constructive feedback. We hope that our responses have adequately addressed your questions. Specifically, we added the additional experiment on scaling up model sizes for STIC and provided discussion on further extending it to various models. We would like to inquire if you have further questions regarding our rebuttal. We are more than happy to discuss any remaining questions and provide additional details. 

Thank you once again for your time and efforts on this paper!